SEP-2106: traceability YAML + JSON Schema 2020-12 conformance checks (#295)
* chore: add SEP-2106 requirement-traceability YAML (tools JSON Schema 2020-12) * Drop structured-content text-fallback row from SEP-2106 That SHOULD predates SEP-2106 (pre-existing context line in tools.mdx); 2106 only relaxes structuredContent to allow non-object values (permissive). Point spec_url at the SEP doc security section where the remaining obligations live. * feat: SEP-2106 JSON Schema 2020-12 vocabulary preservation checks Extend the json-schema-2020-12 server scenario (previously SEP-1613 only) with three checks that verify the broader JSON Schema 2020-12 vocabulary SEP-2106 permits in a tool inputSchema survives tools/list rather than being stripped: composition (allOf/anyOf), conditional (if/then/else), and reference ($anchor) keywords. - everything-server: add these keywords to json_schema_2020_12_tool's raw schema - add a negative-test server (sep-2106-stripped-schema.ts) plus a negative test asserting the checks emit FAILURE when the vocabulary is stripped * feat: SEP-2106 client scenario for network $ref dereferencing Add json-schema-ref-no-deref, a client-suite scenario that advertises a tool whose inputSchema contains a $ref pointing at a canary URL hosted by the scenario. The sep-2106-no-network-ref-deref check fails if the client fetches the canary while processing the schema, covering the SEP-2106 requirement that implementations MUST NOT automatically dereference network-URI $ref values. - register the scenario in the client scenarios list - everything-client: handle the scenario with the basic connect+listTools flow - tests: compliant client passes, a deliberately dereferencing client is caught, and a client that never lists tools cannot pass * Address review findings in SEP-2106 scenarios - json-schema-ref-deref: make getChecks() idempotent (fresh array per call, matching the http-standard-headers convention) and guard the async /mcp handler like the example servers do - json-schema-2020-12: include the SEP-2106 keywords ($anchor, allOf/anyOf, if/then/else) in the description's example inputSchema so a server that copies it verbatim passes the new checks - sep-2106-stripped-schema: default to port 3007 to match negative.test.ts and avoid sharing a default with sep-2549-no-caching-hints * Soft-gate SEP-2106 keyword checks on the negotiated protocol version The json-schema-2020-12 scenario is introducedIn 2025-11-25 (SEP-1613) but the three SEP-2106 keyword-preservation checks test draft-era behavior. Rather than failing servers that only negotiated 2025-11-25, gate the failure on the wire protocolVersion from the initialize handshake: preserved keywords are SUCCESS at any version, stripped keywords are FAILURE only when the server negotiated the draft version and SKIPPED otherwise. The published SDK does not advertise the draft version yet, so the negative test asserts SKIPPED for the stripped-schema server; the FAILURE branch is covered by unit tests on sep2106KeywordCheckStatus.
This commit is contained in:
Paul Carleton
GitHub
parent
f12baed382
commit
ed1ff0ca03
2 changed files with 680 additions and 9 deletions
|
|
@ -1,25 +1,57 @@
|
|||
/**
|
||||
* JSON Schema 2020-12 conformance test scenario (SEP-1613)
|
||||
* JSON Schema 2020-12 conformance test scenario (SEP-1613, SEP-2106)
|
||||
*
|
||||
* Validates that MCP servers correctly preserve JSON Schema 2020-12 keywords
|
||||
* in tool definitions, ensuring implementations don't strip $schema, $defs,
|
||||
* or additionalProperties fields.
|
||||
* or additionalProperties fields (SEP-1613).
|
||||
*
|
||||
* SEP-2106 broadened inputSchema to permit the full JSON Schema 2020-12
|
||||
* vocabulary alongside the required root `type: "object"`. This scenario also
|
||||
* verifies that composition (allOf/anyOf), conditional (if/then/else), and
|
||||
* reference ($anchor) keywords survive tools/list rather than being stripped.
|
||||
*/
|
||||
|
||||
import { ClientScenario, ConformanceCheck } from '../../types.js';
|
||||
import {
|
||||
CheckStatus,
|
||||
ClientScenario,
|
||||
ConformanceCheck,
|
||||
DRAFT_PROTOCOL_VERSION
|
||||
} from '../../types.js';
|
||||
import { connectToServer } from './client-helper.js';
|
||||
|
||||
const EXPECTED_TOOL_NAME = 'json_schema_2020_12_tool';
|
||||
const EXPECTED_SCHEMA_DIALECT = 'https://json-schema.org/draft/2020-12/schema';
|
||||
|
||||
/**
|
||||
* Soft version gate for the SEP-2106 keyword-preservation checks.
|
||||
*
|
||||
* Preserving the broader JSON Schema 2020-12 vocabulary is good behavior at
|
||||
* any protocol version, so a server that preserves the keywords gets SUCCESS
|
||||
* regardless of what it negotiated. Stripping them is only a conformance
|
||||
* FAILURE once the server claims a protocol version that includes SEP-2106
|
||||
* (the draft); a server that only negotiated an earlier dated version is not
|
||||
* yet subject to the requirement, so the check is SKIPPED rather than failed.
|
||||
*/
|
||||
export function sep2106KeywordCheckStatus(
|
||||
preserved: boolean,
|
||||
negotiatedProtocolVersion: string | undefined
|
||||
): CheckStatus {
|
||||
if (preserved) {
|
||||
return 'SUCCESS';
|
||||
}
|
||||
return negotiatedProtocolVersion === DRAFT_PROTOCOL_VERSION
|
||||
? 'FAILURE'
|
||||
: 'SKIPPED';
|
||||
}
|
||||
|
||||
export class JsonSchema2020_12Scenario implements ClientScenario {
|
||||
name = 'json-schema-2020-12';
|
||||
readonly source = { introducedIn: '2025-11-25' } as const;
|
||||
description = `Validates JSON Schema 2020-12 keyword preservation (SEP-1613).
|
||||
description = `Validates JSON Schema 2020-12 keyword preservation (SEP-1613, SEP-2106).
|
||||
|
||||
**Server Implementation Requirements:**
|
||||
|
||||
Implement tool \`${EXPECTED_TOOL_NAME}\` with inputSchema containing JSON Schema 2020-12 features:
|
||||
Implement tool \`${EXPECTED_TOOL_NAME}\` with inputSchema containing JSON Schema 2020-12 features, including the broader vocabulary permitted by SEP-2106 (an \`$anchor\` inside \`$defs\`, composition keywords \`allOf\`/\`anyOf\`, and conditional keywords \`if\`/\`then\`/\`else\`):
|
||||
|
||||
\`\`\`json
|
||||
{
|
||||
|
|
@ -30,6 +62,7 @@ Implement tool \`${EXPECTED_TOOL_NAME}\` with inputSchema containing JSON Schema
|
|||
"type": "object",
|
||||
"$defs": {
|
||||
"address": {
|
||||
"$anchor": "addressDef",
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"street": { "type": "string" },
|
||||
|
|
@ -39,14 +72,26 @@ Implement tool \`${EXPECTED_TOOL_NAME}\` with inputSchema containing JSON Schema
|
|||
},
|
||||
"properties": {
|
||||
"name": { "type": "string" },
|
||||
"address": { "$ref": "#/$defs/address" }
|
||||
"address": { "$ref": "#/$defs/address" },
|
||||
"contactMethod": { "type": "string", "enum": ["phone", "email"] },
|
||||
"phone": { "type": "string" },
|
||||
"email": { "type": "string" }
|
||||
},
|
||||
"allOf": [
|
||||
{ "anyOf": [{ "required": ["phone"] }, { "required": ["email"] }] }
|
||||
],
|
||||
"if": {
|
||||
"properties": { "contactMethod": { "const": "phone" } },
|
||||
"required": ["contactMethod"]
|
||||
},
|
||||
"then": { "required": ["phone"] },
|
||||
"else": { "required": ["email"] },
|
||||
"additionalProperties": false
|
||||
}
|
||||
}
|
||||
\`\`\`
|
||||
|
||||
**Verification**: The test verifies that \`$schema\`, \`$defs\`, and \`additionalProperties\` are preserved in the tool listing response.`;
|
||||
**Verification**: The test verifies that \`$schema\`, \`$defs\`, and \`additionalProperties\` are preserved (SEP-1613), and that the composition (\`allOf\`/\`anyOf\`), conditional (\`if\`/\`then\`/\`else\`), and \`$anchor\` keywords are preserved (SEP-2106), in the tool listing response.`;
|
||||
|
||||
async run(serverUrl: string): Promise<ConformanceCheck[]> {
|
||||
const checks: ConformanceCheck[] = [];
|
||||
|
|
@ -56,9 +101,20 @@ Implement tool \`${EXPECTED_TOOL_NAME}\` with inputSchema containing JSON Schema
|
|||
url: 'https://github.com/modelcontextprotocol/specification/pull/655'
|
||||
}
|
||||
];
|
||||
const sep2106References = [
|
||||
{
|
||||
id: 'SEP-2106',
|
||||
url: 'https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2106'
|
||||
}
|
||||
];
|
||||
|
||||
try {
|
||||
const connection = await connectToServer(serverUrl);
|
||||
// Negotiated wire protocolVersion from the initialize handshake; used to
|
||||
// soft-gate the SEP-2106 checks below.
|
||||
const negotiatedVersion = (
|
||||
connection.client.transport as { protocolVersion?: string } | undefined
|
||||
)?.protocolVersion;
|
||||
const result = await connection.client.listTools();
|
||||
|
||||
// Find the test tool
|
||||
|
|
@ -164,6 +220,110 @@ Implement tool \`${EXPECTED_TOOL_NAME}\` with inputSchema containing JSON Schema
|
|||
}
|
||||
});
|
||||
|
||||
// SEP-2106: the full JSON Schema 2020-12 vocabulary is permitted in
|
||||
// inputSchema and must survive tools/list rather than being stripped to
|
||||
// properties/required. These checks are soft-gated on the negotiated
|
||||
// protocol version (see sep2106KeywordCheckStatus): stripping the
|
||||
// keywords is only a FAILURE for servers that negotiated the draft
|
||||
// protocol version, and SKIPPED otherwise.
|
||||
const skippedSuffix = ` (server negotiated protocol version ${negotiatedVersion ?? 'unknown'}; SEP-2106 applies from ${DRAFT_PROTOCOL_VERSION})`;
|
||||
|
||||
// Check 5: composition keywords (allOf / anyOf) preserved
|
||||
const allOf = inputSchema['allOf'];
|
||||
const hasAllOf = Array.isArray(allOf) && allOf.length > 0;
|
||||
const hasNestedAnyOf =
|
||||
Array.isArray(allOf) &&
|
||||
allOf.some(
|
||||
(sub) =>
|
||||
typeof sub === 'object' &&
|
||||
sub !== null &&
|
||||
Array.isArray((sub as Record<string, unknown>)['anyOf'])
|
||||
);
|
||||
const compositionPreserved = hasAllOf && hasNestedAnyOf;
|
||||
const compositionStatus = sep2106KeywordCheckStatus(
|
||||
compositionPreserved,
|
||||
negotiatedVersion
|
||||
);
|
||||
|
||||
checks.push({
|
||||
id: 'sep-2106-composition-keywords-preserved',
|
||||
name: 'JsonSchema2020_12CompositionKeywords',
|
||||
description:
|
||||
'inputSchema composition keywords (allOf/anyOf) preserved (SEP-2106)',
|
||||
status: compositionStatus,
|
||||
timestamp: new Date().toISOString(),
|
||||
errorMessage: compositionPreserved
|
||||
? undefined
|
||||
: (!hasAllOf
|
||||
? 'allOf keyword missing from inputSchema - composition keyword was likely stripped'
|
||||
: 'nested anyOf missing from allOf - composition keyword was likely stripped') +
|
||||
(compositionStatus === 'SKIPPED' ? skippedSuffix : ''),
|
||||
specReferences: sep2106References,
|
||||
details: {
|
||||
hasAllOf,
|
||||
hasNestedAnyOf,
|
||||
negotiatedProtocolVersion: negotiatedVersion
|
||||
}
|
||||
});
|
||||
|
||||
// Check 6: conditional keywords (if / then / else) preserved
|
||||
const hasIf = 'if' in inputSchema;
|
||||
const hasThen = 'then' in inputSchema;
|
||||
const hasElse = 'else' in inputSchema;
|
||||
const conditionalPreserved = hasIf && hasThen && hasElse;
|
||||
const conditionalStatus = sep2106KeywordCheckStatus(
|
||||
conditionalPreserved,
|
||||
negotiatedVersion
|
||||
);
|
||||
|
||||
checks.push({
|
||||
id: 'sep-2106-conditional-keywords-preserved',
|
||||
name: 'JsonSchema2020_12ConditionalKeywords',
|
||||
description:
|
||||
'inputSchema conditional keywords (if/then/else) preserved (SEP-2106)',
|
||||
status: conditionalStatus,
|
||||
timestamp: new Date().toISOString(),
|
||||
errorMessage: conditionalPreserved
|
||||
? undefined
|
||||
: `Conditional keywords missing (if=${hasIf}, then=${hasThen}, else=${hasElse}) - likely stripped` +
|
||||
(conditionalStatus === 'SKIPPED' ? skippedSuffix : ''),
|
||||
specReferences: sep2106References,
|
||||
details: {
|
||||
hasIf,
|
||||
hasThen,
|
||||
hasElse,
|
||||
negotiatedProtocolVersion: negotiatedVersion
|
||||
}
|
||||
});
|
||||
|
||||
// Check 7: reference keyword ($anchor) preserved within $defs.address
|
||||
const addressDef = defsValue?.['address'] as
|
||||
| Record<string, unknown>
|
||||
| undefined;
|
||||
const hasAnchor = !!addressDef && '$anchor' in addressDef;
|
||||
const anchorStatus = sep2106KeywordCheckStatus(
|
||||
hasAnchor,
|
||||
negotiatedVersion
|
||||
);
|
||||
|
||||
checks.push({
|
||||
id: 'sep-2106-anchor-keyword-preserved',
|
||||
name: 'JsonSchema2020_12AnchorKeyword',
|
||||
description:
|
||||
'inputSchema reference keyword ($anchor) preserved in $defs (SEP-2106)',
|
||||
status: anchorStatus,
|
||||
timestamp: new Date().toISOString(),
|
||||
errorMessage: hasAnchor
|
||||
? undefined
|
||||
: '$anchor missing from $defs.address - reference keyword was likely stripped' +
|
||||
(anchorStatus === 'SKIPPED' ? skippedSuffix : ''),
|
||||
specReferences: sep2106References,
|
||||
details: {
|
||||
hasAnchor,
|
||||
anchorValue: addressDef?.['$anchor']
|
||||
}
|
||||
});
|
||||
|
||||
await connection.close();
|
||||
} catch (error) {
|
||||
checks.push({
|
||||
|
|
|
|||
|
|
@ -3,6 +3,11 @@ import path from 'path';
|
|||
import { DNSRebindingProtectionScenario } from './dns-rebinding';
|
||||
import { ResourcesNotFoundErrorScenario } from './resources';
|
||||
import { CachingScenario } from './caching';
|
||||
import {
|
||||
JsonSchema2020_12Scenario,
|
||||
sep2106KeywordCheckStatus
|
||||
} from './json-schema-2020-12';
|
||||
import { DRAFT_PROTOCOL_VERSION, LATEST_SPEC_VERSION } from '../../types';
|
||||
|
||||
function startServer(scriptPath: string, port: number): Promise<ChildProcess> {
|
||||
return new Promise((resolve, reject) => {
|
||||
|
|
@ -148,4 +153,75 @@ describe('Server scenario negative tests', () => {
|
|||
}
|
||||
}, 15000);
|
||||
});
|
||||
|
||||
describe('json-schema-2020-12 (SEP-2106)', () => {
|
||||
let serverProcess: ChildProcess | null = null;
|
||||
const PORT = 3007;
|
||||
|
||||
beforeAll(async () => {
|
||||
serverProcess = await startServer(
|
||||
path.join(
|
||||
process.cwd(),
|
||||
'examples/servers/typescript/sep-2106-stripped-schema.ts'
|
||||
),
|
||||
PORT
|
||||
);
|
||||
}, 35000);
|
||||
|
||||
afterAll(async () => {
|
||||
await stopServer(serverProcess);
|
||||
});
|
||||
|
||||
it('flags SEP-2106 keyword-preservation checks against a server that strips the 2020-12 vocabulary', async () => {
|
||||
const scenario = new JsonSchema2020_12Scenario();
|
||||
const checks = await scenario.run(`http://localhost:${PORT}/mcp`);
|
||||
|
||||
// The tool is still advertised, so it must be found...
|
||||
const found = checks.find(
|
||||
(c) => c.id === 'json-schema-2020-12-tool-found'
|
||||
);
|
||||
expect(found?.status).toBe('SUCCESS');
|
||||
|
||||
// ...but the stripped 2020-12 keywords must be flagged. The stripped
|
||||
// server negotiates 2025-11-25 (the published SDK does not advertise the
|
||||
// draft protocol version), so the soft version gate reports SKIPPED
|
||||
// rather than FAILURE — see sep2106KeywordCheckStatus.
|
||||
const composition = checks.find(
|
||||
(c) => c.id === 'sep-2106-composition-keywords-preserved'
|
||||
);
|
||||
expect(composition?.status).toBe('SKIPPED');
|
||||
|
||||
const conditional = checks.find(
|
||||
(c) => c.id === 'sep-2106-conditional-keywords-preserved'
|
||||
);
|
||||
expect(conditional?.status).toBe('SKIPPED');
|
||||
|
||||
const anchor = checks.find(
|
||||
(c) => c.id === 'sep-2106-anchor-keyword-preserved'
|
||||
);
|
||||
expect(anchor?.status).toBe('SKIPPED');
|
||||
}, 10000);
|
||||
});
|
||||
|
||||
describe('sep2106KeywordCheckStatus (soft version gate)', () => {
|
||||
it('passes preserved keywords at any negotiated version', () => {
|
||||
expect(sep2106KeywordCheckStatus(true, DRAFT_PROTOCOL_VERSION)).toBe(
|
||||
'SUCCESS'
|
||||
);
|
||||
expect(sep2106KeywordCheckStatus(true, LATEST_SPEC_VERSION)).toBe(
|
||||
'SUCCESS'
|
||||
);
|
||||
expect(sep2106KeywordCheckStatus(true, undefined)).toBe('SUCCESS');
|
||||
});
|
||||
|
||||
it('fails stripped keywords only when the server negotiated the draft version', () => {
|
||||
expect(sep2106KeywordCheckStatus(false, DRAFT_PROTOCOL_VERSION)).toBe(
|
||||
'FAILURE'
|
||||
);
|
||||
expect(sep2106KeywordCheckStatus(false, LATEST_SPEC_VERSION)).toBe(
|
||||
'SKIPPED'
|
||||
);
|
||||
expect(sep2106KeywordCheckStatus(false, undefined)).toBe('SKIPPED');
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
|
|||
Loading…
Reference in a new issue