89c2c0452e* Vendor schema/{version}/schema.json alongside schema.ts
sync-schema now copies the spec repo's JSON Schema for each version into
src/spec-types/{version}.schema.json (same pinned SHA recorded in SOURCE:
modelcontextprotocol@f817239f4d6b1efff2c4dfc2f7af85c985d73076). These are
the inputs for runtime wire-message validation. Vendored verbatim, so they
join schema.ts in .prettierignore.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JHNrpU3Vr6Q1rPZiji7zLY
* Validate all wire messages against the per-version spec JSON schema
Every JSON-RPC message the harness sends or receives is now validated
(ajv) against the vendored schema.json for the run's spec version, at the
wire choke points: sendStatelessRequest (outbound request, JSON response,
SSE events), connectStateful (request/result/notifications), and both mock
servers (inbound client requests, outbound responses/handler results).
Beyond the JSONRPCMessage envelope, messages are validated against the
most specific definition the schema has: typed requests/notifications by
method const, typed error responses by error.code const (e.g. -32021 ->
MissingRequiredClientCapabilityError, the #376 hallucination), and typed
results via the XxxRequest -> XxxResult pair, with SEP-2322
InputRequiredResult discriminated on resultType.
Violations are recorded per scenario and surfaced by the runners as a
synthetic wire-schema-valid check (FAILURE when the implementation under
test sent an invalid message) plus a wire-schema-harness-error check when
the invalid message was harness-authored. A vitest-wide afterEach guard
fails any self-test that recorded violations, so hallucinated fixtures
can't ship (#376, #271). Opt-outs are explicit and greppable: per-call
skipValidation for deliberately malformed traffic, takeWireViolations()
drains in tests that exercise deliberately broken fixtures.
Offenders the new validation caught, now fixed:
- everything-server stateless path omitted resultType (and the caching
hints on server/discover) required by the draft schema
- sep-2164-empty-contents fixture omitted resultType/ttlMs/cacheScope
(its intended violation, empty contents, is schema-valid prose-only)
- connection/mock-server unit-test fixtures used schema-invalid stubs
(tools/call without name, ListToolsResult without required members,
Tool without inputSchema, ProgressNotification without progressToken,
ElicitRequest without message/requestedSchema)
- http-standard-headers test mocks returned bare { tools: [] } results
- input-required-result-validate-input deliberately sends malformed
inputResponses; now marked skipValidation
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JHNrpU3Vr6Q1rPZiji7zLY
* Validate the stateful wire at the SDK transport, close review-panel gaps
Address the review-panel findings on the wire-schema validation PR:
- Stateful wire: hook the SDK transport inside connectToServer, wrapping
send/onmessage so every raw wire message in both directions is
validated for the scenario's spec version -- including the initialize
handshake, server->client requests (elicitation/sampling), and the
harness's responses to them. Request ids are correlated per direction
so responses validate against their typed result definition (e.g. an
outbound ElicitResult). The reconstructed stand-ins in
connectStateful (fake id: 0 envelopes, Zod-reparsed results) are
removed; the hook sees the real bytes. The four scenarios that call
connectToServer directly (elicitation-defaults, elicitation-enums,
tools, lifecycle) now pass ctx.specVersion and get validation for
free; the runner surfaces their violations through the existing
recorder. Known remaining gap (documented in the module doc): the
client-auth scenarios' bespoke express mock is not instrumented.
- skipValidation now skips only the request direction; the
implementation's response to a deliberately malformed request is
still validated, with a narrow carve-out tolerating the JSON-RPC 2.0
id: null an error response must carry when the request could not be
processed (the MCP schema's RequestId forbids null). Covered by new
unit tests pinning both the still-validated response and the
narrowness of the carve-out.
- negative-mrtr.test.ts: the file-wide drain now asserts every recorded
violation is implementation-origin (the deliberately broken fixture)
and throws on any harness-origin violation.
- everything-server: hoist the cacheable-methods set to module scope
and replace the per-request res.json monkey-patch with an explicit
sendStatelessJson(res, method, payload) helper applied at each
dispatch site.
- Pin the extracted dispatch maps (methodDefs/errorDefs, e.g.
-32021 -> MissingRequiredClientCapabilityError under the draft) per
spec version in wire-schema.test.ts, so a schema sync that breaks the
const extraction fails loudly instead of degrading to envelope-only
validation. Also note the batch branch's requestMethod forwarding
limitation.
Offender the transport hook flushed out: the everything-server's
test_elicitation_sep1034_defaults tool sent a number-typed elicitation
field with default: 95.5, which the 2025-11-25 schema forbids
(NumberSchema.default is typed integer there; widened to number only in
the draft). The fixture and the scenario's expected default are now the
integer-valued 95, with comments explaining why.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JHNrpU3Vr6Q1rPZiji7zLY
* Condense comments to three lines max
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JHNrpU3Vr6Q1rPZiji7zLY
* Re-vendor spec schemas at pinned SHA 71e30695
Regenerate src/spec-types/*.schema.json via npm run sync-schema at the
SHA already recorded in SOURCE. The previously vendored draft schema
predated spec PR #3002, so it still required DiscoverResult.serverInfo
at the top level; the mock server now correctly reports serverInfo via
result _meta, which the stale schema rejected.
* Scope the wire-violation recorder per scenario for parallel suite runs
Client --suite runs execute scenarios concurrently over what was a
module-global violation recorder, so scenarios could steal or wipe each
other's recorded violations. Give each suite scenario its own recorder
via AsyncLocalStorage; code outside a scope (single-scenario runs, the
sequential server suite, vitest hooks) falls back to the global
recorder. Also condense a four-line comment in connection/stateful.ts
to three lines.
* Patch the 2025-11-25 NumberSchema generator bug as a load-time erratum
The released 2025-11-25 schema.json types NumberSchema
minimum/maximum/default as integer, contradicting its own schema.ts
(default?: number) — a typescript-json-schema artifact, fixed for draft
in modelcontextprotocol#2710 and proposed for 2025-11-25 in
modelcontextprotocol#3139. Patch the schema at load time so SDK
fixtures with fractional number defaults (all tier-1 SDKs send 95.5)
are not flagged; restore the sep-1034 scenario and example-server
fixture to 95.5. Delete the erratum when #3139 lands and the schemas
are re-vendored.
Also scope the sequential server-suite loop with withWireRecorder so a
connection leaked by a failed scenario cannot attribute late traffic to
the next scenario.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
64 lines
1.9 KiB
JSON
64 lines
1.9 KiB
JSON
{
|
|
"name": "@modelcontextprotocol/conformance",
|
|
"version": "0.2.0-alpha.10",
|
|
"type": "module",
|
|
"license": "MIT",
|
|
"author": "Anthropic, PBC (https://anthropic.com)",
|
|
"homepage": "https://modelcontextprotocol.io",
|
|
"bugs": "https://github.com/modelcontextprotocol/conformance/issues",
|
|
"repository": {
|
|
"type": "git",
|
|
"url": "git+https://github.com/modelcontextprotocol/conformance.git"
|
|
},
|
|
"scripts": {
|
|
"start": "tsx src/index.ts",
|
|
"test": "vitest run",
|
|
"test:watch": "vitest",
|
|
"build": "tsdown src/index.ts --minify --clean --target node20 --no-fixed-extension",
|
|
"lint": "eslint src/ examples/ && prettier --check .",
|
|
"lint:fix": "eslint src/ examples/ --fix && prettier --write .",
|
|
"lint:fix_check": "npm run lint:fix && git diff --exit-code --quiet",
|
|
"tier-check": "node dist/index.js tier-check",
|
|
"traceability": "tsx src/index.ts traceability",
|
|
"sync-schema": "tsx scripts/sync-schema.ts",
|
|
"check": "npm run typecheck && npm run lint",
|
|
"typecheck": "tsgo --noEmit",
|
|
"prepack": "npm run build",
|
|
"prepare": "npm run build"
|
|
},
|
|
"files": [
|
|
"dist"
|
|
],
|
|
"bin": {
|
|
"conformance": "dist/index.js"
|
|
},
|
|
"devDependencies": {
|
|
"@eslint/js": "^9.39.4",
|
|
"@types/express": "^5.0.3",
|
|
"@types/node": "^25.6.0",
|
|
"@typescript/native-preview": "^7.0.0-dev.20260424.1",
|
|
"cors": "^2.8.5",
|
|
"eslint": "^9.39.4",
|
|
"eslint-config-prettier": "^10.1.8",
|
|
"lefthook": "^2.1.6",
|
|
"prettier": "^3.8.3",
|
|
"tsdown": "^0.21.10",
|
|
"tsx": "^4.7.0",
|
|
"typescript": "^5.5.4",
|
|
"typescript-eslint": "^8.59.0",
|
|
"vitest": "^4.1.5"
|
|
},
|
|
"dependencies": {
|
|
"@modelcontextprotocol/sdk": "^1.29.0",
|
|
"@octokit/rest": "^22.0.0",
|
|
"ajv": "^8.20.0",
|
|
"ajv-formats": "^3.0.1",
|
|
"commander": "^14.0.2",
|
|
"eventsource-parser": "^3.0.8",
|
|
"express": "^5.1.0",
|
|
"jose": "^6.1.2",
|
|
"undici": "^7.25.0",
|
|
"yaml": "^2.8.2",
|
|
"zod": "^4.3.6"
|
|
}
|
|
}
|