There is no way to answer "which scenarios must my implementation pass
to conform to spec revision X". The suite accumulates scenarios
continuously, so what an implementation is measured against keeps
growing after the revision shipped, and it can drop below 100% while
standing still. --spec-version does not answer it: it filters on which
revision a scenario targets, not on when the scenario became a
requirement.
Add requirements/<revision>.yaml naming the scenarios a revision
requires, and --requirements <revision> on the client, server, list and
sdk commands to run exactly that set. The revision also fixes the wire
version its scenarios speak: the dated revisions through 2025-11-25 use
the stateful initialize handshake and 2026-07-28 is stateless with
per-request _meta, so a scenario belonging to two revisions runs once
under each and one run does not cover the other. Passing no flag keeps
today's behaviour of running everything.
tier-check gains --sdk <name[@ref]> / --sdk-path <dir>: for each claimed
revision it resolves that revision's build, server invocation and client
command from the sdk-runner's KNOWN_SDKS specOverrides, manages the
server, runs both legs with cwd = the SDK checkout, and merges. Every
claimed revision must pass for Tier 1. This is what makes multi-revision
tiering correct for SDKs whose wire era is fixed per server process
(go-sdk only speaks >= 2026-07-28 when constructed stateless) or per
endpoint (csharp-sdk serves it at /stateless): one
--conformance-server-url cannot describe them, and their config already
does. The single-URL mode remains for SDKs not in the config.
Ship two sets. 2026-07-28 is a snapshot of 0.2.0-alpha.10, published the
day before that revision shipped. 2025-11-25 is a reconstruction from
the same anchor and its header says so: the release current on its ship
date predates spec-version awareness entirely. not_scored entries run
and are reported with a reason (extension per SEP-1730, or
added-after-release) but never affect a pass rate or the exit code, and
--expected-failures judges scored scenarios only. Reports label every
entry with its revision, and JSON passed/failed/total describe the
scored set so they always agree with pass_rate.
tier-check's exit code now gates on the machine-checkable half: nonzero
when a scored scenario fails or a leg could not be measured. A child run
that cannot happen at all (a set naming a renamed scenario, an
unreachable server) reports as not measured rather than as 0% against
the implementation. Conformance runs to completion before the GitHub
checks, so an API failure can no longer interrupt a leg and orphan a
managed server, and the legs spawn their child asynchronously: the
previous synchronous exec blocked the event loop, the parent stopped
draining the managed server's log pipes, and a chatty server (rust-sdk's
per-request tracing) froze on a full pipe and was scored as failing.
Driving every tier-1 SDK through the new path also fixed three configs:
csharp-sdk's server now launches from its build-output directory so
ASP.NET finds appsettings.json (its AllowedHosts filter, which
dns-rebinding-protection tests, silently never loaded from the repo
root); rust-sdk's STATELESS=1 override is removed because its CI runs
one default server for both revisions and the forced mode fails its
SEP-2575 scenarios; python-sdk's build mirrors its CI's two targeted
syncs because --all-packages is unbuildable on main. withManagedServer
now waits for its child to exit before returning, so consecutive runs on
one port cannot race a dying predecessor.
* feat(sdk-runner): add rust-sdk to KNOWN_SDKS
The mcp-conformance package's fixture bins follow the harness contract
directly (URL argv + MCP_CONFORMANCE_SCENARIO env), and the server
selects the SEP-2575 stateless lifecycle via STATELESS=1. Requires the
repo's pinned Rust toolchain (rustup installs it on demand).
* feat(sdk-runner): per-spec-version config overlays (specOverrides)
Several KNOWN_SDKS entries needed a different invocation when targeting
2026-07-28 — go-sdk's -stateless flag, csharp-sdk's /stateless URL,
rust-sdk's STATELESS=1 env, python-sdk's per-revision baseline — and each
carried a comment telling the caller which flag to paste. Encode those
variants in the entry instead: specOverrides maps a canonical spec
version to a partial config that is merged over the base entry when a
run targets that version (--spec-version, after 'draft' alias
resolution, or the entry's own specVersion default). Explicit CLI flags
still win.
Keys are validated against the known spec versions at schema level so a
'draft' key or a typo'd date fails loudly instead of silently never
matching.
* style: prettier fixes in sdk-runner tests
* refactor(sdk-runner): reuse isSpecVersion for specOverrides key validation
Reuses the canonical spec-version list from #421 instead of keeping a
separate copy; SPEC_VERSION_TIMELINE is now exported so the validation
message can enumerate the valid keys.
Follow-ups from the #399 review:
- Stateless mock: validate error replies with a placeholder id so the
JSON-RPC-mandated id:null is not recorded as a harness violation, and
skip inbound validation when express left the body unparsed (wrong or
missing Content-Type) — both were false positives blamed on the wrong
party. Same unparsed-body guard in the stateful mock.
- Stateful mock: validate client traffic against the protocol version
the client declares via MCP-Protocol-Version, falling back to
2025-03-26 when the header is absent (it only exists from 2025-06-18,
whose spec says to assume 2025-03-26), instead of always using the
run version.
- sdk-client: validate batches once as a whole array instead of also
per element, which double-recorded every violation.
- Make specVersion required on connectToServer/connectStateful/
createServerStateful; add isSpecVersion to types.ts.
- Extend the NumberSchema erratum to 2025-06-18 (same generator bug on
minimum/maximum, modelcontextprotocol#3139) and add a tripwire test
that forces erratum deletion once the schemas are re-vendored.
- Document the wire-schema-valid / wire-schema-harness-error check IDs
in the README.
* feat(sdk-runner): expand KNOWN_SDKS to all tier 1 SDKs
- go-sdk: point at the relocated conformance/ fixtures (the old
examples/server/conformance path is gone on main), add the
everything-client fixture that now exists upstream, and wire
conformance/baseline.yml as the expected-failures baseline.
- python-sdk: new entry for main, which targets the 2026-07-28
revision (same uv workspace layout as the v1.x entry).
- csharp-sdk: new entry driving the ConformanceClient /
ConformanceServer fixtures via the .NET 10 SDK. The client takes the
scenario as argv[0], so the command bridges MCP_CONFORMANCE_SCENARIO
into argv; the server exposes the SEP-2575 stateless lifecycle at
/stateless.
* docs: match the sdk override example's lead-in to its content
The example now shows field overrides on a built-in entry rather than a
one-off ref, so the introducing sentence shouldn't promise a ref
override.
A baseline entry names a scenario, but a scenario is many checks:
server-stateless is 29. Excusing the one or two a spec change breaks
means baselining the whole scenario, which stops enforcing the rest.
An entry may now be '<scenario>:<check-id>', in which case every failing
check in that scenario is judged on its own. Bare scenario entries are
unchanged, and YAML parses '- a:b' as a plain string, so every existing
baseline keeps working untouched.
loadExpectedFailures now owns the entry grammar and returns parsed
BaselineEntry values, so evaluateBaseline only matches and cannot throw.
Rejecting a bad entry and interpreting a good one no longer live in
separate layers.
A check id matches all of its occurrences, since ids repeat within a run;
they collapse to one verdict, most-severe first. Absent and SKIPPED are
tolerated, matching how the runner already treats them as green.
Move collapseDuplicateChecks to src/checks/collapse.ts with its tests:
expected-failures.ts needs the reducer and cannot import dpop.ts's auth
server stack. No behavior change.
Reject two configs that previously coerced to an entry matching nothing:
a mapping ('- scenario: check-id', with a space) and an empty list item.
* chore: sync spec-types to modelcontextprotocol@0168c57f
Refreshes the vendored schema copies via npm run sync-schema. The draft
schema now declares LATEST_PROTOCOL_VERSION = "2026-07-28" (previously
the DRAFT-2026-v1 placeholder) and picks up the latest draft doc-comment
and type updates. The dated schemas are unchanged.
* fix: update the draft wire version to 2026-07-28
The spec repo's draft schema now declares
LATEST_PROTOCOL_VERSION = "2026-07-28", but the harness still asserted
the old DRAFT-2026-v1 placeholder, so an SDK that sends the version the
draft spec actually declares cannot pass the draft suite.
- bump DRAFT_PROTOCOL_VERSION in src/types.ts to 2026-07-28
- update the hard-coded literals in tests, example servers, and README
- add src/types.test.ts pinning DRAFT_PROTOCOL_VERSION to the vendored
draft schema's LATEST_PROTOCOL_VERSION so the two cannot drift, and
asserting the draft identifier stays distinct from the dated versions
* feat: vendor spec schema types per-version
Copies schema/{version}/schema.ts from the modelcontextprotocol spec
repo into src/spec-types/{version}.ts so the conformance suite can type
against draft spec versions before any SDK ships them.
npm run sync-schema -- <ref> refreshes the copies and records the spec
commit in src/spec-types/SOURCE.
* feat: add Connection abstraction and RunContext
Connection encapsulates how the conformance suite talks to a
server-under-test for a given spec version:
- connectStateful: 2025-x lifecycle. Thin adapter over the SDK Client
(initialize handshake, session id, SSE handled by the SDK).
- connectStateless: 2026-x lifecycle (SEP-2575). Raw fetch with
per-request _meta + MCP-Protocol-Version header. Decoupled from the
SDK so the suite can test draft spec versions before the SDK
implements them.
connectFor(specVersion) picks the implementation. RunContext bundles
serverUrl, specVersion and a bound connect() for the runner to hand to
each scenario.
Nothing uses this yet; wiring follows in the next commit.
* refactor: thread RunContext through ClientScenario.run
ClientScenario.run(serverUrl) becomes run(ctx: RunContext). The runner
builds the context from --spec-version and the server URL; scenarios
destructure ctx.serverUrl and otherwise behave identically.
No scenario uses ctx.connect() yet, so behaviour is unchanged: 214/214
tests pass, all-scenarios.test.ts still drives the everything-server
fixture exactly as before.
Test files use a testContext(url) helper to construct a RunContext.
The authorization-server scenario list is retyped to
ClientScenarioForAuthorizationServer since those scenarios test an
OAuth server, not an MCP server, and keep run(serverUrl).
* refactor: migrate server scenarios to ctx.connect() + conn.request()
22 carry-forward and lifecycle scenarios now go through the Connection
abstraction instead of connectToServer + SDK Client:
- tools.ts (8): list, call x6, with-progress
- prompts.ts (5)
- resources.ts (7): list, read x3, subscribe x2, not-found
- utils.ts (3): completion, ping, set-level
- json-schema-2020-12.ts, caching.ts, http-standard-headers.ts
Result types come from spec-types/{introducedIn}. Same scenario code
now passes under both --spec-version 2025-11-25 (SDK-backed stateful)
and --spec-version draft (raw stateless).
ToolsCallSampling/Elicitation/WithLogging and elicitation-* keep
connectToServer (need setRequestHandler/setLoggingLevel SDK surface);
they are tagged removedIn: DRAFT in the next commit. stateless.ts and
input-required-result.ts keep their sendRpc helper; migrating those is
deferred to the DRAFT-scenario coherence pass.
* fix: normalize Connection error to JsonRpcError; clean up RunContext import
connectStateful now catches the SDK's McpError and rethrows as
JsonRpcError so scenarios always see the same error class regardless
of which Connection impl ran. ResourcesNotFoundError uses
instanceof JsonRpcError instead of duck-typing.
types.ts uses a normal top-level import for RunContext instead of an
inline import() type.
* fix(dns-rebinding): use version-appropriate probe body
The 'valid Host accepted' check was sending an initialize body, which
a 2026 server returns 404 for. Probe body is now picked from
ctx.specVersion: initialize for the stateful lifecycle, server/discover
with _meta for the stateless lifecycle. The Host/Origin rejection check
is unchanged since rejection happens before body parsing.
* feat(everything-server): route stateless carry-forward methods to McpServer
Adds an in-memory dispatch client connected to the same McpServer the
stateful path uses. Stateless requests for tools/call, resources/*,
prompts/get and completion/complete that fall through the MRTR-specific
handlers are routed to it, so the fixture serves the carry-forward
scenarios under --spec-version draft without duplicating ~500 lines of
tool/resource/prompt registrations.
tools/list now merges the McpServer's tool list with the MRTR-only
stubs so json-schema-2020-12 finds its tool.
draft suite against the fixture: 36/39 (was 13/39). Remaining 3 are
fixture-side SEP gaps (no SSE forwarding for progress, no SEP-2549
ttlMs, no SEP-2243 Mcp-Method validation in the stateless path).
* refactor(connection): drop unused RequestOptions; move sdk-client; add tests
Addresses self-review findings on the new connection module:
- RequestOptions.handlers/.meta and ServerRequestHandler removed: zero
callers (the scenarios that motivated them are deferred). They can be
reintroduced when something actually uses them.
- scenarios/server/client-helper.ts moved to connection/sdk-client.ts so
the connection module no longer depends on the scenarios module.
- connectStateless: handle CRLF SSE event separators; throw a useful
error for non-JSON/non-SSE responses instead of a JSON parse error.
- JSONRPCNotification consistently imported from spec-types/2025-11-25.
- New connection.test.ts (8 tests) covering connectFor selection,
_meta injection, error mapping, SSE LF/CRLF parsing, and the
server-request-on-stream rejection.
* fix: address bughunt findings (response.ok check; targetVersion naming)
- connectStateless: throw on non-2xx responses that lack a JSON-RPC
error envelope (e.g. gateway/framework error JSON), matching the
stateful path's behavior. Previously such a response would return
undefined as the result.
- json-schema-2020-12: rename negotiatedVersion to targetVersion and
reword the skip message and details field. The value is ctx.specVersion
(the run's --spec-version), not a negotiated version; the old wording
was misleading. Drop dead 'unknown' fallback (specVersion is required)
and the corresponding undefined test cases.
* fix(sse-multiple-streams): keep scenario in draft; version-aware request scaffolding
Concurrent POSTs each answered with JSON or their own SSE stream are
core transport behavior in the draft spec too, so removedIn was wrong
for this scenario. Only the request scaffolding is version-specific:
stateful runs keep the initialize/session-id setup, draft runs send
_meta + the MCP-Protocol-Version header on each request instead. The
two checks are unchanged.
* feat: add MockServer abstraction and ScenarioContext
MockServer encapsulates the lifecycle scaffold a client-conformance
scenario presents to the client-under-test:
- createServerStateful: 2025-x lifecycle. SDK Server +
StreamableHTTPServerTransport (sessionless mode); the SDK handles the
initialize handshake.
- createServerStateless: 2026-x lifecycle (SEP-2575). Raw express app
that validates _meta + MCP-Protocol-Version on every request, serves
server/discover, routes other methods to the supplied handlers.
createServerFor(specVersion) picks the implementation. ScenarioContext
bundles specVersion and a bound createServer() for the runner to hand
to each scenario.
This is the client-conformance mirror of src/connection (PR #318).
Nothing uses it yet; wiring follows in the next commit.
* refactor: thread ScenarioContext through Scenario.start()
Scenario.start() becomes start(ctx: ScenarioContext). The runner builds
the context from --spec-version (defaulting to LATEST_SPEC_VERSION) and
passes it through; scenarios receive it as _ctx and otherwise behave
identically.
No scenario uses ctx.createServer() yet, so behaviour is unchanged:
231/231 tests pass.
Test files use a testScenarioContext() helper. The runner already
threads MCP_CONFORMANCE_PROTOCOL_VERSION to the spawned client process,
so the fixture-side env wiring is unchanged.
* refactor: migrate tools_call to ctx.createServer; tag 2025-only client scenarios
ToolsCallScenario now goes through ctx.createServer() instead of an
inline express + SDK Server build. Same handlers, same checks; the
assertion now reads from srv.recorded so it works regardless of which
lifecycle scaffold the runner picked.
initialize, sse-retry, and elicitation-defaults are tagged
removedIn: DRAFT (initialize/GET-SSE/SSE-embedded-elicitation are gone
in the 2026 lifecycle; the MRTR sibling for elicitation-defaults is a
follow-up).
spec-version.test.ts: the 'draft is a superset of latest' invariant no
longer holds once removedIn: DRAFT exists; the test now asserts that
any scenario in latest-but-not-draft is explicitly removedIn.
* feat(auth): make createServer helper version-aware via ScenarioContext
The auth helper now takes ctx: ScenarioContext as its first argument
and branches on ctx.specVersion inside the /mcp route: the stateful
path (SDK Server + StreamableHTTPServerTransport) is unchanged; under
the draft version a raw stateless handler validates _meta + the
MCP-Protocol-Version header, serves server/discover, and routes the
same tools/list and tools/call responses.
The PRM endpoint, bearer-auth middleware, and request logger sit above
the branch and are version-independent.
All 25 call sites across the 12 auth scenario files pass ctx through;
ServerLifecycle and the express.Application return type are unchanged
so stop()/getChecks() are untouched.
Deviation from the MockServer wrapper approach: keeping the helper's
return type as express.Application avoids restructuring 25 call sites'
ServerLifecycle handling in this PR. Folding the auth seam onto
ctx.createServer() fully is a follow-up once the lifecycle ownership
moves into MockServer.
* feat(everything-client): pick stateless requester by MCP_CONFORMANCE_PROTOCOL_VERSION
Adds a statelessRequest(serverUrl, method, params) helper that POSTs
with _meta + MCP-Protocol-Version (the SEP-2575 lifecycle), shimming
around the SDK Client not yet supporting stateless mode. The
runRequestMetadataClient handler's meta constants are extracted to share
with the helper.
runBasicClient (initialize, tools_call, json-schema-ref-no-deref) now
branches on MCP_CONFORMANCE_PROTOCOL_VERSION: for the draft version it
uses statelessRequest to call tools/list then tools/call; for dated
versions it keeps the SDK Client path.
The runner already passes MCP_CONFORMANCE_PROTOCOL_VERSION to the
spawned client, so no runner change is needed.
* fix: address review findings on MockServer (dead opts, shared validator, capability derivation, recorded parity, specVersion threading)
- MockServerOptions removed (capabilities/configure had zero callers); opts
param dropped from createServerStateful/Stateless/For and ScenarioContext.
- validateStatelessRequest extracted from mock-server/stateless and exported;
both the stateless MockServer and auth/helpers/createServer.ts call it so
_meta/header/version validation cannot drift.
- isStatefulVersion exported from connection/select; mock-server/select uses
it instead of duplicating the version set.
- runner/client.ts: env MCP_CONFORMANCE_PROTOCOL_VERSION set unconditionally
to the resolved version; runInteractiveMode now takes specVersion and the
CLI passes it.
- createServerStateful: capabilities derived from handler method prefixes;
newServer() moved inside the try so a capability mismatch surfaces as
JSON-RPC -32603 instead of an HTML 500. Recording moved to the express
layer so unregistered methods are captured (parity with stateless).
- readFinalSseMessage return type now declares error.data.
- Tests added for the capability derivation and unregistered-method recording.
* fix(mock-server): record stateless requests before validation; document preamble exclusions
The stateless impl pushed to recorded[] only after validateStatelessRequest
passed, so requests rejected for a missing header or _meta never appeared,
diverging from the stateful impl which records at the HTTP layer before SDK
dispatch. Record before validation, excluding only the server/discover
lifecycle preamble (mirroring the stateful initialize/notifications-initialized
exclusion), and fix the MockServer.recorded docstring which wrongly claimed
stateless excludes nothing.
* refactor(mock-server): tri-state result for validateStatelessRequest (reject/handled/route)
A successful server/discover was previously modeled as {ok: false},
which read as a validation failure but actually meant "already
answered, do not route". Replace the boolean discriminant with an
explicit kind: 'reject' (validation failed), 'handled' (valid and
already answered), or 'route' (dispatch to handlers).
Both consumers (the stateless /mcp route and the auth helper's
handleStateless) still write status/body for reject and handled and
route only on 'route', so responses are unchanged. Add direct unit
tests covering each branch of the new union.
* feat(runner): single-source the version→lifecycle mapping; export MCP_CONFORMANCE_LIFECYCLE
The stateless mock and the example client both hardcoded the draft
protocol version, so dating the draft (or adding a second stateless
version) would silently break them: the mock would keep rejecting
everything but the old constant, and the client would fall back to the
stateful path.
- Derive STATELESS_SPEC_VERSIONS and a lifecycleFor() helper from
isStatefulVersion() in connection/select.ts, making it the single
source of truth for the version→lifecycle mapping.
- Thread the resolved spec version into the stateless mock:
createServerFor() binds it into createServerStateless(), and
validateStatelessRequest() now takes the supported-versions list as a
parameter. A mock bound to a version accepts exactly that version and
echoes it in -32004 rejections and server/discover; unbound mocks
accept every known stateless version.
- Export MCP_CONFORMANCE_LIFECYCLE (stateful|stateless) to client
processes alongside MCP_CONFORMANCE_PROTOCOL_VERSION, in both the
runner and the in-process test helper, so clients in any language can
pick the lifecycle without keeping their own version map.
- everything-client: import DRAFT_PROTOCOL_VERSION from src/types
instead of duplicating the string, branch on MCP_CONFORMANCE_LIFECYCLE
(falling back to the version comparison for older runners), and send
the runner-resolved version on stateless requests.
- Document MCP_CONFORMANCE_LIFECYCLE in the README and ignore the local
.claude/ workspace dir in .prettierignore so prettier --check . is not
tripped by untracked worktree checkouts.
* fix(everything-client): list-only flow for json-schema-ref-no-deref
runBasicClient now follows tools/list with a tools/call on the first
listed tool, but the json-schema-ref-no-deref scenario mock only serves
tools/list, so the call got -32601 and the client exited non-zero.
Register the scenario with a dedicated handler that stops after listing
tools, keeping the existing lifecycle branch (raw stateless request vs
SDK client). Listing is the whole flow for SEP-2106: the check is that
the client never fetches the network $ref canary while processing the
advertised schema.
* fix(tools_call): make getChecks() idempotent
getChecks() pushed into an instance array on every call, so calling it
more than once accumulated duplicate check entries. The runner is
allowed to call getChecks() repeatedly, and other scenarios already
build their result fresh per call for this reason.
Return a freshly built array instead and drop the now-unused instance
field. Add a unit test covering the not-called FAILURE case and
asserting repeated getChecks() calls do not grow the result.
* fix: ensure dispatch and transport cleanup runs when handlers throw
The stateless tools/list and prompts/list handlers in the everything
server awaited dispatch.close() after the in-memory request, so a throw
from dispatch.client.request() leaked the Client+Server+Transport pair.
Wrap both in try/finally, matching the tools/call and catch-all
handlers in the same file.
In the stateful mock server, the res 'close' listener that tears down
the per-request SDK transport and server was registered after the
handleRequest await; a throw there skipped registration and leaked the
pair. Register the listener before handing the request to the
transport.
* fix(everything-client): send Accept header on stateless requests
The draft transport spec requires POST requests to accept both
application/json and text/event-stream. Servers built on the SDK's
StreamableHTTPServerTransport already enforce this and reject
requests without it. Does not yet make json-schema-ref-no-deref pass
under --spec-version draft: the SDK transport also rejects the draft
protocol version itself, which is the known fixture gap until the
2026-native scenarios migrate onto MockServer.
* chore: exclude .claude/ from vitest globs
Local agent worktree checkouts under .claude/worktrees/ carry their
own copies of the test suite, which the bare **/*.test.ts include was
adding to every run (and to the pre-push hook). Mirrors the existing
.prettierignore entry.
* fix(request-metadata): use spec error code -32004 for unsupported protocol version
The draft spec defines -32004 (UnsupportedProtocolVersionError) with
data { supported, requested } and HTTP 400 for protocol version
rejections; -32001 (HeaderMismatch) is reserved for HTTP header
validation failures.
- request-metadata scenario: return -32004 with spec-shaped data for
the simulated version-negotiation rejection (was -32001)
- everything-client: trigger negotiation on -32004 (was -32001)
- stateless mock server: include the requested version in the -32004
error data alongside the supported list
- update tests to match and assert the new requested field
* fix(auth): register transport cleanup before handleRequest in createServer
The res 'close' listener that tears down the transport/server pair was
registered after awaiting handleRequest, so a throw mid-request skipped
cleanup registration entirely. Register it first, matching the stateful
mock server.
* fix(everything-server): return JSON-RPC errors from stateless list dispatch on failure
The stateless tools/list and prompts/list dispatch handlers had
try/finally but no catch, so a dispatch failure produced an unhandled
rejection instead of a response. Add the same catch-and-respond block
the carry-forward dispatch handler already uses.
* refactor(runner): drop MCP_CONFORMANCE_LIFECYCLE; derive lifecycle from protocol version
The protocol version fully determines the lifecycle, and deriving it is
part of being a conformant client. The in-repo example client imports
the stateless version set from src/, so there is no drift risk, and a
single env var cannot disagree with itself. README documents the
version→lifecycle mapping for clients in other languages.
* fix(everything-server): return JSON-RPC errors from stateless resource list dispatch on failure
resources/list and resources/templates/list arrived from main with
try/finally but no catch, unlike the sibling tools/list and
prompts/list handlers. A throwing dispatch would escape to Express's
HTML 500 instead of a JSON-RPC error. Mirrors the existing pattern.
* fix(everything-server): include requested field in -32004 error data
The spec's UnsupportedProtocolVersionError requires data to carry both
supported and requested. The mock-server and request-metadata -32004
producers already include it; this was the remaining producer.
* feat(runner): client-side spec-version inference, skip, and --force
Mirrors the server runner's policy from #318: an explicitly requested
--spec-version outside a scenario's applicability window skips it
(exit 0) unless --force; when --spec-version is omitted the version is
inferred from the scenario's applicability instead of defaulting to
the latest dated version.
* fix(auth): gate SEP-837 application_type check to spec versions that include it
The sep-837-application-type-present check fired on every Dynamic Client
Registration regardless of the targeted spec version, but SEP-837 only
exists in the draft spec. Clients conforming to 2025-11-25 or earlier were
failed on a requirement that is not part of the version under test.
Thread the ScenarioContext into createAuthServer (mirroring createServer)
and only emit the check when the resolved spec version includes SEP-837,
i.e. the stateless draft lifecycle. At dated versions the check is not
emitted at all. Audited the other auth helper checks (PKCE, metadata
discovery, token verification); all of them enforce requirements that are
already in the dated specs, so SEP-837 was the only draft-only check.
Unit tests cover both sides: the check fails an omitting client at the
draft version and is absent at 2025-11-25.
* feat(sdk-runner): --spec-version passthrough with per-SDK default
The sdk command forwarded --scenario/--suite/--timeout to the underlying
client/server invocation but had no way to set the spec version, so SDK
runs always used the suite default. Add a --spec-version flag that is
passed through to the inner run.
Also add an optional specVersion field to the per-SDK config, used as the
default when the flag isn't given; an explicit --spec-version overrides
it. Set typescript-sdk-v1 to 2025-11-25: the v1 line targets the latest
dated spec, which keeps draft-only scenarios out of its runs by default.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* feat: split draft-spec scenarios into their own suite
- add --suite draft to the server command, selecting scenarios whose
source.introducedIn is the draft protocol version
- exclude draft scenarios from the default active suite until the draft
spec is published as a dated release
- derive the client command's draft suite from source.introducedIn so the
non-auth draft scenarios (SEP-2243/2575/2322/2106) are included alongside
the auth ones
- keep everything-server self-test coverage for draft scenarios in
all-scenarios.test.ts (active + non-pending draft)
- document available suites in the README and add suite-membership tests
* fix: repair broken specReferences URLs
- SEP-986: SEP/SEP-986.md no longer exists in the spec repo; point at the
Tool Names section of the published 2025-11-25 tools page instead
- SEP-990: the ext-auth spec file is enterprise-managed-authorization.mdx,
not enterprise-oauth.mdx
Fixes#313
* feat: add sdk subcommand to run conformance against any SDK ref (#250)
* Revise sdk runner: explicit --mode, KNOWN_SDKS-only config, v1/v2 entries
Addresses review feedback on the sdk subcommand:
- Require --mode (client|server) and remove "both". Each invocation now
tests exactly one side with its own exit code; the old default ran
client then server but combined exit codes with ||=, which skipped the
server side entirely whenever the client run failed.
- Resolve build/run config from KNOWN_SDKS + CLI flags only; drop the
conformance.config.yaml loader (no SDK ships one yet). The Zod schema
stays as the type for the built-in entries.
- Split the typescript entry by major version: typescript-sdk (v2/main,
pnpm install + build:all, expected-failures.yaml) and typescript-sdk-v1
(v1.x, npm ci + build, conformance-baseline.yml). An entry may set
`repo` (real clone target for an alias) and `defaultRef` (branch used
when no @ref is given). parseSdkSpec now leaves ref undefined when
omitted so defaultRef can apply; a trailing @ is treated as no ref.
- Key the clone cache by ref (<repo>/<ref>) so different refs of the same
repo no longer share one checkout.
- Bound the server readiness probe with a per-request AbortSignal timeout
so a server that accepts the socket but never responds can't hang past
the overall deadline.
* fix(sdk-runner): resolve -o to absolute path; add --expected-failures override; replaceAll for safeName
* feat: support "draft" as a first-class spec version target
- Add LATEST_SPEC_VERSION and DATED_SPEC_VERSIONS constants so the next
spec release is a one-line change in types.ts.
- Accept "draft" as a valid protocolVersion in the initialize check and
mock-server response.
- --spec-version draft now selects latest-dated scenarios plus draft-tagged
ones, so SEP authors can run the full suite against an SDK tracking the
in-progress spec without retagging core scenarios.
- Forward --spec-version to the client process via
MCP_CONFORMANCE_SPEC_VERSION so SDK examples can pick the matching
protocolVersion.
Closes#253
* review: collapse SpecVersion to wire strings; dedup NEGOTIABLE_PROTOCOL_VERSIONS; split out 'extension'
Addresses review feedback from @felixweinberger and @mikekistler:
- SpecVersion is now always a wire protocolVersion string
(DatedSpecVersion | typeof DRAFT_PROTOCOL_VERSION). The separate
'draft' tag literal is gone from the type system; 'draft' survives
only as a CLI input alias in resolveSpecVersion. Removes the
tag-vs-wire confusion and the specVersionToProtocolVersion mapping.
- NEGOTIABLE_PROTOCOL_VERSIONS in types.ts is the single source for
what the mock server accepts on initialize (was duplicated in
checks/client.ts and scenarios/client/initialize.ts).
- 'extension' moved out of SpecVersion into a separate ScenarioSpecTag
type. --spec-version extension is no longer valid; extension scenarios
remain reachable via --suite extensions.
- Draft-tagged scenarios now use the DRAFT_PROTOCOL_VERSION constant
so a draft revision bump is a one-line change in types.ts.
* feat: add tier-check CLI for SDK tier assessment
Adds a 'tier-check' subcommand to the conformance tool that automates
SDK tier assessment against SEP-1730 criteria.
Checks performed:
- Conformance test pass rate (via everything-server)
- GitHub label taxonomy (priority/status/area labels)
- Issue triage SLA compliance
- P0 bug resolution tracking
- Stable release detection
- Required file existence (CHANGELOG, SECURITY, etc.)
- Spec tracking (SDK release within 30d of spec release)
Also includes a Claude Code skill (skills/mcp-sdk-tier-audit/) for
judgment-based checks that require codebase analysis (feature coverage,
docs quality, policy evaluation).
Usage:
npx tsx src/index.ts tier-check --repo modelcontextprotocol/typescript-sdk
npx tsx src/index.ts tier-check --repo ... --conformance-server-cmd '...' \
--conformance-server-cwd ... --conformance-server-url ... --output json
* refactor: revise tier-check CLI and skill based on review feedback
- Move skill to .claude/skills/ so it's auto-available in Claude Code
- Remove feature-coverage subagent (redundant with conformance tests)
- Remove hardcoded ~/src/mcp paths from all skill files
- Trim conformance server table to TS + Python only
- Rename file_existence check to policy_signals (informational, not blocking)
- Add GitHub native issue types detection to labels check
- Add missing features to docs-coverage checklist (tasks, elicitation URL mode, JSON Schema 2020-12)
- Add README with CLI quick start and escape hatch for non-Claude-Code users
- Use --limit 500 instead of --limit 100 for gh issue list
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: address review feedback on skill and README
- Use npx @modelcontextprotocol/conformance instead of node dist/index.js
- Add full GitHub auth instructions (gh auth login, GITHUB_TOKEN, --token)
- Point TS SDK conformance server to typescript-sdk/test/conformance/
- Fix Python SDK URL to localhost:3001/mcp (not TBD)
- Remove manual gh issue list / gh release list from SKILL.md (CLI handles it)
- Remove Claude Code-specific subagent_type references
- Assume user is already in conformance repo
- Clean up policy-evaluation-prompt.md: remove redundant grep commands,
focus on content evaluation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve lint errors in conformance.ts
Remove unused variable assignments flagged by eslint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: apply prettier formatting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: add npm run tier-check script, update docs with examples
- Add "tier-check" npm script so users can run `npm run tier-check --`
instead of `node dist/index.js tier-check`
- Update SKILL.md, README, and skill README to use npm run tier-check
- Add full conformance examples with --conformance-server-cmd/cwd/url
flags and realistic paths (~/src/mcp/typescript-sdk)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: fix prettier formatting in SKILL.md
* fix: add per-scenario timeout, support url-only conformance, fix stdout pollution
- Add 30s per-scenario timeout to prevent tier-check from hanging
- Allow --conformance-server-url without --conformance-server-cmd (server already running)
- Move runner status logs to stderr so --output json produces clean JSON
- Update SKILL.md and README with --silent flag and pre-start server workflow
* refactor: skill takes local path + server URL instead of repo name
The skill now requires two arguments:
1. Local path to the SDK checkout (for direct file inspection)
2. URL where the everything server is already running
The GitHub owner/repo is derived from git remote. This eliminates
cloning, server startup complexity, and branch confusion (v1.x vs main).
* refactor: write detailed reports to files, show concise summary
Reports go to results/tier-audits/<sdk>-<date>/ (already gitignored).
Claude's console output is now just the tier classification,
pass/fail summary line, top 3 actions, and file paths.
* docs: update READMEs for new skill interface and pre-start workflow
* simplify: flat file output instead of nested directory
* fix: remediation always shows path to Tier 2 and Tier 1
* feat: add client conformance testing to tier-check CLI and skill
- Add checkClientConformance() that runs core client scenarios
(initialize, tools_call, elicitation-defaults, sse-retry, auth)
by spawning the SDK's conformance client via --client-cmd
- Add client_conformance to TierScorecard type
- Wire --client-cmd option into the CLI
- Update tier logic: both server + client conformance feed into
Tier 1 (100%) and Tier 2 (>=80%) requirements
- Update terminal and markdown output to show both conformance types
- Update skill to auto-detect conformance client or accept explicit
client-cmd argument
- Update README with new option and examples
* improve: table summary output, write reports via subagents
- Change executive summary from pipe-delimited line to a readable
table with T2/T1 columns
- Move assessment and remediation file writing into parallel
subagents to keep the main conversation thread clean
* improve: list tier gaps as numbered items instead of one-line blob
* improve: finalize summary format with separator, high-priority fixes, numbered gaps
* improve: add pre-flight checks for gh auth and server reachability
Fail fast with clear error messages if GitHub CLI is not authenticated
or if the conformance server URL is not reachable, rather than failing
deep into the scorecard run.
* docs: improve README and fix skill auto-detection paths
- Claude Code section: explain client-cmd auto-detection for TS/Python,
show explicit 3-arg form for other SDKs, add examples for all three
- Fix TypeScript build command (npm run build, not pnpm build:all)
- Fix Python server command (add --port, use uv sync --package)
- Fix Python client path (.github/actions/conformance/client.py)
- Expand 'Other SDKs' section with guidance on everything server
- Add gh auth login prerequisite to Claude Code steps
* simplify: remove client-cmd auto-detection, require explicit argument
Client command is now always passed as the third argument. If omitted,
client conformance is skipped and noted as a gap. No more magic path
detection — clearer and more predictable.
* fix: align docs table with canonical list (48 features), simplify policy eval
Docs coverage:
- Table now has numbered rows matching all 48 non-experimental features
from the canonical list (was missing 7: tools text/image/audio/embedded/
error/notifications, protocol version negotiation)
- Hardcode total as 48 in summary so agents don't miscount
Policy evaluation:
- Simplified from deep content analysis to file-existence checks
- Dependency policy: DEPENDENCY_POLICY.md, dependabot.yml, or CONTRIBUTING.md section
- Roadmap: ROADMAP.md must exist (GitHub milestones alone not sufficient)
- Versioning: VERSIONING.md or CONTRIBUTING.md section
- Removed GitHub API calls for milestones and releases from policy eval
* refactor: extract canonical feature list into single source of truth
Create references/feature-list.md with all 48 non-experimental + 5
experimental features. The docs-coverage prompt now references this
file instead of duplicating the list. One place to update when
features change.
* fix: separate deterministic file checks from AI content evaluation
CLI (files.ts): now checks all policy files deterministically —
DEPENDENCY_POLICY.md, docs/dependency-policy.md, dependabot.yml,
renovate.json, ROADMAP.md, docs/roadmap.md, VERSIONING.md,
docs/versioning.md, BREAKING_CHANGES.md (in addition to existing
CHANGELOG.md, SECURITY.md, CONTRIBUTING.md).
AI policy eval: receives CLI output showing which files exist,
then reads ONLY those files to judge content quality. No longer
searches the repo for files — clean separation of concerns.
* style: apply prettier formatting
* revert: undo unrelated console.log change in runner/server.ts
* refactor: shell out to conformance CLI instead of reimplementing runner
Address PR feedback: conformance.ts was duplicating the normal conformance
running code. Now shells out to 'node dist/index.js server/client' with
-o to save results to a temp dir, then parses the checks.json files.
Also removes --conformance-server-cmd and --conformance-server-cwd options
since the server must be pre-started.
* docs: add Go and C# SDK examples to README and SKILL.md
* fix: add --framework net9.0 to C# server command
* rename: conformance.ts -> test-conformance-results.ts
Avoids confusion with src/runner/ (the actual conformance runner).
This file just invokes the CLI and parses output.
* style: prettier formatting
* fix: reconcile conformance results against full scenario list
The tier-check CLI was only counting scenarios that produced a
checks.json file. Scenarios that crashed or failed to run (e.g., auth
scenarios when OAuth is not implemented) were invisible, making the
denominator artificially small (e.g., 4/4 instead of 4/23).
Now both checkConformance and checkClientConformance reconcile their
parsed results against the known scenario lists, adding failure entries
for any expected scenario that didn't produce results.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: tighten documentation evaluation criteria
Clarify what counts as documented vs just having code:
- Conformance test servers don't count as docs or examples
- Examples without prose = PARTIAL, not PASS
- Go Example* test functions explicitly allowed
- Clear PASS/PARTIAL/FAIL verdict definitions
* docs: add Labels and Spec Tracking rows to audit report templates
The executive summary and assessment report were missing two SEP-1730
requirements: label taxonomy compliance and spec tracking (new protocol
features timeline).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: reuse ConformanceCheck type from src/types.ts instead of redefining
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* docs: add SDK integration guide and fix spec references
- Add SDK_INTEGRATION.md with guide for SDK maintainers on integrating conformance tests
- Link to guide from README.md
- Update spec references to use 2025-11-25 instead of draft URLs
* fix: prettier formatting and correct action version to v0.1.10
Add support for SDKs that don't yet pass all conformance tests to still
run conformance in CI without failing, by specifying a YAML baseline file
of known/expected failures.
CLI changes:
- Add --expected-failures <path> flag to both server and client commands
- Exit 0 if only expected failures occur
- Exit 1 if unexpected failures occur (regression)
- Exit 1 if an expected failure now passes (stale baseline, must update)
GitHub Action (action.yml):
- Composite action that builds from source and runs conformance tests
- SDKs reference it as: uses: modelcontextprotocol/conformance@<ref>
- Inputs: mode, url, command, expected-failures, suite, scenario, etc.
New files:
- src/expected-failures.ts: YAML loading, baseline evaluation, reporting
- src/expected-failures.test.ts: 16 unit tests
- action.yml: composite GitHub Action
Closes#99
* Refactor client testing to pass scenario name and server URL
The CLI now passes both scenario name and server URL as arguments to the
client command (instead of just the server URL). This enables a single
"everything" client that can handle all scenarios based on the scenario
name it receives.
Changes:
- Update executeClient() to pass scenarioName as first argument to client
- Add everything-client.ts that routes to appropriate behavior based on
scenario name
- Update README with new usage examples and documentation
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Use env var for scenario name instead of positional arg
Pass scenario name via MCP_CONFORMANCE_SCENARIO env var instead of as a
positional argument. This is non-breaking for existing clients that just
accept <server-url>.
The env var approach is consistent with how context (including private
keys for JWT auth) is already passed via MCP_CONFORMANCE_CONTEXT.
* Add comment explaining env var design decision
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Olivier Chafik <ochafik@anthropic.com>
* spec for servers
* initial version of everything server and tests - tets are very simple
* some clean up
* lint etc.
* fix package lock agian
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
Felix Weinberger
Paul Carleton
Max Isbey
Marcelo Trylesinski
Matt
Inna Harper
Paul Carleton