* chore: sync spec-types to modelcontextprotocol@0168c57f
Refreshes the vendored schema copies via npm run sync-schema. The draft
schema now declares LATEST_PROTOCOL_VERSION = "2026-07-28" (previously
the DRAFT-2026-v1 placeholder) and picks up the latest draft doc-comment
and type updates. The dated schemas are unchanged.
* fix: update the draft wire version to 2026-07-28
The spec repo's draft schema now declares
LATEST_PROTOCOL_VERSION = "2026-07-28", but the harness still asserted
the old DRAFT-2026-v1 placeholder, so an SDK that sends the version the
draft spec actually declares cannot pass the draft suite.
- bump DRAFT_PROTOCOL_VERSION in src/types.ts to 2026-07-28
- update the hard-coded literals in tests, example servers, and README
- add src/types.test.ts pinning DRAFT_PROTOCOL_VERSION to the vendored
draft schema's LATEST_PROTOCOL_VERSION so the two cannot drift, and
asserting the draft identifier stays distinct from the dated versions
* feat: vendor spec schema types per-version
Copies schema/{version}/schema.ts from the modelcontextprotocol spec
repo into src/spec-types/{version}.ts so the conformance suite can type
against draft spec versions before any SDK ships them.
npm run sync-schema -- <ref> refreshes the copies and records the spec
commit in src/spec-types/SOURCE.
* feat: add Connection abstraction and RunContext
Connection encapsulates how the conformance suite talks to a
server-under-test for a given spec version:
- connectStateful: 2025-x lifecycle. Thin adapter over the SDK Client
(initialize handshake, session id, SSE handled by the SDK).
- connectStateless: 2026-x lifecycle (SEP-2575). Raw fetch with
per-request _meta + MCP-Protocol-Version header. Decoupled from the
SDK so the suite can test draft spec versions before the SDK
implements them.
connectFor(specVersion) picks the implementation. RunContext bundles
serverUrl, specVersion and a bound connect() for the runner to hand to
each scenario.
Nothing uses this yet; wiring follows in the next commit.
* refactor: thread RunContext through ClientScenario.run
ClientScenario.run(serverUrl) becomes run(ctx: RunContext). The runner
builds the context from --spec-version and the server URL; scenarios
destructure ctx.serverUrl and otherwise behave identically.
No scenario uses ctx.connect() yet, so behaviour is unchanged: 214/214
tests pass, all-scenarios.test.ts still drives the everything-server
fixture exactly as before.
Test files use a testContext(url) helper to construct a RunContext.
The authorization-server scenario list is retyped to
ClientScenarioForAuthorizationServer since those scenarios test an
OAuth server, not an MCP server, and keep run(serverUrl).
* refactor: migrate server scenarios to ctx.connect() + conn.request()
22 carry-forward and lifecycle scenarios now go through the Connection
abstraction instead of connectToServer + SDK Client:
- tools.ts (8): list, call x6, with-progress
- prompts.ts (5)
- resources.ts (7): list, read x3, subscribe x2, not-found
- utils.ts (3): completion, ping, set-level
- json-schema-2020-12.ts, caching.ts, http-standard-headers.ts
Result types come from spec-types/{introducedIn}. Same scenario code
now passes under both --spec-version 2025-11-25 (SDK-backed stateful)
and --spec-version draft (raw stateless).
ToolsCallSampling/Elicitation/WithLogging and elicitation-* keep
connectToServer (need setRequestHandler/setLoggingLevel SDK surface);
they are tagged removedIn: DRAFT in the next commit. stateless.ts and
input-required-result.ts keep their sendRpc helper; migrating those is
deferred to the DRAFT-scenario coherence pass.
* fix: normalize Connection error to JsonRpcError; clean up RunContext import
connectStateful now catches the SDK's McpError and rethrows as
JsonRpcError so scenarios always see the same error class regardless
of which Connection impl ran. ResourcesNotFoundError uses
instanceof JsonRpcError instead of duck-typing.
types.ts uses a normal top-level import for RunContext instead of an
inline import() type.
* fix(dns-rebinding): use version-appropriate probe body
The 'valid Host accepted' check was sending an initialize body, which
a 2026 server returns 404 for. Probe body is now picked from
ctx.specVersion: initialize for the stateful lifecycle, server/discover
with _meta for the stateless lifecycle. The Host/Origin rejection check
is unchanged since rejection happens before body parsing.
* feat(everything-server): route stateless carry-forward methods to McpServer
Adds an in-memory dispatch client connected to the same McpServer the
stateful path uses. Stateless requests for tools/call, resources/*,
prompts/get and completion/complete that fall through the MRTR-specific
handlers are routed to it, so the fixture serves the carry-forward
scenarios under --spec-version draft without duplicating ~500 lines of
tool/resource/prompt registrations.
tools/list now merges the McpServer's tool list with the MRTR-only
stubs so json-schema-2020-12 finds its tool.
draft suite against the fixture: 36/39 (was 13/39). Remaining 3 are
fixture-side SEP gaps (no SSE forwarding for progress, no SEP-2549
ttlMs, no SEP-2243 Mcp-Method validation in the stateless path).
* refactor(connection): drop unused RequestOptions; move sdk-client; add tests
Addresses self-review findings on the new connection module:
- RequestOptions.handlers/.meta and ServerRequestHandler removed: zero
callers (the scenarios that motivated them are deferred). They can be
reintroduced when something actually uses them.
- scenarios/server/client-helper.ts moved to connection/sdk-client.ts so
the connection module no longer depends on the scenarios module.
- connectStateless: handle CRLF SSE event separators; throw a useful
error for non-JSON/non-SSE responses instead of a JSON parse error.
- JSONRPCNotification consistently imported from spec-types/2025-11-25.
- New connection.test.ts (8 tests) covering connectFor selection,
_meta injection, error mapping, SSE LF/CRLF parsing, and the
server-request-on-stream rejection.
* fix: address bughunt findings (response.ok check; targetVersion naming)
- connectStateless: throw on non-2xx responses that lack a JSON-RPC
error envelope (e.g. gateway/framework error JSON), matching the
stateful path's behavior. Previously such a response would return
undefined as the result.
- json-schema-2020-12: rename negotiatedVersion to targetVersion and
reword the skip message and details field. The value is ctx.specVersion
(the run's --spec-version), not a negotiated version; the old wording
was misleading. Drop dead 'unknown' fallback (specVersion is required)
and the corresponding undefined test cases.
* fix(sse-multiple-streams): keep scenario in draft; version-aware request scaffolding
Concurrent POSTs each answered with JSON or their own SSE stream are
core transport behavior in the draft spec too, so removedIn was wrong
for this scenario. Only the request scaffolding is version-specific:
stateful runs keep the initialize/session-id setup, draft runs send
_meta + the MCP-Protocol-Version header on each request instead. The
two checks are unchanged.
* feat: add MockServer abstraction and ScenarioContext
MockServer encapsulates the lifecycle scaffold a client-conformance
scenario presents to the client-under-test:
- createServerStateful: 2025-x lifecycle. SDK Server +
StreamableHTTPServerTransport (sessionless mode); the SDK handles the
initialize handshake.
- createServerStateless: 2026-x lifecycle (SEP-2575). Raw express app
that validates _meta + MCP-Protocol-Version on every request, serves
server/discover, routes other methods to the supplied handlers.
createServerFor(specVersion) picks the implementation. ScenarioContext
bundles specVersion and a bound createServer() for the runner to hand
to each scenario.
This is the client-conformance mirror of src/connection (PR #318).
Nothing uses it yet; wiring follows in the next commit.
* refactor: thread ScenarioContext through Scenario.start()
Scenario.start() becomes start(ctx: ScenarioContext). The runner builds
the context from --spec-version (defaulting to LATEST_SPEC_VERSION) and
passes it through; scenarios receive it as _ctx and otherwise behave
identically.
No scenario uses ctx.createServer() yet, so behaviour is unchanged:
231/231 tests pass.
Test files use a testScenarioContext() helper. The runner already
threads MCP_CONFORMANCE_PROTOCOL_VERSION to the spawned client process,
so the fixture-side env wiring is unchanged.
* refactor: migrate tools_call to ctx.createServer; tag 2025-only client scenarios
ToolsCallScenario now goes through ctx.createServer() instead of an
inline express + SDK Server build. Same handlers, same checks; the
assertion now reads from srv.recorded so it works regardless of which
lifecycle scaffold the runner picked.
initialize, sse-retry, and elicitation-defaults are tagged
removedIn: DRAFT (initialize/GET-SSE/SSE-embedded-elicitation are gone
in the 2026 lifecycle; the MRTR sibling for elicitation-defaults is a
follow-up).
spec-version.test.ts: the 'draft is a superset of latest' invariant no
longer holds once removedIn: DRAFT exists; the test now asserts that
any scenario in latest-but-not-draft is explicitly removedIn.
* feat(auth): make createServer helper version-aware via ScenarioContext
The auth helper now takes ctx: ScenarioContext as its first argument
and branches on ctx.specVersion inside the /mcp route: the stateful
path (SDK Server + StreamableHTTPServerTransport) is unchanged; under
the draft version a raw stateless handler validates _meta + the
MCP-Protocol-Version header, serves server/discover, and routes the
same tools/list and tools/call responses.
The PRM endpoint, bearer-auth middleware, and request logger sit above
the branch and are version-independent.
All 25 call sites across the 12 auth scenario files pass ctx through;
ServerLifecycle and the express.Application return type are unchanged
so stop()/getChecks() are untouched.
Deviation from the MockServer wrapper approach: keeping the helper's
return type as express.Application avoids restructuring 25 call sites'
ServerLifecycle handling in this PR. Folding the auth seam onto
ctx.createServer() fully is a follow-up once the lifecycle ownership
moves into MockServer.
* feat(everything-client): pick stateless requester by MCP_CONFORMANCE_PROTOCOL_VERSION
Adds a statelessRequest(serverUrl, method, params) helper that POSTs
with _meta + MCP-Protocol-Version (the SEP-2575 lifecycle), shimming
around the SDK Client not yet supporting stateless mode. The
runRequestMetadataClient handler's meta constants are extracted to share
with the helper.
runBasicClient (initialize, tools_call, json-schema-ref-no-deref) now
branches on MCP_CONFORMANCE_PROTOCOL_VERSION: for the draft version it
uses statelessRequest to call tools/list then tools/call; for dated
versions it keeps the SDK Client path.
The runner already passes MCP_CONFORMANCE_PROTOCOL_VERSION to the
spawned client, so no runner change is needed.
* fix: address review findings on MockServer (dead opts, shared validator, capability derivation, recorded parity, specVersion threading)
- MockServerOptions removed (capabilities/configure had zero callers); opts
param dropped from createServerStateful/Stateless/For and ScenarioContext.
- validateStatelessRequest extracted from mock-server/stateless and exported;
both the stateless MockServer and auth/helpers/createServer.ts call it so
_meta/header/version validation cannot drift.
- isStatefulVersion exported from connection/select; mock-server/select uses
it instead of duplicating the version set.
- runner/client.ts: env MCP_CONFORMANCE_PROTOCOL_VERSION set unconditionally
to the resolved version; runInteractiveMode now takes specVersion and the
CLI passes it.
- createServerStateful: capabilities derived from handler method prefixes;
newServer() moved inside the try so a capability mismatch surfaces as
JSON-RPC -32603 instead of an HTML 500. Recording moved to the express
layer so unregistered methods are captured (parity with stateless).
- readFinalSseMessage return type now declares error.data.
- Tests added for the capability derivation and unregistered-method recording.
* fix(mock-server): record stateless requests before validation; document preamble exclusions
The stateless impl pushed to recorded[] only after validateStatelessRequest
passed, so requests rejected for a missing header or _meta never appeared,
diverging from the stateful impl which records at the HTTP layer before SDK
dispatch. Record before validation, excluding only the server/discover
lifecycle preamble (mirroring the stateful initialize/notifications-initialized
exclusion), and fix the MockServer.recorded docstring which wrongly claimed
stateless excludes nothing.
* refactor(mock-server): tri-state result for validateStatelessRequest (reject/handled/route)
A successful server/discover was previously modeled as {ok: false},
which read as a validation failure but actually meant "already
answered, do not route". Replace the boolean discriminant with an
explicit kind: 'reject' (validation failed), 'handled' (valid and
already answered), or 'route' (dispatch to handlers).
Both consumers (the stateless /mcp route and the auth helper's
handleStateless) still write status/body for reject and handled and
route only on 'route', so responses are unchanged. Add direct unit
tests covering each branch of the new union.
* feat(runner): single-source the version→lifecycle mapping; export MCP_CONFORMANCE_LIFECYCLE
The stateless mock and the example client both hardcoded the draft
protocol version, so dating the draft (or adding a second stateless
version) would silently break them: the mock would keep rejecting
everything but the old constant, and the client would fall back to the
stateful path.
- Derive STATELESS_SPEC_VERSIONS and a lifecycleFor() helper from
isStatefulVersion() in connection/select.ts, making it the single
source of truth for the version→lifecycle mapping.
- Thread the resolved spec version into the stateless mock:
createServerFor() binds it into createServerStateless(), and
validateStatelessRequest() now takes the supported-versions list as a
parameter. A mock bound to a version accepts exactly that version and
echoes it in -32004 rejections and server/discover; unbound mocks
accept every known stateless version.
- Export MCP_CONFORMANCE_LIFECYCLE (stateful|stateless) to client
processes alongside MCP_CONFORMANCE_PROTOCOL_VERSION, in both the
runner and the in-process test helper, so clients in any language can
pick the lifecycle without keeping their own version map.
- everything-client: import DRAFT_PROTOCOL_VERSION from src/types
instead of duplicating the string, branch on MCP_CONFORMANCE_LIFECYCLE
(falling back to the version comparison for older runners), and send
the runner-resolved version on stateless requests.
- Document MCP_CONFORMANCE_LIFECYCLE in the README and ignore the local
.claude/ workspace dir in .prettierignore so prettier --check . is not
tripped by untracked worktree checkouts.
* fix(everything-client): list-only flow for json-schema-ref-no-deref
runBasicClient now follows tools/list with a tools/call on the first
listed tool, but the json-schema-ref-no-deref scenario mock only serves
tools/list, so the call got -32601 and the client exited non-zero.
Register the scenario with a dedicated handler that stops after listing
tools, keeping the existing lifecycle branch (raw stateless request vs
SDK client). Listing is the whole flow for SEP-2106: the check is that
the client never fetches the network $ref canary while processing the
advertised schema.
* fix(tools_call): make getChecks() idempotent
getChecks() pushed into an instance array on every call, so calling it
more than once accumulated duplicate check entries. The runner is
allowed to call getChecks() repeatedly, and other scenarios already
build their result fresh per call for this reason.
Return a freshly built array instead and drop the now-unused instance
field. Add a unit test covering the not-called FAILURE case and
asserting repeated getChecks() calls do not grow the result.
* fix: ensure dispatch and transport cleanup runs when handlers throw
The stateless tools/list and prompts/list handlers in the everything
server awaited dispatch.close() after the in-memory request, so a throw
from dispatch.client.request() leaked the Client+Server+Transport pair.
Wrap both in try/finally, matching the tools/call and catch-all
handlers in the same file.
In the stateful mock server, the res 'close' listener that tears down
the per-request SDK transport and server was registered after the
handleRequest await; a throw there skipped registration and leaked the
pair. Register the listener before handing the request to the
transport.
* fix(everything-client): send Accept header on stateless requests
The draft transport spec requires POST requests to accept both
application/json and text/event-stream. Servers built on the SDK's
StreamableHTTPServerTransport already enforce this and reject
requests without it. Does not yet make json-schema-ref-no-deref pass
under --spec-version draft: the SDK transport also rejects the draft
protocol version itself, which is the known fixture gap until the
2026-native scenarios migrate onto MockServer.
* chore: exclude .claude/ from vitest globs
Local agent worktree checkouts under .claude/worktrees/ carry their
own copies of the test suite, which the bare **/*.test.ts include was
adding to every run (and to the pre-push hook). Mirrors the existing
.prettierignore entry.
* fix(request-metadata): use spec error code -32004 for unsupported protocol version
The draft spec defines -32004 (UnsupportedProtocolVersionError) with
data { supported, requested } and HTTP 400 for protocol version
rejections; -32001 (HeaderMismatch) is reserved for HTTP header
validation failures.
- request-metadata scenario: return -32004 with spec-shaped data for
the simulated version-negotiation rejection (was -32001)
- everything-client: trigger negotiation on -32004 (was -32001)
- stateless mock server: include the requested version in the -32004
error data alongside the supported list
- update tests to match and assert the new requested field
* fix(auth): register transport cleanup before handleRequest in createServer
The res 'close' listener that tears down the transport/server pair was
registered after awaiting handleRequest, so a throw mid-request skipped
cleanup registration entirely. Register it first, matching the stateful
mock server.
* fix(everything-server): return JSON-RPC errors from stateless list dispatch on failure
The stateless tools/list and prompts/list dispatch handlers had
try/finally but no catch, so a dispatch failure produced an unhandled
rejection instead of a response. Add the same catch-and-respond block
the carry-forward dispatch handler already uses.
* refactor(runner): drop MCP_CONFORMANCE_LIFECYCLE; derive lifecycle from protocol version
The protocol version fully determines the lifecycle, and deriving it is
part of being a conformant client. The in-repo example client imports
the stateless version set from src/, so there is no drift risk, and a
single env var cannot disagree with itself. README documents the
version→lifecycle mapping for clients in other languages.
* fix(everything-server): return JSON-RPC errors from stateless resource list dispatch on failure
resources/list and resources/templates/list arrived from main with
try/finally but no catch, unlike the sibling tools/list and
prompts/list handlers. A throwing dispatch would escape to Express's
HTML 500 instead of a JSON-RPC error. Mirrors the existing pattern.
* fix(everything-server): include requested field in -32004 error data
The spec's UnsupportedProtocolVersionError requires data to carry both
supported and requested. The mock-server and request-metadata -32004
producers already include it; this was the remaining producer.
* feat(runner): client-side spec-version inference, skip, and --force
Mirrors the server runner's policy from #318: an explicitly requested
--spec-version outside a scenario's applicability window skips it
(exit 0) unless --force; when --spec-version is omitted the version is
inferred from the scenario's applicability instead of defaulting to
the latest dated version.
* fix(auth): gate SEP-837 application_type check to spec versions that include it
The sep-837-application-type-present check fired on every Dynamic Client
Registration regardless of the targeted spec version, but SEP-837 only
exists in the draft spec. Clients conforming to 2025-11-25 or earlier were
failed on a requirement that is not part of the version under test.
Thread the ScenarioContext into createAuthServer (mirroring createServer)
and only emit the check when the resolved spec version includes SEP-837,
i.e. the stateless draft lifecycle. At dated versions the check is not
emitted at all. Audited the other auth helper checks (PKCE, metadata
discovery, token verification); all of them enforce requirements that are
already in the dated specs, so SEP-837 was the only draft-only check.
Unit tests cover both sides: the check fails an omitting client at the
draft version and is absent at 2025-11-25.
* feat(sdk-runner): --spec-version passthrough with per-SDK default
The sdk command forwarded --scenario/--suite/--timeout to the underlying
client/server invocation but had no way to set the spec version, so SDK
runs always used the suite default. Add a --spec-version flag that is
passed through to the inner run.
Also add an optional specVersion field to the per-SDK config, used as the
default when the flag isn't given; an explicit --spec-version overrides
it. Set typescript-sdk-v1 to 2025-11-25: the v1 line targets the latest
dated spec, which keeps draft-only scenarios out of its runs by default.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* feat(everything-server): serve resources and caching hints on the stateless draft path
The stateless (SEP-2575) request path now answers resources/list,
resources/templates/list and resources/read — including the SEP-2164
-32602 + data.uri error for unknown URIs — and includes the SEP-2549
ttlMs/cacheScope hints on its cacheable list results, matching what the
draft-spec scenarios exercise.
* feat: make the harness's own draft-spec traffic conformant
Fixes the class of issues where a strictly-conformant SDK rejects the
harness's requests for reasons unrelated to the behaviour under test
(#311, #312, #315).
- add a shared stateless draft request helper (draft-client.ts) that puts
the cross-cutting SEP-2243 headers (Mcp-Method, Mcp-Name,
MCP-Protocol-Version, Accept) and SEP-2575 _meta (protocolVersion,
clientInfo, clientCapabilities) on every request, with overrides for
negative cases and the SHOULD-level retry on
UnsupportedProtocolVersionError
- migrate the draft scenarios onto it: the SEP-2322 helpers and
server-stateless now send Mcp-Method/Mcp-Name (#312); the SEP-2243
server scenarios drop the legacy initialize handshake and carry _meta on
every raw request (#311); caching and sep-2164-resource-not-found are
stateless DRAFT-2026-v1 requests instead of an SDK 2025-11-25 session
(#315)
- add a draft self-conformance gate (draft-self-conformance.test.ts) that
points the harness's own draft drivers at the existing client-testing
judge scenarios (request-metadata, http-standard-headers) and requires
zero failures or warnings
- align the request-metadata mock's unsupported-version rejection with the
draft schema (-32004) and teach the everything-client to recognize it
- rewrite the SEP-2549/SEP-2164 negative example servers as stateless
draft servers so they keep exercising the violation, not the transport
* refactor: rename the stateless request helpers and harden them per review
- rename draft-client.ts to stateless-client.ts and name the helpers by
feature (sendStatelessRequest, buildStandardHeaders, buildRequestMeta,
withRequestMeta, CONFORMANCE_CLIENT_INFO); 'draft' now only appears in
DRAFT_PROTOCOL_VERSION, the one name that tracks the moving target
- retry only genuine UnsupportedProtocolVersionError rejections (-32004,
tolerating -32001/-32602) and only with recognized versions; record the
retry on the response
- parse text/event-stream responses incrementally and resolve on the
matching event instead of blocking until the timeout
- keep the MCP-Protocol-Version header and _meta protocolVersion in sync
when the version is overridden via meta overrides
- make server-stateless's sendRpc SSE-tolerant
- emit explicit setup failures in the SEP-2243 server scenarios when the
tools/list discovery fails instead of misleading per-case results
- latch the worst status per check in the request-metadata judge so an
early violation cannot be masked by a later conformant request
- rename the gate to harness-traffic-conformance.test.ts and derive its
driver list from the draft suite with documented per-judge exclusions
(35 pairings)
- everything-server: advertise resources in stateless server/discover and
use -32004 for unsupported protocol version
- add unit tests for the helper (retry guard, SSE handling, header/_meta
version sync, default headers)
* refactor: slim the stateless client helper for the initial merge
* feat: split draft-spec scenarios into their own suite
- add --suite draft to the server command, selecting scenarios whose
source.introducedIn is the draft protocol version
- exclude draft scenarios from the default active suite until the draft
spec is published as a dated release
- derive the client command's draft suite from source.introducedIn so the
non-auth draft scenarios (SEP-2243/2575/2322/2106) are included alongside
the auth ones
- keep everything-server self-test coverage for draft scenarios in
all-scenarios.test.ts (active + non-pending draft)
- document available suites in the README and add suite-membership tests
* fix: repair broken specReferences URLs
- SEP-986: SEP/SEP-986.md no longer exists in the spec repo; point at the
Tool Names section of the published 2025-11-25 tools page instead
- SEP-990: the ext-auth spec file is enterprise-managed-authorization.mdx,
not enterprise-oauth.mdx
Fixes#313
* chore: add SEP-2106 requirement-traceability YAML (tools JSON Schema 2020-12)
* Drop structured-content text-fallback row from SEP-2106
That SHOULD predates SEP-2106 (pre-existing context line in tools.mdx); 2106
only relaxes structuredContent to allow non-object values (permissive). Point
spec_url at the SEP doc security section where the remaining obligations live.
* feat: SEP-2106 JSON Schema 2020-12 vocabulary preservation checks
Extend the json-schema-2020-12 server scenario (previously SEP-1613 only) with
three checks that verify the broader JSON Schema 2020-12 vocabulary SEP-2106
permits in a tool inputSchema survives tools/list rather than being stripped:
composition (allOf/anyOf), conditional (if/then/else), and reference ($anchor)
keywords.
- everything-server: add these keywords to json_schema_2020_12_tool's raw schema
- add a negative-test server (sep-2106-stripped-schema.ts) plus a negative test
asserting the checks emit FAILURE when the vocabulary is stripped
* feat: SEP-2106 client scenario for network $ref dereferencing
Add json-schema-ref-no-deref, a client-suite scenario that advertises a tool
whose inputSchema contains a $ref pointing at a canary URL hosted by the
scenario. The sep-2106-no-network-ref-deref check fails if the client fetches
the canary while processing the schema, covering the SEP-2106 requirement that
implementations MUST NOT automatically dereference network-URI $ref values.
- register the scenario in the client scenarios list
- everything-client: handle the scenario with the basic connect+listTools flow
- tests: compliant client passes, a deliberately dereferencing client is
caught, and a client that never lists tools cannot pass
* Address review findings in SEP-2106 scenarios
- json-schema-ref-deref: make getChecks() idempotent (fresh array per call,
matching the http-standard-headers convention) and guard the async /mcp
handler like the example servers do
- json-schema-2020-12: include the SEP-2106 keywords ($anchor, allOf/anyOf,
if/then/else) in the description's example inputSchema so a server that
copies it verbatim passes the new checks
- sep-2106-stripped-schema: default to port 3007 to match negative.test.ts and
avoid sharing a default with sep-2549-no-caching-hints
* Soft-gate SEP-2106 keyword checks on the negotiated protocol version
The json-schema-2020-12 scenario is introducedIn 2025-11-25 (SEP-1613) but the
three SEP-2106 keyword-preservation checks test draft-era behavior. Rather than
failing servers that only negotiated 2025-11-25, gate the failure on the wire
protocolVersion from the initialize handshake: preserved keywords are SUCCESS at
any version, stripped keywords are FAILURE only when the server negotiated the
draft version and SKIPPED otherwise.
The published SDK does not advertise the draft version yet, so the negative test
asserts SKIPPED for the stripped-schema server; the FAILURE branch is covered by
unit tests on sep2106KeywordCheckStatus.
Add two client scenarios for the SEP-2468 requirements that had no
conformance coverage:
- auth/metadata-issuer-mismatch: the AS metadata document declares an
issuer that differs from the issuer identifier used to construct the
well-known URL; the client must not use the metadata
(sep-2468-client-validate-metadata-issuer).
- auth/iss-normalized: the authorization redirect carries an iss value
that only matches the recorded issuer after URL normalization; the
client must compare with simple string comparison and reject
(sep-2468-client-no-normalization).
The two authorization-server-side rows (AS SHOULD include iss, AS MUST
advertise authorization_response_iss_parameter_supported) are excluded:
observing iss in an authorization response requires driving an
Authorization Code Grant against the AS under test, which the
authorization-server suite does not implement yet (#208).
The x-mcp-header / Mcp-Param / Base64 half of SEP-2243 was already tested,
but the scenarios emitted one check ID per test case while sep-2243.yaml
declares one per normative requirement, so the traceability join (exact ID
match) reported 14 of 20 requirements as untested. #287 collapsed the
standard-header IDs but left the custom-header ones per-case.
This finishes the rename and adds DECLARED_CHECK_IDS backfill (the
request-metadata.ts pattern) so every requirement-level ID is emitted on
every run — exercised, backfilled as FAILURE, or SKIPPED when the server
under test exposes no x-mcp-header tool. SEP-2243 traceability goes from
6/20 to 18/20 tested; the remaining 2 (server-not-expect-null,
server-reject-missing-required) have no test case yet.
Client scenarios:
- param-header-{name} -> client-mirrors-designated-params /
client-encode-values / client-base64-unsafe, picked by what the
parameter exercises
- no-mirror-unannotated -> client-mirrors-designated-params
- param-header-tool-call-gate -> client-supports-custom-headers, now also
emitted SUCCESS when the annotated tool is called with Mcp-Param headers
- the invalid-tool loop emits the specific x-mcp-header constraint each
tool violates (not-empty / charset / unique / primitive-only)
Server scenarios:
- accepts-valid-base64 -> server-decode-base64
- rejects-invalid-base64-{padding,chars} -> server-reject-invalid-param-chars
- literal-missing-base64-{prefix,suffix} -> server-validate-param-match
- rejects-missing-custom-header -> server-validate-param-match
- every custom-header rejection's -32001 check -> server-reject-param-mismatch
- standard-header reject cases pass the coarse ID through the catch path too
* chore: additional server conformance tests for SEP-2575
* fix(sep-2575): use the spec's notifications filter object in subscriptions/listen
The scenario sent params.subscriptions: [{type: 'tools/list-changed'}], a
shape that does not exist in the SEP. SubscriptionsListenRequestParams
requires params.notifications: {toolsListChanged?, promptsListChanged?, ...}
(see schema.ts SubscriptionFilter and the canonical
SubscriptionsListenRequest example). A compliant server would either reject
the old request as invalid params or treat it as subscribing to nothing,
failing the ack/list-changed checks.
Also makes the everything-server's acknowledgment echo the agreed
notifications filter, as the SubscriptionsAcknowledgedNotification schema
requires.
* fix(sep-2575): match the spec's notifications/*/list_changed method names
The list-changed checks and the everything-server used
notifications/tools/list-changed (hyphen); the spec's method is
notifications/tools/list_changed (underscore) per ToolListChangedNotification
/ PromptListChangedNotification in schema.ts. With the hyphen form the
notification-filter check cannot catch a real leak, and the list-changed
checks fail a server that emits the correct name.
Also drops the f.params.toolsListChanged === true fallback match: that field
is the subscriptions/listen request filter, not a notification parameter, so
it never appears on a compliant server's notification.
* fix(sep-2575): keep collected frames when the listen stream read times out
listenToStream aborts the fetch after timeoutMs. If the abort fired while
blocked in reader.read(), the rejection propagated to the outer catch and the
helper returned [] — discarding every frame already received. A compliant
server holds a subscriptions/listen stream open indefinitely, so the timeout
is the *normal* way these reads end; previously every subscription check
would report a false 'no frames received' FAILURE against such a server.
The current everything-server only avoided this by res.end()ing immediately
after the ack.
* fix(sep-2575): open the listen stream before triggering list mutations
The notification-filter and list-changed checks fired the mutation trigger
*before* opening the subscriptions/listen stream. A compliant server only
delivers list-changed notifications to streams that are open at the time of
the change, so the SHOULD checks would fail any server that does not replay
past changes to new subscribers, and the filter check could never observe a
real leak.
listenToStream now takes an onFirstFrame hook; the three trigger-based
checks open the stream, wait for the acknowledgment, then fire the mutation
on a separate connection and keep reading.
The everything-server now keeps subscriptions/listen streams open, registers
them with their filters, and fans the list_changed notification out to open
streams when a trigger tool runs - instead of unconditionally emitting a
list-changed frame on every stream open (which itself violated the
notification-filter requirement in spirit: it announced a change when
nothing had changed).
* fix(sep-2575): check the subscription id on every listen-stream notification
server-tags-subscription-id only inspected the acknowledgment frame, so a
server that tags the ack but omits io.modelcontextprotocol/subscriptionId
from subsequent notifications passed. The check now triggers a tool-list
change once the stream is acknowledged and asserts every notification frame
on the stream carries the id in params._meta.
Also drops the f.body.method / f.params.method fallbacks when resolving a
frame's method - those shapes don't exist in JSON-RPC and only masked
malformed frames.
* fix(sep-2575): report WARNING, not FAILURE, for the SHOULD-level list-changed checks
The spec text behind server-sends-{tools,prompts}-list-changed-on-subscription
is a SHOULD ('servers that declared the listChanged capability SHOULD send a
notification...'). Severity follows the spec keyword: MUST -> FAILURE,
SHOULD -> WARNING. runCheck now accepts a warning flag and the two
list-changed checks use it on their failure paths.
* fix(sep-2575): skip stream checks when the diagnostic tool is not exposed
http-server-no-independent-requests-on-stream and
server-no-log-without-loglevel reported SUCCESS when the server rejected the
tools/call outright (e.g. test_streaming_elicitation / test_logging_tool do
not exist): a single error frame contains no independent request and no log
notification, so the check passed without exercising anything. The sibling
list-changed checks already SKIP when their trigger tool is missing; these
now do the same.
* chore(sep-2575): exclude the four unobservable server requirements from traceability
Marks the section-C rows of #296 as excluded rather than untested:
no-prior-context, no-connection-reuse-required, disconnect-is-cancel and
stops-on-cancel describe internal server state that a black-box harness
cannot observe on the wire. With these excluded and the part-B checks landed,
every testable server-side SEP-2575 requirement is covered; the remaining
untested rows are all client-side and blocked on a SEP-2575-aware reference
client.
* fix(sep-2575): emit the request-metadata checks even when the client never connects
The reference SDK conformance client has no handler registered for the
request-metadata scenario, so it exits without sending a request, the
scenario emits zero checks, the suite reports "0 passed, 0 failed", and the
traceability manifest records all seven client-side requirements as
untested.
Declare the scenario check IDs up front and backfill any that were never
emitted as FAILURE from getChecks(): the emitted ID set is now the same for
every client, and a client that never connects fails loudly. A check that
is legitimately not applicable opts out by being emitted as SKIPPED
explicitly (version-header-matches-meta now does this when either version
field is absent). The positive-path test pins the declared list against the
reference client in both directions.
Also refresh the traceability manifest from a run against
typescript-sdk@22595b96: the seven client rows flip to tested and the
manifest catches up with the server-side scenarios from #297.
---------
Co-authored-by: Yuan Teoh <yuanteoh@google.com>
* Add SEP-2350 scope-accumulation check to auth/scope-step-up
SEP-2350 clarifies that on step-up re-authorization, clients SHOULD
compute the union of previously-granted and newly-challenged scopes so
they don't lose permissions for other operations.
- Traceability yaml: 1 client check (two spec sentences merged), 1 server
check tracked for a future server-side scenario, 1 excluded reword
- ScopeStepUpAuthScenario now challenges with only the missing scope so
union accumulation is observable on the second authorization request
- New check sep-2350-scope-union-on-reauth (WARNING when the previously
granted scope is dropped)
- everything-client withOAuthRetry now accumulates prior token scope into
the re-auth scope (passing example)
- New auth-test-echo-scope negative client + vitest case
* feat: add client conformance tests for SEP-2575 (#270)
* feat: merge latest request-metadata scenario stacked branch
* test: implement optional capability conditional checks and simulated negotiation retry
* fix(sep-2243): collapse check IDs to one-per-requirement (#287)
The sep-2243 scenario check IDs drifted from the requirement-traceability
yaml in #259 — the code emitted one ID per test case while the yaml
declares one per normative requirement, so the IDs no longer matched.
Rename the emitted IDs so a check ID maps to a single MUST/SHOULD and is
emitted once per case (the per-case detail moves to name/description),
matching the repo's existing 'same id, vary status/message' convention:
- mcp-method-header-* / mcp-name-header-* -> client-includes-standard-headers
- reject-invalid-tool-* / keep-valid-tool -> client-reject-invalid-tool
- server-accepts-{lower,upper}case-name -> header-name-case-insensitive
- server reject status checks (mismatch/missing/case x method/name)
-> server-reject-invalid-headers
- their error-code variants -> server-reject-error-code
Merge the yaml's server-reject-mismatch + server-reject-status into one
server-reject-invalid-headers MUST (HTTP 400 on a header-validation
failure); keep server-reject-error-code as the SHOULD. Base64/custom-header
rejection checks keep their own param-validation IDs (out of scope here).
Gates and the whitespace-acceptance check keep their scenario-matching
names. No behavior change — only check IDs, descriptions, and the yaml.
* feat: `sdk` subcommand to run local conformance against any SDK ref (#277)
* feat: add sdk subcommand to run conformance against any SDK ref (#250)
* Revise sdk runner: explicit --mode, KNOWN_SDKS-only config, v1/v2 entries
Addresses review feedback on the sdk subcommand:
- Require --mode (client|server) and remove "both". Each invocation now
tests exactly one side with its own exit code; the old default ran
client then server but combined exit codes with ||=, which skipped the
server side entirely whenever the client run failed.
- Resolve build/run config from KNOWN_SDKS + CLI flags only; drop the
conformance.config.yaml loader (no SDK ships one yet). The Zod schema
stays as the type for the built-in entries.
- Split the typescript entry by major version: typescript-sdk (v2/main,
pnpm install + build:all, expected-failures.yaml) and typescript-sdk-v1
(v1.x, npm ci + build, conformance-baseline.yml). An entry may set
`repo` (real clone target for an alias) and `defaultRef` (branch used
when no @ref is given). parseSdkSpec now leaves ref undefined when
omitted so defaultRef can apply; a trailing @ is treated as no ref.
- Key the clone cache by ref (<repo>/<ref>) so different refs of the same
repo no longer share one checkout.
- Bound the server readiness probe with a per-request AbortSignal timeout
so a server that accepts the socket but never responds can't hang past
the overall deadline.
* fix(sdk-runner): resolve -o to absolute path; add --expected-failures override; replaceAll for safeName
* feat: add SEP conformance traceability manifest (#288)
squashed; see PR #288 body
* feat: add server conformance tests for SEP-2575 (#271)
* feat: add server conformance tests for SEP-2575
* refactor and add new checks
* Fix stateless scenario source field and report SKIPPED for inapplicable capability checks
- Add the required `source` field (DRAFT) so the scenario typechecks and is
selectable via --spec-version. Without it the class did not satisfy the
ClientScenario interface and s.source was dereferenced by the spec-version
filter at list time.
- Teach runCheck about a SKIPPED status and use it for the two client-capability
checks when the server does not return -32003, instead of reporting a green
PASS for a requirement that was never exercised.
* Remove unimplemented placeholder checks from stateless server scenario
The subscriptions/listen, statelessness-invariant, list-changed and
disconnect-is-cancel checks emitted SUCCESS without probing the server,
which inflated coverage in the traceability report. Drop them until they
have real assertions; the corresponding rows remain declared in
src/seps/sep-2575.yaml so traceability shows them as not-yet-covered.
Also fix the checkErrorId helper to push under the
sep-2575-http-server-error-jsonrpc-id slug so its failure path actually
short-circuits the aggregate SUCCESS check at the end of the scenario.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* SEP-2352: authorization-server migration scenario (#286)
* Add SEP-2352 authorization-server migration scenario
SEP-2352 requires that client credentials are bound to the issuing
authorization server: when PRM authorization_servers changes to a new
issuer, clients MUST re-register and MUST NOT reuse the previous AS's
client credentials.
- Traceability yaml: 3 checks, 3 excluded (internal state / UI)
- New auth/authorization-server-migration scenario (draft suite): two
auth servers; PRM flips from AS1 to AS2 after the first authenticated
request; AS2 asserts it received a fresh /register and never saw AS1's
client_id at /authorize or /token
- ConformanceOAuthProvider gains invalidateCredentials and bindIssuer so
the everything-client can key credentials by issuer (passing example)
- everything-client adds an issuer-aware handler for this scenario that
re-reads PRM on each 401 and rebinds before re-authorizing
- auth-test-reuse-credentials negative client + vitest case
* Drop application_type from DCR metadata (not in SDK OAuthClientMetadata type)
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* ci(traceability): enable corepack so the SDK build can use pnpm (#289)
The refresh run failed with 'pnpm: not found' — the reference SDK's
build command (typescript-sdk: pnpm install && pnpm run build:all) needs
pnpm on PATH. Add 'corepack enable' to the run job.
* ci(traceability): tolerate SDK conformance failures in the run step (#290)
The `sdk` command exits non-zero when the SDK has conformance failures
not in its baseline. The traceability manifest only needs the emitted
check IDs (written regardless of pass/fail), so a failing SDK must not
fail the refresh. Add `|| true`; the existing 'no results produced'
guard catches a genuinely broken run.
* chore: refresh SEP traceability manifest (typescript-sdk@main) (#291)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Make scopesSupported disjoint from challenge scopes so the SEP-2350 union check can't be satisfied by scopes_supported ∪ challenge
A client that unions scopes_supported with the challenge — instead of its
prior grant with the challenge — would have requested mcp:basic mcp:write
and falsely passed sep-2350-scope-union-on-reauth, since scopesSupported
happened to coincide with the previously-granted scope. Setting it to a
disjoint value makes the check actually require the prior token's scope.
Spec backing: clients MUST NOT assume any set relationship between the
challenged scope set and scopes_supported, so a disjoint advertisement is
realistic.
* SEP-837: application_type check in DCR registration (#284)
* Add SEP-837 application_type check to DCR registration
SEP-837 requires MCP clients to specify an appropriate application_type
during Dynamic Client Registration so OIDC authorization servers can
apply the correct redirect-URI constraints.
- Traceability yaml: 1 check (presence + valid value), 4 excluded
(class-specific SHOULDs unobservable; UI/robustness)
- Check added in the shared createAuthServer DCR handler so it fires in
every auth scenario that performs DCR (no new scenario)
- withOAuthRetry now sets application_type: native (passing example;
the conformance example clients are CLI tools)
- New auth-test-no-application-type negative client + vitest case
* Widen ConformanceOAuthProvider metadata type for application_type
Fixes CI typecheck (TS2353 at withOAuthRetry.ts:76): the SDK's
OAuthClientMetadataSchema doesn't include application_type yet.
registerClient() spreads clientMetadata verbatim into the /register
POST body, so a local type intersection is sufficient to get the
field on the wire without an SDK release.
* Set application_type in runAuthMigrationClient
The SEP-2352 authorization-server-migration handler constructs its own
ConformanceOAuthProvider (not via withOAuthRetry), so it was missing
application_type after rebasing onto main. The scenario does DCR twice
(old AS, new AS) and the SEP-837 check fires on each.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* feat: add conformance tests for iss parameter (SEP-2468) (#220)
* feat: add conformance tests for iss parameter (SEP-2468)
Adds 5 draft conformance scenarios testing RFC 9207 issuer parameter
validation in OAuth authorization responses:
- auth/iss-supported: server advertises support and sends correct iss
- auth/iss-not-advertised: server omits iss parameter entirely
- auth/iss-supported-missing: client must reject missing iss when required
- auth/iss-wrong-issuer: client must reject mismatched iss value
- auth/iss-unexpected: client must reject iss when not advertised
Also adds auth-test-iss-validation.ts, a reference client that correctly
validates iss per RFC 9207, and negative tests confirming the standard
client fails all three rejection scenarios.
TODO: Update RFC_9207_ISS_PARAMETER spec reference once SEP-2468
(modelcontextprotocol/modelcontextprotocol#2468) is merged.
* update scenarios
* fix: createAuthServer iss option type/guard and NotAdvertised scenario duplication
The doc comments said 'Default: not included' but the destructure defaulted
to true/'correct', and the `!== undefined` guard at L155 was unreachable —
so there was no way to omit the metadata field, and IssParameterNotAdvertised
silently advertised support (a duplicate of IssParameterSupported).
Kept the on-by-default behavior (mock AS models a well-behaved server) but
made issParameterSupported `boolean | null` so callers pass null to omit,
matching the codeChallengeMethodsSupported pattern. Doc comments now match.
Scenarios that need omission pass null/'omit' explicitly.
* fix: rejection scenarios silently pass when client never reaches auth endpoint
correctlyRejected = !tokenRequestMade reports SUCCESS if the client errors
out before hitting /authorize. Gate on authReached so a setup failure shows
as FAILURE with authReached:false in details.
* fix: iss-unexpected scenario contradicts SEP-2468 spec table row 3
The spec table says: supported=false/absent + iss present -> *Compare* to
the recorded issuer (not reject). The scenario sent a *correct* iss and
FAILed compliant clients for proceeding after a successful comparison.
Now sends a mismatched iss so the comparison fails and rejection is the
spec-required outcome. Reference client updated to compare-when-present
instead of throw-on-presence.
* refactor: replace harness-config checks with client-proceeded checks
iss-advertised-in-metadata / iss-sent-in-redirect (and the not-* variants)
fired in onAuthorizationRequest before the redirect happened, asserting only
that the harness was configured correctly — a client that ignores iss passes
identically. Replaced with one check per scenario keyed on tokenRequestMade,
which observes that the client actually proceeded through the iss path.
* refactor: rename check IDs to sep-2468-* and align with spec table rows
One ID per spec table row; auth/iss-supported and auth/iss-wrong-issuer
both emit sep-2468-client-compare-iss-supported (same comparison, opposite
input) per the same-slug-for-SUCCESS-and-FAIL convention.
* feat: add sep-2468.yaml requirement traceability
8 check rows (4 client table-row checks, 1 metadata-issuer, 2 AS-side,
1 no-normalization), 1 excluded (error-display is UI-facing). The
record-issuer MUST is merged into the compare-iss-supported row text
since it has no independent wire observation.
* fix: migrate iss scenarios specVersions->source (post-#265)
Replaces `specVersions: ['draft']` with `source: { introducedIn: DRAFT_PROTOCOL_VERSION }`
in the 5 iss-parameter scenarios.
This commit typechecks once the stack is rebased onto main >= #265 (the
ScenarioSource migration). Adding it now so the rebase is mechanical.
* fix: include application_type in iss-validation example DCR (post-#284)
The SEP-837 application_type check now runs in every auth scenario; the
hand-rolled DCR in auth-test-iss-validation.ts was omitting the field.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
---------
Co-authored-by: Anubhav Dhawan <anubhavdhawan@google.com>
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Max Gerber <89937743+max-stytch@users.noreply.github.com>
* feat: add conformance tests for iss parameter (SEP-2468)
Adds 5 draft conformance scenarios testing RFC 9207 issuer parameter
validation in OAuth authorization responses:
- auth/iss-supported: server advertises support and sends correct iss
- auth/iss-not-advertised: server omits iss parameter entirely
- auth/iss-supported-missing: client must reject missing iss when required
- auth/iss-wrong-issuer: client must reject mismatched iss value
- auth/iss-unexpected: client must reject iss when not advertised
Also adds auth-test-iss-validation.ts, a reference client that correctly
validates iss per RFC 9207, and negative tests confirming the standard
client fails all three rejection scenarios.
TODO: Update RFC_9207_ISS_PARAMETER spec reference once SEP-2468
(modelcontextprotocol/modelcontextprotocol#2468) is merged.
* update scenarios
* fix: createAuthServer iss option type/guard and NotAdvertised scenario duplication
The doc comments said 'Default: not included' but the destructure defaulted
to true/'correct', and the `!== undefined` guard at L155 was unreachable —
so there was no way to omit the metadata field, and IssParameterNotAdvertised
silently advertised support (a duplicate of IssParameterSupported).
Kept the on-by-default behavior (mock AS models a well-behaved server) but
made issParameterSupported `boolean | null` so callers pass null to omit,
matching the codeChallengeMethodsSupported pattern. Doc comments now match.
Scenarios that need omission pass null/'omit' explicitly.
* fix: rejection scenarios silently pass when client never reaches auth endpoint
correctlyRejected = !tokenRequestMade reports SUCCESS if the client errors
out before hitting /authorize. Gate on authReached so a setup failure shows
as FAILURE with authReached:false in details.
* fix: iss-unexpected scenario contradicts SEP-2468 spec table row 3
The spec table says: supported=false/absent + iss present -> *Compare* to
the recorded issuer (not reject). The scenario sent a *correct* iss and
FAILed compliant clients for proceeding after a successful comparison.
Now sends a mismatched iss so the comparison fails and rejection is the
spec-required outcome. Reference client updated to compare-when-present
instead of throw-on-presence.
* refactor: replace harness-config checks with client-proceeded checks
iss-advertised-in-metadata / iss-sent-in-redirect (and the not-* variants)
fired in onAuthorizationRequest before the redirect happened, asserting only
that the harness was configured correctly — a client that ignores iss passes
identically. Replaced with one check per scenario keyed on tokenRequestMade,
which observes that the client actually proceeded through the iss path.
* refactor: rename check IDs to sep-2468-* and align with spec table rows
One ID per spec table row; auth/iss-supported and auth/iss-wrong-issuer
both emit sep-2468-client-compare-iss-supported (same comparison, opposite
input) per the same-slug-for-SUCCESS-and-FAIL convention.
* feat: add sep-2468.yaml requirement traceability
8 check rows (4 client table-row checks, 1 metadata-issuer, 2 AS-side,
1 no-normalization), 1 excluded (error-display is UI-facing). The
record-issuer MUST is merged into the compare-iss-supported row text
since it has no independent wire observation.
* fix: migrate iss scenarios specVersions->source (post-#265)
Replaces `specVersions: ['draft']` with `source: { introducedIn: DRAFT_PROTOCOL_VERSION }`
in the 5 iss-parameter scenarios.
This commit typechecks once the stack is rebased onto main >= #265 (the
ScenarioSource migration). Adding it now so the rebase is mechanical.
* fix: include application_type in iss-validation example DCR (post-#284)
The SEP-837 application_type check now runs in every auth scenario; the
hand-rolled DCR in auth-test-iss-validation.ts was omitting the field.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* Add SEP-837 application_type check to DCR registration
SEP-837 requires MCP clients to specify an appropriate application_type
during Dynamic Client Registration so OIDC authorization servers can
apply the correct redirect-URI constraints.
- Traceability yaml: 1 check (presence + valid value), 4 excluded
(class-specific SHOULDs unobservable; UI/robustness)
- Check added in the shared createAuthServer DCR handler so it fires in
every auth scenario that performs DCR (no new scenario)
- withOAuthRetry now sets application_type: native (passing example;
the conformance example clients are CLI tools)
- New auth-test-no-application-type negative client + vitest case
* Widen ConformanceOAuthProvider metadata type for application_type
Fixes CI typecheck (TS2353 at withOAuthRetry.ts:76): the SDK's
OAuthClientMetadataSchema doesn't include application_type yet.
registerClient() spreads clientMetadata verbatim into the /register
POST body, so a local type intersection is sufficient to get the
field on the wire without an SDK release.
* Set application_type in runAuthMigrationClient
The SEP-2352 authorization-server-migration handler constructs its own
ConformanceOAuthProvider (not via withOAuthRetry), so it was missing
application_type after rebasing onto main. The scenario does DCR twice
(old AS, new AS) and the SEP-837 check fires on each.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* Add SEP-2352 authorization-server migration scenario
SEP-2352 requires that client credentials are bound to the issuing
authorization server: when PRM authorization_servers changes to a new
issuer, clients MUST re-register and MUST NOT reuse the previous AS's
client credentials.
- Traceability yaml: 3 checks, 3 excluded (internal state / UI)
- New auth/authorization-server-migration scenario (draft suite): two
auth servers; PRM flips from AS1 to AS2 after the first authenticated
request; AS2 asserts it received a fresh /register and never saw AS1's
client_id at /authorize or /token
- ConformanceOAuthProvider gains invalidateCredentials and bindIssuer so
the everything-client can key credentials by issuer (passing example)
- everything-client adds an issuer-aware handler for this scenario that
re-reads PRM on each 401 and rebinds before re-authorizing
- auth-test-reuse-credentials negative client + vitest case
* Drop application_type from DCR metadata (not in SDK OAuthClientMetadata type)
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
The sep-2243 scenario check IDs drifted from the requirement-traceability
yaml in #259 — the code emitted one ID per test case while the yaml
declares one per normative requirement, so the IDs no longer matched.
Rename the emitted IDs so a check ID maps to a single MUST/SHOULD and is
emitted once per case (the per-case detail moves to name/description),
matching the repo's existing 'same id, vary status/message' convention:
- mcp-method-header-* / mcp-name-header-* -> client-includes-standard-headers
- reject-invalid-tool-* / keep-valid-tool -> client-reject-invalid-tool
- server-accepts-{lower,upper}case-name -> header-name-case-insensitive
- server reject status checks (mismatch/missing/case x method/name)
-> server-reject-invalid-headers
- their error-code variants -> server-reject-error-code
Merge the yaml's server-reject-mismatch + server-reject-status into one
server-reject-invalid-headers MUST (HTTP 400 on a header-validation
failure); keep server-reject-error-code as the SHOULD. Base64/custom-header
rejection checks keep their own param-validation IDs (out of scope here).
Gates and the whitespace-acceptance check keep their scenario-matching
names. No behavior change — only check IDs, descriptions, and the yaml.
* Add prepare script for git dependency installs
* Add conformance tests for SEP-2243 HTTP Standardization
* Add traceability file for SEP-2243
* Fix prettier formatting in http-standard-headers.ts
* Improve SEP-2243 conformance test coverage
Close gaps in HTTP header conformance scenarios:
Client standard headers (http-standard-headers.ts):
- Enforce all expected methods in checks, failing any that were not observed
- Track Mcp-Name header per-method separately from Mcp-Method
- Advertise resources and prompts capabilities so clients exercise those endpoints
- Add a prompt entry for prompts/get testing
Client custom headers (http-custom-headers.ts):
- Add Base64 encoding checks for non-ASCII, whitespace, and control-char values
- Add null/omitted parameter test (second tool call with null value)
- Add client-keeps-valid-tool check verifying clients still call valid tools
after filtering out invalid ones
Server header validation (server/http-standard-headers.ts):
- Replace fetch-based sendRawRequest with http.request to preserve exact
header casing on the wire (fetch/Headers lowercases names)
- Compute defaultArgs and defaultHeaders from tool schema so test requests
satisfy all required parameters while varying only the param under test
Traceability (sep-2243.yaml):
- Add 7 new spec-to-check mappings (nonascii, whitespace, controlchar,
keeps-valid-tool, literal-missing-base64-prefix/suffix, no-mirror-unannotated)
- Fix 2 incorrect existing mappings
* Fill SEP-2243 conformance test coverage gaps
Add missing test cases identified by comparing the branch against the
SEP-2243 spec's Conformance Test Cases section:
- Mcp-Method on notifications: add notifications/initialized to
expectedMethods so clients are checked for header on notifications
- Boolean true: add debug=true param to complement verbose=false
- Leading-space-only and trailing-space-only: separate checks for
Base64 encoding when only one edge has whitespace
- Internal spaces only: verify plain ASCII (no Base64) for values
like 'us west 1' that have spaces only in the middle
- CRLF string: add line1\r\nline2 test (complements existing \n test)
- Leading tab: add \tindented test for tab-triggered Base64 encoding
- Server rejects invalid Mcp-Param chars: mark as excluded in YAML
since HTTP itself prevents these characters in headers
Register all new checks in sep-2243.yaml.
* Remove case-insensitive =?base64? prefix test
The spec states header values are case-sensitive (RFC 9110). The
=?base64? prefix is part of the header value, not the header name,
so it should be matched case-sensitively. This was identified as a
bug in the conformance tests per feedback on SEP-2243.
Changes:
- Remove server-accepts-case-insensitive-base64 check from server
validation scenario
- Change base64 prefix regex from case-insensitive (/i) to
case-sensitive in client header validation
- Remove corresponding entry from sep-2243.yaml
* Remove nested x-mcp-header rejection test
The spec constrains x-mcp-header to primitive types but does not
restrict it to top-level properties only. A nested string property
with x-mcp-header is valid per the spec. This was confirmed by
mikekistler in SEP-2243 PR discussion.
Removes invalid_nested_header from the invalid tool definitions
list and its corresponding sep-2243.yaml entry.
* Use numeric comparison for number header values
Compare number header values numerically instead of as strings to
accommodate cross-SDK floating point representation differences
(e.g., '42' vs '42.0'). For integers, exact numeric match is
required. For decimals, a tolerance of 1e-9 is allowed.
See SEP-2243 discussion on number precision.
* client/http-standard-headers: SKIPPED (not FAILURE) for unexercised methods; make getChecks() idempotent
A client that never calls prompts/list isn't violating SEP-2243 — the spec
sentence is 'The client MUST include the standard MCP request headers on each
POST request', and a request that was never sent can't fail it. The Mcp-Method
requirement is already proven by initialize/tools/list etc., so there's nothing
unique to prompts/list. SKIPPED keeps the gap visible without a false red.
Separately: getChecks() was pushing into this.checks on every call. The runner
may call it more than once (e.g. progress + final report); that produced
duplicate rows. Build a fresh array per call instead.
* client/http-standard-headers: de-dup guard in checkMcpNameHeader
checkMcpMethodHeader already has 'if (this.methodHeaderChecks.has(method)) return'.
checkMcpNameHeader was missing the equivalent. The test server advertises two
tools and two resources; a client that calls both produces two
'client-mcp-name-header-tools-call' / '...-resources-read' rows.
* client/http-standard-headers: replace -> /\//g for slug generation
String.replace with a string arg only replaces the first occurrence.
'notifications/initialized' is fine (one slash) but a method like
'notifications/resources/updated' would yield 'notifications-resources/updated'
in the check id. (replaceAll isn't in this project's TS lib target.)
* client/http-custom-headers: base64 regex (.+) -> (.*) so empty string round-trips
The spec doesn't forbid base64-encoding values that don't strictly need it.
A client that always wraps would send '=?base64??=' for empty_val: ''. The
'.+' wouldn't match, so the harness would compare '=?base64??=' === ''
literally and FAIL a compliant client. '(.*)' lets the empty payload decode
to '' and match.
* client/http-custom-headers: emit SUCCESS for no-mirror-unannotated
This check only pushed on FAILURE; a correct client produced no row at all,
so the pass was invisible in reports and couldn't be counted toward coverage.
Same-slug SUCCESS/FAILURE pair is the repo convention.
* client/http-custom-headers: drop redundant optional-present check
checkParamHeader('Verbose', ...) already pushes a FAILURE when the header is
missing. The explicit 'optional-present' block re-tests the same condition
under a second check id, doubling the row.
* server/http-standard-headers: defaultArgs must use real number/boolean, not strings
defaultArgs feeds the JSON-RPC body. With Record<string, string> + '0'/'false',
a server that validates inputSchema rejects on type mismatch — which is also a
400. Every header-rejection check then false-PASSes (server rejected for the
wrong reason), and server-accepts-valid-base64 false-FAILs a compliant server
because it never gets past schema validation.
Also handle 'integer' (JSON Schema distinguishes it from 'number').
* server/http-standard-headers: createAcceptanceCheck must also assert no JSON-RPC error in body
A server can return HTTP 200 with {error: {code: -32001, ...}} in the body.
Status-only acceptance lets that through as a pass.
* use DRAFT_PROTOCOL_VERSION constant instead of 'DRAFT-2026-v1' literal
The literal will rot when the draft cycle rolls over. types.ts already exports
DRAFT_PROTOCOL_VERSION for this.
* sep-2243.yaml: spec-first rewrite + move to src/seps/
Regenerated from the SEP-2243 spec diff (transports.mdx + tools.mdx) rather
than from the scenario implementations: 21 check rows + 4 excluded vs the
previous 46 + 2. Differences:
- one check id per spec sentence (test variants go in 'details', not new ids)
- 'text:' quotes the spec sentence verbatim instead of paraphrasing
- '-reject-status' (MUST 400) split from '-reject-error-code' (SHOULD -32001
for standard headers, MUST for custom)
- rows with no spec backing dropped (whitespace, base64-padding/chars, prefix/
suffix-literal, control-char-name)
- two more SHOULD excludes (log-warning, no-sensitive-params)
- check ids use sep-2243-<slug> convention
- check: key first, excludes grouped at bottom (matches sep-2164)
Moved to src/seps/ to match #272 layout (this branch predates it; will
reconcile cleanly on rebase).
* add negative test for HttpStandardHeadersScenario
Self-contained vitest that POSTs initialize with and without Mcp-Method,
asserts FAILURE/SUCCESS on the pinned check id, and proves getChecks() is
idempotent. No example client/server file needed for this one because the
scenario *is* the server — the test crafts the bad request directly.
This is the negative-test half of AGENTS.md §Testing your scenario; the
everything-client passing-example half is still TODO.
* server/http-standard-headers: severity fixes for whitespace + malformed-base64 tests
These five tests assert behavior the SEP-2243 text doesn't pin down. Kept
because they're useful consistency signals across SDKs, but adjusted so they
don't FAIL spec-compliant servers:
- server-accepts-whitespace-header-value: this IS a MUST, but per RFC 9110
§5.5 (field parsing MUST exclude OWS), not SEP-2243. Kept as FAILURE,
specReference now cites RFC 9110.
- server-rejects-invalid-base64-padding / -chars: downgraded to INFO.
SEP-2243 says only 'MUST decode them accordingly' without picking RFC 4648
strict vs lenient. Node's Buffer.from() accepts unpadded/dirty input and
decodes to a matching value (server accepts); .NET's Convert.FromBase64String
throws (server rejects). Either is currently compliant. WARNING would force
Tier-1 SDKs into non-default strict decoding (#245); INFO records the
behavior without prescribing it, so cross-SDK divergence is visible
without tier impact.
- server-literal-missing-base64-prefix / -suffix: unchanged. The wrapper
syntax is '=?base64?{x}?=' complete; treating partial wrappers as literal
is the natural reading.
Plumbing: createRejectionCheck gains an optional failSeverity param;
testBase64Case gains a 'reject-info' mode. (This is also the hook for the
larger 400-MUST / -32001-SHOULD split that still needs doing.)
* server/http-standard-headers: split rejection check into status (MUST) + error-code (SHOULD/MUST)
SEP-2243 §Server Validation: 'servers MUST return HTTP status 400 ... and
SHOULD include a JSON-RPC error response using ... -32001'. So a server that
returns 400 with -32600 (or no error body) is compliant for standard headers.
The single conflated check FAILed it.
For custom headers (§Server Behavior for Custom Headers) -32001 IS a MUST,
so both halves stay FAILURE there.
createRejectionCheck → createRejectionChecks returning two rows: '<id>' for
the 400 status and '<id>-error-code' for -32001. testCase (standard) sets
errorCodeSeverity=WARNING; testBase64Case/testMissingCustomHeader (custom)
keep FAILURE; the two malformed-base64 INFO probes set both halves to INFO.
* rename check ids to sep-2243-* prefix; kebab-case throughout
Aligns with the sep-2243.yaml traceability ids and the repo convention
(sep-2164-*, sep-2207-*). Variant suffixes are kept (per-method, per-tool)
so per-case visibility isn't lost; the YAML's row id is a prefix of the
emitted ids, which is fine — extra ids not in the YAML are allowed.
Also fixes the mixed hyphen/underscore in reject-invalid-tool ids
(toolName has underscores; replace to hyphens).
* client: extract BaseHttpScenario to client/http-base.ts; both client scenarios extend it
http-standard-headers.ts had its own copy of the start/stop/handleRequest
boilerplate (~100 lines) that http-custom-headers.ts already abstracted as
BaseHttpScenario. Moved the abstract class to a shared file; both scenario
files now import it. HttpStandardHeadersScenario implements handlePost()
instead of handleRequest(); its handleInitialize() uses the base
sendInitialize() with the resources/prompts capability flags it needs.
Net: 465 → 361 lines for http-standard-headers.ts; the third inline copy of
the same HTTP-server scaffold is gone.
* use req.setEncoding('utf8') instead of per-chunk Buffer.toString()
Per-chunk .toString() corrupts a multi-byte UTF-8 char that straddles a TCP
chunk boundary (e.g. 0xC3 | 0xBC decodes to two replacement chars instead of
'ü'). The custom-headers scenario sends '日本語'/'naïve' in the body, so this
is reachable in principle. setEncoding('utf8') makes 'data' emit strings with
boundary handling done by Node's StringDecoder.
Fixed in both places: BaseHttpScenario.handleRequest (request body) and
sendRawRequest (response body).
* fix: migrate SEP-2243 scenarios from specVersions to source (introducedIn/removedIn)
The introducedIn/removedIn refactor (#265) replaced the specVersions array
with a source property on all scenario interfaces. The SEP-2243 scenarios
still used the old specVersions field, causing matchesSpecVersion() to
receive undefined and crash in spec-version.test.ts.
Replace specVersions with source = { introducedIn: DRAFT_PROTOCOL_VERSION }
in BaseHttpScenario and the two server ClientScenario classes. Remove the
now-unused SpecVersion imports.
* server/http-standard-headers: malformed-base64 padding/chars back to FAILURE
Per discussion on #259: the SEP-2243 conformance-test-case table is the
approved source of truth and lists these as reject cases, so they're
FAILURE-level even though the spec body itself only says 'MUST decode them
accordingly'. SDKs whose stdlib base64 is lenient (Node, browsers) will need
to validate before decoding; if that's burdensome we'll revisit.
Removes the now-unused 'reject-info' mode and statusSeverity/INFO plumbing
introduced for these two probes.
---------
Co-authored-by: Tarek Mahmoud Sayed <tarekms@microsoft.com>
Co-authored-by: Tarek Mahmoud Sayed <tarekms@ntdev.microsoft.com>
Co-authored-by: Tarek Mahmoud Sayed <10833894+tarekgh@users.noreply.github.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* refactor: replace specVersions list with introducedIn/removedIn range tagging
Replaces per-scenario `specVersions: SpecVersion[]` arrays with `introducedIn`
and optional `removedIn` range fields on all three scenario interfaces. Extension
scenarios get an orthogonal `extension?: boolean` flag instead of the former
`'extension'` tag in the versions list.
matchesSpecVersion() is rewritten as a range comparison
(introducedIn <= target && (!removedIn || target < removedIn)), with draft
sorting after all dated versions. getScenarioSpecVersions() reconstructs the
effective version list from the range for backward-compat with tier-check.
Closes#256.
* refactor: nest spec-version range under `source: ScenarioSource` union
The flat `introducedIn`/`removedIn`/`extension?` shape forced extension
scenarios to set a placeholder `introducedIn` that was never read. Nesting
the version-range fields inside a `source` property typed as a union lets
the compiler enforce introducedIn XOR extensionId — no runtime invariant
test needed.
type ScenarioSource =
| { introducedIn: SpecVersion; removedIn?: SpecVersion }
| { extensionId: ExtensionId };
A class can't `implements` a union type directly, so the union lives inside
a property and `Scenario` stays a plain interface — `class implements
Scenario` keeps working everywhere. `EXTENSION_IDS` is a const array
mirroring the `DATED_SPEC_VERSIONS` pattern.
Also renames the cross-app-access scenario to enterprise-managed-auth to
match the extension's canonical name: file, class, slug, context schema
literal, and the everything-client runner.
* refactor: use canonical SEP-2133 extension identifiers
EXTENSION_IDS now holds the wire identifiers from the spec's extension docs
(client-matrix.mdx, docs/extensions/auth/*.mdx) so extensionId values can be
cross-referenced against capabilities.extensions keys directly:
io.modelcontextprotocol/oauth-client-credentials
io.modelcontextprotocol/enterprise-managed-authorization
The previous commit used a shortened 'enterprise-managed-auth' which doesn't
match the canonical '-authorization'. Re-renames the scenario file/class/slug
and downstream references to align.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat: support "draft" as a first-class spec version target
- Add LATEST_SPEC_VERSION and DATED_SPEC_VERSIONS constants so the next
spec release is a one-line change in types.ts.
- Accept "draft" as a valid protocolVersion in the initialize check and
mock-server response.
- --spec-version draft now selects latest-dated scenarios plus draft-tagged
ones, so SEP authors can run the full suite against an SDK tracking the
in-progress spec without retagging core scenarios.
- Forward --spec-version to the client process via
MCP_CONFORMANCE_SPEC_VERSION so SDK examples can pick the matching
protocolVersion.
Closes#253
* review: collapse SpecVersion to wire strings; dedup NEGOTIABLE_PROTOCOL_VERSIONS; split out 'extension'
Addresses review feedback from @felixweinberger and @mikekistler:
- SpecVersion is now always a wire protocolVersion string
(DatedSpecVersion | typeof DRAFT_PROTOCOL_VERSION). The separate
'draft' tag literal is gone from the type system; 'draft' survives
only as a CLI input alias in resolveSpecVersion. Removes the
tag-vs-wire confusion and the specVersionToProtocolVersion mapping.
- NEGOTIABLE_PROTOCOL_VERSIONS in types.ts is the single source for
what the mock server accepts on initialize (was duplicated in
checks/client.ts and scenarios/client/initialize.ts).
- 'extension' moved out of SpecVersion into a separate ScenarioSpecTag
type. --spec-version extension is no longer valid; extension scenarios
remain reachable via --suite extensions.
- Draft-tagged scenarios now use the DRAFT_PROTOCOL_VERSION constant
so a draft revision bump is a one-line change in types.ts.
* feat: add sep-2207 client checks
* Update description for the offline access test
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
---------
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
* feat: add specVersions classification to conformance scenarios
Each scenario declares which spec versions it applies to as a list.
Scenarios that carry forward (e.g. initialize) list all applicable versions
['2025-06-18', '2025-11-25']. Scenarios removed from newer specs (e.g.
backcompat auth) only list their original version ['2025-03-26'].
- specVersions list on Scenario and ClientScenario interfaces
- --spec-version CLI filter uses simple .includes()
- Tier-check conformance matrix (Server / Client: Core / Client: Auth)
with per-version columns and unique All* count
- 7 unit tests for specVersions helpers
- Updated tier-audit skill docs with matrix format
* fix: fix console output template for tier audit skill
The template had an orphan table header (Check | Value | T2 | T1)
with no rows above the conformance matrix, causing an empty table
to render. The scorecard rows below the matrix also lacked their
own header.
Fix: two self-contained tables with clear labels — 'Conformance:'
for the per-version matrix, 'Scorecard:' for the check rows.
* fix: align skill console template with tier-check script output
Add asterisk footnote ('unique scenarios — a scenario may apply to
multiple spec versions') and rename 'Scorecard' to 'Repository Health'
to match the labels used by the tier-check CLI output.
* fix: add specVersions to CrossAppAccessCompleteFlowScenario
Added in 83c446d on main after this branch diverged.
The SDK's `requireBearerAuth` middleware only converts `InvalidTokenError`
instances to HTTP 401 responses. Generic `Error` instances fall through
as HTTP 500, which prevents clients from detecting authentication failures
and initiating the OAuth refresh/re-auth flow.
This was discovered while building token refresh conformance scenarios —
the mock server was returning 500 for expired/invalid tokens instead of
the expected 401.
Co-authored-by: JD Maturen <70791+jdmaturen@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: add conformance tests for SEP-990
* Resolving review changes: Removed redundant tests, updated audience params
* fix: unused serverUrl parameter in runCrossAppAccessTokenExchange
* chore: apply prettier formatting
* fix: address PR review comments for SEP-990 conformance tests
- Delete unused separate token-exchange and jwt-bearer scenarios,
keeping only the complete e2e flow (review comment)
- Add missing required token exchange params per SEP-990 spec:
requested_token_type, audience, resource (review comment)
- Use ctx.idp_client_id for token exchange client_id instead of
AS client_id (review comment)
- Client discovers resource and auth server via PRM metadata
instead of receiving auth_server_url via context (review comment)
- Server IdP handler verifies all required token exchange params
with detailed error messages (review comment)
- Add resource, client_id, jti claims to ID-JAG per SEP-990 spec
- Verify ID-JAG typ header (oauth-id-jag+jwt) in JWT bearer handler
- Remove auth_server_url from context schema
* feat: add client auth and ID-JAG validation to XAA conformance test
Server-side (AS) now verifies:
- client_secret_basic authentication on JWT bearer grant
- ID-JAG typ header is oauth-id-jag+jwt
- ID-JAG client_id claim matches the authenticating client (Section 5.1)
- ID-JAG resource claim matches the MCP server resource identifier
- Client credentials provided via context (client_secret)
Server-side (IdP) now:
- Sets ID-JAG client_id to the MCP Client's AS client_id (not the
IdP client_id), per Section 6.1
Example client now:
- Authenticates to AS via client_secret_basic (Authorization: Basic)
instead of sending client_id in body
- Checks AS metadata grant_types_supported includes jwt-bearer
before attempting the flow
* fix: share MockTokenVerifier and remove unadvertised auth method
- Add shared MockTokenVerifier between AS and MCP server so the MCP
server only accepts tokens actually issued by the auth server,
matching the pattern used by all other auth scenarios
- Remove private_key_jwt from tokenEndpointAuthMethodsSupported since
the handler only implements client_secret_basic
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
The auth test server helper and no-dns-rebinding-protection example
reused a single Server instance across requests, calling connect()
on each new transport. SDK v1.26.0's security fix (GHSA-345p-7cg4-v4c7)
now throws "Already connected to a transport" in this case.
Fix by wrapping Server creation in a factory function called per
request, matching the pattern already used by tools_call.ts and
everything-server.ts.
Also updates no-dns-rebinding-protection.ts to use registerTool()
instead of the deprecated tool() API.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix: bump SDK to 1.26.0, add session management to tools_call scenario
SDK 1.26.0 made Protocol.connect() throw if already connected to a
transport. The tools_call scenario was calling server.connect(transport)
on every request with the same Server instance, which now throws.
Fixed by adding proper session management: each initialize request
creates a new Server + Transport pair, and subsequent requests are
routed to the correct transport via the mcp-session-id header.
* fix: return 404 for invalid session IDs in tools_call POST handler
The else branch was returning 400 for both missing-session and
invalid/stale-session cases. The MCP spec requires 404 for invalid
session IDs and 400 only for non-initialization requests without
any session ID.
* refactor: keep tools_call stateless, create fresh server per request
Instead of adding session management, simply create a new Server
instance per request. This preserves the original stateless design
while fixing the SDK 1.26.0 Protocol.connect() restriction.
Move auth/2025-03-26-oauth-metadata-backcompat and
auth/2025-03-26-oauth-endpoint-fallback out of the required
authScenariosList into a new backcompatScenariosList.
These test backward compatibility with the old 2025-03-26 auth spec
(no PRM, OAuth metadata at server root) which is not part of the
current spec requirements.
- Add new 'backcompat' client suite
- Back-compat scenarios remain in the 'all' suite
- Removed from 'core' and 'auth' suites
- Add separate test describe block for back-compat scenarios
Closes#126
* Add allowClientError to Scenario interface for expected client failures
* Apply suggestion from @pcarleton
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
---------
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
* feat: add resource parameter validation tests (RFC 8707)
Adds conformance tests for OAuth Resource Indicators (RFC 8707) implementation:
1. Resource parameter checks added to token-endpoint-auth-basic scenario:
- resource-parameter-in-authorization: Verify resource in auth request
- resource-parameter-in-token: Verify resource in token request
- resource-parameter-valid-uri: Verify valid canonical URI
- resource-parameter-consistency: Verify consistency between requests
2. New auth/resource-mismatch scenario:
- Tests that client rejects when PRM resource doesn't match server URL
- Server returns mismatched resource in PRM
- Test passes if client does NOT proceed with authorization
Also adds spec references for RFC 8707 and MCP resource parameter spec.
Closes#33
* fix: make resource consistency check a FAILURE and remove dead code
- Change resource parameter consistency from WARNING to FAILURE
- Remove unreachable protocol check in validateCanonicalUri
(URL constructor already validates scheme presence)
* feat: add PKCE conformance tests
Adds conformance checks for PKCE requirements from the MCP auth spec:
- pkce-code-challenge-sent: Verifies client sends code_challenge in auth request
- pkce-s256-method-used: Verifies client uses S256 code challenge method
- pkce-code-verifier-sent: Verifies client sends code_verifier in token request
- pkce-verifier-matches-challenge: Validates BASE64URL(SHA256(verifier)) = challenge
Also adds auth/pkce-no-s256-support scenario that tests clients correctly
refuse when S256 is not in code_challenge_methods_supported.
All PKCE checks are added to the createAuthServer helper, so they automatically
apply to all existing auth scenarios.
Fixes#77
* feat: add bad client example that skips PKCE
Adds auth-test-no-pkce.ts which implements a custom OAuth flow that
deliberately skips PKCE (no code_challenge in auth request, no
code_verifier in token request).
Also adds negative test to verify the conformance suite correctly
detects this non-compliant behavior.
* chore: skip pkce-no-s256-support scenario for now
* chore: fix formatting
* refactor: remove pkce-no-s256-support scenario and always validate PKCE
- Remove pkce-no-s256-support scenario (was skipped anyway)
- Make pkce-verifier-matches-challenge fail if either code_challenge
or code_verifier is missing, rather than only running when both present
- Update negative test to expect the new failure
* docs: add SDK integration guide and fix spec references
- Add SDK_INTEGRATION.md with guide for SDK maintainers on integrating conformance tests
- Link to guide from README.md
- Update spec references to use 2025-11-25 instead of draft URLs
* fix: prettier formatting and correct action version to v0.1.10
* feat: add pre-registration conformance test
Add conformance test for OAuth pre-registration flow where the server
does NOT support Dynamic Client Registration (DCR) and clients must
use pre-configured static credentials.
This addresses issue #34 (Client Registration Methods) - specifically
the pre-registration section that was previously not covered.
Changes:
- Add disableDynamicRegistration option to createAuthServer helper
- Add auth/pre-registration scenario with pre-registered client creds
- Add MCP_PREREGISTRATION spec reference
- Add context schema for pre-registration credentials
- Add withOAuthRetryWithProvider helper for pre-configured providers
- Add negative test client (auth-test-attempts-dcr.ts) that ignores
pre-registered credentials and fails
The scenario verifies that when a server does not advertise
registration_endpoint in its OAuth metadata, compliant clients
use pre-registered credentials passed via context instead of
attempting DCR.
Closes#34
* fix: add redirect_uris to saveClientInformation call
* chore: remove unused WithOAuthRetryOptions interface
* feat: add core and extensions suites for SDK tiering
Add new suites to support SDK tiering requirements:
Client suites:
- core (19 scenarios): Tier 1 requirements (base + auth)
- extensions (2 scenarios): Optional protocol extensions (client credentials)
- all (21 scenarios): Everything (core + extensions)
- auth (15 scenarios): Auth scenarios only (excludes extensions)
Server suites:
- core: Alias for 'active' (tier 1 requirements)
This allows SDK developers to run --suite core for tier 1 certification
while keeping extension tests available but not blocking.
* feat: promote elicitation tests from pending to active
Move 3 elicitation scenarios to active suite:
- elicitation-sep1330-enums
- tools-call-elicitation
- elicitation-sep1034-defaults
These tests are now passing in the Python SDK and ready for
SDK tiering requirements.
Remaining pending tests (2):
- json-schema-2020-12 (blocked on SDK PR #1135)
- server-sse-polling (blocked on SDK PR #1129)
* Add client credentials support to everything-client
- Add ConformanceContextSchema with discriminated union for auth contexts
- Add runClientCredentialsJwt and runClientCredentialsBasic handlers
- Update runner to include scenario name in context for discriminated union parsing
- Update test helpers to set env vars for inline client runners
- Fix JWT audience validation to match SDK behavior (no trailing slash)
- Remove client credentials scenarios from skip list
* Accept both trailing slash forms for JWT audience per RFC 3986
Per RFC 3986, URLs with and without trailing slash are equivalent.
The MCP spec recommends (SHOULD) the form without trailing slash, but
since it's not a MUST, the conformance test should accept both forms
for interoperability with clients like Pydantic that normalize URLs.
* Remove .vscode from tracking and add to gitignore
* Address review feedback
- Remove unused issuerUrl variable and comment
- Simplify audience array to exactly 2 values with clearer naming
- Add comment explaining strip-then-add-back logic
- Rename ConformanceContextSchema to ClientConformanceContextSchema
* Remove dist from tracking and add to gitignore
* Fix .gitignore formatting
* Add getHandler to everything-client for cleaner test imports
Instead of importing individual handlers, tests can now import getHandler
and look up handlers by scenario name.
* Use getHandler exclusively, remove goodClient import
- Add CIMD support to runAuthClient in everything-client
- Add auth fallback to getHandler for unregistered auth/* scenarios
- Remove goodClient import from test file
* Remove runAuthClient fallback, register all auth scenarios explicitly
* [auth] Add metadata suite for running metadata discovery tests
Adds a new 'metadata' suite that runs just the auth/metadata-* scenarios
for faster iteration when testing metadata discovery specifically.
Usage: node dist/index.mjs client --suite metadata --command "..."
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Don't require lefthook to be installed
Set assert_lefthook_installed to false so the hooks gracefully skip
if lefthook is not installed on the system.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add token endpoint auth method conformance tests
Add three new scenarios to test that clients correctly use the
appropriate authentication method based on server metadata:
- auth/token-endpoint-auth-basic: Tests client_secret_basic (HTTP Basic)
- auth/token-endpoint-auth-post: Tests client_secret_post
- auth/token-endpoint-auth-none: Tests public client (no auth)
Each scenario configures the server to only support one auth method
and verifies the client uses the correct method in token requests.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Refactor token endpoint auth tests to use shared createAuthServer
Extend createAuthServer helper with:
- tokenEndpointAuthMethodsSupported option for metadata
- onTokenRequest callback now receives full Request object
- onRegistrationRequest callback for custom client credentials
This eliminates the duplicate auth server implementation in
token-endpoint-auth.ts and reduces code by ~140 lines.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix onTokenRequest callback to use new interface
The createAuthServer onTokenRequest callback now passes an object with
authorizationHeader, body, timestamp etc. instead of (req, timestamp).
Also added missing return value for TokenRequestResult.
* Update @modelcontextprotocol/sdk to 1.25.1
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Olivier Chafik <ochafik@anthropic.com>
- Update client initialization check to accept both 2025-06-18 and 2025-11-25
- Server echoes back client's version if valid (proper version negotiation)
- Fixes compatibility with SDK 1.25.1+ which sends 2025-11-25
🦉 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* fix: validate scope checks ran and report overall test status
- Add FAILURE checks to scope scenarios when authorization flow doesn't
complete (ScopeFromWwwAuthenticateScenario, ScopeFromScopesSupportedScenario,
ScopeOmittedWhenUndefinedScenario)
- Report client timeout and exit errors in test results
- Add OVERALL: PASSED/FAILED summary at end of test output
- Exit with code 1 on any failure (check failures, client timeout, or exit error)
Previously, if the OAuth authorization flow didn't complete, the scope
checks were never triggered and the test would incorrectly report success.
Now these scenarios emit a FAILURE if the expected scope check wasn't recorded.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Count warnings as failures in overall test result
Warnings now contribute to the overall failure status, causing the test
to exit with code 1 when any warnings are present. Also displays warning
checks in the output summary, similar to how failures are displayed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Count warnings as failures in suite runner
Update the suite summary to show ✗ for scenarios with warnings,
and exit with code 1 if any warnings are present across the suite.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Add client credentials conformance scenarios with dynamic keypair generation
Implements SEP-1046 client credentials conformance tests:
- auth/client-credentials-jwt: Tests private_key_jwt authentication
- auth/client-credentials-basic: Tests client_secret_basic authentication
Key changes:
- Generate EC P-256 keypair dynamically at test start (no hardcoded keys)
- Pass credentials to client via MCP_CONFORMANCE_CONTEXT environment variable
- Add context field to ScenarioUrls interface for scenario-specific data
- Update client runner to pass context as env var to spawned client process
The MCP_CONFORMANCE_CONTEXT env var contains a JSON object with:
- client_id: The expected client identifier
- private_key_pem: PEM-encoded private key (for JWT scenarios)
- client_secret: Client secret (for basic auth scenarios)
- signing_algorithm: JWT signing algorithm (defaults to ES256)
* Fix client-credentials scenarios for conformance testing
- Generate EC keypair with extractable: true so private key can be
exported and passed to clients via MCP_CONFORMANCE_CONTEXT
- Fix client_secret_basic scenario to use authorizationHeader param
instead of non-existent headers object
* Fix jose type import - use CryptoKey instead of KeyLike
* Add OAuth 2.1 section 4.2 spec reference to client credentials checks
Add OAUTH_2_1_CLIENT_CREDENTIALS spec reference pointing to OAuth 2.1
draft section 4.2 (Client Credentials Grant) and include it in all
client_credentials conformance checks for both JWT and basic auth flows.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add JWT iss claim verification for client credentials
Per RFC 7523, verify that the JWT issuer (iss) claim matches the
expected client_id, in addition to the existing sub claim check.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* lefthook: move deps installs to optional local change
* add test scenario for infinite retry
and negative test
* Add broken client for retry limit negative test and fix lefthook formatting
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix www-authenticate in scope scenario
* add suites and shift pending to reflect current pending
* lint
---------
Co-authored-by: Claude <noreply@anthropic.com>
* add cimd test and negative test
* Improve auth-test-no-cimd.ts documentation for clarity
Address review feedback: expanded comments to better explain what CIMD is,
why this client is non-compliant, and what behavior it's testing.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Remove .js extensions from relative imports
tsx doesn't require .js extensions for TypeScript imports.
Keeps .js for @modelcontextprotocol/sdk imports as those are external packages.
Also improves auth-test-no-cimd.ts documentation for clarity.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Skip auth/basic-cimd test until SDK support lands
Waiting on typescript-sdk PR #1127 for CIMD support.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix trailing whitespace in auth-test-no-cimd.ts
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* [auth] Add metadata suite for running metadata discovery tests
Adds a new 'metadata' suite that runs just the auth/metadata-* scenarios
for faster iteration when testing metadata discovery specifically.
Usage: node dist/index.mjs client --suite metadata --command "..."
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Don't require lefthook to be installed
Set assert_lefthook_installed to false so the hooks gracefully skip
if lefthook is not installed on the system.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* wip scope change
* rm useless bool
* fix-me: package changes for linking
* negative test
* negative test for multiple scopes
* fix displays of warnings
* wip step-up
* negative test for step up auth
* move warning into main request flow
* clean up more checks
* cleanup step-up scenario
* cleanup and simplify
* cleanup middleware
* inline negative tests
* inline the other test
* refactor logging and inlining for standalone execution too
* skip pending tests
* clean up constructor calls for server lifecycle
* add back compat test
* fix dup logging, rm prm
* avoid fallbacks
* better error from example client
* add final endpoint method
* add test for OASM priority order
* s/prepack/prepare/ for npx
* try switching to .js
* make tsdown version happy default
* fix lock
* fix prmPath issue
* back to main package-lock
* fix bin path
* Add comprehensive auth scenario test suite
- Add index.test.ts to test all auth scenarios with auth-test.ts client
- Upgrade @modelcontextprotocol/sdk from 1.20.1 to 1.22.0
- All 5 auth scenarios now pass including auth/basic-metadata-var2
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* consolidate tests into index.test.ts
* rename metadata
* fix comments
* rework auth scenario listing
* add 3rd variation of metadata
* fix some type issues introduced by version bump
* add reusable spec references
* more refs
* fix up spec references
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Add SEP-1330 conformance test for elicitation enum schemas
Adds server-side conformance test that validates servers properly request
elicitation with SEP-1330 enum schema improvements:
- Untitled single-select enum (type: string, enum: [...])
- Titled single-select enum (oneOf with const/title)
- Legacy titled enum (enumNames for backward compatibility)
- Untitled multi-select enum (type: array, items.enum)
- Titled multi-select enum (items.anyOf with const/title)
Test expects server to implement `test_elicitation_sep1330_enums` tool.
* Add reference server implementation for SEP-1330 conformance test
Adds test_elicitation_sep1330_enums tool to everything-server that
requests elicitation with all 5 enum schema variants. Also updates
SERVER_REQUIREMENTS.md with full specification.
* Fix prettier formatting
* only test active tests (#24)
* refactor maps to pull names from the test
* bring names back
* separate active from pending
* only test active
---------
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
* feat: add client-side SEP-1034 conformance test for elicitation defaults
Adds a conformance test that validates client SDKs properly apply
default values for omitted fields in elicitation responses per SEP-1034.
The test spins up a server that sends elicitation requests with defaults,
and validates that the client returns content with those defaults applied.
* feat: use elicitInput() for SEP-1034 default validation
Updates client-side test to validate server-side default application
using elicitInput() helper, which applies defaults via validation layer.
This aligns with the TypeScript SDK's SEP-1034 implementation approach.
The test now validates:
- Server uses elicitInput() with schemas containing defaults
- Client returns empty/partial content
- Server automatically applies defaults during validation
- Defaults are correctly merged for all primitive types
* refactor: use server.request() instead of elicitInput() for client-side defaults test
Ensures conformance test validates true client-side default application
rather than server-side validation behavior.
* feat: add applyDefaults capability flag for opt-in default application
Enables conformance test to opt into client-side elicitation defaults
by setting the applyDefaults capability flag.
* readme for auth tests
* wip prm test
* add example helper for local oauth w/o browser
* working through dangling timeout issue
* add better test function
* pretty printing
* add mcp method
* some more logging tweaks
* refactor logger into 1 class
* use common request logger
* fix base scenario
* some renaming
* refactor into helpers
* refactor to support second test
* rm readme
* add comment about www-authenticate
Felix Weinberger
Paul Carleton
Caitie McCaffrey
Den Delimarsky
Max Gerber
Anubhav Dhawan
Mike Kistler
Wils Dawson
Daniel Garnier-Moiroux
jd
sagar-okta
Ola Hungerford
Matt