* fix(validation): accept extension result envelopes
This fixes#424
Signed-off-by: King Star <mcxin.y@gmail.com>
* validation: only known extension resultTypes take the envelope-only path
Restrict the generic-envelope fallback to resultType values a known
extension defines (today: 'task', the tasks extension's CreateTaskResult).
Any other non-core resultType is still validated as the request method's
own result — the discriminator is open, so it may be a private extension —
and a resulting violation names the unrecognised value so a typo or an
unknown extension is obvious instead of silently passing envelope-only.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Signed-off-by: King Star <mcxin.y@gmail.com>
Co-authored-by: triage <paulc@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
SEP-2243 permits `x-mcp-header` only on integer, string and boolean
parameters — "Parameters with type `number` are not permitted" — but the
harness never verified the `number` case and its requirement text still
described `number` as permitted.
PR #371 fixed the positive `http-custom-headers` fixture (priority is now
`integer`, `float_val` is served unannotated) but left the two remaining
items from #344:
- `HttpInvalidToolHeadersScenario` serves invalid tools for every other
x-mcp-header constraint but had no number-typed case, so nothing checked
that a client excludes such a tool.
- `sep-2243.yaml` claimed primitive types are "(number, string, or
boolean)" and that "number values" get decimal-string encoding, both of
which contradict the released 2026-07-28 spec.
Add an `invalid_number_header` tool mapped to the existing
`sep-2243-x-mcp-header-primitive-only` requirement (no new check IDs) and
correct both requirement texts. traceability.json is updated in place
since it embeds the yaml text verbatim; check IDs and statuses are
unchanged.
Verified against typescript-sdk@main: the new check fails there because
the SDK deliberately allow-lists `number` to satisfy the old fixture — a
workaround its own source notes as "tracked upstream" and can now drop.
The positive scenario still passes 18/18.
Closes#344
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* feat(tools-list): check deterministic tools/list ordering on the 2026-07-28 wire
The 2026-07-28 revision adds to server/tools.mdx: "Servers SHOULD return
tools in a deterministic order (i.e., the same ordering across requests
when the underlying set of tools has not changed)." Nothing in the suite
exercised it.
Add a tools-list-deterministic-order check to the existing tools-list
scenario. From the 2026-07-28 wire on, the scenario takes two more
consecutive tools/list snapshots and compares the three name sequences.
Same multiset of names, different order is WARNING (SHOULD). A set that
changed between probes is reported as untestable rather than as a
violation, because the spec scopes the SHOULD to an unchanged set. Fewer
than two tools on every probe is INFO. The check carries
source.introducedIn '2026-07-28', written as a literal since that revision
is published and dated, so runners drop it on earlier wires.
Prove it fails: examples/servers/typescript/tools-list-rotated-order.ts
advertises the same four tools and rotates the list by one position per
call; a negative.test.ts case asserts WARNING while tools-list itself
still passes. Unit tests cover the helper that builds the check.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjeC3sCvfHRpdEPz2x41B2
* tools-list: inline the deterministic-order check id and introducedIn literal
Check ids and introducedIn revisions are written inline everywhere else in
src/scenarios (tests pin the literal slug on purpose, since SDK baselines
reference it), so drop the two exported constants.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* fix(tools-list): align tools-name-format with 2025-11-25 spec prose
Enforce dated spec Tool Names SHOULD rules (128 max, no slash), WARNING
severity, 2025-11-25+ gate, negative fixture, and core-spec specReferences.
Fixes#379
* test(tools-list): cast Connection.request mock for tsgo typecheck
* test(tools-list): pin WARNING to offending tool name in negative test
* test(tools-list): rename negative fixture to invalid-tool-names
Document SEP-986 markdown vs integrated spec divergence in fixture header.
* chore: trim explanatory comments to the essentials
Keep the rule statement and the note that the dated spec (not the SEP-986
markdown) is authoritative; drop the process narration.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: triage <paulc@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
Adds resource-parameter-matches-prm to the RFC 8707 checks, emitted by the
token-endpoint-auth and metadata-discovery scenarios; auth/metadata-var2 serves
a pathless resource, so a client that re-serializes it through URL.href fails.
Negative example client included. The everything-client (SDK 1.x) is baselined
on that scenario until typescript-sdk#1972 ships.
Bumps the npm_and_yarn group with 1 update in the / directory: [ip-address](https://github.com/beaugunderson/ip-address).
Bumps the npm_and_yarn group with 1 update in the /examples/servers/typescript directory: [ip-address](https://github.com/beaugunderson/ip-address).
Updates `ip-address` from 10.1.0 to 10.2.0
- [Commits](https://github.com/beaugunderson/ip-address/commits)
Updates `ip-address` from 10.1.0 to 10.2.0
- [Commits](https://github.com/beaugunderson/ip-address/commits)
---
updated-dependencies:
- dependency-name: ip-address
dependency-version: 10.2.0
dependency-type: indirect
dependency-group: npm_and_yarn
- dependency-name: ip-address
dependency-version: 10.2.0
dependency-type: indirect
dependency-group: npm_and_yarn
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
The retry MUST ("waiting the given number of milliseconds before
attempting to reconnect") is a lower bound; no spec text bounds reconnect
delay from above. The slightly-late WARNING and >2x FAILURE bands gated
environment latency, which CI load inflates - in python-sdk CI the warning
band fired on ~4-5% of runs under the suite's parallel client spawns.
- delete the late bands: FAILURE only when the reconnect arrives earlier
than retry minus the 50ms tolerance
- demote the could-not-measure path from WARNING to INFO
- negotiate protocolVersion 2025-11-25, the version whose MUST is under
test (was 2025-03-26; same fix as 4686e06 for server-sse-polling)
Strictly relaxing: no client that passes today can start failing.
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
The client runner has had a 30s per-scenario timeout since it was written,
but `runServerConformanceTest` awaited `scenario.run(ctx)` with no bound. A
server under test that accepts the TCP connection and then never answers an
initialize POST, a tools call, or an SSE read stalled the entire suite run,
not just its own scenario.
#316 bounded the session-termination DELETEs in teardown, but that is a
call-site fix: covering the rest that way needs the same treatment on every
raw fetch in every scenario. Bounding at the runner makes those per-call
bounds defense-in-depth instead of the only protection.
`--timeout` now applies to server mode too, with the same 30s default as
client mode. On expiry the scenario reports a `scenario-timeout` FAILURE
naming the elapsed bound, so a wedged server reads as a failing scenario
rather than a hung process.
The timed-out scenario promise is left pending deliberately — a read blocked
on a socket has no cancellation channel — with its rejection swallowed so a
late failure can't surface against whichever scenario is running by then.
Fixes#427
Co-authored-by: Avi Seth <aviseth@users.noreply.github.com>
A negative check that reads only the final verdict scores SUCCESS whenever
the client fails to reach the requirement at all: "declined" and "never got
far enough to decide" are the same observation. auth/resource-mismatch
decided on !authorizationRequestMade alone, so a client that threw before
making a single HTTP request passed it.
Gate each verdict on the prerequisite the harness already records:
prm-pathbased-requested for the PRM resource comparison, and
authorization-server-metadata for the RFC 9207 iss comparisons, which
SEP-2468 conditions on the issuer recorded from the validated metadata
document. When the prerequisite is not observed the requirement was never
exercised, so report it via notTestable() (#248) rather than SUCCESS.
auth/metadata-issuer-mismatch already gated on the metadata fetch but
reported the unreached case as a plain violation; it now reports untestable
too. All six checks record propertyReached and stopReason in details.
Adds an inert client and a vitest case, so the checks are shown to catch
something and not merely to avoid false-positiving.
Version detection stripped only a leading `v`, so the monorepo tags
Cargo workspaces publish (`rmcp-v3.0.1`, `rmcp-macros-v3.0.1`) parsed
to NaN and scored `stable_release: fail`. rust-sdk was reported as
having no stable release despite a non-prerelease `rmcp-v3.0.1`.
Match the version at the end of the tag instead, so any package-name
prefix is ignored while bare (`1.2.3`), `v`-prefixed and prerelease
(`v2.0.0-rc1`) tags parse as they did before. Tags carrying no
recognizable version are returned unchanged rather than dropped, so
they still surface in the check output.
Fixes#425
`input-required-result-capability-check` guarded its elicitation scan with
`isInputRequiredResult(result) && result.inputRequests`. When a server answered
without `inputRequests`, that arm was skipped, and the following
`isCompleteResult` arm is false by construction for an input_required result, so
nothing was recorded and the check passed. An empty `inputRequests` object passed
too: the arm was taken, but the loop had nothing to scan.
`inputRequests` is optional — a result carrying only `requestState` still
satisfies "at least one of inputRequests or requestState" — so such a server has
not violated "servers MUST NOT send an inputRequests that the client has not
declared support for in its capabilities". It has not exercised it either.
Reporting SUCCESS makes an unverified MUST read as green, which is the failure
mode #248 and #372 established should be reported as a failure instead.
So report it through `notTestable()` with `details.untestable`, matching the
convention in AGENTS.md and the call sites in stateless.ts, rather than claiming a
violation the server did not commit. Servers that legitimately have nothing to ask
for now go red on this check; per #248 the expected-failures baseline is the
escape hatch.
Adds the case to sep-2322-mrtr-broken-server so the existing MRTR negative suite
covers it. The fixture returns a conformant envelope (`requestState` present, no
input requests) rather than a bare one, so it isolates this check rather than also
tripping sep-2322-request-state-incomplete.
Verified against --spec-version 2026-07-28:
everything-server 2/2 passed
sep-2322-mrtr-broken-server FAILURE Not testable: server returned no
inputRequests, so the capability restriction
was never exercised
Refs #439
Co-authored-by: Marcus <238969485+marcus-kepler-92@users.noreply.github.com>
The visible-ASCII session-id check sends a raw initialize POST outside the
SDK transport, because the transport does not expose the MCP-Session-Id
response header. That fetch carried no abort signal and its body was never
read or cancelled.
A server that accepts the connection and never responds hung the scenario
here, before teardown, so the DELETE bound added in #316 never came into
play. And since the probe's `Accept` includes `text/event-stream`, a server
answering with an open SSE stream left the body unconsumed and leaked the
connection for the rest of the run.
Bounding the probe at 5s matches the existing session-termination bound.
#427 is the structural fix for the hang class; this is still worth having
for fast failure and for attributing the failure to this check rather than
to whatever the runner-level timeout happens to interrupt.
Fixes#428
Co-authored-by: Avi Seth <aviseth@users.noreply.github.com>
The BaseHttpScenario generic fallback answered any unrouted request
with only the required draft result fields, so a standard list method
a scenario does not route came back without its required list member.
http-standard-headers advertises resources but has no
resources/templates/list route; the reply's missing resourceTemplates
fails schema validation (draft and 2025-11-25 both require it), and
strictly-validating clients drop the connection before the scenario's
header checks run.
Merge schema-valid empty results for the six list-shaped methods
(tools, resources, resource templates, prompts, roots, tasks) into
the fallback, keyed by a ReadonlyMap so method names that collide
with Object.prototype ("constructor", "toString", ...) miss instead
of poisoning the reply with a prototype function. tasks/list has a
typed result only at 2025-11-25; the empty member is harmless on the
draft wire. Non-list results (tools/call, resources/read,
prompts/get, ...) have no meaningful empty default and keep the bare
stamped fallback, so a route a scenario forgot surfaces instead of
being masked.
Tests pin the fixed behaviour on all three http scenario classes,
the prototype-name lookups, and the spec schema's rejection of the
pre-fix payload.
Fixes#474
Registers modelcontextprotocol/ruby-sdk in KNOWN_SDKS so `tier-check --sdk-path` and `sdk ruby-sdk` manage
the build and per-revision runs themselves, instead of requiring the --conformance-server-url/--client-cmd
fallback for unknown SDKs. This is the registry path the Ruby SDK's SEP-1730 Tier 1 application
(modelcontextprotocol/modelcontextprotocol#3247) was scored through: with the entry in place, reviewers can
reproduce the submission with a single `tier-check --sdk-path <checkout> --requirements 2025-11-25,2026-07-28`,
the same way every other registered SDK is checked.
The entry drives the same fixtures the SDK's own CI runs through `rake conformance`: `conformance/server.rb`
behind the `conformance:server` rake task (PORT from the environment; PORT=3000 matches the registry's port convention)
and `conformance/client.rb`, which reads MCP_CONFORMANCE_SCENARIO from the environment and the server URL from argv,
per the runner's contract. The baseline is `conformance/expected_failures.yml`. One dual-era server serves every revision
from the same /mcp endpoint (stateful dated-spec handshake and the SEP-2575 stateless lifecycle side by side),
so the entry needs no specOverrides.
Testing: `npm test` (525 passing, including a new known-sdks entry test), `npm run check`, and `npm run build` pass.
The entry has already carried four full end-to-end audit runs: the 2026-08-14 and 2026-08-15 tier-check runs against
a local ruby-sdk checkout (`--sdk-path` with `--requirements 2025-11-25,2026-07-28`) each built the SDK, started the
server on PORT=3000, and completed both legs at both revisions, producing the 67/67 server and 50/50 client results
submitted in modelcontextprotocol/modelcontextprotocol#3247. Individually, `sdk ruby-sdk --path <checkout> --mode
server --requirements 2025-11-25` exits 0 with 78/78 checks, and `--mode client --requirements 2025-11-25` exits 0
with every scored scenario passing (failures confined to not_scored extensions).
Routes review of /src/scenarios/authorization-server/ and
/src/runner/authorization-server.ts to the conformance-auth team
(managed in modelcontextprotocol/access).
Claude-Session: https://claude.ai/code/session_01FDRFtxQbYLnBcpwndGXF7W
Co-authored-by: Claude <noreply@anthropic.com>
Lockstep SDKs publish in the hours before the spec release is tagged
(for 2026-07-28: python-sdk 3h early, go-sdk 3.5h, typescript-sdk the
evening before), so "first SDK release at or after the spec" read them
as not having released and would fail them 30 days later. Allow a 24h
lead.
There is no way to answer "which scenarios must my implementation pass
to conform to spec revision X". The suite accumulates scenarios
continuously, so what an implementation is measured against keeps
growing after the revision shipped, and it can drop below 100% while
standing still. --spec-version does not answer it: it filters on which
revision a scenario targets, not on when the scenario became a
requirement.
Add requirements/<revision>.yaml naming the scenarios a revision
requires, and --requirements <revision> on the client, server, list and
sdk commands to run exactly that set. The revision also fixes the wire
version its scenarios speak: the dated revisions through 2025-11-25 use
the stateful initialize handshake and 2026-07-28 is stateless with
per-request _meta, so a scenario belonging to two revisions runs once
under each and one run does not cover the other. Passing no flag keeps
today's behaviour of running everything.
tier-check gains --sdk <name[@ref]> / --sdk-path <dir>: for each claimed
revision it resolves that revision's build, server invocation and client
command from the sdk-runner's KNOWN_SDKS specOverrides, manages the
server, runs both legs with cwd = the SDK checkout, and merges. Every
claimed revision must pass for Tier 1. This is what makes multi-revision
tiering correct for SDKs whose wire era is fixed per server process
(go-sdk only speaks >= 2026-07-28 when constructed stateless) or per
endpoint (csharp-sdk serves it at /stateless): one
--conformance-server-url cannot describe them, and their config already
does. The single-URL mode remains for SDKs not in the config.
Ship two sets. 2026-07-28 is a snapshot of 0.2.0-alpha.10, published the
day before that revision shipped. 2025-11-25 is a reconstruction from
the same anchor and its header says so: the release current on its ship
date predates spec-version awareness entirely. not_scored entries run
and are reported with a reason (extension per SEP-1730, or
added-after-release) but never affect a pass rate or the exit code, and
--expected-failures judges scored scenarios only. Reports label every
entry with its revision, and JSON passed/failed/total describe the
scored set so they always agree with pass_rate.
tier-check's exit code now gates on the machine-checkable half: nonzero
when a scored scenario fails or a leg could not be measured. A child run
that cannot happen at all (a set naming a renamed scenario, an
unreachable server) reports as not measured rather than as 0% against
the implementation. Conformance runs to completion before the GitHub
checks, so an API failure can no longer interrupt a leg and orphan a
managed server, and the legs spawn their child asynchronously: the
previous synchronous exec blocked the event loop, the parent stopped
draining the managed server's log pipes, and a chatty server (rust-sdk's
per-request tracing) froze on a full pipe and was scored as failing.
Driving every tier-1 SDK through the new path also fixed three configs:
csharp-sdk's server now launches from its build-output directory so
ASP.NET finds appsettings.json (its AllowedHosts filter, which
dns-rebinding-protection tests, silently never loaded from the repo
root); rust-sdk's STATELESS=1 override is removed because its CI runs
one default server for both revisions and the forced mode fails its
SEP-2575 scenarios; python-sdk's build mirrors its CI's two targeted
syncs because --all-packages is unbuildable on main. withManagedServer
now waits for its child to exit before returning, so consecutive runs on
one port cannot race a dying predecessor.
* feat: add client-side JSON Schema 2020-12 preservation scenario
The mock server, using the version-aware `MockServer` abstraction from #321,
first advertises a focal tool with a detailed inputSchema and also offers a
permissive `json_schema_echo` tool. The client under test retrieves the
inputSchema via `tools/list`, then sends the same inputSchema back to the server
using the echo tool, completing the round-trip.
The scenario diffs from the echoed schema against the canonical fixture. It
emits eight per-keyword conformance checks. SEP-2106 vocabulary checks are
soft-gated using the existing sep2106KeywordCheckStatus helper. They only flip
to FAILURE during DRAFT-2026-v1 runs.
Signed-off-by: Satoshi Ito <satoshi.ito.tf@hitachi.com>
* Drive draft-target preservation tests through the stateless lifecycle
The stateful test helper passed createServerStateful bare, which no longer
compiles now that it takes a second specVersion argument, and forcing the
stateful mock at the draft version trips wire-schema validation (draft
results require the SEP-2549 resultType/ttlMs/cacheScope members). Use
testScenarioContext(DRAFT_PROTOCOL_VERSION), which selects the stateless
mock server, and drive it with sendStatelessRequest, matching the pattern
in json-schema-ref-deref.test.ts.
* Reword client keyword-preservation rows as issue-derived expectations
SEP-2106 §1 contains no normative preservation sentence, so the six new
sep-2106.yaml rows should not attribute a spec MUST to it. Reword the
remaining rows as expectations derived from issue #101. Also drop the
three rows whose check IDs are not sep-prefixed: the traceability
manifest only collects IDs matching /^sep-\d+-/, so those rows could
never be marked tested.
* Replace stale DRAFT-2026-v1 references with the 2026-07-28 draft name
The draft protocol identifier was renamed in #331. Interpolate
DRAFT_PROTOCOL_VERSION into the SKIPPED errorMessage suffix (matching
the server-side scenario) and update the header comment.
---------
Signed-off-by: Satoshi Ito <satoshi.ito.tf@hitachi.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* feat: add session-lifecycle conformance scenario for streamable HTTP
This commits a `server-session-lifecycle` scenario that verifies the server honors the two RFC 2119 statements the Streamable HTTP transport spec places on session termination:
1. After receiving an HTTP DELETE bearing the issued Mcp-Session-Id, the server accepts it (2xx) or signals that explicit termination is not supported (405).
2. Subsequent requests bearing the terminated session ID MUST get HTTP 404 Not Found.
The scenario inlines a raw `fetch` initialize/terminate/probe flow so the DELETE is the test action (not background cleanup).
Stateless servers that never issue a session ID are reported as INFO, and the lifecycle checks are SKIPPED.
Servers that return 405 on DELETE skip both checks without flagging a failure — the spec allows servers to refuse explicit termination.
Refs #79
Signed-off-by: Satoshi Ito <satoshi.ito.tf@hitachi.com>
* fix: address review feedback on session-lifecycle scenario
- Adapt run() to the RunContext signature introduced in #318.
- Send the protocol version the server negotiated in the initialize
result on follow-up requests instead of hardcoding 2025-11-25,
falling back to the requested spec version (the sse-multiple-streams
pattern from #415/#416). Parsing the initialize body is bounded by a
5s timeout, and every fetch shares one AbortController aborted when
the scenario ends, so a server answering with a never-ending SSE
stream can neither hang the run nor leak a held-open connection.
- Fail the scenario when initialize itself fails; previously a broken
server (401/500) was reported as stateless and passed.
- Check the initialized notification response and fail when it is
rejected, emitting SKIPPED placeholders for the lifecycle checks so
per-check aggregation keeps its rows.
- Report unexpected DELETE statuses as WARNING instead of FAILURE (the
spec never pins a success status for session termination).
- Treat 404 on DELETE as an already-terminated session only when the
follow-up request confirms the session is gone; a 404 from a server
whose session still responds (likely no DELETE route) is a WARNING
with the 404 check skipped, rather than a misattributed failure.
- Exclude the scenario from the draft spec version via removedIn:
sessions were removed entirely by SEP-2575.
---------
Signed-off-by: Satoshi Ito <satoshi.ito.tf@hitachi.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* fix: send HTTP DELETE to terminate server sessions after each scenario
The conformance test harness opened sessions against servers under test but never issued an HTTP DELETE to terminate them, leaving a dangling session on the server after each scenario.
This violates the "hermetic" expectation called out in the Streamable HTTP transport spec.
- `connection/sdk-client.connectToServer` now calls `transport.terminateSession()` before `client.close()`. A 405 response (DELETE not supported) is tolerated because the spec allows the server to refuse.
- A new `terminateSessionRaw` helper handles scenarios that bypass the SDK client and open sessions via raw `fetch` (`lifecycle.ts`, `http-standard-headers.ts`).
- `sse-polling.ts` and `sse-multiple-streams.ts`, which manage their own transport, now terminate the session in their existing `finally` block before closing the client.
- `lifecycle.test.ts` gains two cases asserting that DELETE is sent with the issued session ID and that no DELETE is sent when the server did not assign one.
Verified manually against `examples/servers/typescript/everything-server.ts` that running a server scenario now causes the server to log `Received session termination request for session <id>`.
Fixes#79
Signed-off-by: Satoshi Ito <satoshi.ito.tf@hitachi.com>
* Use negotiated protocol version for teardown DELETE and bound it
Review follow-ups on the session-termination change:
- terminateSessionRaw: abort the DELETE after 5s so a wedged server
cannot stall suite teardown.
- terminateSessionBestEffort: shared SDK-path teardown helper that
delegates to terminateSessionRaw with the transport's negotiated
protocol version, so the socket is truly released at the bound and
the version-mismatch 400 on strict servers (the failure mode of
issue #79) goes away. Used by connectToServer close() and both SSE
scenarios, replacing their inline try/catch blocks.
- lifecycle.ts: the raw session-id probe and its teardown DELETE now
use ctx.specVersion instead of a hardcoded 2025-11-25, matching the
negotiated-version convention from #415/#416.
- Add sdk-client.test.ts regression tests asserting close() sends a
DELETE with the issued session ID (and none without one), so a bad
future merge cannot silently drop the fix.
---------
Signed-off-by: Satoshi Ito <satoshi.ito.tf@hitachi.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* feat(sdk-runner): add rust-sdk to KNOWN_SDKS
The mcp-conformance package's fixture bins follow the harness contract
directly (URL argv + MCP_CONFORMANCE_SCENARIO env), and the server
selects the SEP-2575 stateless lifecycle via STATELESS=1. Requires the
repo's pinned Rust toolchain (rustup installs it on demand).
* feat(sdk-runner): per-spec-version config overlays (specOverrides)
Several KNOWN_SDKS entries needed a different invocation when targeting
2026-07-28 — go-sdk's -stateless flag, csharp-sdk's /stateless URL,
rust-sdk's STATELESS=1 env, python-sdk's per-revision baseline — and each
carried a comment telling the caller which flag to paste. Encode those
variants in the entry instead: specOverrides maps a canonical spec
version to a partial config that is merged over the base entry when a
run targets that version (--spec-version, after 'draft' alias
resolution, or the entry's own specVersion default). Explicit CLI flags
still win.
Keys are validated against the known spec versions at schema level so a
'draft' key or a typo'd date fails loudly instead of silently never
matching.
* style: prettier fixes in sdk-runner tests
* refactor(sdk-runner): reuse isSpecVersion for specOverrides key validation
Reuses the canonical spec-version list from #421 instead of keeping a
separate copy; SPEC_VERSION_TIMELINE is now exported so the validation
message can enumerate the valid keys.
Follow-ups from the #399 review:
- Stateless mock: validate error replies with a placeholder id so the
JSON-RPC-mandated id:null is not recorded as a harness violation, and
skip inbound validation when express left the body unparsed (wrong or
missing Content-Type) — both were false positives blamed on the wrong
party. Same unparsed-body guard in the stateful mock.
- Stateful mock: validate client traffic against the protocol version
the client declares via MCP-Protocol-Version, falling back to
2025-03-26 when the header is absent (it only exists from 2025-06-18,
whose spec says to assume 2025-03-26), instead of always using the
run version.
- sdk-client: validate batches once as a whole array instead of also
per element, which double-recorded every violation.
- Make specVersion required on connectToServer/connectStateful/
createServerStateful; add isSpecVersion to types.ts.
- Extend the NumberSchema erratum to 2025-06-18 (same generator bug on
minimum/maximum, modelcontextprotocol#3139) and add a tripwire test
that forces erratum deletion once the schemas are re-vendored.
- Document the wire-schema-valid / wire-schema-harness-error check IDs
in the README.
* Vendor schema/{version}/schema.json alongside schema.ts
sync-schema now copies the spec repo's JSON Schema for each version into
src/spec-types/{version}.schema.json (same pinned SHA recorded in SOURCE:
modelcontextprotocol@f817239f4d6b1efff2c4dfc2f7af85c985d73076). These are
the inputs for runtime wire-message validation. Vendored verbatim, so they
join schema.ts in .prettierignore.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JHNrpU3Vr6Q1rPZiji7zLY
* Validate all wire messages against the per-version spec JSON schema
Every JSON-RPC message the harness sends or receives is now validated
(ajv) against the vendored schema.json for the run's spec version, at the
wire choke points: sendStatelessRequest (outbound request, JSON response,
SSE events), connectStateful (request/result/notifications), and both mock
servers (inbound client requests, outbound responses/handler results).
Beyond the JSONRPCMessage envelope, messages are validated against the
most specific definition the schema has: typed requests/notifications by
method const, typed error responses by error.code const (e.g. -32021 ->
MissingRequiredClientCapabilityError, the #376 hallucination), and typed
results via the XxxRequest -> XxxResult pair, with SEP-2322
InputRequiredResult discriminated on resultType.
Violations are recorded per scenario and surfaced by the runners as a
synthetic wire-schema-valid check (FAILURE when the implementation under
test sent an invalid message) plus a wire-schema-harness-error check when
the invalid message was harness-authored. A vitest-wide afterEach guard
fails any self-test that recorded violations, so hallucinated fixtures
can't ship (#376, #271). Opt-outs are explicit and greppable: per-call
skipValidation for deliberately malformed traffic, takeWireViolations()
drains in tests that exercise deliberately broken fixtures.
Offenders the new validation caught, now fixed:
- everything-server stateless path omitted resultType (and the caching
hints on server/discover) required by the draft schema
- sep-2164-empty-contents fixture omitted resultType/ttlMs/cacheScope
(its intended violation, empty contents, is schema-valid prose-only)
- connection/mock-server unit-test fixtures used schema-invalid stubs
(tools/call without name, ListToolsResult without required members,
Tool without inputSchema, ProgressNotification without progressToken,
ElicitRequest without message/requestedSchema)
- http-standard-headers test mocks returned bare { tools: [] } results
- input-required-result-validate-input deliberately sends malformed
inputResponses; now marked skipValidation
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JHNrpU3Vr6Q1rPZiji7zLY
* Validate the stateful wire at the SDK transport, close review-panel gaps
Address the review-panel findings on the wire-schema validation PR:
- Stateful wire: hook the SDK transport inside connectToServer, wrapping
send/onmessage so every raw wire message in both directions is
validated for the scenario's spec version -- including the initialize
handshake, server->client requests (elicitation/sampling), and the
harness's responses to them. Request ids are correlated per direction
so responses validate against their typed result definition (e.g. an
outbound ElicitResult). The reconstructed stand-ins in
connectStateful (fake id: 0 envelopes, Zod-reparsed results) are
removed; the hook sees the real bytes. The four scenarios that call
connectToServer directly (elicitation-defaults, elicitation-enums,
tools, lifecycle) now pass ctx.specVersion and get validation for
free; the runner surfaces their violations through the existing
recorder. Known remaining gap (documented in the module doc): the
client-auth scenarios' bespoke express mock is not instrumented.
- skipValidation now skips only the request direction; the
implementation's response to a deliberately malformed request is
still validated, with a narrow carve-out tolerating the JSON-RPC 2.0
id: null an error response must carry when the request could not be
processed (the MCP schema's RequestId forbids null). Covered by new
unit tests pinning both the still-validated response and the
narrowness of the carve-out.
- negative-mrtr.test.ts: the file-wide drain now asserts every recorded
violation is implementation-origin (the deliberately broken fixture)
and throws on any harness-origin violation.
- everything-server: hoist the cacheable-methods set to module scope
and replace the per-request res.json monkey-patch with an explicit
sendStatelessJson(res, method, payload) helper applied at each
dispatch site.
- Pin the extracted dispatch maps (methodDefs/errorDefs, e.g.
-32021 -> MissingRequiredClientCapabilityError under the draft) per
spec version in wire-schema.test.ts, so a schema sync that breaks the
const extraction fails loudly instead of degrading to envelope-only
validation. Also note the batch branch's requestMethod forwarding
limitation.
Offender the transport hook flushed out: the everything-server's
test_elicitation_sep1034_defaults tool sent a number-typed elicitation
field with default: 95.5, which the 2025-11-25 schema forbids
(NumberSchema.default is typed integer there; widened to number only in
the draft). The fixture and the scenario's expected default are now the
integer-valued 95, with comments explaining why.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JHNrpU3Vr6Q1rPZiji7zLY
* Condense comments to three lines max
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JHNrpU3Vr6Q1rPZiji7zLY
* Re-vendor spec schemas at pinned SHA 71e30695
Regenerate src/spec-types/*.schema.json via npm run sync-schema at the
SHA already recorded in SOURCE. The previously vendored draft schema
predated spec PR #3002, so it still required DiscoverResult.serverInfo
at the top level; the mock server now correctly reports serverInfo via
result _meta, which the stale schema rejected.
* Scope the wire-violation recorder per scenario for parallel suite runs
Client --suite runs execute scenarios concurrently over what was a
module-global violation recorder, so scenarios could steal or wipe each
other's recorded violations. Give each suite scenario its own recorder
via AsyncLocalStorage; code outside a scope (single-scenario runs, the
sequential server suite, vitest hooks) falls back to the global
recorder. Also condense a four-line comment in connection/stateful.ts
to three lines.
* Patch the 2025-11-25 NumberSchema generator bug as a load-time erratum
The released 2025-11-25 schema.json types NumberSchema
minimum/maximum/default as integer, contradicting its own schema.ts
(default?: number) — a typescript-json-schema artifact, fixed for draft
in modelcontextprotocol#2710 and proposed for 2025-11-25 in
modelcontextprotocol#3139. Patch the schema at load time so SDK
fixtures with fractional number defaults (all tier-1 SDKs send 95.5)
are not flagged; restore the sep-1034 scenario and example-server
fixture to 95.5. Delete the erratum when #3139 lands and the schemas
are re-vendored.
Also scope the sequential server-suite loop with withWireRecorder so a
connection leaked by a failed scenario cannot attribute late traffic to
the next scenario.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* feat(sdk-runner): expand KNOWN_SDKS to all tier 1 SDKs
- go-sdk: point at the relocated conformance/ fixtures (the old
examples/server/conformance path is gone on main), add the
everything-client fixture that now exists upstream, and wire
conformance/baseline.yml as the expected-failures baseline.
- python-sdk: new entry for main, which targets the 2026-07-28
revision (same uv workspace layout as the v1.x entry).
- csharp-sdk: new entry driving the ConformanceClient /
ConformanceServer fixtures via the .NET 10 SDK. The client takes the
scenario as argv[0], so the command bridges MCP_CONFORMANCE_SCENARIO
into argv; the server exposes the SEP-2575 stateless lifecycle at
/stateless.
* docs: match the sdk override example's lead-in to its content
The example now shows field overrides on a built-in entry rather than a
one-off ref, so the introducing sentence shouldn't promise a ref
override.
* fix: use negotiated protocol version in sse-polling raw requests
The server-sse-polling scenario hard-coded mcp-protocol-version:
2025-11-25 on its raw tools/call POST and Last-Event-ID reconnection
GET instead of the version negotiated during initialize — the same
bug class as #412. A server that enforces the negotiated session
version rejects those requests once it negotiates any other version.
Read transport.protocolVersion after client.connect() and use it for
both raw request headers, falling back to the run's spec version,
mirroring the sse-multiple-streams fix (#415). Add a regression test
whose fixture server down-negotiates to 2025-06-18 and 400s any
request whose MCP-Protocol-Version header does not match, so a fix
that hard-codes the latest version cannot pass.
Follow-up to #415.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0178cz8nAyFRHQyq9cg2XgZz
* test: harden sse-polling regression fixture per review panel
Enforce session id alongside protocol version in the fixture, record
Last-Event-ID presence per GET so the SDK's automatic standalone GET
cannot satisfy the reconnect assertion, normalize the header type
instead of casting, return a descriptive 400 body on the GET path,
add Allow to 405s, use the spec-shaped empty-data priming event, and
read sessionId via the SDK's public getter in the scenario.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0178cz8nAyFRHQyq9cg2XgZz
---------
Co-authored-by: Claude <noreply@anthropic.com>
Tool handlers issued sampling/createMessage and elicitation/create via
mcpServer.server.request(), which carries no relatedRequestId. The SDK's
Streamable HTTP transport routes such requests to the standalone GET SSE
stream; if the client hasn't opened it yet, the request is silently parked
in the event store and the scenario times out after 60s.
Switch test_sampling, test_elicitation, test_elicitation_sep1034_defaults,
and test_elicitation_sep1330_enums to the handler's extra.sendRequest(),
which sets relatedRequestId so the request rides the originating tools/call
POST SSE stream.
Fixes#407
Claude-Session: https://claude.ai/code/session_01Qz5WunPTnDHVot4C2J6sFz
Co-authored-by: Claude <noreply@anthropic.com>
The stateful server-sse-multiple-streams scenario hard-coded
mcp-protocol-version: 2025-03-26 on its three raw concurrent POSTs
instead of the version negotiated during initialize. A server that
enforces the negotiated session version correctly returns 400 to all
three POSTs, so the scenario false-failed before exercising the
multi-stream behavior under test.
Read transport.protocolVersion after client.connect() and use it for
the raw POST headers, falling back to the run's spec version if the
transport does not expose one. Add a regression test driving the
scenario against a fixture server that 400s any request whose
MCP-Protocol-Version header does not match what it negotiated.
Fixes#412
Claude-Session: https://claude.ai/code/session_0178cz8nAyFRHQyq9cg2XgZz
Co-authored-by: Claude <noreply@anthropic.com>
A baseline entry names a scenario, but a scenario is many checks:
server-stateless is 29. Excusing the one or two a spec change breaks
means baselining the whole scenario, which stops enforcing the rest.
An entry may now be '<scenario>:<check-id>', in which case every failing
check in that scenario is judged on its own. Bare scenario entries are
unchanged, and YAML parses '- a:b' as a plain string, so every existing
baseline keeps working untouched.
loadExpectedFailures now owns the entry grammar and returns parsed
BaselineEntry values, so evaluateBaseline only matches and cannot throw.
Rejecting a bad entry and interpreting a good one no longer live in
separate layers.
A check id matches all of its occurrences, since ids repeat within a run;
they collapse to one verdict, most-severe first. Absent and SKIPPED are
tolerated, matching how the runner already treats them as green.
Move collapseDuplicateChecks to src/checks/collapse.ts with its tests:
expected-failures.ts needs the reducer and cannot import dpop.ts's auth
server stack. No behavior change.
Reject two configs that previously coerced to an entry matching nothing:
a mapping ('- scenario: check-id', with a space) and an empty list item.
Spec PR #3002 (part of the final 2026-07-28 revision) moved serverInfo from
the DiscoverResult body to _meta['io.modelcontextprotocol/serverInfo'] and
demoted the request envelope's clientInfo from required to SHOULD. The
checks still asserted the old shape, failing SDKs that follow the final
revision.
- Vendor schema types at spec commit 71e30695.
- server-stateless: drop the missing-clientInfo rejection case and assert
the opposite (a clientInfo-less request must be served, new check
sep-2575-request-meta-client-info-optional); discover no longer requires
body serverInfo; new SHOULD-level check
sep-2575-server-identifies-in-result-meta (WARNING when the _meta
identity is absent, SKIPPED when discover itself failed).
- request-metadata (client side): the clientInfo SHOULD moves out of the
MUST check into its own id sep-2575-client-sends-client-info (WARNING on
absence), so requirement levels never blend in one check id.
- Mock servers and the everything-server example emit the _meta identity
and accept clientInfo-less requests.
Move the DPoP scenarios (auth/dpop, auth/dpop-nonce) and the WIF
JWT-bearer scenario (auth/wif-jwt-bearer) off the draft spec timeline
and register them as the io.modelcontextprotocol/auth/dpop and
io.modelcontextprotocol/auth/wif protocol extensions (SEP-2133
reverse-DNS ids), so they run under --suite extensions rather than at
--spec-version draft.
Claude-Session: https://claude.ai/code/session_012E7BriGaLimJxQr7B7ChiM
Co-authored-by: Claude <noreply@anthropic.com>
* Add DPoP client conformance scenario + shared DPoP helpers (SEP-1932, #368)
Client-anchored packaging: carries the shared DPoP foundation (dpopProof /
dpopToken helpers, the createAuthServer DPoP core, and sep-1932.yaml) together
with the client scenario, so it can merge first; the server (#369) and
authorization-server (#370) scenarios are follow-ups that depend on it.
DPoP is treated as a draft/not-yet-official feature (mirroring the WIF SEP-1933
scenario): the scenario's source is `introducedIn: DRAFT_PROTOCOL_VERSION` and
it is registered in draftScenariosList (informational, not scored). It is NOT
registered in EXTENSION_IDS, since the ext-auth DPoP extension is still on a
branch rather than merged.
Client checks (the two extension MUSTs, RFC 9449 §7.1 / §4.2-4.3):
- sep-1932-client-dpop-auth-scheme — token presented with the DPoP scheme.
- sep-1932-client-fresh-proof — a fresh, well-formed DPoP proof per request.
The test MCP server judges the client via an opt-in DPoP middleware
(dpopResourceAuth) passed through createServer's authMiddleware hook; a compliant
example client (covered by the Client Draft Scenarios loop) plus deliberately-
broken bearer/replay variants drive the pass/fail acceptance tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* client/dpop: split into nonce-less baseline + nonce-required scenario
Server-provided nonces are optional in RFC 9449 (AS §8 "MAY", RS §9 "can
also choose"), and the nonce-less flow is the common case. The single
auth/dpop scenario hard-wired nonce enforcement on both the test AS and
MCP server, so a plain nonce-less proof was never accepted end-to-end and
a client that only does plain proofs was never exercised on its happy path.
Parameterize DPoPClientScenario with `requireNonce` and register it twice:
- auth/dpop (requireNonce=false) — nonce-less baseline; emits the
three baseline checks (token-request-proof, dpop-auth-scheme, fresh-proof).
- auth/dpop-nonce (requireNonce=true) — prior behavior; adds as-nonce and
rs-nonce (five checks).
Check IDs are reused (no new IDs, no sep-1932.yaml change); the two nonce
checks are emitted only by auth/dpop-nonce, so traceability stays intact.
Add auth-test-dpop-no-nonce.ts (nonce-incapable but otherwise compliant)
asserting SUCCESS against auth/dpop — proof that a client with no nonce
handling still completes DPoP when the server does not require a nonce.
Baseline negatives stay on auth/dpop (nonce-independent); the two nonce
negatives are re-pointed to auth/dpop-nonce.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* client/dpop: round-4 nonce-surface correctness fixes
Address review findings on the auth/dpop-nonce posture:
- Record the token-request proof observation BEFORE the §8 nonce challenge,
so a client that is challenged but does not retry is no longer mis-reported
by token-request-proof as "never completed a token request" (its proof was
just validated). (#2)
- Record the resource-request jti/replay on ANY valid proof (before the §9
nonce gate), so fresh-proof is never asserted on zero evidence and a client
that never honours the RS nonce fails ONLY rs-nonce. (#5)
- Require challengeIssued && honored for as-nonce/rs-nonce SUCCESS, closing a
vacuous-pass path (a client that pre-sends the nonce is never challenged). (#3a)
- Gate the AS nonce observation on the authorization_code exchange, matching
recordTokenRequestProof (a refresh exchange must not satisfy §8). (#3c)
- Collapse duplicate shared token-flow check IDs (token-request, pkce-*) that
the §8 round-trip re-POST would otherwise emit twice. (#4)
- Example client: retry the token request only on a use_dpop_nonce error code,
not any 400 carrying a DPoP-Nonce header (consistent with the RS path). (#6)
- Fix the DpopTokenRequestObservation docstring ("last write wins" →
sticky-failure). (#7)
Adds an expectedSuccessSlugs option to the client test helper and tightens the
two nonce negative tests to pin the now-clean behavior (token-request-proof
SUCCESS for no-as-nonce; only rs-nonce fails for no-rs-nonce).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* client/dpop: round-5 fixes — dedupe masking, wording, replay decision
- dedupeSharedChecks was masking failures: it kept the last occurrence of a
duplicate check ID regardless of status and ran in BOTH postures, so a
FAILURE on a challenged POST (or an earlier attempt in a restarted flow) was
silently dropped. Replace with an exported, unit-tested collapseDuplicateChecks
that prefers the MOST-SEVERE occurrence, and only apply it in the nonce
posture (the baseline keeps genuinely distinct repeated attempts). (R5 #1)
- Reword the token-request-proof description: it can be SUCCESS for a client
that presented a valid proof but never obtained a token (challenged, no
retry), so it no longer claims "obtaining a DPoP-bound access token". (R5 #2)
- Reword the as-nonce/rs-nonce failure messages to cover a client that DID
retry but whose retry proof was rejected (not only "did not retry"). (R5 #3)
- Document the deliberate replay decision: recording jti on challenged requests
means re-sending the identical proof across the challenge/retry boundary trips
replay detection — a genuine RFC 9449 §4.2 jti-uniqueness violation. (R5 #5)
Adds src/scenarios/client/auth/dpop.test.ts unit-testing collapseDuplicateChecks
(prefers failures, keeps INFO, preserves order) — the dedupe fix is now pinned.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* client/dpop: round-6 polish — pin collapse wiring; docstring
- Add an integration assertion that the §8/§9 double-POST is collapsed to one
token-request/pkce-* entry each in auth/dpop-nonce, so the requireNonce gating
of collapseDuplicateChecks can't be silently deleted or inverted (previously
mutation-survivable). runClientAgainstScenario now returns the emitted checks
so callers can make occurrence-level assertions (backward-compatible).
- Docstring: note the severity ladder ranks SKIPPED below SUCCESS, and that
equal-severity ties keep the last occurrence.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The scenario's hand-rolled server/discover handler advertises
supportedVersions: ['2026-07-28'], but every other method is delegated
to StreamableHTTPServerTransport from the pinned SDK (^1.29.0), whose
version whitelist tops out at 2025-11-25. A SEP-2575 client that honors
the negotiated version then gets HTTP 400 on tools/list, and the
sep-2106-no-network-ref-deref check can never run.
Extend the existing server/discover workaround to the version check:
rewrite an incoming MCP-Protocol-Version: 2026-07-28 header to the
SDK's LATEST_PROTOCOL_VERSION before delegating (patching rawHeaders
too, since the SDK's Node adapter rebuilds its web-standard Request
from rawHeaders). Add a stateless-draft-client regression test.
Fixes#397
Claude-Session: https://claude.ai/code/session_01N4TdyLB13XSsPwNScYHf4C
Co-authored-by: Claude <noreply@anthropic.com>
* feat(auth): add JWT-bearer helper for WIF client conformance (SEP-1933)
Adds createWorkloadJwt and generateWorkloadKeypair to provide reusable,
tested JWT signing infrastructure for the upcoming wif-jwt-bearer scenario
(PR #2). Also extracts JWT_BEARER_GRANT_TYPE constant and migrates
cross-app-access.ts to use it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(auth): add JWT-bearer helper for WIF client conformance (SEP-1933)
Adds createWorkloadJwt and generateWorkloadKeypair to provide reusable,
tested JWT signing infrastructure for the upcoming wif-jwt-bearer scenario
(PR #2). Also extracts JWT_BEARER_GRANT_TYPE constant and migrates
cross-app-access.ts to use it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(auth): add WIF JWT-bearer scenario and negative tests (SEP-1933)
Adds the auth/wif-jwt-bearer client conformance scenario using the RFC 7523
JWT-bearer grant (urn:ietf:params:oauth:grant-type:jwt-bearer).
The scenario pre-signs valid, wrong-audience, and expired JWTs on start()
to simulate cloud workload identity tokens. The conformance AS verifies the
assertion and emits per-class checks (wif-assertion-verified,
wif-assertion-missing, wif-assertion-audience, wif-assertion-expired,
wif-assertion-malformed). Broken example clients exercise the missing-assertion
and wrong-audience failure paths.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: apply prettier formatting to wif-jwt-bearer.ts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(auth): add expired-assertion negative test; clarify wif-jwt-bearer comments
- Add runWifJwtBearerExpiredAssertion + auth-test-wif-expired-assertion.ts
to exercise the wif-assertion-expired check path (was dead code)
- Clarify WifAssertionVerified description to note iss is not validated
- Add comment explaining clockTolerance: 5 is intentional (same-run keypair)
- Add comment explaining numeric expiresIn is absolute epoch seconds
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(wif-jwt-bearer): disable DCR, detect retries, improve getChecks description
- Add WIF_CLIENT_ID constant; pre-seed provider's clientInformation() so
the SDK skips Dynamic Client Registration entirely (disableDynamicRegistration
on the auth server + pre-seeded client_id on the provider side)
- Add failedOnce/tokenRequestReceived tracking on the scenario; reject and
record wif-no-retry FAILURE if the client attempts a second token request
after the first fails
- Add hasAttempted guard in WifJwtBearerProvider.prepareTokenRequest() to
throw on retry from the client side
- Fix getChecks() sentinel description: distinguish "no request made" from
"request made but verification failed"
- Add client_id to context and Zod schema
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: prettier formatting on everything-client.ts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(wif-jwt-bearer): let errors propagate from broken example clients
The try/catch blocks were silently swallowing auth failures, so the
negative tests passed purely because expectedFailureSlugs found the
AS-emitted check — not because client error-surfacing was verified.
allowClientError: true on the test cases handles the non-zero exit.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(wif-jwt-bearer): tag as draft, add scope-rejected check, add SEP traceability
- Change specVersions from ['extension'] to [DRAFT_PROTOCOL_VERSION] and add
source = { introducedIn: DRAFT_PROTOCOL_VERSION } so the scenario is
reachable via --spec-version draft (extension tag excluded it from all
spec-version runs)
- Move registration from extensionScenariosList to draftScenariosList in
index.ts to match the tag/list convention enforced by spec-version.test.ts
- Add wif-assertion-scope-rejected check: AS returns invalid_scope for a
valid JWT when the client requests the reserved 'wif.rejected' scope;
verify client surfaces the error and does not retry
- Add runWifJwtBearerScopeRejected to everything-client.ts; add optional
scope param to WifJwtBearerProvider; add CLI entry point and vitest case
- Add src/seps/sep-1933.yaml with requirements mapped to each check ID,
covering the three deferred checks (iss, sub, jti) with exclusion rationale
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: prettier formatting
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(wif-jwt-bearer): add wif-grant-fallback check for unauthorized_client fallback
Client bug class: WIF client receives unauthorized_client from JWT-bearer grant
and silently switches to authorization_code instead of surfacing the error.
The MCP SDK retries after UnauthorizedClientError (auth.js:152-154), calling
prepareTokenRequest() a second time. WifGrantFallbackProvider exploits this by
returning authorization_code params on the second call.
Spec anchor: RFC 7523 §2.1 — clients MUST use the JWT-bearer grant type;
silent grant-type switching hides authentication failures.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: rename unused scope param in WifGrantFallbackProvider
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(wif-jwt-bearer): use realistic SPIFFE and K8s PSAT token formats
valid_jwt uses a SPIFFE JWT-SVID subject (spiffe://conformance-test.local/...)
issued by a SPIRE-style issuer. wrong_audience_jwt uses a Kubernetes PSAT
subject and kubernetes.io claims, mirroring how K8s projected service-account
tokens look in production. Token verification and client behaviour are
unchanged; the formats ground the scenario in real workload identity platforms.
k8s_issuer and k8s_subject added to context and schema for external clients
that want to construct their own K8s-style assertions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: prettier formatting on wif-jwt-bearer.ts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore(wif-jwt-bearer): drop informational k8s_issuer/k8s_subject from context
These fields were not used by any client handler and are redundant since
the K8s claims are already baked into wrong_audience_jwt.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(wif-jwt-bearer): address pre-merge review feedback
- Downgrade wif-no-retry, wif-grant-fallback, and wif-assertion-scope-rejected
to WARNING; RFC 7523 is silent on client retry and grant-type switching, so
FAILURE status was not traceable to spec text
- Add WifRetryProvider broken client that re-sends JWT-bearer after
unauthorized_client, making wif-no-retry actually exercisable as a WARNING
- Drop getChecks() sentinel (non-standard pattern that overloaded the
wif-assertion-verified check ID); test harness detects empty check sets
- Add comment on DRAFT_PROTOCOL_VERSION workaround for runner limitation
- Add comment explaining why both slash/no-slash audience forms are accepted
- Tighten schema comments for audience URL constraint and ES256 literal
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(wif-jwt-bearer): drop decorative fields and neutralise JWT fixtures
- Remove issuer and subject from context schema and scenario return value;
neither is validated by the AS or used by the client, making them
misleading. The iss exclusion rationale is updated to explain why.
- Replace SPIFFE/K8s-PSAT constants and the kubernetes.io additionalClaims
with neutral values across all three JWT fixtures; no check inspects
token format, so format flavour was decorative and invited false
assumptions about fixture semantics.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(wif-jwt-bearer): remove workaround comment, drop signing_algorithm, annotate retry provider
- Remove self-admitted workaround comment on DRAFT_PROTOCOL_VERSION; the
classification stands on its own and the comment was inviting challenge
- Drop signing_algorithm from context schema and scenario return; the field
was redundant once the value was fixed to ES256, and the assertion is opaque
to the client regardless
- Add comment to WifRetryProvider explaining it deliberately omits the
hasAttempted guard so the SDK retry reaches the AS
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(wif-jwt-bearer): export scope constants, remove unused AS field, annotate state machine
- Export WIF_TRIGGER_UNAUTHORIZED_SCOPE and WIF_REJECTED_SCOPE from
createWorkloadJwt.ts and import them in everything-client.ts; removes
duplicate bare-string declarations that could silently diverge
- Remove tokenEndpointAuthSigningAlgValuesSupported from mock AS config;
the SDK does not consume this field in the JWT-bearer flow
- Add comment to onTokenRequest clarifying wif-no-retry and wif-grant-fallback
fire on any second request after any failure, not only post-unauthorized_client
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(wif-jwt-bearer): address pcarleton review comments
- Move all broken WIF provider classes and runners to wif-broken-clients.ts;
everything-client.ts now contains only the conformant WifJwtBearerProvider
- Add sentinel in getChecks() for !tokenRequestReceived to prevent a silent
pass when the client never sends a request
- Drop wif-assertion-scope-rejected check; the no-retry behaviour after
invalid_scope is already covered by wif-no-retry, and sep-1933.yaml updated
accordingly
- Remove audience from context schema and scenario return; the field was
unused by any client implementation
- Expand scenario description to tell a client implementer what to implement
- Apply suggested description fix to wif-grant-fallback (does not assume
unauthorized_client was the specific error that preceded the fallback)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(wif-jwt-bearer): drop number type from expiresIn, use jose duration strings
Jose supports negative duration strings ('-60s', '60 seconds ago') that produce
already-expired tokens without needing absolute epoch arithmetic. Drop the
number overload, update the expired_jwt call site to '-60s', and update the
unit test accordingly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
The sep-2575-server-rejects-undeclared-capability check required
error.data.requiredCapabilities to be an array containing 'sampling'. The
schema's MissingRequiredClientCapabilityError defines it as a
ClientCapabilities object ({ "sampling": {} }), as do this repo's own
spec-types and the tasks/required-task-error scenario, so a spec-correct
server could not pass the check. It shipped because the positive -32021
branch had no self-test.
- stateless.ts: accept the object shape; each capability value must itself
be an object ({ "sampling": null } fails). State the shape in the
scenario's implementer-facing description.
- examples/servers/typescript/everything-server.ts: emit the object.
- stateless.test.ts: self-tests for object-passes / array-fails /
null-value-fails, sharing one capability-probe mock helper.
A check that cannot be exercised — the server under test lacks a
diagnostic fixture tool, or advertises a feature it then rejects — was
reported SKIPPED, which is excluded from pass/fail counts, exit codes,
and the expected-failures baseline. A server could dodge the entire
SEP-2575 capability-constraint and subscription surface (7 MUST-level
checks) and still read green, and a baselined failing scenario whose
checks degraded to SKIPPED flipped its baseline entry to stale, which
reads as fixed.
Such checks now fail with a 'Not testable:' message naming the missing
prerequisite and details.untestable for tooling, per the policy in
issue #248 (severity follows the requirement's keyword: MUST -> FAILURE,
SHOULD -> WARNING). Legitimate skips remain: optional capabilities the
server never declared, runner-level spec-version inapplicability, and
the documented tasks-status-notifications harness gap.
Also fixes a test-suite bug found en route: the stream mock's reader
lacked releaseLock, so listenToStream's teardown threw and every
stream-based assertion silently ran against zero frames.
* fix(http-custom-headers): use spec-permitted x-mcp-header types
The http-custom-headers fixture annotated three `number`-typed properties
(priority and float_val) with x-mcp-header. The SEP-2243 spec forbids this:
"MUST only be applied to parameters with primitive types (integer, string,
boolean). Parameters with type `number` are not permitted."
A spec-conformant client MUST reject (exclude from tools/list) any tool whose
x-mcp-header annotations violate the constraints, so it would drop both test
tools and never call them, failing ClientSupportsCustomHeaders through no fault
of its own.
- priority: number -> integer (its value, 42/1, and header are unchanged).
- float_val: drop the x-mcp-header annotation; a floating-point header param is
exactly what the spec prohibits and has no valid replacement. It stays in the
body and is now asserted NOT to be mirrored, like the unannotated query param.
- Treat 'integer' like 'number' in header-value validation and check-id
classification so Priority keeps its cross-SDK numeric tolerance.
* Reword float_val check to the rule actually under test
The new ClientCustomHeaderNoMirrorNumber check asserts absence of a header
for an *unannotated* parameter — the same designated-params-only rule the
query check covers. The previous description stated a 'MUST NOT mirror
number-typed params' requirement that SEP-2243 does not contain (the spec
forbids the annotation on number, with rejection as the consequence; it does
not define a client-side type filter). Reword the description and comment to
the rule actually under test, keeping the number/SEP-2243 context as the
explanatory why. Also trim the float_val schema description to match the
sibling query param style.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
The scenario has introducedIn: '2025-11-25' and tests SEP-1699 priming events,
which were introduced in that version. Sending '2025-03-26' as the client's
protocol version makes a correctly-behaving server suppress the priming event
(older clients cannot handle empty-data SSE), so the scenario WARNs against a
conformant server.
Verified against typescript-sdk@v2-2026-07-28: with this fix the scenario goes
from 'WARNING: Server did not send priming event' to '3 passed, 0 failed'.
* feat: add positive tests for the Authorization Code Grant
* fix(auth): use import type for AuthorizationServerOptions in types.ts
* fix(auth): make client-id/client-secret optional; rename --secret→--client-secret
* fix(auth): generate PKCE verifier per-run; derive S256 challenge
* fix(auth): build authorization URL with URLSearchParams; drop hardcoded resource
* fix(auth): default token_endpoint_auth_methods_supported to client_secret_basic; add 'none' branch
* fix(auth): check auth-method support before starting callback server
* fix(auth): check error param first; treat state mismatch as fatal; drop fabricated code_challenge assertion
* fix(auth): redact tokens from check details
* fix(auth): harden callback server (path-scoped, close(), error handler, clear timeout)
* feat(auth): print redirect URI and timeout hint before browser prompt
---------
Co-authored-by: Michito Okai <michito.okai.zn@hitachi.com>
The ${{ github.action_path }} template expression resolves to a backslash
path on Windows (D:\a\_actions\...\v0.1.14). When inlined into a bash run
block, bash treats backslashes as escape sequences, mangling the path.
Switching to the auto-injected $GITHUB_ACTION_PATH env var routes through
shell variable expansion, which preserves the literal value.
Fixes#150.
Co-authored-by: aditya <aditya@rapidcircuitry.com>
* feat: add JSON-based setup option to authorization server conformance test
* refactor(auth): single-schema validation for --file; drop duplicate schema/merge
* chore: move authorization-server settings sample to examples/
* test(auth): cover --file merge precedence and validation
* fix(auth): validate --file standalone via .strict(); reject partial/typo'd files
The settings file must be a complete, valid AuthorizationServerOptions on
its own; CLI flags are optional overrides on top. .strict() rejects unknown
keys so a typo'd field surfaces as an error instead of being silently
ignored. ZodError from file parsing is reported with the file path.
* chore: drop placeholder --file merge test pending CLI test pattern
The test re-implemented the merge logic locally, which can drift from
src/index.ts. Proper coverage needs either programmatic commander.parse()
or an extracted loadAuthorizationServerOptions() helper; deferring that
decision to a follow-up rather than establishing the pattern here.
---------
Co-authored-by: Michito Okai <michito.okai.zn@hitachi.com>
