* chore: add SEP-2640 requirement-traceability YAML (Skills Extension)
* style: apply yaml formatter pass
* chore(sep-2640): record spec source provenance
* chore(sep-2640): re-extract against SEP HEAD 556154c (drops mcp-resource-template)
Re-extract sep-2640.yaml against current SEP-2640 HEAD. The SEP removed
the `mcp-resource-template` index entry type in two commits on
2026-06-04 (fd50cc91 "Remove mcp-resource-template entries from skill
index", 556154c0 "Remove remaining resource template mentions from SDK
and rationale sections"), after this extraction was first captured on
2026-06-03 at b77fdfe8. Provenance comment bumped accordingly.
Resulting changes:
- sep-2640-index-entry-type-enum: drop `"mcp-resource-template"` from
the enum.
- sep-2640-index-name-required: drop the "is Required for ... Omitted
for mcp-resource-template" conditional language; the column is now
unconditional Required=Yes at SEP HEAD.
- sep-2640-index-digest-required: same simplification; also align
"hexadecimal" -> "hex" with the HEAD table text.
- sep-2640-template-resource-template-registered: removed. The SEP no
longer defines a template entry type, so the SHOULD on registering an
MCP resource template is gone.
- Excluded "Hosts SHOULD surface template entries..." excerpt: removed.
The sentence no longer exists in the SEP.
Sweep verified: zero "template" mentions in the SEP at HEAD. Security
Implications section sentences referenced by the remaining excluded
entries are intact. No other check rows touched.
* feat(sep-2640): ResourcesDirectoryReadScenario for resources/directory/read
Adds the conformance scenario for the SEP-2640 directoryRead surface that
landed in spec commit 2e04c48d (2026-06-09). Per AGENTS.md "fewer scenarios,
more checks", a single ResourcesDirectoryReadScenario emits 6
ConformanceChecks, one per new sep-2640.yaml requirement row.
Class named for the wire method (resources/directory/read), matching the
existing ResourcesListScenario / ResourcesReadTextScenario / etc. family in
src/scenarios/server/resources.ts. The runner-facing name field stays as
'sep-2640-skills' (umbrella) so mcpkit's conformance/Makefile entry
--scenario sep-2640-skills keeps working without a cross-repo race.
src/seps/sep-2640.yaml
- File-level provenance held at 556154c (the existing PR 330 baseline).
- 6 new check rows appended for the directoryRead additions, each
carrying a verbatim, grep-F-searchable excerpt from the SEP at
2e04c48da90224000e750ffd54a3611f2824fbc0:
- sep-2640-capability-directory-read-flag
- sep-2640-directory-read-method-registered
- sep-2640-directory-read-subdir-mimetype
- sep-2640-directory-read-result-resources-shape
- sep-2640-directory-read-invalid-params
- sep-2640-directory-read-pagination
- forward_reference header comment notes the asymmetry: PR 97 schema
rewrite (360123d0, 2026-06-08) made 3 existing rows stale and drifted
~11 others' wording; full re-extraction at SEP HEAD is mcpkit#780's
lifecycle. Provenance deliberately holds at 556154c until that lands.
src/types.ts
- Adds io.modelcontextprotocol/skills to EXTENSION_IDS so the scenario's
source: { extensionId: ... } tag type-checks.
src/scenarios/server/directory.ts (new)
- Capability discovery via wire-observable signal: -32601 method-not-found
is the only definitive "server did not declare directoryRead" signal.
- Fixture assumption: server exposes skill://acme/billing/refunds/templates
with at least one subdirectory child (mcpkit examples/skills layout).
No skill:// resources at all -> every check emits SKIPPED so the
scenario stays green against upstream's everything-server fixture.
- 6 checks (1:1 with the YAML rows above):
1. directoryRead declared (derived from method registration)
2. method registered (happy-path call succeeds)
3. result.resources shape matches resources/list
4. subdir child carries mimeType: "inode/directory"
5. non-directory URI returns -32602
6. nextCursor round-trips (single-page is conformant)
src/scenarios/index.ts
- Imports + registers ResourcesDirectoryReadScenario in
allClientScenariosList, matching the registration name 'sep-2640-skills'
that mcpkit's conformance/Makefile already passes to --scenario.
Verified end-to-end against mcpkit examples/skills:
cd ~/newstack/mcpkit/main && \
MCPCONFORMANCE_SKILLS_PATH=~/newstack/mcpkit/conf-skills \
make -C conformance testconf-skills
-> 6/6 SUCCESS, 0 failed, 0 warnings.
Out of scope:
- PR 97 schema rewrite YAML refresh (separate ticket, mcpkit#780 lifecycle).
- Negative-capability fixture (server without directoryRead) - mcpkit
examples/skills has no flag for it; ext/skills/client_directory_test.go
already covers the SDK pre-call guard.
* feat(sep-2640): expand skills conformance — index + manifest scenarios, drift fixes
Builds on the directory-read scenario with the server-observable skills surface
and brings the whole set up to the current framework conventions.
New scenarios (src/scenarios/server/skills/):
- SkillsIndexScenario (sep-2640-skills-index): skill://index.json is exposed and
readable; every skills[].type is in the SEP enum; every entry has a non-empty
name; a present digest matches sha256:{64hex}; entry URLs use skill://.
- SkillsManifestScenario (sep-2640-skills-manifest): a skill-md SKILL.md is
markdown-typed; frontmatter name/description surface on the resource; the
final skill-path segment equals the frontmatter name; skill _meta keys use the
reverse-domain prefix.
directory.ts (moved into skills/): capability is now read from server/discover
instead of inferred from -32601; an undeclared optional capability SKIPs rather
than FAILs; the untestable.ts helpers back the not-exercisable paths; all
mcpkit-specific fixture URIs are gone — the directory to exercise is discovered
dynamically, so the scenario is brand-neutral.
Capability gating accepts directoryRead at either extensions[id].directoryRead
(SEP-2640's inline example) or extensions[id].config.directoryRead (the SEP-2133
negotiation envelope SEP-2640 defers to); the two SEPs are inconsistent on
nesting, so the check accepts either rather than privileging one reading.
All three scenarios verified green against a live SEP-2640 server (6/6, 6/6, 7/7);
typecheck, eslint, prettier, and the 504-test vitest suite pass. sep-2640.yaml
gains a header note mapping each scenario to the rows it now backs.
* feat(sep-2640): re-extract against the 2026-08-21 rewrite
The prior extraction held at SEP commit 556154c (2026-06-05) by an explicit
in-file decision to wait for the index-schema refactor. The 08-21 core-maintainer
rewrite made that pin actively wrong rather than merely behind.
Removed as no longer normative: the skill://index.json well-known resource
(absent from the current text), the entry type enum, the no-nested-skills rule
(now reversed - skills MAY nest), and the five archive rows (archives are
deferred to an appendix). Reworded the two index-field rows, the empty-listing
row and the verify-digest row.
Added the new surface: skills/list and skills/get with their pagination,
atomicity and caching attributes; the {uri, frontmatter, resources} entry
schema; resources completeness, the "dynamic" sentinel and per-file size; the
512-entry and 16 MiB limits; and the host obligations around lazy retrieval,
content-bound approval and frontmatter re-verification.
index.ts is replaced by enumeration.ts, which drives both new methods. manifest.ts
and directory.ts swap their skill://index.json discovery fallback for skills/list.
The capability helper no longer accepts a config envelope. SEP-2133 is Final and
defines the extensions field as a map of identifiers to settings objects, with no
envelope and no slot for id/specVersion/stability; SEP-2640's capability block
matches. The earlier leniency was based on a misreading.
77 rows: 65 checks (39 wire-tested by the three server scenarios) and 12 excluded
as host-internal.
* feat(sep-2640): close the extraction gaps found by a normative-sentence audit
The first pass extracted the Specification sections but under-covered Security
Implications, which carries 21 of the SEP's 97 RFC-2119 sentences. Sweeping
every normative sentence in the source surfaced rows that were missing entirely:
- origin tagging at the point content enters model context (the L179 host MUST)
- allowed-tools MUST be ignored for MCP-origin skills
- reads bound to the originating server, so a skill from A cannot drive a read
against B, and any cross-origin read gated behind per-call approval
- servers identified by a host-assigned label, never serverInfo.name
- the three disk-cache rules: write-isolation or re-hash, exclusion from
filesystem-skill discovery paths, and removal when a server is removed
- a resources/read of a SKILL.md by another route is not a load
- a nested SKILL.md read as supporting content is ordinary markdown
- the registry-assembly and virtual-mount no-prefetch restatements
- the SDK affordances guidance, which is the row mcpkit itself is measured by
Five rows previously filed as excluded are now declared instead. They are real
requirements that a host-side harness could check; calling them unobservable was
a limitation of this scenario set, not of the requirement.
Adds one wire check, names-should-be-unique, since a within-listing collision is
observable and tells an operator the listing needs disambiguating.
96 rows: 89 checks (40 wire-tested) and 7 excluded.
* style: apply prettier to the SEP-2640 scenarios and yaml
Quote-style normalisation in the yaml (escaped single quotes become
double-quoted scalars) and one return-type wrap in helpers.ts. The parsed
yaml is byte-identical in content; verified by comparing the parse before
and after.
* docs(sep-2640): note why resultType is not declared in this yaml
It appears in all three result examples but is a base-protocol field on the
common Result interface, not a SEP-2640 requirement; its caching-hint
obligations are already owned by sep-2549.yaml.
* fix(sep-2640): follow nextCursor on resources/directory/read
The directory scenario issued one resources/directory/read and every check
below it inspected that single page: the result-shape check, the
subdirectory-mimeType check, and the target selection for the -32602 probe.
SEP-2640 says directory-read pagination mirrors resources/list, so a
conformant server MAY split a directory across pages, and reading only the
first made "no subdirectory here" indistinguishable from "the subdirectory
is on page two".
Reported by Sam Bloomberg against the go-sdk implementation, which exposes
DirectoryOptions.PageSize. Rebuilding that server with PageSize 1 fails
sep-2640-directory-read-subdir-mimetype; with this change it passes 7/7,
and both servers still pass at their default page size.
Only findable against a second implementation. mcpkit hardcoded its page
size to zero with no override, so it never emitted a cursor and the defect
was invisible to every run against it.
listAllResources already did this for resources/list. directoryReadAll
applies the same shape to directory reads, with a page bound and a
repeated-cursor guard.
* feat(sep-2640): client scenario for the no-prefetch MUST NOT
The traceability file filed retrieval policy under "host obligations a
harness cannot observe", alongside 48 other rows. That was wrong for this
one: "hosts MUST NOT retrieve a skill's files ahead of need" reduces to
whether a request arrived, which makes it among the most wire-observable
rules in the SEP.
Adds a Scenario (harness stands up the server, client is the SUT) that
serves a listing naming a SKILL.md and a supporting file, then fails if
either is read. The client's contract, keyed on MCP_CONFORMANCE_SCENARIO,
is to connect, enumerate, and stop.
SKIPs rather than passes when the client never calls skills/list, since
without a listing there is no window in which prefetching is possible and
a pass would prove nothing.
Verified both directions against mcpkit's client: SUCCESS as written, and
FAILURE with a deliberate ReadFromEntry added to the driver. A check that
has never been shown to fail is the same trap as the directory-pagination
false negative this suite shipped for months.
* feat(sep-2640): client scenarios for the four read-time verification MUSTs
Adds digest, size, frontmatter and unlisted-URI variants alongside the
no-prefetch scenario. Each serves an internally honest listing, tampers with
exactly one thing on the wire, and watches whether the client notices.
Rejection is detected by absence, following auth/resource-mismatch: the
client's contract is "load the skill, then read its supporting file", and a
client that verifies aborts on the load and never reaches the supporting
file. Reaching it is the violation. The unlisted variant is proven by the
read never appearing on the wire at all, since a conformant client refuses
locally.
The frontmatter variant recomputes digest and size for the tampered body, so
the frontmatter comparison is the only thing left that can fail. Without
that the digest check would mask it and the scenario would pass for the
wrong reason.
allowClientError is set on all four: a verifying client exits non-zero
because it correctly aborted, and that is the pass.
Verified both directions against mcpkit's client. All four SUCCESS as
written; all four FAILURE against a driver rewritten to read through
ReadSkillURI, which skips verification entirely.
* fix(sep-2640): gate the cache-attributes check on the negotiated version
The check graded ttlMs and cacheScope without looking at the protocol
version, which made it wrong in both directions. Below 2026-07-28 those
fields are not defined by the negotiated schema at all, so a server that
omits them is correct and the WARNING was a false positive. From 2026-07-28
they are required, so a missing one is a failure rather than a warning.
Now SKIPPED with a version reason below 2026-07-28, and SUCCESS or FAILURE
above it.
Reported by Sam Bloomberg against the Go SDK, which was correct on both
wires while this check mis-graded it on both. Second finding from the same
reviewer against this branch, and the second that only a second
implementation could surface: mcpkit emits the attributes on every version
it supports, so it could never have produced the false positive.
Verified three ways: mcpkit at 2026-07-28 SUCCESS, the Go SDK at 2025-11-25
SKIPPED where it previously warned, and a deliberately stripped mcpkit build
at 2026-07-28 FAILURE.
* docs(sep-2640): how to run the skills scenarios against any implementation
Three implementations have now been cross-checked and every one needed a
different invocation, so the knowledge was living in chat rather than the
repo. Records the two flags and why both are needed, how to read totals that
include wire-schema-valid, and the per-SDK setup.
The two traps are worth writing down because both have already cost someone
time. Extension scenarios do not match --spec-version alone, so --force is
always needed. And --spec-version selects the wire lifecycle rather than
filtering, so omitting it against a pre-2026-07-28 server produces a -32022
that reads like a missing method.
Also records that the go-sdk serves 2026-07-28 only when its streamable
transport is Stateless, which is correct per SEP-2575 and which I briefly
mistook for a defect.
* fix(sep-2640): advertise skills on server/discover, not only initialize
Both client scenarios declared the skills extension and `resources` in their
`initialize` reply, but `BaseHttpScenario` intercepts `server/discover` before
`handlePost` runs. On the 2026-07-28 lifecycle a discover-first client
therefore saw the base class default of `{tools: {}}` and had nothing to gate
a `skills/list` call on.
The scenarios still passed, because `server/discover` is optional for clients
and neither client we ran used it. That is the failure mode where a check goes
green because its condition never arose, so both lifecycles are now asserted
in capabilities.test.ts rather than left to the next implementation to find.
Overriding `discoverCapabilities()` covers both paths at once, since
`sendInitialize` already defaults to it.
* fix(sep-2640): give the capability and naming checks something to fail on
`sep-2640-capability-empty-object` reported a hardcoded SUCCESS, so it could
not fail and was checking nothing. It now asserts the observable half of "an
empty object indicates support for the extension with no optional features":
the declared value must be an object at all.
That also closes a false SKIP. `skillsCapability` folded a malformed
declaration into `undefined`, which downstream read as "extension not
declared" and skipped the whole suite, so a server declaring the extension as
`true` produced a clean run. `declaredSkillsCapability` separates absent from
malformed; the inline/envelope check now SKIPs on a malformed value, since
there are no settings to place either way. The base `DiscoverResult` schema
rejects a non-object too, so both halves are asserted.
`SKILL_NAME_PATTERN` accepted consecutive hyphens. The Agent Skills
specification that SEP-2640 defers to for naming lists four rules and we
encoded three, so `pdf--processing` passed. Its own valid and invalid examples
are now the test cases.
Negative controls recorded for all three: with the check back to a hardcoded
SUCCESS the four malformed-declaration cases fail, and with the previous regex
the consecutive-hyphen case fails.
* docs(sep-2640): name skills/list in the discovery prose, not index.json
The 2026-08-21 revision replaced the `skill://index.json` well-known resource
with `skills/list`, and the code follows it — `discoverDirectory` and the
manifest fallback both call `skillsListAll`. Only the prose lagged, including
three strings a user actually sees: two untestable-check reasons and the
directory scenario's description.
The remaining mentions in helpers.ts, enumeration.ts and sep-2640.yaml are
deliberate history, describing what was replaced.
* fix(sep-2640): grade unlisted reads against the entry, not one fixture URI
The `unlisted` client scenario asserted `reads.includes(UNLISTED_URI)`, so it
only caught a client that probed the exact path this fixture happens to serve.
Any client choosing a different unlisted path passed without being tested, and
the check would have reported SUCCESS for a client that reads unlisted files
freely.
The requirement is "hosts MUST resolve reads of the skill's files only to URIs
listed in that entry's resources", so the permitted set now comes from the
entry the client was handed and any read outside it is the violation. That is
the sentence itself rather than a proxy for it, and it lets a SUT derive its
own probe instead of hardcoding ours.
Negative control: a client reading a synthesized `mcpkit-unlisted-probe-0.md`
is now a FAILURE, where the previous check reported SUCCESS.
* fix(sep-2640): require the manifest read, and retire the unlisted scenario
Two changes to the client-side verification scenarios, one of which fixes a
hole the other one shares.
**The manifest read is now a prerequisite.** Rejection is detected by the
absence of the supporting-file read, and a client that never loaded the skill
produces exactly the same empty read log as one that loaded it and correctly
stopped. Both reported SUCCESS, so `listCalled` was the only gate and a client
implementing `skills/list` and nothing else passed every variant here. It
scored 5/5 on this suite while doing nothing.
Absence now means something only after the client has read the tampered
`SKILL.md`. A missing prerequisite reports untestable, which per #248 is a
FAILURE rather than a SKIP: SKIPPED is excluded from pass/fail counts and exit
codes, so it would read as green and hide the gap.
Negative control: a driver that lists and exits goes from 1/1 to 0/1 on all
three variants.
**`-verify-unlisted` is retired**, on Peter Alexander's call. Nothing was
tampered in it, so the only thing under test was whether the driver asked for
a URI it should not have, which grades the driver rather than the client. It
is also the rule an inspection tool answers differently from a host, since
refusing a read requires having a load to refuse, which MCP Inspector 2.6.0
reported as "surfaced" rather than passed.
The requirement is not deleted. It moves to an `excluded:` row alongside the
acting-on window that scopes it, and the reason records that it is excluded by
construction rather than in principle: a `SKILL.md` body linking to a file
absent from `resources` would make the harness adversarial again, since a host
that follows the link fetches an unlisted file on its own initiative. The
scenario name and check id are named there so a future reintroduction lines up
with historical reports rather than reading as a new requirement.
Check rows go from 89 to 88.
* feat(sep-2640): check skills/get cache attributes and the resources capability
Both requirements landed when the stable spec page merged ext-skills#139
today, and both are checkable on the wire.
**`sep-2640-skills-get-cache-attributes`.** `GetSkillResult extends
CacheableResult`, so `ttlMs` and `cacheScope` are REQUIRED as they are on
`resources/read`. Held out of this PR until the page merged rather than
grading against text that was still open: SEP-2640 says of these fields that
"whether the result should also carry the base protocol's caching attributes
... is left open", so this is a behaviour change and a server conformant to
the SEP text alone omits both. It is a reading not everyone shares. The MCP
Inspector's SEP-2640 writeup argues the opposite, that the SEP forecloses it
and inventing the requirement would report a reasonable server as broken.
Gated the same way as the skills/list check: below 2026-07-28 `CacheableResult`
is undefined by the negotiated schema, so omission is correct and the check
SKIPs with a version reason rather than failing a conformant legacy server.
**`sep-2640-capability-requires-resources`.** A server declaring the extension
MUST also declare the base `resources` capability, since skill files are served
through `resources/read`. The page states it as a consequence of the base
Resources specification rather than a free-standing new rule, and the
declaration is on the wire either way.
Verified against mcpkit: 32/32 on 2026-07-28 and 30/30 on 2025-11-25, where the
get-cache check SKIPs below the floor as designed. Negative control recorded: a
build with the attributes stripped from `handleSkillsGet` reports 31/32 with
`sep-2640-skills-get-cache-attributes` FAILURE.
Check rows go from 88 to 90.
* fix(sep-2640): compare frontmatter by content, not key order
entry-frontmatter-identical compared each top-level key with JSON.stringify,
which is key-order sensitive for nested objects. A server whose encoder sorts
map keys (Go's encoding/json does) failed this MUST for any SKILL.md whose
metadata keys are not in alphabetical order, although the content is
identical.
Compare structurally instead. A real difference still fails; both cases are
pinned in negative.test.ts.
* fix(sep-2640): make the client-scenario mocks answer like a conformant server
The skills/list and resources/read replies carried only resultType, so on the
2026-07-28 wire they lacked the ttlMs/cacheScope a ReadResourceResult
requires, and skills/list lacked the attributes the enumeration scenario
requires of servers.
initialize always answered 2026-07-28, while the runner hands these extension
scenarios' clients 2025-11-25 by default. A client that checks the negotiated
version disconnected before listing, and every verify variant reported
SKIPPED whether or not the client verified anything.
The mocks now stamp the caching members and answer initialize at the
requested version when the suite can speak it. The runner default is left
alone because it also applies to the auth extension scenarios. Pinned in
draft-result-fields.test.ts next to the other hand-rolled mocks.
* docs(sep-2640): run from the published package; --force only with --spec-version
The run notes pointed at a fork branch that goes stale on merge, and said
--force is always needed. Extension scenarios only SKIP when an explicit
--spec-version is passed; without it they run on the draft wire.
* docs(sep-2640): drop RUNNING_SEP2640.md
Per-run notes and SDK result tables belong in the PR description, not a
root-level doc that goes stale on merge.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
The runner is the MCP client in the `authorization` direction, and the
spec requires clients to send `resource` in both the authorization
request and the token request. It sent neither.
Adds `--resource <uri>` and a matching `resource` field on
AuthorizationServerOptionsSchema, so `--file` supplies it too. The value
must be an absolute URI with no fragment. The parameter is only sent
when set, so existing runs are unchanged.
No new check IDs: in this direction the runner is the client, so a check
that it sent `resource` could never fail. The existing grant checks now
reach a path they couldn't before.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The transport spec (2025-11-25 § 2.1, point 4) requires that a server
reply to a JSON-RPC notification with 202 Accepted and no body. The
fall-through branch in the initialize scenario test server returned
200 with `{"jsonrpc":"2.0","id":null,"result":{}}` for all non-initialize,
non-tools/list requests - including `notifications/initialized`.
Branch on `request.id == null` (notification) and emit 202 with no body
in that case. Non-notification requests still get the prior placeholder
200-response behavior.
Closes#274.
Co-authored-by: aditya <aditya@rapidcircuitry.com>
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
* fix(validation): accept extension result envelopes
This fixes#424
Signed-off-by: King Star <mcxin.y@gmail.com>
* validation: only known extension resultTypes take the envelope-only path
Restrict the generic-envelope fallback to resultType values a known
extension defines (today: 'task', the tasks extension's CreateTaskResult).
Any other non-core resultType is still validated as the request method's
own result — the discriminator is open, so it may be a private extension —
and a resulting violation names the unrecognised value so a typo or an
unknown extension is obvious instead of silently passing envelope-only.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Signed-off-by: King Star <mcxin.y@gmail.com>
Co-authored-by: triage <paulc@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
SEP-2243 permits `x-mcp-header` only on integer, string and boolean
parameters — "Parameters with type `number` are not permitted" — but the
harness never verified the `number` case and its requirement text still
described `number` as permitted.
PR #371 fixed the positive `http-custom-headers` fixture (priority is now
`integer`, `float_val` is served unannotated) but left the two remaining
items from #344:
- `HttpInvalidToolHeadersScenario` serves invalid tools for every other
x-mcp-header constraint but had no number-typed case, so nothing checked
that a client excludes such a tool.
- `sep-2243.yaml` claimed primitive types are "(number, string, or
boolean)" and that "number values" get decimal-string encoding, both of
which contradict the released 2026-07-28 spec.
Add an `invalid_number_header` tool mapped to the existing
`sep-2243-x-mcp-header-primitive-only` requirement (no new check IDs) and
correct both requirement texts. traceability.json is updated in place
since it embeds the yaml text verbatim; check IDs and statuses are
unchanged.
Verified against typescript-sdk@main: the new check fails there because
the SDK deliberately allow-lists `number` to satisfy the old fixture — a
workaround its own source notes as "tracked upstream" and can now drop.
The positive scenario still passes 18/18.
Closes#344
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* feat(tools-list): check deterministic tools/list ordering on the 2026-07-28 wire
The 2026-07-28 revision adds to server/tools.mdx: "Servers SHOULD return
tools in a deterministic order (i.e., the same ordering across requests
when the underlying set of tools has not changed)." Nothing in the suite
exercised it.
Add a tools-list-deterministic-order check to the existing tools-list
scenario. From the 2026-07-28 wire on, the scenario takes two more
consecutive tools/list snapshots and compares the three name sequences.
Same multiset of names, different order is WARNING (SHOULD). A set that
changed between probes is reported as untestable rather than as a
violation, because the spec scopes the SHOULD to an unchanged set. Fewer
than two tools on every probe is INFO. The check carries
source.introducedIn '2026-07-28', written as a literal since that revision
is published and dated, so runners drop it on earlier wires.
Prove it fails: examples/servers/typescript/tools-list-rotated-order.ts
advertises the same four tools and rotates the list by one position per
call; a negative.test.ts case asserts WARNING while tools-list itself
still passes. Unit tests cover the helper that builds the check.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjeC3sCvfHRpdEPz2x41B2
* tools-list: inline the deterministic-order check id and introducedIn literal
Check ids and introducedIn revisions are written inline everywhere else in
src/scenarios (tests pin the literal slug on purpose, since SDK baselines
reference it), so drop the two exported constants.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* fix(tools-list): align tools-name-format with 2025-11-25 spec prose
Enforce dated spec Tool Names SHOULD rules (128 max, no slash), WARNING
severity, 2025-11-25+ gate, negative fixture, and core-spec specReferences.
Fixes#379
* test(tools-list): cast Connection.request mock for tsgo typecheck
* test(tools-list): pin WARNING to offending tool name in negative test
* test(tools-list): rename negative fixture to invalid-tool-names
Document SEP-986 markdown vs integrated spec divergence in fixture header.
* chore: trim explanatory comments to the essentials
Keep the rule statement and the note that the dated spec (not the SEP-986
markdown) is authoritative; drop the process narration.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: triage <paulc@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
Adds resource-parameter-matches-prm to the RFC 8707 checks, emitted by the
token-endpoint-auth and metadata-discovery scenarios; auth/metadata-var2 serves
a pathless resource, so a client that re-serializes it through URL.href fails.
Negative example client included. The everything-client (SDK 1.x) is baselined
on that scenario until typescript-sdk#1972 ships.
Bumps the npm_and_yarn group with 1 update in the / directory: [ip-address](https://github.com/beaugunderson/ip-address).
Bumps the npm_and_yarn group with 1 update in the /examples/servers/typescript directory: [ip-address](https://github.com/beaugunderson/ip-address).
Updates `ip-address` from 10.1.0 to 10.2.0
- [Commits](https://github.com/beaugunderson/ip-address/commits)
Updates `ip-address` from 10.1.0 to 10.2.0
- [Commits](https://github.com/beaugunderson/ip-address/commits)
---
updated-dependencies:
- dependency-name: ip-address
dependency-version: 10.2.0
dependency-type: indirect
dependency-group: npm_and_yarn
- dependency-name: ip-address
dependency-version: 10.2.0
dependency-type: indirect
dependency-group: npm_and_yarn
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
The retry MUST ("waiting the given number of milliseconds before
attempting to reconnect") is a lower bound; no spec text bounds reconnect
delay from above. The slightly-late WARNING and >2x FAILURE bands gated
environment latency, which CI load inflates - in python-sdk CI the warning
band fired on ~4-5% of runs under the suite's parallel client spawns.
- delete the late bands: FAILURE only when the reconnect arrives earlier
than retry minus the 50ms tolerance
- demote the could-not-measure path from WARNING to INFO
- negotiate protocolVersion 2025-11-25, the version whose MUST is under
test (was 2025-03-26; same fix as 4686e06 for server-sse-polling)
Strictly relaxing: no client that passes today can start failing.
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
The client runner has had a 30s per-scenario timeout since it was written,
but `runServerConformanceTest` awaited `scenario.run(ctx)` with no bound. A
server under test that accepts the TCP connection and then never answers an
initialize POST, a tools call, or an SSE read stalled the entire suite run,
not just its own scenario.
#316 bounded the session-termination DELETEs in teardown, but that is a
call-site fix: covering the rest that way needs the same treatment on every
raw fetch in every scenario. Bounding at the runner makes those per-call
bounds defense-in-depth instead of the only protection.
`--timeout` now applies to server mode too, with the same 30s default as
client mode. On expiry the scenario reports a `scenario-timeout` FAILURE
naming the elapsed bound, so a wedged server reads as a failing scenario
rather than a hung process.
The timed-out scenario promise is left pending deliberately — a read blocked
on a socket has no cancellation channel — with its rejection swallowed so a
late failure can't surface against whichever scenario is running by then.
Fixes#427
Co-authored-by: Avi Seth <aviseth@users.noreply.github.com>
A negative check that reads only the final verdict scores SUCCESS whenever
the client fails to reach the requirement at all: "declined" and "never got
far enough to decide" are the same observation. auth/resource-mismatch
decided on !authorizationRequestMade alone, so a client that threw before
making a single HTTP request passed it.
Gate each verdict on the prerequisite the harness already records:
prm-pathbased-requested for the PRM resource comparison, and
authorization-server-metadata for the RFC 9207 iss comparisons, which
SEP-2468 conditions on the issuer recorded from the validated metadata
document. When the prerequisite is not observed the requirement was never
exercised, so report it via notTestable() (#248) rather than SUCCESS.
auth/metadata-issuer-mismatch already gated on the metadata fetch but
reported the unreached case as a plain violation; it now reports untestable
too. All six checks record propertyReached and stopReason in details.
Adds an inert client and a vitest case, so the checks are shown to catch
something and not merely to avoid false-positiving.
Version detection stripped only a leading `v`, so the monorepo tags
Cargo workspaces publish (`rmcp-v3.0.1`, `rmcp-macros-v3.0.1`) parsed
to NaN and scored `stable_release: fail`. rust-sdk was reported as
having no stable release despite a non-prerelease `rmcp-v3.0.1`.
Match the version at the end of the tag instead, so any package-name
prefix is ignored while bare (`1.2.3`), `v`-prefixed and prerelease
(`v2.0.0-rc1`) tags parse as they did before. Tags carrying no
recognizable version are returned unchanged rather than dropped, so
they still surface in the check output.
Fixes#425
`input-required-result-capability-check` guarded its elicitation scan with
`isInputRequiredResult(result) && result.inputRequests`. When a server answered
without `inputRequests`, that arm was skipped, and the following
`isCompleteResult` arm is false by construction for an input_required result, so
nothing was recorded and the check passed. An empty `inputRequests` object passed
too: the arm was taken, but the loop had nothing to scan.
`inputRequests` is optional — a result carrying only `requestState` still
satisfies "at least one of inputRequests or requestState" — so such a server has
not violated "servers MUST NOT send an inputRequests that the client has not
declared support for in its capabilities". It has not exercised it either.
Reporting SUCCESS makes an unverified MUST read as green, which is the failure
mode #248 and #372 established should be reported as a failure instead.
So report it through `notTestable()` with `details.untestable`, matching the
convention in AGENTS.md and the call sites in stateless.ts, rather than claiming a
violation the server did not commit. Servers that legitimately have nothing to ask
for now go red on this check; per #248 the expected-failures baseline is the
escape hatch.
Adds the case to sep-2322-mrtr-broken-server so the existing MRTR negative suite
covers it. The fixture returns a conformant envelope (`requestState` present, no
input requests) rather than a bare one, so it isolates this check rather than also
tripping sep-2322-request-state-incomplete.
Verified against --spec-version 2026-07-28:
everything-server 2/2 passed
sep-2322-mrtr-broken-server FAILURE Not testable: server returned no
inputRequests, so the capability restriction
was never exercised
Refs #439
Co-authored-by: Marcus <238969485+marcus-kepler-92@users.noreply.github.com>
The visible-ASCII session-id check sends a raw initialize POST outside the
SDK transport, because the transport does not expose the MCP-Session-Id
response header. That fetch carried no abort signal and its body was never
read or cancelled.
A server that accepts the connection and never responds hung the scenario
here, before teardown, so the DELETE bound added in #316 never came into
play. And since the probe's `Accept` includes `text/event-stream`, a server
answering with an open SSE stream left the body unconsumed and leaked the
connection for the rest of the run.
Bounding the probe at 5s matches the existing session-termination bound.
#427 is the structural fix for the hang class; this is still worth having
for fast failure and for attributing the failure to this check rather than
to whatever the runner-level timeout happens to interrupt.
Fixes#428
Co-authored-by: Avi Seth <aviseth@users.noreply.github.com>
The BaseHttpScenario generic fallback answered any unrouted request
with only the required draft result fields, so a standard list method
a scenario does not route came back without its required list member.
http-standard-headers advertises resources but has no
resources/templates/list route; the reply's missing resourceTemplates
fails schema validation (draft and 2025-11-25 both require it), and
strictly-validating clients drop the connection before the scenario's
header checks run.
Merge schema-valid empty results for the six list-shaped methods
(tools, resources, resource templates, prompts, roots, tasks) into
the fallback, keyed by a ReadonlyMap so method names that collide
with Object.prototype ("constructor", "toString", ...) miss instead
of poisoning the reply with a prototype function. tasks/list has a
typed result only at 2025-11-25; the empty member is harmless on the
draft wire. Non-list results (tools/call, resources/read,
prompts/get, ...) have no meaningful empty default and keep the bare
stamped fallback, so a route a scenario forgot surfaces instead of
being masked.
Tests pin the fixed behaviour on all three http scenario classes,
the prototype-name lookups, and the spec schema's rejection of the
pre-fix payload.
Fixes#474
Registers modelcontextprotocol/ruby-sdk in KNOWN_SDKS so `tier-check --sdk-path` and `sdk ruby-sdk` manage
the build and per-revision runs themselves, instead of requiring the --conformance-server-url/--client-cmd
fallback for unknown SDKs. This is the registry path the Ruby SDK's SEP-1730 Tier 1 application
(modelcontextprotocol/modelcontextprotocol#3247) was scored through: with the entry in place, reviewers can
reproduce the submission with a single `tier-check --sdk-path <checkout> --requirements 2025-11-25,2026-07-28`,
the same way every other registered SDK is checked.
The entry drives the same fixtures the SDK's own CI runs through `rake conformance`: `conformance/server.rb`
behind the `conformance:server` rake task (PORT from the environment; PORT=3000 matches the registry's port convention)
and `conformance/client.rb`, which reads MCP_CONFORMANCE_SCENARIO from the environment and the server URL from argv,
per the runner's contract. The baseline is `conformance/expected_failures.yml`. One dual-era server serves every revision
from the same /mcp endpoint (stateful dated-spec handshake and the SEP-2575 stateless lifecycle side by side),
so the entry needs no specOverrides.
Testing: `npm test` (525 passing, including a new known-sdks entry test), `npm run check`, and `npm run build` pass.
The entry has already carried four full end-to-end audit runs: the 2026-08-14 and 2026-08-15 tier-check runs against
a local ruby-sdk checkout (`--sdk-path` with `--requirements 2025-11-25,2026-07-28`) each built the SDK, started the
server on PORT=3000, and completed both legs at both revisions, producing the 67/67 server and 50/50 client results
submitted in modelcontextprotocol/modelcontextprotocol#3247. Individually, `sdk ruby-sdk --path <checkout> --mode
server --requirements 2025-11-25` exits 0 with 78/78 checks, and `--mode client --requirements 2025-11-25` exits 0
with every scored scenario passing (failures confined to not_scored extensions).
Routes review of /src/scenarios/authorization-server/ and
/src/runner/authorization-server.ts to the conformance-auth team
(managed in modelcontextprotocol/access).
Claude-Session: https://claude.ai/code/session_01FDRFtxQbYLnBcpwndGXF7W
Co-authored-by: Claude <noreply@anthropic.com>
Lockstep SDKs publish in the hours before the spec release is tagged
(for 2026-07-28: python-sdk 3h early, go-sdk 3.5h, typescript-sdk the
evening before), so "first SDK release at or after the spec" read them
as not having released and would fail them 30 days later. Allow a 24h
lead.
There is no way to answer "which scenarios must my implementation pass
to conform to spec revision X". The suite accumulates scenarios
continuously, so what an implementation is measured against keeps
growing after the revision shipped, and it can drop below 100% while
standing still. --spec-version does not answer it: it filters on which
revision a scenario targets, not on when the scenario became a
requirement.
Add requirements/<revision>.yaml naming the scenarios a revision
requires, and --requirements <revision> on the client, server, list and
sdk commands to run exactly that set. The revision also fixes the wire
version its scenarios speak: the dated revisions through 2025-11-25 use
the stateful initialize handshake and 2026-07-28 is stateless with
per-request _meta, so a scenario belonging to two revisions runs once
under each and one run does not cover the other. Passing no flag keeps
today's behaviour of running everything.
tier-check gains --sdk <name[@ref]> / --sdk-path <dir>: for each claimed
revision it resolves that revision's build, server invocation and client
command from the sdk-runner's KNOWN_SDKS specOverrides, manages the
server, runs both legs with cwd = the SDK checkout, and merges. Every
claimed revision must pass for Tier 1. This is what makes multi-revision
tiering correct for SDKs whose wire era is fixed per server process
(go-sdk only speaks >= 2026-07-28 when constructed stateless) or per
endpoint (csharp-sdk serves it at /stateless): one
--conformance-server-url cannot describe them, and their config already
does. The single-URL mode remains for SDKs not in the config.
Ship two sets. 2026-07-28 is a snapshot of 0.2.0-alpha.10, published the
day before that revision shipped. 2025-11-25 is a reconstruction from
the same anchor and its header says so: the release current on its ship
date predates spec-version awareness entirely. not_scored entries run
and are reported with a reason (extension per SEP-1730, or
added-after-release) but never affect a pass rate or the exit code, and
--expected-failures judges scored scenarios only. Reports label every
entry with its revision, and JSON passed/failed/total describe the
scored set so they always agree with pass_rate.
tier-check's exit code now gates on the machine-checkable half: nonzero
when a scored scenario fails or a leg could not be measured. A child run
that cannot happen at all (a set naming a renamed scenario, an
unreachable server) reports as not measured rather than as 0% against
the implementation. Conformance runs to completion before the GitHub
checks, so an API failure can no longer interrupt a leg and orphan a
managed server, and the legs spawn their child asynchronously: the
previous synchronous exec blocked the event loop, the parent stopped
draining the managed server's log pipes, and a chatty server (rust-sdk's
per-request tracing) froze on a full pipe and was scored as failing.
Driving every tier-1 SDK through the new path also fixed three configs:
csharp-sdk's server now launches from its build-output directory so
ASP.NET finds appsettings.json (its AllowedHosts filter, which
dns-rebinding-protection tests, silently never loaded from the repo
root); rust-sdk's STATELESS=1 override is removed because its CI runs
one default server for both revisions and the forced mode fails its
SEP-2575 scenarios; python-sdk's build mirrors its CI's two targeted
syncs because --all-packages is unbuildable on main. withManagedServer
now waits for its child to exit before returning, so consecutive runs on
one port cannot race a dying predecessor.
* feat: add client-side JSON Schema 2020-12 preservation scenario
The mock server, using the version-aware `MockServer` abstraction from #321,
first advertises a focal tool with a detailed inputSchema and also offers a
permissive `json_schema_echo` tool. The client under test retrieves the
inputSchema via `tools/list`, then sends the same inputSchema back to the server
using the echo tool, completing the round-trip.
The scenario diffs from the echoed schema against the canonical fixture. It
emits eight per-keyword conformance checks. SEP-2106 vocabulary checks are
soft-gated using the existing sep2106KeywordCheckStatus helper. They only flip
to FAILURE during DRAFT-2026-v1 runs.
Signed-off-by: Satoshi Ito <satoshi.ito.tf@hitachi.com>
* Drive draft-target preservation tests through the stateless lifecycle
The stateful test helper passed createServerStateful bare, which no longer
compiles now that it takes a second specVersion argument, and forcing the
stateful mock at the draft version trips wire-schema validation (draft
results require the SEP-2549 resultType/ttlMs/cacheScope members). Use
testScenarioContext(DRAFT_PROTOCOL_VERSION), which selects the stateless
mock server, and drive it with sendStatelessRequest, matching the pattern
in json-schema-ref-deref.test.ts.
* Reword client keyword-preservation rows as issue-derived expectations
SEP-2106 §1 contains no normative preservation sentence, so the six new
sep-2106.yaml rows should not attribute a spec MUST to it. Reword the
remaining rows as expectations derived from issue #101. Also drop the
three rows whose check IDs are not sep-prefixed: the traceability
manifest only collects IDs matching /^sep-\d+-/, so those rows could
never be marked tested.
* Replace stale DRAFT-2026-v1 references with the 2026-07-28 draft name
The draft protocol identifier was renamed in #331. Interpolate
DRAFT_PROTOCOL_VERSION into the SKIPPED errorMessage suffix (matching
the server-side scenario) and update the header comment.
---------
Signed-off-by: Satoshi Ito <satoshi.ito.tf@hitachi.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* feat: add session-lifecycle conformance scenario for streamable HTTP
This commits a `server-session-lifecycle` scenario that verifies the server honors the two RFC 2119 statements the Streamable HTTP transport spec places on session termination:
1. After receiving an HTTP DELETE bearing the issued Mcp-Session-Id, the server accepts it (2xx) or signals that explicit termination is not supported (405).
2. Subsequent requests bearing the terminated session ID MUST get HTTP 404 Not Found.
The scenario inlines a raw `fetch` initialize/terminate/probe flow so the DELETE is the test action (not background cleanup).
Stateless servers that never issue a session ID are reported as INFO, and the lifecycle checks are SKIPPED.
Servers that return 405 on DELETE skip both checks without flagging a failure — the spec allows servers to refuse explicit termination.
Refs #79
Signed-off-by: Satoshi Ito <satoshi.ito.tf@hitachi.com>
* fix: address review feedback on session-lifecycle scenario
- Adapt run() to the RunContext signature introduced in #318.
- Send the protocol version the server negotiated in the initialize
result on follow-up requests instead of hardcoding 2025-11-25,
falling back to the requested spec version (the sse-multiple-streams
pattern from #415/#416). Parsing the initialize body is bounded by a
5s timeout, and every fetch shares one AbortController aborted when
the scenario ends, so a server answering with a never-ending SSE
stream can neither hang the run nor leak a held-open connection.
- Fail the scenario when initialize itself fails; previously a broken
server (401/500) was reported as stateless and passed.
- Check the initialized notification response and fail when it is
rejected, emitting SKIPPED placeholders for the lifecycle checks so
per-check aggregation keeps its rows.
- Report unexpected DELETE statuses as WARNING instead of FAILURE (the
spec never pins a success status for session termination).
- Treat 404 on DELETE as an already-terminated session only when the
follow-up request confirms the session is gone; a 404 from a server
whose session still responds (likely no DELETE route) is a WARNING
with the 404 check skipped, rather than a misattributed failure.
- Exclude the scenario from the draft spec version via removedIn:
sessions were removed entirely by SEP-2575.
---------
Signed-off-by: Satoshi Ito <satoshi.ito.tf@hitachi.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* fix: send HTTP DELETE to terminate server sessions after each scenario
The conformance test harness opened sessions against servers under test but never issued an HTTP DELETE to terminate them, leaving a dangling session on the server after each scenario.
This violates the "hermetic" expectation called out in the Streamable HTTP transport spec.
- `connection/sdk-client.connectToServer` now calls `transport.terminateSession()` before `client.close()`. A 405 response (DELETE not supported) is tolerated because the spec allows the server to refuse.
- A new `terminateSessionRaw` helper handles scenarios that bypass the SDK client and open sessions via raw `fetch` (`lifecycle.ts`, `http-standard-headers.ts`).
- `sse-polling.ts` and `sse-multiple-streams.ts`, which manage their own transport, now terminate the session in their existing `finally` block before closing the client.
- `lifecycle.test.ts` gains two cases asserting that DELETE is sent with the issued session ID and that no DELETE is sent when the server did not assign one.
Verified manually against `examples/servers/typescript/everything-server.ts` that running a server scenario now causes the server to log `Received session termination request for session <id>`.
Fixes#79
Signed-off-by: Satoshi Ito <satoshi.ito.tf@hitachi.com>
* Use negotiated protocol version for teardown DELETE and bound it
Review follow-ups on the session-termination change:
- terminateSessionRaw: abort the DELETE after 5s so a wedged server
cannot stall suite teardown.
- terminateSessionBestEffort: shared SDK-path teardown helper that
delegates to terminateSessionRaw with the transport's negotiated
protocol version, so the socket is truly released at the bound and
the version-mismatch 400 on strict servers (the failure mode of
issue #79) goes away. Used by connectToServer close() and both SSE
scenarios, replacing their inline try/catch blocks.
- lifecycle.ts: the raw session-id probe and its teardown DELETE now
use ctx.specVersion instead of a hardcoded 2025-11-25, matching the
negotiated-version convention from #415/#416.
- Add sdk-client.test.ts regression tests asserting close() sends a
DELETE with the issued session ID (and none without one), so a bad
future merge cannot silently drop the fix.
---------
Signed-off-by: Satoshi Ito <satoshi.ito.tf@hitachi.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* feat(sdk-runner): add rust-sdk to KNOWN_SDKS
The mcp-conformance package's fixture bins follow the harness contract
directly (URL argv + MCP_CONFORMANCE_SCENARIO env), and the server
selects the SEP-2575 stateless lifecycle via STATELESS=1. Requires the
repo's pinned Rust toolchain (rustup installs it on demand).
* feat(sdk-runner): per-spec-version config overlays (specOverrides)
Several KNOWN_SDKS entries needed a different invocation when targeting
2026-07-28 — go-sdk's -stateless flag, csharp-sdk's /stateless URL,
rust-sdk's STATELESS=1 env, python-sdk's per-revision baseline — and each
carried a comment telling the caller which flag to paste. Encode those
variants in the entry instead: specOverrides maps a canonical spec
version to a partial config that is merged over the base entry when a
run targets that version (--spec-version, after 'draft' alias
resolution, or the entry's own specVersion default). Explicit CLI flags
still win.
Keys are validated against the known spec versions at schema level so a
'draft' key or a typo'd date fails loudly instead of silently never
matching.
* style: prettier fixes in sdk-runner tests
* refactor(sdk-runner): reuse isSpecVersion for specOverrides key validation
Reuses the canonical spec-version list from #421 instead of keeping a
separate copy; SPEC_VERSION_TIMELINE is now exported so the validation
message can enumerate the valid keys.
Follow-ups from the #399 review:
- Stateless mock: validate error replies with a placeholder id so the
JSON-RPC-mandated id:null is not recorded as a harness violation, and
skip inbound validation when express left the body unparsed (wrong or
missing Content-Type) — both were false positives blamed on the wrong
party. Same unparsed-body guard in the stateful mock.
- Stateful mock: validate client traffic against the protocol version
the client declares via MCP-Protocol-Version, falling back to
2025-03-26 when the header is absent (it only exists from 2025-06-18,
whose spec says to assume 2025-03-26), instead of always using the
run version.
- sdk-client: validate batches once as a whole array instead of also
per element, which double-recorded every violation.
- Make specVersion required on connectToServer/connectStateful/
createServerStateful; add isSpecVersion to types.ts.
- Extend the NumberSchema erratum to 2025-06-18 (same generator bug on
minimum/maximum, modelcontextprotocol#3139) and add a tripwire test
that forces erratum deletion once the schemas are re-vendored.
- Document the wire-schema-valid / wire-schema-harness-error check IDs
in the README.
* Vendor schema/{version}/schema.json alongside schema.ts
sync-schema now copies the spec repo's JSON Schema for each version into
src/spec-types/{version}.schema.json (same pinned SHA recorded in SOURCE:
modelcontextprotocol@f817239f4d6b1efff2c4dfc2f7af85c985d73076). These are
the inputs for runtime wire-message validation. Vendored verbatim, so they
join schema.ts in .prettierignore.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JHNrpU3Vr6Q1rPZiji7zLY
* Validate all wire messages against the per-version spec JSON schema
Every JSON-RPC message the harness sends or receives is now validated
(ajv) against the vendored schema.json for the run's spec version, at the
wire choke points: sendStatelessRequest (outbound request, JSON response,
SSE events), connectStateful (request/result/notifications), and both mock
servers (inbound client requests, outbound responses/handler results).
Beyond the JSONRPCMessage envelope, messages are validated against the
most specific definition the schema has: typed requests/notifications by
method const, typed error responses by error.code const (e.g. -32021 ->
MissingRequiredClientCapabilityError, the #376 hallucination), and typed
results via the XxxRequest -> XxxResult pair, with SEP-2322
InputRequiredResult discriminated on resultType.
Violations are recorded per scenario and surfaced by the runners as a
synthetic wire-schema-valid check (FAILURE when the implementation under
test sent an invalid message) plus a wire-schema-harness-error check when
the invalid message was harness-authored. A vitest-wide afterEach guard
fails any self-test that recorded violations, so hallucinated fixtures
can't ship (#376, #271). Opt-outs are explicit and greppable: per-call
skipValidation for deliberately malformed traffic, takeWireViolations()
drains in tests that exercise deliberately broken fixtures.
Offenders the new validation caught, now fixed:
- everything-server stateless path omitted resultType (and the caching
hints on server/discover) required by the draft schema
- sep-2164-empty-contents fixture omitted resultType/ttlMs/cacheScope
(its intended violation, empty contents, is schema-valid prose-only)
- connection/mock-server unit-test fixtures used schema-invalid stubs
(tools/call without name, ListToolsResult without required members,
Tool without inputSchema, ProgressNotification without progressToken,
ElicitRequest without message/requestedSchema)
- http-standard-headers test mocks returned bare { tools: [] } results
- input-required-result-validate-input deliberately sends malformed
inputResponses; now marked skipValidation
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JHNrpU3Vr6Q1rPZiji7zLY
* Validate the stateful wire at the SDK transport, close review-panel gaps
Address the review-panel findings on the wire-schema validation PR:
- Stateful wire: hook the SDK transport inside connectToServer, wrapping
send/onmessage so every raw wire message in both directions is
validated for the scenario's spec version -- including the initialize
handshake, server->client requests (elicitation/sampling), and the
harness's responses to them. Request ids are correlated per direction
so responses validate against their typed result definition (e.g. an
outbound ElicitResult). The reconstructed stand-ins in
connectStateful (fake id: 0 envelopes, Zod-reparsed results) are
removed; the hook sees the real bytes. The four scenarios that call
connectToServer directly (elicitation-defaults, elicitation-enums,
tools, lifecycle) now pass ctx.specVersion and get validation for
free; the runner surfaces their violations through the existing
recorder. Known remaining gap (documented in the module doc): the
client-auth scenarios' bespoke express mock is not instrumented.
- skipValidation now skips only the request direction; the
implementation's response to a deliberately malformed request is
still validated, with a narrow carve-out tolerating the JSON-RPC 2.0
id: null an error response must carry when the request could not be
processed (the MCP schema's RequestId forbids null). Covered by new
unit tests pinning both the still-validated response and the
narrowness of the carve-out.
- negative-mrtr.test.ts: the file-wide drain now asserts every recorded
violation is implementation-origin (the deliberately broken fixture)
and throws on any harness-origin violation.
- everything-server: hoist the cacheable-methods set to module scope
and replace the per-request res.json monkey-patch with an explicit
sendStatelessJson(res, method, payload) helper applied at each
dispatch site.
- Pin the extracted dispatch maps (methodDefs/errorDefs, e.g.
-32021 -> MissingRequiredClientCapabilityError under the draft) per
spec version in wire-schema.test.ts, so a schema sync that breaks the
const extraction fails loudly instead of degrading to envelope-only
validation. Also note the batch branch's requestMethod forwarding
limitation.
Offender the transport hook flushed out: the everything-server's
test_elicitation_sep1034_defaults tool sent a number-typed elicitation
field with default: 95.5, which the 2025-11-25 schema forbids
(NumberSchema.default is typed integer there; widened to number only in
the draft). The fixture and the scenario's expected default are now the
integer-valued 95, with comments explaining why.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JHNrpU3Vr6Q1rPZiji7zLY
* Condense comments to three lines max
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JHNrpU3Vr6Q1rPZiji7zLY
* Re-vendor spec schemas at pinned SHA 71e30695
Regenerate src/spec-types/*.schema.json via npm run sync-schema at the
SHA already recorded in SOURCE. The previously vendored draft schema
predated spec PR #3002, so it still required DiscoverResult.serverInfo
at the top level; the mock server now correctly reports serverInfo via
result _meta, which the stale schema rejected.
* Scope the wire-violation recorder per scenario for parallel suite runs
Client --suite runs execute scenarios concurrently over what was a
module-global violation recorder, so scenarios could steal or wipe each
other's recorded violations. Give each suite scenario its own recorder
via AsyncLocalStorage; code outside a scope (single-scenario runs, the
sequential server suite, vitest hooks) falls back to the global
recorder. Also condense a four-line comment in connection/stateful.ts
to three lines.
* Patch the 2025-11-25 NumberSchema generator bug as a load-time erratum
The released 2025-11-25 schema.json types NumberSchema
minimum/maximum/default as integer, contradicting its own schema.ts
(default?: number) — a typescript-json-schema artifact, fixed for draft
in modelcontextprotocol#2710 and proposed for 2025-11-25 in
modelcontextprotocol#3139. Patch the schema at load time so SDK
fixtures with fractional number defaults (all tier-1 SDKs send 95.5)
are not flagged; restore the sep-1034 scenario and example-server
fixture to 95.5. Delete the erratum when #3139 lands and the schemas
are re-vendored.
Also scope the sequential server-suite loop with withWireRecorder so a
connection leaked by a failed scenario cannot attribute late traffic to
the next scenario.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* feat(sdk-runner): expand KNOWN_SDKS to all tier 1 SDKs
- go-sdk: point at the relocated conformance/ fixtures (the old
examples/server/conformance path is gone on main), add the
everything-client fixture that now exists upstream, and wire
conformance/baseline.yml as the expected-failures baseline.
- python-sdk: new entry for main, which targets the 2026-07-28
revision (same uv workspace layout as the v1.x entry).
- csharp-sdk: new entry driving the ConformanceClient /
ConformanceServer fixtures via the .NET 10 SDK. The client takes the
scenario as argv[0], so the command bridges MCP_CONFORMANCE_SCENARIO
into argv; the server exposes the SEP-2575 stateless lifecycle at
/stateless.
* docs: match the sdk override example's lead-in to its content
The example now shows field overrides on a built-in entry rather than a
one-off ref, so the introducing sentence shouldn't promise a ref
override.
* fix: use negotiated protocol version in sse-polling raw requests
The server-sse-polling scenario hard-coded mcp-protocol-version:
2025-11-25 on its raw tools/call POST and Last-Event-ID reconnection
GET instead of the version negotiated during initialize — the same
bug class as #412. A server that enforces the negotiated session
version rejects those requests once it negotiates any other version.
Read transport.protocolVersion after client.connect() and use it for
both raw request headers, falling back to the run's spec version,
mirroring the sse-multiple-streams fix (#415). Add a regression test
whose fixture server down-negotiates to 2025-06-18 and 400s any
request whose MCP-Protocol-Version header does not match, so a fix
that hard-codes the latest version cannot pass.
Follow-up to #415.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0178cz8nAyFRHQyq9cg2XgZz
* test: harden sse-polling regression fixture per review panel
Enforce session id alongside protocol version in the fixture, record
Last-Event-ID presence per GET so the SDK's automatic standalone GET
cannot satisfy the reconnect assertion, normalize the header type
instead of casting, return a descriptive 400 body on the GET path,
add Allow to 405s, use the spec-shaped empty-data priming event, and
read sessionId via the SDK's public getter in the scenario.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0178cz8nAyFRHQyq9cg2XgZz
---------
Co-authored-by: Claude <noreply@anthropic.com>
Tool handlers issued sampling/createMessage and elicitation/create via
mcpServer.server.request(), which carries no relatedRequestId. The SDK's
Streamable HTTP transport routes such requests to the standalone GET SSE
stream; if the client hasn't opened it yet, the request is silently parked
in the event store and the scenario times out after 60s.
Switch test_sampling, test_elicitation, test_elicitation_sep1034_defaults,
and test_elicitation_sep1330_enums to the handler's extra.sendRequest(),
which sets relatedRequestId so the request rides the originating tools/call
POST SSE stream.
Fixes#407
Claude-Session: https://claude.ai/code/session_01Qz5WunPTnDHVot4C2J6sFz
Co-authored-by: Claude <noreply@anthropic.com>
The stateful server-sse-multiple-streams scenario hard-coded
mcp-protocol-version: 2025-03-26 on its three raw concurrent POSTs
instead of the version negotiated during initialize. A server that
enforces the negotiated session version correctly returns 400 to all
three POSTs, so the scenario false-failed before exercising the
multi-stream behavior under test.
Read transport.protocolVersion after client.connect() and use it for
the raw POST headers, falling back to the run's spec version if the
transport does not expose one. Add a regression test driving the
scenario against a fixture server that 400s any request whose
MCP-Protocol-Version header does not match what it negotiated.
Fixes#412
Claude-Session: https://claude.ai/code/session_0178cz8nAyFRHQyq9cg2XgZz
Co-authored-by: Claude <noreply@anthropic.com>
A baseline entry names a scenario, but a scenario is many checks:
server-stateless is 29. Excusing the one or two a spec change breaks
means baselining the whole scenario, which stops enforcing the rest.
An entry may now be '<scenario>:<check-id>', in which case every failing
check in that scenario is judged on its own. Bare scenario entries are
unchanged, and YAML parses '- a:b' as a plain string, so every existing
baseline keeps working untouched.
loadExpectedFailures now owns the entry grammar and returns parsed
BaselineEntry values, so evaluateBaseline only matches and cannot throw.
Rejecting a bad entry and interpreting a good one no longer live in
separate layers.
A check id matches all of its occurrences, since ids repeat within a run;
they collapse to one verdict, most-severe first. Absent and SKIPPED are
tolerated, matching how the runner already treats them as green.
Move collapseDuplicateChecks to src/checks/collapse.ts with its tests:
expected-failures.ts needs the reducer and cannot import dpop.ts's auth
server stack. No behavior change.
Reject two configs that previously coerced to an entry matching nothing:
a mapping ('- scenario: check-id', with a space) and an empty list item.
Spec PR #3002 (part of the final 2026-07-28 revision) moved serverInfo from
the DiscoverResult body to _meta['io.modelcontextprotocol/serverInfo'] and
demoted the request envelope's clientInfo from required to SHOULD. The
checks still asserted the old shape, failing SDKs that follow the final
revision.
- Vendor schema types at spec commit 71e30695.
- server-stateless: drop the missing-clientInfo rejection case and assert
the opposite (a clientInfo-less request must be served, new check
sep-2575-request-meta-client-info-optional); discover no longer requires
body serverInfo; new SHOULD-level check
sep-2575-server-identifies-in-result-meta (WARNING when the _meta
identity is absent, SKIPPED when discover itself failed).
- request-metadata (client side): the clientInfo SHOULD moves out of the
MUST check into its own id sep-2575-client-sends-client-info (WARNING on
absence), so requirement levels never blend in one check id.
- Mock servers and the everything-server example emit the _meta identity
and accept clientInfo-less requests.
Move the DPoP scenarios (auth/dpop, auth/dpop-nonce) and the WIF
JWT-bearer scenario (auth/wif-jwt-bearer) off the draft spec timeline
and register them as the io.modelcontextprotocol/auth/dpop and
io.modelcontextprotocol/auth/wif protocol extensions (SEP-2133
reverse-DNS ids), so they run under --suite extensions rather than at
--spec-version draft.
Claude-Session: https://claude.ai/code/session_012E7BriGaLimJxQr7B7ChiM
Co-authored-by: Claude <noreply@anthropic.com>
* Add DPoP client conformance scenario + shared DPoP helpers (SEP-1932, #368)
Client-anchored packaging: carries the shared DPoP foundation (dpopProof /
dpopToken helpers, the createAuthServer DPoP core, and sep-1932.yaml) together
with the client scenario, so it can merge first; the server (#369) and
authorization-server (#370) scenarios are follow-ups that depend on it.
DPoP is treated as a draft/not-yet-official feature (mirroring the WIF SEP-1933
scenario): the scenario's source is `introducedIn: DRAFT_PROTOCOL_VERSION` and
it is registered in draftScenariosList (informational, not scored). It is NOT
registered in EXTENSION_IDS, since the ext-auth DPoP extension is still on a
branch rather than merged.
Client checks (the two extension MUSTs, RFC 9449 §7.1 / §4.2-4.3):
- sep-1932-client-dpop-auth-scheme — token presented with the DPoP scheme.
- sep-1932-client-fresh-proof — a fresh, well-formed DPoP proof per request.
The test MCP server judges the client via an opt-in DPoP middleware
(dpopResourceAuth) passed through createServer's authMiddleware hook; a compliant
example client (covered by the Client Draft Scenarios loop) plus deliberately-
broken bearer/replay variants drive the pass/fail acceptance tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* client/dpop: split into nonce-less baseline + nonce-required scenario
Server-provided nonces are optional in RFC 9449 (AS §8 "MAY", RS §9 "can
also choose"), and the nonce-less flow is the common case. The single
auth/dpop scenario hard-wired nonce enforcement on both the test AS and
MCP server, so a plain nonce-less proof was never accepted end-to-end and
a client that only does plain proofs was never exercised on its happy path.
Parameterize DPoPClientScenario with `requireNonce` and register it twice:
- auth/dpop (requireNonce=false) — nonce-less baseline; emits the
three baseline checks (token-request-proof, dpop-auth-scheme, fresh-proof).
- auth/dpop-nonce (requireNonce=true) — prior behavior; adds as-nonce and
rs-nonce (five checks).
Check IDs are reused (no new IDs, no sep-1932.yaml change); the two nonce
checks are emitted only by auth/dpop-nonce, so traceability stays intact.
Add auth-test-dpop-no-nonce.ts (nonce-incapable but otherwise compliant)
asserting SUCCESS against auth/dpop — proof that a client with no nonce
handling still completes DPoP when the server does not require a nonce.
Baseline negatives stay on auth/dpop (nonce-independent); the two nonce
negatives are re-pointed to auth/dpop-nonce.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* client/dpop: round-4 nonce-surface correctness fixes
Address review findings on the auth/dpop-nonce posture:
- Record the token-request proof observation BEFORE the §8 nonce challenge,
so a client that is challenged but does not retry is no longer mis-reported
by token-request-proof as "never completed a token request" (its proof was
just validated). (#2)
- Record the resource-request jti/replay on ANY valid proof (before the §9
nonce gate), so fresh-proof is never asserted on zero evidence and a client
that never honours the RS nonce fails ONLY rs-nonce. (#5)
- Require challengeIssued && honored for as-nonce/rs-nonce SUCCESS, closing a
vacuous-pass path (a client that pre-sends the nonce is never challenged). (#3a)
- Gate the AS nonce observation on the authorization_code exchange, matching
recordTokenRequestProof (a refresh exchange must not satisfy §8). (#3c)
- Collapse duplicate shared token-flow check IDs (token-request, pkce-*) that
the §8 round-trip re-POST would otherwise emit twice. (#4)
- Example client: retry the token request only on a use_dpop_nonce error code,
not any 400 carrying a DPoP-Nonce header (consistent with the RS path). (#6)
- Fix the DpopTokenRequestObservation docstring ("last write wins" →
sticky-failure). (#7)
Adds an expectedSuccessSlugs option to the client test helper and tightens the
two nonce negative tests to pin the now-clean behavior (token-request-proof
SUCCESS for no-as-nonce; only rs-nonce fails for no-rs-nonce).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* client/dpop: round-5 fixes — dedupe masking, wording, replay decision
- dedupeSharedChecks was masking failures: it kept the last occurrence of a
duplicate check ID regardless of status and ran in BOTH postures, so a
FAILURE on a challenged POST (or an earlier attempt in a restarted flow) was
silently dropped. Replace with an exported, unit-tested collapseDuplicateChecks
that prefers the MOST-SEVERE occurrence, and only apply it in the nonce
posture (the baseline keeps genuinely distinct repeated attempts). (R5 #1)
- Reword the token-request-proof description: it can be SUCCESS for a client
that presented a valid proof but never obtained a token (challenged, no
retry), so it no longer claims "obtaining a DPoP-bound access token". (R5 #2)
- Reword the as-nonce/rs-nonce failure messages to cover a client that DID
retry but whose retry proof was rejected (not only "did not retry"). (R5 #3)
- Document the deliberate replay decision: recording jti on challenged requests
means re-sending the identical proof across the challenge/retry boundary trips
replay detection — a genuine RFC 9449 §4.2 jti-uniqueness violation. (R5 #5)
Adds src/scenarios/client/auth/dpop.test.ts unit-testing collapseDuplicateChecks
(prefers failures, keeps INFO, preserves order) — the dedupe fix is now pinned.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* client/dpop: round-6 polish — pin collapse wiring; docstring
- Add an integration assertion that the §8/§9 double-POST is collapsed to one
token-request/pkce-* entry each in auth/dpop-nonce, so the requireNonce gating
of collapseDuplicateChecks can't be silently deleted or inverted (previously
mutation-survivable). runClientAgainstScenario now returns the emitted checks
so callers can make occurrence-level assertions (backward-compatible).
- Docstring: note the severity ladder ranks SKIPPED below SUCCESS, and that
equal-severity ties keep the last occurrence.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The scenario's hand-rolled server/discover handler advertises
supportedVersions: ['2026-07-28'], but every other method is delegated
to StreamableHTTPServerTransport from the pinned SDK (^1.29.0), whose
version whitelist tops out at 2025-11-25. A SEP-2575 client that honors
the negotiated version then gets HTTP 400 on tools/list, and the
sep-2106-no-network-ref-deref check can never run.
Extend the existing server/discover workaround to the version check:
rewrite an incoming MCP-Protocol-Version: 2026-07-28 header to the
SDK's LATEST_PROTOCOL_VERSION before delegating (patching rawHeaders
too, since the SDK's Node adapter rebuilds its web-standard Request
from rawHeaders). Add a stateless-draft-client regression test.
Fixes#397
Claude-Session: https://claude.ai/code/session_01N4TdyLB13XSsPwNScYHf4C
Co-authored-by: Claude <noreply@anthropic.com>
* feat(auth): add JWT-bearer helper for WIF client conformance (SEP-1933)
Adds createWorkloadJwt and generateWorkloadKeypair to provide reusable,
tested JWT signing infrastructure for the upcoming wif-jwt-bearer scenario
(PR #2). Also extracts JWT_BEARER_GRANT_TYPE constant and migrates
cross-app-access.ts to use it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(auth): add JWT-bearer helper for WIF client conformance (SEP-1933)
Adds createWorkloadJwt and generateWorkloadKeypair to provide reusable,
tested JWT signing infrastructure for the upcoming wif-jwt-bearer scenario
(PR #2). Also extracts JWT_BEARER_GRANT_TYPE constant and migrates
cross-app-access.ts to use it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(auth): add WIF JWT-bearer scenario and negative tests (SEP-1933)
Adds the auth/wif-jwt-bearer client conformance scenario using the RFC 7523
JWT-bearer grant (urn:ietf:params:oauth:grant-type:jwt-bearer).
The scenario pre-signs valid, wrong-audience, and expired JWTs on start()
to simulate cloud workload identity tokens. The conformance AS verifies the
assertion and emits per-class checks (wif-assertion-verified,
wif-assertion-missing, wif-assertion-audience, wif-assertion-expired,
wif-assertion-malformed). Broken example clients exercise the missing-assertion
and wrong-audience failure paths.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: apply prettier formatting to wif-jwt-bearer.ts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(auth): add expired-assertion negative test; clarify wif-jwt-bearer comments
- Add runWifJwtBearerExpiredAssertion + auth-test-wif-expired-assertion.ts
to exercise the wif-assertion-expired check path (was dead code)
- Clarify WifAssertionVerified description to note iss is not validated
- Add comment explaining clockTolerance: 5 is intentional (same-run keypair)
- Add comment explaining numeric expiresIn is absolute epoch seconds
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(wif-jwt-bearer): disable DCR, detect retries, improve getChecks description
- Add WIF_CLIENT_ID constant; pre-seed provider's clientInformation() so
the SDK skips Dynamic Client Registration entirely (disableDynamicRegistration
on the auth server + pre-seeded client_id on the provider side)
- Add failedOnce/tokenRequestReceived tracking on the scenario; reject and
record wif-no-retry FAILURE if the client attempts a second token request
after the first fails
- Add hasAttempted guard in WifJwtBearerProvider.prepareTokenRequest() to
throw on retry from the client side
- Fix getChecks() sentinel description: distinguish "no request made" from
"request made but verification failed"
- Add client_id to context and Zod schema
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: prettier formatting on everything-client.ts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(wif-jwt-bearer): let errors propagate from broken example clients
The try/catch blocks were silently swallowing auth failures, so the
negative tests passed purely because expectedFailureSlugs found the
AS-emitted check — not because client error-surfacing was verified.
allowClientError: true on the test cases handles the non-zero exit.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(wif-jwt-bearer): tag as draft, add scope-rejected check, add SEP traceability
- Change specVersions from ['extension'] to [DRAFT_PROTOCOL_VERSION] and add
source = { introducedIn: DRAFT_PROTOCOL_VERSION } so the scenario is
reachable via --spec-version draft (extension tag excluded it from all
spec-version runs)
- Move registration from extensionScenariosList to draftScenariosList in
index.ts to match the tag/list convention enforced by spec-version.test.ts
- Add wif-assertion-scope-rejected check: AS returns invalid_scope for a
valid JWT when the client requests the reserved 'wif.rejected' scope;
verify client surfaces the error and does not retry
- Add runWifJwtBearerScopeRejected to everything-client.ts; add optional
scope param to WifJwtBearerProvider; add CLI entry point and vitest case
- Add src/seps/sep-1933.yaml with requirements mapped to each check ID,
covering the three deferred checks (iss, sub, jti) with exclusion rationale
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: prettier formatting
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(wif-jwt-bearer): add wif-grant-fallback check for unauthorized_client fallback
Client bug class: WIF client receives unauthorized_client from JWT-bearer grant
and silently switches to authorization_code instead of surfacing the error.
The MCP SDK retries after UnauthorizedClientError (auth.js:152-154), calling
prepareTokenRequest() a second time. WifGrantFallbackProvider exploits this by
returning authorization_code params on the second call.
Spec anchor: RFC 7523 §2.1 — clients MUST use the JWT-bearer grant type;
silent grant-type switching hides authentication failures.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: rename unused scope param in WifGrantFallbackProvider
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(wif-jwt-bearer): use realistic SPIFFE and K8s PSAT token formats
valid_jwt uses a SPIFFE JWT-SVID subject (spiffe://conformance-test.local/...)
issued by a SPIRE-style issuer. wrong_audience_jwt uses a Kubernetes PSAT
subject and kubernetes.io claims, mirroring how K8s projected service-account
tokens look in production. Token verification and client behaviour are
unchanged; the formats ground the scenario in real workload identity platforms.
k8s_issuer and k8s_subject added to context and schema for external clients
that want to construct their own K8s-style assertions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: prettier formatting on wif-jwt-bearer.ts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore(wif-jwt-bearer): drop informational k8s_issuer/k8s_subject from context
These fields were not used by any client handler and are redundant since
the K8s claims are already baked into wrong_audience_jwt.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(wif-jwt-bearer): address pre-merge review feedback
- Downgrade wif-no-retry, wif-grant-fallback, and wif-assertion-scope-rejected
to WARNING; RFC 7523 is silent on client retry and grant-type switching, so
FAILURE status was not traceable to spec text
- Add WifRetryProvider broken client that re-sends JWT-bearer after
unauthorized_client, making wif-no-retry actually exercisable as a WARNING
- Drop getChecks() sentinel (non-standard pattern that overloaded the
wif-assertion-verified check ID); test harness detects empty check sets
- Add comment on DRAFT_PROTOCOL_VERSION workaround for runner limitation
- Add comment explaining why both slash/no-slash audience forms are accepted
- Tighten schema comments for audience URL constraint and ES256 literal
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(wif-jwt-bearer): drop decorative fields and neutralise JWT fixtures
- Remove issuer and subject from context schema and scenario return value;
neither is validated by the AS or used by the client, making them
misleading. The iss exclusion rationale is updated to explain why.
- Replace SPIFFE/K8s-PSAT constants and the kubernetes.io additionalClaims
with neutral values across all three JWT fixtures; no check inspects
token format, so format flavour was decorative and invited false
assumptions about fixture semantics.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(wif-jwt-bearer): remove workaround comment, drop signing_algorithm, annotate retry provider
- Remove self-admitted workaround comment on DRAFT_PROTOCOL_VERSION; the
classification stands on its own and the comment was inviting challenge
- Drop signing_algorithm from context schema and scenario return; the field
was redundant once the value was fixed to ES256, and the assertion is opaque
to the client regardless
- Add comment to WifRetryProvider explaining it deliberately omits the
hasAttempted guard so the SDK retry reaches the AS
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(wif-jwt-bearer): export scope constants, remove unused AS field, annotate state machine
- Export WIF_TRIGGER_UNAUTHORIZED_SCOPE and WIF_REJECTED_SCOPE from
createWorkloadJwt.ts and import them in everything-client.ts; removes
duplicate bare-string declarations that could silently diverge
- Remove tokenEndpointAuthSigningAlgValuesSupported from mock AS config;
the SDK does not consume this field in the JWT-bearer flow
- Add comment to onTokenRequest clarifying wif-no-retry and wif-grant-fallback
fire on any second request after any failure, not only post-unauthorized_client
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(wif-jwt-bearer): address pcarleton review comments
- Move all broken WIF provider classes and runners to wif-broken-clients.ts;
everything-client.ts now contains only the conformant WifJwtBearerProvider
- Add sentinel in getChecks() for !tokenRequestReceived to prevent a silent
pass when the client never sends a request
- Drop wif-assertion-scope-rejected check; the no-retry behaviour after
invalid_scope is already covered by wif-no-retry, and sep-1933.yaml updated
accordingly
- Remove audience from context schema and scenario return; the field was
unused by any client implementation
- Expand scenario description to tell a client implementer what to implement
- Apply suggested description fix to wif-grant-fallback (does not assume
unauthorized_client was the specific error that preceded the fallback)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(wif-jwt-bearer): drop number type from expiresIn, use jose duration strings
Jose supports negative duration strings ('-60s', '60 seconds ago') that produce
already-expired tokens without needing absolute epoch arithmetic. Drop the
number overload, update the expired_jwt call site to '-60s', and update the
unit test accordingly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
The sep-2575-server-rejects-undeclared-capability check required
error.data.requiredCapabilities to be an array containing 'sampling'. The
schema's MissingRequiredClientCapabilityError defines it as a
ClientCapabilities object ({ "sampling": {} }), as do this repo's own
spec-types and the tasks/required-task-error scenario, so a spec-correct
server could not pass the check. It shipped because the positive -32021
branch had no self-test.
- stateless.ts: accept the object shape; each capability value must itself
be an object ({ "sampling": null } fails). State the shape in the
scenario's implementer-facing description.
- examples/servers/typescript/everything-server.ts: emit the object.
- stateless.test.ts: self-tests for object-passes / array-fails /
null-value-fails, sharing one capability-probe mock helper.
A check that cannot be exercised — the server under test lacks a
diagnostic fixture tool, or advertises a feature it then rejects — was
reported SKIPPED, which is excluded from pass/fail counts, exit codes,
and the expected-failures baseline. A server could dodge the entire
SEP-2575 capability-constraint and subscription surface (7 MUST-level
checks) and still read green, and a baselined failing scenario whose
checks degraded to SKIPPED flipped its baseline entry to stale, which
reads as fixed.
Such checks now fail with a 'Not testable:' message naming the missing
prerequisite and details.untestable for tooling, per the policy in
issue #248 (severity follows the requirement's keyword: MUST -> FAILURE,
SHOULD -> WARNING). Legitimate skips remain: optional capabilities the
server never declared, runner-level spec-version inapplicability, and
the documented tasks-status-notifications harness gap.
Also fixes a test-suite bug found en route: the stream mock's reader
lacked releaseLock, so listenToStream's teardown threw and every
stream-based assertion silently ran against zero frames.
* fix(http-custom-headers): use spec-permitted x-mcp-header types
The http-custom-headers fixture annotated three `number`-typed properties
(priority and float_val) with x-mcp-header. The SEP-2243 spec forbids this:
"MUST only be applied to parameters with primitive types (integer, string,
boolean). Parameters with type `number` are not permitted."
A spec-conformant client MUST reject (exclude from tools/list) any tool whose
x-mcp-header annotations violate the constraints, so it would drop both test
tools and never call them, failing ClientSupportsCustomHeaders through no fault
of its own.
- priority: number -> integer (its value, 42/1, and header are unchanged).
- float_val: drop the x-mcp-header annotation; a floating-point header param is
exactly what the spec prohibits and has no valid replacement. It stays in the
body and is now asserted NOT to be mirrored, like the unannotated query param.
- Treat 'integer' like 'number' in header-value validation and check-id
classification so Priority keeps its cross-SDK numeric tolerance.
* Reword float_val check to the rule actually under test
The new ClientCustomHeaderNoMirrorNumber check asserts absence of a header
for an *unannotated* parameter — the same designated-params-only rule the
query check covers. The previous description stated a 'MUST NOT mirror
number-typed params' requirement that SEP-2243 does not contain (the spec
forbids the annotation on number, with rejection as the consequence; it does
not define a client-side type filter). Reword the description and comment to
the rule actually under test, keeping the number/SEP-2243 context as the
explanatory why. Also trim the float_val schema description to match the
sibling query param style.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
The scenario has introducedIn: '2025-11-25' and tests SEP-1699 priming events,
which were introduced in that version. Sending '2025-03-26' as the client's
protocol version makes a correctly-behaving server suppress the priming event
(older clients cannot handle empty-data SSE), so the scenario WARNs against a
conformant server.
Verified against typescript-sdk@v2-2026-07-28: with this fix the scenario goes
from 'WARNING: Server did not send priming event' to '3 passed, 0 failed'.
* feat: add positive tests for the Authorization Code Grant
* fix(auth): use import type for AuthorizationServerOptions in types.ts
* fix(auth): make client-id/client-secret optional; rename --secret→--client-secret
* fix(auth): generate PKCE verifier per-run; derive S256 challenge
* fix(auth): build authorization URL with URLSearchParams; drop hardcoded resource
* fix(auth): default token_endpoint_auth_methods_supported to client_secret_basic; add 'none' branch
* fix(auth): check auth-method support before starting callback server
* fix(auth): check error param first; treat state mismatch as fatal; drop fabricated code_challenge assertion
* fix(auth): redact tokens from check details
* fix(auth): harden callback server (path-scoped, close(), error handler, clear timeout)
* feat(auth): print redirect URI and timeout hint before browser prompt
---------
Co-authored-by: Michito Okai <michito.okai.zn@hitachi.com>
The ${{ github.action_path }} template expression resolves to a backslash
path on Windows (D:\a\_actions\...\v0.1.14). When inlined into a bash run
block, bash treats backslashes as escape sequences, mangling the path.
Switching to the auto-injected $GITHUB_ACTION_PATH env var routes through
shell variable expansion, which preserves the literal value.
Fixes#150.
Co-authored-by: aditya <aditya@rapidcircuitry.com>
* feat: add JSON-based setup option to authorization server conformance test
* refactor(auth): single-schema validation for --file; drop duplicate schema/merge
* chore: move authorization-server settings sample to examples/
* test(auth): cover --file merge precedence and validation
* fix(auth): validate --file standalone via .strict(); reject partial/typo'd files
The settings file must be a complete, valid AuthorizationServerOptions on
its own; CLI flags are optional overrides on top. .strict() rejects unknown
keys so a typo'd field surfaces as an error instead of being silently
ignored. ZodError from file parsing is reported with the file path.
* chore: drop placeholder --file merge test pending CLI test pattern
The test re-implemented the merge logic locally, which can drift from
src/index.ts. Proper coverage needs either programmatic commander.parse()
or an extracted loadAuthorizationServerOptions() helper; deferring that
decision to a follow-up rather than establishing the pattern here.
---------
Co-authored-by: Michito Okai <michito.okai.zn@hitachi.com>
#359 made the metadata `issuer` honor the metadataIssuer override so the
2025-03-26 backcompat scenario is RFC 8414 §3.3-compliant. The authorize
callback's `iss` parameter was still derived as `getAuthBaseUrl()+routePrefix`,
so when a scenario sets metadataIssuer (bare origin) but keeps a routePrefix,
the callback `iss` mismatches the metadata `issuer` and an RFC 9207-conforming
client correctly rejects it.
Extract `resolveIssuer()` and use it for both the metadata `issuer` and the
`'correct'`/`'normalized'` callback `iss` so they always match.
Updates SEP-2549 caching traceability text to match
modelcontextprotocol/modelcontextprotocol#2890:
- Scope the five *-caching-hints requirements to results with
resultType: "complete" (interim input_required results carry no
hints).
- Add two excluded client MUST NOT rows from the new Cache Key
section (cache-key mismatch, multi round-trip retry results).
Neither is wire-observable since a conformant client may simply
never cache.
No check IDs change; traceability.json will be refreshed by the
scheduled traceability workflow.
Claude-Session: https://claude.ai/code/session_01Gbqx5LodtAghT5fRYFxeja
* fix(auth): make 2025-03-26 backcompat mock RFC 8414 §3.3-compliant
Drop the routePrefix from auth/2025-03-26-oauth-metadata-backcompat. The
prefix caused createAuthServer to report issuer:'<origin>/oauth' while
serving the metadata at the root /.well-known/oauth-authorization-server —
an RFC 8414 §3.3 mismatch a conforming client must reject. The MCP server
in this scenario only mounts /mcp, so the unprefixed /authorize, /token,
/register routes do not collide (the sibling endpoint-fallback scenario
already mounts them at root).
The scenario continues to test exactly what it's meant to: 2025-03-26-spec
discovery with no PRM, OAuth metadata at the root well-known path.
* fix(auth): keep /oauth prefix in 2025-03-26 backcompat, override issuer instead
Dropping routePrefix made the metadata-advertised endpoints land on the
2025-03-26 hardcoded fallback paths, so a client that fetches the
well-known doc but ignores its body would now pass. Restore the prefix
and pin the metadata issuer to the bare origin via a lazy
metadataIssuer override so the mock is RFC 8414 §3.3-compliant without
losing the fallback-vs-metadata distinction.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
The tasks extension scenarios hardcoded -32003 (Missing Required Client
Capability) and -32001 (Header Mismatch). The draft spec types renumbered
these to -32021 and -32020 per modelcontextprotocol#2907, so a server
built against current draft.ts now returns the new codes and these checks
would produce false failures.
Import MISSING_REQUIRED_CLIENT_CAPABILITY and HEADER_MISMATCH from
spec-types/draft instead of hardcoding, matching the existing
http-standard-headers scenario. The checks now assert whatever the schema
currently defines, so a future renumber propagates automatically.
Also drop the now-stale code number from the affected check-ID slugs
(for example sep-2663-server-returns-32003-when-required becomes
sep-2663-server-returns-missing-capability-when-required) and update the
sep-2663.yaml requirement text to -32021, keeping it consistent with the
already-updated sep-2575 and sep-2243 manifests.
Scenarios are still in the pending list so default CI is unaffected. The
traceability.json manifest already predates the upstream renumber and is
refreshed separately from a full suite run.
* feat(tasks): SEP-2663 lifecycle scenario (8 checks)
Adds the first scenario for the SEP-2663 io.modelcontextprotocol/tasks
extension — a single TasksLifecycleScenario covering sync vs async
dispatch, DetailedTask shape on tasks/get, tool errors vs protocol
errors, and cancellation semantics. 8 ConformanceCheck records, all
passing against a SEP-2663-conformant Go fixture.
Why "tasks" (not "tasks-v2"): SEP-2663 IS the tasks surface once it
lands; the v2 suffix is only meaningful in implementations that
maintain a v1 surface alongside, which the conformance suite does not.
Layout:
- src/scenarios/server/tasks/lifecycle.ts — scenario class
- src/scenarios/server/tasks/helpers.ts — raw-fetch escape hatch
(the SDK's typed schemas strip resultType/inputRequests/...)
- src/scenarios/server/tasks/lifecycle.test.ts — fork-local vitest
runner. Two modes: spawn a fixture binary via MCPKIT_TASKS_BINARY,
or point at an already-running server via MCPKIT_TASKS_SERVER_URL.
Skips when neither is set so it doesn't break upstream CI runs that
go through everything-server (which doesn't yet implement
io.modelcontextprotocol/tasks).
Scenario is registered in pendingClientScenariosList so
all-scenarios.test.ts skips it; promote to active once the upstream
fixture grows extension support.
Tagged ['extension', DRAFT_PROTOCOL_VERSION] — selectable via
--suite extensions and --spec-version draft.
* style(tasks): apply prettier formatting
* feat(tasks,mrtr): port full SEP-2663 + SEP-2322 scenario suite
Builds out the rest of the tasks scenarios (atop the lifecycle canary)
and adds the SEP-2322 ephemeral MRTR scenario in a sibling folder.
Both target their own fixtures; both runners are brand-neutral and
language-agnostic (TASKS_SERVER_URL / TASKS_SERVER_CMD,
MRTR_SERVER_URL / MRTR_SERVER_CMD; readiness via TCP polling).
Tasks ClientScenario classes:
- TasksLifecycleScenario (8 checks; v2-01..v2-08)
- TasksCapabilityNegotiationScenario (4 checks; v2-11/22/23/25, SEP-2575)
- TasksWireFieldsScenario (3 checks; v2-12/13/21)
- TasksRequestStateScenario (3 checks; v2-14/15/28)
- TasksMRTRInputScenario (3 checks; v2-16/17/29 partial fulfillment)
- TasksRequestHeadersScenario (3 checks; SEP-2243 request-header tolerance)
- TasksDispatchScenario (8 checks; v2-09/10/19/20/26/27/30/31)
- TasksStatusNotificationsScenario (1 check; SEP-2663 §notifications, optional)
MRTR ClientScenario class:
- MrtrEphemeralFlowScenario (7 checks + 1 SKIPPED; mrtr-01..07,
mrtr-08 deferred for spec terminology +
reference-impl reasons)
Both runners spawn the fixture via a shell command and detect readiness
by TCP-polling the URL's host/port — no log-line scanning, no
language-specific assumptions. The same env vars work for any server
implementation.
Scenarios are tagged ['extension', DRAFT_PROTOCOL_VERSION] and registered
in pendingClientScenariosList so all-scenarios.test.ts (which targets
the upstream everything-server) skips them until the fixture grows
SEP-2322 / SEP-2663 support.
* docs(tasks,mrtr): scenario READMEs for upstream porting
Restructured around ClientScenario classes (one row per class with
check-list under it) rather than per-numbered-test slugs. Documents
fixture requirements, env vars, open spec questions, and the
wire-format diff for each suite.
Per AGENTS.md, severity follows spec keyword (MUST/MUST NOT → FAILURE,
SHOULD/SHOULD NOT → WARNING). The READMEs explain why some checks emit
INFO rather than FAILURE (optional emission paths per SEP-2322).
* tasks: assert createdAt + lastUpdatedAt; factor _shared/ helpers
Two reviewer-driven additions:
1. SEP-2663 createdAt / lastUpdatedAt ISO-8601 assertion in
`tasks-server-task-creation` (per Luca's PR #262 review feedback).
The check now flags servers that emit non-ISO timestamps (epoch
seconds, RFC-2822, etc.) on TaskInfoV2 envelopes.
2. Factor cross-cutting test-harness helpers into _shared/:
- `_shared/test-runner.ts` — `waitForServerReady` (renamed from
`waitForTcpReady`; the call site cares about server readiness,
not the TCP-poll mechanism). Imported by tasks/ and mrtr/
all-scenarios.test.ts; replaces ~30 LOC of inline duplication
in each.
- `_shared/wire-format.ts` — `ISO_8601_PATTERN` constant +
`isIso8601(s)` predicate. Documented rationale for choosing a
regex over `Date.parse` (too permissive),
`new Date(s).toISOString()` (too strict), or
`Temporal.Instant.from` (Node 24+ experimental). Future
wire-shape predicates (data URI, percent-encoded filename,
etc.) can land here.
Cherry-pick footprint when graduating to upstream PR is the SEP
folder + the imported `_shared/` files. First PR through carries
them upstream; subsequent feat branches inherit via standard
upstream-sync flow.
All 9 scenario tests still pass against the Go reference fixtures.
* refactor(tasks,mrtr): use SDK Client + AnyResult instead of raw-fetch helpers
Drops initRawSession/rawRequest/rawRequestFull from tasks/helpers.ts in
favor of the SDK's Client + StreamableHTTPClientTransport, paired with
a Zod passthrough schema (AnyResult) that preserves SEP-2663 / SEP-2322
draft fields the SDK's typed schemas would strip.
headers.ts and notifications.ts keep a small inline fetch where the SDK
can't reach: per-request HTTP headers (SEP-2243) and SSE notification
observation. Both reuse the SDK session via transport.sessionId.
All SEP-2663 + MRTR ephemeral-flow scenarios pass against the Go fixture.
* style: prettier formatting on tasks/mrtr scenarios
* tasks: align ttlMs / pollIntervalMs assertions per 2026-05-07 SEP
PR 2663 commit 62758914 standardised every duration field on the Ms
suffix, integer milliseconds. wire-fields.ts now asserts ttlMs and
pollIntervalMs are present on CreateTaskResult, the legacy v1 ttl
and pollInterval keys are absent (already covered), and the interim
ttlSeconds / pollIntervalMilliseconds keys are also absent on a
post-2026-05-07 server. lifecycle.ts and the scenario README pick up
matching prose updates.
Verified by make testconf-tasks-v2 (8/8) against a renamed mcpkit
fixture, and make testconf-mrtr (7/7 + 1 SKIPPED) against the paired
MRTR surface.
* style: prettier column alignment on tasks README
* mrtr: flip MRTR_INPUT_REQUIRED_RESULT_TYPE per merged SEP-2322
SEP-2322 merged on 2026-05-06 with the variant renamed from
IncompleteResult to InputRequiredResult and the resultType discriminator
from "incomplete" to "input_required" (commit de6d76fb, per dsp-ant
request). The MRTR_INCOMPLETE_RESULT_TYPE constant was specifically
designed as a one-line flip point for this scenario.
Renames
- MRTR_INCOMPLETE_RESULT_TYPE = "incomplete" ->
MRTR_INPUT_REQUIRED_RESULT_TYPE = "input_required"
- isIncompleteResult -> isInputRequiredResult
- All "IncompleteResult" -> "InputRequiredResult" in scenario prose
and check descriptions (ephemeral-flow.ts, README.md)
SEP-2663 had not yet flipped its discriminator literal as of PR head
82fb2c4d (5/7 21:52 UTC). Caitie's 5/15 RC commitment (issue comment
4384052694 on PR 2322) tracks the alignment to "input_required" both
sides. The constant remains the one-line flip point in case the 2663
follow-up surprises us.
Tested via mcpkit's make testconf-mrtr (7/7 + 1 SKIPPED green against
a renamed mcpkit fixture) and make testconf-tasks-v2 (8/8 still
green, no regressions on the paired surface).
* mrtr: prettier formatting + fix README rename typo
Lefthook prettier reformatted column alignment on first push attempt;
README also had a stale "renamed from InputRequiredResult" — should
read "renamed from IncompleteResult". Fix both.
* mrtr-08: refresh SKIPPED prose post SEP-2322 merge
Two stale references in the mrtr-tasks-composition SKIPPED check:
- Comment block + errorMessage framed blocker (a) as "spec authors
disagree" / "input_required vs incomplete". SEP-2322 merged
2026-05-06 with "input_required" (commit de6d76fb). The blocker
now reads as "SEP-2663 has not yet aligned to the merged 2322
literal" — Caitie's 5/15 RC commitment (PR 2322 issue comment
4384052694) tracks the alignment.
- errorMessage referenced "IsIncomplete signal" — that field was
renamed to IsInputRequired on the mcpkit side in lockstep with
the SEP-2322 wire-variant rename. Updated to match.
Status stays SKIPPED because blocker (b) — the mcpkit middleware
refactor (issue 347) — is still open.
* mrtr-08: drop the resolved-discriminator framing from the SKIPPED message
After SEP-2322 merged with "input_required", the only blocker that
actually keeps mrtr-08 SKIPPED is the eager-task-creation pattern in
reference-server middleware (panyam/mcpkit issue 347). The earlier
two-blocker framing read as if the test were waiting on both, but
blocker (a) is effectively resolved for any server that emits the
merged-2322 literal — leaving (b) as the sole gate.
Tighten the comment block + description + errorMessage to lead with
the middleware refactor and demote the discriminator history to a
parenthetical aside.
* style: prettier line-break on mrtr-08 description string
* tasks-wire-fields: address PR 262 review feedback
Two requested changes from the SEP-2663 author's review pass on
modelcontextprotocol/conformance PR 262
(pullrequestreview-4254601106).
(1) Drop the interim-key absence checks. The transition window between
"ttlSeconds / pollIntervalMilliseconds" and the final
"ttlMs / pollIntervalMs" wording is over now that the merged
spec settled on the Ms suffix. Useful while the spec was in
flight, noise once it stabilized. Removes the absence checks
plus the surrounding comment + description + details fields.
(2) Add Number.isInteger() to ttlMs and pollIntervalMs validation.
Spec says integer milliseconds; the previous typeof + range check
would have allowed fractional values. Now both fields fail if
they're not integers.
README scenario table tightened: "ttlMs + pollIntervalMs present and
integer-valued; legacy ttl / pollInterval keys absent".
* style: prettier column alignment on wire-fields README row
* feat(tasks): align scenarios with the Final-merged SEP-2663 spec
SEP-2663 merged Final on 2026-05-15. Four normative wire changes baked in at merge time. This commit updates the tasks server-conformance scenarios so the suite validates each one as a MUST rather than tolerating divergence as INFO/MAY.
1. notifications/tasks/status renamed to notifications/tasks. The status-notifications scenario now FAILs on observing the legacy method on the v2 surface, and validates the new name's payload shape when emitted.
2. notifications/progress and notifications/message MUST NOT be sent for tasks. Two new absence-asserts on the status-notifications scenario, reusing the existing SSE-observation harness.
3. requestState removed from the tasks-v2 wire. The previous request-state scenario tested the deleted "Request State Management" section; replaced with absence-asserts on CreateTaskResult, DetailedTask, and notifications/tasks payloads. SEP-2322's InputRequiredResult on the MRTR surface still carries requestState and is unchanged.
4. -32003 (Missing Required Client Capability) for required-task tools when the client did not negotiate the extension. New scenario TasksRequiredTaskErrorScenario initializes without declaring the extension, calls a TaskSupport=required tool, and asserts the rejection code plus the structured requiredCapabilities payload. Documents `failing_job` as a fixture requirement.
Verified locally against panyam/mcpkit examples/tasks-v2: 8/9 scenarios pass; the request-state-removal scenario intentionally FAILs against the current mcpkit main because the implementation still emits requestState on DetailedTask. That gap is the next item on the mcpkit catch-up plan.
* chore(tasks): address PR 262 review feedback
- notifications.ts: skip the entire scenario; SEP-2663 routes
notifications/tasks over the SEP-2575 subscriptions/listen stream,
not the tools/call POST SSE the prior harness observed. Body
preserved in git blame for the rewrite follow-up.
- request-state.ts: keep the absence-asserts (per the asymmetric
recommendation in review) and reframe the doc block around the
SEP-2322 lexical-adjacency motivation rather than pre-merge SEP
history.
- required-task-error.ts: cite SEP-2575 §"Missing Required
Capabilities" as the canonical home of -32003; add SEP_2575_REF to
specReferences on the -32003 checks.
- Sweep decontextualized "merged spec dropped X" / "pre-merge SEP-2663
said Y" framing from tasks/* doc blocks and inline check
descriptions.
* chore(tasks): prettier — strip trailing blank line in notifications.ts
* chore(tasks): drop `requestState` from `tasks/update` params
`taskId` is the session-continuation handle on the tasks-v2 MRTR-resume
path; `requestState` lives only on the SEP-2322 ephemeral
InputRequiredResult surface, not on the tasks-v2 wire.
* chore(tasks): loosen strict ack equality on tasks/cancel + tasks/update
The cancel-empty-ack and tasks/update-ack checks were comparing the
whole response against the literal `{resultType:"complete"}` via
`JSON.stringify` equality. That rejected acks carrying `_meta` or any
other future result-shape metadata even though the spec permits them.
Now structural: assert `resultType === "complete"` plus absence of the
task-envelope fields (`taskId`, `status`, `result`, `error`,
`inputRequests`). `_meta` and other metadata pass through.
* chore(tasks): poll for terminal status after tasks/cancel
The cancel-empty-ack check observed status via an immediate tasks/get
after the cancel ack. Cancellation is eventually-consistent, so the
task could still be `working` (or transitioning) at the moment of
observation, making the assertion flaky against any reasonable server.
Now polls via `waitForTerminal()` and asserts the terminal status is
`cancelled`. README's `slow_compute` fixture row documents the
matching tool contract ("MUST settle to `cancelled` when
`tasks/cancel` arrives while running") so the test has a deterministic
anchor.
* chore(tasks): tasks/cancel on a terminal task is idempotent
The previous check asserted `-32602` when cancelling an already-terminal
task. That forces every client to handle the race between observing a
task's terminal status and racing in a cancel request — the task can
terminate between the two and the cancel then errors instead of acking.
Check 8 renamed to `tasks-cancel-terminal-idempotent-ack` and now
asserts the same empty-ack shape (`resultType:"complete"` + no task-
envelope fields) as on an active task. README's lifecycle table row
updated to match.
* docs(tasks): drop `requestState` from SEP-2663 wire-field listings
`requestState` belongs to the SEP-2322 ephemeral MRTR result shape
(`InputRequiredResult`), not to the tasks-v2 wire. The README listed
it under SEP-2663 wire fields in four spots — top blurb, the SEP-2663
row of the Specs table, the v1↔v2 wire-format diff, and the raw-fetch
design note. All four now describe only the actual SEP-2663 shapes;
the raw-fetch note still calls out the SEP-2322 ephemeral
`requestState` separately so the rationale for stripping it via raw
fetch stays intact. The SEP-2322 row of the Specs table is unchanged
since `requestState` is correctly listed there as an MRTR base type.
* docs(tasks): rewrite stale request-state table to match absence-asserts
The README's `tasks-request-state` table still described the prior
scenario shape (shape / echo / stale-tolerance checks), which had
assumed `requestState` was on the tasks-v2 wire. The scenario itself
was rewritten earlier into the two-check absence assertion
(`tasks-create-result-no-request-state` /
`tasks-get-detailed-no-request-state`); the README now matches.
A motivational paragraph above the table explains why a negative
test exists for a field the spec never defines (the SEP-2322
lexical-adjacency confusion vector).
* chore(tasks): assert -32003 (not -32601) for gated tasks methods
When a client calls tasks/get / tasks/update / tasks/cancel without
having negotiated the io.modelcontextprotocol/tasks extension, the
suite now asserts -32003 (Missing Required Client Capability,
SEP-2575 §"Missing Required Capabilities") instead of -32601.
The spec doesn't currently mandate this code for the gated-method
path, so the assertion is forward-looking: it follows the SEP-2575
pattern that already governs `required` tools (and that
subscriptions/listen for tasks is expected to use). README's
open-questions list calls this out so readers know it's a
spec-clarification ahead-of-the-curve.
Drops the now-obsolete "Invalid requestState — silent ack vs -32602"
open-questions item: the previous request-state scenario covered
that, but the rewritten absence-asserts scenario no longer exercises
that path.
* chore(tasks): prettier — realign README table pipes after row edits
* feat(tasks/mrtr): initRawSession helper for draft-version handshake
The SDK's Client.connect() pins protocolVersion to LATEST_PROTOCOL_VERSION
in the initialize body. Scenarios tagged DRAFT_PROTOCOL_VERSION would
still negotiate the previous stable on the wire, which means a strict
draft-only server rejects the handshake with unsupported_protocol_version.
initRawSession sidesteps the SDK: a raw fetch initialize that carries the
draft version, captures the session ID, sends notifications/initialized,
and exposes the small surface scenarios actually use - request,
requestFull, notification, close. Errors are thrown as McpError so
existing instanceof / .code checks keep working. SSE response parsing
folds in the per-scenario raw helper from headers.ts.
All 8 SEP-2663 tasks scenarios + the MRTR ephemeral-flow scenario migrate
to the raw session. tsc clean.
* feat(tasks/mrtr): wire-mode matrix; SEP-2575 _meta + SEP-2243 routing
headers
initRawSession grows a `stateless` option that picks between the
legacy session wire (initialize handshake + Mcp-Session-Id) and the
SEP-2575 stateless wire (no initialize; server/discover up front;
per-request `_meta.io.modelcontextprotocol/{protocolVersion,
clientInfo, clientCapabilities}` envelope; `MCP-Protocol-Version`
header on every call).
Tasks behavior is wire-independent in spec, so the harness can run
each scenario against both wires. To keep the legacy run green
through this PR's refactor (and to surface stateless-mode gaps
deliberately rather than as a regression), the default is legacy-
only — set TASKS_WIRE_MODES=legacy,stateless (or just stateless) to
opt the matrix on. Same toggle for MRTR via MRTR_WIRE_MODES.
Two follow-on cleanups in the same commit so the legacy run actually
passes against an SEP-2243-enforcing server:
- All request paths now auto-emit Mcp-Method (and Mcp-Name for
tools/call.params.name / resources/read.params.uri) on protocol
versions in `SEP_2243_ENFORCED_VERSIONS` (today DRAFT-2026-v1).
Scenarios that probe the mismatch path can still override via
`extraHeaders`. Empty record on older versions so the helper stays
silent when the spec doesn't require it.
- `tasks-headers-body-method-authoritative` was written under an
older SEP-2243 reading where Mcp-Method was tolerant. The current
spec (and mcpkit's PR 477 implementation, plus the upstream
http-header-validation scenario) says the server MUST reject
mismatches with -32001 HeaderMismatch. Renamed to
`tasks-headers-reject-mismatched-method` and the assertion flipped
to expect the rejection.
Verification: testconf-tasks-v2 + testconf-mrtr both green on
legacy wire against mcpkit's tasks-v2 fixture (9/9 + 1/1).
* rename TASKS_WIRE_MODES / MRTR_WIRE_MODES → MCP_WIRE_MODES
Wire-mode choice is a cross-cutting conformance concern, not a
per-suite knob. Operators wouldn't realistically toggle tasks and
mrtr to different wires (both fixtures speak the same MCP server,
either it supports both wires or one). Single env var read by both
harnesses; default unchanged at ['legacy'].
* default MCP_WIRE_MODES to legacy+stateless
The conformance suite is brand-neutral — every SDK consuming it
should validate both MCP wires for SEP-2663 / SEP-2322 behavior by
default. Pinning the default to legacy-only let mcpkit's CI stay
green while gaps in its stateless dispatcher pended, but quietly
under-tested any other SDK that adopts the suite.
Default now runs every tasks-v2 scenario × 2 wires and the mrtr
ephemeral-flow scenario × 2 wires. SDKs with only one wire
implemented pin via MCP_WIRE_MODES=legacy or =stateless. mcpkit's
own audit will need that pin (or to close panyam/mcpkit issue 480)
until the stateless dispatcher consumes per-request
_meta.clientCapabilities for the tasks extension declaration.
* helpers: send MCP-Protocol-Version on every post-initialize POST
Per 2025-11-25 transport spec §Protocol-Version Header, clients MUST
include MCP-Protocol-Version on every subsequent HTTP request after
initialize. Universal post-initialize requirement; applies to both
the legacy session wire and the SEP-2575 stateless wire.
The helper was dropping the header on legacy follow-up calls as a
workaround for our server's detectWireKind treating the header as a
stateless-wire signal. That workaround was non-spec-compliant; the
right fix is for our server to drop the header from its wire-detection
precedence list and rely on body-level _meta.protocolVersion as the
SEP-2575 signal. Filed separately.
Comment thread: PR 5 review (Luca, 2026-05-27).
* types/helpers: parameterize wire mode via ScenarioRunOptions
Drop the module-level defaultStateless flag + setDefaultWireStateless
shim that the harness was toggling in beforeEach. Wire mode now flows
as an explicit parameter through ClientScenario.run, so tests can be
parallelized safely (no shared mutable state between concurrent runs).
ClientScenario.run gains an optional ScenarioRunOptions second arg.
Existing scenarios that ignore the arg satisfy the interface as-is
(TypeScript permits an implementation to declare fewer parameters than
the interface). The 10 task/mrtr scenarios that consume wire mode add
one line each to thread `opts?.stateless` into initRawSession.
Harness loops call `scenario.run(SERVER_URL, { stateless: wire === 'stateless' })`
directly; no more beforeEach setter.
Comment thread: PR 5 review (Luca, 2026-05-27, comments 2 and 7 on parallelism).
* harness: use describe.each + it.each for the (wire x scenario) matrix
Replaces the nested for-loops in both the tasks and mrtr harnesses
with vitest's table-driven describe.each + it.each idioms. Same N x M
cardinality of tests, but vitest sees the parameter table directly:
- Reporter labels render the wire row inline:
`... > stateless wire > 'tasks-lifecycle' - all checks ...`
- Each (wire, scenario) becomes a leaf test vitest can schedule
independently, opening the door to test.concurrent / pool config
later without restructuring.
WIRE_TABLE pairs each WireMode with the boolean flag the harness
needs to pass via ScenarioRunOptions, so the `it.each` callback
receives the prepared `stateless` directly rather than deriving it
inside every test body.
No on-wire change. Behavior identical against the reference fixture
(9 task scenarios + 1 mrtr scenario, each per wire).
Comment thread: PR 5 review (Luca, 2026-05-27, comments 5, 6, 9, 10).
* helpers: extend Mcp-Name to prompts/get + tasks/* (SEP-2243 + SEP-2663)
routingHeaders now auto-emits Mcp-Name for the full standard-header
surface the spec defines:
- prompts/get -> params.name (SEP-2243 §Standard Headers)
- tasks/get -> params.taskId (SEP-2663 §Streamable HTTP routing headers)
- tasks/update -> params.taskId (SEP-2663)
- tasks/cancel -> params.taskId (SEP-2663)
tools/call (params.name) and resources/read (params.uri) were already
covered. Switch from if/else chain to a method-name switch since the
surface is now wider; cleaner to extend further.
Helper doc updated with the full surface list and a TODO for SEP-2243
§Custom Headers from Tool Parameters (Mcp-Param-* mirroring from
x-mcp-header annotations), which requires caching the tools/list
schema. Tracked as a separate follow-up; no current tasks/mrtr
scenario exercises a tool with x-mcp-header annotated inputs.
Comment thread: PR 5 review (Luca, 2026-05-27, comment 4).
* _shared/wire-mode: hoist parseWireModes out of the per-suite harnesses
Tasks and mrtr harnesses had byte-identical copies of WireMode type,
VALID_MODES, DEFAULT_WIRE_MODES, parseWireModes, and WIRE_MODES.
Hoisted to src/scenarios/server/_shared/wire-mode.ts so the single
parser drives both suites. Each harness now does:
import { parseWireModes, type WireMode } from '../_shared/wire-mode';
const WIRE_MODES: WireMode[] = parseWireModes();
No behavior change. MCP_WIRE_MODES env var semantics identical;
default still legacy+stateless.
Companion TODO landed in tasks/helpers.ts: most of that module
(initRawSession, RawSession, routingHeaders, SEP_2243_ENFORCED_VERSIONS,
readJsonRpcResponse, AnyResult, the failureCheck/skipCheck/errMsg test
helpers) isn't actually tasks-specific and belongs in _shared too.
Holding that file reshuffle for a separate follow-up PR so the move
gets its own focused review thread without crowding the SEP work.
Comment thread: PR 5 review (Luca, 2026-05-27, comment 8).
* refactor: hoist raw-session + sep-refs + checks helpers into _shared/
Picks up the TODO left in tasks/helpers.ts on PR 5: the raw fetch
session machinery, the SEP-2243 routing-header builder, the SEP_*
reference constants, and the failureCheck / skipCheck / errMsg /
AnyResult test scaffolding were never tasks-specific. They lived in
tasks/helpers.ts only because tasks/ was the first server suite to
need them. mrtr/helpers.ts re-declared SEP_2322_REF + errMsg +
failureCheck for the same reason.
Move them to src/scenarios/server/_shared/:
- sep-refs.ts: SEP_2243_REF, SEP_2322_REF, SEP_2575_REF, SEP_2663_REF
- checks.ts: errMsg, failureCheck, skipCheck, AnyResult
- raw-session.ts: JsonRpcResponse, RawSession, InitRawOptions,
initRawSession (legacy + stateless paths),
routingHeaders, SEP_2243_ENFORCED_VERSIONS,
readJsonRpcResponse, nextRawId
tasks/helpers.ts shrinks to the tasks-shaped pieces it actually owns:
TASKS_EXTENSION_ID and the waitForTerminal / waitForStatus polling
loops. mrtr/helpers.ts drops the duplicate SEP ref + the duplicate
test-scaffolding wrappers and re-imports from _shared/.
skipCheck and failureCheck in _shared/checks.ts make specReferences
required (a shared scaffold can't pick a sensible default), so all
call sites now pass the SEP they're grading against explicitly.
Pure file move + import-path churn. No behavioral change. Typecheck
+ lint clean.
* feat: emit SEP-2243 Mcp-Param-* headers from tool inputSchema annotations
Closes the TODO in routingHeaders. SEP-2243 §"Custom Headers from
Tool Parameters" requires the client to mirror primitive-typed
tools/call arguments as `Mcp-Param-<Suffix>` HTTP headers when the
tool's inputSchema annotates the property with
`x-mcp-header: "<Suffix>"`. Servers behind L7 routers / WAFs use
these headers to make routing or policy decisions without parsing
the JSON body.
Implementation:
- RawSession gains a `toolSchemas: Map<string, unknown>` cache.
- `initRawSession` issues a best-effort `tools/list` after the
handshake (after `notifications/initialized` for legacy; after
`server/discover` for SEP-2575 stateless) and populates the
cache from the response. Failures are swallowed — empty cache is
the safe default.
- `routingHeaders` takes an optional `toolSchemas` and walks the
cached schema for tools/call: for each primitive-typed property
(string / number / integer / boolean) with `x-mcp-header`, it
encodes the argument value per SEP-2243 §value-encoding (plain
printable ASCII verbatim; everything else wrapped as
`=?base64?{b64-utf8}?=`) and emits `Mcp-Param-<Suffix>`.
- Both wire flavours (legacy + SEP-2575) call routingHeaders the
same way, so the same Mcp-Param-* headers ship regardless of
transport mode.
Behavior wash for tools without `x-mcp-header` annotations — they
emit only Mcp-Method + Mcp-Name as before. Behavior wash for
non-tools/call methods. No new dependencies (uses
`globalThis.btoa` + `TextEncoder`, both standard in Node 20+).
Typecheck + lint + the existing vitest suite stay green.
* chore(_shared/raw-session): drop reference-impl naming from doc comments
Tightens three doc-comment passages that named or hinted at a
specific server-side implementation as the canonical reference. The
conformance suite is positioned brand-neutral; the doc comments now
describe the wire behavior in spec-anchored language only.
No behavioral change.
* sep-refs: point at rendered SEP pages on modelcontextprotocol.io
PR URLs freeze at merge commit; the rendered SEP pages on the spec
website absorb post-merge amendments. Switch all four refs:
SEP-2243 → /seps/2243-http-standardization
SEP-2322 → /seps/2322-MRTR
SEP-2575 → /seps/2575-stateless-mcp
SEP-2663 → /seps/2663-tasks-extension
* raw-session: use eventsource-parser for SSE parsing
Delegate the SSE branch of readJsonRpcResponse to eventsource-parser
(the same parser the MCP TS SDK uses). The hand-rolled splitter
worked for the single-frame JSON-RPC response we need but didn't
cover the full WHATWG SSE semantics — multi-line `data:`
continuation, CR/CRLF line endings, comment lines, the
`event:`/`id:`/`retry:` fields. The library covers all of it.
Already a direct dependency (3.x), so no new transitive footprint.
* test: pin readJsonRpcResponse contract for SSE + JSON branches
12 vitest cases against `_shared/raw-session.ts`'s content-type
dispatcher: application/json (id-match, mismatch, error frames) and
text/event-stream (single-frame extraction, id-filter across multi-
event streams, multi-line `data:` continuation, CRLF tolerance,
comment/retry/id field handling, error-frame return, empty body,
non-JSON data interleaving).
Builds synthetic `Response` objects locally; no fixture server
required, runs in every CI. Tests pin the contract the
eventsource-parser swap inherits — guards against future parser-
version regressions and against any subsequent rewrite that subtly
changes the SSE branch.
* tasks: isolate negative-path dispatch checks (conformance#262)
The removed-method (-32601) and unknown-taskId (-32602) checks
in tasks-dispatch-and-envelope previously sent requests missing
Mcp-Name, so strict servers validating routing headers per
SEP-2243 §Server Behavior could reject before reaching method
dispatch — making the asserted error code unreachable.
- routingHeaders(): emit Mcp-Name on tasks/result so the
removed-method check fails specifically on -32601, not
-32001 HeaderMismatch.
- validTasksParams() helper: baseline params per
tasks-namespace method; negative checks override one field
to express isolation intent declaratively.
- ISOLATION_HINT on wrong-code failures points at the most
likely cause (routing headers / _meta / params shape
short-circuit before dispatch).
Forward-safe under either reading of the open SEP-2663
client-MUST → server-MUST-reject elevation question: the added
Mcp-Name matches the body, so SEP-2243's universal mismatch
rule can't fire either way.
Verified: 18/18 tasks scenarios pass.
* tasks: rename scenario check IDs to sep-2663-* slugs
Aligns 13 tasks scenario check IDs with the SEP-2484 traceability
convention used by sep-2322.yaml and others: the slug emitted at
runtime matches the check: row in the SEP's requirement-traceability
yaml, so plan.modelcontextprotocol.io can link the SEP requirement
to its implementing scenario.
Rename map:
tasks-tools-call-without-extension-sync
→ sep-2663-server-rejects-undeclared-client
tasks-methods-gated-without-extension
→ sep-2663-tasks-methods-non-declaring-32003
tasks-server-task-creation
→ sep-2663-result-type-task-on-create
tasks-get-during-working
→ sep-2663-tasks-get-status-working
tasks-get-terminal-inlined-result
→ sep-2663-tasks-get-status-completed
tasks-tool-error-completed-iserror
→ sep-2663-tool-error-uses-completed-status
tasks-protocol-error-failed-shape
→ sep-2663-tasks-get-status-failed
tasks-cancel-empty-ack
→ sep-2663-cancel-ack-empty-result
tasks-removed-tasks-result
→ sep-2663-tasks-result-removed-method-not-found
tasks-legacy-task-param-ignored
→ sep-2663-legacy-task-param-ignored
tasks-strong-consistency-immediate-get
→ sep-2663-durable-create-strong-consistency
tasks-get-unknown-task-id-rejected
→ sep-2663-tasks-get-invalid-task-id-32602
tasks-mrtr-input-requests-on-tasks-get
→ sep-2663-tasks-get-status-input-required
Wire behavior unchanged — only the reported slugs change. Verified
18/18 tasks scenarios still pass.
Untouched (intentionally): scenario-flow gates like
tasks-extension-advertised, tasks-sync-tool-call, tasks-wire-field-renames,
tasks-immediate-result-shortcut etc., which test end-to-end flows
rather than a specific RFC-2119 sentence and surface in the
traceability manifest as untracked (same pattern as sep-2322.yaml).
* seps: add sep-2663.yaml requirement-traceability declaration
Declares the 37 conformance checks (and 12 excluded requirements)
the SEP-2663 Tasks Extension surface should produce, mapping each
normative MUST/SHOULD sentence on the SEP page to either a check ID
emitted by an existing scenario or a documented reason for exclusion.
Exclusions are all client-internal, host-internal, or architectural
requirements that aren't observable at the protocol level — confirmed
during the new-sep skill's exclusion round.
The 13 scenario checks renamed in the previous commit now match the
check: slugs declared here; the remaining ~12 yaml rows describe
requirements without a scenario emitter yet — those are gaps to fill
in follow-up work (notifications via subscriptions/listen, key
uniqueness across MRTR rounds, legacy-capability migration probes).
spec_url points at the SEP page itself (status: Final) — SEP-2663's
normative content lives on /seps/2663-tasks-extension and is mirrored
in modelcontextprotocol/experimental-ext-tasks for source-control
purposes. Every quoted text: field was verified copy-paste findable
against the rendered SEP page after stripping markdown formatting.
* tasks: enforce SEP-2663 routing-header validation on tasks surface
Two related fixes:
1. tasks/headers.ts description previously said servers "MUST NOT
require" the Mcp-Method / Mcp-Name routing headers on the tasks
surface and "MUST NOT change dispatch behavior based on them —
including when the headers disagree with the body." Both claims
are spec-incorrect. SEP-2663 author confirmed in the conformance
review thread that the client-MUST in §"Streamable HTTP: Routing
Headers" elevates Mcp-Name to "required standard header" status
for tasks/get / tasks/update / tasks/cancel, so SEP-2243's
§"Server Behavior" mismatch + missing-header rejection applies.
Description rewritten to match the spec: routing headers are
REQUIRED, server MUST reject missing or mismatched with -32001.
2. Add a negative-path check covering the most common failure mode
on the tasks surface: tasks/get with mismatched Mcp-Name MUST
return -32001 HeaderMismatch. Probes tasks/get specifically;
tasks/update / tasks/cancel follow by extension. Missing-header
case is deferred (requires raw-session enhancement to suppress
a header entirely; current extraHeaders mechanism only overrides).
3. New yaml row pinning the rule to the SEP-2243 §"Server Behavior"
sentence via row-level url override (the rule itself is SEP-2243;
the scope of application on the tasks surface is what's
distinctively SEP-2663).
Known impact: surfaces a real gap in SDKs that derived their
routing-header validation from the SEP-2243 base table only (which
lists tasks/* as NOT requiring Mcp-Name). Those SDKs will see this
check go FAILURE until they extend their validation to cover
tasks-namespace methods per SEP-2663 §540.
CI against everything-server is unaffected (skips the tasks suite
entirely; everything-server doesn't implement the extension).
* mrtr: activate mrtr-08 composition check + fold into sep-2663.yaml
Folds PR 7 (panyam/mcpconformance) into PR 262:
- Cherry-picks the mrtr-tasks-composition activation from
feat/sep-2663-mrtr-08-composition (ad4f267 upstream): real end-to-end
check driving the SEP-2663 commit 451f5e1 promotion flow
(InputRequiredResult rounds gather user_name via elicitation/create,
then CreateTaskResult on the final round escalates to async; the
final tasks/get result reflects the answer gathered during MRTR).
- Addresses LucaButBoring's PR 7 nit: replaces the inline SEP-2663
reference literal with the shared SEP_2663_REF constant (now also
pointing at the rendered SEP page).
- Renames the scenario's emitted check ID from `mrtr-tasks-composition`
to `sep-2663-mrtr-synchronous-before-task-creation` to match the
corresponding row in sep-2663.yaml — closes the gap between
declared requirement and implementing check.
- Updates mrtr/README.md: refreshes the Specs-covered table and the
per-check description table for the new slug, and drops the "Open
issues" section (both blockers are resolved — discriminator value
aligned upstream; mcpkit middleware refactored per panyam/mcpkit#347).
- Updates the sep-2663.yaml flow-gate comment block to note that
the ephemeral mrtr-* flow gates also fall into the untracked
category for the same reason as the tasks-* flow gates.
One fewer skipped check on PR 262. Verified 123/123 server-suite
tests pass against mcpkit's tasks-v2 + mrtr fixtures (post
panyam/mcpkit#499 + #501).
Closespanyam/mcpconformance#7.
* tasks/mrtr: drop legacy wire when targeting DRAFT-2026-v1
SEP-2575 (Accepted) removes the initialize handshake on DRAFT-2026-v1
and replaces it with self-contained per-request _meta. Driving the
legacy initialize+Mcp-Session-Id wire against a draft fixture is
therefore a logical contradiction: the harness asserts
`protocolVersion: "DRAFT-2026-v1"` while doing the handshake the
version removed. Servers that strictly enforce SEP-2575 correctly
reject these requests with -32602 (see Randgalt's report on
conformance PR 262, 2026-05-31); lenient servers silently accept
them and mask the spec gap.
Both effects are wrong for the harness — we should only emit
spec-permitted traffic.
Adds `effectiveWireModes(protocolVersion)` in _shared/wire-mode.ts.
It wraps `parseWireModes()` and filters `legacy` out when the target
version is in POST_SEP_2575_VERSIONS (currently just DRAFT-2026-v1;
widen when a future dated release picks SEP-2575 up). If the user
explicitly pinned MCP_WIRE_MODES=legacy against a post-SEP-2575
version, the helper warns and falls back to ['stateless'] so the
suite still runs.
Both draft-only suites (tasks, mrtr) switch to
`effectiveWireModes(DRAFT_PROTOCOL_VERSION)`. The MCP_WIRE_MODES
override remains honored within the spec-permitted set.
Verified end-to-end with a strict mcpkit fixture (panyam/mcpkit
fix/sep-2575-enforce-meta-on-draft):
- pre-fix harness → 7/18 fail with the spec-required error
- post-fix harness → 9/9 pass (stateless wire only)
* tasks/mrtr: derive wire from spec version, drop RunContext.wire
`RunContext.wire` was a parallel knob to `specVersion` that the tasks
and MRTR harnesses used to loop scenarios across both the legacy
session wire and the SEP-2575 stateless wire. After SEP-2575 collapsed
the legacy wire on DRAFT-2026-v1 (a6f7c27), the matrix shrank to a
single column on draft — and an explicit `wire` field was both
redundant and unsafe: it let callers go out of sync with the
`specVersion` they declared on the wire.
The CLI's `runServerConformanceTest` never set the field, so every
`npm start -- server --scenario tasks-* --url ...` invocation silently
emitted the legacy `initialize` handshake against a draft server,
producing requests with no `_meta.io.modelcontextprotocol/*` envelope.
Strict SEP-2575 servers correctly rejected the traffic; lenient ones
accepted it and masked the gap.
Changes:
- Drop `wire?` from `RunContext`.
- Add `isStateless(ctx)` in `_shared/wire-mode.ts`; scenarios derive
`stateless` from `POST_SEP_2575_VERSIONS.has(ctx.specVersion)`.
- Extend the CLI's spec-version inference to default extension
scenarios to draft (today every extension scenario in the repo lives
there). Without this, the inference stayed on `LATEST_SPEC_VERSION`
for tasks scenarios and kept emitting legacy traffic.
- Flatten the now-1-element wire loop in the tasks + mrtr harnesses;
remove the parseWireModes/effectiveWireModes/MCP_WIRE_MODES infra
that drove it (no remaining consumers).
Adds a vitest regression in `runner/server.test.ts` that spins up an
in-process recording HTTP server, drives `runServerConformanceTest`
at `tasks-lifecycle`, and asserts the first outgoing call is
`server/discover` carrying `_meta.io.modelcontextprotocol/*` — the
SEP-2575 fingerprint — so the CLI wire selection can't silently
regress.
* scenarios + connection: address conformance#262 layout review
Folds the parallel mini-framework the tasks/MRTR suites had grown into
the existing harness shape pcarleton called out in conformance PR 262.
No new spec coverage; existing scenarios still pass 241/241 unit tests
and the regression added for the wire-derivation fix still pins the CLI
fingerprint.
Layout
scenarios/server/_shared/ is gone. The helper bundle moves to
scenarios/server/tasks-mrtr-helpers.ts (matching the existing
input-required-result-helpers.ts precedent); raw-session.ts and its
test fold into connection/stateless.ts.
Per-suite runners + READMEs (tasks/all-scenarios.test.ts,
mrtr/all-scenarios.test.ts, _shared/test-runner.ts, both READMEs) are
removed. The scenarios are already registered in
scenarios/index.ts pendingClientScenariosList, so the CLI runner
(`npm start -- server --scenario tasks-lifecycle --url ...` or
`--suite pending --url ...`) is the supported entry point. Per
AGENTS.md: don't reimplement the runner.
Connection extension
Connection grows the three things tasks/MRTR scenarios actually need
beyond the existing surface:
- `request(method, params, extraHeaders?)` for the SEP-2243
header-mismatch tests.
- `discover(): Promise<Record<string, unknown>>` for
capability-negotiation scenarios. Resolved eagerly from the SDK
accessors on the stateful wire; issued lazily as `server/discover`
on the stateless wire and memoized.
- `ConnectOptions { capabilities?, clientInfo? }` on
RunContext.connect() so scenarios can negotiate extensions
(`extensions: { 'io.modelcontextprotocol/tasks': {} }`) at
bootstrap on either wire.
Crucially the stateless wire does NOT eagerly emit `server/discover`
when capabilities are declared. SEP-2575 says per-request
`_meta.io.modelcontextprotocol/clientCapabilities` is authoritative;
a server that gates extensions on a prior `server/discover` is
non-conformant, and the conformance suite's job is to surface that,
not paper over it.
Scenario migration
Every tasks/MRTR scenario switches from `initRawSession(...)` to
`await ctx.connect({ capabilities })`, and the capability scenario
switches from `session.initializeResult.capabilities` to
`await withExt.discover()`. The wire choice now flows entirely from
`specVersion` — `isStateless(ctx)` (moved to connection/select.ts
next to `connectFor`) is the single source of truth.
Description sweep
Each scenario's `description` string now spells out the required
server fixtures (tool names + behavior) directly. Previously that
content lived only in the file-header comment, so an implementer
reading the runner's emitted description was missing it.
Reviewer notes
- 23 files changed, +455 / -291 (net + because the Connection
abstraction now carries surface that used to live in raw-session;
the raw-session deletion was 692 lines on its own).
- Unit tests: 241 pass (the 12 deleted tests were redundant with
connection-level coverage; the genuinely-unique SSE multi-line
`data:` test pinned behavior the connection parser doesn't have).
- Reference fixture runs: tasks scenarios surface a fixture-side
spec gap (per-request `_meta` not honored without a prior
`server/discover`). That's the suite working; the gap will be
filed upstream against the fixture, not papered over here.
* tasks: dedupe MRTR ephemeral-flow against #188, fold composition into tasks/
Of the 8 checks in mrtr/ephemeral-flow.ts, 7 substantially overlapped
the SEP-2322 InputRequiredResult* scenarios that #188 already shipped
(basic-elicitation, sampling, list-roots, request-state, multiple-input-
requests, multi-round, wrong-input-key). Only sep-2663-mrtr-synchronous-
before-task-creation — the SEP-2663 commit 451f5e1 composition path
where the MRTR loop's final round returns a CreateTaskResult — is
genuinely new versus #188's coverage, and it's SEP-2663-specific, not
SEP-2322-specific.
Moves that one check to its own tasks scenario in tasks/composition.ts
(extensionId = io.modelcontextprotocol/tasks, registered alongside the
other tasks-* scenarios in scenarios/index.ts) and deletes the entire
mrtr/ directory.
Reuses the existing #188 helpers (isInputRequiredResult, mockElicitResponse
from input-required-result-helpers.ts) rather than reintroducing the
parallel mrtr/helpers.ts ones.
* address review on conformance#262 follow-up
- connectStateless.discover() now reuses the request() pipeline (send +
drainEvents + unwrap) instead of inlining a duplicate copy. The local
memoization stays via discoverPromise ??=.
- runner/server.test.ts mock server no longer silently swallows
unparseable bodies. They land in a parseFailures sidecar that the
test asserts is empty — a non-JSON body to this mock is exactly the
kind of regression the test exists to catch.
- Drop the tautological connectFor identity tests. They asserted
`typeof === 'function'` and `not.toBe(connectStateful)`, both of which
hold trivially once connectFor wraps in a closure. Wire-shape behavior
is already covered by the connectStateless mock-fetch suite and by
runner/server.test.ts.
* tasks/mrtr: address PR 262 audit before review
Pre-emptive cleanup against the upstream patterns established by recent
merges (#318 Connection abstraction, #319 draft-spec consistency, #303
SEP-2243 check IDs, #321 MockServer surface). No behavior change for
scenarios — every check is still emitted; only the shape of how they're
emitted moves toward upstream's house style.
Surface
- Revert withRequestMeta / sendStatelessRequest signatures to match
upstream byte-for-byte. The need that pushed me to widen those
(per-request _meta.clientCapabilities for extension declaration on
the SEP-2575 stateless wire) is now handled inside connectStateless:
ConnectOptions.capabilities / clientInfo are folded into params._meta
via the trailing _meta spread that withRequestMeta already supports.
This keeps the new "extension declaration" concept on the Connection
layer only — the lower-level helpers stay shared with every other
scenario in the suite.
- Document that Connection.request's extraHeaders throws on the
stateful wire instead of "ignored with a warning" — the SDK transport
manages headers internally, so silent drop in a conformance harness
would mask test correctness.
Layout
- src/scenarios/server/tasks-mrtr-helpers.ts moves under tasks/ as
tasks/mrtr-helpers.ts. All 10 importers live in tasks/; the file's
prior placement next to input-required-result-helpers.ts no longer
applies once those scenarios moved into tasks/.
- Drop the unused AnyResult export + zod import. The Connection wrapper
doesn't take a zod schema, so the helper was dead on arrival.
- SEP-2322 spec reference now re-exports MRTR_SPEC_REFERENCES[0] from
input-required-result-helpers so the codebase holds one URL per SEP
instead of two for SEP-2322.
Naming
- All 10 tasks/*.ts switch to `readonly source = { ... } as const` to
match the convention used by caching.ts, resources.ts,
http-standard-headers.ts. Drops the ScenarioSource import where now
unused.
- Scenario-scope each scenario's connect-failure check ID
(tasks-lifecycle-bootstrap, tasks-headers-bootstrap, …) instead of
sharing tasks-session-bootstrap. Matches the upstream caching.ts
pattern (sep-2549-caching-connection) where each scenario owns its
own bootstrap-failure ID.
Traceability
- sep-2663.yaml gains the positive-path counterpart to the existing
rejection requirement under Streamable HTTP: Routing Headers, so
every sep-2663-* check ID emitted by tasks/*.ts is declared in the
manifest.
Verification
- Conformance vitest: 270/270.
- Build: clean.
- mcpkit testconf-tasks-v2 (47 fork scenarios + 1 sentinel): pass.
- mcpkit testconf-mrtr (3 fork scenarios + 1 sentinel): pass.
* tasks/required-task-error: anchor -32003 checks to sep-2663.yaml rows
The two checks in required-task-error.ts each verified a distinct
normative requirement that sep-2663.yaml already declared, but the
check IDs (tasks-required-task-error-code, tasks-required-task-error-data-shape)
used the tasks-* flow-gate prefix and went untracked by the traceability
manifest.
Promote both check IDs to sep-2663-*:
- tasks-required-task-error-code → sep-2663-server-returns-32003-when-required
- tasks-required-task-error-data-shape → sep-2663-server-returns-32003-data-shape
Split the existing YAML entry to mirror the split. The spec sentence
glues two observable requirements with a trailing colon (the JSON-RPC
code, and the data-shape payload that names the missing extension);
each gets its own row so each is verifiable independently.
Remove the now-stale tasks-required-task-error-code reference from the
sep-2663.yaml flow-gate comment block — that check is no longer a
flow-gate.
No behavior change. 270/270 vitest still pass.
* runner: forward ConnectOptions from RunContext.connect
server.ts built the RunContext with `connect: () => connectFor(...)(serverUrl)`,
dropping the opts argument scenarios pass to ctx.connect(). TypeScript
doesn't catch this because `() => T` is assignable to `(opts?) => T`.
Result: scenarios declaring capabilities sent DEFAULT_CLIENT_CAPABILITIES
on the wire in CLI runs while testContext() forwarded opts correctly,
so unit tests passed but `npm start -- server --scenario tasks-*` against
a strict-draft server failed with missing-capability errors.
Tightens the regression test to assert scenario-passed capabilities reach
_meta.clientCapabilities.extensions, not just that defaults are present.
* tasks: align scenario descriptions and cancel-ack check with SEP-2663
capability.ts: description said non-declaring tasks/* requests get
-32601 MethodNotFound; the assertion checks MissingRequiredClientCapability
per SEP-2663 §Error Handling. Reworded to the named error type and
per-request _meta/server-discover framing (SEP-2575).
lifecycle.ts: description claimed cancel-on-terminal MUST return
-32602; Check 8 in the same file asserts an idempotent ack and the
SEP has no -32602 carve-out for terminal tasks. Removed the bullet.
lifecycle.ts Check 7: SEP-2663 §Task Cancellation says transition to
`cancelled` is not guaranteed; stop failing the ack-shape check on
the settled status and record it in details instead.
* connection: fix stale docstring and drop redundant eta-expansion
stateless.ts docstring referenced `connection.initializeResult`; the
implementation exposes `discover()`.
select.ts wrapped connectStateful in an identity arrow; pass it
directly now that signatures match.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
The error-code allocation policy in modelcontextprotocol#2907 renumbers
the draft-only codes: HeaderMismatch -32001 -> -32020,
MissingRequiredClientCapability -32003 -> -32021,
UnsupportedProtocolVersion -32004 -> -32022. This updates every
reference in the suite (mocks, scenarios, fixtures, tests, traceability
yaml) and re-syncs the vendored draft schema to that commit.
src/seps/traceability.json is generated and intentionally left for the
scheduled refresh workflow.
Re-vendors schema/draft/schema.ts from the spec repo at the merge commit
of PR #2891 (2fb207da428f43a4981513ec2aef218b7533c5b3), which also
includes #2889 and #2890.
Type changes picked up:
- Add NotificationMetaObject (with io.modelcontextprotocol/subscriptionId)
- Add HEADER_MISMATCH (-32001) const and HeaderMismatchError interface
- CancelledNotificationParams.requestId is now required
- ClientNotification narrowed to CancelledNotification only
- ListRootsRequest.params narrowed (drops progressToken)
- Remove ElicitationCompleteNotification and elicitationId
Also replaces the local -32001 literal in
src/scenarios/server/http-standard-headers.ts with the new exported
HEADER_MISMATCH constant.
Claude-Session: https://claude.ai/code/session_01Gbqx5LodtAghT5fRYFxeja
The SEP-2243, SEP-2322, and SEP-2106 client-direction scenarios build
their own mock servers and did not answer server/discover, so a
2026-07-28 client that negotiates via discover before its scenario calls
hit a -32601 or generic fallback and had to synthesize the response
itself. The shared stateless mock and the request-metadata and auth
scenarios already served it.
BaseHttpScenario now answers server/discover with a valid DiscoverResult
(supportedVersions, capabilities from an overridable
discoverCapabilities(), serverInfo, plus the cacheable-result members)
before delegating to handlePost; http-standard-headers overrides the
capabilities to match the methods it serves. mrtr-client and
json-schema-ref-no-deref answer it directly. New tests pin the response
shape for all five scenarios.
The auth/scope-step-up scenario emitted sep-2350-scope-union-on-reauth at
every spec version, but the set-wise union requirement (SEP-2350) was only
introduced in 2026-07-28. This wrongly warned clients that are conformant at
2025-11-25 and drop the previously-granted scope on re-authorization (e.g.
python-sdk v1, surfaced while wiring up its runner in #329).
Gate the check behind specVersionAtLeast(specVersion, DRAFT_PROTOCOL_VERSION),
a small timeline-ordering helper added to types.ts (mirrors the ordering used
by scenario applicability). Add tests pinning both sides of the branch:
enforced at draft, silent at 2025-11-25.
Follow-up to #329.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The everything-server returned its -32602 missing-_meta rejection with
HTTP 200; the draft spec requires 400 Bad Request. It also applied the
per-request _meta validation to sessionless requests carrying a legacy
session-era version header, although per-request metadata is a
2026-07-28 requirement; such requests now fall through to the session
path.
The server-stateless scenario gains a companion check
(sep-2575-http-server-meta-invalid-400) requiring HTTP 400 on those
rejections, and sep-2575-server-unsupported-version-error now also
requires error.data.requested to echo the requested version. Two new
negative tests pin both.
At the 2026-07-28 revision every server result must include resultType,
and cacheable results (server/discover, tools/list, prompts/list,
resources/list, resources/templates/list, resources/read) must also
include ttlMs and cacheScope. The client-direction mock servers omitted
these, so a strictly-validating client was failed by the suite's own
non-conformant mock.
The shared stateless mock now stamps the missing members onto handler
results and its server/discover response (withRequiredDraftResultFields),
and the hand-rolled per-scenario servers set them explicitly or route
their fallbacks through the helper. The deliberately resultType-less
probe in sep-2322-client-request-state is unchanged. New tests pin the
response shapes.
* feat: add python-sdk as a built-in SDK runner target
Wire modelcontextprotocol/python-sdk into the `conformance sdk` runner
alongside typescript-sdk and go-sdk, using the SDK's existing conformance
fixtures for both client and server modes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: target python-sdk v1.x on spec 2025-11-25
Convert the built-in python-sdk entry from v2/main to python-sdk-v1,
mirroring typescript-sdk-v1: clone the v1.x branch, default specVersion
to 2025-11-25 so draft-only scenarios/checks are excluded, and point at
the SDK's own expected-failures baseline. v2/main is mid-refactor and
noisy; v1.x is the stable published line.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: default sdk-runner server mode to the `active` suite
Server runs previously fell through to the underlying `server` command's
default; make it explicit and consistent with tiering, which runs the
`active` suite (excludes pending/draft). Avoids surfacing intentionally
-deferred `pending` scenarios (e.g. json-schema-2020-12, server-sse
-polling) as failures. Runner-wide default for all SDKs; callers can
still override with --suite.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: sync spec-types to modelcontextprotocol@0168c57f
Refreshes the vendored schema copies via npm run sync-schema. The draft
schema now declares LATEST_PROTOCOL_VERSION = "2026-07-28" (previously
the DRAFT-2026-v1 placeholder) and picks up the latest draft doc-comment
and type updates. The dated schemas are unchanged.
* fix: update the draft wire version to 2026-07-28
The spec repo's draft schema now declares
LATEST_PROTOCOL_VERSION = "2026-07-28", but the harness still asserted
the old DRAFT-2026-v1 placeholder, so an SDK that sends the version the
draft spec actually declares cannot pass the draft suite.
- bump DRAFT_PROTOCOL_VERSION in src/types.ts to 2026-07-28
- update the hard-coded literals in tests, example servers, and README
- add src/types.test.ts pinning DRAFT_PROTOCOL_VERSION to the vendored
draft schema's LATEST_PROTOCOL_VERSION so the two cannot drift, and
asserting the draft identifier stays distinct from the dated versions
* feat: vendor spec schema types per-version
Copies schema/{version}/schema.ts from the modelcontextprotocol spec
repo into src/spec-types/{version}.ts so the conformance suite can type
against draft spec versions before any SDK ships them.
npm run sync-schema -- <ref> refreshes the copies and records the spec
commit in src/spec-types/SOURCE.
* feat: add Connection abstraction and RunContext
Connection encapsulates how the conformance suite talks to a
server-under-test for a given spec version:
- connectStateful: 2025-x lifecycle. Thin adapter over the SDK Client
(initialize handshake, session id, SSE handled by the SDK).
- connectStateless: 2026-x lifecycle (SEP-2575). Raw fetch with
per-request _meta + MCP-Protocol-Version header. Decoupled from the
SDK so the suite can test draft spec versions before the SDK
implements them.
connectFor(specVersion) picks the implementation. RunContext bundles
serverUrl, specVersion and a bound connect() for the runner to hand to
each scenario.
Nothing uses this yet; wiring follows in the next commit.
* refactor: thread RunContext through ClientScenario.run
ClientScenario.run(serverUrl) becomes run(ctx: RunContext). The runner
builds the context from --spec-version and the server URL; scenarios
destructure ctx.serverUrl and otherwise behave identically.
No scenario uses ctx.connect() yet, so behaviour is unchanged: 214/214
tests pass, all-scenarios.test.ts still drives the everything-server
fixture exactly as before.
Test files use a testContext(url) helper to construct a RunContext.
The authorization-server scenario list is retyped to
ClientScenarioForAuthorizationServer since those scenarios test an
OAuth server, not an MCP server, and keep run(serverUrl).
* refactor: migrate server scenarios to ctx.connect() + conn.request()
22 carry-forward and lifecycle scenarios now go through the Connection
abstraction instead of connectToServer + SDK Client:
- tools.ts (8): list, call x6, with-progress
- prompts.ts (5)
- resources.ts (7): list, read x3, subscribe x2, not-found
- utils.ts (3): completion, ping, set-level
- json-schema-2020-12.ts, caching.ts, http-standard-headers.ts
Result types come from spec-types/{introducedIn}. Same scenario code
now passes under both --spec-version 2025-11-25 (SDK-backed stateful)
and --spec-version draft (raw stateless).
ToolsCallSampling/Elicitation/WithLogging and elicitation-* keep
connectToServer (need setRequestHandler/setLoggingLevel SDK surface);
they are tagged removedIn: DRAFT in the next commit. stateless.ts and
input-required-result.ts keep their sendRpc helper; migrating those is
deferred to the DRAFT-scenario coherence pass.
* fix: normalize Connection error to JsonRpcError; clean up RunContext import
connectStateful now catches the SDK's McpError and rethrows as
JsonRpcError so scenarios always see the same error class regardless
of which Connection impl ran. ResourcesNotFoundError uses
instanceof JsonRpcError instead of duck-typing.
types.ts uses a normal top-level import for RunContext instead of an
inline import() type.
* fix(dns-rebinding): use version-appropriate probe body
The 'valid Host accepted' check was sending an initialize body, which
a 2026 server returns 404 for. Probe body is now picked from
ctx.specVersion: initialize for the stateful lifecycle, server/discover
with _meta for the stateless lifecycle. The Host/Origin rejection check
is unchanged since rejection happens before body parsing.
* feat(everything-server): route stateless carry-forward methods to McpServer
Adds an in-memory dispatch client connected to the same McpServer the
stateful path uses. Stateless requests for tools/call, resources/*,
prompts/get and completion/complete that fall through the MRTR-specific
handlers are routed to it, so the fixture serves the carry-forward
scenarios under --spec-version draft without duplicating ~500 lines of
tool/resource/prompt registrations.
tools/list now merges the McpServer's tool list with the MRTR-only
stubs so json-schema-2020-12 finds its tool.
draft suite against the fixture: 36/39 (was 13/39). Remaining 3 are
fixture-side SEP gaps (no SSE forwarding for progress, no SEP-2549
ttlMs, no SEP-2243 Mcp-Method validation in the stateless path).
* refactor(connection): drop unused RequestOptions; move sdk-client; add tests
Addresses self-review findings on the new connection module:
- RequestOptions.handlers/.meta and ServerRequestHandler removed: zero
callers (the scenarios that motivated them are deferred). They can be
reintroduced when something actually uses them.
- scenarios/server/client-helper.ts moved to connection/sdk-client.ts so
the connection module no longer depends on the scenarios module.
- connectStateless: handle CRLF SSE event separators; throw a useful
error for non-JSON/non-SSE responses instead of a JSON parse error.
- JSONRPCNotification consistently imported from spec-types/2025-11-25.
- New connection.test.ts (8 tests) covering connectFor selection,
_meta injection, error mapping, SSE LF/CRLF parsing, and the
server-request-on-stream rejection.
* fix: address bughunt findings (response.ok check; targetVersion naming)
- connectStateless: throw on non-2xx responses that lack a JSON-RPC
error envelope (e.g. gateway/framework error JSON), matching the
stateful path's behavior. Previously such a response would return
undefined as the result.
- json-schema-2020-12: rename negotiatedVersion to targetVersion and
reword the skip message and details field. The value is ctx.specVersion
(the run's --spec-version), not a negotiated version; the old wording
was misleading. Drop dead 'unknown' fallback (specVersion is required)
and the corresponding undefined test cases.
* fix(sse-multiple-streams): keep scenario in draft; version-aware request scaffolding
Concurrent POSTs each answered with JSON or their own SSE stream are
core transport behavior in the draft spec too, so removedIn was wrong
for this scenario. Only the request scaffolding is version-specific:
stateful runs keep the initialize/session-id setup, draft runs send
_meta + the MCP-Protocol-Version header on each request instead. The
two checks are unchanged.
* feat: add MockServer abstraction and ScenarioContext
MockServer encapsulates the lifecycle scaffold a client-conformance
scenario presents to the client-under-test:
- createServerStateful: 2025-x lifecycle. SDK Server +
StreamableHTTPServerTransport (sessionless mode); the SDK handles the
initialize handshake.
- createServerStateless: 2026-x lifecycle (SEP-2575). Raw express app
that validates _meta + MCP-Protocol-Version on every request, serves
server/discover, routes other methods to the supplied handlers.
createServerFor(specVersion) picks the implementation. ScenarioContext
bundles specVersion and a bound createServer() for the runner to hand
to each scenario.
This is the client-conformance mirror of src/connection (PR #318).
Nothing uses it yet; wiring follows in the next commit.
* refactor: thread ScenarioContext through Scenario.start()
Scenario.start() becomes start(ctx: ScenarioContext). The runner builds
the context from --spec-version (defaulting to LATEST_SPEC_VERSION) and
passes it through; scenarios receive it as _ctx and otherwise behave
identically.
No scenario uses ctx.createServer() yet, so behaviour is unchanged:
231/231 tests pass.
Test files use a testScenarioContext() helper. The runner already
threads MCP_CONFORMANCE_PROTOCOL_VERSION to the spawned client process,
so the fixture-side env wiring is unchanged.
* refactor: migrate tools_call to ctx.createServer; tag 2025-only client scenarios
ToolsCallScenario now goes through ctx.createServer() instead of an
inline express + SDK Server build. Same handlers, same checks; the
assertion now reads from srv.recorded so it works regardless of which
lifecycle scaffold the runner picked.
initialize, sse-retry, and elicitation-defaults are tagged
removedIn: DRAFT (initialize/GET-SSE/SSE-embedded-elicitation are gone
in the 2026 lifecycle; the MRTR sibling for elicitation-defaults is a
follow-up).
spec-version.test.ts: the 'draft is a superset of latest' invariant no
longer holds once removedIn: DRAFT exists; the test now asserts that
any scenario in latest-but-not-draft is explicitly removedIn.
* feat(auth): make createServer helper version-aware via ScenarioContext
The auth helper now takes ctx: ScenarioContext as its first argument
and branches on ctx.specVersion inside the /mcp route: the stateful
path (SDK Server + StreamableHTTPServerTransport) is unchanged; under
the draft version a raw stateless handler validates _meta + the
MCP-Protocol-Version header, serves server/discover, and routes the
same tools/list and tools/call responses.
The PRM endpoint, bearer-auth middleware, and request logger sit above
the branch and are version-independent.
All 25 call sites across the 12 auth scenario files pass ctx through;
ServerLifecycle and the express.Application return type are unchanged
so stop()/getChecks() are untouched.
Deviation from the MockServer wrapper approach: keeping the helper's
return type as express.Application avoids restructuring 25 call sites'
ServerLifecycle handling in this PR. Folding the auth seam onto
ctx.createServer() fully is a follow-up once the lifecycle ownership
moves into MockServer.
* feat(everything-client): pick stateless requester by MCP_CONFORMANCE_PROTOCOL_VERSION
Adds a statelessRequest(serverUrl, method, params) helper that POSTs
with _meta + MCP-Protocol-Version (the SEP-2575 lifecycle), shimming
around the SDK Client not yet supporting stateless mode. The
runRequestMetadataClient handler's meta constants are extracted to share
with the helper.
runBasicClient (initialize, tools_call, json-schema-ref-no-deref) now
branches on MCP_CONFORMANCE_PROTOCOL_VERSION: for the draft version it
uses statelessRequest to call tools/list then tools/call; for dated
versions it keeps the SDK Client path.
The runner already passes MCP_CONFORMANCE_PROTOCOL_VERSION to the
spawned client, so no runner change is needed.
* fix: address review findings on MockServer (dead opts, shared validator, capability derivation, recorded parity, specVersion threading)
- MockServerOptions removed (capabilities/configure had zero callers); opts
param dropped from createServerStateful/Stateless/For and ScenarioContext.
- validateStatelessRequest extracted from mock-server/stateless and exported;
both the stateless MockServer and auth/helpers/createServer.ts call it so
_meta/header/version validation cannot drift.
- isStatefulVersion exported from connection/select; mock-server/select uses
it instead of duplicating the version set.
- runner/client.ts: env MCP_CONFORMANCE_PROTOCOL_VERSION set unconditionally
to the resolved version; runInteractiveMode now takes specVersion and the
CLI passes it.
- createServerStateful: capabilities derived from handler method prefixes;
newServer() moved inside the try so a capability mismatch surfaces as
JSON-RPC -32603 instead of an HTML 500. Recording moved to the express
layer so unregistered methods are captured (parity with stateless).
- readFinalSseMessage return type now declares error.data.
- Tests added for the capability derivation and unregistered-method recording.
* fix(mock-server): record stateless requests before validation; document preamble exclusions
The stateless impl pushed to recorded[] only after validateStatelessRequest
passed, so requests rejected for a missing header or _meta never appeared,
diverging from the stateful impl which records at the HTTP layer before SDK
dispatch. Record before validation, excluding only the server/discover
lifecycle preamble (mirroring the stateful initialize/notifications-initialized
exclusion), and fix the MockServer.recorded docstring which wrongly claimed
stateless excludes nothing.
* refactor(mock-server): tri-state result for validateStatelessRequest (reject/handled/route)
A successful server/discover was previously modeled as {ok: false},
which read as a validation failure but actually meant "already
answered, do not route". Replace the boolean discriminant with an
explicit kind: 'reject' (validation failed), 'handled' (valid and
already answered), or 'route' (dispatch to handlers).
Both consumers (the stateless /mcp route and the auth helper's
handleStateless) still write status/body for reject and handled and
route only on 'route', so responses are unchanged. Add direct unit
tests covering each branch of the new union.
* feat(runner): single-source the version→lifecycle mapping; export MCP_CONFORMANCE_LIFECYCLE
The stateless mock and the example client both hardcoded the draft
protocol version, so dating the draft (or adding a second stateless
version) would silently break them: the mock would keep rejecting
everything but the old constant, and the client would fall back to the
stateful path.
- Derive STATELESS_SPEC_VERSIONS and a lifecycleFor() helper from
isStatefulVersion() in connection/select.ts, making it the single
source of truth for the version→lifecycle mapping.
- Thread the resolved spec version into the stateless mock:
createServerFor() binds it into createServerStateless(), and
validateStatelessRequest() now takes the supported-versions list as a
parameter. A mock bound to a version accepts exactly that version and
echoes it in -32004 rejections and server/discover; unbound mocks
accept every known stateless version.
- Export MCP_CONFORMANCE_LIFECYCLE (stateful|stateless) to client
processes alongside MCP_CONFORMANCE_PROTOCOL_VERSION, in both the
runner and the in-process test helper, so clients in any language can
pick the lifecycle without keeping their own version map.
- everything-client: import DRAFT_PROTOCOL_VERSION from src/types
instead of duplicating the string, branch on MCP_CONFORMANCE_LIFECYCLE
(falling back to the version comparison for older runners), and send
the runner-resolved version on stateless requests.
- Document MCP_CONFORMANCE_LIFECYCLE in the README and ignore the local
.claude/ workspace dir in .prettierignore so prettier --check . is not
tripped by untracked worktree checkouts.
* fix(everything-client): list-only flow for json-schema-ref-no-deref
runBasicClient now follows tools/list with a tools/call on the first
listed tool, but the json-schema-ref-no-deref scenario mock only serves
tools/list, so the call got -32601 and the client exited non-zero.
Register the scenario with a dedicated handler that stops after listing
tools, keeping the existing lifecycle branch (raw stateless request vs
SDK client). Listing is the whole flow for SEP-2106: the check is that
the client never fetches the network $ref canary while processing the
advertised schema.
* fix(tools_call): make getChecks() idempotent
getChecks() pushed into an instance array on every call, so calling it
more than once accumulated duplicate check entries. The runner is
allowed to call getChecks() repeatedly, and other scenarios already
build their result fresh per call for this reason.
Return a freshly built array instead and drop the now-unused instance
field. Add a unit test covering the not-called FAILURE case and
asserting repeated getChecks() calls do not grow the result.
* fix: ensure dispatch and transport cleanup runs when handlers throw
The stateless tools/list and prompts/list handlers in the everything
server awaited dispatch.close() after the in-memory request, so a throw
from dispatch.client.request() leaked the Client+Server+Transport pair.
Wrap both in try/finally, matching the tools/call and catch-all
handlers in the same file.
In the stateful mock server, the res 'close' listener that tears down
the per-request SDK transport and server was registered after the
handleRequest await; a throw there skipped registration and leaked the
pair. Register the listener before handing the request to the
transport.
* fix(everything-client): send Accept header on stateless requests
The draft transport spec requires POST requests to accept both
application/json and text/event-stream. Servers built on the SDK's
StreamableHTTPServerTransport already enforce this and reject
requests without it. Does not yet make json-schema-ref-no-deref pass
under --spec-version draft: the SDK transport also rejects the draft
protocol version itself, which is the known fixture gap until the
2026-native scenarios migrate onto MockServer.
* chore: exclude .claude/ from vitest globs
Local agent worktree checkouts under .claude/worktrees/ carry their
own copies of the test suite, which the bare **/*.test.ts include was
adding to every run (and to the pre-push hook). Mirrors the existing
.prettierignore entry.
* fix(request-metadata): use spec error code -32004 for unsupported protocol version
The draft spec defines -32004 (UnsupportedProtocolVersionError) with
data { supported, requested } and HTTP 400 for protocol version
rejections; -32001 (HeaderMismatch) is reserved for HTTP header
validation failures.
- request-metadata scenario: return -32004 with spec-shaped data for
the simulated version-negotiation rejection (was -32001)
- everything-client: trigger negotiation on -32004 (was -32001)
- stateless mock server: include the requested version in the -32004
error data alongside the supported list
- update tests to match and assert the new requested field
* fix(auth): register transport cleanup before handleRequest in createServer
The res 'close' listener that tears down the transport/server pair was
registered after awaiting handleRequest, so a throw mid-request skipped
cleanup registration entirely. Register it first, matching the stateful
mock server.
* fix(everything-server): return JSON-RPC errors from stateless list dispatch on failure
The stateless tools/list and prompts/list dispatch handlers had
try/finally but no catch, so a dispatch failure produced an unhandled
rejection instead of a response. Add the same catch-and-respond block
the carry-forward dispatch handler already uses.
* refactor(runner): drop MCP_CONFORMANCE_LIFECYCLE; derive lifecycle from protocol version
The protocol version fully determines the lifecycle, and deriving it is
part of being a conformant client. The in-repo example client imports
the stateless version set from src/, so there is no drift risk, and a
single env var cannot disagree with itself. README documents the
version→lifecycle mapping for clients in other languages.
* fix(everything-server): return JSON-RPC errors from stateless resource list dispatch on failure
resources/list and resources/templates/list arrived from main with
try/finally but no catch, unlike the sibling tools/list and
prompts/list handlers. A throwing dispatch would escape to Express's
HTML 500 instead of a JSON-RPC error. Mirrors the existing pattern.
* fix(everything-server): include requested field in -32004 error data
The spec's UnsupportedProtocolVersionError requires data to carry both
supported and requested. The mock-server and request-metadata -32004
producers already include it; this was the remaining producer.
* feat(runner): client-side spec-version inference, skip, and --force
Mirrors the server runner's policy from #318: an explicitly requested
--spec-version outside a scenario's applicability window skips it
(exit 0) unless --force; when --spec-version is omitted the version is
inferred from the scenario's applicability instead of defaulting to
the latest dated version.
* fix(auth): gate SEP-837 application_type check to spec versions that include it
The sep-837-application-type-present check fired on every Dynamic Client
Registration regardless of the targeted spec version, but SEP-837 only
exists in the draft spec. Clients conforming to 2025-11-25 or earlier were
failed on a requirement that is not part of the version under test.
Thread the ScenarioContext into createAuthServer (mirroring createServer)
and only emit the check when the resolved spec version includes SEP-837,
i.e. the stateless draft lifecycle. At dated versions the check is not
emitted at all. Audited the other auth helper checks (PKCE, metadata
discovery, token verification); all of them enforce requirements that are
already in the dated specs, so SEP-837 was the only draft-only check.
Unit tests cover both sides: the check fails an omitting client at the
draft version and is absent at 2025-11-25.
* feat(sdk-runner): --spec-version passthrough with per-SDK default
The sdk command forwarded --scenario/--suite/--timeout to the underlying
client/server invocation but had no way to set the spec version, so SDK
runs always used the suite default. Add a --spec-version flag that is
passed through to the inner run.
Also add an optional specVersion field to the per-SDK config, used as the
default when the flag isn't given; an explicit --spec-version overrides
it. Set typescript-sdk-v1 to 2025-11-25: the v1 line targets the latest
dated spec, which keeps draft-only scenarios out of its runs by default.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* feat: vendor spec schema types per-version
Copies schema/{version}/schema.ts from the modelcontextprotocol spec
repo into src/spec-types/{version}.ts so the conformance suite can type
against draft spec versions before any SDK ships them.
npm run sync-schema -- <ref> refreshes the copies and records the spec
commit in src/spec-types/SOURCE.
* feat: add Connection abstraction and RunContext
Connection encapsulates how the conformance suite talks to a
server-under-test for a given spec version:
- connectStateful: 2025-x lifecycle. Thin adapter over the SDK Client
(initialize handshake, session id, SSE handled by the SDK).
- connectStateless: 2026-x lifecycle (SEP-2575). Raw fetch with
per-request _meta + MCP-Protocol-Version header. Decoupled from the
SDK so the suite can test draft spec versions before the SDK
implements them.
connectFor(specVersion) picks the implementation. RunContext bundles
serverUrl, specVersion and a bound connect() for the runner to hand to
each scenario.
Nothing uses this yet; wiring follows in the next commit.
* refactor: thread RunContext through ClientScenario.run
ClientScenario.run(serverUrl) becomes run(ctx: RunContext). The runner
builds the context from --spec-version and the server URL; scenarios
destructure ctx.serverUrl and otherwise behave identically.
No scenario uses ctx.connect() yet, so behaviour is unchanged: 214/214
tests pass, all-scenarios.test.ts still drives the everything-server
fixture exactly as before.
Test files use a testContext(url) helper to construct a RunContext.
The authorization-server scenario list is retyped to
ClientScenarioForAuthorizationServer since those scenarios test an
OAuth server, not an MCP server, and keep run(serverUrl).
* refactor: migrate server scenarios to ctx.connect() + conn.request()
22 carry-forward and lifecycle scenarios now go through the Connection
abstraction instead of connectToServer + SDK Client:
- tools.ts (8): list, call x6, with-progress
- prompts.ts (5)
- resources.ts (7): list, read x3, subscribe x2, not-found
- utils.ts (3): completion, ping, set-level
- json-schema-2020-12.ts, caching.ts, http-standard-headers.ts
Result types come from spec-types/{introducedIn}. Same scenario code
now passes under both --spec-version 2025-11-25 (SDK-backed stateful)
and --spec-version draft (raw stateless).
ToolsCallSampling/Elicitation/WithLogging and elicitation-* keep
connectToServer (need setRequestHandler/setLoggingLevel SDK surface);
they are tagged removedIn: DRAFT in the next commit. stateless.ts and
input-required-result.ts keep their sendRpc helper; migrating those is
deferred to the DRAFT-scenario coherence pass.
* fix: normalize Connection error to JsonRpcError; clean up RunContext import
connectStateful now catches the SDK's McpError and rethrows as
JsonRpcError so scenarios always see the same error class regardless
of which Connection impl ran. ResourcesNotFoundError uses
instanceof JsonRpcError instead of duck-typing.
types.ts uses a normal top-level import for RunContext instead of an
inline import() type.
* fix(dns-rebinding): use version-appropriate probe body
The 'valid Host accepted' check was sending an initialize body, which
a 2026 server returns 404 for. Probe body is now picked from
ctx.specVersion: initialize for the stateful lifecycle, server/discover
with _meta for the stateless lifecycle. The Host/Origin rejection check
is unchanged since rejection happens before body parsing.
* feat(everything-server): route stateless carry-forward methods to McpServer
Adds an in-memory dispatch client connected to the same McpServer the
stateful path uses. Stateless requests for tools/call, resources/*,
prompts/get and completion/complete that fall through the MRTR-specific
handlers are routed to it, so the fixture serves the carry-forward
scenarios under --spec-version draft without duplicating ~500 lines of
tool/resource/prompt registrations.
tools/list now merges the McpServer's tool list with the MRTR-only
stubs so json-schema-2020-12 finds its tool.
draft suite against the fixture: 36/39 (was 13/39). Remaining 3 are
fixture-side SEP gaps (no SSE forwarding for progress, no SEP-2549
ttlMs, no SEP-2243 Mcp-Method validation in the stateless path).
* refactor(connection): drop unused RequestOptions; move sdk-client; add tests
Addresses self-review findings on the new connection module:
- RequestOptions.handlers/.meta and ServerRequestHandler removed: zero
callers (the scenarios that motivated them are deferred). They can be
reintroduced when something actually uses them.
- scenarios/server/client-helper.ts moved to connection/sdk-client.ts so
the connection module no longer depends on the scenarios module.
- connectStateless: handle CRLF SSE event separators; throw a useful
error for non-JSON/non-SSE responses instead of a JSON parse error.
- JSONRPCNotification consistently imported from spec-types/2025-11-25.
- New connection.test.ts (8 tests) covering connectFor selection,
_meta injection, error mapping, SSE LF/CRLF parsing, and the
server-request-on-stream rejection.
* fix: address bughunt findings (response.ok check; targetVersion naming)
- connectStateless: throw on non-2xx responses that lack a JSON-RPC
error envelope (e.g. gateway/framework error JSON), matching the
stateful path's behavior. Previously such a response would return
undefined as the result.
- json-schema-2020-12: rename negotiatedVersion to targetVersion and
reword the skip message and details field. The value is ctx.specVersion
(the run's --spec-version), not a negotiated version; the old wording
was misleading. Drop dead 'unknown' fallback (specVersion is required)
and the corresponding undefined test cases.
* fix(sse-multiple-streams): keep scenario in draft; version-aware request scaffolding
Concurrent POSTs each answered with JSON or their own SSE stream are
core transport behavior in the draft spec too, so removedIn was wrong
for this scenario. Only the request scaffolding is version-specific:
stateful runs keep the initialize/session-id setup, draft runs send
_meta + the MCP-Protocol-Version header on each request instead. The
two checks are unchanged.
* refactor: fold stateless-client helpers into connection/stateless
The two stateless request paths were ~90% duplicated:
- scenarios/server/stateless-client.ts (main #319): buildStandardHeaders,
withRequestMeta, readSseJsonRpcResponse, sendStatelessRequest -> raw
{status, body}; sends Mcp-Method/Mcp-Name (SEP-2243); has timeout
- connection/stateless.ts (this PR): connectStateless -> Connection.request
-> throws JsonRpcError; missing Mcp-Method/Mcp-Name; no timeout
connection/stateless.ts now exports both layers:
- low-level sendStatelessRequest() (and the building blocks) moved verbatim
from stateless-client.ts
- connectStateless() rebuilt as a thin wrapper over sendStatelessRequest():
classifies SSE events into the notification sink, surfaces server->client
requests on the response stream as a spec violation, throws JsonRpcError
on error responses
So connectStateless() now also picks up the SEP-2243 headers and the 10s
timeout that the previous impl lacked.
stateless-client.ts deleted; its 4 importers + test file repointed at
'../../connection'. Test file moved to connection/stateless.test.ts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(everything-server): make stateless resources reach the carry-forward dispatch
The merge of main left two competing handler sets for resources/read,
resources/list and resources/templates/list: the inline stateless handlers
(test://stateless-static-text + a catch-all -32602) and the carry-forward
dispatch to the McpServer-registered resources. The catch-all made the
carry-forward unreachable, so test://static-text, test://static-binary and
the templates errored under --spec-version draft.
Now resources/list and resources/templates/list dispatch to the McpServer and
merge with the inline entries (same pattern as tools/list and prompts/list),
and resources/read keeps the inline static-text response but falls through to
the carry-forward for unknown URIs (preserving SEP-2164 data.uri on errors).
Also wrap the tools/list and prompts/list dispatch in try/finally so
dispatch.close() always runs, matching tools/call.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(scenarios): send SEP-2243 Mcp-Method header on hand-rolled stateless requests
sse-multiple-streams and dns-rebinding bypass sendStatelessRequest() because
they need raw fetch/undici for low-level inspection, but their hand-built
draft-path headers omitted Mcp-Method, so a strictly-conformant draft server
would 400 them before the behaviour under test runs. Build the headers via
buildStandardHeaders() instead, layering the scenario-specific overrides
(Accept ordering, Host/Origin) on top.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(runner): infer draft spec version for draft-only scenarios
When --spec-version is omitted, the runner defaulted to LATEST_SPEC_VERSION,
so draft-only scenarios (caching, resources-not-found, ...) silently ran over
the stateful connection instead of the stateless draft transport they were
written for; all-scenarios.test.ts had the same problem via testContext().
Infer the spec version from scenario.source.introducedIn (as negative.test.ts
already does) when no explicit version is given.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(connection): only throw JsonRpcError for properly-shaped error objects
connectStateless() destructured response.body.error unconditionally, so a
proxy returning {"error": "upstream timeout"} produced a JsonRpcError with
undefined code/message and lost the HTTP status. Require an object with a
numeric code before treating it as a JSON-RPC error; otherwise surface the
HTTP status and raw body in the thrown Error. Adds a unit test for the
non-conformant 502 case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(connection): align connectStateful normalization with connectStateless
Two consistency nits from automated review:
- Collected notifications were missing the jsonrpc field (the SDK's Zod
parsing strips it), violating the JSONRPCNotification type contract.
- McpError messages kept the SDK's "MCP error <code>: " prefix, so
JsonRpcError.message differed between the stateful and stateless impls.
Also restore progressToken filtering in tools-call-with-progress, which was
lost when the scenario migrated from the SDK onprogress callback to
conn.notifications: the server must echo the request's token back, and
notifications for other tokens are now ignored.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(runner): skip inapplicable scenario/spec-version combos; add --force; plumb spec version to the wire
Running a draft-only scenario with an explicit dated --spec-version (e.g.
server-stateless --spec-version 2025-11-25) used to silently pass: the flag
only affected ctx.connect(), which hand-rolling scenarios never call, so
they sent draft requests regardless and the run looked green while testing
nothing the flag claimed.
Three changes:
- buildStandardHeaders / withRequestMeta / sendStatelessRequest /
connectStateless accept an optional specVersion (default: draft), and
hand-rolling scenarios pass ctx.specVersion through, so requests declare
the version the run was invoked with.
- The runner skips (exit 0) when an explicit --spec-version falls outside
the scenario's introducedIn/removedIn window, with a message instead of
a misleading green run.
- A new --force flag overrides the skip for deliberate mismatch probing;
combined with the plumbing, a forced server-stateless run at 2025-11-25
now genuinely sends that version (8/20 checks pass against the
everything-server instead of a vacuous 26/26).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(everything-server): serve resources and caching hints on the stateless draft path
The stateless (SEP-2575) request path now answers resources/list,
resources/templates/list and resources/read — including the SEP-2164
-32602 + data.uri error for unknown URIs — and includes the SEP-2549
ttlMs/cacheScope hints on its cacheable list results, matching what the
draft-spec scenarios exercise.
* feat: make the harness's own draft-spec traffic conformant
Fixes the class of issues where a strictly-conformant SDK rejects the
harness's requests for reasons unrelated to the behaviour under test
(#311, #312, #315).
- add a shared stateless draft request helper (draft-client.ts) that puts
the cross-cutting SEP-2243 headers (Mcp-Method, Mcp-Name,
MCP-Protocol-Version, Accept) and SEP-2575 _meta (protocolVersion,
clientInfo, clientCapabilities) on every request, with overrides for
negative cases and the SHOULD-level retry on
UnsupportedProtocolVersionError
- migrate the draft scenarios onto it: the SEP-2322 helpers and
server-stateless now send Mcp-Method/Mcp-Name (#312); the SEP-2243
server scenarios drop the legacy initialize handshake and carry _meta on
every raw request (#311); caching and sep-2164-resource-not-found are
stateless DRAFT-2026-v1 requests instead of an SDK 2025-11-25 session
(#315)
- add a draft self-conformance gate (draft-self-conformance.test.ts) that
points the harness's own draft drivers at the existing client-testing
judge scenarios (request-metadata, http-standard-headers) and requires
zero failures or warnings
- align the request-metadata mock's unsupported-version rejection with the
draft schema (-32004) and teach the everything-client to recognize it
- rewrite the SEP-2549/SEP-2164 negative example servers as stateless
draft servers so they keep exercising the violation, not the transport
* refactor: rename the stateless request helpers and harden them per review
- rename draft-client.ts to stateless-client.ts and name the helpers by
feature (sendStatelessRequest, buildStandardHeaders, buildRequestMeta,
withRequestMeta, CONFORMANCE_CLIENT_INFO); 'draft' now only appears in
DRAFT_PROTOCOL_VERSION, the one name that tracks the moving target
- retry only genuine UnsupportedProtocolVersionError rejections (-32004,
tolerating -32001/-32602) and only with recognized versions; record the
retry on the response
- parse text/event-stream responses incrementally and resolve on the
matching event instead of blocking until the timeout
- keep the MCP-Protocol-Version header and _meta protocolVersion in sync
when the version is overridden via meta overrides
- make server-stateless's sendRpc SSE-tolerant
- emit explicit setup failures in the SEP-2243 server scenarios when the
tools/list discovery fails instead of misleading per-case results
- latch the worst status per check in the request-metadata judge so an
early violation cannot be masked by a later conformant request
- rename the gate to harness-traffic-conformance.test.ts and derive its
driver list from the draft suite with documented per-judge exclusions
(35 pairings)
- everything-server: advertise resources in stateless server/discover and
use -32004 for unsupported protocol version
- add unit tests for the helper (retry guard, SSE handling, header/_meta
version sync, default headers)
* refactor: slim the stateless client helper for the initial merge
* feat: split draft-spec scenarios into their own suite
- add --suite draft to the server command, selecting scenarios whose
source.introducedIn is the draft protocol version
- exclude draft scenarios from the default active suite until the draft
spec is published as a dated release
- derive the client command's draft suite from source.introducedIn so the
non-auth draft scenarios (SEP-2243/2575/2322/2106) are included alongside
the auth ones
- keep everything-server self-test coverage for draft scenarios in
all-scenarios.test.ts (active + non-pending draft)
- document available suites in the README and add suite-membership tests
* fix: repair broken specReferences URLs
- SEP-986: SEP/SEP-986.md no longer exists in the spec repo; point at the
Tool Names section of the published 2025-11-25 tools page instead
- SEP-990: the ext-auth spec file is enterprise-managed-authorization.mdx,
not enterprise-oauth.mdx
Fixes#313
* chore: refresh SEP traceability manifest after #303 and #304
Regenerated from a full client+server suite run against
typescript-sdk@258f1a04a5d3 on current main:
- sep-2468: the two new client checks are emitted (tested) and the two
authorization-server rows move to excluded (#304)
- sep-2243: 12 rows flip to tested now that the custom-header check IDs
match their requirements (#303)
* chore: regenerate the manifest on top of SEP-2106 (#295)
Re-ran both suites after rebasing onto main so the manifest also picks
up the new sep-2106 block (1 tested, 4 excluded, 3 untracked check IDs
emitted by the server scenario with no yaml row yet).
* chore: add SEP-2106 requirement-traceability YAML (tools JSON Schema 2020-12)
* Drop structured-content text-fallback row from SEP-2106
That SHOULD predates SEP-2106 (pre-existing context line in tools.mdx); 2106
only relaxes structuredContent to allow non-object values (permissive). Point
spec_url at the SEP doc security section where the remaining obligations live.
* feat: SEP-2106 JSON Schema 2020-12 vocabulary preservation checks
Extend the json-schema-2020-12 server scenario (previously SEP-1613 only) with
three checks that verify the broader JSON Schema 2020-12 vocabulary SEP-2106
permits in a tool inputSchema survives tools/list rather than being stripped:
composition (allOf/anyOf), conditional (if/then/else), and reference ($anchor)
keywords.
- everything-server: add these keywords to json_schema_2020_12_tool's raw schema
- add a negative-test server (sep-2106-stripped-schema.ts) plus a negative test
asserting the checks emit FAILURE when the vocabulary is stripped
* feat: SEP-2106 client scenario for network $ref dereferencing
Add json-schema-ref-no-deref, a client-suite scenario that advertises a tool
whose inputSchema contains a $ref pointing at a canary URL hosted by the
scenario. The sep-2106-no-network-ref-deref check fails if the client fetches
the canary while processing the schema, covering the SEP-2106 requirement that
implementations MUST NOT automatically dereference network-URI $ref values.
- register the scenario in the client scenarios list
- everything-client: handle the scenario with the basic connect+listTools flow
- tests: compliant client passes, a deliberately dereferencing client is
caught, and a client that never lists tools cannot pass
* Address review findings in SEP-2106 scenarios
- json-schema-ref-deref: make getChecks() idempotent (fresh array per call,
matching the http-standard-headers convention) and guard the async /mcp
handler like the example servers do
- json-schema-2020-12: include the SEP-2106 keywords ($anchor, allOf/anyOf,
if/then/else) in the description's example inputSchema so a server that
copies it verbatim passes the new checks
- sep-2106-stripped-schema: default to port 3007 to match negative.test.ts and
avoid sharing a default with sep-2549-no-caching-hints
* Soft-gate SEP-2106 keyword checks on the negotiated protocol version
The json-schema-2020-12 scenario is introducedIn 2025-11-25 (SEP-1613) but the
three SEP-2106 keyword-preservation checks test draft-era behavior. Rather than
failing servers that only negotiated 2025-11-25, gate the failure on the wire
protocolVersion from the initialize handshake: preserved keywords are SUCCESS at
any version, stripped keywords are FAILURE only when the server negotiated the
draft version and SKIPPED otherwise.
The published SDK does not advertise the draft version yet, so the negative test
asserts SKIPPED for the stripped-schema server; the FAILURE branch is covered by
unit tests on sep2106KeywordCheckStatus.
Add two client scenarios for the SEP-2468 requirements that had no
conformance coverage:
- auth/metadata-issuer-mismatch: the AS metadata document declares an
issuer that differs from the issuer identifier used to construct the
well-known URL; the client must not use the metadata
(sep-2468-client-validate-metadata-issuer).
- auth/iss-normalized: the authorization redirect carries an iss value
that only matches the recorded issuer after URL normalization; the
client must compare with simple string comparison and reject
(sep-2468-client-no-normalization).
The two authorization-server-side rows (AS SHOULD include iss, AS MUST
advertise authorization_response_iss_parameter_supported) are excluded:
observing iss in an authorization response requires driving an
Authorization Code Grant against the AS under test, which the
authorization-server suite does not implement yet (#208).
The x-mcp-header / Mcp-Param / Base64 half of SEP-2243 was already tested,
but the scenarios emitted one check ID per test case while sep-2243.yaml
declares one per normative requirement, so the traceability join (exact ID
match) reported 14 of 20 requirements as untested. #287 collapsed the
standard-header IDs but left the custom-header ones per-case.
This finishes the rename and adds DECLARED_CHECK_IDS backfill (the
request-metadata.ts pattern) so every requirement-level ID is emitted on
every run — exercised, backfilled as FAILURE, or SKIPPED when the server
under test exposes no x-mcp-header tool. SEP-2243 traceability goes from
6/20 to 18/20 tested; the remaining 2 (server-not-expect-null,
server-reject-missing-required) have no test case yet.
Client scenarios:
- param-header-{name} -> client-mirrors-designated-params /
client-encode-values / client-base64-unsafe, picked by what the
parameter exercises
- no-mirror-unannotated -> client-mirrors-designated-params
- param-header-tool-call-gate -> client-supports-custom-headers, now also
emitted SUCCESS when the annotated tool is called with Mcp-Param headers
- the invalid-tool loop emits the specific x-mcp-header constraint each
tool violates (not-empty / charset / unique / primitive-only)
Server scenarios:
- accepts-valid-base64 -> server-decode-base64
- rejects-invalid-base64-{padding,chars} -> server-reject-invalid-param-chars
- literal-missing-base64-{prefix,suffix} -> server-validate-param-match
- rejects-missing-custom-header -> server-validate-param-match
- every custom-header rejection's -32001 check -> server-reject-param-mismatch
- standard-header reject cases pass the coarse ID through the catch path too
'All scopes required for the current operation' has no harness-observable
ground truth: the WWW-Authenticate challenge is the only place the server
declares what an operation requires, so there is nothing independent to
compare it against. Detecting the negative (incremental challenging) would
need server-side auth scenarios that do not exist yet and would exercise the
example app's scope configuration rather than SDK behavior, and the spec
explicitly permits dynamic per-request scope determination, so a second
challenge is not conclusively non-conformant.
The client-side half of SEP-2350 (scope union on re-authorization) remains
tested by auth/scope-step-up.
* chore: refresh SEP traceability manifest (typescript-sdk@main)
Regenerated from a client+server suite run against typescript-sdk@5fc42e9be115
following the recipe in .github/workflows/traceability.yml.
New entries since the last refresh (typescript-sdk@22595b96):
- SEP-2322 (MRTR, #188): 17 tested, 0 untested, 16 excluded, 3 untracked
- SEP-2549 (TTL for list results, #275): 7 tested, 0 untested, 13 excluded
- SEP-2260: 12 excluded rows, no checks
- SEP-2207: yaml rows added since the last refresh now appear
(1 tested, 1 untested: sep-2207-server-no-offline-access)
No previously-tested requirement regressed.
* Exclude sep-2207 server offline_access guidance until RS auth scenarios exist
sep-2207-server-no-offline-access was declared in the yaml but no scenario
emits it, so it surfaced as the only untested requirement in the refreshed
manifest. The check needs to probe the SDK server's Protected Resource
Metadata scopes_supported and WWW-Authenticate challenge scope, and the
server suite does not yet exercise the SDK server as an OAuth protected
resource at all.
Mark the requirement excluded with a pointer to #116 (server-side
authorization baseline) rather than leaving it as a permanently-untested
row; revisit when server-side authorization scenarios land.
* chore: additional server conformance tests for SEP-2575
* fix(sep-2575): use the spec's notifications filter object in subscriptions/listen
The scenario sent params.subscriptions: [{type: 'tools/list-changed'}], a
shape that does not exist in the SEP. SubscriptionsListenRequestParams
requires params.notifications: {toolsListChanged?, promptsListChanged?, ...}
(see schema.ts SubscriptionFilter and the canonical
SubscriptionsListenRequest example). A compliant server would either reject
the old request as invalid params or treat it as subscribing to nothing,
failing the ack/list-changed checks.
Also makes the everything-server's acknowledgment echo the agreed
notifications filter, as the SubscriptionsAcknowledgedNotification schema
requires.
* fix(sep-2575): match the spec's notifications/*/list_changed method names
The list-changed checks and the everything-server used
notifications/tools/list-changed (hyphen); the spec's method is
notifications/tools/list_changed (underscore) per ToolListChangedNotification
/ PromptListChangedNotification in schema.ts. With the hyphen form the
notification-filter check cannot catch a real leak, and the list-changed
checks fail a server that emits the correct name.
Also drops the f.params.toolsListChanged === true fallback match: that field
is the subscriptions/listen request filter, not a notification parameter, so
it never appears on a compliant server's notification.
* fix(sep-2575): keep collected frames when the listen stream read times out
listenToStream aborts the fetch after timeoutMs. If the abort fired while
blocked in reader.read(), the rejection propagated to the outer catch and the
helper returned [] — discarding every frame already received. A compliant
server holds a subscriptions/listen stream open indefinitely, so the timeout
is the *normal* way these reads end; previously every subscription check
would report a false 'no frames received' FAILURE against such a server.
The current everything-server only avoided this by res.end()ing immediately
after the ack.
* fix(sep-2575): open the listen stream before triggering list mutations
The notification-filter and list-changed checks fired the mutation trigger
*before* opening the subscriptions/listen stream. A compliant server only
delivers list-changed notifications to streams that are open at the time of
the change, so the SHOULD checks would fail any server that does not replay
past changes to new subscribers, and the filter check could never observe a
real leak.
listenToStream now takes an onFirstFrame hook; the three trigger-based
checks open the stream, wait for the acknowledgment, then fire the mutation
on a separate connection and keep reading.
The everything-server now keeps subscriptions/listen streams open, registers
them with their filters, and fans the list_changed notification out to open
streams when a trigger tool runs - instead of unconditionally emitting a
list-changed frame on every stream open (which itself violated the
notification-filter requirement in spirit: it announced a change when
nothing had changed).
* fix(sep-2575): check the subscription id on every listen-stream notification
server-tags-subscription-id only inspected the acknowledgment frame, so a
server that tags the ack but omits io.modelcontextprotocol/subscriptionId
from subsequent notifications passed. The check now triggers a tool-list
change once the stream is acknowledged and asserts every notification frame
on the stream carries the id in params._meta.
Also drops the f.body.method / f.params.method fallbacks when resolving a
frame's method - those shapes don't exist in JSON-RPC and only masked
malformed frames.
* fix(sep-2575): report WARNING, not FAILURE, for the SHOULD-level list-changed checks
The spec text behind server-sends-{tools,prompts}-list-changed-on-subscription
is a SHOULD ('servers that declared the listChanged capability SHOULD send a
notification...'). Severity follows the spec keyword: MUST -> FAILURE,
SHOULD -> WARNING. runCheck now accepts a warning flag and the two
list-changed checks use it on their failure paths.
* fix(sep-2575): skip stream checks when the diagnostic tool is not exposed
http-server-no-independent-requests-on-stream and
server-no-log-without-loglevel reported SUCCESS when the server rejected the
tools/call outright (e.g. test_streaming_elicitation / test_logging_tool do
not exist): a single error frame contains no independent request and no log
notification, so the check passed without exercising anything. The sibling
list-changed checks already SKIP when their trigger tool is missing; these
now do the same.
* chore(sep-2575): exclude the four unobservable server requirements from traceability
Marks the section-C rows of #296 as excluded rather than untested:
no-prior-context, no-connection-reuse-required, disconnect-is-cancel and
stops-on-cancel describe internal server state that a black-box harness
cannot observe on the wire. With these excluded and the part-B checks landed,
every testable server-side SEP-2575 requirement is covered; the remaining
untested rows are all client-side and blocked on a SEP-2575-aware reference
client.
* fix(sep-2575): emit the request-metadata checks even when the client never connects
The reference SDK conformance client has no handler registered for the
request-metadata scenario, so it exits without sending a request, the
scenario emits zero checks, the suite reports "0 passed, 0 failed", and the
traceability manifest records all seven client-side requirements as
untested.
Declare the scenario check IDs up front and backfill any that were never
emitted as FAILURE from getChecks(): the emitted ID set is now the same for
every client, and a client that never connects fails loudly. A check that
is legitimately not applicable opts out by being emitted as SKIPPED
explicitly (version-header-matches-meta now does this when either version
field is absent). The positive-path test pins the declared list against the
reference client in both directions.
Also refresh the traceability manifest from a run against
typescript-sdk@22595b96: the seven client rows flip to tested and the
manifest catches up with the server-side scenarios from #297.
---------
Co-authored-by: Yuan Teoh <yuanteoh@google.com>
* chore: additional server conformance tests for SEP-2575
* fix(sep-2575): use the spec's notifications filter object in subscriptions/listen
The scenario sent params.subscriptions: [{type: 'tools/list-changed'}], a
shape that does not exist in the SEP. SubscriptionsListenRequestParams
requires params.notifications: {toolsListChanged?, promptsListChanged?, ...}
(see schema.ts SubscriptionFilter and the canonical
SubscriptionsListenRequest example). A compliant server would either reject
the old request as invalid params or treat it as subscribing to nothing,
failing the ack/list-changed checks.
Also makes the everything-server's acknowledgment echo the agreed
notifications filter, as the SubscriptionsAcknowledgedNotification schema
requires.
* fix(sep-2575): match the spec's notifications/*/list_changed method names
The list-changed checks and the everything-server used
notifications/tools/list-changed (hyphen); the spec's method is
notifications/tools/list_changed (underscore) per ToolListChangedNotification
/ PromptListChangedNotification in schema.ts. With the hyphen form the
notification-filter check cannot catch a real leak, and the list-changed
checks fail a server that emits the correct name.
Also drops the f.params.toolsListChanged === true fallback match: that field
is the subscriptions/listen request filter, not a notification parameter, so
it never appears on a compliant server's notification.
* fix(sep-2575): keep collected frames when the listen stream read times out
listenToStream aborts the fetch after timeoutMs. If the abort fired while
blocked in reader.read(), the rejection propagated to the outer catch and the
helper returned [] — discarding every frame already received. A compliant
server holds a subscriptions/listen stream open indefinitely, so the timeout
is the *normal* way these reads end; previously every subscription check
would report a false 'no frames received' FAILURE against such a server.
The current everything-server only avoided this by res.end()ing immediately
after the ack.
* fix(sep-2575): open the listen stream before triggering list mutations
The notification-filter and list-changed checks fired the mutation trigger
*before* opening the subscriptions/listen stream. A compliant server only
delivers list-changed notifications to streams that are open at the time of
the change, so the SHOULD checks would fail any server that does not replay
past changes to new subscribers, and the filter check could never observe a
real leak.
listenToStream now takes an onFirstFrame hook; the three trigger-based
checks open the stream, wait for the acknowledgment, then fire the mutation
on a separate connection and keep reading.
The everything-server now keeps subscriptions/listen streams open, registers
them with their filters, and fans the list_changed notification out to open
streams when a trigger tool runs - instead of unconditionally emitting a
list-changed frame on every stream open (which itself violated the
notification-filter requirement in spirit: it announced a change when
nothing had changed).
* fix(sep-2575): check the subscription id on every listen-stream notification
server-tags-subscription-id only inspected the acknowledgment frame, so a
server that tags the ack but omits io.modelcontextprotocol/subscriptionId
from subsequent notifications passed. The check now triggers a tool-list
change once the stream is acknowledged and asserts every notification frame
on the stream carries the id in params._meta.
Also drops the f.body.method / f.params.method fallbacks when resolving a
frame's method - those shapes don't exist in JSON-RPC and only masked
malformed frames.
* fix(sep-2575): report WARNING, not FAILURE, for the SHOULD-level list-changed checks
The spec text behind server-sends-{tools,prompts}-list-changed-on-subscription
is a SHOULD ('servers that declared the listChanged capability SHOULD send a
notification...'). Severity follows the spec keyword: MUST -> FAILURE,
SHOULD -> WARNING. runCheck now accepts a warning flag and the two
list-changed checks use it on their failure paths.
* fix(sep-2575): skip stream checks when the diagnostic tool is not exposed
http-server-no-independent-requests-on-stream and
server-no-log-without-loglevel reported SUCCESS when the server rejected the
tools/call outright (e.g. test_streaming_elicitation / test_logging_tool do
not exist): a single error frame contains no independent request and no log
notification, so the check passed without exercising anything. The sibling
list-changed checks already SKIP when their trigger tool is missing; these
now do the same.
* chore(sep-2575): exclude the four unobservable server requirements from traceability
Marks the section-C rows of #296 as excluded rather than untested:
no-prior-context, no-connection-reuse-required, disconnect-is-cancel and
stops-on-cancel describe internal server state that a black-box harness
cannot observe on the wire. With these excluded and the part-B checks landed,
every testable server-side SEP-2575 requirement is covered; the remaining
untested rows are all client-side and blocked on a SEP-2575-aware reference
client.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* Add SEP-2350 scope-accumulation check to auth/scope-step-up
SEP-2350 clarifies that on step-up re-authorization, clients SHOULD
compute the union of previously-granted and newly-challenged scopes so
they don't lose permissions for other operations.
- Traceability yaml: 1 client check (two spec sentences merged), 1 server
check tracked for a future server-side scenario, 1 excluded reword
- ScopeStepUpAuthScenario now challenges with only the missing scope so
union accumulation is observable on the second authorization request
- New check sep-2350-scope-union-on-reauth (WARNING when the previously
granted scope is dropped)
- everything-client withOAuthRetry now accumulates prior token scope into
the re-auth scope (passing example)
- New auth-test-echo-scope negative client + vitest case
* feat: add client conformance tests for SEP-2575 (#270)
* feat: merge latest request-metadata scenario stacked branch
* test: implement optional capability conditional checks and simulated negotiation retry
* fix(sep-2243): collapse check IDs to one-per-requirement (#287)
The sep-2243 scenario check IDs drifted from the requirement-traceability
yaml in #259 — the code emitted one ID per test case while the yaml
declares one per normative requirement, so the IDs no longer matched.
Rename the emitted IDs so a check ID maps to a single MUST/SHOULD and is
emitted once per case (the per-case detail moves to name/description),
matching the repo's existing 'same id, vary status/message' convention:
- mcp-method-header-* / mcp-name-header-* -> client-includes-standard-headers
- reject-invalid-tool-* / keep-valid-tool -> client-reject-invalid-tool
- server-accepts-{lower,upper}case-name -> header-name-case-insensitive
- server reject status checks (mismatch/missing/case x method/name)
-> server-reject-invalid-headers
- their error-code variants -> server-reject-error-code
Merge the yaml's server-reject-mismatch + server-reject-status into one
server-reject-invalid-headers MUST (HTTP 400 on a header-validation
failure); keep server-reject-error-code as the SHOULD. Base64/custom-header
rejection checks keep their own param-validation IDs (out of scope here).
Gates and the whitespace-acceptance check keep their scenario-matching
names. No behavior change — only check IDs, descriptions, and the yaml.
* feat: `sdk` subcommand to run local conformance against any SDK ref (#277)
* feat: add sdk subcommand to run conformance against any SDK ref (#250)
* Revise sdk runner: explicit --mode, KNOWN_SDKS-only config, v1/v2 entries
Addresses review feedback on the sdk subcommand:
- Require --mode (client|server) and remove "both". Each invocation now
tests exactly one side with its own exit code; the old default ran
client then server but combined exit codes with ||=, which skipped the
server side entirely whenever the client run failed.
- Resolve build/run config from KNOWN_SDKS + CLI flags only; drop the
conformance.config.yaml loader (no SDK ships one yet). The Zod schema
stays as the type for the built-in entries.
- Split the typescript entry by major version: typescript-sdk (v2/main,
pnpm install + build:all, expected-failures.yaml) and typescript-sdk-v1
(v1.x, npm ci + build, conformance-baseline.yml). An entry may set
`repo` (real clone target for an alias) and `defaultRef` (branch used
when no @ref is given). parseSdkSpec now leaves ref undefined when
omitted so defaultRef can apply; a trailing @ is treated as no ref.
- Key the clone cache by ref (<repo>/<ref>) so different refs of the same
repo no longer share one checkout.
- Bound the server readiness probe with a per-request AbortSignal timeout
so a server that accepts the socket but never responds can't hang past
the overall deadline.
* fix(sdk-runner): resolve -o to absolute path; add --expected-failures override; replaceAll for safeName
* feat: add SEP conformance traceability manifest (#288)
squashed; see PR #288 body
* feat: add server conformance tests for SEP-2575 (#271)
* feat: add server conformance tests for SEP-2575
* refactor and add new checks
* Fix stateless scenario source field and report SKIPPED for inapplicable capability checks
- Add the required `source` field (DRAFT) so the scenario typechecks and is
selectable via --spec-version. Without it the class did not satisfy the
ClientScenario interface and s.source was dereferenced by the spec-version
filter at list time.
- Teach runCheck about a SKIPPED status and use it for the two client-capability
checks when the server does not return -32003, instead of reporting a green
PASS for a requirement that was never exercised.
* Remove unimplemented placeholder checks from stateless server scenario
The subscriptions/listen, statelessness-invariant, list-changed and
disconnect-is-cancel checks emitted SUCCESS without probing the server,
which inflated coverage in the traceability report. Drop them until they
have real assertions; the corresponding rows remain declared in
src/seps/sep-2575.yaml so traceability shows them as not-yet-covered.
Also fix the checkErrorId helper to push under the
sep-2575-http-server-error-jsonrpc-id slug so its failure path actually
short-circuits the aggregate SUCCESS check at the end of the scenario.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* SEP-2352: authorization-server migration scenario (#286)
* Add SEP-2352 authorization-server migration scenario
SEP-2352 requires that client credentials are bound to the issuing
authorization server: when PRM authorization_servers changes to a new
issuer, clients MUST re-register and MUST NOT reuse the previous AS's
client credentials.
- Traceability yaml: 3 checks, 3 excluded (internal state / UI)
- New auth/authorization-server-migration scenario (draft suite): two
auth servers; PRM flips from AS1 to AS2 after the first authenticated
request; AS2 asserts it received a fresh /register and never saw AS1's
client_id at /authorize or /token
- ConformanceOAuthProvider gains invalidateCredentials and bindIssuer so
the everything-client can key credentials by issuer (passing example)
- everything-client adds an issuer-aware handler for this scenario that
re-reads PRM on each 401 and rebinds before re-authorizing
- auth-test-reuse-credentials negative client + vitest case
* Drop application_type from DCR metadata (not in SDK OAuthClientMetadata type)
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* ci(traceability): enable corepack so the SDK build can use pnpm (#289)
The refresh run failed with 'pnpm: not found' — the reference SDK's
build command (typescript-sdk: pnpm install && pnpm run build:all) needs
pnpm on PATH. Add 'corepack enable' to the run job.
* ci(traceability): tolerate SDK conformance failures in the run step (#290)
The `sdk` command exits non-zero when the SDK has conformance failures
not in its baseline. The traceability manifest only needs the emitted
check IDs (written regardless of pass/fail), so a failing SDK must not
fail the refresh. Add `|| true`; the existing 'no results produced'
guard catches a genuinely broken run.
* chore: refresh SEP traceability manifest (typescript-sdk@main) (#291)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Make scopesSupported disjoint from challenge scopes so the SEP-2350 union check can't be satisfied by scopes_supported ∪ challenge
A client that unions scopes_supported with the challenge — instead of its
prior grant with the challenge — would have requested mcp:basic mcp:write
and falsely passed sep-2350-scope-union-on-reauth, since scopesSupported
happened to coincide with the previously-granted scope. Setting it to a
disjoint value makes the check actually require the prior token's scope.
Spec backing: clients MUST NOT assume any set relationship between the
challenged scope set and scopes_supported, so a disjoint advertisement is
realistic.
* SEP-837: application_type check in DCR registration (#284)
* Add SEP-837 application_type check to DCR registration
SEP-837 requires MCP clients to specify an appropriate application_type
during Dynamic Client Registration so OIDC authorization servers can
apply the correct redirect-URI constraints.
- Traceability yaml: 1 check (presence + valid value), 4 excluded
(class-specific SHOULDs unobservable; UI/robustness)
- Check added in the shared createAuthServer DCR handler so it fires in
every auth scenario that performs DCR (no new scenario)
- withOAuthRetry now sets application_type: native (passing example;
the conformance example clients are CLI tools)
- New auth-test-no-application-type negative client + vitest case
* Widen ConformanceOAuthProvider metadata type for application_type
Fixes CI typecheck (TS2353 at withOAuthRetry.ts:76): the SDK's
OAuthClientMetadataSchema doesn't include application_type yet.
registerClient() spreads clientMetadata verbatim into the /register
POST body, so a local type intersection is sufficient to get the
field on the wire without an SDK release.
* Set application_type in runAuthMigrationClient
The SEP-2352 authorization-server-migration handler constructs its own
ConformanceOAuthProvider (not via withOAuthRetry), so it was missing
application_type after rebasing onto main. The scenario does DCR twice
(old AS, new AS) and the SEP-837 check fires on each.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* feat: add conformance tests for iss parameter (SEP-2468) (#220)
* feat: add conformance tests for iss parameter (SEP-2468)
Adds 5 draft conformance scenarios testing RFC 9207 issuer parameter
validation in OAuth authorization responses:
- auth/iss-supported: server advertises support and sends correct iss
- auth/iss-not-advertised: server omits iss parameter entirely
- auth/iss-supported-missing: client must reject missing iss when required
- auth/iss-wrong-issuer: client must reject mismatched iss value
- auth/iss-unexpected: client must reject iss when not advertised
Also adds auth-test-iss-validation.ts, a reference client that correctly
validates iss per RFC 9207, and negative tests confirming the standard
client fails all three rejection scenarios.
TODO: Update RFC_9207_ISS_PARAMETER spec reference once SEP-2468
(modelcontextprotocol/modelcontextprotocol#2468) is merged.
* update scenarios
* fix: createAuthServer iss option type/guard and NotAdvertised scenario duplication
The doc comments said 'Default: not included' but the destructure defaulted
to true/'correct', and the `!== undefined` guard at L155 was unreachable —
so there was no way to omit the metadata field, and IssParameterNotAdvertised
silently advertised support (a duplicate of IssParameterSupported).
Kept the on-by-default behavior (mock AS models a well-behaved server) but
made issParameterSupported `boolean | null` so callers pass null to omit,
matching the codeChallengeMethodsSupported pattern. Doc comments now match.
Scenarios that need omission pass null/'omit' explicitly.
* fix: rejection scenarios silently pass when client never reaches auth endpoint
correctlyRejected = !tokenRequestMade reports SUCCESS if the client errors
out before hitting /authorize. Gate on authReached so a setup failure shows
as FAILURE with authReached:false in details.
* fix: iss-unexpected scenario contradicts SEP-2468 spec table row 3
The spec table says: supported=false/absent + iss present -> *Compare* to
the recorded issuer (not reject). The scenario sent a *correct* iss and
FAILed compliant clients for proceeding after a successful comparison.
Now sends a mismatched iss so the comparison fails and rejection is the
spec-required outcome. Reference client updated to compare-when-present
instead of throw-on-presence.
* refactor: replace harness-config checks with client-proceeded checks
iss-advertised-in-metadata / iss-sent-in-redirect (and the not-* variants)
fired in onAuthorizationRequest before the redirect happened, asserting only
that the harness was configured correctly — a client that ignores iss passes
identically. Replaced with one check per scenario keyed on tokenRequestMade,
which observes that the client actually proceeded through the iss path.
* refactor: rename check IDs to sep-2468-* and align with spec table rows
One ID per spec table row; auth/iss-supported and auth/iss-wrong-issuer
both emit sep-2468-client-compare-iss-supported (same comparison, opposite
input) per the same-slug-for-SUCCESS-and-FAIL convention.
* feat: add sep-2468.yaml requirement traceability
8 check rows (4 client table-row checks, 1 metadata-issuer, 2 AS-side,
1 no-normalization), 1 excluded (error-display is UI-facing). The
record-issuer MUST is merged into the compare-iss-supported row text
since it has no independent wire observation.
* fix: migrate iss scenarios specVersions->source (post-#265)
Replaces `specVersions: ['draft']` with `source: { introducedIn: DRAFT_PROTOCOL_VERSION }`
in the 5 iss-parameter scenarios.
This commit typechecks once the stack is rebased onto main >= #265 (the
ScenarioSource migration). Adding it now so the rebase is mechanical.
* fix: include application_type in iss-validation example DCR (post-#284)
The SEP-837 application_type check now runs in every auth scenario; the
hand-rolled DCR in auth-test-iss-validation.ts was omitting the field.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
---------
Co-authored-by: Anubhav Dhawan <anubhavdhawan@google.com>
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Max Gerber <89937743+max-stytch@users.noreply.github.com>
* feat: add conformance tests for iss parameter (SEP-2468)
Adds 5 draft conformance scenarios testing RFC 9207 issuer parameter
validation in OAuth authorization responses:
- auth/iss-supported: server advertises support and sends correct iss
- auth/iss-not-advertised: server omits iss parameter entirely
- auth/iss-supported-missing: client must reject missing iss when required
- auth/iss-wrong-issuer: client must reject mismatched iss value
- auth/iss-unexpected: client must reject iss when not advertised
Also adds auth-test-iss-validation.ts, a reference client that correctly
validates iss per RFC 9207, and negative tests confirming the standard
client fails all three rejection scenarios.
TODO: Update RFC_9207_ISS_PARAMETER spec reference once SEP-2468
(modelcontextprotocol/modelcontextprotocol#2468) is merged.
* update scenarios
* fix: createAuthServer iss option type/guard and NotAdvertised scenario duplication
The doc comments said 'Default: not included' but the destructure defaulted
to true/'correct', and the `!== undefined` guard at L155 was unreachable —
so there was no way to omit the metadata field, and IssParameterNotAdvertised
silently advertised support (a duplicate of IssParameterSupported).
Kept the on-by-default behavior (mock AS models a well-behaved server) but
made issParameterSupported `boolean | null` so callers pass null to omit,
matching the codeChallengeMethodsSupported pattern. Doc comments now match.
Scenarios that need omission pass null/'omit' explicitly.
* fix: rejection scenarios silently pass when client never reaches auth endpoint
correctlyRejected = !tokenRequestMade reports SUCCESS if the client errors
out before hitting /authorize. Gate on authReached so a setup failure shows
as FAILURE with authReached:false in details.
* fix: iss-unexpected scenario contradicts SEP-2468 spec table row 3
The spec table says: supported=false/absent + iss present -> *Compare* to
the recorded issuer (not reject). The scenario sent a *correct* iss and
FAILed compliant clients for proceeding after a successful comparison.
Now sends a mismatched iss so the comparison fails and rejection is the
spec-required outcome. Reference client updated to compare-when-present
instead of throw-on-presence.
* refactor: replace harness-config checks with client-proceeded checks
iss-advertised-in-metadata / iss-sent-in-redirect (and the not-* variants)
fired in onAuthorizationRequest before the redirect happened, asserting only
that the harness was configured correctly — a client that ignores iss passes
identically. Replaced with one check per scenario keyed on tokenRequestMade,
which observes that the client actually proceeded through the iss path.
* refactor: rename check IDs to sep-2468-* and align with spec table rows
One ID per spec table row; auth/iss-supported and auth/iss-wrong-issuer
both emit sep-2468-client-compare-iss-supported (same comparison, opposite
input) per the same-slug-for-SUCCESS-and-FAIL convention.
* feat: add sep-2468.yaml requirement traceability
8 check rows (4 client table-row checks, 1 metadata-issuer, 2 AS-side,
1 no-normalization), 1 excluded (error-display is UI-facing). The
record-issuer MUST is merged into the compare-iss-supported row text
since it has no independent wire observation.
* fix: migrate iss scenarios specVersions->source (post-#265)
Replaces `specVersions: ['draft']` with `source: { introducedIn: DRAFT_PROTOCOL_VERSION }`
in the 5 iss-parameter scenarios.
This commit typechecks once the stack is rebased onto main >= #265 (the
ScenarioSource migration). Adding it now so the rebase is mechanical.
* fix: include application_type in iss-validation example DCR (post-#284)
The SEP-837 application_type check now runs in every auth scenario; the
hand-rolled DCR in auth-test-iss-validation.ts was omitting the field.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* Add SEP-837 application_type check to DCR registration
SEP-837 requires MCP clients to specify an appropriate application_type
during Dynamic Client Registration so OIDC authorization servers can
apply the correct redirect-URI constraints.
- Traceability yaml: 1 check (presence + valid value), 4 excluded
(class-specific SHOULDs unobservable; UI/robustness)
- Check added in the shared createAuthServer DCR handler so it fires in
every auth scenario that performs DCR (no new scenario)
- withOAuthRetry now sets application_type: native (passing example;
the conformance example clients are CLI tools)
- New auth-test-no-application-type negative client + vitest case
* Widen ConformanceOAuthProvider metadata type for application_type
Fixes CI typecheck (TS2353 at withOAuthRetry.ts:76): the SDK's
OAuthClientMetadataSchema doesn't include application_type yet.
registerClient() spreads clientMetadata verbatim into the /register
POST body, so a local type intersection is sufficient to get the
field on the wire without an SDK release.
* Set application_type in runAuthMigrationClient
The SEP-2352 authorization-server-migration handler constructs its own
ConformanceOAuthProvider (not via withOAuthRetry), so it was missing
application_type after rebasing onto main. The scenario does DCR twice
(old AS, new AS) and the SEP-837 check fires on each.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
The `sdk` command exits non-zero when the SDK has conformance failures
not in its baseline. The traceability manifest only needs the emitted
check IDs (written regardless of pass/fail), so a failing SDK must not
fail the refresh. Add `|| true`; the existing 'no results produced'
guard catches a genuinely broken run.
The refresh run failed with 'pnpm: not found' — the reference SDK's
build command (typescript-sdk: pnpm install && pnpm run build:all) needs
pnpm on PATH. Add 'corepack enable' to the run job.
* Add SEP-2352 authorization-server migration scenario
SEP-2352 requires that client credentials are bound to the issuing
authorization server: when PRM authorization_servers changes to a new
issuer, clients MUST re-register and MUST NOT reuse the previous AS's
client credentials.
- Traceability yaml: 3 checks, 3 excluded (internal state / UI)
- New auth/authorization-server-migration scenario (draft suite): two
auth servers; PRM flips from AS1 to AS2 after the first authenticated
request; AS2 asserts it received a fresh /register and never saw AS1's
client_id at /authorize or /token
- ConformanceOAuthProvider gains invalidateCredentials and bindIssuer so
the everything-client can key credentials by issuer (passing example)
- everything-client adds an issuer-aware handler for this scenario that
re-reads PRM on each 401 and rebinds before re-authorizing
- auth-test-reuse-credentials negative client + vitest case
* Drop application_type from DCR metadata (not in SDK OAuthClientMetadata type)
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* feat: add server conformance tests for SEP-2575
* refactor and add new checks
* Fix stateless scenario source field and report SKIPPED for inapplicable capability checks
- Add the required `source` field (DRAFT) so the scenario typechecks and is
selectable via --spec-version. Without it the class did not satisfy the
ClientScenario interface and s.source was dereferenced by the spec-version
filter at list time.
- Teach runCheck about a SKIPPED status and use it for the two client-capability
checks when the server does not return -32003, instead of reporting a green
PASS for a requirement that was never exercised.
* Remove unimplemented placeholder checks from stateless server scenario
The subscriptions/listen, statelessness-invariant, list-changed and
disconnect-is-cancel checks emitted SUCCESS without probing the server,
which inflated coverage in the traceability report. Drop them until they
have real assertions; the corresponding rows remain declared in
src/seps/sep-2575.yaml so traceability shows them as not-yet-covered.
Also fix the checkErrorId helper to push under the
sep-2575-http-server-error-jsonrpc-id slug so its failure path actually
short-circuits the aggregate SUCCESS check at the end of the scenario.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* feat: add sdk subcommand to run conformance against any SDK ref (#250)
* Revise sdk runner: explicit --mode, KNOWN_SDKS-only config, v1/v2 entries
Addresses review feedback on the sdk subcommand:
- Require --mode (client|server) and remove "both". Each invocation now
tests exactly one side with its own exit code; the old default ran
client then server but combined exit codes with ||=, which skipped the
server side entirely whenever the client run failed.
- Resolve build/run config from KNOWN_SDKS + CLI flags only; drop the
conformance.config.yaml loader (no SDK ships one yet). The Zod schema
stays as the type for the built-in entries.
- Split the typescript entry by major version: typescript-sdk (v2/main,
pnpm install + build:all, expected-failures.yaml) and typescript-sdk-v1
(v1.x, npm ci + build, conformance-baseline.yml). An entry may set
`repo` (real clone target for an alias) and `defaultRef` (branch used
when no @ref is given). parseSdkSpec now leaves ref undefined when
omitted so defaultRef can apply; a trailing @ is treated as no ref.
- Key the clone cache by ref (<repo>/<ref>) so different refs of the same
repo no longer share one checkout.
- Bound the server readiness probe with a per-request AbortSignal timeout
so a server that accepts the socket but never responds can't hang past
the overall deadline.
* fix(sdk-runner): resolve -o to absolute path; add --expected-failures override; replaceAll for safeName
The sep-2243 scenario check IDs drifted from the requirement-traceability
yaml in #259 — the code emitted one ID per test case while the yaml
declares one per normative requirement, so the IDs no longer matched.
Rename the emitted IDs so a check ID maps to a single MUST/SHOULD and is
emitted once per case (the per-case detail moves to name/description),
matching the repo's existing 'same id, vary status/message' convention:
- mcp-method-header-* / mcp-name-header-* -> client-includes-standard-headers
- reject-invalid-tool-* / keep-valid-tool -> client-reject-invalid-tool
- server-accepts-{lower,upper}case-name -> header-name-case-insensitive
- server reject status checks (mismatch/missing/case x method/name)
-> server-reject-invalid-headers
- their error-code variants -> server-reject-error-code
Merge the yaml's server-reject-mismatch + server-reject-status into one
server-reject-invalid-headers MUST (HTTP 400 on a header-validation
failure); keep server-reject-error-code as the SHOULD. Base64/custom-header
rejection checks keep their own param-validation IDs (out of scope here).
Gates and the whitespace-acceptance check keep their scenario-matching
names. No behavior change — only check IDs, descriptions, and the yaml.
* Add prepare script for git dependency installs
* Add conformance tests for SEP-2243 HTTP Standardization
* Add traceability file for SEP-2243
* Fix prettier formatting in http-standard-headers.ts
* Improve SEP-2243 conformance test coverage
Close gaps in HTTP header conformance scenarios:
Client standard headers (http-standard-headers.ts):
- Enforce all expected methods in checks, failing any that were not observed
- Track Mcp-Name header per-method separately from Mcp-Method
- Advertise resources and prompts capabilities so clients exercise those endpoints
- Add a prompt entry for prompts/get testing
Client custom headers (http-custom-headers.ts):
- Add Base64 encoding checks for non-ASCII, whitespace, and control-char values
- Add null/omitted parameter test (second tool call with null value)
- Add client-keeps-valid-tool check verifying clients still call valid tools
after filtering out invalid ones
Server header validation (server/http-standard-headers.ts):
- Replace fetch-based sendRawRequest with http.request to preserve exact
header casing on the wire (fetch/Headers lowercases names)
- Compute defaultArgs and defaultHeaders from tool schema so test requests
satisfy all required parameters while varying only the param under test
Traceability (sep-2243.yaml):
- Add 7 new spec-to-check mappings (nonascii, whitespace, controlchar,
keeps-valid-tool, literal-missing-base64-prefix/suffix, no-mirror-unannotated)
- Fix 2 incorrect existing mappings
* Fill SEP-2243 conformance test coverage gaps
Add missing test cases identified by comparing the branch against the
SEP-2243 spec's Conformance Test Cases section:
- Mcp-Method on notifications: add notifications/initialized to
expectedMethods so clients are checked for header on notifications
- Boolean true: add debug=true param to complement verbose=false
- Leading-space-only and trailing-space-only: separate checks for
Base64 encoding when only one edge has whitespace
- Internal spaces only: verify plain ASCII (no Base64) for values
like 'us west 1' that have spaces only in the middle
- CRLF string: add line1\r\nline2 test (complements existing \n test)
- Leading tab: add \tindented test for tab-triggered Base64 encoding
- Server rejects invalid Mcp-Param chars: mark as excluded in YAML
since HTTP itself prevents these characters in headers
Register all new checks in sep-2243.yaml.
* Remove case-insensitive =?base64? prefix test
The spec states header values are case-sensitive (RFC 9110). The
=?base64? prefix is part of the header value, not the header name,
so it should be matched case-sensitively. This was identified as a
bug in the conformance tests per feedback on SEP-2243.
Changes:
- Remove server-accepts-case-insensitive-base64 check from server
validation scenario
- Change base64 prefix regex from case-insensitive (/i) to
case-sensitive in client header validation
- Remove corresponding entry from sep-2243.yaml
* Remove nested x-mcp-header rejection test
The spec constrains x-mcp-header to primitive types but does not
restrict it to top-level properties only. A nested string property
with x-mcp-header is valid per the spec. This was confirmed by
mikekistler in SEP-2243 PR discussion.
Removes invalid_nested_header from the invalid tool definitions
list and its corresponding sep-2243.yaml entry.
* Use numeric comparison for number header values
Compare number header values numerically instead of as strings to
accommodate cross-SDK floating point representation differences
(e.g., '42' vs '42.0'). For integers, exact numeric match is
required. For decimals, a tolerance of 1e-9 is allowed.
See SEP-2243 discussion on number precision.
* client/http-standard-headers: SKIPPED (not FAILURE) for unexercised methods; make getChecks() idempotent
A client that never calls prompts/list isn't violating SEP-2243 — the spec
sentence is 'The client MUST include the standard MCP request headers on each
POST request', and a request that was never sent can't fail it. The Mcp-Method
requirement is already proven by initialize/tools/list etc., so there's nothing
unique to prompts/list. SKIPPED keeps the gap visible without a false red.
Separately: getChecks() was pushing into this.checks on every call. The runner
may call it more than once (e.g. progress + final report); that produced
duplicate rows. Build a fresh array per call instead.
* client/http-standard-headers: de-dup guard in checkMcpNameHeader
checkMcpMethodHeader already has 'if (this.methodHeaderChecks.has(method)) return'.
checkMcpNameHeader was missing the equivalent. The test server advertises two
tools and two resources; a client that calls both produces two
'client-mcp-name-header-tools-call' / '...-resources-read' rows.
* client/http-standard-headers: replace -> /\//g for slug generation
String.replace with a string arg only replaces the first occurrence.
'notifications/initialized' is fine (one slash) but a method like
'notifications/resources/updated' would yield 'notifications-resources/updated'
in the check id. (replaceAll isn't in this project's TS lib target.)
* client/http-custom-headers: base64 regex (.+) -> (.*) so empty string round-trips
The spec doesn't forbid base64-encoding values that don't strictly need it.
A client that always wraps would send '=?base64??=' for empty_val: ''. The
'.+' wouldn't match, so the harness would compare '=?base64??=' === ''
literally and FAIL a compliant client. '(.*)' lets the empty payload decode
to '' and match.
* client/http-custom-headers: emit SUCCESS for no-mirror-unannotated
This check only pushed on FAILURE; a correct client produced no row at all,
so the pass was invisible in reports and couldn't be counted toward coverage.
Same-slug SUCCESS/FAILURE pair is the repo convention.
* client/http-custom-headers: drop redundant optional-present check
checkParamHeader('Verbose', ...) already pushes a FAILURE when the header is
missing. The explicit 'optional-present' block re-tests the same condition
under a second check id, doubling the row.
* server/http-standard-headers: defaultArgs must use real number/boolean, not strings
defaultArgs feeds the JSON-RPC body. With Record<string, string> + '0'/'false',
a server that validates inputSchema rejects on type mismatch — which is also a
400. Every header-rejection check then false-PASSes (server rejected for the
wrong reason), and server-accepts-valid-base64 false-FAILs a compliant server
because it never gets past schema validation.
Also handle 'integer' (JSON Schema distinguishes it from 'number').
* server/http-standard-headers: createAcceptanceCheck must also assert no JSON-RPC error in body
A server can return HTTP 200 with {error: {code: -32001, ...}} in the body.
Status-only acceptance lets that through as a pass.
* use DRAFT_PROTOCOL_VERSION constant instead of 'DRAFT-2026-v1' literal
The literal will rot when the draft cycle rolls over. types.ts already exports
DRAFT_PROTOCOL_VERSION for this.
* sep-2243.yaml: spec-first rewrite + move to src/seps/
Regenerated from the SEP-2243 spec diff (transports.mdx + tools.mdx) rather
than from the scenario implementations: 21 check rows + 4 excluded vs the
previous 46 + 2. Differences:
- one check id per spec sentence (test variants go in 'details', not new ids)
- 'text:' quotes the spec sentence verbatim instead of paraphrasing
- '-reject-status' (MUST 400) split from '-reject-error-code' (SHOULD -32001
for standard headers, MUST for custom)
- rows with no spec backing dropped (whitespace, base64-padding/chars, prefix/
suffix-literal, control-char-name)
- two more SHOULD excludes (log-warning, no-sensitive-params)
- check ids use sep-2243-<slug> convention
- check: key first, excludes grouped at bottom (matches sep-2164)
Moved to src/seps/ to match #272 layout (this branch predates it; will
reconcile cleanly on rebase).
* add negative test for HttpStandardHeadersScenario
Self-contained vitest that POSTs initialize with and without Mcp-Method,
asserts FAILURE/SUCCESS on the pinned check id, and proves getChecks() is
idempotent. No example client/server file needed for this one because the
scenario *is* the server — the test crafts the bad request directly.
This is the negative-test half of AGENTS.md §Testing your scenario; the
everything-client passing-example half is still TODO.
* server/http-standard-headers: severity fixes for whitespace + malformed-base64 tests
These five tests assert behavior the SEP-2243 text doesn't pin down. Kept
because they're useful consistency signals across SDKs, but adjusted so they
don't FAIL spec-compliant servers:
- server-accepts-whitespace-header-value: this IS a MUST, but per RFC 9110
§5.5 (field parsing MUST exclude OWS), not SEP-2243. Kept as FAILURE,
specReference now cites RFC 9110.
- server-rejects-invalid-base64-padding / -chars: downgraded to INFO.
SEP-2243 says only 'MUST decode them accordingly' without picking RFC 4648
strict vs lenient. Node's Buffer.from() accepts unpadded/dirty input and
decodes to a matching value (server accepts); .NET's Convert.FromBase64String
throws (server rejects). Either is currently compliant. WARNING would force
Tier-1 SDKs into non-default strict decoding (#245); INFO records the
behavior without prescribing it, so cross-SDK divergence is visible
without tier impact.
- server-literal-missing-base64-prefix / -suffix: unchanged. The wrapper
syntax is '=?base64?{x}?=' complete; treating partial wrappers as literal
is the natural reading.
Plumbing: createRejectionCheck gains an optional failSeverity param;
testBase64Case gains a 'reject-info' mode. (This is also the hook for the
larger 400-MUST / -32001-SHOULD split that still needs doing.)
* server/http-standard-headers: split rejection check into status (MUST) + error-code (SHOULD/MUST)
SEP-2243 §Server Validation: 'servers MUST return HTTP status 400 ... and
SHOULD include a JSON-RPC error response using ... -32001'. So a server that
returns 400 with -32600 (or no error body) is compliant for standard headers.
The single conflated check FAILed it.
For custom headers (§Server Behavior for Custom Headers) -32001 IS a MUST,
so both halves stay FAILURE there.
createRejectionCheck → createRejectionChecks returning two rows: '<id>' for
the 400 status and '<id>-error-code' for -32001. testCase (standard) sets
errorCodeSeverity=WARNING; testBase64Case/testMissingCustomHeader (custom)
keep FAILURE; the two malformed-base64 INFO probes set both halves to INFO.
* rename check ids to sep-2243-* prefix; kebab-case throughout
Aligns with the sep-2243.yaml traceability ids and the repo convention
(sep-2164-*, sep-2207-*). Variant suffixes are kept (per-method, per-tool)
so per-case visibility isn't lost; the YAML's row id is a prefix of the
emitted ids, which is fine — extra ids not in the YAML are allowed.
Also fixes the mixed hyphen/underscore in reject-invalid-tool ids
(toolName has underscores; replace to hyphens).
* client: extract BaseHttpScenario to client/http-base.ts; both client scenarios extend it
http-standard-headers.ts had its own copy of the start/stop/handleRequest
boilerplate (~100 lines) that http-custom-headers.ts already abstracted as
BaseHttpScenario. Moved the abstract class to a shared file; both scenario
files now import it. HttpStandardHeadersScenario implements handlePost()
instead of handleRequest(); its handleInitialize() uses the base
sendInitialize() with the resources/prompts capability flags it needs.
Net: 465 → 361 lines for http-standard-headers.ts; the third inline copy of
the same HTTP-server scaffold is gone.
* use req.setEncoding('utf8') instead of per-chunk Buffer.toString()
Per-chunk .toString() corrupts a multi-byte UTF-8 char that straddles a TCP
chunk boundary (e.g. 0xC3 | 0xBC decodes to two replacement chars instead of
'ü'). The custom-headers scenario sends '日本語'/'naïve' in the body, so this
is reachable in principle. setEncoding('utf8') makes 'data' emit strings with
boundary handling done by Node's StringDecoder.
Fixed in both places: BaseHttpScenario.handleRequest (request body) and
sendRawRequest (response body).
* fix: migrate SEP-2243 scenarios from specVersions to source (introducedIn/removedIn)
The introducedIn/removedIn refactor (#265) replaced the specVersions array
with a source property on all scenario interfaces. The SEP-2243 scenarios
still used the old specVersions field, causing matchesSpecVersion() to
receive undefined and crash in spec-version.test.ts.
Replace specVersions with source = { introducedIn: DRAFT_PROTOCOL_VERSION }
in BaseHttpScenario and the two server ClientScenario classes. Remove the
now-unused SpecVersion imports.
* server/http-standard-headers: malformed-base64 padding/chars back to FAILURE
Per discussion on #259: the SEP-2243 conformance-test-case table is the
approved source of truth and lists these as reject cases, so they're
FAILURE-level even though the spec body itself only says 'MUST decode them
accordingly'. SDKs whose stdlib base64 is lenient (Node, browsers) will need
to validate before decoding; if that's burdensome we'll revisit.
Removes the now-unused 'reject-info' mode and statusSeverity/INFO plumbing
introduced for these two probes.
---------
Co-authored-by: Tarek Mahmoud Sayed <tarekms@microsoft.com>
Co-authored-by: Tarek Mahmoud Sayed <tarekms@ntdev.microsoft.com>
Co-authored-by: Tarek Mahmoud Sayed <10833894+tarekgh@users.noreply.github.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* refactor: replace specVersions list with introducedIn/removedIn range tagging
Replaces per-scenario `specVersions: SpecVersion[]` arrays with `introducedIn`
and optional `removedIn` range fields on all three scenario interfaces. Extension
scenarios get an orthogonal `extension?: boolean` flag instead of the former
`'extension'` tag in the versions list.
matchesSpecVersion() is rewritten as a range comparison
(introducedIn <= target && (!removedIn || target < removedIn)), with draft
sorting after all dated versions. getScenarioSpecVersions() reconstructs the
effective version list from the range for backward-compat with tier-check.
Closes#256.
* refactor: nest spec-version range under `source: ScenarioSource` union
The flat `introducedIn`/`removedIn`/`extension?` shape forced extension
scenarios to set a placeholder `introducedIn` that was never read. Nesting
the version-range fields inside a `source` property typed as a union lets
the compiler enforce introducedIn XOR extensionId — no runtime invariant
test needed.
type ScenarioSource =
| { introducedIn: SpecVersion; removedIn?: SpecVersion }
| { extensionId: ExtensionId };
A class can't `implements` a union type directly, so the union lives inside
a property and `Scenario` stays a plain interface — `class implements
Scenario` keeps working everywhere. `EXTENSION_IDS` is a const array
mirroring the `DATED_SPEC_VERSIONS` pattern.
Also renames the cross-app-access scenario to enterprise-managed-auth to
match the extension's canonical name: file, class, slug, context schema
literal, and the everything-client runner.
* refactor: use canonical SEP-2133 extension identifiers
EXTENSION_IDS now holds the wire identifiers from the spec's extension docs
(client-matrix.mdx, docs/extensions/auth/*.mdx) so extensionId values can be
cross-referenced against capabilities.extensions keys directly:
io.modelcontextprotocol/oauth-client-credentials
io.modelcontextprotocol/enterprise-managed-authorization
The previous commit used a shortened 'enterprise-managed-auth' which doesn't
match the canonical '-authorization'. Re-renames the scenario file/class/slug
and downstream references to align.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat: add new-sep scaffolding command
Related #243.
This commit adds the new-sep <NNNN> subcommand, which writes the file src/scenarios/<target>/sep-<NNNN>.yaml. This file contains the "sep", "spec_url", and "requirements" rows in the same style as sep-2164.yaml, with single quotes and two spaces. The spec source resolves in order: --spec-url → --spec-path → GitHub API against modelcontextprotocol/modelcontextprotocol (explicit --pr or title search for "SEP-NNNN").
The target directory is inferred from the spec path (`server/`, `client/`, or `basic/authorization*`); the --target flag overrides this. It refuses to overwrite without the --force flag.
Token resolution reuses the "gh auth token" fallback pattern from tier-check.
It also ships the .claude/skills/new-sep directory, which drives the CLI. It fetches the spec diff via "gh api" and extracts RFC 2119 sentences. There is a warning against regex-only matching. Bullets inherit keywords from their lead-in sentence. It rewrites the placeholder rows per the severity rules in AGENTS.md. MUST/SHALL/REQUIRED are checked for FAILURE. SHOULD is checked for WARNING. MAY/OPTIONAL are excluded.
Signed-off-by: Satoshi Ito <satoshi.ito.tf@hitachi.com>
* dropped --pr arg, list changed files, support multi-file seps w url override
* Consolidate SEP yamls into src/seps/ and tighten yaml ordering
- All sep-NNNN.yaml files now live in src/seps/ regardless of which
scenario suite their checks run in
- Drop --target / inferTarget; output dir is fixed
- renderYaml: check rows emit `check:` first, excluded rows grouped at
bottom with a blank-line separator
- SKILL.md: document key/row ordering, expand exclusion-confirmation step
- Move src/scenarios/server/sep-2164.yaml -> src/seps/
---------
Signed-off-by: Satoshi Ito <satoshi.ito.tf@hitachi.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* feat: support "draft" as a first-class spec version target
- Add LATEST_SPEC_VERSION and DATED_SPEC_VERSIONS constants so the next
spec release is a one-line change in types.ts.
- Accept "draft" as a valid protocolVersion in the initialize check and
mock-server response.
- --spec-version draft now selects latest-dated scenarios plus draft-tagged
ones, so SEP authors can run the full suite against an SDK tracking the
in-progress spec without retagging core scenarios.
- Forward --spec-version to the client process via
MCP_CONFORMANCE_SPEC_VERSION so SDK examples can pick the matching
protocolVersion.
Closes#253
* review: collapse SpecVersion to wire strings; dedup NEGOTIABLE_PROTOCOL_VERSIONS; split out 'extension'
Addresses review feedback from @felixweinberger and @mikekistler:
- SpecVersion is now always a wire protocolVersion string
(DatedSpecVersion | typeof DRAFT_PROTOCOL_VERSION). The separate
'draft' tag literal is gone from the type system; 'draft' survives
only as a CLI input alias in resolveSpecVersion. Removes the
tag-vs-wire confusion and the specVersionToProtocolVersion mapping.
- NEGOTIABLE_PROTOCOL_VERSIONS in types.ts is the single source for
what the mock server accepts on initialize (was duplicated in
checks/client.ts and scenarios/client/initialize.ts).
- 'extension' moved out of SpecVersion into a separate ScenarioSpecTag
type. --spec-version extension is no longer valid; extension scenarios
remain reachable via --suite extensions.
- Draft-tagged scenarios now use the DRAFT_PROTOCOL_VERSION constant
so a draft revision bump is a one-line change in types.ts.
Conformance is a leaf CLI; scheduled version bumps are churn. Deleting
the config disables version-update PRs entirely. Security advisories
remain enabled via repo settings (Code security > Dependabot).
Adds src/scenarios/server/negative.test.ts with spawn helpers. Wires
two cases: the previously-orphaned no-dns-rebinding-protection.ts and
a new sep-2164-empty-contents.ts. Strengthens the AGENTS.md guidance
to point at the harness.
* feat: add conformance scenario for SEP-2164 resource-not-found error code
Adds a draft-scoped scenario verifying servers return -32602 (Invalid
Params) for non-existent resources instead of the legacy -32002 or an
empty contents array. Includes a sep-2164.yaml traceability file mapping
each MUST/MUST NOT/SHOULD to its check per the proposed SEP-2484 process.
* fix: align SEP-2164 check severities with spec text (#246)
Splits sep-2164-error-code into:
- sep-2164-no-empty-contents (MUST NOT -> FAILURE)
- sep-2164-error-code (SHOULD -> WARNING)
Updates sep-2164.yaml to quote the spec text verbatim and adds an
excluded row for the client-side -32002 backcompat SHOULD.
See #245 for the SHOULD/WARNING tier-scoring policy discussion.
---------
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
Fixes a bug where the Markdown output format for the Repository Health section lacked a table header, causing it to render as raw text instead of a table on GitHub.
* feat: add --spec-version option to tier-check
* refactor: share resolveSpecVersion between CLI and tier-check
Move resolveSpecVersion to src/scenarios/index.ts so tier-check can
reuse it instead of duplicating the validation logic.
---------
Co-authored-by: Felix Weinberger <fweinberger@anthropic.com>
* Fix Windows compatibility in tier-check client conformance
Two issues prevented client conformance tests from running on Windows:
1. Single-quote quoting in execSync: The --command argument was wrapped in single quotes, which cmd.exe treats as literal characters rather than string delimiters. This caused the CLI to receive only the first word of the command. Switched to execFileSync with an args array to bypass shell quoting entirely.
2. Backslash path separators in parseOutputDir: dirname() on Windows returns auth\scenario-name but reconcileWithExpected compares against auth/scenario-name, causing auth scenarios to be double-counted. Normalize backslashes to forward slashes in parsed scenario names.
* fix: use process.argv[1] for CLI path in subprocess calls
Combines the npx-compatibility fix from #175 with this PR's execFileSync
approach. process.argv[1] resolves to the absolute path of the running CLI
regardless of working directory, so tier-check works when invoked via npx
or from any cwd.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
Major bumps of devDeps (eslint, typescript, prettier) need manual config
migration and provide no value for conformance testing. Security updates
still come through regardless; scheduled updates are now minor/patch only.
* docs: add AGENTS.md and CONTRIBUTING.md
Codifies scenario design guidelines from past PR reviews:
- fewer scenarios with more checks (CI cost multiplies per-SDK)
- same check id for SUCCESS and FAIL, optimize for Ctrl+F
- extend the everything-client/server rather than adding new example files
- prove it passes and fails
- reuse the CLI runner, don't add parallel entry points
Also asks contributors to open an issue for discussion before a PR, and
to target specific spec MUSTs or open issues rather than running generic
AI bug-hunts against the harness.
* docs: broaden issue-first guidance to cover bugs, not just scenarios
* feat: add sep-2207 client checks
* Update description for the offline access test
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
---------
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
With Node.js 20 the action fails with:
```
SyntaxError: The requested module 'fs' does not provide an export named 'globSync'
at ModuleJob._instantiate (node:internal/modules/esm/module_job:213:21)
at async ModuleJob.run (node:internal/modules/esm/module_job:320:5)
at async ModuleLoader.import (node:internal/modules/esm/loader:606:24)
at async asyncRunEntryPointWithESMLoader (node:internal/modules/run_main:117:5)
```
`fs.globSync` was added in Node.js 22.
* fix: pass --branch to tier-check CLI in tier-audit skill
The skill was not forwarding the --branch argument to the tier-check
CLI, causing policy signal checks to always run against the repo's
default branch. Files on feature branches showed as 'Not found.'
Now derives the branch from the local checkout if not explicitly
provided, and always passes it to the CLI.
* fix: exclude draft/extension scenarios from tier-scoring conformance rates
SEP-1730 says date-versioned scenarios count toward tier scoring while
draft and extension scenarios are informational. The CLI was including
all scenarios in pass_rate, causing extension-only failures to block
Tier 1.
Changes:
- pass_rate now only counts scenarios with at least one date-versioned
spec version
- Terminal and markdown output split into a tier-scoring matrix
(date-versioned + All*) and an informational section (draft/extension)
that only renders when there are draft/extension scenarios
* feat: add specVersions classification to conformance scenarios
Each scenario declares which spec versions it applies to as a list.
Scenarios that carry forward (e.g. initialize) list all applicable versions
['2025-06-18', '2025-11-25']. Scenarios removed from newer specs (e.g.
backcompat auth) only list their original version ['2025-03-26'].
- specVersions list on Scenario and ClientScenario interfaces
- --spec-version CLI filter uses simple .includes()
- Tier-check conformance matrix (Server / Client: Core / Client: Auth)
with per-version columns and unique All* count
- 7 unit tests for specVersions helpers
- Updated tier-audit skill docs with matrix format
* fix: fix console output template for tier audit skill
The template had an orphan table header (Check | Value | T2 | T1)
with no rows above the conformance matrix, causing an empty table
to render. The scorecard rows below the matrix also lacked their
own header.
Fix: two self-contained tables with clear labels — 'Conformance:'
for the per-version matrix, 'Scorecard:' for the check rows.
* fix: align skill console template with tier-check script output
Add asterisk footnote ('unique scenarios — a scenario may apply to
multiple spec versions') and rename 'Scorecard' to 'Repository Health'
to match the labels used by the tier-check CLI output.
* fix: add specVersions to CrossAppAccessCompleteFlowScenario
Added in 83c446d on main after this branch diverged.
The SDK's `requireBearerAuth` middleware only converts `InvalidTokenError`
instances to HTTP 401 responses. Generic `Error` instances fall through
as HTTP 500, which prevents clients from detecting authentication failures
and initiating the OAuth refresh/re-auth flow.
This was discovered while building token refresh conformance scenarios —
the mock server was returning 500 for expired/invalid tokens instead of
the expected 401.
Co-authored-by: JD Maturen <70791+jdmaturen@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: add conformance tests for SEP-990
* Resolving review changes: Removed redundant tests, updated audience params
* fix: unused serverUrl parameter in runCrossAppAccessTokenExchange
* chore: apply prettier formatting
* fix: address PR review comments for SEP-990 conformance tests
- Delete unused separate token-exchange and jwt-bearer scenarios,
keeping only the complete e2e flow (review comment)
- Add missing required token exchange params per SEP-990 spec:
requested_token_type, audience, resource (review comment)
- Use ctx.idp_client_id for token exchange client_id instead of
AS client_id (review comment)
- Client discovers resource and auth server via PRM metadata
instead of receiving auth_server_url via context (review comment)
- Server IdP handler verifies all required token exchange params
with detailed error messages (review comment)
- Add resource, client_id, jti claims to ID-JAG per SEP-990 spec
- Verify ID-JAG typ header (oauth-id-jag+jwt) in JWT bearer handler
- Remove auth_server_url from context schema
* feat: add client auth and ID-JAG validation to XAA conformance test
Server-side (AS) now verifies:
- client_secret_basic authentication on JWT bearer grant
- ID-JAG typ header is oauth-id-jag+jwt
- ID-JAG client_id claim matches the authenticating client (Section 5.1)
- ID-JAG resource claim matches the MCP server resource identifier
- Client credentials provided via context (client_secret)
Server-side (IdP) now:
- Sets ID-JAG client_id to the MCP Client's AS client_id (not the
IdP client_id), per Section 6.1
Example client now:
- Authenticates to AS via client_secret_basic (Authorization: Basic)
instead of sending client_id in body
- Checks AS metadata grant_types_supported includes jwt-bearer
before attempting the flow
* fix: share MockTokenVerifier and remove unadvertised auth method
- Add shared MockTokenVerifier between AS and MCP server so the MCP
server only accepts tokens actually issued by the auth server,
matching the pattern used by all other auth scenarios
- Remove private_key_jwt from tokenEndpointAuthMethodsSupported since
the handler only implements client_secret_basic
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
parseOutputDir only read one level deep, so scenarios with '/' in their
name (e.g. auth/metadata-default) were stored in nested subdirectories
and never found. This caused tier-check to report 4/23 client conformance
when the actual pass rate was 23/23.
Also removes accidentally committed pnpm-lock.yaml.
* feat: add tier-check CLI for SDK tier assessment
Adds a 'tier-check' subcommand to the conformance tool that automates
SDK tier assessment against SEP-1730 criteria.
Checks performed:
- Conformance test pass rate (via everything-server)
- GitHub label taxonomy (priority/status/area labels)
- Issue triage SLA compliance
- P0 bug resolution tracking
- Stable release detection
- Required file existence (CHANGELOG, SECURITY, etc.)
- Spec tracking (SDK release within 30d of spec release)
Also includes a Claude Code skill (skills/mcp-sdk-tier-audit/) for
judgment-based checks that require codebase analysis (feature coverage,
docs quality, policy evaluation).
Usage:
npx tsx src/index.ts tier-check --repo modelcontextprotocol/typescript-sdk
npx tsx src/index.ts tier-check --repo ... --conformance-server-cmd '...' \
--conformance-server-cwd ... --conformance-server-url ... --output json
* refactor: revise tier-check CLI and skill based on review feedback
- Move skill to .claude/skills/ so it's auto-available in Claude Code
- Remove feature-coverage subagent (redundant with conformance tests)
- Remove hardcoded ~/src/mcp paths from all skill files
- Trim conformance server table to TS + Python only
- Rename file_existence check to policy_signals (informational, not blocking)
- Add GitHub native issue types detection to labels check
- Add missing features to docs-coverage checklist (tasks, elicitation URL mode, JSON Schema 2020-12)
- Add README with CLI quick start and escape hatch for non-Claude-Code users
- Use --limit 500 instead of --limit 100 for gh issue list
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: address review feedback on skill and README
- Use npx @modelcontextprotocol/conformance instead of node dist/index.js
- Add full GitHub auth instructions (gh auth login, GITHUB_TOKEN, --token)
- Point TS SDK conformance server to typescript-sdk/test/conformance/
- Fix Python SDK URL to localhost:3001/mcp (not TBD)
- Remove manual gh issue list / gh release list from SKILL.md (CLI handles it)
- Remove Claude Code-specific subagent_type references
- Assume user is already in conformance repo
- Clean up policy-evaluation-prompt.md: remove redundant grep commands,
focus on content evaluation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve lint errors in conformance.ts
Remove unused variable assignments flagged by eslint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: apply prettier formatting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: add npm run tier-check script, update docs with examples
- Add "tier-check" npm script so users can run `npm run tier-check --`
instead of `node dist/index.js tier-check`
- Update SKILL.md, README, and skill README to use npm run tier-check
- Add full conformance examples with --conformance-server-cmd/cwd/url
flags and realistic paths (~/src/mcp/typescript-sdk)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: fix prettier formatting in SKILL.md
* fix: add per-scenario timeout, support url-only conformance, fix stdout pollution
- Add 30s per-scenario timeout to prevent tier-check from hanging
- Allow --conformance-server-url without --conformance-server-cmd (server already running)
- Move runner status logs to stderr so --output json produces clean JSON
- Update SKILL.md and README with --silent flag and pre-start server workflow
* refactor: skill takes local path + server URL instead of repo name
The skill now requires two arguments:
1. Local path to the SDK checkout (for direct file inspection)
2. URL where the everything server is already running
The GitHub owner/repo is derived from git remote. This eliminates
cloning, server startup complexity, and branch confusion (v1.x vs main).
* refactor: write detailed reports to files, show concise summary
Reports go to results/tier-audits/<sdk>-<date>/ (already gitignored).
Claude's console output is now just the tier classification,
pass/fail summary line, top 3 actions, and file paths.
* docs: update READMEs for new skill interface and pre-start workflow
* simplify: flat file output instead of nested directory
* fix: remediation always shows path to Tier 2 and Tier 1
* feat: add client conformance testing to tier-check CLI and skill
- Add checkClientConformance() that runs core client scenarios
(initialize, tools_call, elicitation-defaults, sse-retry, auth)
by spawning the SDK's conformance client via --client-cmd
- Add client_conformance to TierScorecard type
- Wire --client-cmd option into the CLI
- Update tier logic: both server + client conformance feed into
Tier 1 (100%) and Tier 2 (>=80%) requirements
- Update terminal and markdown output to show both conformance types
- Update skill to auto-detect conformance client or accept explicit
client-cmd argument
- Update README with new option and examples
* improve: table summary output, write reports via subagents
- Change executive summary from pipe-delimited line to a readable
table with T2/T1 columns
- Move assessment and remediation file writing into parallel
subagents to keep the main conversation thread clean
* improve: list tier gaps as numbered items instead of one-line blob
* improve: finalize summary format with separator, high-priority fixes, numbered gaps
* improve: add pre-flight checks for gh auth and server reachability
Fail fast with clear error messages if GitHub CLI is not authenticated
or if the conformance server URL is not reachable, rather than failing
deep into the scorecard run.
* docs: improve README and fix skill auto-detection paths
- Claude Code section: explain client-cmd auto-detection for TS/Python,
show explicit 3-arg form for other SDKs, add examples for all three
- Fix TypeScript build command (npm run build, not pnpm build:all)
- Fix Python server command (add --port, use uv sync --package)
- Fix Python client path (.github/actions/conformance/client.py)
- Expand 'Other SDKs' section with guidance on everything server
- Add gh auth login prerequisite to Claude Code steps
* simplify: remove client-cmd auto-detection, require explicit argument
Client command is now always passed as the third argument. If omitted,
client conformance is skipped and noted as a gap. No more magic path
detection — clearer and more predictable.
* fix: align docs table with canonical list (48 features), simplify policy eval
Docs coverage:
- Table now has numbered rows matching all 48 non-experimental features
from the canonical list (was missing 7: tools text/image/audio/embedded/
error/notifications, protocol version negotiation)
- Hardcode total as 48 in summary so agents don't miscount
Policy evaluation:
- Simplified from deep content analysis to file-existence checks
- Dependency policy: DEPENDENCY_POLICY.md, dependabot.yml, or CONTRIBUTING.md section
- Roadmap: ROADMAP.md must exist (GitHub milestones alone not sufficient)
- Versioning: VERSIONING.md or CONTRIBUTING.md section
- Removed GitHub API calls for milestones and releases from policy eval
* refactor: extract canonical feature list into single source of truth
Create references/feature-list.md with all 48 non-experimental + 5
experimental features. The docs-coverage prompt now references this
file instead of duplicating the list. One place to update when
features change.
* fix: separate deterministic file checks from AI content evaluation
CLI (files.ts): now checks all policy files deterministically —
DEPENDENCY_POLICY.md, docs/dependency-policy.md, dependabot.yml,
renovate.json, ROADMAP.md, docs/roadmap.md, VERSIONING.md,
docs/versioning.md, BREAKING_CHANGES.md (in addition to existing
CHANGELOG.md, SECURITY.md, CONTRIBUTING.md).
AI policy eval: receives CLI output showing which files exist,
then reads ONLY those files to judge content quality. No longer
searches the repo for files — clean separation of concerns.
* style: apply prettier formatting
* revert: undo unrelated console.log change in runner/server.ts
* refactor: shell out to conformance CLI instead of reimplementing runner
Address PR feedback: conformance.ts was duplicating the normal conformance
running code. Now shells out to 'node dist/index.js server/client' with
-o to save results to a temp dir, then parses the checks.json files.
Also removes --conformance-server-cmd and --conformance-server-cwd options
since the server must be pre-started.
* docs: add Go and C# SDK examples to README and SKILL.md
* fix: add --framework net9.0 to C# server command
* rename: conformance.ts -> test-conformance-results.ts
Avoids confusion with src/runner/ (the actual conformance runner).
This file just invokes the CLI and parses output.
* style: prettier formatting
* fix: reconcile conformance results against full scenario list
The tier-check CLI was only counting scenarios that produced a
checks.json file. Scenarios that crashed or failed to run (e.g., auth
scenarios when OAuth is not implemented) were invisible, making the
denominator artificially small (e.g., 4/4 instead of 4/23).
Now both checkConformance and checkClientConformance reconcile their
parsed results against the known scenario lists, adding failure entries
for any expected scenario that didn't produce results.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: tighten documentation evaluation criteria
Clarify what counts as documented vs just having code:
- Conformance test servers don't count as docs or examples
- Examples without prose = PARTIAL, not PASS
- Go Example* test functions explicitly allowed
- Clear PASS/PARTIAL/FAIL verdict definitions
* docs: add Labels and Spec Tracking rows to audit report templates
The executive summary and assessment report were missing two SEP-1730
requirements: label taxonomy compliance and spec tracking (new protocol
features timeline).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: reuse ConformanceCheck type from src/types.ts instead of redefining
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The auth test server helper and no-dns-rebinding-protection example
reused a single Server instance across requests, calling connect()
on each new transport. SDK v1.26.0's security fix (GHSA-345p-7cg4-v4c7)
now throws "Already connected to a transport" in this case.
Fix by wrapping Server creation in a factory function called per
request, matching the pattern already used by tools_call.ts and
everything-server.ts.
Also updates no-dns-rebinding-protection.ts to use registerTool()
instead of the deprecated tool() API.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix: bump SDK to 1.26.0, add session management to tools_call scenario
SDK 1.26.0 made Protocol.connect() throw if already connected to a
transport. The tools_call scenario was calling server.connect(transport)
on every request with the same Server instance, which now throws.
Fixed by adding proper session management: each initialize request
creates a new Server + Transport pair, and subsequent requests are
routed to the correct transport via the mcp-session-id header.
* fix: return 404 for invalid session IDs in tools_call POST handler
The else branch was returning 400 for both missing-session and
invalid/stale-session cases. The MCP spec requires 404 for invalid
session IDs and 400 only for non-initialization requests without
any session ID.
* refactor: keep tools_call stateless, create fresh server per request
Instead of adding session management, simply create a new Server
instance per request. This preserves the original stateless design
while fixing the SDK 1.26.0 Protocol.connect() restriction.
Assess any MCP SDK repository against [SEP-1730](https://github.com/modelcontextprotocol/modelcontextprotocol/issues/1730) (the SDK Tiering System). Produces a tier classification (1/2/3) with an evidence-backed scorecard.
Two components work together:
- **`tier-check` CLI** — runs deterministic checks (server + client conformance pass rate, issue triage speed, P0 resolution, labels, releases, policy signals). Works standalone, no AI needed.
- **AI-assisted assessment** — an agent uses the CLI scorecard plus judgment-based evaluation (documentation coverage, dependency policy, roadmap) to produce a full tier report with remediation guide.
## Quick Start: CLI
For an SDK listed in `src/sdk-runner/known-sdks.ts`, tier-check manages the whole conformance side itself — clone/build, then for each shipped revision the server invocation that SDK declares for that revision:
For public repos, any authenticated token works (no special scopes needed — authentication just avoids rate limits). For a [fine-grained personal access token](https://github.com/settings/personal-access-tokens/new), select **Public Repositories (read-only)** with no additional permissions.
### CLI Options
```
--repo <owner/repo> GitHub repository (required)
--branch <branch> Branch to check
--skip-conformance Skip conformance tests
--conformance-server-url <url> URL of the already-running conformance server
--client-cmd <cmd> Command to run the SDK conformance client (for client conformance tests)
Use `--output json` to get machine-readable results, or `--output markdown` for a report you can paste into an issue.
## Full AI-Assisted Assessment
The CLI produces a deterministic scorecard, but some SEP-1730 requirements need judgment: documentation quality, dependency policy, roadmap substance. An AI agent can evaluate these by reading the repo.
### Claude Code
The skill lives in `.claude/skills/` in this repo, so if you open [Claude Code](https://docs.anthropic.com/en/docs/claude-code) in the conformance repo it's already available.
1. Make sure `gh auth login` is done (the skill checks this upfront)
2. Start the SDK's everything server in a separate terminal
Pass the client command as the third argument to include client conformance testing. If omitted, client conformance is skipped and noted as a gap in the report.
**Pass `--requirements` with every revision the SDK claims**, comma-separated. Each revision's scenarios run at that revision's own wire version, and all of them must pass for Tier 1: the dated revisions through `2025-11-25` use the stateful initialize handshake while `2026-07-28` is stateless, so a scenario belonging to both has to work on both and one run does not cover the other. It also means a scenario added to the suite after a revision shipped cannot fail an SDK that had no opportunity to adopt it. Without the flag, scoring uses the suite as it stands today, which is not a tier claim. See [Conformance Requirements](../../../README.md#conformance-requirements), and run `conformance list --requirements 2025-11-25,2026-07-28` to see both sets.
**Known SDKs — no server to start, no commands to look up:**
2. **Evaluate documentation coverage** — check whether MCP features (tools, resources, prompts, sampling, transports, etc.) are documented with examples. See [`references/docs-coverage-prompt.md`](references/docs-coverage-prompt.md) for the full checklist.
3. **Evaluate policies** — check for dependency update policy, roadmap, and versioning/breaking-change policy. See [`references/policy-evaluation-prompt.md`](references/policy-evaluation-prompt.md) for criteria.
4. **Apply tier logic** — combine scorecard + evaluations against the thresholds in [`references/tier-requirements.md`](references/tier-requirements.md).
5. **Generate report** — use [`references/report-template.md`](references/report-template.md) for the output format.
### Manual Review
Run the CLI for the scorecard, then review docs and policies yourself using the tier requirements as a checklist:
| Issue triage | Within 2 business days | Within 1 month |
| P0 resolution | Within 7 days | Within 2 weeks |
| Stable release | >= 1.0.0 with clear versioning | At least one >= 1.0.0 |
| Documentation | All features with examples | Core features documented |
| Dependency policy | Published | Published |
| Roadmap | Published with spec tracking | Plan toward Tier 1 |
## Running Conformance Tests
For any SDK in [`src/sdk-runner/known-sdks.ts`](../../../src/sdk-runner/known-sdks.ts), don't start servers by hand — the config runs the right invocation per revision:
# a single leg at a single revision, for debugging
npx @modelcontextprotocol/conformance sdk --path <sdk checkout> --mode server --requirements 2026-07-28
```
To reproduce one leg fully by hand, copy the build/server/client commands from that SDK's `known-sdks.ts` entry (including its `specOverrides` for the revision you're running) rather than from a README: the entries are exercised by the test suite and per-revision, prose examples are neither.
**Other SDKs:** Your SDK needs an "everything server" — an HTTP server implementing the [Streamable HTTP transport](https://modelcontextprotocol.io/specification/draft/basic/transports.md) with all MCP features (tools, resources, prompts, etc.). See the implementations above as reference.
Start your everything server, then pass `--conformance-server-url`. Pass `--client-cmd` if your SDK has a conformance client. If neither exists yet, use `--skip-conformance` — the scorecard will note this as a gap.
## Reference Files
These files in [`references/`](references/) contain the detailed criteria and prompts:
You are performing a comprehensive tier assessment for an MCP SDK repository against SEP-1730 (the SDK Tiering System). Your goal is to produce a definitive tier classification (Tier 1, 2, or 3) backed by evidence.
## Step 0: Pre-flight Checks
Before doing anything else, verify GitHub CLI authentication:
```bash
gh auth status 2>&1
```
If this fails (exit code non-zero or shows "not logged in"), stop immediately and tell the user:
> GitHub authentication is required for this skill. Please run `gh auth login` first, then re-run the skill.
Do NOT proceed to any other step if this check fails.
**Known SDKs need no server from you.** When the SDK is in `src/sdk-runner/known-sdks.ts` (typescript-sdk, python-sdk, go-sdk, rust-sdk, csharp-sdk, ruby-sdk, and variants), skip the server preflight entirely: Step 2's `--sdk-path` mode builds the SDK and starts the right server per revision itself. The reachability check below applies only to the URL fallback for unknown SDKs.
After parsing arguments (Step 1), for the URL fallback only, verify the conformance server is reachable:
If the server is not reachable, stop and tell the user:
> Conformance server at `<url>` is not reachable. Make sure the everything server is running before invoking this skill.
## Step 1: Parse Arguments
Extract from the user's input:
- **local-path**: absolute path to the SDK checkout (e.g. `~/src/mcp/typescript-sdk`)
- **conformance-server-url** (only for SDKs not in `known-sdks.ts`): URL where the SDK's everything server is already running (e.g. `http://localhost:3000/mcp`). That one endpoint must serve every claimed revision at its own wire version; SDKs that vary the server per revision cannot be driven this way
- **client-cmd** (optional): command to run the SDK's conformance client (e.g. `npx tsx test/conformance/src/everythingClient.ts`). If not provided, client conformance tests are skipped and noted as a gap in the report.
- **requirements** (optional): spec revisions to score against, comma-separated, e.g. `--requirements 2025-11-25,2026-07-28`. Each revision's scenarios run at that revision's wire version, and every listed revision must pass for Tier 1. Scores the SDK against exactly the scenarios that revision required when it shipped, rather than everything the suite carries today. Prefer it whenever the question is "does this SDK conform to revision X". Without it, scoring uses today's suite, which can fail an SDK for a scenario added after it shipped. Run `conformance list --requirements <revision>` to see the set.
- **branch** (optional): Git branch to check on GitHub (e.g. `--branch fweinberger/v1x-governance-docs`). If not provided, derive from the local checkout's current branch: `cd <local-path> && git rev-parse --abbrev-ref HEAD`. This is passed to the tier-check CLI so that policy signal file checks use the correct branch instead of the repo's default branch.
The first two arguments are required. If either is missing, ask the user to provide it.
Derive the GitHub `owner/repo` from the local checkout:
```bash
cd <local-path> && git remote get-url origin | sed 's#.*github.com[:/]##; s#\.git$##'
```
## Step 2: Run the Deterministic Scorecard
The `tier-check` CLI handles all deterministic checks — server conformance, client conformance, labels, triage, P0 resolution, releases, policy signals, and spec tracking. You are already in the conformance repo, so run it directly.
For a known SDK, this manages everything — build, per-revision server, both legs:
```bash
npm run --silent tier-check -- \
--sdk-path <local-path> \
--requirements <revisions> \
--output json
```
`--repo` and `--branch` are derived from the SDK config and checkout; pass them only to override. For an SDK not in `known-sdks.ts`, fall back to driving it directly:
Omit `--requirements` only if the user did not name a revision. When it is set the
scorecard reports `requirements_revision`, both pass rates count exactly the
scenarios that revision requires, and anything run but not scored carries a
`notScoredReason` of `extension` or `added-after-release`. `requirements_revisions`
lists every revision scored, and each detail carries the `revision` it came from. Quote the revision
alongside any conformance number, and report the not-scored failures separately
rather than folding them into the score or omitting them.
If no client-cmd was detected, omit the `--client-cmd` flag (client conformance will be skipped). The `--branch` flag should always be included (derived from the local checkout if not explicitly provided).
The CLI output includes server conformance pass rate, client conformance pass rate (with per-spec-version breakdown), issue triage compliance, P0 resolution times, label taxonomy, stable release status, policy signal files, and spec tracking gap. Parse the JSON output to feed into Step 4.
The conformance results now include a `specVersions` field on each detail entry, enabling per-version pass rate analysis. The `list` command also shows spec version tags: `node dist/index.js list` shows `[2025-06-18]`, `[2025-11-25]`, `[draft]`, or `[extension]` next to each scenario.
### Conformance Baseline Check
After running the CLI, check for an expected-failures baseline file in the SDK repo:
If found, read the file. It lists known/expected conformance failures. This context is essential for interpreting raw pass rates — a 20% client pass rate due entirely to unimplemented OAuth scenarios is very different from 20% due to broken core functionality.
## Step 3: Launch Parallel Evaluations
Launch 2 evaluations in parallel. Each reads the SDK from the local checkout path.
**IMPORTANT**: Launch both evaluations at the same time (in the same response) so they run in parallel.
### Evaluation 1: Documentation Coverage
Use the prompt from `references/docs-coverage-prompt.md`. Pass the local path.
This evaluation checks:
- Whether all non-experimental features are documented with examples (Tier 1 requirement)
- Whether core features are documented (Tier 2 requirement)
- Produces an evidence table with file:line references
### Evaluation 2: Policy Evaluation
Use the prompt from `references/policy-evaluation-prompt.md`. Pass the local path, the derived `owner/repo`, and the `policy_signals` section from the CLI JSON output.
The CLI has already checked which policy files exist (ROADMAP.md, DEPENDENCY_POLICY.md, dependabot.yml, VERSIONING.md, etc.). The AI evaluation reads only the files the CLI found to judge whether the content is substantive — it does NOT search for files in other locations.
This evaluation checks:
- Dependency update policy (required for Tier 1 and Tier 2)
- Published roadmap (required for Tier 1; plan-toward-Tier-1 for Tier 2)
- Clear versioning with documented breaking change policy (required for Tier 1)
- Produces evidence tables for each policy area
## Step 4: Compute Final Tier
Combine the deterministic scorecard (from the CLI) with the evaluation results (docs, policies). Apply the tier logic:
### Tier 1 requires ALL of:
- Server conformance test pass rate == 100% across every requirement set given to `--requirements` (each run at its own wire), otherwise of date-versioned scenarios only
- Client conformance test pass rate == 100%, on the same basis
- Issue triage compliance >= 90% within 2 business days
- All P0 bugs resolved within 7 days
- Stable release >= 1.0.0 with no pre-release suffix
- Clear versioning with documented breaking change policy (evaluation)
- All non-experimental features documented with examples (evaluation)
- Published dependency update policy (evaluation)
- Published roadmap with concrete steps tracking spec components (evaluation)
### Tier 2 requires ALL of:
- Server conformance test pass rate >= 80% (date-versioned scenarios only)
- Published plan toward Tier 1 or explanation for remaining Tier 2 (evaluation)
### Otherwise: Tier 3
If any Tier 2 requirement is not met, the SDK is Tier 3.
**Important edge cases:**
- If GitHub issue labels are not set up per SEP-1730, triage metrics cannot be computed. Note this as a gap. However, repos may use GitHub's native issue types instead of type labels — the CLI checks for both.
- If client conformance was skipped (no client command found), note this as a gap but do not block tier advancement based on it alone.
**Conformance Breakdown:**
The **full suite** pass rates (server total, client total) are used for tier threshold checks. To interpret them, present a single conformance matrix combining server and client results. Each detail entry in the tier-check JSON has a `specVersions` field; client category is derived from the scenario name (`auth/` prefix = Auth, everything else = Core). Server scenarios are all Core.
The tier-scoring table only includes date-versioned scenarios. `draft` and `extension` scenarios are shown separately as informational — they do not affect tier advancement.
This immediately shows where failures concentrate. Failures clustered in Client: Auth / `2025-11-25` means "new auth features not yet implemented" — a scope gap, not a quality problem. Failures in Server or Client: Core are more concerning.
If the SDK has a `baseline.yml` or expected-failures file, cross-reference with the matrix to identify whether baselined failures cluster in a specific cell (e.g. all in `2025-11-25` / Client: Auth = scope gap).
**P0 Label Audit Guidance:**
When evaluating P0 metrics, flag potentially mislabeled P0 issues:
- If P0 count is high (>2) but other Tier 2 metrics (conformance, triage compliance, docs) are strong, this may indicate P0 labels are being used for enhancements, lower-priority work, or feature requests rather than actual critical bugs.
- In such cases, recommend a P0 label audit as a remediation action. Review open P0 issues to verify they represent genuine blocking defects vs. misclassified work.
- Document this finding in the remediation output with specific issue numbers and suggested re-triage actions.
- Do not treat high P0 count as an automatic hard blocker if the audit reveals mislabeling; instead, note it as a process improvement opportunity.
## Step 5: Generate Output
Write detailed reports to files using subagents, then show a concise summary to the user.
### Output files (write via subagents)
**IMPORTANT**: Write both report files using parallel subagents (Task tool) so the file-writing work does not pollute the main conversation thread. Launch both subagents at the same time.
Write two files to `results/` in the conformance repo:
For example: `results/2026-02-11-typescript-sdk-assessment.md`
#### Assessment subagent
Pass all the gathered data (CLI scorecard JSON, docs evaluation results, policy evaluation results) to a subagent and instruct it to write the assessment file using the template from `references/report-template.md`. This file contains the full requirements table, conformance test details (both server and client), triage metrics, documentation coverage table, and policy evaluation evidence.
#### Remediation subagent
Pass all the gathered data to a subagent and instruct it to write the remediation file using the template from `references/report-template.md`. This file always includes both:
- **Path to Tier 2** (if current tier is 3) -- what's needed to reach Tier 2
- **Path to Tier 1** (always) -- what's needed to reach Tier 1
### Console output (shown to the user)
After the subagents finish, output a short executive summary directly to the user:
| Documentation | <pass>/<total> features | ✓/✗ | ✓/✗ |
| Dependency Policy | <summary> | ✓/✗ | ✓/✗ |
| Roadmap | <summary> | ✓/✗ | ✓/✗ |
| Versioning Policy | <summary> | N/A | ✓/✗ |
| Stable Release | <version> | ✓/✗ | ✓/✗ |
---
**High-Priority Fixes:**
1. <fix description>
**For Tier 2:**
1. <gap description>
2. <gap description>
**For Tier 1:**
1. <gap description>
2. <gap description>
Reports:
- results/<date>-<sdk-name>-assessment.md
- results/<date>-<sdk-name>-remediation.md
```
Use ✓ for pass and ✗ for fail.
**High-Priority Fixes**: List any issues that need urgent attention (e.g., P0 label audit if P0 count is >2 but other metrics are strong, suggesting mislabeled issues). If none, omit this section.
**For Tier 2 / For Tier 1**: List each gap as a separate numbered item. Use "All requirements met" if there are no gaps for that tier. Each item should be a concise action (e.g., "Re-triage mislabeled P0s", "Document 16 undocumented core features").
## Reference Files
The following reference files are available in the `references/` directory alongside this skill:
- `references/feature-list.md` -- Canonical list of 48 non-experimental + 5 experimental features (single source of truth)
- `references/tier-requirements.md` -- Full SEP-1730 requirements table with exact thresholds
- `references/report-template.md` -- Output format template for the audit report
- `references/docs-coverage-prompt.md` -- Evaluation prompt for documentation coverage
- `references/policy-evaluation-prompt.md` -- Evaluation prompt for policy review
Read these reference files when you need the detailed content for evaluation prompts or report formatting.
You are evaluating the documentation coverage of an MCP SDK repository for the SEP-1730 tier assessment.
## Input
- **SDK path**: {local-path} (absolute path to local SDK checkout)
## Your Task
Evaluate the documentation quality and coverage of this MCP SDK against the canonical feature list. You need to determine:
1. **Tier 1 compliance**: Are ALL non-experimental features documented with examples?
2. **Tier 2 compliance**: Are core features documented (basic docs)?
## Steps
### 1. Read the canonical feature list
Read `references/feature-list.md` for the definitive list of 48 non-experimental features (plus 5 experimental) to evaluate. That file is the single source of truth — use every feature listed there, in order.
### 2. Find all documentation sources
The SDK is available at `{local-path}`. Search for documentation in these locations:
- `README.md` (root and any subdirectory READMEs)
- `docs/` directory
- `documentation/` directory
- `examples/` directory
- API documentation (generated or hand-written)
- `CONTRIBUTING.md`
- Inline code comments and docstrings on public API surfaces
- {If FAIL: list the core features missing documentation}
```
## What Counts as "Documented"
A feature is "documented" only if there is **prose documentation** (in README, docs/, or similar) explaining what the feature does, when to use it, and how it works. The following do **not** count as documentation on their own:
- Example code without accompanying prose explanation
- Conformance test servers or test fixtures
- Source code, even with comments or docstrings
- Mere existence of an API (e.g., a function existing in the SDK)
**Examples supplement documentation but do not replace it.** A feature with a working example in `examples/` but no prose explaining the feature is PARTIAL, not PASS. A feature with only a conformance server implementation and no user-facing docs is FAIL.
### Verdict criteria
- **PASS**: Prose documentation exists explaining the feature AND at least one runnable or near-runnable code example
- **PARTIAL**: Either prose docs exist but no examples, OR examples exist but no prose docs
- **FAIL**: No prose documentation and no examples. Also use FAIL if the feature is only demonstrated in test/conformance code with no user-facing docs or examples
### What counts as an "example"
- Runnable code in an `examples/` directory
- Code snippets embedded in prose documentation (README, docs/\*.md)
- Go `Example*` test functions (these render on pkg.go.dev and are a language convention)
- Examples in test files count only if they are clearly labeled as examples or referenced from documentation
### What does NOT count as an example
- Conformance test server implementations
- Internal test fixtures
- Source code of the SDK itself
## Other Important Notes
- If the SDK does not implement a feature at all, mark it as "FAIL" for documentation but note "Not implemented" in the Where column.
- Be thorough: check README, docs/, examples/, API references, and inline docstrings.
- Apply these criteria consistently across all features. Do not give credit for documentation that doesn't exist.
You are evaluating the governance and policy documentation of an MCP SDK repository for the SEP-1730 tier assessment.
## Input
- **SDK path**: {local-path} (absolute path to local SDK checkout)
- **Repository**: {repo} (GitHub `owner/repo`, derived from git remote)
- **CLI policy_signals**: {policy_signals_json} (from the tier-check CLI output — shows which files exist)
## Your Task
The CLI has already determined which policy files exist in the repository. Your job is to **read and evaluate the content** of the files that were found. Do NOT search for files in other locations — only evaluate what the CLI reported as present.
Three policy areas to evaluate:
1. **Dependency update policy** (required for Tier 1 and Tier 2)
2. **Roadmap** (Tier 1: published roadmap; Tier 2: published plan toward Tier 1)
| 3 | Critical Bug Resolution | All P0s within 7 days | All P0s within 2 weeks | {open P0 count} open | {PASS/FAIL} | {PASS/FAIL} | {detail or "None"} |
| 4 | Stable Release | Required + clear versioning | At least one stable release | {version} | {PASS/FAIL} | {PASS/FAIL} | {detail or "None"} |
| 4b | Spec Tracking | Timeline agreed per release | Within 6 months | {days_gap}d gap ({PASS/FAIL}) | {PASS/FAIL} | {PASS/FAIL} | {detail or "None"} |
| 5 | Documentation | Comprehensive w/ examples | Basic docs for core features | {pass}/{total} features | {PASS/FAIL} | {PASS/FAIL} | {detail or "None"} |
| 6 | Dependency Policy | Published update policy | Published update policy | {Found/Not found} | {PASS/FAIL} | {PASS/FAIL} | {detail or "None"} |
| 7 | Roadmap | Published roadmap | Plan toward Tier 1 | {Found/Not found} | {PASS/FAIL} | {PASS/FAIL} | {detail or "None"} |
| **New Protocol Features** | Before new spec version release, timeline agreed per release based on feature complexity | Within 6 months | No timeline commitment |
| **Issue Triage** | Within 2 business days | Within a month | No requirement |
| **Critical Bug Resolution** | Within 7 days | Within two weeks | No requirement |
| **Stable Release** | Required with clear versioning | At least one stable release | Not required |
| **Documentation** | Comprehensive with examples for all features | Basic documentation covering core features | No minimum |
| **Dependency Policy** | Published update policy | Published update policy | Not required |
| **Roadmap** | Published roadmap | Published plan toward Tier 1 or explanation for remaining Tier 2 | Not required |
## Exact Thresholds for Automated Checking
| Metric | Tier 1 Threshold | Tier 2 Threshold | How to Measure |
| Issue triage time | <= 2 business days | <= 1 month (30 calendar days) | Time from issue creation to first label application |
| P0 resolution time | <= 7 calendar days | <= 14 calendar days | Time from P0 label application to issue close |
| Stable release version | >= 1.0.0, no pre-release suffix | >= 1.0.0 (at least one) | Check `gh release list` for version matching `^[0-9]+\.[0-9]+\.[0-9]+$` where major >= 1 |
| Documentation coverage | All non-experimental features documented with examples | Core features documented | Subagent evaluation |
| Dependency policy | Published and findable in repo | Published and findable in repo | Subagent evaluation |
| Roadmap | Published with concrete steps tracking spec components | Published plan toward Tier 1 | Subagent evaluation |
Every scenario in the conformance suite has a `specVersions` field indicating which spec version it targets. The valid values are defined as the `SpecVersion` type (as a list) in `src/types.ts` — run `node dist/index.js list` to see the current mapping of scenarios to spec versions.
Date-versioned scenarios (e.g. `2025-06-18`, `2025-11-25`) count toward tier scoring. `draft` and `extension` scenarios are listed separately as informational.
The `--spec-version` CLI flag filters scenarios cumulatively for date versions (e.g. `--spec-version 2025-06-18` includes `2025-03-26` + `2025-06-18`). For `draft`/`extension`, it returns exact matches only.
The tier-check output includes a per-version pass rate breakdown alongside the aggregate.
## Tier Relegation Rules
- **Tier 1 to Tier 2**: Any conformance test fails continuously for 4 weeks
- **Tier 2 to Tier 3**: More than 20% of conformance tests fail continuously for 4 weeks
## Issue Triage Label Taxonomy
SDK repositories must use these consistent labels to enable automated reporting.
### Type Labels (pick one)
| Label | Description |
| ------------- | ----------------------------- |
| `bug` | Something isn't working |
| `enhancement` | Request for new feature |
| `question` | Further information requested |
Note: Repositories using GitHub's native issue types satisfy this requirement without needing type labels.
**Total: 12 labels** (3 type + 5 status + 4 priority)
## Key Definitions
### Issue Triage
Labeling and determining whether an issue is valid. This is NOT the same as resolving the issue. An issue is considered triaged when it receives its first label.
### Critical Bug (P0)
- **Security vulnerabilities** with CVSS score >= 7.0 (High or Critical severity)
- **Core functionality failures** that prevent basic MCP operations: connection establishment, message exchange, or use of core primitives (tools, resources, prompts)
### Stable Release
A published version explicitly marked as production-ready. Specifically: version `1.0.0` or higher without pre-release identifiers like `-alpha`, `-beta`, or `-rc`.
### Clear Versioning
Following idiomatic versioning patterns with documented breaking change policies, so users can understand compatibility expectations when upgrading.
### Roadmap
Outlines concrete steps and work items that track implementation of required MCP specification components (non-experimental features and optional capabilities), giving users visibility into upcoming feature support.
Scaffold a sep-NNNN.yaml requirement-traceability file for the MCP
conformance repo from a SEP PR's spec diff. Runs the new-sep CLI, then
parses the modelcontextprotocol/modelcontextprotocol spec diff to populate
`requirements[]` with the RFC 2119 sentences and proposed check IDs.
argument-hint: '<sep-number>'
---
# new-sep: SEP traceability YAML scaffolding
You are bootstrapping a `sep-NNNN.yaml` file for a new SEP in the MCP conformance repo. The output is the requirement-traceability file specified by SEP-2484: a YAML that maps each normative sentence from the SEP's spec diff to a `check:` ID (testable) or an `excluded:` reason (not testable). The CLI gets the skeleton; you fill in the rows by reading the spec diff.
## Step 0: Pre-flight checks
Before doing anything else, verify GitHub CLI authentication:
```bash
gh auth status 2>&1
```
If this fails, stop immediately and tell the user:
> GitHub authentication is required for this skill. Please run `gh auth login` first, then re-run.
Verify you're running inside the conformance repo:
```bash
test -f package.json && jq -r '.name' package.json
```
The name should be `@modelcontextprotocol/conformance`. If not, stop and ask the user to `cd` into the conformance repo first.
## Step 1: Parse arguments
Extract from the user's input:
- **sep-number** (required): the SEP number, e.g. `2164`. This is also the PR number in `modelcontextprotocol/modelcontextprotocol` by convention.
## Step 2: Generate the skeleton
Run the CLI:
```bash
npm run --silent build
node dist/index.js new-sep <NNNN>
```
(For development against a non-built source tree: `npx tsx src/index.ts new-sep ...`.)
The CLI writes `src/seps/sep-<NNNN>.yaml` with `sep`, `spec_url`, and two TODO `requirements[]` rows. Capture the output path from the CLI's `Wrote …` line and remember it as `$YAML`.
If the CLI errors with "does not change any docs/specification/draft/\*.mdx", the SEP's spec changes landed in a separate PR — ask the user for the spec file path and rerun with `--spec-path docs/specification/draft/<path>`. Do not guess.
## Step 3: Fetch the spec diff
`AGENTS.md` (lines 64–72) is explicit that severity must come from the spec text itself, not the SEP markdown or the conformance PR description:
```bash
gh api "repos/modelcontextprotocol/modelcontextprotocol/pulls/<NNNN>/files" \
For each file, pull the added (`+`-prefixed) lines from `patch`. If `patch` is truncated for a large file, fall back to fetching the whole file at the PR's head ref:
```bash
gh api "repos/modelcontextprotocol/modelcontextprotocol/contents/<path>?ref=<sep-branch>" \
--jq '.content' | base64 -d
```
## Step 4: Extract RFC 2119 requirements
Walk the added lines and identify sentences containing the keywords: **MUST**, **MUST NOT**, **SHOULD**, **SHOULD NOT**, **REQUIRED**, **SHALL**, **SHALL NOT**, **MAY**, **OPTIONAL**.
**Quote the whole sentence**, not just the matched line. The matched word may sit inside a bullet point whose lead-in sentence supplies the keyword by inheritance — e.g.:
> Servers SHOULD return standard JSON-RPC errors for common failure cases:
>
> - Resource not found: -32602 (Invalid Params)
The bullet inherits `SHOULD`. The yaml row should quote the _combined_ obligation: `'Servers SHOULD return standard JSON-RPC errors for common failure cases: Resource not found: -32602 (Invalid Params)'` — see `src/seps/sep-2164.yaml` for the canonical example.
**Regex alone is insufficient** (this is called out in Issue #243). Read for context: pronouns, "the server", and "such cases" all refer back to the lead-in.
| MUST / MUST NOT / SHALL / SHALL NOT / REQUIRED | FAILURE | `check: sep-<NNNN>-<slug>` |
| SHOULD / SHOULD NOT | WARNING | `check: sep-<NNNN>-<slug>` |
| MAY / OPTIONAL | — | _no row — skip entirely_ |
MAY / OPTIONAL sentences are noted in Step 4 only so you consciously skip them — they never produce a yaml row.
A row is `excluded:` when a MUST/SHOULD requirement can't be protocol-observed by the harness. Do **not** write any `excluded:` row on your own authority — every exclusion goes through Step 6.
While classifying, sort each MUST/SHOULD row into one of three buckets:
- **`check:`** — observably testable on the wire.
- **clearly-excluded** — you're confident it can't be observed (e.g. "clients SHOULD also accept -32002" when the harness only drives servers).
- **borderline** — you'd default to `check:` but observability is questionable. Markers:
- _Internal state_ — verbs like _record_, _store_, _associate_, _track_, _cache_. The harness sees wire traffic, not memory; usually only observable via a downstream row already in your list.
- _Precondition phrasing_ — "Before doing X, the implementation MUST Y" where X is itself another row.
Slug convention: lowercase-kebab, derived from the verb phrase. Examples from `sep-2164.yaml`: `no-empty-contents`, `error-code`. Same `id` is used for SUCCESS and FAILURE (`AGENTS.md:52`).
## Step 6: Confirm exclusions with the user
Nothing becomes `excluded:` without sign-off. Two rounds:
**Round 1 — clearly-excluded, single batch question.** One `AskUserQuestion` listing all clearly-excluded rows in the question body (slug + one-line reason each). Options:
- `Exclude all as listed (Recommended)`
- `Flip all to check:`
- `Let me adjust per-row` — if chosen, append these rows to round 2.
Skip this round if the bucket is empty.
**Round 2 — borderline, one question per row.** One `AskUserQuestion` call with a question per borderline row (loop in batches of 4 if needed). For each:
- header: the proposed slug
- question: quote the requirement sentence + your one-line observability concern
- options (list `check:` first — it's the default for borderline):
- `check:` — keep as a testable check
- `excluded: <reason>` — drop to excluded with your stated reason
- `merge into <other-slug>` — offer when the row is a precondition for another row already in the list
Apply the answers before writing. For any `excluded:` outcome, write the reason verbatim into the yaml and add an `issue:` URL if the user supplies one. A `merge` outcome means: drop this row, and append its `text:` to the surviving row's `text:` separated by `/` so the traceability isn't lost.
## Step 7: Rewrite the YAML
Replace the two TODO rows the CLI generated with one row per extracted requirement. Preserve the CLI's quoting style (single quotes, two-space indent — see `src/seps/sep-2164.yaml`).
**Key order within each row** — for `check:` rows the **`check:` key comes first**, then `text:`, then any optional `url:`. Scanning the left margin should reveal every check ID without reading the quoted sentences. For `excluded:` rows the order is **`text:` first**, then `excluded:`, then optional `issue:` — there's no ID to scan for, so lead with the requirement.
**Row order in the file** — all `check:` rows first (in spec-diff order), then **all `excluded:` rows grouped at the bottom**, separated from the checks by **one blank line**. Do not interleave.
If a requirement is ambiguous or you're not confident, leave it as a `TODO:` row rather than guessing — humans review this yaml before scenarios get written.
Also fix the `spec_url`: the CLI emits the page URL with no anchor. If the requirements you extracted live under a specific spec subsection (e.g. `#error-handling`), append it.
If a requirement comes from a **different spec page** than `spec_url` (the SEP touched multiple `.mdx` files — the CLI prints these as "PR also changes N other spec file(s)"), give that row a full `url:` override:
A row's effective spec reference is `row.url ?? file.spec_url`.
Write the result back to `$YAML`.
## Step 8: Suggest a host scenario
`AGENTS.md` prefers **fewer scenarios with more checks** over one-scenario-per-check. Before telling the user to write a new scenario, look for an existing one the new checks could be folded into.
Determine the suite directory from the requirement subjects ("MCP clients MUST…" → `client/`, "Servers MUST…" → `server/`, "authorization servers MUST…" → `authorization-server/`; a SEP may map to more than one). Then search that directory for scenarios touching the same spec area:
Pick 2–3 domain terms from the SEP's subject matter (for a discovery SEP: `metadata`, `well-known`; for an auth-response SEP: `redirect`, `callback`, `pkce`). For each hit, pull the scenario's `name`/`description` to confirm relevance:
```bash
rg -A1 'name:|description:' <hit.ts>
```
If you find a plausible host, recommend it by path. If nothing fits, say so explicitly — a new scenario file is then the right call.
## Step 9: Hand-off
Report to the user, in this order:
1. Path to the generated yaml.
2. Row counts: "`N check:` rows, `M excluded:` rows" — and note which exclusions the user signed off in Step 6.
3. Any requirements you left as `TODO:` and why.
4. **Host-scenario recommendation** from Step 8 — either "consider adding these checks to `src/scenarios/<suite>/<file>.ts` (it already exercises _X_)" or "no existing scenario covers this area; a new file is appropriate".
5. Remaining next steps the user owns:
- add the checks to the host scenario (or create one) under `src/scenarios/{client,server,authorization-server}/`,
- register any new scenario in `src/scenarios/index.ts` (`AGENTS.md:48`),
- add a passing example to the everything-client/server and a negative test, per `AGENTS.md:74-81`.
Do **not** generate or edit scenario `.ts` files or touch `src/scenarios/index.ts`. The skill's scope ends at the yaml plus the recommendation.
description: Use when reviewing a conformance PR that adds or changes scenario .ts files for a SEP — before approving, before requesting changes, or as a self-check before opening one.
---
# review-scenario
## What to check
**Spec diff is ground truth.** Pull the SEP's actual spec changes and read the RFC-2119 sentences yourself — don't trust the PR description or SEP summary for keyword levels:
```bash
gh api "repos/modelcontextprotocol/modelcontextprotocol/pulls/<SEP>/files" \
If the SEP includes a conformance-test-case table, that table is authoritative for the cases it lists. A table/prose mismatch is a spec gap to flag, not something to silently resolve either way.
**Traceability YAML.** `src/seps/sep-<SEP>.yaml` should exist (run `/new-sep <SEP>` first if not). Diff its rows against the spec sentences you extracted; flag rows that paraphrase rather than quote, claim a keyword level the spec doesn't, or assert something the spec never says. Check IDs follow `sep-<NNNN>-<kebab-slug>`.
**Per-scenario-file:**
- **Spec backing** — would a fully spec-compliant implementation FAIL this check? If yes — or if two compliant SDKs in different languages would get different results — the spec hasn't pinned the behavior; note it as a gap rather than enforce it.
- **Dead checks** — emits FAILURE with no reachable SUCCESS counterpart, or sits behind an always-false guard.
- **Logic** — does a missing/malformed input silently pass? Does the assertion distinguish "rejected for the right reason" from "rejected at all"?
**Coverage.** Count YAML `check:` rows vs how many the PR's scenarios actually exercise; list the gaps.
**Proof it runs.** The PR should reference at least one real implementation the scenario ran green against — the in-repo everything-client/server, or an external SDK via `npx https://pkg.pr.new/@modelcontextprotocol/conformance@<PR>`. No run referenced → ask for one before approving.
**Negative test.** Pins the specific failing slugs, not just `failures.length > 0` (AGENTS.md §Examples: prove it passes and fails).
## Output
This is a first pass for a human reviewer — give them what they need to verify each finding without re-deriving it.
**Open with a summary:** N scenarios added/changed, M distinct check IDs emitted, X/Y YAML `check:` rows covered, and which implementation it was run against.
**Then one bullet per finding.** Each bullet makes its own case — the reviewer should be able to confirm or refute it from the bullet alone:
> **`<check-id>`** — [`file.ts:Lnn`](https://github.com/modelcontextprotocol/conformance/blob/<HEAD-SHA>/path/file.ts#Lnn) — claim. Spec: _"quoted normative sentence"_ ([page#anchor](https://modelcontextprotocol.io/specification/draft/...#anchor)). Consequence: what a compliant impl would do and how this check would mis-report it.
e.g.
> **`client-consistent-version`** — [`stateless.ts:86`](…/blob/abc123/src/scenarios/client/stateless.ts#L86) — no spec backing. Spec: _"Servers MUST NOT rely on prior requests over the same connection to establish context (e.g., capabilities, protocol version)"_ ([lifecycle#stateless](…)). A compliant client may change `protocolVersion` per request; this check FAILs it. The `flippingVersionClient` negative test enforces a non-requirement.
Get `<HEAD-SHA>` once via `gh pr view <PR> --json headRefOid -q .headRefOid` and use it for all permalinks so they don't drift on rebase.
Order: spec-backing → logic/dead → coverage (gap list) → conventions. Put spec gaps in a separate trailing list — those go upstream, not to the PR author.
**Self-review:** fix in place and re-run.
**If asked to push fixes** (stacked diff on top of the PR head): one commit per finding, commit message is the finding. Leave design-level items (scenario count, refactors) as prose.
Guidance for AI agents (and humans) contributing to the MCP conformance test framework.
## What this repo is
A test harness that exercises MCP SDK implementations against the protocol spec. The coverage number that matters here is **spec coverage** — how much of the protocol the scenarios test.
Uses **npm** (not pnpm/yarn). Don't commit `pnpm-lock.yaml` or `yarn.lock`.
## Where to start
**Open an issue first** — whether you've hit a bug in the harness or want to propose a new scenario. For scenarios, sketch which part of the spec you want to cover and roughly how; for bugs, include the command you ran and the output. Either way, a short discussion up front beats review churn on a PR that overlaps existing work or heads in a direction we're not going.
**Don't point an agent at the repo and ask it to "find bugs."** Generic bug-hunting on a test harness produces low-signal PRs (typo fixes, unused-variable cleanups, speculative refactors). If you want to contribute via an agent, give it a concrete target:
- Pick a specific MUST or SHOULD from the [MCP spec](https://modelcontextprotocol.io/specification/) that has no scenario yet, and ask the agent to draft one.
- Pick an [open issue](https://github.com/modelcontextprotocol/conformance/issues) and work on that.
The valuable contribution here is **spec coverage**, not harness polish.
## Scenario design: fewer scenarios, more checks
**The strongest rule in this repo:** prefer one scenario with many checks over many scenarios with one check each.
Why:
- Each scenario often spins up its own HTTP server. These suites run in CI on every push for every SDK, so per-scenario overhead multiplies fast.
- Less code to maintain and update when the spec shifts.
- Progress on making an SDK better shows up as "pass 7/10 checks" rather than "pass 1 test, fail another" — finer-grained signal from the same run.
### Granularity heuristic
Ask: **"Would it make sense for someone to implement a server/client that does just this scenario?"**
If two scenarios would always be implemented together, merge them. Examples:
- `tools/list` + a simple `tools/call` → one scenario
- All content-type variants (image, audio, mixed, resource) → one scenario
- Full OAuth flow with token refresh → one scenario, not separate "basic" + "refresh" scenarios. A client that passes "basic" but not "refresh" just shows up as passing N−2 checks.
Keep scenarios separate when they're genuinely independent features or when they're mutually exclusive (e.g., an SDK should support writing a server that _doesn't_ implement certain stateful features).
### When a PR adds scenarios
- Start with **one end-to-end scenario** covering the happy path with many checks along the way.
- Don't add "step 1 only" and "step 1+2" as separate scenarios — the second subsumes the first.
- Register the scenario in the appropriate suite list in `src/scenarios/index.ts` (`core`, `extensions`, `backcompat`, etc.).
## Check conventions
- **Same `id` for SUCCESS and FAIL.** A check should use one slug and flip `status` + `errorMessage`, not branch into `foo-success` vs `foo-failure` slugs.
- **Optimize for Ctrl+F on the slug.** Repetitive check blocks are fine — easier to find the failing one than to unwind a clever helper.
- Reuse `ConformanceCheck` and other types from `src/types.ts` rather than defining parallel shapes.
- **Don't reimplement the runner.** New subcommands that need to "select scenarios → run them → print summary → compute exit code" must go through the existing `client` / `server` commands (subprocess via `process.execPath` like `tier-check` and `sdk` do) or call shared helpers — never a parallel suite-map / summary loop.
- Include `specReferences` pointing to the relevant spec section.
- **Severity follows the spec keyword:** MUST / MUST NOT → `FAILURE`; SHOULD / SHOULD NOT → `WARNING`. (CI treats WARNING as a failure, so Tier-1 SDKs still need to satisfy SHOULDs — see #245.)
- **A missing prerequisite is a failure, not a skip** (#248). When a check cannot be exercised — the server under test lacks a diagnostic fixture tool, rejects the probe, or advertises a feature it does not serve — report it failed via `notTestable()` / `untestableCheck()` from `src/scenarios/untestable.ts`, with the missing prerequisite named in `errorMessage` and `details.untestable: true`. SKIPPED reads as green in every consumer (pass counts, exit codes, expected-failures), so it is reserved for checks that are legitimately not applicable: an optional capability the server never declared, spec-version inapplicability handled by the runner, or a documented harness gap the suite itself tracks (e.g. `tasks-status-notifications` pending its subscriptions/listen rewrite). The untestable severity follows the underlying requirement's keyword (MUST → `FAILURE`, SHOULD → `WARNING`); the expected-failures baseline is the escape hatch for SDKs that haven't built the fixture yet.
## Descriptions and wording
Be precise about what's **required** vs **optional**. A scenario description that tests optional behavior should make that clear — e.g. "Tests that a client _that wants a refresh token_ handles offline_access scope…" not "Tests that a client handles offline_access scope…". Don't accidentally promote a MAY/SHOULD to a MUST in the prose.
When in doubt about spec details (OAuth parameters, audiences, grant types), check the actual spec in `modelcontextprotocol` rather than guessing.
## Reviewing PRs
### SEP scenarios
Verify requirement levels against the SEP's **spec diff** — the change to `docs/specification/draft/` in the SEP's PR — not the SEP markdown summary or the conformance PR's description. The keyword that governs check severity is the one in the spec text; a bullet under a "Servers SHOULD…" sentence is SHOULD-level even if the SEP's title says "standardize."
```sh
gh api "repos/modelcontextprotocol/modelcontextprotocol/contents/docs/specification/draft/<path>?ref=<sep-branch>" --jq '.content' | base64 -d
The command looks up PR #`<NNNN>` in `modelcontextprotocol/modelcontextprotocol` (SEP numbers are PR numbers), derives `spec_url` from the `docs/specification/draft/*.mdx` file it changes, and writes `src/seps/sep-<NNNN>.yaml` with TODO `requirements[]` rows. Use `--spec-path` or `--spec-url` to skip the lookup. The `new-sep` Claude Code skill drives the same flow end-to-end, parses the spec diff, and fills in the requirement rows.
### Traceability manifest
`src/seps/traceability.json` is a generated map of, per SEP, which declared `check:` IDs are actually emitted when the conformance suite runs against the reference SDK. It is consumed by plan.modelcontextprotocol.io to track SEP-2484 progress.
The emitted check IDs come from a real suite run (not a source scan), so dynamic (template-literal) IDs resolve to their concrete values. Generate the manifest from a results directory:
```sh
# 1. Run the suite against the reference SDK, collecting checks.json files:
node dist/index.js server --url '<sdk conformance server url>' --suite all -o results
# 2. Build the manifest from those results:
npm run traceability -- --results results
npm run traceability -- --results results --strict # exit 1 on any untested (advisory)
```
Manifest shape: `{ schemaVersion, docs, source, seps }`, where `seps` is keyed by SEP number. Each requirement is `tested` (its check ID was emitted) or `untested` (declared but never emitted — a real gap, or a check that only fires against a deliberately-broken impl, i.e. it needs a negative test). `"tested" means a scenario emitted the check ID, NOT that any SDK passes it` — per-SDK results live in `tier-check`. Matching is exact, so a scenario's emitted check IDs must match the requirement slugs in the yaml (one check ID per MUST/SHOULD, emitted once per case). `source` records what was run against (e.g. `typescript-sdk@<sha>`); the `docs` field points back here.
Contract for consumers (plan.mcp.io): a SEP appears only if it has a traceability yaml or emits `sep-NNNN-*` check IDs. **A SEP absent from the manifest has no conformance artifacts — treat it as not-started** (diff against your own SEP list to find them). `untracked` lists emitted IDs with no yaml row (usually scenario gates).
The manifest is refreshed by `.github/workflows/traceability.yml` (manual/scheduled), which runs the suite against typescript-sdk and opens a PR with the diff — it is **not** a PR gate. Untested checks are advisory for now; the intended future policy is that an untested check must be backed by a negative test.
## Examples: prove it passes and fails
A new scenario should come with:
1. **A passing example** — usually by extending `examples/clients/typescript/everything-client.ts` or the everything-server, not a new file.
2. **A negative test** — a deliberately-broken implementation in `examples/{clients,servers}/typescript/` plus a vitest case asserting the check emits `FAILURE`/`WARNING` against it. See `src/scenarios/client/auth/index.test.ts` and `src/scenarios/server/negative.test.ts` for the pattern. A passing run against the everything-server proves the check doesn't false-positive, but not that it catches anything.
Delete unused example scenarios. If a scenario key in the everything-client has no corresponding test, remove it.
## Don't add new ways to run tests
Use the existing CLI runner (`npx @modelcontextprotocol/conformance client|server ...`). If you need a feature the runner doesn't have, add it to the runner rather than building a parallel entry point.
## Before opening a PR
- `npm run build` passes
- `npm test` passes
- For non-trivial scenario changes, run against at least one real SDK (typescript-sdk or python-sdk) to see actual output. For changes to shared infrastructure (runner, tier-check), test against go-sdk or csharp-sdk too.
- Scenario is registered in the right suite in `src/scenarios/index.ts`
- If you changed a `sep-*.yaml` or scenario check IDs, `src/seps/traceability.json` will drift; the traceability workflow refreshes it via PR (or regenerate locally with `--results` from a suite run)
Thanks for helping improve the MCP conformance suite!
The most valuable contributions are **new conformance scenarios** that cover under-tested parts of the [MCP spec](https://modelcontextprotocol.io/specification/). If you're not sure where to start, ask in `#conformance-testing-wg` on the MCP Contributors Discord.
## Before you start
**Open an issue first** — whether you've found a bug or want to propose a new scenario. A short discussion up front saves everyone time on PRs that overlap existing work or head in a direction we're not going.
Then read **[AGENTS.md](./AGENTS.md)** — it's the design guide for scenarios and checks. The short version:
- **Fewer scenarios, more checks.** Each scenario spins up its own server and runs in CI for every SDK. One scenario with 10 checks beats 10 scenarios with one check each.
- **Prove it passes and fails.** Extend the existing everything-client/server to pass your scenario, and show (or include) a failing case.
- **Reuse the CLI runner.** Don't add parallel entry points.
If you're using an AI agent to help, please **don't** point it at the repo with a generic "find bugs" prompt — give it a specific MUST from the spec or an open issue to work on. See AGENTS.md for details.
## Setup
```sh
npm install
npm run build
npm test
```
This repo uses **npm** — don't commit `pnpm-lock.yaml` or `yarn.lock`.
## Running your scenario
Run a client scenario against the bundled example client:
A framework for testing MCP (Model Context Protocol) client and server implementations against the specification.
> [!WARNING]
> This repository is a work in progress and is unstable. Join the conversation in the #conformance-testing-wg in the MCP Contributors discord.
**For SDK maintainers:** See [SDK Integration Guide](./SDK_INTEGRATION.md) for a streamlined guide on integrating conformance tests into your SDK repository.
- `--command` - The command to run your MCP client (can include flags)
- `--scenario` - The test scenario to run (e.g., "initialize")
- `--suite` - Run a suite of tests in parallel (e.g., "auth")
- `--suite` - Run a suite of tests in parallel: `all`, `core`, `extensions`, `backcompat`, `auth`, `metadata`, `draft` (scenarios targeting the in-progress draft spec), or `sep-835`
- `--spec-version <version>` - Filter scenarios by spec version (e.g., `2025-11-25`, `2026-07-28`; `draft` is accepted as an alias for the current draft identifier). The draft version selects the latest dated release plus any draft-only scenarios. When omitted, the version is inferred from the scenario's spec applicability (draft-only scenarios run at the draft version, everything else at the latest dated release); an explicitly requested version outside a scenario's applicability window skips the scenario (exit 0) unless `--force` is passed
- `--force` - Run a scenario even if it is not applicable at the requested `--spec-version`
- `--requirements <revision>` - Run exactly what a spec revision requires, frozen at its release (see [Conformance Requirements](#conformance-requirements))
- `--expected-failures <path>` - Path to YAML baseline file of known failures (see [Expected Failures](#expected-failures))
- `--timeout` - Timeout in milliseconds (default: 30000)
- `--verbose` - Show verbose output
The framework appends `<server-url>` as an argument to your command and sets the `MCP_CONFORMANCE_SCENARIO` environment variable to the scenario name. For scenarios that require additional context (e.g., client credentials), the `MCP_CONFORMANCE_CONTEXT` environment variable contains a JSON object with scenario-specific data.
The framework appends `<server-url>` as an argument to your command and sets the `MCP_CONFORMANCE_SCENARIO` environment variable to the scenario name. For scenarios that require additional context (e.g., client credentials), the `MCP_CONFORMANCE_CONTEXT` environment variable contains a JSON object with scenario-specific data. When `--spec-version` is passed, its resolved value is forwarded to the client process as `MCP_CONFORMANCE_PROTOCOL_VERSION`; example clients can use this value directly as their `protocolVersion`. SDKs that hard-code their protocol version can ignore it. Clients under test must derive the lifecycle from the protocol version they are asked to run: dated versions through `2025-11-25` use the stateful lifecycle (initialize handshake), while the 2026 draft (`2026-07-28`) uses the stateless lifecycle (per-request `_meta`).
- `--scenario <scenario>` - Test scenario to run (e.g., "server-initialize"). Runs all available scenarios by default
- `--suite <suite>` - Suite to run: "active" (default), "all", or "pending"
- `--suite <suite>` - Suite to run: "active" (default; excludes pending and draft-spec scenarios), "all", "draft" (scenarios targeting the in-progress draft spec), or "pending"
- `--requirements <revision>` - Run exactly what a spec revision requires, frozen at its release (see [Conformance Requirements](#conformance-requirements))
- `--expected-failures <path>` - Path to YAML baseline file of known failures (see [Expected Failures](#expected-failures))
| `extension` | Optional by definition. SEP-1730: "Experimental features and protocol extensions (such as Tasks and MCP Apps) are not required for any tier." |
| `added-after-release` | The scenario did not exist when the revision shipped, so no implementation could have been passing it. |
| `pending` | The suite's own reference fixture cannot pass it yet, so it cannot be required — but the implementation under test may pass it, so it runs for visibility. |
Both still run, so a failing extension stays visible in the report; neither moves
the pass rate. Promoting an entry into the required lists is a deliberate,
reviewable change, which is how the suite grows without retroactively failing
anyone.
This is the project's contract and lives in this repository. It is the opposite
of an [expected-failures](#expected-failures) baseline, which lives in an
implementation's own repository and records what that implementation knows it
fails. A baselined failure is still a failure against a requirement set.
## Expected Failures
SDKs that don't yet pass all conformance tests can specify a baseline of known failures. This allows running conformance tests in CI without failing, while still catching regressions.
@ -135,6 +222,40 @@ This ensures:
- CI fails on new regressions (unexpected failures)
- CI fails when a fix lands but the baseline isn't updated (stale entries)
### Baselining a single check
A scenario is many checks — `server-stateless` alone is over twenty — so baselining the
whole scenario to excuse one of them stops enforcing the other nineteen. An entry can
instead name a single check, as `<scenario>:<check-id>`:
```yaml
server:
- tasks-lifecycle # whole scenario may fail
- server-stateless:sep-2575-server-implements-discover # only this check may fail
```
The check id is the left-hand column the runner already prints for each check, so it can
be copied straight out of a failing run.
With a per-check entry, every failing check in that scenario is judged on its own: the
named one is excused, and any other failure is still an unexpected regression. The four
exit-code rules above apply per check rather than per scenario, with one addition — a
baselined check that is absent or skipped is tolerated, because a scenario that bails on
a failed prerequisite legitimately never reaches its later checks, and the prerequisite
reports its own failure anyway.
Two things to know:
- **A check id addresses every occurrence of that id.** Ids repeat within a run (a loop,
a retried flow), and the occurrences collapse to one verdict, most-severe first. So
baselining a repeated id excuses all of its occurrences — coarser than ideal, still far
narrower than baselining the scenario.
- **Mind the space.**`- scenario:check-id` is a string; `- scenario: check-id` is YAML
for a mapping and is rejected with an error.
A scenario cannot be listed both wholesale and per-check — the wholesale entry already
excuses everything, so the pair is contradictory and is rejected.
## GitHub Action
This repo provides a composite GitHub Action so SDK repos don't need to write their own conformance scripts.
@ -212,6 +333,150 @@ Run `npx @modelcontextprotocol/conformance list --server` to see all available s
The `sdk` subcommand clones an SDK repository at a given ref, builds it, and runs the **local** conformance build against it. This is the inner-loop tool for scenario authors and the basis for cross-SDK CI. Examples below use `npm start --` so they run from source — no `npm run build` between edits.
`--mode client` or `--mode server` is required — each invocation tests exactly one side, so client and server are run (and pass/fail) independently.
```bash
# Run the client conformance suite against typescript-sdk @main (v2)
npm start -- sdk typescript-sdk --mode client
# Run the server conformance suite (separate invocation)
npm start -- sdk typescript-sdk --mode server
# A specific main-line SHA or branch (v2 monorepo)
Build/run commands for each official SDK are looked up by name from [`src/sdk-runner/known-sdks.ts`](src/sdk-runner/known-sdks.ts) — no config file is required in the SDK repo. Resolution order is **CLI flag > built-in entry**, so any field can be overridden on the command line for refs that diverge from the built-in.
An SDK can have more than one entry when its layout differs across major versions — e.g. `typescript-sdk` (v2, the `main` monorepo) and `typescript-sdk-v1` (the published npm v1.x line). An entry may set `defaultRef` (the branch used when you don't pass `@<ref>`) and `repo` (the real clone target when the entry name is an alias).
When the right invocation depends on the spec version being targeted, the entry carries it in `specOverrides` instead of a comment to copy from. The matching entry is merged over the base config when you pass `--spec-version` (or when the entry's own `specVersion` default applies), so version-specific runs need no extra flags:
```bash
# go-sdk's server pins -stateless=false for the dated-spec suites; its
# specOverrides['2026-07-28'] entry swaps in the stateless invocation, so
# this is the whole command:
npm start -- sdk go-sdk --mode server --suite all --spec-version 2026-07-28
# same for csharp-sdk (stateless URL), rust-sdk (STATELESS=1 env), and
To add a new SDK to the matrix, add an entry to `KNOWN_SDKS`.
Clones are cached under `.sdk-under-test/` and reused (fetched) on subsequent runs.
## SDK Tier Assessment
The `tier-check` subcommand evaluates an MCP SDK repository against [SEP-1730](https://github.com/modelcontextprotocol/modelcontextprotocol/issues/1730) (the SDK Tiering System). There are two ways to run it, and they answer different questions.
### 1. The CLI, for the deterministic half
Conformance pass rates, issue triage, P0 resolution, labels, releases, policy files. No AI, no judgment, reproducible.
For a known SDK (see `src/sdk-runner/known-sdks.ts`), one command does the whole
thing — clone, build, and for **each** shipped revision start the server
invocation that SDK's config declares for that revision, run both legs at that
--client-cmd '<command that runs the SDK conformance client>' \
--requirements 2025-11-25,2026-07-28
```
Omit `--client-cmd` and the client leg is skipped and reported as a gap. Omit
`--requirements` and scoring falls back to the suite as it stands today, which is
not what you want for a tier claim; see [Conformance Requirements](#conformance-requirements).
The exit code is a CI verdict on the machine-checkable half: nonzero when any
scored conformance scenario fails or a leg could not be measured. Governance
findings (triage, P0s, policy files) shape the tier but never the exit code.
### 2. The skill, for the whole assessment
The CLI cannot judge documentation coverage, dependency policy or roadmap quality, and those decide the tier as much as conformance does. The [`mcp-sdk-tier-audit`](.claude/skills/mcp-sdk-tier-audit/README.md) skill runs the CLI, adds those evaluations, and writes a full report with a remediation plan. In Claude Code, from a checkout of this repo:
- **Required scenarios** are the only ones that move the number. `67/67` spans every revision listed: a scenario belonging to both runs once per revision, on that revision's wire, and both have to pass.
- **Not scored** ran and is reported so you can see it, but cannot fail the tier. `extension` means optional by definition; `added-after-release` means the scenario did not exist when that revision shipped. A failure here is information, not a blocker.
- **Not measured** is different from either, and means the run could not happen at all, e.g. a requirement set naming a scenario this build no longer has. Treat it as a broken invocation, never as an SDK failure.
- **Tier 1 blockers** lists every requirement short of Tier 1. Conformance absent from that list means the SDK met every requirement each listed revision imposes.
The exit code follows the same rule: it reflects required scenarios only, so an implementation that meets a revision's requirements exits 0 even with failing extensions.
In `--output json`, `passed` / `failed` / `total` describe the scored set and so always agree with `pass_rate`; anything run without being scored is counted separately under `not_scored`.
## Architecture
See `src/runner/DESIGN.md` for detailed architecture documentation.
