modelcontextprotocol/conformance
Watch
1
0
Fork
You've already forked conformance
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Freeze conformance requirements per spec revision, run each at its wire (#447)

Browse source 
There is no way to answer "which scenarios must my implementation pass
to conform to spec revision X". The suite accumulates scenarios
continuously, so what an implementation is measured against keeps
growing after the revision shipped, and it can drop below 100% while
standing still. --spec-version does not answer it: it filters on which
revision a scenario targets, not on when the scenario became a
requirement.

Add requirements/<revision>.yaml naming the scenarios a revision
requires, and --requirements <revision> on the client, server, list and
sdk commands to run exactly that set. The revision also fixes the wire
version its scenarios speak: the dated revisions through 2025-11-25 use
the stateful initialize handshake and 2026-07-28 is stateless with
per-request _meta, so a scenario belonging to two revisions runs once
under each and one run does not cover the other. Passing no flag keeps
today's behaviour of running everything.

tier-check gains --sdk <name[@ref]> / --sdk-path <dir>: for each claimed
revision it resolves that revision's build, server invocation and client
command from the sdk-runner's KNOWN_SDKS specOverrides, manages the
server, runs both legs with cwd = the SDK checkout, and merges. Every
claimed revision must pass for Tier 1. This is what makes multi-revision
tiering correct for SDKs whose wire era is fixed per server process
(go-sdk only speaks >= 2026-07-28 when constructed stateless) or per
endpoint (csharp-sdk serves it at /stateless): one
--conformance-server-url cannot describe them, and their config already
does. The single-URL mode remains for SDKs not in the config.

Ship two sets. 2026-07-28 is a snapshot of 0.2.0-alpha.10, published the
day before that revision shipped. 2025-11-25 is a reconstruction from
the same anchor and its header says so: the release current on its ship
date predates spec-version awareness entirely. not_scored entries run
and are reported with a reason (extension per SEP-1730, or
added-after-release) but never affect a pass rate or the exit code, and
--expected-failures judges scored scenarios only. Reports label every
entry with its revision, and JSON passed/failed/total describe the
scored set so they always agree with pass_rate.

tier-check's exit code now gates on the machine-checkable half: nonzero
when a scored scenario fails or a leg could not be measured. A child run
that cannot happen at all (a set naming a renamed scenario, an
unreachable server) reports as not measured rather than as 0% against
the implementation. Conformance runs to completion before the GitHub
checks, so an API failure can no longer interrupt a leg and orphan a
managed server, and the legs spawn their child asynchronously: the
previous synchronous exec blocked the event loop, the parent stopped
draining the managed server's log pipes, and a chatty server (rust-sdk's
per-request tracing) froze on a full pipe and was scored as failing.

Driving every tier-1 SDK through the new path also fixed three configs:
csharp-sdk's server now launches from its build-output directory so
ASP.NET finds appsettings.json (its AllowedHosts filter, which
dns-rebinding-protection tests, silently never loaded from the repo
root); rust-sdk's STATELESS=1 override is removed because its CI runs
one default server for both revisions and the forced mode fails its
SEP-2575 scenarios; python-sdk's build mirrors its CI's two targeted
syncs because --all-packages is unbuildable on main. withManagedServer
now waits for its child to exit before returning, so consecutive runs on
one port cannot race a dying predecessor.
This commit is contained in:
Felix Weinberger 2026-08-07 14:55:10 +01:00 • committed by GitHub
commit 232a901445
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 2135 additions and 356 deletions
Download patch file Download diff file Expand all files Collapse all files

148
README.md
View file

@ -66,6 +66,7 @@ npx @modelcontextprotocol/conformance client --command "<client-command>" --scen
- `--suite` - Run a suite of tests in parallel: `all`, `core`, `extensions`, `backcompat`, `auth`, `metadata`, `draft` (scenarios targeting the in-progress draft spec), or `sep-835`
- `--spec-version <version>` - Filter scenarios by spec version (e.g., `2025-11-25`, `2026-07-28`; `draft` is accepted as an alias for the current draft identifier). The draft version selects the latest dated release plus any draft-only scenarios. When omitted, the version is inferred from the scenario's spec applicability (draft-only scenarios run at the draft version, everything else at the latest dated release); an explicitly requested version outside a scenario's applicability window skips the scenario (exit 0) unless `--force` is passed
- `--force` - Run a scenario even if it is not applicable at the requested `--spec-version`
- `--requirements <revision>` - Run exactly what a spec revision requires, frozen at its release (see [Conformance Requirements](#conformance-requirements))
- `--expected-failures <path>` - Path to YAML baseline file of known failures (see [Expected Failures](#expected-failures))
- `--timeout` - Timeout in milliseconds (default: 30000)
- `--verbose` - Show verbose output
@ -83,6 +84,7 @@ npx @modelcontextprotocol/conformance server --url <url> [--scenario <scenario>]
- `--url` - URL of the server to test
- `--scenario <scenario>` - Test scenario to run (e.g., "server-initialize"). Runs all available scenarios by default
- `--suite <suite>` - Suite to run: "active" (default; excludes pending and draft-spec scenarios), "all", "draft" (scenarios targeting the in-progress draft spec), or "pending"
- `--requirements <revision>` - Run exactly what a spec revision requires, frozen at its release (see [Conformance Requirements](#conformance-requirements))
- `--expected-failures <path>` - Path to YAML baseline file of known failures (see [Expected Failures](#expected-failures))
- `--verbose` - Show verbose output
@ -116,6 +118,74 @@ Scenarios that exchange no instrumented wire traffic (see issue #418) emit
neither check. Like any other check, `wire-schema-valid` can be baselined via
the expected-failures file.
## Conformance Requirements
`--suite` and `--spec-version` describe the suite as it is today. Neither answers
"which scenarios did I need to pass to conform to the spec released on
2026-07-28", because the suite keeps growing: a scenario merged after a revision
ships still carries that revision's applicability tag, so it is
indistinguishable from one that existed at release.
A requirement set answers that question. Each `requirements/<revision>.yaml`
names the scenarios a revision requires, for the two roles the specification
defines: an MCP server acting as an OAuth resource server, and an MCP client
acting as an OAuth client. It deliberately covers no authorization-server
scenarios, because the specification puts authorization server implementation
beyond its own scope, so those scenarios serve people deploying an authorization
server rather than implementations of MCP itself.
**Scenarios run at their revision's wire version.** That is the point of a
per-revision set, not a label on it: the dated revisions through `2025-11-25`
use the stateful initialize handshake and `2026-07-28` is stateless with
per-request `_meta`, and a scenario emits different checks under each. A
scenario belonging to both revisions must therefore be run twice, once under
each set. Passing it on one wire says nothing about the other:
```bash
# what does conforming to 2026-07-28 actually require?
npx @modelcontextprotocol/conformance list --requirements 2026-07-28
# run exactly that
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --requirements 2026-07-28
```
`--requirements` replaces `--suite`, `--spec-version` and `--scenario`, since the
set already names every scenario that runs and the revision fixes the wire they
run at. Without it nothing changes: the default is still to run everything, which
is where completeness lives.
`tier-check` takes several at once, and every one of them must pass for Tier 1:
```bash
npx @modelcontextprotocol/conformance tier-check --repo <owner/repo> \
--conformance-server-url http://localhost:3000/mcp \
--requirements 2025-11-25,2026-07-28
```
Only scenarios a revision actually requires decide the exit code and the pass
rate. Anything run without being scored is reported separately and cannot fail
the run.
Requirement sets are frozen, and `not_scored` holds what a revision runs and
reports without counting. Two reasons qualify, and the report names which
applies:
| Reason | Meaning |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `extension` | Optional by definition. SEP-1730: "Experimental features and protocol extensions (such as Tasks and MCP Apps) are not required for any tier." |
| `added-after-release` | The scenario did not exist when the revision shipped, so no implementation could have been passing it. |
| `pending` | The suite's own reference fixture cannot pass it yet, so it cannot be required — but the implementation under test may pass it, so it runs for visibility. |
Both still run, so a failing extension stays visible in the report; neither moves
the pass rate. Promoting an entry into the required lists is a deliberate,
reviewable change, which is how the suite grows without retroactively failing
anyone.
This is the project's contract and lives in this repository. It is the opposite
of an [expected-failures](#expected-failures) baseline, which lives in an
implementation's own repository and records what that implementation knows it
fails. A baselined failure is still a failure against a requirement set.
## Expected Failures
SDKs that don't yet pass all conformance tests can specify a baseline of known failures. This allows running conformance tests in CI without failing, while still catching regressions.
@ -326,26 +396,86 @@ Clones are cached under `.sdk-under-test/` and reused (fetched) on subsequent ru
## SDK Tier Assessment
The `tier-check` subcommand evaluates an MCP SDK repository against [SEP-1730](https://github.com/modelcontextprotocol/modelcontextprotocol/issues/1730) (the SDK Tiering System):
The `tier-check` subcommand evaluates an MCP SDK repository against [SEP-1730](https://github.com/modelcontextprotocol/modelcontextprotocol/issues/1730) (the SDK Tiering System). There are two ways to run it, and they answer different questions.
### 1. The CLI, for the deterministic half
Conformance pass rates, issue triage, P0 resolution, labels, releases, policy files. No AI, no judgment, reproducible.
For a known SDK (see `src/sdk-runner/known-sdks.ts`), one command does the whole
thing — clone, build, and for **each** shipped revision start the server
invocation that SDK's config declares for that revision, run both legs at that
revision's wire, and merge:
```bash
# Without conformance tests (fastest)
gh auth login
npm run --silent tier-check -- --repo modelcontextprotocol/typescript-sdk --skip-conformance
npx @modelcontextprotocol/conformance tier-check --sdk go-sdk
# or against a checkout you already have:
npx @modelcontextprotocol/conformance tier-check --sdk-path ~/src/go-sdk
```
# With conformance tests (start the everything server first)
npm run --silent tier-check -- \
Per-revision resolution is the point, not a convenience. Some SDKs fix their
wire era per server process (go-sdk only speaks 2026-07-28 when started
stateless) or per endpoint (csharp-sdk serves it at `/stateless`), so no single
server URL can be measured against two revisions. The per-SDK, per-revision
invocations live in `known-sdks.ts` `specOverrides`; adding an SDK there is how
it joins the matrix.
For an SDK not in the config, drive it directly — but then the one endpoint you
give must serve every claimed revision at its own wire, and the server must
already be running:
```bash
npx @modelcontextprotocol/conformance tier-check \
--repo modelcontextprotocol/typescript-sdk \
--conformance-server-url http://localhost:3000/mcp
--conformance-server-url http://localhost:3000/mcp \
--client-cmd '<command that runs the SDK conformance client>' \
--requirements 2025-11-25,2026-07-28
```
For a full AI-assisted assessment with remediation guide, use Claude Code:
Omit `--client-cmd` and the client leg is skipped and reported as a gap. Omit
`--requirements` and scoring falls back to the suite as it stands today, which is
not what you want for a tier claim; see [Conformance Requirements](#conformance-requirements).
The exit code is a CI verdict on the machine-checkable half: nonzero when any
scored conformance scenario fails or a leg could not be measured. Governance
findings (triage, P0s, policy files) shape the tier but never the exit code.
### 2. The skill, for the whole assessment
The CLI cannot judge documentation coverage, dependency policy or roadmap quality, and those decide the tier as much as conformance does. The [`mcp-sdk-tier-audit`](.claude/skills/mcp-sdk-tier-audit/README.md) skill runs the CLI, adds those evaluations, and writes a full report with a remediation plan. In Claude Code, from a checkout of this repo:
```
/mcp-sdk-tier-audit <local-sdk-path> <conformance-server-url>
/mcp-sdk-tier-audit <local-sdk-path> <conformance-server-url> '<client-cmd>' --requirements 2025-11-25,2026-07-28
```
See [`.claude/skills/mcp-sdk-tier-audit/README.md`](.claude/skills/mcp-sdk-tier-audit/README.md) for full documentation.
The server must already be running and stay up for the whole audit. Expect a few minutes.
### Reading the result
```
Scored against 2025-11-25 and 2026-07-28, each run at its own wire version.
Server 67/67 required scenarios (100%)
Client 50/50 required scenarios (100%)
Not scored (8 run, 4 failing, no effect on tier):
✗ auth/dpop (extension)
✗ json-schema-2020-12-preservation (added-after-release)
Tier 1 Blockers:
• triage
• p0_resolution
```
- **Required scenarios** are the only ones that move the number. `67/67` spans every revision listed: a scenario belonging to both runs once per revision, on that revision's wire, and both have to pass.
- **Not scored** ran and is reported so you can see it, but cannot fail the tier. `extension` means optional by definition; `added-after-release` means the scenario did not exist when that revision shipped. A failure here is information, not a blocker.
- **Not measured** is different from either, and means the run could not happen at all, e.g. a requirement set naming a scenario this build no longer has. Treat it as a broken invocation, never as an SDK failure.
- **Tier 1 blockers** lists every requirement short of Tier 1. Conformance absent from that list means the SDK met every requirement each listed revision imposes.
The exit code follows the same rule: it reflects required scenarios only, so an implementation that meets a revision's requirements exits 0 even with failing extensions.
In `--output json`, `passed` / `failed` / `total` describe the scored set and so always agree with `pass_rate`; anything run without being scored is counted separately under `not_scored`.
## Architecture