modelcontextprotocol/conformance
Watch
1
0
Fork
You've already forked conformance
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Freeze conformance requirements per spec revision, run each at its wire (#447)

Browse source 
There is no way to answer "which scenarios must my implementation pass
to conform to spec revision X". The suite accumulates scenarios
continuously, so what an implementation is measured against keeps
growing after the revision shipped, and it can drop below 100% while
standing still. --spec-version does not answer it: it filters on which
revision a scenario targets, not on when the scenario became a
requirement.

Add requirements/<revision>.yaml naming the scenarios a revision
requires, and --requirements <revision> on the client, server, list and
sdk commands to run exactly that set. The revision also fixes the wire
version its scenarios speak: the dated revisions through 2025-11-25 use
the stateful initialize handshake and 2026-07-28 is stateless with
per-request _meta, so a scenario belonging to two revisions runs once
under each and one run does not cover the other. Passing no flag keeps
today's behaviour of running everything.

tier-check gains --sdk <name[@ref]> / --sdk-path <dir>: for each claimed
revision it resolves that revision's build, server invocation and client
command from the sdk-runner's KNOWN_SDKS specOverrides, manages the
server, runs both legs with cwd = the SDK checkout, and merges. Every
claimed revision must pass for Tier 1. This is what makes multi-revision
tiering correct for SDKs whose wire era is fixed per server process
(go-sdk only speaks >= 2026-07-28 when constructed stateless) or per
endpoint (csharp-sdk serves it at /stateless): one
--conformance-server-url cannot describe them, and their config already
does. The single-URL mode remains for SDKs not in the config.

Ship two sets. 2026-07-28 is a snapshot of 0.2.0-alpha.10, published the
day before that revision shipped. 2025-11-25 is a reconstruction from
the same anchor and its header says so: the release current on its ship
date predates spec-version awareness entirely. not_scored entries run
and are reported with a reason (extension per SEP-1730, or
added-after-release) but never affect a pass rate or the exit code, and
--expected-failures judges scored scenarios only. Reports label every
entry with its revision, and JSON passed/failed/total describe the
scored set so they always agree with pass_rate.

tier-check's exit code now gates on the machine-checkable half: nonzero
when a scored scenario fails or a leg could not be measured. A child run
that cannot happen at all (a set naming a renamed scenario, an
unreachable server) reports as not measured rather than as 0% against
the implementation. Conformance runs to completion before the GitHub
checks, so an API failure can no longer interrupt a leg and orphan a
managed server, and the legs spawn their child asynchronously: the
previous synchronous exec blocked the event loop, the parent stopped
draining the managed server's log pipes, and a chatty server (rust-sdk's
per-request tracing) froze on a full pipe and was scored as failing.

Driving every tier-1 SDK through the new path also fixed three configs:
csharp-sdk's server now launches from its build-output directory so
ASP.NET finds appsettings.json (its AllowedHosts filter, which
dns-rebinding-protection tests, silently never loaded from the repo
root); rust-sdk's STATELESS=1 override is removed because its CI runs
one default server for both revisions and the forced mode fails its
SEP-2575 scenarios; python-sdk's build mirrors its CI's two targeted
syncs because --all-packages is unbuildable on main. withManagedServer
now waits for its child to exit before returning, so consecutive runs on
one port cannot race a dying predecessor.
This commit is contained in:
Felix Weinberger 2026-08-07 14:55:10 +01:00 • committed by GitHub
commit 232a901445
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 2135 additions and 356 deletions
Download patch file Download diff file Expand all files Collapse all files

127
.claude/skills/mcp-sdk-tier-audit/README.md
View file

@ -9,6 +9,15 @@ Two components work together:
## Quick Start: CLI
For an SDK listed in `src/sdk-runner/known-sdks.ts`, tier-check manages the whole conformance side itself — clone/build, then for each shipped revision the server invocation that SDK declares for that revision:
```bash
gh auth login
npx @modelcontextprotocol/conformance tier-check --sdk go-sdk
```
The per-SDK sections below are for the manual path: driving a server you started yourself.
The CLI is a subcommand of the [MCP Conformance](https://github.com/modelcontextprotocol/conformance) tool.
```bash
@ -95,56 +104,33 @@ The skill lives in `.claude/skills/` in this repo, so if you open [Claude Code](
3. Run the skill:
```
/mcp-sdk-tier-audit <local-sdk-path> <conformance-server-url> [client-cmd]
/mcp-sdk-tier-audit <local-sdk-path> <conformance-server-url> [client-cmd] [--requirements <revision>]
```
Pass the client command as the third argument to include client conformance testing. If omitted, client conformance is skipped and noted as a gap in the report.
**TypeScript SDK example:**
**Pass `--requirements` with every revision the SDK claims**, comma-separated. Each revision's scenarios run at that revision's own wire version, and all of them must pass for Tier 1: the dated revisions through `2025-11-25` use the stateful initialize handshake while `2026-07-28` is stateless, so a scenario belonging to both has to work on both and one run does not cover the other. It also means a scenario added to the suite after a revision shipped cannot fail an SDK that had no opportunity to adopt it. Without the flag, scoring uses the suite as it stands today, which is not a tier claim. See [Conformance Requirements](../../../README.md#conformance-requirements), and run `conformance list --requirements 2025-11-25,2026-07-28` to see both sets.
```bash
# Terminal 1: start the everything server (build first: npm run build)
cd ~/src/mcp/typescript-sdk && npm run test:conformance:server:run
**Known SDKs — no server to start, no commands to look up:**
# Terminal 2: run the audit (from the conformance repo)
/mcp-sdk-tier-audit ~/src/mcp/typescript-sdk http://localhost:3000/mcp "npx tsx ~/src/mcp/typescript-sdk/test/conformance/src/everythingClient.ts"
```
/mcp-sdk-tier-audit <sdk checkout> --requirements 2025-11-25,2026-07-28
```
**Python SDK example:**
The skill drives `tier-check --sdk-path`, which builds the SDK and starts the
server invocation its config declares **per revision** — the per-SDK commands
live in [`src/sdk-runner/known-sdks.ts`](../../../src/sdk-runner/known-sdks.ts),
not here. That matters because a hand-run example cannot be correct for every
SDK: go-sdk needs a different server process per revision (`-stateless` defaults
to true, so a bare invocation mis-measures 2025-11-25 — [#446](https://github.com/modelcontextprotocol/conformance/issues/446)),
and csharp-sdk serves the two revisions at different endpoints. Prose copies of
those invocations drift; the config is tested.
```bash
# Terminal 1: install and start the everything server
cd ~/src/mcp/python-sdk && uv sync --frozen --all-extras --package mcp-everything-server
uv run mcp-everything-server --port 3001
**An SDK not in `known-sdks.ts`:** start its everything server yourself and pass
the URL and client command explicitly — see [Other SDKs](#running-conformance-tests)
below. The one endpoint you give must serve every claimed revision at its own
wire, or the numbers will be wrong for the revisions it does not speak.
# Terminal 2: run the audit (from the conformance repo)
/mcp-sdk-tier-audit ~/src/mcp/python-sdk http://localhost:3001/mcp "uv run python ~/src/mcp/python-sdk/.github/actions/conformance/client.py"
```
**Go SDK example:**
```bash
# Terminal 1: build and start the everything server
cd ~/src/mcp/go-sdk && go build -o /tmp/go-conformance-server ./conformance/everything-server
go build -o /tmp/go-conformance-client ./conformance/everything-client
/tmp/go-conformance-server -http="localhost:3002"
# Terminal 2: run the audit (from the conformance repo)
/mcp-sdk-tier-audit ~/src/mcp/go-sdk http://localhost:3002 "/tmp/go-conformance-client"
```
**C# SDK example:**
```bash
# Terminal 1: start the everything server (requires .NET SDK)
cd ~/src/mcp/csharp-sdk
dotnet run --project tests/ModelContextProtocol.ConformanceServer --framework net9.0 -- --urls http://localhost:3003
# Terminal 2: run the audit (from the conformance repo)
/mcp-sdk-tier-audit ~/src/mcp/csharp-sdk http://localhost:3003 "dotnet run --project ~/src/mcp/csharp-sdk/tests/ModelContextProtocol.ConformanceClient"
```
The skill derives `owner/repo` from git remote, runs the CLI, launches parallel evaluations for docs and policy, and writes detailed reports to `results/`.
### Any Other AI Coding Agent
@ -181,66 +167,17 @@ Run the CLI for the scorecard, then review docs and policies yourself using the
## Running Conformance Tests
To include conformance test results, start the SDK's everything server first, then pass the URL to the CLI. To also run client conformance tests, pass `--client-cmd` with the command to launch the SDK's conformance client.
**TypeScript SDK**:
For any SDK in [`src/sdk-runner/known-sdks.ts`](../../../src/sdk-runner/known-sdks.ts), don't start servers by hand — the config runs the right invocation per revision:
```bash
# Terminal 1: start the server (SDK must be built first)
cd ~/src/mcp/typescript-sdk && npm run build
npm run test:conformance:server:run # starts on port 3000
# both legs, every revision, one verdict
npx @modelcontextprotocol/conformance tier-check --sdk go-sdk
# Terminal 2: run tier-check (server + client conformance)
npm run --silent tier-check -- \
--repo modelcontextprotocol/typescript-sdk \
--conformance-server-url http://localhost:3000/mcp \
--client-cmd 'npx tsx ~/src/mcp/typescript-sdk/test/conformance/src/everythingClient.ts'
# a single leg at a single revision, for debugging
npx @modelcontextprotocol/conformance sdk --path <sdk checkout> --mode server --requirements 2026-07-28
```
**Python SDK**:
```bash
# Terminal 1: install and start the server
cd ~/src/mcp/python-sdk
uv sync --frozen --all-extras --package mcp-everything-server
uv run mcp-everything-server --port 3001 # specify port to avoid conflicts
# Terminal 2: run tier-check (server + client conformance)
npm run --silent tier-check -- \
--repo modelcontextprotocol/python-sdk \
--conformance-server-url http://localhost:3001/mcp \
--client-cmd 'uv run python ~/src/mcp/python-sdk/.github/actions/conformance/client.py'
```
**Go SDK**:
```bash
# Terminal 1: build and start the server
cd ~/src/mcp/go-sdk
go build -o /tmp/go-conformance-server ./conformance/everything-server
go build -o /tmp/go-conformance-client ./conformance/everything-client
/tmp/go-conformance-server -http="localhost:3002"
# Terminal 2: run tier-check (server + client conformance)
npm run --silent tier-check -- \
--repo modelcontextprotocol/go-sdk \
--conformance-server-url http://localhost:3002 \
--client-cmd '/tmp/go-conformance-client'
```
**C# SDK**:
```bash
# Terminal 1: start the server (requires .NET SDK)
cd ~/src/mcp/csharp-sdk
dotnet run --project tests/ModelContextProtocol.ConformanceServer --framework net9.0 -- --urls http://localhost:3003
# Terminal 2: run tier-check (server + client conformance)
npm run --silent tier-check -- \
--repo modelcontextprotocol/csharp-sdk \
--conformance-server-url http://localhost:3003 \
--client-cmd 'dotnet run --project ~/src/mcp/csharp-sdk/tests/ModelContextProtocol.ConformanceClient'
```
To reproduce one leg fully by hand, copy the build/server/client commands from that SDK's `known-sdks.ts` entry (including its `specOverrides` for the revision you're running) rather than from a README: the entries are exercised by the test suite and per-revision, prose examples are neither.
**Other SDKs:** Your SDK needs an "everything server" — an HTTP server implementing the [Streamable HTTP transport](https://modelcontextprotocol.io/specification/draft/basic/transports.md) with all MCP features (tools, resources, prompts, etc.). See the implementations above as reference.

33
.claude/skills/mcp-sdk-tier-audit/SKILL.md
View file

@ -5,7 +5,7 @@ description: >-
Produces tier classification (1/2/3) with evidence table, gap list, and
remediation guide. Works for any official MCP SDK (TypeScript, Python, Go,
C#, Java, Kotlin, PHP, Swift, Rust, Ruby).
argument-hint: '<local-path> <conformance-server-url> [client-cmd] [--branch <branch>]'
argument-hint: '<local-path> [conformance-server-url] [client-cmd] [--requirements <revisions>] [--branch <branch>]'
---
# MCP SDK Tier Audit
@ -26,7 +26,9 @@ If this fails (exit code non-zero or shows "not logged in"), stop immediately an
Do NOT proceed to any other step if this check fails.
After parsing arguments (Step 1), also verify the conformance server is reachable:
**Known SDKs need no server from you.** When the SDK is in `src/sdk-runner/known-sdks.ts` (typescript-sdk, python-sdk, go-sdk, rust-sdk, csharp-sdk, and variants), skip the server preflight entirely: Step 2's `--sdk-path` mode builds the SDK and starts the right server per revision itself. The reachability check below applies only to the URL fallback for unknown SDKs.
After parsing arguments (Step 1), for the URL fallback only, verify the conformance server is reachable:
```bash
curl -sf <conformance-server-url> -o /dev/null -w '%{http_code}' 2>&1 || true
@ -41,8 +43,9 @@ If the server is not reachable, stop and tell the user:
Extract from the user's input:
- **local-path**: absolute path to the SDK checkout (e.g. `~/src/mcp/typescript-sdk`)
- **conformance-server-url**: URL where the SDK's everything server is already running (e.g. `http://localhost:3000/mcp`)
- **conformance-server-url** (only for SDKs not in `known-sdks.ts`): URL where the SDK's everything server is already running (e.g. `http://localhost:3000/mcp`). That one endpoint must serve every claimed revision at its own wire version; SDKs that vary the server per revision cannot be driven this way
- **client-cmd** (optional): command to run the SDK's conformance client (e.g. `npx tsx test/conformance/src/everythingClient.ts`). If not provided, client conformance tests are skipped and noted as a gap in the report.
- **requirements** (optional): spec revisions to score against, comma-separated, e.g. `--requirements 2025-11-25,2026-07-28`. Each revision's scenarios run at that revision's wire version, and every listed revision must pass for Tier 1. Scores the SDK against exactly the scenarios that revision required when it shipped, rather than everything the suite carries today. Prefer it whenever the question is "does this SDK conform to revision X". Without it, scoring uses today's suite, which can fail an SDK for a scenario added after it shipped. Run `conformance list --requirements <revision>` to see the set.
- **branch** (optional): Git branch to check on GitHub (e.g. `--branch fweinberger/v1x-governance-docs`). If not provided, derive from the local checkout's current branch: `cd <local-path> && git rev-parse --abbrev-ref HEAD`. This is passed to the tier-check CLI so that policy signal file checks use the correct branch instead of the repo's default branch.
The first two arguments are required. If either is missing, ask the user to provide it.
@ -57,15 +60,35 @@ cd <local-path> && git remote get-url origin | sed 's#.*github.com[:/]##; s#\.gi
The `tier-check` CLI handles all deterministic checks — server conformance, client conformance, labels, triage, P0 resolution, releases, policy signals, and spec tracking. You are already in the conformance repo, so run it directly.
For a known SDK, this manages everything — build, per-revision server, both legs:
```bash
npm run --silent tier-check -- \
--sdk-path <local-path> \
--requirements <revisions> \
--output json
```
`--repo` and `--branch` are derived from the SDK config and checkout; pass them only to override. For an SDK not in `known-sdks.ts`, fall back to driving it directly:
```bash
npm run --silent tier-check -- \
--repo <owner/repo> \
--branch <branch> \
--requirements <revisions> \
--conformance-server-url <conformance-server-url> \
--client-cmd '<client-cmd>' \
--output json
```
Omit `--requirements` only if the user did not name a revision. When it is set the
scorecard reports `requirements_revision`, both pass rates count exactly the
scenarios that revision requires, and anything run but not scored carries a
`notScoredReason` of `extension` or `added-after-release`. `requirements_revisions`
lists every revision scored, and each detail carries the `revision` it came from. Quote the revision
alongside any conformance number, and report the not-scored failures separately
rather than folding them into the score or omitting them.
If no client-cmd was detected, omit the `--client-cmd` flag (client conformance will be skipped). The `--branch` flag should always be included (derived from the local checkout if not explicitly provided).
The CLI output includes server conformance pass rate, client conformance pass rate (with per-spec-version breakdown), issue triage compliance, P0 resolution times, label taxonomy, stable release status, policy signal files, and spec tracking gap. Parse the JSON output to feed into Step 4.
@ -117,8 +140,8 @@ Combine the deterministic scorecard (from the CLI) with the evaluation results (
### Tier 1 requires ALL of:
- Server conformance test pass rate == 100% (date-versioned scenarios only; `draft` and `extension` are informational and not scored)
- Client conformance test pass rate == 100% (date-versioned scenarios only; `draft` and `extension` are informational and not scored)
- Server conformance test pass rate == 100% across every requirement set given to `--requirements` (each run at its own wire), otherwise of date-versioned scenarios only
- Client conformance test pass rate == 100%, on the same basis
- Issue triage compliance >= 90% within 2 business days
- All P0 bugs resolved within 7 days
- Stable release >= 1.0.0 with no pre-release suffix