Freeze conformance requirements per spec revision, run each at its wire (#447)

There is no way to answer "which scenarios must my implementation pass
to conform to spec revision X". The suite accumulates scenarios
continuously, so what an implementation is measured against keeps
growing after the revision shipped, and it can drop below 100% while
standing still. --spec-version does not answer it: it filters on which
revision a scenario targets, not on when the scenario became a
requirement.

Add requirements/<revision>.yaml naming the scenarios a revision
requires, and --requirements <revision> on the client, server, list and
sdk commands to run exactly that set. The revision also fixes the wire
version its scenarios speak: the dated revisions through 2025-11-25 use
the stateful initialize handshake and 2026-07-28 is stateless with
per-request _meta, so a scenario belonging to two revisions runs once
under each and one run does not cover the other. Passing no flag keeps
today's behaviour of running everything.

tier-check gains --sdk <name[@ref]> / --sdk-path <dir>: for each claimed
revision it resolves that revision's build, server invocation and client
command from the sdk-runner's KNOWN_SDKS specOverrides, manages the
server, runs both legs with cwd = the SDK checkout, and merges. Every
claimed revision must pass for Tier 1. This is what makes multi-revision
tiering correct for SDKs whose wire era is fixed per server process
(go-sdk only speaks >= 2026-07-28 when constructed stateless) or per
endpoint (csharp-sdk serves it at /stateless): one
--conformance-server-url cannot describe them, and their config already
does. The single-URL mode remains for SDKs not in the config.

Ship two sets. 2026-07-28 is a snapshot of 0.2.0-alpha.10, published the
day before that revision shipped. 2025-11-25 is a reconstruction from
the same anchor and its header says so: the release current on its ship
date predates spec-version awareness entirely. not_scored entries run
and are reported with a reason (extension per SEP-1730, or
added-after-release) but never affect a pass rate or the exit code, and
--expected-failures judges scored scenarios only. Reports label every
entry with its revision, and JSON passed/failed/total describe the
scored set so they always agree with pass_rate.

tier-check's exit code now gates on the machine-checkable half: nonzero
when a scored scenario fails or a leg could not be measured. A child run
that cannot happen at all (a set naming a renamed scenario, an
unreachable server) reports as not measured rather than as 0% against
the implementation. Conformance runs to completion before the GitHub
checks, so an API failure can no longer interrupt a leg and orphan a
managed server, and the legs spawn their child asynchronously: the
previous synchronous exec blocked the event loop, the parent stopped
draining the managed server's log pipes, and a chatty server (rust-sdk's
per-request tracing) froze on a full pipe and was scored as failing.

Driving every tier-1 SDK through the new path also fixed three configs:
csharp-sdk's server now launches from its build-output directory so
ASP.NET finds appsettings.json (its AllowedHosts filter, which
dns-rebinding-protection tests, silently never loaded from the repo
root); rust-sdk's STATELESS=1 override is removed because its CI runs
one default server for both revisions and the forced mode fails its
SEP-2575 scenarios; python-sdk's build mirrors its CI's two targeted
syncs because --all-packages is unbuildable on main. withManagedServer
now waits for its child to exit before returning, so consecutive runs on
one port cannot race a dying predecessor.
This commit is contained in:
Felix Weinberger 2026-08-07 14:55:10 +01:00 • committed by GitHub
commit 232a901445
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

View file

@ -5,7 +5,7 @@ description: >-
Produces tier classification (1/2/3) with evidence table, gap list, and
remediation guide. Works for any official MCP SDK (TypeScript, Python, Go,
C#, Java, Kotlin, PHP, Swift, Rust, Ruby).
argument-hint: '<local-path> <conformance-server-url> [client-cmd] [--branch <branch>]'
argument-hint: '<local-path> [conformance-server-url] [client-cmd] [--requirements <revisions>] [--branch <branch>]'
---
# MCP SDK Tier Audit
@ -26,7 +26,9 @@ If this fails (exit code non-zero or shows "not logged in"), stop immediately an
Do NOT proceed to any other step if this check fails.
After parsing arguments (Step 1), also verify the conformance server is reachable:
**Known SDKs need no server from you.** When the SDK is in `src/sdk-runner/known-sdks.ts` (typescript-sdk, python-sdk, go-sdk, rust-sdk, csharp-sdk, and variants), skip the server preflight entirely: Step 2's `--sdk-path` mode builds the SDK and starts the right server per revision itself. The reachability check below applies only to the URL fallback for unknown SDKs.
After parsing arguments (Step 1), for the URL fallback only, verify the conformance server is reachable:
```bash
curl -sf <conformance-server-url> -o /dev/null -w '%{http_code}' 2>&1 || true
@ -41,8 +43,9 @@ If the server is not reachable, stop and tell the user:
Extract from the user's input:
- **local-path**: absolute path to the SDK checkout (e.g. `~/src/mcp/typescript-sdk`)
- **conformance-server-url**: URL where the SDK's everything server is already running (e.g. `http://localhost:3000/mcp`)
- **conformance-server-url** (only for SDKs not in `known-sdks.ts`): URL where the SDK's everything server is already running (e.g. `http://localhost:3000/mcp`). That one endpoint must serve every claimed revision at its own wire version; SDKs that vary the server per revision cannot be driven this way
- **client-cmd** (optional): command to run the SDK's conformance client (e.g. `npx tsx test/conformance/src/everythingClient.ts`). If not provided, client conformance tests are skipped and noted as a gap in the report.
- **requirements** (optional): spec revisions to score against, comma-separated, e.g. `--requirements 2025-11-25,2026-07-28`. Each revision's scenarios run at that revision's wire version, and every listed revision must pass for Tier 1. Scores the SDK against exactly the scenarios that revision required when it shipped, rather than everything the suite carries today. Prefer it whenever the question is "does this SDK conform to revision X". Without it, scoring uses today's suite, which can fail an SDK for a scenario added after it shipped. Run `conformance list --requirements <revision>` to see the set.
- **branch** (optional): Git branch to check on GitHub (e.g. `--branch fweinberger/v1x-governance-docs`). If not provided, derive from the local checkout's current branch: `cd <local-path> && git rev-parse --abbrev-ref HEAD`. This is passed to the tier-check CLI so that policy signal file checks use the correct branch instead of the repo's default branch.
The first two arguments are required. If either is missing, ask the user to provide it.
@ -57,15 +60,35 @@ cd <local-path> && git remote get-url origin | sed 's#.*github.com[:/]##; s#\.gi
The `tier-check` CLI handles all deterministic checks — server conformance, client conformance, labels, triage, P0 resolution, releases, policy signals, and spec tracking. You are already in the conformance repo, so run it directly.
For a known SDK, this manages everything — build, per-revision server, both legs:
```bash
npm run --silent tier-check -- \
--sdk-path <local-path> \
--requirements <revisions> \
--output json
```
`--repo` and `--branch` are derived from the SDK config and checkout; pass them only to override. For an SDK not in `known-sdks.ts`, fall back to driving it directly:
```bash
npm run --silent tier-check -- \
--repo <owner/repo> \
--branch <branch> \
--requirements <revisions> \
--conformance-server-url <conformance-server-url> \
--client-cmd '<client-cmd>' \
--output json
```
Omit `--requirements` only if the user did not name a revision. When it is set the
scorecard reports `requirements_revision`, both pass rates count exactly the
scenarios that revision requires, and anything run but not scored carries a
`notScoredReason` of `extension` or `added-after-release`. `requirements_revisions`
lists every revision scored, and each detail carries the `revision` it came from. Quote the revision
alongside any conformance number, and report the not-scored failures separately
rather than folding them into the score or omitting them.
If no client-cmd was detected, omit the `--client-cmd` flag (client conformance will be skipped). The `--branch` flag should always be included (derived from the local checkout if not explicitly provided).
The CLI output includes server conformance pass rate, client conformance pass rate (with per-spec-version breakdown), issue triage compliance, P0 resolution times, label taxonomy, stable release status, policy signal files, and spec tracking gap. Parse the JSON output to feed into Step 4.
@ -117,8 +140,8 @@ Combine the deterministic scorecard (from the CLI) with the evaluation results (
### Tier 1 requires ALL of:
- Server conformance test pass rate == 100% (date-versioned scenarios only; `draft` and `extension` are informational and not scored)
- Client conformance test pass rate == 100% (date-versioned scenarios only; `draft` and `extension` are informational and not scored)
- Server conformance test pass rate == 100% across every requirement set given to `--requirements` (each run at its own wire), otherwise of date-versioned scenarios only
- Client conformance test pass rate == 100%, on the same basis
- Issue triage compliance >= 90% within 2 business days
- All P0 bugs resolved within 7 days
- Stable release >= 1.0.0 with no pre-release suffix