232a901445There is no way to answer "which scenarios must my implementation pass to conform to spec revision X". The suite accumulates scenarios continuously, so what an implementation is measured against keeps growing after the revision shipped, and it can drop below 100% while standing still. --spec-version does not answer it: it filters on which revision a scenario targets, not on when the scenario became a requirement. Add requirements/<revision>.yaml naming the scenarios a revision requires, and --requirements <revision> on the client, server, list and sdk commands to run exactly that set. The revision also fixes the wire version its scenarios speak: the dated revisions through 2025-11-25 use the stateful initialize handshake and 2026-07-28 is stateless with per-request _meta, so a scenario belonging to two revisions runs once under each and one run does not cover the other. Passing no flag keeps today's behaviour of running everything. tier-check gains --sdk <name[@ref]> / --sdk-path <dir>: for each claimed revision it resolves that revision's build, server invocation and client command from the sdk-runner's KNOWN_SDKS specOverrides, manages the server, runs both legs with cwd = the SDK checkout, and merges. Every claimed revision must pass for Tier 1. This is what makes multi-revision tiering correct for SDKs whose wire era is fixed per server process (go-sdk only speaks >= 2026-07-28 when constructed stateless) or per endpoint (csharp-sdk serves it at /stateless): one --conformance-server-url cannot describe them, and their config already does. The single-URL mode remains for SDKs not in the config. Ship two sets. 2026-07-28 is a snapshot of 0.2.0-alpha.10, published the day before that revision shipped. 2025-11-25 is a reconstruction from the same anchor and its header says so: the release current on its ship date predates spec-version awareness entirely. not_scored entries run and are reported with a reason (extension per SEP-1730, or added-after-release) but never affect a pass rate or the exit code, and --expected-failures judges scored scenarios only. Reports label every entry with its revision, and JSON passed/failed/total describe the scored set so they always agree with pass_rate. tier-check's exit code now gates on the machine-checkable half: nonzero when a scored scenario fails or a leg could not be measured. A child run that cannot happen at all (a set naming a renamed scenario, an unreachable server) reports as not measured rather than as 0% against the implementation. Conformance runs to completion before the GitHub checks, so an API failure can no longer interrupt a leg and orphan a managed server, and the legs spawn their child asynchronously: the previous synchronous exec blocked the event loop, the parent stopped draining the managed server's log pipes, and a chatty server (rust-sdk's per-request tracing) froze on a full pipe and was scored as failing. Driving every tier-1 SDK through the new path also fixed three configs: csharp-sdk's server now launches from its build-output directory so ASP.NET finds appsettings.json (its AllowedHosts filter, which dns-rebinding-protection tests, silently never loaded from the repo root); rust-sdk's STATELESS=1 override is removed because its CI runs one default server for both revisions and the forced mode fails its SEP-2575 scenarios; python-sdk's build mirrors its CI's two targeted syncs because --all-packages is unbuildable on main. withManagedServer now waits for its child to exit before returning, so consecutive runs on one port cannot race a dying predecessor.
129 lines
4.9 KiB
YAML
129 lines
4.9 KiB
YAML
# Conformance requirements for the 2025-11-25 specification revision.
|
|
#
|
|
# This file is the canonical answer to "which scenarios must my implementation pass
|
|
# to conform to 2025-11-25".
|
|
#
|
|
# Anchor, and READ THIS BEFORE TRUSTING IT AS A SNAPSHOT: unlike the 2026-07-28 set,
|
|
# this one could not be frozen at its own ship date. The release current on 2025-11-25
|
|
# was 0.1.7 (published 2025-11-20), which had no --spec-version flag and no concept of
|
|
# which revision a scenario belonged to, so there is nothing to snapshot. This set is
|
|
# instead derived from @modelcontextprotocol/conformance@0.2.0-alpha.10 filtered to
|
|
# 2025-11-25, and therefore contains scenarios written after 2025-11-25 shipped. It is
|
|
# frozen from here on, but it is a reconstruction rather than a contemporaneous record.
|
|
#
|
|
# `server` and `client` are named for the subcommand that runs them, and list what
|
|
# conformance to this revision requires. Scenarios run at THIS revision's wire
|
|
# version: the dated revisions through 2025-11-25 use the stateful initialize
|
|
# handshake and 2026-07-28 is stateless with per-request _meta, so a scenario that
|
|
# applies to both must be run once under each and one run does not cover the other.
|
|
#
|
|
# There is deliberately no authorization-server section: the MCP specification puts
|
|
# authorization-server implementation beyond its own scope, so those scenarios serve
|
|
# people deploying an authorization server, not implementations of MCP itself.
|
|
#
|
|
# `not_scored` is run and reported but never counts toward a pass rate; each entry
|
|
# says why. `extension` is optional by definition (SEP-1730: "Experimental features
|
|
# and protocol extensions ... are not required for any tier"). `added-after-release`
|
|
# means added to the suite AFTER THE ANCHOR RELEASE this file is derived from
|
|
# (0.2.0-alpha.10) — not after the revision's own ship date. This file is a
|
|
# reconstruction (see above), so several SCORED scenarios also postdate
|
|
# 2025-11-25 itself (ping, dns-rebinding-protection, the token-endpoint-auth
|
|
# trio, auth/pre-registration); they are scored because they entered the suite
|
|
# well before the anchor and every current implementation passes them. The line
|
|
# this file freezes is the anchor, and it draws it consistently. Promoting an
|
|
# entry into the lists above is a deliberate, reviewable change.
|
|
#
|
|
# Scenarios that were PENDING in the source release are never scored — SEP-1730
|
|
# scores "applicable required tests" only, and pending means the suite's own
|
|
# reference fixture cannot pass them yet. They still RUN, under not_scored with
|
|
# reason: pending, because the implementation under test may well pass what the
|
|
# reference fixture cannot, and invisible coverage is how gaps hide. (The tasks
|
|
# extension suite attaches to 2026-07-28 only and has no entries here.)
|
|
|
|
server:
|
|
- server-initialize
|
|
- logging-set-level
|
|
- ping
|
|
- completion-complete
|
|
- tools-list
|
|
- tools-call-simple-text
|
|
- tools-call-image
|
|
- tools-call-audio
|
|
- tools-call-embedded-resource
|
|
- tools-call-mixed-content
|
|
- tools-call-with-logging
|
|
- tools-call-error
|
|
- tools-call-with-progress
|
|
- tools-call-sampling
|
|
- tools-call-elicitation
|
|
- elicitation-sep1034-defaults
|
|
- server-sse-multiple-streams
|
|
- elicitation-sep1330-enums
|
|
- resources-list
|
|
- resources-read-text
|
|
- resources-read-binary
|
|
- resources-templates-read
|
|
- resources-subscribe
|
|
- resources-unsubscribe
|
|
- prompts-list
|
|
- prompts-get-simple
|
|
- prompts-get-with-args
|
|
- prompts-get-embedded-resource
|
|
- prompts-get-with-image
|
|
- dns-rebinding-protection
|
|
|
|
client:
|
|
- initialize
|
|
- tools_call
|
|
- elicitation-sep1034-client-defaults
|
|
- sse-retry
|
|
- auth/metadata-default
|
|
- auth/metadata-var1
|
|
- auth/metadata-var2
|
|
- auth/metadata-var3
|
|
- auth/basic-cimd
|
|
- auth/scope-from-www-authenticate
|
|
- auth/scope-from-scopes-supported
|
|
- auth/scope-omitted-when-undefined
|
|
- auth/scope-step-up
|
|
- auth/scope-retry-limit
|
|
- auth/token-endpoint-auth-basic
|
|
- auth/token-endpoint-auth-post
|
|
- auth/token-endpoint-auth-none
|
|
- auth/pre-registration
|
|
|
|
not_scored:
|
|
- scenario: auth/client-credentials-jwt
|
|
leg: client
|
|
reason: extension
|
|
- scenario: auth/client-credentials-basic
|
|
leg: client
|
|
reason: extension
|
|
- scenario: auth/enterprise-managed-authorization
|
|
leg: client
|
|
reason: extension
|
|
- scenario: auth/dpop
|
|
leg: client
|
|
reason: extension
|
|
- scenario: auth/dpop-nonce
|
|
leg: client
|
|
reason: extension
|
|
- scenario: auth/wif-jwt-bearer
|
|
leg: client
|
|
reason: extension
|
|
- scenario: server-session-lifecycle
|
|
leg: server
|
|
reason: added-after-release
|
|
- scenario: json-schema-2020-12-preservation
|
|
leg: client
|
|
reason: added-after-release
|
|
- scenario: json-schema-2020-12
|
|
leg: server
|
|
reason: pending
|
|
note: >-
|
|
the reference fixture cannot pass it yet; the implementation under test might
|
|
- scenario: server-sse-polling
|
|
leg: server
|
|
reason: pending
|
|
note: >-
|
|
on hold pending server-side SSE improvements in the reference fixture
|