- TypeScript 98.1%
- JavaScript 1.9%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
7169291ec0* chore: add SEP-2640 requirement-traceability YAML (Skills Extension)
* style: apply yaml formatter pass
* chore(sep-2640): record spec source provenance
* chore(sep-2640): re-extract against SEP HEAD 556154c (drops mcp-resource-template)
Re-extract sep-2640.yaml against current SEP-2640 HEAD. The SEP removed
the `mcp-resource-template` index entry type in two commits on
2026-06-04 (fd50cc91 "Remove mcp-resource-template entries from skill
index", 556154c0 "Remove remaining resource template mentions from SDK
and rationale sections"), after this extraction was first captured on
2026-06-03 at b77fdfe8. Provenance comment bumped accordingly.
Resulting changes:
- sep-2640-index-entry-type-enum: drop `"mcp-resource-template"` from
the enum.
- sep-2640-index-name-required: drop the "is Required for ... Omitted
for mcp-resource-template" conditional language; the column is now
unconditional Required=Yes at SEP HEAD.
- sep-2640-index-digest-required: same simplification; also align
"hexadecimal" -> "hex" with the HEAD table text.
- sep-2640-template-resource-template-registered: removed. The SEP no
longer defines a template entry type, so the SHOULD on registering an
MCP resource template is gone.
- Excluded "Hosts SHOULD surface template entries..." excerpt: removed.
The sentence no longer exists in the SEP.
Sweep verified: zero "template" mentions in the SEP at HEAD. Security
Implications section sentences referenced by the remaining excluded
entries are intact. No other check rows touched.
* feat(sep-2640): ResourcesDirectoryReadScenario for resources/directory/read
Adds the conformance scenario for the SEP-2640 directoryRead surface that
landed in spec commit 2e04c48d (2026-06-09). Per AGENTS.md "fewer scenarios,
more checks", a single ResourcesDirectoryReadScenario emits 6
ConformanceChecks, one per new sep-2640.yaml requirement row.
Class named for the wire method (resources/directory/read), matching the
existing ResourcesListScenario / ResourcesReadTextScenario / etc. family in
src/scenarios/server/resources.ts. The runner-facing name field stays as
'sep-2640-skills' (umbrella) so mcpkit's conformance/Makefile entry
--scenario sep-2640-skills keeps working without a cross-repo race.
src/seps/sep-2640.yaml
- File-level provenance held at 556154c (the existing PR 330 baseline).
- 6 new check rows appended for the directoryRead additions, each
carrying a verbatim, grep-F-searchable excerpt from the SEP at
2e04c48da90224000e750ffd54a3611f2824fbc0:
- sep-2640-capability-directory-read-flag
- sep-2640-directory-read-method-registered
- sep-2640-directory-read-subdir-mimetype
- sep-2640-directory-read-result-resources-shape
- sep-2640-directory-read-invalid-params
- sep-2640-directory-read-pagination
- forward_reference header comment notes the asymmetry: PR 97 schema
rewrite (360123d0, 2026-06-08) made 3 existing rows stale and drifted
~11 others' wording; full re-extraction at SEP HEAD is mcpkit#780's
lifecycle. Provenance deliberately holds at 556154c until that lands.
src/types.ts
- Adds io.modelcontextprotocol/skills to EXTENSION_IDS so the scenario's
source: { extensionId: ... } tag type-checks.
src/scenarios/server/directory.ts (new)
- Capability discovery via wire-observable signal: -32601 method-not-found
is the only definitive "server did not declare directoryRead" signal.
- Fixture assumption: server exposes skill://acme/billing/refunds/templates
with at least one subdirectory child (mcpkit examples/skills layout).
No skill:// resources at all -> every check emits SKIPPED so the
scenario stays green against upstream's everything-server fixture.
- 6 checks (1:1 with the YAML rows above):
1. directoryRead declared (derived from method registration)
2. method registered (happy-path call succeeds)
3. result.resources shape matches resources/list
4. subdir child carries mimeType: "inode/directory"
5. non-directory URI returns -32602
6. nextCursor round-trips (single-page is conformant)
src/scenarios/index.ts
- Imports + registers ResourcesDirectoryReadScenario in
allClientScenariosList, matching the registration name 'sep-2640-skills'
that mcpkit's conformance/Makefile already passes to --scenario.
Verified end-to-end against mcpkit examples/skills:
cd ~/newstack/mcpkit/main && \
MCPCONFORMANCE_SKILLS_PATH=~/newstack/mcpkit/conf-skills \
make -C conformance testconf-skills
-> 6/6 SUCCESS, 0 failed, 0 warnings.
Out of scope:
- PR 97 schema rewrite YAML refresh (separate ticket, mcpkit#780 lifecycle).
- Negative-capability fixture (server without directoryRead) - mcpkit
examples/skills has no flag for it; ext/skills/client_directory_test.go
already covers the SDK pre-call guard.
* feat(sep-2640): expand skills conformance — index + manifest scenarios, drift fixes
Builds on the directory-read scenario with the server-observable skills surface
and brings the whole set up to the current framework conventions.
New scenarios (src/scenarios/server/skills/):
- SkillsIndexScenario (sep-2640-skills-index): skill://index.json is exposed and
readable; every skills[].type is in the SEP enum; every entry has a non-empty
name; a present digest matches sha256:{64hex}; entry URLs use skill://.
- SkillsManifestScenario (sep-2640-skills-manifest): a skill-md SKILL.md is
markdown-typed; frontmatter name/description surface on the resource; the
final skill-path segment equals the frontmatter name; skill _meta keys use the
reverse-domain prefix.
directory.ts (moved into skills/): capability is now read from server/discover
instead of inferred from -32601; an undeclared optional capability SKIPs rather
than FAILs; the untestable.ts helpers back the not-exercisable paths; all
mcpkit-specific fixture URIs are gone — the directory to exercise is discovered
dynamically, so the scenario is brand-neutral.
Capability gating accepts directoryRead at either extensions[id].directoryRead
(SEP-2640's inline example) or extensions[id].config.directoryRead (the SEP-2133
negotiation envelope SEP-2640 defers to); the two SEPs are inconsistent on
nesting, so the check accepts either rather than privileging one reading.
All three scenarios verified green against a live SEP-2640 server (6/6, 6/6, 7/7);
typecheck, eslint, prettier, and the 504-test vitest suite pass. sep-2640.yaml
gains a header note mapping each scenario to the rows it now backs.
* feat(sep-2640): re-extract against the 2026-08-21 rewrite
The prior extraction held at SEP commit 556154c (2026-06-05) by an explicit
in-file decision to wait for the index-schema refactor. The 08-21 core-maintainer
rewrite made that pin actively wrong rather than merely behind.
Removed as no longer normative: the skill://index.json well-known resource
(absent from the current text), the entry type enum, the no-nested-skills rule
(now reversed - skills MAY nest), and the five archive rows (archives are
deferred to an appendix). Reworded the two index-field rows, the empty-listing
row and the verify-digest row.
Added the new surface: skills/list and skills/get with their pagination,
atomicity and caching attributes; the {uri, frontmatter, resources} entry
schema; resources completeness, the "dynamic" sentinel and per-file size; the
512-entry and 16 MiB limits; and the host obligations around lazy retrieval,
content-bound approval and frontmatter re-verification.
index.ts is replaced by enumeration.ts, which drives both new methods. manifest.ts
and directory.ts swap their skill://index.json discovery fallback for skills/list.
The capability helper no longer accepts a config envelope. SEP-2133 is Final and
defines the extensions field as a map of identifiers to settings objects, with no
envelope and no slot for id/specVersion/stability; SEP-2640's capability block
matches. The earlier leniency was based on a misreading.
77 rows: 65 checks (39 wire-tested by the three server scenarios) and 12 excluded
as host-internal.
* feat(sep-2640): close the extraction gaps found by a normative-sentence audit
The first pass extracted the Specification sections but under-covered Security
Implications, which carries 21 of the SEP's 97 RFC-2119 sentences. Sweeping
every normative sentence in the source surfaced rows that were missing entirely:
- origin tagging at the point content enters model context (the L179 host MUST)
- allowed-tools MUST be ignored for MCP-origin skills
- reads bound to the originating server, so a skill from A cannot drive a read
against B, and any cross-origin read gated behind per-call approval
- servers identified by a host-assigned label, never serverInfo.name
- the three disk-cache rules: write-isolation or re-hash, exclusion from
filesystem-skill discovery paths, and removal when a server is removed
- a resources/read of a SKILL.md by another route is not a load
- a nested SKILL.md read as supporting content is ordinary markdown
- the registry-assembly and virtual-mount no-prefetch restatements
- the SDK affordances guidance, which is the row mcpkit itself is measured by
Five rows previously filed as excluded are now declared instead. They are real
requirements that a host-side harness could check; calling them unobservable was
a limitation of this scenario set, not of the requirement.
Adds one wire check, names-should-be-unique, since a within-listing collision is
observable and tells an operator the listing needs disambiguating.
96 rows: 89 checks (40 wire-tested) and 7 excluded.
* style: apply prettier to the SEP-2640 scenarios and yaml
Quote-style normalisation in the yaml (escaped single quotes become
double-quoted scalars) and one return-type wrap in helpers.ts. The parsed
yaml is byte-identical in content; verified by comparing the parse before
and after.
* docs(sep-2640): note why resultType is not declared in this yaml
It appears in all three result examples but is a base-protocol field on the
common Result interface, not a SEP-2640 requirement; its caching-hint
obligations are already owned by sep-2549.yaml.
* fix(sep-2640): follow nextCursor on resources/directory/read
The directory scenario issued one resources/directory/read and every check
below it inspected that single page: the result-shape check, the
subdirectory-mimeType check, and the target selection for the -32602 probe.
SEP-2640 says directory-read pagination mirrors resources/list, so a
conformant server MAY split a directory across pages, and reading only the
first made "no subdirectory here" indistinguishable from "the subdirectory
is on page two".
Reported by Sam Bloomberg against the go-sdk implementation, which exposes
DirectoryOptions.PageSize. Rebuilding that server with PageSize 1 fails
sep-2640-directory-read-subdir-mimetype; with this change it passes 7/7,
and both servers still pass at their default page size.
Only findable against a second implementation. mcpkit hardcoded its page
size to zero with no override, so it never emitted a cursor and the defect
was invisible to every run against it.
listAllResources already did this for resources/list. directoryReadAll
applies the same shape to directory reads, with a page bound and a
repeated-cursor guard.
* feat(sep-2640): client scenario for the no-prefetch MUST NOT
The traceability file filed retrieval policy under "host obligations a
harness cannot observe", alongside 48 other rows. That was wrong for this
one: "hosts MUST NOT retrieve a skill's files ahead of need" reduces to
whether a request arrived, which makes it among the most wire-observable
rules in the SEP.
Adds a Scenario (harness stands up the server, client is the SUT) that
serves a listing naming a SKILL.md and a supporting file, then fails if
either is read. The client's contract, keyed on MCP_CONFORMANCE_SCENARIO,
is to connect, enumerate, and stop.
SKIPs rather than passes when the client never calls skills/list, since
without a listing there is no window in which prefetching is possible and
a pass would prove nothing.
Verified both directions against mcpkit's client: SUCCESS as written, and
FAILURE with a deliberate ReadFromEntry added to the driver. A check that
has never been shown to fail is the same trap as the directory-pagination
false negative this suite shipped for months.
* feat(sep-2640): client scenarios for the four read-time verification MUSTs
Adds digest, size, frontmatter and unlisted-URI variants alongside the
no-prefetch scenario. Each serves an internally honest listing, tampers with
exactly one thing on the wire, and watches whether the client notices.
Rejection is detected by absence, following auth/resource-mismatch: the
client's contract is "load the skill, then read its supporting file", and a
client that verifies aborts on the load and never reaches the supporting
file. Reaching it is the violation. The unlisted variant is proven by the
read never appearing on the wire at all, since a conformant client refuses
locally.
The frontmatter variant recomputes digest and size for the tampered body, so
the frontmatter comparison is the only thing left that can fail. Without
that the digest check would mask it and the scenario would pass for the
wrong reason.
allowClientError is set on all four: a verifying client exits non-zero
because it correctly aborted, and that is the pass.
Verified both directions against mcpkit's client. All four SUCCESS as
written; all four FAILURE against a driver rewritten to read through
ReadSkillURI, which skips verification entirely.
* fix(sep-2640): gate the cache-attributes check on the negotiated version
The check graded ttlMs and cacheScope without looking at the protocol
version, which made it wrong in both directions. Below 2026-07-28 those
fields are not defined by the negotiated schema at all, so a server that
omits them is correct and the WARNING was a false positive. From 2026-07-28
they are required, so a missing one is a failure rather than a warning.
Now SKIPPED with a version reason below 2026-07-28, and SUCCESS or FAILURE
above it.
Reported by Sam Bloomberg against the Go SDK, which was correct on both
wires while this check mis-graded it on both. Second finding from the same
reviewer against this branch, and the second that only a second
implementation could surface: mcpkit emits the attributes on every version
it supports, so it could never have produced the false positive.
Verified three ways: mcpkit at 2026-07-28 SUCCESS, the Go SDK at 2025-11-25
SKIPPED where it previously warned, and a deliberately stripped mcpkit build
at 2026-07-28 FAILURE.
* docs(sep-2640): how to run the skills scenarios against any implementation
Three implementations have now been cross-checked and every one needed a
different invocation, so the knowledge was living in chat rather than the
repo. Records the two flags and why both are needed, how to read totals that
include wire-schema-valid, and the per-SDK setup.
The two traps are worth writing down because both have already cost someone
time. Extension scenarios do not match --spec-version alone, so --force is
always needed. And --spec-version selects the wire lifecycle rather than
filtering, so omitting it against a pre-2026-07-28 server produces a -32022
that reads like a missing method.
Also records that the go-sdk serves 2026-07-28 only when its streamable
transport is Stateless, which is correct per SEP-2575 and which I briefly
mistook for a defect.
* fix(sep-2640): advertise skills on server/discover, not only initialize
Both client scenarios declared the skills extension and `resources` in their
`initialize` reply, but `BaseHttpScenario` intercepts `server/discover` before
`handlePost` runs. On the 2026-07-28 lifecycle a discover-first client
therefore saw the base class default of `{tools: {}}` and had nothing to gate
a `skills/list` call on.
The scenarios still passed, because `server/discover` is optional for clients
and neither client we ran used it. That is the failure mode where a check goes
green because its condition never arose, so both lifecycles are now asserted
in capabilities.test.ts rather than left to the next implementation to find.
Overriding `discoverCapabilities()` covers both paths at once, since
`sendInitialize` already defaults to it.
* fix(sep-2640): give the capability and naming checks something to fail on
`sep-2640-capability-empty-object` reported a hardcoded SUCCESS, so it could
not fail and was checking nothing. It now asserts the observable half of "an
empty object indicates support for the extension with no optional features":
the declared value must be an object at all.
That also closes a false SKIP. `skillsCapability` folded a malformed
declaration into `undefined`, which downstream read as "extension not
declared" and skipped the whole suite, so a server declaring the extension as
`true` produced a clean run. `declaredSkillsCapability` separates absent from
malformed; the inline/envelope check now SKIPs on a malformed value, since
there are no settings to place either way. The base `DiscoverResult` schema
rejects a non-object too, so both halves are asserted.
`SKILL_NAME_PATTERN` accepted consecutive hyphens. The Agent Skills
specification that SEP-2640 defers to for naming lists four rules and we
encoded three, so `pdf--processing` passed. Its own valid and invalid examples
are now the test cases.
Negative controls recorded for all three: with the check back to a hardcoded
SUCCESS the four malformed-declaration cases fail, and with the previous regex
the consecutive-hyphen case fails.
* docs(sep-2640): name skills/list in the discovery prose, not index.json
The 2026-08-21 revision replaced the `skill://index.json` well-known resource
with `skills/list`, and the code follows it — `discoverDirectory` and the
manifest fallback both call `skillsListAll`. Only the prose lagged, including
three strings a user actually sees: two untestable-check reasons and the
directory scenario's description.
The remaining mentions in helpers.ts, enumeration.ts and sep-2640.yaml are
deliberate history, describing what was replaced.
* fix(sep-2640): grade unlisted reads against the entry, not one fixture URI
The `unlisted` client scenario asserted `reads.includes(UNLISTED_URI)`, so it
only caught a client that probed the exact path this fixture happens to serve.
Any client choosing a different unlisted path passed without being tested, and
the check would have reported SUCCESS for a client that reads unlisted files
freely.
The requirement is "hosts MUST resolve reads of the skill's files only to URIs
listed in that entry's resources", so the permitted set now comes from the
entry the client was handed and any read outside it is the violation. That is
the sentence itself rather than a proxy for it, and it lets a SUT derive its
own probe instead of hardcoding ours.
Negative control: a client reading a synthesized `mcpkit-unlisted-probe-0.md`
is now a FAILURE, where the previous check reported SUCCESS.
* fix(sep-2640): require the manifest read, and retire the unlisted scenario
Two changes to the client-side verification scenarios, one of which fixes a
hole the other one shares.
**The manifest read is now a prerequisite.** Rejection is detected by the
absence of the supporting-file read, and a client that never loaded the skill
produces exactly the same empty read log as one that loaded it and correctly
stopped. Both reported SUCCESS, so `listCalled` was the only gate and a client
implementing `skills/list` and nothing else passed every variant here. It
scored 5/5 on this suite while doing nothing.
Absence now means something only after the client has read the tampered
`SKILL.md`. A missing prerequisite reports untestable, which per #248 is a
FAILURE rather than a SKIP: SKIPPED is excluded from pass/fail counts and exit
codes, so it would read as green and hide the gap.
Negative control: a driver that lists and exits goes from 1/1 to 0/1 on all
three variants.
**`-verify-unlisted` is retired**, on Peter Alexander's call. Nothing was
tampered in it, so the only thing under test was whether the driver asked for
a URI it should not have, which grades the driver rather than the client. It
is also the rule an inspection tool answers differently from a host, since
refusing a read requires having a load to refuse, which MCP Inspector 2.6.0
reported as "surfaced" rather than passed.
The requirement is not deleted. It moves to an `excluded:` row alongside the
acting-on window that scopes it, and the reason records that it is excluded by
construction rather than in principle: a `SKILL.md` body linking to a file
absent from `resources` would make the harness adversarial again, since a host
that follows the link fetches an unlisted file on its own initiative. The
scenario name and check id are named there so a future reintroduction lines up
with historical reports rather than reading as a new requirement.
Check rows go from 89 to 88.
* feat(sep-2640): check skills/get cache attributes and the resources capability
Both requirements landed when the stable spec page merged ext-skills#139
today, and both are checkable on the wire.
**`sep-2640-skills-get-cache-attributes`.** `GetSkillResult extends
CacheableResult`, so `ttlMs` and `cacheScope` are REQUIRED as they are on
`resources/read`. Held out of this PR until the page merged rather than
grading against text that was still open: SEP-2640 says of these fields that
"whether the result should also carry the base protocol's caching attributes
... is left open", so this is a behaviour change and a server conformant to
the SEP text alone omits both. It is a reading not everyone shares. The MCP
Inspector's SEP-2640 writeup argues the opposite, that the SEP forecloses it
and inventing the requirement would report a reasonable server as broken.
Gated the same way as the skills/list check: below 2026-07-28 `CacheableResult`
is undefined by the negotiated schema, so omission is correct and the check
SKIPs with a version reason rather than failing a conformant legacy server.
**`sep-2640-capability-requires-resources`.** A server declaring the extension
MUST also declare the base `resources` capability, since skill files are served
through `resources/read`. The page states it as a consequence of the base
Resources specification rather than a free-standing new rule, and the
declaration is on the wire either way.
Verified against mcpkit: 32/32 on 2026-07-28 and 30/30 on 2025-11-25, where the
get-cache check SKIPs below the floor as designed. Negative control recorded: a
build with the attributes stripped from `handleSkillsGet` reports 31/32 with
`sep-2640-skills-get-cache-attributes` FAILURE.
Check rows go from 88 to 90.
* fix(sep-2640): compare frontmatter by content, not key order
entry-frontmatter-identical compared each top-level key with JSON.stringify,
which is key-order sensitive for nested objects. A server whose encoder sorts
map keys (Go's encoding/json does) failed this MUST for any SKILL.md whose
metadata keys are not in alphabetical order, although the content is
identical.
Compare structurally instead. A real difference still fails; both cases are
pinned in negative.test.ts.
* fix(sep-2640): make the client-scenario mocks answer like a conformant server
The skills/list and resources/read replies carried only resultType, so on the
2026-07-28 wire they lacked the ttlMs/cacheScope a ReadResourceResult
requires, and skills/list lacked the attributes the enumeration scenario
requires of servers.
initialize always answered 2026-07-28, while the runner hands these extension
scenarios' clients 2025-11-25 by default. A client that checks the negotiated
version disconnected before listing, and every verify variant reported
SKIPPED whether or not the client verified anything.
The mocks now stamp the caching members and answer initialize at the
requested version when the suite can speak it. The runner default is left
alone because it also applies to the auth extension scenarios. Pinned in
draft-result-fields.test.ts next to the other hand-rolled mocks.
* docs(sep-2640): run from the published package; --force only with --spec-version
The run notes pointed at a fork branch that goes stale on merge, and said
--force is always needed. Extension scenarios only SKIP when an explicit
--spec-version is passed; without it they run on the draft wire.
* docs(sep-2640): drop RUNNING_SEP2640.md
Per-run notes and SDK result tables belong in the PR description, not a
root-level doc that goes stale on merge.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
|
||
| .claude/skills | ||
| .github | ||
| examples | ||
| requirements | ||
| scripts | ||
| src | ||
| .gitattributes | ||
| .gitignore | ||
| .prettierignore | ||
| .prettierrc.json | ||
| action.yml | ||
| AGENTS.md | ||
| CONTRIBUTING.md | ||
| eslint.config.mjs | ||
| lefthook-local.example.yml | ||
| lefthook.yml | ||
| LICENSE | ||
| MAINTAINERS.md | ||
| package-lock.json | ||
| package.json | ||
| README.md | ||
| SDK_INTEGRATION.md | ||
| tsconfig.json | ||
| vitest.config.ts | ||
MCP Conformance Test Framework
A framework for testing MCP (Model Context Protocol) client and server implementations against the specification.
For SDK maintainers: See SDK Integration Guide for a streamlined guide on integrating conformance tests into your SDK repository.
Quick Start
Testing Clients
# Using the everything-client (recommended)
npx @modelcontextprotocol/conformance client --command "tsx examples/clients/typescript/everything-client.ts" --scenario initialize
# Run an entire suite of tests
npx @modelcontextprotocol/conformance client --command "tsx examples/clients/typescript/everything-client.ts" --suite auth
Testing Servers
# Run all server scenarios (default)
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp
# Run a single scenario
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --scenario server-initialize
List Available Scenarios
npx @modelcontextprotocol/conformance list
Overview
The conformance test framework validates MCP implementations by:
For Clients:
- Starting a test server for the specified scenario
- Running the client implementation with the test server URL
- Capturing MCP protocol interactions
- Running conformance checks against the specification
- Generating detailed test results
For Servers:
- Connecting to the running server as an MCP client
- Sending test requests and capturing responses
- Running conformance checks against server behavior
- Generating detailed test results
Usage
Client Testing
npx @modelcontextprotocol/conformance client --command "<client-command>" --scenario <scenario-name> [options]
Options:
--command- The command to run your MCP client (can include flags)--scenario- The test scenario to run (e.g., "initialize")--suite- Run a suite of tests in parallel:all,core,extensions,backcompat,auth,metadata,draft(scenarios targeting the in-progress draft spec), orsep-835--spec-version <version>- Filter scenarios by spec version (e.g.,2025-11-25,2026-07-28;draftis accepted as an alias for the current draft identifier). The draft version selects the latest dated release plus any draft-only scenarios. When omitted, the version is inferred from the scenario's spec applicability (draft-only scenarios run at the draft version, everything else at the latest dated release); an explicitly requested version outside a scenario's applicability window skips the scenario (exit 0) unless--forceis passed--force- Run a scenario even if it is not applicable at the requested--spec-version--requirements <revision>- Run exactly what a spec revision requires, frozen at its release (see Conformance Requirements)--expected-failures <path>- Path to YAML baseline file of known failures (see Expected Failures)--timeout- Timeout in milliseconds (default: 30000)--verbose- Show verbose output
The framework appends <server-url> as an argument to your command and sets the MCP_CONFORMANCE_SCENARIO environment variable to the scenario name. For scenarios that require additional context (e.g., client credentials), the MCP_CONFORMANCE_CONTEXT environment variable contains a JSON object with scenario-specific data. When --spec-version is passed, its resolved value is forwarded to the client process as MCP_CONFORMANCE_PROTOCOL_VERSION; example clients can use this value directly as their protocolVersion. SDKs that hard-code their protocol version can ignore it. Clients under test must derive the lifecycle from the protocol version they are asked to run: dated versions through 2025-11-25 use the stateful lifecycle (initialize handshake), while the 2026 draft (2026-07-28) uses the stateless lifecycle (per-request _meta).
Server Testing
npx @modelcontextprotocol/conformance server --url <url> [--scenario <scenario>]
Options:
--url- URL of the server to test--scenario <scenario>- Test scenario to run (e.g., "server-initialize"). Runs all available scenarios by default--suite <suite>- Suite to run: "active" (default; excludes pending and draft-spec scenarios), "all", "draft" (scenarios targeting the in-progress draft spec), or "pending"--requirements <revision>- Run exactly what a spec revision requires, frozen at its release (see Conformance Requirements)--expected-failures <path>- Path to YAML baseline file of known failures (see Expected Failures)--verbose- Show verbose output
Test Results
Client Testing - Results are saved to results/<scenario>-<timestamp>/:
checks.json- Array of conformance check results with pass/fail statusstdout.txt- Client stdout outputstderr.txt- Client stderr output
Server Testing - Results are saved to results/server-<scenario>-<timestamp>/:
checks.json- Array of conformance check results with pass/fail status
Wire-schema checks
Every scenario also validates each JSON-RPC message on the wire against the spec's JSON schema for the negotiated spec version, and emits up to two synthetic checks alongside the scenario's own:
wire-schema-valid- fails when a message the implementation under test sent violates the spec JSON schema. The failure details include every violating message and its schema errors.wire-schema-harness-error- fails when the harness itself sent an invalid message. This indicates a bug in the conformance suite (or a deliberately nonconformant fixture), not in the implementation under test; please report it.
Scenarios that exchange no instrumented wire traffic (see issue #418) emit
neither check. Like any other check, wire-schema-valid can be baselined via
the expected-failures file.
Conformance Requirements
--suite and --spec-version describe the suite as it is today. Neither answers
"which scenarios did I need to pass to conform to the spec released on
2026-07-28", because the suite keeps growing: a scenario merged after a revision
ships still carries that revision's applicability tag, so it is
indistinguishable from one that existed at release.
A requirement set answers that question. Each requirements/<revision>.yaml
names the scenarios a revision requires, for the two roles the specification
defines: an MCP server acting as an OAuth resource server, and an MCP client
acting as an OAuth client. It deliberately covers no authorization-server
scenarios, because the specification puts authorization server implementation
beyond its own scope, so those scenarios serve people deploying an authorization
server rather than implementations of MCP itself.
Scenarios run at their revision's wire version. That is the point of a
per-revision set, not a label on it: the dated revisions through 2025-11-25
use the stateful initialize handshake and 2026-07-28 is stateless with
per-request _meta, and a scenario emits different checks under each. A
scenario belonging to both revisions must therefore be run twice, once under
each set. Passing it on one wire says nothing about the other:
# what does conforming to 2026-07-28 actually require?
npx @modelcontextprotocol/conformance list --requirements 2026-07-28
# run exactly that
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --requirements 2026-07-28
--requirements replaces --suite, --spec-version and --scenario, since the
set already names every scenario that runs and the revision fixes the wire they
run at. Without it nothing changes: the default is still to run everything, which
is where completeness lives.
tier-check takes several at once, and every one of them must pass for Tier 1:
npx @modelcontextprotocol/conformance tier-check --repo <owner/repo> \
--conformance-server-url http://localhost:3000/mcp \
--requirements 2025-11-25,2026-07-28
Only scenarios a revision actually requires decide the exit code and the pass rate. Anything run without being scored is reported separately and cannot fail the run.
Requirement sets are frozen, and not_scored holds what a revision runs and
reports without counting. Two reasons qualify, and the report names which
applies:
| Reason | Meaning |
|---|---|
extension |
Optional by definition. SEP-1730: "Experimental features and protocol extensions (such as Tasks and MCP Apps) are not required for any tier." |
added-after-release |
The scenario did not exist when the revision shipped, so no implementation could have been passing it. |
pending |
The suite's own reference fixture cannot pass it yet, so it cannot be required — but the implementation under test may pass it, so it runs for visibility. |
Both still run, so a failing extension stays visible in the report; neither moves the pass rate. Promoting an entry into the required lists is a deliberate, reviewable change, which is how the suite grows without retroactively failing anyone.
This is the project's contract and lives in this repository. It is the opposite of an expected-failures baseline, which lives in an implementation's own repository and records what that implementation knows it fails. A baselined failure is still a failure against a requirement set.
Expected Failures
SDKs that don't yet pass all conformance tests can specify a baseline of known failures. This allows running conformance tests in CI without failing, while still catching regressions.
Create a YAML file listing expected failures by mode:
# conformance-baseline.yml
server:
- tools-call-with-progress
- resources-subscribe
client:
- sse-retry
Then pass it to the CLI:
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --expected-failures ./conformance-baseline.yml
Exit code behavior:
| Scenario Result | In Baseline? | Outcome |
|---|---|---|
| Fails | Yes | Exit 0 — expected failure |
| Fails | No | Exit 1 — unexpected regression |
| Passes | Yes | Exit 1 — stale baseline, remove the entry |
| Passes | No | Exit 0 — normal pass |
This ensures:
- CI passes when only known failures occur
- CI fails on new regressions (unexpected failures)
- CI fails when a fix lands but the baseline isn't updated (stale entries)
Baselining a single check
A scenario is many checks — server-stateless alone is over twenty — so baselining the
whole scenario to excuse one of them stops enforcing the other nineteen. An entry can
instead name a single check, as <scenario>:<check-id>:
server:
- tasks-lifecycle # whole scenario may fail
- server-stateless:sep-2575-server-implements-discover # only this check may fail
The check id is the left-hand column the runner already prints for each check, so it can be copied straight out of a failing run.
With a per-check entry, every failing check in that scenario is judged on its own: the named one is excused, and any other failure is still an unexpected regression. The four exit-code rules above apply per check rather than per scenario, with one addition — a baselined check that is absent or skipped is tolerated, because a scenario that bails on a failed prerequisite legitimately never reaches its later checks, and the prerequisite reports its own failure anyway.
Two things to know:
- A check id addresses every occurrence of that id. Ids repeat within a run (a loop, a retried flow), and the occurrences collapse to one verdict, most-severe first. So baselining a repeated id excuses all of its occurrences — coarser than ideal, still far narrower than baselining the scenario.
- Mind the space.
- scenario:check-idis a string;- scenario: check-idis YAML for a mapping and is rejected with an error.
A scenario cannot be listed both wholesale and per-check — the wholesale entry already excuses everything, so the pair is contradictory and is rejected.
GitHub Action
This repo provides a composite GitHub Action so SDK repos don't need to write their own conformance scripts.
Server Testing
steps:
- uses: actions/checkout@v4
# Start your server (SDK-specific)
- run: |
my-server --port 3001 &
timeout 15 bash -c 'until curl -s http://localhost:3001/mcp; do sleep 0.5; done'
- uses: modelcontextprotocol/conformance@v0.1.11
with:
mode: server
url: http://localhost:3001/mcp
expected-failures: ./conformance-baseline.yml # optional
Client Testing
steps:
- uses: actions/checkout@v4
- uses: modelcontextprotocol/conformance@v0.1.11
with:
mode: client
command: 'python tests/conformance/client.py'
expected-failures: ./conformance-baseline.yml # optional
Action Inputs
| Input | Required | Description |
|---|---|---|
mode |
Yes | server or client |
url |
Server mode | URL of the server to test |
command |
Client mode | Command to run the client under test |
expected-failures |
No | Path to YAML baseline file |
suite |
No | Test suite to run |
scenario |
No | Run a single scenario by name |
timeout |
No | Timeout in ms for client tests (default: 30000) |
verbose |
No | Show verbose output (default: false) |
node-version |
No | Node.js version (default: 20) |
Example Clients
examples/clients/typescript/everything-client.ts- Single client that handles all scenarios based on scenario name (recommended)examples/clients/typescript/test1.ts- Simple MCP client (for reference)examples/clients/typescript/auth-test.ts- Well-behaved OAuth client (for reference)
Available Scenarios
Client Scenarios
- initialize - Tests MCP client initialization handshake
- Validates protocol version
- Validates clientInfo (name and version)
- Validates server response handling
- tools-call - Tests tool invocation
- auth/basic-dcr - Tests OAuth Dynamic Client Registration flow
- auth/basic-metadata-var1 - Tests OAuth with authorization metadata
Server Scenarios
Run npx @modelcontextprotocol/conformance list --server to see all available server scenarios, including:
- server-initialize - Tests server initialization and capabilities
- tools-list - Tests tool listing endpoint
- tools-call-* - Various tool invocation scenarios
- resources-* - Resource management scenarios
- prompts-* - Prompt management scenarios
Running Against an SDK at a Specific Ref
The sdk subcommand clones an SDK repository at a given ref, builds it, and runs the local conformance build against it. This is the inner-loop tool for scenario authors and the basis for cross-SDK CI. Examples below use npm start -- so they run from source — no npm run build between edits.
--mode client or --mode server is required — each invocation tests exactly one side, so client and server are run (and pass/fail) independently.
# Run the client conformance suite against typescript-sdk @main (v2)
npm start -- sdk typescript-sdk --mode client
# Run the server conformance suite (separate invocation)
npm start -- sdk typescript-sdk --mode server
# A specific main-line SHA or branch (v2 monorepo)
npm start -- sdk typescript-sdk@abc123f --mode client
npm start -- sdk typescript-sdk@some-branch --mode server
# The published v1.x line — separate entry (npm build), defaults to the v1.x branch
npm start -- sdk typescript-sdk-v1 --mode client
npm start -- sdk typescript-sdk-v1@v1.29.0 --mode server
# Use an existing local checkout (no clone, no fetch)
npm start -- sdk --path ../typescript-sdk --skip-build --mode client
# Narrow to one scenario / suite
npm start -- sdk --path ../typescript-sdk --mode server --scenario server-initialize
npm start -- sdk typescript-sdk --mode client --suite auth
# Target a specific spec version (passed through to the underlying run).
# When omitted, the SDK's `specVersion` from KNOWN_SDKS is used, if set —
# e.g. typescript-sdk-v1 defaults to 2025-11-25.
npm start -- sdk typescript-sdk --mode client --spec-version draft
Build/run commands for each official SDK are looked up by name from src/sdk-runner/known-sdks.ts — no config file is required in the SDK repo. Resolution order is CLI flag > built-in entry, so any field can be overridden on the command line for refs that diverge from the built-in.
An SDK can have more than one entry when its layout differs across major versions — e.g. typescript-sdk (v2, the main monorepo) and typescript-sdk-v1 (the published npm v1.x line). An entry may set defaultRef (the branch used when you don't pass @<ref>) and repo (the real clone target when the entry name is an alias).
When the right invocation depends on the spec version being targeted, the entry carries it in specOverrides instead of a comment to copy from. The matching entry is merged over the base config when you pass --spec-version (or when the entry's own specVersion default applies), so version-specific runs need no extra flags:
# go-sdk's server pins -stateless=false for the dated-spec suites; its
# specOverrides['2026-07-28'] entry swaps in the stateless invocation, so
# this is the whole command:
npm start -- sdk go-sdk --mode server --suite all --spec-version 2026-07-28
# same for csharp-sdk (stateless URL), rust-sdk (STATELESS=1 env), and
# python-sdk (per-revision expected-failures baseline)
Explicit CLI flags still beat everything, config included — overriding a field for a one-off run:
npm start -- sdk go-sdk@my-fork-branch --mode server \
--build-cmd 'go build -o ./.conformance-server ./experimental/server'
To add a new SDK to the matrix, add an entry to KNOWN_SDKS.
Clones are cached under .sdk-under-test/ and reused (fetched) on subsequent runs.
SDK Tier Assessment
The tier-check subcommand evaluates an MCP SDK repository against SEP-1730 (the SDK Tiering System). There are two ways to run it, and they answer different questions.
1. The CLI, for the deterministic half
Conformance pass rates, issue triage, P0 resolution, labels, releases, policy files. No AI, no judgment, reproducible.
For a known SDK (see src/sdk-runner/known-sdks.ts), one command does the whole
thing — clone, build, and for each shipped revision start the server
invocation that SDK's config declares for that revision, run both legs at that
revision's wire, and merge:
gh auth login
npx @modelcontextprotocol/conformance tier-check --sdk go-sdk
# or against a checkout you already have:
npx @modelcontextprotocol/conformance tier-check --sdk-path ~/src/go-sdk
Per-revision resolution is the point, not a convenience. Some SDKs fix their
wire era per server process (go-sdk only speaks 2026-07-28 when started
stateless) or per endpoint (csharp-sdk serves it at /stateless), so no single
server URL can be measured against two revisions. The per-SDK, per-revision
invocations live in known-sdks.ts specOverrides; adding an SDK there is how
it joins the matrix.
For an SDK not in the config, drive it directly — but then the one endpoint you give must serve every claimed revision at its own wire, and the server must already be running:
npx @modelcontextprotocol/conformance tier-check \
--repo modelcontextprotocol/typescript-sdk \
--conformance-server-url http://localhost:3000/mcp \
--client-cmd '<command that runs the SDK conformance client>' \
--requirements 2025-11-25,2026-07-28
Omit --client-cmd and the client leg is skipped and reported as a gap. Omit
--requirements and scoring falls back to the suite as it stands today, which is
not what you want for a tier claim; see Conformance Requirements.
The exit code is a CI verdict on the machine-checkable half: nonzero when any scored conformance scenario fails or a leg could not be measured. Governance findings (triage, P0s, policy files) shape the tier but never the exit code.
2. The skill, for the whole assessment
The CLI cannot judge documentation coverage, dependency policy or roadmap quality, and those decide the tier as much as conformance does. The mcp-sdk-tier-audit skill runs the CLI, adds those evaluations, and writes a full report with a remediation plan. In Claude Code, from a checkout of this repo:
/mcp-sdk-tier-audit <local-sdk-path> <conformance-server-url> '<client-cmd>' --requirements 2025-11-25,2026-07-28
The server must already be running and stay up for the whole audit. Expect a few minutes.
Reading the result
Scored against 2025-11-25 and 2026-07-28, each run at its own wire version.
Server 67/67 required scenarios (100%)
Client 50/50 required scenarios (100%)
Not scored (8 run, 4 failing, no effect on tier):
✗ auth/dpop (extension)
✗ json-schema-2020-12-preservation (added-after-release)
Tier 1 Blockers:
• triage
• p0_resolution
- Required scenarios are the only ones that move the number.
67/67spans every revision listed: a scenario belonging to both runs once per revision, on that revision's wire, and both have to pass. - Not scored ran and is reported so you can see it, but cannot fail the tier.
extensionmeans optional by definition;added-after-releasemeans the scenario did not exist when that revision shipped. A failure here is information, not a blocker. - Not measured is different from either, and means the run could not happen at all, e.g. a requirement set naming a scenario this build no longer has. Treat it as a broken invocation, never as an SDK failure.
- Tier 1 blockers lists every requirement short of Tier 1. Conformance absent from that list means the SDK met every requirement each listed revision imposes.
The exit code follows the same rule: it reflects required scenarios only, so an implementation that meets a revision's requirements exits 0 even with failing extensions.
In --output json, passed / failed / total describe the scored set and so always agree with pass_rate; anything run without being scored is counted separately under not_scored.
Architecture
See src/runner/DESIGN.md for detailed architecture documentation.
Key Components
- Runner (
src/runner/) - Orchestrates test execution and result generationclient.ts- Client testing implementationserver.ts- Server testing implementationutils.ts- Shared utilitiesindex.ts- Public API exports
- CLI (
src/index.ts) - Command-line interface using Commander.js - Scenarios (
src/scenarios/) - Test scenarios with expected behaviors - Checks (
src/checks/) - Conformance validation functions - Types (
src/types.ts) - Shared type definitions
Adding New Scenarios
- Create a new directory in
src/scenarios/<scenario-name>/ - Implement the
Scenariointerface withstart(),stop(), andgetChecks() - Register the scenario in
src/scenarios/index.ts
See src/scenarios/initialize/ for a reference implementation.