Mirror of https://github.com/modelcontextprotocol/conformance — Conformance Tests for MCP
- TypeScript 98.1%
- JavaScript 1.9%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
82c1195f8a* feat: vendor spec schema types per-version
Copies schema/{version}/schema.ts from the modelcontextprotocol spec
repo into src/spec-types/{version}.ts so the conformance suite can type
against draft spec versions before any SDK ships them.
npm run sync-schema -- <ref> refreshes the copies and records the spec
commit in src/spec-types/SOURCE.
* feat: add Connection abstraction and RunContext
Connection encapsulates how the conformance suite talks to a
server-under-test for a given spec version:
- connectStateful: 2025-x lifecycle. Thin adapter over the SDK Client
(initialize handshake, session id, SSE handled by the SDK).
- connectStateless: 2026-x lifecycle (SEP-2575). Raw fetch with
per-request _meta + MCP-Protocol-Version header. Decoupled from the
SDK so the suite can test draft spec versions before the SDK
implements them.
connectFor(specVersion) picks the implementation. RunContext bundles
serverUrl, specVersion and a bound connect() for the runner to hand to
each scenario.
Nothing uses this yet; wiring follows in the next commit.
* refactor: thread RunContext through ClientScenario.run
ClientScenario.run(serverUrl) becomes run(ctx: RunContext). The runner
builds the context from --spec-version and the server URL; scenarios
destructure ctx.serverUrl and otherwise behave identically.
No scenario uses ctx.connect() yet, so behaviour is unchanged: 214/214
tests pass, all-scenarios.test.ts still drives the everything-server
fixture exactly as before.
Test files use a testContext(url) helper to construct a RunContext.
The authorization-server scenario list is retyped to
ClientScenarioForAuthorizationServer since those scenarios test an
OAuth server, not an MCP server, and keep run(serverUrl).
* refactor: migrate server scenarios to ctx.connect() + conn.request()
22 carry-forward and lifecycle scenarios now go through the Connection
abstraction instead of connectToServer + SDK Client:
- tools.ts (8): list, call x6, with-progress
- prompts.ts (5)
- resources.ts (7): list, read x3, subscribe x2, not-found
- utils.ts (3): completion, ping, set-level
- json-schema-2020-12.ts, caching.ts, http-standard-headers.ts
Result types come from spec-types/{introducedIn}. Same scenario code
now passes under both --spec-version 2025-11-25 (SDK-backed stateful)
and --spec-version draft (raw stateless).
ToolsCallSampling/Elicitation/WithLogging and elicitation-* keep
connectToServer (need setRequestHandler/setLoggingLevel SDK surface);
they are tagged removedIn: DRAFT in the next commit. stateless.ts and
input-required-result.ts keep their sendRpc helper; migrating those is
deferred to the DRAFT-scenario coherence pass.
* fix: normalize Connection error to JsonRpcError; clean up RunContext import
connectStateful now catches the SDK's McpError and rethrows as
JsonRpcError so scenarios always see the same error class regardless
of which Connection impl ran. ResourcesNotFoundError uses
instanceof JsonRpcError instead of duck-typing.
types.ts uses a normal top-level import for RunContext instead of an
inline import() type.
* fix(dns-rebinding): use version-appropriate probe body
The 'valid Host accepted' check was sending an initialize body, which
a 2026 server returns 404 for. Probe body is now picked from
ctx.specVersion: initialize for the stateful lifecycle, server/discover
with _meta for the stateless lifecycle. The Host/Origin rejection check
is unchanged since rejection happens before body parsing.
* feat(everything-server): route stateless carry-forward methods to McpServer
Adds an in-memory dispatch client connected to the same McpServer the
stateful path uses. Stateless requests for tools/call, resources/*,
prompts/get and completion/complete that fall through the MRTR-specific
handlers are routed to it, so the fixture serves the carry-forward
scenarios under --spec-version draft without duplicating ~500 lines of
tool/resource/prompt registrations.
tools/list now merges the McpServer's tool list with the MRTR-only
stubs so json-schema-2020-12 finds its tool.
draft suite against the fixture: 36/39 (was 13/39). Remaining 3 are
fixture-side SEP gaps (no SSE forwarding for progress, no SEP-2549
ttlMs, no SEP-2243 Mcp-Method validation in the stateless path).
* refactor(connection): drop unused RequestOptions; move sdk-client; add tests
Addresses self-review findings on the new connection module:
- RequestOptions.handlers/.meta and ServerRequestHandler removed: zero
callers (the scenarios that motivated them are deferred). They can be
reintroduced when something actually uses them.
- scenarios/server/client-helper.ts moved to connection/sdk-client.ts so
the connection module no longer depends on the scenarios module.
- connectStateless: handle CRLF SSE event separators; throw a useful
error for non-JSON/non-SSE responses instead of a JSON parse error.
- JSONRPCNotification consistently imported from spec-types/2025-11-25.
- New connection.test.ts (8 tests) covering connectFor selection,
_meta injection, error mapping, SSE LF/CRLF parsing, and the
server-request-on-stream rejection.
* fix: address bughunt findings (response.ok check; targetVersion naming)
- connectStateless: throw on non-2xx responses that lack a JSON-RPC
error envelope (e.g. gateway/framework error JSON), matching the
stateful path's behavior. Previously such a response would return
undefined as the result.
- json-schema-2020-12: rename negotiatedVersion to targetVersion and
reword the skip message and details field. The value is ctx.specVersion
(the run's --spec-version), not a negotiated version; the old wording
was misleading. Drop dead 'unknown' fallback (specVersion is required)
and the corresponding undefined test cases.
* fix(sse-multiple-streams): keep scenario in draft; version-aware request scaffolding
Concurrent POSTs each answered with JSON or their own SSE stream are
core transport behavior in the draft spec too, so removedIn was wrong
for this scenario. Only the request scaffolding is version-specific:
stateful runs keep the initialize/session-id setup, draft runs send
_meta + the MCP-Protocol-Version header on each request instead. The
two checks are unchanged.
* feat: add MockServer abstraction and ScenarioContext
MockServer encapsulates the lifecycle scaffold a client-conformance
scenario presents to the client-under-test:
- createServerStateful: 2025-x lifecycle. SDK Server +
StreamableHTTPServerTransport (sessionless mode); the SDK handles the
initialize handshake.
- createServerStateless: 2026-x lifecycle (SEP-2575). Raw express app
that validates _meta + MCP-Protocol-Version on every request, serves
server/discover, routes other methods to the supplied handlers.
createServerFor(specVersion) picks the implementation. ScenarioContext
bundles specVersion and a bound createServer() for the runner to hand
to each scenario.
This is the client-conformance mirror of src/connection (PR #318).
Nothing uses it yet; wiring follows in the next commit.
* refactor: thread ScenarioContext through Scenario.start()
Scenario.start() becomes start(ctx: ScenarioContext). The runner builds
the context from --spec-version (defaulting to LATEST_SPEC_VERSION) and
passes it through; scenarios receive it as _ctx and otherwise behave
identically.
No scenario uses ctx.createServer() yet, so behaviour is unchanged:
231/231 tests pass.
Test files use a testScenarioContext() helper. The runner already
threads MCP_CONFORMANCE_PROTOCOL_VERSION to the spawned client process,
so the fixture-side env wiring is unchanged.
* refactor: migrate tools_call to ctx.createServer; tag 2025-only client scenarios
ToolsCallScenario now goes through ctx.createServer() instead of an
inline express + SDK Server build. Same handlers, same checks; the
assertion now reads from srv.recorded so it works regardless of which
lifecycle scaffold the runner picked.
initialize, sse-retry, and elicitation-defaults are tagged
removedIn: DRAFT (initialize/GET-SSE/SSE-embedded-elicitation are gone
in the 2026 lifecycle; the MRTR sibling for elicitation-defaults is a
follow-up).
spec-version.test.ts: the 'draft is a superset of latest' invariant no
longer holds once removedIn: DRAFT exists; the test now asserts that
any scenario in latest-but-not-draft is explicitly removedIn.
* feat(auth): make createServer helper version-aware via ScenarioContext
The auth helper now takes ctx: ScenarioContext as its first argument
and branches on ctx.specVersion inside the /mcp route: the stateful
path (SDK Server + StreamableHTTPServerTransport) is unchanged; under
the draft version a raw stateless handler validates _meta + the
MCP-Protocol-Version header, serves server/discover, and routes the
same tools/list and tools/call responses.
The PRM endpoint, bearer-auth middleware, and request logger sit above
the branch and are version-independent.
All 25 call sites across the 12 auth scenario files pass ctx through;
ServerLifecycle and the express.Application return type are unchanged
so stop()/getChecks() are untouched.
Deviation from the MockServer wrapper approach: keeping the helper's
return type as express.Application avoids restructuring 25 call sites'
ServerLifecycle handling in this PR. Folding the auth seam onto
ctx.createServer() fully is a follow-up once the lifecycle ownership
moves into MockServer.
* feat(everything-client): pick stateless requester by MCP_CONFORMANCE_PROTOCOL_VERSION
Adds a statelessRequest(serverUrl, method, params) helper that POSTs
with _meta + MCP-Protocol-Version (the SEP-2575 lifecycle), shimming
around the SDK Client not yet supporting stateless mode. The
runRequestMetadataClient handler's meta constants are extracted to share
with the helper.
runBasicClient (initialize, tools_call, json-schema-ref-no-deref) now
branches on MCP_CONFORMANCE_PROTOCOL_VERSION: for the draft version it
uses statelessRequest to call tools/list then tools/call; for dated
versions it keeps the SDK Client path.
The runner already passes MCP_CONFORMANCE_PROTOCOL_VERSION to the
spawned client, so no runner change is needed.
* fix: address review findings on MockServer (dead opts, shared validator, capability derivation, recorded parity, specVersion threading)
- MockServerOptions removed (capabilities/configure had zero callers); opts
param dropped from createServerStateful/Stateless/For and ScenarioContext.
- validateStatelessRequest extracted from mock-server/stateless and exported;
both the stateless MockServer and auth/helpers/createServer.ts call it so
_meta/header/version validation cannot drift.
- isStatefulVersion exported from connection/select; mock-server/select uses
it instead of duplicating the version set.
- runner/client.ts: env MCP_CONFORMANCE_PROTOCOL_VERSION set unconditionally
to the resolved version; runInteractiveMode now takes specVersion and the
CLI passes it.
- createServerStateful: capabilities derived from handler method prefixes;
newServer() moved inside the try so a capability mismatch surfaces as
JSON-RPC -32603 instead of an HTML 500. Recording moved to the express
layer so unregistered methods are captured (parity with stateless).
- readFinalSseMessage return type now declares error.data.
- Tests added for the capability derivation and unregistered-method recording.
* fix(mock-server): record stateless requests before validation; document preamble exclusions
The stateless impl pushed to recorded[] only after validateStatelessRequest
passed, so requests rejected for a missing header or _meta never appeared,
diverging from the stateful impl which records at the HTTP layer before SDK
dispatch. Record before validation, excluding only the server/discover
lifecycle preamble (mirroring the stateful initialize/notifications-initialized
exclusion), and fix the MockServer.recorded docstring which wrongly claimed
stateless excludes nothing.
* refactor(mock-server): tri-state result for validateStatelessRequest (reject/handled/route)
A successful server/discover was previously modeled as {ok: false},
which read as a validation failure but actually meant "already
answered, do not route". Replace the boolean discriminant with an
explicit kind: 'reject' (validation failed), 'handled' (valid and
already answered), or 'route' (dispatch to handlers).
Both consumers (the stateless /mcp route and the auth helper's
handleStateless) still write status/body for reject and handled and
route only on 'route', so responses are unchanged. Add direct unit
tests covering each branch of the new union.
* feat(runner): single-source the version→lifecycle mapping; export MCP_CONFORMANCE_LIFECYCLE
The stateless mock and the example client both hardcoded the draft
protocol version, so dating the draft (or adding a second stateless
version) would silently break them: the mock would keep rejecting
everything but the old constant, and the client would fall back to the
stateful path.
- Derive STATELESS_SPEC_VERSIONS and a lifecycleFor() helper from
isStatefulVersion() in connection/select.ts, making it the single
source of truth for the version→lifecycle mapping.
- Thread the resolved spec version into the stateless mock:
createServerFor() binds it into createServerStateless(), and
validateStatelessRequest() now takes the supported-versions list as a
parameter. A mock bound to a version accepts exactly that version and
echoes it in -32004 rejections and server/discover; unbound mocks
accept every known stateless version.
- Export MCP_CONFORMANCE_LIFECYCLE (stateful|stateless) to client
processes alongside MCP_CONFORMANCE_PROTOCOL_VERSION, in both the
runner and the in-process test helper, so clients in any language can
pick the lifecycle without keeping their own version map.
- everything-client: import DRAFT_PROTOCOL_VERSION from src/types
instead of duplicating the string, branch on MCP_CONFORMANCE_LIFECYCLE
(falling back to the version comparison for older runners), and send
the runner-resolved version on stateless requests.
- Document MCP_CONFORMANCE_LIFECYCLE in the README and ignore the local
.claude/ workspace dir in .prettierignore so prettier --check . is not
tripped by untracked worktree checkouts.
* fix(everything-client): list-only flow for json-schema-ref-no-deref
runBasicClient now follows tools/list with a tools/call on the first
listed tool, but the json-schema-ref-no-deref scenario mock only serves
tools/list, so the call got -32601 and the client exited non-zero.
Register the scenario with a dedicated handler that stops after listing
tools, keeping the existing lifecycle branch (raw stateless request vs
SDK client). Listing is the whole flow for SEP-2106: the check is that
the client never fetches the network $ref canary while processing the
advertised schema.
* fix(tools_call): make getChecks() idempotent
getChecks() pushed into an instance array on every call, so calling it
more than once accumulated duplicate check entries. The runner is
allowed to call getChecks() repeatedly, and other scenarios already
build their result fresh per call for this reason.
Return a freshly built array instead and drop the now-unused instance
field. Add a unit test covering the not-called FAILURE case and
asserting repeated getChecks() calls do not grow the result.
* fix: ensure dispatch and transport cleanup runs when handlers throw
The stateless tools/list and prompts/list handlers in the everything
server awaited dispatch.close() after the in-memory request, so a throw
from dispatch.client.request() leaked the Client+Server+Transport pair.
Wrap both in try/finally, matching the tools/call and catch-all
handlers in the same file.
In the stateful mock server, the res 'close' listener that tears down
the per-request SDK transport and server was registered after the
handleRequest await; a throw there skipped registration and leaked the
pair. Register the listener before handing the request to the
transport.
* fix(everything-client): send Accept header on stateless requests
The draft transport spec requires POST requests to accept both
application/json and text/event-stream. Servers built on the SDK's
StreamableHTTPServerTransport already enforce this and reject
requests without it. Does not yet make json-schema-ref-no-deref pass
under --spec-version draft: the SDK transport also rejects the draft
protocol version itself, which is the known fixture gap until the
2026-native scenarios migrate onto MockServer.
* chore: exclude .claude/ from vitest globs
Local agent worktree checkouts under .claude/worktrees/ carry their
own copies of the test suite, which the bare **/*.test.ts include was
adding to every run (and to the pre-push hook). Mirrors the existing
.prettierignore entry.
* fix(request-metadata): use spec error code -32004 for unsupported protocol version
The draft spec defines -32004 (UnsupportedProtocolVersionError) with
data { supported, requested } and HTTP 400 for protocol version
rejections; -32001 (HeaderMismatch) is reserved for HTTP header
validation failures.
- request-metadata scenario: return -32004 with spec-shaped data for
the simulated version-negotiation rejection (was -32001)
- everything-client: trigger negotiation on -32004 (was -32001)
- stateless mock server: include the requested version in the -32004
error data alongside the supported list
- update tests to match and assert the new requested field
* fix(auth): register transport cleanup before handleRequest in createServer
The res 'close' listener that tears down the transport/server pair was
registered after awaiting handleRequest, so a throw mid-request skipped
cleanup registration entirely. Register it first, matching the stateful
mock server.
* fix(everything-server): return JSON-RPC errors from stateless list dispatch on failure
The stateless tools/list and prompts/list dispatch handlers had
try/finally but no catch, so a dispatch failure produced an unhandled
rejection instead of a response. Add the same catch-and-respond block
the carry-forward dispatch handler already uses.
* refactor(runner): drop MCP_CONFORMANCE_LIFECYCLE; derive lifecycle from protocol version
The protocol version fully determines the lifecycle, and deriving it is
part of being a conformant client. The in-repo example client imports
the stateless version set from src/, so there is no drift risk, and a
single env var cannot disagree with itself. README documents the
version→lifecycle mapping for clients in other languages.
* fix(everything-server): return JSON-RPC errors from stateless resource list dispatch on failure
resources/list and resources/templates/list arrived from main with
try/finally but no catch, unlike the sibling tools/list and
prompts/list handlers. A throwing dispatch would escape to Express's
HTML 500 instead of a JSON-RPC error. Mirrors the existing pattern.
* fix(everything-server): include requested field in -32004 error data
The spec's UnsupportedProtocolVersionError requires data to carry both
supported and requested. The mock-server and request-metadata -32004
producers already include it; this was the remaining producer.
* feat(runner): client-side spec-version inference, skip, and --force
Mirrors the server runner's policy from #318: an explicitly requested
--spec-version outside a scenario's applicability window skips it
(exit 0) unless --force; when --spec-version is omitted the version is
inferred from the scenario's applicability instead of defaulting to
the latest dated version.
* fix(auth): gate SEP-837 application_type check to spec versions that include it
The sep-837-application-type-present check fired on every Dynamic Client
Registration regardless of the targeted spec version, but SEP-837 only
exists in the draft spec. Clients conforming to 2025-11-25 or earlier were
failed on a requirement that is not part of the version under test.
Thread the ScenarioContext into createAuthServer (mirroring createServer)
and only emit the check when the resolved spec version includes SEP-837,
i.e. the stateless draft lifecycle. At dated versions the check is not
emitted at all. Audited the other auth helper checks (PKCE, metadata
discovery, token verification); all of them enforce requirements that are
already in the dated specs, so SEP-837 was the only draft-only check.
Unit tests cover both sides: the check fails an omitting client at the
draft version and is absent at 2025-11-25.
* feat(sdk-runner): --spec-version passthrough with per-SDK default
The sdk command forwarded --scenario/--suite/--timeout to the underlying
client/server invocation but had no way to set the spec version, so SDK
runs always used the suite default. Add a --spec-version flag that is
passed through to the inner run.
Also add an optional specVersion field to the per-SDK config, used as the
default when the flag isn't given; an explicit --spec-version overrides
it. Set typescript-sdk-v1 to 2025-11-25: the v1 line targets the latest
dated spec, which keeps draft-only scenarios out of its runs by default.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
|
||
| .claude/skills | ||
| .github/workflows | ||
| examples | ||
| scripts | ||
| src | ||
| .gitattributes | ||
| .gitignore | ||
| .prettierignore | ||
| .prettierrc.json | ||
| action.yml | ||
| AGENTS.md | ||
| CONTRIBUTING.md | ||
| eslint.config.mjs | ||
| lefthook-local.example.yml | ||
| lefthook.yml | ||
| LICENSE | ||
| MAINTAINERS.md | ||
| package-lock.json | ||
| package.json | ||
| README.md | ||
| SDK_INTEGRATION.md | ||
| tsconfig.json | ||
| vitest.config.ts | ||
MCP Conformance Test Framework
A framework for testing MCP (Model Context Protocol) client and server implementations against the specification.
For SDK maintainers: See SDK Integration Guide for a streamlined guide on integrating conformance tests into your SDK repository.
Quick Start
Testing Clients
# Using the everything-client (recommended)
npx @modelcontextprotocol/conformance client --command "tsx examples/clients/typescript/everything-client.ts" --scenario initialize
# Run an entire suite of tests
npx @modelcontextprotocol/conformance client --command "tsx examples/clients/typescript/everything-client.ts" --suite auth
Testing Servers
# Run all server scenarios (default)
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp
# Run a single scenario
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --scenario server-initialize
List Available Scenarios
npx @modelcontextprotocol/conformance list
Overview
The conformance test framework validates MCP implementations by:
For Clients:
- Starting a test server for the specified scenario
- Running the client implementation with the test server URL
- Capturing MCP protocol interactions
- Running conformance checks against the specification
- Generating detailed test results
For Servers:
- Connecting to the running server as an MCP client
- Sending test requests and capturing responses
- Running conformance checks against server behavior
- Generating detailed test results
Usage
Client Testing
npx @modelcontextprotocol/conformance client --command "<client-command>" --scenario <scenario-name> [options]
Options:
--command- The command to run your MCP client (can include flags)--scenario- The test scenario to run (e.g., "initialize")--suite- Run a suite of tests in parallel:all,core,extensions,backcompat,auth,metadata,draft(scenarios targeting the in-progress draft spec), orsep-835--spec-version <version>- Filter scenarios by spec version (e.g.,2025-11-25,DRAFT-2026-v1;draftis accepted as an alias for the current draft identifier). The draft version selects the latest dated release plus any draft-only scenarios. When omitted, the version is inferred from the scenario's spec applicability (draft-only scenarios run at the draft version, everything else at the latest dated release); an explicitly requested version outside a scenario's applicability window skips the scenario (exit 0) unless--forceis passed--force- Run a scenario even if it is not applicable at the requested--spec-version--expected-failures <path>- Path to YAML baseline file of known failures (see Expected Failures)--timeout- Timeout in milliseconds (default: 30000)--verbose- Show verbose output
The framework appends <server-url> as an argument to your command and sets the MCP_CONFORMANCE_SCENARIO environment variable to the scenario name. For scenarios that require additional context (e.g., client credentials), the MCP_CONFORMANCE_CONTEXT environment variable contains a JSON object with scenario-specific data. When --spec-version is passed, its resolved value is forwarded to the client process as MCP_CONFORMANCE_PROTOCOL_VERSION; example clients can use this value directly as their protocolVersion. SDKs that hard-code their protocol version can ignore it. Clients under test must derive the lifecycle from the protocol version they are asked to run: dated versions through 2025-11-25 use the stateful lifecycle (initialize handshake), while the 2026 draft (DRAFT-2026-v1) uses the stateless lifecycle (per-request _meta).
Server Testing
npx @modelcontextprotocol/conformance server --url <url> [--scenario <scenario>]
Options:
--url- URL of the server to test--scenario <scenario>- Test scenario to run (e.g., "server-initialize"). Runs all available scenarios by default--suite <suite>- Suite to run: "active" (default; excludes pending and draft-spec scenarios), "all", "draft" (scenarios targeting the in-progress draft spec), or "pending"--expected-failures <path>- Path to YAML baseline file of known failures (see Expected Failures)--verbose- Show verbose output
Test Results
Client Testing - Results are saved to results/<scenario>-<timestamp>/:
checks.json- Array of conformance check results with pass/fail statusstdout.txt- Client stdout outputstderr.txt- Client stderr output
Server Testing - Results are saved to results/server-<scenario>-<timestamp>/:
checks.json- Array of conformance check results with pass/fail status
Expected Failures
SDKs that don't yet pass all conformance tests can specify a baseline of known failures. This allows running conformance tests in CI without failing, while still catching regressions.
Create a YAML file listing expected failures by mode:
# conformance-baseline.yml
server:
- tools-call-with-progress
- resources-subscribe
client:
- sse-retry
Then pass it to the CLI:
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --expected-failures ./conformance-baseline.yml
Exit code behavior:
| Scenario Result | In Baseline? | Outcome |
|---|---|---|
| Fails | Yes | Exit 0 — expected failure |
| Fails | No | Exit 1 — unexpected regression |
| Passes | Yes | Exit 1 — stale baseline, remove the entry |
| Passes | No | Exit 0 — normal pass |
This ensures:
- CI passes when only known failures occur
- CI fails on new regressions (unexpected failures)
- CI fails when a fix lands but the baseline isn't updated (stale entries)
GitHub Action
This repo provides a composite GitHub Action so SDK repos don't need to write their own conformance scripts.
Server Testing
steps:
- uses: actions/checkout@v4
# Start your server (SDK-specific)
- run: |
my-server --port 3001 &
timeout 15 bash -c 'until curl -s http://localhost:3001/mcp; do sleep 0.5; done'
- uses: modelcontextprotocol/conformance@v0.1.11
with:
mode: server
url: http://localhost:3001/mcp
expected-failures: ./conformance-baseline.yml # optional
Client Testing
steps:
- uses: actions/checkout@v4
- uses: modelcontextprotocol/conformance@v0.1.11
with:
mode: client
command: 'python tests/conformance/client.py'
expected-failures: ./conformance-baseline.yml # optional
Action Inputs
| Input | Required | Description |
|---|---|---|
mode |
Yes | server or client |
url |
Server mode | URL of the server to test |
command |
Client mode | Command to run the client under test |
expected-failures |
No | Path to YAML baseline file |
suite |
No | Test suite to run |
scenario |
No | Run a single scenario by name |
timeout |
No | Timeout in ms for client tests (default: 30000) |
verbose |
No | Show verbose output (default: false) |
node-version |
No | Node.js version (default: 20) |
Example Clients
examples/clients/typescript/everything-client.ts- Single client that handles all scenarios based on scenario name (recommended)examples/clients/typescript/test1.ts- Simple MCP client (for reference)examples/clients/typescript/auth-test.ts- Well-behaved OAuth client (for reference)
Available Scenarios
Client Scenarios
- initialize - Tests MCP client initialization handshake
- Validates protocol version
- Validates clientInfo (name and version)
- Validates server response handling
- tools-call - Tests tool invocation
- auth/basic-dcr - Tests OAuth Dynamic Client Registration flow
- auth/basic-metadata-var1 - Tests OAuth with authorization metadata
Server Scenarios
Run npx @modelcontextprotocol/conformance list --server to see all available server scenarios, including:
- server-initialize - Tests server initialization and capabilities
- tools-list - Tests tool listing endpoint
- tools-call-* - Various tool invocation scenarios
- resources-* - Resource management scenarios
- prompts-* - Prompt management scenarios
Running Against an SDK at a Specific Ref
The sdk subcommand clones an SDK repository at a given ref, builds it, and runs the local conformance build against it. This is the inner-loop tool for scenario authors and the basis for cross-SDK CI. Examples below use npm start -- so they run from source — no npm run build between edits.
--mode client or --mode server is required — each invocation tests exactly one side, so client and server are run (and pass/fail) independently.
# Run the client conformance suite against typescript-sdk @main (v2)
npm start -- sdk typescript-sdk --mode client
# Run the server conformance suite (separate invocation)
npm start -- sdk typescript-sdk --mode server
# A specific main-line SHA or branch (v2 monorepo)
npm start -- sdk typescript-sdk@abc123f --mode client
npm start -- sdk typescript-sdk@some-branch --mode server
# The published v1.x line — separate entry (npm build), defaults to the v1.x branch
npm start -- sdk typescript-sdk-v1 --mode client
npm start -- sdk typescript-sdk-v1@v1.29.0 --mode server
# Use an existing local checkout (no clone, no fetch)
npm start -- sdk --path ../typescript-sdk --skip-build --mode client
# Narrow to one scenario / suite
npm start -- sdk --path ../typescript-sdk --mode server --scenario server-initialize
npm start -- sdk typescript-sdk --mode client --suite auth
# Target a specific spec version (passed through to the underlying run).
# When omitted, the SDK's `specVersion` from KNOWN_SDKS is used, if set —
# e.g. typescript-sdk-v1 defaults to 2025-11-25.
npm start -- sdk typescript-sdk --mode client --spec-version draft
Build/run commands for each official SDK are looked up by name from src/sdk-runner/known-sdks.ts — no config file is required in the SDK repo. Resolution order is CLI flag > built-in entry, so any field can be overridden on the command line for refs that diverge from the built-in.
An SDK can have more than one entry when its layout differs across major versions — e.g. typescript-sdk (v2, the main monorepo) and typescript-sdk-v1 (the published npm v1.x line). An entry may set defaultRef (the branch used when you don't pass @<ref>) and repo (the real clone target when the entry name is an alias). Overriding for a one-off ref:
npm start -- sdk owner/go-sdk@some-branch \
--mode client \
--build-cmd 'go build -tags mcp_go_client_oauth -o ./.conformance-client ./conformance/everything-client' \
--client-cmd './.conformance-client'
To add a new SDK to the matrix, add an entry to KNOWN_SDKS.
Clones are cached under .sdk-under-test/ and reused (fetched) on subsequent runs.
SDK Tier Assessment
The tier-check subcommand evaluates an MCP SDK repository against SEP-1730 (the SDK Tiering System):
# Without conformance tests (fastest)
gh auth login
npm run --silent tier-check -- --repo modelcontextprotocol/typescript-sdk --skip-conformance
# With conformance tests (start the everything server first)
npm run --silent tier-check -- \
--repo modelcontextprotocol/typescript-sdk \
--conformance-server-url http://localhost:3000/mcp
For a full AI-assisted assessment with remediation guide, use Claude Code:
/mcp-sdk-tier-audit <local-sdk-path> <conformance-server-url>
See .claude/skills/mcp-sdk-tier-audit/README.md for full documentation.
Architecture
See src/runner/DESIGN.md for detailed architecture documentation.
Key Components
- Runner (
src/runner/) - Orchestrates test execution and result generationclient.ts- Client testing implementationserver.ts- Server testing implementationutils.ts- Shared utilitiesindex.ts- Public API exports
- CLI (
src/index.ts) - Command-line interface using Commander.js - Scenarios (
src/scenarios/) - Test scenarios with expected behaviors - Checks (
src/checks/) - Conformance validation functions - Types (
src/types.ts) - Shared type definitions
Adding New Scenarios
- Create a new directory in
src/scenarios/<scenario-name>/ - Implement the
Scenariointerface withstart(),stop(), andgetChecks() - Register the scenario in
src/scenarios/index.ts
See src/scenarios/initialize/ for a reference implementation.