Mirror of https://github.com/modelcontextprotocol/conformance — Conformance Tests for MCP
- TypeScript 98.1%
- JavaScript 1.9%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
39f34b2c58* Add prepare script for git dependency installs
* Add conformance tests for SEP-2243 HTTP Standardization
* Add traceability file for SEP-2243
* Fix prettier formatting in http-standard-headers.ts
* Improve SEP-2243 conformance test coverage
Close gaps in HTTP header conformance scenarios:
Client standard headers (http-standard-headers.ts):
- Enforce all expected methods in checks, failing any that were not observed
- Track Mcp-Name header per-method separately from Mcp-Method
- Advertise resources and prompts capabilities so clients exercise those endpoints
- Add a prompt entry for prompts/get testing
Client custom headers (http-custom-headers.ts):
- Add Base64 encoding checks for non-ASCII, whitespace, and control-char values
- Add null/omitted parameter test (second tool call with null value)
- Add client-keeps-valid-tool check verifying clients still call valid tools
after filtering out invalid ones
Server header validation (server/http-standard-headers.ts):
- Replace fetch-based sendRawRequest with http.request to preserve exact
header casing on the wire (fetch/Headers lowercases names)
- Compute defaultArgs and defaultHeaders from tool schema so test requests
satisfy all required parameters while varying only the param under test
Traceability (sep-2243.yaml):
- Add 7 new spec-to-check mappings (nonascii, whitespace, controlchar,
keeps-valid-tool, literal-missing-base64-prefix/suffix, no-mirror-unannotated)
- Fix 2 incorrect existing mappings
* Fill SEP-2243 conformance test coverage gaps
Add missing test cases identified by comparing the branch against the
SEP-2243 spec's Conformance Test Cases section:
- Mcp-Method on notifications: add notifications/initialized to
expectedMethods so clients are checked for header on notifications
- Boolean true: add debug=true param to complement verbose=false
- Leading-space-only and trailing-space-only: separate checks for
Base64 encoding when only one edge has whitespace
- Internal spaces only: verify plain ASCII (no Base64) for values
like 'us west 1' that have spaces only in the middle
- CRLF string: add line1\r\nline2 test (complements existing \n test)
- Leading tab: add \tindented test for tab-triggered Base64 encoding
- Server rejects invalid Mcp-Param chars: mark as excluded in YAML
since HTTP itself prevents these characters in headers
Register all new checks in sep-2243.yaml.
* Remove case-insensitive =?base64? prefix test
The spec states header values are case-sensitive (RFC 9110). The
=?base64? prefix is part of the header value, not the header name,
so it should be matched case-sensitively. This was identified as a
bug in the conformance tests per feedback on SEP-2243.
Changes:
- Remove server-accepts-case-insensitive-base64 check from server
validation scenario
- Change base64 prefix regex from case-insensitive (/i) to
case-sensitive in client header validation
- Remove corresponding entry from sep-2243.yaml
* Remove nested x-mcp-header rejection test
The spec constrains x-mcp-header to primitive types but does not
restrict it to top-level properties only. A nested string property
with x-mcp-header is valid per the spec. This was confirmed by
mikekistler in SEP-2243 PR discussion.
Removes invalid_nested_header from the invalid tool definitions
list and its corresponding sep-2243.yaml entry.
* Use numeric comparison for number header values
Compare number header values numerically instead of as strings to
accommodate cross-SDK floating point representation differences
(e.g., '42' vs '42.0'). For integers, exact numeric match is
required. For decimals, a tolerance of 1e-9 is allowed.
See SEP-2243 discussion on number precision.
* client/http-standard-headers: SKIPPED (not FAILURE) for unexercised methods; make getChecks() idempotent
A client that never calls prompts/list isn't violating SEP-2243 — the spec
sentence is 'The client MUST include the standard MCP request headers on each
POST request', and a request that was never sent can't fail it. The Mcp-Method
requirement is already proven by initialize/tools/list etc., so there's nothing
unique to prompts/list. SKIPPED keeps the gap visible without a false red.
Separately: getChecks() was pushing into this.checks on every call. The runner
may call it more than once (e.g. progress + final report); that produced
duplicate rows. Build a fresh array per call instead.
* client/http-standard-headers: de-dup guard in checkMcpNameHeader
checkMcpMethodHeader already has 'if (this.methodHeaderChecks.has(method)) return'.
checkMcpNameHeader was missing the equivalent. The test server advertises two
tools and two resources; a client that calls both produces two
'client-mcp-name-header-tools-call' / '...-resources-read' rows.
* client/http-standard-headers: replace -> /\//g for slug generation
String.replace with a string arg only replaces the first occurrence.
'notifications/initialized' is fine (one slash) but a method like
'notifications/resources/updated' would yield 'notifications-resources/updated'
in the check id. (replaceAll isn't in this project's TS lib target.)
* client/http-custom-headers: base64 regex (.+) -> (.*) so empty string round-trips
The spec doesn't forbid base64-encoding values that don't strictly need it.
A client that always wraps would send '=?base64??=' for empty_val: ''. The
'.+' wouldn't match, so the harness would compare '=?base64??=' === ''
literally and FAIL a compliant client. '(.*)' lets the empty payload decode
to '' and match.
* client/http-custom-headers: emit SUCCESS for no-mirror-unannotated
This check only pushed on FAILURE; a correct client produced no row at all,
so the pass was invisible in reports and couldn't be counted toward coverage.
Same-slug SUCCESS/FAILURE pair is the repo convention.
* client/http-custom-headers: drop redundant optional-present check
checkParamHeader('Verbose', ...) already pushes a FAILURE when the header is
missing. The explicit 'optional-present' block re-tests the same condition
under a second check id, doubling the row.
* server/http-standard-headers: defaultArgs must use real number/boolean, not strings
defaultArgs feeds the JSON-RPC body. With Record<string, string> + '0'/'false',
a server that validates inputSchema rejects on type mismatch — which is also a
400. Every header-rejection check then false-PASSes (server rejected for the
wrong reason), and server-accepts-valid-base64 false-FAILs a compliant server
because it never gets past schema validation.
Also handle 'integer' (JSON Schema distinguishes it from 'number').
* server/http-standard-headers: createAcceptanceCheck must also assert no JSON-RPC error in body
A server can return HTTP 200 with {error: {code: -32001, ...}} in the body.
Status-only acceptance lets that through as a pass.
* use DRAFT_PROTOCOL_VERSION constant instead of 'DRAFT-2026-v1' literal
The literal will rot when the draft cycle rolls over. types.ts already exports
DRAFT_PROTOCOL_VERSION for this.
* sep-2243.yaml: spec-first rewrite + move to src/seps/
Regenerated from the SEP-2243 spec diff (transports.mdx + tools.mdx) rather
than from the scenario implementations: 21 check rows + 4 excluded vs the
previous 46 + 2. Differences:
- one check id per spec sentence (test variants go in 'details', not new ids)
- 'text:' quotes the spec sentence verbatim instead of paraphrasing
- '-reject-status' (MUST 400) split from '-reject-error-code' (SHOULD -32001
for standard headers, MUST for custom)
- rows with no spec backing dropped (whitespace, base64-padding/chars, prefix/
suffix-literal, control-char-name)
- two more SHOULD excludes (log-warning, no-sensitive-params)
- check ids use sep-2243-<slug> convention
- check: key first, excludes grouped at bottom (matches sep-2164)
Moved to src/seps/ to match #272 layout (this branch predates it; will
reconcile cleanly on rebase).
* add negative test for HttpStandardHeadersScenario
Self-contained vitest that POSTs initialize with and without Mcp-Method,
asserts FAILURE/SUCCESS on the pinned check id, and proves getChecks() is
idempotent. No example client/server file needed for this one because the
scenario *is* the server — the test crafts the bad request directly.
This is the negative-test half of AGENTS.md §Testing your scenario; the
everything-client passing-example half is still TODO.
* server/http-standard-headers: severity fixes for whitespace + malformed-base64 tests
These five tests assert behavior the SEP-2243 text doesn't pin down. Kept
because they're useful consistency signals across SDKs, but adjusted so they
don't FAIL spec-compliant servers:
- server-accepts-whitespace-header-value: this IS a MUST, but per RFC 9110
§5.5 (field parsing MUST exclude OWS), not SEP-2243. Kept as FAILURE,
specReference now cites RFC 9110.
- server-rejects-invalid-base64-padding / -chars: downgraded to INFO.
SEP-2243 says only 'MUST decode them accordingly' without picking RFC 4648
strict vs lenient. Node's Buffer.from() accepts unpadded/dirty input and
decodes to a matching value (server accepts); .NET's Convert.FromBase64String
throws (server rejects). Either is currently compliant. WARNING would force
Tier-1 SDKs into non-default strict decoding (#245); INFO records the
behavior without prescribing it, so cross-SDK divergence is visible
without tier impact.
- server-literal-missing-base64-prefix / -suffix: unchanged. The wrapper
syntax is '=?base64?{x}?=' complete; treating partial wrappers as literal
is the natural reading.
Plumbing: createRejectionCheck gains an optional failSeverity param;
testBase64Case gains a 'reject-info' mode. (This is also the hook for the
larger 400-MUST / -32001-SHOULD split that still needs doing.)
* server/http-standard-headers: split rejection check into status (MUST) + error-code (SHOULD/MUST)
SEP-2243 §Server Validation: 'servers MUST return HTTP status 400 ... and
SHOULD include a JSON-RPC error response using ... -32001'. So a server that
returns 400 with -32600 (or no error body) is compliant for standard headers.
The single conflated check FAILed it.
For custom headers (§Server Behavior for Custom Headers) -32001 IS a MUST,
so both halves stay FAILURE there.
createRejectionCheck → createRejectionChecks returning two rows: '<id>' for
the 400 status and '<id>-error-code' for -32001. testCase (standard) sets
errorCodeSeverity=WARNING; testBase64Case/testMissingCustomHeader (custom)
keep FAILURE; the two malformed-base64 INFO probes set both halves to INFO.
* rename check ids to sep-2243-* prefix; kebab-case throughout
Aligns with the sep-2243.yaml traceability ids and the repo convention
(sep-2164-*, sep-2207-*). Variant suffixes are kept (per-method, per-tool)
so per-case visibility isn't lost; the YAML's row id is a prefix of the
emitted ids, which is fine — extra ids not in the YAML are allowed.
Also fixes the mixed hyphen/underscore in reject-invalid-tool ids
(toolName has underscores; replace to hyphens).
* client: extract BaseHttpScenario to client/http-base.ts; both client scenarios extend it
http-standard-headers.ts had its own copy of the start/stop/handleRequest
boilerplate (~100 lines) that http-custom-headers.ts already abstracted as
BaseHttpScenario. Moved the abstract class to a shared file; both scenario
files now import it. HttpStandardHeadersScenario implements handlePost()
instead of handleRequest(); its handleInitialize() uses the base
sendInitialize() with the resources/prompts capability flags it needs.
Net: 465 → 361 lines for http-standard-headers.ts; the third inline copy of
the same HTTP-server scaffold is gone.
* use req.setEncoding('utf8') instead of per-chunk Buffer.toString()
Per-chunk .toString() corrupts a multi-byte UTF-8 char that straddles a TCP
chunk boundary (e.g. 0xC3 | 0xBC decodes to two replacement chars instead of
'ü'). The custom-headers scenario sends '日本語'/'naïve' in the body, so this
is reachable in principle. setEncoding('utf8') makes 'data' emit strings with
boundary handling done by Node's StringDecoder.
Fixed in both places: BaseHttpScenario.handleRequest (request body) and
sendRawRequest (response body).
* fix: migrate SEP-2243 scenarios from specVersions to source (introducedIn/removedIn)
The introducedIn/removedIn refactor (#265) replaced the specVersions array
with a source property on all scenario interfaces. The SEP-2243 scenarios
still used the old specVersions field, causing matchesSpecVersion() to
receive undefined and crash in spec-version.test.ts.
Replace specVersions with source = { introducedIn: DRAFT_PROTOCOL_VERSION }
in BaseHttpScenario and the two server ClientScenario classes. Remove the
now-unused SpecVersion imports.
* server/http-standard-headers: malformed-base64 padding/chars back to FAILURE
Per discussion on #259: the SEP-2243 conformance-test-case table is the
approved source of truth and lists these as reject cases, so they're
FAILURE-level even though the spec body itself only says 'MUST decode them
accordingly'. SDKs whose stdlib base64 is lenient (Node, browsers) will need
to validate before decoding; if that's burdensome we'll revisit.
Removes the now-unused 'reject-info' mode and statusSeverity/INFO plumbing
introduced for these two probes.
---------
Co-authored-by: Tarek Mahmoud Sayed <tarekms@microsoft.com>
Co-authored-by: Tarek Mahmoud Sayed <tarekms@ntdev.microsoft.com>
Co-authored-by: Tarek Mahmoud Sayed <10833894+tarekgh@users.noreply.github.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
|
||
| .claude/skills | ||
| .github/workflows | ||
| examples | ||
| src | ||
| .gitattributes | ||
| .gitignore | ||
| .prettierrc.json | ||
| action.yml | ||
| AGENTS.md | ||
| CONTRIBUTING.md | ||
| eslint.config.mjs | ||
| lefthook-local.example.yml | ||
| lefthook.yml | ||
| LICENSE | ||
| MAINTAINERS.md | ||
| package-lock.json | ||
| package.json | ||
| README.md | ||
| SDK_INTEGRATION.md | ||
| tsconfig.json | ||
| vitest.config.ts | ||
MCP Conformance Test Framework
A framework for testing MCP (Model Context Protocol) client and server implementations against the specification.
For SDK maintainers: See SDK Integration Guide for a streamlined guide on integrating conformance tests into your SDK repository.
Quick Start
Testing Clients
# Using the everything-client (recommended)
npx @modelcontextprotocol/conformance client --command "tsx examples/clients/typescript/everything-client.ts" --scenario initialize
# Run an entire suite of tests
npx @modelcontextprotocol/conformance client --command "tsx examples/clients/typescript/everything-client.ts" --suite auth
Testing Servers
# Run all server scenarios (default)
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp
# Run a single scenario
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --scenario server-initialize
List Available Scenarios
npx @modelcontextprotocol/conformance list
Overview
The conformance test framework validates MCP implementations by:
For Clients:
- Starting a test server for the specified scenario
- Running the client implementation with the test server URL
- Capturing MCP protocol interactions
- Running conformance checks against the specification
- Generating detailed test results
For Servers:
- Connecting to the running server as an MCP client
- Sending test requests and capturing responses
- Running conformance checks against server behavior
- Generating detailed test results
Usage
Client Testing
npx @modelcontextprotocol/conformance client --command "<client-command>" --scenario <scenario-name> [options]
Options:
--command- The command to run your MCP client (can include flags)--scenario- The test scenario to run (e.g., "initialize")--suite- Run a suite of tests in parallel (e.g., "auth")--spec-version <version>- Filter scenarios by spec version (e.g.,2025-11-25,DRAFT-2026-v1;draftis accepted as an alias for the current draft identifier). The draft version selects the latest dated release plus any draft-only scenarios--expected-failures <path>- Path to YAML baseline file of known failures (see Expected Failures)--timeout- Timeout in milliseconds (default: 30000)--verbose- Show verbose output
The framework appends <server-url> as an argument to your command and sets the MCP_CONFORMANCE_SCENARIO environment variable to the scenario name. For scenarios that require additional context (e.g., client credentials), the MCP_CONFORMANCE_CONTEXT environment variable contains a JSON object with scenario-specific data. When --spec-version is passed, its resolved value is forwarded to the client process as MCP_CONFORMANCE_PROTOCOL_VERSION; example clients can use this value directly as their protocolVersion. SDKs that hard-code their protocol version can ignore it.
Server Testing
npx @modelcontextprotocol/conformance server --url <url> [--scenario <scenario>]
Options:
--url- URL of the server to test--scenario <scenario>- Test scenario to run (e.g., "server-initialize"). Runs all available scenarios by default--suite <suite>- Suite to run: "active" (default), "all", or "pending"--expected-failures <path>- Path to YAML baseline file of known failures (see Expected Failures)--verbose- Show verbose output
Test Results
Client Testing - Results are saved to results/<scenario>-<timestamp>/:
checks.json- Array of conformance check results with pass/fail statusstdout.txt- Client stdout outputstderr.txt- Client stderr output
Server Testing - Results are saved to results/server-<scenario>-<timestamp>/:
checks.json- Array of conformance check results with pass/fail status
Expected Failures
SDKs that don't yet pass all conformance tests can specify a baseline of known failures. This allows running conformance tests in CI without failing, while still catching regressions.
Create a YAML file listing expected failures by mode:
# conformance-baseline.yml
server:
- tools-call-with-progress
- resources-subscribe
client:
- sse-retry
Then pass it to the CLI:
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --expected-failures ./conformance-baseline.yml
Exit code behavior:
| Scenario Result | In Baseline? | Outcome |
|---|---|---|
| Fails | Yes | Exit 0 — expected failure |
| Fails | No | Exit 1 — unexpected regression |
| Passes | Yes | Exit 1 — stale baseline, remove the entry |
| Passes | No | Exit 0 — normal pass |
This ensures:
- CI passes when only known failures occur
- CI fails on new regressions (unexpected failures)
- CI fails when a fix lands but the baseline isn't updated (stale entries)
GitHub Action
This repo provides a composite GitHub Action so SDK repos don't need to write their own conformance scripts.
Server Testing
steps:
- uses: actions/checkout@v4
# Start your server (SDK-specific)
- run: |
my-server --port 3001 &
timeout 15 bash -c 'until curl -s http://localhost:3001/mcp; do sleep 0.5; done'
- uses: modelcontextprotocol/conformance@v0.1.11
with:
mode: server
url: http://localhost:3001/mcp
expected-failures: ./conformance-baseline.yml # optional
Client Testing
steps:
- uses: actions/checkout@v4
- uses: modelcontextprotocol/conformance@v0.1.11
with:
mode: client
command: 'python tests/conformance/client.py'
expected-failures: ./conformance-baseline.yml # optional
Action Inputs
| Input | Required | Description |
|---|---|---|
mode |
Yes | server or client |
url |
Server mode | URL of the server to test |
command |
Client mode | Command to run the client under test |
expected-failures |
No | Path to YAML baseline file |
suite |
No | Test suite to run |
scenario |
No | Run a single scenario by name |
timeout |
No | Timeout in ms for client tests (default: 30000) |
verbose |
No | Show verbose output (default: false) |
node-version |
No | Node.js version (default: 20) |
Example Clients
examples/clients/typescript/everything-client.ts- Single client that handles all scenarios based on scenario name (recommended)examples/clients/typescript/test1.ts- Simple MCP client (for reference)examples/clients/typescript/auth-test.ts- Well-behaved OAuth client (for reference)
Available Scenarios
Client Scenarios
- initialize - Tests MCP client initialization handshake
- Validates protocol version
- Validates clientInfo (name and version)
- Validates server response handling
- tools-call - Tests tool invocation
- auth/basic-dcr - Tests OAuth Dynamic Client Registration flow
- auth/basic-metadata-var1 - Tests OAuth with authorization metadata
Server Scenarios
Run npx @modelcontextprotocol/conformance list --server to see all available server scenarios, including:
- server-initialize - Tests server initialization and capabilities
- tools-list - Tests tool listing endpoint
- tools-call-* - Various tool invocation scenarios
- resources-* - Resource management scenarios
- prompts-* - Prompt management scenarios
SDK Tier Assessment
The tier-check subcommand evaluates an MCP SDK repository against SEP-1730 (the SDK Tiering System):
# Without conformance tests (fastest)
gh auth login
npm run --silent tier-check -- --repo modelcontextprotocol/typescript-sdk --skip-conformance
# With conformance tests (start the everything server first)
npm run --silent tier-check -- \
--repo modelcontextprotocol/typescript-sdk \
--conformance-server-url http://localhost:3000/mcp
For a full AI-assisted assessment with remediation guide, use Claude Code:
/mcp-sdk-tier-audit <local-sdk-path> <conformance-server-url>
See .claude/skills/mcp-sdk-tier-audit/README.md for full documentation.
Architecture
See src/runner/DESIGN.md for detailed architecture documentation.
Key Components
- Runner (
src/runner/) - Orchestrates test execution and result generationclient.ts- Client testing implementationserver.ts- Server testing implementationutils.ts- Shared utilitiesindex.ts- Public API exports
- CLI (
src/index.ts) - Command-line interface using Commander.js - Scenarios (
src/scenarios/) - Test scenarios with expected behaviors - Checks (
src/checks/) - Conformance validation functions - Types (
src/types.ts) - Shared type definitions
Adding New Scenarios
- Create a new directory in
src/scenarios/<scenario-name>/ - Implement the
Scenariointerface withstart(),stop(), andgetChecks() - Register the scenario in
src/scenarios/index.ts
See src/scenarios/initialize/ for a reference implementation.