modelcontextprotocol/conformance
Watch
1
0
Fork
You've already forked conformance
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: add SDK integration guide and fix spec references (#121)

Browse source 
* docs: add SDK integration guide and fix spec references

- Add SDK_INTEGRATION.md with guide for SDK maintainers on integrating conformance tests
- Link to guide from README.md
- Update spec references to use 2025-11-25 instead of draft URLs

* fix: prettier formatting and correct action version to v0.1.10
This commit is contained in:
Paul Carleton 2026-01-23 12:50:11 +00:00 • committed by GitHub
commit 2d52de5924
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 218 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

2
README.md
View file

@ -5,6 +5,8 @@ A framework for testing MCP (Model Context Protocol) client and server implement
> [!WARNING]
> This repository is a work in progress and is unstable. Join the conversation in the #conformance-testing-wg in the MCP Contributors discord.
**For SDK maintainers:** See [SDK Integration Guide](./SDK_INTEGRATION.md) for a streamlined guide on integrating conformance tests into your SDK repository.
## Quick Start
### Testing Clients

208
SDK_INTEGRATION.md Normal file
View file

@ -0,0 +1,208 @@
# Using MCP Conformance Tests in SDK Repositories
This guide explains how to integrate the MCP conformance test suite into your language SDK repository. The conformance framework tests your MCP implementation against the protocol specification to ensure compatibility.
## Quick Start
Install and run conformance tests:
```bash
# Client testing (framework starts a test server, runs your client against it)
npx @modelcontextprotocol/conformance client --command "your-client-command" --scenario initialize
# Server testing (your server must already be running)
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --scenario server-initialize
```
## Two Testing Modes
### Client Testing
The framework **starts a test server** and spawns your client against it. Your client receives the server URL as its final command-line argument.
```bash
# Run a single scenario
npx @modelcontextprotocol/conformance client \
--command "python tests/conformance/client.py" \
--scenario initialize
# Run a suite of tests
npx @modelcontextprotocol/conformance client \
--command "python tests/conformance/client.py" \
--suite auth
```
**Available client suites:** `all`, `core`, `extensions`, `auth`, `metadata`, `sep-835`
Your client should:
1. Accept the server URL as its last argument
2. Read `MCP_CONFORMANCE_SCENARIO` env var to determine which scenario is being tested
3. Read `MCP_CONFORMANCE_CONTEXT` env var for scenario-specific data (e.g., OAuth credentials)
### Server Testing
Your server must be **running before** invoking the conformance tool. The framework connects to it as an MCP client.
```bash
# Start your server first
your-server --port 3001 &
# Then run conformance tests
npx @modelcontextprotocol/conformance server \
--url http://localhost:3001/mcp \
--suite active
```
**Available server suites:** `active` (default), `all`, `pending`
**Note:** Server testing requires you to manage server lifecycle (start, health-check, cleanup) yourself.
---
## Expected Failures (Baseline) File
The expected-failures feature lets your CI pass while you work on fixing known issues. It catches regressions by failing when:
- A previously passing test starts failing (regression)
- A previously failing test starts passing (stale baseline - remove the entry)
### File Format
Create a YAML file (e.g., `conformance-baseline.yml`):
```yaml
server:
- tools-call-with-progress
- resources-subscribe
client:
- auth/client-credentials-jwt
```
### Usage
```bash
npx @modelcontextprotocol/conformance server \
--url http://localhost:3000/mcp \
--expected-failures ./conformance-baseline.yml
```
### Exit Code Behavior
| Scenario Result | In Baseline? | Exit Code | Meaning |
| --------------- | ------------ | --------- | ----------------------------- |
| Fails | Yes | 0 | Expected failure |
| Fails | No | 1 | Unexpected regression |
| Passes | Yes | 1 | Stale baseline - remove entry |
| Passes | No | 0 | Normal pass |
---
## GitHub Action
The conformance repo provides a reusable GitHub Action that handles Node.js setup and conformance execution.
### Client Testing Example
```yaml
name: Conformance Tests
on: [push, pull_request]
jobs:
conformance:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up your SDK
run: |
# Your SDK setup (pip install, npm install, etc.)
pip install -e .
- uses: modelcontextprotocol/conformance@v0.1.10
with:
mode: client
command: 'python tests/conformance/client.py'
suite: auth
expected-failures: ./conformance-baseline.yml
```
### Server Testing Example
```yaml
name: Conformance Tests
on: [push, pull_request]
jobs:
conformance:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up and start server
run: |
pip install -e .
python -m myserver --port 3001 &
# Wait for server to be ready
timeout 15 bash -c 'until curl -s http://localhost:3001/mcp; do sleep 0.5; done'
- uses: modelcontextprotocol/conformance@v0.1.10
with:
mode: server
url: http://localhost:3001/mcp
suite: active
expected-failures: ./conformance-baseline.yml
```
### Action Inputs
| Input | Required | Description |
| ------------------- | ----------- | ----------------------------------------------- |
| `mode` | Yes | `server` or `client` |
| `url` | Server mode | URL of the server to test |
| `command` | Client mode | Command to run the client |
| `expected-failures` | No | Path to YAML baseline file |
| `suite` | No | Test suite to run |
| `scenario` | No | Run a single scenario by name |
| `timeout` | No | Timeout in ms for client tests (default: 30000) |
| `verbose` | No | Show verbose output (default: false) |
| `node-version` | No | Node.js version (default: 20) |
---
## Writing Conformance Clients/Servers
### Example Client Pattern
See [`src/conformance/everything-client.ts`](https://github.com/modelcontextprotocol/typescript-sdk/blob/main/src/conformance/everything-client.ts) in the TypeScript SDK for a reference implementation. The recommended pattern is a single client that routes behavior based on the scenario:
```python
import os
import sys
import json
def main():
server_url = sys.argv[-1] # URL passed as last argument
scenario = os.environ.get("MCP_CONFORMANCE_SCENARIO", "")
context = json.loads(os.environ.get("MCP_CONFORMANCE_CONTEXT", "{}"))
if scenario.startswith("auth/"):
run_auth_scenario(server_url, scenario, context)
else:
run_default_scenario(server_url)
if __name__ == "__main__":
main()
```
### Example Server Pattern
See [`src/conformance/everything-server.ts`](https://github.com/modelcontextprotocol/typescript-sdk/blob/main/src/conformance/everything-server.ts) in the TypeScript SDK for a reference implementation that handles all server scenarios.
---
## Additional Resources
- [Conformance README](./README.md)
- [Design documentation](./src/runner/DESIGN.md)
- [TypeScript SDK conformance examples](https://github.com/modelcontextprotocol/typescript-sdk/tree/main/src/conformance)

16
src/scenarios/client/auth/spec-references.ts
View file

@ -19,15 +19,15 @@ export const SpecReferences: { [key: string]: SpecReference } = {
},
MCP_PRM_DISCOVERY: {
id: 'MCP-2025-06-18-PRM-discovery',
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#protected-resource-metadata-discovery-requirements'
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#protected-resource-metadata-discovery-requirements'
},
MCP_AUTH_DISCOVERY: {
id: 'MCP-Authorization-metadata-discovery',
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#authorization-server-metadata-discovery'
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#authorization-server-metadata-discovery'
},
MCP_DCR: {
id: 'MCP-Dynamic-client-registration',
url: 'https://modelcontextprotocol.io/specification/draft/basic/client#dynamic-client-registration'
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/client#dynamic-client-registration'
},
OAUTH_2_1_AUTHORIZATION_ENDPOINT: {
id: 'OAUTH-2.1-authorization-endpoint',
@ -39,23 +39,23 @@ export const SpecReferences: { [key: string]: SpecReference } = {
},
MCP_ACCESS_TOKEN_USAGE: {
id: 'MCP-Access-token-usage',
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#access-token-usage'
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#access-token-usage'
},
MCP_SCOPE_SELECTION_STRATEGY: {
id: 'MCP-Scope-selection-strategy',
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#scope-selection-strategy'
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#scope-selection-strategy'
},
MCP_SCOPE_CHALLENGE_HANDLING: {
id: 'MCP-Scope-challenge-handling',
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#scope-challenge-handling'
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#scope-challenge-handling'
},
MCP_AUTH_ERROR_HANDLING: {
id: 'MCP-Auth-error-handling',
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#error-handling'
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#error-handling'
},
MCP_CLIENT_ID_METADATA_DOCUMENTS: {
id: 'MCP-Client-ID-Metadata-Documents',
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#client-id-metadata-documents'
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#client-id-metadata-documents'
},
IETF_CIMD: {
id: 'IETF-OAuth-Client-ID-Metadata-Document',