docs: add SDK integration guide and fix spec references (#121)
* docs: add SDK integration guide and fix spec references - Add SDK_INTEGRATION.md with guide for SDK maintainers on integrating conformance tests - Link to guide from README.md - Update spec references to use 2025-11-25 instead of draft URLs * fix: prettier formatting and correct action version to v0.1.10
This commit is contained in:
Paul Carleton
GitHub
parent
245e44e882
commit
2d52de5924
1 changed files with 218 additions and 8 deletions
208
SDK_INTEGRATION.md
Normal file
208
SDK_INTEGRATION.md
Normal file
|
|
@ -0,0 +1,208 @@
|
|||
# Using MCP Conformance Tests in SDK Repositories
|
||||
|
||||
This guide explains how to integrate the MCP conformance test suite into your language SDK repository. The conformance framework tests your MCP implementation against the protocol specification to ensure compatibility.
|
||||
|
||||
## Quick Start
|
||||
|
||||
Install and run conformance tests:
|
||||
|
||||
```bash
|
||||
# Client testing (framework starts a test server, runs your client against it)
|
||||
npx @modelcontextprotocol/conformance client --command "your-client-command" --scenario initialize
|
||||
|
||||
# Server testing (your server must already be running)
|
||||
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --scenario server-initialize
|
||||
```
|
||||
|
||||
## Two Testing Modes
|
||||
|
||||
### Client Testing
|
||||
|
||||
The framework **starts a test server** and spawns your client against it. Your client receives the server URL as its final command-line argument.
|
||||
|
||||
```bash
|
||||
# Run a single scenario
|
||||
npx @modelcontextprotocol/conformance client \
|
||||
--command "python tests/conformance/client.py" \
|
||||
--scenario initialize
|
||||
|
||||
# Run a suite of tests
|
||||
npx @modelcontextprotocol/conformance client \
|
||||
--command "python tests/conformance/client.py" \
|
||||
--suite auth
|
||||
```
|
||||
|
||||
**Available client suites:** `all`, `core`, `extensions`, `auth`, `metadata`, `sep-835`
|
||||
|
||||
Your client should:
|
||||
|
||||
1. Accept the server URL as its last argument
|
||||
2. Read `MCP_CONFORMANCE_SCENARIO` env var to determine which scenario is being tested
|
||||
3. Read `MCP_CONFORMANCE_CONTEXT` env var for scenario-specific data (e.g., OAuth credentials)
|
||||
|
||||
### Server Testing
|
||||
|
||||
Your server must be **running before** invoking the conformance tool. The framework connects to it as an MCP client.
|
||||
|
||||
```bash
|
||||
# Start your server first
|
||||
your-server --port 3001 &
|
||||
|
||||
# Then run conformance tests
|
||||
npx @modelcontextprotocol/conformance server \
|
||||
--url http://localhost:3001/mcp \
|
||||
--suite active
|
||||
```
|
||||
|
||||
**Available server suites:** `active` (default), `all`, `pending`
|
||||
|
||||
**Note:** Server testing requires you to manage server lifecycle (start, health-check, cleanup) yourself.
|
||||
|
||||
---
|
||||
|
||||
## Expected Failures (Baseline) File
|
||||
|
||||
The expected-failures feature lets your CI pass while you work on fixing known issues. It catches regressions by failing when:
|
||||
|
||||
- A previously passing test starts failing (regression)
|
||||
- A previously failing test starts passing (stale baseline - remove the entry)
|
||||
|
||||
### File Format
|
||||
|
||||
Create a YAML file (e.g., `conformance-baseline.yml`):
|
||||
|
||||
```yaml
|
||||
server:
|
||||
- tools-call-with-progress
|
||||
- resources-subscribe
|
||||
client:
|
||||
- auth/client-credentials-jwt
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
npx @modelcontextprotocol/conformance server \
|
||||
--url http://localhost:3000/mcp \
|
||||
--expected-failures ./conformance-baseline.yml
|
||||
```
|
||||
|
||||
### Exit Code Behavior
|
||||
|
||||
| Scenario Result | In Baseline? | Exit Code | Meaning |
|
||||
| --------------- | ------------ | --------- | ----------------------------- |
|
||||
| Fails | Yes | 0 | Expected failure |
|
||||
| Fails | No | 1 | Unexpected regression |
|
||||
| Passes | Yes | 1 | Stale baseline - remove entry |
|
||||
| Passes | No | 0 | Normal pass |
|
||||
|
||||
---
|
||||
|
||||
## GitHub Action
|
||||
|
||||
The conformance repo provides a reusable GitHub Action that handles Node.js setup and conformance execution.
|
||||
|
||||
### Client Testing Example
|
||||
|
||||
```yaml
|
||||
name: Conformance Tests
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
conformance:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up your SDK
|
||||
run: |
|
||||
# Your SDK setup (pip install, npm install, etc.)
|
||||
pip install -e .
|
||||
|
||||
- uses: modelcontextprotocol/conformance@v0.1.10
|
||||
with:
|
||||
mode: client
|
||||
command: 'python tests/conformance/client.py'
|
||||
suite: auth
|
||||
expected-failures: ./conformance-baseline.yml
|
||||
```
|
||||
|
||||
### Server Testing Example
|
||||
|
||||
```yaml
|
||||
name: Conformance Tests
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
conformance:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up and start server
|
||||
run: |
|
||||
pip install -e .
|
||||
python -m myserver --port 3001 &
|
||||
# Wait for server to be ready
|
||||
timeout 15 bash -c 'until curl -s http://localhost:3001/mcp; do sleep 0.5; done'
|
||||
|
||||
- uses: modelcontextprotocol/conformance@v0.1.10
|
||||
with:
|
||||
mode: server
|
||||
url: http://localhost:3001/mcp
|
||||
suite: active
|
||||
expected-failures: ./conformance-baseline.yml
|
||||
```
|
||||
|
||||
### Action Inputs
|
||||
|
||||
| Input | Required | Description |
|
||||
| ------------------- | ----------- | ----------------------------------------------- |
|
||||
| `mode` | Yes | `server` or `client` |
|
||||
| `url` | Server mode | URL of the server to test |
|
||||
| `command` | Client mode | Command to run the client |
|
||||
| `expected-failures` | No | Path to YAML baseline file |
|
||||
| `suite` | No | Test suite to run |
|
||||
| `scenario` | No | Run a single scenario by name |
|
||||
| `timeout` | No | Timeout in ms for client tests (default: 30000) |
|
||||
| `verbose` | No | Show verbose output (default: false) |
|
||||
| `node-version` | No | Node.js version (default: 20) |
|
||||
|
||||
---
|
||||
|
||||
## Writing Conformance Clients/Servers
|
||||
|
||||
### Example Client Pattern
|
||||
|
||||
See [`src/conformance/everything-client.ts`](https://github.com/modelcontextprotocol/typescript-sdk/blob/main/src/conformance/everything-client.ts) in the TypeScript SDK for a reference implementation. The recommended pattern is a single client that routes behavior based on the scenario:
|
||||
|
||||
```python
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
|
||||
def main():
|
||||
server_url = sys.argv[-1] # URL passed as last argument
|
||||
scenario = os.environ.get("MCP_CONFORMANCE_SCENARIO", "")
|
||||
context = json.loads(os.environ.get("MCP_CONFORMANCE_CONTEXT", "{}"))
|
||||
|
||||
if scenario.startswith("auth/"):
|
||||
run_auth_scenario(server_url, scenario, context)
|
||||
else:
|
||||
run_default_scenario(server_url)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
```
|
||||
|
||||
### Example Server Pattern
|
||||
|
||||
See [`src/conformance/everything-server.ts`](https://github.com/modelcontextprotocol/typescript-sdk/blob/main/src/conformance/everything-server.ts) in the TypeScript SDK for a reference implementation that handles all server scenarios.
|
||||
|
||||
---
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- [Conformance README](./README.md)
|
||||
- [Design documentation](./src/runner/DESIGN.md)
|
||||
- [TypeScript SDK conformance examples](https://github.com/modelcontextprotocol/typescript-sdk/tree/main/src/conformance)
|
||||
Loading…
Reference in a new issue