9603b2fe93
## Motivation and Context PR #248 introduced server conformance testing using `@modelcontextprotocol/conformance`. This extends coverage to the client side so that `MCP::Client` behavior can also be validated against the MCP specification. ### Scope Client conformance is driven by the conformance runner, which spawns test servers for each scenario and invokes `conformance/client.rb` against them. The client script delegates the MCP initialize handshake to `MCP::Client#connect` and executes scenario-specific operations (for example listing tools or calling a tool). ### Usage Run all conformance tests (server + client): ```console bundle exec rake conformance ``` Run a single client scenario: ```console bundle exec rake conformance SCENARIO=initialize ``` List available scenarios: ```console bundle exec rake conformance_list ``` ## How Has This Been Tested? Ran `bundle exec rake conformance` locally. - Server conformance: 40 passed, 0 failed. - Client conformance: 3 passed, 54 failed (all 54 listed in `expected_failures.yml`). ## Breaking Change No. This change only adds test infrastructure under `conformance/` and does not modify SDK behavior.
105 lines
3.7 KiB
Markdown
105 lines
3.7 KiB
Markdown
# MCP Conformance Tests
|
|
|
|
Validates the Ruby SDK's conformance to the MCP specification using [`@modelcontextprotocol/conformance`](https://github.com/modelcontextprotocol/conformance).
|
|
|
|
## Prerequisites
|
|
|
|
- Node.js (for `npx`)
|
|
- `bundle install` completed
|
|
|
|
## Running the Tests
|
|
|
|
### Run all scenarios
|
|
|
|
```bash
|
|
bundle exec rake conformance
|
|
```
|
|
|
|
Runs both server and client conformance tests in sequence. Server conformance starts the
|
|
conformance server and tests it. Client conformance spawns test servers for each scenario
|
|
and invokes `conformance/client.rb` against them. Scenarios listed in `expected_failures.yml`
|
|
are allowed to fail without affecting the exit code.
|
|
|
|
### Environment variables
|
|
|
|
| Variable | Description | Default |
|
|
|----------------|---------------------------------------|---------|
|
|
| `PORT` | Server port (server conformance only) | `9292` |
|
|
| `SCENARIO` | Run a single scenario by name | (all) |
|
|
| `SPEC_VERSION` | Filter scenarios by spec version | (all) |
|
|
| `VERBOSE` | Show raw JSON output when set | (off) |
|
|
|
|
```bash
|
|
# Run a single server scenario
|
|
bundle exec rake conformance SCENARIO=ping
|
|
|
|
# Use a different port with verbose output
|
|
bundle exec rake conformance PORT=3000 VERBOSE=1
|
|
|
|
# Start the server on a specific port
|
|
bundle exec rake conformance_server PORT=3000
|
|
```
|
|
|
|
### Start the server and test separately
|
|
|
|
```bash
|
|
# Terminal 1: start the server
|
|
bundle exec rake conformance_server
|
|
|
|
# Terminal 2: run all scenarios
|
|
npx @modelcontextprotocol/conformance server --url http://localhost:9292/mcp
|
|
|
|
# Terminal 2: run a single scenario
|
|
npx @modelcontextprotocol/conformance server --url http://localhost:9292/mcp --scenario ping
|
|
```
|
|
|
|
Keeps the server alive between test runs, which avoids the startup overhead when iterating
|
|
on a single scenario. Stop the server with Ctrl+C when done.
|
|
|
|
### List available scenarios
|
|
|
|
```bash
|
|
bundle exec rake conformance_list
|
|
```
|
|
|
|
Prints all scenario names that can be passed to `SCENARIO`.
|
|
|
|
## SDK Tier Report
|
|
|
|
The [MCP SDK Tier system](https://modelcontextprotocol.io/community/sdk-tiers) requires SDK
|
|
maintainers to self-assess and report results to the SDK Working Group via
|
|
[modelcontextprotocol/modelcontextprotocol issues](https://github.com/modelcontextprotocol/modelcontextprotocol/issues).
|
|
|
|
To generate a full tier assessment report, use the `/mcp-sdk-tier-audit` slash command from
|
|
the [modelcontextprotocol/conformance](https://github.com/modelcontextprotocol/conformance)
|
|
repository with the conformance server running:
|
|
|
|
```bash
|
|
# Terminal 1 (this repository): start the conformance server
|
|
bundle exec rake conformance_server
|
|
|
|
# Terminal 2 (conformance repository): run the tier audit skill as a slash command in Claude Code
|
|
/mcp-sdk-tier-audit /path/to/modelcontextprotocol/ruby-sdk http://localhost:9292/mcp
|
|
```
|
|
|
|
The skill evaluates conformance pass rate, issue label taxonomy, triage metrics, documentation
|
|
coverage, and policy compliance, then produces a markdown report suitable for tier advancement
|
|
submissions.
|
|
|
|
## File Structure
|
|
|
|
```
|
|
conformance/
|
|
server.rb # Conformance server (Rack + Puma, default port 9292)
|
|
server_runner.rb # Starts the server, runs npx conformance server, exits with result code
|
|
client.rb # Conformance client (invoked by npx conformance client)
|
|
client_runner.rb # Runs npx conformance client, exits with result code
|
|
expected_failures.yml # Baseline of known-failing scenarios (server and client)
|
|
README.md # This file
|
|
```
|
|
|
|
## Known Limitations
|
|
|
|
Known-failing scenarios are registered in `conformance/expected_failures.yml`. They are allowed to
|
|
fail without affecting the exit code and are tracked to catch regressions.
|
|
These are shown in the output of `bundle exec rake conformance`.
|