920707cd7e
## Motivation and Context The three conformance tasks were named `conformance`, `conformance_list`, and `conformance_server`, spelling out with underscores the grouping that Rake expresses with namespaces. They are now `conformance:test`, `conformance:list`, and `conformance:server`. The bare `conformance` task is kept as an alias for `conformance:test`. ## How Has This Been Tested? By running `rake -T` to confirm the three tasks are grouped under `conformance:` with the alias listed beside them, `rake --dry-run default` to confirm that the default task resolves `conformance:test`, and `rake conformance:list` to exercise a renamed task end to end. ## Breaking Changes None. These are development-only Rake tasks.
105 lines
3.8 KiB
Markdown
105 lines
3.8 KiB
Markdown
# MCP Conformance Tests
|
|
|
|
Validates the Ruby SDK's conformance to the MCP specification using [`@modelcontextprotocol/conformance`](https://github.com/modelcontextprotocol/conformance).
|
|
|
|
## Prerequisites
|
|
|
|
- Node.js (for `npx`)
|
|
- `bundle install` completed
|
|
|
|
## Running the Tests
|
|
|
|
### Run all scenarios
|
|
|
|
```bash
|
|
bundle exec rake conformance:test
|
|
```
|
|
|
|
Runs both server and client conformance tests in sequence. Server conformance starts the
|
|
conformance server and tests it. Client conformance spawns test servers for each scenario
|
|
and invokes `conformance/client.rb` against them. Scenarios listed in `expected_failures.yml`
|
|
are allowed to fail without affecting the exit code.
|
|
|
|
### Environment variables
|
|
|
|
| Variable | Description | Default |
|
|
|----------------|---------------------------------------|---------|
|
|
| `PORT` | Server port (server conformance only) | `9292` |
|
|
| `SCENARIO` | Run a single scenario by name | (all) |
|
|
| `SPEC_VERSION` | Filter scenarios by spec version | (all) |
|
|
| `VERBOSE` | Show raw JSON output when set | (off) |
|
|
|
|
```bash
|
|
# Run a single server scenario
|
|
bundle exec rake conformance:test SCENARIO=ping
|
|
|
|
# Use a different port with verbose output
|
|
bundle exec rake conformance:test PORT=3000 VERBOSE=1
|
|
|
|
# Start the server on a specific port
|
|
bundle exec rake conformance:server PORT=3000
|
|
```
|
|
|
|
### Start the server and test separately
|
|
|
|
```bash
|
|
# Terminal 1: start the server
|
|
bundle exec rake conformance:server
|
|
|
|
# Terminal 2: run all scenarios
|
|
npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp
|
|
|
|
# Terminal 2: run a single scenario
|
|
npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp --scenario ping
|
|
```
|
|
|
|
Keeps the server alive between test runs, which avoids the startup overhead when iterating
|
|
on a single scenario. Stop the server with Ctrl+C when done.
|
|
|
|
### List available scenarios
|
|
|
|
```bash
|
|
bundle exec rake conformance:list
|
|
```
|
|
|
|
Prints all scenario names that can be passed to `SCENARIO`.
|
|
|
|
## SDK Tier Report
|
|
|
|
The [MCP SDK Tier system](https://modelcontextprotocol.io/community/sdk-tiers) requires SDK
|
|
maintainers to self-assess and report results to the SDK Working Group via
|
|
[modelcontextprotocol/modelcontextprotocol issues](https://github.com/modelcontextprotocol/modelcontextprotocol/issues).
|
|
|
|
To generate a full tier assessment report, use the `/mcp-sdk-tier-audit` slash command from
|
|
the [modelcontextprotocol/conformance](https://github.com/modelcontextprotocol/conformance)
|
|
repository with the conformance server running:
|
|
|
|
```bash
|
|
# Terminal 1 (this repository): start the conformance server
|
|
bundle exec rake conformance:server
|
|
|
|
# Terminal 2 (conformance repository): run the tier audit skill as a slash command in Claude Code
|
|
/mcp-sdk-tier-audit /path/to/modelcontextprotocol/ruby-sdk http://localhost:9292/mcp
|
|
```
|
|
|
|
The skill evaluates conformance pass rate, issue label taxonomy, triage metrics, documentation
|
|
coverage, and policy compliance, then produces a markdown report suitable for tier advancement
|
|
submissions.
|
|
|
|
## File Structure
|
|
|
|
```
|
|
conformance/
|
|
server.rb # Conformance server (Rack + Puma, default port 9292)
|
|
server_runner.rb # Starts the server, runs npx conformance server, exits with result code
|
|
client.rb # Conformance client (invoked by npx conformance client)
|
|
client_runner.rb # Runs npx conformance client, exits with result code
|
|
expected_failures.yml # Baseline of known-failing scenarios (server and client)
|
|
README.md # This file
|
|
```
|
|
|
|
## Known Limitations
|
|
|
|
Known-failing scenarios are registered in `conformance/expected_failures.yml`. They are allowed to
|
|
fail without affecting the exit code and are tracked to catch regressions.
|
|
These are shown in the output of `bundle exec rake conformance:test`.
|