13bbd2937d
## Motivation To assess and track the Ruby SDK's compliance with the MCP specification, integrate `@modelcontextprotocol/conformance` as the standard evaluation harness. This makes it possible to measure conformance objectively, catch regressions early, and produce the kind of evaluation data needed for SDK tier assessments. ## Scope Only server conformance is included at this point. Client conformance requires protocol-level features that MCP::Client does not yet support (initialize handshake, notifications, ping, etc.). ## Usage Run conformance tests: ```console bundle exec rake conformance ``` Start the conformance server in the foreground for iterating on a single scenario without the per-run startup overhead: ```console bundle exec rake conformance_server ``` ## Default Rake Task (`bundle exec rake`) It is configured to run in order of lower execution cost: `rubocop`, `test`, and then `conformance`. ## CI `.github/workflows/conformance.yml` file has been added to run the conformance tests in CI. ## SDK Tier Report The MCP SDK Tier system (https://modelcontextprotocol.io/community/sdk-tiers) requires SDK maintainers to self-assess and report results to the SDK Working Group via https://github.com/modelcontextprotocol/modelcontextprotocol/issues. To generate a full tier assessment report, use the `/mcp-sdk-tier-audit` slash command from the https://github.com/modelcontextprotocol/conformance repository with the conformance server running: ```console # Terminal 1 (this repository): start the conformance server bundle exec rake conformance_server # Terminal 2 (conformance repository): run the tier audit skill as a slash command in Claude Code /mcp-sdk-tier-audit /path/to/modelcontextprotocol/ruby-sdk http://localhost:9292/mcp ``` The skill evaluates conformance pass rate, issue label taxonomy, triage metrics, documentation coverage, and policy compliance, then produces a markdown report suitable for tier advancement submissions.
103 lines
3.5 KiB
Markdown
103 lines
3.5 KiB
Markdown
# MCP Conformance Tests
|
|
|
|
Validates the Ruby SDK's conformance to the MCP specification using [`@modelcontextprotocol/conformance`](https://github.com/modelcontextprotocol/conformance).
|
|
|
|
## Prerequisites
|
|
|
|
- Node.js (for `npx`)
|
|
- `bundle install` completed
|
|
|
|
## Running the Tests
|
|
|
|
### Run all scenarios
|
|
|
|
```bash
|
|
bundle exec rake conformance
|
|
```
|
|
|
|
Starts the conformance server, runs all active scenarios against it, prints a pass/fail
|
|
summary for each scenario, and exits with a non-zero status code if any unexpected failures
|
|
are detected. Scenarios listed in `expected_failures.yml` are allowed to fail without
|
|
affecting the exit code.
|
|
|
|
### Environment variables
|
|
|
|
| Variable | Description | Default |
|
|
|----------------|--------------------------------------|---------|
|
|
| `PORT` | Server port | `9292` |
|
|
| `SCENARIO` | Run a single scenario by name | (all) |
|
|
| `SPEC_VERSION` | Filter scenarios by spec version | (all) |
|
|
| `VERBOSE` | Show raw JSON output when set | (off) |
|
|
|
|
```bash
|
|
# Run a single scenario
|
|
bundle exec rake conformance SCENARIO=ping
|
|
|
|
# Use a different port with verbose output
|
|
bundle exec rake conformance PORT=3000 VERBOSE=1
|
|
|
|
# Start the server on a specific port
|
|
bundle exec rake conformance_server PORT=3000
|
|
```
|
|
|
|
### Start the server and test separately
|
|
|
|
```bash
|
|
# Terminal 1: start the server
|
|
bundle exec rake conformance_server
|
|
|
|
# Terminal 2: run all scenarios
|
|
npx @modelcontextprotocol/conformance server --url http://localhost:9292/mcp
|
|
|
|
# Terminal 2: run a single scenario
|
|
npx @modelcontextprotocol/conformance server --url http://localhost:9292/mcp --scenario ping
|
|
```
|
|
|
|
Keeps the server alive between test runs, which avoids the startup overhead when iterating
|
|
on a single scenario. Stop the server with Ctrl+C when done.
|
|
|
|
### List available scenarios
|
|
|
|
```bash
|
|
bundle exec rake conformance_list
|
|
```
|
|
|
|
Prints all scenario names that can be passed to `SCENARIO`.
|
|
|
|
## SDK Tier Report
|
|
|
|
The [MCP SDK Tier system](https://modelcontextprotocol.io/community/sdk-tiers) requires SDK
|
|
maintainers to self-assess and report results to the SDK Working Group via
|
|
[modelcontextprotocol/modelcontextprotocol issues](https://github.com/modelcontextprotocol/modelcontextprotocol/issues).
|
|
|
|
To generate a full tier assessment report, use the `/mcp-sdk-tier-audit` slash command from
|
|
the [modelcontextprotocol/conformance](https://github.com/modelcontextprotocol/conformance)
|
|
repository with the conformance server running:
|
|
|
|
```bash
|
|
# Terminal 1 (this repository): start the conformance server
|
|
bundle exec rake conformance_server
|
|
|
|
# Terminal 2 (conformance repository): run the tier audit skill as a slash command in Claude Code
|
|
/mcp-sdk-tier-audit /path/to/modelcontextprotocol/ruby-sdk http://localhost:9292/mcp
|
|
```
|
|
|
|
The skill evaluates conformance pass rate, issue label taxonomy, triage metrics, documentation
|
|
coverage, and policy compliance, then produces a markdown report suitable for tier advancement
|
|
submissions.
|
|
|
|
## File Structure
|
|
|
|
```
|
|
conformance/
|
|
server.rb # Conformance server (Rack + Puma, default port 9292)
|
|
runner.rb # Starts the server, runs npx conformance, exits with result code
|
|
expected_failures.yml # Baseline of known-failing scenarios
|
|
README.md # This file
|
|
```
|
|
|
|
## Known Limitations
|
|
|
|
Known-failing scenarios are registered in `conformance/expected_failures.yml`. They are allowed to
|
|
fail without affecting the exit code and are tracked to catch regressions.
|
|
These are shown in the output of `bundle exec rake conformance`.
|