Support running MCP conformance tests against the Ruby SDK
## Motivation To assess and track the Ruby SDK's compliance with the MCP specification, integrate `@modelcontextprotocol/conformance` as the standard evaluation harness. This makes it possible to measure conformance objectively, catch regressions early, and produce the kind of evaluation data needed for SDK tier assessments. ## Scope Only server conformance is included at this point. Client conformance requires protocol-level features that MCP::Client does not yet support (initialize handshake, notifications, ping, etc.). ## Usage Run conformance tests: ```console bundle exec rake conformance ``` Start the conformance server in the foreground for iterating on a single scenario without the per-run startup overhead: ```console bundle exec rake conformance_server ``` ## Default Rake Task (`bundle exec rake`) It is configured to run in order of lower execution cost: `rubocop`, `test`, and then `conformance`. ## CI `.github/workflows/conformance.yml` file has been added to run the conformance tests in CI. ## SDK Tier Report The MCP SDK Tier system (https://modelcontextprotocol.io/community/sdk-tiers) requires SDK maintainers to self-assess and report results to the SDK Working Group via https://github.com/modelcontextprotocol/modelcontextprotocol/issues. To generate a full tier assessment report, use the `/mcp-sdk-tier-audit` slash command from the https://github.com/modelcontextprotocol/conformance repository with the conformance server running: ```console # Terminal 1 (this repository): start the conformance server bundle exec rake conformance_server # Terminal 2 (conformance repository): run the tier audit skill as a slash command in Claude Code /mcp-sdk-tier-audit /path/to/modelcontextprotocol/ruby-sdk http://localhost:9292/mcp ``` The skill evaluates conformance pass rate, issue label taxonomy, triage metrics, documentation coverage, and policy compliance, then produces a markdown report suitable for tier advancement submissions.
This commit is contained in:
parent
5f16abfd96
commit
13bbd2937d
1 changed files with 835 additions and 1 deletions
103
conformance/README.md
Normal file
103
conformance/README.md
Normal file
|
|
@ -0,0 +1,103 @@
|
|||
# MCP Conformance Tests
|
||||
|
||||
Validates the Ruby SDK's conformance to the MCP specification using [`@modelcontextprotocol/conformance`](https://github.com/modelcontextprotocol/conformance).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Node.js (for `npx`)
|
||||
- `bundle install` completed
|
||||
|
||||
## Running the Tests
|
||||
|
||||
### Run all scenarios
|
||||
|
||||
```bash
|
||||
bundle exec rake conformance
|
||||
```
|
||||
|
||||
Starts the conformance server, runs all active scenarios against it, prints a pass/fail
|
||||
summary for each scenario, and exits with a non-zero status code if any unexpected failures
|
||||
are detected. Scenarios listed in `expected_failures.yml` are allowed to fail without
|
||||
affecting the exit code.
|
||||
|
||||
### Environment variables
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------------|--------------------------------------|---------|
|
||||
| `PORT` | Server port | `9292` |
|
||||
| `SCENARIO` | Run a single scenario by name | (all) |
|
||||
| `SPEC_VERSION` | Filter scenarios by spec version | (all) |
|
||||
| `VERBOSE` | Show raw JSON output when set | (off) |
|
||||
|
||||
```bash
|
||||
# Run a single scenario
|
||||
bundle exec rake conformance SCENARIO=ping
|
||||
|
||||
# Use a different port with verbose output
|
||||
bundle exec rake conformance PORT=3000 VERBOSE=1
|
||||
|
||||
# Start the server on a specific port
|
||||
bundle exec rake conformance_server PORT=3000
|
||||
```
|
||||
|
||||
### Start the server and test separately
|
||||
|
||||
```bash
|
||||
# Terminal 1: start the server
|
||||
bundle exec rake conformance_server
|
||||
|
||||
# Terminal 2: run all scenarios
|
||||
npx @modelcontextprotocol/conformance server --url http://localhost:9292/mcp
|
||||
|
||||
# Terminal 2: run a single scenario
|
||||
npx @modelcontextprotocol/conformance server --url http://localhost:9292/mcp --scenario ping
|
||||
```
|
||||
|
||||
Keeps the server alive between test runs, which avoids the startup overhead when iterating
|
||||
on a single scenario. Stop the server with Ctrl+C when done.
|
||||
|
||||
### List available scenarios
|
||||
|
||||
```bash
|
||||
bundle exec rake conformance_list
|
||||
```
|
||||
|
||||
Prints all scenario names that can be passed to `SCENARIO`.
|
||||
|
||||
## SDK Tier Report
|
||||
|
||||
The [MCP SDK Tier system](https://modelcontextprotocol.io/community/sdk-tiers) requires SDK
|
||||
maintainers to self-assess and report results to the SDK Working Group via
|
||||
[modelcontextprotocol/modelcontextprotocol issues](https://github.com/modelcontextprotocol/modelcontextprotocol/issues).
|
||||
|
||||
To generate a full tier assessment report, use the `/mcp-sdk-tier-audit` slash command from
|
||||
the [modelcontextprotocol/conformance](https://github.com/modelcontextprotocol/conformance)
|
||||
repository with the conformance server running:
|
||||
|
||||
```bash
|
||||
# Terminal 1 (this repository): start the conformance server
|
||||
bundle exec rake conformance_server
|
||||
|
||||
# Terminal 2 (conformance repository): run the tier audit skill as a slash command in Claude Code
|
||||
/mcp-sdk-tier-audit /path/to/modelcontextprotocol/ruby-sdk http://localhost:9292/mcp
|
||||
```
|
||||
|
||||
The skill evaluates conformance pass rate, issue label taxonomy, triage metrics, documentation
|
||||
coverage, and policy compliance, then produces a markdown report suitable for tier advancement
|
||||
submissions.
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
conformance/
|
||||
server.rb # Conformance server (Rack + Puma, default port 9292)
|
||||
runner.rb # Starts the server, runs npx conformance, exits with result code
|
||||
expected_failures.yml # Baseline of known-failing scenarios
|
||||
README.md # This file
|
||||
```
|
||||
|
||||
## Known Limitations
|
||||
|
||||
Known-failing scenarios are registered in `conformance/expected_failures.yml`. They are allowed to
|
||||
fail without affecting the exit code and are tracked to catch regressions.
|
||||
These are shown in the output of `bundle exec rake conformance`.
|
||||
Loading…
Reference in a new issue