5932521552
## Motivation and Context The npm `latest` dist-tag of `@modelcontextprotocol/conformance` points at 0.1.16, the old 0.1 line, while the 2026-07-28 requirement revisions and the frozen-requirements framework live in the 0.2.0 alpha line (`alpha` currently resolves 0.2.0-alpha.11). The runners invoked the bare package name, so `rake conformance` and the conformance CI job resolve whatever `latest` points at: a fresh machine silently downgrades to the old suite while a warm npx cache may keep serving a newer one, and the suite actually in use depends on the environment. Pinning the `alpha` dist-tag puts every run on the line the frozen conformance requirements are anchored to (0.2.0-alpha.10, published 2026-07-27). Running the real alpha suite surfaced the actual client-leg state, so the expected-failures baseline is updated to match: `initialize` now passes (the harness accepts 2026-07-28, fulfilling the TODO on the old entry), and the scenarios pending client-side work are recorded with their reasons: the 2026-07-28 stateless lifecycle and the SEP-2322 MRTR driver, SEP-2243 custom-header mirroring, client-side JSON Schema preservation, the DPoP and WIF authorization extensions, and one `auth/authorization-server-migration` check added to the suite after the alpha.10 anchor. The manual `npx` examples in `conformance/README.md` carry the same pin so local reproduction runs the same suite as the rake task. ## How Has This Been Tested? `npx --yes @modelcontextprotocol/conformance@alpha --version` resolves 0.2.0-alpha.11, and `bundle exec rake conformance` passes the baseline check on both legs: the server leg reports 73 passed with 0 failed and no expected failures, and the client leg fails exactly the scenarios the baseline records. ## Breaking Changes None. The change is confined to the conformance tooling; the shipped gem is untouched.
105 lines
3.8 KiB
Markdown
105 lines
3.8 KiB
Markdown
# MCP Conformance Tests
|
|
|
|
Validates the Ruby SDK's conformance to the MCP specification using [`@modelcontextprotocol/conformance`](https://github.com/modelcontextprotocol/conformance).
|
|
|
|
## Prerequisites
|
|
|
|
- Node.js (for `npx`)
|
|
- `bundle install` completed
|
|
|
|
## Running the Tests
|
|
|
|
### Run all scenarios
|
|
|
|
```bash
|
|
bundle exec rake conformance
|
|
```
|
|
|
|
Runs both server and client conformance tests in sequence. Server conformance starts the
|
|
conformance server and tests it. Client conformance spawns test servers for each scenario
|
|
and invokes `conformance/client.rb` against them. Scenarios listed in `expected_failures.yml`
|
|
are allowed to fail without affecting the exit code.
|
|
|
|
### Environment variables
|
|
|
|
| Variable | Description | Default |
|
|
|----------------|---------------------------------------|---------|
|
|
| `PORT` | Server port (server conformance only) | `9292` |
|
|
| `SCENARIO` | Run a single scenario by name | (all) |
|
|
| `SPEC_VERSION` | Filter scenarios by spec version | (all) |
|
|
| `VERBOSE` | Show raw JSON output when set | (off) |
|
|
|
|
```bash
|
|
# Run a single server scenario
|
|
bundle exec rake conformance SCENARIO=ping
|
|
|
|
# Use a different port with verbose output
|
|
bundle exec rake conformance PORT=3000 VERBOSE=1
|
|
|
|
# Start the server on a specific port
|
|
bundle exec rake conformance_server PORT=3000
|
|
```
|
|
|
|
### Start the server and test separately
|
|
|
|
```bash
|
|
# Terminal 1: start the server
|
|
bundle exec rake conformance_server
|
|
|
|
# Terminal 2: run all scenarios
|
|
npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp
|
|
|
|
# Terminal 2: run a single scenario
|
|
npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp --scenario ping
|
|
```
|
|
|
|
Keeps the server alive between test runs, which avoids the startup overhead when iterating
|
|
on a single scenario. Stop the server with Ctrl+C when done.
|
|
|
|
### List available scenarios
|
|
|
|
```bash
|
|
bundle exec rake conformance_list
|
|
```
|
|
|
|
Prints all scenario names that can be passed to `SCENARIO`.
|
|
|
|
## SDK Tier Report
|
|
|
|
The [MCP SDK Tier system](https://modelcontextprotocol.io/community/sdk-tiers) requires SDK
|
|
maintainers to self-assess and report results to the SDK Working Group via
|
|
[modelcontextprotocol/modelcontextprotocol issues](https://github.com/modelcontextprotocol/modelcontextprotocol/issues).
|
|
|
|
To generate a full tier assessment report, use the `/mcp-sdk-tier-audit` slash command from
|
|
the [modelcontextprotocol/conformance](https://github.com/modelcontextprotocol/conformance)
|
|
repository with the conformance server running:
|
|
|
|
```bash
|
|
# Terminal 1 (this repository): start the conformance server
|
|
bundle exec rake conformance_server
|
|
|
|
# Terminal 2 (conformance repository): run the tier audit skill as a slash command in Claude Code
|
|
/mcp-sdk-tier-audit /path/to/modelcontextprotocol/ruby-sdk http://localhost:9292/mcp
|
|
```
|
|
|
|
The skill evaluates conformance pass rate, issue label taxonomy, triage metrics, documentation
|
|
coverage, and policy compliance, then produces a markdown report suitable for tier advancement
|
|
submissions.
|
|
|
|
## File Structure
|
|
|
|
```
|
|
conformance/
|
|
server.rb # Conformance server (Rack + Puma, default port 9292)
|
|
server_runner.rb # Starts the server, runs npx conformance server, exits with result code
|
|
client.rb # Conformance client (invoked by npx conformance client)
|
|
client_runner.rb # Runs npx conformance client, exits with result code
|
|
expected_failures.yml # Baseline of known-failing scenarios (server and client)
|
|
README.md # This file
|
|
```
|
|
|
|
## Known Limitations
|
|
|
|
Known-failing scenarios are registered in `conformance/expected_failures.yml`. They are allowed to
|
|
fail without affecting the exit code and are tracked to catch regressions.
|
|
These are shown in the output of `bundle exec rake conformance`.
|