modelcontextprotocol/ruby-sdk
Watch
1
0
Fork
You've already forked ruby-sdk
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
ruby-sdk/conformance
History
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Koichi ITO c8116fbcf0 Drive the http-standard-headers scenario in the conformance client 
## Motivation and Context

The 2026-07-28 requirement set scores `http-standard-headers` (SEP-2243) on the client leg,
but `conformance/client.rb` had no case for it: the scenario hit the unknown-scenario abort,
the client exited 1, and every `Mcp-Method` / `Mcp-Name` check reported SKIPPED
("Client did not send a ... request"), failing the scenario in tier-check.
The 2026-08-14 tier audit flagged this as the only scored conformance failure (client leg 49/50).

The transport already sends the standard headers on the modern path; what was missing is
a driver that exercises each request kind the harness observes. The new case issues `tools/list`,
`tools/call` (the `test_headers` fixture tool), `resources/list`, `resources/read`
(the first listed resource URI), `prompts/list`, and `prompts/get` (the first listed prompt),
letting the harness see the `Mcp-Method` header on every POST and the `Mcp-Name` header on
the three name-carrying requests. The `initialize` / `notifications/initialized` rows stay SKIPPED
by design: the 2026-07-28 wire is stateless, so those requests are never sent.

## How Has This Been Tested?

`conformance client --scenario http-standard-headers --spec-version 2026-07-28` passes 9/9 checks,
and the full `--requirements 2026-07-28` client leg exits 0 with every scored scenario passing
(435 checks; the only failures are the baselined not-scored DPoP/WIF extensions). RuboCop is clean.

## Breaking Changes

None. The change only adds a scenario driver to the conformance client fixture; library code is untouched.
2026-08-14 04:22:23 +09:00
..
client.rb Drive the http-standard-headers scenario in the conformance client 2026-08-14 04:22:23 +09:00
client_runner.rb Pin the conformance CLI to the alpha dist-tag 2026-08-08 18:47:24 +09:00
expected_failures.yml Drive multi round-trip input_required results on the client per SEP-2322 2026-08-14 02:04:21 +09:00
README.md Pin the conformance CLI to the alpha dist-tag 2026-08-08 18:47:24 +09:00
server.rb Merge pull request #495 from koic/subscriptions_listen 2026-08-13 23:05:35 +09:00
server_runner.rb Pin the conformance CLI to the alpha dist-tag 2026-08-08 18:47:24 +09:00

README.md

MCP Conformance Tests

Validates the Ruby SDK's conformance to the MCP specification using @modelcontextprotocol/conformance.

Prerequisites

  • Node.js (for npx)
  • bundle install completed

Running the Tests

Run all scenarios

bundle exec rake conformance

Runs both server and client conformance tests in sequence. Server conformance starts the conformance server and tests it. Client conformance spawns test servers for each scenario and invokes conformance/client.rb against them. Scenarios listed in expected_failures.yml are allowed to fail without affecting the exit code.

Environment variables

Variable Description Default
PORT Server port (server conformance only) 9292
SCENARIO Run a single scenario by name (all)
SPEC_VERSION Filter scenarios by spec version (all)
VERBOSE Show raw JSON output when set (off) 
# Run a single server scenario
bundle exec rake conformance SCENARIO=ping

# Use a different port with verbose output
bundle exec rake conformance PORT=3000 VERBOSE=1

# Start the server on a specific port
bundle exec rake conformance_server PORT=3000

Start the server and test separately

# Terminal 1: start the server
bundle exec rake conformance_server

# Terminal 2: run all scenarios
npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp

# Terminal 2: run a single scenario
npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp --scenario ping

Keeps the server alive between test runs, which avoids the startup overhead when iterating on a single scenario. Stop the server with Ctrl+C when done.

List available scenarios

bundle exec rake conformance_list

Prints all scenario names that can be passed to SCENARIO.

SDK Tier Report

The MCP SDK Tier system requires SDK maintainers to self-assess and report results to the SDK Working Group via modelcontextprotocol/modelcontextprotocol issues.

To generate a full tier assessment report, use the /mcp-sdk-tier-audit slash command from the modelcontextprotocol/conformance repository with the conformance server running:

# Terminal 1 (this repository): start the conformance server
bundle exec rake conformance_server

# Terminal 2 (conformance repository): run the tier audit skill as a slash command in Claude Code
/mcp-sdk-tier-audit /path/to/modelcontextprotocol/ruby-sdk http://localhost:9292/mcp

The skill evaluates conformance pass rate, issue label taxonomy, triage metrics, documentation coverage, and policy compliance, then produces a markdown report suitable for tier advancement submissions.

File Structure

conformance/
  server.rb              # Conformance server (Rack + Puma, default port 9292)
  server_runner.rb       # Starts the server, runs npx conformance server, exits with result code
  client.rb              # Conformance client (invoked by npx conformance client)
  client_runner.rb       # Runs npx conformance client, exits with result code
  expected_failures.yml  # Baseline of known-failing scenarios (server and client)
  README.md              # This file

Known Limitations

Known-failing scenarios are registered in conformance/expected_failures.yml. They are allowed to fail without affecting the exit code and are tracked to catch regressions. These are shown in the output of bundle exec rake conformance.