Watch

History

Repository files (latest commit first)
Filename	Latest commit message	Latest commit date
Koichi ITO 920707cd7e Group the conformance Rake tasks under a `conformance` namespace ## Motivation and Context The three conformance tasks were named `conformance`, `conformance_list`, and `conformance_server`, spelling out with underscores the grouping that Rake expresses with namespaces. They are now `conformance:test`, `conformance:list`, and `conformance:server`. The bare `conformance` task is kept as an alias for `conformance:test`. ## How Has This Been Tested? By running `rake -T` to confirm the three tasks are grouped under `conformance:` with the alias listed beside them, `rake --dry-run default` to confirm that the default task resolves `conformance:test`, and `rake conformance:list` to exercise a renamed task end to end. ## Breaking Changes None. These are development-only Rake tasks.		2026-08-30 02:24:52 +09:00
..
client.rb	Drive the `http-standard-headers` scenario in the conformance client	2026-08-14 04:22:23 +09:00
client_runner.rb	Pin the conformance CLI to the `alpha` dist-tag	2026-08-08 18:47:24 +09:00
expected_failures.yml	Drive multi round-trip `input_required` results on the client per SEP-2322	2026-08-14 02:04:21 +09:00
README.md	Group the conformance Rake tasks under a `conformance` namespace	2026-08-30 02:24:52 +09:00
server.rb	Merge pull request #495 from koic/subscriptions_listen	2026-08-13 23:05:35 +09:00
server_runner.rb	Pin the conformance CLI to the `alpha` dist-tag	2026-08-08 18:47:24 +09:00

README.md

MCP Conformance Tests

Validates the Ruby SDK's conformance to the MCP specification using @modelcontextprotocol/conformance.

Prerequisites

Node.js (for npx)
bundle install completed

Running the Tests

Run all scenarios

bundle exec rake conformance:test

Runs both server and client conformance tests in sequence. Server conformance starts the conformance server and tests it. Client conformance spawns test servers for each scenario and invokes conformance/client.rb against them. Scenarios listed in expected_failures.yml are allowed to fail without affecting the exit code.

Environment variables

Variable	Description	Default
`PORT`	Server port (server conformance only)	`9292`
`SCENARIO`	Run a single scenario by name	(all)
`SPEC_VERSION`	Filter scenarios by spec version	(all)
`VERBOSE`	Show raw JSON output when set	(off)

# Run a single server scenario
bundle exec rake conformance:test SCENARIO=ping

# Use a different port with verbose output
bundle exec rake conformance:test PORT=3000 VERBOSE=1

# Start the server on a specific port
bundle exec rake conformance:server PORT=3000

Start the server and test separately

# Terminal 1: start the server
bundle exec rake conformance:server

# Terminal 2: run all scenarios
npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp

# Terminal 2: run a single scenario
npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp --scenario ping

Keeps the server alive between test runs, which avoids the startup overhead when iterating on a single scenario. Stop the server with Ctrl+C when done.

List available scenarios

bundle exec rake conformance:list

Prints all scenario names that can be passed to SCENARIO.

SDK Tier Report

The MCP SDK Tier system requires SDK maintainers to self-assess and report results to the SDK Working Group via modelcontextprotocol/modelcontextprotocol issues.

To generate a full tier assessment report, use the /mcp-sdk-tier-audit slash command from the modelcontextprotocol/conformance repository with the conformance server running:

# Terminal 1 (this repository): start the conformance server
bundle exec rake conformance:server

# Terminal 2 (conformance repository): run the tier audit skill as a slash command in Claude Code
/mcp-sdk-tier-audit /path/to/modelcontextprotocol/ruby-sdk http://localhost:9292/mcp

The skill evaluates conformance pass rate, issue label taxonomy, triage metrics, documentation coverage, and policy compliance, then produces a markdown report suitable for tier advancement submissions.

File Structure

conformance/
  server.rb              # Conformance server (Rack + Puma, default port 9292)
  server_runner.rb       # Starts the server, runs npx conformance server, exits with result code
  client.rb              # Conformance client (invoked by npx conformance client)
  client_runner.rb       # Runs npx conformance client, exits with result code
  expected_failures.yml  # Baseline of known-failing scenarios (server and client)
  README.md              # This file

Known Limitations

Known-failing scenarios are registered in conformance/expected_failures.yml. They are allowed to fail without affecting the exit code and are tracked to catch regressions. These are shown in the output of bundle exec rake conformance:test.