# MCP Conformance Tests

Validates the Ruby SDK's conformance to the MCP specification using [`@modelcontextprotocol/conformance`](https://github.com/modelcontextprotocol/conformance).

## Prerequisites

- Node.js (for `npx`)
- `bundle install` completed

## Running the Tests

### Run all scenarios

```bash
bundle exec rake conformance
```

Runs both server and client conformance tests in sequence. Server conformance starts the
conformance server and tests it. Client conformance spawns test servers for each scenario
and invokes `conformance/client.rb` against them. Scenarios listed in `expected_failures.yml`
are allowed to fail without affecting the exit code.

### Environment variables

| Variable       | Description                           | Default |
|----------------|---------------------------------------|---------|
| `PORT`         | Server port (server conformance only) | `9292`  |
| `SCENARIO`     | Run a single scenario by name         | (all)   |
| `SPEC_VERSION` | Filter scenarios by spec version      | (all)   |
| `VERBOSE`      | Show raw JSON output when set         | (off)   |

```bash
# Run a single server scenario
bundle exec rake conformance SCENARIO=ping

# Use a different port with verbose output
bundle exec rake conformance PORT=3000 VERBOSE=1

# Start the server on a specific port
bundle exec rake conformance_server PORT=3000
```

### Start the server and test separately

```bash
# Terminal 1: start the server
bundle exec rake conformance_server

# Terminal 2: run all scenarios
npx @modelcontextprotocol/conformance server --url http://localhost:9292/mcp

# Terminal 2: run a single scenario
npx @modelcontextprotocol/conformance server --url http://localhost:9292/mcp --scenario ping
```

Keeps the server alive between test runs, which avoids the startup overhead when iterating
on a single scenario. Stop the server with Ctrl+C when done.

### List available scenarios

```bash
bundle exec rake conformance_list
```

Prints all scenario names that can be passed to `SCENARIO`.

## SDK Tier Report

The [MCP SDK Tier system](https://modelcontextprotocol.io/community/sdk-tiers) requires SDK
maintainers to self-assess and report results to the SDK Working Group via
[modelcontextprotocol/modelcontextprotocol issues](https://github.com/modelcontextprotocol/modelcontextprotocol/issues).

To generate a full tier assessment report, use the `/mcp-sdk-tier-audit` slash command from
the [modelcontextprotocol/conformance](https://github.com/modelcontextprotocol/conformance)
repository with the conformance server running:

```bash
# Terminal 1 (this repository): start the conformance server
bundle exec rake conformance_server

# Terminal 2 (conformance repository): run the tier audit skill as a slash command in Claude Code
/mcp-sdk-tier-audit /path/to/modelcontextprotocol/ruby-sdk http://localhost:9292/mcp
```

The skill evaluates conformance pass rate, issue label taxonomy, triage metrics, documentation
coverage, and policy compliance, then produces a markdown report suitable for tier advancement
submissions.

## File Structure

```
conformance/
  server.rb              # Conformance server (Rack + Puma, default port 9292)
  server_runner.rb       # Starts the server, runs npx conformance server, exits with result code
  client.rb              # Conformance client (invoked by npx conformance client)
  client_runner.rb       # Runs npx conformance client, exits with result code
  expected_failures.yml  # Baseline of known-failing scenarios (server and client)
  README.md              # This file
```

## Known Limitations

Known-failing scenarios are registered in `conformance/expected_failures.yml`. They are allowed to
fail without affecting the exit code and are tracked to catch regressions.
These are shown in the output of `bundle exec rake conformance`.