Commit graph ruby-sdk/conformance/README.md
Author SHA1 Message Date
Koichi ITO
9603b2fe93 Support running MCP client conformance tests against the Ruby SDK
## Motivation and Context

PR #248 introduced server conformance testing using `@modelcontextprotocol/conformance`.
This extends coverage to the client side so that `MCP::Client` behavior can also be validated against
the MCP specification.

### Scope

Client conformance is driven by the conformance runner, which spawns test servers for each scenario
and invokes `conformance/client.rb` against them. The client script delegates the MCP initialize
handshake to `MCP::Client#connect` and executes scenario-specific operations (for example listing
tools or calling a tool).

### Usage

Run all conformance tests (server + client):

```console
bundle exec rake conformance
```

Run a single client scenario:

```console
bundle exec rake conformance SCENARIO=initialize
```

List available scenarios:

```console
bundle exec rake conformance_list
```

## How Has This Been Tested?

Ran `bundle exec rake conformance` locally.

- Server conformance: 40 passed, 0 failed.
- Client conformance: 3 passed, 54 failed (all 54 listed in `expected_failures.yml`).

## Breaking Change

No. This change only adds test infrastructure under `conformance/` and does not modify SDK behavior.
2026-05-08 15:11:28 +09:00
Koichi ITO
13bbd2937d Support running MCP conformance tests against the Ruby SDK
## Motivation

To assess and track the Ruby SDK's compliance with the MCP
specification, integrate `@modelcontextprotocol/conformance` as the
standard evaluation harness. This makes it possible to measure
conformance objectively, catch regressions early, and produce the kind
of evaluation data needed for SDK tier assessments.

## Scope

Only server conformance is included at this point. Client conformance
requires protocol-level features that MCP::Client does not yet
support (initialize handshake, notifications, ping, etc.).

## Usage

Run conformance tests:

```console
bundle exec rake conformance
```

Start the conformance server in the foreground for iterating on a single
scenario without the per-run startup overhead:

```console
bundle exec rake conformance_server
```

## Default Rake Task (`bundle exec rake`)

It is configured to run in order of lower execution cost: `rubocop`, `test`, and then `conformance`.

## CI

`.github/workflows/conformance.yml` file has been added to run the conformance tests in CI.

## SDK Tier Report

The MCP SDK Tier system (https://modelcontextprotocol.io/community/sdk-tiers)
requires SDK maintainers to self-assess and report results to the SDK Working
Group via https://github.com/modelcontextprotocol/modelcontextprotocol/issues.

To generate a full tier assessment report, use the `/mcp-sdk-tier-audit`
slash command from the https://github.com/modelcontextprotocol/conformance repository with
the conformance server running:

```console
# Terminal 1 (this repository): start the conformance server
bundle exec rake conformance_server

# Terminal 2 (conformance repository): run the tier audit skill as a slash command in Claude Code
/mcp-sdk-tier-audit /path/to/modelcontextprotocol/ruby-sdk http://localhost:9292/mcp
```

The skill evaluates conformance pass rate, issue label taxonomy, triage
metrics, documentation coverage, and policy compliance, then produces a
markdown report suitable for tier advancement submissions.
2026-03-02 14:50:03 +09:00