Support running MCP conformance tests against the Ruby SDK

## Motivation

To assess and track the Ruby SDK's compliance with the MCP
specification, integrate `@modelcontextprotocol/conformance` as the
standard evaluation harness. This makes it possible to measure
conformance objectively, catch regressions early, and produce the kind
of evaluation data needed for SDK tier assessments.

## Scope

Only server conformance is included at this point. Client conformance
requires protocol-level features that MCP::Client does not yet
support (initialize handshake, notifications, ping, etc.).

## Usage

Run conformance tests:

```console
bundle exec rake conformance
```

Start the conformance server in the foreground for iterating on a single
scenario without the per-run startup overhead:

```console
bundle exec rake conformance_server
```

## Default Rake Task (`bundle exec rake`)

It is configured to run in order of lower execution cost: `rubocop`, `test`, and then `conformance`.

## CI

`.github/workflows/conformance.yml` file has been added to run the conformance tests in CI.

## SDK Tier Report

The MCP SDK Tier system (https://modelcontextprotocol.io/community/sdk-tiers)
requires SDK maintainers to self-assess and report results to the SDK Working
Group via https://github.com/modelcontextprotocol/modelcontextprotocol/issues.

To generate a full tier assessment report, use the `/mcp-sdk-tier-audit`
slash command from the https://github.com/modelcontextprotocol/conformance repository with
the conformance server running:

```console
# Terminal 1 (this repository): start the conformance server
bundle exec rake conformance_server

# Terminal 2 (conformance repository): run the tier audit skill as a slash command in Claude Code
/mcp-sdk-tier-audit /path/to/modelcontextprotocol/ruby-sdk http://localhost:9292/mcp
```

The skill evaluates conformance pass rate, issue label taxonomy, triage
metrics, documentation coverage, and policy compliance, then produces a
markdown report suitable for tier advancement submissions.
This commit is contained in:
Koichi ITO 2026-03-01 11:16:07 +09:00
commit 13bbd2937d

29
.github/workflows/conformance.yml vendored Normal file
View file

@ -0,0 +1,29 @@
name: Conformance Tests
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
concurrency:
group: conformance-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
server-conformance:
runs-on: ubuntu-latest
continue-on-error: true
steps:
- uses: actions/checkout@v6
- uses: ruby/setup-ruby@v1
with:
ruby-version: '4.0' # Specify the latest supported Ruby version.
bundler-cache: true
- uses: actions/setup-node@v4
with:
node-version: '24' # Specify the latest Node.js version.
- run: bundle exec rake conformance