modelcontextprotocol/ruby-sdk
Watch
1
0
Fork
You've already forked ruby-sdk
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Support running MCP client conformance tests against the Ruby SDK

Browse source 
## Motivation and Context

PR #248 introduced server conformance testing using `@modelcontextprotocol/conformance`.
This extends coverage to the client side so that `MCP::Client` behavior can also be validated against
the MCP specification.

### Scope

Client conformance is driven by the conformance runner, which spawns test servers for each scenario
and invokes `conformance/client.rb` against them. The client script delegates the MCP initialize
handshake to `MCP::Client#connect` and executes scenario-specific operations (for example listing
tools or calling a tool).

### Usage

Run all conformance tests (server + client):

```console
bundle exec rake conformance
```

Run a single client scenario:

```console
bundle exec rake conformance SCENARIO=initialize
```

List available scenarios:

```console
bundle exec rake conformance_list
```

## How Has This Been Tested?

Ran `bundle exec rake conformance` locally.

- Server conformance: 40 passed, 0 failed.
- Client conformance: 3 passed, 54 failed (all 54 listed in `expected_failures.yml`).

## Breaking Change

No. This change only adds test infrastructure under `conformance/` and does not modify SDK behavior.
This commit is contained in:
Koichi ITO 2026-03-13 03:33:13 +09:00
commit 9603b2fe93
1 changed files with 131 additions and 17 deletions
Download patch file Download diff file Expand all files Collapse all files

28
conformance/README.md
View file

@ -15,22 +15,22 @@ Validates the Ruby SDK's conformance to the MCP specification using [`@modelcont
bundle exec rake conformance
```
Starts the conformance server, runs all active scenarios against it, prints a pass/fail
summary for each scenario, and exits with a non-zero status code if any unexpected failures
are detected. Scenarios listed in `expected_failures.yml` are allowed to fail without
affecting the exit code.
Runs both server and client conformance tests in sequence. Server conformance starts the
conformance server and tests it. Client conformance spawns test servers for each scenario
and invokes `conformance/client.rb` against them. Scenarios listed in `expected_failures.yml`
are allowed to fail without affecting the exit code.
### Environment variables
| Variable | Description | Default |
|----------------|--------------------------------------|---------|
| `PORT` | Server port | `9292` |
| `SCENARIO` | Run a single scenario by name | (all) |
| `SPEC_VERSION` | Filter scenarios by spec version | (all) |
| `VERBOSE` | Show raw JSON output when set | (off) |
| Variable | Description | Default |
|----------------|---------------------------------------|---------|
| `PORT` | Server port (server conformance only) | `9292` |
| `SCENARIO` | Run a single scenario by name | (all) |
| `SPEC_VERSION` | Filter scenarios by spec version | (all) |
| `VERBOSE` | Show raw JSON output when set | (off) |
```bash
# Run a single scenario
# Run a single server scenario
bundle exec rake conformance SCENARIO=ping
# Use a different port with verbose output
@ -91,8 +91,10 @@ submissions.
```
conformance/
server.rb # Conformance server (Rack + Puma, default port 9292)
runner.rb # Starts the server, runs npx conformance, exits with result code
expected_failures.yml # Baseline of known-failing scenarios
server_runner.rb # Starts the server, runs npx conformance server, exits with result code
client.rb # Conformance client (invoked by npx conformance client)
client_runner.rb # Runs npx conformance client, exits with result code
expected_failures.yml # Baseline of known-failing scenarios (server and client)
README.md # This file
```