Commit graph ruby-sdk/conformance/client_runner.rb
Author SHA1 Message Date
Koichi ITO
5932521552 Pin the conformance CLI to the alpha dist-tag
## Motivation and Context

The npm `latest` dist-tag of `@modelcontextprotocol/conformance` points at 0.1.16, the old 0.1 line,
while the 2026-07-28 requirement revisions and the frozen-requirements framework live in the 0.2.0 alpha line
(`alpha` currently resolves 0.2.0-alpha.11). The runners invoked the bare package name, so `rake conformance`
and the conformance CI job resolve whatever `latest` points at: a fresh machine silently downgrades to
the old suite while a warm npx cache may keep serving a newer one, and the suite actually in use depends
on the environment. Pinning the `alpha` dist-tag puts every run on the line the frozen conformance requirements
are anchored to (0.2.0-alpha.10, published 2026-07-27).

Running the real alpha suite surfaced the actual client-leg state, so the expected-failures baseline is
updated to match: `initialize` now passes (the harness accepts 2026-07-28, fulfilling the TODO on the old entry),
and the scenarios pending client-side work are recorded with their reasons: the 2026-07-28 stateless lifecycle
and the SEP-2322 MRTR driver, SEP-2243 custom-header mirroring, client-side JSON Schema preservation,
the DPoP and WIF authorization extensions, and one `auth/authorization-server-migration` check added to
the suite after the alpha.10 anchor.

The manual `npx` examples in `conformance/README.md` carry the same pin so local reproduction runs
the same suite as the rake task.

## How Has This Been Tested?

`npx --yes @modelcontextprotocol/conformance@alpha --version` resolves 0.2.0-alpha.11,
and `bundle exec rake conformance` passes the baseline check on both legs: the server leg reports 73 passed with 0 failed
and no expected failures, and the client leg fails exactly the scenarios the baseline records.

## Breaking Changes

None. The change is confined to the conformance tooling; the shipped gem is untouched.
2026-08-08 18:47:24 +09:00
Koichi ITO
9603b2fe93 Support running MCP client conformance tests against the Ruby SDK
## Motivation and Context

PR #248 introduced server conformance testing using `@modelcontextprotocol/conformance`.
This extends coverage to the client side so that `MCP::Client` behavior can also be validated against
the MCP specification.

### Scope

Client conformance is driven by the conformance runner, which spawns test servers for each scenario
and invokes `conformance/client.rb` against them. The client script delegates the MCP initialize
handshake to `MCP::Client#connect` and executes scenario-specific operations (for example listing
tools or calling a tool).

### Usage

Run all conformance tests (server + client):

```console
bundle exec rake conformance
```

Run a single client scenario:

```console
bundle exec rake conformance SCENARIO=initialize
```

List available scenarios:

```console
bundle exec rake conformance_list
```

## How Has This Been Tested?

Ran `bundle exec rake conformance` locally.

- Server conformance: 40 passed, 0 failed.
- Client conformance: 3 passed, 54 failed (all 54 listed in `expected_failures.yml`).

## Breaking Change

No. This change only adds test infrastructure under `conformance/` and does not modify SDK behavior.
2026-05-08 15:11:28 +09:00