Pin the conformance CLI to the alpha dist-tag

## Motivation and Context

The npm `latest` dist-tag of `@modelcontextprotocol/conformance` points at 0.1.16, the old 0.1 line,
while the 2026-07-28 requirement revisions and the frozen-requirements framework live in the 0.2.0 alpha line
(`alpha` currently resolves 0.2.0-alpha.11). The runners invoked the bare package name, so `rake conformance`
and the conformance CI job resolve whatever `latest` points at: a fresh machine silently downgrades to
the old suite while a warm npx cache may keep serving a newer one, and the suite actually in use depends
on the environment. Pinning the `alpha` dist-tag puts every run on the line the frozen conformance requirements
are anchored to (0.2.0-alpha.10, published 2026-07-27).

Running the real alpha suite surfaced the actual client-leg state, so the expected-failures baseline is
updated to match: `initialize` now passes (the harness accepts 2026-07-28, fulfilling the TODO on the old entry),
and the scenarios pending client-side work are recorded with their reasons: the 2026-07-28 stateless lifecycle
and the SEP-2322 MRTR driver, SEP-2243 custom-header mirroring, client-side JSON Schema preservation,
the DPoP and WIF authorization extensions, and one `auth/authorization-server-migration` check added to
the suite after the alpha.10 anchor.

The manual `npx` examples in `conformance/README.md` carry the same pin so local reproduction runs
the same suite as the rake task.

## How Has This Been Tested?

`npx --yes @modelcontextprotocol/conformance@alpha --version` resolves 0.2.0-alpha.11,
and `bundle exec rake conformance` passes the baseline check on both legs: the server leg reports 73 passed with 0 failed
and no expected failures, and the client leg fails exactly the scenarios the baseline records.

## Breaking Changes

None. The change is confined to the conformance tooling; the shipped gem is untouched.
This commit is contained in:
Koichi ITO 2026-08-08 18:00:43 +09:00
commit 5932521552
4 changed files with 23 additions and 10 deletions

View file

@ -47,10 +47,10 @@ bundle exec rake conformance_server PORT=3000
bundle exec rake conformance_server
# Terminal 2: run all scenarios
npx @modelcontextprotocol/conformance server --url http://localhost:9292/mcp
npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp
# Terminal 2: run a single scenario
npx @modelcontextprotocol/conformance server --url http://localhost:9292/mcp --scenario ping
npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp --scenario ping
```
Keeps the server alive between test runs, which avoids the startup overhead when iterating

View file

@ -1,6 +1,6 @@
# frozen_string_literal: true
# Runs `npx @modelcontextprotocol/conformance client` against the conformance client script.
# Runs `npx @modelcontextprotocol/conformance@alpha client` against the conformance client script.
require "English"
module Conformance
@ -29,7 +29,7 @@ module Conformance
npx_command = [
"npx",
"--yes",
"@modelcontextprotocol/conformance",
"@modelcontextprotocol/conformance@alpha",
"client",
"--command",
"bundle exec ruby #{client_script}",

View file

@ -1,6 +1,19 @@
server: []
client:
# TODO: The conformance framework's client-initialization check pins the expected requested version to 2025-11-25,
# while this SDK's latest protocol version is 2026-07-28. Remove once the @modelcontextprotocol/conformance package
# accepts 2026-07-28.
- initialize
# Pending client-side support for the 2026-07-28 stateless lifecycle: the conformance client
# still connects through the legacy handshake and has no SEP-2322 MRTR driver yet.
- request-metadata
- sep-2322-client-request-state
# SEP-2243 client-side custom-header mirroring and invalid-tool header gating
# are not implemented in `MCP::Client::HTTP` yet.
- http-custom-headers
- http-invalid-tool-headers
# Client-side JSON Schema preservation gaps: `$ref` values must reach the caller unresolved.
- json-schema-ref-no-deref
- json-schema-2020-12-preservation
# Unimplemented authorization extensions.
- auth/dpop
- auth/dpop-nonce
- auth/wif-jwt-bearer
# Fails one check added to the suite after the 0.2.0-alpha.10 requirements anchor.
- auth/authorization-server-migration

View file

@ -1,6 +1,6 @@
# frozen_string_literal: true
# Starts the conformance server and runs `npx @modelcontextprotocol/conformance` against it.
# Starts the conformance server and runs `npx @modelcontextprotocol/conformance@alpha` against it.
require "English"
require "net/http"
require_relative "server"
@ -31,7 +31,7 @@ module Conformance
def build_command
expected_failures_yml = File.expand_path("expected_failures.yml", __dir__)
npx_command = ["npx", "--yes", "@modelcontextprotocol/conformance", "server", "--url", "http://localhost:#{@port}/mcp"]
npx_command = ["npx", "--yes", "@modelcontextprotocol/conformance@alpha", "server", "--url", "http://localhost:#{@port}/mcp"]
npx_command += ["--scenario", @scenario] if @scenario
npx_command += ["--spec-version", @spec_version] if @spec_version
npx_command += ["--verbose"] if @verbose