Support running MCP conformance tests against the Ruby SDK
## Motivation To assess and track the Ruby SDK's compliance with the MCP specification, integrate `@modelcontextprotocol/conformance` as the standard evaluation harness. This makes it possible to measure conformance objectively, catch regressions early, and produce the kind of evaluation data needed for SDK tier assessments. ## Scope Only server conformance is included at this point. Client conformance requires protocol-level features that MCP::Client does not yet support (initialize handshake, notifications, ping, etc.). ## Usage Run conformance tests: ```console bundle exec rake conformance ``` Start the conformance server in the foreground for iterating on a single scenario without the per-run startup overhead: ```console bundle exec rake conformance_server ``` ## Default Rake Task (`bundle exec rake`) It is configured to run in order of lower execution cost: `rubocop`, `test`, and then `conformance`. ## CI `.github/workflows/conformance.yml` file has been added to run the conformance tests in CI. ## SDK Tier Report The MCP SDK Tier system (https://modelcontextprotocol.io/community/sdk-tiers) requires SDK maintainers to self-assess and report results to the SDK Working Group via https://github.com/modelcontextprotocol/modelcontextprotocol/issues. To generate a full tier assessment report, use the `/mcp-sdk-tier-audit` slash command from the https://github.com/modelcontextprotocol/conformance repository with the conformance server running: ```console # Terminal 1 (this repository): start the conformance server bundle exec rake conformance_server # Terminal 2 (conformance repository): run the tier audit skill as a slash command in Claude Code /mcp-sdk-tier-audit /path/to/modelcontextprotocol/ruby-sdk http://localhost:9292/mcp ``` The skill evaluates conformance pass rate, issue label taxonomy, triage metrics, documentation coverage, and policy compliance, then produces a markdown report suitable for tier advancement submissions.
This commit is contained in:
parent
5f16abfd96
commit
13bbd2937d
1 changed files with 835 additions and 1 deletions
9
conformance/expected_failures.yml
Normal file
9
conformance/expected_failures.yml
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
server:
|
||||
# TODO: Server-to-client requests (sampling/createMessage, elicitation/create) are not implemented.
|
||||
# `Transport#send_request` does not exist in the current SDK.
|
||||
- tools-call-sampling
|
||||
- tools-call-elicitation
|
||||
- elicitation-sep1034-defaults
|
||||
- elicitation-sep1330-enums
|
||||
# TODO: The SDK does not extract `_meta.progressToken` from tool call requests or deliver `notifications/progress` to tools.
|
||||
- tools-call-with-progress
|
||||
Loading…
Reference in a new issue