13bbd2937d
## Motivation To assess and track the Ruby SDK's compliance with the MCP specification, integrate `@modelcontextprotocol/conformance` as the standard evaluation harness. This makes it possible to measure conformance objectively, catch regressions early, and produce the kind of evaluation data needed for SDK tier assessments. ## Scope Only server conformance is included at this point. Client conformance requires protocol-level features that MCP::Client does not yet support (initialize handshake, notifications, ping, etc.). ## Usage Run conformance tests: ```console bundle exec rake conformance ``` Start the conformance server in the foreground for iterating on a single scenario without the per-run startup overhead: ```console bundle exec rake conformance_server ``` ## Default Rake Task (`bundle exec rake`) It is configured to run in order of lower execution cost: `rubocop`, `test`, and then `conformance`. ## CI `.github/workflows/conformance.yml` file has been added to run the conformance tests in CI. ## SDK Tier Report The MCP SDK Tier system (https://modelcontextprotocol.io/community/sdk-tiers) requires SDK maintainers to self-assess and report results to the SDK Working Group via https://github.com/modelcontextprotocol/modelcontextprotocol/issues. To generate a full tier assessment report, use the `/mcp-sdk-tier-audit` slash command from the https://github.com/modelcontextprotocol/conformance repository with the conformance server running: ```console # Terminal 1 (this repository): start the conformance server bundle exec rake conformance_server # Terminal 2 (conformance repository): run the tier audit skill as a slash command in Claude Code /mcp-sdk-tier-audit /path/to/modelcontextprotocol/ruby-sdk http://localhost:9292/mcp ``` The skill evaluates conformance pass rate, issue label taxonomy, triage metrics, documentation coverage, and policy compliance, then produces a markdown report suitable for tier advancement submissions.
29 lines
646 B
YAML
29 lines
646 B
YAML
name: Conformance Tests
|
|
|
|
on:
|
|
push:
|
|
branches: [main]
|
|
pull_request:
|
|
workflow_dispatch:
|
|
|
|
concurrency:
|
|
group: conformance-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
server-conformance:
|
|
runs-on: ubuntu-latest
|
|
continue-on-error: true
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
- uses: ruby/setup-ruby@v1
|
|
with:
|
|
ruby-version: '4.0' # Specify the latest supported Ruby version.
|
|
bundler-cache: true
|
|
- uses: actions/setup-node@v4
|
|
with:
|
|
node-version: '24' # Specify the latest Node.js version.
|
|
- run: bundle exec rake conformance
|