13bbd2937d
## Motivation To assess and track the Ruby SDK's compliance with the MCP specification, integrate `@modelcontextprotocol/conformance` as the standard evaluation harness. This makes it possible to measure conformance objectively, catch regressions early, and produce the kind of evaluation data needed for SDK tier assessments. ## Scope Only server conformance is included at this point. Client conformance requires protocol-level features that MCP::Client does not yet support (initialize handshake, notifications, ping, etc.). ## Usage Run conformance tests: ```console bundle exec rake conformance ``` Start the conformance server in the foreground for iterating on a single scenario without the per-run startup overhead: ```console bundle exec rake conformance_server ``` ## Default Rake Task (`bundle exec rake`) It is configured to run in order of lower execution cost: `rubocop`, `test`, and then `conformance`. ## CI `.github/workflows/conformance.yml` file has been added to run the conformance tests in CI. ## SDK Tier Report The MCP SDK Tier system (https://modelcontextprotocol.io/community/sdk-tiers) requires SDK maintainers to self-assess and report results to the SDK Working Group via https://github.com/modelcontextprotocol/modelcontextprotocol/issues. To generate a full tier assessment report, use the `/mcp-sdk-tier-audit` slash command from the https://github.com/modelcontextprotocol/conformance repository with the conformance server running: ```console # Terminal 1 (this repository): start the conformance server bundle exec rake conformance_server # Terminal 2 (conformance repository): run the tier audit skill as a slash command in Claude Code /mcp-sdk-tier-audit /path/to/modelcontextprotocol/ruby-sdk http://localhost:9292/mcp ``` The skill evaluates conformance pass rate, issue label taxonomy, triage metrics, documentation coverage, and policy compliance, then produces a markdown report suitable for tier advancement submissions.
56 lines
1.4 KiB
Ruby
56 lines
1.4 KiB
Ruby
# frozen_string_literal: true
|
|
|
|
require "bundler/gem_tasks"
|
|
require "rake/testtask"
|
|
|
|
Rake::TestTask.new(:test) do |t|
|
|
t.ruby_opts = ["-W0", "-W:deprecated"]
|
|
t.libs << "test"
|
|
t.libs << "lib"
|
|
t.test_files = FileList["test/**/*_test.rb"]
|
|
end
|
|
|
|
require "rubocop/rake_task"
|
|
|
|
RuboCop::RakeTask.new
|
|
|
|
task default: [:rubocop, :test, :conformance]
|
|
|
|
desc "Run MCP conformance tests (PORT, SCENARIO, SPEC_VERSION, VERBOSE)"
|
|
task :conformance do |t|
|
|
next unless npx_available?(t.name)
|
|
|
|
require_relative "conformance/runner"
|
|
|
|
options = {}
|
|
options[:port] = Integer(ENV["PORT"]) if ENV["PORT"]
|
|
options[:scenario] = ENV["SCENARIO"] if ENV["SCENARIO"]
|
|
options[:spec_version] = ENV["SPEC_VERSION"] if ENV["SPEC_VERSION"]
|
|
options[:verbose] = true if ENV["VERBOSE"]
|
|
|
|
Conformance::Runner.new(**options).run
|
|
end
|
|
|
|
desc "List available conformance scenarios"
|
|
task :conformance_list do |t|
|
|
next unless npx_available?(t.name)
|
|
|
|
system("npx", "--yes", "@modelcontextprotocol/conformance", "list", "--server")
|
|
end
|
|
|
|
desc "Start the conformance server (PORT)"
|
|
task :conformance_server do
|
|
require_relative "conformance/server"
|
|
|
|
options = {}
|
|
options[:port] = Integer(ENV["PORT"]) if ENV["PORT"]
|
|
|
|
Conformance::Server.new(**options).start
|
|
end
|
|
|
|
def npx_available?(task_name)
|
|
return true if system("which", "npx", out: File::NULL, err: File::NULL)
|
|
|
|
warn("Skipping #{task_name}: npx is not installed. Install Node.js to run this task: https://nodejs.org/")
|
|
false
|
|
end
|