modelcontextprotocol/ruby-sdk
Watch
1
0
Fork
You've already forked ruby-sdk
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

[Conformance] Report expected client terminal failures quietly

Browse source 
## Motivation and Context

Negative auth scenarios such as `auth/resource-mismatch` and `auth/scope-retry-limit` are
designed to drive the client into a terminal failure: the SDK correctly refuses
a mismatched protected-resource metadata `resource` and stops after the scope step-up retry limit.
The conformance harness asserts on the observed protocol exchange and already marks these scenarios as passed,
but the client script had no top-level rescue, so each run leaked a raw Ruby backtrace to stderr inside a noisy
"Client exited with code 1" block.

This change wraps the connect and scenario dispatch in a rescue that reports the expected `AuthorizationError`,
`RequestHandlerError`, and `Faraday::Error` outcomes as a concise one-line reason on stderr and exits with status 0.
The harness echoes the client's output only when the exit status is non-zero and does not consult the exit status
for its verdict, so the expected failures no longer produce any output block at all; the one-line reason remains
visible in manual runs and in the harness `--output-dir` stderr artifact. Any unexpected error still raises with
a full backtrace and a failing exit status so genuine SDK bugs remain visible.

## How Has This Been Tested?

Ran `bundle exec rake conformance`: the server suite passes 40/40 and the client suite passes 280/280
with no "Client exited with code" or "Stderr:" blocks in the output. Exercised `auth/resource-mismatch`
and `auth/scope-retry-limit` individually via `Conformance::ClientRunner`; both pass with clean output.

## Breaking Changes

None. Only the conformance client script output changes; no library code is affected.
This commit is contained in:
Koichi ITO 2026-07-07 03:42:37 +09:00
commit f3ffe43cdb
1 changed files with 61 additions and 49 deletions
Download patch file Download diff file Expand all files Collapse all files

12
conformance/client.rb
View file

@ -163,6 +163,14 @@ oauth = scenario.start_with?("auth/") ? build_provider_for(scenario, conformance
transport = MCP::Client::HTTP.new(url: server_url, oauth: oauth) transport = MCP::Client::HTTP.new(url: server_url, oauth: oauth)
client = MCP::Client.new(transport: transport) client = MCP::Client.new(transport: transport)
capabilities = scenario == "elicitation-sep1034-client-defaults" ? { elicitation: {} } : {} capabilities = scenario == "elicitation-sep1034-client-defaults" ? { elicitation: {} } : {}
# The conformance harness asserts on the observed protocol exchange, not on the client's exit status,
# and echoes the client's output only when the exit status is non-zero. Several negative auth scenarios
# (e.g. `auth/resource-mismatch`, `auth/scope-retry-limit`) are expected to drive the client into
# a terminal failure: the SDK correctly refuses a mismatched resource or stops after the scope step-up
# retry limit. Report such an expected failure as a one-line reason and exit cleanly so the run stays
# free of noise, while letting any unexpected error (a real SDK bug) still raise with a full backtrace
# and a failing exit status.
begin
client.connect( client.connect(
client_info: { name: "ruby-sdk-conformance-client", version: MCP::VERSION }, client_info: { name: "ruby-sdk-conformance-client", version: MCP::VERSION },
capabilities: capabilities, capabilities: capabilities,
@ -215,3 +223,7 @@ when %r|\Aauth/|
else else
abort("Unknown or unsupported scenario: #{scenario}") abort("Unknown or unsupported scenario: #{scenario}")
end end
rescue MCP::Client::OAuth::Flow::AuthorizationError, MCP::Client::RequestHandlerError, Faraday::Error => e
warn("#{scenario}: client ended in the expected terminal failure (#{e.class}: #{e.message})")
exit
end