## Motivation and Context
The MCP authorization specification (2025-11-25 scope-selection-strategy)
and RFC 6750 Section 3.1 describe OAuth 2.0 step-up authentication:
the client already holds a valid access token, but the server returns a 403
with `WWW-Authenticate: Bearer error="insufficient_scope", scope="..."`
to demand a higher-privileged scope for a specific operation.
The client is expected to run a fresh authorization request with
the escalated scope and retry.
Until now the Ruby SDK only ran the OAuth flow on 401 responses.
A 403 surfaced unchanged as a `RequestHandlerError` and the operation
that needed the escalated scope failed permanently, even with a valid token
in storage. This lined up with the `auth/scope-step-up` conformance
scenario being listed in `conformance/expected_failures.yml`.
The change adds:
- `MCP::Client::HTTP#send_request` recognises a 403 carrying
an `insufficient_scope` Bearer challenge as a step-up signal,
re-runs a full authorization request with the scope from the challenge,
and retries the original request. Retries are capped at
one per `send_request` call so a server that repeatedly demands more
scope cannot loop the transport.
- `run_step_up_flow!` deliberately bypasses the refresh-token path that
`run_oauth_flow!` uses on 401. Refreshing would re-issue the same
scope set the existing token already has and the server already
rejected. Only a fresh `authorization_code` grant with the escalated
scope can succeed.
- A plain 403 with no `insufficient_scope` challenge continues to
surface as `RequestHandlerError` with `error_type: :forbidden`.
- `conformance/client.rb` invokes a tool for the `auth/scope-step-up`
scenario; the scenario's authorization middleware only requires
the escalated scope on `tools/call`, so the previous `tools/list`-only
client could never trigger the second authorization.
- `auth/scope-step-up` is removed from `conformance/expected_failures.yml`.
Resolves#383.
## How Has This Been Tested?
New `HTTPOAuthTest` cases cover: the 403 insufficient_scope path
re-authorizes with the escalated scope from the challenge and retries
the original request with the new bearer token; a plain 403 without
the `insufficient_scope` challenge is surfaced as
`error_type: :forbidden` with no OAuth side effects; repeated
`insufficient_scope` responses retry at most once and then surface the 403;
a provider with a saved `refresh_token` still goes through
the full authorization request rather than exchanging the refresh token,
because the existing scope set is precisely what was rejected.
Conformance: `auth/scope-step-up` now passes 22/22 with no warnings.
`bundle exec rake test`, `bundle exec rake rubocop`, and
`bundle exec rake conformance` are all green.
## Breaking Changes
None. Behaviour on 401 is unchanged. A 403 without the `insufficient_scope`
Bearer challenge still raises the same `RequestHandlerError` as before.
The new retry only fires when both the WWW-Authenticate header signals step-up
and the transport was constructed with an OAuth provider.
a407cea086