Add OAuth 2.1 client support for MCP authorization flow

## Motivation and Context

The MCP Authorization spec (2025-11-25) requires clients to support
Protected Resource Metadata discovery (RFC 9728), Authorization
Server Metadata discovery (RFC 8414), Dynamic Client Registration,
and OAuth 2.1 Authorization Code grant with PKCE (S256). The Ruby SDK
previously had no client-side OAuth support: the HTTP transport raised
`RequestHandlerError` on a 401 with no recovery, and 21 `auth/*` scenarios
were listed as known failures in `conformance/expected_failures.yml`.

This aligns the Ruby SDK with the Python and TypeScript SDKs,
both of which expose a pluggable OAuth client provider that drives
the full discovery + DCR + authorization + token exchange flow.
`MCP::Client::OAuth::Provider` mirrors that surface: callers supply
`client_metadata`, a `redirect_uri`, and pluggable `redirect_handler`
/ `callback_handler` hooks plus a token/client information storage.
`MCP::Client::OAuth::InMemoryStorage` is provided as a default.

`MCP::Client::HTTP` accepts a new optional `oauth:` keyword. When present,
the transport attaches `Authorization: Bearer` to every request,
and on a 401 parses `WWW-Authenticate` for the `resource_metadata` URL
and `scope` challenge, drives the discovery + DCR + authorization code flow
through the provider, and retries the failed request once with the acquired token.

Spec compliance details:

- The `WWW-Authenticate` parser handles multiple challenges
  (e.g. `Basic ..., Bearer ...` or `Bearer ..., DPoP ...`), extracting only
  the parameters that belong to the Bearer challenge.
- The PRM `resource` value is required to cover the MCP server URL
  (same scheme/host/port, with the PRM path as a prefix of the server path)
  before the flow continues, preventing redirection of issued tokens to a different audience.
- Both authorization and token requests always include a `resource`
  parameter (RFC 8707), defaulting to a canonicalized form of
  the MCP server URL when PRM does not advertise one.
- `client_secret_basic`, `client_secret_post`, and `none` token endpoint
  authentication methods are supported. When pre-registered client information
  specifies no `token_endpoint_auth_method`, the transport defaults to
  `client_secret_basic` per RFC 6749 §2.3.1 an dthe TS/Python SDK convention.
  Pre-registered client credentials supplied via `Storage#save_client_information`
  with either string or symbol keys are honored, skipping DCR.
- Scope resolution follows the spec order: the `WWW-Authenticate` challenge scope wins first,
  then PRM `scopes_supported`, and `Provider#scope` is the last-resort fallback only
  if neither is supplied (the provider-supplied scope must not pre-empt server-advertised scope).
- Token endpoint error responses (RFC 6749 §5.2) are parsed: a 4x with
  `error: "invalid_grant"` raises `Flow::InvalidGrantError`,
  so `MCP::Client::HTTP` can discard the stored refresh token only when
  the AS truly says it is dead. Generic `AuthorizationError`
  (5xx, network failures, other error codes) leaves the refresh token
  intact, so a transient AS outage does not force interactive reauth
  on the next attempt.
- The OAuth `state` parameter is compared via
  `OpenSSL.fixed_length_secure_compare` (with a `bytesize` guard) to
  prevent timing-based discovery of the expected value.
- The 401 retry guard is a request-scoped local variable, so an MCP server
  that keeps returning 401 is surfaced as an error after one
  retry rather than triggering an infinite re-authorization loop.

## How Has This Been Tested?

Unit tests cover PKCE generation, multi-challenge `WWW-Authenticate` parsing,
PRM and Authorization Server metadata URL builders, the full Flow orchestrator
(success path, state mismatch, missing PKCE support, skipping DCR for both string
and symbol-keyed preregistered client information, `client_secret_basic`
token endpoint authentication, PRM `resource` mismatch rejection, fallback to
the server URL when PRM omits `resource`, scope resolution preferring PRM
`scopes_supported` over `Provider#scope`, default `client_secret_basic` for
confidential clients without an explicit `token_endpoint_auth_method`,
and the new `InvalidGrantError` vs generic `AuthorizationError` split for
token endpoint rejections), and the HTTP transport behavior
(401 recovery + retry, no-provider fallthrough, no infinite loop on repeated 401,
no infinite loop on flow failure, refresh-token retention on transient 5xx from
the token endpoint).

Conformance: `bundle exec rake conformance` newly passes 16 `auth/*`
scenarios beyond the existing baseline, including
`auth/metadata-default`, `metadata-var{1,2,3}`,
`scope-from-www-authenticate`, `scope-from-scopes-supported`,
`scope-omitted-when-undefined`, `scope-retry-limit`,
`token-endpoint-auth-{basic,post,none}`, `pre-registration`,
`resource-mismatch`, and `offline-access-not-supported`. Remaining
`auth/*` scenarios stay in `expected_failures.yml`.
`bundle exec rake test rubocop` is clean.

## Breaking Changes

None. `MCP::Client::HTTP#initialize` accepts a new optional `oauth:`
keyword that defaults to `nil`, preserving existing behavior. When unset,
the transport raises `MCP::Client::RequestHandlerError` on a 401 exactly as before.

conformance/client.rb

@@ -8,6 +8,8 @@
 # The scenario name is read from the MCP_CONFORMANCE_SCENARIO environment variable,
 # which is set automatically by the conformance test runner.
 
+require "faraday"
+require "json"
 require_relative "../lib/mcp"
 
 scenario = ENV["MCP_CONFORMANCE_SCENARIO"]
@@ -17,7 +19,82 @@ unless scenario && server_url
   abort("Usage: MCP_CONFORMANCE_SCENARIO=<scenario> ruby conformance/client.rb <server-url>")
 end
 
-transport = MCP::Client::HTTP.new(url: server_url)
+# The conformance harness optionally injects scenario-specific data via
+# the `MCP_CONFORMANCE_CONTEXT` environment variable as a JSON document. The shape is
+# defined by the harness, not the MCP spec, and has varied between versions:
+#
+# - Newer (`@modelcontextprotocol/conformance` >= 0.x): scenario fields are
+#   spread at the top level alongside `name`, e.g.
+#   `{"name":"auth/pre-registration","client_id":"...","client_secret":"..."}`.
+# - Older: a nested `context` object: `{"name":"...","context":{...}}`.
+#
+# Both shapes are accepted so the client conforms to whichever harness version
+# the developer has on hand.
+def conformance_context
+  raw = ENV["MCP_CONFORMANCE_CONTEXT"]
+  return {} if raw.nil? || raw.empty?
+
+  parsed = JSON.parse(raw)
+  return {} unless parsed.is_a?(Hash)
+
+  if parsed["context"].is_a?(Hash)
+    parsed["context"]
+  else
+    parsed.reject { |key, _| key == "name" }
+  end
+rescue JSON::ParserError
+  {}
+end
+
+# Builds an OAuth provider that drives the authorization code + PKCE + DCR flow
+# non-interactively against the conformance test's auth server. The conformance
+# `/authorize` endpoint redirects synchronously to `redirect_uri` with
+# `code=test-auth-code`, so we follow it manually instead of opening a browser.
+def build_oauth_provider(context)
+  callback_holder = {}
+  redirect_uri = "http://localhost:0/callback"
+
+  redirect_handler = ->(authorization_url) do
+    response = Faraday.new.get(authorization_url) do |req|
+      req.options.params_encoder = nil
+    end
+    location = response.headers["location"] || response.headers["Location"]
+    abort("Authorization request did not redirect: #{response.status}.") unless location
+
+    callback_holder[:url] = URI.parse(location)
+  end
+
+  callback_handler = -> do
+    query = URI.decode_www_form(callback_holder.fetch(:url).query).to_h
+    [query["code"], query["state"]]
+  end
+
+  storage = MCP::Client::OAuth::InMemoryStorage.new
+  if context["client_id"]
+    storage.save_client_information(
+      "client_id" => context["client_id"],
+      "client_secret" => context["client_secret"],
+      "token_endpoint_auth_method" => context["token_endpoint_auth_method"] || "client_secret_basic",
+    )
+  end
+
+  MCP::Client::OAuth::Provider.new(
+    client_metadata: {
+      client_name: "ruby-sdk-conformance-client",
+      redirect_uris: [redirect_uri],
+      grant_types: ["authorization_code"],
+      response_types: ["code"],
+      token_endpoint_auth_method: "none",
+    },
+    redirect_uri: redirect_uri,
+    redirect_handler: redirect_handler,
+    callback_handler: callback_handler,
+    storage: storage,
+  )
+end
+
+oauth = scenario.start_with?("auth/") ? build_oauth_provider(conformance_context) : nil
+transport = MCP::Client::HTTP.new(url: server_url, oauth: oauth)
 client = MCP::Client.new(transport: transport)
 client.connect(client_info: { name: "ruby-sdk-conformance-client", version: MCP::VERSION })
 
@@ -29,6 +106,11 @@ when "tools_call"
   add_numbers = tools.find { |t| t.name == "add_numbers" }
   abort("Tool add_numbers not found") unless add_numbers
   client.call_tool(tool: add_numbers, arguments: { a: 1, b: 2 })
+when %r|\Aauth/|
+  # Auth-only scenarios: the protocol-level checks (PRM/AS metadata, DCR, PKCE, token usage)
+  # are observed by the conformance server during `connect` and the subsequent request below.
+  # Listing tools forces a second authenticated MCP request so the bearer token usage check fires.
+  client.tools
 else
   abort("Unknown or unsupported scenario: #{scenario}")
 end

conformance/expected_failures.yml

@@ -4,25 +4,12 @@ client:
   - sse-retry
   # TODO: Elicitation not implemented in Ruby client.
   - elicitation-sep1034-client-defaults
-  # TODO: OAuth/auth not implemented in Ruby client.
-  - auth/metadata-default
-  - auth/metadata-var1
-  - auth/metadata-var2
-  - auth/metadata-var3
+  # TODO: Remaining OAuth/auth scenarios not yet implemented in Ruby client.
   - auth/basic-cimd
-  - auth/scope-from-www-authenticate
-  - auth/scope-from-scopes-supported
-  - auth/scope-omitted-when-undefined
   - auth/scope-step-up
-  - auth/scope-retry-limit
-  - auth/token-endpoint-auth-basic
-  - auth/token-endpoint-auth-post
-  - auth/token-endpoint-auth-none
-  - auth/pre-registration
   - auth/2025-03-26-oauth-metadata-backcompat
   - auth/2025-03-26-oauth-endpoint-fallback
   - auth/client-credentials-jwt
   - auth/client-credentials-basic
   - auth/cross-app-access-complete-flow
   - auth/offline-access-scope
-  - auth/offline-access-not-supported