[CI] Matrix conformance jobs over both spec revisions (#458)

* [CI] Matrix the conformance jobs over both spec revisions

Runs server/client conformance against both 2025-11-25 and 2026-07-28,
each against its own endpoint and baseline. PR pipeline pins an exact
conformance package version; conformance-weekly tracks the moving
target on both dist-tags.

* [CI] Run the draft conformance suite from the alpha package

The conformance repo now ships 2026-07-28 scenarios on its alpha
dist-tag, so the draft targets no longer need a local checkout.
Pinned to the same version the pipeline runs.

* [CI] Baseline the failures --suite all newly exposes

The old workflow ran without --suite all/--spec-version, so several
pre-existing gaps never surfaced in CI. Now that both revisions run
the full suite:

- server: json-schema-2020-12 and resources-templates-read are
  fixture bugs shared by both revisions; 2026-07-28 also needs
  resources/subscribe and resources/unsubscribe declared removed
  under the stateless wire, which the server doesn't yet do
- client: the example client doesn't call add_numbers or echo back
  a tool's inputSchema; against 2026-07-28 it doesn't speak the
  lifecycle or OAuth at all yet, so every such scenario fails

Verified locally: all four conformance jobs (server/client x
2025-11-25/2026-07-28) pass against these baselines.
This commit is contained in:
Christopher Hertel 2026-08-18 23:30:57 +02:00 • committed by GitHub
commit 7ffc739fe7
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 161 additions and 39 deletions

View file

@ -25,4 +25,43 @@ server:
- json-schema-2020-12:json-schema-2020-12-tool-found
- resources-templates-read:wire-schema-valid
client: []
# The client does not speak 2026-07-28 at all yet: it still opens with
# `initialize` regardless of revision, so it never reaches the stateless wire
# (no per-request `_meta`, no Mcp-Method/Mcp-Name headers, no client-side
# requestState) — and separately, the SDK ships no OAuth client, so every
# scenario driving an authorization flow has nothing to exercise.
client:
- request-metadata
- http-standard-headers
- http-custom-headers
- http-invalid-tool-headers:sep-2243-invalid-tool-tools-list-gate
- http-invalid-tool-headers:sep-2243-client-reject-invalid-tool
- sep-2322-client-request-state
- tools_call:tool-add-numbers
- json-schema-2020-12-preservation:json-schema-2020-12-client-tool-found
- json-schema-2020-12-preservation:json-schema-2020-12-client-echo-completed
- auth/metadata-default
- auth/metadata-var1
- auth/metadata-var2
- auth/metadata-var3
- auth/metadata-issuer-mismatch
- auth/basic-cimd
- auth/pre-registration
- auth/scope-from-www-authenticate
- auth/scope-from-scopes-supported
- auth/scope-omitted-when-undefined
- auth/scope-step-up
- auth/scope-retry-limit
- auth/token-endpoint-auth-basic
- auth/token-endpoint-auth-post
- auth/token-endpoint-auth-none
- auth/offline-access-scope
- auth/offline-access-not-supported
- auth/authorization-server-migration:sep-2352-reregister-on-as-change
- auth/iss-supported
- auth/iss-not-advertised
- auth/iss-supported-missing
- auth/iss-wrong-issuer
- auth/iss-unexpected
- auth/iss-normalized

View file

@ -1,8 +1,19 @@
server: []
# Both pre-existing fixture bugs, not lifecycle-specific: json-schema-2020-12
# wants a `json_schema_2020_12_tool` fixture neither conformance server
# defines, and resources-templates-read emits a resource-template URI in a
# shape the schema rejects. Reproduce identically on the 2026-07-28 endpoint
# (see conformance-baseline-2026-07-28.yml).
server:
- json-schema-2020-12:json-schema-2020-12-tool-found
- resources-templates-read:wire-schema-valid
client:
- elicitation-sep1034-client-defaults
- sse-retry
# The example client doesn't call add_numbers or echo the tool's inputSchema
# back, so these checks have nothing to exercise.
- tools_call:tool-add-numbers
- json-schema-2020-12-preservation:json-schema-2020-12-client-echo-completed
- auth/metadata-default
- auth/metadata-var1
- auth/metadata-var2