[CI] Matrix conformance jobs over both spec revisions (#458)
* [CI] Matrix the conformance jobs over both spec revisions Runs server/client conformance against both 2025-11-25 and 2026-07-28, each against its own endpoint and baseline. PR pipeline pins an exact conformance package version; conformance-weekly tracks the moving target on both dist-tags. * [CI] Run the draft conformance suite from the alpha package The conformance repo now ships 2026-07-28 scenarios on its alpha dist-tag, so the draft targets no longer need a local checkout. Pinned to the same version the pipeline runs. * [CI] Baseline the failures --suite all newly exposes The old workflow ran without --suite all/--spec-version, so several pre-existing gaps never surfaced in CI. Now that both revisions run the full suite: - server: json-schema-2020-12 and resources-templates-read are fixture bugs shared by both revisions; 2026-07-28 also needs resources/subscribe and resources/unsubscribe declared removed under the stateless wire, which the server doesn't yet do - client: the example client doesn't call add_numbers or echo back a tool's inputSchema; against 2026-07-28 it doesn't speak the lifecycle or OAuth at all yet, so every such scenario fails Verified locally: all four conformance jobs (server/client x 2025-11-25/2026-07-28) pass against these baselines.
This commit is contained in:
parent
dd3783a720
commit
7ffc739fe7
2 changed files with 161 additions and 39 deletions
|
|
@ -25,4 +25,43 @@ server:
|
|||
- json-schema-2020-12:json-schema-2020-12-tool-found
|
||||
- resources-templates-read:wire-schema-valid
|
||||
|
||||
client: []
|
||||
# The client does not speak 2026-07-28 at all yet: it still opens with
|
||||
# `initialize` regardless of revision, so it never reaches the stateless wire
|
||||
# (no per-request `_meta`, no Mcp-Method/Mcp-Name headers, no client-side
|
||||
# requestState) — and separately, the SDK ships no OAuth client, so every
|
||||
# scenario driving an authorization flow has nothing to exercise.
|
||||
client:
|
||||
- request-metadata
|
||||
- http-standard-headers
|
||||
- http-custom-headers
|
||||
- http-invalid-tool-headers:sep-2243-invalid-tool-tools-list-gate
|
||||
- http-invalid-tool-headers:sep-2243-client-reject-invalid-tool
|
||||
- sep-2322-client-request-state
|
||||
- tools_call:tool-add-numbers
|
||||
- json-schema-2020-12-preservation:json-schema-2020-12-client-tool-found
|
||||
- json-schema-2020-12-preservation:json-schema-2020-12-client-echo-completed
|
||||
|
||||
- auth/metadata-default
|
||||
- auth/metadata-var1
|
||||
- auth/metadata-var2
|
||||
- auth/metadata-var3
|
||||
- auth/metadata-issuer-mismatch
|
||||
- auth/basic-cimd
|
||||
- auth/pre-registration
|
||||
- auth/scope-from-www-authenticate
|
||||
- auth/scope-from-scopes-supported
|
||||
- auth/scope-omitted-when-undefined
|
||||
- auth/scope-step-up
|
||||
- auth/scope-retry-limit
|
||||
- auth/token-endpoint-auth-basic
|
||||
- auth/token-endpoint-auth-post
|
||||
- auth/token-endpoint-auth-none
|
||||
- auth/offline-access-scope
|
||||
- auth/offline-access-not-supported
|
||||
- auth/authorization-server-migration:sep-2352-reregister-on-as-change
|
||||
- auth/iss-supported
|
||||
- auth/iss-not-advertised
|
||||
- auth/iss-supported-missing
|
||||
- auth/iss-wrong-issuer
|
||||
- auth/iss-unexpected
|
||||
- auth/iss-normalized
|
||||
|
|
|
|||
|
|
@ -1,8 +1,19 @@
|
|||
server: []
|
||||
# Both pre-existing fixture bugs, not lifecycle-specific: json-schema-2020-12
|
||||
# wants a `json_schema_2020_12_tool` fixture neither conformance server
|
||||
# defines, and resources-templates-read emits a resource-template URI in a
|
||||
# shape the schema rejects. Reproduce identically on the 2026-07-28 endpoint
|
||||
# (see conformance-baseline-2026-07-28.yml).
|
||||
server:
|
||||
- json-schema-2020-12:json-schema-2020-12-tool-found
|
||||
- resources-templates-read:wire-schema-valid
|
||||
|
||||
client:
|
||||
- elicitation-sep1034-client-defaults
|
||||
- sse-retry
|
||||
# The example client doesn't call add_numbers or echo the tool's inputSchema
|
||||
# back, so these checks have nothing to exercise.
|
||||
- tools_call:tool-add-numbers
|
||||
- json-schema-2020-12-preservation:json-schema-2020-12-client-echo-completed
|
||||
- auth/metadata-default
|
||||
- auth/metadata-var1
|
||||
- auth/metadata-var2
|
||||
|
|
|
|||
Loading…
Reference in a new issue