[CI] Matrix conformance jobs over both spec revisions (#458)

* [CI] Matrix the conformance jobs over both spec revisions

Runs server/client conformance against both 2025-11-25 and 2026-07-28,
each against its own endpoint and baseline. PR pipeline pins an exact
conformance package version; conformance-weekly tracks the moving
target on both dist-tags.

* [CI] Run the draft conformance suite from the alpha package

The conformance repo now ships 2026-07-28 scenarios on its alpha
dist-tag, so the draft targets no longer need a local checkout.
Pinned to the same version the pipeline runs.

* [CI] Baseline the failures --suite all newly exposes

The old workflow ran without --suite all/--spec-version, so several
pre-existing gaps never surfaced in CI. Now that both revisions run
the full suite:

- server: json-schema-2020-12 and resources-templates-read are
  fixture bugs shared by both revisions; 2026-07-28 also needs
  resources/subscribe and resources/unsubscribe declared removed
  under the stateless wire, which the server doesn't yet do
- client: the example client doesn't call add_numbers or echo back
  a tool's inputSchema; against 2026-07-28 it doesn't speak the
  lifecycle or OAuth at all yet, so every such scenario fails

Verified locally: all four conformance jobs (server/client x
2025-11-25/2026-07-28) pass against these baselines.
This commit is contained in:
Christopher Hertel 2026-08-18 23:30:57 +02:00 • committed by GitHub
commit 7ffc739fe7
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

View file

@ -5,6 +5,12 @@ permissions:
contents: read
pull-requests: write
env:
# Pinned so a PR only goes red for reasons in the PR. The 2026-07-28
# scenarios ship on the `alpha` dist-tag; `latest` (0.1.x) has none of them.
# conformance-weekly.yaml is what tracks upstream releases.
CONFORMANCE_PKG: "@modelcontextprotocol/conformance@0.2.0-alpha.11"
jobs:
unit:
runs-on: ubuntu-latest
@ -95,8 +101,24 @@ jobs:
run: vendor/bin/phpunit --testsuite=inspector
conformance-server:
name: conformance / server
name: conformance / server (${{ matrix.spec-version }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
# One entry per lifecycle, not per spec revision: `--spec-version` is
# cumulative across the dated revisions, so 2025-11-25 already covers
# 2025-06-18 and 2025-03-26. 2026-07-28 is not cumulative with them — it
# is the SEP-2575 stateless wire, served from its own endpoint with its
# own baseline, so it needs its own run.
matrix:
include:
- spec-version: '2025-11-25'
path: '/'
baseline: conformance-baseline.yml
- spec-version: '2026-07-28'
path: '/stateless'
baseline: conformance-baseline-2026-07-28.yml
steps:
- name: Checkout
uses: actions/checkout@v7
@ -118,7 +140,12 @@ jobs:
- name: Run conformance tests
working-directory: ./tests/Conformance
run: npx @modelcontextprotocol/conformance server --url http://localhost:8000/ --expected-failures conformance-baseline.yml
run: >-
npx --yes ${{ env.CONFORMANCE_PKG }} server
--url http://localhost:8000${{ matrix.path }}
--suite all
--spec-version ${{ matrix.spec-version }}
--expected-failures ${{ matrix.baseline }}
- name: Show logs on failure
if: failure()
@ -146,8 +173,17 @@ jobs:
run: docker compose -f tests/Conformance/Fixtures/docker-compose.yml down
conformance-client:
name: conformance / client
name: conformance / client (${{ matrix.spec-version }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- spec-version: '2025-11-25'
baseline: conformance-baseline.yml
- spec-version: '2026-07-28'
baseline: conformance-baseline-2026-07-28.yml
steps:
- name: Checkout
uses: actions/checkout@v7
@ -171,7 +207,12 @@ jobs:
- name: Run client conformance tests
working-directory: ./tests/Conformance
run: npx @modelcontextprotocol/conformance client --command "php ${{ github.workspace }}/tests/Conformance/client.php" --suite all --expected-failures conformance-baseline.yml
run: >-
npx --yes ${{ env.CONFORMANCE_PKG }} client
--command "php ${{ github.workspace }}/tests/Conformance/client.php"
--suite all
--spec-version ${{ matrix.spec-version }}
--expected-failures ${{ matrix.baseline }}
- name: Show logs on failure
if: failure()