modelcontextprotocol/php-sdk
Watch
1
0
Fork
You've already forked php-sdk
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

[CI] Version both baseline files and split conformance badges per spec (#459)

Browse source 
- Rename conformance-baseline.yml to conformance-baseline-2025-11-25.yml
  so both baseline files carry their spec revision symmetrically, and
  bring conformance-server/conformance-client up to the same rigor
  (--suite all, pinned CONFORMANCE, baseline check) as the draft targets
- score.php now takes a required spec-version and writes
  <suite>-conformance-<spec-version>.json, so 2025-11-25 and 2026-07-28
  each get their own badge instead of only the dated one
- conformance-weekly.yaml generates and publishes badges for both
  matrix entries instead of gating on 2025-11-25 only, and drops the
  now-stale unversioned badge files from the badges branch
- README shows all four badges alongside the existing packagist/CI/PHP
  version/license row
- Trimmed workflow comments to the minimum needed
This commit is contained in:
Christopher Hertel 2026-08-18 23:42:36 +02:00 • committed by GitHub
commit 78a8926f17
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 66 additions and 64 deletions
Download patch file Download diff file Expand all files Collapse all files

55
.github/workflows/conformance-weekly.yaml vendored
View file

@ -1,13 +1,8 @@
name: conformance-weekly
# Runs the MCP conformance suite weekly against the newest published
# @modelcontextprotocol/conformance. The on:pull_request pipeline pins an exact
# version; this schedule catches upstream releases that add scenarios between
# PRs, so the pin never silently goes stale.
#
# It also scores each run and publishes the client/server pass-rate as
# shields.io endpoint JSON to the orphan `badges` branch (consumed by the
# README); that branch is created on the first run.
# Tracks the newest published @modelcontextprotocol/conformance; the
# on:pull_request pipeline pins an exact version instead. Also publishes
# score badges to the orphan `badges` branch, consumed by the README.
on:
schedule:
- cron: '0 6 * * 1' # Mondays 06:00 UTC
@ -17,10 +12,6 @@ permissions:
contents: write
issues: write
# The PR pipeline pins an exact conformance version so a PR only goes red for
# reasons in the PR. That pin is only safe because this job tracks the moving
# target instead — on both revisions, since each rides a different dist-tag:
# the dated one is on `latest`, the draft one only on `alpha`.
jobs:
server:
name: conformance / server (${{ matrix.spec-version }}, ${{ matrix.dist-tag }})
@ -32,7 +23,7 @@ jobs:
- spec-version: '2025-11-25'
dist-tag: latest
path: '/'
baseline: conformance-baseline.yml
baseline: conformance-baseline-2025-11-25.yml
- spec-version: '2026-07-28'
dist-tag: alpha
path: '/stateless'
@ -58,11 +49,9 @@ jobs:
--spec-version ${{ matrix.spec-version }}
--expected-failures ${{ matrix.baseline }}
--output-dir results
# The badge tracks the released revision, so only the dated entry feeds
# it; the draft entry runs purely to catch upstream drift.
- name: Generate score badge
if: always() && matrix.spec-version == '2025-11-25'
run: php tests/Conformance/score.php server
if: always()
run: php tests/Conformance/score.php server ${{ matrix.spec-version }}
- name: Show docker logs on failure
if: failure()
run: docker compose -f tests/Conformance/Fixtures/docker-compose.yml logs
@ -75,11 +64,11 @@ jobs:
tests/Conformance/logs
tests/Conformance/results
- name: Upload score badge
if: always() && matrix.spec-version == '2025-11-25'
if: always()
uses: actions/upload-artifact@v7
with:
name: server-badge
path: tests/Conformance/server-conformance.json
name: badge-server-${{ matrix.spec-version }}
path: tests/Conformance/server-conformance-${{ matrix.spec-version }}.json
client:
name: conformance / client (${{ matrix.spec-version }}, ${{ matrix.dist-tag }})
@ -90,7 +79,7 @@ jobs:
include:
- spec-version: '2025-11-25'
dist-tag: latest
baseline: conformance-baseline.yml
baseline: conformance-baseline-2025-11-25.yml
- spec-version: '2026-07-28'
dist-tag: alpha
baseline: conformance-baseline-2026-07-28.yml
@ -115,8 +104,8 @@ jobs:
--expected-failures ${{ matrix.baseline }}
--output-dir results
- name: Generate score badge
if: always() && matrix.spec-version == '2025-11-25'
run: php tests/Conformance/score.php client
if: always()
run: php tests/Conformance/score.php client ${{ matrix.spec-version }}
- name: Upload conformance results
if: failure()
uses: actions/upload-artifact@v7
@ -126,11 +115,11 @@ jobs:
tests/Conformance/logs
tests/Conformance/results
- name: Upload score badge
if: always() && matrix.spec-version == '2025-11-25'
if: always()
uses: actions/upload-artifact@v7
with:
name: client-badge
path: tests/Conformance/client-conformance.json
name: badge-client-${{ matrix.spec-version }}
path: tests/Conformance/client-conformance-${{ matrix.spec-version }}.json
notify:
name: Open issue on failure
@ -156,26 +145,21 @@ jobs:
- Run: $RUN_URL
- Triggered: $(date -u +%FT%TZ)
Upstream likely published a release whose scenarios the SDK does not satisfy. Either fix the SDK, update the conformance fixtures, or add the new failure to \`tests/Conformance/conformance-baseline.yml\`."
Upstream likely published a release whose scenarios the SDK does not satisfy. Either fix the SDK, update the conformance fixtures, or add the new failure to \`tests/Conformance/conformance-baseline-2025-11-25.yml\`."
fi
publish:
name: Publish conformance badges
runs-on: ubuntu-latest
needs: [server, client]
# Publish even when the suite regressed (the badge should reflect reality);
# skip on forks, which cannot push the `badges` branch.
if: ${{ !cancelled() && github.repository == 'modelcontextprotocol/php-sdk' }}
steps:
- uses: actions/checkout@v7
- uses: actions/download-artifact@v8
with:
name: server-badge
path: badges-in
- uses: actions/download-artifact@v8
with:
name: client-badge
pattern: badge-*
path: badges-in
merge-multiple: true
- name: Publish to badges branch
run: |
git config user.name "github-actions[bot]"
@ -190,7 +174,8 @@ jobs:
git -C badges-wt rm -rf --quiet . >/dev/null 2>&1 || true
fi
cp badges-in/server-conformance.json badges-in/client-conformance.json badges-wt/
rm -f badges-wt/server-conformance.json badges-wt/client-conformance.json
cp badges-in/*.json badges-wt/
cd badges-wt
git add -A

13
.github/workflows/pipeline.yaml vendored
View file

@ -6,9 +6,7 @@ permissions:
pull-requests: write
env:
# Pinned so a PR only goes red for reasons in the PR. The 2026-07-28
# scenarios ship on the `alpha` dist-tag; `latest` (0.1.x) has none of them.
# conformance-weekly.yaml is what tracks upstream releases.
# Pinned exact version; conformance-weekly.yaml tracks upstream releases.
CONFORMANCE_PKG: "@modelcontextprotocol/conformance@0.2.0-alpha.11"
jobs:
@ -105,16 +103,11 @@ jobs:
runs-on: ubuntu-latest
strategy:
fail-fast: false
# One entry per lifecycle, not per spec revision: `--spec-version` is
# cumulative across the dated revisions, so 2025-11-25 already covers
# 2025-06-18 and 2025-03-26. 2026-07-28 is not cumulative with them — it
# is the SEP-2575 stateless wire, served from its own endpoint with its
# own baseline, so it needs its own run.
matrix:
include:
- spec-version: '2025-11-25'
path: '/'
baseline: conformance-baseline.yml
baseline: conformance-baseline-2025-11-25.yml
- spec-version: '2026-07-28'
path: '/stateless'
baseline: conformance-baseline-2026-07-28.yml
@ -180,7 +173,7 @@ jobs:
matrix:
include:
- spec-version: '2025-11-25'
baseline: conformance-baseline.yml
baseline: conformance-baseline-2025-11-25.yml
- spec-version: '2026-07-28'
baseline: conformance-baseline-2026-07-28.yml