7ffc739fe7* [CI] Matrix the conformance jobs over both spec revisions Runs server/client conformance against both 2025-11-25 and 2026-07-28, each against its own endpoint and baseline. PR pipeline pins an exact conformance package version; conformance-weekly tracks the moving target on both dist-tags. * [CI] Run the draft conformance suite from the alpha package The conformance repo now ships 2026-07-28 scenarios on its alpha dist-tag, so the draft targets no longer need a local checkout. Pinned to the same version the pipeline runs. * [CI] Baseline the failures --suite all newly exposes The old workflow ran without --suite all/--spec-version, so several pre-existing gaps never surfaced in CI. Now that both revisions run the full suite: - server: json-schema-2020-12 and resources-templates-read are fixture bugs shared by both revisions; 2026-07-28 also needs resources/subscribe and resources/unsubscribe declared removed under the stateless wire, which the server doesn't yet do - client: the example client doesn't call add_numbers or echo back a tool's inputSchema; against 2026-07-28 it doesn't speak the lifecycle or OAuth at all yet, so every such scenario fails Verified locally: all four conformance jobs (server/client x 2025-11-25/2026-07-28) pass against these baselines.
202 lines
7.7 KiB
YAML
202 lines
7.7 KiB
YAML
name: conformance-weekly
|
|
|
|
# Runs the MCP conformance suite weekly against the newest published
|
|
# @modelcontextprotocol/conformance. The on:pull_request pipeline pins an exact
|
|
# version; this schedule catches upstream releases that add scenarios between
|
|
# PRs, so the pin never silently goes stale.
|
|
#
|
|
# It also scores each run and publishes the client/server pass-rate as
|
|
# shields.io endpoint JSON to the orphan `badges` branch (consumed by the
|
|
# README); that branch is created on the first run.
|
|
on:
|
|
schedule:
|
|
- cron: '0 6 * * 1' # Mondays 06:00 UTC
|
|
workflow_dispatch:
|
|
|
|
permissions:
|
|
contents: write
|
|
issues: write
|
|
|
|
# The PR pipeline pins an exact conformance version so a PR only goes red for
|
|
# reasons in the PR. That pin is only safe because this job tracks the moving
|
|
# target instead — on both revisions, since each rides a different dist-tag:
|
|
# the dated one is on `latest`, the draft one only on `alpha`.
|
|
jobs:
|
|
server:
|
|
name: conformance / server (${{ matrix.spec-version }}, ${{ matrix.dist-tag }})
|
|
runs-on: ubuntu-latest
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- spec-version: '2025-11-25'
|
|
dist-tag: latest
|
|
path: '/'
|
|
baseline: conformance-baseline.yml
|
|
- spec-version: '2026-07-28'
|
|
dist-tag: alpha
|
|
path: '/stateless'
|
|
baseline: conformance-baseline-2026-07-28.yml
|
|
steps:
|
|
- uses: actions/checkout@v7
|
|
- uses: actions/setup-node@v7
|
|
with:
|
|
node-version: '22'
|
|
- run: composer install --prefer-dist --no-progress --no-interaction
|
|
- name: Start conformance server
|
|
run: |
|
|
mkdir -p tests/Conformance/sessions tests/Conformance/logs
|
|
chmod -R 777 tests/Conformance/sessions tests/Conformance/logs
|
|
docker compose -f tests/Conformance/Fixtures/docker-compose.yml up -d
|
|
sleep 5
|
|
- name: Run conformance tests
|
|
working-directory: ./tests/Conformance
|
|
run: >-
|
|
npx --yes @modelcontextprotocol/conformance@${{ matrix.dist-tag }} server
|
|
--url http://localhost:8000${{ matrix.path }}
|
|
--suite all
|
|
--spec-version ${{ matrix.spec-version }}
|
|
--expected-failures ${{ matrix.baseline }}
|
|
--output-dir results
|
|
# The badge tracks the released revision, so only the dated entry feeds
|
|
# it; the draft entry runs purely to catch upstream drift.
|
|
- name: Generate score badge
|
|
if: always() && matrix.spec-version == '2025-11-25'
|
|
run: php tests/Conformance/score.php server
|
|
- name: Show docker logs on failure
|
|
if: failure()
|
|
run: docker compose -f tests/Conformance/Fixtures/docker-compose.yml logs
|
|
- name: Upload conformance results
|
|
if: failure()
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: conformance-server-results-${{ matrix.spec-version }}
|
|
path: |
|
|
tests/Conformance/logs
|
|
tests/Conformance/results
|
|
- name: Upload score badge
|
|
if: always() && matrix.spec-version == '2025-11-25'
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: server-badge
|
|
path: tests/Conformance/server-conformance.json
|
|
|
|
client:
|
|
name: conformance / client (${{ matrix.spec-version }}, ${{ matrix.dist-tag }})
|
|
runs-on: ubuntu-latest
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- spec-version: '2025-11-25'
|
|
dist-tag: latest
|
|
baseline: conformance-baseline.yml
|
|
- spec-version: '2026-07-28'
|
|
dist-tag: alpha
|
|
baseline: conformance-baseline-2026-07-28.yml
|
|
steps:
|
|
- uses: actions/checkout@v7
|
|
- uses: shivammathur/setup-php@v2
|
|
with:
|
|
php-version: '8.4'
|
|
coverage: none
|
|
- uses: actions/setup-node@v7
|
|
with:
|
|
node-version: '22'
|
|
- run: composer install --prefer-dist --no-progress --no-interaction
|
|
- run: mkdir -p tests/Conformance/logs
|
|
- name: Run conformance tests
|
|
working-directory: ./tests/Conformance
|
|
run: >-
|
|
npx --yes @modelcontextprotocol/conformance@${{ matrix.dist-tag }} client
|
|
--command "php ${{ github.workspace }}/tests/Conformance/client.php"
|
|
--suite all
|
|
--spec-version ${{ matrix.spec-version }}
|
|
--expected-failures ${{ matrix.baseline }}
|
|
--output-dir results
|
|
- name: Generate score badge
|
|
if: always() && matrix.spec-version == '2025-11-25'
|
|
run: php tests/Conformance/score.php client
|
|
- name: Upload conformance results
|
|
if: failure()
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: conformance-client-results-${{ matrix.spec-version }}
|
|
path: |
|
|
tests/Conformance/logs
|
|
tests/Conformance/results
|
|
- name: Upload score badge
|
|
if: always() && matrix.spec-version == '2025-11-25'
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: client-badge
|
|
path: tests/Conformance/client-conformance.json
|
|
|
|
notify:
|
|
name: Open issue on failure
|
|
runs-on: ubuntu-latest
|
|
needs: [server, client]
|
|
if: failure() && github.event_name == 'schedule'
|
|
env:
|
|
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
GH_REPO: ${{ github.repository }}
|
|
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
|
steps:
|
|
- name: File or comment tracking issue
|
|
run: |
|
|
existing=$(gh issue list --label conformance-weekly --state open --json number --jq '.[0].number // empty')
|
|
if [ -n "$existing" ]; then
|
|
gh issue comment "$existing" --body "New failure on $(date -u +%FT%TZ): $RUN_URL"
|
|
else
|
|
gh issue create \
|
|
--title '[conformance] Weekly conformance run failed' \
|
|
--label conformance-weekly \
|
|
--body "Weekly conformance against \`@modelcontextprotocol/conformance@latest\` failed.
|
|
|
|
- Run: $RUN_URL
|
|
- Triggered: $(date -u +%FT%TZ)
|
|
|
|
Upstream likely published a release whose scenarios the SDK does not satisfy. Either fix the SDK, update the conformance fixtures, or add the new failure to \`tests/Conformance/conformance-baseline.yml\`."
|
|
fi
|
|
|
|
publish:
|
|
name: Publish conformance badges
|
|
runs-on: ubuntu-latest
|
|
needs: [server, client]
|
|
# Publish even when the suite regressed (the badge should reflect reality);
|
|
# skip on forks, which cannot push the `badges` branch.
|
|
if: ${{ !cancelled() && github.repository == 'modelcontextprotocol/php-sdk' }}
|
|
steps:
|
|
- uses: actions/checkout@v7
|
|
- uses: actions/download-artifact@v8
|
|
with:
|
|
name: server-badge
|
|
path: badges-in
|
|
- uses: actions/download-artifact@v8
|
|
with:
|
|
name: client-badge
|
|
path: badges-in
|
|
- name: Publish to badges branch
|
|
run: |
|
|
git config user.name "github-actions[bot]"
|
|
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
|
|
|
|
if git ls-remote --exit-code --heads origin badges >/dev/null 2>&1; then
|
|
git fetch origin badges
|
|
git worktree add badges-wt badges
|
|
else
|
|
git worktree add --detach badges-wt
|
|
git -C badges-wt checkout --orphan badges
|
|
git -C badges-wt rm -rf --quiet . >/dev/null 2>&1 || true
|
|
fi
|
|
|
|
cp badges-in/server-conformance.json badges-in/client-conformance.json badges-wt/
|
|
|
|
cd badges-wt
|
|
git add -A
|
|
if git diff --cached --quiet; then
|
|
echo "Conformance scores unchanged."
|
|
else
|
|
git commit -m "Update conformance score badges"
|
|
git push origin badges
|
|
fi
|