modelcontextprotocol/conformance
Watch
1
0
Fork
You've already forked conformance
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Allow expected-failures entries to name a single check (#406)

Browse source 
A baseline entry names a scenario, but a scenario is many checks:
server-stateless is 29. Excusing the one or two a spec change breaks
means baselining the whole scenario, which stops enforcing the rest.

An entry may now be '<scenario>:<check-id>', in which case every failing
check in that scenario is judged on its own. Bare scenario entries are
unchanged, and YAML parses '- a:b' as a plain string, so every existing
baseline keeps working untouched.

loadExpectedFailures now owns the entry grammar and returns parsed
BaselineEntry values, so evaluateBaseline only matches and cannot throw.
Rejecting a bad entry and interpreting a good one no longer live in
separate layers.

A check id matches all of its occurrences, since ids repeat within a run;
they collapse to one verdict, most-severe first. Absent and SKIPPED are
tolerated, matching how the runner already treats them as green.

Move collapseDuplicateChecks to src/checks/collapse.ts with its tests:
expected-failures.ts needs the reducer and cannot import dpop.ts's auth
server stack. No behavior change.

Reject two configs that previously coerced to an entry matching nothing:
a mapping ('- scenario: check-id', with a space) and an empty list item.
This commit is contained in:
Felix Weinberger 2026-07-20 16:20:32 +01:00 • committed by GitHub
commit da56f6631c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 564 additions and 95 deletions
Download patch file Download diff file Expand all files Collapse all files

34
README.md
View file

@ -134,6 +134,40 @@ This ensures:
- CI fails on new regressions (unexpected failures)
- CI fails when a fix lands but the baseline isn't updated (stale entries)
### Baselining a single check
A scenario is many checks — `server-stateless` alone is over twenty — so baselining the
whole scenario to excuse one of them stops enforcing the other nineteen. An entry can
instead name a single check, as `<scenario>:<check-id>`:
```yaml
server:
- tasks-lifecycle # whole scenario may fail
- server-stateless:sep-2575-server-implements-discover # only this check may fail
```
The check id is the left-hand column the runner already prints for each check, so it can
be copied straight out of a failing run.
With a per-check entry, every failing check in that scenario is judged on its own: the
named one is excused, and any other failure is still an unexpected regression. The four
exit-code rules above apply per check rather than per scenario, with one addition — a
baselined check that is absent or skipped is tolerated, because a scenario that bails on
a failed prerequisite legitimately never reaches its later checks, and the prerequisite
reports its own failure anyway.
Two things to know:
- **A check id addresses every occurrence of that id.** Ids repeat within a run (a loop,
a retried flow), and the occurrences collapse to one verdict, most-severe first. So
baselining a repeated id excuses all of its occurrences — coarser than ideal, still far
narrower than baselining the scenario.
- **Mind the space.** `- scenario:check-id` is a string; `- scenario: check-id` is YAML
for a mapping and is rejected with an error.
A scenario cannot be listed both wholesale and per-check — the wholesale entry already
excuses everything, so the pair is contradictory and is rejected.
## GitHub Action
This repo provides a composite GitHub Action so SDK repos don't need to write their own conformance scripts.