modelcontextprotocol/conformance
Watch
1
0
Fork
You've already forked conformance
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Allow expected-failures entries to name a single check (#406)

Browse source 
A baseline entry names a scenario, but a scenario is many checks:
server-stateless is 29. Excusing the one or two a spec change breaks
means baselining the whole scenario, which stops enforcing the rest.

An entry may now be '<scenario>:<check-id>', in which case every failing
check in that scenario is judged on its own. Bare scenario entries are
unchanged, and YAML parses '- a:b' as a plain string, so every existing
baseline keeps working untouched.

loadExpectedFailures now owns the entry grammar and returns parsed
BaselineEntry values, so evaluateBaseline only matches and cannot throw.
Rejecting a bad entry and interpreting a good one no longer live in
separate layers.

A check id matches all of its occurrences, since ids repeat within a run;
they collapse to one verdict, most-severe first. Absent and SKIPPED are
tolerated, matching how the runner already treats them as green.

Move collapseDuplicateChecks to src/checks/collapse.ts with its tests:
expected-failures.ts needs the reducer and cannot import dpop.ts's auth
server stack. No behavior change.

Reject two configs that previously coerced to an entry matching nothing:
a mapping ('- scenario: check-id', with a space) and an empty list item.
This commit is contained in:
Felix Weinberger 2026-07-20 16:20:32 +01:00 • committed by GitHub
commit da56f6631c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 564 additions and 95 deletions
Download patch file Download diff file Expand all files Collapse all files

33
src/scenarios/client/auth/dpop.ts
View file

@ -18,6 +18,7 @@ import {
type DpopClientObservations
} from './helpers/dpopResourceAuth';
import { SpecReferences } from './spec-references';
import { collapseDuplicateChecks } from '../../../checks/collapse';
const PRM_PATH = '/.well-known/oauth-protected-resource/mcp';
@ -111,38 +112,6 @@ function newTokenReqObs(): DpopTokenRequestObservation {
};
}
/**
* Collapse duplicate non-INFO check IDs to a single entry, preferring the
* MOST-SEVERE occurrence (FAILURE > WARNING > SUCCESS > any other status, e.g.
* SKIPPED) so a real failure is never masked. Equal-severity ties keep the LAST
* occurrence (for the nonce round-trip that is the retry's diagnostic, which is
* status-equivalent to the first). Per-request INFO log entries are always kept.
*
* The RFC 9449 §8/§9 nonce round-trip re-POSTs /token (challenge → retry), so
* the shared token-flow conformance checks (`token-request`, `pkce-*`) are
* appended twice; this reports each once without hiding a failure recorded on
* either attempt. Exported so the behaviour is unit-tested directly.
*/
export function collapseDuplicateChecks(
checks: ConformanceCheck[]
): ConformanceCheck[] {
const severity = (s: CheckStatus): number =>
s === 'FAILURE' ? 3 : s === 'WARNING' ? 2 : s === 'SUCCESS' ? 1 : 0;
// Winning index per non-INFO id: highest severity, ties → last occurrence.
const winner = new Map<string, number>();
checks.forEach((c, i) => {
if (c.status === 'INFO') return;
const cur = winner.get(c.id);
if (
cur === undefined ||
severity(c.status) >= severity(checks[cur].status)
) {
winner.set(c.id, i);
}
});
return checks.filter((c, i) => c.status === 'INFO' || winner.get(c.id) === i);
}
export class DPoPClientScenario implements Scenario {
readonly name: string;
readonly source = {