modelcontextprotocol/conformance
Watch
1
0
Fork
You've already forked conformance
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Allow expected-failures entries to name a single check (#406)

Browse source 
A baseline entry names a scenario, but a scenario is many checks:
server-stateless is 29. Excusing the one or two a spec change breaks
means baselining the whole scenario, which stops enforcing the rest.

An entry may now be '<scenario>:<check-id>', in which case every failing
check in that scenario is judged on its own. Bare scenario entries are
unchanged, and YAML parses '- a:b' as a plain string, so every existing
baseline keeps working untouched.

loadExpectedFailures now owns the entry grammar and returns parsed
BaselineEntry values, so evaluateBaseline only matches and cannot throw.
Rejecting a bad entry and interpreting a good one no longer live in
separate layers.

A check id matches all of its occurrences, since ids repeat within a run;
they collapse to one verdict, most-severe first. Absent and SKIPPED are
tolerated, matching how the runner already treats them as green.

Move collapseDuplicateChecks to src/checks/collapse.ts with its tests:
expected-failures.ts needs the reducer and cannot import dpop.ts's auth
server stack. No behavior change.

Reject two configs that previously coerced to an entry matching nothing:
a mapping ('- scenario: check-id', with a space) and an empty list item.
This commit is contained in:
Felix Weinberger 2026-07-20 16:20:32 +01:00 • committed by GitHub
commit da56f6631c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 564 additions and 95 deletions
Download patch file Download diff file Expand all files Collapse all files

85
src/scenarios/client/auth/dpop.test.ts
View file

@ -1,85 +0,0 @@
import { collapseDuplicateChecks } from './dpop';
import type { ConformanceCheck, CheckStatus } from '../../../types';
/** Minimal check factory for the dedupe unit tests. */
function chk(id: string, status: CheckStatus, tag?: string): ConformanceCheck {
return {
id,
name: id,
description: tag ?? id,
status,
timestamp: '2026-07-06T00:00:00.000Z',
specReferences: []
};
}
describe('collapseDuplicateChecks (DPoP nonce-posture shared-check dedupe)', () => {
it('collapses duplicate SUCCESS ids to a single entry', () => {
const out = collapseDuplicateChecks([
chk('token-request', 'SUCCESS'),
chk('pkce-code-verifier-sent', 'SUCCESS'),
chk('token-request', 'SUCCESS')
]);
expect(out.map((c) => c.id)).toEqual([
'pkce-code-verifier-sent',
'token-request'
]);
});
it('never masks a FAILURE — keeps the failing occurrence over a later SUCCESS', () => {
// The regression this guards: a client that fails a check on the challenged
// POST but passes on the nonce retry must still be reported FAILURE.
const out = collapseDuplicateChecks([
chk('pkce-verifier-matches-challenge', 'FAILURE', 'first attempt'),
chk('pkce-verifier-matches-challenge', 'SUCCESS', 'retry')
]);
expect(out).toHaveLength(1);
expect(out[0].status).toBe('FAILURE');
expect(out[0].description).toBe('first attempt');
});
it('keeps the failing occurrence regardless of order (SUCCESS then FAILURE)', () => {
const out = collapseDuplicateChecks([
chk('token-request', 'SUCCESS'),
chk('token-request', 'FAILURE', 'later failure')
]);
expect(out).toHaveLength(1);
expect(out[0].status).toBe('FAILURE');
expect(out[0].description).toBe('later failure');
});
it('prefers WARNING over SUCCESS but FAILURE over WARNING', () => {
expect(
collapseDuplicateChecks([chk('x', 'SUCCESS'), chk('x', 'WARNING')])[0]
.status
).toBe('WARNING');
expect(
collapseDuplicateChecks([chk('x', 'WARNING'), chk('x', 'FAILURE')])[0]
.status
).toBe('FAILURE');
});
it('keeps every INFO log entry, even duplicates', () => {
const out = collapseDuplicateChecks([
chk('incoming-request', 'INFO'),
chk('incoming-request', 'INFO'),
chk('incoming-request', 'INFO')
]);
expect(out).toHaveLength(3);
});
it('preserves order and leaves a duplicate-free list untouched', () => {
const input = [
chk('a', 'SUCCESS'),
chk('b', 'FAILURE'),
chk('incoming', 'INFO'),
chk('c', 'SUCCESS')
];
expect(collapseDuplicateChecks(input).map((c) => c.id)).toEqual([
'a',
'b',
'incoming',
'c'
]);
});
});

33
src/scenarios/client/auth/dpop.ts
View file

@ -18,6 +18,7 @@ import {
type DpopClientObservations
} from './helpers/dpopResourceAuth';
import { SpecReferences } from './spec-references';
import { collapseDuplicateChecks } from '../../../checks/collapse';
const PRM_PATH = '/.well-known/oauth-protected-resource/mcp';
@ -111,38 +112,6 @@ function newTokenReqObs(): DpopTokenRequestObservation {
};
}
/**
* Collapse duplicate non-INFO check IDs to a single entry, preferring the
* MOST-SEVERE occurrence (FAILURE > WARNING > SUCCESS > any other status, e.g.
* SKIPPED) so a real failure is never masked. Equal-severity ties keep the LAST
* occurrence (for the nonce round-trip that is the retry's diagnostic, which is
* status-equivalent to the first). Per-request INFO log entries are always kept.
*
* The RFC 9449 §8/§9 nonce round-trip re-POSTs /token (challenge → retry), so
* the shared token-flow conformance checks (`token-request`, `pkce-*`) are
* appended twice; this reports each once without hiding a failure recorded on
* either attempt. Exported so the behaviour is unit-tested directly.
*/
export function collapseDuplicateChecks(
checks: ConformanceCheck[]
): ConformanceCheck[] {
const severity = (s: CheckStatus): number =>
s === 'FAILURE' ? 3 : s === 'WARNING' ? 2 : s === 'SUCCESS' ? 1 : 0;
// Winning index per non-INFO id: highest severity, ties → last occurrence.
const winner = new Map<string, number>();
checks.forEach((c, i) => {
if (c.status === 'INFO') return;
const cur = winner.get(c.id);
if (
cur === undefined ||
severity(c.status) >= severity(checks[cur].status)
) {
winner.set(c.id, i);
}
});
return checks.filter((c, i) => c.status === 'INFO' || winner.get(c.id) === i);
}
export class DPoPClientScenario implements Scenario {
readonly name: string;
readonly source = {