Allow expected-failures entries to name a single check (#406)
A baseline entry names a scenario, but a scenario is many checks:
server-stateless is 29. Excusing the one or two a spec change breaks
means baselining the whole scenario, which stops enforcing the rest.
An entry may now be '<scenario>:<check-id>', in which case every failing
check in that scenario is judged on its own. Bare scenario entries are
unchanged, and YAML parses '- a:b' as a plain string, so every existing
baseline keeps working untouched.
loadExpectedFailures now owns the entry grammar and returns parsed
BaselineEntry values, so evaluateBaseline only matches and cannot throw.
Rejecting a bad entry and interpreting a good one no longer live in
separate layers.
A check id matches all of its occurrences, since ids repeat within a run;
they collapse to one verdict, most-severe first. Absent and SKIPPED are
tolerated, matching how the runner already treats them as green.
Move collapseDuplicateChecks to src/checks/collapse.ts with its tests:
expected-failures.ts needs the reducer and cannot import dpop.ts's auth
server stack. No behavior change.
Reject two configs that previously coerced to an entry matching nothing:
a mapping ('- scenario: check-id', with a space) and an empty list item.
This commit is contained in:
parent
d1c0b95917
commit
da56f6631c
1 changed files with 564 additions and 95 deletions
35
src/checks/collapse.ts
Normal file
35
src/checks/collapse.ts
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
import type { CheckStatus, ConformanceCheck } from '../types';
|
||||
|
||||
/**
|
||||
* Collapse duplicate non-INFO check IDs to a single entry, preferring the
|
||||
* MOST-SEVERE occurrence (FAILURE > WARNING > SUCCESS > any other status, e.g.
|
||||
* SKIPPED) so a real failure is never masked. Equal-severity ties keep the LAST
|
||||
* occurrence. Per-request INFO log entries are always kept.
|
||||
*
|
||||
* A check ID is not unique within a run: scenarios re-emit a shared ID when a
|
||||
* flow repeats. The RFC 9449 §8/§9 nonce round-trip re-POSTs /token (challenge
|
||||
* → retry), so the shared token-flow checks (`token-request`, `pkce-*`) are
|
||||
* appended twice; `sep-2575-http-server-meta-invalid-400` is emitted once per
|
||||
* iteration of the `_meta` test-case loop. Collapsing reports each ID once
|
||||
* without hiding a failure recorded on any occurrence, which is what lets an
|
||||
* expected-failures baseline address a check by ID.
|
||||
*/
|
||||
export function collapseDuplicateChecks(
|
||||
checks: ConformanceCheck[]
|
||||
): ConformanceCheck[] {
|
||||
const severity = (s: CheckStatus): number =>
|
||||
s === 'FAILURE' ? 3 : s === 'WARNING' ? 2 : s === 'SUCCESS' ? 1 : 0;
|
||||
// Winning index per non-INFO id: highest severity, ties → last occurrence.
|
||||
const winner = new Map<string, number>();
|
||||
checks.forEach((c, i) => {
|
||||
if (c.status === 'INFO') return;
|
||||
const cur = winner.get(c.id);
|
||||
if (
|
||||
cur === undefined ||
|
||||
severity(c.status) >= severity(checks[cur].status)
|
||||
) {
|
||||
winner.set(c.id, i);
|
||||
}
|
||||
});
|
||||
return checks.filter((c, i) => c.status === 'INFO' || winner.get(c.id) === i);
|
||||
}
|
||||
Loading…
Reference in a new issue