fix: fail checks whose prerequisite is missing instead of skipping them (#372)

A check that cannot be exercised — the server under test lacks a
diagnostic fixture tool, or advertises a feature it then rejects — was
reported SKIPPED, which is excluded from pass/fail counts, exit codes,
and the expected-failures baseline. A server could dodge the entire
SEP-2575 capability-constraint and subscription surface (7 MUST-level
checks) and still read green, and a baselined failing scenario whose
checks degraded to SKIPPED flipped its baseline entry to stale, which
reads as fixed.

Such checks now fail with a 'Not testable:' message naming the missing
prerequisite and details.untestable for tooling, per the policy in
issue #248 (severity follows the requirement's keyword: MUST -> FAILURE,
SHOULD -> WARNING). Legitimate skips remain: optional capabilities the
server never declared, runner-level spec-version inapplicability, and
the documented tasks-status-notifications harness gap.

Also fixes a test-suite bug found en route: the stream mock's reader
lacked releaseLock, so listenToStream's teardown threw and every
stream-based assertion silently ran against zero frames.
This commit is contained in:
Felix Weinberger 2026-06-30 11:42:01 +01:00 • committed by GitHub
commit 977813a130
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

View file

@ -55,6 +55,7 @@ Keep scenarios separate when they're genuinely independent features or when they
- **Don't reimplement the runner.** New subcommands that need to "select scenarios → run them → print summary → compute exit code" must go through the existing `client` / `server` commands (subprocess via `process.execPath` like `tier-check` and `sdk` do) or call shared helpers — never a parallel suite-map / summary loop.
- Include `specReferences` pointing to the relevant spec section.
- **Severity follows the spec keyword:** MUST / MUST NOT → `FAILURE`; SHOULD / SHOULD NOT → `WARNING`. (CI treats WARNING as a failure, so Tier-1 SDKs still need to satisfy SHOULDs — see #245.)
- **A missing prerequisite is a failure, not a skip** (#248). When a check cannot be exercised — the server under test lacks a diagnostic fixture tool, rejects the probe, or advertises a feature it does not serve — report it failed via `notTestable()` / `untestableCheck()` from `src/scenarios/untestable.ts`, with the missing prerequisite named in `errorMessage` and `details.untestable: true`. SKIPPED reads as green in every consumer (pass counts, exit codes, expected-failures), so it is reserved for checks that are legitimately not applicable: an optional capability the server never declared, spec-version inapplicability handled by the runner, or a documented harness gap the suite itself tracks (e.g. `tasks-status-notifications` pending its subscriptions/listen rewrite). The untestable severity follows the underlying requirement's keyword (MUST → `FAILURE`, SHOULD → `WARNING`); the expected-failures baseline is the escape hatch for SDKs that haven't built the fixture yet.
## Descriptions and wording