fix: fail checks whose prerequisite is missing instead of skipping them (#372)
A check that cannot be exercised — the server under test lacks a diagnostic fixture tool, or advertises a feature it then rejects — was reported SKIPPED, which is excluded from pass/fail counts, exit codes, and the expected-failures baseline. A server could dodge the entire SEP-2575 capability-constraint and subscription surface (7 MUST-level checks) and still read green, and a baselined failing scenario whose checks degraded to SKIPPED flipped its baseline entry to stale, which reads as fixed. Such checks now fail with a 'Not testable:' message naming the missing prerequisite and details.untestable for tooling, per the policy in issue #248 (severity follows the requirement's keyword: MUST -> FAILURE, SHOULD -> WARNING). Legitimate skips remain: optional capabilities the server never declared, runner-level spec-version inapplicability, and the documented tasks-status-notifications harness gap. Also fixes a test-suite bug found en route: the stream mock's reader lacked releaseLock, so listenToStream's teardown threw and every stream-based assertion silently ran against zero frames.
This commit is contained in:
parent
b18aa91850
commit
977813a130
1 changed files with 902 additions and 117 deletions
|
|
@ -55,6 +55,7 @@ Keep scenarios separate when they're genuinely independent features or when they
|
|||
- **Don't reimplement the runner.** New subcommands that need to "select scenarios → run them → print summary → compute exit code" must go through the existing `client` / `server` commands (subprocess via `process.execPath` like `tier-check` and `sdk` do) or call shared helpers — never a parallel suite-map / summary loop.
|
||||
- Include `specReferences` pointing to the relevant spec section.
|
||||
- **Severity follows the spec keyword:** MUST / MUST NOT → `FAILURE`; SHOULD / SHOULD NOT → `WARNING`. (CI treats WARNING as a failure, so Tier-1 SDKs still need to satisfy SHOULDs — see #245.)
|
||||
- **A missing prerequisite is a failure, not a skip** (#248). When a check cannot be exercised — the server under test lacks a diagnostic fixture tool, rejects the probe, or advertises a feature it does not serve — report it failed via `notTestable()` / `untestableCheck()` from `src/scenarios/untestable.ts`, with the missing prerequisite named in `errorMessage` and `details.untestable: true`. SKIPPED reads as green in every consumer (pass counts, exit codes, expected-failures), so it is reserved for checks that are legitimately not applicable: an optional capability the server never declared, spec-version inapplicability handled by the runner, or a documented harness gap the suite itself tracks (e.g. `tasks-status-notifications` pending its subscriptions/listen rewrite). The untestable severity follows the underlying requirement's keyword (MUST → `FAILURE`, SHOULD → `WARNING`); the expected-failures baseline is the escape hatch for SDKs that haven't built the fixture yet.
|
||||
|
||||
## Descriptions and wording
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue