Review round 4: key on using the artefact, and split the two trigger kinds
Copilot, both accepted. 1. "Anything that puts a built fixture on disk" matches every web test, not the case it was written for: `clients/web`'s `pretest` runs `test-servers:build` ahead of every unit run, so the fixture is on disk for the pure parser and storage tests the very next paragraph excludes. The trigger contradicted its own exclusion list. Re-keyed to depending on the artefact — the test imports, spawns, or points a config at it — with the pretest fact stated as its own ⚠️ so the distinction is legible rather than implied. 2. The closing section still said the fixture-consuming case needs "a real server over a real transport" and a protocol era, which is exactly false for the build-only consumer round 3 added. `smoke:tui` drives no transport, so config choice and protocol era do not apply to it — only the build and staleness half does. The condition is now stated once, with two named sub-cases that say which part of `test-servers` each needs: connecting takes the whole procedure, naming-the-artefact takes the build half. The closing section mirrors that split instead of collapsing them. Re-measured: 100% / 100% hand-off, 63/63 first-move at 100%, full suite at RUNS=5. Unchanged across all four rounds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
0894ab32ef
commit
e85a78f38c
1 changed files with 37 additions and 24 deletions
|
|
@ -16,19 +16,28 @@ test goes, how to run it, and how to clear the gate.
|
||||||
**If it does, load the `test-servers` skill now — that is step one, before
|
**If it does, load the `test-servers` skill now — that is step one, before
|
||||||
choosing a location or writing a line.**
|
choosing a location or writing a line.**
|
||||||
|
|
||||||
The condition is **"does this test connect to, or build and consume, an MCP
|
The condition is **"does this test depend on a fixture from `test-servers/`?"**
|
||||||
fixture?"** — not which tier it lands in, and not which directory it lands in.
|
— not which tier it lands in, and not which directory it lands in. There are two
|
||||||
It does whenever the task is: an integration test **that connects**; an
|
ways to depend on one, and they need different halves of that skill:
|
||||||
end-to-end test that connects; a smoke that drives a **connected** flow; a
|
|
||||||
coverage gap only reachable over a real connection; reproducing a reported bug
|
|
||||||
against a server; **or** anything that puts a built fixture on disk, even
|
|
||||||
without connecting — `smoke:tui` boots the TUI against a catalog whose stdio
|
|
||||||
command *is* `test-servers/build`, so the build and staleness half of that
|
|
||||||
procedure is exactly what it needs.
|
|
||||||
|
|
||||||
It does **not** when the test renders a component from fixture props, exercises
|
- **It connects.** An integration test that connects; an end-to-end test that
|
||||||
a pure function or a parser, or is a smoke that neither connects nor builds a
|
connects; a smoke that drives a connected flow; a coverage gap only reachable
|
||||||
fixture — `smoke:launcher` checks `--help`, and `smoke:web` /
|
over a real connection; reproducing a reported bug against a server. These
|
||||||
|
need the whole procedure — which showcase config, which protocol era, and the
|
||||||
|
staleness hazard.
|
||||||
|
- **It names or runs the built fixture without connecting.** `smoke:tui` boots
|
||||||
|
the TUI against a catalog whose stdio command *is* the built fixture, then
|
||||||
|
asserts it survives. No transport is driven and no protocol era applies, but
|
||||||
|
the **build and staleness** half lands on it in full.
|
||||||
|
|
||||||
|
⚠️ **"A build ran" is not the dependency — using the artefact is.**
|
||||||
|
`clients/web`'s `pretest` runs `test-servers:build` before *every* unit run, so
|
||||||
|
the fixture is on disk for tests that never reference it. What counts is whether
|
||||||
|
the test imports, spawns, or points a config at it.
|
||||||
|
|
||||||
|
So the condition does **not** hold when the test renders a component from
|
||||||
|
fixture props, exercises a pure function or a parser, or is a smoke that touches
|
||||||
|
no fixture — `smoke:launcher` checks `--help`, and `smoke:web` /
|
||||||
`smoke:web:browser` only assert the SPA is served and paints.
|
`smoke:web:browser` only assert the SPA is served and paints.
|
||||||
|
|
||||||
⚠️ **Neither the tier nor the folder decides this.** `src/test/integration/`
|
⚠️ **Neither the tier nor the folder decides this.** `src/test/integration/`
|
||||||
|
|
@ -204,14 +213,18 @@ shared helper that wraps one.
|
||||||
|
|
||||||
The tests that drive MCP behaviour over a transport — the connecting integration
|
The tests that drive MCP behaviour over a transport — the connecting integration
|
||||||
tests and the connected smokes — use a real server rather than a mock. **For
|
tests and the connected smokes — use a real server rather than a mock. **For
|
||||||
those, load the `test-servers` skill to pick, build and run the fixture** —
|
those, load the `test-servers` skill and use all of it**: which showcase config
|
||||||
which showcase config covers the feature, which protocol era to connect with,
|
covers the feature, which protocol era to connect with, how to add a combination
|
||||||
how to add a combination that does not exist yet, and why a fixture can keep
|
that does not exist yet, and why a fixture can keep serving stale code after an
|
||||||
serving stale code after an edit.
|
edit.
|
||||||
|
|
||||||
**Consuming a fixture is a trigger on its own, even without a connection.**
|
**A test that only *names* the built fixture needs that skill too, for a
|
||||||
`smoke:tui` only asserts the TUI boots and survives, but it builds
|
narrower reason.** `smoke:tui` boots the TUI against a catalog whose stdio
|
||||||
`test-servers/` and embeds the result in its catalog — so the staleness hazard
|
command is the build output and asserts it survives — it opens no transport, so
|
||||||
lands on it in full. A pure test that happens to live in the integration
|
config choice and protocol era do not apply to it, but **building the fixture
|
||||||
project, and a smoke that touches no fixture at all, need none of this (see the
|
and the staleness hazard do.** Load the skill and take that half.
|
||||||
tier list above).
|
|
||||||
|
A pure test that happens to live in the integration project, and a smoke that
|
||||||
|
references no fixture, need neither (see the tier list above) — and note that
|
||||||
|
`clients/web`'s `pretest` builds `test-servers/` before every unit run, so its
|
||||||
|
presence on disk says nothing about whether your test depends on it.
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue