Reshape the testing → test-servers pointer into an imperative step

The hand-off measured 33% / 33% at RUNS=3 (#2204's two committed chain
cases). `testing` opened with a one-sentence ⚠️ *classifying* which work
belongs to `test-servers` — a fact, which the model skims — and said it
nowhere else except a bare cross-reference in the last section.

Rewrite it as an action with a trigger ("load the `test-servers` skill
now — that is step one"), enumerate the situations that qualify, and
repeat the pointer at the two later points where the run actually
decides it is writing an integration test: the `integration` project
bullet under "Where the test file goes", and the tier list. The top of a
body is read before the model knows it needs the second skill, so a
pointer that lives only there has already been scrolled past by the time
it matters.

Measured at RUNS=5 on the full suite: the two hand-off cases go to
100% / 80% (100% / 100% on a focused `-- test-servers` run), and all
63 first-move cases stay at 100% — no displacement, as expected, since
neither description changed.

Also update the two places that recorded the old number as the standing
state: the worked example in `docs/skill-authoring.md`, and the
`CHAIN_THRESHOLD` rationale in `scripts/skill-eval.mjs`, which cites the
33% measurement as the reason the bar is 0.5 rather than 0.8. The bar is
unchanged — a reshaped pointer raises the ceiling those two cases reach,
not the floor a new hand-off case should be judged against.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-09-05 14:59:05 -04:00
commit 7d0ae9e9ec
3 changed files with 51 additions and 20 deletions

View file

@ -11,11 +11,18 @@ statements, functions, and branches.** That rule and the React/Mantine
conventions live in [`AGENTS.md`](../../../AGENTS.md); this skill is where a conventions live in [`AGENTS.md`](../../../AGENTS.md); this skill is where a
test goes, how to run it, and how to clear the gate. test goes, how to run it, and how to clear the gate.
⚠️ **Anything that needs a real server to run against — an integration test, a ## Before you write it: does the test need a real server?
smoke, reproducing a bug by hand — is `/test-servers`, and you have to load it.**
Integration and smoke tests here drive a real server over a real transport **If it does, load the `test-servers` skill now — that is step one, before
rather than a mock, so picking, building and connecting to a fixture is a choosing a location or writing a line.** It does, whenever the task is: an
procedure of its own that this skill does not carry. integration test; an end-to-end test; a smoke; a coverage gap that has to be
exercised over a transport; or reproducing a reported bug by hand.
Every one of those drives a **real server over a real transport, never a mock**,
and picking the fixture, building it, and connecting with the right protocol era
is a procedure this skill does not carry. Writing one without `test-servers`
means hand-rolling a fixture that already exists, or mocking the thing the tier
exists to avoid mocking.
## Where the test file goes ## Where the test file goes
@ -38,7 +45,9 @@ web-owned test living under `src/test/` instead is a bug.
`core/` source layout (`mcp/`, `mcp/node/`, `mcp/remote/`, `auth/`, `core/` source layout (`mcp/`, `mcp/node/`, `mcp/remote/`, `auth/`,
`auth/node/`, `storage/`). **Placement is the manifest** — any file under that `auth/node/`, `storage/`). **Placement is the manifest** — any file under that
folder is picked up by the integration project (node env, 30s timeouts) via a folder is picked up by the integration project (node env, 30s timeouts) via a
folder glob; there is no enumeration to keep in sync. folder glob; there is no enumeration to keep in sync. ⚠️ These run against a
real server, so **load the `test-servers` skill before writing one** — the
fixture is half the test.
3. **Shared test infrastructure** — `renderWithMantine.tsx`, `setup.ts`, 3. **Shared test infrastructure** — `renderWithMantine.tsx`, `setup.ts`,
`fixtures/`, `scrollAreaStoryAssertions.ts`. `fixtures/`, `scrollAreaStoryAssertions.ts`.
@ -82,6 +91,10 @@ spawns the built binary) → smokes through the built launcher (`npm run smoke`)
Storybook play functions (`test:storybook`) → the published-tarball check Storybook play functions (`test:storybook`) → the published-tarball check
(`npm run pack:verify`, local/release only — needs network). (`npm run pack:verify`, local/release only — needs network).
Everything from **web integration** rightwards needs a fixture from
`test-servers/` — load the `test-servers` skill as soon as a task puts you at
that tier or deeper.
`validate` runs the per-client `test` scripts — so web **unit** plus cli's `validate` runs the per-client `test` scripts — so web **unit** plus cli's
out-of-process `e2e.test.ts`, but **not** web's integration project, which runs out-of-process `e2e.test.ts`, but **not** web's integration project, which runs
inside the `coverage` gate. CI therefore has no separate `test:integration` step. inside the `coverage` gate. CI therefore has no separate `test:integration` step.
@ -159,5 +172,8 @@ shared helper that wraps one.
## Test servers, not mocks ## Test servers, not mocks
Integration and smoke tests drive a real server over a real transport. See Integration and smoke tests drive a real server over a real transport, never a
`/test-servers` for picking and building one. mock. **Load the `test-servers` skill to pick, build and run the fixture** —
which showcase config covers the feature, which protocol era to connect with,
how to add a combination that does not exist yet, and why a fixture can keep
serving stale code after an edit.

View file

@ -289,13 +289,25 @@ pointed at the second, and the column would stop carrying signal. Read a
hand-off number as a description-strength measurement, not a verdict — and read hand-off number as a description-strength measurement, not a verdict — and read
it at `RUNS=5`, since at `RUNS=3` one sample is worth 33 points. it at `RUNS=5`, since at `RUNS=3` one sample is worth 33 points.
**The committed cases have measured 33% / 33% on one `RUNS=3` run and 100% / **A red hand-off case is a finding about the pointer, not a build break — and
33% on another, and at least one of them being red is the intended state rather the fix is to reshape the pointer, never to lower the bar.** The committed
than an oversight.** `skills:eval` is not a gate (see below), and the number is `testing` -> `test-servers` cases are the worked example. They measured
the finding: `testing` points at `test-servers` in its first paragraph and the 33% / 33% at `RUNS=3` when `testing` opened with a one-sentence ⚠️ *classifying*
model follows that pointer *sometimes*. Strengthening it is its own change which work belongs to `test-servers`; rewriting that into an imperative first
against its own issue (#2247); lowering the bar to turn the column green would step ("load the `test-servers` skill now — that is step one"), and repeating it
throw away the only signal this feature adds. at the two later points where the model actually decides it is writing an
integration test, took them to **100% / 80% at `RUNS=5`** with the prompts
unchanged (#2247) — 100% / 100% on a focused `-- test-servers` run of the same
build, which is the size of the run-to-run noise still present at `RUNS=5`.
Nothing else moved: the two descriptions were not touched, and the same suite
scored **63/63** first-move cases at 100%.
The transferable part is that **a pointer is followed when it reads as an action
with a trigger, and skimmed when it reads as a fact.** #2202 found the same
lever on a *description*'s shape; this is it applied to a body. The corollary is
where to put one: the top of a body is read before the model knows it needs the
second skill, so a pointer that lives only there is a pointer it has already
scrolled past by the time it matters.
⚠️ **Do not read a rise between two `RUNS=3` runs as an improvement.** One ⚠️ **Do not read a rise between two `RUNS=3` runs as an improvement.** One
sample is 33 points there, and the two runs above straddle a 67-point swing on sample is 33 points there, and the two runs above straddle a 67-point swing on
@ -372,7 +384,7 @@ The summary is two lines, never one:
``` ```
7/7 first-move cases at or above 80%. 7/7 first-move cases at or above 80%.
1/2 hand-off cases above 50%. 2/2 hand-off cases above 50%.
``` ```
Narrowing the run never narrows what a **negative** case is scored against — a Narrowing the run never narrows what a **negative** case is scored against — a

View file

@ -57,10 +57,13 @@ const THRESHOLD = Number(process.env.THRESHOLD ?? 0.8);
// acceptable is a separate judgement rather than one inherited from a number // acceptable is a separate judgement rather than one inherited from a number
// tuned for the other measurement. 0.5 is the weakest claim worth asserting — // tuned for the other measurement. 0.5 is the weakest claim worth asserting —
// the pointer is taken more often than not. It is deliberately not 0.8: the // the pointer is taken more often than not. It is deliberately not 0.8: the
// committed `testing` -> `test-servers` cases measure 33% (RUNS=3) against a // committed `testing` -> `test-servers` cases measured 33% (RUNS=3) against a
// pointer that is live and stated in the first paragraph of `testing`'s body, // pointer that was live and stated in the first paragraph of `testing`'s body,
// so an 0.8 bar would mark every hand-off red regardless of how strongly the // so an 0.8 bar would mark a hand-off red regardless of how strongly the first
// first skill points at the second, and the column would stop carrying signal. // skill points at the second, and the column would stop carrying signal.
// (#2247 later reshaped that pointer into an imperative step and took the same
// two cases to 100% at RUNS=5 — which raises the ceiling those cases reach, not
// the floor a *new* hand-off case should be judged against.)
// //
// It is compared STRICTLY, unlike the first-move threshold. "More often than // It is compared STRICTLY, unlike the first-move threshold. "More often than
// not" is `> 0.5`, and an inclusive compare passes exactly half the samples // not" is `> 0.5`, and an inclusive compare passes exactly half the samples