Reshape the testing → test-servers pointer into an imperative step
The hand-off measured 33% / 33% at RUNS=3 (#2204's two committed chain cases). `testing` opened with a one-sentence ⚠️ *classifying* which work belongs to `test-servers` — a fact, which the model skims — and said it nowhere else except a bare cross-reference in the last section. Rewrite it as an action with a trigger ("load the `test-servers` skill now — that is step one"), enumerate the situations that qualify, and repeat the pointer at the two later points where the run actually decides it is writing an integration test: the `integration` project bullet under "Where the test file goes", and the tier list. The top of a body is read before the model knows it needs the second skill, so a pointer that lives only there has already been scrolled past by the time it matters. Measured at RUNS=5 on the full suite: the two hand-off cases go to 100% / 80% (100% / 100% on a focused `-- test-servers` run), and all 63 first-move cases stay at 100% — no displacement, as expected, since neither description changed. Also update the two places that recorded the old number as the standing state: the worked example in `docs/skill-authoring.md`, and the `CHAIN_THRESHOLD` rationale in `scripts/skill-eval.mjs`, which cites the 33% measurement as the reason the bar is 0.5 rather than 0.8. The bar is unchanged — a reshaped pointer raises the ceiling those two cases reach, not the floor a new hand-off case should be judged against. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
ed2a188069
commit
7d0ae9e9ec
3 changed files with 51 additions and 20 deletions
|
|
@ -11,11 +11,18 @@ statements, functions, and branches.** That rule and the React/Mantine
|
||||||
conventions live in [`AGENTS.md`](../../../AGENTS.md); this skill is where a
|
conventions live in [`AGENTS.md`](../../../AGENTS.md); this skill is where a
|
||||||
test goes, how to run it, and how to clear the gate.
|
test goes, how to run it, and how to clear the gate.
|
||||||
|
|
||||||
⚠️ **Anything that needs a real server to run against — an integration test, a
|
## Before you write it: does the test need a real server?
|
||||||
smoke, reproducing a bug by hand — is `/test-servers`, and you have to load it.**
|
|
||||||
Integration and smoke tests here drive a real server over a real transport
|
**If it does, load the `test-servers` skill now — that is step one, before
|
||||||
rather than a mock, so picking, building and connecting to a fixture is a
|
choosing a location or writing a line.** It does, whenever the task is: an
|
||||||
procedure of its own that this skill does not carry.
|
integration test; an end-to-end test; a smoke; a coverage gap that has to be
|
||||||
|
exercised over a transport; or reproducing a reported bug by hand.
|
||||||
|
|
||||||
|
Every one of those drives a **real server over a real transport, never a mock**,
|
||||||
|
and picking the fixture, building it, and connecting with the right protocol era
|
||||||
|
is a procedure this skill does not carry. Writing one without `test-servers`
|
||||||
|
means hand-rolling a fixture that already exists, or mocking the thing the tier
|
||||||
|
exists to avoid mocking.
|
||||||
|
|
||||||
## Where the test file goes
|
## Where the test file goes
|
||||||
|
|
||||||
|
|
@ -38,7 +45,9 @@ web-owned test living under `src/test/` instead is a bug.
|
||||||
`core/` source layout (`mcp/`, `mcp/node/`, `mcp/remote/`, `auth/`,
|
`core/` source layout (`mcp/`, `mcp/node/`, `mcp/remote/`, `auth/`,
|
||||||
`auth/node/`, `storage/`). **Placement is the manifest** — any file under that
|
`auth/node/`, `storage/`). **Placement is the manifest** — any file under that
|
||||||
folder is picked up by the integration project (node env, 30s timeouts) via a
|
folder is picked up by the integration project (node env, 30s timeouts) via a
|
||||||
folder glob; there is no enumeration to keep in sync.
|
folder glob; there is no enumeration to keep in sync. ⚠️ These run against a
|
||||||
|
real server, so **load the `test-servers` skill before writing one** — the
|
||||||
|
fixture is half the test.
|
||||||
3. **Shared test infrastructure** — `renderWithMantine.tsx`, `setup.ts`,
|
3. **Shared test infrastructure** — `renderWithMantine.tsx`, `setup.ts`,
|
||||||
`fixtures/`, `scrollAreaStoryAssertions.ts`.
|
`fixtures/`, `scrollAreaStoryAssertions.ts`.
|
||||||
|
|
||||||
|
|
@ -82,6 +91,10 @@ spawns the built binary) → smokes through the built launcher (`npm run smoke`)
|
||||||
Storybook play functions (`test:storybook`) → the published-tarball check
|
Storybook play functions (`test:storybook`) → the published-tarball check
|
||||||
(`npm run pack:verify`, local/release only — needs network).
|
(`npm run pack:verify`, local/release only — needs network).
|
||||||
|
|
||||||
|
Everything from **web integration** rightwards needs a fixture from
|
||||||
|
`test-servers/` — load the `test-servers` skill as soon as a task puts you at
|
||||||
|
that tier or deeper.
|
||||||
|
|
||||||
`validate` runs the per-client `test` scripts — so web **unit** plus cli's
|
`validate` runs the per-client `test` scripts — so web **unit** plus cli's
|
||||||
out-of-process `e2e.test.ts`, but **not** web's integration project, which runs
|
out-of-process `e2e.test.ts`, but **not** web's integration project, which runs
|
||||||
inside the `coverage` gate. CI therefore has no separate `test:integration` step.
|
inside the `coverage` gate. CI therefore has no separate `test:integration` step.
|
||||||
|
|
@ -159,5 +172,8 @@ shared helper that wraps one.
|
||||||
|
|
||||||
## Test servers, not mocks
|
## Test servers, not mocks
|
||||||
|
|
||||||
Integration and smoke tests drive a real server over a real transport. See
|
Integration and smoke tests drive a real server over a real transport, never a
|
||||||
`/test-servers` for picking and building one.
|
mock. **Load the `test-servers` skill to pick, build and run the fixture** —
|
||||||
|
which showcase config covers the feature, which protocol era to connect with,
|
||||||
|
how to add a combination that does not exist yet, and why a fixture can keep
|
||||||
|
serving stale code after an edit.
|
||||||
|
|
|
||||||
|
|
@ -289,13 +289,25 @@ pointed at the second, and the column would stop carrying signal. Read a
|
||||||
hand-off number as a description-strength measurement, not a verdict — and read
|
hand-off number as a description-strength measurement, not a verdict — and read
|
||||||
it at `RUNS=5`, since at `RUNS=3` one sample is worth 33 points.
|
it at `RUNS=5`, since at `RUNS=3` one sample is worth 33 points.
|
||||||
|
|
||||||
**The committed cases have measured 33% / 33% on one `RUNS=3` run and 100% /
|
**A red hand-off case is a finding about the pointer, not a build break — and
|
||||||
33% on another, and at least one of them being red is the intended state rather
|
the fix is to reshape the pointer, never to lower the bar.** The committed
|
||||||
than an oversight.** `skills:eval` is not a gate (see below), and the number is
|
`testing` -> `test-servers` cases are the worked example. They measured
|
||||||
the finding: `testing` points at `test-servers` in its first paragraph and the
|
33% / 33% at `RUNS=3` when `testing` opened with a one-sentence ⚠️ *classifying*
|
||||||
model follows that pointer *sometimes*. Strengthening it is its own change
|
which work belongs to `test-servers`; rewriting that into an imperative first
|
||||||
against its own issue (#2247); lowering the bar to turn the column green would
|
step ("load the `test-servers` skill now — that is step one"), and repeating it
|
||||||
throw away the only signal this feature adds.
|
at the two later points where the model actually decides it is writing an
|
||||||
|
integration test, took them to **100% / 80% at `RUNS=5`** with the prompts
|
||||||
|
unchanged (#2247) — 100% / 100% on a focused `-- test-servers` run of the same
|
||||||
|
build, which is the size of the run-to-run noise still present at `RUNS=5`.
|
||||||
|
Nothing else moved: the two descriptions were not touched, and the same suite
|
||||||
|
scored **63/63** first-move cases at 100%.
|
||||||
|
|
||||||
|
The transferable part is that **a pointer is followed when it reads as an action
|
||||||
|
with a trigger, and skimmed when it reads as a fact.** #2202 found the same
|
||||||
|
lever on a *description*'s shape; this is it applied to a body. The corollary is
|
||||||
|
where to put one: the top of a body is read before the model knows it needs the
|
||||||
|
second skill, so a pointer that lives only there is a pointer it has already
|
||||||
|
scrolled past by the time it matters.
|
||||||
|
|
||||||
⚠️ **Do not read a rise between two `RUNS=3` runs as an improvement.** One
|
⚠️ **Do not read a rise between two `RUNS=3` runs as an improvement.** One
|
||||||
sample is 33 points there, and the two runs above straddle a 67-point swing on
|
sample is 33 points there, and the two runs above straddle a 67-point swing on
|
||||||
|
|
@ -372,7 +384,7 @@ The summary is two lines, never one:
|
||||||
|
|
||||||
```
|
```
|
||||||
7/7 first-move cases at or above 80%.
|
7/7 first-move cases at or above 80%.
|
||||||
1/2 hand-off cases above 50%.
|
2/2 hand-off cases above 50%.
|
||||||
```
|
```
|
||||||
|
|
||||||
Narrowing the run never narrows what a **negative** case is scored against — a
|
Narrowing the run never narrows what a **negative** case is scored against — a
|
||||||
|
|
|
||||||
|
|
@ -57,10 +57,13 @@ const THRESHOLD = Number(process.env.THRESHOLD ?? 0.8);
|
||||||
// acceptable is a separate judgement rather than one inherited from a number
|
// acceptable is a separate judgement rather than one inherited from a number
|
||||||
// tuned for the other measurement. 0.5 is the weakest claim worth asserting —
|
// tuned for the other measurement. 0.5 is the weakest claim worth asserting —
|
||||||
// the pointer is taken more often than not. It is deliberately not 0.8: the
|
// the pointer is taken more often than not. It is deliberately not 0.8: the
|
||||||
// committed `testing` -> `test-servers` cases measure 33% (RUNS=3) against a
|
// committed `testing` -> `test-servers` cases measured 33% (RUNS=3) against a
|
||||||
// pointer that is live and stated in the first paragraph of `testing`'s body,
|
// pointer that was live and stated in the first paragraph of `testing`'s body,
|
||||||
// so an 0.8 bar would mark every hand-off red regardless of how strongly the
|
// so an 0.8 bar would mark a hand-off red regardless of how strongly the first
|
||||||
// first skill points at the second, and the column would stop carrying signal.
|
// skill points at the second, and the column would stop carrying signal.
|
||||||
|
// (#2247 later reshaped that pointer into an imperative step and took the same
|
||||||
|
// two cases to 100% at RUNS=5 — which raises the ceiling those cases reach, not
|
||||||
|
// the floor a *new* hand-off case should be judged against.)
|
||||||
//
|
//
|
||||||
// It is compared STRICTLY, unlike the first-move threshold. "More often than
|
// It is compared STRICTLY, unlike the first-move threshold. "More often than
|
||||||
// not" is `> 0.5`, and an inclusive compare passes exactly half the samples
|
// not" is `> 0.5`, and an inclusive compare passes exactly half the samples
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue