docs: address Copilot review round 4 on #2203

Corrects the rationale for the five-positive floor, which was wrong. The
claim was that five prompts make a skill's rate flake-tolerant. The
evaluator scores each prompt on its own `passes / RUNS`, so the number of
prompts never enters any rate and cannot affect a threshold crossing —
`RUNS` is the only knob that does. The floor is worth keeping for
breadth: one prompt measures one phrasing, and a description that fires
on a single narrow shape would pass on its one lucky case.

The bad reasoning had reached five places — AGENTS.md, the authoring
checklist, the JSDoc, the check-site comment and the regression test —
plus the emitted message, which asserted it too. All corrected.

Moves `MIN_POSITIVE_CASES` above `validateEvalCases`'s JSDoc. It had been
inserted between the two, so the `@param`/`@returns` block documented the
constant and the function had none.

Replaces the two remaining cases answerable from `AGENTS.md`: `local-dev`'s
web-client-startup case (the rules file gives the exact command) and
`project-structure`'s clients/web-versus-core case (its tree answers it).
Both replacements probed 3/3 and measure 100%.

RUNS=5 skills:eval: 63/63, every local-dev and project-structure case at
100%. local:gate passes, 34/34.

Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-09-01 11:14:06 -04:00
commit 4acab97e63

View file

@ -243,9 +243,11 @@ break.
(`disable-model-invocation: true`, as `release` is) has no eval file at all:
nothing can measure a trigger that only a human types, so an eval there would
be unrunnable rather than merely redundant. `verify:skills` enforces the
floor — fewer than five positives fails the gate, because at `RUNS=5` one
case is 20 points of that skill's reading and a two-case skill reports a rate
a single flake can move across the threshold.
floor — fewer than five positives fails the gate. The reason is **breadth,
not variance**: each prompt is scored on its own `passes / RUNS`, so extra
prompts do not steady any rate (`RUNS` is the knob for that). Five prompts
cover five ways someone might arrive at the skill, which is what catches a
description that fires on one narrow phrasing and nothing else.
5. `npm run verify:skills` passes and the listing is under budget.
6. `RUNS=5 npm run skills:eval` — the **whole** suite — is ≥80% on every case,
including the skills you did not touch.