docs: address Copilot review round 2 on #2203

Enforces the five-positive standard this PR established rather than only
describing it. `validateEvalCases` checked `positives === 0`, so a skill
could regress to a single prompt and still pass the gate. At RUNS=5 one
case is 20 points of that skill's reading, so a one- or two-case skill
reports a rate a single flake can move across the threshold — well-formed
to the checker, meaningless as a measurement. Zero positives keeps its
own clearer message so the common mistake keeps its diagnosis.

Scopes the eval requirement to model-invoked skills in the checklist. A
name-only skill has no eval file because nothing can measure a trigger
only a human types, so an eval there is unrunnable rather than redundant.

Aligns checklist item 1 with the `#` hazard from round 1; it named only
the unquoted colon, which is the louder of the two failure modes.

Four test fixtures asserted on one-positive eval files and legitimately
went red; they now build positives from the exported MIN_POSITIVE_CASES.
The all-positive row keeps testing the missing-negative path, with a new
row for the below-floor case. 388/388 script tests pass.

No skill description or eval case changed, so trigger behaviour is
unaffected and skills:eval is unchanged from the previous round.

Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-09-01 10:32:41 -04:00
commit 880be3acdd

View file

@ -222,7 +222,9 @@ that from happening:
`evals/evals.json` — positives **and negatives**. Seeing a skill fire once tells
you Claude found it, not that it finds it reliably, and a skill that fires on
everything is a context regression that nobody notices by hand. `verify:skills`
requires both kinds; `npm run skills:eval` actually runs them (it needs the
requires both kinds — and at least **five** positives, since at `RUNS=5` a
skill measured by one or two prompts reports a rate a single flake can move
across the threshold; `npm run skills:eval` actually runs them (it needs the
`claude` CLI and real model calls, so it is deliberately **not** in the gate —
run it when adding a skill or editing a model-invoked description).
Two things learned writing the first set, both of which make a case measure