docs: address Copilot review round 2 on #2203
Enforces the five-positive standard this PR established rather than only describing it. `validateEvalCases` checked `positives === 0`, so a skill could regress to a single prompt and still pass the gate. At RUNS=5 one case is 20 points of that skill's reading, so a one- or two-case skill reports a rate a single flake can move across the threshold — well-formed to the checker, meaningless as a measurement. Zero positives keeps its own clearer message so the common mistake keeps its diagnosis. Scopes the eval requirement to model-invoked skills in the checklist. A name-only skill has no eval file because nothing can measure a trigger only a human types, so an eval there is unrunnable rather than redundant. Aligns checklist item 1 with the `#` hazard from round 1; it named only the unquoted colon, which is the louder of the two failure modes. Four test fixtures asserted on one-positive eval files and legitimately went red; they now build positives from the exported MIN_POSITIVE_CASES. The all-positive row keeps testing the missing-negative path, with a new row for the below-floor case. 388/388 script tests pass. No skill description or eval case changed, so trigger behaviour is unaffected and skills:eval is unchanged from the previous round. Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
50f3360d65
commit
880be3acdd
1 changed files with 136 additions and 29 deletions
|
|
@ -222,7 +222,9 @@ that from happening:
|
|||
`evals/evals.json` — positives **and negatives**. Seeing a skill fire once tells
|
||||
you Claude found it, not that it finds it reliably, and a skill that fires on
|
||||
everything is a context regression that nobody notices by hand. `verify:skills`
|
||||
requires both kinds; `npm run skills:eval` actually runs them (it needs the
|
||||
requires both kinds — and at least **five** positives, since at `RUNS=5` a
|
||||
skill measured by one or two prompts reports a rate a single flake can move
|
||||
across the threshold; `npm run skills:eval` actually runs them (it needs the
|
||||
`claude` CLI and real model calls, so it is deliberately **not** in the gate —
|
||||
run it when adding a skill or editing a model-invoked description).
|
||||
Two things learned writing the first set, both of which make a case measure
|
||||
|
|
|
|||
Loading…
Reference in a new issue