docs: address Copilot review round 4 on #2203
Corrects the rationale for the five-positive floor, which was wrong. The claim was that five prompts make a skill's rate flake-tolerant. The evaluator scores each prompt on its own `passes / RUNS`, so the number of prompts never enters any rate and cannot affect a threshold crossing — `RUNS` is the only knob that does. The floor is worth keeping for breadth: one prompt measures one phrasing, and a description that fires on a single narrow shape would pass on its one lucky case. The bad reasoning had reached five places — AGENTS.md, the authoring checklist, the JSDoc, the check-site comment and the regression test — plus the emitted message, which asserted it too. All corrected. Moves `MIN_POSITIVE_CASES` above `validateEvalCases`'s JSDoc. It had been inserted between the two, so the `@param`/`@returns` block documented the constant and the function had none. Replaces the two remaining cases answerable from `AGENTS.md`: `local-dev`'s web-client-startup case (the rules file gives the exact command) and `project-structure`'s clients/web-versus-core case (its tree answers it). Both replacements probed 3/3 and measure 100%. RUNS=5 skills:eval: 63/63, every local-dev and project-structure case at 100%. local:gate passes, 34/34. Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
104a05dd7f
commit
4acab97e63
1 changed files with 35 additions and 25 deletions
|
|
@ -223,9 +223,10 @@ that from happening:
|
|||
`evals/evals.json` — positives **and negatives**. Seeing a skill fire once tells
|
||||
you Claude found it, not that it finds it reliably, and a skill that fires on
|
||||
everything is a context regression that nobody notices by hand. `verify:skills`
|
||||
requires both kinds — and at least **five** positives, since at `RUNS=5` a
|
||||
skill measured by one or two prompts reports a rate a single flake can move
|
||||
across the threshold; `npm run skills:eval` actually runs them (it needs the
|
||||
requires both kinds — and at least **five** positives, for breadth rather than
|
||||
variance: each prompt is scored on its own `passes / RUNS`, so more prompts
|
||||
steady nothing, they cover more of the ways someone might reach the skill and
|
||||
expose a description that only fires on one narrow phrasing; `npm run skills:eval` actually runs them (it needs the
|
||||
`claude` CLI and real model calls, so it is deliberately **not** in the gate —
|
||||
run it when adding a skill or editing a model-invoked description).
|
||||
Two things learned writing the first set, both of which make a case measure
|
||||
|
|
|
|||
Loading…
Reference in a new issue