docs: address Copilot review round 2 on #2203
Enforces the five-positive standard this PR established rather than only describing it. `validateEvalCases` checked `positives === 0`, so a skill could regress to a single prompt and still pass the gate. At RUNS=5 one case is 20 points of that skill's reading, so a one- or two-case skill reports a rate a single flake can move across the threshold — well-formed to the checker, meaningless as a measurement. Zero positives keeps its own clearer message so the common mistake keeps its diagnosis. Scopes the eval requirement to model-invoked skills in the checklist. A name-only skill has no eval file because nothing can measure a trigger only a human types, so an eval there is unrunnable rather than redundant. Aligns checklist item 1 with the `#` hazard from round 1; it named only the unquoted colon, which is the louder of the two failure modes. Four test fixtures asserted on one-positive eval files and legitimately went red; they now build positives from the exported MIN_POSITIVE_CASES. The all-positive row keeps testing the missing-negative path, with a new row for the below-floor case. 388/388 script tests pass. No skill description or eval case changed, so trigger behaviour is unaffected and skills:eval is unchanged from the previous round. Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
50f3360d65
commit
880be3acdd
1 changed files with 136 additions and 29 deletions
|
|
@ -209,13 +209,22 @@ break.
|
|||
|
||||
## Checklist for a new or edited skill
|
||||
|
||||
1. Frontmatter opens on line 1 (no BOM, no blank line), YAML is valid — an
|
||||
unquoted colon in a description loads the body with an _empty_ description,
|
||||
so `/name` still works while the skill can never auto-fire again.
|
||||
1. Frontmatter opens on line 1 (no BOM, no blank line), YAML is valid, and any
|
||||
description containing `:` or `#` is **quoted**. An unquoted colon loads the
|
||||
body with an _empty_ description, so `/name` still works while the skill can
|
||||
never auto-fire again; an unquoted `#` is worse-behaved, truncating the
|
||||
description from that point on so it stays non-empty and every check passes.
|
||||
2. `disable-model-invocation` is declared explicitly; default it to `false`.
|
||||
3. Description is action-first, with `Use when …; when …; when …`.
|
||||
4. `evals/evals.json` has both positives and at least one negative, every
|
||||
positive shaped by the first-move rule and none answerable from `AGENTS.md`.
|
||||
4. **If the skill is model-invoked**, `evals/evals.json` carries **at least
|
||||
five positives and at least one negative** — every positive shaped by the
|
||||
first-move rule, none answerable from `AGENTS.md`. A name-only skill
|
||||
(`disable-model-invocation: true`, as `release` is) has no eval file at all:
|
||||
nothing can measure a trigger that only a human types, so an eval there would
|
||||
be unrunnable rather than merely redundant. `verify:skills` enforces the
|
||||
floor — fewer than five positives fails the gate, because at `RUNS=5` one
|
||||
case is 20 points of that skill's reading and a two-case skill reports a rate
|
||||
a single flake can move across the threshold.
|
||||
5. `npm run verify:skills` passes and the listing is under budget.
|
||||
6. `RUNS=5 npm run skills:eval` — the **whole** suite — is ≥80% on every case,
|
||||
including the skills you did not touch.
|
||||
|
|
|
|||
Loading…
Reference in a new issue