docs: address Copilot review round 2 on #2203

Enforces the five-positive standard this PR established rather than only
describing it. `validateEvalCases` checked `positives === 0`, so a skill
could regress to a single prompt and still pass the gate. At RUNS=5 one
case is 20 points of that skill's reading, so a one- or two-case skill
reports a rate a single flake can move across the threshold — well-formed
to the checker, meaningless as a measurement. Zero positives keeps its
own clearer message so the common mistake keeps its diagnosis.

Scopes the eval requirement to model-invoked skills in the checklist. A
name-only skill has no eval file because nothing can measure a trigger
only a human types, so an eval there is unrunnable rather than redundant.

Aligns checklist item 1 with the `#` hazard from round 1; it named only
the unquoted colon, which is the louder of the two failure modes.

Four test fixtures asserted on one-positive eval files and legitimately
went red; they now build positives from the exported MIN_POSITIVE_CASES.
The all-positive row keeps testing the missing-negative path, with a new
row for the below-floor case. 388/388 script tests pass.

No skill description or eval case changed, so trigger behaviour is
unaffected and skills:eval is unchanged from the previous round.

Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-09-01 10:32:41 -04:00
commit 880be3acdd

View file

@ -209,13 +209,22 @@ break.
## Checklist for a new or edited skill
1. Frontmatter opens on line 1 (no BOM, no blank line), YAML is valid — an
unquoted colon in a description loads the body with an _empty_ description,
so `/name` still works while the skill can never auto-fire again.
1. Frontmatter opens on line 1 (no BOM, no blank line), YAML is valid, and any
description containing `:` or `#` is **quoted**. An unquoted colon loads the
body with an _empty_ description, so `/name` still works while the skill can
never auto-fire again; an unquoted `#` is worse-behaved, truncating the
description from that point on so it stays non-empty and every check passes.
2. `disable-model-invocation` is declared explicitly; default it to `false`.
3. Description is action-first, with `Use when …; when …; when …`.
4. `evals/evals.json` has both positives and at least one negative, every
positive shaped by the first-move rule and none answerable from `AGENTS.md`.
4. **If the skill is model-invoked**, `evals/evals.json` carries **at least
five positives and at least one negative** — every positive shaped by the
first-move rule, none answerable from `AGENTS.md`. A name-only skill
(`disable-model-invocation: true`, as `release` is) has no eval file at all:
nothing can measure a trigger that only a human types, so an eval there would
be unrunnable rather than merely redundant. `verify:skills` enforces the
floor — fewer than five positives fails the gate, because at `RUNS=5` one
case is 20 points of that skill's reading and a two-case skill reports a rate
a single flake can move across the threshold.
5. `npm run verify:skills` passes and the listing is under budget.
6. `RUNS=5 npm run skills:eval` — the **whole** suite — is ≥80% on every case,
including the skills you did not touch.