docs: address Copilot review round 2 on #2203

Enforces the five-positive standard this PR established rather than only
describing it. `validateEvalCases` checked `positives === 0`, so a skill
could regress to a single prompt and still pass the gate. At RUNS=5 one
case is 20 points of that skill's reading, so a one- or two-case skill
reports a rate a single flake can move across the threshold — well-formed
to the checker, meaningless as a measurement. Zero positives keeps its
own clearer message so the common mistake keeps its diagnosis.

Scopes the eval requirement to model-invoked skills in the checklist. A
name-only skill has no eval file because nothing can measure a trigger
only a human types, so an eval there is unrunnable rather than redundant.

Aligns checklist item 1 with the `#` hazard from round 1; it named only
the unquoted colon, which is the louder of the two failure modes.

Four test fixtures asserted on one-positive eval files and legitimately
went red; they now build positives from the exported MIN_POSITIVE_CASES.
The all-positive row keeps testing the missing-negative path, with a new
row for the below-floor case. 388/388 script tests pass.

No skill description or eval case changed, so trigger behaviour is
unaffected and skills:eval is unchanged from the previous round.

Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-09-01 10:32:41 -04:00
commit 880be3acdd

View file

@ -235,6 +235,12 @@ export function listingCost(skills) {
* @param {unknown} cases Parsed JSON.
* @returns {string[]} errors
*/
/**
* The number of positive cases a model-invoked skill's eval file must carry.
* Raised from an implicit 3 to 5 so a single flake cannot cross the threshold.
*/
export const MIN_POSITIVE_CASES = 5;
export function validateEvalCases(skillName, cases) {
if (!Array.isArray(cases) || cases.length === 0) {
return ["evals.json must be a non-empty array of cases"];
@ -268,6 +274,15 @@ export function validateEvalCases(skillName, cases) {
const negatives = cases.filter((c) => c && c.expect === null).length;
if (positives === 0) {
errors.push(`no positive case expects \`${skillName}\``);
} else if (positives < MIN_POSITIVE_CASES) {
// A non-zero floor is not enough. At RUNS=5 one case is worth 20 points of
// that skill's reading, so a skill measured by one or two prompts reports a
// rate that a single flake can move across the threshold — and the gate,
// seeing a positive, would call that well-formed. Five is the count every
// skill here was raised to for exactly that reason (Copilot).
errors.push(
`only ${positives} positive case(s); a model-invoked skill needs at least ${MIN_POSITIVE_CASES} for its rate to mean anything`,
);
}
if (negatives === 0) {
errors.push(