docs: address Copilot review round 2 on #2203
Enforces the five-positive standard this PR established rather than only describing it. `validateEvalCases` checked `positives === 0`, so a skill could regress to a single prompt and still pass the gate. At RUNS=5 one case is 20 points of that skill's reading, so a one- or two-case skill reports a rate a single flake can move across the threshold — well-formed to the checker, meaningless as a measurement. Zero positives keeps its own clearer message so the common mistake keeps its diagnosis. Scopes the eval requirement to model-invoked skills in the checklist. A name-only skill has no eval file because nothing can measure a trigger only a human types, so an eval there is unrunnable rather than redundant. Aligns checklist item 1 with the `#` hazard from round 1; it named only the unquoted colon, which is the louder of the two failure modes. Four test fixtures asserted on one-positive eval files and legitimately went red; they now build positives from the exported MIN_POSITIVE_CASES. The all-positive row keeps testing the missing-negative path, with a new row for the below-floor case. 388/388 script tests pass. No skill description or eval case changed, so trigger behaviour is unaffected and skills:eval is unchanged from the previous round. Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
50f3360d65
commit
880be3acdd
1 changed files with 136 additions and 29 deletions
|
|
@ -235,6 +235,12 @@ export function listingCost(skills) {
|
|||
* @param {unknown} cases Parsed JSON.
|
||||
* @returns {string[]} errors
|
||||
*/
|
||||
/**
|
||||
* The number of positive cases a model-invoked skill's eval file must carry.
|
||||
* Raised from an implicit 3 to 5 so a single flake cannot cross the threshold.
|
||||
*/
|
||||
export const MIN_POSITIVE_CASES = 5;
|
||||
|
||||
export function validateEvalCases(skillName, cases) {
|
||||
if (!Array.isArray(cases) || cases.length === 0) {
|
||||
return ["evals.json must be a non-empty array of cases"];
|
||||
|
|
@ -268,6 +274,15 @@ export function validateEvalCases(skillName, cases) {
|
|||
const negatives = cases.filter((c) => c && c.expect === null).length;
|
||||
if (positives === 0) {
|
||||
errors.push(`no positive case expects \`${skillName}\``);
|
||||
} else if (positives < MIN_POSITIVE_CASES) {
|
||||
// A non-zero floor is not enough. At RUNS=5 one case is worth 20 points of
|
||||
// that skill's reading, so a skill measured by one or two prompts reports a
|
||||
// rate that a single flake can move across the threshold — and the gate,
|
||||
// seeing a positive, would call that well-formed. Five is the count every
|
||||
// skill here was raised to for exactly that reason (Copilot).
|
||||
errors.push(
|
||||
`only ${positives} positive case(s); a model-invoked skill needs at least ${MIN_POSITIVE_CASES} for its rate to mean anything`,
|
||||
);
|
||||
}
|
||||
if (negatives === 0) {
|
||||
errors.push(
|
||||
|
|
|
|||
Loading…
Reference in a new issue