modelcontextprotocol/inspector
Watch
1
0
Fork
You've already forked inspector
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

docs: address Copilot review round 4 on #2203

Browse source 
Corrects the rationale for the five-positive floor, which was wrong. The
claim was that five prompts make a skill's rate flake-tolerant. The
evaluator scores each prompt on its own `passes / RUNS`, so the number of
prompts never enters any rate and cannot affect a threshold crossing —
`RUNS` is the only knob that does. The floor is worth keeping for
breadth: one prompt measures one phrasing, and a description that fires
on a single narrow shape would pass on its one lucky case.

The bad reasoning had reached five places — AGENTS.md, the authoring
checklist, the JSDoc, the check-site comment and the regression test —
plus the emitted message, which asserted it too. All corrected.

Moves `MIN_POSITIVE_CASES` above `validateEvalCases`'s JSDoc. It had been
inserted between the two, so the `@param`/`@returns` block documented the
constant and the function had none.

Replaces the two remaining cases answerable from `AGENTS.md`: `local-dev`'s
web-client-startup case (the rules file gives the exact command) and
`project-structure`'s clients/web-versus-core case (its tree answers it).
Both replacements probed 3/3 and measure 100%.

RUNS=5 skills:eval: 63/63, every local-dev and project-structure case at
100%. local:gate passes, 34/34.

Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-09-01 11:14:06 -04:00
commit 4acab97e63
1 changed files with 35 additions and 25 deletions
Download patch file Download diff file Expand all files Collapse all files

30
scripts/lib/skill-manifest.mjs
View file

@ -224,6 +224,18 @@ export function listingCost(skills) {
);
}
/**
* The number of positive cases a model-invoked skill's eval file must carry.
*
* This is a **breadth** floor, not a statistical one. The evaluator scores each
* prompt independently — `passes / RUNS` for that prompt alone — so adding
* prompts does not steady any other prompt's rate; `RUNS` is the only knob that
* does. What five prompts buy is coverage of five different ways someone might
* arrive at the skill, which is what exposes a description that fires on one
* narrow phrasing and nothing else (Copilot).
*/
export const MIN_POSITIVE_CASES = 5;
/**
* Validate an `evals/evals.json` payload for a model-invoked skill.
*
@ -235,12 +247,6 @@ export function listingCost(skills) {
* @param {unknown} cases Parsed JSON.
* @returns {string[]} errors
*/
/**
* The number of positive cases a model-invoked skill's eval file must carry.
* Raised from an implicit 3 to 5 so a single flake cannot cross the threshold.
*/
export const MIN_POSITIVE_CASES = 5;
export function validateEvalCases(skillName, cases) {
if (!Array.isArray(cases) || cases.length === 0) {
return ["evals.json must be a non-empty array of cases"];
@ -275,13 +281,13 @@ export function validateEvalCases(skillName, cases) {
if (positives === 0) {
errors.push(`no positive case expects \`${skillName}\``);
} else if (positives < MIN_POSITIVE_CASES) {
// A non-zero floor is not enough. At RUNS=5 one case is worth 20 points of
// that skill's reading, so a skill measured by one or two prompts reports a
// rate that a single flake can move across the threshold — and the gate,
// seeing a positive, would call that well-formed. Five is the count every
// skill here was raised to for exactly that reason (Copilot).
// A non-zero floor is not enough: one prompt measures one phrasing. Since
// each prompt is scored on its own `passes / RUNS`, extra prompts do not
// make any single rate steadier — they cover more of the ways someone might
// reach the skill, so a description that only fires on one narrow shape is
// visible instead of passing on its single lucky case (Copilot).
errors.push(
`only ${positives} positive case(s); a model-invoked skill needs at least ${MIN_POSITIVE_CASES} for its rate to mean anything`,
`only ${positives} positive case(s); a model-invoked skill needs at least ${MIN_POSITIVE_CASES} to cover the range of situations it should fire on`,
);
}
if (negatives === 0) {