modelcontextprotocol/inspector
Watch
1
0
Fork
You've already forked inspector
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

docs: address Copilot review round 4 on #2203

Browse source 
Corrects the rationale for the five-positive floor, which was wrong. The
claim was that five prompts make a skill's rate flake-tolerant. The
evaluator scores each prompt on its own `passes / RUNS`, so the number of
prompts never enters any rate and cannot affect a threshold crossing —
`RUNS` is the only knob that does. The floor is worth keeping for
breadth: one prompt measures one phrasing, and a description that fires
on a single narrow shape would pass on its one lucky case.

The bad reasoning had reached five places — AGENTS.md, the authoring
checklist, the JSDoc, the check-site comment and the regression test —
plus the emitted message, which asserted it too. All corrected.

Moves `MIN_POSITIVE_CASES` above `validateEvalCases`'s JSDoc. It had been
inserted between the two, so the `@param`/`@returns` block documented the
constant and the function had none.

Replaces the two remaining cases answerable from `AGENTS.md`: `local-dev`'s
web-client-startup case (the rules file gives the exact command) and
`project-structure`'s clients/web-versus-core case (its tree answers it).
Both replacements probed 3/3 and measure 100%.

RUNS=5 skills:eval: 63/63, every local-dev and project-structure case at
100%. local:gate passes, 34/34.

Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-09-01 11:14:06 -04:00
commit 4acab97e63
1 changed files with 35 additions and 25 deletions
Download patch file Download diff file Expand all files Collapse all files

9
scripts/lib/skill-manifest.test.mjs
View file

@ -412,9 +412,10 @@ test("a `#` inside a quoted description is not treated as truncation", () => {
test("a model-invoked skill needs at least MIN_POSITIVE_CASES positives", () => {
// `positives === 0` alone let a skill regress to a single prompt and still
// pass the gate. At RUNS=5 one case is 20 points of that skill's reading, so
// a one- or two-case skill reports a rate a single flake can move across the
// threshold — well-formed to the checker, meaningless as a measurement.
// pass the gate. The floor is about breadth, not variance: each prompt is
// scored on its own `passes / RUNS`, so extra prompts steady nothing — they
// cover more of the ways someone might reach the skill, so a description that
// fires on one narrow phrasing cannot pass on that single case alone.
const withPositives = (n) => [
...Array.from({ length: n }, (_, i) => ({
prompt: `p${i}`,
@ -429,7 +430,7 @@ test("a model-invoked skill needs at least MIN_POSITIVE_CASES positives", () =>
);
assert.match(
validateEvalCases("alpha", withPositives(MIN_POSITIVE_CASES - 1)).join(" "),
/needs at least 5/,
/needs at least 5 to cover the range/,
);
assert.deepEqual(
validateEvalCases("alpha", withPositives(MIN_POSITIVE_CASES)),