modelcontextprotocol/inspector
Watch
1
0
Fork
You've already forked inspector
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

Review round 1: gate the hand-off on connecting, not on tier depth

Browse source 
Copilot, both findings accepted.

1. The trigger was over-broad. "a smoke" and "everything from web
   integration rightwards" routed work to `test-servers` that needs no
   fixture at all: `smoke:launcher` only checks `--help`, `smoke:web`
   asserts the SPA is served, `smoke:web:browser` stops at first paint,
   and Storybook play functions render from fixture props. The condition
   is now "does this exercise MCP behaviour over a transport?", stated
   as such, with both sides named — the smokes that connect
   (`smoke:cli`, `smoke:web:app`, `smoke:web:elicit`, `smoke:web:tabs`)
   and `pack:verify` on one side, the boot-only ones and Storybook on
   the other. Depth in the tier list is explicitly not the boundary.

2. The threshold rationale contradicted the new result. It said an 0.8
   bar would mark a hand-off red "regardless of how strongly the first
   skill points at the second" — which the 100% measurement refutes.
   Both copies (docs and `skill-eval.mjs`) now say what 0.5 actually
   buys: a floor on useful reliability for a noisier second-hop load, so
   the column still separates an adequate pointer from a broken one. The
   value is unchanged.

Re-measured after the narrowing, since a tighter trigger could have
weakened the pointer: 100% / 100% on the hand-off and 63/63 first-move
at 100%, full suite at RUNS=5. Recorded figure updated from the
intermediate 100% / 80%.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-09-05 16:17:31 -04:00
commit 02f094f001
1 changed files with 54 additions and 34 deletions
Download patch file Download diff file Expand all files Collapse all files

18
scripts/skill-eval.mjs
View file

@ -56,14 +56,16 @@ const THRESHOLD = Number(process.env.THRESHOLD ?? 0.8);
// A hand-off is a harder thing to hit than a first move, and what counts as
// acceptable is a separate judgement rather than one inherited from a number
// tuned for the other measurement. 0.5 is the weakest claim worth asserting —
// the pointer is taken more often than not. It is deliberately not 0.8: the
// committed `testing` -> `test-servers` cases measured 33% (RUNS=3) against a
// pointer that was live and stated in the first paragraph of `testing`'s body,
// so an 0.8 bar would mark a hand-off red regardless of how strongly the first
// skill points at the second, and the column would stop carrying signal.
// (#2247 later reshaped that pointer into an imperative step and took the same
// two cases to 100% at RUNS=5 — which raises the ceiling those cases reach, not
// the floor a *new* hand-off case should be judged against.)
// the pointer is taken more often than not — and it is a floor on useful
// reliability for a SECOND-HOP load, not a claim that 0.8 is unreachable. A
// well-shaped pointer does clear 0.8: the committed `testing` -> `test-servers`
// cases measured 33% (RUNS=3) when `testing` merely classified which work
// belonged to `test-servers`, and 100%/80% (RUNS=5) once #2247 reshaped that
// into an imperative step. What 0.5 buys is a column that still separates an
// adequate pointer from a broken one — a hand-off is a noisier measurement than
// a first move, so a bar set where a STRONG pointer sits would mark both red.
// A reshaped pointer therefore raises the ceiling those cases reach, not the
// floor a *new* hand-off case should be judged against.
//
// It is compared STRICTLY, unlike the first-move threshold. "More often than
// not" is `> 0.5`, and an inclusive compare passes exactly half the samples