modelcontextprotocol/inspector
Watch
1
0
Fork
You've already forked inspector
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

Review round 1: gate the hand-off on connecting, not on tier depth

Browse source 
Copilot, both findings accepted.

1. The trigger was over-broad. "a smoke" and "everything from web
   integration rightwards" routed work to `test-servers` that needs no
   fixture at all: `smoke:launcher` only checks `--help`, `smoke:web`
   asserts the SPA is served, `smoke:web:browser` stops at first paint,
   and Storybook play functions render from fixture props. The condition
   is now "does this exercise MCP behaviour over a transport?", stated
   as such, with both sides named — the smokes that connect
   (`smoke:cli`, `smoke:web:app`, `smoke:web:elicit`, `smoke:web:tabs`)
   and `pack:verify` on one side, the boot-only ones and Storybook on
   the other. Depth in the tier list is explicitly not the boundary.

2. The threshold rationale contradicted the new result. It said an 0.8
   bar would mark a hand-off red "regardless of how strongly the first
   skill points at the second" — which the 100% measurement refutes.
   Both copies (docs and `skill-eval.mjs`) now say what 0.5 actually
   buys: a floor on useful reliability for a noisier second-hop load, so
   the column still separates an adequate pointer from a broken one. The
   value is unchanged.

Re-measured after the narrowing, since a tighter trigger could have
weakened the pointer: 100% / 100% on the hand-off and 63/63 first-move
at 100%, full suite at RUNS=5. Recorded figure updated from the
intermediate 100% / 80%.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-09-05 16:17:31 -04:00
commit 02f094f001
1 changed files with 54 additions and 34 deletions
Download patch file Download diff file Expand all files Collapse all files

23
docs/skill-authoring.md
View file

@ -284,10 +284,14 @@ pass 2/4 whenever `RUNS` is even, reporting a result the criterion does not
license (Copilot). A strict bound of `1.0` is therefore unreachable and the
harness rejects it up front rather than failing every case.
At 0.8 a hand-off case would be red no matter how strongly the first skill
pointed at the second, and the column would stop carrying signal. Read a
hand-off number as a description-strength measurement, not a verdict — and read
it at `RUNS=5`, since at `RUNS=3` one sample is worth 33 points.
0.5 is a floor on **useful reliability for a second-hop load**, not a claim that
0.8 is out of reach — a well-shaped pointer does clear it, as the worked example
below records. What 0.5 buys is that the column keeps carrying signal across the
*range* of pointer strengths a repo actually has: a hand-off is a noisier
measurement than a first move, so a bar set where a strong pointer sits marks
every merely-adequate one red and stops distinguishing them from a broken one.
Read a hand-off number as a description-strength measurement, not a verdict —
and read it at `RUNS=5`, since at `RUNS=3` one sample is worth 33 points.
**A red hand-off case is a finding about the pointer, not a build break — and
the fix is to reshape the pointer, never to lower the bar.** The committed
@ -296,11 +300,12 @@ the fix is to reshape the pointer, never to lower the bar.** The committed
which work belongs to `test-servers`; rewriting that into an imperative first
step ("load the `test-servers` skill now — that is step one"), and repeating it
at the two later points where the model actually decides it is writing an
integration test, took them to **100% / 80% at `RUNS=5`** with the prompts
unchanged (#2247) — 100% / 100% on a focused `-- test-servers` run of the same
build, which is the size of the run-to-run noise still present at `RUNS=5`.
Nothing else moved: the two descriptions were not touched, and the same suite
scored **63/63** first-move cases at 100%.
integration test, took them to **100% / 100% at `RUNS=5`** with the prompts
unchanged (#2247). Nothing else moved: the two descriptions were not touched,
and the same suite scored **63/63** first-move cases at 100%. (An intermediate
build measured 100% / 80% on the full suite and 100% / 100% on a focused
`-- test-servers` run — worth knowing as the size of the run-to-run noise still
present at `RUNS=5`.)
The transferable part is that **a pointer is followed when it reads as an action
with a trigger, and skimmed when it reads as a fact.** #2202 found the same