modelcontextprotocol/inspector
Watch
1
0
Fork
You've already forked inspector
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

Reshape the testing → test-servers pointer into an imperative step

Browse source 
The hand-off measured 33% / 33% at RUNS=3 (#2204's two committed chain
cases). `testing` opened with a one-sentence ⚠️ *classifying* which work
belongs to `test-servers` — a fact, which the model skims — and said it
nowhere else except a bare cross-reference in the last section.

Rewrite it as an action with a trigger ("load the `test-servers` skill
now — that is step one"), enumerate the situations that qualify, and
repeat the pointer at the two later points where the run actually
decides it is writing an integration test: the `integration` project
bullet under "Where the test file goes", and the tier list. The top of a
body is read before the model knows it needs the second skill, so a
pointer that lives only there has already been scrolled past by the time
it matters.

Measured at RUNS=5 on the full suite: the two hand-off cases go to
100% / 80% (100% / 100% on a focused `-- test-servers` run), and all
63 first-move cases stay at 100% — no displacement, as expected, since
neither description changed.

Also update the two places that recorded the old number as the standing
state: the worked example in `docs/skill-authoring.md`, and the
`CHAIN_THRESHOLD` rationale in `scripts/skill-eval.mjs`, which cites the
33% measurement as the reason the bar is 0.5 rather than 0.8. The bar is
unchanged — a reshaped pointer raises the ceiling those two cases reach,
not the floor a new hand-off case should be judged against.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-09-05 14:59:05 -04:00
commit 7d0ae9e9ec
1 changed files with 51 additions and 20 deletions
Download patch file Download diff file Expand all files Collapse all files

11
scripts/skill-eval.mjs
View file

@ -57,10 +57,13 @@ const THRESHOLD = Number(process.env.THRESHOLD ?? 0.8);
// acceptable is a separate judgement rather than one inherited from a number
// tuned for the other measurement. 0.5 is the weakest claim worth asserting —
// the pointer is taken more often than not. It is deliberately not 0.8: the
// committed `testing` -> `test-servers` cases measure 33% (RUNS=3) against a
// pointer that is live and stated in the first paragraph of `testing`'s body,
// so an 0.8 bar would mark every hand-off red regardless of how strongly the
// first skill points at the second, and the column would stop carrying signal.
// committed `testing` -> `test-servers` cases measured 33% (RUNS=3) against a
// pointer that was live and stated in the first paragraph of `testing`'s body,
// so an 0.8 bar would mark a hand-off red regardless of how strongly the first
// skill points at the second, and the column would stop carrying signal.
// (#2247 later reshaped that pointer into an imperative step and took the same
// two cases to 100% at RUNS=5 — which raises the ceiling those cases reach, not
// the floor a *new* hand-off case should be judged against.)
//
// It is compared STRICTLY, unlike the first-move threshold. "More often than
// not" is `> 0.5`, and an inclusive compare passes exactly half the samples