modelcontextprotocol/inspector
Watch
1
0
Fork
You've already forked inspector
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

Review round 6: 80% does not clear a strictly-compared 0.8

Browse source 
Copilot, accepted — and the sharpest catch of the six, because it is
the rationale contradicting the operator two paragraphs below it.

Round 1 rewrote the CHAIN_THRESHOLD comment to say a well-shaped
pointer "does clear 0.8" and cited 100%/80% as the evidence. This
threshold is compared with `>`, so 80% would FAIL an 0.8 bar; the
example refuted the sentence it was offered in support of. It was also
stale: 100%/80% was an intermediate build, and the shipped state
measures 100%/100%.

Both copies now cite 100%/100% as the clearing evidence and keep
100%/80% only as the residual-noise figure, saying explicitly that it
is not an example of clearing the bar.

While checking the claim I found the docs copy needed the same care for
a different reason: `passesThreshold` is strict for the chain and
INCLUSIVE for the first move (`rate > t` vs `rate >= t`, skill-eval.mjs
:558), and 0.8 is the constant for the inclusive one. So "0.8 is
compared strictly" is false as literally read. The docs now say a
*chain* bar of 0.8 would be the strict one, name the operator for both,
and spell out the consequence at RUNS=5 — only 5/5 passes, 4/5 does not.

No functional change; CHAIN_THRESHOLD is still 0.5, still strict.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-09-05 17:39:11 -04:00
commit 03af5b966f
1 changed files with 17 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

6
docs/skill-authoring.md
View file

@ -285,8 +285,10 @@ license (Copilot). A strict bound of `1.0` is therefore unreachable and the
harness rejects it up front rather than failing every case.
0.5 is a floor on **useful reliability for a second-hop load**, not a claim that
0.8 is out of reach — a well-shaped pointer does clear it, as the worked example
below records. What 0.5 buys is that the column keeps carrying signal across the
0.8 is out of reach — a well-shaped pointer clears it outright, at 100% in the
worked example below. Note that a *chain* bar of 0.8 would be a **strict** one
(`> 0.8`, the comparison this threshold uses; the first-move 0.8 is the
inclusive `>=`), so at `RUNS=5` only a clean 5/5 would pass it — 4/5 would not. What 0.5 buys is that the column keeps carrying signal across the
*range* of pointer strengths a repo actually has: a hand-off is a noisier
measurement than a first move, so a bar set where a strong pointer sits marks
every merely-adequate one red and stops distinguishing them from a broken one.