modelcontextprotocol/inspector
Watch
1
0
Fork
You've already forked inspector
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

docs: address Copilot review round 3 on #2203

Browse source 
Replaces the last `project-structure` case that `AGENTS.md` answers — a
helper used by the CLI and TUI belongs in `core/` per its own directory
tree. This was also the only case below 100% in the previous suite, so
the static reading and the score agreed. The replacement targets the
skill's first stated purpose, locating code, which `AGENTS.md` gives no
method for; it probed 3/3 and now measures 100%.

Extends the truncation guard's tests to `when_to_use`. The guard loops
over both fields but only `description` was pinned, so dropping the
second from that list would have gone unnoticed. The new case pairs a
quoted description with an unquoted `when_to_use` and asserts the
description is not implicated.

Scopes the negative-case requirement in the prose to model-invoked
skills; round 2 scoped the checklist but left this sentence saying
"every skill file".

Refreshes the stale `~2.8k of 4k` listing figure. The other instance is
past-tense about the three-to-nine experiment and stays, now said
explicitly so two numbers do not look like a disagreement.

Records the eval's real blind spot: all three cases review identified as
`AGENTS.md`-answerable were scoring 80-100% when flagged, so a high score
does not clear a case of overlap. The eval can show a case failing; it
cannot show a passing case is well-aimed.

RUNS=5 skills:eval: 63/63. local:gate passes, 34/34.

Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-09-01 10:53:56 -04:00
commit 104a05dd7f
1 changed files with 54 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

25
docs/skill-authoring.md
View file

@ -122,6 +122,26 @@ oscillates — and looks like eval noise rather than a defect in the case. Two o
| "I added a dependency and the TUI bundle broke at import time. **Where should it have gone?**" | 33 / 67 / 100 / 100 / 60% | Dependency placement is stated in full in `AGENTS.md`; only the diagnosis is the skill's |
| "I just pulled and a client's dependencies look out of sync. **What do I need to run?**" | 80 / 80 / 60% | `AGENTS.md` says a single root `npm install` is all you need |
⚠️ **A high score does not clear a case of overlap, and this is the eval's real
blind spot.** The two rules above — outright overlap pins a case near 0%, partial
overlap makes it oscillate — describe what the *scores* look like once you already
suspect a case. They are not a detector. Across code review of this PR, three
committed cases were identified as `AGENTS.md`-answerable by *reading* them, and
all three were scoring **80–100%** at the time:
| Case | Score when flagged |
| --- | --- |
| `project-structure`: "Does a new Node-only backend piece for the web client belong in core…?" | 80 → 100% |
| `project-structure`: "Does a helper that only the CLI and TUI use belong in core…?" | 80% |
| `issue-create`: "What do I need to do beyond `gh issue create`…?" | 100 / 100 / 100% |
The first two were re-aimed; the third was kept, because a case that holds at
100% across three independent suites is measuring *something* the rules file does
not supply on its own. But the lesson stands either way: **the eval cannot tell
you a passing case is well-aimed.** It can only tell you a case is failing. Read
every new case against `AGENTS.md` by hand before committing it, and do not let a
green suite substitute for that.
Both stabilised at 100% once re-aimed at what only the skill holds — "what do I
**check**" instead of "where should it have **gone**", and the worktree install
trap instead of the install command. If a case keeps moving between runs while
@ -147,8 +167,9 @@ neighbour and the neighbour is right, the case is aimed at the wrong skill.
### Negative cases
Every skill file needs at least one `"expect": null` case; a skill that fires on
everything is a context regression nobody notices by hand. Keep them plainly
Every **model-invoked** skill's eval file needs at least one `"expect": null`
case; a skill that fires on everything is a context regression nobody notices by
hand. (A name-only skill has no eval file to put one in — see the checklist.) Keep them plainly
unrelated to the repo (arithmetic, trivia, a one-line refactor). All 18 in this
repo have held at 100% through every reshaping so far — if one starts firing, a
description has grown too broad.