feat(skills): measure a skill reached from another skill (#2204)

`skills:eval` ran every case with `--max-turns 1`, so it measured one
thing: whether a skill is the model's FIRST tool call. That is right for a
skill a user reaches directly, and it leaves a whole class unmeasured — a
skill reached from inside another skill's body. `testing` tells the model
that picking a fixture is `/test-servers` and that it has to load it, and
`test-servers` scores 5/5 while every one of those five cases asks for it
by name. A skill only ever reachable through a hand-off would score a
clean 100% with the hand-off silently never firing.

An eval case is now one of two shapes, and exactly one:

  { "prompt": "…", "expect": "test-servers" }              first move, 1 turn
  { "prompt": "…", "chain": ["testing", "test-servers"] }  hand-off, 14 turns

A chain must END with the skill whose file it lives in. The case exists to
measure whether THIS skill is reachable, so the file that goes red is the
one belonging to the skill that stopped being reached; anchoring on the
first link would file a `test-servers` measurement under `testing`, where
a `test-servers` description edit would never be seen. Links are checked
against the repo's model-invoked set, which is why both `verify:skills`
and `collectCases` now validate in a second pass — `test-servers` sorts
before `testing`, so a per-directory check would reject a live chain as
unknown purely because of where the alphabet put it.

Scoring is an ordered SUBSEQUENCE, not a prefix and not a contiguous run:
the model may load something before the chain starts and something
unrelated in between, and neither changes the claim that A led to B.
`collectSkillInvocations` therefore returns an ordered array rather than a
Set — a B, A, B run has to stay distinguishable from one that never
reached B from A.

The two rates are reported in separate columns and never summed. A
hand-off is a second-hop load over many turns; a first-move rate is the
model's opening move. `CHAIN_MAX_TURNS` (14) and `CHAIN_THRESHOLD` are
their own knobs for the same reason, and `CHAIN_THRESHOLD` defaults to
**0.5** rather than inheriting 0.8: see the measurement below.

Read-only containment no longer leans on `--max-turns 1`, which was doing
much of it by itself. The deny list gains the agentic and network tools —
`Task` in particular, whose subagent the flag does not reach.

Measured, `skills:eval -- test-servers`:

  First move (1 turn)     7/7 at 100%, unchanged
  Hand-off (14 turns)     67% and 33% at RUNS=3

So the hand-off is real and unreliable — which is the fact nothing could
observe before this change, and the reason 0.5 rather than 0.8 is the
default bar: at 0.8 both committed cases are red no matter how strongly
the first skill points at the second, and the column stops carrying
signal. A `["pr-flow", "test-servers"]` probe measured 0% and was dropped
rather than kept: `pr-flow` says nothing about fixtures, so the case had
no lever short of broadening a description onto another skill's ground.
Both findings are written up in `docs/skill-authoring.md`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-09-04 19:01:19 -04:00
commit 75940976f8

View file

@ -262,6 +262,20 @@ that from happening:
("how does the `@inspector/core` alias resolve?") invites a `Read`, which is
a *better* answer than a skill. Good cases are "how do I / where does this go"
questions whose answer is a procedure.
**A pointer from one skill's body to another is measured by a `chain` case,
not an `expect` one.** A first-move case can only observe the model's opening
tool call, so a skill reached only *through* another scores a clean 100% on
its direct cases while the hand-off silently never fires (#2204). A chained
case names the ordered skills one run should load, **ending with the skill
whose file it lives in** — so the file that goes red is the one belonging to
the skill that stopped being reached. It runs on a wider turn budget, is
scored against its own `CHAIN_THRESHOLD`, and is **reported in its own
column**: a hand-off rate and a first-move rate are not comparable, and
folding them together would move a headline everyone reads as trigger
reliability. It counts toward neither the five-positive floor nor the
negative requirement, and it is only worth writing where the first link's
body actually points at the target — a chain through a skill that says
nothing about it is a permanent 0% with no lever.
⚠️ **The gate cannot catch a description that never matches.** `verify:skills`
checks that a skill is well-formed and that its cases exist; only
`skills:eval` observes whether it actually fires, and that cannot be gated —