feat(skills): address Copilot review round 3 on #2204

One finding, and a sharp one: position in the stream is not causation.

`collectSkillInvocations` flattened every `tool_use` block across every
assistant event into one ordered array, and `chainHit` read position
alone. But the model can emit several `Skill` blocks in a SINGLE assistant
message, and it has not seen the first skill's body when it does — those
are parallel guesses, not a hand-off. A run that emitted `testing` and
`test-servers` together in one message scored as `A` led to `B`, and would
have gone on scoring after the pointer in `testing`'s body was deleted:
exactly the false pass this feature exists to rule out, arriving by a
second route (Copilot).

Each recorded invocation now carries the assistant event it came from —
`{payload, turn}` — and `chainHit` requires every link after the first to
land in a STRICTLY later turn. The turn count is threaded through
`collectSkillInvocations`'s `turnOffset`/`nextTurn` so a stream read in
chunks stays monotonic rather than restarting per read; without that, a
chunk boundary would reset the numbering and reintroduce the same bug on
long runs.

The scan stays greedy, which is still correct under the constraint: taking
the earliest occurrence of a link can only leave more room for the rest,
so no later starting point succeeds where the greedy one fails.

`sampleHit` is deliberately unaffected — a first-move case asks only
whether a skill fired at all, so it reads names and ignores turns.

Four new assertions in one case: a same-turn pair scores false; the same
two skills a turn apart score true; a same-turn pair does not poison a
later genuine hand-off; and only the chain's first link is unconstrained.
Plus a `collectSkillInvocations` case pinning the turn tagging and the
cross-chunk offset.

`docs/skill-authoring.md` now lists this as the second way a chained case
can false-pass, alongside a prompt carrying the target's own trigger.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-09-04 20:28:37 -04:00
commit 313a580982

View file

@ -216,7 +216,7 @@ for everything else — a prompt someone would actually type to reach this skill
directly is a first-move case even when a hand-off could also get there, and it
is the cheaper measurement by an order of magnitude.
Five rules the shape enforces, each for a reason worth knowing:
Six rules the shape enforces, each for a reason worth knowing:
- **The chain ends with the skill whose file it lives in.** The case exists to
measure whether _this_ skill is reachable, so the file that must go red when
@ -232,6 +232,14 @@ Five rules the shape enforces, each for a reason worth knowing:
run.** The model may load something before the chain starts and something
unrelated in between; neither changes the claim that A led to B. What does not
score is the reverse order.
- **Every link after the first must land in a later assistant turn.** Position
in the stream is not causation: the model can emit several `tool_use` blocks
in one message, and it has not seen the first skill's body when it does — so
two `Skill` calls in the same turn are parallel guesses, not a hand-off, and
a flat index would score them as one (Copilot). This is the difference
between "B was loaded after A" and "A led to B", and it is the second way a
chained case can false-pass — the first being a prompt that carries the
target's own trigger, below. Only the chain's *first* link is unconstrained.
- **Repeats and unknown links are rejected.** A repeated link cannot be
observed, and a link naming a skill the model cannot invoke can never fire —
it would score a permanent 0% that reads as a description problem.