inspector/scripts
Repository files (latest commit first)
Filename Latest commit message Latest commit date
cliffhall 313a580982 feat(skills): address Copilot review round 3 on #2204
One finding, and a sharp one: position in the stream is not causation.

`collectSkillInvocations` flattened every `tool_use` block across every
assistant event into one ordered array, and `chainHit` read position
alone. But the model can emit several `Skill` blocks in a SINGLE assistant
message, and it has not seen the first skill's body when it does — those
are parallel guesses, not a hand-off. A run that emitted `testing` and
`test-servers` together in one message scored as `A` led to `B`, and would
have gone on scoring after the pointer in `testing`'s body was deleted:
exactly the false pass this feature exists to rule out, arriving by a
second route (Copilot).

Each recorded invocation now carries the assistant event it came from —
`{payload, turn}` — and `chainHit` requires every link after the first to
land in a STRICTLY later turn. The turn count is threaded through
`collectSkillInvocations`'s `turnOffset`/`nextTurn` so a stream read in
chunks stays monotonic rather than restarting per read; without that, a
chunk boundary would reset the numbering and reintroduce the same bug on
long runs.

The scan stays greedy, which is still correct under the constraint: taking
the earliest occurrence of a link can only leave more room for the rest,
so no later starting point succeeds where the greedy one fails.

`sampleHit` is deliberately unaffected — a first-move case asks only
whether a skill fired at all, so it reads names and ignores turns.

Four new assertions in one case: a same-turn pair scores false; the same
two skills a turn apart score true; a same-turn pair does not poison a
later genuine hand-off; and only the chain's first link is unconstrained.
Plus a `collectSkillInvocations` case pinning the turn tagging and the
cross-chunk offset.

`docs/skill-authoring.md` now lists this as the second way a chained case
can false-pass, alongside a prompt carrying the target's own trigger.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
2026-09-04 20:28:37 -04:00
..
lib feat(skills): measure a skill reached from another skill (#2204) 2026-09-04 19:07:26 -04:00
dependabot-alerts.mjs chore(deps): address Copilot review round 15 on #2243 2026-09-04 09:26:20 -04:00
dependabot-alerts.test.mjs chore(deps): address Copilot review round 15 on #2243 2026-09-04 09:26:20 -04:00
dependency-refresh.mjs chore(deps): address Copilot review round 1 on #2243 2026-09-04 09:06:32 -04:00
dependency-refresh.test.mjs chore(deps): paginate the release list so the highest version is actual 2026-09-04 09:06:31 -04:00
install-clients.mjs feat: port CLI, TUI, and launcher from v1.5 into v2 (#1452) 2026-06-14 20:27:49 -04:00
install-smoke-browser.mjs fix(smoke): reject an empty SMOKE_BROWSER; test the installer's argument logic 2026-08-25 22:52:02 -04:00
install-smoke-browser.test.mjs fix(smoke): reject an empty SMOKE_BROWSER; test the installer's argument logic 2026-08-25 22:52:02 -04:00
pack-and-verify.mjs ci: drop the Firefox smoke job; keep the engines as an on-demand tool 2026-08-25 23:55:19 -04:00
run-engine-smokes.mjs fix(smoke): route the Chromium tier through ENGINE_SMOKES too 2026-08-26 08:55:51 -04:00
run-engine-smokes.test.mjs Merge remote-tracking branch 'origin/v2/main' into v2/chore/2146-rename-local-gate 2026-08-27 00:16:14 -04:00
skill-eval.mjs feat(skills): address Copilot review round 3 on #2204 2026-09-04 20:28:37 -04:00
skill-eval.test.mjs feat(skills): address Copilot review round 3 on #2204 2026-09-04 20:28:37 -04:00
smoke-cli.mjs chore(scripts): build test-servers unconditionally from one shared helper 2026-08-25 00:57:48 -04:00
smoke-launcher.mjs refactor: self-validating clients, fast validate + separate coverage/smoke (#1504) (#1505) 2026-06-21 11:41:33 -04:00
smoke-tui.mjs fix(scripts): make the test stubs drain, and the raw-mode guard testable (#2147) 2026-08-26 22:13:35 -04:00
smoke-web-app.mjs chore(scripts): rename the local pre-push gate off ci, and guard it out of CI (#2146) 2026-08-26 23:56:34 -04:00
smoke-web-browser.mjs chore(scripts): rename the local pre-push gate off ci, and guard it out of CI (#2146) 2026-08-26 23:56:34 -04:00
smoke-web-elicitation.mjs Merge remote-tracking branch 'origin/v2/main' into v2/chore/2146-rename-local-gate 2026-08-27 00:16:14 -04:00
smoke-web-tabs.mjs Merge remote-tracking branch 'origin/v2/main' into v2/chore/2146-rename-local-gate 2026-08-27 00:16:14 -04:00
smoke-web.mjs fix(smoke): await the server's exit before removing its catalog, and test it 2026-08-11 14:32:20 -04:00
verify-build-gate.mjs chore(scripts): rename the local pre-push gate off ci, and guard it out of CI (#2146) 2026-08-26 23:56:34 -04:00
verify-bundle-externals.mjs test(proxy): isolate proxy env per file; word the guard failure by candidate source 2026-08-23 23:00:28 -04:00
verify-bundle-externals.test.mjs test(proxy): isolate proxy env per file; word the guard failure by candidate source 2026-08-23 23:00:28 -04:00
verify-dep-lockstep.main.test.mjs Merge origin/v2/main: move the #1939 npx fix into the shared tsc-program lib 2026-08-16 17:07:50 -07:00
verify-dep-lockstep.mjs chore(scripts): compare only the copies a program actually met 2026-08-16 12:35:38 -04:00
verify-dep-lockstep.test.mjs chore(scripts): compare only the copies a program actually met 2026-08-16 12:35:38 -04:00
verify-format-coverage.main.test.mjs chore: address Copilot review round 20 2026-08-29 05:32:12 -04:00
verify-format-coverage.mjs chore: address Copilot review round 9 2026-08-29 03:04:03 -04:00
verify-skills-cli.mjs chore: address Copilot review round 17 2026-08-29 04:54:04 -04:00
verify-skills-cli.test.mjs chore: address Copilot review round 16 2026-08-29 04:40:48 -04:00
verify-skills.main.test.mjs feat(skills): measure a skill reached from another skill (#2204) 2026-09-04 19:07:26 -04:00
verify-skills.mjs feat(skills): measure a skill reached from another skill (#2204) 2026-09-04 19:07:26 -04:00
verify-typecheck-coverage.main.test.mjs Merge origin/v2/main: move the #1939 npx fix into the shared tsc-program lib 2026-08-16 17:07:50 -07:00
verify-typecheck-coverage.mjs Merge origin/v2/main: move the #1939 npx fix into the shared tsc-program lib 2026-08-16 17:07:50 -07:00
verify-typecheck-coverage.test.mjs chore(scripts): derive dep-lockstep candidates from what enters each tsc program 2026-08-16 12:17:15 -04:00