One finding, and a sharp one: position in the stream is not causation.
`collectSkillInvocations` flattened every `tool_use` block across every
assistant event into one ordered array, and `chainHit` read position
alone. But the model can emit several `Skill` blocks in a SINGLE assistant
message, and it has not seen the first skill's body when it does — those
are parallel guesses, not a hand-off. A run that emitted `testing` and
`test-servers` together in one message scored as `A` led to `B`, and would
have gone on scoring after the pointer in `testing`'s body was deleted:
exactly the false pass this feature exists to rule out, arriving by a
second route (Copilot).
Each recorded invocation now carries the assistant event it came from —
`{payload, turn}` — and `chainHit` requires every link after the first to
land in a STRICTLY later turn. The turn count is threaded through
`collectSkillInvocations`'s `turnOffset`/`nextTurn` so a stream read in
chunks stays monotonic rather than restarting per read; without that, a
chunk boundary would reset the numbering and reintroduce the same bug on
long runs.
The scan stays greedy, which is still correct under the constraint: taking
the earliest occurrence of a link can only leave more room for the rest,
so no later starting point succeeds where the greedy one fails.
`sampleHit` is deliberately unaffected — a first-move case asks only
whether a skill fired at all, so it reads names and ignores turns.
Four new assertions in one case: a same-turn pair scores false; the same
two skills a turn apart score true; a same-turn pair does not poison a
later genuine hand-off; and only the chain's first link is unconstrained.
Plus a `collectSkillInvocations` case pinning the turn tagging and the
cross-chunk offset.
`docs/skill-authoring.md` now lists this as the second way a chained case
can false-pass, alongside a prompt carrying the target's own trigger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
Two findings, both correct, and the first one means round 1's containment
fix was only half a fix.
**`--allowedTools` is not the availability filter.** It grants permission;
it does not restrict which tools exist. So a tool a user's or a plugin's
settings already permit stayed reachable across all 14 turns, and the
bound still rested on the deny list being complete — the exact property
round 1 set out to stop depending on. `--tools` is the real restriction
(it selects from the built-in set), and both flags now carry the same
four names: `--tools` to bound, `--allowedTools` so those four never need
a permission prompt no headless run can answer. Verified against the real
CLI — the full flag set runs to a clean `end_turn`. The test asserts both
flags, and both probe snippets in `docs/skill-authoring.md` carry
`--tools` with a ⚠️ saying which one does the bounding.
**The threshold guard let `NaN` and negatives through.**
`CHAIN_THRESHOLD=abc` is `NaN`, which fails every comparison and prints an
`above NaN%` summary; a negative bar passes every chain unconditionally.
Both turn an advertised env knob into a measurement that quietly means
nothing. `CHAIN_THRESHOLD` must now be in `[0, 1)` — half-open because the
chain bar is strict — and `THRESHOLD` in `[0, 1]`, inclusive because a
first-move floor of 1.0 is meetable. The message names the offending
value. A new case spawns the script for each bad value rather than
asserting on the predicate, since these guards live in `main`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
Five findings, all real.
**The chain prompts carried the target skill's own trigger.** `test-servers`
claims "a change needs a real server to exercise it", and both cases said
"against a real/live server" — so the model could load `testing`, then pick
`test-servers` from the ORIGINAL prompt, in that order, and score a hit that
would have survived deleting the pointer from `testing` entirely. Rewritten to
carry no server cue ("end to end"), and the measured rate fell from 100% / 67%
to 33% / 33%. That is the size of the artefact, and 33% is what the pointer
alone is worth. Both cases are now red against the 0.5 bar and stay that way:
`skills:eval` is not a gate, the number IS the finding, and lowering the bar to
turn the column green would discard the only signal this feature adds. Filed as
#2247 with the trap written down so a fix cannot re-measure the artefact.
**A deny list cannot bound a 14-turn run.** It only names tools known when it
was written, and this checkout configures an HTTP `mcp-docs` server in
`.mcp.json` while a contributor's own MCP servers and plugins add more. Now
`--allowedTools Read,Glob,Grep,Skill` (what the harness actually needs) plus
`--strict-mcp-config` with no `--mcp-config`, which drops every configured
server. The deny list stays alongside, since a deny is unconditional while an
allow list governs pre-approval. What remains outside all three is a
contributor's plugin tools; `--bare` would remove those and skills with them,
which would measure nothing — so that residual is stated rather than papered
over.
**The chain threshold compared inclusively.** "More often than not" is `> 0.5`,
and `>=` passes exactly half whenever RUNS is even — 2/4 would report a result
the criterion does not license. `passesThreshold` is now strict for chains and a
floor for first moves, and a strict bound of 1.0 is rejected up front rather
than failing every case while looking like a trigger problem.
**The doc's probe command still denied only four tools**, contradicting the
section's own promise that a probe and a scored case see the same policy.
Updated, with a note that it must move with the harness, plus a hand-off probe
form.
**The reporting had no automated coverage** — the one thing that could silently
merge the two measurements lived inside `main`. Extracted as `formatReport` and
tested: both headings, per-group thresholds, the two separate summaries, the
combined exit, and both single-kind selections.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
`skills:eval` ran every case with `--max-turns 1`, so it measured one
thing: whether a skill is the model's FIRST tool call. That is right for a
skill a user reaches directly, and it leaves a whole class unmeasured — a
skill reached from inside another skill's body. `testing` tells the model
that picking a fixture is `/test-servers` and that it has to load it, and
`test-servers` scores 5/5 while every one of those five cases asks for it
by name. A skill only ever reachable through a hand-off would score a
clean 100% with the hand-off silently never firing.
An eval case is now one of two shapes, and exactly one:
{ "prompt": "…", "expect": "test-servers" } first move, 1 turn
{ "prompt": "…", "chain": ["testing", "test-servers"] } hand-off, 14 turns
A chain must END with the skill whose file it lives in. The case exists to
measure whether THIS skill is reachable, so the file that goes red is the
one belonging to the skill that stopped being reached; anchoring on the
first link would file a `test-servers` measurement under `testing`, where
a `test-servers` description edit would never be seen. Links are checked
against the repo's model-invoked set, which is why both `verify:skills`
and `collectCases` now validate in a second pass — `test-servers` sorts
before `testing`, so a per-directory check would reject a live chain as
unknown purely because of where the alphabet put it.
Scoring is an ordered SUBSEQUENCE, not a prefix and not a contiguous run:
the model may load something before the chain starts and something
unrelated in between, and neither changes the claim that A led to B.
`collectSkillInvocations` therefore returns an ordered array rather than a
Set — a B, A, B run has to stay distinguishable from one that never
reached B from A.
The two rates are reported in separate columns and never summed. A
hand-off is a second-hop load over many turns; a first-move rate is the
model's opening move. `CHAIN_MAX_TURNS` (14) and `CHAIN_THRESHOLD` are
their own knobs for the same reason, and `CHAIN_THRESHOLD` defaults to
**0.5** rather than inheriting 0.8: see the measurement below.
Read-only containment no longer leans on `--max-turns 1`, which was doing
much of it by itself. The deny list gains the agentic and network tools —
`Task` in particular, whose subagent the flag does not reach.
Measured, `skills:eval -- test-servers`:
First move (1 turn) 7/7 at 100%, unchanged
Hand-off (14 turns) 67% and 33% at RUNS=3
So the hand-off is real and unreliable — which is the fact nothing could
observe before this change, and the reason 0.5 rather than 0.8 is the
default bar: at 0.8 both committed cases are red no matter how strongly
the first skill points at the second, and the column stops carrying
signal. A `["pr-flow", "test-servers"]` probe measured 0% and was dropped
rather than kept: `pr-flow` says nothing about fixtures, so the case had
no lever short of broadening a description onto another skill's ground.
Both findings are written up in `docs/skill-authoring.md`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
Enforces the five-positive standard this PR established rather than only
describing it. `validateEvalCases` checked `positives === 0`, so a skill
could regress to a single prompt and still pass the gate. At RUNS=5 one
case is 20 points of that skill's reading, so a one- or two-case skill
reports a rate a single flake can move across the threshold — well-formed
to the checker, meaningless as a measurement. Zero positives keeps its
own clearer message so the common mistake keeps its diagnosis.
Scopes the eval requirement to model-invoked skills in the checklist. A
name-only skill has no eval file because nothing can measure a trigger
only a human types, so an eval there is unrunnable rather than redundant.
Aligns checklist item 1 with the `#` hazard from round 1; it named only
the unquoted colon, which is the louder of the two failure modes.
Four test fixtures asserted on one-positive eval files and legitimately
went red; they now build positives from the exported MIN_POSITIVE_CASES.
The all-positive row keeps testing the missing-negative path, with a new
row for the below-floor case. 388/388 script tests pass.
No skill description or eval case changed, so trigger behaviour is
unaffected and skills:eval is unchanged from the previous round.
Signed-off-by: cliffhall <cliff@futurescale.com>
Adds a multi-name filter to `npm run skills:eval` so a set of related
skills can be iterated together, and makes an unknown name a hard error
rather than an empty run — a typo would otherwise report a green 0/0
that reads exactly like a clean pass of the skill that was meant.
Re-aims two `local-dev` cases and one `test-servers` case that partially
overlapped `AGENTS.md`. Partial overlap oscillates across runs rather
than sitting at zero, so it reads as eval noise instead of a defect in
the case; `docs/skill-authoring.md` now records that failure mode and
the observed scores.
Widens the `test-servers` description to the "Use when …" shape and
points the `testing` skill at it, since anything needing a real server
to run against is a procedure `testing` does not carry. That growth
moves the skill listing from 2,921 to 3,144 of 4,000 characters, so the
figure recorded in AGENTS.md is updated to match.
Signed-off-by: cliffhall <cliff@futurescale.com>
- skill-eval: `only` no longer narrows `ours`. A negative case means "no skill
of this repo fired", so filtering the repo's other model-invoked skills out of
that set let `skills:eval -- testing` score a `local-dev` invocation as
somebody else's skill and pass a negative case it should fail. Every
model-invoked skill is now inspected; `only` applies when enqueueing.
`collectCases` takes a skills directory so the focused-mode regression is
actually testable — verified by mutation: restoring the early filter fails it.
- The workflow's `validate` step comment said two coverage guards; it runs four.
Signed-off-by: cliffhall <cliff@futurescale.com>
Node has refused shell-free `.cmd`/`.bat` spawns since CVE-2024-27980, and an
npm-installed `claude` is a `.cmd` shim on Windows — so every shell-free spawn
of it reported ENOENT there, each in a quiet way: `verify:skills` skipped its
authoritative hand-off, `verify:skills:cli` downloaded the pinned package it
already had (and failed offline), and `skills:eval` said "not on PATH".
- New `scripts/lib/claude-cli.mjs` is the one place that decides how `claude` is
spawned: `shell` on win32, arguments through the existing `winShellArgs`
quoting so `cmd.exe` cannot re-parse them as syntax (#1939). Unit-tested with
the platform injected, since a win32-only bug no runner here executes is
exactly the kind that ships.
- `verify-skills`, `verify-skills-cli` and `skill-eval` all route their probes
and spawns through it.
- skill-eval sends the prompt on **stdin** rather than in argv. `claude -p` reads
a piped prompt (verified end to end), which sidesteps `cmd.exe` metacharacter
re-parsing entirely rather than relying on quoting a model-visible string, and
keeps the prompt out of the process table.
Signed-off-by: cliffhall <cliff@futurescale.com>
- verify:skills:cli — the authoritative validator is now a GUARANTEED step in
`local:gate` and CI, not a best-effort hand-off. It resolves the CLI itself
(an installed one at 2.1.233+, else a pinned `npx -y`), so both tiers run the
same script and cannot drift. `verify:skills` keeps skipping when the CLI is
absent, which is right for `validate`: fast, offline, no Claude Code required.
- skill-manifest: the 1,536-char listing-entry cap covers `description` +
`when_to_use` together; checking the description alone passed an over-cap
entry. `when_to_use` is now parsed, validated and counted, with boundary tests
at 1,536/1,537.
- verify-skills: report the budget as *this repo's contribution*. The real
listing budget is shared with bundled skills and the contributor's own
~/.claude/skills, none of which is visible from here, so the number was
claiming more than it measured.
- skill-eval: a negative case now means "no skill OF THIS REPO'S fired". It
failed on any Skill invocation, so a contributor with an unrelated global
skill got a false failure about their environment rather than these skills.
- docs/architecture + local-dev: the Node backend runner gets its
`@inspector/core` alias from `clients/web/tsup.runner.config.ts`, not from
Vite. The old text sent runner work to the wrong config.
- local-dev: the tarball does not ship "only each client's build/" — the
allowlist also carries `clients/web/dist`, `clients/web/static` and the
install helper. States the invariant that actually matters instead.
- issue-triage: the test for needing triage is "no board card". Adding "and no
milestone" would skip the approved-but-unboarded case the same section then
handles. Also corrected the `$M` note: the lookup is `--state all`, so closed
issues DO have entries — `isopen` is about the invariants, not the lookup.
- issue-create: replaced "all five" with a per-version table. Steps 3 and 6
(milestone, Priority) do not exist for v1, so the old list opened with a rule
the same skill later tells you to violate.
Signed-off-by: cliffhall <cliff@futurescale.com>
- CI now installs a pinned `claude` CLI and runs `claude plugin validate` as its
own step. The guard skips the authoritative validator when the CLI is absent,
which without this meant "always, in CI" — so the acceptance criterion was
aspirational. Verified it needs no authentication (clean HOME).
- skill-eval: classify a run from its terminal `result` event, not its exit
code. With `--max-turns 1` a run in which a skill FIRES necessarily hits the
limit and the CLI exits 1 (`error_max_turns`, `num_turns: 2`), so last round's
blanket nonzero rejection threw away exactly the observations the eval counts.
A run with no terminal `result`, or an unusable subtype, still rejects.
- skill-eval: fail collection on a skill that does not parse, a model-invoked
skill with no eval file, unreadable JSON, or invalid cases. Skipping them let
`skills:eval` exit 0 while silently omitting the skill just broken.
- verify-skills: `main()` takes an optional skills directory, and
`verify-skills.main.test.mjs` drives it over fixtures — missing SKILL.md,
malformed frontmatter, undeclared invocation mode, missing/invalid evals,
budget overflow (and that name-only skills do not count toward it), and
multi-offender reporting. Matches the sibling guards' *.main.test.mjs
precedent; without it the gate could stop enforcing while the parser tests
stayed green.
- issue-create: the note claimed an issue arrives unlabeled, contradicting
.github/ISSUE_TEMPLATE — blank issues are disabled, the bug form applies
`bug`, the feature form applies `enhancement` + `v2`. Now a table of what each
form sets and what triage still has to add.
Signed-off-by: cliffhall <cliff@futurescale.com>
- verify-skills: version-gate the optional `claude plugin validate`. Probing
only for the executable failed the MANDATORY gate for anyone on a CLI older
than 2.1.233, which has no such subcommand and exits nonzero on it. Skips
below the floor; a failure now means a supported validator rejected the skills.
- skill-eval: match the invoked skill structurally. A substring test counted
`{"skill":"not-testing"}` as a hit for `testing` and inflated the rate.
- skill-eval: `runOnce` → exported `runPrompt` with an injectable `spawnFn`, so
the nonzero-exit branch is actually pinned. Reverting it now fails a test
(verified by mutation) rather than leaving the suite green.
- board-ops: filter the item lookups by repository. Issue numbers are repo-local
and these are org boards — #11 really does carry a `servers` card — so an
unfiltered `select` can move or delete the wrong card, or pass two ids.
- issue-triage: the audit's `own()` filter dropped draft cards, which carry no
`.content.repository`, so the "non-Issue on a board" check reported 0 while
the invariant was being violated. Verified against a fixture.
- pr-flow: cut the branch from the base it will target. The blanket "cut from
v2/main" was wrong for v1, whose history is unrelated.
Signed-off-by: cliffhall <cliff@futurescale.com>
- skill-manifest: the closing `---` fence must occupy a whole line. `---oops`
was accepted as a terminator, silently truncating the frontmatter and passing
validation — the exact failure the guard exists to catch. Regression cases
added (prefix, trailing whitespace, EOF, CRLF).
- skill-eval: reject a nonzero `claude -p` exit instead of resolving it as an
empty observation. An auth error or rate limit was scoring as "no skill
invoked", which passes every negative case and reads as a trigger miss on
every positive one.
- skill-eval: extract `collectSkillInvocations` / `sampleHit` behind a `main()`
guard and cover them in `skill-eval.test.mjs` — the stream parsing and the
process-outcome handling are unreachable from an eval run's happy path.
- docs/test-servers: annotate the era on every catalogue row. Six legacy rows
carried no annotation under a "unless noted, Modern" default, so the doc sent
readers to the wrong era for the feature they were trying to see.
- docs/quality-gate: `validate` runs four guards, not three; add the
`verify:skills` row and the two new `test:scripts` suites; fix the `#publishing`
fragment left dangling by the split.
- issue-create / pr-flow: delegate the card steps to `/board-ops` rather than
restating its option IDs, which are explicitly unstable.
- issue-triage: the audit counted presence, not cardinality — an issue carrying
both `v1` and `v2` passed a check that claims "exactly one". Same for the five
type labels. Also scope the two milestone checks to board #28.
- AGENTS.md: codify the v1 milestone exception. Every milestone is a v2 release
bucket, so a v1 issue has none to take and the Incoming-milestone invariant
cannot apply to board #11.
- clients/launcher/README: its pointer promised an explanation of why the cli/tui
manifests ship, which the extracted guide does not make.
Signed-off-by: cliffhall <cliff@futurescale.com>
