- Reword the CI guarantee in AGENTS.md, docs/quality-gate.md and the
pre-push-gate skill: a green gate means every check CI applies has
passed here — the strongest predictor of green CI, not a proof — and
name the residual (a test that passes only when slowed) as #1596's
class to fix rather than headroom to keep.
- docs/quality-gate.md: cli's validate DID build, through `test`'s
`pretest` hook; say that the build goes with the dropped `test` leg and
`coverage:cli` builds once, later — in both places.
- Update the two source comments that still described `local:gate` as
running `validate` (verify-skills.mjs, verify-build-gate.mjs).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PsmUAZ6aLUAFqenjJKSSRp
Signed-off-by: cliffhall <cliff@futurescale.com>
`skills:eval` ran every case with `--max-turns 1`, so it measured one
thing: whether a skill is the model's FIRST tool call. That is right for a
skill a user reaches directly, and it leaves a whole class unmeasured — a
skill reached from inside another skill's body. `testing` tells the model
that picking a fixture is `/test-servers` and that it has to load it, and
`test-servers` scores 5/5 while every one of those five cases asks for it
by name. A skill only ever reachable through a hand-off would score a
clean 100% with the hand-off silently never firing.
An eval case is now one of two shapes, and exactly one:
{ "prompt": "…", "expect": "test-servers" } first move, 1 turn
{ "prompt": "…", "chain": ["testing", "test-servers"] } hand-off, 14 turns
A chain must END with the skill whose file it lives in. The case exists to
measure whether THIS skill is reachable, so the file that goes red is the
one belonging to the skill that stopped being reached; anchoring on the
first link would file a `test-servers` measurement under `testing`, where
a `test-servers` description edit would never be seen. Links are checked
against the repo's model-invoked set, which is why both `verify:skills`
and `collectCases` now validate in a second pass — `test-servers` sorts
before `testing`, so a per-directory check would reject a live chain as
unknown purely because of where the alphabet put it.
Scoring is an ordered SUBSEQUENCE, not a prefix and not a contiguous run:
the model may load something before the chain starts and something
unrelated in between, and neither changes the claim that A led to B.
`collectSkillInvocations` therefore returns an ordered array rather than a
Set — a B, A, B run has to stay distinguishable from one that never
reached B from A.
The two rates are reported in separate columns and never summed. A
hand-off is a second-hop load over many turns; a first-move rate is the
model's opening move. `CHAIN_MAX_TURNS` (14) and `CHAIN_THRESHOLD` are
their own knobs for the same reason, and `CHAIN_THRESHOLD` defaults to
**0.5** rather than inheriting 0.8: see the measurement below.
Read-only containment no longer leans on `--max-turns 1`, which was doing
much of it by itself. The deny list gains the agentic and network tools —
`Task` in particular, whose subagent the flag does not reach.
Measured, `skills:eval -- test-servers`:
First move (1 turn) 7/7 at 100%, unchanged
Hand-off (14 turns) 67% and 33% at RUNS=3
So the hand-off is real and unreliable — which is the fact nothing could
observe before this change, and the reason 0.5 rather than 0.8 is the
default bar: at 0.8 both committed cases are red no matter how strongly
the first skill points at the second, and the column stops carrying
signal. A `["pr-flow", "test-servers"]` probe measured 0% and was dropped
rather than kept: `pr-flow` says nothing about fixtures, so the case had
no lever short of broadening a description onto another skill's ground.
Both findings are written up in `docs/skill-authoring.md`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
All three findings are the same shape: a check that accepts a command which
exists but cannot gate.
- Splitting on `|` and `&` conflated `||`, so both failure-masking forms counted:
`npm run X || true` swallows a rejection and `true || npm run X` never runs
the validator. Both `runsCommand` and the chain walk now split on `\n`, `;`
and `&&` only, which refuses each without special-casing either.
- A CI step (or job) carrying `continue-on-error` asserts nothing, since a
rejected skill leaves the workflow green. Both are now skipped, treating a
string `"true"` and an unevaluable `${{ }}` expression as masking on the same
conservative principle as the `if:` checks; an explicit `false` still counts.
- `rootReachesScript` was substring-based, so a `validate` reading `echo npm run
verify:skills` satisfied the sibling vouch while the guard never executed —
the cycle protecting nothing. It now uses the exact chain walk, which is
therefore shared in `lib/npm-scripts.mjs` and fixes the same hole in all four
vouches, not just this PR's. `reachableScripts` keeps its substring behavior
for the harvesting callers, where over-inclusion is safe.
The chain walk also follows npm's implicit pre/post hooks, with a test — a guard
invoked from a `prevalidate` is genuinely run, and reporting it missing would be
a false failure introduced by the fix.
Signed-off-by: cliffhall <cliff@futurescale.com>
The `local:gate` half of the wiring vouch used `reachableScripts`, which
extracts any `npm run …` substring — so `echo npm run verify:skills:cli && npm
run coverage` satisfied it while the validator never ran. Same over-match the CI
half had in round 15, from the other side.
`scriptChainRuns` walks the script graph instead, splitting each body on the
separators that begin a new command and requiring an exact invocation of the
target. A segment naming some other script is followed with or without trailing
flags, since `npm run coverage --silent` still runs it, and `seen` makes a
cyclic graph terminate. Regression cases for the mention, a longer script name,
and the cycle; verified by mutation.
Signed-off-by: cliffhall <cliff@futurescale.com>
- ciRunsUnconditionally never looked at the workflow-level `on:` trigger, so a
workflow switched to `on: release` still contained the step and reported as
wired while no PR ever ran the validator. It now requires `push` or
`pull_request`, handling all three shapes `on:` takes. The positive fixtures
had no trigger at all, so they could not assert the "on every push" half of
the contract; they carry one now, alongside a release-only negative. Verified
by mutation.
- Stale JSDoc from the round-16 version change: `probeClaudeVersion` is now
generic in its parser's result rather than hard-coding `number[]`,
`validatorCommand` documents `{ parts, prerelease }`, and `runValidator`
documents the `platform` option it already accepted. Annotations that lie make
valid call sites look type-invalid.
Signed-off-by: cliffhall <cliff@futurescale.com>
`parseClaudeVersion` dropped SemVer suffixes, so an installed `2.1.250-beta.1`
parsed as `[2, 1, 250]` and compared EQUAL to the stable pin — a different
validator schema running locally while the log claimed an exact match, which is
the cross-machine disagreement the pin exists to prevent.
It now returns `{ parts, prerelease }`, because the two callers want different
things: `isPinnedVersion` rejects any prerelease (build metadata still matches,
per SemVer precedence), while the eval's availability probe only needs some
usable CLI. `formatClaudeVersion` keeps the suffix in the messages, so a
prerelease is reported as what it is rather than as the pin.
Signed-off-by: cliffhall <cliff@futurescale.com>
- The CI wiring guard matched a substring, and last round's justification for
that was wrong: it accepts a longer script name
(`npm run verify:skills:cli:disabled`) and a mere mention (`echo npm run
verify:skills:cli`), either of which lets the real validator be removed while
the guard stays green. `runsCommand` now splits the script on the separators
that begin a new command and requires a segment to equal the invocation
exactly, with both counterexamples as regression cases.
- `local-dev`'s own description and its AGENTS.md index row both still promised
"where a new dependency must be declared" — the rules round 13 moved out of it
entirely. Both now describe what the skill actually carries: the reasoning
behind those rules. Recorded listing share updated to 1,199/4,000.
Signed-off-by: cliffhall <cliff@futurescale.com>
- The CI wiring vouch accepted any syntactically-executable `run:`, so a step
moved into a release-only job or put behind `if: false` still reported as
wired while PR CI validated nothing. It now requires an UNCONDITIONAL step:
any `if:` on the job or the step disqualifies it, since this guard cannot
evaluate an Actions expression and "runs sometimes" is not the property being
asserted. Seven predicate cases, including the two conditional shapes and a
multi-line `run:` that must still count.
- The `local:gate` stage list now has one owner, `docs/quality-gate.md`. It had
three copies, and they drifted apart across rounds 5, 6 and 8 of this review —
which is the argument the PR itself is making, applied to itself.
Signed-off-by: cliffhall <cliff@futurescale.com>
The CI half of the wiring vouch matched the workflow's raw text, so a YAML
comment, a step `name:`, or an unused anchor mentioning `npm run
verify:skills:cli` satisfied it — including this guard's own docblock wording.
Deleting the real step would then have left `verify:skills` green, which is the
vacuous-guard failure the vouch exists to prevent.
It now reads the workflow's executable regions via `extractExecutableRegions`,
the same helper the #2146 workflow gate is built on, and matches only a `run:`.
Covered by three should-not-count cases (comment, step name, workflow input)
plus a multi-line `run:` that must count; verified by mutation — restoring the
substring check fails them.
Signed-off-by: cliffhall <cliff@futurescale.com>
- Enrol both skill gates in the repo's gate-integrity vouch cycle. A gate that
stops being invoked fails silently with every fixture test green, which is the
failure the existing cycle exists to prevent. `verify:format-coverage` now
vouches for `verify:skills`, `verify:skills` vouches back, and it additionally
asserts the two links nothing else could see: `verify:skills:cli` present in
`local:gate` and in the workflow — neither is reachable from `validate` or
from any test. `checkWiring` is pure and unit-tested, with a live case against
the real package.json and workflow so the fixtures cannot drift. All four
links verified by mutation.
- The existing sibling-vouch fixture is now a table over every sibling, so
adding a fourth guard later does not need a new test.
- release skill: stop hard-coding `2.3.0`, which has already shipped — the tag
command aborted on an existing tag and the branch template named the wrong
release. Both now read the version from `package.json` (the tag from
`origin/main`'s), with a warning never to copy a version out of the file.
Signed-off-by: cliffhall <cliff@futurescale.com>
Node has refused shell-free `.cmd`/`.bat` spawns since CVE-2024-27980, and an
npm-installed `claude` is a `.cmd` shim on Windows — so every shell-free spawn
of it reported ENOENT there, each in a quiet way: `verify:skills` skipped its
authoritative hand-off, `verify:skills:cli` downloaded the pinned package it
already had (and failed offline), and `skills:eval` said "not on PATH".
- New `scripts/lib/claude-cli.mjs` is the one place that decides how `claude` is
spawned: `shell` on win32, arguments through the existing `winShellArgs`
quoting so `cmd.exe` cannot re-parse them as syntax (#1939). Unit-tested with
the platform injected, since a win32-only bug no runner here executes is
exactly the kind that ships.
- `verify-skills`, `verify-skills-cli` and `skill-eval` all route their probes
and spawns through it.
- skill-eval sends the prompt on **stdin** rather than in argv. `claude -p` reads
a piped prompt (verified end to end), which sidesteps `cmd.exe` metacharacter
re-parsing entirely rather than relying on quoting a model-visible string, and
keeps the prompt out of the process table.
Signed-off-by: cliffhall <cliff@futurescale.com>
- verify-skills: the best-effort `plugin validate` hand-off now also requires
the EXACT pinned CLI. `local:gate` runs `validate` before the pinned
`verify:skills:cli` step, so a newer local CLI could reject skills that CI's
pinned validator accepts — and the gate would exit there, never reaching the
reproducible check. One schema everywhere, or none.
- Dropped `PLUGIN_VALIDATE_MIN_VERSION`: with both validators on the exact pin
it was read only by its own test, and a constant nothing enforces is a claim
nothing enforces. Its test now pins what actually matters — that equality is
distinguishable from newer as well as older.
- The CI step's comment described the old floor behavior.
- issue-create: the duplicate check searched only open issues, so a completed or
rejected match was invisible to a procedure that says never to duplicate.
Searches all states and reports state/stateReason.
Signed-off-by: cliffhall <cliff@futurescale.com>
- verify:skills:cli — the authoritative validator is now a GUARANTEED step in
`local:gate` and CI, not a best-effort hand-off. It resolves the CLI itself
(an installed one at 2.1.233+, else a pinned `npx -y`), so both tiers run the
same script and cannot drift. `verify:skills` keeps skipping when the CLI is
absent, which is right for `validate`: fast, offline, no Claude Code required.
- skill-manifest: the 1,536-char listing-entry cap covers `description` +
`when_to_use` together; checking the description alone passed an over-cap
entry. `when_to_use` is now parsed, validated and counted, with boundary tests
at 1,536/1,537.
- verify-skills: report the budget as *this repo's contribution*. The real
listing budget is shared with bundled skills and the contributor's own
~/.claude/skills, none of which is visible from here, so the number was
claiming more than it measured.
- skill-eval: a negative case now means "no skill OF THIS REPO'S fired". It
failed on any Skill invocation, so a contributor with an unrelated global
skill got a false failure about their environment rather than these skills.
- docs/architecture + local-dev: the Node backend runner gets its
`@inspector/core` alias from `clients/web/tsup.runner.config.ts`, not from
Vite. The old text sent runner work to the wrong config.
- local-dev: the tarball does not ship "only each client's build/" — the
allowlist also carries `clients/web/dist`, `clients/web/static` and the
install helper. States the invariant that actually matters instead.
- issue-triage: the test for needing triage is "no board card". Adding "and no
milestone" would skip the approved-but-unboarded case the same section then
handles. Also corrected the `$M` note: the lookup is `--state all`, so closed
issues DO have entries — `isopen` is about the invariants, not the lookup.
- issue-create: replaced "all five" with a per-version table. Steps 3 and 6
(milestone, Priority) do not exist for v1, so the old list opened with a rule
the same skill later tells you to violate.
Signed-off-by: cliffhall <cliff@futurescale.com>
- CI now installs a pinned `claude` CLI and runs `claude plugin validate` as its
own step. The guard skips the authoritative validator when the CLI is absent,
which without this meant "always, in CI" — so the acceptance criterion was
aspirational. Verified it needs no authentication (clean HOME).
- skill-eval: classify a run from its terminal `result` event, not its exit
code. With `--max-turns 1` a run in which a skill FIRES necessarily hits the
limit and the CLI exits 1 (`error_max_turns`, `num_turns: 2`), so last round's
blanket nonzero rejection threw away exactly the observations the eval counts.
A run with no terminal `result`, or an unusable subtype, still rejects.
- skill-eval: fail collection on a skill that does not parse, a model-invoked
skill with no eval file, unreadable JSON, or invalid cases. Skipping them let
`skills:eval` exit 0 while silently omitting the skill just broken.
- verify-skills: `main()` takes an optional skills directory, and
`verify-skills.main.test.mjs` drives it over fixtures — missing SKILL.md,
malformed frontmatter, undeclared invocation mode, missing/invalid evals,
budget overflow (and that name-only skills do not count toward it), and
multi-offender reporting. Matches the sibling guards' *.main.test.mjs
precedent; without it the gate could stop enforcing while the parser tests
stayed green.
- issue-create: the note claimed an issue arrives unlabeled, contradicting
.github/ISSUE_TEMPLATE — blank issues are disabled, the bug form applies
`bug`, the feature form applies `enhancement` + `v2`. Now a table of what each
form sets and what triage still has to add.
Signed-off-by: cliffhall <cliff@futurescale.com>
- verify-skills: version-gate the optional `claude plugin validate`. Probing
only for the executable failed the MANDATORY gate for anyone on a CLI older
than 2.1.233, which has no such subcommand and exits nonzero on it. Skips
below the floor; a failure now means a supported validator rejected the skills.
- skill-eval: match the invoked skill structurally. A substring test counted
`{"skill":"not-testing"}` as a hit for `testing` and inflated the rate.
- skill-eval: `runOnce` → exported `runPrompt` with an injectable `spawnFn`, so
the nonzero-exit branch is actually pinned. Reverting it now fails a test
(verified by mutation) rather than leaving the suite green.
- board-ops: filter the item lookups by repository. Issue numbers are repo-local
and these are org boards — #11 really does carry a `servers` card — so an
unfiltered `select` can move or delete the wrong card, or pass two ids.
- issue-triage: the audit's `own()` filter dropped draft cards, which carry no
`.content.repository`, so the "non-Issue on a board" check reported 0 while
the invariant was being violated. Verified against a fixture.
- pr-flow: cut the branch from the base it will target. The blanket "cut from
v2/main" was wrong for v1, whose history is unrelated.
Signed-off-by: cliffhall <cliff@futurescale.com>
Closes#2163
`AGENTS.md` was one 1,223-line contract loaded in full on every turn, whether
the task was a one-line CSS fix or a release cut. This splits it: rules stay
resident, procedures move to `.claude/skills/`.
- Ten skills under `.claude/skills/`, seven `disable-model-invocation: true`
(invoked by name only, so they cost no listing context and have no trigger
question) and three model-invoked with committed eval cases.
- `AGENTS.md` 1,223 → 380 lines: rules, a skills index, and a project tree
reduced to one line per entry. The removed recipes are not duplicated.
- `README.md` 811 → 105 lines, split into `docs/{architecture,quality-gate,
test-servers,publishing,docker}.md`, and dropped from `CLAUDE.md`'s auto-load.
- `.github/copilot-instructions.md` and its mirroring rule removed — Copilot
reads `AGENTS.md`.
- `npm run verify:skills` (in `validate`, so in `local:gate` and CI) parses each
SKILL.md the way Claude Code does, requires an explicit invocation mode and
eval cases for model-invoked skills, and budgets the listing.
- `npm run skills:eval` runs the committed trigger cases headless.
Signed-off-by: cliffhall <cliff@futurescale.com>
