Adds a multi-name filter to `npm run skills:eval` so a set of related
skills can be iterated together, and makes an unknown name a hard error
rather than an empty run — a typo would otherwise report a green 0/0
that reads exactly like a clean pass of the skill that was meant.
Re-aims two `local-dev` cases and one `test-servers` case that partially
overlapped `AGENTS.md`. Partial overlap oscillates across runs rather
than sitting at zero, so it reads as eval noise instead of a defect in
the case; `docs/skill-authoring.md` now records that failure mode and
the observed scores.
Widens the `test-servers` description to the "Use when …" shape and
points the `testing` skill at it, since anything needing a real server
to run against is a procedure `testing` does not carry. That growth
moves the skill listing from 2,921 to 3,144 of 4,000 characters, so the
figure recorded in AGENTS.md is updated to match.
Signed-off-by: cliffhall <cliff@futurescale.com>
`skills:eval` was at 38/46 with several skills measuring 0-33%. The low
scores were mostly bad eval cases rather than bad descriptions, for two
reasons that were nowhere written down:
- The harness runs each case with `--max-turns 1`, so the skill has to be
the model's FIRST tool call. A prompt asking "how does X work here", or
naming a concrete file, sends the model to Read/Grep first — a better
answer to the question, scored as a miss. Every 0% case was one of these.
- `AGENTS.md` is resident on every turn, so a prompt it already answers is
answered without any skill and the case can never pass.
Replace the cases that measured the wrong thing, expand every model-invoked
skill from 3 to 5 positives (negatives kept) so one flake cannot cross the
threshold, and reshape `local-dev`'s description to the action-first form.
Add `docs/skill-authoring.md` with the case shapes that fire, the ones that
cannot, and the probe-then-measure loop.
RUNS=5 npm run skills:eval: 63/63 at or above threshold, 60 at 100%,
all 18 negatives clean.
Closes#2202
Signed-off-by: cliffhall <cliff@futurescale.com>
Two suppressed comments, both accurate.
- Pin zod, zod/v4 and open to the root in vitest.shared.mts, and add matching
paths to the three clients/web tsconfigs. The claim they replace — that these
resolve unambiguously by walk-up — was wrong: both still sit at the top level
of clients/web/node_modules as transitive copies (zod under
eslint-plugin-react-hooks, open under Storybook), so an unpinned bare import
from a web test resolved the client copy while core/ and the SDK packages
resolved the root's. For open that is only wasteful; for zod it is a schema
built by one copy and an instanceof check made by the other across the whole
@modelcontextprotocol/* surface. zod is deduped as well, and zod/v4 gets its
own entry because first-party code imports both specifiers and pinning one
would split the package rather than collapse it.
- Update the consumer comments that #2195 falsified. clients/web/vite.config.ts
(three places) and server/start-vite-dev-server.ts all still said the shared
aliases resolve pino/hono and friends from clients/web/node_modules; they now
state the split — react and react-dom from the client install, everything
root-declared from the repo root.
- local-dev skill: replace the wrong 'resolves unambiguously' list with the
distinction that actually matters. Deleting a declaration is not the same as
the package resolving from the root, because chokidar, zod, open and react
all sit in a client install transitively; check the install, not the manifest.
ajv, commander and undici carry no pin because no client install has a copy.
npm run local:gate passes from a cold tsc build.
Signed-off-by: cliffhall <cliff@futurescale.com>
- clients/web/tsconfig.{app,node,test}.json: repoint the pino, hono,
hono/streaming, @hono/node-server, atomically, chokidar and @napi-rs/keyring
path mappings from ./node_modules to ../../node_modules. The stale mappings
were not failing the build — an unresolvable paths entry falls back to
ordinary walk-up resolution, which finds the root copy — but chokidar is the
case where the fallback never runs: Vite's transitive copy still sits under
clients/web, so tsc was type-checking against the client's copy while Vitest
and the shipped bundle both resolved the root's. Same version today, silent
skew tomorrow. Both path prefixes now carry a comment saying which ownership
each one asserts.
- local-dev skill: the consolidation rationale no longer groups commander, open
and @hono/node-server (client-imported) with the core/ set, no longer claims
#2195 moved undici (it was already root-only), and no longer implies every
root-declared package has an alias.
npm run local:gate passes with clients/web/node_modules/.tmp cleared, so the
tsc results are from a cold build rather than an incremental cache.
Signed-off-by: cliffhall <cliff@futurescale.com>
- Complete the bundler external lists against the AGENTS.md invariant rather
than against today's reachability: chokidar, hono and react to clients/cli,
chokidar and hono to clients/tui, react to the web runner. Every root-declared
package core/ imports at runtime is now named in all three, so a new import in
core/ cannot silently inline one.
- AGENTS.md: split the single over-broad bullet in two. commander, open,
@hono/node-server, vite and @vitejs/plugin-react are root dependencies reached
only from client code, so they are not part of the core/ set and need naming
only in the external list of the client that imports them. Give the
vitest.shared.mts pins their own rule, stating the react/react-dom exception
instead of implying every root-owned package is aliased to the root.
- vitest.shared.mts: the umbrella comment now says root-OWNED rather than root
runtime dependency — express is test-only and lives in the root
devDependencies — and notes that the list is deliberately not the complete
root set, since a package that resolves unambiguously needs no pin.
- local-dev skill: document why react/react-dom stay client-pinned, which the
previous text contradicted.
npm run local:gate and npm run verify:bundle-externals pass.
Signed-off-by: cliffhall <cliff@futurescale.com>
Every runtime dependency core/ imports is now declared once, in the repo-root
package.json, and nowhere else. Deleting the client-side copies does not merely
align them: a package installs only into an install root that declares it, so
with no second declaration a second copy cannot be created at all — the failure
class #1896 was opened for and #1965/#2029 built verify:dep-lockstep to detect.
- clients/{web,cli,tui,launcher}: dropped ajv, atomically, chokidar, commander,
open, pino, zod, @napi-rs/keyring, hono, @hono/node-server and react, as each
applied. clients/cli and clients/launcher now declare no runtime dependencies.
- Root hono floor raised ^4.12.18 -> ^4.13.1, keeping the higher of the two
ranges that were in the tree rather than silently lowering web's.
- tsup external lists: added ajv and zod to all three bundles, plus atomically
and open (cli) and atomically (tui). tsup externalizes only what the nearest
manifest declares, so a package that moves to the root is inlined unless it is
named explicitly.
- vitest.shared.mts: repointed the pino, hono, @hono/node-server, atomically,
chokidar and @napi-rs/keyring pins from <client>/node_modules to the repo
root, joining express, yaml and proper-lockfile. Left behind they would
resolve to a directory that no longer exists, or to a transitive copy an
unrelated dependency dragged in. react and react-dom stay client-pinned so
the pair stays version-matched within an install.
- Docs: AGENTS.md dependency placement, the root README and the local-dev skill
describe the consolidated layout.
verify:dep-lockstep candidate set: 4 -> 2 (hono and pino fell out; only
@standard-schema/spec and zod still reach one tsc program from two installs).
npm run local:gate and npm run pack:verify both pass.
Signed-off-by: cliffhall <cliff@futurescale.com>
- The CI wiring guard matched a substring, and last round's justification for
that was wrong: it accepts a longer script name
(`npm run verify:skills:cli:disabled`) and a mere mention (`echo npm run
verify:skills:cli`), either of which lets the real validator be removed while
the guard stays green. `runsCommand` now splits the script on the separators
that begin a new command and requires a segment to equal the invocation
exactly, with both counterexamples as regression cases.
- `local-dev`'s own description and its AGENTS.md index row both still promised
"where a new dependency must be declared" — the rules round 13 moved out of it
entirely. Both now describe what the skill actually carries: the reasoning
behind those rules. Recorded listing share updated to 1,199/4,000.
Signed-off-by: cliffhall <cliff@futurescale.com>
- skill-manifest: an eval file may only expect its own skill. A foreign
`expect: "other-skill"` passed whenever that other skill fired, so the file
reported a measurement of something it does not describe — and it satisfied
the positive-case requirement while doing so, masking the absence of a real
positive. Both halves covered by a regression case.
- local-dev: stop restating the dependency-placement rules. They are normative
and belong only in AGENTS.md; the skill now points there and carries what a
rule cannot — what each one is defending against, what it looked like when it
was violated (#1970, #2082, #2067, #1952, #1896, #2058), and how to tell you
have hit it. Including the two traps that make this class hard to see: the
ordinary test tiers cannot catch a bundling miss, and probing one by hand with
`node -e` falsely succeeds.
Signed-off-by: cliffhall <cliff@futurescale.com>
- verify:skills:cli — the authoritative validator is now a GUARANTEED step in
`local:gate` and CI, not a best-effort hand-off. It resolves the CLI itself
(an installed one at 2.1.233+, else a pinned `npx -y`), so both tiers run the
same script and cannot drift. `verify:skills` keeps skipping when the CLI is
absent, which is right for `validate`: fast, offline, no Claude Code required.
- skill-manifest: the 1,536-char listing-entry cap covers `description` +
`when_to_use` together; checking the description alone passed an over-cap
entry. `when_to_use` is now parsed, validated and counted, with boundary tests
at 1,536/1,537.
- verify-skills: report the budget as *this repo's contribution*. The real
listing budget is shared with bundled skills and the contributor's own
~/.claude/skills, none of which is visible from here, so the number was
claiming more than it measured.
- skill-eval: a negative case now means "no skill OF THIS REPO'S fired". It
failed on any Skill invocation, so a contributor with an unrelated global
skill got a false failure about their environment rather than these skills.
- docs/architecture + local-dev: the Node backend runner gets its
`@inspector/core` alias from `clients/web/tsup.runner.config.ts`, not from
Vite. The old text sent runner work to the wrong config.
- local-dev: the tarball does not ship "only each client's build/" — the
allowlist also carries `clients/web/dist`, `clients/web/static` and the
install helper. States the invariant that actually matters instead.
- issue-triage: the test for needing triage is "no board card". Adding "and no
milestone" would skip the approved-but-unboarded case the same section then
handles. Also corrected the `$M` note: the lookup is `--state all`, so closed
issues DO have entries — `isopen` is about the invariants, not the lookup.
- issue-create: replaced "all five" with a per-version table. Steps 3 and 6
(milestone, Priority) do not exist for v1, so the old list opened with a rule
the same skill later tells you to violate.
Signed-off-by: cliffhall <cliff@futurescale.com>
Closes#2163
`AGENTS.md` was one 1,223-line contract loaded in full on every turn, whether
the task was a one-line CSS fix or a release cut. This splits it: rules stay
resident, procedures move to `.claude/skills/`.
- Ten skills under `.claude/skills/`, seven `disable-model-invocation: true`
(invoked by name only, so they cost no listing context and have no trigger
question) and three model-invoked with committed eval cases.
- `AGENTS.md` 1,223 → 380 lines: rules, a skills index, and a project tree
reduced to one line per entry. The removed recipes are not duplicated.
- `README.md` 811 → 105 lines, split into `docs/{architecture,quality-gate,
test-servers,publishing,docker}.md`, and dropped from `CLAUDE.md`'s auto-load.
- `.github/copilot-instructions.md` and its mirroring rule removed — Copilot
reads `AGENTS.md`.
- `npm run verify:skills` (in `validate`, so in `local:gate` and CI) parses each
SKILL.md the way Claude Code does, requires an explicit invocation mode and
eval cases for model-invoked skills, and budgets the listing.
- `npm run skills:eval` runs the committed trigger cases headless.
Signed-off-by: cliffhall <cliff@futurescale.com>