The hand-off measured 33% / 33% at RUNS=3 (#2204's two committed chain
cases). `testing` opened with a one-sentence ⚠️ *classifying* which work
belongs to `test-servers` — a fact, which the model skims — and said it
nowhere else except a bare cross-reference in the last section.
Rewrite it as an action with a trigger ("load the `test-servers` skill
now — that is step one"), enumerate the situations that qualify, and
repeat the pointer at the two later points where the run actually
decides it is writing an integration test: the `integration` project
bullet under "Where the test file goes", and the tier list. The top of a
body is read before the model knows it needs the second skill, so a
pointer that lives only there has already been scrolled past by the time
it matters.
Measured at RUNS=5 on the full suite: the two hand-off cases go to
100% / 80% (100% / 100% on a focused `-- test-servers` run), and all
63 first-move cases stay at 100% — no displacement, as expected, since
neither description changed.
Also update the two places that recorded the old number as the standing
state: the worked example in `docs/skill-authoring.md`, and the
`CHAIN_THRESHOLD` rationale in `scripts/skill-eval.mjs`, which cites the
33% measurement as the reason the bar is 0.5 rather than 0.8. The bar is
unchanged — a reshaped pointer raises the ceiling those two cases reach,
not the floor a new hand-off case should be judged against.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
Adds a multi-name filter to `npm run skills:eval` so a set of related
skills can be iterated together, and makes an unknown name a hard error
rather than an empty run — a typo would otherwise report a green 0/0
that reads exactly like a clean pass of the skill that was meant.
Re-aims two `local-dev` cases and one `test-servers` case that partially
overlapped `AGENTS.md`. Partial overlap oscillates across runs rather
than sitting at zero, so it reads as eval noise instead of a defect in
the case; `docs/skill-authoring.md` now records that failure mode and
the observed scores.
Widens the `test-servers` description to the "Use when …" shape and
points the `testing` skill at it, since anything needing a real server
to run against is a procedure `testing` does not carry. That growth
moves the skill listing from 2,921 to 3,144 of 4,000 characters, so the
figure recorded in AGENTS.md is updated to match.
Signed-off-by: cliffhall <cliff@futurescale.com>
`skills:eval` was at 38/46 with several skills measuring 0-33%. The low
scores were mostly bad eval cases rather than bad descriptions, for two
reasons that were nowhere written down:
- The harness runs each case with `--max-turns 1`, so the skill has to be
the model's FIRST tool call. A prompt asking "how does X work here", or
naming a concrete file, sends the model to Read/Grep first — a better
answer to the question, scored as a miss. Every 0% case was one of these.
- `AGENTS.md` is resident on every turn, so a prompt it already answers is
answered without any skill and the case can never pass.
Replace the cases that measured the wrong thing, expand every model-invoked
skill from 3 to 5 positives (negatives kept) so one flake cannot cross the
threshold, and reshape `local-dev`'s description to the action-first form.
Add `docs/skill-authoring.md` with the case shapes that fire, the ones that
cannot, and the probe-then-measure loop.
RUNS=5 npm run skills:eval: 63/63 at or above threshold, 60 at 100%,
all 18 negatives clean.
Closes#2202
Signed-off-by: cliffhall <cliff@futurescale.com>
Measured before and after at the same sample size (n=4), with the six other
model-invoked skills unchanged:
which command runs the TUI suite 25% → 100%
asserting on a Mantine transition 50% → 100%
which tier spawns the built binary 25% → 100%
The lever is the description's SHAPE, not its length: leading with the actions
and then enumerating concrete situations ("Use when … ; when … ; when …") beats
a noun-phrase list of contents ending in a generic "use for any question
about …". The same edit took `pre-push-gate` from 3/5 to 5/5 earlier.
Re-ran the full eval to check the change did not simply move attention off
another skill, which is the failure the previous commit recorded: 40/46, up from
38/46, and the failing set is strictly smaller — `testing` went from three
failures to one, nothing else moved. That remaining case measured 100% in
isolation and 67% in the full run, which is sampling variance rather than a
finding.
AGENTS.md now extends the re-run rule to cover a description edit, not just a
flip, and records the shape that works.
Signed-off-by: cliffhall <cliff@futurescale.com>
Closes#2163
`AGENTS.md` was one 1,223-line contract loaded in full on every turn, whether
the task was a one-line CSS fix or a release cut. This splits it: rules stay
resident, procedures move to `.claude/skills/`.
- Ten skills under `.claude/skills/`, seven `disable-model-invocation: true`
(invoked by name only, so they cost no listing context and have no trigger
question) and three model-invoked with committed eval cases.
- `AGENTS.md` 1,223 → 380 lines: rules, a skills index, and a project tree
reduced to one line per entry. The removed recipes are not duplicated.
- `README.md` 811 → 105 lines, split into `docs/{architecture,quality-gate,
test-servers,publishing,docker}.md`, and dropped from `CLAUDE.md`'s auto-load.
- `.github/copilot-instructions.md` and its mirroring rule removed — Copilot
reads `AGENTS.md`.
- `npm run verify:skills` (in `validate`, so in `local:gate` and CI) parses each
SKILL.md the way Claude Code does, requires an explicit invocation mode and
eval cases for model-invoked skills, and budgets the listing.
- `npm run skills:eval` runs the committed trigger cases headless.
Signed-off-by: cliffhall <cliff@futurescale.com>
