From f95abb27159de5643e19b068ef07d2610a0b4cea Mon Sep 17 00:00:00 2001 From: cliffhall Date: Sun, 30 Aug 2026 00:53:07 -0400 Subject: [PATCH] chore: tighten the testing skill's description MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured before and after at the same sample size (n=4), with the six other model-invoked skills unchanged: which command runs the TUI suite 25% → 100% asserting on a Mantine transition 50% → 100% which tier spawns the built binary 25% → 100% The lever is the description's SHAPE, not its length: leading with the actions and then enumerating concrete situations ("Use when … ; when … ; when …") beats a noun-phrase list of contents ending in a generic "use for any question about …". The same edit took `pre-push-gate` from 3/5 to 5/5 earlier. Re-ran the full eval to check the change did not simply move attention off another skill, which is the failure the previous commit recorded: 40/46, up from 38/46, and the failing set is strictly smaller — `testing` went from three failures to one, nothing else moved. That remaining case measured 100% in isolation and 67% in the full run, which is sampling variance rather than a finding. AGENTS.md now extends the re-run rule to cover a description edit, not just a flip, and records the shape that works. Signed-off-by: cliffhall --- .claude/skills/testing/SKILL.md | 2 +- AGENTS.md | 9 +++++++-- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/.claude/skills/testing/SKILL.md b/.claude/skills/testing/SKILL.md index 0afcbea0..dbbb4438 100644 --- a/.claude/skills/testing/SKILL.md +++ b/.claude/skills/testing/SKILL.md @@ -1,6 +1,6 @@ --- name: testing -description: Testing procedures for this repo — which npm command runs which suite (web unit, web integration, Storybook, cli, tui, launcher, scripts), the test tiers from unit through smoke and pack:verify, where a new test file goes (side-by-side vs src/test/ vs __tests__/), diagnosing the per-file 90% coverage gate, and the renderWithMantine / renderWithMantineTransitions rules for React and Mantine tests. Use for any question about running, placing, or fixing a test here. +description: Run, place and fix tests in this repo. Use when choosing which npm command runs a given suite (web unit, web integration, Storybook, cli, tui, launcher, scripts); when deciding where a new test file belongs — beside its source, under src/test/, or in a client's __tests__/; when a per-file coverage check fails or a v8 ignore is in question; when asking which test tier spawns the built binary rather than importing it; or when rendering, mounting or asserting on Mantine components and their transitions in a test. disable-model-invocation: false --- diff --git a/AGENTS.md b/AGENTS.md index f1ba56a1..a733ee4b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -187,8 +187,13 @@ that from happening: measured 100% and every negative case stayed clean. So the ceiling is attention, not characters — we were at 2.8k of a 4k budget throughout. Adding a skill therefore has a cost paid by the *existing* ones, which only - `skills:eval` can see. **Re-run the full eval after any flip**, not just the - new skill's own cases. + `skills:eval` can see. **Re-run the full eval after any flip _or description + edit_**, not just the changed skill's own cases. + The lever that works is the description's *shape*. Leading with the actions + and then enumerating concrete situations ("Use when … ; when … ; when …") + beats a noun-phrase list of contents: it took `pre-push-gate` from 3/5 to 5/5 + and `testing`'s three cases from 25/50/25% to 100% each (n=4), displacing + nothing else. ⚠️ **`paths` is not a free win.** It looks like the deterministic option, and it does gate loading to matching files — but measured against the `testing` skill's own eval cases, adding it roughly **halved** the rate at which the same