chore: make issue-create and issue-triage model-invocable
`release` is now the only skill reachable by name alone, which is right: nobody cuts a release by implication, whereas "file an issue for this" and "triage the new issues" are both ordinary prose instructions. Eval cases committed for both; each measured 5/5 at RUNS=3. One initial case was rebuilt rather than tuned: "file an issue for the flaky Storybook test" scored 0% because its SUBJECT was unresolved — the model went off grepping for which test was flaky instead of reaching for the filing procedure. A trigger case must not make the model investigate before it can act. The full eval across all nine model-invoked skills reports a cost worth having in writing, and it contradicts the justification given for the previous flip. Every negative case still passes 100%, so there is no over-triggering — but the skills that were already model-invoked measurably LOST trigger rate: `project-structure` fell from 100% to 0% on two cases (n=4) and `testing` from 3/5 to 2/5, while all six newly flipped skills measured 100%. So the ceiling is attention, not characters: the listing sat at 2.8k of 4k throughout, and "the budget is not tight" was the wrong reason to think flipping was free. AGENTS.md now records this, and requires the FULL eval to be re-run after any flip rather than just the new skill's own cases. Two of project-structure's three cases are also the invalid-case pattern this file already documents — `core/auth/` is described in the AGENTS.md tree, so a model answering "which package owns the secret storage backends" without a skill is correct rather than failing. `testing`'s cases are not, and it was the weaker of the two before this change as well. Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
f8c1d0c93f
commit
26d70e00e7
1 changed files with 62 additions and 8 deletions
20
AGENTS.md
20
AGENTS.md
|
|
@ -19,8 +19,8 @@ users invoke them by name.
|
|||
| [`local-dev`](.claude/skills/local-dev/SKILL.md) | Install and run each client; the `@inspector/core` alias; and the **reasoning** behind Dependency placement below — what each rule defends against and how to tell you have hit one (the rules themselves stay here) | Model-invoked, or `/local-dev` |
|
||||
| [`project-structure`](.claude/skills/project-structure/SKILL.md) | Which client owns which surface, what is in `core/`, where a new file belongs | Model-invoked only |
|
||||
| [`testing`](.claude/skills/testing/SKILL.md) | Where a test file goes, which command runs it, the tiers, clearing the coverage gate, `renderWithMantine` | Model-invoked, or `/testing` |
|
||||
| [`issue-create`](.claude/skills/issue-create/SKILL.md) | The five-step create flow: version label, type label, milestone, board card, Status + Priority | `/issue-create` |
|
||||
| [`issue-triage`](.claude/skills/issue-triage/SKILL.md) | The two-pass sweep of unboarded issues, the priority rubric and its score comment, the board audit | `/issue-triage` |
|
||||
| [`issue-create`](.claude/skills/issue-create/SKILL.md) | The five-step create flow: version label, type label, milestone, board card, Status + Priority | Model-invoked, or `/issue-create` |
|
||||
| [`issue-triage`](.claude/skills/issue-triage/SKILL.md) | The two-pass sweep of unboarded issues, the priority rubric and its score comment, the board audit | Model-invoked, or `/issue-triage` |
|
||||
| [`board-ops`](.claude/skills/board-ops/SKILL.md) | `gh project` recipes and the field/option IDs for boards #28 and #11; the option-deletion hazard and its recovery | Model-invoked, or `/board-ops` |
|
||||
| [`pr-flow`](.claude/skills/pr-flow/SKILL.md) | Branch naming, DCO signoff, screenshots, opening the PR, requesting a Copilot review, responding, closing out | Model-invoked, or `/pr-flow` |
|
||||
| [`pre-push-gate`](.claude/skills/pre-push-gate/SKILL.md) | Running `npm run local:gate` and diagnosing a failing stage | Model-invoked, or `/pre-push-gate` |
|
||||
|
|
@ -174,11 +174,21 @@ that from happening:
|
|||
skill at all: it is absent from the listing and the Skill tool refuses it. The
|
||||
costs are asymmetric — a spurious load costs ~250 characters, a missed one
|
||||
costs a wrong base branch or an unsigned commit — and the budget is not tight
|
||||
(all ten skills would total ~2.8k of 4k). Reserve `true` for a procedure that
|
||||
is genuinely only ever started deliberately, as `/release` is.
|
||||
(nine of the ten are model-invoked today and total ~2.8k of 4k). Reserve
|
||||
`true` for a procedure that is genuinely only ever started deliberately —
|
||||
`release` is the only one left, because nobody cuts a release by implication.
|
||||
⚠️ **A `true` skill cannot be reached by another skill either.** If a
|
||||
model-invocable skill says "see `/board-ops`", that pointer is a dead end for
|
||||
the model unless `board-ops` is model-invocable too.
|
||||
⚠️ **Flipping is not free, and the listing budget is not what costs.** Going
|
||||
from three model-invoked skills to nine measurably *lowered* the trigger rate
|
||||
of the ones already there: `project-structure` fell from 100% to 0% on two
|
||||
cases (n=4) and `testing` from 3/5 to 2/5, while the six new skills all
|
||||
measured 100% and every negative case stayed clean. So the ceiling is
|
||||
attention, not characters — we were at 2.8k of a 4k budget throughout. Adding
|
||||
a skill therefore has a cost paid by the *existing* ones, which only
|
||||
`skills:eval` can see. **Re-run the full eval after any flip**, not just the
|
||||
new skill's own cases.
|
||||
⚠️ **`paths` is not a free win.** It looks like the deterministic option, and
|
||||
it does gate loading to matching files — but measured against the `testing`
|
||||
skill's own eval cases, adding it roughly **halved** the rate at which the same
|
||||
|
|
@ -212,7 +222,7 @@ that from happening:
|
|||
overflows, and drops the least-invoked entries **first** — which are exactly the
|
||||
model-invoked skills that must fire on their own. `verify:skills` prints the
|
||||
current cost against the budget recorded in `scripts/lib/skill-manifest.mjs`
|
||||
(2,256/4,000 characters as of this writing) and fails when it is exceeded. Raise
|
||||
(2,834/4,000 characters as of this writing) and fails when it is exceeded. Raise
|
||||
the budget deliberately, or tighten a description; each entry is capped at 1,536
|
||||
characters regardless, so **put the key use case first**.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue