chore: make issue-create and issue-triage model-invocable
`release` is now the only skill reachable by name alone, which is right: nobody cuts a release by implication, whereas "file an issue for this" and "triage the new issues" are both ordinary prose instructions. Eval cases committed for both; each measured 5/5 at RUNS=3. One initial case was rebuilt rather than tuned: "file an issue for the flaky Storybook test" scored 0% because its SUBJECT was unresolved — the model went off grepping for which test was flaky instead of reaching for the filing procedure. A trigger case must not make the model investigate before it can act. The full eval across all nine model-invoked skills reports a cost worth having in writing, and it contradicts the justification given for the previous flip. Every negative case still passes 100%, so there is no over-triggering — but the skills that were already model-invoked measurably LOST trigger rate: `project-structure` fell from 100% to 0% on two cases (n=4) and `testing` from 3/5 to 2/5, while all six newly flipped skills measured 100%. So the ceiling is attention, not characters: the listing sat at 2.8k of 4k throughout, and "the budget is not tight" was the wrong reason to think flipping was free. AGENTS.md now records this, and requires the FULL eval to be re-run after any flip rather than just the new skill's own cases. Two of project-structure's three cases are also the invalid-case pattern this file already documents — `core/auth/` is described in the AGENTS.md tree, so a model answering "which package owns the secret storage backends" without a skill is correct rather than failing. `testing`'s cases are not, and it was the weaker of the two before this change as well. Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
f8c1d0c93f
commit
26d70e00e7
4 changed files with 62 additions and 8 deletions
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
name: issue-create
|
||||
description: Create a tracked Inspector issue end to end — version label, type label, milestone, board card, Status and Priority. Use when filing a new issue, or when work discovered mid-task needs an issue of its own.
|
||||
disable-model-invocation: true
|
||||
description: Create a tracked issue in this repo end to end — version label, type label, milestone, board card, Status and Priority. Use when asked to file, create, open or raise an issue; when picking its labels or its milestone; when checking whether one already exists; or when work discovered mid-task needs an issue of its own.
|
||||
disable-model-invocation: false
|
||||
---
|
||||
|
||||
# Creating an issue
|
||||
|
|
|
|||
22
.claude/skills/issue-create/evals/evals.json
Normal file
22
.claude/skills/issue-create/evals/evals.json
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
[
|
||||
{
|
||||
"prompt": "Create an issue titled 'TUI crashes on an empty catalog' and set it up the way this repo wants.",
|
||||
"expect": "issue-create"
|
||||
},
|
||||
{
|
||||
"prompt": "Which milestone should a new issue go in, and how do I find the current one?",
|
||||
"expect": "issue-create"
|
||||
},
|
||||
{
|
||||
"prompt": "Before I file this, how do I check nobody has reported it already?",
|
||||
"expect": "issue-create"
|
||||
},
|
||||
{
|
||||
"prompt": "What is the chemical symbol for potassium?",
|
||||
"expect": null
|
||||
},
|
||||
{
|
||||
"prompt": "Reverse this string: stressed",
|
||||
"expect": null
|
||||
}
|
||||
]
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
name: issue-triage
|
||||
description: Triage unboarded Inspector issues — the two-pass sweep, the priority rubric and its score comment, and the board audit that checks the whole board against its invariants. Use for "triage new issues" or when scoring an issue's Priority.
|
||||
disable-model-invocation: true
|
||||
disable-model-invocation: false
|
||||
---
|
||||
|
||||
# Triaging issues
|
||||
|
|
|
|||
22
.claude/skills/issue-triage/evals/evals.json
Normal file
22
.claude/skills/issue-triage/evals/evals.json
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
[
|
||||
{
|
||||
"prompt": "Triage the new issues.",
|
||||
"expect": "issue-triage"
|
||||
},
|
||||
{
|
||||
"prompt": "What Priority should issue 2189 be given?",
|
||||
"expect": "issue-triage"
|
||||
},
|
||||
{
|
||||
"prompt": "Audit the project board for anything that has drifted.",
|
||||
"expect": "issue-triage"
|
||||
},
|
||||
{
|
||||
"prompt": "Who wrote the novel Moby-Dick?",
|
||||
"expect": null
|
||||
},
|
||||
{
|
||||
"prompt": "Convert 45 degrees Celsius to Fahrenheit.",
|
||||
"expect": null
|
||||
}
|
||||
]
|
||||
Loading…
Reference in a new issue