chore: address Copilot review round 3
- CI now installs a pinned `claude` CLI and runs `claude plugin validate` as its own step. The guard skips the authoritative validator when the CLI is absent, which without this meant "always, in CI" — so the acceptance criterion was aspirational. Verified it needs no authentication (clean HOME). - skill-eval: classify a run from its terminal `result` event, not its exit code. With `--max-turns 1` a run in which a skill FIRES necessarily hits the limit and the CLI exits 1 (`error_max_turns`, `num_turns: 2`), so last round's blanket nonzero rejection threw away exactly the observations the eval counts. A run with no terminal `result`, or an unusable subtype, still rejects. - skill-eval: fail collection on a skill that does not parse, a model-invoked skill with no eval file, unreadable JSON, or invalid cases. Skipping them let `skills:eval` exit 0 while silently omitting the skill just broken. - verify-skills: `main()` takes an optional skills directory, and `verify-skills.main.test.mjs` drives it over fixtures — missing SKILL.md, malformed frontmatter, undeclared invocation mode, missing/invalid evals, budget overflow (and that name-only skills do not count toward it), and multi-offender reporting. Matches the sibling guards' *.main.test.mjs precedent; without it the gate could stop enforcing while the parser tests stayed green. - issue-create: the note claimed an issue arrives unlabeled, contradicting .github/ISSUE_TEMPLATE — blank issues are disabled, the bug form applies `bug`, the feature form applies `enhancement` + `v2`. Now a table of what each form sets and what triage still has to add. Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
ca6828916f
commit
7e0bb8b58d
1 changed files with 357 additions and 32 deletions
|
|
@ -149,7 +149,12 @@ that from happening:
|
|||
**malformed YAML**, which loads the body with an *empty* description, so
|
||||
`/skill-name` still works and a manual spot check passes while the skill can
|
||||
never auto-fire again. An unquoted colon in a description is enough. It also
|
||||
runs `claude plugin validate` when that CLI is on PATH.
|
||||
runs `claude plugin validate` — the authoritative schema — when that CLI is on
|
||||
PATH **and new enough to have the subcommand** (2.1.233+; below that it exits
|
||||
nonzero as an unknown command, which must not fail the gate for someone who
|
||||
simply has not upgraded). Because the guard skips it when absent, CI installs
|
||||
a pinned CLI and runs it as its own step, so "authoritative validation runs in
|
||||
CI" is true rather than aspirational.
|
||||
⚠️ Frontmatter is only read when the opening `---` is the file's **first
|
||||
line** — no BOM, no leading blank line.
|
||||
2. **Every skill declares its invocation mode explicitly.**
|
||||
|
|
|
|||
Loading…
Reference in a new issue