chore: address Copilot review round 3

- CI now installs a pinned `claude` CLI and runs `claude plugin validate` as its
  own step. The guard skips the authoritative validator when the CLI is absent,
  which without this meant "always, in CI" — so the acceptance criterion was
  aspirational. Verified it needs no authentication (clean HOME).
- skill-eval: classify a run from its terminal `result` event, not its exit
  code. With `--max-turns 1` a run in which a skill FIRES necessarily hits the
  limit and the CLI exits 1 (`error_max_turns`, `num_turns: 2`), so last round's
  blanket nonzero rejection threw away exactly the observations the eval counts.
  A run with no terminal `result`, or an unusable subtype, still rejects.
- skill-eval: fail collection on a skill that does not parse, a model-invoked
  skill with no eval file, unreadable JSON, or invalid cases. Skipping them let
  `skills:eval` exit 0 while silently omitting the skill just broken.
- verify-skills: `main()` takes an optional skills directory, and
  `verify-skills.main.test.mjs` drives it over fixtures — missing SKILL.md,
  malformed frontmatter, undeclared invocation mode, missing/invalid evals,
  budget overflow (and that name-only skills do not count toward it), and
  multi-offender reporting. Matches the sibling guards' *.main.test.mjs
  precedent; without it the gate could stop enforcing while the parser tests
  stayed green.
- issue-create: the note claimed an issue arrives unlabeled, contradicting
  .github/ISSUE_TEMPLATE — blank issues are disabled, the bug form applies
  `bug`, the feature form applies `enhancement` + `v2`. Now a table of what each
  form sets and what triage still has to add.

Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-08-29 01:26:19 -04:00
commit 7e0bb8b58d

View file

@ -149,7 +149,12 @@ that from happening:
**malformed YAML**, which loads the body with an *empty* description, so
`/skill-name` still works and a manual spot check passes while the skill can
never auto-fire again. An unquoted colon in a description is enough. It also
runs `claude plugin validate` when that CLI is on PATH.
runs `claude plugin validate` — the authoritative schema — when that CLI is on
PATH **and new enough to have the subcommand** (2.1.233+; below that it exits
nonzero as an unknown command, which must not fail the gate for someone who
simply has not upgraded). Because the guard skips it when absent, CI installs
a pinned CLI and runs it as its own step, so "authoritative validation runs in
CI" is true rather than aspirational.
⚠️ Frontmatter is only read when the opening `---` is the file's **first
line** — no BOM, no leading blank line.
2. **Every skill declares its invocation mode explicitly.**