chore: address Copilot review round 3

- CI now installs a pinned `claude` CLI and runs `claude plugin validate` as its
  own step. The guard skips the authoritative validator when the CLI is absent,
  which without this meant "always, in CI" — so the acceptance criterion was
  aspirational. Verified it needs no authentication (clean HOME).
- skill-eval: classify a run from its terminal `result` event, not its exit
  code. With `--max-turns 1` a run in which a skill FIRES necessarily hits the
  limit and the CLI exits 1 (`error_max_turns`, `num_turns: 2`), so last round's
  blanket nonzero rejection threw away exactly the observations the eval counts.
  A run with no terminal `result`, or an unusable subtype, still rejects.
- skill-eval: fail collection on a skill that does not parse, a model-invoked
  skill with no eval file, unreadable JSON, or invalid cases. Skipping them let
  `skills:eval` exit 0 while silently omitting the skill just broken.
- verify-skills: `main()` takes an optional skills directory, and
  `verify-skills.main.test.mjs` drives it over fixtures — missing SKILL.md,
  malformed frontmatter, undeclared invocation mode, missing/invalid evals,
  budget overflow (and that name-only skills do not count toward it), and
  multi-offender reporting. Matches the sibling guards' *.main.test.mjs
  precedent; without it the gate could stop enforcing while the parser tests
  stayed green.
- issue-create: the note claimed an issue arrives unlabeled, contradicting
  .github/ISSUE_TEMPLATE — blank issues are disabled, the bug form applies
  `bug`, the feature form applies `enhancement` + `v2`. Now a table of what each
  form sets and what triage still has to add.

Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-08-29 01:26:19 -04:00
commit 7e0bb8b58d

View file

@ -41,7 +41,7 @@ import {
} from "./lib/skill-manifest.mjs";
const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
const SKILLS_DIR = path.join(ROOT, ".claude", "skills");
const DEFAULT_SKILLS_DIR = path.join(ROOT, ".claude", "skills");
/** Directory names under `.claude/skills` that contain a SKILL.md. */
export function skillDirs(
@ -54,7 +54,12 @@ export function skillDirs(
.sort();
}
function main() {
function main(argv = process.argv.slice(2)) {
// An explicit directory is the seam the fixture tests drive; without it this
// guard could only ever be exercised against the repo's own (green) skills,
// so it could stop enforcing anything while every unit test stayed green.
const override = argv[0];
const SKILLS_DIR = override ? path.resolve(override) : DEFAULT_SKILLS_DIR;
const failures = [];
if (!existsSync(SKILLS_DIR)) {
@ -124,6 +129,8 @@ function main() {
`(${modelInvoked.join(", ")}); listing ${cost}/${LISTING_BUDGET} chars.`,
);
if (override) return;
// The authoritative parse, when it is available — strictly a bonus. An older
// CLI does not have `plugin validate` and exits nonzero on it as an unknown
// command, which would fail the MANDATORY gate for a contributor whose only