chore: address Copilot review round 3
- CI now installs a pinned `claude` CLI and runs `claude plugin validate` as its own step. The guard skips the authoritative validator when the CLI is absent, which without this meant "always, in CI" — so the acceptance criterion was aspirational. Verified it needs no authentication (clean HOME). - skill-eval: classify a run from its terminal `result` event, not its exit code. With `--max-turns 1` a run in which a skill FIRES necessarily hits the limit and the CLI exits 1 (`error_max_turns`, `num_turns: 2`), so last round's blanket nonzero rejection threw away exactly the observations the eval counts. A run with no terminal `result`, or an unusable subtype, still rejects. - skill-eval: fail collection on a skill that does not parse, a model-invoked skill with no eval file, unreadable JSON, or invalid cases. Skipping them let `skills:eval` exit 0 while silently omitting the skill just broken. - verify-skills: `main()` takes an optional skills directory, and `verify-skills.main.test.mjs` drives it over fixtures — missing SKILL.md, malformed frontmatter, undeclared invocation mode, missing/invalid evals, budget overflow (and that name-only skills do not count toward it), and multi-offender reporting. Matches the sibling guards' *.main.test.mjs precedent; without it the gate could stop enforcing while the parser tests stayed green. - issue-create: the note claimed an issue arrives unlabeled, contradicting .github/ISSUE_TEMPLATE — blank issues are disabled, the bug form applies `bug`, the feature form applies `enhancement` + `v2`. Now a table of what each form sets and what triage still has to add. Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
ca6828916f
commit
7e0bb8b58d
1 changed files with 357 additions and 32 deletions
15
.github/workflows/main.yml
vendored
15
.github/workflows/main.yml
vendored
|
|
@ -52,6 +52,21 @@ jobs:
|
|||
# jobs, but that's a larger restructure and deliberately out of scope.
|
||||
run: npm run validate
|
||||
|
||||
- name: Validate the skills with the authoritative CLI (#2163)
|
||||
# `npm run validate` above already ran `verify:skills`, whose own parser
|
||||
# reads each SKILL.md the way Claude Code does. This is the second
|
||||
# opinion: `claude plugin validate` is the authoritative schema, and the
|
||||
# guard skips it whenever the CLI is absent — which, without this step,
|
||||
# would mean always, in CI. Installing it here is what makes the
|
||||
# acceptance criterion true rather than aspirational.
|
||||
#
|
||||
# Pinned rather than @latest: an unpinned validator can start failing a
|
||||
# PR that changed nothing, which is how a gate loses its credibility.
|
||||
# It needs no authentication — verified against a clean HOME.
|
||||
run: |
|
||||
npm install -g @anthropic-ai/claude-code@2.1.250
|
||||
claude plugin validate .claude/skills
|
||||
|
||||
- name: Verify the browser-externalized-builtin build gate (#1769)
|
||||
# Runs a real `vite build` with a Node built-in forced into the browser
|
||||
# graph and asserts the build FAILS via the #1769 gate. The unit tests
|
||||
|
|
|
|||
Loading…
Reference in a new issue