Four findings. The first ends a pattern rather than patching it again.
1. The tool grants did not enforce the boundary they claimed, for two
reasons. `--allowedTools` only PRE-APPROVES; `--tools` is what
restricts availability — a distinction this repo had already learned
and written down in `scripts/skill-eval.mjs`, and which I failed to
apply here. And a `Bash(...)` rule can match a COMPOUND command
(`gh release view … && curl …`), so the denials could be walked past.
That is the fourth grant in this review whose real surface was wider
than it looked, after `gh api`, `gh issue comment --body-file` and
`npm view --registry`. Each fix scoped the command and left something
else open, so this one removes the capability instead: THE MODEL NOW
HAS NO BASH AT ALL.
Upstream release notes are prefetched by a deterministic `run:` step
into `upstream-release-notes.md` (the 30 most recent releases via
`gh api`), and the prompt points the model at that file and tells it
to say so plainly if the range it needs is absent. Verified against
the real ext-apps releases API. The model's tools are now
`--tools "Read,Grep,Glob"` for availability, the same three in
`--allowedTools` so no headless run stalls on an unanswerable prompt,
and `--disallowedTools` naming Bash and the write/network tools as a
third layer — the same three-part shape `skill-eval.mjs` uses.
2. The security documentation asserted something false. `sweep` inherits
`issues: write` because filing issues is its purpose, so "the ONLY job
holding a token that can write" was never true. Both AGENTS.md and the
workflow comment now state the actual invariant: NO MODEL RUNS IN A
WRITE-CAPABLE JOB. Two of three jobs write; neither runs a model.
The test changed with it, and this is the substantive half: it was
asserting `analyze.permissions === {contents: read}`, which is one
job's slice and would have passed happily if someone added a model
step to `post`. It now iterates EVERY job and, for each that runs
`claude-code-action`, asserts no write scope and no posting step —
falling back to the inherited top-level permissions, so an omitted
`permissions:` block fails rather than defaulting silently.
3. (Suppressed.) The generated checklist told maintainers to edit
`package.json` unconditionally, which is wrong for the one case this
script models separately: `ext-apps` is `^1.7.4`, which already admits
a 1.8.0 target, so only the lockfile moves. `needsManifestEdit` now
selects the checklist items by asking whether each behind row's
declared range admits the target — exact pins need the edit, satisfied
ranges do not, and an unparseable declaration errs toward asking for a
look.
4. (Suppressed.) The `post` job comment carried the same false claim as
finding 2; corrected in the same change.
Mutation-checked, each failing a different set:
needsManifestEdit -> true -> its unit test + the ext-apps body test
--tools line deleted -> the tool-availability test
61 tests in this file; `npm run local:gate` green end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019JWUZxLDnPgLQzjBrtk1x9
Signed-off-by: cliffhall <cliff@futurescale.com>
cbf84ceae9