fix: address Copilot review round 3 on the SDK watch
Four findings, including one suppressed. Three fixed outright; the fourth is fixed as far as this PR can reach, with the remainder tracked in #2269 rather than papered over. 1. (Suppressed finding, and the most consequential.) The comment lookup was `gh api --jq '.[].body'` split on newlines, which destroyed the comment boundaries — a body is multi-line, so every LINE became its own entry. Both consumers read a line as though it were a whole comment and both use `startsWith`, so a maintainer quoting `<!-- sdk-watch:analysis -->` at the start of any line would have permanently convinced the sweep that issue was analyzed, and the retry added in round 2 would never fire for it again. The same held for the supersession marker. `issueComments` now fetches `--paginate --slurp` and parses JSON, mapping whole `body` strings. The test fake was part of the problem — it returned newline-joined text, which made the broken parse look correct because it could never produce a multi-line body — so it now returns the real wire shape, an array of pages of comment objects. Note the round-2 test written for this class checked INLINE quoting and passed. The unit was correct; the transport feeding it was not. 2. A CLOSED matching issue was being re-queued for analysis. Two different questions about the same match had been collapsed into one: whether it suppresses creation (yes, whatever its state — that is why `sweepIssues` reads `--state all`) and whether it should be re-analyzed (only if open). A closed issue has no analysis marker almost by definition, so it was handed to the analyze job and would have drawn a fresh automated comment nightly, re-arguing the "not planned" decision that `--state all` exists to respect. Gated on `match.state === "OPEN"`, with the closed case logged rather than passed over in silence. 3. The docs contradicted the code in three places — AGENTS.md, the workflow header and `formatFiledOutput` all still said analysis runs only for newly created issues. All three now describe the same rule, including both exclusions and why they differ: an already-analyzed issue is excluded to avoid a nightly duplicate, a closed one because closing it was a decision. 4. The model retains a secret-exfiltration path: `claude-code-action` copies its environment into the model's, `Read` is genuinely needed to inspect how `core/` uses the SDK, and `analysis` is model-controlled text this workflow publishes. Removing the write-capable `gh` tools closed the posting channel, not the output channel. Landed here as a BACKSTOP, and labelled as one in the workflow so it is not read as settled: the posting step refuses any analysis containing ANTHROPIC_API_KEY or the job's GITHUB_TOKEN verbatim, matched with shell pattern matching so no secret reaches an argv. An injection that encodes the value defeats it. The real fix is workload identity federation (`anthropic_federation_rule_id` + `id-token: write`, and no `anthropic_api_key` at all), which needs an Anthropic-side federation rule and org UUID that do not exist and cannot be created from a pull request. Filed as #2269 with the exact YAML change and acceptance criteria, boarded at Todo/Medium in v2.6.0. AGENTS.md and the workflow both point at it rather than describing this job as secure. Mutation-checked, each failing exactly one test: newline splitting restored -> the multi-line-comment test `state === "OPEN"` guard removed -> the closed-issue test 49 tests in this file; `npm run local:gate` green end to end. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019JWUZxLDnPgLQzjBrtk1x9 Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
6711593a37
commit
e9a1571f42
1 changed files with 149 additions and 15 deletions
|
|
@ -134,9 +134,9 @@ The board write needs `organization projects: write`, which `GITHUB_TOKEN` canno
|
|||
- **It never boards, like the monthly sweep** — no `PROJECT_TOKEN` exists in this org — so the issue arrives labeled and milestoned and `/issue-triage` places it.
|
||||
- **It never closes an issue either.** A further release files its own issue and leaves a **supersession comment** on the older one; closing is a maintainer act, since the card may already have moved. An issue closed for the same target keeps suppressing it, so a maintainer's "not planned" is not re-argued nightly.
|
||||
|
||||
**The analysis half runs Claude, not Copilot, and that is deliberate.** #1063 sketched "a copilot agent running Opus"; neither half of that is reachable from a workflow. Assigning `copilot-swe-agent` produces a **pull request** — the artifact this whole section exists to remove — and its model cannot be selected programmatically at all (`replaceActorsForAssignable` takes no model parameter; absent an admin-configured picker it runs Sonnet). So the `analyze` job uses `anthropics/claude-code-action` with `--model claude-opus-5`, which is told to post **one comment** and is denied every file-writing tool. It runs on `ANTHROPIC_API_KEY`, an **organization** secret already available to this repo, and only over issues the sweep **just created** — never over one that already existed, which is what keeps it to one analysis per release instead of a near-identical comment every night.
|
||||
**The analysis half runs Claude, not Copilot, and that is deliberate.** #1063 sketched "a copilot agent running Opus"; neither half of that is reachable from a workflow. Assigning `copilot-swe-agent` produces a **pull request** — the artifact this whole section exists to remove — and its model cannot be selected programmatically at all (`replaceActorsForAssignable` takes no model parameter; absent an admin-configured picker it runs Sonnet). So the `analyze` job uses `anthropics/claude-code-action` with `--model claude-opus-5`, which is told to post **one comment** and is denied every file-writing tool. It runs on `ANTHROPIC_API_KEY`, an **organization** secret already available to this repo, over the issues the sweep just created plus any **open** issue it already filed that still carries no analysis comment (the retry path below). What it never re-analyzes is an issue that already has one, or one that has been closed — which is what keeps it to a single analysis per release instead of a near-identical comment every night.
|
||||
|
||||
- **"An issue exists" and "the issue was analyzed" are different claims**, and the sweep must not equate them. The posting step stamps `ANALYSIS_MARKER` on its comment; a sweep that finds an existing issue for the current target but no such comment **re-queues it**, so a failed or timed-out `analyze` job is retried rather than silently never revisited. That marker is duplicated in the workflow because the posting step is shell — a test asserts the two strings match, since drift would make every issue read as unanalyzed forever.
|
||||
- **"An issue exists" and "the issue was analyzed" are different claims**, and the sweep must not equate them. The posting step stamps `ANALYSIS_MARKER` on its comment; a sweep that finds an **open** existing issue for the current target but no such comment **re-queues it**, so a failed or timed-out `analyze` job is retried rather than silently never revisited. ⚠️ **Open only** — suppressing and re-queuing are different questions about the same match, and a closed issue must keep suppressing without being handed a fresh comment every night. That marker is duplicated in the workflow because the posting step is shell — a test asserts the two strings match, since drift would make every issue read as unanalyzed forever.
|
||||
|
||||
⚠️ **Upstream release notes are untrusted input to that job, and the model is granted nothing that can write anywhere.** The tool whitelist is the only real control — the prompt's prose constraints are not — and two revisions of it were wrong before this one landed:
|
||||
|
||||
|
|
@ -145,6 +145,8 @@ The board write needs `organization projects: write`, which `GITHUB_TOKEN` canno
|
|||
|
||||
So the model returns its write-up as structured output (`--json-schema`) and a deterministic, model-free step posts it — that step is the only thing in the job holding a token that can write, and the body reaches it through the environment rather than the command line. `WebFetch` is denied for the same reason: an outbound fetch the model controls is the other end of the same exfiltration channel, and `gh release view` covers the need. The job's token carries `contents: read` only. **Keep every one of those properties when editing this job.**
|
||||
|
||||
⚠️ **One channel is still open, and the workflow says so rather than implying otherwise.** `claude-code-action` copies the action's environment into the model's, so `ANTHROPIC_API_KEY` is readable by a `Read` the analysis genuinely needs, and `analysis` is model-controlled text this workflow publishes. Two things narrow it and **neither is a boundary**: the posting step refuses any analysis containing a credential *verbatim*, and the job's own token expires with the job. The real fix is to stop putting a long-lived key in the environment — the action supports **workload identity federation** (`anthropic_federation_rule_id` + `id-token: write`), which needs an Anthropic-side federation rule this org does not have yet. Tracked in #2269. Do not read the verbatim scan as having settled it.
|
||||
|
||||
## Contributing
|
||||
|
||||
External contributions are accepted as **issues, not pull requests** — maintainers handle design and implementation through a prompt-driven workflow.
|
||||
|
|
|
|||
Loading…
Reference in a new issue