* chore: land the #2247 round-9 review findings on v2/main
Copilot filed a ninth review round on #2247 after it had merged, so none of
it reached v2/main. All five findings are addressed here.
1. The testing -> test-servers hand-off delivered the wrong half of the
procedure. test-servers documented only the two-process manual path
(server-composable.js --config <name>.json plus a second Inspector
process), but an integration test builds the server in-process from the
@modelcontextprotocol/inspector-test-server API and owns its lifecycle.
Adds a "Two ways to use a fixture — pick the right one first" section
with the in-process recipe, the four mechanics that bite (one barrel,
start/stop in afterEach, read started.url, era is a constructor option),
and a pointer to the reference test; renames "Run one" to "Run one by
hand (two processes)" so the manual path names itself. testing now says
in the connecting bullet that an automated test does NOT spawn.
2. testing's heading asked "does the test need a real server?", which its
own hand-rolled-server exception contradicts. It now asks whether the
test uses a test-servers/ fixture.
3. "imports" was too broad a dependency test — test-server-scope.test.ts
imports the package to unit-test scope middleware as a pure function and
never starts a server. The condition is now "starts, spawns, or
configures", with that test named as the counterexample.
4. docs/skill-authoring.md attributed the 33% -> 100% rise to imperative
wording and placement independently. The #2247 edit changed both at
once, so only the combination is established; the split into two levers
is now labelled the plausible mechanism, not a measured result.
5. The CHAIN_THRESHOLD comment generalised "a well-shaped pointer clears
0.8" from one two-case run of that same multi-variable edit. It now
reads as an existence proof about the ceiling, not a property of
pointers in general.
Also records the headline lesson where it can be found again: a green
hand-off number proves skill B loaded, never that B answered the question —
in AGENTS.md's chain-case rules and in the skill-authoring checklist.
Docs, skill bodies and one comment only; no runtime code changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018zwYWanPhvgu6wiKoeGg4N
Signed-off-by: cliffhall <cliff@futurescale.com>
* fix: address Copilot review round 1 on #2264
The new section claimed two shapes and there are three — the automated
stdio path was missing, and it is the repo's established pattern
(getTestMcpServerCommand() into a stdio transport, used by five web
integration files and the whole CLI suite). Telling callers an automated
test "does not spawn" told them not to use it.
- test-servers: "Two ways" -> "Three ways", with a table and a subsection
for the spawned-stdio shape — the transport owns the child (disconnect,
not stop()), no config can be selected because that entry point runs the
stdio default, and it is still the build. Names a reference test for it.
- test-servers: the build is no longer scoped to the manual path. All three
shapes resolve test-servers/build/ — two through the barrel alias, one by
running the emitted .js — so the staleness hazard is called out as
applying to every shape, and `Build first` no longer opens by claiming
the only reason is subprocess spawning.
- testing: the "does not spawn" instruction is replaced by the two
automated shapes and when to pick each. What they share, and what the
hand-off actually needed to say, is that neither picks a showcase config.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018zwYWanPhvgu6wiKoeGg4N
Signed-off-by: cliffhall <cliff@futurescale.com>
* fix: address Copilot review round 2 on #2264
Round 1 fixed the missing stdio shape but kept the wrong axis. Cutting
the taxonomy by "automated vs by hand" excluded the config-driven smokes
— smoke-web-elicitation.mjs spawns server-composable.js --config
directly, and lib/mcp-app-flow.mjs does it for smoke:web:app,
smoke:web:tabs and pack:verify. Those are automated AND need every bit
of the config and protocol-era guidance a person at two terminals does,
so the skill was routing their authors away from it.
- test-servers: the cut is now how the SERVER IS STOOD UP, stated as
such. Third shape is "spawned composable HTTP", whose consumers are
the smokes, pack:verify, and a person by hand — the config/era
guidance belongs to the shape, whoever starts it. Table gains a
Consumers column and loses the Who column that caused this.
- test-servers: the stale "an automated test builds the server
in-process instead" line under `Run one by hand` is gone (suppressed
comment); that heading is now the same shape driven by a person.
- testing: routes by what is driving — integration/CLI test to
in-process HTTP, stdio-is-the-point to spawned stdio, and a smoke to
the composable-config path with the era guidance it actually needs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018zwYWanPhvgu6wiKoeGg4N
Signed-off-by: cliffhall <cliff@futurescale.com>
* fix: address Copilot review round 3 on #2264
Round 2 replaced one wrong axis with a narrower one: it routed "a smoke"
wholesale to the config-driven composable path, and smoke:cli does not
use it. smoke-cli.mjs starts an in-process createTestServerHttp for the
header round-trip (so it can read back what the CLI sent) and writes the
built stdio entry into a --catalog for the connect checks; smoke:tui uses
only the latter. Neither picks a config.
- test-servers: consumers filled in per row — in-process HTTP gains CLI
tests and smoke:cli, spawned stdio gains smoke:cli and smoke:tui, and
the composable row is narrowed to the WEB smokes. A warning states the
point directly: "a smoke" is not a shape, pick by the row.
- test-servers: the stdio shape now covers naming the built entry in a
catalog without connecting, which is what those two smokes do.
- testing: same narrowing, plus the reason spelled out so the word
"smoke" stops being a routing key.
- testing: the dependency criterion said "starts, spawns, or configures",
which excluded the build-only smoke:tui case documented directly above
it — the skill both required and rejected the hand-off for it
(suppressed comment). It now reads "starts, spawns, configures, or
hands a built entry to the subject under test".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018zwYWanPhvgu6wiKoeGg4N
Signed-off-by: cliffhall <cliff@futurescale.com>
---------
Signed-off-by: cliffhall <cliff@futurescale.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The release skill described a v2 release as three steps and left four
things a release actually depends on implicit or absent: that it is two
pull requests against two different bases, the production smoke of the
release candidate, the ledger artifact the maintainers review, and where
a smoke finding gets fixed. It also gave only the `git tag` path, which
is the fallback rather than the norm.
Restructure it around PR 1 -> PR 2 -> Release:
- a "The shape" table contrasting the two PRs by branch, base, contents,
verification and merge condition, plus why they must not be folded
together and why PR 1 merges before PR 2 is opened;
- step 2 broken into the tree-hash check and production smoke, the
ledger's structure, and the fix-on-`v2/main` rule that keeps the merge
tree byte-identical to `origin/v2/main`;
- step 3 leading with the GitHub UI path, keeping the CLI commands as the
by-hand equivalent.
Docs only. The `npm audit fix`, `--no-git-tag-version`, tag-`origin/main`,
no-`v`-prefix and #2010 warnings are all preserved.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RsmR1iQstcrEzJFmgZXGLi
Signed-off-by: cliffhall <cliff@futurescale.com>
- The recovery text told users to move the endpoint to `::1`. A bare IPv6
literal is not a legal URL host — `new URL("http://::1/token")` throws — so
anyone copying that into the Token URL override got a parse error from advice
meant to unblock them. Now `[::1]`, in both the notice and the manual
reproduction guide.
The exemption list still reads `::1`, because that is the host the SDK
compares; only the remedy is bracketed, because that is what a user types.
The comment says so, so neither gets "corrected" into the other.
- Dropped SEP-2207 from all nine files it had reached. I took the label from
the SDK's own source comments, which attribute this check to it — but this
repo already uses SEP-2207 for OIDC refresh / offline_access
(specification/v2_auth_hardening.md, plus an e2e test), so grepping it here
would have turned up two unrelated things. Replaced with plain description
rather than a different SEP number: having guessed wrong once, guessing again
is not the fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018MvNbmAtQuWhDvzi4yXDju
Signed-off-by: cliffhall <cliff@futurescale.com>
Closes#2280
InsecureTokenEndpointError appeared nowhere in our source, so it fell through
to the generic auth-failure path and rendered a "Re-authentication required"
banner with a Re-authenticate button. That button can never work: the SDK's
assertSecureTokenEndpoint runs inside executeTokenRequest, the error does not
extend OAuthError, and auth() special-cases it to rethrow rather than start a
fresh /authorize redirect. Clicking re-ran the same flow to the same refusal,
under the raw SDK message, which names the three exempt host literals and
nothing actionable. A terminal configuration error was being presented as a
retryable auth error.
Not specific to *.localhost: it fires for any endpoint outside the SDK's
exemption — host.docker.internal (#1911), a LAN hostname, a reverse-proxy
name, a mistyped scheme.
It is now recognized (core/auth/insecureTokenEndpoint.ts, mirroring
issuerBinding.ts's brand-plus-name classifier and walking cause / data.cause,
since era negotiation and the transport wrappers bury the rejection) and
surfaced as the configuration error it is, naming the endpoint and both ways
out, with no action affordance.
Six paths reach the refusal and all six classify it — the connect handshake,
authenticate() during connect, the satisfied-challenge connect retry, the
post-redirect callback, the command/background path, the deferred tab-visible
resume (which re-armed on every focus, an unbounded loop on a terminal error),
and the banner action. They share one wrapper so a seventh cannot omit the
banner clear, and every clear is scoped to its own serverId: these paths are
asynchronous, so a late continuation for one server must not erase a banner
another raised in the meantime.
Adds oauth-insecure-token-endpoint-http.json to reproduce it, whose issuer is
http://localhost.:8091 — the root-anchored spelling every resolver sends to
loopback but which is none of the SDK's exempt literals. transport.strictPort
keeps that fixture honest, since its port is hard-coded inside the issuer.
Making such an endpoint work has to land in the SDK (typescript-sdk#2591);
this changes the reporting, not the outcome.
Split out of #2282, where it was tangled with the *.localhost origin work for
#1944 — a separate question the reporter is still clarifying.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018MvNbmAtQuWhDvzi4yXDju
Signed-off-by: cliffhall <cliff@futurescale.com>
Ink writes styling as escape sequences inside the styled run, so an
accelerator underline splits the word it decorates: `<Text underline>I</Text>nfo`
reaches the frame buffer with escapes between `I` and `nfo`, and a plain
`toContain("Info")` fails against a component that is rendering correctly.
Chalk only emits color when it detects a TTY, so CI stays green and the six
failures land only on a developer whose shell exports FORCE_COLOR.
Add `__tests__/helpers/renderTui.tsx` — ink-testing-library's `render` with
`lastFrame()` and `frames` ANSI-stripped — and point all 16 renderer test
files at it. `frames` is a getter reading through to the live array, since
Ink appends to it as the component re-renders; the raw bytes stay reachable
as `stdout.lastFrame()`.
Pinning FORCE_COLOR=0 would turn the six green while hiding the class: any
future assertion on styled text has the same problem, and the suite would
stop covering the styled path at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFp2LVH7UJST5i8bDBJghw
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 2 on #2273: the list endpoint pages at 30, so a large
review round would silently drop later comments — and a workflow whose whole
point is replying to every comment cannot be documented with a command that
returns only the first thirty. --paginate combines with --jq fine (only --slurp
does not), verified against a live review.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017nLLLTGc37hQiC8s3mNyhq
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 1 on #2273: the text said only a reply in the thread "can
resolve it", which is wrong — resolution is a separate act (the Resolve
conversation button, or the resolveReviewThread GraphQL mutation) and it is the
reviewer's to make. Reading it as satisfied by the documented REST reply is
exactly the wrong conclusion for a PR whose whole point is what a reply does.
State what the reply actually buys: it is the only thing a reviewer reading that
thread sees, and it is what makes resolving the thread defensible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017nLLLTGc37hQiC8s3mNyhq
Signed-off-by: cliffhall <cliff@futurescale.com>
pr-flow §8's "mirror each round at PR level" was being read as permission to
skip the inline reply, so the rollup got posted and the per-thread replies never
happened. A bullet elsewhere on the page cannot be connected back to the thread
it answers: the thread stays open showing a finding and silence, and by round
three matching bullets to comments is reconstruction rather than reading.
- AGENTS.md, "Responding to Code Reviews": says where the response goes, which
was the ambiguity being resolved the wrong way — inline in the comment's own
thread, with the rollup in addition, never instead.
- pr-flow §8 reorders the two so the primary is unmistakable, and keeps the
reason the mirror exists (threads go outdated once the fix is pushed) as a
rationale for it rather than an alternative to the inline reply.
- Both name the exception: a finding in the "Suppressed comments" block has no
comment id and so no thread to reply into, making the mirror its only home.
- §8 gains the two mechanics, so the easy path is not the only documented one:
the reply endpoint keyed by comment id, and fetching comments per review by
review id, since the unpaginated /reviews listing hides later rounds behind
your own replies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017nLLLTGc37hQiC8s3mNyhq
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 2 on #2272: a jq failure on an unexpected shape leaves n
empty, `[ "" -ge 1 ]` then exits non-zero, break never fires, and the loop
sleeps and retries forever — the same unbounded wait as the awk pipe, reached
from the other end. Verified locally: jq exits 5 and the test exits 1.
Propagate the jq failure, guard that n is numeric, and generalize the prose to
say every step which can fail exits rather than retries.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017nLLLTGc37hQiC8s3mNyhq
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 1 on #2272: piping the count through awk made awk's status
the loop's, so an auth or API failure read as a count of 0 and the job would
retry forever on a poll that can never succeed. The example also used sleep 20
against the 30s remote-API floor the same PR adds to AGENTS.md.
Capture the response first and exit non-zero on a gh failure; jq runs separately
because --slurp cannot be combined with --jq. sleep 30.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017nLLLTGc37hQiC8s3mNyhq
Signed-off-by: cliffhall <cliff@futurescale.com>
Agents routinely spend turns tight-polling for a condition that was already
going to notify them — ~80 consecutive no-op turns waiting out one local:gate
run on #2250, with a completion notifier for that very process already armed.
The convention existed only in a per-user memory note, which is invisible to
anyone else and cannot be relied on mid-session.
- AGENTS.md gains a "Waiting on long-running work" section: arm a notifier and
stop; one notification is a backgrounded command or a single `until` loop, many
is a Monitor. It states the one genuine exception — state the harness cannot
observe — and says where the poll belongs in that case, inside one backgrounded
loop rather than spread across turns.
- pre-push-gate points at it where the gate is actually run.
- pr-flow §7 points at it where the Copilot review is actually waited on, with
the loop written out. The count-not-existence trap is called out: on round two
the first round's review is still there, so `-ge 1` returns immediately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017nLLLTGc37hQiC8s3mNyhq
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 2 on #2271: `scripts/smoke-web-*.mjs` overclaimed —
smoke-web-app.mjs and smoke-web-browser.mjs create pages with no viewport at
all, so only the tabs and elicitation smokes back the number. Name those two
explicitly and say the other two set none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017nLLLTGc37hQiC8s3mNyhq
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 1 on #2271:
- §5 covers the TUI as well as the web UI, but a TUI capture has no Playwright
viewport and no fullPage mode. §5a is now explicitly the browser path, with a
one-line TUI instruction (size the terminal so nothing wraps) that hands off
to §5b, which is retitled to say it applies to both.
- The claim that every existing shot was taken at 1280x900 was wrong —
specification/screenshots holds 1554x1374 and 3184x1516 among others. 1280x900
is now presented as the standard being adopted, grounded in the web smokes,
with the assorted older sizes named as the problem rather than the precedent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017nLLLTGc37hQiC8s3mNyhq
Signed-off-by: cliffhall <cliff@futurescale.com>
`/pr-flow` §5 said what to capture and where to put it, but nothing about the
viewport it is captured at — so screenshots clipped their contents, most
visibly the Monitor sidebar, and #2234 needed three full re-captures.
Split §5 into capture / read-back / upload, and add:
- A named capture size, 1280x900 full page — the viewport the web smokes
already use, so shots are comparable across PRs.
- The trap that widening the window does not widen the Monitor sidebar, with
both levers: seeding `inspector.monitor.width` before load, or driving the
ARIA separator with ArrowLeft.
- A no-clipping read-back before uploading.
- The crossfade wait and the focus-marking mechanic, previously unwritten.
The upload mechanics are unchanged, just moved under §5c.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017nLLLTGc37hQiC8s3mNyhq
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, all three accepted.
1. Connecting still is not the condition. A few integration tests
deliberately hand-roll a JSON-RPC server because the composable
fixture *cannot* produce what they assert on —
`inspectorClient-malformed-list.test.ts` says so in its own header:
"the SDK's own server cannot emit these shapes: they are exactly what
its types forbid". Same for `listSalvage-era.test.ts`. Real
transport, real client, zero `test-servers/` dependency.
Round 4 already stated the right condition at the top — "does this
test depend on a fixture from `test-servers/`" — and then three
downstream sites went on saying "connects". Those are now scoped to
fixture use, and the hand-rolled exception is stated where the
trigger is defined rather than left to be discovered, with both files
named. Connecting is kept as a hint, since it usually does imply a
fixture, with the instruction to check whether one can express the
case first.
2. "description-strength measurement" was the wrong surface, and this
PR is what made it wrong: a chained prompt is deliberately written so
the target's *description* cannot fire on it, so the number measures
the pointer in the first skill's body and nothing else. Left as-is it
would send the next person tuning a red hand-off at the description.
3. "an 0.8 bar" -> "a 0.8 bar".
Re-measured: 100% / 100% hand-off, 63/63 first-move at 100%, RUNS=5.
Staged by explicit path, not `git add -A` — see ec6895a5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, accepted. Round 4 split the trigger into two branches and
fixed the closing section, but left the paragraph that ends the opening
section speaking for both — asserting a real transport and a protocol
era for the build-only case that lines 28-31 had just established has
neither.
Scoped explicitly: "In the connecting case …" keeps the transport and
protocol-era guidance, "In the build-only case …" says that guidance
does not apply and names what is actually wanted from `test-servers`
there — how to build the fixture, and why a stale build keeps serving
old code.
Re-measured: 100% / 100% hand-off, 63/63 first-move at 100%, full suite
at RUNS=5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, both accepted.
1. "Anything that puts a built fixture on disk" matches every web test,
not the case it was written for: `clients/web`'s `pretest` runs
`test-servers:build` ahead of every unit run, so the fixture is on
disk for the pure parser and storage tests the very next paragraph
excludes. The trigger contradicted its own exclusion list.
Re-keyed to depending on the artefact — the test imports, spawns, or
points a config at it — with the pretest fact stated as its own ⚠️ so
the distinction is legible rather than implied.
2. The closing section still said the fixture-consuming case needs "a
real server over a real transport" and a protocol era, which is
exactly false for the build-only consumer round 3 added. `smoke:tui`
drives no transport, so config choice and protocol era do not apply
to it — only the build and staleness half does.
The condition is now stated once, with two named sub-cases that say
which part of `test-servers` each needs: connecting takes the whole
procedure, naming-the-artefact takes the build half. The closing
section mirrors that split instead of collapsing them.
Re-measured: 100% / 100% hand-off, 63/63 first-move at 100%, full suite
at RUNS=5. Unchanged across all four rounds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
Two factual corrections, both suppressed comments.
- `.claude/skills/local-dev/SKILL.md`: the react/react-dom exception cited
"19.2.8 today, against the root's 19.2.7" — a skew this PR removes. Dropped
the version pair; the point being made is about which install resolves the
pair, and it stands without a number that goes stale on the next bump.
- `verify-dep-lockstep.mjs`: the `devDependencies` bullet claimed all four
surfaced skews were a devDependency somewhere. `react` is a root runtime
dependency whose client copies are peer-installed, so it is three of four —
which is a better illustration of why the union spans fields as well as
installs, and the bullet now says so.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J86a6vMcA9csPtdsg7ZcCZ
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, both nits accepted — the first is a correctness hole, not a nit.
1. `smoke:tui` never asserts a round trip, so round 2 filed it under
"boot-only, needs nothing". It calls
`ensureTestServers({ requires: ["stdio"] })` and hands the built
fixture to the TUI as its catalog's stdio command
(scripts/smoke-tui.mjs:105-124) — which means the staleness hazard,
the single most expensive thing in `test-servers`, lands on it in
full. Connecting was the wrong sole condition.
The trigger is now "does this test **connect to, or build and
consume**, an MCP fixture?", with `smoke:tui` named at all three
decision points and the fixture-consumption case given its own
paragraph in the closing section.
2. The "29 of 74" figure was brittle and I could not reproduce it
against a different pattern: 29 by my grep, 32 by a direct
`@modelcontextprotocol/inspector-test-server` import, 39 by a wider
union. A number whose value is a function of the grep does not belong
in a skill body that nobody re-runs. Dropped; the concrete
counter-examples stay, which is what actually carries the point —
`storage/store-id.test.ts` validates a string and `mcp/import/*`
parses config files, both sitting beside live-connection tests.
Re-measured after broadening the condition: 100% / 100% hand-off,
63/63 first-move at 100%, full suite at RUNS=5. Unchanged across all
three rounds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, both findings accepted — and the first is the same defect one
level down from round 1.
1. Round 1 replaced "a smoke" with a transport condition but left "an
integration test" unqualified, and left the tier note treating web
integration as wholly fixture-dependent. It is not: **29 of the 74
files in `src/test/integration/` reference a server at all.**
`storage/store-id.test.ts` validates a string; `mcp/import/*` parses
config files. They sit there for the node env and the 30s timeout —
placement is the project manifest, which is exactly why it cannot
also be the fixture trigger.
All four sites now say "an integration test **that connects**" and
name the counter-examples, including the folder-is-not-the-trigger
point at the `integration` bullet itself, and the tier ⚠️ now states
that the boundary cuts *across* the tiers rather than along them.
2. Replacing the worked example orphaned the warning after it: "the two
runs above straddle a 67-point swing … with no change to the pointer"
had nothing left to refer to, since the 33%/33% and 100%/33% pair it
names was gone and the runs on show differed by 20 points across a
pointer rewrite. The weak-pointer pair is restored as its own
paragraph, the warning names both figures explicitly, and the 20-point
intermediate is now framed as RUNS=5 residual noise — contrasted with
the 67, rather than standing in for it.
Re-measured, since "that connects" narrows the phrasing the two chain
prompts have to match: 100% / 100% hand-off, 63/63 first-move at 100%,
full suite at RUNS=5. Unchanged from round 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, both findings accepted.
1. The trigger was over-broad. "a smoke" and "everything from web
integration rightwards" routed work to `test-servers` that needs no
fixture at all: `smoke:launcher` only checks `--help`, `smoke:web`
asserts the SPA is served, `smoke:web:browser` stops at first paint,
and Storybook play functions render from fixture props. The condition
is now "does this exercise MCP behaviour over a transport?", stated
as such, with both sides named — the smokes that connect
(`smoke:cli`, `smoke:web:app`, `smoke:web:elicit`, `smoke:web:tabs`)
and `pack:verify` on one side, the boot-only ones and Storybook on
the other. Depth in the tier list is explicitly not the boundary.
2. The threshold rationale contradicted the new result. It said an 0.8
bar would mark a hand-off red "regardless of how strongly the first
skill points at the second" — which the 100% measurement refutes.
Both copies (docs and `skill-eval.mjs`) now say what 0.5 actually
buys: a floor on useful reliability for a noisier second-hop load, so
the column still separates an adequate pointer from a broken one. The
value is unchanged.
Re-measured after the narrowing, since a tighter trigger could have
weakened the pointer: 100% / 100% on the hand-off and 63/63 first-move
at 100%, full suite at RUNS=5. Recorded figure updated from the
intermediate 100% / 80%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
- Include `optionalDependencies` in the declared tier's candidate union. A
peer range constrains the consumer's host; an optional range is a direct
declaration npm attempts to install, so omitting it left an optional-only
package free to skew unseen. `peerDependencies` stays excluded, with the
reason stated.
- Make the end-to-end fixture's client copy UNDECLARED — only the root
manifest names `solo`, while `clients/web` merely holds a top-level copy.
That is #2226's defining shape, and it is what makes the candidate set a
union across installs rather than a per-install intersection; the old
fixture would have passed a per-install implementation. Adds the mirror
case (declared only by a client, held by the root) so the union is pinned
in both directions.
- Update the two docs that still described the single-tier guard:
`.claude/skills/local-dev/SKILL.md` said "nothing gates either of those,
and `verify:dep-lockstep` is not it" and cited the live `@types/node`
drift in the present tense; `docs/quality-gate.md` described only the
program tier. Both now state what each tier covers and the two limits
that remain.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J86a6vMcA9csPtdsg7ZcCZ
Signed-off-by: cliffhall <cliff@futurescale.com>
The hand-off measured 33% / 33% at RUNS=3 (#2204's two committed chain
cases). `testing` opened with a one-sentence ⚠️ *classifying* which work
belongs to `test-servers` — a fact, which the model skims — and said it
nowhere else except a bare cross-reference in the last section.
Rewrite it as an action with a trigger ("load the `test-servers` skill
now — that is step one"), enumerate the situations that qualify, and
repeat the pointer at the two later points where the run actually
decides it is writing an integration test: the `integration` project
bullet under "Where the test file goes", and the tier list. The top of a
body is read before the model knows it needs the second skill, so a
pointer that lives only there has already been scrolled past by the time
it matters.
Measured at RUNS=5 on the full suite: the two hand-off cases go to
100% / 80% (100% / 100% on a focused `-- test-servers` run), and all
63 first-move cases stay at 100% — no displacement, as expected, since
neither description changed.
Also update the two places that recorded the old number as the standing
state: the worked example in `docs/skill-authoring.md`, and the
`CHAIN_THRESHOLD` rationale in `scripts/skill-eval.mjs`, which cites the
33% measurement as the reason the bar is 0.5 rather than 0.8. The bar is
unchanged — a reshaped pointer raises the ceiling those two cases reach,
not the floor a new hand-off case should be judged against.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
Five findings, all real.
**The chain prompts carried the target skill's own trigger.** `test-servers`
claims "a change needs a real server to exercise it", and both cases said
"against a real/live server" — so the model could load `testing`, then pick
`test-servers` from the ORIGINAL prompt, in that order, and score a hit that
would have survived deleting the pointer from `testing` entirely. Rewritten to
carry no server cue ("end to end"), and the measured rate fell from 100% / 67%
to 33% / 33%. That is the size of the artefact, and 33% is what the pointer
alone is worth. Both cases are now red against the 0.5 bar and stay that way:
`skills:eval` is not a gate, the number IS the finding, and lowering the bar to
turn the column green would discard the only signal this feature adds. Filed as
#2247 with the trap written down so a fix cannot re-measure the artefact.
**A deny list cannot bound a 14-turn run.** It only names tools known when it
was written, and this checkout configures an HTTP `mcp-docs` server in
`.mcp.json` while a contributor's own MCP servers and plugins add more. Now
`--allowedTools Read,Glob,Grep,Skill` (what the harness actually needs) plus
`--strict-mcp-config` with no `--mcp-config`, which drops every configured
server. The deny list stays alongside, since a deny is unconditional while an
allow list governs pre-approval. What remains outside all three is a
contributor's plugin tools; `--bare` would remove those and skills with them,
which would measure nothing — so that residual is stated rather than papered
over.
**The chain threshold compared inclusively.** "More often than not" is `> 0.5`,
and `>=` passes exactly half whenever RUNS is even — 2/4 would report a result
the criterion does not license. `passesThreshold` is now strict for chains and a
floor for first moves, and a strict bound of 1.0 is rejected up front rather
than failing every case while looking like a trigger problem.
**The doc's probe command still denied only four tools**, contradicting the
section's own promise that a probe and a scored case see the same policy.
Updated, with a note that it must move with the harness, plus a hand-off probe
form.
**The reporting had no automated coverage** — the one thing that could silently
merge the two measurements lived inside `main`. Extracted as `formatReport` and
tested: both headings, per-group thresholds, the two separate summaries, the
combined exit, and both single-kind selections.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
`skills:eval` ran every case with `--max-turns 1`, so it measured one
thing: whether a skill is the model's FIRST tool call. That is right for a
skill a user reaches directly, and it leaves a whole class unmeasured — a
skill reached from inside another skill's body. `testing` tells the model
that picking a fixture is `/test-servers` and that it has to load it, and
`test-servers` scores 5/5 while every one of those five cases asks for it
by name. A skill only ever reachable through a hand-off would score a
clean 100% with the hand-off silently never firing.
An eval case is now one of two shapes, and exactly one:
{ "prompt": "…", "expect": "test-servers" } first move, 1 turn
{ "prompt": "…", "chain": ["testing", "test-servers"] } hand-off, 14 turns
A chain must END with the skill whose file it lives in. The case exists to
measure whether THIS skill is reachable, so the file that goes red is the
one belonging to the skill that stopped being reached; anchoring on the
first link would file a `test-servers` measurement under `testing`, where
a `test-servers` description edit would never be seen. Links are checked
against the repo's model-invoked set, which is why both `verify:skills`
and `collectCases` now validate in a second pass — `test-servers` sorts
before `testing`, so a per-directory check would reject a live chain as
unknown purely because of where the alphabet put it.
Scoring is an ordered SUBSEQUENCE, not a prefix and not a contiguous run:
the model may load something before the chain starts and something
unrelated in between, and neither changes the claim that A led to B.
`collectSkillInvocations` therefore returns an ordered array rather than a
Set — a B, A, B run has to stay distinguishable from one that never
reached B from A.
The two rates are reported in separate columns and never summed. A
hand-off is a second-hop load over many turns; a first-move rate is the
model's opening move. `CHAIN_MAX_TURNS` (14) and `CHAIN_THRESHOLD` are
their own knobs for the same reason, and `CHAIN_THRESHOLD` defaults to
**0.5** rather than inheriting 0.8: see the measurement below.
Read-only containment no longer leans on `--max-turns 1`, which was doing
much of it by itself. The deny list gains the agentic and network tools —
`Task` in particular, whose subagent the flag does not reach.
Measured, `skills:eval -- test-servers`:
First move (1 turn) 7/7 at 100%, unchanged
Hand-off (14 turns) 67% and 33% at RUNS=3
So the hand-off is real and unreliable — which is the fact nothing could
observe before this change, and the reason 0.5 rather than 0.8 is the
default bar: at 0.8 both committed cases are red no matter how strongly
the first skill points at the second, and the column stops carrying
signal. A `["pr-flow", "test-servers"]` probe measured 0% and was dropped
rather than kept: `pr-flow` says nothing about fixtures, so the case had
no lever short of broadening a description onto another skill's ground.
Both findings are written up in `docs/skill-authoring.md`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
Five findings, all valid.
`runOutdated` accepted every exit status and returned `result.stdout ?? ""`.
A failed `npm outdated` (exit 2 on a registry or config error) also prints
nothing, so the empty string parsed to an empty package list — and across five
installs a total outage produced output byte-identical to a clean sweep. It now
throws on any status other than the documented 0/1.
`main()` was untested, which is how that reached review: the helper-only suite
passed while the silent-success path was live. It now takes its spawn function
as a parameter and is driven by a fake, covering npm failure, create vs. edit,
milestone handling, both no-op paths and the missing-repo guard.
The no-op path returned before looking for the marker issue, so once every
install caught up, a still-open tracking issue kept its obsolete package table
indefinitely — contradicting the body's own promise to update in place. It now
rewrites through `buildClearedBody()`. It deliberately does not close the
issue: the sweep takes no board actions, and closing one whose card a
maintainer may have moved would make the board assert work shipped that this
script cannot verify shipped.
The release step mandated `npm audit fix`, which AGENTS.md forbids. The reason
is not `--force`: plain `audit fix` resolves an advisory with no upward escape
in range by silently downgrading, as it did to esbuild across three installs in
#2058, and `local:gate` has no version-regression check to catch it. The step
is now `npm audit` report-only, with fixes applied deliberately via a direct
bump or an `overrides` entry.
An unmilestoned issue is swept into Incoming, not Todo — Todo asserts a
maintainer signed off. Message corrected and pinned by a test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWMJENjs1mExXE5XPJTEei
Signed-off-by: cliffhall <cliff@futurescale.com>
Adds npm audit fix (root + every client) as the first sub-step of the
release skill's bump step, before npm version, so a release is never
gated on remembering to check separately. Never --force; anything
audit fix can't resolve is left to the dependabot-alert pipeline
(#2229) or a follow-up issue.
Disables Dependabot npm version updates in dependabot.yml (a
version-update PR carries no issue and no board card) and replaces
them with a monthly scheduled sweep that runs npm outdated across the
root install and every client, filing or updating one idempotent
tracking issue instead of an auto-generated PR.
Closes#2231
Part of #2229
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017N9ha27Qg6LHpBQP7G7zPY
Signed-off-by: cliffhall <cliff@futurescale.com>
Corrects the rationale for the five-positive floor, which was wrong. The
claim was that five prompts make a skill's rate flake-tolerant. The
evaluator scores each prompt on its own `passes / RUNS`, so the number of
prompts never enters any rate and cannot affect a threshold crossing —
`RUNS` is the only knob that does. The floor is worth keeping for
breadth: one prompt measures one phrasing, and a description that fires
on a single narrow shape would pass on its one lucky case.
The bad reasoning had reached five places — AGENTS.md, the authoring
checklist, the JSDoc, the check-site comment and the regression test —
plus the emitted message, which asserted it too. All corrected.
Moves `MIN_POSITIVE_CASES` above `validateEvalCases`'s JSDoc. It had been
inserted between the two, so the `@param`/`@returns` block documented the
constant and the function had none.
Replaces the two remaining cases answerable from `AGENTS.md`: `local-dev`'s
web-client-startup case (the rules file gives the exact command) and
`project-structure`'s clients/web-versus-core case (its tree answers it).
Both replacements probed 3/3 and measure 100%.
RUNS=5 skills:eval: 63/63, every local-dev and project-structure case at
100%. local:gate passes, 34/34.
Signed-off-by: cliffhall <cliff@futurescale.com>
Replaces the last `project-structure` case that `AGENTS.md` answers — a
helper used by the CLI and TUI belongs in `core/` per its own directory
tree. This was also the only case below 100% in the previous suite, so
the static reading and the score agreed. The replacement targets the
skill's first stated purpose, locating code, which `AGENTS.md` gives no
method for; it probed 3/3 and now measures 100%.
Extends the truncation guard's tests to `when_to_use`. The guard loops
over both fields but only `description` was pinned, so dropping the
second from that list would have gone unnoticed. The new case pairs a
quoted description with an unquoted `when_to_use` and asserts the
description is not implicated.
Scopes the negative-case requirement in the prose to model-invoked
skills; round 2 scoped the checklist but left this sentence saying
"every skill file".
Refreshes the stale `~2.8k of 4k` listing figure. The other instance is
past-tense about the three-to-nine experiment and stays, now said
explicitly so two numbers do not look like a disagreement.
Records the eval's real blind spot: all three cases review identified as
`AGENTS.md`-answerable were scoring 80-100% when flagged, so a high score
does not clear a case of overlap. The eval can show a case failing; it
cannot show a passing case is well-aimed.
RUNS=5 skills:eval: 63/63. local:gate passes, 34/34.
Signed-off-by: cliffhall <cliff@futurescale.com>
Fixes an unquoted `#` in `board-ops`'s description. YAML reads ` #28`
onward as a comment, so the parser — and Claude Code, which parses the
same way — only ever saw the description up to `Covers board`. Both
board numbers and the option-deletion hazard were absent from the
listing that decides whether the skill fires, and every check stayed
green because they look for an *empty* description and a truncated one
is still non-empty. Quoting restores all 271 characters, which is the
whole of the 3,144 vs 3,234 discrepancy the review flagged.
`parseSkill` now compares the raw scalar against what YAML kept and
errors on any loss, so the next one fails the gate rather than shipping.
Tests cover the truncated case and a legitimately quoted `#`.
Re-aims the `project-structure` Node-backend case, which `AGENTS.md`
answers via its tree and which had oscillated 80 -> 100% across runs —
the partial-overlap signature this PR documents. The replacement names
no internals and probed 4/4.
Makes the documented probe reproduce the harness: the same
`--disallowedTools` flags as `runPrompt` rather than `--allowedTools`,
`printf` instead of zsh's `print -r`, and `jq` over `tool_use` blocks
instead of a `grep` that also matched MCP server names in the init
event. The first-tool-call table is relabelled as an unrestricted probe,
since the eval cannot reach for Bash at all.
RUNS=5 skills:eval: 63/63, 62 at 100%; restoring those 90 characters
displaced nothing.
Signed-off-by: cliffhall <cliff@futurescale.com>
Adds a multi-name filter to `npm run skills:eval` so a set of related
skills can be iterated together, and makes an unknown name a hard error
rather than an empty run — a typo would otherwise report a green 0/0
that reads exactly like a clean pass of the skill that was meant.
Re-aims two `local-dev` cases and one `test-servers` case that partially
overlapped `AGENTS.md`. Partial overlap oscillates across runs rather
than sitting at zero, so it reads as eval noise instead of a defect in
the case; `docs/skill-authoring.md` now records that failure mode and
the observed scores.
Widens the `test-servers` description to the "Use when …" shape and
points the `testing` skill at it, since anything needing a real server
to run against is a procedure `testing` does not carry. That growth
moves the skill listing from 2,921 to 3,144 of 4,000 characters, so the
figure recorded in AGENTS.md is updated to match.
Signed-off-by: cliffhall <cliff@futurescale.com>
`skills:eval` was at 38/46 with several skills measuring 0-33%. The low
scores were mostly bad eval cases rather than bad descriptions, for two
reasons that were nowhere written down:
- The harness runs each case with `--max-turns 1`, so the skill has to be
the model's FIRST tool call. A prompt asking "how does X work here", or
naming a concrete file, sends the model to Read/Grep first — a better
answer to the question, scored as a miss. Every 0% case was one of these.
- `AGENTS.md` is resident on every turn, so a prompt it already answers is
answered without any skill and the case can never pass.
Replace the cases that measured the wrong thing, expand every model-invoked
skill from 3 to 5 positives (negatives kept) so one flake cannot cross the
threshold, and reshape `local-dev`'s description to the action-first form.
Add `docs/skill-authoring.md` with the case shapes that fire, the ones that
cannot, and the probe-then-measure loop.
RUNS=5 npm run skills:eval: 63/63 at or above threshold, 60 at 100%,
all 18 negatives clean.
Closes#2202
Signed-off-by: cliffhall <cliff@futurescale.com>
Two suppressed comments, both accurate.
- Pin zod, zod/v4 and open to the root in vitest.shared.mts, and add matching
paths to the three clients/web tsconfigs. The claim they replace — that these
resolve unambiguously by walk-up — was wrong: both still sit at the top level
of clients/web/node_modules as transitive copies (zod under
eslint-plugin-react-hooks, open under Storybook), so an unpinned bare import
from a web test resolved the client copy while core/ and the SDK packages
resolved the root's. For open that is only wasteful; for zod it is a schema
built by one copy and an instanceof check made by the other across the whole
@modelcontextprotocol/* surface. zod is deduped as well, and zod/v4 gets its
own entry because first-party code imports both specifiers and pinning one
would split the package rather than collapse it.
- Update the consumer comments that #2195 falsified. clients/web/vite.config.ts
(three places) and server/start-vite-dev-server.ts all still said the shared
aliases resolve pino/hono and friends from clients/web/node_modules; they now
state the split — react and react-dom from the client install, everything
root-declared from the repo root.
- local-dev skill: replace the wrong 'resolves unambiguously' list with the
distinction that actually matters. Deleting a declaration is not the same as
the package resolving from the root, because chokidar, zod, open and react
all sit in a client install transitively; check the install, not the manifest.
ajv, commander and undici carry no pin because no client install has a copy.
npm run local:gate passes from a cold tsc build.
Signed-off-by: cliffhall <cliff@futurescale.com>
- clients/web/tsconfig.{app,node,test}.json: repoint the pino, hono,
hono/streaming, @hono/node-server, atomically, chokidar and @napi-rs/keyring
path mappings from ./node_modules to ../../node_modules. The stale mappings
were not failing the build — an unresolvable paths entry falls back to
ordinary walk-up resolution, which finds the root copy — but chokidar is the
case where the fallback never runs: Vite's transitive copy still sits under
clients/web, so tsc was type-checking against the client's copy while Vitest
and the shipped bundle both resolved the root's. Same version today, silent
skew tomorrow. Both path prefixes now carry a comment saying which ownership
each one asserts.
- local-dev skill: the consolidation rationale no longer groups commander, open
and @hono/node-server (client-imported) with the core/ set, no longer claims
#2195 moved undici (it was already root-only), and no longer implies every
root-declared package has an alias.
npm run local:gate passes with clients/web/node_modules/.tmp cleared, so the
tsc results are from a cold build rather than an incremental cache.
Signed-off-by: cliffhall <cliff@futurescale.com>
- Complete the bundler external lists against the AGENTS.md invariant rather
than against today's reachability: chokidar, hono and react to clients/cli,
chokidar and hono to clients/tui, react to the web runner. Every root-declared
package core/ imports at runtime is now named in all three, so a new import in
core/ cannot silently inline one.
- AGENTS.md: split the single over-broad bullet in two. commander, open,
@hono/node-server, vite and @vitejs/plugin-react are root dependencies reached
only from client code, so they are not part of the core/ set and need naming
only in the external list of the client that imports them. Give the
vitest.shared.mts pins their own rule, stating the react/react-dom exception
instead of implying every root-owned package is aliased to the root.
- vitest.shared.mts: the umbrella comment now says root-OWNED rather than root
runtime dependency — express is test-only and lives in the root
devDependencies — and notes that the list is deliberately not the complete
root set, since a package that resolves unambiguously needs no pin.
- local-dev skill: document why react/react-dom stay client-pinned, which the
previous text contradicted.
npm run local:gate and npm run verify:bundle-externals pass.
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 3. Docs only.
The round-2 wording said every surviving copy is pinned by its holder's peer
range, in the same paragraph that documents cli's @types/node 24.13.1 against
the root's 24.13.3 — a transitive, which no peer range constrains. The rationale
contradicted its own example.
Both files now split the two mechanisms and grade them:
- A peer copy is constrained by its holder's peer range, which is a real pin only
when that range is exact. @vitest/browser-playwright pins vitest to a single
version (hence the exact trio). eslint-plugin-react-refresh accepts ^9 || ^10,
so the eslint copies agree only because npm resolves the same latest in both
installs — a coincidence, not a guarantee.
- A transitive copy is constrained by nothing of ours, and cli's @types/node has
already diverged. It was 24.13.1 on v2/main too: a declared ^24.12.4 still
loses to a nearer transitive.
Nothing gates either, and verify:dep-lockstep is not that gate.
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 2. Docs only; no manifest or lockfile change.
Two claims were wrong:
- "A tool one client runs stays in that client" is contradicted by the
manifests — tsup is declared in web, cli and tui, and vite in web and tui on
top of the root runtime dependency that --web --dev needs. The boundary is
"used by every client", not "used by one"; anything narrower stays put whether
one client declares it or several do. Both are out of scope here and say so.
- verify:dep-lockstep was named as the guard that the surviving copies agree. It
is not: it derives its candidate set from what each tsc program resolves, so a
tool binary never enters a program and is outside the set no matter how far it
drifts — and cli's @types/node 24.13.1 against the root's 24.13.3 is a live
skew that passes for the same reason. Both files now state the narrower
guarantee, say that nothing gates the binaries automatically, note that each
surviving copy is pinned by its holder's peer range, and give
`npm exec -- which` as the manual check.
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 1.
@vitest/browser-playwright declares an EXACT peer on vitest ("vitest":
"4.1.10"), so that package — not the root's range — decides which vitest lands in
clients/web. Dropping web's own vitest declaration left the root free to resolve
4.1.11 while web's peer stayed at 4.1.10, so web's tests ran on one vitest while
loading a @vitest/coverage-v8 provider built against another. Both still passed,
which is the bad part.
vitest and @vitest/coverage-v8 at the root and @vitest/browser-playwright in
clients/web are now pinned exactly to 4.1.10 and move together, the same
discipline as the exact prettier pin (#1790).
The docs also claimed more than the change delivers. npm auto-installs an unmet
peer into the install that needs it, and a client install cannot see the root's
tree — so web's eslint-plugin-react-refresh / -storybook and the TUI's
eslint-plugin-react-hooks each pull a client-local eslint, and web's
Storybook/Vitest stack pulls a local typescript and vitest; @types/express
hoists an @types/node into web and cli. Those copies sit nearer than the root's
and win. AGENTS.md and the local-dev skill now say so, and state what the
consolidation actually buys: one declaration and one place to bump, with
verify:dep-lockstep as the guard that the copies agree.
Signed-off-by: cliffhall <cliff@futurescale.com>
Every runtime dependency core/ imports is now declared once, in the repo-root
package.json, and nowhere else. Deleting the client-side copies does not merely
align them: a package installs only into an install root that declares it, so
with no second declaration a second copy cannot be created at all — the failure
class #1896 was opened for and #1965/#2029 built verify:dep-lockstep to detect.
- clients/{web,cli,tui,launcher}: dropped ajv, atomically, chokidar, commander,
open, pino, zod, @napi-rs/keyring, hono, @hono/node-server and react, as each
applied. clients/cli and clients/launcher now declare no runtime dependencies.
- Root hono floor raised ^4.12.18 -> ^4.13.1, keeping the higher of the two
ranges that were in the tree rather than silently lowering web's.
- tsup external lists: added ajv and zod to all three bundles, plus atomically
and open (cli) and atomically (tui). tsup externalizes only what the nearest
manifest declares, so a package that moves to the root is inlined unless it is
named explicitly.
- vitest.shared.mts: repointed the pino, hono, @hono/node-server, atomically,
chokidar and @napi-rs/keyring pins from <client>/node_modules to the repo
root, joining express, yaml and proper-lockfile. Left behind they would
resolve to a directory that no longer exists, or to a transitive copy an
unrelated dependency dragged in. react and react-dom stay client-pinned so
the pair stays version-matched within an install.
- Docs: AGENTS.md dependency placement, the root README and the local-dev skill
describe the consolidated layout.
verify:dep-lockstep candidate set: 4 -> 2 (hono and pino fell out; only
@standard-schema/spec and zod still reach one tsc program from two installs).
npm run local:gate and npm run pack:verify both pass.
Signed-off-by: cliffhall <cliff@futurescale.com>
Declare the toolchain every client uses once, at the repo root, and delete the
per-client copies: eslint, @eslint/js, typescript-eslint, globals, prettier and
typescript (all five manifests), plus vitest, @vitest/coverage-v8 and
@types/node (four). Adds the latter three to the root manifest; the first six
were already there.
Every client reaches the root copy by walk-up — npm run puts each ancestor
node_modules/.bin on PATH, and Node and TypeScript resolve modules and @types
the same way — so nothing about the client scripts changes. clients/launcher now
declares no devDependencies at all and its validate is unchanged.
The copies had already drifted: globals sat at ^17.7.0 at the root against
^17.4.0 in all four clients, and typescript-eslint at ^8.65.0 against ^8.56.1.
Consolidating takes the root's version in both cases, and makes the exact
prettier pin (#1790) a single pin rather than five that can diverge.
Tools only one client runs stay where they are — tsup, tsx, vite, vite-node,
playwright, storybook, happy-dom, ink-testing-library, and each client's own
@types/*.
Documents the rule in AGENTS.md's Dependency placement section and the reasoning
in the local-dev skill.
Signed-off-by: cliffhall <cliff@futurescale.com>
Measured before and after at the same sample size (n=4), with the six other
model-invoked skills unchanged:
which command runs the TUI suite 25% → 100%
asserting on a Mantine transition 50% → 100%
which tier spawns the built binary 25% → 100%
The lever is the description's SHAPE, not its length: leading with the actions
and then enumerating concrete situations ("Use when … ; when … ; when …") beats
a noun-phrase list of contents ending in a generic "use for any question
about …". The same edit took `pre-push-gate` from 3/5 to 5/5 earlier.
Re-ran the full eval to check the change did not simply move attention off
another skill, which is the failure the previous commit recorded: 40/46, up from
38/46, and the failing set is strictly smaller — `testing` went from three
failures to one, nothing else moved. That remaining case measured 100% in
isolation and 67% in the full run, which is sampling variance rather than a
finding.
AGENTS.md now extends the re-run rule to cover a description edit, not just a
flip, and records the shape that works.
Signed-off-by: cliffhall <cliff@futurescale.com>
`release` is now the only skill reachable by name alone, which is right: nobody
cuts a release by implication, whereas "file an issue for this" and "triage the
new issues" are both ordinary prose instructions.
Eval cases committed for both; each measured 5/5 at RUNS=3. One initial case
was rebuilt rather than tuned: "file an issue for the flaky Storybook test"
scored 0% because its SUBJECT was unresolved — the model went off grepping for
which test was flaky instead of reaching for the filing procedure. A trigger
case must not make the model investigate before it can act.
The full eval across all nine model-invoked skills reports a cost worth having
in writing, and it contradicts the justification given for the previous flip.
Every negative case still passes 100%, so there is no over-triggering — but the
skills that were already model-invoked measurably LOST trigger rate:
`project-structure` fell from 100% to 0% on two cases (n=4) and `testing` from
3/5 to 2/5, while all six newly flipped skills measured 100%.
So the ceiling is attention, not characters: the listing sat at 2.8k of 4k
throughout, and "the budget is not tight" was the wrong reason to think flipping
was free. AGENTS.md now records this, and requires the FULL eval to be re-run
after any flip rather than just the new skill's own cases.
Two of project-structure's three cases are also the invalid-case pattern this
file already documents — `core/auth/` is described in the AGENTS.md tree, so a
model answering "which package owns the secret storage backends" without a skill
is correct rather than failing. `testing`'s cases are not, and it was the weaker
of the two before this change as well.
Signed-off-by: cliffhall <cliff@futurescale.com>
`disable-model-invocation: true` does not merely make a skill prefer `/name` —
the model cannot invoke it at all, and its description is absent from the skill
listing. So "create a PR for #2163", which is how that work actually starts,
could never reach `pr-flow`. The original rationale ("procedures with side
effects we'd invoke by name anyway") conflates invoking a skill, which only
loads instructions, with performing the procedure it describes.
Flipped `pr-flow`, `pre-push-gate`, `board-ops` and `test-servers` to
model-invocable. `user-invocable` is an independent axis, so this is purely
additive — every `/name` still works. `issue-create`, `issue-triage` and
`release` stay deliberate.
`board-ops` is in the set because `pr-flow` and `issue-create` delegate their
card steps to it: a pointer from a model-invocable skill to a manual-only one is
a dead end for the model.
Costs weighed: the listing share goes 1,199 → 2,256 of 4,000, and a spurious
load costs ~250 characters where a missed one costs a wrong base branch or an
unsigned commit. The rules stay resident in AGENTS.md either way — this improves
the odds of getting the procedure too, it does not become the guarantee.
Eval cases committed for all four, measured at RUNS=3: pre-push-gate, board-ops
and test-servers 5/5; pr-flow 4/5, with "take it through to a pull request" at
100% and the Copilot-review case borderline. Three initial cases were replaced
rather than tuned green: their answers are resident in AGENTS.md, so the model
answering without the skill is correct behaviour, not a miss — the rule this
file already records about what makes a valid trigger case.
AGENTS.md's maintenance rule now says to default to `false` and explains why,
so the advice this change walks back does not get re-applied.
Signed-off-by: cliffhall <cliff@futurescale.com>
Three conflicts, all in the files this PR refactored. The code from #1955 and
#2161 (core/react's useSyncExternalStore conversion) merged cleanly.
- `.github/copilot-instructions.md` — modify/delete. This PR deletes it, and
the four lines v2/main added there are a distillation of AGENTS.md rules that
arrive in AGENTS.md anyway. Kept deleted.
- `AGENTS.md` — the incoming change expanded the `core/react/` entry in the
project tree by eleven lines; this PR compacted that tree to one line per
entry on purpose. Took the compact tree and carried the substance onto its
one line. The other incoming hunk — the store-subscription rules under React
instructions — merged cleanly, which is the split working: it is a RULE, so
it lands in the section that stayed.
- `README.md` — same shape, one tree line. This README no longer enumerates
`core/`'s subdirectories at all, so the substance went where that content now
lives: `docs/architecture.md` and the `project-structure` skill's core table.
Nothing from v2/main was dropped; every incoming statement is now in exactly one
place rather than the three it used to occupy.
Signed-off-by: cliffhall <cliff@futurescale.com>
- board-ops: write board dumps to a temp dir, not the repo root. The boards are
private, so a snapshot is a full export of item IDs and every card's Status
and Priority — and the recipe left three such files in the worktree, one
`git add -A` from being published in a PR. All three now go to `$BOARD_TMP`,
and the snapshot step prints the path, since recovery needs it.
- release: branch from `origin/v2/main` after fetching, and read the version
once you are on it. A default clone has only `main` checked out, so the local
ref may not exist and `package.json` was `main`'s — the released version
rather than the one being bumped from. Added the one-liner for reading it
before branching, since the branch name needs it.
Signed-off-by: cliffhall <cliff@futurescale.com>
- The CI wiring guard matched a substring, and last round's justification for
that was wrong: it accepts a longer script name
(`npm run verify:skills:cli:disabled`) and a mere mention (`echo npm run
verify:skills:cli`), either of which lets the real validator be removed while
the guard stays green. `runsCommand` now splits the script on the separators
that begin a new command and requires a segment to equal the invocation
exactly, with both counterexamples as regression cases.
- `local-dev`'s own description and its AGENTS.md index row both still promised
"where a new dependency must be declared" — the rules round 13 moved out of it
entirely. Both now describe what the skill actually carries: the reasoning
behind those rules. Recorded listing share updated to 1,199/4,000.
Signed-off-by: cliffhall <cliff@futurescale.com>
