Corrects the rationale for the five-positive floor, which was wrong. The
claim was that five prompts make a skill's rate flake-tolerant. The
evaluator scores each prompt on its own `passes / RUNS`, so the number of
prompts never enters any rate and cannot affect a threshold crossing —
`RUNS` is the only knob that does. The floor is worth keeping for
breadth: one prompt measures one phrasing, and a description that fires
on a single narrow shape would pass on its one lucky case.
The bad reasoning had reached five places — AGENTS.md, the authoring
checklist, the JSDoc, the check-site comment and the regression test —
plus the emitted message, which asserted it too. All corrected.
Moves `MIN_POSITIVE_CASES` above `validateEvalCases`'s JSDoc. It had been
inserted between the two, so the `@param`/`@returns` block documented the
constant and the function had none.
Replaces the two remaining cases answerable from `AGENTS.md`: `local-dev`'s
web-client-startup case (the rules file gives the exact command) and
`project-structure`'s clients/web-versus-core case (its tree answers it).
Both replacements probed 3/3 and measure 100%.
RUNS=5 skills:eval: 63/63, every local-dev and project-structure case at
100%. local:gate passes, 34/34.
Signed-off-by: cliffhall <cliff@futurescale.com>
Replaces the last `project-structure` case that `AGENTS.md` answers — a
helper used by the CLI and TUI belongs in `core/` per its own directory
tree. This was also the only case below 100% in the previous suite, so
the static reading and the score agreed. The replacement targets the
skill's first stated purpose, locating code, which `AGENTS.md` gives no
method for; it probed 3/3 and now measures 100%.
Extends the truncation guard's tests to `when_to_use`. The guard loops
over both fields but only `description` was pinned, so dropping the
second from that list would have gone unnoticed. The new case pairs a
quoted description with an unquoted `when_to_use` and asserts the
description is not implicated.
Scopes the negative-case requirement in the prose to model-invoked
skills; round 2 scoped the checklist but left this sentence saying
"every skill file".
Refreshes the stale `~2.8k of 4k` listing figure. The other instance is
past-tense about the three-to-nine experiment and stays, now said
explicitly so two numbers do not look like a disagreement.
Records the eval's real blind spot: all three cases review identified as
`AGENTS.md`-answerable were scoring 80-100% when flagged, so a high score
does not clear a case of overlap. The eval can show a case failing; it
cannot show a passing case is well-aimed.
RUNS=5 skills:eval: 63/63. local:gate passes, 34/34.
Signed-off-by: cliffhall <cliff@futurescale.com>
Enforces the five-positive standard this PR established rather than only
describing it. `validateEvalCases` checked `positives === 0`, so a skill
could regress to a single prompt and still pass the gate. At RUNS=5 one
case is 20 points of that skill's reading, so a one- or two-case skill
reports a rate a single flake can move across the threshold — well-formed
to the checker, meaningless as a measurement. Zero positives keeps its
own clearer message so the common mistake keeps its diagnosis.
Scopes the eval requirement to model-invoked skills in the checklist. A
name-only skill has no eval file because nothing can measure a trigger
only a human types, so an eval there is unrunnable rather than redundant.
Aligns checklist item 1 with the `#` hazard from round 1; it named only
the unquoted colon, which is the louder of the two failure modes.
Four test fixtures asserted on one-positive eval files and legitimately
went red; they now build positives from the exported MIN_POSITIVE_CASES.
The all-positive row keeps testing the missing-negative path, with a new
row for the below-floor case. 388/388 script tests pass.
No skill description or eval case changed, so trigger behaviour is
unaffected and skills:eval is unchanged from the previous round.
Signed-off-by: cliffhall <cliff@futurescale.com>
Fixes an unquoted `#` in `board-ops`'s description. YAML reads ` #28`
onward as a comment, so the parser — and Claude Code, which parses the
same way — only ever saw the description up to `Covers board`. Both
board numbers and the option-deletion hazard were absent from the
listing that decides whether the skill fires, and every check stayed
green because they look for an *empty* description and a truncated one
is still non-empty. Quoting restores all 271 characters, which is the
whole of the 3,144 vs 3,234 discrepancy the review flagged.
`parseSkill` now compares the raw scalar against what YAML kept and
errors on any loss, so the next one fails the gate rather than shipping.
Tests cover the truncated case and a legitimately quoted `#`.
Re-aims the `project-structure` Node-backend case, which `AGENTS.md`
answers via its tree and which had oscillated 80 -> 100% across runs —
the partial-overlap signature this PR documents. The replacement names
no internals and probed 4/4.
Makes the documented probe reproduce the harness: the same
`--disallowedTools` flags as `runPrompt` rather than `--allowedTools`,
`printf` instead of zsh's `print -r`, and `jq` over `tool_use` blocks
instead of a `grep` that also matched MCP server names in the init
event. The first-tool-call table is relabelled as an unrestricted probe,
since the eval cannot reach for Bash at all.
RUNS=5 skills:eval: 63/63, 62 at 100%; restoring those 90 characters
displaced nothing.
Signed-off-by: cliffhall <cliff@futurescale.com>
Adds a multi-name filter to `npm run skills:eval` so a set of related
skills can be iterated together, and makes an unknown name a hard error
rather than an empty run — a typo would otherwise report a green 0/0
that reads exactly like a clean pass of the skill that was meant.
Re-aims two `local-dev` cases and one `test-servers` case that partially
overlapped `AGENTS.md`. Partial overlap oscillates across runs rather
than sitting at zero, so it reads as eval noise instead of a defect in
the case; `docs/skill-authoring.md` now records that failure mode and
the observed scores.
Widens the `test-servers` description to the "Use when …" shape and
points the `testing` skill at it, since anything needing a real server
to run against is a procedure `testing` does not carry. That growth
moves the skill listing from 2,921 to 3,144 of 4,000 characters, so the
figure recorded in AGENTS.md is updated to match.
Signed-off-by: cliffhall <cliff@futurescale.com>
`skills:eval` was at 38/46 with several skills measuring 0-33%. The low
scores were mostly bad eval cases rather than bad descriptions, for two
reasons that were nowhere written down:
- The harness runs each case with `--max-turns 1`, so the skill has to be
the model's FIRST tool call. A prompt asking "how does X work here", or
naming a concrete file, sends the model to Read/Grep first — a better
answer to the question, scored as a miss. Every 0% case was one of these.
- `AGENTS.md` is resident on every turn, so a prompt it already answers is
answered without any skill and the case can never pass.
Replace the cases that measured the wrong thing, expand every model-invoked
skill from 3 to 5 positives (negatives kept) so one flake cannot cross the
threshold, and reshape `local-dev`'s description to the action-first form.
Add `docs/skill-authoring.md` with the case shapes that fire, the ones that
cannot, and the probe-then-measure loop.
RUNS=5 npm run skills:eval: 63/63 at or above threshold, 60 at 100%,
all 18 negatives clean.
Closes#2202
Signed-off-by: cliffhall <cliff@futurescale.com>
- Complete the bundler external lists against the AGENTS.md invariant rather
than against today's reachability: chokidar, hono and react to clients/cli,
chokidar and hono to clients/tui, react to the web runner. Every root-declared
package core/ imports at runtime is now named in all three, so a new import in
core/ cannot silently inline one.
- AGENTS.md: split the single over-broad bullet in two. commander, open,
@hono/node-server, vite and @vitejs/plugin-react are root dependencies reached
only from client code, so they are not part of the core/ set and need naming
only in the external list of the client that imports them. Give the
vitest.shared.mts pins their own rule, stating the react/react-dom exception
instead of implying every root-owned package is aliased to the root.
- vitest.shared.mts: the umbrella comment now says root-OWNED rather than root
runtime dependency — express is test-only and lives in the root
devDependencies — and notes that the list is deliberately not the complete
root set, since a package that resolves unambiguously needs no pin.
- local-dev skill: document why react/react-dom stay client-pinned, which the
previous text contradicted.
npm run local:gate and npm run verify:bundle-externals pass.
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 3. Docs only.
The round-2 wording said every surviving copy is pinned by its holder's peer
range, in the same paragraph that documents cli's @types/node 24.13.1 against
the root's 24.13.3 — a transitive, which no peer range constrains. The rationale
contradicted its own example.
Both files now split the two mechanisms and grade them:
- A peer copy is constrained by its holder's peer range, which is a real pin only
when that range is exact. @vitest/browser-playwright pins vitest to a single
version (hence the exact trio). eslint-plugin-react-refresh accepts ^9 || ^10,
so the eslint copies agree only because npm resolves the same latest in both
installs — a coincidence, not a guarantee.
- A transitive copy is constrained by nothing of ours, and cli's @types/node has
already diverged. It was 24.13.1 on v2/main too: a declared ^24.12.4 still
loses to a nearer transitive.
Nothing gates either, and verify:dep-lockstep is not that gate.
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 2. Docs only; no manifest or lockfile change.
Two claims were wrong:
- "A tool one client runs stays in that client" is contradicted by the
manifests — tsup is declared in web, cli and tui, and vite in web and tui on
top of the root runtime dependency that --web --dev needs. The boundary is
"used by every client", not "used by one"; anything narrower stays put whether
one client declares it or several do. Both are out of scope here and say so.
- verify:dep-lockstep was named as the guard that the surviving copies agree. It
is not: it derives its candidate set from what each tsc program resolves, so a
tool binary never enters a program and is outside the set no matter how far it
drifts — and cli's @types/node 24.13.1 against the root's 24.13.3 is a live
skew that passes for the same reason. Both files now state the narrower
guarantee, say that nothing gates the binaries automatically, note that each
surviving copy is pinned by its holder's peer range, and give
`npm exec -- which` as the manual check.
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 1.
@vitest/browser-playwright declares an EXACT peer on vitest ("vitest":
"4.1.10"), so that package — not the root's range — decides which vitest lands in
clients/web. Dropping web's own vitest declaration left the root free to resolve
4.1.11 while web's peer stayed at 4.1.10, so web's tests ran on one vitest while
loading a @vitest/coverage-v8 provider built against another. Both still passed,
which is the bad part.
vitest and @vitest/coverage-v8 at the root and @vitest/browser-playwright in
clients/web are now pinned exactly to 4.1.10 and move together, the same
discipline as the exact prettier pin (#1790).
The docs also claimed more than the change delivers. npm auto-installs an unmet
peer into the install that needs it, and a client install cannot see the root's
tree — so web's eslint-plugin-react-refresh / -storybook and the TUI's
eslint-plugin-react-hooks each pull a client-local eslint, and web's
Storybook/Vitest stack pulls a local typescript and vitest; @types/express
hoists an @types/node into web and cli. Those copies sit nearer than the root's
and win. AGENTS.md and the local-dev skill now say so, and state what the
consolidation actually buys: one declaration and one place to bump, with
verify:dep-lockstep as the guard that the copies agree.
Signed-off-by: cliffhall <cliff@futurescale.com>
Every runtime dependency core/ imports is now declared once, in the repo-root
package.json, and nowhere else. Deleting the client-side copies does not merely
align them: a package installs only into an install root that declares it, so
with no second declaration a second copy cannot be created at all — the failure
class #1896 was opened for and #1965/#2029 built verify:dep-lockstep to detect.
- clients/{web,cli,tui,launcher}: dropped ajv, atomically, chokidar, commander,
open, pino, zod, @napi-rs/keyring, hono, @hono/node-server and react, as each
applied. clients/cli and clients/launcher now declare no runtime dependencies.
- Root hono floor raised ^4.12.18 -> ^4.13.1, keeping the higher of the two
ranges that were in the tree rather than silently lowering web's.
- tsup external lists: added ajv and zod to all three bundles, plus atomically
and open (cli) and atomically (tui). tsup externalizes only what the nearest
manifest declares, so a package that moves to the root is inlined unless it is
named explicitly.
- vitest.shared.mts: repointed the pino, hono, @hono/node-server, atomically,
chokidar and @napi-rs/keyring pins from <client>/node_modules to the repo
root, joining express, yaml and proper-lockfile. Left behind they would
resolve to a directory that no longer exists, or to a transitive copy an
unrelated dependency dragged in. react and react-dom stay client-pinned so
the pair stays version-matched within an install.
- Docs: AGENTS.md dependency placement, the root README and the local-dev skill
describe the consolidated layout.
verify:dep-lockstep candidate set: 4 -> 2 (hono and pino fell out; only
@standard-schema/spec and zod still reach one tsc program from two installs).
npm run local:gate and npm run pack:verify both pass.
Signed-off-by: cliffhall <cliff@futurescale.com>
Declare the toolchain every client uses once, at the repo root, and delete the
per-client copies: eslint, @eslint/js, typescript-eslint, globals, prettier and
typescript (all five manifests), plus vitest, @vitest/coverage-v8 and
@types/node (four). Adds the latter three to the root manifest; the first six
were already there.
Every client reaches the root copy by walk-up — npm run puts each ancestor
node_modules/.bin on PATH, and Node and TypeScript resolve modules and @types
the same way — so nothing about the client scripts changes. clients/launcher now
declares no devDependencies at all and its validate is unchanged.
The copies had already drifted: globals sat at ^17.7.0 at the root against
^17.4.0 in all four clients, and typescript-eslint at ^8.65.0 against ^8.56.1.
Consolidating takes the root's version in both cases, and makes the exact
prettier pin (#1790) a single pin rather than five that can diverge.
Tools only one client runs stay where they are — tsup, tsx, vite, vite-node,
playwright, storybook, happy-dom, ink-testing-library, and each client's own
@types/*.
Documents the rule in AGENTS.md's Dependency placement section and the reasoning
in the local-dev skill.
Signed-off-by: cliffhall <cliff@futurescale.com>
Brings in #2192 (react-hooks lint over `core/react/`) and #2194. All the code,
`eslint.config.js` and the new root devDependency merged cleanly; three
conflicts, all in files this PR refactored.
- `.github/copilot-instructions.md` — modify/delete again. Kept deleted; the
line v2/main added there restates AGENTS.md rules that arrive in AGENTS.md.
- `AGENTS.md` — two of the three incoming hunks merged cleanly, both of them
RULES, into the sections that survived the refactor: `exhaustive-deps` is
`error` in every React scope including `core/react/**`, and
`set-state-in-effect` now covers `core/react/` with `clients/tui` called out
as the exception. The third was a `validate:core` bullet inside the pre-push
gate section this PR moved wholesale to docs/quality-gate.md, so its mechanism
went there instead — including why the plugin has to be a ROOT devDependency
(Node resolution walks up, not down, so a client-local copy is unreachable).
- `README.md` — the incoming paragraph was added to the quality-gate section
this README no longer carries. Moved verbatim to docs/quality-gate.md, beside
the type-aware-lint paragraph it belongs with.
Ran `npm install` for the new devDependency; `npm run validate` passes.
Signed-off-by: cliffhall <cliff@futurescale.com>
Measured before and after at the same sample size (n=4), with the six other
model-invoked skills unchanged:
which command runs the TUI suite 25% → 100%
asserting on a Mantine transition 50% → 100%
which tier spawns the built binary 25% → 100%
The lever is the description's SHAPE, not its length: leading with the actions
and then enumerating concrete situations ("Use when … ; when … ; when …") beats
a noun-phrase list of contents ending in a generic "use for any question
about …". The same edit took `pre-push-gate` from 3/5 to 5/5 earlier.
Re-ran the full eval to check the change did not simply move attention off
another skill, which is the failure the previous commit recorded: 40/46, up from
38/46, and the failing set is strictly smaller — `testing` went from three
failures to one, nothing else moved. That remaining case measured 100% in
isolation and 67% in the full run, which is sampling variance rather than a
finding.
AGENTS.md now extends the re-run rule to cover a description edit, not just a
flip, and records the shape that works.
Signed-off-by: cliffhall <cliff@futurescale.com>
`release` is now the only skill reachable by name alone, which is right: nobody
cuts a release by implication, whereas "file an issue for this" and "triage the
new issues" are both ordinary prose instructions.
Eval cases committed for both; each measured 5/5 at RUNS=3. One initial case
was rebuilt rather than tuned: "file an issue for the flaky Storybook test"
scored 0% because its SUBJECT was unresolved — the model went off grepping for
which test was flaky instead of reaching for the filing procedure. A trigger
case must not make the model investigate before it can act.
The full eval across all nine model-invoked skills reports a cost worth having
in writing, and it contradicts the justification given for the previous flip.
Every negative case still passes 100%, so there is no over-triggering — but the
skills that were already model-invoked measurably LOST trigger rate:
`project-structure` fell from 100% to 0% on two cases (n=4) and `testing` from
3/5 to 2/5, while all six newly flipped skills measured 100%.
So the ceiling is attention, not characters: the listing sat at 2.8k of 4k
throughout, and "the budget is not tight" was the wrong reason to think flipping
was free. AGENTS.md now records this, and requires the FULL eval to be re-run
after any flip rather than just the new skill's own cases.
Two of project-structure's three cases are also the invalid-case pattern this
file already documents — `core/auth/` is described in the AGENTS.md tree, so a
model answering "which package owns the secret storage backends" without a skill
is correct rather than failing. `testing`'s cases are not, and it was the weaker
of the two before this change as well.
Signed-off-by: cliffhall <cliff@futurescale.com>
`disable-model-invocation: true` does not merely make a skill prefer `/name` —
the model cannot invoke it at all, and its description is absent from the skill
listing. So "create a PR for #2163", which is how that work actually starts,
could never reach `pr-flow`. The original rationale ("procedures with side
effects we'd invoke by name anyway") conflates invoking a skill, which only
loads instructions, with performing the procedure it describes.
Flipped `pr-flow`, `pre-push-gate`, `board-ops` and `test-servers` to
model-invocable. `user-invocable` is an independent axis, so this is purely
additive — every `/name` still works. `issue-create`, `issue-triage` and
`release` stay deliberate.
`board-ops` is in the set because `pr-flow` and `issue-create` delegate their
card steps to it: a pointer from a model-invocable skill to a manual-only one is
a dead end for the model.
Costs weighed: the listing share goes 1,199 → 2,256 of 4,000, and a spurious
load costs ~250 characters where a missed one costs a wrong base branch or an
unsigned commit. The rules stay resident in AGENTS.md either way — this improves
the odds of getting the procedure too, it does not become the guarantee.
Eval cases committed for all four, measured at RUNS=3: pre-push-gate, board-ops
and test-servers 5/5; pr-flow 4/5, with "take it through to a pull request" at
100% and the Copilot-review case borderline. Three initial cases were replaced
rather than tuned green: their answers are resident in AGENTS.md, so the model
answering without the skill is correct behaviour, not a miss — the rule this
file already records about what makes a valid trigger case.
AGENTS.md's maintenance rule now says to default to `false` and explains why,
so the advice this change walks back does not get re-applied.
Signed-off-by: cliffhall <cliff@futurescale.com>
- `useEmaIdpLoginState`: the effect-triggered `refresh()` was discarded with
`void`, but it does not own its failures — `getEmaIdpLoginState` awaits
`storage.getIdpSession`, which rejects when the backend is unreachable.
Terminate with `.catch()` and leave `loginState` untouched, as `logout`
already does. Covered by a new test that also asserts no unhandled
rejection escapes.
- Correct the history: the plugin was *client-local*, declared by both
`clients/web` and `clients/tui`. The root could reach neither, since Node
resolution walks up rather than down — that, not web-exclusivity, is why
`core/react/` went unlinted.
- Correct the enforcement claim: `clients/tui` registers the plugin but
deliberately enables only `rules-of-hooks` and `exhaustive-deps`, so
`set-state-in-effect` and `refs` hold for `clients/web` and `core/react/`
only. Says so in AGENTS.md and in the Copilot mirror, which would
otherwise have reviewers cite rules the TUI gate does not run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013hzwzS8UvBsr4yZm7o7sev
Signed-off-by: cliffhall <cliff@futurescale.com>
`eslint-plugin-react-hooks` was a `clients/web` devDependency only, and each
client's `eslint .` is scoped to its own directory — so the whole hook rule
set had never looked at `core/react/`, a directory of nothing but React
hooks. "Neither surface has JSX" was the stated reason and was never a good
one: the rules judge hooks, not JSX.
Add the plugin at the root and apply it in a `core/react/**` block (scoped
there rather than to all of `core/**`, since the rules key off the `use`
prefix), with `exhaustive-deps` promoted to `error` as `clients/web` does.
Resolve the 15 findings it then reports:
- `useSettingsDraft` / `useClientSettingsDraft` seed their draft during
render instead of in a `targetId`/`opened`-keyed effect, so the previous
target's values never reach the DOM — the same rule `clients/web` spells
`useValueChange`. Their "latest value" refs move from render-time writes
into an effect; every reader (the debounce timer, `flush`, `onChange`)
runs after a commit, so a post-commit write is what they need.
- The three async loaders keep their effects behind justified inline
disables: `set-state-in-effect` follows a call into an async function
without modelling the `await`, verified against a minimal hook whose only
setState is in a continuation.
- `useInitialConfig`'s cleanup bump keeps reading `generation.current` at
teardown — the rule's "copy it into a variable" advice would restore a
stale request token and let a superseded response commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013hzwzS8UvBsr4yZm7o7sev
Signed-off-by: cliffhall <cliff@futurescale.com>
Three conflicts, all in the files this PR refactored. The code from #1955 and
#2161 (core/react's useSyncExternalStore conversion) merged cleanly.
- `.github/copilot-instructions.md` — modify/delete. This PR deletes it, and
the four lines v2/main added there are a distillation of AGENTS.md rules that
arrive in AGENTS.md anyway. Kept deleted.
- `AGENTS.md` — the incoming change expanded the `core/react/` entry in the
project tree by eleven lines; this PR compacted that tree to one line per
entry on purpose. Took the compact tree and carried the substance onto its
one line. The other incoming hunk — the store-subscription rules under React
instructions — merged cleanly, which is the split working: it is a RULE, so
it lands in the section that stayed.
- `README.md` — same shape, one tree line. This README no longer enumerates
`core/`'s subdirectories at all, so the substance went where that content now
lives: `docs/architecture.md` and the `project-structure` skill's core table.
Nothing from v2/main was dropped; every incoming statement is now in exactly one
place rather than the three it used to occupy.
Signed-off-by: cliffhall <cliff@futurescale.com>
- The CI wiring guard matched a substring, and last round's justification for
that was wrong: it accepts a longer script name
(`npm run verify:skills:cli:disabled`) and a mere mention (`echo npm run
verify:skills:cli`), either of which lets the real validator be removed while
the guard stays green. `runsCommand` now splits the script on the separators
that begin a new command and requires a segment to equal the invocation
exactly, with both counterexamples as regression cases.
- `local-dev`'s own description and its AGENTS.md index row both still promised
"where a new dependency must be declared" — the rules round 13 moved out of it
entirely. Both now describe what the skill actually carries: the reasoning
behind those rules. Recorded listing share updated to 1,199/4,000.
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 2.
The contract claimed an unstable `read`/`whenAbsent` trips React's "The
result of getSnapshot should be cached" error. It does not. Measured on
React 19: an inline `read` closure throws nothing, logs nothing, and forces
no extra render pass. React's dev check calls getSnapshot twice within a
single render, where the closure is unchanged, so the cache answers both
calls identically and the warning never fires.
The real cost is quiet — a fresh value every render, defeating downstream
useMemo/React.memo/effect deps — which is a better reason to state the
contract than an error that never arrives.
Also stop requiring module scope for every fallback. A primitive
`whenAbsent` (`false`, `undefined`, `"disconnected"`) is stable under
Object.is and is passed inline throughout these hooks, so the rule
contradicted the code it was describing. Only object/array fallbacks need a
constant, which is what NO_TOOLS and NO_PAGINATION are.
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 1.
The cache was keyed on the store and its revision alone, so a changed
`read` was answered with the previous reader's output, and with a null
store — where the revision is pinned at 0 — a changed `whenAbsent` was
ignored outright. Key on `event`, `read` and `whenAbsent` too.
This makes an unstable argument defeat the cache rather than silently
serve a value derived from the previous one, so React fails it with "The
result of getSnapshot should be cached" at first render. That is the
intended failure mode and is now documented as such: a loud error beats a
wrong list nobody can account for.
Also correct the docs, which described the helper as universal. That
misread useListError as a violation: its snapshot is the stored Error
instance, already referentially stable, so it subscribes with
useSyncExternalStore directly and needs no cache. The rule is
read-during-render; the helper is for getters that return a fresh value
per read.
Signed-off-by: cliffhall <cliff@futurescale.com>
- verify-skills: the best-effort `plugin validate` hand-off now also requires
the EXACT pinned CLI. `local:gate` runs `validate` before the pinned
`verify:skills:cli` step, so a newer local CLI could reject skills that CI's
pinned validator accepts — and the gate would exit there, never reaching the
reproducible check. One schema everywhere, or none.
- Dropped `PLUGIN_VALIDATE_MIN_VERSION`: with both validators on the exact pin
it was read only by its own test, and a constant nothing enforces is a claim
nothing enforces. Its test now pins what actually matters — that equality is
distinguishable from newer as well as older.
- The CI step's comment described the old floor behavior.
- issue-create: the duplicate check searched only open issues, so a completed or
rejected match was invisible to a procedure that says never to duplicate.
Searches all states and reports state/stateReason.
Signed-off-by: cliffhall <cliff@futurescale.com>
Every state-subscription hook in core/react/ seeded local React state from
its store prop and re-synced it inside a useEffect. That shape paints one
frame of the previous store's data when the prop changes (switching
servers), and loses any event dispatched between the render that took the
snapshot and the effect that attached the listener.
Convert all 16 to useSyncExternalStore through a new shared
useStoreSnapshot, following useManagedListError (#1954).
useSyncExternalStore needs a referentially stable snapshot, and these
getters hand back a defensive copy, so TypedEventTarget now keeps a
per-event-type dispatch counter (getEventRevision) that the snapshot is
cached against. A contents comparison would not do: MessageLogState folds
a response into its request entry by mutating that entry in place, which a
shallow compare reports as unchanged. The bump lives on the dispatchEvent
override so no dispatch — including a hand-rolled CustomEvent — can bypass
it.
useInspectorClient is included though the issue did not list it; its nine
subscribed slices made the stale frame most visible. Its lastError stays
useState (accumulated from events, no getter to read back) but resets on
client swap during render rather than in an effect.
The settings-draft and async fetch hooks are deliberately untouched: the
former hold genuinely local editable state, the latter are I/O rather than
store subscriptions.
Also answers the issue's open question. react-hooks/set-state-in-effect
never fired here because eslint-plugin-react-hooks has never linted
core/react/ at all — it is a clients/web devDependency and the root config
loads no React plugin. Corrected the config comment that gave "no JSX" as
the reason; the gap itself is #2192, measured at 15 findings, none of them
in the hooks converted here.
Closes#1955
Signed-off-by: cliffhall <cliff@futurescale.com>
- verify-skills-cli: use a local CLI only when it matches the pin EXACTLY. A
floor let a maintainer on a newer CLI validate against a different schema than
CI's, so the same `local:gate` could disagree across machines — the failure a
pin exists to prevent.
- verify-skills-cli: extract `runValidator({probe, spawn, log, error})` and
cover the orchestration — validator accepts, validator rejects, spawn fails,
and the three unusable-probe shapes. On a machine with a working CLI an actual
run only ever walks the happy path.
- pr-flow: stop putting the `gh` token in curl's argv, where any local process
can read it off the process table. Feed the header through `curl --config -`.
- docs/quality-gate: three places had drifted — the CI tier row, the
`local:gate` chain, and the `test:scripts` inventory (which now names all four
suites this PR adds, not two).
Signed-off-by: cliffhall <cliff@futurescale.com>
- verify:skills:cli — the authoritative validator is now a GUARANTEED step in
`local:gate` and CI, not a best-effort hand-off. It resolves the CLI itself
(an installed one at 2.1.233+, else a pinned `npx -y`), so both tiers run the
same script and cannot drift. `verify:skills` keeps skipping when the CLI is
absent, which is right for `validate`: fast, offline, no Claude Code required.
- skill-manifest: the 1,536-char listing-entry cap covers `description` +
`when_to_use` together; checking the description alone passed an over-cap
entry. `when_to_use` is now parsed, validated and counted, with boundary tests
at 1,536/1,537.
- verify-skills: report the budget as *this repo's contribution*. The real
listing budget is shared with bundled skills and the contributor's own
~/.claude/skills, none of which is visible from here, so the number was
claiming more than it measured.
- skill-eval: a negative case now means "no skill OF THIS REPO'S fired". It
failed on any Skill invocation, so a contributor with an unrelated global
skill got a false failure about their environment rather than these skills.
- docs/architecture + local-dev: the Node backend runner gets its
`@inspector/core` alias from `clients/web/tsup.runner.config.ts`, not from
Vite. The old text sent runner work to the wrong config.
- local-dev: the tarball does not ship "only each client's build/" — the
allowlist also carries `clients/web/dist`, `clients/web/static` and the
install helper. States the invariant that actually matters instead.
- issue-triage: the test for needing triage is "no board card". Adding "and no
milestone" would skip the approved-but-unboarded case the same section then
handles. Also corrected the `$M` note: the lookup is `--state all`, so closed
issues DO have entries — `isopen` is about the invariants, not the lookup.
- issue-create: replaced "all five" with a per-version table. Steps 3 and 6
(milestone, Priority) do not exist for v1, so the old list opened with a rule
the same skill later tells you to violate.
Signed-off-by: cliffhall <cliff@futurescale.com>
- CI now installs a pinned `claude` CLI and runs `claude plugin validate` as its
own step. The guard skips the authoritative validator when the CLI is absent,
which without this meant "always, in CI" — so the acceptance criterion was
aspirational. Verified it needs no authentication (clean HOME).
- skill-eval: classify a run from its terminal `result` event, not its exit
code. With `--max-turns 1` a run in which a skill FIRES necessarily hits the
limit and the CLI exits 1 (`error_max_turns`, `num_turns: 2`), so last round's
blanket nonzero rejection threw away exactly the observations the eval counts.
A run with no terminal `result`, or an unusable subtype, still rejects.
- skill-eval: fail collection on a skill that does not parse, a model-invoked
skill with no eval file, unreadable JSON, or invalid cases. Skipping them let
`skills:eval` exit 0 while silently omitting the skill just broken.
- verify-skills: `main()` takes an optional skills directory, and
`verify-skills.main.test.mjs` drives it over fixtures — missing SKILL.md,
malformed frontmatter, undeclared invocation mode, missing/invalid evals,
budget overflow (and that name-only skills do not count toward it), and
multi-offender reporting. Matches the sibling guards' *.main.test.mjs
precedent; without it the gate could stop enforcing while the parser tests
stayed green.
- issue-create: the note claimed an issue arrives unlabeled, contradicting
.github/ISSUE_TEMPLATE — blank issues are disabled, the bug form applies
`bug`, the feature form applies `enhancement` + `v2`. Now a table of what each
form sets and what triage still has to add.
Signed-off-by: cliffhall <cliff@futurescale.com>
`verify:skills` checks that a skill is well-formed and that its cases exist;
only `skills:eval` observes whether it actually fires, and that cannot be
gated. Say so in the script header and in AGENTS.md rather than leaving the
split to be inferred.
Signed-off-by: cliffhall <cliff@futurescale.com>
- skill-manifest: the closing `---` fence must occupy a whole line. `---oops`
was accepted as a terminator, silently truncating the frontmatter and passing
validation — the exact failure the guard exists to catch. Regression cases
added (prefix, trailing whitespace, EOF, CRLF).
- skill-eval: reject a nonzero `claude -p` exit instead of resolving it as an
empty observation. An auth error or rate limit was scoring as "no skill
invoked", which passes every negative case and reads as a trigger miss on
every positive one.
- skill-eval: extract `collectSkillInvocations` / `sampleHit` behind a `main()`
guard and cover them in `skill-eval.test.mjs` — the stream parsing and the
process-outcome handling are unreachable from an eval run's happy path.
- docs/test-servers: annotate the era on every catalogue row. Six legacy rows
carried no annotation under a "unless noted, Modern" default, so the doc sent
readers to the wrong era for the feature they were trying to see.
- docs/quality-gate: `validate` runs four guards, not three; add the
`verify:skills` row and the two new `test:scripts` suites; fix the `#publishing`
fragment left dangling by the split.
- issue-create / pr-flow: delegate the card steps to `/board-ops` rather than
restating its option IDs, which are explicitly unstable.
- issue-triage: the audit counted presence, not cardinality — an issue carrying
both `v1` and `v2` passed a check that claims "exactly one". Same for the five
type labels. Also scope the two milestone checks to board #28.
- AGENTS.md: codify the v1 milestone exception. Every milestone is a v2 release
bucket, so a v1 issue has none to take and the Incoming-milestone invariant
cannot apply to board #11.
- clients/launcher/README: its pointer promised an explanation of why the cli/tui
manifests ship, which the extracted guide does not make.
Signed-off-by: cliffhall <cliff@futurescale.com>
Closes#2163
`AGENTS.md` was one 1,223-line contract loaded in full on every turn, whether
the task was a one-line CSS fix or a release cut. This splits it: rules stay
resident, procedures move to `.claude/skills/`.
- Ten skills under `.claude/skills/`, seven `disable-model-invocation: true`
(invoked by name only, so they cost no listing context and have no trigger
question) and three model-invoked with committed eval cases.
- `AGENTS.md` 1,223 → 380 lines: rules, a skills index, and a project tree
reduced to one line per entry. The removed recipes are not duplicated.
- `README.md` 811 → 105 lines, split into `docs/{architecture,quality-gate,
test-servers,publishing,docker}.md`, and dropped from `CLAUDE.md`'s auto-load.
- `.github/copilot-instructions.md` and its mirroring rule removed — Copilot
reads `AGENTS.md`.
- `npm run verify:skills` (in `validate`, so in `local:gate` and CI) parses each
SKILL.md the way Claude Code does, requires an explicit invocation mode and
eval cases for model-invoked skills, and budgets the listing.
- `npm run skills:eval` runs the committed trigger cases headless.
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 19.
- Parsing the preconfigured client registration sat outside the per-slot
salvage, so one malformed fallback aborted the whole plan and grants carrying
their own valid credentials were never revoked. It is a *fallback*; a bad one
is now recorded and skipped, matching every other slot.
- `sendPlans` preserved an earlier success when a later key was never
attempted, contradicting the failure-first rule two lines below — the
unattempted key's grant may still be live. That branch turns out to be a
bound rather than a path (`withDeadline` fails any plan that reaches the
deadline, so a preceding plan cannot both succeed and leave zero budget), so
it carries a justified `v8 ignore` saying exactly that.
- `clearStoredAuthForRelogin` takes an optional `budgetMs`, which is what lets
the shared-budget behavior be tested without a five-second test.
- Three stale references to `planOAuthRevocation`, which no longer exists —
two JSDoc links and the `AGENTS.md` entry, which additionally still told
callers to make a separate `storage.clear()` call. Following that text would
have reintroduced the second clear the atomic step exists to remove.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177mPHAdECD18nLLCTwR5rh
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 13. Three of its four findings were one root cause, and
it is a design flaw in the premise this PR started from.
Revoking *before* the clear meant the clear waited up to five seconds on the
network — and anything that wrote OAuth state in that window was then deleted
by a clear reasoning about the grant it replaced. A same-server reauthorization
in the TUI (whose Disconnect/Connect accelerators stay live), a stored-only web
clear for a server that becomes active mid-flight, and another CLI/TUI process
writing to the shared file-backed store all hit it.
`revokeStoredOAuthTokens` is therefore split into `planOAuthRevocation` and
`executeOAuthRevocation`, and every caller runs plan → `storage.clear()` →
execute. The snapshot still precedes the clear, because the store is where the
token, the credentials and the endpoint live; but the clear no longer waits on
anything. There is deliberately no single-call wrapper left, so no caller can
reintroduce the racy order by accident.
Fourth finding, same file: `handleConnect` now retires `clearOAuthAttemptRef`
alongside `disconnectAttemptRef`. The server-name check does not cover
disconnect/reconnect to the SAME server — the name still matches on the other
side of it — so a clear still in flight could tear down the session the connect
had just established.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177mPHAdECD18nLLCTwR5rh
Signed-off-by: cliffhall <cliff@futurescale.com>
Two more review findings, both real.
The shim wrapped `effectiveAuthFetch` only, but the transport is constructed
with the base `this.fetchFn`, and the SDK runs its own discovery from inside
the transport on the 401/refresh leg — so a reconnect with an existing auth
provider could still hit the upstream failure. It now wraps the base fetch,
the same seam `withOAuthEndpointOverrides` uses and the only one that reaches
both paths.
That trades away the previous claim that a captured Network entry shows the
real 404: below the trackers, what they record is the substitution. Rather
than leave that silently misleading, a substituted response now carries
`x-inspector-oauth-metadata-source` naming the URL its body was actually
fetched from — the transport builds its tracker internally, so there is no
seam above it to install into instead. An integration test captures the fetch
the transport is handed and drives it against a real server, since nothing
else proves that wiring.
The wrapper also keyed on URL shape alone, but nothing stops an authorization
or token endpoint from living under `/.well-known/oauth-authorization-server/`;
a POST there returning an ordinary OAuth 4xx would have had its error replaced
by a metadata document. Metadata discovery is a GET, so anything else is now
left alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UL9retfvAXRgvi6EWk4SY4
Signed-off-by: cliffhall <cliff@futurescale.com>
Clearing the Inspector's OAuth state deleted the local copy of the tokens
and stopped there. The access token — and the refresh token, which is
long-lived by design — stayed valid at the authorization server until they
expired on their own, so a day of connect/disconnect iteration left it
holding grants for sessions that ended hours ago. RFC 7009 §1 describes
exactly that case, and nothing in the repo constructed a revocation
request, so a server author implementing `revocation_endpoint` had no way
to exercise it with the tool that exists to exercise this.
core/auth/revocation.ts builds and sends the request; the three clear paths
call it, so the behavior is written once — web's "Clear OAuth state and
disconnect" (both the active and stored-only paths), the TUI's Auth tab,
and the CLI's --relogin. It runs BEFORE the local clear, because the token,
the client credentials and the discovered `revocation_endpoint` all live in
the store the clear is about to empty. The request names the refresh token
when there is one: §2.1 asks the AS to also invalidate the access tokens
issued under the same grant, so one request covers both halves.
Best-effort by construction. Every path returns a TokenRevocationOutcome
rather than throwing: no advertised endpoint means nothing is sent and that
authorization server behaves exactly as before, and a network error, a
non-2xx, a 5s timeout or an unreadable store is reported and nothing more.
Forgetting the tokens is what the user asked for. `lost_authorization_state`
recovery deliberately skips it — it clears a half-finished flow to retry it,
and an authorization that never completed has no grant to revoke.
Client authentication resolves the preregistered slot first, mirroring
BaseOAuthClientProvider.clientInformation; reading only the dynamic slot
would send no authentication for a server configured with `oauth.clientId`.
The Basic credential is byte-identical to the SDK's applyBasicAuth so this
request and the token request cannot present the same secret differently.
Opt out per server with `oauth.revokeOnClear` (default on, only `false`
written to disk) or per CLI run with --no-revoke. Read at clear time rather
than connect time, so toggling needs no reconnect. Turning it off is a
testing affordance: a client that walks away still holding live tokens is a
case a server author may want to reproduce.
test-servers now advertises `revocation_endpoint` and serves POST
/oauth/revoke, faithful on §2.2 (an unknown token is a success) and §2.1
(revoking a refresh token invalidates the access tokens under the same
grant), with oauth-revocation-http.json / oauth-no-revocation-http.json as
showcase configs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177mPHAdECD18nLLCTwR5rh
Signed-off-by: cliffhall <cliff@futurescale.com>
The SDK's `discoverAuthorizationServerMetadata` picks its validation schema
from the well-known filename that resolved rather than from the document that
came back, so anything found at `/.well-known/openid-configuration` is
validated as an OpenID provider document — requiring `jwks_uri`,
`subject_types_supported` and `id_token_signing_alg_values_supported`, three
fields RFC 8414 does not define. RFC 8414 §5 permits that filename for general
OAuth metadata, so a conforming plain OAuth 2.0 authorization server is
rejected; and because the parse throws instead of continuing the candidate
loop, discovery aborts and the connection fails outright.
Filed upstream as modelcontextprotocol/typescript-sdk#2733.
`core/auth/oidcDiscoveryCompat.ts` works around it without fabricating a
field: on a failed RFC 8414 candidate it probes the OIDC candidates the SDK
would try next and, when one returns a document that satisfies
`OAuthMetadataSchema` but fails the OIDC schema, serves that body as the RFC
8414 response so the SDK picks the schema that describes it. A genuine OpenID
provider document is left to the SDK's own OIDC leg. Issuer validation runs
unchanged, since the substituted document is the one the server published.
It wraps `effectiveAuthFetch` — above the fetch tracker, the opposite of
`withOAuthEndpointOverrides` — so the Network tab still records the real 404
and the real probe rather than the substitution.
The candidate derivation mirrors the SDK's `buildDiscoveryUrls`, which a fetch
wrapper cannot call, and a test pins it against that export so the two cannot
drift.
Adds `oauth.asMetadataPath` to the composable test server (mirroring
`oauth.resourceMetadataPath`) and the `oauth-rfc8414-at-oidc-path-http.json`
showcase config that reproduces the failure.
Closes#2172
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UL9retfvAXRgvi6EWk4SY4
Signed-off-by: cliffhall <cliff@futurescale.com>
A modern-era `subscriptions/listen` is a long-lived request: the first
message on its stream must be `notifications/subscriptions/acknowledged`,
and the JSON-RPC `result` for the listen id is reserved as the
graceful-closure marker. A server that answers the listen request with a
bare `result` is saying "acknowledged and closed in the same breath" — the
SDK settles the subscription `graceful` and rejects the pending `listen()`.
The Inspector read that rejection as an ordinary drop: eight re-lists over
roughly a minute, the badge flickering `Reconnecting…`, and a final bare
`Stream ended` that said nothing about why. Accepting the message was
never wrong — it is a well-formed graceful-closure response. The silence
was.
- `core/mcp/subscriptionAck.ts` owns the predicate. The SDK gives this
close no code of its own — it is the same `SdkError(ConnectionClosed)`
any pre-ack close carries — so the predicate reads the message, since
the alternative is refusing to retry a genuinely transient drop.
- The condition is deterministic, so the stream ends on the first
occurrence instead of spending the whole backoff run. Both routes are
covered: the one a subscribe/unsubscribe reaches, and the one a
reconnect reaches when an acknowledged stream drops and the server has
since started refusing.
- A `"never-acknowledged"` status of its own, not `"ended"` — `ended`
means an expected close. It surfaces as the thrown error's message, an
orange "Not acknowledged" badge, and a notice in the panel.
- `subscriptions-never-acknowledged-http.json` reproduces it. It
acknowledges the first listen that subscribes to a resource and refuses
every one after, which is what makes the badge reachable by hand.
Closes#2097
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, ninth and tenth reviews:
- A custom `shell:` is a COMMAND TEMPLATE that Actions runs around every
script (`{0}` is the script path), so `shell: npm run local:gate && bash {0}`
invoked the gate while the `run:` value beside it looked harmless. Step
`shell`, and `defaults.run.shell` at both workflow and job level, are
command regions now. An ordinary `bash`/`pwsh`/`cmd` matches no rule, so
naming them costs nothing.
- GitHub's multiline environment-file syntax (`SMOKE_BROWSER<<EOF`, value,
delimiter) bypassed the assignment patterns. It is reported as an override
whose value the guard cannot READ rather than parsed: the value can arrive
through any number of shapes — a brace group redirected to $GITHUB_ENV, a
heredoc nested in another — and a parser handling some of them would be a
coverage claim that holds only for the shapes someone thought of. Nobody
spells `chromium` that way, and if they did, `SMOKE_BROWSER=chromium` is the
fix.
Signed-off-by: Cliff Hall <cliff@futurescale.com>
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, seventh review:
- The unreadable-name rules only knew GitHub expression syntax, so
`ENGINE: firefox` plus `npm run smoke:web:$ENGINE` ran a forbidden pass with
the guard silent — the shell resolves the name, which makes it exactly as
unreadable as a ${{ }} expression and forbidden for the same reason. The
pattern keys off a leading $ or %, covering $VAR, ${VAR}, $env:VAR and cmd's
%VAR% in one, for both the engine and `local:` families.
- The env-key comparison was case-sensitive, but Windows environment variables
are not: `env: { smoke_browser: firefox }` on a windows-latest job sets the
very variable Node reads as SMOKE_BROWSER. Compared case-insensitively now,
as is the POSIX assignment pattern. On a POSIX runner that spelling would be
a different variable and harmless, so this is stricter there than it needs to
be — the right direction for a guard, and the alternative is a verdict that
depends on a `runs-on` which may itself be an expression.
Signed-off-by: Cliff Hall <cliff@futurescale.com>
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, sixth review: the run-script rule matched POSIX `NAME=value` only.
PowerShell is the DEFAULT shell on a Windows runner, so
`$env:SMOKE_BROWSER = "firefox"; npm run smoke` redirected the suite with the
guard silent — on exactly the workflow least likely to be read closely.
Three patterns now, one per dialect: POSIX (which also covers cmd's
`set NAME=…` and the GitHub-native `>> $GITHUB_ENV` append, since both carry
the same substring), PowerShell's `$env:NAME =` and `${env:NAME} =`, and the
.NET SetEnvironmentVariable call reachable from PowerShell. Regression cases
for all five spellings, plus a negative for a PowerShell assignment of
chromium.
Signed-off-by: Cliff Hall <cliff@futurescale.com>
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, fifth review:
- The SMOKE_BROWSER rule was a blanket text match across every region, so a
script that merely PRINTS the variable — `echo "SMOKE_BROWSER: firefox"` —
or a `with:` input mentioning it was reported as an override. It is
kind-aware now: an `env:` entry counts only when SMOKE_BROWSER is literally
its key (recognized structurally, not by regex), a `run:` script only on a
shell assignment, and a `with:` value not at all — whatever it names is
already covered by the script rules. A shell assignment inside an `echo` is
indistinguishable without parsing the shell, and inside `run:` erring toward
the finding is the safe direction; that is stated in the header.
- `jobs.<id>.container.env` is job-wide too — every step in the container sees
it — so an override there redirected `npm run smoke` with the guard silent.
Now traversed, and a `container:` given as a bare image string is ignored
rather than tripping over.
- Nothing asserted the FIRST acceptance criterion: reintroducing a `ci` or
`ci:*` root script left test:scripts green. Asserted directly over the
manifest's keys now, and checked to actually bite by reviving `ci` and
watching the suite fail.
Signed-off-by: Cliff Hall <cliff@futurescale.com>
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, fourth review — four findings, all real:
- Matching pairs by KEY NAME anywhere in the tree scanned things that execute
nothing. Workflow input ids are user-defined, so an input legitimately named
`env` put its own description and default in front of the rules — failing
the gate on documentation OF the gate. The walk is structural now: workflow
`env`, job `env`/`with`, step `run`/`env`/`with`, and nothing else.
- YAML aliases were read as scalars, which have no `.value` on an alias node —
so `run: *command`, with the command anchored in a metadata scalar the walk
never visits, invoked anything at all while the guard saw an empty string.
Aliases are resolved, and reported at the line where they are USED.
- `file` was accepted for diagnostics and then not forwarded, so a malformed
real workflow was reported as `<workflow>`.
- One negative fixture was a YAML comment, which the parser strips — so it
asserted nothing. Inside a `run:` block the comment text really does survive
into the scalar, so the "a comment invokes nothing" rule is now applied
there, where it actually has to be, and the fixture is a real block scalar.
The table's fixtures are schema-valid workflows now rather than bare `- run:`
fragments, which a structural walk correctly cannot see.
Signed-off-by: Cliff Hall <cliff@futurescale.com>
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, third review — and the second finding is really a verdict on the
approach, so the fix is the approach:
- `SMOKE_BROWSER:` with no value is valid YAML that sets the variable to the
empty string, which `resolveBrowserName` rejects as present-but-empty rather
than treating as unset. The value pattern required \S+, so the guard said
nothing. It is \S* now, with its own message.
- The hand-rolled opener could not see `run: | # explanation` or the flow
mapping `env: { SMOKE_BROWSER: firefox }`. That is the third spelling it has
missed in three rounds, each one a forbidden invocation that would have
passed while test:scripts stayed green — worse than no guard, because it
reports a coverage that is not there. YAML has too many spellings to
recognize by indentation, so extractExecutableRegions now parses with the
`yaml` package. It is already a root dependency, and this file is not in the
published files allowlist, so nothing about the tarball changes.
A parse error now throws rather than yielding no regions: a workflow the guard
cannot read must not pass as a workflow with nothing in it.
Verified by injecting a flow-mapping SMOKE_BROWSER override into the real
main.yml — the suite fails, and goes green again when removed.
Signed-off-by: Cliff Hall <cliff@futurescale.com>
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, second review — both findings were real:
- A `${{ … }}` body can itself contain braces, so the `[^}]*\}\}` arm added in
the previous round stopped at the first `}` and missed
`smoke:web:${{ format('{0}', matrix.browser) }}` — the very case it was
added for. It now keys off the `${{` prefix, which nesting cannot defeat.
- Scanning every non-comment line made ordinary metadata a finding:
`name: Explain why smoke:web:firefox stays local` executes nothing and was
reported as an engine invocation, contradicting the header's "what it
forbids, and nothing more". The scan now walks executable positions only —
`run:` scalars (inline and block) and `env:`/`with:` values — via an
exported `extractExecutableRegions`, hand-rolled because `test:scripts` is
dependency-free by design. The narrowing is a stated limit, not a hidden
one: an execution vector outside those three positions is not seen.
Verified by injecting a nested-brace SMOKE_BROWSER expression into the real
main.yml — the suite fails, and goes green again when removed.
Signed-off-by: Cliff Hall <cliff@futurescale.com>
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, first review:
- The guard read only literal names, so `npm run smoke:web:${{ matrix.browser }}`
and `npm run local:${{ matrix.task }}` walked past it — the one form in which
a forbidden engine reaches CI without anyone typing its name. It already
refused an expression-valued SMOKE_BROWSER for that exact reason; this applies
the reasoning consistently. Its limit is now stated: a fully opaque
`npm run ${{ matrix.script }}` names no family and is not detectable.
- The module header claimed the gate adds "the non-Chromium engine passes";
it adds Firefox only, and WebKit belongs to neither tier.
- eslint.config.js:33 still said `npm run ci` — the last stale reference.
- AGENTS.md read "everything from `smoke` rightward is `local:gate` only",
which is two claims wrong: smoke and Storybook run in GitHub CI too, and
`pack:verify` is in neither tier.
Signed-off-by: Cliff Hall <cliff@futurescale.com>
Signed-off-by: cliffhall <cliff@futurescale.com>
The root `ci` script was never what runs in GitHub CI, and the name worked
against us three ways: it collides with the built-in `npm ci` (which does
something else entirely and is one keystroke away), it invites the
cross-engine passes into a workflow by looking like the thing a workflow
should call, and it blurs "GitHub CI" and "the local gate" in every doc
that names it.
- Rename `ci` → `local:gate` and `ci:storybook` → `local:storybook`, with
no back-compat alias: `npm run ci` now fails with npm's missing-script
error, which prints the available scripts.
- Add `scripts/lib/workflow-gate.mjs` + its `test:scripts` suite, which
runs over the repo's real `.github/workflows/**` and fails on an
invocation of a `local:*` script, of a non-Chromium engine pass (derived
from SUPPORTED_BROWSERS), or of `smoke:web:engine`, or on `SMOKE_BROWSER`
set to anything but a literal `chromium`. It does not forbid
`npm run smoke`, `smoke:web:chromium` or `smoke:tui` — those belong in CI
and are there today. Comments are not scanned, so the workflow can still
explain itself. This is the durable half; the rest is prose.
- Document the two-tier model once, in the root README ("Two tiers: GitHub
CI and the local gate"), and point the other docs at it rather than
restating the split.
- Sweep the ~40 stale references across README, AGENTS.md,
.github/copilot-instructions.md, the client READMEs, the roadmap doc and
the script headers. AGENTS.md and copilot-instructions.md change
together, per the mirror rule.
No gate changes what it actually runs.
Signed-off-by: Cliff Hall <cliff@futurescale.com>
Signed-off-by: cliffhall <cliff@futurescale.com>
