chore(scripts): scan only executable positions, and match on the expression prefix (#2146)

Copilot, second review — both findings were real:

- A `${{ … }}` body can itself contain braces, so the `[^}]*\}\}` arm added in
  the previous round stopped at the first `}` and missed
  `smoke:web:${{ format('{0}', matrix.browser) }}` — the very case it was
  added for. It now keys off the `${{` prefix, which nesting cannot defeat.

- Scanning every non-comment line made ordinary metadata a finding:
  `name: Explain why smoke:web:firefox stays local` executes nothing and was
  reported as an engine invocation, contradicting the header's "what it
  forbids, and nothing more". The scan now walks executable positions only —
  `run:` scalars (inline and block) and `env:`/`with:` values — via an
  exported `extractExecutableRegions`, hand-rolled because `test:scripts` is
  dependency-free by design. The narrowing is a stated limit, not a hidden
  one: an execution vector outside those three positions is not seen.

Verified by injecting a nested-brace SMOKE_BROWSER expression into the real
main.yml — the suite fails, and goes green again when removed.

Signed-off-by: Cliff Hall <cliff@futurescale.com>
Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-08-27 00:35:47 -04:00
commit 398d074ad9
5 changed files with 156 additions and 19 deletions

View file

@ -102,7 +102,7 @@ Both exist and do different jobs. Theme files (`src/theme/<Component>.ts`) custo
## Gates and PR hygiene
- `npm run format` before committing; **`npm run local:gate` before pushing** (`validate` → `coverage` → `verify:build-gate` → `verify:bundle-externals` → `smoke` → `smoke:web:firefox` → `local:storybook`; the Firefox step is local-only and not mirrored in GitHub CI). The gate is named `local:` rather than `ci` on purpose and has **no alias** (#2146) — `npm run ci` is gone, and a workflow that invokes a `local:*` script, a non-Chromium engine pass, `smoke:web:engine`, or sets `SMOKE_BROWSER` to a non-Chromium value fails `test:scripts` via `scripts/lib/workflow-gate.mjs` — as does naming either family through a workflow expression (`smoke:web:${{ matrix.browser }}`). Flag any diff that adds one of those to `.github/workflows/**`, or that reintroduces a `ci`/`ci:*` root script. `npm run validate` is the fast inner-loop check and is **not** a substitute — it runs `test`, not `test:coverage`, so it does zero coverage gating. The canonical CI-vs-local table is in the root README ("Two tiers: GitHub CI and the local gate").
- `npm run format` before committing; **`npm run local:gate` before pushing** (`validate` → `coverage` → `verify:build-gate` → `verify:bundle-externals` → `smoke` → `smoke:web:firefox` → `local:storybook`; the Firefox step is local-only and not mirrored in GitHub CI). The gate is named `local:` rather than `ci` on purpose and has **no alias** (#2146) — `npm run ci` is gone, and a workflow that invokes a `local:*` script, a non-Chromium engine pass, `smoke:web:engine`, or sets `SMOKE_BROWSER` to a non-Chromium value fails `test:scripts` via `scripts/lib/workflow-gate.mjs` — as does naming either family through a workflow expression (`smoke:web:${{ matrix.browser }}`). It scans only `run:` scalars and `env:`/`with:` values, so metadata is not a finding. Flag any diff that adds one of those to `.github/workflows/**`, or that reintroduces a `ci`/`ci:*` root script. `npm run validate` is the fast inner-loop check and is **not** a substitute — it runs `test`, not `test:coverage`, so it does zero coverage gating. The canonical CI-vs-local table is in the root README ("Two tiers: GitHub CI and the local gate").
- **A dependency bump must land in every install that declares it.** v2 is not a workspace — the root and each `clients/*` have their own `node_modules`, and a client's test project compiles `core/` and `test-servers/src` (which resolve from the **root**) alongside the client's own sources. Bumping a shared dependency in one manifest only puts two versions of it in one `tsc` program; for a recursive-generic surface like zod that exhausts the tsc heap (#1896). `verify:dep-lockstep` fails the build on this — it derives its candidates from what each `tsc` program actually resolves (`tsc --listFilesOnly`, keeping packages that reach one program from two installs), so a package reached only through another package's `.d.ts` counts too (#1965) — so a PR bumping a package the shared sources pull in should update the root **and every client that already lists it** — not every client unconditionally, since a package absent from an install can't skew and adding it there would be a spurious dependency.
- **A test or smoke must not touch real user state.** The web smokes run against a throwaway catalog via the shared `scripts/lib/prod-web-server.mjs` helper, never the developer's `~/.mcp-inspector/mcp.json` (#1977); the cli/tui smokes drive a temp `--catalog`; `pack:verify`'s `--web` child sets its own `MCP_CATALOG_PATH` for the same reason (#2003 — its App deep link persists a server row). Anything that boots the web backend and then *navigates* it needs that isolation, not just the scripts named `smoke:*`. A new smoke spawning its own server, or teardown that removes a work dir without first awaiting `stopChild` (the #1801 race — `child-cleanup.mjs` exports both halves and both are required), should be flagged.
- **A smoke must assert the thing survives, not that it started.** `smoke:tui` runs the Ink TUI under a pseudoterminal (`scripts/lib/pty.mjs`) and requires it to still be running `SMOKE_TUI_SURVIVE_MS` after its first frame (`scripts/lib/render-smoke.mjs`) — because it previously settled OK on first paint while the TUI was exiting 1 ~40ms later with `Raw mode is not supported`, and won that race on every machine (#2147). Flag anything that collapses the survival wait back into "resolve on the marker", and any new smoke whose only assertion is that a process printed something once. A smoke walks only its own happy path, so the branch that matters — started, then died — belongs in a `scripts/lib/*.test.mjs` driven against a stub (as with `announced-child.mjs` #2000 and `ensure-test-servers.mjs` #2111), not in the smoke.

View file

@ -1003,7 +1003,7 @@ Everything about the selection lives in **`scripts/lib/headless-browser.mjs`**
- **An unrecognized `SMOKE_BROWSER` is an error, never a fallback.** Falling back to Chromium would report a green Chromium run under a job labelled `webkit` — coverage claimed but not run, which is worse than none.
- **A launch failure names the engine that failed** and a remedy that works **from where the caller actually is**: `npm run smoke:web:<engine>`, or `npx playwright install --with-deps <engine>` run *from `clients/web`*. Naming `chromium` while WebKit was missing would send the reader to install a browser they already have — and naming a bare root-level `npx playwright` would send them to a version that is not the pinned one, which can install a browser revision the pinned Playwright still cannot launch.
- **Firefox is gated locally, not in CI, and that asymmetry is deliberate.** A GitHub Actions job was trialled and removed. It was cheap — ~2 minutes, parallel with the 15-minute `build` job, so zero added wall-clock — but across a dozen runs Firefox never once disagreed with Chromium, so it spent runner minutes on every push, from every branch, carrying a real flake surface (`playwright install --with-deps` runs `apt-get update`, which fails whenever a third-party repo in the runner image breaks) to re-confirm a result already in hand. Moving it into `npm run local:gate` keeps the check where a human is about to push a change they can still reason about, and pays for it once rather than on every push. **Don't re-add the CI job on the argument that cross-engine coverage is good in principle** — that argument was accepted, and the pre-push gate is what serves it. Re-add it on evidence: a cross-engine regression that reached `v2/main` because someone skipped the gate.
- **A guard enforces the split, so it cannot be undone by accident (#2146).** `scripts/lib/workflow-gate.mjs` is run by `npm run test:scripts` over every file under `.github/workflows/`, and fails the suite on: an invocation of a `local:*` script (the prefix *means* local-only, so a future sibling is covered without editing the guard); an invocation of a non-Chromium engine pass, derived from `SUPPORTED_BROWSERS` so a fourth engine extends the guard rather than opening a hole in it; an invocation of `smoke:web:engine`, whose engine comes from the environment and so cannot be read off the workflow at all; or `SMOKE_BROWSER` set to anything but a literal `chromium` — the back door that would redirect an otherwise innocent `npm run smoke`. A name built from a workflow **expression** counts as forbidden too (`smoke:web:${{ matrix.browser }}`, `local:${{ matrix.task }}`): a matrix the guard cannot read may well contain `firefox`, so an unreadable name is not waved through. Its limit is worth knowing — a fully opaque `npm run ${{ matrix.script }}` names no family at all and is not detectable. It deliberately does **not** forbid `npm run smoke`, `smoke:web:chromium`, or `smoke:tui`: those belong in CI and are there today, and `smoke:tui` self-skips under `process.env.CI` on its own. Lines that are comments are not scanned — a comment invokes nothing, and a guard that made the workflow undocumentable would be traded for the wrong thing.
- **A guard enforces the split, so it cannot be undone by accident (#2146).** `scripts/lib/workflow-gate.mjs` is run by `npm run test:scripts` over every file under `.github/workflows/`, and fails the suite on: an invocation of a `local:*` script (the prefix *means* local-only, so a future sibling is covered without editing the guard); an invocation of a non-Chromium engine pass, derived from `SUPPORTED_BROWSERS` so a fourth engine extends the guard rather than opening a hole in it; an invocation of `smoke:web:engine`, whose engine comes from the environment and so cannot be read off the workflow at all; or `SMOKE_BROWSER` set to anything but a literal `chromium` — the back door that would redirect an otherwise innocent `npm run smoke`. A name built from a workflow **expression** counts as forbidden too (`smoke:web:${{ matrix.browser }}`, `local:${{ matrix.task }}`): a matrix the guard cannot read may well contain `firefox`, so an unreadable name is not waved through. Its limit is worth knowing — a fully opaque `npm run ${{ matrix.script }}` names no family at all and is not detectable. It deliberately does **not** forbid `npm run smoke`, `smoke:web:chromium`, or `smoke:tui`: those belong in CI and are there today, and `smoke:tui` self-skips under `process.env.CI` on its own. It reads only the positions that execute something — `run:` scalars (inline and block) and the values of an `env:` or `with:` mapping — so ordinary metadata such as a step `name:` explaining the rule is not reported as an invocation of it (Copilot); comments are skipped for the same reason, and because a guard that made the workflow undocumentable would be traded for the wrong thing. That scoping is a stated limit rather than a hidden one: an execution vector outside those three positions is not seen.
- **This makes `npm run local:gate` a strict superset of GitHub CI rather than a mirror of it.** That was already the direction (`smoke:tui` self-skips on CI and runs only locally); Firefox is the second such step. The invariant that still holds, and the one that matters, is the useful direction: **passing `npm run local:gate` locally means CI's gates will pass.**
- **`pack:verify` stays pinned to Chromium**, passing it explicitly rather than reading `SMOKE_BROWSER`. It is a _packaging_ check; the engine question belongs where the sandbox is under test, and pinning it also means it can't be pointed at an engine its npm script never installed.
- **WebKit fails the two App smokes**, for reasons nobody has identified — the failure stage even varies between runs. The investigation was dropped as not worth the effort once it became clear it does not reproduce in real Safari, and that an isolated repro of the suspected mechanism did not reproduce it under Playwright's WebKit either. Treat a WebKit failure as unexplained rather than as a defect until someone has actually looked.

View file

@ -502,7 +502,7 @@ So the direction that matters holds: **passing `npm run local:gate` means CI's g
**The gate is deliberately not named `ci`** ([#2146](https://github.com/modelcontextprotocol/inspector/issues/2146)). That name collided with `npm ci` — a built-in that clean-installs from the lockfile, does **not** run this script, and is one keystroke away — and it invited the cross-engine passes into GitHub CI by looking like the thing a workflow ought to call. There is **no back-compat alias**: `npm run ci` now fails with npm's missing-script error, which prints the available scripts.
That is the readable half, and prose rots. The enforced half is `scripts/lib/workflow-gate.mjs`, run by `npm run test:scripts` over every file under `.github/workflows/`. It fails the suite if a workflow invokes a `local:*` script, a non-Chromium engine pass (`smoke:web:firefox`, `smoke:web:webkit`), or `smoke:web:engine` (whose engine comes from the environment and so cannot be read off the workflow at all), or sets `SMOKE_BROWSER` to anything but a literal `chromium`. It treats a name built from a workflow **expression** the same way — `smoke:web:${{ matrix.browser }}`, `local:${{ matrix.task }}` — since a matrix the guard cannot read may well contain `firefox`. It deliberately does **not** forbid `npm run smoke`, `smoke:web:chromium`, or `smoke:tui` — those belong in CI and are there today. Comments are not scanned, so the workflow can still explain itself.
That is the readable half, and prose rots. The enforced half is `scripts/lib/workflow-gate.mjs`, run by `npm run test:scripts` over every file under `.github/workflows/`. It fails the suite if a workflow invokes a `local:*` script, a non-Chromium engine pass (`smoke:web:firefox`, `smoke:web:webkit`), or `smoke:web:engine` (whose engine comes from the environment and so cannot be read off the workflow at all), or sets `SMOKE_BROWSER` to anything but a literal `chromium`. It treats a name built from a workflow **expression** the same way — `smoke:web:${{ matrix.browser }}`, `local:${{ matrix.task }}` — since a matrix the guard cannot read may well contain `firefox`. It deliberately does **not** forbid `npm run smoke`, `smoke:web:chromium`, or `smoke:tui` — those belong in CI and are there today. It reads only the positions that actually execute something — `run:` scalars and the values of an `env:` or `with:` mapping — so a step *name* explaining the rule is not mistaken for an invocation of it, and the workflow can still document itself.
| Script | What it does |
| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

View file

@ -48,11 +48,21 @@
* Those belong in CI and are there today — `smoke:tui` self-skips under
* `process.env.CI` on its own, so it needs no guarding.
*
* COMMENTS ARE NOT SCANNED. A line whose first non-whitespace character is `#`
* is skipped, whether it is a YAML comment or a shell comment inside a `run:`
* block. A comment cannot invoke anything, and the workflow's value is largely
* in its prose — a guard that made the two tiers undocumentable in the file
* that implements one of them would be traded for the wrong thing.
* ONLY EXECUTABLE POSITIONS ARE SCANNED — `run:` scalars (inline and block),
* and the values of an `env:` or `with:` mapping. Everything else in a workflow
* is metadata that runs nothing, so scanning it produces findings that are
* simply false: `name: Explain why smoke:web:firefox stays local` executes
* nothing, and failing the suite on it would contradict the "what it forbids,
* and nothing more" contract above (Copilot). Comments are skipped for the same
* reason and one more: the workflow's value is largely in its prose, and a
* guard that made the two tiers undocumentable in the file implementing one of
* them would be traded for the wrong thing.
*
* That is a real limit, stated rather than hidden: an execution vector outside
* those three positions — a custom action that reads a script name from
* somewhere else, say — is not seen. The alternative, scanning everything, was
* measured against this repo's own workflow and produces false positives on
* ordinary step names, which is the faster way to get a guard deleted.
*/
import { SUPPORTED_BROWSERS, DEFAULT_BROWSER } from "./headless-browser.mjs";
@ -81,10 +91,24 @@ export const VIOLATION = {
// The name may be spelled out (`local:gate`) or built from an expression
// (`local:${{ matrix.task }}`) — both are `local:` scripts, and neither may be
// invoked from a workflow.
const LOCAL_SCRIPT_RE = /\blocal:(?:[a-z0-9][a-z0-9:-]*|\$\{\{[^}]*\}\})/gi;
//
// The expression arm keys off the `${{` PREFIX and then runs to the end of the
// line, rather than trying to find the matching `}}`. An expression body can
// itself contain braces — `${{ format('{0}', matrix.browser) }}` — so a
// `[^}]*\}\}` body stops at the first `}` and misses exactly the case worth
// catching (Copilot). Matching the prefix cannot be defeated by nesting, and
// since these are display strings the greedy tail costs nothing.
const EXPRESSION = String.raw`\$\{\{[^\n]*`;
const LOCAL_SCRIPT_RE = new RegExp(
String.raw`\blocal:(?:[a-z0-9][a-z0-9:-]*|${EXPRESSION})`,
"gi",
);
// Same reasoning one level down: `smoke:web:` followed by an expression could
// resolve to any engine, so it is read as a forbidden one.
const DYNAMIC_ENGINE_RE = /\bsmoke:web:\$\{\{[^}]*\}\}/gi;
const DYNAMIC_ENGINE_RE = new RegExp(
String.raw`\bsmoke:web:${EXPRESSION}`,
"gi",
);
// The value may be quoted (`SMOKE_BROWSER: "firefox"`), assigned
// (`SMOKE_BROWSER=firefox`), or an expression — capture whatever follows.
const BROWSER_ENV_RE = /\bSMOKE_BROWSER\s*[:=]\s*(\S+)/gi;
@ -93,6 +117,61 @@ function isComment(line) {
return line.trimStart().startsWith("#");
}
function indentOf(line) {
return line.length - line.trimStart().length;
}
/** `run:` with an inline command, optionally as the first key of a `- ` item. */
const RUN_INLINE_RE = /^\s*(?:-\s+)?run:\s*(.*)$/;
/** A key that opens a block whose nested lines are executable: `run: |`, `env:`, `with:`. */
const BLOCK_OPEN_RE = /^\s*(?:-\s+)?(run|env|with):\s*(?:[|>][-+]?\d*)?\s*$/;
/**
* The executable scalars of a workflow, with the line each came from.
*
* Deliberately hand-rolled rather than YAML-parsed: the guard runs in
* `test:scripts`, which is `node --test` with no dependencies by design, and
* the three shapes it needs are simple enough to recognize by indentation.
*
* @param {string} text
* @returns {Array<{line: number, kind: string, text: string}>}
*/
export function extractExecutableRegions(text) {
const regions = [];
let blockKind = null;
let blockIndent = 0;
text.split(/\r?\n/).forEach((line, index) => {
if (isComment(line)) return;
const lineNumber = index + 1;
if (blockKind !== null) {
// A blank line neither ends a block scalar nor carries anything.
if (line.trim() === "") return;
if (indentOf(line) > blockIndent) {
regions.push({ line: lineNumber, kind: blockKind, text: line });
return;
}
// Dedented out of the block — fall through and re-read this line as a key.
blockKind = null;
}
const block = BLOCK_OPEN_RE.exec(line);
if (block) {
blockKind = block[1];
blockIndent = indentOf(line);
return;
}
const run = RUN_INLINE_RE.exec(line);
if (run && run[1].trim() !== "") {
regions.push({ line: lineNumber, kind: "run", text: run[1] });
}
});
return regions;
}
function stripQuotes(value) {
const trimmed = value.trim().replace(/[,;]+$/, "");
const quoted = /^(["'])(.*)\1$/.exec(trimmed);
@ -111,12 +190,10 @@ function stripQuotes(value) {
*/
export function findWorkflowViolations(text, file = "<workflow>") {
const findings = [];
const lines = text.split(/\r?\n/);
lines.forEach((line, index) => {
if (isComment(line)) return;
const lineNumber = index + 1;
for (const { line: lineNumber, text: line } of extractExecutableRegions(
text,
)) {
for (const match of line.matchAll(LOCAL_SCRIPT_RE)) {
findings.push({
file,
@ -174,7 +251,7 @@ export function findWorkflowViolations(text, file = "<workflow>") {
`checked statically. Leave it unset; the default is \`${DEFAULT_BROWSER}\`.`,
});
}
});
}
return findings;
}

View file

@ -19,6 +19,7 @@ import { describe, it } from "node:test";
import {
FORBIDDEN_ENGINE_SCRIPTS,
VIOLATION,
extractExecutableRegions,
findWorkflowViolations,
formatWorkflowViolations,
} from "./workflow-gate.mjs";
@ -72,7 +73,7 @@ describe("findWorkflowViolations", () => {
},
{
name: "flags a quoted SMOKE_BROWSER value",
text: ' SMOKE_BROWSER: "webkit"\n',
text: ' env:\n SMOKE_BROWSER: "webkit"\n',
rules: [VIOLATION.BROWSER_ENV],
},
{
@ -82,12 +83,12 @@ describe("findWorkflowViolations", () => {
},
{
name: "flags an expression, which cannot be checked statically",
text: " SMOKE_BROWSER: ${{ matrix.browser }}\n",
text: " env:\n SMOKE_BROWSER: ${{ matrix.browser }}\n",
rules: [VIOLATION.BROWSER_ENV],
},
{
name: "allows SMOKE_BROWSER set explicitly to chromium",
text: " SMOKE_BROWSER: chromium\n",
text: " env:\n SMOKE_BROWSER: chromium\n",
rules: [],
},
{
@ -125,6 +126,41 @@ describe("findWorkflowViolations", () => {
].join("\n"),
rules: [],
},
{
// Copilot, second review: an expression body can itself contain braces,
// so a brace-balanced regex stops at the first `}` and misses the one
// case worth catching. The rule keys off the `${{` prefix instead.
name: "flags an expression whose own body contains braces",
text: " - run: npm run smoke:web:${{ format('{0}', matrix.browser) }}\n",
rules: [VIOLATION.ENGINE_SCRIPT],
},
{
name: "flags a command inside a block scalar, not just an inline run",
text: " - run: |\n npm run local:gate\n",
rules: [VIOLATION.LOCAL_SCRIPT],
},
{
name: "flags a command passed as an action input",
text: " with:\n args: npm run local:gate\n",
rules: [VIOLATION.LOCAL_SCRIPT],
},
{
// Copilot, second review: metadata executes nothing, so a finding there
// is simply false — and the fastest way to get a guard deleted.
name: "does not read a step name as an invocation",
text: " - name: Explain why smoke:web:firefox stays local\n",
rules: [],
},
{
name: "does not read an `if:` condition as one either",
text: " if: github.event_name == 'push'\n",
rules: [],
},
{
name: "does not read a bare mapping line outside env/with/run",
text: " SMOKE_BROWSER-ish-doc: firefox\n",
rules: [],
},
{
name: "does not read a comment as an invocation",
text: " # smoke:web:firefox runs in npm run local:gate, never here\n",
@ -169,6 +205,30 @@ describe("findWorkflowViolations", () => {
});
});
describe("extractExecutableRegions", () => {
it("ends a block when the next line dedents, and re-reads that line", () => {
// The table cases cannot isolate this: a block that never closed would
// still produce the right findings for them, and would then read the whole
// rest of the file as executable.
const regions = extractExecutableRegions(
[
" env:",
" SMOKE_BROWSER: chromium",
" - name: smoke:web:firefox is local-only",
" - run: npm run smoke",
"",
].join("\n"),
);
assert.deepEqual(
regions.map((r) => [r.kind, r.text.trim()]),
[
["env", "SMOKE_BROWSER: chromium"],
["run", "npm run smoke"],
],
);
});
});
describe(".github/workflows", () => {
const files = readdirSync(workflowDir).filter((f) => /\.ya?ml$/.test(f));