ci: drop the Firefox smoke job; keep the engines as an on-demand tool

The job was trialled and did not earn its place. It was genuinely cheap —
~2 minutes, running in parallel with the 15-minute build job, so zero
added wall-clock — but across a dozen runs Firefox never once disagreed
with Chromium. It was paying runner time and carrying a real flake
surface (playwright install --with-deps runs apt-get update, which fails
whenever a third-party repo in the runner image breaks; that is what the
one red run was) to re-confirm a result already in hand.

Everything that made cross-engine checking possible stays: SMOKE_BROWSER,
the shared headless-browser helper, the installer, and the tests. So
"SMOKE_BROWSER=firefox npm run smoke:web:engine" is there for the moment
it is worth running — before touching the sandbox, the CSP builder or the
proxy page — without taxing every push to re-derive the same answer.

The workflow is now byte-identical to v2/main again; the publish jobs go
back to needing only build.

Docs say what is true rather than what was intended, and AGENTS.md
records the reasoning so the job is not re-added on the general argument
that cross-engine coverage is good — that argument was already accepted,
and the on-demand path is what serves it. Re-add it on evidence: a real
cross-engine regression a gate would have caught.

Also swept the now-stale "matrix" language out of the smokes and the
resolver, including the empty-SMOKE_BROWSER rationale, which no longer
leans on a matrix key that does not exist.

Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-08-25 23:54:51 -04:00
commit 8a7ade7b08

View file

@ -479,9 +479,11 @@ SMOKE_BROWSER=webkit npm run smoke:web:app # one smoke, one engine
SMOKE_BROWSER=firefox npm run smoke:web:engine # all three smokes, one engine
```
Unset, the engine is `chromium` — so `npm run ci` and `npm run smoke` are unchanged. CI runs Chromium inside the `build` job's `npm run smoke`, and every other gated engine in a separate `Sandbox smokes (<engine>)` matrix job — today that matrix holds **Firefox alone**; see the paragraph below for where WebKit stands. Splitting it that way, rather than running the whole `build` job once per engine, is deliberate: validate, the coverage gate, the two verify gates and Storybook are all engine-independent. An unrecognized value is an error, not a fallback: a silent fallback would report a green Chromium run under a job labelled `webkit`. A missing browser binary fails naming the engine and its `npx playwright install --with-deps <engine>`.
Unset, the engine is `chromium` — so `npm run ci` and `npm run smoke` are unchanged, and **CI runs Chromium only**. The other engines are a tool you reach for, not a gate: running the sandbox smokes under Firefox before touching the MCP Apps sandbox, the CSP builder, or the proxy page is cheap and worth doing, but nothing runs it for you. An unrecognized value is an error, not a fallback: a silent fallback would report a green Chromium run for a command that asked for `webkit`. A missing browser binary fails naming the engine and its `npx playwright install --with-deps <engine>`.
**CI gates Chromium and Firefox. WebKit is supported by the tooling but is not gated**, because the two App smokes fail under it for reasons nobody has identified. Two things are known: it does **not** reproduce in real Safari (an MCP App opens there normally), and an isolated repro of the mechanism it was first blamed on did not reproduce it under Playwright's WebKit either. So it reads as a property of that particular build rather than a bug users hit, and chasing it further was judged not worth the effort. Run WebKit locally if you want the extra signal; treat a failure there as unexplained rather than as a defect until someone has looked. An engine is either green in CI or absent from it — a `continue-on-error` job would report coverage nobody is held to.
**Firefox passes all three smokes. WebKit fails the two App smokes**, for reasons nobody has identified. Two things are known: it does **not** reproduce in real Safari (an MCP App opens there normally), and an isolated repro of the mechanism it was first blamed on did not reproduce it under Playwright's WebKit either. So it reads as a property of that particular build rather than a bug users hit, and chasing it further was judged not worth the effort — treat a WebKit failure as unexplained rather than as a defect until someone has looked.
**Why none of this is gated in CI.** A Firefox job was trialled and was cheap — about two minutes, running in parallel with the 15-minute `build` job, so no added wall-clock. It was dropped anyway, on the honest count: across a dozen runs it never once disagreed with Chromium, so it was paying real runner time and a real flake surface (`playwright install --with-deps` runs `apt-get update`, which fails whenever a third-party repo in the runner image breaks) to re-confirm a result we already had. Making it available on demand keeps the value — a cross-engine check right when you are changing engine-sensitive code — without a permanent tax on every push. If a cross-engine regression ever does turn up, that is the evidence for putting the job back; it is a one-job diff.
**Why these smokes specifically.** Most of the web client's behavior is React and Mantine, where a second engine buys little. The MCP Apps sandbox is the exception — it is built out of the primitives that genuinely diverge between engines: a CSP `<meta>` injected as the first `<head>` child of a `srcdoc` document, a nested sandboxed iframe, a `Permissions-Policy` `allow` attribute, and `postMessage` origin discipline across those two frames. Nothing else covers that: `sandbox-csp.test.ts` asserts which policy _string_ is built, which passes identically on an engine that ignores `<meta>` CSP entirely, and no Storybook story reaches the sandbox at all (all three App stories point the iframe at a `data:` placeholder and hand the renderer a mock bridge). Storybook itself remains Chromium-only — broadening it covers a much larger and differently-shaped surface, and is a separate decision.