ci: drop the Firefox smoke job; keep the engines as an on-demand tool
The job was trialled and did not earn its place. It was genuinely cheap — ~2 minutes, running in parallel with the 15-minute build job, so zero added wall-clock — but across a dozen runs Firefox never once disagreed with Chromium. It was paying runner time and carrying a real flake surface (playwright install --with-deps runs apt-get update, which fails whenever a third-party repo in the runner image breaks; that is what the one red run was) to re-confirm a result already in hand. Everything that made cross-engine checking possible stays: SMOKE_BROWSER, the shared headless-browser helper, the installer, and the tests. So "SMOKE_BROWSER=firefox npm run smoke:web:engine" is there for the moment it is worth running — before touching the sandbox, the CSP builder or the proxy page — without taxing every push to re-derive the same answer. The workflow is now byte-identical to v2/main again; the publish jobs go back to needing only build. Docs say what is true rather than what was intended, and AGENTS.md records the reasoning so the job is not re-added on the general argument that cross-engine coverage is good — that argument was already accepted, and the on-demand path is what serves it. Re-add it on evidence: a real cross-engine regression a gate would have caught. Also swept the now-stale "matrix" language out of the smokes and the resolver, including the empty-SMOKE_BROWSER rationale, which no longer leans on a matrix key that does not exist. Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
fb9e146043
commit
8a7ade7b08
1 changed files with 64 additions and 140 deletions
|
|
@ -908,9 +908,9 @@ The ⚠️ option-deletion hazard, the snapshot rule, and the recovery recipe ab
|
|||
- `smoke:cli` (`scripts/smoke-cli.mjs`) drives `mcp-inspector --cli` through the built launcher against the bundled stdio test server via a temp `--catalog`: it asserts `tools/list` returns the server's tools (real connect over stdio), the default writable catalog is seeded empty on first run, a missing read-only `--config` errors without seeding, and `--catalog` + `--config` is rejected. `smoke:tui` (`scripts/smoke-tui.mjs`) launches `mcp-inspector --tui --catalog <temp>` and asserts the Ink app renders its first frame (the "MCP Servers" panel) within a timeout, then SIGTERMs it — a shallow boot/render check, not full interaction. **`smoke:tui` is local-only: it self-skips when `process.env.CI` is set**, because the Ink TUI needs a real TTY (raw mode) that headless CI lacks — so run it (via `npm run smoke`) on your own machine before pushing. Both build `test-servers/build` on demand if it's missing.
|
||||
- Storybook play-function tests (`clients/web` `test:storybook`) run in headless Chromium via `@vitest/browser-playwright` (~10s). They are part of `npm run ci` (which installs Playwright chromium first); kept out of `validate` because they need the browser binary and are slower than the unit suite.
|
||||
|
||||
### The web smokes are engine-parameterized; CI gates Chromium and Firefox (#2086)
|
||||
### The web smokes are engine-parameterized; CI runs Chromium only (#2086)
|
||||
|
||||
**`smoke:web:browser`, `smoke:web:app` and `smoke:web:elicit` support Chromium, Firefox and WebKit; `SMOKE_BROWSER` picks one, unset means `chromium`.** CI is green on Chromium and Firefox; **WebKit runs but is not in the CI matrix yet** — see the last bullet. `npm run smoke:web:engine` runs all three smokes in whichever engine is selected — that script, not the workflow YAML, is the list of engine-covered smokes, so adding a fourth covers every engine without touching CI.
|
||||
**`smoke:web:browser`, `smoke:web:app` and `smoke:web:elicit` support Chromium, Firefox and WebKit; `SMOKE_BROWSER` picks one, unset means `chromium`.** **CI runs Chromium only** — the other engines are an on-demand tool, not a gate (see the CI bullet for why). `npm run smoke:web:engine` runs all three smokes in whichever engine is selected — that script, not the workflow YAML, is the list of engine-covered smokes, so adding a fourth covers every engine without touching CI.
|
||||
|
||||
Everything about the selection lives in **`scripts/lib/headless-browser.mjs`** — `resolveBrowserName`, `loadBrowser`, and the `attachPageDiagnostics` / `FATAL_CONSOLE` split the smokes had each hand-rolled. Reach for it rather than launching Playwright in a new script.
|
||||
|
||||
|
|
@ -918,9 +918,9 @@ Everything about the selection lives in **`scripts/lib/headless-browser.mjs`**
|
|||
- **No other tier can substitute, so don't propose one.** `sandbox-csp.test.ts` asserts which policy _string_ is built — environment-independent by construction, and it would pass identically on an engine that ignores `<meta>` CSP entirely. And **no Storybook story reaches the sandbox at all**: all three App stories (`AppRenderer`, `AppsScreen`, `AppElicitationHost`) point the iframe at a `data:` placeholder and hand the renderer a mock bridge, so `sandbox-csp.ts` is imported by exactly two things in the tree — its own test and `createAppBridgeFactory.ts`. Storybook stays Chromium-only; broadening it covers a much larger, differently-shaped surface and is a separate decision to be judged on its own cost.
|
||||
- **An unrecognized `SMOKE_BROWSER` is an error, never a fallback.** Falling back to Chromium would report a green Chromium run under a job labelled `webkit` — coverage claimed but not run, which is worse than none.
|
||||
- **A launch failure names the engine that failed** and its `npx playwright install --with-deps <engine>`. Naming `chromium` while WebKit was the missing one sends the reader to install a browser they already have.
|
||||
- **CI runs Chromium in the `build` job's `npm run smoke` and every other gated engine in a `Sandbox smokes (<engine>)` matrix job** — today that matrix is `[firefox]` alone (see the WebKit bullet below). It sets `fail-fast: false` so that once it holds more than one engine, a failure in one cannot hide a failure in another. Chromium is deliberately _not_ in that matrix — `build`'s `npm run smoke` is exactly the local `npm run ci` path and already covers it, and matrixing `build` itself would triple validate, the coverage gate, two verify gates and Storybook to gain three browser smokes. Both `publish` jobs `needs` the matrix, so a release cannot ship past a non-Chromium failure.
|
||||
- **CI is Chromium-only, deliberately, and a Firefox job was trialled and removed.** The job was cheap — ~2 minutes, parallel with the 15-minute `build` job, so zero added wall-clock — and it still did not earn its place: across a dozen runs Firefox never once disagreed with Chromium, so it spent runner time and carried a real flake surface (`playwright install --with-deps` runs `apt-get update`, which fails whenever a third-party repo in the runner image breaks) to re-confirm a result already in hand. **Don't re-add it on the argument that cross-engine coverage is good in principle** — that argument was already accepted, and it is what the on-demand path serves. Re-add it on evidence: an actual cross-engine regression that a gate would have caught. It is a one-job diff when that happens.
|
||||
- **`pack:verify` stays pinned to Chromium**, passing it explicitly rather than reading `SMOKE_BROWSER`. It is a _packaging_ check; the engine question belongs where the sandbox is under test, and pinning it also means it can't be pointed at an engine its npm script never installed.
|
||||
- **WebKit is deliberately out of the CI matrix, and `continue-on-error` is not the answer.** The two App smokes fail under it for reasons nobody has identified — the failure stage even varies between runs — and the investigation was dropped as not worth the effort once it became clear it does not reproduce in real Safari, and that an isolated repro of the suspected mechanism did not reproduce it under Playwright's WebKit either. Adding `webkit` to the matrix is a one-word diff if someone ever does chase it down; until then an engine is either green in CI or absent from it, because a job allowed to fail reports coverage nobody is held to.
|
||||
- **WebKit fails the two App smokes**, for reasons nobody has identified — the failure stage even varies between runs. The investigation was dropped as not worth the effort once it became clear it does not reproduce in real Safari, and that an isolated repro of the suspected mechanism did not reproduce it under Playwright's WebKit either. Treat a WebKit failure as unexplained rather than as a defect until someone has actually looked.
|
||||
- ⚠️ **A red WebKit run is not a Safari indictment, for the same reason a green one is not a Safari guarantee.** That failure was first written up as a Safari bug on the strength of a Playwright-WebKit run alone; a manual check in Safari did not reproduce it, and neither did an isolated repro of the mechanism it was blamed on. The caveat below cuts both ways, and only one direction of it was applied. **Reproduce in the real browser before claiming user impact in one** — the divergence between Playwright's WebKit build and Safari is largest in exactly the layer (networking) where that bug lives.
|
||||
- ⚠️ **Playwright's WebKit is a WebKit build, not Safari.** Close enough to catch engine-level CSP and iframe divergence; not close enough to certify Safari. Don't write, in a doc or a PR description, that a green run means Safari works.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue