test(smoke): run the headless web smokes in Firefox and WebKit, not just Chromium (#2086)

Every browser-driven check in this repo launched Chromium and nothing else.
That is fine for most of the web client, whose behavior is React and Mantine.
It is not fine for the MCP Apps sandbox, which is built out of the primitives
that genuinely diverge between engines: a CSP <meta> injected as the first
<head> child of a srcdoc document, a nested sandboxed iframe, a
Permissions-Policy allow attribute, and postMessage origin discipline across
those two frames. Nothing else covers that surface — sandbox-csp.test.ts
asserts which policy STRING is built (environment-independent by
construction), and no Storybook story reaches the sandbox at all, since all
three App stories use a data: placeholder iframe and a mock bridge.

The engine is now a parameter:

  SMOKE_BROWSER=webkit npm run smoke:web:app
  SMOKE_BROWSER=firefox npm run smoke:web:engine   # all three smokes

Unset means chromium, so `npm run ci` and `npm run smoke` are unchanged. An
unrecognized value is an error rather than a fallback — falling back would
report a green Chromium run under a job labelled "webkit", claiming coverage
that never ran — and a launch failure names the engine that failed and its own
`playwright install` invocation.

Selection, launching, and the fatal-vs-benign page-diagnostics split now live
in one shared scripts/lib/headless-browser.mjs. The three smokes had each
hand-rolled the diagnostics split and two had their own copy of loadChromium;
pack:verify goes through the same helper, pinned to Chromium explicitly since
it is a packaging check.

The npm scripts drop their `cd clients/web && npx playwright install chromium`
prefix for scripts/install-smoke-browser.mjs. The engine is a variable now, and
a ${VAR:-default} expansion does not expand under Windows' cmd.exe; npx is also
a .cmd shim there that a shell-free spawn cannot start, so this resolves the
Playwright CLI through resolveNodeBin — the #1939 problem solved the #1939 way.

CI gains a `Sandbox smokes (<engine>)` matrix job rather than matrixing `build`,
which also runs validate, the coverage gate, two verify gates and Storybook —
all engine-independent. Both publish jobs now need it.

WebKit is supported by the tooling but deliberately NOT in the CI matrix yet:
pointing the smokes at it immediately found #2132, a pre-existing Safari
incompatibility in the web client's SSE transport that strands the last message
on /api/mcp/events, so an MCP App never leaves "loading". That is a product bug
affecting real Safari users, with its own diagnosis and a verified fix, and it
is not a matrix change. Adding `webkit` here is a one-word diff once it lands.
An engine is either green in CI or absent from it; a continue-on-error job would
report coverage nobody is held to.

Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-08-25 20:43:55 -04:00
commit ef4259a544

View file

@ -22,14 +22,12 @@
* **test server** is always a repo fixture — it is not in the tarball and
* should not be.
*
* Playwright is resolved with a `createRequire` based at
* clients/web/package.json rather than a bare `import("playwright")`: a bare ESM
* specifier resolves relative to `scripts/`, not the cwd, so a `cd clients/web`
* in the npm script would NOT make it resolvable. Same gotcha as
* smoke-web-browser.mjs; see its header.
* Launching the browser is NOT this module's job — `lib/headless-browser.mjs`
* owns that, so the engine is a parameter rather than a hard-coded Chromium
* (#2086) and so `smoke:web:browser`, which does not drive an App at all, can
* share it without importing this file.
*/
import { createRequire } from "node:module";
import { join } from "node:path";
import { startAnnouncedChild } from "./announced-child.mjs";
import {
@ -40,16 +38,6 @@ import {
/** The App tool the `mcp-app-http.json` fixture serves. */
export const APP_TOOL = "mcp_app_demo";
/**
* Console messages that are the async half of the uncaught-crash class (an
* unhandled rejection or a failed dynamic import). Hard failures; every other
* console error is a diagnostic, so benign font-CDN / React-warning noise can't
* flake CI. Kept identical to smoke-web-browser.mjs, which documents the
* reasoning at length.
*/
export const FATAL_CONSOLE =
/^Uncaught\b|Failed to fetch dynamically imported module/;
/** Path to the composable test server build, relative to the repo root. */
export function composableServerPath(repoRoot) {
return testServerEntryPath(repoRoot, "composable");
@ -143,59 +131,6 @@ export function buildAppDeepLink({
);
}
/**
* Attach the two error channels a headless page reports on.
*
* An uncaught *synchronous* page error fires `pageerror`; its async twin — an
* unhandled rejection or a failed dynamic import — is not a `pageerror` at all,
* Chromium reports it on the console channel instead. Both are captured; only
* `fatal()` is a failure, so ordinary console noise stays a diagnostic.
*/
export function attachPageDiagnostics(page) {
const pageErrors = [];
const consoleErrors = [];
page.on("pageerror", (err) =>
pageErrors.push(err instanceof Error ? err.message : String(err)),
);
page.on("console", (msg) => {
if (msg.type() === "error") consoleErrors.push(msg.text());
});
return {
pageErrors,
consoleErrors,
fatalConsole: () => consoleErrors.filter((m) => FATAL_CONSOLE.test(m)),
benignConsole: () => consoleErrors.filter((m) => !FATAL_CONSOLE.test(m)),
fatal: () => [
...pageErrors,
...consoleErrors.filter((m) => FATAL_CONSOLE.test(m)),
],
};
}
/** Launch headless Chromium, resolved from the web client's install. */
export async function loadChromium(repoRoot) {
const requireFromWeb = createRequire(
join(repoRoot, "clients", "web", "package.json"),
);
let chromium;
try {
({ chromium } = requireFromWeb("playwright"));
} catch (err) {
// Not resolvable means devDependencies are missing — fixed by `npm install`
// at the repo root, NOT by `playwright install` (which fetches binaries).
throw new Error(
`could not resolve the Playwright package from clients/web — run \`npm install\` at the repo root (${err instanceof Error ? err.message : String(err)})`,
);
}
try {
return await chromium.launch({ headless: true });
} catch (err) {
throw new Error(
`chromium failed to launch — on a bare Linux box run \`npx playwright install --with-deps chromium\` for the system libraries (${err instanceof Error ? err.message : String(err)})`,
);
}
}
/**
* Drive **connect → open app → widget ready** on an already-open page.
*