test(smoke): run the headless web smokes in Firefox and WebKit, not just Chromium (#2086)

Every browser-driven check in this repo launched Chromium and nothing else.
That is fine for most of the web client, whose behavior is React and Mantine.
It is not fine for the MCP Apps sandbox, which is built out of the primitives
that genuinely diverge between engines: a CSP <meta> injected as the first
<head> child of a srcdoc document, a nested sandboxed iframe, a
Permissions-Policy allow attribute, and postMessage origin discipline across
those two frames. Nothing else covers that surface — sandbox-csp.test.ts
asserts which policy STRING is built (environment-independent by
construction), and no Storybook story reaches the sandbox at all, since all
three App stories use a data: placeholder iframe and a mock bridge.

The engine is now a parameter:

  SMOKE_BROWSER=webkit npm run smoke:web:app
  SMOKE_BROWSER=firefox npm run smoke:web:engine   # all three smokes

Unset means chromium, so `npm run ci` and `npm run smoke` are unchanged. An
unrecognized value is an error rather than a fallback — falling back would
report a green Chromium run under a job labelled "webkit", claiming coverage
that never ran — and a launch failure names the engine that failed and its own
`playwright install` invocation.

Selection, launching, and the fatal-vs-benign page-diagnostics split now live
in one shared scripts/lib/headless-browser.mjs. The three smokes had each
hand-rolled the diagnostics split and two had their own copy of loadChromium;
pack:verify goes through the same helper, pinned to Chromium explicitly since
it is a packaging check.

The npm scripts drop their `cd clients/web && npx playwright install chromium`
prefix for scripts/install-smoke-browser.mjs. The engine is a variable now, and
a ${VAR:-default} expansion does not expand under Windows' cmd.exe; npx is also
a .cmd shim there that a shell-free spawn cannot start, so this resolves the
Playwright CLI through resolveNodeBin — the #1939 problem solved the #1939 way.

CI gains a `Sandbox smokes (<engine>)` matrix job rather than matrixing `build`,
which also runs validate, the coverage gate, two verify gates and Storybook —
all engine-independent. Both publish jobs now need it.

WebKit is supported by the tooling but deliberately NOT in the CI matrix yet:
pointing the smokes at it immediately found #2132, a pre-existing Safari
incompatibility in the web client's SSE transport that strands the last message
on /api/mcp/events, so an MCP App never leaves "loading". That is a product bug
affecting real Safari users, with its own diagnosis and a verified fix, and it
is not a matrix change. Adding `webkit` here is a one-word diff once it lands.
An engine is either green in CI or absent from it; a continue-on-error job would
report coverage nobody is held to.

Signed-off-by: cliffhall <cliff@futurescale.com>

scripts/smoke-web-elicitation.mjs

@@ -22,9 +22,14 @@
  * therefore needs `frameLocator(...).frameLocator(...)`, not one hop.
  *
  * Set `SMOKE_SCREENSHOT_DIR` to capture PNGs of each state (used to attach
- * proof to a PR); unset, it asserts only. Playwright is resolved with a
- * `createRequire` based at clients/web/package.json for the reason documented at
- * length in smoke-web-browser.mjs.
+ * proof to a PR); unset, it asserts only.
+ *
+ * `SMOKE_BROWSER` picks the engine (`chromium` — the default — `firefox`, or
+ * `webkit`); CI runs all three (#2086). Along with `smoke:web:app` this is one
+ * of the two places the MCP Apps sandbox is actually loaded, and the two nested
+ * frames below are precisely the surface that diverges between engines. See
+ * `lib/headless-browser.mjs`, including why a green WebKit run is not a Safari
+ * guarantee.
  *
  * Expects `clients/web/dist` and `clients/launcher/build` to be built first.
  * `test-servers/build` is rebuilt on every run, as in smoke:web:app — see
@@ -33,20 +38,35 @@
 
 import { spawn } from "node:child_process";
 import { mkdirSync } from "node:fs";
-import { createRequire } from "node:module";
 import { setTimeout as delay } from "node:timers/promises";
 import { join, resolve } from "node:path";
 import { startProdWebServer } from "./lib/prod-web-server.mjs";
 import { stopChild } from "./lib/child-cleanup.mjs";
+import {
+  attachPageDiagnostics,
+  loadBrowser,
+  resolveBrowserName,
+} from "./lib/headless-browser.mjs";
 import {
   ensureTestServers,
   testServerEntryPath,
 } from "./lib/ensure-test-servers.mjs";
 
 const repoRoot = resolve(import.meta.dirname, "..");
-const requireFromWeb = createRequire(
-  resolve(repoRoot, "clients/web/package.json"),
-);
+
+// Resolved before anything is started, so an unsupported SMOKE_BROWSER fails
+// immediately rather than after a web server and two MCP servers are up. Every
+// message carries the engine, so a matrix failure names which one broke.
+let BROWSER;
+try {
+  BROWSER = resolveBrowserName();
+} catch (err) {
+  console.error(
+    `smoke:web:elicit FAILED — ${err instanceof Error ? err.message : String(err)}`,
+  );
+  process.exit(1);
+}
+const LABEL = `smoke:web:elicit [${BROWSER}]`;
 
 const composableServer = testServerEntryPath(repoRoot, "composable");
 const configPath = (name) =>
@@ -59,9 +79,6 @@ const PORT = process.env.SMOKE_WEB_ELICIT_PORT ?? "6296";
 const TOKEN = "smoke-web-elicit-token";
 const TOOL = "app_choose_option";
 const SHOT_DIR = process.env.SMOKE_SCREENSHOT_DIR;
-// The async half of the uncaught-crash class. Kept identical to
-// smoke-web-browser.mjs / smoke-web-app.mjs.
-const FATAL_CONSOLE = /^Uncaught\b|Failed to fetch dynamically imported module/;
 
 const servers = [];
 let browser = null;
@@ -69,7 +86,7 @@ const web = startProdWebServer({
   host: HOST,
   port: PORT,
   token: TOKEN,
-  label: "smoke:web:elicit",
+  label: LABEL,
 });
 
 async function shutdown() {
@@ -84,14 +101,14 @@ async function shutdown() {
   await web.stop();
   while (servers.length) {
     await stopChild(servers.pop(), {
-      label: "smoke:web:elicit",
+      label: LABEL,
       what: "MCP test server",
     });
   }
 }
 
 async function fail(message) {
-  console.error(`smoke:web:elicit FAILED — ${message}`);
+  console.error(`${LABEL} FAILED — ${message}`);
   await shutdown();
   process.exit(1);
 }
@@ -134,24 +151,6 @@ async function startMcpServer(configName) {
   throw new Error(`MCP test server did not start within 30s:\n${out}`);
 }
 
-async function loadChromium() {
-  let chromium;
-  try {
-    ({ chromium } = requireFromWeb("playwright"));
-  } catch (err) {
-    throw new Error(
-      `could not resolve the Playwright package from clients/web — run \`npm install\` at the repo root (${err instanceof Error ? err.message : String(err)})`,
-    );
-  }
-  try {
-    return await chromium.launch({ headless: true });
-  } catch (err) {
-    throw new Error(
-      `chromium failed to launch — on a bare Linux box run \`npx playwright install --with-deps chromium\` for the system libraries (${err instanceof Error ? err.message : String(err)})`,
-    );
-  }
-}
-
 async function shot(page, name) {
   if (!SHOT_DIR) return;
   mkdirSync(SHOT_DIR, { recursive: true });
@@ -159,7 +158,7 @@ async function shot(page, name) {
     path: join(SHOT_DIR, `${name}.png`),
     fullPage: false,
   });
-  console.log(`smoke:web:elicit — captured ${name}.png`);
+  console.log(`${LABEL} — captured ${name}.png`);
 }
 
 /** Connect to `mcpUrl` through the deep link and wait for the Tools list. */
@@ -207,31 +206,24 @@ try {
   // Rebuilt on every run — presence is not freshness (#2111).
   ensureTestServers({
     repoRoot,
-    label: "smoke:web:elicit",
+    label: LABEL,
     requires: ["composable"],
   });
   const appUrl = await startMcpServer("app-elicitation-http");
   const nativeUrl = await startMcpServer("app-elicitation-native-http");
   await web.waitForReady();
-  browser = await loadChromium();
+  browser = await loadBrowser(repoRoot, BROWSER);
   const page = await browser.newPage({
     viewport: { width: 1280, height: 900 },
   });
 
-  // Uncaught *synchronous* page errors, and their *async* twin (an unhandled
-  // rejection or a failed dynamic import), which Chromium reports on the
-  // console channel instead. Both are hard failures; every other console error
-  // is only a diagnostic, so benign font/React noise cannot flake CI. Same
-  // split as smoke:web:browser and smoke:web:app, which document it at length.
-  const pageErrors = [];
-  const consoleErrors = [];
-  page.on("pageerror", (err) =>
-    pageErrors.push(err instanceof Error ? err.message : String(err)),
-  );
-  page.on("console", (msg) => {
-    if (msg.type() === "error") consoleErrors.push(msg.text());
-  });
-  const fatalConsole = () => consoleErrors.filter((m) => FATAL_CONSOLE.test(m));
+  // Uncaught *synchronous* page errors, plus their *async* twin (an unhandled
+  // rejection or a failed dynamic import), which arrives on the console channel
+  // instead. Both are hard failures; every other console error is only a
+  // diagnostic, so benign font/React noise cannot flake CI. Shared with
+  // smoke:web:app rather than re-hand-rolled — the split is subtle enough that
+  // two copies would drift, and it is documented at length on the helper.
+  const diagnostics = attachPageDiagnostics(page);
 
   const drive = async () => {
     // ── 1. Negotiated: the server's app answers the elicitation ────────────
@@ -297,27 +289,25 @@ try {
   try {
     await Promise.race([web.whenChildExits(), drive()]);
   } catch (err) {
-    const diagnostics = [
-      ...pageErrors,
-      ...fatalConsole().map((m) => `console: ${m}`),
+    const notes = [
+      ...diagnostics.pageErrors,
+      ...diagnostics.fatalConsole().map((m) => `console: ${m}`),
     ];
     await fail(
       `${err instanceof Error ? err.message : String(err)}${
-        diagnostics.length
-          ? ` — page diagnostics: ${diagnostics.join("; ")}`
-          : ""
+        notes.length ? ` — page diagnostics: ${notes.join("; ")}` : ""
       }`,
     );
   }
 
   // A drive that reached all of its assertions still fails if the page threw on
   // the way: without this the smoke prints OK over a broken bundle.
-  const fatal = [...pageErrors, ...fatalConsole()];
+  const fatal = diagnostics.fatal();
   if (fatal.length > 0) {
     await fail(`page logged uncaught error(s): ${fatal.join("; ")}`);
   }
 
-  console.log("smoke:web:elicit OK — app-rendered and native paths both drive");
+  console.log(`${LABEL} OK — app-rendered and native paths both drive`);
   await shutdown();
 } catch (err) {
   await fail(err instanceof Error ? err.message : String(err));