modelcontextprotocol/inspector
Watch
1
0
Fork
You've already forked inspector
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

test(smoke): run the headless web smokes in Firefox and WebKit, not just Chromium (#2086)

Browse source 
Every browser-driven check in this repo launched Chromium and nothing else.
That is fine for most of the web client, whose behavior is React and Mantine.
It is not fine for the MCP Apps sandbox, which is built out of the primitives
that genuinely diverge between engines: a CSP <meta> injected as the first
<head> child of a srcdoc document, a nested sandboxed iframe, a
Permissions-Policy allow attribute, and postMessage origin discipline across
those two frames. Nothing else covers that surface — sandbox-csp.test.ts
asserts which policy STRING is built (environment-independent by
construction), and no Storybook story reaches the sandbox at all, since all
three App stories use a data: placeholder iframe and a mock bridge.

The engine is now a parameter:

  SMOKE_BROWSER=webkit npm run smoke:web:app
  SMOKE_BROWSER=firefox npm run smoke:web:engine   # all three smokes

Unset means chromium, so `npm run ci` and `npm run smoke` are unchanged. An
unrecognized value is an error rather than a fallback — falling back would
report a green Chromium run under a job labelled "webkit", claiming coverage
that never ran — and a launch failure names the engine that failed and its own
`playwright install` invocation.

Selection, launching, and the fatal-vs-benign page-diagnostics split now live
in one shared scripts/lib/headless-browser.mjs. The three smokes had each
hand-rolled the diagnostics split and two had their own copy of loadChromium;
pack:verify goes through the same helper, pinned to Chromium explicitly since
it is a packaging check.

The npm scripts drop their `cd clients/web && npx playwright install chromium`
prefix for scripts/install-smoke-browser.mjs. The engine is a variable now, and
a ${VAR:-default} expansion does not expand under Windows' cmd.exe; npx is also
a .cmd shim there that a shell-free spawn cannot start, so this resolves the
Playwright CLI through resolveNodeBin — the #1939 problem solved the #1939 way.

CI gains a `Sandbox smokes (<engine>)` matrix job rather than matrixing `build`,
which also runs validate, the coverage gate, two verify gates and Storybook —
all engine-independent. Both publish jobs now need it.

WebKit is supported by the tooling but deliberately NOT in the CI matrix yet:
pointing the smokes at it immediately found #2132, a pre-existing Safari
incompatibility in the web client's SSE transport that strands the last message
on /api/mcp/events, so an MCP App never leaves "loading". That is a product bug
affecting real Safari users, with its own diagnosis and a verified fix, and it
is not a matrix change. Adding `webkit` here is a one-word diff once it lands.
An engine is either green in CI or absent from it; a continue-on-error job would
report coverage nobody is held to.

Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-08-25 20:43:55 -04:00
commit ef4259a544
8 changed files with 652 additions and 222 deletions
Download patch file Download diff file Expand all files Collapse all files

84
scripts/install-smoke-browser.mjs Normal file
View file

@ -0,0 +1,84 @@
#!/usr/bin/env node
/**
* Fetch the Playwright browser binary a headless web smoke is about to launch
* (#2086).
*
* This replaces the `cd clients/web && npx playwright install chromium` prefix
* each smoke's npm script used to carry. Two reasons it is a script rather than
* a longer npm-script line:
*
* - The engine is now a variable. `npx playwright install ${SMOKE_BROWSER:-…}`
* is a POSIX-shell expansion that does not expand under Windows' `cmd.exe`,
* which npm uses there — it would silently try to install a browser named
* `${SMOKE_BROWSER:-chromium}`.
* - `npx` is a `.cmd` shim on Windows, which a shell-free spawn cannot start.
* `resolveNodeBin` walks to the JS entry behind the package's `bin` and runs
* it with `process.execPath` instead — the same fix #1939 made for the
* verify scripts, and it keeps the resolution pinned to the repo's own
* Playwright rather than whatever `npx` might fetch.
*
* Usage: `node scripts/install-smoke-browser.mjs [engine]`. With no argument the
* engine comes from `SMOKE_BROWSER` (default `chromium`); pass one explicitly
* where the consumer's engine is fixed regardless of that variable, as
* `pack:verify` does.
*/
import { spawnSync } from "node:child_process";
import { join, resolve } from "node:path";
import {
SUPPORTED_BROWSERS,
resolveBrowserName,
} from "./lib/headless-browser.mjs";
import { resolveNodeBin } from "./lib/resolve-node-bin.mjs";
const repoRoot = resolve(import.meta.dirname, "..");
const webDir = join(repoRoot, "clients", "web");
const requested = process.argv[2];
let browserName;
try {
if (requested === undefined) {
browserName = resolveBrowserName();
} else if (SUPPORTED_BROWSERS.includes(requested)) {
browserName = requested;
} else {
throw new Error(
`unsupported browser "${requested}" — expected one of ${SUPPORTED_BROWSERS.join(", ")}`,
);
}
} catch (err) {
console.error(
`install-smoke-browser: ${err instanceof Error ? err.message : String(err)}`,
);
process.exit(1);
}
let cli;
try {
cli = resolveNodeBin("playwright", "playwright", webDir);
} catch (err) {
console.error(
"install-smoke-browser: could not resolve the Playwright CLI from clients/web — " +
`run \`npm install\` at the repo root (${err instanceof Error ? err.message : String(err)})`,
);
process.exit(1);
}
// `install` is a no-op when the binary is already present, so this is cheap on
// a warm machine and on a CI runner whose Playwright cache was restored.
const result = spawnSync(process.execPath, [cli, "install", browserName], {
cwd: webDir,
stdio: "inherit",
});
if (result.error) {
console.error(
`install-smoke-browser: could not run the Playwright CLI: ${result.error.message}`,
);
process.exit(1);
}
if (result.status !== 0) {
console.error(
`install-smoke-browser: \`playwright install ${browserName}\` exited ${result.status}`,
);
process.exit(result.status ?? 1);
}

146
scripts/lib/headless-browser.mjs Normal file
View file

@ -0,0 +1,146 @@
/**
* The headless browser the web smokes drive: which engine, how it is launched,
* and the two error channels a page reports on (#2086).
*
* All three smokes (`smoke:web:browser`, `smoke:web:app`, `smoke:web:elicit`)
* and `pack:verify` go through here, so the engine is a parameter in exactly
* one place and the diagnostics split is stated once rather than hand-rolled
* four times.
*
* Every browser-driven check in this repo used to be Chromium-only. That is
* fine for most of the web client, whose behavior is React and Mantine — and it
* is NOT fine for the MCP Apps sandbox, which is built out of exactly the
* primitives that diverge between engines: a CSP `<meta>` injected into a
* `srcdoc` document, a nested sandboxed iframe, a `Permissions-Policy` `allow`
* attribute, and `postMessage` origin discipline across those two frames. A
* regression in any of those is invisible unless it also reproduces in Chromium.
*
* The unit tests cannot cover it either: `sandbox-csp.test.ts` asserts which
* policy STRING is built, which would pass identically on an engine that
* ignores `<meta>` CSP entirely. And no Storybook story reaches the sandbox at
* all — all three App stories point the iframe at a `data:` placeholder and hand
* the renderer a mock bridge. So the smokes are the only place the sandbox is
* genuinely exercised, and this module is what lets them be pointed at another
* engine.
*
* ⚠️ Playwright's WebKit is a WebKit build, not Safari. It is close enough to
* catch engine-level CSP and iframe divergence, and not close enough to certify
* Safari specifically — a green run here is not a Safari guarantee.
*
* Playwright is resolved with a `createRequire` based at
* clients/web/package.json rather than a bare `import("playwright")`: a bare ESM
* specifier resolves relative to `scripts/`, not the cwd, so a `cd clients/web`
* in the npm script would NOT make it resolvable. See smoke-web-browser.mjs's
* header for the full gotcha.
*/
import { createRequire } from "node:module";
import { join } from "node:path";
/**
* Console messages that are the async half of the uncaught-crash class (an
* unhandled rejection or a failed dynamic import). Hard failures; every other
* console error is a diagnostic, so benign font-CDN / React-warning noise can't
* flake CI. smoke-web-browser.mjs's header documents the reasoning at length.
*/
export const FATAL_CONSOLE =
/^Uncaught\b|Failed to fetch dynamically imported module/;
/**
* Attach the two error channels a headless page reports on.
*
* An uncaught *synchronous* page error fires `pageerror`; its async twin — an
* unhandled rejection or a failed dynamic import — is not a `pageerror` at all
* and arrives on the console channel instead. Both are captured, and both are
* read on every engine rather than branching per browser: which channel an
* engine picks is exactly the kind of detail that differs between them, and
* reading both makes that difference irrelevant. Only `fatal()` is a failure, so
* ordinary console noise stays a diagnostic.
*/
export function attachPageDiagnostics(page) {
const pageErrors = [];
const consoleErrors = [];
page.on("pageerror", (err) =>
pageErrors.push(err instanceof Error ? err.message : String(err)),
);
page.on("console", (msg) => {
if (msg.type() === "error") consoleErrors.push(msg.text());
});
return {
pageErrors,
consoleErrors,
fatalConsole: () => consoleErrors.filter((m) => FATAL_CONSOLE.test(m)),
benignConsole: () => consoleErrors.filter((m) => !FATAL_CONSOLE.test(m)),
fatal: () => [
...pageErrors,
...consoleErrors.filter((m) => FATAL_CONSOLE.test(m)),
],
};
}
/** The supported engine set. Anything else is a typo, not a request. */
export const SUPPORTED_BROWSERS = ["chromium", "firefox", "webkit"];
/** The engine used when nothing asks for another one. */
export const DEFAULT_BROWSER = "chromium";
/** The env var each smoke reads to pick its engine. */
export const BROWSER_ENV_VAR = "SMOKE_BROWSER";
/**
* Resolve the engine name from the environment, deny-by-default.
*
* An unrecognized value FAILS rather than falling back to Chromium: a silent
* fallback would report a green Chromium run under a job labelled "webkit",
* which is worse than no coverage — it claims coverage that never ran.
*/
export function resolveBrowserName(env = process.env) {
const raw = env[BROWSER_ENV_VAR];
if (raw === undefined || raw.trim() === "") return DEFAULT_BROWSER;
const name = raw.trim().toLowerCase();
if (!SUPPORTED_BROWSERS.includes(name)) {
throw new Error(
`${BROWSER_ENV_VAR}="${raw}" is not a supported browser — expected one of ${SUPPORTED_BROWSERS.join(", ")}`,
);
}
return name;
}
/**
* Launch a headless browser of `browserName`, resolved from the web client's
* install.
*
* The launch-failure message names the engine that failed and the
* `playwright install` invocation that fixes it — naming "chromium" while
* WebKit was the thing missing sends the reader off to install a browser they
* already have.
*/
export async function loadBrowser(repoRoot, browserName = DEFAULT_BROWSER) {
if (!SUPPORTED_BROWSERS.includes(browserName)) {
throw new Error(
`unsupported browser "${browserName}" — expected one of ${SUPPORTED_BROWSERS.join(", ")}`,
);
}
const requireFromWeb = createRequire(
join(repoRoot, "clients", "web", "package.json"),
);
let playwright;
try {
playwright = requireFromWeb("playwright");
} catch (err) {
// Not resolvable means devDependencies are missing — fixed by `npm install`
// at the repo root, NOT by `playwright install` (which fetches binaries).
throw new Error(
`could not resolve the Playwright package from clients/web — run \`npm install\` at the repo root (${err instanceof Error ? err.message : String(err)})`,
);
}
try {
return await playwright[browserName].launch({ headless: true });
} catch (err) {
throw new Error(
`${browserName} failed to launch — run \`npx playwright install --with-deps ${browserName}\`, ` +
`which fetches the browser and (on a bare Linux box) its system libraries ` +
`(${err instanceof Error ? err.message : String(err)})`,
);
}
}

148
scripts/lib/headless-browser.test.mjs Normal file
View file

@ -0,0 +1,148 @@
/**
* Unit tests for the headless-browser engine selection (#2086).
*
* The smokes cannot check this themselves: each one resolves exactly one engine
* per process and then spends its whole run inside the happy path, so the branch
* that matters most — an unrecognized `SMOKE_BROWSER` — is dead code from their
* point of view. It is also the branch whose failure is silent rather than loud:
* a fallback to Chromium there would report a green run under a job labelled
* "webkit", claiming coverage that never ran.
*
* `loadBrowser`'s own launch path is deliberately not covered here — it takes a
* real browser binary, which is what the smokes are for. Only its argument
* validation, which fails before any of that, is.
*/
import assert from "node:assert/strict";
import { describe, it } from "node:test";
import {
BROWSER_ENV_VAR,
DEFAULT_BROWSER,
SUPPORTED_BROWSERS,
attachPageDiagnostics,
loadBrowser,
resolveBrowserName,
} from "./headless-browser.mjs";
describe("resolveBrowserName", () => {
it("defaults to chromium when the variable is unset", () => {
assert.equal(resolveBrowserName({}), DEFAULT_BROWSER);
});
it("treats an empty or whitespace-only value as unset", () => {
// A CI expression that resolves to nothing (`SMOKE_BROWSER: ${{ … }}` with
// an undefined matrix key) sets the variable to "" rather than removing it.
for (const raw of ["", " "]) {
assert.equal(
resolveBrowserName({ [BROWSER_ENV_VAR]: raw }),
DEFAULT_BROWSER,
);
}
});
it("accepts every supported engine", () => {
for (const name of SUPPORTED_BROWSERS) {
assert.equal(resolveBrowserName({ [BROWSER_ENV_VAR]: name }), name);
}
});
it("normalizes surrounding whitespace and case", () => {
assert.equal(
resolveBrowserName({ [BROWSER_ENV_VAR]: " WebKit " }),
"webkit",
);
});
it("rejects an unrecognized engine rather than falling back", () => {
assert.throws(
() => resolveBrowserName({ [BROWSER_ENV_VAR]: "safari" }),
// The message must quote what was asked for AND list what is accepted —
// "safari" is the most likely typo here and its fix is not guessable.
(err) =>
/safari/.test(err.message) &&
SUPPORTED_BROWSERS.every((name) => err.message.includes(name)),
);
});
});
describe("loadBrowser", () => {
it("rejects an unsupported engine before touching Playwright", async () => {
await assert.rejects(
// A repo root that does not exist: reaching the `createRequire` would
// throw a different (resolution) error, so this also pins the ORDER —
// validation first, so the message names the real mistake.
() => loadBrowser("/nonexistent-repo-root", "safari"),
/unsupported browser "safari"/,
);
});
});
describe("attachPageDiagnostics", () => {
/** Minimal Playwright `page` stand-in: records handlers, replays events. */
function fakePage() {
const handlers = {};
return {
on: (event, fn) => {
(handlers[event] ??= []).push(fn);
},
emit: (event, arg) => handlers[event]?.forEach((fn) => fn(arg)),
};
}
const consoleMessage = (type, text) => ({
type: () => type,
text: () => text,
});
it("splits fatal console errors from benign noise", () => {
const page = fakePage();
const diagnostics = attachPageDiagnostics(page);
page.emit("console", consoleMessage("error", "Uncaught (in promise) boom"));
page.emit(
"console",
consoleMessage("error", "Failed to fetch dynamically imported module x"),
);
// The two shapes that used to flake CI, and must stay diagnostics.
page.emit(
"console",
consoleMessage("error", "Failed to load resource: net::ERR_FAILED"),
);
page.emit(
"console",
consoleMessage("error", "Warning: each child needs a key"),
);
// Non-error console output is ignored entirely.
page.emit("console", consoleMessage("log", "Uncaught looking but a log"));
assert.deepEqual(diagnostics.fatalConsole(), [
"Uncaught (in promise) boom",
"Failed to fetch dynamically imported module x",
]);
assert.deepEqual(diagnostics.benignConsole(), [
"Failed to load resource: net::ERR_FAILED",
"Warning: each child needs a key",
]);
});
it("counts every pageerror as fatal, whatever it was thrown as", () => {
const page = fakePage();
const diagnostics = attachPageDiagnostics(page);
page.emit("pageerror", new Error("sync boom"));
page.emit("pageerror", "thrown as a string");
page.emit("console", consoleMessage("error", "benign"));
assert.deepEqual(diagnostics.pageErrors, [
"sync boom",
"thrown as a string",
]);
assert.deepEqual(diagnostics.fatal(), ["sync boom", "thrown as a string"]);
});
it("reports nothing on a clean page", () => {
const diagnostics = attachPageDiagnostics(fakePage());
assert.deepEqual(diagnostics.fatal(), []);
assert.deepEqual(diagnostics.benignConsole(), []);
});
});

73
scripts/lib/mcp-app-flow.mjs
View file

@ -22,14 +22,12 @@
* **test server** is always a repo fixture — it is not in the tarball and
* should not be.
*
* Playwright is resolved with a `createRequire` based at
* clients/web/package.json rather than a bare `import("playwright")`: a bare ESM
* specifier resolves relative to `scripts/`, not the cwd, so a `cd clients/web`
* in the npm script would NOT make it resolvable. Same gotcha as
* smoke-web-browser.mjs; see its header.
* Launching the browser is NOT this module's job — `lib/headless-browser.mjs`
* owns that, so the engine is a parameter rather than a hard-coded Chromium
* (#2086) and so `smoke:web:browser`, which does not drive an App at all, can
* share it without importing this file.
*/
import { createRequire } from "node:module";
import { join } from "node:path";
import { startAnnouncedChild } from "./announced-child.mjs";
import {
@ -40,16 +38,6 @@ import {
/** The App tool the `mcp-app-http.json` fixture serves. */
export const APP_TOOL = "mcp_app_demo";
/**
* Console messages that are the async half of the uncaught-crash class (an
* unhandled rejection or a failed dynamic import). Hard failures; every other
* console error is a diagnostic, so benign font-CDN / React-warning noise can't
* flake CI. Kept identical to smoke-web-browser.mjs, which documents the
* reasoning at length.
*/
export const FATAL_CONSOLE =
/^Uncaught\b|Failed to fetch dynamically imported module/;
/** Path to the composable test server build, relative to the repo root. */
export function composableServerPath(repoRoot) {
return testServerEntryPath(repoRoot, "composable");
@ -143,59 +131,6 @@ export function buildAppDeepLink({
);
}
/**
* Attach the two error channels a headless page reports on.
*
* An uncaught *synchronous* page error fires `pageerror`; its async twin — an
* unhandled rejection or a failed dynamic import — is not a `pageerror` at all,
* Chromium reports it on the console channel instead. Both are captured; only
* `fatal()` is a failure, so ordinary console noise stays a diagnostic.
*/
export function attachPageDiagnostics(page) {
const pageErrors = [];
const consoleErrors = [];
page.on("pageerror", (err) =>
pageErrors.push(err instanceof Error ? err.message : String(err)),
);
page.on("console", (msg) => {
if (msg.type() === "error") consoleErrors.push(msg.text());
});
return {
pageErrors,
consoleErrors,
fatalConsole: () => consoleErrors.filter((m) => FATAL_CONSOLE.test(m)),
benignConsole: () => consoleErrors.filter((m) => !FATAL_CONSOLE.test(m)),
fatal: () => [
...pageErrors,
...consoleErrors.filter((m) => FATAL_CONSOLE.test(m)),
],
};
}
/** Launch headless Chromium, resolved from the web client's install. */
export async function loadChromium(repoRoot) {
const requireFromWeb = createRequire(
join(repoRoot, "clients", "web", "package.json"),
);
let chromium;
try {
({ chromium } = requireFromWeb("playwright"));
} catch (err) {
// Not resolvable means devDependencies are missing — fixed by `npm install`
// at the repo root, NOT by `playwright install` (which fetches binaries).
throw new Error(
`could not resolve the Playwright package from clients/web — run \`npm install\` at the repo root (${err instanceof Error ? err.message : String(err)})`,
);
}
try {
return await chromium.launch({ headless: true });
} catch (err) {
throw new Error(
`chromium failed to launch — on a bare Linux box run \`npx playwright install --with-deps chromium\` for the system libraries (${err instanceof Error ? err.message : String(err)})`,
);
}
}
/**
* Drive **connect → open app → widget ready** on an already-open page.
*

9
scripts/pack-and-verify.mjs
View file

@ -60,12 +60,11 @@ import {
ensureTestServers,
testServerEntryPath,
} from "./lib/ensure-test-servers.mjs";
import { attachPageDiagnostics, loadBrowser } from "./lib/headless-browser.mjs";
import {
APP_TOOL,
attachPageDiagnostics,
buildAppDeepLink,
driveAppFlow,
loadChromium,
startMcpAppServer,
} from "./lib/mcp-app-flow.mjs";
import { winShellArgs } from "./lib/win-shell-args.mjs";
@ -567,7 +566,11 @@ async function verifyAppRender(baseUrl, token, whenWebServerExits) {
},
label: LABEL,
});
browser = await loadChromium(repoRoot);
// Chromium explicitly, not `resolveBrowserName()`: this is a *packaging*
// check, and the engine matrix (#2086) belongs to the smokes, where the
// sandbox surface is what is under test. Pinning it also means `pack:verify`
// cannot be pointed at an engine its npm script never installed.
browser = await loadBrowser(repoRoot, "chromium");
const page = await browser.newPage();
const diagnostics = attachPageDiagnostics(page);

35
scripts/smoke-web-app.mjs
View file

@ -41,6 +41,16 @@
* renamed* without its reader being updated, a repo-tree failure pack:verify
* would only find later.
*
* ── Which engine ────────────────────────────────────────────────────────────
*
* `SMOKE_BROWSER` picks the engine (`chromium` — the default — `firefox`, or
* `webkit`); CI runs all three (#2086). This smoke is one of the two places the
* MCP Apps sandbox is genuinely exercised, and the sandbox is built out of the
* primitives that actually diverge between engines — `srcdoc` CSP inheritance,
* nested sandboxed iframes, `Permissions-Policy`, cross-frame `postMessage`. See
* `lib/headless-browser.mjs`, including why a green WebKit run is not a Safari
* guarantee.
*
* Expects `clients/web/dist` and `clients/launcher/build` to be built first —
* the validate / CI ordering guarantees this. `test-servers/build` is rebuilt on
* every run, as in smoke:cli — see `scripts/lib/ensure-test-servers.mjs` for why
@ -54,17 +64,34 @@ import { join, resolve } from "node:path";
import { startProdWebServer } from "./lib/prod-web-server.mjs";
import { stopChild } from "./lib/child-cleanup.mjs";
import {
APP_TOOL,
attachPageDiagnostics,
loadBrowser,
resolveBrowserName,
} from "./lib/headless-browser.mjs";
import {
APP_TOOL,
buildAppDeepLink,
driveAppFlow,
loadChromium,
sandboxProxyPageFor,
startMcpAppServer,
} from "./lib/mcp-app-flow.mjs";
const repoRoot = resolve(import.meta.dirname, "..");
const LABEL = "smoke:web:app";
// Resolved before anything is started, so an unsupported SMOKE_BROWSER fails
// immediately rather than after a web server and two MCP servers are up. Every
// message this smoke prints carries the engine, so a matrix failure names which
// one broke without the reader having to match it to a job title.
let BROWSER;
try {
BROWSER = resolveBrowserName();
} catch (err) {
console.error(
`smoke:web:app FAILED — ${err instanceof Error ? err.message : String(err)}`,
);
process.exit(1);
}
const LABEL = `smoke:web:app [${BROWSER}]`;
// Resolved exactly as the runtime does, from the built runner's directory.
const sandboxProxyPage = sandboxProxyPageFor(
@ -141,7 +168,7 @@ try {
label: LABEL,
});
await server.waitForReady();
browser = await loadChromium(repoRoot);
browser = await loadBrowser(repoRoot, BROWSER);
const page = await browser.newPage();
const diagnostics = attachPageDiagnostics(page);

127
scripts/smoke-web-browser.mjs
View file

@ -29,12 +29,13 @@
* Two channels carry the failure. A *synchronous* uncaught exception (the
* CASE-1 shape above — a stub call during module init) fires `pageerror`. Its
* *async* twin — the same `TypeError` reached through an `await`/`.then()`, or a
* failed dynamic import (this app lazy-loads chunks) — is NOT a `pageerror`;
* Chromium logs it on the **console** channel as `Uncaught (in promise) …` /
* `Failed to fetch dynamically imported module`. Both are hard failures.
* failed dynamic import (this app lazy-loads chunks) — is NOT a `pageerror`; it
* arrives on the **console** channel as `Uncaught (in promise) …` /
* `Failed to fetch dynamically imported module`. Both are hard failures, and
* both channels are read on every engine rather than branching per browser.
*
* Every *other* `console.error` is NOT a hard failure: the console is where
* Chromium also reports benign things a boot smoke shouldn't fail on — a failed
* Every *other* `console.error` is NOT a hard failure: the console is also where
* benign things a boot smoke shouldn't fail on land — a failed
* subresource load (e.g. the Google-Fonts `<link>` in index.html on a
* network-restricted box) or a React key/prop warning. Those are printed as
* diagnostics. The `Uncaught` / dynamic-import prefixes are unambiguous — a
@ -42,29 +43,44 @@
* never starts with `Uncaught` — so hard-failing on them can't reintroduce that
* flake.
*
* Playwright lives in clients/web's node_modules, so it's resolved with a
* `createRequire` based at clients/web/package.json rather than a bare
* `import("playwright")`. A bare ESM specifier resolves relative to *this
* script's* directory (scripts/), not the cwd — so `cd clients/web` in the npm
* script would NOT make it resolvable (it only appeared to work locally when an
* ancestor node_modules happened to carry playwright; it fails in CI, which has
* none). createRequire pins resolution to clients/web regardless of cwd.
* Launching the browser (and resolving Playwright from clients/web, which has
* its own gotcha — see `lib/headless-browser.mjs`) is delegated to that module,
* which is also where `SMOKE_BROWSER` picks the engine: `chromium` (the
* default), `firefox`, or `webkit`, all three of which CI runs (#2086). The
* engine question here is narrower than in the App smokes — this asserts a clean
* first paint, i.e. that the shipped bundle's syntax and API level are
* *reachable* on the engine at all, rather than anything about the sandbox — but
* it is real, and it is nearly free to include.
*
* Expects `clients/web/dist` and `clients/launcher/build` to be built first —
* the validate / CI ordering guarantees this.
*/
import { createRequire } from "node:module";
import { setTimeout as delay } from "node:timers/promises";
import { fileURLToPath } from "node:url";
import { dirname, resolve } from "node:path";
import { startProdWebServer } from "./lib/prod-web-server.mjs";
import {
attachPageDiagnostics,
loadBrowser,
resolveBrowserName,
} from "./lib/headless-browser.mjs";
const scriptDir = dirname(fileURLToPath(import.meta.url));
// Resolve playwright from clients/web (where it's installed) no matter the cwd.
const requireFromWeb = createRequire(
resolve(scriptDir, "..", "clients/web/package.json"),
);
const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
// Resolved before the web server is started, so an unsupported SMOKE_BROWSER
// fails immediately. Every message carries the engine, so a matrix failure names
// which one broke.
let BROWSER;
try {
BROWSER = resolveBrowserName();
} catch (err) {
console.error(
`smoke:web:browser FAILED — ${err instanceof Error ? err.message : String(err)}`,
);
process.exit(1);
}
const LABEL = `smoke:web:browser [${BROWSER}]`;
const HOST = "127.0.0.1";
// Distinct from smoke:web's SMOKE_WEB_PORT so overriding one doesn't make both
@ -72,17 +88,11 @@ const HOST = "127.0.0.1";
const PORT = process.env.SMOKE_WEB_BROWSER_PORT ?? "6298";
const TOKEN = "smoke-web-browser-token";
// Console messages that are the async half of the uncaught-crash class (an
// unhandled promise rejection or a failed dynamic import). Hard failures, unlike
// benign console noise. See the header comment for why this can't reintroduce
// the font/CDN flake.
const FATAL_CONSOLE = /^Uncaught\b|Failed to fetch dynamically imported module/;
const server = startProdWebServer({
host: HOST,
port: PORT,
token: TOKEN,
label: "smoke:web:browser",
label: LABEL,
});
let browser = null;
@ -99,51 +109,22 @@ async function shutdown() {
}
async function fail(message) {
console.error(`smoke:web:browser FAILED — ${message}`);
console.error(`${LABEL} FAILED — ${message}`);
await shutdown();
process.exit(1);
}
async function loadChromium() {
let chromium;
try {
({ chromium } = requireFromWeb("playwright"));
} catch (err) {
// A failure here means the playwright *npm package* isn't resolvable from
// clients/web — fixed by installing devDependencies, NOT by `playwright
// install` (which fetches browser binaries). `npm install` at the repo root
// cascades into clients/web via postinstall.
throw new Error(
`could not resolve the Playwright package from clients/web — run \`npm install\` at the repo root (${err instanceof Error ? err.message : String(err)})`,
);
}
try {
return await chromium.launch({ headless: true });
} catch (err) {
throw new Error(
`chromium failed to launch — on a bare Linux box run \`npx playwright install --with-deps chromium\` for the system libraries (${err instanceof Error ? err.message : String(err)})`,
);
}
}
try {
await server.waitForReady();
browser = await loadChromium();
browser = await loadBrowser(repoRoot, BROWSER);
const page = await browser.newPage();
// Uncaught (synchronous) page errors are a hard failure — a Node-only module
// reaching the browser bundle surfaces here as a TypeError when its empty stub
// is called during module init.
const pageErrors = [];
// Console errors are diagnostic only EXCEPT those matching FATAL_CONSOLE (the
// async half of the same crash class) — see the header comment.
const consoleErrors = [];
page.on("pageerror", (err) =>
pageErrors.push(err instanceof Error ? err.message : String(err)),
);
page.on("console", (msg) => {
if (msg.type() === "error") consoleErrors.push(msg.text());
});
// is called during module init. Console errors are diagnostic only EXCEPT the
// async half of that same crash class; see the header comment, and the shared
// helper for the split.
const diagnostics = attachPageDiagnostics(page);
const render = async () => {
// Token is injected into index.html by the prod server, so a bare `/` load
@ -175,39 +156,35 @@ try {
try {
await Promise.race([server.whenChildExits(), render()]);
} catch (err) {
const diagnostics = [
...pageErrors,
...consoleErrors.map((m) => `console: ${m}`),
const notes = [
...diagnostics.pageErrors,
...diagnostics.consoleErrors.map((m) => `console: ${m}`),
];
await fail(
`${err instanceof Error ? err.message : String(err)}${
diagnostics.length
? ` — page diagnostics: ${diagnostics.join("; ")}`
: ""
notes.length ? ` — page diagnostics: ${notes.join("; ")}` : ""
}`,
);
}
// Hard failures: any uncaught (sync) page error, plus console errors that are
// the async half of the class (unhandled rejection / failed dynamic import).
const fatalConsole = consoleErrors.filter((m) => FATAL_CONSOLE.test(m));
if (pageErrors.length > 0 || fatalConsole.length > 0) {
await fail(
`app logged uncaught error(s): ${[...pageErrors, ...fatalConsole].join("; ")}`,
);
const fatal = diagnostics.fatal();
if (fatal.length > 0) {
await fail(`app logged uncaught error(s): ${fatal.join("; ")}`);
}
// Non-fatal console errors: surface them so a real problem isn't invisible,
// without failing the smoke on benign subresource/warning noise.
const benignConsole = consoleErrors.filter((m) => !FATAL_CONSOLE.test(m));
if (benignConsole.length > 0) {
const benign = diagnostics.benignConsole();
if (benign.length > 0) {
console.log(
`smoke:web:browser note — ${benignConsole.length} non-fatal console error(s): ${benignConsole.join("; ")}`,
`${LABEL} note — ${benign.length} non-fatal console error(s): ${benign.join("; ")}`,
);
}
console.log(
`smoke:web:browser OK — app booted at ${server.baseUrl}, rendered "Add Servers" with no uncaught errors (sync page error or unhandled rejection)`,
`${LABEL} OK — app booted at ${server.baseUrl}, rendered "Add Servers" with no uncaught errors (sync page error or unhandled rejection)`,
);
await shutdown();
process.exit(0);

102
scripts/smoke-web-elicitation.mjs
View file

@ -22,9 +22,14 @@
* therefore needs `frameLocator(...).frameLocator(...)`, not one hop.
*
* Set `SMOKE_SCREENSHOT_DIR` to capture PNGs of each state (used to attach
* proof to a PR); unset, it asserts only. Playwright is resolved with a
* `createRequire` based at clients/web/package.json for the reason documented at
* length in smoke-web-browser.mjs.
* proof to a PR); unset, it asserts only.
*
* `SMOKE_BROWSER` picks the engine (`chromium` — the default — `firefox`, or
* `webkit`); CI runs all three (#2086). Along with `smoke:web:app` this is one
* of the two places the MCP Apps sandbox is actually loaded, and the two nested
* frames below are precisely the surface that diverges between engines. See
* `lib/headless-browser.mjs`, including why a green WebKit run is not a Safari
* guarantee.
*
* Expects `clients/web/dist` and `clients/launcher/build` to be built first.
* `test-servers/build` is rebuilt on every run, as in smoke:web:app — see
@ -33,20 +38,35 @@
import { spawn } from "node:child_process";
import { mkdirSync } from "node:fs";
import { createRequire } from "node:module";
import { setTimeout as delay } from "node:timers/promises";
import { join, resolve } from "node:path";
import { startProdWebServer } from "./lib/prod-web-server.mjs";
import { stopChild } from "./lib/child-cleanup.mjs";
import {
attachPageDiagnostics,
loadBrowser,
resolveBrowserName,
} from "./lib/headless-browser.mjs";
import {
ensureTestServers,
testServerEntryPath,
} from "./lib/ensure-test-servers.mjs";
const repoRoot = resolve(import.meta.dirname, "..");
const requireFromWeb = createRequire(
resolve(repoRoot, "clients/web/package.json"),
);
// Resolved before anything is started, so an unsupported SMOKE_BROWSER fails
// immediately rather than after a web server and two MCP servers are up. Every
// message carries the engine, so a matrix failure names which one broke.
let BROWSER;
try {
BROWSER = resolveBrowserName();
} catch (err) {
console.error(
`smoke:web:elicit FAILED — ${err instanceof Error ? err.message : String(err)}`,
);
process.exit(1);
}
const LABEL = `smoke:web:elicit [${BROWSER}]`;
const composableServer = testServerEntryPath(repoRoot, "composable");
const configPath = (name) =>
@ -59,9 +79,6 @@ const PORT = process.env.SMOKE_WEB_ELICIT_PORT ?? "6296";
const TOKEN = "smoke-web-elicit-token";
const TOOL = "app_choose_option";
const SHOT_DIR = process.env.SMOKE_SCREENSHOT_DIR;
// The async half of the uncaught-crash class. Kept identical to
// smoke-web-browser.mjs / smoke-web-app.mjs.
const FATAL_CONSOLE = /^Uncaught\b|Failed to fetch dynamically imported module/;
const servers = [];
let browser = null;
@ -69,7 +86,7 @@ const web = startProdWebServer({
host: HOST,
port: PORT,
token: TOKEN,
label: "smoke:web:elicit",
label: LABEL,
});
async function shutdown() {
@ -84,14 +101,14 @@ async function shutdown() {
await web.stop();
while (servers.length) {
await stopChild(servers.pop(), {
label: "smoke:web:elicit",
label: LABEL,
what: "MCP test server",
});
}
}
async function fail(message) {
console.error(`smoke:web:elicit FAILED — ${message}`);
console.error(`${LABEL} FAILED — ${message}`);
await shutdown();
process.exit(1);
}
@ -134,24 +151,6 @@ async function startMcpServer(configName) {
throw new Error(`MCP test server did not start within 30s:\n${out}`);
}
async function loadChromium() {
let chromium;
try {
({ chromium } = requireFromWeb("playwright"));
} catch (err) {
throw new Error(
`could not resolve the Playwright package from clients/web — run \`npm install\` at the repo root (${err instanceof Error ? err.message : String(err)})`,
);
}
try {
return await chromium.launch({ headless: true });
} catch (err) {
throw new Error(
`chromium failed to launch — on a bare Linux box run \`npx playwright install --with-deps chromium\` for the system libraries (${err instanceof Error ? err.message : String(err)})`,
);
}
}
async function shot(page, name) {
if (!SHOT_DIR) return;
mkdirSync(SHOT_DIR, { recursive: true });
@ -159,7 +158,7 @@ async function shot(page, name) {
path: join(SHOT_DIR, `${name}.png`),
fullPage: false,
});
console.log(`smoke:web:elicit — captured ${name}.png`);
console.log(`${LABEL} — captured ${name}.png`);
}
/** Connect to `mcpUrl` through the deep link and wait for the Tools list. */
@ -207,31 +206,24 @@ try {
// Rebuilt on every run — presence is not freshness (#2111).
ensureTestServers({
repoRoot,
label: "smoke:web:elicit",
label: LABEL,
requires: ["composable"],
});
const appUrl = await startMcpServer("app-elicitation-http");
const nativeUrl = await startMcpServer("app-elicitation-native-http");
await web.waitForReady();
browser = await loadChromium();
browser = await loadBrowser(repoRoot, BROWSER);
const page = await browser.newPage({
viewport: { width: 1280, height: 900 },
});
// Uncaught *synchronous* page errors, and their *async* twin (an unhandled
// rejection or a failed dynamic import), which Chromium reports on the
// console channel instead. Both are hard failures; every other console error
// is only a diagnostic, so benign font/React noise cannot flake CI. Same
// split as smoke:web:browser and smoke:web:app, which document it at length.
const pageErrors = [];
const consoleErrors = [];
page.on("pageerror", (err) =>
pageErrors.push(err instanceof Error ? err.message : String(err)),
);
page.on("console", (msg) => {
if (msg.type() === "error") consoleErrors.push(msg.text());
});
const fatalConsole = () => consoleErrors.filter((m) => FATAL_CONSOLE.test(m));
// Uncaught *synchronous* page errors, plus their *async* twin (an unhandled
// rejection or a failed dynamic import), which arrives on the console channel
// instead. Both are hard failures; every other console error is only a
// diagnostic, so benign font/React noise cannot flake CI. Shared with
// smoke:web:app rather than re-hand-rolled — the split is subtle enough that
// two copies would drift, and it is documented at length on the helper.
const diagnostics = attachPageDiagnostics(page);
const drive = async () => {
// ── 1. Negotiated: the server's app answers the elicitation ────────────
@ -297,27 +289,25 @@ try {
try {
await Promise.race([web.whenChildExits(), drive()]);
} catch (err) {
const diagnostics = [
...pageErrors,
...fatalConsole().map((m) => `console: ${m}`),
const notes = [
...diagnostics.pageErrors,
...diagnostics.fatalConsole().map((m) => `console: ${m}`),
];
await fail(
`${err instanceof Error ? err.message : String(err)}${
diagnostics.length
? ` — page diagnostics: ${diagnostics.join("; ")}`
: ""
notes.length ? ` — page diagnostics: ${notes.join("; ")}` : ""
}`,
);
}
// A drive that reached all of its assertions still fails if the page threw on
// the way: without this the smoke prints OK over a broken bundle.
const fatal = [...pageErrors, ...fatalConsole()];
const fatal = diagnostics.fatal();
if (fatal.length > 0) {
await fail(`page logged uncaught error(s): ${fatal.join("; ")}`);
}
console.log("smoke:web:elicit OK — app-rendered and native paths both drive");
console.log(`${LABEL} OK — app-rendered and native paths both drive`);
await shutdown();
} catch (err) {
await fail(err instanceof Error ? err.message : String(err));