modelcontextprotocol/inspector
Watch
1
0
Fork
You've already forked inspector
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

test(smoke): run the headless web smokes in Firefox and WebKit, not just Chromium (#2086)

Browse source 
Every browser-driven check in this repo launched Chromium and nothing else.
That is fine for most of the web client, whose behavior is React and Mantine.
It is not fine for the MCP Apps sandbox, which is built out of the primitives
that genuinely diverge between engines: a CSP <meta> injected as the first
<head> child of a srcdoc document, a nested sandboxed iframe, a
Permissions-Policy allow attribute, and postMessage origin discipline across
those two frames. Nothing else covers that surface — sandbox-csp.test.ts
asserts which policy STRING is built (environment-independent by
construction), and no Storybook story reaches the sandbox at all, since all
three App stories use a data: placeholder iframe and a mock bridge.

The engine is now a parameter:

  SMOKE_BROWSER=webkit npm run smoke:web:app
  SMOKE_BROWSER=firefox npm run smoke:web:engine   # all three smokes

Unset means chromium, so `npm run ci` and `npm run smoke` are unchanged. An
unrecognized value is an error rather than a fallback — falling back would
report a green Chromium run under a job labelled "webkit", claiming coverage
that never ran — and a launch failure names the engine that failed and its own
`playwright install` invocation.

Selection, launching, and the fatal-vs-benign page-diagnostics split now live
in one shared scripts/lib/headless-browser.mjs. The three smokes had each
hand-rolled the diagnostics split and two had their own copy of loadChromium;
pack:verify goes through the same helper, pinned to Chromium explicitly since
it is a packaging check.

The npm scripts drop their `cd clients/web && npx playwright install chromium`
prefix for scripts/install-smoke-browser.mjs. The engine is a variable now, and
a ${VAR:-default} expansion does not expand under Windows' cmd.exe; npx is also
a .cmd shim there that a shell-free spawn cannot start, so this resolves the
Playwright CLI through resolveNodeBin — the #1939 problem solved the #1939 way.

CI gains a `Sandbox smokes (<engine>)` matrix job rather than matrixing `build`,
which also runs validate, the coverage gate, two verify gates and Storybook —
all engine-independent. Both publish jobs now need it.

WebKit is supported by the tooling but deliberately NOT in the CI matrix yet:
pointing the smokes at it immediately found #2132, a pre-existing Safari
incompatibility in the web client's SSE transport that strands the last message
on /api/mcp/events, so an MCP App never leaves "loading". That is a product bug
affecting real Safari users, with its own diagnosis and a verified fix, and it
is not a matrix change. Adding `webkit` here is a one-word diff once it lands.
An engine is either green in CI or absent from it; a continue-on-error job would
report coverage nobody is held to.

Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-08-25 20:43:55 -04:00
commit ef4259a544
1 changed files with 652 additions and 222 deletions
Download patch file Download diff file Expand all files Collapse all files

127
scripts/smoke-web-browser.mjs
View file

@ -29,12 +29,13 @@
* Two channels carry the failure. A *synchronous* uncaught exception (the
* CASE-1 shape above — a stub call during module init) fires `pageerror`. Its
* *async* twin — the same `TypeError` reached through an `await`/`.then()`, or a
* failed dynamic import (this app lazy-loads chunks) — is NOT a `pageerror`;
* Chromium logs it on the **console** channel as `Uncaught (in promise) …` /
* `Failed to fetch dynamically imported module`. Both are hard failures.
* failed dynamic import (this app lazy-loads chunks) — is NOT a `pageerror`; it
* arrives on the **console** channel as `Uncaught (in promise) …` /
* `Failed to fetch dynamically imported module`. Both are hard failures, and
* both channels are read on every engine rather than branching per browser.
*
* Every *other* `console.error` is NOT a hard failure: the console is where
* Chromium also reports benign things a boot smoke shouldn't fail on — a failed
* Every *other* `console.error` is NOT a hard failure: the console is also where
* benign things a boot smoke shouldn't fail on land — a failed
* subresource load (e.g. the Google-Fonts `<link>` in index.html on a
* network-restricted box) or a React key/prop warning. Those are printed as
* diagnostics. The `Uncaught` / dynamic-import prefixes are unambiguous — a
@ -42,29 +43,44 @@
* never starts with `Uncaught` — so hard-failing on them can't reintroduce that
* flake.
*
* Playwright lives in clients/web's node_modules, so it's resolved with a
* `createRequire` based at clients/web/package.json rather than a bare
* `import("playwright")`. A bare ESM specifier resolves relative to *this
* script's* directory (scripts/), not the cwd — so `cd clients/web` in the npm
* script would NOT make it resolvable (it only appeared to work locally when an
* ancestor node_modules happened to carry playwright; it fails in CI, which has
* none). createRequire pins resolution to clients/web regardless of cwd.
* Launching the browser (and resolving Playwright from clients/web, which has
* its own gotcha — see `lib/headless-browser.mjs`) is delegated to that module,
* which is also where `SMOKE_BROWSER` picks the engine: `chromium` (the
* default), `firefox`, or `webkit`, all three of which CI runs (#2086). The
* engine question here is narrower than in the App smokes — this asserts a clean
* first paint, i.e. that the shipped bundle's syntax and API level are
* *reachable* on the engine at all, rather than anything about the sandbox — but
* it is real, and it is nearly free to include.
*
* Expects `clients/web/dist` and `clients/launcher/build` to be built first —
* the validate / CI ordering guarantees this.
*/
import { createRequire } from "node:module";
import { setTimeout as delay } from "node:timers/promises";
import { fileURLToPath } from "node:url";
import { dirname, resolve } from "node:path";
import { startProdWebServer } from "./lib/prod-web-server.mjs";
import {
attachPageDiagnostics,
loadBrowser,
resolveBrowserName,
} from "./lib/headless-browser.mjs";
const scriptDir = dirname(fileURLToPath(import.meta.url));
// Resolve playwright from clients/web (where it's installed) no matter the cwd.
const requireFromWeb = createRequire(
resolve(scriptDir, "..", "clients/web/package.json"),
);
const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
// Resolved before the web server is started, so an unsupported SMOKE_BROWSER
// fails immediately. Every message carries the engine, so a matrix failure names
// which one broke.
let BROWSER;
try {
BROWSER = resolveBrowserName();
} catch (err) {
console.error(
`smoke:web:browser FAILED — ${err instanceof Error ? err.message : String(err)}`,
);
process.exit(1);
}
const LABEL = `smoke:web:browser [${BROWSER}]`;
const HOST = "127.0.0.1";
// Distinct from smoke:web's SMOKE_WEB_PORT so overriding one doesn't make both
@ -72,17 +88,11 @@ const HOST = "127.0.0.1";
const PORT = process.env.SMOKE_WEB_BROWSER_PORT ?? "6298";
const TOKEN = "smoke-web-browser-token";
// Console messages that are the async half of the uncaught-crash class (an
// unhandled promise rejection or a failed dynamic import). Hard failures, unlike
// benign console noise. See the header comment for why this can't reintroduce
// the font/CDN flake.
const FATAL_CONSOLE = /^Uncaught\b|Failed to fetch dynamically imported module/;
const server = startProdWebServer({
host: HOST,
port: PORT,
token: TOKEN,
label: "smoke:web:browser",
label: LABEL,
});
let browser = null;
@ -99,51 +109,22 @@ async function shutdown() {
}
async function fail(message) {
console.error(`smoke:web:browser FAILED — ${message}`);
console.error(`${LABEL} FAILED — ${message}`);
await shutdown();
process.exit(1);
}
async function loadChromium() {
let chromium;
try {
({ chromium } = requireFromWeb("playwright"));
} catch (err) {
// A failure here means the playwright *npm package* isn't resolvable from
// clients/web — fixed by installing devDependencies, NOT by `playwright
// install` (which fetches browser binaries). `npm install` at the repo root
// cascades into clients/web via postinstall.
throw new Error(
`could not resolve the Playwright package from clients/web — run \`npm install\` at the repo root (${err instanceof Error ? err.message : String(err)})`,
);
}
try {
return await chromium.launch({ headless: true });
} catch (err) {
throw new Error(
`chromium failed to launch — on a bare Linux box run \`npx playwright install --with-deps chromium\` for the system libraries (${err instanceof Error ? err.message : String(err)})`,
);
}
}
try {
await server.waitForReady();
browser = await loadChromium();
browser = await loadBrowser(repoRoot, BROWSER);
const page = await browser.newPage();
// Uncaught (synchronous) page errors are a hard failure — a Node-only module
// reaching the browser bundle surfaces here as a TypeError when its empty stub
// is called during module init.
const pageErrors = [];
// Console errors are diagnostic only EXCEPT those matching FATAL_CONSOLE (the
// async half of the same crash class) — see the header comment.
const consoleErrors = [];
page.on("pageerror", (err) =>
pageErrors.push(err instanceof Error ? err.message : String(err)),
);
page.on("console", (msg) => {
if (msg.type() === "error") consoleErrors.push(msg.text());
});
// is called during module init. Console errors are diagnostic only EXCEPT the
// async half of that same crash class; see the header comment, and the shared
// helper for the split.
const diagnostics = attachPageDiagnostics(page);
const render = async () => {
// Token is injected into index.html by the prod server, so a bare `/` load
@ -175,39 +156,35 @@ try {
try {
await Promise.race([server.whenChildExits(), render()]);
} catch (err) {
const diagnostics = [
...pageErrors,
...consoleErrors.map((m) => `console: ${m}`),
const notes = [
...diagnostics.pageErrors,
...diagnostics.consoleErrors.map((m) => `console: ${m}`),
];
await fail(
`${err instanceof Error ? err.message : String(err)}${
diagnostics.length
? ` — page diagnostics: ${diagnostics.join("; ")}`
: ""
notes.length ? ` — page diagnostics: ${notes.join("; ")}` : ""
}`,
);
}
// Hard failures: any uncaught (sync) page error, plus console errors that are
// the async half of the class (unhandled rejection / failed dynamic import).
const fatalConsole = consoleErrors.filter((m) => FATAL_CONSOLE.test(m));
if (pageErrors.length > 0 || fatalConsole.length > 0) {
await fail(
`app logged uncaught error(s): ${[...pageErrors, ...fatalConsole].join("; ")}`,
);
const fatal = diagnostics.fatal();
if (fatal.length > 0) {
await fail(`app logged uncaught error(s): ${fatal.join("; ")}`);
}
// Non-fatal console errors: surface them so a real problem isn't invisible,
// without failing the smoke on benign subresource/warning noise.
const benignConsole = consoleErrors.filter((m) => !FATAL_CONSOLE.test(m));
if (benignConsole.length > 0) {
const benign = diagnostics.benignConsole();
if (benign.length > 0) {
console.log(
`smoke:web:browser note — ${benignConsole.length} non-fatal console error(s): ${benignConsole.join("; ")}`,
`${LABEL} note — ${benign.length} non-fatal console error(s): ${benign.join("; ")}`,
);
}
console.log(
`smoke:web:browser OK — app booted at ${server.baseUrl}, rendered "Add Servers" with no uncaught errors (sync page error or unhandled rejection)`,
`${LABEL} OK — app booted at ${server.baseUrl}, rendered "Add Servers" with no uncaught errors (sync page error or unhandled rejection)`,
);
await shutdown();
process.exit(0);