docs(smoke): stop describing the WebKit smoke failure as a Safari bug

#2132 was filed asserting that a Safari user opening an MCP App gets a
permanently 'loading' widget. That was never tested. A manual check in
real Safari renders the App normally, so the claim is unsupported and is
removed from all six places it reached.

The failure under Playwright's WebKit is real and still keeps that engine
out of the CI matrix. What is not established is that any shipping
browser behaves this way — and Playwright ships its own WebKit build
whose networking stack is not Safari's, which is exactly the layer the
failure lives in.

headless-browser.mjs already carried the caveat that a green WebKit run
is not a Safari guarantee. The same sentence forbids the inverse
inference, which is the one that was made; it now says so explicitly, and
AGENTS.md carries the rule: reproduce in the real browser before claiming
user impact in one.

No behavior change.

Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-08-25 22:32:47 -04:00
commit 25eb2fb3bf
7 changed files with 26 additions and 14 deletions

View file

@ -105,7 +105,7 @@ Both exist and do different jobs. Theme files (`src/theme/<Component>.ts`) custo
- `npm run format` before committing; **`npm run ci` before pushing** (`validate` → `coverage` → `verify:build-gate` → `verify:bundle-externals` → `smoke` → Storybook). `npm run validate` is the fast inner-loop check and is **not** a substitute — it runs `test`, not `test:coverage`, so it does zero coverage gating.
- **A dependency bump must land in every install that declares it.** v2 is not a workspace — the root and each `clients/*` have their own `node_modules`, and a client's test project compiles `core/` and `test-servers/src` (which resolve from the **root**) alongside the client's own sources. Bumping a shared dependency in one manifest only puts two versions of it in one `tsc` program; for a recursive-generic surface like zod that exhausts the tsc heap (#1896). `verify:dep-lockstep` fails the build on this — it derives its candidates from what each `tsc` program actually resolves (`tsc --listFilesOnly`, keeping packages that reach one program from two installs), so a package reached only through another package's `.d.ts` counts too (#1965) — so a PR bumping a package the shared sources pull in should update the root **and every client that already lists it** — not every client unconditionally, since a package absent from an install can't skew and adding it there would be a spurious dependency.
- **A test or smoke must not touch real user state.** The web smokes run against a throwaway catalog via the shared `scripts/lib/prod-web-server.mjs` helper, never the developer's `~/.mcp-inspector/mcp.json` (#1977); the cli/tui smokes drive a temp `--catalog`; `pack:verify`'s `--web` child sets its own `MCP_CATALOG_PATH` for the same reason (#2003 — its App deep link persists a server row). Anything that boots the web backend and then *navigates* it needs that isolation, not just the scripts named `smoke:*`. A new smoke spawning its own server, or teardown that removes a work dir without first awaiting `stopChild` (the #1801 race — `child-cleanup.mjs` exports both halves and both are required), should be flagged.
- **The headless web smokes are engine-parameterized, not Chromium-only.** `smoke:web:browser` / `smoke:web:app` / `smoke:web:elicit` take their engine from `SMOKE_BROWSER` (chromium — the default — firefox, webkit) and CI covers Chromium plus Firefox (#2086; WebKit runs locally but is out of the matrix pending #2132, a real Safari SSE hang it found), because the MCP Apps sandbox is built out of the primitives that diverge between engines: `srcdoc` CSP inheritance, nested sandboxed iframes, `Permissions-Policy`, cross-frame `postMessage`. Nothing else covers that — `sandbox-csp.test.ts` asserts a policy *string*, and no Storybook story reaches the sandbox at all. Flag a new browser-driven script that launches Playwright itself instead of going through `scripts/lib/headless-browser.mjs`, an unrecognized-engine path that falls back to Chromium rather than failing (it would claim coverage that never ran), or a launch-failure message naming the wrong engine. `pack:verify` is pinned to Chromium on purpose — it is a packaging check.
- **The headless web smokes are engine-parameterized, not Chromium-only.** `smoke:web:browser` / `smoke:web:app` / `smoke:web:elicit` take their engine from `SMOKE_BROWSER` (chromium — the default — firefox, webkit) and CI covers Chromium plus Firefox (#2086; WebKit runs locally but is out of the matrix pending #2132, an undelivered-SSE-tail hang under Playwright's WebKit — not reproduced in real Safari, so don't cite it as one), because the MCP Apps sandbox is built out of the primitives that diverge between engines: `srcdoc` CSP inheritance, nested sandboxed iframes, `Permissions-Policy`, cross-frame `postMessage`. Nothing else covers that — `sandbox-csp.test.ts` asserts a policy *string*, and no Storybook story reaches the sandbox at all. Flag a new browser-driven script that launches Playwright itself instead of going through `scripts/lib/headless-browser.mjs`, an unrecognized-engine path that falls back to Chromium rather than failing (it would claim coverage that never ran), or a launch-failure message naming the wrong engine. `pack:verify` is pinned to Chromium on purpose — it is a packaging check.
- **Build output is never a gate target.** Lint, format, and typecheck read first-party source only; everything a build writes (`clients/*/build`, `clients/web/dist`, `storybook-static`, `coverage`, `test-servers/build`, `core/**/{build,dist}`, `*.tsbuildinfo`) stays out via each scope's `globalIgnores`, `format` globs, and tsconfig `include`. Gating generated code reports defects in vendored third-party source that nobody can fix, and a rule promotion turns that warning into a `validate` failure (#2043). Flag a PR that adds a build location without ignoring it in the same change, that widens an ignore to silence a finding in first-party code, or that adds a build directory to a tsconfig `include` to make a generated `.d.ts` resolve. Note the coverage guards don't catch this — they assert source is _covered_, not that output is _excluded_.
- **Lint has no warning tier.** Every `lint` script runs `--max-warnings 0`, so a warning fails `validate` exactly as an error does (#2085) — a `warn`-level `react-hooks/exhaustive-deps` finding otherwise let a stale-closure bug pass the pre-push gate and reach review. Flag a PR that silences a finding to make the gate pass (widening a `globalIgnores`, dropping a rule, or an inline disable with no justification comment); the fix is the defect, not the message. A rule meant to be enforced should be set to `error` rather than left at `warn` and carried by the flag.

View file

@ -128,8 +128,8 @@ jobs:
#
# This is the only place the MCP Apps sandbox is exercised on a non-Chromium
# engine, and it earned its keep immediately: pointing the smokes at WebKit is
# what found #2132, a Safari-only hang in the web client's SSE transport that
# every Chromium-only tier had been green through.
# what found #2132, an undelivered-SSE-tail hang no Chromium-only tier could
# see. (Note #2132 is NOT known to affect Safari — see the matrix comment.)
#
# The unit tests cannot substitute — `sandbox-csp.test.ts` asserts
# which policy STRING is built, which passes identically on an engine that
@ -149,11 +149,12 @@ jobs:
matrix:
# `webkit` belongs here and is deliberately absent: the smokes RUN in it
# (`SMOKE_BROWSER=webkit` works, and smoke:web:browser passes), but the
# two App smokes fail on a real, pre-existing Safari incompatibility in
# the web client's SSE transport — #2132, which carries the diagnosis and
# the verified fix. Adding it here is a one-word diff once that lands.
# An engine is either green or absent; a `continue-on-error` job would
# report coverage nobody is holding to a standard.
# two App smokes fail on #2132 — the SSE stream's last message is not
# delivered, so an App never leaves "loading". That is NOT reproduced in
# real Safari, so read it as a property of Playwright's WebKit build
# rather than a browser bug. Adding it here is a one-word diff once it is
# resolved. An engine is either green or absent; a `continue-on-error`
# job would report coverage nobody is holding to a standard.
browser: [firefox]
name: Sandbox smokes (${{ matrix.browser }})
steps:

View file

@ -920,7 +920,8 @@ Everything about the selection lives in **`scripts/lib/headless-browser.mjs`**
- **A launch failure names the engine that failed** and its `npx playwright install --with-deps <engine>`. Naming `chromium` while WebKit was the missing one sends the reader to install a browser they already have.
- **CI runs Chromium in the `build` job's `npm run smoke` and every other gated engine in a `Sandbox smokes (<engine>)` matrix job** — today that matrix is `[firefox]` alone (see the WebKit bullet below). It sets `fail-fast: false` so that once it holds more than one engine, a failure in one cannot hide a failure in another. Chromium is deliberately _not_ in that matrix — `build`'s `npm run smoke` is exactly the local `npm run ci` path and already covers it, and matrixing `build` itself would triple validate, the coverage gate, two verify gates and Storybook to gain three browser smokes. Both `publish` jobs `needs` the matrix, so a release cannot ship past a non-Chromium failure.
- **`pack:verify` stays pinned to Chromium**, passing it explicitly rather than reading `SMOKE_BROWSER`. It is a _packaging_ check; the engine question belongs where the sandbox is under test, and pinning it also means it can't be pointed at an engine its npm script never installed.
- **WebKit is deliberately out of the CI matrix for now, and `continue-on-error` is not the answer.** The two App smokes fail there on a real, pre-existing Safari incompatibility in the web client's SSE transport — [#2132](https://github.com/modelcontextprotocol/inspector/issues/2132) carries the diagnosis and a verified fix. Adding `webkit` to the matrix is a one-word diff once that lands; until then an engine is either green in CI or absent from it, because a job allowed to fail reports coverage nobody is held to. Worth noting the matrix found this on its first outing, against a bug every Chromium-only tier had been green through — and it is a _product_ bug, not a test one.
- **WebKit is deliberately out of the CI matrix for now, and `continue-on-error` is not the answer.** The two App smokes fail there — [#2132](https://github.com/modelcontextprotocol/inspector/issues/2132): the SSE stream's last message is not delivered, so an App never leaves "loading". Adding `webkit` to the matrix is a one-word diff once that is resolved; until then an engine is either green in CI or absent from it, because a job allowed to fail reports coverage nobody is held to.
- ⚠️ **A red WebKit run is not a Safari indictment, for the same reason a green one is not a Safari guarantee.** #2132 was first written up as a Safari bug on the strength of a Playwright-WebKit failure alone; a manual check in Safari did not reproduce it. The caveat below cuts both ways, and only one direction of it was applied. **Reproduce in the real browser before claiming user impact in one** — the divergence between Playwright's WebKit build and Safari is largest in exactly the layer (networking) where that bug lives.
- ⚠️ **Playwright's WebKit is a WebKit build, not Safari.** Close enough to catch engine-level CSP and iframe divergence; not close enough to certify Safari. Don't write, in a doc or a PR description, that a green run means Safari works.
### Build output is never a gate target

View file

@ -481,7 +481,7 @@ SMOKE_BROWSER=firefox npm run smoke:web:engine # all three smokes, one engine
Unset, the engine is `chromium` — so `npm run ci` and `npm run smoke` are unchanged. CI runs Chromium inside the `build` job's `npm run smoke`, and every other gated engine in a separate `Sandbox smokes (<engine>)` matrix job — today that matrix holds **Firefox alone**; see the paragraph below for where WebKit stands. Splitting it that way, rather than running the whole `build` job once per engine, is deliberate: validate, the coverage gate, the two verify gates and Storybook are all engine-independent. An unrecognized value is an error, not a fallback: a silent fallback would report a green Chromium run under a job labelled `webkit`. A missing browser binary fails naming the engine and its `npx playwright install --with-deps <engine>`.
**CI green today: Chromium and Firefox.** WebKit is supported by the tooling and worth running locally, but it is **not yet in the CI matrix** — the two App smokes fail there on a real, pre-existing Safari incompatibility in the web client's SSE transport, tracked with its diagnosis and a verified fix in [#2132](https://github.com/modelcontextprotocol/inspector/issues/2132). That is the matrix doing its job on its first outing: the bug affects the shipped web UI in Safari, and every Chromium-only tier had been green through it. An engine is either green in CI or absent from it — a `continue-on-error` job would report coverage nobody is held to.
**CI green today: Chromium and Firefox.** WebKit is supported by the tooling and worth running locally, but it is **not yet in the CI matrix** — the two App smokes fail there, tracked in [#2132](https://github.com/modelcontextprotocol/inspector/issues/2132): the SSE stream's last message is not delivered, so an App never leaves "loading". **This has not been reproduced in real Safari** — an MCP App opens there normally — so on current evidence it is a property of Playwright's WebKit build (whose network stack is not Safari's) rather than a browser bug users hit. It still blocks gating the engine, and it is worth understanding before WebKit joins the matrix. An engine is either green in CI or absent from it — a `continue-on-error` job would report coverage nobody is held to.
**Why these smokes specifically.** Most of the web client's behavior is React and Mantine, where a second engine buys little. The MCP Apps sandbox is the exception — it is built out of the primitives that genuinely diverge between engines: a CSP `<meta>` injected as the first `<head>` child of a `srcdoc` document, a nested sandboxed iframe, a `Permissions-Policy` `allow` attribute, and `postMessage` origin discipline across those two frames. Nothing else covers that: `sandbox-csp.test.ts` asserts which policy _string_ is built, which passes identically on an engine that ignores `<meta>` CSP entirely, and no Storybook story reaches the sandbox at all (all three App stories point the iframe at a `data:` placeholder and hand the renderer a mock bridge). Storybook itself remains Chromium-only — broadening it covers a much larger and differently-shaped surface, and is a separate decision.

View file

@ -27,6 +27,12 @@
* catch engine-level CSP and iframe divergence, and not close enough to certify
* Safari specifically — a green run here is not a Safari guarantee.
*
* **And that cuts both ways: a RED run here is not a Safari indictment.** #2132
* was written up as a Safari bug on the strength of a failure in this build
* alone; a manual check in Safari did not reproduce it. The divergence is widest
* in the networking layer, which is exactly where that failure lives. Reproduce
* in the real browser before describing a finding as one users hit.
*
* Playwright is resolved with a `createRequire` based at
* clients/web/package.json rather than a bare `import("playwright")`: a bare ESM
* specifier resolves relative to `scripts/`, not the cwd, so a `cd clients/web`

View file

@ -46,8 +46,10 @@
* `SMOKE_BROWSER` picks the engine (`chromium` — the default — `firefox`, or
* `webkit`). CI gates **Chromium** (in the `build` job's `npm run smoke`) and
* **Firefox** (in the `Sandbox smokes` matrix job); WebKit runs here but is not
* gated yet — it fails on #2132, a Safari incompatibility in the web client's
* SSE transport that this smoke is what found (#2086).
* gated yet — it fails on #2132, where the SSE stream's last message is not
* delivered under Playwright's WebKit. That has NOT been reproduced in real
* Safari (an MCP App opens there fine), so treat it as a property of that
* build, not as a browser bug, until someone shows otherwise (#2086).
*
* This smoke is one of the two places the
* MCP Apps sandbox is genuinely exercised, and the sandbox is built out of the

View file

@ -27,8 +27,10 @@
* `SMOKE_BROWSER` picks the engine (`chromium` — the default — `firefox`, or
* `webkit`). CI gates **Chromium** (in the `build` job's `npm run smoke`) and
* **Firefox** (in the `Sandbox smokes` matrix job); WebKit runs here but is not
* gated yet — it fails on #2132, a Safari incompatibility in the web client's
* SSE transport that this smoke is what found (#2086).
* gated yet — it fails on #2132, where the SSE stream's last message is not
* delivered under Playwright's WebKit. That has NOT been reproduced in real
* Safari (an MCP App opens there fine), so treat it as a property of that
* build, not as a browser bug, until someone shows otherwise (#2086).
*
* Along with `smoke:web:app` this is one
* of the two places the MCP Apps sandbox is actually loaded, and the two nested