The web client had exactly one connected-flow smoke and it was the Apps
tab. `smoke:web:browser` proves the bundle boots and paints but never
connects; `smoke:web:app` / `smoke:web:elicit` connect but only drive the
Apps surface. So the three highest-traffic tabs were unexercised through
the prod bundle end to end — everything covering them (unit tests,
Storybook, the integration project) mocks or bypasses the prod server, so
a regression living only in the built bundle or in the prod Hono
backend's wiring of a tab passed all of them.
`scripts/smoke-web-tabs.mjs` drives all three against one server on one
browser launch (~5s added wall-clock): Tools runs `list_items` and
asserts its `structuredContent` section rendered (#1908), Resources reads
`events` with the template list asserted separately — `resources/list`
and `resources/templates/list` are different calls and either can fail
alone — and Prompts fetches the argument-less `simple_prompt`.
Every assertion is on a `data-*` attribute, never on copy, mirroring the
`data-app-status` precedent: each screen now carries a readiness contract
(`data-tool-count` / `data-call-status` and its Resources / Prompts
equivalents), tabulated in the web README and pinned by screen unit tests
so a rename fails loudly there rather than as an opaque timeout here.
The deep-link connect step is extracted to `lib/deep-link-connect.mjs`
rather than copied a third time. That matters because `parseDeepLink`
ignores a link it cannot validate rather than reporting one, so a drifted
copy fails as an opaque connect timeout in whichever smoke was not
updated. `smoke:web:elicit`'s hand-rolled copy is retired onto it.
Chromium-only and deliberately not in `ENGINE_SMOKES`: the engine tiers
exist for the MCP Apps sandbox's genuinely divergent primitives, and
these tabs are ordinary React and Mantine. Network, Protocol, Logs and
Tasks remain uncovered — a stated gap, not an oversight.
Closes#2148
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UL9retfvAXRgvi6EWk4SY4
Signed-off-by: cliffhall <cliff@futurescale.com>
Every browser-driven check in this repo launched Chromium and nothing else.
That is fine for most of the web client, whose behavior is React and Mantine.
It is not fine for the MCP Apps sandbox, which is built out of the primitives
that genuinely diverge between engines: a CSP <meta> injected as the first
<head> child of a srcdoc document, a nested sandboxed iframe, a
Permissions-Policy allow attribute, and postMessage origin discipline across
those two frames. Nothing else covers that surface — sandbox-csp.test.ts
asserts which policy STRING is built (environment-independent by
construction), and no Storybook story reaches the sandbox at all, since all
three App stories use a data: placeholder iframe and a mock bridge.
The engine is now a parameter:
SMOKE_BROWSER=webkit npm run smoke:web:app
SMOKE_BROWSER=firefox npm run smoke:web:engine # all three smokes
Unset means chromium, so `npm run ci` and `npm run smoke` are unchanged. An
unrecognized value is an error rather than a fallback — falling back would
report a green Chromium run under a job labelled "webkit", claiming coverage
that never ran — and a launch failure names the engine that failed and its own
`playwright install` invocation.
Selection, launching, and the fatal-vs-benign page-diagnostics split now live
in one shared scripts/lib/headless-browser.mjs. The three smokes had each
hand-rolled the diagnostics split and two had their own copy of loadChromium;
pack:verify goes through the same helper, pinned to Chromium explicitly since
it is a packaging check.
The npm scripts drop their `cd clients/web && npx playwright install chromium`
prefix for scripts/install-smoke-browser.mjs. The engine is a variable now, and
a ${VAR:-default} expansion does not expand under Windows' cmd.exe; npx is also
a .cmd shim there that a shell-free spawn cannot start, so this resolves the
Playwright CLI through resolveNodeBin — the #1939 problem solved the #1939 way.
CI gains a `Sandbox smokes (<engine>)` matrix job rather than matrixing `build`,
which also runs validate, the coverage gate, two verify gates and Storybook —
all engine-independent. Both publish jobs now need it.
WebKit is supported by the tooling but deliberately NOT in the CI matrix yet:
pointing the smokes at it immediately found #2132, a pre-existing Safari
incompatibility in the web client's SSE transport that strands the last message
on /api/mcp/events, so an MCP App never leaves "loading". That is a product bug
affecting real Safari users, with its own diagnosis and a verified fix, and it
is not a matrix change. Adding `webkit` here is a one-word diff once it lands.
An engine is either green in CI or absent from it; a continue-on-error job would
report coverage nobody is held to.
Signed-off-by: cliffhall <cliff@futurescale.com>
Five `scripts/*.mjs` consumers each carried their own `ensureTestServer()`
that returned early when `test-servers/build` was already on disk. Presence
is not freshness: once that directory exists — which it does on any machine
that has run the smokes once — an edit to `test-servers/src` was never
picked up again, and the smoke went on driving the previously-built fixture.
The failure is silent and misattributed. It does not read as "your fixture
is stale"; it reads as a product bug in whatever the smoke is checking.
`scripts/lib/ensure-test-servers.mjs` is now the one place that build is
produced for a script consumer, and it builds unconditionally — memoized
once per process per repo root, so pack-and-verify's two asks stay one tsc
pass while a fresh run always rebuilds. Callers name the entries they
depend on and get one actionable failure message, replacing five that had
drifted. pack-and-verify also stops shelling out to `npx tsc`, which is
what made it the one script unrunnable on Windows (#1939).
`test-servers/tsconfig.json` gains `incremental` so the up-to-date case is
cheap: 3.0s cold, 1.6s incremental here.
Unconditional emit is still not a clean — `--noCheck` leaves the stale .js
of a deleted src file behind — which is unchanged from before and recorded
in the helper's header and AGENTS.md.
Closes#2111
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JTHVxSu8AUgHRLvo1ntZ8H
Signed-off-by: cliffhall <cliff@futurescale.com>
One conflict, scripts/smoke-web-app.mjs: #2056 added a second phase to the
smoke (the same app declaring _meta.ui.domain, asserting it renders from the
dedicated app origin rather than an opaque one) while this branch extracted
the smoke's flow into scripts/lib/mcp-app-flow.mjs.
Resolved by keeping both: phase 2 now runs through the shared helper too,
which grows three knobs it needs and pack:verify does not —
- expectDeepLink: phase 2 re-navigates the SAME page, so the token gate is
already proven and a rejected re-navigate would fail at the connect wait
regardless;
- what: so a phase-2 ready timeout doesn't read as a phase-1 one;
- extraDiagnostics: phase 2 appends the frame list, which is what separates
"never rendered" from "rendered at the wrong origin".
Each is covered by a new unit test, since neither consumer exercises a
failure branch.
pack:verify deliberately drives phase 1 only. The dedicated origin is served
by the same runner the packaging assertions already cover and reads nothing
new from disk (app-origin-controller.ts has no file reads), so a second
browser launch there would buy no packaging signal.
Verified on the merged tree: smoke:web:app passes both phases (dedicated
origin http://127.0.0.1:6278/app-document/<id>), and pack:verify passes end
to end against the real tarball.
Signed-off-by: cliffhall <cliff@futurescale.com>
Nothing covered the intersection #1859 fell through. `smoke:web:app` drives
the App path but against the repo build tree; `pack:verify` runs against the
installed tarball but only asserted the sandbox proxy page *exists*. So a
packaging regression that ships the file present-but-unreachable — a rename
with a stale reader, a wrong position relative to clients/web/build, a future
"files"/.npmignore edit — passes both.
Extract the drive-the-App-flow half of the smoke into
scripts/lib/mcp-app-flow.mjs and call it from both, rather than copying it:
the deep-link shape is the part that rots silently, and a drifted link fails
as an opaque timeout rather than a mismatch. pack:verify then drives
connect -> open app -> data-app-status="ready" in headless Chromium against
the installed `--web` boot it already starts.
The client under test comes from the install; the App test server stays a
repo fixture.
Also isolates pack:verify's `--web` catalog (the #1977 treatment the smokes
already have). Untidy while the step only did `GET /`; the App deep link
persists a server row, which would make it destructive and non-deterministic
against the developer's real ~/.mcp-inspector/mcp.json.
Unit tests cover what neither consumer can check itself — both CSRF gates and
the appArgs encoding on the deep link, and driveAppFlow's failure branches
against a stand-in page, all dead code from a happy-path smoke's view.
Verified: pack:verify passes end to end against the real tarball, and fails
at the new step alone when sandbox-controller.ts is pointed at a renamed
proxy page (the tarball and on-disk assertions both still pass — the exact
gap this closes).
Signed-off-by: cliffhall <cliff@futurescale.com>
