ci: drop the Firefox smoke job; keep the engines as an on-demand tool
The job was trialled and did not earn its place. It was genuinely cheap — ~2 minutes, running in parallel with the 15-minute build job, so zero added wall-clock — but across a dozen runs Firefox never once disagreed with Chromium. It was paying runner time and carrying a real flake surface (playwright install --with-deps runs apt-get update, which fails whenever a third-party repo in the runner image breaks; that is what the one red run was) to re-confirm a result already in hand. Everything that made cross-engine checking possible stays: SMOKE_BROWSER, the shared headless-browser helper, the installer, and the tests. So "SMOKE_BROWSER=firefox npm run smoke:web:engine" is there for the moment it is worth running — before touching the sandbox, the CSP builder or the proxy page — without taxing every push to re-derive the same answer. The workflow is now byte-identical to v2/main again; the publish jobs go back to needing only build. Docs say what is true rather than what was intended, and AGENTS.md records the reasoning so the job is not re-added on the general argument that cross-engine coverage is good — that argument was already accepted, and the on-demand path is what serves it. Re-add it on evidence: a real cross-engine regression a gate would have caught. Also swept the now-stale "matrix" language out of the smokes and the resolver, including the empty-SMOKE_BROWSER rationale, which no longer leans on a matrix key that does not exist. Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall
parent
fb9e146043
commit
8a7ade7b08
10 changed files with 64 additions and 140 deletions
2
.github/copilot-instructions.md
vendored
2
.github/copilot-instructions.md
vendored
|
|
@ -105,7 +105,7 @@ Both exist and do different jobs. Theme files (`src/theme/<Component>.ts`) custo
|
|||
- `npm run format` before committing; **`npm run ci` before pushing** (`validate` → `coverage` → `verify:build-gate` → `verify:bundle-externals` → `smoke` → Storybook). `npm run validate` is the fast inner-loop check and is **not** a substitute — it runs `test`, not `test:coverage`, so it does zero coverage gating.
|
||||
- **A dependency bump must land in every install that declares it.** v2 is not a workspace — the root and each `clients/*` have their own `node_modules`, and a client's test project compiles `core/` and `test-servers/src` (which resolve from the **root**) alongside the client's own sources. Bumping a shared dependency in one manifest only puts two versions of it in one `tsc` program; for a recursive-generic surface like zod that exhausts the tsc heap (#1896). `verify:dep-lockstep` fails the build on this — it derives its candidates from what each `tsc` program actually resolves (`tsc --listFilesOnly`, keeping packages that reach one program from two installs), so a package reached only through another package's `.d.ts` counts too (#1965) — so a PR bumping a package the shared sources pull in should update the root **and every client that already lists it** — not every client unconditionally, since a package absent from an install can't skew and adding it there would be a spurious dependency.
|
||||
- **A test or smoke must not touch real user state.** The web smokes run against a throwaway catalog via the shared `scripts/lib/prod-web-server.mjs` helper, never the developer's `~/.mcp-inspector/mcp.json` (#1977); the cli/tui smokes drive a temp `--catalog`; `pack:verify`'s `--web` child sets its own `MCP_CATALOG_PATH` for the same reason (#2003 — its App deep link persists a server row). Anything that boots the web backend and then *navigates* it needs that isolation, not just the scripts named `smoke:*`. A new smoke spawning its own server, or teardown that removes a work dir without first awaiting `stopChild` (the #1801 race — `child-cleanup.mjs` exports both halves and both are required), should be flagged.
|
||||
- **The headless web smokes are engine-parameterized, not Chromium-only.** `smoke:web:browser` / `smoke:web:app` / `smoke:web:elicit` take their engine from `SMOKE_BROWSER` (chromium — the default — firefox, webkit) and CI covers Chromium plus Firefox (#2086; WebKit runs locally but is out of the matrix — the App smokes fail under it for unidentified reasons that do not reproduce in real Safari, and chasing it was dropped), because the MCP Apps sandbox is built out of the primitives that diverge between engines: `srcdoc` CSP inheritance, nested sandboxed iframes, `Permissions-Policy`, cross-frame `postMessage`. Nothing else covers that — `sandbox-csp.test.ts` asserts a policy *string*, and no Storybook story reaches the sandbox at all. Flag a new browser-driven script that launches Playwright itself instead of going through `scripts/lib/headless-browser.mjs`, an unrecognized-engine path that falls back to Chromium rather than failing (it would claim coverage that never ran), or a launch-failure message naming the wrong engine. `pack:verify` is pinned to Chromium on purpose — it is a packaging check.
|
||||
- **The headless web smokes are engine-parameterized, not Chromium-only.** `smoke:web:browser` / `smoke:web:app` / `smoke:web:elicit` take their engine from `SMOKE_BROWSER` (chromium — the default — firefox, webkit). **CI runs Chromium only**; the others are an on-demand tool, deliberately not a gate (#2086 — a Firefox job was trialled and removed for never disagreeing with Chromium). They exist because the MCP Apps sandbox is built out of the primitives that diverge between engines: `srcdoc` CSP inheritance, nested sandboxed iframes, `Permissions-Policy`, cross-frame `postMessage`. Nothing else covers that — `sandbox-csp.test.ts` asserts a policy *string*, and no Storybook story reaches the sandbox at all. Flag a new browser-driven script that launches Playwright itself instead of going through `scripts/lib/headless-browser.mjs`, an unrecognized-engine path that falls back to Chromium rather than failing (it would claim coverage that never ran), or a launch-failure message naming the wrong engine. `pack:verify` is pinned to Chromium on purpose — it is a packaging check.
|
||||
|
||||
- **Build output is never a gate target.** Lint, format, and typecheck read first-party source only; everything a build writes (`clients/*/build`, `clients/web/dist`, `storybook-static`, `coverage`, `test-servers/build`, `core/**/{build,dist}`, `*.tsbuildinfo`) stays out via each scope's `globalIgnores`, `format` globs, and tsconfig `include`. Gating generated code reports defects in vendored third-party source that nobody can fix, and a rule promotion turns that warning into a `validate` failure (#2043). Flag a PR that adds a build location without ignoring it in the same change, that widens an ignore to silence a finding in first-party code, or that adds a build directory to a tsconfig `include` to make a generated `.d.ts` resolve. Note the coverage guards don't catch this — they assert source is _covered_, not that output is _excluded_.
|
||||
- **Lint has no warning tier.** Every `lint` script runs `--max-warnings 0`, so a warning fails `validate` exactly as an error does (#2085) — a `warn`-level `react-hooks/exhaustive-deps` finding otherwise let a stale-closure bug pass the pre-push gate and reach review. Flag a PR that silences a finding to make the gate pass (widening a `globalIgnores`, dropping a rule, or an inline disable with no justification comment); the fix is the defect, not the message. A rule meant to be enforced should be set to `error` rather than left at `warn` and carried by the flag.
|
||||
|
|
|
|||
94
.github/workflows/main.yml
vendored
94
.github/workflows/main.yml
vendored
|
|
@ -28,8 +28,8 @@ jobs:
|
|||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v7
|
||||
with:
|
||||
node-version: "22.x"
|
||||
cache: "npm"
|
||||
node-version: '22.x'
|
||||
cache: 'npm'
|
||||
|
||||
- name: Install dependencies (root + all clients)
|
||||
# The root postinstall (scripts/install-clients.mjs) cascades
|
||||
|
|
@ -115,86 +115,6 @@ jobs:
|
|||
working-directory: ./clients/web
|
||||
run: npm run test:storybook
|
||||
|
||||
# The same headless web smokes the `build` job already runs in Chromium, run
|
||||
# again in the other engines this repo supports (#2086).
|
||||
#
|
||||
# Why this is a job of its own rather than a matrix over `build`: `build` also
|
||||
# runs validate, the coverage gate, two verify gates and Storybook, none of
|
||||
# which are engine-dependent — matrixing it would triple all of that to gain
|
||||
# three browser smokes. Chromium is deliberately NOT in the matrix here for
|
||||
# the same reason: `npm run smoke` inside `build` is exactly the local
|
||||
# `npm run ci` path, and it already covers it. Supported set = the Chromium run
|
||||
# there plus whatever the matrix below names.
|
||||
#
|
||||
# This is the only place the MCP Apps sandbox is exercised on a non-Chromium
|
||||
# engine — which is why Firefox is gated here rather than assumed to behave
|
||||
# like Chromium.
|
||||
#
|
||||
# The unit tests cannot substitute — `sandbox-csp.test.ts` asserts
|
||||
# which policy STRING is built, which passes identically on an engine that
|
||||
# ignores `<meta>` CSP entirely — and no Storybook story reaches the sandbox at
|
||||
# all (all three App stories use a `data:` placeholder iframe and a mock
|
||||
# bridge). Note Playwright's WebKit is a WebKit build, not Safari: close enough
|
||||
# to catch engine-level CSP and iframe divergence, not close enough to certify
|
||||
# Safari.
|
||||
browser-engine-smokes:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
# Kept on despite the matrix currently holding one engine: the moment a
|
||||
# second is added back (see below), failing fast would hide one engine's
|
||||
# regression behind another's — which is the exact distinction this job
|
||||
# exists to draw, so the flag should not have to be remembered then.
|
||||
fail-fast: false
|
||||
matrix:
|
||||
# `webkit` belongs here and is deliberately absent: the smokes RUN in it
|
||||
# (`SMOKE_BROWSER=webkit` works, and smoke:web:browser passes), but the
|
||||
# two App smokes fail there for reasons nobody has identified. It does
|
||||
# NOT reproduce in real Safari, and an isolated SSE repro did not
|
||||
# reproduce it under Playwright's WebKit either, so it reads as a
|
||||
# property of that build rather than a browser bug — and chasing it
|
||||
# further was judged not worth the effort. Adding it here is a one-word diff once it is
|
||||
# resolved. An engine is either green or absent; a `continue-on-error`
|
||||
# job would report coverage nobody is holding to a standard.
|
||||
browser: [firefox]
|
||||
name: Sandbox smokes (${{ matrix.browser }})
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v7
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v7
|
||||
with:
|
||||
node-version: "22.x"
|
||||
cache: "npm"
|
||||
|
||||
- name: Install dependencies (root + all clients)
|
||||
run: npm install
|
||||
|
||||
- name: Build all clients
|
||||
# The smokes need clients/web/dist and the cli/tui/launcher bundles.
|
||||
# `build` rather than `validate` — the format/lint/typecheck half is
|
||||
# engine-independent and already ran in the `build` job.
|
||||
run: npm run build
|
||||
|
||||
- name: Cache Playwright browsers
|
||||
uses: actions/cache@v6
|
||||
with:
|
||||
path: ~/.cache/ms-playwright
|
||||
key: playwright-${{ matrix.browser }}-${{ runner.os }}-${{ hashFiles('clients/web/package-lock.json') }}
|
||||
|
||||
- name: Install Playwright ${{ matrix.browser }}
|
||||
working-directory: ./clients/web
|
||||
# `--with-deps` for the system libraries a bare runner lacks; WebKit is
|
||||
# the large download here and dominates this step.
|
||||
run: npx playwright install --with-deps ${{ matrix.browser }}
|
||||
|
||||
- name: Run the headless web smokes in ${{ matrix.browser }}
|
||||
# The set of smokes lives in the `smoke:web:engine` npm script, not
|
||||
# enumerated here, so adding one covers every engine automatically.
|
||||
env:
|
||||
SMOKE_BROWSER: ${{ matrix.browser }}
|
||||
run: npm run smoke:web:engine
|
||||
|
||||
# Publish the single `@modelcontextprotocol/inspector` package to npm on a
|
||||
# published GitHub release. v2 is not an npm workspace, so there is no
|
||||
# `publish-all` / `--workspaces` (v1) — just one `npm publish`, whose `prepack`
|
||||
|
|
@ -207,7 +127,7 @@ jobs:
|
|||
runs-on: ubuntu-latest
|
||||
if: github.event_name == 'release'
|
||||
environment: release
|
||||
needs: [build, browser-engine-smokes]
|
||||
needs: build
|
||||
# Serialize publishes so two releases cut in quick succession can't run
|
||||
# overlapping `npm publish`es. Never cancel an in-flight publish.
|
||||
concurrency:
|
||||
|
|
@ -225,9 +145,9 @@ jobs:
|
|||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v7
|
||||
with:
|
||||
node-version: "22.x"
|
||||
cache: "npm"
|
||||
registry-url: "https://registry.npmjs.org"
|
||||
node-version: '22.x'
|
||||
cache: 'npm'
|
||||
registry-url: 'https://registry.npmjs.org'
|
||||
|
||||
- name: Assert release tag matches package version
|
||||
# `npm publish` ships whatever `version` is in the root package.json,
|
||||
|
|
@ -300,7 +220,7 @@ jobs:
|
|||
runs-on: ubuntu-latest
|
||||
if: github.event_name == 'release'
|
||||
environment: release
|
||||
needs: [build, browser-engine-smokes]
|
||||
needs: build
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
|
|
|
|||
|
|
@ -908,9 +908,9 @@ The ⚠️ option-deletion hazard, the snapshot rule, and the recovery recipe ab
|
|||
- `smoke:cli` (`scripts/smoke-cli.mjs`) drives `mcp-inspector --cli` through the built launcher against the bundled stdio test server via a temp `--catalog`: it asserts `tools/list` returns the server's tools (real connect over stdio), the default writable catalog is seeded empty on first run, a missing read-only `--config` errors without seeding, and `--catalog` + `--config` is rejected. `smoke:tui` (`scripts/smoke-tui.mjs`) launches `mcp-inspector --tui --catalog <temp>` and asserts the Ink app renders its first frame (the "MCP Servers" panel) within a timeout, then SIGTERMs it — a shallow boot/render check, not full interaction. **`smoke:tui` is local-only: it self-skips when `process.env.CI` is set**, because the Ink TUI needs a real TTY (raw mode) that headless CI lacks — so run it (via `npm run smoke`) on your own machine before pushing. Both build `test-servers/build` on demand if it's missing.
|
||||
- Storybook play-function tests (`clients/web` `test:storybook`) run in headless Chromium via `@vitest/browser-playwright` (~10s). They are part of `npm run ci` (which installs Playwright chromium first); kept out of `validate` because they need the browser binary and are slower than the unit suite.
|
||||
|
||||
### The web smokes are engine-parameterized; CI gates Chromium and Firefox (#2086)
|
||||
### The web smokes are engine-parameterized; CI runs Chromium only (#2086)
|
||||
|
||||
**`smoke:web:browser`, `smoke:web:app` and `smoke:web:elicit` support Chromium, Firefox and WebKit; `SMOKE_BROWSER` picks one, unset means `chromium`.** CI is green on Chromium and Firefox; **WebKit runs but is not in the CI matrix yet** — see the last bullet. `npm run smoke:web:engine` runs all three smokes in whichever engine is selected — that script, not the workflow YAML, is the list of engine-covered smokes, so adding a fourth covers every engine without touching CI.
|
||||
**`smoke:web:browser`, `smoke:web:app` and `smoke:web:elicit` support Chromium, Firefox and WebKit; `SMOKE_BROWSER` picks one, unset means `chromium`.** **CI runs Chromium only** — the other engines are an on-demand tool, not a gate (see the CI bullet for why). `npm run smoke:web:engine` runs all three smokes in whichever engine is selected — that script, not the workflow YAML, is the list of engine-covered smokes, so adding a fourth covers every engine without touching CI.
|
||||
|
||||
Everything about the selection lives in **`scripts/lib/headless-browser.mjs`** — `resolveBrowserName`, `loadBrowser`, and the `attachPageDiagnostics` / `FATAL_CONSOLE` split the smokes had each hand-rolled. Reach for it rather than launching Playwright in a new script.
|
||||
|
||||
|
|
@ -918,9 +918,9 @@ Everything about the selection lives in **`scripts/lib/headless-browser.mjs`**
|
|||
- **No other tier can substitute, so don't propose one.** `sandbox-csp.test.ts` asserts which policy _string_ is built — environment-independent by construction, and it would pass identically on an engine that ignores `<meta>` CSP entirely. And **no Storybook story reaches the sandbox at all**: all three App stories (`AppRenderer`, `AppsScreen`, `AppElicitationHost`) point the iframe at a `data:` placeholder and hand the renderer a mock bridge, so `sandbox-csp.ts` is imported by exactly two things in the tree — its own test and `createAppBridgeFactory.ts`. Storybook stays Chromium-only; broadening it covers a much larger, differently-shaped surface and is a separate decision to be judged on its own cost.
|
||||
- **An unrecognized `SMOKE_BROWSER` is an error, never a fallback.** Falling back to Chromium would report a green Chromium run under a job labelled `webkit` — coverage claimed but not run, which is worse than none.
|
||||
- **A launch failure names the engine that failed** and its `npx playwright install --with-deps <engine>`. Naming `chromium` while WebKit was the missing one sends the reader to install a browser they already have.
|
||||
- **CI runs Chromium in the `build` job's `npm run smoke` and every other gated engine in a `Sandbox smokes (<engine>)` matrix job** — today that matrix is `[firefox]` alone (see the WebKit bullet below). It sets `fail-fast: false` so that once it holds more than one engine, a failure in one cannot hide a failure in another. Chromium is deliberately _not_ in that matrix — `build`'s `npm run smoke` is exactly the local `npm run ci` path and already covers it, and matrixing `build` itself would triple validate, the coverage gate, two verify gates and Storybook to gain three browser smokes. Both `publish` jobs `needs` the matrix, so a release cannot ship past a non-Chromium failure.
|
||||
- **CI is Chromium-only, deliberately, and a Firefox job was trialled and removed.** The job was cheap — ~2 minutes, parallel with the 15-minute `build` job, so zero added wall-clock — and it still did not earn its place: across a dozen runs Firefox never once disagreed with Chromium, so it spent runner time and carried a real flake surface (`playwright install --with-deps` runs `apt-get update`, which fails whenever a third-party repo in the runner image breaks) to re-confirm a result already in hand. **Don't re-add it on the argument that cross-engine coverage is good in principle** — that argument was already accepted, and it is what the on-demand path serves. Re-add it on evidence: an actual cross-engine regression that a gate would have caught. It is a one-job diff when that happens.
|
||||
- **`pack:verify` stays pinned to Chromium**, passing it explicitly rather than reading `SMOKE_BROWSER`. It is a _packaging_ check; the engine question belongs where the sandbox is under test, and pinning it also means it can't be pointed at an engine its npm script never installed.
|
||||
- **WebKit is deliberately out of the CI matrix, and `continue-on-error` is not the answer.** The two App smokes fail under it for reasons nobody has identified — the failure stage even varies between runs — and the investigation was dropped as not worth the effort once it became clear it does not reproduce in real Safari, and that an isolated repro of the suspected mechanism did not reproduce it under Playwright's WebKit either. Adding `webkit` to the matrix is a one-word diff if someone ever does chase it down; until then an engine is either green in CI or absent from it, because a job allowed to fail reports coverage nobody is held to.
|
||||
- **WebKit fails the two App smokes**, for reasons nobody has identified — the failure stage even varies between runs. The investigation was dropped as not worth the effort once it became clear it does not reproduce in real Safari, and that an isolated repro of the suspected mechanism did not reproduce it under Playwright's WebKit either. Treat a WebKit failure as unexplained rather than as a defect until someone has actually looked.
|
||||
- ⚠️ **A red WebKit run is not a Safari indictment, for the same reason a green one is not a Safari guarantee.** That failure was first written up as a Safari bug on the strength of a Playwright-WebKit run alone; a manual check in Safari did not reproduce it, and neither did an isolated repro of the mechanism it was blamed on. The caveat below cuts both ways, and only one direction of it was applied. **Reproduce in the real browser before claiming user impact in one** — the divergence between Playwright's WebKit build and Safari is largest in exactly the layer (networking) where that bug lives.
|
||||
- ⚠️ **Playwright's WebKit is a WebKit build, not Safari.** Close enough to catch engine-level CSP and iframe divergence; not close enough to certify Safari. Don't write, in a doc or a PR description, that a green run means Safari works.
|
||||
|
||||
|
|
|
|||
|
|
@ -479,9 +479,11 @@ SMOKE_BROWSER=webkit npm run smoke:web:app # one smoke, one engine
|
|||
SMOKE_BROWSER=firefox npm run smoke:web:engine # all three smokes, one engine
|
||||
```
|
||||
|
||||
Unset, the engine is `chromium` — so `npm run ci` and `npm run smoke` are unchanged. CI runs Chromium inside the `build` job's `npm run smoke`, and every other gated engine in a separate `Sandbox smokes (<engine>)` matrix job — today that matrix holds **Firefox alone**; see the paragraph below for where WebKit stands. Splitting it that way, rather than running the whole `build` job once per engine, is deliberate: validate, the coverage gate, the two verify gates and Storybook are all engine-independent. An unrecognized value is an error, not a fallback: a silent fallback would report a green Chromium run under a job labelled `webkit`. A missing browser binary fails naming the engine and its `npx playwright install --with-deps <engine>`.
|
||||
Unset, the engine is `chromium` — so `npm run ci` and `npm run smoke` are unchanged, and **CI runs Chromium only**. The other engines are a tool you reach for, not a gate: running the sandbox smokes under Firefox before touching the MCP Apps sandbox, the CSP builder, or the proxy page is cheap and worth doing, but nothing runs it for you. An unrecognized value is an error, not a fallback: a silent fallback would report a green Chromium run for a command that asked for `webkit`. A missing browser binary fails naming the engine and its `npx playwright install --with-deps <engine>`.
|
||||
|
||||
**CI gates Chromium and Firefox. WebKit is supported by the tooling but is not gated**, because the two App smokes fail under it for reasons nobody has identified. Two things are known: it does **not** reproduce in real Safari (an MCP App opens there normally), and an isolated repro of the mechanism it was first blamed on did not reproduce it under Playwright's WebKit either. So it reads as a property of that particular build rather than a bug users hit, and chasing it further was judged not worth the effort. Run WebKit locally if you want the extra signal; treat a failure there as unexplained rather than as a defect until someone has looked. An engine is either green in CI or absent from it — a `continue-on-error` job would report coverage nobody is held to.
|
||||
**Firefox passes all three smokes. WebKit fails the two App smokes**, for reasons nobody has identified. Two things are known: it does **not** reproduce in real Safari (an MCP App opens there normally), and an isolated repro of the mechanism it was first blamed on did not reproduce it under Playwright's WebKit either. So it reads as a property of that particular build rather than a bug users hit, and chasing it further was judged not worth the effort — treat a WebKit failure as unexplained rather than as a defect until someone has looked.
|
||||
|
||||
**Why none of this is gated in CI.** A Firefox job was trialled and was cheap — about two minutes, running in parallel with the 15-minute `build` job, so no added wall-clock. It was dropped anyway, on the honest count: across a dozen runs it never once disagreed with Chromium, so it was paying real runner time and a real flake surface (`playwright install --with-deps` runs `apt-get update`, which fails whenever a third-party repo in the runner image breaks) to re-confirm a result we already had. Making it available on demand keeps the value — a cross-engine check right when you are changing engine-sensitive code — without a permanent tax on every push. If a cross-engine regression ever does turn up, that is the evidence for putting the job back; it is a one-job diff.
|
||||
|
||||
**Why these smokes specifically.** Most of the web client's behavior is React and Mantine, where a second engine buys little. The MCP Apps sandbox is the exception — it is built out of the primitives that genuinely diverge between engines: a CSP `<meta>` injected as the first `<head>` child of a `srcdoc` document, a nested sandboxed iframe, a `Permissions-Policy` `allow` attribute, and `postMessage` origin discipline across those two frames. Nothing else covers that: `sandbox-csp.test.ts` asserts which policy _string_ is built, which passes identically on an engine that ignores `<meta>` CSP entirely, and no Storybook story reaches the sandbox at all (all three App stories point the iframe at a `data:` placeholder and hand the renderer a mock bridge). Storybook itself remains Chromium-only — broadening it covers a much larger and differently-shaped surface, and is a separate decision.
|
||||
|
||||
|
|
|
|||
|
|
@ -102,13 +102,14 @@ export const BROWSER_ENV_VAR = "SMOKE_BROWSER";
|
|||
* which is worse than no coverage — it claims coverage that never ran.
|
||||
*
|
||||
* **Only an ABSENT variable selects the default. A present-but-empty one is an
|
||||
* error**, because that is what an unresolved CI expression looks like: GitHub
|
||||
* Actions renders `SMOKE_BROWSER: ${{ matrix.browsr }}` (or any undefined key)
|
||||
* as the empty string rather than omitting the variable. Treating that as "not
|
||||
* set" would run Chromium inside a job named for another engine — precisely the
|
||||
* false-coverage failure this resolver exists to prevent, arriving through a
|
||||
* typo instead of a bad value. Unsetting the variable is how you ask for the
|
||||
* default; setting it to nothing is not.
|
||||
* error**, because empty is what a variable set from something that did not
|
||||
* resolve looks like — `SMOKE_BROWSER="$ENGINE"` with `ENGINE` unset, or a CI
|
||||
* expression naming a key that does not exist (GitHub Actions renders those as
|
||||
* the empty string rather than omitting the variable). Treating that as "not
|
||||
* set" would silently run Chromium for a caller who asked for something else,
|
||||
* which is the same false-coverage failure the unrecognized-value branch exists
|
||||
* to prevent, arriving through a typo instead of a bad value. Unsetting the
|
||||
* variable is how you ask for the default; setting it to nothing is not.
|
||||
*/
|
||||
export function resolveBrowserName(env = process.env) {
|
||||
const raw = env[BROWSER_ENV_VAR];
|
||||
|
|
@ -116,8 +117,8 @@ export function resolveBrowserName(env = process.env) {
|
|||
const name = raw.trim().toLowerCase();
|
||||
if (name === "") {
|
||||
throw new Error(
|
||||
`${BROWSER_ENV_VAR} is set but empty — an unresolved CI expression ` +
|
||||
`(a misspelled \`matrix\` key renders as "") looks exactly like this. ` +
|
||||
`${BROWSER_ENV_VAR} is set but empty — a variable set from something ` +
|
||||
`that did not resolve looks exactly like this. ` +
|
||||
`Unset it to use ${DEFAULT_BROWSER}, or set one of ${SUPPORTED_BROWSERS.join(", ")}.`,
|
||||
);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,8 +5,8 @@
|
|||
* per process and then spends its whole run inside the happy path, so the branch
|
||||
* that matters most — an unrecognized `SMOKE_BROWSER` — is dead code from their
|
||||
* point of view. It is also the branch whose failure is silent rather than loud:
|
||||
* a fallback to Chromium there would report a green run under a job labelled
|
||||
* "webkit", claiming coverage that never ran.
|
||||
* a fallback to Chromium there would report a green run to a caller who asked
|
||||
* for another engine — coverage claimed that never ran.
|
||||
*
|
||||
* `loadBrowser`'s failure branches are covered here too, through its injectable
|
||||
* `loadPlaywright`. The smokes cannot reach them by construction —
|
||||
|
|
@ -36,20 +36,20 @@ describe("resolveBrowserName", () => {
|
|||
});
|
||||
|
||||
it("rejects a present-but-empty value instead of defaulting", () => {
|
||||
// This is the case worth being strict about. A CI expression that resolves
|
||||
// to nothing (`SMOKE_BROWSER: ${{ matrix.browsr }}`, an undefined key) sets
|
||||
// the variable to "" rather than removing it — so defaulting here would run
|
||||
// Chromium inside a job named for another engine, which is the very
|
||||
// false-coverage outcome the unrecognized-value branch exists to stop. Only
|
||||
// an ABSENT variable may select the default.
|
||||
// This is the case worth being strict about. `SMOKE_BROWSER="$ENGINE"` with
|
||||
// ENGINE unset — or a CI expression naming a key that does not exist — sets
|
||||
// the variable to "" rather than removing it, so defaulting here would
|
||||
// silently run Chromium for a caller who asked for something else. That is
|
||||
// the same false-coverage outcome the unrecognized-value branch exists to
|
||||
// stop. Only an ABSENT variable may select the default.
|
||||
for (const raw of ["", " ", "\t\n"]) {
|
||||
assert.throws(
|
||||
() => resolveBrowserName({ [BROWSER_ENV_VAR]: raw }),
|
||||
(err) => {
|
||||
assert.match(err.message, /set but empty/);
|
||||
// Names the likely cause and the actual remedy — "unset it", which is
|
||||
// different from "set it to nothing" and is the whole point here.
|
||||
assert.match(err.message, /matrix/);
|
||||
// Names the actual remedy — "unset it", which is different from
|
||||
// "set it to nothing" and is the whole point here.
|
||||
assert.match(err.message, /did not resolve/);
|
||||
assert.match(err.message, /Unset it/);
|
||||
return true;
|
||||
},
|
||||
|
|
|
|||
|
|
@ -567,7 +567,7 @@ async function verifyAppRender(baseUrl, token, whenWebServerExits) {
|
|||
label: LABEL,
|
||||
});
|
||||
// Chromium explicitly, not `resolveBrowserName()`: this is a *packaging*
|
||||
// check, and the engine matrix (#2086) belongs to the smokes, where the
|
||||
// check, and the engine question (#2086) belongs to the smokes, where the
|
||||
// sandbox surface is what is under test. Pinning it also means `pack:verify`
|
||||
// cannot be pointed at an engine its npm script never installed.
|
||||
browser = await loadBrowser(repoRoot, "chromium");
|
||||
|
|
|
|||
|
|
@ -44,14 +44,15 @@
|
|||
* ── Which engine ────────────────────────────────────────────────────────────
|
||||
*
|
||||
* `SMOKE_BROWSER` picks the engine (`chromium` — the default — `firefox`, or
|
||||
* `webkit`). CI gates **Chromium** (in the `build` job's `npm run smoke`) and
|
||||
* **Firefox** (in the `Sandbox smokes` matrix job); WebKit runs here but is not
|
||||
* gated: this smoke fails under Playwright's WebKit for reasons nobody has
|
||||
* identified, and nobody is currently investigating. It does NOT reproduce in
|
||||
* real Safari, and an isolated SSE repro did not reproduce it in Playwright's
|
||||
* WebKit either — so it is a property of that build, not a browser bug, and it
|
||||
* was judged not worth chasing. Run WebKit locally if you want the signal; do
|
||||
* not read a failure here as a defect until someone has looked (#2086).
|
||||
* `webkit`). **CI runs Chromium only.** The other engines are an on-demand tool,
|
||||
* not a gate: `SMOKE_BROWSER=firefox npm run smoke:web:engine` before touching
|
||||
* the sandbox is cheap and worth doing, but nothing runs it for you (#2086).
|
||||
*
|
||||
* Firefox passes. WebKit fails this smoke for reasons nobody has identified and
|
||||
* nobody is investigating: it does not reproduce in real Safari, and an isolated
|
||||
* SSE repro did not reproduce it under Playwright's WebKit either, so it reads
|
||||
* as a property of that build rather than a browser bug. Do not read a WebKit
|
||||
* failure here as a defect until someone has actually looked.
|
||||
*
|
||||
* This smoke is one of the two places the
|
||||
* MCP Apps sandbox is genuinely exercised, and the sandbox is built out of the
|
||||
|
|
@ -89,8 +90,8 @@ const repoRoot = resolve(import.meta.dirname, "..");
|
|||
|
||||
// Resolved before anything is started, so an unsupported SMOKE_BROWSER fails
|
||||
// immediately rather than after a web server and two MCP servers are up. Every
|
||||
// message this smoke prints carries the engine, so a matrix failure names which
|
||||
// one broke without the reader having to match it to a job title.
|
||||
// message this smoke prints carries the engine, so a failure names which one
|
||||
// broke rather than leaving the reader to remember what they invoked it with.
|
||||
let BROWSER;
|
||||
try {
|
||||
BROWSER = resolveBrowserName();
|
||||
|
|
|
|||
|
|
@ -46,10 +46,9 @@
|
|||
* Launching the browser (and resolving Playwright from clients/web, which has
|
||||
* its own gotcha — see `lib/headless-browser.mjs`) is delegated to that module,
|
||||
* which is also where `SMOKE_BROWSER` picks the engine: `chromium` (the
|
||||
* default), `firefox`, or `webkit` (#2086). CI gates Chromium (in the `build`
|
||||
* job's `npm run smoke`) and Firefox (in the `Sandbox smokes` matrix job);
|
||||
* WebKit runs but is not gated — note this smoke PASSES in WebKit, it is the two
|
||||
* App smokes that do not (see their headers).
|
||||
* default), `firefox`, or `webkit` (#2086). **CI runs Chromium only**; the other
|
||||
* engines are an on-demand tool rather than a gate. This smoke passes in all
|
||||
* three — it is the two App smokes that fail under WebKit (see their headers).
|
||||
*
|
||||
* The engine question here is narrower than in the App smokes — this asserts a clean
|
||||
* first paint, i.e. that the shipped bundle's syntax and API level are
|
||||
|
|
@ -73,8 +72,8 @@ import {
|
|||
const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
|
||||
// Resolved before the web server is started, so an unsupported SMOKE_BROWSER
|
||||
// fails immediately. Every message carries the engine, so a matrix failure names
|
||||
// which one broke.
|
||||
// fails immediately. Every message carries the engine, so a failure names which
|
||||
// one broke.
|
||||
let BROWSER;
|
||||
try {
|
||||
BROWSER = resolveBrowserName();
|
||||
|
|
|
|||
|
|
@ -25,14 +25,15 @@
|
|||
* proof to a PR); unset, it asserts only.
|
||||
*
|
||||
* `SMOKE_BROWSER` picks the engine (`chromium` — the default — `firefox`, or
|
||||
* `webkit`). CI gates **Chromium** (in the `build` job's `npm run smoke`) and
|
||||
* **Firefox** (in the `Sandbox smokes` matrix job); WebKit runs here but is not
|
||||
* gated: this smoke fails under Playwright's WebKit for reasons nobody has
|
||||
* identified, and nobody is currently investigating. It does NOT reproduce in
|
||||
* real Safari, and an isolated SSE repro did not reproduce it in Playwright's
|
||||
* WebKit either — so it is a property of that build, not a browser bug, and it
|
||||
* was judged not worth chasing. Run WebKit locally if you want the signal; do
|
||||
* not read a failure here as a defect until someone has looked (#2086).
|
||||
* `webkit`). **CI runs Chromium only.** The other engines are an on-demand tool,
|
||||
* not a gate: `SMOKE_BROWSER=firefox npm run smoke:web:engine` before touching
|
||||
* the sandbox is cheap and worth doing, but nothing runs it for you (#2086).
|
||||
*
|
||||
* Firefox passes. WebKit fails this smoke for reasons nobody has identified and
|
||||
* nobody is investigating: it does not reproduce in real Safari, and an isolated
|
||||
* SSE repro did not reproduce it under Playwright's WebKit either, so it reads
|
||||
* as a property of that build rather than a browser bug. Do not read a WebKit
|
||||
* failure here as a defect until someone has actually looked.
|
||||
*
|
||||
* Along with `smoke:web:app` this is one
|
||||
* of the two places the MCP Apps sandbox is actually loaded, and the two nested
|
||||
|
|
@ -65,7 +66,7 @@ const repoRoot = resolve(import.meta.dirname, "..");
|
|||
|
||||
// Resolved before anything is started, so an unsupported SMOKE_BROWSER fails
|
||||
// immediately rather than after a web server and two MCP servers are up. Every
|
||||
// message carries the engine, so a matrix failure names which one broke.
|
||||
// message carries the engine, so a failure names which one broke.
|
||||
let BROWSER;
|
||||
try {
|
||||
BROWSER = resolveBrowserName();
|
||||
|
|
|
|||
Loading…
Reference in a new issue