fix(smoke): assert the requested engine is the one launched; fix the remedy

Three findings, all from Copilot's suppressed block.

1. The tests never checked that loadBrowser launches the engine it was
   asked for. The failure-path stand-in rejects identically for every
   engine, so loadBrowser(root, "firefox") could have called
   chromium.launch() and still produced a correctly Firefox-labelled
   error. The smokes could not catch it either: the Firefox gate runs
   after Chromium is already installed, so a mis-dispatched launch would
   SUCCEED and report Firefox coverage that never happened.

   Added a success-path test with distinct per-engine spies asserting the
   requested engine is invoked, that nothing else is, and that its handle
   is what comes back. Mutation-verified: hard-coding playwright.chromium
   fails it.

2. The launch-failure message told the reader to run
   `npx playwright install --with-deps <engine>`, which is wrong now that
   the npm scripts run from the repo root rather than clients/web -- I
   introduced that when I removed the `cd`. Playwright is pinned in
   clients/web, so a bare root-level npx can fetch a different version and
   install a browser revision the pinned one still cannot launch. It now
   points at `npm run smoke:web:<engine>` first, and scopes the raw
   command to clients/web with the reason.

   That message named smoke:web:webkit, which did not exist -- only the
   chromium and firefox scripts did. Added it, and a test asserting every
   supported engine has one, since a remedy nobody can run is worse than
   no remedy. Mutation-verified.

3. AGENTS.md said npm run ci "mirrors" the workflow and then, two
   sentences later, that it is "a superset, not a mirror". It now leads
   with strict superset, which is the topology this PR is documenting.

The README and AGENTS.md remediation wording is corrected the same way as
the runtime message, so a reader gets the same advice wherever they find
it.

Addresses Copilot review 5030668906 on #2133.

Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-08-26 09:33:21 -04:00
commit 93bc7df360
3 changed files with 73 additions and 7 deletions

View file

@ -187,8 +187,11 @@ export async function loadBrowser(
return await playwright[browserName].launch({ headless: true });
} catch (err) {
throw new Error(
`${browserName} failed to launch — run \`npx playwright install --with-deps ${browserName}\`, ` +
`which fetches the browser and (on a bare Linux box) its system libraries ` +
`${browserName} failed to launch — run \`npm run smoke:web:${browserName}\`, ` +
`or from clients/web run \`npx playwright install --with-deps ${browserName}\` ` +
`if you also need its system libraries. Playwright is pinned in ` +
`clients/web, so running that from the repo root can fetch a different ` +
`version and install a browser revision this one still cannot launch ` +
`(${err instanceof Error ? err.message : String(err)})`,
);
}

View file

@ -16,7 +16,10 @@
* naming *that engine* and its own `playwright install` command, which is
* exactly the kind of string that rots into naming the wrong one.
*
* Only the successful launch is left to the smokes, since it needs a real binary.
* The successful path is covered here too, with distinct per-engine spies — the
* failure-path stand-in rejects identically for every engine, so it cannot tell
* a correct dispatch from a mis-dispatch that happens to print the right label.
* Only a launch against a REAL binary is left to the smokes.
*/
import assert from "node:assert/strict";
@ -108,6 +111,44 @@ describe("loadBrowser", () => {
assert.equal(loaded, false);
});
it("launches the engine that was asked for, and returns it", async () => {
// The load-bearing assertion, and the one the failure-path tests below
// CANNOT make (Copilot, #2133): their stand-in rejects identically for every
// engine, so `loadBrowser(root, "firefox")` could call `chromium.launch()`
// and still produce a correctly Firefox-labelled error. The smokes cannot
// catch that either — the Firefox gate runs after Chromium is already
// installed, so a mis-dispatched launch would succeed and report Firefox
// coverage that never happened. Only distinct per-engine spies pin it.
const launched = [];
const spyPlaywright = () =>
Object.fromEntries(
SUPPORTED_BROWSERS.map((name) => [
name,
{
launch: async (options) => {
launched.push({ name, options });
return { engine: name };
},
},
]),
);
for (const name of SUPPORTED_BROWSERS) {
launched.length = 0;
const browser = await loadBrowser("/repo", name, {
loadPlaywright: spyPlaywright,
});
assert.deepEqual(
launched.map((l) => l.name),
[name],
`loadBrowser(…, "${name}") must launch ${name} and nothing else`,
);
// The returned handle is that engine's, not some other engine's.
assert.deepEqual(browser, { engine: name });
assert.deepEqual(launched[0].options, { headless: true });
}
});
it("names the engine that failed, and its own install command", async () => {
// The whole point of #2086's acceptance criterion: a reader whose WebKit
// binary is missing must not be sent to install chromium.
@ -119,10 +160,16 @@ describe("loadBrowser", () => {
}),
(err) => {
assert.match(err.message, new RegExp(`^${name} failed to launch`));
// Names a remedy that works from where the caller is: the npm
// script, and the raw command scoped to clients/web (Playwright is
// pinned there; a bare root-level `npx playwright` can fetch a
// different version and install a revision this cannot launch).
assert.match(err.message, new RegExp(`npm run smoke:web:${name}`));
assert.match(err.message, /from clients\/web/);
assert.match(
err.message,
new RegExp(
`npx playwright install --with-deps ${name}\\\`(?![\\s\\S]*--with-deps (?!${name}))`,
`--with-deps ${name}\\\`(?![\\s\\S]*--with-deps (?!${name}))`,
),
);
// The underlying cause survives, so the reader can tell a missing

View file

@ -16,6 +16,7 @@ import { existsSync, readFileSync } from "node:fs";
import { join } from "node:path";
import { describe, it } from "node:test";
import { ENGINE_SMOKES } from "./run-engine-smokes.mjs";
import { SUPPORTED_BROWSERS } from "./lib/headless-browser.mjs";
const scriptDir = import.meta.dirname;
const scripts = JSON.parse(
@ -75,6 +76,20 @@ describe("every engine tier consumes ENGINE_SMOKES", () => {
assert.match(scripts.ci, /smoke:web:firefox/);
});
it("every supported engine has a `smoke:web:<engine>` script", () => {
// loadBrowser's launch-failure message tells the reader to run
// `npm run smoke:web:<engine>`. If a supported engine has no such script,
// that remedy is unrunnable — which is how the message shipped naming
// `smoke:web:webkit` before it existed (Copilot, #2133).
for (const name of SUPPORTED_BROWSERS) {
assert.match(
scripts[`smoke:web:${name}`] ?? "",
new RegExp(`run-engine-smokes\\.mjs ${name}$`),
`npm run smoke:web:${name} is promised by the launch-failure message`,
);
}
});
it("each gated tier names its engine explicitly rather than reading the env", () => {
// An ambient SMOKE_BROWSER must not be able to redirect a gate: without the
// literal engine, `SMOKE_BROWSER=firefox npm run ci` would run Firefox twice