fix(smoke): assert the requested engine is the one launched; fix the remedy
Three findings, all from Copilot's suppressed block. 1. The tests never checked that loadBrowser launches the engine it was asked for. The failure-path stand-in rejects identically for every engine, so loadBrowser(root, "firefox") could have called chromium.launch() and still produced a correctly Firefox-labelled error. The smokes could not catch it either: the Firefox gate runs after Chromium is already installed, so a mis-dispatched launch would SUCCEED and report Firefox coverage that never happened. Added a success-path test with distinct per-engine spies asserting the requested engine is invoked, that nothing else is, and that its handle is what comes back. Mutation-verified: hard-coding playwright.chromium fails it. 2. The launch-failure message told the reader to run `npx playwright install --with-deps <engine>`, which is wrong now that the npm scripts run from the repo root rather than clients/web -- I introduced that when I removed the `cd`. Playwright is pinned in clients/web, so a bare root-level npx can fetch a different version and install a browser revision the pinned one still cannot launch. It now points at `npm run smoke:web:<engine>` first, and scopes the raw command to clients/web with the reason. That message named smoke:web:webkit, which did not exist -- only the chromium and firefox scripts did. Added it, and a test asserting every supported engine has one, since a remedy nobody can run is worse than no remedy. Mutation-verified. 3. AGENTS.md said npm run ci "mirrors" the workflow and then, two sentences later, that it is "a superset, not a mirror". It now leads with strict superset, which is the topology this PR is documenting. The README and AGENTS.md remediation wording is corrected the same way as the runtime message, so a reader gets the same advice wherever they find it. Addresses Copilot review 5030668906 on #2133. Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
1cebe71761
commit
93bc7df360
3 changed files with 73 additions and 7 deletions
|
|
@ -187,8 +187,11 @@ export async function loadBrowser(
|
|||
return await playwright[browserName].launch({ headless: true });
|
||||
} catch (err) {
|
||||
throw new Error(
|
||||
`${browserName} failed to launch — run \`npx playwright install --with-deps ${browserName}\`, ` +
|
||||
`which fetches the browser and (on a bare Linux box) its system libraries ` +
|
||||
`${browserName} failed to launch — run \`npm run smoke:web:${browserName}\`, ` +
|
||||
`or from clients/web run \`npx playwright install --with-deps ${browserName}\` ` +
|
||||
`if you also need its system libraries. Playwright is pinned in ` +
|
||||
`clients/web, so running that from the repo root can fetch a different ` +
|
||||
`version and install a browser revision this one still cannot launch ` +
|
||||
`(${err instanceof Error ? err.message : String(err)})`,
|
||||
);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -16,7 +16,10 @@
|
|||
* naming *that engine* and its own `playwright install` command, which is
|
||||
* exactly the kind of string that rots into naming the wrong one.
|
||||
*
|
||||
* Only the successful launch is left to the smokes, since it needs a real binary.
|
||||
* The successful path is covered here too, with distinct per-engine spies — the
|
||||
* failure-path stand-in rejects identically for every engine, so it cannot tell
|
||||
* a correct dispatch from a mis-dispatch that happens to print the right label.
|
||||
* Only a launch against a REAL binary is left to the smokes.
|
||||
*/
|
||||
|
||||
import assert from "node:assert/strict";
|
||||
|
|
@ -108,6 +111,44 @@ describe("loadBrowser", () => {
|
|||
assert.equal(loaded, false);
|
||||
});
|
||||
|
||||
it("launches the engine that was asked for, and returns it", async () => {
|
||||
// The load-bearing assertion, and the one the failure-path tests below
|
||||
// CANNOT make (Copilot, #2133): their stand-in rejects identically for every
|
||||
// engine, so `loadBrowser(root, "firefox")` could call `chromium.launch()`
|
||||
// and still produce a correctly Firefox-labelled error. The smokes cannot
|
||||
// catch that either — the Firefox gate runs after Chromium is already
|
||||
// installed, so a mis-dispatched launch would succeed and report Firefox
|
||||
// coverage that never happened. Only distinct per-engine spies pin it.
|
||||
const launched = [];
|
||||
const spyPlaywright = () =>
|
||||
Object.fromEntries(
|
||||
SUPPORTED_BROWSERS.map((name) => [
|
||||
name,
|
||||
{
|
||||
launch: async (options) => {
|
||||
launched.push({ name, options });
|
||||
return { engine: name };
|
||||
},
|
||||
},
|
||||
]),
|
||||
);
|
||||
|
||||
for (const name of SUPPORTED_BROWSERS) {
|
||||
launched.length = 0;
|
||||
const browser = await loadBrowser("/repo", name, {
|
||||
loadPlaywright: spyPlaywright,
|
||||
});
|
||||
assert.deepEqual(
|
||||
launched.map((l) => l.name),
|
||||
[name],
|
||||
`loadBrowser(…, "${name}") must launch ${name} and nothing else`,
|
||||
);
|
||||
// The returned handle is that engine's, not some other engine's.
|
||||
assert.deepEqual(browser, { engine: name });
|
||||
assert.deepEqual(launched[0].options, { headless: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("names the engine that failed, and its own install command", async () => {
|
||||
// The whole point of #2086's acceptance criterion: a reader whose WebKit
|
||||
// binary is missing must not be sent to install chromium.
|
||||
|
|
@ -119,10 +160,16 @@ describe("loadBrowser", () => {
|
|||
}),
|
||||
(err) => {
|
||||
assert.match(err.message, new RegExp(`^${name} failed to launch`));
|
||||
// Names a remedy that works from where the caller is: the npm
|
||||
// script, and the raw command scoped to clients/web (Playwright is
|
||||
// pinned there; a bare root-level `npx playwright` can fetch a
|
||||
// different version and install a revision this cannot launch).
|
||||
assert.match(err.message, new RegExp(`npm run smoke:web:${name}`));
|
||||
assert.match(err.message, /from clients\/web/);
|
||||
assert.match(
|
||||
err.message,
|
||||
new RegExp(
|
||||
`npx playwright install --with-deps ${name}\\\`(?![\\s\\S]*--with-deps (?!${name}))`,
|
||||
`--with-deps ${name}\\\`(?![\\s\\S]*--with-deps (?!${name}))`,
|
||||
),
|
||||
);
|
||||
// The underlying cause survives, so the reader can tell a missing
|
||||
|
|
|
|||
|
|
@ -16,6 +16,7 @@ import { existsSync, readFileSync } from "node:fs";
|
|||
import { join } from "node:path";
|
||||
import { describe, it } from "node:test";
|
||||
import { ENGINE_SMOKES } from "./run-engine-smokes.mjs";
|
||||
import { SUPPORTED_BROWSERS } from "./lib/headless-browser.mjs";
|
||||
|
||||
const scriptDir = import.meta.dirname;
|
||||
const scripts = JSON.parse(
|
||||
|
|
@ -75,6 +76,20 @@ describe("every engine tier consumes ENGINE_SMOKES", () => {
|
|||
assert.match(scripts.ci, /smoke:web:firefox/);
|
||||
});
|
||||
|
||||
it("every supported engine has a `smoke:web:<engine>` script", () => {
|
||||
// loadBrowser's launch-failure message tells the reader to run
|
||||
// `npm run smoke:web:<engine>`. If a supported engine has no such script,
|
||||
// that remedy is unrunnable — which is how the message shipped naming
|
||||
// `smoke:web:webkit` before it existed (Copilot, #2133).
|
||||
for (const name of SUPPORTED_BROWSERS) {
|
||||
assert.match(
|
||||
scripts[`smoke:web:${name}`] ?? "",
|
||||
new RegExp(`run-engine-smokes\\.mjs ${name}$`),
|
||||
`npm run smoke:web:${name} is promised by the launch-failure message`,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it("each gated tier names its engine explicitly rather than reading the env", () => {
|
||||
// An ambient SMOKE_BROWSER must not be able to redirect a gate: without the
|
||||
// literal engine, `SMOKE_BROWSER=firefox npm run ci` would run Firefox twice
|
||||
|
|
|
|||
Loading…
Reference in a new issue