fix(smoke): assert the requested engine is the one launched; fix the remedy

Three findings, all from Copilot's suppressed block.

1. The tests never checked that loadBrowser launches the engine it was
   asked for. The failure-path stand-in rejects identically for every
   engine, so loadBrowser(root, "firefox") could have called
   chromium.launch() and still produced a correctly Firefox-labelled
   error. The smokes could not catch it either: the Firefox gate runs
   after Chromium is already installed, so a mis-dispatched launch would
   SUCCEED and report Firefox coverage that never happened.

   Added a success-path test with distinct per-engine spies asserting the
   requested engine is invoked, that nothing else is, and that its handle
   is what comes back. Mutation-verified: hard-coding playwright.chromium
   fails it.

2. The launch-failure message told the reader to run
   `npx playwright install --with-deps <engine>`, which is wrong now that
   the npm scripts run from the repo root rather than clients/web -- I
   introduced that when I removed the `cd`. Playwright is pinned in
   clients/web, so a bare root-level npx can fetch a different version and
   install a browser revision the pinned one still cannot launch. It now
   points at `npm run smoke:web:<engine>` first, and scopes the raw
   command to clients/web with the reason.

   That message named smoke:web:webkit, which did not exist -- only the
   chromium and firefox scripts did. Added it, and a test asserting every
   supported engine has one, since a remedy nobody can run is worse than
   no remedy. Mutation-verified.

3. AGENTS.md said npm run ci "mirrors" the workflow and then, two
   sentences later, that it is "a superset, not a mirror". It now leads
   with strict superset, which is the topology this PR is documenting.

The README and AGENTS.md remediation wording is corrected the same way as
the runtime message, so a reader gets the same advice wherever they find
it.

Addresses Copilot review 5030668906 on #2133.

Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-08-26 09:33:21 -04:00
commit 93bc7df360

View file

@ -479,7 +479,7 @@ SMOKE_BROWSER=webkit npm run smoke:web:app # one smoke, one engine
SMOKE_BROWSER=firefox npm run smoke:web:engine # all three smokes, one engine
```
Unset, the engine is `chromium`, so `npm run smoke` is unchanged. **`npm run ci` — the mandatory pre-push gate — additionally runs all three smokes under Firefox** via `smoke:web:firefox`; **GitHub CI does not.** An unrecognized `SMOKE_BROWSER` is an error, not a fallback: a silent fallback would report a green Chromium run for a command that asked for `webkit`. A missing browser binary fails naming the engine and its `npx playwright install --with-deps <engine>`.
Unset, the engine is `chromium`, so `npm run smoke` is unchanged. **`npm run ci` — the mandatory pre-push gate — additionally runs all three smokes under Firefox** via `smoke:web:firefox`; **GitHub CI does not.** An unrecognized `SMOKE_BROWSER` is an error, not a fallback: a silent fallback would report a green Chromium run for a command that asked for `webkit`. A missing browser binary fails naming the engine and a remedy that works from where you are — `npm run smoke:web:<engine>`, or `npx playwright install --with-deps <engine>` run from `clients/web`, where Playwright is pinned.
**Firefox passes all three smokes. WebKit fails the two App smokes**, for reasons nobody has identified. Two things are known: it does **not** reproduce in real Safari (an MCP App opens there normally), and an isolated repro of the mechanism it was first blamed on did not reproduce it under Playwright's WebKit either. So it reads as a property of that particular build rather than a bug users hit, and chasing it further was judged not worth the effort — treat a WebKit failure as unexplained rather than as a defect until someone has looked.