chore(scripts): walk the workflow schema's executable paths, and resolve aliases (#2146)
Copilot, fourth review — four findings, all real: - Matching pairs by KEY NAME anywhere in the tree scanned things that execute nothing. Workflow input ids are user-defined, so an input legitimately named `env` put its own description and default in front of the rules — failing the gate on documentation OF the gate. The walk is structural now: workflow `env`, job `env`/`with`, step `run`/`env`/`with`, and nothing else. - YAML aliases were read as scalars, which have no `.value` on an alias node — so `run: *command`, with the command anchored in a metadata scalar the walk never visits, invoked anything at all while the guard saw an empty string. Aliases are resolved, and reported at the line where they are USED. - `file` was accepted for diagnostics and then not forwarded, so a malformed real workflow was reported as `<workflow>`. - One negative fixture was a YAML comment, which the parser strips — so it asserted nothing. Inside a `run:` block the comment text really does survive into the scalar, so the "a comment invokes nothing" rule is now applied there, where it actually has to be, and the fixture is a real block scalar. The table's fixtures are schema-valid workflows now rather than bare `- run:` fragments, which a structural walk correctly cannot see. Signed-off-by: Cliff Hall <cliff@futurescale.com> Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
b8d3cfdc3d
commit
8b02329703
1 changed files with 287 additions and 128 deletions
|
|
@ -1003,7 +1003,7 @@ Everything about the selection lives in **`scripts/lib/headless-browser.mjs`**
|
|||
- **An unrecognized `SMOKE_BROWSER` is an error, never a fallback.** Falling back to Chromium would report a green Chromium run under a job labelled `webkit` — coverage claimed but not run, which is worse than none.
|
||||
- **A launch failure names the engine that failed** and a remedy that works **from where the caller actually is**: `npm run smoke:web:<engine>`, or `npx playwright install --with-deps <engine>` run *from `clients/web`*. Naming `chromium` while WebKit was missing would send the reader to install a browser they already have — and naming a bare root-level `npx playwright` would send them to a version that is not the pinned one, which can install a browser revision the pinned Playwright still cannot launch.
|
||||
- **Firefox is gated locally, not in CI, and that asymmetry is deliberate.** A GitHub Actions job was trialled and removed. It was cheap — ~2 minutes, parallel with the 15-minute `build` job, so zero added wall-clock — but across a dozen runs Firefox never once disagreed with Chromium, so it spent runner minutes on every push, from every branch, carrying a real flake surface (`playwright install --with-deps` runs `apt-get update`, which fails whenever a third-party repo in the runner image breaks) to re-confirm a result already in hand. Moving it into `npm run local:gate` keeps the check where a human is about to push a change they can still reason about, and pays for it once rather than on every push. **Don't re-add the CI job on the argument that cross-engine coverage is good in principle** — that argument was accepted, and the pre-push gate is what serves it. Re-add it on evidence: a cross-engine regression that reached `v2/main` because someone skipped the gate.
|
||||
- **A guard enforces the split, so it cannot be undone by accident (#2146).** `scripts/lib/workflow-gate.mjs` is run by `npm run test:scripts` over every file under `.github/workflows/`, and fails the suite on: an invocation of a `local:*` script (the prefix *means* local-only, so a future sibling is covered without editing the guard); an invocation of a non-Chromium engine pass, derived from `SUPPORTED_BROWSERS` so a fourth engine extends the guard rather than opening a hole in it; an invocation of `smoke:web:engine`, whose engine comes from the environment and so cannot be read off the workflow at all; or `SMOKE_BROWSER` set to anything but a literal `chromium` — the back door that would redirect an otherwise innocent `npm run smoke`, and one that includes setting it to *nothing*, since present-and-empty is rejected rather than defaulted. A name built from a workflow **expression** counts as forbidden too (`smoke:web:${{ matrix.browser }}`, `local:${{ matrix.task }}`): a matrix the guard cannot read may well contain `firefox`, so an unreadable name is not waved through. Its limit is worth knowing — a fully opaque `npm run ${{ matrix.script }}` names no family at all and is not detectable. It deliberately does **not** forbid `npm run smoke`, `smoke:web:chromium`, or `smoke:tui`: those belong in CI and are there today, and `smoke:tui` self-skips under `process.env.CI` on its own. It **parses** the workflow (via the root-declared `yaml` package) and reads only the positions that execute something — `run:` scalars (inline and block) and the values of an `env:` or `with:` mapping — so ordinary metadata such as a step `name:` explaining the rule is not reported as an invocation of it (Copilot); comments are skipped for the same reason, and because a guard that made the workflow undocumentable would be traded for the wrong thing. That scoping is a stated limit rather than a hidden one: an execution vector outside those three positions is not seen.
|
||||
- **A guard enforces the split, so it cannot be undone by accident (#2146).** `scripts/lib/workflow-gate.mjs` is run by `npm run test:scripts` over every file under `.github/workflows/`, and fails the suite on: an invocation of a `local:*` script (the prefix *means* local-only, so a future sibling is covered without editing the guard); an invocation of a non-Chromium engine pass, derived from `SUPPORTED_BROWSERS` so a fourth engine extends the guard rather than opening a hole in it; an invocation of `smoke:web:engine`, whose engine comes from the environment and so cannot be read off the workflow at all; or `SMOKE_BROWSER` set to anything but a literal `chromium` — the back door that would redirect an otherwise innocent `npm run smoke`, and one that includes setting it to *nothing*, since present-and-empty is rejected rather than defaulted. A name built from a workflow **expression** counts as forbidden too (`smoke:web:${{ matrix.browser }}`, `local:${{ matrix.task }}`): a matrix the guard cannot read may well contain `firefox`, so an unreadable name is not waved through. Its limit is worth knowing — a fully opaque `npm run ${{ matrix.script }}` names no family at all and is not detectable. It deliberately does **not** forbid `npm run smoke`, `smoke:web:chromium`, or `smoke:tui`: those belong in CI and are there today, and `smoke:tui` self-skips under `process.env.CI` on its own. It **parses** the workflow (via the root-declared `yaml` package) and descends the schema's executable paths — workflow `env`, job `env`/`with`, step `run`/`env`/`with`, with YAML aliases resolved — rather than matching pairs by key name anywhere in the tree, which would scan a workflow input legitimately named `env` and fail the gate on documentation of the gate. So it reads only what executes — so ordinary metadata such as a step `name:` explaining the rule is not reported as an invocation of it (Copilot); comments are skipped for the same reason, and because a guard that made the workflow undocumentable would be traded for the wrong thing. That scoping is a stated limit rather than a hidden one: an execution vector outside those three positions is not seen.
|
||||
- **This makes `npm run local:gate` a strict superset of GitHub CI rather than a mirror of it.** That was already the direction (`smoke:tui` self-skips on CI and runs only locally); Firefox is the second such step. The invariant that still holds, and the one that matters, is the useful direction: **passing `npm run local:gate` locally means CI's gates will pass.**
|
||||
- **`pack:verify` stays pinned to Chromium**, passing it explicitly rather than reading `SMOKE_BROWSER`. It is a _packaging_ check; the engine question belongs where the sandbox is under test, and pinning it also means it can't be pointed at an engine its npm script never installed.
|
||||
- **WebKit fails the two App smokes**, for reasons nobody has identified — the failure stage even varies between runs. The investigation was dropped as not worth the effort once it became clear it does not reproduce in real Safari, and that an isolated repro of the suspected mechanism did not reproduce it under Playwright's WebKit either. Treat a WebKit failure as unexplained rather than as a defect until someone has actually looked.
|
||||
|
|
|
|||
Loading…
Reference in a new issue