fix: address Copilot review round 5
Both findings are the same shape as round 2's: a guard that documents an
invariant while checking one narrow expression of it.
`retry` was only rejected at the project level, so `it("x", { retry: 2 })` and a
`--retry` flag in an npm script both walked through a guard whose own header
said "no retry anywhere". Enforcement is extended rather than the claim
narrowed, because the decision is about the behavior, not about a config key —
all three places Vitest accepts one are now scanned, and the header names the
one that remains out of reach (a human typing --retry into their own shell),
rather than overclaiming.
The test-definition scan is deliberately positional: the retry must sit in an
options object following the test name, which is the only place Vitest reads it
from. A bare word match would fire on fixture data, on `client.retry`, on an
unrelated helper's options and on a mock — all of which exist here, and a guard
that cries wolf gets disabled. Both directions are tested. The first pattern
also missed `it.each([...])("name", { retry })` entirely, so every parameterized
suite was a blind spot; the table argument is now part of the pattern.
Config discovery named only the two `.ts` spellings this repo happens to use,
while Vitest loads twelve. A `clients/foo/vitest.config.mts` was therefore
invisible, and an invisible config yields no project to reject — the same hole
round 1 found in the hand-written CONFIG_ROOTS, one level down. Both discovery
and resolution now use the full set, ordered as Vitest orders it.
Both checks were observed firing before being trusted: a planted retry in a
launcher test and a planted vitest.config.mts each fail the guard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PsmUAZ6aLUAFqenjJKSSRp
Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall
parent
7b794de0b0
commit
f4d5b6a292
3 changed files with 187 additions and 11 deletions
|
|
@ -29,7 +29,7 @@ That is the readable half, and prose rots. The enforced half is `scripts/lib/wor
|
|||
|
||||
| Script | What it does |
|
||||
| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `npm run validate` | Runs the five durable guards first — `verify:format-coverage` (every tracked source file is format-gated), `verify:skills` (every `.claude/skills` manifest parses and declares its invocation mode; the **model-invoked** ones also carry eval cases — the name-only skills intentionally have none), `verify:typecheck-coverage` (every tracked TS file lands in a tsconfig project), `verify:dep-lockstep` (two tiers: no dependency reaching one `tsc` program from two installs skews across them, and — since #2226 — no **declared** package holds two different top-level versions across installs at all), `verify:test-timeouts` (every Vitest project resolves to a budget somebody chose, both web projects configure Testing Library's `asyncUtilTimeout`, and `retry` stays unset) — then `test:scripts` (the guards' own parser unit tests), then `validate:core` (the shared `core/` `format:check` + `lint` gate — including the `core/react/**` React-hooks block added by #2192, whose plugin is a **root** devDependency because a client-local copy is unreachable from the root: Node resolution walks up, not down), then per client: `format:check` + `lint` + **`typecheck`** (cli/tui/launcher; web typechecks via `tsc -b` inside its `build`) + `build` + fast unit tests. The quick inner-loop check. |
|
||||
| `npm run validate` | Runs the five durable guards first — `verify:format-coverage` (every tracked source file is format-gated), `verify:skills` (every `.claude/skills` manifest parses and declares its invocation mode; the **model-invoked** ones also carry eval cases — the name-only skills intentionally have none), `verify:typecheck-coverage` (every tracked TS file lands in a tsconfig project), `verify:dep-lockstep` (two tiers: no dependency reaching one `tsc` program from two installs skews across them, and — since #2226 — no **declared** package holds two different top-level versions across installs at all), `verify:test-timeouts` (every Vitest project resolves to a budget somebody chose, both web projects configure Testing Library's `asyncUtilTimeout`, and `retry` stays unset in every project, test and script) — then `test:scripts` (the guards' own parser unit tests), then `validate:core` (the shared `core/` `format:check` + `lint` gate — including the `core/react/**` React-hooks block added by #2192, whose plugin is a **root** devDependency because a client-local copy is unreachable from the root: Node resolution walks up, not down), then per client: `format:check` + `lint` + **`typecheck`** (cli/tui/launcher; web typechecks via `tsc -b` inside its `build`) + `build` + fast unit tests. The quick inner-loop check. |
|
||||
| `npm run coverage` | The **per-file ≥90% gate** (lines/statements/functions/branches) under v8 instrumentation, per client. CI-enforced. For web this also runs the integration project and covers the shared `core/` runtime (including `core/json` and `core/client`). |
|
||||
| `npm run smoke` | End-to-end smokes through the built launcher (`--help` dispatch + prod cli/tui/web — the TUI one runs under a **pseudoterminal** and asserts the Ink app is still running two seconds after its first frame, not merely that it painted one, [#2147](https://github.com/modelcontextprotocol/inspector/issues/2147)), plus three headless-browser smokes: a boot smoke that runs the prod web bundle and asserts a clean first render (no uncaught error — sync exception or unhandled rejection, how a Node built-in reaching the browser bundle manifests), and an **MCP Apps** smoke (`smoke:web:app`) that drives connect → open app → `data-app-status="ready"` against a composable App server, covering the sandbox proxy and UI-protocol bridge, and an **app-rendered elicitation** smoke (`smoke:web:elicit`) that drives one end to end — call the tool, answer inside the sandboxed app, see the app's `ElicitResult` reach the server — and then the same tool against a server that never advertised the capability, which must fall back to the native elicitation form. Those three take their engine from `SMOKE_BROWSER` — see [Supported browsers](#supported-browsers). A fourth, **`smoke:web:tabs`**, drives the core tabs against one server on one browser launch — Tools runs a tool and asserts its `structuredContent` section, Resources reads a resource with templates listed, Prompts fetches one ([#2148](https://github.com/modelcontextprotocol/inspector/issues/2148)) — asserting the `data-*` contract in [the web README](../clients/web/README.md#core-tab-automation-contract) rather than visible copy. It is **Chromium-only**: those tabs are ordinary React and Mantine, so unlike the sandbox they are not engine-sensitive. |
|
||||
| `npm run verify:build-gate` | Runs a real `vite build` with a Node built-in forced into the browser graph and asserts the build **fails** via the #1769 gate (which turns Vite's browser-externalization warning into a hard error). Guards against the warning phrasing drifting in a Vite bump and silently disabling the gate. Part of `npm run local:gate` and of GitHub CI. |
|
||||
|
|
@ -40,7 +40,7 @@ That is the readable half, and prose rots. The enforced half is `scripts/lib/wor
|
|||
| `npm run test:scripts` | Table-driven unit tests (`node --test`) for the guard's own pure parsers (`scripts/lib/npm-scripts.mjs`, `scripts/lib/tsc-program.mjs` + the exported helpers of `verify-typecheck-coverage.mjs` and `verify-dep-lockstep.mjs`), one case per rule they encode, plus two suites over shared `scripts/lib` helpers that no smoke can check itself: `resolve-node-bin.test.mjs` — the cross-platform bin resolver (#1939), pinned against the real `bin`/`exports` shapes of the packages the scripts actually spawn — and `announced-child.test.mjs` — the spawn/readiness ownership helper (#2000), which drives real `node -e` children to prove a child that never announces is still published to the caller before the timeout throws, and so is reachable by teardown rather than orphaned. Four more do the same: `mcp-app-flow.test.mjs` covers the shared MCP Apps flow (#2003) — the deep link's two CSRF gates and `appArgs` encoding, plus `driveAppFlow`'s failure branches against a stand-in page, all of which are dead code from the happy-path smokes' point of view and would otherwise surface only as opaque timeouts; `render-smoke.test.mjs` and `pty.test.mjs` cover the TUI boot harness ([#2147](https://github.com/modelcontextprotocol/inspector/issues/2147)) — the former driving real `node -e` stubs to prove that a child which paints the marker and *then* dies is a **failure**, which the old harness reported as OK and which no fixed TUI can reproduce; the latter pinning the three mutually-incompatible `script(1)` invocations, where a wrong guess stops the pseudoterminal from being allocated at all; and `ensure-test-servers.test.mjs` pins the [#2111](https://github.com/modelcontextprotocol/inspector/issues/2111) invariant — that `test-servers/build` is rebuilt **even when it already exists** — which no smoke can assert about itself, since one driving a stale fixture reports a product failure rather than a staleness one. `workflow-gate.test.mjs` is a different shape again ([#2146](https://github.com/modelcontextprotocol/inspector/issues/2146)): besides the table of parser cases it runs the parser over the repo's real `.github/workflows/**`, so a workflow that invokes the local-only gate or a non-Chromium engine pass fails here rather than in a CI run nobody expected to be red. Five more arrive with the skills (#2163). `skill-manifest.test.mjs` pins the frontmatter contract one case per rule — most importantly that a fence which is not a whole line is not a terminator, and that malformed YAML is an error rather than an empty description. `verify-skills.main.test.mjs` drives that guard's `main()` over fixture directories, the sibling guards' `*.main.test.mjs` pattern, so it cannot stop enforcing while the parser tests stay green. `skill-eval.test.mjs` covers the trigger eval's stream parsing and its run classification, which an eval run's happy path never reaches and which would otherwise report a plausible hit rate for a CLI that failed to run. `verify-skills-cli.test.mjs` covers which validator gets run — a local CLI only at the exact pin — and the orchestration around it, neither of which an ordinary run on a working machine exercises. `claude-cli.test.mjs` covers the one place that decides how the CLI is spawned — the Windows `.cmd` shim needs a shell, and its arguments then need quoting — with the platform **injected**, since a win32-only bug that no runner here executes is exactly the kind that ships. Runs in `validate` — and `verify:typecheck-coverage` guards *this* gate in turn (reachable from `validate`, non-empty test set, every test file matched by the `test:scripts` glob), since `node --test` silently skips a file its glob misses and still exits 0. |
|
||||
| `npm run verify:typecheck-coverage` | The typecheck-coverage analog of the above (#1791): for each Node client (auto-discovered from disk — enrolled via its `typecheck` script's projects, or for a `tsc -b` client like `clients/web` via its `tsconfig.json` `references`) it runs those projects with `tsc --listFilesOnly`, unions them, and **fails** listing any tracked `.ts`/`.tsx`/`.mts`/`.cts` under the client that lands in no project (so a new top-level config/helper can't silently go untypechecked). It also requires, deny-by-default, the first-party TS no client owns (`test-servers/src`, the root `vitest.shared.mts`, all of `core/`, and any new top-level location) to land in some client project's tsc pass — so a `core` `*.tsx` web's projects don't reach is caught too. Also asserts the gate is wired (each client's typecheck pass — its `typecheck` script, or web's `tsc -b` — is reachable from its `validate`, and the root chain runs each client's `validate`). Runs in `validate`. |
|
||||
| `npm run verify:dep-lockstep` | Guards the "one version per install-crossing dependency" invariant (#1896). v2 is not a workspace, so a client's test project compiles the shared first-party TypeScript — `core/`, `test-servers/src`, and the root-owned `vitest.shared.mts`, all of which resolve their dependencies from the **root** install — alongside the client's own sources, putting the same package in one `tsc` program twice. At the same version that's harmless; skewed, TypeScript must relate two structurally-distinct copies of every type, which for a recursive-generic surface is exponential (zod `4.3.6` vs `4.4.3` exhausted the 4GB tsc heap in `clients/web`). Derives its candidate set from **what actually enters each program** (#1965) — every client tsconfig project listed with `tsc --listFilesOnly` via the shared `scripts/lib/tsc-program.mjs`, each resolved `node_modules` file mapped to its owning install, keeping the packages that reach one program from two installs (a package whose declarations arrive only through another package's `.d.ts`, as `@modelcontextprotocol/sdk`'s do, is invisible to a scan of first-party imports). Prices each copy from the lockfile entry for the exact install path the program resolved, compares only the installs that met in one program, and **fails deny-by-default** on any disagreement not in the annotated `TOLERATED_SKEW` allowlist — empty today — with an allowlisted package tolerated only *within a major version*. A **second tier** (#2226) runs alongside it, asking the weaker but broader question the `AGENTS.md` rule actually states: does a package this repo *declares* anywhere resolve to two versions across our installs at all? Its candidate set is every name in any install's `dependencies`/`devDependencies`/`optionalDependencies` — unioned across the root and all four clients, so a copy declared by only one of them still counts — that **more than one install holds a top-level copy of** (17 packages today). Nested copies are excluded: one exists because some dependency asked for a different version, so it is that dependency's range to govern, not ours. Neither tier subsumes the other — the program tier sees a copy no manifest names (`@modelcontextprotocol/sdk`, arriving through another package's `.d.ts`), while the declared tier sees a **transitive** copy no program loads (cli's `@types/node`, hoisted via `@types/express` — the case that motivated it), two **clients** disagreeing with no root copy involved (`@types/react`, web against tui), and the peer shadows `eslint`/`typescript`/`vitest` that never enter a program. Same deny-by-default and same within-a-major rule, against its own `TOLERATED_DECLARED_SKEW` — also empty. Two limits: it reads **lockfiles**, so an uncommitted hand-installed copy is invisible, and it compares only declared names, so a purely transitive package no manifest names stays the first tier's business. Runs in `validate`.
|
||||
| `npm run verify:test-timeouts` | Guards the wall-clock budgets the test gates run under (#2323). The class it encodes against is a budget **nobody chose**: three of the six Vitest projects ran on Vitest's own `testTimeout: 5000` and five on its `hookTimeout`/`teardownTimeout: 10000`, sized for an idle machine rather than the one this team works on — three or four concurrent agent sessions in separate worktrees, each free to run a full `local:gate`, on eight logical cores. A correct, deterministic test cut off by such a budget fails a gate its diff did not break, which is the same "channel nobody trusts" failure [Lint has no warning tier](../AGENTS.md#lint-has-no-warning-tier) describes from the other direction; #2292, #1942 and #1742 were each an instance, found one site at a time. Deliberately **not** cited here: #2278 (a missing condition wait around a geometry read) and #2250 (a real race in a test's own timing) were fixed by making the test wait for the right thing, and presenting a race fix as evidence for a larger ceiling would argue against #1596. It asks **Vitest itself** to resolve each of the four configs and reads the number a test actually gets, rather than checking that a key is absent from a config block — which would pass just as happily on a config that had stopped being loaded. Three properties beyond the numbers: a **seventh project** with no row is an error rather than a silent skip, since an unstated budget is precisely the drift being prevented; **`retry` must stay unset**, because a retry turns a load-induced red into a silent green on the only pre-push gate here and would re-open #1596; and both web projects must `configure({ asyncUtilTimeout })`, the Testing Library half that no Vitest config can see — it governs 788 `waitFor`/`findBy*` sites and is the binding bound on any async assertion, being tighter than any per-test budget here. What is enforced there is that the value is **stated**, not that it is large: raising it to 5000 was tried and measured worse (three arms of three full unit runs — unmodified `v2/main` 3/3 green, the raise 1/4/0 files timing out, the raise reverted 3/3 green), because a wait that is *meant* to expire spends its whole budget on the happy path, so a 5x raise is a 5x cost on exactly those tests and starves the worker pool. The reasoning is recorded at the call site. Observed **red** against the pre-#2323 config before it was trusted. Runs in `validate`; its pure decision logic has a sibling `verify-test-timeouts.test.mjs` under `test:scripts`.
|
||||
| `npm run verify:test-timeouts` | Guards the wall-clock budgets the test gates run under (#2323). The class it encodes against is a budget **nobody chose**: three of the six Vitest projects ran on Vitest's own `testTimeout: 5000` and five on its `hookTimeout`/`teardownTimeout: 10000`, sized for an idle machine rather than the one this team works on — three or four concurrent agent sessions in separate worktrees, each free to run a full `local:gate`, on eight logical cores. A correct, deterministic test cut off by such a budget fails a gate its diff did not break, which is the same "channel nobody trusts" failure [Lint has no warning tier](../AGENTS.md#lint-has-no-warning-tier) describes from the other direction; #2292, #1942 and #1742 were each an instance, found one site at a time. Deliberately **not** cited here: #2278 (a missing condition wait around a geometry read) and #2250 (a real race in a test's own timing) were fixed by making the test wait for the right thing, and presenting a race fix as evidence for a larger ceiling would argue against #1596. It asks **Vitest itself** to resolve each of the four configs and reads the number a test actually gets, rather than checking that a key is absent from a config block — which would pass just as happily on a config that had stopped being loaded. Three properties beyond the numbers: a **seventh project** with no row is an error rather than a silent skip, since an unstated budget is precisely the drift being prevented; **`retry` must stay unset** — enforced in all three places Vitest accepts one, since the decision is about the behavior rather than about a config key: the resolved project, an individual `it`/`describe` options object (the parameterized `it.each` form included), and a `--retry` flag in an npm script. A human typing `--retry` in their own shell is the one thing no committed check can see, and the guard says so rather than overclaiming. A retry turns a load-induced red into a silent green on the only pre-push gate here and would re-open #1596; and both web projects must `configure({ asyncUtilTimeout })`, the Testing Library half that no Vitest config can see — it governs 788 `waitFor`/`findBy*` sites and is the binding bound on any async assertion, being tighter than any per-test budget here. What is enforced there is that the value is **stated**, not that it is large: raising it to 5000 was tried and measured worse (three arms of three full unit runs — unmodified `v2/main` 3/3 green, the raise 1/4/0 files timing out, the raise reverted 3/3 green), because a wait that is *meant* to expire spends its whole budget on the happy path, so a 5x raise is a 5x cost on exactly those tests and starves the worker pool. The reasoning is recorded at the call site. Observed **red** against the pre-#2323 config before it was trusted. Runs in `validate`; its pure decision logic has a sibling `verify-test-timeouts.test.mjs` under `test:scripts`.
|
||||
| `npm run local:gate` | **Mandatory pre-push command.** `validate` → `verify:skills:cli` → `coverage` → `verify:build-gate` → `verify:bundle-externals` → `smoke` → `smoke:web:firefox` → `local:storybook`. A strict superset of GitHub CI — see [Two tiers](#two-tiers-github-ci-and-the-local-gate). Named `local:` rather than `ci` on purpose (#2146); there is no `npm run ci` alias. |
|
||||
| `npm run pack:verify` | Publish smoke — see [Publishing](./publishing.md). |
|
||||
|
||||
|
|
|
|||
|
|
@ -31,7 +31,12 @@
|
|||
* 3. **`retry` must stay unset.** A retry converts a load-induced red into a
|
||||
* silent green on the only pre-push gate this repo has, and would re-open
|
||||
* #1596 by hiding a real race behind a second attempt. That was a deliberate
|
||||
* "do not raise" decision, so it is enforced rather than remembered.
|
||||
* "do not raise" decision, so it is enforced rather than remembered — and
|
||||
* enforced in all three places Vitest accepts one, since the decision is
|
||||
* about the behavior rather than about a config key: the resolved project,
|
||||
* an individual `it`/`describe` options object, and a `--retry` flag in an
|
||||
* npm script (Copilot). What stays outside its reach is a human typing
|
||||
* `--retry` into their own shell, which no committed check can see.
|
||||
*
|
||||
* It also asserts the Testing Library half, which no Vitest config can see:
|
||||
* `asyncUtilTimeout` governs every `waitFor` / `findBy*` in the web projects
|
||||
|
|
@ -46,6 +51,7 @@
|
|||
* both `asyncUtilTimeout` assertions unmet.
|
||||
*/
|
||||
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { existsSync, readdirSync, readFileSync } from "node:fs";
|
||||
import { dirname, join, relative, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
|
@ -139,6 +145,22 @@ export const EXPECTED_ASYNC_UTIL_TIMEOUT = 1_000;
|
|||
*/
|
||||
export const CLIENTS_DIR = "clients";
|
||||
|
||||
/**
|
||||
* Every filename Vitest will load a config from, most specific first.
|
||||
*
|
||||
* The full set, not the two spellings this repo happens to use: discovery is
|
||||
* only deny-by-default if it sees every config Vitest would (Copilot). A
|
||||
* `clients/foo/vitest.config.mts` that this list did not name would be
|
||||
* invisible, and an invisible config yields no project to reject — the same
|
||||
* hole as a hand-written `CONFIG_ROOTS`, one level down. Order matters at
|
||||
* resolution time, where the first match wins, and mirrors Vitest's own
|
||||
* preference for a `vitest.config.*` over a `vite.config.*`.
|
||||
*/
|
||||
export const VITEST_CONFIG_FILENAMES = Object.freeze([
|
||||
...["ts", "mts", "cts", "js", "mjs", "cjs"].map((e) => `vitest.config.${e}`),
|
||||
...["ts", "mts", "cts", "js", "mjs", "cjs"].map((e) => `vite.config.${e}`),
|
||||
]);
|
||||
|
||||
/**
|
||||
* Compare one resolved project against its row.
|
||||
*
|
||||
|
|
@ -340,6 +362,48 @@ export function checkSetupFilesWiring(
|
|||
];
|
||||
}
|
||||
|
||||
/**
|
||||
* A `retry` declared on an individual test or suite, or passed on a command
|
||||
* line — the two places a project-level check cannot see.
|
||||
*
|
||||
* The "no retry" decision is about the behavior, not about one config key, so a
|
||||
* guard that only reads `project.config.retry` leaves `it("…", { retry: 2 })`
|
||||
* and `--retry=2` in an npm script wide open (Copilot). Both are scanned here.
|
||||
*
|
||||
* The test-definition pattern deliberately requires the `retry` to sit in an
|
||||
* **options object that follows the test name**, which is the only position
|
||||
* Vitest reads it from. That keeps a `retry` field in fixture data, a variable
|
||||
* named `retry`, or a mocked API's option out of it — this repo has several, and
|
||||
* a bare `/\bretry\b/` scan would fail on all of them.
|
||||
*
|
||||
* @param {string} source
|
||||
* @returns {string[]} the matched declarations, empty when there are none
|
||||
*/
|
||||
export function findTestLevelRetries(source) {
|
||||
const code = stripComments(source);
|
||||
// The optional group between the chain and the name is `it.each`'s table —
|
||||
// `it.each([...])("name", …)` or its tagged-template form — which otherwise
|
||||
// pushes the name out of the position this pattern looks for, so every
|
||||
// parameterized suite would be a blind spot.
|
||||
const re =
|
||||
/\b(?:it|test|describe|suite|bench)(?:\.\w+)*(?:\s*(?:\([^()]*\)|`[^`]*`))?\s*\(\s*(?:"[^"]*"|'[^']*'|`[^`]*`)\s*,\s*\{[^}]*?\bretry\s*:\s*([^,}\s]+)/g;
|
||||
return [...code.matchAll(re)].map((m) => `retry: ${m[1]}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* A `retry` flag passed to vitest from an npm script.
|
||||
*
|
||||
* @param {Record<string, unknown>} scripts
|
||||
* @returns {string[]} `"<name>: <script>"` for each offender
|
||||
*/
|
||||
export function findScriptRetries(scripts) {
|
||||
return Object.entries(scripts ?? {})
|
||||
.filter(
|
||||
([, cmd]) => typeof cmd === "string" && /(^|\s)--retry(=|\s|$)/.test(cmd),
|
||||
)
|
||||
.map(([name, cmd]) => `${name}: ${String(cmd)}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Every Vitest config under `clients/`, as repo-relative directories.
|
||||
*
|
||||
|
|
@ -360,9 +424,7 @@ export function discoverConfigRoots(root = repoRoot) {
|
|||
.filter((e) => e.isDirectory())
|
||||
.map((e) => join(clients, e.name))
|
||||
.filter((dir) =>
|
||||
["vitest.config.ts", "vite.config.ts"].some((f) =>
|
||||
existsSync(join(dir, f)),
|
||||
),
|
||||
VITEST_CONFIG_FILENAMES.some((f) => existsSync(join(dir, f))),
|
||||
)
|
||||
.map((dir) => relative(root, dir).split("\\").join("/"))
|
||||
.sort();
|
||||
|
|
@ -400,14 +462,14 @@ export function checkConfigRootCoverage(discovered, configured = CONFIG_ROOTS) {
|
|||
/**
|
||||
* Resolve every project of one config, via Vitest's own resolver.
|
||||
*
|
||||
* @param {string} root absolute directory holding the `vitest.config.ts` or `vite.config.ts`
|
||||
* @param {string} root absolute directory holding one of `VITEST_CONFIG_FILENAMES`
|
||||
* @returns {Promise<{name: string, config: Record<string, unknown>}[]>}
|
||||
*/
|
||||
async function resolveProjects(root) {
|
||||
const { createVitest } = await import("vitest/node");
|
||||
const config = ["vitest.config.ts", "vite.config.ts"]
|
||||
.map((f) => join(root, f))
|
||||
.find((f) => existsSync(f));
|
||||
const config = VITEST_CONFIG_FILENAMES.map((f) => join(root, f)).find((f) =>
|
||||
existsSync(f),
|
||||
);
|
||||
if (!config) {
|
||||
throw new Error(`no vitest/vite config under ${root}`);
|
||||
}
|
||||
|
|
@ -482,6 +544,43 @@ async function main() {
|
|||
}
|
||||
}
|
||||
|
||||
// `retry`, in the two places a resolved project cannot show it. Tracked files
|
||||
// only — an untracked scratch test is not something this repo ships.
|
||||
const testFiles = execFileSync(
|
||||
"git",
|
||||
["ls-files", "*.test.ts", "*.test.tsx", "*.test.mts", "*.stories.tsx"],
|
||||
{ cwd: repoRoot, encoding: "utf-8" },
|
||||
)
|
||||
.split("\n")
|
||||
.filter(Boolean);
|
||||
for (const file of testFiles) {
|
||||
const found = findTestLevelRetries(
|
||||
readFileSync(resolve(repoRoot, file), "utf-8"),
|
||||
);
|
||||
for (const decl of found) {
|
||||
failures.push(
|
||||
`${file} declares \`${decl}\` on a test or suite — a retry turns a ` +
|
||||
`load-induced red into a silent green on the only pre-push gate here (#1596)`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const manifests = execFileSync(
|
||||
"git",
|
||||
["ls-files", "package.json", "*/package.json", "*/*/package.json"],
|
||||
{ cwd: repoRoot, encoding: "utf-8" },
|
||||
)
|
||||
.split("\n")
|
||||
.filter(Boolean);
|
||||
for (const file of manifests) {
|
||||
const { scripts } = JSON.parse(
|
||||
readFileSync(resolve(repoRoot, file), "utf-8"),
|
||||
);
|
||||
for (const offender of findScriptRetries(scripts)) {
|
||||
failures.push(`${file} passes --retry from a script — ${offender}`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const { file, from, project } of ASYNC_UTIL_SITES) {
|
||||
const abs = resolve(repoRoot, file);
|
||||
if (!existsSync(abs)) {
|
||||
|
|
@ -513,7 +612,9 @@ async function main() {
|
|||
|
||||
console.log(
|
||||
`verify:test-timeouts OK — ${checked} Vitest projects on stated budgets, ` +
|
||||
`no retry, ${ASYNC_UTIL_SITES.length} asyncUtilTimeout sites configured.`,
|
||||
`no retry in any project, test or script (${testFiles.length} test files, ` +
|
||||
`${manifests.length} manifests), ` +
|
||||
`${ASYNC_UTIL_SITES.length} asyncUtilTimeout sites configured.`,
|
||||
);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -27,8 +27,11 @@ import {
|
|||
checkProject,
|
||||
checkSetupFilesWiring,
|
||||
discoverConfigRoots,
|
||||
findScriptRetries,
|
||||
findTestLevelRetries,
|
||||
identifyProject,
|
||||
stripComments,
|
||||
VITEST_CONFIG_FILENAMES,
|
||||
} from "./verify-test-timeouts.mjs";
|
||||
|
||||
const ok = { ...EXPECTED_TIMEOUTS };
|
||||
|
|
@ -311,3 +314,75 @@ test("discovery finds exactly the configs this guard is set up to check", () =>
|
|||
);
|
||||
assert.deepEqual(checkConfigRootCoverage(discoverConfigRoots()), []);
|
||||
});
|
||||
|
||||
test("a retry on an individual test or suite is caught", () => {
|
||||
// The project-level check cannot see these, and the decision is about the
|
||||
// behavior rather than about one config key (Copilot).
|
||||
for (const src of [
|
||||
`it("x", { retry: 2 }, async () => {});`,
|
||||
`test("x", { retry: 1 }, () => {});`,
|
||||
`describe("x", { retry: 3 }, () => {});`,
|
||||
`it.each([1])("x %i", { retry: 2 }, () => {});`,
|
||||
'describe.each([[1, 2]])("x", { retry: 1 }, () => {});',
|
||||
`it('single quoted', { timeout: 100, retry: 2 }, () => {});`,
|
||||
]) {
|
||||
assert.equal(findTestLevelRetries(src).length, 1, src);
|
||||
}
|
||||
});
|
||||
|
||||
test("a retry that is not a test option is not a false positive", () => {
|
||||
// All four shapes exist in this repo. A bare /\bretry\b/ scan would fail on
|
||||
// every one of them, and a guard that cries wolf gets disabled.
|
||||
for (const src of [
|
||||
`const fixture = { retry: 2 };`,
|
||||
`expect(client.retry).toBe(2);`,
|
||||
`it("x", async () => { await withRetries({ retry: 2 }); });`,
|
||||
`vi.mock("x", () => ({ retry: 2 }));`,
|
||||
]) {
|
||||
assert.deepEqual(findTestLevelRetries(src), [], src);
|
||||
}
|
||||
});
|
||||
|
||||
test("a retry hidden in a comment is not reported", () => {
|
||||
assert.deepEqual(
|
||||
findTestLevelRetries(`// it("x", { retry: 2 }, () => {});`),
|
||||
[],
|
||||
);
|
||||
});
|
||||
|
||||
test("a --retry flag in an npm script is caught", () => {
|
||||
assert.deepEqual(findScriptRetries({ test: "vitest run --retry=2" }), [
|
||||
"test: vitest run --retry=2",
|
||||
]);
|
||||
assert.equal(findScriptRetries({ test: "vitest run --retry 2" }).length, 1);
|
||||
});
|
||||
|
||||
test("a script that merely contains the word retry is not a false positive", () => {
|
||||
assert.deepEqual(
|
||||
findScriptRetries({ test: "vitest run retry-helper.ts" }),
|
||||
[],
|
||||
);
|
||||
assert.deepEqual(findScriptRetries({ test: "vitest run --no-retry-x" }), []);
|
||||
assert.deepEqual(findScriptRetries({}), []);
|
||||
assert.deepEqual(findScriptRetries(undefined), []);
|
||||
});
|
||||
|
||||
test("discovery knows every filename Vitest loads a config from", () => {
|
||||
// Deny-by-default only holds if discovery sees every config Vitest would; a
|
||||
// `vitest.config.mts` this list did not name would be invisible (Copilot).
|
||||
for (const ext of ["ts", "mts", "cts", "js", "mjs", "cjs"]) {
|
||||
assert.ok(
|
||||
VITEST_CONFIG_FILENAMES.includes(`vitest.config.${ext}`),
|
||||
`vitest.config.${ext}`,
|
||||
);
|
||||
assert.ok(
|
||||
VITEST_CONFIG_FILENAMES.includes(`vite.config.${ext}`),
|
||||
`vite.config.${ext}`,
|
||||
);
|
||||
}
|
||||
// vitest.config.* is preferred over vite.config.*, as Vitest itself does.
|
||||
assert.ok(
|
||||
VITEST_CONFIG_FILENAMES.indexOf("vitest.config.cjs") <
|
||||
VITEST_CONFIG_FILENAMES.indexOf("vite.config.ts"),
|
||||
);
|
||||
});
|
||||
|
|
|
|||
Loading…
Reference in a new issue