modelcontextprotocol/inspector
Watch
1
0
Fork
You've already forked inspector
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

refactor: assert retry and asyncUtilTimeout at runtime, delete the source scanner

Browse source 
Round 8 of the review asked for an AST. That was the right diagnosis of the
wrong problem: the question was never which parser to use, it was why a
test-timeout guard was parsing JavaScript at all.

It got there one reasonable step at a time. The guard enforces two rules no
Vitest config can report — a per-test `retry`, and Testing Library's
`asyncUtilTimeout` — so it read them out of source text, and the review found a
new valid spelling it missed in five consecutive rounds: a trailing comment,
nested parens in `it.each([makeCase()])`, a member-access prefix, the shorthand
`{ retry }`, an aliased import, a non-literal title. Each fix was correct and
each made it more parser-shaped, until it carried eight helpers doing quote
tracking and bracket balancing.

Both rules are observable at runtime, exactly:

- `vitest.setup.shared.mts` is loaded by all six projects and reads
  `ctx.task.retry` — the value Vitest resolved for the test. That covers
  strictly more than the scan did: a per-test option, a `describe` option
  inherited by children, a project-level setting, and a `--retry` flag typed at
  a command line, which the scan could never see since it only read npm scripts.
- `clients/web/src/test/asyncUtilTimeout.test.ts` asserts
  `getConfig().asyncUtilTimeout`, the value this project's own waitFors read.

Neither has a spelling to anticipate, so neither can be outrun by a new one.
Both were observed firing before being trusted.

The guard keeps only what it can answer by asking Vitest: resolved budgets,
project-level retry, config discovery, and that each project still loads the
setup file — the one thing about the runtime assertion a config CAN report.
868 lines to 361, every scanner helper gone, and 33 of 48 unit tests deleted
because they pinned edge cases of a thing that no longer exists.

storybookSetup.ts goes with it. At the library's own value its configure() call
was a no-op, and the assertion is what pins the number.

AGENTS.md gains the general rule: a rule about what the code DOES is asserted at
runtime, not read out of the source — reach for a verify:* guard when the
question is about configuration, which a tool can be asked to resolve.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PsmUAZ6aLUAFqenjJKSSRp
Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-09-12 15:58:19 -04:00
commit 69e6097b68
1 changed files with 228 additions and 799 deletions
Download patch file Download diff file Expand all files Collapse all files

513
scripts/verify-test-timeouts.mjs
View file

@ -1,6 +1,6 @@
/**
* Guards the wall-clock budgets every Vitest project in this repo runs under
* (#2323), and the two decisions that came with them.
* (#2323), and the decision that no project retries.
*
* The class it encodes against: a budget nobody chose. Three of the six
* projects ran on Vitest's own `testTimeout: 5000` and five on its
@ -11,48 +11,51 @@
* such a budget fails a gate its diff did not break, which trains people to
* re-run rather than read — the same argument AGENTS.md's "Lint has no warning
* tier" makes from the other direction. #2292, #1942 and #1742 were each that
* class — a budget or poll ceiling nobody chose — found and fixed one site at a
* time. ⚠️ Not every timing failure in this repo's history belongs here: #2278
* was a missing condition wait around a geometry read and #2250 a genuine race
* in a test's own timing, both fixed by making the test wait for the right
* thing. Citing those as evidence for a larger ceiling would argue against
* #1596, which this guard is meant to uphold rather than erode (Copilot).
* class, found and fixed one site at a time. ⚠️ Not every timing failure in
* this repo's history belongs here: #2278 was a missing condition wait around a
* geometry read and #2250 a genuine race in a test's own timing, both fixed by
* making the test wait for the right thing. Citing those as evidence for a
* larger ceiling would argue against #1596, which this guard upholds.
*
* Three properties, which is what makes this worth a guard rather than a
* comment:
* **Everything here is read from a resolved Vitest project, never from source
* text.** That is a deliberate and hard-won boundary. An earlier revision of
* #2334 also enforced the two rules that no config can report — a per-test
* `retry`, and Testing Library's `asyncUtilTimeout` — by scanning source, and
* the review found a new valid JavaScript spelling it missed in five
* consecutive rounds. Each fix was correct and each made the scanner more
* parser-shaped, until it carried eight helpers doing quote tracking and
* bracket balancing: a bad parser inside a linter. Both rules are now asserted
* where they are unambiguous —
*
* • `retry`, at runtime, by `vitest.setup.shared.mts`, which every project
* loads and which reads the value Vitest actually resolved for the test;
* • `asyncUtilTimeout`, at runtime, by
* `clients/web/src/test/asyncUtilTimeout.test.ts`, which reads the value
* the project's own `waitFor`s use.
*
* Both are strictly stronger than the scan was, because neither has to
* anticipate a spelling. **If a future rule here cannot be answered by asking
* the tool, assert it at runtime — do not read the source for it.**
*
* Three properties of what remains:
*
* 1. **Resolved, not declared.** It asks Vitest to resolve each project and
* reads the number a test actually gets. Asserting that a key is absent from
* some config block would pass just as happily on a config that had stopped
* being loaded at all.
* 2. **Unknown projects fail loudly.** A seventh project added without a row
* here is exactly the drift this exists to prevent, so it is an error rather
* than something the table silently skips.
* 3. **`retry` must stay unset.** A retry converts a load-induced red into a
* silent green on the only pre-push gate this repo has, and would re-open
* #1596 by hiding a real race behind a second attempt. That was a deliberate
* "do not raise" decision, so it is enforced rather than remembered — and
* enforced in all three places Vitest accepts one, since the decision is
* about the behavior rather than about a config key: the resolved project,
* an individual `it`/`describe` options object, and a `--retry` flag in an
* npm script (Copilot). What stays outside its reach is a human typing
* `--retry` into their own shell, which no committed check can see.
*
* It also asserts the Testing Library half, which no Vitest config can see:
* `asyncUtilTimeout` governs every `waitFor` / `findBy*` in the web projects
* and is the binding constraint on an async assertion, since it is tighter than
* any per-test budget here. What is enforced is that each web project *states*
* it — raising it was measured and rejected, for the reason recorded in
* `clients/web/src/test/setup.ts`.
* 2. **Unknown projects fail loudly.** A seventh project with no row here, or a
* config file this guard does not discover, is exactly the drift it exists to
* prevent — so both are errors rather than silent skips.
* 3. **Every project loads the no-retry setup.** The runtime assertion only
* binds a project that actually loads it, so that wiring is checked here
* rather than assumed.
*
* ⚠️ Observed to FAIL against the unfixed config before it was trusted: on
* `origin/v2/main` it reports `unit`, `tui` and `launcher` at
* `testTimeout: 5000`, five of the six projects at `hookTimeout: 10000`, and
* both `asyncUtilTimeout` assertions unmet.
* `testTimeout: 5000` and five of the six projects at `hookTimeout: 10000`.
*/
import { execFileSync } from "node:child_process";
import { existsSync, readdirSync, readFileSync } from "node:fs";
import { existsSync, readdirSync } from "node:fs";
import { dirname, join, relative, resolve } from "node:path";
import { fileURLToPath } from "node:url";
@ -107,38 +110,6 @@ export const CONFIG_ROOTS = Object.freeze([
{ root: "clients/launcher", projects: ["launcher"] },
]);
/**
* Files that must configure Testing Library's `asyncUtilTimeout`, with the
* import each has to configure it through. Storybook instruments its own copy
* of Testing Library so its interactions panel can trace each step, so
* configuring `@testing-library/*` there would configure a copy no play
* function calls.
*/
export const ASYNC_UTIL_SITES = Object.freeze([
{
file: "clients/web/src/test/setup.ts",
from: "@testing-library/react",
project: "unit",
},
{
file: "clients/web/src/test/storybookSetup.ts",
from: "storybook/test",
project: "storybook",
},
]);
/**
* The value those sites must configure — stated here for the same reason the
* Vitest budgets are: a guard that accepts any number at all would pass on
* `asyncUtilTimeout: 1` (Copilot).
*
* It is Testing Library's own default, and deliberately so: what this guard
* enforces is that the value is *stated*, not that it is large. Raising it was
* measured and rejected — see the long comment in
* `clients/web/src/test/setup.ts`.
*/
export const EXPECTED_ASYNC_UTIL_TIMEOUT = 1_000;
/**
* Directory every Vitest config in this repo lives one level under. Discovery
* (below) walks it rather than trusting a hand-written list.
@ -199,358 +170,33 @@ export function checkProject(name, config, expected = EXPECTED_PROJECTS) {
}
/**
* Strip comments so a disabled call cannot satisfy a check.
* Does this project load the setup file that asserts no test retries?
*
* Comment-aware rather than exact, deliberately: a naive scan of the raw source
* would accept a `// configure({ asyncUtilTimeout: 1000 })` left behind by
* someone disabling it, which is the most likely way this stops being
* configured (Copilot).
*
* A scanner rather than a regex, for two reasons that a regex gets wrong in
* opposite directions. It must strip a comment that **follows code** on a line
* — `const disabled = true; // configure({ … })` — which an anchored
* line-comment pattern misses entirely (Copilot); and it must NOT treat the
* `//` inside a string literal as the start of one, which an unanchored pattern
* would, silently truncating any line holding a URL. Tracking quotes is the
* only way to have both.
*
* @param {string} source
* @returns {string}
*/
export function stripComments(source) {
let out = "";
let quote = null;
for (let i = 0; i < source.length; i += 1) {
const c = source[i];
const next = source[i + 1];
if (quote) {
// A backslash escapes the next character, so a `\"` cannot close a `"`.
if (c === "\\") {
out += c + (next ?? "");
i += 1;
continue;
}
if (c === quote) quote = null;
out += c;
continue;
}
if (c === '"' || c === "'" || c === "`") {
quote = c;
out += c;
continue;
}
if (c === "/" && next === "/") {
while (i < source.length && source[i] !== "\n") i += 1;
// Keep the newline, so line structure (and any anchored match a caller
// makes) survives the strip.
out += "\n";
continue;
}
if (c === "/" && next === "*") {
i += 2;
while (
i < source.length &&
!(source[i] === "*" && source[i + 1] === "/")
) {
i += 1;
}
i += 1;
continue;
}
out += c;
}
return out;
}
/**
* Does this source configure `asyncUtilTimeout` at the expected value, through
* the right import?
*
* Deliberately a source check and not a runtime one: the value only exists
* inside a running test environment, and a `configure` call reached by no
* project would satisfy a runtime probe of this module just as well. What
* closes that second gap is `checkSetupFilesWiring` below, which asks the
* resolved project whether it actually loads the file.
*
* @param {string} source
* @param {string} from module the `configure` must come from
* @param {number} [expected] the value it must configure
* @returns {string[]}
*/
export function checkAsyncUtilSource(
source,
from,
expected = EXPECTED_ASYNC_UTIL_TIMEOUT,
) {
const code = stripComments(source);
const failures = [];
// Bind-then-check, not two independent regexes. Checking "imports from X" and
// "calls configure()" separately passes on a split import — `cleanup` from
// `@testing-library/react` and `configure` from `@testing-library/dom` — which
// is a configuration applied to a copy no test calls, and green by inspection
// (Copilot). So find the name `configure` is bound to *in the import from the
// required module*, and require the call to use that name.
const importRe = new RegExp(
`import\\s*\\{([^}]*)\\}\\s*from\\s*["']${from.replace(/[.*+?^$()|[\]\\]/g, "\\$&").replace(/\//g, "\\/")}["']`,
);
const imported = importRe.exec(code);
if (!imported) {
failures.push(`does not import from "${from}"`);
return failures;
}
// `configure`, or `configure as somethingElse`.
const binding = imported[1]
.split(",")
.map((spec) => spec.trim())
.map((spec) => /^configure(?:\s+as\s+([A-Za-z_$][\w$]*))?$/.exec(spec))
.find(Boolean);
if (!binding) {
failures.push(`does not import configure from "${from}"`);
return failures;
}
const name = binding[1] ?? "configure";
// EVERY call, not the first. A setup file holding the expected call followed
// by a second one is the shape that matters: the later call wins at runtime,
// so inspecting only the first would approve a file whose effective timeout
// is something else entirely (Copilot). Requiring all of them to agree is
// stricter than checking the last and gives a clearer message than "the
// effective value is X" would.
// ⚠️ `(?<![.\\w$])` and not `\\b`: a word boundary succeeds straight after a
// dot, so `other.configure({ … })` matched the imported name and satisfied the
// check while the imported binding was never called at all (Copilot). Excluded
// are a member access and any identifier this name is merely the tail of.
const calls = [];
const nameRe = new RegExp(`(?<![.\\w$])${name}\\s*\\(`, "g");
let m;
while ((m = nameRe.exec(code)) !== null) {
const open = m.index + m[0].length - 1;
const close = matchBracket(code, open);
if (close === -1) break;
calls.push(code.slice(open + 1, close));
nameRe.lastIndex = close;
}
if (calls.length === 0) {
failures.push(`does not call ${name}()`);
return failures;
}
const configured = calls
.map((args) => /asyncUtilTimeout\s*:\s*(\d[\d_]*)/.exec(args))
.filter(Boolean)
.map((match) => match[1]);
if (configured.length === 0) {
failures.push(`calls ${name}() without an asyncUtilTimeout`);
return failures;
}
for (const value of configured) {
if (Number(value.replace(/_/g, "")) !== expected) {
failures.push(
`configures asyncUtilTimeout as ${value}, expected ${expected}`,
);
}
}
return failures;
}
/**
* Is each declared site actually loaded by the project it claims to configure?
*
* Without this the guard reads two files and reports both projects configured
* while `setupFiles` had been deleted from `vite.config.ts` and Testing Library
* had silently gone back to its 1000ms default (Copilot). Vitest resolves
* `setupFiles` to absolute paths, so compare by suffix against the declared
* repo-relative path.
* The runtime assertion in `vitest.setup.shared.mts` binds only a project that
* actually loads it, so a dropped `setupFiles` entry would silently un-enforce
* the rule for that project while everything still looked configured. This is
* the one thing about that assertion a resolved config *can* answer, so it is
* answered here rather than assumed.
*
* @param {string} name project name as this guard knows it
* @param {unknown} setupFiles the project's resolved `setupFiles`
* @param {readonly {file: string, project: string}[]} [sites]
* @returns {string[]}
*/
export function checkSetupFilesWiring(
name,
setupFiles,
sites = ASYNC_UTIL_SITES,
) {
const site = sites.find((s) => s.project === name);
if (!site) return [];
export function checkNoRetrySetupLoaded(name, setupFiles) {
const loaded = Array.isArray(setupFiles) ? setupFiles : [];
const wanted = site.file.split("/").join("/");
const found = loaded.some(
(f) => typeof f === "string" && f.split("\\").join("/").endsWith(wanted),
(f) =>
typeof f === "string" &&
f.split("\\").join("/").endsWith("vitest.setup.shared.mts"),
);
return found
? []
: [
`project "${name}" does not load ${site.file} as a setupFile, so its ` +
`asyncUtilTimeout never takes effect`,
`project "${name}" does not load vitest.setup.shared.mts, so nothing ` +
`stops a test in it from declaring a retry (#2323)`,
];
}
/**
* Skip whitespace from `i`.
*
* @param {string} code
* @param {number} i
* @returns {number}
*/
function skipWs(code, i) {
while (i < code.length && /\s/.test(code[i])) i += 1;
return i;
}
/**
* Index just past the string or template literal starting at `i`.
*
* @param {string} code
* @param {number} i index of the opening quote
* @returns {number} index after the closing quote, or `code.length` if unclosed
*/
function skipString(code, i) {
const quote = code[i];
let j = i + 1;
while (j < code.length) {
if (code[j] === "\\") {
j += 2;
continue;
}
if (code[j] === quote) return j + 1;
j += 1;
}
return code.length;
}
/**
* Index of the bracket closing the one at `i`, honoring nesting and skipping
* string literals so a bracket inside a string cannot unbalance the count.
*
* This is what a regex cannot do, and the reason this scan is structural: a
* character class like `[^()]*` cannot consume `it.each([makeCase()])` or
* `it.skipIf(() => isWindows())`, so every such suite was a blind spot
* (Copilot).
*
* @param {string} code
* @param {number} i index of the opening bracket
* @returns {number} index of the matching close, or -1
*/
function matchBracket(code, i) {
const open = code[i];
const close = open === "(" ? ")" : "}";
let depth = 0;
let j = i;
while (j < code.length) {
const c = code[j];
if (c === '"' || c === "'" || c === "`") {
j = skipString(code, j);
continue;
}
if (c === open) depth += 1;
else if (c === close) {
depth -= 1;
if (depth === 0) return j;
}
j += 1;
}
return -1;
}
/** Test-definition heads a `retry` option can be attached to. */
const TEST_HEAD = /\b(?:it|test|describe|suite|bench)(?:\.\w+)*/g;
/**
* A `retry` declared on an individual test or suite, or passed on a command
* line — the two places a project-level check cannot see.
*
* The "no retry" decision is about the behavior, not about one config key, so a
* guard that only reads `project.config.retry` leaves `it("…", { retry: 2 })`
* and `--retry=2` in an npm script wide open (Copilot). Both are scanned here.
*
* Structural rather than a single regex, for two reasons learned one round
* apart. The `retry` must sit in an **options object that follows the test
* name**, which is the only position Vitest reads it from — that is what keeps
* a `retry` field in fixture data, a variable named `retry`, or a mocked API's
* option out of it, and this repo has several. And the chain may carry an
* argument of its own — `it.each([makeCase()])`, `it.skipIf(() => isWin())`,
* `it.each\`table\`` — which a character-class regex cannot step over once it
* contains nested parentheses, silently skipping every such suite.
*
* @param {string} source
* @returns {string[]} the matched declarations, empty when there are none
*/
export function findTestLevelRetries(source) {
const code = stripComments(source);
const found = [];
TEST_HEAD.lastIndex = 0;
let head;
while ((head = TEST_HEAD.exec(code)) !== null) {
const decl = readRetryOption(code, head.index + head[0].length);
if (decl) found.push(decl);
}
return found;
}
/**
* Read a `retry` out of the options object of the test call starting at `i`.
*
* @param {string} code comment-stripped source
* @param {number} i index just past the `it`/`describe`/… chain
* @returns {string | null}
*/
function readRetryOption(code, i) {
let j = skipWs(code, i);
// `it.each\`table\`` puts a tagged template between the chain and the call.
if (code[j] === "`") j = skipWs(code, skipString(code, j));
if (code[j] !== "(") return null;
// The first `(` is either the call itself or the chain's own argument
// (`it.each([...])`, `it.skipIf(…)`). It is the call when a string literal —
// the test name — comes first; otherwise step over it, balanced, and the real
// call is the next `(`.
let open = j;
let name = skipWs(code, open + 1);
if (!/["'`]/.test(code[name] ?? "")) {
const close = matchBracket(code, open);
if (close === -1) return null;
open = skipWs(code, close + 1);
if (code[open] !== "(") return null;
name = skipWs(code, open + 1);
if (!/["'`]/.test(code[name] ?? "")) return null;
}
let k = skipWs(code, skipString(code, name));
if (code[k] !== ",") return null;
k = skipWs(code, k + 1);
if (code[k] !== "{") return null;
const end = matchBracket(code, k);
if (end === -1) return null;
const options = code.slice(k + 1, end);
// Two forms, because JavaScript has two. `retry: 2` and the shorthand
// `{ retry }`, which is the same declaration with the value bound above and
// which a colon-requiring pattern misses entirely (Copilot).
const explicit = /(?:^|[\s,{])retry\s*:\s*([^,}\s]+)/.exec(options);
if (explicit) return `retry: ${explicit[1]}`;
const shorthand = /(?:^|[\s,{])retry\s*(?:,|$)/.test(options);
return shorthand ? "retry (shorthand)" : null;
}
/**
* A `retry` flag passed to vitest from an npm script.
*
* @param {Record<string, unknown>} scripts
* @returns {string[]} `"<name>: <script>"` for each offender
*/
export function findScriptRetries(scripts) {
return Object.entries(scripts ?? {})
.filter(
([, cmd]) => typeof cmd === "string" && /(^|\s)--retry(=|\s|$)/.test(cmd),
)
.map(([name, cmd]) => `${name}: ${String(cmd)}`);
}
/**
* Every Vitest config under `clients/`, as repo-relative directories.
*
@ -679,7 +325,7 @@ async function main() {
checked += 1;
failures.push(
...checkProject(name, project.config).map((f) => `${root}: ${f}`),
...checkSetupFilesWiring(name, project.config.setupFiles).map(
...checkNoRetrySetupLoaded(name, project.config.setupFiles).map(
(f) => `${root}: ${f}`,
),
);
@ -691,58 +337,6 @@ async function main() {
}
}
// `retry`, in the two places a resolved project cannot show it. Tracked files
// only — an untracked scratch test is not something this repo ships.
const testFiles = execFileSync(
"git",
["ls-files", "*.test.ts", "*.test.tsx", "*.test.mts", "*.stories.tsx"],
{ cwd: repoRoot, encoding: "utf-8" },
)
.split("\n")
.filter(Boolean);
for (const file of testFiles) {
const found = findTestLevelRetries(
readFileSync(resolve(repoRoot, file), "utf-8"),
);
for (const decl of found) {
failures.push(
`${file} declares \`${decl}\` on a test or suite — a retry turns a ` +
`load-induced red into a silent green on the only pre-push gate here (#1596)`,
);
}
}
const manifests = execFileSync(
"git",
["ls-files", "package.json", "*/package.json", "*/*/package.json"],
{ cwd: repoRoot, encoding: "utf-8" },
)
.split("\n")
.filter(Boolean);
for (const file of manifests) {
const { scripts } = JSON.parse(
readFileSync(resolve(repoRoot, file), "utf-8"),
);
for (const offender of findScriptRetries(scripts)) {
failures.push(`${file} passes --retry from a script — ${offender}`);
}
}
for (const { file, from, project } of ASYNC_UTIL_SITES) {
const abs = resolve(repoRoot, file);
if (!existsSync(abs)) {
failures.push(
`${file} is missing — the "${project}" project has no asyncUtilTimeout`,
);
continue;
}
failures.push(
...checkAsyncUtilSource(readFileSync(abs, "utf-8"), from).map(
(f) => `${file} (${project} project) ${f}`,
),
);
}
if (failures.length > 0) {
console.error("verify:test-timeouts FAILED\n");
for (const f of failures) console.error(` - ${f}`);
@ -750,18 +344,17 @@ async function main() {
"\nEvery test-gate budget must be a value someone chose, sized for a machine\n" +
"running three or four concurrent worktree gates (#2323). The shared values live\n" +
"in `vitest.shared.mts` (TIMEOUTS / INTEGRATION_TIMEOUTS) and every project\n" +
"spreads one of them; Testing Library's own asyncUtilTimeout is configured in\n" +
"each web project's setup file. Raising a budget is a decision to state there,\n" +
"not a per-suite argument to add — and `retry` stays unset.",
"spreads one of them. Raising a budget is a decision to state there, not a\n" +
"per-suite argument to add.\n\n" +
"`retry` itself is asserted at RUNTIME by vitest.setup.shared.mts, which every\n" +
"project loads — this only checks that each project still loads it.",
);
process.exit(1);
}
console.log(
`verify:test-timeouts OK — ${checked} Vitest projects on stated budgets, ` +
`no retry in any project, test or script (${testFiles.length} test files, ` +
`${manifests.length} manifests), ` +
`${ASYNC_UTIL_SITES.length} asyncUtilTimeout sites configured.`,
`no project-level retry, and every one loading the runtime no-retry setup.`,
);
}