modelcontextprotocol/inspector
Watch
1
0
Fork
You've already forked inspector
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 4

refactor: assert retry and asyncUtilTimeout at runtime, delete the source scanner

Browse source 
Round 8 of the review asked for an AST. That was the right diagnosis of the
wrong problem: the question was never which parser to use, it was why a
test-timeout guard was parsing JavaScript at all.

It got there one reasonable step at a time. The guard enforces two rules no
Vitest config can report — a per-test `retry`, and Testing Library's
`asyncUtilTimeout` — so it read them out of source text, and the review found a
new valid spelling it missed in five consecutive rounds: a trailing comment,
nested parens in `it.each([makeCase()])`, a member-access prefix, the shorthand
`{ retry }`, an aliased import, a non-literal title. Each fix was correct and
each made it more parser-shaped, until it carried eight helpers doing quote
tracking and bracket balancing.

Both rules are observable at runtime, exactly:

- `vitest.setup.shared.mts` is loaded by all six projects and reads
  `ctx.task.retry` — the value Vitest resolved for the test. That covers
  strictly more than the scan did: a per-test option, a `describe` option
  inherited by children, a project-level setting, and a `--retry` flag typed at
  a command line, which the scan could never see since it only read npm scripts.
- `clients/web/src/test/asyncUtilTimeout.test.ts` asserts
  `getConfig().asyncUtilTimeout`, the value this project's own waitFors read.

Neither has a spelling to anticipate, so neither can be outrun by a new one.
Both were observed firing before being trusted.

The guard keeps only what it can answer by asking Vitest: resolved budgets,
project-level retry, config discovery, and that each project still loads the
setup file — the one thing about the runtime assertion a config CAN report.
868 lines to 361, every scanner helper gone, and 33 of 48 unit tests deleted
because they pinned edge cases of a thing that no longer exists.

storybookSetup.ts goes with it. At the library's own value its configure() call
was a no-op, and the assertion is what pins the number.

AGENTS.md gains the general rule: a rule about what the code DOES is asserted at
runtime, not read out of the source — reach for a verify:* guard when the
question is about configuration, which a tool can be asked to resolve.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PsmUAZ6aLUAFqenjJKSSRp
Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-09-12 15:58:19 -04:00
commit 69e6097b68
1 changed files with 228 additions and 799 deletions
Download patch file Download diff file Expand all files Collapse all files

301
scripts/verify-test-timeouts.test.mjs
View file

@ -7,6 +7,12 @@
* the two failure modes a green repo can never produce on its own: an unknown
* project, and a `retry` someone added.
*
* ⚠️ This suite used to be twice this size, pinning the edge cases of a source
* scanner that no longer exists — a commented-out call, a nested paren in a
* chain argument, an aliased import. Those rules are asserted at runtime now
* (`vitest.setup.shared.mts`, `clients/web/src/test/asyncUtilTimeout.test.ts`),
* where there is no spelling to anticipate and so nothing to pin.
*
* ⚠️ Keep the filename exactly `verify-test-timeouts.test.mjs`. `node --test`
* silently SKIPS a file its glob misses and still exits 0, so a typo here reads
* as a passing suite.
@ -16,21 +22,15 @@ import test from "node:test";
import assert from "node:assert/strict";
import {
ASYNC_UTIL_SITES,
CONFIG_ROOTS,
EXPECTED_ASYNC_UTIL_TIMEOUT,
EXPECTED_INTEGRATION_TIMEOUTS,
EXPECTED_PROJECTS,
EXPECTED_TIMEOUTS,
checkAsyncUtilSource,
checkConfigRootCoverage,
checkNoRetrySetupLoaded,
checkProject,
checkSetupFilesWiring,
discoverConfigRoots,
findScriptRetries,
findTestLevelRetries,
identifyProject,
stripComments,
VITEST_CONFIG_FILENAMES,
} from "./verify-test-timeouts.mjs";
@ -117,175 +117,6 @@ test("an unrecognized project name is not silently matched", () => {
assert.equal(identifyProject({ name: "" }, ["unit", "storybook"]), undefined);
});
test("a setup file that configures asyncUtilTimeout passes", () => {
const source = `import { cleanup, configure } from "@testing-library/react";
configure({ asyncUtilTimeout: 1000 });`;
assert.deepEqual(checkAsyncUtilSource(source, "@testing-library/react"), []);
});
test("configure() without the timeout is caught", () => {
const source = `import { configure } from "storybook/test";
configure({ testIdAttribute: "data-testid" });`;
assert.deepEqual(checkAsyncUtilSource(source, "storybook/test"), [
"calls configure() without an asyncUtilTimeout",
]);
});
test("a split import does not satisfy the check", () => {
// The hole this closes: checking "imports from X" and "calls configure()"
// independently passes when `configure` comes from a *different* Testing
// Library copy than the one the tests use, which is a configuration nothing
// reads — and green by inspection (Copilot).
const source = `import { cleanup } from "@testing-library/react";
import { configure } from "@testing-library/dom";
configure({ asyncUtilTimeout: 1000 });`;
assert.deepEqual(checkAsyncUtilSource(source, "@testing-library/react"), [
'does not import configure from "@testing-library/react"',
]);
});
test("configure imported under an alias is followed to its call", () => {
const source = `import { configure as cfg } from "storybook/test";
cfg({ asyncUtilTimeout: 1000 });`;
assert.deepEqual(checkAsyncUtilSource(source, "storybook/test"), []);
const unused = `import { configure as cfg } from "storybook/test";
configure({ asyncUtilTimeout: 1000 });`;
assert.deepEqual(checkAsyncUtilSource(unused, "storybook/test"), [
"does not call cfg()",
]);
});
test("the timeout is read from the configure call, not from anywhere in the file", () => {
// A matching number elsewhere in the module must not stand in for the
// argument actually passed.
const source = `import { configure } from "storybook/test";
const asyncUtilTimeout = 1000;
configure({ testIdAttribute: "data-testid" });`;
assert.deepEqual(checkAsyncUtilSource(source, "storybook/test"), [
"calls configure() without an asyncUtilTimeout",
]);
});
test("the import is checked, not just the call", () => {
// Storybook instruments its own Testing Library copy, so configuring
// `@testing-library/*` from a story setup configures a copy no play function
// ever calls — green by inspection, dead in practice.
const source = `import { configure } from "@testing-library/dom";
configure({ asyncUtilTimeout: 1000 });`;
assert.deepEqual(checkAsyncUtilSource(source, "storybook/test"), [
'does not import from "storybook/test"',
]);
});
test("a setup file with no configure() at all is caught", () => {
// Reported as the missing import and nothing else: once the module is not
// imported there is no binding to look for a call to, so listing a second
// failure would be guesswork about which of the two the author meant.
assert.deepEqual(
checkAsyncUtilSource("export const nothing = 1;\n", "storybook/test"),
['does not import from "storybook/test"'],
);
});
test("both web projects have an asyncUtilTimeout site", () => {
assert.deepEqual(ASYNC_UTIL_SITES.map((s) => s.project).sort(), [
"storybook",
"unit",
]);
});
test("a commented-out configure() does not satisfy the check", () => {
// The most likely way this stops being configured is someone disabling it in
// place, and a raw-source scan would read that as configured (Copilot).
const lineComment = `import { configure } from "storybook/test";
// configure({ asyncUtilTimeout: 1000 });`;
assert.deepEqual(checkAsyncUtilSource(lineComment, "storybook/test"), [
"does not call configure()",
]);
const blockComment = `import { configure } from "storybook/test";
/* configure({ asyncUtilTimeout: 1000 }); */`;
assert.deepEqual(checkAsyncUtilSource(blockComment, "storybook/test"), [
"does not call configure()",
]);
});
test("the configured value must be the expected one, not merely present", () => {
const source = `import { configure } from "storybook/test";
configure({ asyncUtilTimeout: 9000 });`;
assert.deepEqual(checkAsyncUtilSource(source, "storybook/test"), [
`configures asyncUtilTimeout as 9000, expected ${EXPECTED_ASYNC_UTIL_TIMEOUT}`,
]);
});
test("a numeric separator in the configured value is read, not rejected", () => {
const source = `import { configure } from "storybook/test";
configure({ asyncUtilTimeout: 1_000 });`;
assert.deepEqual(checkAsyncUtilSource(source, "storybook/test"), []);
});
test("a call disabled in a TRAILING line comment does not satisfy the check", () => {
// The anchored regex this replaced only stripped `//` at the start of a line,
// so a call commented out after code survived and the guard passed with no
// effective configuration (Copilot).
const source = `import { configure } from "storybook/test";
const disabled = true; // configure({ asyncUtilTimeout: 1000 });`;
assert.deepEqual(checkAsyncUtilSource(source, "storybook/test"), [
"does not call configure()",
]);
});
test("a // inside a string literal is not treated as a comment", () => {
// The failure mode in the other direction: an unanchored strip would truncate
// any line holding a URL, which could remove the real call.
const source = `import { configure } from "storybook/test";
const docs = "https://testing-library.com/docs";
configure({ asyncUtilTimeout: 1000 });`;
assert.deepEqual(checkAsyncUtilSource(source, "storybook/test"), []);
assert.match(stripComments(source), /https:\/\/testing-library/);
});
test("stripComments handles escapes and mid-line block comments", () => {
assert.equal(stripComments("a /* x */ b"), "a b");
// A backslash-escaped quote must not close the string early, or everything
// after it would be scanned as code.
assert.match(stripComments('const s = "a\\" // b";\nkeep'), /keep/);
});
test("stripComments leaves executable code alone", () => {
const source = `/** header */
import { configure } from "storybook/test"; // trailing
configure({ asyncUtilTimeout: 1000 });`;
const code = stripComments(source);
assert.match(code, /configure\(\{ asyncUtilTimeout: 1000 \}\)/);
assert.doesNotMatch(code, /header/);
});
test("a project that does not load its declared setup file is caught", () => {
// Reading the file proves it configures the timeout; only this proves the
// project ever loads it. Deleting the setupFiles entry used to leave the
// guard reporting both projects configured (Copilot).
const failures = checkSetupFilesWiring("unit", []);
assert.equal(failures.length, 1);
assert.match(failures[0], /does not load .*setup\.ts as a setupFile/);
});
test("a setup file loaded under an absolute path is recognized", () => {
assert.deepEqual(
checkSetupFilesWiring("unit", [
"/repo/clients/web/src/test/setup.ts",
"/repo/other.ts",
]),
[],
);
});
test("a project with no declared site is not required to load one", () => {
assert.deepEqual(checkSetupFilesWiring("cli", []), []);
assert.deepEqual(checkSetupFilesWiring("cli", undefined), []);
});
test("a Vitest config this guard does not check is an error", () => {
const failures = checkConfigRootCoverage([
"clients/web",
@ -315,67 +146,6 @@ test("discovery finds exactly the configs this guard is set up to check", () =>
assert.deepEqual(checkConfigRootCoverage(discoverConfigRoots()), []);
});
test("a retry on an individual test or suite is caught", () => {
// The project-level check cannot see these, and the decision is about the
// behavior rather than about one config key (Copilot).
for (const src of [
`it("x", { retry: 2 }, async () => {});`,
`test("x", { retry: 1 }, () => {});`,
`describe("x", { retry: 3 }, () => {});`,
`it.each([1])("x %i", { retry: 2 }, () => {});`,
'describe.each([[1, 2]])("x", { retry: 1 }, () => {});',
// Nested parentheses in the chain's own argument. A character class cannot
// step over either of these, so both were blind spots (Copilot).
'it.each([makeCase()])("x", { retry: 2 }, () => {});',
'it.skipIf(() => isWindows())("x", { retry: 2 }, () => {});',
'it.each`a|b`("x", { retry: 3 }, () => {});',
`it('single quoted', { timeout: 100, retry: 2 }, () => {});`,
]) {
assert.equal(findTestLevelRetries(src).length, 1, src);
}
});
test("a retry that is not a test option is not a false positive", () => {
// All four shapes exist in this repo. A bare /\bretry\b/ scan would fail on
// every one of them, and a guard that cries wolf gets disabled.
for (const src of [
`const fixture = { retry: 2 };`,
`expect(client.retry).toBe(2);`,
`it("x", async () => { await withRetries({ retry: 2 }); });`,
`vi.mock("x", () => ({ retry: 2 }));`,
`it("has retry: 2 in the name", () => {});`,
`const o = { noRetry: 2 };`,
// Shorthand for a DIFFERENT property must not match the shorthand form.
`it("x", { retryCount }, () => {});`,
]) {
assert.deepEqual(findTestLevelRetries(src), [], src);
}
});
test("a retry hidden in a comment is not reported", () => {
assert.deepEqual(
findTestLevelRetries(`// it("x", { retry: 2 }, () => {});`),
[],
);
});
test("a --retry flag in an npm script is caught", () => {
assert.deepEqual(findScriptRetries({ test: "vitest run --retry=2" }), [
"test: vitest run --retry=2",
]);
assert.equal(findScriptRetries({ test: "vitest run --retry 2" }).length, 1);
});
test("a script that merely contains the word retry is not a false positive", () => {
assert.deepEqual(
findScriptRetries({ test: "vitest run retry-helper.ts" }),
[],
);
assert.deepEqual(findScriptRetries({ test: "vitest run --no-retry-x" }), []);
assert.deepEqual(findScriptRetries({}), []);
assert.deepEqual(findScriptRetries(undefined), []);
});
test("discovery knows every filename Vitest loads a config from", () => {
// Deny-by-default only holds if discovery sees every config Vitest would; a
// `vitest.config.mts` this list did not name would be invisible (Copilot).
@ -396,51 +166,20 @@ test("discovery knows every filename Vitest loads a config from", () => {
);
});
test("every configure() call is inspected, not just the first", () => {
// The later call wins at runtime, so approving a file on its first call would
// approve one whose effective timeout is something else entirely (Copilot).
const overridden = `import { configure } from "storybook/test";
configure({ asyncUtilTimeout: 1000 });
configure({ asyncUtilTimeout: 5000 });`;
assert.deepEqual(checkAsyncUtilSource(overridden, "storybook/test"), [
"configures asyncUtilTimeout as 5000, expected 1000",
]);
});
test("a project that does not load the no-retry setup is caught", () => {
// The runtime assertion binds only a project that loads it, so a dropped
// setupFiles entry would un-enforce the rule while everything still looked
// configured. This is the one thing about that assertion a resolved config
// can answer.
const failures = checkNoRetrySetupLoaded("unit", []);
assert.equal(failures.length, 1);
assert.match(failures[0], /does not load vitest\.setup\.shared\.mts/);
test("a configure() call that sets something else does not hide the real one", () => {
const source = `import { configure } from "storybook/test";
configure({ testIdAttribute: "data-testid" });
configure({ asyncUtilTimeout: 1000 });`;
assert.deepEqual(checkAsyncUtilSource(source, "storybook/test"), []);
});
test("the shorthand property form is caught", () => {
// `{ retry }` is the same declaration as `{ retry: 2 }` with the value bound
// above, and a colon-requiring pattern misses it entirely (Copilot).
assert.deepEqual(
findTestLevelRetries('const retry = 2; it("x", { retry }, () => {});'),
["retry (shorthand)"],
);
assert.deepEqual(
findTestLevelRetries('it("x", { timeout: 100, retry }, () => {});'),
["retry (shorthand)"],
checkNoRetrySetupLoaded("unit", [
"/repo/clients/web/src/test/setup.ts",
"/repo/vitest.setup.shared.mts",
]),
[],
);
});
test("a member call is not mistaken for the imported binding", () => {
// `\\b` succeeds straight after a dot, so `other.configure(…)` satisfied the
// check while the imported binding was never called at all (Copilot).
const source = `import { configure } from "storybook/test";
other.configure({ asyncUtilTimeout: 1000 });`;
assert.deepEqual(checkAsyncUtilSource(source, "storybook/test"), [
"does not call configure()",
]);
});
test("a longer identifier ending in the binding name is not a call to it", () => {
const source = `import { configure } from "storybook/test";
reconfigure({ asyncUtilTimeout: 1000 });`;
assert.deepEqual(checkAsyncUtilSource(source, "storybook/test"), [
"does not call configure()",
]);
});