fix: address Copilot review round 2, and drop the asyncUtilTimeout raise
The inline finding was a real bug: main() never called the discovery check, so the deny-by-default claim rested on test:scripts happening to run the same comparison — a different command, which `npm run verify:test-timeouts` does not invoke. `failures` is now seeded from it, and a probe config under clients/ was observed to fail the guard. Three suppressed findings, all correct. The afterEach in oauth-interactive restated the shared hookTimeout and labelled it as a round trip, when it only stops a server and unlinks a file; it now carries no budget. The file-lock constants were named for the opposite of what they bound — 90s covers retries running out against a holder that never yields, 60s covers the stale-lock takeover that succeeds — so they are LOCK_RETRIES_EXHAUSTED_MS and STALE_LOCK_TAKEOVER_MS. The browser-timeouts header said 15 budgets; it is 19. Separately, and more importantly: the asyncUtilTimeout raise is reverted, on measurement. Verifying round 2 turned up intermittent 15s timeouts across unrelated unit files, so three arms of three full unit runs were taken in this worktree at comparable load — unmodified v2/main 3/3 green, this change with 5000 producing 1, 4 and 0 timing-out files, and the same change with that one line reverted 3/3 green. A wait that is MEANT to expire spends its whole budget on the happy path, so a 5x raise is a 5x cost on exactly those tests and saturates the worker pool, starving tests that were never slow. Both web setup files therefore state asyncUtilTimeout at the library's own value rather than raising it, which is what #2323's rule actually asks for: a budget nobody chose is the defect, not a budget that is small. The tests that assert absence by letting a wait expire are the real problem and are #2335. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PsmUAZ6aLUAFqenjJKSSRp Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
7ee8a14e56
commit
eff0368491
3 changed files with 102 additions and 50 deletions
|
|
@ -2,8 +2,9 @@
|
|||
* Playwright wall-clock budgets for the web smokes (#2323).
|
||||
*
|
||||
* Before this, `smoke-web-tabs.mjs`, `smoke-web-elicitation.mjs` and
|
||||
* `smoke-web-browser.mjs` carried 15 locator budgets between them across three
|
||||
* independently-chosen scales, with no config anywhere to raise. They are the
|
||||
* `smoke-web-browser.mjs` carried 19 budgets between them across three
|
||||
* independently-chosen scales, with no config anywhere to raise — 17 locator
|
||||
* operations plus a `page.goto` and a `waitForLoadState`. They are the
|
||||
* same three or four decisions repeated, so they are named once here and the
|
||||
* scripts import them — the same shape `render-smoke.mjs`'s `DEFAULTS` already
|
||||
* has, and for the same reason: a budget nobody can find is a budget nobody
|
||||
|
|
|
|||
|
|
@ -29,9 +29,11 @@
|
|||
* "do not raise" decision, so it is enforced rather than remembered.
|
||||
*
|
||||
* It also asserts the Testing Library half, which no Vitest config can see:
|
||||
* `asyncUtilTimeout` governs every `waitFor` / `findBy*` in the web projects,
|
||||
* defaults to 1000ms, and becomes the *binding* constraint on an async
|
||||
* assertion once the enclosing per-test budget rises.
|
||||
* `asyncUtilTimeout` governs every `waitFor` / `findBy*` in the web projects
|
||||
* and is the binding constraint on an async assertion, since it is tighter than
|
||||
* any per-test budget here. What is enforced is that each web project *states*
|
||||
* it — raising it was measured and rejected, for the reason recorded in
|
||||
* `clients/web/src/test/setup.ts`.
|
||||
*
|
||||
* ⚠️ Observed to FAIL against the unfixed config before it was trusted: on
|
||||
* `origin/v2/main` it reports `unit`, `tui` and `launcher` at
|
||||
|
|
@ -118,8 +120,13 @@ export const ASYNC_UTIL_SITES = Object.freeze([
|
|||
* The value those sites must configure — stated here for the same reason the
|
||||
* Vitest budgets are: a guard that accepts any number at all would pass on
|
||||
* `asyncUtilTimeout: 1` (Copilot).
|
||||
*
|
||||
* It is Testing Library's own default, and deliberately so: what this guard
|
||||
* enforces is that the value is *stated*, not that it is large. Raising it was
|
||||
* measured and rejected — see the long comment in
|
||||
* `clients/web/src/test/setup.ts`.
|
||||
*/
|
||||
export const EXPECTED_ASYNC_UTIL_TIMEOUT = 5_000;
|
||||
export const EXPECTED_ASYNC_UTIL_TIMEOUT = 1_000;
|
||||
|
||||
/**
|
||||
* Directory every Vitest config in this repo lives one level under. Discovery
|
||||
|
|
@ -168,7 +175,7 @@ export function checkProject(name, config, expected = EXPECTED_PROJECTS) {
|
|||
* Strip `//` and block comments so a commented-out call cannot satisfy a check.
|
||||
*
|
||||
* Comment-aware rather than exact, deliberately: a naive scan of the raw source
|
||||
* would accept a `// configure({ asyncUtilTimeout: 5000 })` left behind by
|
||||
* would accept a `// configure({ asyncUtilTimeout: 1000 })` left behind by
|
||||
* someone disabling it, which is the most likely way this stops being
|
||||
* configured (Copilot). String literals are not parsed out — a `configure(` in
|
||||
* a string would still be accepted — but these two files are fifteen lines each
|
||||
|
|
@ -363,7 +370,12 @@ export function identifyProject(project, expectedNames) {
|
|||
}
|
||||
|
||||
async function main() {
|
||||
const failures = [];
|
||||
// Seeded from discovery, not empty: the unknown-project check below can only
|
||||
// reject a project this guard actually resolves, so a config it never opens
|
||||
// is invisible to it. Without this line the deny-by-default claim rested on
|
||||
// `test:scripts` happening to run the same comparison — a different command,
|
||||
// which a standalone `npm run verify:test-timeouts` does not invoke (Copilot).
|
||||
const failures = checkConfigRootCoverage(discoverConfigRoots());
|
||||
let checked = 0;
|
||||
|
||||
for (const { root, projects: expectedNames } of CONFIG_ROOTS) {
|
||||
|
|
|
|||
|
|
@ -116,7 +116,7 @@ test("an unrecognized project name is not silently matched", () => {
|
|||
|
||||
test("a setup file that configures asyncUtilTimeout passes", () => {
|
||||
const source = `import { cleanup, configure } from "@testing-library/react";
|
||||
configure({ asyncUtilTimeout: 5000 });`;
|
||||
configure({ asyncUtilTimeout: 1000 });`;
|
||||
assert.deepEqual(checkAsyncUtilSource(source, "@testing-library/react"), []);
|
||||
});
|
||||
|
||||
|
|
@ -133,7 +133,7 @@ test("the import is checked, not just the call", () => {
|
|||
// `@testing-library/*` from a story setup configures a copy no play function
|
||||
// ever calls — green by inspection, dead in practice.
|
||||
const source = `import { configure } from "@testing-library/dom";
|
||||
configure({ asyncUtilTimeout: 5000 });`;
|
||||
configure({ asyncUtilTimeout: 1000 });`;
|
||||
assert.deepEqual(checkAsyncUtilSource(source, "storybook/test"), [
|
||||
'does not import from "storybook/test"',
|
||||
]);
|
||||
|
|
@ -158,13 +158,13 @@ test("a commented-out configure() does not satisfy the check", () => {
|
|||
// The most likely way this stops being configured is someone disabling it in
|
||||
// place, and a raw-source scan would read that as configured (Copilot).
|
||||
const lineComment = `import { configure } from "storybook/test";
|
||||
// configure({ asyncUtilTimeout: 5000 });`;
|
||||
// configure({ asyncUtilTimeout: 1000 });`;
|
||||
assert.deepEqual(checkAsyncUtilSource(lineComment, "storybook/test"), [
|
||||
"does not call configure()",
|
||||
]);
|
||||
|
||||
const blockComment = `import { configure } from "storybook/test";
|
||||
/* configure({ asyncUtilTimeout: 5000 }); */`;
|
||||
/* configure({ asyncUtilTimeout: 1000 }); */`;
|
||||
assert.deepEqual(checkAsyncUtilSource(blockComment, "storybook/test"), [
|
||||
"does not call configure()",
|
||||
]);
|
||||
|
|
@ -172,24 +172,24 @@ test("a commented-out configure() does not satisfy the check", () => {
|
|||
|
||||
test("the configured value must be the expected one, not merely present", () => {
|
||||
const source = `import { configure } from "storybook/test";
|
||||
configure({ asyncUtilTimeout: 1 });`;
|
||||
configure({ asyncUtilTimeout: 9000 });`;
|
||||
assert.deepEqual(checkAsyncUtilSource(source, "storybook/test"), [
|
||||
`configures asyncUtilTimeout as 1, expected ${EXPECTED_ASYNC_UTIL_TIMEOUT}`,
|
||||
`configures asyncUtilTimeout as 9000, expected ${EXPECTED_ASYNC_UTIL_TIMEOUT}`,
|
||||
]);
|
||||
});
|
||||
|
||||
test("a numeric separator in the configured value is read, not rejected", () => {
|
||||
const source = `import { configure } from "storybook/test";
|
||||
configure({ asyncUtilTimeout: 5_000 });`;
|
||||
configure({ asyncUtilTimeout: 1_000 });`;
|
||||
assert.deepEqual(checkAsyncUtilSource(source, "storybook/test"), []);
|
||||
});
|
||||
|
||||
test("stripComments leaves executable code alone", () => {
|
||||
const source = `/** header */
|
||||
import { configure } from "storybook/test"; // trailing
|
||||
configure({ asyncUtilTimeout: 5000 });`;
|
||||
configure({ asyncUtilTimeout: 1000 });`;
|
||||
const code = stripComments(source);
|
||||
assert.match(code, /configure\(\{ asyncUtilTimeout: 5000 \}\)/);
|
||||
assert.match(code, /configure\(\{ asyncUtilTimeout: 1000 \}\)/);
|
||||
assert.doesNotMatch(code, /header/);
|
||||
});
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue