feat: queue concurrent local:gate runs on a machine-wide lease (#2339)
`npm run local:gate` now runs its stages under `scripts/gate-lease.mjs`, a
`proper-lockfile` lease under `os.tmpdir()` (or `$XDG_RUNTIME_DIR`), so a gate
started in a second worktree waits for the first instead of overlapping it.
Measured on v2/main @ aa56551b: a quiet gate is 257s green; two started
together had one fail at 279s (`smoke:web:chromium` found port 6298 held by
the other gate's smoke) and the survivor take 338s. Under the lease, three
gates started within seconds ran back to back (259s, then 262s, then the
third), each green with zero timeouts.
The lease grants rather than polls a threshold, so two waiters cannot
deadlock on each other; a holder that dies stops refreshing and is taken over
after 30s; a signal to the wrapper stops the whole process group, releases,
and exits 128+signal; the wait names the holder's pid and worktree and gives
up after 45 minutes. `INSPECTOR_SKIP_GATE_LEASE=1` bypasses it. A lock that
cannot be created runs the gate unleased with a warning.
`scriptChainRuns` looks through the wrapper (and only it) so the existing
vouches about what `local:gate` runs keep reading the real chain.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PsmUAZ6aLUAFqenjJKSSRp
Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall
parent
aa56551be2
commit
f99402918d
6 changed files with 849 additions and 12 deletions
401
scripts/gate-lease.mjs
Normal file
401
scripts/gate-lease.mjs
Normal file
|
|
@ -0,0 +1,401 @@
|
|||
#!/usr/bin/env node
|
||||
/**
|
||||
* Run a command under a machine-wide lease, so that concurrent `npm run
|
||||
* local:gate` runs in different worktrees queue instead of contending (#2339).
|
||||
*
|
||||
* **Two things go wrong when gates overlap, and both were measured** on
|
||||
* `v2/main` at `aa56551b` (M3, 8 logical cores). A quiet gate took 257s and
|
||||
* passed. Two started together in separate worktrees: one **failed** at 279s
|
||||
* — `smoke:web:chromium` found its fixed port (6298) held by the other gate's
|
||||
* copy of the same smoke — and the survivor took 338s, 1.3x, with the machine
|
||||
* half-free for its last minute. So the first problem is deterministic, not a
|
||||
* load effect: the web smokes bind fixed ports (6296–6299, plus the sandbox
|
||||
* and app-origin ports), so two gates reaching the same smoke together produce
|
||||
* a red run whose diff cannot have caused it — exactly the failure #2338
|
||||
* exists to stop. The second is load: with three or four sessions (#2323
|
||||
* measured sustained load averages of 50–70) every test budget is the wrong
|
||||
* number. Run back to back, the same two gates finish green in ~2x257s, and
|
||||
* each is measured against the quiet baseline every other #2338 aspect is
|
||||
* tuned to. So a second gate waits.
|
||||
*
|
||||
* **Why a lease and not a load threshold.** Two sessions each polling "is the
|
||||
* machine quiet yet?" deadlock on each other — neither ever clears, since each
|
||||
* is the reason the other is waiting (#2323). A lease *grants*: exactly one
|
||||
* waiter wins the `mkdir`, runs, and releases; the rest keep asking. There is
|
||||
* nothing to wait *for* except a release, and a release always comes — see the
|
||||
* crash case below.
|
||||
*
|
||||
* **Why `proper-lockfile`.** `core/auth/node/file-lock.ts` (#2082) records why
|
||||
* a hand-rolled election loses to it: `mkdir` is atomic, a live holder
|
||||
* refreshes the lock's mtime at `stale / 2` for as long as it lives, and a
|
||||
* holder that dies without releasing — a killed terminal, an OOM'd session —
|
||||
* stops refreshing, goes stale, and is taken over by the next waiter. That is
|
||||
* the whole "how does a crashed holder release" answer, and it is the
|
||||
* library's, not ours. The file-lock module's caveats about stale takeover
|
||||
* being non-single-winner apply here too and matter less: the worst case is
|
||||
* two gates running at once, which is exactly today's behaviour.
|
||||
*
|
||||
* **Why the lease is machine-wide.** The lock lives under `os.tmpdir()`
|
||||
* (`$XDG_RUNTIME_DIR` where a desktop session sets it), never inside a
|
||||
* worktree — a lock in the repo would be one per worktree, which is one per
|
||||
* session, which coordinates nothing.
|
||||
*
|
||||
* **Why default-on.** The sessions that need it are the ones that did not
|
||||
* think to opt in. `INSPECTOR_SKIP_GATE_LEASE=1` bypasses it, and the first
|
||||
* "waiting" line names the holder, its worktree, and that variable, so a wait
|
||||
* is never a mystery. It is also never a gate: a lock that cannot be *created*
|
||||
* (an unwritable tmpdir) runs the command unleased with a warning, because a
|
||||
* coordination aid must not acquire a new way to fail the gate it wraps.
|
||||
*
|
||||
* **Why the whole gate and not its parallel stages.** The serial stages are
|
||||
* not free — `lint` is type-aware and `build` is a full Vite bundle — and a
|
||||
* per-stage lease would let two gates interleave into the same 2.5x. One
|
||||
* lease around the whole run is the smallest thing that keeps the measured
|
||||
* baseline meaningful.
|
||||
*
|
||||
* Usage: `node scripts/gate-lease.mjs <command> [args...]`. The command runs
|
||||
* with inherited stdio in its own process group, so a signal to this process
|
||||
* stops the whole tree — `npm run` nests four deep and `sh` forwards nothing —
|
||||
* and the lease is released before this process exits with `128 + signal`.
|
||||
*/
|
||||
|
||||
import { spawn } from "node:child_process";
|
||||
import { mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { constants as osConstants, tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { fileURLToPath } from "node:url";
|
||||
// CJS-only package; a default import is the shape every loader agrees on.
|
||||
import properLockfile from "proper-lockfile";
|
||||
|
||||
/** Set (to anything but `0` or empty) to run without taking the lease. */
|
||||
export const SKIP_ENV = "INSPECTOR_SKIP_GATE_LEASE";
|
||||
|
||||
/** Overrides where the lease lives. Mostly a test seam; also a shared-box escape. */
|
||||
export const DIR_ENV = "INSPECTOR_GATE_LEASE_DIR";
|
||||
|
||||
/**
|
||||
* How long a lease may go unrefreshed before a waiter may take it over.
|
||||
*
|
||||
* `proper-lockfile` refreshes the lock's mtime at `stale / 2`, so this is not
|
||||
* "how long a gate may run" — it is how long after a holder *dies* the queue
|
||||
* stays blocked. 30s rather than the library's 10s default, on purpose: the
|
||||
* refresh is a timer in an otherwise idle process, and the cost of a timer
|
||||
* firing late (a laptop waking from sleep, an event loop starved by the very
|
||||
* load this exists to manage) is a false takeover — two gates running at once,
|
||||
* the thing being prevented — whereas the cost of the longer window is 20
|
||||
* more seconds of waiting after a crash, against a gate that takes minutes.
|
||||
*/
|
||||
export const STALE_MS = 30_000;
|
||||
|
||||
/** How often a waiter re-asks. The acquire is one `mkdir`; this is not a hot loop. */
|
||||
export const POLL_MS = 2_000;
|
||||
|
||||
/** How often a waiter says it is still waiting, so a long queue is visibly alive. */
|
||||
export const PROGRESS_MS = 60_000;
|
||||
|
||||
/**
|
||||
* How long a waiter waits before giving up.
|
||||
*
|
||||
* A dead holder releases within {@link STALE_MS}, so this only ever expires
|
||||
* against a *live* holder that is still running after 45 minutes — four
|
||||
* queued quiet gates are ~18 minutes, so a holder that long is a hung gate, and
|
||||
* the right outcome is a loud failure naming it rather than a fifth process
|
||||
* joining the pile. An unbounded wait would be a task that looks like progress
|
||||
* and can never succeed.
|
||||
*/
|
||||
export const MAX_WAIT_MS = 45 * 60_000;
|
||||
|
||||
/** Where the lease lives. `$XDG_RUNTIME_DIR` is per-user and per-session where it exists. */
|
||||
export function leaseDir(env = process.env) {
|
||||
if (env[DIR_ENV]) return path.resolve(env[DIR_ENV]);
|
||||
return path.join(env.XDG_RUNTIME_DIR || tmpdir(), "mcp-inspector-gate-lease");
|
||||
}
|
||||
|
||||
/**
|
||||
* The lease target: `proper-lockfile` locks `<target>.lock` beside it and
|
||||
* never opens the target itself, so the target doubles as the holder record
|
||||
* (pid, worktree, start time) a waiter prints. It need not exist to be locked
|
||||
* (`realpath: false` below).
|
||||
*/
|
||||
export function leaseTarget(dir) {
|
||||
return path.join(dir, "local-gate");
|
||||
}
|
||||
|
||||
/** The lock directory `proper-lockfile` creates for {@link leaseTarget}. */
|
||||
export function lockPathOf(dir) {
|
||||
return `${leaseTarget(dir)}.lock`;
|
||||
}
|
||||
|
||||
/** `INSPECTOR_SKIP_GATE_LEASE=0` and an empty value both mean "not skipped". */
|
||||
export function isSkipped(env = process.env) {
|
||||
const value = env[SKIP_ENV];
|
||||
return value !== undefined && value !== "" && value !== "0";
|
||||
}
|
||||
|
||||
/** `95s` → `1m35s`; sub-minute values stay in seconds. */
|
||||
export function formatDuration(ms) {
|
||||
const s = Math.max(0, Math.round(ms / 1000));
|
||||
if (s < 60) return `${s}s`;
|
||||
return `${Math.floor(s / 60)}m${String(s % 60).padStart(2, "0")}s`;
|
||||
}
|
||||
|
||||
/**
|
||||
* The exit code to report for a finished child: its own, or `128 + signal`
|
||||
* (the shell convention) when a signal ended it. `code` and `signal` are
|
||||
* mutually exclusive on a real ChildProcess exit.
|
||||
*/
|
||||
export function exitCodeFor({ code, signal }) {
|
||||
if (typeof code === "number") return code;
|
||||
return 128 + (osConstants.signals[signal] ?? 0);
|
||||
}
|
||||
|
||||
/** The holder record the current holder wrote, or `null` if unreadable or malformed. */
|
||||
export function readHolder(dir) {
|
||||
try {
|
||||
const record = JSON.parse(readFileSync(leaseTarget(dir), "utf8"));
|
||||
if (
|
||||
typeof record?.pid === "number" &&
|
||||
typeof record?.cwd === "string" &&
|
||||
typeof record?.startedAt === "number"
|
||||
) {
|
||||
return record;
|
||||
}
|
||||
} catch {
|
||||
// Missing, unreadable, or not JSON — all mean "nothing to report".
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** One line a waiter can act on: who holds it, where, and for how long. */
|
||||
export function describeHolder(holder, now = Date.now()) {
|
||||
if (holder === null) return "another local:gate";
|
||||
return `pid ${holder.pid} in ${holder.cwd}, running for ${formatDuration(now - holder.startedAt)}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Send `signal` to the child's whole process group (it was spawned as a group
|
||||
* leader), falling back to the child alone where groups are unavailable or the
|
||||
* group is already gone.
|
||||
*/
|
||||
function signalTree(child, signal) {
|
||||
try {
|
||||
if (process.platform !== "win32") {
|
||||
process.kill(-child.pid, signal);
|
||||
return;
|
||||
}
|
||||
} catch {
|
||||
// ESRCH: the group is already gone, or the child never became a leader.
|
||||
}
|
||||
try {
|
||||
child.kill(signal);
|
||||
} catch {
|
||||
// Already exited; nothing to signal.
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Take the lease, waiting for a holder to release or go stale.
|
||||
*
|
||||
* Returns the release function, or `null` when a lock cannot be created here
|
||||
* at all — the caller then runs unleased. Throws only when the wait budget is
|
||||
* exhausted against a live holder.
|
||||
*/
|
||||
async function acquireLease({ dir, log, pollMs, progressMs, maxWaitMs }) {
|
||||
const target = leaseTarget(dir);
|
||||
const startedWaiting = Date.now();
|
||||
let lastProgress = startedWaiting;
|
||||
let announced = false;
|
||||
for (;;) {
|
||||
try {
|
||||
return await properLockfile.lock(target, {
|
||||
realpath: false,
|
||||
stale: STALE_MS,
|
||||
retries: 0,
|
||||
// The library's default throws from a timer with no caller on the
|
||||
// stack, which would take the *holder* down mid-gate. A compromised
|
||||
// lease means another gate is now running alongside this one — worth
|
||||
// saying, not worth killing a gate that is otherwise fine.
|
||||
onCompromised: (err) =>
|
||||
log(
|
||||
`gate-lease: another process took the lease over while this gate was running (${err.message}); continuing without it.`,
|
||||
),
|
||||
});
|
||||
} catch (err) {
|
||||
if (err?.code !== "ELOCKED") {
|
||||
log(
|
||||
`gate-lease: could not take the lease at ${lockPathOf(dir)} (${err?.message ?? err}); running without it.`,
|
||||
);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
const waited = Date.now() - startedWaiting;
|
||||
if (!announced) {
|
||||
announced = true;
|
||||
log(
|
||||
`gate-lease: ${describeHolder(readHolder(dir))} holds the gate lease; waiting for it to finish so the two do not contend. ${SKIP_ENV}=1 runs anyway.`,
|
||||
);
|
||||
} else if (Date.now() - lastProgress >= progressMs) {
|
||||
lastProgress = Date.now();
|
||||
log(
|
||||
`gate-lease: still waiting (${formatDuration(waited)}) on ${describeHolder(readHolder(dir))}.`,
|
||||
);
|
||||
}
|
||||
if (waited >= maxWaitMs) {
|
||||
throw new Error(
|
||||
`gate-lease: gave up after ${formatDuration(waited)} — ${describeHolder(readHolder(dir))} still holds ${lockPathOf(dir)}. If that gate is hung, stop it; ${SKIP_ENV}=1 runs without the lease.`,
|
||||
);
|
||||
}
|
||||
await delay(pollMs);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Run `command args` holding the lease, and resolve with the exit code to
|
||||
* report. Signals to this process stop the child's whole tree first and are
|
||||
* reported as `128 + signal` after the lease is released.
|
||||
*
|
||||
* Every knob is injectable so the tests can drive a real lock, a real child
|
||||
* and a real signal without waiting real minutes.
|
||||
*
|
||||
* @param {object} opts
|
||||
* @param {string} opts.command
|
||||
* @param {string[]} opts.args
|
||||
* @param {string} [opts.dir] lease directory; default {@link leaseDir}
|
||||
* @param {NodeJS.ProcessEnv} [opts.env]
|
||||
* @param {(line: string) => void} [opts.log]
|
||||
* @param {number} [opts.pollMs]
|
||||
* @param {number} [opts.progressMs]
|
||||
* @param {number} [opts.maxWaitMs]
|
||||
* @param {number} [opts.graceMs] SIGTERM → SIGKILL escalation on a signal
|
||||
* @param {import("node:child_process").StdioOptions} [opts.stdio]
|
||||
* @returns {Promise<number>}
|
||||
*/
|
||||
export async function runUnderLease({
|
||||
command,
|
||||
args,
|
||||
dir = leaseDir(),
|
||||
env = process.env,
|
||||
log = (line) => console.error(line),
|
||||
pollMs = POLL_MS,
|
||||
progressMs = PROGRESS_MS,
|
||||
maxWaitMs = MAX_WAIT_MS,
|
||||
graceMs = 5_000,
|
||||
stdio = "inherit",
|
||||
}) {
|
||||
let release = null;
|
||||
let waited = 0;
|
||||
if (isSkipped(env)) {
|
||||
log(`gate-lease: ${SKIP_ENV} is set; running without the lease.`);
|
||||
} else {
|
||||
let usable = true;
|
||||
try {
|
||||
mkdirSync(dir, { recursive: true });
|
||||
} catch (err) {
|
||||
usable = false;
|
||||
log(
|
||||
`gate-lease: could not create ${dir} (${err?.message ?? err}); running without the lease.`,
|
||||
);
|
||||
}
|
||||
if (usable) {
|
||||
const startedWaiting = Date.now();
|
||||
release = await acquireLease({ dir, log, pollMs, progressMs, maxWaitMs });
|
||||
waited = Date.now() - startedWaiting;
|
||||
}
|
||||
if (release !== null) {
|
||||
if (waited >= pollMs) {
|
||||
log(`gate-lease: acquired after ${formatDuration(waited)}.`);
|
||||
}
|
||||
try {
|
||||
writeFileSync(
|
||||
leaseTarget(dir),
|
||||
JSON.stringify({
|
||||
pid: process.pid,
|
||||
cwd: process.cwd(),
|
||||
startedAt: Date.now(),
|
||||
}),
|
||||
);
|
||||
} catch {
|
||||
// Best effort: the record only improves a waiter's message.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const startedRunning = Date.now();
|
||||
const child = spawn(command, args, {
|
||||
stdio,
|
||||
env,
|
||||
// Its own group, so one signal reaches every descendant. See the header.
|
||||
detached: process.platform !== "win32",
|
||||
});
|
||||
|
||||
const signals = ["SIGINT", "SIGTERM", "SIGHUP"];
|
||||
let stoppedBy = null;
|
||||
let escalation = null;
|
||||
const onSignal = (signal) => {
|
||||
if (stoppedBy !== null) return;
|
||||
stoppedBy = signal;
|
||||
log(`gate-lease: received ${signal}; stopping the gate.`);
|
||||
signalTree(child, signal);
|
||||
escalation = setTimeout(() => {
|
||||
log(
|
||||
`gate-lease: the gate did not exit within ${graceMs}ms of ${signal}; sending SIGKILL.`,
|
||||
);
|
||||
signalTree(child, "SIGKILL");
|
||||
}, graceMs);
|
||||
};
|
||||
const handlers = signals.map((signal) => {
|
||||
const handler = () => onSignal(signal);
|
||||
process.on(signal, handler);
|
||||
return [signal, handler];
|
||||
});
|
||||
|
||||
const outcome = await new Promise((resolve, reject) => {
|
||||
child.once("error", reject);
|
||||
child.once("exit", (code, signal) => resolve({ code, signal }));
|
||||
}).finally(() => {
|
||||
clearTimeout(escalation);
|
||||
for (const [signal, handler] of handlers) process.off(signal, handler);
|
||||
});
|
||||
|
||||
if (release !== null) {
|
||||
// The record is only meaningful while the lock beside it is held; a
|
||||
// waiter reading a fresh lock must not be told about the previous holder.
|
||||
rmSync(leaseTarget(dir), { force: true });
|
||||
try {
|
||||
await release();
|
||||
} catch (err) {
|
||||
log(
|
||||
`gate-lease: could not release the lease (${err?.message ?? err}); it goes stale on its own after ${formatDuration(STALE_MS)}.`,
|
||||
);
|
||||
}
|
||||
log(
|
||||
`gate-lease: released after ${formatDuration(Date.now() - startedRunning)}${waited >= pollMs ? ` (waited ${formatDuration(waited)} first)` : ""}.`,
|
||||
);
|
||||
}
|
||||
return stoppedBy !== null
|
||||
? exitCodeFor({ code: null, signal: stoppedBy })
|
||||
: exitCodeFor(outcome);
|
||||
}
|
||||
|
||||
/** `node scripts/gate-lease.mjs <command> [args...]` */
|
||||
export async function main(argv = process.argv.slice(2)) {
|
||||
const [command, ...args] = argv;
|
||||
if (!command) {
|
||||
console.error("usage: node scripts/gate-lease.mjs <command> [args...]");
|
||||
return 2;
|
||||
}
|
||||
try {
|
||||
return await runUnderLease({ command, args });
|
||||
} catch (err) {
|
||||
console.error(err?.message ?? err);
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
if (
|
||||
process.argv[1] &&
|
||||
path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)
|
||||
) {
|
||||
process.exitCode = await main();
|
||||
}
|
||||
327
scripts/gate-lease.test.mjs
Normal file
327
scripts/gate-lease.test.mjs
Normal file
|
|
@ -0,0 +1,327 @@
|
|||
// Tests for the local:gate lease (#2339). The pure helpers are table-driven;
|
||||
// the lease itself is exercised against a REAL `proper-lockfile` lock, a real
|
||||
// child process and — for the signal path — a real signal to a real wrapper
|
||||
// process, each against a throwaway lease directory so no test can queue
|
||||
// behind (or block) a developer's actual gate. Run via `npm run test:scripts`
|
||||
// (node:test; the root has no vitest harness).
|
||||
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { spawn } from "node:child_process";
|
||||
import {
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
mkdtempSync,
|
||||
readFileSync,
|
||||
utimesSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
import {
|
||||
DIR_ENV,
|
||||
SKIP_ENV,
|
||||
STALE_MS,
|
||||
describeHolder,
|
||||
exitCodeFor,
|
||||
formatDuration,
|
||||
isSkipped,
|
||||
leaseDir,
|
||||
leaseTarget,
|
||||
lockPathOf,
|
||||
main,
|
||||
readHolder,
|
||||
runUnderLease,
|
||||
} from "./gate-lease.mjs";
|
||||
|
||||
const SCRIPT = fileURLToPath(new URL("./gate-lease.mjs", import.meta.url));
|
||||
|
||||
/** A fresh lease directory per test, so tests never contend with each other. */
|
||||
function freshDir() {
|
||||
return mkdtempSync(join(tmpdir(), "gate-lease-test-"));
|
||||
}
|
||||
|
||||
function collectLog() {
|
||||
const lines = [];
|
||||
return { lines, log: (line) => lines.push(line) };
|
||||
}
|
||||
|
||||
/** Run `node -e <source>` under the lease with quiet stdio and fast knobs. */
|
||||
function runNode(source, opts = {}) {
|
||||
return runUnderLease({
|
||||
command: process.execPath,
|
||||
args: ["-e", source],
|
||||
stdio: "ignore",
|
||||
pollMs: 25,
|
||||
progressMs: 60_000,
|
||||
...opts,
|
||||
});
|
||||
}
|
||||
|
||||
/** Poll until `predicate()` holds or `timeoutMs` passes. */
|
||||
async function waitFor(predicate, timeoutMs = 5_000) {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (!predicate()) {
|
||||
assert.ok(Date.now() < deadline, "condition not met in time");
|
||||
await delay(25);
|
||||
}
|
||||
}
|
||||
|
||||
function isAlive(pid) {
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
test("leaseDir: explicit override, then XDG_RUNTIME_DIR, then tmpdir", () => {
|
||||
assert.equal(leaseDir({ [DIR_ENV]: "/x/y" }), "/x/y");
|
||||
assert.equal(
|
||||
leaseDir({ XDG_RUNTIME_DIR: "/run/user/1" }),
|
||||
join("/run/user/1", "mcp-inspector-gate-lease"),
|
||||
);
|
||||
assert.equal(leaseDir({}), join(tmpdir(), "mcp-inspector-gate-lease"));
|
||||
// The lock is a sibling of the target, never inside it — a file inside the
|
||||
// lock directory would make its removal (and so stale takeover) fail.
|
||||
assert.equal(lockPathOf("/d"), `${leaseTarget("/d")}.lock`);
|
||||
});
|
||||
|
||||
test("isSkipped: unset, empty and `0` mean the lease is on", () => {
|
||||
for (const [env, expected] of [
|
||||
[{}, false],
|
||||
[{ [SKIP_ENV]: "" }, false],
|
||||
[{ [SKIP_ENV]: "0" }, false],
|
||||
[{ [SKIP_ENV]: "1" }, true],
|
||||
[{ [SKIP_ENV]: "true" }, true],
|
||||
]) {
|
||||
assert.equal(isSkipped(env), expected, JSON.stringify(env));
|
||||
}
|
||||
});
|
||||
|
||||
test("formatDuration: seconds under a minute, m/ss above", () => {
|
||||
for (const [ms, expected] of [
|
||||
[0, "0s"],
|
||||
[-500, "0s"],
|
||||
[4_400, "4s"],
|
||||
[59_499, "59s"],
|
||||
[60_000, "1m00s"],
|
||||
[95_000, "1m35s"],
|
||||
[3_605_000, "60m05s"],
|
||||
]) {
|
||||
assert.equal(formatDuration(ms), expected, String(ms));
|
||||
}
|
||||
});
|
||||
|
||||
test("exitCodeFor: the child's code, else 128 + signal", () => {
|
||||
assert.equal(exitCodeFor({ code: 0, signal: null }), 0);
|
||||
assert.equal(exitCodeFor({ code: 7, signal: null }), 7);
|
||||
assert.equal(exitCodeFor({ code: null, signal: "SIGTERM" }), 143);
|
||||
assert.equal(exitCodeFor({ code: null, signal: "SIGINT" }), 130);
|
||||
assert.equal(exitCodeFor({ code: null, signal: "SIGNOPE" }), 128);
|
||||
});
|
||||
|
||||
test("readHolder/describeHolder: a missing or malformed record is not an error", () => {
|
||||
const dir = freshDir();
|
||||
assert.equal(readHolder(dir), null);
|
||||
assert.equal(describeHolder(null), "another local:gate");
|
||||
writeFileSync(leaseTarget(dir), "not json");
|
||||
assert.equal(readHolder(dir), null);
|
||||
writeFileSync(leaseTarget(dir), JSON.stringify({ pid: "12", cwd: "/w" }));
|
||||
assert.equal(readHolder(dir), null);
|
||||
const record = { pid: 12, cwd: "/w", startedAt: 1_000 };
|
||||
writeFileSync(leaseTarget(dir), JSON.stringify(record));
|
||||
assert.deepEqual(readHolder(dir), record);
|
||||
assert.equal(
|
||||
describeHolder(record, 96_000),
|
||||
"pid 12 in /w, running for 1m35s",
|
||||
);
|
||||
});
|
||||
|
||||
test("two concurrent runs serialize: the second starts after the first ends", async () => {
|
||||
const dir = freshDir();
|
||||
const marks = join(dir, "marks.log");
|
||||
const { lines, log } = collectLog();
|
||||
const body = (label) =>
|
||||
`const fs=require("fs");const f=${JSON.stringify(marks)};` +
|
||||
`fs.appendFileSync(f,"${label} start "+Date.now()+"\\n");` +
|
||||
`setTimeout(()=>fs.appendFileSync(f,"${label} end "+Date.now()+"\\n"),300)`;
|
||||
const first = runNode(body("a"), { dir, log });
|
||||
// Give the first run the lock — and its holder record, which is written
|
||||
// just after — before the second asks, so which one waits, and what it is
|
||||
// told, are not themselves races the assertions have to allow for.
|
||||
await waitFor(() => existsSync(leaseTarget(dir)));
|
||||
const second = runNode(body("b"), { dir, log });
|
||||
assert.deepEqual(await Promise.all([first, second]), [0, 0]);
|
||||
|
||||
const marksByLabel = Object.fromEntries(
|
||||
readFileSync(marks, "utf8")
|
||||
.trim()
|
||||
.split("\n")
|
||||
.map((line) => line.split(" "))
|
||||
.map(([label, phase, at]) => [`${label} ${phase}`, Number(at)]),
|
||||
);
|
||||
assert.ok(
|
||||
marksByLabel["b start"] >= marksByLabel["a end"],
|
||||
`b started (${marksByLabel["b start"]}) before a ended (${marksByLabel["a end"]})`,
|
||||
);
|
||||
// The waiter said who it was waiting on, named the bypass, and reported the
|
||||
// wait when it finally acquired; the uncontended run said nothing on entry.
|
||||
assert.equal(
|
||||
lines.filter((l) => l.includes("holds the gate lease")).length,
|
||||
1,
|
||||
);
|
||||
assert.match(
|
||||
lines.find((l) => l.includes("holds the gate lease")),
|
||||
/pid \d+ in .+, running for/,
|
||||
);
|
||||
assert.match(
|
||||
lines.find((l) => l.includes("holds the gate lease")),
|
||||
new RegExp(`${SKIP_ENV}=1`),
|
||||
);
|
||||
assert.equal(
|
||||
lines.filter((l) => l.startsWith("gate-lease: acquired after")).length,
|
||||
1,
|
||||
);
|
||||
assert.equal(
|
||||
lines.filter((l) => l.startsWith("gate-lease: released after")).length,
|
||||
2,
|
||||
);
|
||||
assert.ok(!existsSync(lockPathOf(dir)), "the lock is released at the end");
|
||||
assert.ok(!existsSync(leaseTarget(dir)), "the holder record goes with it");
|
||||
});
|
||||
|
||||
test("the child's exit code is the run's exit code", async () => {
|
||||
const dir = freshDir();
|
||||
assert.equal(await runNode("process.exit(7)", { dir, log: () => {} }), 7);
|
||||
});
|
||||
|
||||
test("a stale lock (its holder died) is taken over, not waited on", async () => {
|
||||
const dir = freshDir();
|
||||
mkdirSync(lockPathOf(dir), { recursive: true });
|
||||
const dead = new Date(Date.now() - STALE_MS - 1_000);
|
||||
utimesSync(lockPathOf(dir), dead, dead);
|
||||
const { lines, log } = collectLog();
|
||||
assert.equal(await runNode("", { dir, log }), 0);
|
||||
assert.equal(
|
||||
lines.filter((l) => l.includes("holds the gate lease")).length,
|
||||
0,
|
||||
);
|
||||
assert.ok(!existsSync(lockPathOf(dir)));
|
||||
});
|
||||
|
||||
test("a live lock that never releases fails the wait loudly, naming the holder", async () => {
|
||||
const dir = freshDir();
|
||||
// A fresh lock directory nobody refreshes is indistinguishable from a live
|
||||
// holder for STALE_MS, which is longer than this test is prepared to wait.
|
||||
mkdirSync(lockPathOf(dir), { recursive: true });
|
||||
writeFileSync(
|
||||
leaseTarget(dir),
|
||||
JSON.stringify({
|
||||
pid: 424242,
|
||||
cwd: "/some/worktree",
|
||||
startedAt: Date.now(),
|
||||
}),
|
||||
);
|
||||
const { lines, log } = collectLog();
|
||||
await assert.rejects(runNode("", { dir, log, maxWaitMs: 0 }), (err) => {
|
||||
assert.match(err.message, /gave up after/);
|
||||
assert.match(err.message, /pid 424242 in \/some\/worktree/);
|
||||
assert.match(err.message, new RegExp(`${SKIP_ENV}=1`));
|
||||
return true;
|
||||
});
|
||||
assert.equal(
|
||||
lines.filter((l) => l.includes("holds the gate lease")).length,
|
||||
1,
|
||||
);
|
||||
// The lock was somebody else's; giving up must not remove it.
|
||||
assert.ok(existsSync(lockPathOf(dir)));
|
||||
});
|
||||
|
||||
test("a waiter reports progress while it waits", async () => {
|
||||
const dir = freshDir();
|
||||
mkdirSync(lockPathOf(dir), { recursive: true });
|
||||
const { lines, log } = collectLog();
|
||||
await assert.rejects(
|
||||
runNode("", { dir, log, maxWaitMs: 120, pollMs: 30, progressMs: 0 }),
|
||||
/gave up/,
|
||||
);
|
||||
assert.ok(
|
||||
lines.some((l) => l.startsWith("gate-lease: still waiting")),
|
||||
lines.join("\n"),
|
||||
);
|
||||
});
|
||||
|
||||
test(`${SKIP_ENV}=1 runs without touching the lease`, async () => {
|
||||
const dir = freshDir();
|
||||
const { lines, log } = collectLog();
|
||||
const env = { ...process.env, [SKIP_ENV]: "1" };
|
||||
assert.equal(await runNode("", { dir, log, env }), 0);
|
||||
assert.match(lines.join("\n"), new RegExp(`${SKIP_ENV} is set`));
|
||||
assert.ok(!existsSync(lockPathOf(dir)));
|
||||
});
|
||||
|
||||
test("a lease directory that cannot exist degrades to an unleased run", async () => {
|
||||
const parent = freshDir();
|
||||
const file = join(parent, "a-file");
|
||||
writeFileSync(file, "");
|
||||
const { lines, log } = collectLog();
|
||||
assert.equal(await runNode("", { dir: join(file, "nested"), log }), 0);
|
||||
assert.match(
|
||||
lines.join("\n"),
|
||||
/could not create .*; running without the lease/,
|
||||
);
|
||||
});
|
||||
|
||||
test("main: no command is a usage error", async () => {
|
||||
const { error } = console;
|
||||
const lines = [];
|
||||
console.error = (line) => lines.push(line);
|
||||
try {
|
||||
assert.equal(await main([]), 2);
|
||||
} finally {
|
||||
console.error = error;
|
||||
}
|
||||
assert.match(lines.join("\n"), /usage:/);
|
||||
});
|
||||
|
||||
test("a signal to the wrapper stops the whole tree, releases the lease and exits 128+signal", async () => {
|
||||
const dir = freshDir();
|
||||
const pidFile = join(dir, "grandchild.pid");
|
||||
// The gate stand-in: records its pid, then idles forever unless signalled.
|
||||
const idle =
|
||||
`require("fs").writeFileSync(${JSON.stringify(pidFile)}, String(process.pid));` +
|
||||
`setInterval(() => {}, 1000)`;
|
||||
const wrapper = spawn(
|
||||
process.execPath,
|
||||
[SCRIPT, process.execPath, "-e", idle],
|
||||
{
|
||||
env: { ...process.env, [DIR_ENV]: dir },
|
||||
stdio: ["ignore", "ignore", "pipe"],
|
||||
},
|
||||
);
|
||||
let stderr = "";
|
||||
wrapper.stderr.on("data", (chunk) => (stderr += chunk));
|
||||
const exited = new Promise((resolve) =>
|
||||
wrapper.once("exit", (code, signal) => resolve({ code, signal })),
|
||||
);
|
||||
|
||||
await waitFor(() => existsSync(pidFile));
|
||||
const grandchild = Number(readFileSync(pidFile, "utf8"));
|
||||
assert.ok(isAlive(grandchild));
|
||||
assert.ok(existsSync(lockPathOf(dir)), "the wrapper held the lease");
|
||||
|
||||
wrapper.kill("SIGTERM");
|
||||
const outcome = await exited;
|
||||
assert.deepEqual(outcome, { code: 143, signal: null }, stderr);
|
||||
await waitFor(() => !isAlive(grandchild));
|
||||
assert.ok(
|
||||
!existsSync(lockPathOf(dir)),
|
||||
"the lease was released on the way out",
|
||||
);
|
||||
assert.match(stderr, /received SIGTERM; stopping the gate/);
|
||||
});
|
||||
|
|
@ -50,6 +50,16 @@ export function reachableScripts(scripts, entry = "validate") {
|
|||
return reached;
|
||||
}
|
||||
|
||||
/** The lease wrapper `local:gate` runs under (#2339); see `scripts/gate-lease.mjs`. */
|
||||
export const GATE_LEASE_WRAPPER = "node scripts/gate-lease.mjs ";
|
||||
|
||||
/** `node scripts/gate-lease.mjs npm run X` → `npm run X`; anything else unchanged. */
|
||||
function unwrapGateLease(segment) {
|
||||
return segment.startsWith(GATE_LEASE_WRAPPER)
|
||||
? segment.slice(GATE_LEASE_WRAPPER.length).trim()
|
||||
: segment;
|
||||
}
|
||||
|
||||
/** The command strings of every script reachable from the root `validate`. */
|
||||
function rootReachedCommands(rootScripts) {
|
||||
return [...reachableScripts(rootScripts)]
|
||||
|
|
@ -108,6 +118,12 @@ export function rootReachesScript(rootScripts, scriptName) {
|
|||
* flags, as are npm's implicit `pre`/`post` hooks. Only the target is matched
|
||||
* exactly, which suits the argument-less scripts a vouch asks about.
|
||||
*
|
||||
* One wrapper is transparent: `node scripts/gate-lease.mjs npm run X` runs
|
||||
* `npm run X` under the local:gate lease (#2339) and nothing else, so a vouch
|
||||
* about what `local:gate` runs looks through it. It is matched by its exact
|
||||
* leading tokens — a wrapper with flags, or any other wrapper, still hides the
|
||||
* invocation, because this helper cannot know what an arbitrary wrapper does.
|
||||
*
|
||||
* @param {Record<string, string>} scripts
|
||||
* @param {string} entry
|
||||
* @param {string} target
|
||||
|
|
@ -124,7 +140,9 @@ export function scriptChainRuns(scripts, entry, target) {
|
|||
if (typeof scripts?.[hook] === "string") queue.push(hook);
|
||||
const body = scripts?.[name];
|
||||
if (typeof body !== "string") continue;
|
||||
for (const segment of body.split(/\n|;|&&/).map((part) => part.trim())) {
|
||||
for (const segment of body
|
||||
.split(/\n|;|&&/)
|
||||
.map((part) => unwrapGateLease(part.trim()))) {
|
||||
if (segment === `npm run ${target}`) return true;
|
||||
const tokens = segment.split(/\s+/);
|
||||
if (tokens[0] === "npm" && tokens[1] === "run" && tokens[2]) {
|
||||
|
|
|
|||
|
|
@ -23,7 +23,11 @@ import {
|
|||
findWorkflowViolations,
|
||||
formatWorkflowViolations,
|
||||
} from "./workflow-gate.mjs";
|
||||
import { reachableScripts } from "./npm-scripts.mjs";
|
||||
import {
|
||||
GATE_LEASE_WRAPPER,
|
||||
reachableScripts,
|
||||
scriptChainRuns,
|
||||
} from "./npm-scripts.mjs";
|
||||
|
||||
const repoRoot = join(import.meta.dirname, "..", "..");
|
||||
const workflowDir = join(repoRoot, ".github", "workflows");
|
||||
|
|
@ -607,7 +611,22 @@ describe("the gate's name", () => {
|
|||
|
||||
it("is `local:gate`, and runs the local-only Storybook step", () => {
|
||||
assert.ok(scripts["local:gate"], "the pre-push gate must exist");
|
||||
assert.match(scripts["local:gate"], /local:storybook/);
|
||||
assert.match(scripts["local:gate:stages"], /local:storybook/);
|
||||
});
|
||||
|
||||
it("runs its stages under the lease wrapper, and nothing else (#2339)", () => {
|
||||
// The wrapper serializes gates across worktrees. It must be the ONLY thing
|
||||
// `local:gate` does — a stage placed beside it would run outside the
|
||||
// lease — and it must run exactly the stages script, so every assertion
|
||||
// below about what the gate runs can read `local:gate:stages` directly.
|
||||
assert.equal(
|
||||
scripts["local:gate"],
|
||||
`${GATE_LEASE_WRAPPER}npm run local:gate:stages`,
|
||||
);
|
||||
assert.ok(
|
||||
scriptChainRuns(scripts, "local:gate", "local:storybook"),
|
||||
"the wrapper is transparent to a chain vouch",
|
||||
);
|
||||
});
|
||||
|
||||
describe("runs each client's test suite once, not twice (#2341)", () => {
|
||||
|
|
@ -631,8 +650,11 @@ describe("the gate's name", () => {
|
|||
);
|
||||
|
||||
it("starts from `local:validate`, and never from `validate`", () => {
|
||||
assert.match(scripts["local:gate"], /^npm run local:validate && /);
|
||||
assert.doesNotMatch(scripts["local:gate"], /\brun validate(?=$|[\s&;])/);
|
||||
assert.match(scripts["local:gate:stages"], /^npm run local:validate && /);
|
||||
assert.doesNotMatch(
|
||||
scripts["local:gate:stages"],
|
||||
/\brun validate(?=$|[\s&;])/,
|
||||
);
|
||||
// Nothing reachable from the WHOLE gate — not just its first stage —
|
||||
// may run `validate` or a client's `validate:*`: those are the doors
|
||||
// through which the bare `test` leg would come back, and a later stage
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@ import { join } from "node:path";
|
|||
import { describe, it } from "node:test";
|
||||
import { ENGINE_SMOKES } from "./run-engine-smokes.mjs";
|
||||
import { SUPPORTED_BROWSERS } from "./lib/headless-browser.mjs";
|
||||
import { scriptChainRuns } from "./lib/npm-scripts.mjs";
|
||||
|
||||
const scriptDir = import.meta.dirname;
|
||||
const scripts = JSON.parse(
|
||||
|
|
@ -73,7 +74,8 @@ describe("every engine tier consumes ENGINE_SMOKES", () => {
|
|||
|
||||
it("the Firefox tier goes through the runner and is in the pre-push gate", () => {
|
||||
assert.match(scripts["smoke:web:firefox"], /run-engine-smokes\.mjs/);
|
||||
assert.match(scripts["local:gate"], /smoke:web:firefox/);
|
||||
// Through the #2339 lease wrapper, which `scriptChainRuns` looks past.
|
||||
assert.ok(scriptChainRuns(scripts, "local:gate", "smoke:web:firefox"));
|
||||
});
|
||||
|
||||
it("every supported engine has a `smoke:web:<engine>` script", () => {
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@ import {
|
|||
ciRunsUnconditionally,
|
||||
runsCommand,
|
||||
} from "./verify-skills.mjs";
|
||||
import { scriptChainRuns } from "./lib/npm-scripts.mjs";
|
||||
import { GATE_LEASE_WRAPPER, scriptChainRuns } from "./lib/npm-scripts.mjs";
|
||||
|
||||
const SCRIPT = path.join(
|
||||
path.dirname(fileURLToPath(import.meta.url)),
|
||||
|
|
@ -491,6 +491,43 @@ test("scriptChainRuns follows the chain to a real invocation", () => {
|
|||
);
|
||||
});
|
||||
|
||||
test("scriptChainRuns looks through the local:gate lease wrapper, and only it", () => {
|
||||
// `local:gate` runs its stages under `scripts/gate-lease.mjs` (#2339). The
|
||||
// wrapper runs exactly the `npm run …` it is handed, so a vouch sees past
|
||||
// it; a wrapper carrying flags, or any other wrapper, still hides the chain.
|
||||
const T = "verify:skills:cli";
|
||||
const stages = `npm run validate && npm run ${T}`;
|
||||
assert.equal(
|
||||
scriptChainRuns(
|
||||
{
|
||||
"local:gate": `${GATE_LEASE_WRAPPER}npm run local:gate:stages`,
|
||||
"local:gate:stages": stages,
|
||||
},
|
||||
"local:gate",
|
||||
T,
|
||||
),
|
||||
true,
|
||||
);
|
||||
assert.equal(
|
||||
scriptChainRuns(
|
||||
{ "local:gate": `${GATE_LEASE_WRAPPER}npm run ${T}` },
|
||||
"local:gate",
|
||||
T,
|
||||
),
|
||||
true,
|
||||
);
|
||||
for (const wrapped of [
|
||||
`node scripts/gate-lease.mjs --quiet npm run ${T}`,
|
||||
`node scripts/other-wrapper.mjs npm run ${T}`,
|
||||
]) {
|
||||
assert.equal(
|
||||
scriptChainRuns({ "local:gate": wrapped }, "local:gate", T),
|
||||
false,
|
||||
wrapped,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test("scriptChainRuns is not satisfied by a mention or a longer name", () => {
|
||||
// `reachableScripts` extracts any `npm run …` substring, so both of these
|
||||
// passed the old check while the validator never ran.
|
||||
|
|
|
|||
Loading…
Reference in a new issue