fix: address Copilot review round 4
- Reword the CI guarantee in AGENTS.md, docs/quality-gate.md and the pre-push-gate skill: a green gate means every check CI applies has passed here — the strongest predictor of green CI, not a proof — and name the residual (a test that passes only when slowed) as #1596's class to fix rather than headroom to keep. - docs/quality-gate.md: cli's validate DID build, through `test`'s `pretest` hook; say that the build goes with the dropped `test` leg and `coverage:cli` builds once, later — in both places. - Update the two source comments that still described `local:gate` as running `validate` (verify-skills.mjs, verify-build-gate.mjs). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PsmUAZ6aLUAFqenjJKSSRp Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall
parent
ae9ba01bb6
commit
9cff77e6d9
5 changed files with 12 additions and 9 deletions
|
|
@ -32,8 +32,10 @@ answer.
|
|||
|
||||
It runs **every check** GitHub CI runs (which additionally runs `npm install`,
|
||||
and runs `coverage` as a parallel job), plus two local-only steps. So the
|
||||
direction that matters holds: **passing `local:gate` locally means CI's gates
|
||||
will pass.** The reverse does not.
|
||||
direction that matters holds: **passing `local:gate` locally means every check
|
||||
CI applies has already passed on your machine** — the strongest predictor of a
|
||||
green CI there is here, though not a proof (a different OS, and the bare test
|
||||
pass noted below). The reverse does not hold at all.
|
||||
|
||||
One difference in *invocations*, not checks: CI runs each client's unit suite
|
||||
twice — bare inside `validate`, instrumented inside `coverage` — on two
|
||||
|
|
|
|||
|
|
@ -473,7 +473,7 @@ free to run a full `local:gate`). Raising a budget nobody chose hides no race.
|
|||
## Mandatory pre-push gate
|
||||
|
||||
- **ALWAYS run `npm run format` before committing.** The **root** `format` auto-fixes `core/`, the root `scripts/` tooling, the root shared surface, and every client's scope in one shot. `validate` runs the non-fixing `format:check` and will fail in CI on any unformatted file, so run the auto-fixer first rather than letting `format:check` catch it.
|
||||
- **`npm run local:gate` is the mandatory pre-push command.** It runs **every check** `.github/workflows/main.yml` runs, plus two local-only steps, so passing it locally means CI's gates will pass. Expect several minutes.
|
||||
- **`npm run local:gate` is the mandatory pre-push command.** It runs **every check** `.github/workflows/main.yml` runs, plus two local-only steps, so a green run here is the strongest predictor of a green CI this repo has: every check CI applies has already passed on your machine. It is not a proof — CI runs on a different OS and, since #2341, runs each client's suite bare where the gate runs it only instrumented — so the one residual is a test that passes *only* when slowed down, which is a race (#1596) to fix, never headroom to keep. Expect several minutes.
|
||||
- **The gate runs each client's test suite once, instrumented; CI runs it twice (#2341).** CI's `build` job runs the bare `test` inside `validate` and its parallel `coverage` job runs `test:coverage`, which costs CI no wall clock. Serially in one process the bare pass was ~80s of a ~370s gate, re-running exactly the files the coverage pass runs a few minutes later — so the gate calls **`local:validate`**, which is `validate` minus each client's `test` leg (every client's `validate` is `check && test`, and `local:validate` runs the `check` half). "Every check" is therefore a claim about checks, not invocations, and it holds because `@vitest/coverage-v8` collects coverage from V8's own profiler (`Profiler.takePreciseCoverage`) and rewrites no source: a test file sees identical code either way, and the instrumented run is only *slower*, which makes it the stricter of the two for the failure class this repo actually sees (a correct test cut off under load). A test that passed only *because* it ran slower would be a race — #1596's class, a defect wherever it surfaces — and CI's bare pass still runs it. **`npm run validate` is unchanged**, because CI runs it directly and it is the inner-loop check; `verify:*` guards that ask "is this reachable from `validate`" are unaffected. `local:validate` lives in the `local:` namespace so the workflow guard keeps it out of CI by construction.
|
||||
- **`npm run validate` is the fast inner-loop check and is NOT an acceptable substitute.** It runs `test`, not `test:coverage`, so it does **zero** coverage gating, no smokes, and no Storybook tests. Skipping the gate is how a push passes every fast local check and still fails CI.
|
||||
- There is deliberately **no `npm run ci`** — that name collided with the `npm ci` built-in, which clean-installs from the lockfile and does not run this script.
|
||||
|
|
|
|||
|
|
@ -21,9 +21,9 @@ The local gate runs **every check** CI runs, and is not a mirror. Two of its ste
|
|||
| `smoke:web:firefox` | Trialled as a CI job and removed ([#2086](https://github.com/modelcontextprotocol/inspector/issues/2086)): across a dozen runs it never once disagreed with Chromium, and `playwright install --with-deps` carries a real flake surface. Kept in front of a human about to push instead. `smoke:web:webkit` is worse still — it fails two of the three smokes for reasons nobody has identified — so it is in neither tier. See [Supported browsers](#supported-browsers). |
|
||||
| `smoke:tui` | The Ink TUI needs a real TTY. It _is_ invoked in CI via `npm run smoke` and self-skips there on `process.env.CI`, so it needs no guarding — it handles itself. |
|
||||
|
||||
And one CI *invocation* the gate deliberately does not repeat: CI runs each client's unit suite **twice** — bare inside `validate` in the `build` job, instrumented inside `coverage` in the parallel job — at no wall-clock cost, because the two jobs run on separate runners. Serially in one process that second pass was ~80s of a ~370s gate, re-running exactly the test files the coverage pass runs a few minutes later ([#2341](https://github.com/modelcontextprotocol/inspector/issues/2341)). So the gate's first stage is **`local:validate`**: the same guards and `validate:core`, then each client's `check` — `format:check` + `lint` + `typecheck`, plus `build` where the client's `validate` ran one (web, tui, launcher; cli's binary is built by `coverage:cli`, as it always was) — instead of its `validate` (`check` + `test`). The suites still run once, under `coverage`. That subsumes the bare pass because `@vitest/coverage-v8` reads V8's own precise-coverage profiler and rewrites nothing — the code under test is byte-identical, and the instrumented run is only slower, which is the stricter direction for the failures this repo sees. `npm run validate` and both CI jobs are unchanged.
|
||||
And one CI *invocation* the gate deliberately does not repeat: CI runs each client's unit suite **twice** — bare inside `validate` in the `build` job, instrumented inside `coverage` in the parallel job — at no wall-clock cost, because the two jobs run on separate runners. Serially in one process that second pass was ~80s of a ~370s gate, re-running exactly the test files the coverage pass runs a few minutes later ([#2341](https://github.com/modelcontextprotocol/inspector/issues/2341)). So the gate's first stage is **`local:validate`**: the same guards and `validate:core`, then each client's `check` — `format:check` + `lint` + `typecheck`, plus `build` for web, tui and launcher, which name one explicitly; cli's `validate` built only through `test`'s `pretest` hook, so dropping the `test` leg drops that build too, and `coverage:cli` builds the binary once, later — instead of its `validate` (`check` + `test`). The suites still run once, under `coverage`. That subsumes the bare pass because `@vitest/coverage-v8` reads V8's own precise-coverage profiler and rewrites nothing — the code under test is byte-identical, and the instrumented run is only slower, which is the stricter direction for the failures this repo sees. `npm run validate` and both CI jobs are unchanged.
|
||||
|
||||
So the direction that matters holds: **passing `npm run local:gate` means CI's gates will pass.** The reverse does not.
|
||||
So the direction that matters holds: **passing `npm run local:gate` means every check CI applies has already passed on your machine** — the strongest predictor of a green CI this repo has, though not a proof: CI runs on a different OS, and the bare pass above is the one invocation it has that the gate does not, so a test that passes *only* when instrumentation slows it down would surface there first. That test is a race ([#1596](https://github.com/modelcontextprotocol/inspector/issues/1596)) to fix, not a reason to put the second pass back. The reverse direction does not hold at all: CI green says nothing about the Firefox pass or the real `smoke:tui`.
|
||||
|
||||
**The gate is deliberately not named `ci`** ([#2146](https://github.com/modelcontextprotocol/inspector/issues/2146)). That name collided with `npm ci` — a built-in that clean-installs from the lockfile, does **not** run this script, and is one keystroke away — and it invited the cross-engine passes into GitHub CI by looking like the thing a workflow ought to call. There is **no back-compat alias**: `npm run ci` now fails with npm's missing-script error, which prints the available scripts.
|
||||
|
||||
|
|
@ -44,7 +44,7 @@ That is the readable half, and prose rots. The enforced half is `scripts/lib/wor
|
|||
| `npm run verify:dep-lockstep` | Guards the "one version per install-crossing dependency" invariant (#1896). v2 is not a workspace, so a client's test project compiles the shared first-party TypeScript — `core/`, `test-servers/src`, and the root-owned `vitest.shared.mts`, all of which resolve their dependencies from the **root** install — alongside the client's own sources, putting the same package in one `tsc` program twice. At the same version that's harmless; skewed, TypeScript must relate two structurally-distinct copies of every type, which for a recursive-generic surface is exponential (zod `4.3.6` vs `4.4.3` exhausted the 4GB tsc heap in `clients/web`). Derives its candidate set from **what actually enters each program** (#1965) — every client tsconfig project listed with `tsc --listFilesOnly` via the shared `scripts/lib/tsc-program.mjs`, each resolved `node_modules` file mapped to its owning install, keeping the packages that reach one program from two installs (a package whose declarations arrive only through another package's `.d.ts`, as `@modelcontextprotocol/sdk`'s do, is invisible to a scan of first-party imports). Prices each copy from the lockfile entry for the exact install path the program resolved, compares only the installs that met in one program, and **fails deny-by-default** on any disagreement not in the annotated `TOLERATED_SKEW` allowlist — empty today — with an allowlisted package tolerated only *within a major version*. A **second tier** (#2226) runs alongside it, asking the weaker but broader question the `AGENTS.md` rule actually states: does a package this repo *declares* anywhere resolve to two versions across our installs at all? Its candidate set is every name in any install's `dependencies`/`devDependencies`/`optionalDependencies` — unioned across the root and all four clients, so a copy declared by only one of them still counts — that **more than one install holds a top-level copy of** (17 packages today). Nested copies are excluded: one exists because some dependency asked for a different version, so it is that dependency's range to govern, not ours. Neither tier subsumes the other — the program tier sees a copy no manifest names (`@modelcontextprotocol/sdk`, arriving through another package's `.d.ts`), while the declared tier sees a **transitive** copy no program loads (cli's `@types/node`, hoisted via `@types/express` — the case that motivated it), two **clients** disagreeing with no root copy involved (`@types/react`, web against tui), and the peer shadows `eslint`/`typescript`/`vitest` that never enter a program. Same deny-by-default and same within-a-major rule, against its own `TOLERATED_DECLARED_SKEW` — also empty. Two limits: it reads **lockfiles**, so an uncommitted hand-installed copy is invisible, and it compares only declared names, so a purely transitive package no manifest names stays the first tier's business. Runs in `validate`.
|
||||
| `npm run verify:test-timeouts` | Guards the wall-clock budgets the test gates run under (#2323). The class it encodes against is a budget **nobody chose**: three of the six Vitest projects ran on Vitest's own `testTimeout: 5000` and five on its `hookTimeout`/`teardownTimeout: 10000`, sized for an idle machine rather than the one this team works on — three or four concurrent agent sessions in separate worktrees, each free to run a full `local:gate`, on eight logical cores. A correct, deterministic test cut off by such a budget fails a gate its diff did not break, which is the same "channel nobody trusts" failure [Lint has no warning tier](../AGENTS.md#lint-has-no-warning-tier) describes from the other direction; #2292, #1942 and #1742 were each an instance, found one site at a time. Deliberately **not** cited: #2278 (a missing condition wait around a geometry read) and #2250 (a real race in a test's own timing) were fixed by making the test wait for the right thing, and presenting a race fix as evidence for a larger ceiling would argue against #1596. It asks **Vitest itself** to resolve each of the four configs and reads the number a test actually gets, rather than checking that a key is absent from a config block — which would pass just as happily on a config that had stopped being loaded. A **seventh project** with no row, or a config file it does not discover (it knows all twelve filenames Vitest accepts, not just the two this repo uses), is an error rather than a silent skip. It also checks that every project loads `vitest.setup.shared.mts`. ⚠️ **Everything it reads comes from a resolved project, never from source text** — the two rules no config can report are asserted at runtime instead: `retry` by `vitest.setup.shared.mts` (which reads the value Vitest resolved, so a per-test option, a `describe` option, a project setting and a `--retry` flag are one check) and Testing Library's `asyncUtilTimeout` by `clients/web/src/test/asyncUtilTimeout.test.ts` (which reads what the project's own `waitFor`s use). Both began as source scanning in #2334 and the review found a new valid spelling missed in five consecutive rounds, so the boundary is deliberate: if a rule cannot be answered by asking the tool, assert it at runtime rather than reading the source for it. Observed **red** against the pre-#2323 config before it was trusted. Runs in `validate`; its decision logic has a sibling `verify-test-timeouts.test.mjs` under `test:scripts`.
|
||||
| `npm run local:gate` | **Mandatory pre-push command.** `local:validate` → `verify:skills:cli` → `coverage` → `verify:build-gate` → `verify:bundle-externals` → `smoke` → `smoke:web:firefox` → `local:storybook`. Every GitHub CI check plus two local-only ones — see [Two tiers](#two-tiers-github-ci-and-the-local-gate). Named `local:` rather than `ci` on purpose (#2146); there is no `npm run ci` alias. |
|
||||
| `npm run local:validate` | The gate's first stage: `validate` with each client's `test` leg removed (#2341) — the same `validate:guards` and `validate:core`, then every client's `check` — `format:check` + `lint` + `typecheck`, plus `build` for web, tui and launcher (cli has never built in `validate`; `coverage:cli` builds its binary) — each client's `validate` being `check && test`. The suites run once, under `coverage`, instead of bare here and instrumented there. In the `local:` namespace so the workflow guard keeps it out of CI, where the bare pass is free. |
|
||||
| `npm run local:validate` | The gate's first stage: `validate` with each client's `test` leg removed (#2341) — the same `validate:guards` and `validate:core`, then every client's `check` — `format:check` + `lint` + `typecheck`, plus `build` for web, tui and launcher, which name one explicitly (cli's only validate-time build was `test`'s `pretest` hook, so it goes with the `test` leg and `coverage:cli` builds the binary once, later) — each client's `validate` being `check && test`. The suites run once, under `coverage`, instead of bare here and instrumented there. In the `local:` namespace so the workflow guard keeps it out of CI, where the bare pass is free. |
|
||||
| `npm run pack:verify` | Publish smoke — see [Publishing](./publishing.md). |
|
||||
|
||||
Per-client scripts exist too (`validate:web`, `coverage:cli`, `smoke:tui`, …), plus root `validate:core` / `format:core` for the shared `core/` package, `format:scripts` for the root `scripts/` tooling, and `format:shared` / `lint:shared` for the root "shared" surface (`test-servers/src/**`, `vitest.shared.mts`, the root `eslint.config.js`). Run `npm run format` before committing — the root `format` fixes `core/`, the root `scripts/`, the shared surface, and every client; `validate` runs the non-fixing `format:check` and fails CI on any unformatted file.
|
||||
|
|
|
|||
|
|
@ -350,8 +350,9 @@ if (!output.includes(ERROR_PREFIX)) {
|
|||
// so PROBE_MODULE must appear. Without this, a repo that already leaks a built-in
|
||||
// would report OK even if the probe was tree-shaken or entryPath went stale —
|
||||
// branch (c)'s failure, silently inverted into a pass. (Unreachable inside
|
||||
// `npm run local:gate`: `validate`'s `build:web` fails first on a pre-existing leak — but
|
||||
// this script is a documented standalone command, run exactly when debugging one.)
|
||||
// `npm run local:gate`: web's `check` — its `build`, run by `local:validate` —
|
||||
// fails first on a pre-existing leak; but this script is a documented
|
||||
// standalone command, run exactly when debugging one.)
|
||||
if (!output.includes(PROBE_MODULE)) {
|
||||
fail(
|
||||
`the #1769 gate fired, but ${PROBE_MODULE} isn't among the offenders — it ` +
|
||||
|
|
|
|||
|
|
@ -330,7 +330,7 @@ function main(argv = process.argv.slice(2)) {
|
|||
// The authoritative parse, when it is available — strictly a bonus, and only
|
||||
// from the EXACT pinned CLI.
|
||||
//
|
||||
// `local:gate` runs `validate` (and therefore this) before the pinned
|
||||
// `local:gate` runs `local:validate` (and therefore this) before the pinned
|
||||
// `verify:skills:cli` step, so a local CLI on any other version could reject
|
||||
// skills that CI's pinned validator accepts — and the gate would exit here,
|
||||
// never reaching the reproducible step (Copilot). Accepting merely "new enough"
|
||||
|
|
|
|||
Loading…
Reference in a new issue