inspector/.claude/skills/testing/SKILL.md
cliffhall 0d947ad569 Review round 8: using a fixture, not connecting, is the trigger
Copilot, all three accepted.

1. Connecting still is not the condition. A few integration tests
   deliberately hand-roll a JSON-RPC server because the composable
   fixture *cannot* produce what they assert on —
   `inspectorClient-malformed-list.test.ts` says so in its own header:
   "the SDK's own server cannot emit these shapes: they are exactly what
   its types forbid". Same for `listSalvage-era.test.ts`. Real
   transport, real client, zero `test-servers/` dependency.

   Round 4 already stated the right condition at the top — "does this
   test depend on a fixture from `test-servers/`" — and then three
   downstream sites went on saying "connects". Those are now scoped to
   fixture use, and the hand-rolled exception is stated where the
   trigger is defined rather than left to be discovered, with both files
   named. Connecting is kept as a hint, since it usually does imply a
   fixture, with the instruction to check whether one can express the
   case first.

2. "description-strength measurement" was the wrong surface, and this
   PR is what made it wrong: a chained prompt is deliberately written so
   the target's *description* cannot fire on it, so the number measures
   the pointer in the first skill's body and nothing else. Left as-is it
   would send the next person tuning a red hand-off at the description.

3. "an 0.8 bar" -> "a 0.8 bar".

Re-measured: 100% / 100% hand-off, 63/63 first-move at 100%, RUNS=5.

Staged by explicit path, not `git add -A` — see ec6895a5.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
2026-09-05 18:03:37 -04:00

13 KiB

name description disable-model-invocation
testing Run, place and fix tests in this repo. Use when choosing which npm command runs a given suite (web unit, web integration, Storybook, cli, tui, launcher, scripts); when deciding where a new test file belongs — beside its source, under src/test/, or in a client's __tests__/; when a per-file coverage check fails or a v8 ignore is in question; when asking which test tier spawns the built binary rather than importing it; or when rendering, mounting or asserting on Mantine components and their transitions in a test. false

Testing

Every change needs tests, and every file must clear ≥90% on lines, statements, functions, and branches. That rule and the React/Mantine conventions live in AGENTS.md; this skill is where a test goes, how to run it, and how to clear the gate.

Before you write it: does the test need a real server?

If it does, load the test-servers skill now — that is step one, before choosing a location or writing a line.

The condition is "does this test depend on a fixture from test-servers/?" — not which tier it lands in, and not which directory it lands in. There are two ways to depend on one, and they need different halves of that skill:

  • It connects to a fixture. An integration test that connects; an end-to-end test that connects; a smoke that drives a connected flow; a coverage gap only reachable over a real connection; reproducing a reported bug against a server. These need the whole procedure — which showcase config, which protocol era, and the staleness hazard. ⚠️ Connecting is a strong hint, not the rule. A few integration tests deliberately hand-roll a JSON-RPC server because the composable fixture cannot produce what they assert on — inspectorClient-malformed-list.test.ts and listSalvage-era.test.ts need wire shapes the SDK's own server refuses to emit. Real transport, real client, no test-servers/ dependency. Check whether a fixture can express the case before reaching for one.
  • It names or runs the built fixture without connecting. smoke:tui boots the TUI against a catalog whose stdio command is the built fixture, then asserts it survives. No transport is driven and no protocol era applies, but the build and staleness half lands on it in full.

⚠️ "A build ran" is not the dependency — using the artefact is. clients/web's pretest runs test-servers:build before every unit run, so the fixture is on disk for tests that never reference it. What counts is whether the test imports, spawns, or points a config at it.

So the condition does not hold when the test renders a component from fixture props, exercises a pure function or a parser, or is a smoke that touches no fixture — smoke:launcher checks --help, and smoke:web / smoke:web:browser only assert the SPA is served and paints.

⚠️ Neither the tier nor the folder decides this. src/test/integration/ holds storage/store-id.test.ts, which validates a string, and mcp/import/*, which parses config files, right beside the tests that drive a live connection. They sit there for the node env and the 30s timeout, not because they connect — placement is the project manifest, so it cannot also be the fixture trigger. Ask what the test does, not where it lives.

In the connecting case, the test drives a real server over a real transport, never a mock, and picking the fixture, building it, and connecting with the right protocol era is a procedure this skill does not carry. Writing one without test-servers means hand-rolling a fixture that already exists, or mocking the thing the tier exists to avoid mocking. In the build-only case, none of the transport or protocol-era guidance applies — what you need from test-servers is how to build the fixture and why a stale build keeps serving old code.

Where the test file goes

Side-by-side by default; src/test/ only for what can't be co-located; and the Node clients are different.

clients/web — side-by-side

<Name>.test.tsx (or .test.ts for non-React modules) next to the source. Components, hooks, lib/, utils/. This is the overwhelming majority; a web-owned test living under src/test/ instead is a bug.

clients/web/src/test/ is for the three things that cannot be co-located:

  1. Tests of the repo-root core/ package → src/test/core/…, mirroring the core/ folder layout. core/ physically lives outside clients/web/, is consumed via the @inspector/core alias, and has no test harness of its own. This includes core/json/* and core/client/*.
  2. The integration project → src/test/integration/…, mirroring the core/ source layout (mcp/, mcp/node/, mcp/remote/, auth/, auth/node/, storage/). Placement is the manifest — any file under that folder is picked up by the integration project (node env, 30s timeouts) via a folder glob; there is no enumeration to keep in sync. ⚠️ Placement is not the fixture trigger, though — this folder holds pure parser and storage tests alongside the connecting ones. If the test you are adding here needs a fixture from test-servers/, load that skill first; the fixture is half of that test. Connecting is a strong hint but not the rule — see the hand-rolled-server exception above.
  3. Shared test infrastructure — renderWithMantine.tsx, setup.ts, fixtures/, scrollAreaStoryAssertions.ts.

clients/cli, clients/tui, clients/launcher — a top-level __tests__/

All their tests, not beside their source. Their tsconfig.json excludes **/*.test.* and their tsconfig.test.json includes __tests__/**/*, so a co-located src/**/*.test.* lands in no tsconfig project and fails npm run verify:typecheck-coverage (#1791).

Root tooling — scripts/*.test.mjs

A new scripts/*.mjs helper with pure logic gets a sibling *.test.mjs, run by npm run test:scripts (node's built-in runner; the root has no vitest harness by design). ⚠️ Keep the filename *.test.mjs — node --test silently skips a file its glob misses and still exits 0.

Running them

Scope From Command
Web unit clients/web npm run test (test:watch while iterating)
Web integration clients/web npm run test:integration
Web Storybook play fns clients/web npm run test:storybook
CLI clients/cli npm run test (pretest builds test-servers + the bin)
TUI clients/tui npm run test
Launcher clients/launcher npm run test
Root tooling repo root npm run test:scripts
Everything, fast repo root npm run validate
The coverage gate repo root npm run coverage

There is no aggregate root test script — each client self-validates.

In unit tests that expect error output, suppress it from the console.

The tiers, shallowest first

unit (test, per client) → web integration (test:integration, real transports/servers) → out-of-process (clients/cli/__tests__/e2e.test.ts, spawns the built binary) → smokes through the built launcher (npm run smoke) → Storybook play functions (test:storybook) → the published-tarball check (npm run pack:verify, local/release only — needs network).

⚠️ Depth in that list is not the fixture boundary, and the boundary cuts across the tiers rather than along them. Needing test-servers/: the web integration tests that drive one, the out-of-process CLI tests, the smokes that connect (smoke:cli, smoke:web:app, smoke:web:elicit, smoke:web:tabs), pack:verify, and smoke:tui — which never asserts a round trip but calls ensureTestServers({ requires: ["stdio"] }) and hands the built fixture to the TUI as its catalog's stdio command. Not needing it: the pure tests inside the same integration project, the connecting tests that deliberately hand-roll a server, smoke:launcher, smoke:web and smoke:web:browser (all three stop at boot without a fixture), and every Storybook play function (fixture props). Load the test-servers skill as soon as a task puts you on the fixture side of that line — whichever tier it sits in.

validate runs the per-client test scripts — so web unit plus cli's out-of-process e2e.test.ts, but not web's integration project, which runs inside the coverage gate. CI therefore has no separate test:integration step.

The coverage gate

Per-file ≥90 on all four dimensions, CI-enforced, across web, cli, tui and launcher. New code must clear 90 on every dimension.

Scope notes:

  • The web coverage include (in clients/web/vite.config.ts) also covers the shared core/ runtime the browser consumes — core/mcp, core/react, core/auth, core/storage, core/logging, core/node, core/json, core/client.
  • ⚠️ That include is a whitelist naming components/hooks/theme/ lib/utils/server. A module placed outside those directories falls out of the gate entirely, silently. The documented exceptions are src/App.tsx (a composition root at ~42% branch coverage — gating it is a dedicated decomposition effort) and the src/main.tsx / src/index.ts bootstraps.
  • CLI tests run in-process by importing runCli() (__tests__/helpers/cli-runner.ts) so src is measured; src/index.ts is the only exclusion. commander uses .exitOverride() so a parse error throws instead of tearing down the test worker.
  • TUI covers all of src/**, React surface included. Components mount through ink-testing-library with the passthrough doubles in __tests__/helpers/; keypresses are driven through stdin. The only exclusion is src/tui-servers.ts (a pure re-export, excluded so it doesn't surface as a misleading 0/0 row).

When a v8 ignore is justified

A genuinely-unreachable branch is annotated at the source rather than waved through by lowering the gate. The acceptable reasons are enumerated in AGENTS.md — do not reach for one that is not on that list.

React tests: renderWithMantine

Always render through renderWithMantine (src/test/renderWithMantine.tsx) — it wraps in MantineProvider with the project theme. Do not hand-roll a bare MantineProvider; it skips the project theme and the helper's options and drifts from every other test.

Note the justification has changed and the old one is wrong: it sets env="test", which makes Mantine skip the animated render — but it does not stop the timers. env is read only at Transition.mjs's render branch, while useTransition() runs before that check and still schedules real window.setTimeouts (opening a <Modal> schedules three 200ms timers). What prevents a timer outliving its file — and throwing an uncaught ReferenceError: window is not defined that fails the whole run, attributed to an innocent file (#1760) — is the leaked-timer safety net in src/test/setup.ts, which is global and covers every unit test however it renders. The rule stands on consistency, not on timer safety.

  • Forced color scheme: pass the option — renderWithMantine(ui, { colorScheme: "dark" }) — rather than hand-rolling defaultColorScheme="dark".
  • Mid-flight transition state (e.g. asserting a data-anim="out" cell during an exit crossfade) is the only reason to use renderWithMantineTransitions. Pass settleMs derived from the component's real animation duration (HEADER_ANIM_MS + 200), do not also use vi.useFakeTimers() in that test (the auto-settle no-ops under fake timers), and if the test unmounts the tree itself use the unmount() the helper returns. The mechanism is documented at length on the helper — read there before changing it.

Storybook play functions

Every screen and element component has a *.stories.tsx; play functions double as interaction tests, run headless in CI and in the local gate.

⚠️ expect(...) from storybook/test returns a promise. Storybook instruments it, so every expect in a play function is awaited — as is any shared helper that wraps one.

Test servers, not mocks

The tests that drive MCP behaviour over a transport use a real server rather than a mock, and for the ones that get that server from test-servers/, load the skill and use all of it: which showcase config covers the feature, which protocol era to connect with, how to add a combination that does not exist yet, and why a fixture can keep serving stale code after an edit.

A test that only names the built fixture needs that skill too, for a narrower reason. smoke:tui boots the TUI against a catalog whose stdio command is the build output and asserts it survives — it opens no transport, so config choice and protocol era do not apply to it, but building the fixture and the staleness hazard do. Load the skill and take that half.

A pure test that happens to live in the integration project, and a smoke that references no fixture, need neither (see the tier list above) — and note that clients/web's pretest builds test-servers/ before every unit run, so its presence on disk says nothing about whether your test depends on it.