test(cli,tui): add in-process source coverage gates (#1484) (#1502)

* test(cli,tui): add in-process source coverage gates (#1484)

CLI and TUI source had no coverage gate: CLI tests spawned the built
binary out-of-process (coverage never saw clients/cli/src) and the TUI
had only two smoke tests. Regressions there were invisible to CI.

CLI:
- Rewrite __tests__/helpers/cli-runner.ts to invoke runCli() in-process
  (capture process.std*.write, map thrown errors to an exit code),
  preserving the CliResult shape so all existing tests run unchanged and
  clients/cli/src is now measured under v8 instrumentation.
- cli.ts: program.exitOverride() so a commander usage error throws
  (catchable in-process) instead of tearing down the worker; --help /
  --version still exit cleanly. Drop the dead, never-imported main().
- Add a thin out-of-process e2e.test.ts that spawns the built binary for
  the shebang / process.exit paths the in-process runner can't reach.
- Add @vitest/coverage-v8, a per-file gate (90/85/80/50) excluding only
  src/index.ts (binary bootstrap), and pool: 'forks' for capture
  isolation. cli.ts lands at ~98% lines / 100% functions.

TUI:
- Add @vitest/coverage-v8 and an interim per-file gate over the
  non-React logic (logger.ts, components/tabsConfig.ts, utils/*); the
  Ink components, App.tsx, and hooks/ are excluded with a documented
  follow-up (#1501). Add tests covering the included modules (100%).

Wiring + docs:
- CI runs each client's test:coverage; root test:cli / test:tui (used by
  validate) and root test:coverage now enforce the gates.
- gitignore clients/**/coverage; update AGENTS.md and client READMEs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(cli): clarify in-process runner footguns from PR #1502 review

Review notes on the in-process CLI test runner (no behavior change):
- Warn in runCli's docblock that --help/--version must NOT go through the
  in-process runner: exitOverride only re-throws non-zero exits, so exitCode
  0 falls through to process.exit(0) and would kill the vitest worker. Those
  belong in the out-of-process e2e.test.ts layer.
- Document that a timeout orphans the uncancellable invokeCli promise (its
  stdio subprocess stays alive; later writes hit the restored stdout) — a
  known, acceptable trade-off on the hang-only failure path.
- Fix a stale reference to the removed main() (errors now route through
  index.ts).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(cli): fix remaining stale main() reference in vitest exclusion comment

PR #1502 re-review caught a second stale `main()` mention (the first was
fixed in 52c23e41). src/index.ts inlines the bootstrap as
runCli(process.argv).then(...).catch(handleError) — there is no main().
Comment-only; no behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: compose root test:coverage from per-client scripts

Replace the brittle inline cd-chain with a composition of the existing
test:cli / test:tui scripts plus a new matching test:web. Behavior is
unchanged (web → cli → tui coverage gates) but there's a single source of
truth for each client's coverage invocation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Cliff Hall 2026-06-20 15:05:58 -04:00 • committed by GitHub
commit 388ea5a7bd
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

View file

@ -3,10 +3,13 @@
## Running Tests
```bash
# Run all tests
# Run all tests (pretest builds test-servers + the CLI binary)
npm test
# Run in watch mode (useful for test file changes; won't work on CLI source changes without rebuild)
# Run all tests under the per-file coverage gate
npm run test:coverage
# Run in watch mode (useful for test file changes)
npm run test:watch
# Run specific test file
@ -16,29 +19,51 @@ npm run test:cli-headers # headers.test.ts
npm run test:cli-metadata # metadata.test.ts
```
## How the CLI is exercised
Most tests run the CLI **in-process**: `helpers/cli-runner.ts` imports `runCli()`
from `../src/cli.ts` and invokes it directly, capturing `process.stdout`/`stderr`
and mapping thrown errors to an exit code. This is what lets vitest's v8 coverage
instrument `clients/cli/src` (#1484). Because `commander` uses `.exitOverride()`,
a parse/usage error throws instead of calling `process.exit` and tearing down the
worker.
`e2e.test.ts` is the deliberately thin **out-of-process** layer: it spawns the
**built** `build/index.js` as a real subprocess to cover the shebang, the
`index.ts` `isMain` bootstrap, and the actual `process.exit` codes — the only
parts the in-process runner can't reach. (`scripts/smoke-cli.mjs` at the repo
root provides a further end-to-end check of the binary.)
## Test Files
- `cli.test.ts` - Basic CLI functionality: CLI mode, environment variables, config files, resources, prompts, logging, transport types
- `tools.test.ts` - Tool-related tests: Tool discovery, JSON argument parsing, error handling, prompts
- `headers.test.ts` - Header parsing and validation
- `metadata.test.ts` - Metadata functionality: General metadata, tool-specific metadata, parsing, merging, validation
- `methods.test.ts` - Method/option-validation paths not covered elsewhere (resource templates, missing/invalid options, the `--` target separator)
- `error-handler.test.ts` - The binary's `handleError` error sink, exercised in-process with `process.exit` stubbed
- `e2e.test.ts` - Out-of-process spawn of the built binary (exit codes + boot)
## Helpers
The `helpers/` directory contains shared utilities:
- `cli-runner.ts` - Spawns CLI as subprocess and captures output
- `test-mcp-server.ts` - Standalone stdio MCP server script for stdio transport testing
- `instrumented-server.ts` - In-process MCP test server for HTTP/SSE transports with request recording
- `cli-runner.ts` - Invokes `runCli()` in-process and captures stdout/stderr + exit code
- `assertions.ts` - Custom assertion helpers for CLI output validation
- `fixtures.ts` - Test config file generators and temporary directory management
- `fixtures.ts` - Test config/catalog file generators and temporary directory management
HTTP/SSE test servers (for transport tests) come from the
`@modelcontextprotocol/inspector-test-server` package (`createTestServerHttp`),
and the bundled stdio server is launched via `getTestMcpServerCommand()`.
## Notes
- Tests run in parallel across files (Vitest default)
- Tests within a file run sequentially (we have isolated config files and ports, so we could get more aggressive if desired)
- Config files use `crypto.randomUUID()` for uniqueness in parallel execution
- Test files run in separate forked processes (`pool: 'forks'`), so the in-process
runner's `process.std*.write` patching never overlaps across files; tests within
a file run sequentially
- Config files use `crypto.randomUUID()` for uniqueness
- HTTP/SSE servers use dynamic port allocation to avoid conflicts
- Coverage is not used because much of the code that we want to measure is run by a spawned process, so it can't be tracked by Vitest
- /sample-config.json is no longer used by tests - not clear if this file serves some other purpose so leaving it for now
- All tests now use built-in MCP test servers, there are no external dependencies on servers from a registry
- Coverage is enforced per-file (lines ≥ 90, statements ≥ 85, functions ≥ 80,
branches ≥ 50). `src/index.ts` (the binary bootstrap) is excluded because it
only runs in the spawned binary, which coverage can't instrument
- All tests use the built-in MCP test servers — no external/registry dependencies