`local:gate` ran every client's unit suite bare inside `validate` and
again instrumented inside `coverage` — ~80s of a ~370s run re-executing
exactly the files the coverage pass runs a few minutes later. CI pays
nothing for that duplication because its two jobs run in parallel; the
gate runs serially, so it paid all of it.
The gate's first stage is now `local:validate`: the same `validate:guards`
and `validate:core`, then each client's new `check` script (`format:check`
+ `lint` + `typecheck` + `build`) instead of its `validate`, which is now
literally `check && test`. `npm run validate` and both CI jobs are
unchanged; `local:validate` sits in the `local:` namespace so the
workflow guard keeps it out of CI by construction.
The "strict superset" claim in AGENTS.md is reworded to what is now
true — every CI *check*, not every CI *invocation* — with the reasoning:
`@vitest/coverage-v8` reads V8's precise-coverage profiler and rewrites
no source, so the instrumented pass runs byte-identical code and is only
slower, the stricter direction for the failure class this repo sees.
Measured on the same worktree: 370s -> 262s wall (EXIT=0 both), the
four removed legs summing to 78.5s in the baseline run's own profile.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PsmUAZ6aLUAFqenjJKSSRp
Signed-off-by: cliffhall <cliff@futurescale.com>
Round 8 of the review asked for an AST. That was the right diagnosis of the
wrong problem: the question was never which parser to use, it was why a
test-timeout guard was parsing JavaScript at all.
It got there one reasonable step at a time. The guard enforces two rules no
Vitest config can report — a per-test `retry`, and Testing Library's
`asyncUtilTimeout` — so it read them out of source text, and the review found a
new valid spelling it missed in five consecutive rounds: a trailing comment,
nested parens in `it.each([makeCase()])`, a member-access prefix, the shorthand
`{ retry }`, an aliased import, a non-literal title. Each fix was correct and
each made it more parser-shaped, until it carried eight helpers doing quote
tracking and bracket balancing.
Both rules are observable at runtime, exactly:
- `vitest.setup.shared.mts` is loaded by all six projects and reads
`ctx.task.retry` — the value Vitest resolved for the test. That covers
strictly more than the scan did: a per-test option, a `describe` option
inherited by children, a project-level setting, and a `--retry` flag typed at
a command line, which the scan could never see since it only read npm scripts.
- `clients/web/src/test/asyncUtilTimeout.test.ts` asserts
`getConfig().asyncUtilTimeout`, the value this project's own waitFors read.
Neither has a spelling to anticipate, so neither can be outrun by a new one.
Both were observed firing before being trusted.
The guard keeps only what it can answer by asking Vitest: resolved budgets,
project-level retry, config discovery, and that each project still loads the
setup file — the one thing about the runtime assertion a config CAN report.
868 lines to 361, every scanner helper gone, and 33 of 48 unit tests deleted
because they pinned edge cases of a thing that no longer exists.
storybookSetup.ts goes with it. At the library's own value its configure() call
was a no-op, and the assertion is what pins the number.
AGENTS.md gains the general rule: a rule about what the code DOES is asserted at
runtime, not read out of the source — reach for a verify:* guard when the
question is about configuration, which a tool can be asked to resolve.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PsmUAZ6aLUAFqenjJKSSRp
Signed-off-by: cliffhall <cliff@futurescale.com>
Both findings are the same shape as round 2's: a guard that documents an
invariant while checking one narrow expression of it.
`retry` was only rejected at the project level, so `it("x", { retry: 2 })` and a
`--retry` flag in an npm script both walked through a guard whose own header
said "no retry anywhere". Enforcement is extended rather than the claim
narrowed, because the decision is about the behavior, not about a config key —
all three places Vitest accepts one are now scanned, and the header names the
one that remains out of reach (a human typing --retry into their own shell),
rather than overclaiming.
The test-definition scan is deliberately positional: the retry must sit in an
options object following the test name, which is the only place Vitest reads it
from. A bare word match would fire on fixture data, on `client.retry`, on an
unrelated helper's options and on a mock — all of which exist here, and a guard
that cries wolf gets disabled. Both directions are tested. The first pattern
also missed `it.each([...])("name", { retry })` entirely, so every parameterized
suite was a blind spot; the table argument is now part of the pattern.
Config discovery named only the two `.ts` spellings this repo happens to use,
while Vitest loads twelve. A `clients/foo/vitest.config.mts` was therefore
invisible, and an invisible config yields no project to reject — the same hole
round 1 found in the hand-written CONFIG_ROOTS, one level down. Both discovery
and resolution now use the full set, ordered as Vitest orders it.
Both checks were observed firing before being trusted: a planted retry in a
launcher test and a planted vitest.config.mts each fail the guard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PsmUAZ6aLUAFqenjJKSSRp
Signed-off-by: cliffhall <cliff@futurescale.com>
The comment stripper had a real hole: the line-comment regex was anchored, so it
only removed `//` at the start of a line and a call disabled after code —
`const disabled = true; // configure({ asyncUtilTimeout: 1000 });` — survived it
and the guard passed with no effective configuration. Replaced with a scanner,
because a regex gets this wrong in both directions: it has to strip a comment
that follows code, and it must NOT treat the `//` inside a string literal as one,
which would silently truncate any line holding a URL. Quote tracking (with
escapes) is the only way to have both. Three regression tests.
Two documentation claims were also wrong. The guard header and quality-gate.md
cited #2278 and #2250 as instances of a budget nobody chose; they were not —
#2278 was a missing condition wait around a geometry read and #2250 a real race
in a test's own timing, both fixed by making the test wait for the right thing.
Presenting a race fix as evidence for a larger ceiling argues against #1596,
which this guard exists to uphold. Only the budget and poll-ceiling cases are
cited now, and both files say explicitly why those two are excluded.
And browser-timeouts.mjs overstated its own behavior twice: `bestEffort` can
spend its full budget on an otherwise passing smoke precisely because its expiry
is caught, and raising any budget does delay the report when the thing waited on
never arrives. The diagnostics improve the eventual message; they do not make
the wait shorter. Both now stated plainly, so the header is something to weigh a
change against rather than a reassurance.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PsmUAZ6aLUAFqenjJKSSRp
Signed-off-by: cliffhall <cliff@futurescale.com>
The inline finding was a real bug: main() never called the discovery check, so
the deny-by-default claim rested on test:scripts happening to run the same
comparison — a different command, which `npm run verify:test-timeouts` does not
invoke. `failures` is now seeded from it, and a probe config under clients/ was
observed to fail the guard.
Three suppressed findings, all correct. The afterEach in oauth-interactive
restated the shared hookTimeout and labelled it as a round trip, when it only
stops a server and unlinks a file; it now carries no budget. The file-lock
constants were named for the opposite of what they bound — 90s covers retries
running out against a holder that never yields, 60s covers the stale-lock
takeover that succeeds — so they are LOCK_RETRIES_EXHAUSTED_MS and
STALE_LOCK_TAKEOVER_MS. The browser-timeouts header said 15 budgets; it is 19.
Separately, and more importantly: the asyncUtilTimeout raise is reverted, on
measurement. Verifying round 2 turned up intermittent 15s timeouts across
unrelated unit files, so three arms of three full unit runs were taken in this
worktree at comparable load — unmodified v2/main 3/3 green, this change with
5000 producing 1, 4 and 0 timing-out files, and the same change with that one
line reverted 3/3 green. A wait that is MEANT to expire spends its whole budget
on the happy path, so a 5x raise is a 5x cost on exactly those tests and
saturates the worker pool, starving tests that were never slow.
Both web setup files therefore state asyncUtilTimeout at the library's own
value rather than raising it, which is what #2323's rule actually asks for: a
budget nobody chose is the defect, not a budget that is small. The tests that
assert absence by letting a wait expire are the real problem and are #2335.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PsmUAZ6aLUAFqenjJKSSRp
Signed-off-by: cliffhall <cliff@futurescale.com>
Three of the six Vitest projects ran on Vitest's own testTimeout: 5000 and
five on its hookTimeout/teardownTimeout: 10000, and all 788 Testing Library
waits in the web unit project ran on asyncUtilTimeout: 1000. None of those
numbers was chosen here, and all of them were sized for an idle machine rather
than for three or four concurrent worktree sessions each running a full
local:gate on eight cores. #2292, #2278, #2250, #1942 and #1742 were each that
class, diagnosed one site at a time.
This is the complement of #1596, not a retreat from it: every fix that issue
produced stands, and no race is hidden here. What changes is that a correct,
deterministic test is no longer cut off by a budget nobody picked.
- vitest.shared.mts exports frozen TIMEOUTS / INTEGRATION_TIMEOUTS; all six
projects spread one of them, clients/launcher importing the shared module for
the first time to do so.
- Both web projects configure Testing Library's asyncUtilTimeout. The storybook
one needs its own setup file, deliberately outside .storybook/ and free of
setProjectAnnotations so the automatic preview-annotation provisioning
(#1898) is untouched.
- 28 per-suite restatements of the integration project's own budget are gone,
so raising the shared value now moves every site under it. The handful that
genuinely differ — cross-process lock contention, a full OAuth round trip —
keep their budget as a named constant that says why.
- AppRenderer's waitFor no longer asks for exactly its enclosing budget, which
it could never win (#2292's shape); cli's e2e child timer drops below the
project budget so the bound carrying the diagnostic is the one that fires.
- The transition auto-settle's slack term is named RAF_SLACK_MS and raised;
the term derived from HEADER_ANIM_MS is untouched. The two answer to
different things and must not be scaled as one number.
- The 19 Playwright budgets scattered across three web smokes become four
named constants in scripts/lib/browser-timeouts.mjs.
scripts/verify-test-timeouts.mjs resolves each project through Vitest itself
and asserts the budgets, the asyncUtilTimeout sites, and that retry stays
unset. It was observed red against the pre-change config: unit/tui/launcher at
testTimeout 5000, five projects at hookTimeout 10000, both asyncUtilTimeout
sites unmet.
Closes#2323
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PsmUAZ6aLUAFqenjJKSSRp
Signed-off-by: cliffhall <cliff@futurescale.com>
* chore: land the #2247 round-9 review findings on v2/main
Copilot filed a ninth review round on #2247 after it had merged, so none of
it reached v2/main. All five findings are addressed here.
1. The testing -> test-servers hand-off delivered the wrong half of the
procedure. test-servers documented only the two-process manual path
(server-composable.js --config <name>.json plus a second Inspector
process), but an integration test builds the server in-process from the
@modelcontextprotocol/inspector-test-server API and owns its lifecycle.
Adds a "Two ways to use a fixture — pick the right one first" section
with the in-process recipe, the four mechanics that bite (one barrel,
start/stop in afterEach, read started.url, era is a constructor option),
and a pointer to the reference test; renames "Run one" to "Run one by
hand (two processes)" so the manual path names itself. testing now says
in the connecting bullet that an automated test does NOT spawn.
2. testing's heading asked "does the test need a real server?", which its
own hand-rolled-server exception contradicts. It now asks whether the
test uses a test-servers/ fixture.
3. "imports" was too broad a dependency test — test-server-scope.test.ts
imports the package to unit-test scope middleware as a pure function and
never starts a server. The condition is now "starts, spawns, or
configures", with that test named as the counterexample.
4. docs/skill-authoring.md attributed the 33% -> 100% rise to imperative
wording and placement independently. The #2247 edit changed both at
once, so only the combination is established; the split into two levers
is now labelled the plausible mechanism, not a measured result.
5. The CHAIN_THRESHOLD comment generalised "a well-shaped pointer clears
0.8" from one two-case run of that same multi-variable edit. It now
reads as an existence proof about the ceiling, not a property of
pointers in general.
Also records the headline lesson where it can be found again: a green
hand-off number proves skill B loaded, never that B answered the question —
in AGENTS.md's chain-case rules and in the skill-authoring checklist.
Docs, skill bodies and one comment only; no runtime code changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018zwYWanPhvgu6wiKoeGg4N
Signed-off-by: cliffhall <cliff@futurescale.com>
* fix: address Copilot review round 1 on #2264
The new section claimed two shapes and there are three — the automated
stdio path was missing, and it is the repo's established pattern
(getTestMcpServerCommand() into a stdio transport, used by five web
integration files and the whole CLI suite). Telling callers an automated
test "does not spawn" told them not to use it.
- test-servers: "Two ways" -> "Three ways", with a table and a subsection
for the spawned-stdio shape — the transport owns the child (disconnect,
not stop()), no config can be selected because that entry point runs the
stdio default, and it is still the build. Names a reference test for it.
- test-servers: the build is no longer scoped to the manual path. All three
shapes resolve test-servers/build/ — two through the barrel alias, one by
running the emitted .js — so the staleness hazard is called out as
applying to every shape, and `Build first` no longer opens by claiming
the only reason is subprocess spawning.
- testing: the "does not spawn" instruction is replaced by the two
automated shapes and when to pick each. What they share, and what the
hand-off actually needed to say, is that neither picks a showcase config.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018zwYWanPhvgu6wiKoeGg4N
Signed-off-by: cliffhall <cliff@futurescale.com>
* fix: address Copilot review round 2 on #2264
Round 1 fixed the missing stdio shape but kept the wrong axis. Cutting
the taxonomy by "automated vs by hand" excluded the config-driven smokes
— smoke-web-elicitation.mjs spawns server-composable.js --config
directly, and lib/mcp-app-flow.mjs does it for smoke:web:app,
smoke:web:tabs and pack:verify. Those are automated AND need every bit
of the config and protocol-era guidance a person at two terminals does,
so the skill was routing their authors away from it.
- test-servers: the cut is now how the SERVER IS STOOD UP, stated as
such. Third shape is "spawned composable HTTP", whose consumers are
the smokes, pack:verify, and a person by hand — the config/era
guidance belongs to the shape, whoever starts it. Table gains a
Consumers column and loses the Who column that caused this.
- test-servers: the stale "an automated test builds the server
in-process instead" line under `Run one by hand` is gone (suppressed
comment); that heading is now the same shape driven by a person.
- testing: routes by what is driving — integration/CLI test to
in-process HTTP, stdio-is-the-point to spawned stdio, and a smoke to
the composable-config path with the era guidance it actually needs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018zwYWanPhvgu6wiKoeGg4N
Signed-off-by: cliffhall <cliff@futurescale.com>
* fix: address Copilot review round 3 on #2264
Round 2 replaced one wrong axis with a narrower one: it routed "a smoke"
wholesale to the config-driven composable path, and smoke:cli does not
use it. smoke-cli.mjs starts an in-process createTestServerHttp for the
header round-trip (so it can read back what the CLI sent) and writes the
built stdio entry into a --catalog for the connect checks; smoke:tui uses
only the latter. Neither picks a config.
- test-servers: consumers filled in per row — in-process HTTP gains CLI
tests and smoke:cli, spawned stdio gains smoke:cli and smoke:tui, and
the composable row is narrowed to the WEB smokes. A warning states the
point directly: "a smoke" is not a shape, pick by the row.
- test-servers: the stdio shape now covers naming the built entry in a
catalog without connecting, which is what those two smokes do.
- testing: same narrowing, plus the reason spelled out so the word
"smoke" stops being a routing key.
- testing: the dependency criterion said "starts, spawns, or configures",
which excluded the build-only smoke:tui case documented directly above
it — the skill both required and rejected the hand-off for it
(suppressed comment). It now reads "starts, spawns, configures, or
hands a built entry to the subject under test".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018zwYWanPhvgu6wiKoeGg4N
Signed-off-by: cliffhall <cliff@futurescale.com>
---------
Signed-off-by: cliffhall <cliff@futurescale.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot review round 5. Two of three findings taken; the third rests on a false
premise about the SDK and is declined with evidence in the PR thread.
The hint and the input description are part of this fix's user-facing contract —
an HTTPS-only instruction there puts the local-fixture workflow back out of
reach even while the validator stays permissive — and nothing asserted them.
Two cases now do, plus one asserting a loopback http URL raises no error in the
form. Each mutation (restoring either piece of old copy) fails exactly one of
them.
The docs said there was "no loopback exemption at all" before #2305, which
contradicts the sentence above it: the runtime already tolerated an `http://`
already-stored `client_id`, and that asymmetry is what the issue is about. The
gap was in the config/form validator, and the wording now says so.
Also added: a note to use the canonical spelling of the host. `127.1` and
`2130706433` are accepted here and by the SDK, since both canonicalize to
`127.0.0.1` — but a CIMD `client_id` is string-compared by the authorization
server against the URL it dereferenced, so an exotic spelling is a real footgun
independent of what either validator allows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GfHQ8vMm5NBgzbUJ6UwxmB
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 4, four findings, all correct.
The CIMD-off / CIMD-on pair the smoke guide documents is not a control without
clearing OAuth state first: tokens and the registered client persist in
oauth.json independently of the install-wide CIMD toggle, and the Inspector
reuses valid stored tokens before prompting — so a leftover grant from an
earlier CIMD-on run makes the CIMD-off phase connect and the negative control
silently proves nothing. Both the spec's pair and the walkthrough in
docs/test-servers.md now say to clear state (or isolate MCP_STORAGE_DIR) first.
The composable server announces its MCP endpoint, not the metadata document —
`server-composable.ts` prints a URL ending in `/mcp`. The guide called the
announced URL a metadata URL; it now says to take the origin from that line and
append `/client-metadata.json`.
Two general statements elsewhere in the same file still defined CIMD as always
HTTPS (the glossary row, and the Stytch section's opening), so a reader
following them would have rejected the loopback fixture the same file now
documents. Both note the exemption; the Stytch examples stay HTTPS, which they
are.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GfHQ8vMm5NBgzbUJ6UwxmB
Signed-off-by: cliffhall <cliff@futurescale.com>
`getCimdClientMetadataUrlError` rejected every non-HTTPS CIMD client metadata
URL with no loopback exemption, and the same check runs against `client.json`
on disk through `refineCimdMetadataUrl`, so it could not be worked around by
hand-editing the file either. Every test server in this repo speaks plain HTTP,
so CIMD could not be driven against any local fixture: #2242 shipped verified by
tests alone, and reproducing it by hand meant standing up a self-signed HTTPS
listener plus `NODE_TLS_REJECT_UNAUTHORIZED=0` in the test server's environment.
Accept plain `http:` on the three literals the SDK itself exempts from its
token-endpoint TLS assertion — `localhost`, `127.0.0.1` and `[::1]`. That is
option 1 in #2305 and matches what the runtime already tolerated:
`core/auth/cimd.ts` notes that an already-stored `client_id` may be an `http://`
URL used by local dev/test metadata servers. Anything outside those three,
`localhost.` and `tenant.app.localhost` included, is still rejected — matching
the SDK literal for literal keeps the two allow-lists from disagreeing about one
URL.
The exemption is a literal set rather than `isLoopbackHost` because
`core/client/config-parse.ts` is browser-safe and `core/node/hostUrl.ts` imports
`node:net`.
The field error now names the exemption instead of saying only "must use
HTTPS", and the form's hint and description say the same, so a developer
pointing the field at a local fixture is told which spellings work.
Closes#2305
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GfHQ8vMm5NBgzbUJ6UwxmB
Signed-off-by: cliffhall <cliff@futurescale.com>
Validate `oauth.clientMetadataPath` the way the other two metadata paths
are validated — in `loadConfig` for the JSON route and again at server
setup for a `ServerConfig` built in code. This path is not merely
advertised: it becomes the served document's own `client_id`, so an
off-origin or query-bearing value would publish a client id this server
cannot honour.
Derive that `client_id` from `req.originalUrl` rather than from the
registered route, so a document fetched with a query string answers with
a `client_id` byte-identical to the URL it was fetched from — the
equality CIMD turns on.
Correct the docs' fixed-port warning: this fixture configures no
`issuerUrl` and derives the `client_id` from the request, which is
exactly why the integration test can drive it on a harness-chosen port.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWGH2ytPzSPxnp4cibaV3Z
Signed-off-by: cliffhall <cliff@futurescale.com>
In CIMD (SEP-991) the `client_id` *is* a URL the authorization server
dereferences to learn the client's metadata. Nothing in this repo served such a
document, so a fixture could advertise `client_id_metadata_document_supported`
and still be unusable — exercising CIMD meant standing up a second host by hand.
That is why #2242 shipped verified by its tests alone, and why the v2.6.0
release smoke recorded it as the one contribution with an observable UI surface
that could not be reached.
The composable server now hosts the document itself when `oauth.clientMetadata`
is set, gated on `supportCIMD` — advertising a client the server would then
refuse to honour is a worse fixture than serving nothing. The document's
`client_id` is derived from the **request** rather than from a configured
issuer, so it stays correct when the harness picks the port, as the integration
test does.
`supportDCR: false` in the fixture is load-bearing rather than incidental. With
DCR available a CIMD failure silently succeeds via dynamic registration and the
reproduction proves nothing — during development a misconfigured run connected
happily and reported `Dynamic (DCR)` with a `test_client_…` id, which reads as
success until you check the client id. With DCR off, CIMD is the only path that
can complete, so reaching a connected state is itself the assertion; the test
pins the absent `registration_endpoint` so it stays that way.
Verified end to end against the fixture: Connection Info read
`Client registration — Client ID Metadata (CIMD)` with the client id equal to
the metadata URL. That drive needed a self-signed HTTPS listener, because the
Inspector requires the CIMD metadata URL to be HTTPS with no loopback
exemption — filed separately as #2305, since it is a validation gap rather than
a fixture one.
Closes#2306
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RsmR1iQstcrEzJFmgZXGLi
Signed-off-by: cliffhall <cliff@futurescale.com>
**Verification reads were unbounded.** The 512-entry limit is checked by
`checkSkillConformance` — as a warning, since the SEP makes it an
interoperability bound rather than a MUST — but checking it constrains
nothing, so a hostile server advertising a million entries had the tool
perform a million sequential reads after the report already knew the manifest
was over. Capped at `SKILL_MAX_RESOURCE_ENTRIES`, with the finding still
reported so bounding the work does not silence the reason it was
unnecessary. `manifestListsSelf` is computed over the READ slice, so a
self-entry pushed past the cap still reaches the fallback: the frontmatter
comparison is mandatory and must not be lost to a limit that bounds other
files.
**A malformed skill URI still produced a directory root.** `skillUriIdentity`
falls back to the raw string, so `not a uri/SKILL.md` yielded the root `not a
uri` and enabled the Directory section — letting the UI build a request from
a URI the conformance checks had already rejected. The comment above it
claimed otherwise. `normalizeSkillUri` now, with the rule written down:
identity is for COMPARING two spellings, not for deciding a URI is
well-formed enough to build a request from.
The existing regression test passed against a weaker input that failed for
lacking the suffix and never reached the fallback; it is a table of three
shapes now.
**The test-server guide overstated its own coverage,** claiming four of the
five conformance scenarios map onto fixtures. Three do: there is no
size-mismatch fixture, because the fixture can override an advertised digest
and nothing else, so that path is covered by unit tests. Corrected, with what
adding one would take.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ND46Ci64DC3tzBy9aG81Nc
Signed-off-by: cliffhall <cliff@futurescale.com>
Two live findings. The review ran against 4fa00d1c — the commit before the
round-2 fixes — so its other three were re-reports of already-fixed issues,
verified present at HEAD rather than re-fixed.
**The TUI printed "Conformance: conforms" directly above "Verification
FAILED".** Self-contradictory in the one place a reader looks for a verdict:
the static checks pass on a skill whose advertised digest is well-formed,
while its bytes do not hash to it. The heading is now "Listing checks", which
names what the section covers — the checks against the entry the listing
returned, which say nothing about the bytes served. Same problem, and the
same resolution, as the web screen's "No structural issues".
**The test-server guide claimed every result carries the full modern
envelope**, which the new directory handler made untrue: `readDirectoryPage`
deliberately emits `resultType` alone, because SEP-2640 states the caching
attributes for a modern `skills/list` and says nothing of the kind here, and
its one worked example carries `resultType` only. The claim is narrowed to
the two `skills/*` results and the exception is documented as the deliberate
choice it is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ND46Ci64DC3tzBy9aG81Nc
Signed-off-by: cliffhall <cliff@futurescale.com>
SEP-2640: hosts MUST NOT assume name uniqueness, and when two entries in one
listing collide on `name` a host MUST disambiguate them rather than silently
discarding or preferring one.
`checkSkillConformance` structurally cannot see this — it takes one entry and
a collision is a property of the pair — so `checkSkillNameCollisions` walks
the listing and returns a finding per colliding entry, each naming the
others. All three clients merge it into the entry's own findings, so it
carries through the header badge, the CLI report and the TUI row marks with
no new surface.
**It is a `warning`, not an `error`,** and that is the severity split doing
its job. The obligation is on the *consumer*: a server may legitimately
publish two skills with the same name under different paths, and the SEP's
own `acme/billing/refunds` example is exactly that shape. Calling it an error
would tell a conforming author their catalog is invalid. `--verify` therefore
still exits 0 for a collision.
In the web screen it renders as a banner directly under the Conformance
header and is filtered out of the findings list, the same treatment
`dynamic-resources` gets: it changes how everything below it should be read,
and stating one fact twice reads as two findings.
Fixed while adding it: an entry whose only finding was a banner one rendered
an EMPTY findings container instead of "no structural issues", because the
list's presence was decided on the unfiltered set while its contents were
filtered. `listedIssues` is now derived once and used for both.
The fixture gains `acme/reports` + `globex/reports`, both fully conforming
and sharing the name `reports` — also the only fixture with a multi-segment
skill path, which nothing else exercised. Eight skills over four pages now,
and the integration test walks the cursor to exhaustion rather than asserting
a fixed page count, so a future fixture does not require editing it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ND46Ci64DC3tzBy9aG81Nc
Signed-off-by: cliffhall <cliff@futurescale.com>
Five findings, all real.
**Critical — `verifySkills` selected `contents[0]`.** These bytes are hashed
against the file's advertised digest, so accepting a block the server
labelled something else verifies one file's content against another file's
digest, and can report that as `verified`. A false pass is worse than a
missing check. `contentsFor(result, uri)` now selects by NORMALIZED identity,
which keeps the canonicalized-echo case that motivated the original code
while rejecting an unrelated block, and reports a read error when nothing
answers for the URI. This is what `onReadSkillFile` already did; the two
paths hash bytes against digests and must not disagree about which bytes.
**`skills/get` was dispatched without the extension gate** that `skills/list`
had. Hoisted into `assertSkillsSupported` so the two cannot drift — declaring
the extension commits a server to both. Without it, an undeclared server's
-32601 is indistinguishable to a script from the -32602 a declared server
returns for a URI it does not serve.
**The TUI could strand a user on a hidden Skills tab.** The tab left the bar
when the gate went false but `activeTab` did not, so the pane kept rendering
for a server that never declared the extension. Reset to `info`, following
the Auth precedent — additionally gated on `connected`, because this gate
reads a server declaration and would otherwise fire during a reconnect.
**The TUI keyed a verdict by URI**, so a refresh replacing the manifest under
the same URI left hashes computed for the previous snapshot describing the
new one. Keyed on the entry now, as the web screen already was.
**A failed "Load more" discarded the pages on screen and the retry cursor.**
Both are preserved now, and the success and failure paths share one
staleness-guarded `commit` helper so they cannot drift on which results they
may write — which is how they came to disagree.
Two nits: the six-skill counts. The docs sentence was more wrong than
reported (five awkward skills of six, three outright violations) and is
rewritten rather than renumbered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ND46Ci64DC3tzBy9aG81Nc
Signed-off-by: cliffhall <cliff@futurescale.com>
Closes#2248.
Completes SEP-2640 support across all three clients, and closes the one
obligation #2234 deliberately left open.
- `checkSkillFrontmatterMatch` compares a served SKILL.md's own YAML
frontmatter against the entry the listing advertised, field by field. No
digest can cover this: a digest is taken over the bytes the server served,
so it proves the file was not altered in transit and says nothing about
whether the listing described it honestly. Needs a real YAML parser; `yaml`
was already a root dependency, so this adds no package — but it does newly
put it on core/'s import graph, hence the three bundler `external` lists.
- `resources/directory/read`: result schemas defined against the normative
text, `InspectorClient.readResourceDirectory`, a Directory section on the
Skills screen, and `--method resources/directory/read` in the CLI. The
client refuses the call locally when the server did not declare
`directoryRead`, which is the SEP's MUST NOT. A child the directory lists
but the entry does not is marked `not listed` rather than merged into the
manifest — the SEP calls a directory result a live observation and forbids
treating it as extending the manifest.
- CLI: `skills/list`, `skills/get` and `resources/directory/read`, plus
`--verify` — one NDJSON report per skill, a summary on stderr, exit 7 on a
violation. Reads are sequential; a conforming manifest may declare 512
entries.
- TUI: a Skills pane, shown only when the server declares the extension. Each
row carries its conformance verdict as a glyph as well as a colour, since
the pane is read over ssh and through script(1). Enter verifies.
- `verifySkills` re-throws `AuthRecoveryRequiredError` rather than recording
it per file: it says the session's authorization expired, so absorbing it
would report N identical read failures and swallow the error the TUI and
the web commands key off to reauthorize.
Two open questions settled, both in code comments where they will be found:
`skills/get` carries no caching attributes because SEP-2640 leaves the
question open in as many words, so requiring them would fail a conforming
server; and there is no `PagedSkillsState` because every consumer of this
list is a whole-catalog verdict — computed over page one of three, "this
server's skills conform" is not merely partial but wrong.
The fixture grows two skills, both for checks that were otherwise
undemonstrable: `lying-listing` (listing and file disagree, digest still
verifies) and `stale-manifest` (serves a file its manifest does not declare).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ND46Ci64DC3tzBy9aG81Nc
Signed-off-by: cliffhall <cliff@futurescale.com>
In round 2 I claimed a single wrapper "stops a future path omitting the banner
clear". That was only true inside useOAuthRecovery. useConnectionLifecycle
hand-wrote the notice-plus-scoped-clear pair three times, so the exact failure
mode I said was impossible was one careless edit away — in the PR that claimed
it.
reportTerminalInsecureTokenEndpoint now lives in lib/ and both hooks route
through it; zero hand-written pairs remain. It is generic over the banner shape
rather than importing ReAuthBannerState, since useOAuthRecovery already imports
this module and naming its type would close a cycle. All it needs is a serverId
to compare.
The helper's header also still said "the three OAuth failure paths". There are
seven. Rather than update the count I removed the enumeration and recorded why:
a list of callers is a comment that rots on the next round, which is what just
happened to it.
Also fixes "a /etc/hosts" -> "an /etc/hosts".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018MvNbmAtQuWhDvzi4yXDju
Signed-off-by: cliffhall <cliff@futurescale.com>
- The recovery text told users to move the endpoint to `::1`. A bare IPv6
literal is not a legal URL host — `new URL("http://::1/token")` throws — so
anyone copying that into the Token URL override got a parse error from advice
meant to unblock them. Now `[::1]`, in both the notice and the manual
reproduction guide.
The exemption list still reads `::1`, because that is the host the SDK
compares; only the remedy is bracketed, because that is what a user types.
The comment says so, so neither gets "corrected" into the other.
- Dropped SEP-2207 from all nine files it had reached. I took the label from
the SDK's own source comments, which attribute this check to it — but this
repo already uses SEP-2207 for OIDC refresh / offline_access
(specification/v2_auth_hardening.md, plus an e2e test), so grepping it here
would have turned up two unrelated things. Replaced with plain description
rather than a different SEP number: having guessed wrong once, guessing again
is not the fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018MvNbmAtQuWhDvzi4yXDju
Signed-off-by: cliffhall <cliff@futurescale.com>
strictPort was a new test-server config option — plus its validation and 18
tests — invented solely to keep one fixture's hard-coded issuer port honest.
That is infrastructure built for a screenshot, so it goes.
test-servers/src is now byte-identical to v2/main again. The fixture keeps its
fixed port and docs/test-servers.md carries the caveat instead, written to be
actionable rather than merely cautionary: the failure mode is confusing rather
than obvious, since a relocated server leaves discovery pointing at whatever
unrelated process holds 8091, so the note says to check the port before
believing what the flow does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018MvNbmAtQuWhDvzi4yXDju
Signed-off-by: cliffhall <cliff@futurescale.com>
Closes#2280
InsecureTokenEndpointError appeared nowhere in our source, so it fell through
to the generic auth-failure path and rendered a "Re-authentication required"
banner with a Re-authenticate button. That button can never work: the SDK's
assertSecureTokenEndpoint runs inside executeTokenRequest, the error does not
extend OAuthError, and auth() special-cases it to rethrow rather than start a
fresh /authorize redirect. Clicking re-ran the same flow to the same refusal,
under the raw SDK message, which names the three exempt host literals and
nothing actionable. A terminal configuration error was being presented as a
retryable auth error.
Not specific to *.localhost: it fires for any endpoint outside the SDK's
exemption — host.docker.internal (#1911), a LAN hostname, a reverse-proxy
name, a mistyped scheme.
It is now recognized (core/auth/insecureTokenEndpoint.ts, mirroring
issuerBinding.ts's brand-plus-name classifier and walking cause / data.cause,
since era negotiation and the transport wrappers bury the rejection) and
surfaced as the configuration error it is, naming the endpoint and both ways
out, with no action affordance.
Six paths reach the refusal and all six classify it — the connect handshake,
authenticate() during connect, the satisfied-challenge connect retry, the
post-redirect callback, the command/background path, the deferred tab-visible
resume (which re-armed on every focus, an unbounded loop on a terminal error),
and the banner action. They share one wrapper so a seventh cannot omit the
banner clear, and every clear is scoped to its own serverId: these paths are
asynchronous, so a late continuation for one server must not erase a banner
another raised in the meantime.
Adds oauth-insecure-token-endpoint-http.json to reproduce it, whose issuer is
http://localhost.:8091 — the root-anchored spelling every resolver sends to
loopback but which is none of the SDK's exempt literals. transport.strictPort
keeps that fixture honest, since its port is hard-coded inside the issuer.
Making such an endpoint work has to land in the SDK (typescript-sdk#2591);
this changes the reporting, not the outcome.
Split out of #2282, where it was tangled with the *.localhost origin work for
#1944 — a separate question the reporter is still clarifying.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018MvNbmAtQuWhDvzi4yXDju
Signed-off-by: cliffhall <cliff@futurescale.com>
The Tools tab rendered every schema-portability finding, always open, above
the argument form. On a server whose schemas are broadly unportable that put
the same wall of text ahead of every tool's first input — and the findings
address the server author, not the caller trying to fill the form.
The section now opens collapsed behind the same disclosure heading the Skills
pane's Conformance section uses, with an `N error(s), M warning(s)` badge. The
expand choice is global rather than per tool (`useSchemaFindingsExpanded`,
stored as `inspector.schemaFindings.expanded`): this panel is reused across
selections, so a per-tool disclosure would re-collapse on every click and
reproduce the same scrolling. The badge stays visible while collapsed and the
tool-list severity icon is untouched, so nothing about a tool's standing is
hidden by the closed state.
Two neighbouring fixes on the same panel:
- `JsonEditor` frames Ace in a bordered `Paper`. Ace paints its background edge
to edge with nothing around it, so "Edit as JSON" replaced a column of
bordered inputs with one borderless slab.
- `ToolDetailPanel`'s body `ScrollArea` overrode the app-wide `type="scroll"`
default with `type="auto"`, parking a permanent bar down the side of any tool
whose form is taller than the panel. `offsetScrollbars` stays, so the form
does not shift sideways when the bar fades in.
`test-servers/configs/unportable-schemas-many-http.json` reproduces the
original complaint: the same four presets as `unportable-schemas-http.json`
carrying 26 findings rather than 5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DU3QYnc95g6EM4vo3hYNet
Signed-off-by: cliffhall <cliff@futurescale.com>
Adds a third entry to `DEFAULT_SEED_CONFIG`, `example-server-default`, a
Streamable HTTP card pointing at the MCP org's own feature-reference server
(https://example-server.modelcontextprotocol.io/mcp). Until now every seeded
server was a local stdio process and every remote example in the docs was a
placeholder, so trying a remote connection meant finding or standing up a
server first.
The endpoint is protected by OAuth, but its authorization server advertises
Dynamic Client Registration and fronts a *mock* upstream IdP, so the whole
round trip completes with no account, no API key and no pre-registered client
— which is what makes it viable as a shipped default. Verified end to end
against a prod build: the card connects, negotiates MCP 2025-11-25 and lists
the reference server's tools.
The seed carries no `protocolEra`. That is not a default-by-omission: the
server answers the plain `initialize` handshake and rejects `server/discover`,
so it has no modern (SEP-2663) era to negotiate — `"auto"` would only buy a
failed probe and `"modern"` would fail the connection outright.
Docs updated alongside: the seeded-catalog block and per-client table in
`docs/mcp-server-configuration.md`, the Streamable HTTP example there (which
now leads with the live endpoint and keeps a placeholder to illustrate
`headers`), and the seed description in `clients/launcher/README.md`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012x3rC3JXU17aW3uSwjPoi5
Signed-off-by: cliffhall <cliff@futurescale.com>
The Resources sidebar keyed every row on its `uri` (and templates on
`uriTemplate`), but nothing in MCP makes those unique — a server may return
the same URI twice, and `ManagedListState.applyItems` replaces the list
wholesale, so a repeat is what the server actually sent. React logged
`Encountered two children with the same key` on every render, and a
filtered-out row survived reconciliation instead of unmounting.
This is the resources counterpart of #1957. The row identity is now the same
`listRowKey(id, sourceIndex)` the Tools sidebar has used since then, extracted
so the two formats cannot drift; `toolRowKey` delegates to it. Applied to all
three Resources lists and to `RootsTable`, whose user-maintained list can
collect a duplicate URI the same way.
Selection stays keyed on the URI, deliberately: unlike a duplicated tool name
(#2001), duplicate rows here denote the same resource — a `resources/read`
takes the URI — so highlighting both is correct. Only the React key needs to
tell them apart.
Adds `ServerConfig.duplicateResourceUris` and the
`duplicate-resource-uris-http.json` showcase config, the `resources/list`
mirror of `duplicateToolNames`, so the defect has a fixture to reproduce it —
appended rather than adjacent, for the reason #1957 documented.
Closes#2206
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018yVgAJz4wpkS4mkGk7DQnp
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot round 7. The copyable job inherited the repository's default
GITHUB_TOKEN scopes and left that token in the runner's git config while
`npx` executed a package downloaded from npm.
Adds `permissions: contents: read` and `persist-credentials: false`,
each with a comment giving its reason.
The review cited `.github/workflows/main.yml:11-18` as precedent for
both. It supports one: main.yml does set `permissions: contents: read`
at workflow level, for exactly the stated reason, and every workflow in
this repo carries a block. It does not support the other —
`persist-credentials` appears nowhere in .github/workflows. That half is
taken on its own merit rather than as convention: the job uses git for
nothing after checkout, so the credential is only attack surface while
third-party code runs.
Both matter more in this example than in a workflow we own, because it
is written to be copied into a repository whose defaults the reader
cannot see — which is now stated in the guide.
The edited block was parsed as YAML to confirm both keys land where
intended rather than eyeballing the indentation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KsKC3HniVF9dCZtDVbYp5B
Signed-off-by: cliffhall <cliff@futurescale.com>
Two findings, both real, both mine — and both found by a review round
requested after an explicit "approval recommended", which is the
argument for not stopping at one quiet round.
1. The guide said the `hasApp:false` case is reported "through exit code
2 rather than through a key". It is reported BOTH ways:
`{"appInfo":{"hasApp":false,"toolName":"echo"}}` with exit 2. This
was the round-3 fix over-correcting — that round established there is
no `result` key under --app-info, and I generalized it into "no key
reports this", which does not follow. Now documents both signals and
when each is the right one to branch on: the exit code short-circuits
an && chain without parsing, while the field is the only one of the
two that survives `tools/list --app-info`, whose NDJSON reports every
tool over one connection and exits 0 regardless.
2. §8 claimed "the CLI's own exit code propagates through set -e". It
does not. `pipefail` reports the RIGHTMOST non-zero status, so a
tools/call returning isError:true — which exits 5 and also falsifies
the assertion — surfaces as 1, and the §5 `case` block the guide
pointed at would print the catch-all instead of naming the tool
error. Verified: left=5/right=1 → 1; left=5/right=0 → 5; and the real
CLI piped into `jq -e '.result.isError != true'` → 1.
Added the status table and corrected §8 to say the job carries
pass/fail but not the class, with the capture-then-assert form as the
way to get it. Verified that form recovers exit 5 where the pipeline
reported 1, and that the happy path still passes.
Worth recording: the first attempt to reproduce finding 2 was itself
wrong — feeding jq invalid JSON made it exit on a parse error and the
pipeline returned 5, appearing to refute the claim. The controlled test
and the real CLI both confirm it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KsKC3HniVF9dCZtDVbYp5B
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot round 4. The complete script created an empty isolated OAuth
store and then always passed --stored-auth-only, so against an
OAuth-protected server it could never authenticate — exit 3 by
construction — while being presented as the general "arbitrary MCP
server" template.
The two requirements are genuinely in tension: §6 wants isolation so a
run cannot read or rotate real tokens, and --stored-auth-only wants a
token to reuse. Round 2 made the isolation airtight without reconciling
that, which is what turned a latent gap into a guaranteed one.
Resolved by making credential provisioning explicit rather than by
scoping the script down to unauthenticated servers: it now takes a
bearer token from the CI secret store when MCP_TOKEN is set — the
approach §6 already recommends for unattended runs — and is unchanged
when it is not. Added a callout naming the tension, wired MCP_TOKEN
through the Actions job, and noted that a grant requiring genuine
interactive OAuth does not belong in a CI smoke test at all.
The conditional argument uses "${auth[@]+"${auth[@]}"}" because an empty
array is an unbound variable under `set -u` in bash before 4.4, and
macOS still ships 3.2 (verified locally). Documented why, so it is not
simplified into ${MCP_TOKEN:+--header "..."} — that form happens to
preserve its inner quoting in bash, but as a shell-specific subtlety a
reader cannot check rather than something the script states.
Verified against a live HTTP test server both ways: MCP_TOKEN unset and
MCP_TOKEN set to a value containing a space both pass all three
assertions. Argv construction checked directly — the header arrives as
one word with the token intact, and no empty argument is passed when the
variable is unset.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KsKC3HniVF9dCZtDVbYp5B
Signed-off-by: cliffhall <cliff@futurescale.com>
Two findings, both real; the first was introduced by the round-2 fix.
1. The envelope table said `result` is present "Always". It is not:
`emitResult` returns early for `--app-info` and writes `{appInfo}`
alone, with no `result` key — in the `hasApp:false` case too, which
is signalled through exit code 2 rather than through a key. So a
consumer that requires `.result` breaks on every probe. Documented
as a distinct shape rather than a footnote on the table, since that
is what it is. Verified against the mcp-app-http fixture: keys are
["appInfo"] with --app-info for a tool with and without an app, and
["appInfo","result"] for the same tool called without it.
Worth noting this was self-inflicted — the table arrived in the
round-2 fix for `schemaFindings`, and stating a shape more precisely
is exactly where an over-broad "always" slips in.
2. The Actions job ran `./smoke.sh` while the guide never tells anyone
to chmod +x, and a file copied out of a code block does not carry the
mode bit. Now `bash smoke.sh`, with a comment saying why so it is not
tidied back. Verified: ./nonexec.sh fails with "permission denied",
bash nonexec.sh runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KsKC3HniVF9dCZtDVbYp5B
Signed-off-by: cliffhall <cliff@futurescale.com>
Four distinct issues across 2 inline and 3 suppressed comments. All were
verified against the CLI's own code and behavior before being accepted;
one was diagnosed correctly but explained wrongly, and is documented as
it actually behaves rather than as the comment stated.
1. MCP_STORAGE_DIR alone is not isolation. MCP_INSPECTOR_OAUTH_STATE_PATH
is checked first, so an inherited value silently wins and a smoke run
still reaches the real token file — the guide documented that
precedence and then ignored it one paragraph later. Both variables are
now exported, in precedence order, in §6 and in the §8 script, with a
--list-stored-auth check that prints the path actually resolved.
Verified: with a poisoned MCP_INSPECTOR_OAUTH_STATE_PATH in the
environment, the two-line export wins.
2. `--format json` can carry a third top-level key. `emitResult` sets
`schemaFindings` whenever --strict is passed to tools/list and there
is at least one finding, so "that is the whole contract" was wrong.
Replaced with a table of all three keys and a note to parse by key.
3. --strict findings under --format json. The actionable half is right —
a JSON pipeline should read `schemaFindings` from stdout. The stated
mechanism is not: `writeSchemaLintReport` runs unconditionally, so
--format json --strict emits the full human report on stderr AND the
structured copy on stdout. Documented both streams as measured, with
a jq gate over the per-tool grouping. The `?` operators in that filter
are load-bearing: the key is absent on a clean run, and without them
jq errors instead of passing. Both arms verified.
4. The ErrorEnvelope snippet returned 0. `cmd || { … }` makes the brace
group's status the list's, so a successful jq turned a failed run
green — the same failure class the guide warns about twice elsewhere,
which makes shipping it worse. Now captures, reports, and exits with
the status; verified propagating exit 4.
The §8 script was re-run end to end against a live HTTP test server with
the added export and still passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KsKC3HniVF9dCZtDVbYp5B
Signed-off-by: cliffhall <cliff@futurescale.com>
An MCP cursor is an opaque string: the spec constrains neither its content
nor its length, so `""` is a `nextCursor` a server may legitimately return
and the client is required to send back verbatim.
`listPrompts`, `listResources`, `listResourceTemplates` and
`listRequestorTasks` built their request params with `cursor ? { cursor } : {}`,
a truthiness check that cannot distinguish "no cursor" from "the cursor is
the empty string". Against a server that paginates with `""` they dropped it
and re-requested page one — a list that silently stops at the first page, or
a walk that loops on it forever. Nothing surfaced an error, because the
request was well-formed; it just asked the wrong question. `listTools` (and
the later `listSkills`) already used `cursor !== undefined`; this brings the
other four into line.
`getReplayableParams` in `protocolReplay.ts` existed partly to mirror the
asymmetry — its `method === "tools/list" || params.cursor !== ""` special
case collapses to `typeof params.cursor === "string"` now that the adapters
agree, and its long comment is updated rather than left describing a state
that no longer exists.
Coverage:
- `inspectorClient-list-cursor.test.ts` asserts the outbound params of all
five adapters — `""` carried, an absent cursor sending no `cursor` key,
a non-empty cursor forwarded unchanged. `listTools` is the control.
- `test-servers/configs/empty-cursor-http.json` (`emptyStringCursor`) hands
out `""` as the cursor for page two, the shape no other fixture produces,
and `empty-cursor.test.ts` walks all three of its lists end to end.
- Removing the fix from any one adapter fails exactly its two cases.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EN2UfrkQZvtrviMtqURRm4
Signed-off-by: cliffhall <cliff@futurescale.com>
Three findings, all real.
1. The secret-scan snippet swallowed the CLI's exit code. `out=$(…)`
with no status capture means a non-matching `grep` returns 1, the
`if` is simply not taken, and the step exits 0 even though the tool
call failed — the scan silently becomes the only assertion. Under
`set -e` the opposite happens: the script dies at the assignment and
never scans the error output, which is exactly where a leaked
credential is most likely to appear. Now captures with `|| status=$?`,
scans, then propagates, with a callout explaining both failure modes.
2. `@2` is a major-version range, not a pin, and it was written directly
under a sentence claiming it pinned the version. Both occurrences now
use an exact published release (2.5.0, the current `latest`), and the
prerequisites note says outright that a range is not a pin. `--yes`
added to the prerequisite form so an unattended job cannot hang on
the first-run install prompt.
3. `--stored-auth-only` was described as failing when the store has no
token. It is a no-op against a server that never challenges — the
guide's own verification run passed with an empty store against a
non-auth server. Reworded to say exit 3 happens only when a challenge
cannot be satisfied, and added the way to actually assert that
authentication happened: one run with an isolated empty
MCP_STORAGE_DIR that requires exit 3.
Both corrected shell snippets were executed: secret+failure exits 1,
failure without a secret propagates exit 4.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KsKC3HniVF9dCZtDVbYp5B
Signed-off-by: cliffhall <cliff@futurescale.com>
Adds `docs/cli-smoke-testing.md` — a copyable connect → list → call →
assert workflow for testing an arbitrary MCP server from a shell or a CI
job, plus index rows in the root README and a pointer from the CLI
README.
The gap this fills is a user-facing one. `clients/cli/README.md` is a
flag reference and `scripts/smoke-cli.mjs` is an internal end-to-end
check of the launcher → CLI path; neither tells someone how to compose
those flags into a check they run on every commit against their own
server.
Covers, in order: connecting over stdio/HTTP/SSE (including the `--`
separator, which splits the opposite way under `--cli`); `--format json`
and its `{result[, appInfo]}` envelope; asserting a tool exists with
`jq -e`; one representative `tools/call` and the `--tool-arg` vs
`--tool-args-json` coercion difference; the exit-code map and how to
branch on it; keeping OAuth non-interactive with `--stored-auth-only`
and an isolated `MCP_STORAGE_DIR`; and negative assertions — refusals,
a secret-shape scan of captured output, and `--strict` schema
portability. Ends with a complete script and a GitHub Actions job.
Every command and output shape in the guide was verified against the
bundled composable test servers over both stdio and HTTP rather than
transcribed from the reference, which corrected three claims that would
otherwise have shipped wrong:
- `--tool-arg zip=012` does not send `12`. `012` is not valid JSON, so
it is sent as the literal string. The real trap is `zip=10001`, which
does coerce to a number and is then rejected by a string schema.
- `if ! cmd; then status=$?` yields `0`, because `!` inverts the
pipeline's status — every failure class would look identical. The
guide uses `|| status=$?` and says why.
- The sample script does not report a failure class on its own; that is
the opt-in `case` block from §5.
The §8 script was run end to end against a live HTTP test server, and
exits 0/3/4/5/6 were each reproduced against a real server.
Closes#1886
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KsKC3HniVF9dCZtDVbYp5B
Signed-off-by: cliffhall <cliff@futurescale.com>
`isOfferable` in `core/json/rootUnion.ts` checked that a merged branch carries
at least one property and that every property value is readable, but never that
each name in `required` resolves to a property a form can render. A branch like
`{ properties: { kind }, required: ["kind", "payload"] }` was therefore offered
in the picker, rendered no control for `payload`, and had the submit-time check
report it missing permanently — the exact dead end the function's own comment
already reasons through for a `false`-schema required field, one case short.
Judge the merge, so a branch requiring only names the ROOT declares —
`anyOf: [{ required: ["email"] }, …]` — stays offerable as before. `hasOwn`
rather than `in`, since an argument legally named `constructor` would otherwise
resolve through the prototype and read as declared.
Declining every branch of a union leaves the web form with neither a picker nor
a field, which is a different dead end. `SchemaForm` now seeds the raw-JSON
editor (#2151) in that one case — a root union that resolved to no branches and
no properties — so the arguments stay expressible. Seeded rather than forced, so
the switch still works, and re-seeded during render (never in an effect) because
the form is reused across tools rather than remounted. A tool that genuinely
takes no arguments is untouched. The TUI has no such editor; there the union is
simply not offered, which is already better than a section that can never be
submitted.
Adds a `record_shipment_by` preset and wires it into the
`root-union-schemas-http.json` showcase so the shape is reproducible by hand,
with docs/test-servers.md covering it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYT4eKdnx1yRw7x4bW7Ts
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, all three accepted.
1. Connecting still is not the condition. A few integration tests
deliberately hand-roll a JSON-RPC server because the composable
fixture *cannot* produce what they assert on —
`inspectorClient-malformed-list.test.ts` says so in its own header:
"the SDK's own server cannot emit these shapes: they are exactly what
its types forbid". Same for `listSalvage-era.test.ts`. Real
transport, real client, zero `test-servers/` dependency.
Round 4 already stated the right condition at the top — "does this
test depend on a fixture from `test-servers/`" — and then three
downstream sites went on saying "connects". Those are now scoped to
fixture use, and the hand-rolled exception is stated where the
trigger is defined rather than left to be discovered, with both files
named. Connecting is kept as a hint, since it usually does imply a
fixture, with the instruction to check whether one can express the
case first.
2. "description-strength measurement" was the wrong surface, and this
PR is what made it wrong: a chained prompt is deliberately written so
the target's *description* cannot fire on it, so the number measures
the pointer in the first skill's body and nothing else. Left as-is it
would send the next person tuning a red hand-off at the description.
3. "an 0.8 bar" -> "a 0.8 bar".
Re-measured: 100% / 100% hand-off, 63/63 first-move at 100%, RUNS=5.
Staged by explicit path, not `git add -A` — see ec6895a5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, accepted — and the sharpest catch of the six, because it is
the rationale contradicting the operator two paragraphs below it.
Round 1 rewrote the CHAIN_THRESHOLD comment to say a well-shaped
pointer "does clear 0.8" and cited 100%/80% as the evidence. This
threshold is compared with `>`, so 80% would FAIL an 0.8 bar; the
example refuted the sentence it was offered in support of. It was also
stale: 100%/80% was an intermediate build, and the shipped state
measures 100%/100%.
Both copies now cite 100%/100% as the clearing evidence and keep
100%/80% only as the residual-noise figure, saying explicitly that it
is not an example of clearing the bar.
While checking the claim I found the docs copy needed the same care for
a different reason: `passesThreshold` is strict for the chain and
INCLUSIVE for the first move (`rate > t` vs `rate >= t`, skill-eval.mjs
:558), and 0.8 is the constant for the inclusive one. So "0.8 is
compared strictly" is false as literally read. The docs now say a
*chain* bar of 0.8 would be the strict one, name the operator for both,
and spell out the consequence at RUNS=5 — only 5/5 passes, 4/5 does not.
No functional change; CHAIN_THRESHOLD is still 0.5, still strict.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, both nits accepted — the first is a correctness hole, not a nit.
1. `smoke:tui` never asserts a round trip, so round 2 filed it under
"boot-only, needs nothing". It calls
`ensureTestServers({ requires: ["stdio"] })` and hands the built
fixture to the TUI as its catalog's stdio command
(scripts/smoke-tui.mjs:105-124) — which means the staleness hazard,
the single most expensive thing in `test-servers`, lands on it in
full. Connecting was the wrong sole condition.
The trigger is now "does this test **connect to, or build and
consume**, an MCP fixture?", with `smoke:tui` named at all three
decision points and the fixture-consumption case given its own
paragraph in the closing section.
2. The "29 of 74" figure was brittle and I could not reproduce it
against a different pattern: 29 by my grep, 32 by a direct
`@modelcontextprotocol/inspector-test-server` import, 39 by a wider
union. A number whose value is a function of the grep does not belong
in a skill body that nobody re-runs. Dropped; the concrete
counter-examples stay, which is what actually carries the point —
`storage/store-id.test.ts` validates a string and `mcp/import/*`
parses config files, both sitting beside live-connection tests.
Re-measured after broadening the condition: 100% / 100% hand-off,
63/63 first-move at 100%, full suite at RUNS=5. Unchanged across all
three rounds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, both findings accepted — and the first is the same defect one
level down from round 1.
1. Round 1 replaced "a smoke" with a transport condition but left "an
integration test" unqualified, and left the tier note treating web
integration as wholly fixture-dependent. It is not: **29 of the 74
files in `src/test/integration/` reference a server at all.**
`storage/store-id.test.ts` validates a string; `mcp/import/*` parses
config files. They sit there for the node env and the 30s timeout —
placement is the project manifest, which is exactly why it cannot
also be the fixture trigger.
All four sites now say "an integration test **that connects**" and
name the counter-examples, including the folder-is-not-the-trigger
point at the `integration` bullet itself, and the tier ⚠️ now states
that the boundary cuts *across* the tiers rather than along them.
2. Replacing the worked example orphaned the warning after it: "the two
runs above straddle a 67-point swing … with no change to the pointer"
had nothing left to refer to, since the 33%/33% and 100%/33% pair it
names was gone and the runs on show differed by 20 points across a
pointer rewrite. The weak-pointer pair is restored as its own
paragraph, the warning names both figures explicitly, and the 20-point
intermediate is now framed as RUNS=5 residual noise — contrasted with
the 67, rather than standing in for it.
Re-measured, since "that connects" narrows the phrasing the two chain
prompts have to match: 100% / 100% hand-off, 63/63 first-move at 100%,
full suite at RUNS=5. Unchanged from round 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
All four suppressed comments, none with an inline thread to answer.
- `docs/quality-gate.md`: the second-tier explanation was a paragraph
*inside* the script table, which breaks the table — the `local:gate` and
`pack:verify` rows below it would have rendered as pipe-delimited text.
Folded back into the `verify:dep-lockstep` cell.
- `verify-dep-lockstep.mjs`: the module header and `declaredPackages`'
summary line both still named two fields after the implementation grew a
third. Both now name all three, and the header says which field is
excluded and points at the reason.
- `AGENTS.md`: "Nothing keeps the surviving copies aligned, and nothing
gates them" contradicted the sentence added later in the same paragraph.
Split the two claims — nothing aligns them *automatically*, and the guard
now rejects the drift; the bump is still by hand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J86a6vMcA9csPtdsg7ZcCZ
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, both findings accepted.
1. The trigger was over-broad. "a smoke" and "everything from web
integration rightwards" routed work to `test-servers` that needs no
fixture at all: `smoke:launcher` only checks `--help`, `smoke:web`
asserts the SPA is served, `smoke:web:browser` stops at first paint,
and Storybook play functions render from fixture props. The condition
is now "does this exercise MCP behaviour over a transport?", stated
as such, with both sides named — the smokes that connect
(`smoke:cli`, `smoke:web:app`, `smoke:web:elicit`, `smoke:web:tabs`)
and `pack:verify` on one side, the boot-only ones and Storybook on
the other. Depth in the tier list is explicitly not the boundary.
2. The threshold rationale contradicted the new result. It said an 0.8
bar would mark a hand-off red "regardless of how strongly the first
skill points at the second" — which the 100% measurement refutes.
Both copies (docs and `skill-eval.mjs`) now say what 0.5 actually
buys: a floor on useful reliability for a noisier second-hop load, so
the column still separates an adequate pointer from a broken one. The
value is unchanged.
Re-measured after the narrowing, since a tighter trigger could have
weakened the pointer: 100% / 100% on the hand-off and 63/63 first-move
at 100%, full suite at RUNS=5. Recorded figure updated from the
intermediate 100% / 80%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
- Include `optionalDependencies` in the declared tier's candidate union. A
peer range constrains the consumer's host; an optional range is a direct
declaration npm attempts to install, so omitting it left an optional-only
package free to skew unseen. `peerDependencies` stays excluded, with the
reason stated.
- Make the end-to-end fixture's client copy UNDECLARED — only the root
manifest names `solo`, while `clients/web` merely holds a top-level copy.
That is #2226's defining shape, and it is what makes the candidate set a
union across installs rather than a per-install intersection; the old
fixture would have passed a per-install implementation. Adds the mirror
case (declared only by a client, held by the root) so the union is pinned
in both directions.
- Update the two docs that still described the single-tier guard:
`.claude/skills/local-dev/SKILL.md` said "nothing gates either of those,
and `verify:dep-lockstep` is not it" and cited the live `@types/node`
drift in the present tense; `docs/quality-gate.md` described only the
program tier. Both now state what each tier covers and the two limits
that remain.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J86a6vMcA9csPtdsg7ZcCZ
Signed-off-by: cliffhall <cliff@futurescale.com>
The hand-off measured 33% / 33% at RUNS=3 (#2204's two committed chain
cases). `testing` opened with a one-sentence ⚠️ *classifying* which work
belongs to `test-servers` — a fact, which the model skims — and said it
nowhere else except a bare cross-reference in the last section.
Rewrite it as an action with a trigger ("load the `test-servers` skill
now — that is step one"), enumerate the situations that qualify, and
repeat the pointer at the two later points where the run actually
decides it is writing an integration test: the `integration` project
bullet under "Where the test file goes", and the tier list. The top of a
body is read before the model knows it needs the second skill, so a
pointer that lives only there has already been scrolled past by the time
it matters.
Measured at RUNS=5 on the full suite: the two hand-off cases go to
100% / 80% (100% / 100% on a focused `-- test-servers` run), and all
63 first-move cases stay at 100% — no displacement, as expected, since
neither description changed.
Also update the two places that recorded the old number as the standing
state: the worked example in `docs/skill-authoring.md`, and the
`CHAIN_THRESHOLD` rationale in `scripts/skill-eval.mjs`, which cites the
33% measurement as the reason the bar is 0.5 rather than 0.8. The bar is
unchanged — a reshaped pointer raises the ceiling those two cases reach,
not the floor a new hand-off case should be judged against.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
Most of this round re-reported code the previous commit already changed
(it generated 0 new comments). Two findings were genuinely open.
- SkillsScreen: give "View SKILL.md" and "Fetch with skills/get" the same
attempt ordering `verifyRow` got in round 3. Both were still
last-completion-wins within one manifest: a double click whose older
request settled last replaced the newer result. One shared `isStale`
helper now covers all three actions — a result is dropped when its
manifest was invalidated OR a later click already wrote.
- test-servers/skills.ts + docs: stop calling all three edge cases
"non-conforming". `resources: "dynamic"` is a LEGAL wire form for
generated content — it is unverifiable, not invalid, which is exactly
why the checker reports it as a warning. The prose contradicted the
code it was describing. Now: one legal-but-unverifiable case and two
actual violations.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018kfrH3uRNUM4JAk1reKtyY
Signed-off-by: cliffhall <cliff@futurescale.com>
