Commit graph inspector/.claude/skills/test-servers/SKILL.md
Author SHA1 Message Date
Cliff Hall
ab80221ce6
chore: land the #2247 round-9 review findings on v2/main (#2322)
* chore: land the #2247 round-9 review findings on v2/main

Copilot filed a ninth review round on #2247 after it had merged, so none of
it reached v2/main. All five findings are addressed here.

1. The testing -> test-servers hand-off delivered the wrong half of the
   procedure. test-servers documented only the two-process manual path
   (server-composable.js --config <name>.json plus a second Inspector
   process), but an integration test builds the server in-process from the
   @modelcontextprotocol/inspector-test-server API and owns its lifecycle.
   Adds a "Two ways to use a fixture — pick the right one first" section
   with the in-process recipe, the four mechanics that bite (one barrel,
   start/stop in afterEach, read started.url, era is a constructor option),
   and a pointer to the reference test; renames "Run one" to "Run one by
   hand (two processes)" so the manual path names itself. testing now says
   in the connecting bullet that an automated test does NOT spawn.

2. testing's heading asked "does the test need a real server?", which its
   own hand-rolled-server exception contradicts. It now asks whether the
   test uses a test-servers/ fixture.

3. "imports" was too broad a dependency test — test-server-scope.test.ts
   imports the package to unit-test scope middleware as a pure function and
   never starts a server. The condition is now "starts, spawns, or
   configures", with that test named as the counterexample.

4. docs/skill-authoring.md attributed the 33% -> 100% rise to imperative
   wording and placement independently. The #2247 edit changed both at
   once, so only the combination is established; the split into two levers
   is now labelled the plausible mechanism, not a measured result.

5. The CHAIN_THRESHOLD comment generalised "a well-shaped pointer clears
   0.8" from one two-case run of that same multi-variable edit. It now
   reads as an existence proof about the ceiling, not a property of
   pointers in general.

Also records the headline lesson where it can be found again: a green
hand-off number proves skill B loaded, never that B answered the question —
in AGENTS.md's chain-case rules and in the skill-authoring checklist.

Docs, skill bodies and one comment only; no runtime code changes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018zwYWanPhvgu6wiKoeGg4N
Signed-off-by: cliffhall <cliff@futurescale.com>

* fix: address Copilot review round 1 on #2264

The new section claimed two shapes and there are three — the automated
stdio path was missing, and it is the repo's established pattern
(getTestMcpServerCommand() into a stdio transport, used by five web
integration files and the whole CLI suite). Telling callers an automated
test "does not spawn" told them not to use it.

- test-servers: "Two ways" -> "Three ways", with a table and a subsection
  for the spawned-stdio shape — the transport owns the child (disconnect,
  not stop()), no config can be selected because that entry point runs the
  stdio default, and it is still the build. Names a reference test for it.
- test-servers: the build is no longer scoped to the manual path. All three
  shapes resolve test-servers/build/ — two through the barrel alias, one by
  running the emitted .js — so the staleness hazard is called out as
  applying to every shape, and `Build first` no longer opens by claiming
  the only reason is subprocess spawning.
- testing: the "does not spawn" instruction is replaced by the two
  automated shapes and when to pick each. What they share, and what the
  hand-off actually needed to say, is that neither picks a showcase config.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018zwYWanPhvgu6wiKoeGg4N
Signed-off-by: cliffhall <cliff@futurescale.com>

* fix: address Copilot review round 2 on #2264

Round 1 fixed the missing stdio shape but kept the wrong axis. Cutting
the taxonomy by "automated vs by hand" excluded the config-driven smokes
— smoke-web-elicitation.mjs spawns server-composable.js --config
directly, and lib/mcp-app-flow.mjs does it for smoke:web:app,
smoke:web:tabs and pack:verify. Those are automated AND need every bit
of the config and protocol-era guidance a person at two terminals does,
so the skill was routing their authors away from it.

- test-servers: the cut is now how the SERVER IS STOOD UP, stated as
  such. Third shape is "spawned composable HTTP", whose consumers are
  the smokes, pack:verify, and a person by hand — the config/era
  guidance belongs to the shape, whoever starts it. Table gains a
  Consumers column and loses the Who column that caused this.
- test-servers: the stale "an automated test builds the server
  in-process instead" line under `Run one by hand` is gone (suppressed
  comment); that heading is now the same shape driven by a person.
- testing: routes by what is driving — integration/CLI test to
  in-process HTTP, stdio-is-the-point to spawned stdio, and a smoke to
  the composable-config path with the era guidance it actually needs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018zwYWanPhvgu6wiKoeGg4N
Signed-off-by: cliffhall <cliff@futurescale.com>

* fix: address Copilot review round 3 on #2264

Round 2 replaced one wrong axis with a narrower one: it routed "a smoke"
wholesale to the config-driven composable path, and smoke:cli does not
use it. smoke-cli.mjs starts an in-process createTestServerHttp for the
header round-trip (so it can read back what the CLI sent) and writes the
built stdio entry into a --catalog for the connect checks; smoke:tui uses
only the latter. Neither picks a config.

- test-servers: consumers filled in per row — in-process HTTP gains CLI
  tests and smoke:cli, spawned stdio gains smoke:cli and smoke:tui, and
  the composable row is narrowed to the WEB smokes. A warning states the
  point directly: "a smoke" is not a shape, pick by the row.
- test-servers: the stdio shape now covers naming the built entry in a
  catalog without connecting, which is what those two smokes do.
- testing: same narrowing, plus the reason spelled out so the word
  "smoke" stops being a routing key.
- testing: the dependency criterion said "starts, spawns, or configures",
  which excluded the build-only smoke:tui case documented directly above
  it — the skill both required and rejected the hand-off for it
  (suppressed comment). It now reads "starts, spawns, configures, or
  hands a built entry to the subject under test".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018zwYWanPhvgu6wiKoeGg4N
Signed-off-by: cliffhall <cliff@futurescale.com>

---------

Signed-off-by: cliffhall <cliff@futurescale.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 08:22:16 -04:00
cliffhall
084a083e2d fix: usable IPv6 spelling, and drop a SEP label that means something else here (round 2)
- The recovery text told users to move the endpoint to `::1`. A bare IPv6
  literal is not a legal URL host — `new URL("http://::1/token")` throws — so
  anyone copying that into the Token URL override got a parse error from advice
  meant to unblock them. Now `[::1]`, in both the notice and the manual
  reproduction guide.

  The exemption list still reads `::1`, because that is the host the SDK
  compares; only the remedy is bracketed, because that is what a user types.
  The comment says so, so neither gets "corrected" into the other.

- Dropped SEP-2207 from all nine files it had reached. I took the label from
  the SDK's own source comments, which attribute this check to it — but this
  repo already uses SEP-2207 for OIDC refresh / offline_access
  (specification/v2_auth_hardening.md, plus an e2e test), so grepping it here
  would have turned up two unrelated things. Replaced with plain description
  rather than a different SEP number: having guessed wrong once, guessing again
  is not the fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018MvNbmAtQuWhDvzi4yXDju
Signed-off-by: cliffhall <cliff@futurescale.com>
2026-09-07 09:54:17 -04:00
cliffhall
b4b82493b1 fix: report a terminal OAuth token-endpoint refusal instead of a dead retry
Closes #2280

InsecureTokenEndpointError appeared nowhere in our source, so it fell through
to the generic auth-failure path and rendered a "Re-authentication required"
banner with a Re-authenticate button. That button can never work: the SDK's
assertSecureTokenEndpoint runs inside executeTokenRequest, the error does not
extend OAuthError, and auth() special-cases it to rethrow rather than start a
fresh /authorize redirect. Clicking re-ran the same flow to the same refusal,
under the raw SDK message, which names the three exempt host literals and
nothing actionable. A terminal configuration error was being presented as a
retryable auth error.

Not specific to *.localhost: it fires for any endpoint outside the SDK's
exemption — host.docker.internal (#1911), a LAN hostname, a reverse-proxy
name, a mistyped scheme.

It is now recognized (core/auth/insecureTokenEndpoint.ts, mirroring
issuerBinding.ts's brand-plus-name classifier and walking cause / data.cause,
since era negotiation and the transport wrappers bury the rejection) and
surfaced as the configuration error it is, naming the endpoint and both ways
out, with no action affordance.

Six paths reach the refusal and all six classify it — the connect handshake,
authenticate() during connect, the satisfied-challenge connect retry, the
post-redirect callback, the command/background path, the deferred tab-visible
resume (which re-armed on every focus, an unbounded loop on a terminal error),
and the banner action. They share one wrapper so a seventh cannot omit the
banner clear, and every clear is scoped to its own serverId: these paths are
asynchronous, so a late continuation for one server must not erase a banner
another raised in the meantime.

Adds oauth-insecure-token-endpoint-http.json to reproduce it, whose issuer is
http://localhost.:8091 — the root-anchored spelling every resolver sends to
loopback but which is none of the SDK's exempt literals. transport.strictPort
keeps that fixture honest, since its port is hard-coded inside the issuer.

Making such an endpoint work has to land in the SDK (typescript-sdk#2591);
this changes the reporting, not the outcome.

Split out of #2282, where it was tangled with the *.localhost origin work for
#1944 — a separate question the reporter is still clarifying.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018MvNbmAtQuWhDvzi4yXDju
Signed-off-by: cliffhall <cliff@futurescale.com>
2026-09-07 09:09:54 -04:00
cliffhall
a344225e23 docs: accept several skill names in skills:eval and re-aim overlapping cases
Adds a multi-name filter to `npm run skills:eval` so a set of related
skills can be iterated together, and makes an unknown name a hard error
rather than an empty run — a typo would otherwise report a green 0/0
that reads exactly like a clean pass of the skill that was meant.

Re-aims two `local-dev` cases and one `test-servers` case that partially
overlapped `AGENTS.md`. Partial overlap oscillates across runs rather
than sitting at zero, so it reads as eval noise instead of a defect in
the case; `docs/skill-authoring.md` now records that failure mode and
the observed scores.

Widens the `test-servers` description to the "Use when …" shape and
points the `testing` skill at it, since anything needing a real server
to run against is a procedure `testing` does not carry. That growth
moves the skill listing from 2,921 to 3,144 of 4,000 characters, so the
figure recorded in AGENTS.md is updated to match.

Signed-off-by: cliffhall <cliff@futurescale.com>
2026-09-01 00:18:43 -04:00
cliffhall
94a71f48bc chore: make the workflow skills model-invocable
`disable-model-invocation: true` does not merely make a skill prefer `/name` —
the model cannot invoke it at all, and its description is absent from the skill
listing. So "create a PR for #2163", which is how that work actually starts,
could never reach `pr-flow`. The original rationale ("procedures with side
effects we'd invoke by name anyway") conflates invoking a skill, which only
loads instructions, with performing the procedure it describes.

Flipped `pr-flow`, `pre-push-gate`, `board-ops` and `test-servers` to
model-invocable. `user-invocable` is an independent axis, so this is purely
additive — every `/name` still works. `issue-create`, `issue-triage` and
`release` stay deliberate.

`board-ops` is in the set because `pr-flow` and `issue-create` delegate their
card steps to it: a pointer from a model-invocable skill to a manual-only one is
a dead end for the model.

Costs weighed: the listing share goes 1,199 → 2,256 of 4,000, and a spurious
load costs ~250 characters where a missed one costs a wrong base branch or an
unsigned commit. The rules stay resident in AGENTS.md either way — this improves
the odds of getting the procedure too, it does not become the guarantee.

Eval cases committed for all four, measured at RUNS=3: pre-push-gate, board-ops
and test-servers 5/5; pr-flow 4/5, with "take it through to a pull request" at
100% and the Copilot-review case borderline. Three initial cases were replaced
rather than tuned green: their answers are resident in AGENTS.md, so the model
answering without the skill is correct behaviour, not a miss — the rule this
file already records about what makes a valid trigger case.

AGENTS.md's maintenance rule now says to default to `false` and explains why,
so the advice this change walks back does not get re-applied.

Signed-off-by: cliffhall <cliff@futurescale.com>
2026-08-30 00:00:14 -04:00
cliffhall
a51e62dfa1 chore: address Copilot review round 1
- skill-manifest: the closing `---` fence must occupy a whole line. `---oops`
  was accepted as a terminator, silently truncating the frontmatter and passing
  validation — the exact failure the guard exists to catch. Regression cases
  added (prefix, trailing whitespace, EOF, CRLF).
- skill-eval: reject a nonzero `claude -p` exit instead of resolving it as an
  empty observation. An auth error or rate limit was scoring as "no skill
  invoked", which passes every negative case and reads as a trigger miss on
  every positive one.
- skill-eval: extract `collectSkillInvocations` / `sampleHit` behind a `main()`
  guard and cover them in `skill-eval.test.mjs` — the stream parsing and the
  process-outcome handling are unreachable from an eval run's happy path.
- docs/test-servers: annotate the era on every catalogue row. Six legacy rows
  carried no annotation under a "unless noted, Modern" default, so the doc sent
  readers to the wrong era for the feature they were trying to see.
- docs/quality-gate: `validate` runs four guards, not three; add the
  `verify:skills` row and the two new `test:scripts` suites; fix the `#publishing`
  fragment left dangling by the split.
- issue-create / pr-flow: delegate the card steps to `/board-ops` rather than
  restating its option IDs, which are explicitly unstable.
- issue-triage: the audit counted presence, not cardinality — an issue carrying
  both `v1` and `v2` passed a check that claims "exactly one". Same for the five
  type labels. Also scope the two milestone checks to board #28.
- AGENTS.md: codify the v1 milestone exception. Every milestone is a v2 release
  bucket, so a v1 issue has none to take and the Incoming-milestone invariant
  cannot apply to board #11.
- clients/launcher/README: its pointer promised an explanation of why the cli/tui
  manifests ship, which the extracted guide does not make.

Signed-off-by: cliffhall <cliff@futurescale.com>
2026-08-29 00:52:45 -04:00
cliffhall
73447c72d1 chore: extract AGENTS.md recipes into agent skills
Closes #2163

`AGENTS.md` was one 1,223-line contract loaded in full on every turn, whether
the task was a one-line CSS fix or a release cut. This splits it: rules stay
resident, procedures move to `.claude/skills/`.

- Ten skills under `.claude/skills/`, seven `disable-model-invocation: true`
  (invoked by name only, so they cost no listing context and have no trigger
  question) and three model-invoked with committed eval cases.
- `AGENTS.md` 1,223 → 380 lines: rules, a skills index, and a project tree
  reduced to one line per entry. The removed recipes are not duplicated.
- `README.md` 811 → 105 lines, split into `docs/{architecture,quality-gate,
  test-servers,publishing,docker}.md`, and dropped from `CLAUDE.md`'s auto-load.
- `.github/copilot-instructions.md` and its mirroring rule removed — Copilot
  reads `AGENTS.md`.
- `npm run verify:skills` (in `validate`, so in `local:gate` and CI) parses each
  SKILL.md the way Claude Code does, requires an explicit invocation mode and
  eval cases for model-invoked skills, and budgets the listing.
- `npm run skills:eval` runs the committed trigger cases headless.

Signed-off-by: cliffhall <cliff@futurescale.com>
2026-08-29 00:36:37 -04:00