Three findings, all real.
1. The secret-scan snippet swallowed the CLI's exit code. `out=$(…)`
with no status capture means a non-matching `grep` returns 1, the
`if` is simply not taken, and the step exits 0 even though the tool
call failed — the scan silently becomes the only assertion. Under
`set -e` the opposite happens: the script dies at the assignment and
never scans the error output, which is exactly where a leaked
credential is most likely to appear. Now captures with `|| status=$?`,
scans, then propagates, with a callout explaining both failure modes.
2. `@2` is a major-version range, not a pin, and it was written directly
under a sentence claiming it pinned the version. Both occurrences now
use an exact published release (2.5.0, the current `latest`), and the
prerequisites note says outright that a range is not a pin. `--yes`
added to the prerequisite form so an unattended job cannot hang on
the first-run install prompt.
3. `--stored-auth-only` was described as failing when the store has no
token. It is a no-op against a server that never challenges — the
guide's own verification run passed with an empty store against a
non-auth server. Reworded to say exit 3 happens only when a challenge
cannot be satisfied, and added the way to actually assert that
authentication happened: one run with an isolated empty
MCP_STORAGE_DIR that requires exit 3.
Both corrected shell snippets were executed: secret+failure exits 1,
failure without a secret propagates exit 4.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KsKC3HniVF9dCZtDVbYp5B
Signed-off-by: cliffhall <cliff@futurescale.com>
Adds `docs/cli-smoke-testing.md` — a copyable connect → list → call →
assert workflow for testing an arbitrary MCP server from a shell or a CI
job, plus index rows in the root README and a pointer from the CLI
README.
The gap this fills is a user-facing one. `clients/cli/README.md` is a
flag reference and `scripts/smoke-cli.mjs` is an internal end-to-end
check of the launcher → CLI path; neither tells someone how to compose
those flags into a check they run on every commit against their own
server.
Covers, in order: connecting over stdio/HTTP/SSE (including the `--`
separator, which splits the opposite way under `--cli`); `--format json`
and its `{result[, appInfo]}` envelope; asserting a tool exists with
`jq -e`; one representative `tools/call` and the `--tool-arg` vs
`--tool-args-json` coercion difference; the exit-code map and how to
branch on it; keeping OAuth non-interactive with `--stored-auth-only`
and an isolated `MCP_STORAGE_DIR`; and negative assertions — refusals,
a secret-shape scan of captured output, and `--strict` schema
portability. Ends with a complete script and a GitHub Actions job.
Every command and output shape in the guide was verified against the
bundled composable test servers over both stdio and HTTP rather than
transcribed from the reference, which corrected three claims that would
otherwise have shipped wrong:
- `--tool-arg zip=012` does not send `12`. `012` is not valid JSON, so
it is sent as the literal string. The real trap is `zip=10001`, which
does coerce to a number and is then rejected by a string schema.
- `if ! cmd; then status=$?` yields `0`, because `!` inverts the
pipeline's status — every failure class would look identical. The
guide uses `|| status=$?` and says why.
- The sample script does not report a failure class on its own; that is
the opt-in `case` block from §5.
The §8 script was run end to end against a live HTTP test server, and
exits 0/3/4/5/6 were each reproduced against a real server.
Closes#1886
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KsKC3HniVF9dCZtDVbYp5B
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, all three accepted.
1. Connecting still is not the condition. A few integration tests
deliberately hand-roll a JSON-RPC server because the composable
fixture *cannot* produce what they assert on —
`inspectorClient-malformed-list.test.ts` says so in its own header:
"the SDK's own server cannot emit these shapes: they are exactly what
its types forbid". Same for `listSalvage-era.test.ts`. Real
transport, real client, zero `test-servers/` dependency.
Round 4 already stated the right condition at the top — "does this
test depend on a fixture from `test-servers/`" — and then three
downstream sites went on saying "connects". Those are now scoped to
fixture use, and the hand-rolled exception is stated where the
trigger is defined rather than left to be discovered, with both files
named. Connecting is kept as a hint, since it usually does imply a
fixture, with the instruction to check whether one can express the
case first.
2. "description-strength measurement" was the wrong surface, and this
PR is what made it wrong: a chained prompt is deliberately written so
the target's *description* cannot fire on it, so the number measures
the pointer in the first skill's body and nothing else. Left as-is it
would send the next person tuning a red hand-off at the description.
3. "an 0.8 bar" -> "a 0.8 bar".
Re-measured: 100% / 100% hand-off, 63/63 first-move at 100%, RUNS=5.
Staged by explicit path, not `git add -A` — see ec6895a5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, accepted — and the sharpest catch of the six, because it is
the rationale contradicting the operator two paragraphs below it.
Round 1 rewrote the CHAIN_THRESHOLD comment to say a well-shaped
pointer "does clear 0.8" and cited 100%/80% as the evidence. This
threshold is compared with `>`, so 80% would FAIL an 0.8 bar; the
example refuted the sentence it was offered in support of. It was also
stale: 100%/80% was an intermediate build, and the shipped state
measures 100%/100%.
Both copies now cite 100%/100% as the clearing evidence and keep
100%/80% only as the residual-noise figure, saying explicitly that it
is not an example of clearing the bar.
While checking the claim I found the docs copy needed the same care for
a different reason: `passesThreshold` is strict for the chain and
INCLUSIVE for the first move (`rate > t` vs `rate >= t`, skill-eval.mjs
:558), and 0.8 is the constant for the inclusive one. So "0.8 is
compared strictly" is false as literally read. The docs now say a
*chain* bar of 0.8 would be the strict one, name the operator for both,
and spell out the consequence at RUNS=5 — only 5/5 passes, 4/5 does not.
No functional change; CHAIN_THRESHOLD is still 0.5, still strict.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, both nits accepted — the first is a correctness hole, not a nit.
1. `smoke:tui` never asserts a round trip, so round 2 filed it under
"boot-only, needs nothing". It calls
`ensureTestServers({ requires: ["stdio"] })` and hands the built
fixture to the TUI as its catalog's stdio command
(scripts/smoke-tui.mjs:105-124) — which means the staleness hazard,
the single most expensive thing in `test-servers`, lands on it in
full. Connecting was the wrong sole condition.
The trigger is now "does this test **connect to, or build and
consume**, an MCP fixture?", with `smoke:tui` named at all three
decision points and the fixture-consumption case given its own
paragraph in the closing section.
2. The "29 of 74" figure was brittle and I could not reproduce it
against a different pattern: 29 by my grep, 32 by a direct
`@modelcontextprotocol/inspector-test-server` import, 39 by a wider
union. A number whose value is a function of the grep does not belong
in a skill body that nobody re-runs. Dropped; the concrete
counter-examples stay, which is what actually carries the point —
`storage/store-id.test.ts` validates a string and `mcp/import/*`
parses config files, both sitting beside live-connection tests.
Re-measured after broadening the condition: 100% / 100% hand-off,
63/63 first-move at 100%, full suite at RUNS=5. Unchanged across all
three rounds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, both findings accepted — and the first is the same defect one
level down from round 1.
1. Round 1 replaced "a smoke" with a transport condition but left "an
integration test" unqualified, and left the tier note treating web
integration as wholly fixture-dependent. It is not: **29 of the 74
files in `src/test/integration/` reference a server at all.**
`storage/store-id.test.ts` validates a string; `mcp/import/*` parses
config files. They sit there for the node env and the 30s timeout —
placement is the project manifest, which is exactly why it cannot
also be the fixture trigger.
All four sites now say "an integration test **that connects**" and
name the counter-examples, including the folder-is-not-the-trigger
point at the `integration` bullet itself, and the tier ⚠️ now states
that the boundary cuts *across* the tiers rather than along them.
2. Replacing the worked example orphaned the warning after it: "the two
runs above straddle a 67-point swing … with no change to the pointer"
had nothing left to refer to, since the 33%/33% and 100%/33% pair it
names was gone and the runs on show differed by 20 points across a
pointer rewrite. The weak-pointer pair is restored as its own
paragraph, the warning names both figures explicitly, and the 20-point
intermediate is now framed as RUNS=5 residual noise — contrasted with
the 67, rather than standing in for it.
Re-measured, since "that connects" narrows the phrasing the two chain
prompts have to match: 100% / 100% hand-off, 63/63 first-move at 100%,
full suite at RUNS=5. Unchanged from round 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
All four suppressed comments, none with an inline thread to answer.
- `docs/quality-gate.md`: the second-tier explanation was a paragraph
*inside* the script table, which breaks the table — the `local:gate` and
`pack:verify` rows below it would have rendered as pipe-delimited text.
Folded back into the `verify:dep-lockstep` cell.
- `verify-dep-lockstep.mjs`: the module header and `declaredPackages`'
summary line both still named two fields after the implementation grew a
third. Both now name all three, and the header says which field is
excluded and points at the reason.
- `AGENTS.md`: "Nothing keeps the surviving copies aligned, and nothing
gates them" contradicted the sentence added later in the same paragraph.
Split the two claims — nothing aligns them *automatically*, and the guard
now rejects the drift; the bump is still by hand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J86a6vMcA9csPtdsg7ZcCZ
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot, both findings accepted.
1. The trigger was over-broad. "a smoke" and "everything from web
integration rightwards" routed work to `test-servers` that needs no
fixture at all: `smoke:launcher` only checks `--help`, `smoke:web`
asserts the SPA is served, `smoke:web:browser` stops at first paint,
and Storybook play functions render from fixture props. The condition
is now "does this exercise MCP behaviour over a transport?", stated
as such, with both sides named — the smokes that connect
(`smoke:cli`, `smoke:web:app`, `smoke:web:elicit`, `smoke:web:tabs`)
and `pack:verify` on one side, the boot-only ones and Storybook on
the other. Depth in the tier list is explicitly not the boundary.
2. The threshold rationale contradicted the new result. It said an 0.8
bar would mark a hand-off red "regardless of how strongly the first
skill points at the second" — which the 100% measurement refutes.
Both copies (docs and `skill-eval.mjs`) now say what 0.5 actually
buys: a floor on useful reliability for a noisier second-hop load, so
the column still separates an adequate pointer from a broken one. The
value is unchanged.
Re-measured after the narrowing, since a tighter trigger could have
weakened the pointer: 100% / 100% on the hand-off and 63/63 first-move
at 100%, full suite at RUNS=5. Recorded figure updated from the
intermediate 100% / 80%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
- Include `optionalDependencies` in the declared tier's candidate union. A
peer range constrains the consumer's host; an optional range is a direct
declaration npm attempts to install, so omitting it left an optional-only
package free to skew unseen. `peerDependencies` stays excluded, with the
reason stated.
- Make the end-to-end fixture's client copy UNDECLARED — only the root
manifest names `solo`, while `clients/web` merely holds a top-level copy.
That is #2226's defining shape, and it is what makes the candidate set a
union across installs rather than a per-install intersection; the old
fixture would have passed a per-install implementation. Adds the mirror
case (declared only by a client, held by the root) so the union is pinned
in both directions.
- Update the two docs that still described the single-tier guard:
`.claude/skills/local-dev/SKILL.md` said "nothing gates either of those,
and `verify:dep-lockstep` is not it" and cited the live `@types/node`
drift in the present tense; `docs/quality-gate.md` described only the
program tier. Both now state what each tier covers and the two limits
that remain.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J86a6vMcA9csPtdsg7ZcCZ
Signed-off-by: cliffhall <cliff@futurescale.com>
The hand-off measured 33% / 33% at RUNS=3 (#2204's two committed chain
cases). `testing` opened with a one-sentence ⚠️ *classifying* which work
belongs to `test-servers` — a fact, which the model skims — and said it
nowhere else except a bare cross-reference in the last section.
Rewrite it as an action with a trigger ("load the `test-servers` skill
now — that is step one"), enumerate the situations that qualify, and
repeat the pointer at the two later points where the run actually
decides it is writing an integration test: the `integration` project
bullet under "Where the test file goes", and the tier list. The top of a
body is read before the model knows it needs the second skill, so a
pointer that lives only there has already been scrolled past by the time
it matters.
Measured at RUNS=5 on the full suite: the two hand-off cases go to
100% / 80% (100% / 100% on a focused `-- test-servers` run), and all
63 first-move cases stay at 100% — no displacement, as expected, since
neither description changed.
Also update the two places that recorded the old number as the standing
state: the worked example in `docs/skill-authoring.md`, and the
`CHAIN_THRESHOLD` rationale in `scripts/skill-eval.mjs`, which cites the
33% measurement as the reason the bar is 0.5 rather than 0.8. The bar is
unchanged — a reshaped pointer raises the ceiling those two cases reach,
not the floor a new hand-off case should be judged against.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EQVdemwgUhh45yzeaRSLhW
Signed-off-by: cliffhall <cliff@futurescale.com>
Most of this round re-reported code the previous commit already changed
(it generated 0 new comments). Two findings were genuinely open.
- SkillsScreen: give "View SKILL.md" and "Fetch with skills/get" the same
attempt ordering `verifyRow` got in round 3. Both were still
last-completion-wins within one manifest: a double click whose older
request settled last replaced the newer result. One shared `isStale`
helper now covers all three actions — a result is dropped when its
manifest was invalidated OR a later click already wrote.
- test-servers/skills.ts + docs: stop calling all three edge cases
"non-conforming". `resources: "dynamic"` is a LEGAL wire form for
generated content — it is unverifiable, not invalid, which is exactly
why the checker reports it as a warning. The prose contradicted the
code it was describing. Now: one legal-but-unverifiable case and two
actual violations.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018kfrH3uRNUM4JAk1reKtyY
Signed-off-by: cliffhall <cliff@futurescale.com>
Two of these reverse round 4, which asked for the opposite. Where the
readings conflict I took the permissive one: a wrong tightening rejects
conforming servers, a wrong loosening only fails to report something.
- skills.ts: stop requiring the `skill:` scheme. SEP-2640 says a server
SHOULD use `skill://` and explicitly allows a domain-native scheme, so
round 4's requirement handed a conforming `github://…` skill a false
`malformed-uri` and skipped its name and root checks. Containment is
scheme-independent — it compares against THIS entry's own root — so the
traversal fix that requirement came packaged with is unaffected.
- skills.ts: the 512-entry and 16 MiB limits are warnings. Both are
SHOULD NOTs for a server and MAYs for a host, so calling them errors
contradicted the "error = a MUST was broken" contract round 4
established and told authors a permitted skill was invalid.
- skills.ts: `skillEntriesMatch` compares canonically — keys sorted,
manifest sorted by URI. `JSON.stringify` treated key order and manifest
order as differences, though neither carries meaning.
- SkillsScreen: a differing `skills/get` is "a different snapshot"
(yellow), not a disagreement (red). The SEP defines it as a fresh
point-in-time read, so a skill that changed since the listing
legitimately differs.
- useServerCommands: route `onGetSkill` through the command-scoped OAuth
recovery like every other server command. An expired authorization
rendered an error and stopped there — no reauthorization, no retry.
- test-servers/skills.ts: stamp the modern base result envelope
(`resultType` / `ttlMs` / `cacheScope`) on every skills result. The SDK
stamps it only for methods in its own codec, and `skills/*` are
consumer-owned, so a 2026-era connection was receiving results without
it. Unconditional rather than per era: the modern leg builds a server
per request, so there is no era to branch on at registration time, and
on legacy they are three members no codec inspects.
- composable-test-server: remove the `skills.directoryRead` option. It was
publicly settable and advertised a method nothing serves, producing the
exact false capability the fixture exists to help catch. `skills` is now
a plain boolean; the flag returns in phase 3 (#2248) with its handler.
Still not changed: whether a modern `skills/list` RESULT SCHEMA must
require that envelope. Stamping it server-side is free; requiring it
client-side would reject servers on a reading nobody has checked against
the normative text. #2248 owns settling it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018kfrH3uRNUM4JAk1reKtyY
Signed-off-by: cliffhall <cliff@futurescale.com>
- skills.ts: report a manifest that omits the skill's own SKILL.md
(an empty list included), duplicate URIs, entries outside the skill
root, and a missing size. `Conforms` was reachable for manifests that
break invariants SEP-2640 states.
- skills.ts: cross-check the declared byte length before hashing. A size
that disagrees fails verification on its own — the digest is taken over
the bytes the server served, so agreeing with it says nothing about
whether the manifest describes them.
- skills.ts: copy the view instead of slicing its backing store in
`sha256Digest`. `SharedArrayBuffer.prototype.slice()` returns another
SharedArrayBuffer, which `crypto.subtle.digest` rejects — the cast
claimed to handle the exact input that would have thrown. No cast now.
- skills.ts: state the one obligation NOT checked here — that an entry's
frontmatter matches the fetched SKILL.md's. The digest cannot cover it,
and closing it needs a YAML parser, so it is tracked on #2248.
- managedSkillsState: cap the walk at LIST_MAX_PAGES. The repeated-cursor
guard only catches a server stuck on one cursor; endlessly unique ones
walked forever. Raises rather than truncating, like the salvage walk.
- managedSkillsState: gate every write on a session generation, so a walk
that resolves after a disconnect or destroy cannot repopulate a cleared
store or deliver the previous session's skills into the next.
- inspectorClient: send a cursor when it is `!== undefined`, not when it
is truthy. An opaque cursor may be `""`, and dropping it re-requested
page one — which the store then reported as a repeated-cursor failure.
- SkillsScreen: key verdict invalidation on the manifest (URI + digests +
sizes), not the URI alone. A Refresh that changed the manifest left a
green badge attached to a digest nothing had checked.
- SkillsScreen: epoch-guard every read continuation, so a fetch that
resolves after the selection moved on cannot write into the new one.
- SkillsScreen: bound "Verify all" to 4 concurrent reads. A conforming
manifest may hold 512 files.
- skillsSchemas: drop the guessed `resources/directory/read` result
schema. Nothing calls it, so an unverified shape could sit wrong
indefinitely; phase 3 adds it against the normative text.
- skills-http.json: declare the extension bare. It advertised
`directoryRead: true` with no handler, so Connection Info reported
"Supported" for a method that answers -32601.
- types.ts: restore the Tasks doc comment the Skills interface displaced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018kfrH3uRNUM4JAk1reKtyY
Signed-off-by: cliffhall <cliff@futurescale.com>
Adds detection, enumeration, and conformance checking for the Skills
extension (`io.modelcontextprotocol/skills`) — phases 1 and 2 of #2234,
which is everything its Acceptance list names.
Skills is a *server*-declared extension, read off the connecting server's
`capabilities.extensions`, so it is deliberately absent from
`ADVERTISABLE_EXTENSIONS` — that registry is what the Inspector advertises
and the user toggles, and an entry there would be a meaningless toggle.
`skills/list` and `skills/get` are consumer-owned extension methods that
neither era codec defines, so the SDK's era gate skips them and they go out
as ordinary `client.request` calls with explicit result schemas. The
raw-wire channel modern `tasks/*` needs is not used, and the Skills tab is
not era-gated: a legacy-era server that declares the extension is serving it.
The checks are the point, not the list view. `core/mcp/skills.ts` reports
each obligation SEP-2640 states — the name/path invariant, the digest
format, the 512-entry and 16 MiB limits, and `resources: "dynamic"`, which
means integrity cannot be verified at all. Digest verification hashes the
fetched bytes with WebCrypto and returns a mismatch with both digests
attached rather than throwing, because showing a mismatch loudly is the
whole value proposition.
Files are fetched on demand: SEP-2640 is explicit that a `resources/read`
of a `SKILL.md` is not a load and confers no standing, so none of the SEP's
host machinery is implemented.
The `skills-http` fixture serves four skills over two pages, three of them
deliberately non-conforming — without those the verification code is
untestable.
Phase 3 (CLI, TUI, `resources/directory/read`, paged mode) is #2248.
Closes#2234
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018kfrH3uRNUM4JAk1reKtyY
Signed-off-by: cliffhall <cliff@futurescale.com>
Re-measured the two committed chain cases under the stricter turn-boundary
scoring from 313a5809: 100% and 33% at RUNS=3, against 33% and 33% before.
That rise is NOT an effect of the change and the doc now says so. The turn
rule only ever rejects matches a flatter reading accepted, so it cannot
raise a score — the two runs simply straddle a 67-point swing on the same
prompt at RUNS=3, where one sample is worth 33 points. Reading it as an
improvement would be exactly the mistake the eval exists to prevent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
One finding, and a sharp one: position in the stream is not causation.
`collectSkillInvocations` flattened every `tool_use` block across every
assistant event into one ordered array, and `chainHit` read position
alone. But the model can emit several `Skill` blocks in a SINGLE assistant
message, and it has not seen the first skill's body when it does — those
are parallel guesses, not a hand-off. A run that emitted `testing` and
`test-servers` together in one message scored as `A` led to `B`, and would
have gone on scoring after the pointer in `testing`'s body was deleted:
exactly the false pass this feature exists to rule out, arriving by a
second route (Copilot).
Each recorded invocation now carries the assistant event it came from —
`{payload, turn}` — and `chainHit` requires every link after the first to
land in a STRICTLY later turn. The turn count is threaded through
`collectSkillInvocations`'s `turnOffset`/`nextTurn` so a stream read in
chunks stays monotonic rather than restarting per read; without that, a
chunk boundary would reset the numbering and reintroduce the same bug on
long runs.
The scan stays greedy, which is still correct under the constraint: taking
the earliest occurrence of a link can only leave more room for the rest,
so no later starting point succeeds where the greedy one fails.
`sampleHit` is deliberately unaffected — a first-move case asks only
whether a skill fired at all, so it reads names and ignores turns.
Four new assertions in one case: a same-turn pair scores false; the same
two skills a turn apart score true; a same-turn pair does not poison a
later genuine hand-off; and only the chain's first link is unconstrained.
Plus a `collectSkillInvocations` case pinning the turn tagging and the
cross-chunk offset.
`docs/skill-authoring.md` now lists this as the second way a chained case
can false-pass, alongside a prompt carrying the target's own trigger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
Two findings, both correct, and the first one means round 1's containment
fix was only half a fix.
**`--allowedTools` is not the availability filter.** It grants permission;
it does not restrict which tools exist. So a tool a user's or a plugin's
settings already permit stayed reachable across all 14 turns, and the
bound still rested on the deny list being complete — the exact property
round 1 set out to stop depending on. `--tools` is the real restriction
(it selects from the built-in set), and both flags now carry the same
four names: `--tools` to bound, `--allowedTools` so those four never need
a permission prompt no headless run can answer. Verified against the real
CLI — the full flag set runs to a clean `end_turn`. The test asserts both
flags, and both probe snippets in `docs/skill-authoring.md` carry
`--tools` with a ⚠️ saying which one does the bounding.
**The threshold guard let `NaN` and negatives through.**
`CHAIN_THRESHOLD=abc` is `NaN`, which fails every comparison and prints an
`above NaN%` summary; a negative bar passes every chain unconditionally.
Both turn an advertised env knob into a measurement that quietly means
nothing. `CHAIN_THRESHOLD` must now be in `[0, 1)` — half-open because the
chain bar is strict — and `THRESHOLD` in `[0, 1]`, inclusive because a
first-move floor of 1.0 is meetable. The message names the offending
value. A new case spawns the script for each bad value rather than
asserting on the predicate, since these guards live in `main`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
Five findings, all real.
**The chain prompts carried the target skill's own trigger.** `test-servers`
claims "a change needs a real server to exercise it", and both cases said
"against a real/live server" — so the model could load `testing`, then pick
`test-servers` from the ORIGINAL prompt, in that order, and score a hit that
would have survived deleting the pointer from `testing` entirely. Rewritten to
carry no server cue ("end to end"), and the measured rate fell from 100% / 67%
to 33% / 33%. That is the size of the artefact, and 33% is what the pointer
alone is worth. Both cases are now red against the 0.5 bar and stay that way:
`skills:eval` is not a gate, the number IS the finding, and lowering the bar to
turn the column green would discard the only signal this feature adds. Filed as
#2247 with the trap written down so a fix cannot re-measure the artefact.
**A deny list cannot bound a 14-turn run.** It only names tools known when it
was written, and this checkout configures an HTTP `mcp-docs` server in
`.mcp.json` while a contributor's own MCP servers and plugins add more. Now
`--allowedTools Read,Glob,Grep,Skill` (what the harness actually needs) plus
`--strict-mcp-config` with no `--mcp-config`, which drops every configured
server. The deny list stays alongside, since a deny is unconditional while an
allow list governs pre-approval. What remains outside all three is a
contributor's plugin tools; `--bare` would remove those and skills with them,
which would measure nothing — so that residual is stated rather than papered
over.
**The chain threshold compared inclusively.** "More often than not" is `> 0.5`,
and `>=` passes exactly half whenever RUNS is even — 2/4 would report a result
the criterion does not license. `passesThreshold` is now strict for chains and a
floor for first moves, and a strict bound of 1.0 is rejected up front rather
than failing every case while looking like a trigger problem.
**The doc's probe command still denied only four tools**, contradicting the
section's own promise that a probe and a scored case see the same policy.
Updated, with a note that it must move with the harness, plus a hand-off probe
form.
**The reporting had no automated coverage** — the one thing that could silently
merge the two measurements lived inside `main`. Extracted as `formatReport` and
tested: both headings, per-group thresholds, the two separate summaries, the
combined exit, and both single-kind selections.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
`skills:eval` ran every case with `--max-turns 1`, so it measured one
thing: whether a skill is the model's FIRST tool call. That is right for a
skill a user reaches directly, and it leaves a whole class unmeasured — a
skill reached from inside another skill's body. `testing` tells the model
that picking a fixture is `/test-servers` and that it has to load it, and
`test-servers` scores 5/5 while every one of those five cases asks for it
by name. A skill only ever reachable through a hand-off would score a
clean 100% with the hand-off silently never firing.
An eval case is now one of two shapes, and exactly one:
{ "prompt": "…", "expect": "test-servers" } first move, 1 turn
{ "prompt": "…", "chain": ["testing", "test-servers"] } hand-off, 14 turns
A chain must END with the skill whose file it lives in. The case exists to
measure whether THIS skill is reachable, so the file that goes red is the
one belonging to the skill that stopped being reached; anchoring on the
first link would file a `test-servers` measurement under `testing`, where
a `test-servers` description edit would never be seen. Links are checked
against the repo's model-invoked set, which is why both `verify:skills`
and `collectCases` now validate in a second pass — `test-servers` sorts
before `testing`, so a per-directory check would reject a live chain as
unknown purely because of where the alphabet put it.
Scoring is an ordered SUBSEQUENCE, not a prefix and not a contiguous run:
the model may load something before the chain starts and something
unrelated in between, and neither changes the claim that A led to B.
`collectSkillInvocations` therefore returns an ordered array rather than a
Set — a B, A, B run has to stay distinguishable from one that never
reached B from A.
The two rates are reported in separate columns and never summed. A
hand-off is a second-hop load over many turns; a first-move rate is the
model's opening move. `CHAIN_MAX_TURNS` (14) and `CHAIN_THRESHOLD` are
their own knobs for the same reason, and `CHAIN_THRESHOLD` defaults to
**0.5** rather than inheriting 0.8: see the measurement below.
Read-only containment no longer leans on `--max-turns 1`, which was doing
much of it by itself. The deny list gains the agentic and network tools —
`Task` in particular, whose subagent the flag does not reach.
Measured, `skills:eval -- test-servers`:
First move (1 turn) 7/7 at 100%, unchanged
Hand-off (14 turns) 67% and 33% at RUNS=3
So the hand-off is real and unreliable — which is the fact nothing could
observe before this change, and the reason 0.5 rather than 0.8 is the
default bar: at 0.8 both committed cases are red no matter how strongly
the first skill points at the second, and the column stops carrying
signal. A `["pr-flow", "test-servers"]` probe measured 0% and was dropped
rather than kept: `pr-flow` says nothing about fixtures, so the case had
no lever short of broadening a description onto another skill's ground.
Both findings are written up in `docs/skill-authoring.md`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
Restrict the truncation guard to PLAIN scalars. `yaml` attaches a legitimate
trailing comment to quoted and block scalars too, so keying on `.comment` alone
rejected valid frontmatter — `description: "Complete value" # note` was reported
as truncated, telling the author to quote a value that was already quoted. The
regex this replaced tested the same thing by reading the first character; moving
to the node dropped the check rather than encoding it. Verified all three
shapes (double-quoted, single-quoted, block) against the parser, and added the
regression cases.
Also correct the `--max-turns 1` bullet in docs/skill-authoring.md, which
claimed the skill must be the model's first tool CALL. `collectSkillInvocations`
scans every tool_use block in the assistant event without regard to order, so a
turn calling Read and Skill together scores a hit. What actually loses the
sample is needing a tool RESULT first — that decision lands in turn two, which
never runs. The practical advice is unchanged; the mechanism now matches the
collector.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SyBPs7caoMcyvnp9ArMBhV
Signed-off-by: cliffhall <cliff@futurescale.com>
Corrects the rationale for the five-positive floor, which was wrong. The
claim was that five prompts make a skill's rate flake-tolerant. The
evaluator scores each prompt on its own `passes / RUNS`, so the number of
prompts never enters any rate and cannot affect a threshold crossing —
`RUNS` is the only knob that does. The floor is worth keeping for
breadth: one prompt measures one phrasing, and a description that fires
on a single narrow shape would pass on its one lucky case.
The bad reasoning had reached five places — AGENTS.md, the authoring
checklist, the JSDoc, the check-site comment and the regression test —
plus the emitted message, which asserted it too. All corrected.
Moves `MIN_POSITIVE_CASES` above `validateEvalCases`'s JSDoc. It had been
inserted between the two, so the `@param`/`@returns` block documented the
constant and the function had none.
Replaces the two remaining cases answerable from `AGENTS.md`: `local-dev`'s
web-client-startup case (the rules file gives the exact command) and
`project-structure`'s clients/web-versus-core case (its tree answers it).
Both replacements probed 3/3 and measure 100%.
RUNS=5 skills:eval: 63/63, every local-dev and project-structure case at
100%. local:gate passes, 34/34.
Signed-off-by: cliffhall <cliff@futurescale.com>
Replaces the last `project-structure` case that `AGENTS.md` answers — a
helper used by the CLI and TUI belongs in `core/` per its own directory
tree. This was also the only case below 100% in the previous suite, so
the static reading and the score agreed. The replacement targets the
skill's first stated purpose, locating code, which `AGENTS.md` gives no
method for; it probed 3/3 and now measures 100%.
Extends the truncation guard's tests to `when_to_use`. The guard loops
over both fields but only `description` was pinned, so dropping the
second from that list would have gone unnoticed. The new case pairs a
quoted description with an unquoted `when_to_use` and asserts the
description is not implicated.
Scopes the negative-case requirement in the prose to model-invoked
skills; round 2 scoped the checklist but left this sentence saying
"every skill file".
Refreshes the stale `~2.8k of 4k` listing figure. The other instance is
past-tense about the three-to-nine experiment and stays, now said
explicitly so two numbers do not look like a disagreement.
Records the eval's real blind spot: all three cases review identified as
`AGENTS.md`-answerable were scoring 80-100% when flagged, so a high score
does not clear a case of overlap. The eval can show a case failing; it
cannot show a passing case is well-aimed.
RUNS=5 skills:eval: 63/63. local:gate passes, 34/34.
Signed-off-by: cliffhall <cliff@futurescale.com>
Enforces the five-positive standard this PR established rather than only
describing it. `validateEvalCases` checked `positives === 0`, so a skill
could regress to a single prompt and still pass the gate. At RUNS=5 one
case is 20 points of that skill's reading, so a one- or two-case skill
reports a rate a single flake can move across the threshold — well-formed
to the checker, meaningless as a measurement. Zero positives keeps its
own clearer message so the common mistake keeps its diagnosis.
Scopes the eval requirement to model-invoked skills in the checklist. A
name-only skill has no eval file because nothing can measure a trigger
only a human types, so an eval there is unrunnable rather than redundant.
Aligns checklist item 1 with the `#` hazard from round 1; it named only
the unquoted colon, which is the louder of the two failure modes.
Four test fixtures asserted on one-positive eval files and legitimately
went red; they now build positives from the exported MIN_POSITIVE_CASES.
The all-positive row keeps testing the missing-negative path, with a new
row for the below-floor case. 388/388 script tests pass.
No skill description or eval case changed, so trigger behaviour is
unaffected and skills:eval is unchanged from the previous round.
Signed-off-by: cliffhall <cliff@futurescale.com>
Fixes an unquoted `#` in `board-ops`'s description. YAML reads ` #28`
onward as a comment, so the parser — and Claude Code, which parses the
same way — only ever saw the description up to `Covers board`. Both
board numbers and the option-deletion hazard were absent from the
listing that decides whether the skill fires, and every check stayed
green because they look for an *empty* description and a truncated one
is still non-empty. Quoting restores all 271 characters, which is the
whole of the 3,144 vs 3,234 discrepancy the review flagged.
`parseSkill` now compares the raw scalar against what YAML kept and
errors on any loss, so the next one fails the gate rather than shipping.
Tests cover the truncated case and a legitimately quoted `#`.
Re-aims the `project-structure` Node-backend case, which `AGENTS.md`
answers via its tree and which had oscillated 80 -> 100% across runs —
the partial-overlap signature this PR documents. The replacement names
no internals and probed 4/4.
Makes the documented probe reproduce the harness: the same
`--disallowedTools` flags as `runPrompt` rather than `--allowedTools`,
`printf` instead of zsh's `print -r`, and `jq` over `tool_use` blocks
instead of a `grep` that also matched MCP server names in the init
event. The first-tool-call table is relabelled as an unrestricted probe,
since the eval cannot reach for Bash at all.
RUNS=5 skills:eval: 63/63, 62 at 100%; restoring those 90 characters
displaced nothing.
Signed-off-by: cliffhall <cliff@futurescale.com>
Adds a multi-name filter to `npm run skills:eval` so a set of related
skills can be iterated together, and makes an unknown name a hard error
rather than an empty run — a typo would otherwise report a green 0/0
that reads exactly like a clean pass of the skill that was meant.
Re-aims two `local-dev` cases and one `test-servers` case that partially
overlapped `AGENTS.md`. Partial overlap oscillates across runs rather
than sitting at zero, so it reads as eval noise instead of a defect in
the case; `docs/skill-authoring.md` now records that failure mode and
the observed scores.
Widens the `test-servers` description to the "Use when …" shape and
points the `testing` skill at it, since anything needing a real server
to run against is a procedure `testing` does not carry. That growth
moves the skill listing from 2,921 to 3,144 of 4,000 characters, so the
figure recorded in AGENTS.md is updated to match.
Signed-off-by: cliffhall <cliff@futurescale.com>
`skills:eval` was at 38/46 with several skills measuring 0-33%. The low
scores were mostly bad eval cases rather than bad descriptions, for two
reasons that were nowhere written down:
- The harness runs each case with `--max-turns 1`, so the skill has to be
the model's FIRST tool call. A prompt asking "how does X work here", or
naming a concrete file, sends the model to Read/Grep first — a better
answer to the question, scored as a miss. Every 0% case was one of these.
- `AGENTS.md` is resident on every turn, so a prompt it already answers is
answered without any skill and the case can never pass.
Replace the cases that measured the wrong thing, expand every model-invoked
skill from 3 to 5 positives (negatives kept) so one flake cannot cross the
threshold, and reshape `local-dev`'s description to the action-first form.
Add `docs/skill-authoring.md` with the case shapes that fire, the ones that
cannot, and the probe-then-measure loop.
RUNS=5 npm run skills:eval: 63/63 at or above threshold, 60 at 100%,
all 18 negatives clean.
Closes#2202
Signed-off-by: cliffhall <cliff@futurescale.com>
Brings in #2192 (react-hooks lint over `core/react/`) and #2194. All the code,
`eslint.config.js` and the new root devDependency merged cleanly; three
conflicts, all in files this PR refactored.
- `.github/copilot-instructions.md` — modify/delete again. Kept deleted; the
line v2/main added there restates AGENTS.md rules that arrive in AGENTS.md.
- `AGENTS.md` — two of the three incoming hunks merged cleanly, both of them
RULES, into the sections that survived the refactor: `exhaustive-deps` is
`error` in every React scope including `core/react/**`, and
`set-state-in-effect` now covers `core/react/` with `clients/tui` called out
as the exception. The third was a `validate:core` bullet inside the pre-push
gate section this PR moved wholesale to docs/quality-gate.md, so its mechanism
went there instead — including why the plugin has to be a ROOT devDependency
(Node resolution walks up, not down, so a client-local copy is unreachable).
- `README.md` — the incoming paragraph was added to the quality-gate section
this README no longer carries. Moved verbatim to docs/quality-gate.md, beside
the type-aware-lint paragraph it belongs with.
Ran `npm install` for the new devDependency; `npm run validate` passes.
Signed-off-by: cliffhall <cliff@futurescale.com>
Three conflicts, all in the files this PR refactored. The code from #1955 and
#2161 (core/react's useSyncExternalStore conversion) merged cleanly.
- `.github/copilot-instructions.md` — modify/delete. This PR deletes it, and
the four lines v2/main added there are a distillation of AGENTS.md rules that
arrive in AGENTS.md anyway. Kept deleted.
- `AGENTS.md` — the incoming change expanded the `core/react/` entry in the
project tree by eleven lines; this PR compacted that tree to one line per
entry on purpose. Took the compact tree and carried the substance onto its
one line. The other incoming hunk — the store-subscription rules under React
instructions — merged cleanly, which is the split working: it is a RULE, so
it lands in the section that stayed.
- `README.md` — same shape, one tree line. This README no longer enumerates
`core/`'s subdirectories at all, so the substance went where that content now
lives: `docs/architecture.md` and the `project-structure` skill's core table.
Nothing from v2/main was dropped; every incoming statement is now in exactly one
place rather than the three it used to occupy.
Signed-off-by: cliffhall <cliff@futurescale.com>
- skill-manifest: accept CRLF on the opening fence. A Windows checkout with
`core.autocrlf` has `---\r\n`, the fence is still the first line, and the
authoritative validator accepts it — so `npm run validate` was failing a valid
manifest, and failing it *before* that validator ever ran. The closing fence
now strips a preceding CR too, so the YAML handed to the parser is clean.
Regression cases for CRLF, mixed endings, and CRLF-with-no-body, plus cases
proving the first-line rule and the whole-line terminator rule still hold with
CRLF. Verified by mutation.
- docs/quality-gate: `verify:skills` requires eval cases only from model-invoked
skills; the seven name-only ones intentionally have none, and the summary read
as an instruction to add them everywhere.
- docs/quality-gate: the `test:scripts` inventory said four new suites;
`claude-cli.test.mjs` makes five.
Signed-off-by: cliffhall <cliff@futurescale.com>
- verify-skills: the best-effort `plugin validate` hand-off now also requires
the EXACT pinned CLI. `local:gate` runs `validate` before the pinned
`verify:skills:cli` step, so a newer local CLI could reject skills that CI's
pinned validator accepts — and the gate would exit there, never reaching the
reproducible check. One schema everywhere, or none.
- Dropped `PLUGIN_VALIDATE_MIN_VERSION`: with both validators on the exact pin
it was read only by its own test, and a constant nothing enforces is a claim
nothing enforces. Its test now pins what actually matters — that equality is
distinguishable from newer as well as older.
- The CI step's comment described the old floor behavior.
- issue-create: the duplicate check searched only open issues, so a completed or
rejected match was invisible to a procedure that says never to duplicate.
Searches all states and reports state/stateReason.
Signed-off-by: cliffhall <cliff@futurescale.com>
- verify-skills-cli: use a local CLI only when it matches the pin EXACTLY. A
floor let a maintainer on a newer CLI validate against a different schema than
CI's, so the same `local:gate` could disagree across machines — the failure a
pin exists to prevent.
- verify-skills-cli: extract `runValidator({probe, spawn, log, error})` and
cover the orchestration — validator accepts, validator rejects, spawn fails,
and the three unusable-probe shapes. On a machine with a working CLI an actual
run only ever walks the happy path.
- pr-flow: stop putting the `gh` token in curl's argv, where any local process
can read it off the process table. Feed the header through `curl --config -`.
- docs/quality-gate: three places had drifted — the CI tier row, the
`local:gate` chain, and the `test:scripts` inventory (which now names all four
suites this PR adds, not two).
Signed-off-by: cliffhall <cliff@futurescale.com>
- verify:skills:cli — the authoritative validator is now a GUARANTEED step in
`local:gate` and CI, not a best-effort hand-off. It resolves the CLI itself
(an installed one at 2.1.233+, else a pinned `npx -y`), so both tiers run the
same script and cannot drift. `verify:skills` keeps skipping when the CLI is
absent, which is right for `validate`: fast, offline, no Claude Code required.
- skill-manifest: the 1,536-char listing-entry cap covers `description` +
`when_to_use` together; checking the description alone passed an over-cap
entry. `when_to_use` is now parsed, validated and counted, with boundary tests
at 1,536/1,537.
- verify-skills: report the budget as *this repo's contribution*. The real
listing budget is shared with bundled skills and the contributor's own
~/.claude/skills, none of which is visible from here, so the number was
claiming more than it measured.
- skill-eval: a negative case now means "no skill OF THIS REPO'S fired". It
failed on any Skill invocation, so a contributor with an unrelated global
skill got a false failure about their environment rather than these skills.
- docs/architecture + local-dev: the Node backend runner gets its
`@inspector/core` alias from `clients/web/tsup.runner.config.ts`, not from
Vite. The old text sent runner work to the wrong config.
- local-dev: the tarball does not ship "only each client's build/" — the
allowlist also carries `clients/web/dist`, `clients/web/static` and the
install helper. States the invariant that actually matters instead.
- issue-triage: the test for needing triage is "no board card". Adding "and no
milestone" would skip the approved-but-unboarded case the same section then
handles. Also corrected the `$M` note: the lookup is `--state all`, so closed
issues DO have entries — `isopen` is about the invariants, not the lookup.
- issue-create: replaced "all five" with a per-version table. Steps 3 and 6
(milestone, Priority) do not exist for v1, so the old list opened with a rule
the same skill later tells you to violate.
Signed-off-by: cliffhall <cliff@futurescale.com>
- CI now installs a pinned `claude` CLI and runs `claude plugin validate` as its
own step. The guard skips the authoritative validator when the CLI is absent,
which without this meant "always, in CI" — so the acceptance criterion was
aspirational. Verified it needs no authentication (clean HOME).
- skill-eval: classify a run from its terminal `result` event, not its exit
code. With `--max-turns 1` a run in which a skill FIRES necessarily hits the
limit and the CLI exits 1 (`error_max_turns`, `num_turns: 2`), so last round's
blanket nonzero rejection threw away exactly the observations the eval counts.
A run with no terminal `result`, or an unusable subtype, still rejects.
- skill-eval: fail collection on a skill that does not parse, a model-invoked
skill with no eval file, unreadable JSON, or invalid cases. Skipping them let
`skills:eval` exit 0 while silently omitting the skill just broken.
- verify-skills: `main()` takes an optional skills directory, and
`verify-skills.main.test.mjs` drives it over fixtures — missing SKILL.md,
malformed frontmatter, undeclared invocation mode, missing/invalid evals,
budget overflow (and that name-only skills do not count toward it), and
multi-offender reporting. Matches the sibling guards' *.main.test.mjs
precedent; without it the gate could stop enforcing while the parser tests
stayed green.
- issue-create: the note claimed an issue arrives unlabeled, contradicting
.github/ISSUE_TEMPLATE — blank issues are disabled, the bug form applies
`bug`, the feature form applies `enhancement` + `v2`. Now a table of what each
form sets and what triage still has to add.
Signed-off-by: cliffhall <cliff@futurescale.com>
- skill-manifest: the closing `---` fence must occupy a whole line. `---oops`
was accepted as a terminator, silently truncating the frontmatter and passing
validation — the exact failure the guard exists to catch. Regression cases
added (prefix, trailing whitespace, EOF, CRLF).
- skill-eval: reject a nonzero `claude -p` exit instead of resolving it as an
empty observation. An auth error or rate limit was scoring as "no skill
invoked", which passes every negative case and reads as a trigger miss on
every positive one.
- skill-eval: extract `collectSkillInvocations` / `sampleHit` behind a `main()`
guard and cover them in `skill-eval.test.mjs` — the stream parsing and the
process-outcome handling are unreachable from an eval run's happy path.
- docs/test-servers: annotate the era on every catalogue row. Six legacy rows
carried no annotation under a "unless noted, Modern" default, so the doc sent
readers to the wrong era for the feature they were trying to see.
- docs/quality-gate: `validate` runs four guards, not three; add the
`verify:skills` row and the two new `test:scripts` suites; fix the `#publishing`
fragment left dangling by the split.
- issue-create / pr-flow: delegate the card steps to `/board-ops` rather than
restating its option IDs, which are explicitly unstable.
- issue-triage: the audit counted presence, not cardinality — an issue carrying
both `v1` and `v2` passed a check that claims "exactly one". Same for the five
type labels. Also scope the two milestone checks to board #28.
- AGENTS.md: codify the v1 milestone exception. Every milestone is a v2 release
bucket, so a v1 issue has none to take and the Incoming-milestone invariant
cannot apply to board #11.
- clients/launcher/README: its pointer promised an explanation of why the cli/tui
manifests ship, which the extracted guide does not make.
Signed-off-by: cliffhall <cliff@futurescale.com>
Closes#2163
`AGENTS.md` was one 1,223-line contract loaded in full on every turn, whether
the task was a one-line CSS fix or a release cut. This splits it: rules stay
resident, procedures move to `.claude/skills/`.
- Ten skills under `.claude/skills/`, seven `disable-model-invocation: true`
(invoked by name only, so they cost no listing context and have no trigger
question) and three model-invoked with committed eval cases.
- `AGENTS.md` 1,223 → 380 lines: rules, a skills index, and a project tree
reduced to one line per entry. The removed recipes are not duplicated.
- `README.md` 811 → 105 lines, split into `docs/{architecture,quality-gate,
test-servers,publishing,docker}.md`, and dropped from `CLAUDE.md`'s auto-load.
- `.github/copilot-instructions.md` and its mirroring rule removed — Copilot
reads `AGENTS.md`.
- `npm run verify:skills` (in `validate`, so in `local:gate` and CI) parses each
SKILL.md the way Claude Code does, requires an explicit invocation mode and
eval cases for model-invoked skills, and budgets the listing.
- `npm run skills:eval` runs the committed trigger cases headless.
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 15.
The issuer check still failed open for the legacy unkeyed grant. "Unkeyed"
means its authorization server is UNKNOWN, not that the cached endpoint is
proven to own it: server metadata is a single slot a later discovery
overwrites, while a pre-SEP-2352 token survives until the first issuer-stamped
save. So a clear could post an old bearer credential to an authorization server
that never minted it.
Such a grant is now cleared without being revoked, and reported as such with a
message saying re-authorizing makes future clears revocable. The cost is that
a pre-issuer-binding entry gets no revocation — which is exactly the behavior
it had before this feature existed, so nothing regresses — and it earns
revocation back on the next authorization. Not sending is the only answer that
cannot be wrong.
Every seed in the suite is now issuer-bound, which is what a real flow writes;
the two tests that were about the legacy grant now assert the refusal, since
that is the behavior worth pinning.
Also renamed the CLI test that still described the pre-round-13 ordering.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177mPHAdECD18nLLCTwR5rh
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 14. Two code fixes and a documentation sweep.
- `buildRevocationRequest` ran outside `revokeToken`'s best-effort try.
`encodeURIComponent` throws on a lone UTF-16 surrogate — valid JSON, so it
can reach there from a persisted client id or secret — and every caller has
already cleared its local state by then, so a rejection instead of a `failed`
outcome would break the documented guarantee. Construction is inside the try
now, and the catch reports `params.endpoint` rather than a `url` that may not
exist on that path.
- The fixture's Basic decode turned a malformed percent escape into an Express
500, reporting a bad credential as a server fault rather than as the
`invalid_client` 401 the endpoint means. Decoding failure is failed
authentication.
- Ten places still said revocation happens *before* the clear, which the
reorder in the previous commit made false: the settings-form description, the
`--no-revoke` help text, both READMEs, `docs/mcp-server-configuration.md`,
three JSDoc blocks, and the PR body. Swept, and the ordering is stated where
it is load-bearing rather than asserted in passing. Also merged the two
adjacent JSDoc blocks on `InspectorClient.clearOAuthTokens`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177mPHAdECD18nLLCTwR5rh
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 13. Three of its four findings were one root cause, and
it is a design flaw in the premise this PR started from.
Revoking *before* the clear meant the clear waited up to five seconds on the
network — and anything that wrote OAuth state in that window was then deleted
by a clear reasoning about the grant it replaced. A same-server reauthorization
in the TUI (whose Disconnect/Connect accelerators stay live), a stored-only web
clear for a server that becomes active mid-flight, and another CLI/TUI process
writing to the shared file-backed store all hit it.
`revokeStoredOAuthTokens` is therefore split into `planOAuthRevocation` and
`executeOAuthRevocation`, and every caller runs plan → `storage.clear()` →
execute. The snapshot still precedes the clear, because the store is where the
token, the credentials and the endpoint live; but the clear no longer waits on
anything. There is deliberately no single-call wrapper left, so no caller can
reintroduce the racy order by accident.
Fourth finding, same file: `handleConnect` now retires `clearOAuthAttemptRef`
alongside `disconnectAttemptRef`. The server-name check does not cover
disconnect/reconnect to the SAME server — the name still matches on the other
side of it — so a clear still in flight could tear down the session the connect
had just established.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177mPHAdECD18nLLCTwR5rh
Signed-off-by: cliffhall <cliff@futurescale.com>
Clearing the Inspector's OAuth state deleted the local copy of the tokens
and stopped there. The access token — and the refresh token, which is
long-lived by design — stayed valid at the authorization server until they
expired on their own, so a day of connect/disconnect iteration left it
holding grants for sessions that ended hours ago. RFC 7009 §1 describes
exactly that case, and nothing in the repo constructed a revocation
request, so a server author implementing `revocation_endpoint` had no way
to exercise it with the tool that exists to exercise this.
core/auth/revocation.ts builds and sends the request; the three clear paths
call it, so the behavior is written once — web's "Clear OAuth state and
disconnect" (both the active and stored-only paths), the TUI's Auth tab,
and the CLI's --relogin. It runs BEFORE the local clear, because the token,
the client credentials and the discovered `revocation_endpoint` all live in
the store the clear is about to empty. The request names the refresh token
when there is one: §2.1 asks the AS to also invalidate the access tokens
issued under the same grant, so one request covers both halves.
Best-effort by construction. Every path returns a TokenRevocationOutcome
rather than throwing: no advertised endpoint means nothing is sent and that
authorization server behaves exactly as before, and a network error, a
non-2xx, a 5s timeout or an unreadable store is reported and nothing more.
Forgetting the tokens is what the user asked for. `lost_authorization_state`
recovery deliberately skips it — it clears a half-finished flow to retry it,
and an authorization that never completed has no grant to revoke.
Client authentication resolves the preregistered slot first, mirroring
BaseOAuthClientProvider.clientInformation; reading only the dynamic slot
would send no authentication for a server configured with `oauth.clientId`.
The Basic credential is byte-identical to the SDK's applyBasicAuth so this
request and the token request cannot present the same secret differently.
Opt out per server with `oauth.revokeOnClear` (default on, only `false`
written to disk) or per CLI run with --no-revoke. Read at clear time rather
than connect time, so toggling needs no reconnect. Turning it off is a
testing affordance: a client that walks away still holding live tokens is a
case a server author may want to reproduce.
test-servers now advertises `revocation_endpoint` and serves POST
/oauth/revoke, faithful on §2.2 (an unknown token is a success) and §2.1
(revoking a refresh token invalidates the access tokens under the same
grant), with oauth-revocation-http.json / oauth-no-revocation-http.json as
showcase configs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177mPHAdECD18nLLCTwR5rh
Signed-off-by: cliffhall <cliff@futurescale.com>
The root `ci` script was never what runs in GitHub CI, and the name worked
against us three ways: it collides with the built-in `npm ci` (which does
something else entirely and is one keystroke away), it invites the
cross-engine passes into a workflow by looking like the thing a workflow
should call, and it blurs "GitHub CI" and "the local gate" in every doc
that names it.
- Rename `ci` → `local:gate` and `ci:storybook` → `local:storybook`, with
no back-compat alias: `npm run ci` now fails with npm's missing-script
error, which prints the available scripts.
- Add `scripts/lib/workflow-gate.mjs` + its `test:scripts` suite, which
runs over the repo's real `.github/workflows/**` and fails on an
invocation of a `local:*` script, of a non-Chromium engine pass (derived
from SUPPORTED_BROWSERS), or of `smoke:web:engine`, or on `SMOKE_BROWSER`
set to anything but a literal `chromium`. It does not forbid
`npm run smoke`, `smoke:web:chromium` or `smoke:tui` — those belong in CI
and are there today. Comments are not scanned, so the workflow can still
explain itself. This is the durable half; the rest is prose.
- Document the two-tier model once, in the root README ("Two tiers: GitHub
CI and the local gate"), and point the other docs at it rather than
restating the split.
- Sweep the ~40 stale references across README, AGENTS.md,
.github/copilot-instructions.md, the client READMEs, the roadmap doc and
the script headers. AGENTS.md and copilot-instructions.md change
together, per the mirror rule.
No gate changes what it actually runs.
Signed-off-by: Cliff Hall <cliff@futurescale.com>
Signed-off-by: cliffhall <cliff@futurescale.com>
The self-healing claim was still overstated after b339ad49 fixed the
code: storage is untouched at filter time, but it does update on the
next successful grant, including when the token response omits `scope`.
Markdown only — prettier ran clean; pushed without the full gate on the
maintainer's call, since `npm run ci` checks formatting rather than
prose.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013hzwzS8UvBsr4yZm7o7sev
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 6 on #2114.
The early return handed back the persisted scope unchanged, so an
`offline_access` the user had just typed into Scopes never reached the
request when storage predated the change — and storage routinely does,
since `createOAuthProvider` preserves a stored scope rather than
reseeding from current settings. The docs promised configured scopes
are honored and the form warned that `offline_access` "is still
requested"; both were false in exactly that state.
Honoring it means adding it, not merely declining to strip it.
Only the token itself is added, not the rest of the configured scope:
unioning unrelated configured tokens into the request is outside what
this setting governs and could push an otherwise-satisfied connection
into a step-up authorization.
Three tests, all verified to fail against the previous behavior.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013hzwzS8UvBsr4yZm7o7sev
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 5 on #2114 — a hole in last round's fix.
When the persisted scope was `offline_access` alone,
`scopeForDeclinedRefreshGrant` returned undefined.
`OAuthManager.createOAuthProvider` reads that as "nothing stored" and
seeds storage with the configured scope, so the request-only filter
silently overwrote the value it promised to leave alone — and
re-enabling the grant could no longer restore it.
It now falls back to the configured scope, which is both the right
thing to request and inert to that branch. The previous "storage is
never rewritten" test could not see this: it ran at the provider level,
while the seeding branch lives in the manager. The new regression runs
there, and fails when the fallback is removed.
Also corrects the EMA rationale, which was wrong even though its
conclusion held. `createOAuthProviderForTransport` does build a
`BaseOAuthClientProvider` and wrap it, and `EmaTransportOAuthProvider`
forwards `clientMetadata` — so the setting is not bypassed outright.
What it cannot change is the IdP authorization leg's fixed
`openid offline_access`, which is the leg the user signs in through and
therefore the reason the control cannot affect EMA's consent behavior.
Fixed in the docs, the code comment, the UI string and the test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013hzwzS8UvBsr4yZm7o7sev
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 4 on #2114 — a real gap, not a wording one.
Declining the grant stops the SDK *adding* `offline_access`, but the
Inspector's own storage puts it back. A successful default-on
authorization persists the scope the AS granted
(`resolvePersistedScopeAfterGrant`), `offline_access` included;
`prepareForAuth` reloads it and `authenticate` hands it to the SDK as
`scope`, which sees the token in scope and appends `prompt=consent`
again. So any server that had authorized once could uncheck the box and
still hit AADSTS90094 — the users most likely to reach for the setting
were the ones it did not help.
`scopeForDeclinedRefreshGrant` filters it out of the *request* only,
applied in the provider's `scope` getter — the one seam both
`clientMetadata.scope` and `authenticate`'s `scope:` argument read
through. An `offline_access` the user configured is honored; storage is
never rewritten, so re-checking the box restores the old behavior
instead of finding the value destroyed, and scope-satisfaction checks
still see the real grant. Verified by reverting the filter: the new e2e
regression fails.
Also from that round, both previously suppressed:
The checkbox looked effective under enterprise-managed authorization,
which never builds this provider at all — it authorizes against the
enterprise IdP with a fixed `openid offline_access`
(`ema/constants.ts:16`). It now annotates itself the way the endpoint
overrides do. The existing count assertion covered two such notices and
was named for the overrides alone; it is renamed and expects three, with
a dedicated case pinning the notice to this control so the count cannot
be met by three copies elsewhere.
And "obtains no new refresh token" was another overstatement — the AS
decides issuance, and a preconfigured client is reused with whatever its
registration already permits. Replaced with what the setting actually
controls.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013hzwzS8UvBsr4yZm7o7sev
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review round 2 on #2114. The setting was described as
guaranteeing "no offline_access, no forced prompt", which is not what
the SDK does.
`startAuthorization` appends `prompt=consent` for any `offline_access`
in the effective scope and never consults `grant_types`, and
`determineScope` skips its augmentation when the scope already carries
it. So the prompt still goes out when the user configured
`offline_access` in Scopes, or left Scopes blank and the resource
advertises it in `scopes_supported`. The flag removes the SDK's
*automatic* augmentation and nothing more; the docs, the help text and
the type say that now.
The form also flags the case a user can act on: opted out while
`offline_access` is still listed in Scopes. Whole-token match, so
`offline_access_extra` does not trip it.
Corrects the Clear stored OAuth state claim too. It clears the local
tokens and client information — it is not a revocation and cannot
reach the AS. A dynamically registered client re-registers on the next
connect (the new registration declaring only `authorization_code`); a
preconfigured `oauth.clientId` is reused unchanged.
Adds the two missing seam tests, both confirmed to fail without the
code they cover: App's connect wiring (deleting the spread fails two of
the three) and an /api/servers POST followed by a read-back, since a
200 alone never proved the opt-out persisted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013hzwzS8UvBsr4yZm7o7sev
Signed-off-by: cliffhall <cliff@futurescale.com>
Copilot review on #2114. The setting was described as taking effect
"immediately", which is wrong twice over.
`applyLiveServerSettings` treats OAuth as connection-time-only, so a
connected client keeps the metadata it was built with — deliberately,
and identically to every other field in that section (client id,
secret, scopes, authorizationParams, endpoint overrides). Not wiring
this one field live; saying "next connect" instead, in the docs, the
checkbox description, and the type.
And two pieces of state outlive the setting: the registration already
held at the authorization server still lists the grant, and a refresh
token issued earlier stays usable, because the SDK's
`refreshAuthorization()` posts the stored token without consulting
`grant_types`. So the stated cost is not guaranteed until stored OAuth
state is cleared. Documented, rather than revoking credentials as a
side effect of a checkbox — Clear stored OAuth state already does that
explicitly.
Also adds the manager->provider seam test the other bridges have:
without it, deleting `requestRefreshToken:` from `createOAuthProvider`
would leave the feature dead with every other test green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013hzwzS8UvBsr4yZm7o7sev
Signed-off-by: cliffhall <cliff@futurescale.com>
