chore: address Copilot review round 1
- skill-manifest: the closing `---` fence must occupy a whole line. `---oops` was accepted as a terminator, silently truncating the frontmatter and passing validation — the exact failure the guard exists to catch. Regression cases added (prefix, trailing whitespace, EOF, CRLF). - skill-eval: reject a nonzero `claude -p` exit instead of resolving it as an empty observation. An auth error or rate limit was scoring as "no skill invoked", which passes every negative case and reads as a trigger miss on every positive one. - skill-eval: extract `collectSkillInvocations` / `sampleHit` behind a `main()` guard and cover them in `skill-eval.test.mjs` — the stream parsing and the process-outcome handling are unreachable from an eval run's happy path. - docs/test-servers: annotate the era on every catalogue row. Six legacy rows carried no annotation under a "unless noted, Modern" default, so the doc sent readers to the wrong era for the feature they were trying to see. - docs/quality-gate: `validate` runs four guards, not three; add the `verify:skills` row and the two new `test:scripts` suites; fix the `#publishing` fragment left dangling by the split. - issue-create / pr-flow: delegate the card steps to `/board-ops` rather than restating its option IDs, which are explicitly unstable. - issue-triage: the audit counted presence, not cardinality — an issue carrying both `v1` and `v2` passed a check that claims "exactly one". Same for the five type labels. Also scope the two milestone checks to board #28. - AGENTS.md: codify the v1 milestone exception. Every milestone is a v2 release bucket, so a v1 issue has none to take and the Incoming-milestone invariant cannot apply to board #11. - clients/launcher/README: its pointer promised an explanation of why the cli/tui manifests ship, which the extracted guide does not make. Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall
parent
73447c72d1
commit
a51e62dfa1
12 changed files with 248 additions and 83 deletions
|
|
@ -89,18 +89,16 @@ in **Todo** with its milestone already set — not in Incoming, which is the que
|
|||
for issues nobody has evaluated yet. Work you are starting immediately goes
|
||||
straight to **In Progress**.
|
||||
|
||||
```sh
|
||||
ITEM_ID=$(gh project item-add 28 --owner modelcontextprotocol --url <issue-url> --format json --jq '.id')
|
||||
# Status → Todo
|
||||
gh project item-edit --project-id PVT_kwDOCt2Azc4BJVxt --id "$ITEM_ID" \
|
||||
--field-id PVTSSF_lADOCt2Azc4BJVxtzg5iI8c --single-select-option-id fbdaf21e
|
||||
# Priority → Medium (score it with the rubric in /issue-triage; don't eyeball it)
|
||||
gh project item-edit --project-id PVT_kwDOCt2Azc4BJVxt --id "$ITEM_ID" \
|
||||
--field-id PVTSSF_lADOCt2Azc4BJVxtzg5iJE4 --single-select-option-id da944a9c
|
||||
```
|
||||
Run the **add-card recipe in `/board-ops`** with **Status → Todo** and the
|
||||
Priority you scored. The project, field and option IDs live there and only
|
||||
there: an option ID is regenerated whenever its field's option list is edited,
|
||||
so a second copy here would go stale silently and break issue creation even
|
||||
after `board-ops` was fixed.
|
||||
|
||||
Full ID tables, the v1 board's recipe, and the option-deletion hazard are in
|
||||
`/board-ops`. The priority rubric is in `/issue-triage`.
|
||||
For a **v1** issue, the same recipe against board #11 — no Priority, that board
|
||||
has no such field.
|
||||
|
||||
The priority rubric is in `/issue-triage`.
|
||||
|
||||
## Note on issues that arrive from elsewhere
|
||||
|
||||
|
|
|
|||
|
|
@ -200,11 +200,11 @@ count means the board contradicts a rule, not that the rule needs revisiting.
|
|||
| Double-boarded | An issue has a card on **one** board, the one matching its version label | Delete the wrong-board card |
|
||||
| Non-Issue items | **Only issues go on a board** — never PRs, never drafts | Delete the item |
|
||||
| No Status | Every card carries a Status | Set one — `Incoming` if unmilestoned, else by where it actually is |
|
||||
| `Incoming` **with** a milestone | Incoming ⇔ no milestone | Approval was never recorded: move to **Todo**, or clear the milestone |
|
||||
| Past Incoming **without** a milestone | Everything past Incoming ⇔ milestoned | Claims an approval nobody made: milestone it, or move back to Incoming |
|
||||
| `Incoming` **with** a milestone (#28) | Incoming ⇔ no milestone | Approval was never recorded: move to **Todo**, or clear the milestone |
|
||||
| Past Incoming **without** a milestone (#28) | Everything past Incoming ⇔ milestoned | Claims an approval nobody made: milestone it, or move back to Incoming |
|
||||
| Wrong board for label | `v1` → #11, `v2` → #28 | Move the card to the right board |
|
||||
| No version label | Every issue carries exactly one of `v1`/`v2` | Apply it |
|
||||
| No type label | Every issue carries one of the five types | Classify it |
|
||||
| Not exactly one version label | Every issue carries **exactly one** of `v1`/`v2` | Apply the missing one, or remove the extra |
|
||||
| Not exactly one type label | Every issue carries **exactly one** of the five types | Classify it, or remove the extra |
|
||||
| No Priority (#28) | Every board item is prioritized | Score it with the rubric |
|
||||
| Closed, not shipped, still carded | **Done means the work shipped** | Delete the card |
|
||||
|
||||
|
|
@ -231,17 +231,21 @@ jq -nr --slurpfile o "$D/i.json" --slurpfile a "$D/b28.json" --slurpfile b "$D/b
|
|||
"double-boarded": [$B28[].n | select(. as $n | [$B11[].n]|index($n))],
|
||||
"non-Issue on a board": [(own($a)[], own($b)[]) | select(.content.type!="Issue") | .content.number],
|
||||
"no Status": [($B28[], $B11[]) | select(.s==null) | .n],
|
||||
"Incoming w/ milestone": [($B28[], $B11[]) | select(.s=="Incoming" and ms(.n)!=null) | .n],
|
||||
"past Incoming, no ms": [($B28[], $B11[]) | select(.s!=null and .s!="Incoming" and .s!="Done"
|
||||
and isopen(.n) and ms(.n)==null) | .n],
|
||||
"Incoming w/ milestone": [$B28[] | select(.s=="Incoming" and ms(.n)!=null) | .n],
|
||||
"past Incoming, no ms": [$B28[] | select(.s!=null and .s!="Incoming" and .s!="Done"
|
||||
and isopen(.n) and ms(.n)==null) | .n],
|
||||
"v1 label on #28": [$B28[] | select(isopen(.n) and (lab(.n)|index("v1"))) | .n],
|
||||
"v2 label on #11": [$B11[] | select(isopen(.n) and (lab(.n)|index("v2"))) | .n],
|
||||
"open, no version label":[$o[0][] | select(.state=="OPEN")
|
||||
| select(([.labels[].name]|index("v1") or index("v2"))|not) | .number],
|
||||
"open, no type label": [$o[0][] | select(.state=="OPEN")
|
||||
| select(([.labels[].name] | index("bug") or index("enhancement")
|
||||
or index("documentation") or index("chore")
|
||||
or index("question"))|not) | .number],
|
||||
"open, not exactly 1 version label":
|
||||
[$o[0][] | select(.state=="OPEN")
|
||||
| select(([.labels[].name] | map(select(IN("v1","v2"))) | length) != 1)
|
||||
| .number],
|
||||
"open, not exactly 1 type label":
|
||||
[$o[0][] | select(.state=="OPEN")
|
||||
| select(([.labels[].name]
|
||||
| map(select(IN("bug","enhancement","documentation","chore","question")))
|
||||
| length) != 1)
|
||||
| .number],
|
||||
"#28 open, no Priority": [$B28[] | select(.p==null and isopen(.n)) | .n],
|
||||
"closed unshipped, still carded":
|
||||
[($B28[], $B11[]) | select(I(.n)!=null and (isopen(.n)|not)
|
||||
|
|
@ -256,6 +260,14 @@ Two things the queries must account for, both learned the hard way:
|
|||
`modelcontextprotocol/servers` issue. Without the `.content.repository` filter
|
||||
it reads as a statusless-card defect, and "fixing" it would mean editing
|
||||
another repo's tracking.
|
||||
- **The two milestone checks are #28-only.** Every milestone in this repo is a
|
||||
v2 release bucket, so a `v1` issue has none it could take — running the
|
||||
Incoming⇔milestone invariant over board #11 would flag every card on it for a
|
||||
state it cannot reach.
|
||||
- **Count the labels; don't test for presence.** The invariant is *exactly
|
||||
one*, so a predicate that only asks "is any version label present" passes an
|
||||
issue carrying **both** `v1` and `v2` — which belongs to two lines at once
|
||||
and shows up in both filtered queries. Same for the five type labels (Copilot).
|
||||
- **Only open issues have a `$I` entry.** A closed issue still has a card
|
||||
(correctly, in `Done`), so a check that treats "no milestone found" as a
|
||||
violation must gate on `isopen(.n)` or it flags every closed card.
|
||||
|
|
|
|||
|
|
@ -173,7 +173,9 @@ Poll for the review with a `startswith` match — the review login carries a
|
|||
## 9. Merge and close out
|
||||
|
||||
**On merge of a v2 PR, manually close its issue and move the board item to
|
||||
Done** (`259d6aab`), since auto-close won't fire on `v2/main`.
|
||||
Done**, since auto-close won't fire on `v2/main`. Use the move-a-card recipe in
|
||||
`/board-ops` — the option IDs are unstable, so this file names the column and
|
||||
nothing else.
|
||||
|
||||
`Done` is only for work that **shipped** — a merged PR, or a parent whose last
|
||||
sub-issue closed. Anything else (duplicate, won't fix, not planned, obsolete)
|
||||
|
|
|
|||
|
|
@ -43,15 +43,28 @@ the cache.
|
|||
|
||||
## Run one
|
||||
|
||||
Two processes: the test server, then the Inspector.
|
||||
|
||||
```sh
|
||||
# From the repo root, with a built launcher:
|
||||
node clients/launcher/build/index.js --web
|
||||
# then in the Inspector, add the server the config announces
|
||||
# 1. The server, from the repo root, with the config you picked:
|
||||
node test-servers/build/server-composable.js --config test-servers/configs/<name>.json
|
||||
```
|
||||
|
||||
The server prints its URL on **stderr**, and the bound port is **not
|
||||
necessarily** the config's — `findAvailablePort()` walks upward when the
|
||||
configured port is taken. Parse the announced URL rather than assuming.
|
||||
```sh
|
||||
# 2. The Inspector, in another terminal (needs a built launcher — `npm run build`):
|
||||
node clients/launcher/build/index.js --web
|
||||
```
|
||||
|
||||
Then add the server in the Inspector using the URL the first process announced.
|
||||
|
||||
Two mechanics that bite:
|
||||
|
||||
- **The server announces its URL on _stderr_**, not stdout (`console.error` in
|
||||
`server-composable.ts`). Watching stdout alone looks like a server that never
|
||||
started.
|
||||
- **The bound port is not necessarily the config's.** `createTestServerHttp`
|
||||
resolves through `findAvailablePort()`, which walks upward when the configured
|
||||
port is taken — so read the announced URL rather than assuming.
|
||||
|
||||
## Pick the right protocol era
|
||||
|
||||
|
|
|
|||
|
|
@ -211,9 +211,9 @@ skills; the rules are here.
|
|||
- **Only issues go on a board — never PRs.** A PR gets the `v2` label but is tracked through its linked issue's card (via `Closes #N`), not its own board item.
|
||||
- **Label by version — every issue and every PR, no exceptions.** Exactly one of `v1` (work targeting `v1/main`, the deprecated security-fix-only line) or `v2` (active development; the default for anything new). There is no unlabeled state and no "decide later": an issue with neither label belongs to no version line and is invisible to every version-filtered query. Set it at **create time** (`gh issue create --label v2 …`), never by backfilling. **If the target version isn't obvious, it's `v2`.**
|
||||
- **Label by type — exactly one of `bug` / `enhancement` / `documentation` / `chore` / `question`** on every issue you create or triage. The version label says which line the work belongs to; the type label says what kind of work it is, and the two are independent. Don't force the binary: pressing a docs task or a dependency pin into `enhancement` degrades it to "not a bug", at which point filtering by it stops telling you anything. A **PR** needs no type label — it is classified through the issue it closes.
|
||||
- **Every issue you create gets a milestone.** Milestones are *release* buckets, so pick by when the work ships. Never leave an issue you filed unmilestoned pending a decision. (An issue that arrives **unboarded** is the deliberate exception: it stays unmilestoned in `Incoming` until a maintainer approves it — there, the *absence* of a milestone is the signal.)
|
||||
- **Every v2 issue you create gets a milestone.** Milestones are *release* buckets, so pick by when the work ships. Never leave a v2 issue you filed unmilestoned pending a decision. Two exceptions, both deliberate: an issue that arrives **unboarded** stays unmilestoned in `Incoming` until a maintainer approves it — there, the *absence* of a milestone is the signal; and **every milestone is a v2 release bucket, so a `v1` issue has none to take**. Say so when filing one rather than dropping it in a v2.x bucket.
|
||||
- **Every v2 board item has a Priority.** Priority is a **board field**, not a label, so an unboarded issue has nowhere to store it. Derive it with the rubric in the `issue-triage` skill rather than asserting it. Board #11 has no Priority field; a v1 issue gets a Status and nothing else.
|
||||
- **`Incoming` ⇔ no milestone; everything past it ⇔ milestoned.** Assigning the milestone *is* the approval act, so the two always go together. `Todo` asserts a maintainer signed off, so never park an unreviewed issue there — that erases the distinction and quietly promotes unreviewed work into the queue. An issue created through the documented flow skips `Incoming` entirely, because filing it *was* the approval.
|
||||
- **`Incoming` ⇔ no milestone; everything past it ⇔ milestoned — on board #28.** Board #11 is exempt for the reason above: a v1 issue has no bucket to take, so its Status is set on its own and the audit's milestone checks do not apply to it. The rest of the invariant is unchanged: assigning the milestone *is* the approval act, so the two always go together. `Todo` asserts a maintainer signed off, so never park an unreviewed issue there — that erases the distinction and quietly promotes unreviewed work into the queue. An issue created through the documented flow skips `Incoming` entirely, because filing it *was* the approval.
|
||||
- **`Done` means the work shipped.** Exactly two things earn a card a place in Done: its **PR merged**, or it is a **parent whose last sub-issue closed**. Anything else — duplicate, won't fix, not planned, obsolete, superseded — means nothing shipped, so the card is **deleted**. Done is read as the record of what a milestone actually delivered; a duplicate sitting there makes that record wrong in a way nobody can detect later. Deleting a card touches the board only — the issue keeps its labels and comments and stays searchable forever.
|
||||
- **When work begins**, create a feature branch and set Status to **In Progress**. **Branch names start with the target version segment** — `v2/fix/2071-oauth-resource-metadata`, `v1/fix/proxy-ssrf-pin` — matching the base branches themselves.
|
||||
- **When work is complete**, run `npm run format` then `npm run local:gate`, **sign off every commit** (`git commit -s` — the DCO check is a hard merge gate with no partial credit), open a PR against the matching base branch with **`Closes #<ISSUE_NUMBER>` as the body's first line**, and set Status to **In Review**.
|
||||
|
|
|
|||
|
|
@ -120,7 +120,7 @@ repo-root `validate:launcher` simply delegates here (`cd clients/launcher && npm
|
|||
|
||||
## Publishing
|
||||
|
||||
The launcher provides the `mcp-inspector` bin for the single `@modelcontextprotocol/inspector` tarball. Packaging is a whole-repo concern — how the one-package/single-version tarball is assembled, the `"files"` allowlist invariants (no source maps, why `clients/web/build` needs `.npmignore`, why the cli/tui `package.json`s ship), and the `npm run pack:verify` publish smoke — is documented in the [Publishing](../../docs/publishing.md).
|
||||
The launcher provides the `mcp-inspector` bin for the single `@modelcontextprotocol/inspector` tarball. Packaging is a whole-repo concern — how the one-package/single-version tarball is assembled, the `"files"` allowlist invariants (no source maps, why `clients/web/build` needs `.npmignore`, why `clients/web/static` must ship at that exact path), and the `npm run pack:verify` publish smoke — is documented in [Publishing](../../docs/publishing.md).
|
||||
|
||||
## Architecture
|
||||
|
||||
|
|
|
|||
|
|
@ -29,17 +29,18 @@ That is the readable half, and prose rots. The enforced half is `scripts/lib/wor
|
|||
|
||||
| Script | What it does |
|
||||
| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `npm run validate` | Runs the three durable guards first — `verify:format-coverage` (every tracked source file is format-gated), `verify:typecheck-coverage` (every one lands in a tsconfig project), `verify:dep-lockstep` (no dependency reaching one `tsc` program from two installs skews across them) — then `test:scripts` (the guards' own parser unit tests), then `validate:core` (the shared `core/` `format:check` + `lint` gate), then per client: `format:check` + `lint` + **`typecheck`** (cli/tui/launcher; web typechecks via `tsc -b` inside its `build`) + `build` + fast unit tests. The quick inner-loop check. |
|
||||
| `npm run validate` | Runs the four durable guards first — `verify:format-coverage` (every tracked source file is format-gated), `verify:skills` (every `.claude/skills` manifest parses, declares its invocation mode, and carries eval cases), `verify:typecheck-coverage` (every tracked TS file lands in a tsconfig project), `verify:dep-lockstep` (no dependency reaching one `tsc` program from two installs skews across them) — then `test:scripts` (the guards' own parser unit tests), then `validate:core` (the shared `core/` `format:check` + `lint` gate), then per client: `format:check` + `lint` + **`typecheck`** (cli/tui/launcher; web typechecks via `tsc -b` inside its `build`) + `build` + fast unit tests. The quick inner-loop check. |
|
||||
| `npm run coverage` | The **per-file ≥90% gate** (lines/statements/functions/branches) under v8 instrumentation, per client. CI-enforced. For web this also runs the integration project and covers the shared `core/` runtime (including `core/json` and `core/client`). |
|
||||
| `npm run smoke` | End-to-end smokes through the built launcher (`--help` dispatch + prod cli/tui/web — the TUI one runs under a **pseudoterminal** and asserts the Ink app is still running two seconds after its first frame, not merely that it painted one, [#2147](https://github.com/modelcontextprotocol/inspector/issues/2147)), plus three headless-browser smokes: a boot smoke that runs the prod web bundle and asserts a clean first render (no uncaught error — sync exception or unhandled rejection, how a Node built-in reaching the browser bundle manifests), and an **MCP Apps** smoke (`smoke:web:app`) that drives connect → open app → `data-app-status="ready"` against a composable App server, covering the sandbox proxy and UI-protocol bridge, and an **app-rendered elicitation** smoke (`smoke:web:elicit`) that drives one end to end — call the tool, answer inside the sandboxed app, see the app's `ElicitResult` reach the server — and then the same tool against a server that never advertised the capability, which must fall back to the native elicitation form. Those three take their engine from `SMOKE_BROWSER` — see [Supported browsers](#supported-browsers). A fourth, **`smoke:web:tabs`**, drives the core tabs against one server on one browser launch — Tools runs a tool and asserts its `structuredContent` section, Resources reads a resource with templates listed, Prompts fetches one ([#2148](https://github.com/modelcontextprotocol/inspector/issues/2148)) — asserting the `data-*` contract in [the web README](../clients/web/README.md#core-tab-automation-contract) rather than visible copy. It is **Chromium-only**: those tabs are ordinary React and Mantine, so unlike the sandbox they are not engine-sensitive. |
|
||||
| `npm run verify:build-gate` | Runs a real `vite build` with a Node built-in forced into the browser graph and asserts the build **fails** via the #1769 gate (which turns Vite's browser-externalization warning into a hard error). Guards against the warning phrasing drifting in a Vite bump and silently disabling the gate. Part of `npm run local:gate` and of GitHub CI. |
|
||||
| `npm run verify:bundle-externals` | Guards the must-not-bundle invariant (#2067): for each tsup-bundled client it reads the **built** `build/` output and fails if any package that must stay external was inlined anyway. Candidates are the union of the client's own `external` array **and the root manifest's `dependencies`** — the latter because #2067 was a *missing* `external` entry, which a self-referential check would have passed. Detection is via esbuild's `// <path>` module banners, so it covers both shapes — a separate `<pkg>-HASH.js` chunk (what a dynamically `import()`ed CommonJS package produces) and a statically-imported package folded straight into `index.js`, which emits no chunk at all. A build with no banners fails as such rather than passing clean, so enabling `minify` cannot silently retire the check. `undici` was declared only in the root and `clients/cli` manifests, and tsup auto-externalizes only what the _nearest_ manifest declares, so the web and TUI bundles inlined 1.05MB of it — and CommonJS inlined into an ESM bundle throws `Dynamic require of "assert" is not supported` on first use, from a specifier no user-side install can satisfy. Reads the output rather than the config because those two disagreed for four releases. Part of `npm run local:gate` and of GitHub CI. |
|
||||
| `npm run verify:format-coverage` | Parses the `format:check` globs out of every `package.json` (only those reachable from `validate`), enumerates all tracked source files, and **fails** listing any not covered by a glob — the durable guard for the "every first-party source file is format-gated" invariant (#1792). Runs first in `validate`. |
|
||||
| `npm run test:scripts` | Table-driven unit tests (`node --test`) for the guard's own pure parsers (`scripts/lib/npm-scripts.mjs`, `scripts/lib/tsc-program.mjs` + the exported helpers of `verify-typecheck-coverage.mjs` and `verify-dep-lockstep.mjs`), one case per rule they encode, plus two suites over shared `scripts/lib` helpers that no smoke can check itself: `resolve-node-bin.test.mjs` — the cross-platform bin resolver (#1939), pinned against the real `bin`/`exports` shapes of the packages the scripts actually spawn — and `announced-child.test.mjs` — the spawn/readiness ownership helper (#2000), which drives real `node -e` children to prove a child that never announces is still published to the caller before the timeout throws, and so is reachable by teardown rather than orphaned. Four more do the same: `mcp-app-flow.test.mjs` covers the shared MCP Apps flow (#2003) — the deep link's two CSRF gates and `appArgs` encoding, plus `driveAppFlow`'s failure branches against a stand-in page, all of which are dead code from the happy-path smokes' point of view and would otherwise surface only as opaque timeouts; `render-smoke.test.mjs` and `pty.test.mjs` cover the TUI boot harness ([#2147](https://github.com/modelcontextprotocol/inspector/issues/2147)) — the former driving real `node -e` stubs to prove that a child which paints the marker and *then* dies is a **failure**, which the old harness reported as OK and which no fixed TUI can reproduce; the latter pinning the three mutually-incompatible `script(1)` invocations, where a wrong guess stops the pseudoterminal from being allocated at all; and `ensure-test-servers.test.mjs` pins the [#2111](https://github.com/modelcontextprotocol/inspector/issues/2111) invariant — that `test-servers/build` is rebuilt **even when it already exists** — which no smoke can assert about itself, since one driving a stale fixture reports a product failure rather than a staleness one. `workflow-gate.test.mjs` is a different shape again ([#2146](https://github.com/modelcontextprotocol/inspector/issues/2146)): besides the table of parser cases it runs the parser over the repo's real `.github/workflows/**`, so a workflow that invokes the local-only gate or a non-Chromium engine pass fails here rather than in a CI run nobody expected to be red. Runs in `validate` — and `verify:typecheck-coverage` guards *this* gate in turn (reachable from `validate`, non-empty test set, every test file matched by the `test:scripts` glob), since `node --test` silently skips a file its glob misses and still exits 0. |
|
||||
| `npm run verify:skills` | Guards `.claude/skills` ([#2163](https://github.com/modelcontextprotocol/inspector/issues/2163)). A skill's body loads only when it is invoked, so one that stops being reachable loses behavior **silently** — and the way that happens is frontmatter Claude Code cannot parse, which loads the body with an *empty* description: `/skill-name` still works, so a manual spot check passes, while nothing is left for the model to match against. This parses each `SKILL.md` the same way, requires an explicit invocation mode on every skill and committed eval cases (positives **and** negatives) on every model-invoked one, and fails when the skill listing exceeds its recorded character budget. Also runs `claude plugin validate` when that CLI is on PATH — but does not depend on it, since it is absent in CI. Runs in `validate`. |
|
||||
| `npm run test:scripts` | Table-driven unit tests (`node --test`) for the guard's own pure parsers (`scripts/lib/npm-scripts.mjs`, `scripts/lib/tsc-program.mjs` + the exported helpers of `verify-typecheck-coverage.mjs` and `verify-dep-lockstep.mjs`), one case per rule they encode, plus two suites over shared `scripts/lib` helpers that no smoke can check itself: `resolve-node-bin.test.mjs` — the cross-platform bin resolver (#1939), pinned against the real `bin`/`exports` shapes of the packages the scripts actually spawn — and `announced-child.test.mjs` — the spawn/readiness ownership helper (#2000), which drives real `node -e` children to prove a child that never announces is still published to the caller before the timeout throws, and so is reachable by teardown rather than orphaned. Four more do the same: `mcp-app-flow.test.mjs` covers the shared MCP Apps flow (#2003) — the deep link's two CSRF gates and `appArgs` encoding, plus `driveAppFlow`'s failure branches against a stand-in page, all of which are dead code from the happy-path smokes' point of view and would otherwise surface only as opaque timeouts; `render-smoke.test.mjs` and `pty.test.mjs` cover the TUI boot harness ([#2147](https://github.com/modelcontextprotocol/inspector/issues/2147)) — the former driving real `node -e` stubs to prove that a child which paints the marker and *then* dies is a **failure**, which the old harness reported as OK and which no fixed TUI can reproduce; the latter pinning the three mutually-incompatible `script(1)` invocations, where a wrong guess stops the pseudoterminal from being allocated at all; and `ensure-test-servers.test.mjs` pins the [#2111](https://github.com/modelcontextprotocol/inspector/issues/2111) invariant — that `test-servers/build` is rebuilt **even when it already exists** — which no smoke can assert about itself, since one driving a stale fixture reports a product failure rather than a staleness one. `workflow-gate.test.mjs` is a different shape again ([#2146](https://github.com/modelcontextprotocol/inspector/issues/2146)): besides the table of parser cases it runs the parser over the repo's real `.github/workflows/**`, so a workflow that invokes the local-only gate or a non-Chromium engine pass fails here rather than in a CI run nobody expected to be red. Two more arrive with the skills (#2163): `skill-manifest.test.mjs` pins the frontmatter contract one case per rule — most importantly that a fence which is not a whole line is not a terminator, and that malformed YAML is an error rather than an empty description — and `skill-eval.test.mjs` covers the trigger eval's stream parsing and its process-outcome handling, which an eval run's happy path never reaches and which would otherwise report a plausible hit rate for a CLI that failed to run. Runs in `validate` — and `verify:typecheck-coverage` guards *this* gate in turn (reachable from `validate`, non-empty test set, every test file matched by the `test:scripts` glob), since `node --test` silently skips a file its glob misses and still exits 0. |
|
||||
| `npm run verify:typecheck-coverage` | The typecheck-coverage analog of the above (#1791): for each Node client (auto-discovered from disk — enrolled via its `typecheck` script's projects, or for a `tsc -b` client like `clients/web` via its `tsconfig.json` `references`) it runs those projects with `tsc --listFilesOnly`, unions them, and **fails** listing any tracked `.ts`/`.tsx`/`.mts`/`.cts` under the client that lands in no project (so a new top-level config/helper can't silently go untypechecked). It also requires, deny-by-default, the first-party TS no client owns (`test-servers/src`, the root `vitest.shared.mts`, all of `core/`, and any new top-level location) to land in some client project's tsc pass — so a `core` `*.tsx` web's projects don't reach is caught too. Also asserts the gate is wired (each client's typecheck pass — its `typecheck` script, or web's `tsc -b` — is reachable from its `validate`, and the root chain runs each client's `validate`). Runs in `validate`. |
|
||||
| `npm run verify:dep-lockstep` | Guards the "one version per install-crossing dependency" invariant (#1896). v2 is not a workspace, so a client's test project compiles the shared first-party TypeScript — `core/`, `test-servers/src`, and the root-owned `vitest.shared.mts`, all of which resolve their dependencies from the **root** install — alongside the client's own sources, putting the same package in one `tsc` program twice. At the same version that's harmless; skewed, TypeScript must relate two structurally-distinct copies of every type, which for a recursive-generic surface is exponential (zod `4.3.6` vs `4.4.3` exhausted the 4GB tsc heap in `clients/web`). Derives its candidate set from **what actually enters each program** (#1965) — every client tsconfig project listed with `tsc --listFilesOnly` via the shared `scripts/lib/tsc-program.mjs`, each resolved `node_modules` file mapped to its owning install, keeping the packages that reach one program from two installs (a package whose declarations arrive only through another package's `.d.ts`, as `@modelcontextprotocol/sdk`'s do, is invisible to a scan of first-party imports). Prices each copy from the lockfile entry for the exact install path the program resolved, compares only the installs that met in one program, and **fails deny-by-default** on any disagreement not in the annotated `TOLERATED_SKEW` allowlist — empty today — with an allowlisted package tolerated only *within a major version*. Runs in `validate`.
|
||||
| `npm run local:gate` | **Mandatory pre-push command.** `validate` → `coverage` → `verify:build-gate` → `verify:bundle-externals` → `smoke` → `smoke:web:firefox` → `local:storybook`. A strict superset of GitHub CI — see [Two tiers](#two-tiers-github-ci-and-the-local-gate). Named `local:` rather than `ci` on purpose (#2146); there is no `npm run ci` alias. |
|
||||
| `npm run pack:verify` | Publish smoke — see [Publishing](#publishing). |
|
||||
| `npm run pack:verify` | Publish smoke — see [Publishing](./publishing.md). |
|
||||
|
||||
Per-client scripts exist too (`validate:web`, `coverage:cli`, `smoke:tui`, …), plus root `validate:core` / `format:core` for the shared `core/` package, `format:scripts` for the root `scripts/` tooling, and `format:shared` / `lint:shared` for the root "shared" surface (`test-servers/src/**`, `vitest.shared.mts`, the root `eslint.config.js`). Run `npm run format` before committing — the root `format` fixes `core/`, the root `scripts/`, the shared surface, and every client; `validate` runs the non-fixing `format:check` and fails CI on any unformatted file.
|
||||
|
||||
|
|
|
|||
|
|
@ -26,33 +26,36 @@ This is what lets an Inspector connection negotiating `protocolEra: "auto" | "mo
|
|||
|
||||
## Showcase configs
|
||||
|
||||
Each config below is a ready-made server for exercising one feature by hand. Load one with `--config`, and unless noted, connect with **Protocol Era = Modern**.
|
||||
Each config below is a ready-made server for exercising one feature by hand.
|
||||
Load one with `--config` and connect with the era its row names — every row says
|
||||
which, because the two are not interchangeable and the wrong one usually presents
|
||||
as a missing capability rather than an error.
|
||||
|
||||
| Config | Demonstrates | Issue |
|
||||
| ----------------------------------------- | --------------------------------------------------- | ---------------------------------------------------------------------- |
|
||||
| `mcp-app-http.json` **(legacy era)** | An MCP App (UI resource + app tool) in the Apps tab | [#1859](https://github.com/modelcontextprotocol/inspector/issues/1859) |
|
||||
| `app-elicitation-http.json` **(legacy era)** | An MCP App rendering a form elicitation | [#1854](https://github.com/modelcontextprotocol/inspector/issues/1854) |
|
||||
| `mcp-app-domain-http.json` **(legacy era)** | An MCP App asking for a dedicated origin (`_meta.ui.domain`) | [#2056](https://github.com/modelcontextprotocol/inspector/issues/2056) |
|
||||
| `modern-mrtr-http.json` | A single MRTR round-trip | — |
|
||||
| `mrtr-showcase-http.json` | Every MRTR preset in one server | [#1860](https://github.com/modelcontextprotocol/inspector/issues/1860) |
|
||||
| `modern-network-http.json` | Network tab: `Mcp-*` headers + error taxonomy | [#1628](https://github.com/modelcontextprotocol/inspector/issues/1628) |
|
||||
| `xmcpheader-modern-http.json` | Tools tab: `x-mcp-header` mirroring and exclusions | [#1632](https://github.com/modelcontextprotocol/inspector/issues/1632) |
|
||||
| `pagination-http.json` | Page-by-page list fetching | [#1721](https://github.com/modelcontextprotocol/inspector/issues/1721) |
|
||||
| `structured-output-http.json` | Tools tab: a result's `structuredContent` section | [#1908](https://github.com/modelcontextprotocol/inspector/issues/1908) |
|
||||
| `duplicate-tool-names-http.json` | A `tools/list` that repeats a tool name | [#1957](https://github.com/modelcontextprotocol/inspector/issues/1957) |
|
||||
| `nullable-fields-http.json` | Tools tab: nullable (`anyOf` + `null`) arguments | [#1928](https://github.com/modelcontextprotocol/inspector/issues/1928) |
|
||||
| `modern-mrtr-http.json` **(modern era)** | A single MRTR round-trip | — |
|
||||
| `mrtr-showcase-http.json` **(modern era)** | Every MRTR preset in one server | [#1860](https://github.com/modelcontextprotocol/inspector/issues/1860) |
|
||||
| `modern-network-http.json` **(modern era)** | Network tab: `Mcp-*` headers + error taxonomy | [#1628](https://github.com/modelcontextprotocol/inspector/issues/1628) |
|
||||
| `xmcpheader-modern-http.json` **(modern era)** | Tools tab: `x-mcp-header` mirroring and exclusions | [#1632](https://github.com/modelcontextprotocol/inspector/issues/1632) |
|
||||
| `pagination-http.json` **(legacy era)** | Page-by-page list fetching | [#1721](https://github.com/modelcontextprotocol/inspector/issues/1721) |
|
||||
| `structured-output-http.json` **(legacy era)** | Tools tab: a result's `structuredContent` section | [#1908](https://github.com/modelcontextprotocol/inspector/issues/1908) |
|
||||
| `duplicate-tool-names-http.json` **(legacy era)** | A `tools/list` that repeats a tool name | [#1957](https://github.com/modelcontextprotocol/inspector/issues/1957) |
|
||||
| `nullable-fields-http.json` **(legacy era)** | Tools tab: nullable (`anyOf` + `null`) arguments | [#1928](https://github.com/modelcontextprotocol/inspector/issues/1928) |
|
||||
| `root-union-schemas-http.json` **(legacy era)** | Tool schemas whose arguments are a root `anyOf` / `oneOf` | [#2123](https://github.com/modelcontextprotocol/inspector/issues/2123) |
|
||||
| `unportable-schemas-http.json` **(legacy era)** | Tool schemas a real client rejects, flagged in all three clients | [#1005](https://github.com/modelcontextprotocol/inspector/issues/1005) |
|
||||
| `rfc6570-templates-http.json` | Resources tab: RFC 6570 resource-template expansion | [#1919](https://github.com/modelcontextprotocol/inspector/issues/1919) |
|
||||
| `advertised-extensions-http.json` | Tool registration gated on advertised extensions | [#1739](https://github.com/modelcontextprotocol/inspector/issues/1739) |
|
||||
| `rfc6570-templates-http.json` **(legacy era)** | Resources tab: RFC 6570 resource-template expansion | [#1919](https://github.com/modelcontextprotocol/inspector/issues/1919) |
|
||||
| `advertised-extensions-http.json` **(legacy era)** | Tool registration gated on advertised extensions | [#1739](https://github.com/modelcontextprotocol/inspector/issues/1739) |
|
||||
| `oauth-custom-resource-metadata-http.json` **(legacy era)** | OAuth discovery driven by the challenge's `resource_metadata` | [#2071](https://github.com/modelcontextprotocol/inspector/issues/2071) |
|
||||
| `oauth-revocation-http.json` / `oauth-no-revocation-http.json` **(legacy era)** | RFC 7009 token revocation on clear, with and without a `revocation_endpoint` | [#2144](https://github.com/modelcontextprotocol/inspector/issues/2144) |
|
||||
| `oauth-rfc8414-at-oidc-path-http.json` **(legacy era)** | Plain OAuth 2.0 AS metadata served at the OIDC well-known path | [#2172](https://github.com/modelcontextprotocol/inspector/issues/2172) |
|
||||
| `logging-{legacy,modern}-http.json` | Logging, both eras | [#1629](https://github.com/modelcontextprotocol/inspector/issues/1629) |
|
||||
| `subscriptions-{legacy,modern}-http.json` | Resource subscriptions, both eras | [#1630](https://github.com/modelcontextprotocol/inspector/issues/1630) |
|
||||
| `subscriptions-never-acknowledged-http.json` | A `subscriptions/listen` answered with a bare result | [#2097](https://github.com/modelcontextprotocol/inspector/issues/2097) |
|
||||
| `tasks-{legacy,modern}-http.json` | Tasks, both eras | [#1631](https://github.com/modelcontextprotocol/inspector/issues/1631) |
|
||||
| `cancellation-modern-http.json` | Cancelling a call by closing its response stream | [#2140](https://github.com/modelcontextprotocol/inspector/issues/2140) |
|
||||
| `logging-{legacy,modern}-http.json` **(era per file)** | Logging, both eras | [#1629](https://github.com/modelcontextprotocol/inspector/issues/1629) |
|
||||
| `subscriptions-{legacy,modern}-http.json` **(era per file)** | Resource subscriptions, both eras | [#1630](https://github.com/modelcontextprotocol/inspector/issues/1630) |
|
||||
| `subscriptions-never-acknowledged-http.json` **(modern era)** | A `subscriptions/listen` answered with a bare result | [#2097](https://github.com/modelcontextprotocol/inspector/issues/2097) |
|
||||
| `tasks-{legacy,modern}-http.json` **(era per file)** | Tasks, both eras | [#1631](https://github.com/modelcontextprotocol/inspector/issues/1631) |
|
||||
| `cancellation-modern-http.json` **(modern era)** | Cancelling a call by closing its response stream | [#2140](https://github.com/modelcontextprotocol/inspector/issues/2140) |
|
||||
|
||||
## Cancelling a call
|
||||
|
||||
|
|
|
|||
|
|
@ -49,12 +49,15 @@ export function splitFrontmatter(text) {
|
|||
};
|
||||
}
|
||||
const rest = text.slice(FRONTMATTER_FENCE.length + 1);
|
||||
const end = rest.indexOf("\n" + FRONTMATTER_FENCE);
|
||||
if (end === -1)
|
||||
// The closing fence must occupy the WHOLE line. Matching a mere prefix would
|
||||
// accept `---oops` as a terminator, silently truncating the frontmatter to
|
||||
// whatever preceded it — the very failure this function exists to catch.
|
||||
const close = /\n---[ \t]*(\r?\n|$)/.exec(rest);
|
||||
if (close === null)
|
||||
return { error: "frontmatter is never closed by a `---` line" };
|
||||
return {
|
||||
frontmatter: rest.slice(0, end),
|
||||
body: rest.slice(end + FRONTMATTER_FENCE.length + 2),
|
||||
frontmatter: rest.slice(0, close.index),
|
||||
body: rest.slice(close.index + close[0].length),
|
||||
};
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -28,6 +28,25 @@ test("splitFrontmatter requires the fence on the very first line", () => {
|
|||
assert.equal(splitFrontmatter(fm("name: x")).error, undefined);
|
||||
});
|
||||
|
||||
test("splitFrontmatter requires the closing fence to be a whole line", () => {
|
||||
// `---oops` is not a terminator. Accepting it would silently truncate the
|
||||
// frontmatter to whatever preceded it and report the file as valid.
|
||||
assert.match(
|
||||
splitFrontmatter("---\nname: x\n---oops\nbody\n").error ?? "",
|
||||
/never closed/,
|
||||
);
|
||||
// Trailing whitespace on the fence is still a fence, and so is EOF.
|
||||
assert.equal(
|
||||
splitFrontmatter("---\nname: x\n--- \nbody\n").frontmatter,
|
||||
"name: x",
|
||||
);
|
||||
assert.equal(splitFrontmatter("---\nname: x\n---").frontmatter, "name: x");
|
||||
assert.equal(
|
||||
splitFrontmatter("---\nname: x\n---\r\nbody").frontmatter,
|
||||
"name: x",
|
||||
);
|
||||
});
|
||||
|
||||
test("splitFrontmatter reports an unterminated block", () => {
|
||||
assert.match(splitFrontmatter("---\nname: x\n").error ?? "", /never closed/);
|
||||
});
|
||||
|
|
|
|||
|
|
@ -54,6 +54,53 @@ function collectCases(only) {
|
|||
return cases;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract the payloads the `Skill` tool was invoked with from a chunk of
|
||||
* `--output-format stream-json` output.
|
||||
*
|
||||
* Pure and separately tested: everything below spawns a real CLI, so the
|
||||
* parsing and the process-outcome handling are unreachable from the happy path
|
||||
* of an eval run and would otherwise only ever be exercised by the thing they
|
||||
* are supposed to measure.
|
||||
*
|
||||
* @param {string} text One or more newline-delimited JSON events. A trailing
|
||||
* partial line is ignored, so this can be fed incrementally.
|
||||
* @returns {{ invoked: Set<string>, rest: string }}
|
||||
*/
|
||||
export function collectSkillInvocations(text) {
|
||||
const lines = text.split("\n");
|
||||
const rest = lines.pop() ?? "";
|
||||
const invoked = new Set();
|
||||
for (const line of lines) {
|
||||
if (!line.trim()) continue;
|
||||
let evt;
|
||||
try {
|
||||
evt = JSON.parse(line);
|
||||
} catch {
|
||||
// A malformed line is noise from the CLI, not an observation.
|
||||
continue;
|
||||
}
|
||||
if (evt?.type !== "assistant") continue;
|
||||
for (const block of evt.message?.content ?? []) {
|
||||
if (block?.type !== "tool_use" || block.name !== "Skill") continue;
|
||||
// Don't assume the input field's name — match on the whole payload.
|
||||
invoked.add(JSON.stringify(block.input ?? {}));
|
||||
}
|
||||
}
|
||||
return { invoked, rest };
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether one sample satisfies a case.
|
||||
*
|
||||
* @param {string | null} expect Skill name, or null for a negative case.
|
||||
* @param {Set<string>} invoked
|
||||
*/
|
||||
export function sampleHit(expect, invoked) {
|
||||
if (expect === null) return invoked.size === 0;
|
||||
return [...invoked].some((payload) => payload.includes(expect));
|
||||
}
|
||||
|
||||
/** Returns the payloads the Skill tool was called with in one fresh session. */
|
||||
function runOnce(prompt) {
|
||||
return new Promise((resolve, reject) => {
|
||||
|
|
@ -72,33 +119,28 @@ function runOnce(prompt) {
|
|||
"--disallowedTools",
|
||||
"Bash,Write,Edit,NotebookEdit",
|
||||
],
|
||||
{ cwd: ROOT, stdio: ["ignore", "pipe", "ignore"] },
|
||||
{ cwd: ROOT, stdio: ["ignore", "pipe", "inherit"] },
|
||||
);
|
||||
|
||||
let buf = "";
|
||||
const invoked = new Set();
|
||||
p.stdout.on("data", (chunk) => {
|
||||
buf += chunk.toString();
|
||||
const lines = buf.split("\n");
|
||||
buf = lines.pop() ?? "";
|
||||
for (const line of lines) {
|
||||
if (!line.trim()) continue;
|
||||
let evt;
|
||||
try {
|
||||
evt = JSON.parse(line);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (evt?.type !== "assistant") continue;
|
||||
for (const block of evt.message?.content ?? []) {
|
||||
if (block?.type !== "tool_use" || block.name !== "Skill") continue;
|
||||
// Don't assume the input field's name — match on the whole payload.
|
||||
invoked.add(JSON.stringify(block.input ?? {}));
|
||||
}
|
||||
}
|
||||
const parsed = collectSkillInvocations(buf + chunk.toString());
|
||||
buf = parsed.rest;
|
||||
for (const payload of parsed.invoked) invoked.add(payload);
|
||||
});
|
||||
p.on("error", reject);
|
||||
p.on("close", () => resolve(invoked));
|
||||
p.on("close", (code) => {
|
||||
// A CLI that failed to run observed NOTHING. Resolving it as an empty set
|
||||
// would silently pass every negative case and read as a trigger miss on
|
||||
// every positive one, so an auth error or a rate limit would come back as
|
||||
// a plausible-looking hit rate (Copilot).
|
||||
if (code !== 0) {
|
||||
reject(new Error(`\`claude -p\` exited ${code} for prompt: ${prompt}`));
|
||||
return;
|
||||
}
|
||||
resolve(invoked);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
|
|
@ -142,9 +184,7 @@ async function main() {
|
|||
let failed = 0;
|
||||
for (const c of cases) {
|
||||
const mine = results.filter((r) => r.c === c);
|
||||
const passes = mine.filter((r) =>
|
||||
c.expect === null ? r.invoked.size === 0 : hit(r.invoked, c.expect),
|
||||
).length;
|
||||
const passes = mine.filter((r) => sampleHit(c.expect, r.invoked)).length;
|
||||
const rate = passes / mine.length;
|
||||
const ok = rate >= THRESHOLD;
|
||||
if (!ok) failed++;
|
||||
|
|
@ -160,7 +200,12 @@ async function main() {
|
|||
process.exit(failed > 0 ? 1 : 0);
|
||||
}
|
||||
|
||||
main().catch((e) => {
|
||||
console.error(e);
|
||||
process.exit(1);
|
||||
});
|
||||
if (
|
||||
process.argv[1] &&
|
||||
path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)
|
||||
) {
|
||||
main().catch((e) => {
|
||||
console.error(e.message ?? e);
|
||||
process.exit(1);
|
||||
});
|
||||
}
|
||||
|
|
|
|||
69
scripts/skill-eval.test.mjs
Normal file
69
scripts/skill-eval.test.mjs
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
// Tests for the pure halves of the skill trigger eval (#2163).
|
||||
//
|
||||
// Everything else in `skill-eval.mjs` spawns a real `claude` CLI, so its stream
|
||||
// parsing and its process-outcome handling are unreachable from an eval run's
|
||||
// happy path — they would only ever be exercised by the very measurement they
|
||||
// are supposed to make trustworthy. A nonzero exit read as "no skill invoked"
|
||||
// (the bug these cases pin) passes every negative case and reads as a trigger
|
||||
// miss on every positive one, so a rate-limited run reports a plausible number
|
||||
// instead of failing.
|
||||
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { collectSkillInvocations, sampleHit } from "./skill-eval.mjs";
|
||||
|
||||
const assistant = (...blocks) =>
|
||||
JSON.stringify({ type: "assistant", message: { content: blocks } });
|
||||
const skillUse = (name) => ({
|
||||
type: "tool_use",
|
||||
name: "Skill",
|
||||
input: { skill: name },
|
||||
});
|
||||
|
||||
test("collectSkillInvocations finds Skill tool_use payloads", () => {
|
||||
const { invoked } = collectSkillInvocations(
|
||||
assistant(skillUse("testing")) + "\n",
|
||||
);
|
||||
assert.equal(invoked.size, 1);
|
||||
assert.ok([...invoked][0].includes("testing"));
|
||||
});
|
||||
|
||||
test("collectSkillInvocations ignores other tools and other event types", () => {
|
||||
const text =
|
||||
assistant({ type: "tool_use", name: "Read", input: { file_path: "a" } }) +
|
||||
"\n" +
|
||||
JSON.stringify({ type: "result", result: "Skill" }) +
|
||||
"\n";
|
||||
assert.equal(collectSkillInvocations(text).invoked.size, 0);
|
||||
});
|
||||
|
||||
test("collectSkillInvocations survives malformed and blank lines", () => {
|
||||
const text = "not json\n\n" + assistant(skillUse("local-dev")) + "\n";
|
||||
const { invoked } = collectSkillInvocations(text);
|
||||
assert.equal(invoked.size, 1);
|
||||
});
|
||||
|
||||
test("collectSkillInvocations holds back a trailing partial line", () => {
|
||||
const whole = assistant(skillUse("local-dev"));
|
||||
const first = collectSkillInvocations(whole.slice(0, 20));
|
||||
assert.equal(first.invoked.size, 0);
|
||||
assert.equal(first.rest, whole.slice(0, 20));
|
||||
// Feeding the remainder back with the held-over prefix completes the event.
|
||||
const second = collectSkillInvocations(first.rest + whole.slice(20) + "\n");
|
||||
assert.equal(second.invoked.size, 1);
|
||||
});
|
||||
|
||||
test("collectSkillInvocations tolerates a tool_use with no input", () => {
|
||||
const text = assistant({ type: "tool_use", name: "Skill" }) + "\n";
|
||||
assert.deepEqual([...collectSkillInvocations(text).invoked], ["{}"]);
|
||||
});
|
||||
|
||||
test("sampleHit scores positive and negative cases", () => {
|
||||
const fired = new Set(['{"skill":"testing"}']);
|
||||
const none = new Set();
|
||||
assert.equal(sampleHit("testing", fired), true);
|
||||
assert.equal(sampleHit("local-dev", fired), false);
|
||||
assert.equal(sampleHit(null, none), true);
|
||||
assert.equal(sampleHit(null, fired), false);
|
||||
assert.equal(sampleHit("testing", none), false);
|
||||
});
|
||||
Loading…
Reference in a new issue