Commit graph conformance/package-lock.json
Author SHA1 Message Date
claude[bot]
89c2c0452e
Validate all wire messages against the per-version spec JSON schema (#399)
* Vendor schema/{version}/schema.json alongside schema.ts

sync-schema now copies the spec repo's JSON Schema for each version into
src/spec-types/{version}.schema.json (same pinned SHA recorded in SOURCE:
modelcontextprotocol@f817239f4d6b1efff2c4dfc2f7af85c985d73076). These are
the inputs for runtime wire-message validation. Vendored verbatim, so they
join schema.ts in .prettierignore.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JHNrpU3Vr6Q1rPZiji7zLY

* Validate all wire messages against the per-version spec JSON schema

Every JSON-RPC message the harness sends or receives is now validated
(ajv) against the vendored schema.json for the run's spec version, at the
wire choke points: sendStatelessRequest (outbound request, JSON response,
SSE events), connectStateful (request/result/notifications), and both mock
servers (inbound client requests, outbound responses/handler results).

Beyond the JSONRPCMessage envelope, messages are validated against the
most specific definition the schema has: typed requests/notifications by
method const, typed error responses by error.code const (e.g. -32021 ->
MissingRequiredClientCapabilityError, the #376 hallucination), and typed
results via the XxxRequest -> XxxResult pair, with SEP-2322
InputRequiredResult discriminated on resultType.

Violations are recorded per scenario and surfaced by the runners as a
synthetic wire-schema-valid check (FAILURE when the implementation under
test sent an invalid message) plus a wire-schema-harness-error check when
the invalid message was harness-authored. A vitest-wide afterEach guard
fails any self-test that recorded violations, so hallucinated fixtures
can't ship (#376, #271). Opt-outs are explicit and greppable: per-call
skipValidation for deliberately malformed traffic, takeWireViolations()
drains in tests that exercise deliberately broken fixtures.

Offenders the new validation caught, now fixed:
- everything-server stateless path omitted resultType (and the caching
  hints on server/discover) required by the draft schema
- sep-2164-empty-contents fixture omitted resultType/ttlMs/cacheScope
  (its intended violation, empty contents, is schema-valid prose-only)
- connection/mock-server unit-test fixtures used schema-invalid stubs
  (tools/call without name, ListToolsResult without required members,
  Tool without inputSchema, ProgressNotification without progressToken,
  ElicitRequest without message/requestedSchema)
- http-standard-headers test mocks returned bare { tools: [] } results
- input-required-result-validate-input deliberately sends malformed
  inputResponses; now marked skipValidation

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JHNrpU3Vr6Q1rPZiji7zLY

* Validate the stateful wire at the SDK transport, close review-panel gaps

Address the review-panel findings on the wire-schema validation PR:

- Stateful wire: hook the SDK transport inside connectToServer, wrapping
  send/onmessage so every raw wire message in both directions is
  validated for the scenario's spec version -- including the initialize
  handshake, server->client requests (elicitation/sampling), and the
  harness's responses to them. Request ids are correlated per direction
  so responses validate against their typed result definition (e.g. an
  outbound ElicitResult). The reconstructed stand-ins in
  connectStateful (fake id: 0 envelopes, Zod-reparsed results) are
  removed; the hook sees the real bytes. The four scenarios that call
  connectToServer directly (elicitation-defaults, elicitation-enums,
  tools, lifecycle) now pass ctx.specVersion and get validation for
  free; the runner surfaces their violations through the existing
  recorder. Known remaining gap (documented in the module doc): the
  client-auth scenarios' bespoke express mock is not instrumented.

- skipValidation now skips only the request direction; the
  implementation's response to a deliberately malformed request is
  still validated, with a narrow carve-out tolerating the JSON-RPC 2.0
  id: null an error response must carry when the request could not be
  processed (the MCP schema's RequestId forbids null). Covered by new
  unit tests pinning both the still-validated response and the
  narrowness of the carve-out.

- negative-mrtr.test.ts: the file-wide drain now asserts every recorded
  violation is implementation-origin (the deliberately broken fixture)
  and throws on any harness-origin violation.

- everything-server: hoist the cacheable-methods set to module scope
  and replace the per-request res.json monkey-patch with an explicit
  sendStatelessJson(res, method, payload) helper applied at each
  dispatch site.

- Pin the extracted dispatch maps (methodDefs/errorDefs, e.g.
  -32021 -> MissingRequiredClientCapabilityError under the draft) per
  spec version in wire-schema.test.ts, so a schema sync that breaks the
  const extraction fails loudly instead of degrading to envelope-only
  validation. Also note the batch branch's requestMethod forwarding
  limitation.

Offender the transport hook flushed out: the everything-server's
test_elicitation_sep1034_defaults tool sent a number-typed elicitation
field with default: 95.5, which the 2025-11-25 schema forbids
(NumberSchema.default is typed integer there; widened to number only in
the draft). The fixture and the scenario's expected default are now the
integer-valued 95, with comments explaining why.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JHNrpU3Vr6Q1rPZiji7zLY

* Condense comments to three lines max

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JHNrpU3Vr6Q1rPZiji7zLY

* Re-vendor spec schemas at pinned SHA 71e30695

Regenerate src/spec-types/*.schema.json via npm run sync-schema at the
SHA already recorded in SOURCE. The previously vendored draft schema
predated spec PR #3002, so it still required DiscoverResult.serverInfo
at the top level; the mock server now correctly reports serverInfo via
result _meta, which the stale schema rejected.

* Scope the wire-violation recorder per scenario for parallel suite runs

Client --suite runs execute scenarios concurrently over what was a
module-global violation recorder, so scenarios could steal or wipe each
other's recorded violations. Give each suite scenario its own recorder
via AsyncLocalStorage; code outside a scope (single-scenario runs, the
sequential server suite, vitest hooks) falls back to the global
recorder. Also condense a four-line comment in connection/stateful.ts
to three lines.

* Patch the 2025-11-25 NumberSchema generator bug as a load-time erratum

The released 2025-11-25 schema.json types NumberSchema
minimum/maximum/default as integer, contradicting its own schema.ts
(default?: number) — a typescript-json-schema artifact, fixed for draft
in modelcontextprotocol#2710 and proposed for 2025-11-25 in
modelcontextprotocol#3139. Patch the schema at load time so SDK
fixtures with fractional number defaults (all tier-1 SDKs send 95.5)
are not flagged; restore the sep-1034 scenario and example-server
fixture to 95.5. Delete the erratum when #3139 lands and the schemas
are re-vendored.

Also scope the sequential server-suite loop with withWireRecorder so a
connection leaked by a failed scenario cannot attribute late traffic to
the next scenario.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
2026-07-27 15:58:43 +01:00
Paul Carleton
468e53547d
chore: bump version to 0.2.0-alpha.10 (#413) 2026-07-27 12:35:42 +02:00
Felix Weinberger
794dcab99e
chore: bump version to 0.2.0-alpha.9 (#377) 2026-07-01 19:09:47 +01:00
Paul Carleton
4944b26806
chore: bump version to 0.2.0-alpha.8 (#373) 2026-06-30 11:50:37 +01:00
Felix Weinberger
5d7abd5e70
chore: bump version to 0.2.0-alpha.7 (#367) 2026-06-25 11:26:47 +01:00
Felix Weinberger
1d197a9930
chore: bump version to 0.2.0-alpha.6 (#362) 2026-06-24 11:44:20 +01:00
Felix Weinberger
38c4852d8c
chore: bump version to 0.2.0-alpha.5 (#357)
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
2026-06-23 11:57:58 +00:00
Paul Carleton
7a620cb192
chore: bump version to 0.2.0-alpha.4 (#342) 2026-06-16 17:29:40 +01:00
Paul Carleton
70b17b3945
chore: bump version to 0.2.0-alpha.3 (#334) 2026-06-08 18:06:00 +01:00
Paul Carleton
25fd44323f
chore: bump version to 0.2.0-alpha.2 (#328) 2026-06-03 15:27:20 +01:00
Paul Carleton
7d3b897618
chore: bump version to 0.2.0-alpha.1 (#320) 2026-05-26 21:12:23 +01:00
Paul Carleton
bcfd400c9a
chore: bump version to 0.2.0-alpha.0 (#306) 2026-05-23 00:35:25 +01:00
dependabot[bot]
7f6eb14835
chore(deps): bump the all-dependencies group across 1 directory with 10 updates (#252)
* chore(deps): bump the all-dependencies group across 1 directory with 10 updates

Bumps the all-dependencies group with 10 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [@modelcontextprotocol/sdk](https://github.com/modelcontextprotocol/typescript-sdk) | `1.27.1` | `1.29.0` |
| [eventsource-parser](https://github.com/rexxars/eventsource-parser) | `3.0.6` | `3.0.8` |
| [undici](https://github.com/nodejs/undici) | `7.24.5` | `7.25.0` |
| [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `25.5.0` | `25.6.0` |
| [@typescript/native-preview](https://github.com/microsoft/typescript-go) | `7.0.0-dev.20260325.1` | `7.0.0-dev.20260424.1` |
| [lefthook](https://github.com/evilmartians/lefthook) | `2.1.4` | `2.1.6` |
| [prettier](https://github.com/prettier/prettier) | `3.8.1` | `3.8.3` |
| [tsdown](https://github.com/rolldown/tsdown) | `0.21.5` | `0.21.10` |
| [typescript-eslint](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/typescript-eslint) | `8.57.2` | `8.59.0` |
| [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) | `4.1.1` | `4.1.5` |



Updates `@modelcontextprotocol/sdk` from 1.27.1 to 1.29.0
- [Release notes](https://github.com/modelcontextprotocol/typescript-sdk/releases)
- [Commits](https://github.com/modelcontextprotocol/typescript-sdk/compare/v1.27.1...v1.29.0)

Updates `eventsource-parser` from 3.0.6 to 3.0.8
- [Release notes](https://github.com/rexxars/eventsource-parser/releases)
- [Changelog](https://github.com/rexxars/eventsource-parser/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rexxars/eventsource-parser/compare/v3.0.6...v3.0.8)

Updates `undici` from 7.24.5 to 7.25.0
- [Release notes](https://github.com/nodejs/undici/releases)
- [Commits](https://github.com/nodejs/undici/compare/v7.24.5...v7.25.0)

Updates `@types/node` from 25.5.0 to 25.6.0
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Updates `@typescript/native-preview` from 7.0.0-dev.20260325.1 to 7.0.0-dev.20260424.1
- [Changelog](https://github.com/microsoft/typescript-go/blob/main/CHANGES.md)
- [Commits](https://github.com/microsoft/typescript-go/commits)

Updates `lefthook` from 2.1.4 to 2.1.6
- [Release notes](https://github.com/evilmartians/lefthook/releases)
- [Changelog](https://github.com/evilmartians/lefthook/blob/master/CHANGELOG.md)
- [Commits](https://github.com/evilmartians/lefthook/compare/v2.1.4...v2.1.6)

Updates `prettier` from 3.8.1 to 3.8.3
- [Release notes](https://github.com/prettier/prettier/releases)
- [Changelog](https://github.com/prettier/prettier/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prettier/prettier/compare/3.8.1...3.8.3)

Updates `tsdown` from 0.21.5 to 0.21.10
- [Release notes](https://github.com/rolldown/tsdown/releases)
- [Commits](https://github.com/rolldown/tsdown/compare/v0.21.5...v0.21.10)

Updates `typescript-eslint` from 8.57.2 to 8.59.0
- [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases)
- [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/typescript-eslint/CHANGELOG.md)
- [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v8.59.0/packages/typescript-eslint)

Updates `vitest` from 4.1.1 to 4.1.5
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.5/packages/vitest)

---
updated-dependencies:
- dependency-name: "@modelcontextprotocol/sdk"
  dependency-version: 1.29.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: all-dependencies
- dependency-name: eventsource-parser
  dependency-version: 3.0.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: all-dependencies
- dependency-name: undici
  dependency-version: 7.25.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: all-dependencies
- dependency-name: "@types/node"
  dependency-version: 25.6.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: all-dependencies
- dependency-name: "@typescript/native-preview"
  dependency-version: 7.0.0-dev.20260424.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: all-dependencies
- dependency-name: lefthook
  dependency-version: 2.1.6
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: all-dependencies
- dependency-name: prettier
  dependency-version: 3.8.3
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: all-dependencies
- dependency-name: tsdown
  dependency-version: 0.21.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: all-dependencies
- dependency-name: typescript-eslint
  dependency-version: 8.59.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: all-dependencies
- dependency-name: vitest
  dependency-version: 4.1.5
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: all-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix: switch moduleResolution to bundler for newer tsgo

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
2026-04-24 12:51:39 +01:00
Paul Carleton
825f2031c8
chore(deps): security rollup — path-to-regexp, defu, vite, hono, @hono/node-server (#251)
Supersedes #204, #209, #210, #218, #229, #230, #231, #232, #233.
Excludes #205 (failing CI; triaged separately).
2026-04-24 12:22:07 +01:00
Paul Carleton
21a9a2febd
chore: bump version to 0.1.16 (#198) 2026-03-27 11:29:13 +00:00
dependabot[bot]
7b5865a874
chore(deps-dev): bump the all-dependencies group with 2 updates (#196)
Bumps the all-dependencies group with 2 updates: [@typescript/native-preview](https://github.com/microsoft/typescript-go) and [tsdown](https://github.com/rolldown/tsdown).


Updates `@typescript/native-preview` from 7.0.0-dev.20260323.1 to 7.0.0-dev.20260325.1
- [Changelog](https://github.com/microsoft/typescript-go/blob/main/CHANGES.md)
- [Commits](https://github.com/microsoft/typescript-go/commits)

Updates `tsdown` from 0.21.4 to 0.21.5
- [Release notes](https://github.com/rolldown/tsdown/releases)
- [Commits](https://github.com/rolldown/tsdown/compare/v0.21.4...v0.21.5)

---
updated-dependencies:
- dependency-name: "@typescript/native-preview"
  dependency-version: 7.0.0-dev.20260325.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: all-dependencies
- dependency-name: tsdown
  dependency-version: 0.21.5
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: all-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-25 11:49:15 +00:00
Paul Carleton
8f3994c75f
chore(deps): consolidate dependabot updates (#192)
* chore(deps): consolidate dependabot updates

Bumps all pending dependency updates into one commit:

Root package:
- zod 3.25.76 → 4.3.6 (major)
- @types/node 22 → 25 (major)
- @modelcontextprotocol/sdk 1.26.0 → 1.27.1
- tsdown 0.15.12 → 0.21.4
- prettier 3.6.2 → 3.8.1
- typescript-eslint 8.48.0 → 8.57.1
- lefthook 2.0.2 → 2.1.4
- eslint/@eslint/js → 9.39.4 (kept at v9, typescript-eslint lacks v10 support)
- Regenerated lockfile to pick up transitive security fixes
  (hono, @hono/node-server, express-rate-limit, flatted, rollup,
  minimatch, ajv, qs, undici)

examples/servers/typescript:
- @modelcontextprotocol/sdk → 1.27.1
- Removed zod-to-json-schema (broken with zod v4)
- Added zod ^4 as direct dep
- Regenerated lockfile

GitHub Actions:
- actions/setup-node v4 → v6 in action.yml

Code changes for zod v4:
- ZodError.errors → ZodError.issues (src/index.ts)
- .refine() callback signature updated (src/schemas.ts)
- everything-server: replaced zodToJsonSchema with SDK's
  toJsonSchemaCompat helper (handles zod v3/v4/v4-mini)

Build fix for tsdown 0.21:
- Added --no-fixed-extension to preserve dist/index.js output
  (tsdown 0.21 defaults to .mjs for type:module packages)

* fix: include all platform binaries in lockfile

The lockfile was generated on macOS which caused npm to only resolve
darwin-arm64 entries for platform-specific optional dependencies
(esbuild, rolldown bindings). This broke npm ci on Linux CI runners.

Regenerated from a clean state to include all 26 esbuild platforms
and all 15 rolldown platform bindings.
2026-03-24 16:18:54 +00:00
Paul Carleton
a2855b0358
0.1.15 (#156) 2026-02-20 14:41:15 +00:00
Paul Carleton
8ab0831130
0.1.14 (#149) 2026-02-13 19:13:12 +00:00
Felix Weinberger
3b4a92baad
feat: SDK tier assessment CLI and skill (#142)
* feat: add tier-check CLI for SDK tier assessment

Adds a 'tier-check' subcommand to the conformance tool that automates
SDK tier assessment against SEP-1730 criteria.

Checks performed:
- Conformance test pass rate (via everything-server)
- GitHub label taxonomy (priority/status/area labels)
- Issue triage SLA compliance
- P0 bug resolution tracking
- Stable release detection
- Required file existence (CHANGELOG, SECURITY, etc.)
- Spec tracking (SDK release within 30d of spec release)

Also includes a Claude Code skill (skills/mcp-sdk-tier-audit/) for
judgment-based checks that require codebase analysis (feature coverage,
docs quality, policy evaluation).

Usage:
  npx tsx src/index.ts tier-check --repo modelcontextprotocol/typescript-sdk
  npx tsx src/index.ts tier-check --repo ... --conformance-server-cmd '...' \
    --conformance-server-cwd ... --conformance-server-url ... --output json

* refactor: revise tier-check CLI and skill based on review feedback

- Move skill to .claude/skills/ so it's auto-available in Claude Code
- Remove feature-coverage subagent (redundant with conformance tests)
- Remove hardcoded ~/src/mcp paths from all skill files
- Trim conformance server table to TS + Python only
- Rename file_existence check to policy_signals (informational, not blocking)
- Add GitHub native issue types detection to labels check
- Add missing features to docs-coverage checklist (tasks, elicitation URL mode, JSON Schema 2020-12)
- Add README with CLI quick start and escape hatch for non-Claude-Code users
- Use --limit 500 instead of --limit 100 for gh issue list

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address review feedback on skill and README

- Use npx @modelcontextprotocol/conformance instead of node dist/index.js
- Add full GitHub auth instructions (gh auth login, GITHUB_TOKEN, --token)
- Point TS SDK conformance server to typescript-sdk/test/conformance/
- Fix Python SDK URL to localhost:3001/mcp (not TBD)
- Remove manual gh issue list / gh release list from SKILL.md (CLI handles it)
- Remove Claude Code-specific subagent_type references
- Assume user is already in conformance repo
- Clean up policy-evaluation-prompt.md: remove redundant grep commands,
  focus on content evaluation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: resolve lint errors in conformance.ts

Remove unused variable assignments flagged by eslint.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style: apply prettier formatting

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: add npm run tier-check script, update docs with examples

- Add "tier-check" npm script so users can run `npm run tier-check --`
  instead of `node dist/index.js tier-check`
- Update SKILL.md, README, and skill README to use npm run tier-check
- Add full conformance examples with --conformance-server-cmd/cwd/url
  flags and realistic paths (~/src/mcp/typescript-sdk)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style: fix prettier formatting in SKILL.md

* fix: add per-scenario timeout, support url-only conformance, fix stdout pollution

- Add 30s per-scenario timeout to prevent tier-check from hanging
- Allow --conformance-server-url without --conformance-server-cmd (server already running)
- Move runner status logs to stderr so --output json produces clean JSON
- Update SKILL.md and README with --silent flag and pre-start server workflow

* refactor: skill takes local path + server URL instead of repo name

The skill now requires two arguments:
1. Local path to the SDK checkout (for direct file inspection)
2. URL where the everything server is already running

The GitHub owner/repo is derived from git remote. This eliminates
cloning, server startup complexity, and branch confusion (v1.x vs main).

* refactor: write detailed reports to files, show concise summary

Reports go to results/tier-audits/<sdk>-<date>/ (already gitignored).
Claude's console output is now just the tier classification,
pass/fail summary line, top 3 actions, and file paths.

* docs: update READMEs for new skill interface and pre-start workflow

* simplify: flat file output instead of nested directory

* fix: remediation always shows path to Tier 2 and Tier 1

* feat: add client conformance testing to tier-check CLI and skill

- Add checkClientConformance() that runs core client scenarios
  (initialize, tools_call, elicitation-defaults, sse-retry, auth)
  by spawning the SDK's conformance client via --client-cmd
- Add client_conformance to TierScorecard type
- Wire --client-cmd option into the CLI
- Update tier logic: both server + client conformance feed into
  Tier 1 (100%) and Tier 2 (>=80%) requirements
- Update terminal and markdown output to show both conformance types
- Update skill to auto-detect conformance client or accept explicit
  client-cmd argument
- Update README with new option and examples

* improve: table summary output, write reports via subagents

- Change executive summary from pipe-delimited line to a readable
  table with T2/T1 columns
- Move assessment and remediation file writing into parallel
  subagents to keep the main conversation thread clean

* improve: list tier gaps as numbered items instead of one-line blob

* improve: finalize summary format with separator, high-priority fixes, numbered gaps

* improve: add pre-flight checks for gh auth and server reachability

Fail fast with clear error messages if GitHub CLI is not authenticated
or if the conformance server URL is not reachable, rather than failing
deep into the scorecard run.

* docs: improve README and fix skill auto-detection paths

- Claude Code section: explain client-cmd auto-detection for TS/Python,
  show explicit 3-arg form for other SDKs, add examples for all three
- Fix TypeScript build command (npm run build, not pnpm build:all)
- Fix Python server command (add --port, use uv sync --package)
- Fix Python client path (.github/actions/conformance/client.py)
- Expand 'Other SDKs' section with guidance on everything server
- Add gh auth login prerequisite to Claude Code steps

* simplify: remove client-cmd auto-detection, require explicit argument

Client command is now always passed as the third argument. If omitted,
client conformance is skipped and noted as a gap. No more magic path
detection — clearer and more predictable.

* fix: align docs table with canonical list (48 features), simplify policy eval

Docs coverage:
- Table now has numbered rows matching all 48 non-experimental features
  from the canonical list (was missing 7: tools text/image/audio/embedded/
  error/notifications, protocol version negotiation)
- Hardcode total as 48 in summary so agents don't miscount

Policy evaluation:
- Simplified from deep content analysis to file-existence checks
- Dependency policy: DEPENDENCY_POLICY.md, dependabot.yml, or CONTRIBUTING.md section
- Roadmap: ROADMAP.md must exist (GitHub milestones alone not sufficient)
- Versioning: VERSIONING.md or CONTRIBUTING.md section
- Removed GitHub API calls for milestones and releases from policy eval

* refactor: extract canonical feature list into single source of truth

Create references/feature-list.md with all 48 non-experimental + 5
experimental features. The docs-coverage prompt now references this
file instead of duplicating the list. One place to update when
features change.

* fix: separate deterministic file checks from AI content evaluation

CLI (files.ts): now checks all policy files deterministically —
DEPENDENCY_POLICY.md, docs/dependency-policy.md, dependabot.yml,
renovate.json, ROADMAP.md, docs/roadmap.md, VERSIONING.md,
docs/versioning.md, BREAKING_CHANGES.md (in addition to existing
CHANGELOG.md, SECURITY.md, CONTRIBUTING.md).

AI policy eval: receives CLI output showing which files exist,
then reads ONLY those files to judge content quality. No longer
searches the repo for files — clean separation of concerns.

* style: apply prettier formatting

* revert: undo unrelated console.log change in runner/server.ts

* refactor: shell out to conformance CLI instead of reimplementing runner

Address PR feedback: conformance.ts was duplicating the normal conformance
running code. Now shells out to 'node dist/index.js server/client' with
-o to save results to a temp dir, then parses the checks.json files.

Also removes --conformance-server-cmd and --conformance-server-cwd options
since the server must be pre-started.

* docs: add Go and C# SDK examples to README and SKILL.md

* fix: add --framework net9.0 to C# server command

* rename: conformance.ts -> test-conformance-results.ts

Avoids confusion with src/runner/ (the actual conformance runner).
This file just invokes the CLI and parses output.

* style: prettier formatting

* fix: reconcile conformance results against full scenario list

The tier-check CLI was only counting scenarios that produced a
checks.json file. Scenarios that crashed or failed to run (e.g., auth
scenarios when OAuth is not implemented) were invisible, making the
denominator artificially small (e.g., 4/4 instead of 4/23).

Now both checkConformance and checkClientConformance reconcile their
parsed results against the known scenario lists, adding failure entries
for any expected scenario that didn't produce results.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: tighten documentation evaluation criteria

Clarify what counts as documented vs just having code:
- Conformance test servers don't count as docs or examples
- Examples without prose = PARTIAL, not PASS
- Go Example* test functions explicitly allowed
- Clear PASS/PARTIAL/FAIL verdict definitions

* docs: add Labels and Spec Tracking rows to audit report templates

The executive summary and assessment report were missing two SEP-1730
requirements: label taxonomy compliance and spec tracking (new protocol
features timeline).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: reuse ConformanceCheck type from src/types.ts instead of redefining

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 17:59:06 +00:00
Paul Carleton
a1b8aaae5e
0.1.13 (#136) 2026-02-05 16:10:50 +00:00
Paul Carleton
22196ca809
fix: bump SDK to 1.26.0, add session management to tools_call scenario (#134)
* fix: bump SDK to 1.26.0, add session management to tools_call scenario

SDK 1.26.0 made Protocol.connect() throw if already connected to a
transport. The tools_call scenario was calling server.connect(transport)
on every request with the same Server instance, which now throws.

Fixed by adding proper session management: each initialize request
creates a new Server + Transport pair, and subsequent requests are
routed to the correct transport via the mcp-session-id header.

* fix: return 404 for invalid session IDs in tools_call POST handler

The else branch was returning 400 for both missing-session and
invalid/stale-session cases. The MCP spec requires 404 for invalid
session IDs and 400 only for non-initialization requests without
any session ID.

* refactor: keep tools_call stateless, create fresh server per request

Instead of adding session management, simply create a new Server
instance per request. This preserves the original stateless design
while fixing the SDK 1.26.0 Protocol.connect() restriction.
2026-02-05 15:43:21 +00:00
dependabot[bot]
189a31dc47
Bump hono from 4.11.4 to 4.11.7 (#124)
Bumps [hono](https://github.com/honojs/hono) from 4.11.4 to 4.11.7.
- [Release notes](https://github.com/honojs/hono/releases)
- [Commits](https://github.com/honojs/hono/compare/v4.11.4...v4.11.7)

---
updated-dependencies:
- dependency-name: hono
  dependency-version: 4.11.7
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-30 09:59:34 +00:00
Paul Carleton
4bfafd5e31
chore: 0.1.12 (#128)
* chore: bump version to 0.1.12-alpha.0 for OIDC trusted publishing test

* ci: rename prerelease checkbox to publish_alpha, clarify no-publish default

* 0.1.12
2026-01-29 18:19:35 +00:00
Paul Carleton
c2f3fdaf78
0.1.11 (#122) 2026-01-23 18:33:26 +00:00
Paul Carleton
d0b2b4914d
feat: add DNS rebinding protection conformance tests (#115)
* feat: add DNS rebinding protection conformance tests

Add a new server conformance scenario to verify that localhost MCP servers
properly validate Host headers to prevent DNS rebinding attacks.

Checks:
- localhost-host-rebinding-rejected: Verifies server returns 403 for
  non-localhost Host headers (e.g., evil.example.com)
- localhost-host-valid-accepted: Verifies server accepts requests with
  valid localhost Host headers

Also updates the everything-server example to use createMcpExpressApp()
which includes DNS rebinding protection by default.

Closes #103

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: add negative test case for DNS rebinding protection

Add a minimal MCP server that intentionally omits DNS rebinding protection
to serve as a negative test case. This server is expected to FAIL the
dns-rebinding-protection scenario.

Also update README to document the negative test case.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: accept any 4xx status code for DNS rebinding rejection

Update the DNS rebinding protection test to accept any 4xx status code
as a valid rejection response, not just 403. This accommodates different
HTTP semantics:
- 403 Forbidden: Per MCP spec
- 421 Misdirected Request: RFC 7540 (semantically correct for DNS rebinding)
- Other 4xx: Implementation-specific error codes

* refactor: address PR review comments

- Simplify negative test server (remove event store, sessions)
- Update spec reference URL to security_best_practices
- Clarify scope: localhost servers without HTTPS/auth
- DRY up checks.push using spread operator
- Fix valid host check to require 2xx response

* fix: include error details in catch block

* refactor: simplify negative test server to stateless

* refactor: check both Host and Origin headers for DNS rebinding protection

- Send both Host and Origin headers in test requests so servers checking
  either header will pass the conformance test
- Add second spec reference for Origin check (transports#security-warning)
- Update descriptions to mention both Host and Origin headers
- Fix protocol version to 2025-11-25
- Remove unnecessary 'as const' type assertions

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-22 18:37:48 +00:00
Max Isbey
83ff334e0e
feat: add --expected-failures baseline and composite GitHub Action (#113)
Add support for SDKs that don't yet pass all conformance tests to still
run conformance in CI without failing, by specifying a YAML baseline file
of known/expected failures.

CLI changes:
- Add --expected-failures <path> flag to both server and client commands
- Exit 0 if only expected failures occur
- Exit 1 if unexpected failures occur (regression)
- Exit 1 if an expected failure now passes (stale baseline, must update)

GitHub Action (action.yml):
- Composite action that builds from source and runs conformance tests
- SDKs reference it as: uses: modelcontextprotocol/conformance@<ref>
- Inputs: mode, url, command, expected-failures, suite, scenario, etc.

New files:
- src/expected-failures.ts: YAML loading, baseline evaluation, reporting
- src/expected-failures.test.ts: 16 unit tests
- action.yml: composite GitHub Action

Closes #99
2026-01-22 10:41:04 +00:00
Paul Carleton
7a3b13a404
0.1.10 2026-01-20 15:15:11 +00:00
Konstantin Konstantinov
0675e9b382
update npm audit (#108) 2026-01-19 16:46:09 +00:00
Paul Carleton
a90896b7d3
Add token endpoint auth method conformance tests (#48)
* [auth] Add metadata suite for running metadata discovery tests

Adds a new 'metadata' suite that runs just the auth/metadata-* scenarios
for faster iteration when testing metadata discovery specifically.

Usage: node dist/index.mjs client --suite metadata --command "..."

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Don't require lefthook to be installed

Set assert_lefthook_installed to false so the hooks gracefully skip
if lefthook is not installed on the system.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Add token endpoint auth method conformance tests

Add three new scenarios to test that clients correctly use the
appropriate authentication method based on server metadata:
- auth/token-endpoint-auth-basic: Tests client_secret_basic (HTTP Basic)
- auth/token-endpoint-auth-post: Tests client_secret_post
- auth/token-endpoint-auth-none: Tests public client (no auth)

Each scenario configures the server to only support one auth method
and verifies the client uses the correct method in token requests.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Refactor token endpoint auth tests to use shared createAuthServer

Extend createAuthServer helper with:
- tokenEndpointAuthMethodsSupported option for metadata
- onTokenRequest callback now receives full Request object
- onRegistrationRequest callback for custom client credentials

This eliminates the duplicate auth server implementation in
token-endpoint-auth.ts and reduces code by ~140 lines.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Fix onTokenRequest callback to use new interface

The createAuthServer onTokenRequest callback now passes an object with
authorizationHeader, body, timestamp etc. instead of (req, timestamp).
Also added missing return value for TokenRequestResult.

* Update @modelcontextprotocol/sdk to 1.25.1

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Olivier Chafik <ochafik@anthropic.com>
2026-01-05 14:32:35 +00:00
dependabot[bot]
0d1e9c7104
Bump vitest from 4.0.5 to 4.0.13 (#69)
Bumps [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) from 4.0.5 to 4.0.13.
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Commits](https://github.com/vitest-dev/vitest/commits/v4.0.13/packages/vitest)

---
updated-dependencies:
- dependency-name: vitest
  dependency-version: 4.0.13
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
2026-01-05 14:17:28 +00:00
dependabot[bot]
619fa77b92
Bump typescript-eslint from 8.46.2 to 8.48.0 (#65)
Bumps [typescript-eslint](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/typescript-eslint) from 8.46.2 to 8.48.0.
- [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases)
- [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/typescript-eslint/CHANGELOG.md)
- [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v8.48.0/packages/typescript-eslint)

---
updated-dependencies:
- dependency-name: typescript-eslint
  dependency-version: 8.48.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-05 14:02:13 +00:00
dependabot[bot]
31e26d0dd4
Bump @eslint/js from 9.38.0 to 9.39.1 (#67)
Bumps [@eslint/js](https://github.com/eslint/eslint/tree/HEAD/packages/js) from 9.38.0 to 9.39.1.
- [Release notes](https://github.com/eslint/eslint/releases)
- [Commits](https://github.com/eslint/eslint/commits/v9.39.1/packages/js)

---
updated-dependencies:
- dependency-name: "@eslint/js"
  dependency-version: 9.39.1
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-05 14:02:10 +00:00
dependabot[bot]
ef8c0844ad
Bump @typescript/native-preview (#70)
Bumps [@typescript/native-preview](https://github.com/microsoft/typescript-go) from 7.0.0-dev.20251030.1 to 7.0.0-dev.20251124.1.
- [Changelog](https://github.com/microsoft/typescript-go/blob/main/CHANGES.md)
- [Commits](https://github.com/microsoft/typescript-go/commits)

---
updated-dependencies:
- dependency-name: "@typescript/native-preview"
  dependency-version: 7.0.0-dev.20251124.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-05 14:01:40 +00:00
dependabot[bot]
29c01d5158
Bump body-parser in the npm_and_yarn group across 1 directory (#71)
Bumps the npm_and_yarn group with 1 update in the / directory: [body-parser](https://github.com/expressjs/body-parser).


Updates `body-parser` from 2.2.0 to 2.2.1
- [Release notes](https://github.com/expressjs/body-parser/releases)
- [Changelog](https://github.com/expressjs/body-parser/blob/master/HISTORY.md)
- [Commits](https://github.com/expressjs/body-parser/compare/v2.2.0...v2.2.1)

---
updated-dependencies:
- dependency-name: body-parser
  dependency-version: 2.2.1
  dependency-type: indirect
  dependency-group: npm_and_yarn
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-05 14:01:26 +00:00
Paul Carleton
b96a29a962
Add client credentials conformance tests (JWT and Basic auth) (#55)
* Add client credentials conformance scenarios with dynamic keypair generation

Implements SEP-1046 client credentials conformance tests:
- auth/client-credentials-jwt: Tests private_key_jwt authentication
- auth/client-credentials-basic: Tests client_secret_basic authentication

Key changes:
- Generate EC P-256 keypair dynamically at test start (no hardcoded keys)
- Pass credentials to client via MCP_CONFORMANCE_CONTEXT environment variable
- Add context field to ScenarioUrls interface for scenario-specific data
- Update client runner to pass context as env var to spawned client process

The MCP_CONFORMANCE_CONTEXT env var contains a JSON object with:
- client_id: The expected client identifier
- private_key_pem: PEM-encoded private key (for JWT scenarios)
- client_secret: Client secret (for basic auth scenarios)
- signing_algorithm: JWT signing algorithm (defaults to ES256)

* Fix client-credentials scenarios for conformance testing

- Generate EC keypair with extractable: true so private key can be
  exported and passed to clients via MCP_CONFORMANCE_CONTEXT
- Fix client_secret_basic scenario to use authorizationHeader param
  instead of non-existent headers object

* Fix jose type import - use CryptoKey instead of KeyLike

* Add OAuth 2.1 section 4.2 spec reference to client credentials checks

Add OAUTH_2_1_CLIENT_CREDENTIALS spec reference pointing to OAuth 2.1
draft section 4.2 (Client Credentials Grant) and include it in all
client_credentials conformance checks for both JWT and basic auth flows.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Add JWT iss claim verification for client credentials

Per RFC 7523, verify that the JWT issuer (iss) claim matches the
expected client_id, in addition to the existing sub claim check.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-25 20:07:05 +00:00
Felix Weinberger
f0e948187d
feat: add conformance tests for SEP-1699 SSE polling (#47) 2025-11-24 18:34:28 +00:00
Paul Carleton
14905af980
0.1.8 2025-11-21 19:05:58 +00:00
Paul Carleton
b64e054918
Add sep-835 limit retry scenario, add suite options to server command and update pending scenarios (#49)
* lefthook: move deps installs to optional local change

* add test scenario for infinite retry
and negative test

* Add broken client for retry limit negative test and fix lefthook formatting

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix www-authenticate in scope scenario

* add suites and shift pending to reflect current pending

* lint

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-21 16:49:38 +00:00
Paul Carleton
f2aaedaaa4
0.1.7 2025-11-20 21:02:52 +00:00
Felix Weinberger
8ef618bfe5
fix: sync package-lock.json after lefthook moved to devDeps (#46) 2025-11-20 17:18:35 +00:00
Paul Carleton
4a4c188cee
0.1.6 2025-11-19 15:02:42 +00:00
Paul Carleton
5c151133bb
[auth] Scenarios for scope selection (#36)
* wip scope change

* rm useless bool

* fix-me: package changes for linking

* negative test

* negative test for multiple scopes

* fix displays of warnings

* wip step-up

* negative test for step up auth

* move warning into main request flow

* clean up more checks

* cleanup step-up scenario

* cleanup and simplify

* cleanup middleware

* inline negative tests

* inline the other test

* refactor logging and inlining for standalone execution too

* skip pending tests
2025-11-18 17:16:23 +00:00
dependabot[bot]
253ede03ad
Bump js-yaml in the npm_and_yarn group across 1 directory (#31)
Bumps the npm_and_yarn group with 1 update in the / directory: [js-yaml](https://github.com/nodeca/js-yaml).


Updates `js-yaml` from 4.1.0 to 4.1.1
- [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nodeca/js-yaml/compare/4.1.0...4.1.1)

---
updated-dependencies:
- dependency-name: js-yaml
  dependency-version: 4.1.1
  dependency-type: indirect
  dependency-group: npm_and_yarn
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-17 11:19:16 +00:00
Paul Carleton
f7c8c4460f
[scenario] authz metadata discovery tests: compatibility with 2025-03-26 spec, prm root, and www-authenticate (#28)
* clean up constructor calls for server lifecycle

* add back compat test

* fix dup logging, rm prm

* avoid fallbacks

* better error from example client

* add final endpoint method

* add test for OASM priority order

* s/prepack/prepare/ for npx

* try switching to .js

* make tsdown version happy default

* fix lock

* fix prmPath issue

* back to main package-lock

* fix bin path

* Add comprehensive auth scenario test suite

- Add index.test.ts to test all auth scenarios with auth-test.ts client
- Upgrade @modelcontextprotocol/sdk from 1.20.1 to 1.22.0
- All 5 auth scenarios now pass including auth/basic-metadata-var2

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* consolidate tests into index.test.ts

* rename metadata

* fix comments

* rework auth scenario listing

* add 3rd variation of metadata

* fix some type issues introduced by version bump

* add reusable spec references

* more refs

* fix up spec references

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-14 18:36:07 +00:00
Inna Harper
38443d1a43
remove server requirements, move requirements to schenario description (#30) 2025-11-14 16:50:07 +00:00
Max Isbey
ee60717a06
Update dependencies in package.json and package-lock.json (#27)
Downgrade @typescript/native-preview and tsdown to previous versions.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-13 15:49:58 +00:00
Max Isbey
f2339987df
Upgrade vitest to v4.0.5 (#26)
Updates vitest from v2.1.8 to v4.0.5. All tests pass successfully with no code changes required.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-13 14:28:37 +00:00
Matt
aeceefa9cf
move to vitest (#18)
* feat: move to vitest

* fix: lock file

* fix: tests running sequentially on same port

* fix lock file

---------

Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
2025-11-13 13:32:07 +00:00
Paul Carleton
091a95d059
0.1.5 2025-11-11 17:31:54 +00:00