* feat: add specVersions classification to conformance scenarios
Each scenario declares which spec versions it applies to as a list.
Scenarios that carry forward (e.g. initialize) list all applicable versions
['2025-06-18', '2025-11-25']. Scenarios removed from newer specs (e.g.
backcompat auth) only list their original version ['2025-03-26'].
- specVersions list on Scenario and ClientScenario interfaces
- --spec-version CLI filter uses simple .includes()
- Tier-check conformance matrix (Server / Client: Core / Client: Auth)
with per-version columns and unique All* count
- 7 unit tests for specVersions helpers
- Updated tier-audit skill docs with matrix format
* fix: fix console output template for tier audit skill
The template had an orphan table header (Check | Value | T2 | T1)
with no rows above the conformance matrix, causing an empty table
to render. The scorecard rows below the matrix also lacked their
own header.
Fix: two self-contained tables with clear labels — 'Conformance:'
for the per-version matrix, 'Scorecard:' for the check rows.
* fix: align skill console template with tier-check script output
Add asterisk footnote ('unique scenarios — a scenario may apply to
multiple spec versions') and rename 'Scorecard' to 'Repository Health'
to match the labels used by the tier-check CLI output.
* fix: add specVersions to CrossAppAccessCompleteFlowScenario
Added in 83c446d on main after this branch diverged.
Move auth/2025-03-26-oauth-metadata-backcompat and
auth/2025-03-26-oauth-endpoint-fallback out of the required
authScenariosList into a new backcompatScenariosList.
These test backward compatibility with the old 2025-03-26 auth spec
(no PRM, OAuth metadata at server root) which is not part of the
current spec requirements.
- Add new 'backcompat' client suite
- Back-compat scenarios remain in the 'all' suite
- Removed from 'core' and 'auth' suites
- Add separate test describe block for back-compat scenarios
Closes#126
* feat: add core and extensions suites for SDK tiering
Add new suites to support SDK tiering requirements:
Client suites:
- core (19 scenarios): Tier 1 requirements (base + auth)
- extensions (2 scenarios): Optional protocol extensions (client credentials)
- all (21 scenarios): Everything (core + extensions)
- auth (15 scenarios): Auth scenarios only (excludes extensions)
Server suites:
- core: Alias for 'active' (tier 1 requirements)
This allows SDK developers to run --suite core for tier 1 certification
while keeping extension tests available but not blocking.
* feat: promote elicitation tests from pending to active
Move 3 elicitation scenarios to active suite:
- elicitation-sep1330-enums
- tools-call-elicitation
- elicitation-sep1034-defaults
These tests are now passing in the Python SDK and ready for
SDK tiering requirements.
Remaining pending tests (2):
- json-schema-2020-12 (blocked on SDK PR #1135)
- server-sse-polling (blocked on SDK PR #1129)
* feat: add DNS rebinding protection conformance tests
Add a new server conformance scenario to verify that localhost MCP servers
properly validate Host headers to prevent DNS rebinding attacks.
Checks:
- localhost-host-rebinding-rejected: Verifies server returns 403 for
non-localhost Host headers (e.g., evil.example.com)
- localhost-host-valid-accepted: Verifies server accepts requests with
valid localhost Host headers
Also updates the everything-server example to use createMcpExpressApp()
which includes DNS rebinding protection by default.
Closes#103
Co-Authored-By: Claude <noreply@anthropic.com>
* feat: add negative test case for DNS rebinding protection
Add a minimal MCP server that intentionally omits DNS rebinding protection
to serve as a negative test case. This server is expected to FAIL the
dns-rebinding-protection scenario.
Also update README to document the negative test case.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: accept any 4xx status code for DNS rebinding rejection
Update the DNS rebinding protection test to accept any 4xx status code
as a valid rejection response, not just 403. This accommodates different
HTTP semantics:
- 403 Forbidden: Per MCP spec
- 421 Misdirected Request: RFC 7540 (semantically correct for DNS rebinding)
- Other 4xx: Implementation-specific error codes
* refactor: address PR review comments
- Simplify negative test server (remove event store, sessions)
- Update spec reference URL to security_best_practices
- Clarify scope: localhost servers without HTTPS/auth
- DRY up checks.push using spread operator
- Fix valid host check to require 2xx response
* fix: include error details in catch block
* refactor: simplify negative test server to stateless
* refactor: check both Host and Origin headers for DNS rebinding protection
- Send both Host and Origin headers in test requests so servers checking
either header will pass the conformance test
- Add second spec reference for Origin check (transports#security-warning)
- Update descriptions to mention both Host and Origin headers
- Fix protocol version to 2025-11-25
- Remove unnecessary 'as const' type assertions
---------
Co-authored-by: Claude <noreply@anthropic.com>
* lefthook: move deps installs to optional local change
* add test scenario for infinite retry
and negative test
* Add broken client for retry limit negative test and fix lefthook formatting
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix www-authenticate in scope scenario
* add suites and shift pending to reflect current pending
* lint
---------
Co-authored-by: Claude <noreply@anthropic.com>
* add cimd test and negative test
* Improve auth-test-no-cimd.ts documentation for clarity
Address review feedback: expanded comments to better explain what CIMD is,
why this client is non-compliant, and what behavior it's testing.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Remove .js extensions from relative imports
tsx doesn't require .js extensions for TypeScript imports.
Keeps .js for @modelcontextprotocol/sdk imports as those are external packages.
Also improves auth-test-no-cimd.ts documentation for clarity.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Skip auth/basic-cimd test until SDK support lands
Waiting on typescript-sdk PR #1127 for CIMD support.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix trailing whitespace in auth-test-no-cimd.ts
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Add conformance test to validate that MCP servers correctly preserve
JSON Schema 2020-12 keywords ($schema, $defs, additionalProperties)
in tool definitions.
Changes:
- Add JsonSchema2020_12Scenario in src/scenarios/server/json-schema-2020-12.ts
- Add json_schema_2020_12_tool to everything-server with tools/list override
to return raw JSON Schema instead of Zod-converted schema
- Register scenario in pendingClientScenariosList (requires SDK PR #1135)
The test is added to pending because the client SDK currently strips
the JSON Schema 2020-12 keywords. Once PR #1135 is merged into the SDK,
this scenario can be moved to the active list.
* [auth] Add metadata suite for running metadata discovery tests
Adds a new 'metadata' suite that runs just the auth/metadata-* scenarios
for faster iteration when testing metadata discovery specifically.
Usage: node dist/index.mjs client --suite metadata --command "..."
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Don't require lefthook to be installed
Set assert_lefthook_installed to false so the hooks gracefully skip
if lefthook is not installed on the system.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Adds a --suite option to the client command that runs groups of related
tests concurrently and outputs a summary. Currently supports the 'auth' suite.
Usage: client --suite auth --command "tsx examples/clients/typescript/auth-test.ts"
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-authored-by: Claude <noreply@anthropic.com>
* clean up constructor calls for server lifecycle
* add back compat test
* fix dup logging, rm prm
* avoid fallbacks
* better error from example client
* add final endpoint method
* add test for OASM priority order
* s/prepack/prepare/ for npx
* try switching to .js
* make tsdown version happy default
* fix lock
* fix prmPath issue
* back to main package-lock
* fix bin path
* Add comprehensive auth scenario test suite
- Add index.test.ts to test all auth scenarios with auth-test.ts client
- Upgrade @modelcontextprotocol/sdk from 1.20.1 to 1.22.0
- All 5 auth scenarios now pass including auth/basic-metadata-var2
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* consolidate tests into index.test.ts
* rename metadata
* fix comments
* rework auth scenario listing
* add 3rd variation of metadata
* fix some type issues introduced by version bump
* add reusable spec references
* more refs
* fix up spec references
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Add SEP-1330 conformance test for elicitation enum schemas
Adds server-side conformance test that validates servers properly request
elicitation with SEP-1330 enum schema improvements:
- Untitled single-select enum (type: string, enum: [...])
- Titled single-select enum (oneOf with const/title)
- Legacy titled enum (enumNames for backward compatibility)
- Untitled multi-select enum (type: array, items.enum)
- Titled multi-select enum (items.anyOf with const/title)
Test expects server to implement `test_elicitation_sep1330_enums` tool.
* Add reference server implementation for SEP-1330 conformance test
Adds test_elicitation_sep1330_enums tool to everything-server that
requests elicitation with all 5 enum schema variants. Also updates
SERVER_REQUIREMENTS.md with full specification.
* Fix prettier formatting
* only test active tests (#24)
* refactor maps to pull names from the test
* bring names back
* separate active from pending
* only test active
---------
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
* feat: add client-side SEP-1034 conformance test for elicitation defaults
Adds a conformance test that validates client SDKs properly apply
default values for omitted fields in elicitation responses per SEP-1034.
The test spins up a server that sends elicitation requests with defaults,
and validates that the client returns content with those defaults applied.
* feat: use elicitInput() for SEP-1034 default validation
Updates client-side test to validate server-side default application
using elicitInput() helper, which applies defaults via validation layer.
This aligns with the TypeScript SDK's SEP-1034 implementation approach.
The test now validates:
- Server uses elicitInput() with schemas containing defaults
- Client returns empty/partial content
- Server automatically applies defaults during validation
- Defaults are correctly merged for all primitive types
* refactor: use server.request() instead of elicitInput() for client-side defaults test
Ensures conformance test validates true client-side default application
rather than server-side validation behavior.
* feat: add applyDefaults capability flag for opt-in default application
Enables conformance test to opt into client-side elicitation defaults
by setting the applyDefaults capability flag.
Add comprehensive conformance test for SEP-1034 which extends MCP
elicitation to support default values across all primitive types.
Changes:
- Add test_elicitation_sep1034_defaults tool to reference server
- Create ElicitationDefaultsScenario with 5 granular checks:
* String schema default value
* Integer schema default value
* Number schema default value
* Enum schema default value (validates it's a valid enum member)
* Boolean schema default value (regression test)
- Register scenario in index.ts
The test validates that servers properly include default values in
elicitation requestedSchema per SEP-1034 specification.
Test results: All 5 checks pass against TypeScript reference server
* readme for auth tests
* wip prm test
* add example helper for local oauth w/o browser
* working through dangling timeout issue
* add better test function
* pretty printing
* add mcp method
* some more logging tweaks
* refactor logger into 1 class
* use common request logger
* fix base scenario
* some renaming
* refactor into helpers
* refactor to support second test
* rm readme
* add comment about www-authenticate
* spec for servers
* initial version of everything server and tests - tets are very simple
* some clean up
* lint etc.
* fix package lock agian
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
