Add -o, --output-dir option to both client and server commands.
By default, no results directory is created - only console output.
When -o is specified, results are saved to that directory with
timestamped subdirectories.
This eliminates the need to add 'results/' to .gitignore in every
repo where conformance tests are run.
Claude-Generated-By: Claude Code (cli/claude-opus-4-5=100%)
Claude-Steers: 0
Claude-Permission-Prompts: 2
Claude-Escapes: 0
- Switch from weekly to monthly schedule
- Group all npm dependencies into single PR
- Group all GitHub actions into single PR
- Add explicit config for examples/servers/typescript directory
- Remove cooldown config (less relevant with monthly schedule)
* Add client credentials support to everything-client
- Add ConformanceContextSchema with discriminated union for auth contexts
- Add runClientCredentialsJwt and runClientCredentialsBasic handlers
- Update runner to include scenario name in context for discriminated union parsing
- Update test helpers to set env vars for inline client runners
- Fix JWT audience validation to match SDK behavior (no trailing slash)
- Remove client credentials scenarios from skip list
* Accept both trailing slash forms for JWT audience per RFC 3986
Per RFC 3986, URLs with and without trailing slash are equivalent.
The MCP spec recommends (SHOULD) the form without trailing slash, but
since it's not a MUST, the conformance test should accept both forms
for interoperability with clients like Pydantic that normalize URLs.
* Remove .vscode from tracking and add to gitignore
* Address review feedback
- Remove unused issuerUrl variable and comment
- Simplify audience array to exactly 2 values with clearer naming
- Add comment explaining strip-then-add-back logic
- Rename ConformanceContextSchema to ClientConformanceContextSchema
* Remove dist from tracking and add to gitignore
* Fix .gitignore formatting
* Add getHandler to everything-client for cleaner test imports
Instead of importing individual handlers, tests can now import getHandler
and look up handlers by scenario name.
* Use getHandler exclusively, remove goodClient import
- Add CIMD support to runAuthClient in everything-client
- Add auth fallback to getHandler for unregistered auth/* scenarios
- Remove goodClient import from test file
* Remove runAuthClient fallback, register all auth scenarios explicitly
* [auth] Add metadata suite for running metadata discovery tests
Adds a new 'metadata' suite that runs just the auth/metadata-* scenarios
for faster iteration when testing metadata discovery specifically.
Usage: node dist/index.mjs client --suite metadata --command "..."
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Don't require lefthook to be installed
Set assert_lefthook_installed to false so the hooks gracefully skip
if lefthook is not installed on the system.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add token endpoint auth method conformance tests
Add three new scenarios to test that clients correctly use the
appropriate authentication method based on server metadata:
- auth/token-endpoint-auth-basic: Tests client_secret_basic (HTTP Basic)
- auth/token-endpoint-auth-post: Tests client_secret_post
- auth/token-endpoint-auth-none: Tests public client (no auth)
Each scenario configures the server to only support one auth method
and verifies the client uses the correct method in token requests.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Refactor token endpoint auth tests to use shared createAuthServer
Extend createAuthServer helper with:
- tokenEndpointAuthMethodsSupported option for metadata
- onTokenRequest callback now receives full Request object
- onRegistrationRequest callback for custom client credentials
This eliminates the duplicate auth server implementation in
token-endpoint-auth.ts and reduces code by ~140 lines.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix onTokenRequest callback to use new interface
The createAuthServer onTokenRequest callback now passes an object with
authorizationHeader, body, timestamp etc. instead of (req, timestamp).
Also added missing return value for TokenRequestResult.
* Update @modelcontextprotocol/sdk to 1.25.1
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Olivier Chafik <ochafik@anthropic.com>
* Refactor client testing to pass scenario name and server URL
The CLI now passes both scenario name and server URL as arguments to the
client command (instead of just the server URL). This enables a single
"everything" client that can handle all scenarios based on the scenario
name it receives.
Changes:
- Update executeClient() to pass scenarioName as first argument to client
- Add everything-client.ts that routes to appropriate behavior based on
scenario name
- Update README with new usage examples and documentation
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Use env var for scenario name instead of positional arg
Pass scenario name via MCP_CONFORMANCE_SCENARIO env var instead of as a
positional argument. This is non-breaking for existing clients that just
accept <server-url>.
The env var approach is consistent with how context (including private
keys for JWT auth) is already passed via MCP_CONFORMANCE_CONTEXT.
* Add comment explaining env var design decision
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Olivier Chafik <ochafik@anthropic.com>
- Update client initialization check to accept both 2025-06-18 and 2025-11-25
- Server echoes back client's version if valid (proper version negotiation)
- Fixes compatibility with SDK 1.25.1+ which sends 2025-11-25
🦉 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
spawn('npx', ...) fails on Windows because npx is actually npx.cmd.
Adding shell: true only on Windows fixes this while preserving
Unix signal handling behavior.
🦉 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* fix: validate scope checks ran and report overall test status
- Add FAILURE checks to scope scenarios when authorization flow doesn't
complete (ScopeFromWwwAuthenticateScenario, ScopeFromScopesSupportedScenario,
ScopeOmittedWhenUndefinedScenario)
- Report client timeout and exit errors in test results
- Add OVERALL: PASSED/FAILED summary at end of test output
- Exit with code 1 on any failure (check failures, client timeout, or exit error)
Previously, if the OAuth authorization flow didn't complete, the scope
checks were never triggered and the test would incorrectly report success.
Now these scenarios emit a FAILURE if the expected scope check wasn't recorded.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Count warnings as failures in overall test result
Warnings now contribute to the overall failure status, causing the test
to exit with code 1 when any warnings are present. Also displays warning
checks in the output summary, similar to how failures are displayed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Count warnings as failures in suite runner
Update the suite summary to show ✗ for scenarios with warnings,
and exit with code 1 if any warnings are present across the suite.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Add client credentials conformance scenarios with dynamic keypair generation
Implements SEP-1046 client credentials conformance tests:
- auth/client-credentials-jwt: Tests private_key_jwt authentication
- auth/client-credentials-basic: Tests client_secret_basic authentication
Key changes:
- Generate EC P-256 keypair dynamically at test start (no hardcoded keys)
- Pass credentials to client via MCP_CONFORMANCE_CONTEXT environment variable
- Add context field to ScenarioUrls interface for scenario-specific data
- Update client runner to pass context as env var to spawned client process
The MCP_CONFORMANCE_CONTEXT env var contains a JSON object with:
- client_id: The expected client identifier
- private_key_pem: PEM-encoded private key (for JWT scenarios)
- client_secret: Client secret (for basic auth scenarios)
- signing_algorithm: JWT signing algorithm (defaults to ES256)
* Fix client-credentials scenarios for conformance testing
- Generate EC keypair with extractable: true so private key can be
exported and passed to clients via MCP_CONFORMANCE_CONTEXT
- Fix client_secret_basic scenario to use authorizationHeader param
instead of non-existent headers object
* Fix jose type import - use CryptoKey instead of KeyLike
* Add OAuth 2.1 section 4.2 spec reference to client credentials checks
Add OAUTH_2_1_CLIENT_CREDENTIALS spec reference pointing to OAuth 2.1
draft section 4.2 (Client Credentials Grant) and include it in all
client_credentials conformance checks for both JWT and basic auth flows.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add JWT iss claim verification for client credentials
Per RFC 7523, verify that the JWT issuer (iss) claim matches the
expected client_id, in addition to the existing sub claim check.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* lefthook: move deps installs to optional local change
* add test scenario for infinite retry
and negative test
* Add broken client for retry limit negative test and fix lefthook formatting
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix www-authenticate in scope scenario
* add suites and shift pending to reflect current pending
* lint
---------
Co-authored-by: Claude <noreply@anthropic.com>
* add cimd test and negative test
* Improve auth-test-no-cimd.ts documentation for clarity
Address review feedback: expanded comments to better explain what CIMD is,
why this client is non-compliant, and what behavior it's testing.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Remove .js extensions from relative imports
tsx doesn't require .js extensions for TypeScript imports.
Keeps .js for @modelcontextprotocol/sdk imports as those are external packages.
Also improves auth-test-no-cimd.ts documentation for clarity.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Skip auth/basic-cimd test until SDK support lands
Waiting on typescript-sdk PR #1127 for CIMD support.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix trailing whitespace in auth-test-no-cimd.ts
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Add conformance test to validate that MCP servers correctly preserve
JSON Schema 2020-12 keywords ($schema, $defs, additionalProperties)
in tool definitions.
Changes:
- Add JsonSchema2020_12Scenario in src/scenarios/server/json-schema-2020-12.ts
- Add json_schema_2020_12_tool to everything-server with tools/list override
to return raw JSON Schema instead of Zod-converted schema
- Register scenario in pendingClientScenariosList (requires SDK PR #1135)
The test is added to pending because the client SDK currently strips
the JSON Schema 2020-12 keywords. Once PR #1135 is merged into the SDK,
this scenario can be moved to the active list.
* fix server elicitation conf
* Remove empty elicitation object from server config
* Use ElicitResultSchema instead of z.any() for proper validation
The elicitation/create response should be validated against ElicitResultSchema
which ensures the response has the correct structure (action, content fields)
rather than accepting any response.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
* [auth] Add metadata suite for running metadata discovery tests
Adds a new 'metadata' suite that runs just the auth/metadata-* scenarios
for faster iteration when testing metadata discovery specifically.
Usage: node dist/index.mjs client --suite metadata --command "..."
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Don't require lefthook to be installed
Set assert_lefthook_installed to false so the hooks gracefully skip
if lefthook is not installed on the system.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Adds a --suite option to the client command that runs groups of related
tests concurrently and outputs a summary. Currently supports the 'auth' suite.
Usage: client --suite auth --command "tsx examples/clients/typescript/auth-test.ts"
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-authored-by: Claude <noreply@anthropic.com>
This shows `npx` commands instead of `npm i` in PR preview comments,
which is more appropriate for this CLI tool.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
Lefthook was incorrectly listed as a production dependency, causing it
to install git hooks in users' repositories when they install this
conformance testing package. This is undesirable behavior - lefthook
should only be used by contributors to this repo.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
* wip scope change
* rm useless bool
* fix-me: package changes for linking
* negative test
* negative test for multiple scopes
* fix displays of warnings
* wip step-up
* negative test for step up auth
* move warning into main request flow
* clean up more checks
* cleanup step-up scenario
* cleanup and simplify
* cleanup middleware
* inline negative tests
* inline the other test
* refactor logging and inlining for standalone execution too
* skip pending tests
* clean up constructor calls for server lifecycle
* add back compat test
* fix dup logging, rm prm
* avoid fallbacks
* better error from example client
* add final endpoint method
* add test for OASM priority order
* s/prepack/prepare/ for npx
* try switching to .js
* make tsdown version happy default
* fix lock
* fix prmPath issue
* back to main package-lock
* fix bin path
* Add comprehensive auth scenario test suite
- Add index.test.ts to test all auth scenarios with auth-test.ts client
- Upgrade @modelcontextprotocol/sdk from 1.20.1 to 1.22.0
- All 5 auth scenarios now pass including auth/basic-metadata-var2
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* consolidate tests into index.test.ts
* rename metadata
* fix comments
* rework auth scenario listing
* add 3rd variation of metadata
* fix some type issues introduced by version bump
* add reusable spec references
* more refs
* fix up spec references
---------
Co-authored-by: Claude <noreply@anthropic.com>
Downgrade @typescript/native-preview and tsdown to previous versions.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
Updates vitest from v2.1.8 to v4.0.5. All tests pass successfully with no code changes required.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
