* feat: add tier-check CLI for SDK tier assessment
Adds a 'tier-check' subcommand to the conformance tool that automates
SDK tier assessment against SEP-1730 criteria.
Checks performed:
- Conformance test pass rate (via everything-server)
- GitHub label taxonomy (priority/status/area labels)
- Issue triage SLA compliance
- P0 bug resolution tracking
- Stable release detection
- Required file existence (CHANGELOG, SECURITY, etc.)
- Spec tracking (SDK release within 30d of spec release)
Also includes a Claude Code skill (skills/mcp-sdk-tier-audit/) for
judgment-based checks that require codebase analysis (feature coverage,
docs quality, policy evaluation).
Usage:
npx tsx src/index.ts tier-check --repo modelcontextprotocol/typescript-sdk
npx tsx src/index.ts tier-check --repo ... --conformance-server-cmd '...' \
--conformance-server-cwd ... --conformance-server-url ... --output json
* refactor: revise tier-check CLI and skill based on review feedback
- Move skill to .claude/skills/ so it's auto-available in Claude Code
- Remove feature-coverage subagent (redundant with conformance tests)
- Remove hardcoded ~/src/mcp paths from all skill files
- Trim conformance server table to TS + Python only
- Rename file_existence check to policy_signals (informational, not blocking)
- Add GitHub native issue types detection to labels check
- Add missing features to docs-coverage checklist (tasks, elicitation URL mode, JSON Schema 2020-12)
- Add README with CLI quick start and escape hatch for non-Claude-Code users
- Use --limit 500 instead of --limit 100 for gh issue list
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: address review feedback on skill and README
- Use npx @modelcontextprotocol/conformance instead of node dist/index.js
- Add full GitHub auth instructions (gh auth login, GITHUB_TOKEN, --token)
- Point TS SDK conformance server to typescript-sdk/test/conformance/
- Fix Python SDK URL to localhost:3001/mcp (not TBD)
- Remove manual gh issue list / gh release list from SKILL.md (CLI handles it)
- Remove Claude Code-specific subagent_type references
- Assume user is already in conformance repo
- Clean up policy-evaluation-prompt.md: remove redundant grep commands,
focus on content evaluation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve lint errors in conformance.ts
Remove unused variable assignments flagged by eslint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: apply prettier formatting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: add npm run tier-check script, update docs with examples
- Add "tier-check" npm script so users can run `npm run tier-check --`
instead of `node dist/index.js tier-check`
- Update SKILL.md, README, and skill README to use npm run tier-check
- Add full conformance examples with --conformance-server-cmd/cwd/url
flags and realistic paths (~/src/mcp/typescript-sdk)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: fix prettier formatting in SKILL.md
* fix: add per-scenario timeout, support url-only conformance, fix stdout pollution
- Add 30s per-scenario timeout to prevent tier-check from hanging
- Allow --conformance-server-url without --conformance-server-cmd (server already running)
- Move runner status logs to stderr so --output json produces clean JSON
- Update SKILL.md and README with --silent flag and pre-start server workflow
* refactor: skill takes local path + server URL instead of repo name
The skill now requires two arguments:
1. Local path to the SDK checkout (for direct file inspection)
2. URL where the everything server is already running
The GitHub owner/repo is derived from git remote. This eliminates
cloning, server startup complexity, and branch confusion (v1.x vs main).
* refactor: write detailed reports to files, show concise summary
Reports go to results/tier-audits/<sdk>-<date>/ (already gitignored).
Claude's console output is now just the tier classification,
pass/fail summary line, top 3 actions, and file paths.
* docs: update READMEs for new skill interface and pre-start workflow
* simplify: flat file output instead of nested directory
* fix: remediation always shows path to Tier 2 and Tier 1
* feat: add client conformance testing to tier-check CLI and skill
- Add checkClientConformance() that runs core client scenarios
(initialize, tools_call, elicitation-defaults, sse-retry, auth)
by spawning the SDK's conformance client via --client-cmd
- Add client_conformance to TierScorecard type
- Wire --client-cmd option into the CLI
- Update tier logic: both server + client conformance feed into
Tier 1 (100%) and Tier 2 (>=80%) requirements
- Update terminal and markdown output to show both conformance types
- Update skill to auto-detect conformance client or accept explicit
client-cmd argument
- Update README with new option and examples
* improve: table summary output, write reports via subagents
- Change executive summary from pipe-delimited line to a readable
table with T2/T1 columns
- Move assessment and remediation file writing into parallel
subagents to keep the main conversation thread clean
* improve: list tier gaps as numbered items instead of one-line blob
* improve: finalize summary format with separator, high-priority fixes, numbered gaps
* improve: add pre-flight checks for gh auth and server reachability
Fail fast with clear error messages if GitHub CLI is not authenticated
or if the conformance server URL is not reachable, rather than failing
deep into the scorecard run.
* docs: improve README and fix skill auto-detection paths
- Claude Code section: explain client-cmd auto-detection for TS/Python,
show explicit 3-arg form for other SDKs, add examples for all three
- Fix TypeScript build command (npm run build, not pnpm build:all)
- Fix Python server command (add --port, use uv sync --package)
- Fix Python client path (.github/actions/conformance/client.py)
- Expand 'Other SDKs' section with guidance on everything server
- Add gh auth login prerequisite to Claude Code steps
* simplify: remove client-cmd auto-detection, require explicit argument
Client command is now always passed as the third argument. If omitted,
client conformance is skipped and noted as a gap. No more magic path
detection — clearer and more predictable.
* fix: align docs table with canonical list (48 features), simplify policy eval
Docs coverage:
- Table now has numbered rows matching all 48 non-experimental features
from the canonical list (was missing 7: tools text/image/audio/embedded/
error/notifications, protocol version negotiation)
- Hardcode total as 48 in summary so agents don't miscount
Policy evaluation:
- Simplified from deep content analysis to file-existence checks
- Dependency policy: DEPENDENCY_POLICY.md, dependabot.yml, or CONTRIBUTING.md section
- Roadmap: ROADMAP.md must exist (GitHub milestones alone not sufficient)
- Versioning: VERSIONING.md or CONTRIBUTING.md section
- Removed GitHub API calls for milestones and releases from policy eval
* refactor: extract canonical feature list into single source of truth
Create references/feature-list.md with all 48 non-experimental + 5
experimental features. The docs-coverage prompt now references this
file instead of duplicating the list. One place to update when
features change.
* fix: separate deterministic file checks from AI content evaluation
CLI (files.ts): now checks all policy files deterministically —
DEPENDENCY_POLICY.md, docs/dependency-policy.md, dependabot.yml,
renovate.json, ROADMAP.md, docs/roadmap.md, VERSIONING.md,
docs/versioning.md, BREAKING_CHANGES.md (in addition to existing
CHANGELOG.md, SECURITY.md, CONTRIBUTING.md).
AI policy eval: receives CLI output showing which files exist,
then reads ONLY those files to judge content quality. No longer
searches the repo for files — clean separation of concerns.
* style: apply prettier formatting
* revert: undo unrelated console.log change in runner/server.ts
* refactor: shell out to conformance CLI instead of reimplementing runner
Address PR feedback: conformance.ts was duplicating the normal conformance
running code. Now shells out to 'node dist/index.js server/client' with
-o to save results to a temp dir, then parses the checks.json files.
Also removes --conformance-server-cmd and --conformance-server-cwd options
since the server must be pre-started.
* docs: add Go and C# SDK examples to README and SKILL.md
* fix: add --framework net9.0 to C# server command
* rename: conformance.ts -> test-conformance-results.ts
Avoids confusion with src/runner/ (the actual conformance runner).
This file just invokes the CLI and parses output.
* style: prettier formatting
* fix: reconcile conformance results against full scenario list
The tier-check CLI was only counting scenarios that produced a
checks.json file. Scenarios that crashed or failed to run (e.g., auth
scenarios when OAuth is not implemented) were invisible, making the
denominator artificially small (e.g., 4/4 instead of 4/23).
Now both checkConformance and checkClientConformance reconcile their
parsed results against the known scenario lists, adding failure entries
for any expected scenario that didn't produce results.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: tighten documentation evaluation criteria
Clarify what counts as documented vs just having code:
- Conformance test servers don't count as docs or examples
- Examples without prose = PARTIAL, not PASS
- Go Example* test functions explicitly allowed
- Clear PASS/PARTIAL/FAIL verdict definitions
* docs: add Labels and Spec Tracking rows to audit report templates
The executive summary and assessment report were missing two SEP-1730
requirements: label taxonomy compliance and spec tracking (new protocol
features timeline).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: reuse ConformanceCheck type from src/types.ts instead of redefining
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix: bump SDK to 1.26.0, add session management to tools_call scenario
SDK 1.26.0 made Protocol.connect() throw if already connected to a
transport. The tools_call scenario was calling server.connect(transport)
on every request with the same Server instance, which now throws.
Fixed by adding proper session management: each initialize request
creates a new Server + Transport pair, and subsequent requests are
routed to the correct transport via the mcp-session-id header.
* fix: return 404 for invalid session IDs in tools_call POST handler
The else branch was returning 400 for both missing-session and
invalid/stale-session cases. The MCP spec requires 404 for invalid
session IDs and 400 only for non-initialization requests without
any session ID.
* refactor: keep tools_call stateless, create fresh server per request
Instead of adding session management, simply create a new Server
instance per request. This preserves the original stateless design
while fixing the SDK 1.26.0 Protocol.connect() restriction.
* chore: bump version to 0.1.12-alpha.0 for OIDC trusted publishing test
* ci: rename prerelease checkbox to publish_alpha, clarify no-publish default
* 0.1.12
* feat: add DNS rebinding protection conformance tests
Add a new server conformance scenario to verify that localhost MCP servers
properly validate Host headers to prevent DNS rebinding attacks.
Checks:
- localhost-host-rebinding-rejected: Verifies server returns 403 for
non-localhost Host headers (e.g., evil.example.com)
- localhost-host-valid-accepted: Verifies server accepts requests with
valid localhost Host headers
Also updates the everything-server example to use createMcpExpressApp()
which includes DNS rebinding protection by default.
Closes#103
Co-Authored-By: Claude <noreply@anthropic.com>
* feat: add negative test case for DNS rebinding protection
Add a minimal MCP server that intentionally omits DNS rebinding protection
to serve as a negative test case. This server is expected to FAIL the
dns-rebinding-protection scenario.
Also update README to document the negative test case.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: accept any 4xx status code for DNS rebinding rejection
Update the DNS rebinding protection test to accept any 4xx status code
as a valid rejection response, not just 403. This accommodates different
HTTP semantics:
- 403 Forbidden: Per MCP spec
- 421 Misdirected Request: RFC 7540 (semantically correct for DNS rebinding)
- Other 4xx: Implementation-specific error codes
* refactor: address PR review comments
- Simplify negative test server (remove event store, sessions)
- Update spec reference URL to security_best_practices
- Clarify scope: localhost servers without HTTPS/auth
- DRY up checks.push using spread operator
- Fix valid host check to require 2xx response
* fix: include error details in catch block
* refactor: simplify negative test server to stateless
* refactor: check both Host and Origin headers for DNS rebinding protection
- Send both Host and Origin headers in test requests so servers checking
either header will pass the conformance test
- Add second spec reference for Origin check (transports#security-warning)
- Update descriptions to mention both Host and Origin headers
- Fix protocol version to 2025-11-25
- Remove unnecessary 'as const' type assertions
---------
Co-authored-by: Claude <noreply@anthropic.com>
Add support for SDKs that don't yet pass all conformance tests to still
run conformance in CI without failing, by specifying a YAML baseline file
of known/expected failures.
CLI changes:
- Add --expected-failures <path> flag to both server and client commands
- Exit 0 if only expected failures occur
- Exit 1 if unexpected failures occur (regression)
- Exit 1 if an expected failure now passes (stale baseline, must update)
GitHub Action (action.yml):
- Composite action that builds from source and runs conformance tests
- SDKs reference it as: uses: modelcontextprotocol/conformance@<ref>
- Inputs: mode, url, command, expected-failures, suite, scenario, etc.
New files:
- src/expected-failures.ts: YAML loading, baseline evaluation, reporting
- src/expected-failures.test.ts: 16 unit tests
- action.yml: composite GitHub Action
Closes#99
* [auth] Add metadata suite for running metadata discovery tests
Adds a new 'metadata' suite that runs just the auth/metadata-* scenarios
for faster iteration when testing metadata discovery specifically.
Usage: node dist/index.mjs client --suite metadata --command "..."
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Don't require lefthook to be installed
Set assert_lefthook_installed to false so the hooks gracefully skip
if lefthook is not installed on the system.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add token endpoint auth method conformance tests
Add three new scenarios to test that clients correctly use the
appropriate authentication method based on server metadata:
- auth/token-endpoint-auth-basic: Tests client_secret_basic (HTTP Basic)
- auth/token-endpoint-auth-post: Tests client_secret_post
- auth/token-endpoint-auth-none: Tests public client (no auth)
Each scenario configures the server to only support one auth method
and verifies the client uses the correct method in token requests.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Refactor token endpoint auth tests to use shared createAuthServer
Extend createAuthServer helper with:
- tokenEndpointAuthMethodsSupported option for metadata
- onTokenRequest callback now receives full Request object
- onRegistrationRequest callback for custom client credentials
This eliminates the duplicate auth server implementation in
token-endpoint-auth.ts and reduces code by ~140 lines.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix onTokenRequest callback to use new interface
The createAuthServer onTokenRequest callback now passes an object with
authorizationHeader, body, timestamp etc. instead of (req, timestamp).
Also added missing return value for TokenRequestResult.
* Update @modelcontextprotocol/sdk to 1.25.1
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Olivier Chafik <ochafik@anthropic.com>
* Add client credentials conformance scenarios with dynamic keypair generation
Implements SEP-1046 client credentials conformance tests:
- auth/client-credentials-jwt: Tests private_key_jwt authentication
- auth/client-credentials-basic: Tests client_secret_basic authentication
Key changes:
- Generate EC P-256 keypair dynamically at test start (no hardcoded keys)
- Pass credentials to client via MCP_CONFORMANCE_CONTEXT environment variable
- Add context field to ScenarioUrls interface for scenario-specific data
- Update client runner to pass context as env var to spawned client process
The MCP_CONFORMANCE_CONTEXT env var contains a JSON object with:
- client_id: The expected client identifier
- private_key_pem: PEM-encoded private key (for JWT scenarios)
- client_secret: Client secret (for basic auth scenarios)
- signing_algorithm: JWT signing algorithm (defaults to ES256)
* Fix client-credentials scenarios for conformance testing
- Generate EC keypair with extractable: true so private key can be
exported and passed to clients via MCP_CONFORMANCE_CONTEXT
- Fix client_secret_basic scenario to use authorizationHeader param
instead of non-existent headers object
* Fix jose type import - use CryptoKey instead of KeyLike
* Add OAuth 2.1 section 4.2 spec reference to client credentials checks
Add OAUTH_2_1_CLIENT_CREDENTIALS spec reference pointing to OAuth 2.1
draft section 4.2 (Client Credentials Grant) and include it in all
client_credentials conformance checks for both JWT and basic auth flows.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add JWT iss claim verification for client credentials
Per RFC 7523, verify that the JWT issuer (iss) claim matches the
expected client_id, in addition to the existing sub claim check.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* lefthook: move deps installs to optional local change
* add test scenario for infinite retry
and negative test
* Add broken client for retry limit negative test and fix lefthook formatting
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix www-authenticate in scope scenario
* add suites and shift pending to reflect current pending
* lint
---------
Co-authored-by: Claude <noreply@anthropic.com>
* wip scope change
* rm useless bool
* fix-me: package changes for linking
* negative test
* negative test for multiple scopes
* fix displays of warnings
* wip step-up
* negative test for step up auth
* move warning into main request flow
* clean up more checks
* cleanup step-up scenario
* cleanup and simplify
* cleanup middleware
* inline negative tests
* inline the other test
* refactor logging and inlining for standalone execution too
* skip pending tests
* clean up constructor calls for server lifecycle
* add back compat test
* fix dup logging, rm prm
* avoid fallbacks
* better error from example client
* add final endpoint method
* add test for OASM priority order
* s/prepack/prepare/ for npx
* try switching to .js
* make tsdown version happy default
* fix lock
* fix prmPath issue
* back to main package-lock
* fix bin path
* Add comprehensive auth scenario test suite
- Add index.test.ts to test all auth scenarios with auth-test.ts client
- Upgrade @modelcontextprotocol/sdk from 1.20.1 to 1.22.0
- All 5 auth scenarios now pass including auth/basic-metadata-var2
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* consolidate tests into index.test.ts
* rename metadata
* fix comments
* rework auth scenario listing
* add 3rd variation of metadata
* fix some type issues introduced by version bump
* add reusable spec references
* more refs
* fix up spec references
---------
Co-authored-by: Claude <noreply@anthropic.com>
Downgrade @typescript/native-preview and tsdown to previous versions.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
Updates vitest from v2.1.8 to v4.0.5. All tests pass successfully with no code changes required.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
* readme for auth tests
* wip prm test
* add example helper for local oauth w/o browser
* working through dangling timeout issue
* add better test function
* pretty printing
* add mcp method
* some more logging tweaks
* refactor logger into 1 class
* use common request logger
* fix base scenario
* some renaming
* refactor into helpers
* refactor to support second test
* rm readme
* add comment about www-authenticate
