3b4a92baad* feat: add tier-check CLI for SDK tier assessment
Adds a 'tier-check' subcommand to the conformance tool that automates
SDK tier assessment against SEP-1730 criteria.
Checks performed:
- Conformance test pass rate (via everything-server)
- GitHub label taxonomy (priority/status/area labels)
- Issue triage SLA compliance
- P0 bug resolution tracking
- Stable release detection
- Required file existence (CHANGELOG, SECURITY, etc.)
- Spec tracking (SDK release within 30d of spec release)
Also includes a Claude Code skill (skills/mcp-sdk-tier-audit/) for
judgment-based checks that require codebase analysis (feature coverage,
docs quality, policy evaluation).
Usage:
npx tsx src/index.ts tier-check --repo modelcontextprotocol/typescript-sdk
npx tsx src/index.ts tier-check --repo ... --conformance-server-cmd '...' \
--conformance-server-cwd ... --conformance-server-url ... --output json
* refactor: revise tier-check CLI and skill based on review feedback
- Move skill to .claude/skills/ so it's auto-available in Claude Code
- Remove feature-coverage subagent (redundant with conformance tests)
- Remove hardcoded ~/src/mcp paths from all skill files
- Trim conformance server table to TS + Python only
- Rename file_existence check to policy_signals (informational, not blocking)
- Add GitHub native issue types detection to labels check
- Add missing features to docs-coverage checklist (tasks, elicitation URL mode, JSON Schema 2020-12)
- Add README with CLI quick start and escape hatch for non-Claude-Code users
- Use --limit 500 instead of --limit 100 for gh issue list
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: address review feedback on skill and README
- Use npx @modelcontextprotocol/conformance instead of node dist/index.js
- Add full GitHub auth instructions (gh auth login, GITHUB_TOKEN, --token)
- Point TS SDK conformance server to typescript-sdk/test/conformance/
- Fix Python SDK URL to localhost:3001/mcp (not TBD)
- Remove manual gh issue list / gh release list from SKILL.md (CLI handles it)
- Remove Claude Code-specific subagent_type references
- Assume user is already in conformance repo
- Clean up policy-evaluation-prompt.md: remove redundant grep commands,
focus on content evaluation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve lint errors in conformance.ts
Remove unused variable assignments flagged by eslint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: apply prettier formatting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: add npm run tier-check script, update docs with examples
- Add "tier-check" npm script so users can run `npm run tier-check --`
instead of `node dist/index.js tier-check`
- Update SKILL.md, README, and skill README to use npm run tier-check
- Add full conformance examples with --conformance-server-cmd/cwd/url
flags and realistic paths (~/src/mcp/typescript-sdk)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: fix prettier formatting in SKILL.md
* fix: add per-scenario timeout, support url-only conformance, fix stdout pollution
- Add 30s per-scenario timeout to prevent tier-check from hanging
- Allow --conformance-server-url without --conformance-server-cmd (server already running)
- Move runner status logs to stderr so --output json produces clean JSON
- Update SKILL.md and README with --silent flag and pre-start server workflow
* refactor: skill takes local path + server URL instead of repo name
The skill now requires two arguments:
1. Local path to the SDK checkout (for direct file inspection)
2. URL where the everything server is already running
The GitHub owner/repo is derived from git remote. This eliminates
cloning, server startup complexity, and branch confusion (v1.x vs main).
* refactor: write detailed reports to files, show concise summary
Reports go to results/tier-audits/<sdk>-<date>/ (already gitignored).
Claude's console output is now just the tier classification,
pass/fail summary line, top 3 actions, and file paths.
* docs: update READMEs for new skill interface and pre-start workflow
* simplify: flat file output instead of nested directory
* fix: remediation always shows path to Tier 2 and Tier 1
* feat: add client conformance testing to tier-check CLI and skill
- Add checkClientConformance() that runs core client scenarios
(initialize, tools_call, elicitation-defaults, sse-retry, auth)
by spawning the SDK's conformance client via --client-cmd
- Add client_conformance to TierScorecard type
- Wire --client-cmd option into the CLI
- Update tier logic: both server + client conformance feed into
Tier 1 (100%) and Tier 2 (>=80%) requirements
- Update terminal and markdown output to show both conformance types
- Update skill to auto-detect conformance client or accept explicit
client-cmd argument
- Update README with new option and examples
* improve: table summary output, write reports via subagents
- Change executive summary from pipe-delimited line to a readable
table with T2/T1 columns
- Move assessment and remediation file writing into parallel
subagents to keep the main conversation thread clean
* improve: list tier gaps as numbered items instead of one-line blob
* improve: finalize summary format with separator, high-priority fixes, numbered gaps
* improve: add pre-flight checks for gh auth and server reachability
Fail fast with clear error messages if GitHub CLI is not authenticated
or if the conformance server URL is not reachable, rather than failing
deep into the scorecard run.
* docs: improve README and fix skill auto-detection paths
- Claude Code section: explain client-cmd auto-detection for TS/Python,
show explicit 3-arg form for other SDKs, add examples for all three
- Fix TypeScript build command (npm run build, not pnpm build:all)
- Fix Python server command (add --port, use uv sync --package)
- Fix Python client path (.github/actions/conformance/client.py)
- Expand 'Other SDKs' section with guidance on everything server
- Add gh auth login prerequisite to Claude Code steps
* simplify: remove client-cmd auto-detection, require explicit argument
Client command is now always passed as the third argument. If omitted,
client conformance is skipped and noted as a gap. No more magic path
detection — clearer and more predictable.
* fix: align docs table with canonical list (48 features), simplify policy eval
Docs coverage:
- Table now has numbered rows matching all 48 non-experimental features
from the canonical list (was missing 7: tools text/image/audio/embedded/
error/notifications, protocol version negotiation)
- Hardcode total as 48 in summary so agents don't miscount
Policy evaluation:
- Simplified from deep content analysis to file-existence checks
- Dependency policy: DEPENDENCY_POLICY.md, dependabot.yml, or CONTRIBUTING.md section
- Roadmap: ROADMAP.md must exist (GitHub milestones alone not sufficient)
- Versioning: VERSIONING.md or CONTRIBUTING.md section
- Removed GitHub API calls for milestones and releases from policy eval
* refactor: extract canonical feature list into single source of truth
Create references/feature-list.md with all 48 non-experimental + 5
experimental features. The docs-coverage prompt now references this
file instead of duplicating the list. One place to update when
features change.
* fix: separate deterministic file checks from AI content evaluation
CLI (files.ts): now checks all policy files deterministically —
DEPENDENCY_POLICY.md, docs/dependency-policy.md, dependabot.yml,
renovate.json, ROADMAP.md, docs/roadmap.md, VERSIONING.md,
docs/versioning.md, BREAKING_CHANGES.md (in addition to existing
CHANGELOG.md, SECURITY.md, CONTRIBUTING.md).
AI policy eval: receives CLI output showing which files exist,
then reads ONLY those files to judge content quality. No longer
searches the repo for files — clean separation of concerns.
* style: apply prettier formatting
* revert: undo unrelated console.log change in runner/server.ts
* refactor: shell out to conformance CLI instead of reimplementing runner
Address PR feedback: conformance.ts was duplicating the normal conformance
running code. Now shells out to 'node dist/index.js server/client' with
-o to save results to a temp dir, then parses the checks.json files.
Also removes --conformance-server-cmd and --conformance-server-cwd options
since the server must be pre-started.
* docs: add Go and C# SDK examples to README and SKILL.md
* fix: add --framework net9.0 to C# server command
* rename: conformance.ts -> test-conformance-results.ts
Avoids confusion with src/runner/ (the actual conformance runner).
This file just invokes the CLI and parses output.
* style: prettier formatting
* fix: reconcile conformance results against full scenario list
The tier-check CLI was only counting scenarios that produced a
checks.json file. Scenarios that crashed or failed to run (e.g., auth
scenarios when OAuth is not implemented) were invisible, making the
denominator artificially small (e.g., 4/4 instead of 4/23).
Now both checkConformance and checkClientConformance reconcile their
parsed results against the known scenario lists, adding failure entries
for any expected scenario that didn't produce results.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: tighten documentation evaluation criteria
Clarify what counts as documented vs just having code:
- Conformance test servers don't count as docs or examples
- Examples without prose = PARTIAL, not PASS
- Go Example* test functions explicitly allowed
- Clear PASS/PARTIAL/FAIL verdict definitions
* docs: add Labels and Spec Tracking rows to audit report templates
The executive summary and assessment report were missing two SEP-1730
requirements: label taxonomy compliance and spec tracking (new protocol
features timeline).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: reuse ConformanceCheck type from src/types.ts instead of redefining
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
12 KiB
12 KiB
MCP SDK Tier Audit
Assess any MCP SDK repository against SEP-1730 (the SDK Tiering System). Produces a tier classification (1/2/3) with an evidence-backed scorecard.
Two components work together:
tier-checkCLI — runs deterministic checks (server + client conformance pass rate, issue triage speed, P0 resolution, labels, releases, policy signals). Works standalone, no AI needed.- AI-assisted assessment — an agent uses the CLI scorecard plus judgment-based evaluation (documentation coverage, dependency policy, roadmap) to produce a full tier report with remediation guide.
Quick Start: CLI
The CLI is a subcommand of the MCP Conformance tool.
# Clone and build
git clone https://github.com/modelcontextprotocol/conformance.git
cd conformance
npm install
npm run build
# Authenticate with GitHub (needed for API access)
gh auth login
# Run against any MCP SDK repo (without conformance tests)
npm run --silent tier-check -- --repo modelcontextprotocol/typescript-sdk --skip-conformance
The CLI uses the GitHub API (read-only) for issue metrics, labels, and release checks. Authenticate via one of:
- GitHub CLI (recommended):
gh auth login— the CLI picks up your token automatically - Environment variable:
export GITHUB_TOKEN=ghp_... - Flag:
--token ghp_...
For public repos, any authenticated token works (no special scopes needed — authentication just avoids rate limits). For a fine-grained personal access token, select Public Repositories (read-only) with no additional permissions.
CLI Options
--repo <owner/repo> GitHub repository (required)
--branch <branch> Branch to check
--skip-conformance Skip conformance tests
--conformance-server-url <url> URL of the already-running conformance server
--client-cmd <cmd> Command to run the SDK conformance client (for client conformance tests)
--days <n> Limit triage analysis to last N days
--output <format> json | markdown | terminal (default: terminal)
--token <token> GitHub token (defaults to GITHUB_TOKEN or gh auth token)
What the CLI Checks
| Check | What it measures |
|---|---|
| Server Conformance | Pass rate of server implementation against the conformance test suite |
| Client Conformance | Pass rate of client implementation against the conformance test suite |
| Labels | Whether SEP-1730 label taxonomy is set up (supports GitHub native issue types) |
| Triage | How quickly issues get labeled after creation |
| P0 Resolution | Whether critical bugs are resolved within SLA |
| Stable Release | Whether a stable release >= 1.0.0 exists |
| Policy Signals | Presence of CHANGELOG, SECURITY, CONTRIBUTING, dependabot, ROADMAP |
| Spec Tracking | Gap between latest spec release and SDK release |
Example Output
Tier Assessment: Tier 2
Repo: modelcontextprotocol/typescript-sdk
Timestamp: 2026-02-10T12:00:00Z
Check Results:
✓ Server Conformance 45/45 (100%)
✓ Client Conformance 4/4 (100%)
✗ Labels 9/12 required labels
Missing: needs confirmation, needs repro, ready for work
✓ Triage 92% within 2BD (150 issues, median 8h)
✓ P0 Resolution 0 open, 3/3 closed within 7d
✓ Stable Release 2.3.1
~ Policy Signals ✓ CHANGELOG.md, ✗ SECURITY.md, ✓ CONTRIBUTING.md, ✓ .github/dependabot.yml, ✗ ROADMAP.md
✓ Spec Tracking 2d gap
Use --output json to get machine-readable results, or --output markdown for a report you can paste into an issue.
Full AI-Assisted Assessment
The CLI produces a deterministic scorecard, but some SEP-1730 requirements need judgment: documentation quality, dependency policy, roadmap substance. An AI agent can evaluate these by reading the repo.
Claude Code
The skill lives in .claude/skills/ in this repo, so if you open Claude Code in the conformance repo it's already available.
- Make sure
gh auth loginis done (the skill checks this upfront) - Start the SDK's everything server in a separate terminal
- Run the skill:
/mcp-sdk-tier-audit <local-sdk-path> <conformance-server-url> [client-cmd]
Pass the client command as the third argument to include client conformance testing. If omitted, client conformance is skipped and noted as a gap in the report.
TypeScript SDK example:
# Terminal 1: start the everything server (build first: npm run build)
cd ~/src/mcp/typescript-sdk && npm run test:conformance:server:run
# Terminal 2: run the audit (from the conformance repo)
/mcp-sdk-tier-audit ~/src/mcp/typescript-sdk http://localhost:3000/mcp "npx tsx ~/src/mcp/typescript-sdk/test/conformance/src/everythingClient.ts"
Python SDK example:
# Terminal 1: install and start the everything server
cd ~/src/mcp/python-sdk && uv sync --frozen --all-extras --package mcp-everything-server
uv run mcp-everything-server --port 3001
# Terminal 2: run the audit (from the conformance repo)
/mcp-sdk-tier-audit ~/src/mcp/python-sdk http://localhost:3001/mcp "uv run python ~/src/mcp/python-sdk/.github/actions/conformance/client.py"
Go SDK example:
# Terminal 1: build and start the everything server
cd ~/src/mcp/go-sdk && go build -o /tmp/go-conformance-server ./conformance/everything-server
go build -o /tmp/go-conformance-client ./conformance/everything-client
/tmp/go-conformance-server -http="localhost:3002"
# Terminal 2: run the audit (from the conformance repo)
/mcp-sdk-tier-audit ~/src/mcp/go-sdk http://localhost:3002 "/tmp/go-conformance-client"
C# SDK example:
# Terminal 1: start the everything server (requires .NET SDK)
cd ~/src/mcp/csharp-sdk
dotnet run --project tests/ModelContextProtocol.ConformanceServer --framework net9.0 -- --urls http://localhost:3003
# Terminal 2: run the audit (from the conformance repo)
/mcp-sdk-tier-audit ~/src/mcp/csharp-sdk http://localhost:3003 "dotnet run --project ~/src/mcp/csharp-sdk/tests/ModelContextProtocol.ConformanceClient"
The skill derives owner/repo from git remote, runs the CLI, launches parallel evaluations for docs and policy, and writes detailed reports to results/.
Any Other AI Coding Agent
If you use a different agent (Codex, Cursor, Aider, OpenCode, etc.), give it these instructions:
-
Run the CLI to get the deterministic scorecard:
node dist/index.js tier-check --repo <repo> --conformance-server-url <url> --output json -
Evaluate documentation coverage — check whether MCP features (tools, resources, prompts, sampling, transports, etc.) are documented with examples. See
references/docs-coverage-prompt.mdfor the full checklist. -
Evaluate policies — check for dependency update policy, roadmap, and versioning/breaking-change policy. See
references/policy-evaluation-prompt.mdfor criteria. -
Apply tier logic — combine scorecard + evaluations against the thresholds in
references/tier-requirements.md. -
Generate report — use
references/report-template.mdfor the output format.
Manual Review
Run the CLI for the scorecard, then review docs and policies yourself using the tier requirements as a checklist:
| Requirement | Tier 1 | Tier 2 |
|---|---|---|
| Server Conformance | 100% pass | >= 80% pass |
| Client Conformance | 100% pass | >= 80% pass |
| Issue triage | Within 2 business days | Within 1 month |
| P0 resolution | Within 7 days | Within 2 weeks |
| Stable release | >= 1.0.0 with clear versioning | At least one >= 1.0.0 |
| Documentation | All features with examples | Core features documented |
| Dependency policy | Published | Published |
| Roadmap | Published with spec tracking | Plan toward Tier 1 |
Running Conformance Tests
To include conformance test results, start the SDK's everything server first, then pass the URL to the CLI. To also run client conformance tests, pass --client-cmd with the command to launch the SDK's conformance client.
TypeScript SDK:
# Terminal 1: start the server (SDK must be built first)
cd ~/src/mcp/typescript-sdk && npm run build
npm run test:conformance:server:run # starts on port 3000
# Terminal 2: run tier-check (server + client conformance)
npm run --silent tier-check -- \
--repo modelcontextprotocol/typescript-sdk \
--conformance-server-url http://localhost:3000/mcp \
--client-cmd 'npx tsx ~/src/mcp/typescript-sdk/test/conformance/src/everythingClient.ts'
Python SDK:
# Terminal 1: install and start the server
cd ~/src/mcp/python-sdk
uv sync --frozen --all-extras --package mcp-everything-server
uv run mcp-everything-server --port 3001 # specify port to avoid conflicts
# Terminal 2: run tier-check (server + client conformance)
npm run --silent tier-check -- \
--repo modelcontextprotocol/python-sdk \
--conformance-server-url http://localhost:3001/mcp \
--client-cmd 'uv run python ~/src/mcp/python-sdk/.github/actions/conformance/client.py'
Go SDK:
# Terminal 1: build and start the server
cd ~/src/mcp/go-sdk
go build -o /tmp/go-conformance-server ./conformance/everything-server
go build -o /tmp/go-conformance-client ./conformance/everything-client
/tmp/go-conformance-server -http="localhost:3002"
# Terminal 2: run tier-check (server + client conformance)
npm run --silent tier-check -- \
--repo modelcontextprotocol/go-sdk \
--conformance-server-url http://localhost:3002 \
--client-cmd '/tmp/go-conformance-client'
C# SDK:
# Terminal 1: start the server (requires .NET SDK)
cd ~/src/mcp/csharp-sdk
dotnet run --project tests/ModelContextProtocol.ConformanceServer --framework net9.0 -- --urls http://localhost:3003
# Terminal 2: run tier-check (server + client conformance)
npm run --silent tier-check -- \
--repo modelcontextprotocol/csharp-sdk \
--conformance-server-url http://localhost:3003 \
--client-cmd 'dotnet run --project ~/src/mcp/csharp-sdk/tests/ModelContextProtocol.ConformanceClient'
Other SDKs: Your SDK needs an "everything server" — an HTTP server implementing the Streamable HTTP transport with all MCP features (tools, resources, prompts, etc.). See the implementations above as reference.
Start your everything server, then pass --conformance-server-url. Pass --client-cmd if your SDK has a conformance client. If neither exists yet, use --skip-conformance — the scorecard will note this as a gap.
Reference Files
These files in references/ contain the detailed criteria and prompts:
| File | Purpose |
|---|---|
tier-requirements.md |
Full SEP-1730 requirements with exact thresholds |
docs-coverage-prompt.md |
Feature checklist for documentation evaluation |
policy-evaluation-prompt.md |
Criteria for dependency, roadmap, and versioning policy |
report-template.md |
Output format for the full audit report |