feat: SDK tier assessment CLI and skill (#142)

* feat: add tier-check CLI for SDK tier assessment

Adds a 'tier-check' subcommand to the conformance tool that automates
SDK tier assessment against SEP-1730 criteria.

Checks performed:
- Conformance test pass rate (via everything-server)
- GitHub label taxonomy (priority/status/area labels)
- Issue triage SLA compliance
- P0 bug resolution tracking
- Stable release detection
- Required file existence (CHANGELOG, SECURITY, etc.)
- Spec tracking (SDK release within 30d of spec release)

Also includes a Claude Code skill (skills/mcp-sdk-tier-audit/) for
judgment-based checks that require codebase analysis (feature coverage,
docs quality, policy evaluation).

Usage:
  npx tsx src/index.ts tier-check --repo modelcontextprotocol/typescript-sdk
  npx tsx src/index.ts tier-check --repo ... --conformance-server-cmd '...' \
    --conformance-server-cwd ... --conformance-server-url ... --output json

* refactor: revise tier-check CLI and skill based on review feedback

- Move skill to .claude/skills/ so it's auto-available in Claude Code
- Remove feature-coverage subagent (redundant with conformance tests)
- Remove hardcoded ~/src/mcp paths from all skill files
- Trim conformance server table to TS + Python only
- Rename file_existence check to policy_signals (informational, not blocking)
- Add GitHub native issue types detection to labels check
- Add missing features to docs-coverage checklist (tasks, elicitation URL mode, JSON Schema 2020-12)
- Add README with CLI quick start and escape hatch for non-Claude-Code users
- Use --limit 500 instead of --limit 100 for gh issue list

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address review feedback on skill and README

- Use npx @modelcontextprotocol/conformance instead of node dist/index.js
- Add full GitHub auth instructions (gh auth login, GITHUB_TOKEN, --token)
- Point TS SDK conformance server to typescript-sdk/test/conformance/
- Fix Python SDK URL to localhost:3001/mcp (not TBD)
- Remove manual gh issue list / gh release list from SKILL.md (CLI handles it)
- Remove Claude Code-specific subagent_type references
- Assume user is already in conformance repo
- Clean up policy-evaluation-prompt.md: remove redundant grep commands,
  focus on content evaluation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: resolve lint errors in conformance.ts

Remove unused variable assignments flagged by eslint.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style: apply prettier formatting

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: add npm run tier-check script, update docs with examples

- Add "tier-check" npm script so users can run `npm run tier-check --`
  instead of `node dist/index.js tier-check`
- Update SKILL.md, README, and skill README to use npm run tier-check
- Add full conformance examples with --conformance-server-cmd/cwd/url
  flags and realistic paths (~/src/mcp/typescript-sdk)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style: fix prettier formatting in SKILL.md

* fix: add per-scenario timeout, support url-only conformance, fix stdout pollution

- Add 30s per-scenario timeout to prevent tier-check from hanging
- Allow --conformance-server-url without --conformance-server-cmd (server already running)
- Move runner status logs to stderr so --output json produces clean JSON
- Update SKILL.md and README with --silent flag and pre-start server workflow

* refactor: skill takes local path + server URL instead of repo name

The skill now requires two arguments:
1. Local path to the SDK checkout (for direct file inspection)
2. URL where the everything server is already running

The GitHub owner/repo is derived from git remote. This eliminates
cloning, server startup complexity, and branch confusion (v1.x vs main).

* refactor: write detailed reports to files, show concise summary

Reports go to results/tier-audits/<sdk>-<date>/ (already gitignored).
Claude's console output is now just the tier classification,
pass/fail summary line, top 3 actions, and file paths.

* docs: update READMEs for new skill interface and pre-start workflow

* simplify: flat file output instead of nested directory

* fix: remediation always shows path to Tier 2 and Tier 1

* feat: add client conformance testing to tier-check CLI and skill

- Add checkClientConformance() that runs core client scenarios
  (initialize, tools_call, elicitation-defaults, sse-retry, auth)
  by spawning the SDK's conformance client via --client-cmd
- Add client_conformance to TierScorecard type
- Wire --client-cmd option into the CLI
- Update tier logic: both server + client conformance feed into
  Tier 1 (100%) and Tier 2 (>=80%) requirements
- Update terminal and markdown output to show both conformance types
- Update skill to auto-detect conformance client or accept explicit
  client-cmd argument
- Update README with new option and examples

* improve: table summary output, write reports via subagents

- Change executive summary from pipe-delimited line to a readable
  table with T2/T1 columns
- Move assessment and remediation file writing into parallel
  subagents to keep the main conversation thread clean

* improve: list tier gaps as numbered items instead of one-line blob

* improve: finalize summary format with separator, high-priority fixes, numbered gaps

* improve: add pre-flight checks for gh auth and server reachability

Fail fast with clear error messages if GitHub CLI is not authenticated
or if the conformance server URL is not reachable, rather than failing
deep into the scorecard run.

* docs: improve README and fix skill auto-detection paths

- Claude Code section: explain client-cmd auto-detection for TS/Python,
  show explicit 3-arg form for other SDKs, add examples for all three
- Fix TypeScript build command (npm run build, not pnpm build:all)
- Fix Python server command (add --port, use uv sync --package)
- Fix Python client path (.github/actions/conformance/client.py)
- Expand 'Other SDKs' section with guidance on everything server
- Add gh auth login prerequisite to Claude Code steps

* simplify: remove client-cmd auto-detection, require explicit argument

Client command is now always passed as the third argument. If omitted,
client conformance is skipped and noted as a gap. No more magic path
detection — clearer and more predictable.

* fix: align docs table with canonical list (48 features), simplify policy eval

Docs coverage:
- Table now has numbered rows matching all 48 non-experimental features
  from the canonical list (was missing 7: tools text/image/audio/embedded/
  error/notifications, protocol version negotiation)
- Hardcode total as 48 in summary so agents don't miscount

Policy evaluation:
- Simplified from deep content analysis to file-existence checks
- Dependency policy: DEPENDENCY_POLICY.md, dependabot.yml, or CONTRIBUTING.md section
- Roadmap: ROADMAP.md must exist (GitHub milestones alone not sufficient)
- Versioning: VERSIONING.md or CONTRIBUTING.md section
- Removed GitHub API calls for milestones and releases from policy eval

* refactor: extract canonical feature list into single source of truth

Create references/feature-list.md with all 48 non-experimental + 5
experimental features. The docs-coverage prompt now references this
file instead of duplicating the list. One place to update when
features change.

* fix: separate deterministic file checks from AI content evaluation

CLI (files.ts): now checks all policy files deterministically —
DEPENDENCY_POLICY.md, docs/dependency-policy.md, dependabot.yml,
renovate.json, ROADMAP.md, docs/roadmap.md, VERSIONING.md,
docs/versioning.md, BREAKING_CHANGES.md (in addition to existing
CHANGELOG.md, SECURITY.md, CONTRIBUTING.md).

AI policy eval: receives CLI output showing which files exist,
then reads ONLY those files to judge content quality. No longer
searches the repo for files — clean separation of concerns.

* style: apply prettier formatting

* revert: undo unrelated console.log change in runner/server.ts

* refactor: shell out to conformance CLI instead of reimplementing runner

Address PR feedback: conformance.ts was duplicating the normal conformance
running code. Now shells out to 'node dist/index.js server/client' with
-o to save results to a temp dir, then parses the checks.json files.

Also removes --conformance-server-cmd and --conformance-server-cwd options
since the server must be pre-started.

* docs: add Go and C# SDK examples to README and SKILL.md

* fix: add --framework net9.0 to C# server command

* rename: conformance.ts -> test-conformance-results.ts

Avoids confusion with src/runner/ (the actual conformance runner).
This file just invokes the CLI and parses output.

* style: prettier formatting

* fix: reconcile conformance results against full scenario list

The tier-check CLI was only counting scenarios that produced a
checks.json file. Scenarios that crashed or failed to run (e.g., auth
scenarios when OAuth is not implemented) were invisible, making the
denominator artificially small (e.g., 4/4 instead of 4/23).

Now both checkConformance and checkClientConformance reconcile their
parsed results against the known scenario lists, adding failure entries
for any expected scenario that didn't produce results.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: tighten documentation evaluation criteria

Clarify what counts as documented vs just having code:
- Conformance test servers don't count as docs or examples
- Examples without prose = PARTIAL, not PASS
- Go Example* test functions explicitly allowed
- Clear PASS/PARTIAL/FAIL verdict definitions

* docs: add Labels and Spec Tracking rows to audit report templates

The executive summary and assessment report were missing two SEP-1730
requirements: label taxonomy compliance and spec tracking (new protocol
features timeline).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: reuse ConformanceCheck type from src/types.ts instead of redefining

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Felix Weinberger 2026-02-12 17:59:06 +00:00 • committed by GitHub
commit 3b4a92baad
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

View file

@ -0,0 +1,276 @@
---
name: mcp-sdk-tier-audit
description: >-
Comprehensive tier assessment for an MCP SDK repository against SEP-1730.
Produces tier classification (1/2/3) with evidence table, gap list, and
remediation guide. Works for any official MCP SDK (TypeScript, Python, Go,
C#, Java, Kotlin, PHP, Swift, Rust, Ruby).
argument-hint: '<local-path> <conformance-server-url> [client-cmd]'
---
# MCP SDK Tier Audit
You are performing a comprehensive tier assessment for an MCP SDK repository against SEP-1730 (the SDK Tiering System). Your goal is to produce a definitive tier classification (Tier 1, 2, or 3) backed by evidence.
## Step 0: Pre-flight Checks
Before doing anything else, verify GitHub CLI authentication:
```bash
gh auth status 2>&1
```
If this fails (exit code non-zero or shows "not logged in"), stop immediately and tell the user:
> GitHub authentication is required for this skill. Please run `gh auth login` first, then re-run the skill.
Do NOT proceed to any other step if this check fails.
After parsing arguments (Step 1), also verify the conformance server is reachable:
```bash
curl -sf <conformance-server-url> -o /dev/null -w '%{http_code}' 2>&1 || true
```
If the server is not reachable, stop and tell the user:
> Conformance server at `<url>` is not reachable. Make sure the everything server is running before invoking this skill.
## Step 1: Parse Arguments
Extract from the user's input:
- **local-path**: absolute path to the SDK checkout (e.g. `~/src/mcp/typescript-sdk`)
- **conformance-server-url**: URL where the SDK's everything server is already running (e.g. `http://localhost:3000/mcp`)
- **client-cmd** (optional): command to run the SDK's conformance client (e.g. `npx tsx test/conformance/src/everythingClient.ts`). If not provided, client conformance tests are skipped and noted as a gap in the report.
The first two arguments are required. If either is missing, ask the user to provide it.
Derive the GitHub `owner/repo` from the local checkout:
```bash
cd <local-path> && git remote get-url origin | sed 's#.*github.com[:/]##; s#\.git$##'
```
## Step 2: Run the Deterministic Scorecard
The `tier-check` CLI handles all deterministic checks — server conformance, client conformance, labels, triage, P0 resolution, releases, policy signals, and spec tracking. You are already in the conformance repo, so run it directly.
```bash
npm run --silent tier-check -- \
--repo <owner/repo> \
--conformance-server-url <conformance-server-url> \
--client-cmd '<client-cmd>' \
--output json
```
If no client-cmd was detected, omit the `--client-cmd` flag (client conformance will be skipped).
The CLI output includes server conformance pass rate, client conformance pass rate, issue triage compliance, P0 resolution times, label taxonomy, stable release status, policy signal files, and spec tracking gap. Parse the JSON output to feed into Step 4.
### Conformance Baseline Check
After running the CLI, check for an expected-failures baseline file in the SDK repo:
```bash
find <local-path> -name "baseline.yml" -o -name "expected-failures.yml" 2>/dev/null | head -5
```
If found, read the file. It lists known/expected conformance failures. This context is essential for interpreting raw pass rates — a 20% client pass rate due entirely to unimplemented OAuth scenarios is very different from 20% due to broken core functionality.
## Step 3: Launch Parallel Evaluations
Launch 2 evaluations in parallel. Each reads the SDK from the local checkout path.
**IMPORTANT**: Launch both evaluations at the same time (in the same response) so they run in parallel.
### Evaluation 1: Documentation Coverage
Use the prompt from `references/docs-coverage-prompt.md`. Pass the local path.
This evaluation checks:
- Whether all non-experimental features are documented with examples (Tier 1 requirement)
- Whether core features are documented (Tier 2 requirement)
- Produces an evidence table with file:line references
### Evaluation 2: Policy Evaluation
Use the prompt from `references/policy-evaluation-prompt.md`. Pass the local path, the derived `owner/repo`, and the `policy_signals` section from the CLI JSON output.
The CLI has already checked which policy files exist (ROADMAP.md, DEPENDENCY_POLICY.md, dependabot.yml, VERSIONING.md, etc.). The AI evaluation reads only the files the CLI found to judge whether the content is substantive — it does NOT search for files in other locations.
This evaluation checks:
- Dependency update policy (required for Tier 1 and Tier 2)
- Published roadmap (required for Tier 1; plan-toward-Tier-1 for Tier 2)
- Clear versioning with documented breaking change policy (required for Tier 1)
- Produces evidence tables for each policy area
## Step 4: Compute Final Tier
Combine the deterministic scorecard (from the CLI) with the evaluation results (docs, policies). Apply the tier logic:
### Tier 1 requires ALL of:
- Server conformance test pass rate == 100%
- Client conformance test pass rate == 100%
- Issue triage compliance >= 90% within 2 business days
- All P0 bugs resolved within 7 days
- Stable release >= 1.0.0 with no pre-release suffix
- Clear versioning with documented breaking change policy (evaluation)
- All non-experimental features documented with examples (evaluation)
- Published dependency update policy (evaluation)
- Published roadmap with concrete steps tracking spec components (evaluation)
### Tier 2 requires ALL of:
- Server conformance test pass rate >= 80%
- Client conformance test pass rate >= 80%
- Issue triage compliance >= 80% within 1 month
- P0 bugs resolved within 2 weeks
- At least one stable release >= 1.0.0
- Basic docs covering core features (evaluation)
- Published dependency update policy (evaluation)
- Published plan toward Tier 1 or explanation for remaining Tier 2 (evaluation)
### Otherwise: Tier 3
If any Tier 2 requirement is not met, the SDK is Tier 3.
**Important edge cases:**
- If GitHub issue labels are not set up per SEP-1730, triage metrics cannot be computed. Note this as a gap. However, repos may use GitHub's native issue types instead of type labels — the CLI checks for both.
- If client conformance was skipped (no client command found), note this as a gap but do not block tier advancement based on it alone.
**Client Conformance Splits:**
When reporting client conformance, always break results into three categories:
1. **Core suite** — Non-auth scenarios (e.g. initialize, tools_call, elicitation, sse-retry)
2. **Auth suite** — OAuth/authorization scenarios (any scenario starting with `auth/`)
3. **Full suite** — All scenarios combined
The **full suite** number is used for tier threshold checks. However, the core vs auth split provides essential context. Always present both numbers in the report.
If the SDK has a `baseline.yml` or expected-failures file, note which failures are known/tracked vs. unexpected regressions. A low full-suite score where all failures are auth scenarios documented in the baseline is a scope gap (OAuth not yet implemented), not a quality problem — flag it accordingly in the assessment.
**P0 Label Audit Guidance:**
When evaluating P0 metrics, flag potentially mislabeled P0 issues:
- If P0 count is high (>2) but other Tier 2 metrics (conformance, triage compliance, docs) are strong, this may indicate P0 labels are being used for enhancements, lower-priority work, or feature requests rather than actual critical bugs.
- In such cases, recommend a P0 label audit as a remediation action. Review open P0 issues to verify they represent genuine blocking defects vs. misclassified work.
- Document this finding in the remediation output with specific issue numbers and suggested re-triage actions.
- Do not treat high P0 count as an automatic hard blocker if the audit reveals mislabeling; instead, note it as a process improvement opportunity.
## Step 5: Generate Output
Write detailed reports to files using subagents, then show a concise summary to the user.
### Output files (write via subagents)
**IMPORTANT**: Write both report files using parallel subagents (Task tool) so the file-writing work does not pollute the main conversation thread. Launch both subagents at the same time.
Write two files to `results/` in the conformance repo:
- `results/<YYYY-MM-DD>-<sdk-name>-assessment.md`
- `results/<YYYY-MM-DD>-<sdk-name>-remediation.md`
For example: `results/2026-02-11-typescript-sdk-assessment.md`
#### Assessment subagent
Pass all the gathered data (CLI scorecard JSON, docs evaluation results, policy evaluation results) to a subagent and instruct it to write the assessment file using the template from `references/report-template.md`. This file contains the full requirements table, conformance test details (both server and client), triage metrics, documentation coverage table, and policy evaluation evidence.
#### Remediation subagent
Pass all the gathered data to a subagent and instruct it to write the remediation file using the template from `references/report-template.md`. This file always includes both:
- **Path to Tier 2** (if current tier is 3) -- what's needed to reach Tier 2
- **Path to Tier 1** (always) -- what's needed to reach Tier 1
### Console output (shown to the user)
After the subagents finish, output a short executive summary directly to the user:
```
## <sdk-name> — Tier <X>
| Check | Value | T2 | T1 |
|-------|-------|----|----|
| Server Conformance | <passed>/<total> (<rate>%) | ✓/✗ | ✓/✗ |
| Client Conformance (full) | <passed>/<total> (<rate>%) | ✓/✗ | ✓/✗ |
| — Core scenarios | <core_pass>/<core_total> (<rate>%) | — | — |
| — Auth scenarios | <auth_pass>/<auth_total> (<rate>%) | — | — |
| Issue Triage | <rate>% (<triaged>/<total>) | ✓/✗ | ✓/✗ |
| Labels | <present>/<required> | ✓/✗ | ✓/✗ |
| P0 Resolution | <count> open | ✓/✗ | ✓/✗ |
| Spec Tracking | <days>d gap | ✓/✗ | ✓/✗ |
| Documentation | <pass>/<total> features | ✓/✗ | ✓/✗ |
| Dependency Policy | <summary> | ✓/✗ | ✓/✗ |
| Roadmap | <summary> | ✓/✗ | ✓/✗ |
| Versioning Policy | <summary> | N/A | ✓/✗ |
| Stable Release | <version> | ✓/✗ | ✓/✗ |
If a baseline file was found, add a note below the table:
> **Baseline**: {N} failures in `baseline.yml` ({list of categories, e.g. "18 auth scenarios"}). Core suite: {core_rate}%.
---
**High-Priority Fixes:**
1. <fix description>
**For Tier 2:**
1. <gap description>
2. <gap description>
**For Tier 1:**
1. <gap description>
2. <gap description>
Reports:
- results/<date>-<sdk-name>-assessment.md
- results/<date>-<sdk-name>-remediation.md
```
Use ✓ for pass and ✗ for fail.
**High-Priority Fixes**: List any issues that need urgent attention (e.g., P0 label audit if P0 count is >2 but other metrics are strong, suggesting mislabeled issues). If none, omit this section.
**For Tier 2 / For Tier 1**: List each gap as a separate numbered item. Use "All requirements met" if there are no gaps for that tier. Each item should be a concise action (e.g., "Re-triage mislabeled P0s", "Document 16 undocumented core features").
## Reference Files
The following reference files are available in the `references/` directory alongside this skill:
- `references/feature-list.md` -- Canonical list of 48 non-experimental + 5 experimental features (single source of truth)
- `references/tier-requirements.md` -- Full SEP-1730 requirements table with exact thresholds
- `references/report-template.md` -- Output format template for the audit report
- `references/docs-coverage-prompt.md` -- Evaluation prompt for documentation coverage
- `references/policy-evaluation-prompt.md` -- Evaluation prompt for policy review
Read these reference files when you need the detailed content for evaluation prompts or report formatting.
## Usage Examples
```
# TypeScript SDK — server + client conformance
/mcp-sdk-tier-audit ~/src/mcp/typescript-sdk http://localhost:3000/mcp "npx tsx ~/src/mcp/typescript-sdk/test/conformance/src/everythingClient.ts"
# Python SDK — server + client conformance
/mcp-sdk-tier-audit ~/src/mcp/python-sdk http://localhost:3001/mcp "uv run python ~/src/mcp/python-sdk/.github/actions/conformance/client.py"
# Go SDK — server + client conformance
/mcp-sdk-tier-audit ~/src/mcp/go-sdk http://localhost:3002 "/tmp/go-conformance-client"
# C# SDK — server + client conformance
# Two C#-specific requirements in the client-cmd:
# --framework net9.0 : required because the project targets net8.0/net9.0/net10.0
# -- $MCP_CONFORMANCE_SCENARIO : the runner sets this env var and uses shell:true, so the
# shell expands it; dotnet passes [scenario, url] to the program
/mcp-sdk-tier-audit ~/src/mcp/csharp-sdk http://localhost:3003 "dotnet run --project ~/src/mcp/csharp-sdk/tests/ModelContextProtocol.ConformanceClient --framework net9.0 -- $MCP_CONFORMANCE_SCENARIO"
# Any SDK — server conformance only (no client)
/mcp-sdk-tier-audit ~/src/mcp/some-sdk http://localhost:3004
```