Compare commits
289 changed files with 6985 additions and 80372 deletions
|
|
@ -1,195 +0,0 @@
|
|||
# MCP SDK Tier Audit
|
||||
|
||||
Assess any MCP SDK repository against [SEP-1730](https://github.com/modelcontextprotocol/modelcontextprotocol/issues/1730) (the SDK Tiering System). Produces a tier classification (1/2/3) with an evidence-backed scorecard.
|
||||
|
||||
Two components work together:
|
||||
|
||||
- **`tier-check` CLI** — runs deterministic checks (server + client conformance pass rate, issue triage speed, P0 resolution, labels, releases, policy signals). Works standalone, no AI needed.
|
||||
- **AI-assisted assessment** — an agent uses the CLI scorecard plus judgment-based evaluation (documentation coverage, dependency policy, roadmap) to produce a full tier report with remediation guide.
|
||||
|
||||
## Quick Start: CLI
|
||||
|
||||
For an SDK listed in `src/sdk-runner/known-sdks.ts`, tier-check manages the whole conformance side itself — clone/build, then for each shipped revision the server invocation that SDK declares for that revision:
|
||||
|
||||
```bash
|
||||
gh auth login
|
||||
npx @modelcontextprotocol/conformance tier-check --sdk go-sdk
|
||||
```
|
||||
|
||||
The per-SDK sections below are for the manual path: driving a server you started yourself.
|
||||
|
||||
The CLI is a subcommand of the [MCP Conformance](https://github.com/modelcontextprotocol/conformance) tool.
|
||||
|
||||
```bash
|
||||
# Clone and build
|
||||
git clone https://github.com/modelcontextprotocol/conformance.git
|
||||
cd conformance
|
||||
npm install
|
||||
npm run build
|
||||
|
||||
# Authenticate with GitHub (needed for API access)
|
||||
gh auth login
|
||||
|
||||
# Run against any MCP SDK repo (without conformance tests)
|
||||
npm run --silent tier-check -- --repo modelcontextprotocol/typescript-sdk --skip-conformance
|
||||
```
|
||||
|
||||
The CLI uses the GitHub API (read-only) for issue metrics, labels, and release checks. Authenticate via one of:
|
||||
|
||||
- **GitHub CLI** (recommended): `gh auth login` — the CLI picks up your token automatically
|
||||
- **Environment variable**: `export GITHUB_TOKEN=ghp_...`
|
||||
- **Flag**: `--token ghp_...`
|
||||
|
||||
For public repos, any authenticated token works (no special scopes needed — authentication just avoids rate limits). For a [fine-grained personal access token](https://github.com/settings/personal-access-tokens/new), select **Public Repositories (read-only)** with no additional permissions.
|
||||
|
||||
### CLI Options
|
||||
|
||||
```
|
||||
--repo <owner/repo> GitHub repository (required)
|
||||
--branch <branch> Branch to check
|
||||
--skip-conformance Skip conformance tests
|
||||
--conformance-server-url <url> URL of the already-running conformance server
|
||||
--client-cmd <cmd> Command to run the SDK conformance client (for client conformance tests)
|
||||
--days <n> Limit triage analysis to last N days
|
||||
--output <format> json | markdown | terminal (default: terminal)
|
||||
--token <token> GitHub token (defaults to GITHUB_TOKEN or gh auth token)
|
||||
```
|
||||
|
||||
### What the CLI Checks
|
||||
|
||||
| Check | What it measures |
|
||||
| ------------------ | ------------------------------------------------------------------------------ |
|
||||
| Server Conformance | Pass rate of server implementation against the conformance test suite |
|
||||
| Client Conformance | Pass rate of client implementation against the conformance test suite |
|
||||
| Labels | Whether SEP-1730 label taxonomy is set up (supports GitHub native issue types) |
|
||||
| Triage | How quickly issues get labeled after creation |
|
||||
| P0 Resolution | Whether critical bugs are resolved within SLA |
|
||||
| Stable Release | Whether a stable release >= 1.0.0 exists |
|
||||
| Policy Signals | Presence of CHANGELOG, SECURITY, CONTRIBUTING, dependabot, ROADMAP |
|
||||
| Spec Tracking | Gap between latest spec release and SDK release |
|
||||
|
||||
### Example Output
|
||||
|
||||
```
|
||||
Tier Assessment: Tier 2
|
||||
|
||||
Repo: modelcontextprotocol/typescript-sdk
|
||||
Timestamp: 2026-02-10T12:00:00Z
|
||||
|
||||
Check Results:
|
||||
|
||||
✓ Server Conformance 45/45 (100%)
|
||||
✓ Client Conformance 4/4 (100%)
|
||||
✗ Labels 9/12 required labels
|
||||
Missing: needs confirmation, needs repro, ready for work
|
||||
✓ Triage 92% within 2BD (150 issues, median 8h)
|
||||
✓ P0 Resolution 0 open, 3/3 closed within 7d
|
||||
✓ Stable Release 2.3.1
|
||||
~ Policy Signals ✓ CHANGELOG.md, ✗ SECURITY.md, ✓ CONTRIBUTING.md, ✓ .github/dependabot.yml, ✗ ROADMAP.md
|
||||
✓ Spec Tracking 2d gap
|
||||
```
|
||||
|
||||
Use `--output json` to get machine-readable results, or `--output markdown` for a report you can paste into an issue.
|
||||
|
||||
## Full AI-Assisted Assessment
|
||||
|
||||
The CLI produces a deterministic scorecard, but some SEP-1730 requirements need judgment: documentation quality, dependency policy, roadmap substance. An AI agent can evaluate these by reading the repo.
|
||||
|
||||
### Claude Code
|
||||
|
||||
The skill lives in `.claude/skills/` in this repo, so if you open [Claude Code](https://docs.anthropic.com/en/docs/claude-code) in the conformance repo it's already available.
|
||||
|
||||
1. Make sure `gh auth login` is done (the skill checks this upfront)
|
||||
2. Start the SDK's everything server in a separate terminal
|
||||
3. Run the skill:
|
||||
|
||||
```
|
||||
/mcp-sdk-tier-audit <local-sdk-path> <conformance-server-url> [client-cmd] [--requirements <revision>]
|
||||
```
|
||||
|
||||
Pass the client command as the third argument to include client conformance testing. If omitted, client conformance is skipped and noted as a gap in the report.
|
||||
|
||||
**Pass `--requirements` with every revision the SDK claims**, comma-separated. Each revision's scenarios run at that revision's own wire version, and all of them must pass for Tier 1: the dated revisions through `2025-11-25` use the stateful initialize handshake while `2026-07-28` is stateless, so a scenario belonging to both has to work on both and one run does not cover the other. It also means a scenario added to the suite after a revision shipped cannot fail an SDK that had no opportunity to adopt it. Without the flag, scoring uses the suite as it stands today, which is not a tier claim. See [Conformance Requirements](../../../README.md#conformance-requirements), and run `conformance list --requirements 2025-11-25,2026-07-28` to see both sets.
|
||||
|
||||
**Known SDKs — no server to start, no commands to look up:**
|
||||
|
||||
```
|
||||
/mcp-sdk-tier-audit <sdk checkout> --requirements 2025-11-25,2026-07-28
|
||||
```
|
||||
|
||||
The skill drives `tier-check --sdk-path`, which builds the SDK and starts the
|
||||
server invocation its config declares **per revision** — the per-SDK commands
|
||||
live in [`src/sdk-runner/known-sdks.ts`](../../../src/sdk-runner/known-sdks.ts),
|
||||
not here. That matters because a hand-run example cannot be correct for every
|
||||
SDK: go-sdk needs a different server process per revision (`-stateless` defaults
|
||||
to true, so a bare invocation mis-measures 2025-11-25 — [#446](https://github.com/modelcontextprotocol/conformance/issues/446)),
|
||||
and csharp-sdk serves the two revisions at different endpoints. Prose copies of
|
||||
those invocations drift; the config is tested.
|
||||
|
||||
**An SDK not in `known-sdks.ts`:** start its everything server yourself and pass
|
||||
the URL and client command explicitly — see [Other SDKs](#running-conformance-tests)
|
||||
below. The one endpoint you give must serve every claimed revision at its own
|
||||
wire, or the numbers will be wrong for the revisions it does not speak.
|
||||
|
||||
|
||||
### Any Other AI Coding Agent
|
||||
|
||||
If you use a different agent (Codex, Cursor, Aider, OpenCode, etc.), give it these instructions:
|
||||
|
||||
1. **Run the CLI** to get the deterministic scorecard:
|
||||
|
||||
```bash
|
||||
node dist/index.js tier-check --repo <repo> --conformance-server-url <url> --output json
|
||||
```
|
||||
|
||||
2. **Evaluate documentation coverage** — check whether MCP features (tools, resources, prompts, sampling, transports, etc.) are documented with examples. See [`references/docs-coverage-prompt.md`](references/docs-coverage-prompt.md) for the full checklist.
|
||||
|
||||
3. **Evaluate policies** — check for dependency update policy, roadmap, and versioning/breaking-change policy. See [`references/policy-evaluation-prompt.md`](references/policy-evaluation-prompt.md) for criteria.
|
||||
|
||||
4. **Apply tier logic** — combine scorecard + evaluations against the thresholds in [`references/tier-requirements.md`](references/tier-requirements.md).
|
||||
|
||||
5. **Generate report** — use [`references/report-template.md`](references/report-template.md) for the output format.
|
||||
|
||||
### Manual Review
|
||||
|
||||
Run the CLI for the scorecard, then review docs and policies yourself using the tier requirements as a checklist:
|
||||
|
||||
| Requirement | Tier 1 | Tier 2 |
|
||||
| ------------------ | ------------------------------ | ------------------------ |
|
||||
| Server Conformance | 100% pass | >= 80% pass |
|
||||
| Client Conformance | 100% pass | >= 80% pass |
|
||||
| Issue triage | Within 2 business days | Within 1 month |
|
||||
| P0 resolution | Within 7 days | Within 2 weeks |
|
||||
| Stable release | >= 1.0.0 with clear versioning | At least one >= 1.0.0 |
|
||||
| Documentation | All features with examples | Core features documented |
|
||||
| Dependency policy | Published | Published |
|
||||
| Roadmap | Published with spec tracking | Plan toward Tier 1 |
|
||||
|
||||
## Running Conformance Tests
|
||||
|
||||
For any SDK in [`src/sdk-runner/known-sdks.ts`](../../../src/sdk-runner/known-sdks.ts), don't start servers by hand — the config runs the right invocation per revision:
|
||||
|
||||
```bash
|
||||
# both legs, every revision, one verdict
|
||||
npx @modelcontextprotocol/conformance tier-check --sdk go-sdk
|
||||
|
||||
# a single leg at a single revision, for debugging
|
||||
npx @modelcontextprotocol/conformance sdk --path <sdk checkout> --mode server --requirements 2026-07-28
|
||||
```
|
||||
|
||||
To reproduce one leg fully by hand, copy the build/server/client commands from that SDK's `known-sdks.ts` entry (including its `specOverrides` for the revision you're running) rather than from a README: the entries are exercised by the test suite and per-revision, prose examples are neither.
|
||||
|
||||
**Other SDKs:** Your SDK needs an "everything server" — an HTTP server implementing the [Streamable HTTP transport](https://modelcontextprotocol.io/specification/draft/basic/transports.md) with all MCP features (tools, resources, prompts, etc.). See the implementations above as reference.
|
||||
|
||||
Start your everything server, then pass `--conformance-server-url`. Pass `--client-cmd` if your SDK has a conformance client. If neither exists yet, use `--skip-conformance` — the scorecard will note this as a gap.
|
||||
|
||||
## Reference Files
|
||||
|
||||
These files in [`references/`](references/) contain the detailed criteria and prompts:
|
||||
|
||||
| File | Purpose |
|
||||
| ----------------------------- | ------------------------------------------------------- |
|
||||
| `tier-requirements.md` | Full SEP-1730 requirements with exact thresholds |
|
||||
| `docs-coverage-prompt.md` | Feature checklist for documentation evaluation |
|
||||
| `policy-evaluation-prompt.md` | Criteria for dependency, roadmap, and versioning policy |
|
||||
| `report-template.md` | Output format for the full audit report |
|
||||
|
|
@ -1,330 +0,0 @@
|
|||
---
|
||||
name: mcp-sdk-tier-audit
|
||||
description: >-
|
||||
Comprehensive tier assessment for an MCP SDK repository against SEP-1730.
|
||||
Produces tier classification (1/2/3) with evidence table, gap list, and
|
||||
remediation guide. Works for any official MCP SDK (TypeScript, Python, Go,
|
||||
C#, Java, Kotlin, PHP, Swift, Rust, Ruby).
|
||||
argument-hint: '<local-path> [conformance-server-url] [client-cmd] [--requirements <revisions>] [--branch <branch>]'
|
||||
---
|
||||
|
||||
# MCP SDK Tier Audit
|
||||
|
||||
You are performing a comprehensive tier assessment for an MCP SDK repository against SEP-1730 (the SDK Tiering System). Your goal is to produce a definitive tier classification (Tier 1, 2, or 3) backed by evidence.
|
||||
|
||||
## Step 0: Pre-flight Checks
|
||||
|
||||
Before doing anything else, verify GitHub CLI authentication:
|
||||
|
||||
```bash
|
||||
gh auth status 2>&1
|
||||
```
|
||||
|
||||
If this fails (exit code non-zero or shows "not logged in"), stop immediately and tell the user:
|
||||
|
||||
> GitHub authentication is required for this skill. Please run `gh auth login` first, then re-run the skill.
|
||||
|
||||
Do NOT proceed to any other step if this check fails.
|
||||
|
||||
**Known SDKs need no server from you.** When the SDK is in `src/sdk-runner/known-sdks.ts` (typescript-sdk, python-sdk, go-sdk, rust-sdk, csharp-sdk, ruby-sdk, and variants), skip the server preflight entirely: Step 2's `--sdk-path` mode builds the SDK and starts the right server per revision itself. The reachability check below applies only to the URL fallback for unknown SDKs.
|
||||
|
||||
After parsing arguments (Step 1), for the URL fallback only, verify the conformance server is reachable:
|
||||
|
||||
```bash
|
||||
curl -sf <conformance-server-url> -o /dev/null -w '%{http_code}' 2>&1 || true
|
||||
```
|
||||
|
||||
If the server is not reachable, stop and tell the user:
|
||||
|
||||
> Conformance server at `<url>` is not reachable. Make sure the everything server is running before invoking this skill.
|
||||
|
||||
## Step 1: Parse Arguments
|
||||
|
||||
Extract from the user's input:
|
||||
|
||||
- **local-path**: absolute path to the SDK checkout (e.g. `~/src/mcp/typescript-sdk`)
|
||||
- **conformance-server-url** (only for SDKs not in `known-sdks.ts`): URL where the SDK's everything server is already running (e.g. `http://localhost:3000/mcp`). That one endpoint must serve every claimed revision at its own wire version; SDKs that vary the server per revision cannot be driven this way
|
||||
- **client-cmd** (optional): command to run the SDK's conformance client (e.g. `npx tsx test/conformance/src/everythingClient.ts`). If not provided, client conformance tests are skipped and noted as a gap in the report.
|
||||
- **requirements** (optional): spec revisions to score against, comma-separated, e.g. `--requirements 2025-11-25,2026-07-28`. Each revision's scenarios run at that revision's wire version, and every listed revision must pass for Tier 1. Scores the SDK against exactly the scenarios that revision required when it shipped, rather than everything the suite carries today. Prefer it whenever the question is "does this SDK conform to revision X". Without it, scoring uses today's suite, which can fail an SDK for a scenario added after it shipped. Run `conformance list --requirements <revision>` to see the set.
|
||||
- **branch** (optional): Git branch to check on GitHub (e.g. `--branch fweinberger/v1x-governance-docs`). If not provided, derive from the local checkout's current branch: `cd <local-path> && git rev-parse --abbrev-ref HEAD`. This is passed to the tier-check CLI so that policy signal file checks use the correct branch instead of the repo's default branch.
|
||||
|
||||
The first two arguments are required. If either is missing, ask the user to provide it.
|
||||
|
||||
Derive the GitHub `owner/repo` from the local checkout:
|
||||
|
||||
```bash
|
||||
cd <local-path> && git remote get-url origin | sed 's#.*github.com[:/]##; s#\.git$##'
|
||||
```
|
||||
|
||||
## Step 2: Run the Deterministic Scorecard
|
||||
|
||||
The `tier-check` CLI handles all deterministic checks — server conformance, client conformance, labels, triage, P0 resolution, releases, policy signals, and spec tracking. You are already in the conformance repo, so run it directly.
|
||||
|
||||
For a known SDK, this manages everything — build, per-revision server, both legs:
|
||||
|
||||
```bash
|
||||
npm run --silent tier-check -- \
|
||||
--sdk-path <local-path> \
|
||||
--requirements <revisions> \
|
||||
--output json
|
||||
```
|
||||
|
||||
`--repo` and `--branch` are derived from the SDK config and checkout; pass them only to override. For an SDK not in `known-sdks.ts`, fall back to driving it directly:
|
||||
|
||||
```bash
|
||||
npm run --silent tier-check -- \
|
||||
--repo <owner/repo> \
|
||||
--branch <branch> \
|
||||
--requirements <revisions> \
|
||||
--conformance-server-url <conformance-server-url> \
|
||||
--client-cmd '<client-cmd>' \
|
||||
--output json
|
||||
```
|
||||
|
||||
Omit `--requirements` only if the user did not name a revision. When it is set the
|
||||
scorecard reports `requirements_revision`, both pass rates count exactly the
|
||||
scenarios that revision requires, and anything run but not scored carries a
|
||||
`notScoredReason` of `extension` or `added-after-release`. `requirements_revisions`
|
||||
lists every revision scored, and each detail carries the `revision` it came from. Quote the revision
|
||||
alongside any conformance number, and report the not-scored failures separately
|
||||
rather than folding them into the score or omitting them.
|
||||
|
||||
If no client-cmd was detected, omit the `--client-cmd` flag (client conformance will be skipped). The `--branch` flag should always be included (derived from the local checkout if not explicitly provided).
|
||||
|
||||
The CLI output includes server conformance pass rate, client conformance pass rate (with per-spec-version breakdown), issue triage compliance, P0 resolution times, label taxonomy, stable release status, policy signal files, and spec tracking gap. Parse the JSON output to feed into Step 4.
|
||||
|
||||
The conformance results now include a `specVersions` field on each detail entry, enabling per-version pass rate analysis. The `list` command also shows spec version tags: `node dist/index.js list` shows `[2025-06-18]`, `[2025-11-25]`, `[draft]`, or `[extension]` next to each scenario.
|
||||
|
||||
### Conformance Baseline Check
|
||||
|
||||
After running the CLI, check for an expected-failures baseline file in the SDK repo:
|
||||
|
||||
```bash
|
||||
find <local-path> -name "baseline.yml" -o -name "expected-failures.yml" 2>/dev/null | head -5
|
||||
```
|
||||
|
||||
If found, read the file. It lists known/expected conformance failures. This context is essential for interpreting raw pass rates — a 20% client pass rate due entirely to unimplemented OAuth scenarios is very different from 20% due to broken core functionality.
|
||||
|
||||
## Step 3: Launch Parallel Evaluations
|
||||
|
||||
Launch 2 evaluations in parallel. Each reads the SDK from the local checkout path.
|
||||
|
||||
**IMPORTANT**: Launch both evaluations at the same time (in the same response) so they run in parallel.
|
||||
|
||||
### Evaluation 1: Documentation Coverage
|
||||
|
||||
Use the prompt from `references/docs-coverage-prompt.md`. Pass the local path.
|
||||
|
||||
This evaluation checks:
|
||||
|
||||
- Whether all non-experimental features are documented with examples (Tier 1 requirement)
|
||||
- Whether core features are documented (Tier 2 requirement)
|
||||
- Produces an evidence table with file:line references
|
||||
|
||||
### Evaluation 2: Policy Evaluation
|
||||
|
||||
Use the prompt from `references/policy-evaluation-prompt.md`. Pass the local path, the derived `owner/repo`, and the `policy_signals` section from the CLI JSON output.
|
||||
|
||||
The CLI has already checked which policy files exist (ROADMAP.md, DEPENDENCY_POLICY.md, dependabot.yml, VERSIONING.md, etc.). The AI evaluation reads only the files the CLI found to judge whether the content is substantive — it does NOT search for files in other locations.
|
||||
|
||||
This evaluation checks:
|
||||
|
||||
- Dependency update policy (required for Tier 1 and Tier 2)
|
||||
- Published roadmap (required for Tier 1; plan-toward-Tier-1 for Tier 2)
|
||||
- Clear versioning with documented breaking change policy (required for Tier 1)
|
||||
- Produces evidence tables for each policy area
|
||||
|
||||
## Step 4: Compute Final Tier
|
||||
|
||||
Combine the deterministic scorecard (from the CLI) with the evaluation results (docs, policies). Apply the tier logic:
|
||||
|
||||
### Tier 1 requires ALL of:
|
||||
|
||||
- Server conformance test pass rate == 100% across every requirement set given to `--requirements` (each run at its own wire), otherwise of date-versioned scenarios only
|
||||
- Client conformance test pass rate == 100%, on the same basis
|
||||
- Issue triage compliance >= 90% within 2 business days
|
||||
- All P0 bugs resolved within 7 days
|
||||
- Stable release >= 1.0.0 with no pre-release suffix
|
||||
- Clear versioning with documented breaking change policy (evaluation)
|
||||
- All non-experimental features documented with examples (evaluation)
|
||||
- Published dependency update policy (evaluation)
|
||||
- Published roadmap with concrete steps tracking spec components (evaluation)
|
||||
|
||||
### Tier 2 requires ALL of:
|
||||
|
||||
- Server conformance test pass rate >= 80% (date-versioned scenarios only)
|
||||
- Client conformance test pass rate >= 80% (date-versioned scenarios only)
|
||||
- Issue triage compliance >= 80% within 1 month
|
||||
- P0 bugs resolved within 2 weeks
|
||||
- At least one stable release >= 1.0.0
|
||||
- Basic docs covering core features (evaluation)
|
||||
- Published dependency update policy (evaluation)
|
||||
- Published plan toward Tier 1 or explanation for remaining Tier 2 (evaluation)
|
||||
|
||||
### Otherwise: Tier 3
|
||||
|
||||
If any Tier 2 requirement is not met, the SDK is Tier 3.
|
||||
|
||||
**Important edge cases:**
|
||||
|
||||
- If GitHub issue labels are not set up per SEP-1730, triage metrics cannot be computed. Note this as a gap. However, repos may use GitHub's native issue types instead of type labels — the CLI checks for both.
|
||||
- If client conformance was skipped (no client command found), note this as a gap but do not block tier advancement based on it alone.
|
||||
|
||||
**Conformance Breakdown:**
|
||||
|
||||
The **full suite** pass rates (server total, client total) are used for tier threshold checks. To interpret them, present a single conformance matrix combining server and client results. Each detail entry in the tier-check JSON has a `specVersions` field; client category is derived from the scenario name (`auth/` prefix = Auth, everything else = Core). Server scenarios are all Core.
|
||||
|
||||
Example:
|
||||
|
||||
| | 2025-03-26 | 2025-06-18 | 2025-11-25 | All\* |
|
||||
| ------------ | ---------- | ---------- | ---------- | ------------ |
|
||||
| Server | — | 26/26 | 4/4 | 30/30 (100%) |
|
||||
| Client: Core | — | 2/2 | 2/2 | 4/4 (100%) |
|
||||
| Client: Auth | 2/2 | 3/3 | 6/11 | 8/16 (50%) |
|
||||
|
||||
Informational (not scored for tier):
|
||||
|
||||
| | draft | extension |
|
||||
| ------------ | ----- | --------- |
|
||||
| Client: Auth | 0/1 | 0/2 |
|
||||
|
||||
The tier-scoring table only includes date-versioned scenarios. `draft` and `extension` scenarios are shown separately as informational — they do not affect tier advancement.
|
||||
|
||||
This immediately shows where failures concentrate. Failures clustered in Client: Auth / `2025-11-25` means "new auth features not yet implemented" — a scope gap, not a quality problem. Failures in Server or Client: Core are more concerning.
|
||||
|
||||
If the SDK has a `baseline.yml` or expected-failures file, cross-reference with the matrix to identify whether baselined failures cluster in a specific cell (e.g. all in `2025-11-25` / Client: Auth = scope gap).
|
||||
|
||||
**P0 Label Audit Guidance:**
|
||||
|
||||
When evaluating P0 metrics, flag potentially mislabeled P0 issues:
|
||||
|
||||
- If P0 count is high (>2) but other Tier 2 metrics (conformance, triage compliance, docs) are strong, this may indicate P0 labels are being used for enhancements, lower-priority work, or feature requests rather than actual critical bugs.
|
||||
- In such cases, recommend a P0 label audit as a remediation action. Review open P0 issues to verify they represent genuine blocking defects vs. misclassified work.
|
||||
- Document this finding in the remediation output with specific issue numbers and suggested re-triage actions.
|
||||
- Do not treat high P0 count as an automatic hard blocker if the audit reveals mislabeling; instead, note it as a process improvement opportunity.
|
||||
|
||||
## Step 5: Generate Output
|
||||
|
||||
Write detailed reports to files using subagents, then show a concise summary to the user.
|
||||
|
||||
### Output files (write via subagents)
|
||||
|
||||
**IMPORTANT**: Write both report files using parallel subagents (Task tool) so the file-writing work does not pollute the main conversation thread. Launch both subagents at the same time.
|
||||
|
||||
Write two files to `results/` in the conformance repo:
|
||||
|
||||
- `results/<YYYY-MM-DD>-<sdk-name>-assessment.md`
|
||||
- `results/<YYYY-MM-DD>-<sdk-name>-remediation.md`
|
||||
|
||||
For example: `results/2026-02-11-typescript-sdk-assessment.md`
|
||||
|
||||
#### Assessment subagent
|
||||
|
||||
Pass all the gathered data (CLI scorecard JSON, docs evaluation results, policy evaluation results) to a subagent and instruct it to write the assessment file using the template from `references/report-template.md`. This file contains the full requirements table, conformance test details (both server and client), triage metrics, documentation coverage table, and policy evaluation evidence.
|
||||
|
||||
#### Remediation subagent
|
||||
|
||||
Pass all the gathered data to a subagent and instruct it to write the remediation file using the template from `references/report-template.md`. This file always includes both:
|
||||
|
||||
- **Path to Tier 2** (if current tier is 3) -- what's needed to reach Tier 2
|
||||
- **Path to Tier 1** (always) -- what's needed to reach Tier 1
|
||||
|
||||
### Console output (shown to the user)
|
||||
|
||||
After the subagents finish, output a short executive summary directly to the user:
|
||||
|
||||
```
|
||||
## <sdk-name> — Tier <X>
|
||||
|
||||
Conformance:
|
||||
|
||||
| | 2025-03-26 | 2025-06-18 | 2025-11-25 | All* | T2 | T1 |
|
||||
|--------------|------------|------------|------------|------|----|----|
|
||||
| Server | — | pass/total | pass/total | pass/total (rate%) | ✓/✗ | ✓/✗ |
|
||||
| Client: Core | — | pass/total | pass/total | pass/total (rate%) | — | — |
|
||||
| Client: Auth | pass/total | pass/total | pass/total | pass/total (rate%) | — | — |
|
||||
| **Client Total** | | | | **pass/total (rate%)** | **✓/✗** | **✓/✗** |
|
||||
|
||||
\* unique scenarios — a scenario may apply to multiple spec versions
|
||||
|
||||
Informational (not scored for tier):
|
||||
|
||||
| | draft | extension |
|
||||
|--------------|-------|-----------|
|
||||
| Client: Auth | pass/total | pass/total |
|
||||
|
||||
If a baseline file was found, add a note below the conformance table:
|
||||
> **Baseline**: {N} failures in `baseline.yml` ({list by cell, e.g. "6 in Client: Auth/2025-11-25, 2 in Client: Auth/extension"}).
|
||||
|
||||
Repository Health:
|
||||
|
||||
| Check | Value | T2 | T1 |
|
||||
|-------|-------|----|----|
|
||||
| Issue Triage | <rate>% (<triaged>/<total>) | ✓/✗ | ✓/✗ |
|
||||
| Labels | <present>/<required> | ✓/✗ | ✓/✗ |
|
||||
| P0 Resolution | <count> open | ✓/✗ | ✓/✗ |
|
||||
| Spec Tracking | <days>d gap | ✓/✗ | ✓/✗ |
|
||||
| Documentation | <pass>/<total> features | ✓/✗ | ✓/✗ |
|
||||
| Dependency Policy | <summary> | ✓/✗ | ✓/✗ |
|
||||
| Roadmap | <summary> | ✓/✗ | ✓/✗ |
|
||||
| Versioning Policy | <summary> | N/A | ✓/✗ |
|
||||
| Stable Release | <version> | ✓/✗ | ✓/✗ |
|
||||
|
||||
---
|
||||
|
||||
**High-Priority Fixes:**
|
||||
1. <fix description>
|
||||
|
||||
**For Tier 2:**
|
||||
1. <gap description>
|
||||
2. <gap description>
|
||||
|
||||
**For Tier 1:**
|
||||
1. <gap description>
|
||||
2. <gap description>
|
||||
|
||||
Reports:
|
||||
- results/<date>-<sdk-name>-assessment.md
|
||||
- results/<date>-<sdk-name>-remediation.md
|
||||
```
|
||||
|
||||
Use ✓ for pass and ✗ for fail.
|
||||
|
||||
**High-Priority Fixes**: List any issues that need urgent attention (e.g., P0 label audit if P0 count is >2 but other metrics are strong, suggesting mislabeled issues). If none, omit this section.
|
||||
|
||||
**For Tier 2 / For Tier 1**: List each gap as a separate numbered item. Use "All requirements met" if there are no gaps for that tier. Each item should be a concise action (e.g., "Re-triage mislabeled P0s", "Document 16 undocumented core features").
|
||||
|
||||
## Reference Files
|
||||
|
||||
The following reference files are available in the `references/` directory alongside this skill:
|
||||
|
||||
- `references/feature-list.md` -- Canonical list of 48 non-experimental + 5 experimental features (single source of truth)
|
||||
- `references/tier-requirements.md` -- Full SEP-1730 requirements table with exact thresholds
|
||||
- `references/report-template.md` -- Output format template for the audit report
|
||||
- `references/docs-coverage-prompt.md` -- Evaluation prompt for documentation coverage
|
||||
- `references/policy-evaluation-prompt.md` -- Evaluation prompt for policy review
|
||||
|
||||
Read these reference files when you need the detailed content for evaluation prompts or report formatting.
|
||||
|
||||
## Usage Examples
|
||||
|
||||
```
|
||||
# TypeScript SDK — server + client conformance
|
||||
/mcp-sdk-tier-audit ~/src/mcp/typescript-sdk http://localhost:3000/mcp "npx tsx ~/src/mcp/typescript-sdk/test/conformance/src/everythingClient.ts"
|
||||
|
||||
# Python SDK — server + client conformance
|
||||
/mcp-sdk-tier-audit ~/src/mcp/python-sdk http://localhost:3001/mcp "uv run python ~/src/mcp/python-sdk/.github/actions/conformance/client.py"
|
||||
|
||||
# Go SDK — server + client conformance
|
||||
/mcp-sdk-tier-audit ~/src/mcp/go-sdk http://localhost:3002 "/tmp/go-conformance-client"
|
||||
|
||||
# C# SDK — server + client conformance
|
||||
# Two C#-specific requirements in the client-cmd:
|
||||
# --framework net9.0 : required because the project targets net8.0/net9.0/net10.0
|
||||
# -- $MCP_CONFORMANCE_SCENARIO : the runner sets this env var and uses shell:true, so the
|
||||
# shell expands it; dotnet passes [scenario, url] to the program
|
||||
/mcp-sdk-tier-audit ~/src/mcp/csharp-sdk http://localhost:3003 "dotnet run --project ~/src/mcp/csharp-sdk/tests/ModelContextProtocol.ConformanceClient --framework net9.0 -- $MCP_CONFORMANCE_SCENARIO"
|
||||
|
||||
# Any SDK — server conformance only (no client)
|
||||
/mcp-sdk-tier-audit ~/src/mcp/some-sdk http://localhost:3004
|
||||
```
|
||||
|
|
@ -1,139 +0,0 @@
|
|||
# Documentation Coverage Subagent Prompt
|
||||
|
||||
You are evaluating the documentation coverage of an MCP SDK repository for the SEP-1730 tier assessment.
|
||||
|
||||
## Input
|
||||
|
||||
- **SDK path**: {local-path} (absolute path to local SDK checkout)
|
||||
|
||||
## Your Task
|
||||
|
||||
Evaluate the documentation quality and coverage of this MCP SDK against the canonical feature list. You need to determine:
|
||||
|
||||
1. **Tier 1 compliance**: Are ALL non-experimental features documented with examples?
|
||||
2. **Tier 2 compliance**: Are core features documented (basic docs)?
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Read the canonical feature list
|
||||
|
||||
Read `references/feature-list.md` for the definitive list of 48 non-experimental features (plus 5 experimental) to evaluate. That file is the single source of truth — use every feature listed there, in order.
|
||||
|
||||
### 2. Find all documentation sources
|
||||
|
||||
The SDK is available at `{local-path}`. Search for documentation in these locations:
|
||||
|
||||
- `README.md` (root and any subdirectory READMEs)
|
||||
- `docs/` directory
|
||||
- `documentation/` directory
|
||||
- `examples/` directory
|
||||
- API documentation (generated or hand-written)
|
||||
- `CONTRIBUTING.md`
|
||||
- Inline code comments and docstrings on public API surfaces
|
||||
- Any `*.md` files in the repo
|
||||
|
||||
```bash
|
||||
# Find all markdown files
|
||||
find {local-path} -name "*.md" -not -path "*/node_modules/*" -not -path "*/.git/*"
|
||||
|
||||
# Find example files
|
||||
find {local-path} -path "*/examples/*" -not -path "*/node_modules/*"
|
||||
|
||||
# Find API docs
|
||||
find {local-path} -path "*/docs/*" -not -path "*/node_modules/*"
|
||||
```
|
||||
|
||||
### 3. Evaluate each feature
|
||||
|
||||
For each of the 48 non-experimental features in the canonical list, determine:
|
||||
|
||||
- **Documented?**: Is there documentation explaining this feature? (Yes/No)
|
||||
- **Where**: File path and line numbers where documentation exists
|
||||
- **Has Examples?**: Are there code examples showing how to use this feature? (Yes/No/N/A)
|
||||
- **Verdict**: PASS (documented with examples), PARTIAL (documented but no examples), or FAIL (not documented)
|
||||
|
||||
## Required Output Format
|
||||
|
||||
Produce your assessment in this exact format:
|
||||
|
||||
```markdown
|
||||
### Documentation Coverage Assessment
|
||||
|
||||
**SDK path**: {local-path}
|
||||
**Documentation locations found**:
|
||||
|
||||
- {path1}: {description}
|
||||
- {path2}: {description}
|
||||
- ...
|
||||
|
||||
#### Feature Documentation Table
|
||||
|
||||
One row per feature from `references/feature-list.md`. Use the exact feature numbers and names from that file.
|
||||
|
||||
| # | Feature | Documented? | Where | Has Examples? | Verdict |
|
||||
| --- | -------------------------- | ----------- | -------------- | ----------------------- | ----------------- |
|
||||
| 1 | Tools - listing | Yes/No | {file}:{lines} | Yes ({N} examples) / No | PASS/PARTIAL/FAIL |
|
||||
| 2 | Tools - calling | Yes/No | {file}:{lines} | Yes ({N} examples) / No | PASS/PARTIAL/FAIL |
|
||||
| ... | ... | ... | ... | ... | ... |
|
||||
| 48 | JSON Schema 2020-12 | Yes/No | {file}:{lines} | Yes ({N} examples) / No | PASS/PARTIAL/FAIL |
|
||||
| — | Tasks - get (experimental) | Yes/No | {file}:{lines} | Yes ({N} examples) / No | INFO |
|
||||
| ... | ... | ... | ... | ... | ... |
|
||||
|
||||
All 48 non-experimental features MUST appear in the table. Do not skip or merge rows.
|
||||
|
||||
#### Summary
|
||||
|
||||
**Total non-experimental features**: 48
|
||||
**PASS (documented with examples)**: {N}/48
|
||||
**PARTIAL (documented, no examples)**: {N}/48
|
||||
**FAIL (not documented)**: {N}/48
|
||||
|
||||
**Core features documented**: {N}/{total core} ({percentage}%)
|
||||
**All features documented with examples**: {N}/48 ({percentage}%)
|
||||
|
||||
#### Tier Verdicts
|
||||
|
||||
**Tier 1** (all non-experimental features documented with examples): **PASS/FAIL**
|
||||
|
||||
- {If FAIL: list the features missing documentation or examples}
|
||||
|
||||
**Tier 2** (basic docs covering core features): **PASS/FAIL**
|
||||
|
||||
- {If FAIL: list the core features missing documentation}
|
||||
```
|
||||
|
||||
## What Counts as "Documented"
|
||||
|
||||
A feature is "documented" only if there is **prose documentation** (in README, docs/, or similar) explaining what the feature does, when to use it, and how it works. The following do **not** count as documentation on their own:
|
||||
|
||||
- Example code without accompanying prose explanation
|
||||
- Conformance test servers or test fixtures
|
||||
- Source code, even with comments or docstrings
|
||||
- Mere existence of an API (e.g., a function existing in the SDK)
|
||||
|
||||
**Examples supplement documentation but do not replace it.** A feature with a working example in `examples/` but no prose explaining the feature is PARTIAL, not PASS. A feature with only a conformance server implementation and no user-facing docs is FAIL.
|
||||
|
||||
### Verdict criteria
|
||||
|
||||
- **PASS**: Prose documentation exists explaining the feature AND at least one runnable or near-runnable code example
|
||||
- **PARTIAL**: Either prose docs exist but no examples, OR examples exist but no prose docs
|
||||
- **FAIL**: No prose documentation and no examples. Also use FAIL if the feature is only demonstrated in test/conformance code with no user-facing docs or examples
|
||||
|
||||
### What counts as an "example"
|
||||
|
||||
- Runnable code in an `examples/` directory
|
||||
- Code snippets embedded in prose documentation (README, docs/\*.md)
|
||||
- Go `Example*` test functions (these render on pkg.go.dev and are a language convention)
|
||||
- Examples in test files count only if they are clearly labeled as examples or referenced from documentation
|
||||
|
||||
### What does NOT count as an example
|
||||
|
||||
- Conformance test server implementations
|
||||
- Internal test fixtures
|
||||
- Source code of the SDK itself
|
||||
|
||||
## Other Important Notes
|
||||
|
||||
- If the SDK does not implement a feature at all, mark it as "FAIL" for documentation but note "Not implemented" in the Where column.
|
||||
- Be thorough: check README, docs/, examples/, API references, and inline docstrings.
|
||||
- Apply these criteria consistently across all features. Do not give credit for documentation that doesn't exist.
|
||||
|
|
@ -1,80 +0,0 @@
|
|||
# MCP SDK Canonical Feature List
|
||||
|
||||
Single source of truth for all MCP features evaluated in the tier audit. **48 non-experimental features** plus 5 experimental (informational only).
|
||||
|
||||
When updating this list, also update the total count referenced in `docs-coverage-prompt.md`.
|
||||
|
||||
## Non-Experimental Features (48 total)
|
||||
|
||||
### Core Features (36)
|
||||
|
||||
| # | Feature | Protocol Method |
|
||||
| --- | ----------------------------------- | -------------------------------------- |
|
||||
| 1 | Tools - listing | `tools/list` |
|
||||
| 2 | Tools - calling | `tools/call` |
|
||||
| 3 | Tools - text results | |
|
||||
| 4 | Tools - image results | |
|
||||
| 5 | Tools - audio results | |
|
||||
| 6 | Tools - embedded resources | |
|
||||
| 7 | Tools - error handling | |
|
||||
| 8 | Tools - change notifications | `notifications/tools/list_changed` |
|
||||
| 9 | Resources - listing | `resources/list` |
|
||||
| 10 | Resources - reading text | `resources/read` |
|
||||
| 11 | Resources - reading binary | `resources/read` |
|
||||
| 12 | Resources - templates | `resources/templates/list` |
|
||||
| 13 | Resources - template reading | |
|
||||
| 14 | Resources - subscribing | `resources/subscribe` |
|
||||
| 15 | Resources - unsubscribing | `resources/unsubscribe` |
|
||||
| 16 | Resources - change notifications | `notifications/resources/list_changed` |
|
||||
| 17 | Prompts - listing | `prompts/list` |
|
||||
| 18 | Prompts - getting simple | `prompts/get` |
|
||||
| 19 | Prompts - getting with arguments | `prompts/get` |
|
||||
| 20 | Prompts - embedded resources | |
|
||||
| 21 | Prompts - image content | |
|
||||
| 22 | Prompts - change notifications | `notifications/prompts/list_changed` |
|
||||
| 23 | Sampling - creating messages | `sampling/createMessage` |
|
||||
| 24 | Elicitation - form mode | `elicitation/create` |
|
||||
| 25 | Elicitation - URL mode | `elicitation/create` (mode: "url") |
|
||||
| 26 | Elicitation - schema validation | |
|
||||
| 27 | Elicitation - default values | |
|
||||
| 28 | Elicitation - enum values | |
|
||||
| 29 | Elicitation - complete notification | `notifications/elicitation/complete` |
|
||||
| 30 | Roots - listing | `roots/list` |
|
||||
| 31 | Roots - change notifications | `notifications/roots/list_changed` |
|
||||
| 32 | Logging - sending log messages | `notifications/message` |
|
||||
| 33 | Logging - setting level | `logging/setLevel` |
|
||||
| 34 | Completions - resource argument | `completion/complete` |
|
||||
| 35 | Completions - prompt argument | `completion/complete` |
|
||||
| 36 | Ping | `ping` |
|
||||
|
||||
### Transport Features (6)
|
||||
|
||||
| # | Feature |
|
||||
| --- | ---------------------------------- |
|
||||
| 37 | Streamable HTTP transport (client) |
|
||||
| 38 | Streamable HTTP transport (server) |
|
||||
| 39 | SSE transport - legacy (client) |
|
||||
| 40 | SSE transport - legacy (server) |
|
||||
| 41 | stdio transport (client) |
|
||||
| 42 | stdio transport (server) |
|
||||
|
||||
### Protocol Features (6)
|
||||
|
||||
| # | Feature |
|
||||
| --- | ---------------------------- |
|
||||
| 43 | Progress notifications |
|
||||
| 44 | Cancellation |
|
||||
| 45 | Pagination |
|
||||
| 46 | Capability negotiation |
|
||||
| 47 | Protocol version negotiation |
|
||||
| 48 | JSON Schema 2020-12 support |
|
||||
|
||||
## Experimental Features (5, informational only)
|
||||
|
||||
| # | Feature | Protocol Method |
|
||||
| --- | ---------------------------- | ---------------------------- |
|
||||
| — | Tasks - get | `tasks/get` |
|
||||
| — | Tasks - result | `tasks/result` |
|
||||
| — | Tasks - cancel | `tasks/cancel` |
|
||||
| — | Tasks - list | `tasks/list` |
|
||||
| — | Tasks - status notifications | `notifications/tasks/status` |
|
||||
|
|
@ -1,145 +0,0 @@
|
|||
# Policy Evaluation Prompt
|
||||
|
||||
You are evaluating the governance and policy documentation of an MCP SDK repository for the SEP-1730 tier assessment.
|
||||
|
||||
## Input
|
||||
|
||||
- **SDK path**: {local-path} (absolute path to local SDK checkout)
|
||||
- **Repository**: {repo} (GitHub `owner/repo`, derived from git remote)
|
||||
- **CLI policy_signals**: {policy_signals_json} (from the tier-check CLI output — shows which files exist)
|
||||
|
||||
## Your Task
|
||||
|
||||
The CLI has already determined which policy files exist in the repository. Your job is to **read and evaluate the content** of the files that were found. Do NOT search for files in other locations — only evaluate what the CLI reported as present.
|
||||
|
||||
Three policy areas to evaluate:
|
||||
|
||||
1. **Dependency update policy** (required for Tier 1 and Tier 2)
|
||||
2. **Roadmap** (Tier 1: published roadmap; Tier 2: published plan toward Tier 1)
|
||||
3. **Versioning policy** (Tier 1 only: documented breaking change policy)
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Identify which files exist from CLI output
|
||||
|
||||
From the `policy_signals.files` object in the CLI JSON output, note which files have `true` (exist) vs `false` (missing).
|
||||
|
||||
The CLI checks these files:
|
||||
|
||||
**Dependency policy**: `DEPENDENCY_POLICY.md`, `docs/dependency-policy.md`, `.github/dependabot.yml`, `.github/renovate.json`, `renovate.json`
|
||||
|
||||
**Roadmap**: `ROADMAP.md`, `docs/roadmap.md`
|
||||
|
||||
**Versioning**: `VERSIONING.md`, `docs/versioning.md`, `BREAKING_CHANGES.md`
|
||||
|
||||
**General** (may contain relevant sections): `CONTRIBUTING.md`
|
||||
|
||||
### 2. Read and evaluate files that exist
|
||||
|
||||
For each file that the CLI reported as present, read its content at `{local-path}/{file}` and evaluate:
|
||||
|
||||
- Is the content substantive (not just a placeholder title)?
|
||||
- Does it meet the criteria below?
|
||||
|
||||
**Do NOT** search the repo for policy information in other files. If the dedicated file doesn't exist, the policy is not published.
|
||||
|
||||
## Evaluation Criteria
|
||||
|
||||
### Dependency Update Policy
|
||||
|
||||
**PASS** if any of these exist with substantive content:
|
||||
|
||||
- `DEPENDENCY_POLICY.md` or `docs/dependency-policy.md` — must describe how and when dependencies are updated
|
||||
- `.github/dependabot.yml` or `.github/renovate.json` or `renovate.json` — automated tooling counts as a published policy in practice
|
||||
|
||||
**FAIL** if none of the above exist (per CLI output).
|
||||
|
||||
### Roadmap
|
||||
|
||||
**PASS for Tier 1**: `ROADMAP.md` or `docs/roadmap.md` exists with concrete work items tracking MCP spec components.
|
||||
|
||||
**PASS for Tier 2**: Same file exists with at least a plan toward Tier 1, or explanation for remaining at Tier 2.
|
||||
|
||||
**FAIL** if no roadmap file exists (per CLI output).
|
||||
|
||||
### Versioning Policy
|
||||
|
||||
**PASS for Tier 1** if any of these exist with substantive content:
|
||||
|
||||
- `VERSIONING.md` or `docs/versioning.md` or `BREAKING_CHANGES.md`
|
||||
- A clearly labeled "Versioning" or "Breaking Changes" section in `CONTRIBUTING.md` (only check if CONTRIBUTING.md exists per CLI output)
|
||||
|
||||
The content must describe: what constitutes a breaking change, how breaking changes are communicated, and the versioning scheme.
|
||||
|
||||
**Not required for Tier 2.**
|
||||
|
||||
**FAIL** if no versioning documentation found in the above files.
|
||||
|
||||
## Required Output Format
|
||||
|
||||
```markdown
|
||||
### Policy Evaluation Assessment
|
||||
|
||||
**SDK path**: {local-path}
|
||||
**Repository**: {repo}
|
||||
|
||||
---
|
||||
|
||||
#### 1. Dependency Update Policy: {PASS/FAIL}
|
||||
|
||||
| File | Exists (CLI) | Content Verdict |
|
||||
| ------------------------- | ------------ | ------------------------------- |
|
||||
| DEPENDENCY_POLICY.md | Yes/No | Substantive / Placeholder / N/A |
|
||||
| docs/dependency-policy.md | Yes/No | Substantive / Placeholder / N/A |
|
||||
| .github/dependabot.yml | Yes/No | Configured / N/A |
|
||||
| .github/renovate.json | Yes/No | Configured / N/A |
|
||||
|
||||
**Verdict**: **PASS/FAIL** — {one-line explanation}
|
||||
|
||||
---
|
||||
|
||||
#### 2. Roadmap: {PASS/FAIL}
|
||||
|
||||
| File | Exists (CLI) | Content Verdict |
|
||||
| --------------- | ------------ | ------------------------------- |
|
||||
| ROADMAP.md | Yes/No | Substantive / Placeholder / N/A |
|
||||
| docs/roadmap.md | Yes/No | Substantive / Placeholder / N/A |
|
||||
|
||||
**Verdict**:
|
||||
|
||||
- **Tier 1**: **PASS/FAIL** — {one-line explanation}
|
||||
- **Tier 2**: **PASS/FAIL** — {one-line explanation}
|
||||
|
||||
---
|
||||
|
||||
#### 3. Versioning Policy: {PASS/FAIL}
|
||||
|
||||
| File | Exists (CLI) | Content Verdict |
|
||||
| ------------------------------------ | ------------ | ------------------------------- |
|
||||
| VERSIONING.md | Yes/No | Substantive / Placeholder / N/A |
|
||||
| docs/versioning.md | Yes/No | Substantive / Placeholder / N/A |
|
||||
| BREAKING_CHANGES.md | Yes/No | Substantive / Placeholder / N/A |
|
||||
| CONTRIBUTING.md (versioning section) | Yes/No | Found / Not found / N/A |
|
||||
|
||||
**Verdict**:
|
||||
|
||||
- **Tier 1**: **PASS/FAIL** — {one-line explanation}
|
||||
- **Tier 2**: **N/A** — only requires stable release
|
||||
|
||||
---
|
||||
|
||||
#### Overall Policy Summary
|
||||
|
||||
| Policy Area | Tier 1 | Tier 2 |
|
||||
| ------------------------ | --------- | --------- |
|
||||
| Dependency Update Policy | PASS/FAIL | PASS/FAIL |
|
||||
| Roadmap | PASS/FAIL | PASS/FAIL |
|
||||
| Versioning Policy | PASS/FAIL | N/A |
|
||||
```
|
||||
|
||||
## Important Notes
|
||||
|
||||
- Only evaluate files the CLI reported as existing. Do not search the repo for alternatives.
|
||||
- If a file exists but is just a placeholder (e.g., only has a title with no content), mark it as "Placeholder" and FAIL.
|
||||
- Dependabot/Renovate config files pass automatically if they exist and are properly configured.
|
||||
- CHANGELOG.md showing past releases does NOT count as a roadmap.
|
||||
|
|
@ -1,144 +0,0 @@
|
|||
# Report Templates
|
||||
|
||||
Write two files to `results/` in the conformance repo:
|
||||
|
||||
- `results/<YYYY-MM-DD>-<sdk-name>-assessment.md`
|
||||
- `results/<YYYY-MM-DD>-<sdk-name>-remediation.md`
|
||||
|
||||
## assessment.md
|
||||
|
||||
```markdown
|
||||
# MCP SDK Tier Audit: {repo}
|
||||
|
||||
**Date**: {date}
|
||||
**Branch**: {branch}
|
||||
**Auditor**: mcp-sdk-tier-audit skill (automated + subagent evaluation)
|
||||
|
||||
## Tier Assessment: Tier {X}
|
||||
|
||||
{Brief 1-2 sentence summary of the overall assessment and key factors.}
|
||||
|
||||
### Requirements Summary
|
||||
|
||||
| # | Requirement | Tier 1 Standard | Tier 2 Standard | Current Value | T1? | T2? | Gap |
|
||||
| --- | ----------------------- | --------------------------------- | ---------------------------- | --------------------------------- | ----------- | ----------- | ------------------ |
|
||||
| 1a | Server Conformance | 100% pass rate | >= 80% pass rate | {X}% ({passed}/{total}) | {PASS/FAIL} | {PASS/FAIL} | {detail or "None"} |
|
||||
| 1b | Client Conformance | 100% pass rate | >= 80% pass rate | {X}% ({passed}/{total}) | {PASS/FAIL} | {PASS/FAIL} | {detail or "None"} |
|
||||
| 2 | Issue Triage | >= 90% within 2 biz days | >= 80% within 1 month | {compliance}% ({triaged}/{total}) | {PASS/FAIL} | {PASS/FAIL} | {detail or "None"} |
|
||||
| 2b | Labels | 12 required labels | 12 required labels | {present}/{required} | {PASS/FAIL} | {PASS/FAIL} | {detail or "None"} |
|
||||
| 3 | Critical Bug Resolution | All P0s within 7 days | All P0s within 2 weeks | {open P0 count} open | {PASS/FAIL} | {PASS/FAIL} | {detail or "None"} |
|
||||
| 4 | Stable Release | Required + clear versioning | At least one stable release | {version} | {PASS/FAIL} | {PASS/FAIL} | {detail or "None"} |
|
||||
| 4b | Spec Tracking | Timeline agreed per release | Within 6 months | {days_gap}d gap ({PASS/FAIL}) | {PASS/FAIL} | {PASS/FAIL} | {detail or "None"} |
|
||||
| 5 | Documentation | Comprehensive w/ examples | Basic docs for core features | {pass}/{total} features | {PASS/FAIL} | {PASS/FAIL} | {detail or "None"} |
|
||||
| 6 | Dependency Policy | Published update policy | Published update policy | {Found/Not found} | {PASS/FAIL} | {PASS/FAIL} | {detail or "None"} |
|
||||
| 7 | Roadmap | Published roadmap | Plan toward Tier 1 | {Found/Not found} | {PASS/FAIL} | {PASS/FAIL} | {detail or "None"} |
|
||||
| 8 | Versioning Policy | Documented breaking change policy | N/A | {Found/Not found} | {PASS/FAIL} | N/A | {detail or "None"} |
|
||||
|
||||
### Tier Determination
|
||||
|
||||
- Tier 1: {PASS/FAIL} -- {count}/8 requirements met (failing: {list})
|
||||
- Tier 2: {PASS/FAIL} -- {count}/7 requirements met (failing: {list})
|
||||
- **Final Tier: {X}**
|
||||
|
||||
---
|
||||
|
||||
## Server Conformance Details
|
||||
|
||||
Pass rate: {X}% ({passed}/{total})
|
||||
|
||||
| Scenario | Status | Checks |
|
||||
| -------- | ----------- | ---------------- |
|
||||
| {name} | {PASS/FAIL} | {passed}/{total} |
|
||||
| ... | ... | ... |
|
||||
|
||||
---
|
||||
|
||||
## Client Conformance Details
|
||||
|
||||
Full suite pass rate: {X}% ({passed}/{total})
|
||||
|
||||
> **Suite breakdown**: Core: {core_pass}/{core_total} ({core_rate}%), Auth: {auth_pass}/{auth_total} ({auth_rate}%)
|
||||
> **Baseline**: {N} known expected failures documented in `{baseline_file}` ({categories})
|
||||
|
||||
### Core Scenarios
|
||||
|
||||
| Scenario | Status | Checks |
|
||||
| -------- | ----------- | ---------------- |
|
||||
| {name} | {PASS/FAIL} | {passed}/{total} |
|
||||
| ... | ... | ... |
|
||||
|
||||
### Auth Scenarios
|
||||
|
||||
| Scenario | Status | Checks | Notes |
|
||||
| -------- | ----------- | ---------------- | --------------------------- |
|
||||
| {name} | {PASS/FAIL} | {passed}/{total} | {in baseline? / unexpected} |
|
||||
| ... | ... | ... | ... |
|
||||
|
||||
---
|
||||
|
||||
## Issue Triage Details
|
||||
|
||||
Analysis period: Last {N} issues
|
||||
Labels: {present/missing list}
|
||||
|
||||
| Metric | Value | T1 Req | T2 Req | Verdict |
|
||||
| --------------- | ----- | ------ | ------ | --------- |
|
||||
| Compliance rate | {X}% | >= 90% | >= 80% | {verdict} |
|
||||
| Exceeding SLA | {N} | -- | -- | -- |
|
||||
| Open P0s | {N} | 0 | 0 | {verdict} |
|
||||
|
||||
{If open P0s, list them with issue number, title, age}
|
||||
|
||||
---
|
||||
|
||||
## Documentation Coverage
|
||||
|
||||
{Paste subagent 1 output: feature table with Documented/Where/Examples/Verdict columns}
|
||||
|
||||
---
|
||||
|
||||
## Policy Evaluation
|
||||
|
||||
{Paste subagent 2 output: dependency policy, roadmap, versioning policy sections with evidence tables}
|
||||
```
|
||||
|
||||
## remediation.md
|
||||
|
||||
```markdown
|
||||
# Remediation Guide: {repo}
|
||||
|
||||
**Date**: {date}
|
||||
**Current Tier**: {X}
|
||||
|
||||
## Path to Tier 2
|
||||
|
||||
{Only include this section if current tier is 3. List every requirement not met for Tier 2.}
|
||||
|
||||
| # | Action | Requirement | Effort | Where |
|
||||
| --- | ------------- | ------------- | -------------------- | ------------ |
|
||||
| 1 | {description} | {requirement} | {Small/Medium/Large} | {file paths} |
|
||||
| ... | ... | ... | ... | ... |
|
||||
|
||||
## Path to Tier 1
|
||||
|
||||
{Always include this section. List every requirement not met for Tier 1 (including any Tier 2 gaps).}
|
||||
|
||||
| # | Action | Requirement | Effort | Where |
|
||||
| --- | ------------- | ------------- | -------------------- | ------------ |
|
||||
| 1 | {description} | {requirement} | {Small/Medium/Large} | {file paths} |
|
||||
| ... | ... | ... | ... | ... |
|
||||
|
||||
## Recommended Next Steps
|
||||
|
||||
1. {First priority action with brief rationale}
|
||||
2. {Second priority action}
|
||||
3. {Third priority action}
|
||||
```
|
||||
|
||||
## Formatting Rules
|
||||
|
||||
1. Every PASS/FAIL must be based on evidence, not assumption.
|
||||
2. If data is unavailable, mark as "N/A - {reason}" and note in remediation.
|
||||
3. All file references must include file path and line numbers where possible.
|
||||
4. Remediation items ordered by impact: tier-advancing items first.
|
||||
5. Effort estimates: Small (< 1 day), Medium (1-3 days), Large (> 3 days).
|
||||
|
|
@ -1,104 +0,0 @@
|
|||
# SEP-1730: SDK Tier Requirements Reference
|
||||
|
||||
This is the authoritative reference table for MCP SDK tiering requirements, extracted from SEP-1730.
|
||||
|
||||
Source: `modelcontextprotocol/docs/community/sdk-tiers.mdx` in the spec repository
|
||||
|
||||
## Full Requirements Table
|
||||
|
||||
| Requirement | Tier 1: Fully Supported | Tier 2: Commitment to Full Support | Tier 3: Experimental |
|
||||
| --------------------------- | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | ---------------------- |
|
||||
| **Conformance Tests** | 100% pass rate | >= 80% pass rate | No minimum |
|
||||
| **New Protocol Features** | Before new spec version release, timeline agreed per release based on feature complexity | Within 6 months | No timeline commitment |
|
||||
| **Issue Triage** | Within 2 business days | Within a month | No requirement |
|
||||
| **Critical Bug Resolution** | Within 7 days | Within two weeks | No requirement |
|
||||
| **Stable Release** | Required with clear versioning | At least one stable release | Not required |
|
||||
| **Documentation** | Comprehensive with examples for all features | Basic documentation covering core features | No minimum |
|
||||
| **Dependency Policy** | Published update policy | Published update policy | Not required |
|
||||
| **Roadmap** | Published roadmap | Published plan toward Tier 1 or explanation for remaining Tier 2 | Not required |
|
||||
|
||||
## Exact Thresholds for Automated Checking
|
||||
|
||||
| Metric | Tier 1 Threshold | Tier 2 Threshold | How to Measure |
|
||||
| ---------------------- | ------------------------------------------------------ | ------------------------------- | ---------------------------------------------------------------------------------------- |
|
||||
| Conformance pass rate | == 100% | >= 80% | `passed / (passed + failed) * 100` from conformance suite |
|
||||
| Issue triage time | <= 2 business days | <= 1 month (30 calendar days) | Time from issue creation to first label application |
|
||||
| P0 resolution time | <= 7 calendar days | <= 14 calendar days | Time from P0 label application to issue close |
|
||||
| Stable release version | >= 1.0.0, no pre-release suffix | >= 1.0.0 (at least one) | Check `gh release list` for version matching `^[0-9]+\.[0-9]+\.[0-9]+$` where major >= 1 |
|
||||
| Documentation coverage | All non-experimental features documented with examples | Core features documented | Subagent evaluation |
|
||||
| Dependency policy | Published and findable in repo | Published and findable in repo | Subagent evaluation |
|
||||
| Roadmap | Published with concrete steps tracking spec components | Published plan toward Tier 1 | Subagent evaluation |
|
||||
| Versioning policy | Documented breaking change policy | N/A (just needs stable release) | Subagent evaluation |
|
||||
|
||||
## Conformance Score Calculation
|
||||
|
||||
Every scenario in the conformance suite has a `specVersions` field indicating which spec version it targets. The valid values are defined as the `SpecVersion` type (as a list) in `src/types.ts` — run `node dist/index.js list` to see the current mapping of scenarios to spec versions.
|
||||
|
||||
Date-versioned scenarios (e.g. `2025-06-18`, `2025-11-25`) count toward tier scoring. `draft` and `extension` scenarios are listed separately as informational.
|
||||
|
||||
The `--spec-version` CLI flag filters scenarios cumulatively for date versions (e.g. `--spec-version 2025-06-18` includes `2025-03-26` + `2025-06-18`). For `draft`/`extension`, it returns exact matches only.
|
||||
|
||||
The tier-check output includes a per-version pass rate breakdown alongside the aggregate.
|
||||
|
||||
## Tier Relegation Rules
|
||||
|
||||
- **Tier 1 to Tier 2**: Any conformance test fails continuously for 4 weeks
|
||||
- **Tier 2 to Tier 3**: More than 20% of conformance tests fail continuously for 4 weeks
|
||||
|
||||
## Issue Triage Label Taxonomy
|
||||
|
||||
SDK repositories must use these consistent labels to enable automated reporting.
|
||||
|
||||
### Type Labels (pick one)
|
||||
|
||||
| Label | Description |
|
||||
| ------------- | ----------------------------- |
|
||||
| `bug` | Something isn't working |
|
||||
| `enhancement` | Request for new feature |
|
||||
| `question` | Further information requested |
|
||||
|
||||
Note: Repositories using GitHub's native issue types satisfy this requirement without needing type labels.
|
||||
|
||||
### Status Labels (pick one)
|
||||
|
||||
| Label | Description |
|
||||
| -------------------- | ------------------------------------------------------- |
|
||||
| `needs confirmation` | Unclear if still relevant |
|
||||
| `needs repro` | Insufficient information to reproduce |
|
||||
| `ready for work` | Has enough information to start |
|
||||
| `good first issue` | Good for newcomers |
|
||||
| `help wanted` | Contributions welcome from those familiar with codebase |
|
||||
|
||||
### Priority Labels (only if actionable)
|
||||
|
||||
| Label | Description |
|
||||
| ----- | --------------------------------------------------------------- |
|
||||
| `P0` | Critical: core functionality failures or high-severity security |
|
||||
| `P1` | Significant bug affecting many users |
|
||||
| `P2` | Moderate issues, valuable feature requests |
|
||||
| `P3` | Nice to haves, rare edge cases |
|
||||
|
||||
**Total: 12 labels** (3 type + 5 status + 4 priority)
|
||||
|
||||
## Key Definitions
|
||||
|
||||
### Issue Triage
|
||||
|
||||
Labeling and determining whether an issue is valid. This is NOT the same as resolving the issue. An issue is considered triaged when it receives its first label.
|
||||
|
||||
### Critical Bug (P0)
|
||||
|
||||
- **Security vulnerabilities** with CVSS score >= 7.0 (High or Critical severity)
|
||||
- **Core functionality failures** that prevent basic MCP operations: connection establishment, message exchange, or use of core primitives (tools, resources, prompts)
|
||||
|
||||
### Stable Release
|
||||
|
||||
A published version explicitly marked as production-ready. Specifically: version `1.0.0` or higher without pre-release identifiers like `-alpha`, `-beta`, or `-rc`.
|
||||
|
||||
### Clear Versioning
|
||||
|
||||
Following idiomatic versioning patterns with documented breaking change policies, so users can understand compatibility expectations when upgrading.
|
||||
|
||||
### Roadmap
|
||||
|
||||
Outlines concrete steps and work items that track implementation of required MCP specification components (non-experimental features and optional capabilities), giving users visibility into upcoming feature support.
|
||||
|
|
@ -1,201 +0,0 @@
|
|||
---
|
||||
name: new-sep
|
||||
description: >-
|
||||
Scaffold a sep-NNNN.yaml requirement-traceability file for the MCP
|
||||
conformance repo from a SEP PR's spec diff. Runs the new-sep CLI, then
|
||||
parses the modelcontextprotocol/modelcontextprotocol spec diff to populate
|
||||
`requirements[]` with the RFC 2119 sentences and proposed check IDs.
|
||||
argument-hint: '<sep-number>'
|
||||
---
|
||||
|
||||
# new-sep: SEP traceability YAML scaffolding
|
||||
|
||||
You are bootstrapping a `sep-NNNN.yaml` file for a new SEP in the MCP conformance repo. The output is the requirement-traceability file specified by SEP-2484: a YAML that maps each normative sentence from the SEP's spec diff to a `check:` ID (testable) or an `excluded:` reason (not testable). The CLI gets the skeleton; you fill in the rows by reading the spec diff.
|
||||
|
||||
## Step 0: Pre-flight checks
|
||||
|
||||
Before doing anything else, verify GitHub CLI authentication:
|
||||
|
||||
```bash
|
||||
gh auth status 2>&1
|
||||
```
|
||||
|
||||
If this fails, stop immediately and tell the user:
|
||||
|
||||
> GitHub authentication is required for this skill. Please run `gh auth login` first, then re-run.
|
||||
|
||||
Verify you're running inside the conformance repo:
|
||||
|
||||
```bash
|
||||
test -f package.json && jq -r '.name' package.json
|
||||
```
|
||||
|
||||
The name should be `@modelcontextprotocol/conformance`. If not, stop and ask the user to `cd` into the conformance repo first.
|
||||
|
||||
## Step 1: Parse arguments
|
||||
|
||||
Extract from the user's input:
|
||||
|
||||
- **sep-number** (required): the SEP number, e.g. `2164`. This is also the PR number in `modelcontextprotocol/modelcontextprotocol` by convention.
|
||||
|
||||
## Step 2: Generate the skeleton
|
||||
|
||||
Run the CLI:
|
||||
|
||||
```bash
|
||||
npm run --silent build
|
||||
node dist/index.js new-sep <NNNN>
|
||||
```
|
||||
|
||||
(For development against a non-built source tree: `npx tsx src/index.ts new-sep ...`.)
|
||||
|
||||
The CLI writes `src/seps/sep-<NNNN>.yaml` with `sep`, `spec_url`, and two TODO `requirements[]` rows. Capture the output path from the CLI's `Wrote …` line and remember it as `$YAML`.
|
||||
|
||||
If the CLI errors with "does not change any docs/specification/draft/\*.mdx", the SEP's spec changes landed in a separate PR — ask the user for the spec file path and rerun with `--spec-path docs/specification/draft/<path>`. Do not guess.
|
||||
|
||||
## Step 3: Fetch the spec diff
|
||||
|
||||
`AGENTS.md` (lines 64–72) is explicit that severity must come from the spec text itself, not the SEP markdown or the conformance PR description:
|
||||
|
||||
```bash
|
||||
gh api "repos/modelcontextprotocol/modelcontextprotocol/pulls/<NNNN>/files" \
|
||||
--jq '.[] | select(.filename | test("^docs/specification/draft/.*\\.mdx$")) | {filename, patch}'
|
||||
```
|
||||
|
||||
For each file, pull the added (`+`-prefixed) lines from `patch`. If `patch` is truncated for a large file, fall back to fetching the whole file at the PR's head ref:
|
||||
|
||||
```bash
|
||||
gh api "repos/modelcontextprotocol/modelcontextprotocol/contents/<path>?ref=<sep-branch>" \
|
||||
--jq '.content' | base64 -d
|
||||
```
|
||||
|
||||
## Step 4: Extract RFC 2119 requirements
|
||||
|
||||
Walk the added lines and identify sentences containing the keywords: **MUST**, **MUST NOT**, **SHOULD**, **SHOULD NOT**, **REQUIRED**, **SHALL**, **SHALL NOT**, **MAY**, **OPTIONAL**.
|
||||
|
||||
**Quote the whole sentence**, not just the matched line. The matched word may sit inside a bullet point whose lead-in sentence supplies the keyword by inheritance — e.g.:
|
||||
|
||||
> Servers SHOULD return standard JSON-RPC errors for common failure cases:
|
||||
>
|
||||
> - Resource not found: -32602 (Invalid Params)
|
||||
|
||||
The bullet inherits `SHOULD`. The yaml row should quote the _combined_ obligation: `'Servers SHOULD return standard JSON-RPC errors for common failure cases: Resource not found: -32602 (Invalid Params)'` — see `src/seps/sep-2164.yaml` for the canonical example.
|
||||
|
||||
**Regex alone is insufficient** (this is called out in Issue #243). Read for context: pronouns, "the server", and "such cases" all refer back to the lead-in.
|
||||
|
||||
## Step 5: Map severity → check vs. excluded
|
||||
|
||||
From `AGENTS.md:50-56`:
|
||||
|
||||
| Keyword | Severity | YAML field |
|
||||
| ---------------------------------------------- | -------- | -------------------------- |
|
||||
| MUST / MUST NOT / SHALL / SHALL NOT / REQUIRED | FAILURE | `check: sep-<NNNN>-<slug>` |
|
||||
| SHOULD / SHOULD NOT | WARNING | `check: sep-<NNNN>-<slug>` |
|
||||
| MAY / OPTIONAL | — | _no row — skip entirely_ |
|
||||
|
||||
MAY / OPTIONAL sentences are noted in Step 4 only so you consciously skip them — they never produce a yaml row.
|
||||
|
||||
A row is `excluded:` when a MUST/SHOULD requirement can't be protocol-observed by the harness. Do **not** write any `excluded:` row on your own authority — every exclusion goes through Step 6.
|
||||
|
||||
While classifying, sort each MUST/SHOULD row into one of three buckets:
|
||||
|
||||
- **`check:`** — observably testable on the wire.
|
||||
- **clearly-excluded** — you're confident it can't be observed (e.g. "clients SHOULD also accept -32002" when the harness only drives servers).
|
||||
- **borderline** — you'd default to `check:` but observability is questionable. Markers:
|
||||
- _Internal state_ — verbs like _record_, _store_, _associate_, _track_, _cache_. The harness sees wire traffic, not memory; usually only observable via a downstream row already in your list.
|
||||
- _UI / human-facing_ — _display_, _show_, _render_, _prompt the user_.
|
||||
- _Precondition phrasing_ — "Before doing X, the implementation MUST Y" where X is itself another row.
|
||||
|
||||
Slug convention: lowercase-kebab, derived from the verb phrase. Examples from `sep-2164.yaml`: `no-empty-contents`, `error-code`. Same `id` is used for SUCCESS and FAILURE (`AGENTS.md:52`).
|
||||
|
||||
## Step 6: Confirm exclusions with the user
|
||||
|
||||
Nothing becomes `excluded:` without sign-off. Two rounds:
|
||||
|
||||
**Round 1 — clearly-excluded, single batch question.** One `AskUserQuestion` listing all clearly-excluded rows in the question body (slug + one-line reason each). Options:
|
||||
|
||||
- `Exclude all as listed (Recommended)`
|
||||
- `Flip all to check:`
|
||||
- `Let me adjust per-row` — if chosen, append these rows to round 2.
|
||||
|
||||
Skip this round if the bucket is empty.
|
||||
|
||||
**Round 2 — borderline, one question per row.** One `AskUserQuestion` call with a question per borderline row (loop in batches of 4 if needed). For each:
|
||||
|
||||
- header: the proposed slug
|
||||
- question: quote the requirement sentence + your one-line observability concern
|
||||
- options (list `check:` first — it's the default for borderline):
|
||||
- `check:` — keep as a testable check
|
||||
- `excluded: <reason>` — drop to excluded with your stated reason
|
||||
- `merge into <other-slug>` — offer when the row is a precondition for another row already in the list
|
||||
|
||||
Apply the answers before writing. For any `excluded:` outcome, write the reason verbatim into the yaml and add an `issue:` URL if the user supplies one. A `merge` outcome means: drop this row, and append its `text:` to the surviving row's `text:` separated by `/` so the traceability isn't lost.
|
||||
|
||||
## Step 7: Rewrite the YAML
|
||||
|
||||
Replace the two TODO rows the CLI generated with one row per extracted requirement. Preserve the CLI's quoting style (single quotes, two-space indent — see `src/seps/sep-2164.yaml`).
|
||||
|
||||
**Key order within each row** — for `check:` rows the **`check:` key comes first**, then `text:`, then any optional `url:`. Scanning the left margin should reveal every check ID without reading the quoted sentences. For `excluded:` rows the order is **`text:` first**, then `excluded:`, then optional `issue:` — there's no ID to scan for, so lead with the requirement.
|
||||
|
||||
**Row order in the file** — all `check:` rows first (in spec-diff order), then **all `excluded:` rows grouped at the bottom**, separated from the checks by **one blank line**. Do not interleave.
|
||||
|
||||
```yaml
|
||||
requirements:
|
||||
- check: sep-NNNN-first-slug
|
||||
text: '...'
|
||||
- check: sep-NNNN-second-slug
|
||||
text: '...'
|
||||
|
||||
- text: '...'
|
||||
excluded: 'reason'
|
||||
issue: https://github.com/modelcontextprotocol/conformance/issues/<NNNN>
|
||||
```
|
||||
|
||||
If a requirement is ambiguous or you're not confident, leave it as a `TODO:` row rather than guessing — humans review this yaml before scenarios get written.
|
||||
|
||||
Also fix the `spec_url`: the CLI emits the page URL with no anchor. If the requirements you extracted live under a specific spec subsection (e.g. `#error-handling`), append it.
|
||||
|
||||
If a requirement comes from a **different spec page** than `spec_url` (the SEP touched multiple `.mdx` files — the CLI prints these as "PR also changes N other spec file(s)"), give that row a full `url:` override:
|
||||
|
||||
```yaml
|
||||
- check: sep-NNNN-slug
|
||||
text: '...'
|
||||
url: https://modelcontextprotocol.io/specification/draft/other/page#anchor
|
||||
```
|
||||
|
||||
A row's effective spec reference is `row.url ?? file.spec_url`.
|
||||
|
||||
Write the result back to `$YAML`.
|
||||
|
||||
## Step 8: Suggest a host scenario
|
||||
|
||||
`AGENTS.md` prefers **fewer scenarios with more checks** over one-scenario-per-check. Before telling the user to write a new scenario, look for an existing one the new checks could be folded into.
|
||||
|
||||
Determine the suite directory from the requirement subjects ("MCP clients MUST…" → `client/`, "Servers MUST…" → `server/`, "authorization servers MUST…" → `authorization-server/`; a SEP may map to more than one). Then search that directory for scenarios touching the same spec area:
|
||||
|
||||
```bash
|
||||
rg -l -i '<domain-term>|<domain-term-2>' src/scenarios/<suite>/ --type ts
|
||||
```
|
||||
|
||||
Pick 2–3 domain terms from the SEP's subject matter (for a discovery SEP: `metadata`, `well-known`; for an auth-response SEP: `redirect`, `callback`, `pkce`). For each hit, pull the scenario's `name`/`description` to confirm relevance:
|
||||
|
||||
```bash
|
||||
rg -A1 'name:|description:' <hit.ts>
|
||||
```
|
||||
|
||||
If you find a plausible host, recommend it by path. If nothing fits, say so explicitly — a new scenario file is then the right call.
|
||||
|
||||
## Step 9: Hand-off
|
||||
|
||||
Report to the user, in this order:
|
||||
|
||||
1. Path to the generated yaml.
|
||||
2. Row counts: "`N check:` rows, `M excluded:` rows" — and note which exclusions the user signed off in Step 6.
|
||||
3. Any requirements you left as `TODO:` and why.
|
||||
4. **Host-scenario recommendation** from Step 8 — either "consider adding these checks to `src/scenarios/<suite>/<file>.ts` (it already exercises _X_)" or "no existing scenario covers this area; a new file is appropriate".
|
||||
5. Remaining next steps the user owns:
|
||||
- add the checks to the host scenario (or create one) under `src/scenarios/{client,server,authorization-server}/`,
|
||||
- register any new scenario in `src/scenarios/index.ts` (`AGENTS.md:48`),
|
||||
- add a passing example to the everything-client/server and a negative test, per `AGENTS.md:74-81`.
|
||||
|
||||
Do **not** generate or edit scenario `.ts` files or touch `src/scenarios/index.ts`. The skill's scope ends at the yaml plus the recommendation.
|
||||
|
|
@ -1,53 +0,0 @@
|
|||
---
|
||||
name: review-scenario
|
||||
description: Use when reviewing a conformance PR that adds or changes scenario .ts files for a SEP — before approving, before requesting changes, or as a self-check before opening one.
|
||||
---
|
||||
|
||||
# review-scenario
|
||||
|
||||
## What to check
|
||||
|
||||
**Spec diff is ground truth.** Pull the SEP's actual spec changes and read the RFC-2119 sentences yourself — don't trust the PR description or SEP summary for keyword levels:
|
||||
|
||||
```bash
|
||||
gh api "repos/modelcontextprotocol/modelcontextprotocol/pulls/<SEP>/files" \
|
||||
--jq '.[] | select(.filename | test("^docs/specification/draft/.*\\.mdx$")) | {filename, patch}'
|
||||
```
|
||||
|
||||
If the SEP includes a conformance-test-case table, that table is authoritative for the cases it lists. A table/prose mismatch is a spec gap to flag, not something to silently resolve either way.
|
||||
|
||||
**Traceability YAML.** `src/seps/sep-<SEP>.yaml` should exist (run `/new-sep <SEP>` first if not). Diff its rows against the spec sentences you extracted; flag rows that paraphrase rather than quote, claim a keyword level the spec doesn't, or assert something the spec never says. Check IDs follow `sep-<NNNN>-<kebab-slug>`.
|
||||
|
||||
**Per-scenario-file:**
|
||||
|
||||
- **Spec backing** — would a fully spec-compliant implementation FAIL this check? If yes — or if two compliant SDKs in different languages would get different results — the spec hasn't pinned the behavior; note it as a gap rather than enforce it.
|
||||
- **Dead checks** — emits FAILURE with no reachable SUCCESS counterpart, or sits behind an always-false guard.
|
||||
- **Logic** — does a missing/malformed input silently pass? Does the assertion distinguish "rejected for the right reason" from "rejected at all"?
|
||||
|
||||
**Coverage.** Count YAML `check:` rows vs how many the PR's scenarios actually exercise; list the gaps.
|
||||
|
||||
**Proof it runs.** The PR should reference at least one real implementation the scenario ran green against — the in-repo everything-client/server, or an external SDK via `npx https://pkg.pr.new/@modelcontextprotocol/conformance@<PR>`. No run referenced → ask for one before approving.
|
||||
|
||||
**Negative test.** Pins the specific failing slugs, not just `failures.length > 0` (AGENTS.md §Examples: prove it passes and fails).
|
||||
|
||||
## Output
|
||||
|
||||
This is a first pass for a human reviewer — give them what they need to verify each finding without re-deriving it.
|
||||
|
||||
**Open with a summary:** N scenarios added/changed, M distinct check IDs emitted, X/Y YAML `check:` rows covered, and which implementation it was run against.
|
||||
|
||||
**Then one bullet per finding.** Each bullet makes its own case — the reviewer should be able to confirm or refute it from the bullet alone:
|
||||
|
||||
> **`<check-id>`** — [`file.ts:Lnn`](https://github.com/modelcontextprotocol/conformance/blob/<HEAD-SHA>/path/file.ts#Lnn) — claim. Spec: _"quoted normative sentence"_ ([page#anchor](https://modelcontextprotocol.io/specification/draft/...#anchor)). Consequence: what a compliant impl would do and how this check would mis-report it.
|
||||
|
||||
e.g.
|
||||
|
||||
> **`client-consistent-version`** — [`stateless.ts:86`](…/blob/abc123/src/scenarios/client/stateless.ts#L86) — no spec backing. Spec: _"Servers MUST NOT rely on prior requests over the same connection to establish context (e.g., capabilities, protocol version)"_ ([lifecycle#stateless](…)). A compliant client may change `protocolVersion` per request; this check FAILs it. The `flippingVersionClient` negative test enforces a non-requirement.
|
||||
|
||||
Get `<HEAD-SHA>` once via `gh pr view <PR> --json headRefOid -q .headRefOid` and use it for all permalinks so they don't drift on rebase.
|
||||
|
||||
Order: spec-backing → logic/dead → coverage (gap list) → conventions. Put spec gaps in a separate trailing list — those go upstream, not to the PR author.
|
||||
|
||||
**Self-review:** fix in place and re-run.
|
||||
|
||||
**If asked to push fixes** (stacked diff on top of the PR head): one commit per finding, commit message is the finding. Leave design-level items (scenario count, refactors) as prose.
|
||||
3
.gitattributes
vendored
3
.gitattributes
vendored
|
|
@ -1,3 +0,0 @@
|
|||
# Enforce LF line endings for all text files
|
||||
* text=auto eol=lf
|
||||
|
||||
3
.github/CODEOWNERS
vendored
3
.github/CODEOWNERS
vendored
|
|
@ -1,3 +0,0 @@
|
|||
# Authorization server conformance area — team managed in modelcontextprotocol/access
|
||||
/src/scenarios/authorization-server/ @modelcontextprotocol/conformance-auth
|
||||
/src/runner/authorization-server.ts @modelcontextprotocol/conformance-auth
|
||||
27
.github/workflows/ci.yml
vendored
27
.github/workflows/ci.yml
vendored
|
|
@ -5,12 +5,6 @@ on:
|
|||
pull_request:
|
||||
release:
|
||||
types: [published]
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
publish_alpha:
|
||||
description: 'Publish to npm as alpha prerelease. If unchecked, only tests run (no publish). Version must already be bumped in package.json on the selected branch.'
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
|
@ -24,8 +18,8 @@ jobs:
|
|||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- uses: actions/setup-node@v6
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 24
|
||||
cache: npm
|
||||
|
|
@ -37,7 +31,7 @@ jobs:
|
|||
|
||||
publish:
|
||||
runs-on: ubuntu-latest
|
||||
if: github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.publish_alpha == 'true')
|
||||
if: github.event_name == 'release'
|
||||
environment: release
|
||||
needs: [test]
|
||||
|
||||
|
|
@ -46,19 +40,14 @@ jobs:
|
|||
id-token: write
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- uses: actions/setup-node@v6
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 24
|
||||
cache: npm
|
||||
registry-url: 'https://registry.npmjs.org'
|
||||
|
||||
- run: npm ci
|
||||
|
||||
- name: Publish to npm
|
||||
run: |
|
||||
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
|
||||
npm publish --access public --tag alpha
|
||||
else
|
||||
npm publish --access public
|
||||
fi
|
||||
- run: npm publish --provenance --access public
|
||||
env:
|
||||
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
|
||||
|
|
|
|||
6
.github/workflows/pr-publish.yml
vendored
6
.github/workflows/pr-publish.yml
vendored
|
|
@ -14,8 +14,8 @@ jobs:
|
|||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- uses: actions/setup-node@v6
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 24
|
||||
cache: npm
|
||||
|
|
@ -24,4 +24,4 @@ jobs:
|
|||
- name: Build
|
||||
run: npm run build
|
||||
- name: Publish
|
||||
run: npx pkg-pr-new publish --bin
|
||||
run: npx pkg-pr-new publish
|
||||
|
|
|
|||
130
.github/workflows/traceability.yml
vendored
130
.github/workflows/traceability.yml
vendored
|
|
@ -1,130 +0,0 @@
|
|||
name: Refresh SEP traceability manifest
|
||||
|
||||
# Regenerates src/seps/traceability.json by running the conformance suite against
|
||||
# the reference SDK and recording which check IDs were emitted, then opens a PR
|
||||
# with the diff. NOT a PR gate — runs on demand / on a schedule and proposes an
|
||||
# update for review. plan.modelcontextprotocol.io reads the committed file from
|
||||
# main.
|
||||
#
|
||||
# Depends on the `conformance sdk` subcommand (#277), which clones+builds the SDK
|
||||
# and runs the client+server suites. The `run` job executes third-party SDK code,
|
||||
# so it has NO repo write token (read-only perms, persist-credentials: false) and
|
||||
# only uploads results as an artifact; the separate `propose` job holds the
|
||||
# write/PR permissions and never executes SDK code.
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
sdk:
|
||||
description: 'SDK ref to run against (e.g. typescript-sdk@<sha>)'
|
||||
default: 'typescript-sdk@main'
|
||||
schedule:
|
||||
- cron: '0 6 * * 1' # Weekly, Monday 06:00 UTC.
|
||||
|
||||
concurrency:
|
||||
group: traceability-refresh
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
run:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
env:
|
||||
SDK_REF: ${{ inputs.sdk || 'typescript-sdk@main' }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
persist-credentials: false # no git token while SDK code runs
|
||||
|
||||
- uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version: 24
|
||||
cache: npm
|
||||
|
||||
# The SDK's own build (e.g. typescript-sdk uses `pnpm install && pnpm run
|
||||
# build:all`) needs pnpm on PATH; corepack provides it.
|
||||
- run: corepack enable
|
||||
|
||||
- run: npm ci
|
||||
- run: npm run build
|
||||
|
||||
- name: Run conformance suites against the reference SDK
|
||||
# `sdk` requires --mode client|server; run both into the same results dir
|
||||
# (the second reuses the cached checkout + build via --skip-build).
|
||||
# `|| true`: the manifest only needs the emitted check IDs (written
|
||||
# regardless of pass/fail), so SDK conformance failures must not fail
|
||||
# this step. The "no results produced" guard below is the real safety net.
|
||||
run: |
|
||||
node dist/index.js sdk "$SDK_REF" --mode client --suite all -o results || true
|
||||
node dist/index.js sdk "$SDK_REF" --mode server --suite all --skip-build -o results || true
|
||||
|
||||
- name: Fail if no results were produced
|
||||
run: |
|
||||
if [ -z "$(find results -name checks.json -print -quit 2>/dev/null)" ]; then
|
||||
echo "No checks.json produced — the suite run failed; not proposing a manifest."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: conformance-results
|
||||
path: results
|
||||
retention-days: 7
|
||||
|
||||
propose:
|
||||
needs: run
|
||||
runs-on: ubuntu-latest
|
||||
# Requires the repo/org setting "Allow GitHub Actions to create and approve
|
||||
# pull requests" to be enabled, otherwise `gh pr create` fails.
|
||||
permissions:
|
||||
contents: write
|
||||
pull-requests: write
|
||||
env:
|
||||
SDK_REF: ${{ inputs.sdk || 'typescript-sdk@main' }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version: 24
|
||||
cache: npm
|
||||
- run: npm ci
|
||||
- run: npm run build
|
||||
|
||||
- uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: conformance-results
|
||||
path: results
|
||||
|
||||
- name: Regenerate manifest
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# Record the resolved sha (stable per SDK commit) so the manifest's
|
||||
# `source` only changes when the SDK actually advances — no per-run noise.
|
||||
ref="${SDK_REF#*@}"
|
||||
sha="$(git ls-remote https://github.com/modelcontextprotocol/typescript-sdk.git "$ref" | cut -f1)"
|
||||
node dist/index.js traceability --results results \
|
||||
--source "typescript-sdk@${sha:0:12}"
|
||||
|
||||
- name: Open/update the rolling refresh PR
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if git diff --quiet -- src/seps/traceability.json; then
|
||||
echo "traceability.json unchanged"
|
||||
exit 0
|
||||
fi
|
||||
# One rolling branch/PR, force-updated each run, so the schedule does
|
||||
# not accrue a new PR every week.
|
||||
branch="traceability-refresh"
|
||||
git config user.name 'github-actions[bot]'
|
||||
git config user.email 'github-actions[bot]@users.noreply.github.com'
|
||||
git checkout -B "$branch"
|
||||
git add src/seps/traceability.json
|
||||
git commit -m "chore: refresh SEP traceability manifest ($SDK_REF)"
|
||||
git push --force origin "$branch"
|
||||
gh pr view "$branch" >/dev/null 2>&1 || gh pr create \
|
||||
--head "$branch" \
|
||||
--title 'chore: refresh SEP traceability manifest' \
|
||||
--body 'Automated refresh from a conformance run against the reference SDK. Review the coverage diff before merging.'
|
||||
7
.gitignore
vendored
7
.gitignore
vendored
|
|
@ -1,9 +1,4 @@
|
|||
node_modules
|
||||
results/
|
||||
lefthook-local.yml
|
||||
dist/
|
||||
.vscode/
|
||||
.idea/
|
||||
.claude/settings.local.json
|
||||
.sdk-under-test/
|
||||
.sync-schema-tmp/
|
||||
dist
|
||||
|
|
@ -1,15 +0,0 @@
|
|||
# Generated by `conformance traceability` — formatting is owned by the
|
||||
# generator (deterministic JSON.stringify), not Prettier. Without this, the
|
||||
# repo's `prettier --check .` would reformat the file and fight the generator's
|
||||
# output (and the refresh workflow's `git diff` check).
|
||||
src/seps/traceability.json
|
||||
|
||||
# Vendored verbatim from modelcontextprotocol/schema/{version}/schema.ts and
|
||||
# schema.json via `npm run sync-schema`. Keep byte-identical with upstream so
|
||||
# the SOURCE pin is meaningful and re-syncing produces a clean diff.
|
||||
src/spec-types/*.ts
|
||||
src/spec-types/*.schema.json
|
||||
|
||||
# Local tooling workspaces (not part of the repo).
|
||||
.claude/
|
||||
.sdk-under-test/
|
||||
|
|
@ -1,5 +1,4 @@
|
|||
{
|
||||
"endOfLine": "lf",
|
||||
"singleQuote": true,
|
||||
"trailingComma": "none",
|
||||
"overrides": [
|
||||
|
|
|
|||
126
AGENTS.md
126
AGENTS.md
|
|
@ -1,126 +0,0 @@
|
|||
# AGENTS.md
|
||||
|
||||
Guidance for AI agents (and humans) contributing to the MCP conformance test framework.
|
||||
|
||||
## What this repo is
|
||||
|
||||
A test harness that exercises MCP SDK implementations against the protocol spec. The coverage number that matters here is **spec coverage** — how much of the protocol the scenarios test.
|
||||
|
||||
Uses **npm** (not pnpm/yarn). Don't commit `pnpm-lock.yaml` or `yarn.lock`.
|
||||
|
||||
## Where to start
|
||||
|
||||
**Open an issue first** — whether you've hit a bug in the harness or want to propose a new scenario. For scenarios, sketch which part of the spec you want to cover and roughly how; for bugs, include the command you ran and the output. Either way, a short discussion up front beats review churn on a PR that overlaps existing work or heads in a direction we're not going.
|
||||
|
||||
**Don't point an agent at the repo and ask it to "find bugs."** Generic bug-hunting on a test harness produces low-signal PRs (typo fixes, unused-variable cleanups, speculative refactors). If you want to contribute via an agent, give it a concrete target:
|
||||
|
||||
- Pick a specific MUST or SHOULD from the [MCP spec](https://modelcontextprotocol.io/specification/) that has no scenario yet, and ask the agent to draft one.
|
||||
- Pick an [open issue](https://github.com/modelcontextprotocol/conformance/issues) and work on that.
|
||||
|
||||
The valuable contribution here is **spec coverage**, not harness polish.
|
||||
|
||||
## Scenario design: fewer scenarios, more checks
|
||||
|
||||
**The strongest rule in this repo:** prefer one scenario with many checks over many scenarios with one check each.
|
||||
|
||||
Why:
|
||||
|
||||
- Each scenario often spins up its own HTTP server. These suites run in CI on every push for every SDK, so per-scenario overhead multiplies fast.
|
||||
- Less code to maintain and update when the spec shifts.
|
||||
- Progress on making an SDK better shows up as "pass 7/10 checks" rather than "pass 1 test, fail another" — finer-grained signal from the same run.
|
||||
|
||||
### Granularity heuristic
|
||||
|
||||
Ask: **"Would it make sense for someone to implement a server/client that does just this scenario?"**
|
||||
|
||||
If two scenarios would always be implemented together, merge them. Examples:
|
||||
|
||||
- `tools/list` + a simple `tools/call` → one scenario
|
||||
- All content-type variants (image, audio, mixed, resource) → one scenario
|
||||
- Full OAuth flow with token refresh → one scenario, not separate "basic" + "refresh" scenarios. A client that passes "basic" but not "refresh" just shows up as passing N−2 checks.
|
||||
|
||||
Keep scenarios separate when they're genuinely independent features or when they're mutually exclusive (e.g., an SDK should support writing a server that _doesn't_ implement certain stateful features).
|
||||
|
||||
### When a PR adds scenarios
|
||||
|
||||
- Start with **one end-to-end scenario** covering the happy path with many checks along the way.
|
||||
- Don't add "step 1 only" and "step 1+2" as separate scenarios — the second subsumes the first.
|
||||
- Register the scenario in the appropriate suite list in `src/scenarios/index.ts` (`core`, `extensions`, `backcompat`, etc.).
|
||||
|
||||
## Check conventions
|
||||
|
||||
- **Same `id` for SUCCESS and FAIL.** A check should use one slug and flip `status` + `errorMessage`, not branch into `foo-success` vs `foo-failure` slugs.
|
||||
- **Optimize for Ctrl+F on the slug.** Repetitive check blocks are fine — easier to find the failing one than to unwind a clever helper.
|
||||
- Reuse `ConformanceCheck` and other types from `src/types.ts` rather than defining parallel shapes.
|
||||
- **Don't reimplement the runner.** New subcommands that need to "select scenarios → run them → print summary → compute exit code" must go through the existing `client` / `server` commands (subprocess via `process.execPath` like `tier-check` and `sdk` do) or call shared helpers — never a parallel suite-map / summary loop.
|
||||
- Include `specReferences` pointing to the relevant spec section.
|
||||
- **Severity follows the spec keyword:** MUST / MUST NOT → `FAILURE`; SHOULD / SHOULD NOT → `WARNING`. (CI treats WARNING as a failure, so Tier-1 SDKs still need to satisfy SHOULDs — see #245.)
|
||||
- **A missing prerequisite is a failure, not a skip** (#248). When a check cannot be exercised — the server under test lacks a diagnostic fixture tool, rejects the probe, or advertises a feature it does not serve — report it failed via `notTestable()` / `untestableCheck()` from `src/scenarios/untestable.ts`, with the missing prerequisite named in `errorMessage` and `details.untestable: true`. SKIPPED reads as green in every consumer (pass counts, exit codes, expected-failures), so it is reserved for checks that are legitimately not applicable: an optional capability the server never declared, spec-version inapplicability handled by the runner, or a documented harness gap the suite itself tracks (e.g. `tasks-status-notifications` pending its subscriptions/listen rewrite). The untestable severity follows the underlying requirement's keyword (MUST → `FAILURE`, SHOULD → `WARNING`); the expected-failures baseline is the escape hatch for SDKs that haven't built the fixture yet.
|
||||
|
||||
## Descriptions and wording
|
||||
|
||||
Be precise about what's **required** vs **optional**. A scenario description that tests optional behavior should make that clear — e.g. "Tests that a client _that wants a refresh token_ handles offline_access scope…" not "Tests that a client handles offline_access scope…". Don't accidentally promote a MAY/SHOULD to a MUST in the prose.
|
||||
|
||||
When in doubt about spec details (OAuth parameters, audiences, grant types), check the actual spec in `modelcontextprotocol` rather than guessing.
|
||||
|
||||
## Reviewing PRs
|
||||
|
||||
### SEP scenarios
|
||||
|
||||
Verify requirement levels against the SEP's **spec diff** — the change to `docs/specification/draft/` in the SEP's PR — not the SEP markdown summary or the conformance PR's description. The keyword that governs check severity is the one in the spec text; a bullet under a "Servers SHOULD…" sentence is SHOULD-level even if the SEP's title says "standardize."
|
||||
|
||||
```sh
|
||||
gh api "repos/modelcontextprotocol/modelcontextprotocol/contents/docs/specification/draft/<path>?ref=<sep-branch>" --jq '.content' | base64 -d
|
||||
```
|
||||
|
||||
### Adding a new SEP
|
||||
|
||||
Scaffold the requirement-traceability YAML with:
|
||||
|
||||
```sh
|
||||
npx @modelcontextprotocol/conformance new-sep <NNNN>
|
||||
```
|
||||
|
||||
The command looks up PR #`<NNNN>` in `modelcontextprotocol/modelcontextprotocol` (SEP numbers are PR numbers), derives `spec_url` from the `docs/specification/draft/*.mdx` file it changes, and writes `src/seps/sep-<NNNN>.yaml` with TODO `requirements[]` rows. Use `--spec-path` or `--spec-url` to skip the lookup. The `new-sep` Claude Code skill drives the same flow end-to-end, parses the spec diff, and fills in the requirement rows.
|
||||
|
||||
### Traceability manifest
|
||||
|
||||
`src/seps/traceability.json` is a generated map of, per SEP, which declared `check:` IDs are actually emitted when the conformance suite runs against the reference SDK. It is consumed by plan.modelcontextprotocol.io to track SEP-2484 progress.
|
||||
|
||||
The emitted check IDs come from a real suite run (not a source scan), so dynamic (template-literal) IDs resolve to their concrete values. Generate the manifest from a results directory:
|
||||
|
||||
```sh
|
||||
# 1. Run the suite against the reference SDK, collecting checks.json files:
|
||||
node dist/index.js client --command '<sdk conformance client>' --suite all -o results
|
||||
node dist/index.js server --url '<sdk conformance server url>' --suite all -o results
|
||||
# 2. Build the manifest from those results:
|
||||
npm run traceability -- --results results
|
||||
npm run traceability -- --results results --strict # exit 1 on any untested (advisory)
|
||||
```
|
||||
|
||||
Manifest shape: `{ schemaVersion, docs, source, seps }`, where `seps` is keyed by SEP number. Each requirement is `tested` (its check ID was emitted) or `untested` (declared but never emitted — a real gap, or a check that only fires against a deliberately-broken impl, i.e. it needs a negative test). `"tested" means a scenario emitted the check ID, NOT that any SDK passes it` — per-SDK results live in `tier-check`. Matching is exact, so a scenario's emitted check IDs must match the requirement slugs in the yaml (one check ID per MUST/SHOULD, emitted once per case). `source` records what was run against (e.g. `typescript-sdk@<sha>`); the `docs` field points back here.
|
||||
|
||||
Contract for consumers (plan.mcp.io): a SEP appears only if it has a traceability yaml or emits `sep-NNNN-*` check IDs. **A SEP absent from the manifest has no conformance artifacts — treat it as not-started** (diff against your own SEP list to find them). `untracked` lists emitted IDs with no yaml row (usually scenario gates).
|
||||
|
||||
The manifest is refreshed by `.github/workflows/traceability.yml` (manual/scheduled), which runs the suite against typescript-sdk and opens a PR with the diff — it is **not** a PR gate. Untested checks are advisory for now; the intended future policy is that an untested check must be backed by a negative test.
|
||||
|
||||
## Examples: prove it passes and fails
|
||||
|
||||
A new scenario should come with:
|
||||
|
||||
1. **A passing example** — usually by extending `examples/clients/typescript/everything-client.ts` or the everything-server, not a new file.
|
||||
2. **A negative test** — a deliberately-broken implementation in `examples/{clients,servers}/typescript/` plus a vitest case asserting the check emits `FAILURE`/`WARNING` against it. See `src/scenarios/client/auth/index.test.ts` and `src/scenarios/server/negative.test.ts` for the pattern. A passing run against the everything-server proves the check doesn't false-positive, but not that it catches anything.
|
||||
|
||||
Delete unused example scenarios. If a scenario key in the everything-client has no corresponding test, remove it.
|
||||
|
||||
## Don't add new ways to run tests
|
||||
|
||||
Use the existing CLI runner (`npx @modelcontextprotocol/conformance client|server ...`). If you need a feature the runner doesn't have, add it to the runner rather than building a parallel entry point.
|
||||
|
||||
## Before opening a PR
|
||||
|
||||
- `npm run build` passes
|
||||
- `npm test` passes
|
||||
- For non-trivial scenario changes, run against at least one real SDK (typescript-sdk or python-sdk) to see actual output. For changes to shared infrastructure (runner, tier-check), test against go-sdk or csharp-sdk too.
|
||||
- Scenario is registered in the right suite in `src/scenarios/index.ts`
|
||||
- If you changed a `sep-*.yaml` or scenario check IDs, `src/seps/traceability.json` will drift; the traceability workflow refreshes it via PR (or regenerate locally with `--results` from a suite run)
|
||||
|
|
@ -1,71 +0,0 @@
|
|||
# Contributing
|
||||
|
||||
Thanks for helping improve the MCP conformance suite!
|
||||
|
||||
The most valuable contributions are **new conformance scenarios** that cover under-tested parts of the [MCP spec](https://modelcontextprotocol.io/specification/). If you're not sure where to start, ask in `#conformance-testing-wg` on the MCP Contributors Discord.
|
||||
|
||||
## Before you start
|
||||
|
||||
**Open an issue first** — whether you've found a bug or want to propose a new scenario. A short discussion up front saves everyone time on PRs that overlap existing work or head in a direction we're not going.
|
||||
|
||||
Then read **[AGENTS.md](./AGENTS.md)** — it's the design guide for scenarios and checks. The short version:
|
||||
|
||||
- **Fewer scenarios, more checks.** Each scenario spins up its own server and runs in CI for every SDK. One scenario with 10 checks beats 10 scenarios with one check each.
|
||||
- **Prove it passes and fails.** Extend the existing everything-client/server to pass your scenario, and show (or include) a failing case.
|
||||
- **Reuse the CLI runner.** Don't add parallel entry points.
|
||||
|
||||
If you're using an AI agent to help, please **don't** point it at the repo with a generic "find bugs" prompt — give it a specific MUST from the spec or an open issue to work on. See AGENTS.md for details.
|
||||
|
||||
## Setup
|
||||
|
||||
```sh
|
||||
npm install
|
||||
npm run build
|
||||
npm test
|
||||
```
|
||||
|
||||
This repo uses **npm** — don't commit `pnpm-lock.yaml` or `yarn.lock`.
|
||||
|
||||
## Running your scenario
|
||||
|
||||
Run a client scenario against the bundled example client:
|
||||
|
||||
```sh
|
||||
npm run build
|
||||
node dist/index.js client --command "tsx examples/clients/typescript/everything-client.ts" --scenario <your-scenario>
|
||||
```
|
||||
|
||||
Run a server scenario against any server running locally (replace 3000 with your server's port, if necessary):
|
||||
|
||||
```sh
|
||||
# Against a server
|
||||
node dist/index.js server --url http://localhost:3000/mcp --scenario <your-scenario>
|
||||
```
|
||||
|
||||
Run a server scenario against the latest TypeScript SDK:
|
||||
|
||||
```sh
|
||||
# In another directory, clone and prepare the SDK once
|
||||
cd ..
|
||||
git clone https://github.com/modelcontextprotocol/typescript-sdk.git
|
||||
cd typescript-sdk
|
||||
pnpm install
|
||||
pnpm run build:all
|
||||
|
||||
# Start the SDK's conformance server in one terminal
|
||||
cd test/conformance
|
||||
PORT=3100 pnpm run test:conformance:server:run
|
||||
|
||||
# Back in this repo, run your local conformance build against it
|
||||
cd /path/to/conformance
|
||||
npm run build
|
||||
node dist/index.js server --url http://localhost:3100/mcp --scenario <your-scenario>
|
||||
```
|
||||
|
||||
See the [README](./README.md) for full CLI options and the [SDK Integration Guide](./SDK_INTEGRATION.md) for more on testing against a real SDK.
|
||||
|
||||
## Pull requests
|
||||
|
||||
- Register your scenario in the right suite in `src/scenarios/index.ts`
|
||||
- Run against at least one real SDK (see above) before opening the PR — we'll ask what the output looked like
|
||||
- Keep PRs focused; one feature or scenario group at a time
|
||||
199
LICENSE
199
LICENSE
|
|
@ -1,193 +1,6 @@
|
|||
The MCP project is undergoing a licensing transition from the MIT License to the Apache License, Version 2.0 ("Apache-2.0"). All new code and specification contributions to the project are licensed under Apache-2.0. Documentation contributions (excluding specifications) are licensed under CC-BY-4.0.
|
||||
|
||||
Contributions for which relicensing consent has been obtained are licensed under Apache-2.0. Contributions made by authors who originally licensed their work under the MIT License and who have not yet granted explicit permission to relicense remain licensed under the MIT License.
|
||||
|
||||
No rights beyond those granted by the applicable original license are conveyed for such contributions.
|
||||
|
||||
---
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to the Licensor for inclusion in the Work by the copyright
|
||||
owner or by an individual or Legal Entity authorized to submit on behalf
|
||||
of the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
---
|
||||
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2024-2025 Model Context Protocol a Series of LF Projects, LLC.
|
||||
Copyright (c) 2024 Anthropic, PBC
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
|
|
@ -205,12 +18,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
||||
---
|
||||
|
||||
Creative Commons Attribution 4.0 International (CC-BY-4.0)
|
||||
|
||||
Documentation in this project (excluding specifications) is licensed under
|
||||
CC-BY-4.0. See https://creativecommons.org/licenses/by/4.0/legalcode for
|
||||
the full license text.
|
||||
SOFTWARE.
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue