modelcontextprotocol/conformance
Watch
1
0
Fork
You've already forked conformance
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat: SDK tier assessment CLI and skill (#142)

Browse source 
* feat: add tier-check CLI for SDK tier assessment

Adds a 'tier-check' subcommand to the conformance tool that automates
SDK tier assessment against SEP-1730 criteria.

Checks performed:
- Conformance test pass rate (via everything-server)
- GitHub label taxonomy (priority/status/area labels)
- Issue triage SLA compliance
- P0 bug resolution tracking
- Stable release detection
- Required file existence (CHANGELOG, SECURITY, etc.)
- Spec tracking (SDK release within 30d of spec release)

Also includes a Claude Code skill (skills/mcp-sdk-tier-audit/) for
judgment-based checks that require codebase analysis (feature coverage,
docs quality, policy evaluation).

Usage:
  npx tsx src/index.ts tier-check --repo modelcontextprotocol/typescript-sdk
  npx tsx src/index.ts tier-check --repo ... --conformance-server-cmd '...' \
    --conformance-server-cwd ... --conformance-server-url ... --output json

* refactor: revise tier-check CLI and skill based on review feedback

- Move skill to .claude/skills/ so it's auto-available in Claude Code
- Remove feature-coverage subagent (redundant with conformance tests)
- Remove hardcoded ~/src/mcp paths from all skill files
- Trim conformance server table to TS + Python only
- Rename file_existence check to policy_signals (informational, not blocking)
- Add GitHub native issue types detection to labels check
- Add missing features to docs-coverage checklist (tasks, elicitation URL mode, JSON Schema 2020-12)
- Add README with CLI quick start and escape hatch for non-Claude-Code users
- Use --limit 500 instead of --limit 100 for gh issue list

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address review feedback on skill and README

- Use npx @modelcontextprotocol/conformance instead of node dist/index.js
- Add full GitHub auth instructions (gh auth login, GITHUB_TOKEN, --token)
- Point TS SDK conformance server to typescript-sdk/test/conformance/
- Fix Python SDK URL to localhost:3001/mcp (not TBD)
- Remove manual gh issue list / gh release list from SKILL.md (CLI handles it)
- Remove Claude Code-specific subagent_type references
- Assume user is already in conformance repo
- Clean up policy-evaluation-prompt.md: remove redundant grep commands,
  focus on content evaluation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: resolve lint errors in conformance.ts

Remove unused variable assignments flagged by eslint.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style: apply prettier formatting

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: add npm run tier-check script, update docs with examples

- Add "tier-check" npm script so users can run `npm run tier-check --`
  instead of `node dist/index.js tier-check`
- Update SKILL.md, README, and skill README to use npm run tier-check
- Add full conformance examples with --conformance-server-cmd/cwd/url
  flags and realistic paths (~/src/mcp/typescript-sdk)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style: fix prettier formatting in SKILL.md

* fix: add per-scenario timeout, support url-only conformance, fix stdout pollution

- Add 30s per-scenario timeout to prevent tier-check from hanging
- Allow --conformance-server-url without --conformance-server-cmd (server already running)
- Move runner status logs to stderr so --output json produces clean JSON
- Update SKILL.md and README with --silent flag and pre-start server workflow

* refactor: skill takes local path + server URL instead of repo name

The skill now requires two arguments:
1. Local path to the SDK checkout (for direct file inspection)
2. URL where the everything server is already running

The GitHub owner/repo is derived from git remote. This eliminates
cloning, server startup complexity, and branch confusion (v1.x vs main).

* refactor: write detailed reports to files, show concise summary

Reports go to results/tier-audits/<sdk>-<date>/ (already gitignored).
Claude's console output is now just the tier classification,
pass/fail summary line, top 3 actions, and file paths.

* docs: update READMEs for new skill interface and pre-start workflow

* simplify: flat file output instead of nested directory

* fix: remediation always shows path to Tier 2 and Tier 1

* feat: add client conformance testing to tier-check CLI and skill

- Add checkClientConformance() that runs core client scenarios
  (initialize, tools_call, elicitation-defaults, sse-retry, auth)
  by spawning the SDK's conformance client via --client-cmd
- Add client_conformance to TierScorecard type
- Wire --client-cmd option into the CLI
- Update tier logic: both server + client conformance feed into
  Tier 1 (100%) and Tier 2 (>=80%) requirements
- Update terminal and markdown output to show both conformance types
- Update skill to auto-detect conformance client or accept explicit
  client-cmd argument
- Update README with new option and examples

* improve: table summary output, write reports via subagents

- Change executive summary from pipe-delimited line to a readable
  table with T2/T1 columns
- Move assessment and remediation file writing into parallel
  subagents to keep the main conversation thread clean

* improve: list tier gaps as numbered items instead of one-line blob

* improve: finalize summary format with separator, high-priority fixes, numbered gaps

* improve: add pre-flight checks for gh auth and server reachability

Fail fast with clear error messages if GitHub CLI is not authenticated
or if the conformance server URL is not reachable, rather than failing
deep into the scorecard run.

* docs: improve README and fix skill auto-detection paths

- Claude Code section: explain client-cmd auto-detection for TS/Python,
  show explicit 3-arg form for other SDKs, add examples for all three
- Fix TypeScript build command (npm run build, not pnpm build:all)
- Fix Python server command (add --port, use uv sync --package)
- Fix Python client path (.github/actions/conformance/client.py)
- Expand 'Other SDKs' section with guidance on everything server
- Add gh auth login prerequisite to Claude Code steps

* simplify: remove client-cmd auto-detection, require explicit argument

Client command is now always passed as the third argument. If omitted,
client conformance is skipped and noted as a gap. No more magic path
detection — clearer and more predictable.

* fix: align docs table with canonical list (48 features), simplify policy eval

Docs coverage:
- Table now has numbered rows matching all 48 non-experimental features
  from the canonical list (was missing 7: tools text/image/audio/embedded/
  error/notifications, protocol version negotiation)
- Hardcode total as 48 in summary so agents don't miscount

Policy evaluation:
- Simplified from deep content analysis to file-existence checks
- Dependency policy: DEPENDENCY_POLICY.md, dependabot.yml, or CONTRIBUTING.md section
- Roadmap: ROADMAP.md must exist (GitHub milestones alone not sufficient)
- Versioning: VERSIONING.md or CONTRIBUTING.md section
- Removed GitHub API calls for milestones and releases from policy eval

* refactor: extract canonical feature list into single source of truth

Create references/feature-list.md with all 48 non-experimental + 5
experimental features. The docs-coverage prompt now references this
file instead of duplicating the list. One place to update when
features change.

* fix: separate deterministic file checks from AI content evaluation

CLI (files.ts): now checks all policy files deterministically —
DEPENDENCY_POLICY.md, docs/dependency-policy.md, dependabot.yml,
renovate.json, ROADMAP.md, docs/roadmap.md, VERSIONING.md,
docs/versioning.md, BREAKING_CHANGES.md (in addition to existing
CHANGELOG.md, SECURITY.md, CONTRIBUTING.md).

AI policy eval: receives CLI output showing which files exist,
then reads ONLY those files to judge content quality. No longer
searches the repo for files — clean separation of concerns.

* style: apply prettier formatting

* revert: undo unrelated console.log change in runner/server.ts

* refactor: shell out to conformance CLI instead of reimplementing runner

Address PR feedback: conformance.ts was duplicating the normal conformance
running code. Now shells out to 'node dist/index.js server/client' with
-o to save results to a temp dir, then parses the checks.json files.

Also removes --conformance-server-cmd and --conformance-server-cwd options
since the server must be pre-started.

* docs: add Go and C# SDK examples to README and SKILL.md

* fix: add --framework net9.0 to C# server command

* rename: conformance.ts -> test-conformance-results.ts

Avoids confusion with src/runner/ (the actual conformance runner).
This file just invokes the CLI and parses output.

* style: prettier formatting

* fix: reconcile conformance results against full scenario list

The tier-check CLI was only counting scenarios that produced a
checks.json file. Scenarios that crashed or failed to run (e.g., auth
scenarios when OAuth is not implemented) were invisible, making the
denominator artificially small (e.g., 4/4 instead of 4/23).

Now both checkConformance and checkClientConformance reconcile their
parsed results against the known scenario lists, adding failure entries
for any expected scenario that didn't produce results.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: tighten documentation evaluation criteria

Clarify what counts as documented vs just having code:
- Conformance test servers don't count as docs or examples
- Examples without prose = PARTIAL, not PASS
- Go Example* test functions explicitly allowed
- Clear PASS/PARTIAL/FAIL verdict definitions

* docs: add Labels and Spec Tracking rows to audit report templates

The executive summary and assessment report were missing two SEP-1730
requirements: label taxonomy compliance and spec tracking (new protocol
features timeline).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: reuse ConformanceCheck type from src/types.ts instead of redefining

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Felix Weinberger 2026-02-12 17:59:06 +00:00 • committed by GitHub
commit 3b4a92baad
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
12 changed files with 3147 additions and 1139 deletions
Download patch file Download diff file Expand all files Collapse all files

4
src/index.ts
View file

@ -28,6 +28,7 @@ import {
evaluateBaseline,
printBaselineResults
} from './expected-failures';
import { createTierCheckCommand } from './tier-check';
import packageJson from '../package.json';
const program = new Command();
@ -383,6 +384,9 @@ program
}
});
// Tier check command
program.addCommand(createTierCheckCommand());
// List scenarios command
program
.command('list')

58
src/tier-check/checks/files.ts Normal file
View file

@ -0,0 +1,58 @@
import { Octokit } from '@octokit/rest';
import { PolicySignalsResult } from '../types';
// Policy files checked deterministically by the CLI.
// The AI policy evaluation then reads ONLY files that exist here
// to judge whether content is substantive — it does not search for
// files in other locations.
const POLICY_SIGNAL_FILES = [
// General project health
'CHANGELOG.md',
'SECURITY.md',
'CONTRIBUTING.md',
// Dependency update policy
'DEPENDENCY_POLICY.md',
'docs/dependency-policy.md',
'.github/dependabot.yml',
'.github/renovate.json',
'renovate.json',
// Roadmap
'ROADMAP.md',
'docs/roadmap.md',
// Versioning / breaking change policy
'VERSIONING.md',
'docs/versioning.md',
'BREAKING_CHANGES.md'
];
export async function checkPolicySignals(
octokit: Octokit,
owner: string,
repo: string,
branch?: string
): Promise<PolicySignalsResult> {
const files: Record<string, boolean> = {};
for (const filePath of POLICY_SIGNAL_FILES) {
try {
await octokit.repos.getContent({
owner,
repo,
path: filePath,
...(branch ? { ref: branch } : {})
});
files[filePath] = true;
} catch {
files[filePath] = false;
}
}
return {
status: Object.values(files).every((v) => v)
? 'pass'
: Object.values(files).some((v) => v)
? 'partial'
: 'fail',
files
};
}

72
src/tier-check/checks/labels.ts Normal file
View file

@ -0,0 +1,72 @@
import { Octokit } from '@octokit/rest';
import { LabelsResult } from '../types';
// Type labels can be satisfied by GitHub's native issue types (Bug, Enhancement, Question)
const TYPE_LABELS = ['bug', 'enhancement', 'question'];
const STATUS_LABELS = [
'needs confirmation',
'needs repro',
'ready for work',
'good first issue',
'help wanted'
];
const PRIORITY_LABELS = ['P0', 'P1', 'P2', 'P3'];
export async function checkLabels(
octokit: Octokit,
owner: string,
repo: string
): Promise<LabelsResult> {
const labels: string[] = [];
let page = 1;
while (true) {
const { data } = await octokit.issues.listLabelsForRepo({
owner,
repo,
per_page: 100,
page
});
labels.push(...data.map((l) => l.name));
if (data.length < 100) break;
page++;
}
// Check if the repo uses GitHub's native issue types
// If so, type labels (bug/enhancement/question) are satisfied
let usesIssueTypes = false;
try {
const { data: repoData } = await octokit.request(
'GET /repos/{owner}/{repo}',
{ owner, repo }
);
// Repos with issue types enabled have them configured at the org or repo level.
// We detect this by checking for the presence of issue type configuration.
// As a heuristic: if the repo has no type labels but has issues, it likely uses types.
usesIssueTypes = !!(repoData as Record<string, unknown>).issue_types;
} catch {
// If we can't determine, assume labels are needed
}
const labelSet = new Set(labels.map((l) => l.toLowerCase()));
// Build required labels list, excluding type labels if issue types are used
const requiredLabels = [
...(usesIssueTypes ? [] : TYPE_LABELS),
...STATUS_LABELS,
...PRIORITY_LABELS
];
const missing = requiredLabels.filter((l) => !labelSet.has(l.toLowerCase()));
const found = requiredLabels.filter((l) => labelSet.has(l.toLowerCase()));
return {
status: missing.length === 0 ? 'pass' : 'fail',
present: found.length,
required: requiredLabels.length,
missing,
found,
uses_issue_types: usesIssueTypes
};
}

87
src/tier-check/checks/p0.ts Normal file
View file

@ -0,0 +1,87 @@
import { Octokit } from '@octokit/rest';
import { P0Result } from '../types';
export async function checkP0Resolution(
octokit: Octokit,
owner: string,
repo: string
): Promise<P0Result> {
// Fetch all issues with P0 label
const p0Issues: Array<{
number: number;
title: string;
state: string;
created_at: string;
closed_at: string | null;
labels: Array<{ name: string }>;
}> = [];
let page = 1;
while (true) {
const { data } = await octokit.issues.listForRepo({
owner,
repo,
labels: 'P0',
state: 'all',
per_page: 100,
page
});
for (const issue of data) {
if (issue.pull_request) continue;
p0Issues.push({
number: issue.number,
title: issue.title,
state: issue.state,
created_at: issue.created_at,
closed_at: issue.closed_at ?? null,
labels: issue.labels.filter(
(l): l is { name: string } =>
typeof l === 'object' && l !== null && 'name' in l
)
});
}
if (data.length < 100) break;
page++;
}
const openP0s = p0Issues.filter((i) => i.state === 'open');
const closedP0s = p0Issues.filter((i) => i.state === 'closed' && i.closed_at);
let closedWithin7d = 0;
let closedWithin14d = 0;
for (const issue of closedP0s) {
const daysToClose =
(new Date(issue.closed_at!).getTime() -
new Date(issue.created_at).getTime()) /
(1000 * 60 * 60 * 24);
if (daysToClose <= 7) closedWithin7d++;
if (daysToClose <= 14) closedWithin14d++;
}
const openP0Details = openP0s.map((i) => ({
number: i.number,
title: i.title,
age_days: Math.round(
(Date.now() - new Date(i.created_at).getTime()) / (1000 * 60 * 60 * 24)
)
}));
const allResolved7d =
openP0s.length === 0 &&
(closedP0s.length === 0 || closedWithin7d === closedP0s.length);
const allResolved14d =
openP0s.length === 0 &&
(closedP0s.length === 0 || closedWithin14d === closedP0s.length);
return {
status: allResolved7d ? 'pass' : allResolved14d ? 'partial' : 'fail',
open_p0s: openP0s.length,
open_p0_details: openP0Details,
closed_within_7d: closedWithin7d,
closed_within_14d: closedWithin14d,
closed_total: closedP0s.length,
all_p0s_resolved_within_7d: allResolved7d,
all_p0s_resolved_within_14d: allResolved14d
};
}

59
src/tier-check/checks/release.ts Normal file
View file

@ -0,0 +1,59 @@
import { Octokit } from '@octokit/rest';
import { ReleaseResult } from '../types';
export async function checkStableRelease(
octokit: Octokit,
owner: string,
repo: string
): Promise<ReleaseResult> {
try {
const { data: releases } = await octokit.repos.listReleases({
owner,
repo,
per_page: 20
});
if (releases.length === 0) {
return {
status: 'fail',
version: null,
is_stable: false,
is_prerelease: false
};
}
// Find latest non-draft release
const latest = releases.find((r) => !r.draft);
if (!latest) {
return {
status: 'fail',
version: null,
is_stable: false,
is_prerelease: false
};
}
const version = latest.tag_name.replace(/^v/, '');
const isPrerelease =
latest.prerelease ||
/-(alpha|beta|rc|dev|preview|snapshot)/i.test(version);
// Check if version is >= 1.0.0
const parts = version.split('.').map((p) => parseInt(p, 10));
const isStable = !isPrerelease && parts.length >= 2 && parts[0] >= 1;
return {
status: isStable ? 'pass' : 'fail',
version,
is_stable: isStable,
is_prerelease: isPrerelease
};
} catch {
return {
status: 'fail',
version: null,
is_stable: false,
is_prerelease: false
};
}
}

79
src/tier-check/checks/spec-tracking.ts Normal file
View file

@ -0,0 +1,79 @@
import { Octokit } from '@octokit/rest';
import { SpecTrackingResult } from '../types';
export async function checkSpecTracking(
octokit: Octokit,
owner: string,
repo: string
): Promise<SpecTrackingResult> {
try {
// Get latest spec release from modelcontextprotocol/modelcontextprotocol
const { data: specReleases } = await octokit.repos.listReleases({
owner: 'modelcontextprotocol',
repo: 'modelcontextprotocol',
per_page: 5
});
const latestSpec = specReleases.find((r) => !r.draft && !r.prerelease);
// Get SDK releases (API returns newest-first)
const { data: sdkReleases } = await octokit.repos.listReleases({
owner,
repo,
per_page: 50
});
const nonDraftSdkReleases = sdkReleases.filter((r) => !r.draft);
if (!latestSpec || nonDraftSdkReleases.length === 0) {
return {
status: 'skipped',
latest_spec_release: latestSpec?.published_at || null,
latest_sdk_release: nonDraftSdkReleases[0]?.published_at || null,
sdk_release_within_30d: null,
days_gap: null
};
}
const specDate = new Date(latestSpec.published_at!);
// Reverse so oldest-first, then find the FIRST SDK release after the spec
const oldestFirst = [...nonDraftSdkReleases].reverse();
const firstSdkAfterSpec = oldestFirst.find(
(r) => new Date(r.published_at!) >= specDate
);
if (!firstSdkAfterSpec) {
// No SDK release after the latest spec release
const daysSinceSpec = Math.round(
(Date.now() - specDate.getTime()) / (1000 * 60 * 60 * 24)
);
return {
status: daysSinceSpec <= 30 ? 'pass' : 'fail',
latest_spec_release: latestSpec.published_at,
latest_sdk_release: nonDraftSdkReleases[0]?.published_at || null,
sdk_release_within_30d: daysSinceSpec <= 30,
days_gap: daysSinceSpec
};
}
const sdkDate = new Date(firstSdkAfterSpec.published_at!);
const daysGap = Math.round(
(sdkDate.getTime() - specDate.getTime()) / (1000 * 60 * 60 * 24)
);
return {
status: daysGap <= 30 ? 'pass' : 'fail',
latest_spec_release: latestSpec.published_at,
latest_sdk_release: firstSdkAfterSpec.published_at,
sdk_release_within_30d: daysGap <= 30,
days_gap: daysGap
};
} catch {
return {
status: 'skipped',
latest_spec_release: null,
latest_sdk_release: null,
sdk_release_within_30d: null,
days_gap: null
};
}
}

203
src/tier-check/checks/test-conformance-results.ts Normal file
View file

@ -0,0 +1,203 @@
import { execSync } from 'child_process';
import { mkdtempSync, readFileSync, readdirSync, existsSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { ConformanceResult } from '../types';
import { listScenarios, listActiveClientScenarios } from '../../scenarios';
import { ConformanceCheck } from '../../types';
/**
* Parse conformance results from an output directory.
* The conformance CLI saves checks.json per scenario under outputDir/<scenario>/server/ or client/.
*/
function parseOutputDir(outputDir: string): ConformanceResult {
if (!existsSync(outputDir)) {
return {
status: 'fail',
pass_rate: 0,
passed: 0,
failed: 0,
total: 0,
details: []
};
}
const details: ConformanceResult['details'] = [];
let totalPassed = 0;
let totalFailed = 0;
const entries = readdirSync(outputDir);
for (const scenarioName of entries) {
const checksPath = join(outputDir, scenarioName, 'checks.json');
if (!existsSync(checksPath)) continue;
try {
const checks: ConformanceCheck[] = JSON.parse(
readFileSync(checksPath, 'utf-8')
);
const passed = checks.filter((c) => c.status === 'SUCCESS').length;
const failed = checks.filter((c) => c.status === 'FAILURE').length;
const scenarioPassed = failed === 0 && passed > 0;
totalPassed += scenarioPassed ? 1 : 0;
totalFailed += scenarioPassed ? 0 : 1;
details.push({
scenario: scenarioName,
passed: scenarioPassed,
checks_passed: passed,
checks_failed: failed
});
} catch {
totalFailed++;
details.push({
scenario: scenarioName,
passed: false,
checks_passed: 0,
checks_failed: 1
});
}
}
const total = totalPassed + totalFailed;
const pass_rate = total > 0 ? totalPassed / total : 0;
return {
status: pass_rate >= 1.0 ? 'pass' : pass_rate >= 0.8 ? 'partial' : 'fail',
pass_rate,
passed: totalPassed,
failed: totalFailed,
total,
details
};
}
/**
* Strip the timestamp suffix from a result directory name.
* Result dirs are named `{scenario}-{ISO timestamp}` where the timestamp
* has colons/dots replaced with dashes (e.g., `initialize-2026-02-12T16-08-37-806Z`).
* Server scenarios also have a `server-` prefix (e.g., `server-ping-2026-02-12T16-08-37-806Z`).
*/
function stripTimestamp(dirName: string): string {
return dirName.replace(/-\d{4}-\d{2}-\d{2}T[\d-]+Z$/, '');
}
/**
* Reconcile parsed results against the full list of expected scenarios.
* Any expected scenario that didn't produce results is counted as a failure.
* This ensures the denominator reflects the full test suite, not just
* scenarios that ran successfully enough to write checks.json.
*/
function reconcileWithExpected(
result: ConformanceResult,
expectedScenarios: string[],
resultPrefix?: string
): ConformanceResult {
const reportedNames = new Set(
result.details.map((d) => {
let name = stripTimestamp(d.scenario);
if (resultPrefix) {
name = name.replace(new RegExp(`^${resultPrefix}-`), '');
}
return name;
})
);
for (const expected of expectedScenarios) {
if (!reportedNames.has(expected)) {
result.failed++;
result.total++;
result.details.push({
scenario: expected,
passed: false,
checks_passed: 0,
checks_failed: 0
});
}
}
result.pass_rate = result.total > 0 ? result.passed / result.total : 0;
result.status =
result.pass_rate >= 1.0
? 'pass'
: result.pass_rate >= 0.8
? 'partial'
: 'fail';
return result;
}
/**
* Run server conformance tests by shelling out to the conformance CLI.
*/
export async function checkConformance(options: {
serverUrl?: string;
skip?: boolean;
}): Promise<ConformanceResult> {
if (options.skip || !options.serverUrl) {
return {
status: 'skipped',
pass_rate: 0,
passed: 0,
failed: 0,
total: 0,
details: []
};
}
const outputDir = mkdtempSync(join(tmpdir(), 'tier-check-server-'));
try {
execSync(
`node dist/index.js server --url ${options.serverUrl} -o ${outputDir}`,
{
cwd: process.cwd(),
stdio: ['pipe', 'pipe', 'pipe'],
timeout: 120_000
}
);
} catch {
// Non-zero exit is expected when tests fail — results are still in outputDir
}
return reconcileWithExpected(
parseOutputDir(outputDir),
listActiveClientScenarios(),
'server'
);
}
/**
* Run client conformance tests by shelling out to the conformance CLI.
*/
export async function checkClientConformance(options: {
clientCmd?: string;
skip?: boolean;
}): Promise<ConformanceResult> {
if (options.skip || !options.clientCmd) {
return {
status: 'skipped',
pass_rate: 0,
passed: 0,
failed: 0,
total: 0,
details: []
};
}
const outputDir = mkdtempSync(join(tmpdir(), 'tier-check-client-'));
try {
execSync(
`node dist/index.js client --command '${options.clientCmd}' --suite all -o ${outputDir}`,
{
cwd: process.cwd(),
stdio: ['pipe', 'pipe', 'pipe'],
timeout: 120_000
}
);
} catch {
// Non-zero exit is expected when tests fail — results are still in outputDir
}
return reconcileWithExpected(parseOutputDir(outputDir), listScenarios());
}

109
src/tier-check/checks/triage.ts Normal file
View file

@ -0,0 +1,109 @@
import { Octokit } from '@octokit/rest';
import { TriageResult } from '../types';
export async function checkTriage(
octokit: Octokit,
owner: string,
repo: string,
days?: number
): Promise<TriageResult> {
const since = days
? new Date(Date.now() - days * 24 * 60 * 60 * 1000).toISOString()
: undefined;
// Fetch issues (not PRs) — the API returns labels inline
const issues: Array<{
number: number;
created_at: string;
labels: string[];
}> = [];
let page = 1;
while (true) {
const { data } = await octokit.issues.listForRepo({
owner,
repo,
state: 'open',
...(since ? { since } : {}),
per_page: 100,
page,
sort: 'created',
direction: 'desc'
});
if (data.length === 0) break;
for (const issue of data) {
if (issue.pull_request) continue;
if (since && new Date(issue.created_at) < new Date(since)) continue;
issues.push({
number: issue.number,
created_at: issue.created_at,
labels: issue.labels
.filter(
(l): l is { name: string } =>
typeof l === 'object' && l !== null && 'name' in l
)
.map((l) => l.name)
});
}
if (data.length < 100) break;
page++;
}
if (issues.length === 0) {
return {
status: 'pass',
compliance_rate: 1,
total_issues: 0,
triaged_within_sla: 0,
exceeding_sla: 0,
median_hours: 0,
p95_hours: 0,
days_analyzed: days
};
}
// An issue is "triaged" if it has at least one label
const triaged = issues.filter((i) => i.labels.length > 0);
const untriaged = issues.filter((i) => i.labels.length === 0);
// For untriaged issues, compute how long they've been open without a label
const untriagedAgeHours = untriaged.map(
(i) => (Date.now() - new Date(i.created_at).getTime()) / (1000 * 60 * 60)
);
untriagedAgeHours.sort((a, b) => a - b);
const total = issues.length;
const triagedRate = total > 0 ? triaged.length / total : 1;
// For SLA: issues without labels that are older than 2BD are SLA violations
const TWO_BUSINESS_DAYS_HOURS = 2 * 24;
const exceeding = untriagedAgeHours.filter(
(h) => h > TWO_BUSINESS_DAYS_HOURS
).length;
// Median/p95 of untriaged issue ages (0 if all triaged)
const median =
untriagedAgeHours.length > 0
? untriagedAgeHours[Math.floor(untriagedAgeHours.length / 2)]
: 0;
const p95 =
untriagedAgeHours.length > 0
? untriagedAgeHours[Math.floor(untriagedAgeHours.length * 0.95)]
: 0;
let status: 'pass' | 'partial' | 'fail';
if (triagedRate >= 0.9) status = 'pass';
else if (triagedRate >= 0.8) status = 'partial';
else status = 'fail';
return {
status,
compliance_rate: triagedRate,
total_issues: total,
triaged_within_sla: triaged.length,
exceeding_sla: exceeding,
median_hours: Math.round(median * 10) / 10,
p95_hours: Math.round(p95 * 10) / 10,
days_analyzed: days
};
}

201
src/tier-check/index.ts Normal file
View file

@ -0,0 +1,201 @@
import { Command } from 'commander';
import { Octokit } from '@octokit/rest';
import {
checkConformance,
checkClientConformance
} from './checks/test-conformance-results';
import { checkLabels } from './checks/labels';
import { checkTriage } from './checks/triage';
import { checkP0Resolution } from './checks/p0';
import { checkStableRelease } from './checks/release';
import { checkPolicySignals } from './checks/files';
import { checkSpecTracking } from './checks/spec-tracking';
import { computeTier } from './tier-logic';
import { formatJson, formatMarkdown, formatTerminal } from './output';
import { TierScorecard } from './types';
function parseRepo(repo: string): { owner: string; repo: string } {
const parts = repo.split('/');
if (parts.length !== 2)
throw new Error(`Invalid repo format: ${repo}. Expected owner/repo`);
return { owner: parts[0], repo: parts[1] };
}
export function createTierCheckCommand(): Command {
const tierCheck = new Command('tier-check')
.description('Run SDK tier assessment checks against a GitHub repository')
.requiredOption(
'--repo <owner/repo>',
'GitHub repository (e.g., modelcontextprotocol/typescript-sdk)'
)
.option('--branch <branch>', 'Branch to check')
.option(
'--conformance-server-url <url>',
'URL of the already-running conformance server'
)
.option(
'--client-cmd <cmd>',
'Command to run the SDK conformance client (for client conformance tests)'
)
.option('--skip-conformance', 'Skip conformance tests')
.option('--days <n>', 'Limit triage check to issues created in last N days')
.option(
'--output <format>',
'Output format: json, markdown, terminal',
'terminal'
)
.option(
'--token <token>',
'GitHub token (defaults to GITHUB_TOKEN env var)'
)
.action(async (options) => {
const { owner, repo } = parseRepo(options.repo);
let token = options.token || process.env.GITHUB_TOKEN;
if (!token) {
// Try to get token from GitHub CLI
try {
const { execSync } = await import('child_process');
token = execSync('gh auth token', { encoding: 'utf-8' }).trim();
} catch {
// gh not installed or not authenticated
}
}
if (!token) {
console.error(
'GitHub token required. Either:\n' +
' gh auth login\n' +
' export GITHUB_TOKEN=$(gh auth token)\n' +
' or pass --token <token>'
);
process.exit(1);
}
const octokit = new Octokit({ auth: token });
const days = options.days ? parseInt(options.days, 10) : undefined;
console.error('Running tier assessment checks...\n');
// Run all checks
const [
conformance,
clientConformance,
labels,
triage,
p0,
release,
files,
specTracking
] = await Promise.all([
checkConformance({
serverUrl: options.conformanceServerUrl,
skip: options.skipConformance
}).then((r) => {
console.error(' ✓ Server Conformance');
return r;
}),
checkClientConformance({
clientCmd: options.clientCmd,
skip: options.skipConformance || !options.clientCmd
}).then((r) => {
console.error(' ✓ Client Conformance');
return r;
}),
checkLabels(octokit, owner, repo).then((r) => {
console.error(' ✓ Labels');
return r;
}),
checkTriage(octokit, owner, repo, days).then((r) => {
console.error(' \u2713 Triage');
return r;
}),
checkP0Resolution(octokit, owner, repo).then((r) => {
console.error(' \u2713 P0 Resolution');
return r;
}),
checkStableRelease(octokit, owner, repo).then((r) => {
console.error(' \u2713 Stable Release');
return r;
}),
checkPolicySignals(octokit, owner, repo, options.branch).then((r) => {
console.error(' \u2713 Policy Signals');
return r;
}),
checkSpecTracking(octokit, owner, repo).then((r) => {
console.error(' \u2713 Spec Tracking');
return r;
})
]);
const checks = {
conformance,
client_conformance: clientConformance,
labels,
triage,
p0_resolution: p0,
stable_release: release,
policy_signals: files,
spec_tracking: specTracking
};
const implied_tier = computeTier(checks);
const scorecard: TierScorecard = {
repo: options.repo,
branch: options.branch || null,
timestamp: new Date().toISOString(),
version: release.version,
checks,
implied_tier
};
switch (options.output) {
case 'json':
console.log(formatJson(scorecard));
break;
case 'markdown':
console.log(formatMarkdown(scorecard));
break;
default:
formatTerminal(scorecard);
}
});
// Subcommands for individual checks
tierCheck
.command('labels')
.description('Check label taxonomy')
.requiredOption('--repo <owner/repo>', 'GitHub repository')
.option('--token <token>', 'GitHub token')
.action(async (options) => {
const { owner, repo } = parseRepo(options.repo);
const octokit = new Octokit({
auth: options.token || process.env.GITHUB_TOKEN
});
const result = await checkLabels(octokit, owner, repo);
console.log(JSON.stringify(result, null, 2));
});
tierCheck
.command('triage')
.description('Check issue triage speed')
.requiredOption('--repo <owner/repo>', 'GitHub repository')
.option('--days <n>', 'Limit triage check to issues created in last N days')
.option('--token <token>', 'GitHub token')
.action(async (options) => {
const { owner, repo } = parseRepo(options.repo);
const octokit = new Octokit({
auth: options.token || process.env.GITHUB_TOKEN
});
const result = await checkTriage(
octokit,
owner,
repo,
options.days ? parseInt(options.days, 10) : undefined
);
console.log(JSON.stringify(result, null, 2));
});
return tierCheck;
}

153
src/tier-check/output.ts Normal file
View file

@ -0,0 +1,153 @@
import { TierScorecard, CheckStatus } from './types';
const COLORS = {
RESET: '\x1b[0m',
GREEN: '\x1b[32m',
YELLOW: '\x1b[33m',
RED: '\x1b[31m',
BLUE: '\x1b[36m',
BOLD: '\x1b[1m',
DIM: '\x1b[2m'
};
function statusIcon(status: CheckStatus): string {
switch (status) {
case 'pass':
return `${COLORS.GREEN}\u2713${COLORS.RESET}`;
case 'fail':
return `${COLORS.RED}\u2717${COLORS.RESET}`;
case 'partial':
return `${COLORS.YELLOW}~${COLORS.RESET}`;
case 'skipped':
return `${COLORS.DIM}-${COLORS.RESET}`;
}
}
export function formatJson(scorecard: TierScorecard): string {
return JSON.stringify(scorecard, null, 2);
}
export function formatMarkdown(scorecard: TierScorecard): string {
const lines: string[] = [];
const c = scorecard.checks;
lines.push(`# Tier Assessment: Tier ${scorecard.implied_tier.tier}`);
lines.push('');
lines.push(`**Repo**: ${scorecard.repo}`);
if (scorecard.branch) lines.push(`**Branch**: ${scorecard.branch}`);
if (scorecard.version) lines.push(`**Version**: ${scorecard.version}`);
lines.push(`**Timestamp**: ${scorecard.timestamp}`);
lines.push('');
lines.push('## Check Results');
lines.push('');
lines.push('| Check | Status | Detail |');
lines.push('|-------|--------|--------|');
lines.push(
`| Server Conformance | ${c.conformance.status} | ${c.conformance.passed}/${c.conformance.total} scenarios pass (${Math.round(c.conformance.pass_rate * 100)}%) |`
);
lines.push(
`| Client Conformance | ${c.client_conformance.status} | ${c.client_conformance.passed}/${c.client_conformance.total} scenarios pass (${Math.round(c.client_conformance.pass_rate * 100)}%) |`
);
lines.push(
`| Labels | ${c.labels.status} | ${c.labels.present}/${c.labels.required} required labels${c.labels.missing.length > 0 ? ` (missing: ${c.labels.missing.join(', ')})` : ''} |`
);
lines.push(
`| Triage | ${c.triage.status} | ${Math.round(c.triage.compliance_rate * 100)}% within 2BD, median ${c.triage.median_hours}h, p95 ${c.triage.p95_hours}h |`
);
lines.push(
`| P0 Resolution | ${c.p0_resolution.status} | ${c.p0_resolution.open_p0s} open, ${c.p0_resolution.closed_within_7d}/${c.p0_resolution.closed_total} closed within 7d |`
);
lines.push(
`| Stable Release | ${c.stable_release.status} | ${c.stable_release.version || 'none'} (stable: ${c.stable_release.is_stable}) |`
);
lines.push(
`| Policy Signals | ${c.policy_signals.status} | ${Object.entries(
c.policy_signals.files
)
.map(([f, e]) => `${f}: ${e ? '\u2713' : '\u2717'}`)
.join(', ')} |`
);
lines.push(
`| Spec Tracking | ${c.spec_tracking.status} | ${c.spec_tracking.days_gap !== null ? `${c.spec_tracking.days_gap}d gap` : 'N/A'} |`
);
lines.push('');
if (scorecard.implied_tier.tier1_blockers.length > 0) {
lines.push('## Tier 1 Blockers');
lines.push('');
for (const blocker of scorecard.implied_tier.tier1_blockers) {
lines.push(`- ${blocker}`);
}
lines.push('');
}
lines.push(`> ${scorecard.implied_tier.note}`);
return lines.join('\n');
}
export function formatTerminal(scorecard: TierScorecard): void {
const c = scorecard.checks;
const tier = scorecard.implied_tier.tier;
const tierColor =
tier === 1 ? COLORS.GREEN : tier === 2 ? COLORS.YELLOW : COLORS.RED;
console.log(
`\n${COLORS.BOLD}Tier Assessment: ${tierColor}Tier ${tier}${COLORS.RESET}\n`
);
console.log(`Repo: ${scorecard.repo}`);
if (scorecard.branch) console.log(`Branch: ${scorecard.branch}`);
if (scorecard.version) console.log(`Version: ${scorecard.version}`);
console.log(`Timestamp: ${scorecard.timestamp}\n`);
console.log(`${COLORS.BOLD}Check Results:${COLORS.RESET}\n`);
console.log(
` ${statusIcon(c.conformance.status)} Server Conformance ${c.conformance.passed}/${c.conformance.total} (${Math.round(c.conformance.pass_rate * 100)}%)`
);
console.log(
` ${statusIcon(c.client_conformance.status)} Client Conformance ${c.client_conformance.passed}/${c.client_conformance.total} (${Math.round(c.client_conformance.pass_rate * 100)}%)`
);
console.log(
` ${statusIcon(c.labels.status)} Labels ${c.labels.present}/${c.labels.required} required labels`
);
if (c.labels.missing.length > 0)
console.log(
` ${COLORS.DIM}Missing: ${c.labels.missing.join(', ')}${COLORS.RESET}`
);
console.log(
` ${statusIcon(c.triage.status)} Triage ${Math.round(c.triage.compliance_rate * 100)}% within 2BD (${c.triage.total_issues} issues, median ${c.triage.median_hours}h)`
);
console.log(
` ${statusIcon(c.p0_resolution.status)} P0 Resolution ${c.p0_resolution.open_p0s} open, ${c.p0_resolution.closed_within_7d}/${c.p0_resolution.closed_total} closed within 7d`
);
if (c.p0_resolution.open_p0_details.length > 0) {
for (const p0 of c.p0_resolution.open_p0_details) {
console.log(
` ${COLORS.RED}#${p0.number} (${p0.age_days}d old): ${p0.title}${COLORS.RESET}`
);
}
}
console.log(
` ${statusIcon(c.stable_release.status)} Stable Release ${c.stable_release.version || 'none'}`
);
console.log(
` ${statusIcon(c.policy_signals.status)} Policy Signals ${Object.entries(
c.policy_signals.files
)
.map(([f, e]) => `${e ? '\u2713' : '\u2717'} ${f}`)
.join(', ')}`
);
console.log(
` ${statusIcon(c.spec_tracking.status)} Spec Tracking ${c.spec_tracking.days_gap !== null ? `${c.spec_tracking.days_gap}d gap` : 'N/A'}`
);
if (scorecard.implied_tier.tier1_blockers.length > 0) {
console.log(`\n${COLORS.BOLD}Tier 1 Blockers:${COLORS.RESET}`);
for (const blocker of scorecard.implied_tier.tier1_blockers) {
console.log(` ${COLORS.RED}\u2022${COLORS.RESET} ${blocker}`);
}
}
console.log(`\n${COLORS.DIM}${scorecard.implied_tier.note}${COLORS.RESET}\n`);
}

66
src/tier-check/tier-logic.ts Normal file
View file

@ -0,0 +1,66 @@
import { TierScorecard } from './types';
export function computeTier(
checks: TierScorecard['checks']
): TierScorecard['implied_tier'] {
const tier1Blockers: string[] = [];
// Check Tier 1 requirements — server conformance
if (checks.conformance.status === 'skipped') {
tier1Blockers.push('server_conformance (skipped)');
} else if (checks.conformance.pass_rate < 1.0) {
tier1Blockers.push('server_conformance');
}
// Check Tier 1 requirements — client conformance
if (checks.client_conformance.status === 'skipped') {
tier1Blockers.push('client_conformance (skipped)');
} else if (checks.client_conformance.pass_rate < 1.0) {
tier1Blockers.push('client_conformance');
}
if (checks.triage.compliance_rate < 0.9) {
tier1Blockers.push('triage');
}
if (!checks.p0_resolution.all_p0s_resolved_within_7d) {
tier1Blockers.push('p0_resolution');
}
if (!checks.stable_release.is_stable) {
tier1Blockers.push('stable_release');
}
// Policy signals (CHANGELOG, SECURITY, etc.) are informational evidence —
// they feed into the skill's judgment-based evaluation but don't independently
// block tier advancement since SEP-1730 doesn't list specific files.
if (checks.spec_tracking.status === 'fail') {
tier1Blockers.push('spec_tracking');
}
if (checks.labels.missing.length > 0) {
tier1Blockers.push('labels');
}
// Check Tier 2 requirements
const tier2Met =
(checks.conformance.status === 'skipped' ||
checks.conformance.pass_rate >= 0.8) &&
(checks.client_conformance.status === 'skipped' ||
checks.client_conformance.pass_rate >= 0.8) &&
checks.p0_resolution.all_p0s_resolved_within_14d &&
checks.stable_release.is_stable;
const tier = tier1Blockers.length === 0 ? 1 : tier2Met ? 2 : 3;
return {
tier,
tier1_blockers: tier1Blockers,
tier2_met: tier2Met,
note:
tier === 1
? 'All deterministic checks pass. Judgment-based checks (docs, policy, roadmap) require /mcp-sdk-tier-audit skill.'
: 'Partial assessment — judgment-based checks require /mcp-sdk-tier-audit skill'
};
}

87
src/tier-check/types.ts Normal file
View file

@ -0,0 +1,87 @@
export type CheckStatus = 'pass' | 'fail' | 'partial' | 'skipped';
export interface CheckResult {
status: CheckStatus;
[key: string]: unknown;
}
export interface ConformanceResult extends CheckResult {
pass_rate: number;
passed: number;
failed: number;
total: number;
details: Array<{
scenario: string;
passed: boolean;
checks_passed: number;
checks_failed: number;
}>;
}
export interface LabelsResult extends CheckResult {
present: number;
required: number;
missing: string[];
found: string[];
uses_issue_types: boolean;
}
export interface TriageResult extends CheckResult {
compliance_rate: number;
total_issues: number;
triaged_within_sla: number;
exceeding_sla: number;
median_hours: number;
p95_hours: number;
days_analyzed: number | undefined;
}
export interface P0Result extends CheckResult {
open_p0s: number;
open_p0_details: Array<{ number: number; title: string; age_days: number }>;
closed_within_7d: number;
closed_within_14d: number;
closed_total: number;
all_p0s_resolved_within_7d: boolean;
all_p0s_resolved_within_14d: boolean;
}
export interface ReleaseResult extends CheckResult {
version: string | null;
is_stable: boolean;
is_prerelease: boolean;
}
export interface PolicySignalsResult extends CheckResult {
files: Record<string, boolean>;
}
export interface SpecTrackingResult extends CheckResult {
latest_spec_release: string | null;
latest_sdk_release: string | null;
sdk_release_within_30d: boolean | null;
days_gap: number | null;
}
export interface TierScorecard {
repo: string;
branch: string | null;
timestamp: string;
version: string | null;
checks: {
conformance: ConformanceResult;
client_conformance: ConformanceResult;
labels: LabelsResult;
triage: TriageResult;
p0_resolution: P0Result;
stable_release: ReleaseResult;
policy_signals: PolicySignalsResult;
spec_tracking: SpecTrackingResult;
};
implied_tier: {
tier: 1 | 2 | 3;
tier1_blockers: string[];
tier2_met: boolean;
note: string;
};
}