tsdown + cli with commander (#17)

* feat: tsdown

* fix: formatting

* automate version updates in cli

* fix: docs

* scenarios

* fix: commander deps
This commit is contained in:
Matt 2025-11-11 11:51:55 +00:00 • committed by GitHub
commit 7f1515c475
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
16 changed files with 1600 additions and 539 deletions

3
.gitignore vendored
View file

@ -1,3 +1,4 @@
node_modules node_modules
results/ results/
lefthook-local.yml lefthook-local.yml
dist

View file

@ -1,6 +1,6 @@
# MCP Conformance Test Framework # MCP Conformance Test Framework
A framework for testing MCP (Model Context Protocol) client implementations against the specification. A framework for testing MCP (Model Context Protocol) client and server implementations against the specification.
> [!WARNING] This repository is a work in progress and is unstable. Join the conversation in the #conformance-testing-wg in the MCP Contributors discord. > [!WARNING] This repository is a work in progress and is unstable. Join the conversation in the #conformance-testing-wg in the MCP Contributors discord.
@ -9,19 +9,30 @@ A framework for testing MCP (Model Context Protocol) client implementations agai
### Testing Clients ### Testing Clients
```bash ```bash
npm install npx @modelcontextprotocol/conformance client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize
npm run start -- --command "tsx examples/clients/typescript/test1.ts" --scenario initialize
``` ```
### Testing Servers ### Testing Servers
```bash ```bash
npm run test:server -- --server-url http://localhost:3000/mcp --all # Run all server scenarios (default)
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp
# Run a single scenario
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --scenario server-initialize
```
### List Available Scenarios
```bash
npx @modelcontextprotocol/conformance list
``` ```
## Overview ## Overview
The conformance test framework validates MCP client implementations by: The conformance test framework validates MCP implementations by:
**For Clients:**
1. Starting a test server for the specified scenario 1. Starting a test server for the specified scenario
2. Running the client implementation with the test server URL 2. Running the client implementation with the test server URL
@ -29,25 +40,53 @@ The conformance test framework validates MCP client implementations by:
4. Running conformance checks against the specification 4. Running conformance checks against the specification
5. Generating detailed test results 5. Generating detailed test results
**For Servers:**
1. Connecting to the running server as an MCP client
2. Sending test requests and capturing responses
3. Running conformance checks against server behavior
4. Generating detailed test results
## Usage ## Usage
### Client Testing
```bash ```bash
npm run start -- --command "<client-command>" --scenario <scenario-name> npx @modelcontextprotocol/conformance client --command "<client-command>" --scenario <scenario-name> [options]
``` ```
**Options:**
- `--command` - The command to run your MCP client (can include flags) - `--command` - The command to run your MCP client (can include flags)
- `--scenario` - The test scenario to run (e.g., "initialize") - `--scenario` - The test scenario to run (e.g., "initialize")
- `--timeout` - Timeout in milliseconds (default: 30000)
- `--verbose` - Show verbose output
The framework appends the server URL as the final argument to your command. The framework appends the server URL as the final argument to your command.
### Server Testing
```bash
npx @modelcontextprotocol/conformance server --url <url> [--scenario <scenario>]
```
**Options:**
- `--url` - URL of the server to test
- `--scenario <scenario>` - Test scenario to run (e.g., "server-initialize". Runs all available scenarios by default
## Test Results ## Test Results
Results are saved to `results/<scenario>-<timestamp>/`: **Client Testing** - Results are saved to `results/<scenario>-<timestamp>/`:
- `checks.json` - Array of conformance check results with pass/fail status - `checks.json` - Array of conformance check results with pass/fail status
- `stdout.txt` - Client stdout output - `stdout.txt` - Client stdout output
- `stderr.txt` - Client stderr output - `stderr.txt` - Client stderr output
**Server Testing** - Results are saved to `results/server-<scenario>-<timestamp>/`:
- `checks.json` - Array of conformance check results with pass/fail status
## Example Clients ## Example Clients
- `examples/clients/typescript/test1.ts` - Valid MCP client (passes all checks) - `examples/clients/typescript/test1.ts` - Valid MCP client (passes all checks)
@ -55,10 +94,25 @@ Results are saved to `results/<scenario>-<timestamp>/`:
## Available Scenarios ## Available Scenarios
### Client Scenarios
- **initialize** - Tests MCP client initialization handshake - **initialize** - Tests MCP client initialization handshake
- Validates protocol version - Validates protocol version
- Validates clientInfo (name and version) - Validates clientInfo (name and version)
- Validates server response handling - Validates server response handling
- **tools-call** - Tests tool invocation
- **auth/basic-dcr** - Tests OAuth Dynamic Client Registration flow
- **auth/basic-metadata-var1** - Tests OAuth with authorization metadata
### Server Scenarios
Run `npx @modelcontextprotocol/conformance list --server` to see all available server scenarios, including:
- **server-initialize** - Tests server initialization and capabilities
- **tools-list** - Tests tool listing endpoint
- **tools-call-\*** - Various tool invocation scenarios
- **resources-\*** - Resource management scenarios
- **prompts-\*** - Prompt management scenarios
## Architecture ## Architecture
@ -67,8 +121,13 @@ See `src/runner/DESIGN.md` for detailed architecture documentation.
### Key Components ### Key Components
- **Runner** (`src/runner/`) - Orchestrates test execution and result generation - **Runner** (`src/runner/`) - Orchestrates test execution and result generation
- **Scenarios** (`src/scenarios/`) - Test scenarios with custom server implementations - `client.ts` - Client testing implementation
- **Checks** (`src/checks.ts`) - Conformance validation functions - `server.ts` - Server testing implementation
- `utils.ts` - Shared utilities
- `index.ts` - Public API exports
- **CLI** (`src/index.ts`) - Command-line interface using Commander.js
- **Scenarios** (`src/scenarios/`) - Test scenarios with expected behaviors
- **Checks** (`src/checks/`) - Conformance validation functions
- **Types** (`src/types.ts`) - Shared type definitions - **Types** (`src/types.ts`) - Shared type definitions
## Adding New Scenarios ## Adding New Scenarios

View file

@ -1022,8 +1022,8 @@ MCP Conformance Test Server running on http://localhost:3000
```bash ```bash
# Single test # Single test
npm run test:server -- --server-url http://localhost:3000/mcp --scenario server-initialize npm run start -- server --url http://localhost:3000/mcp --scenario server-initialize
# All tests # All tests
npm run test:server -- --server-url http://localhost:3000/mcp --all npm run start -- server --url http://localhost:3000/mcp
``` ```

View file

@ -147,8 +147,7 @@ This server implements the MCP Server Conformance Requirements specified in `../
To run conformance tests against this server: To run conformance tests against this server:
```bash ```bash
cd ../../../ npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp
npm run test:server -- --server-url http://localhost:3000/mcp --all
``` ```
## Implementation Notes ## Implementation Notes

788
package-lock.json generated

File diff suppressed because it is too large Load diff

View file

@ -3,13 +3,19 @@
"version": "0.1.0", "version": "0.1.0",
"type": "module", "type": "module",
"scripts": { "scripts": {
"start": "tsx src/runner.ts", "start": "tsx src/index.ts",
"test:server": "tsx src/server-runner.ts",
"test": "jest", "test": "jest",
"build": "tsdown src/index.ts --minify --clean --target node20",
"lint": "eslint src/ && prettier --check .", "lint": "eslint src/ && prettier --check .",
"lint:fix": "eslint src/ --fix && prettier --write .", "lint:fix": "eslint src/ --fix && prettier --write .",
"lint:fix_check": "npm run lint:fix && git diff --exit-code --quiet" "lint:fix_check": "npm run lint:fix && git diff --exit-code --quiet"
}, },
"files": [
"dist"
],
"bin": {
"conformance": "dist/index.mjs"
},
"devDependencies": { "devDependencies": {
"@eslint/js": "^9.8.0", "@eslint/js": "^9.8.0",
"@jest/globals": "^29.7.0", "@jest/globals": "^29.7.0",
@ -20,12 +26,14 @@
"jest": "^29.7.0", "jest": "^29.7.0",
"prettier": "3.6.2", "prettier": "3.6.2",
"ts-jest": "^29.1.1", "ts-jest": "^29.1.1",
"tsdown": "^0.16.2",
"tsx": "^4.7.0", "tsx": "^4.7.0",
"typescript": "^5.5.4", "typescript": "^5.5.4",
"typescript-eslint": "^8.0.0" "typescript-eslint": "^8.0.0"
}, },
"dependencies": { "dependencies": {
"@modelcontextprotocol/sdk": "^1.20.1", "@modelcontextprotocol/sdk": "^1.20.1",
"commander": "^14.0.2",
"express": "^5.1.0", "express": "^5.1.0",
"lefthook": "^2.0.2", "lefthook": "^2.0.2",
"zod": "^3.25.76" "zod": "^3.25.76"

164
src/index.ts Normal file
View file

@ -0,0 +1,164 @@
#!/usr/bin/env node
import { Command } from 'commander';
import { ZodError } from 'zod';
import {
runConformanceTest,
printClientResults,
runServerConformanceTest,
printServerResults,
printServerSummary
} from './runner';
import { listScenarios, listClientScenarios } from './scenarios';
import { ConformanceCheck } from './types';
import { ClientOptionsSchema, ServerOptionsSchema } from './schemas';
import packageJson from '../package.json';
const program = new Command();
program
.name('conformance')
.description('MCP Conformance Test Suite')
.version(packageJson.version);
// Client command - tests a client implementation against scenarios
program
.command('client')
.description('Run conformance tests against a client implementation')
.requiredOption('--command <command>', 'Command to run the client')
.requiredOption('--scenario <scenario>', 'Scenario to test')
.option('--timeout <ms>', 'Timeout in milliseconds', '30000')
.option('--verbose', 'Show verbose output')
.action(async (options) => {
try {
// Validate options with Zod
const validated = ClientOptionsSchema.parse(options);
const result = await runConformanceTest(
validated.command,
validated.scenario,
validated.timeout
);
const { failed } = printClientResults(
result.checks,
validated.verbose ?? false
);
process.exit(failed > 0 ? 1 : 0);
} catch (error) {
if (error instanceof ZodError) {
console.error('Validation error:');
error.errors.forEach((err) => {
console.error(` ${err.path.join('.')}: ${err.message}`);
});
console.error('\nAvailable client scenarios:');
listScenarios().forEach((s) => console.error(` - ${s}`));
process.exit(1);
}
console.error('Client test error:', error);
process.exit(1);
}
});
// Server command - tests a server implementation
program
.command('server')
.description('Run conformance tests against a server implementation')
.requiredOption('--url <url>', 'URL of the server to test')
.option(
'--scenario <scenario>',
'Scenario to test (defaults to all scenarios if not specified)'
)
.action(async (options) => {
try {
// Validate options with Zod
const validated = ServerOptionsSchema.parse(options);
// If a single scenario is specified, run just that one
if (validated.scenario) {
const result = await runServerConformanceTest(
validated.url,
validated.scenario
);
const { failed } = printServerResults(result.checks);
process.exit(failed > 0 ? 1 : 0);
} else {
// Run all scenarios
const scenarios = listClientScenarios();
console.log(
`Running ${scenarios.length} scenarios against ${validated.url}\n`
);
const allResults: { scenario: string; checks: ConformanceCheck[] }[] =
[];
for (const scenarioName of scenarios) {
console.log(`\n=== Running scenario: ${scenarioName} ===`);
try {
const result = await runServerConformanceTest(
validated.url,
scenarioName
);
allResults.push({ scenario: scenarioName, checks: result.checks });
} catch (error) {
console.error(`Failed to run scenario ${scenarioName}:`, error);
allResults.push({
scenario: scenarioName,
checks: [
{
id: scenarioName,
name: scenarioName,
description: 'Failed to run scenario',
status: 'FAILURE',
timestamp: new Date().toISOString(),
errorMessage:
error instanceof Error ? error.message : String(error)
}
]
});
}
}
const { totalFailed } = printServerSummary(allResults);
process.exit(totalFailed > 0 ? 1 : 0);
}
} catch (error) {
if (error instanceof ZodError) {
console.error('Validation error:');
error.errors.forEach((err) => {
console.error(` ${err.path.join('.')}: ${err.message}`);
});
console.error('\nAvailable server scenarios:');
listClientScenarios().forEach((s) => console.error(` - ${s}`));
process.exit(1);
}
console.error('Server test error:', error);
process.exit(1);
}
});
// List scenarios command
program
.command('list')
.description('List available test scenarios')
.option('--client', 'List client scenarios')
.option('--server', 'List server scenarios')
.action((options) => {
if (options.server || (!options.client && !options.server)) {
console.log('Server scenarios (test against a server):');
const serverScenarios = listClientScenarios();
serverScenarios.forEach((s) => console.log(` - ${s}`));
}
if (options.client || (!options.client && !options.server)) {
if (options.server || (!options.client && !options.server)) {
console.log('');
}
console.log('Client scenarios (test against a client):');
const clientScenarios = listScenarios();
clientScenarios.forEach((s) => console.log(` - ${s}`));
}
});
program.parse();

View file

@ -1,3 +0,0 @@
import { runConformanceTest } from './runner/index.js';
export { runConformanceTest };

View file

@ -2,20 +2,45 @@
## Overview ## Overview
The conformance test runner is a framework for testing MCP client implementations against the MCP specification. It executes client implementations in controlled scenarios and validates their behavior through a series of conformance checks. The conformance test runner is a framework for testing both MCP client and server implementations against the MCP specification. It provides two testing modes:
1. **Client Testing** - Executes client implementations in controlled scenarios and validates their behavior
2. **Server Testing** - Tests server implementations by acting as an MCP client and validating responses
## Architecture ## Architecture
### File Structure
```
src/runner/
├── index.ts # Exports all public functions
├── client.ts # Client testing implementation
├── server.ts # Server testing implementation
└── utils.ts # Shared utilities (formatting, file I/O, ANSI colors)
```
### Components ### Components
**Client Testing:**
1. **Runner** - Orchestrates test execution 1. **Runner** - Orchestrates test execution
2. **Test Server** - MCP server (and optionally auth server for auth scenarios) 2. **Test Server** - MCP server (and optionally auth server for auth scenarios)
3. **Client Process** - The MCP client implementation under test 3. **Client Process** - The MCP client implementation under test
4. **Scenario** - A specific test case with expected behaviors 4. **Scenario** - A specific test case with expected behaviors
5. **Checks** - Validation functions that produce ConformanceCheck results 5. **Checks** - Validation functions that produce ConformanceCheck results
**Server Testing:**
1. **Runner** - Orchestrates test execution
2. **Test Client** - MCP SDK client that connects to the server under test
3. **Server Process** - The MCP server implementation under test (external)
4. **Scenario** - A specific test case with expected server behaviors
5. **Checks** - Validation functions that produce ConformanceCheck results
### Execution Flow ### Execution Flow
**Client Testing:**
``` ```
1. Runner starts test server(s) on available port(s) 1. Runner starts test server(s) on available port(s)
2. Runner spawns client process with server URL as final argument 2. Runner spawns client process with server URL as final argument
@ -25,44 +50,124 @@ The conformance test runner is a framework for testing MCP client implementation
6. Results are written to results/<scenario>-<timestamp>/ 6. Results are written to results/<scenario>-<timestamp>/
``` ```
## CLI Interface **Server Testing:**
```bash ```
npm run start -- --command "tsx examples/clients/typescript/test1.ts --verbose" --scenario initialize 1. User starts their server implementation
2. Runner connects to server URL as an MCP client
3. Runner sends requests and captures responses
4. Scenario-specific checks validate server behavior
5. Results are written to results/server-<scenario>-<timestamp>/
``` ```
### Arguments ## CLI Interface
The conformance suite provides a unified CLI with two main commands:
### Client Testing
```bash
# Run a single client scenario
npm run start -- client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize
# With verbose output
npm run start -- client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize --verbose
# With custom timeout
npm run start -- client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize --timeout 60000
```
**Arguments:**
- `--command` - The command to run the client (can include existing flags) - `--command` - The command to run the client (can include existing flags)
- `--scenario` - The scenario to test (e.g., "initialize", "list-tools", "call-tool") - `--scenario` - The scenario to test (e.g., "initialize", "tools-call")
- `--timeout` - Timeout in milliseconds (default: 30000)
- `--verbose` - Show verbose output (JSON format)
The runner will append the server URL as the final argument to the command. The runner will append the server URL as the final argument to the command.
### Server Testing
```bash
# Run a single server scenario
npm run start -- server --url http://localhost:3000/mcp --scenario server-initialize
# Run all server scenarios (default when no --scenario specified)
npm run start -- server --url http://localhost:3000/mcp
```
**Arguments:**
- `--url` - URL of the server to test
- `--scenario <scenario>` - Scenario to test (optional, defaults to all scenarios if not specified)
### List Available Scenarios
```bash
# List all scenarios
npm run start -- list
# List only client scenarios
npm run start -- list --client
# List only server scenarios
npm run start -- list --server
```
## Validation
All CLI arguments are validated using Zod schemas (`src/schemas.ts`) before being passed to runner functions:
**Client Validation:**
- Command is non-empty
- Scenario exists in available scenarios
- Timeout is a positive integer
**Server Validation:**
- URL is a valid HTTP/HTTPS URL
- All specified scenarios exist (if provided)
- Defaults to all scenarios when no `--scenario` is specified
## Scenarios ## Scenarios
A scenario represents a specific test case that validates one or more aspects of MCP client behavior. Each scenario: A scenario represents a specific test case that validates one or more aspects of MCP behavior. Each scenario:
- Configures the test server with expected behavior - Configures the test environment with expected behavior
- May run multiple conformance checks - May run multiple conformance checks
- Validates both client behavior and server responses - Validates both request/response patterns and protocol compliance
Example scenarios: ### Client Scenario Examples
- `initialize` - Tests client initialization handshake - `initialize` - Tests client initialization handshake
- `list-tools` - Tests tool discovery - `tools-call` - Tests tool invocation
- `call-tool` - Tests tool invocation - `auth/basic-dcr` - Tests OAuth Dynamic Client Registration flow
- `auth-flow` - Tests OAuth/authorization flows (requires auth server) - `auth/basic-metadata-var1` - Tests OAuth with authorization metadata variation 1
### Server Scenario Examples
- `server-initialize` - Tests server initialization and capabilities
- `tools-list` - Tests tool listing endpoint
- `tools-call-simple-text` - Tests tool invocation with text response
- `resources-list` - Tests resource listing
- `prompts-list` - Tests prompt listing
- `logging-set-level` - Tests logging level configuration
## Output Structure ## Output Structure
Results are written to: `results/<scenario>-<timestamp>/` Results are written to: `results/<scenario>-<timestamp>/` or `results/server-<scenario>-<timestamp>/`
Files: **Client Testing Files:**
- `checks.json` - Array of ConformanceCheck objects with validation results - `checks.json` - Array of ConformanceCheck objects with validation results
- `stdout.txt` - Complete stdout from the client process - `stdout.txt` - Complete stdout from the client process
- `stderr.txt` - Complete stderr from the client process - `stderr.txt` - Complete stderr from the client process
**Server Testing Files:**
- `checks.json` - Array of ConformanceCheck objects with validation results
### checks.json Format ### checks.json Format
```json ```json
@ -84,6 +189,30 @@ Files:
] ]
``` ```
## Programmatic Usage
The runner can also be used programmatically:
```typescript
import { runConformanceTest, printClientResults, runServerConformanceTest, printServerResults } from './runner/index.js';
// Test a client
const clientResult = await runConformanceTest('tsx my-client.ts', 'initialize', 30000);
const { failed } = printClientResults(clientResult.checks);
// Test a server
const serverResult = await runServerConformanceTest('http://localhost:3000/mcp', 'server-initialize');
const { failed } = printServerResults(serverResult.checks);
```
## Shared Utilities
The `utils.ts` module provides shared functionality:
- **File Operations:** `ensureResultsDir()`, `createResultDir()`
- **Formatting:** `formatPrettyChecks()` for colored console output
- **Styling:** ANSI color constants and helpers
## Future Enhancements ## Future Enhancements
- **Test Suites** - Group multiple scenarios for convenience - **Test Suites** - Group multiple scenarios for convenience
@ -91,3 +220,5 @@ Files:
- **Parallel Execution** - Run multiple scenarios concurrently - **Parallel Execution** - Run multiple scenarios concurrently
- **Custom Servers** - Allow custom MCP server implementations for advanced scenarios - **Custom Servers** - Allow custom MCP server implementations for advanced scenarios
- **Report Formats** - HTML, Markdown, or other human-readable formats - **Report Formats** - HTML, Markdown, or other human-readable formats
- **Watch Mode** - Automatically re-run tests on file changes
- **Coverage Tracking** - Track which parts of the spec are tested

207
src/runner/client.ts Normal file
View file

@ -0,0 +1,207 @@
import { spawn } from 'child_process';
import { promises as fs } from 'fs';
import path from 'path';
import { ConformanceCheck } from '../types';
import { getScenario } from '../scenarios';
import { ensureResultsDir, createResultDir, formatPrettyChecks } from './utils';
export interface ClientExecutionResult {
exitCode: number;
stdout: string;
stderr: string;
timedOut: boolean;
}
async function executeClient(
command: string,
serverUrl: string,
timeout: number = 30000
): Promise<ClientExecutionResult> {
const commandParts = command.split(' ');
const executable = commandParts[0];
const args = [...commandParts.slice(1), serverUrl];
let stdout = '';
let stderr = '';
let timedOut = false;
return new Promise((resolve) => {
const process = spawn(executable, args, {
shell: true,
stdio: 'pipe'
});
const timeoutHandle = setTimeout(() => {
timedOut = true;
process.kill();
}, timeout);
if (process.stdout) {
process.stdout.on('data', (data) => {
stdout += data.toString();
});
}
if (process.stderr) {
process.stderr.on('data', (data) => {
stderr += data.toString();
});
}
process.on('close', (code) => {
clearTimeout(timeoutHandle);
resolve({
exitCode: code || 0,
stdout,
stderr,
timedOut
});
});
process.on('error', (error) => {
clearTimeout(timeoutHandle);
resolve({
exitCode: -1,
stdout,
stderr: stderr + `\nProcess error: ${error.message}`,
timedOut
});
});
});
}
export async function runConformanceTest(
clientCommand: string,
scenarioName: string,
timeout: number = 30000
): Promise<{
checks: ConformanceCheck[];
clientOutput: ClientExecutionResult;
resultDir: string;
}> {
await ensureResultsDir();
const resultDir = createResultDir(scenarioName);
await fs.mkdir(resultDir, { recursive: true });
// Scenario is guaranteed to exist by CLI validation
const scenario = getScenario(scenarioName)!;
console.log(`Starting scenario: ${scenarioName}`);
const urls = await scenario.start();
console.log(`Executing client: ${clientCommand} ${urls.serverUrl}`);
try {
const clientOutput = await executeClient(
clientCommand,
urls.serverUrl,
timeout
);
// Print stdout/stderr if client exited with nonzero code
if (clientOutput.exitCode !== 0) {
console.error(`\nClient exited with code ${clientOutput.exitCode}`);
if (clientOutput.stdout) {
console.error(`\nStdout:\n${clientOutput.stdout}`);
}
if (clientOutput.stderr) {
console.error(`\nStderr:\n${clientOutput.stderr}`);
}
}
if (clientOutput.timedOut) {
console.error(`\nClient timed out after ${timeout}ms`);
}
const checks = scenario.getChecks();
await fs.writeFile(
path.join(resultDir, 'checks.json'),
JSON.stringify(checks, null, 2)
);
await fs.writeFile(path.join(resultDir, 'stdout.txt'), clientOutput.stdout);
await fs.writeFile(path.join(resultDir, 'stderr.txt'), clientOutput.stderr);
console.log(`Results saved to ${resultDir}`);
return {
checks,
clientOutput,
resultDir
};
} finally {
await scenario.stop();
}
}
export function printClientResults(
checks: ConformanceCheck[],
verbose: boolean = false
): { passed: number; failed: number; denominator: number } {
const denominator = checks.filter(
(c) => c.status === 'SUCCESS' || c.status === 'FAILURE'
).length;
const passed = checks.filter((c) => c.status === 'SUCCESS').length;
const failed = checks.filter((c) => c.status === 'FAILURE').length;
if (verbose) {
console.log(`Checks:\n${JSON.stringify(checks, null, 2)}`);
} else {
console.log(`Checks:\n${formatPrettyChecks(checks)}`);
}
console.log(`\nTest Results:`);
console.log(`Passed: ${passed}/${denominator}, ${failed} failed`);
if (failed > 0) {
console.log('\nFailed Checks:');
checks
.filter((c) => c.status === 'FAILURE')
.forEach((c) => {
console.log(` - ${c.name}: ${c.description}`);
if (c.errorMessage) {
console.log(` Error: ${c.errorMessage}`);
}
});
}
return { passed, failed, denominator };
}
export async function runInteractiveMode(scenarioName: string): Promise<void> {
await ensureResultsDir();
const resultDir = createResultDir(scenarioName);
await fs.mkdir(resultDir, { recursive: true });
// Scenario is guaranteed to exist by CLI validation
const scenario = getScenario(scenarioName)!;
console.log(`Starting scenario: ${scenarioName}`);
const urls = await scenario.start();
console.log(`Server URL: ${urls.serverUrl}`);
console.log('Press Ctrl+C to stop and save checks...');
const handleShutdown = async () => {
console.log('\nShutting down...');
const checks = scenario.getChecks();
await fs.writeFile(
path.join(resultDir, 'checks.json'),
JSON.stringify(checks, null, 2)
);
console.log(`\nChecks:\n${JSON.stringify(checks, null, 2)}`);
console.log(`\nChecks saved to ${resultDir}/checks.json`);
await scenario.stop();
process.exit(0);
};
process.on('SIGINT', handleShutdown);
process.on('SIGTERM', handleShutdown);
await new Promise(() => {});
}

View file

@ -1,312 +1,23 @@
import { spawn } from 'child_process'; // Export client functions
import { promises as fs } from 'fs'; export {
import path from 'path'; runConformanceTest,
import { ConformanceCheck } from '../types.js'; printClientResults,
import { getScenario } from '../scenarios/index.js'; runInteractiveMode,
type ClientExecutionResult
} from './client';
// ANSI color codes // Export server functions
const COLORS = { export {
RESET: '\x1b[0m', runServerConformanceTest,
GRAY: '\x1b[90m', printServerResults,
GREEN: '\x1b[32m', printServerSummary
YELLOW: '\x1b[33m', } from './server';
RED: '\x1b[31m',
BLUE: '\x1b[36m'
};
function getStatusColor(status: string): string { // Export utilities
switch (status) { export {
case 'SUCCESS': ensureResultsDir,
return COLORS.GREEN; createResultDir,
case 'FAILURE': formatPrettyChecks,
return COLORS.RED; getStatusColor,
case 'INFO': COLORS
return COLORS.BLUE; } from './utils';
default:
return COLORS.RESET;
}
}
function formatPrettyChecks(checks: ConformanceCheck[]): string {
// Find the longest id and status for column alignment
const maxIdLength = Math.max(...checks.map((c) => c.id.length));
const maxStatusLength = Math.max(...checks.map((c) => c.status.length));
return checks
.map((check) => {
const timestamp = `${COLORS.GRAY}${check.timestamp}${COLORS.RESET}`;
const id = check.id.padEnd(maxIdLength);
const statusColor = getStatusColor(check.status);
const status = `${statusColor}${check.status.padEnd(maxStatusLength)}${COLORS.RESET}`;
const description = check.description;
const line = `${timestamp} [${id}] ${status} ${description}`;
// Add newline after outgoing responses for better visual separation
return (
line +
(check.id.includes('outgoing') && check.id.includes('response')
? '\n'
: '')
);
})
.join('\n');
}
export interface ClientExecutionResult {
exitCode: number;
stdout: string;
stderr: string;
timedOut: boolean;
}
async function ensureResultsDir(): Promise<string> {
const resultsDir = path.join(process.cwd(), 'results');
await fs.mkdir(resultsDir, { recursive: true });
return resultsDir;
}
function createResultDir(scenario: string): string {
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
return path.join('results', `${scenario}-${timestamp}`);
}
async function executeClient(
command: string,
serverUrl: string,
timeout: number = 30000
): Promise<ClientExecutionResult> {
const commandParts = command.split(' ');
const executable = commandParts[0];
const args = [...commandParts.slice(1), serverUrl];
let stdout = '';
let stderr = '';
let timedOut = false;
return new Promise((resolve) => {
const process = spawn(executable, args, {
shell: true,
stdio: 'pipe'
});
const timeoutHandle = setTimeout(() => {
timedOut = true;
process.kill();
}, timeout);
if (process.stdout) {
process.stdout.on('data', (data) => {
stdout += data.toString();
});
}
if (process.stderr) {
process.stderr.on('data', (data) => {
stderr += data.toString();
});
}
process.on('close', (code) => {
clearTimeout(timeoutHandle);
resolve({
exitCode: code || 0,
stdout,
stderr,
timedOut
});
});
process.on('error', (error) => {
clearTimeout(timeoutHandle);
resolve({
exitCode: -1,
stdout,
stderr: stderr + `\nProcess error: ${error.message}`,
timedOut
});
});
});
}
export async function runConformanceTest(
clientCommand: string,
scenarioName: string,
timeout: number = 30000
): Promise<{
checks: ConformanceCheck[];
clientOutput: ClientExecutionResult;
resultDir: string;
}> {
await ensureResultsDir();
const resultDir = createResultDir(scenarioName);
await fs.mkdir(resultDir, { recursive: true });
const scenario = getScenario(scenarioName);
if (!scenario) {
throw new Error(`Unknown scenario: ${scenarioName}`);
}
console.log(`Starting scenario: ${scenarioName}`);
const urls = await scenario.start();
console.log(`Executing client: ${clientCommand} ${urls.serverUrl}`);
try {
const clientOutput = await executeClient(
clientCommand,
urls.serverUrl,
timeout
);
// Print stdout/stderr if client exited with nonzero code
if (clientOutput.exitCode !== 0) {
console.error(`\nClient exited with code ${clientOutput.exitCode}`);
if (clientOutput.stdout) {
console.error(`\nStdout:\n${clientOutput.stdout}`);
}
if (clientOutput.stderr) {
console.error(`\nStderr:\n${clientOutput.stderr}`);
}
}
if (clientOutput.timedOut) {
console.error(`\nClient timed out after ${timeout}ms`);
}
const checks = scenario.getChecks();
await fs.writeFile(
path.join(resultDir, 'checks.json'),
JSON.stringify(checks, null, 2)
);
await fs.writeFile(path.join(resultDir, 'stdout.txt'), clientOutput.stdout);
await fs.writeFile(path.join(resultDir, 'stderr.txt'), clientOutput.stderr);
console.log(`Results saved to ${resultDir}`);
return {
checks,
clientOutput,
resultDir
};
} finally {
await scenario.stop();
}
}
async function runInteractiveMode(scenarioName: string): Promise<void> {
await ensureResultsDir();
const resultDir = createResultDir(scenarioName);
await fs.mkdir(resultDir, { recursive: true });
const scenario = getScenario(scenarioName);
if (!scenario) {
throw new Error(`Unknown scenario: ${scenarioName}`);
}
console.log(`Starting scenario: ${scenarioName}`);
const urls = await scenario.start();
console.log(`Server URL: ${urls.serverUrl}`);
console.log('Press Ctrl+C to stop and save checks...');
const handleShutdown = async () => {
console.log('\nShutting down...');
const checks = scenario.getChecks();
await fs.writeFile(
path.join(resultDir, 'checks.json'),
JSON.stringify(checks, null, 2)
);
console.log(`\nChecks:\n${JSON.stringify(checks, null, 2)}`);
console.log(`\nChecks saved to ${resultDir}/checks.json`);
await scenario.stop();
process.exit(0);
};
process.on('SIGINT', handleShutdown);
process.on('SIGTERM', handleShutdown);
await new Promise(() => {});
}
async function main(): Promise<void> {
const args = process.argv.slice(2);
let command: string | null = null;
let scenario: string | null = null;
let verbose = false;
for (let i = 0; i < args.length; i++) {
if (args[i] === '--command' && i + 1 < args.length) {
command = args[i + 1];
i++;
} else if (args[i] === '--scenario' && i + 1 < args.length) {
scenario = args[i + 1];
i++;
} else if (args[i] === '--verbose') {
verbose = true;
}
}
if (!scenario) {
console.error(
'Usage: runner --scenario <scenario> [--command "<command>"] [--verbose]'
);
console.error(
'Example: runner --scenario initialize --command "tsx examples/clients/typescript/test1.ts"'
);
console.error('Or run without --command for interactive mode');
process.exit(1);
}
if (!command) {
try {
await runInteractiveMode(scenario);
} catch (error) {
console.error('Interactive mode error:', error);
process.exit(1);
}
return;
}
try {
const result = await runConformanceTest(command, scenario);
const denominator = result.checks.filter(
(c) => c.status === 'SUCCESS' || c.status == 'FAILURE'
).length;
const passed = result.checks.filter((c) => c.status === 'SUCCESS').length;
const failed = result.checks.filter((c) => c.status === 'FAILURE').length;
if (verbose) {
console.log(`Checks:\n${JSON.stringify(result.checks, null, 2)}`);
} else {
console.log(`Checks:\n${formatPrettyChecks(result.checks)}`);
}
console.log(`\nTest Results:`);
console.log(`Passed: ${passed}/${denominator}, ${failed} failed`);
if (failed > 0) {
console.log('\nFailed Checks:');
result.checks
.filter((c) => c.status === 'FAILURE')
.forEach((c) => {
console.log(` - ${c.name}: ${c.description}`);
if (c.errorMessage) {
console.log(` Error: ${c.errorMessage}`);
}
});
}
process.exit(failed > 0 ? 1 : 0);
} catch (error) {
console.error('Test runner error:', error);
process.exit(1);
}
}
main();

93
src/runner/server.ts Normal file
View file

@ -0,0 +1,93 @@
import { promises as fs } from 'fs';
import path from 'path';
import { ConformanceCheck } from '../types';
import { getClientScenario } from '../scenarios';
import { ensureResultsDir, createResultDir } from './utils';
export async function runServerConformanceTest(
serverUrl: string,
scenarioName: string
): Promise<{
checks: ConformanceCheck[];
resultDir: string;
}> {
await ensureResultsDir();
const resultDir = createResultDir(scenarioName, 'server');
await fs.mkdir(resultDir, { recursive: true });
// Scenario is guaranteed to exist by CLI validation
const scenario = getClientScenario(scenarioName)!;
console.log(
`Running client scenario '${scenarioName}' against server: ${serverUrl}`
);
const checks = await scenario.run(serverUrl);
await fs.writeFile(
path.join(resultDir, 'checks.json'),
JSON.stringify(checks, null, 2)
);
console.log(`Results saved to ${resultDir}`);
return {
checks,
resultDir
};
}
export function printServerResults(checks: ConformanceCheck[]): {
passed: number;
failed: number;
denominator: number;
} {
const denominator = checks.filter(
(c) => c.status === 'SUCCESS' || c.status === 'FAILURE'
).length;
const passed = checks.filter((c) => c.status === 'SUCCESS').length;
const failed = checks.filter((c) => c.status === 'FAILURE').length;
console.log(`Checks:\n${JSON.stringify(checks, null, 2)}`);
console.log(`\nTest Results:`);
console.log(`Passed: ${passed}/${denominator}, ${failed} failed`);
if (failed > 0) {
console.log('\nFailed Checks:');
checks
.filter((c) => c.status === 'FAILURE')
.forEach((c) => {
console.log(` - ${c.name}: ${c.description}`);
if (c.errorMessage) {
console.log(` Error: ${c.errorMessage}`);
}
});
}
return { passed, failed, denominator };
}
export function printServerSummary(
allResults: { scenario: string; checks: ConformanceCheck[] }[]
): { totalPassed: number; totalFailed: number } {
console.log('\n\n=== SUMMARY ===');
let totalPassed = 0;
let totalFailed = 0;
for (const result of allResults) {
const passed = result.checks.filter((c) => c.status === 'SUCCESS').length;
const failed = result.checks.filter((c) => c.status === 'FAILURE').length;
totalPassed += passed;
totalFailed += failed;
const status = failed === 0 ? '✓' : '✗';
console.log(
`${status} ${result.scenario}: ${passed} passed, ${failed} failed`
);
}
console.log(`\nTotal: ${totalPassed} passed, ${totalFailed} failed`);
return { totalPassed, totalFailed };
}

62
src/runner/utils.ts Normal file
View file

@ -0,0 +1,62 @@
import { promises as fs } from 'fs';
import path from 'path';
import { ConformanceCheck } from '../types';
// ANSI color codes
export const COLORS = {
RESET: '\x1b[0m',
GRAY: '\x1b[90m',
GREEN: '\x1b[32m',
YELLOW: '\x1b[33m',
RED: '\x1b[31m',
BLUE: '\x1b[36m'
};
export function getStatusColor(status: string): string {
switch (status) {
case 'SUCCESS':
return COLORS.GREEN;
case 'FAILURE':
return COLORS.RED;
case 'INFO':
return COLORS.BLUE;
default:
return COLORS.RESET;
}
}
export function formatPrettyChecks(checks: ConformanceCheck[]): string {
// Find the longest id and status for column alignment
const maxIdLength = Math.max(...checks.map((c) => c.id.length));
const maxStatusLength = Math.max(...checks.map((c) => c.status.length));
return checks
.map((check) => {
const timestamp = `${COLORS.GRAY}${check.timestamp}${COLORS.RESET}`;
const id = check.id.padEnd(maxIdLength);
const statusColor = getStatusColor(check.status);
const status = `${statusColor}${check.status.padEnd(maxStatusLength)}${COLORS.RESET}`;
const description = check.description;
const line = `${timestamp} [${id}] ${status} ${description}`;
// Add newline after outgoing responses for better visual separation
return (
line +
(check.id.includes('outgoing') && check.id.includes('response')
? '\n'
: '')
);
})
.join('\n');
}
export async function ensureResultsDir(): Promise<string> {
const resultsDir = path.join(process.cwd(), 'results');
await fs.mkdir(resultsDir, { recursive: true });
return resultsDir;
}
export function createResultDir(scenario: string, prefix = ''): string {
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
const dirName = prefix ? `${prefix}-${scenario}` : scenario;
return path.join('results', `${dirName}-${timestamp}`);
}

44
src/schemas.ts Normal file
View file

@ -0,0 +1,44 @@
import { z } from 'zod';
import { getScenario, getClientScenario } from './scenarios';
// Client command options schema
export const ClientOptionsSchema = z.object({
command: z.string().min(1, 'Command cannot be empty'),
scenario: z
.string()
.min(1, 'Scenario cannot be empty')
.refine(
(scenario) => getScenario(scenario) !== undefined,
(scenario) => ({
message: `Unknown scenario '${scenario}'`
})
),
timeout: z
.string()
.transform((val) => parseInt(val, 10))
.pipe(
z
.number()
.positive('Timeout must be a positive number')
.int('Timeout must be an integer')
),
verbose: z.boolean().optional()
});
export type ClientOptions = z.infer<typeof ClientOptionsSchema>;
// Server command options schema
export const ServerOptionsSchema = z.object({
url: z.string().url('Invalid server URL'),
scenario: z
.string()
.refine(
(scenario) => getClientScenario(scenario) !== undefined,
(scenario) => ({
message: `Unknown scenario '${scenario}'`
})
)
.optional()
});
export type ServerOptions = z.infer<typeof ServerOptionsSchema>;

View file

@ -1,186 +0,0 @@
import { promises as fs } from 'fs';
import path from 'path';
import { ConformanceCheck } from './types.js';
import { getClientScenario } from './scenarios/index.js';
async function ensureResultsDir(): Promise<string> {
const resultsDir = path.join(process.cwd(), 'results');
await fs.mkdir(resultsDir, { recursive: true });
return resultsDir;
}
function createResultDir(scenario: string): string {
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
return path.join('results', `server-${scenario}-${timestamp}`);
}
export async function runServerConformanceTest(
serverUrl: string,
scenarioName: string
): Promise<{
checks: ConformanceCheck[];
resultDir: string;
}> {
await ensureResultsDir();
const resultDir = createResultDir(scenarioName);
await fs.mkdir(resultDir, { recursive: true });
const scenario = getClientScenario(scenarioName);
if (!scenario) {
throw new Error(`Unknown client scenario: ${scenarioName}`);
}
console.log(
`Running client scenario '${scenarioName}' against server: ${serverUrl}`
);
const checks = await scenario.run(serverUrl);
await fs.writeFile(
path.join(resultDir, 'checks.json'),
JSON.stringify(checks, null, 2)
);
console.log(`Results saved to ${resultDir}`);
return {
checks,
resultDir
};
}
async function main(): Promise<void> {
const args = process.argv.slice(2);
let serverUrl: string | null = null;
let scenario: string | null = null;
let runAll = false;
for (let i = 0; i < args.length; i++) {
if (args[i] === '--server-url' && i + 1 < args.length) {
serverUrl = args[i + 1];
i++;
} else if (args[i] === '--scenario' && i + 1 < args.length) {
scenario = args[i + 1];
i++;
} else if (args[i] === '--all') {
runAll = true;
}
}
if (!serverUrl) {
console.error(
'Usage: server-runner --server-url <url> [--scenario <scenario> | --all]'
);
console.error(
'Example: server-runner --server-url http://localhost:3000/mcp --scenario server-initialize'
);
console.error(
'Example: server-runner --server-url http://localhost:3000/mcp --all'
);
process.exit(1);
}
if (!scenario && !runAll) {
console.error('Must specify either --scenario or --all');
process.exit(1);
}
try {
const allResults: { scenario: string; checks: ConformanceCheck[] }[] = [];
if (runAll) {
// Get all server scenarios
const { listClientScenarios } = await import('./scenarios/index.js');
const scenarios = listClientScenarios();
console.log(
`Running ${scenarios.length} scenarios against ${serverUrl}\n`
);
for (const scenarioName of scenarios) {
console.log(`\n=== Running scenario: ${scenarioName} ===`);
try {
const result = await runServerConformanceTest(
serverUrl,
scenarioName
);
allResults.push({ scenario: scenarioName, checks: result.checks });
} catch (error) {
console.error(`Failed to run scenario ${scenarioName}:`, error);
allResults.push({
scenario: scenarioName,
checks: [
{
id: scenarioName,
name: scenarioName,
description: `Failed to run scenario`,
status: 'FAILURE',
timestamp: new Date().toISOString(),
errorMessage:
error instanceof Error ? error.message : String(error)
}
]
});
}
}
// Summary
console.log('\n\n=== SUMMARY ===');
let totalPassed = 0;
let totalFailed = 0;
for (const result of allResults) {
const passed = result.checks.filter(
(c) => c.status === 'SUCCESS'
).length;
const failed = result.checks.filter(
(c) => c.status === 'FAILURE'
).length;
totalPassed += passed;
totalFailed += failed;
const status = failed === 0 ? '✓' : '✗';
console.log(
`${status} ${result.scenario}: ${passed} passed, ${failed} failed`
);
}
console.log(`\nTotal: ${totalPassed} passed, ${totalFailed} failed`);
process.exit(totalFailed > 0 ? 1 : 0);
} else if (scenario) {
const result = await runServerConformanceTest(serverUrl, scenario);
const denominator = result.checks.filter(
(c) => c.status === 'SUCCESS' || c.status == 'FAILURE'
).length;
const passed = result.checks.filter((c) => c.status === 'SUCCESS').length;
const failed = result.checks.filter((c) => c.status === 'FAILURE').length;
console.log(`Checks:\n${JSON.stringify(result.checks, null, 2)}`);
console.log(`\nTest Results:`);
console.log(`Passed: ${passed}/${denominator}, ${failed} failed`);
if (failed > 0) {
console.log('\nFailed Checks:');
result.checks
.filter((c) => c.status === 'FAILURE')
.forEach((c) => {
console.log(` - ${c.name}: ${c.description}`);
if (c.errorMessage) {
console.log(` Error: ${c.errorMessage}`);
}
});
}
process.exit(failed > 0 ? 1 : 0);
}
} catch (error) {
console.error('Server test runner error:', error);
process.exit(1);
}
}
if (import.meta.url === `file://${process.argv[1]}`) {
main();
}

View file

@ -3,6 +3,7 @@
"target": "ES2020", "target": "ES2020",
"module": "ES2020", "module": "ES2020",
"moduleResolution": "node", "moduleResolution": "node",
"resolveJsonModule": true,
"strict": true, "strict": true,
"esModuleInterop": true, "esModuleInterop": true,
"skipLibCheck": true, "skipLibCheck": true,