tsdown + cli with commander (#17)
* feat: tsdown * fix: formatting * automate version updates in cli * fix: docs * scenarios * fix: commander deps
This commit is contained in:
Matt
GitHub
parent
18f48e5dea
commit
7f1515c475
16 changed files with 1600 additions and 539 deletions
3
.gitignore
vendored
3
.gitignore
vendored
|
|
@ -1,3 +1,4 @@
|
|||
node_modules
|
||||
results/
|
||||
lefthook-local.yml
|
||||
lefthook-local.yml
|
||||
dist
|
||||
77
README.md
77
README.md
|
|
@ -1,6 +1,6 @@
|
|||
# MCP Conformance Test Framework
|
||||
|
||||
A framework for testing MCP (Model Context Protocol) client implementations against the specification.
|
||||
A framework for testing MCP (Model Context Protocol) client and server implementations against the specification.
|
||||
|
||||
> [!WARNING] This repository is a work in progress and is unstable. Join the conversation in the #conformance-testing-wg in the MCP Contributors discord.
|
||||
|
||||
|
|
@ -9,19 +9,30 @@ A framework for testing MCP (Model Context Protocol) client implementations agai
|
|||
### Testing Clients
|
||||
|
||||
```bash
|
||||
npm install
|
||||
npm run start -- --command "tsx examples/clients/typescript/test1.ts" --scenario initialize
|
||||
npx @modelcontextprotocol/conformance client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize
|
||||
```
|
||||
|
||||
### Testing Servers
|
||||
|
||||
```bash
|
||||
npm run test:server -- --server-url http://localhost:3000/mcp --all
|
||||
# Run all server scenarios (default)
|
||||
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp
|
||||
|
||||
# Run a single scenario
|
||||
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --scenario server-initialize
|
||||
```
|
||||
|
||||
### List Available Scenarios
|
||||
|
||||
```bash
|
||||
npx @modelcontextprotocol/conformance list
|
||||
```
|
||||
|
||||
## Overview
|
||||
|
||||
The conformance test framework validates MCP client implementations by:
|
||||
The conformance test framework validates MCP implementations by:
|
||||
|
||||
**For Clients:**
|
||||
|
||||
1. Starting a test server for the specified scenario
|
||||
2. Running the client implementation with the test server URL
|
||||
|
|
@ -29,25 +40,53 @@ The conformance test framework validates MCP client implementations by:
|
|||
4. Running conformance checks against the specification
|
||||
5. Generating detailed test results
|
||||
|
||||
**For Servers:**
|
||||
|
||||
1. Connecting to the running server as an MCP client
|
||||
2. Sending test requests and capturing responses
|
||||
3. Running conformance checks against server behavior
|
||||
4. Generating detailed test results
|
||||
|
||||
## Usage
|
||||
|
||||
### Client Testing
|
||||
|
||||
```bash
|
||||
npm run start -- --command "<client-command>" --scenario <scenario-name>
|
||||
npx @modelcontextprotocol/conformance client --command "<client-command>" --scenario <scenario-name> [options]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
||||
- `--command` - The command to run your MCP client (can include flags)
|
||||
- `--scenario` - The test scenario to run (e.g., "initialize")
|
||||
- `--timeout` - Timeout in milliseconds (default: 30000)
|
||||
- `--verbose` - Show verbose output
|
||||
|
||||
The framework appends the server URL as the final argument to your command.
|
||||
|
||||
### Server Testing
|
||||
|
||||
```bash
|
||||
npx @modelcontextprotocol/conformance server --url <url> [--scenario <scenario>]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
||||
- `--url` - URL of the server to test
|
||||
- `--scenario <scenario>` - Test scenario to run (e.g., "server-initialize". Runs all available scenarios by default
|
||||
|
||||
## Test Results
|
||||
|
||||
Results are saved to `results/<scenario>-<timestamp>/`:
|
||||
**Client Testing** - Results are saved to `results/<scenario>-<timestamp>/`:
|
||||
|
||||
- `checks.json` - Array of conformance check results with pass/fail status
|
||||
- `stdout.txt` - Client stdout output
|
||||
- `stderr.txt` - Client stderr output
|
||||
|
||||
**Server Testing** - Results are saved to `results/server-<scenario>-<timestamp>/`:
|
||||
|
||||
- `checks.json` - Array of conformance check results with pass/fail status
|
||||
|
||||
## Example Clients
|
||||
|
||||
- `examples/clients/typescript/test1.ts` - Valid MCP client (passes all checks)
|
||||
|
|
@ -55,10 +94,25 @@ Results are saved to `results/<scenario>-<timestamp>/`:
|
|||
|
||||
## Available Scenarios
|
||||
|
||||
### Client Scenarios
|
||||
|
||||
- **initialize** - Tests MCP client initialization handshake
|
||||
- Validates protocol version
|
||||
- Validates clientInfo (name and version)
|
||||
- Validates server response handling
|
||||
- **tools-call** - Tests tool invocation
|
||||
- **auth/basic-dcr** - Tests OAuth Dynamic Client Registration flow
|
||||
- **auth/basic-metadata-var1** - Tests OAuth with authorization metadata
|
||||
|
||||
### Server Scenarios
|
||||
|
||||
Run `npx @modelcontextprotocol/conformance list --server` to see all available server scenarios, including:
|
||||
|
||||
- **server-initialize** - Tests server initialization and capabilities
|
||||
- **tools-list** - Tests tool listing endpoint
|
||||
- **tools-call-\*** - Various tool invocation scenarios
|
||||
- **resources-\*** - Resource management scenarios
|
||||
- **prompts-\*** - Prompt management scenarios
|
||||
|
||||
## Architecture
|
||||
|
||||
|
|
@ -67,8 +121,13 @@ See `src/runner/DESIGN.md` for detailed architecture documentation.
|
|||
### Key Components
|
||||
|
||||
- **Runner** (`src/runner/`) - Orchestrates test execution and result generation
|
||||
- **Scenarios** (`src/scenarios/`) - Test scenarios with custom server implementations
|
||||
- **Checks** (`src/checks.ts`) - Conformance validation functions
|
||||
- `client.ts` - Client testing implementation
|
||||
- `server.ts` - Server testing implementation
|
||||
- `utils.ts` - Shared utilities
|
||||
- `index.ts` - Public API exports
|
||||
- **CLI** (`src/index.ts`) - Command-line interface using Commander.js
|
||||
- **Scenarios** (`src/scenarios/`) - Test scenarios with expected behaviors
|
||||
- **Checks** (`src/checks/`) - Conformance validation functions
|
||||
- **Types** (`src/types.ts`) - Shared type definitions
|
||||
|
||||
## Adding New Scenarios
|
||||
|
|
|
|||
|
|
@ -1022,8 +1022,8 @@ MCP Conformance Test Server running on http://localhost:3000
|
|||
|
||||
```bash
|
||||
# Single test
|
||||
npm run test:server -- --server-url http://localhost:3000/mcp --scenario server-initialize
|
||||
npm run start -- server --url http://localhost:3000/mcp --scenario server-initialize
|
||||
|
||||
# All tests
|
||||
npm run test:server -- --server-url http://localhost:3000/mcp --all
|
||||
npm run start -- server --url http://localhost:3000/mcp
|
||||
```
|
||||
|
|
|
|||
|
|
@ -147,8 +147,7 @@ This server implements the MCP Server Conformance Requirements specified in `../
|
|||
To run conformance tests against this server:
|
||||
|
||||
```bash
|
||||
cd ../../../
|
||||
npm run test:server -- --server-url http://localhost:3000/mcp --all
|
||||
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp
|
||||
```
|
||||
|
||||
## Implementation Notes
|
||||
|
|
|
|||
788
package-lock.json
generated
788
package-lock.json
generated
File diff suppressed because it is too large
Load diff
12
package.json
12
package.json
|
|
@ -3,13 +3,19 @@
|
|||
"version": "0.1.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"start": "tsx src/runner.ts",
|
||||
"test:server": "tsx src/server-runner.ts",
|
||||
"start": "tsx src/index.ts",
|
||||
"test": "jest",
|
||||
"build": "tsdown src/index.ts --minify --clean --target node20",
|
||||
"lint": "eslint src/ && prettier --check .",
|
||||
"lint:fix": "eslint src/ --fix && prettier --write .",
|
||||
"lint:fix_check": "npm run lint:fix && git diff --exit-code --quiet"
|
||||
},
|
||||
"files": [
|
||||
"dist"
|
||||
],
|
||||
"bin": {
|
||||
"conformance": "dist/index.mjs"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@eslint/js": "^9.8.0",
|
||||
"@jest/globals": "^29.7.0",
|
||||
|
|
@ -20,12 +26,14 @@
|
|||
"jest": "^29.7.0",
|
||||
"prettier": "3.6.2",
|
||||
"ts-jest": "^29.1.1",
|
||||
"tsdown": "^0.16.2",
|
||||
"tsx": "^4.7.0",
|
||||
"typescript": "^5.5.4",
|
||||
"typescript-eslint": "^8.0.0"
|
||||
},
|
||||
"dependencies": {
|
||||
"@modelcontextprotocol/sdk": "^1.20.1",
|
||||
"commander": "^14.0.2",
|
||||
"express": "^5.1.0",
|
||||
"lefthook": "^2.0.2",
|
||||
"zod": "^3.25.76"
|
||||
|
|
|
|||
164
src/index.ts
Normal file
164
src/index.ts
Normal file
|
|
@ -0,0 +1,164 @@
|
|||
#!/usr/bin/env node
|
||||
|
||||
import { Command } from 'commander';
|
||||
import { ZodError } from 'zod';
|
||||
import {
|
||||
runConformanceTest,
|
||||
printClientResults,
|
||||
runServerConformanceTest,
|
||||
printServerResults,
|
||||
printServerSummary
|
||||
} from './runner';
|
||||
import { listScenarios, listClientScenarios } from './scenarios';
|
||||
import { ConformanceCheck } from './types';
|
||||
import { ClientOptionsSchema, ServerOptionsSchema } from './schemas';
|
||||
import packageJson from '../package.json';
|
||||
|
||||
const program = new Command();
|
||||
|
||||
program
|
||||
.name('conformance')
|
||||
.description('MCP Conformance Test Suite')
|
||||
.version(packageJson.version);
|
||||
|
||||
// Client command - tests a client implementation against scenarios
|
||||
program
|
||||
.command('client')
|
||||
.description('Run conformance tests against a client implementation')
|
||||
.requiredOption('--command <command>', 'Command to run the client')
|
||||
.requiredOption('--scenario <scenario>', 'Scenario to test')
|
||||
.option('--timeout <ms>', 'Timeout in milliseconds', '30000')
|
||||
.option('--verbose', 'Show verbose output')
|
||||
.action(async (options) => {
|
||||
try {
|
||||
// Validate options with Zod
|
||||
const validated = ClientOptionsSchema.parse(options);
|
||||
|
||||
const result = await runConformanceTest(
|
||||
validated.command,
|
||||
validated.scenario,
|
||||
validated.timeout
|
||||
);
|
||||
|
||||
const { failed } = printClientResults(
|
||||
result.checks,
|
||||
validated.verbose ?? false
|
||||
);
|
||||
process.exit(failed > 0 ? 1 : 0);
|
||||
} catch (error) {
|
||||
if (error instanceof ZodError) {
|
||||
console.error('Validation error:');
|
||||
error.errors.forEach((err) => {
|
||||
console.error(` ${err.path.join('.')}: ${err.message}`);
|
||||
});
|
||||
console.error('\nAvailable client scenarios:');
|
||||
listScenarios().forEach((s) => console.error(` - ${s}`));
|
||||
process.exit(1);
|
||||
}
|
||||
console.error('Client test error:', error);
|
||||
process.exit(1);
|
||||
}
|
||||
});
|
||||
|
||||
// Server command - tests a server implementation
|
||||
program
|
||||
.command('server')
|
||||
.description('Run conformance tests against a server implementation')
|
||||
.requiredOption('--url <url>', 'URL of the server to test')
|
||||
.option(
|
||||
'--scenario <scenario>',
|
||||
'Scenario to test (defaults to all scenarios if not specified)'
|
||||
)
|
||||
.action(async (options) => {
|
||||
try {
|
||||
// Validate options with Zod
|
||||
const validated = ServerOptionsSchema.parse(options);
|
||||
|
||||
// If a single scenario is specified, run just that one
|
||||
if (validated.scenario) {
|
||||
const result = await runServerConformanceTest(
|
||||
validated.url,
|
||||
validated.scenario
|
||||
);
|
||||
|
||||
const { failed } = printServerResults(result.checks);
|
||||
process.exit(failed > 0 ? 1 : 0);
|
||||
} else {
|
||||
// Run all scenarios
|
||||
const scenarios = listClientScenarios();
|
||||
console.log(
|
||||
`Running ${scenarios.length} scenarios against ${validated.url}\n`
|
||||
);
|
||||
|
||||
const allResults: { scenario: string; checks: ConformanceCheck[] }[] =
|
||||
[];
|
||||
|
||||
for (const scenarioName of scenarios) {
|
||||
console.log(`\n=== Running scenario: ${scenarioName} ===`);
|
||||
try {
|
||||
const result = await runServerConformanceTest(
|
||||
validated.url,
|
||||
scenarioName
|
||||
);
|
||||
allResults.push({ scenario: scenarioName, checks: result.checks });
|
||||
} catch (error) {
|
||||
console.error(`Failed to run scenario ${scenarioName}:`, error);
|
||||
allResults.push({
|
||||
scenario: scenarioName,
|
||||
checks: [
|
||||
{
|
||||
id: scenarioName,
|
||||
name: scenarioName,
|
||||
description: 'Failed to run scenario',
|
||||
status: 'FAILURE',
|
||||
timestamp: new Date().toISOString(),
|
||||
errorMessage:
|
||||
error instanceof Error ? error.message : String(error)
|
||||
}
|
||||
]
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const { totalFailed } = printServerSummary(allResults);
|
||||
process.exit(totalFailed > 0 ? 1 : 0);
|
||||
}
|
||||
} catch (error) {
|
||||
if (error instanceof ZodError) {
|
||||
console.error('Validation error:');
|
||||
error.errors.forEach((err) => {
|
||||
console.error(` ${err.path.join('.')}: ${err.message}`);
|
||||
});
|
||||
console.error('\nAvailable server scenarios:');
|
||||
listClientScenarios().forEach((s) => console.error(` - ${s}`));
|
||||
process.exit(1);
|
||||
}
|
||||
console.error('Server test error:', error);
|
||||
process.exit(1);
|
||||
}
|
||||
});
|
||||
|
||||
// List scenarios command
|
||||
program
|
||||
.command('list')
|
||||
.description('List available test scenarios')
|
||||
.option('--client', 'List client scenarios')
|
||||
.option('--server', 'List server scenarios')
|
||||
.action((options) => {
|
||||
if (options.server || (!options.client && !options.server)) {
|
||||
console.log('Server scenarios (test against a server):');
|
||||
const serverScenarios = listClientScenarios();
|
||||
serverScenarios.forEach((s) => console.log(` - ${s}`));
|
||||
}
|
||||
|
||||
if (options.client || (!options.client && !options.server)) {
|
||||
if (options.server || (!options.client && !options.server)) {
|
||||
console.log('');
|
||||
}
|
||||
console.log('Client scenarios (test against a client):');
|
||||
const clientScenarios = listScenarios();
|
||||
clientScenarios.forEach((s) => console.log(` - ${s}`));
|
||||
}
|
||||
});
|
||||
|
||||
program.parse();
|
||||
|
|
@ -1,3 +0,0 @@
|
|||
import { runConformanceTest } from './runner/index.js';
|
||||
|
||||
export { runConformanceTest };
|
||||
|
|
@ -2,20 +2,45 @@
|
|||
|
||||
## Overview
|
||||
|
||||
The conformance test runner is a framework for testing MCP client implementations against the MCP specification. It executes client implementations in controlled scenarios and validates their behavior through a series of conformance checks.
|
||||
The conformance test runner is a framework for testing both MCP client and server implementations against the MCP specification. It provides two testing modes:
|
||||
|
||||
1. **Client Testing** - Executes client implementations in controlled scenarios and validates their behavior
|
||||
2. **Server Testing** - Tests server implementations by acting as an MCP client and validating responses
|
||||
|
||||
## Architecture
|
||||
|
||||
### File Structure
|
||||
|
||||
```
|
||||
src/runner/
|
||||
├── index.ts # Exports all public functions
|
||||
├── client.ts # Client testing implementation
|
||||
├── server.ts # Server testing implementation
|
||||
└── utils.ts # Shared utilities (formatting, file I/O, ANSI colors)
|
||||
```
|
||||
|
||||
### Components
|
||||
|
||||
**Client Testing:**
|
||||
|
||||
1. **Runner** - Orchestrates test execution
|
||||
2. **Test Server** - MCP server (and optionally auth server for auth scenarios)
|
||||
3. **Client Process** - The MCP client implementation under test
|
||||
4. **Scenario** - A specific test case with expected behaviors
|
||||
5. **Checks** - Validation functions that produce ConformanceCheck results
|
||||
|
||||
**Server Testing:**
|
||||
|
||||
1. **Runner** - Orchestrates test execution
|
||||
2. **Test Client** - MCP SDK client that connects to the server under test
|
||||
3. **Server Process** - The MCP server implementation under test (external)
|
||||
4. **Scenario** - A specific test case with expected server behaviors
|
||||
5. **Checks** - Validation functions that produce ConformanceCheck results
|
||||
|
||||
### Execution Flow
|
||||
|
||||
**Client Testing:**
|
||||
|
||||
```
|
||||
1. Runner starts test server(s) on available port(s)
|
||||
2. Runner spawns client process with server URL as final argument
|
||||
|
|
@ -25,44 +50,124 @@ The conformance test runner is a framework for testing MCP client implementation
|
|||
6. Results are written to results/<scenario>-<timestamp>/
|
||||
```
|
||||
|
||||
## CLI Interface
|
||||
**Server Testing:**
|
||||
|
||||
```bash
|
||||
npm run start -- --command "tsx examples/clients/typescript/test1.ts --verbose" --scenario initialize
|
||||
```
|
||||
1. User starts their server implementation
|
||||
2. Runner connects to server URL as an MCP client
|
||||
3. Runner sends requests and captures responses
|
||||
4. Scenario-specific checks validate server behavior
|
||||
5. Results are written to results/server-<scenario>-<timestamp>/
|
||||
```
|
||||
|
||||
### Arguments
|
||||
## CLI Interface
|
||||
|
||||
The conformance suite provides a unified CLI with two main commands:
|
||||
|
||||
### Client Testing
|
||||
|
||||
```bash
|
||||
# Run a single client scenario
|
||||
npm run start -- client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize
|
||||
|
||||
# With verbose output
|
||||
npm run start -- client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize --verbose
|
||||
|
||||
# With custom timeout
|
||||
npm run start -- client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize --timeout 60000
|
||||
```
|
||||
|
||||
**Arguments:**
|
||||
|
||||
- `--command` - The command to run the client (can include existing flags)
|
||||
- `--scenario` - The scenario to test (e.g., "initialize", "list-tools", "call-tool")
|
||||
- `--scenario` - The scenario to test (e.g., "initialize", "tools-call")
|
||||
- `--timeout` - Timeout in milliseconds (default: 30000)
|
||||
- `--verbose` - Show verbose output (JSON format)
|
||||
|
||||
The runner will append the server URL as the final argument to the command.
|
||||
|
||||
### Server Testing
|
||||
|
||||
```bash
|
||||
# Run a single server scenario
|
||||
npm run start -- server --url http://localhost:3000/mcp --scenario server-initialize
|
||||
|
||||
# Run all server scenarios (default when no --scenario specified)
|
||||
npm run start -- server --url http://localhost:3000/mcp
|
||||
```
|
||||
|
||||
**Arguments:**
|
||||
|
||||
- `--url` - URL of the server to test
|
||||
- `--scenario <scenario>` - Scenario to test (optional, defaults to all scenarios if not specified)
|
||||
|
||||
### List Available Scenarios
|
||||
|
||||
```bash
|
||||
# List all scenarios
|
||||
npm run start -- list
|
||||
|
||||
# List only client scenarios
|
||||
npm run start -- list --client
|
||||
|
||||
# List only server scenarios
|
||||
npm run start -- list --server
|
||||
```
|
||||
|
||||
## Validation
|
||||
|
||||
All CLI arguments are validated using Zod schemas (`src/schemas.ts`) before being passed to runner functions:
|
||||
|
||||
**Client Validation:**
|
||||
|
||||
- Command is non-empty
|
||||
- Scenario exists in available scenarios
|
||||
- Timeout is a positive integer
|
||||
|
||||
**Server Validation:**
|
||||
|
||||
- URL is a valid HTTP/HTTPS URL
|
||||
- All specified scenarios exist (if provided)
|
||||
- Defaults to all scenarios when no `--scenario` is specified
|
||||
|
||||
## Scenarios
|
||||
|
||||
A scenario represents a specific test case that validates one or more aspects of MCP client behavior. Each scenario:
|
||||
A scenario represents a specific test case that validates one or more aspects of MCP behavior. Each scenario:
|
||||
|
||||
- Configures the test server with expected behavior
|
||||
- Configures the test environment with expected behavior
|
||||
- May run multiple conformance checks
|
||||
- Validates both client behavior and server responses
|
||||
- Validates both request/response patterns and protocol compliance
|
||||
|
||||
Example scenarios:
|
||||
### Client Scenario Examples
|
||||
|
||||
- `initialize` - Tests client initialization handshake
|
||||
- `list-tools` - Tests tool discovery
|
||||
- `call-tool` - Tests tool invocation
|
||||
- `auth-flow` - Tests OAuth/authorization flows (requires auth server)
|
||||
- `tools-call` - Tests tool invocation
|
||||
- `auth/basic-dcr` - Tests OAuth Dynamic Client Registration flow
|
||||
- `auth/basic-metadata-var1` - Tests OAuth with authorization metadata variation 1
|
||||
|
||||
### Server Scenario Examples
|
||||
|
||||
- `server-initialize` - Tests server initialization and capabilities
|
||||
- `tools-list` - Tests tool listing endpoint
|
||||
- `tools-call-simple-text` - Tests tool invocation with text response
|
||||
- `resources-list` - Tests resource listing
|
||||
- `prompts-list` - Tests prompt listing
|
||||
- `logging-set-level` - Tests logging level configuration
|
||||
|
||||
## Output Structure
|
||||
|
||||
Results are written to: `results/<scenario>-<timestamp>/`
|
||||
Results are written to: `results/<scenario>-<timestamp>/` or `results/server-<scenario>-<timestamp>/`
|
||||
|
||||
Files:
|
||||
**Client Testing Files:**
|
||||
|
||||
- `checks.json` - Array of ConformanceCheck objects with validation results
|
||||
- `stdout.txt` - Complete stdout from the client process
|
||||
- `stderr.txt` - Complete stderr from the client process
|
||||
|
||||
**Server Testing Files:**
|
||||
|
||||
- `checks.json` - Array of ConformanceCheck objects with validation results
|
||||
|
||||
### checks.json Format
|
||||
|
||||
```json
|
||||
|
|
@ -84,6 +189,30 @@ Files:
|
|||
]
|
||||
```
|
||||
|
||||
## Programmatic Usage
|
||||
|
||||
The runner can also be used programmatically:
|
||||
|
||||
```typescript
|
||||
import { runConformanceTest, printClientResults, runServerConformanceTest, printServerResults } from './runner/index.js';
|
||||
|
||||
// Test a client
|
||||
const clientResult = await runConformanceTest('tsx my-client.ts', 'initialize', 30000);
|
||||
const { failed } = printClientResults(clientResult.checks);
|
||||
|
||||
// Test a server
|
||||
const serverResult = await runServerConformanceTest('http://localhost:3000/mcp', 'server-initialize');
|
||||
const { failed } = printServerResults(serverResult.checks);
|
||||
```
|
||||
|
||||
## Shared Utilities
|
||||
|
||||
The `utils.ts` module provides shared functionality:
|
||||
|
||||
- **File Operations:** `ensureResultsDir()`, `createResultDir()`
|
||||
- **Formatting:** `formatPrettyChecks()` for colored console output
|
||||
- **Styling:** ANSI color constants and helpers
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- **Test Suites** - Group multiple scenarios for convenience
|
||||
|
|
@ -91,3 +220,5 @@ Files:
|
|||
- **Parallel Execution** - Run multiple scenarios concurrently
|
||||
- **Custom Servers** - Allow custom MCP server implementations for advanced scenarios
|
||||
- **Report Formats** - HTML, Markdown, or other human-readable formats
|
||||
- **Watch Mode** - Automatically re-run tests on file changes
|
||||
- **Coverage Tracking** - Track which parts of the spec are tested
|
||||
|
|
|
|||
207
src/runner/client.ts
Normal file
207
src/runner/client.ts
Normal file
|
|
@ -0,0 +1,207 @@
|
|||
import { spawn } from 'child_process';
|
||||
import { promises as fs } from 'fs';
|
||||
import path from 'path';
|
||||
import { ConformanceCheck } from '../types';
|
||||
import { getScenario } from '../scenarios';
|
||||
import { ensureResultsDir, createResultDir, formatPrettyChecks } from './utils';
|
||||
|
||||
export interface ClientExecutionResult {
|
||||
exitCode: number;
|
||||
stdout: string;
|
||||
stderr: string;
|
||||
timedOut: boolean;
|
||||
}
|
||||
|
||||
async function executeClient(
|
||||
command: string,
|
||||
serverUrl: string,
|
||||
timeout: number = 30000
|
||||
): Promise<ClientExecutionResult> {
|
||||
const commandParts = command.split(' ');
|
||||
const executable = commandParts[0];
|
||||
const args = [...commandParts.slice(1), serverUrl];
|
||||
|
||||
let stdout = '';
|
||||
let stderr = '';
|
||||
let timedOut = false;
|
||||
|
||||
return new Promise((resolve) => {
|
||||
const process = spawn(executable, args, {
|
||||
shell: true,
|
||||
stdio: 'pipe'
|
||||
});
|
||||
|
||||
const timeoutHandle = setTimeout(() => {
|
||||
timedOut = true;
|
||||
process.kill();
|
||||
}, timeout);
|
||||
|
||||
if (process.stdout) {
|
||||
process.stdout.on('data', (data) => {
|
||||
stdout += data.toString();
|
||||
});
|
||||
}
|
||||
|
||||
if (process.stderr) {
|
||||
process.stderr.on('data', (data) => {
|
||||
stderr += data.toString();
|
||||
});
|
||||
}
|
||||
|
||||
process.on('close', (code) => {
|
||||
clearTimeout(timeoutHandle);
|
||||
resolve({
|
||||
exitCode: code || 0,
|
||||
stdout,
|
||||
stderr,
|
||||
timedOut
|
||||
});
|
||||
});
|
||||
|
||||
process.on('error', (error) => {
|
||||
clearTimeout(timeoutHandle);
|
||||
resolve({
|
||||
exitCode: -1,
|
||||
stdout,
|
||||
stderr: stderr + `\nProcess error: ${error.message}`,
|
||||
timedOut
|
||||
});
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
export async function runConformanceTest(
|
||||
clientCommand: string,
|
||||
scenarioName: string,
|
||||
timeout: number = 30000
|
||||
): Promise<{
|
||||
checks: ConformanceCheck[];
|
||||
clientOutput: ClientExecutionResult;
|
||||
resultDir: string;
|
||||
}> {
|
||||
await ensureResultsDir();
|
||||
const resultDir = createResultDir(scenarioName);
|
||||
await fs.mkdir(resultDir, { recursive: true });
|
||||
|
||||
// Scenario is guaranteed to exist by CLI validation
|
||||
const scenario = getScenario(scenarioName)!;
|
||||
|
||||
console.log(`Starting scenario: ${scenarioName}`);
|
||||
const urls = await scenario.start();
|
||||
|
||||
console.log(`Executing client: ${clientCommand} ${urls.serverUrl}`);
|
||||
|
||||
try {
|
||||
const clientOutput = await executeClient(
|
||||
clientCommand,
|
||||
urls.serverUrl,
|
||||
timeout
|
||||
);
|
||||
|
||||
// Print stdout/stderr if client exited with nonzero code
|
||||
if (clientOutput.exitCode !== 0) {
|
||||
console.error(`\nClient exited with code ${clientOutput.exitCode}`);
|
||||
if (clientOutput.stdout) {
|
||||
console.error(`\nStdout:\n${clientOutput.stdout}`);
|
||||
}
|
||||
if (clientOutput.stderr) {
|
||||
console.error(`\nStderr:\n${clientOutput.stderr}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (clientOutput.timedOut) {
|
||||
console.error(`\nClient timed out after ${timeout}ms`);
|
||||
}
|
||||
|
||||
const checks = scenario.getChecks();
|
||||
|
||||
await fs.writeFile(
|
||||
path.join(resultDir, 'checks.json'),
|
||||
JSON.stringify(checks, null, 2)
|
||||
);
|
||||
|
||||
await fs.writeFile(path.join(resultDir, 'stdout.txt'), clientOutput.stdout);
|
||||
|
||||
await fs.writeFile(path.join(resultDir, 'stderr.txt'), clientOutput.stderr);
|
||||
|
||||
console.log(`Results saved to ${resultDir}`);
|
||||
|
||||
return {
|
||||
checks,
|
||||
clientOutput,
|
||||
resultDir
|
||||
};
|
||||
} finally {
|
||||
await scenario.stop();
|
||||
}
|
||||
}
|
||||
|
||||
export function printClientResults(
|
||||
checks: ConformanceCheck[],
|
||||
verbose: boolean = false
|
||||
): { passed: number; failed: number; denominator: number } {
|
||||
const denominator = checks.filter(
|
||||
(c) => c.status === 'SUCCESS' || c.status === 'FAILURE'
|
||||
).length;
|
||||
const passed = checks.filter((c) => c.status === 'SUCCESS').length;
|
||||
const failed = checks.filter((c) => c.status === 'FAILURE').length;
|
||||
|
||||
if (verbose) {
|
||||
console.log(`Checks:\n${JSON.stringify(checks, null, 2)}`);
|
||||
} else {
|
||||
console.log(`Checks:\n${formatPrettyChecks(checks)}`);
|
||||
}
|
||||
|
||||
console.log(`\nTest Results:`);
|
||||
console.log(`Passed: ${passed}/${denominator}, ${failed} failed`);
|
||||
|
||||
if (failed > 0) {
|
||||
console.log('\nFailed Checks:');
|
||||
checks
|
||||
.filter((c) => c.status === 'FAILURE')
|
||||
.forEach((c) => {
|
||||
console.log(` - ${c.name}: ${c.description}`);
|
||||
if (c.errorMessage) {
|
||||
console.log(` Error: ${c.errorMessage}`);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
return { passed, failed, denominator };
|
||||
}
|
||||
|
||||
export async function runInteractiveMode(scenarioName: string): Promise<void> {
|
||||
await ensureResultsDir();
|
||||
const resultDir = createResultDir(scenarioName);
|
||||
await fs.mkdir(resultDir, { recursive: true });
|
||||
|
||||
// Scenario is guaranteed to exist by CLI validation
|
||||
const scenario = getScenario(scenarioName)!;
|
||||
|
||||
console.log(`Starting scenario: ${scenarioName}`);
|
||||
const urls = await scenario.start();
|
||||
|
||||
console.log(`Server URL: ${urls.serverUrl}`);
|
||||
console.log('Press Ctrl+C to stop and save checks...');
|
||||
|
||||
const handleShutdown = async () => {
|
||||
console.log('\nShutting down...');
|
||||
|
||||
const checks = scenario.getChecks();
|
||||
await fs.writeFile(
|
||||
path.join(resultDir, 'checks.json'),
|
||||
JSON.stringify(checks, null, 2)
|
||||
);
|
||||
|
||||
console.log(`\nChecks:\n${JSON.stringify(checks, null, 2)}`);
|
||||
console.log(`\nChecks saved to ${resultDir}/checks.json`);
|
||||
|
||||
await scenario.stop();
|
||||
process.exit(0);
|
||||
};
|
||||
|
||||
process.on('SIGINT', handleShutdown);
|
||||
process.on('SIGTERM', handleShutdown);
|
||||
|
||||
await new Promise(() => {});
|
||||
}
|
||||
|
|
@ -1,312 +1,23 @@
|
|||
import { spawn } from 'child_process';
|
||||
import { promises as fs } from 'fs';
|
||||
import path from 'path';
|
||||
import { ConformanceCheck } from '../types.js';
|
||||
import { getScenario } from '../scenarios/index.js';
|
||||
// Export client functions
|
||||
export {
|
||||
runConformanceTest,
|
||||
printClientResults,
|
||||
runInteractiveMode,
|
||||
type ClientExecutionResult
|
||||
} from './client';
|
||||
|
||||
// ANSI color codes
|
||||
const COLORS = {
|
||||
RESET: '\x1b[0m',
|
||||
GRAY: '\x1b[90m',
|
||||
GREEN: '\x1b[32m',
|
||||
YELLOW: '\x1b[33m',
|
||||
RED: '\x1b[31m',
|
||||
BLUE: '\x1b[36m'
|
||||
};
|
||||
// Export server functions
|
||||
export {
|
||||
runServerConformanceTest,
|
||||
printServerResults,
|
||||
printServerSummary
|
||||
} from './server';
|
||||
|
||||
function getStatusColor(status: string): string {
|
||||
switch (status) {
|
||||
case 'SUCCESS':
|
||||
return COLORS.GREEN;
|
||||
case 'FAILURE':
|
||||
return COLORS.RED;
|
||||
case 'INFO':
|
||||
return COLORS.BLUE;
|
||||
default:
|
||||
return COLORS.RESET;
|
||||
}
|
||||
}
|
||||
|
||||
function formatPrettyChecks(checks: ConformanceCheck[]): string {
|
||||
// Find the longest id and status for column alignment
|
||||
const maxIdLength = Math.max(...checks.map((c) => c.id.length));
|
||||
const maxStatusLength = Math.max(...checks.map((c) => c.status.length));
|
||||
|
||||
return checks
|
||||
.map((check) => {
|
||||
const timestamp = `${COLORS.GRAY}${check.timestamp}${COLORS.RESET}`;
|
||||
const id = check.id.padEnd(maxIdLength);
|
||||
const statusColor = getStatusColor(check.status);
|
||||
const status = `${statusColor}${check.status.padEnd(maxStatusLength)}${COLORS.RESET}`;
|
||||
const description = check.description;
|
||||
const line = `${timestamp} [${id}] ${status} ${description}`;
|
||||
// Add newline after outgoing responses for better visual separation
|
||||
return (
|
||||
line +
|
||||
(check.id.includes('outgoing') && check.id.includes('response')
|
||||
? '\n'
|
||||
: '')
|
||||
);
|
||||
})
|
||||
.join('\n');
|
||||
}
|
||||
|
||||
export interface ClientExecutionResult {
|
||||
exitCode: number;
|
||||
stdout: string;
|
||||
stderr: string;
|
||||
timedOut: boolean;
|
||||
}
|
||||
|
||||
async function ensureResultsDir(): Promise<string> {
|
||||
const resultsDir = path.join(process.cwd(), 'results');
|
||||
await fs.mkdir(resultsDir, { recursive: true });
|
||||
return resultsDir;
|
||||
}
|
||||
|
||||
function createResultDir(scenario: string): string {
|
||||
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
|
||||
return path.join('results', `${scenario}-${timestamp}`);
|
||||
}
|
||||
|
||||
async function executeClient(
|
||||
command: string,
|
||||
serverUrl: string,
|
||||
timeout: number = 30000
|
||||
): Promise<ClientExecutionResult> {
|
||||
const commandParts = command.split(' ');
|
||||
const executable = commandParts[0];
|
||||
const args = [...commandParts.slice(1), serverUrl];
|
||||
|
||||
let stdout = '';
|
||||
let stderr = '';
|
||||
let timedOut = false;
|
||||
|
||||
return new Promise((resolve) => {
|
||||
const process = spawn(executable, args, {
|
||||
shell: true,
|
||||
stdio: 'pipe'
|
||||
});
|
||||
|
||||
const timeoutHandle = setTimeout(() => {
|
||||
timedOut = true;
|
||||
process.kill();
|
||||
}, timeout);
|
||||
|
||||
if (process.stdout) {
|
||||
process.stdout.on('data', (data) => {
|
||||
stdout += data.toString();
|
||||
});
|
||||
}
|
||||
|
||||
if (process.stderr) {
|
||||
process.stderr.on('data', (data) => {
|
||||
stderr += data.toString();
|
||||
});
|
||||
}
|
||||
|
||||
process.on('close', (code) => {
|
||||
clearTimeout(timeoutHandle);
|
||||
resolve({
|
||||
exitCode: code || 0,
|
||||
stdout,
|
||||
stderr,
|
||||
timedOut
|
||||
});
|
||||
});
|
||||
|
||||
process.on('error', (error) => {
|
||||
clearTimeout(timeoutHandle);
|
||||
resolve({
|
||||
exitCode: -1,
|
||||
stdout,
|
||||
stderr: stderr + `\nProcess error: ${error.message}`,
|
||||
timedOut
|
||||
});
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
export async function runConformanceTest(
|
||||
clientCommand: string,
|
||||
scenarioName: string,
|
||||
timeout: number = 30000
|
||||
): Promise<{
|
||||
checks: ConformanceCheck[];
|
||||
clientOutput: ClientExecutionResult;
|
||||
resultDir: string;
|
||||
}> {
|
||||
await ensureResultsDir();
|
||||
const resultDir = createResultDir(scenarioName);
|
||||
await fs.mkdir(resultDir, { recursive: true });
|
||||
|
||||
const scenario = getScenario(scenarioName);
|
||||
if (!scenario) {
|
||||
throw new Error(`Unknown scenario: ${scenarioName}`);
|
||||
}
|
||||
|
||||
console.log(`Starting scenario: ${scenarioName}`);
|
||||
const urls = await scenario.start();
|
||||
|
||||
console.log(`Executing client: ${clientCommand} ${urls.serverUrl}`);
|
||||
|
||||
try {
|
||||
const clientOutput = await executeClient(
|
||||
clientCommand,
|
||||
urls.serverUrl,
|
||||
timeout
|
||||
);
|
||||
|
||||
// Print stdout/stderr if client exited with nonzero code
|
||||
if (clientOutput.exitCode !== 0) {
|
||||
console.error(`\nClient exited with code ${clientOutput.exitCode}`);
|
||||
if (clientOutput.stdout) {
|
||||
console.error(`\nStdout:\n${clientOutput.stdout}`);
|
||||
}
|
||||
if (clientOutput.stderr) {
|
||||
console.error(`\nStderr:\n${clientOutput.stderr}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (clientOutput.timedOut) {
|
||||
console.error(`\nClient timed out after ${timeout}ms`);
|
||||
}
|
||||
|
||||
const checks = scenario.getChecks();
|
||||
|
||||
await fs.writeFile(
|
||||
path.join(resultDir, 'checks.json'),
|
||||
JSON.stringify(checks, null, 2)
|
||||
);
|
||||
|
||||
await fs.writeFile(path.join(resultDir, 'stdout.txt'), clientOutput.stdout);
|
||||
|
||||
await fs.writeFile(path.join(resultDir, 'stderr.txt'), clientOutput.stderr);
|
||||
|
||||
console.log(`Results saved to ${resultDir}`);
|
||||
|
||||
return {
|
||||
checks,
|
||||
clientOutput,
|
||||
resultDir
|
||||
};
|
||||
} finally {
|
||||
await scenario.stop();
|
||||
}
|
||||
}
|
||||
|
||||
async function runInteractiveMode(scenarioName: string): Promise<void> {
|
||||
await ensureResultsDir();
|
||||
const resultDir = createResultDir(scenarioName);
|
||||
await fs.mkdir(resultDir, { recursive: true });
|
||||
|
||||
const scenario = getScenario(scenarioName);
|
||||
if (!scenario) {
|
||||
throw new Error(`Unknown scenario: ${scenarioName}`);
|
||||
}
|
||||
|
||||
console.log(`Starting scenario: ${scenarioName}`);
|
||||
const urls = await scenario.start();
|
||||
|
||||
console.log(`Server URL: ${urls.serverUrl}`);
|
||||
console.log('Press Ctrl+C to stop and save checks...');
|
||||
|
||||
const handleShutdown = async () => {
|
||||
console.log('\nShutting down...');
|
||||
|
||||
const checks = scenario.getChecks();
|
||||
await fs.writeFile(
|
||||
path.join(resultDir, 'checks.json'),
|
||||
JSON.stringify(checks, null, 2)
|
||||
);
|
||||
|
||||
console.log(`\nChecks:\n${JSON.stringify(checks, null, 2)}`);
|
||||
console.log(`\nChecks saved to ${resultDir}/checks.json`);
|
||||
|
||||
await scenario.stop();
|
||||
process.exit(0);
|
||||
};
|
||||
|
||||
process.on('SIGINT', handleShutdown);
|
||||
process.on('SIGTERM', handleShutdown);
|
||||
|
||||
await new Promise(() => {});
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const args = process.argv.slice(2);
|
||||
let command: string | null = null;
|
||||
let scenario: string | null = null;
|
||||
let verbose = false;
|
||||
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
if (args[i] === '--command' && i + 1 < args.length) {
|
||||
command = args[i + 1];
|
||||
i++;
|
||||
} else if (args[i] === '--scenario' && i + 1 < args.length) {
|
||||
scenario = args[i + 1];
|
||||
i++;
|
||||
} else if (args[i] === '--verbose') {
|
||||
verbose = true;
|
||||
}
|
||||
}
|
||||
|
||||
if (!scenario) {
|
||||
console.error(
|
||||
'Usage: runner --scenario <scenario> [--command "<command>"] [--verbose]'
|
||||
);
|
||||
console.error(
|
||||
'Example: runner --scenario initialize --command "tsx examples/clients/typescript/test1.ts"'
|
||||
);
|
||||
console.error('Or run without --command for interactive mode');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!command) {
|
||||
try {
|
||||
await runInteractiveMode(scenario);
|
||||
} catch (error) {
|
||||
console.error('Interactive mode error:', error);
|
||||
process.exit(1);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const result = await runConformanceTest(command, scenario);
|
||||
|
||||
const denominator = result.checks.filter(
|
||||
(c) => c.status === 'SUCCESS' || c.status == 'FAILURE'
|
||||
).length;
|
||||
const passed = result.checks.filter((c) => c.status === 'SUCCESS').length;
|
||||
const failed = result.checks.filter((c) => c.status === 'FAILURE').length;
|
||||
|
||||
if (verbose) {
|
||||
console.log(`Checks:\n${JSON.stringify(result.checks, null, 2)}`);
|
||||
} else {
|
||||
console.log(`Checks:\n${formatPrettyChecks(result.checks)}`);
|
||||
}
|
||||
|
||||
console.log(`\nTest Results:`);
|
||||
console.log(`Passed: ${passed}/${denominator}, ${failed} failed`);
|
||||
|
||||
if (failed > 0) {
|
||||
console.log('\nFailed Checks:');
|
||||
result.checks
|
||||
.filter((c) => c.status === 'FAILURE')
|
||||
.forEach((c) => {
|
||||
console.log(` - ${c.name}: ${c.description}`);
|
||||
if (c.errorMessage) {
|
||||
console.log(` Error: ${c.errorMessage}`);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
process.exit(failed > 0 ? 1 : 0);
|
||||
} catch (error) {
|
||||
console.error('Test runner error:', error);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
main();
|
||||
// Export utilities
|
||||
export {
|
||||
ensureResultsDir,
|
||||
createResultDir,
|
||||
formatPrettyChecks,
|
||||
getStatusColor,
|
||||
COLORS
|
||||
} from './utils';
|
||||
|
|
|
|||
93
src/runner/server.ts
Normal file
93
src/runner/server.ts
Normal file
|
|
@ -0,0 +1,93 @@
|
|||
import { promises as fs } from 'fs';
|
||||
import path from 'path';
|
||||
import { ConformanceCheck } from '../types';
|
||||
import { getClientScenario } from '../scenarios';
|
||||
import { ensureResultsDir, createResultDir } from './utils';
|
||||
|
||||
export async function runServerConformanceTest(
|
||||
serverUrl: string,
|
||||
scenarioName: string
|
||||
): Promise<{
|
||||
checks: ConformanceCheck[];
|
||||
resultDir: string;
|
||||
}> {
|
||||
await ensureResultsDir();
|
||||
const resultDir = createResultDir(scenarioName, 'server');
|
||||
await fs.mkdir(resultDir, { recursive: true });
|
||||
|
||||
// Scenario is guaranteed to exist by CLI validation
|
||||
const scenario = getClientScenario(scenarioName)!;
|
||||
|
||||
console.log(
|
||||
`Running client scenario '${scenarioName}' against server: ${serverUrl}`
|
||||
);
|
||||
|
||||
const checks = await scenario.run(serverUrl);
|
||||
|
||||
await fs.writeFile(
|
||||
path.join(resultDir, 'checks.json'),
|
||||
JSON.stringify(checks, null, 2)
|
||||
);
|
||||
|
||||
console.log(`Results saved to ${resultDir}`);
|
||||
|
||||
return {
|
||||
checks,
|
||||
resultDir
|
||||
};
|
||||
}
|
||||
|
||||
export function printServerResults(checks: ConformanceCheck[]): {
|
||||
passed: number;
|
||||
failed: number;
|
||||
denominator: number;
|
||||
} {
|
||||
const denominator = checks.filter(
|
||||
(c) => c.status === 'SUCCESS' || c.status === 'FAILURE'
|
||||
).length;
|
||||
const passed = checks.filter((c) => c.status === 'SUCCESS').length;
|
||||
const failed = checks.filter((c) => c.status === 'FAILURE').length;
|
||||
|
||||
console.log(`Checks:\n${JSON.stringify(checks, null, 2)}`);
|
||||
|
||||
console.log(`\nTest Results:`);
|
||||
console.log(`Passed: ${passed}/${denominator}, ${failed} failed`);
|
||||
|
||||
if (failed > 0) {
|
||||
console.log('\nFailed Checks:');
|
||||
checks
|
||||
.filter((c) => c.status === 'FAILURE')
|
||||
.forEach((c) => {
|
||||
console.log(` - ${c.name}: ${c.description}`);
|
||||
if (c.errorMessage) {
|
||||
console.log(` Error: ${c.errorMessage}`);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
return { passed, failed, denominator };
|
||||
}
|
||||
|
||||
export function printServerSummary(
|
||||
allResults: { scenario: string; checks: ConformanceCheck[] }[]
|
||||
): { totalPassed: number; totalFailed: number } {
|
||||
console.log('\n\n=== SUMMARY ===');
|
||||
let totalPassed = 0;
|
||||
let totalFailed = 0;
|
||||
|
||||
for (const result of allResults) {
|
||||
const passed = result.checks.filter((c) => c.status === 'SUCCESS').length;
|
||||
const failed = result.checks.filter((c) => c.status === 'FAILURE').length;
|
||||
totalPassed += passed;
|
||||
totalFailed += failed;
|
||||
|
||||
const status = failed === 0 ? '✓' : '✗';
|
||||
console.log(
|
||||
`${status} ${result.scenario}: ${passed} passed, ${failed} failed`
|
||||
);
|
||||
}
|
||||
|
||||
console.log(`\nTotal: ${totalPassed} passed, ${totalFailed} failed`);
|
||||
|
||||
return { totalPassed, totalFailed };
|
||||
}
|
||||
62
src/runner/utils.ts
Normal file
62
src/runner/utils.ts
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
import { promises as fs } from 'fs';
|
||||
import path from 'path';
|
||||
import { ConformanceCheck } from '../types';
|
||||
|
||||
// ANSI color codes
|
||||
export const COLORS = {
|
||||
RESET: '\x1b[0m',
|
||||
GRAY: '\x1b[90m',
|
||||
GREEN: '\x1b[32m',
|
||||
YELLOW: '\x1b[33m',
|
||||
RED: '\x1b[31m',
|
||||
BLUE: '\x1b[36m'
|
||||
};
|
||||
|
||||
export function getStatusColor(status: string): string {
|
||||
switch (status) {
|
||||
case 'SUCCESS':
|
||||
return COLORS.GREEN;
|
||||
case 'FAILURE':
|
||||
return COLORS.RED;
|
||||
case 'INFO':
|
||||
return COLORS.BLUE;
|
||||
default:
|
||||
return COLORS.RESET;
|
||||
}
|
||||
}
|
||||
|
||||
export function formatPrettyChecks(checks: ConformanceCheck[]): string {
|
||||
// Find the longest id and status for column alignment
|
||||
const maxIdLength = Math.max(...checks.map((c) => c.id.length));
|
||||
const maxStatusLength = Math.max(...checks.map((c) => c.status.length));
|
||||
|
||||
return checks
|
||||
.map((check) => {
|
||||
const timestamp = `${COLORS.GRAY}${check.timestamp}${COLORS.RESET}`;
|
||||
const id = check.id.padEnd(maxIdLength);
|
||||
const statusColor = getStatusColor(check.status);
|
||||
const status = `${statusColor}${check.status.padEnd(maxStatusLength)}${COLORS.RESET}`;
|
||||
const description = check.description;
|
||||
const line = `${timestamp} [${id}] ${status} ${description}`;
|
||||
// Add newline after outgoing responses for better visual separation
|
||||
return (
|
||||
line +
|
||||
(check.id.includes('outgoing') && check.id.includes('response')
|
||||
? '\n'
|
||||
: '')
|
||||
);
|
||||
})
|
||||
.join('\n');
|
||||
}
|
||||
|
||||
export async function ensureResultsDir(): Promise<string> {
|
||||
const resultsDir = path.join(process.cwd(), 'results');
|
||||
await fs.mkdir(resultsDir, { recursive: true });
|
||||
return resultsDir;
|
||||
}
|
||||
|
||||
export function createResultDir(scenario: string, prefix = ''): string {
|
||||
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
|
||||
const dirName = prefix ? `${prefix}-${scenario}` : scenario;
|
||||
return path.join('results', `${dirName}-${timestamp}`);
|
||||
}
|
||||
44
src/schemas.ts
Normal file
44
src/schemas.ts
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
import { z } from 'zod';
|
||||
import { getScenario, getClientScenario } from './scenarios';
|
||||
|
||||
// Client command options schema
|
||||
export const ClientOptionsSchema = z.object({
|
||||
command: z.string().min(1, 'Command cannot be empty'),
|
||||
scenario: z
|
||||
.string()
|
||||
.min(1, 'Scenario cannot be empty')
|
||||
.refine(
|
||||
(scenario) => getScenario(scenario) !== undefined,
|
||||
(scenario) => ({
|
||||
message: `Unknown scenario '${scenario}'`
|
||||
})
|
||||
),
|
||||
timeout: z
|
||||
.string()
|
||||
.transform((val) => parseInt(val, 10))
|
||||
.pipe(
|
||||
z
|
||||
.number()
|
||||
.positive('Timeout must be a positive number')
|
||||
.int('Timeout must be an integer')
|
||||
),
|
||||
verbose: z.boolean().optional()
|
||||
});
|
||||
|
||||
export type ClientOptions = z.infer<typeof ClientOptionsSchema>;
|
||||
|
||||
// Server command options schema
|
||||
export const ServerOptionsSchema = z.object({
|
||||
url: z.string().url('Invalid server URL'),
|
||||
scenario: z
|
||||
.string()
|
||||
.refine(
|
||||
(scenario) => getClientScenario(scenario) !== undefined,
|
||||
(scenario) => ({
|
||||
message: `Unknown scenario '${scenario}'`
|
||||
})
|
||||
)
|
||||
.optional()
|
||||
});
|
||||
|
||||
export type ServerOptions = z.infer<typeof ServerOptionsSchema>;
|
||||
|
|
@ -1,186 +0,0 @@
|
|||
import { promises as fs } from 'fs';
|
||||
import path from 'path';
|
||||
import { ConformanceCheck } from './types.js';
|
||||
import { getClientScenario } from './scenarios/index.js';
|
||||
|
||||
async function ensureResultsDir(): Promise<string> {
|
||||
const resultsDir = path.join(process.cwd(), 'results');
|
||||
await fs.mkdir(resultsDir, { recursive: true });
|
||||
return resultsDir;
|
||||
}
|
||||
|
||||
function createResultDir(scenario: string): string {
|
||||
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
|
||||
return path.join('results', `server-${scenario}-${timestamp}`);
|
||||
}
|
||||
|
||||
export async function runServerConformanceTest(
|
||||
serverUrl: string,
|
||||
scenarioName: string
|
||||
): Promise<{
|
||||
checks: ConformanceCheck[];
|
||||
resultDir: string;
|
||||
}> {
|
||||
await ensureResultsDir();
|
||||
const resultDir = createResultDir(scenarioName);
|
||||
await fs.mkdir(resultDir, { recursive: true });
|
||||
|
||||
const scenario = getClientScenario(scenarioName);
|
||||
if (!scenario) {
|
||||
throw new Error(`Unknown client scenario: ${scenarioName}`);
|
||||
}
|
||||
|
||||
console.log(
|
||||
`Running client scenario '${scenarioName}' against server: ${serverUrl}`
|
||||
);
|
||||
|
||||
const checks = await scenario.run(serverUrl);
|
||||
|
||||
await fs.writeFile(
|
||||
path.join(resultDir, 'checks.json'),
|
||||
JSON.stringify(checks, null, 2)
|
||||
);
|
||||
|
||||
console.log(`Results saved to ${resultDir}`);
|
||||
|
||||
return {
|
||||
checks,
|
||||
resultDir
|
||||
};
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const args = process.argv.slice(2);
|
||||
let serverUrl: string | null = null;
|
||||
let scenario: string | null = null;
|
||||
let runAll = false;
|
||||
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
if (args[i] === '--server-url' && i + 1 < args.length) {
|
||||
serverUrl = args[i + 1];
|
||||
i++;
|
||||
} else if (args[i] === '--scenario' && i + 1 < args.length) {
|
||||
scenario = args[i + 1];
|
||||
i++;
|
||||
} else if (args[i] === '--all') {
|
||||
runAll = true;
|
||||
}
|
||||
}
|
||||
|
||||
if (!serverUrl) {
|
||||
console.error(
|
||||
'Usage: server-runner --server-url <url> [--scenario <scenario> | --all]'
|
||||
);
|
||||
console.error(
|
||||
'Example: server-runner --server-url http://localhost:3000/mcp --scenario server-initialize'
|
||||
);
|
||||
console.error(
|
||||
'Example: server-runner --server-url http://localhost:3000/mcp --all'
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!scenario && !runAll) {
|
||||
console.error('Must specify either --scenario or --all');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
try {
|
||||
const allResults: { scenario: string; checks: ConformanceCheck[] }[] = [];
|
||||
|
||||
if (runAll) {
|
||||
// Get all server scenarios
|
||||
const { listClientScenarios } = await import('./scenarios/index.js');
|
||||
const scenarios = listClientScenarios();
|
||||
|
||||
console.log(
|
||||
`Running ${scenarios.length} scenarios against ${serverUrl}\n`
|
||||
);
|
||||
|
||||
for (const scenarioName of scenarios) {
|
||||
console.log(`\n=== Running scenario: ${scenarioName} ===`);
|
||||
try {
|
||||
const result = await runServerConformanceTest(
|
||||
serverUrl,
|
||||
scenarioName
|
||||
);
|
||||
allResults.push({ scenario: scenarioName, checks: result.checks });
|
||||
} catch (error) {
|
||||
console.error(`Failed to run scenario ${scenarioName}:`, error);
|
||||
allResults.push({
|
||||
scenario: scenarioName,
|
||||
checks: [
|
||||
{
|
||||
id: scenarioName,
|
||||
name: scenarioName,
|
||||
description: `Failed to run scenario`,
|
||||
status: 'FAILURE',
|
||||
timestamp: new Date().toISOString(),
|
||||
errorMessage:
|
||||
error instanceof Error ? error.message : String(error)
|
||||
}
|
||||
]
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Summary
|
||||
console.log('\n\n=== SUMMARY ===');
|
||||
let totalPassed = 0;
|
||||
let totalFailed = 0;
|
||||
|
||||
for (const result of allResults) {
|
||||
const passed = result.checks.filter(
|
||||
(c) => c.status === 'SUCCESS'
|
||||
).length;
|
||||
const failed = result.checks.filter(
|
||||
(c) => c.status === 'FAILURE'
|
||||
).length;
|
||||
totalPassed += passed;
|
||||
totalFailed += failed;
|
||||
|
||||
const status = failed === 0 ? '✓' : '✗';
|
||||
console.log(
|
||||
`${status} ${result.scenario}: ${passed} passed, ${failed} failed`
|
||||
);
|
||||
}
|
||||
|
||||
console.log(`\nTotal: ${totalPassed} passed, ${totalFailed} failed`);
|
||||
process.exit(totalFailed > 0 ? 1 : 0);
|
||||
} else if (scenario) {
|
||||
const result = await runServerConformanceTest(serverUrl, scenario);
|
||||
|
||||
const denominator = result.checks.filter(
|
||||
(c) => c.status === 'SUCCESS' || c.status == 'FAILURE'
|
||||
).length;
|
||||
const passed = result.checks.filter((c) => c.status === 'SUCCESS').length;
|
||||
const failed = result.checks.filter((c) => c.status === 'FAILURE').length;
|
||||
|
||||
console.log(`Checks:\n${JSON.stringify(result.checks, null, 2)}`);
|
||||
|
||||
console.log(`\nTest Results:`);
|
||||
console.log(`Passed: ${passed}/${denominator}, ${failed} failed`);
|
||||
|
||||
if (failed > 0) {
|
||||
console.log('\nFailed Checks:');
|
||||
result.checks
|
||||
.filter((c) => c.status === 'FAILURE')
|
||||
.forEach((c) => {
|
||||
console.log(` - ${c.name}: ${c.description}`);
|
||||
if (c.errorMessage) {
|
||||
console.log(` Error: ${c.errorMessage}`);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
process.exit(failed > 0 ? 1 : 0);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('Server test runner error:', error);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
if (import.meta.url === `file://${process.argv[1]}`) {
|
||||
main();
|
||||
}
|
||||
|
|
@ -3,6 +3,7 @@
|
|||
"target": "ES2020",
|
||||
"module": "ES2020",
|
||||
"moduleResolution": "node",
|
||||
"resolveJsonModule": true,
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
|
|
|
|||
Loading…
Reference in a new issue