tsdown + cli with commander (#17)
* feat: tsdown * fix: formatting * automate version updates in cli * fix: docs * scenarios * fix: commander deps
This commit is contained in:
Matt
GitHub
parent
18f48e5dea
commit
7f1515c475
1 changed files with 1600 additions and 539 deletions
|
|
@ -2,20 +2,45 @@
|
|||
|
||||
## Overview
|
||||
|
||||
The conformance test runner is a framework for testing MCP client implementations against the MCP specification. It executes client implementations in controlled scenarios and validates their behavior through a series of conformance checks.
|
||||
The conformance test runner is a framework for testing both MCP client and server implementations against the MCP specification. It provides two testing modes:
|
||||
|
||||
1. **Client Testing** - Executes client implementations in controlled scenarios and validates their behavior
|
||||
2. **Server Testing** - Tests server implementations by acting as an MCP client and validating responses
|
||||
|
||||
## Architecture
|
||||
|
||||
### File Structure
|
||||
|
||||
```
|
||||
src/runner/
|
||||
├── index.ts # Exports all public functions
|
||||
├── client.ts # Client testing implementation
|
||||
├── server.ts # Server testing implementation
|
||||
└── utils.ts # Shared utilities (formatting, file I/O, ANSI colors)
|
||||
```
|
||||
|
||||
### Components
|
||||
|
||||
**Client Testing:**
|
||||
|
||||
1. **Runner** - Orchestrates test execution
|
||||
2. **Test Server** - MCP server (and optionally auth server for auth scenarios)
|
||||
3. **Client Process** - The MCP client implementation under test
|
||||
4. **Scenario** - A specific test case with expected behaviors
|
||||
5. **Checks** - Validation functions that produce ConformanceCheck results
|
||||
|
||||
**Server Testing:**
|
||||
|
||||
1. **Runner** - Orchestrates test execution
|
||||
2. **Test Client** - MCP SDK client that connects to the server under test
|
||||
3. **Server Process** - The MCP server implementation under test (external)
|
||||
4. **Scenario** - A specific test case with expected server behaviors
|
||||
5. **Checks** - Validation functions that produce ConformanceCheck results
|
||||
|
||||
### Execution Flow
|
||||
|
||||
**Client Testing:**
|
||||
|
||||
```
|
||||
1. Runner starts test server(s) on available port(s)
|
||||
2. Runner spawns client process with server URL as final argument
|
||||
|
|
@ -25,44 +50,124 @@ The conformance test runner is a framework for testing MCP client implementation
|
|||
6. Results are written to results/<scenario>-<timestamp>/
|
||||
```
|
||||
|
||||
## CLI Interface
|
||||
**Server Testing:**
|
||||
|
||||
```bash
|
||||
npm run start -- --command "tsx examples/clients/typescript/test1.ts --verbose" --scenario initialize
|
||||
```
|
||||
1. User starts their server implementation
|
||||
2. Runner connects to server URL as an MCP client
|
||||
3. Runner sends requests and captures responses
|
||||
4. Scenario-specific checks validate server behavior
|
||||
5. Results are written to results/server-<scenario>-<timestamp>/
|
||||
```
|
||||
|
||||
### Arguments
|
||||
## CLI Interface
|
||||
|
||||
The conformance suite provides a unified CLI with two main commands:
|
||||
|
||||
### Client Testing
|
||||
|
||||
```bash
|
||||
# Run a single client scenario
|
||||
npm run start -- client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize
|
||||
|
||||
# With verbose output
|
||||
npm run start -- client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize --verbose
|
||||
|
||||
# With custom timeout
|
||||
npm run start -- client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize --timeout 60000
|
||||
```
|
||||
|
||||
**Arguments:**
|
||||
|
||||
- `--command` - The command to run the client (can include existing flags)
|
||||
- `--scenario` - The scenario to test (e.g., "initialize", "list-tools", "call-tool")
|
||||
- `--scenario` - The scenario to test (e.g., "initialize", "tools-call")
|
||||
- `--timeout` - Timeout in milliseconds (default: 30000)
|
||||
- `--verbose` - Show verbose output (JSON format)
|
||||
|
||||
The runner will append the server URL as the final argument to the command.
|
||||
|
||||
### Server Testing
|
||||
|
||||
```bash
|
||||
# Run a single server scenario
|
||||
npm run start -- server --url http://localhost:3000/mcp --scenario server-initialize
|
||||
|
||||
# Run all server scenarios (default when no --scenario specified)
|
||||
npm run start -- server --url http://localhost:3000/mcp
|
||||
```
|
||||
|
||||
**Arguments:**
|
||||
|
||||
- `--url` - URL of the server to test
|
||||
- `--scenario <scenario>` - Scenario to test (optional, defaults to all scenarios if not specified)
|
||||
|
||||
### List Available Scenarios
|
||||
|
||||
```bash
|
||||
# List all scenarios
|
||||
npm run start -- list
|
||||
|
||||
# List only client scenarios
|
||||
npm run start -- list --client
|
||||
|
||||
# List only server scenarios
|
||||
npm run start -- list --server
|
||||
```
|
||||
|
||||
## Validation
|
||||
|
||||
All CLI arguments are validated using Zod schemas (`src/schemas.ts`) before being passed to runner functions:
|
||||
|
||||
**Client Validation:**
|
||||
|
||||
- Command is non-empty
|
||||
- Scenario exists in available scenarios
|
||||
- Timeout is a positive integer
|
||||
|
||||
**Server Validation:**
|
||||
|
||||
- URL is a valid HTTP/HTTPS URL
|
||||
- All specified scenarios exist (if provided)
|
||||
- Defaults to all scenarios when no `--scenario` is specified
|
||||
|
||||
## Scenarios
|
||||
|
||||
A scenario represents a specific test case that validates one or more aspects of MCP client behavior. Each scenario:
|
||||
A scenario represents a specific test case that validates one or more aspects of MCP behavior. Each scenario:
|
||||
|
||||
- Configures the test server with expected behavior
|
||||
- Configures the test environment with expected behavior
|
||||
- May run multiple conformance checks
|
||||
- Validates both client behavior and server responses
|
||||
- Validates both request/response patterns and protocol compliance
|
||||
|
||||
Example scenarios:
|
||||
### Client Scenario Examples
|
||||
|
||||
- `initialize` - Tests client initialization handshake
|
||||
- `list-tools` - Tests tool discovery
|
||||
- `call-tool` - Tests tool invocation
|
||||
- `auth-flow` - Tests OAuth/authorization flows (requires auth server)
|
||||
- `tools-call` - Tests tool invocation
|
||||
- `auth/basic-dcr` - Tests OAuth Dynamic Client Registration flow
|
||||
- `auth/basic-metadata-var1` - Tests OAuth with authorization metadata variation 1
|
||||
|
||||
### Server Scenario Examples
|
||||
|
||||
- `server-initialize` - Tests server initialization and capabilities
|
||||
- `tools-list` - Tests tool listing endpoint
|
||||
- `tools-call-simple-text` - Tests tool invocation with text response
|
||||
- `resources-list` - Tests resource listing
|
||||
- `prompts-list` - Tests prompt listing
|
||||
- `logging-set-level` - Tests logging level configuration
|
||||
|
||||
## Output Structure
|
||||
|
||||
Results are written to: `results/<scenario>-<timestamp>/`
|
||||
Results are written to: `results/<scenario>-<timestamp>/` or `results/server-<scenario>-<timestamp>/`
|
||||
|
||||
Files:
|
||||
**Client Testing Files:**
|
||||
|
||||
- `checks.json` - Array of ConformanceCheck objects with validation results
|
||||
- `stdout.txt` - Complete stdout from the client process
|
||||
- `stderr.txt` - Complete stderr from the client process
|
||||
|
||||
**Server Testing Files:**
|
||||
|
||||
- `checks.json` - Array of ConformanceCheck objects with validation results
|
||||
|
||||
### checks.json Format
|
||||
|
||||
```json
|
||||
|
|
@ -84,6 +189,30 @@ Files:
|
|||
]
|
||||
```
|
||||
|
||||
## Programmatic Usage
|
||||
|
||||
The runner can also be used programmatically:
|
||||
|
||||
```typescript
|
||||
import { runConformanceTest, printClientResults, runServerConformanceTest, printServerResults } from './runner/index.js';
|
||||
|
||||
// Test a client
|
||||
const clientResult = await runConformanceTest('tsx my-client.ts', 'initialize', 30000);
|
||||
const { failed } = printClientResults(clientResult.checks);
|
||||
|
||||
// Test a server
|
||||
const serverResult = await runServerConformanceTest('http://localhost:3000/mcp', 'server-initialize');
|
||||
const { failed } = printServerResults(serverResult.checks);
|
||||
```
|
||||
|
||||
## Shared Utilities
|
||||
|
||||
The `utils.ts` module provides shared functionality:
|
||||
|
||||
- **File Operations:** `ensureResultsDir()`, `createResultDir()`
|
||||
- **Formatting:** `formatPrettyChecks()` for colored console output
|
||||
- **Styling:** ANSI color constants and helpers
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- **Test Suites** - Group multiple scenarios for convenience
|
||||
|
|
@ -91,3 +220,5 @@ Files:
|
|||
- **Parallel Execution** - Run multiple scenarios concurrently
|
||||
- **Custom Servers** - Allow custom MCP server implementations for advanced scenarios
|
||||
- **Report Formats** - HTML, Markdown, or other human-readable formats
|
||||
- **Watch Mode** - Automatically re-run tests on file changes
|
||||
- **Coverage Tracking** - Track which parts of the spec are tested
|
||||
|
|
|
|||
Loading…
Reference in a new issue