tsdown + cli with commander (#17)
* feat: tsdown * fix: formatting * automate version updates in cli * fix: docs * scenarios * fix: commander deps
This commit is contained in:
parent
18f48e5dea
commit
7f1515c475
16 changed files with 1600 additions and 539 deletions
1
.gitignore
vendored
1
.gitignore
vendored
|
|
@ -1,3 +1,4 @@
|
||||||
node_modules
|
node_modules
|
||||||
results/
|
results/
|
||||||
lefthook-local.yml
|
lefthook-local.yml
|
||||||
|
dist
|
||||||
77
README.md
77
README.md
|
|
@ -1,6 +1,6 @@
|
||||||
# MCP Conformance Test Framework
|
# MCP Conformance Test Framework
|
||||||
|
|
||||||
A framework for testing MCP (Model Context Protocol) client implementations against the specification.
|
A framework for testing MCP (Model Context Protocol) client and server implementations against the specification.
|
||||||
|
|
||||||
> [!WARNING] This repository is a work in progress and is unstable. Join the conversation in the #conformance-testing-wg in the MCP Contributors discord.
|
> [!WARNING] This repository is a work in progress and is unstable. Join the conversation in the #conformance-testing-wg in the MCP Contributors discord.
|
||||||
|
|
||||||
|
|
@ -9,19 +9,30 @@ A framework for testing MCP (Model Context Protocol) client implementations agai
|
||||||
### Testing Clients
|
### Testing Clients
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
npm install
|
npx @modelcontextprotocol/conformance client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize
|
||||||
npm run start -- --command "tsx examples/clients/typescript/test1.ts" --scenario initialize
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Testing Servers
|
### Testing Servers
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
npm run test:server -- --server-url http://localhost:3000/mcp --all
|
# Run all server scenarios (default)
|
||||||
|
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp
|
||||||
|
|
||||||
|
# Run a single scenario
|
||||||
|
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --scenario server-initialize
|
||||||
|
```
|
||||||
|
|
||||||
|
### List Available Scenarios
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npx @modelcontextprotocol/conformance list
|
||||||
```
|
```
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
The conformance test framework validates MCP client implementations by:
|
The conformance test framework validates MCP implementations by:
|
||||||
|
|
||||||
|
**For Clients:**
|
||||||
|
|
||||||
1. Starting a test server for the specified scenario
|
1. Starting a test server for the specified scenario
|
||||||
2. Running the client implementation with the test server URL
|
2. Running the client implementation with the test server URL
|
||||||
|
|
@ -29,25 +40,53 @@ The conformance test framework validates MCP client implementations by:
|
||||||
4. Running conformance checks against the specification
|
4. Running conformance checks against the specification
|
||||||
5. Generating detailed test results
|
5. Generating detailed test results
|
||||||
|
|
||||||
|
**For Servers:**
|
||||||
|
|
||||||
|
1. Connecting to the running server as an MCP client
|
||||||
|
2. Sending test requests and capturing responses
|
||||||
|
3. Running conformance checks against server behavior
|
||||||
|
4. Generating detailed test results
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
|
### Client Testing
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
npm run start -- --command "<client-command>" --scenario <scenario-name>
|
npx @modelcontextprotocol/conformance client --command "<client-command>" --scenario <scenario-name> [options]
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Options:**
|
||||||
|
|
||||||
- `--command` - The command to run your MCP client (can include flags)
|
- `--command` - The command to run your MCP client (can include flags)
|
||||||
- `--scenario` - The test scenario to run (e.g., "initialize")
|
- `--scenario` - The test scenario to run (e.g., "initialize")
|
||||||
|
- `--timeout` - Timeout in milliseconds (default: 30000)
|
||||||
|
- `--verbose` - Show verbose output
|
||||||
|
|
||||||
The framework appends the server URL as the final argument to your command.
|
The framework appends the server URL as the final argument to your command.
|
||||||
|
|
||||||
|
### Server Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npx @modelcontextprotocol/conformance server --url <url> [--scenario <scenario>]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Options:**
|
||||||
|
|
||||||
|
- `--url` - URL of the server to test
|
||||||
|
- `--scenario <scenario>` - Test scenario to run (e.g., "server-initialize". Runs all available scenarios by default
|
||||||
|
|
||||||
## Test Results
|
## Test Results
|
||||||
|
|
||||||
Results are saved to `results/<scenario>-<timestamp>/`:
|
**Client Testing** - Results are saved to `results/<scenario>-<timestamp>/`:
|
||||||
|
|
||||||
- `checks.json` - Array of conformance check results with pass/fail status
|
- `checks.json` - Array of conformance check results with pass/fail status
|
||||||
- `stdout.txt` - Client stdout output
|
- `stdout.txt` - Client stdout output
|
||||||
- `stderr.txt` - Client stderr output
|
- `stderr.txt` - Client stderr output
|
||||||
|
|
||||||
|
**Server Testing** - Results are saved to `results/server-<scenario>-<timestamp>/`:
|
||||||
|
|
||||||
|
- `checks.json` - Array of conformance check results with pass/fail status
|
||||||
|
|
||||||
## Example Clients
|
## Example Clients
|
||||||
|
|
||||||
- `examples/clients/typescript/test1.ts` - Valid MCP client (passes all checks)
|
- `examples/clients/typescript/test1.ts` - Valid MCP client (passes all checks)
|
||||||
|
|
@ -55,10 +94,25 @@ Results are saved to `results/<scenario>-<timestamp>/`:
|
||||||
|
|
||||||
## Available Scenarios
|
## Available Scenarios
|
||||||
|
|
||||||
|
### Client Scenarios
|
||||||
|
|
||||||
- **initialize** - Tests MCP client initialization handshake
|
- **initialize** - Tests MCP client initialization handshake
|
||||||
- Validates protocol version
|
- Validates protocol version
|
||||||
- Validates clientInfo (name and version)
|
- Validates clientInfo (name and version)
|
||||||
- Validates server response handling
|
- Validates server response handling
|
||||||
|
- **tools-call** - Tests tool invocation
|
||||||
|
- **auth/basic-dcr** - Tests OAuth Dynamic Client Registration flow
|
||||||
|
- **auth/basic-metadata-var1** - Tests OAuth with authorization metadata
|
||||||
|
|
||||||
|
### Server Scenarios
|
||||||
|
|
||||||
|
Run `npx @modelcontextprotocol/conformance list --server` to see all available server scenarios, including:
|
||||||
|
|
||||||
|
- **server-initialize** - Tests server initialization and capabilities
|
||||||
|
- **tools-list** - Tests tool listing endpoint
|
||||||
|
- **tools-call-\*** - Various tool invocation scenarios
|
||||||
|
- **resources-\*** - Resource management scenarios
|
||||||
|
- **prompts-\*** - Prompt management scenarios
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
|
|
@ -67,8 +121,13 @@ See `src/runner/DESIGN.md` for detailed architecture documentation.
|
||||||
### Key Components
|
### Key Components
|
||||||
|
|
||||||
- **Runner** (`src/runner/`) - Orchestrates test execution and result generation
|
- **Runner** (`src/runner/`) - Orchestrates test execution and result generation
|
||||||
- **Scenarios** (`src/scenarios/`) - Test scenarios with custom server implementations
|
- `client.ts` - Client testing implementation
|
||||||
- **Checks** (`src/checks.ts`) - Conformance validation functions
|
- `server.ts` - Server testing implementation
|
||||||
|
- `utils.ts` - Shared utilities
|
||||||
|
- `index.ts` - Public API exports
|
||||||
|
- **CLI** (`src/index.ts`) - Command-line interface using Commander.js
|
||||||
|
- **Scenarios** (`src/scenarios/`) - Test scenarios with expected behaviors
|
||||||
|
- **Checks** (`src/checks/`) - Conformance validation functions
|
||||||
- **Types** (`src/types.ts`) - Shared type definitions
|
- **Types** (`src/types.ts`) - Shared type definitions
|
||||||
|
|
||||||
## Adding New Scenarios
|
## Adding New Scenarios
|
||||||
|
|
|
||||||
|
|
@ -1022,8 +1022,8 @@ MCP Conformance Test Server running on http://localhost:3000
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Single test
|
# Single test
|
||||||
npm run test:server -- --server-url http://localhost:3000/mcp --scenario server-initialize
|
npm run start -- server --url http://localhost:3000/mcp --scenario server-initialize
|
||||||
|
|
||||||
# All tests
|
# All tests
|
||||||
npm run test:server -- --server-url http://localhost:3000/mcp --all
|
npm run start -- server --url http://localhost:3000/mcp
|
||||||
```
|
```
|
||||||
|
|
|
||||||
|
|
@ -147,8 +147,7 @@ This server implements the MCP Server Conformance Requirements specified in `../
|
||||||
To run conformance tests against this server:
|
To run conformance tests against this server:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd ../../../
|
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp
|
||||||
npm run test:server -- --server-url http://localhost:3000/mcp --all
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Implementation Notes
|
## Implementation Notes
|
||||||
|
|
|
||||||
788
package-lock.json
generated
788
package-lock.json
generated
File diff suppressed because it is too large
Load diff
12
package.json
12
package.json
|
|
@ -3,13 +3,19 @@
|
||||||
"version": "0.1.0",
|
"version": "0.1.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"start": "tsx src/runner.ts",
|
"start": "tsx src/index.ts",
|
||||||
"test:server": "tsx src/server-runner.ts",
|
|
||||||
"test": "jest",
|
"test": "jest",
|
||||||
|
"build": "tsdown src/index.ts --minify --clean --target node20",
|
||||||
"lint": "eslint src/ && prettier --check .",
|
"lint": "eslint src/ && prettier --check .",
|
||||||
"lint:fix": "eslint src/ --fix && prettier --write .",
|
"lint:fix": "eslint src/ --fix && prettier --write .",
|
||||||
"lint:fix_check": "npm run lint:fix && git diff --exit-code --quiet"
|
"lint:fix_check": "npm run lint:fix && git diff --exit-code --quiet"
|
||||||
},
|
},
|
||||||
|
"files": [
|
||||||
|
"dist"
|
||||||
|
],
|
||||||
|
"bin": {
|
||||||
|
"conformance": "dist/index.mjs"
|
||||||
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"@eslint/js": "^9.8.0",
|
"@eslint/js": "^9.8.0",
|
||||||
"@jest/globals": "^29.7.0",
|
"@jest/globals": "^29.7.0",
|
||||||
|
|
@ -20,12 +26,14 @@
|
||||||
"jest": "^29.7.0",
|
"jest": "^29.7.0",
|
||||||
"prettier": "3.6.2",
|
"prettier": "3.6.2",
|
||||||
"ts-jest": "^29.1.1",
|
"ts-jest": "^29.1.1",
|
||||||
|
"tsdown": "^0.16.2",
|
||||||
"tsx": "^4.7.0",
|
"tsx": "^4.7.0",
|
||||||
"typescript": "^5.5.4",
|
"typescript": "^5.5.4",
|
||||||
"typescript-eslint": "^8.0.0"
|
"typescript-eslint": "^8.0.0"
|
||||||
},
|
},
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@modelcontextprotocol/sdk": "^1.20.1",
|
"@modelcontextprotocol/sdk": "^1.20.1",
|
||||||
|
"commander": "^14.0.2",
|
||||||
"express": "^5.1.0",
|
"express": "^5.1.0",
|
||||||
"lefthook": "^2.0.2",
|
"lefthook": "^2.0.2",
|
||||||
"zod": "^3.25.76"
|
"zod": "^3.25.76"
|
||||||
|
|
|
||||||
164
src/index.ts
Normal file
164
src/index.ts
Normal file
|
|
@ -0,0 +1,164 @@
|
||||||
|
#!/usr/bin/env node
|
||||||
|
|
||||||
|
import { Command } from 'commander';
|
||||||
|
import { ZodError } from 'zod';
|
||||||
|
import {
|
||||||
|
runConformanceTest,
|
||||||
|
printClientResults,
|
||||||
|
runServerConformanceTest,
|
||||||
|
printServerResults,
|
||||||
|
printServerSummary
|
||||||
|
} from './runner';
|
||||||
|
import { listScenarios, listClientScenarios } from './scenarios';
|
||||||
|
import { ConformanceCheck } from './types';
|
||||||
|
import { ClientOptionsSchema, ServerOptionsSchema } from './schemas';
|
||||||
|
import packageJson from '../package.json';
|
||||||
|
|
||||||
|
const program = new Command();
|
||||||
|
|
||||||
|
program
|
||||||
|
.name('conformance')
|
||||||
|
.description('MCP Conformance Test Suite')
|
||||||
|
.version(packageJson.version);
|
||||||
|
|
||||||
|
// Client command - tests a client implementation against scenarios
|
||||||
|
program
|
||||||
|
.command('client')
|
||||||
|
.description('Run conformance tests against a client implementation')
|
||||||
|
.requiredOption('--command <command>', 'Command to run the client')
|
||||||
|
.requiredOption('--scenario <scenario>', 'Scenario to test')
|
||||||
|
.option('--timeout <ms>', 'Timeout in milliseconds', '30000')
|
||||||
|
.option('--verbose', 'Show verbose output')
|
||||||
|
.action(async (options) => {
|
||||||
|
try {
|
||||||
|
// Validate options with Zod
|
||||||
|
const validated = ClientOptionsSchema.parse(options);
|
||||||
|
|
||||||
|
const result = await runConformanceTest(
|
||||||
|
validated.command,
|
||||||
|
validated.scenario,
|
||||||
|
validated.timeout
|
||||||
|
);
|
||||||
|
|
||||||
|
const { failed } = printClientResults(
|
||||||
|
result.checks,
|
||||||
|
validated.verbose ?? false
|
||||||
|
);
|
||||||
|
process.exit(failed > 0 ? 1 : 0);
|
||||||
|
} catch (error) {
|
||||||
|
if (error instanceof ZodError) {
|
||||||
|
console.error('Validation error:');
|
||||||
|
error.errors.forEach((err) => {
|
||||||
|
console.error(` ${err.path.join('.')}: ${err.message}`);
|
||||||
|
});
|
||||||
|
console.error('\nAvailable client scenarios:');
|
||||||
|
listScenarios().forEach((s) => console.error(` - ${s}`));
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
console.error('Client test error:', error);
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// Server command - tests a server implementation
|
||||||
|
program
|
||||||
|
.command('server')
|
||||||
|
.description('Run conformance tests against a server implementation')
|
||||||
|
.requiredOption('--url <url>', 'URL of the server to test')
|
||||||
|
.option(
|
||||||
|
'--scenario <scenario>',
|
||||||
|
'Scenario to test (defaults to all scenarios if not specified)'
|
||||||
|
)
|
||||||
|
.action(async (options) => {
|
||||||
|
try {
|
||||||
|
// Validate options with Zod
|
||||||
|
const validated = ServerOptionsSchema.parse(options);
|
||||||
|
|
||||||
|
// If a single scenario is specified, run just that one
|
||||||
|
if (validated.scenario) {
|
||||||
|
const result = await runServerConformanceTest(
|
||||||
|
validated.url,
|
||||||
|
validated.scenario
|
||||||
|
);
|
||||||
|
|
||||||
|
const { failed } = printServerResults(result.checks);
|
||||||
|
process.exit(failed > 0 ? 1 : 0);
|
||||||
|
} else {
|
||||||
|
// Run all scenarios
|
||||||
|
const scenarios = listClientScenarios();
|
||||||
|
console.log(
|
||||||
|
`Running ${scenarios.length} scenarios against ${validated.url}\n`
|
||||||
|
);
|
||||||
|
|
||||||
|
const allResults: { scenario: string; checks: ConformanceCheck[] }[] =
|
||||||
|
[];
|
||||||
|
|
||||||
|
for (const scenarioName of scenarios) {
|
||||||
|
console.log(`\n=== Running scenario: ${scenarioName} ===`);
|
||||||
|
try {
|
||||||
|
const result = await runServerConformanceTest(
|
||||||
|
validated.url,
|
||||||
|
scenarioName
|
||||||
|
);
|
||||||
|
allResults.push({ scenario: scenarioName, checks: result.checks });
|
||||||
|
} catch (error) {
|
||||||
|
console.error(`Failed to run scenario ${scenarioName}:`, error);
|
||||||
|
allResults.push({
|
||||||
|
scenario: scenarioName,
|
||||||
|
checks: [
|
||||||
|
{
|
||||||
|
id: scenarioName,
|
||||||
|
name: scenarioName,
|
||||||
|
description: 'Failed to run scenario',
|
||||||
|
status: 'FAILURE',
|
||||||
|
timestamp: new Date().toISOString(),
|
||||||
|
errorMessage:
|
||||||
|
error instanceof Error ? error.message : String(error)
|
||||||
|
}
|
||||||
|
]
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const { totalFailed } = printServerSummary(allResults);
|
||||||
|
process.exit(totalFailed > 0 ? 1 : 0);
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
if (error instanceof ZodError) {
|
||||||
|
console.error('Validation error:');
|
||||||
|
error.errors.forEach((err) => {
|
||||||
|
console.error(` ${err.path.join('.')}: ${err.message}`);
|
||||||
|
});
|
||||||
|
console.error('\nAvailable server scenarios:');
|
||||||
|
listClientScenarios().forEach((s) => console.error(` - ${s}`));
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
console.error('Server test error:', error);
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// List scenarios command
|
||||||
|
program
|
||||||
|
.command('list')
|
||||||
|
.description('List available test scenarios')
|
||||||
|
.option('--client', 'List client scenarios')
|
||||||
|
.option('--server', 'List server scenarios')
|
||||||
|
.action((options) => {
|
||||||
|
if (options.server || (!options.client && !options.server)) {
|
||||||
|
console.log('Server scenarios (test against a server):');
|
||||||
|
const serverScenarios = listClientScenarios();
|
||||||
|
serverScenarios.forEach((s) => console.log(` - ${s}`));
|
||||||
|
}
|
||||||
|
|
||||||
|
if (options.client || (!options.client && !options.server)) {
|
||||||
|
if (options.server || (!options.client && !options.server)) {
|
||||||
|
console.log('');
|
||||||
|
}
|
||||||
|
console.log('Client scenarios (test against a client):');
|
||||||
|
const clientScenarios = listScenarios();
|
||||||
|
clientScenarios.forEach((s) => console.log(` - ${s}`));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
program.parse();
|
||||||
|
|
@ -1,3 +0,0 @@
|
||||||
import { runConformanceTest } from './runner/index.js';
|
|
||||||
|
|
||||||
export { runConformanceTest };
|
|
||||||
|
|
@ -2,20 +2,45 @@
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
The conformance test runner is a framework for testing MCP client implementations against the MCP specification. It executes client implementations in controlled scenarios and validates their behavior through a series of conformance checks.
|
The conformance test runner is a framework for testing both MCP client and server implementations against the MCP specification. It provides two testing modes:
|
||||||
|
|
||||||
|
1. **Client Testing** - Executes client implementations in controlled scenarios and validates their behavior
|
||||||
|
2. **Server Testing** - Tests server implementations by acting as an MCP client and validating responses
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
|
### File Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
src/runner/
|
||||||
|
├── index.ts # Exports all public functions
|
||||||
|
├── client.ts # Client testing implementation
|
||||||
|
├── server.ts # Server testing implementation
|
||||||
|
└── utils.ts # Shared utilities (formatting, file I/O, ANSI colors)
|
||||||
|
```
|
||||||
|
|
||||||
### Components
|
### Components
|
||||||
|
|
||||||
|
**Client Testing:**
|
||||||
|
|
||||||
1. **Runner** - Orchestrates test execution
|
1. **Runner** - Orchestrates test execution
|
||||||
2. **Test Server** - MCP server (and optionally auth server for auth scenarios)
|
2. **Test Server** - MCP server (and optionally auth server for auth scenarios)
|
||||||
3. **Client Process** - The MCP client implementation under test
|
3. **Client Process** - The MCP client implementation under test
|
||||||
4. **Scenario** - A specific test case with expected behaviors
|
4. **Scenario** - A specific test case with expected behaviors
|
||||||
5. **Checks** - Validation functions that produce ConformanceCheck results
|
5. **Checks** - Validation functions that produce ConformanceCheck results
|
||||||
|
|
||||||
|
**Server Testing:**
|
||||||
|
|
||||||
|
1. **Runner** - Orchestrates test execution
|
||||||
|
2. **Test Client** - MCP SDK client that connects to the server under test
|
||||||
|
3. **Server Process** - The MCP server implementation under test (external)
|
||||||
|
4. **Scenario** - A specific test case with expected server behaviors
|
||||||
|
5. **Checks** - Validation functions that produce ConformanceCheck results
|
||||||
|
|
||||||
### Execution Flow
|
### Execution Flow
|
||||||
|
|
||||||
|
**Client Testing:**
|
||||||
|
|
||||||
```
|
```
|
||||||
1. Runner starts test server(s) on available port(s)
|
1. Runner starts test server(s) on available port(s)
|
||||||
2. Runner spawns client process with server URL as final argument
|
2. Runner spawns client process with server URL as final argument
|
||||||
|
|
@ -25,44 +50,124 @@ The conformance test runner is a framework for testing MCP client implementation
|
||||||
6. Results are written to results/<scenario>-<timestamp>/
|
6. Results are written to results/<scenario>-<timestamp>/
|
||||||
```
|
```
|
||||||
|
|
||||||
## CLI Interface
|
**Server Testing:**
|
||||||
|
|
||||||
```bash
|
```
|
||||||
npm run start -- --command "tsx examples/clients/typescript/test1.ts --verbose" --scenario initialize
|
1. User starts their server implementation
|
||||||
|
2. Runner connects to server URL as an MCP client
|
||||||
|
3. Runner sends requests and captures responses
|
||||||
|
4. Scenario-specific checks validate server behavior
|
||||||
|
5. Results are written to results/server-<scenario>-<timestamp>/
|
||||||
```
|
```
|
||||||
|
|
||||||
### Arguments
|
## CLI Interface
|
||||||
|
|
||||||
|
The conformance suite provides a unified CLI with two main commands:
|
||||||
|
|
||||||
|
### Client Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run a single client scenario
|
||||||
|
npm run start -- client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize
|
||||||
|
|
||||||
|
# With verbose output
|
||||||
|
npm run start -- client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize --verbose
|
||||||
|
|
||||||
|
# With custom timeout
|
||||||
|
npm run start -- client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize --timeout 60000
|
||||||
|
```
|
||||||
|
|
||||||
|
**Arguments:**
|
||||||
|
|
||||||
- `--command` - The command to run the client (can include existing flags)
|
- `--command` - The command to run the client (can include existing flags)
|
||||||
- `--scenario` - The scenario to test (e.g., "initialize", "list-tools", "call-tool")
|
- `--scenario` - The scenario to test (e.g., "initialize", "tools-call")
|
||||||
|
- `--timeout` - Timeout in milliseconds (default: 30000)
|
||||||
|
- `--verbose` - Show verbose output (JSON format)
|
||||||
|
|
||||||
The runner will append the server URL as the final argument to the command.
|
The runner will append the server URL as the final argument to the command.
|
||||||
|
|
||||||
|
### Server Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run a single server scenario
|
||||||
|
npm run start -- server --url http://localhost:3000/mcp --scenario server-initialize
|
||||||
|
|
||||||
|
# Run all server scenarios (default when no --scenario specified)
|
||||||
|
npm run start -- server --url http://localhost:3000/mcp
|
||||||
|
```
|
||||||
|
|
||||||
|
**Arguments:**
|
||||||
|
|
||||||
|
- `--url` - URL of the server to test
|
||||||
|
- `--scenario <scenario>` - Scenario to test (optional, defaults to all scenarios if not specified)
|
||||||
|
|
||||||
|
### List Available Scenarios
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# List all scenarios
|
||||||
|
npm run start -- list
|
||||||
|
|
||||||
|
# List only client scenarios
|
||||||
|
npm run start -- list --client
|
||||||
|
|
||||||
|
# List only server scenarios
|
||||||
|
npm run start -- list --server
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
All CLI arguments are validated using Zod schemas (`src/schemas.ts`) before being passed to runner functions:
|
||||||
|
|
||||||
|
**Client Validation:**
|
||||||
|
|
||||||
|
- Command is non-empty
|
||||||
|
- Scenario exists in available scenarios
|
||||||
|
- Timeout is a positive integer
|
||||||
|
|
||||||
|
**Server Validation:**
|
||||||
|
|
||||||
|
- URL is a valid HTTP/HTTPS URL
|
||||||
|
- All specified scenarios exist (if provided)
|
||||||
|
- Defaults to all scenarios when no `--scenario` is specified
|
||||||
|
|
||||||
## Scenarios
|
## Scenarios
|
||||||
|
|
||||||
A scenario represents a specific test case that validates one or more aspects of MCP client behavior. Each scenario:
|
A scenario represents a specific test case that validates one or more aspects of MCP behavior. Each scenario:
|
||||||
|
|
||||||
- Configures the test server with expected behavior
|
- Configures the test environment with expected behavior
|
||||||
- May run multiple conformance checks
|
- May run multiple conformance checks
|
||||||
- Validates both client behavior and server responses
|
- Validates both request/response patterns and protocol compliance
|
||||||
|
|
||||||
Example scenarios:
|
### Client Scenario Examples
|
||||||
|
|
||||||
- `initialize` - Tests client initialization handshake
|
- `initialize` - Tests client initialization handshake
|
||||||
- `list-tools` - Tests tool discovery
|
- `tools-call` - Tests tool invocation
|
||||||
- `call-tool` - Tests tool invocation
|
- `auth/basic-dcr` - Tests OAuth Dynamic Client Registration flow
|
||||||
- `auth-flow` - Tests OAuth/authorization flows (requires auth server)
|
- `auth/basic-metadata-var1` - Tests OAuth with authorization metadata variation 1
|
||||||
|
|
||||||
|
### Server Scenario Examples
|
||||||
|
|
||||||
|
- `server-initialize` - Tests server initialization and capabilities
|
||||||
|
- `tools-list` - Tests tool listing endpoint
|
||||||
|
- `tools-call-simple-text` - Tests tool invocation with text response
|
||||||
|
- `resources-list` - Tests resource listing
|
||||||
|
- `prompts-list` - Tests prompt listing
|
||||||
|
- `logging-set-level` - Tests logging level configuration
|
||||||
|
|
||||||
## Output Structure
|
## Output Structure
|
||||||
|
|
||||||
Results are written to: `results/<scenario>-<timestamp>/`
|
Results are written to: `results/<scenario>-<timestamp>/` or `results/server-<scenario>-<timestamp>/`
|
||||||
|
|
||||||
Files:
|
**Client Testing Files:**
|
||||||
|
|
||||||
- `checks.json` - Array of ConformanceCheck objects with validation results
|
- `checks.json` - Array of ConformanceCheck objects with validation results
|
||||||
- `stdout.txt` - Complete stdout from the client process
|
- `stdout.txt` - Complete stdout from the client process
|
||||||
- `stderr.txt` - Complete stderr from the client process
|
- `stderr.txt` - Complete stderr from the client process
|
||||||
|
|
||||||
|
**Server Testing Files:**
|
||||||
|
|
||||||
|
- `checks.json` - Array of ConformanceCheck objects with validation results
|
||||||
|
|
||||||
### checks.json Format
|
### checks.json Format
|
||||||
|
|
||||||
```json
|
```json
|
||||||
|
|
@ -84,6 +189,30 @@ Files:
|
||||||
]
|
]
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Programmatic Usage
|
||||||
|
|
||||||
|
The runner can also be used programmatically:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import { runConformanceTest, printClientResults, runServerConformanceTest, printServerResults } from './runner/index.js';
|
||||||
|
|
||||||
|
// Test a client
|
||||||
|
const clientResult = await runConformanceTest('tsx my-client.ts', 'initialize', 30000);
|
||||||
|
const { failed } = printClientResults(clientResult.checks);
|
||||||
|
|
||||||
|
// Test a server
|
||||||
|
const serverResult = await runServerConformanceTest('http://localhost:3000/mcp', 'server-initialize');
|
||||||
|
const { failed } = printServerResults(serverResult.checks);
|
||||||
|
```
|
||||||
|
|
||||||
|
## Shared Utilities
|
||||||
|
|
||||||
|
The `utils.ts` module provides shared functionality:
|
||||||
|
|
||||||
|
- **File Operations:** `ensureResultsDir()`, `createResultDir()`
|
||||||
|
- **Formatting:** `formatPrettyChecks()` for colored console output
|
||||||
|
- **Styling:** ANSI color constants and helpers
|
||||||
|
|
||||||
## Future Enhancements
|
## Future Enhancements
|
||||||
|
|
||||||
- **Test Suites** - Group multiple scenarios for convenience
|
- **Test Suites** - Group multiple scenarios for convenience
|
||||||
|
|
@ -91,3 +220,5 @@ Files:
|
||||||
- **Parallel Execution** - Run multiple scenarios concurrently
|
- **Parallel Execution** - Run multiple scenarios concurrently
|
||||||
- **Custom Servers** - Allow custom MCP server implementations for advanced scenarios
|
- **Custom Servers** - Allow custom MCP server implementations for advanced scenarios
|
||||||
- **Report Formats** - HTML, Markdown, or other human-readable formats
|
- **Report Formats** - HTML, Markdown, or other human-readable formats
|
||||||
|
- **Watch Mode** - Automatically re-run tests on file changes
|
||||||
|
- **Coverage Tracking** - Track which parts of the spec are tested
|
||||||
|
|
|
||||||
207
src/runner/client.ts
Normal file
207
src/runner/client.ts
Normal file
|
|
@ -0,0 +1,207 @@
|
||||||
|
import { spawn } from 'child_process';
|
||||||
|
import { promises as fs } from 'fs';
|
||||||
|
import path from 'path';
|
||||||
|
import { ConformanceCheck } from '../types';
|
||||||
|
import { getScenario } from '../scenarios';
|
||||||
|
import { ensureResultsDir, createResultDir, formatPrettyChecks } from './utils';
|
||||||
|
|
||||||
|
export interface ClientExecutionResult {
|
||||||
|
exitCode: number;
|
||||||
|
stdout: string;
|
||||||
|
stderr: string;
|
||||||
|
timedOut: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function executeClient(
|
||||||
|
command: string,
|
||||||
|
serverUrl: string,
|
||||||
|
timeout: number = 30000
|
||||||
|
): Promise<ClientExecutionResult> {
|
||||||
|
const commandParts = command.split(' ');
|
||||||
|
const executable = commandParts[0];
|
||||||
|
const args = [...commandParts.slice(1), serverUrl];
|
||||||
|
|
||||||
|
let stdout = '';
|
||||||
|
let stderr = '';
|
||||||
|
let timedOut = false;
|
||||||
|
|
||||||
|
return new Promise((resolve) => {
|
||||||
|
const process = spawn(executable, args, {
|
||||||
|
shell: true,
|
||||||
|
stdio: 'pipe'
|
||||||
|
});
|
||||||
|
|
||||||
|
const timeoutHandle = setTimeout(() => {
|
||||||
|
timedOut = true;
|
||||||
|
process.kill();
|
||||||
|
}, timeout);
|
||||||
|
|
||||||
|
if (process.stdout) {
|
||||||
|
process.stdout.on('data', (data) => {
|
||||||
|
stdout += data.toString();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
if (process.stderr) {
|
||||||
|
process.stderr.on('data', (data) => {
|
||||||
|
stderr += data.toString();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
process.on('close', (code) => {
|
||||||
|
clearTimeout(timeoutHandle);
|
||||||
|
resolve({
|
||||||
|
exitCode: code || 0,
|
||||||
|
stdout,
|
||||||
|
stderr,
|
||||||
|
timedOut
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
process.on('error', (error) => {
|
||||||
|
clearTimeout(timeoutHandle);
|
||||||
|
resolve({
|
||||||
|
exitCode: -1,
|
||||||
|
stdout,
|
||||||
|
stderr: stderr + `\nProcess error: ${error.message}`,
|
||||||
|
timedOut
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function runConformanceTest(
|
||||||
|
clientCommand: string,
|
||||||
|
scenarioName: string,
|
||||||
|
timeout: number = 30000
|
||||||
|
): Promise<{
|
||||||
|
checks: ConformanceCheck[];
|
||||||
|
clientOutput: ClientExecutionResult;
|
||||||
|
resultDir: string;
|
||||||
|
}> {
|
||||||
|
await ensureResultsDir();
|
||||||
|
const resultDir = createResultDir(scenarioName);
|
||||||
|
await fs.mkdir(resultDir, { recursive: true });
|
||||||
|
|
||||||
|
// Scenario is guaranteed to exist by CLI validation
|
||||||
|
const scenario = getScenario(scenarioName)!;
|
||||||
|
|
||||||
|
console.log(`Starting scenario: ${scenarioName}`);
|
||||||
|
const urls = await scenario.start();
|
||||||
|
|
||||||
|
console.log(`Executing client: ${clientCommand} ${urls.serverUrl}`);
|
||||||
|
|
||||||
|
try {
|
||||||
|
const clientOutput = await executeClient(
|
||||||
|
clientCommand,
|
||||||
|
urls.serverUrl,
|
||||||
|
timeout
|
||||||
|
);
|
||||||
|
|
||||||
|
// Print stdout/stderr if client exited with nonzero code
|
||||||
|
if (clientOutput.exitCode !== 0) {
|
||||||
|
console.error(`\nClient exited with code ${clientOutput.exitCode}`);
|
||||||
|
if (clientOutput.stdout) {
|
||||||
|
console.error(`\nStdout:\n${clientOutput.stdout}`);
|
||||||
|
}
|
||||||
|
if (clientOutput.stderr) {
|
||||||
|
console.error(`\nStderr:\n${clientOutput.stderr}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (clientOutput.timedOut) {
|
||||||
|
console.error(`\nClient timed out after ${timeout}ms`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const checks = scenario.getChecks();
|
||||||
|
|
||||||
|
await fs.writeFile(
|
||||||
|
path.join(resultDir, 'checks.json'),
|
||||||
|
JSON.stringify(checks, null, 2)
|
||||||
|
);
|
||||||
|
|
||||||
|
await fs.writeFile(path.join(resultDir, 'stdout.txt'), clientOutput.stdout);
|
||||||
|
|
||||||
|
await fs.writeFile(path.join(resultDir, 'stderr.txt'), clientOutput.stderr);
|
||||||
|
|
||||||
|
console.log(`Results saved to ${resultDir}`);
|
||||||
|
|
||||||
|
return {
|
||||||
|
checks,
|
||||||
|
clientOutput,
|
||||||
|
resultDir
|
||||||
|
};
|
||||||
|
} finally {
|
||||||
|
await scenario.stop();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function printClientResults(
|
||||||
|
checks: ConformanceCheck[],
|
||||||
|
verbose: boolean = false
|
||||||
|
): { passed: number; failed: number; denominator: number } {
|
||||||
|
const denominator = checks.filter(
|
||||||
|
(c) => c.status === 'SUCCESS' || c.status === 'FAILURE'
|
||||||
|
).length;
|
||||||
|
const passed = checks.filter((c) => c.status === 'SUCCESS').length;
|
||||||
|
const failed = checks.filter((c) => c.status === 'FAILURE').length;
|
||||||
|
|
||||||
|
if (verbose) {
|
||||||
|
console.log(`Checks:\n${JSON.stringify(checks, null, 2)}`);
|
||||||
|
} else {
|
||||||
|
console.log(`Checks:\n${formatPrettyChecks(checks)}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`\nTest Results:`);
|
||||||
|
console.log(`Passed: ${passed}/${denominator}, ${failed} failed`);
|
||||||
|
|
||||||
|
if (failed > 0) {
|
||||||
|
console.log('\nFailed Checks:');
|
||||||
|
checks
|
||||||
|
.filter((c) => c.status === 'FAILURE')
|
||||||
|
.forEach((c) => {
|
||||||
|
console.log(` - ${c.name}: ${c.description}`);
|
||||||
|
if (c.errorMessage) {
|
||||||
|
console.log(` Error: ${c.errorMessage}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
return { passed, failed, denominator };
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function runInteractiveMode(scenarioName: string): Promise<void> {
|
||||||
|
await ensureResultsDir();
|
||||||
|
const resultDir = createResultDir(scenarioName);
|
||||||
|
await fs.mkdir(resultDir, { recursive: true });
|
||||||
|
|
||||||
|
// Scenario is guaranteed to exist by CLI validation
|
||||||
|
const scenario = getScenario(scenarioName)!;
|
||||||
|
|
||||||
|
console.log(`Starting scenario: ${scenarioName}`);
|
||||||
|
const urls = await scenario.start();
|
||||||
|
|
||||||
|
console.log(`Server URL: ${urls.serverUrl}`);
|
||||||
|
console.log('Press Ctrl+C to stop and save checks...');
|
||||||
|
|
||||||
|
const handleShutdown = async () => {
|
||||||
|
console.log('\nShutting down...');
|
||||||
|
|
||||||
|
const checks = scenario.getChecks();
|
||||||
|
await fs.writeFile(
|
||||||
|
path.join(resultDir, 'checks.json'),
|
||||||
|
JSON.stringify(checks, null, 2)
|
||||||
|
);
|
||||||
|
|
||||||
|
console.log(`\nChecks:\n${JSON.stringify(checks, null, 2)}`);
|
||||||
|
console.log(`\nChecks saved to ${resultDir}/checks.json`);
|
||||||
|
|
||||||
|
await scenario.stop();
|
||||||
|
process.exit(0);
|
||||||
|
};
|
||||||
|
|
||||||
|
process.on('SIGINT', handleShutdown);
|
||||||
|
process.on('SIGTERM', handleShutdown);
|
||||||
|
|
||||||
|
await new Promise(() => {});
|
||||||
|
}
|
||||||
|
|
@ -1,312 +1,23 @@
|
||||||
import { spawn } from 'child_process';
|
// Export client functions
|
||||||
import { promises as fs } from 'fs';
|
export {
|
||||||
import path from 'path';
|
runConformanceTest,
|
||||||
import { ConformanceCheck } from '../types.js';
|
printClientResults,
|
||||||
import { getScenario } from '../scenarios/index.js';
|
runInteractiveMode,
|
||||||
|
type ClientExecutionResult
|
||||||
|
} from './client';
|
||||||
|
|
||||||
// ANSI color codes
|
// Export server functions
|
||||||
const COLORS = {
|
export {
|
||||||
RESET: '\x1b[0m',
|
runServerConformanceTest,
|
||||||
GRAY: '\x1b[90m',
|
printServerResults,
|
||||||
GREEN: '\x1b[32m',
|
printServerSummary
|
||||||
YELLOW: '\x1b[33m',
|
} from './server';
|
||||||
RED: '\x1b[31m',
|
|
||||||
BLUE: '\x1b[36m'
|
|
||||||
};
|
|
||||||
|
|
||||||
function getStatusColor(status: string): string {
|
// Export utilities
|
||||||
switch (status) {
|
export {
|
||||||
case 'SUCCESS':
|
ensureResultsDir,
|
||||||
return COLORS.GREEN;
|
createResultDir,
|
||||||
case 'FAILURE':
|
formatPrettyChecks,
|
||||||
return COLORS.RED;
|
getStatusColor,
|
||||||
case 'INFO':
|
COLORS
|
||||||
return COLORS.BLUE;
|
} from './utils';
|
||||||
default:
|
|
||||||
return COLORS.RESET;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function formatPrettyChecks(checks: ConformanceCheck[]): string {
|
|
||||||
// Find the longest id and status for column alignment
|
|
||||||
const maxIdLength = Math.max(...checks.map((c) => c.id.length));
|
|
||||||
const maxStatusLength = Math.max(...checks.map((c) => c.status.length));
|
|
||||||
|
|
||||||
return checks
|
|
||||||
.map((check) => {
|
|
||||||
const timestamp = `${COLORS.GRAY}${check.timestamp}${COLORS.RESET}`;
|
|
||||||
const id = check.id.padEnd(maxIdLength);
|
|
||||||
const statusColor = getStatusColor(check.status);
|
|
||||||
const status = `${statusColor}${check.status.padEnd(maxStatusLength)}${COLORS.RESET}`;
|
|
||||||
const description = check.description;
|
|
||||||
const line = `${timestamp} [${id}] ${status} ${description}`;
|
|
||||||
// Add newline after outgoing responses for better visual separation
|
|
||||||
return (
|
|
||||||
line +
|
|
||||||
(check.id.includes('outgoing') && check.id.includes('response')
|
|
||||||
? '\n'
|
|
||||||
: '')
|
|
||||||
);
|
|
||||||
})
|
|
||||||
.join('\n');
|
|
||||||
}
|
|
||||||
|
|
||||||
export interface ClientExecutionResult {
|
|
||||||
exitCode: number;
|
|
||||||
stdout: string;
|
|
||||||
stderr: string;
|
|
||||||
timedOut: boolean;
|
|
||||||
}
|
|
||||||
|
|
||||||
async function ensureResultsDir(): Promise<string> {
|
|
||||||
const resultsDir = path.join(process.cwd(), 'results');
|
|
||||||
await fs.mkdir(resultsDir, { recursive: true });
|
|
||||||
return resultsDir;
|
|
||||||
}
|
|
||||||
|
|
||||||
function createResultDir(scenario: string): string {
|
|
||||||
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
|
|
||||||
return path.join('results', `${scenario}-${timestamp}`);
|
|
||||||
}
|
|
||||||
|
|
||||||
async function executeClient(
|
|
||||||
command: string,
|
|
||||||
serverUrl: string,
|
|
||||||
timeout: number = 30000
|
|
||||||
): Promise<ClientExecutionResult> {
|
|
||||||
const commandParts = command.split(' ');
|
|
||||||
const executable = commandParts[0];
|
|
||||||
const args = [...commandParts.slice(1), serverUrl];
|
|
||||||
|
|
||||||
let stdout = '';
|
|
||||||
let stderr = '';
|
|
||||||
let timedOut = false;
|
|
||||||
|
|
||||||
return new Promise((resolve) => {
|
|
||||||
const process = spawn(executable, args, {
|
|
||||||
shell: true,
|
|
||||||
stdio: 'pipe'
|
|
||||||
});
|
|
||||||
|
|
||||||
const timeoutHandle = setTimeout(() => {
|
|
||||||
timedOut = true;
|
|
||||||
process.kill();
|
|
||||||
}, timeout);
|
|
||||||
|
|
||||||
if (process.stdout) {
|
|
||||||
process.stdout.on('data', (data) => {
|
|
||||||
stdout += data.toString();
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
if (process.stderr) {
|
|
||||||
process.stderr.on('data', (data) => {
|
|
||||||
stderr += data.toString();
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
process.on('close', (code) => {
|
|
||||||
clearTimeout(timeoutHandle);
|
|
||||||
resolve({
|
|
||||||
exitCode: code || 0,
|
|
||||||
stdout,
|
|
||||||
stderr,
|
|
||||||
timedOut
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
process.on('error', (error) => {
|
|
||||||
clearTimeout(timeoutHandle);
|
|
||||||
resolve({
|
|
||||||
exitCode: -1,
|
|
||||||
stdout,
|
|
||||||
stderr: stderr + `\nProcess error: ${error.message}`,
|
|
||||||
timedOut
|
|
||||||
});
|
|
||||||
});
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
export async function runConformanceTest(
|
|
||||||
clientCommand: string,
|
|
||||||
scenarioName: string,
|
|
||||||
timeout: number = 30000
|
|
||||||
): Promise<{
|
|
||||||
checks: ConformanceCheck[];
|
|
||||||
clientOutput: ClientExecutionResult;
|
|
||||||
resultDir: string;
|
|
||||||
}> {
|
|
||||||
await ensureResultsDir();
|
|
||||||
const resultDir = createResultDir(scenarioName);
|
|
||||||
await fs.mkdir(resultDir, { recursive: true });
|
|
||||||
|
|
||||||
const scenario = getScenario(scenarioName);
|
|
||||||
if (!scenario) {
|
|
||||||
throw new Error(`Unknown scenario: ${scenarioName}`);
|
|
||||||
}
|
|
||||||
|
|
||||||
console.log(`Starting scenario: ${scenarioName}`);
|
|
||||||
const urls = await scenario.start();
|
|
||||||
|
|
||||||
console.log(`Executing client: ${clientCommand} ${urls.serverUrl}`);
|
|
||||||
|
|
||||||
try {
|
|
||||||
const clientOutput = await executeClient(
|
|
||||||
clientCommand,
|
|
||||||
urls.serverUrl,
|
|
||||||
timeout
|
|
||||||
);
|
|
||||||
|
|
||||||
// Print stdout/stderr if client exited with nonzero code
|
|
||||||
if (clientOutput.exitCode !== 0) {
|
|
||||||
console.error(`\nClient exited with code ${clientOutput.exitCode}`);
|
|
||||||
if (clientOutput.stdout) {
|
|
||||||
console.error(`\nStdout:\n${clientOutput.stdout}`);
|
|
||||||
}
|
|
||||||
if (clientOutput.stderr) {
|
|
||||||
console.error(`\nStderr:\n${clientOutput.stderr}`);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (clientOutput.timedOut) {
|
|
||||||
console.error(`\nClient timed out after ${timeout}ms`);
|
|
||||||
}
|
|
||||||
|
|
||||||
const checks = scenario.getChecks();
|
|
||||||
|
|
||||||
await fs.writeFile(
|
|
||||||
path.join(resultDir, 'checks.json'),
|
|
||||||
JSON.stringify(checks, null, 2)
|
|
||||||
);
|
|
||||||
|
|
||||||
await fs.writeFile(path.join(resultDir, 'stdout.txt'), clientOutput.stdout);
|
|
||||||
|
|
||||||
await fs.writeFile(path.join(resultDir, 'stderr.txt'), clientOutput.stderr);
|
|
||||||
|
|
||||||
console.log(`Results saved to ${resultDir}`);
|
|
||||||
|
|
||||||
return {
|
|
||||||
checks,
|
|
||||||
clientOutput,
|
|
||||||
resultDir
|
|
||||||
};
|
|
||||||
} finally {
|
|
||||||
await scenario.stop();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async function runInteractiveMode(scenarioName: string): Promise<void> {
|
|
||||||
await ensureResultsDir();
|
|
||||||
const resultDir = createResultDir(scenarioName);
|
|
||||||
await fs.mkdir(resultDir, { recursive: true });
|
|
||||||
|
|
||||||
const scenario = getScenario(scenarioName);
|
|
||||||
if (!scenario) {
|
|
||||||
throw new Error(`Unknown scenario: ${scenarioName}`);
|
|
||||||
}
|
|
||||||
|
|
||||||
console.log(`Starting scenario: ${scenarioName}`);
|
|
||||||
const urls = await scenario.start();
|
|
||||||
|
|
||||||
console.log(`Server URL: ${urls.serverUrl}`);
|
|
||||||
console.log('Press Ctrl+C to stop and save checks...');
|
|
||||||
|
|
||||||
const handleShutdown = async () => {
|
|
||||||
console.log('\nShutting down...');
|
|
||||||
|
|
||||||
const checks = scenario.getChecks();
|
|
||||||
await fs.writeFile(
|
|
||||||
path.join(resultDir, 'checks.json'),
|
|
||||||
JSON.stringify(checks, null, 2)
|
|
||||||
);
|
|
||||||
|
|
||||||
console.log(`\nChecks:\n${JSON.stringify(checks, null, 2)}`);
|
|
||||||
console.log(`\nChecks saved to ${resultDir}/checks.json`);
|
|
||||||
|
|
||||||
await scenario.stop();
|
|
||||||
process.exit(0);
|
|
||||||
};
|
|
||||||
|
|
||||||
process.on('SIGINT', handleShutdown);
|
|
||||||
process.on('SIGTERM', handleShutdown);
|
|
||||||
|
|
||||||
await new Promise(() => {});
|
|
||||||
}
|
|
||||||
|
|
||||||
async function main(): Promise<void> {
|
|
||||||
const args = process.argv.slice(2);
|
|
||||||
let command: string | null = null;
|
|
||||||
let scenario: string | null = null;
|
|
||||||
let verbose = false;
|
|
||||||
|
|
||||||
for (let i = 0; i < args.length; i++) {
|
|
||||||
if (args[i] === '--command' && i + 1 < args.length) {
|
|
||||||
command = args[i + 1];
|
|
||||||
i++;
|
|
||||||
} else if (args[i] === '--scenario' && i + 1 < args.length) {
|
|
||||||
scenario = args[i + 1];
|
|
||||||
i++;
|
|
||||||
} else if (args[i] === '--verbose') {
|
|
||||||
verbose = true;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (!scenario) {
|
|
||||||
console.error(
|
|
||||||
'Usage: runner --scenario <scenario> [--command "<command>"] [--verbose]'
|
|
||||||
);
|
|
||||||
console.error(
|
|
||||||
'Example: runner --scenario initialize --command "tsx examples/clients/typescript/test1.ts"'
|
|
||||||
);
|
|
||||||
console.error('Or run without --command for interactive mode');
|
|
||||||
process.exit(1);
|
|
||||||
}
|
|
||||||
|
|
||||||
if (!command) {
|
|
||||||
try {
|
|
||||||
await runInteractiveMode(scenario);
|
|
||||||
} catch (error) {
|
|
||||||
console.error('Interactive mode error:', error);
|
|
||||||
process.exit(1);
|
|
||||||
}
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
try {
|
|
||||||
const result = await runConformanceTest(command, scenario);
|
|
||||||
|
|
||||||
const denominator = result.checks.filter(
|
|
||||||
(c) => c.status === 'SUCCESS' || c.status == 'FAILURE'
|
|
||||||
).length;
|
|
||||||
const passed = result.checks.filter((c) => c.status === 'SUCCESS').length;
|
|
||||||
const failed = result.checks.filter((c) => c.status === 'FAILURE').length;
|
|
||||||
|
|
||||||
if (verbose) {
|
|
||||||
console.log(`Checks:\n${JSON.stringify(result.checks, null, 2)}`);
|
|
||||||
} else {
|
|
||||||
console.log(`Checks:\n${formatPrettyChecks(result.checks)}`);
|
|
||||||
}
|
|
||||||
|
|
||||||
console.log(`\nTest Results:`);
|
|
||||||
console.log(`Passed: ${passed}/${denominator}, ${failed} failed`);
|
|
||||||
|
|
||||||
if (failed > 0) {
|
|
||||||
console.log('\nFailed Checks:');
|
|
||||||
result.checks
|
|
||||||
.filter((c) => c.status === 'FAILURE')
|
|
||||||
.forEach((c) => {
|
|
||||||
console.log(` - ${c.name}: ${c.description}`);
|
|
||||||
if (c.errorMessage) {
|
|
||||||
console.log(` Error: ${c.errorMessage}`);
|
|
||||||
}
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
process.exit(failed > 0 ? 1 : 0);
|
|
||||||
} catch (error) {
|
|
||||||
console.error('Test runner error:', error);
|
|
||||||
process.exit(1);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
main();
|
|
||||||
|
|
|
||||||
93
src/runner/server.ts
Normal file
93
src/runner/server.ts
Normal file
|
|
@ -0,0 +1,93 @@
|
||||||
|
import { promises as fs } from 'fs';
|
||||||
|
import path from 'path';
|
||||||
|
import { ConformanceCheck } from '../types';
|
||||||
|
import { getClientScenario } from '../scenarios';
|
||||||
|
import { ensureResultsDir, createResultDir } from './utils';
|
||||||
|
|
||||||
|
export async function runServerConformanceTest(
|
||||||
|
serverUrl: string,
|
||||||
|
scenarioName: string
|
||||||
|
): Promise<{
|
||||||
|
checks: ConformanceCheck[];
|
||||||
|
resultDir: string;
|
||||||
|
}> {
|
||||||
|
await ensureResultsDir();
|
||||||
|
const resultDir = createResultDir(scenarioName, 'server');
|
||||||
|
await fs.mkdir(resultDir, { recursive: true });
|
||||||
|
|
||||||
|
// Scenario is guaranteed to exist by CLI validation
|
||||||
|
const scenario = getClientScenario(scenarioName)!;
|
||||||
|
|
||||||
|
console.log(
|
||||||
|
`Running client scenario '${scenarioName}' against server: ${serverUrl}`
|
||||||
|
);
|
||||||
|
|
||||||
|
const checks = await scenario.run(serverUrl);
|
||||||
|
|
||||||
|
await fs.writeFile(
|
||||||
|
path.join(resultDir, 'checks.json'),
|
||||||
|
JSON.stringify(checks, null, 2)
|
||||||
|
);
|
||||||
|
|
||||||
|
console.log(`Results saved to ${resultDir}`);
|
||||||
|
|
||||||
|
return {
|
||||||
|
checks,
|
||||||
|
resultDir
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export function printServerResults(checks: ConformanceCheck[]): {
|
||||||
|
passed: number;
|
||||||
|
failed: number;
|
||||||
|
denominator: number;
|
||||||
|
} {
|
||||||
|
const denominator = checks.filter(
|
||||||
|
(c) => c.status === 'SUCCESS' || c.status === 'FAILURE'
|
||||||
|
).length;
|
||||||
|
const passed = checks.filter((c) => c.status === 'SUCCESS').length;
|
||||||
|
const failed = checks.filter((c) => c.status === 'FAILURE').length;
|
||||||
|
|
||||||
|
console.log(`Checks:\n${JSON.stringify(checks, null, 2)}`);
|
||||||
|
|
||||||
|
console.log(`\nTest Results:`);
|
||||||
|
console.log(`Passed: ${passed}/${denominator}, ${failed} failed`);
|
||||||
|
|
||||||
|
if (failed > 0) {
|
||||||
|
console.log('\nFailed Checks:');
|
||||||
|
checks
|
||||||
|
.filter((c) => c.status === 'FAILURE')
|
||||||
|
.forEach((c) => {
|
||||||
|
console.log(` - ${c.name}: ${c.description}`);
|
||||||
|
if (c.errorMessage) {
|
||||||
|
console.log(` Error: ${c.errorMessage}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
return { passed, failed, denominator };
|
||||||
|
}
|
||||||
|
|
||||||
|
export function printServerSummary(
|
||||||
|
allResults: { scenario: string; checks: ConformanceCheck[] }[]
|
||||||
|
): { totalPassed: number; totalFailed: number } {
|
||||||
|
console.log('\n\n=== SUMMARY ===');
|
||||||
|
let totalPassed = 0;
|
||||||
|
let totalFailed = 0;
|
||||||
|
|
||||||
|
for (const result of allResults) {
|
||||||
|
const passed = result.checks.filter((c) => c.status === 'SUCCESS').length;
|
||||||
|
const failed = result.checks.filter((c) => c.status === 'FAILURE').length;
|
||||||
|
totalPassed += passed;
|
||||||
|
totalFailed += failed;
|
||||||
|
|
||||||
|
const status = failed === 0 ? '✓' : '✗';
|
||||||
|
console.log(
|
||||||
|
`${status} ${result.scenario}: ${passed} passed, ${failed} failed`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`\nTotal: ${totalPassed} passed, ${totalFailed} failed`);
|
||||||
|
|
||||||
|
return { totalPassed, totalFailed };
|
||||||
|
}
|
||||||
62
src/runner/utils.ts
Normal file
62
src/runner/utils.ts
Normal file
|
|
@ -0,0 +1,62 @@
|
||||||
|
import { promises as fs } from 'fs';
|
||||||
|
import path from 'path';
|
||||||
|
import { ConformanceCheck } from '../types';
|
||||||
|
|
||||||
|
// ANSI color codes
|
||||||
|
export const COLORS = {
|
||||||
|
RESET: '\x1b[0m',
|
||||||
|
GRAY: '\x1b[90m',
|
||||||
|
GREEN: '\x1b[32m',
|
||||||
|
YELLOW: '\x1b[33m',
|
||||||
|
RED: '\x1b[31m',
|
||||||
|
BLUE: '\x1b[36m'
|
||||||
|
};
|
||||||
|
|
||||||
|
export function getStatusColor(status: string): string {
|
||||||
|
switch (status) {
|
||||||
|
case 'SUCCESS':
|
||||||
|
return COLORS.GREEN;
|
||||||
|
case 'FAILURE':
|
||||||
|
return COLORS.RED;
|
||||||
|
case 'INFO':
|
||||||
|
return COLORS.BLUE;
|
||||||
|
default:
|
||||||
|
return COLORS.RESET;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function formatPrettyChecks(checks: ConformanceCheck[]): string {
|
||||||
|
// Find the longest id and status for column alignment
|
||||||
|
const maxIdLength = Math.max(...checks.map((c) => c.id.length));
|
||||||
|
const maxStatusLength = Math.max(...checks.map((c) => c.status.length));
|
||||||
|
|
||||||
|
return checks
|
||||||
|
.map((check) => {
|
||||||
|
const timestamp = `${COLORS.GRAY}${check.timestamp}${COLORS.RESET}`;
|
||||||
|
const id = check.id.padEnd(maxIdLength);
|
||||||
|
const statusColor = getStatusColor(check.status);
|
||||||
|
const status = `${statusColor}${check.status.padEnd(maxStatusLength)}${COLORS.RESET}`;
|
||||||
|
const description = check.description;
|
||||||
|
const line = `${timestamp} [${id}] ${status} ${description}`;
|
||||||
|
// Add newline after outgoing responses for better visual separation
|
||||||
|
return (
|
||||||
|
line +
|
||||||
|
(check.id.includes('outgoing') && check.id.includes('response')
|
||||||
|
? '\n'
|
||||||
|
: '')
|
||||||
|
);
|
||||||
|
})
|
||||||
|
.join('\n');
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function ensureResultsDir(): Promise<string> {
|
||||||
|
const resultsDir = path.join(process.cwd(), 'results');
|
||||||
|
await fs.mkdir(resultsDir, { recursive: true });
|
||||||
|
return resultsDir;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createResultDir(scenario: string, prefix = ''): string {
|
||||||
|
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
|
||||||
|
const dirName = prefix ? `${prefix}-${scenario}` : scenario;
|
||||||
|
return path.join('results', `${dirName}-${timestamp}`);
|
||||||
|
}
|
||||||
44
src/schemas.ts
Normal file
44
src/schemas.ts
Normal file
|
|
@ -0,0 +1,44 @@
|
||||||
|
import { z } from 'zod';
|
||||||
|
import { getScenario, getClientScenario } from './scenarios';
|
||||||
|
|
||||||
|
// Client command options schema
|
||||||
|
export const ClientOptionsSchema = z.object({
|
||||||
|
command: z.string().min(1, 'Command cannot be empty'),
|
||||||
|
scenario: z
|
||||||
|
.string()
|
||||||
|
.min(1, 'Scenario cannot be empty')
|
||||||
|
.refine(
|
||||||
|
(scenario) => getScenario(scenario) !== undefined,
|
||||||
|
(scenario) => ({
|
||||||
|
message: `Unknown scenario '${scenario}'`
|
||||||
|
})
|
||||||
|
),
|
||||||
|
timeout: z
|
||||||
|
.string()
|
||||||
|
.transform((val) => parseInt(val, 10))
|
||||||
|
.pipe(
|
||||||
|
z
|
||||||
|
.number()
|
||||||
|
.positive('Timeout must be a positive number')
|
||||||
|
.int('Timeout must be an integer')
|
||||||
|
),
|
||||||
|
verbose: z.boolean().optional()
|
||||||
|
});
|
||||||
|
|
||||||
|
export type ClientOptions = z.infer<typeof ClientOptionsSchema>;
|
||||||
|
|
||||||
|
// Server command options schema
|
||||||
|
export const ServerOptionsSchema = z.object({
|
||||||
|
url: z.string().url('Invalid server URL'),
|
||||||
|
scenario: z
|
||||||
|
.string()
|
||||||
|
.refine(
|
||||||
|
(scenario) => getClientScenario(scenario) !== undefined,
|
||||||
|
(scenario) => ({
|
||||||
|
message: `Unknown scenario '${scenario}'`
|
||||||
|
})
|
||||||
|
)
|
||||||
|
.optional()
|
||||||
|
});
|
||||||
|
|
||||||
|
export type ServerOptions = z.infer<typeof ServerOptionsSchema>;
|
||||||
|
|
@ -1,186 +0,0 @@
|
||||||
import { promises as fs } from 'fs';
|
|
||||||
import path from 'path';
|
|
||||||
import { ConformanceCheck } from './types.js';
|
|
||||||
import { getClientScenario } from './scenarios/index.js';
|
|
||||||
|
|
||||||
async function ensureResultsDir(): Promise<string> {
|
|
||||||
const resultsDir = path.join(process.cwd(), 'results');
|
|
||||||
await fs.mkdir(resultsDir, { recursive: true });
|
|
||||||
return resultsDir;
|
|
||||||
}
|
|
||||||
|
|
||||||
function createResultDir(scenario: string): string {
|
|
||||||
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
|
|
||||||
return path.join('results', `server-${scenario}-${timestamp}`);
|
|
||||||
}
|
|
||||||
|
|
||||||
export async function runServerConformanceTest(
|
|
||||||
serverUrl: string,
|
|
||||||
scenarioName: string
|
|
||||||
): Promise<{
|
|
||||||
checks: ConformanceCheck[];
|
|
||||||
resultDir: string;
|
|
||||||
}> {
|
|
||||||
await ensureResultsDir();
|
|
||||||
const resultDir = createResultDir(scenarioName);
|
|
||||||
await fs.mkdir(resultDir, { recursive: true });
|
|
||||||
|
|
||||||
const scenario = getClientScenario(scenarioName);
|
|
||||||
if (!scenario) {
|
|
||||||
throw new Error(`Unknown client scenario: ${scenarioName}`);
|
|
||||||
}
|
|
||||||
|
|
||||||
console.log(
|
|
||||||
`Running client scenario '${scenarioName}' against server: ${serverUrl}`
|
|
||||||
);
|
|
||||||
|
|
||||||
const checks = await scenario.run(serverUrl);
|
|
||||||
|
|
||||||
await fs.writeFile(
|
|
||||||
path.join(resultDir, 'checks.json'),
|
|
||||||
JSON.stringify(checks, null, 2)
|
|
||||||
);
|
|
||||||
|
|
||||||
console.log(`Results saved to ${resultDir}`);
|
|
||||||
|
|
||||||
return {
|
|
||||||
checks,
|
|
||||||
resultDir
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
async function main(): Promise<void> {
|
|
||||||
const args = process.argv.slice(2);
|
|
||||||
let serverUrl: string | null = null;
|
|
||||||
let scenario: string | null = null;
|
|
||||||
let runAll = false;
|
|
||||||
|
|
||||||
for (let i = 0; i < args.length; i++) {
|
|
||||||
if (args[i] === '--server-url' && i + 1 < args.length) {
|
|
||||||
serverUrl = args[i + 1];
|
|
||||||
i++;
|
|
||||||
} else if (args[i] === '--scenario' && i + 1 < args.length) {
|
|
||||||
scenario = args[i + 1];
|
|
||||||
i++;
|
|
||||||
} else if (args[i] === '--all') {
|
|
||||||
runAll = true;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (!serverUrl) {
|
|
||||||
console.error(
|
|
||||||
'Usage: server-runner --server-url <url> [--scenario <scenario> | --all]'
|
|
||||||
);
|
|
||||||
console.error(
|
|
||||||
'Example: server-runner --server-url http://localhost:3000/mcp --scenario server-initialize'
|
|
||||||
);
|
|
||||||
console.error(
|
|
||||||
'Example: server-runner --server-url http://localhost:3000/mcp --all'
|
|
||||||
);
|
|
||||||
process.exit(1);
|
|
||||||
}
|
|
||||||
|
|
||||||
if (!scenario && !runAll) {
|
|
||||||
console.error('Must specify either --scenario or --all');
|
|
||||||
process.exit(1);
|
|
||||||
}
|
|
||||||
|
|
||||||
try {
|
|
||||||
const allResults: { scenario: string; checks: ConformanceCheck[] }[] = [];
|
|
||||||
|
|
||||||
if (runAll) {
|
|
||||||
// Get all server scenarios
|
|
||||||
const { listClientScenarios } = await import('./scenarios/index.js');
|
|
||||||
const scenarios = listClientScenarios();
|
|
||||||
|
|
||||||
console.log(
|
|
||||||
`Running ${scenarios.length} scenarios against ${serverUrl}\n`
|
|
||||||
);
|
|
||||||
|
|
||||||
for (const scenarioName of scenarios) {
|
|
||||||
console.log(`\n=== Running scenario: ${scenarioName} ===`);
|
|
||||||
try {
|
|
||||||
const result = await runServerConformanceTest(
|
|
||||||
serverUrl,
|
|
||||||
scenarioName
|
|
||||||
);
|
|
||||||
allResults.push({ scenario: scenarioName, checks: result.checks });
|
|
||||||
} catch (error) {
|
|
||||||
console.error(`Failed to run scenario ${scenarioName}:`, error);
|
|
||||||
allResults.push({
|
|
||||||
scenario: scenarioName,
|
|
||||||
checks: [
|
|
||||||
{
|
|
||||||
id: scenarioName,
|
|
||||||
name: scenarioName,
|
|
||||||
description: `Failed to run scenario`,
|
|
||||||
status: 'FAILURE',
|
|
||||||
timestamp: new Date().toISOString(),
|
|
||||||
errorMessage:
|
|
||||||
error instanceof Error ? error.message : String(error)
|
|
||||||
}
|
|
||||||
]
|
|
||||||
});
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Summary
|
|
||||||
console.log('\n\n=== SUMMARY ===');
|
|
||||||
let totalPassed = 0;
|
|
||||||
let totalFailed = 0;
|
|
||||||
|
|
||||||
for (const result of allResults) {
|
|
||||||
const passed = result.checks.filter(
|
|
||||||
(c) => c.status === 'SUCCESS'
|
|
||||||
).length;
|
|
||||||
const failed = result.checks.filter(
|
|
||||||
(c) => c.status === 'FAILURE'
|
|
||||||
).length;
|
|
||||||
totalPassed += passed;
|
|
||||||
totalFailed += failed;
|
|
||||||
|
|
||||||
const status = failed === 0 ? '✓' : '✗';
|
|
||||||
console.log(
|
|
||||||
`${status} ${result.scenario}: ${passed} passed, ${failed} failed`
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
console.log(`\nTotal: ${totalPassed} passed, ${totalFailed} failed`);
|
|
||||||
process.exit(totalFailed > 0 ? 1 : 0);
|
|
||||||
} else if (scenario) {
|
|
||||||
const result = await runServerConformanceTest(serverUrl, scenario);
|
|
||||||
|
|
||||||
const denominator = result.checks.filter(
|
|
||||||
(c) => c.status === 'SUCCESS' || c.status == 'FAILURE'
|
|
||||||
).length;
|
|
||||||
const passed = result.checks.filter((c) => c.status === 'SUCCESS').length;
|
|
||||||
const failed = result.checks.filter((c) => c.status === 'FAILURE').length;
|
|
||||||
|
|
||||||
console.log(`Checks:\n${JSON.stringify(result.checks, null, 2)}`);
|
|
||||||
|
|
||||||
console.log(`\nTest Results:`);
|
|
||||||
console.log(`Passed: ${passed}/${denominator}, ${failed} failed`);
|
|
||||||
|
|
||||||
if (failed > 0) {
|
|
||||||
console.log('\nFailed Checks:');
|
|
||||||
result.checks
|
|
||||||
.filter((c) => c.status === 'FAILURE')
|
|
||||||
.forEach((c) => {
|
|
||||||
console.log(` - ${c.name}: ${c.description}`);
|
|
||||||
if (c.errorMessage) {
|
|
||||||
console.log(` Error: ${c.errorMessage}`);
|
|
||||||
}
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
process.exit(failed > 0 ? 1 : 0);
|
|
||||||
}
|
|
||||||
} catch (error) {
|
|
||||||
console.error('Server test runner error:', error);
|
|
||||||
process.exit(1);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (import.meta.url === `file://${process.argv[1]}`) {
|
|
||||||
main();
|
|
||||||
}
|
|
||||||
|
|
@ -3,6 +3,7 @@
|
||||||
"target": "ES2020",
|
"target": "ES2020",
|
||||||
"module": "ES2020",
|
"module": "ES2020",
|
||||||
"moduleResolution": "node",
|
"moduleResolution": "node",
|
||||||
|
"resolveJsonModule": true,
|
||||||
"strict": true,
|
"strict": true,
|
||||||
"esModuleInterop": true,
|
"esModuleInterop": true,
|
||||||
"skipLibCheck": true,
|
"skipLibCheck": true,
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue