diff --git a/.gitignore b/.gitignore index 2db5275..4a1642b 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ node_modules results/ -lefthook-local.yml \ No newline at end of file +lefthook-local.yml +dist \ No newline at end of file diff --git a/README.md b/README.md index 2942831..741b0b7 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # MCP Conformance Test Framework -A framework for testing MCP (Model Context Protocol) client implementations against the specification. +A framework for testing MCP (Model Context Protocol) client and server implementations against the specification. > [!WARNING] This repository is a work in progress and is unstable. Join the conversation in the #conformance-testing-wg in the MCP Contributors discord. @@ -9,19 +9,30 @@ A framework for testing MCP (Model Context Protocol) client implementations agai ### Testing Clients ```bash -npm install -npm run start -- --command "tsx examples/clients/typescript/test1.ts" --scenario initialize +npx @modelcontextprotocol/conformance client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize ``` ### Testing Servers ```bash -npm run test:server -- --server-url http://localhost:3000/mcp --all +# Run all server scenarios (default) +npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp + +# Run a single scenario +npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --scenario server-initialize +``` + +### List Available Scenarios + +```bash +npx @modelcontextprotocol/conformance list ``` ## Overview -The conformance test framework validates MCP client implementations by: +The conformance test framework validates MCP implementations by: + +**For Clients:** 1. Starting a test server for the specified scenario 2. Running the client implementation with the test server URL @@ -29,25 +40,53 @@ The conformance test framework validates MCP client implementations by: 4. Running conformance checks against the specification 5. Generating detailed test results +**For Servers:** + +1. Connecting to the running server as an MCP client +2. Sending test requests and capturing responses +3. Running conformance checks against server behavior +4. Generating detailed test results + ## Usage +### Client Testing + ```bash -npm run start -- --command "" --scenario +npx @modelcontextprotocol/conformance client --command "" --scenario [options] ``` +**Options:** + - `--command` - The command to run your MCP client (can include flags) - `--scenario` - The test scenario to run (e.g., "initialize") +- `--timeout` - Timeout in milliseconds (default: 30000) +- `--verbose` - Show verbose output The framework appends the server URL as the final argument to your command. +### Server Testing + +```bash +npx @modelcontextprotocol/conformance server --url [--scenario ] +``` + +**Options:** + +- `--url` - URL of the server to test +- `--scenario ` - Test scenario to run (e.g., "server-initialize". Runs all available scenarios by default + ## Test Results -Results are saved to `results/-/`: +**Client Testing** - Results are saved to `results/-/`: - `checks.json` - Array of conformance check results with pass/fail status - `stdout.txt` - Client stdout output - `stderr.txt` - Client stderr output +**Server Testing** - Results are saved to `results/server--/`: + +- `checks.json` - Array of conformance check results with pass/fail status + ## Example Clients - `examples/clients/typescript/test1.ts` - Valid MCP client (passes all checks) @@ -55,10 +94,25 @@ Results are saved to `results/-/`: ## Available Scenarios +### Client Scenarios + - **initialize** - Tests MCP client initialization handshake - Validates protocol version - Validates clientInfo (name and version) - Validates server response handling +- **tools-call** - Tests tool invocation +- **auth/basic-dcr** - Tests OAuth Dynamic Client Registration flow +- **auth/basic-metadata-var1** - Tests OAuth with authorization metadata + +### Server Scenarios + +Run `npx @modelcontextprotocol/conformance list --server` to see all available server scenarios, including: + +- **server-initialize** - Tests server initialization and capabilities +- **tools-list** - Tests tool listing endpoint +- **tools-call-\*** - Various tool invocation scenarios +- **resources-\*** - Resource management scenarios +- **prompts-\*** - Prompt management scenarios ## Architecture @@ -67,8 +121,13 @@ See `src/runner/DESIGN.md` for detailed architecture documentation. ### Key Components - **Runner** (`src/runner/`) - Orchestrates test execution and result generation -- **Scenarios** (`src/scenarios/`) - Test scenarios with custom server implementations -- **Checks** (`src/checks.ts`) - Conformance validation functions + - `client.ts` - Client testing implementation + - `server.ts` - Server testing implementation + - `utils.ts` - Shared utilities + - `index.ts` - Public API exports +- **CLI** (`src/index.ts`) - Command-line interface using Commander.js +- **Scenarios** (`src/scenarios/`) - Test scenarios with expected behaviors +- **Checks** (`src/checks/`) - Conformance validation functions - **Types** (`src/types.ts`) - Shared type definitions ## Adding New Scenarios diff --git a/SERVER_REQUIREMENTS.md b/SERVER_REQUIREMENTS.md index 3561ab8..1c8de1f 100644 --- a/SERVER_REQUIREMENTS.md +++ b/SERVER_REQUIREMENTS.md @@ -1022,8 +1022,8 @@ MCP Conformance Test Server running on http://localhost:3000 ```bash # Single test -npm run test:server -- --server-url http://localhost:3000/mcp --scenario server-initialize +npm run start -- server --url http://localhost:3000/mcp --scenario server-initialize # All tests -npm run test:server -- --server-url http://localhost:3000/mcp --all +npm run start -- server --url http://localhost:3000/mcp ``` diff --git a/examples/servers/typescript/README.md b/examples/servers/typescript/README.md index ba5963f..5beacf1 100644 --- a/examples/servers/typescript/README.md +++ b/examples/servers/typescript/README.md @@ -147,8 +147,7 @@ This server implements the MCP Server Conformance Requirements specified in `../ To run conformance tests against this server: ```bash -cd ../../../ -npm run test:server -- --server-url http://localhost:3000/mcp --all +npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp ``` ## Implementation Notes diff --git a/package-lock.json b/package-lock.json index c29a11b..2f8e60c 100644 --- a/package-lock.json +++ b/package-lock.json @@ -9,10 +9,14 @@ "version": "0.1.0", "dependencies": { "@modelcontextprotocol/sdk": "^1.20.1", + "commander": "^14.0.2", "express": "^5.1.0", "lefthook": "^2.0.2", "zod": "^3.25.76" }, + "bin": { + "conformance": "dist/index.mjs" + }, "devDependencies": { "@eslint/js": "^9.8.0", "@jest/globals": "^29.7.0", @@ -23,6 +27,7 @@ "jest": "^29.7.0", "prettier": "3.6.2", "ts-jest": "^29.1.1", + "tsdown": "^0.16.2", "tsx": "^4.7.0", "typescript": "^5.5.4", "typescript-eslint": "^8.0.0" @@ -59,7 +64,6 @@ "integrity": "sha512-e7jT4DxYvIDLk1ZHmU/m/mB19rex9sv0c2ftBtjSBv+kVM/902eh0fINUzD7UwLLNR+jU585GxUJ8/EBfAM5fw==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { "@babel/code-frame": "^7.27.1", "@babel/generator": "^7.28.5", @@ -525,6 +529,40 @@ "dev": true, "license": "MIT" }, + "node_modules/@emnapi/core": { + "version": "1.7.0", + "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.7.0.tgz", + "integrity": "sha512-pJdKGq/1iquWYtv1RRSljZklxHCOCAJFJrImO5ZLKPJVJlVUcs8yFwNQlqS0Lo8xT1VAXXTCZocF9n26FWEKsw==", + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "@emnapi/wasi-threads": "1.1.0", + "tslib": "^2.4.0" + } + }, + "node_modules/@emnapi/runtime": { + "version": "1.7.0", + "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.7.0.tgz", + "integrity": "sha512-oAYoQnCYaQZKVS53Fq23ceWMRxq5EhQsE0x0RdQ55jT7wagMu5k+fS39v1fiSLrtrLQlXwVINenqhLMtTrV/1Q==", + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/@emnapi/wasi-threads": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-1.1.0.tgz", + "integrity": "sha512-WI0DdZ8xFSbgMjR1sFsKABJ/C5OnRrjT06JXbZKexJGrDuPTzZdDYfFlsgcCXCyf+suG5QU2e/y1Wo2V/OapLQ==", + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, "node_modules/@esbuild/aix-ppc64": { "version": "0.25.12", "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.25.12.tgz", @@ -1678,6 +1716,19 @@ } } }, + "node_modules/@napi-rs/wasm-runtime": { + "version": "1.0.7", + "resolved": "https://registry.npmjs.org/@napi-rs/wasm-runtime/-/wasm-runtime-1.0.7.tgz", + "integrity": "sha512-SeDnOO0Tk7Okiq6DbXmmBODgOAb9dp9gjlphokTUxmt8U3liIP1ZsozBahH69j/RJv+Rfs6IwUKHTgQYJ/HBAw==", + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "@emnapi/core": "^1.5.0", + "@emnapi/runtime": "^1.5.0", + "@tybys/wasm-util": "^0.10.1" + } + }, "node_modules/@nodelib/fs.scandir": { "version": "2.1.5", "resolved": "https://registry.npmjs.org/@nodelib/fs.scandir/-/fs.scandir-2.1.5.tgz", @@ -1716,6 +1767,284 @@ "node": ">= 8" } }, + "node_modules/@oxc-project/runtime": { + "version": "0.96.0", + "resolved": "https://registry.npmjs.org/@oxc-project/runtime/-/runtime-0.96.0.tgz", + "integrity": "sha512-34lh4o9CcSw09Hx6fKihPu85+m+4pmDlkXwJrLvN5nMq5JrcGhhihVM415zDqT8j8IixO1PYYdQZRN4SwQCncg==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-project/types": { + "version": "0.96.0", + "resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.96.0.tgz", + "integrity": "sha512-r/xkmoXA0xEpU6UGtn18CNVjXH6erU3KCpCDbpLmbVxBFor1U9MqN5Z2uMmCHJuXjJzlnDR+hWY+yPoLo8oHDw==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/Boshen" + } + }, + "node_modules/@quansync/fs": { + "version": "0.1.5", + "resolved": "https://registry.npmjs.org/@quansync/fs/-/fs-0.1.5.tgz", + "integrity": "sha512-lNS9hL2aS2NZgNW7BBj+6EBl4rOf8l+tQ0eRY6JWCI8jI2kc53gSoqbjojU0OnAWhzoXiOjFyGsHcDGePB3lhA==", + "dev": true, + "license": "MIT", + "dependencies": { + "quansync": "^0.2.11" + }, + "funding": { + "url": "https://github.com/sponsors/sxzz" + } + }, + "node_modules/@rolldown/binding-android-arm64": { + "version": "1.0.0-beta.49", + "resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.0-beta.49.tgz", + "integrity": "sha512-xKQEOmqOet0vFHt/aqcoQGWvoDJhfSO8EBhuST0CDnxQRmnVzbI8keeeX62vi53ZyICKZxczyfx4A8dUY3dqKw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-darwin-arm64": { + "version": "1.0.0-beta.49", + "resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.0-beta.49.tgz", + "integrity": "sha512-kN0N/8m8HUYO13PqlIwxcXD7fu2E6GKu0J4iH7wUJw3T3QK+nvrc20rxtTZ0J6sA1sGCE8UYvvvnurDwMUp0dg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-darwin-x64": { + "version": "1.0.0-beta.49", + "resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.0-beta.49.tgz", + "integrity": "sha512-29qmvsgY2A4ymfy8sQkFFOFc13m04SLUcYn1iil41gpkYrAspBLkvsOQMHPCs3rQCOImgweT4tFotqTAonwphQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-freebsd-x64": { + "version": "1.0.0-beta.49", + "resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.0-beta.49.tgz", + "integrity": "sha512-fY+esrHjgt6+RAnDPuUk39RvFNmYhJekGyC6wr0HWXGTBed07Feap9BrYINSh6x5xFlNpOPs6tImKnV0zVDuWQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-arm-gnueabihf": { + "version": "1.0.0-beta.49", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.0-beta.49.tgz", + "integrity": "sha512-oQNAqB+XrRM2AZaSPyudQETsPhzCZqgPICQu80fJuNyBFYoz6nonNNZtm3BJ9uP+HZfUk9NfOn9vPoCNuk6gAw==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-arm64-gnu": { + "version": "1.0.0-beta.49", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.0-beta.49.tgz", + "integrity": "sha512-bJinAiuWUJvlBxPa8ZmRnWkmmAoUlSWtZT4pRkWi/QX3HlgHfUUbhF+d7aZLciai+iFfbiPqOwCL2tqNXXrUsA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-arm64-musl": { + "version": "1.0.0-beta.49", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.0-beta.49.tgz", + "integrity": "sha512-gwueY8EJU7afq5tNwKSjYy5JqTR/0MNzZfv6s5dX+rMgeUpTNhwIToLO1F41TPYEa+6LRTXUWG23DO/ONPzUJA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-x64-gnu": { + "version": "1.0.0-beta.49", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.0-beta.49.tgz", + "integrity": "sha512-VXYkjzzEZh5N5Ue1IEcBgL8RuJu5jWrIKmg8WY6hhCbnNJ1IOsObT4HFW+rE8ZaKNjoIXzImoiYi1UAkKiQRYA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-x64-musl": { + "version": "1.0.0-beta.49", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.0-beta.49.tgz", + "integrity": "sha512-S5Yw6g/ftiW7MpNpnOM5vSIlDzGuohDY8y7VOI47+92HhO6WqsNfcMkDZXm3G5l6YIfUNStGBV86NWrzasp+sw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-openharmony-arm64": { + "version": "1.0.0-beta.49", + "resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.0-beta.49.tgz", + "integrity": "sha512-bhRoMO2oP46W1UDd/PTrSdoIYfvLS2jiFAned0SOzOO0tcait9u+b9i8h4ZugbT2IK4qUXNezovbHJs7hKJOEQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openharmony" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-wasm32-wasi": { + "version": "1.0.0-beta.49", + "resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.0-beta.49.tgz", + "integrity": "sha512-Owp6Y1RQ84UMOV8hrg5e1Fmu8Po1IUXWytAHUtPcc00+ty6Gr9g5GgLLw0oblu7QovBr4848ozvkMcEj3vDKgA==", + "cpu": [ + "wasm32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "@napi-rs/wasm-runtime": "^1.0.7" + }, + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/@rolldown/binding-win32-arm64-msvc": { + "version": "1.0.0-beta.49", + "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.0-beta.49.tgz", + "integrity": "sha512-dnheX8aXsN9P12uwPOW3TVvqSnQ1cfjKQlYgU2dTkrRpnco0kTGvqE1nEWybGukTyuPdzVvrGElgSGEJ7crcSQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-win32-ia32-msvc": { + "version": "1.0.0-beta.49", + "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-ia32-msvc/-/binding-win32-ia32-msvc-1.0.0-beta.49.tgz", + "integrity": "sha512-Blt1aODXiAuEdZBqHYXGJwVFlonXKkVEJy5hhxOgnAVi/0mzFNWDxc8qVlxl7dpQjQdboW/wXdgMHpTDfomicg==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-win32-x64-msvc": { + "version": "1.0.0-beta.49", + "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.0-beta.49.tgz", + "integrity": "sha512-sSu4qUlL/62QJrR3P+Bd+EblD8tUpnovUz65qow3PA7YxH+f5NFDbCJMR1m5b8zBuVZwZIHfzbuawz+Vl34/xg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/pluginutils": { + "version": "1.0.0-beta.49", + "resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-beta.49.tgz", + "integrity": "sha512-HLlu3Qn3ePmNCbfehwKWXQMzX/2rzcL6Jmpo+Dl3xnq46TGMyJAgO+IsS8ka7IDLeD3wcoOhjJwxTdIdbrFhGw==", + "dev": true, + "license": "MIT" + }, "node_modules/@sinclair/typebox": { "version": "0.27.8", "resolved": "https://registry.npmjs.org/@sinclair/typebox/-/typebox-0.27.8.tgz", @@ -1743,6 +2072,17 @@ "@sinonjs/commons": "^3.0.0" } }, + "node_modules/@tybys/wasm-util": { + "version": "0.10.1", + "resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.10.1.tgz", + "integrity": "sha512-9tTaPJLSiejZKx+Bmog4uSubteqTvFrVrURwkmHixBo0G4seD0zUxp98E1DzUBJxLQ3NPwXrGKDiVjwx/DpPsg==", + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, "node_modules/@types/babel__core": { "version": "7.20.5", "resolved": "https://registry.npmjs.org/@types/babel__core/-/babel__core-7.20.5.tgz", @@ -2037,7 +2377,6 @@ "integrity": "sha512-6m1I5RmHBGTnUGS113G04DMu3CpSdxCAU/UvtjNWL4Nuf3MW9tQhiJqRlHzChIkhy6kZSAQmc+I1bcGjE3yNKg==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { "@typescript-eslint/scope-manager": "8.46.3", "@typescript-eslint/types": "8.46.3", @@ -2282,7 +2621,6 @@ "integrity": "sha512-NZyJarBfL7nWwIq+FDL6Zp/yHEhePMNnnJ0y3qfieCrmNvYct8uvtiV41UvlSe6apAfk0fY1FbWx+NwfmpvtTg==", "dev": true, "license": "MIT", - "peer": true, "bin": { "acorn": "bin/acorn" }, @@ -2375,6 +2713,16 @@ "url": "https://github.com/chalk/ansi-styles?sponsor=1" } }, + "node_modules/ansis": { + "version": "4.2.0", + "resolved": "https://registry.npmjs.org/ansis/-/ansis-4.2.0.tgz", + "integrity": "sha512-HqZ5rWlFjGiV0tDm3UxxgNRqsOTniqoKZu0pIAfh7TZQMGuZK+hH0drySty0si0QXj1ieop4+SkSfPZBPPkHig==", + "dev": true, + "license": "ISC", + "engines": { + "node": ">=14" + } + }, "node_modules/anymatch": { "version": "3.1.3", "resolved": "https://registry.npmjs.org/anymatch/-/anymatch-3.1.3.tgz", @@ -2396,6 +2744,23 @@ "dev": true, "license": "Python-2.0" }, + "node_modules/ast-kit": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/ast-kit/-/ast-kit-2.2.0.tgz", + "integrity": "sha512-m1Q/RaVOnTp9JxPX+F+Zn7IcLYMzM8kZofDImfsKZd8MbR+ikdOzTeztStWqfrqIxZnYWryyI9ePm3NGjnZgGw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@babel/parser": "^7.28.5", + "pathe": "^2.0.3" + }, + "engines": { + "node": ">=20.19.0" + }, + "funding": { + "url": "https://github.com/sponsors/sxzz" + } + }, "node_modules/babel-jest": { "version": "29.7.0", "resolved": "https://registry.npmjs.org/babel-jest/-/babel-jest-29.7.0.tgz", @@ -2529,6 +2894,16 @@ "baseline-browser-mapping": "dist/cli.js" } }, + "node_modules/birpc": { + "version": "2.8.0", + "resolved": "https://registry.npmjs.org/birpc/-/birpc-2.8.0.tgz", + "integrity": "sha512-Bz2a4qD/5GRhiHSwj30c/8kC8QGj12nNDwz3D4ErQ4Xhy35dsSDvF+RA/tWpjyU0pdGtSDiEk6B5fBGE1qNVhw==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/antfu" + } + }, "node_modules/body-parser": { "version": "2.2.0", "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-2.2.0.tgz", @@ -2593,7 +2968,6 @@ } ], "license": "MIT", - "peer": true, "dependencies": { "baseline-browser-mapping": "^2.8.19", "caniuse-lite": "^1.0.30001751", @@ -2647,6 +3021,16 @@ "node": ">= 0.8" } }, + "node_modules/cac": { + "version": "6.7.14", + "resolved": "https://registry.npmjs.org/cac/-/cac-6.7.14.tgz", + "integrity": "sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=8" + } + }, "node_modules/call-bind-apply-helpers": { "version": "1.0.2", "resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz", @@ -2744,6 +3128,22 @@ "node": ">=10" } }, + "node_modules/chokidar": { + "version": "4.0.3", + "resolved": "https://registry.npmjs.org/chokidar/-/chokidar-4.0.3.tgz", + "integrity": "sha512-Qgzu8kfBvo+cA4962jnP1KkS6Dop5NS6g7R5LFYJr4b8Ub94PPQXUksCw9PvXoeXPRRddRNC5C1JQUR2SMGtnA==", + "dev": true, + "license": "MIT", + "dependencies": { + "readdirp": "^4.0.1" + }, + "engines": { + "node": ">= 14.16.0" + }, + "funding": { + "url": "https://paulmillr.com/funding/" + } + }, "node_modules/ci-info": { "version": "3.9.0", "resolved": "https://registry.npmjs.org/ci-info/-/ci-info-3.9.0.tgz", @@ -2820,6 +3220,15 @@ "dev": true, "license": "MIT" }, + "node_modules/commander": { + "version": "14.0.2", + "resolved": "https://registry.npmjs.org/commander/-/commander-14.0.2.tgz", + "integrity": "sha512-TywoWNNRbhoD0BXs1P3ZEScW8W5iKrnbithIl0YH+uCmBd0QpPOA8yc82DS3BIE5Ma6FnBVUsJ7wVUDz4dvOWQ==", + "license": "MIT", + "engines": { + "node": ">=20" + } + }, "node_modules/concat-map": { "version": "0.0.1", "resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz", @@ -2990,6 +3399,16 @@ "node": ">=8" } }, + "node_modules/diff": { + "version": "8.0.2", + "resolved": "https://registry.npmjs.org/diff/-/diff-8.0.2.tgz", + "integrity": "sha512-sSuxWU5j5SR9QQji/o2qMvqRNYRDOcBTgsJ/DeCf4iSN4gW+gNMXM7wFIP+fdXZxoNiAnHUTGjCr+TSWXdRDKg==", + "dev": true, + "license": "BSD-3-Clause", + "engines": { + "node": ">=0.3.1" + } + }, "node_modules/diff-sequences": { "version": "29.6.3", "resolved": "https://registry.npmjs.org/diff-sequences/-/diff-sequences-29.6.3.tgz", @@ -3000,6 +3419,27 @@ "node": "^14.15.0 || ^16.10.0 || >=18.0.0" } }, + "node_modules/dts-resolver": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/dts-resolver/-/dts-resolver-2.1.3.tgz", + "integrity": "sha512-bihc7jPC90VrosXNzK0LTE2cuLP6jr0Ro8jk+kMugHReJVLIpHz/xadeq3MhuwyO4TD4OA3L1Q8pBBFRc08Tsw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=20.19.0" + }, + "funding": { + "url": "https://github.com/sponsors/sxzz" + }, + "peerDependencies": { + "oxc-resolver": ">=11.0.0" + }, + "peerDependenciesMeta": { + "oxc-resolver": { + "optional": true + } + } + }, "node_modules/dunder-proto": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz", @@ -3047,6 +3487,16 @@ "dev": true, "license": "MIT" }, + "node_modules/empathic": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/empathic/-/empathic-2.0.0.tgz", + "integrity": "sha512-i6UzDscO/XfAcNYD75CfICkmfLedpyPDdozrLMmQc5ORaQcdMoc21OnlEylMIqI7U8eniKrPMxxtj8k0vhmJhA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=14" + } + }, "node_modules/encodeurl": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz", @@ -3173,7 +3623,6 @@ "integrity": "sha512-BhHmn2yNOFA9H9JmmIVKJmd288g9hrVRDkdoIgRCRuSySRUHH7r/DI6aAXW9T1WwUuY3DFgrcaqB+deURBLR5g==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { "@eslint-community/eslint-utils": "^4.8.0", "@eslint-community/regexpp": "^4.12.1", @@ -3461,7 +3910,6 @@ "resolved": "https://registry.npmjs.org/express/-/express-5.1.0.tgz", "integrity": "sha512-DT9ck5YIRU+8GYzzU5kT3eHGA5iL+1Zd0EutOmTE9Dtk+Tvuzd23VBU+ec7HPNSTxXYO55gPV/hq4pSBJDjFpA==", "license": "MIT", - "peer": true, "dependencies": { "accepts": "^2.0.0", "body-parser": "^2.2.0", @@ -3953,6 +4401,13 @@ "node": ">= 0.4" } }, + "node_modules/hookable": { + "version": "5.5.3", + "resolved": "https://registry.npmjs.org/hookable/-/hookable-5.5.3.tgz", + "integrity": "sha512-Yc+BQe8SvoXH1643Qez1zqLRmbA5rCL+sSmk6TVos0LWVfNIB7PGncdlId77WzLGSIB5KaWgTaNTs2lNVEI6VQ==", + "dev": true, + "license": "MIT" + }, "node_modules/html-escaper": { "version": "2.0.2", "resolved": "https://registry.npmjs.org/html-escaper/-/html-escaper-2.0.2.tgz", @@ -4282,7 +4737,6 @@ "integrity": "sha512-NIy3oAFp9shda19hy4HK0HRTWKtPJmGdnvywu01nOqNC2vZg+Z+fvJDxpMQA88eb2I9EcafcdjYgsDthnYTvGw==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { "@jest/core": "^29.7.0", "@jest/types": "^29.6.3", @@ -5189,6 +5643,16 @@ "yallist": "^3.0.2" } }, + "node_modules/magic-string": { + "version": "0.30.21", + "resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.21.tgz", + "integrity": "sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/sourcemap-codec": "^1.5.5" + } + }, "node_modules/make-dir": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/make-dir/-/make-dir-4.0.0.tgz", @@ -5621,6 +6085,13 @@ "url": "https://opencollective.com/express" } }, + "node_modules/pathe": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/pathe/-/pathe-2.0.3.tgz", + "integrity": "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w==", + "dev": true, + "license": "MIT" + }, "node_modules/picocolors": { "version": "1.1.1", "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", @@ -5852,6 +6323,23 @@ "url": "https://github.com/sponsors/ljharb" } }, + "node_modules/quansync": { + "version": "0.2.11", + "resolved": "https://registry.npmjs.org/quansync/-/quansync-0.2.11.tgz", + "integrity": "sha512-AifT7QEbW9Nri4tAwR5M/uzpBuqfZf+zwaEM/QkzEjj7NBuFD2rBuy0K3dE+8wltbezDV7JMA0WfnCPYRSYbXA==", + "dev": true, + "funding": [ + { + "type": "individual", + "url": "https://github.com/sponsors/antfu" + }, + { + "type": "individual", + "url": "https://github.com/sponsors/sxzz" + } + ], + "license": "MIT" + }, "node_modules/queue-microtask": { "version": "1.2.3", "resolved": "https://registry.npmjs.org/queue-microtask/-/queue-microtask-1.2.3.tgz", @@ -5920,6 +6408,20 @@ "dev": true, "license": "MIT" }, + "node_modules/readdirp": { + "version": "4.1.2", + "resolved": "https://registry.npmjs.org/readdirp/-/readdirp-4.1.2.tgz", + "integrity": "sha512-GDhwkLfywWL2s6vEjyhri+eXmfH6j1L7JE27WhqLeYzoh/A3DBaYGEj2H/HFZCn/kMfim73FXxEJTw06WtxQwg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 14.18.0" + }, + "funding": { + "type": "individual", + "url": "https://paulmillr.com/funding/" + } + }, "node_modules/require-directory": { "version": "2.1.1", "resolved": "https://registry.npmjs.org/require-directory/-/require-directory-2.1.1.tgz", @@ -6024,6 +6526,84 @@ "node": ">=0.10.0" } }, + "node_modules/rolldown": { + "version": "1.0.0-beta.49", + "resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.0-beta.49.tgz", + "integrity": "sha512-Bfmdn3ZqyCwi1LxG39KBrSlil9a/xnrOrAj+jqqN2YTR/WJIEOOfwNKgDALQvr0xlO9bG/i1C883KGd4nd7SrA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@oxc-project/types": "=0.96.0", + "@rolldown/pluginutils": "1.0.0-beta.49" + }, + "bin": { + "rolldown": "bin/cli.mjs" + }, + "engines": { + "node": "^20.19.0 || >=22.12.0" + }, + "optionalDependencies": { + "@rolldown/binding-android-arm64": "1.0.0-beta.49", + "@rolldown/binding-darwin-arm64": "1.0.0-beta.49", + "@rolldown/binding-darwin-x64": "1.0.0-beta.49", + "@rolldown/binding-freebsd-x64": "1.0.0-beta.49", + "@rolldown/binding-linux-arm-gnueabihf": "1.0.0-beta.49", + "@rolldown/binding-linux-arm64-gnu": "1.0.0-beta.49", + "@rolldown/binding-linux-arm64-musl": "1.0.0-beta.49", + "@rolldown/binding-linux-x64-gnu": "1.0.0-beta.49", + "@rolldown/binding-linux-x64-musl": "1.0.0-beta.49", + "@rolldown/binding-openharmony-arm64": "1.0.0-beta.49", + "@rolldown/binding-wasm32-wasi": "1.0.0-beta.49", + "@rolldown/binding-win32-arm64-msvc": "1.0.0-beta.49", + "@rolldown/binding-win32-ia32-msvc": "1.0.0-beta.49", + "@rolldown/binding-win32-x64-msvc": "1.0.0-beta.49" + } + }, + "node_modules/rolldown-plugin-dts": { + "version": "0.17.5", + "resolved": "https://registry.npmjs.org/rolldown-plugin-dts/-/rolldown-plugin-dts-0.17.5.tgz", + "integrity": "sha512-dYzjLdhgsSIPmOCPJdDXiD6AUotAHVkGLNlSMdi3VmvTqk7O9H7uGzG2WGzgnDxZq6kZh31vZUbp9ZFoOhpySA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@babel/generator": "^7.28.5", + "@babel/parser": "^7.28.5", + "@babel/types": "^7.28.5", + "ast-kit": "^2.2.0", + "birpc": "^2.8.0", + "debug": "^4.4.3", + "dts-resolver": "^2.1.3", + "get-tsconfig": "^4.13.0", + "magic-string": "^0.30.21" + }, + "engines": { + "node": ">=20.18.0" + }, + "funding": { + "url": "https://github.com/sponsors/sxzz" + }, + "peerDependencies": { + "@ts-macro/tsc": "^0.3.6", + "@typescript/native-preview": ">=7.0.0-dev.20250601.1", + "rolldown": "^1.0.0-beta.44", + "typescript": "^5.0.0", + "vue-tsc": "~3.1.0" + }, + "peerDependenciesMeta": { + "@ts-macro/tsc": { + "optional": true + }, + "@typescript/native-preview": { + "optional": true + }, + "typescript": { + "optional": true + }, + "vue-tsc": { + "optional": true + } + } + }, "node_modules/router": { "version": "2.2.0", "resolved": "https://registry.npmjs.org/router/-/router-2.2.0.tgz", @@ -6436,6 +7016,64 @@ "node": ">=8" } }, + "node_modules/tinyexec": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-1.0.2.tgz", + "integrity": "sha512-W/KYk+NFhkmsYpuHq5JykngiOCnxeVL8v8dFnqxSD8qEEdRfXk1SDM6JzNqcERbcGYj9tMrDQBYV9cjgnunFIg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18" + } + }, + "node_modules/tinyglobby": { + "version": "0.2.15", + "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.15.tgz", + "integrity": "sha512-j2Zq4NyQYG5XMST4cbs02Ak8iJUdxRM0XI5QyxXuZOzKOINmWurp3smXu3y5wDcJrptwpSjgXHzIQxR0omXljQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "fdir": "^6.5.0", + "picomatch": "^4.0.3" + }, + "engines": { + "node": ">=12.0.0" + }, + "funding": { + "url": "https://github.com/sponsors/SuperchupuDev" + } + }, + "node_modules/tinyglobby/node_modules/fdir": { + "version": "6.5.0", + "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.5.0.tgz", + "integrity": "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12.0.0" + }, + "peerDependencies": { + "picomatch": "^3 || ^4" + }, + "peerDependenciesMeta": { + "picomatch": { + "optional": true + } + } + }, + "node_modules/tinyglobby/node_modules/picomatch": { + "version": "4.0.3", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.3.tgz", + "integrity": "sha512-5gTmgEY/sqK6gFXLIsQNH19lWb4ebPDLA4SdLP7dsWkIXHWlG66oPuVvXSGFPppYZz8ZDZq0dYYrbHfBCVUb1Q==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/jonschlinkert" + } + }, "node_modules/tmpl": { "version": "1.0.5", "resolved": "https://registry.npmjs.org/tmpl/-/tmpl-1.0.5.tgz", @@ -6465,6 +7103,16 @@ "node": ">=0.6" } }, + "node_modules/tree-kill": { + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/tree-kill/-/tree-kill-1.2.2.tgz", + "integrity": "sha512-L0Orpi8qGpRG//Nd+H90vFB+3iHnue1zSSGmNOOCh1GLJ7rUKVwV2HvijphGQS2UmhUZewS9VgvxYIdgr+fG1A==", + "dev": true, + "license": "MIT", + "bin": { + "tree-kill": "cli.js" + } + }, "node_modules/ts-api-utils": { "version": "2.1.0", "resolved": "https://registry.npmjs.org/ts-api-utils/-/ts-api-utils-2.1.0.tgz", @@ -6557,6 +7205,88 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/tsdown": { + "version": "0.16.2", + "resolved": "https://registry.npmjs.org/tsdown/-/tsdown-0.16.2.tgz", + "integrity": "sha512-ZyHFO5DduN2TxY47hHPGZRZz5XSQbDYFl8L4BkLJlzYeHuX0cx5mZkonq7Xm4OC/z8BxWY+ai4kwvevup4Lwlw==", + "dev": true, + "license": "MIT", + "dependencies": { + "ansis": "^4.2.0", + "cac": "^6.7.14", + "chokidar": "^4.0.3", + "debug": "^4.4.3", + "diff": "^8.0.2", + "empathic": "^2.0.0", + "hookable": "^5.5.3", + "rolldown": "1.0.0-beta.49", + "rolldown-plugin-dts": "^0.17.5", + "semver": "^7.7.3", + "tinyexec": "^1.0.2", + "tinyglobby": "^0.2.15", + "tree-kill": "^1.2.2", + "unconfig-core": "^7.4.1", + "unrun": "^0.2.7" + }, + "bin": { + "tsdown": "dist/run.mjs" + }, + "engines": { + "node": ">=20.19.0" + }, + "funding": { + "url": "https://github.com/sponsors/sxzz" + }, + "peerDependencies": { + "@arethetypeswrong/core": "^0.18.1", + "@vitejs/devtools": "^0.0.0-alpha.17", + "publint": "^0.3.0", + "typescript": "^5.0.0", + "unplugin-lightningcss": "^0.4.0", + "unplugin-unused": "^0.5.0" + }, + "peerDependenciesMeta": { + "@arethetypeswrong/core": { + "optional": true + }, + "@vitejs/devtools": { + "optional": true + }, + "publint": { + "optional": true + }, + "typescript": { + "optional": true + }, + "unplugin-lightningcss": { + "optional": true + }, + "unplugin-unused": { + "optional": true + } + } + }, + "node_modules/tsdown/node_modules/semver": { + "version": "7.7.3", + "resolved": "https://registry.npmjs.org/semver/-/semver-7.7.3.tgz", + "integrity": "sha512-SdsKMrI9TdgjdweUSR9MweHA4EJ8YxHn8DFaDisvhVlUOe4BF1tLD7GAj0lIqWVl+dPb/rExr0Btby5loQm20Q==", + "dev": true, + "license": "ISC", + "bin": { + "semver": "bin/semver.js" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/tslib": { + "version": "2.8.1", + "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", + "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", + "dev": true, + "license": "0BSD", + "optional": true + }, "node_modules/tsx": { "version": "4.20.6", "resolved": "https://registry.npmjs.org/tsx/-/tsx-4.20.6.tgz", @@ -6633,7 +7363,6 @@ "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==", "dev": true, "license": "Apache-2.0", - "peer": true, "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" @@ -6680,6 +7409,20 @@ "node": ">=0.8.0" } }, + "node_modules/unconfig-core": { + "version": "7.4.1", + "resolved": "https://registry.npmjs.org/unconfig-core/-/unconfig-core-7.4.1.tgz", + "integrity": "sha512-Bp/bPZjV2Vl/fofoA2OYLSnw1Z0MOhCX7zHnVCYrazpfZvseBbGhwcNQMxsg185Mqh7VZQqK3C8hFG/Dyng+yA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@quansync/fs": "^0.1.5", + "quansync": "^0.2.11" + }, + "funding": { + "url": "https://github.com/sponsors/antfu" + } + }, "node_modules/undici-types": { "version": "7.16.0", "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.16.0.tgz", @@ -6696,6 +7439,34 @@ "node": ">= 0.8" } }, + "node_modules/unrun": { + "version": "0.2.8", + "resolved": "https://registry.npmjs.org/unrun/-/unrun-0.2.8.tgz", + "integrity": "sha512-2HnCvemvMN7qQm0/1uhhi8JoVHQxfkAhZTCdFNWAK35dXKJlOHpcJ6ayN23JKJZyAXjAwpVZZr878flfRiaiFQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@oxc-project/runtime": "^0.96.0", + "rolldown": "1.0.0-beta.49" + }, + "bin": { + "unrun": "dist/cli.mjs" + }, + "engines": { + "node": ">=20.19.0" + }, + "funding": { + "url": "https://github.com/sponsors/Gugustinette" + }, + "peerDependencies": { + "synckit": "^0.11.11" + }, + "peerDependenciesMeta": { + "synckit": { + "optional": true + } + } + }, "node_modules/update-browserslist-db": { "version": "1.1.4", "resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.1.4.tgz", @@ -6905,7 +7676,6 @@ "resolved": "https://registry.npmjs.org/zod/-/zod-3.25.76.tgz", "integrity": "sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ==", "license": "MIT", - "peer": true, "funding": { "url": "https://github.com/sponsors/colinhacks" } diff --git a/package.json b/package.json index 199a95c..ef93bc8 100644 --- a/package.json +++ b/package.json @@ -3,13 +3,19 @@ "version": "0.1.0", "type": "module", "scripts": { - "start": "tsx src/runner.ts", - "test:server": "tsx src/server-runner.ts", + "start": "tsx src/index.ts", "test": "jest", + "build": "tsdown src/index.ts --minify --clean --target node20", "lint": "eslint src/ && prettier --check .", "lint:fix": "eslint src/ --fix && prettier --write .", "lint:fix_check": "npm run lint:fix && git diff --exit-code --quiet" }, + "files": [ + "dist" + ], + "bin": { + "conformance": "dist/index.mjs" + }, "devDependencies": { "@eslint/js": "^9.8.0", "@jest/globals": "^29.7.0", @@ -20,12 +26,14 @@ "jest": "^29.7.0", "prettier": "3.6.2", "ts-jest": "^29.1.1", + "tsdown": "^0.16.2", "tsx": "^4.7.0", "typescript": "^5.5.4", "typescript-eslint": "^8.0.0" }, "dependencies": { "@modelcontextprotocol/sdk": "^1.20.1", + "commander": "^14.0.2", "express": "^5.1.0", "lefthook": "^2.0.2", "zod": "^3.25.76" diff --git a/src/index.ts b/src/index.ts new file mode 100644 index 0000000..d35c676 --- /dev/null +++ b/src/index.ts @@ -0,0 +1,164 @@ +#!/usr/bin/env node + +import { Command } from 'commander'; +import { ZodError } from 'zod'; +import { + runConformanceTest, + printClientResults, + runServerConformanceTest, + printServerResults, + printServerSummary +} from './runner'; +import { listScenarios, listClientScenarios } from './scenarios'; +import { ConformanceCheck } from './types'; +import { ClientOptionsSchema, ServerOptionsSchema } from './schemas'; +import packageJson from '../package.json'; + +const program = new Command(); + +program + .name('conformance') + .description('MCP Conformance Test Suite') + .version(packageJson.version); + +// Client command - tests a client implementation against scenarios +program + .command('client') + .description('Run conformance tests against a client implementation') + .requiredOption('--command ', 'Command to run the client') + .requiredOption('--scenario ', 'Scenario to test') + .option('--timeout ', 'Timeout in milliseconds', '30000') + .option('--verbose', 'Show verbose output') + .action(async (options) => { + try { + // Validate options with Zod + const validated = ClientOptionsSchema.parse(options); + + const result = await runConformanceTest( + validated.command, + validated.scenario, + validated.timeout + ); + + const { failed } = printClientResults( + result.checks, + validated.verbose ?? false + ); + process.exit(failed > 0 ? 1 : 0); + } catch (error) { + if (error instanceof ZodError) { + console.error('Validation error:'); + error.errors.forEach((err) => { + console.error(` ${err.path.join('.')}: ${err.message}`); + }); + console.error('\nAvailable client scenarios:'); + listScenarios().forEach((s) => console.error(` - ${s}`)); + process.exit(1); + } + console.error('Client test error:', error); + process.exit(1); + } + }); + +// Server command - tests a server implementation +program + .command('server') + .description('Run conformance tests against a server implementation') + .requiredOption('--url ', 'URL of the server to test') + .option( + '--scenario ', + 'Scenario to test (defaults to all scenarios if not specified)' + ) + .action(async (options) => { + try { + // Validate options with Zod + const validated = ServerOptionsSchema.parse(options); + + // If a single scenario is specified, run just that one + if (validated.scenario) { + const result = await runServerConformanceTest( + validated.url, + validated.scenario + ); + + const { failed } = printServerResults(result.checks); + process.exit(failed > 0 ? 1 : 0); + } else { + // Run all scenarios + const scenarios = listClientScenarios(); + console.log( + `Running ${scenarios.length} scenarios against ${validated.url}\n` + ); + + const allResults: { scenario: string; checks: ConformanceCheck[] }[] = + []; + + for (const scenarioName of scenarios) { + console.log(`\n=== Running scenario: ${scenarioName} ===`); + try { + const result = await runServerConformanceTest( + validated.url, + scenarioName + ); + allResults.push({ scenario: scenarioName, checks: result.checks }); + } catch (error) { + console.error(`Failed to run scenario ${scenarioName}:`, error); + allResults.push({ + scenario: scenarioName, + checks: [ + { + id: scenarioName, + name: scenarioName, + description: 'Failed to run scenario', + status: 'FAILURE', + timestamp: new Date().toISOString(), + errorMessage: + error instanceof Error ? error.message : String(error) + } + ] + }); + } + } + + const { totalFailed } = printServerSummary(allResults); + process.exit(totalFailed > 0 ? 1 : 0); + } + } catch (error) { + if (error instanceof ZodError) { + console.error('Validation error:'); + error.errors.forEach((err) => { + console.error(` ${err.path.join('.')}: ${err.message}`); + }); + console.error('\nAvailable server scenarios:'); + listClientScenarios().forEach((s) => console.error(` - ${s}`)); + process.exit(1); + } + console.error('Server test error:', error); + process.exit(1); + } + }); + +// List scenarios command +program + .command('list') + .description('List available test scenarios') + .option('--client', 'List client scenarios') + .option('--server', 'List server scenarios') + .action((options) => { + if (options.server || (!options.client && !options.server)) { + console.log('Server scenarios (test against a server):'); + const serverScenarios = listClientScenarios(); + serverScenarios.forEach((s) => console.log(` - ${s}`)); + } + + if (options.client || (!options.client && !options.server)) { + if (options.server || (!options.client && !options.server)) { + console.log(''); + } + console.log('Client scenarios (test against a client):'); + const clientScenarios = listScenarios(); + clientScenarios.forEach((s) => console.log(` - ${s}`)); + } + }); + +program.parse(); diff --git a/src/runner.ts b/src/runner.ts deleted file mode 100644 index ff1d929..0000000 --- a/src/runner.ts +++ /dev/null @@ -1,3 +0,0 @@ -import { runConformanceTest } from './runner/index.js'; - -export { runConformanceTest }; diff --git a/src/runner/DESIGN.md b/src/runner/DESIGN.md index e5bec10..fb714d5 100644 --- a/src/runner/DESIGN.md +++ b/src/runner/DESIGN.md @@ -2,20 +2,45 @@ ## Overview -The conformance test runner is a framework for testing MCP client implementations against the MCP specification. It executes client implementations in controlled scenarios and validates their behavior through a series of conformance checks. +The conformance test runner is a framework for testing both MCP client and server implementations against the MCP specification. It provides two testing modes: + +1. **Client Testing** - Executes client implementations in controlled scenarios and validates their behavior +2. **Server Testing** - Tests server implementations by acting as an MCP client and validating responses ## Architecture +### File Structure + +``` +src/runner/ +├── index.ts # Exports all public functions +├── client.ts # Client testing implementation +├── server.ts # Server testing implementation +└── utils.ts # Shared utilities (formatting, file I/O, ANSI colors) +``` + ### Components +**Client Testing:** + 1. **Runner** - Orchestrates test execution 2. **Test Server** - MCP server (and optionally auth server for auth scenarios) 3. **Client Process** - The MCP client implementation under test 4. **Scenario** - A specific test case with expected behaviors 5. **Checks** - Validation functions that produce ConformanceCheck results +**Server Testing:** + +1. **Runner** - Orchestrates test execution +2. **Test Client** - MCP SDK client that connects to the server under test +3. **Server Process** - The MCP server implementation under test (external) +4. **Scenario** - A specific test case with expected server behaviors +5. **Checks** - Validation functions that produce ConformanceCheck results + ### Execution Flow +**Client Testing:** + ``` 1. Runner starts test server(s) on available port(s) 2. Runner spawns client process with server URL as final argument @@ -25,44 +50,124 @@ The conformance test runner is a framework for testing MCP client implementation 6. Results are written to results/-/ ``` -## CLI Interface +**Server Testing:** -```bash -npm run start -- --command "tsx examples/clients/typescript/test1.ts --verbose" --scenario initialize +``` +1. User starts their server implementation +2. Runner connects to server URL as an MCP client +3. Runner sends requests and captures responses +4. Scenario-specific checks validate server behavior +5. Results are written to results/server--/ ``` -### Arguments +## CLI Interface + +The conformance suite provides a unified CLI with two main commands: + +### Client Testing + +```bash +# Run a single client scenario +npm run start -- client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize + +# With verbose output +npm run start -- client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize --verbose + +# With custom timeout +npm run start -- client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize --timeout 60000 +``` + +**Arguments:** - `--command` - The command to run the client (can include existing flags) -- `--scenario` - The scenario to test (e.g., "initialize", "list-tools", "call-tool") +- `--scenario` - The scenario to test (e.g., "initialize", "tools-call") +- `--timeout` - Timeout in milliseconds (default: 30000) +- `--verbose` - Show verbose output (JSON format) The runner will append the server URL as the final argument to the command. +### Server Testing + +```bash +# Run a single server scenario +npm run start -- server --url http://localhost:3000/mcp --scenario server-initialize + +# Run all server scenarios (default when no --scenario specified) +npm run start -- server --url http://localhost:3000/mcp +``` + +**Arguments:** + +- `--url` - URL of the server to test +- `--scenario ` - Scenario to test (optional, defaults to all scenarios if not specified) + +### List Available Scenarios + +```bash +# List all scenarios +npm run start -- list + +# List only client scenarios +npm run start -- list --client + +# List only server scenarios +npm run start -- list --server +``` + +## Validation + +All CLI arguments are validated using Zod schemas (`src/schemas.ts`) before being passed to runner functions: + +**Client Validation:** + +- Command is non-empty +- Scenario exists in available scenarios +- Timeout is a positive integer + +**Server Validation:** + +- URL is a valid HTTP/HTTPS URL +- All specified scenarios exist (if provided) +- Defaults to all scenarios when no `--scenario` is specified + ## Scenarios -A scenario represents a specific test case that validates one or more aspects of MCP client behavior. Each scenario: +A scenario represents a specific test case that validates one or more aspects of MCP behavior. Each scenario: -- Configures the test server with expected behavior +- Configures the test environment with expected behavior - May run multiple conformance checks -- Validates both client behavior and server responses +- Validates both request/response patterns and protocol compliance -Example scenarios: +### Client Scenario Examples - `initialize` - Tests client initialization handshake -- `list-tools` - Tests tool discovery -- `call-tool` - Tests tool invocation -- `auth-flow` - Tests OAuth/authorization flows (requires auth server) +- `tools-call` - Tests tool invocation +- `auth/basic-dcr` - Tests OAuth Dynamic Client Registration flow +- `auth/basic-metadata-var1` - Tests OAuth with authorization metadata variation 1 + +### Server Scenario Examples + +- `server-initialize` - Tests server initialization and capabilities +- `tools-list` - Tests tool listing endpoint +- `tools-call-simple-text` - Tests tool invocation with text response +- `resources-list` - Tests resource listing +- `prompts-list` - Tests prompt listing +- `logging-set-level` - Tests logging level configuration ## Output Structure -Results are written to: `results/-/` +Results are written to: `results/-/` or `results/server--/` -Files: +**Client Testing Files:** - `checks.json` - Array of ConformanceCheck objects with validation results - `stdout.txt` - Complete stdout from the client process - `stderr.txt` - Complete stderr from the client process +**Server Testing Files:** + +- `checks.json` - Array of ConformanceCheck objects with validation results + ### checks.json Format ```json @@ -84,6 +189,30 @@ Files: ] ``` +## Programmatic Usage + +The runner can also be used programmatically: + +```typescript +import { runConformanceTest, printClientResults, runServerConformanceTest, printServerResults } from './runner/index.js'; + +// Test a client +const clientResult = await runConformanceTest('tsx my-client.ts', 'initialize', 30000); +const { failed } = printClientResults(clientResult.checks); + +// Test a server +const serverResult = await runServerConformanceTest('http://localhost:3000/mcp', 'server-initialize'); +const { failed } = printServerResults(serverResult.checks); +``` + +## Shared Utilities + +The `utils.ts` module provides shared functionality: + +- **File Operations:** `ensureResultsDir()`, `createResultDir()` +- **Formatting:** `formatPrettyChecks()` for colored console output +- **Styling:** ANSI color constants and helpers + ## Future Enhancements - **Test Suites** - Group multiple scenarios for convenience @@ -91,3 +220,5 @@ Files: - **Parallel Execution** - Run multiple scenarios concurrently - **Custom Servers** - Allow custom MCP server implementations for advanced scenarios - **Report Formats** - HTML, Markdown, or other human-readable formats +- **Watch Mode** - Automatically re-run tests on file changes +- **Coverage Tracking** - Track which parts of the spec are tested diff --git a/src/runner/client.ts b/src/runner/client.ts new file mode 100644 index 0000000..8f3ff1a --- /dev/null +++ b/src/runner/client.ts @@ -0,0 +1,207 @@ +import { spawn } from 'child_process'; +import { promises as fs } from 'fs'; +import path from 'path'; +import { ConformanceCheck } from '../types'; +import { getScenario } from '../scenarios'; +import { ensureResultsDir, createResultDir, formatPrettyChecks } from './utils'; + +export interface ClientExecutionResult { + exitCode: number; + stdout: string; + stderr: string; + timedOut: boolean; +} + +async function executeClient( + command: string, + serverUrl: string, + timeout: number = 30000 +): Promise { + const commandParts = command.split(' '); + const executable = commandParts[0]; + const args = [...commandParts.slice(1), serverUrl]; + + let stdout = ''; + let stderr = ''; + let timedOut = false; + + return new Promise((resolve) => { + const process = spawn(executable, args, { + shell: true, + stdio: 'pipe' + }); + + const timeoutHandle = setTimeout(() => { + timedOut = true; + process.kill(); + }, timeout); + + if (process.stdout) { + process.stdout.on('data', (data) => { + stdout += data.toString(); + }); + } + + if (process.stderr) { + process.stderr.on('data', (data) => { + stderr += data.toString(); + }); + } + + process.on('close', (code) => { + clearTimeout(timeoutHandle); + resolve({ + exitCode: code || 0, + stdout, + stderr, + timedOut + }); + }); + + process.on('error', (error) => { + clearTimeout(timeoutHandle); + resolve({ + exitCode: -1, + stdout, + stderr: stderr + `\nProcess error: ${error.message}`, + timedOut + }); + }); + }); +} + +export async function runConformanceTest( + clientCommand: string, + scenarioName: string, + timeout: number = 30000 +): Promise<{ + checks: ConformanceCheck[]; + clientOutput: ClientExecutionResult; + resultDir: string; +}> { + await ensureResultsDir(); + const resultDir = createResultDir(scenarioName); + await fs.mkdir(resultDir, { recursive: true }); + + // Scenario is guaranteed to exist by CLI validation + const scenario = getScenario(scenarioName)!; + + console.log(`Starting scenario: ${scenarioName}`); + const urls = await scenario.start(); + + console.log(`Executing client: ${clientCommand} ${urls.serverUrl}`); + + try { + const clientOutput = await executeClient( + clientCommand, + urls.serverUrl, + timeout + ); + + // Print stdout/stderr if client exited with nonzero code + if (clientOutput.exitCode !== 0) { + console.error(`\nClient exited with code ${clientOutput.exitCode}`); + if (clientOutput.stdout) { + console.error(`\nStdout:\n${clientOutput.stdout}`); + } + if (clientOutput.stderr) { + console.error(`\nStderr:\n${clientOutput.stderr}`); + } + } + + if (clientOutput.timedOut) { + console.error(`\nClient timed out after ${timeout}ms`); + } + + const checks = scenario.getChecks(); + + await fs.writeFile( + path.join(resultDir, 'checks.json'), + JSON.stringify(checks, null, 2) + ); + + await fs.writeFile(path.join(resultDir, 'stdout.txt'), clientOutput.stdout); + + await fs.writeFile(path.join(resultDir, 'stderr.txt'), clientOutput.stderr); + + console.log(`Results saved to ${resultDir}`); + + return { + checks, + clientOutput, + resultDir + }; + } finally { + await scenario.stop(); + } +} + +export function printClientResults( + checks: ConformanceCheck[], + verbose: boolean = false +): { passed: number; failed: number; denominator: number } { + const denominator = checks.filter( + (c) => c.status === 'SUCCESS' || c.status === 'FAILURE' + ).length; + const passed = checks.filter((c) => c.status === 'SUCCESS').length; + const failed = checks.filter((c) => c.status === 'FAILURE').length; + + if (verbose) { + console.log(`Checks:\n${JSON.stringify(checks, null, 2)}`); + } else { + console.log(`Checks:\n${formatPrettyChecks(checks)}`); + } + + console.log(`\nTest Results:`); + console.log(`Passed: ${passed}/${denominator}, ${failed} failed`); + + if (failed > 0) { + console.log('\nFailed Checks:'); + checks + .filter((c) => c.status === 'FAILURE') + .forEach((c) => { + console.log(` - ${c.name}: ${c.description}`); + if (c.errorMessage) { + console.log(` Error: ${c.errorMessage}`); + } + }); + } + + return { passed, failed, denominator }; +} + +export async function runInteractiveMode(scenarioName: string): Promise { + await ensureResultsDir(); + const resultDir = createResultDir(scenarioName); + await fs.mkdir(resultDir, { recursive: true }); + + // Scenario is guaranteed to exist by CLI validation + const scenario = getScenario(scenarioName)!; + + console.log(`Starting scenario: ${scenarioName}`); + const urls = await scenario.start(); + + console.log(`Server URL: ${urls.serverUrl}`); + console.log('Press Ctrl+C to stop and save checks...'); + + const handleShutdown = async () => { + console.log('\nShutting down...'); + + const checks = scenario.getChecks(); + await fs.writeFile( + path.join(resultDir, 'checks.json'), + JSON.stringify(checks, null, 2) + ); + + console.log(`\nChecks:\n${JSON.stringify(checks, null, 2)}`); + console.log(`\nChecks saved to ${resultDir}/checks.json`); + + await scenario.stop(); + process.exit(0); + }; + + process.on('SIGINT', handleShutdown); + process.on('SIGTERM', handleShutdown); + + await new Promise(() => {}); +} diff --git a/src/runner/index.ts b/src/runner/index.ts index b042aeb..d48a291 100644 --- a/src/runner/index.ts +++ b/src/runner/index.ts @@ -1,312 +1,23 @@ -import { spawn } from 'child_process'; -import { promises as fs } from 'fs'; -import path from 'path'; -import { ConformanceCheck } from '../types.js'; -import { getScenario } from '../scenarios/index.js'; +// Export client functions +export { + runConformanceTest, + printClientResults, + runInteractiveMode, + type ClientExecutionResult +} from './client'; -// ANSI color codes -const COLORS = { - RESET: '\x1b[0m', - GRAY: '\x1b[90m', - GREEN: '\x1b[32m', - YELLOW: '\x1b[33m', - RED: '\x1b[31m', - BLUE: '\x1b[36m' -}; +// Export server functions +export { + runServerConformanceTest, + printServerResults, + printServerSummary +} from './server'; -function getStatusColor(status: string): string { - switch (status) { - case 'SUCCESS': - return COLORS.GREEN; - case 'FAILURE': - return COLORS.RED; - case 'INFO': - return COLORS.BLUE; - default: - return COLORS.RESET; - } -} - -function formatPrettyChecks(checks: ConformanceCheck[]): string { - // Find the longest id and status for column alignment - const maxIdLength = Math.max(...checks.map((c) => c.id.length)); - const maxStatusLength = Math.max(...checks.map((c) => c.status.length)); - - return checks - .map((check) => { - const timestamp = `${COLORS.GRAY}${check.timestamp}${COLORS.RESET}`; - const id = check.id.padEnd(maxIdLength); - const statusColor = getStatusColor(check.status); - const status = `${statusColor}${check.status.padEnd(maxStatusLength)}${COLORS.RESET}`; - const description = check.description; - const line = `${timestamp} [${id}] ${status} ${description}`; - // Add newline after outgoing responses for better visual separation - return ( - line + - (check.id.includes('outgoing') && check.id.includes('response') - ? '\n' - : '') - ); - }) - .join('\n'); -} - -export interface ClientExecutionResult { - exitCode: number; - stdout: string; - stderr: string; - timedOut: boolean; -} - -async function ensureResultsDir(): Promise { - const resultsDir = path.join(process.cwd(), 'results'); - await fs.mkdir(resultsDir, { recursive: true }); - return resultsDir; -} - -function createResultDir(scenario: string): string { - const timestamp = new Date().toISOString().replace(/[:.]/g, '-'); - return path.join('results', `${scenario}-${timestamp}`); -} - -async function executeClient( - command: string, - serverUrl: string, - timeout: number = 30000 -): Promise { - const commandParts = command.split(' '); - const executable = commandParts[0]; - const args = [...commandParts.slice(1), serverUrl]; - - let stdout = ''; - let stderr = ''; - let timedOut = false; - - return new Promise((resolve) => { - const process = spawn(executable, args, { - shell: true, - stdio: 'pipe' - }); - - const timeoutHandle = setTimeout(() => { - timedOut = true; - process.kill(); - }, timeout); - - if (process.stdout) { - process.stdout.on('data', (data) => { - stdout += data.toString(); - }); - } - - if (process.stderr) { - process.stderr.on('data', (data) => { - stderr += data.toString(); - }); - } - - process.on('close', (code) => { - clearTimeout(timeoutHandle); - resolve({ - exitCode: code || 0, - stdout, - stderr, - timedOut - }); - }); - - process.on('error', (error) => { - clearTimeout(timeoutHandle); - resolve({ - exitCode: -1, - stdout, - stderr: stderr + `\nProcess error: ${error.message}`, - timedOut - }); - }); - }); -} - -export async function runConformanceTest( - clientCommand: string, - scenarioName: string, - timeout: number = 30000 -): Promise<{ - checks: ConformanceCheck[]; - clientOutput: ClientExecutionResult; - resultDir: string; -}> { - await ensureResultsDir(); - const resultDir = createResultDir(scenarioName); - await fs.mkdir(resultDir, { recursive: true }); - - const scenario = getScenario(scenarioName); - if (!scenario) { - throw new Error(`Unknown scenario: ${scenarioName}`); - } - - console.log(`Starting scenario: ${scenarioName}`); - const urls = await scenario.start(); - - console.log(`Executing client: ${clientCommand} ${urls.serverUrl}`); - - try { - const clientOutput = await executeClient( - clientCommand, - urls.serverUrl, - timeout - ); - - // Print stdout/stderr if client exited with nonzero code - if (clientOutput.exitCode !== 0) { - console.error(`\nClient exited with code ${clientOutput.exitCode}`); - if (clientOutput.stdout) { - console.error(`\nStdout:\n${clientOutput.stdout}`); - } - if (clientOutput.stderr) { - console.error(`\nStderr:\n${clientOutput.stderr}`); - } - } - - if (clientOutput.timedOut) { - console.error(`\nClient timed out after ${timeout}ms`); - } - - const checks = scenario.getChecks(); - - await fs.writeFile( - path.join(resultDir, 'checks.json'), - JSON.stringify(checks, null, 2) - ); - - await fs.writeFile(path.join(resultDir, 'stdout.txt'), clientOutput.stdout); - - await fs.writeFile(path.join(resultDir, 'stderr.txt'), clientOutput.stderr); - - console.log(`Results saved to ${resultDir}`); - - return { - checks, - clientOutput, - resultDir - }; - } finally { - await scenario.stop(); - } -} - -async function runInteractiveMode(scenarioName: string): Promise { - await ensureResultsDir(); - const resultDir = createResultDir(scenarioName); - await fs.mkdir(resultDir, { recursive: true }); - - const scenario = getScenario(scenarioName); - if (!scenario) { - throw new Error(`Unknown scenario: ${scenarioName}`); - } - - console.log(`Starting scenario: ${scenarioName}`); - const urls = await scenario.start(); - - console.log(`Server URL: ${urls.serverUrl}`); - console.log('Press Ctrl+C to stop and save checks...'); - - const handleShutdown = async () => { - console.log('\nShutting down...'); - - const checks = scenario.getChecks(); - await fs.writeFile( - path.join(resultDir, 'checks.json'), - JSON.stringify(checks, null, 2) - ); - - console.log(`\nChecks:\n${JSON.stringify(checks, null, 2)}`); - console.log(`\nChecks saved to ${resultDir}/checks.json`); - - await scenario.stop(); - process.exit(0); - }; - - process.on('SIGINT', handleShutdown); - process.on('SIGTERM', handleShutdown); - - await new Promise(() => {}); -} - -async function main(): Promise { - const args = process.argv.slice(2); - let command: string | null = null; - let scenario: string | null = null; - let verbose = false; - - for (let i = 0; i < args.length; i++) { - if (args[i] === '--command' && i + 1 < args.length) { - command = args[i + 1]; - i++; - } else if (args[i] === '--scenario' && i + 1 < args.length) { - scenario = args[i + 1]; - i++; - } else if (args[i] === '--verbose') { - verbose = true; - } - } - - if (!scenario) { - console.error( - 'Usage: runner --scenario [--command ""] [--verbose]' - ); - console.error( - 'Example: runner --scenario initialize --command "tsx examples/clients/typescript/test1.ts"' - ); - console.error('Or run without --command for interactive mode'); - process.exit(1); - } - - if (!command) { - try { - await runInteractiveMode(scenario); - } catch (error) { - console.error('Interactive mode error:', error); - process.exit(1); - } - return; - } - - try { - const result = await runConformanceTest(command, scenario); - - const denominator = result.checks.filter( - (c) => c.status === 'SUCCESS' || c.status == 'FAILURE' - ).length; - const passed = result.checks.filter((c) => c.status === 'SUCCESS').length; - const failed = result.checks.filter((c) => c.status === 'FAILURE').length; - - if (verbose) { - console.log(`Checks:\n${JSON.stringify(result.checks, null, 2)}`); - } else { - console.log(`Checks:\n${formatPrettyChecks(result.checks)}`); - } - - console.log(`\nTest Results:`); - console.log(`Passed: ${passed}/${denominator}, ${failed} failed`); - - if (failed > 0) { - console.log('\nFailed Checks:'); - result.checks - .filter((c) => c.status === 'FAILURE') - .forEach((c) => { - console.log(` - ${c.name}: ${c.description}`); - if (c.errorMessage) { - console.log(` Error: ${c.errorMessage}`); - } - }); - } - - process.exit(failed > 0 ? 1 : 0); - } catch (error) { - console.error('Test runner error:', error); - process.exit(1); - } -} - -main(); +// Export utilities +export { + ensureResultsDir, + createResultDir, + formatPrettyChecks, + getStatusColor, + COLORS +} from './utils'; diff --git a/src/runner/server.ts b/src/runner/server.ts new file mode 100644 index 0000000..e153d54 --- /dev/null +++ b/src/runner/server.ts @@ -0,0 +1,93 @@ +import { promises as fs } from 'fs'; +import path from 'path'; +import { ConformanceCheck } from '../types'; +import { getClientScenario } from '../scenarios'; +import { ensureResultsDir, createResultDir } from './utils'; + +export async function runServerConformanceTest( + serverUrl: string, + scenarioName: string +): Promise<{ + checks: ConformanceCheck[]; + resultDir: string; +}> { + await ensureResultsDir(); + const resultDir = createResultDir(scenarioName, 'server'); + await fs.mkdir(resultDir, { recursive: true }); + + // Scenario is guaranteed to exist by CLI validation + const scenario = getClientScenario(scenarioName)!; + + console.log( + `Running client scenario '${scenarioName}' against server: ${serverUrl}` + ); + + const checks = await scenario.run(serverUrl); + + await fs.writeFile( + path.join(resultDir, 'checks.json'), + JSON.stringify(checks, null, 2) + ); + + console.log(`Results saved to ${resultDir}`); + + return { + checks, + resultDir + }; +} + +export function printServerResults(checks: ConformanceCheck[]): { + passed: number; + failed: number; + denominator: number; +} { + const denominator = checks.filter( + (c) => c.status === 'SUCCESS' || c.status === 'FAILURE' + ).length; + const passed = checks.filter((c) => c.status === 'SUCCESS').length; + const failed = checks.filter((c) => c.status === 'FAILURE').length; + + console.log(`Checks:\n${JSON.stringify(checks, null, 2)}`); + + console.log(`\nTest Results:`); + console.log(`Passed: ${passed}/${denominator}, ${failed} failed`); + + if (failed > 0) { + console.log('\nFailed Checks:'); + checks + .filter((c) => c.status === 'FAILURE') + .forEach((c) => { + console.log(` - ${c.name}: ${c.description}`); + if (c.errorMessage) { + console.log(` Error: ${c.errorMessage}`); + } + }); + } + + return { passed, failed, denominator }; +} + +export function printServerSummary( + allResults: { scenario: string; checks: ConformanceCheck[] }[] +): { totalPassed: number; totalFailed: number } { + console.log('\n\n=== SUMMARY ==='); + let totalPassed = 0; + let totalFailed = 0; + + for (const result of allResults) { + const passed = result.checks.filter((c) => c.status === 'SUCCESS').length; + const failed = result.checks.filter((c) => c.status === 'FAILURE').length; + totalPassed += passed; + totalFailed += failed; + + const status = failed === 0 ? '✓' : '✗'; + console.log( + `${status} ${result.scenario}: ${passed} passed, ${failed} failed` + ); + } + + console.log(`\nTotal: ${totalPassed} passed, ${totalFailed} failed`); + + return { totalPassed, totalFailed }; +} diff --git a/src/runner/utils.ts b/src/runner/utils.ts new file mode 100644 index 0000000..3b2b0ae --- /dev/null +++ b/src/runner/utils.ts @@ -0,0 +1,62 @@ +import { promises as fs } from 'fs'; +import path from 'path'; +import { ConformanceCheck } from '../types'; + +// ANSI color codes +export const COLORS = { + RESET: '\x1b[0m', + GRAY: '\x1b[90m', + GREEN: '\x1b[32m', + YELLOW: '\x1b[33m', + RED: '\x1b[31m', + BLUE: '\x1b[36m' +}; + +export function getStatusColor(status: string): string { + switch (status) { + case 'SUCCESS': + return COLORS.GREEN; + case 'FAILURE': + return COLORS.RED; + case 'INFO': + return COLORS.BLUE; + default: + return COLORS.RESET; + } +} + +export function formatPrettyChecks(checks: ConformanceCheck[]): string { + // Find the longest id and status for column alignment + const maxIdLength = Math.max(...checks.map((c) => c.id.length)); + const maxStatusLength = Math.max(...checks.map((c) => c.status.length)); + + return checks + .map((check) => { + const timestamp = `${COLORS.GRAY}${check.timestamp}${COLORS.RESET}`; + const id = check.id.padEnd(maxIdLength); + const statusColor = getStatusColor(check.status); + const status = `${statusColor}${check.status.padEnd(maxStatusLength)}${COLORS.RESET}`; + const description = check.description; + const line = `${timestamp} [${id}] ${status} ${description}`; + // Add newline after outgoing responses for better visual separation + return ( + line + + (check.id.includes('outgoing') && check.id.includes('response') + ? '\n' + : '') + ); + }) + .join('\n'); +} + +export async function ensureResultsDir(): Promise { + const resultsDir = path.join(process.cwd(), 'results'); + await fs.mkdir(resultsDir, { recursive: true }); + return resultsDir; +} + +export function createResultDir(scenario: string, prefix = ''): string { + const timestamp = new Date().toISOString().replace(/[:.]/g, '-'); + const dirName = prefix ? `${prefix}-${scenario}` : scenario; + return path.join('results', `${dirName}-${timestamp}`); +} diff --git a/src/schemas.ts b/src/schemas.ts new file mode 100644 index 0000000..5f22fc6 --- /dev/null +++ b/src/schemas.ts @@ -0,0 +1,44 @@ +import { z } from 'zod'; +import { getScenario, getClientScenario } from './scenarios'; + +// Client command options schema +export const ClientOptionsSchema = z.object({ + command: z.string().min(1, 'Command cannot be empty'), + scenario: z + .string() + .min(1, 'Scenario cannot be empty') + .refine( + (scenario) => getScenario(scenario) !== undefined, + (scenario) => ({ + message: `Unknown scenario '${scenario}'` + }) + ), + timeout: z + .string() + .transform((val) => parseInt(val, 10)) + .pipe( + z + .number() + .positive('Timeout must be a positive number') + .int('Timeout must be an integer') + ), + verbose: z.boolean().optional() +}); + +export type ClientOptions = z.infer; + +// Server command options schema +export const ServerOptionsSchema = z.object({ + url: z.string().url('Invalid server URL'), + scenario: z + .string() + .refine( + (scenario) => getClientScenario(scenario) !== undefined, + (scenario) => ({ + message: `Unknown scenario '${scenario}'` + }) + ) + .optional() +}); + +export type ServerOptions = z.infer; diff --git a/src/server-runner.ts b/src/server-runner.ts deleted file mode 100644 index bac3202..0000000 --- a/src/server-runner.ts +++ /dev/null @@ -1,186 +0,0 @@ -import { promises as fs } from 'fs'; -import path from 'path'; -import { ConformanceCheck } from './types.js'; -import { getClientScenario } from './scenarios/index.js'; - -async function ensureResultsDir(): Promise { - const resultsDir = path.join(process.cwd(), 'results'); - await fs.mkdir(resultsDir, { recursive: true }); - return resultsDir; -} - -function createResultDir(scenario: string): string { - const timestamp = new Date().toISOString().replace(/[:.]/g, '-'); - return path.join('results', `server-${scenario}-${timestamp}`); -} - -export async function runServerConformanceTest( - serverUrl: string, - scenarioName: string -): Promise<{ - checks: ConformanceCheck[]; - resultDir: string; -}> { - await ensureResultsDir(); - const resultDir = createResultDir(scenarioName); - await fs.mkdir(resultDir, { recursive: true }); - - const scenario = getClientScenario(scenarioName); - if (!scenario) { - throw new Error(`Unknown client scenario: ${scenarioName}`); - } - - console.log( - `Running client scenario '${scenarioName}' against server: ${serverUrl}` - ); - - const checks = await scenario.run(serverUrl); - - await fs.writeFile( - path.join(resultDir, 'checks.json'), - JSON.stringify(checks, null, 2) - ); - - console.log(`Results saved to ${resultDir}`); - - return { - checks, - resultDir - }; -} - -async function main(): Promise { - const args = process.argv.slice(2); - let serverUrl: string | null = null; - let scenario: string | null = null; - let runAll = false; - - for (let i = 0; i < args.length; i++) { - if (args[i] === '--server-url' && i + 1 < args.length) { - serverUrl = args[i + 1]; - i++; - } else if (args[i] === '--scenario' && i + 1 < args.length) { - scenario = args[i + 1]; - i++; - } else if (args[i] === '--all') { - runAll = true; - } - } - - if (!serverUrl) { - console.error( - 'Usage: server-runner --server-url [--scenario | --all]' - ); - console.error( - 'Example: server-runner --server-url http://localhost:3000/mcp --scenario server-initialize' - ); - console.error( - 'Example: server-runner --server-url http://localhost:3000/mcp --all' - ); - process.exit(1); - } - - if (!scenario && !runAll) { - console.error('Must specify either --scenario or --all'); - process.exit(1); - } - - try { - const allResults: { scenario: string; checks: ConformanceCheck[] }[] = []; - - if (runAll) { - // Get all server scenarios - const { listClientScenarios } = await import('./scenarios/index.js'); - const scenarios = listClientScenarios(); - - console.log( - `Running ${scenarios.length} scenarios against ${serverUrl}\n` - ); - - for (const scenarioName of scenarios) { - console.log(`\n=== Running scenario: ${scenarioName} ===`); - try { - const result = await runServerConformanceTest( - serverUrl, - scenarioName - ); - allResults.push({ scenario: scenarioName, checks: result.checks }); - } catch (error) { - console.error(`Failed to run scenario ${scenarioName}:`, error); - allResults.push({ - scenario: scenarioName, - checks: [ - { - id: scenarioName, - name: scenarioName, - description: `Failed to run scenario`, - status: 'FAILURE', - timestamp: new Date().toISOString(), - errorMessage: - error instanceof Error ? error.message : String(error) - } - ] - }); - } - } - - // Summary - console.log('\n\n=== SUMMARY ==='); - let totalPassed = 0; - let totalFailed = 0; - - for (const result of allResults) { - const passed = result.checks.filter( - (c) => c.status === 'SUCCESS' - ).length; - const failed = result.checks.filter( - (c) => c.status === 'FAILURE' - ).length; - totalPassed += passed; - totalFailed += failed; - - const status = failed === 0 ? '✓' : '✗'; - console.log( - `${status} ${result.scenario}: ${passed} passed, ${failed} failed` - ); - } - - console.log(`\nTotal: ${totalPassed} passed, ${totalFailed} failed`); - process.exit(totalFailed > 0 ? 1 : 0); - } else if (scenario) { - const result = await runServerConformanceTest(serverUrl, scenario); - - const denominator = result.checks.filter( - (c) => c.status === 'SUCCESS' || c.status == 'FAILURE' - ).length; - const passed = result.checks.filter((c) => c.status === 'SUCCESS').length; - const failed = result.checks.filter((c) => c.status === 'FAILURE').length; - - console.log(`Checks:\n${JSON.stringify(result.checks, null, 2)}`); - - console.log(`\nTest Results:`); - console.log(`Passed: ${passed}/${denominator}, ${failed} failed`); - - if (failed > 0) { - console.log('\nFailed Checks:'); - result.checks - .filter((c) => c.status === 'FAILURE') - .forEach((c) => { - console.log(` - ${c.name}: ${c.description}`); - if (c.errorMessage) { - console.log(` Error: ${c.errorMessage}`); - } - }); - } - - process.exit(failed > 0 ? 1 : 0); - } - } catch (error) { - console.error('Server test runner error:', error); - process.exit(1); - } -} - -if (import.meta.url === `file://${process.argv[1]}`) { - main(); -} diff --git a/tsconfig.json b/tsconfig.json index 640d43d..a31599e 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -3,6 +3,7 @@ "target": "ES2020", "module": "ES2020", "moduleResolution": "node", + "resolveJsonModule": true, "strict": true, "esModuleInterop": true, "skipLibCheck": true,