modelcontextprotocol/conformance
Watch
1
0
Fork
You've already forked conformance
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
Mirror of https://github.com/modelcontextprotocol/conformance — Conformance Tests for MCP
28 commits 1 branch 19 tags 2 MiB
  • TypeScript 98.1%
  • JavaScript 1.9%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Felix Weinberger 0e8f13ba1e
 feat: add SEP-1034 conformance test for elicitation defaults (#14) 
Add comprehensive conformance test for SEP-1034 which extends MCP
elicitation to support default values across all primitive types.

Changes:
- Add test_elicitation_sep1034_defaults tool to reference server
- Create ElicitationDefaultsScenario with 5 granular checks:
  * String schema default value
  * Integer schema default value
  * Number schema default value
  * Enum schema default value (validates it's a valid enum member)
  * Boolean schema default value (regression test)
- Register scenario in index.ts

The test validates that servers properly include default values in
elicitation requestedSchema per SEP-1034 specification.

Test results: All 5 checks pass against TypeScript reference server
2025-11-10 16:04:09 +00:00
.github/workflows Add test that ensures example server passes conformance (#9) 2025-11-04 15:43:52 +00:00
examples feat: add SEP-1034 conformance test for elicitation defaults (#14) 2025-11-10 16:04:09 +00:00
src feat: add SEP-1034 conformance test for elicitation defaults (#14) 2025-11-10 16:04:09 +00:00
.gitignore Add lefthook for linting and testing hooks, and optional pre-commit hook via local (#7) 2025-11-04 14:50:17 +00:00
.prettierrc.json adjust prettier to use more defaults (#8) 2025-11-04 15:34:27 +00:00
eslint.config.mjs adjust prettier to use more defaults (#8) 2025-11-04 15:34:27 +00:00
jest.config.js adjust prettier to use more defaults (#8) 2025-11-04 15:34:27 +00:00
lefthook-local.example.yml adjust prettier to use more defaults (#8) 2025-11-04 15:34:27 +00:00
lefthook.yml adjust prettier to use more defaults (#8) 2025-11-04 15:34:27 +00:00
package-lock.json [scenario] OAuth basic & PRM variation (#11) 2025-11-10 11:51:30 +00:00
package.json Add test that ensures example server passes conformance (#9) 2025-11-04 15:43:52 +00:00
README.md adjust prettier to use more defaults (#8) 2025-11-04 15:34:27 +00:00
SERVER_REQUIREMENTS.md docs: update server requirements (#10) 2025-11-09 14:35:41 +00:00
tsconfig.json adjust prettier to use more defaults (#8) 2025-11-04 15:34:27 +00:00

README.md

MCP Conformance Test Framework

A framework for testing MCP (Model Context Protocol) client implementations against the specification.

[!WARNING] This repository is a work in progress and is unstable. Join the conversation in the #conformance-testing-wg in the MCP Contributors discord.

Quick Start

Testing Clients

npm install
npm run start -- --command "tsx examples/clients/typescript/test1.ts" --scenario initialize

Testing Servers

npm run test:server -- --server-url http://localhost:3000/mcp --all

Overview

The conformance test framework validates MCP client implementations by:

  1. Starting a test server for the specified scenario
  2. Running the client implementation with the test server URL
  3. Capturing MCP protocol interactions
  4. Running conformance checks against the specification
  5. Generating detailed test results

Usage

npm run start -- --command "<client-command>" --scenario <scenario-name>
  • --command - The command to run your MCP client (can include flags)
  • --scenario - The test scenario to run (e.g., "initialize")

The framework appends the server URL as the final argument to your command.

Test Results

Results are saved to results/<scenario>-<timestamp>/:

  • checks.json - Array of conformance check results with pass/fail status
  • stdout.txt - Client stdout output
  • stderr.txt - Client stderr output

Example Clients

  • examples/clients/typescript/test1.ts - Valid MCP client (passes all checks)
  • examples/clients/typescript/test-broken.ts - Invalid client missing required fields (fails checks)

Available Scenarios

  • initialize - Tests MCP client initialization handshake
    • Validates protocol version
    • Validates clientInfo (name and version)
    • Validates server response handling

Architecture

See src/runner/DESIGN.md for detailed architecture documentation.

Key Components

  • Runner (src/runner/) - Orchestrates test execution and result generation
  • Scenarios (src/scenarios/) - Test scenarios with custom server implementations
  • Checks (src/checks.ts) - Conformance validation functions
  • Types (src/types.ts) - Shared type definitions

Adding New Scenarios

  1. Create a new directory in src/scenarios/<scenario-name>/
  2. Implement the Scenario interface with start(), stop(), and getChecks()
  3. Register the scenario in src/scenarios/index.ts

See src/scenarios/initialize/ for a reference implementation.