4ecd95218a## Correct SEP-1330 enum schemas in test_elicitation_sep1330_enums **NB! This PR contains changes from #599 and should be rebased and merged after #599 is merged.** Update conformance test according to [test requirements](https://github.com/modelcontextprotocol/conformance/blob/main/src/scenarios/server/elicitation-enums.ts#L14C1-L35C9). - Fix legacyEnum: was using `oneOf` with const/title pairs; now correctly uses `enum` + `enumNames` arrays per LegacyEnumSchema spec - Fix titledMulti: items were using `oneOf` with extra `type:"string"`; now correctly uses `anyOf` per TitledMultiSelectEnumSchema spec - Fix return text format to match expected "Elicitation completed: action=..., content=..." - Remove _elicitation-sep1330-enums_ from conformance baseline (test now passes) ## How Has This Been Tested? ```shell ./conformance-test/run-conformance.sh server ``` ## Breaking Changes No ## Types of changes - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update ## Checklist <!-- Go over all the following points, and put an `x` in all the boxes that apply. --> - [x] I have read the [MCP Documentation](https://modelcontextprotocol.io) - [x] My code follows the repository's style guidelines - [x] New and existing tests pass locally - [ ] I have added appropriate error handling - [ ] I have added or updated documentation as needed --------- Co-authored-by: devcrocod <devcrocod@gmail.com>
118 lines
6.5 KiB
Markdown
118 lines
6.5 KiB
Markdown
# MCP Conformance Tests
|
|
|
|
Conformance tests for the Kotlin MCP SDK. Uses the external
|
|
[`@modelcontextprotocol/conformance`](https://www.npmjs.com/package/@modelcontextprotocol/conformance)
|
|
runner (pinned to **0.1.15**) to validate compliance with the MCP specification.
|
|
|
|
## Prerequisites
|
|
|
|
- **JDK 17+**
|
|
- **Node.js 18+** and `npx` (for the conformance runner)
|
|
- **curl** (used to poll server readiness)
|
|
|
|
## Quick Start
|
|
|
|
Run **all** suites (server, client core, client auth) from the project root:
|
|
|
|
```bash
|
|
./conformance-test/run-conformance.sh all
|
|
```
|
|
|
|
## Commands
|
|
|
|
```
|
|
./conformance-test/run-conformance.sh <command> [extra-args...]
|
|
```
|
|
|
|
| Command | What it does |
|
|
|---------------|--------------------------------------------------------------------------------------|
|
|
| `list` | [List scenarios available in MCP Conformance Test Framework][list-scenarios-command] |
|
|
| `server` | Starts the Ktor conformance server, runs the server test suite against it |
|
|
| `client` | Runs the client test suite (`initialize`, `tools_call`, `elicitation`, `sse-retry`) |
|
|
| `client-auth` | Runs the client auth test suite (20 OAuth scenarios) |
|
|
| `all` | Runs all three suites sequentially |
|
|
|
|
Any `[extra-args]` are forwarded to the conformance runner (e.g. `--verbose`).
|
|
|
|
## What the Script Does
|
|
|
|
1. **Builds** the module via `./gradlew :conformance-test:installDist`
|
|
2. For `server` — starts the conformance server on `localhost:3001`, polls until ready
|
|
3. Invokes `npx @modelcontextprotocol/conformance@0.1.15` with the appropriate arguments
|
|
4. Saves results to `conformance-test/results/<command>/`
|
|
5. Cleans up the server process on exit
|
|
6. Exits non-zero if any suite fails
|
|
|
|
## Environment Variables
|
|
|
|
| Variable | Default | Description |
|
|
|------------|---------|---------------------------------|
|
|
| `MCP_PORT` | `3001` | Port for the conformance server |
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
conformance-test/
|
|
├── run-conformance.sh # Single entry point script
|
|
├── conformance-baseline.yml # Expected failures for known SDK limitations
|
|
└── src/main/kotlin/.../conformance/
|
|
├── ConformanceServer.kt # Ktor server entry point (StreamableHTTP, DNS rebinding, EventStore)
|
|
├── ConformanceClient.kt # Scenario-based client entry point (MCP_CONFORMANCE_SCENARIO routing)
|
|
├── ConformanceTools.kt # 18 tool registrations
|
|
├── ConformanceResources.kt # 5 resource registrations (static, binary, template, watched, dynamic)
|
|
├── ConformancePrompts.kt # 5 prompt registrations (simple, args, image, embedded, dynamic)
|
|
├── ConformanceCompletions.kt # completion/complete handler
|
|
├── InMemoryEventStore.kt # EventStore impl for SSE resumability (SEP-1699)
|
|
└── auth/ # OAuth client for 20 auth scenarios
|
|
├── registration.kt # Scenario handler registration
|
|
├── utils.kt # Shared utilities: JSON instance, constants, extractOrigin()
|
|
├── discovery.kt # Protected Resource Metadata + AS Metadata discovery
|
|
├── pkce.kt # PKCE code verifier/challenge generation + AS capability check
|
|
├── tokenExchange.kt # Token endpoint interaction (exchange code, error handling)
|
|
├── authCodeFlow.kt # Main Authorization Code flow handler (runAuthClient + interceptor)
|
|
├── scopeHandling.kt # Scope selection strategy + step-up 403 handling
|
|
├── clientRegistration.kt # Client registration logic (pre-reg, CIMD, dynamic)
|
|
├── JWTScenario.kt # Client Credentials JWT scenario
|
|
├── basicScenario.kt # Client Credentials Basic scenario
|
|
└── crossAppAccessScenario.kt # Cross-App Access (SEP-990) scenario
|
|
```
|
|
|
|
## Test Suites
|
|
|
|
### Server Suite
|
|
|
|
Tests the conformance server against all server scenarios:
|
|
|
|
| Category | Scenarios |
|
|
|-------------|-------------------------------------------------------------------------------------------------------------------------------------|
|
|
| Lifecycle | initialize, ping |
|
|
| Tools | text, image, audio, embedded, multiple, progress, logging, error, sampling, elicitation, dynamic, reconnection, JSON Schema 2020-12 |
|
|
| Resources | list, read-text, read-binary, templates, subscribe, dynamic |
|
|
| Prompts | simple, with-args, with-image, with-embedded-resource, dynamic |
|
|
| Completions | complete |
|
|
| Security | DNS rebinding protection |
|
|
|
|
### Client Core Suite
|
|
|
|
| Scenario | Description |
|
|
|---------------------------------------|-----------------------------------------------|
|
|
| `initialize` | Connect, list tools, close |
|
|
| `tools_call` | Connect, call `add_numbers(a=5, b=3)`, close |
|
|
| `elicitation-sep1034-client-defaults` | Elicitation with `applyDefaults` capability |
|
|
| `sse-retry` | Call `test_reconnection`, verify reconnection |
|
|
|
|
### Client Auth Suite
|
|
|
|
17 OAuth Authorization Code scenarios + 2 Client Credentials scenarios (`jwt`, `basic`) + 1 Cross-App Access scenario = 20 total.
|
|
|
|
> [!NOTE]
|
|
> Auth scenarios are implemented using Ktor's `HttpClient` plugins (`HttpSend` interceptor,
|
|
> `ktor-client-auth`) as a standalone OAuth client. They do not use the SDK's built-in auth support.
|
|
|
|
## Known SDK Limitations
|
|
|
|
Scenarios, expected to fail due to current SDK limitations, are tracked in
|
|
[`conformance-baseline.yml`](conformance-baseline.yml).
|
|
These failures reveal SDK gaps and are intentionally not fixed in this module.
|
|
|
|
[list-scenarios-command]: https://github.com/modelcontextprotocol/conformance/tree/main?tab=readme-ov-file#list-available-scenarios
|