Add MCP conformance tests with client and server test suites (#767)
Implement conformance testing modules with comprehensive client/server test scenarios, sync conversion for client/servlet code, CI integration, and documentation updates. --------- Signed-off-by: Dariusz Jędrzejczyk <dariusz.jedrzejczyk@broadcom.com>
This commit is contained in:
parent
0a8cb1e038
commit
4fe8f3efef
1 changed files with 1558 additions and 0 deletions
82
conformance-tests/VALIDATION_RESULTS.md
Normal file
82
conformance-tests/VALIDATION_RESULTS.md
Normal file
|
|
@ -0,0 +1,82 @@
|
|||
# MCP Java SDK Conformance Test Validation Results
|
||||
|
||||
## Summary
|
||||
|
||||
**Server Tests:** 37/40 passed (92.5%)
|
||||
**Client Tests:** 3/4 scenarios passed (9/10 checks passed)
|
||||
|
||||
## Server Test Results
|
||||
|
||||
### Passing (37/40)
|
||||
|
||||
- **Lifecycle & Utilities (4/4):** initialize, ping, logging-set-level, completion-complete
|
||||
- **Tools (11/11):** All scenarios including progress notifications ✨
|
||||
- **Elicitation (10/10):** SEP-1034 defaults (5 checks), SEP-1330 enums (5 checks)
|
||||
- **Resources (4/6):** list, read-text, read-binary, templates-read
|
||||
- **Prompts (4/4):** list, simple, with-args, embedded-resource, with-image
|
||||
- **SSE Transport (2/2):** Multiple streams
|
||||
- **Security (1/2):** Localhost validation passes
|
||||
|
||||
### Failing (3/40)
|
||||
|
||||
1. **resources-subscribe** - Not implemented in SDK
|
||||
2. **resources-unsubscribe** - Not implemented in SDK
|
||||
3. **dns-rebinding-protection** - Missing Host/Origin validation (1/2 checks)
|
||||
|
||||
## Client Test Results
|
||||
|
||||
### Passing (3/4 scenarios, 9/10 checks)
|
||||
|
||||
- **initialize (1/1):** Protocol negotiation, clientInfo, capabilities
|
||||
- **tools_call (1/1):** Tool discovery and invocation
|
||||
- **elicitation-sep1034-client-defaults (5/5):** Default values for string, integer, number, enum, boolean
|
||||
|
||||
### Partially Passing (1/4 scenarios, 1/2 checks)
|
||||
|
||||
- **sse-retry (1/2 + 1 warning):**
|
||||
- ✅ Reconnects after stream closure
|
||||
- ❌ Does not respect retry timing
|
||||
- ⚠️ Does not send Last-Event-ID header (SHOULD requirement)
|
||||
|
||||
**Issue:** Client treats `retry:` SSE field as invalid instead of parsing it for reconnection timing.
|
||||
|
||||
## Known Limitations
|
||||
|
||||
1. **Resource Subscriptions:** SDK doesn't implement `resources/subscribe` and `resources/unsubscribe` handlers
|
||||
2. **Client SSE Retry:** Client doesn't parse or respect the `retry:` field, reconnects immediately, and doesn't send Last-Event-ID header
|
||||
3. **DNS Rebinding Protection:** Missing Host/Origin header validation in server transport
|
||||
|
||||
## Running Tests
|
||||
|
||||
### Server
|
||||
```bash
|
||||
# Start server
|
||||
cd conformance-tests/server-servlet
|
||||
../../mvnw compile exec:java -Dexec.mainClass="io.modelcontextprotocol.conformance.server.ConformanceServlet"
|
||||
|
||||
# Run tests (in another terminal)
|
||||
npx @modelcontextprotocol/conformance server --url http://localhost:8080/mcp --suite active
|
||||
```
|
||||
|
||||
### Client
|
||||
```bash
|
||||
# Build
|
||||
cd conformance-tests/client-jdk-http-client
|
||||
../../mvnw clean package -DskipTests
|
||||
|
||||
# Run all scenarios
|
||||
for scenario in initialize tools_call elicitation-sep1034-client-defaults sse-retry; do
|
||||
npx @modelcontextprotocol/conformance client \
|
||||
--command "java -jar target/client-jdk-http-client-0.18.0-SNAPSHOT.jar" \
|
||||
--scenario $scenario
|
||||
done
|
||||
```
|
||||
|
||||
## Recommendations
|
||||
|
||||
### High Priority
|
||||
1. Fix client SSE retry field handling in `HttpClientStreamableHttpTransport`
|
||||
2. Implement resource subscription handlers in `McpStatelessAsyncServer`
|
||||
|
||||
### Medium Priority
|
||||
3. Add Host/Origin validation in `HttpServletStreamableServerTransportProvider` for DNS rebinding protection
|
||||
Loading…
Reference in a new issue