2fd159a79a
- Add CHANGELOG.md with release lines and history - Refresh ROADMAP.md: 2.x stable line, 3.0.x for 2026-07-28 spec (first milestones Sep 2026), Tier 2 commitment, 1.x/0.x security-patches-only - Refresh VALIDATION_RESULTS.md: 2026-08-17 run on conformance suite 0.2.0-alpha.11 (server 73/73, auth 193/193, client 3/4 w/ sse-retry baselined) - Bump conformance CI action pins v0.1.11 -> v0.1.16 - Update docs with missing features Signed-off-by: Christian Tzolov <christian.tzolov@broadcom.com>
113 lines
4.5 KiB
Markdown
113 lines
4.5 KiB
Markdown
# MCP Java SDK Conformance Test Validation Results
|
|
|
|
Last validated: **2026-08-17** against conformance suite
|
|
**`@modelcontextprotocol/conformance@0.2.0-alpha.11`** (SDK at `main`, 2.0.1-SNAPSHOT), targetting
|
|
version 2025-11-25 (`--spec-version 2025-11-25`).
|
|
|
|
## Summary
|
|
|
|
**Server Tests (active suite):** 73/73 checks passed (31 scenarios, 100%)
|
|
**Server Tests (SEP-1613 `json-schema-2020-12`):** 5/5 checks passed (SEP-2106 checks skipped — post-2025-11-25 spec additions)
|
|
**Client Tests:** 3/4 scenarios passed; `sse-retry` fails (tracked in `conformance-baseline.yml`)
|
|
**Auth Tests:** 14/14 scenarios passing (193 checks, 0 failed, 0 warnings)
|
|
|
|
Baseline check passed on every run: all failures are expected per
|
|
[`conformance-baseline.yml`](conformance-baseline.yml).
|
|
|
|
## Server Test Results
|
|
|
|
### Active Suite — Passing (31/31 scenarios, 73/73 checks)
|
|
|
|
- **Lifecycle & Utilities:** initialize, ping, logging-set-level, completion-complete
|
|
- **Tools (13/13):** all scenarios including progress notifications, sampling, elicitation
|
|
- **Elicitation:** SEP-1034 defaults (6 checks), SEP-1330 enums (6 checks)
|
|
- **Resources:** list, read-text, read-binary, templates-read, subscribe, unsubscribe
|
|
- **Prompts:** list, simple, with-args, embedded-resource, with-image
|
|
- **SSE Transport:** multiple streams
|
|
- **Security:** DNS rebinding protection
|
|
|
|
### SEP-1613 — JSON Schema 2020-12 (5/5 checks)
|
|
|
|
- `json_schema_2020_12_tool` found; `$schema`, `$defs`, and `additionalProperties`
|
|
fields preserved; every JSON-RPC message valid per the spec JSON schema for the
|
|
negotiated spec version (`wire-schema-valid`)
|
|
- SEP-2106 checks (composition/conditional/anchor keywords) reported SKIPPED:
|
|
they postdate the 2025-11-25 spec release and are excluded from scoring
|
|
|
|
## Client Test Results
|
|
|
|
### Passing (3/4 scenarios)
|
|
|
|
- **initialize (1/1):** protocol negotiation, clientInfo, capabilities
|
|
- **tools_call (2/2):** tool discovery and invocation
|
|
- **elicitation-sep1034-client-defaults (5/5):** default values for string, integer, number, enum, boolean
|
|
|
|
### Failing — in baseline (1/4 scenarios)
|
|
|
|
- **sse-retry:** client does not parse/respect the `retry:` SSE field timing and
|
|
does not send the `Last-Event-ID` header (SHOULD requirement). Expected failure,
|
|
listed in `conformance-baseline.yml`.
|
|
|
|
## Auth Test Results (Spring HTTP Client)
|
|
|
|
**Status: 193 checks passed, 0 failed, 0 warnings across 14 scenarios**
|
|
|
|
Uses the `client-spring-http-client` module with Spring Security OAuth2 and the
|
|
[mcp-client-security](https://github.com/springaicommunity/mcp-client-security) library.
|
|
|
|
Fully passing: metadata-default, metadata-var1/2/3, basic-cimd,
|
|
scope-from-www-authenticate, scope-from-scopes-supported, scope-omitted-when-undefined,
|
|
scope-step-up, scope-retry-limit, token-endpoint-auth-basic/post/none, pre-registration.
|
|
|
|
Note: `auth/resource-mismatch` (present in earlier suite versions) is no longer part
|
|
of the 0.2.0-alpha auth suite.
|
|
|
|
## Known Limitations
|
|
|
|
1. **Client SSE Retry:** client doesn't parse or respect the `retry:` field,
|
|
reconnects immediately, and doesn't send the `Last-Event-ID` header
|
|
|
|
## Running Tests
|
|
|
|
### Server (active suite)
|
|
```bash
|
|
# Build and start server
|
|
./mvnw clean install -DskipTests
|
|
mvn exec:java -pl conformance-tests/server-servlet \
|
|
-Dexec.mainClass="io.modelcontextprotocol.conformance.server.ConformanceServlet"
|
|
|
|
# Run tests (in another terminal, from the repo root)
|
|
npx @modelcontextprotocol/conformance@0.2.0-alpha.11 server \
|
|
--url http://localhost:8080/mcp --suite active \
|
|
--expected-failures ./conformance-tests/conformance-baseline.yml
|
|
```
|
|
|
|
### Server (SEP-1613 scenario)
|
|
```bash
|
|
npx @modelcontextprotocol/conformance@0.2.0-alpha.11 server \
|
|
--url http://localhost:8080/mcp --scenario json-schema-2020-12
|
|
```
|
|
|
|
### Client
|
|
```bash
|
|
for scenario in initialize tools_call elicitation-sep1034-client-defaults sse-retry; do
|
|
npx @modelcontextprotocol/conformance@0.2.0-alpha.11 client \
|
|
--command "java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-*.jar" \
|
|
--scenario $scenario \
|
|
--expected-failures ./conformance-tests/conformance-baseline.yml
|
|
done
|
|
```
|
|
|
|
### Auth (Spring HTTP Client)
|
|
```bash
|
|
npx @modelcontextprotocol/conformance@0.2.0-alpha.11 client \
|
|
--spec-version 2025-11-25 \
|
|
--command "java -jar conformance-tests/client-spring-http-client/target/client-spring-http-client-*.jar" \
|
|
--suite auth \
|
|
--expected-failures ./conformance-tests/conformance-baseline.yml
|
|
```
|
|
|
|
## Recommendations
|
|
|
|
### High Priority
|
|
1. Fix client SSE retry field handling in `HttpClientStreamableHttpTransport`
|