Prepare SDK Tier 2 assessment submission
- Add CHANGELOG.md with release lines and history - Refresh ROADMAP.md: 2.x stable line, 3.0.x for 2026-07-28 spec (first milestones Sep 2026), Tier 2 commitment, 1.x/0.x security-patches-only - Refresh VALIDATION_RESULTS.md: 2026-08-17 run on conformance suite 0.2.0-alpha.11 (server 73/73, auth 193/193, client 3/4 w/ sse-retry baselined) - Bump conformance CI action pins v0.1.11 -> v0.1.16 - Update docs with missing features Signed-off-by: Christian Tzolov <christian.tzolov@broadcom.com>
This commit is contained in:
Christian Tzolov
Daniel Garnier-Moiroux
parent
8ee8ccbc17
commit
2fd159a79a
2 changed files with 432 additions and 105 deletions
|
|
@ -1,127 +1,110 @@
|
|||
# MCP Java SDK Conformance Test Validation Results
|
||||
|
||||
Last validated: **2026-08-17** against conformance suite
|
||||
**`@modelcontextprotocol/conformance@0.2.0-alpha.11`** (SDK at `main`, 2.0.1-SNAPSHOT), targetting
|
||||
version 2025-11-25 (`--spec-version 2025-11-25`).
|
||||
|
||||
## Summary
|
||||
|
||||
**Server Tests (active suite):** 44/44 passed (31 scenarios, 100%)
|
||||
**Server Tests (spec 2025-11-25):** 4/4 passed — SEP-1613 `json-schema-2020-12` scenario ✨
|
||||
**Client Tests:** 3/4 scenarios passed (9/10 checks passed)
|
||||
**Auth Tests:** 15/15 scenarios fully passing (195 passed, 0 failed, 0 warnings, 100% scenarios, 100% checks)
|
||||
**Server Tests (active suite):** 73/73 checks passed (31 scenarios, 100%)
|
||||
**Server Tests (SEP-1613 `json-schema-2020-12`):** 5/5 checks passed (SEP-2106 checks skipped — post-2025-11-25 spec additions)
|
||||
**Client Tests:** 3/4 scenarios passed; `sse-retry` fails (tracked in `conformance-baseline.yml`)
|
||||
**Auth Tests:** 14/14 scenarios passing (193 checks, 0 failed, 0 warnings)
|
||||
|
||||
Baseline check passed on every run: all failures are expected per
|
||||
[`conformance-baseline.yml`](conformance-baseline.yml).
|
||||
|
||||
## Server Test Results
|
||||
|
||||
### Active Suite — Passing (31/31 scenarios, 44/44 checks)
|
||||
### Active Suite — Passing (31/31 scenarios, 73/73 checks)
|
||||
|
||||
- **Lifecycle & Utilities (4/4):** initialize, ping, logging-set-level, completion-complete
|
||||
- **Tools (13/13):** All scenarios including progress notifications, sampling, elicitation ✨
|
||||
- **Elicitation (10/10):** SEP-1034 defaults (5 checks), SEP-1330 enums (5 checks)
|
||||
- **Resources (7/7):** list, read-text, read-binary, templates-read, subscribe, unsubscribe, SEP-2164 resource-not-found
|
||||
- **Prompts (5/5):** list, simple, with-args, embedded-resource, with-image
|
||||
- **SSE Transport (2/2):** Multiple streams
|
||||
- **Security (2/2):** Localhost validation passes, DNS rebinding protection
|
||||
- **Lifecycle & Utilities:** initialize, ping, logging-set-level, completion-complete
|
||||
- **Tools (13/13):** all scenarios including progress notifications, sampling, elicitation
|
||||
- **Elicitation:** SEP-1034 defaults (6 checks), SEP-1330 enums (6 checks)
|
||||
- **Resources:** list, read-text, read-binary, templates-read, subscribe, unsubscribe
|
||||
- **Prompts:** list, simple, with-args, embedded-resource, with-image
|
||||
- **SSE Transport:** multiple streams
|
||||
- **Security:** DNS rebinding protection
|
||||
|
||||
### Spec 2025-11-25 Scenarios — Passing (1/1 scenario, 4/4 checks)
|
||||
### SEP-1613 — JSON Schema 2020-12 (5/5 checks)
|
||||
|
||||
- **JSON Schema 2020-12 (SEP-1613) (4/4):** ✨
|
||||
- `json_schema_2020_12_tool` found
|
||||
- `inputSchema.$schema` field preserved
|
||||
- `inputSchema.$defs` field preserved
|
||||
- `inputSchema.additionalProperties` field preserved
|
||||
- `json_schema_2020_12_tool` found; `$schema`, `$defs`, and `additionalProperties`
|
||||
fields preserved; every JSON-RPC message valid per the spec JSON schema for the
|
||||
negotiated spec version (`wire-schema-valid`)
|
||||
- SEP-2106 checks (composition/conditional/anchor keywords) reported SKIPPED:
|
||||
they postdate the 2025-11-25 spec release and are excluded from scoring
|
||||
|
||||
## Client Test Results
|
||||
|
||||
### Passing (3/4 scenarios, 9/10 checks)
|
||||
### Passing (3/4 scenarios)
|
||||
|
||||
- **initialize (1/1):** Protocol negotiation, clientInfo, capabilities
|
||||
- **tools_call (1/1):** Tool discovery and invocation
|
||||
- **elicitation-sep1034-client-defaults (5/5):** Default values for string, integer, number, enum, boolean
|
||||
- **initialize (1/1):** protocol negotiation, clientInfo, capabilities
|
||||
- **tools_call (2/2):** tool discovery and invocation
|
||||
- **elicitation-sep1034-client-defaults (5/5):** default values for string, integer, number, enum, boolean
|
||||
|
||||
### Partially Passing (1/4 scenarios, 1/2 checks)
|
||||
### Failing — in baseline (1/4 scenarios)
|
||||
|
||||
- **sse-retry (1/2 + 1 warning):**
|
||||
- ✅ Reconnects after stream closure
|
||||
- ❌ Does not respect retry timing
|
||||
- ⚠️ Does not send Last-Event-ID header (SHOULD requirement)
|
||||
|
||||
**Issue:** Client treats `retry:` SSE field as invalid instead of parsing it for reconnection timing.
|
||||
- **sse-retry:** client does not parse/respect the `retry:` SSE field timing and
|
||||
does not send the `Last-Event-ID` header (SHOULD requirement). Expected failure,
|
||||
listed in `conformance-baseline.yml`.
|
||||
|
||||
## Auth Test Results (Spring HTTP Client)
|
||||
|
||||
**Status: 195 passed, 0 failed, 0 warnings across 15 scenarios**
|
||||
**Status: 193 checks passed, 0 failed, 0 warnings across 14 scenarios**
|
||||
|
||||
Uses the `client-spring-http-client` module with Spring Security OAuth2 and the [mcp-client-security](https://github.com/springaicommunity/mcp-client-security) library.
|
||||
Uses the `client-spring-http-client` module with Spring Security OAuth2 and the
|
||||
[mcp-client-security](https://github.com/springaicommunity/mcp-client-security) library.
|
||||
|
||||
### Fully Passing (15/15 scenarios)
|
||||
Fully passing: metadata-default, metadata-var1/2/3, basic-cimd,
|
||||
scope-from-www-authenticate, scope-from-scopes-supported, scope-omitted-when-undefined,
|
||||
scope-step-up, scope-retry-limit, token-endpoint-auth-basic/post/none, pre-registration.
|
||||
|
||||
- **auth/metadata-default (13/13):** Default metadata discovery
|
||||
- **auth/metadata-var1 (13/13):** Metadata discovery variant 1
|
||||
- **auth/metadata-var2 (13/13):** Metadata discovery variant 2
|
||||
- **auth/metadata-var3 (13/13):** Metadata discovery variant 3
|
||||
- **auth/basic-cimd (12/12):** Basic Client-Initiated Metadata Discovery
|
||||
- **auth/scope-from-www-authenticate (14/14):** Scope extraction from WWW-Authenticate header
|
||||
- **auth/scope-from-scopes-supported (14/14):** Scope extraction from scopes_supported
|
||||
- **auth/scope-omitted-when-undefined (14/14):** Scope omitted when not defined
|
||||
- **auth/scope-step-up (16/16):** Scope step-up challenge
|
||||
- **auth/scope-retry-limit (11/11):** Scope retry limit handling
|
||||
- **auth/token-endpoint-auth-basic (18/18):** Token endpoint with HTTP Basic auth
|
||||
- **auth/token-endpoint-auth-post (18/18):** Token endpoint with POST body auth
|
||||
- **auth/token-endpoint-auth-none (18/18):** Token endpoint with no client auth
|
||||
- **auth/resource-mismatch (2/2):** Resource mismatch handling
|
||||
- **auth/pre-registration (6/6):** Pre-registered client credentials flow
|
||||
Note: `auth/resource-mismatch` (present in earlier suite versions) is no longer part
|
||||
of the 0.2.0-alpha auth suite.
|
||||
|
||||
## Known Limitations
|
||||
|
||||
1. **Client SSE Retry:** Client doesn't parse or respect the `retry:` field, reconnects immediately, and doesn't send Last-Event-ID header
|
||||
1. **Client SSE Retry:** client doesn't parse or respect the `retry:` field,
|
||||
reconnects immediately, and doesn't send the `Last-Event-ID` header
|
||||
|
||||
## Running Tests
|
||||
|
||||
### Server (active suite)
|
||||
```bash
|
||||
# Start server
|
||||
./mvnw compile -pl conformance-tests/server-servlet -am exec:java
|
||||
# Build and start server
|
||||
./mvnw clean install -DskipTests
|
||||
mvn exec:java -pl conformance-tests/server-servlet \
|
||||
-Dexec.mainClass="io.modelcontextprotocol.conformance.server.ConformanceServlet"
|
||||
|
||||
# Run tests (in another terminal)
|
||||
npx @modelcontextprotocol/conformance server --url http://localhost:8080/mcp --suite active
|
||||
# Run tests (in another terminal, from the repo root)
|
||||
npx @modelcontextprotocol/conformance@0.2.0-alpha.11 server \
|
||||
--url http://localhost:8080/mcp --suite active \
|
||||
--expected-failures ./conformance-tests/conformance-baseline.yml
|
||||
```
|
||||
|
||||
### Server (spec 2025-11-25 scenarios — SEP-1613)
|
||||
### Server (SEP-1613 scenario)
|
||||
```bash
|
||||
# Start server (if not already running)
|
||||
./mvnw compile -pl conformance-tests/server-servlet -am exec:java
|
||||
|
||||
# Run json-schema-2020-12 scenario
|
||||
cd ../conformance && node --import tsx/esm src/index.ts server \
|
||||
--url http://localhost:8080/mcp \
|
||||
--scenario json-schema-2020-12
|
||||
npx @modelcontextprotocol/conformance@0.2.0-alpha.11 server \
|
||||
--url http://localhost:8080/mcp --scenario json-schema-2020-12
|
||||
```
|
||||
|
||||
### Client
|
||||
```bash
|
||||
# Build
|
||||
cd conformance-tests/client-jdk-http-client
|
||||
../../mvnw clean package -DskipTests
|
||||
|
||||
# Run all scenarios
|
||||
for scenario in initialize tools_call elicitation-sep1034-client-defaults sse-retry; do
|
||||
npx @modelcontextprotocol/conformance client \
|
||||
--command "java -jar target/client-jdk-http-client-1.1.0-SNAPSHOT.jar" \
|
||||
--scenario $scenario
|
||||
npx @modelcontextprotocol/conformance@0.2.0-alpha.11 client \
|
||||
--command "java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-*.jar" \
|
||||
--scenario $scenario \
|
||||
--expected-failures ./conformance-tests/conformance-baseline.yml
|
||||
done
|
||||
```
|
||||
|
||||
### Auth (Spring HTTP Client)
|
||||
|
||||
Ensure you run with the conformance testing suite `0.1.15` or higher.
|
||||
|
||||
```bash
|
||||
# Build
|
||||
cd conformance-tests/client-spring-http-client
|
||||
../../mvnw clean package -DskipTests
|
||||
|
||||
# Run auth suite
|
||||
npx @modelcontextprotocol/conformance@0.1.15 client \
|
||||
npx @modelcontextprotocol/conformance@0.2.0-alpha.11 client \
|
||||
--spec-version 2025-11-25 \
|
||||
--command "java -jar target/client-spring-http-client-1.1.0-SNAPSHOT.jar" \
|
||||
--suite auth
|
||||
--command "java -jar conformance-tests/client-spring-http-client/target/client-spring-http-client-*.jar" \
|
||||
--suite auth \
|
||||
--expected-failures ./conformance-tests/conformance-baseline.yml
|
||||
```
|
||||
|
||||
## Recommendations
|
||||
|
|
|
|||
|
|
@ -7,3 +7,6 @@ client:
|
|||
# - Client does not parse or respect retry: field timing
|
||||
# - Client does not send Last-Event-ID header
|
||||
- sse-retry
|
||||
# Not supporting versions > 1 year old
|
||||
- auth/2025-03-26-oauth-metadata-backcompat
|
||||
- auth/2025-03-26-oauth-endpoint-fallback
|
||||
|
|
|
|||
Loading…
Reference in a new issue