Add MCP conformance tests with client and server test suites (#767)
Implement conformance testing modules with comprehensive client/server test scenarios, sync conversion for client/servlet code, CI integration, and documentation updates. --------- Signed-off-by: Dariusz Jędrzejczyk <dariusz.jedrzejczyk@broadcom.com>
This commit is contained in:
Dariusz Jędrzejczyk
GitHub
parent
0a8cb1e038
commit
4fe8f3efef
4 changed files with 1558 additions and 0 deletions
135
conformance-tests/client-jdk-http-client/README.md
Normal file
135
conformance-tests/client-jdk-http-client/README.md
Normal file
|
|
@ -0,0 +1,135 @@
|
|||
# MCP Conformance Tests - JDK HTTP Client
|
||||
|
||||
This module provides a conformance test client implementation for the Java MCP SDK using the JDK HTTP Client with Streamable HTTP transport.
|
||||
|
||||
## Overview
|
||||
|
||||
The conformance test client is designed to work with the [MCP Conformance Test Framework](https://github.com/modelcontextprotocol/conformance). It validates that the Java MCP SDK client properly implements the MCP specification.
|
||||
|
||||
## Architecture
|
||||
|
||||
The client reads test scenarios from environment variables and accepts the server URL as a command-line argument, following the conformance framework's conventions:
|
||||
|
||||
- **MCP_CONFORMANCE_SCENARIO**: Environment variable specifying which test scenario to run
|
||||
- **Server URL**: Passed as the last command-line argument
|
||||
|
||||
## Supported Scenarios
|
||||
|
||||
Currently implemented scenarios:
|
||||
|
||||
- **initialize**: Tests the MCP client initialization handshake only
|
||||
- ✅ Validates protocol version negotiation
|
||||
- ✅ Validates clientInfo (name and version)
|
||||
- ✅ Validates proper handling of server capabilities
|
||||
- Does NOT call any tools or perform additional operations
|
||||
|
||||
- **tools_call**: Tests tool discovery and invocation
|
||||
- ✅ Initializes the client
|
||||
- ✅ Lists available tools from the server
|
||||
- ✅ Calls the `add_numbers` tool with test arguments (a=5, b=3)
|
||||
- ✅ Validates the tool result
|
||||
|
||||
- **elicitation-sep1034-client-defaults**: Tests client applies default values for omitted elicitation fields (SEP-1034)
|
||||
- ✅ Initializes the client
|
||||
- ✅ Lists available tools from the server
|
||||
- ✅ Calls the `test_client_elicitation_defaults` tool
|
||||
- ✅ Validates that the client properly applies default values from JSON schema to elicitation responses (5/5 checks pass)
|
||||
|
||||
- **sse-retry**: Tests client respects SSE retry field timing and reconnects properly (SEP-1699)
|
||||
- ⚠️ Initializes the client
|
||||
- ⚠️ Lists available tools from the server
|
||||
- ⚠️ Calls the `test_reconnection` tool which triggers SSE stream closure
|
||||
- ✅ Client reconnects after stream closure (PASSING)
|
||||
- ❌ Client does not respect retry timing (FAILING)
|
||||
- ⚠️ Client does not send Last-Event-ID header (WARNING - SHOULD requirement)
|
||||
|
||||
## Building
|
||||
|
||||
Build the executable JAR:
|
||||
|
||||
```bash
|
||||
cd conformance-tests/client-jdk-http-client
|
||||
../../mvnw clean package -DskipTests
|
||||
```
|
||||
|
||||
This creates an executable JAR at:
|
||||
```
|
||||
target/client-jdk-http-client-0.18.0-SNAPSHOT.jar
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
### Using the Conformance Framework
|
||||
|
||||
Run a single scenario:
|
||||
|
||||
```bash
|
||||
npx @modelcontextprotocol/conformance client \
|
||||
--command "java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar" \
|
||||
--scenario initialize
|
||||
|
||||
npx @modelcontextprotocol/conformance client \
|
||||
--command "java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar" \
|
||||
--scenario tools_call
|
||||
|
||||
npx @modelcontextprotocol/conformance client \
|
||||
--command "java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar" \
|
||||
--scenario elicitation-sep1034-client-defaults
|
||||
|
||||
npx @modelcontextprotocol/conformance client \
|
||||
--command "java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar" \
|
||||
--scenario sse-retry
|
||||
```
|
||||
|
||||
Run with verbose output:
|
||||
|
||||
```bash
|
||||
npx @modelcontextprotocol/conformance client \
|
||||
--command "java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar" \
|
||||
--scenario initialize \
|
||||
--verbose
|
||||
```
|
||||
|
||||
### Manual Testing
|
||||
|
||||
You can also run the client manually if you have a test server:
|
||||
|
||||
```bash
|
||||
export MCP_CONFORMANCE_SCENARIO=initialize
|
||||
java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar http://localhost:3000/mcp
|
||||
```
|
||||
|
||||
## Test Results
|
||||
|
||||
The conformance framework generates test results showing:
|
||||
|
||||
**Current Status (3/4 scenarios passing):**
|
||||
- ✅ initialize: 1/1 checks passed
|
||||
- ✅ tools_call: 1/1 checks passed
|
||||
- ✅ elicitation-sep1034-client-defaults: 5/5 checks passed
|
||||
- ⚠️ sse-retry: 1/2 checks passed, 1 warning
|
||||
|
||||
Test result files are generated in `results/<scenario>-<timestamp>/`:
|
||||
- `checks.json`: Array of conformance check results with pass/fail status
|
||||
- `stdout.txt`: Client stdout output
|
||||
- `stderr.txt`: Client stderr output
|
||||
|
||||
### Known Issue: SSE Retry Handling
|
||||
|
||||
The `sse-retry` scenario currently fails because:
|
||||
1. The client treats the SSE `retry:` field as invalid instead of parsing it
|
||||
2. The client does not implement retry timing (reconnects immediately)
|
||||
3. The client does not send the Last-Event-ID header on reconnection
|
||||
|
||||
This is a known limitation in the `HttpClientStreamableHttpTransport` implementation.
|
||||
|
||||
## Next Steps
|
||||
|
||||
Future enhancements:
|
||||
|
||||
- Fix SSE retry field handling (SEP-1699) to properly parse and respect retry timing
|
||||
- Implement Last-Event-ID header on reconnection for resumability
|
||||
- Add auth scenarios (currently excluded as per requirements)
|
||||
- Implement a comprehensive "everything-client" pattern
|
||||
- Add to CI/CD pipeline
|
||||
- Create expected-failures baseline for known issues
|
||||
78
conformance-tests/client-jdk-http-client/pom.xml
Normal file
78
conformance-tests/client-jdk-http-client/pom.xml
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<parent>
|
||||
<groupId>io.modelcontextprotocol.sdk</groupId>
|
||||
<artifactId>conformance-tests</artifactId>
|
||||
<version>0.18.0-SNAPSHOT</version>
|
||||
</parent>
|
||||
<artifactId>client-jdk-http-client</artifactId>
|
||||
<packaging>jar</packaging>
|
||||
<name>MCP Conformance Tests - JDK HTTP Client</name>
|
||||
<description>JDK HTTP Client conformance tests for the Java MCP SDK</description>
|
||||
<url>https://github.com/modelcontextprotocol/java-sdk</url>
|
||||
|
||||
<scm>
|
||||
<url>https://github.com/modelcontextprotocol/java-sdk</url>
|
||||
<connection>git://github.com/modelcontextprotocol/java-sdk.git</connection>
|
||||
<developerConnection>git@github.com/modelcontextprotocol/java-sdk.git</developerConnection>
|
||||
</scm>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>io.modelcontextprotocol.sdk</groupId>
|
||||
<artifactId>mcp</artifactId>
|
||||
<version>0.18.0-SNAPSHOT</version>
|
||||
</dependency>
|
||||
|
||||
<!-- Logging -->
|
||||
<dependency>
|
||||
<groupId>ch.qos.logback</groupId>
|
||||
<artifactId>logback-classic</artifactId>
|
||||
<version>${logback.version}</version>
|
||||
<scope>runtime</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<!-- Maven Shade Plugin for creating executable JAR with dependencies -->
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-shade-plugin</artifactId>
|
||||
<version>3.5.1</version>
|
||||
<executions>
|
||||
<execution>
|
||||
<phase>package</phase>
|
||||
<goals>
|
||||
<goal>shade</goal>
|
||||
</goals>
|
||||
<configuration>
|
||||
<transformers>
|
||||
<transformer
|
||||
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
|
||||
<mainClass>io.modelcontextprotocol.conformance.client.ConformanceJdkClientMcpClient</mainClass>
|
||||
</transformer>
|
||||
<transformer
|
||||
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
|
||||
</transformers>
|
||||
<filters>
|
||||
<filter>
|
||||
<artifact>*:*</artifact>
|
||||
<excludes>
|
||||
<exclude>META-INF/*.SF</exclude>
|
||||
<exclude>META-INF/*.DSA</exclude>
|
||||
<exclude>META-INF/*.RSA</exclude>
|
||||
</excludes>
|
||||
</filter>
|
||||
</filters>
|
||||
</configuration>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
</project>
|
||||
|
|
@ -0,0 +1,286 @@
|
|||
package io.modelcontextprotocol.conformance.client;
|
||||
|
||||
import java.time.Duration;
|
||||
|
||||
import io.modelcontextprotocol.client.McpClient;
|
||||
import io.modelcontextprotocol.client.McpSyncClient;
|
||||
import io.modelcontextprotocol.client.transport.HttpClientStreamableHttpTransport;
|
||||
import io.modelcontextprotocol.spec.McpSchema;
|
||||
|
||||
/**
|
||||
* MCP Conformance Test Client - JDK HTTP Client Implementation
|
||||
*
|
||||
* <p>
|
||||
* This client is designed to work with the MCP conformance test framework. It reads the
|
||||
* test scenario from the MCP_CONFORMANCE_SCENARIO environment variable and the server URL
|
||||
* from command-line arguments.
|
||||
*
|
||||
* <p>
|
||||
* Usage: ConformanceJdkClientMcpClient <server-url>
|
||||
*
|
||||
* @see <a href= "https://github.com/modelcontextprotocol/conformance">MCP Conformance
|
||||
* Test Framework</a>
|
||||
*/
|
||||
public class ConformanceJdkClientMcpClient {
|
||||
|
||||
public static void main(String[] args) {
|
||||
if (args.length == 0) {
|
||||
System.err.println("Usage: ConformanceJdkClientMcpClient <server-url>");
|
||||
System.err.println("The server URL must be provided as the last command-line argument.");
|
||||
System.err.println("The MCP_CONFORMANCE_SCENARIO environment variable must be set.");
|
||||
System.exit(1);
|
||||
}
|
||||
|
||||
String scenario = System.getenv("MCP_CONFORMANCE_SCENARIO");
|
||||
if (scenario == null || scenario.isEmpty()) {
|
||||
System.err.println("Error: MCP_CONFORMANCE_SCENARIO environment variable is not set");
|
||||
System.exit(1);
|
||||
}
|
||||
|
||||
String serverUrl = args[args.length - 1];
|
||||
|
||||
try {
|
||||
switch (scenario) {
|
||||
case "initialize":
|
||||
runInitializeScenario(serverUrl);
|
||||
break;
|
||||
case "tools_call":
|
||||
runToolsCallScenario(serverUrl);
|
||||
break;
|
||||
case "elicitation-sep1034-client-defaults":
|
||||
runElicitationDefaultsScenario(serverUrl);
|
||||
break;
|
||||
case "sse-retry":
|
||||
runSSERetryScenario(serverUrl);
|
||||
break;
|
||||
default:
|
||||
System.err.println("Unknown scenario: " + scenario);
|
||||
System.err.println("Available scenarios:");
|
||||
System.err.println(" - initialize");
|
||||
System.err.println(" - tools_call");
|
||||
System.err.println(" - elicitation-sep1034-client-defaults");
|
||||
System.err.println(" - sse-retry");
|
||||
System.exit(1);
|
||||
}
|
||||
System.exit(0);
|
||||
}
|
||||
catch (Exception e) {
|
||||
System.err.println("Error: " + e.getMessage());
|
||||
e.printStackTrace();
|
||||
System.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Helper method to create and configure an MCP client with transport.
|
||||
* @param serverUrl the URL of the MCP server
|
||||
* @return configured McpSyncClient instance
|
||||
*/
|
||||
private static McpSyncClient createClient(String serverUrl) {
|
||||
HttpClientStreamableHttpTransport transport = HttpClientStreamableHttpTransport.builder(serverUrl).build();
|
||||
|
||||
return McpClient.sync(transport)
|
||||
.clientInfo(new McpSchema.Implementation("test-client", "1.0.0"))
|
||||
.requestTimeout(Duration.ofSeconds(30))
|
||||
.build();
|
||||
}
|
||||
|
||||
/**
|
||||
* Helper method to create and configure an MCP client with elicitation support.
|
||||
* @param serverUrl the URL of the MCP server
|
||||
* @return configured McpSyncClient instance with elicitation handler
|
||||
*/
|
||||
private static McpSyncClient createClientWithElicitation(String serverUrl) {
|
||||
HttpClientStreamableHttpTransport transport = HttpClientStreamableHttpTransport.builder(serverUrl).build();
|
||||
|
||||
// Build client capabilities with elicitation support
|
||||
var capabilities = McpSchema.ClientCapabilities.builder().elicitation().build();
|
||||
|
||||
return McpClient.sync(transport)
|
||||
.clientInfo(new McpSchema.Implementation("test-client", "1.0.0"))
|
||||
.requestTimeout(Duration.ofSeconds(30))
|
||||
.capabilities(capabilities)
|
||||
.elicitation(request -> {
|
||||
// Apply default values from the schema to create the content
|
||||
var content = new java.util.HashMap<String, Object>();
|
||||
var schema = request.requestedSchema();
|
||||
|
||||
if (schema != null && schema.containsKey("properties")) {
|
||||
@SuppressWarnings("unchecked")
|
||||
var properties = (java.util.Map<String, Object>) schema.get("properties");
|
||||
|
||||
// Apply defaults for each property
|
||||
properties.forEach((key, propDef) -> {
|
||||
@SuppressWarnings("unchecked")
|
||||
var propMap = (java.util.Map<String, Object>) propDef;
|
||||
if (propMap.containsKey("default")) {
|
||||
content.put(key, propMap.get("default"));
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// Return accept action with the defaults applied
|
||||
return new McpSchema.ElicitResult(McpSchema.ElicitResult.Action.ACCEPT, content, null);
|
||||
})
|
||||
.build();
|
||||
}
|
||||
|
||||
/**
|
||||
* Initialize scenario: Tests MCP client initialization handshake.
|
||||
* @param serverUrl the URL of the MCP server
|
||||
* @throws Exception if any error occurs during execution
|
||||
*/
|
||||
private static void runInitializeScenario(String serverUrl) throws Exception {
|
||||
McpSyncClient client = createClient(serverUrl);
|
||||
|
||||
try {
|
||||
// Initialize client
|
||||
client.initialize();
|
||||
|
||||
System.out.println("Successfully connected to MCP server");
|
||||
}
|
||||
finally {
|
||||
// Close the client (which will close the transport)
|
||||
client.close();
|
||||
System.out.println("Connection closed successfully");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Tools call scenario: Tests tool listing and invocation functionality.
|
||||
* @param serverUrl the URL of the MCP server
|
||||
* @throws Exception if any error occurs during execution
|
||||
*/
|
||||
private static void runToolsCallScenario(String serverUrl) throws Exception {
|
||||
McpSyncClient client = createClient(serverUrl);
|
||||
|
||||
try {
|
||||
// Initialize client
|
||||
client.initialize();
|
||||
|
||||
System.out.println("Successfully connected to MCP server");
|
||||
|
||||
// List available tools
|
||||
McpSchema.ListToolsResult toolsResult = client.listTools();
|
||||
System.out.println("Successfully listed tools");
|
||||
|
||||
// Call the add_numbers tool if it exists
|
||||
if (toolsResult != null && toolsResult.tools() != null) {
|
||||
for (McpSchema.Tool tool : toolsResult.tools()) {
|
||||
if ("add_numbers".equals(tool.name())) {
|
||||
// Call the add_numbers tool with test arguments
|
||||
var arguments = new java.util.HashMap<String, Object>();
|
||||
arguments.put("a", 5);
|
||||
arguments.put("b", 3);
|
||||
|
||||
McpSchema.CallToolResult result = client
|
||||
.callTool(new McpSchema.CallToolRequest("add_numbers", arguments));
|
||||
|
||||
System.out.println("Successfully called add_numbers tool");
|
||||
if (result != null && result.content() != null) {
|
||||
System.out.println("Tool result: " + result.content());
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
finally {
|
||||
// Close the client (which will close the transport)
|
||||
client.close();
|
||||
System.out.println("Connection closed successfully");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Elicitation defaults scenario: Tests client applies default values for omitted
|
||||
* elicitation fields (SEP-1034).
|
||||
* @param serverUrl the URL of the MCP server
|
||||
* @throws Exception if any error occurs during execution
|
||||
*/
|
||||
private static void runElicitationDefaultsScenario(String serverUrl) throws Exception {
|
||||
McpSyncClient client = createClientWithElicitation(serverUrl);
|
||||
|
||||
try {
|
||||
// Initialize client
|
||||
client.initialize();
|
||||
|
||||
System.out.println("Successfully connected to MCP server");
|
||||
|
||||
// List available tools
|
||||
McpSchema.ListToolsResult toolsResult = client.listTools();
|
||||
System.out.println("Successfully listed tools");
|
||||
|
||||
// Call the test_client_elicitation_defaults tool if it exists
|
||||
if (toolsResult != null && toolsResult.tools() != null) {
|
||||
for (McpSchema.Tool tool : toolsResult.tools()) {
|
||||
if ("test_client_elicitation_defaults".equals(tool.name())) {
|
||||
// Call the tool which will trigger an elicitation request
|
||||
var arguments = new java.util.HashMap<String, Object>();
|
||||
|
||||
McpSchema.CallToolResult result = client
|
||||
.callTool(new McpSchema.CallToolRequest("test_client_elicitation_defaults", arguments));
|
||||
|
||||
System.out.println("Successfully called test_client_elicitation_defaults tool");
|
||||
if (result != null && result.content() != null) {
|
||||
System.out.println("Tool result: " + result.content());
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
finally {
|
||||
// Close the client (which will close the transport)
|
||||
client.close();
|
||||
System.out.println("Connection closed successfully");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* SSE retry scenario: Tests client respects SSE retry field timing and reconnects
|
||||
* properly (SEP-1699).
|
||||
* @param serverUrl the URL of the MCP server
|
||||
* @throws Exception if any error occurs during execution
|
||||
*/
|
||||
private static void runSSERetryScenario(String serverUrl) throws Exception {
|
||||
McpSyncClient client = createClient(serverUrl);
|
||||
|
||||
try {
|
||||
// Initialize client
|
||||
client.initialize();
|
||||
|
||||
System.out.println("Successfully connected to MCP server");
|
||||
|
||||
// List available tools
|
||||
McpSchema.ListToolsResult toolsResult = client.listTools();
|
||||
System.out.println("Successfully listed tools");
|
||||
|
||||
// Call the test_reconnection tool if it exists
|
||||
if (toolsResult != null && toolsResult.tools() != null) {
|
||||
for (McpSchema.Tool tool : toolsResult.tools()) {
|
||||
if ("test_reconnection".equals(tool.name())) {
|
||||
// Call the tool which will trigger SSE stream closure and
|
||||
// reconnection
|
||||
var arguments = new java.util.HashMap<String, Object>();
|
||||
|
||||
McpSchema.CallToolResult result = client
|
||||
.callTool(new McpSchema.CallToolRequest("test_reconnection", arguments));
|
||||
|
||||
System.out.println("Successfully called test_reconnection tool");
|
||||
if (result != null && result.content() != null) {
|
||||
System.out.println("Tool result: " + result.content());
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
finally {
|
||||
// Close the client (which will close the transport)
|
||||
client.close();
|
||||
System.out.println("Connection closed successfully");
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<configuration>
|
||||
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
|
||||
<encoder>
|
||||
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
|
||||
</encoder>
|
||||
</appender>
|
||||
|
||||
<!-- Set default log level -->
|
||||
<root level="INFO">
|
||||
<appender-ref ref="STDOUT" />
|
||||
</root>
|
||||
|
||||
<!-- More verbose logging for MCP client -->
|
||||
<logger name="io.modelcontextprotocol" level="DEBUG" />
|
||||
</configuration>
|
||||
Loading…
Reference in a new issue