f04ccfd162<!-- Provide a brief summary of your changes -->
## Motivation and Context
1. The currently failing checks are commented out in `ConformanceTest`,
see the TODOs.
2. I added detailed logging for Gradle task, Server, and Client to allow
easier output reading when viewing tests results outside of IDE.
3. As a future improvement, instead of hard-coding scenarios, we can get
them with `npx @modelcontextprotocol/conformance list` and run all of
them.
4. The conformance framework writes its result into
`kotlin-sdk-test/results` directory and contains JSONs like the one
below:
```json
[
{
"id": "mcp-client-initialization",
"name": "MCPClientInitialization",
"description": "Validates that MCP client properly initializes with server",
"status": "SUCCESS",
"timestamp": "2025-11-28T14:38:36.718Z",
"specReferences": [
{
"id": "MCP-Lifecycle",
"url": "https://modelcontextprotocol.io/specification/2025-06-18/basic/lifecycle"
}
],
"details": {
"protocolVersionSent": "2025-06-18",
"expectedSpecVersion": "2025-06-18",
"versionMatch": true,
"clientName": "kotlin-conformance-client",
"clientVersion": "1.0.0"
}
},
{
"id": "server-info",
"name": "ServerInfo",
"description": "Test server info returned to client",
"status": "INFO",
"timestamp": "2025-11-28T14:38:36.718Z",
"specReferences": [
{
"id": "MCP-Lifecycle",
"url": "https://modelcontextprotocol.io/specification/2025-06-18/basic/lifecycle"
}
],
"details": {
"serverName": "test-server",
"serverVersion": "1.0.0"
}
}
]
```
As another future improvement, we can do somethig with these JSON files,
like putting them into a job summary on GitHub.
5. The client test also generated stderr.txt file in
`kotlin-sdk-test/results/initialize-.......` which contains errors
because it tries to call a tool. I haven't investigated if it's just the
way I wrote the client or related to `StreamableHttpClientTransport`.
6. The server is the simplified version of the existing server for
Kotlin<--->Typescript tests. If conformance tests extend significantly,
we can either combine the server implementations or ditch the
integration tests.
## How Has This Been Tested?
<!-- Have you tested this in a real application? Which scenarios were
tested? -->
## Breaking Changes
<!-- Will users need to update their code or configurations? -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [x] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [x] My code follows the repository's style guidelines
- [ ] New and existing tests pass locally
- [ ] I have added appropriate error handling
- [ ] I have added or updated documentation as needed
## Additional context
<!-- Add any other context, implementation notes, or design decisions
-->
---------
Co-authored-by: Konstantin Pavlov <1517853+kpavlov@users.noreply.github.com>
50 lines
1.3 KiB
YAML
50 lines
1.3 KiB
YAML
name: Conformance Tests
|
|
|
|
on:
|
|
workflow_dispatch:
|
|
pull_request:
|
|
branches: [ main ]
|
|
push:
|
|
branches: [ main ]
|
|
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
|
# Cancel only when the run is NOT on `main` branch
|
|
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
|
|
|
|
jobs:
|
|
run-conformance:
|
|
runs-on: macos-latest-xlarge
|
|
name: Run Conformance Tests
|
|
timeout-minutes: 20
|
|
env:
|
|
JAVA_OPTS: "-Xmx8g -Dfile.encoding=UTF-8 -Djava.awt.headless=true -Dkotlin.daemon.jvm.options=-Xmx6g"
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
|
|
- name: Set up JDK 21
|
|
uses: actions/setup-java@v5
|
|
with:
|
|
java-version: '21'
|
|
distribution: 'temurin'
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@v4
|
|
with:
|
|
node-version: '24.11.1'
|
|
|
|
- name: Setup Gradle
|
|
uses: gradle/actions/setup-gradle@v5
|
|
with:
|
|
add-job-summary: 'always'
|
|
cache-read-only: true
|
|
|
|
- name: Run Conformance Tests
|
|
run: ./gradlew --no-daemon :conformance-test:test
|
|
|
|
- name: Upload Conformance Results
|
|
if: ${{ !cancelled() }}
|
|
uses: actions/upload-artifact@v5
|
|
with:
|
|
name: conformance-results
|
|
path: conformance-test/results/
|