f587167452* Assert on structured output in the smoke tests The smoke tests connected to each server, listed its tools and stopped there, so they could not catch a broken structured result. mcp-test-client now calls every tool that declares an outputSchema and checks the answer against it: the result must carry structuredContent, and a tool with an array-rooted schema must return a top-level JSON array rather than an object wrapping one. Tools with no output schema are left alone, so this is not a new requirement on any example. The client negotiates with mode "auto", so it exercises servers of either protocol era. Tool calls reach the live NWS API; an error result is reported as a skip rather than a failure. mock-mcp-server moves to the low-level Server and advertises one array-rooted and one object-rooted tool, with the schemas written out literally. A client that compiles every declared outputSchema at connect time fails against the array-rooted one if it assumes output schemas are always objects. utils.sh no longer swallows build output. A failed build left no binary behind and the test reported a downstream spawn ENOENT instead of the compiler error, which is what makes a broken build hard to diagnose from CI logs; #143 is the current example. check_dependency returns instead of exiting, so a missing toolchain fails its own test and the suite still prints its summary. Binaries resolve through a helper that accounts for the .exe suffix. The test helpers move to the 2.0 MCP SDK packages, where the 2026-07-28 support lives; the examples themselves are untouched. Those require Node 20 or newer, so CI moves off Node 18 to Node 24, the Active LTS - Node 20 reached end of life in April 2026. The Go examples and the Rust client stay uncovered here. Both clients abort when no .env file is present, so that coverage lands with those examples. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Update to the stable 2.0 SDK packages and address review feedback @modelcontextprotocol/{client,server} 2.0.0 shipped on 2026-07-27, so the helpers move off 2.0.0-beta.5. Review fixes: - strip the UTF-8 BOM that preceded smoke-test.sh's shebang - ensure_built and ensure_helpers_built return rather than exit, so one project failing to build no longer aborts the suite before the summary - engines.node and the documented Node requirement go to 20, matching what the 2.0 SDK packages actually require - close the client on the failure path, so a failing test cannot leave the spawned server running - the README claimed Go and Rust client coverage that did not exist Adds the Go weather server to the smoke test, with the Go toolchain CI was missing. The Go and Rust clients stay uncovered: on main both abort without a .env, so credential-free startup has to land in their own directories. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
131 lines
4.9 KiB
Markdown
131 lines
4.9 KiB
Markdown
# MCP Quickstart Smoke Tests
|
|
|
|
This directory contains smoke tests for the MCP quickstart examples. These tests verify that all example servers and clients can start and respond correctly.
|
|
|
|
## Overview
|
|
|
|
The smoke tests verify:
|
|
|
|
- **Servers**: Each weather server (Python, TypeScript, Rust, Go) can start, respond to MCP protocol requests, and honour the output schemas it advertises
|
|
- **Clients**: The Python and TypeScript MCP clients can connect to a mock server and list tools
|
|
|
|
The Go and Rust clients are not covered here: on `main` both abort when no `.env` file is present, so they cannot be driven without credentials. Making them start credential-free is a change in their own directories, so their coverage lands with those changes rather than here. The Ruby examples are not covered either — the `mcp` gem cannot negotiate protocol revision `2026-07-28`.
|
|
|
|
## Structured content
|
|
|
|
Listing tools is not enough to catch a broken structured result, so each server test also **calls** every tool that declares an `outputSchema` and checks the answer:
|
|
|
|
- the result must carry `structuredContent`, and it must conform to the declared schema (the SDK validates this and throws on a mismatch);
|
|
- a tool with an array-rooted schema must return a **top-level JSON array**, and one with an object-rooted schema must return an object.
|
|
|
|
The array case is the one worth guarding. A server that advertises `{"type": "array"}` and then answers `{"result": [...]}` passes a tools/list-only test and fails this one.
|
|
|
|
Tool calls reach the live NWS API. When it is unreachable the tools return an error result, which the test reports as a skip rather than a failure — someone else's outage should not fail the build.
|
|
|
|
## Running Tests
|
|
|
|
```bash
|
|
./tests/smoke-test.sh
|
|
```
|
|
|
|
## Requirements
|
|
|
|
- **Node.js** 20+ (required by the 2.0 MCP SDK packages)
|
|
- **npm** (for Node.js dependencies)
|
|
- **Python** 3.10+
|
|
- **uv** (Python package manager)
|
|
- **Rust** stable
|
|
- **Cargo** (for Rust builds)
|
|
- **Go** 1.25+
|
|
|
|
## How It Works
|
|
|
|
### Server Tests
|
|
|
|
Each server test:
|
|
|
|
1. Builds/prepares the server if needed (a failed build prints its compiler output rather than swallowing it)
|
|
2. Uses `mcp-test-client.ts` to connect to the server via stdio
|
|
3. Negotiates a protocol era with `mode: "auto"` — one `server/discover` probe, falling back to the `2025-11-25` `initialize` handshake
|
|
4. Lists tools, then calls each tool that declares an `outputSchema` and checks the structured result against it
|
|
5. Reports pass/fail
|
|
|
|
### Client Tests
|
|
|
|
Each client test:
|
|
|
|
1. Builds/prepares the client if needed
|
|
2. Runs the client CLI without an ANTHROPIC_API_KEY
|
|
3. The client connects to a mock server, lists tools, and exits gracefully
|
|
4. Verifies the client can connect and communicate via MCP protocol
|
|
5. Reports pass/fail
|
|
|
|
**Note**: Client tests run the actual CLI programs without an Anthropic API key. The clients are designed to handle missing API keys gracefully by listing available tools and exiting, which is perfect for smoke testing the MCP connectivity without requiring external API calls.
|
|
|
|
## Test Helpers
|
|
|
|
### mcp-test-client.ts
|
|
|
|
A minimal MCP client that connects to a server, initializes the session, and lists available tools. Used to test servers without requiring a full client implementation.
|
|
|
|
**Usage**:
|
|
|
|
```bash
|
|
node tests/helpers/build/mcp-test-client.js <command> [args...]
|
|
```
|
|
|
|
**Example**:
|
|
|
|
```bash
|
|
node tests/helpers/build/mcp-test-client.js python weather.py
|
|
```
|
|
|
|
### mock-mcp-server.ts
|
|
|
|
A minimal MCP server that verifies clients call the `tools/list` method. Used to test clients without requiring a real weather server. Exits with an error if the client doesn't call `tools/list`.
|
|
|
|
It advertises two tools whose output schemas cover both shapes a structured result can take: an object root, and an array root. The array-rooted one is deliberate — a client that compiles every declared `outputSchema` up front, as the Go and Rust quickstart clients do, will fail here if it assumes an output schema is always `{"type": "object"}`.
|
|
|
|
**Usage**:
|
|
|
|
```bash
|
|
node tests/helpers/build/mock-mcp-server.js
|
|
```
|
|
|
|
## CI/CD Integration
|
|
|
|
Tests run automatically on pull requests via GitHub Actions. See `.github/workflows/ci.yml` for the CI configuration.
|
|
|
|
## Troubleshooting
|
|
|
|
### Dependencies missing
|
|
|
|
Install required dependencies:
|
|
|
|
```bash
|
|
# Python/uv
|
|
curl -LsSf https://astral.sh/uv/install.sh | sh
|
|
|
|
# Node.js (via nvm)
|
|
nvm install 24
|
|
|
|
# Rust
|
|
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
|
```
|
|
|
|
## Adding New Tests
|
|
|
|
To add a new test:
|
|
|
|
1. Add a new test function in `smoke-test.sh` (e.g., `test_new_feature()`)
|
|
2. Include dependency checks, builds, and test execution in the function
|
|
3. Add a `run_test` call in the "Run all tests" section
|
|
4. Update this README
|
|
|
|
## Maintenance
|
|
|
|
These tests are designed to be simple and low-maintenance:
|
|
|
|
- Shell scripts for orchestration (language-agnostic)
|
|
- Minimal TypeScript helpers for test infrastructure
|
|
- No external API dependencies
|