modelcontextprotocol/quickstart-resources
Watch
1
0
Fork
You've already forked quickstart-resources
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Assert on structured output in the smoke tests and update to v2 (#163)

Browse source 
* Assert on structured output in the smoke tests

The smoke tests connected to each server, listed its tools and stopped there,
so they could not catch a broken structured result.

mcp-test-client now calls every tool that declares an outputSchema and checks
the answer against it: the result must carry structuredContent, and a tool
with an array-rooted schema must return a top-level JSON array rather than an
object wrapping one. Tools with no output schema are left alone, so this is
not a new requirement on any example. The client negotiates with mode "auto",
so it exercises servers of either protocol era. Tool calls reach the live NWS
API; an error result is reported as a skip rather than a failure.

mock-mcp-server moves to the low-level Server and advertises one array-rooted
and one object-rooted tool, with the schemas written out literally. A client
that compiles every declared outputSchema at connect time fails against the
array-rooted one if it assumes output schemas are always objects.

utils.sh no longer swallows build output. A failed build left no binary behind
and the test reported a downstream spawn ENOENT instead of the compiler error,
which is what makes a broken build hard to diagnose from CI logs; #143 is the
current example. check_dependency returns instead of exiting, so a missing
toolchain fails its own test and the suite still prints its summary. Binaries
resolve through a helper that accounts for the .exe suffix.

The test helpers move to the 2.0 MCP SDK packages, where the 2026-07-28
support lives; the examples themselves are untouched. Those require Node 20 or
newer, so CI moves off Node 18 to Node 24, the Active LTS - Node 20 reached
end of life in April 2026.

The Go examples and the Rust client stay uncovered here. Both clients abort
when no .env file is present, so that coverage lands with those examples.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Update to the stable 2.0 SDK packages and address review feedback

@modelcontextprotocol/{client,server} 2.0.0 shipped on 2026-07-27, so the
helpers move off 2.0.0-beta.5.

Review fixes:
- strip the UTF-8 BOM that preceded smoke-test.sh's shebang
- ensure_built and ensure_helpers_built return rather than exit, so one
  project failing to build no longer aborts the suite before the summary
- engines.node and the documented Node requirement go to 20, matching what
  the 2.0 SDK packages actually require
- close the client on the failure path, so a failing test cannot leave the
  spawned server running
- the README claimed Go and Rust client coverage that did not exist

Adds the Go weather server to the smoke test, with the Go toolchain CI was
missing. The Go and Rust clients stay uncovered: on main both abort without
a .env, so credential-free startup has to land in their own directories.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Ola Hungerford 2026-08-03 10:42:57 -07:00 • committed by GitHub
commit f587167452
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 378 additions and 1114 deletions
Download patch file Download diff file Expand all files Collapse all files

9
.github/workflows/ci.yml vendored
View file

@ -14,10 +14,12 @@ jobs:
- name: Checkout repository
uses: actions/checkout@v4
# Node 24 is the Active LTS. The 2.0 MCP SDK packages require >=20, but
# Node 20 reached end of life in April 2026.
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: "18"
node-version: "24"
- name: Set up Python
uses: actions/setup-python@v5
@ -34,5 +36,10 @@ jobs:
with:
toolchain: stable
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: "1.25"
- name: Run smoke tests
run: ./tests/smoke-test.sh

8
.gitignore vendored
View file

@ -19,6 +19,14 @@ Gemfile.lock
# Environment variables file
.env
# Go-generated binaries
# `smoke-test.sh` builds the Go examples as `server`; `go build` with no -o
# names the binary after its directory.
*.exe
/weather-server-go/server
/weather-server-go/weather-server-go
/mcp-client-go/mcp-client-go
# Rust-generated files
debug
target

34
tests/README.md
View file

@ -1,13 +1,26 @@
# MCP Quickstart Smoke Tests
This directory contains smoke tests for the MCP quickstart examples. These tests verify that all example servers and clients can start and respond correctly, without calling external APIs.
This directory contains smoke tests for the MCP quickstart examples. These tests verify that all example servers and clients can start and respond correctly.
## Overview
The smoke tests verify:
- **Servers**: Each weather server (Python, TypeScript, Rust) can start and respond to MCP protocol requests
- **Clients**: Each MCP client (Python, TypeScript) can connect to a mock server and list tools
- **Servers**: Each weather server (Python, TypeScript, Rust, Go) can start, respond to MCP protocol requests, and honour the output schemas it advertises
- **Clients**: The Python and TypeScript MCP clients can connect to a mock server and list tools
The Go and Rust clients are not covered here: on `main` both abort when no `.env` file is present, so they cannot be driven without credentials. Making them start credential-free is a change in their own directories, so their coverage lands with those changes rather than here. The Ruby examples are not covered either — the `mcp` gem cannot negotiate protocol revision `2026-07-28`.
## Structured content
Listing tools is not enough to catch a broken structured result, so each server test also **calls** every tool that declares an `outputSchema` and checks the answer:
- the result must carry `structuredContent`, and it must conform to the declared schema (the SDK validates this and throws on a mismatch);
- a tool with an array-rooted schema must return a **top-level JSON array**, and one with an object-rooted schema must return an object.
The array case is the one worth guarding. A server that advertises `{"type": "array"}` and then answers `{"result": [...]}` passes a tools/list-only test and fails this one.
Tool calls reach the live NWS API. When it is unreachable the tools return an error result, which the test reports as a skip rather than a failure — someone else's outage should not fail the build.
## Running Tests
@ -17,12 +30,13 @@ The smoke tests verify:
## Requirements
- **Node.js** 16+
- **Node.js** 20+ (required by the 2.0 MCP SDK packages)
- **npm** (for Node.js dependencies)
- **Python** 3.10+
- **uv** (Python package manager)
- **Rust** stable
- **Cargo** (for Rust builds)
- **Go** 1.25+
## How It Works
@ -30,10 +44,10 @@ The smoke tests verify:
Each server test:
1. Builds/prepares the server if needed
1. Builds/prepares the server if needed (a failed build prints its compiler output rather than swallowing it)
2. Uses `mcp-test-client.ts` to connect to the server via stdio
3. Sends MCP initialize and `tools/list` requests
4. Verifies the server responds with a valid tool list
3. Negotiates a protocol era with `mode: "auto"` — one `server/discover` probe, falling back to the `2025-11-25` `initialize` handshake
4. Lists tools, then calls each tool that declares an `outputSchema` and checks the structured result against it
5. Reports pass/fail
### Client Tests
@ -68,7 +82,9 @@ node tests/helpers/build/mcp-test-client.js python weather.py
### mock-mcp-server.ts
A minimal MCP server that verifies clients call the `tools/list` method and returns an empty tool list. Used to test clients without requiring a real weather server. Exits with an error if the client doesn't call `tools/list`.
A minimal MCP server that verifies clients call the `tools/list` method. Used to test clients without requiring a real weather server. Exits with an error if the client doesn't call `tools/list`.
It advertises two tools whose output schemas cover both shapes a structured result can take: an object root, and an array root. The array-rooted one is deliberate — a client that compiles every declared `outputSchema` up front, as the Go and Rust quickstart clients do, will fail here if it assumes an output schema is always `{"type": "object"}`.
**Usage**:
@ -91,7 +107,7 @@ Install required dependencies:
curl -LsSf https://astral.sh/uv/install.sh | sh
# Node.js (via nvm)
nvm install 18
nvm install 24
# Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

109
tests/helpers/mcp-test-client.ts
View file

@ -1,45 +1,120 @@
#!/usr/bin/env node
/**
* Minimal MCP Test Client for testing servers
* Connects to a server, initializes, and lists tools
*
* Connects to a server, initializes, lists tools, and then checks the tools
* actually honour what they advertise:
*
* - every tool that declares an `outputSchema` is called, and the result must
* carry `structuredContent` (the SDK validates it against the schema for us
* and throws on a mismatch);
* - a tool whose `outputSchema` is array-rooted must answer with a top-level
* JSON array, not an object wrapping one.
*
* The second check is the point of the exercise. Listing tools alone would not
* notice a server that advertises `{"type": "array"}` and then returns
* `{"result": [...]}`.
*/
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";
import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
/** Arguments to call a tool with, chosen by matching its name. */
const TOOL_ARGUMENTS: { match: RegExp; args: Record<string, unknown> }[] = [
{ match: /alert/i, args: { state: "CA" } },
{ match: /forecast/i, args: { latitude: 38.5816, longitude: -121.4944 } },
];
function argumentsFor(name: string): Record<string, unknown> | undefined {
return TOOL_ARGUMENTS.find(({ match }) => match.test(name))?.args;
}
/** The root `type` of a JSON Schema, when it declares a single one. */
function rootType(schema: unknown): string | undefined {
if (typeof schema !== "object" || schema === null) return undefined;
const type = (schema as { type?: unknown }).type;
return typeof type === "string" ? type : undefined;
}
async function testServer(command: string, args: string[]) {
console.error(`Testing server: ${command} ${args.join(" ")}`);
const transport = new StdioClientTransport({
command,
args,
});
const transport = new StdioClientTransport({ command, args });
// `auto` probes for 2026-07-28 and falls back to the 2025-11-25 handshake, so
// this one helper tests servers of either era.
const client = new Client(
{
name: "mcp-test-client",
version: "1.0.0",
},
{
capabilities: {},
}
{ name: "mcp-test-client", version: "1.0.0" },
{ capabilities: {}, versionNegotiation: { mode: "auto" } },
);
try {
// Connect to server
await client.connect(transport);
console.error("✓ Connected to server");
// List tools
const { tools } = await client.listTools();
console.error(`✓ Listed ${tools.length} tools`);
// Success
let checked = 0;
for (const tool of tools) {
if (!tool.outputSchema) continue;
const toolArgs = argumentsFor(tool.name);
if (!toolArgs) {
console.error(` - ${tool.name}: no known arguments, skipping call`);
continue;
}
// A throw here is the SDK rejecting the result against `outputSchema`.
const result = await client.callTool({
name: tool.name,
arguments: toolArgs,
});
// Upstream (api.weather.gov) being unreachable surfaces as an error
// result, which is a legitimate answer and carries no structured data.
// Skip rather than fail the build on someone else's outage.
if (result.isError) {
console.error(` - ${tool.name}: returned an error result, skipping`);
continue;
}
if (result.structuredContent === undefined) {
throw new Error(
`${tool.name} declares an output schema but returned no structuredContent`,
);
}
const expected = rootType(tool.outputSchema);
const isArray = Array.isArray(result.structuredContent);
if (expected === "array" && !isArray) {
throw new Error(
`${tool.name} declares an array-rooted output schema but returned ` +
`${JSON.stringify(result.structuredContent).slice(0, 80)}`,
);
}
if (expected === "object" && isArray) {
throw new Error(
`${tool.name} declares an object-rooted output schema but returned an array`,
);
}
console.error(
`✓ ${tool.name}: structuredContent is ${isArray ? "a top-level array" : "an object"}, matching its schema`,
);
checked += 1;
}
console.error(`✓ Verified structured content for ${checked} tools`);
console.error("✓ Server test passed");
await client.close();
process.exit(0);
} catch (error) {
console.error(`✗ Server test failed: ${error}`);
// Close on the way out too, or the spawned server outlives this process and
// a failing test hangs instead of reporting.
await client.close().catch(() => {});
process.exit(1);
}
}

131
tests/helpers/mock-mcp-server.ts
View file

@ -1,40 +1,116 @@
#!/usr/bin/env node
/**
* Mock MCP Server for testing clients
* Verifies that clients call the tools/list method and returns an empty tool list
*
* Verifies that clients call `tools/list`, and advertises tools whose output
* schemas cover both shapes a structured result can take: an object root, and
* an array root (allowed as of protocol revision 2026-07-28).
*
* The array-rooted tool is deliberate. Clients that compile every declared
* `outputSchema` up front — as the Go and Rust quickstart clients do — will
* fail here if they assume an output schema is always `{"type": "object"}`.
*
* This uses the low-level `Server` rather than `McpServer` so the schemas are
* written out literally, which is the point of a mock: what goes on the wire is
* exactly what is in this file.
*/
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
const server = new McpServer(
{
name: "mock-test-server",
version: "1.0.0",
},
{
capabilities: {
tools: {},
},
}
);
import type { Tool } from "@modelcontextprotocol/server";
import { Server } from "@modelcontextprotocol/server";
import { serveStdio } from "@modelcontextprotocol/server/stdio";
// Track whether tools/list was called
let toolsListCalled = false;
// Override the default tools/list handler to track calls
server.server.setRequestHandler(ListToolsRequestSchema, async () => {
toolsListCalled = true;
return { tools: [] };
});
const TOOLS: Tool[] = [
{
name: "get_alerts",
description: "Mock alerts, returning a top-level array",
inputSchema: {
type: "object",
properties: { state: { type: "string" } },
required: ["state"],
},
// Array root: legal as of 2026-07-28, rejected by older revisions.
outputSchema: {
type: "array",
items: {
type: "object",
properties: { event: { type: "string" }, area: { type: "string" } },
required: ["event", "area"],
},
},
},
{
name: "get_forecast",
description: "Mock forecast, returning an object",
inputSchema: {
type: "object",
properties: {
latitude: { type: "number" },
longitude: { type: "number" },
},
required: ["latitude", "longitude"],
},
outputSchema: {
type: "object",
properties: {
latitude: { type: "number" },
longitude: { type: "number" },
summary: { type: "string" },
},
required: ["latitude", "longitude", "summary"],
},
},
];
async function main() {
const transport = new StdioServerTransport();
await server.connect(transport);
console.error("Mock MCP Server running on stdio");
function buildServer(): Server {
const server = new Server(
{ name: "mock-test-server", version: "1.0.0" },
{ capabilities: { tools: {} } },
);
server.setRequestHandler("tools/list", async () => {
toolsListCalled = true;
return { tools: TOOLS };
});
server.setRequestHandler("tools/call", async (request) => {
const { name, arguments: args = {} } = request.params;
if (name === "get_alerts") {
const state = String((args as { state?: unknown }).state ?? "??");
const alerts = [{ event: "Mock Warning", area: state }];
return {
content: [{ type: "text", text: `1 alert for ${state}` }],
structuredContent: alerts,
};
}
if (name === "get_forecast") {
const { latitude = 0, longitude = 0 } = args as {
latitude?: number;
longitude?: number;
};
const forecast = { latitude, longitude, summary: "Mock conditions" };
return {
content: [{ type: "text", text: forecast.summary }],
structuredContent: forecast,
};
}
return {
content: [{ type: "text", text: `Unknown tool: ${name}` }],
isError: true,
};
});
return server;
}
serveStdio(buildServer);
console.error("Mock MCP Server running on stdio");
// Verify that tools/list was called when the connection closes
process.stdin.on("end", () => {
if (!toolsListCalled) {
@ -42,8 +118,3 @@ process.stdin.on("end", () => {
process.exit(1);
}
});
main().catch((error) => {
console.error("Server error:", error);
process.exit(1);
});

1083
tests/helpers/package-lock.json generated
View file

File diff suppressed because it is too large Load diff

10
tests/helpers/package.json
View file

@ -7,14 +7,16 @@
"build": "tsc",
"clean": "rm -rf build"
},
"dependencies": {
"@modelcontextprotocol/sdk": "^1.26.0"
},
"devDependencies": {
"@types/node": "^22.13.4",
"typescript": "^5.7.3"
},
"engines": {
"node": ">=16.0.0"
"node": ">=20.0.0"
},
"dependencies": {
"@modelcontextprotocol/client": "^2.0.0",
"@modelcontextprotocol/server": "^2.0.0",
"zod": "^4.4.3"
}
}

52
tests/smoke-test.sh
View file

@ -38,57 +38,79 @@ run_test() {
# Test: Python weather server
test_weather_server_python() {
check_dependency uv
check_dependency uv || return 1
local server_dir="${PROJECT_ROOT}/weather-server-python"
node "${TEST_CLIENT}" uv --directory "${server_dir}" run weather.py
}
# Test: TypeScript weather server
test_weather_server_typescript() {
check_dependency node
check_dependency npm
check_dependency node || return 1
check_dependency npm || return 1
local server_dir="${PROJECT_ROOT}/weather-server-typescript"
ensure_built "${server_dir}"
ensure_built "${server_dir}" || return 1
node "${TEST_CLIENT}" node "${server_dir}/build/index.js"
}
# Test: Rust weather server
test_weather_server_rust() {
check_dependency cargo
check_dependency cargo || return 1
local server_dir="${PROJECT_ROOT}/weather-server-rust"
ensure_built "${server_dir}"
ensure_built "${server_dir}" || return 1
# Determine which binary to use
if [ -f "${server_dir}/target/release/weather" ]; then
local server_bin="${server_dir}/target/release/weather"
else
local server_bin="${server_dir}/target/debug/weather"
fi
local server_bin
server_bin=$(resolve_binary "${server_dir}/target/release/weather" \
|| resolve_binary "${server_dir}/target/debug/weather") || {
print_error "no weather binary found in ${server_dir}/target"
return 1
}
node "${TEST_CLIENT}" "${server_bin}"
}
# Test: Go weather server
test_weather_server_go() {
check_dependency go || return 1
local server_dir="${PROJECT_ROOT}/weather-server-go"
ensure_built "${server_dir}" || return 1
local server_bin
server_bin=$(resolve_binary "${server_dir}/server") || {
print_error "no server binary found in ${server_dir}"
return 1
}
node "${TEST_CLIENT}" "${server_bin}"
}
# Test: Python MCP client
test_mcp_client_python() {
check_dependency uv
check_dependency uv || return 1
local client_dir="${PROJECT_ROOT}/mcp-client-python"
uv --directory "${client_dir}" run python "${client_dir}/client.py" "${MOCK_SERVER}" >/dev/null 2>&1
}
# Test: TypeScript MCP client
test_mcp_client_typescript() {
check_dependency node
check_dependency npm
check_dependency node || return 1
check_dependency npm || return 1
local client_dir="${PROJECT_ROOT}/mcp-client-typescript"
ensure_built "${client_dir}"
ensure_built "${client_dir}" || return 1
node "${client_dir}/build/index.js" "${MOCK_SERVER}" >/dev/null 2>&1
}
# Run all tests
#
# All four servers are covered. The Go and Rust clients are not: on main both
# abort when no .env file is present, so they cannot be driven without
# credentials. Making them start credential-free is a change in their own
# directories, so their coverage lands with those changes rather than here.
print_header "Running smoke tests"
run_test "weather-server-python" test_weather_server_python
run_test "weather-server-typescript" test_weather_server_typescript
run_test "weather-server-rust" test_weather_server_rust
run_test "weather-server-go" test_weather_server_go
run_test "mcp-client-python" test_mcp_client_python
run_test "mcp-client-typescript" test_mcp_client_typescript

74
tests/utils.sh
View file

@ -30,12 +30,39 @@ command_exists() {
command -v "$1" >/dev/null 2>&1
}
# Check dependency and exit if not found
# Check dependency and fail the calling test if not found.
#
# Returns rather than exits: a missing toolchain should fail its own test and
# let the rest of the suite run, not abort the whole script before the summary
# is printed.
check_dependency() {
local cmd=$1
if ! command_exists "${cmd}"; then
print_error "${cmd} is not installed"
exit 1
return 1
fi
}
# The executable suffix for this platform: ".exe" under MSYS/Git Bash, empty
# elsewhere. Windows will not run a binary without it, whatever the file is
# actually named.
exe_suffix() {
case "$(uname -s)" in
MINGW* | MSYS* | CYGWIN*) echo ".exe" ;;
*) echo "" ;;
esac
}
# Echo the path to a built binary, accounting for the .exe suffix on Windows.
# Prints nothing and returns 1 when neither exists.
resolve_binary() {
local path=$1
if [ -f "${path}$(exe_suffix)" ]; then
echo "${path}$(exe_suffix)"
elif [ -f "${path}" ]; then
echo "${path}"
else
return 1
fi
}
@ -56,31 +83,58 @@ ensure_helpers_built() {
if [ ! -f "${TEST_CLIENT}" ] || [ ! -f "${MOCK_SERVER}" ]; then
print_error "Test helpers not built"
print_header "Building test helpers..."
cd "${PROJECT_ROOT}/tests/helpers" || exit 1
cd "${PROJECT_ROOT}/tests/helpers" || return 1
npm install >/dev/null 2>&1
npm run build >/dev/null 2>&1
cd - >/dev/null || exit 1
cd - >/dev/null || return 1
print_success "Test helpers built"
fi
}
# Ensure a project directory is built (TypeScript/Rust)
# Run a build step, showing its output only if it fails.
#
# Builds used to be run with `>/dev/null 2>&1` unconditionally, which hid
# compile errors: a failed build left no binary behind and the smoke test
# reported the downstream `spawn ... ENOENT` instead of the actual error.
run_build() {
local what=$1
shift
local output
if ! output=$("$@" 2>&1); then
print_error "${what} failed"
echo "${output}"
return 1
fi
}
# Ensure a project directory is built (TypeScript/Rust/Go)
#
# Returns rather than exits, for the same reason check_dependency does: one
# project failing to build should fail its own test, not abort the suite
# before the summary is printed.
ensure_built() {
local dir=$1
cd "${dir}" || exit 1
cd "${dir}" || return 1
# Install npm dependencies if needed
if [ -f "package.json" ] && [ ! -d "node_modules" ]; then
npm install >/dev/null 2>&1
run_build "npm install in ${dir}" npm install || return 1
fi
# Build TypeScript if needed
if [ -f "tsconfig.json" ] && [ ! -f "build/index.js" ]; then
npm run build >/dev/null 2>&1
run_build "npm run build in ${dir}" npm run build || return 1
fi
# Build Rust if needed
if [ -f "Cargo.toml" ] && [ ! -f "target/release/weather" ] && [ ! -f "target/debug/weather" ]; then
cargo build --release >/dev/null 2>&1
if [ -f "Cargo.toml" ] \
&& ! resolve_binary "target/release/weather" >/dev/null \
&& ! resolve_binary "target/debug/weather" >/dev/null; then
run_build "cargo build in ${dir}" cargo build --release || return 1
fi
# Build Go if needed
if [ -f "go.mod" ] && ! resolve_binary "server" >/dev/null; then
run_build "go build in ${dir}" go build -o "server$(exe_suffix)" . || return 1
fi
}