From c9e37204de2e436076175920e191c303380710a1 Mon Sep 17 00:00:00 2001 From: Olivier Chafik Date: Sat, 13 Dec 2025 20:57:27 +0000 Subject: [PATCH] tests: add Playwright E2E tests with screenshot golden testing (+ fix examples session handling) (#115) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Add Playwright E2E tests with screenshot golden testing - Add E2E tests for all 8 MCP server examples - Screenshot golden images for visual regression testing - CI workflow for running E2E tests - npm scripts: test:e2e, test:e2e:update, test:e2e:ui 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Add default demo code for Three.js example When no code is provided, show a rotating green cube demo. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Add E2E testing documentation to CONTRIBUTING.md 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Update Three.js golden screenshot with 3D cube 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Add explicit permissions to CI workflow Set minimal read-only permissions for security best practices. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Fix test.setTimeout() to be inside describe blocks Playwright requires setTimeout to be called within a test or describe block. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Simplify E2E tests - remove excessive timeouts - Remove explicit setTimeout calls (default 30s is sufficient) - Replace waitForTimeout(5000/6000) with proper waitForAppLoad() - Wait for inner iframe visibility instead of fixed delays - Keep only 500ms stabilization for screenshot animations 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Auto-generate missing snapshots in CI Use updateSnapshots: 'missing' in CI to handle cross-platform screenshot differences (macOS vs Linux). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Use platform-agnostic golden screenshots - Remove -chromium-darwin suffix from snapshot filenames - Configure snapshotPathTemplate for cross-platform compatibility - Increase tolerance to 5% for rendering differences between platforms 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Refactor tests to use forEach instead of for-of loop May help with Playwright test discovery issues in CI. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Use npm ci and list reporter in CI - Use npm ci for exact package versions - Use --reporter=list instead of html to avoid potential re-evaluation issues - Only upload test-results on failure 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Exclude e2e tests from bun test Run bun test only on src/ directory to avoid running Playwright tests with Bun's test runner. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Fix prettier formatting 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Add interaction tests for basic server apps Test that clicking buttons in the MCP App triggers the corresponding host callbacks: - Send Message → host logs "Message from MCP App" - Send Log → host logs "Log message from MCP App" - Open Link → host logs "Open link request" Tests both React and Vanilla JS implementations. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Mask dynamic content in E2E screenshot tests Add Playwright mask option to handle servers with dynamic/random content: - basic-react/vanillajs: mask server time display - system-monitor: mask CPU chart, memory stats, uptime - cohort-heatmap: mask heatmap grid (random data) - customer-segmentation: mask scatter chart (random data) This addresses PR feedback about handling examples with non-deterministic output. Masking replaces the previous 10% tolerance with proper exclusion of dynamic elements, allowing tighter 1% tolerance for the rest of the UI. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Add wiki-explorer to E2E tests with default URL param - Add default URL "https://en.wikipedia.org/wiki/Model_Context_Protocol" to wiki-explorer-server's get-first-degree-links tool inputSchema - Update basic-host to automatically populate input field with tool defaults extracted from inputSchema.properties - Add wiki-explorer-server to E2E test suite with dynamic masking for the force-directed graph 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Use .default() for threejs tool schema instead of .optional() This exposes the default code and height values in the JSON Schema, allowing basic-host to auto-populate the input field with defaults. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Enable parallel E2E tests with timeouts and canvas masking - Enable fullyParallel with 4 workers locally (2 in CI) - Add 30s timeout per test - Mask threejs canvas for stable screenshots - Update snapshots to reflect default values in input fields 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Add pre-commit check for private registry URLs in package-lock.json Same check as in CI to catch issues before push. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Increase screenshot diff tolerance to 6% for cross-platform rendering Font rendering differs between macOS and Linux, causing ~5% pixel differences. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Revert pre-commit artifactory check (moved to separate PR #133) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Format threejs server.ts 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * Update CONTRIBUTING.md to reflect platform-agnostic screenshots 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude * fix(e2e): use factory pattern for MCP servers to support parallel connections McpServer only supports one transport at a time. When multiple browser contexts connected in parallel, calling server.connect(transport) for each session overwrote the previous transport's callbacks, causing connection corruption and test timeouts. Changes: - server-utils.ts: accept factory function instead of single instance - All 9 example servers: wrap server creation in createServer() function - playwright.config.ts: re-enable parallel execution (4 workers) - servers.spec.ts: add toBeEnabled wait for server connection - Update screenshot baselines for basic-vanillajs and cohort-heatmap 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 --------- Co-authored-by: Claude --- .github/workflows/ci.yml | 32 + .gitignore | 5 + CONTRIBUTING.md | 39 + examples/basic-host/src/index.tsx | 31 +- examples/basic-server-react/server.ts | 37 +- examples/basic-server-vanillajs/server.ts | 37 +- examples/budget-allocator-server/server.ts | 132 +- examples/cohort-heatmap-server/server.ts | 18 +- .../customer-segmentation-server/server.ts | 101 +- examples/scenario-modeler-server/server.ts | 129 +- examples/shared/server-utils.ts | 62 +- examples/system-monitor-server/server.ts | 18 +- examples/threejs-server/server.ts | 60 +- examples/threejs-server/src/threejs-app.tsx | 28 +- examples/wiki-explorer-server/server.ts | 24 +- package-lock.json | 2435 ++++++++++++----- package.json | 10 +- playwright.config.ts | 46 + tests/e2e/servers.spec.ts | 182 ++ .../servers.spec.ts-snapshots/basic-react.png | Bin 0 -> 49728 bytes .../basic-vanillajs.png | Bin 0 -> 48776 bytes .../budget-allocator.png | Bin 0 -> 68213 bytes .../cohort-heatmap.png | Bin 0 -> 39849 bytes .../customer-segmentation.png | Bin 0 -> 39306 bytes .../host-initial.png | Bin 0 -> 16120 bytes .../scenario-modeler.png | Bin 0 -> 71454 bytes .../system-monitor.png | Bin 0 -> 37642 bytes .../e2e/servers.spec.ts-snapshots/threejs.png | Bin 0 -> 37725 bytes .../wiki-explorer.png | Bin 0 -> 25253 bytes 29 files changed, 2507 insertions(+), 919 deletions(-) create mode 100644 playwright.config.ts create mode 100644 tests/e2e/servers.spec.ts create mode 100644 tests/e2e/servers.spec.ts-snapshots/basic-react.png create mode 100644 tests/e2e/servers.spec.ts-snapshots/basic-vanillajs.png create mode 100644 tests/e2e/servers.spec.ts-snapshots/budget-allocator.png create mode 100644 tests/e2e/servers.spec.ts-snapshots/cohort-heatmap.png create mode 100644 tests/e2e/servers.spec.ts-snapshots/customer-segmentation.png create mode 100644 tests/e2e/servers.spec.ts-snapshots/host-initial.png create mode 100644 tests/e2e/servers.spec.ts-snapshots/scenario-modeler.png create mode 100644 tests/e2e/servers.spec.ts-snapshots/system-monitor.png create mode 100644 tests/e2e/servers.spec.ts-snapshots/threejs.png create mode 100644 tests/e2e/servers.spec.ts-snapshots/wiki-explorer.png diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9cd3266..dd795a9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -6,6 +6,9 @@ on: pull_request: branches: [main] +permissions: + contents: read + jobs: build: runs-on: ubuntu-latest @@ -35,3 +38,32 @@ jobs: - run: npm test - run: npm run prettier + + e2e: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: oven-sh/setup-bun@v2 + with: + bun-version: latest + + - uses: actions/setup-node@v4 + with: + node-version: "20" + + - run: npm ci + + - name: Install Playwright browsers + run: npx playwright install --with-deps chromium + + - name: Run E2E tests + run: npx playwright test --reporter=list + + - name: Upload test results + uses: actions/upload-artifact@v4 + if: failure() + with: + name: test-results + path: test-results/ + retention-days: 7 diff --git a/.gitignore b/.gitignore index dc9ad99..1d78df5 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,8 @@ bun.lockb .vscode/ docs/api/ tmp/ +intermediate-findings/ + +# Playwright +playwright-report/ +test-results/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 310bc20..81f776a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -40,6 +40,45 @@ Or build and run examples: npm run examples:start ``` +## Testing + +### Unit Tests + +Run unit tests with Bun: + +```bash +npm test +``` + +### E2E Tests + +E2E tests use Playwright to verify all example servers work correctly with screenshot comparisons. + +```bash +# Run all E2E tests +npm run test:e2e + +# Run a specific server's tests +npm run test:e2e -- --grep "Budget Allocator" + +# Run tests in interactive UI mode +npm run test:e2e:ui +``` + +### Updating Golden Screenshots + +When UI changes are intentional, update the golden screenshots: + +```bash +# Update all screenshots +npm run test:e2e:update + +# Update screenshots for a specific server +npm run test:e2e:update -- --grep "Three.js" +``` + +**Note**: Golden screenshots are platform-agnostic. Tests use canvas masking and tolerance thresholds to handle minor cross-platform rendering differences. + ## Code of Conduct This project follows our [Code of Conduct](CODE_OF_CONDUCT.md). Please review it before contributing. diff --git a/examples/basic-host/src/index.tsx b/examples/basic-host/src/index.tsx index 1288a34..afe51e5 100644 --- a/examples/basic-host/src/index.tsx +++ b/examples/basic-host/src/index.tsx @@ -1,9 +1,30 @@ +import type { Tool } from "@modelcontextprotocol/sdk/types.js"; import { Component, type ErrorInfo, type ReactNode, StrictMode, Suspense, use, useEffect, useMemo, useRef, useState } from "react"; import { createRoot } from "react-dom/client"; import { callTool, connectToServer, hasAppHtml, initializeApp, loadSandboxProxy, log, newAppBridge, type ServerInfo, type ToolCallInfo } from "./implementation"; import styles from "./index.module.css"; +/** + * Extract default values from a tool's JSON Schema inputSchema. + * Returns a formatted JSON string with defaults, or "{}" if none found. + */ +function getToolDefaults(tool: Tool | undefined): string { + if (!tool?.inputSchema?.properties) return "{}"; + + const defaults: Record = {}; + for (const [key, prop] of Object.entries(tool.inputSchema.properties)) { + if (prop && typeof prop === "object" && "default" in prop) { + defaults[key] = prop.default; + } + } + + return Object.keys(defaults).length > 0 + ? JSON.stringify(defaults, null, 2) + : "{}"; +} + + // Host passes serversPromise to CallToolPanel interface HostProps { serversPromise: Promise; @@ -74,6 +95,14 @@ function CallToolPanel({ serversPromise, addToolCall }: CallToolPanelProps) { setSelectedServer(server); const [firstTool] = server.tools.keys(); setSelectedTool(firstTool ?? ""); + // Set input JSON to tool defaults (if any) + setInputJson(getToolDefaults(server.tools.get(firstTool ?? ""))); + }; + + const handleToolSelect = (toolName: string) => { + setSelectedTool(toolName); + // Set input JSON to tool defaults (if any) + setInputJson(getToolDefaults(selectedServer?.tools.get(toolName))); }; const handleSubmit = () => { @@ -96,7 +125,7 @@ function CallToolPanel({ serversPromise, addToolCall }: CallToolPanelProps) {