tests: add Playwright E2E tests with screenshot golden testing (+ fix examples session handling) (#115)

* Add Playwright E2E tests with screenshot golden testing

- Add E2E tests for all 8 MCP server examples
- Screenshot golden images for visual regression testing
- CI workflow for running E2E tests
- npm scripts: test:e2e, test:e2e:update, test:e2e:ui

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Add default demo code for Three.js example

When no code is provided, show a rotating green cube demo.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Add E2E testing documentation to CONTRIBUTING.md

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Update Three.js golden screenshot with 3D cube

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Add explicit permissions to CI workflow

Set minimal read-only permissions for security best practices.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Fix test.setTimeout() to be inside describe blocks

Playwright requires setTimeout to be called within a test or describe block.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Simplify E2E tests - remove excessive timeouts

- Remove explicit setTimeout calls (default 30s is sufficient)
- Replace waitForTimeout(5000/6000) with proper waitForAppLoad()
- Wait for inner iframe visibility instead of fixed delays
- Keep only 500ms stabilization for screenshot animations

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Auto-generate missing snapshots in CI

Use updateSnapshots: 'missing' in CI to handle cross-platform
screenshot differences (macOS vs Linux).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Use platform-agnostic golden screenshots

- Remove -chromium-darwin suffix from snapshot filenames
- Configure snapshotPathTemplate for cross-platform compatibility
- Increase tolerance to 5% for rendering differences between platforms

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Refactor tests to use forEach instead of for-of loop

May help with Playwright test discovery issues in CI.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Use npm ci and list reporter in CI

- Use npm ci for exact package versions
- Use --reporter=list instead of html to avoid potential re-evaluation issues
- Only upload test-results on failure

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Exclude e2e tests from bun test

Run bun test only on src/ directory to avoid running Playwright tests
with Bun's test runner.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Fix prettier formatting

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Add interaction tests for basic server apps

Test that clicking buttons in the MCP App triggers the corresponding
host callbacks:
- Send Message → host logs "Message from MCP App"
- Send Log → host logs "Log message from MCP App"
- Open Link → host logs "Open link request"

Tests both React and Vanilla JS implementations.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Mask dynamic content in E2E screenshot tests

Add Playwright mask option to handle servers with dynamic/random content:
- basic-react/vanillajs: mask server time display
- system-monitor: mask CPU chart, memory stats, uptime
- cohort-heatmap: mask heatmap grid (random data)
- customer-segmentation: mask scatter chart (random data)

This addresses PR feedback about handling examples with non-deterministic
output. Masking replaces the previous 10% tolerance with proper exclusion
of dynamic elements, allowing tighter 1% tolerance for the rest of the UI.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Add wiki-explorer to E2E tests with default URL param

- Add default URL "https://en.wikipedia.org/wiki/Model_Context_Protocol" to
  wiki-explorer-server's get-first-degree-links tool inputSchema
- Update basic-host to automatically populate input field with tool defaults
  extracted from inputSchema.properties
- Add wiki-explorer-server to E2E test suite with dynamic masking for the
  force-directed graph

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Use .default() for threejs tool schema instead of .optional()

This exposes the default code and height values in the JSON Schema,
allowing basic-host to auto-populate the input field with defaults.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Enable parallel E2E tests with timeouts and canvas masking

- Enable fullyParallel with 4 workers locally (2 in CI)
- Add 30s timeout per test
- Mask threejs canvas for stable screenshots
- Update snapshots to reflect default values in input fields

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Add pre-commit check for private registry URLs in package-lock.json

Same check as in CI to catch issues before push.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Increase screenshot diff tolerance to 6% for cross-platform rendering

Font rendering differs between macOS and Linux, causing ~5% pixel differences.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Revert pre-commit artifactory check (moved to separate PR #133)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Format threejs server.ts

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Update CONTRIBUTING.md to reflect platform-agnostic screenshots

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(e2e): use factory pattern for MCP servers to support parallel connections

McpServer only supports one transport at a time. When multiple browser
contexts connected in parallel, calling server.connect(transport) for
each session overwrote the previous transport's callbacks, causing
connection corruption and test timeouts.

Changes:
- server-utils.ts: accept factory function instead of single instance
- All 9 example servers: wrap server creation in createServer() function
- playwright.config.ts: re-enable parallel execution (4 workers)
- servers.spec.ts: add toBeEnabled wait for server connection
- Update screenshot baselines for basic-vanillajs and cohort-heatmap

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Olivier Chafik 2025-12-13 20:57:27 +00:00 • committed by GitHub
commit c9e37204de
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

View file

@ -36,7 +36,10 @@
"build": "npm run generate:schemas && bun build.bun.ts",
"prepack": "npm run build",
"build:all": "npm run build && npm run examples:build",
"test": "bun test",
"test": "bun test src",
"test:e2e": "playwright test",
"test:e2e:update": "playwright test --update-snapshots",
"test:e2e:ui": "playwright test --ui",
"examples:build": "bun examples/run-all.ts build",
"examples:start": "NODE_ENV=development npm run build && bun examples/run-all.ts start",
"examples:dev": "NODE_ENV=development bun examples/run-all.ts dev",
@ -50,10 +53,11 @@
},
"author": "Olivier Chafik",
"devDependencies": {
"@playwright/test": "^1.52.0",
"@types/bun": "^1.3.2",
"bun": "^1.3.2",
"@types/react": "^19.2.2",
"@types/react-dom": "^19.2.2",
"bun": "^1.3.2",
"concurrently": "^9.2.1",
"cors": "^2.8.5",
"esbuild": "^0.25.12",
@ -78,8 +82,8 @@
"optionalDependencies": {
"@rollup/rollup-darwin-arm64": "^4.53.3",
"@rollup/rollup-darwin-x64": "^4.53.3",
"@rollup/rollup-linux-x64-gnu": "^4.53.3",
"@rollup/rollup-linux-arm64-gnu": "^4.53.3",
"@rollup/rollup-linux-x64-gnu": "^4.53.3",
"@rollup/rollup-win32-x64-msvc": "^4.53.3"
}
}