feat: add --expected-failures baseline and composite GitHub Action (#113)

Add support for SDKs that don't yet pass all conformance tests to still run conformance in CI without failing, by specifying a YAML baseline file of known/expected failures. CLI changes: - Add --expected-failures <path> flag to both server and client commands - Exit 0 if only expected failures occur - Exit 1 if unexpected failures occur (regression) - Exit 1 if an expected failure now passes (stale baseline, must update) GitHub Action (action.yml): - Composite action that builds from source and runs conformance tests - SDKs reference it as: uses: modelcontextprotocol/conformance@<ref> - Inputs: mode, url, command, expected-failures, suite, scenario, etc. New files: - src/expected-failures.ts: YAML loading, baseline evaluation, reporting - src/expected-failures.test.ts: 16 unit tests - action.yml: composite GitHub Action Closes #99
2026-01-22 10:41:04 +00:00 · 2026-01-22 10:41:04 +00:00 · 83ff334e0e
commit 83ff334e0e
parent cc784d96e2
7 changed files with 697 additions and 1 deletions
--- a/README.md
+++ b/README.md
@ -65,6 +65,7 @@ npx @modelcontextprotocol/conformance client --command "<client-command>" --scen
 - `--command` - The command to run your MCP client (can include flags)
 - `--scenario` - The test scenario to run (e.g., "initialize")
 - `--suite` - Run a suite of tests in parallel (e.g., "auth")
+- `--expected-failures <path>` - Path to YAML baseline file of known failures (see [Expected Failures](#expected-failures))
 - `--timeout` - Timeout in milliseconds (default: 30000)
 - `--verbose` - Show verbose output

@ -79,7 +80,10 @@ npx @modelcontextprotocol/conformance server --url <url> [--scenario <scenario>]
 **Options:**

 - `--url` - URL of the server to test
- `--scenario <scenario>` - Test scenario to run (e.g., "server-initialize". Runs all available scenarios by default
+- `--scenario <scenario>` - Test scenario to run (e.g., "server-initialize"). Runs all available scenarios by default
+- `--suite <suite>` - Suite to run: "active" (default), "all", or "pending"
+- `--expected-failures <path>` - Path to YAML baseline file of known failures (see [Expected Failures](#expected-failures))
+- `--verbose` - Show verbose output

 ## Test Results

@ -93,6 +97,91 @@ npx @modelcontextprotocol/conformance server --url <url> [--scenario <scenario>]

 - `checks.json` - Array of conformance check results with pass/fail status

+## Expected Failures
+
+SDKs that don't yet pass all conformance tests can specify a baseline of known failures. This allows running conformance tests in CI without failing, while still catching regressions.
+
+Create a YAML file listing expected failures by mode:
+
+```yaml
+# conformance-baseline.yml
+server:
+  - tools-call-with-progress
+  - resources-subscribe
+client:
+  - sse-retry
+```
+
+Then pass it to the CLI:
+
+```bash
+npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --expected-failures ./conformance-baseline.yml
+```
+
+**Exit code behavior:**
+
+| Scenario Result | In Baseline? | Outcome                                   |
+| --------------- | ------------ | ----------------------------------------- |
+| Fails           | Yes          | Exit 0 — expected failure                 |
+| Fails           | No           | Exit 1 — unexpected regression            |
+| Passes          | Yes          | Exit 1 — stale baseline, remove the entry |
+| Passes          | No           | Exit 0 — normal pass                      |
+
+This ensures:
+
+- CI passes when only known failures occur
+- CI fails on new regressions (unexpected failures)
+- CI fails when a fix lands but the baseline isn't updated (stale entries)
+
+## GitHub Action
+
+This repo provides a composite GitHub Action so SDK repos don't need to write their own conformance scripts.
+
+### Server Testing
+
+```yaml
+steps:
+  - uses: actions/checkout@v4
+
+  # Start your server (SDK-specific)
+  - run: |
+      my-server --port 3001 &
+      timeout 15 bash -c 'until curl -s http://localhost:3001/mcp; do sleep 0.5; done'
+
+  - uses: modelcontextprotocol/conformance@v0.1.11
+    with:
+      mode: server
+      url: http://localhost:3001/mcp
+      expected-failures: ./conformance-baseline.yml # optional
+```
+
+### Client Testing
+
+```yaml
+steps:
+  - uses: actions/checkout@v4
+
+  - uses: modelcontextprotocol/conformance@v0.1.11
+    with:
+      mode: client
+      command: 'python tests/conformance/client.py'
+      expected-failures: ./conformance-baseline.yml # optional
+```
+
+### Action Inputs
+
+| Input               | Required    | Description                                     |
+| ------------------- | ----------- | ----------------------------------------------- |
+| `mode`              | Yes         | `server` or `client`                            |
+| `url`               | Server mode | URL of the server to test                       |
+| `command`           | Client mode | Command to run the client under test            |
+| `expected-failures` | No          | Path to YAML baseline file                      |
+| `suite`             | No          | Test suite to run                               |
+| `scenario`          | No          | Run a single scenario by name                   |
+| `timeout`           | No          | Timeout in ms for client tests (default: 30000) |
+| `verbose`           | No          | Show verbose output (default: false)            |
+| `node-version`      | No          | Node.js version (default: 20)                   |
+
 ## Example Clients

 - `examples/clients/typescript/everything-client.ts` - Single client that handles all scenarios based on scenario name (recommended)
--- a/action.yml
+++ b/action.yml
@ -0,0 +1,97 @@
+name: 'MCP Conformance Tests'
+description: 'Run MCP conformance tests against a server or client implementation'
+inputs:
+  mode:
+    description: 'Test mode: "server" or "client"'
+    required: true
+  url:
+    description: 'Server URL to test against (required for server mode)'
+    required: false
+  command:
+    description: 'Command to run the client under test (required for client mode)'
+    required: false
+  expected-failures:
+    description: 'Path to YAML file listing expected failures (baseline)'
+    required: false
+  suite:
+    description: 'Test suite to run (server: "active"|"all"|"pending", client: "all"|"auth"|"metadata"|"sep-835")'
+    required: false
+  scenario:
+    description: 'Run a single scenario by name'
+    required: false
+  timeout:
+    description: 'Timeout in milliseconds for client tests (default: 30000)'
+    required: false
+    default: '30000'
+  verbose:
+    description: 'Show verbose output (default: false)'
+    required: false
+    default: 'false'
+  node-version:
+    description: 'Node.js version to use (default: 20)'
+    required: false
+    default: '20'
+runs:
+  using: 'composite'
+  steps:
+    - name: Setup Node.js
+      uses: actions/setup-node@v4
+      with:
+        node-version: ${{ inputs.node-version }}
+
+    - name: Build conformance tests
+      shell: bash
+      run: |
+        cd "${{ github.action_path }}"
+        npm ci
+        npm run build
+
+    - name: Run conformance tests
+      shell: bash
+      run: |
+        CONFORMANCE="${{ github.action_path }}/dist/index.js"
+
+        # Build the command arguments
+        ARGS="${{ inputs.mode }}"
+
+        # Mode-specific required options
+        if [ "${{ inputs.mode }}" = "server" ]; then
+          if [ -z "${{ inputs.url }}" ]; then
+            echo "::error::The 'url' input is required for server mode"
+            exit 1
+          fi
+          ARGS="${ARGS} --url ${{ inputs.url }}"
+        elif [ "${{ inputs.mode }}" = "client" ]; then
+          if [ -z "${{ inputs.command }}" ]; then
+            echo "::error::The 'command' input is required for client mode"
+            exit 1
+          fi
+          ARGS="${ARGS} --command '${{ inputs.command }}'"
+        else
+          echo "::error::Invalid mode '${{ inputs.mode }}'. Must be 'server' or 'client'."
+          exit 1
+        fi
+
+        # Optional arguments
+        if [ -n "${{ inputs.expected-failures }}" ]; then
+          ARGS="${ARGS} --expected-failures ${{ inputs.expected-failures }}"
+        fi
+
+        if [ -n "${{ inputs.suite }}" ]; then
+          ARGS="${ARGS} --suite ${{ inputs.suite }}"
+        fi
+
+        if [ -n "${{ inputs.scenario }}" ]; then
+          ARGS="${ARGS} --scenario ${{ inputs.scenario }}"
+        fi
+
+        if [ "${{ inputs.mode }}" = "client" ]; then
+          ARGS="${ARGS} --timeout ${{ inputs.timeout }}"
+        fi
+
+        if [ "${{ inputs.verbose }}" = "true" ]; then
+          ARGS="${ARGS} --verbose"
+        fi
+
+        echo "Running: node ${CONFORMANCE} ${ARGS}"
+        eval "node ${CONFORMANCE} ${ARGS}"
--- a/package-lock.json
+++ b/package-lock.json
@ -14,6 +14,7 @@
        "eventsource-parser": "^3.0.6",
        "express": "^5.1.0",
        "jose": "^6.1.2",
+        "yaml": "^2.8.2",
        "zod": "^3.25.76"
      },
      "bin": {
@ -5645,6 +5646,21 @@
      "integrity": "sha1-tSQ9jz7BqjXxNkYFvA0QNuMKtp8=",
      "license": "ISC"
    },
+    "node_modules/yaml": {
+      "version": "2.8.2",
+      "resolved": "https://registry.npmjs.org/yaml/-/yaml-2.8.2.tgz",
+      "integrity": "sha512-mplynKqc1C2hTVYxd0PU2xQAc22TI1vShAYGksCCfxbn/dFwnHTNi1bvYsBTkhdUNtGIf5xNOg938rrSSYvS9A==",
+      "license": "ISC",
+      "bin": {
+        "yaml": "bin.mjs"
+      },
+      "engines": {
+        "node": ">= 14.6"
+      },
+      "funding": {
+        "url": "https://github.com/sponsors/eemeli"
+      }
+    },
    "node_modules/yocto-queue": {
      "version": "0.1.0",
      "resolved": "https://registry.npmjs.org/yocto-queue/-/yocto-queue-0.1.0.tgz",
--- a/package.json
+++ b/package.json
@ -50,6 +50,7 @@
    "eventsource-parser": "^3.0.6",
    "express": "^5.1.0",
    "jose": "^6.1.2",
+    "yaml": "^2.8.2",
    "zod": "^3.25.76"
  }
 }
--- a/src/expected-failures.test.ts
+++ b/src/expected-failures.test.ts
@ -0,0 +1,261 @@
+import { describe, it, expect, beforeEach, afterEach } from 'vitest';
+import { promises as fs } from 'fs';
+import path from 'path';
+import os from 'os';
+import { loadExpectedFailures, evaluateBaseline } from './expected-failures';
+import { ConformanceCheck } from './types';
+
+function makeCheck(
+  id: string,
+  status: 'SUCCESS' | 'FAILURE' | 'WARNING' | 'SKIPPED' | 'INFO'
+): ConformanceCheck {
+  return {
+    id,
+    name: id,
+    description: `Check ${id}`,
+    status,
+    timestamp: new Date().toISOString()
+  };
+}
+
+describe('loadExpectedFailures', () => {
+  let tmpDir: string;
+
+  beforeEach(async () => {
+    tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), 'conformance-test-'));
+  });
+
+  afterEach(async () => {
+    await fs.rm(tmpDir, { recursive: true });
+  });
+
+  it('loads a valid YAML file with both server and client entries', async () => {
+    const filePath = path.join(tmpDir, 'baseline.yml');
+    await fs.writeFile(
+      filePath,
+      `server:
+  - tools-call-with-progress
+  - resources-subscribe
+client:
+  - sse-retry
+  - auth/basic-dcr
+`
+    );
+
+    const result = await loadExpectedFailures(filePath);
+    expect(result.server).toEqual([
+      'tools-call-with-progress',
+      'resources-subscribe'
+    ]);
+    expect(result.client).toEqual(['sse-retry', 'auth/basic-dcr']);
+  });
+
+  it('loads a file with only server entries', async () => {
+    const filePath = path.join(tmpDir, 'baseline.yml');
+    await fs.writeFile(
+      filePath,
+      `server:
+  - tools-call-with-progress
+`
+    );
+
+    const result = await loadExpectedFailures(filePath);
+    expect(result.server).toEqual(['tools-call-with-progress']);
+    expect(result.client).toBeUndefined();
+  });
+
+  it('handles an empty file', async () => {
+    const filePath = path.join(tmpDir, 'baseline.yml');
+    await fs.writeFile(filePath, '');
+
+    const result = await loadExpectedFailures(filePath);
+    expect(result).toEqual({});
+  });
+
+  it('throws on invalid structure (array at top level)', async () => {
+    const filePath = path.join(tmpDir, 'baseline.yml');
+    await fs.writeFile(filePath, '- foo\n- bar\n');
+
+    await expect(loadExpectedFailures(filePath)).rejects.toThrow(
+      'expected an object'
+    );
+  });
+
+  it('throws if server is not an array', async () => {
+    const filePath = path.join(tmpDir, 'baseline.yml');
+    await fs.writeFile(filePath, 'server: not-an-array\n');
+
+    await expect(loadExpectedFailures(filePath)).rejects.toThrow(
+      "'server' must be an array"
+    );
+  });
+
+  it('throws if client is not an array', async () => {
+    const filePath = path.join(tmpDir, 'baseline.yml');
+    await fs.writeFile(filePath, 'client: 123\n');
+
+    await expect(loadExpectedFailures(filePath)).rejects.toThrow(
+      "'client' must be an array"
+    );
+  });
+
+  it('throws on missing file', async () => {
+    await expect(
+      loadExpectedFailures('/nonexistent/path.yml')
+    ).rejects.toThrow();
+  });
+});
+
+describe('evaluateBaseline', () => {
+  it('returns exit 0 when all failures are expected', () => {
+    const results = [
+      {
+        scenario: 'scenario-a',
+        checks: [makeCheck('check1', 'SUCCESS'), makeCheck('check2', 'FAILURE')]
+      },
+      {
+        scenario: 'scenario-b',
+        checks: [makeCheck('check3', 'FAILURE')]
+      }
+    ];
+
+    const result = evaluateBaseline(results, ['scenario-a', 'scenario-b']);
+    expect(result.exitCode).toBe(0);
+    expect(result.expectedFailures).toEqual(['scenario-a', 'scenario-b']);
+    expect(result.unexpectedFailures).toEqual([]);
+    expect(result.staleEntries).toEqual([]);
+  });
+
+  it('returns exit 0 when no failures at all and no baseline', () => {
+    const results = [
+      {
+        scenario: 'scenario-a',
+        checks: [makeCheck('check1', 'SUCCESS')]
+      }
+    ];
+
+    const result = evaluateBaseline(results, []);
+    expect(result.exitCode).toBe(0);
+    expect(result.expectedFailures).toEqual([]);
+    expect(result.unexpectedFailures).toEqual([]);
+    expect(result.staleEntries).toEqual([]);
+  });
+
+  it('returns exit 1 for unexpected failures', () => {
+    const results = [
+      {
+        scenario: 'scenario-a',
+        checks: [makeCheck('check1', 'FAILURE')]
+      }
+    ];
+
+    const result = evaluateBaseline(results, []);
+    expect(result.exitCode).toBe(1);
+    expect(result.unexpectedFailures).toEqual(['scenario-a']);
+  });
+
+  it('returns exit 1 for stale baseline entries (scenario now passes)', () => {
+    const results = [
+      {
+        scenario: 'scenario-a',
+        checks: [makeCheck('check1', 'SUCCESS')]
+      }
+    ];
+
+    const result = evaluateBaseline(results, ['scenario-a']);
+    expect(result.exitCode).toBe(1);
+    expect(result.staleEntries).toEqual(['scenario-a']);
+  });
+
+  it('returns exit 1 when both stale and unexpected failures exist', () => {
+    const results = [
+      {
+        scenario: 'scenario-a',
+        checks: [makeCheck('check1', 'SUCCESS')] // was expected to fail but passes
+      },
+      {
+        scenario: 'scenario-b',
+        checks: [makeCheck('check2', 'FAILURE')] // unexpected failure
+      }
+    ];
+
+    const result = evaluateBaseline(results, ['scenario-a']);
+    expect(result.exitCode).toBe(1);
+    expect(result.staleEntries).toEqual(['scenario-a']);
+    expect(result.unexpectedFailures).toEqual(['scenario-b']);
+  });
+
+  it('handles warnings as failures', () => {
+    const results = [
+      {
+        scenario: 'scenario-a',
+        checks: [makeCheck('check1', 'WARNING')]
+      }
+    ];
+
+    // Not in baseline → unexpected
+    const result1 = evaluateBaseline(results, []);
+    expect(result1.exitCode).toBe(1);
+    expect(result1.unexpectedFailures).toEqual(['scenario-a']);
+
+    // In baseline → expected
+    const result2 = evaluateBaseline(results, ['scenario-a']);
+    expect(result2.exitCode).toBe(0);
+    expect(result2.expectedFailures).toEqual(['scenario-a']);
+  });
+
+  it('ignores baseline entries for scenarios not in the run', () => {
+    const results = [
+      {
+        scenario: 'scenario-a',
+        checks: [makeCheck('check1', 'SUCCESS')]
+      }
+    ];
+
+    // scenario-z is in baseline but not in the results - should not be stale
+    const result = evaluateBaseline(results, ['scenario-z']);
+    expect(result.exitCode).toBe(0);
+    expect(result.staleEntries).toEqual([]);
+  });
+
+  it('handles mixed expected/unexpected/passing scenarios', () => {
+    const results = [
+      {
+        scenario: 'expected-fail',
+        checks: [makeCheck('c1', 'FAILURE')]
+      },
+      {
+        scenario: 'unexpected-fail',
+        checks: [makeCheck('c2', 'FAILURE')]
+      },
+      {
+        scenario: 'normal-pass',
+        checks: [makeCheck('c3', 'SUCCESS')]
+      }
+    ];
+
+    const result = evaluateBaseline(results, ['expected-fail']);
+    expect(result.exitCode).toBe(1);
+    expect(result.expectedFailures).toEqual(['expected-fail']);
+    expect(result.unexpectedFailures).toEqual(['unexpected-fail']);
+    expect(result.staleEntries).toEqual([]);
+  });
+
+  it('skipped and info checks do not count as failures', () => {
+    const results = [
+      {
+        scenario: 'scenario-a',
+        checks: [
+          makeCheck('c1', 'SUCCESS'),
+          makeCheck('c2', 'SKIPPED'),
+          makeCheck('c3', 'INFO')
+        ]
+      }
+    ];
+
+    // In baseline but passes (only SUCCESS/SKIPPED/INFO) → stale
+    const result = evaluateBaseline(results, ['scenario-a']);
+    expect(result.exitCode).toBe(1);
+    expect(result.staleEntries).toEqual(['scenario-a']);
+  });
+});
--- a/src/expected-failures.ts
+++ b/src/expected-failures.ts
@ -0,0 +1,166 @@
+import { promises as fs } from 'fs';
+import { parse as parseYaml } from 'yaml';
+import { ConformanceCheck } from './types';
+import { COLORS } from './runner/utils';
+
+export interface ExpectedFailures {
+  server?: string[];
+  client?: string[];
+}
+
+export interface BaselineResult {
+  /** Exit code: 0 if only expected failures, 1 if unexpected failures or stale baseline */
+  exitCode: number;
+  /** Scenarios that failed unexpectedly (not in baseline) */
+  unexpectedFailures: string[];
+  /** Scenarios in baseline that now pass (stale entries) */
+  staleEntries: string[];
+  /** Scenarios that failed as expected */
+  expectedFailures: string[];
+}
+
+/**
+ * Load and parse an expected-failures YAML file.
+ *
+ * Expected format:
+ * ```yaml
+ * server:
+ *   - scenario-name-1
+ *   - scenario-name-2
+ * client:
+ *   - scenario-name-3
+ * ```
+ */
+export async function loadExpectedFailures(
+  filePath: string
+): Promise<ExpectedFailures> {
+  const content = await fs.readFile(filePath, 'utf-8');
+  const parsed = parseYaml(content);
+
+  if (parsed === null || parsed === undefined) {
+    return {};
+  }
+
+  if (typeof parsed !== 'object' || Array.isArray(parsed)) {
+    throw new Error(
+      `Invalid expected-failures file: expected an object with 'server' and/or 'client' keys`
+    );
+  }
+
+  const result: ExpectedFailures = {};
+
+  if (parsed.server !== undefined) {
+    if (!Array.isArray(parsed.server)) {
+      throw new Error(
+        `Invalid expected-failures file: 'server' must be an array of scenario names`
+      );
+    }
+    result.server = parsed.server.map(String);
+  }
+
+  if (parsed.client !== undefined) {
+    if (!Array.isArray(parsed.client)) {
+      throw new Error(
+        `Invalid expected-failures file: 'client' must be an array of scenario names`
+      );
+    }
+    result.client = parsed.client.map(String);
+  }
+
+  return result;
+}
+
+/**
+ * Evaluate scenario results against an expected-failures baseline.
+ *
+ * Rules:
+ * - Scenario fails and IS in baseline → expected (ok)
+ * - Scenario fails and is NOT in baseline → unexpected failure (exit 1)
+ * - Scenario passes and IS in baseline → stale entry (exit 1, must update baseline)
+ * - Scenario passes and is NOT in baseline → normal pass (ok)
+ */
+export function evaluateBaseline(
+  results: { scenario: string; checks: ConformanceCheck[] }[],
+  expectedScenarios: string[]
+): BaselineResult {
+  const expectedSet = new Set(expectedScenarios);
+  const unexpectedFailures: string[] = [];
+  const staleEntries: string[] = [];
+  const expectedFailures: string[] = [];
+
+  const seenScenarios = new Set<string>();
+
+  for (const result of results) {
+    seenScenarios.add(result.scenario);
+    const hasFailed =
+      result.checks.some((c) => c.status === 'FAILURE') ||
+      result.checks.some((c) => c.status === 'WARNING');
+    const isExpected = expectedSet.has(result.scenario);
+
+    if (hasFailed && isExpected) {
+      expectedFailures.push(result.scenario);
+    } else if (hasFailed && !isExpected) {
+      unexpectedFailures.push(result.scenario);
+    } else if (!hasFailed && isExpected) {
+      staleEntries.push(result.scenario);
+    }
+    // !hasFailed && !isExpected → normal pass, nothing to do
+  }
+
+  // Also check for baseline entries that reference scenarios not in the run
+  // (these are not stale - they might just not be in this suite)
+
+  const exitCode =
+    unexpectedFailures.length > 0 || staleEntries.length > 0 ? 1 : 0;
+
+  return { exitCode, unexpectedFailures, staleEntries, expectedFailures };
+}
+
+/**
+ * Print baseline evaluation results.
+ */
+export function printBaselineResults(result: BaselineResult): void {
+  if (result.expectedFailures.length > 0) {
+    console.log(
+      `\n${COLORS.YELLOW}Expected failures (in baseline):${COLORS.RESET}`
+    );
+    for (const scenario of result.expectedFailures) {
+      console.log(`  ~ ${scenario}`);
+    }
+  }
+
+  if (result.staleEntries.length > 0) {
+    console.log(
+      `\n${COLORS.RED}Stale baseline entries (now passing - remove from baseline):${COLORS.RESET}`
+    );
+    for (const scenario of result.staleEntries) {
+      console.log(`  ✓ ${scenario}`);
+    }
+  }
+
+  if (result.unexpectedFailures.length > 0) {
+    console.log(
+      `\n${COLORS.RED}Unexpected failures (not in baseline):${COLORS.RESET}`
+    );
+    for (const scenario of result.unexpectedFailures) {
+      console.log(`  ✗ ${scenario}`);
+    }
+  }
+
+  if (result.exitCode === 0) {
+    console.log(
+      `\n${COLORS.GREEN}Baseline check passed: all failures are expected.${COLORS.RESET}`
+    );
+  } else {
+    if (result.staleEntries.length > 0) {
+      console.log(
+        `\n${COLORS.RED}Baseline is stale: update your expected-failures file to remove passing scenarios.${COLORS.RESET}`
+      );
+    }
+    if (result.unexpectedFailures.length > 0) {
+      console.log(
+        `\n${COLORS.RED}Unexpected failures detected: these scenarios are not in your expected-failures baseline.${COLORS.RESET}`
+      );
+    }
+  }
+}
--- a/src/index.ts
+++ b/src/index.ts
@ -20,6 +20,11 @@ import {
 } from './scenarios';
 import { ConformanceCheck } from './types';
 import { ClientOptionsSchema, ServerOptionsSchema } from './schemas';
+import {
+  loadExpectedFailures,
+  evaluateBaseline,
+  printBaselineResults
+} from './expected-failures';
 import packageJson from '../package.json';

 const program = new Command();
@ -39,6 +44,10 @@ program
  .option('--scenario <scenario>', 'Scenario to test')
  .option('--suite <suite>', 'Run a suite of tests in parallel (e.g., "auth")')
  .option('--timeout <ms>', 'Timeout in milliseconds', '30000')
+  .option(
+    '--expected-failures <path>',
+    'Path to YAML file listing expected failures (baseline)'
+  )
  .option('-o, --output-dir <path>', 'Save results to this directory')
  .option('--verbose', 'Show verbose output')
  .action(async (options) => {
@ -149,6 +158,17 @@ program
        console.log(
          `\nTotal: ${totalPassed} passed, ${totalFailed} failed, ${totalWarnings} warnings`
        );
+
+        if (options.expectedFailures) {
+          const expectedFailuresConfig = await loadExpectedFailures(
+            options.expectedFailures
+          );
+          const baselineScenarios = expectedFailuresConfig.client ?? [];
+          const baselineResult = evaluateBaseline(results, baselineScenarios);
+          printBaselineResults(baselineResult);
+          process.exit(baselineResult.exitCode);
+        }
+
        process.exit(totalFailed > 0 || totalWarnings > 0 ? 1 : 0);
      }

@ -183,6 +203,20 @@ program
        verbose,
        result.clientOutput
      );
+
+      if (options.expectedFailures) {
+        const expectedFailuresConfig = await loadExpectedFailures(
+          options.expectedFailures
+        );
+        const baselineScenarios = expectedFailuresConfig.client ?? [];
+        const baselineResult = evaluateBaseline(
+          [{ scenario: validated.scenario, checks: result.checks }],
+          baselineScenarios
+        );
+        printBaselineResults(baselineResult);
+        process.exit(baselineResult.exitCode);
+      }
+
      process.exit(overallFailure ? 1 : 0);
    } catch (error) {
      if (error instanceof ZodError) {
@ -213,6 +247,10 @@ program
    'Suite to run: "active" (default, excludes pending), "all", or "pending"',
    'active'
  )
+  .option(
+    '--expected-failures <path>',
+    'Path to YAML file listing expected failures (baseline)'
+  )
  .option('-o, --output-dir <path>', 'Save results to this directory')
  .option('--verbose', 'Show verbose output (JSON instead of pretty print)')
  .action(async (options) => {
@ -236,6 +274,20 @@ program
          result.scenarioDescription,
          verbose
        );
+
+        if (options.expectedFailures) {
+          const expectedFailuresConfig = await loadExpectedFailures(
+            options.expectedFailures
+          );
+          const baselineScenarios = expectedFailuresConfig.server ?? [];
+          const baselineResult = evaluateBaseline(
+            [{ scenario: validated.scenario!, checks: result.checks }],
+            baselineScenarios
+          );
+          printBaselineResults(baselineResult);
+          process.exit(baselineResult.exitCode);
+        }
+
        process.exit(failed > 0 ? 1 : 0);
      } else {
        // Run scenarios based on suite
@ -290,6 +342,20 @@ program
        }

        const { totalFailed } = printServerSummary(allResults);
+
+        if (options.expectedFailures) {
+          const expectedFailuresConfig = await loadExpectedFailures(
+            options.expectedFailures
+          );
+          const baselineScenarios = expectedFailuresConfig.server ?? [];
+          const baselineResult = evaluateBaseline(
+            allResults,
+            baselineScenarios
+          );
+          printBaselineResults(baselineResult);
+          process.exit(baselineResult.exitCode);
+        }
+
        process.exit(totalFailed > 0 ? 1 : 0);
      }
    } catch (error) {