fix: fail checks whose prerequisite is missing instead of skipping them (#372)

A check that cannot be exercised — the server under test lacks a
diagnostic fixture tool, or advertises a feature it then rejects — was
reported SKIPPED, which is excluded from pass/fail counts, exit codes,
and the expected-failures baseline. A server could dodge the entire
SEP-2575 capability-constraint and subscription surface (7 MUST-level
checks) and still read green, and a baselined failing scenario whose
checks degraded to SKIPPED flipped its baseline entry to stale, which
reads as fixed.

Such checks now fail with a 'Not testable:' message naming the missing
prerequisite and details.untestable for tooling, per the policy in
issue #248 (severity follows the requirement's keyword: MUST -> FAILURE,
SHOULD -> WARNING). Legitimate skips remain: optional capabilities the
server never declared, runner-level spec-version inapplicability, and
the documented tasks-status-notifications harness gap.

Also fixes a test-suite bug found en route: the stream mock's reader
lacked releaseLock, so listenToStream's teardown threw and every
stream-based assertion silently ran against zero frames.

AGENTS.md

@@ -55,6 +55,7 @@ Keep scenarios separate when they're genuinely independent features or when they
 - **Don't reimplement the runner.** New subcommands that need to "select scenarios → run them → print summary → compute exit code" must go through the existing `client` / `server` commands (subprocess via `process.execPath` like `tier-check` and `sdk` do) or call shared helpers — never a parallel suite-map / summary loop.
 - Include `specReferences` pointing to the relevant spec section.
 - **Severity follows the spec keyword:** MUST / MUST NOT → `FAILURE`; SHOULD / SHOULD NOT → `WARNING`. (CI treats WARNING as a failure, so Tier-1 SDKs still need to satisfy SHOULDs — see #245.)
+- **A missing prerequisite is a failure, not a skip** (#248). When a check cannot be exercised — the server under test lacks a diagnostic fixture tool, rejects the probe, or advertises a feature it does not serve — report it failed via `notTestable()` / `untestableCheck()` from `src/scenarios/untestable.ts`, with the missing prerequisite named in `errorMessage` and `details.untestable: true`. SKIPPED reads as green in every consumer (pass counts, exit codes, expected-failures), so it is reserved for checks that are legitimately not applicable: an optional capability the server never declared, spec-version inapplicability handled by the runner, or a documented harness gap the suite itself tracks (e.g. `tasks-status-notifications` pending its subscriptions/listen rewrite). The untestable severity follows the underlying requirement's keyword (MUST → `FAILURE`, SHOULD → `WARNING`); the expected-failures baseline is the escape hatch for SDKs that haven't built the fixture yet.
 
 ## Descriptions and wording
 

src/scenarios/server/http-standard-headers.test.ts

@@ -0,0 +1,240 @@
+import { describe, test, expect, afterEach } from 'vitest';
+import { testContext } from '../../connection/testing';
+import {
+  HttpHeaderValidationScenario,
+  HttpCustomHeaderServerValidationScenario,
+  CUSTOM_HEADER_SERVER_DECLARED_CHECK_IDS
+} from './http-standard-headers';
+import type { ConformanceCheck } from '../../types';
+
+/**
+ * Pins the untestable-failure policy (issue #248) for the SEP-2243 server
+ * scenarios: a server that lacks the fixtures these scenarios need must read
+ * red with a "Not testable:" cause, never as a green SKIPPED run.
+ */
+
+const realFetch = global.fetch;
+afterEach(() => {
+  global.fetch = realFetch;
+});
+
+function mockFetchTarget(
+  handler: (reqBody: any, reqHeaders: Record<string, string>) => any
+) {
+  global.fetch = (async (_url: any, init: any) => {
+    const body = JSON.parse(init.body);
+    const headers = init.headers || {};
+    const responseConfig = (await handler(body, headers)) ?? {
+      status: 404,
+      body: {
+        jsonrpc: '2.0',
+        id: body.id,
+        error: { code: -32601, message: 'Not found' }
+      }
+    };
+    const text = JSON.stringify(responseConfig.body);
+    return {
+      status: responseConfig.status ?? 200,
+      headers: { get: () => 'application/json' },
+      json: async () => responseConfig.body,
+      text: async () => text
+    } as unknown as Response;
+  }) as typeof fetch;
+  return 'http://mock-sep2243-server.local';
+}
+
+const findAll = (checks: ConformanceCheck[], id: string) =>
+  checks.filter((c) => c.id === id);
+
+describe('http-custom-header-server-validation — missing fixture policy', () => {
+  test('emits untestable FAILUREs for every declared check when no x-mcp-header tool exists', async () => {
+    const mockUrl = mockFetchTarget((reqBody) => {
+      if (reqBody.method === 'tools/list') {
+        return {
+          status: 200,
+          body: {
+            jsonrpc: '2.0',
+            id: reqBody.id,
+            result: {
+              tools: [
+                {
+                  name: 'plain_tool',
+                  inputSchema: {
+                    type: 'object',
+                    properties: { q: { type: 'string' } }
+                  }
+                }
+              ]
+            }
+          }
+        };
+      }
+    });
+
+    const scenario = new HttpCustomHeaderServerValidationScenario();
+    const checks = await scenario.run(testContext(mockUrl));
+
+    const gate = findAll(checks, 'sep-2243-server-no-xmcp-tool')[0];
+    expect(gate?.status).toBe('FAILURE');
+    expect(gate?.errorMessage).toContain('Not testable:');
+
+    for (const id of CUSTOM_HEADER_SERVER_DECLARED_CHECK_IDS) {
+      const declared = findAll(checks, id)[0];
+      expect(declared?.status, id).toBe('FAILURE');
+      expect(declared?.errorMessage, id).toContain('Not testable:');
+      expect(declared?.details, id).toMatchObject({ untestable: true });
+    }
+
+    // The run must not contain a single SKIPPED row: the whole point is
+    // that this server cannot collect a vacuous green.
+    expect(checks.every((c) => c.status !== 'SKIPPED')).toBe(true);
+  });
+
+  test('emits untestable FAILUREs when the annotated tool has no string parameter', async () => {
+    const mockUrl = mockFetchTarget((reqBody) => {
+      if (reqBody.method === 'tools/list') {
+        return {
+          status: 200,
+          body: {
+            jsonrpc: '2.0',
+            id: reqBody.id,
+            result: {
+              tools: [
+                {
+                  name: 'numeric_only',
+                  inputSchema: {
+                    type: 'object',
+                    properties: {
+                      level: { type: 'number', 'x-mcp-header': 'Level' }
+                    }
+                  }
+                }
+              ]
+            }
+          }
+        };
+      }
+    });
+
+    const scenario = new HttpCustomHeaderServerValidationScenario();
+    const checks = await scenario.run(testContext(mockUrl));
+
+    const gate = findAll(checks, 'sep-2243-server-no-string-param')[0];
+    expect(gate?.status).toBe('FAILURE');
+    expect(gate?.errorMessage).toContain('Not testable:');
+    for (const id of CUSTOM_HEADER_SERVER_DECLARED_CHECK_IDS) {
+      expect(findAll(checks, id)[0]?.status, id).toBe('FAILURE');
+    }
+  });
+});
+
+describe('http-header-validation — zero-tools Mcp-Name cases', () => {
+  // This scenario sends its live cases over raw node http (not fetch), so it
+  // is driven against a real local server rather than a fetch mock.
+  async function startBareServer(
+    handler: (body: any) => { status: number; body: object }
+  ): Promise<{ url: string; close: () => Promise<void> }> {
+    const http = await import('http');
+    const server = http.createServer((req, res) => {
+      let raw = '';
+      req.setEncoding('utf8');
+      req.on('data', (c) => (raw += c));
+      req.on('end', () => {
+        let body: any = {};
+        try {
+          body = JSON.parse(raw);
+        } catch {
+          // Treat unparseable bodies as empty requests.
+        }
+        const out = handler(body);
+        res.writeHead(out.status, { 'Content-Type': 'application/json' });
+        res.end(JSON.stringify(out.body));
+      });
+    });
+    await new Promise<void>((resolve) => server.listen(0, resolve));
+    const address = server.address();
+    const port = typeof address === 'object' && address ? address.port : 0;
+    return {
+      url: `http://localhost:${port}/mcp`,
+      close: () =>
+        new Promise<void>((resolve) => {
+          server.closeAllConnections?.();
+          server.close(() => resolve());
+        })
+    };
+  }
+
+  test('emits untestable FAILUREs for the Mcp-Name cases when the server lists no tools', async () => {
+    const srv = await startBareServer((body) => {
+      if (body.method === 'tools/list') {
+        return {
+          status: 200,
+          body: { jsonrpc: '2.0', id: body.id, result: { tools: [] } }
+        };
+      }
+      return {
+        status: 400,
+        body: {
+          jsonrpc: '2.0',
+          id: body.id ?? null,
+          error: { code: -32020, message: 'Header mismatch' }
+        }
+      };
+    });
+
+    const scenario = new HttpHeaderValidationScenario();
+    let checks: ConformanceCheck[];
+    try {
+      checks = await scenario.run(testContext(srv.url));
+    } finally {
+      await srv.close();
+    }
+
+    // Previously these three cases were silently omitted; now each surfaces
+    // as an untestable FAILURE naming the missing prerequisite.
+    const whitespace = findAll(
+      checks,
+      'sep-2243-server-accepts-whitespace-header-value'
+    );
+    expect(whitespace.length).toBeGreaterThan(0);
+    expect(whitespace[0].status).toBe('FAILURE');
+    expect(whitespace[0].errorMessage).toContain('Not testable:');
+
+    const nameRejects = findAll(
+      checks,
+      'sep-2243-server-reject-invalid-headers'
+    ).filter((c) => c.errorMessage?.startsWith('Not testable:'));
+    expect(nameRejects.map((c) => c.name).sort()).toEqual([
+      'ServerRejectsMismatchedNameHeader',
+      'ServerRejectsMissingNameHeader'
+    ]);
+  });
+
+  test('emits untestable FAILUREs for the Mcp-Name cases when tools/list discovery fails', async () => {
+    const srv = await startBareServer((body) => ({
+      status: 500,
+      body: {
+        jsonrpc: '2.0',
+        id: body.id ?? null,
+        error: { code: -32603, message: 'boom' }
+      }
+    }));
+
+    const scenario = new HttpHeaderValidationScenario();
+    let checks: ConformanceCheck[];
+    try {
+      checks = await scenario.run(testContext(srv.url));
+    } finally {
+      await srv.close();
+    }
+
+    const setup = findAll(checks, 'sep-2243-server-standard-setup')[0];
+    expect(setup?.status).toBe('FAILURE');
+    const whitespace = findAll(
+      checks,
+      'sep-2243-server-accepts-whitespace-header-value'
+    )[0];
+    expect(whitespace?.status).toBe('FAILURE');
+    expect(whitespace?.errorMessage).toContain('Not testable:');
+  });
+});

src/scenarios/server/http-standard-headers.ts

@@ -26,6 +26,7 @@ import {
   type RunContext
 } from '../../connection';
 import { HEADER_MISMATCH } from '../../spec-types/draft';
+import { untestableCheck } from '../untestable';
 
 const SPEC_REFERENCE = {
   id: 'SEP-2243-Server-Validation',
@@ -86,6 +87,45 @@ export const CUSTOM_HEADER_SERVER_DECLARED_CHECK_IDS = [
   'sep-2243-server-reject-param-mismatch'
 ] as const;
 
+/**
+ * The three Mcp-Name validation cases need a callable tool. They are emitted
+ * from two places: as live test cases when the server lists a tool, and as
+ * untestable failures (issue #248) when it does not — keep the identities in
+ * one table so the two paths cannot drift.
+ */
+const MCP_NAME_CASES = [
+  {
+    id: REJECT_STATUS_CHECK_ID,
+    name: 'ServerRejectsMismatchedNameHeader',
+    description:
+      'Server rejects tools/call where Mcp-Name does not match body params.name',
+    specRef: SPEC_REFERENCE
+  },
+  {
+    id: 'sep-2243-server-accepts-whitespace-header-value',
+    name: 'ServerAcceptsWhitespaceHeaderValue',
+    description:
+      'Server MUST accept leading/trailing whitespace in Mcp-Name value (RFC 9110 §5.5: field parsing MUST exclude OWS before evaluating)',
+    specRef: SPEC_REFERENCE_RFC9110_OWS
+  },
+  {
+    id: REJECT_STATUS_CHECK_ID,
+    name: 'ServerRejectsMissingNameHeader',
+    description:
+      'Server MUST reject tools/call with missing Mcp-Name header when body has params.name',
+    specRef: SPEC_REFERENCE
+  }
+] as const;
+
+/** Emit the Mcp-Name cases as untestable failures naming the cause. */
+function untestableMcpNameCases(checks: ConformanceCheck[], reason: string) {
+  for (const c of MCP_NAME_CASES) {
+    checks.push(
+      untestableCheck(c.id, c.name, c.description, reason, [c.specRef])
+    );
+  }
+}
+
 /**
  * Helper to send a raw HTTP POST request with custom headers.
  * Uses Node.js http.request to preserve exact header casing and values,
@@ -294,6 +334,10 @@ export class HttpHeaderValidationScenario implements ClientScenario {
           specReferences: [SPEC_REFERENCE],
           details: { httpStatus: toolsResponse.status, error: rpcError }
         });
+        untestableMcpNameCases(
+          checks,
+          'tools/list discovery failed, so the Mcp-Name header-validation cases could not be exercised'
+        );
         return checks;
       }
       const toolsResult = toolsResponse.body.result as {
@@ -417,6 +461,14 @@ export class HttpHeaderValidationScenario implements ClientScenario {
               'Standard header omitted but value present in body → MUST reject'
           }
         );
+      } else {
+        // No tools to exercise the Mcp-Name dimension: these MUST cases were
+        // previously omitted without a trace. Emit untestable failures so
+        // the gap is visible (issue #248).
+        untestableMcpNameCases(
+          checks,
+          'server lists no tools, so the Mcp-Name header-validation cases could not be exercised'
+        );
       }
 
       // --- Case Sensitivity Tests ---
@@ -608,22 +660,18 @@ export class HttpCustomHeaderServerValidationScenario implements ClientScenario
       });
 
       if (!xMcpTool) {
-        checks.push({
+        checks.push(
-          id: 'sep-2243-server-no-xmcp-tool',
+          untestableCheck(
-          name: 'HttpCustomHeaderServerNoTool',
+            'sep-2243-server-no-xmcp-tool',
-          description:
+            'HttpCustomHeaderServerNoTool',
             'Server has no tools with x-mcp-header annotations to test',
-          status: 'SKIPPED',
+            'server exposes no tool with x-mcp-header annotations, so none of the custom-header validation requirements could be exercised',
-          timestamp: new Date().toISOString(),
+            [SPEC_REFERENCE_CUSTOM]
-          specReferences: [SPEC_REFERENCE_CUSTOM],
+          )
-          details: {
+        );
-            reason:
+        this.untestableDeclaredChecks(
-              'No tools with x-mcp-header found. These tests require at least one tool with x-mcp-header annotations.'
-          }
-        });
-        this.skipDeclaredChecks(
           checks,
-          'Server exposes no tool with x-mcp-header annotations.'
+          'server exposes no tool with x-mcp-header annotations'
         );
         return checks;
       }
@@ -640,18 +688,18 @@ export class HttpCustomHeaderServerValidationScenario implements ClientScenario
           def['x-mcp-header'] !== undefined && (def as any).type === 'string'
       );
       if (!annotatedEntry) {
-        checks.push({
+        checks.push(
-          id: 'sep-2243-server-no-string-param',
+          untestableCheck(
-          name: 'HttpCustomHeaderServerNoStringParam',
+            'sep-2243-server-no-string-param',
-          description:
+            'HttpCustomHeaderServerNoStringParam',
             'Server has no string-typed x-mcp-header parameter to test',
-          status: 'SKIPPED',
+            'server exposes no string-typed x-mcp-header parameter, so the custom-header validation requirements could not be exercised',
-          timestamp: new Date().toISOString(),
+            [SPEC_REFERENCE_CUSTOM]
-          specReferences: [SPEC_REFERENCE_CUSTOM]
+          )
-        });
+        );
-        this.skipDeclaredChecks(
+        this.untestableDeclaredChecks(
           checks,
-          'Server exposes no string-typed x-mcp-header parameter.'
+          'server exposes no string-typed x-mcp-header parameter'
         );
         return checks;
       }
@@ -818,7 +866,7 @@ export class HttpCustomHeaderServerValidationScenario implements ClientScenario
     }
 
     // Declared-but-unemitted -> FAILURE. Reached only when setup threw partway
-    // through (the gate-out paths emit SKIPPED rows and the happy path emits
+    // through (the gate-out paths emit untestable FAILURE rows and the happy path emits
     // every declared ID).
     this.failDeclaredChecks(checks);
 
@@ -846,23 +894,27 @@ export class HttpCustomHeaderServerValidationScenario implements ClientScenario
   }
 
   /**
-   * Emit one SKIPPED row per declared requirement check when the scenario
+   * Emit one untestable FAILURE row per declared requirement check when the
-   * cannot run against this server (no x-mcp-header tool). The IDs must still
+   * scenario cannot run against this server (no x-mcp-header tool). These
-   * reach checks.json so the traceability manifest records that a scenario
+   * are MUST requirements, so a missing fixture must read red, not green
-   * exists for each requirement.
+   * (issue #248); the IDs still reach checks.json so the traceability
+   * manifest records that a scenario exists for each requirement.
    */
-  private skipDeclaredChecks(checks: ConformanceCheck[], reason: string): void {
+  private untestableDeclaredChecks(
+    checks: ConformanceCheck[],
+    reason: string
+  ): void {
     for (const id of CUSTOM_HEADER_SERVER_DECLARED_CHECK_IDS) {
       if (checks.some((c) => c.id === id)) continue;
-      checks.push({
+      checks.push(
-        id,
+        untestableCheck(
-        name: 'NotApplicable',
+          id,
-        description: `Declared check ${id} is not testable against this server`,
+          'NotTestable',
-        status: 'SKIPPED',
+          `Declared check ${id} is not testable against this server`,
-        timestamp: new Date().toISOString(),
+          reason,
-        specReferences: [SPEC_REFERENCE_CUSTOM],
+          [SPEC_REFERENCE_CUSTOM]
-        details: { reason }
+        )
-      });
+      );
     }
   }
 

src/scenarios/server/stateless.test.ts

@@ -66,7 +66,13 @@ describe('Stateless Server Scenario Negative Tests', () => {
             }
             const chunk = new TextEncoder().encode(streamData[frameIndex++]);
             return { value: chunk, done: false };
-          }
+          },
+          // Required by listenToStream's finally block; without it the
+          // reader teardown throws, the stream helper swallows the error,
+          // and every stream-based assertion silently runs against zero
+          // frames — the exact vacuous-pass failure mode this suite exists
+          // to prevent.
+          releaseLock: () => {}
         };
 
         return {
@@ -91,6 +97,24 @@ describe('Stateless Server Scenario Negative Tests', () => {
     return 'http://mock-stateless-mcp-server.local';
   }
 
+  // Shared server/discover stanza for the negative-server mocks below.
+  const discoverResponse = (
+    reqBody: any,
+    capabilities: object,
+    serverName: string
+  ) => ({
+    status: 200,
+    body: {
+      jsonrpc: '2.0',
+      id: reqBody.id,
+      result: {
+        supportedVersions: ['2026-07-28'],
+        capabilities,
+        serverInfo: { name: serverName, version: '1.0.0' }
+      }
+    }
+  });
+
   test('Fails validation if missing required fields in _meta are allowed to pass', async () => {
     // This bad server completely ignores missing params/_meta fields and returns a fake success result
     const mockUrl = mockFetchTarget((reqBody) => {
@@ -127,6 +151,190 @@ describe('Stateless Server Scenario Negative Tests', () => {
     expect(missingVersionCheck?.status).toBe('FAILURE');
   });
 
+  test('Fails (untestable) the capability checks when the diagnostic tool is not listed', async () => {
+    // This server has no test_missing_capability tool at all: the
+    // undeclared-capability MUSTs cannot be exercised and must read red,
+    // not SKIPPED (issue #248).
+    const mockUrl = mockFetchTarget((reqBody) => {
+      if (reqBody.method === 'server/discover') {
+        return discoverResponse(reqBody, { tools: {} }, 'no-diagnostic-tools');
+      }
+      if (reqBody.method === 'tools/list') {
+        return {
+          status: 200,
+          body: { jsonrpc: '2.0', id: reqBody.id, result: { tools: [] } }
+        };
+      }
+    });
+
+    const scenario = new ServerStatelessScenario();
+    const checks = await scenario.run(testContext(mockUrl));
+
+    const rejectCheck = findCheck(
+      checks,
+      'sep-2575-server-rejects-undeclared-capability'
+    );
+    const statusCheck = findCheck(
+      checks,
+      'sep-2575-missing-capability-http-400'
+    );
+
+    expect(rejectCheck?.status).toBe('FAILURE');
+    expect(rejectCheck?.errorMessage).toContain('Not testable:');
+    expect(rejectCheck?.errorMessage).toContain('test_missing_capability');
+    expect(statusCheck?.status).toBe('FAILURE');
+    expect(statusCheck?.errorMessage).toContain('Not testable:');
+  });
+
+  test('Fails the capability check when the listed diagnostic tool executes without -32021', async () => {
+    // This server lists test_missing_capability but happily executes it even
+    // though the client never declared the sampling capability — a genuine
+    // violation of the MUST, not an untestable gap.
+    const mockUrl = mockFetchTarget((reqBody) => {
+      if (reqBody.method === 'server/discover') {
+        return discoverResponse(reqBody, { tools: {} }, 'no-enforcement');
+      }
+      if (reqBody.method === 'tools/list') {
+        return {
+          status: 200,
+          body: {
+            jsonrpc: '2.0',
+            id: reqBody.id,
+            result: { tools: [{ name: 'test_missing_capability' }] }
+          }
+        };
+      }
+      if (
+        reqBody.method === 'tools/call' &&
+        reqBody.params?.name === 'test_missing_capability'
+      ) {
+        return {
+          status: 200,
+          body: {
+            jsonrpc: '2.0',
+            id: reqBody.id,
+            result: { resultType: 'complete', content: [] }
+          }
+        };
+      }
+    });
+
+    const scenario = new ServerStatelessScenario();
+    const checks = await scenario.run(testContext(mockUrl));
+
+    const rejectCheck = findCheck(
+      checks,
+      'sep-2575-server-rejects-undeclared-capability'
+    );
+    expect(rejectCheck?.status).toBe('FAILURE');
+    expect(rejectCheck?.errorMessage).toContain('MUST reject with -32021');
+  });
+
+  test('Fails the subscription checks when listChanged is advertised but listen is rejected', async () => {
+    const mockUrl = mockFetchTarget((reqBody) => {
+      if (reqBody.method === 'server/discover') {
+        return discoverResponse(
+          reqBody,
+          { tools: { listChanged: true } },
+          'claims-subscriptions'
+        );
+      }
+      if (reqBody.method === 'subscriptions/listen') {
+        return {
+          isStream: true,
+          status: 404,
+          frames: [
+            {
+              jsonrpc: '2.0',
+              id: reqBody.id,
+              error: { code: -32601, message: 'Method not found' }
+            }
+          ]
+        };
+      }
+    });
+
+    const scenario = new ServerStatelessScenario();
+    const checks = await scenario.run(testContext(mockUrl));
+
+    const ackCheck = findCheck(
+      checks,
+      'sep-2575-server-sends-subscription-ack'
+    );
+    expect(ackCheck?.status).toBe('FAILURE');
+    expect(ackCheck?.errorMessage).toContain('Not testable:');
+    expect(ackCheck?.errorMessage).toContain('-32601');
+  });
+
+  test('Skips the subscription checks when no subscription capability is advertised', async () => {
+    // A server that never claimed listChanged/subscribe legitimately has
+    // nothing to serve on subscriptions/listen — this is the one case that
+    // stays SKIPPED.
+    const mockUrl = mockFetchTarget((reqBody) => {
+      if (reqBody.method === 'server/discover') {
+        return discoverResponse(reqBody, { tools: {} }, 'no-subscriptions');
+      }
+      if (reqBody.method === 'subscriptions/listen') {
+        return {
+          isStream: true,
+          status: 404,
+          frames: [
+            {
+              jsonrpc: '2.0',
+              id: reqBody.id,
+              error: { code: -32601, message: 'Method not found' }
+            }
+          ]
+        };
+      }
+    });
+
+    const scenario = new ServerStatelessScenario();
+    const checks = await scenario.run(testContext(mockUrl));
+
+    const ackCheck = findCheck(
+      checks,
+      'sep-2575-server-sends-subscription-ack'
+    );
+    const tagCheck = findCheck(checks, 'sep-2575-server-tags-subscription-id');
+    expect(ackCheck?.status).toBe('SKIPPED');
+    expect(tagCheck?.status).toBe('SKIPPED');
+  });
+
+  test('Fails (untestable) the subscription checks when discover was never observed', async () => {
+    // With no server/discover response at all, a -32601 on listen cannot be
+    // attributed to an intentionally absent capability — the legitimate skip
+    // requires an observed advertisement.
+    const mockUrl = mockFetchTarget((reqBody) => {
+      if (reqBody.method === 'subscriptions/listen') {
+        return {
+          isStream: true,
+          status: 404,
+          frames: [
+            {
+              jsonrpc: '2.0',
+              id: reqBody.id,
+              error: { code: -32601, message: 'Method not found' }
+            }
+          ]
+        };
+      }
+    });
+
+    const scenario = new ServerStatelessScenario();
+    const checks = await scenario.run(testContext(mockUrl));
+
+    const ackCheck = findCheck(
+      checks,
+      'sep-2575-server-sends-subscription-ack'
+    );
+    expect(ackCheck?.status).toBe('FAILURE');
+    expect(ackCheck?.errorMessage).toContain('Not testable:');
+    expect(ackCheck?.errorMessage).toContain(
+      'server/discover was not observed'
+    );
+  });
+
   test('Fails validation when missing-_meta rejections are returned with HTTP 200', async () => {
     // This bad server picks the right JSON-RPC error code but the wrong HTTP
     // status: the spec requires 400 Bad Request for malformed requests.

src/scenarios/server/stateless.ts

@@ -12,6 +12,7 @@ import {
   readSseJsonRpcResponse,
   type RunContext
 } from '../../connection';
+import { notTestable } from '../untestable';
 
 const SPEC_REF = [
   {
@@ -114,6 +115,10 @@ export class ServerStatelessScenario implements ClientScenario {
       }
     }
 
+    // Every JSON-RPC error response observed anywhere in the scenario counts
+    // toward the id-echo backfill's "did we see any error at all" condition.
+    let errorResponsesObserved = 0;
+
     // Helper to send raw RPC requests via fetch
     const sendRpc = async (
       method: string,
@@ -156,6 +161,7 @@ export class ServerStatelessScenario implements ClientScenario {
           // Response might not be JSON
         }
       }
+      if (data?.error !== undefined) errorResponsesObserved++;
       return { res, data };
     };
 
@@ -483,6 +489,22 @@ export class ServerStatelessScenario implements ClientScenario {
       }
     );
 
+    // Fetched once here and reused by the capability-constraint checks below
+    // to tell a missing diagnostic tool apart from a genuine violation.
+    const toolsListResponse = await sendRpc(
+      'tools/list',
+      { _meta: validMeta },
+      undefined,
+      202
+    ).catch(() => null);
+    const listedToolNames: string[] = Array.isArray(
+      toolsListResponse?.data?.result?.tools
+    )
+      ? toolsListResponse.data.result.tools
+          .map((t: any) => t?.name)
+          .filter((n: any) => typeof n === 'string')
+      : [];
+
     // Dynamic verification helper to check capability consistency against true handlers
     await runCheck(
       'sep-2575-discover-capabilities-match-handlers',
@@ -493,12 +515,9 @@ export class ServerStatelessScenario implements ClientScenario {
           return {
             error: `Discovery runtime check failed: ${discoverRpcError.message}`
           };
-        const { data: toolsData } = await sendRpc(
+        if (!toolsListResponse)
-          'tools/list',
+          return { error: 'tools/list probe network hit failed' };
-          { _meta: validMeta },
+        const toolsData = toolsListResponse.data;
-          undefined,
-          202
-        );
 
         if (discoverCapabilities.tools) {
           const toolsPassed = Array.isArray(toolsData?.result?.tools);
@@ -632,6 +651,9 @@ export class ServerStatelessScenario implements ClientScenario {
 
     // Determine if this server actively enforces client capabilities
     const serverRequiresCapability = data401?.error?.code === -32021;
+    const capabilityToolListed = listedToolNames.includes(
+      'test_missing_capability'
+    );
 
     await runCheck(
       'sep-2575-server-rejects-undeclared-capability',
@@ -645,14 +667,29 @@ export class ServerStatelessScenario implements ClientScenario {
           };
 
         if (!serverRequiresCapability) {
-          // The server didn't return -32021, so this requirement isn't
+          // Distinguish a genuine violation from a fixture gap, strongest
-          // exercised for this method. Report SKIPPED rather than a green PASS.
+          // evidence first: a result proves the tool executed (violation
+          // regardless of whether tools/list mentioned it); a non--32021
+          // error from a listed tool is the wrong code; an error from an
+          // unlisted tool means the requirement could not be exercised.
+          if (data401?.result !== undefined) {
+            return {
+              error:
+                "Server executed 'test_missing_capability' although the client did not declare the 'sampling' capability — it MUST reject with -32021",
+              details: { response: data401 }
+            };
+          }
+          if (data401?.error && capabilityToolListed) {
+            return {
+              error: `Expected MissingRequiredClientCapabilityError (-32021), got error code ${data401.error.code}`,
+              details: { response: data401 }
+            };
+          }
           return {
-            skipped: true,
+            error: notTestable(
-            details: {
+              "server does not list the diagnostic tool 'test_missing_capability' in tools/list and the probe call did not exercise it (required for the undeclared-capability rejection)"
-              note: 'Skipped requirement tracking: Server returned a non-32021 response, indicating it does not require explicit client capability authorization constraints for this method.',
+            ),
-              response: data401
+            details: { untestable: true, response: data401 }
-            }
           };
         }
 
@@ -680,14 +717,16 @@ export class ServerStatelessScenario implements ClientScenario {
           };
 
         if (!serverRequiresCapability) {
-          // No -32021 means the HTTP-400 requirement doesn't apply here.
+          // Same root cause as the previous check: either the diagnostic
-          // Report SKIPPED rather than a green PASS.
+          // tool is missing (untestable) or the server failed to emit the
+          // -32021 this status requirement attaches to.
           return {
-            skipped: true,
+            error: notTestable(
-            details: {
+              capabilityToolListed
-              note: 'Skipped status tracking: Server did not return a MissingRequiredClientCapabilityError.',
+                ? 'server did not return MissingRequiredClientCapabilityError, so its HTTP status could not be validated'
-              httpStatus: res401.status
+                : "server does not list the diagnostic tool 'test_missing_capability' in tools/list, so the -32021 HTTP status could not be validated"
-            }
+            ),
+            details: { untestable: true, httpStatus: res401.status }
           };
         }
 
@@ -795,16 +834,17 @@ export class ServerStatelessScenario implements ClientScenario {
           };
         }
 
-        // If the call was rejected outright (e.g. the diagnostic tool does
+        // If the call was rejected outright, nothing was streamed and the
-        // not exist), nothing was streamed and the requirement was not
+        // requirement could not be exercised — fail with the cause rather
-        // exercised - report SKIPPED rather than a vacuous SUCCESS.
+        // than reporting a vacuous SUCCESS or an invisible SKIPPED.
         if (frames.every((f) => f?.error !== undefined)) {
           return {
-            skipped: true,
+            error: notTestable(
-            details: {
+              listedToolNames.includes('test_streaming_elicitation')
-              note: 'Server does not expose diagnostic tool test_streaming_elicitation; the response stream could not be exercised.',
+                ? `the 'test_streaming_elicitation' call was rejected (code ${frames[0]?.error?.code}), so the response stream could not be exercised`
-              response: frames[0]
+                : "server does not list the diagnostic tool 'test_streaming_elicitation' in tools/list, so the response stream could not be exercised"
-            }
+            ),
+            details: { untestable: true, response: frames[0] }
           };
         }
 
@@ -841,17 +881,17 @@ export class ServerStatelessScenario implements ClientScenario {
           };
         }
 
-        // If the call was rejected outright (e.g. the diagnostic tool does
+        // If the call was rejected outright, the server never had anything
-        // not exist), the server never had anything to log and the
+        // to log and the requirement could not be exercised — fail with the
-        // requirement was not exercised - report SKIPPED rather than a
+        // cause rather than reporting a vacuous SUCCESS or invisible SKIPPED.
-        // vacuous SUCCESS.
         if (frames.every((f) => f?.error !== undefined)) {
           return {
-            skipped: true,
+            error: notTestable(
-            details: {
+              listedToolNames.includes('test_logging_tool')
-              note: 'Server does not expose diagnostic tool test_logging_tool; the no-log-without-logLevel requirement could not be exercised.',
+                ? `the 'test_logging_tool' call was rejected (code ${frames[0]?.error?.code}), so the no-log-without-logLevel requirement could not be exercised`
-              response: frames[0]
+                : "server does not list the diagnostic tool 'test_logging_tool' in tools/list, so the no-log-without-logLevel requirement could not be exercised"
-            }
+            ),
+            details: { untestable: true, response: frames[0] }
           };
         }
 
@@ -872,6 +912,41 @@ export class ServerStatelessScenario implements ClientScenario {
     // ==========================================
     // 6. Subscription Streams & Filtering (3 Checks)
     // ==========================================
+    // A server that advertises no subscription-delivered capability has
+    // nothing to serve on subscriptions/listen, so a -32601 there is a
+    // legitimate feature absence (SKIPPED). A server that DOES advertise
+    // listChanged/subscribe but rejects the method fails: it claims a
+    // feature it does not serve.
+    const advertisesSubscriptions = !!(
+      discoverCapabilities?.tools?.listChanged ||
+      discoverCapabilities?.prompts?.listChanged ||
+      discoverCapabilities?.resources?.listChanged ||
+      discoverCapabilities?.resources?.subscribe
+    );
+    // The legitimate skip requires an OBSERVED advertisement: when
+    // server/discover itself failed, nothing is known about the server's
+    // capabilities and the gap must not read as an intentional absence.
+    const discoverObserved = !discoverRpcError && discoverResult != null;
+    const listenRejected = (frames: any[]) => {
+      if (frames[0]?.error?.code !== -32601) return null;
+      if (discoverObserved && !advertisesSubscriptions) {
+        return {
+          skipped: true,
+          details: {
+            note: 'Server advertises no subscription-delivered capability; subscriptions/listen is not applicable.'
+          }
+        };
+      }
+      return {
+        error: notTestable(
+          advertisesSubscriptions
+            ? 'server advertises listChanged/subscribe capabilities but answers subscriptions/listen with -32601 (Method not found)'
+            : 'server/discover was not observed, so the -32601 on subscriptions/listen cannot be attributed to an intentionally absent capability'
+        ),
+        details: { untestable: true, response: frames[0] }
+      };
+    };
+
     const subscriptionParams = {
       _meta: validMeta,
       notifications: { toolsListChanged: true }
@@ -904,14 +979,8 @@ export class ServerStatelessScenario implements ClientScenario {
       'ServerSendsSubscriptionAck',
       'notifications/subscriptions/acknowledged is the first message on a subscriptions/listen stream',
       () => {
-        if (streamFrames[0]?.error?.code === -32601) {
+        const rejectedStream = listenRejected(streamFrames);
-          return {
+        if (rejectedStream) return rejectedStream;
-            skipped: true,
-            details: {
-              note: 'Server does not support subscriptions/listen (Method not found)'
-            }
-          };
-        }
         if (streamFrames.length === 0) {
           return {
             error:
@@ -934,14 +1003,8 @@ export class ServerStatelessScenario implements ClientScenario {
       'ServerTagsSubscriptionId',
       'Listen-stream notifications carry _meta.../subscriptionId',
       () => {
-        if (streamFrames[0]?.error?.code === -32601) {
+        const rejectedStream = listenRejected(streamFrames);
-          return {
+        if (rejectedStream) return rejectedStream;
-            skipped: true,
-            details: {
-              note: 'Server does not support subscriptions/listen (Method not found)'
-            }
-          };
-        }
         if (streamFrames.length === 0) {
           return {
             error:
@@ -1015,14 +1078,8 @@ export class ServerStatelessScenario implements ClientScenario {
           }
         );
 
-        if (narrowFrames[0]?.error?.code === -32601) {
+        const rejectedNarrow = listenRejected(narrowFrames);
-          return {
+        if (rejectedNarrow) return rejectedNarrow;
-            skipped: true,
-            details: {
-              note: 'Server does not support subscriptions/listen (Method not found)'
-            }
-          };
-        }
         if (narrowFrames.length === 0) {
           return {
             error:
@@ -1086,11 +1143,13 @@ export class ServerStatelessScenario implements ClientScenario {
           }
         );
         if (trigger?.data?.error?.code === -32601) {
+          // SHOULD-level requirement, so the untestable report is a WARNING.
           return {
-            skipped: true,
+            warning: true,
-            details: {
+            error: notTestable(
-              note: 'Server does not expose diagnostic hook test_trigger_prompt_change to mutate lists'
+              "server declares prompts.listChanged but does not expose the diagnostic hook 'test_trigger_prompt_change' to mutate the prompt list"
-            }
+            ),
+            details: { untestable: true }
           };
         }
 
@@ -1154,11 +1213,13 @@ export class ServerStatelessScenario implements ClientScenario {
           }
         );
         if (trigger?.data?.error?.code === -32601) {
+          // SHOULD-level requirement, so the untestable report is a WARNING.
           return {
-            skipped: true,
+            warning: true,
-            details: {
+            error: notTestable(
-              note: 'Server does not expose diagnostic hook test_trigger_tool_change to mutate lists'
+              "server declares tools.listChanged but does not expose the diagnostic hook 'test_trigger_tool_change' to mutate the tool list"
-            }
+            ),
+            details: { untestable: true }
           };
         }
 
@@ -1185,12 +1246,26 @@ export class ServerStatelessScenario implements ClientScenario {
     );
 
     if (!checks.some((c) => c.id === 'sep-2575-http-server-error-jsonrpc-id')) {
+      // Only report SUCCESS when at least one error response was actually
+      // observed; a server that never produced an error response (because it
+      // wrongly accepted every invalid probe) must not collect a vacuous pass
+      // for the id-echo requirement on top of its other failures.
       checks.push({
         id: 'sep-2575-http-server-error-jsonrpc-id',
         name: 'HttpServerErrorJsonrpcId',
         description: 'All error responses carry the request JSON-RPC id',
-        status: 'SUCCESS',
+        status: errorResponsesObserved > 0 ? 'SUCCESS' : 'FAILURE',
         timestamp,
+        errorMessage:
+          errorResponsesObserved > 0
+            ? undefined
+            : notTestable(
+                'no error responses were observed across the probes, so the id-echo requirement could not be validated'
+              ),
+        details:
+          errorResponsesObserved > 0
+            ? { errorResponsesObserved }
+            : { untestable: true, errorResponsesObserved },
         specReferences: SPEC_REF
       });
     }

src/scenarios/server/tasks/lifecycle.test.ts

@@ -0,0 +1,82 @@
+import { describe, test, expect, afterEach } from 'vitest';
+import { testContext } from '../../../connection/testing';
+import { TasksLifecycleScenario } from './lifecycle';
+import { DRAFT_PROTOCOL_VERSION } from '../../../types';
+import type { ConformanceCheck } from '../../../types';
+
+/**
+ * Pins the untestable-failure policy (issue #248) for the SEP-2663 lifecycle
+ * scenario: when no task is ever created, the downstream task checks must
+ * fail with a "Not testable:" cause instead of reporting SKIPPED.
+ */
+
+const realFetch = global.fetch;
+afterEach(() => {
+  global.fetch = realFetch;
+});
+
+function mockServer() {
+  global.fetch = (async (_url: any, init: any) => {
+    const body = JSON.parse(init.body);
+    let result: any;
+    if (body.method === 'server/discover') {
+      result = {
+        supportedVersions: [DRAFT_PROTOCOL_VERSION],
+        capabilities: { tools: {} },
+        serverInfo: { name: 'taskless-server', version: '1.0.0' }
+      };
+    } else if (body.method === 'tools/list') {
+      result = { tools: [] };
+    } else if (body.method === 'tools/call') {
+      // Always answers synchronously: never creates a task.
+      result = {
+        resultType: 'complete',
+        content: [{ type: 'text', text: 'sync answer' }]
+      };
+    } else {
+      return {
+        status: 404,
+        headers: { get: () => 'application/json' },
+        json: async () => ({
+          jsonrpc: '2.0',
+          id: body.id ?? null,
+          error: { code: -32601, message: 'Method not found' }
+        }),
+        text: async () =>
+          JSON.stringify({
+            jsonrpc: '2.0',
+            id: body.id ?? null,
+            error: { code: -32601, message: 'Method not found' }
+          })
+      } as unknown as Response;
+    }
+    const payload = { jsonrpc: '2.0', id: body.id, result };
+    return {
+      status: 200,
+      headers: { get: () => 'application/json' },
+      json: async () => payload,
+      text: async () => JSON.stringify(payload)
+    } as unknown as Response;
+  }) as typeof fetch;
+  return 'http://mock-taskless-server.local';
+}
+
+describe('tasks-lifecycle — no task created', () => {
+  test('downstream task checks fail as untestable instead of SKIPPED', async () => {
+    const mockUrl = mockServer();
+    const scenario = new TasksLifecycleScenario();
+    const checks: ConformanceCheck[] = await scenario.run(
+      testContext(mockUrl, DRAFT_PROTOCOL_VERSION)
+    );
+
+    const gated = checks.filter((c) =>
+      c.errorMessage?.startsWith('Not testable:')
+    );
+    expect(gated.length).toBeGreaterThan(0);
+    for (const check of gated) {
+      expect(check.status).toBe('FAILURE');
+      expect(check.details).toMatchObject({ untestable: true });
+    }
+    expect(checks.every((c) => c.status !== 'SKIPPED')).toBe(true);
+  });
+});

src/scenarios/server/tasks/lifecycle.ts

@@ -16,8 +16,9 @@
 import { ClientScenario, ConformanceCheck } from '../../../types';
 import type { Connection, RunContext } from '../../../connection';
 import { SEP_2322_REF, SEP_2663_REF } from './mrtr-helpers';
-import { errMsg, failureCheck, skipCheck } from './mrtr-helpers';
+import { errMsg, failureCheck } from './mrtr-helpers';
 import { TASKS_EXTENSION_ID, waitForTerminal } from './helpers';
+import { untestableCheck } from '../../untestable';
 import { isIso8601 } from './mrtr-helpers';
 
 export class TasksLifecycleScenario implements ClientScenario {
@@ -231,7 +232,13 @@ The server MUST advertise \`io.modelcontextprotocol/tasks\` under
         'tasks/get returns status + metadata for an active task';
       if (!workingTaskId) {
         checks.push(
-          skipCheck(id, name, description, 'no task created', [SEP_2663_REF])
+          untestableCheck(
+            id,
+            name,
+            description,
+            'no task was created by the preceding step, so this check could not be exercised',
+            [SEP_2663_REF]
+          )
         );
       } else {
         try {
@@ -271,7 +278,13 @@ The server MUST advertise \`io.modelcontextprotocol/tasks\` under
         'Completed task tasks/get inlines result with content[] (no separate tasks/result method)';
       if (!workingTaskId) {
         checks.push(
-          skipCheck(id, name, description, 'no task created', [SEP_2663_REF])
+          untestableCheck(
+            id,
+            name,
+            description,
+            'no task was created by the preceding step, so this check could not be exercised',
+            [SEP_2663_REF]
+          )
         );
       } else {
         try {

src/scenarios/server/tasks/mrtr-helpers.ts

@@ -38,7 +38,14 @@ export function failureCheck(
   };
 }
 
-/** Build a SKIPPED check (preserves id stability so Ctrl+F still finds it). */
+/**
+ * Build a SKIPPED check (preserves id stability so Ctrl+F still finds it).
+ *
+ * Use ONLY for checks that are legitimately not applicable (e.g. a harness
+ * gap the suite itself tracks). A check whose prerequisite is missing on the
+ * server under test must use untestableCheck from src/scenarios/untestable.ts
+ * instead — SKIPPED reads as green in every consumer (issue #248).
+ */
 export function skipCheck(
   id: string,
   name: string,

src/scenarios/server/tasks/wire-fields.ts

@@ -13,8 +13,9 @@
 import { ClientScenario, ConformanceCheck } from '../../../types';
 import type { Connection, RunContext } from '../../../connection';
 import { SEP_2663_REF } from './mrtr-helpers';
-import { errMsg, failureCheck, skipCheck } from './mrtr-helpers';
+import { errMsg, failureCheck } from './mrtr-helpers';
 import { TASKS_EXTENSION_ID, waitForTerminal } from './helpers';
+import { untestableCheck } from '../../untestable';
 
 export class TasksWireFieldsScenario implements ClientScenario {
   name = 'tasks-wire-fields';
@@ -155,7 +156,13 @@ export class TasksWireFieldsScenario implements ClientScenario {
         'Task remains accessible via tasks/get for the duration of its ttlMs';
       if (!createdTaskId) {
         checks.push(
-          skipCheck(id, name, description, 'no task created', [SEP_2663_REF])
+          untestableCheck(
+            id,
+            name,
+            description,
+            'no task was created by the preceding step, so this check could not be exercised',
+            [SEP_2663_REF]
+          )
         );
       } else {
         try {
@@ -204,7 +211,13 @@ export class TasksWireFieldsScenario implements ClientScenario {
         const taskId = created.taskId;
         if (!taskId) {
           checks.push(
-            skipCheck(id, name, description, 'no task created', [SEP_2663_REF])
+            untestableCheck(
+              id,
+              name,
+              description,
+              'no task was created by the preceding step, so this check could not be exercised',
+              [SEP_2663_REF]
+            )
           );
         } else {
           const terminal = await waitForTerminal(conn, taskId);

src/scenarios/untestable.test.ts

@@ -0,0 +1,39 @@
+import { describe, it, expect } from 'vitest';
+import { notTestable, untestableCheck } from './untestable';
+
+describe('untestable helpers', () => {
+  it('notTestable formats the stable prefix', () => {
+    expect(notTestable('server lacks tool X')).toBe(
+      'Not testable: server lacks tool X'
+    );
+  });
+
+  it('untestableCheck defaults to FAILURE and marks details.untestable', () => {
+    const check = untestableCheck(
+      'some-check-id',
+      'SomeCheck',
+      'A MUST requirement',
+      'server lacks the diagnostic tool',
+      [{ id: 'SEP-0000', url: 'https://example.com' }]
+    );
+    expect(check).toMatchObject({
+      id: 'some-check-id',
+      status: 'FAILURE',
+      errorMessage: 'Not testable: server lacks the diagnostic tool',
+      details: { untestable: true }
+    });
+  });
+
+  it('untestableCheck supports WARNING severity for SHOULD requirements', () => {
+    const check = untestableCheck(
+      'should-check-id',
+      'ShouldCheck',
+      'A SHOULD requirement',
+      'missing hook',
+      [{ id: 'SEP-0000', url: 'https://example.com' }],
+      'WARNING'
+    );
+    expect(check.status).toBe('WARNING');
+    expect(check.errorMessage).toBe('Not testable: missing hook');
+  });
+});

src/scenarios/untestable.ts

@@ -0,0 +1,55 @@
+/**
+ * "Untestable" check policy (issue #248).
+ *
+ * A check whose prerequisite is missing — the server under test lacks a
+ * diagnostic fixture tool, rejects the probe that would exercise the
+ * requirement, or advertises a feature it does not actually serve — MUST NOT
+ * report SKIPPED: SKIPPED is excluded from pass/fail counts, exit codes, and
+ * the expected-failures baseline, so the run reads as green and the gap is
+ * invisible to anyone burning down a conformance list. Instead the check
+ * fails, with an errorMessage that names the missing prerequisite, so the
+ * result is red until the prerequisite exists and the scenario can sit in an
+ * expected-failures baseline meanwhile (the documented escape hatch).
+ *
+ * SKIPPED remains correct only for checks that are legitimately not
+ * applicable: an optional capability the server never claimed (e.g. a SHOULD
+ * check gated on `prompts.listChanged` the server did not declare), or
+ * spec-version inapplicability handled by the runner.
+ */
+
+import type { ConformanceCheck, SpecReference } from '../types';
+
+/**
+ * Format the errorMessage for a check that could not be exercised. The
+ * stable prefix lets reports and dashboards distinguish "requirement was
+ * violated" from "requirement could not be verified" without a new status.
+ */
+export function notTestable(reason: string): string {
+  return `Not testable: ${reason}`;
+}
+
+/**
+ * Build a failing check for a requirement that could not be exercised
+ * because a prerequisite is missing. `severity` follows the spec keyword of
+ * the underlying requirement (MUST -> FAILURE, SHOULD -> WARNING), matching
+ * what the check could have reported had it run.
+ */
+export function untestableCheck(
+  id: string,
+  name: string,
+  description: string,
+  reason: string,
+  specReferences: SpecReference[],
+  severity: 'FAILURE' | 'WARNING' = 'FAILURE'
+): ConformanceCheck {
+  return {
+    id,
+    name,
+    description,
+    status: severity,
+    timestamp: new Date().toISOString(),
+    errorMessage: notTestable(reason),
+    specReferences,
+    details: { untestable: true, reason }
+  };
+}