fix: fail checks whose prerequisite is missing instead of skipping them (#372)
A check that cannot be exercised — the server under test lacks a diagnostic fixture tool, or advertises a feature it then rejects — was reported SKIPPED, which is excluded from pass/fail counts, exit codes, and the expected-failures baseline. A server could dodge the entire SEP-2575 capability-constraint and subscription surface (7 MUST-level checks) and still read green, and a baselined failing scenario whose checks degraded to SKIPPED flipped its baseline entry to stale, which reads as fixed. Such checks now fail with a 'Not testable:' message naming the missing prerequisite and details.untestable for tooling, per the policy in issue #248 (severity follows the requirement's keyword: MUST -> FAILURE, SHOULD -> WARNING). Legitimate skips remain: optional capabilities the server never declared, runner-level spec-version inapplicability, and the documented tasks-status-notifications harness gap. Also fixes a test-suite bug found en route: the stream mock's reader lacked releaseLock, so listenToStream's teardown threw and every stream-based assertion silently ran against zero frames.
This commit is contained in:
Felix Weinberger
GitHub
parent
b18aa91850
commit
977813a130
11 changed files with 902 additions and 117 deletions
|
|
@ -55,6 +55,7 @@ Keep scenarios separate when they're genuinely independent features or when they
|
|||
- **Don't reimplement the runner.** New subcommands that need to "select scenarios → run them → print summary → compute exit code" must go through the existing `client` / `server` commands (subprocess via `process.execPath` like `tier-check` and `sdk` do) or call shared helpers — never a parallel suite-map / summary loop.
|
||||
- Include `specReferences` pointing to the relevant spec section.
|
||||
- **Severity follows the spec keyword:** MUST / MUST NOT → `FAILURE`; SHOULD / SHOULD NOT → `WARNING`. (CI treats WARNING as a failure, so Tier-1 SDKs still need to satisfy SHOULDs — see #245.)
|
||||
- **A missing prerequisite is a failure, not a skip** (#248). When a check cannot be exercised — the server under test lacks a diagnostic fixture tool, rejects the probe, or advertises a feature it does not serve — report it failed via `notTestable()` / `untestableCheck()` from `src/scenarios/untestable.ts`, with the missing prerequisite named in `errorMessage` and `details.untestable: true`. SKIPPED reads as green in every consumer (pass counts, exit codes, expected-failures), so it is reserved for checks that are legitimately not applicable: an optional capability the server never declared, spec-version inapplicability handled by the runner, or a documented harness gap the suite itself tracks (e.g. `tasks-status-notifications` pending its subscriptions/listen rewrite). The untestable severity follows the underlying requirement's keyword (MUST → `FAILURE`, SHOULD → `WARNING`); the expected-failures baseline is the escape hatch for SDKs that haven't built the fixture yet.
|
||||
|
||||
## Descriptions and wording
|
||||
|
||||
|
|
|
|||
240
src/scenarios/server/http-standard-headers.test.ts
Normal file
240
src/scenarios/server/http-standard-headers.test.ts
Normal file
|
|
@ -0,0 +1,240 @@
|
|||
import { describe, test, expect, afterEach } from 'vitest';
|
||||
import { testContext } from '../../connection/testing';
|
||||
import {
|
||||
HttpHeaderValidationScenario,
|
||||
HttpCustomHeaderServerValidationScenario,
|
||||
CUSTOM_HEADER_SERVER_DECLARED_CHECK_IDS
|
||||
} from './http-standard-headers';
|
||||
import type { ConformanceCheck } from '../../types';
|
||||
|
||||
/**
|
||||
* Pins the untestable-failure policy (issue #248) for the SEP-2243 server
|
||||
* scenarios: a server that lacks the fixtures these scenarios need must read
|
||||
* red with a "Not testable:" cause, never as a green SKIPPED run.
|
||||
*/
|
||||
|
||||
const realFetch = global.fetch;
|
||||
afterEach(() => {
|
||||
global.fetch = realFetch;
|
||||
});
|
||||
|
||||
function mockFetchTarget(
|
||||
handler: (reqBody: any, reqHeaders: Record<string, string>) => any
|
||||
) {
|
||||
global.fetch = (async (_url: any, init: any) => {
|
||||
const body = JSON.parse(init.body);
|
||||
const headers = init.headers || {};
|
||||
const responseConfig = (await handler(body, headers)) ?? {
|
||||
status: 404,
|
||||
body: {
|
||||
jsonrpc: '2.0',
|
||||
id: body.id,
|
||||
error: { code: -32601, message: 'Not found' }
|
||||
}
|
||||
};
|
||||
const text = JSON.stringify(responseConfig.body);
|
||||
return {
|
||||
status: responseConfig.status ?? 200,
|
||||
headers: { get: () => 'application/json' },
|
||||
json: async () => responseConfig.body,
|
||||
text: async () => text
|
||||
} as unknown as Response;
|
||||
}) as typeof fetch;
|
||||
return 'http://mock-sep2243-server.local';
|
||||
}
|
||||
|
||||
const findAll = (checks: ConformanceCheck[], id: string) =>
|
||||
checks.filter((c) => c.id === id);
|
||||
|
||||
describe('http-custom-header-server-validation — missing fixture policy', () => {
|
||||
test('emits untestable FAILUREs for every declared check when no x-mcp-header tool exists', async () => {
|
||||
const mockUrl = mockFetchTarget((reqBody) => {
|
||||
if (reqBody.method === 'tools/list') {
|
||||
return {
|
||||
status: 200,
|
||||
body: {
|
||||
jsonrpc: '2.0',
|
||||
id: reqBody.id,
|
||||
result: {
|
||||
tools: [
|
||||
{
|
||||
name: 'plain_tool',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: { q: { type: 'string' } }
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
});
|
||||
|
||||
const scenario = new HttpCustomHeaderServerValidationScenario();
|
||||
const checks = await scenario.run(testContext(mockUrl));
|
||||
|
||||
const gate = findAll(checks, 'sep-2243-server-no-xmcp-tool')[0];
|
||||
expect(gate?.status).toBe('FAILURE');
|
||||
expect(gate?.errorMessage).toContain('Not testable:');
|
||||
|
||||
for (const id of CUSTOM_HEADER_SERVER_DECLARED_CHECK_IDS) {
|
||||
const declared = findAll(checks, id)[0];
|
||||
expect(declared?.status, id).toBe('FAILURE');
|
||||
expect(declared?.errorMessage, id).toContain('Not testable:');
|
||||
expect(declared?.details, id).toMatchObject({ untestable: true });
|
||||
}
|
||||
|
||||
// The run must not contain a single SKIPPED row: the whole point is
|
||||
// that this server cannot collect a vacuous green.
|
||||
expect(checks.every((c) => c.status !== 'SKIPPED')).toBe(true);
|
||||
});
|
||||
|
||||
test('emits untestable FAILUREs when the annotated tool has no string parameter', async () => {
|
||||
const mockUrl = mockFetchTarget((reqBody) => {
|
||||
if (reqBody.method === 'tools/list') {
|
||||
return {
|
||||
status: 200,
|
||||
body: {
|
||||
jsonrpc: '2.0',
|
||||
id: reqBody.id,
|
||||
result: {
|
||||
tools: [
|
||||
{
|
||||
name: 'numeric_only',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
level: { type: 'number', 'x-mcp-header': 'Level' }
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
});
|
||||
|
||||
const scenario = new HttpCustomHeaderServerValidationScenario();
|
||||
const checks = await scenario.run(testContext(mockUrl));
|
||||
|
||||
const gate = findAll(checks, 'sep-2243-server-no-string-param')[0];
|
||||
expect(gate?.status).toBe('FAILURE');
|
||||
expect(gate?.errorMessage).toContain('Not testable:');
|
||||
for (const id of CUSTOM_HEADER_SERVER_DECLARED_CHECK_IDS) {
|
||||
expect(findAll(checks, id)[0]?.status, id).toBe('FAILURE');
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('http-header-validation — zero-tools Mcp-Name cases', () => {
|
||||
// This scenario sends its live cases over raw node http (not fetch), so it
|
||||
// is driven against a real local server rather than a fetch mock.
|
||||
async function startBareServer(
|
||||
handler: (body: any) => { status: number; body: object }
|
||||
): Promise<{ url: string; close: () => Promise<void> }> {
|
||||
const http = await import('http');
|
||||
const server = http.createServer((req, res) => {
|
||||
let raw = '';
|
||||
req.setEncoding('utf8');
|
||||
req.on('data', (c) => (raw += c));
|
||||
req.on('end', () => {
|
||||
let body: any = {};
|
||||
try {
|
||||
body = JSON.parse(raw);
|
||||
} catch {
|
||||
// Treat unparseable bodies as empty requests.
|
||||
}
|
||||
const out = handler(body);
|
||||
res.writeHead(out.status, { 'Content-Type': 'application/json' });
|
||||
res.end(JSON.stringify(out.body));
|
||||
});
|
||||
});
|
||||
await new Promise<void>((resolve) => server.listen(0, resolve));
|
||||
const address = server.address();
|
||||
const port = typeof address === 'object' && address ? address.port : 0;
|
||||
return {
|
||||
url: `http://localhost:${port}/mcp`,
|
||||
close: () =>
|
||||
new Promise<void>((resolve) => {
|
||||
server.closeAllConnections?.();
|
||||
server.close(() => resolve());
|
||||
})
|
||||
};
|
||||
}
|
||||
|
||||
test('emits untestable FAILUREs for the Mcp-Name cases when the server lists no tools', async () => {
|
||||
const srv = await startBareServer((body) => {
|
||||
if (body.method === 'tools/list') {
|
||||
return {
|
||||
status: 200,
|
||||
body: { jsonrpc: '2.0', id: body.id, result: { tools: [] } }
|
||||
};
|
||||
}
|
||||
return {
|
||||
status: 400,
|
||||
body: {
|
||||
jsonrpc: '2.0',
|
||||
id: body.id ?? null,
|
||||
error: { code: -32020, message: 'Header mismatch' }
|
||||
}
|
||||
};
|
||||
});
|
||||
|
||||
const scenario = new HttpHeaderValidationScenario();
|
||||
let checks: ConformanceCheck[];
|
||||
try {
|
||||
checks = await scenario.run(testContext(srv.url));
|
||||
} finally {
|
||||
await srv.close();
|
||||
}
|
||||
|
||||
// Previously these three cases were silently omitted; now each surfaces
|
||||
// as an untestable FAILURE naming the missing prerequisite.
|
||||
const whitespace = findAll(
|
||||
checks,
|
||||
'sep-2243-server-accepts-whitespace-header-value'
|
||||
);
|
||||
expect(whitespace.length).toBeGreaterThan(0);
|
||||
expect(whitespace[0].status).toBe('FAILURE');
|
||||
expect(whitespace[0].errorMessage).toContain('Not testable:');
|
||||
|
||||
const nameRejects = findAll(
|
||||
checks,
|
||||
'sep-2243-server-reject-invalid-headers'
|
||||
).filter((c) => c.errorMessage?.startsWith('Not testable:'));
|
||||
expect(nameRejects.map((c) => c.name).sort()).toEqual([
|
||||
'ServerRejectsMismatchedNameHeader',
|
||||
'ServerRejectsMissingNameHeader'
|
||||
]);
|
||||
});
|
||||
|
||||
test('emits untestable FAILUREs for the Mcp-Name cases when tools/list discovery fails', async () => {
|
||||
const srv = await startBareServer((body) => ({
|
||||
status: 500,
|
||||
body: {
|
||||
jsonrpc: '2.0',
|
||||
id: body.id ?? null,
|
||||
error: { code: -32603, message: 'boom' }
|
||||
}
|
||||
}));
|
||||
|
||||
const scenario = new HttpHeaderValidationScenario();
|
||||
let checks: ConformanceCheck[];
|
||||
try {
|
||||
checks = await scenario.run(testContext(srv.url));
|
||||
} finally {
|
||||
await srv.close();
|
||||
}
|
||||
|
||||
const setup = findAll(checks, 'sep-2243-server-standard-setup')[0];
|
||||
expect(setup?.status).toBe('FAILURE');
|
||||
const whitespace = findAll(
|
||||
checks,
|
||||
'sep-2243-server-accepts-whitespace-header-value'
|
||||
)[0];
|
||||
expect(whitespace?.status).toBe('FAILURE');
|
||||
expect(whitespace?.errorMessage).toContain('Not testable:');
|
||||
});
|
||||
});
|
||||
|
|
@ -26,6 +26,7 @@ import {
|
|||
type RunContext
|
||||
} from '../../connection';
|
||||
import { HEADER_MISMATCH } from '../../spec-types/draft';
|
||||
import { untestableCheck } from '../untestable';
|
||||
|
||||
const SPEC_REFERENCE = {
|
||||
id: 'SEP-2243-Server-Validation',
|
||||
|
|
@ -86,6 +87,45 @@ export const CUSTOM_HEADER_SERVER_DECLARED_CHECK_IDS = [
|
|||
'sep-2243-server-reject-param-mismatch'
|
||||
] as const;
|
||||
|
||||
/**
|
||||
* The three Mcp-Name validation cases need a callable tool. They are emitted
|
||||
* from two places: as live test cases when the server lists a tool, and as
|
||||
* untestable failures (issue #248) when it does not — keep the identities in
|
||||
* one table so the two paths cannot drift.
|
||||
*/
|
||||
const MCP_NAME_CASES = [
|
||||
{
|
||||
id: REJECT_STATUS_CHECK_ID,
|
||||
name: 'ServerRejectsMismatchedNameHeader',
|
||||
description:
|
||||
'Server rejects tools/call where Mcp-Name does not match body params.name',
|
||||
specRef: SPEC_REFERENCE
|
||||
},
|
||||
{
|
||||
id: 'sep-2243-server-accepts-whitespace-header-value',
|
||||
name: 'ServerAcceptsWhitespaceHeaderValue',
|
||||
description:
|
||||
'Server MUST accept leading/trailing whitespace in Mcp-Name value (RFC 9110 §5.5: field parsing MUST exclude OWS before evaluating)',
|
||||
specRef: SPEC_REFERENCE_RFC9110_OWS
|
||||
},
|
||||
{
|
||||
id: REJECT_STATUS_CHECK_ID,
|
||||
name: 'ServerRejectsMissingNameHeader',
|
||||
description:
|
||||
'Server MUST reject tools/call with missing Mcp-Name header when body has params.name',
|
||||
specRef: SPEC_REFERENCE
|
||||
}
|
||||
] as const;
|
||||
|
||||
/** Emit the Mcp-Name cases as untestable failures naming the cause. */
|
||||
function untestableMcpNameCases(checks: ConformanceCheck[], reason: string) {
|
||||
for (const c of MCP_NAME_CASES) {
|
||||
checks.push(
|
||||
untestableCheck(c.id, c.name, c.description, reason, [c.specRef])
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Helper to send a raw HTTP POST request with custom headers.
|
||||
* Uses Node.js http.request to preserve exact header casing and values,
|
||||
|
|
@ -294,6 +334,10 @@ export class HttpHeaderValidationScenario implements ClientScenario {
|
|||
specReferences: [SPEC_REFERENCE],
|
||||
details: { httpStatus: toolsResponse.status, error: rpcError }
|
||||
});
|
||||
untestableMcpNameCases(
|
||||
checks,
|
||||
'tools/list discovery failed, so the Mcp-Name header-validation cases could not be exercised'
|
||||
);
|
||||
return checks;
|
||||
}
|
||||
const toolsResult = toolsResponse.body.result as {
|
||||
|
|
@ -417,6 +461,14 @@ export class HttpHeaderValidationScenario implements ClientScenario {
|
|||
'Standard header omitted but value present in body → MUST reject'
|
||||
}
|
||||
);
|
||||
} else {
|
||||
// No tools to exercise the Mcp-Name dimension: these MUST cases were
|
||||
// previously omitted without a trace. Emit untestable failures so
|
||||
// the gap is visible (issue #248).
|
||||
untestableMcpNameCases(
|
||||
checks,
|
||||
'server lists no tools, so the Mcp-Name header-validation cases could not be exercised'
|
||||
);
|
||||
}
|
||||
|
||||
// --- Case Sensitivity Tests ---
|
||||
|
|
@ -608,22 +660,18 @@ export class HttpCustomHeaderServerValidationScenario implements ClientScenario
|
|||
});
|
||||
|
||||
if (!xMcpTool) {
|
||||
checks.push({
|
||||
id: 'sep-2243-server-no-xmcp-tool',
|
||||
name: 'HttpCustomHeaderServerNoTool',
|
||||
description:
|
||||
checks.push(
|
||||
untestableCheck(
|
||||
'sep-2243-server-no-xmcp-tool',
|
||||
'HttpCustomHeaderServerNoTool',
|
||||
'Server has no tools with x-mcp-header annotations to test',
|
||||
status: 'SKIPPED',
|
||||
timestamp: new Date().toISOString(),
|
||||
specReferences: [SPEC_REFERENCE_CUSTOM],
|
||||
details: {
|
||||
reason:
|
||||
'No tools with x-mcp-header found. These tests require at least one tool with x-mcp-header annotations.'
|
||||
}
|
||||
});
|
||||
this.skipDeclaredChecks(
|
||||
'server exposes no tool with x-mcp-header annotations, so none of the custom-header validation requirements could be exercised',
|
||||
[SPEC_REFERENCE_CUSTOM]
|
||||
)
|
||||
);
|
||||
this.untestableDeclaredChecks(
|
||||
checks,
|
||||
'Server exposes no tool with x-mcp-header annotations.'
|
||||
'server exposes no tool with x-mcp-header annotations'
|
||||
);
|
||||
return checks;
|
||||
}
|
||||
|
|
@ -640,18 +688,18 @@ export class HttpCustomHeaderServerValidationScenario implements ClientScenario
|
|||
def['x-mcp-header'] !== undefined && (def as any).type === 'string'
|
||||
);
|
||||
if (!annotatedEntry) {
|
||||
checks.push({
|
||||
id: 'sep-2243-server-no-string-param',
|
||||
name: 'HttpCustomHeaderServerNoStringParam',
|
||||
description:
|
||||
checks.push(
|
||||
untestableCheck(
|
||||
'sep-2243-server-no-string-param',
|
||||
'HttpCustomHeaderServerNoStringParam',
|
||||
'Server has no string-typed x-mcp-header parameter to test',
|
||||
status: 'SKIPPED',
|
||||
timestamp: new Date().toISOString(),
|
||||
specReferences: [SPEC_REFERENCE_CUSTOM]
|
||||
});
|
||||
this.skipDeclaredChecks(
|
||||
'server exposes no string-typed x-mcp-header parameter, so the custom-header validation requirements could not be exercised',
|
||||
[SPEC_REFERENCE_CUSTOM]
|
||||
)
|
||||
);
|
||||
this.untestableDeclaredChecks(
|
||||
checks,
|
||||
'Server exposes no string-typed x-mcp-header parameter.'
|
||||
'server exposes no string-typed x-mcp-header parameter'
|
||||
);
|
||||
return checks;
|
||||
}
|
||||
|
|
@ -818,7 +866,7 @@ export class HttpCustomHeaderServerValidationScenario implements ClientScenario
|
|||
}
|
||||
|
||||
// Declared-but-unemitted -> FAILURE. Reached only when setup threw partway
|
||||
// through (the gate-out paths emit SKIPPED rows and the happy path emits
|
||||
// through (the gate-out paths emit untestable FAILURE rows and the happy path emits
|
||||
// every declared ID).
|
||||
this.failDeclaredChecks(checks);
|
||||
|
||||
|
|
@ -846,23 +894,27 @@ export class HttpCustomHeaderServerValidationScenario implements ClientScenario
|
|||
}
|
||||
|
||||
/**
|
||||
* Emit one SKIPPED row per declared requirement check when the scenario
|
||||
* cannot run against this server (no x-mcp-header tool). The IDs must still
|
||||
* reach checks.json so the traceability manifest records that a scenario
|
||||
* exists for each requirement.
|
||||
* Emit one untestable FAILURE row per declared requirement check when the
|
||||
* scenario cannot run against this server (no x-mcp-header tool). These
|
||||
* are MUST requirements, so a missing fixture must read red, not green
|
||||
* (issue #248); the IDs still reach checks.json so the traceability
|
||||
* manifest records that a scenario exists for each requirement.
|
||||
*/
|
||||
private skipDeclaredChecks(checks: ConformanceCheck[], reason: string): void {
|
||||
private untestableDeclaredChecks(
|
||||
checks: ConformanceCheck[],
|
||||
reason: string
|
||||
): void {
|
||||
for (const id of CUSTOM_HEADER_SERVER_DECLARED_CHECK_IDS) {
|
||||
if (checks.some((c) => c.id === id)) continue;
|
||||
checks.push({
|
||||
id,
|
||||
name: 'NotApplicable',
|
||||
description: `Declared check ${id} is not testable against this server`,
|
||||
status: 'SKIPPED',
|
||||
timestamp: new Date().toISOString(),
|
||||
specReferences: [SPEC_REFERENCE_CUSTOM],
|
||||
details: { reason }
|
||||
});
|
||||
checks.push(
|
||||
untestableCheck(
|
||||
id,
|
||||
'NotTestable',
|
||||
`Declared check ${id} is not testable against this server`,
|
||||
reason,
|
||||
[SPEC_REFERENCE_CUSTOM]
|
||||
)
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -66,7 +66,13 @@ describe('Stateless Server Scenario Negative Tests', () => {
|
|||
}
|
||||
const chunk = new TextEncoder().encode(streamData[frameIndex++]);
|
||||
return { value: chunk, done: false };
|
||||
}
|
||||
},
|
||||
// Required by listenToStream's finally block; without it the
|
||||
// reader teardown throws, the stream helper swallows the error,
|
||||
// and every stream-based assertion silently runs against zero
|
||||
// frames — the exact vacuous-pass failure mode this suite exists
|
||||
// to prevent.
|
||||
releaseLock: () => {}
|
||||
};
|
||||
|
||||
return {
|
||||
|
|
@ -91,6 +97,24 @@ describe('Stateless Server Scenario Negative Tests', () => {
|
|||
return 'http://mock-stateless-mcp-server.local';
|
||||
}
|
||||
|
||||
// Shared server/discover stanza for the negative-server mocks below.
|
||||
const discoverResponse = (
|
||||
reqBody: any,
|
||||
capabilities: object,
|
||||
serverName: string
|
||||
) => ({
|
||||
status: 200,
|
||||
body: {
|
||||
jsonrpc: '2.0',
|
||||
id: reqBody.id,
|
||||
result: {
|
||||
supportedVersions: ['2026-07-28'],
|
||||
capabilities,
|
||||
serverInfo: { name: serverName, version: '1.0.0' }
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test('Fails validation if missing required fields in _meta are allowed to pass', async () => {
|
||||
// This bad server completely ignores missing params/_meta fields and returns a fake success result
|
||||
const mockUrl = mockFetchTarget((reqBody) => {
|
||||
|
|
@ -127,6 +151,190 @@ describe('Stateless Server Scenario Negative Tests', () => {
|
|||
expect(missingVersionCheck?.status).toBe('FAILURE');
|
||||
});
|
||||
|
||||
test('Fails (untestable) the capability checks when the diagnostic tool is not listed', async () => {
|
||||
// This server has no test_missing_capability tool at all: the
|
||||
// undeclared-capability MUSTs cannot be exercised and must read red,
|
||||
// not SKIPPED (issue #248).
|
||||
const mockUrl = mockFetchTarget((reqBody) => {
|
||||
if (reqBody.method === 'server/discover') {
|
||||
return discoverResponse(reqBody, { tools: {} }, 'no-diagnostic-tools');
|
||||
}
|
||||
if (reqBody.method === 'tools/list') {
|
||||
return {
|
||||
status: 200,
|
||||
body: { jsonrpc: '2.0', id: reqBody.id, result: { tools: [] } }
|
||||
};
|
||||
}
|
||||
});
|
||||
|
||||
const scenario = new ServerStatelessScenario();
|
||||
const checks = await scenario.run(testContext(mockUrl));
|
||||
|
||||
const rejectCheck = findCheck(
|
||||
checks,
|
||||
'sep-2575-server-rejects-undeclared-capability'
|
||||
);
|
||||
const statusCheck = findCheck(
|
||||
checks,
|
||||
'sep-2575-missing-capability-http-400'
|
||||
);
|
||||
|
||||
expect(rejectCheck?.status).toBe('FAILURE');
|
||||
expect(rejectCheck?.errorMessage).toContain('Not testable:');
|
||||
expect(rejectCheck?.errorMessage).toContain('test_missing_capability');
|
||||
expect(statusCheck?.status).toBe('FAILURE');
|
||||
expect(statusCheck?.errorMessage).toContain('Not testable:');
|
||||
});
|
||||
|
||||
test('Fails the capability check when the listed diagnostic tool executes without -32021', async () => {
|
||||
// This server lists test_missing_capability but happily executes it even
|
||||
// though the client never declared the sampling capability — a genuine
|
||||
// violation of the MUST, not an untestable gap.
|
||||
const mockUrl = mockFetchTarget((reqBody) => {
|
||||
if (reqBody.method === 'server/discover') {
|
||||
return discoverResponse(reqBody, { tools: {} }, 'no-enforcement');
|
||||
}
|
||||
if (reqBody.method === 'tools/list') {
|
||||
return {
|
||||
status: 200,
|
||||
body: {
|
||||
jsonrpc: '2.0',
|
||||
id: reqBody.id,
|
||||
result: { tools: [{ name: 'test_missing_capability' }] }
|
||||
}
|
||||
};
|
||||
}
|
||||
if (
|
||||
reqBody.method === 'tools/call' &&
|
||||
reqBody.params?.name === 'test_missing_capability'
|
||||
) {
|
||||
return {
|
||||
status: 200,
|
||||
body: {
|
||||
jsonrpc: '2.0',
|
||||
id: reqBody.id,
|
||||
result: { resultType: 'complete', content: [] }
|
||||
}
|
||||
};
|
||||
}
|
||||
});
|
||||
|
||||
const scenario = new ServerStatelessScenario();
|
||||
const checks = await scenario.run(testContext(mockUrl));
|
||||
|
||||
const rejectCheck = findCheck(
|
||||
checks,
|
||||
'sep-2575-server-rejects-undeclared-capability'
|
||||
);
|
||||
expect(rejectCheck?.status).toBe('FAILURE');
|
||||
expect(rejectCheck?.errorMessage).toContain('MUST reject with -32021');
|
||||
});
|
||||
|
||||
test('Fails the subscription checks when listChanged is advertised but listen is rejected', async () => {
|
||||
const mockUrl = mockFetchTarget((reqBody) => {
|
||||
if (reqBody.method === 'server/discover') {
|
||||
return discoverResponse(
|
||||
reqBody,
|
||||
{ tools: { listChanged: true } },
|
||||
'claims-subscriptions'
|
||||
);
|
||||
}
|
||||
if (reqBody.method === 'subscriptions/listen') {
|
||||
return {
|
||||
isStream: true,
|
||||
status: 404,
|
||||
frames: [
|
||||
{
|
||||
jsonrpc: '2.0',
|
||||
id: reqBody.id,
|
||||
error: { code: -32601, message: 'Method not found' }
|
||||
}
|
||||
]
|
||||
};
|
||||
}
|
||||
});
|
||||
|
||||
const scenario = new ServerStatelessScenario();
|
||||
const checks = await scenario.run(testContext(mockUrl));
|
||||
|
||||
const ackCheck = findCheck(
|
||||
checks,
|
||||
'sep-2575-server-sends-subscription-ack'
|
||||
);
|
||||
expect(ackCheck?.status).toBe('FAILURE');
|
||||
expect(ackCheck?.errorMessage).toContain('Not testable:');
|
||||
expect(ackCheck?.errorMessage).toContain('-32601');
|
||||
});
|
||||
|
||||
test('Skips the subscription checks when no subscription capability is advertised', async () => {
|
||||
// A server that never claimed listChanged/subscribe legitimately has
|
||||
// nothing to serve on subscriptions/listen — this is the one case that
|
||||
// stays SKIPPED.
|
||||
const mockUrl = mockFetchTarget((reqBody) => {
|
||||
if (reqBody.method === 'server/discover') {
|
||||
return discoverResponse(reqBody, { tools: {} }, 'no-subscriptions');
|
||||
}
|
||||
if (reqBody.method === 'subscriptions/listen') {
|
||||
return {
|
||||
isStream: true,
|
||||
status: 404,
|
||||
frames: [
|
||||
{
|
||||
jsonrpc: '2.0',
|
||||
id: reqBody.id,
|
||||
error: { code: -32601, message: 'Method not found' }
|
||||
}
|
||||
]
|
||||
};
|
||||
}
|
||||
});
|
||||
|
||||
const scenario = new ServerStatelessScenario();
|
||||
const checks = await scenario.run(testContext(mockUrl));
|
||||
|
||||
const ackCheck = findCheck(
|
||||
checks,
|
||||
'sep-2575-server-sends-subscription-ack'
|
||||
);
|
||||
const tagCheck = findCheck(checks, 'sep-2575-server-tags-subscription-id');
|
||||
expect(ackCheck?.status).toBe('SKIPPED');
|
||||
expect(tagCheck?.status).toBe('SKIPPED');
|
||||
});
|
||||
|
||||
test('Fails (untestable) the subscription checks when discover was never observed', async () => {
|
||||
// With no server/discover response at all, a -32601 on listen cannot be
|
||||
// attributed to an intentionally absent capability — the legitimate skip
|
||||
// requires an observed advertisement.
|
||||
const mockUrl = mockFetchTarget((reqBody) => {
|
||||
if (reqBody.method === 'subscriptions/listen') {
|
||||
return {
|
||||
isStream: true,
|
||||
status: 404,
|
||||
frames: [
|
||||
{
|
||||
jsonrpc: '2.0',
|
||||
id: reqBody.id,
|
||||
error: { code: -32601, message: 'Method not found' }
|
||||
}
|
||||
]
|
||||
};
|
||||
}
|
||||
});
|
||||
|
||||
const scenario = new ServerStatelessScenario();
|
||||
const checks = await scenario.run(testContext(mockUrl));
|
||||
|
||||
const ackCheck = findCheck(
|
||||
checks,
|
||||
'sep-2575-server-sends-subscription-ack'
|
||||
);
|
||||
expect(ackCheck?.status).toBe('FAILURE');
|
||||
expect(ackCheck?.errorMessage).toContain('Not testable:');
|
||||
expect(ackCheck?.errorMessage).toContain(
|
||||
'server/discover was not observed'
|
||||
);
|
||||
});
|
||||
|
||||
test('Fails validation when missing-_meta rejections are returned with HTTP 200', async () => {
|
||||
// This bad server picks the right JSON-RPC error code but the wrong HTTP
|
||||
// status: the spec requires 400 Bad Request for malformed requests.
|
||||
|
|
|
|||
|
|
@ -12,6 +12,7 @@ import {
|
|||
readSseJsonRpcResponse,
|
||||
type RunContext
|
||||
} from '../../connection';
|
||||
import { notTestable } from '../untestable';
|
||||
|
||||
const SPEC_REF = [
|
||||
{
|
||||
|
|
@ -114,6 +115,10 @@ export class ServerStatelessScenario implements ClientScenario {
|
|||
}
|
||||
}
|
||||
|
||||
// Every JSON-RPC error response observed anywhere in the scenario counts
|
||||
// toward the id-echo backfill's "did we see any error at all" condition.
|
||||
let errorResponsesObserved = 0;
|
||||
|
||||
// Helper to send raw RPC requests via fetch
|
||||
const sendRpc = async (
|
||||
method: string,
|
||||
|
|
@ -156,6 +161,7 @@ export class ServerStatelessScenario implements ClientScenario {
|
|||
// Response might not be JSON
|
||||
}
|
||||
}
|
||||
if (data?.error !== undefined) errorResponsesObserved++;
|
||||
return { res, data };
|
||||
};
|
||||
|
||||
|
|
@ -483,6 +489,22 @@ export class ServerStatelessScenario implements ClientScenario {
|
|||
}
|
||||
);
|
||||
|
||||
// Fetched once here and reused by the capability-constraint checks below
|
||||
// to tell a missing diagnostic tool apart from a genuine violation.
|
||||
const toolsListResponse = await sendRpc(
|
||||
'tools/list',
|
||||
{ _meta: validMeta },
|
||||
undefined,
|
||||
202
|
||||
).catch(() => null);
|
||||
const listedToolNames: string[] = Array.isArray(
|
||||
toolsListResponse?.data?.result?.tools
|
||||
)
|
||||
? toolsListResponse.data.result.tools
|
||||
.map((t: any) => t?.name)
|
||||
.filter((n: any) => typeof n === 'string')
|
||||
: [];
|
||||
|
||||
// Dynamic verification helper to check capability consistency against true handlers
|
||||
await runCheck(
|
||||
'sep-2575-discover-capabilities-match-handlers',
|
||||
|
|
@ -493,12 +515,9 @@ export class ServerStatelessScenario implements ClientScenario {
|
|||
return {
|
||||
error: `Discovery runtime check failed: ${discoverRpcError.message}`
|
||||
};
|
||||
const { data: toolsData } = await sendRpc(
|
||||
'tools/list',
|
||||
{ _meta: validMeta },
|
||||
undefined,
|
||||
202
|
||||
);
|
||||
if (!toolsListResponse)
|
||||
return { error: 'tools/list probe network hit failed' };
|
||||
const toolsData = toolsListResponse.data;
|
||||
|
||||
if (discoverCapabilities.tools) {
|
||||
const toolsPassed = Array.isArray(toolsData?.result?.tools);
|
||||
|
|
@ -632,6 +651,9 @@ export class ServerStatelessScenario implements ClientScenario {
|
|||
|
||||
// Determine if this server actively enforces client capabilities
|
||||
const serverRequiresCapability = data401?.error?.code === -32021;
|
||||
const capabilityToolListed = listedToolNames.includes(
|
||||
'test_missing_capability'
|
||||
);
|
||||
|
||||
await runCheck(
|
||||
'sep-2575-server-rejects-undeclared-capability',
|
||||
|
|
@ -645,14 +667,29 @@ export class ServerStatelessScenario implements ClientScenario {
|
|||
};
|
||||
|
||||
if (!serverRequiresCapability) {
|
||||
// The server didn't return -32021, so this requirement isn't
|
||||
// exercised for this method. Report SKIPPED rather than a green PASS.
|
||||
// Distinguish a genuine violation from a fixture gap, strongest
|
||||
// evidence first: a result proves the tool executed (violation
|
||||
// regardless of whether tools/list mentioned it); a non--32021
|
||||
// error from a listed tool is the wrong code; an error from an
|
||||
// unlisted tool means the requirement could not be exercised.
|
||||
if (data401?.result !== undefined) {
|
||||
return {
|
||||
error:
|
||||
"Server executed 'test_missing_capability' although the client did not declare the 'sampling' capability — it MUST reject with -32021",
|
||||
details: { response: data401 }
|
||||
};
|
||||
}
|
||||
if (data401?.error && capabilityToolListed) {
|
||||
return {
|
||||
error: `Expected MissingRequiredClientCapabilityError (-32021), got error code ${data401.error.code}`,
|
||||
details: { response: data401 }
|
||||
};
|
||||
}
|
||||
return {
|
||||
skipped: true,
|
||||
details: {
|
||||
note: 'Skipped requirement tracking: Server returned a non-32021 response, indicating it does not require explicit client capability authorization constraints for this method.',
|
||||
response: data401
|
||||
}
|
||||
error: notTestable(
|
||||
"server does not list the diagnostic tool 'test_missing_capability' in tools/list and the probe call did not exercise it (required for the undeclared-capability rejection)"
|
||||
),
|
||||
details: { untestable: true, response: data401 }
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -680,14 +717,16 @@ export class ServerStatelessScenario implements ClientScenario {
|
|||
};
|
||||
|
||||
if (!serverRequiresCapability) {
|
||||
// No -32021 means the HTTP-400 requirement doesn't apply here.
|
||||
// Report SKIPPED rather than a green PASS.
|
||||
// Same root cause as the previous check: either the diagnostic
|
||||
// tool is missing (untestable) or the server failed to emit the
|
||||
// -32021 this status requirement attaches to.
|
||||
return {
|
||||
skipped: true,
|
||||
details: {
|
||||
note: 'Skipped status tracking: Server did not return a MissingRequiredClientCapabilityError.',
|
||||
httpStatus: res401.status
|
||||
}
|
||||
error: notTestable(
|
||||
capabilityToolListed
|
||||
? 'server did not return MissingRequiredClientCapabilityError, so its HTTP status could not be validated'
|
||||
: "server does not list the diagnostic tool 'test_missing_capability' in tools/list, so the -32021 HTTP status could not be validated"
|
||||
),
|
||||
details: { untestable: true, httpStatus: res401.status }
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -795,16 +834,17 @@ export class ServerStatelessScenario implements ClientScenario {
|
|||
};
|
||||
}
|
||||
|
||||
// If the call was rejected outright (e.g. the diagnostic tool does
|
||||
// not exist), nothing was streamed and the requirement was not
|
||||
// exercised - report SKIPPED rather than a vacuous SUCCESS.
|
||||
// If the call was rejected outright, nothing was streamed and the
|
||||
// requirement could not be exercised — fail with the cause rather
|
||||
// than reporting a vacuous SUCCESS or an invisible SKIPPED.
|
||||
if (frames.every((f) => f?.error !== undefined)) {
|
||||
return {
|
||||
skipped: true,
|
||||
details: {
|
||||
note: 'Server does not expose diagnostic tool test_streaming_elicitation; the response stream could not be exercised.',
|
||||
response: frames[0]
|
||||
}
|
||||
error: notTestable(
|
||||
listedToolNames.includes('test_streaming_elicitation')
|
||||
? `the 'test_streaming_elicitation' call was rejected (code ${frames[0]?.error?.code}), so the response stream could not be exercised`
|
||||
: "server does not list the diagnostic tool 'test_streaming_elicitation' in tools/list, so the response stream could not be exercised"
|
||||
),
|
||||
details: { untestable: true, response: frames[0] }
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -841,17 +881,17 @@ export class ServerStatelessScenario implements ClientScenario {
|
|||
};
|
||||
}
|
||||
|
||||
// If the call was rejected outright (e.g. the diagnostic tool does
|
||||
// not exist), the server never had anything to log and the
|
||||
// requirement was not exercised - report SKIPPED rather than a
|
||||
// vacuous SUCCESS.
|
||||
// If the call was rejected outright, the server never had anything
|
||||
// to log and the requirement could not be exercised — fail with the
|
||||
// cause rather than reporting a vacuous SUCCESS or invisible SKIPPED.
|
||||
if (frames.every((f) => f?.error !== undefined)) {
|
||||
return {
|
||||
skipped: true,
|
||||
details: {
|
||||
note: 'Server does not expose diagnostic tool test_logging_tool; the no-log-without-logLevel requirement could not be exercised.',
|
||||
response: frames[0]
|
||||
}
|
||||
error: notTestable(
|
||||
listedToolNames.includes('test_logging_tool')
|
||||
? `the 'test_logging_tool' call was rejected (code ${frames[0]?.error?.code}), so the no-log-without-logLevel requirement could not be exercised`
|
||||
: "server does not list the diagnostic tool 'test_logging_tool' in tools/list, so the no-log-without-logLevel requirement could not be exercised"
|
||||
),
|
||||
details: { untestable: true, response: frames[0] }
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -872,6 +912,41 @@ export class ServerStatelessScenario implements ClientScenario {
|
|||
// ==========================================
|
||||
// 6. Subscription Streams & Filtering (3 Checks)
|
||||
// ==========================================
|
||||
// A server that advertises no subscription-delivered capability has
|
||||
// nothing to serve on subscriptions/listen, so a -32601 there is a
|
||||
// legitimate feature absence (SKIPPED). A server that DOES advertise
|
||||
// listChanged/subscribe but rejects the method fails: it claims a
|
||||
// feature it does not serve.
|
||||
const advertisesSubscriptions = !!(
|
||||
discoverCapabilities?.tools?.listChanged ||
|
||||
discoverCapabilities?.prompts?.listChanged ||
|
||||
discoverCapabilities?.resources?.listChanged ||
|
||||
discoverCapabilities?.resources?.subscribe
|
||||
);
|
||||
// The legitimate skip requires an OBSERVED advertisement: when
|
||||
// server/discover itself failed, nothing is known about the server's
|
||||
// capabilities and the gap must not read as an intentional absence.
|
||||
const discoverObserved = !discoverRpcError && discoverResult != null;
|
||||
const listenRejected = (frames: any[]) => {
|
||||
if (frames[0]?.error?.code !== -32601) return null;
|
||||
if (discoverObserved && !advertisesSubscriptions) {
|
||||
return {
|
||||
skipped: true,
|
||||
details: {
|
||||
note: 'Server advertises no subscription-delivered capability; subscriptions/listen is not applicable.'
|
||||
}
|
||||
};
|
||||
}
|
||||
return {
|
||||
error: notTestable(
|
||||
advertisesSubscriptions
|
||||
? 'server advertises listChanged/subscribe capabilities but answers subscriptions/listen with -32601 (Method not found)'
|
||||
: 'server/discover was not observed, so the -32601 on subscriptions/listen cannot be attributed to an intentionally absent capability'
|
||||
),
|
||||
details: { untestable: true, response: frames[0] }
|
||||
};
|
||||
};
|
||||
|
||||
const subscriptionParams = {
|
||||
_meta: validMeta,
|
||||
notifications: { toolsListChanged: true }
|
||||
|
|
@ -904,14 +979,8 @@ export class ServerStatelessScenario implements ClientScenario {
|
|||
'ServerSendsSubscriptionAck',
|
||||
'notifications/subscriptions/acknowledged is the first message on a subscriptions/listen stream',
|
||||
() => {
|
||||
if (streamFrames[0]?.error?.code === -32601) {
|
||||
return {
|
||||
skipped: true,
|
||||
details: {
|
||||
note: 'Server does not support subscriptions/listen (Method not found)'
|
||||
}
|
||||
};
|
||||
}
|
||||
const rejectedStream = listenRejected(streamFrames);
|
||||
if (rejectedStream) return rejectedStream;
|
||||
if (streamFrames.length === 0) {
|
||||
return {
|
||||
error:
|
||||
|
|
@ -934,14 +1003,8 @@ export class ServerStatelessScenario implements ClientScenario {
|
|||
'ServerTagsSubscriptionId',
|
||||
'Listen-stream notifications carry _meta.../subscriptionId',
|
||||
() => {
|
||||
if (streamFrames[0]?.error?.code === -32601) {
|
||||
return {
|
||||
skipped: true,
|
||||
details: {
|
||||
note: 'Server does not support subscriptions/listen (Method not found)'
|
||||
}
|
||||
};
|
||||
}
|
||||
const rejectedStream = listenRejected(streamFrames);
|
||||
if (rejectedStream) return rejectedStream;
|
||||
if (streamFrames.length === 0) {
|
||||
return {
|
||||
error:
|
||||
|
|
@ -1015,14 +1078,8 @@ export class ServerStatelessScenario implements ClientScenario {
|
|||
}
|
||||
);
|
||||
|
||||
if (narrowFrames[0]?.error?.code === -32601) {
|
||||
return {
|
||||
skipped: true,
|
||||
details: {
|
||||
note: 'Server does not support subscriptions/listen (Method not found)'
|
||||
}
|
||||
};
|
||||
}
|
||||
const rejectedNarrow = listenRejected(narrowFrames);
|
||||
if (rejectedNarrow) return rejectedNarrow;
|
||||
if (narrowFrames.length === 0) {
|
||||
return {
|
||||
error:
|
||||
|
|
@ -1086,11 +1143,13 @@ export class ServerStatelessScenario implements ClientScenario {
|
|||
}
|
||||
);
|
||||
if (trigger?.data?.error?.code === -32601) {
|
||||
// SHOULD-level requirement, so the untestable report is a WARNING.
|
||||
return {
|
||||
skipped: true,
|
||||
details: {
|
||||
note: 'Server does not expose diagnostic hook test_trigger_prompt_change to mutate lists'
|
||||
}
|
||||
warning: true,
|
||||
error: notTestable(
|
||||
"server declares prompts.listChanged but does not expose the diagnostic hook 'test_trigger_prompt_change' to mutate the prompt list"
|
||||
),
|
||||
details: { untestable: true }
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -1154,11 +1213,13 @@ export class ServerStatelessScenario implements ClientScenario {
|
|||
}
|
||||
);
|
||||
if (trigger?.data?.error?.code === -32601) {
|
||||
// SHOULD-level requirement, so the untestable report is a WARNING.
|
||||
return {
|
||||
skipped: true,
|
||||
details: {
|
||||
note: 'Server does not expose diagnostic hook test_trigger_tool_change to mutate lists'
|
||||
}
|
||||
warning: true,
|
||||
error: notTestable(
|
||||
"server declares tools.listChanged but does not expose the diagnostic hook 'test_trigger_tool_change' to mutate the tool list"
|
||||
),
|
||||
details: { untestable: true }
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -1185,12 +1246,26 @@ export class ServerStatelessScenario implements ClientScenario {
|
|||
);
|
||||
|
||||
if (!checks.some((c) => c.id === 'sep-2575-http-server-error-jsonrpc-id')) {
|
||||
// Only report SUCCESS when at least one error response was actually
|
||||
// observed; a server that never produced an error response (because it
|
||||
// wrongly accepted every invalid probe) must not collect a vacuous pass
|
||||
// for the id-echo requirement on top of its other failures.
|
||||
checks.push({
|
||||
id: 'sep-2575-http-server-error-jsonrpc-id',
|
||||
name: 'HttpServerErrorJsonrpcId',
|
||||
description: 'All error responses carry the request JSON-RPC id',
|
||||
status: 'SUCCESS',
|
||||
status: errorResponsesObserved > 0 ? 'SUCCESS' : 'FAILURE',
|
||||
timestamp,
|
||||
errorMessage:
|
||||
errorResponsesObserved > 0
|
||||
? undefined
|
||||
: notTestable(
|
||||
'no error responses were observed across the probes, so the id-echo requirement could not be validated'
|
||||
),
|
||||
details:
|
||||
errorResponsesObserved > 0
|
||||
? { errorResponsesObserved }
|
||||
: { untestable: true, errorResponsesObserved },
|
||||
specReferences: SPEC_REF
|
||||
});
|
||||
}
|
||||
|
|
|
|||
82
src/scenarios/server/tasks/lifecycle.test.ts
Normal file
82
src/scenarios/server/tasks/lifecycle.test.ts
Normal file
|
|
@ -0,0 +1,82 @@
|
|||
import { describe, test, expect, afterEach } from 'vitest';
|
||||
import { testContext } from '../../../connection/testing';
|
||||
import { TasksLifecycleScenario } from './lifecycle';
|
||||
import { DRAFT_PROTOCOL_VERSION } from '../../../types';
|
||||
import type { ConformanceCheck } from '../../../types';
|
||||
|
||||
/**
|
||||
* Pins the untestable-failure policy (issue #248) for the SEP-2663 lifecycle
|
||||
* scenario: when no task is ever created, the downstream task checks must
|
||||
* fail with a "Not testable:" cause instead of reporting SKIPPED.
|
||||
*/
|
||||
|
||||
const realFetch = global.fetch;
|
||||
afterEach(() => {
|
||||
global.fetch = realFetch;
|
||||
});
|
||||
|
||||
function mockServer() {
|
||||
global.fetch = (async (_url: any, init: any) => {
|
||||
const body = JSON.parse(init.body);
|
||||
let result: any;
|
||||
if (body.method === 'server/discover') {
|
||||
result = {
|
||||
supportedVersions: [DRAFT_PROTOCOL_VERSION],
|
||||
capabilities: { tools: {} },
|
||||
serverInfo: { name: 'taskless-server', version: '1.0.0' }
|
||||
};
|
||||
} else if (body.method === 'tools/list') {
|
||||
result = { tools: [] };
|
||||
} else if (body.method === 'tools/call') {
|
||||
// Always answers synchronously: never creates a task.
|
||||
result = {
|
||||
resultType: 'complete',
|
||||
content: [{ type: 'text', text: 'sync answer' }]
|
||||
};
|
||||
} else {
|
||||
return {
|
||||
status: 404,
|
||||
headers: { get: () => 'application/json' },
|
||||
json: async () => ({
|
||||
jsonrpc: '2.0',
|
||||
id: body.id ?? null,
|
||||
error: { code: -32601, message: 'Method not found' }
|
||||
}),
|
||||
text: async () =>
|
||||
JSON.stringify({
|
||||
jsonrpc: '2.0',
|
||||
id: body.id ?? null,
|
||||
error: { code: -32601, message: 'Method not found' }
|
||||
})
|
||||
} as unknown as Response;
|
||||
}
|
||||
const payload = { jsonrpc: '2.0', id: body.id, result };
|
||||
return {
|
||||
status: 200,
|
||||
headers: { get: () => 'application/json' },
|
||||
json: async () => payload,
|
||||
text: async () => JSON.stringify(payload)
|
||||
} as unknown as Response;
|
||||
}) as typeof fetch;
|
||||
return 'http://mock-taskless-server.local';
|
||||
}
|
||||
|
||||
describe('tasks-lifecycle — no task created', () => {
|
||||
test('downstream task checks fail as untestable instead of SKIPPED', async () => {
|
||||
const mockUrl = mockServer();
|
||||
const scenario = new TasksLifecycleScenario();
|
||||
const checks: ConformanceCheck[] = await scenario.run(
|
||||
testContext(mockUrl, DRAFT_PROTOCOL_VERSION)
|
||||
);
|
||||
|
||||
const gated = checks.filter((c) =>
|
||||
c.errorMessage?.startsWith('Not testable:')
|
||||
);
|
||||
expect(gated.length).toBeGreaterThan(0);
|
||||
for (const check of gated) {
|
||||
expect(check.status).toBe('FAILURE');
|
||||
expect(check.details).toMatchObject({ untestable: true });
|
||||
}
|
||||
expect(checks.every((c) => c.status !== 'SKIPPED')).toBe(true);
|
||||
});
|
||||
});
|
||||
|
|
@ -16,8 +16,9 @@
|
|||
import { ClientScenario, ConformanceCheck } from '../../../types';
|
||||
import type { Connection, RunContext } from '../../../connection';
|
||||
import { SEP_2322_REF, SEP_2663_REF } from './mrtr-helpers';
|
||||
import { errMsg, failureCheck, skipCheck } from './mrtr-helpers';
|
||||
import { errMsg, failureCheck } from './mrtr-helpers';
|
||||
import { TASKS_EXTENSION_ID, waitForTerminal } from './helpers';
|
||||
import { untestableCheck } from '../../untestable';
|
||||
import { isIso8601 } from './mrtr-helpers';
|
||||
|
||||
export class TasksLifecycleScenario implements ClientScenario {
|
||||
|
|
@ -231,7 +232,13 @@ The server MUST advertise \`io.modelcontextprotocol/tasks\` under
|
|||
'tasks/get returns status + metadata for an active task';
|
||||
if (!workingTaskId) {
|
||||
checks.push(
|
||||
skipCheck(id, name, description, 'no task created', [SEP_2663_REF])
|
||||
untestableCheck(
|
||||
id,
|
||||
name,
|
||||
description,
|
||||
'no task was created by the preceding step, so this check could not be exercised',
|
||||
[SEP_2663_REF]
|
||||
)
|
||||
);
|
||||
} else {
|
||||
try {
|
||||
|
|
@ -271,7 +278,13 @@ The server MUST advertise \`io.modelcontextprotocol/tasks\` under
|
|||
'Completed task tasks/get inlines result with content[] (no separate tasks/result method)';
|
||||
if (!workingTaskId) {
|
||||
checks.push(
|
||||
skipCheck(id, name, description, 'no task created', [SEP_2663_REF])
|
||||
untestableCheck(
|
||||
id,
|
||||
name,
|
||||
description,
|
||||
'no task was created by the preceding step, so this check could not be exercised',
|
||||
[SEP_2663_REF]
|
||||
)
|
||||
);
|
||||
} else {
|
||||
try {
|
||||
|
|
|
|||
|
|
@ -38,7 +38,14 @@ export function failureCheck(
|
|||
};
|
||||
}
|
||||
|
||||
/** Build a SKIPPED check (preserves id stability so Ctrl+F still finds it). */
|
||||
/**
|
||||
* Build a SKIPPED check (preserves id stability so Ctrl+F still finds it).
|
||||
*
|
||||
* Use ONLY for checks that are legitimately not applicable (e.g. a harness
|
||||
* gap the suite itself tracks). A check whose prerequisite is missing on the
|
||||
* server under test must use untestableCheck from src/scenarios/untestable.ts
|
||||
* instead — SKIPPED reads as green in every consumer (issue #248).
|
||||
*/
|
||||
export function skipCheck(
|
||||
id: string,
|
||||
name: string,
|
||||
|
|
|
|||
|
|
@ -13,8 +13,9 @@
|
|||
import { ClientScenario, ConformanceCheck } from '../../../types';
|
||||
import type { Connection, RunContext } from '../../../connection';
|
||||
import { SEP_2663_REF } from './mrtr-helpers';
|
||||
import { errMsg, failureCheck, skipCheck } from './mrtr-helpers';
|
||||
import { errMsg, failureCheck } from './mrtr-helpers';
|
||||
import { TASKS_EXTENSION_ID, waitForTerminal } from './helpers';
|
||||
import { untestableCheck } from '../../untestable';
|
||||
|
||||
export class TasksWireFieldsScenario implements ClientScenario {
|
||||
name = 'tasks-wire-fields';
|
||||
|
|
@ -155,7 +156,13 @@ export class TasksWireFieldsScenario implements ClientScenario {
|
|||
'Task remains accessible via tasks/get for the duration of its ttlMs';
|
||||
if (!createdTaskId) {
|
||||
checks.push(
|
||||
skipCheck(id, name, description, 'no task created', [SEP_2663_REF])
|
||||
untestableCheck(
|
||||
id,
|
||||
name,
|
||||
description,
|
||||
'no task was created by the preceding step, so this check could not be exercised',
|
||||
[SEP_2663_REF]
|
||||
)
|
||||
);
|
||||
} else {
|
||||
try {
|
||||
|
|
@ -204,7 +211,13 @@ export class TasksWireFieldsScenario implements ClientScenario {
|
|||
const taskId = created.taskId;
|
||||
if (!taskId) {
|
||||
checks.push(
|
||||
skipCheck(id, name, description, 'no task created', [SEP_2663_REF])
|
||||
untestableCheck(
|
||||
id,
|
||||
name,
|
||||
description,
|
||||
'no task was created by the preceding step, so this check could not be exercised',
|
||||
[SEP_2663_REF]
|
||||
)
|
||||
);
|
||||
} else {
|
||||
const terminal = await waitForTerminal(conn, taskId);
|
||||
|
|
|
|||
39
src/scenarios/untestable.test.ts
Normal file
39
src/scenarios/untestable.test.ts
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
import { describe, it, expect } from 'vitest';
|
||||
import { notTestable, untestableCheck } from './untestable';
|
||||
|
||||
describe('untestable helpers', () => {
|
||||
it('notTestable formats the stable prefix', () => {
|
||||
expect(notTestable('server lacks tool X')).toBe(
|
||||
'Not testable: server lacks tool X'
|
||||
);
|
||||
});
|
||||
|
||||
it('untestableCheck defaults to FAILURE and marks details.untestable', () => {
|
||||
const check = untestableCheck(
|
||||
'some-check-id',
|
||||
'SomeCheck',
|
||||
'A MUST requirement',
|
||||
'server lacks the diagnostic tool',
|
||||
[{ id: 'SEP-0000', url: 'https://example.com' }]
|
||||
);
|
||||
expect(check).toMatchObject({
|
||||
id: 'some-check-id',
|
||||
status: 'FAILURE',
|
||||
errorMessage: 'Not testable: server lacks the diagnostic tool',
|
||||
details: { untestable: true }
|
||||
});
|
||||
});
|
||||
|
||||
it('untestableCheck supports WARNING severity for SHOULD requirements', () => {
|
||||
const check = untestableCheck(
|
||||
'should-check-id',
|
||||
'ShouldCheck',
|
||||
'A SHOULD requirement',
|
||||
'missing hook',
|
||||
[{ id: 'SEP-0000', url: 'https://example.com' }],
|
||||
'WARNING'
|
||||
);
|
||||
expect(check.status).toBe('WARNING');
|
||||
expect(check.errorMessage).toBe('Not testable: missing hook');
|
||||
});
|
||||
});
|
||||
55
src/scenarios/untestable.ts
Normal file
55
src/scenarios/untestable.ts
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
/**
|
||||
* "Untestable" check policy (issue #248).
|
||||
*
|
||||
* A check whose prerequisite is missing — the server under test lacks a
|
||||
* diagnostic fixture tool, rejects the probe that would exercise the
|
||||
* requirement, or advertises a feature it does not actually serve — MUST NOT
|
||||
* report SKIPPED: SKIPPED is excluded from pass/fail counts, exit codes, and
|
||||
* the expected-failures baseline, so the run reads as green and the gap is
|
||||
* invisible to anyone burning down a conformance list. Instead the check
|
||||
* fails, with an errorMessage that names the missing prerequisite, so the
|
||||
* result is red until the prerequisite exists and the scenario can sit in an
|
||||
* expected-failures baseline meanwhile (the documented escape hatch).
|
||||
*
|
||||
* SKIPPED remains correct only for checks that are legitimately not
|
||||
* applicable: an optional capability the server never claimed (e.g. a SHOULD
|
||||
* check gated on `prompts.listChanged` the server did not declare), or
|
||||
* spec-version inapplicability handled by the runner.
|
||||
*/
|
||||
|
||||
import type { ConformanceCheck, SpecReference } from '../types';
|
||||
|
||||
/**
|
||||
* Format the errorMessage for a check that could not be exercised. The
|
||||
* stable prefix lets reports and dashboards distinguish "requirement was
|
||||
* violated" from "requirement could not be verified" without a new status.
|
||||
*/
|
||||
export function notTestable(reason: string): string {
|
||||
return `Not testable: ${reason}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a failing check for a requirement that could not be exercised
|
||||
* because a prerequisite is missing. `severity` follows the spec keyword of
|
||||
* the underlying requirement (MUST -> FAILURE, SHOULD -> WARNING), matching
|
||||
* what the check could have reported had it run.
|
||||
*/
|
||||
export function untestableCheck(
|
||||
id: string,
|
||||
name: string,
|
||||
description: string,
|
||||
reason: string,
|
||||
specReferences: SpecReference[],
|
||||
severity: 'FAILURE' | 'WARNING' = 'FAILURE'
|
||||
): ConformanceCheck {
|
||||
return {
|
||||
id,
|
||||
name,
|
||||
description,
|
||||
status: severity,
|
||||
timestamp: new Date().toISOString(),
|
||||
errorMessage: notTestable(reason),
|
||||
specReferences,
|
||||
details: { untestable: true, reason }
|
||||
};
|
||||
}
|
||||
Loading…
Reference in a new issue