modelcontextprotocol/conformance
Watch
1
0
Fork
You've already forked conformance
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix: fail checks whose prerequisite is missing instead of skipping them (#372)

Browse source 
A check that cannot be exercised — the server under test lacks a
diagnostic fixture tool, or advertises a feature it then rejects — was
reported SKIPPED, which is excluded from pass/fail counts, exit codes,
and the expected-failures baseline. A server could dodge the entire
SEP-2575 capability-constraint and subscription surface (7 MUST-level
checks) and still read green, and a baselined failing scenario whose
checks degraded to SKIPPED flipped its baseline entry to stale, which
reads as fixed.

Such checks now fail with a 'Not testable:' message naming the missing
prerequisite and details.untestable for tooling, per the policy in
issue #248 (severity follows the requirement's keyword: MUST -> FAILURE,
SHOULD -> WARNING). Legitimate skips remain: optional capabilities the
server never declared, runner-level spec-version inapplicability, and
the documented tasks-status-notifications harness gap.

Also fixes a test-suite bug found en route: the stream mock's reader
lacked releaseLock, so listenToStream's teardown threw and every
stream-based assertion silently ran against zero frames.
This commit is contained in:
Felix Weinberger 2026-06-30 11:42:01 +01:00 • committed by GitHub
commit 977813a130
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 902 additions and 117 deletions
Download patch file Download diff file Expand all files Collapse all files

19
src/scenarios/server/tasks/lifecycle.ts
View file

@ -16,8 +16,9 @@
import { ClientScenario, ConformanceCheck } from '../../../types';
import type { Connection, RunContext } from '../../../connection';
import { SEP_2322_REF, SEP_2663_REF } from './mrtr-helpers';
import { errMsg, failureCheck, skipCheck } from './mrtr-helpers';
import { errMsg, failureCheck } from './mrtr-helpers';
import { TASKS_EXTENSION_ID, waitForTerminal } from './helpers';
import { untestableCheck } from '../../untestable';
import { isIso8601 } from './mrtr-helpers';
export class TasksLifecycleScenario implements ClientScenario {
@ -231,7 +232,13 @@ The server MUST advertise \`io.modelcontextprotocol/tasks\` under
'tasks/get returns status + metadata for an active task';
if (!workingTaskId) {
checks.push(
skipCheck(id, name, description, 'no task created', [SEP_2663_REF])
untestableCheck(
id,
name,
description,
'no task was created by the preceding step, so this check could not be exercised',
[SEP_2663_REF]
)
);
} else {
try {
@ -271,7 +278,13 @@ The server MUST advertise \`io.modelcontextprotocol/tasks\` under
'Completed task tasks/get inlines result with content[] (no separate tasks/result method)';
if (!workingTaskId) {
checks.push(
skipCheck(id, name, description, 'no task created', [SEP_2663_REF])
untestableCheck(
id,
name,
description,
'no task was created by the preceding step, so this check could not be exercised',
[SEP_2663_REF]
)
);
} else {
try {