modelcontextprotocol/conformance
Watch
1
0
Fork
You've already forked conformance
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

chore: additional server conformance tests for SEP-2575 (#297)

Browse source 
* chore: additional server conformance tests for SEP-2575

* fix(sep-2575): use the spec's notifications filter object in subscriptions/listen

The scenario sent params.subscriptions: [{type: 'tools/list-changed'}], a
shape that does not exist in the SEP. SubscriptionsListenRequestParams
requires params.notifications: {toolsListChanged?, promptsListChanged?, ...}
(see schema.ts SubscriptionFilter and the canonical
SubscriptionsListenRequest example). A compliant server would either reject
the old request as invalid params or treat it as subscribing to nothing,
failing the ack/list-changed checks.

Also makes the everything-server's acknowledgment echo the agreed
notifications filter, as the SubscriptionsAcknowledgedNotification schema
requires.

* fix(sep-2575): match the spec's notifications/*/list_changed method names

The list-changed checks and the everything-server used
notifications/tools/list-changed (hyphen); the spec's method is
notifications/tools/list_changed (underscore) per ToolListChangedNotification
/ PromptListChangedNotification in schema.ts. With the hyphen form the
notification-filter check cannot catch a real leak, and the list-changed
checks fail a server that emits the correct name.

Also drops the f.params.toolsListChanged === true fallback match: that field
is the subscriptions/listen request filter, not a notification parameter, so
it never appears on a compliant server's notification.

* fix(sep-2575): keep collected frames when the listen stream read times out

listenToStream aborts the fetch after timeoutMs. If the abort fired while
blocked in reader.read(), the rejection propagated to the outer catch and the
helper returned [] — discarding every frame already received. A compliant
server holds a subscriptions/listen stream open indefinitely, so the timeout
is the *normal* way these reads end; previously every subscription check
would report a false 'no frames received' FAILURE against such a server.
The current everything-server only avoided this by res.end()ing immediately
after the ack.

* fix(sep-2575): open the listen stream before triggering list mutations

The notification-filter and list-changed checks fired the mutation trigger
*before* opening the subscriptions/listen stream. A compliant server only
delivers list-changed notifications to streams that are open at the time of
the change, so the SHOULD checks would fail any server that does not replay
past changes to new subscribers, and the filter check could never observe a
real leak.

listenToStream now takes an onFirstFrame hook; the three trigger-based
checks open the stream, wait for the acknowledgment, then fire the mutation
on a separate connection and keep reading.

The everything-server now keeps subscriptions/listen streams open, registers
them with their filters, and fans the list_changed notification out to open
streams when a trigger tool runs - instead of unconditionally emitting a
list-changed frame on every stream open (which itself violated the
notification-filter requirement in spirit: it announced a change when
nothing had changed).

* fix(sep-2575): check the subscription id on every listen-stream notification

server-tags-subscription-id only inspected the acknowledgment frame, so a
server that tags the ack but omits io.modelcontextprotocol/subscriptionId
from subsequent notifications passed. The check now triggers a tool-list
change once the stream is acknowledged and asserts every notification frame
on the stream carries the id in params._meta.

Also drops the f.body.method / f.params.method fallbacks when resolving a
frame's method - those shapes don't exist in JSON-RPC and only masked
malformed frames.

* fix(sep-2575): report WARNING, not FAILURE, for the SHOULD-level list-changed checks

The spec text behind server-sends-{tools,prompts}-list-changed-on-subscription
is a SHOULD ('servers that declared the listChanged capability SHOULD send a
notification...'). Severity follows the spec keyword: MUST -> FAILURE,
SHOULD -> WARNING. runCheck now accepts a warning flag and the two
list-changed checks use it on their failure paths.

* fix(sep-2575): skip stream checks when the diagnostic tool is not exposed

http-server-no-independent-requests-on-stream and
server-no-log-without-loglevel reported SUCCESS when the server rejected the
tools/call outright (e.g. test_streaming_elicitation / test_logging_tool do
not exist): a single error frame contains no independent request and no log
notification, so the check passed without exercising anything. The sibling
list-changed checks already SKIP when their trigger tool is missing; these
now do the same.

* chore(sep-2575): exclude the four unobservable server requirements from traceability

Marks the section-C rows of #296 as excluded rather than untested:
no-prior-context, no-connection-reuse-required, disconnect-is-cancel and
stops-on-cancel describe internal server state that a black-box harness
cannot observe on the wire. With these excluded and the part-B checks landed,
every testable server-side SEP-2575 requirement is covered; the remaining
untested rows are all client-side and blocked on a SEP-2575-aware reference
client.

---------

Co-authored-by: Paul Carleton <paulc@anthropic.com>
This commit is contained in:
Yuan Teoh 2026-05-21 11:17:31 -07:00 • committed by GitHub
commit c57795e2a3
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 1008 additions and 20 deletions
Download patch file Download diff file Expand all files Collapse all files

168
examples/servers/typescript/everything-server.ts
View file

@ -1041,6 +1041,38 @@ function isInitializeRequest(body: any): boolean {
// Use createMcpExpressApp for DNS rebinding protection on localhost
const app = createMcpExpressApp();
// Open subscriptions/listen streams (SEP-2575). Notifications are delivered
// to whichever streams are open *at the time of the change* and whose filter
// requested that notification type.
interface ListenStream {
res: import('express').Response;
subscriptionId: string;
wantsTools: boolean;
wantsPrompts: boolean;
}
const activeListenStreams: ListenStream[] = [];
function notifyListenStreams(
type: 'tools' | 'prompts',
notificationMethod: string
) {
for (const stream of activeListenStreams) {
const wants = type === 'tools' ? stream.wantsTools : stream.wantsPrompts;
if (!wants) continue;
stream.res.write(
JSON.stringify({
jsonrpc: '2.0',
method: notificationMethod,
params: {
_meta: {
'io.modelcontextprotocol/subscriptionId': stream.subscriptionId
}
}
}) + '\n'
);
}
}
// Configure CORS to expose Mcp-Session-Id header for browser-based clients
app.use(
cors({
@ -1113,6 +1145,54 @@ app.post('/mcp', async (req, res) => {
});
}
// Subscriptions Listening Endpoint Stream Handler (SSE/Chunked Line)
if (method === 'subscriptions/listen') {
res.writeHead(200, {
'Content-Type': 'application/json',
'Cache-Control': 'no-cache',
Connection: 'keep-alive',
'Transfer-Encoding': 'chunked'
});
const requestedNotifications = params.notifications || {};
// The subscription ID matches the JSON-RPC id of the listen request.
const trackingSubId = String(id ?? 'sub-token-stateless-123');
const wantsTools = requestedNotifications.toolsListChanged === true;
const wantsPrompts = requestedNotifications.promptsListChanged === true;
// First message MUST be notifications/subscriptions/acknowledged carrying tracking token in _meta
// The `notifications` field echoes the subset of the requested filter the server honors.
const ackFrame = {
jsonrpc: '2.0',
method: 'notifications/subscriptions/acknowledged',
params: {
_meta: { 'io.modelcontextprotocol/subscriptionId': trackingSubId },
notifications: {
...(wantsTools ? { toolsListChanged: true } : {}),
...(wantsPrompts ? { promptsListChanged: true } : {})
}
}
};
res.write(JSON.stringify(ackFrame) + '\n');
// Keep the stream open and register it so list-changed notifications
// triggered by later requests are delivered to it. The stream ends when
// the client disconnects (which is also how the client cancels it).
const stream: ListenStream = {
res,
subscriptionId: trackingSubId,
wantsTools,
wantsPrompts
};
activeListenStreams.push(stream);
res.on('close', () => {
const index = activeListenStreams.indexOf(stream);
if (index !== -1) activeListenStreams.splice(index, 1);
});
return;
}
if (method === 'server/discover') {
return res.json({
jsonrpc: '2.0',
@ -1120,8 +1200,8 @@ app.post('/mcp', async (req, res) => {
result: {
supportedVersions: ['DRAFT-2026-v1'],
capabilities: {
tools: { listChanged: false }, // Explicitly declare capability flags to resolve check assertions
prompts: { listChanged: false }
tools: { listChanged: true }, // Explicitly announce dynamic capabilities matching Section 7 expectations
prompts: { listChanged: true }
},
serverInfo: { name: 'everything-stateless-server', version: '1.0.0' }
}
@ -1138,6 +1218,17 @@ app.post('/mcp', async (req, res) => {
name: 'test_missing_capability',
description: 'Test tool requiring sampling',
inputSchema: { type: 'object', properties: {} }
},
{
name: 'test_streaming_elicitation',
description:
'Diagnostic tool validating response progress streams',
inputSchema: { type: 'object', properties: {} }
},
{
name: 'test_logging_tool',
description: 'Diagnostic logging validator tool',
inputSchema: { type: 'object', properties: {} }
}
]
}
@ -1176,6 +1267,79 @@ app.post('/mcp', async (req, res) => {
result: { content: [{ type: 'text', text: 'Success' }] }
});
}
// Progressive IncompleteResult Stream Generator Handling
if (name === 'test_streaming_elicitation') {
res.writeHead(200, {
'Content-Type': 'application/json',
'Transfer-Encoding': 'chunked'
});
res.write(
JSON.stringify({
jsonrpc: '2.0',
method: 'notifications/progress', // Emits standard progress notice
params: { progressToken: 'token-abc', total: 100, value: 50 }
}) + '\n'
);
return res.end(
JSON.stringify({
jsonrpc: '2.0',
id,
result: { content: [{ type: 'text', text: 'Streaming complete' }] }
})
);
}
// Contextual Logging Constraints Verification Handler
if (name === 'test_logging_tool') {
res.writeHead(200, {
'Content-Type': 'application/json',
'Transfer-Encoding': 'chunked'
});
// RULE: No logs allowed if meta configuration lacks explicit log level bounds
if (meta && meta['io.modelcontextprotocol/logLevel']) {
res.write(
JSON.stringify({
jsonrpc: '2.0',
method: 'notifications/message',
params: {
level: 'info',
text: 'Diagnostic trace logging activated'
}
}) + '\n'
);
}
return res.end(
JSON.stringify({
jsonrpc: '2.0',
id,
result: { content: [{ type: 'text', text: 'Logging evaluated' }] }
})
);
}
// Helper mutation hooks used by dynamic tests to force stream activity
// evaluation. The list change is fanned out to whichever
// subscriptions/listen streams are currently open and asked for it.
if (
name === 'test_trigger_tool_change' ||
name === 'test_trigger_prompt_change'
) {
if (name === 'test_trigger_tool_change') {
notifyListenStreams('tools', 'notifications/tools/list_changed');
} else {
notifyListenStreams('prompts', 'notifications/prompts/list_changed');
}
return res.json({
jsonrpc: '2.0',
id,
result: { content: [{ type: 'text', text: 'Mutation triggered' }] }
});
}
}
// Removed Methods per SEP-2575 (Changed status from 200 to 400/404 per Transport Spec)

275
src/scenarios/server/stateless.test.ts
View file

@ -13,7 +13,69 @@ describe('Stateless Server Scenario Negative Tests', () => {
global.fetch = async (_url: any, init: any) => {
const body = JSON.parse(init.body);
const headers = init.headers || {};
const responseConfig = await handler(body, headers);
let responseConfig = await handler(body, headers);
// GLOBAL SPEC-COMPLIANT FALLBACKS: Provides successful data loops for downstream checks
// if individual tests are focusing exclusively on separate fields (like discovery or meta checks).
if (!responseConfig) {
if (body.method === 'subscriptions/listen') {
responseConfig = {
isStream: true,
status: 200,
frames: [
{
jsonrpc: '2.0',
method: 'notifications/subscriptions/acknowledged',
params: {
_meta: {
'io.modelcontextprotocol/subscriptionId':
'global-valid-sub-id'
}
}
}
]
};
} else if (body.method === 'tools/call') {
responseConfig = {
isStream: true,
status: 200,
frames: [
{
jsonrpc: '2.0',
result: {
content: [{ type: 'text', text: 'Progress chunk details' }]
}
}
]
};
}
}
if (responseConfig?.isStream) {
const streamData = responseConfig.frames.map(
(f: any) => JSON.stringify(f) + '\n'
);
let frameIndex = 0;
const mockReader = {
read: async () => {
if (frameIndex >= streamData.length) {
return { value: undefined, done: true };
}
const chunk = new TextEncoder().encode(streamData[frameIndex++]);
return { value: chunk, done: false };
}
};
return {
status: responseConfig?.status ?? 200,
body: {
getReader: () => mockReader,
releaseLock: () => {}
}
} as unknown as Response;
}
return {
status: responseConfig?.status ?? 404,
@ -135,4 +197,215 @@ describe('Stateless Server Scenario Negative Tests', () => {
);
expect(negotiationMatchCheck?.status).toBe('FAILURE');
});
test('Fails validation when stream does not send subscription acknowledgement or tracking ID', async () => {
const mockUrl = mockFetchTarget((reqBody) => {
if (reqBody.method === 'subscriptions/listen') {
return {
isStream: true,
status: 200,
frames: [
{
jsonrpc: '2.0',
method: 'notifications/some-unrelated-event',
params: { _meta: {} }
}
]
};
}
});
const scenario = new ServerStatelessScenario();
const checks = await scenario.run(mockUrl);
const ackCheck = findCheck(
checks,
'sep-2575-server-sends-subscription-ack'
);
const idCheck = findCheck(checks, 'sep-2575-server-tags-subscription-id');
expect(ackCheck?.status).toBe('FAILURE');
expect(idCheck?.status).toBe('FAILURE');
});
test('Fails validation when tool response stream leaks raw independent JSON-RPC requests', async () => {
const mockUrl = mockFetchTarget((reqBody) => {
if (
reqBody.method === 'tools/call' &&
reqBody.params?.name === 'test_streaming_elicitation'
) {
return {
isStream: true,
status: 200,
frames: [
{
jsonrpc: '2.0',
id: 'server-driven-id-999',
method: 'client/some-arbitrary-request',
params: {}
}
]
};
}
});
const scenario = new ServerStatelessScenario();
const checks = await scenario.run(mockUrl);
const independentRequestCheck = findCheck(
checks,
'sep-2575-http-server-no-independent-requests-on-stream'
);
expect(independentRequestCheck?.status).toBe('FAILURE');
});
test('Fails validation when logging occurs without an explicit client request logLevel metadata', async () => {
const mockUrl = mockFetchTarget((reqBody) => {
if (
reqBody.method === 'tools/call' &&
reqBody.params?.name === 'test_logging_tool'
) {
return {
isStream: true,
status: 200,
frames: [
{
jsonrpc: '2.0',
method: 'notifications/message',
params: { level: 'debug', text: 'Stray log frame' }
}
]
};
}
});
const scenario = new ServerStatelessScenario();
const checks = await scenario.run(mockUrl);
const logWithoutLevelCheck = findCheck(
checks,
'sep-2575-server-no-log-without-loglevel'
);
expect(logWithoutLevelCheck?.status).toBe('FAILURE');
});
test('Fails validation when server leaks out-of-filter notifications on a subscription stream', async () => {
const mockUrl = mockFetchTarget((reqBody) => {
// The scenario subscribes ONLY to prompts list-changed notifications
if (
reqBody.method === 'subscriptions/listen' &&
reqBody.params?.notifications?.promptsListChanged === true
) {
return {
isStream: true,
status: 200,
frames: [
{
jsonrpc: '2.0',
method: 'notifications/subscriptions/acknowledged',
_meta: {
'io.modelcontextprotocol/subscriptionId': 'sub-leak-test'
}
},
// LEAK VIOLATION: Server sends tools/list_changed on a prompt-only stream
{
jsonrpc: '2.0',
method: 'notifications/tools/list_changed',
_meta: {
'io.modelcontextprotocol/subscriptionId': 'sub-leak-test'
}
}
]
};
}
});
const scenario = new ServerStatelessScenario();
const checks = await scenario.run(mockUrl);
const filterCheck = findCheck(
checks,
'sep-2575-server-honors-notification-filter'
);
expect(filterCheck?.status).toBe('FAILURE');
});
test('Warns when server drops tools or prompts list changed notifications despite declaring capabilities', async () => {
const mockUrl = mockFetchTarget((reqBody) => {
// 1. Declare BOTH capabilities so neither check is skipped
if (reqBody.method === 'server/discover') {
return {
status: 200,
body: {
jsonrpc: '2.0',
id: reqBody.id,
result: {
supportedVersions: ['DRAFT-2026-v1'],
capabilities: {
tools: { listChanged: true },
prompts: { listChanged: true }
},
serverInfo: { name: 'test-server', version: '1.0' }
}
}
};
}
// 2. Allow BOTH mutation triggers to succeed
if (
reqBody.method === 'tools/call' &&
(reqBody.params?.name === 'test_trigger_prompt_change' ||
reqBody.params?.name === 'test_trigger_tool_change')
) {
return {
status: 200,
body: {
jsonrpc: '2.0',
id: reqBody.id,
result: { content: [{ type: 'text', text: 'mutated' }] }
}
};
}
// 3. Provide the streams but withhold the required notifications for BOTH channels
if (reqBody.method === 'subscriptions/listen') {
const filter = reqBody.params?.notifications;
if (
filter?.promptsListChanged === true ||
filter?.toolsListChanged === true
) {
return {
isStream: true,
status: 200,
frames: [
{
jsonrpc: '2.0',
method: 'notifications/subscriptions/acknowledged',
_meta: {
'io.modelcontextprotocol/subscriptionId': 'sub-drop-test'
}
}
// MISSING VIOLATION: We never send the expected notifications/*/list_changed
]
};
}
}
});
const scenario = new ServerStatelessScenario();
const checks = await scenario.run(mockUrl);
const promptsCheck = findCheck(
checks,
'sep-2575-server-sends-prompts-list-changed-on-subscription'
);
const toolsCheck = findCheck(
checks,
'sep-2575-server-sends-tools-list-changed-on-subscription'
);
// These map to SHOULD requirements, so severity is WARNING not FAILURE.
expect(promptsCheck?.status).toBe('WARNING');
expect(toolsCheck?.status).toBe('WARNING');
});
});

565
src/scenarios/server/stateless.ts
View file

@ -38,8 +38,14 @@ export class ServerStatelessScenario implements ClientScenario {
- Absent or altered protocol version header metadata must trigger a \`-32001 Header Mismatch\` error with an HTTP 400 boundary state.
4. **Client Capability Constraints (2 Checks)**
- Accessing platform capabilities without explicit declaration drops requests with a \`-32003 MissingRequiredClientCapabilityError\` containing needed capabilities, returning an HTTP status code \`400 Bad Request\`.
5. **Methods & Routing Mechanics (3 Checks)**
- Removed legacy endpoints (\`initialize\`, \`ping\`, \`logging/setLevel\`, etc.) or generic unknown methods must cleanly yield an HTTP status code \`404 Not Found\` alongside a JSON-RPC \`-32601 Method not found\` payload. All error returns must preserve original request ID mappings.`;
5. **Methods & Routing Mechanics (5 Checks)**
- Removed legacy endpoints (\`initialize\`, \`ping\`, \`logging/setLevel\`, etc.) or generic unknown methods must cleanly yield an HTTP status code \`404 Not Found\` alongside a JSON-RPC \`-32601 Method not found\` payload. All error returns must preserve original request ID mappings.
- Validates response streams contain only \`IncompleteResult\` chunks and never independent top-level JSON-RPC requests, while enforcing that no log messages are emitted when \`_meta.../logLevel\` is omitted.
6. **Subscription Streams & Filtering (3 Checks)**
- Mandates that \`notifications/subscriptions/acknowledged\` is the first message on a \`subscriptions/listen\` stream, and that subsequent notifications carry a matching \`_meta.../subscriptionId\`.
- Verifies strict containment where servers do not dispatch notification types that fall outside the client's explicit requested subscription filter list.
7. **Dynamic List Mutations (2 Checks)**
- Evaluates that list-changed capable servers notify active listen streams with \`promptsListChanged: true\` or \`toolsListChanged: true\` upon live configuration or capability modifications. `;
async run(serverUrl: string): Promise<ConformanceCheck[]> {
const checks: ConformanceCheck[] = [];
@ -51,15 +57,29 @@ export class ServerStatelessScenario implements ClientScenario {
name: string,
description: string,
fn: () =>
| Promise<{ error?: string; skipped?: boolean; details?: any } | void>
| ({ error?: string; skipped?: boolean; details?: any } | void),
| Promise<{
error?: string;
warning?: boolean;
skipped?: boolean;
details?: any;
} | void>
| ({
error?: string;
warning?: boolean;
skipped?: boolean;
details?: any;
} | void),
fallbackDetails = {}
) {
try {
const result = await fn();
const errorMessage = result?.error;
// SHOULD-level requirements report WARNING rather than FAILURE
// (severity follows the spec keyword).
const status = errorMessage
? 'FAILURE'
? result?.warning
? 'WARNING'
: 'FAILURE'
: result?.skipped
? 'SKIPPED'
: 'SUCCESS';
@ -119,6 +139,126 @@ export class ServerStatelessScenario implements ClientScenario {
return { res, data };
};
// Helper to read multi-frame streaming endpoints (like subscriptions/listen).
// `onFirstFrame` runs once after the first frame arrives (i.e. after the
// server has acknowledged the subscription) so callers can trigger
// side effects on a separate connection while this stream is still open.
const listenToStream = async (
method: string,
params?: any,
maxFrames = 3,
timeoutMs = 1000,
onFirstFrame?: () => Promise<void>
): Promise<any[]> => {
const headers = {
'Content-Type': 'application/json',
Accept: 'application/json',
'MCP-Protocol-Version': DRAFT_PROTOCOL_VERSION
};
const body = JSON.stringify({
jsonrpc: '2.0',
id: Date.now(),
method,
...(params !== undefined ? { params } : {})
});
const controller = new AbortController();
const timeoutId = setTimeout(() => {
controller.abort();
}, timeoutMs);
try {
const res = await fetch(serverUrl, {
method: 'POST',
headers,
body,
signal: controller.signal
});
if (!res.body) {
clearTimeout(timeoutId);
return [];
}
const reader = res.body.getReader();
const decoder = new TextDecoder();
const frames: any[] = [];
let buffer = '';
let firstFrameCallbackFired = false;
const maybeFireFirstFrameCallback = async () => {
if (firstFrameCallbackFired || frames.length === 0 || !onFirstFrame)
return;
firstFrameCallbackFired = true;
try {
await onFirstFrame();
} catch {
// The trigger failed; the caller inspects its own captured result.
}
};
try {
while (frames.length < maxFrames) {
let value: Uint8Array | undefined;
let done = false;
try {
({ value, done } = await reader.read());
} catch {
// The stream was aborted (timeout) or dropped. A compliant
// server holds a subscriptions/listen stream open indefinitely,
// so hitting the timeout is the normal way these reads end —
// return whatever frames arrived before it fired.
break;
}
if (value) {
buffer += decoder.decode(value, { stream: true });
const lines = buffer.split(/\r?\n/);
buffer = lines.pop() || '';
for (const line of lines) {
const cleanLine = line.trim();
if (!cleanLine) continue;
const jsonText = cleanLine.startsWith('data:')
? cleanLine.replace(/^data:\s*/, '')
: cleanLine;
try {
frames.push(JSON.parse(jsonText));
} catch {
// Keep segment buffering
}
}
await maybeFireFirstFrameCallback();
}
if (done) {
const finalLine = buffer.trim();
if (finalLine) {
const jsonText = finalLine.startsWith('data:')
? finalLine.replace(/^data:\s*/, '')
: finalLine;
try {
frames.push(JSON.parse(jsonText));
} catch {
// Trailing formatting mismatch
}
}
break;
}
}
} finally {
reader.releaseLock();
}
clearTimeout(timeoutId);
return frames;
} catch {
clearTimeout(timeoutId);
return [];
}
};
const validMeta = {
'io.modelcontextprotocol/protocolVersion': DRAFT_PROTOCOL_VERSION,
'io.modelcontextprotocol/clientInfo': {
@ -512,7 +652,7 @@ export class ServerStatelessScenario implements ClientScenario {
);
// ==========================================
// 5. Methods & Routing Mechanics (3 Checks)
// 5. Methods & Routing Mechanics (5 Checks)
// ==========================================
const expectedSlugs = [
'initialize',
@ -580,7 +720,418 @@ export class ServerStatelessScenario implements ClientScenario {
}
);
// Final catchall ensuring JSON-RPC id integrity validation rules ran successfully
await runCheck(
'sep-2575-http-server-no-independent-requests-on-stream',
'HttpServerNoIndependentRequestsOnStream',
'Request stream contains only IncompleteResult, never independent JSON-RPC requests',
async () => {
const frames = await listenToStream(
'tools/call',
{
name: 'test_streaming_elicitation',
arguments: {},
_meta: validMeta
},
3,
600
);
if (frames.length === 0) {
return {
error:
'Failed to receive progressive stream chunk execution frames from tools/call handler endpoint'
};
}
// If the call was rejected outright (e.g. the diagnostic tool does
// not exist), nothing was streamed and the requirement was not
// exercised - report SKIPPED rather than a vacuous SUCCESS.
if (frames.every((f) => f?.error !== undefined)) {
return {
skipped: true,
details: {
note: 'Server does not expose diagnostic tool test_streaming_elicitation; the response stream could not be exercised.',
response: frames[0]
}
};
}
const hasIndependentRequest = frames.some(
(f) => f.method && !f.method.startsWith('notifications/')
);
if (hasIndependentRequest) {
return {
error:
'Server emitted an independent standard request layout inside a response progress execution block stream context',
details: { frames }
};
}
return { details: { inspectedFramesCount: frames.length } };
}
);
await runCheck(
'sep-2575-server-no-log-without-loglevel',
'ServerNoLogWithoutLogLevel',
"No notifications/message for requests that didn't set _meta.../logLevel",
async () => {
const frames = await listenToStream(
'tools/call',
{ name: 'test_logging_tool', arguments: {}, _meta: validMeta },
3,
500
);
if (frames.length === 0) {
return {
error:
'Logging target endpoint context dropped or failed to yield frame structures'
};
}
// If the call was rejected outright (e.g. the diagnostic tool does
// not exist), the server never had anything to log and the
// requirement was not exercised - report SKIPPED rather than a
// vacuous SUCCESS.
if (frames.every((f) => f?.error !== undefined)) {
return {
skipped: true,
details: {
note: 'Server does not expose diagnostic tool test_logging_tool; the no-log-without-logLevel requirement could not be exercised.',
response: frames[0]
}
};
}
const logFrame = frames.find(
(f) => f.method === 'notifications/message'
);
if (logFrame) {
return {
error:
'Server dispatched a notifications/message payload chunk even though the client request did not authorize a log level context descriptor',
details: { logFrame }
};
}
return { details: { framesFound: frames.length } };
}
);
// ==========================================
// 6. Subscription Streams & Filtering (3 Checks)
// ==========================================
const subscriptionParams = {
_meta: validMeta,
notifications: { toolsListChanged: true }
};
// Open a tools-filtered stream and (best-effort) trigger a tool-list
// change once it is acknowledged, so the stream carries at least one
// post-acknowledgment notification for the subscription-id check.
let streamFrames: any[] = [];
try {
streamFrames = await listenToStream(
'subscriptions/listen',
subscriptionParams,
2,
800,
async () => {
await sendRpc('tools/call', {
name: 'test_trigger_tool_change',
arguments: {},
_meta: validMeta
});
}
);
} catch {
// Stream pipeline tracking failed
}
await runCheck(
'sep-2575-server-sends-subscription-ack',
'ServerSendsSubscriptionAck',
'notifications/subscriptions/acknowledged is the first message on a subscriptions/listen stream',
() => {
if (streamFrames[0]?.error?.code === -32601) {
return {
skipped: true,
details: {
note: 'Server does not support subscriptions/listen (Method not found)'
}
};
}
if (streamFrames.length === 0) {
return {
error:
'Failed to open or receive frames from the subscriptions/listen stream endpoint'
};
}
const firstFrame = streamFrames[0];
if (firstFrame?.method !== 'notifications/subscriptions/acknowledged') {
return {
error: `Expected first frame method to be 'notifications/subscriptions/acknowledged', got '${firstFrame?.method}'`,
details: { firstFrame }
};
}
return { details: { firstFrame } };
}
);
await runCheck(
'sep-2575-server-tags-subscription-id',
'ServerTagsSubscriptionId',
'Listen-stream notifications carry _meta.../subscriptionId',
() => {
if (streamFrames[0]?.error?.code === -32601) {
return {
skipped: true,
details: {
note: 'Server does not support subscriptions/listen (Method not found)'
}
};
}
if (streamFrames.length === 0) {
return {
error:
'Failed to open stream line or tracking frames are missing completely'
};
}
// Every notification delivered on the stream (the acknowledgment and
// anything after it) must carry the subscription id in params._meta.
const notificationFrames = streamFrames.filter(
(f) =>
typeof f?.method === 'string' &&
f.method.startsWith('notifications/')
);
if (notificationFrames.length === 0) {
return {
error:
'subscriptions/listen stream carried no notification frames to inspect',
details: { streamFrames }
};
}
const untaggedFrames = notificationFrames.filter(
(f) => !f?.params?._meta?.['io.modelcontextprotocol/subscriptionId']
);
if (untaggedFrames.length > 0) {
return {
error: `${untaggedFrames.length} of ${notificationFrames.length} listen-stream notification(s) are missing io.modelcontextprotocol/subscriptionId in params._meta`,
details: { untaggedFrames }
};
}
const subId =
notificationFrames[0]?.params?._meta?.[
'io.modelcontextprotocol/subscriptionId'
];
return {
details: {
subscriptionId: subId,
inspectedNotificationCount: notificationFrames.length
}
};
}
);
await runCheck(
'sep-2575-server-honors-notification-filter',
'ServerHonorsNotificationFilter',
"Server doesn't send notification types the client didn't request",
async () => {
const narrowParams = {
_meta: validMeta,
notifications: { promptsListChanged: true }
};
// Open a stream filtered to prompts only, then mutate the *tool* list
// on a separate connection once the subscription is acknowledged. A
// compliant server must not deliver the resulting tools notification
// to this stream.
const narrowFrames = await listenToStream(
'subscriptions/listen',
narrowParams,
3,
1500,
async () => {
await sendRpc('tools/call', {
name: 'test_trigger_tool_change',
arguments: {},
_meta: validMeta
});
}
);
if (narrowFrames[0]?.error?.code === -32601) {
return {
skipped: true,
details: {
note: 'Server does not support subscriptions/listen (Method not found)'
}
};
}
if (narrowFrames.length === 0) {
return {
error:
'Strict subscription filtering line failed to return communication frames'
};
}
const leakedToolFrame = narrowFrames.find(
(f) => f?.method === 'notifications/tools/list_changed'
);
if (leakedToolFrame) {
return {
error:
'Server leaked a tools/list-changed notification over a stream explicitly filtered to prompts only',
details: { leakedToolFrame }
};
}
return { details: { narrowFramesCount: narrowFrames.length } };
}
);
// ==========================================
// 7. Dynamic List Mutations (2 Checks)
// ==========================================
await runCheck(
'sep-2575-server-sends-prompts-list-changed-on-subscription',
'ServerSendsPromptsListChangedOnSubscription',
'List-changed-capable servers notify listen streams with promptsListChanged: true (SHOULD)',
async () => {
// Automatically pass/skip if the server didn't declare this capability during discovery
if (!discoverCapabilities?.prompts?.listChanged) {
return {
skipped: true,
details: {
note: 'Server did not declare prompts.listChanged capability in server/discover'
}
};
}
const promptsParams = {
_meta: validMeta,
notifications: { promptsListChanged: true }
};
// Open the subscription first; mutate the prompt list on a separate
// connection once it is acknowledged. A compliant server only
// notifies streams that are open at the time of the change.
let trigger: any = null;
const frames = await listenToStream(
'subscriptions/listen',
promptsParams,
2,
1500,
async () => {
trigger = await sendRpc('tools/call', {
name: 'test_trigger_prompt_change',
arguments: {},
_meta: validMeta
});
}
);
if (trigger?.data?.error?.code === -32601) {
return {
skipped: true,
details: {
note: 'Server does not expose diagnostic hook test_trigger_prompt_change to mutate lists'
}
};
}
if (frames.length === 0) {
return {
warning: true,
error:
'Failed to open or receive frames from the subscriptions/listen stream endpoint'
};
}
const changeFrame = frames.find(
(f) => f.method === 'notifications/prompts/list_changed'
);
if (!changeFrame) {
return {
warning: true,
error:
'Mutated the prompt list but no notifications/prompts/list_changed arrived on the open subscription stream. This is a SHOULD requirement.'
};
}
return { details: { changeFrame } };
}
);
await runCheck(
'sep-2575-server-sends-tools-list-changed-on-subscription',
'ServerSendsToolsListChangedOnSubscription',
'List-changed-capable servers notify listen streams with toolsListChanged: true (SHOULD)',
async () => {
// Automatically pass/skip if the server didn't declare this capability during discovery
if (!discoverCapabilities?.tools?.listChanged) {
return {
skipped: true,
details: {
note: 'Server did not declare tools.listChanged capability in server/discover'
}
};
}
const toolsParams = {
_meta: validMeta,
notifications: { toolsListChanged: true }
};
// Open the subscription first; mutate the tool list on a separate
// connection once it is acknowledged. A compliant server only
// notifies streams that are open at the time of the change.
let trigger: any = null;
const frames = await listenToStream(
'subscriptions/listen',
toolsParams,
2,
1500,
async () => {
trigger = await sendRpc('tools/call', {
name: 'test_trigger_tool_change',
arguments: {},
_meta: validMeta
});
}
);
if (trigger?.data?.error?.code === -32601) {
return {
skipped: true,
details: {
note: 'Server does not expose diagnostic hook test_trigger_tool_change to mutate lists'
}
};
}
if (frames.length === 0) {
return {
warning: true,
error:
'Failed to open or receive frames from the subscriptions/listen stream endpoint'
};
}
const changeFrame = frames.find(
(f) => f.method === 'notifications/tools/list_changed'
);
if (!changeFrame) {
return {
warning: true,
error:
'Mutated the tool list but no notifications/tools/list_changed arrived on the open subscription stream. This is a SHOULD requirement.'
};
}
return { details: { changeFrame } };
}
);
if (!checks.some((c) => c.id === 'sep-2575-http-server-error-jsonrpc-id')) {
checks.push({
id: 'sep-2575-http-server-error-jsonrpc-id',

20
src/seps/sep-2575.yaml
View file

@ -13,10 +13,6 @@ requirements:
- check: sep-2575-server-tags-subscription-id
text: 'On notifications delivered via a subscriptions/listen stream, the server MUST include io.modelcontextprotocol/subscriptionId in _meta so the client can correlate the notification with the originating subscription request.'
url: https://modelcontextprotocol.io/specification/draft/basic/index#meta
- check: sep-2575-server-stateless-no-prior-context
text: 'A server MUST NOT treat connection or process identity as a proxy for conversation or session continuity. / Servers MUST NOT rely on prior requests over the same connection to establish context (e.g., capabilities, protocol version, client identity).'
- check: sep-2575-server-stateless-no-connection-reuse-required
text: 'Servers MUST NOT require that a client reuse the same connection to perform related operations.'
- check: sep-2575-server-unsupported-version-error
text: 'If the server does not implement the requested version (whether the version is unknown to the server, or is a known version the server has chosen not to support), it MUST respond with an UnsupportedProtocolVersionError listing the versions it does support.'
url: https://modelcontextprotocol.io/specification/draft/basic/lifecycle#protocol-version-negotiation
@ -29,12 +25,6 @@ requirements:
- check: sep-2575-http-server-no-independent-requests-on-stream
text: 'The server MUST NOT send independent JSON-RPC requests on this stream. Server-to-client interactions are embedded as input requests inside an IncompleteResult.'
url: https://modelcontextprotocol.io/specification/draft/basic/transports#receiving-messages-1
- check: sep-2575-http-server-disconnect-is-cancel
text: 'Closing the SSE response stream MUST be treated by the server as cancellation of that request.'
url: https://modelcontextprotocol.io/specification/draft/basic/transports#cancellation-1
- check: sep-2575-http-server-stops-on-cancel
text: 'The server SHOULD stop work on the cancelled request as soon as practical and MUST NOT send any further messages for it [HTTP].'
url: https://modelcontextprotocol.io/specification/draft/basic/transports#cancellation-1
- check: sep-2575-http-client-sends-version-header
text: 'Every POST request to the MCP endpoint MUST include an MCP-Protocol-Version header.'
url: https://modelcontextprotocol.io/specification/draft/basic/transports#protocol-version-header
@ -78,6 +68,16 @@ requirements:
text: 'The server MUST NOT emit notifications/message for a request that does not include [io.modelcontextprotocol/logLevel in _meta].'
url: https://modelcontextprotocol.io/specification/draft/server/utilities/logging#per-request-log-level
- text: 'A server MUST NOT treat connection or process identity as a proxy for conversation or session continuity. / Servers MUST NOT rely on prior requests over the same connection to establish context (e.g., capabilities, protocol version, client identity).'
excluded: 'internal server state, not directly wire-observable; the observable consequence (rejecting requests with incomplete _meta rather than falling back to remembered state) is covered by sep-2575-request-meta-invalid-* — see https://github.com/modelcontextprotocol/conformance/issues/296'
- text: 'Servers MUST NOT require that a client reuse the same connection to perform related operations.'
excluded: 'not observable from a black-box harness; every harness request already arrives on an independent connection — see https://github.com/modelcontextprotocol/conformance/issues/296'
- text: 'Closing the SSE response stream MUST be treated by the server as cancellation of that request.'
excluded: '"treated as cancellation" is internal server state; once the stream is closed there is no channel left on which to observe the effect — see https://github.com/modelcontextprotocol/conformance/issues/296'
url: https://modelcontextprotocol.io/specification/draft/basic/transports#cancellation-1
- text: 'The server SHOULD stop work on the cancelled request as soon as practical and MUST NOT send any further messages for it [HTTP].'
excluded: '"stop work as soon as practical" is unobservable from a black-box harness, and "no further messages" cannot be verified once the response stream is closed — see https://github.com/modelcontextprotocol/conformance/issues/296'
url: https://modelcontextprotocol.io/specification/draft/basic/transports#cancellation-1
- text: 'State that needs to span multiple requests (e.g., long-running tasks, application-level handles) MUST be referenced by an explicit identifier the client passes on each request.'
excluded: 'architectural guidance, observable only via subscriptionId/task-id rows already listed'
- text: 'To distinguish notifications belonging to different concurrent subscriptions, clients MUST correlate notifications using the io.modelcontextprotocol/subscriptionId field carried in _meta.'