modelcontextprotocol/conformance
Watch
1
0
Fork
You've already forked conformance
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Make the draft-spec scenarios internally consistent (fixes #311, #312, #315) (#319)

Browse source 
* feat(everything-server): serve resources and caching hints on the stateless draft path

The stateless (SEP-2575) request path now answers resources/list,
resources/templates/list and resources/read — including the SEP-2164
-32602 + data.uri error for unknown URIs — and includes the SEP-2549
ttlMs/cacheScope hints on its cacheable list results, matching what the
draft-spec scenarios exercise.

* feat: make the harness's own draft-spec traffic conformant

Fixes the class of issues where a strictly-conformant SDK rejects the
harness's requests for reasons unrelated to the behaviour under test
(#311, #312, #315).

- add a shared stateless draft request helper (draft-client.ts) that puts
  the cross-cutting SEP-2243 headers (Mcp-Method, Mcp-Name,
  MCP-Protocol-Version, Accept) and SEP-2575 _meta (protocolVersion,
  clientInfo, clientCapabilities) on every request, with overrides for
  negative cases and the SHOULD-level retry on
  UnsupportedProtocolVersionError
- migrate the draft scenarios onto it: the SEP-2322 helpers and
  server-stateless now send Mcp-Method/Mcp-Name (#312); the SEP-2243
  server scenarios drop the legacy initialize handshake and carry _meta on
  every raw request (#311); caching and sep-2164-resource-not-found are
  stateless DRAFT-2026-v1 requests instead of an SDK 2025-11-25 session
  (#315)
- add a draft self-conformance gate (draft-self-conformance.test.ts) that
  points the harness's own draft drivers at the existing client-testing
  judge scenarios (request-metadata, http-standard-headers) and requires
  zero failures or warnings
- align the request-metadata mock's unsupported-version rejection with the
  draft schema (-32004) and teach the everything-client to recognize it
- rewrite the SEP-2549/SEP-2164 negative example servers as stateless
  draft servers so they keep exercising the violation, not the transport

* refactor: rename the stateless request helpers and harden them per review

- rename draft-client.ts to stateless-client.ts and name the helpers by
  feature (sendStatelessRequest, buildStandardHeaders, buildRequestMeta,
  withRequestMeta, CONFORMANCE_CLIENT_INFO); 'draft' now only appears in
  DRAFT_PROTOCOL_VERSION, the one name that tracks the moving target
- retry only genuine UnsupportedProtocolVersionError rejections (-32004,
  tolerating -32001/-32602) and only with recognized versions; record the
  retry on the response
- parse text/event-stream responses incrementally and resolve on the
  matching event instead of blocking until the timeout
- keep the MCP-Protocol-Version header and _meta protocolVersion in sync
  when the version is overridden via meta overrides
- make server-stateless's sendRpc SSE-tolerant
- emit explicit setup failures in the SEP-2243 server scenarios when the
  tools/list discovery fails instead of misleading per-case results
- latch the worst status per check in the request-metadata judge so an
  early violation cannot be masked by a later conformant request
- rename the gate to harness-traffic-conformance.test.ts and derive its
  driver list from the draft suite with documented per-judge exclusions
  (35 pairings)
- everything-server: advertise resources in stateless server/discover and
  use -32004 for unsupported protocol version
- add unit tests for the helper (retry guard, SSE handling, header/_meta
  version sync, default headers)

* refactor: slim the stateless client helper for the initial merge
This commit is contained in:
Paul Carleton 2026-05-26 21:10:23 +01:00 • committed by GitHub
commit 076b417b23
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 894 additions and 596 deletions
Download patch file Download diff file Expand all files Collapse all files

399
src/scenarios/server/caching.ts
View file

@ -10,14 +10,7 @@ import {
ConformanceCheck,
DRAFT_PROTOCOL_VERSION
} from '../../types';
import { connectToServer } from './client-helper';
import {
ListToolsResultSchema,
ListPromptsResultSchema,
ListResourcesResultSchema,
ListResourceTemplatesResultSchema,
ReadResourceResultSchema
} from '@modelcontextprotocol/sdk/types.js';
import { sendStatelessRequest } from './stateless-client';
const SPEC_REFS = [
{
@ -98,259 +91,181 @@ Servers MUST include \`ttlMs\` (integer >= 0) and \`cacheScope\` ("public" or "p
const checks: ConformanceCheck[] = [];
const allFields: Array<{ endpoint: string; fields: CachingFields }> = [];
try {
const connection = await connectToServer(serverUrl);
// 1. tools/list
// SEP-2549 only exists in the draft spec, so each cacheable endpoint is
// queried over the stateless path (SEP-2575): protocolVersion DRAFT-2026-v1
// plus the cross-cutting _meta and standard headers (issue #315).
const queryEndpoint = async (
checkId: string,
checkName: string,
endpoint: string,
params?: Record<string, unknown>
): Promise<Record<string, unknown> | undefined> => {
const description = `${endpoint} response includes ttlMs and cacheScope caching hints`;
try {
const toolsResult = await connection.client.request(
{ method: 'tools/list', params: {} },
ListToolsResultSchema
const response = await sendStatelessRequest(
serverUrl,
endpoint,
params
);
const fields = extractCachingFields(
toolsResult as Record<string, unknown>
);
allFields.push({ endpoint: 'tools/list', fields });
checks.push(
buildPresenceCheck(
'sep-2549-tools-list-caching-hints',
'ToolsListCachingHints',
'tools/list',
fields
)
);
} catch (error) {
checks.push({
id: 'sep-2549-tools-list-caching-hints',
name: 'ToolsListCachingHints',
description:
'tools/list response includes ttlMs and cacheScope caching hints',
status: 'FAILURE',
timestamp: new Date().toISOString(),
errorMessage: `tools/list request failed: ${error instanceof Error ? error.message : String(error)}`,
specReferences: SPEC_REFS
});
}
// 2. prompts/list
try {
const promptsResult = await connection.client.request(
{ method: 'prompts/list', params: {} },
ListPromptsResultSchema
);
const fields = extractCachingFields(
promptsResult as Record<string, unknown>
);
allFields.push({ endpoint: 'prompts/list', fields });
checks.push(
buildPresenceCheck(
'sep-2549-prompts-list-caching-hints',
'PromptsListCachingHints',
'prompts/list',
fields
)
);
} catch (error) {
checks.push({
id: 'sep-2549-prompts-list-caching-hints',
name: 'PromptsListCachingHints',
description:
'prompts/list response includes ttlMs and cacheScope caching hints',
status: 'FAILURE',
timestamp: new Date().toISOString(),
errorMessage: `prompts/list request failed: ${error instanceof Error ? error.message : String(error)}`,
specReferences: SPEC_REFS
});
}
// 3. resources/list
let firstResourceUri: string | undefined;
try {
const resourcesResult = await connection.client.request(
{ method: 'resources/list', params: {} },
ListResourcesResultSchema
);
const fields = extractCachingFields(
resourcesResult as Record<string, unknown>
);
allFields.push({ endpoint: 'resources/list', fields });
checks.push(
buildPresenceCheck(
'sep-2549-resources-list-caching-hints',
'ResourcesListCachingHints',
'resources/list',
fields
)
);
// Capture the first resource URI for the resources/read check
if (resourcesResult.resources && resourcesResult.resources.length > 0) {
firstResourceUri = resourcesResult.resources[0].uri;
}
} catch (error) {
checks.push({
id: 'sep-2549-resources-list-caching-hints',
name: 'ResourcesListCachingHints',
description:
'resources/list response includes ttlMs and cacheScope caching hints',
status: 'FAILURE',
timestamp: new Date().toISOString(),
errorMessage: `resources/list request failed: ${error instanceof Error ? error.message : String(error)}`,
specReferences: SPEC_REFS
});
}
// 4. resources/templates/list
try {
const templatesResult = await connection.client.request(
{ method: 'resources/templates/list', params: {} },
ListResourceTemplatesResultSchema
);
const fields = extractCachingFields(
templatesResult as Record<string, unknown>
);
allFields.push({ endpoint: 'resources/templates/list', fields });
checks.push(
buildPresenceCheck(
'sep-2549-resources-templates-list-caching-hints',
'ResourcesTemplatesListCachingHints',
'resources/templates/list',
fields
)
);
} catch (error) {
checks.push({
id: 'sep-2549-resources-templates-list-caching-hints',
name: 'ResourcesTemplatesListCachingHints',
description:
'resources/templates/list response includes ttlMs and cacheScope caching hints',
status: 'FAILURE',
timestamp: new Date().toISOString(),
errorMessage: `resources/templates/list request failed: ${error instanceof Error ? error.message : String(error)}`,
specReferences: SPEC_REFS
});
}
// 5. resources/read — use first resource from resources/list
if (firstResourceUri) {
try {
const readResult = await connection.client.request(
{
method: 'resources/read',
params: { uri: firstResourceUri }
},
ReadResourceResultSchema
);
const fields = extractCachingFields(
readResult as Record<string, unknown>
);
allFields.push({ endpoint: 'resources/read', fields });
checks.push(
buildPresenceCheck(
'sep-2549-resources-read-caching-hints',
'ResourcesReadCachingHints',
'resources/read',
fields
)
);
} catch (error) {
const result = response.body?.result;
if (!result) {
const error = response.body?.error;
checks.push({
id: 'sep-2549-resources-read-caching-hints',
name: 'ResourcesReadCachingHints',
description:
'resources/read response includes ttlMs and cacheScope caching hints',
id: checkId,
name: checkName,
description,
status: 'FAILURE',
timestamp: new Date().toISOString(),
errorMessage: `resources/read request failed: ${error instanceof Error ? error.message : String(error)}`,
specReferences: SPEC_REFS
errorMessage: error
? `${endpoint} returned JSON-RPC error ${error.code}: ${error.message}`
: `${endpoint} returned HTTP ${response.status} with no result`,
specReferences: SPEC_REFS,
details: { httpStatus: response.status, error }
});
return undefined;
}
const fields = extractCachingFields(result);
allFields.push({ endpoint, fields });
checks.push(buildPresenceCheck(checkId, checkName, endpoint, fields));
return result;
} catch (error) {
checks.push({
id: checkId,
name: checkName,
description,
status: 'FAILURE',
timestamp: new Date().toISOString(),
errorMessage: `${endpoint} request failed: ${error instanceof Error ? error.message : String(error)}`,
specReferences: SPEC_REFS
});
return undefined;
}
};
// 6. Aggregate: ttlMs must be a non-negative integer
const ttlErrors: string[] = [];
const endpointsWithTtl = allFields.filter((f) => f.fields.hasTtlMs);
if (endpointsWithTtl.length === 0) {
ttlErrors.push('no endpoints returned ttlMs');
} else {
for (const { endpoint, fields } of endpointsWithTtl) {
const val = fields.ttlMs;
if (typeof val !== 'number') {
ttlErrors.push(
`${endpoint}: ttlMs is ${typeof val}, expected number`
);
} else if (!Number.isInteger(val)) {
ttlErrors.push(`${endpoint}: ttlMs is ${val}, expected integer`);
} else if (val < 0) {
ttlErrors.push(`${endpoint}: ttlMs is ${val}, must be >= 0`);
}
}
}
// 1. tools/list
await queryEndpoint(
'sep-2549-tools-list-caching-hints',
'ToolsListCachingHints',
'tools/list'
);
checks.push({
id: 'sep-2549-ttl-non-negative',
name: 'TtlNonNegative',
description: 'All ttlMs values are non-negative integers',
status: ttlErrors.length === 0 ? 'SUCCESS' : 'FAILURE',
timestamp: new Date().toISOString(),
errorMessage: ttlErrors.length > 0 ? ttlErrors.join('; ') : undefined,
specReferences: SPEC_REFS,
details: {
endpoints: allFields.map((f) => ({
endpoint: f.endpoint,
ttlMs: f.fields.ttlMs
}))
}
});
// 2. prompts/list
await queryEndpoint(
'sep-2549-prompts-list-caching-hints',
'PromptsListCachingHints',
'prompts/list'
);
// 7. Aggregate: cacheScope must be "public" or "private"
const scopeErrors: string[] = [];
const endpointsWithScope = allFields.filter(
(f) => f.fields.hasCacheScope
// 3. resources/list
const resourcesResult = await queryEndpoint(
'sep-2549-resources-list-caching-hints',
'ResourcesListCachingHints',
'resources/list'
);
// 4. resources/templates/list
await queryEndpoint(
'sep-2549-resources-templates-list-caching-hints',
'ResourcesTemplatesListCachingHints',
'resources/templates/list'
);
// 5. resources/read — use first resource from resources/list
const resources = resourcesResult?.resources as
| Array<{ uri?: string }>
| undefined;
const firstResourceUri = resources?.[0]?.uri;
if (firstResourceUri) {
await queryEndpoint(
'sep-2549-resources-read-caching-hints',
'ResourcesReadCachingHints',
'resources/read',
{ uri: firstResourceUri }
);
if (endpointsWithScope.length === 0) {
scopeErrors.push('no endpoints returned cacheScope');
} else {
for (const { endpoint, fields } of endpointsWithScope) {
const val = fields.cacheScope;
if (val !== 'public' && val !== 'private') {
scopeErrors.push(
`${endpoint}: cacheScope is ${JSON.stringify(val)}, expected "public" or "private"`
);
}
}
}
} else {
// Keep the emitted check-ID set stable even when there is nothing to
// read (resources/list failed or the server exposes no resources).
checks.push({
id: 'sep-2549-cache-scope-valid',
name: 'CacheScopeValid',
description: 'All cacheScope values are "public" or "private"',
status: scopeErrors.length === 0 ? 'SUCCESS' : 'FAILURE',
id: 'sep-2549-resources-read-caching-hints',
name: 'ResourcesReadCachingHints',
description:
'resources/read response includes ttlMs and cacheScope caching hints',
status: 'SKIPPED',
timestamp: new Date().toISOString(),
errorMessage:
scopeErrors.length > 0 ? scopeErrors.join('; ') : undefined,
specReferences: SPEC_REFS,
details: {
endpoints: allFields.map((f) => ({
endpoint: f.endpoint,
cacheScope: f.fields.cacheScope
}))
}
});
await connection.close();
} catch (error) {
// Connection-level failure — push a single failure check
checks.push({
id: 'sep-2549-caching-connection',
name: 'CachingConnection',
description: 'Caching hints scenario failed to connect',
status: 'FAILURE',
timestamp: new Date().toISOString(),
errorMessage: `Connection failed: ${error instanceof Error ? error.message : String(error)}`,
'resources/read was not exercised: resources/list failed or returned no resources.',
specReferences: SPEC_REFS
});
}
// 6. Aggregate: ttlMs must be a non-negative integer
const ttlErrors: string[] = [];
const endpointsWithTtl = allFields.filter((f) => f.fields.hasTtlMs);
if (endpointsWithTtl.length === 0) {
ttlErrors.push('no endpoints returned ttlMs');
} else {
for (const { endpoint, fields } of endpointsWithTtl) {
const val = fields.ttlMs;
if (typeof val !== 'number') {
ttlErrors.push(
`${endpoint}: ttlMs is ${typeof val}, expected number`
);
} else if (!Number.isInteger(val)) {
ttlErrors.push(`${endpoint}: ttlMs is ${val}, expected integer`);
} else if (val < 0) {
ttlErrors.push(`${endpoint}: ttlMs is ${val}, must be >= 0`);
}
}
}
checks.push({
id: 'sep-2549-ttl-non-negative',
name: 'TtlNonNegative',
description: 'All ttlMs values are non-negative integers',
status: ttlErrors.length === 0 ? 'SUCCESS' : 'FAILURE',
timestamp: new Date().toISOString(),
errorMessage: ttlErrors.length > 0 ? ttlErrors.join('; ') : undefined,
specReferences: SPEC_REFS,
details: {
endpoints: allFields.map((f) => ({
endpoint: f.endpoint,
ttlMs: f.fields.ttlMs
}))
}
});
// 7. Aggregate: cacheScope must be "public" or "private"
const scopeErrors: string[] = [];
const endpointsWithScope = allFields.filter((f) => f.fields.hasCacheScope);
if (endpointsWithScope.length === 0) {
scopeErrors.push('no endpoints returned cacheScope');
} else {
for (const { endpoint, fields } of endpointsWithScope) {
const val = fields.cacheScope;
if (val !== 'public' && val !== 'private') {
scopeErrors.push(
`${endpoint}: cacheScope is ${JSON.stringify(val)}, expected "public" or "private"`
);
}
}
}
checks.push({
id: 'sep-2549-cache-scope-valid',
name: 'CacheScopeValid',
description: 'All cacheScope values are "public" or "private"',
status: scopeErrors.length === 0 ? 'SUCCESS' : 'FAILURE',
timestamp: new Date().toISOString(),
errorMessage: scopeErrors.length > 0 ? scopeErrors.join('; ') : undefined,
specReferences: SPEC_REFS,
details: {
endpoints: allFields.map((f) => ({
endpoint: f.endpoint,
cacheScope: f.fields.cacheScope
}))
}
});
return checks;
}
}