chore: additional server conformance tests for SEP-2575 (#297)
* chore: additional server conformance tests for SEP-2575
* fix(sep-2575): use the spec's notifications filter object in subscriptions/listen
The scenario sent params.subscriptions: [{type: 'tools/list-changed'}], a
shape that does not exist in the SEP. SubscriptionsListenRequestParams
requires params.notifications: {toolsListChanged?, promptsListChanged?, ...}
(see schema.ts SubscriptionFilter and the canonical
SubscriptionsListenRequest example). A compliant server would either reject
the old request as invalid params or treat it as subscribing to nothing,
failing the ack/list-changed checks.
Also makes the everything-server's acknowledgment echo the agreed
notifications filter, as the SubscriptionsAcknowledgedNotification schema
requires.
* fix(sep-2575): match the spec's notifications/*/list_changed method names
The list-changed checks and the everything-server used
notifications/tools/list-changed (hyphen); the spec's method is
notifications/tools/list_changed (underscore) per ToolListChangedNotification
/ PromptListChangedNotification in schema.ts. With the hyphen form the
notification-filter check cannot catch a real leak, and the list-changed
checks fail a server that emits the correct name.
Also drops the f.params.toolsListChanged === true fallback match: that field
is the subscriptions/listen request filter, not a notification parameter, so
it never appears on a compliant server's notification.
* fix(sep-2575): keep collected frames when the listen stream read times out
listenToStream aborts the fetch after timeoutMs. If the abort fired while
blocked in reader.read(), the rejection propagated to the outer catch and the
helper returned [] — discarding every frame already received. A compliant
server holds a subscriptions/listen stream open indefinitely, so the timeout
is the *normal* way these reads end; previously every subscription check
would report a false 'no frames received' FAILURE against such a server.
The current everything-server only avoided this by res.end()ing immediately
after the ack.
* fix(sep-2575): open the listen stream before triggering list mutations
The notification-filter and list-changed checks fired the mutation trigger
*before* opening the subscriptions/listen stream. A compliant server only
delivers list-changed notifications to streams that are open at the time of
the change, so the SHOULD checks would fail any server that does not replay
past changes to new subscribers, and the filter check could never observe a
real leak.
listenToStream now takes an onFirstFrame hook; the three trigger-based
checks open the stream, wait for the acknowledgment, then fire the mutation
on a separate connection and keep reading.
The everything-server now keeps subscriptions/listen streams open, registers
them with their filters, and fans the list_changed notification out to open
streams when a trigger tool runs - instead of unconditionally emitting a
list-changed frame on every stream open (which itself violated the
notification-filter requirement in spirit: it announced a change when
nothing had changed).
* fix(sep-2575): check the subscription id on every listen-stream notification
server-tags-subscription-id only inspected the acknowledgment frame, so a
server that tags the ack but omits io.modelcontextprotocol/subscriptionId
from subsequent notifications passed. The check now triggers a tool-list
change once the stream is acknowledged and asserts every notification frame
on the stream carries the id in params._meta.
Also drops the f.body.method / f.params.method fallbacks when resolving a
frame's method - those shapes don't exist in JSON-RPC and only masked
malformed frames.
* fix(sep-2575): report WARNING, not FAILURE, for the SHOULD-level list-changed checks
The spec text behind server-sends-{tools,prompts}-list-changed-on-subscription
is a SHOULD ('servers that declared the listChanged capability SHOULD send a
notification...'). Severity follows the spec keyword: MUST -> FAILURE,
SHOULD -> WARNING. runCheck now accepts a warning flag and the two
list-changed checks use it on their failure paths.
* fix(sep-2575): skip stream checks when the diagnostic tool is not exposed
http-server-no-independent-requests-on-stream and
server-no-log-without-loglevel reported SUCCESS when the server rejected the
tools/call outright (e.g. test_streaming_elicitation / test_logging_tool do
not exist): a single error frame contains no independent request and no log
notification, so the check passed without exercising anything. The sibling
list-changed checks already SKIP when their trigger tool is missing; these
now do the same.
* chore(sep-2575): exclude the four unobservable server requirements from traceability
Marks the section-C rows of #296 as excluded rather than untested:
no-prior-context, no-connection-reuse-required, disconnect-is-cancel and
stops-on-cancel describe internal server state that a black-box harness
cannot observe on the wire. With these excluded and the part-B checks landed,
every testable server-side SEP-2575 requirement is covered; the remaining
untested rows are all client-side and blocked on a SEP-2575-aware reference
client.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
This commit is contained in:
parent
0371b188bf
commit
c57795e2a3
1 changed files with 1008 additions and 20 deletions
|
|
@ -13,10 +13,6 @@ requirements:
|
|||
- check: sep-2575-server-tags-subscription-id
|
||||
text: 'On notifications delivered via a subscriptions/listen stream, the server MUST include io.modelcontextprotocol/subscriptionId in _meta so the client can correlate the notification with the originating subscription request.'
|
||||
url: https://modelcontextprotocol.io/specification/draft/basic/index#meta
|
||||
- check: sep-2575-server-stateless-no-prior-context
|
||||
text: 'A server MUST NOT treat connection or process identity as a proxy for conversation or session continuity. / Servers MUST NOT rely on prior requests over the same connection to establish context (e.g., capabilities, protocol version, client identity).'
|
||||
- check: sep-2575-server-stateless-no-connection-reuse-required
|
||||
text: 'Servers MUST NOT require that a client reuse the same connection to perform related operations.'
|
||||
- check: sep-2575-server-unsupported-version-error
|
||||
text: 'If the server does not implement the requested version (whether the version is unknown to the server, or is a known version the server has chosen not to support), it MUST respond with an UnsupportedProtocolVersionError listing the versions it does support.'
|
||||
url: https://modelcontextprotocol.io/specification/draft/basic/lifecycle#protocol-version-negotiation
|
||||
|
|
@ -29,12 +25,6 @@ requirements:
|
|||
- check: sep-2575-http-server-no-independent-requests-on-stream
|
||||
text: 'The server MUST NOT send independent JSON-RPC requests on this stream. Server-to-client interactions are embedded as input requests inside an IncompleteResult.'
|
||||
url: https://modelcontextprotocol.io/specification/draft/basic/transports#receiving-messages-1
|
||||
- check: sep-2575-http-server-disconnect-is-cancel
|
||||
text: 'Closing the SSE response stream MUST be treated by the server as cancellation of that request.'
|
||||
url: https://modelcontextprotocol.io/specification/draft/basic/transports#cancellation-1
|
||||
- check: sep-2575-http-server-stops-on-cancel
|
||||
text: 'The server SHOULD stop work on the cancelled request as soon as practical and MUST NOT send any further messages for it [HTTP].'
|
||||
url: https://modelcontextprotocol.io/specification/draft/basic/transports#cancellation-1
|
||||
- check: sep-2575-http-client-sends-version-header
|
||||
text: 'Every POST request to the MCP endpoint MUST include an MCP-Protocol-Version header.'
|
||||
url: https://modelcontextprotocol.io/specification/draft/basic/transports#protocol-version-header
|
||||
|
|
@ -78,6 +68,16 @@ requirements:
|
|||
text: 'The server MUST NOT emit notifications/message for a request that does not include [io.modelcontextprotocol/logLevel in _meta].'
|
||||
url: https://modelcontextprotocol.io/specification/draft/server/utilities/logging#per-request-log-level
|
||||
|
||||
- text: 'A server MUST NOT treat connection or process identity as a proxy for conversation or session continuity. / Servers MUST NOT rely on prior requests over the same connection to establish context (e.g., capabilities, protocol version, client identity).'
|
||||
excluded: 'internal server state, not directly wire-observable; the observable consequence (rejecting requests with incomplete _meta rather than falling back to remembered state) is covered by sep-2575-request-meta-invalid-* — see https://github.com/modelcontextprotocol/conformance/issues/296'
|
||||
- text: 'Servers MUST NOT require that a client reuse the same connection to perform related operations.'
|
||||
excluded: 'not observable from a black-box harness; every harness request already arrives on an independent connection — see https://github.com/modelcontextprotocol/conformance/issues/296'
|
||||
- text: 'Closing the SSE response stream MUST be treated by the server as cancellation of that request.'
|
||||
excluded: '"treated as cancellation" is internal server state; once the stream is closed there is no channel left on which to observe the effect — see https://github.com/modelcontextprotocol/conformance/issues/296'
|
||||
url: https://modelcontextprotocol.io/specification/draft/basic/transports#cancellation-1
|
||||
- text: 'The server SHOULD stop work on the cancelled request as soon as practical and MUST NOT send any further messages for it [HTTP].'
|
||||
excluded: '"stop work as soon as practical" is unobservable from a black-box harness, and "no further messages" cannot be verified once the response stream is closed — see https://github.com/modelcontextprotocol/conformance/issues/296'
|
||||
url: https://modelcontextprotocol.io/specification/draft/basic/transports#cancellation-1
|
||||
- text: 'State that needs to span multiple requests (e.g., long-running tasks, application-level handles) MUST be referenced by an explicit identifier the client passes on each request.'
|
||||
excluded: 'architectural guidance, observable only via subscriptionId/task-id rows already listed'
|
||||
- text: 'To distinguish notifications belonging to different concurrent subscriptions, clients MUST correlate notifications using the io.modelcontextprotocol/subscriptionId field carried in _meta.'
|
||||
|
|
|
|||
Loading…
Reference in a new issue