modelcontextprotocol/python-sdk
Watch
1
0
Fork
You've already forked python-sdk
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

tests/interaction: era-axis machinery for the requirements manifest (#2909)

Browse source
This commit is contained in:
Max 2026-06-19 12:59:39 +01:00 • committed by GitHub
commit 364b762ef6
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 1142 additions and 30 deletions
Download patch file Download diff file Expand all files Collapse all files

79
tests/interaction/README.md
View file

@ -56,10 +56,12 @@ test body — each directory pins its flavour's true output exactly.
Transport-agnostic tests take the `connect` fixture instead of constructing `Client(server)`
directly, and therefore run once per transport: over the in-memory transport, over the server's
real streamable HTTP app driven in-process through the streaming bridge, and over the legacy SSE
transport the same way. A test connects with `async with connect(server, ...) as client:` and
asserts the same output on every leg, because the transport is not supposed to change observable
behaviour. Tests that are tied to one transport do not use the fixture: the wire-recording tests
real streamable HTTP app driven in-process through the streaming bridge (in both stateful and
stateless configurations), and over the legacy SSE transport the same way. A test connects with
`async with connect(server, ...) as client:` and asserts the same output on every leg, because the
transport is not supposed to change observable behaviour. Requirements that need a server-to-client
back-channel or persisted session state are carved out of the stateless arm via `arm_exclusions`.
Tests that are tied to one transport do not use the fixture: the wire-recording tests
(their seam is the in-memory stream pair), the bare-`ClientSession` lifecycle tests, the
real-clock timeout tests (the timeout machinery is transport-independent and must not race
transport latency), and everything under `transports/`, which pins behaviour only observable on
@ -95,6 +97,14 @@ clients can share one session manager.
would require real-time waits the suite refuses.
- **`transports`** names the transports a behaviour applies to; omitted means transport-independent.
- **`issue`** carries the tracking link for a recorded gap once one is filed.
- **`note`** carries free-form context that does not fit `divergence` or `deferred`.
- **`added_in`** / **`removed_in`** bound the spec versions the behaviour exists in, as a half-open
`[added_in, removed_in)` window.
- **`supersedes`** / **`superseded_by`** link a retired entry to its replacement; the link is
bidirectional and both ends must be versioned.
- **`arm_exclusions`** carve specific `(transport, spec_version)` matrix cells out with a typed
`ArmExclusionReason`.
- **`known_failures`** mark specific `(transport, spec_version)` cells as strict xfail.
Tests link themselves to the manifest with a decorator:
@ -126,15 +136,60 @@ This is also the triage key for any rewrite: a test that fails on the new code p
divergence note (the rewrite accidentally fixed a known gap — decide whether to keep the fix) or
it does not (the rewrite broke something that was correct — fix the rewrite).
### When a new spec revision is released
### Spec versions and the era axis
1. Update `SPEC_REVISION` and walk the new revision's changelog.
2. For each changed interaction, find its requirements (the IDs use the wire method strings the
changelog speaks in), re-audit the tests against the new text, and update `source` links and
assertions where behaviour legitimately changed.
3. New interactions get new requirements and new tests; removed interactions get their
requirements deleted along with their tests.
4. A behaviour that is correct under both revisions needs no change beyond the `source` link.
`SPEC_VERSIONS` in `_requirements.py` is the ordered tuple of protocol revisions the suite
exercises. `SPEC_BASE_URL` (and `SPEC_2026_BASE_URL`) are pinned literals — not derived from
`SPEC_VERSIONS` — so growing the active axis never repoints existing `source` links. The
`connect` fixture fans out over `CONNECTABLE_TRANSPORTS × SPEC_VERSIONS`, but the grid is
filtered per test:
`pytest_generate_tests` reads the test's stacked `@requirement` marks and calls `compute_cells()`,
which intersects the admissible cells across every cited requirement — a cell survives only if
**all** of the test's requirements admit it.
`streamable-http-stateless` is the fourth connectable transport: the 2025-era unofficial stateless
mode where each request opens a fresh transport, no session id is issued, and there is no standalone
GET stream. Requirements that need a server→client back-channel or persisted session state are
excluded from that arm via `arm_exclusions` (reasons `server-initiated-request` and
`requires-session`).
What admits or excludes a cell:
- **`added_in` / `removed_in`** gate which spec versions a requirement exists in, as a half-open
`[added_in, removed_in)` window. A test runs only on versions inside every cited requirement's
window.
- **`arm_exclusions`** carve specific `(transport, spec_version)` cells out with a typed
`ArmExclusionReason`. The reason vocabulary doubles as a re-admission checklist: when the gap
closes, grep for the reason string to find every cell to re-admit.
- **`known_failures`** keep a cell in the grid but mark it as a strict xfail — the test runs and
must fail; an unexpected pass fails the suite.
- **`TRANSPORT_SPEC_VERSIONS`** era-locks a transport to a subset of spec versions (currently only
`sse` is locked to `2025-11-25`). A `(transport, version)` cell is dropped if the version is not
in the transport's entry; transports absent from the map serve every spec version. This is the
mechanism for cutting an entire transport off from a new revision (or admitting it).
- **`transports`** is descriptive metadata for the non-`connect` transport-specific suites under
`transports/` and does **not** drive cell generation. Only `arm_exclusions`, `added_in`,
`removed_in`, and `TRANSPORT_SPEC_VERSIONS` filter the grid.
- **`supersedes` / `superseded_by`** link a retired entry to its replacement. `test_coverage.py`
enforces that links are bidirectional and versioned: the retired entry carries `removed_in`, the
replacement carries `added_in`.
Node IDs stay `[transport]` while `len(SPEC_VERSIONS) == 1`, so today's test IDs are
byte-identical to before the era axis existed. They become `[transport-version]` the moment a
second version is appended to `SPEC_VERSIONS`.
When a new spec revision lands:
1. Append the version string to `SPEC_VERSIONS` (and to the `SpecVersion` `Literal`).
2. Walk the new revision's changelog.
3. For each affected requirement: set `removed_in` on retired behaviour, add a new entry with
`added_in` for its replacement, and link the pair with `supersedes` / `superseded_by`.
Behaviour that survives unchanged needs nothing beyond a re-audit of its `source` URL.
4. For requirements that cannot run on the new era's path, add an `arm_exclusions` entry with the
appropriate `ArmExclusionReason`.
5. Review `TRANSPORT_SPEC_VERSIONS`: any era-locked transport will not produce cells on the new
version unless its entry is extended (or removed); add an entry for any transport the new
revision retires.
## Writing a test

20
tests/interaction/_connect.py
View file

@ -9,6 +9,7 @@ server's real Starlette app through the in-process streaming bridge, so the full
from collections.abc import AsyncIterator, Awaitable, Callable, Iterable
from contextlib import AbstractAsyncContextManager, asynccontextmanager
from functools import partial
from typing import Any, Protocol
import httpx
@ -69,6 +70,7 @@ class Connect(Protocol):
message_handler: MessageHandlerFnT | None = None,
client_info: Implementation | None = None,
elicitation_callback: ElicitationFnT | None = None,
protocol_version: str = LATEST_PROTOCOL_VERSION,
) -> AbstractAsyncContextManager[Client]: ...
@ -83,6 +85,7 @@ async def connect_in_memory(
message_handler: MessageHandlerFnT | None = None,
client_info: Implementation | None = None,
elicitation_callback: ElicitationFnT | None = None,
protocol_version: str = LATEST_PROTOCOL_VERSION,
) -> AsyncIterator[Client]:
"""Yield a Client connected to the server over the in-memory transport."""
async with Client(
@ -113,13 +116,15 @@ async def connect_over_streamable_http(
message_handler: MessageHandlerFnT | None = None,
client_info: Implementation | None = None,
elicitation_callback: ElicitationFnT | None = None,
protocol_version: str = LATEST_PROTOCOL_VERSION,
) -> AsyncIterator[Client]:
"""Yield a Client connected to the server's streamable HTTP app, entirely in process.
With the defaults this is the matrix leg (stateful sessions, SSE responses); the
transport-specific tests pass `stateless_http` or `json_response` to select the other
server modes, and the resumability tests pass an `event_store` (with `retry_interval=0` so
the client's reconnection wait is a no-op).
With the defaults this is the matrix leg (stateful sessions, SSE responses); the stateless
matrix arm binds `stateless_http=True` (see `connect_over_streamable_http_stateless`);
transport-specific tests pass `json_response` to select the other server mode, and the
resumability tests pass an `event_store` (with `retry_interval=0` so the client's
reconnection wait is a no-op).
"""
app = server.streamable_http_app(
stateless_http=stateless_http,
@ -145,6 +150,12 @@ async def connect_over_streamable_http(
yield client
connect_over_streamable_http_stateless: Connect = partial(connect_over_streamable_http, stateless_http=True)
"""The streamable-http matrix arm with the server in stateless mode (fresh transport per request,
no session id, no standalone GET stream). The same shared Server instance backs every request --
stateless mode does not require a server factory."""
@asynccontextmanager
async def mounted_app(
server: Server | MCPServer,
@ -326,6 +337,7 @@ async def connect_over_sse(
message_handler: MessageHandlerFnT | None = None,
client_info: Implementation | None = None,
elicitation_callback: ElicitationFnT | None = None,
protocol_version: str = LATEST_PROTOCOL_VERSION,
) -> AsyncIterator[Client]:
"""Yield a Client connected to the server's legacy SSE transport, entirely in process."""
app, _ = build_sse_app(server)

774
tests/interaction/_requirements.py
View file

File diff suppressed because it is too large Load diff

39
tests/interaction/conftest.py
View file

@ -1,23 +1,48 @@
"""Shared fixtures for the interaction suite."""
"""Shared fixtures for the interaction suite.
The ``connect`` fixture is parametrized per-test from the ``@requirement`` marks the test
carries: ``pytest_generate_tests`` looks up each cited requirement in the manifest and computes
the (transport, spec_version) cells via :func:`compute_cells`, applying arm exclusions, version
bounds, and known-failure xfails declaratively.
"""
from functools import partial
import pytest
from tests.interaction._connect import Connect, connect_in_memory, connect_over_sse, connect_over_streamable_http
from tests.interaction._connect import (
Connect,
connect_in_memory,
connect_over_sse,
connect_over_streamable_http,
connect_over_streamable_http_stateless,
)
from tests.interaction._requirements import REQUIREMENTS, compute_cells
_FACTORIES: dict[str, Connect] = {
"in-memory": connect_in_memory,
"streamable-http": connect_over_streamable_http,
"streamable-http-stateless": connect_over_streamable_http_stateless,
"sse": connect_over_sse,
}
@pytest.fixture(params=sorted(_FACTORIES))
def pytest_generate_tests(metafunc: pytest.Metafunc) -> None:
"""Parametrize ``connect`` from the test's stacked ``@requirement`` marks."""
if "connect" not in metafunc.fixturenames:
return
requirements = [REQUIREMENTS[mark.args[0]] for mark in metafunc.definition.iter_markers("requirement")]
metafunc.parametrize("connect", compute_cells(requirements), indirect=True)
@pytest.fixture
def connect(request: pytest.FixtureRequest) -> Connect:
"""The transport-parametrized connection factory: a test using it runs once per transport.
"""The transport-parametrized connection factory: a test using it runs once per matrix cell.
Tests that are tied to one transport (the wire-recording tests, the bare-ClientSession tests,
the transport-specific tests under transports/) do not use this fixture and connect directly.
"""
transport_name = request.param
assert isinstance(transport_name, str)
return _FACTORIES[transport_name]
transport, spec_version = request.param
assert isinstance(transport, str)
assert isinstance(spec_version, str)
return partial(_FACTORIES[transport], protocol_version=spec_version)

258
tests/interaction/test_coverage.py
View file

@ -11,10 +11,29 @@ import importlib
import re
from pathlib import Path
from types import ModuleType
from typing import cast
import pytest
from tests.interaction._requirements import REQUIREMENTS, Requirement, covered_by, requirement
from mcp.shared.version import KNOWN_PROTOCOL_VERSIONS
from mcp.types import LATEST_PROTOCOL_VERSION
from tests.interaction._requirements import (
CONNECTABLE_TRANSPORTS,
REQUIREMENTS,
SPEC_2026_BASE_URL,
SPEC_BASE_URL,
SPEC_VERSIONS,
ArmExclusion,
KnownFailure,
Requirement,
SpecVersion,
Transport,
cell_id,
compute_cells,
covered_by,
requirement,
)
from tests.interaction.conftest import _FACTORIES
_SUITE_ROOT = Path(__file__).parent
_REPO_ROOT = _SUITE_ROOT.parent.parent
@ -93,6 +112,87 @@ def test_deferral_reasons_cite_existing_paths() -> None:
assert not missing, f"Deferral reasons citing paths that do not exist: {missing}"
def test_spec_versions_are_known_and_include_latest() -> None:
"""Every active spec version is one the SDK knows about, and the SDK's latest is on the active axis."""
assert set(SPEC_VERSIONS) <= set(KNOWN_PROTOCOL_VERSIONS)
assert LATEST_PROTOCOL_VERSION in SPEC_VERSIONS
def test_spec_base_urls_are_pinned_to_their_revision() -> None:
"""SPEC_BASE_URL constants are pinned literals, so growing SPEC_VERSIONS cannot repoint existing source links."""
assert SPEC_BASE_URL == "https://modelcontextprotocol.io/specification/2025-11-25"
assert SPEC_2026_BASE_URL == "https://modelcontextprotocol.io/specification/2026-07-28"
def test_connectable_transports_match_connect_factories() -> None:
"""CONNECTABLE_TRANSPORTS and the conftest factory map name exactly the same transports."""
assert set(CONNECTABLE_TRANSPORTS) == set(_FACTORIES)
def test_supersession_links_are_symmetric_and_versioned() -> None:
"""``supersedes``/``superseded_by`` reference real entries, agree in both directions, and carry version bounds."""
broken = [
f"{req_id} -> {target}"
for req_id, req in REQUIREMENTS.items()
for target in req.supersedes
if target not in REQUIREMENTS or REQUIREMENTS[target].superseded_by != req_id or req.added_in is None
] + [
f"{req_id} <- {req.superseded_by}"
for req_id, req in REQUIREMENTS.items()
if req.superseded_by is not None
if req.superseded_by not in REQUIREMENTS
or req_id not in REQUIREMENTS[req.superseded_by].supersedes
or req.removed_in is None
]
assert not broken, f"Broken supersession links (forward '->' or back '<-'): {broken}"
def test_removed_entry_has_disposition() -> None:
"""Every retired requirement carries either a forward link or a prose note explaining the retirement."""
undisposed = [
req_id
for req_id, req in REQUIREMENTS.items()
if req.removed_in is not None and req.superseded_by is None and req.note is None
]
assert not undisposed, f"Requirements with removed_in but no superseded_by or note: {undisposed}"
def test_transport_restriction_has_note() -> None:
"""Every transport-restricted requirement carries a note explaining why it is transport-specific."""
missing = [req_id for req_id, req in REQUIREMENTS.items() if req.transports is not None and req.note is None]
assert not missing, f"Requirements with transports= but no note: {missing}"
def test_every_arm_exclusion_targets_a_reachable_cell() -> None:
"""Every arm exclusion names a connectable transport (or wildcards).
spec_version is type-checked against the SpecVersion Literal and may reference a version not yet
on the active SPEC_VERSIONS axis, so pre-staged exclusions for an upcoming revision are permitted.
"""
unreachable = [
f"{req_id}: {exclusion}"
for req_id, req in REQUIREMENTS.items()
for exclusion in req.arm_exclusions
if exclusion.transport is not None and exclusion.transport not in CONNECTABLE_TRANSPORTS
]
assert not unreachable, f"Arm exclusions targeting unreachable cells: {unreachable}"
def test_every_known_failure_targets_a_reachable_cell() -> None:
"""Every known failure names a connectable transport (or wildcards).
spec_version is type-checked against the SpecVersion Literal and may reference a version not yet
on the active SPEC_VERSIONS axis, so pre-staged exclusions for an upcoming revision are permitted.
"""
unreachable = [
f"{req_id}: {failure}"
for req_id, req in REQUIREMENTS.items()
for failure in req.known_failures
if failure.transport is not None and failure.transport not in CONNECTABLE_TRANSPORTS
]
assert not unreachable, f"Known failures targeting unreachable cells: {unreachable}"
def test_unknown_requirement_id_is_rejected() -> None:
"""Marking a test with an ID that is not in the manifest fails at decoration time."""
with pytest.raises(KeyError, match="Unknown requirement id 'tools:call:does-not-exist'"):
@ -103,3 +203,159 @@ def test_invalid_requirement_source_is_rejected() -> None:
"""A requirement whose source is not a spec URL, 'sdk', or an issue reference fails at construction."""
with pytest.raises(ValueError, match="source must be a specification URL"):
Requirement(source="https://example.com/not-the-spec", behavior="Never constructed.")
def test_arm_exclusion_with_unknown_spec_version_is_rejected() -> None:
"""An arm exclusion naming a spec version outside KNOWN_PROTOCOL_VERSIONS fails at construction."""
with pytest.raises(ValueError, match="is not in KNOWN_PROTOCOL_VERSIONS"):
ArmExclusion(reason="requires-session", spec_version=cast("SpecVersion", "2099-01-01"))
def test_known_failure_with_empty_note_is_rejected() -> None:
"""A known failure with a blank note fails at construction."""
with pytest.raises(ValueError, match="note must be non-empty"):
KnownFailure(note=" ")
def test_known_failure_with_unknown_spec_version_is_rejected() -> None:
"""A known failure naming a spec version outside KNOWN_PROTOCOL_VERSIONS fails at construction."""
with pytest.raises(ValueError, match="is not in KNOWN_PROTOCOL_VERSIONS"):
KnownFailure(note="x", spec_version=cast("SpecVersion", "2099-01-01"))
def test_known_failure_with_malformed_issue_is_rejected() -> None:
"""A known failure whose issue reference is neither '#<n>' nor a GitHub URL fails at construction."""
with pytest.raises(ValueError, match="must be '#<n>' or a GitHub URL"):
KnownFailure(note="x", issue="not-a-link")
def test_requirement_with_unknown_added_in_is_rejected() -> None:
"""A requirement whose added_in is outside KNOWN_PROTOCOL_VERSIONS fails at construction."""
with pytest.raises(ValueError, match="added_in .* is not in KNOWN_PROTOCOL_VERSIONS"):
Requirement(source="sdk", behavior="x", added_in=cast("SpecVersion", "2099-01-01"))
def test_requirement_with_unknown_removed_in_is_rejected() -> None:
"""A requirement whose removed_in is outside KNOWN_PROTOCOL_VERSIONS fails at construction."""
with pytest.raises(ValueError, match="removed_in .* is not in KNOWN_PROTOCOL_VERSIONS"):
Requirement(source="sdk", behavior="x", removed_in=cast("SpecVersion", "2099-01-01"))
def test_requirement_with_empty_version_range_is_rejected() -> None:
"""A requirement whose added_in is not strictly earlier than its removed_in fails at construction."""
with pytest.raises(ValueError, match="must be earlier than"):
Requirement(source="sdk", behavior="x", added_in="2025-11-25", removed_in="2025-11-25")
def _req(
*,
added_in: SpecVersion | None = None,
removed_in: SpecVersion | None = None,
transports: tuple[Transport, ...] | None = None,
arm_exclusions: tuple[ArmExclusion, ...] = (),
known_failures: tuple[KnownFailure, ...] = (),
) -> Requirement:
"""Build a synthetic Requirement for compute_cells() unit tests."""
return Requirement(
source="sdk",
behavior="x",
added_in=added_in,
removed_in=removed_in,
transports=transports,
arm_exclusions=arm_exclusions,
known_failures=known_failures,
)
def test_compute_cells_with_no_requirements_yields_full_grid() -> None:
"""With a single-version axis, an empty requirement list yields one cell per connectable transport."""
cells = compute_cells([], spec_versions=("2025-11-25",))
assert [c.id for c in cells] == ["in-memory", "sse", "streamable-http", "streamable-http-stateless"]
assert [c.values for c in cells] == [
(("in-memory", "2025-11-25"),),
(("sse", "2025-11-25"),),
(("streamable-http", "2025-11-25"),),
(("streamable-http-stateless", "2025-11-25"),),
]
def test_compute_cells_intersects_stacked_version_ranges() -> None:
"""Stacked requirements intersect their [added_in, removed_in) windows: a cell survives only if all admit it."""
cells = compute_cells(
[_req(removed_in="2026-07-28"), _req(added_in="2025-11-25")],
spec_versions=("2025-11-25", "2026-07-28"),
)
assert [c.id for c in cells] == [
"in-memory-2025-11-25",
"sse-2025-11-25",
"streamable-http-2025-11-25",
"streamable-http-stateless-2025-11-25",
]
def test_compute_cells_drops_era_locked_transport_outside_its_versions() -> None:
"""A transport listed in TRANSPORT_SPEC_VERSIONS only appears for the spec versions it serves."""
cells = compute_cells([], spec_versions=("2025-11-25", "2026-07-28"))
assert [c.id for c in cells] == [
"in-memory-2025-11-25",
"sse-2025-11-25",
"streamable-http-2025-11-25",
"streamable-http-stateless-2025-11-25",
"in-memory-2026-07-28",
"streamable-http-2026-07-28",
"streamable-http-stateless-2026-07-28",
]
def test_compute_cells_honours_arm_exclusion_from_any_stacked_requirement() -> None:
"""An arm exclusion on any stacked requirement drops the matching cell even when other requirements have none."""
cells = compute_cells(
[_req(), _req(arm_exclusions=(ArmExclusion(reason="requires-session", transport="sse"),))],
spec_versions=("2025-11-25",),
)
assert [c.id for c in cells] == ["in-memory", "streamable-http", "streamable-http-stateless"]
def test_compute_cells_wildcard_arm_exclusion_drops_every_cell() -> None:
"""An arm exclusion with both transport and spec_version unset matches every cell, leaving none."""
cells = compute_cells([_req(arm_exclusions=(ArmExclusion(reason="requires-session"),))])
assert cells == []
def test_compute_cells_marks_known_failure_as_strict_xfail() -> None:
"""A known failure attaches a strict xfail mark to exactly the matching cell and leaves others unmarked."""
cells = compute_cells(
[_req(known_failures=(KnownFailure(note="broken on sse", transport="sse"),))],
spec_versions=("2025-11-25",),
)
by_id = {c.id: c for c in cells}
assert set(by_id) == {"in-memory", "sse", "streamable-http", "streamable-http-stateless"}
assert by_id["sse"].marks[0].name == "xfail"
assert by_id["sse"].marks[0].kwargs == {"reason": "broken on sse", "strict": True}
assert by_id["in-memory"].marks == ()
assert by_id["streamable-http"].marks == ()
assert by_id["streamable-http-stateless"].marks == ()
def test_compute_cells_wildcard_known_failure_marks_every_cell() -> None:
"""A known failure with both transport and spec_version unset marks every emitted cell as strict xfail."""
cells = compute_cells([_req(known_failures=(KnownFailure(note="all broken"),))], spec_versions=("2025-11-25",))
assert len(cells) == 4
assert all(c.marks[0].name == "xfail" for c in cells)
assert all(c.marks[0].kwargs == {"reason": "all broken", "strict": True} for c in cells)
def test_compute_cells_ignores_transports_field() -> None:
"""Requirement.transports is descriptive metadata only and does not filter the cell grid."""
cells = compute_cells([_req(transports=("stdio",))], spec_versions=("2025-11-25",))
assert [c.id for c in cells] == list(CONNECTABLE_TRANSPORTS)
def test_cell_id_omits_version_when_single_spec_version() -> None:
"""With a single-version axis the cell id is just the transport name."""
assert cell_id("sse", "2025-11-25", spec_versions=("2025-11-25",)) == "sse"
def test_cell_id_appends_version_when_multiple_spec_versions() -> None:
"""With more than one active spec version the cell id gains a -<version> suffix."""
assert cell_id("sse", "2025-11-25", spec_versions=("2025-11-25", "2026-07-28")) == "sse-2025-11-25"