tests/interaction: era-axis machinery for the requirements manifest (#2909)

This commit is contained in:
Max 2026-06-19 12:59:39 +01:00 • committed by GitHub
commit 364b762ef6
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 1142 additions and 30 deletions

View file

@ -56,10 +56,12 @@ test body — each directory pins its flavour's true output exactly.
Transport-agnostic tests take the `connect` fixture instead of constructing `Client(server)` Transport-agnostic tests take the `connect` fixture instead of constructing `Client(server)`
directly, and therefore run once per transport: over the in-memory transport, over the server's directly, and therefore run once per transport: over the in-memory transport, over the server's
real streamable HTTP app driven in-process through the streaming bridge, and over the legacy SSE real streamable HTTP app driven in-process through the streaming bridge (in both stateful and
transport the same way. A test connects with `async with connect(server, ...) as client:` and stateless configurations), and over the legacy SSE transport the same way. A test connects with
asserts the same output on every leg, because the transport is not supposed to change observable `async with connect(server, ...) as client:` and asserts the same output on every leg, because the
behaviour. Tests that are tied to one transport do not use the fixture: the wire-recording tests transport is not supposed to change observable behaviour. Requirements that need a server-to-client
back-channel or persisted session state are carved out of the stateless arm via `arm_exclusions`.
Tests that are tied to one transport do not use the fixture: the wire-recording tests
(their seam is the in-memory stream pair), the bare-`ClientSession` lifecycle tests, the (their seam is the in-memory stream pair), the bare-`ClientSession` lifecycle tests, the
real-clock timeout tests (the timeout machinery is transport-independent and must not race real-clock timeout tests (the timeout machinery is transport-independent and must not race
transport latency), and everything under `transports/`, which pins behaviour only observable on transport latency), and everything under `transports/`, which pins behaviour only observable on
@ -95,6 +97,14 @@ clients can share one session manager.
would require real-time waits the suite refuses. would require real-time waits the suite refuses.
- **`transports`** names the transports a behaviour applies to; omitted means transport-independent. - **`transports`** names the transports a behaviour applies to; omitted means transport-independent.
- **`issue`** carries the tracking link for a recorded gap once one is filed. - **`issue`** carries the tracking link for a recorded gap once one is filed.
- **`note`** carries free-form context that does not fit `divergence` or `deferred`.
- **`added_in`** / **`removed_in`** bound the spec versions the behaviour exists in, as a half-open
`[added_in, removed_in)` window.
- **`supersedes`** / **`superseded_by`** link a retired entry to its replacement; the link is
bidirectional and both ends must be versioned.
- **`arm_exclusions`** carve specific `(transport, spec_version)` matrix cells out with a typed
`ArmExclusionReason`.
- **`known_failures`** mark specific `(transport, spec_version)` cells as strict xfail.
Tests link themselves to the manifest with a decorator: Tests link themselves to the manifest with a decorator:
@ -126,15 +136,60 @@ This is also the triage key for any rewrite: a test that fails on the new code p
divergence note (the rewrite accidentally fixed a known gap — decide whether to keep the fix) or divergence note (the rewrite accidentally fixed a known gap — decide whether to keep the fix) or
it does not (the rewrite broke something that was correct — fix the rewrite). it does not (the rewrite broke something that was correct — fix the rewrite).
### When a new spec revision is released ### Spec versions and the era axis
1. Update `SPEC_REVISION` and walk the new revision's changelog. `SPEC_VERSIONS` in `_requirements.py` is the ordered tuple of protocol revisions the suite
2. For each changed interaction, find its requirements (the IDs use the wire method strings the exercises. `SPEC_BASE_URL` (and `SPEC_2026_BASE_URL`) are pinned literals — not derived from
changelog speaks in), re-audit the tests against the new text, and update `source` links and `SPEC_VERSIONS` — so growing the active axis never repoints existing `source` links. The
assertions where behaviour legitimately changed. `connect` fixture fans out over `CONNECTABLE_TRANSPORTS × SPEC_VERSIONS`, but the grid is
3. New interactions get new requirements and new tests; removed interactions get their filtered per test:
requirements deleted along with their tests. `pytest_generate_tests` reads the test's stacked `@requirement` marks and calls `compute_cells()`,
4. A behaviour that is correct under both revisions needs no change beyond the `source` link. which intersects the admissible cells across every cited requirement — a cell survives only if
**all** of the test's requirements admit it.
`streamable-http-stateless` is the fourth connectable transport: the 2025-era unofficial stateless
mode where each request opens a fresh transport, no session id is issued, and there is no standalone
GET stream. Requirements that need a server→client back-channel or persisted session state are
excluded from that arm via `arm_exclusions` (reasons `server-initiated-request` and
`requires-session`).
What admits or excludes a cell:
- **`added_in` / `removed_in`** gate which spec versions a requirement exists in, as a half-open
`[added_in, removed_in)` window. A test runs only on versions inside every cited requirement's
window.
- **`arm_exclusions`** carve specific `(transport, spec_version)` cells out with a typed
`ArmExclusionReason`. The reason vocabulary doubles as a re-admission checklist: when the gap
closes, grep for the reason string to find every cell to re-admit.
- **`known_failures`** keep a cell in the grid but mark it as a strict xfail — the test runs and
must fail; an unexpected pass fails the suite.
- **`TRANSPORT_SPEC_VERSIONS`** era-locks a transport to a subset of spec versions (currently only
`sse` is locked to `2025-11-25`). A `(transport, version)` cell is dropped if the version is not
in the transport's entry; transports absent from the map serve every spec version. This is the
mechanism for cutting an entire transport off from a new revision (or admitting it).
- **`transports`** is descriptive metadata for the non-`connect` transport-specific suites under
`transports/` and does **not** drive cell generation. Only `arm_exclusions`, `added_in`,
`removed_in`, and `TRANSPORT_SPEC_VERSIONS` filter the grid.
- **`supersedes` / `superseded_by`** link a retired entry to its replacement. `test_coverage.py`
enforces that links are bidirectional and versioned: the retired entry carries `removed_in`, the
replacement carries `added_in`.
Node IDs stay `[transport]` while `len(SPEC_VERSIONS) == 1`, so today's test IDs are
byte-identical to before the era axis existed. They become `[transport-version]` the moment a
second version is appended to `SPEC_VERSIONS`.
When a new spec revision lands:
1. Append the version string to `SPEC_VERSIONS` (and to the `SpecVersion` `Literal`).
2. Walk the new revision's changelog.
3. For each affected requirement: set `removed_in` on retired behaviour, add a new entry with
`added_in` for its replacement, and link the pair with `supersedes` / `superseded_by`.
Behaviour that survives unchanged needs nothing beyond a re-audit of its `source` URL.
4. For requirements that cannot run on the new era's path, add an `arm_exclusions` entry with the
appropriate `ArmExclusionReason`.
5. Review `TRANSPORT_SPEC_VERSIONS`: any era-locked transport will not produce cells on the new
version unless its entry is extended (or removed); add an entry for any transport the new
revision retires.
## Writing a test ## Writing a test

View file

@ -9,6 +9,7 @@ server's real Starlette app through the in-process streaming bridge, so the full
from collections.abc import AsyncIterator, Awaitable, Callable, Iterable from collections.abc import AsyncIterator, Awaitable, Callable, Iterable
from contextlib import AbstractAsyncContextManager, asynccontextmanager from contextlib import AbstractAsyncContextManager, asynccontextmanager
from functools import partial
from typing import Any, Protocol from typing import Any, Protocol
import httpx import httpx
@ -69,6 +70,7 @@ class Connect(Protocol):
message_handler: MessageHandlerFnT | None = None, message_handler: MessageHandlerFnT | None = None,
client_info: Implementation | None = None, client_info: Implementation | None = None,
elicitation_callback: ElicitationFnT | None = None, elicitation_callback: ElicitationFnT | None = None,
protocol_version: str = LATEST_PROTOCOL_VERSION,
) -> AbstractAsyncContextManager[Client]: ... ) -> AbstractAsyncContextManager[Client]: ...
@ -83,6 +85,7 @@ async def connect_in_memory(
message_handler: MessageHandlerFnT | None = None, message_handler: MessageHandlerFnT | None = None,
client_info: Implementation | None = None, client_info: Implementation | None = None,
elicitation_callback: ElicitationFnT | None = None, elicitation_callback: ElicitationFnT | None = None,
protocol_version: str = LATEST_PROTOCOL_VERSION,
) -> AsyncIterator[Client]: ) -> AsyncIterator[Client]:
"""Yield a Client connected to the server over the in-memory transport.""" """Yield a Client connected to the server over the in-memory transport."""
async with Client( async with Client(
@ -113,13 +116,15 @@ async def connect_over_streamable_http(
message_handler: MessageHandlerFnT | None = None, message_handler: MessageHandlerFnT | None = None,
client_info: Implementation | None = None, client_info: Implementation | None = None,
elicitation_callback: ElicitationFnT | None = None, elicitation_callback: ElicitationFnT | None = None,
protocol_version: str = LATEST_PROTOCOL_VERSION,
) -> AsyncIterator[Client]: ) -> AsyncIterator[Client]:
"""Yield a Client connected to the server's streamable HTTP app, entirely in process. """Yield a Client connected to the server's streamable HTTP app, entirely in process.
With the defaults this is the matrix leg (stateful sessions, SSE responses); the With the defaults this is the matrix leg (stateful sessions, SSE responses); the stateless
transport-specific tests pass `stateless_http` or `json_response` to select the other matrix arm binds `stateless_http=True` (see `connect_over_streamable_http_stateless`);
server modes, and the resumability tests pass an `event_store` (with `retry_interval=0` so transport-specific tests pass `json_response` to select the other server mode, and the
the client's reconnection wait is a no-op). resumability tests pass an `event_store` (with `retry_interval=0` so the client's
reconnection wait is a no-op).
""" """
app = server.streamable_http_app( app = server.streamable_http_app(
stateless_http=stateless_http, stateless_http=stateless_http,
@ -145,6 +150,12 @@ async def connect_over_streamable_http(
yield client yield client
connect_over_streamable_http_stateless: Connect = partial(connect_over_streamable_http, stateless_http=True)
"""The streamable-http matrix arm with the server in stateless mode (fresh transport per request,
no session id, no standalone GET stream). The same shared Server instance backs every request --
stateless mode does not require a server factory."""
@asynccontextmanager @asynccontextmanager
async def mounted_app( async def mounted_app(
server: Server | MCPServer, server: Server | MCPServer,
@ -326,6 +337,7 @@ async def connect_over_sse(
message_handler: MessageHandlerFnT | None = None, message_handler: MessageHandlerFnT | None = None,
client_info: Implementation | None = None, client_info: Implementation | None = None,
elicitation_callback: ElicitationFnT | None = None, elicitation_callback: ElicitationFnT | None = None,
protocol_version: str = LATEST_PROTOCOL_VERSION,
) -> AsyncIterator[Client]: ) -> AsyncIterator[Client]:
"""Yield a Client connected to the server's legacy SSE transport, entirely in process.""" """Yield a Client connected to the server's legacy SSE transport, entirely in process."""
app, _ = build_sse_app(server) app, _ = build_sse_app(server)

File diff suppressed because it is too large Load diff

View file

@ -1,23 +1,48 @@
"""Shared fixtures for the interaction suite.""" """Shared fixtures for the interaction suite.
The ``connect`` fixture is parametrized per-test from the ``@requirement`` marks the test
carries: ``pytest_generate_tests`` looks up each cited requirement in the manifest and computes
the (transport, spec_version) cells via :func:`compute_cells`, applying arm exclusions, version
bounds, and known-failure xfails declaratively.
"""
from functools import partial
import pytest import pytest
from tests.interaction._connect import Connect, connect_in_memory, connect_over_sse, connect_over_streamable_http from tests.interaction._connect import (
Connect,
connect_in_memory,
connect_over_sse,
connect_over_streamable_http,
connect_over_streamable_http_stateless,
)
from tests.interaction._requirements import REQUIREMENTS, compute_cells
_FACTORIES: dict[str, Connect] = { _FACTORIES: dict[str, Connect] = {
"in-memory": connect_in_memory, "in-memory": connect_in_memory,
"streamable-http": connect_over_streamable_http, "streamable-http": connect_over_streamable_http,
"streamable-http-stateless": connect_over_streamable_http_stateless,
"sse": connect_over_sse, "sse": connect_over_sse,
} }
@pytest.fixture(params=sorted(_FACTORIES)) def pytest_generate_tests(metafunc: pytest.Metafunc) -> None:
"""Parametrize ``connect`` from the test's stacked ``@requirement`` marks."""
if "connect" not in metafunc.fixturenames:
return
requirements = [REQUIREMENTS[mark.args[0]] for mark in metafunc.definition.iter_markers("requirement")]
metafunc.parametrize("connect", compute_cells(requirements), indirect=True)
@pytest.fixture
def connect(request: pytest.FixtureRequest) -> Connect: def connect(request: pytest.FixtureRequest) -> Connect:
"""The transport-parametrized connection factory: a test using it runs once per transport. """The transport-parametrized connection factory: a test using it runs once per matrix cell.
Tests that are tied to one transport (the wire-recording tests, the bare-ClientSession tests, Tests that are tied to one transport (the wire-recording tests, the bare-ClientSession tests,
the transport-specific tests under transports/) do not use this fixture and connect directly. the transport-specific tests under transports/) do not use this fixture and connect directly.
""" """
transport_name = request.param transport, spec_version = request.param
assert isinstance(transport_name, str) assert isinstance(transport, str)
return _FACTORIES[transport_name] assert isinstance(spec_version, str)
return partial(_FACTORIES[transport], protocol_version=spec_version)

View file

@ -11,10 +11,29 @@ import importlib
import re import re
from pathlib import Path from pathlib import Path
from types import ModuleType from types import ModuleType
from typing import cast
import pytest import pytest
from tests.interaction._requirements import REQUIREMENTS, Requirement, covered_by, requirement from mcp.shared.version import KNOWN_PROTOCOL_VERSIONS
from mcp.types import LATEST_PROTOCOL_VERSION
from tests.interaction._requirements import (
CONNECTABLE_TRANSPORTS,
REQUIREMENTS,
SPEC_2026_BASE_URL,
SPEC_BASE_URL,
SPEC_VERSIONS,
ArmExclusion,
KnownFailure,
Requirement,
SpecVersion,
Transport,
cell_id,
compute_cells,
covered_by,
requirement,
)
from tests.interaction.conftest import _FACTORIES
_SUITE_ROOT = Path(__file__).parent _SUITE_ROOT = Path(__file__).parent
_REPO_ROOT = _SUITE_ROOT.parent.parent _REPO_ROOT = _SUITE_ROOT.parent.parent
@ -93,6 +112,87 @@ def test_deferral_reasons_cite_existing_paths() -> None:
assert not missing, f"Deferral reasons citing paths that do not exist: {missing}" assert not missing, f"Deferral reasons citing paths that do not exist: {missing}"
def test_spec_versions_are_known_and_include_latest() -> None:
"""Every active spec version is one the SDK knows about, and the SDK's latest is on the active axis."""
assert set(SPEC_VERSIONS) <= set(KNOWN_PROTOCOL_VERSIONS)
assert LATEST_PROTOCOL_VERSION in SPEC_VERSIONS
def test_spec_base_urls_are_pinned_to_their_revision() -> None:
"""SPEC_BASE_URL constants are pinned literals, so growing SPEC_VERSIONS cannot repoint existing source links."""
assert SPEC_BASE_URL == "https://modelcontextprotocol.io/specification/2025-11-25"
assert SPEC_2026_BASE_URL == "https://modelcontextprotocol.io/specification/2026-07-28"
def test_connectable_transports_match_connect_factories() -> None:
"""CONNECTABLE_TRANSPORTS and the conftest factory map name exactly the same transports."""
assert set(CONNECTABLE_TRANSPORTS) == set(_FACTORIES)
def test_supersession_links_are_symmetric_and_versioned() -> None:
"""``supersedes``/``superseded_by`` reference real entries, agree in both directions, and carry version bounds."""
broken = [
f"{req_id} -> {target}"
for req_id, req in REQUIREMENTS.items()
for target in req.supersedes
if target not in REQUIREMENTS or REQUIREMENTS[target].superseded_by != req_id or req.added_in is None
] + [
f"{req_id} <- {req.superseded_by}"
for req_id, req in REQUIREMENTS.items()
if req.superseded_by is not None
if req.superseded_by not in REQUIREMENTS
or req_id not in REQUIREMENTS[req.superseded_by].supersedes
or req.removed_in is None
]
assert not broken, f"Broken supersession links (forward '->' or back '<-'): {broken}"
def test_removed_entry_has_disposition() -> None:
"""Every retired requirement carries either a forward link or a prose note explaining the retirement."""
undisposed = [
req_id
for req_id, req in REQUIREMENTS.items()
if req.removed_in is not None and req.superseded_by is None and req.note is None
]
assert not undisposed, f"Requirements with removed_in but no superseded_by or note: {undisposed}"
def test_transport_restriction_has_note() -> None:
"""Every transport-restricted requirement carries a note explaining why it is transport-specific."""
missing = [req_id for req_id, req in REQUIREMENTS.items() if req.transports is not None and req.note is None]
assert not missing, f"Requirements with transports= but no note: {missing}"
def test_every_arm_exclusion_targets_a_reachable_cell() -> None:
"""Every arm exclusion names a connectable transport (or wildcards).
spec_version is type-checked against the SpecVersion Literal and may reference a version not yet
on the active SPEC_VERSIONS axis, so pre-staged exclusions for an upcoming revision are permitted.
"""
unreachable = [
f"{req_id}: {exclusion}"
for req_id, req in REQUIREMENTS.items()
for exclusion in req.arm_exclusions
if exclusion.transport is not None and exclusion.transport not in CONNECTABLE_TRANSPORTS
]
assert not unreachable, f"Arm exclusions targeting unreachable cells: {unreachable}"
def test_every_known_failure_targets_a_reachable_cell() -> None:
"""Every known failure names a connectable transport (or wildcards).
spec_version is type-checked against the SpecVersion Literal and may reference a version not yet
on the active SPEC_VERSIONS axis, so pre-staged exclusions for an upcoming revision are permitted.
"""
unreachable = [
f"{req_id}: {failure}"
for req_id, req in REQUIREMENTS.items()
for failure in req.known_failures
if failure.transport is not None and failure.transport not in CONNECTABLE_TRANSPORTS
]
assert not unreachable, f"Known failures targeting unreachable cells: {unreachable}"
def test_unknown_requirement_id_is_rejected() -> None: def test_unknown_requirement_id_is_rejected() -> None:
"""Marking a test with an ID that is not in the manifest fails at decoration time.""" """Marking a test with an ID that is not in the manifest fails at decoration time."""
with pytest.raises(KeyError, match="Unknown requirement id 'tools:call:does-not-exist'"): with pytest.raises(KeyError, match="Unknown requirement id 'tools:call:does-not-exist'"):
@ -103,3 +203,159 @@ def test_invalid_requirement_source_is_rejected() -> None:
"""A requirement whose source is not a spec URL, 'sdk', or an issue reference fails at construction.""" """A requirement whose source is not a spec URL, 'sdk', or an issue reference fails at construction."""
with pytest.raises(ValueError, match="source must be a specification URL"): with pytest.raises(ValueError, match="source must be a specification URL"):
Requirement(source="https://example.com/not-the-spec", behavior="Never constructed.") Requirement(source="https://example.com/not-the-spec", behavior="Never constructed.")
def test_arm_exclusion_with_unknown_spec_version_is_rejected() -> None:
"""An arm exclusion naming a spec version outside KNOWN_PROTOCOL_VERSIONS fails at construction."""
with pytest.raises(ValueError, match="is not in KNOWN_PROTOCOL_VERSIONS"):
ArmExclusion(reason="requires-session", spec_version=cast("SpecVersion", "2099-01-01"))
def test_known_failure_with_empty_note_is_rejected() -> None:
"""A known failure with a blank note fails at construction."""
with pytest.raises(ValueError, match="note must be non-empty"):
KnownFailure(note=" ")
def test_known_failure_with_unknown_spec_version_is_rejected() -> None:
"""A known failure naming a spec version outside KNOWN_PROTOCOL_VERSIONS fails at construction."""
with pytest.raises(ValueError, match="is not in KNOWN_PROTOCOL_VERSIONS"):
KnownFailure(note="x", spec_version=cast("SpecVersion", "2099-01-01"))
def test_known_failure_with_malformed_issue_is_rejected() -> None:
"""A known failure whose issue reference is neither '#<n>' nor a GitHub URL fails at construction."""
with pytest.raises(ValueError, match="must be '#<n>' or a GitHub URL"):
KnownFailure(note="x", issue="not-a-link")
def test_requirement_with_unknown_added_in_is_rejected() -> None:
"""A requirement whose added_in is outside KNOWN_PROTOCOL_VERSIONS fails at construction."""
with pytest.raises(ValueError, match="added_in .* is not in KNOWN_PROTOCOL_VERSIONS"):
Requirement(source="sdk", behavior="x", added_in=cast("SpecVersion", "2099-01-01"))
def test_requirement_with_unknown_removed_in_is_rejected() -> None:
"""A requirement whose removed_in is outside KNOWN_PROTOCOL_VERSIONS fails at construction."""
with pytest.raises(ValueError, match="removed_in .* is not in KNOWN_PROTOCOL_VERSIONS"):
Requirement(source="sdk", behavior="x", removed_in=cast("SpecVersion", "2099-01-01"))
def test_requirement_with_empty_version_range_is_rejected() -> None:
"""A requirement whose added_in is not strictly earlier than its removed_in fails at construction."""
with pytest.raises(ValueError, match="must be earlier than"):
Requirement(source="sdk", behavior="x", added_in="2025-11-25", removed_in="2025-11-25")
def _req(
*,
added_in: SpecVersion | None = None,
removed_in: SpecVersion | None = None,
transports: tuple[Transport, ...] | None = None,
arm_exclusions: tuple[ArmExclusion, ...] = (),
known_failures: tuple[KnownFailure, ...] = (),
) -> Requirement:
"""Build a synthetic Requirement for compute_cells() unit tests."""
return Requirement(
source="sdk",
behavior="x",
added_in=added_in,
removed_in=removed_in,
transports=transports,
arm_exclusions=arm_exclusions,
known_failures=known_failures,
)
def test_compute_cells_with_no_requirements_yields_full_grid() -> None:
"""With a single-version axis, an empty requirement list yields one cell per connectable transport."""
cells = compute_cells([], spec_versions=("2025-11-25",))
assert [c.id for c in cells] == ["in-memory", "sse", "streamable-http", "streamable-http-stateless"]
assert [c.values for c in cells] == [
(("in-memory", "2025-11-25"),),
(("sse", "2025-11-25"),),
(("streamable-http", "2025-11-25"),),
(("streamable-http-stateless", "2025-11-25"),),
]
def test_compute_cells_intersects_stacked_version_ranges() -> None:
"""Stacked requirements intersect their [added_in, removed_in) windows: a cell survives only if all admit it."""
cells = compute_cells(
[_req(removed_in="2026-07-28"), _req(added_in="2025-11-25")],
spec_versions=("2025-11-25", "2026-07-28"),
)
assert [c.id for c in cells] == [
"in-memory-2025-11-25",
"sse-2025-11-25",
"streamable-http-2025-11-25",
"streamable-http-stateless-2025-11-25",
]
def test_compute_cells_drops_era_locked_transport_outside_its_versions() -> None:
"""A transport listed in TRANSPORT_SPEC_VERSIONS only appears for the spec versions it serves."""
cells = compute_cells([], spec_versions=("2025-11-25", "2026-07-28"))
assert [c.id for c in cells] == [
"in-memory-2025-11-25",
"sse-2025-11-25",
"streamable-http-2025-11-25",
"streamable-http-stateless-2025-11-25",
"in-memory-2026-07-28",
"streamable-http-2026-07-28",
"streamable-http-stateless-2026-07-28",
]
def test_compute_cells_honours_arm_exclusion_from_any_stacked_requirement() -> None:
"""An arm exclusion on any stacked requirement drops the matching cell even when other requirements have none."""
cells = compute_cells(
[_req(), _req(arm_exclusions=(ArmExclusion(reason="requires-session", transport="sse"),))],
spec_versions=("2025-11-25",),
)
assert [c.id for c in cells] == ["in-memory", "streamable-http", "streamable-http-stateless"]
def test_compute_cells_wildcard_arm_exclusion_drops_every_cell() -> None:
"""An arm exclusion with both transport and spec_version unset matches every cell, leaving none."""
cells = compute_cells([_req(arm_exclusions=(ArmExclusion(reason="requires-session"),))])
assert cells == []
def test_compute_cells_marks_known_failure_as_strict_xfail() -> None:
"""A known failure attaches a strict xfail mark to exactly the matching cell and leaves others unmarked."""
cells = compute_cells(
[_req(known_failures=(KnownFailure(note="broken on sse", transport="sse"),))],
spec_versions=("2025-11-25",),
)
by_id = {c.id: c for c in cells}
assert set(by_id) == {"in-memory", "sse", "streamable-http", "streamable-http-stateless"}
assert by_id["sse"].marks[0].name == "xfail"
assert by_id["sse"].marks[0].kwargs == {"reason": "broken on sse", "strict": True}
assert by_id["in-memory"].marks == ()
assert by_id["streamable-http"].marks == ()
assert by_id["streamable-http-stateless"].marks == ()
def test_compute_cells_wildcard_known_failure_marks_every_cell() -> None:
"""A known failure with both transport and spec_version unset marks every emitted cell as strict xfail."""
cells = compute_cells([_req(known_failures=(KnownFailure(note="all broken"),))], spec_versions=("2025-11-25",))
assert len(cells) == 4
assert all(c.marks[0].name == "xfail" for c in cells)
assert all(c.marks[0].kwargs == {"reason": "all broken", "strict": True} for c in cells)
def test_compute_cells_ignores_transports_field() -> None:
"""Requirement.transports is descriptive metadata only and does not filter the cell grid."""
cells = compute_cells([_req(transports=("stdio",))], spec_versions=("2025-11-25",))
assert [c.id for c in cells] == list(CONNECTABLE_TRANSPORTS)
def test_cell_id_omits_version_when_single_spec_version() -> None:
"""With a single-version axis the cell id is just the transport name."""
assert cell_id("sse", "2025-11-25", spec_versions=("2025-11-25",)) == "sse"
def test_cell_id_appends_version_when_multiple_spec_versions() -> None:
"""With more than one active spec version the cell id gains a -<version> suffix."""
assert cell_id("sse", "2025-11-25", spec_versions=("2025-11-25", "2026-07-28")) == "sse-2025-11-25"