tests/interaction: era-axis machinery for the requirements manifest (#2909)

This commit is contained in:
Max 2026-06-19 12:59:39 +01:00 • committed by GitHub
commit 364b762ef6
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

View file

@ -56,10 +56,12 @@ test body — each directory pins its flavour's true output exactly.
Transport-agnostic tests take the `connect` fixture instead of constructing `Client(server)`
directly, and therefore run once per transport: over the in-memory transport, over the server's
real streamable HTTP app driven in-process through the streaming bridge, and over the legacy SSE
transport the same way. A test connects with `async with connect(server, ...) as client:` and
asserts the same output on every leg, because the transport is not supposed to change observable
behaviour. Tests that are tied to one transport do not use the fixture: the wire-recording tests
real streamable HTTP app driven in-process through the streaming bridge (in both stateful and
stateless configurations), and over the legacy SSE transport the same way. A test connects with
`async with connect(server, ...) as client:` and asserts the same output on every leg, because the
transport is not supposed to change observable behaviour. Requirements that need a server-to-client
back-channel or persisted session state are carved out of the stateless arm via `arm_exclusions`.
Tests that are tied to one transport do not use the fixture: the wire-recording tests
(their seam is the in-memory stream pair), the bare-`ClientSession` lifecycle tests, the
real-clock timeout tests (the timeout machinery is transport-independent and must not race
transport latency), and everything under `transports/`, which pins behaviour only observable on
@ -95,6 +97,14 @@ clients can share one session manager.
would require real-time waits the suite refuses.
- **`transports`** names the transports a behaviour applies to; omitted means transport-independent.
- **`issue`** carries the tracking link for a recorded gap once one is filed.
- **`note`** carries free-form context that does not fit `divergence` or `deferred`.
- **`added_in`** / **`removed_in`** bound the spec versions the behaviour exists in, as a half-open
`[added_in, removed_in)` window.
- **`supersedes`** / **`superseded_by`** link a retired entry to its replacement; the link is
bidirectional and both ends must be versioned.
- **`arm_exclusions`** carve specific `(transport, spec_version)` matrix cells out with a typed
`ArmExclusionReason`.
- **`known_failures`** mark specific `(transport, spec_version)` cells as strict xfail.
Tests link themselves to the manifest with a decorator:
@ -126,15 +136,60 @@ This is also the triage key for any rewrite: a test that fails on the new code p
divergence note (the rewrite accidentally fixed a known gap — decide whether to keep the fix) or
it does not (the rewrite broke something that was correct — fix the rewrite).
### When a new spec revision is released
### Spec versions and the era axis
1. Update `SPEC_REVISION` and walk the new revision's changelog.
2. For each changed interaction, find its requirements (the IDs use the wire method strings the
changelog speaks in), re-audit the tests against the new text, and update `source` links and
assertions where behaviour legitimately changed.
3. New interactions get new requirements and new tests; removed interactions get their
requirements deleted along with their tests.
4. A behaviour that is correct under both revisions needs no change beyond the `source` link.
`SPEC_VERSIONS` in `_requirements.py` is the ordered tuple of protocol revisions the suite
exercises. `SPEC_BASE_URL` (and `SPEC_2026_BASE_URL`) are pinned literals — not derived from
`SPEC_VERSIONS` — so growing the active axis never repoints existing `source` links. The
`connect` fixture fans out over `CONNECTABLE_TRANSPORTS × SPEC_VERSIONS`, but the grid is
filtered per test:
`pytest_generate_tests` reads the test's stacked `@requirement` marks and calls `compute_cells()`,
which intersects the admissible cells across every cited requirement — a cell survives only if
**all** of the test's requirements admit it.
`streamable-http-stateless` is the fourth connectable transport: the 2025-era unofficial stateless
mode where each request opens a fresh transport, no session id is issued, and there is no standalone
GET stream. Requirements that need a server→client back-channel or persisted session state are
excluded from that arm via `arm_exclusions` (reasons `server-initiated-request` and
`requires-session`).
What admits or excludes a cell:
- **`added_in` / `removed_in`** gate which spec versions a requirement exists in, as a half-open
`[added_in, removed_in)` window. A test runs only on versions inside every cited requirement's
window.
- **`arm_exclusions`** carve specific `(transport, spec_version)` cells out with a typed
`ArmExclusionReason`. The reason vocabulary doubles as a re-admission checklist: when the gap
closes, grep for the reason string to find every cell to re-admit.
- **`known_failures`** keep a cell in the grid but mark it as a strict xfail — the test runs and
must fail; an unexpected pass fails the suite.
- **`TRANSPORT_SPEC_VERSIONS`** era-locks a transport to a subset of spec versions (currently only
`sse` is locked to `2025-11-25`). A `(transport, version)` cell is dropped if the version is not
in the transport's entry; transports absent from the map serve every spec version. This is the
mechanism for cutting an entire transport off from a new revision (or admitting it).
- **`transports`** is descriptive metadata for the non-`connect` transport-specific suites under
`transports/` and does **not** drive cell generation. Only `arm_exclusions`, `added_in`,
`removed_in`, and `TRANSPORT_SPEC_VERSIONS` filter the grid.
- **`supersedes` / `superseded_by`** link a retired entry to its replacement. `test_coverage.py`
enforces that links are bidirectional and versioned: the retired entry carries `removed_in`, the
replacement carries `added_in`.
Node IDs stay `[transport]` while `len(SPEC_VERSIONS) == 1`, so today's test IDs are
byte-identical to before the era axis existed. They become `[transport-version]` the moment a
second version is appended to `SPEC_VERSIONS`.
When a new spec revision lands:
1. Append the version string to `SPEC_VERSIONS` (and to the `SpecVersion` `Literal`).
2. Walk the new revision's changelog.
3. For each affected requirement: set `removed_in` on retired behaviour, add a new entry with
`added_in` for its replacement, and link the pair with `supersedes` / `superseded_by`.
Behaviour that survives unchanged needs nothing beyond a re-audit of its `source` URL.
4. For requirements that cannot run on the new era's path, add an `arm_exclusions` entry with the
appropriate `ArmExclusionReason`.
5. Review `TRANSPORT_SPEC_VERSIONS`: any era-locked transport will not produce cells on the new
version unless its entry is extended (or removed); add an entry for any transport the new
revision retires.
## Writing a test