tests/interaction: era-axis machinery for the requirements manifest (#2909)
This commit is contained in:
parent
734746a3d9
commit
364b762ef6
1 changed files with 1142 additions and 30 deletions
|
|
@ -56,10 +56,12 @@ test body — each directory pins its flavour's true output exactly.
|
|||
|
||||
Transport-agnostic tests take the `connect` fixture instead of constructing `Client(server)`
|
||||
directly, and therefore run once per transport: over the in-memory transport, over the server's
|
||||
real streamable HTTP app driven in-process through the streaming bridge, and over the legacy SSE
|
||||
transport the same way. A test connects with `async with connect(server, ...) as client:` and
|
||||
asserts the same output on every leg, because the transport is not supposed to change observable
|
||||
behaviour. Tests that are tied to one transport do not use the fixture: the wire-recording tests
|
||||
real streamable HTTP app driven in-process through the streaming bridge (in both stateful and
|
||||
stateless configurations), and over the legacy SSE transport the same way. A test connects with
|
||||
`async with connect(server, ...) as client:` and asserts the same output on every leg, because the
|
||||
transport is not supposed to change observable behaviour. Requirements that need a server-to-client
|
||||
back-channel or persisted session state are carved out of the stateless arm via `arm_exclusions`.
|
||||
Tests that are tied to one transport do not use the fixture: the wire-recording tests
|
||||
(their seam is the in-memory stream pair), the bare-`ClientSession` lifecycle tests, the
|
||||
real-clock timeout tests (the timeout machinery is transport-independent and must not race
|
||||
transport latency), and everything under `transports/`, which pins behaviour only observable on
|
||||
|
|
@ -95,6 +97,14 @@ clients can share one session manager.
|
|||
would require real-time waits the suite refuses.
|
||||
- **`transports`** names the transports a behaviour applies to; omitted means transport-independent.
|
||||
- **`issue`** carries the tracking link for a recorded gap once one is filed.
|
||||
- **`note`** carries free-form context that does not fit `divergence` or `deferred`.
|
||||
- **`added_in`** / **`removed_in`** bound the spec versions the behaviour exists in, as a half-open
|
||||
`[added_in, removed_in)` window.
|
||||
- **`supersedes`** / **`superseded_by`** link a retired entry to its replacement; the link is
|
||||
bidirectional and both ends must be versioned.
|
||||
- **`arm_exclusions`** carve specific `(transport, spec_version)` matrix cells out with a typed
|
||||
`ArmExclusionReason`.
|
||||
- **`known_failures`** mark specific `(transport, spec_version)` cells as strict xfail.
|
||||
|
||||
Tests link themselves to the manifest with a decorator:
|
||||
|
||||
|
|
@ -126,15 +136,60 @@ This is also the triage key for any rewrite: a test that fails on the new code p
|
|||
divergence note (the rewrite accidentally fixed a known gap — decide whether to keep the fix) or
|
||||
it does not (the rewrite broke something that was correct — fix the rewrite).
|
||||
|
||||
### When a new spec revision is released
|
||||
### Spec versions and the era axis
|
||||
|
||||
1. Update `SPEC_REVISION` and walk the new revision's changelog.
|
||||
2. For each changed interaction, find its requirements (the IDs use the wire method strings the
|
||||
changelog speaks in), re-audit the tests against the new text, and update `source` links and
|
||||
assertions where behaviour legitimately changed.
|
||||
3. New interactions get new requirements and new tests; removed interactions get their
|
||||
requirements deleted along with their tests.
|
||||
4. A behaviour that is correct under both revisions needs no change beyond the `source` link.
|
||||
`SPEC_VERSIONS` in `_requirements.py` is the ordered tuple of protocol revisions the suite
|
||||
exercises. `SPEC_BASE_URL` (and `SPEC_2026_BASE_URL`) are pinned literals — not derived from
|
||||
`SPEC_VERSIONS` — so growing the active axis never repoints existing `source` links. The
|
||||
`connect` fixture fans out over `CONNECTABLE_TRANSPORTS × SPEC_VERSIONS`, but the grid is
|
||||
filtered per test:
|
||||
`pytest_generate_tests` reads the test's stacked `@requirement` marks and calls `compute_cells()`,
|
||||
which intersects the admissible cells across every cited requirement — a cell survives only if
|
||||
**all** of the test's requirements admit it.
|
||||
|
||||
`streamable-http-stateless` is the fourth connectable transport: the 2025-era unofficial stateless
|
||||
mode where each request opens a fresh transport, no session id is issued, and there is no standalone
|
||||
GET stream. Requirements that need a server→client back-channel or persisted session state are
|
||||
excluded from that arm via `arm_exclusions` (reasons `server-initiated-request` and
|
||||
`requires-session`).
|
||||
|
||||
What admits or excludes a cell:
|
||||
|
||||
- **`added_in` / `removed_in`** gate which spec versions a requirement exists in, as a half-open
|
||||
`[added_in, removed_in)` window. A test runs only on versions inside every cited requirement's
|
||||
window.
|
||||
- **`arm_exclusions`** carve specific `(transport, spec_version)` cells out with a typed
|
||||
`ArmExclusionReason`. The reason vocabulary doubles as a re-admission checklist: when the gap
|
||||
closes, grep for the reason string to find every cell to re-admit.
|
||||
- **`known_failures`** keep a cell in the grid but mark it as a strict xfail — the test runs and
|
||||
must fail; an unexpected pass fails the suite.
|
||||
- **`TRANSPORT_SPEC_VERSIONS`** era-locks a transport to a subset of spec versions (currently only
|
||||
`sse` is locked to `2025-11-25`). A `(transport, version)` cell is dropped if the version is not
|
||||
in the transport's entry; transports absent from the map serve every spec version. This is the
|
||||
mechanism for cutting an entire transport off from a new revision (or admitting it).
|
||||
- **`transports`** is descriptive metadata for the non-`connect` transport-specific suites under
|
||||
`transports/` and does **not** drive cell generation. Only `arm_exclusions`, `added_in`,
|
||||
`removed_in`, and `TRANSPORT_SPEC_VERSIONS` filter the grid.
|
||||
- **`supersedes` / `superseded_by`** link a retired entry to its replacement. `test_coverage.py`
|
||||
enforces that links are bidirectional and versioned: the retired entry carries `removed_in`, the
|
||||
replacement carries `added_in`.
|
||||
|
||||
Node IDs stay `[transport]` while `len(SPEC_VERSIONS) == 1`, so today's test IDs are
|
||||
byte-identical to before the era axis existed. They become `[transport-version]` the moment a
|
||||
second version is appended to `SPEC_VERSIONS`.
|
||||
|
||||
When a new spec revision lands:
|
||||
|
||||
1. Append the version string to `SPEC_VERSIONS` (and to the `SpecVersion` `Literal`).
|
||||
2. Walk the new revision's changelog.
|
||||
3. For each affected requirement: set `removed_in` on retired behaviour, add a new entry with
|
||||
`added_in` for its replacement, and link the pair with `supersedes` / `superseded_by`.
|
||||
Behaviour that survives unchanged needs nothing beyond a re-audit of its `source` URL.
|
||||
4. For requirements that cannot run on the new era's path, add an `arm_exclusions` entry with the
|
||||
appropriate `ArmExclusionReason`.
|
||||
5. Review `TRANSPORT_SPEC_VERSIONS`: any era-locked transport will not produce cells on the new
|
||||
version unless its entry is extended (or removed); add an entry for any transport the new
|
||||
revision retires.
|
||||
|
||||
## Writing a test
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue