conformance/src
Repository files (latest commit first)
Filename Latest commit message Latest commit date
AKS 542381b7dc
feat(tools-list): check deterministic tools/list ordering on the 2026-07-28 wire (#492)
* feat(tools-list): check deterministic tools/list ordering on the 2026-07-28 wire

The 2026-07-28 revision adds to server/tools.mdx: "Servers SHOULD return
tools in a deterministic order (i.e., the same ordering across requests
when the underlying set of tools has not changed)." Nothing in the suite
exercised it.

Add a tools-list-deterministic-order check to the existing tools-list
scenario. From the 2026-07-28 wire on, the scenario takes two more
consecutive tools/list snapshots and compares the three name sequences.
Same multiset of names, different order is WARNING (SHOULD). A set that
changed between probes is reported as untestable rather than as a
violation, because the spec scopes the SHOULD to an unchanged set. Fewer
than two tools on every probe is INFO. The check carries
source.introducedIn '2026-07-28', written as a literal since that revision
is published and dated, so runners drop it on earlier wires.

Prove it fails: examples/servers/typescript/tools-list-rotated-order.ts
advertises the same four tools and rotates the list by one position per
call; a negative.test.ts case asserts WARNING while tools-list itself
still passes. Unit tests cover the helper that builds the check.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjeC3sCvfHRpdEPz2x41B2

* tools-list: inline the deterministic-order check id and introducedIn literal

Check ids and introducedIn revisions are written inline everywhere else in
src/scenarios (tests pin the literal slug on purpose, since SDK baselines
reference it), so drop the two exported constants.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
2026-09-06 20:13:29 +00:00
..
checks Allow expected-failures entries to name a single check (#406) 2026-07-20 17:20:32 +02:00
connection fix: send HTTP DELETE to terminate server sessions after each scenario (#316) 2026-07-31 17:24:13 +01:00
mock-server Fix wire-validation attribution bugs and extend the schema erratum to 2025-06-18 (#421) 2026-07-27 20:32:17 +01:00
new-sep feat: add new-sep scaffolding command (#272) 2026-05-13 15:53:24 +01:00
runner fix(runner): bound server scenarios with a per-scenario timeout (#454) 2026-09-06 20:14:58 +01:00
scenarios feat(tools-list): check deterministic tools/list ordering on the 2026-07-28 wire (#492) 2026-09-06 20:13:29 +00:00
schemas fix(auth): add issuer to pre-registration context (#423) 2026-09-06 18:44:29 +01:00
sdk-runner feat: add ruby-sdk to the known SDKs registry (#457) 2026-09-06 18:43:45 +01:00
seps feat: add client-side JSON Schema 2020-12 preservation scenario (#335) 2026-07-31 17:24:05 +00:00
spec-types Validate all wire messages against the per-version spec JSON schema (#399) 2026-07-27 15:58:43 +01:00
tier-check fix(tier-check): strip package-name prefixes from monorepo release tags (#463) 2026-09-06 19:10:54 +00:00
traceability feat: add SEP conformance traceability manifest (#288) 2026-05-19 22:22:25 +01:00
validation Fix wire-validation attribution bugs and extend the schema erratum to 2025-06-18 (#421) 2026-07-27 20:32:17 +01:00
expected-failures.test.ts Allow expected-failures entries to name a single check (#406) 2026-07-20 17:20:32 +02:00
expected-failures.ts Allow expected-failures entries to name a single check (#406) 2026-07-20 17:20:32 +02:00
index.ts fix(runner): bound server scenarios with a per-scenario timeout (#454) 2026-09-06 20:14:58 +01:00
requirements.test.ts Freeze conformance requirements per spec revision, run each at its wire (#447) 2026-08-07 14:55:10 +01:00
requirements.ts Freeze conformance requirements per spec revision, run each at its wire (#447) 2026-08-07 14:55:10 +01:00
schemas.ts feat: add positive tests for the Authorization Code Grant (supersedes #267) (#364) 2026-06-24 16:13:55 +01:00
types.test.ts fix: update the draft wire version to 2026-07-28 (#331) 2026-06-08 16:47:17 +01:00
types.ts feat(sdk-runner): add rust-sdk + per-spec-version config overlays (#419) 2026-07-28 10:56:25 +01:00