conformance/src/scenarios
Repository files (latest commit first)
Filename Latest commit message Latest commit date
AKS 542381b7dc
feat(tools-list): check deterministic tools/list ordering on the 2026-07-28 wire (#492)
* feat(tools-list): check deterministic tools/list ordering on the 2026-07-28 wire

The 2026-07-28 revision adds to server/tools.mdx: "Servers SHOULD return
tools in a deterministic order (i.e., the same ordering across requests
when the underlying set of tools has not changed)." Nothing in the suite
exercised it.

Add a tools-list-deterministic-order check to the existing tools-list
scenario. From the 2026-07-28 wire on, the scenario takes two more
consecutive tools/list snapshots and compares the three name sequences.
Same multiset of names, different order is WARNING (SHOULD). A set that
changed between probes is reported as untestable rather than as a
violation, because the spec scopes the SHOULD to an unchanged set. Fewer
than two tools on every probe is INFO. The check carries
source.introducedIn '2026-07-28', written as a literal since that revision
is published and dated, so runners drop it on earlier wires.

Prove it fails: examples/servers/typescript/tools-list-rotated-order.ts
advertises the same four tools and rotates the list by one position per
call; a negative.test.ts case asserts WARNING while tools-list itself
still passes. Unit tests cover the helper that builds the check.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjeC3sCvfHRpdEPz2x41B2

* tools-list: inline the deterministic-order check id and introducedIn literal

Check ids and introducedIn revisions are written inline everywhere else in
src/scenarios (tests pin the literal slug on purpose, since SDK baselines
reference it), so drop the two exported constants.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
2026-09-06 20:13:29 +00:00
..
authorization-server feat: CIMD support check for authorization-server metadata (supersedes #235) (#365) 2026-06-24 16:53:00 +01:00
client feat(client/auth): check the resource parameter matches the PRM-published identifier (#488) 2026-09-06 20:41:25 +01:00
server feat(tools-list): check deterministic tools/list ordering on the 2026-07-28 wire (#492) 2026-09-06 20:13:29 +00:00
index.ts feat: add client-side JSON Schema 2020-12 preservation scenario (#335) 2026-07-31 17:24:05 +00:00
request-logger.ts [auth] Implement CIMD conformance test (#37) 2025-11-20 21:00:13 +00:00
spec-version.test.ts feat: CIMD support check for authorization-server metadata (supersedes #235) (#365) 2026-06-24 16:53:00 +01:00
untestable.test.ts fix: fail checks whose prerequisite is missing instead of skipping them (#372) 2026-06-30 11:42:01 +01:00
untestable.ts fix: fail checks whose prerequisite is missing instead of skipping them (#372) 2026-06-30 11:42:01 +01:00