modelcontextprotocol/conformance
Watch
1
0
Fork
You've already forked conformance
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(sdk-runner): add rust-sdk + per-spec-version config overlays (#419)

Browse source 
* feat(sdk-runner): add rust-sdk to KNOWN_SDKS

The mcp-conformance package's fixture bins follow the harness contract
directly (URL argv + MCP_CONFORMANCE_SCENARIO env), and the server
selects the SEP-2575 stateless lifecycle via STATELESS=1. Requires the
repo's pinned Rust toolchain (rustup installs it on demand).

* feat(sdk-runner): per-spec-version config overlays (specOverrides)

Several KNOWN_SDKS entries needed a different invocation when targeting
2026-07-28 — go-sdk's -stateless flag, csharp-sdk's /stateless URL,
rust-sdk's STATELESS=1 env, python-sdk's per-revision baseline — and each
carried a comment telling the caller which flag to paste. Encode those
variants in the entry instead: specOverrides maps a canonical spec
version to a partial config that is merged over the base entry when a
run targets that version (--spec-version, after 'draft' alias
resolution, or the entry's own specVersion default). Explicit CLI flags
still win.

Keys are validated against the known spec versions at schema level so a
'draft' key or a typo'd date fails loudly instead of silently never
matching.

* style: prettier fixes in sdk-runner tests

* refactor(sdk-runner): reuse isSpecVersion for specOverrides key validation

Reuses the canonical spec-version list from #421 instead of keeping a
separate copy; SPEC_VERSION_TIMELINE is now exported so the validation
message can enumerate the valid keys.
This commit is contained in:
Paul Carleton 2026-07-28 10:56:25 +01:00 • committed by GitHub
commit 49103de6ed
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 241 additions and 28 deletions
Download patch file Download diff file Expand all files Collapse all files

23
README.md
View file

@ -299,14 +299,25 @@ npm start -- sdk typescript-sdk --mode client --spec-version draft
Build/run commands for each official SDK are looked up by name from [`src/sdk-runner/known-sdks.ts`](src/sdk-runner/known-sdks.ts) — no config file is required in the SDK repo. Resolution order is **CLI flag > built-in entry**, so any field can be overridden on the command line for refs that diverge from the built-in.
An SDK can have more than one entry when its layout differs across major versions — e.g. `typescript-sdk` (v2, the `main` monorepo) and `typescript-sdk-v1` (the published npm v1.x line). An entry may set `defaultRef` (the branch used when you don't pass `@<ref>`) and `repo` (the real clone target when the entry name is an alias). Overriding built-in fields for a one-off run:
An SDK can have more than one entry when its layout differs across major versions — e.g. `typescript-sdk` (v2, the `main` monorepo) and `typescript-sdk-v1` (the published npm v1.x line). An entry may set `defaultRef` (the branch used when you don't pass `@<ref>`) and `repo` (the real clone target when the entry name is an alias).
When the right invocation depends on the spec version being targeted, the entry carries it in `specOverrides` instead of a comment to copy from. The matching entry is merged over the base config when you pass `--spec-version` (or when the entry's own `specVersion` default applies), so version-specific runs need no extra flags:
```bash
# e.g. a 2026-07-28 run against go-sdk's server — the built-in server command
# pins -stateless=false (what the dated-spec suites need), so override it to
# get the SEP-2575 stateless lifecycle:
npm start -- sdk go-sdk --mode server --suite all --spec-version 2026-07-28 \
--server-cmd './.conformance-server -http=localhost:3000'
# go-sdk's server pins -stateless=false for the dated-spec suites; its
# specOverrides['2026-07-28'] entry swaps in the stateless invocation, so
# this is the whole command:
npm start -- sdk go-sdk --mode server --suite all --spec-version 2026-07-28
# same for csharp-sdk (stateless URL), rust-sdk (STATELESS=1 env), and
# python-sdk (per-revision expected-failures baseline)
```
Explicit CLI flags still beat everything, config included — overriding a field for a one-off run:
```bash
npm start -- sdk go-sdk@my-fork-branch --mode server \
--build-cmd 'go build -o ./.conformance-server ./experimental/server'
```
To add a new SDK to the matrix, add an entry to `KNOWN_SDKS`.