modelcontextprotocol/conformance
Watch
1
0
Fork
You've already forked conformance
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat: sdk subcommand to run local conformance against any SDK ref (#277)

Browse source 
* feat: add sdk subcommand to run conformance against any SDK ref (#250)

* Revise sdk runner: explicit --mode, KNOWN_SDKS-only config, v1/v2 entries

Addresses review feedback on the sdk subcommand:

- Require --mode (client|server) and remove "both". Each invocation now
  tests exactly one side with its own exit code; the old default ran
  client then server but combined exit codes with ||=, which skipped the
  server side entirely whenever the client run failed.
- Resolve build/run config from KNOWN_SDKS + CLI flags only; drop the
  conformance.config.yaml loader (no SDK ships one yet). The Zod schema
  stays as the type for the built-in entries.
- Split the typescript entry by major version: typescript-sdk (v2/main,
  pnpm install + build:all, expected-failures.yaml) and typescript-sdk-v1
  (v1.x, npm ci + build, conformance-baseline.yml). An entry may set
  `repo` (real clone target for an alias) and `defaultRef` (branch used
  when no @ref is given). parseSdkSpec now leaves ref undefined when
  omitted so defaultRef can apply; a trailing @ is treated as no ref.
- Key the clone cache by ref (<repo>/<ref>) so different refs of the same
  repo no longer share one checkout.
- Bound the server readiness probe with a per-request AbortSignal timeout
  so a server that accepts the socket but never responds can't hang past
  the overall deadline.

* fix(sdk-runner): resolve -o to absolute path; add --expected-failures override; replaceAll for safeName
This commit is contained in:
Paul Carleton 2026-05-19 22:08:06 +01:00 • committed by GitHub
commit cfcecdba2c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 643 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

1
AGENTS.md
View file

@ -52,6 +52,7 @@ Keep scenarios separate when they're genuinely independent features or when they
- **Same `id` for SUCCESS and FAIL.** A check should use one slug and flip `status` + `errorMessage`, not branch into `foo-success` vs `foo-failure` slugs.
- **Optimize for Ctrl+F on the slug.** Repetitive check blocks are fine — easier to find the failing one than to unwind a clever helper.
- Reuse `ConformanceCheck` and other types from `src/types.ts` rather than defining parallel shapes.
- **Don't reimplement the runner.** New subcommands that need to "select scenarios → run them → print summary → compute exit code" must go through the existing `client` / `server` commands (subprocess via `process.execPath` like `tier-check` and `sdk` do) or call shared helpers — never a parallel suite-map / summary loop.
- Include `specReferences` pointing to the relevant spec section.
- **Severity follows the spec keyword:** MUST / MUST NOT → `FAILURE`; SHOULD / SHOULD NOT → `WARNING`. (CI treats WARNING as a failure, so Tier-1 SDKs still need to satisfy SHOULDs — see #245.)