feat: add --expected-failures baseline and composite GitHub Action (#113)

Add support for SDKs that don't yet pass all conformance tests to still
run conformance in CI without failing, by specifying a YAML baseline file
of known/expected failures.

CLI changes:
- Add --expected-failures <path> flag to both server and client commands
- Exit 0 if only expected failures occur
- Exit 1 if unexpected failures occur (regression)
- Exit 1 if an expected failure now passes (stale baseline, must update)

GitHub Action (action.yml):
- Composite action that builds from source and runs conformance tests
- SDKs reference it as: uses: modelcontextprotocol/conformance@<ref>
- Inputs: mode, url, command, expected-failures, suite, scenario, etc.

New files:
- src/expected-failures.ts: YAML loading, baseline evaluation, reporting
- src/expected-failures.test.ts: 16 unit tests
- action.yml: composite GitHub Action

Closes #99
This commit is contained in:
Max Isbey 2026-01-22 10:41:04 +00:00 • committed by GitHub
commit 83ff334e0e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

16
package-lock.json generated
View file

@ -14,6 +14,7 @@
"eventsource-parser": "^3.0.6",
"express": "^5.1.0",
"jose": "^6.1.2",
"yaml": "^2.8.2",
"zod": "^3.25.76"
},
"bin": {
@ -5645,6 +5646,21 @@
"integrity": "sha1-tSQ9jz7BqjXxNkYFvA0QNuMKtp8=",
"license": "ISC"
},
"node_modules/yaml": {
"version": "2.8.2",
"resolved": "https://registry.npmjs.org/yaml/-/yaml-2.8.2.tgz",
"integrity": "sha512-mplynKqc1C2hTVYxd0PU2xQAc22TI1vShAYGksCCfxbn/dFwnHTNi1bvYsBTkhdUNtGIf5xNOg938rrSSYvS9A==",
"license": "ISC",
"bin": {
"yaml": "bin.mjs"
},
"engines": {
"node": ">= 14.6"
},
"funding": {
"url": "https://github.com/sponsors/eemeli"
}
},
"node_modules/yocto-queue": {
"version": "0.1.0",
"resolved": "https://registry.npmjs.org/yocto-queue/-/yocto-queue-0.1.0.tgz",