'All scopes required for the current operation' has no harness-observable
ground truth: the WWW-Authenticate challenge is the only place the server
declares what an operation requires, so there is nothing independent to
compare it against. Detecting the negative (incremental challenging) would
need server-side auth scenarios that do not exist yet and would exercise the
example app's scope configuration rather than SDK behavior, and the spec
explicitly permits dynamic per-request scope determination, so a second
challenge is not conclusively non-conformant.
The client-side half of SEP-2350 (scope union on re-authorization) remains
tested by auth/scope-step-up.
* chore: refresh SEP traceability manifest (typescript-sdk@main)
Regenerated from a client+server suite run against typescript-sdk@5fc42e9be115
following the recipe in .github/workflows/traceability.yml.
New entries since the last refresh (typescript-sdk@22595b96):
- SEP-2322 (MRTR, #188): 17 tested, 0 untested, 16 excluded, 3 untracked
- SEP-2549 (TTL for list results, #275): 7 tested, 0 untested, 13 excluded
- SEP-2260: 12 excluded rows, no checks
- SEP-2207: yaml rows added since the last refresh now appear
(1 tested, 1 untested: sep-2207-server-no-offline-access)
No previously-tested requirement regressed.
* Exclude sep-2207 server offline_access guidance until RS auth scenarios exist
sep-2207-server-no-offline-access was declared in the yaml but no scenario
emits it, so it surfaced as the only untested requirement in the refreshed
manifest. The check needs to probe the SDK server's Protected Resource
Metadata scopes_supported and WWW-Authenticate challenge scope, and the
server suite does not yet exercise the SDK server as an OAuth protected
resource at all.
Mark the requirement excluded with a pointer to #116 (server-side
authorization baseline) rather than leaving it as a permanently-untested
row; revisit when server-side authorization scenarios land.
* chore: additional server conformance tests for SEP-2575
* fix(sep-2575): use the spec's notifications filter object in subscriptions/listen
The scenario sent params.subscriptions: [{type: 'tools/list-changed'}], a
shape that does not exist in the SEP. SubscriptionsListenRequestParams
requires params.notifications: {toolsListChanged?, promptsListChanged?, ...}
(see schema.ts SubscriptionFilter and the canonical
SubscriptionsListenRequest example). A compliant server would either reject
the old request as invalid params or treat it as subscribing to nothing,
failing the ack/list-changed checks.
Also makes the everything-server's acknowledgment echo the agreed
notifications filter, as the SubscriptionsAcknowledgedNotification schema
requires.
* fix(sep-2575): match the spec's notifications/*/list_changed method names
The list-changed checks and the everything-server used
notifications/tools/list-changed (hyphen); the spec's method is
notifications/tools/list_changed (underscore) per ToolListChangedNotification
/ PromptListChangedNotification in schema.ts. With the hyphen form the
notification-filter check cannot catch a real leak, and the list-changed
checks fail a server that emits the correct name.
Also drops the f.params.toolsListChanged === true fallback match: that field
is the subscriptions/listen request filter, not a notification parameter, so
it never appears on a compliant server's notification.
* fix(sep-2575): keep collected frames when the listen stream read times out
listenToStream aborts the fetch after timeoutMs. If the abort fired while
blocked in reader.read(), the rejection propagated to the outer catch and the
helper returned [] — discarding every frame already received. A compliant
server holds a subscriptions/listen stream open indefinitely, so the timeout
is the *normal* way these reads end; previously every subscription check
would report a false 'no frames received' FAILURE against such a server.
The current everything-server only avoided this by res.end()ing immediately
after the ack.
* fix(sep-2575): open the listen stream before triggering list mutations
The notification-filter and list-changed checks fired the mutation trigger
*before* opening the subscriptions/listen stream. A compliant server only
delivers list-changed notifications to streams that are open at the time of
the change, so the SHOULD checks would fail any server that does not replay
past changes to new subscribers, and the filter check could never observe a
real leak.
listenToStream now takes an onFirstFrame hook; the three trigger-based
checks open the stream, wait for the acknowledgment, then fire the mutation
on a separate connection and keep reading.
The everything-server now keeps subscriptions/listen streams open, registers
them with their filters, and fans the list_changed notification out to open
streams when a trigger tool runs - instead of unconditionally emitting a
list-changed frame on every stream open (which itself violated the
notification-filter requirement in spirit: it announced a change when
nothing had changed).
* fix(sep-2575): check the subscription id on every listen-stream notification
server-tags-subscription-id only inspected the acknowledgment frame, so a
server that tags the ack but omits io.modelcontextprotocol/subscriptionId
from subsequent notifications passed. The check now triggers a tool-list
change once the stream is acknowledged and asserts every notification frame
on the stream carries the id in params._meta.
Also drops the f.body.method / f.params.method fallbacks when resolving a
frame's method - those shapes don't exist in JSON-RPC and only masked
malformed frames.
* fix(sep-2575): report WARNING, not FAILURE, for the SHOULD-level list-changed checks
The spec text behind server-sends-{tools,prompts}-list-changed-on-subscription
is a SHOULD ('servers that declared the listChanged capability SHOULD send a
notification...'). Severity follows the spec keyword: MUST -> FAILURE,
SHOULD -> WARNING. runCheck now accepts a warning flag and the two
list-changed checks use it on their failure paths.
* fix(sep-2575): skip stream checks when the diagnostic tool is not exposed
http-server-no-independent-requests-on-stream and
server-no-log-without-loglevel reported SUCCESS when the server rejected the
tools/call outright (e.g. test_streaming_elicitation / test_logging_tool do
not exist): a single error frame contains no independent request and no log
notification, so the check passed without exercising anything. The sibling
list-changed checks already SKIP when their trigger tool is missing; these
now do the same.
* chore(sep-2575): exclude the four unobservable server requirements from traceability
Marks the section-C rows of #296 as excluded rather than untested:
no-prior-context, no-connection-reuse-required, disconnect-is-cancel and
stops-on-cancel describe internal server state that a black-box harness
cannot observe on the wire. With these excluded and the part-B checks landed,
every testable server-side SEP-2575 requirement is covered; the remaining
untested rows are all client-side and blocked on a SEP-2575-aware reference
client.
* fix(sep-2575): emit the request-metadata checks even when the client never connects
The reference SDK conformance client has no handler registered for the
request-metadata scenario, so it exits without sending a request, the
scenario emits zero checks, the suite reports "0 passed, 0 failed", and the
traceability manifest records all seven client-side requirements as
untested.
Declare the scenario check IDs up front and backfill any that were never
emitted as FAILURE from getChecks(): the emitted ID set is now the same for
every client, and a client that never connects fails loudly. A check that
is legitimately not applicable opts out by being emitted as SKIPPED
explicitly (version-header-matches-meta now does this when either version
field is absent). The positive-path test pins the declared list against the
reference client in both directions.
Also refresh the traceability manifest from a run against
typescript-sdk@22595b96: the seven client rows flip to tested and the
manifest catches up with the server-side scenarios from #297.
---------
Co-authored-by: Yuan Teoh <yuanteoh@google.com>
* chore: additional server conformance tests for SEP-2575
* fix(sep-2575): use the spec's notifications filter object in subscriptions/listen
The scenario sent params.subscriptions: [{type: 'tools/list-changed'}], a
shape that does not exist in the SEP. SubscriptionsListenRequestParams
requires params.notifications: {toolsListChanged?, promptsListChanged?, ...}
(see schema.ts SubscriptionFilter and the canonical
SubscriptionsListenRequest example). A compliant server would either reject
the old request as invalid params or treat it as subscribing to nothing,
failing the ack/list-changed checks.
Also makes the everything-server's acknowledgment echo the agreed
notifications filter, as the SubscriptionsAcknowledgedNotification schema
requires.
* fix(sep-2575): match the spec's notifications/*/list_changed method names
The list-changed checks and the everything-server used
notifications/tools/list-changed (hyphen); the spec's method is
notifications/tools/list_changed (underscore) per ToolListChangedNotification
/ PromptListChangedNotification in schema.ts. With the hyphen form the
notification-filter check cannot catch a real leak, and the list-changed
checks fail a server that emits the correct name.
Also drops the f.params.toolsListChanged === true fallback match: that field
is the subscriptions/listen request filter, not a notification parameter, so
it never appears on a compliant server's notification.
* fix(sep-2575): keep collected frames when the listen stream read times out
listenToStream aborts the fetch after timeoutMs. If the abort fired while
blocked in reader.read(), the rejection propagated to the outer catch and the
helper returned [] — discarding every frame already received. A compliant
server holds a subscriptions/listen stream open indefinitely, so the timeout
is the *normal* way these reads end; previously every subscription check
would report a false 'no frames received' FAILURE against such a server.
The current everything-server only avoided this by res.end()ing immediately
after the ack.
* fix(sep-2575): open the listen stream before triggering list mutations
The notification-filter and list-changed checks fired the mutation trigger
*before* opening the subscriptions/listen stream. A compliant server only
delivers list-changed notifications to streams that are open at the time of
the change, so the SHOULD checks would fail any server that does not replay
past changes to new subscribers, and the filter check could never observe a
real leak.
listenToStream now takes an onFirstFrame hook; the three trigger-based
checks open the stream, wait for the acknowledgment, then fire the mutation
on a separate connection and keep reading.
The everything-server now keeps subscriptions/listen streams open, registers
them with their filters, and fans the list_changed notification out to open
streams when a trigger tool runs - instead of unconditionally emitting a
list-changed frame on every stream open (which itself violated the
notification-filter requirement in spirit: it announced a change when
nothing had changed).
* fix(sep-2575): check the subscription id on every listen-stream notification
server-tags-subscription-id only inspected the acknowledgment frame, so a
server that tags the ack but omits io.modelcontextprotocol/subscriptionId
from subsequent notifications passed. The check now triggers a tool-list
change once the stream is acknowledged and asserts every notification frame
on the stream carries the id in params._meta.
Also drops the f.body.method / f.params.method fallbacks when resolving a
frame's method - those shapes don't exist in JSON-RPC and only masked
malformed frames.
* fix(sep-2575): report WARNING, not FAILURE, for the SHOULD-level list-changed checks
The spec text behind server-sends-{tools,prompts}-list-changed-on-subscription
is a SHOULD ('servers that declared the listChanged capability SHOULD send a
notification...'). Severity follows the spec keyword: MUST -> FAILURE,
SHOULD -> WARNING. runCheck now accepts a warning flag and the two
list-changed checks use it on their failure paths.
* fix(sep-2575): skip stream checks when the diagnostic tool is not exposed
http-server-no-independent-requests-on-stream and
server-no-log-without-loglevel reported SUCCESS when the server rejected the
tools/call outright (e.g. test_streaming_elicitation / test_logging_tool do
not exist): a single error frame contains no independent request and no log
notification, so the check passed without exercising anything. The sibling
list-changed checks already SKIP when their trigger tool is missing; these
now do the same.
* chore(sep-2575): exclude the four unobservable server requirements from traceability
Marks the section-C rows of #296 as excluded rather than untested:
no-prior-context, no-connection-reuse-required, disconnect-is-cancel and
stops-on-cancel describe internal server state that a black-box harness
cannot observe on the wire. With these excluded and the part-B checks landed,
every testable server-side SEP-2575 requirement is covered; the remaining
untested rows are all client-side and blocked on a SEP-2575-aware reference
client.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* Add SEP-2350 scope-accumulation check to auth/scope-step-up
SEP-2350 clarifies that on step-up re-authorization, clients SHOULD
compute the union of previously-granted and newly-challenged scopes so
they don't lose permissions for other operations.
- Traceability yaml: 1 client check (two spec sentences merged), 1 server
check tracked for a future server-side scenario, 1 excluded reword
- ScopeStepUpAuthScenario now challenges with only the missing scope so
union accumulation is observable on the second authorization request
- New check sep-2350-scope-union-on-reauth (WARNING when the previously
granted scope is dropped)
- everything-client withOAuthRetry now accumulates prior token scope into
the re-auth scope (passing example)
- New auth-test-echo-scope negative client + vitest case
* feat: add client conformance tests for SEP-2575 (#270)
* feat: merge latest request-metadata scenario stacked branch
* test: implement optional capability conditional checks and simulated negotiation retry
* fix(sep-2243): collapse check IDs to one-per-requirement (#287)
The sep-2243 scenario check IDs drifted from the requirement-traceability
yaml in #259 — the code emitted one ID per test case while the yaml
declares one per normative requirement, so the IDs no longer matched.
Rename the emitted IDs so a check ID maps to a single MUST/SHOULD and is
emitted once per case (the per-case detail moves to name/description),
matching the repo's existing 'same id, vary status/message' convention:
- mcp-method-header-* / mcp-name-header-* -> client-includes-standard-headers
- reject-invalid-tool-* / keep-valid-tool -> client-reject-invalid-tool
- server-accepts-{lower,upper}case-name -> header-name-case-insensitive
- server reject status checks (mismatch/missing/case x method/name)
-> server-reject-invalid-headers
- their error-code variants -> server-reject-error-code
Merge the yaml's server-reject-mismatch + server-reject-status into one
server-reject-invalid-headers MUST (HTTP 400 on a header-validation
failure); keep server-reject-error-code as the SHOULD. Base64/custom-header
rejection checks keep their own param-validation IDs (out of scope here).
Gates and the whitespace-acceptance check keep their scenario-matching
names. No behavior change — only check IDs, descriptions, and the yaml.
* feat: `sdk` subcommand to run local conformance against any SDK ref (#277)
* feat: add sdk subcommand to run conformance against any SDK ref (#250)
* Revise sdk runner: explicit --mode, KNOWN_SDKS-only config, v1/v2 entries
Addresses review feedback on the sdk subcommand:
- Require --mode (client|server) and remove "both". Each invocation now
tests exactly one side with its own exit code; the old default ran
client then server but combined exit codes with ||=, which skipped the
server side entirely whenever the client run failed.
- Resolve build/run config from KNOWN_SDKS + CLI flags only; drop the
conformance.config.yaml loader (no SDK ships one yet). The Zod schema
stays as the type for the built-in entries.
- Split the typescript entry by major version: typescript-sdk (v2/main,
pnpm install + build:all, expected-failures.yaml) and typescript-sdk-v1
(v1.x, npm ci + build, conformance-baseline.yml). An entry may set
`repo` (real clone target for an alias) and `defaultRef` (branch used
when no @ref is given). parseSdkSpec now leaves ref undefined when
omitted so defaultRef can apply; a trailing @ is treated as no ref.
- Key the clone cache by ref (<repo>/<ref>) so different refs of the same
repo no longer share one checkout.
- Bound the server readiness probe with a per-request AbortSignal timeout
so a server that accepts the socket but never responds can't hang past
the overall deadline.
* fix(sdk-runner): resolve -o to absolute path; add --expected-failures override; replaceAll for safeName
* feat: add SEP conformance traceability manifest (#288)
squashed; see PR #288 body
* feat: add server conformance tests for SEP-2575 (#271)
* feat: add server conformance tests for SEP-2575
* refactor and add new checks
* Fix stateless scenario source field and report SKIPPED for inapplicable capability checks
- Add the required `source` field (DRAFT) so the scenario typechecks and is
selectable via --spec-version. Without it the class did not satisfy the
ClientScenario interface and s.source was dereferenced by the spec-version
filter at list time.
- Teach runCheck about a SKIPPED status and use it for the two client-capability
checks when the server does not return -32003, instead of reporting a green
PASS for a requirement that was never exercised.
* Remove unimplemented placeholder checks from stateless server scenario
The subscriptions/listen, statelessness-invariant, list-changed and
disconnect-is-cancel checks emitted SUCCESS without probing the server,
which inflated coverage in the traceability report. Drop them until they
have real assertions; the corresponding rows remain declared in
src/seps/sep-2575.yaml so traceability shows them as not-yet-covered.
Also fix the checkErrorId helper to push under the
sep-2575-http-server-error-jsonrpc-id slug so its failure path actually
short-circuits the aggregate SUCCESS check at the end of the scenario.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* SEP-2352: authorization-server migration scenario (#286)
* Add SEP-2352 authorization-server migration scenario
SEP-2352 requires that client credentials are bound to the issuing
authorization server: when PRM authorization_servers changes to a new
issuer, clients MUST re-register and MUST NOT reuse the previous AS's
client credentials.
- Traceability yaml: 3 checks, 3 excluded (internal state / UI)
- New auth/authorization-server-migration scenario (draft suite): two
auth servers; PRM flips from AS1 to AS2 after the first authenticated
request; AS2 asserts it received a fresh /register and never saw AS1's
client_id at /authorize or /token
- ConformanceOAuthProvider gains invalidateCredentials and bindIssuer so
the everything-client can key credentials by issuer (passing example)
- everything-client adds an issuer-aware handler for this scenario that
re-reads PRM on each 401 and rebinds before re-authorizing
- auth-test-reuse-credentials negative client + vitest case
* Drop application_type from DCR metadata (not in SDK OAuthClientMetadata type)
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* ci(traceability): enable corepack so the SDK build can use pnpm (#289)
The refresh run failed with 'pnpm: not found' — the reference SDK's
build command (typescript-sdk: pnpm install && pnpm run build:all) needs
pnpm on PATH. Add 'corepack enable' to the run job.
* ci(traceability): tolerate SDK conformance failures in the run step (#290)
The `sdk` command exits non-zero when the SDK has conformance failures
not in its baseline. The traceability manifest only needs the emitted
check IDs (written regardless of pass/fail), so a failing SDK must not
fail the refresh. Add `|| true`; the existing 'no results produced'
guard catches a genuinely broken run.
* chore: refresh SEP traceability manifest (typescript-sdk@main) (#291)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Make scopesSupported disjoint from challenge scopes so the SEP-2350 union check can't be satisfied by scopes_supported ∪ challenge
A client that unions scopes_supported with the challenge — instead of its
prior grant with the challenge — would have requested mcp:basic mcp:write
and falsely passed sep-2350-scope-union-on-reauth, since scopesSupported
happened to coincide with the previously-granted scope. Setting it to a
disjoint value makes the check actually require the prior token's scope.
Spec backing: clients MUST NOT assume any set relationship between the
challenged scope set and scopes_supported, so a disjoint advertisement is
realistic.
* SEP-837: application_type check in DCR registration (#284)
* Add SEP-837 application_type check to DCR registration
SEP-837 requires MCP clients to specify an appropriate application_type
during Dynamic Client Registration so OIDC authorization servers can
apply the correct redirect-URI constraints.
- Traceability yaml: 1 check (presence + valid value), 4 excluded
(class-specific SHOULDs unobservable; UI/robustness)
- Check added in the shared createAuthServer DCR handler so it fires in
every auth scenario that performs DCR (no new scenario)
- withOAuthRetry now sets application_type: native (passing example;
the conformance example clients are CLI tools)
- New auth-test-no-application-type negative client + vitest case
* Widen ConformanceOAuthProvider metadata type for application_type
Fixes CI typecheck (TS2353 at withOAuthRetry.ts:76): the SDK's
OAuthClientMetadataSchema doesn't include application_type yet.
registerClient() spreads clientMetadata verbatim into the /register
POST body, so a local type intersection is sufficient to get the
field on the wire without an SDK release.
* Set application_type in runAuthMigrationClient
The SEP-2352 authorization-server-migration handler constructs its own
ConformanceOAuthProvider (not via withOAuthRetry), so it was missing
application_type after rebasing onto main. The scenario does DCR twice
(old AS, new AS) and the SEP-837 check fires on each.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* feat: add conformance tests for iss parameter (SEP-2468) (#220)
* feat: add conformance tests for iss parameter (SEP-2468)
Adds 5 draft conformance scenarios testing RFC 9207 issuer parameter
validation in OAuth authorization responses:
- auth/iss-supported: server advertises support and sends correct iss
- auth/iss-not-advertised: server omits iss parameter entirely
- auth/iss-supported-missing: client must reject missing iss when required
- auth/iss-wrong-issuer: client must reject mismatched iss value
- auth/iss-unexpected: client must reject iss when not advertised
Also adds auth-test-iss-validation.ts, a reference client that correctly
validates iss per RFC 9207, and negative tests confirming the standard
client fails all three rejection scenarios.
TODO: Update RFC_9207_ISS_PARAMETER spec reference once SEP-2468
(modelcontextprotocol/modelcontextprotocol#2468) is merged.
* update scenarios
* fix: createAuthServer iss option type/guard and NotAdvertised scenario duplication
The doc comments said 'Default: not included' but the destructure defaulted
to true/'correct', and the `!== undefined` guard at L155 was unreachable —
so there was no way to omit the metadata field, and IssParameterNotAdvertised
silently advertised support (a duplicate of IssParameterSupported).
Kept the on-by-default behavior (mock AS models a well-behaved server) but
made issParameterSupported `boolean | null` so callers pass null to omit,
matching the codeChallengeMethodsSupported pattern. Doc comments now match.
Scenarios that need omission pass null/'omit' explicitly.
* fix: rejection scenarios silently pass when client never reaches auth endpoint
correctlyRejected = !tokenRequestMade reports SUCCESS if the client errors
out before hitting /authorize. Gate on authReached so a setup failure shows
as FAILURE with authReached:false in details.
* fix: iss-unexpected scenario contradicts SEP-2468 spec table row 3
The spec table says: supported=false/absent + iss present -> *Compare* to
the recorded issuer (not reject). The scenario sent a *correct* iss and
FAILed compliant clients for proceeding after a successful comparison.
Now sends a mismatched iss so the comparison fails and rejection is the
spec-required outcome. Reference client updated to compare-when-present
instead of throw-on-presence.
* refactor: replace harness-config checks with client-proceeded checks
iss-advertised-in-metadata / iss-sent-in-redirect (and the not-* variants)
fired in onAuthorizationRequest before the redirect happened, asserting only
that the harness was configured correctly — a client that ignores iss passes
identically. Replaced with one check per scenario keyed on tokenRequestMade,
which observes that the client actually proceeded through the iss path.
* refactor: rename check IDs to sep-2468-* and align with spec table rows
One ID per spec table row; auth/iss-supported and auth/iss-wrong-issuer
both emit sep-2468-client-compare-iss-supported (same comparison, opposite
input) per the same-slug-for-SUCCESS-and-FAIL convention.
* feat: add sep-2468.yaml requirement traceability
8 check rows (4 client table-row checks, 1 metadata-issuer, 2 AS-side,
1 no-normalization), 1 excluded (error-display is UI-facing). The
record-issuer MUST is merged into the compare-iss-supported row text
since it has no independent wire observation.
* fix: migrate iss scenarios specVersions->source (post-#265)
Replaces `specVersions: ['draft']` with `source: { introducedIn: DRAFT_PROTOCOL_VERSION }`
in the 5 iss-parameter scenarios.
This commit typechecks once the stack is rebased onto main >= #265 (the
ScenarioSource migration). Adding it now so the rebase is mechanical.
* fix: include application_type in iss-validation example DCR (post-#284)
The SEP-837 application_type check now runs in every auth scenario; the
hand-rolled DCR in auth-test-iss-validation.ts was omitting the field.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
---------
Co-authored-by: Anubhav Dhawan <anubhavdhawan@google.com>
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Max Gerber <89937743+max-stytch@users.noreply.github.com>
* feat: add conformance tests for iss parameter (SEP-2468)
Adds 5 draft conformance scenarios testing RFC 9207 issuer parameter
validation in OAuth authorization responses:
- auth/iss-supported: server advertises support and sends correct iss
- auth/iss-not-advertised: server omits iss parameter entirely
- auth/iss-supported-missing: client must reject missing iss when required
- auth/iss-wrong-issuer: client must reject mismatched iss value
- auth/iss-unexpected: client must reject iss when not advertised
Also adds auth-test-iss-validation.ts, a reference client that correctly
validates iss per RFC 9207, and negative tests confirming the standard
client fails all three rejection scenarios.
TODO: Update RFC_9207_ISS_PARAMETER spec reference once SEP-2468
(modelcontextprotocol/modelcontextprotocol#2468) is merged.
* update scenarios
* fix: createAuthServer iss option type/guard and NotAdvertised scenario duplication
The doc comments said 'Default: not included' but the destructure defaulted
to true/'correct', and the `!== undefined` guard at L155 was unreachable —
so there was no way to omit the metadata field, and IssParameterNotAdvertised
silently advertised support (a duplicate of IssParameterSupported).
Kept the on-by-default behavior (mock AS models a well-behaved server) but
made issParameterSupported `boolean | null` so callers pass null to omit,
matching the codeChallengeMethodsSupported pattern. Doc comments now match.
Scenarios that need omission pass null/'omit' explicitly.
* fix: rejection scenarios silently pass when client never reaches auth endpoint
correctlyRejected = !tokenRequestMade reports SUCCESS if the client errors
out before hitting /authorize. Gate on authReached so a setup failure shows
as FAILURE with authReached:false in details.
* fix: iss-unexpected scenario contradicts SEP-2468 spec table row 3
The spec table says: supported=false/absent + iss present -> *Compare* to
the recorded issuer (not reject). The scenario sent a *correct* iss and
FAILed compliant clients for proceeding after a successful comparison.
Now sends a mismatched iss so the comparison fails and rejection is the
spec-required outcome. Reference client updated to compare-when-present
instead of throw-on-presence.
* refactor: replace harness-config checks with client-proceeded checks
iss-advertised-in-metadata / iss-sent-in-redirect (and the not-* variants)
fired in onAuthorizationRequest before the redirect happened, asserting only
that the harness was configured correctly — a client that ignores iss passes
identically. Replaced with one check per scenario keyed on tokenRequestMade,
which observes that the client actually proceeded through the iss path.
* refactor: rename check IDs to sep-2468-* and align with spec table rows
One ID per spec table row; auth/iss-supported and auth/iss-wrong-issuer
both emit sep-2468-client-compare-iss-supported (same comparison, opposite
input) per the same-slug-for-SUCCESS-and-FAIL convention.
* feat: add sep-2468.yaml requirement traceability
8 check rows (4 client table-row checks, 1 metadata-issuer, 2 AS-side,
1 no-normalization), 1 excluded (error-display is UI-facing). The
record-issuer MUST is merged into the compare-iss-supported row text
since it has no independent wire observation.
* fix: migrate iss scenarios specVersions->source (post-#265)
Replaces `specVersions: ['draft']` with `source: { introducedIn: DRAFT_PROTOCOL_VERSION }`
in the 5 iss-parameter scenarios.
This commit typechecks once the stack is rebased onto main >= #265 (the
ScenarioSource migration). Adding it now so the rebase is mechanical.
* fix: include application_type in iss-validation example DCR (post-#284)
The SEP-837 application_type check now runs in every auth scenario; the
hand-rolled DCR in auth-test-iss-validation.ts was omitting the field.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* Add SEP-837 application_type check to DCR registration
SEP-837 requires MCP clients to specify an appropriate application_type
during Dynamic Client Registration so OIDC authorization servers can
apply the correct redirect-URI constraints.
- Traceability yaml: 1 check (presence + valid value), 4 excluded
(class-specific SHOULDs unobservable; UI/robustness)
- Check added in the shared createAuthServer DCR handler so it fires in
every auth scenario that performs DCR (no new scenario)
- withOAuthRetry now sets application_type: native (passing example;
the conformance example clients are CLI tools)
- New auth-test-no-application-type negative client + vitest case
* Widen ConformanceOAuthProvider metadata type for application_type
Fixes CI typecheck (TS2353 at withOAuthRetry.ts:76): the SDK's
OAuthClientMetadataSchema doesn't include application_type yet.
registerClient() spreads clientMetadata verbatim into the /register
POST body, so a local type intersection is sufficient to get the
field on the wire without an SDK release.
* Set application_type in runAuthMigrationClient
The SEP-2352 authorization-server-migration handler constructs its own
ConformanceOAuthProvider (not via withOAuthRetry), so it was missing
application_type after rebasing onto main. The scenario does DCR twice
(old AS, new AS) and the SEP-837 check fires on each.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
The `sdk` command exits non-zero when the SDK has conformance failures
not in its baseline. The traceability manifest only needs the emitted
check IDs (written regardless of pass/fail), so a failing SDK must not
fail the refresh. Add `|| true`; the existing 'no results produced'
guard catches a genuinely broken run.
The refresh run failed with 'pnpm: not found' — the reference SDK's
build command (typescript-sdk: pnpm install && pnpm run build:all) needs
pnpm on PATH. Add 'corepack enable' to the run job.
* Add SEP-2352 authorization-server migration scenario
SEP-2352 requires that client credentials are bound to the issuing
authorization server: when PRM authorization_servers changes to a new
issuer, clients MUST re-register and MUST NOT reuse the previous AS's
client credentials.
- Traceability yaml: 3 checks, 3 excluded (internal state / UI)
- New auth/authorization-server-migration scenario (draft suite): two
auth servers; PRM flips from AS1 to AS2 after the first authenticated
request; AS2 asserts it received a fresh /register and never saw AS1's
client_id at /authorize or /token
- ConformanceOAuthProvider gains invalidateCredentials and bindIssuer so
the everything-client can key credentials by issuer (passing example)
- everything-client adds an issuer-aware handler for this scenario that
re-reads PRM on each 401 and rebinds before re-authorizing
- auth-test-reuse-credentials negative client + vitest case
* Drop application_type from DCR metadata (not in SDK OAuthClientMetadata type)
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* feat: add server conformance tests for SEP-2575
* refactor and add new checks
* Fix stateless scenario source field and report SKIPPED for inapplicable capability checks
- Add the required `source` field (DRAFT) so the scenario typechecks and is
selectable via --spec-version. Without it the class did not satisfy the
ClientScenario interface and s.source was dereferenced by the spec-version
filter at list time.
- Teach runCheck about a SKIPPED status and use it for the two client-capability
checks when the server does not return -32003, instead of reporting a green
PASS for a requirement that was never exercised.
* Remove unimplemented placeholder checks from stateless server scenario
The subscriptions/listen, statelessness-invariant, list-changed and
disconnect-is-cancel checks emitted SUCCESS without probing the server,
which inflated coverage in the traceability report. Drop them until they
have real assertions; the corresponding rows remain declared in
src/seps/sep-2575.yaml so traceability shows them as not-yet-covered.
Also fix the checkErrorId helper to push under the
sep-2575-http-server-error-jsonrpc-id slug so its failure path actually
short-circuits the aggregate SUCCESS check at the end of the scenario.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* feat: add sdk subcommand to run conformance against any SDK ref (#250)
* Revise sdk runner: explicit --mode, KNOWN_SDKS-only config, v1/v2 entries
Addresses review feedback on the sdk subcommand:
- Require --mode (client|server) and remove "both". Each invocation now
tests exactly one side with its own exit code; the old default ran
client then server but combined exit codes with ||=, which skipped the
server side entirely whenever the client run failed.
- Resolve build/run config from KNOWN_SDKS + CLI flags only; drop the
conformance.config.yaml loader (no SDK ships one yet). The Zod schema
stays as the type for the built-in entries.
- Split the typescript entry by major version: typescript-sdk (v2/main,
pnpm install + build:all, expected-failures.yaml) and typescript-sdk-v1
(v1.x, npm ci + build, conformance-baseline.yml). An entry may set
`repo` (real clone target for an alias) and `defaultRef` (branch used
when no @ref is given). parseSdkSpec now leaves ref undefined when
omitted so defaultRef can apply; a trailing @ is treated as no ref.
- Key the clone cache by ref (<repo>/<ref>) so different refs of the same
repo no longer share one checkout.
- Bound the server readiness probe with a per-request AbortSignal timeout
so a server that accepts the socket but never responds can't hang past
the overall deadline.
* fix(sdk-runner): resolve -o to absolute path; add --expected-failures override; replaceAll for safeName
The sep-2243 scenario check IDs drifted from the requirement-traceability
yaml in #259 — the code emitted one ID per test case while the yaml
declares one per normative requirement, so the IDs no longer matched.
Rename the emitted IDs so a check ID maps to a single MUST/SHOULD and is
emitted once per case (the per-case detail moves to name/description),
matching the repo's existing 'same id, vary status/message' convention:
- mcp-method-header-* / mcp-name-header-* -> client-includes-standard-headers
- reject-invalid-tool-* / keep-valid-tool -> client-reject-invalid-tool
- server-accepts-{lower,upper}case-name -> header-name-case-insensitive
- server reject status checks (mismatch/missing/case x method/name)
-> server-reject-invalid-headers
- their error-code variants -> server-reject-error-code
Merge the yaml's server-reject-mismatch + server-reject-status into one
server-reject-invalid-headers MUST (HTTP 400 on a header-validation
failure); keep server-reject-error-code as the SHOULD. Base64/custom-header
rejection checks keep their own param-validation IDs (out of scope here).
Gates and the whitespace-acceptance check keep their scenario-matching
names. No behavior change — only check IDs, descriptions, and the yaml.
* Add prepare script for git dependency installs
* Add conformance tests for SEP-2243 HTTP Standardization
* Add traceability file for SEP-2243
* Fix prettier formatting in http-standard-headers.ts
* Improve SEP-2243 conformance test coverage
Close gaps in HTTP header conformance scenarios:
Client standard headers (http-standard-headers.ts):
- Enforce all expected methods in checks, failing any that were not observed
- Track Mcp-Name header per-method separately from Mcp-Method
- Advertise resources and prompts capabilities so clients exercise those endpoints
- Add a prompt entry for prompts/get testing
Client custom headers (http-custom-headers.ts):
- Add Base64 encoding checks for non-ASCII, whitespace, and control-char values
- Add null/omitted parameter test (second tool call with null value)
- Add client-keeps-valid-tool check verifying clients still call valid tools
after filtering out invalid ones
Server header validation (server/http-standard-headers.ts):
- Replace fetch-based sendRawRequest with http.request to preserve exact
header casing on the wire (fetch/Headers lowercases names)
- Compute defaultArgs and defaultHeaders from tool schema so test requests
satisfy all required parameters while varying only the param under test
Traceability (sep-2243.yaml):
- Add 7 new spec-to-check mappings (nonascii, whitespace, controlchar,
keeps-valid-tool, literal-missing-base64-prefix/suffix, no-mirror-unannotated)
- Fix 2 incorrect existing mappings
* Fill SEP-2243 conformance test coverage gaps
Add missing test cases identified by comparing the branch against the
SEP-2243 spec's Conformance Test Cases section:
- Mcp-Method on notifications: add notifications/initialized to
expectedMethods so clients are checked for header on notifications
- Boolean true: add debug=true param to complement verbose=false
- Leading-space-only and trailing-space-only: separate checks for
Base64 encoding when only one edge has whitespace
- Internal spaces only: verify plain ASCII (no Base64) for values
like 'us west 1' that have spaces only in the middle
- CRLF string: add line1\r\nline2 test (complements existing \n test)
- Leading tab: add \tindented test for tab-triggered Base64 encoding
- Server rejects invalid Mcp-Param chars: mark as excluded in YAML
since HTTP itself prevents these characters in headers
Register all new checks in sep-2243.yaml.
* Remove case-insensitive =?base64? prefix test
The spec states header values are case-sensitive (RFC 9110). The
=?base64? prefix is part of the header value, not the header name,
so it should be matched case-sensitively. This was identified as a
bug in the conformance tests per feedback on SEP-2243.
Changes:
- Remove server-accepts-case-insensitive-base64 check from server
validation scenario
- Change base64 prefix regex from case-insensitive (/i) to
case-sensitive in client header validation
- Remove corresponding entry from sep-2243.yaml
* Remove nested x-mcp-header rejection test
The spec constrains x-mcp-header to primitive types but does not
restrict it to top-level properties only. A nested string property
with x-mcp-header is valid per the spec. This was confirmed by
mikekistler in SEP-2243 PR discussion.
Removes invalid_nested_header from the invalid tool definitions
list and its corresponding sep-2243.yaml entry.
* Use numeric comparison for number header values
Compare number header values numerically instead of as strings to
accommodate cross-SDK floating point representation differences
(e.g., '42' vs '42.0'). For integers, exact numeric match is
required. For decimals, a tolerance of 1e-9 is allowed.
See SEP-2243 discussion on number precision.
* client/http-standard-headers: SKIPPED (not FAILURE) for unexercised methods; make getChecks() idempotent
A client that never calls prompts/list isn't violating SEP-2243 — the spec
sentence is 'The client MUST include the standard MCP request headers on each
POST request', and a request that was never sent can't fail it. The Mcp-Method
requirement is already proven by initialize/tools/list etc., so there's nothing
unique to prompts/list. SKIPPED keeps the gap visible without a false red.
Separately: getChecks() was pushing into this.checks on every call. The runner
may call it more than once (e.g. progress + final report); that produced
duplicate rows. Build a fresh array per call instead.
* client/http-standard-headers: de-dup guard in checkMcpNameHeader
checkMcpMethodHeader already has 'if (this.methodHeaderChecks.has(method)) return'.
checkMcpNameHeader was missing the equivalent. The test server advertises two
tools and two resources; a client that calls both produces two
'client-mcp-name-header-tools-call' / '...-resources-read' rows.
* client/http-standard-headers: replace -> /\//g for slug generation
String.replace with a string arg only replaces the first occurrence.
'notifications/initialized' is fine (one slash) but a method like
'notifications/resources/updated' would yield 'notifications-resources/updated'
in the check id. (replaceAll isn't in this project's TS lib target.)
* client/http-custom-headers: base64 regex (.+) -> (.*) so empty string round-trips
The spec doesn't forbid base64-encoding values that don't strictly need it.
A client that always wraps would send '=?base64??=' for empty_val: ''. The
'.+' wouldn't match, so the harness would compare '=?base64??=' === ''
literally and FAIL a compliant client. '(.*)' lets the empty payload decode
to '' and match.
* client/http-custom-headers: emit SUCCESS for no-mirror-unannotated
This check only pushed on FAILURE; a correct client produced no row at all,
so the pass was invisible in reports and couldn't be counted toward coverage.
Same-slug SUCCESS/FAILURE pair is the repo convention.
* client/http-custom-headers: drop redundant optional-present check
checkParamHeader('Verbose', ...) already pushes a FAILURE when the header is
missing. The explicit 'optional-present' block re-tests the same condition
under a second check id, doubling the row.
* server/http-standard-headers: defaultArgs must use real number/boolean, not strings
defaultArgs feeds the JSON-RPC body. With Record<string, string> + '0'/'false',
a server that validates inputSchema rejects on type mismatch — which is also a
400. Every header-rejection check then false-PASSes (server rejected for the
wrong reason), and server-accepts-valid-base64 false-FAILs a compliant server
because it never gets past schema validation.
Also handle 'integer' (JSON Schema distinguishes it from 'number').
* server/http-standard-headers: createAcceptanceCheck must also assert no JSON-RPC error in body
A server can return HTTP 200 with {error: {code: -32001, ...}} in the body.
Status-only acceptance lets that through as a pass.
* use DRAFT_PROTOCOL_VERSION constant instead of 'DRAFT-2026-v1' literal
The literal will rot when the draft cycle rolls over. types.ts already exports
DRAFT_PROTOCOL_VERSION for this.
* sep-2243.yaml: spec-first rewrite + move to src/seps/
Regenerated from the SEP-2243 spec diff (transports.mdx + tools.mdx) rather
than from the scenario implementations: 21 check rows + 4 excluded vs the
previous 46 + 2. Differences:
- one check id per spec sentence (test variants go in 'details', not new ids)
- 'text:' quotes the spec sentence verbatim instead of paraphrasing
- '-reject-status' (MUST 400) split from '-reject-error-code' (SHOULD -32001
for standard headers, MUST for custom)
- rows with no spec backing dropped (whitespace, base64-padding/chars, prefix/
suffix-literal, control-char-name)
- two more SHOULD excludes (log-warning, no-sensitive-params)
- check ids use sep-2243-<slug> convention
- check: key first, excludes grouped at bottom (matches sep-2164)
Moved to src/seps/ to match #272 layout (this branch predates it; will
reconcile cleanly on rebase).
* add negative test for HttpStandardHeadersScenario
Self-contained vitest that POSTs initialize with and without Mcp-Method,
asserts FAILURE/SUCCESS on the pinned check id, and proves getChecks() is
idempotent. No example client/server file needed for this one because the
scenario *is* the server — the test crafts the bad request directly.
This is the negative-test half of AGENTS.md §Testing your scenario; the
everything-client passing-example half is still TODO.
* server/http-standard-headers: severity fixes for whitespace + malformed-base64 tests
These five tests assert behavior the SEP-2243 text doesn't pin down. Kept
because they're useful consistency signals across SDKs, but adjusted so they
don't FAIL spec-compliant servers:
- server-accepts-whitespace-header-value: this IS a MUST, but per RFC 9110
§5.5 (field parsing MUST exclude OWS), not SEP-2243. Kept as FAILURE,
specReference now cites RFC 9110.
- server-rejects-invalid-base64-padding / -chars: downgraded to INFO.
SEP-2243 says only 'MUST decode them accordingly' without picking RFC 4648
strict vs lenient. Node's Buffer.from() accepts unpadded/dirty input and
decodes to a matching value (server accepts); .NET's Convert.FromBase64String
throws (server rejects). Either is currently compliant. WARNING would force
Tier-1 SDKs into non-default strict decoding (#245); INFO records the
behavior without prescribing it, so cross-SDK divergence is visible
without tier impact.
- server-literal-missing-base64-prefix / -suffix: unchanged. The wrapper
syntax is '=?base64?{x}?=' complete; treating partial wrappers as literal
is the natural reading.
Plumbing: createRejectionCheck gains an optional failSeverity param;
testBase64Case gains a 'reject-info' mode. (This is also the hook for the
larger 400-MUST / -32001-SHOULD split that still needs doing.)
* server/http-standard-headers: split rejection check into status (MUST) + error-code (SHOULD/MUST)
SEP-2243 §Server Validation: 'servers MUST return HTTP status 400 ... and
SHOULD include a JSON-RPC error response using ... -32001'. So a server that
returns 400 with -32600 (or no error body) is compliant for standard headers.
The single conflated check FAILed it.
For custom headers (§Server Behavior for Custom Headers) -32001 IS a MUST,
so both halves stay FAILURE there.
createRejectionCheck → createRejectionChecks returning two rows: '<id>' for
the 400 status and '<id>-error-code' for -32001. testCase (standard) sets
errorCodeSeverity=WARNING; testBase64Case/testMissingCustomHeader (custom)
keep FAILURE; the two malformed-base64 INFO probes set both halves to INFO.
* rename check ids to sep-2243-* prefix; kebab-case throughout
Aligns with the sep-2243.yaml traceability ids and the repo convention
(sep-2164-*, sep-2207-*). Variant suffixes are kept (per-method, per-tool)
so per-case visibility isn't lost; the YAML's row id is a prefix of the
emitted ids, which is fine — extra ids not in the YAML are allowed.
Also fixes the mixed hyphen/underscore in reject-invalid-tool ids
(toolName has underscores; replace to hyphens).
* client: extract BaseHttpScenario to client/http-base.ts; both client scenarios extend it
http-standard-headers.ts had its own copy of the start/stop/handleRequest
boilerplate (~100 lines) that http-custom-headers.ts already abstracted as
BaseHttpScenario. Moved the abstract class to a shared file; both scenario
files now import it. HttpStandardHeadersScenario implements handlePost()
instead of handleRequest(); its handleInitialize() uses the base
sendInitialize() with the resources/prompts capability flags it needs.
Net: 465 → 361 lines for http-standard-headers.ts; the third inline copy of
the same HTTP-server scaffold is gone.
* use req.setEncoding('utf8') instead of per-chunk Buffer.toString()
Per-chunk .toString() corrupts a multi-byte UTF-8 char that straddles a TCP
chunk boundary (e.g. 0xC3 | 0xBC decodes to two replacement chars instead of
'ü'). The custom-headers scenario sends '日本語'/'naïve' in the body, so this
is reachable in principle. setEncoding('utf8') makes 'data' emit strings with
boundary handling done by Node's StringDecoder.
Fixed in both places: BaseHttpScenario.handleRequest (request body) and
sendRawRequest (response body).
* fix: migrate SEP-2243 scenarios from specVersions to source (introducedIn/removedIn)
The introducedIn/removedIn refactor (#265) replaced the specVersions array
with a source property on all scenario interfaces. The SEP-2243 scenarios
still used the old specVersions field, causing matchesSpecVersion() to
receive undefined and crash in spec-version.test.ts.
Replace specVersions with source = { introducedIn: DRAFT_PROTOCOL_VERSION }
in BaseHttpScenario and the two server ClientScenario classes. Remove the
now-unused SpecVersion imports.
* server/http-standard-headers: malformed-base64 padding/chars back to FAILURE
Per discussion on #259: the SEP-2243 conformance-test-case table is the
approved source of truth and lists these as reject cases, so they're
FAILURE-level even though the spec body itself only says 'MUST decode them
accordingly'. SDKs whose stdlib base64 is lenient (Node, browsers) will need
to validate before decoding; if that's burdensome we'll revisit.
Removes the now-unused 'reject-info' mode and statusSeverity/INFO plumbing
introduced for these two probes.
---------
Co-authored-by: Tarek Mahmoud Sayed <tarekms@microsoft.com>
Co-authored-by: Tarek Mahmoud Sayed <tarekms@ntdev.microsoft.com>
Co-authored-by: Tarek Mahmoud Sayed <10833894+tarekgh@users.noreply.github.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* refactor: replace specVersions list with introducedIn/removedIn range tagging
Replaces per-scenario `specVersions: SpecVersion[]` arrays with `introducedIn`
and optional `removedIn` range fields on all three scenario interfaces. Extension
scenarios get an orthogonal `extension?: boolean` flag instead of the former
`'extension'` tag in the versions list.
matchesSpecVersion() is rewritten as a range comparison
(introducedIn <= target && (!removedIn || target < removedIn)), with draft
sorting after all dated versions. getScenarioSpecVersions() reconstructs the
effective version list from the range for backward-compat with tier-check.
Closes#256.
* refactor: nest spec-version range under `source: ScenarioSource` union
The flat `introducedIn`/`removedIn`/`extension?` shape forced extension
scenarios to set a placeholder `introducedIn` that was never read. Nesting
the version-range fields inside a `source` property typed as a union lets
the compiler enforce introducedIn XOR extensionId — no runtime invariant
test needed.
type ScenarioSource =
| { introducedIn: SpecVersion; removedIn?: SpecVersion }
| { extensionId: ExtensionId };
A class can't `implements` a union type directly, so the union lives inside
a property and `Scenario` stays a plain interface — `class implements
Scenario` keeps working everywhere. `EXTENSION_IDS` is a const array
mirroring the `DATED_SPEC_VERSIONS` pattern.
Also renames the cross-app-access scenario to enterprise-managed-auth to
match the extension's canonical name: file, class, slug, context schema
literal, and the everything-client runner.
* refactor: use canonical SEP-2133 extension identifiers
EXTENSION_IDS now holds the wire identifiers from the spec's extension docs
(client-matrix.mdx, docs/extensions/auth/*.mdx) so extensionId values can be
cross-referenced against capabilities.extensions keys directly:
io.modelcontextprotocol/oauth-client-credentials
io.modelcontextprotocol/enterprise-managed-authorization
The previous commit used a shortened 'enterprise-managed-auth' which doesn't
match the canonical '-authorization'. Re-renames the scenario file/class/slug
and downstream references to align.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat: add new-sep scaffolding command
Related #243.
This commit adds the new-sep <NNNN> subcommand, which writes the file src/scenarios/<target>/sep-<NNNN>.yaml. This file contains the "sep", "spec_url", and "requirements" rows in the same style as sep-2164.yaml, with single quotes and two spaces. The spec source resolves in order: --spec-url → --spec-path → GitHub API against modelcontextprotocol/modelcontextprotocol (explicit --pr or title search for "SEP-NNNN").
The target directory is inferred from the spec path (`server/`, `client/`, or `basic/authorization*`); the --target flag overrides this. It refuses to overwrite without the --force flag.
Token resolution reuses the "gh auth token" fallback pattern from tier-check.
It also ships the .claude/skills/new-sep directory, which drives the CLI. It fetches the spec diff via "gh api" and extracts RFC 2119 sentences. There is a warning against regex-only matching. Bullets inherit keywords from their lead-in sentence. It rewrites the placeholder rows per the severity rules in AGENTS.md. MUST/SHALL/REQUIRED are checked for FAILURE. SHOULD is checked for WARNING. MAY/OPTIONAL are excluded.
Signed-off-by: Satoshi Ito <satoshi.ito.tf@hitachi.com>
* dropped --pr arg, list changed files, support multi-file seps w url override
* Consolidate SEP yamls into src/seps/ and tighten yaml ordering
- All sep-NNNN.yaml files now live in src/seps/ regardless of which
scenario suite their checks run in
- Drop --target / inferTarget; output dir is fixed
- renderYaml: check rows emit `check:` first, excluded rows grouped at
bottom with a blank-line separator
- SKILL.md: document key/row ordering, expand exclusion-confirmation step
- Move src/scenarios/server/sep-2164.yaml -> src/seps/
---------
Signed-off-by: Satoshi Ito <satoshi.ito.tf@hitachi.com>
Co-authored-by: Paul Carleton <paulc@anthropic.com>
* feat: support "draft" as a first-class spec version target
- Add LATEST_SPEC_VERSION and DATED_SPEC_VERSIONS constants so the next
spec release is a one-line change in types.ts.
- Accept "draft" as a valid protocolVersion in the initialize check and
mock-server response.
- --spec-version draft now selects latest-dated scenarios plus draft-tagged
ones, so SEP authors can run the full suite against an SDK tracking the
in-progress spec without retagging core scenarios.
- Forward --spec-version to the client process via
MCP_CONFORMANCE_SPEC_VERSION so SDK examples can pick the matching
protocolVersion.
Closes#253
* review: collapse SpecVersion to wire strings; dedup NEGOTIABLE_PROTOCOL_VERSIONS; split out 'extension'
Addresses review feedback from @felixweinberger and @mikekistler:
- SpecVersion is now always a wire protocolVersion string
(DatedSpecVersion | typeof DRAFT_PROTOCOL_VERSION). The separate
'draft' tag literal is gone from the type system; 'draft' survives
only as a CLI input alias in resolveSpecVersion. Removes the
tag-vs-wire confusion and the specVersionToProtocolVersion mapping.
- NEGOTIABLE_PROTOCOL_VERSIONS in types.ts is the single source for
what the mock server accepts on initialize (was duplicated in
checks/client.ts and scenarios/client/initialize.ts).
- 'extension' moved out of SpecVersion into a separate ScenarioSpecTag
type. --spec-version extension is no longer valid; extension scenarios
remain reachable via --suite extensions.
- Draft-tagged scenarios now use the DRAFT_PROTOCOL_VERSION constant
so a draft revision bump is a one-line change in types.ts.
Conformance is a leaf CLI; scheduled version bumps are churn. Deleting
the config disables version-update PRs entirely. Security advisories
remain enabled via repo settings (Code security > Dependabot).
Adds src/scenarios/server/negative.test.ts with spawn helpers. Wires
two cases: the previously-orphaned no-dns-rebinding-protection.ts and
a new sep-2164-empty-contents.ts. Strengthens the AGENTS.md guidance
to point at the harness.
* feat: add conformance scenario for SEP-2164 resource-not-found error code
Adds a draft-scoped scenario verifying servers return -32602 (Invalid
Params) for non-existent resources instead of the legacy -32002 or an
empty contents array. Includes a sep-2164.yaml traceability file mapping
each MUST/MUST NOT/SHOULD to its check per the proposed SEP-2484 process.
* fix: align SEP-2164 check severities with spec text (#246)
Splits sep-2164-error-code into:
- sep-2164-no-empty-contents (MUST NOT -> FAILURE)
- sep-2164-error-code (SHOULD -> WARNING)
Updates sep-2164.yaml to quote the spec text verbatim and adds an
excluded row for the client-side -32002 backcompat SHOULD.
See #245 for the SHOULD/WARNING tier-scoring policy discussion.
---------
Co-authored-by: Paul Carleton <paulcarletonjr@gmail.com>
Fixes a bug where the Markdown output format for the Repository Health section lacked a table header, causing it to render as raw text instead of a table on GitHub.
* feat: add --spec-version option to tier-check
* refactor: share resolveSpecVersion between CLI and tier-check
Move resolveSpecVersion to src/scenarios/index.ts so tier-check can
reuse it instead of duplicating the validation logic.
---------
Co-authored-by: Felix Weinberger <fweinberger@anthropic.com>
* Fix Windows compatibility in tier-check client conformance
Two issues prevented client conformance tests from running on Windows:
1. Single-quote quoting in execSync: The --command argument was wrapped in single quotes, which cmd.exe treats as literal characters rather than string delimiters. This caused the CLI to receive only the first word of the command. Switched to execFileSync with an args array to bypass shell quoting entirely.
2. Backslash path separators in parseOutputDir: dirname() on Windows returns auth\scenario-name but reconcileWithExpected compares against auth/scenario-name, causing auth scenarios to be double-counted. Normalize backslashes to forward slashes in parsed scenario names.
* fix: use process.argv[1] for CLI path in subprocess calls
Combines the npx-compatibility fix from #175 with this PR's execFileSync
approach. process.argv[1] resolves to the absolute path of the running CLI
regardless of working directory, so tier-check works when invoked via npx
or from any cwd.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
Major bumps of devDeps (eslint, typescript, prettier) need manual config
migration and provide no value for conformance testing. Security updates
still come through regardless; scheduled updates are now minor/patch only.
* docs: add AGENTS.md and CONTRIBUTING.md
Codifies scenario design guidelines from past PR reviews:
- fewer scenarios with more checks (CI cost multiplies per-SDK)
- same check id for SUCCESS and FAIL, optimize for Ctrl+F
- extend the everything-client/server rather than adding new example files
- prove it passes and fails
- reuse the CLI runner, don't add parallel entry points
Also asks contributors to open an issue for discussion before a PR, and
to target specific spec MUSTs or open issues rather than running generic
AI bug-hunts against the harness.
* docs: broaden issue-first guidance to cover bugs, not just scenarios
