78 lines
4 KiB
YAML
78 lines
4 KiB
YAML
# Conformance scenarios not yet passing against the Python SDK on main.
|
|
# CI exits 0 if only these fail, exits 1 on unexpected failures or stale entries.
|
|
#
|
|
# Baseline established against the harness pinned via CONFORMANCE_PKG in
|
|
# .github/workflows/conformance.yml. New conformance releases are adopted by
|
|
# deliberately bumping that pin and reconciling both this file and
|
|
# expected-failures.2026-07-28.yml in the same change.
|
|
#
|
|
# Entries are grouped by SEP. As each SEP lands in the SDK the corresponding
|
|
# scenarios start passing and MUST be removed from this list (the runner fails
|
|
# on stale entries), so the baseline burns down per milestone.
|
|
|
|
client:
|
|
# --- Draft-spec scenarios (in `--suite draft`, also part of `--suite all`) ---
|
|
# SEP-2575 (request metadata / _meta envelope): client does not populate the
|
|
# _meta envelope or the MCP-Protocol-Version header semantics yet.
|
|
- request-metadata
|
|
# SEP-2322 (multi-round-trip requests): client does not echo requestState /
|
|
# handle IncompleteResult yet.
|
|
- sep-2322-client-request-state
|
|
# SEP-2243 (HTTP standardization): no fixture handler / client header support yet.
|
|
- http-custom-headers
|
|
- http-invalid-tool-headers
|
|
# SEP-2106 (JSON Schema $ref handling): client still dereferences network $refs.
|
|
- json-schema-ref-no-deref
|
|
# SEP-2352 (authorization server migration): client does not re-register when
|
|
# PRM authorization_servers changes.
|
|
- auth/authorization-server-migration
|
|
# SEP-837 (application_type during DCR): the check fires on every non-legacy
|
|
# spec version (the default LATEST is 2026-07-28). The client omits
|
|
# application_type during Dynamic Client Registration, so every scenario that
|
|
# reaches DCR fails it. SEP-2468 iss validation is implemented, so these now
|
|
# fail only on the application_type check, not on iss.
|
|
- auth/offline-access-not-supported
|
|
- auth/iss-supported
|
|
- auth/iss-not-advertised
|
|
- auth/iss-supported-missing
|
|
- auth/iss-wrong-issuer
|
|
- auth/iss-unexpected
|
|
- auth/iss-normalized
|
|
|
|
# --- Pre-existing scenarios that fail on checks added after conformance 0.1.15 ---
|
|
# SEP-990 (enterprise-managed authorization extension): no fixture handler /
|
|
# client support for the token-exchange + JWT bearer flow.
|
|
- auth/enterprise-managed-authorization
|
|
|
|
server:
|
|
# --- Draft-spec scenarios (in `--suite draft`; the `active` suite is green) ---
|
|
# SEP-2575 (stateless HTTP / _meta envelope): server has no stateless mode,
|
|
# _meta-derived capabilities, error-code mappings, or server/discover yet.
|
|
- server-stateless
|
|
# SEP-2322 (multi-round-trip requests / IncompleteResult): not implemented;
|
|
# most scenarios currently fail early with "Missing session ID" because
|
|
# mcp-everything-server only runs in stateful mode.
|
|
- input-required-result-basic-elicitation
|
|
- input-required-result-basic-sampling
|
|
- input-required-result-basic-list-roots
|
|
- input-required-result-request-state
|
|
- input-required-result-multiple-input-requests
|
|
- input-required-result-multi-round
|
|
- input-required-result-non-tool-request
|
|
- input-required-result-result-type
|
|
- input-required-result-tampered-state
|
|
- input-required-result-capability-check
|
|
# SEP-2243 (HTTP header standardization): -32020 HeaderMismatch handling and
|
|
# case-insensitive/whitespace-trimmed header validation not implemented.
|
|
- http-header-validation
|
|
# WARNING-only entries: these scenarios emit no FAILURE checks, only SHOULD-level
|
|
# WARNINGs, but the expected-failures evaluator counts WARNINGs as failures.
|
|
# SEP-2322 SHOULD-level behaviour (re-request missing inputResponses).
|
|
- input-required-result-missing-input-response
|
|
# SEP-2322 negative-case scenarios: input-required-result-validate-input is
|
|
# now baselined (added when the stateless path landed — the stateless server
|
|
# reaches the handler, so the previous accidental pass via -32600 "Missing
|
|
# session ID" no longer applies). input-required-result-unsupported-methods
|
|
# is intentionally NOT baselined: it still passes for now; add it once it
|
|
# starts failing for real.
|
|
- input-required-result-validate-input
|