feat(sep-2640): skills server conformance against the Accepted SEP, enumeration + manifest + directory (#330)
* chore: add SEP-2640 requirement-traceability YAML (Skills Extension)
* style: apply yaml formatter pass
* chore(sep-2640): record spec source provenance
* chore(sep-2640): re-extract against SEP HEAD 556154c (drops mcp-resource-template)
Re-extract sep-2640.yaml against current SEP-2640 HEAD. The SEP removed
the `mcp-resource-template` index entry type in two commits on
2026-06-04 (fd50cc91 "Remove mcp-resource-template entries from skill
index", 556154c0 "Remove remaining resource template mentions from SDK
and rationale sections"), after this extraction was first captured on
2026-06-03 at b77fdfe8. Provenance comment bumped accordingly.
Resulting changes:
- sep-2640-index-entry-type-enum: drop `"mcp-resource-template"` from
the enum.
- sep-2640-index-name-required: drop the "is Required for ... Omitted
for mcp-resource-template" conditional language; the column is now
unconditional Required=Yes at SEP HEAD.
- sep-2640-index-digest-required: same simplification; also align
"hexadecimal" -> "hex" with the HEAD table text.
- sep-2640-template-resource-template-registered: removed. The SEP no
longer defines a template entry type, so the SHOULD on registering an
MCP resource template is gone.
- Excluded "Hosts SHOULD surface template entries..." excerpt: removed.
The sentence no longer exists in the SEP.
Sweep verified: zero "template" mentions in the SEP at HEAD. Security
Implications section sentences referenced by the remaining excluded
entries are intact. No other check rows touched.
* feat(sep-2640): ResourcesDirectoryReadScenario for resources/directory/read
Adds the conformance scenario for the SEP-2640 directoryRead surface that
landed in spec commit 2e04c48d (2026-06-09). Per AGENTS.md "fewer scenarios,
more checks", a single ResourcesDirectoryReadScenario emits 6
ConformanceChecks, one per new sep-2640.yaml requirement row.
Class named for the wire method (resources/directory/read), matching the
existing ResourcesListScenario / ResourcesReadTextScenario / etc. family in
src/scenarios/server/resources.ts. The runner-facing name field stays as
'sep-2640-skills' (umbrella) so mcpkit's conformance/Makefile entry
--scenario sep-2640-skills keeps working without a cross-repo race.
src/seps/sep-2640.yaml
- File-level provenance held at 556154c (the existing PR 330 baseline).
- 6 new check rows appended for the directoryRead additions, each
carrying a verbatim, grep-F-searchable excerpt from the SEP at
2e04c48da90224000e750ffd54a3611f2824fbc0:
- sep-2640-capability-directory-read-flag
- sep-2640-directory-read-method-registered
- sep-2640-directory-read-subdir-mimetype
- sep-2640-directory-read-result-resources-shape
- sep-2640-directory-read-invalid-params
- sep-2640-directory-read-pagination
- forward_reference header comment notes the asymmetry: PR 97 schema
rewrite (360123d0, 2026-06-08) made 3 existing rows stale and drifted
~11 others' wording; full re-extraction at SEP HEAD is mcpkit#780's
lifecycle. Provenance deliberately holds at 556154c until that lands.
src/types.ts
- Adds io.modelcontextprotocol/skills to EXTENSION_IDS so the scenario's
source: { extensionId: ... } tag type-checks.
src/scenarios/server/directory.ts (new)
- Capability discovery via wire-observable signal: -32601 method-not-found
is the only definitive "server did not declare directoryRead" signal.
- Fixture assumption: server exposes skill://acme/billing/refunds/templates
with at least one subdirectory child (mcpkit examples/skills layout).
No skill:// resources at all -> every check emits SKIPPED so the
scenario stays green against upstream's everything-server fixture.
- 6 checks (1:1 with the YAML rows above):
1. directoryRead declared (derived from method registration)
2. method registered (happy-path call succeeds)
3. result.resources shape matches resources/list
4. subdir child carries mimeType: "inode/directory"
5. non-directory URI returns -32602
6. nextCursor round-trips (single-page is conformant)
src/scenarios/index.ts
- Imports + registers ResourcesDirectoryReadScenario in
allClientScenariosList, matching the registration name 'sep-2640-skills'
that mcpkit's conformance/Makefile already passes to --scenario.
Verified end-to-end against mcpkit examples/skills:
cd ~/newstack/mcpkit/main && \
MCPCONFORMANCE_SKILLS_PATH=~/newstack/mcpkit/conf-skills \
make -C conformance testconf-skills
-> 6/6 SUCCESS, 0 failed, 0 warnings.
Out of scope:
- PR 97 schema rewrite YAML refresh (separate ticket, mcpkit#780 lifecycle).
- Negative-capability fixture (server without directoryRead) - mcpkit
examples/skills has no flag for it; ext/skills/client_directory_test.go
already covers the SDK pre-call guard.
* feat(sep-2640): expand skills conformance — index + manifest scenarios, drift fixes
Builds on the directory-read scenario with the server-observable skills surface
and brings the whole set up to the current framework conventions.
New scenarios (src/scenarios/server/skills/):
- SkillsIndexScenario (sep-2640-skills-index): skill://index.json is exposed and
readable; every skills[].type is in the SEP enum; every entry has a non-empty
name; a present digest matches sha256:{64hex}; entry URLs use skill://.
- SkillsManifestScenario (sep-2640-skills-manifest): a skill-md SKILL.md is
markdown-typed; frontmatter name/description surface on the resource; the
final skill-path segment equals the frontmatter name; skill _meta keys use the
reverse-domain prefix.
directory.ts (moved into skills/): capability is now read from server/discover
instead of inferred from -32601; an undeclared optional capability SKIPs rather
than FAILs; the untestable.ts helpers back the not-exercisable paths; all
mcpkit-specific fixture URIs are gone — the directory to exercise is discovered
dynamically, so the scenario is brand-neutral.
Capability gating accepts directoryRead at either extensions[id].directoryRead
(SEP-2640's inline example) or extensions[id].config.directoryRead (the SEP-2133
negotiation envelope SEP-2640 defers to); the two SEPs are inconsistent on
nesting, so the check accepts either rather than privileging one reading.
All three scenarios verified green against a live SEP-2640 server (6/6, 6/6, 7/7);
typecheck, eslint, prettier, and the 504-test vitest suite pass. sep-2640.yaml
gains a header note mapping each scenario to the rows it now backs.
* feat(sep-2640): re-extract against the 2026-08-21 rewrite
The prior extraction held at SEP commit 556154c (2026-06-05) by an explicit
in-file decision to wait for the index-schema refactor. The 08-21 core-maintainer
rewrite made that pin actively wrong rather than merely behind.
Removed as no longer normative: the skill://index.json well-known resource
(absent from the current text), the entry type enum, the no-nested-skills rule
(now reversed - skills MAY nest), and the five archive rows (archives are
deferred to an appendix). Reworded the two index-field rows, the empty-listing
row and the verify-digest row.
Added the new surface: skills/list and skills/get with their pagination,
atomicity and caching attributes; the {uri, frontmatter, resources} entry
schema; resources completeness, the "dynamic" sentinel and per-file size; the
512-entry and 16 MiB limits; and the host obligations around lazy retrieval,
content-bound approval and frontmatter re-verification.
index.ts is replaced by enumeration.ts, which drives both new methods. manifest.ts
and directory.ts swap their skill://index.json discovery fallback for skills/list.
The capability helper no longer accepts a config envelope. SEP-2133 is Final and
defines the extensions field as a map of identifiers to settings objects, with no
envelope and no slot for id/specVersion/stability; SEP-2640's capability block
matches. The earlier leniency was based on a misreading.
77 rows: 65 checks (39 wire-tested by the three server scenarios) and 12 excluded
as host-internal.
* feat(sep-2640): close the extraction gaps found by a normative-sentence audit
The first pass extracted the Specification sections but under-covered Security
Implications, which carries 21 of the SEP's 97 RFC-2119 sentences. Sweeping
every normative sentence in the source surfaced rows that were missing entirely:
- origin tagging at the point content enters model context (the L179 host MUST)
- allowed-tools MUST be ignored for MCP-origin skills
- reads bound to the originating server, so a skill from A cannot drive a read
against B, and any cross-origin read gated behind per-call approval
- servers identified by a host-assigned label, never serverInfo.name
- the three disk-cache rules: write-isolation or re-hash, exclusion from
filesystem-skill discovery paths, and removal when a server is removed
- a resources/read of a SKILL.md by another route is not a load
- a nested SKILL.md read as supporting content is ordinary markdown
- the registry-assembly and virtual-mount no-prefetch restatements
- the SDK affordances guidance, which is the row mcpkit itself is measured by
Five rows previously filed as excluded are now declared instead. They are real
requirements that a host-side harness could check; calling them unobservable was
a limitation of this scenario set, not of the requirement.
Adds one wire check, names-should-be-unique, since a within-listing collision is
observable and tells an operator the listing needs disambiguating.
96 rows: 89 checks (40 wire-tested) and 7 excluded.
* style: apply prettier to the SEP-2640 scenarios and yaml
Quote-style normalisation in the yaml (escaped single quotes become
double-quoted scalars) and one return-type wrap in helpers.ts. The parsed
yaml is byte-identical in content; verified by comparing the parse before
and after.
* docs(sep-2640): note why resultType is not declared in this yaml
It appears in all three result examples but is a base-protocol field on the
common Result interface, not a SEP-2640 requirement; its caching-hint
obligations are already owned by sep-2549.yaml.
* fix(sep-2640): follow nextCursor on resources/directory/read
The directory scenario issued one resources/directory/read and every check
below it inspected that single page: the result-shape check, the
subdirectory-mimeType check, and the target selection for the -32602 probe.
SEP-2640 says directory-read pagination mirrors resources/list, so a
conformant server MAY split a directory across pages, and reading only the
first made "no subdirectory here" indistinguishable from "the subdirectory
is on page two".
Reported by Sam Bloomberg against the go-sdk implementation, which exposes
DirectoryOptions.PageSize. Rebuilding that server with PageSize 1 fails
sep-2640-directory-read-subdir-mimetype; with this change it passes 7/7,
and both servers still pass at their default page size.
Only findable against a second implementation. mcpkit hardcoded its page
size to zero with no override, so it never emitted a cursor and the defect
was invisible to every run against it.
listAllResources already did this for resources/list. directoryReadAll
applies the same shape to directory reads, with a page bound and a
repeated-cursor guard.
* feat(sep-2640): client scenario for the no-prefetch MUST NOT
The traceability file filed retrieval policy under "host obligations a
harness cannot observe", alongside 48 other rows. That was wrong for this
one: "hosts MUST NOT retrieve a skill's files ahead of need" reduces to
whether a request arrived, which makes it among the most wire-observable
rules in the SEP.
Adds a Scenario (harness stands up the server, client is the SUT) that
serves a listing naming a SKILL.md and a supporting file, then fails if
either is read. The client's contract, keyed on MCP_CONFORMANCE_SCENARIO,
is to connect, enumerate, and stop.
SKIPs rather than passes when the client never calls skills/list, since
without a listing there is no window in which prefetching is possible and
a pass would prove nothing.
Verified both directions against mcpkit's client: SUCCESS as written, and
FAILURE with a deliberate ReadFromEntry added to the driver. A check that
has never been shown to fail is the same trap as the directory-pagination
false negative this suite shipped for months.
* feat(sep-2640): client scenarios for the four read-time verification MUSTs
Adds digest, size, frontmatter and unlisted-URI variants alongside the
no-prefetch scenario. Each serves an internally honest listing, tampers with
exactly one thing on the wire, and watches whether the client notices.
Rejection is detected by absence, following auth/resource-mismatch: the
client's contract is "load the skill, then read its supporting file", and a
client that verifies aborts on the load and never reaches the supporting
file. Reaching it is the violation. The unlisted variant is proven by the
read never appearing on the wire at all, since a conformant client refuses
locally.
The frontmatter variant recomputes digest and size for the tampered body, so
the frontmatter comparison is the only thing left that can fail. Without
that the digest check would mask it and the scenario would pass for the
wrong reason.
allowClientError is set on all four: a verifying client exits non-zero
because it correctly aborted, and that is the pass.
Verified both directions against mcpkit's client. All four SUCCESS as
written; all four FAILURE against a driver rewritten to read through
ReadSkillURI, which skips verification entirely.
* fix(sep-2640): gate the cache-attributes check on the negotiated version
The check graded ttlMs and cacheScope without looking at the protocol
version, which made it wrong in both directions. Below 2026-07-28 those
fields are not defined by the negotiated schema at all, so a server that
omits them is correct and the WARNING was a false positive. From 2026-07-28
they are required, so a missing one is a failure rather than a warning.
Now SKIPPED with a version reason below 2026-07-28, and SUCCESS or FAILURE
above it.
Reported by Sam Bloomberg against the Go SDK, which was correct on both
wires while this check mis-graded it on both. Second finding from the same
reviewer against this branch, and the second that only a second
implementation could surface: mcpkit emits the attributes on every version
it supports, so it could never have produced the false positive.
Verified three ways: mcpkit at 2026-07-28 SUCCESS, the Go SDK at 2025-11-25
SKIPPED where it previously warned, and a deliberately stripped mcpkit build
at 2026-07-28 FAILURE.
* docs(sep-2640): how to run the skills scenarios against any implementation
Three implementations have now been cross-checked and every one needed a
different invocation, so the knowledge was living in chat rather than the
repo. Records the two flags and why both are needed, how to read totals that
include wire-schema-valid, and the per-SDK setup.
The two traps are worth writing down because both have already cost someone
time. Extension scenarios do not match --spec-version alone, so --force is
always needed. And --spec-version selects the wire lifecycle rather than
filtering, so omitting it against a pre-2026-07-28 server produces a -32022
that reads like a missing method.
Also records that the go-sdk serves 2026-07-28 only when its streamable
transport is Stateless, which is correct per SEP-2575 and which I briefly
mistook for a defect.
* fix(sep-2640): advertise skills on server/discover, not only initialize
Both client scenarios declared the skills extension and `resources` in their
`initialize` reply, but `BaseHttpScenario` intercepts `server/discover` before
`handlePost` runs. On the 2026-07-28 lifecycle a discover-first client
therefore saw the base class default of `{tools: {}}` and had nothing to gate
a `skills/list` call on.
The scenarios still passed, because `server/discover` is optional for clients
and neither client we ran used it. That is the failure mode where a check goes
green because its condition never arose, so both lifecycles are now asserted
in capabilities.test.ts rather than left to the next implementation to find.
Overriding `discoverCapabilities()` covers both paths at once, since
`sendInitialize` already defaults to it.
* fix(sep-2640): give the capability and naming checks something to fail on
`sep-2640-capability-empty-object` reported a hardcoded SUCCESS, so it could
not fail and was checking nothing. It now asserts the observable half of "an
empty object indicates support for the extension with no optional features":
the declared value must be an object at all.
That also closes a false SKIP. `skillsCapability` folded a malformed
declaration into `undefined`, which downstream read as "extension not
declared" and skipped the whole suite, so a server declaring the extension as
`true` produced a clean run. `declaredSkillsCapability` separates absent from
malformed; the inline/envelope check now SKIPs on a malformed value, since
there are no settings to place either way. The base `DiscoverResult` schema
rejects a non-object too, so both halves are asserted.
`SKILL_NAME_PATTERN` accepted consecutive hyphens. The Agent Skills
specification that SEP-2640 defers to for naming lists four rules and we
encoded three, so `pdf--processing` passed. Its own valid and invalid examples
are now the test cases.
Negative controls recorded for all three: with the check back to a hardcoded
SUCCESS the four malformed-declaration cases fail, and with the previous regex
the consecutive-hyphen case fails.
* docs(sep-2640): name skills/list in the discovery prose, not index.json
The 2026-08-21 revision replaced the `skill://index.json` well-known resource
with `skills/list`, and the code follows it — `discoverDirectory` and the
manifest fallback both call `skillsListAll`. Only the prose lagged, including
three strings a user actually sees: two untestable-check reasons and the
directory scenario's description.
The remaining mentions in helpers.ts, enumeration.ts and sep-2640.yaml are
deliberate history, describing what was replaced.
* fix(sep-2640): grade unlisted reads against the entry, not one fixture URI
The `unlisted` client scenario asserted `reads.includes(UNLISTED_URI)`, so it
only caught a client that probed the exact path this fixture happens to serve.
Any client choosing a different unlisted path passed without being tested, and
the check would have reported SUCCESS for a client that reads unlisted files
freely.
The requirement is "hosts MUST resolve reads of the skill's files only to URIs
listed in that entry's resources", so the permitted set now comes from the
entry the client was handed and any read outside it is the violation. That is
the sentence itself rather than a proxy for it, and it lets a SUT derive its
own probe instead of hardcoding ours.
Negative control: a client reading a synthesized `mcpkit-unlisted-probe-0.md`
is now a FAILURE, where the previous check reported SUCCESS.
* fix(sep-2640): require the manifest read, and retire the unlisted scenario
Two changes to the client-side verification scenarios, one of which fixes a
hole the other one shares.
**The manifest read is now a prerequisite.** Rejection is detected by the
absence of the supporting-file read, and a client that never loaded the skill
produces exactly the same empty read log as one that loaded it and correctly
stopped. Both reported SUCCESS, so `listCalled` was the only gate and a client
implementing `skills/list` and nothing else passed every variant here. It
scored 5/5 on this suite while doing nothing.
Absence now means something only after the client has read the tampered
`SKILL.md`. A missing prerequisite reports untestable, which per #248 is a
FAILURE rather than a SKIP: SKIPPED is excluded from pass/fail counts and exit
codes, so it would read as green and hide the gap.
Negative control: a driver that lists and exits goes from 1/1 to 0/1 on all
three variants.
**`-verify-unlisted` is retired**, on Peter Alexander's call. Nothing was
tampered in it, so the only thing under test was whether the driver asked for
a URI it should not have, which grades the driver rather than the client. It
is also the rule an inspection tool answers differently from a host, since
refusing a read requires having a load to refuse, which MCP Inspector 2.6.0
reported as "surfaced" rather than passed.
The requirement is not deleted. It moves to an `excluded:` row alongside the
acting-on window that scopes it, and the reason records that it is excluded by
construction rather than in principle: a `SKILL.md` body linking to a file
absent from `resources` would make the harness adversarial again, since a host
that follows the link fetches an unlisted file on its own initiative. The
scenario name and check id are named there so a future reintroduction lines up
with historical reports rather than reading as a new requirement.
Check rows go from 89 to 88.
* feat(sep-2640): check skills/get cache attributes and the resources capability
Both requirements landed when the stable spec page merged ext-skills#139
today, and both are checkable on the wire.
**`sep-2640-skills-get-cache-attributes`.** `GetSkillResult extends
CacheableResult`, so `ttlMs` and `cacheScope` are REQUIRED as they are on
`resources/read`. Held out of this PR until the page merged rather than
grading against text that was still open: SEP-2640 says of these fields that
"whether the result should also carry the base protocol's caching attributes
... is left open", so this is a behaviour change and a server conformant to
the SEP text alone omits both. It is a reading not everyone shares. The MCP
Inspector's SEP-2640 writeup argues the opposite, that the SEP forecloses it
and inventing the requirement would report a reasonable server as broken.
Gated the same way as the skills/list check: below 2026-07-28 `CacheableResult`
is undefined by the negotiated schema, so omission is correct and the check
SKIPs with a version reason rather than failing a conformant legacy server.
**`sep-2640-capability-requires-resources`.** A server declaring the extension
MUST also declare the base `resources` capability, since skill files are served
through `resources/read`. The page states it as a consequence of the base
Resources specification rather than a free-standing new rule, and the
declaration is on the wire either way.
Verified against mcpkit: 32/32 on 2026-07-28 and 30/30 on 2025-11-25, where the
get-cache check SKIPs below the floor as designed. Negative control recorded: a
build with the attributes stripped from `handleSkillsGet` reports 31/32 with
`sep-2640-skills-get-cache-attributes` FAILURE.
Check rows go from 88 to 90.
* fix(sep-2640): compare frontmatter by content, not key order
entry-frontmatter-identical compared each top-level key with JSON.stringify,
which is key-order sensitive for nested objects. A server whose encoder sorts
map keys (Go's encoding/json does) failed this MUST for any SKILL.md whose
metadata keys are not in alphabetical order, although the content is
identical.
Compare structurally instead. A real difference still fails; both cases are
pinned in negative.test.ts.
* fix(sep-2640): make the client-scenario mocks answer like a conformant server
The skills/list and resources/read replies carried only resultType, so on the
2026-07-28 wire they lacked the ttlMs/cacheScope a ReadResourceResult
requires, and skills/list lacked the attributes the enumeration scenario
requires of servers.
initialize always answered 2026-07-28, while the runner hands these extension
scenarios' clients 2025-11-25 by default. A client that checks the negotiated
version disconnected before listing, and every verify variant reported
SKIPPED whether or not the client verified anything.
The mocks now stamp the caching members and answer initialize at the
requested version when the suite can speak it. The runner default is left
alone because it also applies to the auth extension scenarios. Pinned in
draft-result-fields.test.ts next to the other hand-rolled mocks.
* docs(sep-2640): run from the published package; --force only with --spec-version
The run notes pointed at a fork branch that goes stale on merge, and said
--force is always needed. Extension scenarios only SKIP when an explicit
--spec-version is passed; without it they run on the draft wire.
* docs(sep-2640): drop RUNNING_SEP2640.md
Per-run notes and SDK result tables belong in the PR description, not a
root-level doc that goes stale on merge.
---------
Co-authored-by: Paul Carleton <paulc@anthropic.com>
This commit is contained in:
Sri Panyam
GitHub
parent
a983ba93c9
commit
7169291ec0
13 changed files with 3767 additions and 3 deletions
|
|
@ -12,6 +12,8 @@ import type { BaseHttpScenario } from './http-base';
|
|||
import { RequestMetadataScenario } from './request-metadata';
|
||||
import { MRTRClientScenario } from './mrtr-client';
|
||||
import { JsonSchemaRefDerefScenario } from './json-schema-ref-deref';
|
||||
import { SkillsNoPrefetchScenario } from './skills/no-prefetch';
|
||||
import { SkillsVerificationScenario } from './skills/verification';
|
||||
|
||||
/**
|
||||
* Pins that the hand-rolled mock servers used by client-direction scenarios
|
||||
|
|
@ -528,3 +530,96 @@ describe('sep-2322-client-request-state mock results (2026-07-28)', () => {
|
|||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('sep-2640 skills client mock results', () => {
|
||||
const skillsScenarios: Array<{
|
||||
name: string;
|
||||
make: () => BaseHttpScenario;
|
||||
}> = [
|
||||
{
|
||||
name: 'sep-2640-client-no-prefetch',
|
||||
make: () => new SkillsNoPrefetchScenario()
|
||||
},
|
||||
{
|
||||
name: 'sep-2640-client-verify-digest',
|
||||
make: () => new SkillsVerificationScenario('digest')
|
||||
},
|
||||
{
|
||||
name: 'sep-2640-client-verify-size',
|
||||
make: () => new SkillsVerificationScenario('size')
|
||||
},
|
||||
{
|
||||
name: 'sep-2640-client-verify-frontmatter',
|
||||
make: () => new SkillsVerificationScenario('frontmatter')
|
||||
}
|
||||
];
|
||||
|
||||
for (const s of skillsScenarios) {
|
||||
it(`${s.name} carries the draft-required members on skills/list and resources/read`, async () => {
|
||||
const scenario = s.make();
|
||||
const { serverUrl } = await scenario.start(
|
||||
testScenarioContext(DRAFT_PROTOCOL_VERSION)
|
||||
);
|
||||
try {
|
||||
let id = 1;
|
||||
for (const [method, params] of [
|
||||
['skills/list', {}],
|
||||
['resources/read', { uri: 'skill://pdf-processing/SKILL.md' }]
|
||||
] as const) {
|
||||
const { status, body } = await post(
|
||||
serverUrl,
|
||||
{
|
||||
jsonrpc: '2.0',
|
||||
id: id++,
|
||||
method,
|
||||
params: { ...params, _meta: meta }
|
||||
},
|
||||
{
|
||||
'mcp-protocol-version': DRAFT_PROTOCOL_VERSION,
|
||||
'Mcp-Method': method
|
||||
}
|
||||
);
|
||||
expect(status, method).toBe(200);
|
||||
expect(body.result, method).toMatchObject(CACHEABLE_FIELDS);
|
||||
if (method === 'resources/read') {
|
||||
expect(
|
||||
wireSchemaErrors(DRAFT_PROTOCOL_VERSION, body, method),
|
||||
method
|
||||
).toEqual([]);
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
await scenario.stop();
|
||||
}
|
||||
});
|
||||
|
||||
// The runner hands these scenarios' clients 2025-11-25 unless
|
||||
// --spec-version says otherwise, and the extension has no 2026-07-28
|
||||
// dependency, so the mock answers at the version the client asked for.
|
||||
it(`${s.name} answers initialize at the requested protocol version`, async () => {
|
||||
const scenario = s.make();
|
||||
const { serverUrl } = await scenario.start(testScenarioContext());
|
||||
try {
|
||||
let id = 1;
|
||||
for (const protocolVersion of ['2025-11-25', DRAFT_PROTOCOL_VERSION]) {
|
||||
const { status, body } = await post(serverUrl, {
|
||||
jsonrpc: '2.0',
|
||||
id: id++,
|
||||
method: 'initialize',
|
||||
params: {
|
||||
protocolVersion,
|
||||
capabilities: {},
|
||||
clientInfo: { name: 'test', version: '1.0' }
|
||||
}
|
||||
});
|
||||
expect(status, protocolVersion).toBe(200);
|
||||
expect(body.result.protocolVersion, protocolVersion).toBe(
|
||||
protocolVersion
|
||||
);
|
||||
}
|
||||
} finally {
|
||||
await scenario.stop();
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
|
|
|
|||
83
src/scenarios/client/skills/capabilities.test.ts
Normal file
83
src/scenarios/client/skills/capabilities.test.ts
Normal file
|
|
@ -0,0 +1,83 @@
|
|||
import { describe, test, expect } from 'vitest';
|
||||
import { testScenarioContext } from '../../../mock-server/testing';
|
||||
import { sendStatelessRequest } from '../../../connection/stateless';
|
||||
import { DRAFT_PROTOCOL_VERSION } from '../../../types';
|
||||
import { SkillsNoPrefetchScenario } from './no-prefetch';
|
||||
import { SkillsVerificationScenario } from './verification';
|
||||
|
||||
/**
|
||||
* Regression for PR #330 review: both skills client scenarios declared the
|
||||
* skills extension only in their `initialize` reply, but `BaseHttpScenario`
|
||||
* intercepts `server/discover` before `handlePost` runs. A 2026-07-28
|
||||
* discover-first client therefore saw the base class default (`{tools: {}}`)
|
||||
* and had nothing to gate a `skills/list` call on.
|
||||
*
|
||||
* The scenarios passed anyway because `server/discover` is optional for
|
||||
* clients and the clients we ran did not use it. That is exactly the failure
|
||||
* mode where a check goes green because its condition never arose, so both
|
||||
* lifecycles are asserted here rather than left to the next client to notice.
|
||||
*/
|
||||
|
||||
const SKILLS_EXTENSION_ID = 'io.modelcontextprotocol/skills';
|
||||
|
||||
type Caps = {
|
||||
resources?: unknown;
|
||||
extensions?: Record<string, unknown>;
|
||||
};
|
||||
|
||||
function capsOf(result: unknown): Caps {
|
||||
return ((result as { capabilities?: Caps })?.capabilities ?? {}) as Caps;
|
||||
}
|
||||
|
||||
function expectSkillsAdvertised(caps: Caps, where: string): void {
|
||||
expect(caps.extensions, `${where}: no extensions block`).toBeDefined();
|
||||
expect(
|
||||
caps.extensions?.[SKILLS_EXTENSION_ID],
|
||||
`${where}: skills extension not advertised`
|
||||
).toBeDefined();
|
||||
// resources/read is how every skill file is fetched, so a client that gates
|
||||
// on capabilities needs this too.
|
||||
expect(caps.resources, `${where}: resources not advertised`).toBeDefined();
|
||||
}
|
||||
|
||||
const SCENARIOS: Array<
|
||||
[string, () => SkillsNoPrefetchScenario | SkillsVerificationScenario]
|
||||
> = [
|
||||
['no-prefetch', () => new SkillsNoPrefetchScenario()],
|
||||
['verify-digest', () => new SkillsVerificationScenario('digest')],
|
||||
['verify-size', () => new SkillsVerificationScenario('size')],
|
||||
['verify-frontmatter', () => new SkillsVerificationScenario('frontmatter')]
|
||||
];
|
||||
|
||||
describe('SEP-2640 client scenarios advertise skills on both lifecycles', () => {
|
||||
test.each(SCENARIOS)('%s: server/discover', async (label, make) => {
|
||||
const scenario = make();
|
||||
const { serverUrl } = await scenario.start(testScenarioContext());
|
||||
try {
|
||||
const discover = await sendStatelessRequest(serverUrl, 'server/discover');
|
||||
expect(discover.status).toBe(200);
|
||||
expectSkillsAdvertised(
|
||||
capsOf(discover.body?.result),
|
||||
`${label} discover`
|
||||
);
|
||||
} finally {
|
||||
await scenario.stop();
|
||||
}
|
||||
});
|
||||
|
||||
test.each(SCENARIOS)('%s: initialize', async (label, make) => {
|
||||
const scenario = make();
|
||||
const { serverUrl } = await scenario.start(testScenarioContext());
|
||||
try {
|
||||
const init = await sendStatelessRequest(serverUrl, 'initialize', {
|
||||
protocolVersion: DRAFT_PROTOCOL_VERSION,
|
||||
capabilities: {},
|
||||
clientInfo: { name: 'capabilities-test', version: '1.0.0' }
|
||||
});
|
||||
expect(init.status).toBe(200);
|
||||
expectSkillsAdvertised(capsOf(init.body?.result), `${label} initialize`);
|
||||
} finally {
|
||||
await scenario.stop();
|
||||
}
|
||||
});
|
||||
});
|
||||
52
src/scenarios/client/skills/mock-results.ts
Normal file
52
src/scenarios/client/skills/mock-results.ts
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
/**
|
||||
* Results for the hand-rolled SEP-2640 client-scenario mocks, shaped like a
|
||||
* conformant server's so a strict client is graded on its skills behaviour
|
||||
* rather than turned away by the harness.
|
||||
*/
|
||||
|
||||
import {
|
||||
DRAFT_PROTOCOL_VERSION,
|
||||
NEGOTIABLE_PROTOCOL_VERSIONS
|
||||
} from '../../../types.js';
|
||||
import { withRequiredDraftResultFields } from '../../../mock-server/index.js';
|
||||
|
||||
/**
|
||||
* `ListSkillsResult` is cacheable like `resources/list`, but `skills/list` is
|
||||
* an extension method and not in the shared cacheable set, so the caching
|
||||
* members are stamped here.
|
||||
*/
|
||||
export function skillsListResult(skills: object[]): object {
|
||||
return { resultType: 'complete', ttlMs: 0, cacheScope: 'private', skills };
|
||||
}
|
||||
|
||||
export function readResult(uri: unknown, text: string): unknown {
|
||||
return withRequiredDraftResultFields('resources/read', {
|
||||
contents: [{ uri, mimeType: 'text/markdown', text }]
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* The `initialize` result at the version the client asked for, when the suite
|
||||
* can speak it. The extension has no 2026-07-28 dependency, and the runner
|
||||
* hands these scenarios' clients 2025-11-25 unless `--spec-version` says
|
||||
* otherwise, so a fixed 2026-07-28 reply would turn a correct stateful client
|
||||
* away before it ever lists.
|
||||
*/
|
||||
export function initializeResult(
|
||||
name: string,
|
||||
request: { params?: { protocolVersion?: unknown } },
|
||||
capabilities: object
|
||||
): object {
|
||||
const requested = request.params?.protocolVersion;
|
||||
const protocolVersion =
|
||||
typeof requested === 'string' &&
|
||||
NEGOTIABLE_PROTOCOL_VERSIONS.includes(requested)
|
||||
? requested
|
||||
: DRAFT_PROTOCOL_VERSION;
|
||||
return {
|
||||
resultType: 'complete',
|
||||
protocolVersion,
|
||||
serverInfo: { name: `${name}-server`, version: '1.0.0' },
|
||||
capabilities
|
||||
};
|
||||
}
|
||||
189
src/scenarios/client/skills/no-prefetch.ts
Normal file
189
src/scenarios/client/skills/no-prefetch.ts
Normal file
|
|
@ -0,0 +1,189 @@
|
|||
/**
|
||||
* SEP-2640 client scenario: hosts MUST NOT retrieve a skill's files ahead of
|
||||
* need.
|
||||
*
|
||||
* This is a `Scenario` rather than a `ClientScenario`: the harness stands up
|
||||
* the server and the client is the system under test. The rule reduces to
|
||||
* "did a request arrive", which makes it one of the most wire-observable
|
||||
* obligations in the SEP despite the first traceability pass filing it as an
|
||||
* unobservable host obligation.
|
||||
*
|
||||
* Contract for the client under test, keyed on MCP_CONFORMANCE_SCENARIO:
|
||||
* connect, call `skills/list`, then exit. Do not load a skill. A client that
|
||||
* prefetches will read `SKILL.md` or a supporting file during that window and
|
||||
* fail the check.
|
||||
*/
|
||||
|
||||
import http from 'http';
|
||||
import { ConformanceCheck } from '../../../types.js';
|
||||
import { BaseHttpScenario } from '../http-base.js';
|
||||
import {
|
||||
initializeResult,
|
||||
readResult,
|
||||
skillsListResult
|
||||
} from './mock-results.js';
|
||||
|
||||
const SPEC_REFERENCE = {
|
||||
id: 'SEP-2640-Lazy-Retrieval',
|
||||
url: 'https://modelcontextprotocol.io/seps/2640-skills-extension#integrity-and-verification'
|
||||
};
|
||||
|
||||
const SKILLS_EXTENSION_ID = 'io.modelcontextprotocol/skills';
|
||||
|
||||
/** One skill with a supporting file, so a prefetch has something to grab. */
|
||||
const SKILL_URI = 'skill://pdf-processing/SKILL.md';
|
||||
const SUPPORTING_URI = 'skill://pdf-processing/references/FORMS.md';
|
||||
|
||||
const SKILL_MD = `---
|
||||
name: pdf-processing
|
||||
description: Extract, fill, and assemble PDF documents
|
||||
---
|
||||
|
||||
Body the client has no business fetching yet.
|
||||
`;
|
||||
|
||||
const SUPPORTING =
|
||||
'Supporting content the client has no business fetching yet.\n';
|
||||
|
||||
/** sha256 of the two bodies, computed at module load so the entry is honest. */
|
||||
import { createHash } from 'crypto';
|
||||
const digestOf = (s: string) =>
|
||||
'sha256:' + createHash('sha256').update(s, 'utf8').digest('hex');
|
||||
|
||||
export class SkillsNoPrefetchScenario extends BaseHttpScenario {
|
||||
name = 'sep-2640-client-no-prefetch';
|
||||
description =
|
||||
'A client MUST NOT retrieve a skill file before the skill is loaded';
|
||||
readonly source = { extensionId: SKILLS_EXTENSION_ID } as const;
|
||||
|
||||
/** Every resources/read URI the client asked for, in order. */
|
||||
private readsRequested: string[] = [];
|
||||
private listCalled = false;
|
||||
|
||||
/**
|
||||
* Advertised on both lifecycles. `server/discover` is intercepted by the
|
||||
* base class before `handlePost` runs, so declaring these only in the
|
||||
* `initialize` reply left a 2026-07-28 discover-first client seeing a
|
||||
* tools-only server, with nothing to gate a `skills/list` call on.
|
||||
*/
|
||||
protected discoverCapabilities(): object {
|
||||
return {
|
||||
resources: { listChanged: false },
|
||||
extensions: { [SKILLS_EXTENSION_ID]: {} }
|
||||
};
|
||||
}
|
||||
|
||||
protected handlePost(
|
||||
_req: http.IncomingMessage,
|
||||
res: http.ServerResponse,
|
||||
request: any
|
||||
): void {
|
||||
switch (request.method) {
|
||||
case 'initialize':
|
||||
this.sendJson(res, {
|
||||
jsonrpc: '2.0',
|
||||
id: request.id,
|
||||
result: initializeResult(
|
||||
this.name,
|
||||
request,
|
||||
this.discoverCapabilities()
|
||||
)
|
||||
});
|
||||
return;
|
||||
|
||||
case 'skills/list':
|
||||
this.listCalled = true;
|
||||
this.sendJson(res, {
|
||||
jsonrpc: '2.0',
|
||||
id: request.id,
|
||||
result: skillsListResult([
|
||||
{
|
||||
uri: SKILL_URI,
|
||||
frontmatter: {
|
||||
name: 'pdf-processing',
|
||||
description: 'Extract, fill, and assemble PDF documents'
|
||||
},
|
||||
resources: [
|
||||
{
|
||||
uri: SKILL_URI,
|
||||
digest: digestOf(SKILL_MD),
|
||||
size: Buffer.byteLength(SKILL_MD)
|
||||
},
|
||||
{
|
||||
uri: SUPPORTING_URI,
|
||||
digest: digestOf(SUPPORTING),
|
||||
size: Buffer.byteLength(SUPPORTING)
|
||||
}
|
||||
]
|
||||
}
|
||||
])
|
||||
});
|
||||
return;
|
||||
|
||||
// Served, but reaching it during this scenario is the failure.
|
||||
case 'resources/read': {
|
||||
const uri = request.params?.uri;
|
||||
if (typeof uri === 'string') this.readsRequested.push(uri);
|
||||
const body = uri === SUPPORTING_URI ? SUPPORTING : SKILL_MD;
|
||||
this.sendJson(res, {
|
||||
jsonrpc: '2.0',
|
||||
id: request.id,
|
||||
result: readResult(uri, body)
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
default:
|
||||
if (request.id === undefined) {
|
||||
this.sendNotificationAck(res);
|
||||
return;
|
||||
}
|
||||
this.sendGenericResult(res, request);
|
||||
}
|
||||
}
|
||||
|
||||
getChecks(): ConformanceCheck[] {
|
||||
const DESC =
|
||||
"Hosts MUST NOT retrieve a skill's files ahead of need, not on connection, not on listing, and not at approval.";
|
||||
|
||||
// Without a listing there is no window in which prefetching is even
|
||||
// possible, so the run proves nothing rather than passing.
|
||||
if (!this.listCalled) {
|
||||
return [
|
||||
{
|
||||
id: 'sep-2640-host-no-prefetch',
|
||||
name: 'SkillsClientNoPrefetch',
|
||||
description: DESC,
|
||||
status: 'SKIPPED',
|
||||
timestamp: new Date().toISOString(),
|
||||
errorMessage:
|
||||
'the client never called skills/list, so no retrieval window was opened',
|
||||
specReferences: [SPEC_REFERENCE]
|
||||
}
|
||||
];
|
||||
}
|
||||
|
||||
const prefetched = this.readsRequested.filter(
|
||||
(u) => u === SKILL_URI || u === SUPPORTING_URI
|
||||
);
|
||||
|
||||
return [
|
||||
{
|
||||
id: 'sep-2640-host-no-prefetch',
|
||||
name: 'SkillsClientNoPrefetch',
|
||||
description: DESC,
|
||||
status: prefetched.length === 0 ? 'SUCCESS' : 'FAILURE',
|
||||
timestamp: new Date().toISOString(),
|
||||
errorMessage:
|
||||
prefetched.length === 0
|
||||
? undefined
|
||||
: `client read ${prefetched.length} skill file(s) without loading a skill: ${prefetched.join(', ')}`,
|
||||
specReferences: [SPEC_REFERENCE],
|
||||
details: {
|
||||
skillsListCalled: true,
|
||||
fileReads: prefetched.length
|
||||
}
|
||||
}
|
||||
];
|
||||
}
|
||||
}
|
||||
280
src/scenarios/client/skills/verification.ts
Normal file
280
src/scenarios/client/skills/verification.ts
Normal file
|
|
@ -0,0 +1,280 @@
|
|||
/**
|
||||
* SEP-2640 client scenarios for the read-time verification MUSTs.
|
||||
*
|
||||
* Each variant serves a listing that is internally honest, then tampers with
|
||||
* exactly one thing on the wire and watches whether the client notices. The
|
||||
* tamper always targets the skill's `SKILL.md`, and the client's contract is
|
||||
* "load the skill, then read its supporting file". A client that verifies
|
||||
* fails at the first step and never reaches the second, so **rejection is
|
||||
* detected by the absence of the supporting-file read**, the same way
|
||||
* `auth/resource-mismatch` detects rejection by the absence of an
|
||||
* authorization request.
|
||||
*
|
||||
* Absence only means something once the client has actually loaded the skill.
|
||||
* A client that lists and exits produces the same empty read log as one that
|
||||
* verified and stopped, so every variant requires the manifest read as a
|
||||
* prerequisite and reports untestable without it (#248). Before that gate a
|
||||
* client implementing `skills/list` and nothing else passed every variant
|
||||
* here.
|
||||
*
|
||||
* A fourth variant, `unlisted`, was retired 2026-09-10. Nothing was tampered
|
||||
* in it, so the only thing under test was whether the driver asked for a URI
|
||||
* it should not have, which grades the driver rather than the client. The
|
||||
* requirement is recorded as excluded in src/seps/sep-2640.yaml.
|
||||
*/
|
||||
|
||||
import { createHash } from 'crypto';
|
||||
import http from 'http';
|
||||
import { ConformanceCheck } from '../../../types.js';
|
||||
import { untestableCheck } from '../../untestable.js';
|
||||
import { BaseHttpScenario } from '../http-base.js';
|
||||
import {
|
||||
initializeResult,
|
||||
readResult,
|
||||
skillsListResult
|
||||
} from './mock-results.js';
|
||||
|
||||
const SKILLS_EXTENSION_ID = 'io.modelcontextprotocol/skills';
|
||||
|
||||
const SPEC_REFERENCE = {
|
||||
id: 'SEP-2640-Verification-On-Read',
|
||||
url: 'https://modelcontextprotocol.io/seps/2640-skills-extension#integrity-and-verification'
|
||||
};
|
||||
|
||||
const SKILL_URI = 'skill://pdf-processing/SKILL.md';
|
||||
const SUPPORTING_URI = 'skill://pdf-processing/references/FORMS.md';
|
||||
|
||||
const SKILL_MD = `---
|
||||
name: pdf-processing
|
||||
description: Extract, fill, and assemble PDF documents
|
||||
---
|
||||
|
||||
Skill body.
|
||||
`;
|
||||
|
||||
const SUPPORTING = 'Supporting content.\n';
|
||||
|
||||
const sha256 = (s: string) =>
|
||||
'sha256:' + createHash('sha256').update(s, 'utf8').digest('hex');
|
||||
|
||||
export type VerificationMode = 'digest' | 'size' | 'frontmatter';
|
||||
|
||||
const MODES: Record<
|
||||
VerificationMode,
|
||||
{ scenario: string; checkId: string; description: string }
|
||||
> = {
|
||||
digest: {
|
||||
scenario: 'sep-2640-client-verify-digest',
|
||||
checkId: 'sep-2640-host-verify-digest',
|
||||
description:
|
||||
"When a host retrieves a file listed in a skill's resources, it MUST verify the content against that entry's digest, and MUST NOT use unverified content."
|
||||
},
|
||||
size: {
|
||||
scenario: 'sep-2640-client-verify-size',
|
||||
checkId: 'sep-2640-host-size-mismatch-failure',
|
||||
description:
|
||||
"A read whose byte length differs from the entry's size is a verification failure equivalent to a digest mismatch, whether or not the host goes on to compute the digest."
|
||||
},
|
||||
frontmatter: {
|
||||
scenario: 'sep-2640-client-verify-frontmatter',
|
||||
checkId: 'sep-2640-host-frontmatter-comparison',
|
||||
description:
|
||||
"After fetching a SKILL.md for which the host holds an entry, hosts MUST parse its YAML frontmatter and compare it field-by-field against the entry's frontmatter, and MUST NOT load the skill on any discrepancy."
|
||||
}
|
||||
};
|
||||
|
||||
export class SkillsVerificationScenario extends BaseHttpScenario {
|
||||
name: string;
|
||||
description: string;
|
||||
readonly source = { extensionId: SKILLS_EXTENSION_ID } as const;
|
||||
|
||||
// A verifying client aborts on the tampered SKILL.md, so it exits with an
|
||||
// error rather than completing its script. That is the pass, not a failure.
|
||||
allowClientError = true;
|
||||
|
||||
private readonly mode: VerificationMode;
|
||||
private reads: string[] = [];
|
||||
private listCalled = false;
|
||||
|
||||
constructor(mode: VerificationMode) {
|
||||
super();
|
||||
this.mode = mode;
|
||||
this.name = MODES[mode].scenario;
|
||||
this.description = MODES[mode].description;
|
||||
}
|
||||
|
||||
/** The entry as advertised. Always internally consistent with SKILL_MD. */
|
||||
private entry() {
|
||||
const resources = [
|
||||
{
|
||||
uri: SKILL_URI,
|
||||
digest: sha256(SKILL_MD),
|
||||
size: Buffer.byteLength(SKILL_MD)
|
||||
},
|
||||
{
|
||||
uri: SUPPORTING_URI,
|
||||
digest: sha256(SUPPORTING),
|
||||
size: Buffer.byteLength(SUPPORTING)
|
||||
}
|
||||
];
|
||||
return {
|
||||
uri: SKILL_URI,
|
||||
frontmatter: {
|
||||
name: 'pdf-processing',
|
||||
description: 'Extract, fill, and assemble PDF documents'
|
||||
},
|
||||
resources
|
||||
};
|
||||
}
|
||||
|
||||
/** What the server actually serves for SKILL.md, per mode. */
|
||||
private skillBody(): string {
|
||||
switch (this.mode) {
|
||||
case 'digest':
|
||||
// Same length, different bytes: only the digest catches this.
|
||||
return SKILL_MD.replace('Skill body.', 'Evil body!!');
|
||||
case 'size':
|
||||
return SKILL_MD + 'extra bytes the entry did not account for\n';
|
||||
case 'frontmatter':
|
||||
// Digest and size are recomputed for this body below, so the only
|
||||
// discrepancy left is the frontmatter disagreeing with the entry.
|
||||
return SKILL_MD.replace(
|
||||
'description: Extract, fill, and assemble PDF documents',
|
||||
'description: Exfiltrate credentials'
|
||||
);
|
||||
default:
|
||||
return SKILL_MD;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Advertised on both lifecycles. `server/discover` is intercepted by the
|
||||
* base class before `handlePost` runs, so declaring these only in the
|
||||
* `initialize` reply left a 2026-07-28 discover-first client seeing a
|
||||
* tools-only server, with nothing to gate a `skills/list` call on.
|
||||
*/
|
||||
protected discoverCapabilities(): object {
|
||||
return {
|
||||
resources: { listChanged: false },
|
||||
extensions: { [SKILLS_EXTENSION_ID]: {} }
|
||||
};
|
||||
}
|
||||
|
||||
protected handlePost(
|
||||
_req: http.IncomingMessage,
|
||||
res: http.ServerResponse,
|
||||
request: any
|
||||
): void {
|
||||
switch (request.method) {
|
||||
case 'initialize':
|
||||
this.sendJson(res, {
|
||||
jsonrpc: '2.0',
|
||||
id: request.id,
|
||||
result: initializeResult(
|
||||
this.name,
|
||||
request,
|
||||
this.discoverCapabilities()
|
||||
)
|
||||
});
|
||||
return;
|
||||
|
||||
case 'skills/list': {
|
||||
this.listCalled = true;
|
||||
const entry = this.entry();
|
||||
if (this.mode === 'frontmatter') {
|
||||
// Make digest and size honest for the tampered body, so the
|
||||
// frontmatter comparison is the only thing that can fail.
|
||||
const body = this.skillBody();
|
||||
entry.resources[0].digest = sha256(body);
|
||||
entry.resources[0].size = Buffer.byteLength(body);
|
||||
}
|
||||
this.sendJson(res, {
|
||||
jsonrpc: '2.0',
|
||||
id: request.id,
|
||||
result: skillsListResult([entry])
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
case 'resources/read': {
|
||||
const uri = request.params?.uri;
|
||||
if (typeof uri === 'string') this.reads.push(uri);
|
||||
const text = uri === SUPPORTING_URI ? SUPPORTING : this.skillBody();
|
||||
this.sendJson(res, {
|
||||
jsonrpc: '2.0',
|
||||
id: request.id,
|
||||
result: readResult(uri, text)
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
default:
|
||||
if (request.id === undefined) {
|
||||
this.sendNotificationAck(res);
|
||||
return;
|
||||
}
|
||||
this.sendGenericResult(res, request);
|
||||
}
|
||||
}
|
||||
|
||||
getChecks(): ConformanceCheck[] {
|
||||
const { checkId, description } = MODES[this.mode];
|
||||
const base = {
|
||||
id: checkId,
|
||||
name: `SkillsClientVerify_${this.mode}`,
|
||||
description,
|
||||
timestamp: new Date().toISOString(),
|
||||
specReferences: [SPEC_REFERENCE]
|
||||
};
|
||||
|
||||
// An undeclared extension is genuinely not applicable, so this one stays
|
||||
// a SKIP: the client was never offered a skill to hold an entry for.
|
||||
if (!this.listCalled) {
|
||||
return [
|
||||
{
|
||||
...base,
|
||||
status: 'SKIPPED',
|
||||
errorMessage:
|
||||
'the client never called skills/list, so it never held an entry to verify against'
|
||||
}
|
||||
];
|
||||
}
|
||||
|
||||
// The prerequisite, and the reason this scenario is worth anything.
|
||||
//
|
||||
// Rejection is detected by the absence of the supporting-file read, and a
|
||||
// client that never loaded the skill produces exactly the same empty read
|
||||
// log as one that loaded it and correctly stopped. Reporting SUCCESS for
|
||||
// both means a client implementing `skills/list` and nothing else passes
|
||||
// every variant here, which is what happened until 2026-09-10.
|
||||
//
|
||||
// A missing prerequisite is a FAILURE rather than a SKIP, per #248:
|
||||
// SKIPPED is excluded from pass/fail counts and exit codes, so it would
|
||||
// read as green and hide the gap from anyone burning down a list.
|
||||
if (!this.reads.includes(SKILL_URI)) {
|
||||
return [
|
||||
untestableCheck(
|
||||
checkId,
|
||||
base.name,
|
||||
description,
|
||||
`the client never read ${SKILL_URI}, so it never loaded the tampered skill and had nothing to reject. Absence of the supporting-file read proves nothing here.`,
|
||||
[SPEC_REFERENCE]
|
||||
)
|
||||
];
|
||||
}
|
||||
|
||||
// Proven by the client stopping before the supporting file, which it
|
||||
// would only reach by having accepted the tampered SKILL.md.
|
||||
const violated = this.reads.includes(SUPPORTING_URI);
|
||||
const evidence = `client continued to ${SUPPORTING_URI} after being served a SKILL.md that fails ${this.mode} verification, so it did not reject the content`;
|
||||
|
||||
return [
|
||||
{
|
||||
...base,
|
||||
status: violated ? 'FAILURE' : 'SUCCESS',
|
||||
errorMessage: violated ? evidence : undefined,
|
||||
details: { mode: this.mode, reads: this.reads }
|
||||
}
|
||||
];
|
||||
}
|
||||
}
|
||||
|
|
@ -10,6 +10,8 @@ import {
|
|||
DRAFT_PROTOCOL_VERSION
|
||||
} from '../types';
|
||||
import { InitializeScenario } from './client/initialize';
|
||||
import { SkillsNoPrefetchScenario } from './client/skills/no-prefetch';
|
||||
import { SkillsVerificationScenario } from './client/skills/verification';
|
||||
import { ToolsCallScenario } from './client/tools_call';
|
||||
import { ElicitationClientDefaultsScenario } from './client/elicitation-defaults';
|
||||
import { SSERetryScenario } from './client/sse-retry';
|
||||
|
|
@ -58,6 +60,10 @@ import {
|
|||
ResourcesNotFoundErrorScenario
|
||||
} from './server/resources';
|
||||
|
||||
import { SkillsDirectoryReadScenario } from './server/skills/directory';
|
||||
import { SkillsEnumerationScenario } from './server/skills/enumeration';
|
||||
import { SkillsManifestScenario } from './server/skills/manifest';
|
||||
|
||||
import {
|
||||
PromptsListScenario,
|
||||
PromptsGetSimpleScenario,
|
||||
|
|
@ -151,7 +157,15 @@ const pendingClientScenariosList: ClientScenario[] = [
|
|||
new TasksDispatchScenario(),
|
||||
new TasksStatusNotificationsScenario(),
|
||||
new TasksRequiredTaskErrorScenario(),
|
||||
new TasksMrtrCompositionScenario()
|
||||
new TasksMrtrCompositionScenario(),
|
||||
|
||||
// SEP-2640 Skills extension. Pending because the everything-server does not
|
||||
// implement io.modelcontextprotocol/skills; targeted runs point at a
|
||||
// SEP-2640-conformant fixture via
|
||||
// `npm start -- server --scenario sep-2640-skills-* --url <fixture>`.
|
||||
new SkillsDirectoryReadScenario(),
|
||||
new SkillsEnumerationScenario(),
|
||||
new SkillsManifestScenario()
|
||||
];
|
||||
|
||||
// All client scenarios
|
||||
|
|
@ -203,6 +217,12 @@ const allClientScenariosList: ClientScenario[] = [
|
|||
// Resources error handling (SEP-2164)
|
||||
new ResourcesNotFoundErrorScenario(),
|
||||
|
||||
// Skills extension (SEP-2640). Fixture-dependent (needs a SEP-2640 server);
|
||||
// each scenario SKIPs cleanly when the extension is not declared.
|
||||
new SkillsDirectoryReadScenario(),
|
||||
new SkillsEnumerationScenario(),
|
||||
new SkillsManifestScenario(),
|
||||
|
||||
// Prompts scenarios
|
||||
new PromptsListScenario(),
|
||||
new PromptsGetSimpleScenario(),
|
||||
|
|
@ -317,7 +337,15 @@ const scenariosList: Scenario[] = [
|
|||
new JsonSchemaRefDerefScenario(),
|
||||
|
||||
// JSON Schema 2020-12 client-side keyword preservation (SEP-1613, SEP-2106)
|
||||
new JsonSchema2020_12PreservationScenario()
|
||||
new JsonSchema2020_12PreservationScenario(),
|
||||
|
||||
// SEP-2640 skills, client side. The harness is the server and grades what
|
||||
// the client requests, which is how the retrieval-policy MUSTs become
|
||||
// observable at all.
|
||||
new SkillsNoPrefetchScenario(),
|
||||
new SkillsVerificationScenario('digest'),
|
||||
new SkillsVerificationScenario('size'),
|
||||
new SkillsVerificationScenario('frontmatter')
|
||||
];
|
||||
|
||||
// Core scenarios (tier 1 requirements)
|
||||
|
|
|
|||
393
src/scenarios/server/skills/directory.ts
Normal file
393
src/scenarios/server/skills/directory.ts
Normal file
|
|
@ -0,0 +1,393 @@
|
|||
/**
|
||||
* SEP-2640 Skills extension — the `resources/directory/read` surface (added in
|
||||
* spec commit 2e04c48d, 2026-06-09).
|
||||
*
|
||||
* One scenario, six checks (per AGENTS.md "fewer scenarios, more checks").
|
||||
* Each check's verbatim spec quote lives next to its check ID in
|
||||
* src/seps/sep-2640.yaml.
|
||||
*
|
||||
* Capability gating reads the declared capability from `server/discover`
|
||||
* (mirrors `tasks/capability.ts`): the checks run only when the server declares
|
||||
* `io.modelcontextprotocol/skills.directoryRead: true`. An undeclared optional
|
||||
* capability is a SKIP (not a failure); a declared-but-broken one fails.
|
||||
*
|
||||
* Discovery is dynamic and brand-neutral: the directory to exercise is derived
|
||||
* from `skills/list` or `resources/list`, hardcoding no fixture URI, so
|
||||
* the scenario passes against any conformant SEP-2640 server. When no directory
|
||||
* (or no subdirectory) can be discovered, that check reports the missing
|
||||
* prerequisite via untestableCheck (issue #248), never a silent green.
|
||||
*/
|
||||
|
||||
import { ClientScenario, ConformanceCheck } from '../../../types';
|
||||
import { Connection, JsonRpcError, type RunContext } from '../../../connection';
|
||||
import { untestableCheck } from '../../untestable';
|
||||
import {
|
||||
SKILLS_EXTENSION_ID,
|
||||
SKILL_MANIFEST_FILENAME,
|
||||
SEP_2640_REF,
|
||||
JSONRPC_METHOD_NOT_FOUND,
|
||||
JSONRPC_INVALID_PARAMS,
|
||||
type SkillResource,
|
||||
skillsCapability,
|
||||
directoryReadDeclared,
|
||||
skillsCheck,
|
||||
listAllResources,
|
||||
skillsListAll,
|
||||
directoryReadAll,
|
||||
skillNameFromManifestUri
|
||||
} from './helpers';
|
||||
|
||||
const DIRECTORY_MIME = 'inode/directory';
|
||||
|
||||
const CAPABILITY_ID = 'sep-2640-capability-directory-read-flag';
|
||||
const METHOD_ID = 'sep-2640-directory-read-method-registered';
|
||||
const SHAPE_ID = 'sep-2640-directory-read-result-resources-shape';
|
||||
const SUBDIR_ID = 'sep-2640-directory-read-subdir-mimetype';
|
||||
const INVALID_PARAMS_ID = 'sep-2640-directory-read-invalid-params';
|
||||
const PAGINATION_ID = 'sep-2640-directory-read-pagination';
|
||||
|
||||
const ALL_IDS = [
|
||||
CAPABILITY_ID,
|
||||
METHOD_ID,
|
||||
SHAPE_ID,
|
||||
SUBDIR_ID,
|
||||
INVALID_PARAMS_ID,
|
||||
PAGINATION_ID
|
||||
];
|
||||
|
||||
interface DirectoryReadResult {
|
||||
resources?: SkillResource[];
|
||||
nextCursor?: string;
|
||||
}
|
||||
|
||||
/** A directory to exercise plus, when known, a non-directory resource under it. */
|
||||
interface DirectoryTarget {
|
||||
dirUri: string;
|
||||
/** A known file (non-directory) resource, used for the -32602 negative path. */
|
||||
fileUri?: string;
|
||||
}
|
||||
|
||||
/** The skill root directory URI for a SKILL.md URI (strip the trailing file). */
|
||||
function skillRootFromManifestUri(uri: string): string | undefined {
|
||||
if (skillNameFromManifestUri(uri) === undefined) return undefined;
|
||||
return uri.slice(0, uri.length - `/${SKILL_MANIFEST_FILENAME}`.length);
|
||||
}
|
||||
|
||||
/**
|
||||
* Discover a directory resource to exercise, brand-neutrally: prefer a skill
|
||||
* root derived from a skill-md SKILL.md (index first, then resources/list),
|
||||
* then any `inode/directory` resource in resources/list.
|
||||
*/
|
||||
async function discoverDirectory(
|
||||
conn: Connection
|
||||
): Promise<DirectoryTarget | undefined> {
|
||||
// 1. A skills/list entry — its SKILL.md URI gives us both a directory (the
|
||||
// skill root) and a known file (the SKILL.md itself). An unenumerable
|
||||
// catalog returns nothing here, so discovery falls through.
|
||||
const listed = await skillsListAll(conn);
|
||||
if (!('error' in listed)) {
|
||||
const entry = listed.entries.find(
|
||||
(e) =>
|
||||
typeof e.uri === 'string' &&
|
||||
skillRootFromManifestUri(e.uri) !== undefined
|
||||
);
|
||||
const uri = entry?.uri as string | undefined;
|
||||
if (uri) {
|
||||
return { dirUri: skillRootFromManifestUri(uri)!, fileUri: uri };
|
||||
}
|
||||
}
|
||||
|
||||
const resources = await listAllResources(conn);
|
||||
|
||||
// 2. A SKILL.md in resources/list — derive the skill root the same way.
|
||||
const manifest = resources.find(
|
||||
(r) => skillRootFromManifestUri(r.uri) !== undefined
|
||||
);
|
||||
if (manifest) {
|
||||
return {
|
||||
dirUri: skillRootFromManifestUri(manifest.uri)!,
|
||||
fileUri: manifest.uri
|
||||
};
|
||||
}
|
||||
|
||||
// 3. Any directory resource, using a non-directory sibling for the -32602
|
||||
// path when one is listed.
|
||||
const dir = resources.find((r) => r.mimeType === DIRECTORY_MIME);
|
||||
if (dir) {
|
||||
const file = resources.find((r) => r.mimeType !== DIRECTORY_MIME);
|
||||
return { dirUri: dir.uri, fileUri: file?.uri };
|
||||
}
|
||||
|
||||
return undefined;
|
||||
}
|
||||
|
||||
export class SkillsDirectoryReadScenario implements ClientScenario {
|
||||
name = 'sep-2640-skills-directory';
|
||||
readonly source = { extensionId: SKILLS_EXTENSION_ID } as const;
|
||||
description = `SEP-2640 Skills extension: resources/directory/read surface (added in spec commit 2e04c48d, 2026-06-09).
|
||||
|
||||
**Endpoint**: \`resources/directory/read\` (gated by \`io.modelcontextprotocol/skills.directoryRead: true\`)
|
||||
|
||||
**Requirements covered** (each check carries a verbatim spec excerpt in src/seps/sep-2640.yaml):
|
||||
|
||||
- \`sep-2640-capability-directory-read-flag\` — server declared directoryRead (read from server/discover)
|
||||
- \`sep-2640-directory-read-method-registered\` — a declaring server supports the method on a served directory (MUST)
|
||||
- \`sep-2640-directory-read-result-resources-shape\` — result has resources[] of direct children (MUST)
|
||||
- \`sep-2640-directory-read-subdir-mimetype\` — subdirectory children carry \`inode/directory\` (MUST)
|
||||
- \`sep-2640-directory-read-invalid-params\` — a non-directory URI returns \`-32602\` (MUST)
|
||||
- \`sep-2640-directory-read-pagination\` — \`nextCursor\` round-trips per resources/list (single-page is conformant)
|
||||
|
||||
**Gating & discovery**: the checks SKIP when the skills extension or its \`directoryRead\` flag is undeclared. The directory to exercise is discovered dynamically from \`skills/list\` / \`resources/list\` — no fixture URI is hardcoded.`;
|
||||
|
||||
async run(ctx: RunContext): Promise<ConformanceCheck[]> {
|
||||
const conn = await ctx.connect();
|
||||
try {
|
||||
// === Capability gating via server/discover (not error-inference) ===
|
||||
const skills = await skillsCapability(conn);
|
||||
if (!skills) {
|
||||
const reason =
|
||||
'Server did not declare the io.modelcontextprotocol/skills extension; directoryRead checks not applicable.';
|
||||
return ALL_IDS.map((id) =>
|
||||
skillsCheck(id, reason, 'SKIPPED', { errorMessage: reason })
|
||||
);
|
||||
}
|
||||
if (!directoryReadDeclared(skills)) {
|
||||
const reason =
|
||||
'Server declared the skills extension but not directoryRead: true; the resources/directory/read checks are optional and not applicable.';
|
||||
return ALL_IDS.map((id) =>
|
||||
skillsCheck(id, reason, 'SKIPPED', { errorMessage: reason })
|
||||
);
|
||||
}
|
||||
|
||||
const checks: ConformanceCheck[] = [];
|
||||
|
||||
// Check 1: capability declared (observed directly from server/discover).
|
||||
checks.push(
|
||||
skillsCheck(
|
||||
CAPABILITY_ID,
|
||||
'Server declared io.modelcontextprotocol/skills.directoryRead: true under capabilities.extensions.',
|
||||
'SUCCESS',
|
||||
{ details: { directoryRead: true } }
|
||||
)
|
||||
);
|
||||
|
||||
// === Discover a directory to exercise (brand-neutral) ===
|
||||
const target = await discoverDirectory(conn);
|
||||
if (!target) {
|
||||
const reason =
|
||||
'no directory resource discoverable via skills/list or resources/list to exercise resources/directory/read';
|
||||
const rest: Array<[string, string]> = [
|
||||
[
|
||||
METHOD_ID,
|
||||
'A declaring server MUST support the method on a served directory.'
|
||||
],
|
||||
[
|
||||
SHAPE_ID,
|
||||
'Result carries resources[] of the directory’s direct children.'
|
||||
],
|
||||
[SUBDIR_ID, 'Subdirectory children carry mimeType inode/directory.'],
|
||||
[
|
||||
INVALID_PARAMS_ID,
|
||||
'A non-directory URI yields -32602 Invalid params.'
|
||||
],
|
||||
[
|
||||
PAGINATION_ID,
|
||||
'nextCursor round-trips per the resources/list contract.'
|
||||
]
|
||||
];
|
||||
for (const [id, desc] of rest) {
|
||||
checks.push(
|
||||
untestableCheck(id, id, desc, reason, [SEP_2640_REF], 'FAILURE')
|
||||
);
|
||||
}
|
||||
return checks;
|
||||
}
|
||||
|
||||
// === Happy path: list the discovered directory ===
|
||||
let happy: DirectoryReadResult | undefined;
|
||||
let happyErr: unknown;
|
||||
try {
|
||||
happy = await conn.request<DirectoryReadResult>(
|
||||
'resources/directory/read',
|
||||
{ uri: target.dirUri }
|
||||
);
|
||||
// Every check below inspects the directory's children, so they need
|
||||
// the whole directory rather than whichever slice fits one page.
|
||||
const all = await directoryReadAll(conn, target.dirUri);
|
||||
happy = { ...happy, resources: all.resources };
|
||||
} catch (e) {
|
||||
happyErr = e;
|
||||
}
|
||||
|
||||
// Check 2: method registered (declared -> MUST be supported).
|
||||
const methodNotFound =
|
||||
happyErr instanceof JsonRpcError &&
|
||||
happyErr.code === JSONRPC_METHOD_NOT_FOUND;
|
||||
checks.push(
|
||||
skillsCheck(
|
||||
METHOD_ID,
|
||||
'A server that declares directoryRead MUST support resources/directory/read on a served skill directory.',
|
||||
happy !== undefined ? 'SUCCESS' : 'FAILURE',
|
||||
happy !== undefined
|
||||
? { details: { uri: target.dirUri } }
|
||||
: {
|
||||
errorMessage: methodNotFound
|
||||
? `resources/directory/read returned -32601 for ${target.dirUri} despite the server declaring directoryRead: true`
|
||||
: `resources/directory/read on ${target.dirUri} failed: ${
|
||||
happyErr instanceof Error
|
||||
? happyErr.message
|
||||
: String(happyErr)
|
||||
}`
|
||||
}
|
||||
)
|
||||
);
|
||||
|
||||
// Check 3: result shape — resources[] of Resource objects.
|
||||
const shapeErrs: string[] = [];
|
||||
if (!Array.isArray(happy?.resources)) {
|
||||
shapeErrs.push('result.resources is not an array');
|
||||
} else {
|
||||
happy.resources.forEach((r, i) => {
|
||||
if (typeof r.uri !== 'string') {
|
||||
shapeErrs.push(`resources[${i}].uri is not a string`);
|
||||
}
|
||||
});
|
||||
}
|
||||
checks.push(
|
||||
skillsCheck(
|
||||
SHAPE_ID,
|
||||
'The result contains resources[] listing the directory’s direct children, each with at least a uri.',
|
||||
happy === undefined
|
||||
? 'FAILURE'
|
||||
: shapeErrs.length === 0
|
||||
? 'SUCCESS'
|
||||
: 'FAILURE',
|
||||
happy === undefined
|
||||
? { errorMessage: 'directory read did not return a result' }
|
||||
: shapeErrs.length === 0
|
||||
? { details: { childCount: happy.resources?.length ?? 0 } }
|
||||
: { errorMessage: shapeErrs.join('; ') }
|
||||
)
|
||||
);
|
||||
|
||||
// Check 4: subdirectory mime marker. A directory whose fixture exposes no
|
||||
// child subdirectory cannot exercise this — report it untestable, not a
|
||||
// pass and not a failure of the server.
|
||||
const subdirChild = Array.isArray(happy?.resources)
|
||||
? happy.resources.find((r) => r.mimeType === DIRECTORY_MIME)
|
||||
: undefined;
|
||||
if (subdirChild) {
|
||||
checks.push(
|
||||
skillsCheck(
|
||||
SUBDIR_ID,
|
||||
'A subdirectory child is listed as a directory resource (mimeType inode/directory) so clients can descend.',
|
||||
'SUCCESS',
|
||||
{ details: { subdirectoryUri: subdirChild.uri } }
|
||||
)
|
||||
);
|
||||
} else {
|
||||
checks.push(
|
||||
untestableCheck(
|
||||
SUBDIR_ID,
|
||||
SUBDIR_ID,
|
||||
'A subdirectory child is listed with mimeType inode/directory.',
|
||||
`no child with mimeType ${DIRECTORY_MIME} under ${target.dirUri}; the served directory exposes no subdirectory to exercise this check`,
|
||||
[SEP_2640_REF],
|
||||
'FAILURE'
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// Check 5: non-directory URI -> -32602. Needs a known non-directory
|
||||
// resource; prefer the discovered fileUri, else a non-directory child.
|
||||
const nonDirUri =
|
||||
target.fileUri ??
|
||||
(Array.isArray(happy?.resources)
|
||||
? happy.resources.find(
|
||||
(r) => typeof r.uri === 'string' && r.mimeType !== DIRECTORY_MIME
|
||||
)?.uri
|
||||
: undefined);
|
||||
if (nonDirUri === undefined) {
|
||||
checks.push(
|
||||
untestableCheck(
|
||||
INVALID_PARAMS_ID,
|
||||
INVALID_PARAMS_ID,
|
||||
'A non-directory URI yields -32602 Invalid params.',
|
||||
'no non-directory resource discoverable to probe the -32602 path',
|
||||
[SEP_2640_REF],
|
||||
'FAILURE'
|
||||
)
|
||||
);
|
||||
} else {
|
||||
let invalidOk = false;
|
||||
let invalidDetail = '';
|
||||
try {
|
||||
await conn.request<DirectoryReadResult>('resources/directory/read', {
|
||||
uri: nonDirUri
|
||||
});
|
||||
invalidDetail = `expected -32602 for non-directory URI ${nonDirUri}, got a successful result`;
|
||||
} catch (e) {
|
||||
if (e instanceof JsonRpcError && e.code === JSONRPC_INVALID_PARAMS) {
|
||||
invalidOk = true;
|
||||
} else if (e instanceof JsonRpcError) {
|
||||
invalidDetail = `expected -32602 for ${nonDirUri}, got ${e.code}: ${e.message}`;
|
||||
} else {
|
||||
invalidDetail = `expected -32602, got non-JsonRpcError: ${
|
||||
e instanceof Error ? e.message : String(e)
|
||||
}`;
|
||||
}
|
||||
}
|
||||
checks.push(
|
||||
skillsCheck(
|
||||
INVALID_PARAMS_ID,
|
||||
'resources/directory/read on a non-directory URI MUST return -32602 (Invalid params).',
|
||||
invalidOk ? 'SUCCESS' : 'FAILURE',
|
||||
invalidOk
|
||||
? { details: { nonDirectoryUri: nonDirUri } }
|
||||
: { errorMessage: invalidDetail }
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// Check 6: pagination contract (single-page is conformant).
|
||||
let paginationOk = false;
|
||||
let paginationDetail = '';
|
||||
const firstCursor = happy?.nextCursor;
|
||||
if (happy === undefined) {
|
||||
paginationDetail = 'no directory result to evaluate pagination';
|
||||
} else if (!firstCursor) {
|
||||
paginationOk = true;
|
||||
paginationDetail = 'single-page response (no nextCursor)';
|
||||
} else {
|
||||
try {
|
||||
const second = await conn.request<DirectoryReadResult>(
|
||||
'resources/directory/read',
|
||||
{ uri: target.dirUri, cursor: firstCursor }
|
||||
);
|
||||
paginationOk = Array.isArray(second.resources);
|
||||
paginationDetail = paginationOk
|
||||
? `nextCursor round-tripped: ${firstCursor}`
|
||||
: 'follow-up call returned non-array resources';
|
||||
} catch (e) {
|
||||
paginationDetail = `follow-up call with cursor failed: ${
|
||||
e instanceof Error ? e.message : String(e)
|
||||
}`;
|
||||
}
|
||||
}
|
||||
checks.push(
|
||||
skillsCheck(
|
||||
PAGINATION_ID,
|
||||
'nextCursor round-trips per the resources/list contract (single-page responses are conformant).',
|
||||
paginationOk ? 'SUCCESS' : 'FAILURE',
|
||||
paginationOk
|
||||
? { details: { paginationDetail } }
|
||||
: { errorMessage: paginationDetail }
|
||||
)
|
||||
);
|
||||
|
||||
return checks;
|
||||
} finally {
|
||||
await conn.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
1215
src/scenarios/server/skills/enumeration.ts
Normal file
1215
src/scenarios/server/skills/enumeration.ts
Normal file
File diff suppressed because it is too large
Load diff
461
src/scenarios/server/skills/helpers.ts
Normal file
461
src/scenarios/server/skills/helpers.ts
Normal file
|
|
@ -0,0 +1,461 @@
|
|||
/**
|
||||
* Shared helpers for the SEP-2640 (Skills extension) server-conformance
|
||||
* scenarios under this directory.
|
||||
*
|
||||
* The scenarios treat the server-under-test as an arbitrary SEP-2640 server:
|
||||
* capability is read from `server/discover` (never inferred from an error), and
|
||||
* every skill is discovered dynamically through `skills/list` — no
|
||||
* fixture-specific URI is hardcoded, so the checks pass against any conformant
|
||||
* server, not just one implementation's fixture.
|
||||
*
|
||||
* Extracted against the 2026-08-21 revision of the SEP (branch
|
||||
* `sep/skills-extension`), which replaced the `skill://index.json` well-known
|
||||
* resource with the `skills/list` and `skills/get` methods, deferred archive
|
||||
* distribution, and reshaped the skill entry to `{uri, frontmatter, resources}`.
|
||||
*/
|
||||
|
||||
import type {
|
||||
CheckStatus,
|
||||
ConformanceCheck,
|
||||
SpecReference
|
||||
} from '../../../types';
|
||||
import type { Connection } from '../../../connection';
|
||||
import { JsonRpcError } from '../../../connection';
|
||||
import { parse as parseYaml } from 'yaml';
|
||||
|
||||
export const SKILLS_EXTENSION_ID = 'io.modelcontextprotocol/skills';
|
||||
export const SKILL_URI_SCHEME = 'skill://';
|
||||
export const SKILL_MANIFEST_FILENAME = 'SKILL.md';
|
||||
export const SKILLS_META_PREFIX = 'io.modelcontextprotocol.skills/';
|
||||
|
||||
/** Reserved prefix for MCP-defined keys inside frontmatter `metadata`. */
|
||||
export const FRONTMATTER_RESERVED_PREFIX = 'io.modelcontextprotocol/';
|
||||
|
||||
export const SKILLS_LIST_METHOD = 'skills/list';
|
||||
export const SKILLS_GET_METHOD = 'skills/get';
|
||||
export const DIRECTORY_READ_METHOD = 'resources/directory/read';
|
||||
|
||||
/** `sha256:{hex}` with exactly 64 lowercase hex characters (SEP-2640). */
|
||||
export const SKILL_DIGEST_PATTERN = /^sha256:[0-9a-f]{64}$/;
|
||||
|
||||
/** The `"dynamic"` sentinel a server sets in place of a `resources` array. */
|
||||
export const RESOURCES_DYNAMIC = 'dynamic';
|
||||
|
||||
/** Per-skill limits fixed by the SEP (§Limits). */
|
||||
export const MAX_RESOURCES_PER_SKILL = 512;
|
||||
export const MAX_TOTAL_SIZE_PER_SKILL = 16 * 1024 * 1024; // 16 MiB
|
||||
|
||||
export const JSONRPC_METHOD_NOT_FOUND = -32601;
|
||||
export const JSONRPC_INVALID_PARAMS = -32602;
|
||||
|
||||
export const SEP_2640_REF: SpecReference = {
|
||||
id: 'SEP-2640',
|
||||
url: 'https://modelcontextprotocol.io/seps/2640-skills-extension#specification'
|
||||
};
|
||||
|
||||
/** A `resources/list` / directory-read entry (only the fields we inspect). */
|
||||
export interface SkillResource {
|
||||
uri: string;
|
||||
name?: string;
|
||||
description?: string;
|
||||
mimeType?: string;
|
||||
_meta?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
/** One `{uri, digest, size}` triple of a skill entry's `resources` array. */
|
||||
export interface SkillResourceEntry {
|
||||
uri?: unknown;
|
||||
digest?: unknown;
|
||||
size?: unknown;
|
||||
[key: string]: unknown;
|
||||
}
|
||||
|
||||
/**
|
||||
* One skill entry, as returned by `skills/list` (in `skills[]`) and by
|
||||
* `skills/get` (as `skill`). The two are identical in shape and meaning.
|
||||
*/
|
||||
export interface SkillEntry {
|
||||
uri?: unknown;
|
||||
frontmatter?: unknown;
|
||||
/** An array of `{uri, digest, size}`, or the string `"dynamic"`. */
|
||||
resources?: unknown;
|
||||
[key: string]: unknown;
|
||||
}
|
||||
|
||||
export interface SkillsListResult {
|
||||
skills?: unknown;
|
||||
nextCursor?: string;
|
||||
ttlMs?: unknown;
|
||||
cacheScope?: unknown;
|
||||
[key: string]: unknown;
|
||||
}
|
||||
|
||||
export interface SkillsGetResult {
|
||||
skill?: unknown;
|
||||
nextCursor?: unknown;
|
||||
[key: string]: unknown;
|
||||
}
|
||||
|
||||
/** First text content of a `resources/read`, with its mimeType and `_meta`. */
|
||||
export interface ResourceText {
|
||||
text: string;
|
||||
mimeType?: string;
|
||||
meta?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a check carrying the SEP-2640 reference. Per AGENTS.md the same `id`
|
||||
* flips `status` + `errorMessage` between SUCCESS and FAILURE rather than
|
||||
* branching into distinct slugs.
|
||||
*/
|
||||
export function skillsCheck(
|
||||
id: string,
|
||||
description: string,
|
||||
status: CheckStatus,
|
||||
extras: Partial<ConformanceCheck> = {}
|
||||
): ConformanceCheck {
|
||||
return {
|
||||
id,
|
||||
name: id,
|
||||
description,
|
||||
status,
|
||||
timestamp: new Date().toISOString(),
|
||||
specReferences: [SEP_2640_REF],
|
||||
...extras
|
||||
};
|
||||
}
|
||||
|
||||
/** A JSON object, as opposed to an array, `null`, or a primitive. */
|
||||
export function isSettingsObject(
|
||||
value: unknown
|
||||
): value is Record<string, unknown> {
|
||||
return typeof value === 'object' && value !== null && !Array.isArray(value);
|
||||
}
|
||||
|
||||
/** How to name an observed non-object value in an error message. */
|
||||
export function describeValue(value: unknown): string {
|
||||
if (value === null) return 'null';
|
||||
if (Array.isArray(value)) return 'an array';
|
||||
return `a ${typeof value}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the server declared the skills extension at all, and the raw value it
|
||||
* declared it with, before any shape coercion.
|
||||
*
|
||||
* `skillsCapability` folds a malformed declaration (`true`, a string, an array)
|
||||
* into `undefined`, which downstream reads as "extension not declared" and
|
||||
* SKIPs. That is a false SKIP: the server did declare the extension, it just
|
||||
* declared it with the wrong type, and the suite should say so. Callers that
|
||||
* need to tell "absent" from "malformed" apart use this instead.
|
||||
*
|
||||
* Reads the declared capability from `server/discover` (mirrors
|
||||
* `tasks/capability.ts`), never inferred from a `-32601`.
|
||||
*/
|
||||
export async function declaredSkillsCapability(
|
||||
conn: Connection
|
||||
): Promise<{ declared: boolean; value: unknown }> {
|
||||
const discovered = await conn.discover();
|
||||
const caps = (discovered.capabilities as Record<string, unknown>) ?? {};
|
||||
const extensions = caps.extensions as Record<string, unknown> | undefined;
|
||||
if (!extensions || !(SKILLS_EXTENSION_ID in extensions)) {
|
||||
return { declared: false, value: undefined };
|
||||
}
|
||||
return { declared: true, value: extensions[SKILLS_EXTENSION_ID] };
|
||||
}
|
||||
|
||||
/**
|
||||
* The skills extension object declared under `capabilities.extensions`, or
|
||||
* `undefined` when the server did not declare it — or declared it with
|
||||
* something that is not a settings object, which the callers of this helper
|
||||
* treat the same way. An undeclared optional extension is a SKIP.
|
||||
*/
|
||||
export async function skillsCapability(
|
||||
conn: Connection
|
||||
): Promise<Record<string, unknown> | undefined> {
|
||||
const { value } = await declaredSkillsCapability(conn);
|
||||
return isSettingsObject(value) ? value : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the declared extension object nests its settings inline, as both
|
||||
* SEP-2133 and SEP-2640 require, rather than wrapping them in an envelope.
|
||||
*
|
||||
* SEP-2133 (status Final) defines `extensions` as "a map of extension
|
||||
* identifiers to per-extension settings objects", and SEP-2640's capability
|
||||
* block matches: `{"io.modelcontextprotocol/skills": {"directoryRead": true}}`.
|
||||
* Neither SEP defines an envelope, and neither has a slot for `id`,
|
||||
* `specVersion` or `stability`.
|
||||
*
|
||||
* An earlier revision of this helper accepted a `config` envelope alongside the
|
||||
* inline form, on the belief that the two SEPs disagreed. Re-reading SEP-2133
|
||||
* at Final status, they do not. The envelope is a non-conformant shape emitted
|
||||
* by at least one SDK, so it is reported rather than silently accepted.
|
||||
*/
|
||||
export function settingsAreInline(skills: Record<string, unknown>): {
|
||||
inline: boolean;
|
||||
envelopeKeys: string[];
|
||||
} {
|
||||
const envelopeKeys = ['config', 'specVersion', 'stability', 'id'].filter(
|
||||
(k) => k in skills
|
||||
);
|
||||
return { inline: envelopeKeys.length === 0, envelopeKeys };
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the server declares the base `resources` capability.
|
||||
*
|
||||
* The stable spec page (ext-skills#139) states this as a consequence of the
|
||||
* base Resources specification rather than a new obligation: a server that
|
||||
* serves skill files through `resources/read` already has to declare it.
|
||||
*/
|
||||
export async function resourcesCapabilityDeclared(
|
||||
conn: Connection
|
||||
): Promise<boolean> {
|
||||
const discovered = await conn.discover();
|
||||
const caps = (discovered.capabilities as Record<string, unknown>) ?? {};
|
||||
return isSettingsObject(caps.resources);
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the skills extension declares `directoryRead: true`.
|
||||
*
|
||||
* Reads only the inline location the SEPs specify. A server that buries the
|
||||
* flag inside an envelope fails `sep-2640-capability-declaration-inline` and is
|
||||
* treated here as not having declared the optional method, which is the
|
||||
* conservative reading: a client that follows the spec would not see the flag
|
||||
* either, and "clients MUST NOT call resources/directory/read against a server
|
||||
* that has not declared directoryRead: true".
|
||||
*/
|
||||
export function directoryReadDeclared(
|
||||
skills: Record<string, unknown>
|
||||
): boolean {
|
||||
return skills.directoryRead === true;
|
||||
}
|
||||
|
||||
/** Everything from `resources/list`, paginating until `nextCursor` clears. */
|
||||
export async function listAllResources(
|
||||
conn: Connection
|
||||
): Promise<SkillResource[]> {
|
||||
const out: SkillResource[] = [];
|
||||
let cursor: string | undefined;
|
||||
do {
|
||||
const page = await conn.request<{
|
||||
resources?: SkillResource[];
|
||||
nextCursor?: string;
|
||||
}>('resources/list', cursor ? { cursor } : undefined);
|
||||
out.push(...(page.resources ?? []));
|
||||
cursor = page.nextCursor;
|
||||
} while (cursor);
|
||||
return out;
|
||||
}
|
||||
|
||||
/** One page of `skills/list`, kept separate so pagination can be inspected. */
|
||||
export interface SkillsListPage {
|
||||
result: SkillsListResult;
|
||||
entries: SkillEntry[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Call `skills/list` once, optionally with a cursor. Returns a `JsonRpcError`
|
||||
* rather than throwing so a scenario can distinguish "method not implemented"
|
||||
* (a FAILURE, since the method is mandatory for a declaring server) from a
|
||||
* transport fault.
|
||||
*/
|
||||
export async function skillsListPage(
|
||||
conn: Connection,
|
||||
cursor?: string
|
||||
): Promise<SkillsListPage | { error: JsonRpcError }> {
|
||||
try {
|
||||
const result = await conn.request<SkillsListResult>(
|
||||
SKILLS_LIST_METHOD,
|
||||
cursor ? { cursor } : {}
|
||||
);
|
||||
const entries = Array.isArray(result.skills)
|
||||
? (result.skills as SkillEntry[])
|
||||
: [];
|
||||
return { result, entries };
|
||||
} catch (e) {
|
||||
if (e instanceof JsonRpcError) return { error: e };
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Every entry from `skills/list`, following `nextCursor`. `pages` is retained
|
||||
* so the atomic-entry and pagination checks can reason about page boundaries.
|
||||
* Bounded to avoid looping forever against a server that returns a constant
|
||||
* cursor.
|
||||
*/
|
||||
export async function skillsListAll(
|
||||
conn: Connection,
|
||||
maxPages = 50
|
||||
): Promise<
|
||||
| { entries: SkillEntry[]; pages: SkillsListPage[]; truncated: boolean }
|
||||
| { error: JsonRpcError }
|
||||
> {
|
||||
const pages: SkillsListPage[] = [];
|
||||
const entries: SkillEntry[] = [];
|
||||
let cursor: string | undefined;
|
||||
const seenCursors = new Set<string>();
|
||||
|
||||
for (let i = 0; i < maxPages; i++) {
|
||||
const page = await skillsListPage(conn, cursor);
|
||||
if ('error' in page) return page;
|
||||
pages.push(page);
|
||||
entries.push(...page.entries);
|
||||
const next = page.result.nextCursor;
|
||||
if (typeof next !== 'string' || next.length === 0) {
|
||||
return { entries, pages, truncated: false };
|
||||
}
|
||||
if (seenCursors.has(next)) {
|
||||
// A repeating cursor is a server bug; stop rather than spin.
|
||||
return { entries, pages, truncated: true };
|
||||
}
|
||||
seenCursors.add(next);
|
||||
cursor = next;
|
||||
}
|
||||
return { entries, pages, truncated: true };
|
||||
}
|
||||
|
||||
/** Call `skills/get` for one skill URI. */
|
||||
export async function skillsGet(
|
||||
conn: Connection,
|
||||
uri: string
|
||||
): Promise<{ result: SkillsGetResult } | { error: JsonRpcError }> {
|
||||
try {
|
||||
const result = await conn.request<SkillsGetResult>(SKILLS_GET_METHOD, {
|
||||
uri
|
||||
});
|
||||
return { result };
|
||||
} catch (e) {
|
||||
if (e instanceof JsonRpcError) return { error: e };
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
/** Read a resource's first text content plus its mimeType and `_meta`. */
|
||||
export async function readResourceText(
|
||||
conn: Connection,
|
||||
uri: string
|
||||
): Promise<ResourceText | undefined> {
|
||||
const res = await conn.request<{
|
||||
contents?: Array<{
|
||||
text?: string;
|
||||
mimeType?: string;
|
||||
_meta?: Record<string, unknown>;
|
||||
}>;
|
||||
}>('resources/read', { uri });
|
||||
const entry = (res.contents ?? []).find((c) => typeof c.text === 'string');
|
||||
if (!entry || typeof entry.text !== 'string') return undefined;
|
||||
return { text: entry.text, mimeType: entry.mimeType, meta: entry._meta };
|
||||
}
|
||||
|
||||
/**
|
||||
* The skill name recoverable from a `SKILL.md` resource URI: the final segment
|
||||
* of `<skill-path>`, i.e. the last path segment before the trailing
|
||||
* `SKILL.md`. Returns `undefined` when the URI does not end in `/SKILL.md`.
|
||||
*
|
||||
* Scheme-agnostic by design: the SEP is explicit that "no scheme is
|
||||
* privileged" and that the structural constraints "apply regardless of
|
||||
* scheme", so a server serving skills under `github://` is judged by the same
|
||||
* path rule as one using `skill://`.
|
||||
*
|
||||
* skill://org/team/deploy/SKILL.md -> "deploy"
|
||||
* github://acme/repo/lint/SKILL.md -> "lint"
|
||||
*/
|
||||
export function skillNameFromManifestUri(uri: string): string | undefined {
|
||||
const schemeEnd = uri.indexOf('://');
|
||||
if (schemeEnd < 0) return undefined;
|
||||
const parts = uri
|
||||
.slice(schemeEnd + 3)
|
||||
.split('/')
|
||||
.filter((p) => p.length > 0);
|
||||
if (parts.length < 2) return undefined;
|
||||
if (parts[parts.length - 1] !== SKILL_MANIFEST_FILENAME) return undefined;
|
||||
return parts[parts.length - 2];
|
||||
}
|
||||
|
||||
/** The skill's root directory URI: its `SKILL.md` URI with the file removed. */
|
||||
export function skillRootFromManifestUri(uri: string): string | undefined {
|
||||
if (!uri.endsWith(`/${SKILL_MANIFEST_FILENAME}`)) return undefined;
|
||||
return uri.slice(0, -`/${SKILL_MANIFEST_FILENAME}`.length);
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract and parse the YAML frontmatter block at the head of a `SKILL.md`.
|
||||
* Returns `undefined` when there is no leading `---` delimited block or it does
|
||||
* not parse to an object.
|
||||
*/
|
||||
export function parseFrontmatter(
|
||||
markdown: string
|
||||
): Record<string, unknown> | undefined {
|
||||
// Tolerate a leading UTF-8 BOM before the opening `---` fence.
|
||||
const body = markdown.charCodeAt(0) === 0xfeff ? markdown.slice(1) : markdown;
|
||||
const match = body.match(/^---\r?\n([\s\S]*?)\r?\n---[ \t]*(?:\r?\n|$)/);
|
||||
if (!match) return undefined;
|
||||
try {
|
||||
const parsed = parseYaml(match[1]) as unknown;
|
||||
return parsed && typeof parsed === 'object'
|
||||
? (parsed as Record<string, unknown>)
|
||||
: undefined;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
/** True when the entry's `resources` is the `"dynamic"` sentinel. */
|
||||
export function isDynamicResources(entry: SkillEntry): boolean {
|
||||
return entry.resources === RESOURCES_DYNAMIC;
|
||||
}
|
||||
|
||||
/**
|
||||
* The entry's `resources` array, or `undefined` when it is `"dynamic"`, absent,
|
||||
* or any other value. Callers distinguish those cases via `isDynamicResources`.
|
||||
*/
|
||||
export function resourcesArray(
|
||||
entry: SkillEntry
|
||||
): SkillResourceEntry[] | undefined {
|
||||
return Array.isArray(entry.resources)
|
||||
? (entry.resources as SkillResourceEntry[])
|
||||
: undefined;
|
||||
}
|
||||
|
||||
/** A short, stable label for an entry, for error messages. */
|
||||
export function entryLabel(entry: SkillEntry, i: number): string {
|
||||
return typeof entry.uri === 'string' ? entry.uri : `skills[${i}]`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Every child of a directory, following `nextCursor` until it clears.
|
||||
*
|
||||
* SEP-2640 says directory-read pagination mirrors `resources/list`, so a
|
||||
* conformant server MAY split a directory across pages. Reading only the
|
||||
* first page makes "no subdirectory here" indistinguishable from "the
|
||||
* subdirectory is on page two".
|
||||
*/
|
||||
export async function directoryReadAll(
|
||||
conn: Connection,
|
||||
uri: string,
|
||||
maxPages = 50
|
||||
): Promise<{ resources: SkillResource[]; pages: number; truncated: boolean }> {
|
||||
const resources: SkillResource[] = [];
|
||||
const seen = new Set<string>();
|
||||
let cursor: string | undefined;
|
||||
|
||||
for (let i = 0; i < maxPages; i++) {
|
||||
const page = await conn.request<{
|
||||
resources?: SkillResource[];
|
||||
nextCursor?: string;
|
||||
}>('resources/directory/read', cursor ? { uri, cursor } : { uri });
|
||||
resources.push(...(page.resources ?? []));
|
||||
const next = page.nextCursor;
|
||||
if (typeof next !== 'string' || next.length === 0) {
|
||||
return { resources, pages: i + 1, truncated: false };
|
||||
}
|
||||
if (seen.has(next)) return { resources, pages: i + 1, truncated: true };
|
||||
seen.add(next);
|
||||
cursor = next;
|
||||
}
|
||||
return { resources, pages: maxPages, truncated: true };
|
||||
}
|
||||
358
src/scenarios/server/skills/manifest.ts
Normal file
358
src/scenarios/server/skills/manifest.ts
Normal file
|
|
@ -0,0 +1,358 @@
|
|||
/**
|
||||
* SEP-2640 Skills extension — the `SKILL.md` manifest resource.
|
||||
*
|
||||
* One scenario, many checks (per AGENTS.md "fewer scenarios, more checks").
|
||||
* Each check's verbatim spec quote lives next to its check ID in
|
||||
* src/seps/sep-2640.yaml.
|
||||
*
|
||||
* Discovery is dynamic and brand-neutral: the scenario finds a `skill-md`
|
||||
* skill's `SKILL.md` resource from `resources/list` (preferred — it carries the
|
||||
* Resource `name`/`description` metadata) or falls back to the first `skill-md`
|
||||
* entry in `skills/list`, hardcoding no fixture skill. Undeclared
|
||||
* extension SKIPs; a declared extension with no discoverable `SKILL.md` reports
|
||||
* the missing prerequisite via untestableCheck (issue #248), never a silent
|
||||
* green.
|
||||
*/
|
||||
|
||||
import { ClientScenario, ConformanceCheck } from '../../../types';
|
||||
import { JsonRpcError, type RunContext } from '../../../connection';
|
||||
import { untestableCheck } from '../../untestable';
|
||||
import {
|
||||
SKILLS_EXTENSION_ID,
|
||||
SKILLS_META_PREFIX,
|
||||
SEP_2640_REF,
|
||||
type SkillResource,
|
||||
skillsCapability,
|
||||
skillsCheck,
|
||||
listAllResources,
|
||||
skillsListAll,
|
||||
readResourceText,
|
||||
skillNameFromManifestUri,
|
||||
parseFrontmatter
|
||||
} from './helpers';
|
||||
|
||||
const MIMETYPE_ID = 'sep-2640-skillmd-mimetype';
|
||||
const METADATA_NAME_ID = 'sep-2640-skillmd-metadata-name';
|
||||
const METADATA_DESCRIPTION_ID = 'sep-2640-skillmd-metadata-description';
|
||||
const FINAL_SEGMENT_ID = 'sep-2640-final-segment-equals-name';
|
||||
const META_PREFIX_ID = 'sep-2640-meta-prefix';
|
||||
|
||||
const MARKDOWN_MIME = 'text/markdown';
|
||||
|
||||
/** A SKILL.md resource URI is skill://<skill-path>/SKILL.md. */
|
||||
function isManifestUri(uri: string): boolean {
|
||||
return skillNameFromManifestUri(uri) !== undefined;
|
||||
}
|
||||
|
||||
/** A `_meta` key that already carries a reverse-domain namespace (`vendor.tld/…`). */
|
||||
function isNamespacedMetaKey(key: string): boolean {
|
||||
return /^[a-z0-9-]+(\.[a-z0-9-]+)+\//i.test(key);
|
||||
}
|
||||
|
||||
export class SkillsManifestScenario implements ClientScenario {
|
||||
name = 'sep-2640-skills-manifest';
|
||||
readonly source = { extensionId: SKILLS_EXTENSION_ID } as const;
|
||||
description = `SEP-2640 Skills extension: the \`SKILL.md\` manifest resource.
|
||||
|
||||
**Resource**: \`skill://<skill-path>/SKILL.md\` (read via \`resources/read\`)
|
||||
|
||||
**Requirements covered** (each check carries a verbatim spec excerpt in src/seps/sep-2640.yaml):
|
||||
|
||||
- \`sep-2640-skillmd-mimetype\` — the SKILL.md resource \`mimeType\` SHOULD be \`text/markdown\`
|
||||
- \`sep-2640-skillmd-metadata-name\` — the resource \`name\` SHOULD be the frontmatter \`name\`
|
||||
- \`sep-2640-skillmd-metadata-description\` — the resource \`description\` SHOULD be the frontmatter \`description\`
|
||||
- \`sep-2640-final-segment-equals-name\` — the final \`<skill-path>\` segment MUST equal the frontmatter \`name\`
|
||||
- \`sep-2640-meta-prefix\` — un-namespaced skill \`_meta\` keys SHOULD use the \`io.modelcontextprotocol.skills/\` prefix
|
||||
|
||||
**Discovery is dynamic**: the scenario picks the first \`skill-md\` skill it finds. Undeclared extension SKIPs; a declared extension with no discoverable SKILL.md reports the missing prerequisite (not a silent skip).`;
|
||||
|
||||
async run(ctx: RunContext): Promise<ConformanceCheck[]> {
|
||||
const conn = await ctx.connect();
|
||||
try {
|
||||
const skills = await skillsCapability(conn);
|
||||
const allIds = [
|
||||
MIMETYPE_ID,
|
||||
METADATA_NAME_ID,
|
||||
METADATA_DESCRIPTION_ID,
|
||||
FINAL_SEGMENT_ID,
|
||||
META_PREFIX_ID
|
||||
];
|
||||
if (!skills) {
|
||||
const reason =
|
||||
'Server did not declare the io.modelcontextprotocol/skills extension; SKILL.md checks not applicable.';
|
||||
return allIds.map((id) =>
|
||||
skillsCheck(id, reason, 'SKIPPED', { errorMessage: reason })
|
||||
);
|
||||
}
|
||||
|
||||
// === Dynamic discovery: resources/list first (carries Resource
|
||||
// metadata), then skills/list. ===
|
||||
const resources = await listAllResources(conn);
|
||||
const manifestResource: SkillResource | undefined = resources.find((r) =>
|
||||
isManifestUri(r.uri)
|
||||
);
|
||||
let manifestUri = manifestResource?.uri;
|
||||
if (!manifestUri) {
|
||||
const listed = await skillsListAll(conn);
|
||||
if (!('error' in listed)) {
|
||||
const entry = listed.entries.find(
|
||||
(e) => typeof e.uri === 'string' && isManifestUri(e.uri)
|
||||
);
|
||||
manifestUri = entry?.uri as string | undefined;
|
||||
}
|
||||
}
|
||||
|
||||
if (!manifestUri) {
|
||||
const reason =
|
||||
'no skill://<skill-path>/SKILL.md resource found via resources/list or skills/list';
|
||||
return [
|
||||
untestableCheck(
|
||||
MIMETYPE_ID,
|
||||
MIMETYPE_ID,
|
||||
'SKILL.md resource mimeType SHOULD be text/markdown.',
|
||||
reason,
|
||||
[SEP_2640_REF],
|
||||
'WARNING'
|
||||
),
|
||||
untestableCheck(
|
||||
METADATA_NAME_ID,
|
||||
METADATA_NAME_ID,
|
||||
'SKILL.md resource name SHOULD match the frontmatter name.',
|
||||
reason,
|
||||
[SEP_2640_REF],
|
||||
'WARNING'
|
||||
),
|
||||
untestableCheck(
|
||||
METADATA_DESCRIPTION_ID,
|
||||
METADATA_DESCRIPTION_ID,
|
||||
'SKILL.md resource description SHOULD match the frontmatter description.',
|
||||
reason,
|
||||
[SEP_2640_REF],
|
||||
'WARNING'
|
||||
),
|
||||
untestableCheck(
|
||||
FINAL_SEGMENT_ID,
|
||||
FINAL_SEGMENT_ID,
|
||||
'The final <skill-path> segment MUST equal the frontmatter name.',
|
||||
reason,
|
||||
[SEP_2640_REF],
|
||||
'FAILURE'
|
||||
),
|
||||
untestableCheck(
|
||||
META_PREFIX_ID,
|
||||
META_PREFIX_ID,
|
||||
'Skill _meta keys SHOULD use the io.modelcontextprotocol.skills/ prefix.',
|
||||
reason,
|
||||
[SEP_2640_REF],
|
||||
'WARNING'
|
||||
)
|
||||
];
|
||||
}
|
||||
|
||||
const checks: ConformanceCheck[] = [];
|
||||
|
||||
// Read the manifest content (for mimeType, frontmatter, and _meta).
|
||||
let content:
|
||||
| { text: string; mimeType?: string; meta?: Record<string, unknown> }
|
||||
| undefined;
|
||||
let readError: string | undefined;
|
||||
try {
|
||||
content = await readResourceText(conn, manifestUri);
|
||||
if (!content) readError = 'resources/read returned no text content';
|
||||
} catch (e) {
|
||||
readError =
|
||||
e instanceof JsonRpcError
|
||||
? `resources/read failed: code ${e.code}: ${e.message}`
|
||||
: e instanceof Error
|
||||
? e.message
|
||||
: String(e);
|
||||
}
|
||||
|
||||
// === skillmd-mimetype (SHOULD) ===
|
||||
// Prefer the read content's mimeType; fall back to the resources/list
|
||||
// Resource metadata mimeType.
|
||||
const mimeType = content?.mimeType ?? manifestResource?.mimeType;
|
||||
if (mimeType === undefined) {
|
||||
checks.push(
|
||||
untestableCheck(
|
||||
MIMETYPE_ID,
|
||||
MIMETYPE_ID,
|
||||
'SKILL.md resource mimeType SHOULD be text/markdown.',
|
||||
`no mimeType observable for ${manifestUri}${readError ? ` (${readError})` : ''}`,
|
||||
[SEP_2640_REF],
|
||||
'WARNING'
|
||||
)
|
||||
);
|
||||
} else {
|
||||
checks.push(
|
||||
skillsCheck(
|
||||
MIMETYPE_ID,
|
||||
'SKILL.md resource mimeType SHOULD be text/markdown.',
|
||||
mimeType === MARKDOWN_MIME ? 'SUCCESS' : 'WARNING',
|
||||
mimeType === MARKDOWN_MIME
|
||||
? { details: { uri: manifestUri, mimeType } }
|
||||
: {
|
||||
errorMessage: `expected mimeType "${MARKDOWN_MIME}", got ${JSON.stringify(mimeType)}`
|
||||
}
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// Parse the frontmatter once for the name/description/final-segment checks.
|
||||
const frontmatter = content ? parseFrontmatter(content.text) : undefined;
|
||||
const fmName =
|
||||
typeof frontmatter?.name === 'string' ? frontmatter.name : undefined;
|
||||
const fmDescription =
|
||||
typeof frontmatter?.description === 'string'
|
||||
? frontmatter.description
|
||||
: undefined;
|
||||
|
||||
// === final-segment-equals-name (MUST) ===
|
||||
const uriName = skillNameFromManifestUri(manifestUri);
|
||||
if (fmName === undefined || uriName === undefined) {
|
||||
const missing =
|
||||
fmName === undefined
|
||||
? `SKILL.md frontmatter has no string "name"${readError ? ` (${readError})` : ''}`
|
||||
: `could not derive the skill name from URI ${manifestUri}`;
|
||||
checks.push(
|
||||
untestableCheck(
|
||||
FINAL_SEGMENT_ID,
|
||||
FINAL_SEGMENT_ID,
|
||||
'The final <skill-path> segment MUST equal the frontmatter name.',
|
||||
missing,
|
||||
[SEP_2640_REF],
|
||||
'FAILURE'
|
||||
)
|
||||
);
|
||||
} else {
|
||||
checks.push(
|
||||
skillsCheck(
|
||||
FINAL_SEGMENT_ID,
|
||||
'The final <skill-path> segment of the SKILL.md URI MUST equal the frontmatter name.',
|
||||
uriName === fmName ? 'SUCCESS' : 'FAILURE',
|
||||
uriName === fmName
|
||||
? { details: { uri: manifestUri, name: fmName } }
|
||||
: {
|
||||
errorMessage: `final path segment "${uriName}" != frontmatter name "${fmName}"`
|
||||
}
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// === skillmd-metadata-name (SHOULD) — needs the Resource metadata ===
|
||||
if (!manifestResource) {
|
||||
checks.push(
|
||||
untestableCheck(
|
||||
METADATA_NAME_ID,
|
||||
METADATA_NAME_ID,
|
||||
'SKILL.md resource name SHOULD match the frontmatter name.',
|
||||
`SKILL.md ${manifestUri} is not listed in resources/list, so its Resource name metadata is not observable`,
|
||||
[SEP_2640_REF],
|
||||
'WARNING'
|
||||
)
|
||||
);
|
||||
} else if (fmName === undefined) {
|
||||
checks.push(
|
||||
untestableCheck(
|
||||
METADATA_NAME_ID,
|
||||
METADATA_NAME_ID,
|
||||
'SKILL.md resource name SHOULD match the frontmatter name.',
|
||||
`SKILL.md frontmatter has no string "name" to compare against${readError ? ` (${readError})` : ''}`,
|
||||
[SEP_2640_REF],
|
||||
'WARNING'
|
||||
)
|
||||
);
|
||||
} else {
|
||||
checks.push(
|
||||
skillsCheck(
|
||||
METADATA_NAME_ID,
|
||||
'The SKILL.md resource name SHOULD be set from the frontmatter name.',
|
||||
manifestResource.name === fmName ? 'SUCCESS' : 'WARNING',
|
||||
manifestResource.name === fmName
|
||||
? { details: { name: fmName } }
|
||||
: {
|
||||
errorMessage: `resource name ${JSON.stringify(manifestResource.name)} != frontmatter name ${JSON.stringify(fmName)}`
|
||||
}
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// === skillmd-metadata-description (SHOULD) — needs Resource metadata ===
|
||||
if (!manifestResource) {
|
||||
checks.push(
|
||||
untestableCheck(
|
||||
METADATA_DESCRIPTION_ID,
|
||||
METADATA_DESCRIPTION_ID,
|
||||
'SKILL.md resource description SHOULD match the frontmatter description.',
|
||||
`SKILL.md ${manifestUri} is not listed in resources/list, so its Resource description metadata is not observable`,
|
||||
[SEP_2640_REF],
|
||||
'WARNING'
|
||||
)
|
||||
);
|
||||
} else if (fmDescription === undefined) {
|
||||
checks.push(
|
||||
untestableCheck(
|
||||
METADATA_DESCRIPTION_ID,
|
||||
METADATA_DESCRIPTION_ID,
|
||||
'SKILL.md resource description SHOULD match the frontmatter description.',
|
||||
`SKILL.md frontmatter has no string "description" to compare against${readError ? ` (${readError})` : ''}`,
|
||||
[SEP_2640_REF],
|
||||
'WARNING'
|
||||
)
|
||||
);
|
||||
} else {
|
||||
checks.push(
|
||||
skillsCheck(
|
||||
METADATA_DESCRIPTION_ID,
|
||||
'The SKILL.md resource description SHOULD be set from the frontmatter description.',
|
||||
manifestResource.description === fmDescription
|
||||
? 'SUCCESS'
|
||||
: 'WARNING',
|
||||
manifestResource.description === fmDescription
|
||||
? { details: { description: fmDescription } }
|
||||
: {
|
||||
errorMessage: `resource description ${JSON.stringify(manifestResource.description)} != frontmatter description ${JSON.stringify(fmDescription)}`
|
||||
}
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
// === meta-prefix (SHOULD, conditional on _meta keys being present) ===
|
||||
// Union the _meta of the read content and the resources/list Resource.
|
||||
const metaKeys = new Set<string>([
|
||||
...Object.keys(content?.meta ?? {}),
|
||||
...Object.keys(manifestResource?._meta ?? {})
|
||||
]);
|
||||
if (metaKeys.size === 0) {
|
||||
checks.push(
|
||||
skillsCheck(
|
||||
META_PREFIX_ID,
|
||||
'When _meta keys are used for skill resources, they SHOULD use the io.modelcontextprotocol.skills/ reverse-domain prefix.',
|
||||
'SUCCESS',
|
||||
{ details: { note: 'skill resource exposes no _meta keys' } }
|
||||
)
|
||||
);
|
||||
} else {
|
||||
// Only bare (un-namespaced) keys are flagged: a key already carrying a
|
||||
// reverse-domain namespace is the intended shape, whichever vendor.
|
||||
const bareKeys = [...metaKeys].filter((k) => !isNamespacedMetaKey(k));
|
||||
checks.push(
|
||||
skillsCheck(
|
||||
META_PREFIX_ID,
|
||||
'When _meta keys are used for skill resources, they SHOULD use the io.modelcontextprotocol.skills/ reverse-domain prefix.',
|
||||
bareKeys.length === 0 ? 'SUCCESS' : 'WARNING',
|
||||
bareKeys.length === 0
|
||||
? { details: { metaKeys: [...metaKeys] } }
|
||||
: {
|
||||
errorMessage: `un-namespaced skill _meta keys SHOULD use the ${SKILLS_META_PREFIX} prefix: ${bareKeys.join(', ')}`
|
||||
}
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
return checks;
|
||||
} finally {
|
||||
await conn.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
301
src/scenarios/server/skills/negative.test.ts
Normal file
301
src/scenarios/server/skills/negative.test.ts
Normal file
|
|
@ -0,0 +1,301 @@
|
|||
import { describe, test, expect } from 'vitest';
|
||||
import { createHash } from 'crypto';
|
||||
import { createServer, type IncomingMessage, type Server } from 'http';
|
||||
import type { AddressInfo } from 'net';
|
||||
import { testContext } from '../../../connection/testing';
|
||||
import { DRAFT_PROTOCOL_VERSION } from '../../../types';
|
||||
import { withRequiredDraftResultFields } from '../../../mock-server';
|
||||
import {
|
||||
takeWireViolations,
|
||||
formatWireViolation
|
||||
} from '../../../validation/wire-schema';
|
||||
import { SkillsEnumerationScenario } from './enumeration';
|
||||
import { SKILLS_EXTENSION_ID } from './helpers';
|
||||
|
||||
/**
|
||||
* Negative controls for the SEP-2640 capability checks.
|
||||
*
|
||||
* `sep-2640-capability-empty-object` used to report a hardcoded SUCCESS, so it
|
||||
* could not fail and was checking nothing (PR #330 review). It now asserts the
|
||||
* observable half of "an empty object indicates support for the extension with
|
||||
* no optional features": the declared value must be an object at all.
|
||||
*
|
||||
* The same change closes a false SKIP. A server declaring the extension as
|
||||
* `true` used to be folded into "extension not declared" and skipped the whole
|
||||
* suite, which read as a clean run against a server that is plainly wrong.
|
||||
*
|
||||
* A non-object declaration also violates the base `DiscoverResult` schema, so
|
||||
* the harness's wire validation catches it independently. That is belt and
|
||||
* braces, not redundancy: the scenario's job is to stop *masking* the problem
|
||||
* with a SKIP, and the tests below assert both halves. The malformed cases
|
||||
* emit those violations deliberately and drain them.
|
||||
*/
|
||||
|
||||
/** A listed entry that is well formed apart from whatever the test is probing. */
|
||||
function entryFor(name: string) {
|
||||
const uri = `skill://${name}/SKILL.md`;
|
||||
return {
|
||||
uri,
|
||||
frontmatter: { name, description: 'A negative-control fixture skill.' },
|
||||
resources: [
|
||||
{
|
||||
uri,
|
||||
digest: `sha256:${'0'.repeat(64)}`,
|
||||
size: 1
|
||||
}
|
||||
]
|
||||
};
|
||||
}
|
||||
|
||||
function jsonRpcResult(id: unknown, method: string, result: object): string {
|
||||
return JSON.stringify({
|
||||
jsonrpc: '2.0',
|
||||
id,
|
||||
result: withRequiredDraftResultFields(method, result)
|
||||
});
|
||||
}
|
||||
|
||||
async function readJsonBody(
|
||||
req: IncomingMessage
|
||||
): Promise<Record<string, unknown>> {
|
||||
const chunks: Buffer[] = [];
|
||||
for await (const chunk of req) {
|
||||
chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk));
|
||||
}
|
||||
return JSON.parse(Buffer.concat(chunks).toString('utf8')) as Record<
|
||||
string,
|
||||
unknown
|
||||
>;
|
||||
}
|
||||
|
||||
/**
|
||||
* A minimal SEP-2575 stateless server whose only interesting property is the
|
||||
* value it declares the skills extension with. Everything else answers just
|
||||
* enough for the scenario to reach its capability checks.
|
||||
*/
|
||||
function startServerDeclaring(
|
||||
declared: unknown,
|
||||
skills: object[] = [],
|
||||
files: Record<string, string> = {}
|
||||
): Promise<{ url: string; server: Server }> {
|
||||
const server = createServer(async (req, res) => {
|
||||
if (req.method !== 'POST') {
|
||||
res.writeHead(405).end();
|
||||
return;
|
||||
}
|
||||
const body = await readJsonBody(req);
|
||||
const method = body.method as string;
|
||||
const id = body.id;
|
||||
const send = (result: object) => {
|
||||
res.writeHead(200, { 'Content-Type': 'application/json' });
|
||||
res.end(jsonRpcResult(id, method, result));
|
||||
};
|
||||
|
||||
if (method === 'server/discover') {
|
||||
const extensions =
|
||||
declared === undefined ? {} : { [SKILLS_EXTENSION_ID]: declared };
|
||||
send({
|
||||
supportedVersions: [DRAFT_PROTOCOL_VERSION],
|
||||
capabilities: { resources: {}, extensions },
|
||||
serverInfo: { name: 'skills-negative', version: '1.0.0' }
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (method === 'skills/list') {
|
||||
send({ skills });
|
||||
return;
|
||||
}
|
||||
if (method === 'resources/read') {
|
||||
const uri = (body.params as { uri?: string } | undefined)?.uri;
|
||||
if (uri !== undefined && uri in files) {
|
||||
send({
|
||||
contents: [{ uri, mimeType: 'text/markdown', text: files[uri] }]
|
||||
});
|
||||
return;
|
||||
}
|
||||
}
|
||||
if (method === 'resources/list') {
|
||||
send({ resources: [] });
|
||||
return;
|
||||
}
|
||||
res.writeHead(200, { 'Content-Type': 'application/json' });
|
||||
res.end(
|
||||
JSON.stringify({
|
||||
jsonrpc: '2.0',
|
||||
id,
|
||||
error: { code: -32601, message: `Method not found: ${method}` }
|
||||
})
|
||||
);
|
||||
});
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
server.once('error', reject);
|
||||
server.listen(0, () => {
|
||||
const addr = server.address() as AddressInfo;
|
||||
resolve({ url: `http://localhost:${addr.port}/mcp`, server });
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
async function runAgainst(
|
||||
declared: unknown,
|
||||
skills: object[] = [],
|
||||
files: Record<string, string> = {}
|
||||
) {
|
||||
const { url, server } = await startServerDeclaring(declared, skills, files);
|
||||
try {
|
||||
const scenario = new SkillsEnumerationScenario();
|
||||
const checks = await scenario.run(testContext(url, DRAFT_PROTOCOL_VERSION));
|
||||
return {
|
||||
checks: new Map(checks.map((c) => [c.id, c])),
|
||||
// Drained here so an intentionally malformed declaration does not trip
|
||||
// the global vitest hook; the tests assert on the drained result.
|
||||
violations: takeWireViolations().violations
|
||||
};
|
||||
} finally {
|
||||
await new Promise<void>((r) => server.close(() => r()));
|
||||
}
|
||||
}
|
||||
|
||||
async function checksFor(
|
||||
declared: unknown,
|
||||
skills: object[] = [],
|
||||
files: Record<string, string> = {}
|
||||
) {
|
||||
const { checks, violations } = await runAgainst(declared, skills, files);
|
||||
expect(violations, 'unexpected wire-schema violations').toEqual([]);
|
||||
return checks;
|
||||
}
|
||||
|
||||
const EMPTY_OBJECT_ID = 'sep-2640-capability-empty-object';
|
||||
const INLINE_ID = 'sep-2640-capability-declaration-inline';
|
||||
|
||||
describe('SEP-2640 capability declaration', () => {
|
||||
test('an empty object is a valid declaration', async () => {
|
||||
const checks = await checksFor({});
|
||||
expect(checks.get(EMPTY_OBJECT_ID)?.status).toBe('SUCCESS');
|
||||
expect(checks.get(INLINE_ID)?.status).toBe('SUCCESS');
|
||||
});
|
||||
|
||||
test('a non-empty settings object is a valid declaration', async () => {
|
||||
const checks = await checksFor({ directoryRead: true });
|
||||
expect(checks.get(EMPTY_OBJECT_ID)?.status).toBe('SUCCESS');
|
||||
expect(checks.get(INLINE_ID)?.status).toBe('SUCCESS');
|
||||
});
|
||||
|
||||
test.each([
|
||||
['true', true],
|
||||
['a string', 'yes'],
|
||||
['an array', []],
|
||||
['null', null]
|
||||
])(
|
||||
'declaring the extension as %s is a FAILURE, not a SKIP',
|
||||
async (_label, value) => {
|
||||
const { checks, violations } = await runAgainst(value);
|
||||
expect(checks.get(EMPTY_OBJECT_ID)?.status).toBe('FAILURE');
|
||||
// No settings object means no envelope question to answer.
|
||||
expect(checks.get(INLINE_ID)?.status).toBe('SKIPPED');
|
||||
// The base DiscoverResult schema rejects it too; both must fire.
|
||||
expect(
|
||||
violations.some((v) =>
|
||||
/extensions.*must be object/.test(formatWireViolation(v))
|
||||
),
|
||||
'expected a DiscoverResult wire-schema violation'
|
||||
).toBe(true);
|
||||
}
|
||||
);
|
||||
|
||||
test('an envelope instead of inline settings is a FAILURE', async () => {
|
||||
const checks = await checksFor({ config: { directoryRead: true } });
|
||||
expect(checks.get(EMPTY_OBJECT_ID)?.status).toBe('SUCCESS');
|
||||
expect(checks.get(INLINE_ID)?.status).toBe('FAILURE');
|
||||
});
|
||||
|
||||
test('an undeclared extension skips every check', async () => {
|
||||
const checks = await checksFor(undefined);
|
||||
expect(checks.get(EMPTY_OBJECT_ID)?.status).toBe('SKIPPED');
|
||||
expect(checks.get(INLINE_ID)?.status).toBe('SKIPPED');
|
||||
expect([...checks.values()].every((c) => c.status === 'SKIPPED')).toBe(
|
||||
true
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* Valid and invalid `name` examples reproduced from the Agent Skills
|
||||
* specification, which SEP-2640 defers to for naming
|
||||
* (https://agentskills.io/specification#name-field). The consecutive-hyphen
|
||||
* rule is the one the first pass of this check missed.
|
||||
*/
|
||||
const NAMING_ID = 'sep-2640-name-naming-rules';
|
||||
|
||||
describe('SEP-2640 skill naming rules', () => {
|
||||
test.each(['pdf-processing', 'data-analysis', 'code-review', 'a', 'a1'])(
|
||||
'accepts %s',
|
||||
async (name) => {
|
||||
const checks = await checksFor({}, [entryFor(name)]);
|
||||
expect(checks.get(NAMING_ID)?.status).toBe('SUCCESS');
|
||||
}
|
||||
);
|
||||
|
||||
test.each([
|
||||
['consecutive hyphens', 'pdf--processing'],
|
||||
['a leading hyphen', '-pdf'],
|
||||
['a trailing hyphen', 'pdf-'],
|
||||
['65 characters', 'a'.repeat(65)]
|
||||
])('rejects %s', async (_label, name) => {
|
||||
const checks = await checksFor({}, [entryFor(name)]);
|
||||
expect(checks.get(NAMING_ID)?.status).toBe('FAILURE');
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* `entry-frontmatter-identical` compares content, not serialisation. A server
|
||||
* whose JSON encoder orders map keys differently from the YAML source (Go's
|
||||
* encoding/json sorts them) is conformant and must pass.
|
||||
*/
|
||||
const IDENTICAL_ID = 'sep-2640-entry-frontmatter-identical';
|
||||
|
||||
function skillWithFile(
|
||||
yamlMetadata: string,
|
||||
entryMetadata: Record<string, string>
|
||||
) {
|
||||
const uri = 'skill://demo/SKILL.md';
|
||||
const text = `---\nname: demo\ndescription: Demo skill\nmetadata:\n${yamlMetadata}---\n\nBody.\n`;
|
||||
const entry = {
|
||||
uri,
|
||||
frontmatter: {
|
||||
name: 'demo',
|
||||
description: 'Demo skill',
|
||||
metadata: entryMetadata
|
||||
},
|
||||
resources: [
|
||||
{
|
||||
uri,
|
||||
digest: `sha256:${createHash('sha256').update(text).digest('hex')}`,
|
||||
size: Buffer.byteLength(text)
|
||||
}
|
||||
]
|
||||
};
|
||||
return { entry, files: { [uri]: text } };
|
||||
}
|
||||
|
||||
describe('SEP-2640 frontmatter identity', () => {
|
||||
test('nested keys serialised in a different order are identical', async () => {
|
||||
const { entry, files } = skillWithFile(' version: "1.0"\n author: me\n', {
|
||||
author: 'me',
|
||||
version: '1.0'
|
||||
});
|
||||
const checks = await checksFor({}, [entry], files);
|
||||
expect(checks.get(IDENTICAL_ID)?.status).toBe('SUCCESS');
|
||||
});
|
||||
|
||||
test('a differing nested value is a FAILURE', async () => {
|
||||
const { entry, files } = skillWithFile(' version: "1.0"\n author: me\n', {
|
||||
author: 'someone-else',
|
||||
version: '1.0'
|
||||
});
|
||||
const checks = await checksFor({}, [entry], files);
|
||||
expect(checks.get(IDENTICAL_ID)?.status).toBe('FAILURE');
|
||||
});
|
||||
});
|
||||
308
src/seps/sep-2640.yaml
Normal file
308
src/seps/sep-2640.yaml
Normal file
|
|
@ -0,0 +1,308 @@
|
|||
# spec_source: modelcontextprotocol/modelcontextprotocol@sep/skills-extension seps/2640-skills-extension.md
|
||||
# extracted: 2026-08-28
|
||||
#
|
||||
# provenance: re-extracted against the 2026-08-21 revision, which the core
|
||||
# maintainers produced after their second review round and which went back up
|
||||
# for CM vote on 2026-08-25. The prior extraction held at 556154c (2026-06-05)
|
||||
# by an explicit decision to wait for mcpkit#780; that hold expired badly.
|
||||
# The 08-21 revision:
|
||||
# - removed the `skill://index.json` well-known resource entirely (it now
|
||||
# appears nowhere in the SEP) and replaced it with the `skills/list` and
|
||||
# `skills/get` methods, both mandatory for a declaring server;
|
||||
# - deferred archive distribution to an appendix ("Appendix: Deferred
|
||||
# Features"), retiring the `type` enum and every archive requirement;
|
||||
# - reversed the nesting rule: a `SKILL.md` MAY now appear in a descendant
|
||||
# directory, where the June text forbade it;
|
||||
# - reshaped the entry to `{uri, frontmatter, resources}`, where `resources`
|
||||
# is a complete `{uri, digest, size}` array or the string `"dynamic"`;
|
||||
# - fixed per-skill limits of 512 resource entries and 16 MiB;
|
||||
# - added host obligations around lazy retrieval, content-bound approval and
|
||||
# frontmatter re-verification.
|
||||
# Eight rows from the prior extraction were removed as no longer normative
|
||||
# (server-expose-index, index-entry-type-enum, no-nested-skills, and the five
|
||||
# archive rows) and four were reworded. Because the SEP is mid-vote, expect one
|
||||
# more pass at the CM-stamped head.
|
||||
#
|
||||
# coverage: extracted against branch head a3e147ca2710 (2026-08-25), the commit
|
||||
# the CM vote is running on. A keyword sweep of the source finds 97 sentences
|
||||
# carrying an RFC 2119 term; this file declares 89 checks plus 7 excluded rows.
|
||||
# The residual handful are restatements that fold into a declared row rather
|
||||
# than standing alone. Of the 89 declared checks, 40 are exercised on the wire
|
||||
# by the three server scenarios; the other 49 are host obligations (retrieval
|
||||
# policy, approval binding, cache isolation, context handling) that a
|
||||
# server-side harness cannot observe and that would need a host-side scenario
|
||||
# set to test.
|
||||
#
|
||||
# deliberately not declared here: `resultType`, which appears in all three
|
||||
# result examples in this SEP, is a base-protocol field on the common `Result`
|
||||
# interface (schema/2026-07-28/schema.ts), not a SEP-2640 requirement. Servers
|
||||
# MUST include it and clients treat an absent value as "complete". Its
|
||||
# caching-hint obligations belong to sep-2549.yaml. Declaring it here would
|
||||
# double-count a base-protocol rule against this extension.
|
||||
#
|
||||
# backing_scenarios: three server ClientScenarios under
|
||||
# src/scenarios/server/skills/ emit the check IDs below (a row is "tested"
|
||||
# once a scenario emits its check ID; see src/traceability/):
|
||||
# enumeration.ts (sep-2640-skills-enumeration) — the capability rows, the
|
||||
# sep-2640-skills-list-* rows, the entry-schema and resources rows, the two
|
||||
# limit rows, and the sep-2640-skills-get-* rows. Replaces the retired
|
||||
# index.ts, which tested `skill://index.json`.
|
||||
# manifest.ts (sep-2640-skills-manifest) — sep-2640-skillmd-mimetype,
|
||||
# sep-2640-skillmd-metadata-name, sep-2640-skillmd-metadata-description,
|
||||
# sep-2640-final-segment-equals-name, sep-2640-meta-prefix.
|
||||
# directory.ts (sep-2640-skills-directory) —
|
||||
# sep-2640-capability-directory-read-flag and the
|
||||
# sep-2640-directory-read-* rows.
|
||||
# The remaining rows are host-internal or off-wire (host retrieval policy,
|
||||
# digest and frontmatter verification, approval binding, cache isolation) and
|
||||
# stay traceability-only for this server-scenario set. A host-side scenario
|
||||
# set would be needed to exercise them.
|
||||
sep: 2640
|
||||
spec_url: https://modelcontextprotocol.io/seps/2640-skills-extension#specification
|
||||
requirements:
|
||||
# === Skill Format ===
|
||||
- check: sep-2640-skillmd-required
|
||||
text: 'Every skill MUST contain a `SKILL.md` file at its root.'
|
||||
- check: sep-2640-skillmd-frontmatter
|
||||
text: '`SKILL.md` MUST begin with YAML frontmatter containing at minimum the `name` and `description` fields as defined by the Agent Skills specification.'
|
||||
|
||||
# === Resource Mapping ===
|
||||
- check: sep-2640-skill-uri-scheme
|
||||
text: 'Each file within a skill directory is exposed as an MCP resource. Servers SHOULD use the `skill://` URI scheme, under which the resource URI has the form: `skill://<skill-path>/<file-path>`'
|
||||
- check: sep-2640-final-segment-equals-name
|
||||
text: "The final segment of `<skill-path>` MUST equal the skill's `name` as declared in its `SKILL.md` frontmatter."
|
||||
- check: sep-2640-name-naming-rules
|
||||
text: "The final `<skill-path>` segment, being the skill `name`, MUST satisfy the Agent Skills specification's naming rules."
|
||||
- check: sep-2640-authority-reg-name
|
||||
text: 'The first `<skill-path>` segment occupies the authority component and SHOULD be a valid `reg-name` per RFC 3986; any other prefix segments SHOULD be valid URI path segments; no further constraints are imposed on them.'
|
||||
# Reverses the retired sep-2640-no-nested-skills row: the June text said a
|
||||
# SKILL.md MUST NOT appear in any descendant directory.
|
||||
- check: sep-2640-nested-skills-permitted
|
||||
text: 'A `SKILL.md` MAY appear in a descendant directory of a skill — skills can nest.'
|
||||
- check: sep-2640-nested-skillmd-not-acted-on
|
||||
text: "From the enclosing skill's perspective, a nested skill's directory and files are ordinary supporting files, and reading them is ordinary reading. A nested `SKILL.md` read this way is ordinary markdown: hosts MUST NOT act on its frontmatter."
|
||||
- check: sep-2640-nested-publication-flat
|
||||
text: "A nested skill is published like any other: through its own `skills/list` entry, or by explicit reference. The listing remains flat — an entry for a nested skill is an ordinary entry whose `uri` happens to share a path prefix with the enclosing skill's, and nothing in the listing marks nesting."
|
||||
|
||||
# === Resource Metadata ===
|
||||
- check: sep-2640-skillmd-mimetype
|
||||
text: 'For each `skill://<skill-path>/SKILL.md` resource: `mimeType` SHOULD be `text/markdown`.'
|
||||
- check: sep-2640-skillmd-metadata-name
|
||||
text: 'For each `skill://<skill-path>/SKILL.md` resource: `name` SHOULD be set from the `name` field of the `SKILL.md` YAML frontmatter. By the path constraint above, this will equal the final segment of `<skill-path>`.'
|
||||
- check: sep-2640-skillmd-metadata-description
|
||||
text: 'For each `skill://<skill-path>/SKILL.md` resource: `description` SHOULD be set from the `description` field of the `SKILL.md` YAML frontmatter.'
|
||||
- check: sep-2640-meta-prefix
|
||||
text: 'When `_meta` keys are used for skill resources, implementations SHOULD use the `io.modelcontextprotocol.skills/` reverse-domain prefix.'
|
||||
|
||||
# === Capability Declaration ===
|
||||
# The inline-settings row is the check that would have caught panyam/mcpkit#1334.
|
||||
# SEP-2133 (status Final) defines `extensions` as "a map of extension
|
||||
# identifiers to per-extension settings objects" and defines no envelope; the
|
||||
# SEP-2640 capability block matches it.
|
||||
- check: sep-2640-capability-declaration-inline
|
||||
text: 'Per SEP-2133 extension negotiation, servers declare support for this extension in their `initialize` response, mapping the extension identifier directly to its per-extension settings object: `{"capabilities": {"extensions": {"io.modelcontextprotocol/skills": {"directoryRead": true}}}}`.'
|
||||
url: https://modelcontextprotocol.io/seps/2640-skills-extension#capability-declaration
|
||||
- check: sep-2640-capability-commits-to-methods
|
||||
text: 'Declaring the extension itself commits the server to `skills/list` and `skills/get`.'
|
||||
- check: sep-2640-capability-empty-object
|
||||
text: 'An empty object indicates support for the extension with no optional features.'
|
||||
- check: sep-2640-capability-requires-resources
|
||||
text: "This extension depends only on the base protocol's Resources primitive. Skill files are served through `resources/read`, and the base Resources specification requires any server that supports resources to declare the `resources` capability. A server declaring this extension therefore **MUST** also declare the `resources` capability."
|
||||
- check: sep-2640-capability-directory-read-flag
|
||||
text: 'Clients MUST NOT call `resources/directory/read` against a server that has not declared `directoryRead: true`.'
|
||||
|
||||
# === Discovery: enumeration via skills/list ===
|
||||
- check: sep-2640-skills-list-implemented
|
||||
text: 'A server declaring the `io.modelcontextprotocol/skills` extension MUST implement the `skills/list` method, which returns the skills it serves. The result MAY be empty.'
|
||||
- check: sep-2640-skills-list-pagination
|
||||
text: "Pagination mirrors the base protocol's list methods: the request accepts an optional `cursor`, and when the result includes `nextCursor` the client passes it back to retrieve the next page."
|
||||
- check: sep-2640-skills-list-entry-atomic
|
||||
text: "An entry is atomic — a skill's `resources` set is never split across pages."
|
||||
- check: sep-2640-skills-list-cache-attributes
|
||||
text: "In protocol versions 2026-07-28 and later, the result also carries the base protocol's list-caching attributes — `ttlMs` and `cacheScope`, as defined for `tools/list` and `resources/list` (SEP-2549) — with the same semantics: a freshness hint for the listing and a cache-scope marker, not an integrity property."
|
||||
- check: sep-2640-skills-list-may-be-partial
|
||||
text: 'A server whose skill catalog is large, generated on demand, or otherwise unenumerable MAY return an empty or partial listing.'
|
||||
- check: sep-2640-host-no-empty-listing-assumption
|
||||
text: 'Hosts MUST NOT treat an empty or partial listing as proof that a server has no skills.'
|
||||
- check: sep-2640-enumeration-scheme-uniform
|
||||
text: "The method serves entries for a server's skills whatever URI scheme they use — enumeration is uniform across schemes."
|
||||
|
||||
# === The skill entry (shared by skills/list and skills/get) ===
|
||||
- check: sep-2640-entry-uri-required
|
||||
text: "`skills[].uri` (Yes): Resource URI of the skill's `SKILL.md`."
|
||||
- check: sep-2640-entry-frontmatter-required
|
||||
text: "`frontmatter` is the skill's `SKILL.md` YAML frontmatter rendered verbatim as a JSON object — every field the author wrote, not a curated subset. Because the Agent Skills specification requires `name` and `description`, those fields are always present."
|
||||
- check: sep-2640-entry-frontmatter-identical
|
||||
text: 'The `frontmatter` object MUST be identical in content to the frontmatter of the `SKILL.md` it describes.'
|
||||
- check: sep-2640-entry-uri-matches-frontmatter-name
|
||||
text: "The final `<skill-path>` segment of the entry's `uri` MUST equal `frontmatter.name`, per Resource Mapping."
|
||||
- check: sep-2640-metadata-reserved-prefix
|
||||
text: 'Within the frontmatter `metadata` object, keys prefixed with `io.modelcontextprotocol/` are reserved for metadata defined by MCP extensions. This extension currently defines no such keys. Implementations SHOULD ignore keys under this prefix that they do not recognize.'
|
||||
- check: sep-2640-names-should-be-unique
|
||||
text: "Within a server's listing, names SHOULD be unique, but they are not guaranteed to be."
|
||||
- check: sep-2640-host-disambiguate-listing-collision
|
||||
text: 'When two entries in one listing collide on `name`, hosts MUST disambiguate them — for example by their distinguishing path segments — rather than silently discarding or preferring one.'
|
||||
- check: sep-2640-names-not-unique
|
||||
text: "A skill's `name` is a label, not an identifier. Within a server's listing, names SHOULD be unique, but they are not guaranteed to be. Hosts MUST NOT assume name uniqueness. When two entries in one listing collide on `name`, hosts MUST disambiguate them — for example by their distinguishing path segments — rather than silently discarding or preferring one."
|
||||
|
||||
# === resources ===
|
||||
- check: sep-2640-entry-resources-required
|
||||
text: '`resources` is REQUIRED on every skill entry and takes one of two forms: an array enumerating the skill''s files — `SKILL.md` and every supporting file — as `{uri, digest, size}` triples, or the string `"dynamic"`.'
|
||||
- check: sep-2640-resources-complete
|
||||
text: "When present, `resources` MUST be complete: it lists every file of the skill, each exactly once, including an entry matching the skill's top-level `uri` — that entry carries the digest and size of `SKILL.md` itself."
|
||||
- check: sep-2640-resources-uri-within-skill
|
||||
text: "Each `uri` MUST be the skill's `SKILL.md` or a file within the skill's directory."
|
||||
- check: sep-2640-resources-digest-format
|
||||
text: "Digests are SHA-256 hashes of an artifact's raw bytes, formatted as `sha256:{hex}` where `{hex}` is 64 lowercase hexadecimal characters. Each entry in a skill's `resources` carries the digest of the file at its `uri`."
|
||||
- check: sep-2640-resources-size-required
|
||||
text: "Each entry MUST carry `size`: the length in bytes of the file's raw content — the same bytes the `digest` covers."
|
||||
- check: sep-2640-resources-dynamic-marker
|
||||
text: 'When a skill''s content is generated dynamically, such that stable digests cannot be published, the server MUST set `"resources": "dynamic"` instead of an array.'
|
||||
- check: sep-2640-resources-invalid-entry
|
||||
text: 'An entry with no `resources` at all, or with any value other than an array or `"dynamic"`, is invalid, and hosts MUST NOT load it.'
|
||||
- check: sep-2640-resources-nested-completeness
|
||||
text: "Completeness extends to nested skills: from the enclosing skill's perspective their files are supporting files, so the enclosing skill's `resources` lists them too, and the same file may appear in both the enclosing and the nested skill's entries."
|
||||
|
||||
# === Limits ===
|
||||
- check: sep-2640-limit-resources-per-skill
|
||||
text: "Resources per skill: 512 entries, counted over the entries of the skill's `resources`, `SKILL.md` included."
|
||||
- check: sep-2640-limit-total-size
|
||||
text: "Total file size per skill: 16 MiB (16,777,216 bytes), counted over the sum of `size` over the skill's `resources`."
|
||||
- check: sep-2640-limit-host-support
|
||||
text: 'Hosts MUST support skills up to and including these limits, and MAY support larger ones. Servers SHOULD NOT serve a skill that exceeds either limit; a skill that does is not guaranteed to be loadable by any conforming host.'
|
||||
|
||||
# === Retrieval via skills/get ===
|
||||
- check: sep-2640-skills-get-implemented
|
||||
text: 'A server declaring the `io.modelcontextprotocol/skills` extension MUST also implement the `skills/get` method, which returns the entry for a single skill named by its URI.'
|
||||
- check: sep-2640-skills-get-entry-shape
|
||||
text: 'The `skill` object is a skill entry, identical in shape and meaning to an entry of `skills/list` — the same `uri`, `frontmatter`, and `resources` fields, under the same rules.'
|
||||
- check: sep-2640-skills-get-unknown-uri-invalid-params
|
||||
text: 'If the URI does not identify a skill the server serves, the server MUST return error `-32602` (Invalid params) — the same code `resources/read` uses for unknown resources.'
|
||||
- check: sep-2640-skills-get-answers-unlisted
|
||||
text: 'A server MUST answer for every skill it serves, whether or not that skill appears in its `skills/list` result. A skill absent from a partial listing is still retrievable by URI.'
|
||||
- check: sep-2640-skills-get-cache-attributes
|
||||
text: "`GetSkillResult` extends `CacheableResult`, so `ttlMs` and `cacheScope` are **REQUIRED**, as they are on `resources/read`, and carry the same semantics. `ttlMs` is the server's hint for how long a host may treat the entry as current before re-calling `skills/get`. Neither field is an integrity property."
|
||||
- check: sep-2640-skills-get-no-cursor
|
||||
text: 'The result carries no pagination cursor: a single entry is not a list.'
|
||||
|
||||
# === Reading / integrity (host obligations) ===
|
||||
- check: sep-2640-host-load-by-uri
|
||||
text: 'hosts MUST support loading a skill given only its URI'
|
||||
- check: sep-2640-read-is-not-a-load
|
||||
text: "Hosts MUST NOT treat a `resources/read` of a `SKILL.md` that arrives by any other route as a load: it grants no approval, opens no window, and confers no standing on the skill's supporting files."
|
||||
- check: sep-2640-host-verify-digest
|
||||
text: "When a host retrieves a file listed in a skill's `resources`, it MUST verify the content against that entry's digest. Whatever the cause, hosts MUST NOT use the unverified content."
|
||||
- check: sep-2640-host-size-mismatch-failure
|
||||
text: "A read whose byte length differs from the entry's `size` is a verification failure equivalent to a digest mismatch, whether or not the host goes on to compute the digest."
|
||||
- check: sep-2640-host-no-prefetch
|
||||
text: "Hosts MUST NOT retrieve a skill's files ahead of need — not on connection, not on listing, and not at approval. A `SKILL.md` is fetched when the skill is loaded, and a supporting file when it is read."
|
||||
- check: sep-2640-host-frontmatter-comparison
|
||||
text: "After fetching a `SKILL.md` for which the host holds an entry, hosts MUST parse its YAML frontmatter and compare it field-by-field against the entry's `frontmatter`. Any discrepancy MUST be treated as a verification failure equivalent to a digest mismatch, and the skill MUST NOT be loaded."
|
||||
- check: sep-2640-host-digest-not-security-boundary
|
||||
text: 'Digests are unsigned and supplied by the same server that supplies the content. A match proves the two are consistent, not that either is trustworthy. Hosts MUST NOT treat a digest match as a security boundary.'
|
||||
- check: sep-2640-host-skill-identity-pair
|
||||
text: "The identity of an MCP-served skill is the pair of the host's identity for the originating server and the skill's `uri`. Hosts MUST preserve both halves wherever a skill is recorded or addressed — the registry, persisted approvals, the cache, and any tool or path through which the model reaches the skill — and MUST NOT key any of these on the `uri` alone."
|
||||
- check: sep-2640-host-not-skill-by-scheme
|
||||
text: 'A host MUST NOT conclude that a resource is a skill merely because its URI carries a particular scheme.'
|
||||
- check: sep-2640-host-cross-origin-no-shadow
|
||||
text: 'When skills from different origins collide on `name`, hosts MUST resolve the name within a per-origin namespace, identifying servers by a host-assigned label; an MCP-served skill MUST NOT silently shadow, or be silently substituted for, a same-named skill from any other origin.'
|
||||
- check: sep-2640-host-content-bound-approval
|
||||
text: "When a host persists any per-skill user approval, it MUST be bound to the entry's `resources` set — every `uri` and `digest` — observed at the moment of approval. If a subsequent entry for that skill advertises a different set, the host MUST treat the prior approval as revoked and re-prompt before loading or executing."
|
||||
- check: sep-2640-host-dynamic-not-content-bound
|
||||
text: 'A skill whose `resources` is `"dynamic"` cannot be content-bound: hosts MAY decline to load it, and MUST NOT treat a persisted approval as covering whatever content the server currently serves.'
|
||||
- check: sep-2640-host-nested-fresh-consent
|
||||
text: 'Activating a nested skill — loading it as a skill in its own right — requires fresh, explicit user consent; approval of the enclosing skill does not substitute for it. A nested `SKILL.md` read as supporting content is ordinary markdown: hosts MUST NOT act on its frontmatter.'
|
||||
|
||||
- check: sep-2640-host-origin-tag-visible
|
||||
text: 'Hosts MUST tag MCP-served skill content with its originating server identity at the point it enters model context, and MUST NOT present it as indistinguishable from a local filesystem skill.'
|
||||
url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications
|
||||
- check: sep-2640-host-untrusted-input
|
||||
text: 'Hosts MUST treat MCP-served skill content as untrusted model input, subject to the same prompt-injection defenses applied to any server-provided text. A server being connected does not make its skill content authoritative.'
|
||||
url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications
|
||||
- check: sep-2640-host-skills-are-data-not-directives
|
||||
text: 'Hosts MUST NOT treat skill resources as higher-authority than other context. Explicit user policy governs whether a skill is loaded at all.'
|
||||
url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications
|
||||
- check: sep-2640-host-cache-path-encodes-origin
|
||||
text: 'Any path at which a host materializes skill content, whether a cache directory or a virtual mount, MUST encode the server identity as well as the `uri`, so that same-URI skills from different servers land at distinct paths and the originating server is recoverable from the path.'
|
||||
url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications
|
||||
- check: sep-2640-host-higher-risk-surface
|
||||
text: 'Hosts MUST treat MCP-served skills as a higher-risk surface than remote tool invocation.'
|
||||
url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications
|
||||
- check: sep-2640-host-no-implicit-local-execution
|
||||
text: 'Hosts MUST NOT allow MCP-served skill content to cause host-side code execution without explicit per-skill user approval.'
|
||||
url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications
|
||||
- check: sep-2640-host-exec-gate-while-acting
|
||||
text: 'Hosts MUST apply the same approval gate to code-execution tool calls issued while the model is acting on an MCP-served skill.'
|
||||
url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications
|
||||
- check: sep-2640-host-reads-bound-to-origin
|
||||
text: "Hosts MUST bind such reads to the skill's originating server: a skill served by server A MUST NOT cause a `resources/read` against server B."
|
||||
url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications
|
||||
- check: sep-2640-host-cross-origin-read-approval
|
||||
text: 'Any cross-origin read MUST be gated behind explicit per-call user approval naming both servers.'
|
||||
url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications
|
||||
- check: sep-2640-host-label-not-serverinfo-name
|
||||
text: "Hosts MUST identify servers by a host-assigned label, not the server's self-reported `serverInfo.name`."
|
||||
url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications
|
||||
- check: sep-2640-host-no-implicit-permission-grants
|
||||
text: "Hosts MUST NOT honor frontmatter fields that widen the model's tool or filesystem permissions when the skill arrives over MCP."
|
||||
url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications
|
||||
- check: sep-2640-host-ignore-allowed-tools
|
||||
text: 'The Agent Skills `allowed-tools` field, which a filesystem-sourced skill uses to declare the tools available while it runs, MUST be ignored for MCP-origin skills.'
|
||||
url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications
|
||||
- check: sep-2640-intermediary-meta-prefix
|
||||
text: 'Intermediaries MAY attach provenance or verification annotations via `_meta` under their own reverse-domain prefix — not the `io.modelcontextprotocol.skills/` prefix reserved for this extension.'
|
||||
url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications
|
||||
- check: sep-2640-host-cache-write-isolation
|
||||
text: 'Hosts that cache skill content on disk MUST do one of the following for every file served from the cache: keep the cache where nothing but the host can write to it, or re-hash the file against the entry on every read.'
|
||||
url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications
|
||||
- check: sep-2640-host-cache-excluded-from-discovery
|
||||
text: 'Hosts that cache MCP-served skill content on disk MUST also do so in a location excluded from every filesystem-skill discovery path, and MUST treat content loaded from there as MCP-origin.'
|
||||
url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications
|
||||
- check: sep-2640-host-cache-removal-on-server-removal
|
||||
text: "Hosts SHOULD remove a server's cached skill content when the user removes that server."
|
||||
url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications
|
||||
|
||||
# === Implementation Guidelines (host + SDK) ===
|
||||
- check: sep-2640-host-registry-no-fetch
|
||||
text: 'Assembling the registry reads only the listing: the host MUST NOT fetch `SKILL.md` or any supporting file at this stage.'
|
||||
- check: sep-2640-host-virtual-mount-lazy
|
||||
text: "A virtual mount resolves reads on access; it MUST NOT be populated by fetching the skill's files in advance."
|
||||
- check: sep-2640-host-surface-directory-read
|
||||
text: 'When the originating server declares `directoryRead`, the host SHOULD surface this capability to the model.'
|
||||
- check: sep-2640-sdk-convenience-wrappers
|
||||
text: "SDK maintainers SHOULD provide affordances that wrap the underlying resource operations in skill-specific terms. The SDK handles: reading `SKILL.md` frontmatter to populate resource metadata, serving file content on `resources/read`, and answering `skills/get` — and, where the server's skill set is bounded, `skills/list` — computing entry digests and sizes from the registered files, and warning when a registered skill exceeds the Limits."
|
||||
|
||||
# === Directory Listing (resources/directory/read) ===
|
||||
- check: sep-2640-directory-read-method-registered
|
||||
text: 'A server that declares `directoryRead` MUST support the method for every directory within the skill namespaces it serves as individual files.'
|
||||
- check: sep-2640-directory-read-subdir-mimetype
|
||||
text: 'A _directory resource_ is a resource whose `mimeType` is `inode/directory`.'
|
||||
- check: sep-2640-directory-read-result-resources-shape
|
||||
text: 'The result contains every direct child of the directory: files with their ordinary resource metadata, subdirectories listed as directory resources (`mimeType: "inode/directory"`). The listing is not recursive; clients descend by calling the method again on a child directory.'
|
||||
- check: sep-2640-directory-read-invalid-params
|
||||
text: 'The method applies only to directory resources. If the URI does not exist, or exists but is not a directory resource, the server MUST return error `-32602` (Invalid params) — the same code `resources/read` uses for unknown resources.'
|
||||
- check: sep-2640-directory-read-pagination
|
||||
text: 'Pagination mirrors `resources/list`: when the result includes `nextCursor`, the client passes it back as `cursor` to retrieve the next page.'
|
||||
- check: sep-2640-directory-read-empty-dir
|
||||
text: 'An empty directory yields an empty `resources` array.'
|
||||
- check: sep-2640-directory-read-not-manifest-extension
|
||||
text: 'Hosts MUST NOT treat the directory result as extending the manifest. While acting on the skill under the held entry, the host MUST NOT read a newly listed child — an unlisted file is a verification failure, exactly as a digest mismatch is — and MUST NOT surface it to the model as a file of the skill.'
|
||||
|
||||
# === Excluded: not observable on the MCP wire ===
|
||||
- text: 'Per RFC 3986, the first segment of `<skill-path>` occupies the authority component. This carries no special semantics under this convention and clients MUST NOT attempt DNS or network resolution of it.'
|
||||
excluded: 'DNS and network resolution sit below the MCP wire layer; the harness cannot observe whether the client performed name lookups on URI authority components.'
|
||||
- text: "Hosts SHOULD indicate which server a skill originates from when presenting it, SHOULD let users inspect a skill's content before it is loaded into model context"
|
||||
excluded: 'UI presentation requirements (origin indicator, pre-load inspection); the harness cannot observe what the host displays to users.'
|
||||
url: https://modelcontextprotocol.io/seps/2640-skills-extension#security-implications
|
||||
- text: "A host is _acting on_ a skill from the moment it loads the skill's `SKILL.md` into the model's context until, at the earliest, that `SKILL.md` leaves context; hosts MAY hold the window open longer, never shorter."
|
||||
excluded: "The window is defined by what is in the model's context, which is host-internal state the harness cannot observe. It scopes several wire-observable rules but is not itself checkable."
|
||||
- text: "While acting on a skill, a host MUST resolve reads of the skill's files only to URIs listed in that entry's `resources`, and MUST treat a read of an unlisted file within the skill as a verification failure equivalent to a digest mismatch."
|
||||
excluded: 'Scoped by the acting-on window above, which the harness cannot observe. As built it was also the one rule where the harness had nothing adversarial to do: the listing it served was entirely honest, so the only thing under test was whether the driver asked for a URI it should not have, which grades the driver rather than the client. Retired 2026-09-10 after review. Excluded by construction, not in principle: a listing whose `SKILL.md` body links to a file absent from `resources` would make the harness adversarial again, since a host that follows the link fetches an unlisted file on its own initiative. If it returns, it returns under the same scenario name `sep-2640-client-verify-unlisted` and the same check id `sep-2640-host-unlisted-read-failure`, so reports and baselines line up across the gap.'
|
||||
- text: 'Hosts SHOULD instead cache what they do retrieve, and digests make that cache cheap to validate: a cached file whose digest matches the current entry can be served without fetching it again.'
|
||||
excluded: 'Cache hits are the absence of a request. A harness cannot distinguish a compliant cache from a host that simply did not need the file again.'
|
||||
- text: 'A host that declines a skill on this basis SHOULD tell the user why rather than fail silently on a later read.'
|
||||
excluded: 'User-facing messaging; not protocol-observable.'
|
||||
- text: 'For a skill whose `resources` is `"dynamic"`, the entry offers nothing to count. A host that chooses to load such a skill applies the total-size limit to what it actually retrieves and MAY stop loading the skill once that limit is reached.'
|
||||
excluded: 'A host stopping mid-load is indistinguishable on the wire from a host that needed no further files.'
|
||||
- text: 'Hosts SHOULD expect this sequence and present it as such — a skill that has changed and needs re-approval — rather than as a read error.'
|
||||
excluded: 'User-facing presentation of the stale-entry recovery path; not protocol-observable.'
|
||||
|
|
@ -102,7 +102,8 @@ export const EXTENSION_IDS = [
|
|||
'io.modelcontextprotocol/enterprise-managed-authorization',
|
||||
'io.modelcontextprotocol/auth/dpop',
|
||||
'io.modelcontextprotocol/auth/wif',
|
||||
'io.modelcontextprotocol/tasks'
|
||||
'io.modelcontextprotocol/tasks',
|
||||
'io.modelcontextprotocol/skills'
|
||||
] as const;
|
||||
export type ExtensionId = (typeof EXTENSION_IDS)[number];
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue