ext-skills/docs/experimental-findings.md

Experimental Findings

ℹ️ This document is not actively maintained. It captures a snapshot of early findings. Current discussion and decisions are tracked in the meeting notes discussions, Discord #skills-over-mcp-wg, and on the SEP-2640 PR.

Contributing findings? Start with the experimental findings template.

VS Code: MCP-served skills as agent skills (Issue #66)

Written against the pre-v1 draft (August 2026). The index format, type: "skill-md", archive entries, and resource templates discussed below belong to that draft; v1 replaced the index with skills/list/skills/get, dropped archives and template entries (decision log, 2026-07-16), and makes digest verification a host MUST.

Implementation: tobi-oye/vscode#1 — on a microsoft/vscode fork (personal exploration, not submitted upstream)

Added skill:// discovery to VS Code and verified it against the Hugging Face MCP server (https://huggingface.co/mcp). VS Code discovered 8 skills and offered them to the model with no manual attachment.

Findings:

• VS Code already had the whole progressive-disclosure loop; it just had no MCP source. findAgentSkills() → name/description into a <skills> block → a model-facing skill tool. Same shape as skillsdotnet's SkillCatalog. Only the MCP origin was missing.
• The loading half needed no code. mcp-resource:// is already registered with VS Code's IFileService, so mapping skill://…/SKILL.md onto it is enough — the existing skill tool reads it and each read becomes a resources/read. "Treat filesystem and MCP skills identically" falls out for free.
• The index wire format had already moved past the checked-in draft. The live server serves {url, digest, frontmatter:{name, description}}; the draft here specifies top-level name/description and a required type: "skill-md". A parser written to the draft matches zero entries on a real server. Accepting both shapes is three lines.
• Per-context-computation discovery is an accidental DoS. findAgentSkills() runs on every context rebuild; discovery is two round trips per server. A naive version issued 20 index reads for one chat turn. Same shape as the incident behind hf-mcp-server's client denylist (#164, ~100k req/min). Cache the promise, keyed on connection state — not just server id, or a lookup made while a server was stopped pins an empty result.
• Cross-server name collisions are unspecified. Skill names are a flat namespace. The SEP ties the final URI segment to the frontmatter name but says nothing about two servers both serving deploy. Local-over-MCP is defensible; MCP-vs-MCP degenerates to discovery order. Worth an explicit note in the SEP.
• Provenance reaches the UI but not the model. The <skills> block emits only <name>, <description>, <file> — and <file> encodes an opaque server id. Loading is never ambiguous, but a model reasoning across servers has no legible origin signal.
• Archives cost 17 of 25 skills. Archive-only entries have no url to a SKILL.md, so a resource-reading host skips them. Archives were removed from SEP-2640 (decision log, 2026-07-16); until servers follow, such hosts see a fraction of what is offered.
• Adding a skill source touched four type surfaces that must agree (storage enum, a parallel source union, an ext-host DTO, a proposed API type). Missing two produced a runtime throw that broke chat while tsc stayed green.
• Identifying skills by URL alone was enough (#54). skill:// prefix plus /SKILL.md suffix, no _meta read — a two-line filter with nothing to negotiate. Because the last path segment must match the skill name, the name is readable off the URL, so a picker fills without fetching every SKILL.md. The tradeoff is that a URL carries no structured metadata; tags, versions or provenance would still need _meta.
• The index ships a digest that this client ignores. The live server sends sha256: per entry; nothing here verifies it, so a corrupted or swapped skill loads silently. fast-agent does check it. Worth settling whether verification is the host's job — the draft currently leans on it being "the transport's concern over an authenticated MCP connection".

Verification: tsc clean; 10 unit tests (incl. skill:// → mcp-resource:// round trip and index parsing against verbatim live output); 112 existing promptSyntax tests pass; discovery confirmed against the live server.

Note: none of the three defects above were caught by type-checking or unit tests — all surfaced only from running against a real server.

Open:

• Model invocation not demonstrated. Discovery and contribution are confirmed; no run yet shows the model choosing to load an MCP-served skill. Confounded by source builds being unable to reach the Copilot service (OSS product.json ships no OAuth client config) and by a small auto-routed model. Consistent with the adherence problems recorded elsewhere on this page.
• Resource templates parsed but not materialized (need the completion API).
• No resources/subscribe, so mid-session skill updates are missed.

McpGraph: Skills in MCP Server Repo

Date: Not documented

Implementation:

• Repository: TeamSparkAI/mcpGraph
• Author: Bob Dickinson
• Relevant artifacts: mcpgraphtoolkit/SKILL.md (875+ lines)

Approach tested: Related to Approach 5: Server Instructions Reference. The standalone skill lives beside the MCP server but is not formally connected to it.

Setup:

• Clients tested: Claude; specific client not documented
• Models tested: Not documented
• Configuration notes: The skill teaches agents to build directed graphs of MCP nodes and orchestrate tool calls

What was tested: Whether an agent discovers and follows a colocated skill before attempting to use the server tools.

Results:

• What worked: Adding a server instruction that told the agent to read the skill before using the tool caused Claude to load it reliably
• What didn't: Claude initially ignored the skill even when the skill and server had similar descriptions, and only read it after failing to use the server tools
• What was surprising: A single server instruction changed the loading order reliably

Requirements or design questions addressed: Reliable skill discovery and loading, and whether server instructions can connect an MCP server to colocated skill guidance.

Evidence and reproduction: These are community-reported observations; the client version, model version, and runnable reproduction are not documented.

Limitations:

• This workaround works for 1:1 skill-to-server case, but doesn't solve discovery — users installing from a registry don't know to also install the skill
• Distinguishes between "skill required to make the server work at all" vs. "skill that orchestrates tools you could use without it" — potentially different solutions needed

Skilljack MCP

Repo: olaservo/skilljack-mcp

Loads skills into tool descriptions. Uses dynamic tool updates to keep the skills manifest current.

Example eval approach and observations here: https://github.com/olaservo/skilljack-mcp/blob/main/evals/README.md

FastMCP 3.0 Skills Support

URL: gofastmcp.com/servers/providers/skills

FastMCP added skills support in version 3.0. Worth examining for alignment with other approaches.

Update model comparison (Feb 26 office hours):

• FastMCP supports more of a "pull" model for updating resources that have changed
• The skills-as-resources implementation in this repo (PR #16) watches for changes and allows clients to subscribe to resources via resources/subscribe and resources/updated notifications — more of a "push" model
• Both models are worth evaluating; the right choice is likely use-case specific

Related: jlowin/fastmcp#2694

PydanticAI Skills Support

PR: pydantic/pydantic-ai#3780

Introduces support for agent skills with a tools-based approach.

NimbleBrain: skill:// Resource Consolidation

Date: Not documented

Implementation:

• Repositories: mcp-ipinfo, mcp-webfetch, mcp-pdfco, mcp-folk, and mcp-brave-search
• Author: Mat Goldsborough (NimbleBrain)
• Relevant artifacts: Atomic MCP server repositories with skills exposed as skill:// resources

Approach tested: Approach 3: Skills as Tools and/or Resources.

Setup:

• Clients tested: Not documented
• Models tested: Not documented
• Configuration notes: Previously separate MCP server code, skills, and server.json registry metadata were consolidated into one repository per server

What was tested: Whether colocating skills with their MCP servers and exposing them as resources simplifies distribution while preserving or improving agent results.

Results:

• What worked: Consolidating the artifacts simplified build, versioning, and deployment; skills ship atomically with the tools they describe
• What worked: skill:// resources provided ephemeral availability without git cloning or client-side file-system access
• What worked: Quick tests produced the same or better results than injecting skills before the LLM call
• What didn't: Not documented
• What was surprising: Not documented

Requirements or design questions addressed: Installless skill availability, provenance through colocation, atomic versioning, and reuse of existing MCP resource primitives.

Evidence and reproduction: The linked repositories are reference implementations. The comparison with upstream skill injection was reported through community discussion; a test procedure and quantitative results are not documented.

Limitations: Client versions, model versions, test cases, and evaluation criteria are not documented, so the result cannot yet be reproduced precisely.

Sources and attribution:

"Skills living as skill:// resources on the server itself was the natural endpoint of that consolidation. The skill context is colocated with the tools it describes, versioned together, shipped together." — Mat Goldsborough (NimbleBrain), via Discord

Skill Reliability and Adherence

Multiple community members have independently reported that models do not reliably load or follow skill instructions, even when skills are preloaded in context. This is a cross-cutting behavioral problem, not specific to any single implementation approach.

Findings:

• Models appear to frequently ignore available skills, requiring hooks or repeated prompting to trigger skill loading
• Skill adherence appears to be "time-decaying" similar to other model instructions — models follow instructions initially but lose adherence as the context window grows and compaction occurs
• Behavior is model-specific: weaker models show lower success rates with lazy-loaded skills
• One effective workaround observed by Kryspin: wrapping skills in a subagent whose name or description mentions the skill topic
• Community desire for "skill autoloads" and "dynamic memory autoloads" as design patterns

Community input:

"Even Opus 4.6 needs to be constantly bugged to load skills when they're preloaded in the context already. I actually have a hook that reminds it to load skills and it still just doesn't a lot of the time." — Luca (AWS), via Discord

"I also have this problem with skills: they're useful… when used. Which isn't nearly often enough." — Jeremiah (FastMCP), via Discord

"Skills are ephemeral and/or time decaying — it clicks once and then give it some time and they lose the plot." — Kryspin (qcompute), via Discord

"I've seen lazy load skills with various degrees of success, actually looks like it might be model specific… [best pattern is] putting them in with a subagent that similarly named or mentions the topic in their description." — Kryspin (qcompute), via Discord

See also: #37 — Compare skill delivery mechanisms: file-based vs MCP-based

PHP MCP SDK + Symfony AI Mate: Skills as skill:// resources

Written against the pre-v1 draft (June 2026). References to skill://index.json, resource_templates/list, and mcp-resource-template entries describe that draft; v1 replaced the index with skills/list/skills/get and dropped template entries (decision log, 2026-07-16).

Server / SDK: modelcontextprotocol/php-sdk#372 — adds io.modelcontextprotocol/skills support to the official PHP MCP SDK Consumer: symfony/ai#2132 — ships Agent Skills in the Symfony AI "Mate" MCP server Contributor: Johannes Wachter (@wachterjohannes)

First PHP-ecosystem implementation of SEP-2640 (prior documented implementations are Python/TS). The SDK PR adds a one-line server affordance — addSkillsFromDirectory() — that walks a directory, registers each SKILL.md and its supporting files as skill:// resources, derives name/description from YAML frontmatter, enforces the spec's final-path-segment ↔ frontmatter-name rule, guards against path traversal, and serves a skill://index.json discovery index. The Mate PR ships two real skills colocated with the tools they orchestrate, including a multi-file skill with a references/ subdirectory.

Tested (works):

• Serving is covered by MCP Inspector stdio snapshot tests: resources/list, resources/read of a SKILL.md, of a supporting file, and of skill://index.json, plus resource_templates/list. Unit tests cover frontmatter parsing (BOM/CRLF, non-mapping rejection), the name↔segment rule, and resource-name sanitization. PHPStan level 6 and the full suite (792 tests) green.
• The directory model + relative supporting-file URIs resolve correctly in a non-Python implementation — e.g. skill://code-review/references/SECURITY.md is a sibling resource of skill://code-review/SKILL.md. Positive evidence the directory model travels across ecosystems.

_meta prefix — independent convergence (not a gap). Our SDK independently chose io.modelcontextprotocol.skills/ to namespace extra frontmatter fields on the resource descriptor — which matches the prefix SEP-2640 recommends ("When _meta keys are used for skill resources, implementations SHOULD use the io.modelcontextprotocol.skills/ reverse-domain prefix"). Useful corroboration of the recommended prefix. Note: the working-group repo-local draft (docs/sep-draft-skills-extension.md) does not yet include that sentence — its _meta paragraph ends at "…via the resource's _meta object." — so the SEP PR and this repo's copy have drifted and could be synced. (Discussed on SEP-2640.)

SDK implementation notes (not spec gaps):

• Resource-name uniqueness, not charset. The skill name charset ([a-z0-9-], ≤64, no leading/trailing hyphen) is a strict subset of the MCP resource-name charset, so the SKILL.md resource name can carry the frontmatter name directly — "resource name SHOULD equal frontmatter name" is satisfiable. The wrinkle is uniqueness: our SDK registers every resource under a unique name key, including a skill's supporting files (references/SECURITY.md) and skills that share a frontmatter name under different prefixes (acme/billing/refunds vs acme/support/refunds). So we derive a unique name from the URI path and keep the frontmatter name in title. Identity is the URI regardless — an SDK registration detail, not a SEP issue.
• Empty-payload capability serialization trap. An extension advertising an empty {} payload (as Skills does) serialized to [] rather than {} and had to be coerced. A likely footgun for any SDK implementing an empty-payload extension.
• symfony/yaml required for frontmatter parsing — the feature is non-functional without a YAML parser; frontmatter handling is a real dependency, not free.

Client consumption (observed from docs, not yet eval'd):

• Per current Claude Code documentation (June 2026), Claude Code loads skills from the filesystem and plugins only; it does not discover or load MCP-served skill:// resources as skills, and its MCP resource support is user-@-mention attachments, not model-driven resources/read. So end-to-end, model-driven consumption of MCP-served skills is not exercisable in Claude Code today — a data point for #38.
• FastMCP 3.0 (per this repo's existing findings) is the consumer best positioned to validate the serving half against; not yet done.

Remaining / untested:

• No model-adherence eval yet comparing filesystem vs. skill:// delivery.
• mcp-resource-template skill type (parameterized namespaces) is deferred in the SDK PR — the SkillType enum carries the value for forward-compat, but only skill-md entries are emitted; the template path is unimplemented and untested.
• Not yet tested against any client that implements model-driven skill:// loading.