From e4de42cf8d9f6d409aee3a5cf9e2d80d477bd123 Mon Sep 17 00:00:00 2001 From: Ola Hungerford Date: Tue, 3 Mar 2026 13:12:01 -0700 Subject: [PATCH] Add notes to docs from Discord and office hours discussions (#36) * Add community insights from Discord threads and meeting notes Distill Discord discussions (Feb 14-24) and Feb 26 office hours into existing docs: - New findings: NimbleBrain skill:// resource consolidation, skill reliability/adherence issues across implementations - Expanded approaches with distribution/provenance considerations, resources adoption insight, and domain-level discovery (.well-known) - Expanded open questions on dependencies and distribution trust - Added Cloudflare Agent Skills Discovery RFC and NimbleBrain reference implementations to related work Co-Authored-By: Claude Opus 4.6 * Remove trust note since its already mentioned in the doc separately * Clarify attribution for community insights on skill dependencies in open questions * Refine findings on model skill adherence and community workarounds * Remove outdated comparison section and streamline evaluation insights in experimental findings * Remove unnecessary blank line in experimental findings documentation * Add SEP-2093 proposal to related work documentation --------- Co-authored-by: Claude Opus 4.6 --- docs/approaches.md | 23 ++++++++++++ docs/experimental-findings.md | 67 ++++++++++++++++++++++++----------- docs/open-questions.md | 22 ++++++++++++ docs/related-work.md | 5 +++ docs/use-cases.md | 12 ++++++- 5 files changed, 107 insertions(+), 22 deletions(-) diff --git a/docs/approaches.md b/docs/approaches.md index 1653c48..5d862aa 100644 --- a/docs/approaches.md +++ b/docs/approaches.md @@ -11,6 +11,7 @@ Several design considerations are emerging from community discussion: - **Don't assume how tool paradigms will evolve.** The conceptual surface of skills shouldn't bake in assumptions about how tools develop. That doesn't preclude skills being implemented as a well-known tool, but the design should not couple skills to any particular tool evolution path. - **Let the primitive choice follow from the use case.** The answer may not be "resources" or "new primitive" — it may be both, depending on the interaction pattern. Some skills are context for the model. Some are context for the human. Some are both. The delivery mechanism should support that range. ([See related thread on SEP 2076](https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2076#discussion_r2736299627)) - **Minimize ecosystem complexity.** The broader AI tooling ecosystem is experiencing complexity fatigue — too many overlapping concepts (servers, skills, plugins, hooks, agents) erode credibility and adoption. Whatever approach the IG recommends should reuse existing MCP primitives where possible and only introduce new surface area when there's a clear case that existing primitives can't serve the need. ([See related issue](https://github.com/modelcontextprotocol/experimental-ext-skills/issues/14)) +- **Skills are context, and the pattern extends beyond workflows.** The skill format and progressive disclosure pattern apply equally to organizational knowledge and in-context learning — not just tool-usage workflows. ## Central Tension: Convention vs. Protocol Extension @@ -76,12 +77,34 @@ Examples: > "The only slight concern I have is the idea that there are still 'first class skills' (skills that agents recognize as skills, can be presented as skills through the user agent, can be bundled with subagents, etc) and these sort of 'skills as context' approaches where the agent can certainly discover and ingest the skills data, but possibly with some differences compared to how they would apply first class skills." — [Bob Dickinson](https://github.com/TeamSparkAI) +See also [notes from Feb 26th Office Hours](https://github.com/modelcontextprotocol/modelcontextprotocol/discussions/2316) + This approach may also: - Use resource templates for parameterized skill discovery - Use Prompts for explicit skill invocation - Use `tools/listChanged` and other notifications for dynamic updates without server re-initialization +### Distribution and Provenance Considerations + +Several design considerations have been suggested in community discussion and proposals around how skills are distributed over MCP: + +- **Ephemeral availability:** Skills should be available while a server is installed, without requiring a separate permanent install step. Clients could optionally offer to permanently install skills discovered from servers. +- **Provenance metadata:** The server URL for remote servers should be bundled into skill frontmatter metadata, so skills carry their origin and source identity. +- **SDK ergonomics:** It would be valuable at the SDK level to provide frontmatter and body content separately in code, rather than requiring authors to construct a single markdown blob. +- **Trust model alignment:** Skill trust should align with existing MCP trust — based on server trust. The community consensus is to discourage using MCP as a mechanism for providing a skills marketplace for arbitrary third-party content. +- **No OS-level packaging:** MCP servers should not provide platform-specific bundles (tar.gz, etc.); skills should remain text-based context. MCP has no notion of operating system or environment on the receiving side. +- **Git-based distribution:** Versioned distribution via git (tags, pinned refs) can be viable without a formal registry. Clare Liguori (AWS) noted that Terraform operated without a formal registry for a long time — Feb 26 office hours. +- **Domain-level discovery:** The [Agent Skills Discovery RFC](https://github.com/cloudflare/agent-skills-discovery-rfc) proposes `/.well-known/skills/` for organizations to publish skills at predictable URLs with content integrity (SHA-256 digests). This is complementary to MCP — it handles discovery and distribution while MCP handles runtime consumption. + +**Community input:** + +> "Installless/temporary/ephemeral skill availability while server is installed feels like a good pattern. Clients could optionally offer to permanently install." — [Sam Morrow](https://github.com/SamMorrowDrums) (GitHub), via Discord + +> "We should probably stipulate that the server URL for remote servers also be bundled into frontmatter metadata, ideally given the way users may discover these autonomously, encoding source identity in a way that is collocated will be good when things go wrong." — [Sam Morrow](https://github.com/SamMorrowDrums) (GitHub), via Discord + +> "We have no notion of operating system/environment and I don't think MCP servers providing tar.gz bundles of arbitrary content is a great idea… the trust model is same as MCP, based on server trust, so broadly I think we want to discourage using MCP as a mechanism for providing a skills marketplace for arbitrary 3rd party content." — [Sam Morrow](https://github.com/SamMorrowDrums) (GitHub), via Discord + ### Variant: Skills via Sampling Instead of exposing skill tools to the main agent, use MCP's Sampling with Tools capability ([SEP-1577](https://github.com/modelcontextprotocol/modelcontextprotocol/pull/1577)). The server requests a sampling call with skill-specific tools (`read_skill_md`, `execute_script`, etc.) that are only visible during that sampling request. This keeps skill tools hidden from the main agent, addressing tool bloat. The server orchestrates skill execution; the main agent just sees the result. diff --git a/docs/experimental-findings.md b/docs/experimental-findings.md index d20297c..01cf717 100644 --- a/docs/experimental-findings.md +++ b/docs/experimental-findings.md @@ -26,27 +26,7 @@ Bob Dickinson built a standalone SKILL.md file that lives in the same repo as th Loads skills into tool descriptions. Uses dynamic tool updates to keep the skills manifest current. -## worfcat's Three-Way Comparison - -Tested skills exposed in three ways across multiple clients: - -### 1. Skills as MCP Resources (with a simple `validate_skill` tool) - -**Result: Bad.** The model never used them or looked at them unless explicitly asked. - -### 2. Skills as Tools in an MCP Server (`list_skills` and `read_skills`) - -**Result: Mixed.** Needed some tool description tuning to get used. More brittle than native. Worked with Claude Code, OpenAI agent builder, and a few other clients. - -### 3. Native Skills (installing in `skills/` folder of Claude Code) - -**Result: Best.** Easiest, worked out of the box. - -**Key insight:** - -> "There is a many-many relationship between skills and tools. A skill should not require pieces of instruction like 'here's how you use this single tool'. If an instruction of how to use an MCP server is copied across skills, it should live at the server level. Context decomposition basically." - -**Note:** Claude Code's skill implementation is just a tool call — `Skill()` returns the SKILL.md file. Nothing fancy (at least for now). +Example eval approach and observations here: https://github.com/olaservo/skilljack-mcp/blob/main/evals/README.md ## FastMCP 3.0 Skills Support @@ -54,6 +34,12 @@ Tested skills exposed in three ways across multiple clients: FastMCP added skills support in version 3.0. Worth examining for alignment with other approaches. +**Update model comparison (Feb 26 office hours):** + +- FastMCP supports more of a "pull" model for updating resources that have changed +- The skills-as-resources implementation in this repo ([PR #16](https://github.com/modelcontextprotocol/experimental-ext-skills/pull/16)) watches for changes and allows clients to subscribe to resources via `resources/subscribe` and `resources/updated` notifications — more of a "push" model +- Both models are worth evaluating; the right choice is likely use-case specific + **Related:** [jlowin/fastmcp#2694](https://github.com/jlowin/fastmcp/issues/2694) ## PydanticAI Skills Support @@ -61,3 +47,42 @@ FastMCP added skills support in version 3.0. Worth examining for alignment with **PR:** [pydantic/pydantic-ai#3780](https://github.com/pydantic/pydantic-ai/pull/3780) Introduces support for agent skills with a tools-based approach. + +## NimbleBrain: skill:// Resource Consolidation + +[Mat Goldsborough](https://github.com/mgoldsborough) (NimbleBrain) had previously maintained separate components for MCP server code, a skills monorepo, and registry metadata with `server.json`. After community discussion, he consolidated into single atomic repos per server with skills exposed as `skill://` resources directly on the server. + +**Findings:** + +- Collapsing three separate artifacts into one repo simplified build, versioning, and deployment — skills are colocated with the tools they describe and shipped atomically +- `skill://` resources enable ephemeral/installless availability: skill context is present while the server is installed and disappears when it disconnects, with no git cloning or file system access required on the client side +- Quick tests showed same or better results compared to the previous approach of injecting skills upstream before the LLM call +- Validates the skills-as-resources approach documented in [Approach 3](approaches.md#3-skills-as-tools-andor-resources) + +**Reference implementations:** [mcp-ipinfo](https://github.com/NimbleBrainInc/mcp-ipinfo), [mcp-webfetch](https://github.com/NimbleBrainInc/mcp-webfetch), [mcp-pdfco](https://github.com/NimbleBrainInc/mcp-pdfco), [mcp-folk](https://github.com/NimbleBrainInc/mcp-folk), [mcp-brave-search](https://github.com/NimbleBrainInc/mcp-brave-search) + +**Community input:** + +> "Skills living as skill:// resources on the server itself was the natural endpoint of that consolidation. The skill context is colocated with the tools it describes, versioned together, shipped together." — [Mat Goldsborough](https://github.com/mgoldsborough) (NimbleBrain), via Discord + +## Skill Reliability and Adherence + +Multiple community members have independently reported that models do not reliably load or follow skill instructions, even when skills are preloaded in context. This is a cross-cutting behavioral problem, not specific to any single implementation approach. + +**Findings:** + +- Models appear to frequently ignore available skills, requiring hooks or repeated prompting to trigger skill loading +- Skill adherence appears to be "time-decaying" similar to other model instructions — models follow instructions initially but lose adherence as the context window grows and compaction occurs +- Behavior is model-specific: weaker models show lower success rates with lazy-loaded skills +- One effective workaround observed by Kryspin: wrapping skills in a subagent whose name or description mentions the skill topic +- Community desire for "skill autoloads" and "dynamic memory autoloads" as design patterns + +**Community input:** + +> "Even Opus 4.6 needs to be constantly bugged to load skills when they're preloaded in the context already. I actually have a hook that reminds it to load skills and it still just doesn't a lot of the time." — Luca (AWS), via Discord + +> "I also have this problem with skills: they're useful… when used. Which isn't nearly often enough." — Jeremiah (FastMCP), via Discord + +> "Skills are ephemeral and/or time decaying — it clicks once and then give it some time and they lose the plot." — Kryspin (qcompute), via Discord + +> "I've seen lazy load skills with various degrees of success, actually looks like it might be model specific… [best pattern is] putting them in with a subagent that similarly named or mentions the topic in their description." — Kryspin (qcompute), via Discord diff --git a/docs/open-questions.md b/docs/open-questions.md index 0c9838e..29d2687 100644 --- a/docs/open-questions.md +++ b/docs/open-questions.md @@ -4,6 +4,8 @@ Should skills be discoverable through registry metadata ("if you install this server, also install this skill") or contained within the MCP server itself? +A third option is emerging: domain-level discovery via `/.well-known/skills/` (see [Agent Skills Discovery RFC](https://github.com/cloudflare/agent-skills-discovery-rfc)). This decouples skill discovery from both registries and MCP servers — an organization publishes skills at a predictable URL on its own domain. This could complement MCP-level discovery rather than replace it: `.well-known` handles "find available skills," MCP handles "load and use them at runtime." + ## 2. How do "first-class" skills differ from "skills as context"? 'Native' agent skills can be presented through the user agent, bundled with subagents, etc. Do MCP-surfaced skills lose capabilities compared to directly installed skills? This question also ties into a more general topic about context-as-resources discoverability and standardization of client host behavior — it's not just a skills-specific framing question. @@ -18,6 +20,20 @@ Or is the separation between "primitive server" and "skill that uses the primiti A skill orchestrating tools from several servers can't live in any single server's instructions. +**Related:** [agentskills/agentskills#110](https://github.com/agentskills/agentskills/issues/110) — Discusses how skills should declare their tool/server dependencies. The lack of explicit dependency configuration makes multi-server skill execution unpredictable: if required servers and tools aren't already loaded, the skill can't reliably execute. + +**Emerging proposal — host-mediated dependency resolution:** Skills would declare MCP servers and/or tools as dependencies in their frontmatter. The host mediates availability: if required dependencies are not present, the skill frontmatter should not be loaded into model context (it is effectively not an available skill). This model also enables local caching of skills — they can be downloaded once and used offline as long as their dependencies are available. + +The agentskills.io spec currently has a freeform [compatibility field](https://agentskills.io/specification#compatibility-field) but no formal dependency mechanism. Some existing tools (e.g., skills.sh) handle dependencies implicitly by instructing agents to install via bash/npm/uv. Skills may also be composable (skill-to-skill dependencies) — see [Use Case 3](use-cases.md#3-multi-server-composition). + +**Community input:** + +> "If [required tools/servers are] not available then the skill frontmatter shouldn't be loaded into model context, as it is effectively not an available skill. This also means that you can cache skills locally." — [Peder Holdgaard Pedersen](https://github.com/PederHP) (Saxo Bank), via Discord + +> "If there is some standard that we can introduce to specify skill dependencies… and all platforms can read that and load the skill or not based on that would be amazing." — Sunish Sheth (Databricks), via Discord + +> "npx skills is ok for simple use-cases but doesn't work for dependencies across Skill — i.e. if Skill 1 uses Skill 2 — which is one of the benefits of Skills — that they are composable." — [Kaxil Naik](https://github.com/kaxil) (Astronomer), via Discord + ## 5. Do clients actually leverage skills when presented via MCP? Early experiments suggest they do, but more rigorous testing is needed. @@ -64,6 +80,12 @@ If skills can be abused for prompt injection, what mitigations should be spec'd? Proposed mitigations: skills are untrusted docs not directives; clients MUST NOT auto-apply without explicit policy; skills should be presented with provenance and be optionally gated. +The distribution channel itself also has trust implications. Current skill distribution via git repos can be problematic from a security and trust perspective, particularly for enterprises. MCP's authenticated server model provides a more controlled distribution channel, but the trust model should align with existing MCP trust boundaries — not position MCP as a marketplace for arbitrary third-party content. + +> "Current distribution of skills is a nightmare in terms of security and trust — both from an end-user and enterprise point-of-view. A git repo is a problematic distribution channel." — [Peder Holdgaard Pedersen](https://github.com/PederHP) (Saxo Bank), via Discord + +> "The trust model is same as MCP, based on server trust, so broadly I think we want to discourage using MCP as a mechanism for providing a skills marketplace for arbitrary 3rd party content." — [Sam Morrow](https://github.com/SamMorrowDrums) (GitHub), via Discord + ## 11. Should the control model be use-case specific? Perhaps resources (application-controlled) for some use cases, tools (model-controlled) for others? Can a convention support both? diff --git a/docs/related-work.md b/docs/related-work.md index 30b6a67..4ff6d79 100644 --- a/docs/related-work.md +++ b/docs/related-work.md @@ -5,7 +5,9 @@ | Proposal | Venue | Description | | :--- | :--- | :--- | | [SEP-2076](https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2076) | MCP Spec | Agent Skills as a first-class MCP primitive | +| [SEP-2093](https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2093) | MCP Spec | Resource Contents Metadata and Capabilities: scoped `resources/list`, per-resource capabilities, `resources/metadata` endpoint | | [skills.json format proposal](https://github.com/modelcontextprotocol/registry/discussions/895) | MCP Registry | Skills metadata in registry schema | +| [Agent Skills Discovery via Well-Known URIs](https://github.com/cloudflare/agent-skills-discovery-rfc) | Cloudflare | Domain-level skill discovery using `/.well-known/skills/` (RFC 8615); includes integrity verification and progressive disclosure | ## Implementations @@ -29,6 +31,7 @@ Original implementations from external repositories (example implementations in | Strands Agents MCP server | AWS | [github.com/strands-agents/mcp-server](https://github.com/strands-agents/mcp-server/) | Docs-as-MCP: TF-IDF search + doc fetch | | AWS MCP server | AWS | [docs.aws.amazon.com/aws-mcp/…](https://docs.aws.amazon.com/aws-mcp/latest/userguide/understanding-mcp-server-tools.html) | `retrieve_agent_sop` (skills) + `call_aws` (tool) | | Kiro powers directory | Kiro | [github.com/kirodotdev/powers](https://github.com/kirodotdev/powers/) | Plugin directory bundling skills + MCP servers | +| NimbleBrain skill:// servers | NimbleBrain | [github.com/NimbleBrainInc](https://github.com/NimbleBrainInc) | skill:// resource colocation examples: [mcp-ipinfo](https://github.com/NimbleBrainInc/mcp-ipinfo), [mcp-webfetch](https://github.com/NimbleBrainInc/mcp-webfetch), [mcp-pdfco](https://github.com/NimbleBrainInc/mcp-pdfco), [mcp-folk](https://github.com/NimbleBrainInc/mcp-folk), [mcp-brave-search](https://github.com/NimbleBrainInc/mcp-brave-search) | ## External Resources @@ -39,4 +42,6 @@ Original implementations from external repositories (example implementations in - **Apache Airflow AIP-91** (MCP integration): [cwiki.apache.org/…/AIP-91+-+MCP](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-91+-+MCP) - **llms.txt convention:** [llmstxt.org](https://llmstxt.org/) — Convention for making documentation LLM-accessible; used by [MCPDoc](https://github.com/langchain-ai/mcpdoc) in a way similar to how skills work. - **AWS Agent SOPs:** [docs.aws.amazon.com/…/agent-sops](https://docs.aws.amazon.com/aws-mcp/latest/userguide/agent-sops.html) — Pre-built operational workflows as skill-like guidance +- **Skill dependency declaration:** [agentskills/agentskills#110](https://github.com/agentskills/agentskills/issues/110) — Discusses how skills should declare their tool/server dependencies +- **Agent Skills Discovery RFC** (Matt Silverlock / Cloudflare): [github.com/cloudflare/agent-skills-discovery-rfc](https://github.com/cloudflare/agent-skills-discovery-rfc) — Proposes `/.well-known/skills/` for domain-level skill discovery using RFC 8615. Defines an `index.json` discovery index with progressive disclosure (name/description at discovery time, full SKILL.md on activation), SHA-256 content integrity, and optional archive distribution. Complementary to MCP-level discovery: `.well-known` answers "what skills does this domain publish?" while MCP answers "how does the agent consume them at runtime?" - **Video background:** [youtube.com/watch?v=CEvIs9y1uog](https://www.youtube.com/watch?v=CEvIs9y1uog) diff --git a/docs/use-cases.md b/docs/use-cases.md index 7db97e1..8d5cc05 100644 --- a/docs/use-cases.md +++ b/docs/use-cases.md @@ -18,7 +18,9 @@ Workflows that reference tools conditionally based on context, requiring rich st ## 3. Multi-Server Composition -Skills that leverage tools from multiple off-the-shelf servers where you can't (or don't want to) modify their individual instructions. +Skills that leverage tools from multiple off-the-shelf servers where you can't (or don't want to) modify their individual instructions. Skills can also make it more natural to translate ad-hoc tool-call-based workflows into scripted workflows, without fundamentally changing the content of the skill. + +**Related:** [agentskills/agentskills#110](https://github.com/agentskills/agentskills/issues/110) — Skill dependency declaration **Community input:** @@ -26,6 +28,8 @@ Skills that leverage tools from multiple off-the-shelf servers where you can't ( > "The ecosystem has been too focused on the server being the main deliverable in some ways, and actually there's a lot that can be done in terms of composition that we miss by people generally imagining their code as being the server boundary and not providing functionality more as a library." — [Sam Morrow](https://github.com/SamMorrowDrums) +Beyond multi-server tool orchestration, skills themselves may be composable — one skill depending on another skill's output or behavior. This extends the dependency model beyond tool availability to skill availability, and raises questions about declarative dependency metadata. See [Open Question 4](open-questions.md#4-how-should-skills-relate-to-multiple-servers) for the emerging proposal on host-mediated dependency resolution. + ## 4. Progressive Disclosure Skills broken into linked sets of files for effective context management, loaded progressively as the agent needs them rather than all at once. @@ -34,6 +38,12 @@ Skills broken into linked sets of files for effective context management, loaded > "Especially mimicking progressive disclosure via resources and dynamically adding new ones as the agent reads pieces of the skill has been quite handy!" — [Ozz / Juan Antonio Osorio](https://github.com/JAORMX) +**Practical limitation:** In non-visual execution environments (CLI agents, headless), skills lack the "collapse" mechanism of UI-based progressive disclosure. Without a UI affordance to hide/show content, all disclosed context remains in the model's context window, contributing to the adherence decay problem (see [Skill Reliability and Adherence](experimental-findings.md#skill-reliability-and-adherence)). Tool-based expansion partially addresses this, since pruning a tool call also collapses the context, but this is not a general solution. + +**At scale:** A server may expose hundreds or thousands of skills, but a client may only need a handful. This reinforces the need for selective loading via progressive disclosure rather than loading all skill content at once. + +> "A server can contain 100s or 1000s of skills as an extreme but a client might only need handful of them." — [Kaxil Naik](https://github.com/kaxil) (Astronomer), via Discord + **Related:** [Anthropic's guidance on progressive disclosure](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills) ## 5. Server-Skill Pairing