Prepare docs for modelcontextprotocol.io (#728)

This PR reorganizes and extends the existing documentation in
preparation for publishing on https://modelcontextprotocol.io/.

The current plan is for docs to remain in this repo, and be synced to
the `modelcontextprotocol/modelcontextprotocol` whenever necessary
(e.g., accompanying a release of the registry). I have included a script
in
https://github.com/modelcontextprotocol/modelcontextprotocol/pull/1725
(`bin/pull-registry-docs.sh`) that can be run manually to perform the
sync.

If we later decide to permanently move the docs to the
`modelcontextprotocol/modelcontextprotocol` repo, that also works.

This PR puts all https://modelcontextprotocol.io/ docs into
`docs/modelcontextprotocol-io/`. Docs outside that subdirectory, such as
`docs/administration/admin-operations.md`, are intended to be accessible
solely via this repo. One major exception is the
`docs/reference/server-json/` subdirectory, which will depend on the
resolution of
https://github.com/modelcontextprotocol/modelcontextprotocol/issues/1649.
However, I think we could address that in a follow-up PR.

---------

Co-authored-by: Adam Jones <adamj+git@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>

docs/design/design-principles.md

@@ -0,0 +1,39 @@
+# MCP Registry Design Principles
+
+These are the core constraints that guide the design of the MCP Registry. They are not exhaustive, but they are the most important principles that we will use to evaluate design decisions.
+
+## 1. Single Source of Truth
+
+The registry serves as the authoritative metadata repository for publicly-available MCP servers, both locally-run and remote, open source and closed source. Server creators publish once, and all consumers (MCP clients, aggregators, etc.) reference the same canonical data.
+
+## 2. Minimal Operational Burden
+
+- Design for low maintenance and operational overhead
+- Delegate complexity to existing services where possible (GitHub for auth, npm/PyPI/NuGet for packages)
+- Avoid features that require constant human intervention or moderation
+- Build for reasonable downtime tolerance (24h acceptable) by having consumers cache data for their end-users
+
+## 3. Vendor Neutrality
+
+- No preferential treatment for specific servers or organizations
+- No built-in ranking, curation, or quality judgments
+- Let consumers (MCP clients, aggregators) make their own curation decisions
+
+## 4. Meets Industry Security Standards
+
+- Leverage existing package registries (npm, PyPI, NuGet, Docker Hub, etc.) for source code distribution, obviating the need to reinvent source code security
+- Use mechanisms like DNS verification, OAuth to provide base layer of authentication and trust
+- Implement rate limiting, field validation, and blacklisting to prevent abuse
+
+## 6. Reusable, Extensible Shapes; Not Infrastructure
+
+- API shapes (OpenAPI, server.json) designed for reuse
+- Enable private/internal registries using same formats
+- Don't mandate infrastructure reuse - focus on interface compatibility
+
+## 7. Progressive Enhancement
+
+- Start with MVP that provides immediate value
+- Build foundation that supports future features
+- Don't over-engineer for hypothetical needs
+- Each milestone should be independently valuable

docs/design/ecosystem-vision.md

@@ -0,0 +1,56 @@
+# Ecosystem Vision
+
+How the MCP Registry fits into the broader ecosystem and our vision for the future.
+
+## The Registry Ecosystem
+
+The MCP registry provides MCP clients with a list of MCP servers, like an app store for MCP servers. (In the future it might do more, like also hosting a list of clients).
+
+There are two parts to the registry project:
+
+1. 🟦 **The MCP registry spec**: An [API specification](../reference/api/) that allows anyone to implement a registry.
+2. 🟥 **The Official MCP registry**: A hosted registry following the MCP registry spec at [`registry.modelcontextprotocol.io`](https://registry.modelcontextprotocol.io). This serves as the **authoritative repository** for publicly-available MCP servers. Server creators publish once, and all consumers (MCP clients, aggregators, marketplaces) reference the same canonical data. This is owned by the MCP open-source community, backed by major trusted contributors to the MCP ecosystem such as Anthropic, GitHub, PulseMCP and Microsoft.
+
+The registry is built around the [`server.json`](../reference/server-json/) format - a standardized way to describe MCP servers that works across discovery, initialization, and packaging scenarios.
+
+In time, we expect the ecosystem to look a bit like this:
+
+![](./ecosystem-diagram.excalidraw.svg)
+
+Note that MCP registries are _metaregistries_. They host metadata about packages, but not the package code or binaries. Instead, they reference other package registries (like NPM, PyPi or Docker) for this.
+
+Additionally, we expect clients pull from _subregistries_. These subregistries add value to the registry ecosystem by providing curation, or extending it with additional metadata. The Official MCP registry expects a lot of API requests from ETL jobs from these subregistries.
+
+## Registry vs Package Registry
+
+Key distinction: MCP Registries are **metaregistries**.
+
+- **Package registries** (npm, PyPI, Docker Hub) host actual code/binaries
+- **The MCP Registry** hosts metadata pointing to those packages
+
+```
+MCP Registry: "weather-server v1.2.0 is at npm:weather-mcp"
+NPM Registry: [actual weather-mcp package code]
+```
+
+## Official vs Community Registries
+
+**Official MCP Registry** (`registry.modelcontextprotocol.io`):
+- Canonical source for publicly-available servers
+- Community-owned, backed by trusted contributors
+- Focuses on discoverability and basic metadata
+
+**Subregistries** (Smithery, PulseMCP, etc.):
+- Add value through curation, ratings, enhanced metadata
+- ETL from official registry + additional annotations
+- Serve specific communities or use cases
+
+## How Servers Are Represented
+
+Each server entry contains:
+- **Identity**: Unique name (`io.github.user/server-name`)
+- **Packages**: Where to download it (`npm`, `pypi`, `docker`, etc.)
+- **Runtime**: How to execute it (args, env vars)
+- **Metadata**: Description, capabilities, version
+
+This is stored in a standardized `server.json` format that works across discovery, installation, and execution.

docs/design/roadmap.md

@@ -0,0 +1,31 @@
+# MCP Registry Roadmap
+
+This is a high-level roadmap for the MCP Registry. It is subject to change and not exhaustive, but it outlines the general thinking of the sequencing and scope of our work in this repository.
+
+This roadmap may occasionally drift out of date. Please review [Issues](https://github.com/modelcontextprotocol/registry/issues) (and corresponding Labels) for the most current work in progress.
+
+## Current Status
+
+The initial version of the MCP Registry is actively being developed. The initial focus is on delivering a REST API to which server creators can publish, and aggregator/marketplace consumers can ETL.
+
+## Phase 1: MVP/Go-Live (Current Focus)
+
+See the [go-live blocker issues](https://github.com/modelcontextprotocol/registry/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22go-live%20blocker%22).
+
+## Backlog (Future Work, may be moved to out of scope)
+
+- [ ] UI implementation
+- [ ] Store and surface other data besides servers (e.g. [clients](https://modelcontextprotocol.io/clients), resources)
+- [ ] Download count tracking
+- [ ] Internationalization (i18n)
+
+## Out of Scope (Not Planned)
+
+- **Source code hosting**: The registry will never host actual server code
+- **Quality rankings**: No built-in server quality assessments or rankings
+- **Curation**: No editorial decisions about which servers are "better"
+- **Unified runtime**: Not solving how servers are executed
+- **Server hosting**: The registry does not provide hosting for servers
+- **Search engine**: The registry will not provide a commercial grade search engine for servers
+- **Server tags or categories**: Not supported, to reduce moderation burden
+- **Server rankings**: The registry will not rank servers by subjective measures of quality

docs/design/tech-architecture.md

@@ -0,0 +1,202 @@
+# MCP Registry Technical Architecture
+
+This document describes the technical architecture of the MCP Registry, including system components, deployment strategies, and data flows.
+
+## System Overview
+
+The MCP Registry is designed as a lightweight metadata service that bridges MCP server creators with consumers (MCP clients and aggregators).
+
+## Core Components
+
+### REST API (Go)
+
+The main application server implemented in Go, providing:
+- Public read endpoints for server discovery
+- Authenticated write endpoints for server publication
+- GitHub OAuth integration (extensible to other providers)
+- DNS verification system (optional for custom namespaces)
+
+### Database (PostgreSQL)
+
+Primary data store for:
+- Versioned server metadata (server.json contents)
+- User authentication state
+- DNS verification records
+
+### CDN Layer
+
+Critical for scalability:
+- Caches all public read endpoints
+- Reduces load on origin servers
+- Enables global distribution
+- Designed for daily consumer polling patterns
+
+### CLI Tool
+
+Developer interface for:
+- Server publication workflow
+- GitHub OAuth flow
+- DNS verification
+
+## Deployment Architecture
+
+### Kubernetes Deployment (Helm)
+
+The registry is designed to run on Kubernetes using Helm charts:
+
+```mermaid
+graph TB
+    subgraph "Kubernetes Cluster"
+        subgraph "Namespace: mcp-registry"
+            subgraph "Registry Service"
+                LB[Load Balancer<br/>:80]
+                RS[Registry Service<br/>:8080]
+                RP1[Registry Pod 1]
+                RP2[Registry Pod 2]
+                RP3[Registry Pod N]
+            end
+            
+            subgraph "Database Service"
+                DBS[DB Service<br/>:27017]
+                SS[StatefulSet]
+                PV[Persistent Volume]
+            end
+            
+            subgraph "Secrets"
+                GHS[GitHub OAuth Secret]
+            end
+        end
+    end
+    
+    LB --> RS
+    RS --> RP1
+    RS --> RP2
+    RS --> RP3
+    RP1 --> DBS
+    RP2 --> DBS
+    RP3 --> DBS
+    DBS --> SS
+    SS --> PV
+    RP1 -.-> GHS
+    RP2 -.-> GHS
+    RP3 -.-> GHS
+```
+
+## Data Flow Patterns
+
+### 1. Server Publication Flow
+
+```mermaid
+sequenceDiagram
+    participant Dev as Developer
+    participant CLI as CLI Tool
+    participant API as Registry API
+    participant DB as Database
+    participant GH as GitHub
+    participant DNS as DNS Provider
+    
+    Dev->>CLI: mcp publish server.json
+    CLI->>CLI: Validate server.json
+    CLI->>GH: OAuth flow
+    GH-->>CLI: Access token
+    CLI->>API: POST /servers
+    API->>GH: Verify token
+    API->>DNS: Verify domain (if applicable)
+    API->>DB: Store metadata
+    API-->>CLI: Success
+    CLI-->>Dev: Published!
+```
+
+### 2. Consumer Discovery Flow
+
+```mermaid
+sequenceDiagram
+    participant Client as MCP Client Host App
+    participant INT as Intermediary<br/>(Marketplace/Aggregator)
+    participant CDN as CDN Cache
+    participant API as Registry API
+    participant DB as Database
+    
+    Note over INT,CDN: Daily ETL Process
+    INT->>CDN: GET /servers
+    alt Cache Hit
+        CDN-->>INT: Cached response
+    else Cache Miss
+        CDN->>API: GET /servers
+        API->>DB: Query servers
+        DB-->>API: Server list
+        API-->>CDN: Response + cache headers
+        CDN-->>INT: Response
+    end
+    INT->>INT: Process & enhance data
+    INT->>INT: Store in local cache
+    
+    Note over Client,INT: Real-time Client Access
+    Client->>INT: Request server list
+    INT-->>Client: Curated/enhanced data
+```
+
+### 3. DNS Verification Flow
+
+```mermaid
+sequenceDiagram
+    participant User as User
+    participant CLI as CLI Tool
+    participant API as Registry API
+    participant DNS as DNS Provider
+    participant DB as Database
+    
+    User->>CLI: mcp verify-domain example.com
+    CLI->>API: POST /verify-domain
+    API->>API: Generate verification token
+    API->>DB: Store pending verification
+    API-->>CLI: TXT record: mcp-verify=abc123
+    CLI-->>User: Add TXT record to DNS
+    User->>DNS: Configure TXT record
+    User->>CLI: Confirm added
+    CLI->>API: POST /verify-domain/check
+    API->>DNS: Query TXT records
+    DNS-->>API: TXT records
+    API->>API: Validate token
+    API->>DB: Store verification
+    API-->>CLI: Domain verified
+    CLI-->>User: Success!
+```
+
+### 4. Admin OIDC Authentication Flow
+
+For registry administration, users with @modelcontextprotocol.io Google Cloud Identity accounts can authenticate using OIDC:
+
+```mermaid
+sequenceDiagram
+    participant Admin as Admin User
+    participant CLI as Admin CLI
+    participant GCP as Google Cloud Identity
+    participant API as Registry API
+    
+    Admin->>CLI: Request admin token
+    CLI->>GCP: gcloud auth print-identity-token --audiences=mcp-registry
+    GCP-->>CLI: ID Token (with hd: "modelcontextprotocol.io")
+    CLI->>API: POST /v0/auth/oidc {"oidc_token": "eyJ..."}
+    API->>GCP: Verify token signature (JWKS)
+    API->>API: Validate claims (issuer, audience, hd)
+    API->>API: Grant admin permissions (edit: *, publish: *)
+    API-->>CLI: Registry JWT Token
+    CLI->>API: POST /admin/* (with Registry JWT)
+    API->>API: Validate JWT + permissions
+    API-->>CLI: Admin operation success
+```
+
+**Usage:**
+```bash
+# Get Google Cloud Identity token
+ID_TOKEN=$(gcloud auth print-identity-token)
+
+# Exchange for Registry JWT token  
+REGISTRY_TOKEN=$(curl -X POST /v0/auth/oidc \
+  -H "Content-Type: application/json" \
+  -d '{"oidc_token": "'$ID_TOKEN'"}' | jq -r .registry_token)
+
+# Use for admin operations
+curl -H "Authorization: Bearer $REGISTRY_TOKEN" /v0/...
+```