## Summary
Follow-up to #1211 to encode the GCP API and IAM dependencies its
node-SA bindings rely on. After #1211 merged, the staging deploy hit two
undocumented prereqs in sequence — Cloud Resource Manager API not
enabled (because `projects.NewIAMMember` calls SetIamPolicy through
CRM), then the Pulumi service account missing
`roles/resourcemanager.projectIamAdmin`. Both fixed manually to unblock;
this PR makes them explicit so they don't bite again.
## Changes
**`ensureRequiredAPIs` adopts five GCP APIs** as Pulumi-managed
`projects.Service` resources, all with `DisableOnDestroy: false` and
`DisableDependentServices: false` so a Pulumi destroy/refactor can never
disable a shared API:
| API | Why |
|-----|-----|
| `cloudresourcemanager` | `projects.NewIAMMember` → SetIamPolicy |
| `compute` | `compute.GetDefaultServiceAccount` invoke + GKE |
| `container` | GKE cluster |
| `logging` | fluentbit-gke ships container logs |
| `monitoring` | managed Prometheus collector ships metrics |
CRM is created explicitly (not in the loop) so callers get a direct
reference for `pulumi.DependsOn`. Storage is intentionally **not**
Pulumi-managed — Pulumi state itself lives in a GCS bucket,
chicken-and-egg.
**`grantNodeServiceAccountRoles`** now `DependsOn` the CRM
`projects.Service` for the four IAM bindings, and uses
`compute.GetDefaultServiceAccount` to derive the SA email (Compute API
was already required for GKE — avoids a CRM-dependent project-number
lookup).
**`deploy/README.md`** — adds `roles/resourcemanager.projectIamAdmin` to
the Pulumi SA's required roles and clarifies which APIs must be enabled
before the first `pulumi up` (storage, cloudresourcemanager, container)
vs. which `ensureRequiredAPIs` adopts.
## Notes
`projects.Service` is idempotent against already-enabled APIs — on
existing projects (staging/prod), Pulumi adopts the live enablement into
state without re-enabling. This was confirmed live: staging deploy went
green after the manual unblocks above, validating the runtime model.
## Test plan
- [x] `go build ./...` clean for `deploy/`
- [x] `golangci-lint run` clean for `deploy/pkg/providers/gcp/...` (one
pre-existing `nilnil` at line 61, unrelated)
- [x] Manual unblock + green staging deploy verified the runtime
behavior this PR encodes
- [ ] On merge: staging deploy adopts the existing API enablements
without churn, then proceeds normally
- [ ] Prod deploy via #1212 (image promotion) inherits the same setup
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Daily ~17:00 UTC `Publish Endpoint Latency` and `Availability < 95%`
alerts are caused by both registry pods exit-0'ing within seconds of
each other and racing with the still-recovering `registry-pg-rw`
endpoint on restart. Separately, the default compute SA had no project
IAM roles, so `fluentbit-gke` and the GMP collector were silently
dropping every log line and time series — which is why none of this was
visible in Cloud Logging.
All Kubernetes/IAM changes apply uniformly to staging and prod.
## Changes
- **GCP IAM** (`deploy/pkg/providers/gcp/provider.go`) — grant the
default compute SA `logging.logWriter`, `monitoring.metricWriter`,
`monitoring.viewer`, and `stackdriver.resourceMetadata.writer`. Already
applied live; this just encodes it in Pulumi.
- **Pod resilience** (`deploy/pkg/k8s/registry.go`) —
- `PodDisruptionBudget` (`minAvailable: 1`) prevents a synchronized
voluntary disruption.
- `TopologySpreadConstraints` (`ScheduleAnyway`, hostname) keeps the PDB
from deadlocking node drains when both pods co-locate.
- `StartupProbe` (30 × 5s = 150s) protects the DB-retry budget from the
liveness probe.
- Memory bump 128→256Mi request, 256→512Mi limit.
- **Startup retry** (`cmd/registry/main.go`) — replace silent `return`
on initial DB-ping failure with bounded retry-with-backoff (8 attempts,
1→8s capped, 10s per-attempt). A brief endpoint flap no longer cascades
into a restart loop.
- **Diagnostic log** (`internal/service/registry_service.go`) — emit
`validate_ms` on each publish (success and failure). Validate fans out
to npm/PyPI/OCI with 10s per-host timeouts and is the most likely
contributor to publish latency. Total request duration is already
covered by the existing `http.request.duration` histogram.
## Calibration
The IAM, PDB, topology-spread, startup-probe, and DB-retry pieces are
mitigations — they reduce blast radius regardless of root cause. The
`validate_ms` log is the diagnostic for identifying the trigger when the
alert fires next. Single-PG SPOF (`deploy/pkg/k8s/postgres.go:57`)
remains a follow-up; today's evidence shows PG was a victim, not the
cause.
## Test plan
- [x] `go build ./...` and `go vet ./...` clean for both modules
- [x] `make lint` clean (12 pre-existing issues under `deploy/` are not
in `make lint`)
- [x] `go test -race -count=1 ./internal/... ./cmd/...` green
- [x] IAM applied live; `fluentbit-gke` and GMP no longer hitting
`IAM_PERMISSION_DENIED`
- [ ] After deploy, verify the next 17:00 UTC firing surfaces
`validate_ms` in Cloud Logging
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Applied `gofmt -s -w` across the entire codebase to ensure consistent
Go formatting
- This will trigger a deployment with minimal code changes
## Test plan
- [x] Pre-commit hooks pass
- [x] No functional changes, formatting only
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
- Add workflow_dispatch to deploy.yml for manual deployment triggers
- Document that releases don't auto-deploy to production
- Add instructions for deploying releases to production
- Add verification steps for deployment
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
## Summary
Bumps the node machine type from e2-small to e2-standard-2 to provide
more CPU and memory headroom for the cluster.
Also disables PodDisruptionBudgets (PDBs) for single-instance PostgreSQL
clusters to prevent node upgrade operations from becoming blocked.
## Changes
- Increase GKE node machine type from e2-small to e2-standard-2
- Set `enablePDB: false` for registry and Grafana PostgreSQL clusters
## Context
The machine type increase addresses resource constraints causing
deployment failures.
The PDB change prevents a specific issue where GKE node upgrades get
stuck when trying to drain nodes containing single-instance PostgreSQL
pods. With `minAvailable: 1` PDBs, evicting the only database instance
violates the availability requirement, blocking the drain operation
indefinitely. Disabling PDBs for these development/staging
single-instance databases allows node maintenance to proceed smoothly.
For production deployments requiring high availability, the instances
count should be increased to 3 with PDBs enabled.
---------
Co-authored-by: Claude <noreply@anthropic.com>
This includes infra setup for metrics collection and visualisation
## Motivation and Context
It is important to have basic observability setup, so that common issues
can be identified quickly
## How Has This Been Tested?
- Tested on local
- Tested on one of the production setup
## Breaking Changes
No
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [X] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [X] My code follows the repository's style guidelines
- [X] New and existing tests pass locally
- [X] I have added appropriate error handling
- [ ] I have added or updated documentation as needed
## Additional context
- This includes set up of following components:
- Victoriametrics single node cluster for storing metrics data with
persistent storage configuration for local and production env.
- Vmagent with target discovery for mcp-registry pods with scrape
interval of 30s.
- Grafana for visualising metrics and setting alerts.
- Datasource of victoriametrics is pre-configured as config map
- Persistent volume for basic configuration
- Sqlite for local and Postgres for production setup for storing
dashboard, alerts etc details
## Metrics in action
<img width="1422" height="666" alt="Screenshot 2025-08-31 at 4 35 28 AM"
src="https://github.com/user-attachments/assets/e54ce846-653c-4fa3-aa7a-fe277f0810e6"
/>
---------
Co-authored-by: Adam Jones <adamj@anthropic.com>
Co-authored-by: adam jones <adamj+git@anthropic.com>
<!-- Provide a brief summary of your changes -->
Implements automated database backup functionality with Google Cloud
Storage integration, including retention policies and MinIO support for
local development.
## Motivation and Context
<!-- Why is this change needed? What problem does it solve? -->
This change adds critical database backup capabilities to ensure data
durability and disaster recovery. The solution provides automated
backups with configurable retention policies and supports both
production (GCS) and development (MinIO) environments.
## How Has This Been Tested?
<!-- Have you tested this in a real application? Which scenarios were
tested? -->
- Tested backup creation and restoration with MinIO in local development
- Verified GCS bucket lifecycle policies for automatic deletion after 60
days
- Tested backup retention and cleanup logic
## Breaking Changes
<!-- Will users need to update their code or configurations? -->
No breaking changes. This is an additive feature that doesn't affect
existing functionality.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [x] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [x] My code follows the repository's style guidelines
- [x] New and existing tests pass locally
- [x] I have added appropriate error handling
- [x] I have added or updated documentation as needed
## Additional context
<!-- Add any other context, implementation notes, or design decisions
-->
- Implements 60-day retention period for backups as a safety net
- Uses GCS lifecycle rules for automatic cleanup
- Includes MinIO setup instructions for local development testing
- Port-forwarding commands updated for consistency across documentation
- Fixes#184
Adds the Pulumi code to:
- Deploy the registry (and associated services e.g. mongodb) to Google
Cloud Platform (GCP), on top of Google Kubernetes Engine (GKE)
- Sets up proper environments and secrets management
- Uses the real container image, now that it's published in #225. At the
moment attached to latest, we might want to pin the version later (or
perhaps always use `latest` in staging, and pin prod)
- Uses real domains (`staging.registry.modelcontextprotocol.io`) rather
than examples (``)
## Motivation and Context
Setting up infrastructure to deploy it. I set something up in Azure in
#227, although not super robust (e.g. no service accounts etc.). Think
we will use GCP as:
- the maintainers have experience with GCP, but none with Azure
- costs are quite low, and Anthropic is happy to cover them in the short
term
- means we only have to maintain one login system (just Google Cloud
Identity), not two (Google Workspace + Azure)
## How Has This Been Tested?
Deployed this to a staging and production cluster. Try it yourself at:
```bash
curl -H "Host: staging.registry.modelcontextprotocol.io" -k https://35.222.36.75/v0/ping
```
(will be sorting out domains very soon)
## Breaking Changes
NA - just adds support for GCP deployment
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [x] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [x] My code follows the repository's style guidelines
- [ ] New and existing tests pass locally
- [x] I have added appropriate error handling
- [x] I have added or updated documentation as needed
## Additional context
<!-- Add any other context, implementation notes, or design decisions
-->
Expected follow-ups:
- GitHub Action setup to deploy things to the cluster from GitHub, to
avoid gatekeeping to just the people with the secrets.
Original PR: #227
- Add Pulumi-based infrastructure as code for deploying MCP Registry to
Kubernetes
- Support for both local development (minikube) and Azure Kubernetes
Service (AKS)
- Complete deployment orchestration including:
- cluster setup: e.g. you point this at an Azure account, and it can set
up and manage the cluster for you. e.g. K8s version, number of nodes,
type of nodes, ...
- cloud agnostic K8s services: cert-manager, nginx-ingress
- app services: MongoDB, and registry application (currently using nginx
as a placeholder, blocked on #225 (as is #190). but should be a 1 line
change)
## How is this different to #190
- Supports cluster setup and management. This enables:
- Non-hosting maintainers managing many devops workflows (e.g. scaling
up the cluster, or bumping K8s versions). Without this, we'd need to
bug/page the organisation hosting the registry when we need these things
changed.
- Makes it easy to spin up things like staging/temporary clusters, as
well as enables contributors to replicate the stack exactly on their own
Azure accounts.
- Sets up cloud-agnostic services. For example, rather than using the
Azure-managed ingresses and CA, we install nginx-ingress and
cert-manager. This enables:
- Running the entire infra stack can also run locally (e.g. in minikube,
k3s, orbstack, colima) - making it much easier for contributors to test
changes to infra stuff.
- Moving between cloud providers much more easily, e.g. we could shift
from Azure to GCP/AWS/other with minimal hassle.
- Everything stays written in Go, rather than Helm templates. This means
we get things like type-checking etc. for free (which from my experience
makes AI tools wayyy better at editing K8s stuff), and contributors
don't need to learn a new language if they're already using Go.
## Testing
I've got this running well:
- locally in minikube
- on cloud in Azure (my personal Azure account)
<details><summary>Claude written architecture and security
review</summary>
<p>
## Deployment Review & Assessment
### Current Architecture Strengths
**Pulumi IaC Approach**
- Well-structured infrastructure as code using Pulumi
- Multi-provider support (AKS, local) with clean abstraction
- Good separation of concerns in `pkg/` directory
**Security Fundamentals**
- Non-root container execution (`appuser` with UID 10001)
- Secrets properly managed via Kubernetes secrets
- TLS/SSL certificate management with cert-manager and Let's Encrypt
### Critical Issues & High-Priority Improvements
**1. Production Deployment Not Ready** 🚨
The registry deployment uses `nginx:alpine` placeholder image instead of
the actual MCP registry:
- `deploy/pkg/k8s/registry.go:67` - TODO comments indicate incomplete
setup
- Health probes are commented out
- Port mapping doesn't match actual application (80 vs 8080)
**Fix:** Build and publish actual registry container image to GHCR,
update deployment
**2. Database Security Considerations** 🔒
- MongoDB deployed without authentication
- No backup/disaster recovery strategy
- Database credentials hardcoded
*Note: MongoDB is not exposed externally (ClusterIP service), so this is
not a critical security risk but should be addressed for production.*
**3. Monitoring & Observability Gaps** 📊
- No Prometheus/Grafana monitoring stack
- No log aggregation (ELK/Loki)
- No application metrics/health dashboards
- No alerting configured
**4. High Availability & Reliability** ⚠️
- Single MongoDB instance (no replication)
- No persistent volume backup strategy
- Fixed 10Gi storage without growth planning
- Only 2 replicas for registry service
- No pod disruption budgets
- No horizontal pod autoscaling
### Recommended Improvements
**Immediate (High Priority)**
1. Complete Registry Deployment - Build proper container image pipeline,
enable health checks
2. Secure MongoDB - Add authentication credentials, implement backup
strategy
**Medium Priority**
3. Add Monitoring Stack - Prometheus, Grafana deployment
4. Security Hardening (Nice to Have) - RBAC policies, Network Policies,
Pod Security Standards
5. CI/CD Pipeline Enhancement - Container image building/publishing,
automated deployment
**Lower Priority**
6. High Availability - MongoDB replica set, HPA for registry pods
7. Operational Excellence - Kubernetes dashboard, cost optimization
### Configuration Issues
- Production config has test credentials: `deploy/Pulumi.prod.yaml:4-5`
- Missing environment-specific resource sizing
- Hardcoded domain names (`example.com`)
The deployment setup shows good architectural foundations but needs
significant work before production readiness. The most critical issue is
the placeholder nginx container - priority should be completing the
actual registry application deployment before addressing the other
improvements. Security measures like RBAC and Network Policies are nice
to have but not strictly necessary given that MongoDB is not exposed
externally.
🤖 Generated with [Claude Code](https://claude.ai/code)
</p>
</details>
## Metadata
Working towards #91
---------
Co-authored-by: Claude <noreply@anthropic.com>
