## What this does
The publisher CLI now derives the OIDC audience to request from the
`--registry` URL instead of using a fixed constant, and the server
validates the token's `aud` claim against a new per-deployment env var
(`MCP_REGISTRY_GITHUB_OIDC_AUDIENCE`).
Production sets this env var to
`https://registry.modelcontextprotocol.io` (and per-environment URL
elsewhere), so the deployment only accepts tokens that were requested
for it.
## Compatibility
This is a breaking change for older `mcp-publisher` binaries that still
request the legacy hardcoded audience. After this rolls to a deployment,
users on old `mcp-publisher` will see a clear `invalid audience:
expected …, got […]` error and need to upgrade. Users following the
documented `releases/latest` install pattern get the new binary
automatically.
## Test plan
- [ ] CI green
- [ ] Run a synthetic publish from a real GitHub Actions workflow
against staging after deploy
- [ ] Confirm an older `mcp-publisher` against the same staging deploy
returns the expected `invalid audience` error
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Follow-up to #1221. That PR doubled pgxpool `MaxConns` from 30 to 60 per
pod without bumping the registry pod memory limit. pgxpool keeps
per-connection state in the Go process (prepared statement cache,
scratch buffers, row iteration state), so memory grew roughly
proportionally — peak per-pod usage shifted from ~330 MB to ~450 MB
against a 512 MiB limit.
## What surfaced it
Pod `tx49f` was OOMKilled at 17:01 UTC on 2026-04-29 (`exitCode 137`,
`reason: OOMKilled`). Memory history from Cloud Monitoring confirms the
regression:
| Pods | Peak 30-min memory |
|------|-------------------:|
| Pre-#1221 (`MaxConns=30`) | 178–333 MB |
| **Post-#1221** (`MaxConns=60`) | **437–449 MB** |
| Old limit | **512 MiB** ← right at the edge |
Other pod was at 394 MB at the time of inspection — could OOM again.
## Fix
`deploy/pkg/k8s/registry.go:217-225`:
| | Before | After |
|---|-------:|------:|
| `requests.memory` | 256 MiB | **512 MiB** |
| `limits.memory` | 512 MiB | **1 GiB** |
1 GiB is ~2× the current peak under load — gives room for traffic spikes
without immediate OOM risk. Either node has 4–5 GiB free allocatable
memory, so the new request fits comfortably.
## Test plan
- [x] `go build ./...` clean
- [x] `golangci-lint run` clean
- [ ] After release + prod promotion: pod memory stabilises around 450
MB peak with no OOMKills
- [ ] After release + prod promotion: tx49f restartCount stops climbing
(currently at 5)
## Deployment
Zero downtime — Deployment template change only, rolling update with
`maxSurge: 1, maxUnavailable: 0`. **No PG involvement** (no
`max_connections` change, no shared_preload_libraries change). Same
shape as a normal v1.7.x image promotion.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Follow-up to #1215 and #1220. Both of those addressed *individual* slow
queries; this PR addresses *concurrency* — under today's scraper load
the cursor query is fast on average (mean 42ms) but the connection pool
saturates and queue depth blows up at the Go HTTP layer.
## Diagnosis
`pg_stat_statements` (added in #1215) made this possible to see:
```
max_ms mean_ms calls template
10,755 41.8 94,009 plain cursor pagination, no filter
7,277 214.7 14,770 ILIKE substring filter
7,638 330.8 7,656 ILIKE substring + is_latest filter
```
Mean times are healthy. But `max_exec_time` of 7–10s on the cursor
query, combined with sustained ~15 req/s from scrapers (ServiceNow's
148.139.x.x range, anonymous `node` user-agent, etc) saturates
`MaxConns=30 × 2 pods = 60`. New requests queue at the Go HTTP layer;
nginx-side p99 hits 35s; eventually scrapers time out at 60s, retry, and
amplify the queue.
Today's two ongoing alerts are both this pattern:
- 17:35–17:40 UTC `Availability dropped below 95%` — 4,526 GET requests
in 5 min, hundreds of 504s
- 18:17 UTC `Publish Endpoint Latency` re-fire — same scraper
concurrency dragging the publish path
Critically, the symptoms today were also visible during yesterday's
incident, but yesterday's broken cursor (#1215) was the dominant cause.
After #1215 the cursor is fast individually; concurrency now becomes the
next bottleneck.
## Changes
### pgxpool (`internal/database/postgres.go`)
| Setting | Before | After |
|---------|-------:|------:|
| `MaxConns` | 30 | **60** per pod |
| `MinConns` | 5 | **10** per pod |
| `MaxConnIdleTime` | 30 min | (unchanged) |
| `MaxConnLifetime` | 2 h | (unchanged) |
2 pods × 60 = 120 total app connections. Cuts queue depth roughly in
half at current scraper load.
### PG cluster (`deploy/pkg/k8s/postgres.go`)
- `max_connections: 100 → 200` — required to support the larger pool.
120 (app) + ~10 (PG internals: autovacuum, replication, admin) + 70
headroom.
- Added explicit `resources:` block — previously the pod had no resource
limits, making node-level OOM behaviour unpredictable.
| | Request | Limit |
|---|--------:|------:|
| memory | 512Mi | **4Gi** |
| cpu | 200m | 1500m |
## Resource budget
| Node | Now | After |
|------|----:|------:|
| dy89 (PG node) | 39% mem (~2.4 GiB / 6 GiB) | ~65% mem worst-case
(~3.9 GiB) |
| 2yxm | 36% mem | unchanged |
Both nodes well within capacity. CPU usage <50% on both, plenty of
headroom.
PG worst-case memory math:
- 200 conns × ~15 MB per backend = 3 GiB
- + `shared_buffers` 128 MiB
- + `maintenance_work_mem`, OS overhead, etc.
- ≈ 3–4 GiB total
## Deployment caveat
`max_connections` is a postmaster-level setting → CNPG triggers a PG
restart on the change. Same in-pod restart shape as the
pg_stat_statements deploy yesterday — registry pods see ~30s of DB
unavailability, covered by v1.7.1's retry-with-backoff (8 attempts, 1→8s
capped). One registry pod may bounce once before recovering, like
yesterday.
**Time the merge for a low-traffic UTC window.**
Order of operations matters within the deploy itself:
1. Pulumi applies the new CNPG spec → PG restarts with
`max_connections=200`
2. Rolling deploy of registry pods picks up `MaxConns=60` config
3. New conns are accepted up to 200 limit
Pulumi's standard ordering does step 1 before step 2 in this scenario
(Pulumi resource graph: CNPG cluster precedes Deployment). If for any
reason it doesn't, the worst case is `too many connections` errors
during a small window — Self-correcting once the rollout completes.
## Test plan
- [x] `go build ./...` clean for app + deploy
- [x] `make lint` clean
- [x] `go test -race ./internal/database/...` green
- [ ] On merge: staging deploy applies the spec change cleanly; PG
restarts; pool reaches 60 max
- [ ] On prod deploy: same; verify no `too many connections` errors
during the rollout window
- [ ] After deploy: query `SHOW max_connections` returns 200; query
pg_stat_statements after a scraper burst, confirm `max_exec_time` for
cursor query no longer hits 10s
## Out of scope
- **ILIKE substring search** (`server_name ILIKE '%foo%'`) is
unindexable. Three pg_stat_statements variants run with means 141–333ms
and max 4–7s. Worth replacing with a `pg_trgm` GIN index or full-text
search in a separate PR.
- **Per-IP nginx rate limiting** — defends against scraper retry storms
regardless of pool size.
- **`Cache-Control` on `/v0/servers`** — let nginx absorb
scraper-repeated cursors.
- **Pre-existing `superfluous WriteHeader` warnings** — separate Huma
framework issue.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Three changes from the 2026-04-27 incident:
1. **Cursor SQL fix** — `addCursorCondition` rewritten to use a
row-constructor comparison `(server_name, version) > ($1, $2)` instead
of the OR-decomposed form. Postgres can index-seek on the row
constructor; it can't on the OR form, which scanned from the start of
the index and filtered rows before the cursor. Cost grew linearly with
cursor depth.
2. **Per-phase publish timings restored** — `createServerInTransaction`
was simplified during #1211 review to log only `validate_ms`. The
incident showed publishes spending 50+ s in `acquire_lock` /
`version_checks` / `db_create` while `validate_ms` reported a few
hundred ms — the diagnostic signal was hidden. Restored timings for
every phase, refactored into a small `runPhase` helper.
3. **`pg_stat_statements` enabled** in the CNPG cluster spec. We had no
aggregate query visibility during the incident — could only EXPLAIN
queries we happened to suspect.
## Cursor fix evidence
Local benchmark, 100K rows, cold cache:
| Form | Rows filtered | Buffer reads | Time |
|------|--------------:|-------------:|-----:|
| OLD | 80,001 | 7,679 | 31.6 ms |
| NEW | 2 | 1 | **0.05 ms** |
Maps to prod's measured 8,911 buffer hits → 760 ms. End-to-end API:
`/v0/servers?limit=100&cursor=…` returns in 4–7 ms regardless of depth.
Risk check: confirmed `(server_name, version)` index exists in that
column order (both `servers_pkey` and `idx_servers_name_version`), and
both columns are `NOT NULL` so row-constructor comparison can't silently
drop rows.
## Tests
- New
`TestPostgreSQL_PerformanceScenarios/compound_cursor_across_versions_of_same_server`
pins multi-version pagination semantics. The existing cursor tests only
exercised the fallback (single-component cursor) and degenerate (one
version per server) cases.
- `make lint` clean, `go test -race ./internal/... ./cmd/...` green.
## Deployment
The cursor + slog changes are zero-downtime. The CNPG spec change
triggers a brief PG restart (single-instance cluster). After the
restart, run once on prod:
```bash
kubectl exec -i registry-pg-1 -c postgres \
--context gke_mcp-registry-prod_us-central1-b_mcp-registry-prod \
-- psql -U postgres -c "CREATE EXTENSION IF NOT EXISTS pg_stat_statements"
```
Time the merge for a low-traffic UTC window. v1.7.1's DB
retry-with-backoff covers the brief PG restart.
## Out of scope
`MaxConns` bump, per-IP rate limiting at nginx, response caching, the
pre-existing `superfluous WriteHeader` warnings — separate follow-ups.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Follow-up to #1211 to encode the GCP API and IAM dependencies its
node-SA bindings rely on. After #1211 merged, the staging deploy hit two
undocumented prereqs in sequence — Cloud Resource Manager API not
enabled (because `projects.NewIAMMember` calls SetIamPolicy through
CRM), then the Pulumi service account missing
`roles/resourcemanager.projectIamAdmin`. Both fixed manually to unblock;
this PR makes them explicit so they don't bite again.
## Changes
**`ensureRequiredAPIs` adopts five GCP APIs** as Pulumi-managed
`projects.Service` resources, all with `DisableOnDestroy: false` and
`DisableDependentServices: false` so a Pulumi destroy/refactor can never
disable a shared API:
| API | Why |
|-----|-----|
| `cloudresourcemanager` | `projects.NewIAMMember` → SetIamPolicy |
| `compute` | `compute.GetDefaultServiceAccount` invoke + GKE |
| `container` | GKE cluster |
| `logging` | fluentbit-gke ships container logs |
| `monitoring` | managed Prometheus collector ships metrics |
CRM is created explicitly (not in the loop) so callers get a direct
reference for `pulumi.DependsOn`. Storage is intentionally **not**
Pulumi-managed — Pulumi state itself lives in a GCS bucket,
chicken-and-egg.
**`grantNodeServiceAccountRoles`** now `DependsOn` the CRM
`projects.Service` for the four IAM bindings, and uses
`compute.GetDefaultServiceAccount` to derive the SA email (Compute API
was already required for GKE — avoids a CRM-dependent project-number
lookup).
**`deploy/README.md`** — adds `roles/resourcemanager.projectIamAdmin` to
the Pulumi SA's required roles and clarifies which APIs must be enabled
before the first `pulumi up` (storage, cloudresourcemanager, container)
vs. which `ensureRequiredAPIs` adopts.
## Notes
`projects.Service` is idempotent against already-enabled APIs — on
existing projects (staging/prod), Pulumi adopts the live enablement into
state without re-enabling. This was confirmed live: staging deploy went
green after the manual unblocks above, validating the runtime model.
## Test plan
- [x] `go build ./...` clean for `deploy/`
- [x] `golangci-lint run` clean for `deploy/pkg/providers/gcp/...` (one
pre-existing `nilnil` at line 61, unrelated)
- [x] Manual unblock + green staging deploy verified the runtime
behavior this PR encodes
- [ ] On merge: staging deploy adopts the existing API enablements
without churn, then proceeds normally
- [ ] Prod deploy via #1212 (image promotion) inherits the same setup
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Daily ~17:00 UTC `Publish Endpoint Latency` and `Availability < 95%`
alerts are caused by both registry pods exit-0'ing within seconds of
each other and racing with the still-recovering `registry-pg-rw`
endpoint on restart. Separately, the default compute SA had no project
IAM roles, so `fluentbit-gke` and the GMP collector were silently
dropping every log line and time series — which is why none of this was
visible in Cloud Logging.
All Kubernetes/IAM changes apply uniformly to staging and prod.
## Changes
- **GCP IAM** (`deploy/pkg/providers/gcp/provider.go`) — grant the
default compute SA `logging.logWriter`, `monitoring.metricWriter`,
`monitoring.viewer`, and `stackdriver.resourceMetadata.writer`. Already
applied live; this just encodes it in Pulumi.
- **Pod resilience** (`deploy/pkg/k8s/registry.go`) —
- `PodDisruptionBudget` (`minAvailable: 1`) prevents a synchronized
voluntary disruption.
- `TopologySpreadConstraints` (`ScheduleAnyway`, hostname) keeps the PDB
from deadlocking node drains when both pods co-locate.
- `StartupProbe` (30 × 5s = 150s) protects the DB-retry budget from the
liveness probe.
- Memory bump 128→256Mi request, 256→512Mi limit.
- **Startup retry** (`cmd/registry/main.go`) — replace silent `return`
on initial DB-ping failure with bounded retry-with-backoff (8 attempts,
1→8s capped, 10s per-attempt). A brief endpoint flap no longer cascades
into a restart loop.
- **Diagnostic log** (`internal/service/registry_service.go`) — emit
`validate_ms` on each publish (success and failure). Validate fans out
to npm/PyPI/OCI with 10s per-host timeouts and is the most likely
contributor to publish latency. Total request duration is already
covered by the existing `http.request.duration` histogram.
## Calibration
The IAM, PDB, topology-spread, startup-probe, and DB-retry pieces are
mitigations — they reduce blast radius regardless of root cause. The
`validate_ms` log is the diagnostic for identifying the trigger when the
alert fires next. Single-PG SPOF (`deploy/pkg/k8s/postgres.go:57`)
remains a follow-up; today's evidence shows PG was a victim, not the
cause.
## Test plan
- [x] `go build ./...` and `go vet ./...` clean for both modules
- [x] `make lint` clean (12 pre-existing issues under `deploy/` are not
in `make lint`)
- [x] `go test -race -count=1 ./internal/... ./cmd/...` green
- [x] IAM applied live; `fluentbit-gke` and GMP no longer hitting
`IAM_PERMISSION_DENIED`
- [ ] After deploy, verify the next 17:00 UTC firing surfaces
`validate_ms` in Cloud Logging
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
<!-- Provide a brief summary of your changes -->
## Motivation and Context
<!-- Why is this change needed? What problem does it solve? -->
The following PR ensures we don't have downtime when we are doing
rollouts during deployment/promotions.
## How Has This Been Tested?
<!-- Have you tested this in a real application? Which scenarios were
tested? -->
## Breaking Changes
<!-- Will users need to update their code or configurations? -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [ ] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [ ] My code follows the repository's style guidelines
- [ ] New and existing tests pass locally
- [ ] I have added appropriate error handling
- [ ] I have added or updated documentation as needed
## Additional context
<!-- Add any other context, implementation notes, or design decisions
-->
Signed-off-by: Radoslav Dimitrov <radoslav@stacklok.com>
<!-- Provide a brief summary of your changes -->
## Motivation and Context
<!-- Why is this change needed? What problem does it solve? -->
Seems that per ingress annotations were not working, so had to put them
in the ingress configmap. Did tested this manually on staging so it's
confirmed it's working (I was getting 429s instead of 503s) and no
traffic was getting into the registry pod.
## How Has This Been Tested?
<!-- Have you tested this in a real application? Which scenarios were
tested? -->
## Breaking Changes
<!-- Will users need to update their code or configurations? -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [ ] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [ ] My code follows the repository's style guidelines
- [ ] New and existing tests pass locally
- [ ] I have added appropriate error handling
- [ ] I have added or updated documentation as needed
## Additional context
<!-- Add any other context, implementation notes, or design decisions
-->
Signed-off-by: Radoslav Dimitrov <radoslav@stacklok.com>
<!-- Provide a brief summary of your changes -->
## Motivation and Context
<!-- Why is this change needed? What problem does it solve? -->
A few changes related to configuring the rate limiting:
* Set `externalTrafficPolicy: Local` to preserve real client IPs
(otherwise we only see the cluster IPs)
* Disable `use-forwarded-headers` (L4 LB doesn't set `X-Forwarded-For`,
prevents spoofing)
* Scale NGINX to 2 replicas in prod (costs nothing, gives us
zero-downtime deploys to prod)
## How Has This Been Tested?
<!-- Have you tested this in a real application? Which scenarios were
tested? -->
## Breaking Changes
<!-- Will users need to update their code or configurations? -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [ ] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [ ] My code follows the repository's style guidelines
- [ ] New and existing tests pass locally
- [ ] I have added appropriate error handling
- [ ] I have added or updated documentation as needed
## Additional context
<!-- Add any other context, implementation notes, or design decisions
-->
Signed-off-by: Radoslav Dimitrov <radoslav@stacklok.com>
<!-- Provide a brief summary of your changes -->
## Motivation and Context
<!-- Why is this change needed? What problem does it solve? -->
The following is an alternative solution to #833
Goal is to deploy and promote it to prod, then monitor the alerts for a
week to see its effect and reassess. In case it's actual traffic that
we've started blocking we can also up the replicas. I thought it doesn't
make sense to up them now since it would be nice to isolate the result
of this change first.
## How Has This Been Tested?
<!-- Have you tested this in a real application? Which scenarios were
tested? -->
## Breaking Changes
<!-- Will users need to update their code or configurations? -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [ ] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [ ] My code follows the repository's style guidelines
- [ ] New and existing tests pass locally
- [ ] I have added appropriate error handling
- [ ] I have added or updated documentation as needed
## Additional context
<!-- Add any other context, implementation notes, or design decisions
-->
---------
Signed-off-by: Radoslav Dimitrov <radoslav@stacklok.com>
Telemetry to cover failure modes which are not covered by container logs
and metrics for finding resource constraints.
## Motivation and Context
When there is any issue with registry container we should be notified.
## How Has This Been Tested?
- Local seup
## Breaking Changes
- No
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [ ] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [ ] My code follows the repository's style guidelines
- [ ] New and existing tests pass locally
- [ ] I have added appropriate error handling
- [ ] I have added or updated documentation as needed
## Additional context
- No additional exporter is used, taken advantage of opentelemetry
collector
- It covers metrics related to resource constraints, currently only
limited to default namespace.
- Takes cares of kubernetes events as logs which are the source of
figuring out any problem with service, covers all such scenarios where
pod is not able to start yet and get missed because there are no
container logs for such cases. Limited to default namespace.
- Taken care of daemonset deployment i.e. deploying otel collector as
agent by using correct filtering.
- Cardinality contributing factors are only pod ids (but have to observe
more), node ids will not increase cardinality as scale up will lead to
limited nodes.
- Shipping of metrics for resources happens every 60s and list of
metrics that will be emitted
[https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/kubeletstatsreceiver/metadata.yaml](url)
- Container errors
<img width="1440" height="816" alt="Screenshot 2025-10-10 at 1 21 14 AM"
src="https://github.com/user-attachments/assets/ba90a217-2a49-4522-aa44-a98c02adf95b"
/>
- Resource metrics
<img width="1440" height="816" alt="Screenshot 2025-10-10 at 1 23 51 AM"
src="https://github.com/user-attachments/assets/3467be96-db3c-4930-afa2-3cbf5f0ced8b"
/>
<!-- Provide a brief summary of your changes -->
## Motivation and Context
<!-- Why is this change needed? What problem does it solve? -->
The following PR moves away from automatically promoting latest releases
to production in favour of specifying the released version explicitly
via a PR & Review process.
Still sets the image tag to latest for now so we can enable it
separately whenever we decide 👍
## How Has This Been Tested?
<!-- Have you tested this in a real application? Which scenarios were
tested? -->
## Breaking Changes
<!-- Will users need to update their code or configurations? -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [ ] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [ ] My code follows the repository's style guidelines
- [ ] New and existing tests pass locally
- [ ] I have added appropriate error handling
- [ ] I have added or updated documentation as needed
## Additional context
<!-- Add any other context, implementation notes, or design decisions
-->
Signed-off-by: Radoslav Dimitrov <radoslav@stacklok.com>
## Summary
- Applied `gofmt -s -w` across the entire codebase to ensure consistent
Go formatting
- This will trigger a deployment with minimal code changes
## Test plan
- [x] Pre-commit hooks pass
- [x] No functional changes, formatting only
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
- Add workflow_dispatch to deploy.yml for manual deployment triggers
- Document that releases don't auto-deploy to production
- Add instructions for deploying releases to production
- Add verification steps for deployment
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
<!-- Provide a brief summary of your changes -->
## Motivation and Context
<!-- Why is this change needed? What problem does it solve? -->
The following PR updates production to automatically receive new stable
releases when GitHub releases are cut while staging will continue
testing the latest main branch builds
Deails:
- Configures production deployment to **use latest stable** releases
instead of main branch
-`ghcr.io/modelcontextprotocol/registry:latest`(stable releases).
Released CLI matches version of the prod service API.
- Staging continues to **use main branch** for testing latest
development code - `ghcr.io/modelcontextprotocol/registry:main` (latest
development). Building the CLI from main would match the version of the
staging service API.
- Added environment-based Docker image tag selection logic
## How Has This Been Tested?
<!-- Have you tested this in a real application? Which scenarios were
tested? -->
No
## Breaking Changes
<!-- Will users need to update their code or configurations? -->
No
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [ ] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [ ] My code follows the repository's style guidelines
- [ ] New and existing tests pass locally
- [ ] I have added appropriate error handling
- [ ] I have added or updated documentation as needed
## Additional context
<!-- Add any other context, implementation notes, or design decisions
-->
Signed-off-by: Radoslav Dimitrov <radoslav@stacklok.com>
## Summary
Bumps the node machine type from e2-small to e2-standard-2 to provide
more CPU and memory headroom for the cluster.
Also disables PodDisruptionBudgets (PDBs) for single-instance PostgreSQL
clusters to prevent node upgrade operations from becoming blocked.
## Changes
- Increase GKE node machine type from e2-small to e2-standard-2
- Set `enablePDB: false` for registry and Grafana PostgreSQL clusters
## Context
The machine type increase addresses resource constraints causing
deployment failures.
The PDB change prevents a specific issue where GKE node upgrades get
stuck when trying to drain nodes containing single-instance PostgreSQL
pods. With `minAvailable: 1` PDBs, evicting the only database instance
violates the availability requirement, blocking the drain operation
indefinitely. Disabling PDBs for these development/staging
single-instance databases allows node maintenance to proceed smoothly.
For production deployments requiring high availability, the instances
count should be increased to 3 with PDBs enabled.
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Motivation and Context
Logs will help in fixing issues quickly, thereby reducing the downtime
and MTTD.
## How Has This Been Tested?
End to end local setup is tested
## Breaking Changes
No
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [ ] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [x] My code follows the repository's style guidelines
- [x] New and existing tests pass locally
- [ ] I have added appropriate error handling
- [ ] I have added or updated documentation as needed
## Additional context
- It handles container logs for mcp-registry pods.
- It supports k8sattributes resources which helps in identifying from
where the logs are coming.
- Supports victorialogs and it's plugin in grafana.
<img width="1129" height="383" alt="Screenshot 2025-09-15 at 1 43 51 AM"
src="https://github.com/user-attachments/assets/a560c541-721d-415c-a0a8-8cbe008c3af6"
/>
<img width="1134" height="703" alt="Screenshot 2025-09-15 at 1 44 34 AM"
src="https://github.com/user-attachments/assets/8b5b4132-9fc3-4b8e-8027-27c7787070f1"
/>
<img width="1134" height="727" alt="Screenshot 2025-09-15 at 1 46 10 AM"
src="https://github.com/user-attachments/assets/bcdb4d9d-cea1-4d88-8dd5-fa4cdb51031f"
/>
---------
Co-authored-by: Radoslav Dimitrov <radoslav@stacklok.com>
<!-- Provide a brief summary of your changes -->
## Motivation and Context
<!-- Why is this change needed? What problem does it solve? -->
The following PR addresses the feedback comments from #366
## How Has This Been Tested?
<!-- Have you tested this in a real application? Which scenarios were
tested? -->
## Breaking Changes
<!-- Will users need to update their code or configurations? -->
No
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [ ] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [ ] My code follows the repository's style guidelines
- [ ] New and existing tests pass locally
- [ ] I have added appropriate error handling
- [ ] I have added or updated documentation as needed
## Additional context
<!-- Add any other context, implementation notes, or design decisions
-->
Signed-off-by: Radoslav Dimitrov <radoslav@stacklok.com>
## Summary
- Removed Azure deployment section and references from README
- Updated project structure to reflect current GCP prod/staging configs
- Removed Azure Load Balancer annotation from ingress controller
- Updated provider documentation to show GCP and local options only
- Added monitoring.go to project structure documentation
This cleans up outdated Azure/AKS deployment configuration that is no
longer used, leaving only the current GCP-based deployment structure.
## Motivation and Context
Ensures that the redirect url is set properly when logging in with
Google
## How Has This Been Tested?
In staging
## Breaking Changes
None
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Motivation and Context
Consistency with https://staging.registry.modelcontextprotocol.io/ URL
format (i.e. `<env>.<service>`, rather than `<service>.<env>`
## How Has This Been Tested?
Deployed in staging
## Breaking Changes
Grafana URL changes, but it's only just launched so fine.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Motivation and Context
After #328, the monitoring stack was failing to deploy because we hit
resource limits. This PR reduces the resource requests to a level
compatible with the cluster.
## How Has This Been Tested?
I've checked that everything still deploys happily in staging.
## Breaking Changes
None
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
This includes infra setup for metrics collection and visualisation
## Motivation and Context
It is important to have basic observability setup, so that common issues
can be identified quickly
## How Has This Been Tested?
- Tested on local
- Tested on one of the production setup
## Breaking Changes
No
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [X] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [X] My code follows the repository's style guidelines
- [X] New and existing tests pass locally
- [X] I have added appropriate error handling
- [ ] I have added or updated documentation as needed
## Additional context
- This includes set up of following components:
- Victoriametrics single node cluster for storing metrics data with
persistent storage configuration for local and production env.
- Vmagent with target discovery for mcp-registry pods with scrape
interval of 30s.
- Grafana for visualising metrics and setting alerts.
- Datasource of victoriametrics is pre-configured as config map
- Persistent volume for basic configuration
- Sqlite for local and Postgres for production setup for storing
dashboard, alerts etc details
## Metrics in action
<img width="1422" height="666" alt="Screenshot 2025-08-31 at 4 35 28 AM"
src="https://github.com/user-attachments/assets/e54ce846-653c-4fa3-aa7a-fe277f0810e6"
/>
---------
Co-authored-by: Adam Jones <adamj@anthropic.com>
Co-authored-by: adam jones <adamj+git@anthropic.com>
## Summary
This PR implements OIDC authentication support for admin users,
specifically configured for Google Cloud Identity. The implementation
includes:
- Configurable OIDC authentication endpoints and validation
- Google Cloud Identity integration for admin access
- JWT token validation and user session management
- Deployment configuration updates for OIDC provider settings
Fixes#283
<!-- Provide a brief summary of your changes -->
Implements automated database backup functionality with Google Cloud
Storage integration, including retention policies and MinIO support for
local development.
## Motivation and Context
<!-- Why is this change needed? What problem does it solve? -->
This change adds critical database backup capabilities to ensure data
durability and disaster recovery. The solution provides automated
backups with configurable retention policies and supports both
production (GCS) and development (MinIO) environments.
## How Has This Been Tested?
<!-- Have you tested this in a real application? Which scenarios were
tested? -->
- Tested backup creation and restoration with MinIO in local development
- Verified GCS bucket lifecycle policies for automatic deletion after 60
days
- Tested backup retention and cleanup logic
## Breaking Changes
<!-- Will users need to update their code or configurations? -->
No breaking changes. This is an additive feature that doesn't affect
existing functionality.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [x] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [x] My code follows the repository's style guidelines
- [x] New and existing tests pass locally
- [x] I have added appropriate error handling
- [x] I have added or updated documentation as needed
## Additional context
<!-- Add any other context, implementation notes, or design decisions
-->
- Implements 60-day retention period for backups as a safety net
- Uses GCS lifecycle rules for automatic cleanup
- Includes MinIO setup instructions for local development testing
- Port-forwarding commands updated for consistency across documentation
- Fixes#184
Fixes#228
## Motivation and Context
See #19
## How Has This Been Tested?
- Run locally
- Run in docker compose
- Run infra in minikube
- Unit and integration tests all passing :)
## Breaking Changes
Yes: users will not be able to access any data they used to have in
custom mongodb. We are accepting this breaking change, given the
registry is in experimental development status.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [x] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [x] My code follows the repository's style guidelines
- [x] New and existing tests pass locally
- [x] I have added appropriate error handling
- [x] I have added or updated documentation as needed
## Additional context
I have not implemented connection pooling support for simplicity, and
because cloudnative-pg already provides built-in connection pooling with
pgBouncer so if we want it later we can switch to this endpoint easily.
## Motivation and Context
In staging (and prod, for now) we pull the `:latest` mcp-registry image.
But K8s won't actually change anything unless we recreate the pods.
This change ensures the workflow goes from docker -> staging -> prod,
and changes the annotation so that the pods are recreated each deploy.
This ensures they actually pull the latest version.
## How Has This Been Tested?
Deployed locally to minikube
## Breaking Changes
None
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [x] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [x] My code follows the repository's style guidelines
- [x] New and existing tests pass locally
- [x] I have added appropriate error handling
- [x] I have added or updated documentation as needed
Fixes#264. Adds endpoints:
```
/v0/auth/github # exchange a github token for a registry token that gives you access to namespaces io.github.username and io.github.org for your orgs
/v0/auth/none # get a registry token for namespace io.modelcontextprotocol.anonymous (only for testing/local dev purposes)
```
Updates `/v0/publish` to accept one of these registry tokens.
## Motivation and Context
See #264
## How Has This Been Tested?
- Running locally and getting a token from `/v0/auth/none` and then
publishing to `/v0/publish`
- Updating the integration tests that get a token and then publish, and
them passing
## Breaking Changes
This is a breaking change to the auth API. This is fine for registry
given it's not deployed yet.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [x] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [x] My code follows the repository's style guidelines
- [x] New and existing tests pass locally
- [x] I have added appropriate error handling
- [x] I have added or updated documentation as needed
## Additional context
<!-- Add any other context, implementation notes, or design decisions
-->
---------
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: adam jones <domdomegg@users.noreply.github.com>
The Always pull policy is needed for the latest tag to force Kubernetes
to pull the newest image on every pod creation, even if an image with
that tag already exists locally. Without this, pods might use outdated
cached versions since the latest tag is mutable and can point to
different images over time.
Adds the Pulumi code to:
- Deploy the registry (and associated services e.g. mongodb) to Google
Cloud Platform (GCP), on top of Google Kubernetes Engine (GKE)
- Sets up proper environments and secrets management
- Uses the real container image, now that it's published in #225. At the
moment attached to latest, we might want to pin the version later (or
perhaps always use `latest` in staging, and pin prod)
- Uses real domains (`staging.registry.modelcontextprotocol.io`) rather
than examples (``)
## Motivation and Context
Setting up infrastructure to deploy it. I set something up in Azure in
#227, although not super robust (e.g. no service accounts etc.). Think
we will use GCP as:
- the maintainers have experience with GCP, but none with Azure
- costs are quite low, and Anthropic is happy to cover them in the short
term
- means we only have to maintain one login system (just Google Cloud
Identity), not two (Google Workspace + Azure)
## How Has This Been Tested?
Deployed this to a staging and production cluster. Try it yourself at:
```bash
curl -H "Host: staging.registry.modelcontextprotocol.io" -k https://35.222.36.75/v0/ping
```
(will be sorting out domains very soon)
## Breaking Changes
NA - just adds support for GCP deployment
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [x] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [x] My code follows the repository's style guidelines
- [ ] New and existing tests pass locally
- [x] I have added appropriate error handling
- [x] I have added or updated documentation as needed
## Additional context
<!-- Add any other context, implementation notes, or design decisions
-->
Expected follow-ups:
- GitHub Action setup to deploy things to the cluster from GitHub, to
avoid gatekeeping to just the people with the secrets.
Original PR: #227
- Add Pulumi-based infrastructure as code for deploying MCP Registry to
Kubernetes
- Support for both local development (minikube) and Azure Kubernetes
Service (AKS)
- Complete deployment orchestration including:
- cluster setup: e.g. you point this at an Azure account, and it can set
up and manage the cluster for you. e.g. K8s version, number of nodes,
type of nodes, ...
- cloud agnostic K8s services: cert-manager, nginx-ingress
- app services: MongoDB, and registry application (currently using nginx
as a placeholder, blocked on #225 (as is #190). but should be a 1 line
change)
## How is this different to #190
- Supports cluster setup and management. This enables:
- Non-hosting maintainers managing many devops workflows (e.g. scaling
up the cluster, or bumping K8s versions). Without this, we'd need to
bug/page the organisation hosting the registry when we need these things
changed.
- Makes it easy to spin up things like staging/temporary clusters, as
well as enables contributors to replicate the stack exactly on their own
Azure accounts.
- Sets up cloud-agnostic services. For example, rather than using the
Azure-managed ingresses and CA, we install nginx-ingress and
cert-manager. This enables:
- Running the entire infra stack can also run locally (e.g. in minikube,
k3s, orbstack, colima) - making it much easier for contributors to test
changes to infra stuff.
- Moving between cloud providers much more easily, e.g. we could shift
from Azure to GCP/AWS/other with minimal hassle.
- Everything stays written in Go, rather than Helm templates. This means
we get things like type-checking etc. for free (which from my experience
makes AI tools wayyy better at editing K8s stuff), and contributors
don't need to learn a new language if they're already using Go.
## Testing
I've got this running well:
- locally in minikube
- on cloud in Azure (my personal Azure account)
<details><summary>Claude written architecture and security
review</summary>
<p>
## Deployment Review & Assessment
### Current Architecture Strengths
**Pulumi IaC Approach**
- Well-structured infrastructure as code using Pulumi
- Multi-provider support (AKS, local) with clean abstraction
- Good separation of concerns in `pkg/` directory
**Security Fundamentals**
- Non-root container execution (`appuser` with UID 10001)
- Secrets properly managed via Kubernetes secrets
- TLS/SSL certificate management with cert-manager and Let's Encrypt
### Critical Issues & High-Priority Improvements
**1. Production Deployment Not Ready** 🚨
The registry deployment uses `nginx:alpine` placeholder image instead of
the actual MCP registry:
- `deploy/pkg/k8s/registry.go:67` - TODO comments indicate incomplete
setup
- Health probes are commented out
- Port mapping doesn't match actual application (80 vs 8080)
**Fix:** Build and publish actual registry container image to GHCR,
update deployment
**2. Database Security Considerations** 🔒
- MongoDB deployed without authentication
- No backup/disaster recovery strategy
- Database credentials hardcoded
*Note: MongoDB is not exposed externally (ClusterIP service), so this is
not a critical security risk but should be addressed for production.*
**3. Monitoring & Observability Gaps** 📊
- No Prometheus/Grafana monitoring stack
- No log aggregation (ELK/Loki)
- No application metrics/health dashboards
- No alerting configured
**4. High Availability & Reliability** ⚠️
- Single MongoDB instance (no replication)
- No persistent volume backup strategy
- Fixed 10Gi storage without growth planning
- Only 2 replicas for registry service
- No pod disruption budgets
- No horizontal pod autoscaling
### Recommended Improvements
**Immediate (High Priority)**
1. Complete Registry Deployment - Build proper container image pipeline,
enable health checks
2. Secure MongoDB - Add authentication credentials, implement backup
strategy
**Medium Priority**
3. Add Monitoring Stack - Prometheus, Grafana deployment
4. Security Hardening (Nice to Have) - RBAC policies, Network Policies,
Pod Security Standards
5. CI/CD Pipeline Enhancement - Container image building/publishing,
automated deployment
**Lower Priority**
6. High Availability - MongoDB replica set, HPA for registry pods
7. Operational Excellence - Kubernetes dashboard, cost optimization
### Configuration Issues
- Production config has test credentials: `deploy/Pulumi.prod.yaml:4-5`
- Missing environment-specific resource sizing
- Hardcoded domain names (`example.com`)
The deployment setup shows good architectural foundations but needs
significant work before production readiness. The most critical issue is
the placeholder nginx container - priority should be completing the
actual registry application deployment before addressing the other
improvements. Security measures like RBAC and Network Policies are nice
to have but not strictly necessary given that MongoDB is not exposed
externally.
🤖 Generated with [Claude Code](https://claude.ai/code)
</p>
</details>
## Metadata
Working towards #91
---------
Co-authored-by: Claude <noreply@anthropic.com>
