## Summary
Follow-up to #1215 and #1220. Both of those addressed *individual* slow
queries; this PR addresses *concurrency* — under today's scraper load
the cursor query is fast on average (mean 42ms) but the connection pool
saturates and queue depth blows up at the Go HTTP layer.
## Diagnosis
`pg_stat_statements` (added in #1215) made this possible to see:
```
max_ms mean_ms calls template
10,755 41.8 94,009 plain cursor pagination, no filter
7,277 214.7 14,770 ILIKE substring filter
7,638 330.8 7,656 ILIKE substring + is_latest filter
```
Mean times are healthy. But `max_exec_time` of 7–10s on the cursor
query, combined with sustained ~15 req/s from scrapers (ServiceNow's
148.139.x.x range, anonymous `node` user-agent, etc) saturates
`MaxConns=30 × 2 pods = 60`. New requests queue at the Go HTTP layer;
nginx-side p99 hits 35s; eventually scrapers time out at 60s, retry, and
amplify the queue.
Today's two ongoing alerts are both this pattern:
- 17:35–17:40 UTC `Availability dropped below 95%` — 4,526 GET requests
in 5 min, hundreds of 504s
- 18:17 UTC `Publish Endpoint Latency` re-fire — same scraper
concurrency dragging the publish path
Critically, the symptoms today were also visible during yesterday's
incident, but yesterday's broken cursor (#1215) was the dominant cause.
After #1215 the cursor is fast individually; concurrency now becomes the
next bottleneck.
## Changes
### pgxpool (`internal/database/postgres.go`)
| Setting | Before | After |
|---------|-------:|------:|
| `MaxConns` | 30 | **60** per pod |
| `MinConns` | 5 | **10** per pod |
| `MaxConnIdleTime` | 30 min | (unchanged) |
| `MaxConnLifetime` | 2 h | (unchanged) |
2 pods × 60 = 120 total app connections. Cuts queue depth roughly in
half at current scraper load.
### PG cluster (`deploy/pkg/k8s/postgres.go`)
- `max_connections: 100 → 200` — required to support the larger pool.
120 (app) + ~10 (PG internals: autovacuum, replication, admin) + 70
headroom.
- Added explicit `resources:` block — previously the pod had no resource
limits, making node-level OOM behaviour unpredictable.
| | Request | Limit |
|---|--------:|------:|
| memory | 512Mi | **4Gi** |
| cpu | 200m | 1500m |
## Resource budget
| Node | Now | After |
|------|----:|------:|
| dy89 (PG node) | 39% mem (~2.4 GiB / 6 GiB) | ~65% mem worst-case
(~3.9 GiB) |
| 2yxm | 36% mem | unchanged |
Both nodes well within capacity. CPU usage <50% on both, plenty of
headroom.
PG worst-case memory math:
- 200 conns × ~15 MB per backend = 3 GiB
- + `shared_buffers` 128 MiB
- + `maintenance_work_mem`, OS overhead, etc.
- ≈ 3–4 GiB total
## Deployment caveat
`max_connections` is a postmaster-level setting → CNPG triggers a PG
restart on the change. Same in-pod restart shape as the
pg_stat_statements deploy yesterday — registry pods see ~30s of DB
unavailability, covered by v1.7.1's retry-with-backoff (8 attempts, 1→8s
capped). One registry pod may bounce once before recovering, like
yesterday.
**Time the merge for a low-traffic UTC window.**
Order of operations matters within the deploy itself:
1. Pulumi applies the new CNPG spec → PG restarts with
`max_connections=200`
2. Rolling deploy of registry pods picks up `MaxConns=60` config
3. New conns are accepted up to 200 limit
Pulumi's standard ordering does step 1 before step 2 in this scenario
(Pulumi resource graph: CNPG cluster precedes Deployment). If for any
reason it doesn't, the worst case is `too many connections` errors
during a small window — Self-correcting once the rollout completes.
## Test plan
- [x] `go build ./...` clean for app + deploy
- [x] `make lint` clean
- [x] `go test -race ./internal/database/...` green
- [ ] On merge: staging deploy applies the spec change cleanly; PG
restarts; pool reaches 60 max
- [ ] On prod deploy: same; verify no `too many connections` errors
during the rollout window
- [ ] After deploy: query `SHOW max_connections` returns 200; query
pg_stat_statements after a scraper burst, confirm `max_exec_time` for
cursor query no longer hits 10s
## Out of scope
- **ILIKE substring search** (`server_name ILIKE '%foo%'`) is
unindexable. Three pg_stat_statements variants run with means 141–333ms
and max 4–7s. Worth replacing with a `pg_trgm` GIN index or full-text
search in a separate PR.
- **Per-IP nginx rate limiting** — defends against scraper retry storms
regardless of pool size.
- **`Cache-Control` on `/v0/servers`** — let nginx absorb
scraper-repeated cursors.
- **Pre-existing `superfluous WriteHeader` warnings** — separate Huma
framework issue.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Three changes from the 2026-04-27 incident:
1. **Cursor SQL fix** — `addCursorCondition` rewritten to use a
row-constructor comparison `(server_name, version) > ($1, $2)` instead
of the OR-decomposed form. Postgres can index-seek on the row
constructor; it can't on the OR form, which scanned from the start of
the index and filtered rows before the cursor. Cost grew linearly with
cursor depth.
2. **Per-phase publish timings restored** — `createServerInTransaction`
was simplified during #1211 review to log only `validate_ms`. The
incident showed publishes spending 50+ s in `acquire_lock` /
`version_checks` / `db_create` while `validate_ms` reported a few
hundred ms — the diagnostic signal was hidden. Restored timings for
every phase, refactored into a small `runPhase` helper.
3. **`pg_stat_statements` enabled** in the CNPG cluster spec. We had no
aggregate query visibility during the incident — could only EXPLAIN
queries we happened to suspect.
## Cursor fix evidence
Local benchmark, 100K rows, cold cache:
| Form | Rows filtered | Buffer reads | Time |
|------|--------------:|-------------:|-----:|
| OLD | 80,001 | 7,679 | 31.6 ms |
| NEW | 2 | 1 | **0.05 ms** |
Maps to prod's measured 8,911 buffer hits → 760 ms. End-to-end API:
`/v0/servers?limit=100&cursor=…` returns in 4–7 ms regardless of depth.
Risk check: confirmed `(server_name, version)` index exists in that
column order (both `servers_pkey` and `idx_servers_name_version`), and
both columns are `NOT NULL` so row-constructor comparison can't silently
drop rows.
## Tests
- New
`TestPostgreSQL_PerformanceScenarios/compound_cursor_across_versions_of_same_server`
pins multi-version pagination semantics. The existing cursor tests only
exercised the fallback (single-component cursor) and degenerate (one
version per server) cases.
- `make lint` clean, `go test -race ./internal/... ./cmd/...` green.
## Deployment
The cursor + slog changes are zero-downtime. The CNPG spec change
triggers a brief PG restart (single-instance cluster). After the
restart, run once on prod:
```bash
kubectl exec -i registry-pg-1 -c postgres \
--context gke_mcp-registry-prod_us-central1-b_mcp-registry-prod \
-- psql -U postgres -c "CREATE EXTENSION IF NOT EXISTS pg_stat_statements"
```
Time the merge for a low-traffic UTC window. v1.7.1's DB
retry-with-backoff covers the brief PG restart.
## Out of scope
`MaxConns` bump, per-IP rate limiting at nginx, response caching, the
pre-existing `superfluous WriteHeader` warnings — separate follow-ups.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Bumps the node machine type from e2-small to e2-standard-2 to provide
more CPU and memory headroom for the cluster.
Also disables PodDisruptionBudgets (PDBs) for single-instance PostgreSQL
clusters to prevent node upgrade operations from becoming blocked.
## Changes
- Increase GKE node machine type from e2-small to e2-standard-2
- Set `enablePDB: false` for registry and Grafana PostgreSQL clusters
## Context
The machine type increase addresses resource constraints causing
deployment failures.
The PDB change prevents a specific issue where GKE node upgrades get
stuck when trying to drain nodes containing single-instance PostgreSQL
pods. With `minAvailable: 1` PDBs, evicting the only database instance
violates the availability requirement, blocking the drain operation
indefinitely. Disabling PDBs for these development/staging
single-instance databases allows node maintenance to proceed smoothly.
For production deployments requiring high availability, the instances
count should be increased to 3 with PDBs enabled.
---------
Co-authored-by: Claude <noreply@anthropic.com>
This includes infra setup for metrics collection and visualisation
## Motivation and Context
It is important to have basic observability setup, so that common issues
can be identified quickly
## How Has This Been Tested?
- Tested on local
- Tested on one of the production setup
## Breaking Changes
No
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [X] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [X] My code follows the repository's style guidelines
- [X] New and existing tests pass locally
- [X] I have added appropriate error handling
- [ ] I have added or updated documentation as needed
## Additional context
- This includes set up of following components:
- Victoriametrics single node cluster for storing metrics data with
persistent storage configuration for local and production env.
- Vmagent with target discovery for mcp-registry pods with scrape
interval of 30s.
- Grafana for visualising metrics and setting alerts.
- Datasource of victoriametrics is pre-configured as config map
- Persistent volume for basic configuration
- Sqlite for local and Postgres for production setup for storing
dashboard, alerts etc details
## Metrics in action
<img width="1422" height="666" alt="Screenshot 2025-08-31 at 4 35 28 AM"
src="https://github.com/user-attachments/assets/e54ce846-653c-4fa3-aa7a-fe277f0810e6"
/>
---------
Co-authored-by: Adam Jones <adamj@anthropic.com>
Co-authored-by: adam jones <adamj+git@anthropic.com>
<!-- Provide a brief summary of your changes -->
Implements automated database backup functionality with Google Cloud
Storage integration, including retention policies and MinIO support for
local development.
## Motivation and Context
<!-- Why is this change needed? What problem does it solve? -->
This change adds critical database backup capabilities to ensure data
durability and disaster recovery. The solution provides automated
backups with configurable retention policies and supports both
production (GCS) and development (MinIO) environments.
## How Has This Been Tested?
<!-- Have you tested this in a real application? Which scenarios were
tested? -->
- Tested backup creation and restoration with MinIO in local development
- Verified GCS bucket lifecycle policies for automatic deletion after 60
days
- Tested backup retention and cleanup logic
## Breaking Changes
<!-- Will users need to update their code or configurations? -->
No breaking changes. This is an additive feature that doesn't affect
existing functionality.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [x] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [x] My code follows the repository's style guidelines
- [x] New and existing tests pass locally
- [x] I have added appropriate error handling
- [x] I have added or updated documentation as needed
## Additional context
<!-- Add any other context, implementation notes, or design decisions
-->
- Implements 60-day retention period for backups as a safety net
- Uses GCS lifecycle rules for automatic cleanup
- Includes MinIO setup instructions for local development testing
- Port-forwarding commands updated for consistency across documentation
- Fixes#184
Fixes#228
## Motivation and Context
See #19
## How Has This Been Tested?
- Run locally
- Run in docker compose
- Run infra in minikube
- Unit and integration tests all passing :)
## Breaking Changes
Yes: users will not be able to access any data they used to have in
custom mongodb. We are accepting this breaking change, given the
registry is in experimental development status.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [x] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [x] My code follows the repository's style guidelines
- [x] New and existing tests pass locally
- [x] I have added appropriate error handling
- [x] I have added or updated documentation as needed
## Additional context
I have not implemented connection pooling support for simplicity, and
because cloudnative-pg already provides built-in connection pooling with
pgBouncer so if we want it later we can switch to this endpoint easily.
Radoslav Dimitrov
adam jones
pree-dew