fix(deploy): bump registry pod memory to match doubled pgxpool size (#1225)

## Summary

Follow-up to #1221. That PR doubled pgxpool `MaxConns` from 30 to 60 per
pod without bumping the registry pod memory limit. pgxpool keeps
per-connection state in the Go process (prepared statement cache,
scratch buffers, row iteration state), so memory grew roughly
proportionally — peak per-pod usage shifted from ~330 MB to ~450 MB
against a 512 MiB limit.

## What surfaced it

Pod `tx49f` was OOMKilled at 17:01 UTC on 2026-04-29 (`exitCode 137`,
`reason: OOMKilled`). Memory history from Cloud Monitoring confirms the
regression:

| Pods | Peak 30-min memory |
|------|-------------------:|
| Pre-#1221 (`MaxConns=30`) | 178–333 MB |
| **Post-#1221** (`MaxConns=60`) | **437–449 MB** |
| Old limit | **512 MiB** ← right at the edge |

Other pod was at 394 MB at the time of inspection — could OOM again.

## Fix

`deploy/pkg/k8s/registry.go:217-225`:

| | Before | After |
|---|-------:|------:|
| `requests.memory` | 256 MiB | **512 MiB** |
| `limits.memory` | 512 MiB | **1 GiB** |

1 GiB is ~2× the current peak under load — gives room for traffic spikes
without immediate OOM risk. Either node has 4–5 GiB free allocatable
memory, so the new request fits comfortably.

## Test plan

- [x] `go build ./...` clean
- [x] `golangci-lint run` clean
- [ ] After release + prod promotion: pod memory stabilises around 450
MB peak with no OOMKills
- [ ] After release + prod promotion: tx49f restartCount stops climbing
(currently at 5)

## Deployment

Zero downtime — Deployment template change only, rolling update with
`maxSurge: 1, maxUnavailable: 0`. **No PG involvement** (no
`max_connections` change, no shared_preload_libraries change). Same
shape as a normal v1.7.x image promotion.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Radoslav Dimitrov 2026-04-29 14:05:09 -04:00 • committed by GitHub
commit 5dcae5b3ce
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

View file

@ -215,12 +215,20 @@ func DeployMCPRegistry(ctx *pulumi.Context, cluster *providers.ProviderInfo, env
TimeoutSeconds: pulumi.Int(3),
},
Resources: &corev1.ResourceRequirementsArgs{
// Memory was 256Mi/512Mi when pgxpool MaxConns was 30. After
// #1221 doubled MaxConns to 60 per pod, peak per-pod memory
// shifted from ~330MB to ~450MB under scraper load and one
// pod was OOMKilled at 17:01 UTC on 2026-04-29. pgxpool keeps
// per-connection state (prepared statement cache, scratch
// buffers, row iteration state) so memory scales roughly
// linearly with the connection count. Doubled both request
// and limit to match.
Requests: pulumi.StringMap{
"memory": pulumi.String("256Mi"),
"memory": pulumi.String("512Mi"),
"cpu": pulumi.String("100m"),
},
Limits: pulumi.StringMap{
"memory": pulumi.String("512Mi"),
"memory": pulumi.String("1Gi"),
},
},
},