fix(deploy): bump registry pod memory to match doubled pgxpool size (#1225)
## Summary Follow-up to #1221. That PR doubled pgxpool `MaxConns` from 30 to 60 per pod without bumping the registry pod memory limit. pgxpool keeps per-connection state in the Go process (prepared statement cache, scratch buffers, row iteration state), so memory grew roughly proportionally — peak per-pod usage shifted from ~330 MB to ~450 MB against a 512 MiB limit. ## What surfaced it Pod `tx49f` was OOMKilled at 17:01 UTC on 2026-04-29 (`exitCode 137`, `reason: OOMKilled`). Memory history from Cloud Monitoring confirms the regression: | Pods | Peak 30-min memory | |------|-------------------:| | Pre-#1221 (`MaxConns=30`) | 178–333 MB | | **Post-#1221** (`MaxConns=60`) | **437–449 MB** | | Old limit | **512 MiB** ← right at the edge | Other pod was at 394 MB at the time of inspection — could OOM again. ## Fix `deploy/pkg/k8s/registry.go:217-225`: | | Before | After | |---|-------:|------:| | `requests.memory` | 256 MiB | **512 MiB** | | `limits.memory` | 512 MiB | **1 GiB** | 1 GiB is ~2× the current peak under load — gives room for traffic spikes without immediate OOM risk. Either node has 4–5 GiB free allocatable memory, so the new request fits comfortably. ## Test plan - [x] `go build ./...` clean - [x] `golangci-lint run` clean - [ ] After release + prod promotion: pod memory stabilises around 450 MB peak with no OOMKills - [ ] After release + prod promotion: tx49f restartCount stops climbing (currently at 5) ## Deployment Zero downtime — Deployment template change only, rolling update with `maxSurge: 1, maxUnavailable: 0`. **No PG involvement** (no `max_connections` change, no shared_preload_libraries change). Same shape as a normal v1.7.x image promotion. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
0a97e59ecb
commit
5dcae5b3ce
1 changed files with 10 additions and 2 deletions
|
|
@ -215,12 +215,20 @@ func DeployMCPRegistry(ctx *pulumi.Context, cluster *providers.ProviderInfo, env
|
|||
TimeoutSeconds: pulumi.Int(3),
|
||||
},
|
||||
Resources: &corev1.ResourceRequirementsArgs{
|
||||
// Memory was 256Mi/512Mi when pgxpool MaxConns was 30. After
|
||||
// #1221 doubled MaxConns to 60 per pod, peak per-pod memory
|
||||
// shifted from ~330MB to ~450MB under scraper load and one
|
||||
// pod was OOMKilled at 17:01 UTC on 2026-04-29. pgxpool keeps
|
||||
// per-connection state (prepared statement cache, scratch
|
||||
// buffers, row iteration state) so memory scales roughly
|
||||
// linearly with the connection count. Doubled both request
|
||||
// and limit to match.
|
||||
Requests: pulumi.StringMap{
|
||||
"memory": pulumi.String("256Mi"),
|
||||
"memory": pulumi.String("512Mi"),
|
||||
"cpu": pulumi.String("100m"),
|
||||
},
|
||||
Limits: pulumi.StringMap{
|
||||
"memory": pulumi.String("512Mi"),
|
||||
"memory": pulumi.String("1Gi"),
|
||||
},
|
||||
},
|
||||
},
|
||||
|
|
|
|||
Loading…
Reference in a new issue