From 5dcae5b3cee0f5ebfcc3c65c196bc5f89c6a1a68 Mon Sep 17 00:00:00 2001 From: Radoslav Dimitrov Date: Wed, 29 Apr 2026 14:05:09 -0400 Subject: [PATCH] fix(deploy): bump registry pod memory to match doubled pgxpool size (#1225) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Follow-up to #1221. That PR doubled pgxpool `MaxConns` from 30 to 60 per pod without bumping the registry pod memory limit. pgxpool keeps per-connection state in the Go process (prepared statement cache, scratch buffers, row iteration state), so memory grew roughly proportionally — peak per-pod usage shifted from ~330 MB to ~450 MB against a 512 MiB limit. ## What surfaced it Pod `tx49f` was OOMKilled at 17:01 UTC on 2026-04-29 (`exitCode 137`, `reason: OOMKilled`). Memory history from Cloud Monitoring confirms the regression: | Pods | Peak 30-min memory | |------|-------------------:| | Pre-#1221 (`MaxConns=30`) | 178–333 MB | | **Post-#1221** (`MaxConns=60`) | **437–449 MB** | | Old limit | **512 MiB** ← right at the edge | Other pod was at 394 MB at the time of inspection — could OOM again. ## Fix `deploy/pkg/k8s/registry.go:217-225`: | | Before | After | |---|-------:|------:| | `requests.memory` | 256 MiB | **512 MiB** | | `limits.memory` | 512 MiB | **1 GiB** | 1 GiB is ~2× the current peak under load — gives room for traffic spikes without immediate OOM risk. Either node has 4–5 GiB free allocatable memory, so the new request fits comfortably. ## Test plan - [x] `go build ./...` clean - [x] `golangci-lint run` clean - [ ] After release + prod promotion: pod memory stabilises around 450 MB peak with no OOMKills - [ ] After release + prod promotion: tx49f restartCount stops climbing (currently at 5) ## Deployment Zero downtime — Deployment template change only, rolling update with `maxSurge: 1, maxUnavailable: 0`. **No PG involvement** (no `max_connections` change, no shared_preload_libraries change). Same shape as a normal v1.7.x image promotion. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) --- deploy/pkg/k8s/registry.go | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/deploy/pkg/k8s/registry.go b/deploy/pkg/k8s/registry.go index 579f740..48e9124 100644 --- a/deploy/pkg/k8s/registry.go +++ b/deploy/pkg/k8s/registry.go @@ -215,12 +215,20 @@ func DeployMCPRegistry(ctx *pulumi.Context, cluster *providers.ProviderInfo, env TimeoutSeconds: pulumi.Int(3), }, Resources: &corev1.ResourceRequirementsArgs{ + // Memory was 256Mi/512Mi when pgxpool MaxConns was 30. After + // #1221 doubled MaxConns to 60 per pod, peak per-pod memory + // shifted from ~330MB to ~450MB under scraper load and one + // pod was OOMKilled at 17:01 UTC on 2026-04-29. pgxpool keeps + // per-connection state (prepared statement cache, scratch + // buffers, row iteration state) so memory scales roughly + // linearly with the connection count. Doubled both request + // and limit to match. Requests: pulumi.StringMap{ - "memory": pulumi.String("256Mi"), + "memory": pulumi.String("512Mi"), "cpu": pulumi.String("100m"), }, Limits: pulumi.StringMap{ - "memory": pulumi.String("512Mi"), + "memory": pulumi.String("1Gi"), }, }, },