registry/deploy/pkg
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Radoslav Dimitrov 5dcae5b3ce
fix(deploy): bump registry pod memory to match doubled pgxpool size (#1225)
## Summary

Follow-up to #1221. That PR doubled pgxpool `MaxConns` from 30 to 60 per
pod without bumping the registry pod memory limit. pgxpool keeps
per-connection state in the Go process (prepared statement cache,
scratch buffers, row iteration state), so memory grew roughly
proportionally — peak per-pod usage shifted from ~330 MB to ~450 MB
against a 512 MiB limit.

## What surfaced it

Pod `tx49f` was OOMKilled at 17:01 UTC on 2026-04-29 (`exitCode 137`,
`reason: OOMKilled`). Memory history from Cloud Monitoring confirms the
regression:

| Pods | Peak 30-min memory |
|------|-------------------:|
| Pre-#1221 (`MaxConns=30`) | 178–333 MB |
| **Post-#1221** (`MaxConns=60`) | **437–449 MB** |
| Old limit | **512 MiB** ← right at the edge |

Other pod was at 394 MB at the time of inspection — could OOM again.

## Fix

`deploy/pkg/k8s/registry.go:217-225`:

| | Before | After |
|---|-------:|------:|
| `requests.memory` | 256 MiB | **512 MiB** |
| `limits.memory` | 512 MiB | **1 GiB** |

1 GiB is ~2× the current peak under load — gives room for traffic spikes
without immediate OOM risk. Either node has 4–5 GiB free allocatable
memory, so the new request fits comfortably.

## Test plan

- [x] `go build ./...` clean
- [x] `golangci-lint run` clean
- [ ] After release + prod promotion: pod memory stabilises around 450
MB peak with no OOMKills
- [ ] After release + prod promotion: tx49f restartCount stops climbing
(currently at 5)

## Deployment

Zero downtime — Deployment template change only, rolling update with
`maxSurge: 1, maxUnavailable: 0`. **No PG involvement** (no
`max_connections` change, no shared_preload_libraries change). Same
shape as a normal v1.7.x image promotion.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 14:05:09 -04:00
..
k8s fix(deploy): bump registry pod memory to match doubled pgxpool size (#1225) 2026-04-29 14:05:09 -04:00
providers fix(deploy): manage Cloud Resource Manager API enablement in Pulumi (#1213) 2026-04-27 12:32:22 -04:00