registry/deploy/pkg/k8s
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Radoslav Dimitrov 5dcae5b3ce
fix(deploy): bump registry pod memory to match doubled pgxpool size (#1225)
## Summary

Follow-up to #1221. That PR doubled pgxpool `MaxConns` from 30 to 60 per
pod without bumping the registry pod memory limit. pgxpool keeps
per-connection state in the Go process (prepared statement cache,
scratch buffers, row iteration state), so memory grew roughly
proportionally — peak per-pod usage shifted from ~330 MB to ~450 MB
against a 512 MiB limit.

## What surfaced it

Pod `tx49f` was OOMKilled at 17:01 UTC on 2026-04-29 (`exitCode 137`,
`reason: OOMKilled`). Memory history from Cloud Monitoring confirms the
regression:

| Pods | Peak 30-min memory |
|------|-------------------:|
| Pre-#1221 (`MaxConns=30`) | 178–333 MB |
| **Post-#1221** (`MaxConns=60`) | **437–449 MB** |
| Old limit | **512 MiB** ← right at the edge |

Other pod was at 394 MB at the time of inspection — could OOM again.

## Fix

`deploy/pkg/k8s/registry.go:217-225`:

| | Before | After |
|---|-------:|------:|
| `requests.memory` | 256 MiB | **512 MiB** |
| `limits.memory` | 512 MiB | **1 GiB** |

1 GiB is ~2× the current peak under load — gives room for traffic spikes
without immediate OOM risk. Either node has 4–5 GiB free allocatable
memory, so the new request fits comfortably.

## Test plan

- [x] `go build ./...` clean
- [x] `golangci-lint run` clean
- [ ] After release + prod promotion: pod memory stabilises around 450
MB peak with no OOMKills
- [ ] After release + prod promotion: tx49f restartCount stops climbing
(currently at 5)

## Deployment

Zero downtime — Deployment template change only, rolling update with
`maxSurge: 1, maxUnavailable: 0`. **No PG involvement** (no
`max_connections` change, no shared_preload_libraries change). Same
shape as a normal v1.7.x image promotion.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 14:05:09 -04:00
..
backup.go Add database backup functionality with GCS integration (#297) 2025-08-25 17:27:40 +01:00
cert_manager.go Refactor database implementation to use PostgreSQL instead of MongoDB (#289) 2025-08-22 01:40:28 +01:00
deploy.go Add monitoring infrastructure: victoriametrics and grafana (#328) 2025-09-02 12:40:43 +00:00
ingress.go Set 429 status code responses in the config map (#849) 2025-12-13 13:34:12 -07:00
monitoring.go Add failure modes telemetry (#646) 2025-10-20 11:57:49 -07:00
postgres.go fix(db): raise pgxpool MaxConns + PG max_connections + set PG resources (#1221) 2026-04-28 15:21:24 -04:00
registry.go fix(deploy): bump registry pod memory to match doubled pgxpool size (#1225) 2026-04-29 14:05:09 -04:00