fix(deploy): cache bulk registry reads (#1509)

## Summary

- cache successful public list responses for `/v0/servers` and
`/v0.1/servers` at ingress-nginx for 30 seconds
- give only those exact cached paths a 600 RPM per-controller budget
- preserve the existing 180 RPM limit for publish, auth, mutation,
detail, and other routes
- expose `X-Registry-Cache` to make cache MISS/HIT behavior directly
observable

## Why

Production ingress logs show that the current 180 RPM per-controller
limit rejects legitimate bulk pagination before it reaches the
application. The affected traffic is overwhelmingly registry-list reads,
not publishing.

A ten-minute production sample contained 17,836 successful list
responses but only 3,188 unique URLs: 82.1% were repeated requests,
representing roughly 1 GB of duplicate response bodies. Raising the list
limit without caching would increase pressure on the application and its
shared PostgreSQL pool. The short ingress cache absorbs repeated
cursor-page reads before raising their scoped client budget.

Publish latency is a separate concern. Reducing read pressure may help
indirectly, but slow external package validation is intentionally out of
scope for this PR.

Relates to #826.

## Safety

- caching is limited to exact public GET list paths and keyed by host
plus full request URI
- only 200 responses are cached
- cache lifetime is 30 seconds; newly published entries can therefore
take up to 30 seconds to appear on a cached page
- cache locking coalesces simultaneous misses
- stale cached responses may be served while refreshing or during
transient upstream 5xx/timeouts
- each ingress replica uses at most 1 GB of ephemeral on-disk cache

## Validation

- `go test ./...` in `deploy/`
- `go vet ./...` in `deploy/`
- `git diff --check`
- ingress-nginx chart 4.13.0 in a disposable kind cluster
- generated NGINX configuration passed `nginx -t`
- repeated list request verified `X-Registry-Cache: MISS` followed by
`HIT`
- non-list route verified uncached
- exact list locations verified with the higher rate zone while the
catch-all retained 180 RPM

Nothing was applied to production during investigation or validation.
This commit is contained in:
Radoslav Dimitrov 2026-08-07 02:16:52 +03:00 • committed by GitHub
commit f52dc8525a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

View file

@ -65,6 +65,31 @@ func SetupIngressController(ctx *pulumi.Context, cluster *providers.ProviderInfo
"annotations": pulumi.Map{},
},
"config": pulumi.Map{
// Cache the public list endpoints at the ingress. Registry consumers
// repeatedly walk the same cursor pages, so sending every walk to the
// application needlessly consumes application CPU and database pool
// connections. The cache is intentionally short-lived so newly
// published servers become visible quickly.
//
// Use a variable cache zone so this global location snippet remains a
// no-op for publish, auth, mutation, and individual-server routes.
"http-snippet": pulumi.String(`
proxy_cache_path /tmp/registry-read-cache levels=1:2 keys_zone=registry_read_cache:20m max_size=1g inactive=5m use_temp_path=off;
map "$request_method:$uri" $registry_read_cache_zone {
default off;
~^GET:/v0(?:\.1)?/servers$ registry_read_cache;
}
`),
"location-snippet": pulumi.String(`
proxy_cache $registry_read_cache_zone;
proxy_cache_key "$scheme$request_method$host$request_uri";
proxy_cache_valid 200 30s;
proxy_cache_lock on;
proxy_cache_lock_timeout 5s;
proxy_cache_background_update on;
proxy_cache_use_stale updating error timeout http_500 http_502 http_503 http_504;
add_header X-Registry-Cache $upstream_cache_status always;
`),
// Disable strict path validation, to work around a bug in ingress-nginx
// https://cert-manager.io/docs/releases/release-notes/release-notes-1.18/#acme-http01-challenge-paths-now-use-pathtype-exact-in-ingress-routes
// https://github.com/kubernetes/ingress-nginx/issues/11176