fix(deploy): cache bulk registry reads (#1509)
## Summary - cache successful public list responses for `/v0/servers` and `/v0.1/servers` at ingress-nginx for 30 seconds - give only those exact cached paths a 600 RPM per-controller budget - preserve the existing 180 RPM limit for publish, auth, mutation, detail, and other routes - expose `X-Registry-Cache` to make cache MISS/HIT behavior directly observable ## Why Production ingress logs show that the current 180 RPM per-controller limit rejects legitimate bulk pagination before it reaches the application. The affected traffic is overwhelmingly registry-list reads, not publishing. A ten-minute production sample contained 17,836 successful list responses but only 3,188 unique URLs: 82.1% were repeated requests, representing roughly 1 GB of duplicate response bodies. Raising the list limit without caching would increase pressure on the application and its shared PostgreSQL pool. The short ingress cache absorbs repeated cursor-page reads before raising their scoped client budget. Publish latency is a separate concern. Reducing read pressure may help indirectly, but slow external package validation is intentionally out of scope for this PR. Relates to #826. ## Safety - caching is limited to exact public GET list paths and keyed by host plus full request URI - only 200 responses are cached - cache lifetime is 30 seconds; newly published entries can therefore take up to 30 seconds to appear on a cached page - cache locking coalesces simultaneous misses - stale cached responses may be served while refreshing or during transient upstream 5xx/timeouts - each ingress replica uses at most 1 GB of ephemeral on-disk cache ## Validation - `go test ./...` in `deploy/` - `go vet ./...` in `deploy/` - `git diff --check` - ingress-nginx chart 4.13.0 in a disposable kind cluster - generated NGINX configuration passed `nginx -t` - repeated list request verified `X-Registry-Cache: MISS` followed by `HIT` - non-list route verified uncached - exact list locations verified with the higher rate zone while the catch-all retained 180 RPM Nothing was applied to production during investigation or validation.
This commit is contained in:
parent
2b09bde534
commit
f52dc8525a
1 changed files with 112 additions and 5 deletions
|
|
@ -65,6 +65,31 @@ func SetupIngressController(ctx *pulumi.Context, cluster *providers.ProviderInfo
|
|||
"annotations": pulumi.Map{},
|
||||
},
|
||||
"config": pulumi.Map{
|
||||
// Cache the public list endpoints at the ingress. Registry consumers
|
||||
// repeatedly walk the same cursor pages, so sending every walk to the
|
||||
// application needlessly consumes application CPU and database pool
|
||||
// connections. The cache is intentionally short-lived so newly
|
||||
// published servers become visible quickly.
|
||||
//
|
||||
// Use a variable cache zone so this global location snippet remains a
|
||||
// no-op for publish, auth, mutation, and individual-server routes.
|
||||
"http-snippet": pulumi.String(`
|
||||
proxy_cache_path /tmp/registry-read-cache levels=1:2 keys_zone=registry_read_cache:20m max_size=1g inactive=5m use_temp_path=off;
|
||||
map "$request_method:$uri" $registry_read_cache_zone {
|
||||
default off;
|
||||
~^GET:/v0(?:\.1)?/servers$ registry_read_cache;
|
||||
}
|
||||
`),
|
||||
"location-snippet": pulumi.String(`
|
||||
proxy_cache $registry_read_cache_zone;
|
||||
proxy_cache_key "$scheme$request_method$host$request_uri";
|
||||
proxy_cache_valid 200 30s;
|
||||
proxy_cache_lock on;
|
||||
proxy_cache_lock_timeout 5s;
|
||||
proxy_cache_background_update on;
|
||||
proxy_cache_use_stale updating error timeout http_500 http_502 http_503 http_504;
|
||||
add_header X-Registry-Cache $upstream_cache_status always;
|
||||
`),
|
||||
// Disable strict path validation, to work around a bug in ingress-nginx
|
||||
// https://cert-manager.io/docs/releases/release-notes/release-notes-1.18/#acme-http01-challenge-paths-now-use-pathtype-exact-in-ingress-routes
|
||||
// https://github.com/kubernetes/ingress-nginx/issues/11176
|
||||
|
|
|
|||
Loading…
Reference in a new issue