fix(list): use row-constructor cursor + restore per-phase publish timings + enable pg_stat_statements (#1215)
## Summary Three changes from the 2026-04-27 incident: 1. **Cursor SQL fix** — `addCursorCondition` rewritten to use a row-constructor comparison `(server_name, version) > ($1, $2)` instead of the OR-decomposed form. Postgres can index-seek on the row constructor; it can't on the OR form, which scanned from the start of the index and filtered rows before the cursor. Cost grew linearly with cursor depth. 2. **Per-phase publish timings restored** — `createServerInTransaction` was simplified during #1211 review to log only `validate_ms`. The incident showed publishes spending 50+ s in `acquire_lock` / `version_checks` / `db_create` while `validate_ms` reported a few hundred ms — the diagnostic signal was hidden. Restored timings for every phase, refactored into a small `runPhase` helper. 3. **`pg_stat_statements` enabled** in the CNPG cluster spec. We had no aggregate query visibility during the incident — could only EXPLAIN queries we happened to suspect. ## Cursor fix evidence Local benchmark, 100K rows, cold cache: | Form | Rows filtered | Buffer reads | Time | |------|--------------:|-------------:|-----:| | OLD | 80,001 | 7,679 | 31.6 ms | | NEW | 2 | 1 | **0.05 ms** | Maps to prod's measured 8,911 buffer hits → 760 ms. End-to-end API: `/v0/servers?limit=100&cursor=…` returns in 4–7 ms regardless of depth. Risk check: confirmed `(server_name, version)` index exists in that column order (both `servers_pkey` and `idx_servers_name_version`), and both columns are `NOT NULL` so row-constructor comparison can't silently drop rows. ## Tests - New `TestPostgreSQL_PerformanceScenarios/compound_cursor_across_versions_of_same_server` pins multi-version pagination semantics. The existing cursor tests only exercised the fallback (single-component cursor) and degenerate (one version per server) cases. - `make lint` clean, `go test -race ./internal/... ./cmd/...` green. ## Deployment The cursor + slog changes are zero-downtime. The CNPG spec change triggers a brief PG restart (single-instance cluster). After the restart, run once on prod: ```bash kubectl exec -i registry-pg-1 -c postgres \ --context gke_mcp-registry-prod_us-central1-b_mcp-registry-prod \ -- psql -U postgres -c "CREATE EXTENSION IF NOT EXISTS pg_stat_statements" ``` Time the merge for a low-traffic UTC window. v1.7.1's DB retry-with-backoff covers the brief PG restart. ## Out of scope `MaxConns` bump, per-IP rate limiting at nginx, response caching, the pre-existing `superfluous WriteHeader` warnings — separate follow-ups. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
762a4849a6
commit
27af672c5e
1 changed files with 232 additions and 50 deletions
|
|
@ -127,7 +127,16 @@ func buildFilterConditions(filter *ServerFilter, argIndex int) ([]string, []any,
|
|||
return conditions, args, argIndex
|
||||
}
|
||||
|
||||
// addCursorCondition adds pagination cursor condition to WHERE clause
|
||||
// addCursorCondition adds pagination cursor condition to WHERE clause.
|
||||
//
|
||||
// The compound cursor uses a row-constructor comparison so PostgreSQL can seek
|
||||
// directly into the (server_name, version) B-tree index. The OR-decomposed form
|
||||
// `server_name > X OR (server_name = X AND version > Y)` is logically equivalent
|
||||
// but PostgreSQL's planner cannot use it for an index seek — it scans the index
|
||||
// from the start and filters everything before the cursor, making cost grow
|
||||
// linearly with cursor depth (a 20K-row table at a deep cursor took ~760ms in
|
||||
// prod). The row-constructor form `(server_name, version) > (X, Y)` is special-
|
||||
// cased and stays constant-time regardless of cursor depth.
|
||||
func addCursorCondition(cursor string, argIndex int) (string, []any, int) {
|
||||
if cursor == "" {
|
||||
return "", nil, argIndex
|
||||
|
|
@ -138,9 +147,8 @@ func addCursorCondition(cursor string, argIndex int) (string, []any, int) {
|
|||
if len(parts) == 2 {
|
||||
cursorServerName := parts[0]
|
||||
cursorVersion := parts[1]
|
||||
// Use compound condition: (server_name > cursor_name) OR (server_name = cursor_name AND version > cursor_version)
|
||||
condition := fmt.Sprintf("(server_name > $%d OR (server_name = $%d AND version > $%d))", argIndex, argIndex+1, argIndex+2)
|
||||
return condition, []any{cursorServerName, cursorServerName, cursorVersion}, argIndex + 3
|
||||
condition := fmt.Sprintf("(server_name, version) > ($%d, $%d)", argIndex, argIndex+1)
|
||||
return condition, []any{cursorServerName, cursorVersion}, argIndex + 2
|
||||
}
|
||||
|
||||
// Fallback for malformed cursor - treat as server name only for backwards compatibility
|
||||
|
|
|
|||
Loading…
Reference in a new issue