27af672c5e## Summary Three changes from the 2026-04-27 incident: 1. **Cursor SQL fix** — `addCursorCondition` rewritten to use a row-constructor comparison `(server_name, version) > ($1, $2)` instead of the OR-decomposed form. Postgres can index-seek on the row constructor; it can't on the OR form, which scanned from the start of the index and filtered rows before the cursor. Cost grew linearly with cursor depth. 2. **Per-phase publish timings restored** — `createServerInTransaction` was simplified during #1211 review to log only `validate_ms`. The incident showed publishes spending 50+ s in `acquire_lock` / `version_checks` / `db_create` while `validate_ms` reported a few hundred ms — the diagnostic signal was hidden. Restored timings for every phase, refactored into a small `runPhase` helper. 3. **`pg_stat_statements` enabled** in the CNPG cluster spec. We had no aggregate query visibility during the incident — could only EXPLAIN queries we happened to suspect. ## Cursor fix evidence Local benchmark, 100K rows, cold cache: | Form | Rows filtered | Buffer reads | Time | |------|--------------:|-------------:|-----:| | OLD | 80,001 | 7,679 | 31.6 ms | | NEW | 2 | 1 | **0.05 ms** | Maps to prod's measured 8,911 buffer hits → 760 ms. End-to-end API: `/v0/servers?limit=100&cursor=…` returns in 4–7 ms regardless of depth. Risk check: confirmed `(server_name, version)` index exists in that column order (both `servers_pkey` and `idx_servers_name_version`), and both columns are `NOT NULL` so row-constructor comparison can't silently drop rows. ## Tests - New `TestPostgreSQL_PerformanceScenarios/compound_cursor_across_versions_of_same_server` pins multi-version pagination semantics. The existing cursor tests only exercised the fallback (single-component cursor) and degenerate (one version per server) cases. - `make lint` clean, `go test -race ./internal/... ./cmd/...` green. ## Deployment The cursor + slog changes are zero-downtime. The CNPG spec change triggers a brief PG restart (single-instance cluster). After the restart, run once on prod: ```bash kubectl exec -i registry-pg-1 -c postgres \ --context gke_mcp-registry-prod_us-central1-b_mcp-registry-prod \ -- psql -U postgres -c "CREATE EXTENSION IF NOT EXISTS pg_stat_statements" ``` Time the merge for a low-traffic UTC window. v1.7.1's DB retry-with-backoff covers the brief PG restart. ## Out of scope `MaxConns` bump, per-IP rate limiting at nginx, response caching, the pre-existing `superfluous WriteHeader` warnings — separate follow-ups. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
90 lines
3.1 KiB
Go
90 lines
3.1 KiB
Go
package k8s
|
|
|
|
import (
|
|
"github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
|
|
corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
|
|
"github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
|
|
metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
|
|
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
|
|
|
|
"github.com/modelcontextprotocol/registry/deploy/infra/pkg/providers"
|
|
)
|
|
|
|
// DeployPostgresDatabases deploys the CloudNative PostgreSQL operator and PostgreSQL cluster
|
|
func DeployPostgresDatabases(ctx *pulumi.Context, cluster *providers.ProviderInfo, environment string) (*apiextensions.CustomResource, error) {
|
|
// Create cnpg-system namespace
|
|
cnpgNamespace, err := corev1.NewNamespace(ctx, "cnpg-system", &corev1.NamespaceArgs{
|
|
Metadata: &metav1.ObjectMetaArgs{
|
|
Name: pulumi.String("cnpg-system"),
|
|
},
|
|
}, pulumi.Provider(cluster.Provider))
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
// Install cloudnative-pg Helm chart
|
|
cloudNativePG, err := helm.NewChart(ctx, "cloudnative-pg", helm.ChartArgs{
|
|
Chart: pulumi.String("cloudnative-pg"),
|
|
Version: pulumi.String("v0.26.0"),
|
|
FetchArgs: helm.FetchArgs{
|
|
Repo: pulumi.String("https://cloudnative-pg.github.io/charts"),
|
|
},
|
|
Namespace: cnpgNamespace.Metadata.Name().Elem(),
|
|
Values: pulumi.Map{
|
|
"webhooks": pulumi.Map{
|
|
"replicaCount": pulumi.Int(1),
|
|
},
|
|
},
|
|
}, pulumi.Provider(cluster.Provider))
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
// Create PostgreSQL cluster
|
|
pgCluster, err := apiextensions.NewCustomResource(ctx, "registry-pg", &apiextensions.CustomResourceArgs{
|
|
ApiVersion: pulumi.String("postgresql.cnpg.io/v1"),
|
|
Kind: pulumi.String("Cluster"),
|
|
Metadata: &metav1.ObjectMetaArgs{
|
|
Name: pulumi.String("registry-pg"),
|
|
Namespace: pulumi.String("default"),
|
|
Labels: pulumi.StringMap{
|
|
"app": pulumi.String("registry-pg"),
|
|
"environment": pulumi.String(environment),
|
|
},
|
|
},
|
|
OtherFields: map[string]any{
|
|
"spec": map[string]any{
|
|
"instances": 1,
|
|
"enablePDB": false,
|
|
"storage": map[string]any{
|
|
"size": "50Gi",
|
|
},
|
|
// Enable pg_stat_statements so we can attribute slow time to specific
|
|
// queries (the 2026-04-27 incident took an EXPLAIN-the-one-query-I-saw
|
|
// approach because we had no aggregate visibility). CNPG triggers a PG
|
|
// restart on shared_preload_libraries change — with instances: 1 this
|
|
// is brief downtime. CREATE EXTENSION still needs to run once as a
|
|
// superuser on existing clusters; new clusters get it via the
|
|
// postInitApplicationSQL hook below.
|
|
"postgresql": map[string]any{
|
|
"shared_preload_libraries": []any{"pg_stat_statements"},
|
|
"parameters": map[string]any{
|
|
"pg_stat_statements.track": "all",
|
|
},
|
|
},
|
|
"bootstrap": map[string]any{
|
|
"initdb": map[string]any{
|
|
"postInitApplicationSQL": []any{
|
|
"CREATE EXTENSION IF NOT EXISTS pg_stat_statements",
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
}, pulumi.Provider(cluster.Provider), pulumi.DependsOnInputs(cloudNativePG.Ready), pulumi.RetainOnDelete(true))
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
return pgCluster, nil
|
|
}
|