fix: harden registry availability and add publish phase timings (#1211)

## Summary

Daily ~17:00 UTC `Publish Endpoint Latency` and `Availability < 95%`
alerts are caused by both registry pods exit-0'ing within seconds of
each other and racing with the still-recovering `registry-pg-rw`
endpoint on restart. Separately, the default compute SA had no project
IAM roles, so `fluentbit-gke` and the GMP collector were silently
dropping every log line and time series — which is why none of this was
visible in Cloud Logging.

All Kubernetes/IAM changes apply uniformly to staging and prod.

## Changes

- **GCP IAM** (`deploy/pkg/providers/gcp/provider.go`) — grant the
default compute SA `logging.logWriter`, `monitoring.metricWriter`,
`monitoring.viewer`, and `stackdriver.resourceMetadata.writer`. Already
applied live; this just encodes it in Pulumi.
- **Pod resilience** (`deploy/pkg/k8s/registry.go`) —
- `PodDisruptionBudget` (`minAvailable: 1`) prevents a synchronized
voluntary disruption.
- `TopologySpreadConstraints` (`ScheduleAnyway`, hostname) keeps the PDB
from deadlocking node drains when both pods co-locate.
- `StartupProbe` (30 × 5s = 150s) protects the DB-retry budget from the
liveness probe.
  - Memory bump 128→256Mi request, 256→512Mi limit.
- **Startup retry** (`cmd/registry/main.go`) — replace silent `return`
on initial DB-ping failure with bounded retry-with-backoff (8 attempts,
1→8s capped, 10s per-attempt). A brief endpoint flap no longer cascades
into a restart loop.
- **Diagnostic log** (`internal/service/registry_service.go`) — emit
`validate_ms` on each publish (success and failure). Validate fans out
to npm/PyPI/OCI with 10s per-host timeouts and is the most likely
contributor to publish latency. Total request duration is already
covered by the existing `http.request.duration` histogram.

## Calibration

The IAM, PDB, topology-spread, startup-probe, and DB-retry pieces are
mitigations — they reduce blast radius regardless of root cause. The
`validate_ms` log is the diagnostic for identifying the trigger when the
alert fires next. Single-PG SPOF (`deploy/pkg/k8s/postgres.go:57`)
remains a follow-up; today's evidence shows PG was a victim, not the
cause.

## Test plan

- [x] `go build ./...` and `go vet ./...` clean for both modules
- [x] `make lint` clean (12 pre-existing issues under `deploy/` are not
in `make lint`)
- [x] `go test -race -count=1 ./internal/... ./cmd/...` green
- [x] IAM applied live; `fluentbit-gke` and GMP no longer hitting
`IAM_PERMISSION_DENIED`
- [ ] After deploy, verify the next 17:00 UTC firing surfaces
`validate_ms` in Cloud Logging

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

cmd/registry/main.go

@@ -57,15 +57,31 @@ func main() {
 	// Initialize configuration
 	cfg := config.NewConfig()
 
-	// Create a context with timeout for PostgreSQL connection
-	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
-	defer cancel()
-
-	// Connect to PostgreSQL
-	db, err = database.NewPostgreSQL(ctx, cfg.DatabaseURL)
-	if err != nil {
-		log.Printf("Failed to connect to PostgreSQL: %v", err)
-		return
+	// Connect to PostgreSQL with bounded retry. The DB endpoint can flap briefly
+	// during voluntary disruptions (node drains, rollouts) and without retry the
+	// pod exits cleanly, restarts, then races with the still-recovering service
+	// — turning a transient blip into a multi-minute outage.
+	const (
+		maxStartupAttempts = 8
+		perAttemptTimeout  = 10 * time.Second
+		initialBackoff     = 1 * time.Second
+		maxBackoff         = 8 * time.Second
+	)
+	backoff := initialBackoff
+	for attempt := 1; ; attempt++ {
+		attemptCtx, attemptCancel := context.WithTimeout(context.Background(), perAttemptTimeout)
+		db, err = database.NewPostgreSQL(attemptCtx, cfg.DatabaseURL)
+		attemptCancel()
+		if err == nil {
+			break
+		}
+		if attempt >= maxStartupAttempts {
+			log.Printf("Failed to connect to PostgreSQL after %d attempts: %v", attempt, err)
+			return
+		}
+		log.Printf("PostgreSQL not reachable (attempt %d/%d): %v, retrying in %s", attempt, maxStartupAttempts, err, backoff)
+		time.Sleep(backoff)
+		backoff = min(backoff*2, maxBackoff)
 	}
 
 	// Store the PostgreSQL instance for later cleanup