fix: harden registry availability and add publish phase timings (#1211)

## Summary

Daily ~17:00 UTC `Publish Endpoint Latency` and `Availability < 95%`
alerts are caused by both registry pods exit-0'ing within seconds of
each other and racing with the still-recovering `registry-pg-rw`
endpoint on restart. Separately, the default compute SA had no project
IAM roles, so `fluentbit-gke` and the GMP collector were silently
dropping every log line and time series — which is why none of this was
visible in Cloud Logging.

All Kubernetes/IAM changes apply uniformly to staging and prod.

## Changes

- **GCP IAM** (`deploy/pkg/providers/gcp/provider.go`) — grant the
default compute SA `logging.logWriter`, `monitoring.metricWriter`,
`monitoring.viewer`, and `stackdriver.resourceMetadata.writer`. Already
applied live; this just encodes it in Pulumi.
- **Pod resilience** (`deploy/pkg/k8s/registry.go`) —
- `PodDisruptionBudget` (`minAvailable: 1`) prevents a synchronized
voluntary disruption.
- `TopologySpreadConstraints` (`ScheduleAnyway`, hostname) keeps the PDB
from deadlocking node drains when both pods co-locate.
- `StartupProbe` (30 × 5s = 150s) protects the DB-retry budget from the
liveness probe.
  - Memory bump 128→256Mi request, 256→512Mi limit.
- **Startup retry** (`cmd/registry/main.go`) — replace silent `return`
on initial DB-ping failure with bounded retry-with-backoff (8 attempts,
1→8s capped, 10s per-attempt). A brief endpoint flap no longer cascades
into a restart loop.
- **Diagnostic log** (`internal/service/registry_service.go`) — emit
`validate_ms` on each publish (success and failure). Validate fans out
to npm/PyPI/OCI with 10s per-host timeouts and is the most likely
contributor to publish latency. Total request duration is already
covered by the existing `http.request.duration` histogram.

## Calibration

The IAM, PDB, topology-spread, startup-probe, and DB-retry pieces are
mitigations — they reduce blast radius regardless of root cause. The
`validate_ms` log is the diagnostic for identifying the trigger when the
alert fires next. Single-PG SPOF (`deploy/pkg/k8s/postgres.go:57`)
remains a follow-up; today's evidence shows PG was a victim, not the
cause.

## Test plan

- [x] `go build ./...` and `go vet ./...` clean for both modules
- [x] `make lint` clean (12 pre-existing issues under `deploy/` are not
in `make lint`)
- [x] `go test -race -count=1 ./internal/... ./cmd/...` green
- [x] IAM applied live; `fluentbit-gke` and GMP no longer hitting
`IAM_PERMISSION_DENIED`
- [ ] After deploy, verify the next 17:00 UTC firing surfaces
`validate_ms` in Cloud Logging

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

internal/service/registry_service.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"errors"
 	"fmt"
+	"log/slog"
 	"time"
 
 	"github.com/jackc/pgx/v5"
@@ -85,15 +86,29 @@ func (s *registryServiceImpl) CreateServer(ctx context.Context, req *apiv0.Serve
 	})
 }
 
-// createServerInTransaction contains the actual CreateServer logic within a transaction
+// createServerInTransaction contains the actual CreateServer logic within a transaction.
 func (s *registryServiceImpl) createServerInTransaction(ctx context.Context, tx pgx.Tx, req *apiv0.ServerJSON) (*apiv0.ServerResponse, error) {
-	// Validate the request
-	if err := validators.ValidatePublishRequest(ctx, *req, s.cfg); err != nil {
-		return nil, err
+	serverJSON := *req
+
+	// Validate the request. ValidatePublishRequest fans out to npm/PyPI/OCI for
+	// registry-ownership checks with 10s per-host timeouts, so it's the most likely
+	// contributor to publish latency. Log validate_ms on both success and failure
+	// so the next /v0/publish latency alert is diagnostic — a slow upstream that
+	// times out into a validation error is exactly the case we need to see.
+	validateStart := time.Now()
+	validateErr := validators.ValidatePublishRequest(ctx, serverJSON, s.cfg)
+	validateMs := time.Since(validateStart).Milliseconds()
+	if validateErr != nil {
+		slog.WarnContext(ctx, "publish validate failed",
+			"server_name", serverJSON.Name,
+			"version", serverJSON.Version,
+			"validate_ms", validateMs,
+			"error", validateErr.Error(),
+		)
+		return nil, validateErr
 	}
 
 	publishTime := time.Now()
-	serverJSON := *req
 
 	// Acquire advisory lock to prevent concurrent publishes of the same server
 	if err := s.db.AcquirePublishLock(ctx, tx, serverJSON.Name); err != nil {
@@ -161,7 +176,17 @@ func (s *registryServiceImpl) createServerInTransaction(ctx context.Context, tx
 	}
 
 	// Insert new server version
-	return s.db.CreateServer(ctx, tx, &serverJSON, officialMeta)
+	resp, err := s.db.CreateServer(ctx, tx, &serverJSON, officialMeta)
+	if err != nil {
+		return nil, err
+	}
+
+	slog.InfoContext(ctx, "publish complete",
+		"server_name", serverJSON.Name,
+		"version", serverJSON.Version,
+		"validate_ms", validateMs,
+	)
+	return resp, nil
 }
 
 // recalculateLatest picks the highest non-deleted version of the given server and flags it