modelcontextprotocol/registry
Watch
1
0
Fork
You've already forked registry
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Introduce a cron job that syncs the prod base into staging (#633)

Browse source 
<!-- Provide a brief summary of your changes -->

## Motivation and Context
<!-- Why is this change needed? What problem does it solve? -->
The following PR adds a workflow that should:
* Runs daily but also can be manually triggered
* Drops the staging database and copies the prod database into it

This should improve our release confidence by reducing the likelihood of
regressions or unexpected breakages.
 
Copy from Claude:
```bash
What This Does:

1. Connects to prod → Extracts backup credentials
2. Switches to staging → Configures backup access
3. Creates restore PVC → 50GB temporary storage
4. Triggers k8up restore → Downloads latest prod backup
5. Waits for restore → With proper job discovery
6. Finds PostgreSQL PVC → Dynamically
7. Scales down DB → Safely
8. Copies data → With validation and backup
9. Scales up DB → With pod creation waiting
10. Verifies DB → Tests tables and queries
11. Cleans up → Removes temporary resources
```

## How Has This Been Tested?
<!-- Have you tested this in a real application? Which scenarios were
tested? -->
No

## Breaking Changes
<!-- Will users need to update their code or configurations? -->
No (I assume no one relies on staging keeping its state)

## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update

## Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->
- [ ] I have read the [MCP
Documentation](https://modelcontextprotocol.io)
- [ ] My code follows the repository's style guidelines
- [ ] New and existing tests pass locally
- [ ] I have added appropriate error handling
- [ ] I have added or updated documentation as needed

## Additional context
<!-- Add any other context, implementation notes, or design decisions
-->

---------

Signed-off-by: Radoslav Dimitrov <radoslav@stacklok.com>
Co-authored-by: Adam Jones <adamj@anthropic.com>
This commit is contained in:
Radoslav Dimitrov 2025-10-09 14:01:39 +03:00 • committed by GitHub
commit 8de5283995
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 388 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

388
.github/workflows/sync-db.yml vendored Normal file
View file

@ -0,0 +1,388 @@
name: Sync Production DB to Staging (from backups)
on:
schedule:
# Run daily at 2 AM UTC (during low-traffic hours)
- cron: '0 2 * * *'
workflow_dispatch: # Allow manual triggering
permissions:
contents: read
jobs:
sync-database:
name: Sync Prod DB to Staging from k8up Backups
runs-on: ubuntu-latest
environment: staging
concurrency:
group: sync-staging-database
cancel-in-progress: false
steps:
- name: Authenticate to Google Cloud (Production)
uses: google-github-actions/auth@7c6bc770dae815cd3e89ee6cdf493a5fab2cc093
with:
credentials_json: ${{ secrets.GCP_PROD_SERVICE_ACCOUNT_KEY }}
- name: Setup Google Cloud SDK
uses: google-github-actions/setup-gcloud@aa5489c8933f4cc7a4f7d45035b3b1440c9c10db
with:
project_id: mcp-registry-prod
install_components: gke-gcloud-auth-plugin
- name: Get backup credentials from prod cluster
id: backup-creds
run: |
gcloud container clusters get-credentials mcp-registry-prod \
--zone=us-central1-b \
--project=mcp-registry-prod
# Store in outputs (GitHub Actions encrypts these automatically)
kubectl get secret k8up-backup-credentials -n default -o json | jq -r '
"access_key=" + (.data.AWS_ACCESS_KEY_ID | @base64d),
"secret_key=" + (.data.AWS_SECRET_ACCESS_KEY | @base64d)
' >> $GITHUB_OUTPUT
- name: Remove all production access (SAFETY MEASURE)
run: |
# Remove production cluster from kubeconfig
kubectl config delete-context gke_mcp-registry-prod_us-central1-b_mcp-registry-prod 2>/dev/null || true
# Revoke gcloud credentials
gcloud auth revoke --all 2>/dev/null || true
# Clear gcloud configuration
gcloud config unset project 2>/dev/null || true
gcloud config unset account 2>/dev/null || true
# Verify no contexts remain
CONTEXT_COUNT=$(kubectl config get-contexts -o name 2>/dev/null | wc -l)
if [ "$CONTEXT_COUNT" -gt 0 ]; then
echo "❌ ERROR: $CONTEXT_COUNT context(s) still exist after cleanup!"
kubectl config get-contexts
exit 1
fi
- name: Switch to staging cluster
uses: google-github-actions/auth@7c6bc770dae815cd3e89ee6cdf493a5fab2cc093
with:
credentials_json: ${{ secrets.GCP_STAGING_SERVICE_ACCOUNT_KEY }}
- name: Configure staging cluster access
run: |
gcloud config set project mcp-registry-staging
gcloud container clusters get-credentials mcp-registry-staging \
--zone=us-central1-b \
--project=mcp-registry-staging
- name: Create secret for prod backup bucket access
run: |
# Create/update secret in staging with access to prod backups
kubectl create secret generic prod-to-staging-sync-credentials \
--from-literal=AWS_ACCESS_KEY_ID="${{ steps.backup-creds.outputs.access_key }}" \
--from-literal=AWS_SECRET_ACCESS_KEY="${{ steps.backup-creds.outputs.secret_key }}" \
--dry-run=client -o yaml | kubectl apply -f -
- name: Create restore PVC
run: |
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: restore-data-pvc
namespace: default
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
EOF
- name: Trigger k8up restore from prod backups
id: restore
run: |
RESTORE_NAME="restore-from-prod-$(date +%Y%m%d-%H%M%S)"
echo "restore_name=$RESTORE_NAME" >> $GITHUB_OUTPUT
# Create a k8up Restore resource to restore from prod backups
kubectl apply -f - <<EOF
apiVersion: k8up.io/v1
kind: Restore
metadata:
name: $RESTORE_NAME
namespace: default
spec:
snapshot: latest
restoreMethod:
folder:
claimName: restore-data-pvc
backend:
repoPasswordSecretRef:
name: k8up-repo-password
key: password
s3:
bucket: mcp-registry-prod-backups
endpoint: https://storage.googleapis.com
accessKeyIDSecretRef:
name: prod-to-staging-sync-credentials
key: AWS_ACCESS_KEY_ID
secretAccessKeySecretRef:
name: prod-to-staging-sync-credentials
key: AWS_SECRET_ACCESS_KEY
EOF
- name: Wait for k8up restore to complete
run: |
RESTORE_NAME="${{ steps.restore.outputs.restore_name }}"
echo "Waiting for restore job to start..."
sleep 15
# Find the job created by k8up for this restore
for i in {1..30}; do
JOB_NAME=$(kubectl get jobs -n default -l k8up.io/owned-by=restore -o jsonpath='{.items[?(@.metadata.ownerReferences[0].name=="'$RESTORE_NAME'")].metadata.name}' 2>/dev/null)
if [ -n "$JOB_NAME" ]; then
echo "Found restore job: $JOB_NAME"
break
fi
echo "Waiting for job to be created... ($i/30)"
sleep 2
done
if [ -z "$JOB_NAME" ]; then
echo "ERROR: Restore job not found"
kubectl get restore $RESTORE_NAME -n default -o yaml
exit 1
fi
# Wait for the restore job to complete (max 15 minutes)
kubectl wait --for=condition=complete \
job/$JOB_NAME \
--timeout=900s -n default || {
echo "Restore job failed or timed out"
kubectl describe job/$JOB_NAME -n default
kubectl logs job/$JOB_NAME -n default --tail=100
exit 1
}
- name: Find staging PostgreSQL PVC
id: pgdata-pvc
run: |
# Find the PVC used by the PostgreSQL cluster
PVC_NAME=$(kubectl get pvc -n default -l cnpg.io/cluster=registry-pg -o jsonpath='{.items[0].metadata.name}')
if [ -z "$PVC_NAME" ]; then
echo "ERROR: Could not find PostgreSQL PVC"
kubectl get pvc -n default -l cnpg.io/cluster=registry-pg
exit 1
fi
echo "pvc_name=$PVC_NAME" >> $GITHUB_OUTPUT
- name: Scale down staging PostgreSQL
run: |
echo "Scaling down PostgreSQL cluster..."
kubectl patch cluster registry-pg -n default \
--type merge \
--patch '{"spec":{"instances":0}}'
# Wait for pods to terminate
echo "Waiting for pods to terminate..."
kubectl wait --for=delete pod -l cnpg.io/cluster=registry-pg -n default --timeout=300s || true
- name: Verify we are in staging cluster (SAFETY CHECK)
run: |
# Get current cluster context
CURRENT_CONTEXT=$(kubectl config current-context)
CURRENT_PROJECT=$(gcloud config get-value project)
echo "Current kubectl context: $CURRENT_CONTEXT"
echo "Current GCP project: $CURRENT_PROJECT"
# Verify we're in staging
if ! echo "$CURRENT_CONTEXT" | grep -qi "staging"; then
echo "❌ SAFETY CHECK FAILED: Not in staging cluster"
echo "Context: $CURRENT_CONTEXT"
echo "Expected: staging cluster"
exit 1
fi
if [ "$CURRENT_PROJECT" != "mcp-registry-staging" ]; then
echo "❌ SAFETY CHECK FAILED: Not in staging project"
echo "Project: $CURRENT_PROJECT"
echo "Expected: mcp-registry-staging"
exit 1
fi
- name: Replace staging database with restored backup
id: copy-job
run: |
JOB_NAME="copy-pgdata-$(date +%Y%m%d-%H%M%S)"
echo "job_name=$JOB_NAME" >> $GITHUB_OUTPUT
# Create a job to copy the restored backup data to the staging PVC
kubectl apply -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
name: $JOB_NAME
namespace: default
spec:
ttlSecondsAfterFinished: 600
template:
spec:
restartPolicy: Never
containers:
- name: copy-data
image: busybox:latest
command:
- /bin/sh
- -c
- |
set -e
echo "Finding PostgreSQL data in backup..."
echo "Restore structure:"
find /restore -maxdepth 3 -type d 2>/dev/null | head -20
# Try different possible paths for pgdata
PGDATA_SOURCE=""
for path in \$(find /restore -type d -name "pgdata" 2>/dev/null); do
if [ -f "\$path/PG_VERSION" ]; then
PGDATA_SOURCE="\$path"
break
fi
done
if [ -z "\$PGDATA_SOURCE" ]; then
echo "ERROR: Could not find valid pgdata directory with PG_VERSION"
echo "Searched paths:"
find /restore -type d -name "pgdata" 2>/dev/null
exit 1
fi
echo "Found pgdata at: \$PGDATA_SOURCE"
echo "Contents:"
ls -lah \$PGDATA_SOURCE/ | head -10
echo "Backing up existing staging data..."
mkdir -p /pgdata-backup
if [ "\$(ls -A /pgdata)" ]; then
cp -a /pgdata/. /pgdata-backup/ || echo "Warning: Could not backup existing data"
fi
echo "Clearing existing data..."
rm -rf /pgdata/*
echo "Copying backup data to staging PVC..."
cp -a \$PGDATA_SOURCE/. /pgdata/
echo "Setting correct permissions..."
chmod 700 /pgdata
ls -lah /pgdata/ | head -20
echo "PostgreSQL version: \$(cat /pgdata/PG_VERSION)"
volumeMounts:
- name: restore-data
mountPath: /restore
- name: staging-pgdata
mountPath: /pgdata
volumes:
- name: restore-data
persistentVolumeClaim:
claimName: restore-data-pvc
- name: staging-pgdata
persistentVolumeClaim:
claimName: ${{ steps.pgdata-pvc.outputs.pvc_name }}
EOF
- name: Wait for data copy to complete
run: |
JOB_NAME="${{ steps.copy-job.outputs.job_name }}"
# Wait for copy to complete
kubectl wait --for=condition=complete job/$JOB_NAME --timeout=600s -n default || {
echo "Data copy job failed"
kubectl describe job/$JOB_NAME -n default
kubectl logs job/$JOB_NAME -n default --tail=100
exit 1
}
- name: Scale up staging PostgreSQL
run: |
echo "Scaling up PostgreSQL cluster..."
kubectl patch cluster registry-pg -n default \
--type merge \
--patch '{"spec":{"instances":1}}'
# Wait for PostgreSQL pod to be created
echo "Waiting for PostgreSQL pod to be created..."
for i in {1..60}; do
POD_COUNT=$(kubectl get pods -l cnpg.io/cluster=registry-pg -n default --no-headers 2>/dev/null | wc -l)
if [ "$POD_COUNT" -gt 0 ]; then
echo "Pod created"
break
fi
echo "Waiting... ($i/60)"
sleep 2
done
# Wait for PostgreSQL to be ready
echo "Waiting for PostgreSQL to be ready..."
kubectl wait --for=condition=ready pod -l cnpg.io/cluster=registry-pg -n default --timeout=300s
- name: Verify staging DB is functional
run: |
# Create a verification pod
kubectl run pg-verify-$(date +%s) \
--image=postgres:15 \
--rm -i --restart=Never \
--env="PGPASSWORD=$(kubectl get secret registry-pg-superuser -n default -o jsonpath='{.data.password}' | base64 -d)" \
-- bash -c '
echo "Waiting for database to accept connections..."
for i in {1..30}; do
if pg_isready -h registry-pg-rw -U postgres 2>/dev/null; then
break
fi
echo "Waiting... ($i/30)"
sleep 2
done
echo "Querying database..."
TABLE_COUNT=$(psql -h registry-pg-rw -U postgres -d app -tAc "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = '\''public'\'';" 2>&1)
if [ $? -ne 0 ]; then
echo "ERROR: Could not query database"
echo "$TABLE_COUNT"
exit 1
fi
if [ "$TABLE_COUNT" -lt 1 ]; then
echo "ERROR: Staging DB has no tables!"
exit 1
fi
echo "Staging DB has $TABLE_COUNT tables"
echo "Top 10 tables by row count:"
psql -h registry-pg-rw -U postgres -d app \
-c "SELECT schemaname, tablename, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC LIMIT 10;" || true
'
- name: Cleanup
if: always()
run: |
# Clean up jobs first
if [ -n "${{ steps.copy-job.outputs.job_name }}" ]; then
kubectl delete job ${{ steps.copy-job.outputs.job_name }} -n default || true
fi
# Remove restore PVC (will wait for jobs to finish)
kubectl delete pvc restore-data-pvc -n default || true
# Remove prod backup credentials (for security)
kubectl delete secret prod-to-staging-sync-credentials -n default || true
# Clean up old restore resources (keep last 3)
kubectl get restore -n default --sort-by=.metadata.creationTimestamp -o name | head -n -3 | xargs -r kubectl delete || true
# Clean up old copy jobs (keep last 3)
kubectl get jobs -n default --sort-by=.metadata.creationTimestamp -o name | grep 'copy-pgdata-' | head -n -3 | xargs -r kubectl delete -n default || true