Fix sync-db workflow: k8up restore job detection and timing issues (#641)
<!-- Provide a brief summary of your changes --> ## Motivation and Context <!-- Why is this change needed? What problem does it solve? --> The following PR addresses the failing sync job - https://github.com/modelcontextprotocol/registry/actions/runs/18374711984/job/52345853050 **Root Cause** 1. Job detection failure: The workflow was looking for a job with a specific labeling pattern (k8up.io/owned-by=restore), but k8up creates jobs with a naming convention `restore-<restore-name>` 2. Race condition: The cleanup step (with if: always()) was deleting the prod-to-staging-sync-credentials secret even when the restore job hadn't been created yet or was still running **The Fix** 1. Updated job detection logic: Now searches for jobs by name pattern restore-$RESTORE_NAME instead of relying on labels 2. Added better diagnostics: If the job isn't found, the workflow now shows: - Restore resource status - Any existing restore jobs - k8up operator logs 3. Fixed cleanup race condition: The cleanup step now waits up to 2 minutes for any running restore jobs to complete before deleting credentials 4. Improved error handling: Better logging when restore jobs fail, including checking if credentials still exist **The changes ensure:** - The workflow correctly finds the k8up restore job - Credentials aren't deleted while jobs are still running - Better visibility into failures for easier debugging ## How Has This Been Tested? <!-- Have you tested this in a real application? Which scenarios were tested? --> ## Breaking Changes <!-- Will users need to update their code or configurations? --> ## Types of changes <!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: --> - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update ## Checklist <!-- Go over all the following points, and put an `x` in all the boxes that apply. --> - [ ] I have read the [MCP Documentation](https://modelcontextprotocol.io) - [ ] My code follows the repository's style guidelines - [ ] New and existing tests pass locally - [ ] I have added appropriate error handling - [ ] I have added or updated documentation as needed ## Additional context <!-- Add any other context, implementation notes, or design decisions --> --------- Signed-off-by: Radoslav Dimitrov <radoslav@stacklok.com>
This commit is contained in:
Radoslav Dimitrov
GitHub
parent
036c935733
commit
5055c79531
1 changed files with 53 additions and 6 deletions
59
.github/workflows/sync-db.yml
vendored
59
.github/workflows/sync-db.yml
vendored
|
|
@ -139,29 +139,48 @@ jobs:
|
|||
sleep 15
|
||||
|
||||
# Find the job created by k8up for this restore
|
||||
for i in {1..30}; do
|
||||
JOB_NAME=$(kubectl get jobs -n default -l k8up.io/owned-by=restore -o jsonpath='{.items[?(@.metadata.ownerReferences[0].name=="'$RESTORE_NAME'")].metadata.name}' 2>/dev/null)
|
||||
# k8up creates jobs with name pattern "restore-<restore-name>"
|
||||
# Since our restore is named "restore-from-prod-*", the job will be "restore-restore-from-prod-*"
|
||||
for i in {1..60}; do
|
||||
JOB_NAME=$(kubectl get jobs -n default --no-headers 2>/dev/null | grep "^restore-$RESTORE_NAME" | awk '{print $1}' | head -1)
|
||||
if [ -n "$JOB_NAME" ]; then
|
||||
echo "Found restore job: $JOB_NAME"
|
||||
break
|
||||
fi
|
||||
echo "Waiting for job to be created... ($i/30)"
|
||||
echo "Waiting for job to be created... ($i/60)"
|
||||
sleep 2
|
||||
done
|
||||
|
||||
if [ -z "$JOB_NAME" ]; then
|
||||
echo "ERROR: Restore job not found"
|
||||
kubectl get restore $RESTORE_NAME -n default -o yaml
|
||||
echo "ERROR: Restore job not found after 120 seconds"
|
||||
echo "Checking restore resource status:"
|
||||
kubectl get restore $RESTORE_NAME -n default
|
||||
kubectl describe restore $RESTORE_NAME -n default
|
||||
|
||||
echo "Checking for any restore jobs:"
|
||||
kubectl get jobs -n default | grep restore || echo "No restore jobs found"
|
||||
|
||||
echo "Checking k8up operator logs:"
|
||||
kubectl logs deployment/k8up -n default --tail=50 | grep -i restore || echo "No restore logs found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Wait for the restore job to complete (max 15 minutes)
|
||||
echo "Waiting for restore job to complete..."
|
||||
kubectl wait --for=condition=complete \
|
||||
job/$JOB_NAME \
|
||||
--timeout=900s -n default || {
|
||||
echo "Restore job failed or timed out"
|
||||
echo "Job status:"
|
||||
kubectl get job/$JOB_NAME -n default
|
||||
echo "Job details:"
|
||||
kubectl describe job/$JOB_NAME -n default
|
||||
kubectl logs job/$JOB_NAME -n default --tail=100
|
||||
echo "Job logs (if available):"
|
||||
kubectl logs job/$JOB_NAME -n default --tail=100 || echo "No logs available"
|
||||
|
||||
# Check if it's a credential issue
|
||||
echo "Checking if credentials exist:"
|
||||
kubectl get secret prod-to-staging-sync-credentials -n default || echo "Credentials secret missing!"
|
||||
exit 1
|
||||
}
|
||||
|
||||
|
|
@ -370,11 +389,36 @@ jobs:
|
|||
- name: Cleanup
|
||||
if: always()
|
||||
run: |
|
||||
# Wait for any restore jobs to finish before cleanup (max 5 minutes for large database restores)
|
||||
echo "Checking for running restore jobs..."
|
||||
RESTORE_NAME="${{ steps.restore.outputs.restore_name }}"
|
||||
|
||||
for i in {1..60}; do
|
||||
# Check specifically for our restore job in Running or Pending state
|
||||
RUNNING_JOBS=$(kubectl get jobs -n default --no-headers 2>/dev/null | grep "^restore-${RESTORE_NAME}" | grep -E "Running|Pending" | wc -l)
|
||||
if [ "$RUNNING_JOBS" -eq 0 ]; then
|
||||
echo "No running restore jobs found"
|
||||
break
|
||||
fi
|
||||
echo "Waiting for $RUNNING_JOBS restore job(s) to finish... ($i/60)"
|
||||
sleep 5
|
||||
done
|
||||
|
||||
# Clean up jobs first
|
||||
if [ -n "${{ steps.copy-job.outputs.job_name }}" ]; then
|
||||
kubectl delete job ${{ steps.copy-job.outputs.job_name }} -n default || true
|
||||
fi
|
||||
|
||||
# Clean up the restore job if it exists (with proper error handling)
|
||||
if [ -n "$RESTORE_NAME" ]; then
|
||||
if kubectl get job "restore-$RESTORE_NAME" -n default >/dev/null 2>&1; then
|
||||
echo "Deleting restore job: restore-$RESTORE_NAME"
|
||||
kubectl delete job "restore-$RESTORE_NAME" -n default || echo "Failed to delete job, may have already been cleaned up"
|
||||
else
|
||||
echo "Restore job restore-$RESTORE_NAME not found, skipping deletion"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Remove restore PVC (will wait for jobs to finish)
|
||||
kubectl delete pvc restore-data-pvc -n default || true
|
||||
|
||||
|
|
@ -384,5 +428,8 @@ jobs:
|
|||
# Clean up old restore resources (keep last 3)
|
||||
kubectl get restore -n default --sort-by=.metadata.creationTimestamp -o name | head -n -3 | xargs -r kubectl delete || true
|
||||
|
||||
# Clean up old restore jobs (keep last 3)
|
||||
kubectl get jobs -n default --sort-by=.metadata.creationTimestamp -o name | grep '^job.batch/restore-' | head -n -3 | xargs -r kubectl delete -n default || true
|
||||
|
||||
# Clean up old copy jobs (keep last 3)
|
||||
kubectl get jobs -n default --sort-by=.metadata.creationTimestamp -o name | grep 'copy-pgdata-' | head -n -3 | xargs -r kubectl delete -n default || true
|
||||
|
|
|
|||
Loading…
Reference in a new issue