89cb9de125* Repair corrupted GroupMember state so deploys stop failing (#133) Every Deploy to Production run since 2026-06-30 aborts about a minute in with "googleapi: Error 409: Member already exists" and merged changes (e.g. #134, #137) never apply. Two external-email GroupMember records in the maintainers group are recorded in state with an empty member ID ("groups/04f1mdlm38smb30/members/"), so each pulumi up plans a create-before-delete replace whose create conflicts with the live membership and fails the whole update. Five other records corrupted at the same time were already healed by the 2026-07-02 deploy run. Add a one-time TEMP block to the deploy workflow (same precedent as the auth-maintainers repair and the experimental-ext-tasks state delete): drop the two broken records, then re-import each membership by member email so the provider records the numeric member ID. All commands are || true-guarded, and the block must be removed after one successful deploy. Fixes #133 Co-authored-by: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NazA3QKftRn72tdPG5vrpb * Widen state repair to all seven corrupted GroupMember records (#133) * Replace per-URN state deletes with export surgery; verify healing post-deploy (#133) The committed repair used 'pulumi state delete <urn>' per record, but the failed create-before-delete replaces left duplicate same-URN entries (a live record plus a pending-delete leftover) for some of the seven corrupted GroupMember records. 'pulumi state delete' fails on an ambiguous URN, and the '|| true' guard hid that failure, making the repair a silent no-op. - Replace the per-URN deletes with state surgery: export the stack, drop every entry (including pending-delete duplicates) whose URN is one of the seven with jq, and re-import. Unguarded, so a surgery failure is loud; skips the import when nothing matches (idempotent). - Keep the seven imports-by-email ('|| true' so a not-currently-live membership falls back to a create instead of blocking the deploy). - Add a post-deploy verification step that fails if any GroupMember record still has an empty member ID or a duplicate URN. The pinned provider (SamuZad/googleworkspace 0.11.1) writes an empty member ID even on successful creates (fixed upstream in 0.11.2), so a green 'make up' alone does not prove healing; the TEMP block must only be removed after this verification step passes. - Correct the root-cause comment: Google-side membership removals -> refresh dropped the records -> re-create -> provider create bug wrote empty IDs. * Bump googleworkspace provider 0.11.1 -> 0.11.2 (#133) 0.11.2 carries upstream commit "handle edge case with group memberships" (SamuZad/terraform-provider-googleworkspace, 2026-03-12), which fixes the create-path bug that stores an empty member ID for external-email group members — the bug that corrupted the seven records repaired by the TEMP block in deploy.yml and that would re-corrupt any record restored via the fallback create path. CI regenerates the SDK from this pin via 'pulumi install' and typechecks against it. --------- Co-authored-by: Claude <noreply@anthropic.com>
186 lines
10 KiB
YAML
186 lines
10 KiB
YAML
name: Deploy
|
|
|
|
on:
|
|
push:
|
|
branches:
|
|
- main
|
|
workflow_dispatch:
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
env:
|
|
PULUMI_VERSION: "3.197.0"
|
|
|
|
jobs:
|
|
deploy-production:
|
|
name: Deploy to Production
|
|
runs-on: ubuntu-latest
|
|
environment: production
|
|
concurrency:
|
|
group: deploy-production
|
|
cancel-in-progress: false
|
|
steps:
|
|
- name: Checkout code
|
|
uses: actions/checkout@v4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@v4
|
|
with:
|
|
node-version: '22'
|
|
cache: 'npm'
|
|
|
|
- name: Setup Pulumi
|
|
uses: pulumi/actions@v6
|
|
with:
|
|
pulumi-version: ${{ env.PULUMI_VERSION }}
|
|
|
|
- name: Cache Pulumi plugins
|
|
uses: actions/cache@v4
|
|
with:
|
|
path: ~/.pulumi/plugins
|
|
key: pulumi-plugins-${{ hashFiles('Pulumi.yaml') }}
|
|
restore-keys: |
|
|
pulumi-plugins-
|
|
|
|
- name: Install Pulumi packages
|
|
env:
|
|
GITHUB_TOKEN: ${{ github.token }}
|
|
run: pulumi install
|
|
|
|
- name: Install dependencies
|
|
run: npm ci
|
|
|
|
- name: Build
|
|
run: npm run build
|
|
|
|
- name: Authenticate to Google Cloud
|
|
uses: google-github-actions/auth@v2
|
|
with:
|
|
credentials_json: ${{ secrets.GCP_PROD_SERVICE_ACCOUNT_KEY }}
|
|
|
|
- name: Deploy to Production
|
|
env:
|
|
PULUMI_PASSPHRASE: ${{ secrets.PULUMI_PROD_PASSPHRASE }}
|
|
GITHUB_TOKEN: ${{ secrets.PULUMI_GITHUB_TOKEN }}
|
|
ORG_BILLING_EMAIL: ${{ secrets.ORG_BILLING_EMAIL }}
|
|
DISCORD_BOT_TOKEN: ${{ secrets.DISCORD_BOT_TOKEN }}
|
|
DISCORD_GUILD_ID: ${{ secrets.DISCORD_GUILD_ID }}
|
|
run: |
|
|
echo "$PULUMI_PASSPHRASE" > passphrase.prod.txt
|
|
export PULUMI_CONFIG_PASSPHRASE_FILE=passphrase.prod.txt
|
|
pulumi login gs://mcp-access-prod-pulumi-state
|
|
# TEMP: drop stale state for renamed repo (experimental-ext-tasks -> ext-tasks, #125).
|
|
# Delete-on-up 404s because the old repo name is gone. Remove after one successful deploy.
|
|
pulumi state delete 'urn:pulumi:prod::mcp-access::github:index/repositoryCollaborators:RepositoryCollaborators::repo-experimental-ext-tasks' --stack prod --yes || true
|
|
pulumi config set discord:guildId "$DISCORD_GUILD_ID" --stack prod
|
|
pulumi config set discord:botToken "$DISCORD_BOT_TOKEN" --secret --stack prod
|
|
pulumi config set githubBillingEmail "$ORG_BILLING_EMAIL" --secret --stack prod
|
|
# TEMP (#133): one-time state surgery for seven corrupted GroupMember
|
|
# records. Root cause chain: between 2026-06-30 and 2026-07-02 the seven
|
|
# external-email memberships below were removed on the Google side
|
|
# (actor unknown); the deploy's refresh then dropped their records from
|
|
# state; the subsequent up re-created them; and the pinned provider
|
|
# (SamuZad/googleworkspace 0.11.1) has a create-path bug that stores an
|
|
# empty member ID ("groups/<groupId>/members/") even on successful
|
|
# creates (fixed upstream in 0.11.2). The broken ID is immutable, so
|
|
# every deploy since plans a create-before-delete replace whose create
|
|
# step 409s ("Member already exists") against the live membership and
|
|
# aborts the whole update. The failed replaces also left DUPLICATE
|
|
# same-URN entries in state (a live record plus a pending-delete
|
|
# '"delete": true' leftover) for some of the seven, and
|
|
# `pulumi state delete <urn>` FAILS on an ambiguous URN — which is why
|
|
# this block edits an exported copy of the state instead of deleting
|
|
# per URN.
|
|
#
|
|
# Step 1: export the state, drop EVERY entry (including pending-delete
|
|
# duplicates) whose URN is one of the seven, and re-import. These
|
|
# commands are deliberately NOT `|| true`-guarded: if the surgery
|
|
# fails, we want a loud failure, not a silent no-op. If nothing
|
|
# matches (state already repaired), the import is skipped and the
|
|
# deploy proceeds — the block is idempotent.
|
|
GROUPMEMBER_URNS='[
|
|
"urn:pulumi:prod::mcp-access::googleworkspace:index/groupMember:GroupMember::adamj@anthropic.com-catch-all",
|
|
"urn:pulumi:prod::mcp-access::googleworkspace:index/groupMember:GroupMember::bob.dickinson@gmail.com-maintainers",
|
|
"urn:pulumi:prod::mcp-access::googleworkspace:index/groupMember:GroupMember::bob.dickinson@gmail.com-registry-wg",
|
|
"urn:pulumi:prod::mcp-access::googleworkspace:index/groupMember:GroupMember::chugh.tapan@gmail.com-maintainers",
|
|
"urn:pulumi:prod::mcp-access::googleworkspace:index/groupMember:GroupMember::davideramian@anthropic.com-antitrust",
|
|
"urn:pulumi:prod::mcp-access::googleworkspace:index/groupMember:GroupMember::davidsp@anthropic.com-antitrust",
|
|
"urn:pulumi:prod::mcp-access::googleworkspace:index/groupMember:GroupMember::mattsamuels@anthropic.com-antitrust"
|
|
]'
|
|
pulumi stack export --stack prod --file /tmp/state.json
|
|
jq --argjson urns "$GROUPMEMBER_URNS" \
|
|
'.deployment.resources |= map(select(.urn as $u | ($urns | index($u)) | not))' \
|
|
/tmp/state.json > /tmp/state-repaired.json
|
|
before=$(jq '.deployment.resources | length' /tmp/state.json)
|
|
after=$(jq '.deployment.resources | length' /tmp/state-repaired.json)
|
|
echo "State surgery: removing $((before - after)) corrupted GroupMember entries"
|
|
if [ "$before" -eq "$after" ]; then
|
|
echo "No matching entries in state (already repaired); skipping import"
|
|
else
|
|
pulumi stack import --stack prod --file /tmp/state-repaired.json
|
|
fi
|
|
# Step 2: re-import the live memberships by member email (the
|
|
# Directory API accepts an email as the member key, and the provider
|
|
# then stores the real member ID). `|| true` is deliberate here: if a
|
|
# membership is not currently live, its import fails harmlessly and
|
|
# the subsequent up creates it, restoring access. Note that on the
|
|
# 0.11.1 provider such a fallback create re-corrupts the record's
|
|
# member ID — the "Verify GroupMember state healed" step after the
|
|
# deploy catches that instead of letting a green run masquerade as
|
|
# healed. --protect=false keeps the imported records deletable, like
|
|
# every other GroupMember.
|
|
#
|
|
# REMOVAL CRITERION: remove this block (and the verification step
|
|
# below) only after the "Verify GroupMember state healed" step has
|
|
# passed on a deploy — a green `make up` alone does NOT prove the
|
|
# records are healed.
|
|
pulumi import googleworkspace:index/groupMember:GroupMember 'adamj@anthropic.com-catch-all' 'groups/00upglbi31qapnv/members/adamj@anthropic.com' --stack prod --yes --protect=false || true
|
|
pulumi import googleworkspace:index/groupMember:GroupMember 'bob.dickinson@gmail.com-maintainers' 'groups/04f1mdlm38smb30/members/bob.dickinson@gmail.com' --stack prod --yes --protect=false || true
|
|
pulumi import googleworkspace:index/groupMember:GroupMember 'bob.dickinson@gmail.com-registry-wg' 'groups/040ew0vw3g472qb/members/bob.dickinson@gmail.com' --stack prod --yes --protect=false || true
|
|
pulumi import googleworkspace:index/groupMember:GroupMember 'chugh.tapan@gmail.com-maintainers' 'groups/04f1mdlm38smb30/members/chugh.tapan@gmail.com' --stack prod --yes --protect=false || true
|
|
pulumi import googleworkspace:index/groupMember:GroupMember 'davideramian@anthropic.com-antitrust' 'groups/03jtnz0s4hi6gld/members/davideramian@anthropic.com' --stack prod --yes --protect=false || true
|
|
pulumi import googleworkspace:index/groupMember:GroupMember 'davidsp@anthropic.com-antitrust' 'groups/03jtnz0s4hi6gld/members/davidsp@anthropic.com' --stack prod --yes --protect=false || true
|
|
pulumi import googleworkspace:index/groupMember:GroupMember 'mattsamuels@anthropic.com-antitrust' 'groups/03jtnz0s4hi6gld/members/mattsamuels@anthropic.com' --stack prod --yes --protect=false || true
|
|
make up
|
|
|
|
# TEMP (#133): a green deploy alone does NOT prove the GroupMember records
|
|
# are healed — the pinned provider (0.11.1) writes an empty member ID even
|
|
# on successful creates, so the fallback create path in the repair block
|
|
# above can silently re-corrupt state while the run stays green. This step
|
|
# makes that visible: it fails loudly if any GroupMember record still has
|
|
# an empty member ID or a duplicate URN. Remove the TEMP repair block in
|
|
# the deploy step, and this step, only after this step has PASSED on a
|
|
# deploy — not merely after a green run.
|
|
- name: "Verify GroupMember state healed (TEMP, #133)"
|
|
env:
|
|
PULUMI_PASSPHRASE: ${{ secrets.PULUMI_PROD_PASSPHRASE }}
|
|
run: |
|
|
echo "$PULUMI_PASSPHRASE" > passphrase.prod.txt
|
|
export PULUMI_CONFIG_PASSPHRASE_FILE=passphrase.prod.txt
|
|
pulumi login gs://mcp-access-prod-pulumi-state
|
|
pulumi stack export --stack prod --file /tmp/state-verify.json
|
|
empty_ids=$(jq -r '[.deployment.resources[]
|
|
| select(.type == "googleworkspace:index/groupMember:GroupMember")
|
|
| select((.id // "") | endswith("/members/"))
|
|
| .urn] | .[]' /tmp/state-verify.json)
|
|
dup_urns=$(jq -r '[.deployment.resources[]
|
|
| select(.type == "googleworkspace:index/groupMember:GroupMember")
|
|
| .urn] | group_by(.) | map(select(length > 1) | .[0]) | .[]' /tmp/state-verify.json)
|
|
status=0
|
|
if [ -n "$empty_ids" ]; then
|
|
echo "::error::GroupMember records still have an empty member ID (state NOT healed):"
|
|
echo "$empty_ids"
|
|
status=1
|
|
fi
|
|
if [ -n "$dup_urns" ]; then
|
|
echo "::error::Duplicate GroupMember URNs remain in state (pending-delete leftovers):"
|
|
echo "$dup_urns"
|
|
status=1
|
|
fi
|
|
if [ "$status" -ne 0 ]; then
|
|
echo "::error::Deploy succeeded but the state repair did NOT stick. Do not remove the TEMP repair block; see #133."
|
|
exit 1
|
|
fi
|
|
echo "All GroupMember records have real member IDs and unique URNs; state is healed."
|
|
echo "The TEMP repair block (and this step) can now be removed."
|