Test env NATS cluster cut-over

07-maintenance.md

@@ -0,0 +1,79 @@
+# 7. Maintenance
+
+## Health
+
+| Check | Command |
+|-------|---------|
+| Edges | `curl -fsS https://zappier.example.com/health` and middleware `/health` |
+| NATS | `curl -fsS http://127.0.0.1:8222/healthz` on each nats node (localhost) |
+| Fleet | `node src/cli.js status` or http://127.0.0.1:3850/ (loopback) |
+| Tree floor | status: tree-node available ≥ min, none of those paused |
+
+Green / yellow / red on the operator console: https://zapier.georgelambert.org/packages/verae-fleet/docs/CONSOLE.pdf
+
+## Restart
+
+- User systemd (survives reboot with linger):
+  - `bash packages/verae-keep/scripts/install-systemd.sh`
+  - `bash packages/verae-fleet/scripts/install-systemd.sh`
+  - `loginctl enable-linger $USER`
+  - `systemctl --user restart verae-keep-guard verae-fleet`
+- Compose: `docker compose up -d --no-deps middleware`
+- One replica: fleet `restart tree-node-1` (floor spawns a replacement if needed)
+- Pause is **not** off — paused copies do not count toward `keepFloor`
+- Do not run **both** `nohup guard.sh` and the systemd unit
+
+## Backup
+
+Repo: [verae-backup](https://git.georgelambert.org/marchon/verae-backup)
+
+```bash
+export BACKUP_ROOT=/SSD2/backups/verae
+bash scripts/backup.sh
+bash scripts/restore.sh --dry-run
+```
+
+| Data | Default path |
+|------|----------------|
+| zappier SQLite | `~/verae-stack/packages/zappier/zappier.db` |
+| worm / tree blobs | `~/verae-fleet-runtime/data/` |
+| keep state | `~/verae-keep/data` |
+| NATS JetStream | `/var/lib/nats/jetstream` if present |
+| IAM | `~/verae-stack/packages/verae-staff-iam/data` |
+
+User timer: `verae-backup.timer` (03:17 UTC). Do not back up private keys into git. Restore `identityFile` paths on the operator host. After `--apply`, `curl` `/health` on edge and keep.
+
+## Upgrade
+
+Repo: [verae-deploy](https://git.georgelambert.org/marchon/verae-deploy)
+
+```bash
+export VERAE_SRC=$HOME/verae-src
+export DEPLOY_TYPE=ns1-all-in-one
+bash scripts/deploy.sh main          # or a tag vYYYY-MM-DD
+systemctl --user restart verae-keep-guard verae-fleet
+```
+
+1. `host-deps.sh` on **this** OS.
+2. `git fetch` + checkout ref (not force-push).
+3. `npm ci` / `npm rebuild` (never copy macOS `node_modules`).
+4. Restart that unit only; NATS stays up.
+5. Zapier Platform app: `zapier-platform push` from `verae-zapier-app` when HTTPS contracts change.
+
+## Off-box watch
+
+Repo: [verae-uptime](https://git.georgelambert.org/marchon/verae-uptime)
+
+```bash
+node src/watch.js --once     # exit 2 if a public door failed
+```
+
+Run this on a **different** machine than NS1 (laptop cron is enough). `/status` on `:3870` in loop mode.
+
+## Logs
+
+journald for systemd; `docker compose logs -f middleware nats1`. Redact API keys (middleware debug redaction).
+
+## Docs
+
+After doc edits: from the monorepo `npm run docs:site && npm run docs:deploy` so https://zapier.georgelambert.org/ stays current (PDF default, Markdown indexes at `/index-md.html`).