offline / test (push) Waiting to run

Details

Add Proxmox worker CT, off-box watch, backup, and tagged deploy.

NS1 is the Proxmox host. verae-proxmox creates LXC 510 (verae-px-worker
10.10.10.20 on vmbr1) with a private NATS proxy on 10.10.10.1:4222.
verae-uptime GET-watches public doors; verae-backup snapshots SQLite and
worm/tree data; verae-deploy does host-deps + checkout + npm ci.
Fleet overlays/ns1 are checked in (start.sh no longer rewrites JSON).
User systemd + linger for keep and fleet survive reboot.

2026-09-11 23:35:44 -04:00

2.8 KiB

Raw Blame History

7. Maintenance

Health

Check	Command
Edges	`curl -fsS https://zappier.example.com/health` and middleware `/health`
NATS	`curl -fsS http://127.0.0.1:8222/healthz` on each nats node (localhost)
Fleet	`node src/cli.js status` or http://127.0.0.1:3850/ (loopback)
Tree floor	status: tree-node available ≥ min, none of those paused

Green / yellow / red on the operator console: https://zapier.georgelambert.org/packages/verae-fleet/docs/CONSOLE.pdf

Restart

User systemd (survives reboot with linger):
- bash packages/verae-keep/scripts/install-systemd.sh
- bash packages/verae-fleet/scripts/install-systemd.sh
- loginctl enable-linger $USER
- systemctl --user restart verae-keep-guard verae-fleet
Compose: docker compose up -d --no-deps middleware
One replica: fleet restart tree-node-1 (floor spawns a replacement if needed)
Pause is not off — paused copies do not count toward keepFloor
Do not run both nohup guard.sh and the systemd unit

Backup

Repo: verae-backup

export BACKUP_ROOT=/SSD2/backups/verae
bash scripts/backup.sh
bash scripts/restore.sh --dry-run

Data	Default path
zappier SQLite	`~/verae-stack/packages/zappier/zappier.db`
worm / tree blobs	`~/verae-fleet-runtime/data/`
keep state	`~/verae-keep/data`
NATS JetStream	`/var/lib/nats/jetstream` if present
IAM	`~/verae-stack/packages/verae-staff-iam/data`

User timer: verae-backup.timer (03:17 UTC). Do not back up private keys into git. Restore identityFile paths on the operator host. After --apply, curl /health on edge and keep.

Upgrade

Repo: verae-deploy

export VERAE_SRC=$HOME/verae-src
export DEPLOY_TYPE=ns1-all-in-one
bash scripts/deploy.sh main          # or a tag vYYYY-MM-DD
systemctl --user restart verae-keep-guard verae-fleet

host-deps.sh on this OS.
git fetch + checkout ref (not force-push).
npm ci / npm rebuild (never copy macOS node_modules).
Restart that unit only; NATS stays up.
Zapier Platform app: zapier-platform push from verae-zapier-app when HTTPS contracts change.

Off-box watch

Repo: verae-uptime

node src/watch.js --once     # exit 2 if a public door failed

Run this on a different machine than NS1 (laptop cron is enough). /status on :3870 in loop mode.

Logs

journald for systemd; docker compose logs -f middleware nats1. Redact API keys (middleware debug redaction).

Docs

After doc edits: from the monorepo npm run docs:site && npm run docs:deploy so https://zapier.georgelambert.org/ stays current (PDF default, Markdown indexes at /index-md.html).

2.8 KiB Raw Blame History