George Lambert
7a5e25639e
Some checks are pending
offline / test (push) Waiting to run
NS1 is the Proxmox host. verae-proxmox creates LXC 510 (verae-px-worker 10.10.10.20 on vmbr1) with a private NATS proxy on 10.10.10.1:4222. verae-uptime GET-watches public doors; verae-backup snapshots SQLite and worm/tree data; verae-deploy does host-deps + checkout + npm ci. Fleet overlays/ns1 are checked in (start.sh no longer rewrites JSON). User systemd + linger for keep and fleet survive reboot.
79 lines
2.8 KiB
Markdown
79 lines
2.8 KiB
Markdown
# 7. Maintenance
|
|
|
|
## Health
|
|
|
|
| Check | Command |
|
|
|-------|---------|
|
|
| Edges | `curl -fsS https://zappier.example.com/health` and middleware `/health` |
|
|
| NATS | `curl -fsS http://127.0.0.1:8222/healthz` on each nats node (localhost) |
|
|
| Fleet | `node src/cli.js status` or http://127.0.0.1:3850/ (loopback) |
|
|
| Tree floor | status: tree-node available ≥ min, none of those paused |
|
|
|
|
Green / yellow / red on the operator console: https://zapier.georgelambert.org/packages/verae-fleet/docs/CONSOLE.pdf
|
|
|
|
## Restart
|
|
|
|
- User systemd (survives reboot with linger):
|
|
- `bash packages/verae-keep/scripts/install-systemd.sh`
|
|
- `bash packages/verae-fleet/scripts/install-systemd.sh`
|
|
- `loginctl enable-linger $USER`
|
|
- `systemctl --user restart verae-keep-guard verae-fleet`
|
|
- Compose: `docker compose up -d --no-deps middleware`
|
|
- One replica: fleet `restart tree-node-1` (floor spawns a replacement if needed)
|
|
- Pause is **not** off — paused copies do not count toward `keepFloor`
|
|
- Do not run **both** `nohup guard.sh` and the systemd unit
|
|
|
|
## Backup
|
|
|
|
Repo: [verae-backup](https://git.georgelambert.org/marchon/verae-backup)
|
|
|
|
```bash
|
|
export BACKUP_ROOT=/SSD2/backups/verae
|
|
bash scripts/backup.sh
|
|
bash scripts/restore.sh --dry-run
|
|
```
|
|
|
|
| Data | Default path |
|
|
|------|----------------|
|
|
| zappier SQLite | `~/verae-stack/packages/zappier/zappier.db` |
|
|
| worm / tree blobs | `~/verae-fleet-runtime/data/` |
|
|
| keep state | `~/verae-keep/data` |
|
|
| NATS JetStream | `/var/lib/nats/jetstream` if present |
|
|
| IAM | `~/verae-stack/packages/verae-staff-iam/data` |
|
|
|
|
User timer: `verae-backup.timer` (03:17 UTC). Do not back up private keys into git. Restore `identityFile` paths on the operator host. After `--apply`, `curl` `/health` on edge and keep.
|
|
|
|
## Upgrade
|
|
|
|
Repo: [verae-deploy](https://git.georgelambert.org/marchon/verae-deploy)
|
|
|
|
```bash
|
|
export VERAE_SRC=$HOME/verae-src
|
|
export DEPLOY_TYPE=ns1-all-in-one
|
|
bash scripts/deploy.sh main # or a tag vYYYY-MM-DD
|
|
systemctl --user restart verae-keep-guard verae-fleet
|
|
```
|
|
|
|
1. `host-deps.sh` on **this** OS.
|
|
2. `git fetch` + checkout ref (not force-push).
|
|
3. `npm ci` / `npm rebuild` (never copy macOS `node_modules`).
|
|
4. Restart that unit only; NATS stays up.
|
|
5. Zapier Platform app: `zapier-platform push` from `verae-zapier-app` when HTTPS contracts change.
|
|
|
|
## Off-box watch
|
|
|
|
Repo: [verae-uptime](https://git.georgelambert.org/marchon/verae-uptime)
|
|
|
|
```bash
|
|
node src/watch.js --once # exit 2 if a public door failed
|
|
```
|
|
|
|
Run this on a **different** machine than NS1 (laptop cron is enough). `/status` on `:3870` in loop mode.
|
|
|
|
## Logs
|
|
|
|
journald for systemd; `docker compose logs -f middleware nats1`. Redact API keys (middleware debug redaction).
|
|
|
|
## Docs
|
|
|
|
After doc edits: from the monorepo `npm run docs:site && npm run docs:deploy` so https://zapier.georgelambert.org/ stays current (PDF default, Markdown indexes at `/index-md.html`).
|