Add Proxmox worker CT, off-box watch, backup, and tagged deploy.

NS1 is the Proxmox host. verae-proxmox creates LXC 510 (verae-px-worker
10.10.10.20 on vmbr1) with a private NATS proxy on 10.10.10.1:4222.
verae-uptime GET-watches public doors; verae-backup snapshots SQLite and
worm/tree data; verae-deploy does host-deps + checkout + npm ci.
Fleet overlays/ns1 are checked in (start.sh no longer rewrites JSON).
User systemd + linger for keep and fleet survive reboot.

packages/docs-master/README.md

@@ -32,6 +32,10 @@ Summaries, NATS contracts, and message flows for every Verae Time × Zapier modu
 | verae-staff-iam | https://git.georgelambert.org/marchon/verae-staff-iam |
 | verae-keep | https://git.georgelambert.org/marchon/verae-keep |
 | verae-bootstrap | https://git.georgelambert.org/marchon/verae-bootstrap |
+| verae-uptime | https://git.georgelambert.org/marchon/verae-uptime |
+| verae-backup | https://git.georgelambert.org/marchon/verae-backup |
+| verae-deploy | https://git.georgelambert.org/marchon/verae-deploy |
+| verae-proxmox | https://git.georgelambert.org/marchon/verae-proxmox |
 
 Clone (SSH port 2223):
 

packages/docs-master/modules/verae-backup/NATS.md

@@ -0,0 +1,3 @@
+# NATS — verae-backup
+
+Copies JetStream store directory if present. No client connection.

packages/docs-master/modules/verae-backup/SUMMARY.md

@@ -0,0 +1,3 @@
+# verae-backup
+
+Nightly tar of SQLite, IAM, keep, worm/tree runtime, optional JetStream dir. Restore is dry-run unless `--apply`.

packages/docs-master/modules/verae-deploy/NATS.md

@@ -0,0 +1,3 @@
+# NATS — verae-deploy
+
+None during deploy. Archive hosts still need nats-server from host-deps.

packages/docs-master/modules/verae-deploy/SUMMARY.md

@@ -0,0 +1,3 @@
+# verae-deploy
+
+`host-deps` + git checkout tag/branch + npm ci/rebuild on the target OS. Restart keep/fleet after.

packages/docs-master/modules/verae-proxmox/NATS.md

@@ -0,0 +1,3 @@
+# NATS — verae-proxmox
+
+Guest clients use `nats://10.10.10.1:4222` (socat to host loopback). Never bind NATS on vmbr0.

packages/docs-master/modules/verae-proxmox/SUMMARY.md

@@ -0,0 +1,3 @@
+# verae-proxmox
+
+LXC 510 `verae-px-worker` on NS1 `vmbr1` `10.10.10.20`. Extra worm/tree copies. Private NATS proxy `10.10.10.1:4222`.

packages/docs-master/modules/verae-uptime/NATS.md

@@ -0,0 +1,3 @@
+# NATS — verae-uptime
+
+None. HTTPS only.

packages/docs-master/modules/verae-uptime/SUMMARY.md

@@ -0,0 +1,3 @@
+# verae-uptime
+
+Off-box HTTPS GET of public doors. `--once` for cron (exit 2 on failure); loop serves `/status` on `:3870`.

packages/overview/02-modules-and-repos.md

@@ -40,6 +40,10 @@ Each runtime piece is its **own git repo** on Forgejo (`git.georgelambert.org`,
 | **verae-bootstrap** | `packages/verae-bootstrap` | Clone every repo; `host-deps.sh` installs OS/Node/NATS per module |
 | **zapier-docs-master** | `packages/docs-master` | Per-module `SUMMARY.md` + `NATS.md` |
 | **verae-ops** | `packages/verae-ops` | Docker, Proxmox, VMs, dedicated hardware, linking services |
+| **verae-uptime** | `packages/verae-uptime` | Off-box HTTPS watch of public doors |
+| **verae-backup** | `packages/verae-backup` | SQLite / JetStream / worm+tree snapshots |
+| **verae-deploy** | `packages/verae-deploy` | Tagged checkout + npm ci on the target OS |
+| **verae-proxmox** | `packages/verae-proxmox` | LXC 510 worker on NS1 `vmbr1` |
 
 Libraries that are **not** separate Forgejo apps today: `verae-chain-client` (inside middleware), job-poller and webhook-deliver (middleware workers, fleet-spawned).
 

packages/overview/04-uptime.md

@@ -23,7 +23,11 @@ If a tree node is paused, crashes, or fails `/health`, fleet **starts another co
 
 ## Spread across machines
 
-`machines.json` lists hosts (`local`, `ns1` = `marchon@70.88.205.138` with `~/.ssh/id_ed25519`, optional `lan-134`). New replicas go to the **least-loaded** eligible host. Remote spawn/health/kill is SSH; workers bind loopback on the remote box.
+`machines.json` lists hosts (`local`, `ns1` = `marchon@70.88.205.138` with `~/.ssh/id_ed25519`, optional `lan-134`, **`px-worker`** = LXC 510 `10.10.10.20` on `vmbr1`). New replicas go to the **least-loaded** eligible host. Remote spawn/health/kill is SSH; workers bind loopback on the remote box.
+
+NS1 **is** the Proxmox host. Extra worm/tree copies on `px-worker` mean a host-process crash does not take every bloom/tree replica. A second **chassis** is still the next step for disk/PSU failure.
+
+Off-box probe: [verae-uptime](https://git.georgelambert.org/marchon/verae-uptime) (`node src/watch.js --once`). Backup: [verae-backup](https://git.georgelambert.org/marchon/verae-backup). Tagged upgrade: [verae-deploy](https://git.georgelambert.org/marchon/verae-deploy).
 
 ## JetStream
 

packages/overview/10-docs-sync.md

@@ -21,6 +21,7 @@ Validated against the module list in `packages/` and Forgejo repos.
 - `verae-staff-iam`, `verae-staff-session`, `verae-staff-ui`
 - `verae-keep`
 - `verae-bootstrap` (clone + npm per server type; `scripts/host-deps.sh` installs OS/Node/NATS)
+- `verae-uptime`, `verae-backup`, `verae-deploy`, `verae-proxmox` (LXC 510 on NS1 `vmbr1`)
 
 ## Move the catalog to another hostname
 

packages/overview/11-hostnames.md

@@ -9,6 +9,7 @@ All of the running Verae Time × Zapier processes on **2026-09-11** live on one
 | `local` (NS1 all-in-one) | **NS1.GEORGELAMBERT.ORG** | `70.88.205.138` | Control plane + archive workers + keep |
 | `ns1` (SSH worker profile) | **NS1.GEORGELAMBERT.ORG** | `70.88.205.138` | Same box; SSH spawn unused while all-in-one is local |
 | `lan-134` | *(none yet)* | `70.88.205.134` | Disabled until SSH works |
+| `px-worker` | **verae-px-worker** (LXC **510**) | `10.10.10.20` on `vmbr1` | Extra worm/tree; enable after `verae-proxmox` create-worker |
 
 Loopback on that box is `127.0.0.1`. Do **not** publish NATS `:4222`.
 

packages/verae-backup/NATS.md

@@ -0,0 +1,3 @@
+# NATS — verae-backup
+
+Backup copies the JetStream **store directory** if present. It does not publish or subscribe. Restore JetStream only with nats-server stopped.

packages/verae-backup/README.md

@@ -0,0 +1,23 @@
+# verae-backup
+
+Snapshot and restore **zappier SQLite**, staff IAM data, keep state, fleet-runtime (worm/tree blobs), and JetStream store if present.
+
+**Forgejo:** https://git.georgelambert.org/marchon/verae-backup  
+**SSH:** `ssh://git@git.georgelambert.org:2223/marchon/verae-backup.git`
+
+```bash
+# on NS1
+export BACKUP_ROOT=/SSD2/backups/verae
+bash scripts/backup.sh
+bash scripts/restore.sh --dry-run
+# after stopping keep/fleet consumers:
+# bash scripts/restore.sh --apply --archive /SSD2/backups/verae/20260101T031700Z.tgz
+```
+
+Paths: `paths.env`. Missing sources are skipped. Default retain 14 tarballs.
+
+User timer: `systemd/verae-backup.timer` (03:17 UTC). Enable with linger.
+
+Does **not** back up private keys. Restore identityFile **paths** on the operator host by hand.
+
+Does **not** connect to NATS; it copies the JetStream directory if `NATS_STORE` exists.

packages/verae-backup/SUMMARY.md

@@ -0,0 +1,3 @@
+# verae-backup
+
+Tar snapshots of SQLite, IAM, keep, worm/tree runtime, optional JetStream dir. Dry-run restore by default.

packages/verae-backup/package.json

@@ -0,0 +1,11 @@
+{
+  "name": "verae-backup",
+  "version": "0.1.0",
+  "private": true,
+  "description": "Backup and restore SQLite, JetStream, worm/tree, keep, and IAM data",
+  "scripts": {
+    "backup": "bash scripts/backup.sh",
+    "restore": "bash scripts/restore.sh",
+    "test": "bash scripts/test.sh"
+  }
+}

packages/verae-backup/paths.env

@@ -0,0 +1,16 @@
+# Sourced by backup.sh / restore.sh. Override in the environment.
+BACKUP_ROOT="${BACKUP_ROOT:-/SSD2/backups/verae}"
+KEEP=14
+
+# SQLite / IAM
+ZAPPIER_DB="${ZAPPIER_DB:-$HOME/verae-stack/packages/zappier/zappier.db}"
+IAM_DATA="${IAM_DATA:-$HOME/verae-stack/packages/verae-staff-iam/data}"
+
+# Keep + fleet runtime (worm/tree blobs live under fleet-runtime data/)
+KEEP_DATA="${KEEP_DATA:-$HOME/verae-keep/data}"
+FLEET_RUNTIME="${FLEET_RUNTIME:-$HOME/verae-fleet-runtime}"
+FLEET_PKG_DATA="${FLEET_PKG_DATA:-$HOME/verae-stack/packages/verae-fleet/data}"
+
+# JetStream (docker or nats-server). Missing paths are skipped.
+NATS_STORE="${NATS_STORE:-/var/lib/nats/jetstream}"
+NATS_DOCKER_VOL="${NATS_DOCKER_VOL:-}"

packages/verae-backup/scripts/backup.sh

@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+# Snapshot Verae data dirs into $BACKUP_ROOT/<utc-stamp>.tgz
+set -euo pipefail
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+# shellcheck disable=SC1091
+. "$ROOT/paths.env"
+STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
+DEST="$BACKUP_ROOT/$STAMP"
+mkdir -p "$DEST"
+copy_if() {
+  local src="$1" name="$2"
+  if [[ -e "$src" ]]; then
+    mkdir -p "$DEST/$name"
+    cp -a "$src" "$DEST/$name/" 2>/dev/null || sudo cp -a "$src" "$DEST/$name/"
+    echo "copied $src"
+  else
+    echo "skip missing $src"
+  fi
+}
+copy_if "$ZAPPIER_DB" sqlite
+copy_if "$IAM_DATA" iam
+copy_if "$KEEP_DATA" keep
+copy_if "$FLEET_RUNTIME" fleet-runtime
+copy_if "$FLEET_PKG_DATA" fleet-pkg
+copy_if "$NATS_STORE" nats
+if [[ -n "${NATS_DOCKER_VOL:-}" ]]; then copy_if "$NATS_DOCKER_VOL" nats-docker; fi
+tar -C "$BACKUP_ROOT" -czf "$BACKUP_ROOT/$STAMP.tgz" "$STAMP"
+rm -rf "$DEST"
+# retain
+ls -1dt "$BACKUP_ROOT"/*.tgz 2>/dev/null | tail -n +$((KEEP + 1)) | while read -r old; do
+  rm -f "$old"
+done
+echo "backup $BACKUP_ROOT/$STAMP.tgz"
+echo "$BACKUP_ROOT/$STAMP.tgz"

packages/verae-backup/scripts/restore.sh

@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+# Restore a .tgz from backup.sh. Default is --dry-run.
+set -euo pipefail
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+# shellcheck disable=SC1091
+. "$ROOT/paths.env"
+DRY=1
+ARCHIVE=""
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --apply) DRY=0 ;;
+    --dry-run) DRY=1 ;;
+    --archive) ARCHIVE="${2:-}"; shift ;;
+    *) echo "usage: restore.sh [--dry-run|--apply] [--archive file.tgz]" >&2; exit 1 ;;
+  esac
+  shift
+done
+if [[ -z "$ARCHIVE" ]]; then
+  ARCHIVE="$(ls -1t "$BACKUP_ROOT"/*.tgz 2>/dev/null | head -1 || true)"
+fi
+[[ -n "$ARCHIVE" && -f "$ARCHIVE" ]] || { echo "no archive" >&2; exit 1; }
+echo "archive $ARCHIVE dry=$DRY"
+TMP="$(mktemp -d)"
+tar -xzf "$ARCHIVE" -C "$TMP"
+INNER="$(find "$TMP" -mindepth 1 -maxdepth 1 -type d | head -1)"
+restore_one() {
+  local from="$1" to="$2"
+  [[ -e "$from" ]] || { echo "skip $from"; return; }
+  echo "would restore $from -> $to"
+  if [[ $DRY -eq 0 ]]; then
+    mkdir -p "$(dirname "$to")"
+    rm -rf "$to"
+    cp -a "$from" "$to"
+  fi
+}
+restore_one "$INNER/sqlite/$(basename "$ZAPPIER_DB")" "$ZAPPIER_DB"
+restore_one "$INNER/iam/$(basename "$IAM_DATA")" "$IAM_DATA"
+restore_one "$INNER/keep/$(basename "$KEEP_DATA")" "$KEEP_DATA"
+restore_one "$INNER/fleet-runtime/$(basename "$FLEET_RUNTIME")" "$FLEET_RUNTIME"
+restore_one "$INNER/nats/$(basename "$NATS_STORE")" "$NATS_STORE"
+rm -rf "$TMP"
+if [[ $DRY -eq 1 ]]; then
+  echo "dry-run only. re-run with --apply after stopping keep/fleet/nats consumers."
+fi