Add Proxmox worker CT, off-box watch, backup, and tagged deploy.

NS1 is the Proxmox host. verae-proxmox creates LXC 510 (verae-px-worker 10.10.10.20 on vmbr1) with a private NATS proxy on 10.10.10.1:4222. verae-uptime GET-watches public doors; verae-backup snapshots SQLite and worm/tree data; verae-deploy does host-deps + checkout + npm ci. Fleet overlays/ns1 are checked in (start.sh no longer rewrites JSON). User systemd + linger for keep and fleet survive reboot.
2026-09-11 23:35:44 -04:00 · 2026-09-11 23:35:44 -04:00 · 7a5e25639e
commit 7a5e25639e
parent 2d51d7a0dd
69 changed files with 1087 additions and 118 deletions
--- a/packages/verae-proxmox/NATS.md
+++ b/packages/verae-proxmox/NATS.md
@ -0,0 +1,3 @@
+# NATS — verae-proxmox
+
+Guests on `vmbr1` must not use the public NIC. The host runs `socat` `10.10.10.1:4222` → `127.0.0.1:4222`. Zapier cloud still never talks to NATS.
--- a/packages/verae-proxmox/README.md
+++ b/packages/verae-proxmox/README.md
@ -0,0 +1,38 @@
+# verae-proxmox
+
+Provision an **archive worker LXC** on **NS1.GEORGELAMBERT.ORG** (this box **is** the Proxmox host).
+
+| | |
+|--|--|
+| VMID | **510** |
+| Hostname | `verae-px-worker` |
+| Network | `vmbr1` **10.10.10.20/24** gw `10.10.10.1` |
+| Storage | SSD2 16G, unprivileged Ubuntu 24.04 |
+| Roles | `tree-node`, `archive-worm` |
+| NATS | `nats://10.10.10.1:4222` via host socat proxy (loopback NATS stays `127.0.0.1:4222`; **not** on `vmbr0`) |
+
+**Forgejo:** https://git.georgelambert.org/marchon/verae-proxmox  
+**SSH:** `ssh://git@git.georgelambert.org:2223/marchon/verae-proxmox.git`
+
+```bash
+# on NS1
+git clone ssh://git@git.georgelambert.org:2223/marchon/verae-proxmox.git
+cd verae-proxmox
+bash scripts/create-worker.sh
+ssh marchon@10.10.10.20
+```
+
+On the guest:
+
+```bash
+git clone ssh://git@git.georgelambert.org:2223/marchon/verae-bootstrap.git
+cd verae-bootstrap
+bash scripts/host-deps.sh --type proxmox-worker
+export VERAE_SRC=$HOME/verae-src
+export NATS_URL=nats://10.10.10.1:4222
+bash scripts/fetch.sh proxmox-worker
+```
+
+Then enable `px-worker` in `verae-fleet/overlays/ns1/machines.json` and `node src/cli.js ssh-check px-worker`.
+
+Keep on the guest uses `units.px-worker.json` (worm ×1 + tree-node ×1 extra copies). Host keep still runs the NS1 floor of 3.
--- a/packages/verae-proxmox/SUMMARY.md
+++ b/packages/verae-proxmox/SUMMARY.md
@ -0,0 +1,3 @@
+# verae-proxmox
+
+Create Ubuntu LXC 510 on NS1 vmbr1 for extra worm/tree copies. Private NATS proxy on 10.10.10.1:4222.
--- a/packages/verae-proxmox/guest.env
+++ b/packages/verae-proxmox/guest.env
@ -0,0 +1,15 @@
+# Defaults for NS1.GEORGELAMBERT.ORG (Proxmox).
+VMID="${VMID:-510}"
+GUEST_HOSTNAME="${GUEST_HOSTNAME:-verae-px-worker}"
+BRIDGE="${BRIDGE:-vmbr1}"
+IP="${IP:-10.10.10.20/24}"
+GW="${GW:-10.10.10.1}"
+MEMORY="${MEMORY:-2048}"
+CORES="${CORES:-2}"
+DISK="${DISK:-16}"
+STORAGE="${STORAGE:-SSD2}"
+TEMPLATE="${TEMPLATE:-/var/lib/vz/template/cache/ubuntu-24.04-standard_24.04-2_amd64.tar.zst}"
+DNS="${DNS:-8.8.8.8}"
+NATS_PRIVATE="${NATS_PRIVATE:-10.10.10.1}"
+NATS_PORT="${NATS_PORT:-4222}"
+USER_NAME="${USER_NAME:-marchon}"
--- a/packages/verae-proxmox/package.json
+++ b/packages/verae-proxmox/package.json
@ -0,0 +1,10 @@
+{
+  "name": "verae-proxmox",
+  "version": "0.1.0",
+  "private": true,
+  "description": "Provision Verae archive worker LXC guests on NS1 Proxmox (vmbr1)",
+  "scripts": {
+    "create-worker": "bash scripts/create-worker.sh",
+    "test": "bash scripts/test.sh"
+  }
+}
--- a/packages/verae-proxmox/scripts/create-worker.sh
+++ b/packages/verae-proxmox/scripts/create-worker.sh
@ -0,0 +1,80 @@
+#!/usr/bin/env bash
+# Create (or reuse) LXC 510 on this Proxmox host, bootstrap marchon + sshd + host-deps.
+# Run on NS1 as marchon (sudo). Does not publish NATS on vmbr0.
+set -euo pipefail
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+# shellcheck disable=SC1091
+. "$ROOT/guest.env"
+if [[ ! -f /etc/pve/local/.members ]] && [[ ! -d /etc/pve/nodes ]]; then
+  echo "not a Proxmox host" >&2
+  exit 1
+fi
+export PATH="/usr/sbin:/usr/bin:/bin:$PATH"
+if ! command -v pct >/dev/null; then
+  echo "pct missing (need /usr/sbin/pct on the Proxmox host)" >&2
+  exit 1
+fi
+if [[ ! -f "$TEMPLATE" ]]; then
+  echo "missing template $TEMPLATE" >&2
+  exit 1
+fi
+if ! sudo pct status "$VMID" >/dev/null 2>&1; then
+  echo "pct create $VMID $GUEST_HOSTNAME $IP"
+  sudo pct create "$VMID" "$TEMPLATE" \
+    --hostname "$GUEST_HOSTNAME" \
+    --memory "$MEMORY" --cores "$CORES" --swap 512 \
+    --net0 "name=eth0,bridge=${BRIDGE},ip=${IP},gw=${GW},type=veth" \
+    --rootfs "${STORAGE}:${DISK}" \
+    --unprivileged 1 --onboot 1 --nameserver "$DNS" \
+    --features nesting=1 \
+    --ostype ubuntu
+else
+  echo "CT $VMID already exists"
+fi
+sudo pct start "$VMID" 2>/dev/null || true
+for i in $(seq 1 30); do
+  sudo pct exec "$VMID" -- true 2>/dev/null && break
+  sleep 2
+done
+PUB=""
+[[ -f "$HOME/.ssh/id_ed25519.pub" ]] && PUB="$(cat "$HOME/.ssh/id_ed25519.pub")"
+[[ -z "$PUB" && -f "$HOME/.ssh/authorized_keys" ]] && PUB="$(head -1 "$HOME/.ssh/authorized_keys")"
+[[ -n "$PUB" ]] || { echo "no ssh public key for $USER_NAME" >&2; exit 1; }
+if [[ ! -f "$HOME/.ssh/id_ed25519" ]]; then
+  ssh-keygen -t ed25519 -N "" -f "$HOME/.ssh/id_ed25519" -C "ns1-to-px-worker"
+  PUB="$(cat "$HOME/.ssh/id_ed25519.pub")"
+fi
+sudo pct exec "$VMID" -- bash -lc "
+set -e
+export DEBIAN_FRONTEND=noninteractive
+apt-get update -qq
+apt-get install -y --no-install-recommends openssh-server sudo curl git ca-certificates python3 make g++ xz-utils
+id $USER_NAME >/dev/null 2>&1 || useradd -m -s /bin/bash $USER_NAME
+echo '$USER_NAME ALL=(ALL) NOPASSWD:ALL' >/etc/sudoers.d/90-$USER_NAME
+chmod 440 /etc/sudoers.d/90-$USER_NAME
+install -d -m 700 -o $USER_NAME -g $USER_NAME /home/$USER_NAME/.ssh
+grep -qxF '$PUB' /home/$USER_NAME/.ssh/authorized_keys 2>/dev/null || echo '$PUB' >>/home/$USER_NAME/.ssh/authorized_keys
+chown $USER_NAME:$USER_NAME /home/$USER_NAME/.ssh/authorized_keys
+chmod 600 /home/$USER_NAME/.ssh/authorized_keys
+systemctl enable --now ssh
+"
+HOST_IP="${IP%%/*}"
+echo "guest $GUEST_HOSTNAME $HOST_IP"
+# NATS private proxy on the Proxmox host
+if ! ss -lnt | grep -q "10.10.10.1:4222"; then
+  if command -v socat >/dev/null; then
+    sudo cp "$ROOT/scripts/nats-private-proxy.service" /etc/systemd/system/verae-nats-private-proxy.service
+    sudo systemctl daemon-reload
+    sudo systemctl enable --now verae-nats-private-proxy.service || true
+  else
+    sudo apt-get install -y socat || sudo apt-get install -y socat
+    sudo cp "$ROOT/scripts/nats-private-proxy.service" /etc/systemd/system/verae-nats-private-proxy.service
+    sudo systemctl daemon-reload
+    sudo systemctl enable --now verae-nats-private-proxy.service
+  fi
+fi
+ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ConnectTimeout=8 \
+  -i "$HOME/.ssh/id_ed25519" "$USER_NAME@$HOST_IP" 'echo ssh_ok' \
+  || echo "ssh not ready yet — retry: ssh $USER_NAME@$HOST_IP"
+echo "NATS for guests: nats://${NATS_PRIVATE}:${NATS_PORT} (never on vmbr0)"
+echo "next: on the guest, clone verae-bootstrap type proxmox-worker"
--- a/packages/verae-proxmox/scripts/nats-private-proxy.service
+++ b/packages/verae-proxmox/scripts/nats-private-proxy.service
@ -0,0 +1,13 @@
+[Unit]
+Description=Proxy loopback NATS onto vmbr1 (10.10.10.1:4222) for Proxmox guests
+After=network.target
+
+[Service]
+Type=simple
+# Public NIC must stay unbound. Loopback NATS remains 127.0.0.1:4222.
+ExecStart=/usr/bin/socat TCP-LISTEN:4222,bind=10.10.10.1,reuseaddr,fork TCP:127.0.0.1:4222
+Restart=always
+RestartSec=2
+
+[Install]
+WantedBy=multi-user.target
--- a/packages/verae-proxmox/scripts/test.sh
+++ b/packages/verae-proxmox/scripts/test.sh
@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+set -euo pipefail
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+bash -n "$ROOT/scripts/create-worker.sh"
+test -f "$ROOT/guest.env"
+test -f "$ROOT/scripts/nats-private-proxy.service"
+grep -q 'bind=10.10.10.1' "$ROOT/scripts/nats-private-proxy.service"
+grep -q vmbr1 "$ROOT/guest.env"
+echo OK