Add Proxmox worker CT, off-box watch, backup, and tagged deploy.
Some checks are pending
offline / test (push) Waiting to run
Some checks are pending
offline / test (push) Waiting to run
NS1 is the Proxmox host. verae-proxmox creates LXC 510 (verae-px-worker 10.10.10.20 on vmbr1) with a private NATS proxy on 10.10.10.1:4222. verae-uptime GET-watches public doors; verae-backup snapshots SQLite and worm/tree data; verae-deploy does host-deps + checkout + npm ci. Fleet overlays/ns1 are checked in (start.sh no longer rewrites JSON). User systemd + linger for keep and fleet survive reboot.
This commit is contained in:
parent
2d51d7a0dd
commit
7a5e25639e
69 changed files with 1087 additions and 118 deletions
|
|
@ -1,60 +1,50 @@
|
|||
# 3. Proxmox (LXC and QEMU)
|
||||
# 3. Proxmox (NS1 is the hypervisor)
|
||||
|
||||
Target: one **NATS cluster VM/CT** (or three), one **HTTPS edge** VM, one or more **worker** VMs. Fleet SSH-spreads worm/tree-node onto worker hosts.
|
||||
**NS1.GEORGELAMBERT.ORG** (`70.88.205.138`) **is** the Proxmox host. Control plane, Caddy, keep, and fleet run on the host. Extra worm/tree copies run in an **unprivileged Ubuntu LXC** on the private bridge.
|
||||
|
||||
## Suggested guests
|
||||
| Bridge | Address | Use |
|
||||
|--------|---------|-----|
|
||||
| `vmbr0` | `70.88.205.138/28` | Public HTTPS / SSH. **Never NATS.** |
|
||||
| `vmbr1` | `10.10.10.1/24` | Private. PKI `.11`, peergos-a `.12`, peergos-b `.13`, **verae-px-worker `.20`** |
|
||||
|
||||
| Guest | OS | vCPU / RAM / disk | Role |
|
||||
|-------|----|-------------------|------|
|
||||
| `nats-a` `nats-b` `nats-c` | Debian 12 LXC unprivileged | 1 / 1G / 8G | JetStream, bind `127.0.0.1` **or** a vmbr **private** bridge only |
|
||||
| `edge` | Debian 12 QEMU | 2 / 2G / 20G | Caddy + zappier-edge + middleware |
|
||||
| `worker-1` … | Debian 12 LXC | 2 / 2G / 20G | fleet SSH target; worm + tree-node |
|
||||
## Guest we actually deploy
|
||||
|
||||
Use a **private bridge** (`vmbr1`, RFC1918) for NATS `4222`/`6222`. Public `vmbr0` only on `edge` (80/443).
|
||||
| | |
|
||||
|--|--|
|
||||
| VMID | **510** |
|
||||
| Name | `verae-px-worker` |
|
||||
| Template | `ubuntu-24.04-standard_24.04-2_amd64.tar.zst` (`local`) |
|
||||
| Disk | `SSD2` 16G |
|
||||
| RAM / CPU | 2G / 2 |
|
||||
| Roles | `tree-node`, `archive-worm` |
|
||||
| Fleet id | `px-worker` |
|
||||
| NATS | `nats://10.10.10.1:4222` (host `socat` → `127.0.0.1:4222`) |
|
||||
|
||||
## NATS on an LXC
|
||||
Repo: [verae-proxmox](https://git.georgelambert.org/marchon/verae-proxmox)
|
||||
|
||||
```bash
|
||||
apt-get update && apt-get install -y curl
|
||||
# official nats-server binary
|
||||
curl -sSL https://github.com/nats-io/nats-server/releases/latest/download/nats-server-v2.10.26-linux-amd64.tar.gz | tar -xz
|
||||
install nats-server /usr/local/bin/
|
||||
# on NS1
|
||||
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-proxmox.git
|
||||
cd verae-proxmox
|
||||
bash scripts/create-worker.sh
|
||||
ssh -i ~/.ssh/id_ed25519 marchon@10.10.10.20 true
|
||||
```
|
||||
|
||||
`/etc/nats/nats.conf`:
|
||||
On the guest, bootstrap type `proxmox-worker` ([verae-bootstrap](https://git.georgelambert.org/marchon/verae-bootstrap)). Then `verae-fleet ssh-check px-worker` and set `"enabled": true` in `overlays/ns1/machines.json`.
|
||||
|
||||
```
|
||||
jetstream {}
|
||||
listen: 10.10.10.11:4222 # private bridge, not 0.0.0.0 on public
|
||||
cluster {
|
||||
name: verae
|
||||
listen: 10.10.10.11:6222
|
||||
routes: [ nats-route://10.10.10.12:6222, nats-route://10.10.10.13:6222 ]
|
||||
}
|
||||
http: 127.0.0.1:8222
|
||||
```
|
||||
## NATS for guests
|
||||
|
||||
systemd: `ExecStart=/usr/local/bin/nats-server -c /etc/nats/nats.conf`.
|
||||
Loopback NATS on the host stays `127.0.0.1:4222`. `verae-nats-private-proxy.service` binds **only** `10.10.10.1:4222`. Do not listen on `vmbr0`.
|
||||
|
||||
## Edge VM
|
||||
## Suggested future guests (not created by create-worker.sh)
|
||||
|
||||
Install Node 22, clone `zappier-edge` and `verae-middleware`, Caddy reverse-proxy:
|
||||
| Guest | Role |
|
||||
|-------|------|
|
||||
| `nats-b` `nats-c` | Extra JetStream nodes on `vmbr1` when leaving all-in-one |
|
||||
| more `worker-*` | Same recipe as 510 with the next free `10.10.10.2x` |
|
||||
|
||||
```
|
||||
zappier.example.com {
|
||||
reverse_proxy 127.0.0.1:3000
|
||||
}
|
||||
middleware.example.com {
|
||||
reverse_proxy 127.0.0.1:3100
|
||||
}
|
||||
```
|
||||
## Edge / Caddy
|
||||
|
||||
Middleware env: `NATS_URL=nats://10.10.10.11:4222,nats://10.10.10.12:4222,nats://10.10.10.13:4222`.
|
||||
Stay on the **host** (already live). Do not put zappier-edge in the worker CT.
|
||||
|
||||
## Fleet on Proxmox
|
||||
|
||||
On the operator workstation or a jump CT (loopback `:3850`):
|
||||
|
||||
`machines.json` entries: `kind=ssh`, `user=root` or `marchon`, `host=10.10.10.2x`, `identityFile=~/.ssh/id_ed25519` (**path only**). Roles: `tree-node,archive-worm`. Capacity ≥ 3 per worker if you want floors on one host.
|
||||
|
||||
`ssh-check` from the operator console before enabling `keepFloor`.
|
||||
Operator console: `https://fleet.zapier.georgelambert.org/` · overlay `packages/verae-fleet/overlays/ns1/`.
|
||||
|
|
|
|||
|
|
@ -13,28 +13,62 @@ Green / yellow / red on the operator console: https://zapier.georgelambert.org/p
|
|||
|
||||
## Restart
|
||||
|
||||
- systemd: `systemctl restart verae-middleware zappier-edge`
|
||||
- User systemd (survives reboot with linger):
|
||||
- `bash packages/verae-keep/scripts/install-systemd.sh`
|
||||
- `bash packages/verae-fleet/scripts/install-systemd.sh`
|
||||
- `loginctl enable-linger $USER`
|
||||
- `systemctl --user restart verae-keep-guard verae-fleet`
|
||||
- Compose: `docker compose up -d --no-deps middleware`
|
||||
- One replica: fleet `restart tree-node-1` (floor spawns a replacement if needed)
|
||||
- Pause is **not** off — paused copies do not count toward `keepFloor`
|
||||
- Do not run **both** `nohup guard.sh` and the systemd unit
|
||||
|
||||
## Backup
|
||||
|
||||
| Data | Where |
|
||||
|------|--------|
|
||||
| zappier SQLite | `ZAPPIER_DB` path / volume |
|
||||
| worm / tree blobs | instance `data/` dirs |
|
||||
| NATS JetStream | nats store dir |
|
||||
| fleet overlay | `packages/verae-fleet/data/` (gitignored) |
|
||||
Repo: [verae-backup](https://git.georgelambert.org/marchon/verae-backup)
|
||||
|
||||
Do not back up private keys into git. Restore `identityFile` paths on the operator host.
|
||||
```bash
|
||||
export BACKUP_ROOT=/SSD2/backups/verae
|
||||
bash scripts/backup.sh
|
||||
bash scripts/restore.sh --dry-run
|
||||
```
|
||||
|
||||
| Data | Default path |
|
||||
|------|----------------|
|
||||
| zappier SQLite | `~/verae-stack/packages/zappier/zappier.db` |
|
||||
| worm / tree blobs | `~/verae-fleet-runtime/data/` |
|
||||
| keep state | `~/verae-keep/data` |
|
||||
| NATS JetStream | `/var/lib/nats/jetstream` if present |
|
||||
| IAM | `~/verae-stack/packages/verae-staff-iam/data` |
|
||||
|
||||
User timer: `verae-backup.timer` (03:17 UTC). Do not back up private keys into git. Restore `identityFile` paths on the operator host. After `--apply`, `curl` `/health` on edge and keep.
|
||||
|
||||
## Upgrade
|
||||
|
||||
1. `git pull` each independent repo (or refresh the monorepo snapshot).
|
||||
2. `npm test` in that package.
|
||||
3. Restart that unit only; NATS cluster stays up.
|
||||
4. Zapier Platform app: `zapier-platform push` from `verae-zapier-app` when HTTPS contracts change.
|
||||
Repo: [verae-deploy](https://git.georgelambert.org/marchon/verae-deploy)
|
||||
|
||||
```bash
|
||||
export VERAE_SRC=$HOME/verae-src
|
||||
export DEPLOY_TYPE=ns1-all-in-one
|
||||
bash scripts/deploy.sh main # or a tag vYYYY-MM-DD
|
||||
systemctl --user restart verae-keep-guard verae-fleet
|
||||
```
|
||||
|
||||
1. `host-deps.sh` on **this** OS.
|
||||
2. `git fetch` + checkout ref (not force-push).
|
||||
3. `npm ci` / `npm rebuild` (never copy macOS `node_modules`).
|
||||
4. Restart that unit only; NATS stays up.
|
||||
5. Zapier Platform app: `zapier-platform push` from `verae-zapier-app` when HTTPS contracts change.
|
||||
|
||||
## Off-box watch
|
||||
|
||||
Repo: [verae-uptime](https://git.georgelambert.org/marchon/verae-uptime)
|
||||
|
||||
```bash
|
||||
node src/watch.js --once # exit 2 if a public door failed
|
||||
```
|
||||
|
||||
Run this on a **different** machine than NS1 (laptop cron is enough). `/status` on `:3870` in loop mode.
|
||||
|
||||
## Logs
|
||||
|
||||
|
|
|
|||
|
|
@ -33,6 +33,8 @@ The billing and user platform is the standalone **zappier** project (`/Users/mar
|
|||
|
||||
## 15-minute lab (one machine)
|
||||
|
||||
NS1 is Proxmox. Extra worm/tree LXC: [verae-proxmox](https://git.georgelambert.org/marchon/verae-proxmox) `create-worker.sh` (VMID 510, `10.10.10.20`). Off-box watch: [verae-uptime](https://git.georgelambert.org/marchon/verae-uptime). Backup: [verae-backup](https://git.georgelambert.org/marchon/verae-backup). Tagged upgrade: [verae-deploy](https://git.georgelambert.org/marchon/verae-deploy).
|
||||
|
||||
Bare metal / VM (Debian or Alpine): install OS + Node first, then clone.
|
||||
|
||||
```bash
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue