marchon/verae-ops
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Test env NATS cluster cut-over

Browse source
This commit is contained in:
George Lambert 2026-09-12 00:00:03 -04:00
commit a5b5612e84
15 changed files with 986 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

60
03-proxmox.md Normal file
View file

@ -0,0 +1,60 @@
# 3. Proxmox (NS1 is the hypervisor)
**NS1.GEORGELAMBERT.ORG** (`70.88.205.138`) **is** the Proxmox host. Control plane, Caddy, keep, and fleet run on the host. Extra worm/tree copies run in an **unprivileged Ubuntu LXC** on the private bridge.
| Bridge | Address | Use |
|--------|---------|-----|
| `vmbr0` | `70.88.205.138/28` | Public HTTPS / SSH. **Never NATS.** |
| `vmbr1` | `10.10.10.1/24` | Private. PKI `.11`, peergos-a `.12`, peergos-b `.13`, **verae-px-worker `.20`** |
## Guest we actually deploy
| | |
|--|--|
| VMID | **510** |
| Name | `verae-px-worker` |
| Template | `ubuntu-24.04-standard_24.04-2_amd64.tar.zst` (`local`) |
| Disk | `SSD2` 16G |
| RAM / CPU | 2G / 2 |
| Roles | `tree-node`, `archive-worm` |
| Fleet id | `px-worker` |
| NATS | `nats://10.10.10.1:4222` (host `socat` → `127.0.0.1:4222`) |
Repo: [verae-proxmox](https://git.georgelambert.org/marchon/verae-proxmox)
```bash
# on NS1
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-proxmox.git
cd verae-proxmox
bash scripts/create-worker.sh
ssh -i ~/.ssh/id_ed25519 marchon@10.10.10.20 true
```
On the guest, bootstrap type `proxmox-worker` ([verae-bootstrap](https://git.georgelambert.org/marchon/verae-bootstrap)). Then `verae-fleet ssh-check px-worker` and set `"enabled": true` in `overlays/ns1/machines.json`.
## NATS for guests
Loopback NATS on the host stays `127.0.0.1:4222` (current lab bus). `verae-nats-private-proxy.service` binds **only** `10.10.10.1:4222` for the worker CT.
**Three-node test cluster** (distinct LXC, same chassis): [verae-nats-cluster](https://git.georgelambert.org/marchon/verae-nats-cluster)
| Guest | VMID | IP |
|-------|------|-----|
| nats-a | 511 | 10.10.10.21 |
| nats-b | 512 | 10.10.10.22 |
| nats-c | 513 | 10.10.10.23 |
`bash scripts/create-cluster.sh` then `bash scripts/test.sh`. Client URL lists all three. Do not listen on `vmbr0`. Same configs later move to separate hardware.
## Suggested future guests (not created by create-worker.sh)
| Guest | Role |
|-------|------|
| `nats-b` `nats-c` | Extra JetStream nodes on `vmbr1` when leaving all-in-one |
| more `worker-*` | Same recipe as 510 with the next free `10.10.10.2x` |
## Edge / Caddy
Stay on the **host** (already live). Do not put zappier-edge in the worker CT.
Operator console: `https://fleet.zapier.georgelambert.org/` · overlay `packages/verae-fleet/overlays/ns1/`.