George Lambert
a7a7ec86ce
Some checks are pending
offline / test (push) Waiting to run
Same NS1 ladder as 20260912T051237Z. Product streams restored to ZFS; raised cores, RAM, and max_mem stay on 510–513.
73 lines
3.2 KiB
Markdown
73 lines
3.2 KiB
Markdown
# verae-nats-cluster
|
||
|
||
Three **distinct Proxmox LXC** guests running **NATS JetStream** as a cluster, on the private bridge only. This is the test stand before the same layout moves to separate hardware.
|
||
|
||
**Forgejo:** https://git.georgelambert.org/marchon/verae-nats-cluster
|
||
**SSH:** `ssh://git@git.georgelambert.org:2223/marchon/verae-nats-cluster.git`
|
||
|
||
| Guest | VMID | `vmbr1` IP | Client | Cluster | Monitor |
|
||
|-------|------|------------|--------|---------|---------|
|
||
| nats-a | 511 | 10.10.10.21 | `:4222` | `:6222` | `127.0.0.1:8222` |
|
||
| nats-b | 512 | 10.10.10.22 | `:4222` | `:6222` | `127.0.0.1:8222` |
|
||
| nats-c | 513 | 10.10.10.23 | `:4222` | `:6222` | `127.0.0.1:8222` |
|
||
|
||
Cluster name: `verae`. Client URL:
|
||
|
||
```text
|
||
nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
|
||
```
|
||
|
||
**Not** on `vmbr0`. **Not** `0.0.0.0`. Host lab NATS on `127.0.0.1:4222` is left running so current keep/fleet/middleware stay up until you cut over.
|
||
|
||
```bash
|
||
# on NS1 (the Proxmox host)
|
||
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-nats-cluster.git
|
||
cd verae-nats-cluster
|
||
bash scripts/create-cluster.sh
|
||
bash scripts/status.sh
|
||
bash scripts/test.sh
|
||
```
|
||
|
||
Test-env cut-over (keep/fleet/middleware on NS1):
|
||
|
||
```bash
|
||
# 1) create product streams (replicas=3)
|
||
bash scripts/ensure-streams.sh
|
||
# 2) fleet overlay nats.url + services/*.json already list the three URLs
|
||
# 3) restart fleet and keep with NATS_URL from client.env
|
||
```
|
||
|
||
`JETSTREAM_REPLICAS=3` on middleware. Host `127.0.0.1:4222` can stay up unused. Zapier cloud never talks to NATS.
|
||
|
||
nkeys/mTLS: `verae-nats-accounts` still has the INTERNAL/LEAF sketch. Do **not** enable accounts on this cluster until every client passes credentials in `NATS_URL`. Private `vmbr1` is the current isolation.
|
||
|
||
## Message speed (throughput + delay)
|
||
|
||
From a **client that is not a nats-* server** (LXC 510 `verae-px-worker`):
|
||
|
||
```bash
|
||
# on NS1 (hypervisor). Bench client is LXC 510.
|
||
bash scripts/bench.sh
|
||
|
||
# Full study that must run ON NS1 (orchestrator + charts + HTML/PDF):
|
||
bash scripts/study-on-ns1.sh
|
||
|
||
# Maximize cores/RAM + tmpfs JetStream, same ladder, restore ZFS after:
|
||
bash scripts/maximize-ns1-study.sh
|
||
```
|
||
|
||
That writes `results/<utc>/` and copies a summary to [BENCH.md](BENCH.md). Full write-up with charts (HTML + PDF) is in the progress repo: [nats-cluster-bench](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.pdf). Core NATS is fire-and-forget; JetStream rows use **file** storage and **replicas=3**. Ping delay is sequential RTT; flood delay is queueing under burst.
|
||
|
||
Latest run (`20260912T045131Z`):
|
||
|
||
| Kind | Load | Result |
|
||
|------|------|--------|
|
||
| Core 1p1s 50k×128 B | pub/sub | ~1.20M msgs/s aggregate |
|
||
| Core 8p8s 200k×128 B | pub/sub | ~2.01M msgs/s aggregate |
|
||
| Core 4p4s 50k×1 KiB | pub/sub | ~630k msgs/s (~616 MB/s aggregate) |
|
||
| JetStream r=3 file | 1–4 pubs, 128 B | ~16k durable pubs/s |
|
||
| JetStream r=3 file | 4 pubs, 1 KiB | ~13.5k durable pubs/s |
|
||
| Ping 1k×128 B | sequential RTT | avg 0.307 ms, p99 0.734 ms |
|
||
| Flood 8p 20k×128 B | burst queueing | avg 454 ms |
|
||
|
||
Hardware move: same three configs, three boxes, private NIC only — change IPs in `cluster.env` and `conf/nats.conf.tmpl`.
|