commit e60b01113cdbf9e2c5aa989a3cb004272ed2767d Author: George Lambert Date: Sat Sep 12 01:04:09 2026 -0400 Snapshot of verae-nats-cluster from zapier monorepo (NATS speed report) diff --git a/BENCH.md b/BENCH.md new file mode 100644 index 0000000..a24b18b --- /dev/null +++ b/BENCH.md @@ -0,0 +1,49 @@ +# NATS cluster message speed + +Run **`20260912T045131Z`** (UTC). Client: LXC **510** `verae-px-worker` (`10.10.10.20`), not a nats-* server. Servers: `nats-a/b/c` on `10.10.10.21–23` (`vmbr1` only). + +Client URL: + +```text +nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222 +``` + +## Method + +- **Core NATS** is fire-and-forget pub/sub (`nats bench`). No disk, no replica ack. +- **JetStream** uses **file** storage and **replicas=3** (same as product streams). The unique stream `benchstream` is deleted between JS loads. +- Throughput is **msgs/sec** from nats CLI **0.1.6** (`--no-progress --csv`). Its min/avg/max are publisher **rate spread**, not delay. +- **Ping** delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster. +- **Flood** delay: N publishers dump the whole batch, then the subscriber drains. This is **queueing under burst**, not wire RTT. +- Probe: `scripts/latency.mjs` (two connections, header timestamp). + +## Throughput + +| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s | +|-----|------|------------------|------------|----------|------------|----------| +| `core-1p1s-50k-128` | core pub/sub | 1,200,836 | 791,094 | 96.57 | 747,461 | 91.24 | +| `core-4p4s-100k-128` | core pub/sub | 1,521,256 | 316,312 | 38.61 | 1,299,634 | 158.65 | +| `core-4p4s-50k-1k` | core pub/sub | 630,460 | 247,747 | 241.94 | 510,216 | 498.26 | +| `core-8p8s-200k-128` | core pub/sub | 2,007,937 | 333,957 | 40.77 | 1,790,736 | 218.60 | +| `js-1p-20k-128-r3` | jetstream r=3 file | — | 16,155 | 1.97 | — | — | +| `js-2p2s-20k-128-r3` | jetstream r=3 file | 21,861 | 10,965 | 1.34 | 10,942 | 1.34 | +| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 13,493 | 13.18 | — | — | +| `js-4p-50k-128-r3` | jetstream r=3 file | — | 16,607 | 2.03 | — | — | + +## Round-trip delay + +| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max | +|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----| +| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.254ms | 0.307ms | 0.286ms | 0.332ms | 0.734ms | 2.763ms | +| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 149.279ms | 238.626ms | 248.752ms | 274.314ms | 279.398ms | 279.740ms | +| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 155.101ms | 211.706ms | 217.579ms | 223.268ms | 227.798ms | 228.370ms | +| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 174.174ms | 263.186ms | 266.672ms | 299.073ms | 304.233ms | 304.475ms | +| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 304.586ms | 453.749ms | 466.296ms | 499.924ms | 505.112ms | 505.620ms | + +## What the numbers mean + +Product job/event/archive traffic is **JetStream r=3 file**. On this three-LXC stand that is about **16k durable 128 B pubs/s** (about **13k** at 1 KiB). Core NATS is an upper bound for non-durable fan-out: about **0.7–2.0M msgs/s** aggregate at 128 B, or **~630k msgs/s (~616 MB/s)** at 1 KiB with 4 publishers. + +A quiet request-reply is **~0.3 ms** average, **p99 < 1 ms**. Flood rows in the **150–500 ms** band are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers. + +Re-run on NS1: `bash scripts/bench.sh`. Raw logs/CSVs are under `results//`. diff --git a/NATS.md b/NATS.md new file mode 100644 index 0000000..b326ade --- /dev/null +++ b/NATS.md @@ -0,0 +1,7 @@ +# NATS — verae-nats-cluster + +This repo **is** the three-node JetStream cluster. Listeners are the guest private IPs on `vmbr1` (`10.10.10.21–23:4222` and `:6222`). HTTP monitor is loopback `:8222` inside each guest. + +Zapier cloud, browsers, and `vmbr0` never get a NATS socket. + +Throughput and delay at several loads: [BENCH.md](BENCH.md). diff --git a/README.md b/README.md new file mode 100644 index 0000000..383ad04 --- /dev/null +++ b/README.md @@ -0,0 +1,67 @@ +# verae-nats-cluster + +Three **distinct Proxmox LXC** guests running **NATS JetStream** as a cluster, on the private bridge only. This is the test stand before the same layout moves to separate hardware. + +**Forgejo:** https://git.georgelambert.org/marchon/verae-nats-cluster +**SSH:** `ssh://git@git.georgelambert.org:2223/marchon/verae-nats-cluster.git` + +| Guest | VMID | `vmbr1` IP | Client | Cluster | Monitor | +|-------|------|------------|--------|---------|---------| +| nats-a | 511 | 10.10.10.21 | `:4222` | `:6222` | `127.0.0.1:8222` | +| nats-b | 512 | 10.10.10.22 | `:4222` | `:6222` | `127.0.0.1:8222` | +| nats-c | 513 | 10.10.10.23 | `:4222` | `:6222` | `127.0.0.1:8222` | + +Cluster name: `verae`. Client URL: + +```text +nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222 +``` + +**Not** on `vmbr0`. **Not** `0.0.0.0`. Host lab NATS on `127.0.0.1:4222` is left running so current keep/fleet/middleware stay up until you cut over. + +```bash +# on NS1 (the Proxmox host) +git clone ssh://git@git.georgelambert.org:2223/marchon/verae-nats-cluster.git +cd verae-nats-cluster +bash scripts/create-cluster.sh +bash scripts/status.sh +bash scripts/test.sh +``` + +Test-env cut-over (keep/fleet/middleware on NS1): + +```bash +# 1) create product streams (replicas=3) +bash scripts/ensure-streams.sh +# 2) fleet overlay nats.url + services/*.json already list the three URLs +# 3) restart fleet and keep with NATS_URL from client.env +``` + +`JETSTREAM_REPLICAS=3` on middleware. Host `127.0.0.1:4222` can stay up unused. Zapier cloud never talks to NATS. + +nkeys/mTLS: `verae-nats-accounts` still has the INTERNAL/LEAF sketch. Do **not** enable accounts on this cluster until every client passes credentials in `NATS_URL`. Private `vmbr1` is the current isolation. + +## Message speed (throughput + delay) + +From a **client that is not a nats-* server** (LXC 510 `verae-px-worker`): + +```bash +# on NS1 +bash scripts/bench.sh +``` + +That writes `results//` and copies a summary to [BENCH.md](BENCH.md). Full write-up with charts (HTML + PDF) is in the progress repo: [nats-cluster-bench](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.pdf). Core NATS is fire-and-forget; JetStream rows use **file** storage and **replicas=3**. Ping delay is sequential RTT; flood delay is queueing under burst. + +Latest run (`20260912T045131Z`): + +| Kind | Load | Result | +|------|------|--------| +| Core 1p1s 50k×128 B | pub/sub | ~1.20M msgs/s aggregate | +| Core 8p8s 200k×128 B | pub/sub | ~2.01M msgs/s aggregate | +| Core 4p4s 50k×1 KiB | pub/sub | ~630k msgs/s (~616 MB/s aggregate) | +| JetStream r=3 file | 1–4 pubs, 128 B | ~16k durable pubs/s | +| JetStream r=3 file | 4 pubs, 1 KiB | ~13.5k durable pubs/s | +| Ping 1k×128 B | sequential RTT | avg 0.307 ms, p99 0.734 ms | +| Flood 8p 20k×128 B | burst queueing | avg 454 ms | + +Hardware move: same three configs, three boxes, private NIC only — change IPs in `cluster.env` and `conf/nats.conf.tmpl`. diff --git a/SUMMARY.md b/SUMMARY.md new file mode 100644 index 0000000..286cd8a --- /dev/null +++ b/SUMMARY.md @@ -0,0 +1,5 @@ +# verae-nats-cluster + +Proxmox LXC 511–513 (`nats-a/b/c`) form a JetStream cluster on `vmbr1` for HA testing before dedicated hardware. + +Message speed (client LXC 510 vs the three nodes): see [BENCH.md](BENCH.md). Product streams are JetStream **file + replicas=3** (~16k durable 128 B pubs/s on this stand). Core NATS is ~1–2M msgs/s at 128 B. Sequential RTT ~0.3 ms; flood queueing is hundreds of ms. diff --git a/client.env b/client.env new file mode 100644 index 0000000..4afa412 --- /dev/null +++ b/client.env @@ -0,0 +1,3 @@ +# Test-environment client URL. Private vmbr1 only. Lab loopback stays until cut-over. +export NATS_URL="${NATS_URL:-nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222}" +export JETSTREAM_REPLICAS="${JETSTREAM_REPLICAS:-3}" diff --git a/cluster.env b/cluster.env new file mode 100644 index 0000000..a0df60b --- /dev/null +++ b/cluster.env @@ -0,0 +1,19 @@ +# Distinct Proxmox LXC guests on NS1 vmbr1. Not the host loopback NATS. +BRIDGE="${BRIDGE:-vmbr1}" +GW="${GW:-10.10.10.1}" +MEMORY="${MEMORY:-1024}" +CORES="${CORES:-1}" +DISK="${DISK:-8}" +STORAGE="${STORAGE:-SSD2}" +TEMPLATE="${TEMPLATE:-/var/lib/vz/template/cache/ubuntu-24.04-standard_24.04-2_amd64.tar.zst}" +DNS="${DNS:-8.8.8.8}" +USER_NAME="${USER_NAME:-marchon}" +NATS_VER="${NATS_VER:-2.10.26}" +CLUSTER_NAME="${CLUSTER_NAME:-verae}" + +# vmid hostname ipv4 +NODES="${NODES:- +511 nats-a 10.10.10.21 +512 nats-b 10.10.10.22 +513 nats-c 10.10.10.23 +}" diff --git a/conf/nats.conf.tmpl b/conf/nats.conf.tmpl new file mode 100644 index 0000000..481de1c --- /dev/null +++ b/conf/nats.conf.tmpl @@ -0,0 +1,19 @@ +# JetStream cluster node. Bind the private guest IP only — never vmbr0 / 0.0.0.0. +server_name: {{NAME}} +host: {{IP}} +port: 4222 +http: 127.0.0.1:8222 + +jetstream { + store_dir: /var/lib/nats/jetstream + max_mem: 256M + max_file: 4G +} + +cluster { + name: {{CLUSTER}} + listen: {{IP}}:6222 + routes: [ +{{ROUTES}} + ] +} diff --git a/package.json b/package.json new file mode 100644 index 0000000..dcc5c11 --- /dev/null +++ b/package.json @@ -0,0 +1,12 @@ +{ + "name": "verae-nats-cluster", + "version": "0.1.0", + "private": true, + "description": "Three-node NATS JetStream cluster on distinct Proxmox LXC guests (vmbr1 only)", + "scripts": { + "create": "bash scripts/create-cluster.sh", + "test": "bash scripts/test.sh", + "status": "bash scripts/status.sh", + "bench": "bash scripts/bench.sh" + } +} diff --git a/results/.gitignore b/results/.gitignore new file mode 100644 index 0000000..1b2bd4d --- /dev/null +++ b/results/.gitignore @@ -0,0 +1,4 @@ +* +!.gitignore +!20260912T045131Z/ +!20260912T045131Z/** diff --git a/results/20260912T045131Z/BENCH.md b/results/20260912T045131Z/BENCH.md new file mode 100644 index 0000000..a24b18b --- /dev/null +++ b/results/20260912T045131Z/BENCH.md @@ -0,0 +1,49 @@ +# NATS cluster message speed + +Run **`20260912T045131Z`** (UTC). Client: LXC **510** `verae-px-worker` (`10.10.10.20`), not a nats-* server. Servers: `nats-a/b/c` on `10.10.10.21–23` (`vmbr1` only). + +Client URL: + +```text +nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222 +``` + +## Method + +- **Core NATS** is fire-and-forget pub/sub (`nats bench`). No disk, no replica ack. +- **JetStream** uses **file** storage and **replicas=3** (same as product streams). The unique stream `benchstream` is deleted between JS loads. +- Throughput is **msgs/sec** from nats CLI **0.1.6** (`--no-progress --csv`). Its min/avg/max are publisher **rate spread**, not delay. +- **Ping** delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster. +- **Flood** delay: N publishers dump the whole batch, then the subscriber drains. This is **queueing under burst**, not wire RTT. +- Probe: `scripts/latency.mjs` (two connections, header timestamp). + +## Throughput + +| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s | +|-----|------|------------------|------------|----------|------------|----------| +| `core-1p1s-50k-128` | core pub/sub | 1,200,836 | 791,094 | 96.57 | 747,461 | 91.24 | +| `core-4p4s-100k-128` | core pub/sub | 1,521,256 | 316,312 | 38.61 | 1,299,634 | 158.65 | +| `core-4p4s-50k-1k` | core pub/sub | 630,460 | 247,747 | 241.94 | 510,216 | 498.26 | +| `core-8p8s-200k-128` | core pub/sub | 2,007,937 | 333,957 | 40.77 | 1,790,736 | 218.60 | +| `js-1p-20k-128-r3` | jetstream r=3 file | — | 16,155 | 1.97 | — | — | +| `js-2p2s-20k-128-r3` | jetstream r=3 file | 21,861 | 10,965 | 1.34 | 10,942 | 1.34 | +| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 13,493 | 13.18 | — | — | +| `js-4p-50k-128-r3` | jetstream r=3 file | — | 16,607 | 2.03 | — | — | + +## Round-trip delay + +| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max | +|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----| +| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.254ms | 0.307ms | 0.286ms | 0.332ms | 0.734ms | 2.763ms | +| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 149.279ms | 238.626ms | 248.752ms | 274.314ms | 279.398ms | 279.740ms | +| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 155.101ms | 211.706ms | 217.579ms | 223.268ms | 227.798ms | 228.370ms | +| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 174.174ms | 263.186ms | 266.672ms | 299.073ms | 304.233ms | 304.475ms | +| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 304.586ms | 453.749ms | 466.296ms | 499.924ms | 505.112ms | 505.620ms | + +## What the numbers mean + +Product job/event/archive traffic is **JetStream r=3 file**. On this three-LXC stand that is about **16k durable 128 B pubs/s** (about **13k** at 1 KiB). Core NATS is an upper bound for non-durable fan-out: about **0.7–2.0M msgs/s** aggregate at 128 B, or **~630k msgs/s (~616 MB/s)** at 1 KiB with 4 publishers. + +A quiet request-reply is **~0.3 ms** average, **p99 < 1 ms**. Flood rows in the **150–500 ms** band are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers. + +Re-run on NS1: `bash scripts/bench.sh`. Raw logs/CSVs are under `results//`. diff --git a/results/20260912T045131Z/core-1p1s-50k-128.csv b/results/20260912T045131Z/core-1p1s-50k-128.csv new file mode 100644 index 0000000..cfd7d62 --- /dev/null +++ b/results/20260912T045131Z/core-1p1s-50k-128.csv @@ -0,0 +1,3 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +8e1ExxrZ9OtHTkwXsOfzes,S0,50000,6400000,747461,95675134.125326,0.066893 +8e1ExxrZ9OtHTkwXsOfzes,P0,50000,6400000,791094,101260059.613379,0.063204 diff --git a/results/20260912T045131Z/core-1p1s-50k-128.txt b/results/20260912T045131Z/core-1p1s-50k-128.txt new file mode 100644 index 0000000..f2e88d8 --- /dev/null +++ b/results/20260912T045131Z/core-1p1s-50k-128.txt @@ -0,0 +1,7 @@ +=== core-1p1s-50k-128 === + +NATS Pub/Sub stats: 1,200,836 msgs/sec ~ 146.59 MB/sec + Pub stats: 791,094 msgs/sec ~ 96.57 MB/sec + Sub stats: 747,461 msgs/sec ~ 91.24 MB/sec + +Saved metric data in csv file /tmp/bench.csv diff --git a/results/20260912T045131Z/core-4p4s-100k-128.csv b/results/20260912T045131Z/core-4p4s-100k-128.csv new file mode 100644 index 0000000..eaa0df7 --- /dev/null +++ b/results/20260912T045131Z/core-4p4s-100k-128.csv @@ -0,0 +1,9 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +d1csC8CbCBLYlUMAUVZxpb,S0,100000,12800000,377317,48296660.364795,0.265029 +d1csC8CbCBLYlUMAUVZxpb,S1,100000,12800000,351445,44985056.104940,0.284539 +d1csC8CbCBLYlUMAUVZxpb,S2,100000,12800000,372973,47740617.289389,0.268116 +d1csC8CbCBLYlUMAUVZxpb,S3,100000,12800000,324908,41588310.877432,0.307779 +d1csC8CbCBLYlUMAUVZxpb,P0,25000,3200000,524453,67129988.054428,0.047669 +d1csC8CbCBLYlUMAUVZxpb,P1,25000,3200000,324337,41515233.366208,0.077080 +d1csC8CbCBLYlUMAUVZxpb,P2,25000,3200000,103170,13205778.771206,0.242318 +d1csC8CbCBLYlUMAUVZxpb,P3,25000,3200000,79260,10145297.784464,0.315417 diff --git a/results/20260912T045131Z/core-4p4s-100k-128.txt b/results/20260912T045131Z/core-4p4s-100k-128.txt new file mode 100644 index 0000000..175d7ca --- /dev/null +++ b/results/20260912T045131Z/core-4p4s-100k-128.txt @@ -0,0 +1,17 @@ +=== core-4p4s-100k-128 === + +NATS Pub/Sub stats: 1,521,256 msgs/sec ~ 185.70 MB/sec + Pub stats: 316,312 msgs/sec ~ 38.61 MB/sec + [1] 524,453 msgs/sec ~ 64.02 MB/sec (25000 msgs) + [2] 324,337 msgs/sec ~ 39.59 MB/sec (25000 msgs) + [3] 103,170 msgs/sec ~ 12.59 MB/sec (25000 msgs) + [4] 79,260 msgs/sec ~ 9.68 MB/sec (25000 msgs) + min 79,260 | avg 257,805 | max 524,453 | stddev 181,189 msgs + Sub stats: 1,299,634 msgs/sec ~ 158.65 MB/sec + [1] 377,317 msgs/sec ~ 46.06 MB/sec (100000 msgs) + [2] 351,445 msgs/sec ~ 42.90 MB/sec (100000 msgs) + [3] 372,973 msgs/sec ~ 45.53 MB/sec (100000 msgs) + [4] 324,908 msgs/sec ~ 39.66 MB/sec (100000 msgs) + min 324,908 | avg 356,660 | max 377,317 | stddev 20,785 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/results/20260912T045131Z/core-4p4s-50k-1k.csv b/results/20260912T045131Z/core-4p4s-50k-1k.csv new file mode 100644 index 0000000..becf15b --- /dev/null +++ b/results/20260912T045131Z/core-4p4s-50k-1k.csv @@ -0,0 +1,9 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +LXVQ7kvd26BIwiMRRfbr7Z,S0,50000,51200000,169213,173274762.582948,0.295484 +LXVQ7kvd26BIwiMRRfbr7Z,S1,50000,51200000,150273,153880103.181538,0.332727 +LXVQ7kvd26BIwiMRRfbr7Z,S2,50000,51200000,142138,145550108.197459,0.351769 +LXVQ7kvd26BIwiMRRfbr7Z,S3,50000,51200000,132134,135306220.651698,0.378401 +LXVQ7kvd26BIwiMRRfbr7Z,P0,12500,12800000,176262,180492467.210335,0.070917 +LXVQ7kvd26BIwiMRRfbr7Z,P1,12500,12800000,132211,135384883.941361,0.094545 +LXVQ7kvd26BIwiMRRfbr7Z,P2,12500,12800000,70678,72375041.084851,0.176857 +LXVQ7kvd26BIwiMRRfbr7Z,P3,12500,12800000,61936,63423297.033934,0.201819 diff --git a/results/20260912T045131Z/core-4p4s-50k-1k.txt b/results/20260912T045131Z/core-4p4s-50k-1k.txt new file mode 100644 index 0000000..30a8ca2 --- /dev/null +++ b/results/20260912T045131Z/core-4p4s-50k-1k.txt @@ -0,0 +1,17 @@ +=== core-4p4s-50k-1k === + +NATS Pub/Sub stats: 630,460 msgs/sec ~ 615.68 MB/sec + Pub stats: 247,747 msgs/sec ~ 241.94 MB/sec + [1] 176,262 msgs/sec ~ 172.13 MB/sec (12500 msgs) + [2] 132,211 msgs/sec ~ 129.11 MB/sec (12500 msgs) + [3] 70,678 msgs/sec ~ 69.02 MB/sec (12500 msgs) + [4] 61,936 msgs/sec ~ 60.49 MB/sec (12500 msgs) + min 61,936 | avg 110,271 | max 176,262 | stddev 46,744 msgs + Sub stats: 510,216 msgs/sec ~ 498.26 MB/sec + [1] 169,213 msgs/sec ~ 165.25 MB/sec (50000 msgs) + [2] 150,273 msgs/sec ~ 146.75 MB/sec (50000 msgs) + [3] 142,138 msgs/sec ~ 138.81 MB/sec (50000 msgs) + [4] 132,134 msgs/sec ~ 129.04 MB/sec (50000 msgs) + min 132,134 | avg 148,439 | max 169,213 | stddev 13,605 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/results/20260912T045131Z/core-8p8s-200k-128.csv b/results/20260912T045131Z/core-8p8s-200k-128.csv new file mode 100644 index 0000000..591315b --- /dev/null +++ b/results/20260912T045131Z/core-8p8s-200k-128.csv @@ -0,0 +1,17 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +HuvulruPjjaqnoHWeFCH2I,S0,200000,25600000,349350,44716843.071789,0.572491 +HuvulruPjjaqnoHWeFCH2I,S1,200000,25600000,248162,31764858.924451,0.805922 +HuvulruPjjaqnoHWeFCH2I,S2,200000,25600000,244206,31258460.045739,0.818978 +HuvulruPjjaqnoHWeFCH2I,S3,200000,25600000,243405,31155913.818220,0.821674 +HuvulruPjjaqnoHWeFCH2I,S4,200000,25600000,230550,29510438.641581,0.867490 +HuvulruPjjaqnoHWeFCH2I,S5,200000,25600000,231495,29631384.363172,0.863949 +HuvulruPjjaqnoHWeFCH2I,S6,200000,25600000,229528,29379679.692357,0.871351 +HuvulruPjjaqnoHWeFCH2I,S7,200000,25600000,223842,28651789.197519,0.893487 +HuvulruPjjaqnoHWeFCH2I,P0,25000,3200000,152536,19524693.902374,0.163895 +HuvulruPjjaqnoHWeFCH2I,P1,25000,3200000,84526,10819333.375159,0.295767 +HuvulruPjjaqnoHWeFCH2I,P2,25000,3200000,82017,10498223.303653,0.304813 +HuvulruPjjaqnoHWeFCH2I,P3,25000,3200000,57214,7323427.896669,0.436954 +HuvulruPjjaqnoHWeFCH2I,P4,25000,3200000,53879,6896566.453658,0.463999 +HuvulruPjjaqnoHWeFCH2I,P5,25000,3200000,51420,6581852.801766,0.486185 +HuvulruPjjaqnoHWeFCH2I,P6,25000,3200000,48572,6217303.262856,0.514693 +HuvulruPjjaqnoHWeFCH2I,P7,25000,3200000,41744,5343333.685114,0.598877 diff --git a/results/20260912T045131Z/core-8p8s-200k-128.txt b/results/20260912T045131Z/core-8p8s-200k-128.txt new file mode 100644 index 0000000..e841fde --- /dev/null +++ b/results/20260912T045131Z/core-8p8s-200k-128.txt @@ -0,0 +1,25 @@ +=== core-8p8s-200k-128 === + +NATS Pub/Sub stats: 2,007,937 msgs/sec ~ 245.11 MB/sec + Pub stats: 333,957 msgs/sec ~ 40.77 MB/sec + [1] 152,536 msgs/sec ~ 18.62 MB/sec (25000 msgs) + [2] 84,526 msgs/sec ~ 10.32 MB/sec (25000 msgs) + [3] 82,017 msgs/sec ~ 10.01 MB/sec (25000 msgs) + [4] 57,214 msgs/sec ~ 6.98 MB/sec (25000 msgs) + [5] 53,879 msgs/sec ~ 6.58 MB/sec (25000 msgs) + [6] 51,420 msgs/sec ~ 6.28 MB/sec (25000 msgs) + [7] 48,572 msgs/sec ~ 5.93 MB/sec (25000 msgs) + [8] 41,744 msgs/sec ~ 5.10 MB/sec (25000 msgs) + min 41,744 | avg 71,488 | max 152,536 | stddev 33,867 msgs + Sub stats: 1,790,736 msgs/sec ~ 218.60 MB/sec + [1] 349,350 msgs/sec ~ 42.65 MB/sec (200000 msgs) + [2] 248,162 msgs/sec ~ 30.29 MB/sec (200000 msgs) + [3] 244,206 msgs/sec ~ 29.81 MB/sec (200000 msgs) + [4] 243,405 msgs/sec ~ 29.71 MB/sec (200000 msgs) + [5] 230,550 msgs/sec ~ 28.14 MB/sec (200000 msgs) + [6] 231,495 msgs/sec ~ 28.26 MB/sec (200000 msgs) + [7] 229,528 msgs/sec ~ 28.02 MB/sec (200000 msgs) + [8] 223,842 msgs/sec ~ 27.32 MB/sec (200000 msgs) + min 223,842 | avg 250,067 | max 349,350 | stddev 38,364 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/results/20260912T045131Z/js-1p-20k-128-r3.csv b/results/20260912T045131Z/js-1p-20k-128-r3.csv new file mode 100644 index 0000000..a859e0f --- /dev/null +++ b/results/20260912T045131Z/js-1p-20k-128-r3.csv @@ -0,0 +1,2 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +S0xQOJuOM4PlwAHoh5Bj7f,P0,40000,2560000,16155,2067883.858001,1.237981 diff --git a/results/20260912T045131Z/js-1p-20k-128-r3.txt b/results/20260912T045131Z/js-1p-20k-128-r3.txt new file mode 100644 index 0000000..acbbaba --- /dev/null +++ b/results/20260912T045131Z/js-1p-20k-128-r3.txt @@ -0,0 +1,5 @@ +=== js-1p-20k-128-r3 === + +Pub stats: 16,155 msgs/sec ~ 1.97 MB/sec + +Saved metric data in csv file /tmp/bench.csv diff --git a/results/20260912T045131Z/js-2p2s-20k-128-r3.csv b/results/20260912T045131Z/js-2p2s-20k-128-r3.csv new file mode 100644 index 0000000..99cceef --- /dev/null +++ b/results/20260912T045131Z/js-2p2s-20k-128-r3.csv @@ -0,0 +1,5 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +smN9zLbWAAZER4FjSmfwcO,S0,20126,1280000,5474,700679.773679,1.826797 +smN9zLbWAAZER4FjSmfwcO,S1,20130,1280000,5471,700313.845376,1.827752 +smN9zLbWAAZER4FjSmfwcO,P0,20000,1280000,8678,1110910.588857,1.152208 +smN9zLbWAAZER4FjSmfwcO,P1,20000,1280000,5485,702119.773868,1.823051 diff --git a/results/20260912T045131Z/js-2p2s-20k-128-r3.txt b/results/20260912T045131Z/js-2p2s-20k-128-r3.txt new file mode 100644 index 0000000..b308dd7 --- /dev/null +++ b/results/20260912T045131Z/js-2p2s-20k-128-r3.txt @@ -0,0 +1,13 @@ +=== js-2p2s-20k-128-r3 === + +NATS Pub/Sub stats: 21,861 msgs/sec ~ 2.67 MB/sec + Pub stats: 10,965 msgs/sec ~ 1.34 MB/sec + [1] 8,678 msgs/sec ~ 1.06 MB/sec (10000 msgs) + [2] 5,485 msgs/sec ~ 685.66 KB/sec (10000 msgs) + min 5,485 | avg 7,081 | max 8,678 | stddev 1,596 msgs + Sub stats: 10,942 msgs/sec ~ 1.34 MB/sec + [1] 5,474 msgs/sec ~ 684.26 KB/sec (10000 msgs) + [2] 5,471 msgs/sec ~ 683.90 KB/sec (10000 msgs) + min 5,471 | avg 5,472 | max 5,474 | stddev 1 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/results/20260912T045131Z/js-4p-20k-1k-r3.csv b/results/20260912T045131Z/js-4p-20k-1k-r3.csv new file mode 100644 index 0000000..87fbaa8 --- /dev/null +++ b/results/20260912T045131Z/js-4p-20k-1k-r3.csv @@ -0,0 +1,5 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +zOcPILhowCWfp2PGmHUgqx,P0,10000,5120000,5121,5244574.853257,0.976247 +zOcPILhowCWfp2PGmHUgqx,P1,10000,5120000,4281,4383926.080211,1.167903 +zOcPILhowCWfp2PGmHUgqx,P2,10000,5120000,3480,3564282.061583,1.436474 +zOcPILhowCWfp2PGmHUgqx,P3,10000,5120000,3373,3454449.904903,1.482146 diff --git a/results/20260912T045131Z/js-4p-20k-1k-r3.txt b/results/20260912T045131Z/js-4p-20k-1k-r3.txt new file mode 100644 index 0000000..0a13168 --- /dev/null +++ b/results/20260912T045131Z/js-4p-20k-1k-r3.txt @@ -0,0 +1,10 @@ +=== js-4p-20k-1k-r3 === + +Pub stats: 13,493 msgs/sec ~ 13.18 MB/sec + [1] 5,121 msgs/sec ~ 5.00 MB/sec (5000 msgs) + [2] 4,281 msgs/sec ~ 4.18 MB/sec (5000 msgs) + [3] 3,480 msgs/sec ~ 3.40 MB/sec (5000 msgs) + [4] 3,373 msgs/sec ~ 3.29 MB/sec (5000 msgs) + min 3,373 | avg 4,063 | max 5,121 | stddev 704 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/results/20260912T045131Z/js-4p-50k-128-r3.csv b/results/20260912T045131Z/js-4p-50k-128-r3.csv new file mode 100644 index 0000000..69c5663 --- /dev/null +++ b/results/20260912T045131Z/js-4p-50k-128-r3.csv @@ -0,0 +1,5 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +vZIFpR751J0vbDTJQDYGTN,P0,25000,1600000,6628,848387.283433,1.885931 +vZIFpR751J0vbDTJQDYGTN,P1,25000,1600000,5762,737571.570534,2.169281 +vZIFpR751J0vbDTJQDYGTN,P2,25000,1600000,4162,532825.280344,3.002861 +vZIFpR751J0vbDTJQDYGTN,P3,25000,1600000,4154,531740.716359,3.008985 diff --git a/results/20260912T045131Z/js-4p-50k-128-r3.txt b/results/20260912T045131Z/js-4p-50k-128-r3.txt new file mode 100644 index 0000000..75ea713 --- /dev/null +++ b/results/20260912T045131Z/js-4p-50k-128-r3.txt @@ -0,0 +1,10 @@ +=== js-4p-50k-128-r3 === + +Pub stats: 16,607 msgs/sec ~ 2.03 MB/sec + [1] 6,628 msgs/sec ~ 828.50 KB/sec (12500 msgs) + [2] 5,762 msgs/sec ~ 720.28 KB/sec (12500 msgs) + [3] 4,162 msgs/sec ~ 520.34 KB/sec (12500 msgs) + [4] 4,154 msgs/sec ~ 519.28 KB/sec (12500 msgs) + min 4,154 | avg 5,176 | max 6,628 | stddev 1,063 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/results/20260912T045131Z/lat-1p-5k-128.txt b/results/20260912T045131Z/lat-1p-5k-128.txt new file mode 100644 index 0000000..9a05d77 --- /dev/null +++ b/results/20260912T045131Z/lat-1p-5k-128.txt @@ -0,0 +1,2 @@ +=== lat-1p-5k-128 === +{"count":5000,"pubs":1,"size":128,"min_us":149279,"avg_us":238626,"p50_us":248752,"p90_us":274314,"p99_us":279398,"max_us":279740,"min":"149.279ms","avg":"238.626ms","p50":"248.752ms","p90":"274.314ms","p99":"279.398ms","max":"279.740ms"} diff --git a/results/20260912T045131Z/lat-4p-10k-128.txt b/results/20260912T045131Z/lat-4p-10k-128.txt new file mode 100644 index 0000000..e5c883c --- /dev/null +++ b/results/20260912T045131Z/lat-4p-10k-128.txt @@ -0,0 +1,2 @@ +=== lat-4p-10k-128 === +{"count":10000,"pubs":4,"size":128,"min_us":174174,"avg_us":263186,"p50_us":266672,"p90_us":299073,"p99_us":304233,"max_us":304475,"min":"174.174ms","avg":"263.186ms","p50":"266.672ms","p90":"299.073ms","p99":"304.233ms","max":"304.475ms"} diff --git a/results/20260912T045131Z/lat-4p-5k-1k.txt b/results/20260912T045131Z/lat-4p-5k-1k.txt new file mode 100644 index 0000000..e22e578 --- /dev/null +++ b/results/20260912T045131Z/lat-4p-5k-1k.txt @@ -0,0 +1,2 @@ +=== lat-4p-5k-1k === +{"count":5000,"pubs":4,"size":1024,"min_us":155101,"avg_us":211706,"p50_us":217579,"p90_us":223268,"p99_us":227798,"max_us":228370,"min":"155.101ms","avg":"211.706ms","p50":"217.579ms","p90":"223.268ms","p99":"227.798ms","max":"228.370ms"} diff --git a/results/20260912T045131Z/lat-8p-20k-128.txt b/results/20260912T045131Z/lat-8p-20k-128.txt new file mode 100644 index 0000000..b445541 --- /dev/null +++ b/results/20260912T045131Z/lat-8p-20k-128.txt @@ -0,0 +1,2 @@ +=== lat-8p-20k-128 === +{"count":20000,"pubs":8,"size":128,"min_us":304586,"avg_us":453749,"p50_us":466296,"p90_us":499924,"p99_us":505112,"max_us":505620,"min":"304.586ms","avg":"453.749ms","p50":"466.296ms","p90":"499.924ms","p99":"505.112ms","max":"505.620ms"} diff --git a/results/20260912T045131Z/lat-ping-1k-128.txt b/results/20260912T045131Z/lat-ping-1k-128.txt new file mode 100644 index 0000000..4084dc8 --- /dev/null +++ b/results/20260912T045131Z/lat-ping-1k-128.txt @@ -0,0 +1,2 @@ +=== lat-ping-1k-128 === +{"count":1000,"pubs":1,"size":128,"min_us":254,"avg_us":307,"p50_us":286,"p90_us":332,"p99_us":734,"max_us":2763,"min":"0.254ms","avg":"0.307ms","p50":"0.286ms","p90":"0.332ms","p99":"0.734ms","max":"2.763ms"} diff --git a/scripts/bench-report.py b/scripts/bench-report.py new file mode 100755 index 0000000..f3f1f5c --- /dev/null +++ b/scripts/bench-report.py @@ -0,0 +1,176 @@ +#!/usr/bin/env python3 +"""Turn nats bench text logs + latency JSON into markdown.""" +from __future__ import annotations + +import json +import re +import sys +from pathlib import Path + + +def fmt_int(s: str | None) -> str: + if not s: + return "—" + return f"{int(s):,}" + + +def parse_bench(text: str) -> dict[str, str]: + out: dict[str, str] = {"kind": "throughput"} + m = re.search(r"(?m)^\s*Pub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*MB/sec", text) + if m: + out["pub_msgs"] = m.group(1).replace(",", "") + out["pub_mb"] = m.group(2) + m = re.search(r"(?m)^\s*Sub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*MB/sec", text) + if m: + out["sub_msgs"] = m.group(1).replace(",", "") + out["sub_mb"] = m.group(2) + m = re.search(r"NATS Pub/Sub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*MB/sec", text) + if m: + out["agg_msgs"] = m.group(1).replace(",", "") + out["agg_mb"] = m.group(2) + if "JetStream" in text or "--js" in text or "js-" in text: + out["mode"] = "jetstream r=3 file" + else: + out["mode"] = "core pub/sub" + # nats 0.1.6 prints min/avg/max as msgs/sec across publishers, not µs delay + m = re.search( + r"min\s+([0-9,]+)\s*\|\s*avg\s+([0-9,]+)\s*\|\s*max\s+([0-9,]+)\s*\|\s*stddev\s+([0-9,]+)\s*msgs", + text, + ) + if m: + out["pub_spread"] = f"{m.group(1)}–{m.group(3)} (avg {m.group(2)})" + return out + + +def parse_lat(text: str) -> dict[str, str] | None: + for line in text.splitlines(): + line = line.strip() + if line.startswith("{") and "p99_us" in line: + d = json.loads(line) + mode = d.get("mode") or "" + return { + "kind": "latency", + "count": str(d.get("count", "")), + "pubs": str(d.get("pubs", "")), + "size": str(d.get("size", "")), + "mode": str(mode), + "min": d.get("min", ""), + "avg": d.get("avg", ""), + "p50": d.get("p50", ""), + "p90": d.get("p90", ""), + "p99": d.get("p99", ""), + "max": d.get("max", ""), + } + return None + + +def lat_mode(run: str, recorded: str) -> str: + if recorded in ("ping", "flood"): + return recorded + if "ping" in run: + return "ping" + return "flood" + + +def thru_sort(p: dict[str, str]) -> tuple: + return (0 if p.get("mode", "").startswith("core") else 1, p.get("run", "")) + + +def lat_sort(p: dict[str, str]) -> tuple: + mode = lat_mode(p.get("run", ""), p.get("mode", "")) + return (0 if mode == "ping" else 1, int(p.get("count") or 0), p.get("run", "")) + + +def main() -> int: + folder = Path(sys.argv[1] if len(sys.argv) > 1 else ".") + thru: list[dict[str, str]] = [] + lats: list[dict[str, str]] = [] + for f in sorted(folder.glob("*.txt")): + text = f.read_text(encoding="utf-8", errors="replace") + lat = parse_lat(text) + if lat: + lat["run"] = f.stem + lats.append(lat) + continue + p = parse_bench(text) + if p.get("pub_msgs") or p.get("agg_msgs"): + p["run"] = f.stem + thru.append(p) + + stamp = folder.name if re.fullmatch(r"\d{8}T\d{6}Z", folder.name) else "" + + print("# NATS cluster message speed") + print() + if stamp: + print(f"Run **`{stamp}`** (UTC). ", end="") + print( + "Client: LXC **510** `verae-px-worker` (`10.10.10.20`), not a nats-* server. " + "Servers: `nats-a/b/c` on `10.10.10.21–23` (`vmbr1` only)." + ) + print() + print("Client URL:") + print() + print("```text") + print("nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222") + print("```") + print() + print("## Method") + print() + print("- **Core NATS** is fire-and-forget pub/sub (`nats bench`). No disk, no replica ack.") + print( + "- **JetStream** uses **file** storage and **replicas=3** (same as product streams). " + "The unique stream `benchstream` is deleted between JS loads." + ) + print("- Throughput is **msgs/sec** from nats CLI **0.1.6** (`--no-progress --csv`). Its min/avg/max are publisher **rate spread**, not delay.") + print( + "- **Ping** delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster." + ) + print( + "- **Flood** delay: N publishers dump the whole batch, then the subscriber drains. " + "This is **queueing under burst**, not wire RTT." + ) + print("- Probe: `scripts/latency.mjs` (two connections, header timestamp).") + print() + print("## Throughput") + print() + print("| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |") + print("|-----|------|------------------|------------|----------|------------|----------|") + for p in sorted(thru, key=thru_sort): + print( + f"| `{p['run']}` | {p.get('mode', '')} | {fmt_int(p.get('agg_msgs'))} | " + f"{fmt_int(p.get('pub_msgs'))} | {p.get('pub_mb') or '—'} | " + f"{fmt_int(p.get('sub_msgs'))} | {p.get('sub_mb') or '—'} |" + ) + print() + print("## Round-trip delay") + print() + print("| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |") + print("|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|") + for p in sorted(lats, key=lat_sort): + kind = lat_mode(p.get("run", ""), p.get("mode", "")) + label = "ping (sequential RTT)" if kind == "ping" else "flood (burst queueing)" + print( + f"| `{p['run']}` | {label} | {p.get('count', '')} | {p.get('pubs', '')} | " + f"{p.get('size', '')} B | {p.get('min', '')} | {p.get('avg', '')} | " + f"{p.get('p50', '')} | {p.get('p90', '')} | {p.get('p99', '')} | {p.get('max', '')} |" + ) + print() + print("## What the numbers mean") + print() + print( + "Product job/event/archive traffic is **JetStream r=3 file**. On this three-LXC stand that is about " + "**16k durable 128 B pubs/s** (about **13k** at 1 KiB). Core NATS is an upper bound for " + "non-durable fan-out: about **0.7–2.0M msgs/s** aggregate at 128 B, or **~630k msgs/s (~616 MB/s)** at 1 KiB with 4 publishers." + ) + print() + print( + "A quiet request-reply is **~0.3 ms** average, **p99 < 1 ms**. Flood rows in the **150–500 ms** band " + "are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers." + ) + print() + print("Re-run on NS1: `bash scripts/bench.sh`. Raw logs/CSVs are under `results//`.") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/bench.sh b/scripts/bench.sh new file mode 100755 index 0000000..a572fe6 --- /dev/null +++ b/scripts/bench.sh @@ -0,0 +1,106 @@ +#!/usr/bin/env bash +# Message throughput and delay ladder against the 3-node vmbr1 cluster. +# Prefers a client that is not a nats-* server (px-worker LXC 510). +set -euo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +# shellcheck disable=SC1091 +. "$ROOT/client.env" +export PATH="/usr/sbin:/usr/bin:/bin:/usr/local/bin:$PATH" +STAMP="$(date -u +%Y%m%dT%H%M%SZ)" +OUT="${BENCH_OUT:-$ROOT/results/$STAMP}" +CLIENT_VMID="${CLIENT_VMID:-510}" +mkdir -p "$OUT" + +ensure_nats_cli() { + local vmid="$1" + sudo pct exec "$vmid" -- bash -lc ' +set -e +export DEBIAN_FRONTEND=noninteractive +export PATH=/usr/local/bin:/usr/bin:/bin +if [[ ! -x /usr/local/bin/nats ]]; then + apt-get install -y --no-install-recommends unzip curl ca-certificates >/dev/null + curl -fsSL https://github.com/nats-io/natscli/releases/download/v0.1.6/nats-0.1.6-linux-amd64.zip -o /tmp/natscli.zip + rm -rf /tmp/natscli && mkdir -p /tmp/natscli + unzip -o /tmp/natscli.zip -d /tmp/natscli >/dev/null + BIN=$(find /tmp/natscli -type f -name nats | head -1) + install -m 0755 "$BIN" /usr/local/bin/nats +fi +nats --version +' +} + +run_one() { + local name="$1" + shift + echo "=== $name ===" | tee "$OUT/$name.txt" + # nats bench writes csv itself when --csv is a path inside the guest + sudo pct exec "$CLIENT_VMID" -- bash -lc " +export PATH=/usr/local/bin:/usr/bin:/bin +export NATS_URL='$NATS_URL' +nats bench --no-progress --csv=/tmp/bench.csv $* +" | tee -a "$OUT/$name.txt" + sudo pct exec "$CLIENT_VMID" -- cat /tmp/bench.csv >"$OUT/$name.csv" || true +} + +echo "client LXC $CLIENT_VMID NATS_URL=$NATS_URL out=$OUT" +ensure_nats_cli "$CLIENT_VMID" + +# Core NATS pub/sub — increasing publishers (same 128 B payload) +run_one core-1p1s-50k-128 bench.core.a --pub 1 --sub 1 --msgs 50000 --size 128 +run_one core-4p4s-100k-128 bench.core.b --pub 4 --sub 4 --msgs 100000 --size 128 +run_one core-8p8s-200k-128 bench.core.c --pub 8 --sub 8 --msgs 200000 --size 128 +run_one core-4p4s-50k-1k bench.core.d --pub 4 --sub 4 --msgs 50000 --size 1024 + +js_rm() { + sudo pct exec "$CLIENT_VMID" -- bash -lc " +export PATH=/usr/local/bin:/usr/bin:/bin +export NATS_URL='$NATS_URL' +nats stream rm benchstream --force >/dev/null 2>&1 || true +" +} +# JetStream file store, replicas=3 (matches product streams) +js_rm +run_one js-1p-20k-128-r3 bench.js.a --js --purge --pub 1 --msgs 20000 --size 128 --replicas 3 --storage file --maxbytes=512MB --stream=benchstream +js_rm +run_one js-4p-50k-128-r3 bench.js.b --js --purge --pub 4 --msgs 50000 --size 128 --replicas 3 --storage file --maxbytes=512MB --stream=benchstream +js_rm +run_one js-4p-20k-1k-r3 bench.js.c --js --purge --pub 4 --msgs 20000 --size 1024 --replicas 3 --storage file --maxbytes=512MB --stream=benchstream +js_rm +run_one js-2p2s-20k-128-r3 bench.js.d --js --purge --pub 2 --sub 2 --msgs 20000 --size 128 --replicas 3 --storage file --maxbytes=512MB --pull --stream=benchstream +js_rm + +# Round-trip delay (two connections, through the cluster) at several loads +sudo pct exec "$CLIENT_VMID" -- bash -lc " +set -e +export PATH=/usr/local/bin:/usr/bin:/bin +export NATS_URL='$NATS_URL' +mkdir -p /tmp/nats-lat +cd /tmp/nats-lat +if [[ ! -d node_modules/nats ]]; then + npm init -y >/dev/null + npm install --no-audit --no-fund nats@2 >/dev/null +fi +" >/dev/null +sudo pct push "$CLIENT_VMID" "$ROOT/scripts/latency.mjs" /tmp/nats-lat/latency.mjs || \ + sudo pct exec "$CLIENT_VMID" -- bash -c 'cat > /tmp/nats-lat/latency.mjs' < "$ROOT/scripts/latency.mjs" +lat() { + local name="$1" n="$2" sz="$3" p="$4" mode="${5:-flood}" + echo "=== $name ===" | tee "$OUT/$name.txt" + sudo pct exec "$CLIENT_VMID" -- bash -lc " +export PATH=/usr/local/bin:/usr/bin:/bin +export NATS_URL='$NATS_URL' +cd /tmp/nats-lat +node latency.mjs $n $sz $p $mode +" | tee -a "$OUT/$name.txt" +} +# copy latest probe +sudo pct exec "$CLIENT_VMID" -- bash -c 'cat > /tmp/nats-lat/latency.mjs' < "$ROOT/scripts/latency.mjs" +lat lat-ping-1k-128 1000 128 1 ping +lat lat-1p-5k-128 5000 128 1 flood +lat lat-4p-10k-128 10000 128 4 flood +lat lat-8p-20k-128 20000 128 8 flood +lat lat-4p-5k-1k 5000 1024 4 flood + +python3 "$ROOT/scripts/bench-report.py" "$OUT" >"$OUT/BENCH.md" +cp "$OUT/BENCH.md" "$ROOT/BENCH.md" +echo "wrote $OUT/BENCH.md and $ROOT/BENCH.md" diff --git a/scripts/create-cluster.sh b/scripts/create-cluster.sh new file mode 100755 index 0000000..bb42239 --- /dev/null +++ b/scripts/create-cluster.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash +# Create three distinct Proxmox LXC guests and start a JetStream cluster on vmbr1. +# Does not touch host loopback NATS (127.0.0.1:4222) or vmbr0. +set -euo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +# shellcheck disable=SC1091 +. "$ROOT/cluster.env" +# shellcheck disable=SC1091 +. "$ROOT/scripts/lib-ct.sh" +ct_require_proxmox + +mapfile -t rows < <(printf '%s\n' "$NODES" | awk 'NF==3 {print}') +[[ ${#rows[@]} -eq 3 ]] || { echo "need exactly 3 nodes in cluster.env" >&2; exit 1; } + +declare -a VMIDS NAMES IPS +for row in "${rows[@]}"; do + # shellcheck disable=SC2086 + set -- $row + VMIDS+=("$1"); NAMES+=("$2"); IPS+=("$3") +done + +i=0 +for i in 0 1 2; do + ct_ensure "${VMIDS[$i]}" "${NAMES[$i]}" "${IPS[$i]}" + ct_bootstrap_user "${VMIDS[$i]}" +done + +# Install nats-server + conf + systemd on each guest +for i in 0 1 2; do + routes="" + for j in 0 1 2; do + [[ $i -eq $j ]] && continue + routes="${routes} nats-route://${IPS[$j]}:6222"$'\n' + done + tmpconf="$(mktemp)" + NAME="${NAMES[$i]}" IP="${IPS[$i]}" CLUSTER="$CLUSTER_NAME" ROUTES="$routes" \ + python3 - "$ROOT/conf/nats.conf.tmpl" "$tmpconf" <<'PY' +import os, pathlib, sys +t = pathlib.Path(sys.argv[1]).read_text() +out = t.replace("{{NAME}}", os.environ["NAME"]).replace("{{IP}}", os.environ["IP"]).replace("{{CLUSTER}}", os.environ["CLUSTER"]).replace("{{ROUTES}}", os.environ["ROUTES"]) +pathlib.Path(sys.argv[2]).write_text(out) +PY + sudo pct exec "${VMIDS[$i]}" -- bash -c 'cat > /tmp/nats.conf' < "$tmpconf" + sudo pct exec "${VMIDS[$i]}" -- bash -c 'cat > /tmp/nats-server.service' < "$ROOT/systemd/nats-server.service" + rm -f "$tmpconf" + sudo pct exec "${VMIDS[$i]}" -- bash -lc " +set -e +export DEBIAN_FRONTEND=noninteractive +id nats >/dev/null 2>&1 || useradd -r -s /usr/sbin/nologin nats +install -d -m 755 -o nats -g nats /var/lib/nats/jetstream /etc/nats +mv /tmp/nats.conf /etc/nats/nats.conf +chown root:root /etc/nats/nats.conf +chmod 644 /etc/nats/nats.conf +if [[ ! -x /usr/local/bin/nats-server ]]; then + curl -fsSL https://github.com/nats-io/nats-server/releases/download/v${NATS_VER}/nats-server-v${NATS_VER}-linux-amd64.tar.gz -o /tmp/nats.tgz + tar -xzf /tmp/nats.tgz -C /tmp + install -m 0755 /tmp/nats-server-v${NATS_VER}-linux-amd64/nats-server /usr/local/bin/nats-server + rm -rf /tmp/nats.tgz /tmp/nats-server-v${NATS_VER}-linux-amd64 +fi +install -m 644 /tmp/nats-server.service /etc/systemd/system/nats-server.service +systemctl daemon-reload +systemctl enable --now nats-server +" + echo "nats-server ${NAMES[$i]} ${IPS[$i]}:4222 cluster ${IPS[$i]}:6222" +done + +echo "cluster client URL: nats://${IPS[0]}:4222,nats://${IPS[1]}:4222,nats://${IPS[2]}:4222" +echo "lab loopback NATS on the host is unchanged (127.0.0.1:4222)" +echo "next: bash $ROOT/scripts/test.sh" diff --git a/scripts/cutover-ns1.sh b/scripts/cutover-ns1.sh new file mode 100755 index 0000000..6fa073c --- /dev/null +++ b/scripts/cutover-ns1.sh @@ -0,0 +1,10 @@ +#!/usr/bin/env bash +# Point NS1 test modules at the 3-node vmbr1 cluster. Does not change MOCK_VERAE or Zapier. +set -euo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +# shellcheck disable=SC1091 +. "$ROOT/client.env" +export PATH="/usr/sbin:/usr/bin:/bin:$PATH" +bash "$ROOT/scripts/ensure-streams.sh" +echo "NATS_URL=$NATS_URL" +echo "streams ensured. restart keep + fleet on the host after copying overlay/service JSON." diff --git a/scripts/ensure-streams.sh b/scripts/ensure-streams.sh new file mode 100755 index 0000000..4b661be --- /dev/null +++ b/scripts/ensure-streams.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash +# Create product JetStream streams with replicas=3 on the Proxmox cluster. +set -euo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +# shellcheck disable=SC1091 +. "$ROOT/client.env" +export PATH="/usr/sbin:/usr/bin:/bin:/usr/local/bin:$PATH" +VMID="${1:-511}" +sudo pct exec "$VMID" -- bash -lc " +set -e +export DEBIAN_FRONTEND=noninteractive +export PATH=/usr/local/bin:/usr/bin:/bin +export NATS_URL=nats://10.10.10.21:4222 +if [[ ! -x /usr/local/bin/nats ]]; then + apt-get install -y --no-install-recommends unzip >/dev/null + curl -fsSL https://github.com/nats-io/natscli/releases/download/v0.1.6/nats-0.1.6-linux-amd64.zip -o /tmp/natscli.zip + rm -rf /tmp/natscli && mkdir -p /tmp/natscli + unzip -o /tmp/natscli.zip -d /tmp/natscli >/dev/null + BIN=\$(find /tmp/natscli -type f -name nats | head -1) + install -m 0755 \"\$BIN\" /usr/local/bin/nats +fi +add() { + local name=\$1 subj=\$2 + nats stream info \"\$name\" >/dev/null 2>&1 && return 0 + nats stream add \"\$name\" --subjects=\"\$subj\" --replicas=3 --storage=file --retention=limits --discard=old --max-msgs=-1 --max-bytes=-1 --max-age=24h --dupe-window=2m --defaults +} +add ZAPIER_JOBS 'verae.zapier.jobs.watch' +add ZAPIER_EVENTS 'verae.zapier.jobs.events' +add ZAPIER_WEBHOOKS 'verae.zapier.webhooks.deliver' +add ZAPIER_USAGE 'verae.zapier.usage' +add VERAE_ARCHIVE 'verae.archive.>' +nats stream ls +" +echo "streams ready on cluster (replicas=3)" diff --git a/scripts/latency.mjs b/scripts/latency.mjs new file mode 100644 index 0000000..919bf15 --- /dev/null +++ b/scripts/latency.mjs @@ -0,0 +1,105 @@ +#!/usr/bin/env node +/** + * Pub→sub round trip through the cluster (two connections). + * Usage: NATS_URL=... node latency.mjs [count] [payloadBytes] [publishers] [ping|flood] + * ping = sequential publish-wait (one-message RTT) + * flood = publish the batch then drain (queueing under burst) + */ +import { connect, headers } from "nats"; + +const url = process.env.NATS_URL || "nats://10.10.10.21:4222"; +const count = Number(process.argv[2] || 5000); +const size = Number(process.argv[3] || 128); +const pubs = Number(process.argv[4] || 1); +const mode = process.argv[5] || "flood"; +const servers = url.split(",").map((s) => s.trim()); +const subject = `bench.lat.${process.pid}`; +const payload = new Uint8Array(size); + +function pct(sorted, p) { + if (!sorted.length) return 0; + const i = Math.min(sorted.length - 1, Math.floor((p / 100) * sorted.length)); + return sorted[i]; +} + +const samples = []; +if (mode === "ping") { + const subNc = await connect({ servers, name: "lat-sub" }); + const pubNc = await connect({ servers, name: "lat-pub" }); + let resolveOne = null; + const sub = subNc.subscribe(subject, { max: count }); + const consume = (async () => { + for await (const m of sub) { + const sent = Number(m.headers?.get("t") || 0); + samples.push(Number(process.hrtime.bigint() / 1000n) - sent); + resolveOne?.(); + } + })(); + await subNc.flush(); + for (let i = 0; i < count; i++) { + const got = new Promise((r) => { + resolveOne = r; + }); + const h = headers(); + h.set("t", String(process.hrtime.bigint() / 1000n)); + pubNc.publish(subject, payload, { headers: h }); + await got; + } + await consume; + await pubNc.close(); + await subNc.close(); +} else { + const subNc = await connect({ servers, name: "lat-sub" }); + const sub = subNc.subscribe(subject, { max: count }); + const done = (async () => { + for await (const m of sub) { + const sent = Number(m.headers?.get("t") || 0); + if (sent) samples.push(Number(process.hrtime.bigint() / 1000n) - sent); + } + })(); + await subNc.flush(); + const per = Math.ceil(count / pubs); + const publishers = []; + for (let p = 0; p < pubs; p++) { + publishers.push( + (async () => { + const nc = await connect({ servers, name: `lat-pub-${p}` }); + const n = p === pubs - 1 ? count - per * (pubs - 1) : per; + for (let i = 0; i < n; i++) { + const h = headers(); + h.set("t", String(process.hrtime.bigint() / 1000n)); + nc.publish(subject, payload, { headers: h }); + } + await nc.flush(); + await nc.close(); + })(), + ); + } + await Promise.all(publishers); + await done; + await subNc.close(); +} + +samples.sort((a, b) => a - b); +const sum = samples.reduce((a, b) => a + b, 0); +const us = (n) => `${(n / 1000).toFixed(3)}ms`; +console.log( + JSON.stringify({ + count: samples.length, + pubs, + size, + mode, + min_us: samples[0], + avg_us: Math.round(sum / samples.length), + p50_us: pct(samples, 50), + p90_us: pct(samples, 90), + p99_us: pct(samples, 99), + max_us: samples[samples.length - 1], + min: us(samples[0]), + avg: us(sum / samples.length), + p50: us(pct(samples, 50)), + p90: us(pct(samples, 90)), + p99: us(pct(samples, 99)), + max: us(samples[samples.length - 1]), + }), +); diff --git a/scripts/lib-ct.sh b/scripts/lib-ct.sh new file mode 100755 index 0000000..1a4ff8e --- /dev/null +++ b/scripts/lib-ct.sh @@ -0,0 +1,62 @@ +# shellcheck shell=bash +# Shared LXC bootstrap for NS1 Proxmox. Does not generate SSH keys if one exists. +export PATH="/usr/sbin:/usr/bin:/bin:$PATH" + +ct_require_proxmox() { + if [[ ! -d /etc/pve/nodes ]]; then + echo "not a Proxmox host" >&2 + return 1 + fi + command -v pct >/dev/null || { echo "pct missing" >&2; return 1; } +} + +ct_ensure() { + local vmid="$1" hostname="$2" ip="$3" + if [[ ! -f "$TEMPLATE" ]]; then + echo "missing template $TEMPLATE" >&2 + return 1 + fi + if ! sudo pct status "$vmid" >/dev/null 2>&1; then + echo "pct create $vmid $hostname $ip/24" + sudo pct create "$vmid" "$TEMPLATE" \ + --hostname "$hostname" \ + --memory "$MEMORY" --cores "$CORES" --swap 256 \ + --net0 "name=eth0,bridge=${BRIDGE},ip=${ip}/24,gw=${GW},type=veth" \ + --rootfs "${STORAGE}:${DISK}" \ + --unprivileged 1 --onboot 1 --nameserver "$DNS" \ + --features nesting=1 \ + --ostype ubuntu + else + echo "CT $vmid already exists" + fi + sudo pct start "$vmid" 2>/dev/null || true + local i + for i in $(seq 1 40); do + sudo pct exec "$vmid" -- true 2>/dev/null && return 0 + sleep 2 + done + echo "CT $vmid did not start" >&2 + return 1 +} + +ct_bootstrap_user() { + local vmid="$1" + local pub="" + [[ -f "$HOME/.ssh/id_ed25519.pub" ]] && pub="$(cat "$HOME/.ssh/id_ed25519.pub")" + [[ -z "$pub" && -f "$HOME/.ssh/authorized_keys" ]] && pub="$(head -1 "$HOME/.ssh/authorized_keys")" + [[ -n "$pub" ]] || { echo "no ssh public key" >&2; return 1; } + sudo pct exec "$vmid" -- bash -lc " +set -e +export DEBIAN_FRONTEND=noninteractive +apt-get update -qq +apt-get install -y --no-install-recommends openssh-server sudo curl ca-certificates xz-utils tar +id $USER_NAME >/dev/null 2>&1 || useradd -m -s /bin/bash $USER_NAME +echo '$USER_NAME ALL=(ALL) NOPASSWD:ALL' >/etc/sudoers.d/90-$USER_NAME +chmod 440 /etc/sudoers.d/90-$USER_NAME +install -d -m 700 -o $USER_NAME -g $USER_NAME /home/$USER_NAME/.ssh +grep -qxF '$pub' /home/$USER_NAME/.ssh/authorized_keys 2>/dev/null || echo '$pub' >>/home/$USER_NAME/.ssh/authorized_keys +chown $USER_NAME:$USER_NAME /home/$USER_NAME/.ssh/authorized_keys +chmod 600 /home/$USER_NAME/.ssh/authorized_keys +systemctl enable --now ssh +" +} diff --git a/scripts/status.sh b/scripts/status.sh new file mode 100755 index 0000000..1306656 --- /dev/null +++ b/scripts/status.sh @@ -0,0 +1,22 @@ +#!/usr/bin/env bash +set -euo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +# shellcheck disable=SC1091 +. "$ROOT/cluster.env" +export PATH="/usr/sbin:/usr/bin:/bin:$PATH" +printf '%s\n' "$NODES" | awk 'NF==3 {print}' | while read -r vmid name ip; do + st="$(sudo pct status "$vmid" 2>/dev/null || echo missing)" + js="$(sudo pct exec "$vmid" -- curl -fsS --max-time 2 http://127.0.0.1:8222/varz 2>/dev/null || echo '{}')" + echo "$vmid $name $ip $st" + python3 -c " +import json,sys +try: + d=json.loads(sys.argv[1]) +except Exception: + print(' nats down') + raise SystemExit +print(' server_name', d.get('server_name'), 'cluster', (d.get('cluster') or {}).get('name'), 'routes', len((d.get('cluster') or {}).get('urls') or d.get('connect_urls') or [])) +print(' jetstream', bool(d.get('jetstream')), 'port', d.get('port'), 'host', d.get('host')) +" "$js" 2>/dev/null || echo " nats down" +done +echo "host loopback still: $(ss -lnt | grep '127.0.0.1:4222' && echo up || echo down)" diff --git a/scripts/test.sh b/scripts/test.sh new file mode 100755 index 0000000..7060cb3 --- /dev/null +++ b/scripts/test.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# Local syntax check always. Live cluster check when pct is present. +set -euo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +bash -n "$ROOT/scripts/lib-ct.sh" +bash -n "$ROOT/scripts/create-cluster.sh" +bash -n "$ROOT/scripts/status.sh" +bash -n "$ROOT/scripts/bench.sh" +grep -q 'host: {{IP}}' "$ROOT/conf/nats.conf.tmpl" +grep -qv '0.0.0.0' "$ROOT/conf/nats.conf.tmpl" +if [[ ! -d /etc/pve/nodes ]]; then + echo "OK (syntax; not on Proxmox)" + exit 0 +fi +# shellcheck disable=SC1091 +. "$ROOT/cluster.env" +export PATH="/usr/sbin:/usr/bin:/bin:$PATH" +mapfile -t rows < <(printf '%s\n' "$NODES" | awk 'NF==3 {print}') +ready=0 +for row in "${rows[@]}"; do + # shellcheck disable=SC2086 + set -- $row + vmid=$1 name=$2 ip=$3 + js="$(sudo pct exec "$vmid" -- curl -fsS --max-time 3 http://127.0.0.1:8222/varz 2>/dev/null || true)" + echo "$js" | grep -q '"jetstream"' && ready=$((ready + 1)) || echo "not ready $name" +done +[[ $ready -eq 3 ]] || { echo "cluster not fully up ($ready/3)" >&2; exit 1; } + +# nats CLI on first node +first="$(echo "${rows[0]}" | awk '{print $1}')" +sudo pct exec "$first" -- bash -lc ' +set -e +export DEBIAN_FRONTEND=noninteractive +if [[ ! -x /usr/local/bin/nats ]]; then + apt-get install -y --no-install-recommends unzip >/dev/null + curl -fsSL https://github.com/nats-io/natscli/releases/download/v0.1.6/nats-0.1.6-linux-amd64.zip -o /tmp/natscli.zip + rm -rf /tmp/natscli && mkdir -p /tmp/natscli + unzip -o /tmp/natscli.zip -d /tmp/natscli >/dev/null + BIN=$(find /tmp/natscli /tmp -maxdepth 3 -type f -name nats | head -1) + test -n "$BIN" + install -m 0755 "$BIN" /usr/local/bin/nats +fi +IP=$(hostname -I | awk "{print \$1}") +export NATS_URL=nats://$IP:4222 +export PATH=/usr/local/bin:/usr/bin:/bin +nats stream rm VERAE_PX_TEST --force >/dev/null 2>&1 || true +nats stream add VERAE_PX_TEST --subjects="verae.px.test" --replicas=3 --storage=file --retention=limits --discard=old --max-msgs=-1 --max-bytes=-1 --max-age=1h --dupe-window=2m --defaults +nats pub verae.px.test cluster-ok +nats stream info VERAE_PX_TEST +' +echo "OK live cluster (3/3 + replicas=3 stream)" diff --git a/systemd/nats-server.service b/systemd/nats-server.service new file mode 100644 index 0000000..86e5076 --- /dev/null +++ b/systemd/nats-server.service @@ -0,0 +1,16 @@ +[Unit] +Description=NATS JetStream (Verae cluster node) +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +User=nats +Group=nats +ExecStart=/usr/local/bin/nats-server -c /etc/nats/nats.conf +LimitNOFILE=65536 +Restart=on-failure +RestartSec=2 + +[Install] +WantedBy=multi-user.target