Add a NATS cluster message-speed bench (throughput and delay).

Client LXC 510 against nats-a/b/c at several core and JetStream r=3 loads.

packages/verae-nats-cluster/BENCH.md

@@ -0,0 +1,49 @@
+# NATS cluster message speed
+
+Run **`20260912T045131Z`** (UTC). Client: LXC **510** `verae-px-worker` (`10.10.10.20`), not a nats-* server. Servers: `nats-a/b/c` on `10.10.10.21–23` (`vmbr1` only).
+
+Client URL:
+
+```text
+nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+```
+
+## Method
+
+- **Core NATS** is fire-and-forget pub/sub (`nats bench`). No disk, no replica ack.
+- **JetStream** uses **file** storage and **replicas=3** (same as product streams). The unique stream `benchstream` is deleted between JS loads.
+- Throughput is **msgs/sec** from nats CLI **0.1.6** (`--no-progress --csv`). Its min/avg/max are publisher **rate spread**, not delay.
+- **Ping** delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster.
+- **Flood** delay: N publishers dump the whole batch, then the subscriber drains. This is **queueing under burst**, not wire RTT.
+- Probe: `scripts/latency.mjs` (two connections, header timestamp).
+
+## Throughput
+
+| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
+|-----|------|------------------|------------|----------|------------|----------|
+| `core-1p1s-50k-128` | core pub/sub | 1,200,836 | 791,094 | 96.57 | 747,461 | 91.24 |
+| `core-4p4s-100k-128` | core pub/sub | 1,521,256 | 316,312 | 38.61 | 1,299,634 | 158.65 |
+| `core-4p4s-50k-1k` | core pub/sub | 630,460 | 247,747 | 241.94 | 510,216 | 498.26 |
+| `core-8p8s-200k-128` | core pub/sub | 2,007,937 | 333,957 | 40.77 | 1,790,736 | 218.60 |
+| `js-1p-20k-128-r3` | jetstream r=3 file | — | 16,155 | 1.97 | — | — |
+| `js-2p2s-20k-128-r3` | jetstream r=3 file | 21,861 | 10,965 | 1.34 | 10,942 | 1.34 |
+| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 13,493 | 13.18 | — | — |
+| `js-4p-50k-128-r3` | jetstream r=3 file | — | 16,607 | 2.03 | — | — |
+
+## Round-trip delay
+
+| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
+|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|
+| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.254ms | 0.307ms | 0.286ms | 0.332ms | 0.734ms | 2.763ms |
+| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 149.279ms | 238.626ms | 248.752ms | 274.314ms | 279.398ms | 279.740ms |
+| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 155.101ms | 211.706ms | 217.579ms | 223.268ms | 227.798ms | 228.370ms |
+| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 174.174ms | 263.186ms | 266.672ms | 299.073ms | 304.233ms | 304.475ms |
+| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 304.586ms | 453.749ms | 466.296ms | 499.924ms | 505.112ms | 505.620ms |
+
+## What the numbers mean
+
+Product job/event/archive traffic is **JetStream r=3 file**. On this three-LXC stand that is about **16k durable 128 B pubs/s** (about **13k** at 1 KiB). Core NATS is an upper bound for non-durable fan-out: about **0.7–2.0M msgs/s** aggregate at 128 B, or **~630k msgs/s (~616 MB/s)** at 1 KiB with 4 publishers.
+
+A quiet request-reply is **~0.3 ms** average, **p99 < 1 ms**. Flood rows in the **150–500 ms** band are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers.
+
+Re-run on NS1: `bash scripts/bench.sh`. Raw logs/CSVs are under `results/<utc>/`.

packages/verae-nats-cluster/NATS.md

@@ -3,3 +3,5 @@
 This repo **is** the three-node JetStream cluster. Listeners are the guest private IPs on `vmbr1` (`10.10.10.21–23:4222` and `:6222`). HTTP monitor is loopback `:8222` inside each guest.
 
 Zapier cloud, browsers, and `vmbr0` never get a NATS socket.
+
+Throughput and delay at several loads: [BENCH.md](BENCH.md).

packages/verae-nats-cluster/README.md

@@ -41,4 +41,27 @@ bash scripts/ensure-streams.sh
 
 nkeys/mTLS: `verae-nats-accounts` still has the INTERNAL/LEAF sketch. Do **not** enable accounts on this cluster until every client passes credentials in `NATS_URL`. Private `vmbr1` is the current isolation.
 
+## Message speed (throughput + delay)
+
+From a **client that is not a nats-* server** (LXC 510 `verae-px-worker`):
+
+```bash
+# on NS1
+bash scripts/bench.sh
+```
+
+That writes `results/<utc>/` and copies a summary to [BENCH.md](BENCH.md). Core NATS is fire-and-forget; JetStream rows use **file** storage and **replicas=3**. Ping delay is sequential RTT; flood delay is queueing under burst.
+
+Latest run (`20260912T045131Z`):
+
+| Kind | Load | Result |
+|------|------|--------|
+| Core 1p1s 50k×128 B | pub/sub | ~1.20M msgs/s aggregate |
+| Core 8p8s 200k×128 B | pub/sub | ~2.01M msgs/s aggregate |
+| Core 4p4s 50k×1 KiB | pub/sub | ~630k msgs/s (~616 MB/s aggregate) |
+| JetStream r=3 file | 1–4 pubs, 128 B | ~16k durable pubs/s |
+| JetStream r=3 file | 4 pubs, 1 KiB | ~13.5k durable pubs/s |
+| Ping 1k×128 B | sequential RTT | avg 0.307 ms, p99 0.734 ms |
+| Flood 8p 20k×128 B | burst queueing | avg 454 ms |
+
 Hardware move: same three configs, three boxes, private NIC only — change IPs in `cluster.env` and `conf/nats.conf.tmpl`.

packages/verae-nats-cluster/SUMMARY.md

@@ -1,3 +1,5 @@
 # verae-nats-cluster
 
 Proxmox LXC 511–513 (`nats-a/b/c`) form a JetStream cluster on `vmbr1` for HA testing before dedicated hardware.
+
+Message speed (client LXC 510 vs the three nodes): see [BENCH.md](BENCH.md). Product streams are JetStream **file + replicas=3** (~16k durable 128 B pubs/s on this stand). Core NATS is ~1–2M msgs/s at 128 B. Sequential RTT ~0.3 ms; flood queueing is hundreds of ms.

packages/verae-nats-cluster/package.json

@@ -6,6 +6,7 @@
   "scripts": {
     "create": "bash scripts/create-cluster.sh",
     "test": "bash scripts/test.sh",
-    "status": "bash scripts/status.sh"
+    "status": "bash scripts/status.sh",
+    "bench": "bash scripts/bench.sh"
   }
 }

packages/verae-nats-cluster/results/.gitignore

@@ -0,0 +1,4 @@
+*
+!.gitignore
+!20260912T045131Z/
+!20260912T045131Z/**

packages/verae-nats-cluster/results/20260912T045131Z/BENCH.md

@@ -0,0 +1,49 @@
+# NATS cluster message speed
+
+Run **`20260912T045131Z`** (UTC). Client: LXC **510** `verae-px-worker` (`10.10.10.20`), not a nats-* server. Servers: `nats-a/b/c` on `10.10.10.21–23` (`vmbr1` only).
+
+Client URL:
+
+```text
+nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+```
+
+## Method
+
+- **Core NATS** is fire-and-forget pub/sub (`nats bench`). No disk, no replica ack.
+- **JetStream** uses **file** storage and **replicas=3** (same as product streams). The unique stream `benchstream` is deleted between JS loads.
+- Throughput is **msgs/sec** from nats CLI **0.1.6** (`--no-progress --csv`). Its min/avg/max are publisher **rate spread**, not delay.
+- **Ping** delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster.
+- **Flood** delay: N publishers dump the whole batch, then the subscriber drains. This is **queueing under burst**, not wire RTT.
+- Probe: `scripts/latency.mjs` (two connections, header timestamp).
+
+## Throughput
+
+| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
+|-----|------|------------------|------------|----------|------------|----------|
+| `core-1p1s-50k-128` | core pub/sub | 1,200,836 | 791,094 | 96.57 | 747,461 | 91.24 |
+| `core-4p4s-100k-128` | core pub/sub | 1,521,256 | 316,312 | 38.61 | 1,299,634 | 158.65 |
+| `core-4p4s-50k-1k` | core pub/sub | 630,460 | 247,747 | 241.94 | 510,216 | 498.26 |
+| `core-8p8s-200k-128` | core pub/sub | 2,007,937 | 333,957 | 40.77 | 1,790,736 | 218.60 |
+| `js-1p-20k-128-r3` | jetstream r=3 file | — | 16,155 | 1.97 | — | — |
+| `js-2p2s-20k-128-r3` | jetstream r=3 file | 21,861 | 10,965 | 1.34 | 10,942 | 1.34 |
+| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 13,493 | 13.18 | — | — |
+| `js-4p-50k-128-r3` | jetstream r=3 file | — | 16,607 | 2.03 | — | — |
+
+## Round-trip delay
+
+| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
+|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|
+| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.254ms | 0.307ms | 0.286ms | 0.332ms | 0.734ms | 2.763ms |
+| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 149.279ms | 238.626ms | 248.752ms | 274.314ms | 279.398ms | 279.740ms |
+| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 155.101ms | 211.706ms | 217.579ms | 223.268ms | 227.798ms | 228.370ms |
+| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 174.174ms | 263.186ms | 266.672ms | 299.073ms | 304.233ms | 304.475ms |
+| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 304.586ms | 453.749ms | 466.296ms | 499.924ms | 505.112ms | 505.620ms |
+
+## What the numbers mean
+
+Product job/event/archive traffic is **JetStream r=3 file**. On this three-LXC stand that is about **16k durable 128 B pubs/s** (about **13k** at 1 KiB). Core NATS is an upper bound for non-durable fan-out: about **0.7–2.0M msgs/s** aggregate at 128 B, or **~630k msgs/s (~616 MB/s)** at 1 KiB with 4 publishers.
+
+A quiet request-reply is **~0.3 ms** average, **p99 < 1 ms**. Flood rows in the **150–500 ms** band are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers.
+
+Re-run on NS1: `bash scripts/bench.sh`. Raw logs/CSVs are under `results/<utc>/`.

packages/verae-nats-cluster/results/20260912T045131Z/core-1p1s-50k-128.csv

@@ -0,0 +1,3 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+8e1ExxrZ9OtHTkwXsOfzes,S0,50000,6400000,747461,95675134.125326,0.066893
+8e1ExxrZ9OtHTkwXsOfzes,P0,50000,6400000,791094,101260059.613379,0.063204

packages/verae-nats-cluster/results/20260912T045131Z/core-1p1s-50k-128.txt

@@ -0,0 +1,7 @@
+=== core-1p1s-50k-128 ===
+
+NATS Pub/Sub stats: 1,200,836 msgs/sec ~ 146.59 MB/sec
+ Pub stats: 791,094 msgs/sec ~ 96.57 MB/sec
+ Sub stats: 747,461 msgs/sec ~ 91.24 MB/sec
+
+Saved metric data in csv file /tmp/bench.csv

packages/verae-nats-cluster/results/20260912T045131Z/core-4p4s-100k-128.csv

@@ -0,0 +1,9 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+d1csC8CbCBLYlUMAUVZxpb,S0,100000,12800000,377317,48296660.364795,0.265029
+d1csC8CbCBLYlUMAUVZxpb,S1,100000,12800000,351445,44985056.104940,0.284539
+d1csC8CbCBLYlUMAUVZxpb,S2,100000,12800000,372973,47740617.289389,0.268116
+d1csC8CbCBLYlUMAUVZxpb,S3,100000,12800000,324908,41588310.877432,0.307779
+d1csC8CbCBLYlUMAUVZxpb,P0,25000,3200000,524453,67129988.054428,0.047669
+d1csC8CbCBLYlUMAUVZxpb,P1,25000,3200000,324337,41515233.366208,0.077080
+d1csC8CbCBLYlUMAUVZxpb,P2,25000,3200000,103170,13205778.771206,0.242318
+d1csC8CbCBLYlUMAUVZxpb,P3,25000,3200000,79260,10145297.784464,0.315417

packages/verae-nats-cluster/results/20260912T045131Z/core-4p4s-100k-128.txt

@@ -0,0 +1,17 @@
+=== core-4p4s-100k-128 ===
+
+NATS Pub/Sub stats: 1,521,256 msgs/sec ~ 185.70 MB/sec
+ Pub stats: 316,312 msgs/sec ~ 38.61 MB/sec
+  [1] 524,453 msgs/sec ~ 64.02 MB/sec (25000 msgs)
+  [2] 324,337 msgs/sec ~ 39.59 MB/sec (25000 msgs)
+  [3] 103,170 msgs/sec ~ 12.59 MB/sec (25000 msgs)
+  [4] 79,260 msgs/sec ~ 9.68 MB/sec (25000 msgs)
+  min 79,260 | avg 257,805 | max 524,453 | stddev 181,189 msgs
+ Sub stats: 1,299,634 msgs/sec ~ 158.65 MB/sec
+  [1] 377,317 msgs/sec ~ 46.06 MB/sec (100000 msgs)
+  [2] 351,445 msgs/sec ~ 42.90 MB/sec (100000 msgs)
+  [3] 372,973 msgs/sec ~ 45.53 MB/sec (100000 msgs)
+  [4] 324,908 msgs/sec ~ 39.66 MB/sec (100000 msgs)
+  min 324,908 | avg 356,660 | max 377,317 | stddev 20,785 msgs
+
+Saved metric data in csv file /tmp/bench.csv

packages/verae-nats-cluster/results/20260912T045131Z/core-4p4s-50k-1k.csv

@@ -0,0 +1,9 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+LXVQ7kvd26BIwiMRRfbr7Z,S0,50000,51200000,169213,173274762.582948,0.295484
+LXVQ7kvd26BIwiMRRfbr7Z,S1,50000,51200000,150273,153880103.181538,0.332727
+LXVQ7kvd26BIwiMRRfbr7Z,S2,50000,51200000,142138,145550108.197459,0.351769
+LXVQ7kvd26BIwiMRRfbr7Z,S3,50000,51200000,132134,135306220.651698,0.378401
+LXVQ7kvd26BIwiMRRfbr7Z,P0,12500,12800000,176262,180492467.210335,0.070917
+LXVQ7kvd26BIwiMRRfbr7Z,P1,12500,12800000,132211,135384883.941361,0.094545
+LXVQ7kvd26BIwiMRRfbr7Z,P2,12500,12800000,70678,72375041.084851,0.176857
+LXVQ7kvd26BIwiMRRfbr7Z,P3,12500,12800000,61936,63423297.033934,0.201819

packages/verae-nats-cluster/results/20260912T045131Z/core-4p4s-50k-1k.txt

@@ -0,0 +1,17 @@
+=== core-4p4s-50k-1k ===
+
+NATS Pub/Sub stats: 630,460 msgs/sec ~ 615.68 MB/sec
+ Pub stats: 247,747 msgs/sec ~ 241.94 MB/sec
+  [1] 176,262 msgs/sec ~ 172.13 MB/sec (12500 msgs)
+  [2] 132,211 msgs/sec ~ 129.11 MB/sec (12500 msgs)
+  [3] 70,678 msgs/sec ~ 69.02 MB/sec (12500 msgs)
+  [4] 61,936 msgs/sec ~ 60.49 MB/sec (12500 msgs)
+  min 61,936 | avg 110,271 | max 176,262 | stddev 46,744 msgs
+ Sub stats: 510,216 msgs/sec ~ 498.26 MB/sec
+  [1] 169,213 msgs/sec ~ 165.25 MB/sec (50000 msgs)
+  [2] 150,273 msgs/sec ~ 146.75 MB/sec (50000 msgs)
+  [3] 142,138 msgs/sec ~ 138.81 MB/sec (50000 msgs)
+  [4] 132,134 msgs/sec ~ 129.04 MB/sec (50000 msgs)
+  min 132,134 | avg 148,439 | max 169,213 | stddev 13,605 msgs
+
+Saved metric data in csv file /tmp/bench.csv

packages/verae-nats-cluster/results/20260912T045131Z/core-8p8s-200k-128.csv

@@ -0,0 +1,17 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+HuvulruPjjaqnoHWeFCH2I,S0,200000,25600000,349350,44716843.071789,0.572491
+HuvulruPjjaqnoHWeFCH2I,S1,200000,25600000,248162,31764858.924451,0.805922
+HuvulruPjjaqnoHWeFCH2I,S2,200000,25600000,244206,31258460.045739,0.818978
+HuvulruPjjaqnoHWeFCH2I,S3,200000,25600000,243405,31155913.818220,0.821674
+HuvulruPjjaqnoHWeFCH2I,S4,200000,25600000,230550,29510438.641581,0.867490
+HuvulruPjjaqnoHWeFCH2I,S5,200000,25600000,231495,29631384.363172,0.863949
+HuvulruPjjaqnoHWeFCH2I,S6,200000,25600000,229528,29379679.692357,0.871351
+HuvulruPjjaqnoHWeFCH2I,S7,200000,25600000,223842,28651789.197519,0.893487
+HuvulruPjjaqnoHWeFCH2I,P0,25000,3200000,152536,19524693.902374,0.163895
+HuvulruPjjaqnoHWeFCH2I,P1,25000,3200000,84526,10819333.375159,0.295767
+HuvulruPjjaqnoHWeFCH2I,P2,25000,3200000,82017,10498223.303653,0.304813
+HuvulruPjjaqnoHWeFCH2I,P3,25000,3200000,57214,7323427.896669,0.436954
+HuvulruPjjaqnoHWeFCH2I,P4,25000,3200000,53879,6896566.453658,0.463999
+HuvulruPjjaqnoHWeFCH2I,P5,25000,3200000,51420,6581852.801766,0.486185
+HuvulruPjjaqnoHWeFCH2I,P6,25000,3200000,48572,6217303.262856,0.514693
+HuvulruPjjaqnoHWeFCH2I,P7,25000,3200000,41744,5343333.685114,0.598877

packages/verae-nats-cluster/results/20260912T045131Z/core-8p8s-200k-128.txt

@@ -0,0 +1,25 @@
+=== core-8p8s-200k-128 ===
+
+NATS Pub/Sub stats: 2,007,937 msgs/sec ~ 245.11 MB/sec
+ Pub stats: 333,957 msgs/sec ~ 40.77 MB/sec
+  [1] 152,536 msgs/sec ~ 18.62 MB/sec (25000 msgs)
+  [2] 84,526 msgs/sec ~ 10.32 MB/sec (25000 msgs)
+  [3] 82,017 msgs/sec ~ 10.01 MB/sec (25000 msgs)
+  [4] 57,214 msgs/sec ~ 6.98 MB/sec (25000 msgs)
+  [5] 53,879 msgs/sec ~ 6.58 MB/sec (25000 msgs)
+  [6] 51,420 msgs/sec ~ 6.28 MB/sec (25000 msgs)
+  [7] 48,572 msgs/sec ~ 5.93 MB/sec (25000 msgs)
+  [8] 41,744 msgs/sec ~ 5.10 MB/sec (25000 msgs)
+  min 41,744 | avg 71,488 | max 152,536 | stddev 33,867 msgs
+ Sub stats: 1,790,736 msgs/sec ~ 218.60 MB/sec
+  [1] 349,350 msgs/sec ~ 42.65 MB/sec (200000 msgs)
+  [2] 248,162 msgs/sec ~ 30.29 MB/sec (200000 msgs)
+  [3] 244,206 msgs/sec ~ 29.81 MB/sec (200000 msgs)
+  [4] 243,405 msgs/sec ~ 29.71 MB/sec (200000 msgs)
+  [5] 230,550 msgs/sec ~ 28.14 MB/sec (200000 msgs)
+  [6] 231,495 msgs/sec ~ 28.26 MB/sec (200000 msgs)
+  [7] 229,528 msgs/sec ~ 28.02 MB/sec (200000 msgs)
+  [8] 223,842 msgs/sec ~ 27.32 MB/sec (200000 msgs)
+  min 223,842 | avg 250,067 | max 349,350 | stddev 38,364 msgs
+
+Saved metric data in csv file /tmp/bench.csv

packages/verae-nats-cluster/results/20260912T045131Z/js-1p-20k-128-r3.csv

@@ -0,0 +1,2 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+S0xQOJuOM4PlwAHoh5Bj7f,P0,40000,2560000,16155,2067883.858001,1.237981

packages/verae-nats-cluster/results/20260912T045131Z/js-1p-20k-128-r3.txt

@@ -0,0 +1,5 @@
+=== js-1p-20k-128-r3 ===
+
+Pub stats: 16,155 msgs/sec ~ 1.97 MB/sec
+
+Saved metric data in csv file /tmp/bench.csv

packages/verae-nats-cluster/results/20260912T045131Z/js-2p2s-20k-128-r3.csv

@@ -0,0 +1,5 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+smN9zLbWAAZER4FjSmfwcO,S0,20126,1280000,5474,700679.773679,1.826797
+smN9zLbWAAZER4FjSmfwcO,S1,20130,1280000,5471,700313.845376,1.827752
+smN9zLbWAAZER4FjSmfwcO,P0,20000,1280000,8678,1110910.588857,1.152208
+smN9zLbWAAZER4FjSmfwcO,P1,20000,1280000,5485,702119.773868,1.823051

packages/verae-nats-cluster/results/20260912T045131Z/js-2p2s-20k-128-r3.txt

@@ -0,0 +1,13 @@
+=== js-2p2s-20k-128-r3 ===
+
+NATS Pub/Sub stats: 21,861 msgs/sec ~ 2.67 MB/sec
+ Pub stats: 10,965 msgs/sec ~ 1.34 MB/sec
+  [1] 8,678 msgs/sec ~ 1.06 MB/sec (10000 msgs)
+  [2] 5,485 msgs/sec ~ 685.66 KB/sec (10000 msgs)
+  min 5,485 | avg 7,081 | max 8,678 | stddev 1,596 msgs
+ Sub stats: 10,942 msgs/sec ~ 1.34 MB/sec
+  [1] 5,474 msgs/sec ~ 684.26 KB/sec (10000 msgs)
+  [2] 5,471 msgs/sec ~ 683.90 KB/sec (10000 msgs)
+  min 5,471 | avg 5,472 | max 5,474 | stddev 1 msgs
+
+Saved metric data in csv file /tmp/bench.csv

packages/verae-nats-cluster/results/20260912T045131Z/js-4p-20k-1k-r3.csv

@@ -0,0 +1,5 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+zOcPILhowCWfp2PGmHUgqx,P0,10000,5120000,5121,5244574.853257,0.976247
+zOcPILhowCWfp2PGmHUgqx,P1,10000,5120000,4281,4383926.080211,1.167903
+zOcPILhowCWfp2PGmHUgqx,P2,10000,5120000,3480,3564282.061583,1.436474
+zOcPILhowCWfp2PGmHUgqx,P3,10000,5120000,3373,3454449.904903,1.482146

packages/verae-nats-cluster/results/20260912T045131Z/js-4p-20k-1k-r3.txt

@@ -0,0 +1,10 @@
+=== js-4p-20k-1k-r3 ===
+
+Pub stats: 13,493 msgs/sec ~ 13.18 MB/sec
+ [1] 5,121 msgs/sec ~ 5.00 MB/sec (5000 msgs)
+ [2] 4,281 msgs/sec ~ 4.18 MB/sec (5000 msgs)
+ [3] 3,480 msgs/sec ~ 3.40 MB/sec (5000 msgs)
+ [4] 3,373 msgs/sec ~ 3.29 MB/sec (5000 msgs)
+ min 3,373 | avg 4,063 | max 5,121 | stddev 704 msgs
+
+Saved metric data in csv file /tmp/bench.csv

packages/verae-nats-cluster/results/20260912T045131Z/js-4p-50k-128-r3.csv

@@ -0,0 +1,5 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+vZIFpR751J0vbDTJQDYGTN,P0,25000,1600000,6628,848387.283433,1.885931
+vZIFpR751J0vbDTJQDYGTN,P1,25000,1600000,5762,737571.570534,2.169281
+vZIFpR751J0vbDTJQDYGTN,P2,25000,1600000,4162,532825.280344,3.002861
+vZIFpR751J0vbDTJQDYGTN,P3,25000,1600000,4154,531740.716359,3.008985

packages/verae-nats-cluster/results/20260912T045131Z/js-4p-50k-128-r3.txt

@@ -0,0 +1,10 @@
+=== js-4p-50k-128-r3 ===
+
+Pub stats: 16,607 msgs/sec ~ 2.03 MB/sec
+ [1] 6,628 msgs/sec ~ 828.50 KB/sec (12500 msgs)
+ [2] 5,762 msgs/sec ~ 720.28 KB/sec (12500 msgs)
+ [3] 4,162 msgs/sec ~ 520.34 KB/sec (12500 msgs)
+ [4] 4,154 msgs/sec ~ 519.28 KB/sec (12500 msgs)
+ min 4,154 | avg 5,176 | max 6,628 | stddev 1,063 msgs
+
+Saved metric data in csv file /tmp/bench.csv

packages/verae-nats-cluster/results/20260912T045131Z/lat-1p-5k-128.txt

@@ -0,0 +1,2 @@
+=== lat-1p-5k-128 ===
+{"count":5000,"pubs":1,"size":128,"min_us":149279,"avg_us":238626,"p50_us":248752,"p90_us":274314,"p99_us":279398,"max_us":279740,"min":"149.279ms","avg":"238.626ms","p50":"248.752ms","p90":"274.314ms","p99":"279.398ms","max":"279.740ms"}

packages/verae-nats-cluster/results/20260912T045131Z/lat-4p-10k-128.txt

@@ -0,0 +1,2 @@
+=== lat-4p-10k-128 ===
+{"count":10000,"pubs":4,"size":128,"min_us":174174,"avg_us":263186,"p50_us":266672,"p90_us":299073,"p99_us":304233,"max_us":304475,"min":"174.174ms","avg":"263.186ms","p50":"266.672ms","p90":"299.073ms","p99":"304.233ms","max":"304.475ms"}

packages/verae-nats-cluster/results/20260912T045131Z/lat-4p-5k-1k.txt

@@ -0,0 +1,2 @@
+=== lat-4p-5k-1k ===
+{"count":5000,"pubs":4,"size":1024,"min_us":155101,"avg_us":211706,"p50_us":217579,"p90_us":223268,"p99_us":227798,"max_us":228370,"min":"155.101ms","avg":"211.706ms","p50":"217.579ms","p90":"223.268ms","p99":"227.798ms","max":"228.370ms"}

packages/verae-nats-cluster/results/20260912T045131Z/lat-8p-20k-128.txt

@@ -0,0 +1,2 @@
+=== lat-8p-20k-128 ===
+{"count":20000,"pubs":8,"size":128,"min_us":304586,"avg_us":453749,"p50_us":466296,"p90_us":499924,"p99_us":505112,"max_us":505620,"min":"304.586ms","avg":"453.749ms","p50":"466.296ms","p90":"499.924ms","p99":"505.112ms","max":"505.620ms"}

packages/verae-nats-cluster/results/20260912T045131Z/lat-ping-1k-128.txt

@@ -0,0 +1,2 @@
+=== lat-ping-1k-128 ===
+{"count":1000,"pubs":1,"size":128,"min_us":254,"avg_us":307,"p50_us":286,"p90_us":332,"p99_us":734,"max_us":2763,"min":"0.254ms","avg":"0.307ms","p50":"0.286ms","p90":"0.332ms","p99":"0.734ms","max":"2.763ms"}

packages/verae-nats-cluster/scripts/bench-report.py

@@ -0,0 +1,176 @@
+#!/usr/bin/env python3
+"""Turn nats bench text logs + latency JSON into markdown."""
+from __future__ import annotations
+
+import json
+import re
+import sys
+from pathlib import Path
+
+
+def fmt_int(s: str | None) -> str:
+    if not s:
+        return "—"
+    return f"{int(s):,}"
+
+
+def parse_bench(text: str) -> dict[str, str]:
+    out: dict[str, str] = {"kind": "throughput"}
+    m = re.search(r"(?m)^\s*Pub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*MB/sec", text)
+    if m:
+        out["pub_msgs"] = m.group(1).replace(",", "")
+        out["pub_mb"] = m.group(2)
+    m = re.search(r"(?m)^\s*Sub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*MB/sec", text)
+    if m:
+        out["sub_msgs"] = m.group(1).replace(",", "")
+        out["sub_mb"] = m.group(2)
+    m = re.search(r"NATS Pub/Sub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*MB/sec", text)
+    if m:
+        out["agg_msgs"] = m.group(1).replace(",", "")
+        out["agg_mb"] = m.group(2)
+    if "JetStream" in text or "--js" in text or "js-" in text:
+        out["mode"] = "jetstream r=3 file"
+    else:
+        out["mode"] = "core pub/sub"
+    # nats 0.1.6 prints min/avg/max as msgs/sec across publishers, not µs delay
+    m = re.search(
+        r"min\s+([0-9,]+)\s*\|\s*avg\s+([0-9,]+)\s*\|\s*max\s+([0-9,]+)\s*\|\s*stddev\s+([0-9,]+)\s*msgs",
+        text,
+    )
+    if m:
+        out["pub_spread"] = f"{m.group(1)}–{m.group(3)} (avg {m.group(2)})"
+    return out
+
+
+def parse_lat(text: str) -> dict[str, str] | None:
+    for line in text.splitlines():
+        line = line.strip()
+        if line.startswith("{") and "p99_us" in line:
+            d = json.loads(line)
+            mode = d.get("mode") or ""
+            return {
+                "kind": "latency",
+                "count": str(d.get("count", "")),
+                "pubs": str(d.get("pubs", "")),
+                "size": str(d.get("size", "")),
+                "mode": str(mode),
+                "min": d.get("min", ""),
+                "avg": d.get("avg", ""),
+                "p50": d.get("p50", ""),
+                "p90": d.get("p90", ""),
+                "p99": d.get("p99", ""),
+                "max": d.get("max", ""),
+            }
+    return None
+
+
+def lat_mode(run: str, recorded: str) -> str:
+    if recorded in ("ping", "flood"):
+        return recorded
+    if "ping" in run:
+        return "ping"
+    return "flood"
+
+
+def thru_sort(p: dict[str, str]) -> tuple:
+    return (0 if p.get("mode", "").startswith("core") else 1, p.get("run", ""))
+
+
+def lat_sort(p: dict[str, str]) -> tuple:
+    mode = lat_mode(p.get("run", ""), p.get("mode", ""))
+    return (0 if mode == "ping" else 1, int(p.get("count") or 0), p.get("run", ""))
+
+
+def main() -> int:
+    folder = Path(sys.argv[1] if len(sys.argv) > 1 else ".")
+    thru: list[dict[str, str]] = []
+    lats: list[dict[str, str]] = []
+    for f in sorted(folder.glob("*.txt")):
+        text = f.read_text(encoding="utf-8", errors="replace")
+        lat = parse_lat(text)
+        if lat:
+            lat["run"] = f.stem
+            lats.append(lat)
+            continue
+        p = parse_bench(text)
+        if p.get("pub_msgs") or p.get("agg_msgs"):
+            p["run"] = f.stem
+            thru.append(p)
+
+    stamp = folder.name if re.fullmatch(r"\d{8}T\d{6}Z", folder.name) else ""
+
+    print("# NATS cluster message speed")
+    print()
+    if stamp:
+        print(f"Run **`{stamp}`** (UTC). ", end="")
+    print(
+        "Client: LXC **510** `verae-px-worker` (`10.10.10.20`), not a nats-* server. "
+        "Servers: `nats-a/b/c` on `10.10.10.21–23` (`vmbr1` only)."
+    )
+    print()
+    print("Client URL:")
+    print()
+    print("```text")
+    print("nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222")
+    print("```")
+    print()
+    print("## Method")
+    print()
+    print("- **Core NATS** is fire-and-forget pub/sub (`nats bench`). No disk, no replica ack.")
+    print(
+        "- **JetStream** uses **file** storage and **replicas=3** (same as product streams). "
+        "The unique stream `benchstream` is deleted between JS loads."
+    )
+    print("- Throughput is **msgs/sec** from nats CLI **0.1.6** (`--no-progress --csv`). Its min/avg/max are publisher **rate spread**, not delay.")
+    print(
+        "- **Ping** delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster."
+    )
+    print(
+        "- **Flood** delay: N publishers dump the whole batch, then the subscriber drains. "
+        "This is **queueing under burst**, not wire RTT."
+    )
+    print("- Probe: `scripts/latency.mjs` (two connections, header timestamp).")
+    print()
+    print("## Throughput")
+    print()
+    print("| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |")
+    print("|-----|------|------------------|------------|----------|------------|----------|")
+    for p in sorted(thru, key=thru_sort):
+        print(
+            f"| `{p['run']}` | {p.get('mode', '')} | {fmt_int(p.get('agg_msgs'))} | "
+            f"{fmt_int(p.get('pub_msgs'))} | {p.get('pub_mb') or '—'} | "
+            f"{fmt_int(p.get('sub_msgs'))} | {p.get('sub_mb') or '—'} |"
+        )
+    print()
+    print("## Round-trip delay")
+    print()
+    print("| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |")
+    print("|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|")
+    for p in sorted(lats, key=lat_sort):
+        kind = lat_mode(p.get("run", ""), p.get("mode", ""))
+        label = "ping (sequential RTT)" if kind == "ping" else "flood (burst queueing)"
+        print(
+            f"| `{p['run']}` | {label} | {p.get('count', '')} | {p.get('pubs', '')} | "
+            f"{p.get('size', '')} B | {p.get('min', '')} | {p.get('avg', '')} | "
+            f"{p.get('p50', '')} | {p.get('p90', '')} | {p.get('p99', '')} | {p.get('max', '')} |"
+        )
+    print()
+    print("## What the numbers mean")
+    print()
+    print(
+        "Product job/event/archive traffic is **JetStream r=3 file**. On this three-LXC stand that is about "
+        "**16k durable 128 B pubs/s** (about **13k** at 1 KiB). Core NATS is an upper bound for "
+        "non-durable fan-out: about **0.7–2.0M msgs/s** aggregate at 128 B, or **~630k msgs/s (~616 MB/s)** at 1 KiB with 4 publishers."
+    )
+    print()
+    print(
+        "A quiet request-reply is **~0.3 ms** average, **p99 < 1 ms**. Flood rows in the **150–500 ms** band "
+        "are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers."
+    )
+    print()
+    print("Re-run on NS1: `bash scripts/bench.sh`. Raw logs/CSVs are under `results/<utc>/`.")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

packages/verae-nats-cluster/scripts/bench.sh

@@ -0,0 +1,106 @@
+#!/usr/bin/env bash
+# Message throughput and delay ladder against the 3-node vmbr1 cluster.
+# Prefers a client that is not a nats-* server (px-worker LXC 510).
+set -euo pipefail
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+# shellcheck disable=SC1091
+. "$ROOT/client.env"
+export PATH="/usr/sbin:/usr/bin:/bin:/usr/local/bin:$PATH"
+STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
+OUT="${BENCH_OUT:-$ROOT/results/$STAMP}"
+CLIENT_VMID="${CLIENT_VMID:-510}"
+mkdir -p "$OUT"
+
+ensure_nats_cli() {
+  local vmid="$1"
+  sudo pct exec "$vmid" -- bash -lc '
+set -e
+export DEBIAN_FRONTEND=noninteractive
+export PATH=/usr/local/bin:/usr/bin:/bin
+if [[ ! -x /usr/local/bin/nats ]]; then
+  apt-get install -y --no-install-recommends unzip curl ca-certificates >/dev/null
+  curl -fsSL https://github.com/nats-io/natscli/releases/download/v0.1.6/nats-0.1.6-linux-amd64.zip -o /tmp/natscli.zip
+  rm -rf /tmp/natscli && mkdir -p /tmp/natscli
+  unzip -o /tmp/natscli.zip -d /tmp/natscli >/dev/null
+  BIN=$(find /tmp/natscli -type f -name nats | head -1)
+  install -m 0755 "$BIN" /usr/local/bin/nats
+fi
+nats --version
+'
+}
+
+run_one() {
+  local name="$1"
+  shift
+  echo "=== $name ===" | tee "$OUT/$name.txt"
+  # nats bench writes csv itself when --csv is a path inside the guest
+  sudo pct exec "$CLIENT_VMID" -- bash -lc "
+export PATH=/usr/local/bin:/usr/bin:/bin
+export NATS_URL='$NATS_URL'
+nats bench --no-progress --csv=/tmp/bench.csv $*
+" | tee -a "$OUT/$name.txt"
+  sudo pct exec "$CLIENT_VMID" -- cat /tmp/bench.csv >"$OUT/$name.csv" || true
+}
+
+echo "client LXC $CLIENT_VMID  NATS_URL=$NATS_URL  out=$OUT"
+ensure_nats_cli "$CLIENT_VMID"
+
+# Core NATS pub/sub — increasing publishers (same 128 B payload)
+run_one core-1p1s-50k-128   bench.core.a --pub 1 --sub 1 --msgs 50000 --size 128
+run_one core-4p4s-100k-128  bench.core.b --pub 4 --sub 4 --msgs 100000 --size 128
+run_one core-8p8s-200k-128  bench.core.c --pub 8 --sub 8 --msgs 200000 --size 128
+run_one core-4p4s-50k-1k    bench.core.d --pub 4 --sub 4 --msgs 50000 --size 1024
+
+js_rm() {
+  sudo pct exec "$CLIENT_VMID" -- bash -lc "
+export PATH=/usr/local/bin:/usr/bin:/bin
+export NATS_URL='$NATS_URL'
+nats stream rm benchstream --force >/dev/null 2>&1 || true
+"
+}
+# JetStream file store, replicas=3 (matches product streams)
+js_rm
+run_one js-1p-20k-128-r3    bench.js.a --js --purge --pub 1 --msgs 20000 --size 128 --replicas 3 --storage file --maxbytes=512MB --stream=benchstream
+js_rm
+run_one js-4p-50k-128-r3    bench.js.b --js --purge --pub 4 --msgs 50000 --size 128 --replicas 3 --storage file --maxbytes=512MB --stream=benchstream
+js_rm
+run_one js-4p-20k-1k-r3     bench.js.c --js --purge --pub 4 --msgs 20000 --size 1024 --replicas 3 --storage file --maxbytes=512MB --stream=benchstream
+js_rm
+run_one js-2p2s-20k-128-r3  bench.js.d --js --purge --pub 2 --sub 2 --msgs 20000 --size 128 --replicas 3 --storage file --maxbytes=512MB --pull --stream=benchstream
+js_rm
+
+# Round-trip delay (two connections, through the cluster) at several loads
+sudo pct exec "$CLIENT_VMID" -- bash -lc "
+set -e
+export PATH=/usr/local/bin:/usr/bin:/bin
+export NATS_URL='$NATS_URL'
+mkdir -p /tmp/nats-lat
+cd /tmp/nats-lat
+if [[ ! -d node_modules/nats ]]; then
+  npm init -y >/dev/null
+  npm install --no-audit --no-fund nats@2 >/dev/null
+fi
+" >/dev/null
+sudo pct push "$CLIENT_VMID" "$ROOT/scripts/latency.mjs" /tmp/nats-lat/latency.mjs || \
+  sudo pct exec "$CLIENT_VMID" -- bash -c 'cat > /tmp/nats-lat/latency.mjs' < "$ROOT/scripts/latency.mjs"
+lat() {
+  local name="$1" n="$2" sz="$3" p="$4" mode="${5:-flood}"
+  echo "=== $name ===" | tee "$OUT/$name.txt"
+  sudo pct exec "$CLIENT_VMID" -- bash -lc "
+export PATH=/usr/local/bin:/usr/bin:/bin
+export NATS_URL='$NATS_URL'
+cd /tmp/nats-lat
+node latency.mjs $n $sz $p $mode
+" | tee -a "$OUT/$name.txt"
+}
+# copy latest probe
+sudo pct exec "$CLIENT_VMID" -- bash -c 'cat > /tmp/nats-lat/latency.mjs' < "$ROOT/scripts/latency.mjs"
+lat lat-ping-1k-128 1000 128 1 ping
+lat lat-1p-5k-128 5000 128 1 flood
+lat lat-4p-10k-128 10000 128 4 flood
+lat lat-8p-20k-128 20000 128 8 flood
+lat lat-4p-5k-1k 5000 1024 4 flood
+
+python3 "$ROOT/scripts/bench-report.py" "$OUT" >"$OUT/BENCH.md"
+cp "$OUT/BENCH.md" "$ROOT/BENCH.md"
+echo "wrote $OUT/BENCH.md and $ROOT/BENCH.md"

packages/verae-nats-cluster/scripts/latency.mjs

@@ -0,0 +1,105 @@
+#!/usr/bin/env node
+/**
+ * Pub→sub round trip through the cluster (two connections).
+ * Usage: NATS_URL=... node latency.mjs [count] [payloadBytes] [publishers] [ping|flood]
+ * ping  = sequential publish-wait (one-message RTT)
+ * flood = publish the batch then drain (queueing under burst)
+ */
+import { connect, headers } from "nats";
+
+const url = process.env.NATS_URL || "nats://10.10.10.21:4222";
+const count = Number(process.argv[2] || 5000);
+const size = Number(process.argv[3] || 128);
+const pubs = Number(process.argv[4] || 1);
+const mode = process.argv[5] || "flood";
+const servers = url.split(",").map((s) => s.trim());
+const subject = `bench.lat.${process.pid}`;
+const payload = new Uint8Array(size);
+
+function pct(sorted, p) {
+  if (!sorted.length) return 0;
+  const i = Math.min(sorted.length - 1, Math.floor((p / 100) * sorted.length));
+  return sorted[i];
+}
+
+const samples = [];
+if (mode === "ping") {
+  const subNc = await connect({ servers, name: "lat-sub" });
+  const pubNc = await connect({ servers, name: "lat-pub" });
+  let resolveOne = null;
+  const sub = subNc.subscribe(subject, { max: count });
+  const consume = (async () => {
+    for await (const m of sub) {
+      const sent = Number(m.headers?.get("t") || 0);
+      samples.push(Number(process.hrtime.bigint() / 1000n) - sent);
+      resolveOne?.();
+    }
+  })();
+  await subNc.flush();
+  for (let i = 0; i < count; i++) {
+    const got = new Promise((r) => {
+      resolveOne = r;
+    });
+    const h = headers();
+    h.set("t", String(process.hrtime.bigint() / 1000n));
+    pubNc.publish(subject, payload, { headers: h });
+    await got;
+  }
+  await consume;
+  await pubNc.close();
+  await subNc.close();
+} else {
+  const subNc = await connect({ servers, name: "lat-sub" });
+  const sub = subNc.subscribe(subject, { max: count });
+  const done = (async () => {
+    for await (const m of sub) {
+      const sent = Number(m.headers?.get("t") || 0);
+      if (sent) samples.push(Number(process.hrtime.bigint() / 1000n) - sent);
+    }
+  })();
+  await subNc.flush();
+  const per = Math.ceil(count / pubs);
+  const publishers = [];
+  for (let p = 0; p < pubs; p++) {
+    publishers.push(
+      (async () => {
+        const nc = await connect({ servers, name: `lat-pub-${p}` });
+        const n = p === pubs - 1 ? count - per * (pubs - 1) : per;
+        for (let i = 0; i < n; i++) {
+          const h = headers();
+          h.set("t", String(process.hrtime.bigint() / 1000n));
+          nc.publish(subject, payload, { headers: h });
+        }
+        await nc.flush();
+        await nc.close();
+      })(),
+    );
+  }
+  await Promise.all(publishers);
+  await done;
+  await subNc.close();
+}
+
+samples.sort((a, b) => a - b);
+const sum = samples.reduce((a, b) => a + b, 0);
+const us = (n) => `${(n / 1000).toFixed(3)}ms`;
+console.log(
+  JSON.stringify({
+    count: samples.length,
+    pubs,
+    size,
+    mode,
+    min_us: samples[0],
+    avg_us: Math.round(sum / samples.length),
+    p50_us: pct(samples, 50),
+    p90_us: pct(samples, 90),
+    p99_us: pct(samples, 99),
+    max_us: samples[samples.length - 1],
+    min: us(samples[0]),
+    avg: us(sum / samples.length),
+    p50: us(pct(samples, 50)),
+    p90: us(pct(samples, 90)),
+    p99: us(pct(samples, 99)),
+    max: us(samples[samples.length - 1]),
+  }),
+);

packages/verae-nats-cluster/scripts/test.sh

@@ -5,6 +5,7 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd)"
 bash -n "$ROOT/scripts/lib-ct.sh"
 bash -n "$ROOT/scripts/create-cluster.sh"
 bash -n "$ROOT/scripts/status.sh"
+bash -n "$ROOT/scripts/bench.sh"
 grep -q 'host: {{IP}}' "$ROOT/conf/nats.conf.tmpl"
 grep -qv '0.0.0.0' "$ROOT/conf/nats.conf.tmpl"
 if [[ ! -d /etc/pve/nodes ]]; then

packages/zapier-decisions/STATUS.md

@@ -31,6 +31,22 @@ nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
 
 Repo: https://git.georgelambert.org/marchon/verae-nats-cluster
 
+### Cluster message speed (`20260912T045131Z`)
+
+Measured from LXC **510** (not a nats-* server) against `10.10.10.21–23`. Full tables: [BENCH.md](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md) / catalog [BENCH.pdf](https://zapier.georgelambert.org/packages/verae-nats-cluster/BENCH.pdf). Re-run: `bash scripts/bench.sh` on NS1.
+
+| Kind | Load | Result |
+|------|------|--------|
+| Core NATS | 1p1s 50k×128 B | ~1.20M msgs/s aggregate |
+| Core NATS | 8p8s 200k×128 B | ~2.01M msgs/s aggregate |
+| Core NATS | 4p4s 50k×1 KiB | ~630k msgs/s (~616 MB/s) |
+| JetStream file r=3 | 1–4 pubs, 128 B | ~16k durable pubs/s |
+| JetStream file r=3 | 4 pubs, 1 KiB | ~13.5k durable pubs/s |
+| Ping (sequential RTT) | 1k×128 B | avg 0.307 ms, p99 0.734 ms |
+| Flood (burst queueing) | 8p 20k×128 B | avg 454 ms |
+
+Product job/event/archive traffic is the JetStream row. Ping is one-message delay; flood is the mailbox catching up after a burst.
+
 ---
 
 ## Built and checked in (Forgejo, each its own repo)

scripts/build-docs-site.py

@@ -628,6 +628,8 @@ def main() -> None:
                 extras = extras + ("LOG.md", "TODO.md", "STATUS.md")
             if pkg == "verae-bootstrap":
                 extras = extras + ("HOST-DEPS.md",)
+            if pkg == "verae-nats-cluster":
+                extras = extras + ("BENCH.md",)
             for name in extras:
                 p = pkg_root / name
                 if p.exists():
@@ -851,6 +853,7 @@ def main() -> None:
         ("packages/verae-fleet/docs/CONSOLE.pdf", "Operator console"),
         ("packages/ui-docs/UI-REVIEW.pdf", "UI review (screenshots + live doors)"),
         ("packages/zapier-decisions/STATUS.pdf", "Current lab status (2026-09-12)"),
+        ("packages/verae-nats-cluster/BENCH.pdf", "NATS cluster throughput and delay (2026-09-12)"),
         ("overview/README.pdf", "System overview"),
         ("overview/INDEX.pdf", "Documentation index"),
         ("docs/modules-pdf/index.pdf", "Module PDFs (book)"),
@@ -863,6 +866,7 @@ def main() -> None:
         ("docs-master/CATALOG-HOST.html", "Move the catalog to another hostname"),
         ("docs-master/HOST-DEPS.html", "Host OS / Node / NATS installer"),
         ("packages/zapier-decisions/STATUS.html", "Current lab status"),
+        ("packages/verae-nats-cluster/BENCH.html", "NATS cluster throughput and delay"),
         ("docs-master/MESSAGE-FLOWS.html", "Numbered message flows"),
         ("docs-master/modules-and-nats.html", "NATS address table"),
         ("packages/verae-bootstrap/HOST-DEPS.html", "Host OS / Node / NATS installer"),