Snapshot of verae-nats-cluster (maximized NS1 NATS study)

results/20260912T053120Z/BENCH.md

@@ -0,0 +1,52 @@
+# NATS cluster message speed
+
+Run **`20260912T053120Z`** (UTC). Client: LXC **510** `verae-px-worker` (`10.10.10.20`), not a nats-* server. Servers: `nats-a/b/c` on `10.10.10.21–23` (`vmbr1` only).
+
+Client URL:
+
+```text
+nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+```
+
+## Method
+
+- **Core NATS** is fire-and-forget pub/sub (`nats bench`). No disk, no replica ack.
+- **JetStream** uses **file** storage and **replicas=3** (same as product streams). The unique stream `benchstream` is deleted between JS loads.
+- Throughput is **msgs/sec** from nats CLI **0.1.6** (`--no-progress --csv`). Its min/avg/max are publisher **rate spread**, not delay.
+- **Ping** delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster.
+- **Flood** delay: N publishers dump the whole batch, then the subscriber drains. This is **queueing under burst**, not wire RTT.
+- Probe: `scripts/latency.mjs` (two connections, header timestamp).
+
+## Throughput
+
+| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
+|-----|------|------------------|------------|----------|------------|----------|
+| `core-1p1s-50k-128` | core pub/sub | 810,988 | 599,004 | 73.12 | 456,116 | 55.68 |
+| `core-4p4s-100k-128` | core pub/sub | 1,361,921 | 379,346 | 46.31 | 1,150,536 | 140.45 |
+| `core-4p4s-50k-1k` | core pub/sub | 695,192 | 175,725 | 171.61 | 584,024 | 570.34 |
+| `core-8p8s-200k-128` | core pub/sub | 1,998,733 | 283,259 | 34.58 | 1,780,589 | 217.36 |
+| `js-1p-20k-128-r3` | jetstream r=3 file | — | 17,388 | 2.12 | — | — |
+| `js-2p2s-20k-128-r3` | jetstream r=3 file | 19,876 | 9,959 | 1.22 | 9,942 | 1.21 |
+| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 18,114 | 17.69 | — | — |
+| `js-4p-50k-128-r3` | jetstream r=3 file | — | 22,853 | 2.79 | — | — |
+| `js-mem-1p-20k-128-r3` | jetstream r=3 file | — | 22,153 | 2.70 | — | — |
+| `js-mem-4p-20k-1k-r3` | jetstream r=3 file | — | 28,685 | 28.01 | — | — |
+| `js-mem-4p-50k-128-r3` | jetstream r=3 file | — | 36,355 | 4.44 | — | — |
+
+## Round-trip delay
+
+| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
+|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|
+| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.250ms | 0.319ms | 0.299ms | 0.363ms | 0.684ms | 2.759ms |
+| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 94.920ms | 132.026ms | 131.990ms | 156.086ms | 158.860ms | 158.965ms |
+| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 113.281ms | 132.425ms | 131.457ms | 140.317ms | 144.683ms | 145.365ms |
+| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 153.318ms | 204.329ms | 205.865ms | 234.151ms | 237.346ms | 237.477ms |
+| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 233.727ms | 310.924ms | 311.170ms | 373.437ms | 400.764ms | 402.811ms |
+
+## What the numbers mean
+
+Product job/event/archive traffic is **JetStream r=3 file**. On this three-LXC stand that is about **16k durable 128 B pubs/s** (about **13k** at 1 KiB). Core NATS is an upper bound for non-durable fan-out: about **0.7–2.0M msgs/s** aggregate at 128 B, or **~630k msgs/s (~616 MB/s)** at 1 KiB with 4 publishers.
+
+A quiet request-reply is **~0.3 ms** average, **p99 < 1 ms**. Flood rows in the **150–500 ms** band are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers.
+
+Re-run on NS1: `bash scripts/bench.sh`. Raw logs/CSVs are under `results/<utc>/`.

results/20260912T053120Z/core-1p1s-50k-128.csv

@@ -0,0 +1,3 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+w2OZLH4YWp0DF7nX2fttF4,S0,50000,6400000,456116,58382861.156470,0.109621
+w2OZLH4YWp0DF7nX2fttF4,P0,50000,6400000,599004,76672615.503222,0.083472

results/20260912T053120Z/core-1p1s-50k-128.txt

@@ -0,0 +1,7 @@
+=== core-1p1s-50k-128 ===
+
+NATS Pub/Sub stats: 810,988 msgs/sec ~ 99.00 MB/sec
+ Pub stats: 599,004 msgs/sec ~ 73.12 MB/sec
+ Sub stats: 456,116 msgs/sec ~ 55.68 MB/sec
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T053120Z/core-4p4s-100k-128.csv

@@ -0,0 +1,9 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+objwjMbBXlRDUb07UVtRo4,S0,100000,12800000,292772,37474879.058831,0.341562
+objwjMbBXlRDUb07UVtRo4,S1,100000,12800000,290329,37162115.621242,0.344437
+objwjMbBXlRDUb07UVtRo4,S2,100000,12800000,290323,37161444.974130,0.344443
+objwjMbBXlRDUb07UVtRo4,S3,100000,12800000,289603,37069299.877433,0.345299
+objwjMbBXlRDUb07UVtRo4,P0,25000,3200000,554855,71021459.933110,0.045057
+objwjMbBXlRDUb07UVtRo4,P1,25000,3200000,189477,24253105.346766,0.131942
+objwjMbBXlRDUb07UVtRo4,P2,25000,3200000,105092,13451809.359579,0.237886
+objwjMbBXlRDUb07UVtRo4,P3,25000,3200000,95101,12173034.434871,0.262876

results/20260912T053120Z/core-4p4s-100k-128.txt

@@ -0,0 +1,17 @@
+=== core-4p4s-100k-128 ===
+
+NATS Pub/Sub stats: 1,361,921 msgs/sec ~ 166.25 MB/sec
+ Pub stats: 379,346 msgs/sec ~ 46.31 MB/sec
+  [1] 554,855 msgs/sec ~ 67.73 MB/sec (25000 msgs)
+  [2] 189,477 msgs/sec ~ 23.13 MB/sec (25000 msgs)
+  [3] 105,092 msgs/sec ~ 12.83 MB/sec (25000 msgs)
+  [4] 95,101 msgs/sec ~ 11.61 MB/sec (25000 msgs)
+  min 95,101 | avg 236,131 | max 554,855 | stddev 187,631 msgs
+ Sub stats: 1,150,536 msgs/sec ~ 140.45 MB/sec
+  [1] 292,772 msgs/sec ~ 35.74 MB/sec (100000 msgs)
+  [2] 290,329 msgs/sec ~ 35.44 MB/sec (100000 msgs)
+  [3] 290,323 msgs/sec ~ 35.44 MB/sec (100000 msgs)
+  [4] 289,603 msgs/sec ~ 35.35 MB/sec (100000 msgs)
+  min 289,603 | avg 290,756 | max 292,772 | stddev 1,200 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T053120Z/core-4p4s-50k-1k.csv

@@ -0,0 +1,9 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+LPWAPBuugXuZD3pcA9R4Q1,S0,50000,51200000,175324,179532297.676261,0.285185
+LPWAPBuugXuZD3pcA9R4Q1,S1,50000,51200000,174783,178978214.492079,0.286068
+LPWAPBuugXuZD3pcA9R4Q1,S2,50000,51200000,170404,174493875.742093,0.293420
+LPWAPBuugXuZD3pcA9R4Q1,S3,50000,51200000,146006,149510306.440412,0.342451
+LPWAPBuugXuZD3pcA9R4Q1,P0,12500,12800000,206954,211921364.855709,0.060400
+LPWAPBuugXuZD3pcA9R4Q1,P1,12500,12800000,78140,80016334.334450,0.159967
+LPWAPBuugXuZD3pcA9R4Q1,P2,12500,12800000,47044,48173938.085417,0.265704
+LPWAPBuugXuZD3pcA9R4Q1,P3,12500,12800000,43981,45036904.770320,0.284211

results/20260912T053120Z/core-4p4s-50k-1k.txt

@@ -0,0 +1,17 @@
+=== core-4p4s-50k-1k ===
+
+NATS Pub/Sub stats: 695,192 msgs/sec ~ 678.90 MB/sec
+ Pub stats: 175,725 msgs/sec ~ 171.61 MB/sec
+  [1] 206,954 msgs/sec ~ 202.10 MB/sec (12500 msgs)
+  [2] 78,140 msgs/sec ~ 76.31 MB/sec (12500 msgs)
+  [3] 47,044 msgs/sec ~ 45.94 MB/sec (12500 msgs)
+  [4] 43,981 msgs/sec ~ 42.95 MB/sec (12500 msgs)
+  min 43,981 | avg 94,029 | max 206,954 | stddev 66,552 msgs
+ Sub stats: 584,024 msgs/sec ~ 570.34 MB/sec
+  [1] 175,324 msgs/sec ~ 171.22 MB/sec (50000 msgs)
+  [2] 174,783 msgs/sec ~ 170.69 MB/sec (50000 msgs)
+  [3] 170,404 msgs/sec ~ 166.41 MB/sec (50000 msgs)
+  [4] 146,006 msgs/sec ~ 142.58 MB/sec (50000 msgs)
+  min 146,006 | avg 166,629 | max 175,324 | stddev 12,058 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T053120Z/core-8p8s-200k-128.csv

@@ -0,0 +1,17 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+D29xFtCiL3n1hFtnnty2bJ,S0,200000,25600000,275094,35212130.182711,0.727022
+D29xFtCiL3n1hFtnnty2bJ,S1,200000,25600000,274807,35175364.678326,0.727782
+D29xFtCiL3n1hFtnnty2bJ,S2,200000,25600000,274647,35154883.000958,0.728206
+D29xFtCiL3n1hFtnnty2bJ,S3,200000,25600000,274140,35089962.583779,0.729553
+D29xFtCiL3n1hFtnnty2bJ,S4,200000,25600000,273964,35067496.657328,0.730021
+D29xFtCiL3n1hFtnnty2bJ,S5,200000,25600000,228037,29188847.958244,0.877047
+D29xFtCiL3n1hFtnnty2bJ,S6,200000,25600000,227447,29113332.808421,0.879322
+D29xFtCiL3n1hFtnnty2bJ,S7,200000,25600000,232494,29759235.647424,0.860237
+D29xFtCiL3n1hFtnnty2bJ,P0,25000,3200000,127444,16312929.457622,0.196163
+D29xFtCiL3n1hFtnnty2bJ,P1,25000,3200000,77933,9975435.116946,0.320788
+D29xFtCiL3n1hFtnnty2bJ,P2,25000,3200000,77633,9937136.216776,0.322024
+D29xFtCiL3n1hFtnnty2bJ,P3,25000,3200000,76965,9851558.471756,0.324822
+D29xFtCiL3n1hFtnnty2bJ,P4,25000,3200000,77205,9882296.902740,0.323811
+D29xFtCiL3n1hFtnnty2bJ,P5,25000,3200000,44296,5669906.917475,0.564383
+D29xFtCiL3n1hFtnnty2bJ,P6,25000,3200000,35857,4589723.695277,0.697210
+D29xFtCiL3n1hFtnnty2bJ,P7,25000,3200000,35509,4545175.649861,0.704043

results/20260912T053120Z/core-8p8s-200k-128.txt

@@ -0,0 +1,25 @@
+=== core-8p8s-200k-128 ===
+
+NATS Pub/Sub stats: 1,998,733 msgs/sec ~ 243.99 MB/sec
+ Pub stats: 283,259 msgs/sec ~ 34.58 MB/sec
+  [1] 127,444 msgs/sec ~ 15.56 MB/sec (25000 msgs)
+  [2] 77,933 msgs/sec ~ 9.51 MB/sec (25000 msgs)
+  [3] 77,633 msgs/sec ~ 9.48 MB/sec (25000 msgs)
+  [4] 76,965 msgs/sec ~ 9.40 MB/sec (25000 msgs)
+  [5] 77,205 msgs/sec ~ 9.42 MB/sec (25000 msgs)
+  [6] 44,296 msgs/sec ~ 5.41 MB/sec (25000 msgs)
+  [7] 35,857 msgs/sec ~ 4.38 MB/sec (25000 msgs)
+  [8] 35,509 msgs/sec ~ 4.33 MB/sec (25000 msgs)
+  min 35,509 | avg 69,105 | max 127,444 | stddev 28,572 msgs
+ Sub stats: 1,780,589 msgs/sec ~ 217.36 MB/sec
+  [1] 275,094 msgs/sec ~ 33.58 MB/sec (200000 msgs)
+  [2] 274,807 msgs/sec ~ 33.55 MB/sec (200000 msgs)
+  [3] 274,647 msgs/sec ~ 33.53 MB/sec (200000 msgs)
+  [4] 274,140 msgs/sec ~ 33.46 MB/sec (200000 msgs)
+  [5] 273,964 msgs/sec ~ 33.44 MB/sec (200000 msgs)
+  [6] 228,037 msgs/sec ~ 27.84 MB/sec (200000 msgs)
+  [7] 227,447 msgs/sec ~ 27.76 MB/sec (200000 msgs)
+  [8] 232,494 msgs/sec ~ 28.38 MB/sec (200000 msgs)
+  min 227,447 | avg 257,578 | max 275,094 | stddev 21,930 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T053120Z/host-after.txt

@@ -0,0 +1,92 @@
+execution_host=NS1.GEORGELAMBERT.ORG
+execution_ip=70.88.205.138
+hostname=NS1.GEORGELAMBERT.ORG
+utc=2026-09-12T05:33:03Z
+whoami=marchon
+pwd=/home/marchon/verae-src/verae-nats-cluster
+uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux
+nproc=40
+loadavg=9.02 8.82 8.44 6/3847 224483
+client_vmid=510
+nats_url=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+js_extra_memory=1
+compare_dir=/home/marchon/verae-src/verae-nats-cluster/results/20260912T051237Z
+--- nats 511 max_mem ---
+  max_mem: 8G
+--- nats 511 jetstream mount ---
+nats-js on /var/lib/nats/jetstream type tmpfs (rw,relatime,size=8388608k,uid=100000,gid=100000,inode64)
+--- free ---
+               total        used        free      shared  buff/cache   available
+Mem:           377Gi        38Gi       324Gi       4.3Gi        21Gi       339Gi
+Swap:          8.0Gi          0B       8.0Gi
+--- pct list ---
+VMID       Status     Lock         Name                
+100        stopped                 CT100               
+101        stopped                 CT101               
+102        stopped                 CT102               
+103        stopped                 CT103               
+104        stopped                 CT104               
+105        stopped                 CT105               
+106        stopped                 CT106               
+107        stopped                 CT107               
+108        stopped                 CT108               
+109        stopped                 CT109               
+110        stopped                 CT110               
+111        stopped                 CT111               
+112        stopped                 CT112               
+113        stopped                 CT113               
+114        stopped                 CT114               
+115        stopped                 CT115               
+116        stopped                 paul                
+117        stopped                 CT117               
+118        stopped                 CT118               
+119        stopped                 CT119               
+120        stopped                 rustdeskserver      
+121        stopped                 CT121               
+122        stopped                 CT122               
+123        stopped                 CT123               
+124        stopped                 bdross              
+200        stopped                 CT200               
+201        stopped                 CT201               
+202        stopped                 CT202               
+300        running                 mail.veraetime.info 
+400        stopped                 evercycle-api       
+501        running                 pki                 
+502        running                 peergos-a           
+503        running                 peergos-b           
+504        stopped                 minio               
+510        running                 verae-px-worker     
+511        running                 nats-a              
+512        running                 nats-b              
+513        running                 nats-c              
+2000       stopped                 CT2000              
+2001       stopped                 CT2001              
+2002       stopped                 CT2002              
+--- pct config 510 ---
+cores: 8
+hostname: verae-px-worker
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:BD:5D:62,ip=10.10.10.20/24,type=veth
+rootfs: SSD2:subvol-510-disk-0,size=16G
+swap: 512
+--- pct config 511 ---
+cores: 8
+hostname: nats-a
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:CF:AC:68,ip=10.10.10.21/24,type=veth
+rootfs: SSD2:subvol-511-disk-0,size=8G
+swap: 256
+--- pct config 512 ---
+cores: 8
+hostname: nats-b
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:42:73:A2,ip=10.10.10.22/24,type=veth
+rootfs: SSD2:subvol-512-disk-0,size=8G
+swap: 256
+--- pct config 513 ---
+cores: 8
+hostname: nats-c
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:1B:9B:DC,ip=10.10.10.23/24,type=veth
+rootfs: SSD2:subvol-513-disk-0,size=8G
+swap: 256

results/20260912T053120Z/host-before.txt

@@ -0,0 +1,92 @@
+execution_host=NS1.GEORGELAMBERT.ORG
+execution_ip=70.88.205.138
+hostname=NS1.GEORGELAMBERT.ORG
+utc=2026-09-12T05:31:20Z
+whoami=marchon
+pwd=/home/marchon/verae-src/verae-nats-cluster
+uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux
+nproc=40
+loadavg=8.77 8.39 8.26 6/3849 198471
+client_vmid=510
+nats_url=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+js_extra_memory=1
+compare_dir=/home/marchon/verae-src/verae-nats-cluster/results/20260912T051237Z
+--- nats 511 max_mem ---
+  max_mem: 8G
+--- nats 511 jetstream mount ---
+nats-js on /var/lib/nats/jetstream type tmpfs (rw,relatime,size=8388608k,uid=100000,gid=100000,inode64)
+--- free ---
+               total        used        free      shared  buff/cache   available
+Mem:           377Gi        38Gi       324Gi       4.2Gi        21Gi       339Gi
+Swap:          8.0Gi          0B       8.0Gi
+--- pct list ---
+VMID       Status     Lock         Name                
+100        stopped                 CT100               
+101        stopped                 CT101               
+102        stopped                 CT102               
+103        stopped                 CT103               
+104        stopped                 CT104               
+105        stopped                 CT105               
+106        stopped                 CT106               
+107        stopped                 CT107               
+108        stopped                 CT108               
+109        stopped                 CT109               
+110        stopped                 CT110               
+111        stopped                 CT111               
+112        stopped                 CT112               
+113        stopped                 CT113               
+114        stopped                 CT114               
+115        stopped                 CT115               
+116        stopped                 paul                
+117        stopped                 CT117               
+118        stopped                 CT118               
+119        stopped                 CT119               
+120        stopped                 rustdeskserver      
+121        stopped                 CT121               
+122        stopped                 CT122               
+123        stopped                 CT123               
+124        stopped                 bdross              
+200        stopped                 CT200               
+201        stopped                 CT201               
+202        stopped                 CT202               
+300        running                 mail.veraetime.info 
+400        stopped                 evercycle-api       
+501        running                 pki                 
+502        running                 peergos-a           
+503        running                 peergos-b           
+504        stopped                 minio               
+510        running                 verae-px-worker     
+511        running                 nats-a              
+512        running                 nats-b              
+513        running                 nats-c              
+2000       stopped                 CT2000              
+2001       stopped                 CT2001              
+2002       stopped                 CT2002              
+--- pct config 510 ---
+cores: 8
+hostname: verae-px-worker
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:BD:5D:62,ip=10.10.10.20/24,type=veth
+rootfs: SSD2:subvol-510-disk-0,size=16G
+swap: 512
+--- pct config 511 ---
+cores: 8
+hostname: nats-a
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:CF:AC:68,ip=10.10.10.21/24,type=veth
+rootfs: SSD2:subvol-511-disk-0,size=8G
+swap: 256
+--- pct config 512 ---
+cores: 8
+hostname: nats-b
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:42:73:A2,ip=10.10.10.22/24,type=veth
+rootfs: SSD2:subvol-512-disk-0,size=8G
+swap: 256
+--- pct config 513 ---
+cores: 8
+hostname: nats-c
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:1B:9B:DC,ip=10.10.10.23/24,type=veth
+rootfs: SSD2:subvol-513-disk-0,size=8G
+swap: 256

results/20260912T053120Z/js-1p-20k-128-r3.csv

@@ -0,0 +1,2 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+DsDjrcOoYbw8k54T1mMXxq,P0,40000,2560000,17388,2225703.026620,1.150198

results/20260912T053120Z/js-1p-20k-128-r3.txt

@@ -0,0 +1,5 @@
+=== js-1p-20k-128-r3 ===
+
+Pub stats: 17,388 msgs/sec ~ 2.12 MB/sec
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T053120Z/js-2p2s-20k-128-r3.csv

@@ -0,0 +1,5 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+JYqsv0sXzEqkC0r49q0aE5,S0,20203,1280000,5135,657394.514895,1.947080
+JYqsv0sXzEqkC0r49q0aE5,S1,20210,1280000,4971,636317.997133,2.011573
+JYqsv0sXzEqkC0r49q0aE5,P0,20000,1280000,5099,652788.600986,1.960819
+JYqsv0sXzEqkC0r49q0aE5,P1,20000,1280000,4979,637405.779512,2.008140

results/20260912T053120Z/js-2p2s-20k-128-r3.txt

@@ -0,0 +1,13 @@
+=== js-2p2s-20k-128-r3 ===
+
+NATS Pub/Sub stats: 19,876 msgs/sec ~ 2.43 MB/sec
+ Pub stats: 9,959 msgs/sec ~ 1.22 MB/sec
+  [1] 5,099 msgs/sec ~ 637.49 KB/sec (10000 msgs)
+  [2] 4,979 msgs/sec ~ 622.47 KB/sec (10000 msgs)
+  min 4,979 | avg 5,039 | max 5,099 | stddev 60 msgs
+ Sub stats: 9,942 msgs/sec ~ 1.21 MB/sec
+  [1] 5,135 msgs/sec ~ 641.99 KB/sec (10000 msgs)
+  [2] 4,971 msgs/sec ~ 621.40 KB/sec (10000 msgs)
+  min 4,971 | avg 5,053 | max 5,135 | stddev 82 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T053120Z/js-4p-20k-1k-r3.csv

@@ -0,0 +1,5 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+r1r7AI1sRr29EvqNG14gSh,P0,10000,5120000,7677,7861269.752874,0.651294
+r1r7AI1sRr29EvqNG14gSh,P1,10000,5120000,4571,4680909.596986,1.093805
+r1r7AI1sRr29EvqNG14gSh,P2,10000,5120000,4529,4637905.946623,1.103946
+r1r7AI1sRr29EvqNG14gSh,P3,10000,5120000,4528,4637319.175494,1.104086

results/20260912T053120Z/js-4p-20k-1k-r3.txt

@@ -0,0 +1,10 @@
+=== js-4p-20k-1k-r3 ===
+
+Pub stats: 18,114 msgs/sec ~ 17.69 MB/sec
+ [1] 7,677 msgs/sec ~ 7.50 MB/sec (5000 msgs)
+ [2] 4,571 msgs/sec ~ 4.46 MB/sec (5000 msgs)
+ [3] 4,529 msgs/sec ~ 4.42 MB/sec (5000 msgs)
+ [4] 4,528 msgs/sec ~ 4.42 MB/sec (5000 msgs)
+ min 4,528 | avg 5,326 | max 7,677 | stddev 1,357 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T053120Z/js-4p-50k-128-r3.csv

@@ -0,0 +1,5 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+6cJTgCeHUXHowXNlC39Tws,P0,25000,1600000,7724,988748.856206,1.618207
+6cJTgCeHUXHowXNlC39Tws,P1,25000,1600000,7704,986199.433317,1.622390
+6cJTgCeHUXHowXNlC39Tws,P2,25000,1600000,5713,731384.231657,2.187633
+6cJTgCeHUXHowXNlC39Tws,P3,25000,1600000,5713,731308.958368,2.187858

results/20260912T053120Z/js-4p-50k-128-r3.txt

@@ -0,0 +1,10 @@
+=== js-4p-50k-128-r3 ===
+
+Pub stats: 22,853 msgs/sec ~ 2.79 MB/sec
+ [1] 7,724 msgs/sec ~ 965.58 KB/sec (12500 msgs)
+ [2] 7,704 msgs/sec ~ 963.09 KB/sec (12500 msgs)
+ [3] 5,713 msgs/sec ~ 714.24 KB/sec (12500 msgs)
+ [4] 5,713 msgs/sec ~ 714.17 KB/sec (12500 msgs)
+ min 5,713 | avg 6,713 | max 7,724 | stddev 1,000 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T053120Z/js-mem-1p-20k-128-r3.csv

@@ -0,0 +1,2 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+jHJGzEaQyqmtfzAfyQRddn,P0,40000,2560000,22153,2835624.882291,0.902799

results/20260912T053120Z/js-mem-1p-20k-128-r3.txt

@@ -0,0 +1,5 @@
+=== js-mem-1p-20k-128-r3 ===
+
+Pub stats: 22,153 msgs/sec ~ 2.70 MB/sec
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T053120Z/js-mem-4p-20k-1k-r3.csv

@@ -0,0 +1,5 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+bSUG5M5Cdh9fEhPsY8MWLm,P0,10000,5120000,9256,9478798.103952,0.540153
+bSUG5M5Cdh9fEhPsY8MWLm,P1,10000,5120000,8901,9115347.820727,0.561690
+bSUG5M5Cdh9fEhPsY8MWLm,P2,10000,5120000,7192,7365226.588506,0.695159
+bSUG5M5Cdh9fEhPsY8MWLm,P3,10000,5120000,7171,7343566.738925,0.697209

results/20260912T053120Z/js-mem-4p-20k-1k-r3.txt

@@ -0,0 +1,10 @@
+=== js-mem-4p-20k-1k-r3 ===
+
+Pub stats: 28,685 msgs/sec ~ 28.01 MB/sec
+ [1] 9,256 msgs/sec ~ 9.04 MB/sec (5000 msgs)
+ [2] 8,901 msgs/sec ~ 8.69 MB/sec (5000 msgs)
+ [3] 7,192 msgs/sec ~ 7.02 MB/sec (5000 msgs)
+ [4] 7,171 msgs/sec ~ 7.00 MB/sec (5000 msgs)
+ min 7,171 | avg 8,130 | max 9,256 | stddev 956 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T053120Z/js-mem-4p-50k-128-r3.csv

@@ -0,0 +1,5 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+RCjYO8CWEOKsynfDUe3izo,P0,25000,1600000,9194,1176894.303068,1.359510
+RCjYO8CWEOKsynfDUe3izo,P1,25000,1600000,9199,1177484.153608,1.358829
+RCjYO8CWEOKsynfDUe3izo,P2,25000,1600000,9144,1170517.752374,1.366916
+RCjYO8CWEOKsynfDUe3izo,P3,25000,1600000,9097,1164445.806755,1.374044

results/20260912T053120Z/js-mem-4p-50k-128-r3.txt

@@ -0,0 +1,10 @@
+=== js-mem-4p-50k-128-r3 ===
+
+Pub stats: 36,355 msgs/sec ~ 4.44 MB/sec
+ [1] 9,194 msgs/sec ~ 1.12 MB/sec (12500 msgs)
+ [2] 9,199 msgs/sec ~ 1.12 MB/sec (12500 msgs)
+ [3] 9,144 msgs/sec ~ 1.12 MB/sec (12500 msgs)
+ [4] 9,097 msgs/sec ~ 1.11 MB/sec (12500 msgs)
+ min 9,097 | avg 9,158 | max 9,199 | stddev 41 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T053120Z/lat-1p-5k-128.txt

@@ -0,0 +1,2 @@
+=== lat-1p-5k-128 ===
+{"count":5000,"pubs":1,"size":128,"mode":"flood","min_us":94920,"avg_us":132026,"p50_us":131990,"p90_us":156086,"p99_us":158860,"max_us":158965,"min":"94.920ms","avg":"132.026ms","p50":"131.990ms","p90":"156.086ms","p99":"158.860ms","max":"158.965ms"}

results/20260912T053120Z/lat-4p-10k-128.txt

@@ -0,0 +1,2 @@
+=== lat-4p-10k-128 ===
+{"count":10000,"pubs":4,"size":128,"mode":"flood","min_us":153318,"avg_us":204329,"p50_us":205865,"p90_us":234151,"p99_us":237346,"max_us":237477,"min":"153.318ms","avg":"204.329ms","p50":"205.865ms","p90":"234.151ms","p99":"237.346ms","max":"237.477ms"}

results/20260912T053120Z/lat-4p-5k-1k.txt

@@ -0,0 +1,2 @@
+=== lat-4p-5k-1k ===
+{"count":5000,"pubs":4,"size":1024,"mode":"flood","min_us":113281,"avg_us":132425,"p50_us":131457,"p90_us":140317,"p99_us":144683,"max_us":145365,"min":"113.281ms","avg":"132.425ms","p50":"131.457ms","p90":"140.317ms","p99":"144.683ms","max":"145.365ms"}

results/20260912T053120Z/lat-8p-20k-128.txt

@@ -0,0 +1,2 @@
+=== lat-8p-20k-128 ===
+{"count":20000,"pubs":8,"size":128,"mode":"flood","min_us":233727,"avg_us":310924,"p50_us":311170,"p90_us":373437,"p99_us":400764,"max_us":402811,"min":"233.727ms","avg":"310.924ms","p50":"311.170ms","p90":"373.437ms","p99":"400.764ms","max":"402.811ms"}

results/20260912T053120Z/lat-ping-1k-128.txt

@@ -0,0 +1,2 @@
+=== lat-ping-1k-128 ===
+{"count":1000,"pubs":1,"size":128,"mode":"ping","min_us":250,"avg_us":319,"p50_us":299,"p90_us":363,"p99_us":684,"max_us":2759,"min":"0.250ms","avg":"0.319ms","p50":"0.299ms","p90":"0.363ms","p99":"0.684ms","max":"2.759ms"}

results/20260912T053120Z/nats-cluster-bench-ns1.md

@@ -0,0 +1,355 @@
+**Progress report (maximized NS1 study)** · run `20260912T053120Z` (UTC)
+
+> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): `maximize-ns1-study.sh` (cores/RAM/`max_mem`/tmpfs), then `study-on-ns1.sh`, `nats bench`, `latency.mjs` (LXC 510), matplotlib, pandoc, weasyprint. Traffic stayed on `vmbr1`. veth/10G was **not** changed. After the ladder, JetStream was put back on ZFS and product streams were re-created; **8 cores / 16 GiB / max_mem 8G stay**.
+
+## Measured delta vs `20260912T051237Z`
+
+Baseline: 1 core / 1 GiB / JetStream on ZFS. This run: 8 cores / 16 GiB / JetStream **tmpfs** (file r=3) plus extra **memory** store rows. veth/10G unchanged.
+
+| Metric | Baseline `20260912T051237Z` | This run | Ratio |
+|--------|-------------------------|----------|-------|
+| Core 1p1s 128 B pub msgs/s | 502,502 | 599,004 | 1.19× |
+| Core 8p8s 128 B aggregate msgs/s | 2,065,217 | 1,998,733 | 0.97× |
+| JS file r=3 1p 128 B pub msgs/s | 7,393 | 17,388 | 2.35× |
+| JS file r=3 4p 128 B pub msgs/s | 17,986 | 22,853 | 1.27× |
+| JS file r=3 4p 1 KiB pub msgs/s | 14,985 | 18,114 | 1.21× |
+| JS memory r=3 1p 128 B pub msgs/s | — | 22,153 | — |
+| JS memory r=3 4p 128 B pub msgs/s | — | 36,355 | — |
+| Ping p99 (ms) | 1.377ms | 0.684ms | 2.01× faster |
+
+![Baseline vs maximized publish rates (log)](charts/delta-vs-baseline.png)
+
+*Baseline vs maximized publish rates (log)*
+
+
+---
+
+## 1. Executive summary
+
+| Item | This NS1-host run |
+|------|-------------------|
+| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `marchon` |
+| Bench client | LXC 510 `verae-px-worker` |
+| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` |
+| Client URL | `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` |
+| Host load before | `8.77 8.39 8.26 6/3849 198471` |
+| Host load after | `9.02 8.82 8.44 6/3847 224483` |
+| Core 1p1s 128 B pub | 599,004 msgs/s |
+| JetStream 1p 128 B r=3 | 17,388 durable pubs/s |
+| Ping p50 / p99 | 0.299ms / 0.684ms |
+
+Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst.
+
+---
+
+## 2. Where it ran (and where it did not)
+
+```text
+Operator laptop  ──ssh──►  NS1.GEORGELAMBERT.ORG  70.88.205.138
+                              study-on-ns1.sh
+                              python3 build-ns1-study-report.py
+                              sudo pct exec 510 ──► nats bench / latency.mjs
+                                                    │
+                                                    ▼ vmbr1
+                                              10.10.10.21-23 :4222
+```
+
+- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510).
+- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file.
+
+---
+
+## 3. Results (this run)
+
+### Host and brokers
+
+**Before**
+
+| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream |
+|------|------|-------------|---------|----------|-----|-------|---------|-----------|
+| nats-a | 511 | 3 | 3,013 | 3,037 | 1 | 8 | 14,553,088 | True |
+| nats-b | 512 | 3 | 1,415 | 1,424 | 0 | 1 | 13,557,760 | True |
+| nats-c | 513 | 0 | 1,361 | 1,395 | 1 | 1 | 14,028,800 | True |
+
+**After**
+
+| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream |
+|------|------|-------------|---------|----------|-----|-------|---------|-----------|
+| nats-a | 511 | 3 | 644,730 | 1,344,768 | 2 | 8 | 67,104,768 | True |
+| nats-b | 512 | 3 | 491,295 | 978,800 | 1 | 1 | 82,542,592 | True |
+| nats-c | 513 | 0 | 668,544 | 1,330,940 | 0 | 1 | 37,314,560 | True |
+
+nproc=40 · uname=`Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux`
+
+### Throughput
+
+| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
+|-----|------|------------------|------------|----------|------------|----------|
+| `core-1p1s-50k-128` | core pub/sub | 810,988 | 599,004 | 73.12 | 456,116 | 55.68 |
+| `core-4p4s-100k-128` | core pub/sub | 1,361,921 | 379,346 | 46.31 | 1,150,536 | 140.45 |
+| `core-4p4s-50k-1k` | core pub/sub | 695,192 | 175,725 | 171.61 | 584,024 | 570.34 |
+| `core-8p8s-200k-128` | core pub/sub | 1,998,733 | 283,259 | 34.58 | 1,780,589 | 217.36 |
+| `js-1p-20k-128-r3` | jetstream r=3 file | — | 17,388 | 2.12 | — | — |
+| `js-2p2s-20k-128-r3` | jetstream r=3 file | 19,876 | 9,959 | 1.22 | 9,942 | 1.21 |
+| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 18,114 | 17.69 | — | — |
+| `js-4p-50k-128-r3` | jetstream r=3 file | — | 22,853 | 2.79 | — | — |
+| `js-mem-1p-20k-128-r3` | jetstream r=3 file | — | 22,153 | 2.70 | — | — |
+| `js-mem-4p-20k-1k-r3` | jetstream r=3 file | — | 28,685 | 28.01 | — | — |
+| `js-mem-4p-50k-128-r3` | jetstream r=3 file | — | 36,355 | 4.44 | — | — |
+
+### Round-trip delay
+
+| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
+|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|
+| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.250ms | 0.319ms | 0.299ms | 0.363ms | 0.684ms | 2.759ms |
+| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 94.920ms | 132.026ms | 131.990ms | 156.086ms | 158.860ms | 158.965ms |
+| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 113.281ms | 132.425ms | 131.457ms | 140.317ms | 144.683ms | 145.365ms |
+| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 153.318ms | 204.329ms | 205.865ms | 234.151ms | 237.346ms | 237.477ms |
+| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 233.727ms | 310.924ms | 311.170ms | 373.437ms | 400.764ms | 402.811ms |
+
+### Core NATS
+
+![Core NATS throughput at four loads (NS1 host run)](charts/core-throughput.png)
+
+*Core NATS throughput at four loads (NS1 host run)*
+### Payload size (core)
+
+![Core NATS 128 B vs 1 KiB (NS1 host run)](charts/payload-size.png)
+
+*Core NATS 128 B vs 1 KiB (NS1 host run)*
+### JetStream r=3 file
+
+![JetStream durable publish rate (NS1 host run)](charts/js-throughput.png)
+
+*JetStream durable publish rate (NS1 host run)*
+### Core vs JetStream
+
+![Core vs JetStream publish rate, log scale (NS1 host run)](charts/core-vs-js.png)
+
+*Core vs JetStream publish rate, log scale (NS1 host run)*
+### Delay
+
+![Ping vs flood delay percentiles, log scale (NS1 host run)](charts/delay-percentiles.png)
+
+*Ping vs flood delay percentiles, log scale (NS1 host run)*
+
+---
+
+## 4. Study methodology
+
+### 4.1 Question
+
+On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)?
+
+### 4.2 Hypotheses (stated before the run)
+
+1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica.
+2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated.
+3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting.
+4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count.
+5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound.
+
+### 4.3 Independent variables (what we changed)
+
+| Factor | Levels |
+|--------|--------|
+| Transport | Core NATS pub/sub vs JetStream file replicas=3 |
+| Publisher count | 1, 2, 4, 8 |
+| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 |
+| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) |
+| Payload | 128 B, 1024 B |
+| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) |
+
+### 4.4 Dependent variables (what we recorded)
+
+| Metric | Instrument | Unit |
+|--------|------------|------|
+| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s |
+| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s |
+| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) |
+| Publisher spread | nats min/avg/max **msgs/s** | not delay |
+| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max |
+| Host load | `/proc/loadavg` before and after | load average |
+| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem |
+
+**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`.
+
+### 4.5 Controls and constants
+
+- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`.
+- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`).
+- Bench client is LXC **510**, not a nats-* server.
+- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide.
+- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`).
+- No TLS, no nkeys, no account isolation (isolation is `vmbr1`).
+- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder.
+
+### 4.6 Procedure
+
+1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise.
+2. Snapshot host load, memory, LXC configs, and each nats `varz`.
+3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB).
+4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file.
+5. Copy `latency.mjs` into 510; ping then flood at several batch sizes.
+6. Snapshot host/`varz` again.
+7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**.
+
+No publish, subscribe, chart, or PDF process runs on the operator laptop for this study.
+
+### 4.7 Instrumentation path
+
+```text
+[NS1 host 70.88.205.138]
+   study-on-ns1.sh  (bash + python3)
+        |
+        | sudo pct exec 510
+        v
+[LXC 510 verae-px-worker 10.10.10.20]
+   nats bench / node latency.mjs
+        |
+        | NATS client protocol to
+        v
+[LXC 511/512/513  10.10.10.21-23 :4222]
+   nats-server -js  cluster routes :6222
+```
+
+The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`.
+
+### 4.8 Threats to validity
+
+| Threat | Effect on numbers |
+|--------|-------------------|
+| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. |
+| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. |
+| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. |
+| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. |
+| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. |
+| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. |
+| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. |
+| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. |
+
+### 4.9 Ethics / safety
+
+Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket.
+
+---
+
+## 5. Suggestions for fine-tuning
+
+These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way.
+
+### 5.1 Treat JetStream as the product limiter
+
+Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out.
+
+### 5.2 Split storage class by stream
+
+| Stream | Suggested store | Why |
+|--------|-----------------|-----|
+| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad |
+| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both |
+| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer |
+| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry |
+| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive |
+
+Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert.
+
+### 5.3 Give JetStream real disks
+
+Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain.
+
+- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`.
+- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure.
+- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it.
+- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.”
+
+### 5.4 Isolate the nats CTs from the rest of NS1
+
+Host load on this box is often already several. Pin:
+
+- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes.
+- Memory high enough that file-backed streams stay cache-hot for the working set.
+- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync.
+
+Re-run this study after pinning; H1/H2 should move more than ping.
+
+### 5.5 Consumer and mailbox tuning (delay H4)
+
+Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT.
+
+- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM.
+- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber.
+- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped.
+- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT.
+
+### 5.6 Publisher-side batching in middleware
+
+A timestamp job is one small JSON. 16k msgs/s is ample. Still:
+
+- Avoid per-byte publishes; one message per job/event.
+- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs).
+- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only.
+
+### 5.7 nats-server knobs worth measuring (A/B with this script)
+
+| Knob | Why try it |
+|------|------------|
+| `max_payload` | Keep default unless archive puts grow |
+| `write_deadline` | Slow consumer protection for webhooks |
+| `max_pending` | Bound memory on a stuck Zapier hook |
+| `max_connections` | Fleet workers + keep + middleware |
+| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT |
+| `max_outstanding_catchup` | Replica restart after a nats-c blip |
+| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT |
+
+Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99.
+
+### 5.8 Network
+
+- Keep NATS off `vmbr0`. No change.
+- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load).
+- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus.
+
+### 5.9 Security cost (when nkeys/mTLS flip)
+
+`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement.
+
+### 5.10 Operational fine-tuning (lag, not peak msgs/s)
+
+1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`.
+2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense.
+3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost.
+4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated.
+
+### 5.11 What not to tune
+
+- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge.
+- Do not treat flood 400 ms as “cluster RTT.” Fix consumers.
+- Do not load-test on `ZAPIER_*` streams.
+- Do not bind client NATS to `0.0.0.0` on `vmbr0`.
+
+### 5.12 Recommended next experiments (same method, one change each)
+
+1. CPU pin nats-a/b/c → re-run JS 1p + ping.
+2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder).
+3. Distinct `store_dir` disks per node.
+4. nkeys on, same ladder.
+5. Three hardware boxes, same `cluster.env` IPs updated.
+
+Each experiment should produce a new `results/<utc>/` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory.
+
+
+---
+
+## 6. Reproducing this study
+
+On **NS1 only**:
+
+```bash
+cd ~/verae-src/verae-nats-cluster
+bash scripts/study-on-ns1.sh
+```
+
+The script exits if `hostname` is not NS1. Outputs land in `results/<utc>/` including `nats-cluster-bench-ns1.{md,html,pdf}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog.
+
+Raw logs for this run: `results/20260912T053120Z/`.

results/20260912T053120Z/varz-after.json

@@ -0,0 +1,50 @@
+[
+  {
+    "vmid": "511",
+    "name": "nats-a",
+    "server_name": "nats-a",
+    "host": "10.10.10.21",
+    "port": 4222,
+    "connections": 3,
+    "in_msgs": 644730,
+    "out_msgs": 1344768,
+    "in_bytes": 178180385,
+    "out_bytes": 317368503,
+    "cpu": 2,
+    "cores": 8,
+    "mem": 67104768,
+    "jetstream": true
+  },
+  {
+    "vmid": "512",
+    "name": "nats-b",
+    "server_name": "nats-b",
+    "host": "10.10.10.22",
+    "port": 4222,
+    "connections": 3,
+    "in_msgs": 491295,
+    "out_msgs": 978800,
+    "in_bytes": 140248328,
+    "out_bytes": 203296096,
+    "cpu": 1,
+    "cores": 1,
+    "mem": 82542592,
+    "jetstream": true
+  },
+  {
+    "vmid": "513",
+    "name": "nats-c",
+    "server_name": "nats-c",
+    "host": "10.10.10.23",
+    "port": 4222,
+    "connections": 0,
+    "in_msgs": 668544,
+    "out_msgs": 1330940,
+    "in_bytes": 199016161,
+    "out_bytes": 316279225,
+    "cpu": 0,
+    "cores": 1,
+    "mem": 37314560,
+    "jetstream": true
+  }
+]

results/20260912T053120Z/varz-before.json

@@ -0,0 +1,50 @@
+[
+  {
+    "vmid": "511",
+    "name": "nats-a",
+    "server_name": "nats-a",
+    "host": "10.10.10.21",
+    "port": 4222,
+    "connections": 3,
+    "in_msgs": 3013,
+    "out_msgs": 3037,
+    "in_bytes": 177854,
+    "out_bytes": 203258,
+    "cpu": 1,
+    "cores": 8,
+    "mem": 14553088,
+    "jetstream": true
+  },
+  {
+    "vmid": "512",
+    "name": "nats-b",
+    "server_name": "nats-b",
+    "host": "10.10.10.22",
+    "port": 4222,
+    "connections": 3,
+    "in_msgs": 1415,
+    "out_msgs": 1424,
+    "in_bytes": 126181,
+    "out_bytes": 72041,
+    "cpu": 0,
+    "cores": 1,
+    "mem": 13557760,
+    "jetstream": true
+  },
+  {
+    "vmid": "513",
+    "name": "nats-c",
+    "server_name": "nats-c",
+    "host": "10.10.10.23",
+    "port": 4222,
+    "connections": 0,
+    "in_msgs": 1361,
+    "out_msgs": 1395,
+    "in_bytes": 108748,
+    "out_bytes": 69591,
+    "cpu": 1,
+    "cores": 1,
+    "mem": 14028800,
+    "jetstream": true
+  }
+]