Snapshot of verae-nats-cluster (optimal NATS config study)

results/20260912T053120Z/BENCH.md

@@ -0,0 +1,52 @@
+# NATS cluster message speed
+
+Run **`20260912T053120Z`** (UTC). Client: LXC **510** `verae-px-worker` (`10.10.10.20`), not a nats-* server. Servers: `nats-a/b/c` on `10.10.10.21–23` (`vmbr1` only).
+
+Client URL:
+
+```text
+nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+```
+
+## Method
+
+- **Core NATS** is fire-and-forget pub/sub (`nats bench`). No disk, no replica ack.
+- **JetStream** uses **file** storage and **replicas=3** (same as product streams). The unique stream `benchstream` is deleted between JS loads.
+- Throughput is **msgs/sec** from nats CLI **0.1.6** (`--no-progress --csv`). Its min/avg/max are publisher **rate spread**, not delay.
+- **Ping** delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster.
+- **Flood** delay: N publishers dump the whole batch, then the subscriber drains. This is **queueing under burst**, not wire RTT.
+- Probe: `scripts/latency.mjs` (two connections, header timestamp).
+
+## Throughput
+
+| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
+|-----|------|------------------|------------|----------|------------|----------|
+| `core-1p1s-50k-128` | core pub/sub | 810,988 | 599,004 | 73.12 | 456,116 | 55.68 |
+| `core-4p4s-100k-128` | core pub/sub | 1,361,921 | 379,346 | 46.31 | 1,150,536 | 140.45 |
+| `core-4p4s-50k-1k` | core pub/sub | 695,192 | 175,725 | 171.61 | 584,024 | 570.34 |
+| `core-8p8s-200k-128` | core pub/sub | 1,998,733 | 283,259 | 34.58 | 1,780,589 | 217.36 |
+| `js-1p-20k-128-r3` | jetstream r=3 file | — | 17,388 | 2.12 | — | — |
+| `js-2p2s-20k-128-r3` | jetstream r=3 file | 19,876 | 9,959 | 1.22 | 9,942 | 1.21 |
+| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 18,114 | 17.69 | — | — |
+| `js-4p-50k-128-r3` | jetstream r=3 file | — | 22,853 | 2.79 | — | — |
+| `js-mem-1p-20k-128-r3` | jetstream r=3 file | — | 22,153 | 2.70 | — | — |
+| `js-mem-4p-20k-1k-r3` | jetstream r=3 file | — | 28,685 | 28.01 | — | — |
+| `js-mem-4p-50k-128-r3` | jetstream r=3 file | — | 36,355 | 4.44 | — | — |
+
+## Round-trip delay
+
+| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
+|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|
+| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.250ms | 0.319ms | 0.299ms | 0.363ms | 0.684ms | 2.759ms |
+| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 94.920ms | 132.026ms | 131.990ms | 156.086ms | 158.860ms | 158.965ms |
+| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 113.281ms | 132.425ms | 131.457ms | 140.317ms | 144.683ms | 145.365ms |
+| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 153.318ms | 204.329ms | 205.865ms | 234.151ms | 237.346ms | 237.477ms |
+| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 233.727ms | 310.924ms | 311.170ms | 373.437ms | 400.764ms | 402.811ms |
+
+## What the numbers mean
+
+Product job/event/archive traffic is **JetStream r=3 file**. On this three-LXC stand that is about **16k durable 128 B pubs/s** (about **13k** at 1 KiB). Core NATS is an upper bound for non-durable fan-out: about **0.7–2.0M msgs/s** aggregate at 128 B, or **~630k msgs/s (~616 MB/s)** at 1 KiB with 4 publishers.
+
+A quiet request-reply is **~0.3 ms** average, **p99 < 1 ms**. Flood rows in the **150–500 ms** band are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers.
+
+Re-run on NS1: `bash scripts/bench.sh`. Raw logs/CSVs are under `results/<utc>/`.

results/20260912T053120Z/core-1p1s-50k-128.csv

@@ -0,0 +1,3 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+w2OZLH4YWp0DF7nX2fttF4,S0,50000,6400000,456116,58382861.156470,0.109621
+w2OZLH4YWp0DF7nX2fttF4,P0,50000,6400000,599004,76672615.503222,0.083472

results/20260912T053120Z/core-1p1s-50k-128.txt

@@ -0,0 +1,7 @@
+=== core-1p1s-50k-128 ===
+
+NATS Pub/Sub stats: 810,988 msgs/sec ~ 99.00 MB/sec
+ Pub stats: 599,004 msgs/sec ~ 73.12 MB/sec
+ Sub stats: 456,116 msgs/sec ~ 55.68 MB/sec
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T053120Z/core-4p4s-100k-128.csv

@@ -0,0 +1,9 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+objwjMbBXlRDUb07UVtRo4,S0,100000,12800000,292772,37474879.058831,0.341562
+objwjMbBXlRDUb07UVtRo4,S1,100000,12800000,290329,37162115.621242,0.344437
+objwjMbBXlRDUb07UVtRo4,S2,100000,12800000,290323,37161444.974130,0.344443
+objwjMbBXlRDUb07UVtRo4,S3,100000,12800000,289603,37069299.877433,0.345299
+objwjMbBXlRDUb07UVtRo4,P0,25000,3200000,554855,71021459.933110,0.045057
+objwjMbBXlRDUb07UVtRo4,P1,25000,3200000,189477,24253105.346766,0.131942
+objwjMbBXlRDUb07UVtRo4,P2,25000,3200000,105092,13451809.359579,0.237886
+objwjMbBXlRDUb07UVtRo4,P3,25000,3200000,95101,12173034.434871,0.262876

results/20260912T053120Z/core-4p4s-100k-128.txt

@@ -0,0 +1,17 @@
+=== core-4p4s-100k-128 ===
+
+NATS Pub/Sub stats: 1,361,921 msgs/sec ~ 166.25 MB/sec
+ Pub stats: 379,346 msgs/sec ~ 46.31 MB/sec
+  [1] 554,855 msgs/sec ~ 67.73 MB/sec (25000 msgs)
+  [2] 189,477 msgs/sec ~ 23.13 MB/sec (25000 msgs)
+  [3] 105,092 msgs/sec ~ 12.83 MB/sec (25000 msgs)
+  [4] 95,101 msgs/sec ~ 11.61 MB/sec (25000 msgs)
+  min 95,101 | avg 236,131 | max 554,855 | stddev 187,631 msgs
+ Sub stats: 1,150,536 msgs/sec ~ 140.45 MB/sec
+  [1] 292,772 msgs/sec ~ 35.74 MB/sec (100000 msgs)
+  [2] 290,329 msgs/sec ~ 35.44 MB/sec (100000 msgs)
+  [3] 290,323 msgs/sec ~ 35.44 MB/sec (100000 msgs)
+  [4] 289,603 msgs/sec ~ 35.35 MB/sec (100000 msgs)
+  min 289,603 | avg 290,756 | max 292,772 | stddev 1,200 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T053120Z/core-4p4s-50k-1k.csv

@@ -0,0 +1,9 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+LPWAPBuugXuZD3pcA9R4Q1,S0,50000,51200000,175324,179532297.676261,0.285185
+LPWAPBuugXuZD3pcA9R4Q1,S1,50000,51200000,174783,178978214.492079,0.286068
+LPWAPBuugXuZD3pcA9R4Q1,S2,50000,51200000,170404,174493875.742093,0.293420
+LPWAPBuugXuZD3pcA9R4Q1,S3,50000,51200000,146006,149510306.440412,0.342451
+LPWAPBuugXuZD3pcA9R4Q1,P0,12500,12800000,206954,211921364.855709,0.060400
+LPWAPBuugXuZD3pcA9R4Q1,P1,12500,12800000,78140,80016334.334450,0.159967
+LPWAPBuugXuZD3pcA9R4Q1,P2,12500,12800000,47044,48173938.085417,0.265704
+LPWAPBuugXuZD3pcA9R4Q1,P3,12500,12800000,43981,45036904.770320,0.284211

results/20260912T053120Z/core-4p4s-50k-1k.txt

@@ -0,0 +1,17 @@
+=== core-4p4s-50k-1k ===
+
+NATS Pub/Sub stats: 695,192 msgs/sec ~ 678.90 MB/sec
+ Pub stats: 175,725 msgs/sec ~ 171.61 MB/sec
+  [1] 206,954 msgs/sec ~ 202.10 MB/sec (12500 msgs)
+  [2] 78,140 msgs/sec ~ 76.31 MB/sec (12500 msgs)
+  [3] 47,044 msgs/sec ~ 45.94 MB/sec (12500 msgs)
+  [4] 43,981 msgs/sec ~ 42.95 MB/sec (12500 msgs)
+  min 43,981 | avg 94,029 | max 206,954 | stddev 66,552 msgs
+ Sub stats: 584,024 msgs/sec ~ 570.34 MB/sec
+  [1] 175,324 msgs/sec ~ 171.22 MB/sec (50000 msgs)
+  [2] 174,783 msgs/sec ~ 170.69 MB/sec (50000 msgs)
+  [3] 170,404 msgs/sec ~ 166.41 MB/sec (50000 msgs)
+  [4] 146,006 msgs/sec ~ 142.58 MB/sec (50000 msgs)
+  min 146,006 | avg 166,629 | max 175,324 | stddev 12,058 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T053120Z/core-8p8s-200k-128.csv

@@ -0,0 +1,17 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+D29xFtCiL3n1hFtnnty2bJ,S0,200000,25600000,275094,35212130.182711,0.727022
+D29xFtCiL3n1hFtnnty2bJ,S1,200000,25600000,274807,35175364.678326,0.727782
+D29xFtCiL3n1hFtnnty2bJ,S2,200000,25600000,274647,35154883.000958,0.728206
+D29xFtCiL3n1hFtnnty2bJ,S3,200000,25600000,274140,35089962.583779,0.729553
+D29xFtCiL3n1hFtnnty2bJ,S4,200000,25600000,273964,35067496.657328,0.730021
+D29xFtCiL3n1hFtnnty2bJ,S5,200000,25600000,228037,29188847.958244,0.877047
+D29xFtCiL3n1hFtnnty2bJ,S6,200000,25600000,227447,29113332.808421,0.879322
+D29xFtCiL3n1hFtnnty2bJ,S7,200000,25600000,232494,29759235.647424,0.860237
+D29xFtCiL3n1hFtnnty2bJ,P0,25000,3200000,127444,16312929.457622,0.196163
+D29xFtCiL3n1hFtnnty2bJ,P1,25000,3200000,77933,9975435.116946,0.320788
+D29xFtCiL3n1hFtnnty2bJ,P2,25000,3200000,77633,9937136.216776,0.322024
+D29xFtCiL3n1hFtnnty2bJ,P3,25000,3200000,76965,9851558.471756,0.324822
+D29xFtCiL3n1hFtnnty2bJ,P4,25000,3200000,77205,9882296.902740,0.323811
+D29xFtCiL3n1hFtnnty2bJ,P5,25000,3200000,44296,5669906.917475,0.564383
+D29xFtCiL3n1hFtnnty2bJ,P6,25000,3200000,35857,4589723.695277,0.697210
+D29xFtCiL3n1hFtnnty2bJ,P7,25000,3200000,35509,4545175.649861,0.704043

results/20260912T053120Z/core-8p8s-200k-128.txt

@@ -0,0 +1,25 @@
+=== core-8p8s-200k-128 ===
+
+NATS Pub/Sub stats: 1,998,733 msgs/sec ~ 243.99 MB/sec
+ Pub stats: 283,259 msgs/sec ~ 34.58 MB/sec
+  [1] 127,444 msgs/sec ~ 15.56 MB/sec (25000 msgs)
+  [2] 77,933 msgs/sec ~ 9.51 MB/sec (25000 msgs)
+  [3] 77,633 msgs/sec ~ 9.48 MB/sec (25000 msgs)
+  [4] 76,965 msgs/sec ~ 9.40 MB/sec (25000 msgs)
+  [5] 77,205 msgs/sec ~ 9.42 MB/sec (25000 msgs)
+  [6] 44,296 msgs/sec ~ 5.41 MB/sec (25000 msgs)
+  [7] 35,857 msgs/sec ~ 4.38 MB/sec (25000 msgs)
+  [8] 35,509 msgs/sec ~ 4.33 MB/sec (25000 msgs)
+  min 35,509 | avg 69,105 | max 127,444 | stddev 28,572 msgs
+ Sub stats: 1,780,589 msgs/sec ~ 217.36 MB/sec
+  [1] 275,094 msgs/sec ~ 33.58 MB/sec (200000 msgs)
+  [2] 274,807 msgs/sec ~ 33.55 MB/sec (200000 msgs)
+  [3] 274,647 msgs/sec ~ 33.53 MB/sec (200000 msgs)
+  [4] 274,140 msgs/sec ~ 33.46 MB/sec (200000 msgs)
+  [5] 273,964 msgs/sec ~ 33.44 MB/sec (200000 msgs)
+  [6] 228,037 msgs/sec ~ 27.84 MB/sec (200000 msgs)
+  [7] 227,447 msgs/sec ~ 27.76 MB/sec (200000 msgs)
+  [8] 232,494 msgs/sec ~ 28.38 MB/sec (200000 msgs)
+  min 227,447 | avg 257,578 | max 275,094 | stddev 21,930 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T053120Z/host-after.txt

@@ -0,0 +1,92 @@
+execution_host=NS1.GEORGELAMBERT.ORG
+execution_ip=70.88.205.138
+hostname=NS1.GEORGELAMBERT.ORG
+utc=2026-09-12T05:33:03Z
+whoami=marchon
+pwd=/home/marchon/verae-src/verae-nats-cluster
+uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux
+nproc=40
+loadavg=9.02 8.82 8.44 6/3847 224483
+client_vmid=510
+nats_url=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+js_extra_memory=1
+compare_dir=/home/marchon/verae-src/verae-nats-cluster/results/20260912T051237Z
+--- nats 511 max_mem ---
+  max_mem: 8G
+--- nats 511 jetstream mount ---
+nats-js on /var/lib/nats/jetstream type tmpfs (rw,relatime,size=8388608k,uid=100000,gid=100000,inode64)
+--- free ---
+               total        used        free      shared  buff/cache   available
+Mem:           377Gi        38Gi       324Gi       4.3Gi        21Gi       339Gi
+Swap:          8.0Gi          0B       8.0Gi
+--- pct list ---
+VMID       Status     Lock         Name                
+100        stopped                 CT100               
+101        stopped                 CT101               
+102        stopped                 CT102               
+103        stopped                 CT103               
+104        stopped                 CT104               
+105        stopped                 CT105               
+106        stopped                 CT106               
+107        stopped                 CT107               
+108        stopped                 CT108               
+109        stopped                 CT109               
+110        stopped                 CT110               
+111        stopped                 CT111               
+112        stopped                 CT112               
+113        stopped                 CT113               
+114        stopped                 CT114               
+115        stopped                 CT115               
+116        stopped                 paul                
+117        stopped                 CT117               
+118        stopped                 CT118               
+119        stopped                 CT119               
+120        stopped                 rustdeskserver      
+121        stopped                 CT121               
+122        stopped                 CT122               
+123        stopped                 CT123               
+124        stopped                 bdross              
+200        stopped                 CT200               
+201        stopped                 CT201               
+202        stopped                 CT202               
+300        running                 mail.veraetime.info 
+400        stopped                 evercycle-api       
+501        running                 pki                 
+502        running                 peergos-a           
+503        running                 peergos-b           
+504        stopped                 minio               
+510        running                 verae-px-worker     
+511        running                 nats-a              
+512        running                 nats-b              
+513        running                 nats-c              
+2000       stopped                 CT2000              
+2001       stopped                 CT2001              
+2002       stopped                 CT2002              
+--- pct config 510 ---
+cores: 8
+hostname: verae-px-worker
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:BD:5D:62,ip=10.10.10.20/24,type=veth
+rootfs: SSD2:subvol-510-disk-0,size=16G
+swap: 512
+--- pct config 511 ---
+cores: 8
+hostname: nats-a
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:CF:AC:68,ip=10.10.10.21/24,type=veth
+rootfs: SSD2:subvol-511-disk-0,size=8G
+swap: 256
+--- pct config 512 ---
+cores: 8
+hostname: nats-b
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:42:73:A2,ip=10.10.10.22/24,type=veth
+rootfs: SSD2:subvol-512-disk-0,size=8G
+swap: 256
+--- pct config 513 ---
+cores: 8
+hostname: nats-c
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:1B:9B:DC,ip=10.10.10.23/24,type=veth
+rootfs: SSD2:subvol-513-disk-0,size=8G
+swap: 256

results/20260912T053120Z/host-before.txt

@@ -0,0 +1,92 @@
+execution_host=NS1.GEORGELAMBERT.ORG
+execution_ip=70.88.205.138
+hostname=NS1.GEORGELAMBERT.ORG
+utc=2026-09-12T05:31:20Z
+whoami=marchon
+pwd=/home/marchon/verae-src/verae-nats-cluster
+uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux
+nproc=40
+loadavg=8.77 8.39 8.26 6/3849 198471
+client_vmid=510
+nats_url=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+js_extra_memory=1
+compare_dir=/home/marchon/verae-src/verae-nats-cluster/results/20260912T051237Z
+--- nats 511 max_mem ---
+  max_mem: 8G
+--- nats 511 jetstream mount ---
+nats-js on /var/lib/nats/jetstream type tmpfs (rw,relatime,size=8388608k,uid=100000,gid=100000,inode64)
+--- free ---
+               total        used        free      shared  buff/cache   available
+Mem:           377Gi        38Gi       324Gi       4.2Gi        21Gi       339Gi
+Swap:          8.0Gi          0B       8.0Gi
+--- pct list ---
+VMID       Status     Lock         Name                
+100        stopped                 CT100               
+101        stopped                 CT101               
+102        stopped                 CT102               
+103        stopped                 CT103               
+104        stopped                 CT104               
+105        stopped                 CT105               
+106        stopped                 CT106               
+107        stopped                 CT107               
+108        stopped                 CT108               
+109        stopped                 CT109               
+110        stopped                 CT110               
+111        stopped                 CT111               
+112        stopped                 CT112               
+113        stopped                 CT113               
+114        stopped                 CT114               
+115        stopped                 CT115               
+116        stopped                 paul                
+117        stopped                 CT117               
+118        stopped                 CT118               
+119        stopped                 CT119               
+120        stopped                 rustdeskserver      
+121        stopped                 CT121               
+122        stopped                 CT122               
+123        stopped                 CT123               
+124        stopped                 bdross              
+200        stopped                 CT200               
+201        stopped                 CT201               
+202        stopped                 CT202               
+300        running                 mail.veraetime.info 
+400        stopped                 evercycle-api       
+501        running                 pki                 
+502        running                 peergos-a           
+503        running                 peergos-b           
+504        stopped                 minio               
+510        running                 verae-px-worker     
+511        running                 nats-a              
+512        running                 nats-b              
+513        running                 nats-c              
+2000       stopped                 CT2000              
+2001       stopped                 CT2001              
+2002       stopped                 CT2002              
+--- pct config 510 ---
+cores: 8
+hostname: verae-px-worker
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:BD:5D:62,ip=10.10.10.20/24,type=veth
+rootfs: SSD2:subvol-510-disk-0,size=16G
+swap: 512
+--- pct config 511 ---
+cores: 8
+hostname: nats-a
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:CF:AC:68,ip=10.10.10.21/24,type=veth
+rootfs: SSD2:subvol-511-disk-0,size=8G
+swap: 256
+--- pct config 512 ---
+cores: 8
+hostname: nats-b
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:42:73:A2,ip=10.10.10.22/24,type=veth
+rootfs: SSD2:subvol-512-disk-0,size=8G
+swap: 256
+--- pct config 513 ---
+cores: 8
+hostname: nats-c
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:1B:9B:DC,ip=10.10.10.23/24,type=veth
+rootfs: SSD2:subvol-513-disk-0,size=8G
+swap: 256

results/20260912T053120Z/js-1p-20k-128-r3.csv

@@ -0,0 +1,2 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+DsDjrcOoYbw8k54T1mMXxq,P0,40000,2560000,17388,2225703.026620,1.150198

results/20260912T053120Z/js-1p-20k-128-r3.txt

@@ -0,0 +1,5 @@
+=== js-1p-20k-128-r3 ===
+
+Pub stats: 17,388 msgs/sec ~ 2.12 MB/sec
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T053120Z/js-2p2s-20k-128-r3.csv

@@ -0,0 +1,5 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+JYqsv0sXzEqkC0r49q0aE5,S0,20203,1280000,5135,657394.514895,1.947080
+JYqsv0sXzEqkC0r49q0aE5,S1,20210,1280000,4971,636317.997133,2.011573
+JYqsv0sXzEqkC0r49q0aE5,P0,20000,1280000,5099,652788.600986,1.960819
+JYqsv0sXzEqkC0r49q0aE5,P1,20000,1280000,4979,637405.779512,2.008140