Snapshot of verae-nats-cluster (optimal NATS config study)

results/20260912T055851Z/BENCH.md

@@ -0,0 +1,58 @@
+# NATS cluster message speed
+
+Run **`20260912T055851Z`** (UTC). Client: LXC **510** `verae-px-worker` (`10.10.10.20`), not a nats-* server. Servers: `nats-a/b/c` on `10.10.10.21–23` (`vmbr1` only).
+
+Client URL:
+
+```text
+nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+```
+
+## Method
+
+- **Core NATS** is fire-and-forget pub/sub (`nats bench`). No disk, no replica ack.
+- **JetStream** uses **file** storage and **replicas=3** (same as product streams). The unique stream `benchstream` is deleted between JS loads.
+- Throughput is **msgs/sec** from nats CLI **0.1.6** (`--no-progress --csv`). Its min/avg/max are publisher **rate spread**, not delay.
+- **Ping** delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster.
+- **Flood** delay: N publishers dump the whole batch, then the subscriber drains. This is **queueing under burst**, not wire RTT.
+- Probe: `scripts/latency.mjs` (two connections, header timestamp).
+
+## Throughput
+
+| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
+|-----|------|------------------|------------|----------|------------|----------|
+| `core-1p1s-50k-128` | core pub/sub | 835,602 | 662,227 | 80.84 | 472,239 | 57.65 |
+| `core-4p4s-100k-128` | core pub/sub | 1,597,284 | 692,109 | 84.49 | 1,286,093 | 156.99 |
+| `core-4p4s-50k-1k` | core pub/sub | 595,555 | 189,939 | 185.49 | 479,898 | 468.65 |
+| `core-8p8s-200k-128` | core pub/sub | 1,760,599 | 223,078 | 27.23 | 1,570,171 | 191.67 |
+| `js-1p-20k-128-r3` | jetstream r=3 file | — | 14,330 | 1.75 | — | — |
+| `js-2p2s-20k-128-r3` | jetstream r=3 file | 18,449 | 9,252 | 1.13 | 9,228 | 1.13 |
+| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 15,197 | 14.84 | — | — |
+| `js-4p-50k-128-r3` | jetstream r=3 file | — | 19,232 | 2.35 | — | — |
+| `js-file-1p-20k-128-r1` | jetstream r=3 file | — | 18,888 | 2.31 | — | — |
+| `js-file-1p-20k-4k-r3` | jetstream r=3 file | — | 8,673 | 33.88 | — | — |
+| `js-file-4p-50k-128-r1` | jetstream r=3 file | — | 24,560 | 3.00 | — | — |
+| `js-mem-1p-20k-128-r1` | jetstream r=3 file | — | 29,972 | 3.66 | — | — |
+| `js-mem-1p-20k-128-r3` | jetstream r=3 file | — | 20,188 | 2.46 | — | — |
+| `js-mem-4p-20k-1k-r3` | jetstream r=3 file | — | 33,916 | 33.12 | — | — |
+| `js-mem-4p-50k-128-r1` | jetstream r=3 file | — | 64,923 | 7.93 | — | — |
+| `js-mem-4p-50k-128-r3` | jetstream r=3 file | — | 37,736 | 4.61 | — | — |
+
+## Round-trip delay
+
+| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
+|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|
+| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.340ms | 0.530ms | 0.456ms | 0.827ms | 1.140ms | 2.910ms |
+| `lat-reconnect-200-128` | flood (burst queueing) | 200 | 1 | 128 B | 0.366ms | 0.540ms | 0.503ms | 0.619ms | 1.750ms | 3.206ms |
+| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 98.399ms | 124.452ms | 125.413ms | 140.165ms | 145.770ms | 146.002ms |
+| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 125.336ms | 157.915ms | 157.392ms | 176.826ms | 178.015ms | 178.776ms |
+| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 157.463ms | 205.870ms | 207.044ms | 237.035ms | 239.449ms | 240.473ms |
+| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 230.898ms | 361.458ms | 372.512ms | 443.799ms | 458.198ms | 460.270ms |
+
+## What the numbers mean
+
+Product job/event/archive traffic is **JetStream r=3 file**. On this three-LXC stand that is about **16k durable 128 B pubs/s** (about **13k** at 1 KiB). Core NATS is an upper bound for non-durable fan-out: about **0.7–2.0M msgs/s** aggregate at 128 B, or **~630k msgs/s (~616 MB/s)** at 1 KiB with 4 publishers.
+
+A quiet request-reply is **~0.3 ms** average, **p99 < 1 ms**. Flood rows in the **150–500 ms** band are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers.
+
+Re-run on NS1: `bash scripts/bench.sh`. Raw logs/CSVs are under `results/<utc>/`.

results/20260912T055851Z/core-1p1s-50k-128.csv

@@ -0,0 +1,3 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+aWd091p1c7cXCtuUvyZSxt,S0,50000,6400000,472239,60446688.561339,0.105878
+aWd091p1c7cXCtuUvyZSxt,P0,50000,6400000,662227,84765130.062490,0.075503

results/20260912T055851Z/core-1p1s-50k-128.txt

@@ -0,0 +1,7 @@
+=== core-1p1s-50k-128 ===
+
+NATS Pub/Sub stats: 835,602 msgs/sec ~ 102.00 MB/sec
+ Pub stats: 662,227 msgs/sec ~ 80.84 MB/sec
+ Sub stats: 472,239 msgs/sec ~ 57.65 MB/sec
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T055851Z/core-4p4s-100k-128.csv

@@ -0,0 +1,9 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+MH2k3yW0YONZD0HiNpchpL,S0,100000,12800000,409118,52367181.312369,0.244428
+MH2k3yW0YONZD0HiNpchpL,S1,100000,12800000,321711,41179038.222383,0.310838
+MH2k3yW0YONZD0HiNpchpL,S2,100000,12800000,321716,41179732.151708,0.310833
+MH2k3yW0YONZD0HiNpchpL,S3,100000,12800000,321573,41161354.894041,0.310971
+MH2k3yW0YONZD0HiNpchpL,P0,25000,3200000,264684,33879607.417508,0.094452
+MH2k3yW0YONZD0HiNpchpL,P1,25000,3200000,243211,31131121.677273,0.102791
+MH2k3yW0YONZD0HiNpchpL,P2,25000,3200000,183537,23492805.654965,0.136212
+MH2k3yW0YONZD0HiNpchpL,P3,25000,3200000,173063,22152095.916748,0.144456

results/20260912T055851Z/core-4p4s-100k-128.txt

@@ -0,0 +1,17 @@
+=== core-4p4s-100k-128 ===
+
+NATS Pub/Sub stats: 1,597,284 msgs/sec ~ 194.98 MB/sec
+ Pub stats: 692,109 msgs/sec ~ 84.49 MB/sec
+  [1] 264,684 msgs/sec ~ 32.31 MB/sec (25000 msgs)
+  [2] 243,211 msgs/sec ~ 29.69 MB/sec (25000 msgs)
+  [3] 183,537 msgs/sec ~ 22.40 MB/sec (25000 msgs)
+  [4] 173,063 msgs/sec ~ 21.13 MB/sec (25000 msgs)
+  min 173,063 | avg 216,123 | max 264,684 | stddev 38,755 msgs
+ Sub stats: 1,286,093 msgs/sec ~ 156.99 MB/sec
+  [1] 409,118 msgs/sec ~ 49.94 MB/sec (100000 msgs)
+  [2] 321,711 msgs/sec ~ 39.27 MB/sec (100000 msgs)
+  [3] 321,716 msgs/sec ~ 39.27 MB/sec (100000 msgs)
+  [4] 321,573 msgs/sec ~ 39.25 MB/sec (100000 msgs)
+  min 321,573 | avg 343,529 | max 409,118 | stddev 37,867 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T055851Z/core-4p4s-50k-1k.csv

@@ -0,0 +1,9 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+I0O2O7NFVVnTrSOBs8Q38n,S0,50000,51200000,171931,176057368.403530,0.290814
+I0O2O7NFVVnTrSOBs8Q38n,S1,50000,51200000,127850,130919309.123975,0.391081
+I0O2O7NFVVnTrSOBs8Q38n,S2,50000,51200000,120555,123448409.014747,0.414748
+I0O2O7NFVVnTrSOBs8Q38n,S3,50000,51200000,120019,122899963.083444,0.416599
+I0O2O7NFVVnTrSOBs8Q38n,P0,12500,12800000,101551,103988522.006912,0.123091
+I0O2O7NFVVnTrSOBs8Q38n,P1,12500,12800000,49245,50427332.045753,0.253831
+I0O2O7NFVVnTrSOBs8Q38n,P2,12500,12800000,48521,49686273.879592,0.257616
+I0O2O7NFVVnTrSOBs8Q38n,P3,12500,12800000,47497,48637215.616680,0.263173

results/20260912T055851Z/core-4p4s-50k-1k.txt

@@ -0,0 +1,17 @@
+=== core-4p4s-50k-1k ===
+
+NATS Pub/Sub stats: 595,555 msgs/sec ~ 581.60 MB/sec
+ Pub stats: 189,939 msgs/sec ~ 185.49 MB/sec
+  [1] 101,551 msgs/sec ~ 99.17 MB/sec (12500 msgs)
+  [2] 49,245 msgs/sec ~ 48.09 MB/sec (12500 msgs)
+  [3] 48,521 msgs/sec ~ 47.38 MB/sec (12500 msgs)
+  [4] 47,497 msgs/sec ~ 46.38 MB/sec (12500 msgs)
+  min 47,497 | avg 61,703 | max 101,551 | stddev 23,014 msgs
+ Sub stats: 479,898 msgs/sec ~ 468.65 MB/sec
+  [1] 171,931 msgs/sec ~ 167.90 MB/sec (50000 msgs)
+  [2] 127,850 msgs/sec ~ 124.85 MB/sec (50000 msgs)
+  [3] 120,555 msgs/sec ~ 117.73 MB/sec (50000 msgs)
+  [4] 120,019 msgs/sec ~ 117.21 MB/sec (50000 msgs)
+  min 120,019 | avg 135,088 | max 171,931 | stddev 21,494 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T055851Z/core-8p8s-200k-128.csv

@@ -0,0 +1,17 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+PZHMggVrmAZAaABqkdVm12,S0,200000,25600000,221384,28337251.548719,0.903404
+PZHMggVrmAZAaABqkdVm12,S1,200000,25600000,221400,28339297.871777,0.903339
+PZHMggVrmAZAaABqkdVm12,S2,200000,25600000,221358,28333840.935200,0.903513
+PZHMggVrmAZAaABqkdVm12,S3,200000,25600000,196749,25183987.052629,1.016519
+PZHMggVrmAZAaABqkdVm12,S4,200000,25600000,196654,25171832.144418,1.017010
+PZHMggVrmAZAaABqkdVm12,S5,200000,25600000,196517,25154300.186621,1.017719
+PZHMggVrmAZAaABqkdVm12,S6,200000,25600000,196490,25150741.114236,1.017863
+PZHMggVrmAZAaABqkdVm12,S7,200000,25600000,196395,25138656.210795,1.018352
+PZHMggVrmAZAaABqkdVm12,P0,25000,3200000,169409,21684439.538250,0.147571
+PZHMggVrmAZAaABqkdVm12,P1,25000,3200000,137478,17597297.160740,0.181846
+PZHMggVrmAZAaABqkdVm12,P2,25000,3200000,109841,14059658.143542,0.227602
+PZHMggVrmAZAaABqkdVm12,P3,25000,3200000,96040,12293171.802285,0.260307
+PZHMggVrmAZAaABqkdVm12,P4,25000,3200000,75019,9602540.006267,0.333245
+PZHMggVrmAZAaABqkdVm12,P5,25000,3200000,73607,9421821.419509,0.339637
+PZHMggVrmAZAaABqkdVm12,P6,25000,3200000,27890,3569920.156650,0.896379
+PZHMggVrmAZAaABqkdVm12,P7,25000,3200000,27932,3575348.254450,0.895018

results/20260912T055851Z/core-8p8s-200k-128.txt

@@ -0,0 +1,25 @@
+=== core-8p8s-200k-128 ===
+
+NATS Pub/Sub stats: 1,760,599 msgs/sec ~ 214.92 MB/sec
+ Pub stats: 223,078 msgs/sec ~ 27.23 MB/sec
+  [1] 169,409 msgs/sec ~ 20.68 MB/sec (25000 msgs)
+  [2] 137,478 msgs/sec ~ 16.78 MB/sec (25000 msgs)
+  [3] 109,841 msgs/sec ~ 13.41 MB/sec (25000 msgs)
+  [4] 96,040 msgs/sec ~ 11.72 MB/sec (25000 msgs)
+  [5] 75,019 msgs/sec ~ 9.16 MB/sec (25000 msgs)
+  [6] 73,607 msgs/sec ~ 8.99 MB/sec (25000 msgs)
+  [7] 27,890 msgs/sec ~ 3.40 MB/sec (25000 msgs)
+  [8] 27,932 msgs/sec ~ 3.41 MB/sec (25000 msgs)
+  min 27,890 | avg 89,652 | max 169,409 | stddev 46,357 msgs
+ Sub stats: 1,570,171 msgs/sec ~ 191.67 MB/sec
+  [1] 221,384 msgs/sec ~ 27.02 MB/sec (200000 msgs)
+  [2] 221,400 msgs/sec ~ 27.03 MB/sec (200000 msgs)
+  [3] 221,358 msgs/sec ~ 27.02 MB/sec (200000 msgs)
+  [4] 196,749 msgs/sec ~ 24.02 MB/sec (200000 msgs)
+  [5] 196,654 msgs/sec ~ 24.01 MB/sec (200000 msgs)
+  [6] 196,517 msgs/sec ~ 23.99 MB/sec (200000 msgs)
+  [7] 196,490 msgs/sec ~ 23.99 MB/sec (200000 msgs)
+  [8] 196,395 msgs/sec ~ 23.97 MB/sec (200000 msgs)
+  min 196,395 | avg 205,868 | max 221,400 | stddev 12,016 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T055851Z/host-after.txt

@@ -0,0 +1,92 @@
+execution_host=NS1.GEORGELAMBERT.ORG
+execution_ip=70.88.205.138
+hostname=NS1.GEORGELAMBERT.ORG
+utc=2026-09-12T06:01:17Z
+whoami=marchon
+pwd=/home/marchon/verae-src/verae-nats-cluster
+uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux
+nproc=40
+loadavg=8.48 8.52 8.17 5/3860 657839
+client_vmid=510
+nats_url=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+js_extra_memory=1
+compare_dir=/home/marchon/verae-src/verae-nats-cluster/results/20260912T051237Z
+--- nats 511 max_mem ---
+  max_mem: 8G
+--- nats 511 jetstream mount ---
+jetstream on rootfs
+--- free ---
+               total        used        free      shared  buff/cache   available
+Mem:           377Gi        38Gi       324Gi       4.3Gi        21Gi       338Gi
+Swap:          8.0Gi          0B       8.0Gi
+--- pct list ---
+VMID       Status     Lock         Name                
+100        stopped                 CT100               
+101        stopped                 CT101               
+102        stopped                 CT102               
+103        stopped                 CT103               
+104        stopped                 CT104               
+105        stopped                 CT105               
+106        stopped                 CT106               
+107        stopped                 CT107               
+108        stopped                 CT108               
+109        stopped                 CT109               
+110        stopped                 CT110               
+111        stopped                 CT111               
+112        stopped                 CT112               
+113        stopped                 CT113               
+114        stopped                 CT114               
+115        stopped                 CT115               
+116        stopped                 paul                
+117        stopped                 CT117               
+118        stopped                 CT118               
+119        stopped                 CT119               
+120        stopped                 rustdeskserver      
+121        stopped                 CT121               
+122        stopped                 CT122               
+123        stopped                 CT123               
+124        stopped                 bdross              
+200        stopped                 CT200               
+201        stopped                 CT201               
+202        stopped                 CT202               
+300        running                 mail.veraetime.info 
+400        stopped                 evercycle-api       
+501        running                 pki                 
+502        running                 peergos-a           
+503        running                 peergos-b           
+504        stopped                 minio               
+510        running                 verae-px-worker     
+511        running                 nats-a              
+512        running                 nats-b              
+513        running                 nats-c              
+2000       stopped                 CT2000              
+2001       stopped                 CT2001              
+2002       stopped                 CT2002              
+--- pct config 510 ---
+cores: 8
+hostname: verae-px-worker
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:BD:5D:62,ip=10.10.10.20/24,type=veth
+rootfs: SSD2:subvol-510-disk-0,size=16G
+swap: 512
+--- pct config 511 ---
+cores: 8
+hostname: nats-a
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:CF:AC:68,ip=10.10.10.21/24,type=veth
+rootfs: SSD2:subvol-511-disk-0,size=8G
+swap: 256
+--- pct config 512 ---
+cores: 8
+hostname: nats-b
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:42:73:A2,ip=10.10.10.22/24,type=veth
+rootfs: SSD2:subvol-512-disk-0,size=8G
+swap: 256
+--- pct config 513 ---
+cores: 8
+hostname: nats-c
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:1B:9B:DC,ip=10.10.10.23/24,type=veth
+rootfs: SSD2:subvol-513-disk-0,size=8G
+swap: 256

results/20260912T055851Z/host-before.txt

@@ -0,0 +1,92 @@
+execution_host=NS1.GEORGELAMBERT.ORG
+execution_ip=70.88.205.138
+hostname=NS1.GEORGELAMBERT.ORG
+utc=2026-09-12T05:58:55Z
+whoami=marchon
+pwd=/home/marchon/verae-src/verae-nats-cluster
+uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux
+nproc=40
+loadavg=8.70 8.15 8.00 4/3842 621042
+client_vmid=510
+nats_url=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+js_extra_memory=1
+compare_dir=/home/marchon/verae-src/verae-nats-cluster/results/20260912T051237Z
+--- nats 511 max_mem ---
+  max_mem: 8G
+--- nats 511 jetstream mount ---
+jetstream on rootfs
+--- free ---
+               total        used        free      shared  buff/cache   available
+Mem:           377Gi        38Gi       324Gi       4.3Gi        21Gi       339Gi
+Swap:          8.0Gi          0B       8.0Gi
+--- pct list ---
+VMID       Status     Lock         Name                
+100        stopped                 CT100               
+101        stopped                 CT101               
+102        stopped                 CT102               
+103        stopped                 CT103               
+104        stopped                 CT104               
+105        stopped                 CT105               
+106        stopped                 CT106               
+107        stopped                 CT107               
+108        stopped                 CT108               
+109        stopped                 CT109               
+110        stopped                 CT110               
+111        stopped                 CT111               
+112        stopped                 CT112               
+113        stopped                 CT113               
+114        stopped                 CT114               
+115        stopped                 CT115               
+116        stopped                 paul                
+117        stopped                 CT117               
+118        stopped                 CT118               
+119        stopped                 CT119               
+120        stopped                 rustdeskserver      
+121        stopped                 CT121               
+122        stopped                 CT122               
+123        stopped                 CT123               
+124        stopped                 bdross              
+200        stopped                 CT200               
+201        stopped                 CT201               
+202        stopped                 CT202               
+300        running                 mail.veraetime.info 
+400        stopped                 evercycle-api       
+501        running                 pki                 
+502        running                 peergos-a           
+503        running                 peergos-b           
+504        stopped                 minio               
+510        running                 verae-px-worker     
+511        running                 nats-a              
+512        running                 nats-b              
+513        running                 nats-c              
+2000       stopped                 CT2000              
+2001       stopped                 CT2001              
+2002       stopped                 CT2002              
+--- pct config 510 ---
+cores: 8
+hostname: verae-px-worker
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:BD:5D:62,ip=10.10.10.20/24,type=veth
+rootfs: SSD2:subvol-510-disk-0,size=16G
+swap: 512
+--- pct config 511 ---
+cores: 8
+hostname: nats-a
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:CF:AC:68,ip=10.10.10.21/24,type=veth
+rootfs: SSD2:subvol-511-disk-0,size=8G
+swap: 256
+--- pct config 512 ---
+cores: 8
+hostname: nats-b
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:42:73:A2,ip=10.10.10.22/24,type=veth
+rootfs: SSD2:subvol-512-disk-0,size=8G
+swap: 256
+--- pct config 513 ---
+cores: 8
+hostname: nats-c
+memory: 16384
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:1B:9B:DC,ip=10.10.10.23/24,type=veth
+rootfs: SSD2:subvol-513-disk-0,size=8G
+swap: 256

results/20260912T055851Z/js-1p-20k-128-r3.csv

@@ -0,0 +1,2 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+vSyjynJZ5N37lo74IJMJLi,P0,40000,2560000,14330,1834259.456665,1.395659