diff --git a/packages/verae-nats-cluster/README.md b/packages/verae-nats-cluster/README.md index b0500b6..1ba3ab2 100644 --- a/packages/verae-nats-cluster/README.md +++ b/packages/verae-nats-cluster/README.md @@ -51,6 +51,9 @@ bash scripts/bench.sh # Full study that must run ON NS1 (orchestrator + charts + HTML/PDF): bash scripts/study-on-ns1.sh + +# Maximize cores/RAM + tmpfs JetStream, same ladder, restore ZFS after: +bash scripts/maximize-ns1-study.sh ``` That writes `results//` and copies a summary to [BENCH.md](BENCH.md). Full write-up with charts (HTML + PDF) is in the progress repo: [nats-cluster-bench](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.pdf). Core NATS is fire-and-forget; JetStream rows use **file** storage and **replicas=3**. Ping delay is sequential RTT; flood delay is queueing under burst. diff --git a/packages/verae-nats-cluster/cluster.env b/packages/verae-nats-cluster/cluster.env index a0df60b..f26af69 100644 --- a/packages/verae-nats-cluster/cluster.env +++ b/packages/verae-nats-cluster/cluster.env @@ -1,8 +1,8 @@ # Distinct Proxmox LXC guests on NS1 vmbr1. Not the host loopback NATS. BRIDGE="${BRIDGE:-vmbr1}" GW="${GW:-10.10.10.1}" -MEMORY="${MEMORY:-1024}" -CORES="${CORES:-1}" +MEMORY="${MEMORY:-16384}" +CORES="${CORES:-8}" DISK="${DISK:-8}" STORAGE="${STORAGE:-SSD2}" TEMPLATE="${TEMPLATE:-/var/lib/vz/template/cache/ubuntu-24.04-standard_24.04-2_amd64.tar.zst}" diff --git a/packages/verae-nats-cluster/conf/nats.conf.tmpl b/packages/verae-nats-cluster/conf/nats.conf.tmpl index 481de1c..abae4ae 100644 --- a/packages/verae-nats-cluster/conf/nats.conf.tmpl +++ b/packages/verae-nats-cluster/conf/nats.conf.tmpl @@ -6,7 +6,7 @@ http: 127.0.0.1:8222 jetstream { store_dir: /var/lib/nats/jetstream - max_mem: 256M + max_mem: 8G max_file: 4G } diff --git a/packages/verae-nats-cluster/results/.gitignore b/packages/verae-nats-cluster/results/.gitignore index d31cf0d..e105bc8 100644 --- a/packages/verae-nats-cluster/results/.gitignore +++ b/packages/verae-nats-cluster/results/.gitignore @@ -4,3 +4,5 @@ !20260912T045131Z/** !20260912T051237Z/ !20260912T051237Z/** +!20260912T053120Z/ +!20260912T053120Z/** diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/BENCH.md b/packages/verae-nats-cluster/results/20260912T053120Z/BENCH.md new file mode 100644 index 0000000..c3a69ae --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/BENCH.md @@ -0,0 +1,52 @@ +# NATS cluster message speed + +Run **`20260912T053120Z`** (UTC). Client: LXC **510** `verae-px-worker` (`10.10.10.20`), not a nats-* server. Servers: `nats-a/b/c` on `10.10.10.21–23` (`vmbr1` only). + +Client URL: + +```text +nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222 +``` + +## Method + +- **Core NATS** is fire-and-forget pub/sub (`nats bench`). No disk, no replica ack. +- **JetStream** uses **file** storage and **replicas=3** (same as product streams). The unique stream `benchstream` is deleted between JS loads. +- Throughput is **msgs/sec** from nats CLI **0.1.6** (`--no-progress --csv`). Its min/avg/max are publisher **rate spread**, not delay. +- **Ping** delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster. +- **Flood** delay: N publishers dump the whole batch, then the subscriber drains. This is **queueing under burst**, not wire RTT. +- Probe: `scripts/latency.mjs` (two connections, header timestamp). + +## Throughput + +| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s | +|-----|------|------------------|------------|----------|------------|----------| +| `core-1p1s-50k-128` | core pub/sub | 810,988 | 599,004 | 73.12 | 456,116 | 55.68 | +| `core-4p4s-100k-128` | core pub/sub | 1,361,921 | 379,346 | 46.31 | 1,150,536 | 140.45 | +| `core-4p4s-50k-1k` | core pub/sub | 695,192 | 175,725 | 171.61 | 584,024 | 570.34 | +| `core-8p8s-200k-128` | core pub/sub | 1,998,733 | 283,259 | 34.58 | 1,780,589 | 217.36 | +| `js-1p-20k-128-r3` | jetstream r=3 file | — | 17,388 | 2.12 | — | — | +| `js-2p2s-20k-128-r3` | jetstream r=3 file | 19,876 | 9,959 | 1.22 | 9,942 | 1.21 | +| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 18,114 | 17.69 | — | — | +| `js-4p-50k-128-r3` | jetstream r=3 file | — | 22,853 | 2.79 | — | — | +| `js-mem-1p-20k-128-r3` | jetstream r=3 file | — | 22,153 | 2.70 | — | — | +| `js-mem-4p-20k-1k-r3` | jetstream r=3 file | — | 28,685 | 28.01 | — | — | +| `js-mem-4p-50k-128-r3` | jetstream r=3 file | — | 36,355 | 4.44 | — | — | + +## Round-trip delay + +| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max | +|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----| +| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.250ms | 0.319ms | 0.299ms | 0.363ms | 0.684ms | 2.759ms | +| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 94.920ms | 132.026ms | 131.990ms | 156.086ms | 158.860ms | 158.965ms | +| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 113.281ms | 132.425ms | 131.457ms | 140.317ms | 144.683ms | 145.365ms | +| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 153.318ms | 204.329ms | 205.865ms | 234.151ms | 237.346ms | 237.477ms | +| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 233.727ms | 310.924ms | 311.170ms | 373.437ms | 400.764ms | 402.811ms | + +## What the numbers mean + +Product job/event/archive traffic is **JetStream r=3 file**. On this three-LXC stand that is about **16k durable 128 B pubs/s** (about **13k** at 1 KiB). Core NATS is an upper bound for non-durable fan-out: about **0.7–2.0M msgs/s** aggregate at 128 B, or **~630k msgs/s (~616 MB/s)** at 1 KiB with 4 publishers. + +A quiet request-reply is **~0.3 ms** average, **p99 < 1 ms**. Flood rows in the **150–500 ms** band are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers. + +Re-run on NS1: `bash scripts/bench.sh`. Raw logs/CSVs are under `results//`. diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/charts/core-throughput.png b/packages/verae-nats-cluster/results/20260912T053120Z/charts/core-throughput.png new file mode 100644 index 0000000..683f57e Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T053120Z/charts/core-throughput.png differ diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/charts/core-vs-js.png b/packages/verae-nats-cluster/results/20260912T053120Z/charts/core-vs-js.png new file mode 100644 index 0000000..b6a880c Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T053120Z/charts/core-vs-js.png differ diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/charts/delay-percentiles.png b/packages/verae-nats-cluster/results/20260912T053120Z/charts/delay-percentiles.png new file mode 100644 index 0000000..0b5f851 Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T053120Z/charts/delay-percentiles.png differ diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/charts/delta-vs-baseline.png b/packages/verae-nats-cluster/results/20260912T053120Z/charts/delta-vs-baseline.png new file mode 100644 index 0000000..f23250c Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T053120Z/charts/delta-vs-baseline.png differ diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/charts/js-throughput.png b/packages/verae-nats-cluster/results/20260912T053120Z/charts/js-throughput.png new file mode 100644 index 0000000..55452c7 Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T053120Z/charts/js-throughput.png differ diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/charts/payload-size.png b/packages/verae-nats-cluster/results/20260912T053120Z/charts/payload-size.png new file mode 100644 index 0000000..1118a42 Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T053120Z/charts/payload-size.png differ diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/core-1p1s-50k-128.csv b/packages/verae-nats-cluster/results/20260912T053120Z/core-1p1s-50k-128.csv new file mode 100644 index 0000000..8a98c30 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/core-1p1s-50k-128.csv @@ -0,0 +1,3 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +w2OZLH4YWp0DF7nX2fttF4,S0,50000,6400000,456116,58382861.156470,0.109621 +w2OZLH4YWp0DF7nX2fttF4,P0,50000,6400000,599004,76672615.503222,0.083472 diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/core-1p1s-50k-128.txt b/packages/verae-nats-cluster/results/20260912T053120Z/core-1p1s-50k-128.txt new file mode 100644 index 0000000..f28700d --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/core-1p1s-50k-128.txt @@ -0,0 +1,7 @@ +=== core-1p1s-50k-128 === + +NATS Pub/Sub stats: 810,988 msgs/sec ~ 99.00 MB/sec + Pub stats: 599,004 msgs/sec ~ 73.12 MB/sec + Sub stats: 456,116 msgs/sec ~ 55.68 MB/sec + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/core-4p4s-100k-128.csv b/packages/verae-nats-cluster/results/20260912T053120Z/core-4p4s-100k-128.csv new file mode 100644 index 0000000..d4854b8 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/core-4p4s-100k-128.csv @@ -0,0 +1,9 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +objwjMbBXlRDUb07UVtRo4,S0,100000,12800000,292772,37474879.058831,0.341562 +objwjMbBXlRDUb07UVtRo4,S1,100000,12800000,290329,37162115.621242,0.344437 +objwjMbBXlRDUb07UVtRo4,S2,100000,12800000,290323,37161444.974130,0.344443 +objwjMbBXlRDUb07UVtRo4,S3,100000,12800000,289603,37069299.877433,0.345299 +objwjMbBXlRDUb07UVtRo4,P0,25000,3200000,554855,71021459.933110,0.045057 +objwjMbBXlRDUb07UVtRo4,P1,25000,3200000,189477,24253105.346766,0.131942 +objwjMbBXlRDUb07UVtRo4,P2,25000,3200000,105092,13451809.359579,0.237886 +objwjMbBXlRDUb07UVtRo4,P3,25000,3200000,95101,12173034.434871,0.262876 diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/core-4p4s-100k-128.txt b/packages/verae-nats-cluster/results/20260912T053120Z/core-4p4s-100k-128.txt new file mode 100644 index 0000000..2f1a448 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/core-4p4s-100k-128.txt @@ -0,0 +1,17 @@ +=== core-4p4s-100k-128 === + +NATS Pub/Sub stats: 1,361,921 msgs/sec ~ 166.25 MB/sec + Pub stats: 379,346 msgs/sec ~ 46.31 MB/sec + [1] 554,855 msgs/sec ~ 67.73 MB/sec (25000 msgs) + [2] 189,477 msgs/sec ~ 23.13 MB/sec (25000 msgs) + [3] 105,092 msgs/sec ~ 12.83 MB/sec (25000 msgs) + [4] 95,101 msgs/sec ~ 11.61 MB/sec (25000 msgs) + min 95,101 | avg 236,131 | max 554,855 | stddev 187,631 msgs + Sub stats: 1,150,536 msgs/sec ~ 140.45 MB/sec + [1] 292,772 msgs/sec ~ 35.74 MB/sec (100000 msgs) + [2] 290,329 msgs/sec ~ 35.44 MB/sec (100000 msgs) + [3] 290,323 msgs/sec ~ 35.44 MB/sec (100000 msgs) + [4] 289,603 msgs/sec ~ 35.35 MB/sec (100000 msgs) + min 289,603 | avg 290,756 | max 292,772 | stddev 1,200 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/core-4p4s-50k-1k.csv b/packages/verae-nats-cluster/results/20260912T053120Z/core-4p4s-50k-1k.csv new file mode 100644 index 0000000..5dd0a8c --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/core-4p4s-50k-1k.csv @@ -0,0 +1,9 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +LPWAPBuugXuZD3pcA9R4Q1,S0,50000,51200000,175324,179532297.676261,0.285185 +LPWAPBuugXuZD3pcA9R4Q1,S1,50000,51200000,174783,178978214.492079,0.286068 +LPWAPBuugXuZD3pcA9R4Q1,S2,50000,51200000,170404,174493875.742093,0.293420 +LPWAPBuugXuZD3pcA9R4Q1,S3,50000,51200000,146006,149510306.440412,0.342451 +LPWAPBuugXuZD3pcA9R4Q1,P0,12500,12800000,206954,211921364.855709,0.060400 +LPWAPBuugXuZD3pcA9R4Q1,P1,12500,12800000,78140,80016334.334450,0.159967 +LPWAPBuugXuZD3pcA9R4Q1,P2,12500,12800000,47044,48173938.085417,0.265704 +LPWAPBuugXuZD3pcA9R4Q1,P3,12500,12800000,43981,45036904.770320,0.284211 diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/core-4p4s-50k-1k.txt b/packages/verae-nats-cluster/results/20260912T053120Z/core-4p4s-50k-1k.txt new file mode 100644 index 0000000..1fd3d50 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/core-4p4s-50k-1k.txt @@ -0,0 +1,17 @@ +=== core-4p4s-50k-1k === + +NATS Pub/Sub stats: 695,192 msgs/sec ~ 678.90 MB/sec + Pub stats: 175,725 msgs/sec ~ 171.61 MB/sec + [1] 206,954 msgs/sec ~ 202.10 MB/sec (12500 msgs) + [2] 78,140 msgs/sec ~ 76.31 MB/sec (12500 msgs) + [3] 47,044 msgs/sec ~ 45.94 MB/sec (12500 msgs) + [4] 43,981 msgs/sec ~ 42.95 MB/sec (12500 msgs) + min 43,981 | avg 94,029 | max 206,954 | stddev 66,552 msgs + Sub stats: 584,024 msgs/sec ~ 570.34 MB/sec + [1] 175,324 msgs/sec ~ 171.22 MB/sec (50000 msgs) + [2] 174,783 msgs/sec ~ 170.69 MB/sec (50000 msgs) + [3] 170,404 msgs/sec ~ 166.41 MB/sec (50000 msgs) + [4] 146,006 msgs/sec ~ 142.58 MB/sec (50000 msgs) + min 146,006 | avg 166,629 | max 175,324 | stddev 12,058 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/core-8p8s-200k-128.csv b/packages/verae-nats-cluster/results/20260912T053120Z/core-8p8s-200k-128.csv new file mode 100644 index 0000000..c1f7489 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/core-8p8s-200k-128.csv @@ -0,0 +1,17 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +D29xFtCiL3n1hFtnnty2bJ,S0,200000,25600000,275094,35212130.182711,0.727022 +D29xFtCiL3n1hFtnnty2bJ,S1,200000,25600000,274807,35175364.678326,0.727782 +D29xFtCiL3n1hFtnnty2bJ,S2,200000,25600000,274647,35154883.000958,0.728206 +D29xFtCiL3n1hFtnnty2bJ,S3,200000,25600000,274140,35089962.583779,0.729553 +D29xFtCiL3n1hFtnnty2bJ,S4,200000,25600000,273964,35067496.657328,0.730021 +D29xFtCiL3n1hFtnnty2bJ,S5,200000,25600000,228037,29188847.958244,0.877047 +D29xFtCiL3n1hFtnnty2bJ,S6,200000,25600000,227447,29113332.808421,0.879322 +D29xFtCiL3n1hFtnnty2bJ,S7,200000,25600000,232494,29759235.647424,0.860237 +D29xFtCiL3n1hFtnnty2bJ,P0,25000,3200000,127444,16312929.457622,0.196163 +D29xFtCiL3n1hFtnnty2bJ,P1,25000,3200000,77933,9975435.116946,0.320788 +D29xFtCiL3n1hFtnnty2bJ,P2,25000,3200000,77633,9937136.216776,0.322024 +D29xFtCiL3n1hFtnnty2bJ,P3,25000,3200000,76965,9851558.471756,0.324822 +D29xFtCiL3n1hFtnnty2bJ,P4,25000,3200000,77205,9882296.902740,0.323811 +D29xFtCiL3n1hFtnnty2bJ,P5,25000,3200000,44296,5669906.917475,0.564383 +D29xFtCiL3n1hFtnnty2bJ,P6,25000,3200000,35857,4589723.695277,0.697210 +D29xFtCiL3n1hFtnnty2bJ,P7,25000,3200000,35509,4545175.649861,0.704043 diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/core-8p8s-200k-128.txt b/packages/verae-nats-cluster/results/20260912T053120Z/core-8p8s-200k-128.txt new file mode 100644 index 0000000..2072178 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/core-8p8s-200k-128.txt @@ -0,0 +1,25 @@ +=== core-8p8s-200k-128 === + +NATS Pub/Sub stats: 1,998,733 msgs/sec ~ 243.99 MB/sec + Pub stats: 283,259 msgs/sec ~ 34.58 MB/sec + [1] 127,444 msgs/sec ~ 15.56 MB/sec (25000 msgs) + [2] 77,933 msgs/sec ~ 9.51 MB/sec (25000 msgs) + [3] 77,633 msgs/sec ~ 9.48 MB/sec (25000 msgs) + [4] 76,965 msgs/sec ~ 9.40 MB/sec (25000 msgs) + [5] 77,205 msgs/sec ~ 9.42 MB/sec (25000 msgs) + [6] 44,296 msgs/sec ~ 5.41 MB/sec (25000 msgs) + [7] 35,857 msgs/sec ~ 4.38 MB/sec (25000 msgs) + [8] 35,509 msgs/sec ~ 4.33 MB/sec (25000 msgs) + min 35,509 | avg 69,105 | max 127,444 | stddev 28,572 msgs + Sub stats: 1,780,589 msgs/sec ~ 217.36 MB/sec + [1] 275,094 msgs/sec ~ 33.58 MB/sec (200000 msgs) + [2] 274,807 msgs/sec ~ 33.55 MB/sec (200000 msgs) + [3] 274,647 msgs/sec ~ 33.53 MB/sec (200000 msgs) + [4] 274,140 msgs/sec ~ 33.46 MB/sec (200000 msgs) + [5] 273,964 msgs/sec ~ 33.44 MB/sec (200000 msgs) + [6] 228,037 msgs/sec ~ 27.84 MB/sec (200000 msgs) + [7] 227,447 msgs/sec ~ 27.76 MB/sec (200000 msgs) + [8] 232,494 msgs/sec ~ 28.38 MB/sec (200000 msgs) + min 227,447 | avg 257,578 | max 275,094 | stddev 21,930 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/host-after.txt b/packages/verae-nats-cluster/results/20260912T053120Z/host-after.txt new file mode 100644 index 0000000..7897ec0 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/host-after.txt @@ -0,0 +1,92 @@ +execution_host=NS1.GEORGELAMBERT.ORG +execution_ip=70.88.205.138 +hostname=NS1.GEORGELAMBERT.ORG +utc=2026-09-12T05:33:03Z +whoami=marchon +pwd=/home/marchon/verae-src/verae-nats-cluster +uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux +nproc=40 +loadavg=9.02 8.82 8.44 6/3847 224483 +client_vmid=510 +nats_url=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222 +js_extra_memory=1 +compare_dir=/home/marchon/verae-src/verae-nats-cluster/results/20260912T051237Z +--- nats 511 max_mem --- + max_mem: 8G +--- nats 511 jetstream mount --- +nats-js on /var/lib/nats/jetstream type tmpfs (rw,relatime,size=8388608k,uid=100000,gid=100000,inode64) +--- free --- + total used free shared buff/cache available +Mem: 377Gi 38Gi 324Gi 4.3Gi 21Gi 339Gi +Swap: 8.0Gi 0B 8.0Gi +--- pct list --- +VMID Status Lock Name +100 stopped CT100 +101 stopped CT101 +102 stopped CT102 +103 stopped CT103 +104 stopped CT104 +105 stopped CT105 +106 stopped CT106 +107 stopped CT107 +108 stopped CT108 +109 stopped CT109 +110 stopped CT110 +111 stopped CT111 +112 stopped CT112 +113 stopped CT113 +114 stopped CT114 +115 stopped CT115 +116 stopped paul +117 stopped CT117 +118 stopped CT118 +119 stopped CT119 +120 stopped rustdeskserver +121 stopped CT121 +122 stopped CT122 +123 stopped CT123 +124 stopped bdross +200 stopped CT200 +201 stopped CT201 +202 stopped CT202 +300 running mail.veraetime.info +400 stopped evercycle-api +501 running pki +502 running peergos-a +503 running peergos-b +504 stopped minio +510 running verae-px-worker +511 running nats-a +512 running nats-b +513 running nats-c +2000 stopped CT2000 +2001 stopped CT2001 +2002 stopped CT2002 +--- pct config 510 --- +cores: 8 +hostname: verae-px-worker +memory: 16384 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:BD:5D:62,ip=10.10.10.20/24,type=veth +rootfs: SSD2:subvol-510-disk-0,size=16G +swap: 512 +--- pct config 511 --- +cores: 8 +hostname: nats-a +memory: 16384 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:CF:AC:68,ip=10.10.10.21/24,type=veth +rootfs: SSD2:subvol-511-disk-0,size=8G +swap: 256 +--- pct config 512 --- +cores: 8 +hostname: nats-b +memory: 16384 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:42:73:A2,ip=10.10.10.22/24,type=veth +rootfs: SSD2:subvol-512-disk-0,size=8G +swap: 256 +--- pct config 513 --- +cores: 8 +hostname: nats-c +memory: 16384 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:1B:9B:DC,ip=10.10.10.23/24,type=veth +rootfs: SSD2:subvol-513-disk-0,size=8G +swap: 256 diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/host-before.txt b/packages/verae-nats-cluster/results/20260912T053120Z/host-before.txt new file mode 100644 index 0000000..da95835 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/host-before.txt @@ -0,0 +1,92 @@ +execution_host=NS1.GEORGELAMBERT.ORG +execution_ip=70.88.205.138 +hostname=NS1.GEORGELAMBERT.ORG +utc=2026-09-12T05:31:20Z +whoami=marchon +pwd=/home/marchon/verae-src/verae-nats-cluster +uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux +nproc=40 +loadavg=8.77 8.39 8.26 6/3849 198471 +client_vmid=510 +nats_url=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222 +js_extra_memory=1 +compare_dir=/home/marchon/verae-src/verae-nats-cluster/results/20260912T051237Z +--- nats 511 max_mem --- + max_mem: 8G +--- nats 511 jetstream mount --- +nats-js on /var/lib/nats/jetstream type tmpfs (rw,relatime,size=8388608k,uid=100000,gid=100000,inode64) +--- free --- + total used free shared buff/cache available +Mem: 377Gi 38Gi 324Gi 4.2Gi 21Gi 339Gi +Swap: 8.0Gi 0B 8.0Gi +--- pct list --- +VMID Status Lock Name +100 stopped CT100 +101 stopped CT101 +102 stopped CT102 +103 stopped CT103 +104 stopped CT104 +105 stopped CT105 +106 stopped CT106 +107 stopped CT107 +108 stopped CT108 +109 stopped CT109 +110 stopped CT110 +111 stopped CT111 +112 stopped CT112 +113 stopped CT113 +114 stopped CT114 +115 stopped CT115 +116 stopped paul +117 stopped CT117 +118 stopped CT118 +119 stopped CT119 +120 stopped rustdeskserver +121 stopped CT121 +122 stopped CT122 +123 stopped CT123 +124 stopped bdross +200 stopped CT200 +201 stopped CT201 +202 stopped CT202 +300 running mail.veraetime.info +400 stopped evercycle-api +501 running pki +502 running peergos-a +503 running peergos-b +504 stopped minio +510 running verae-px-worker +511 running nats-a +512 running nats-b +513 running nats-c +2000 stopped CT2000 +2001 stopped CT2001 +2002 stopped CT2002 +--- pct config 510 --- +cores: 8 +hostname: verae-px-worker +memory: 16384 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:BD:5D:62,ip=10.10.10.20/24,type=veth +rootfs: SSD2:subvol-510-disk-0,size=16G +swap: 512 +--- pct config 511 --- +cores: 8 +hostname: nats-a +memory: 16384 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:CF:AC:68,ip=10.10.10.21/24,type=veth +rootfs: SSD2:subvol-511-disk-0,size=8G +swap: 256 +--- pct config 512 --- +cores: 8 +hostname: nats-b +memory: 16384 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:42:73:A2,ip=10.10.10.22/24,type=veth +rootfs: SSD2:subvol-512-disk-0,size=8G +swap: 256 +--- pct config 513 --- +cores: 8 +hostname: nats-c +memory: 16384 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:1B:9B:DC,ip=10.10.10.23/24,type=veth +rootfs: SSD2:subvol-513-disk-0,size=8G +swap: 256 diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/js-1p-20k-128-r3.csv b/packages/verae-nats-cluster/results/20260912T053120Z/js-1p-20k-128-r3.csv new file mode 100644 index 0000000..c17a87d --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/js-1p-20k-128-r3.csv @@ -0,0 +1,2 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +DsDjrcOoYbw8k54T1mMXxq,P0,40000,2560000,17388,2225703.026620,1.150198 diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/js-1p-20k-128-r3.txt b/packages/verae-nats-cluster/results/20260912T053120Z/js-1p-20k-128-r3.txt new file mode 100644 index 0000000..fda3f14 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/js-1p-20k-128-r3.txt @@ -0,0 +1,5 @@ +=== js-1p-20k-128-r3 === + +Pub stats: 17,388 msgs/sec ~ 2.12 MB/sec + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/js-2p2s-20k-128-r3.csv b/packages/verae-nats-cluster/results/20260912T053120Z/js-2p2s-20k-128-r3.csv new file mode 100644 index 0000000..270bee9 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/js-2p2s-20k-128-r3.csv @@ -0,0 +1,5 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +JYqsv0sXzEqkC0r49q0aE5,S0,20203,1280000,5135,657394.514895,1.947080 +JYqsv0sXzEqkC0r49q0aE5,S1,20210,1280000,4971,636317.997133,2.011573 +JYqsv0sXzEqkC0r49q0aE5,P0,20000,1280000,5099,652788.600986,1.960819 +JYqsv0sXzEqkC0r49q0aE5,P1,20000,1280000,4979,637405.779512,2.008140 diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/js-2p2s-20k-128-r3.txt b/packages/verae-nats-cluster/results/20260912T053120Z/js-2p2s-20k-128-r3.txt new file mode 100644 index 0000000..4dc4838 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/js-2p2s-20k-128-r3.txt @@ -0,0 +1,13 @@ +=== js-2p2s-20k-128-r3 === + +NATS Pub/Sub stats: 19,876 msgs/sec ~ 2.43 MB/sec + Pub stats: 9,959 msgs/sec ~ 1.22 MB/sec + [1] 5,099 msgs/sec ~ 637.49 KB/sec (10000 msgs) + [2] 4,979 msgs/sec ~ 622.47 KB/sec (10000 msgs) + min 4,979 | avg 5,039 | max 5,099 | stddev 60 msgs + Sub stats: 9,942 msgs/sec ~ 1.21 MB/sec + [1] 5,135 msgs/sec ~ 641.99 KB/sec (10000 msgs) + [2] 4,971 msgs/sec ~ 621.40 KB/sec (10000 msgs) + min 4,971 | avg 5,053 | max 5,135 | stddev 82 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/js-4p-20k-1k-r3.csv b/packages/verae-nats-cluster/results/20260912T053120Z/js-4p-20k-1k-r3.csv new file mode 100644 index 0000000..8c263a1 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/js-4p-20k-1k-r3.csv @@ -0,0 +1,5 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +r1r7AI1sRr29EvqNG14gSh,P0,10000,5120000,7677,7861269.752874,0.651294 +r1r7AI1sRr29EvqNG14gSh,P1,10000,5120000,4571,4680909.596986,1.093805 +r1r7AI1sRr29EvqNG14gSh,P2,10000,5120000,4529,4637905.946623,1.103946 +r1r7AI1sRr29EvqNG14gSh,P3,10000,5120000,4528,4637319.175494,1.104086 diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/js-4p-20k-1k-r3.txt b/packages/verae-nats-cluster/results/20260912T053120Z/js-4p-20k-1k-r3.txt new file mode 100644 index 0000000..13775e4 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/js-4p-20k-1k-r3.txt @@ -0,0 +1,10 @@ +=== js-4p-20k-1k-r3 === + +Pub stats: 18,114 msgs/sec ~ 17.69 MB/sec + [1] 7,677 msgs/sec ~ 7.50 MB/sec (5000 msgs) + [2] 4,571 msgs/sec ~ 4.46 MB/sec (5000 msgs) + [3] 4,529 msgs/sec ~ 4.42 MB/sec (5000 msgs) + [4] 4,528 msgs/sec ~ 4.42 MB/sec (5000 msgs) + min 4,528 | avg 5,326 | max 7,677 | stddev 1,357 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/js-4p-50k-128-r3.csv b/packages/verae-nats-cluster/results/20260912T053120Z/js-4p-50k-128-r3.csv new file mode 100644 index 0000000..ccebd0b --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/js-4p-50k-128-r3.csv @@ -0,0 +1,5 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +6cJTgCeHUXHowXNlC39Tws,P0,25000,1600000,7724,988748.856206,1.618207 +6cJTgCeHUXHowXNlC39Tws,P1,25000,1600000,7704,986199.433317,1.622390 +6cJTgCeHUXHowXNlC39Tws,P2,25000,1600000,5713,731384.231657,2.187633 +6cJTgCeHUXHowXNlC39Tws,P3,25000,1600000,5713,731308.958368,2.187858 diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/js-4p-50k-128-r3.txt b/packages/verae-nats-cluster/results/20260912T053120Z/js-4p-50k-128-r3.txt new file mode 100644 index 0000000..f4b0dec --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/js-4p-50k-128-r3.txt @@ -0,0 +1,10 @@ +=== js-4p-50k-128-r3 === + +Pub stats: 22,853 msgs/sec ~ 2.79 MB/sec + [1] 7,724 msgs/sec ~ 965.58 KB/sec (12500 msgs) + [2] 7,704 msgs/sec ~ 963.09 KB/sec (12500 msgs) + [3] 5,713 msgs/sec ~ 714.24 KB/sec (12500 msgs) + [4] 5,713 msgs/sec ~ 714.17 KB/sec (12500 msgs) + min 5,713 | avg 6,713 | max 7,724 | stddev 1,000 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/js-mem-1p-20k-128-r3.csv b/packages/verae-nats-cluster/results/20260912T053120Z/js-mem-1p-20k-128-r3.csv new file mode 100644 index 0000000..58feb82 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/js-mem-1p-20k-128-r3.csv @@ -0,0 +1,2 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +jHJGzEaQyqmtfzAfyQRddn,P0,40000,2560000,22153,2835624.882291,0.902799 diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/js-mem-1p-20k-128-r3.txt b/packages/verae-nats-cluster/results/20260912T053120Z/js-mem-1p-20k-128-r3.txt new file mode 100644 index 0000000..e43a918 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/js-mem-1p-20k-128-r3.txt @@ -0,0 +1,5 @@ +=== js-mem-1p-20k-128-r3 === + +Pub stats: 22,153 msgs/sec ~ 2.70 MB/sec + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/js-mem-4p-20k-1k-r3.csv b/packages/verae-nats-cluster/results/20260912T053120Z/js-mem-4p-20k-1k-r3.csv new file mode 100644 index 0000000..d528650 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/js-mem-4p-20k-1k-r3.csv @@ -0,0 +1,5 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +bSUG5M5Cdh9fEhPsY8MWLm,P0,10000,5120000,9256,9478798.103952,0.540153 +bSUG5M5Cdh9fEhPsY8MWLm,P1,10000,5120000,8901,9115347.820727,0.561690 +bSUG5M5Cdh9fEhPsY8MWLm,P2,10000,5120000,7192,7365226.588506,0.695159 +bSUG5M5Cdh9fEhPsY8MWLm,P3,10000,5120000,7171,7343566.738925,0.697209 diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/js-mem-4p-20k-1k-r3.txt b/packages/verae-nats-cluster/results/20260912T053120Z/js-mem-4p-20k-1k-r3.txt new file mode 100644 index 0000000..5014c01 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/js-mem-4p-20k-1k-r3.txt @@ -0,0 +1,10 @@ +=== js-mem-4p-20k-1k-r3 === + +Pub stats: 28,685 msgs/sec ~ 28.01 MB/sec + [1] 9,256 msgs/sec ~ 9.04 MB/sec (5000 msgs) + [2] 8,901 msgs/sec ~ 8.69 MB/sec (5000 msgs) + [3] 7,192 msgs/sec ~ 7.02 MB/sec (5000 msgs) + [4] 7,171 msgs/sec ~ 7.00 MB/sec (5000 msgs) + min 7,171 | avg 8,130 | max 9,256 | stddev 956 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/js-mem-4p-50k-128-r3.csv b/packages/verae-nats-cluster/results/20260912T053120Z/js-mem-4p-50k-128-r3.csv new file mode 100644 index 0000000..3bcfc38 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/js-mem-4p-50k-128-r3.csv @@ -0,0 +1,5 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +RCjYO8CWEOKsynfDUe3izo,P0,25000,1600000,9194,1176894.303068,1.359510 +RCjYO8CWEOKsynfDUe3izo,P1,25000,1600000,9199,1177484.153608,1.358829 +RCjYO8CWEOKsynfDUe3izo,P2,25000,1600000,9144,1170517.752374,1.366916 +RCjYO8CWEOKsynfDUe3izo,P3,25000,1600000,9097,1164445.806755,1.374044 diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/js-mem-4p-50k-128-r3.txt b/packages/verae-nats-cluster/results/20260912T053120Z/js-mem-4p-50k-128-r3.txt new file mode 100644 index 0000000..bfe718e --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/js-mem-4p-50k-128-r3.txt @@ -0,0 +1,10 @@ +=== js-mem-4p-50k-128-r3 === + +Pub stats: 36,355 msgs/sec ~ 4.44 MB/sec + [1] 9,194 msgs/sec ~ 1.12 MB/sec (12500 msgs) + [2] 9,199 msgs/sec ~ 1.12 MB/sec (12500 msgs) + [3] 9,144 msgs/sec ~ 1.12 MB/sec (12500 msgs) + [4] 9,097 msgs/sec ~ 1.11 MB/sec (12500 msgs) + min 9,097 | avg 9,158 | max 9,199 | stddev 41 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/lat-1p-5k-128.txt b/packages/verae-nats-cluster/results/20260912T053120Z/lat-1p-5k-128.txt new file mode 100644 index 0000000..849fbc2 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/lat-1p-5k-128.txt @@ -0,0 +1,2 @@ +=== lat-1p-5k-128 === +{"count":5000,"pubs":1,"size":128,"mode":"flood","min_us":94920,"avg_us":132026,"p50_us":131990,"p90_us":156086,"p99_us":158860,"max_us":158965,"min":"94.920ms","avg":"132.026ms","p50":"131.990ms","p90":"156.086ms","p99":"158.860ms","max":"158.965ms"} diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/lat-4p-10k-128.txt b/packages/verae-nats-cluster/results/20260912T053120Z/lat-4p-10k-128.txt new file mode 100644 index 0000000..f394dfa --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/lat-4p-10k-128.txt @@ -0,0 +1,2 @@ +=== lat-4p-10k-128 === +{"count":10000,"pubs":4,"size":128,"mode":"flood","min_us":153318,"avg_us":204329,"p50_us":205865,"p90_us":234151,"p99_us":237346,"max_us":237477,"min":"153.318ms","avg":"204.329ms","p50":"205.865ms","p90":"234.151ms","p99":"237.346ms","max":"237.477ms"} diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/lat-4p-5k-1k.txt b/packages/verae-nats-cluster/results/20260912T053120Z/lat-4p-5k-1k.txt new file mode 100644 index 0000000..66a5c5a --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/lat-4p-5k-1k.txt @@ -0,0 +1,2 @@ +=== lat-4p-5k-1k === +{"count":5000,"pubs":4,"size":1024,"mode":"flood","min_us":113281,"avg_us":132425,"p50_us":131457,"p90_us":140317,"p99_us":144683,"max_us":145365,"min":"113.281ms","avg":"132.425ms","p50":"131.457ms","p90":"140.317ms","p99":"144.683ms","max":"145.365ms"} diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/lat-8p-20k-128.txt b/packages/verae-nats-cluster/results/20260912T053120Z/lat-8p-20k-128.txt new file mode 100644 index 0000000..273544e --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/lat-8p-20k-128.txt @@ -0,0 +1,2 @@ +=== lat-8p-20k-128 === +{"count":20000,"pubs":8,"size":128,"mode":"flood","min_us":233727,"avg_us":310924,"p50_us":311170,"p90_us":373437,"p99_us":400764,"max_us":402811,"min":"233.727ms","avg":"310.924ms","p50":"311.170ms","p90":"373.437ms","p99":"400.764ms","max":"402.811ms"} diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/lat-ping-1k-128.txt b/packages/verae-nats-cluster/results/20260912T053120Z/lat-ping-1k-128.txt new file mode 100644 index 0000000..4604122 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/lat-ping-1k-128.txt @@ -0,0 +1,2 @@ +=== lat-ping-1k-128 === +{"count":1000,"pubs":1,"size":128,"mode":"ping","min_us":250,"avg_us":319,"p50_us":299,"p90_us":363,"p99_us":684,"max_us":2759,"min":"0.250ms","avg":"0.319ms","p50":"0.299ms","p90":"0.363ms","p99":"0.684ms","max":"2.759ms"} diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/nats-cluster-bench-ns1.html b/packages/verae-nats-cluster/results/20260912T053120Z/nats-cluster-bench-ns1.html new file mode 100644 index 0000000..c466b2a --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/nats-cluster-bench-ns1.html @@ -0,0 +1,1385 @@ + + + + + + + NATS cluster message speed — NS1 host study + + + + +
Verae Time × Zapier · progress report · maximized NS1 study

NATS cluster message speed — maximized (RAM disk + 8 cores)

packages/zapier-decisions/reports/nats-cluster-bench-ns1.md
+
+

NATS cluster message speed — NS1 host study

+
+

Progress report (maximized NS1 study) · run +20260912T053120Z (UTC)

+
+

Execution provenance. Every process for this study +ran on NS1.GEORGELAMBERT.ORG +(70.88.205.138): maximize-ns1-study.sh +(cores/RAM/max_mem/tmpfs), then +study-on-ns1.sh, nats bench, +latency.mjs (LXC 510), matplotlib, pandoc, weasyprint. +Traffic stayed on vmbr1. veth/10G was not +changed. After the ladder, JetStream was put back on ZFS and product +streams were re-created; 8 cores / 16 GiB / max_mem 8G +stay.

+
+

Measured delta vs +20260912T051237Z

+

Baseline: 1 core / 1 GiB / JetStream on ZFS. This run: 8 cores / 16 +GiB / JetStream tmpfs (file r=3) plus extra +memory store rows. veth/10G unchanged.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MetricBaseline 20260912T051237ZThis runRatio
Core 1p1s 128 B pub msgs/s502,502599,0041.19×
Core 8p8s 128 B aggregate msgs/s2,065,2171,998,7330.97×
JS file r=3 1p 128 B pub msgs/s7,39317,3882.35×
JS file r=3 4p 128 B pub msgs/s17,98622,8531.27×
JS file r=3 4p 1 KiB pub msgs/s14,98518,1141.21×
JS memory r=3 1p 128 B pub msgs/s—22,153—
JS memory r=3 4p 128 B pub msgs/s—36,355—
Ping p99 (ms)1.377ms0.684ms2.01× faster
+
+ + +
+

Baseline vs maximized publish rates (log)

+
+

1. Executive summary

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ItemThis NS1-host run
Control planeNS1.GEORGELAMBERT.ORG (70.88.205.138), user +marchon
Bench clientLXC 510 verae-px-worker
BrokersLXC 511/512/513 nats-a/b/c on +10.10.10.21–23
Client URLnats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
Host load before8.77 8.39 8.26 6/3849 198471
Host load after9.02 8.82 8.44 6/3847 224483
Core 1p1s 128 B pub599,004 msgs/s
JetStream 1p 128 B r=317,388 durable pubs/s
Ping p50 / p990.299ms / 0.684ms
+

Product traffic is the JetStream row. Ping is one-message delay. +Flood is mailbox catch-up after a burst.

+
+

2. Where it ran (and where it +did not)

+
Operator laptop  ──ssh──►  NS1.GEORGELAMBERT.ORG  70.88.205.138
+                              study-on-ns1.sh
+                              python3 build-ns1-study-report.py
+                              sudo pct exec 510 ──► nats bench / latency.mjs
+                                                    │
+                                                    ▼ vmbr1
+                                              10.10.10.21-23 :4222
+ +
+

3. Results (this run)

+

Host and brokers

+

Before

+ +++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NodeVMIDconnectionsin_msgsout_msgscpucoresmem (B)jetstream
nats-a51133,0133,0371814,553,088True
nats-b51231,4151,4240113,557,760True
nats-c51301,3611,3951114,028,800True
+

After

+ +++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NodeVMIDconnectionsin_msgsout_msgscpucoresmem (B)jetstream
nats-a5113644,7301,344,7682867,104,768True
nats-b5123491,295978,8001182,542,592True
nats-c5130668,5441,330,9400137,314,560True
+

nproc=40 · +uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux

+

Throughput

+ +++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunModeAggregate msgs/sPub msgs/sPub MB/sSub msgs/sSub MB/s
core-1p1s-50k-128core pub/sub810,988599,00473.12456,11655.68
core-4p4s-100k-128core pub/sub1,361,921379,34646.311,150,536140.45
core-4p4s-50k-1kcore pub/sub695,192175,725171.61584,024570.34
core-8p8s-200k-128core pub/sub1,998,733283,25934.581,780,589217.36
js-1p-20k-128-r3jetstream r=3 file—17,3882.12——
js-2p2s-20k-128-r3jetstream r=3 file19,8769,9591.229,9421.21
js-4p-20k-1k-r3jetstream r=3 file—18,11417.69——
js-4p-50k-128-r3jetstream r=3 file—22,8532.79——
js-mem-1p-20k-128-r3jetstream r=3 file—22,1532.70——
js-mem-4p-20k-1k-r3jetstream r=3 file—28,68528.01——
js-mem-4p-50k-128-r3jetstream r=3 file—36,3554.44——
+

Round-trip delay

+ +++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunKindCountPubsSizeminavgp50p90p99max
lat-ping-1k-128ping (sequential RTT)10001128 B0.250ms0.319ms0.299ms0.363ms0.684ms2.759ms
lat-1p-5k-128flood (burst queueing)50001128 B94.920ms132.026ms131.990ms156.086ms158.860ms158.965ms
lat-4p-5k-1kflood (burst queueing)500041024 B113.281ms132.425ms131.457ms140.317ms144.683ms145.365ms
lat-4p-10k-128flood (burst queueing)100004128 B153.318ms204.329ms205.865ms234.151ms237.346ms237.477ms
lat-8p-20k-128flood (burst queueing)200008128 B233.727ms310.924ms311.170ms373.437ms400.764ms402.811ms
+

Core NATS

+
+ + +
+

Core NATS throughput at four loads (NS1 host run) ### +Payload size (core)

+
+ + +
+

Core NATS 128 B vs 1 KiB (NS1 host run) ### JetStream r=3 +file

+
+ + +
+

JetStream durable publish rate (NS1 host run) ### Core vs +JetStream

+
+ + +
+

Core vs JetStream publish rate, log scale (NS1 host run) ### +Delay

+
+ + +
+

Ping vs flood delay percentiles, log scale (NS1 host +run)

+
+

4. Study methodology

+

4.1 Question

+

On the NS1 test stand, what message throughput and +delay does the three-node verae JetStream +cluster deliver at several loads, and which part of the stack is the +limiter for product traffic (jobs, events, webhooks, archive)?

+

4.2 Hypotheses (stated before +the run)

+
    +
  1. H1 — Core vs JetStream. Fire-and-forget core NATS +is at least an order of magnitude faster than JetStream file + +replicas=3, because durable publish waits for a majority disk +replica.
    2. +
  2. H2 — JetStream parallelism. Adding publishers does +not linearly increase JetStream write rate once the +replica log is saturated.
    3. +
  3. H3 — Quiet delay. Sequential pub→sub round trip on +vmbr1 is well under 1 ms p99 when the consumer is +waiting.
    4. +
  4. H4 — Burst delay. If publishers dump a batch before +the subscriber drains, observed delay is queueing time, +roughly linear in backlog, not in cluster hop count.
    5. +
  5. H5 — Payload. Moving 128 B → 1 KiB lowers message +rate and raises byte rate on core NATS; JetStream in this size band +stays replica/fsync bound.
    6. +
+

4.3 Independent variables +(what we changed)

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FactorLevels
TransportCore NATS pub/sub vs JetStream file replicas=3
Publisher count1, 2, 4, 8
Subscriber count0 (JS publish-only), 1, 2, 4, 8
Message count1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step)
Payload128 B, 1024 B
Delay modeping (publish, wait, repeat) vs +flood (publish all, then drain)
+

4.4 Dependent variables +(what we recorded)

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MetricInstrumentUnit
Publish ratenats bench 0.1.6 Pub statsmsgs/s, MB/s
Subscribe ratenats bench Sub statsmsgs/s, MB/s
Aggregatenats bench NATS Pub/Sub statsmsgs/s (fan-out counts both sides)
Publisher spreadnats min/avg/max msgs/snot delay
One-way-ish RTTlatency.mjs header timestampmin, avg, p50, p90, p99, max
Host load/proc/loadavg before and afterload average
Broker countershttp://127.0.0.1:8222/varz inside each nats LXCconnections, in/out msgs, cpu, mem
+

Important: nats CLI 0.1.6 min/avg/max are +rate spread across publishers, not microseconds of +delay. Delay is only latency.mjs.

+

4.5 Controls and constants

+ +

4.6 Procedure

+
    +
  1. Confirm this script is executing on +NS1.GEORGELAMBERT.ORG. Refuse otherwise.
    2. +
  2. Snapshot host load, memory, LXC configs, and each nats +varz.
    3. +
  3. From NS1, pct exec 510 the core ladder (1p1s, 4p4s, +8p8s at 128 B; 4p4s at 1 KiB).
    4. +
  4. Delete benchstream; JS ladder (1p, 4p, 4p×1 KiB, 2p2s +pull) at replicas=3 file.
    5. +
  5. Copy latency.mjs into 510; ping then flood at several +batch sizes.
    6. +
  6. Snapshot host/varz again.
    7. +
  7. Parse logs on this host; draw charts; write HTML +and PDF on this host.
    8. +
+

No publish, subscribe, chart, or PDF process runs on the operator +laptop for this study.

+

4.7 Instrumentation path

+
[NS1 host 70.88.205.138]
+   study-on-ns1.sh  (bash + python3)
+        |
+        | sudo pct exec 510
+        v
+[LXC 510 verae-px-worker 10.10.10.20]
+   nats bench / node latency.mjs
+        |
+        | NATS client protocol to
+        v
+[LXC 511/512/513  10.10.10.21-23 :4222]
+   nats-server -js  cluster routes :6222
+

The hypervisor issues the guest commands. The messages themselves +never leave vmbr1.

+

4.8 Threats to validity

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ThreatEffect on numbers
One physical hostThree “replicas” share CPU, memory, and usually the same datastore. +This measures process/LXC HA, not disk HA.
Shared loadNS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. +Load average during a run is part of the result, not noise to +ignore.
Single bench clientAll publishers live in 510. Per-publisher rate spread is contention +in that guest.
Short runsSeconds of traffic. No compaction, no multi-hour page-cache +eviction, no snapshot during load.
No TLS/nkeysProduction auth will cost CPU. Do not treat these rates as +post-nkeys rates.
Fan-out aggregateCore aggregate msgs/s counts pub+sub. Do not compare that column to +JetStream unique writes.
Flood ≠ RTTMixing flood averages with ping p99 produces a fake “NATS is slow” +story.
Lab onlyNot a Zapier HTTPS bench and not live +api.veraetime.net.
+

4.9 Ethics / safety

+

Bench uses throwaway subjects (bench.core.*, +bench.js.*, bench.lat.*) and a throwaway +stream. It does not purge product streams. Zapier cloud has no NATS +socket.

+
+

5. Suggestions for fine-tuning

+

These follow from the method and from the first ladder on this stand +(JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of +ms). Apply in order of leverage. Re-run this NS1 study +after each change so the delta is measured the same way.

+

5.1 Treat JetStream as +the product limiter

+

Product jobs/events/webhooks/archive are durable. Tuning core NATS to +2M msgs/s will not move a timestamp Zap. Put effort into replica +write path and consumer lag, not core +fan-out.

+

5.2 Split storage class by +stream

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
StreamSuggested storeWhy
ZAPIER_JOBSfile, r=3Work queue; lose-a-job is bad
ZAPIER_EVENTSfile r=3, or memory r=3 if events are rebuildable from job +statusHot waiters; measure both
ZAPIER_WEBHOOKSfile, r=3, workqueueHTTPS to Zapier is the slow consumer
ZAPIER_USAGEfile, r=3, limits + max-ageTelemetry
VERAE_ARCHIVEfile, r=3, on the best diskPuts are larger and must survive
+

Try ZAPIER_EVENTS as memory store in a maintenance +window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms +and durable events still ack at a higher rate, keep it; if a CT restart +drops in-flight waiters, revert.

+

5.3 Give JetStream real disks

+

Today r=3 on three LXC guests on one Proxmox host is +three files, one failure domain.

+ +

5.4 Isolate the nats +CTs from the rest of NS1

+

Host load on this box is often already several. Pin:

+ +

Re-run this study after pinning; H1/H2 should move more than +ping.

+

5.5 Consumer and mailbox +tuning (delay H4)

+

Flood delay is backlog / consume_rate. Fine-tune the +waiters, not the broker RTT.

+ +

5.6 Publisher-side +batching in middleware

+

A timestamp job is one small JSON. 16k msgs/s is ample. Still:

+ +

5.7 +nats-server knobs worth measuring (A/B with this script)

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
KnobWhy try it
max_payloadKeep default unless archive puts grow
write_deadlineSlow consumer protection for webhooks
max_pendingBound memory on a stuck Zapier hook
max_connectionsFleet workers + keep + middleware
JetStream max_file_store / +max_memory_storePrevent one stream from filling the CT
max_outstanding_catchupReplica restart after a nats-c blip
GOMAXPROCS = LXC coresDo not overthread a 2-core CT
+

Change one knob, re-run +study-on-ns1.sh, compare JetStream 1p 128 B and ping +p99.

+

5.8 Network

+ +

5.9 Security cost (when +nkeys/mTLS flip)

+

verae-nats-accounts is still a sketch. Enabling accounts +will add CPU on publish. Budget: re-run this exact study +after creds are in every NATS_URL, and +accept a drop on both core and JS. Do not flip without that +measurement.

+

5.10 Operational +fine-tuning (lag, not peak msgs/s)

+
    +
  1. Scrape varz / jsz from the host over +vmbr1 (not public). Monitor loopback :8222 is +invisible to Prometheus on NS1 unless we add a host-side proxy on +10.10.10.21:8222 bound only to vmbr1.
    2. +
  2. Keep replica floors for webhook-deliver and job-poller — they are +the flood defense.
    3. +
  3. Backup/restore drill of JetStream during idle, then +a short JS 1p run to see catchup cost.
    4. +
  4. A 15–30 minute soak (not in this ladder) for page cache and +compaction; add that as a third study when disks are dedicated.
    5. +
+

5.11 What not to tune

+ + +
    +
  1. CPU pin nats-a/b/c → re-run JS 1p + ping.
    2. +
  2. ZAPIER_EVENTS-shaped memory stream vs file (throwaway +stream, same flags as this JS ladder).
    3. +
  3. Distinct store_dir disks per node.
    4. +
  4. nkeys on, same ladder.
    5. +
  5. Three hardware boxes, same cluster.env IPs +updated.
    6. +
+

Each experiment should produce a new +results/<utc>/ on NS1 and a new progress-repo report +so we can diff H1–H5 instead of arguing from memory.

+
+

6. Reproducing this study

+

On NS1 only:

+
cd ~/verae-src/verae-nats-cluster
+bash scripts/study-on-ns1.sh
+

The script exits if hostname is not NS1. Outputs land in +results/<utc>/ including +nats-cluster-bench-ns1.{md,html,pdf} and +charts/. Copy those into +zapier-decisions/reports/ for the progress repo and +catalog.

+

Raw logs for this run: results/20260912T053120Z/.

+ + diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/nats-cluster-bench-ns1.md b/packages/verae-nats-cluster/results/20260912T053120Z/nats-cluster-bench-ns1.md new file mode 100644 index 0000000..b0d8c69 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/nats-cluster-bench-ns1.md @@ -0,0 +1,355 @@ +**Progress report (maximized NS1 study)** · run `20260912T053120Z` (UTC) + +> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): `maximize-ns1-study.sh` (cores/RAM/`max_mem`/tmpfs), then `study-on-ns1.sh`, `nats bench`, `latency.mjs` (LXC 510), matplotlib, pandoc, weasyprint. Traffic stayed on `vmbr1`. veth/10G was **not** changed. After the ladder, JetStream was put back on ZFS and product streams were re-created; **8 cores / 16 GiB / max_mem 8G stay**. + +## Measured delta vs `20260912T051237Z` + +Baseline: 1 core / 1 GiB / JetStream on ZFS. This run: 8 cores / 16 GiB / JetStream **tmpfs** (file r=3) plus extra **memory** store rows. veth/10G unchanged. + +| Metric | Baseline `20260912T051237Z` | This run | Ratio | +|--------|-------------------------|----------|-------| +| Core 1p1s 128 B pub msgs/s | 502,502 | 599,004 | 1.19× | +| Core 8p8s 128 B aggregate msgs/s | 2,065,217 | 1,998,733 | 0.97× | +| JS file r=3 1p 128 B pub msgs/s | 7,393 | 17,388 | 2.35× | +| JS file r=3 4p 128 B pub msgs/s | 17,986 | 22,853 | 1.27× | +| JS file r=3 4p 1 KiB pub msgs/s | 14,985 | 18,114 | 1.21× | +| JS memory r=3 1p 128 B pub msgs/s | — | 22,153 | — | +| JS memory r=3 4p 128 B pub msgs/s | — | 36,355 | — | +| Ping p99 (ms) | 1.377ms | 0.684ms | 2.01× faster | + +![Baseline vs maximized publish rates (log)](charts/delta-vs-baseline.png) + +*Baseline vs maximized publish rates (log)* + + +--- + +## 1. Executive summary + +| Item | This NS1-host run | +|------|-------------------| +| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `marchon` | +| Bench client | LXC 510 `verae-px-worker` | +| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` | +| Client URL | `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` | +| Host load before | `8.77 8.39 8.26 6/3849 198471` | +| Host load after | `9.02 8.82 8.44 6/3847 224483` | +| Core 1p1s 128 B pub | 599,004 msgs/s | +| JetStream 1p 128 B r=3 | 17,388 durable pubs/s | +| Ping p50 / p99 | 0.299ms / 0.684ms | + +Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst. + +--- + +## 2. Where it ran (and where it did not) + +```text +Operator laptop ──ssh──► NS1.GEORGELAMBERT.ORG 70.88.205.138 + study-on-ns1.sh + python3 build-ns1-study-report.py + sudo pct exec 510 ──► nats bench / latency.mjs + │ + ▼ vmbr1 + 10.10.10.21-23 :4222 +``` + +- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510). +- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file. + +--- + +## 3. Results (this run) + +### Host and brokers + +**Before** + +| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream | +|------|------|-------------|---------|----------|-----|-------|---------|-----------| +| nats-a | 511 | 3 | 3,013 | 3,037 | 1 | 8 | 14,553,088 | True | +| nats-b | 512 | 3 | 1,415 | 1,424 | 0 | 1 | 13,557,760 | True | +| nats-c | 513 | 0 | 1,361 | 1,395 | 1 | 1 | 14,028,800 | True | + +**After** + +| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream | +|------|------|-------------|---------|----------|-----|-------|---------|-----------| +| nats-a | 511 | 3 | 644,730 | 1,344,768 | 2 | 8 | 67,104,768 | True | +| nats-b | 512 | 3 | 491,295 | 978,800 | 1 | 1 | 82,542,592 | True | +| nats-c | 513 | 0 | 668,544 | 1,330,940 | 0 | 1 | 37,314,560 | True | + +nproc=40 · uname=`Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux` + +### Throughput + +| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s | +|-----|------|------------------|------------|----------|------------|----------| +| `core-1p1s-50k-128` | core pub/sub | 810,988 | 599,004 | 73.12 | 456,116 | 55.68 | +| `core-4p4s-100k-128` | core pub/sub | 1,361,921 | 379,346 | 46.31 | 1,150,536 | 140.45 | +| `core-4p4s-50k-1k` | core pub/sub | 695,192 | 175,725 | 171.61 | 584,024 | 570.34 | +| `core-8p8s-200k-128` | core pub/sub | 1,998,733 | 283,259 | 34.58 | 1,780,589 | 217.36 | +| `js-1p-20k-128-r3` | jetstream r=3 file | — | 17,388 | 2.12 | — | — | +| `js-2p2s-20k-128-r3` | jetstream r=3 file | 19,876 | 9,959 | 1.22 | 9,942 | 1.21 | +| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 18,114 | 17.69 | — | — | +| `js-4p-50k-128-r3` | jetstream r=3 file | — | 22,853 | 2.79 | — | — | +| `js-mem-1p-20k-128-r3` | jetstream r=3 file | — | 22,153 | 2.70 | — | — | +| `js-mem-4p-20k-1k-r3` | jetstream r=3 file | — | 28,685 | 28.01 | — | — | +| `js-mem-4p-50k-128-r3` | jetstream r=3 file | — | 36,355 | 4.44 | — | — | + +### Round-trip delay + +| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max | +|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----| +| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.250ms | 0.319ms | 0.299ms | 0.363ms | 0.684ms | 2.759ms | +| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 94.920ms | 132.026ms | 131.990ms | 156.086ms | 158.860ms | 158.965ms | +| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 113.281ms | 132.425ms | 131.457ms | 140.317ms | 144.683ms | 145.365ms | +| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 153.318ms | 204.329ms | 205.865ms | 234.151ms | 237.346ms | 237.477ms | +| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 233.727ms | 310.924ms | 311.170ms | 373.437ms | 400.764ms | 402.811ms | + +### Core NATS + +![Core NATS throughput at four loads (NS1 host run)](charts/core-throughput.png) + +*Core NATS throughput at four loads (NS1 host run)* +### Payload size (core) + +![Core NATS 128 B vs 1 KiB (NS1 host run)](charts/payload-size.png) + +*Core NATS 128 B vs 1 KiB (NS1 host run)* +### JetStream r=3 file + +![JetStream durable publish rate (NS1 host run)](charts/js-throughput.png) + +*JetStream durable publish rate (NS1 host run)* +### Core vs JetStream + +![Core vs JetStream publish rate, log scale (NS1 host run)](charts/core-vs-js.png) + +*Core vs JetStream publish rate, log scale (NS1 host run)* +### Delay + +![Ping vs flood delay percentiles, log scale (NS1 host run)](charts/delay-percentiles.png) + +*Ping vs flood delay percentiles, log scale (NS1 host run)* + +--- + +## 4. Study methodology + +### 4.1 Question + +On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)? + +### 4.2 Hypotheses (stated before the run) + +1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica. +2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated. +3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting. +4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count. +5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound. + +### 4.3 Independent variables (what we changed) + +| Factor | Levels | +|--------|--------| +| Transport | Core NATS pub/sub vs JetStream file replicas=3 | +| Publisher count | 1, 2, 4, 8 | +| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 | +| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) | +| Payload | 128 B, 1024 B | +| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) | + +### 4.4 Dependent variables (what we recorded) + +| Metric | Instrument | Unit | +|--------|------------|------| +| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s | +| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s | +| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) | +| Publisher spread | nats min/avg/max **msgs/s** | not delay | +| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max | +| Host load | `/proc/loadavg` before and after | load average | +| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem | + +**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`. + +### 4.5 Controls and constants + +- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`. +- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`). +- Bench client is LXC **510**, not a nats-* server. +- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide. +- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`). +- No TLS, no nkeys, no account isolation (isolation is `vmbr1`). +- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder. + +### 4.6 Procedure + +1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise. +2. Snapshot host load, memory, LXC configs, and each nats `varz`. +3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB). +4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file. +5. Copy `latency.mjs` into 510; ping then flood at several batch sizes. +6. Snapshot host/`varz` again. +7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**. + +No publish, subscribe, chart, or PDF process runs on the operator laptop for this study. + +### 4.7 Instrumentation path + +```text +[NS1 host 70.88.205.138] + study-on-ns1.sh (bash + python3) + | + | sudo pct exec 510 + v +[LXC 510 verae-px-worker 10.10.10.20] + nats bench / node latency.mjs + | + | NATS client protocol to + v +[LXC 511/512/513 10.10.10.21-23 :4222] + nats-server -js cluster routes :6222 +``` + +The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`. + +### 4.8 Threats to validity + +| Threat | Effect on numbers | +|--------|-------------------| +| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. | +| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. | +| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. | +| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. | +| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. | +| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. | +| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. | +| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. | + +### 4.9 Ethics / safety + +Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket. + +--- + +## 5. Suggestions for fine-tuning + +These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way. + +### 5.1 Treat JetStream as the product limiter + +Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out. + +### 5.2 Split storage class by stream + +| Stream | Suggested store | Why | +|--------|-----------------|-----| +| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad | +| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both | +| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer | +| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry | +| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive | + +Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert. + +### 5.3 Give JetStream real disks + +Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain. + +- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`. +- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure. +- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it. +- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.” + +### 5.4 Isolate the nats CTs from the rest of NS1 + +Host load on this box is often already several. Pin: + +- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes. +- Memory high enough that file-backed streams stay cache-hot for the working set. +- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync. + +Re-run this study after pinning; H1/H2 should move more than ping. + +### 5.5 Consumer and mailbox tuning (delay H4) + +Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT. + +- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM. +- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber. +- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped. +- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT. + +### 5.6 Publisher-side batching in middleware + +A timestamp job is one small JSON. 16k msgs/s is ample. Still: + +- Avoid per-byte publishes; one message per job/event. +- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs). +- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only. + +### 5.7 nats-server knobs worth measuring (A/B with this script) + +| Knob | Why try it | +|------|------------| +| `max_payload` | Keep default unless archive puts grow | +| `write_deadline` | Slow consumer protection for webhooks | +| `max_pending` | Bound memory on a stuck Zapier hook | +| `max_connections` | Fleet workers + keep + middleware | +| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT | +| `max_outstanding_catchup` | Replica restart after a nats-c blip | +| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT | + +Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99. + +### 5.8 Network + +- Keep NATS off `vmbr0`. No change. +- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load). +- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus. + +### 5.9 Security cost (when nkeys/mTLS flip) + +`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement. + +### 5.10 Operational fine-tuning (lag, not peak msgs/s) + +1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`. +2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense. +3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost. +4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated. + +### 5.11 What not to tune + +- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge. +- Do not treat flood 400 ms as “cluster RTT.” Fix consumers. +- Do not load-test on `ZAPIER_*` streams. +- Do not bind client NATS to `0.0.0.0` on `vmbr0`. + +### 5.12 Recommended next experiments (same method, one change each) + +1. CPU pin nats-a/b/c → re-run JS 1p + ping. +2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder). +3. Distinct `store_dir` disks per node. +4. nkeys on, same ladder. +5. Three hardware boxes, same `cluster.env` IPs updated. + +Each experiment should produce a new `results//` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory. + + +--- + +## 6. Reproducing this study + +On **NS1 only**: + +```bash +cd ~/verae-src/verae-nats-cluster +bash scripts/study-on-ns1.sh +``` + +The script exits if `hostname` is not NS1. Outputs land in `results//` including `nats-cluster-bench-ns1.{md,html,pdf}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog. + +Raw logs for this run: `results/20260912T053120Z/`. diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/nats-cluster-bench-ns1.pdf b/packages/verae-nats-cluster/results/20260912T053120Z/nats-cluster-bench-ns1.pdf new file mode 100644 index 0000000..634bd81 Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T053120Z/nats-cluster-bench-ns1.pdf differ diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/varz-after.json b/packages/verae-nats-cluster/results/20260912T053120Z/varz-after.json new file mode 100644 index 0000000..b2f70c8 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/varz-after.json @@ -0,0 +1,50 @@ +[ + { + "vmid": "511", + "name": "nats-a", + "server_name": "nats-a", + "host": "10.10.10.21", + "port": 4222, + "connections": 3, + "in_msgs": 644730, + "out_msgs": 1344768, + "in_bytes": 178180385, + "out_bytes": 317368503, + "cpu": 2, + "cores": 8, + "mem": 67104768, + "jetstream": true + }, + { + "vmid": "512", + "name": "nats-b", + "server_name": "nats-b", + "host": "10.10.10.22", + "port": 4222, + "connections": 3, + "in_msgs": 491295, + "out_msgs": 978800, + "in_bytes": 140248328, + "out_bytes": 203296096, + "cpu": 1, + "cores": 1, + "mem": 82542592, + "jetstream": true + }, + { + "vmid": "513", + "name": "nats-c", + "server_name": "nats-c", + "host": "10.10.10.23", + "port": 4222, + "connections": 0, + "in_msgs": 668544, + "out_msgs": 1330940, + "in_bytes": 199016161, + "out_bytes": 316279225, + "cpu": 0, + "cores": 1, + "mem": 37314560, + "jetstream": true + } +] diff --git a/packages/verae-nats-cluster/results/20260912T053120Z/varz-before.json b/packages/verae-nats-cluster/results/20260912T053120Z/varz-before.json new file mode 100644 index 0000000..0e6e3a1 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T053120Z/varz-before.json @@ -0,0 +1,50 @@ +[ + { + "vmid": "511", + "name": "nats-a", + "server_name": "nats-a", + "host": "10.10.10.21", + "port": 4222, + "connections": 3, + "in_msgs": 3013, + "out_msgs": 3037, + "in_bytes": 177854, + "out_bytes": 203258, + "cpu": 1, + "cores": 8, + "mem": 14553088, + "jetstream": true + }, + { + "vmid": "512", + "name": "nats-b", + "server_name": "nats-b", + "host": "10.10.10.22", + "port": 4222, + "connections": 3, + "in_msgs": 1415, + "out_msgs": 1424, + "in_bytes": 126181, + "out_bytes": 72041, + "cpu": 0, + "cores": 1, + "mem": 13557760, + "jetstream": true + }, + { + "vmid": "513", + "name": "nats-c", + "server_name": "nats-c", + "host": "10.10.10.23", + "port": 4222, + "connections": 0, + "in_msgs": 1361, + "out_msgs": 1395, + "in_bytes": 108748, + "out_bytes": 69591, + "cpu": 1, + "cores": 1, + "mem": 14028800, + "jetstream": true + } +] diff --git a/packages/verae-nats-cluster/scripts/bench.sh b/packages/verae-nats-cluster/scripts/bench.sh index a572fe6..6f057e8 100755 --- a/packages/verae-nats-cluster/scripts/bench.sh +++ b/packages/verae-nats-cluster/scripts/bench.sh @@ -69,6 +69,17 @@ js_rm run_one js-2p2s-20k-128-r3 bench.js.d --js --purge --pub 2 --sub 2 --msgs 20000 --size 128 --replicas 3 --storage file --maxbytes=512MB --pull --stream=benchstream js_rm +# Optional native memory-store ladder (same replica count). Used by maximize-ns1-study.sh. +if [[ "${JS_EXTRA_MEMORY:-0}" == "1" ]]; then + js_rm + run_one js-mem-1p-20k-128-r3 bench.js.m1 --js --purge --pub 1 --msgs 20000 --size 128 --replicas 3 --storage memory --maxbytes=512MB --stream=benchstream + js_rm + run_one js-mem-4p-50k-128-r3 bench.js.m2 --js --purge --pub 4 --msgs 50000 --size 128 --replicas 3 --storage memory --maxbytes=512MB --stream=benchstream + js_rm + run_one js-mem-4p-20k-1k-r3 bench.js.m3 --js --purge --pub 4 --msgs 20000 --size 1024 --replicas 3 --storage memory --maxbytes=512MB --stream=benchstream + js_rm +fi + # Round-trip delay (two connections, through the cluster) at several loads sudo pct exec "$CLIENT_VMID" -- bash -lc " set -e diff --git a/packages/verae-nats-cluster/scripts/build-ns1-study-report.py b/packages/verae-nats-cluster/scripts/build-ns1-study-report.py index 701b6f1..5b79dc7 100755 --- a/packages/verae-nats-cluster/scripts/build-ns1-study-report.py +++ b/packages/verae-nats-cluster/scripts/build-ns1-study-report.py @@ -57,6 +57,9 @@ JS_LABELS = { "js-4p-50k-128-r3": "4p 50k×128 B", "js-4p-20k-1k-r3": "4p 20k×1 KiB", "js-2p2s-20k-128-r3": "2p2s pull 20k×128 B", + "js-mem-1p-20k-128-r3": "mem 1p 128 B", + "js-mem-4p-50k-128-r3": "mem 4p 128 B", + "js-mem-4p-20k-1k-r3": "mem 4p 1 KiB", } LAT_LABELS = { "lat-ping-1k-128": "Ping\n1k×128 B", @@ -286,11 +289,117 @@ def figure(name: str, caption: str) -> str: return f"![{caption}](charts/{name})\n\n*{caption}*" -def write_markdown(folder: Path, thru: list[dict[str, str]], lats: list[dict[str, str]]) -> str: +def ratio(new: str | None, old: str | None) -> str: + if not new or not old: + return "—" + a, b = float(new), float(old) + if b == 0: + return "—" + return f"{a / b:.2f}×" + + +def delay_ms_val(p: dict[str, str] | None, key: str) -> str | None: + if not p or not p.get(key): + return None + return str(ms(p[key])) + + +def delta_table( + thru: list[dict[str, str]], + lats: list[dict[str, str]], + base_thru: list[dict[str, str]], + base_lats: list[dict[str, str]], + base_stamp: str, +) -> str: + bt = {p["run"]: p for p in base_thru} + nt = {p["run"]: p for p in thru} + bl = {p["run"]: p for p in base_lats} + nl = {p["run"]: p for p in lats} + keys = [ + ("core-1p1s-50k-128", "pub", "Core 1p1s 128 B pub msgs/s"), + ("core-8p8s-200k-128", "agg", "Core 8p8s 128 B aggregate msgs/s"), + ("js-1p-20k-128-r3", "pub", "JS file r=3 1p 128 B pub msgs/s"), + ("js-4p-50k-128-r3", "pub", "JS file r=3 4p 128 B pub msgs/s"), + ("js-4p-20k-1k-r3", "pub", "JS file r=3 4p 1 KiB pub msgs/s"), + ("js-mem-1p-20k-128-r3", "pub", "JS memory r=3 1p 128 B pub msgs/s"), + ("js-mem-4p-50k-128-r3", "pub", "JS memory r=3 4p 128 B pub msgs/s"), + ] + lines = [ + f"| Metric | Baseline `{base_stamp}` | This run | Ratio |", + "|--------|-------------------------|----------|-------|", + ] + for run, kind, label in keys: + old, new = bt.get(run), nt.get(run) + ok = "pub_msgs" if kind == "pub" else "agg_msgs" + ov = old.get(ok) if old else None + nv = new.get(ok) if new else None + lines.append(f"| {label} | {fmt_int(ov)} | {fmt_int(nv)} | {ratio(nv, ov)} |") + old_p, new_p = bl.get("lat-ping-1k-128"), nl.get("lat-ping-1k-128") + if old_p or new_p: + ov = delay_ms_val(old_p, "p99") + nv = delay_ms_val(new_p, "p99") + # smaller delay is better — invert ratio label + r = "—" + if ov and nv and float(nv) != 0: + r = f"{float(ov) / float(nv):.2f}× faster" if float(nv) < float(ov) else f"{float(nv) / float(ov):.2f}× slower" + lines.append( + f"| Ping p99 (ms) | {old_p.get('p99') if old_p else '—'} | {new_p.get('p99') if new_p else '—'} | {r} |" + ) + return "\n".join(lines) + + +def chart_delta( + thru: list[dict[str, str]], + base_thru: list[dict[str, str]], + dest: Path, +) -> None: + bt = {p["run"]: p for p in base_thru} + nt = {p["run"]: p for p in thru} + labels = ["Core 1p\n128 B pub", "JS file 1p\n128 B", "JS file 4p\n128 B", "JS mem 1p\n128 B"] + keys = ["core-1p1s-50k-128", "js-1p-20k-128-r3", "js-4p-50k-128-r3", "js-mem-1p-20k-128-r3"] + old = [int(bt[k]["pub_msgs"]) if k in bt and bt[k].get("pub_msgs") else 0 for k in keys] + new = [int(nt[k]["pub_msgs"]) if k in nt and nt[k].get("pub_msgs") else 0 for k in keys] + if not any(new): + return + fig, ax = plt.subplots(figsize=(9.2, 4.4)) + x = list(range(len(labels))) + w = 0.35 + ax.bar([i - w / 2 for i in x], old, w, label="Baseline 1c/1G/ZFS", color=MUTED) + ax.bar([i + w / 2 for i in x], new, w, label="8c/16G/tmpfs (+ mem rows)", color=INDIGO) + ax.set_xticks(x, labels) + ax.set_yscale("log") + ax.set_ylabel("publish messages / second (log)") + ax.set_title("Measured delta vs 20260912T051237Z") + ax.legend(loc="upper right") + ax.set_axisbelow(True) + save(fig, dest / "delta-vs-baseline.png") + + +def write_markdown( + folder: Path, + thru: list[dict[str, str]], + lats: list[dict[str, str]], + compare: Path | None = None, +) -> str: before = kv_file(folder / "host-before.txt") after = kv_file(folder / "host-after.txt") stamp = folder.name method = (Path(__file__).resolve().parent / "ns1-study-methodology.md").read_text(encoding="utf-8") + delta_md = "" + base_thru: list[dict[str, str]] = [] + base_lats: list[dict[str, str]] = [] + if compare and compare.is_dir(): + base_thru, base_lats = load_runs(compare) + delta_md = ( + f"## Measured delta vs `{compare.name}`\n\n" + "Baseline: 1 core / 1 GiB / JetStream on ZFS. This run: 8 cores / 16 GiB / " + "JetStream **tmpfs** (file r=3) plus extra **memory** store rows. veth/10G unchanged.\n\n" + + delta_table(thru, lats, base_thru, base_lats, compare.name) + + "\n" + ) + if (folder / "charts" / "delta-vs-baseline.png").exists(): + delta_md += "\n" + figure("delta-vs-baseline.png", "Baseline vs maximized publish rates (log)") + delta_md += "\n" figs = [] charts_dir = folder / "charts" if (charts_dir / "core-throughput.png").exists(): @@ -308,11 +417,11 @@ def write_markdown(folder: Path, thru: list[dict[str, str]], lats: list[dict[str js1 = next((p for p in thru if p["run"] == "js-1p-20k-128-r3"), None) core1 = next((p for p in thru if p["run"] == "core-1p1s-50k-128"), None) - md = f"""**Progress report (second study)** · run `{stamp}` (UTC) + md = f"""**Progress report (maximized NS1 study)** · run `{stamp}` (UTC) -> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): the orchestrator (`study-on-ns1.sh`), `nats bench`, `latency.mjs` (inside LXC 510 on this hypervisor), charting (`matplotlib`), and HTML/PDF (`pandoc` + `weasyprint`). The operator laptop did **not** publish, subscribe, draw charts, or render the PDF. Traffic stayed on `vmbr1` from LXC **510** to `nats-a/b/c` (**511–513**). +> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): `maximize-ns1-study.sh` (cores/RAM/`max_mem`/tmpfs), then `study-on-ns1.sh`, `nats bench`, `latency.mjs` (LXC 510), matplotlib, pandoc, weasyprint. Traffic stayed on `vmbr1`. veth/10G was **not** changed. After the ladder, JetStream was put back on ZFS and product streams were re-created; **8 cores / 16 GiB / max_mem 8G stay**. -This is a full methodology write-up plus the numbers from that on-host run. The earlier report (`nats-cluster-bench`, run `20260912T045131Z`) used the same cluster but was **orchestrated and rendered off-box**. Use this document when you need “it was all run on 138.” +{delta_md} --- @@ -407,8 +516,8 @@ def render(md_path: Path, html_path: Path, pdf_path: Path) -> None: '
' '' - '
Verae Time × Zapier · progress report · run on NS1.GEORGELAMBERT.ORG
' - "

NATS cluster message speed — NS1 host study

" + '
Verae Time × Zapier · progress report · maximized NS1 study
' + "

NATS cluster message speed — maximized (RAM disk + 8 cores)

" '
packages/zapier-decisions/reports/nats-cluster-bench-ns1.md
' "
\n", encoding="utf-8", @@ -440,9 +549,13 @@ def render(md_path: Path, html_path: Path, pdf_path: Path) -> None: def main() -> int: folder = Path(sys.argv[1] if len(sys.argv) > 1 else ".") + compare = Path(sys.argv[2]) if len(sys.argv) > 2 and sys.argv[2] else None thru, lats = load_runs(folder) charts(thru, lats, folder / "charts") - md = write_markdown(folder, thru, lats) + if compare and compare.is_dir(): + base_thru, _base_lats = load_runs(compare) + chart_delta(thru, base_thru, folder / "charts") + md = write_markdown(folder, thru, lats, compare if compare and compare.is_dir() else None) md_path = folder / "nats-cluster-bench-ns1.md" md_path.write_text(md, encoding="utf-8") html_path = folder / "nats-cluster-bench-ns1.html" diff --git a/packages/verae-nats-cluster/scripts/maximize-ns1-study.sh b/packages/verae-nats-cluster/scripts/maximize-ns1-study.sh new file mode 100755 index 0000000..7799a6d --- /dev/null +++ b/packages/verae-nats-cluster/scripts/maximize-ns1-study.sh @@ -0,0 +1,102 @@ +#!/usr/bin/env bash +# Maximize nats LXC resources + RAM-disk JetStream, run the NS1 study, then +# put product streams back on ZFS. Cores/RAM/max_mem stay raised. +# Must run on NS1.GEORGELAMBERT.ORG. +set -euo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +export PATH="/usr/sbin:/usr/bin:/bin:/usr/local/bin:$PATH" + +HOST="$(hostname -f 2>/dev/null || hostname)" +case "$HOST" in + NS1.GEORGELAMBERT.ORG|NS1|ns1.georgelambert.org|ns1) ;; + *) + echo "refusing: maximize-ns1-study.sh must run on NS1, got '$HOST'" >&2 + exit 1 + ;; +esac + +CORES="${CORES:-8}" +MEMORY="${MEMORY:-16384}" +TMPFS_SIZE="${TMPFS_SIZE:-8G}" +NATS_VMS=(511 512 513) +ALL_VMS=(510 511 512 513) + +apply_resources() { + local v + for v in "${ALL_VMS[@]}"; do + echo "pct set $v --cores $CORES --memory $MEMORY" + sudo pct set "$v" --cores "$CORES" --memory "$MEMORY" + done +} + +patch_max_mem() { + local v + for v in "${NATS_VMS[@]}"; do + sudo pct exec "$v" -- sed -i -E 's/max_mem:[[:space:]]*[0-9]+[MmGg]/max_mem: 8G/' /etc/nats/nats.conf + sudo pct exec "$v" -- grep -n max_mem /etc/nats/nats.conf + done +} + +mount_tmpfs() { + local v + for v in "${NATS_VMS[@]}"; do + sudo pct exec "$v" -- bash -lc " +set -e +systemctl stop nats-server +mkdir -p /var/lib/nats/jetstream +if ! mountpoint -q /var/lib/nats/jetstream; then + mount -t tmpfs -o size=${TMPFS_SIZE} nats-js /var/lib/nats/jetstream +fi +chown nats:nats /var/lib/nats/jetstream +chmod 755 /var/lib/nats/jetstream +systemctl start nats-server +mount | grep jetstream +" + done +} + +unmount_tmpfs() { + local v + for v in "${NATS_VMS[@]}"; do + sudo pct exec "$v" -- bash -lc ' +set -e +systemctl stop nats-server || true +if mountpoint -q /var/lib/nats/jetstream; then + umount /var/lib/nats/jetstream +fi +mkdir -p /var/lib/nats/jetstream +chown nats:nats /var/lib/nats/jetstream +systemctl start nats-server +' + done +} + +wait_cluster() { + local n=0 + until sudo pct exec 511 -- curl -fsS --max-time 2 http://127.0.0.1:8222/varz >/dev/null 2>&1; do + n=$((n + 1)) + [[ $n -lt 30 ]] || { echo "nats-a varz not up" >&2; return 1; } + sleep 1 + done + sleep 2 +} + +restore_durable() { + echo "restoring ZFS JetStream (product streams)" + unmount_tmpfs + wait_cluster + bash "$ROOT/scripts/ensure-streams.sh" || true +} + +apply_resources +patch_max_mem +mount_tmpfs +wait_cluster +trap restore_durable EXIT + +export JS_EXTRA_MEMORY=1 +export COMPARE_DIR="${COMPARE_DIR:-$ROOT/results/20260912T051237Z}" +# recorded in host-before by appending after dump starts — study script reads pct config live +bash "$ROOT/scripts/study-on-ns1.sh" + +echo "maximize study finished; trap will restore ZFS jetstream" diff --git a/packages/verae-nats-cluster/scripts/study-on-ns1.sh b/packages/verae-nats-cluster/scripts/study-on-ns1.sh index 4eec5cc..e4e424f 100755 --- a/packages/verae-nats-cluster/scripts/study-on-ns1.sh +++ b/packages/verae-nats-cluster/scripts/study-on-ns1.sh @@ -37,6 +37,12 @@ dump_env() { echo "loadavg=$(cat /proc/loadavg)" echo "client_vmid=$CLIENT_VMID" echo "nats_url=$NATS_URL" + echo "js_extra_memory=${JS_EXTRA_MEMORY:-0}" + echo "compare_dir=${COMPARE_DIR:-}" + echo "--- nats 511 max_mem ---" + sudo pct exec 511 -- grep max_mem /etc/nats/nats.conf || true + echo "--- nats 511 jetstream mount ---" + sudo pct exec 511 -- mount | grep jetstream || echo "jetstream on rootfs" echo "--- free ---" free -h echo "--- pct list ---" @@ -86,6 +92,6 @@ echo "NS1 study $STAMP out=$OUT" dump_env before BENCH_OUT="$OUT" CLIENT_VMID="$CLIENT_VMID" bash "$ROOT/scripts/bench.sh" dump_env after -python3 "$ROOT/scripts/build-ns1-study-report.py" "$OUT" +python3 "$ROOT/scripts/build-ns1-study-report.py" "$OUT" "${COMPARE_DIR:-}" echo "NS1 study complete $OUT" ls -la "$OUT"/nats-cluster-bench-ns1.* "$OUT"/charts 2>/dev/null || ls -la "$OUT" diff --git a/packages/verae-nats-cluster/scripts/test.sh b/packages/verae-nats-cluster/scripts/test.sh index 9e02ae8..85b6670 100755 --- a/packages/verae-nats-cluster/scripts/test.sh +++ b/packages/verae-nats-cluster/scripts/test.sh @@ -7,6 +7,7 @@ bash -n "$ROOT/scripts/create-cluster.sh" bash -n "$ROOT/scripts/status.sh" bash -n "$ROOT/scripts/bench.sh" bash -n "$ROOT/scripts/study-on-ns1.sh" +bash -n "$ROOT/scripts/maximize-ns1-study.sh" grep -q 'host: {{IP}}' "$ROOT/conf/nats.conf.tmpl" grep -qv '0.0.0.0' "$ROOT/conf/nats.conf.tmpl" if [[ ! -d /etc/pve/nodes ]]; then diff --git a/packages/zapier-decisions/LOG.md b/packages/zapier-decisions/LOG.md index fb7372f..2537314 100644 --- a/packages/zapier-decisions/LOG.md +++ b/packages/zapier-decisions/LOG.md @@ -1,5 +1,12 @@ # Action log +## 2026-09-12 — maximized NS1 NATS study + +- `pct set` 510–513 to 8 cores / 16 GiB; `max_mem: 8G`; tmpfs 8G on JetStream for the ladder only. +- Run `20260912T053120Z`. JS file 1p 128 B **7.4k → 17.4k (2.35×)**; JS memory 4p **36.4k**; ping p99 **1.38 → 0.68 ms**. +- tmpfs unmounted after; product streams on ZFS again. Resources and max_mem **left raised**. +- Report: `reports/ns1-maximize/`. + ## 2026-09-12 — NS1-host NATS study (second run) - All orchestration, nats bench, matplotlib, pandoc, and weasyprint ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`). Laptop not in the path. diff --git a/packages/zapier-decisions/README.md b/packages/zapier-decisions/README.md index c556fab..5232dc7 100644 --- a/packages/zapier-decisions/README.md +++ b/packages/zapier-decisions/README.md @@ -9,6 +9,7 @@ Lab log of architecture decisions, actions taken, and open todos while cleaning | [STATUS.md](STATUS.md) | **Current status** (2026-09-12) | | [reports/nats-cluster-bench.md](reports/nats-cluster-bench.md) | NATS cluster speed report (charts, HTML, PDF) | | [reports/ns1-host/nats-cluster-bench-ns1.md](reports/ns1-host/nats-cluster-bench-ns1.md) | Second study: all code on NS1 (methodology + tuning) | +| [reports/ns1-maximize/nats-cluster-bench-ns1.md](reports/ns1-maximize/nats-cluster-bench-ns1.md) | Maximized: 8c/16G/tmpfs vs baseline | | [LOG.md](LOG.md) | Chronological actions | | [TODO.md](TODO.md) | Open items | | [decisions/](decisions/) | One file per decision | diff --git a/packages/zapier-decisions/STATUS.md b/packages/zapier-decisions/STATUS.md index 81abc25..fb46a01 100644 --- a/packages/zapier-decisions/STATUS.md +++ b/packages/zapier-decisions/STATUS.md @@ -39,6 +39,17 @@ Measured from LXC **510** (not a nats-* server) against `10.10.10.21–23`. **Second study (all code on NS1.GEORGELAMBERT.ORG / 70.88.205.138, run `20260912T051237Z`):** [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.pdf) · [markdown](reports/ns1-host/nats-cluster-bench-ns1.md). Orchestrator, nats bench, charts, HTML, and PDF ran on 138; traffic was LXC 510 → 511–513 on `vmbr1`. Includes full methodology and fine-tuning notes. Host load ~8.5/40 during the run; JetStream 1p 128 B was **7.4k** durable pubs/s (vs ~16k on the quieter first ladder); ping p99 **1.38 ms**. +**Maximized study (`20260912T053120Z`):** 8 cores + 16 GiB on 510–513, `max_mem: 8G`, JetStream **tmpfs** for the ladder (ZFS restored after). veth/10G unchanged. [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.pdf). + +| Metric | Baseline 051237Z | Maximized | Ratio | +|--------|------------------|-----------|-------| +| JS file r=3 1p 128 B | 7,393 pubs/s | **17,388** | **2.35×** | +| JS memory r=3 4p 128 B | — | **36,355** | ~4.9× vs baseline file 1p | +| Core 1p 128 B pub | 502,502 | 599,004 | 1.19× | +| Ping p99 | 1.377 ms | **0.684 ms** | **2.01× faster** | + +Live cluster **keeps 8 cores / 16 GiB / max_mem 8G**. Product streams are back on ZFS (tmpfs was study-only). + Short tables: [BENCH.md](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md). Re-run: `bash scripts/bench.sh` on NS1. | Kind | Load | Result | diff --git a/packages/zapier-decisions/reports/ns1-maximize/charts/core-throughput.png b/packages/zapier-decisions/reports/ns1-maximize/charts/core-throughput.png new file mode 100644 index 0000000..683f57e Binary files /dev/null and b/packages/zapier-decisions/reports/ns1-maximize/charts/core-throughput.png differ diff --git a/packages/zapier-decisions/reports/ns1-maximize/charts/core-vs-js.png b/packages/zapier-decisions/reports/ns1-maximize/charts/core-vs-js.png new file mode 100644 index 0000000..b6a880c Binary files /dev/null and b/packages/zapier-decisions/reports/ns1-maximize/charts/core-vs-js.png differ diff --git a/packages/zapier-decisions/reports/ns1-maximize/charts/delay-percentiles.png b/packages/zapier-decisions/reports/ns1-maximize/charts/delay-percentiles.png new file mode 100644 index 0000000..0b5f851 Binary files /dev/null and b/packages/zapier-decisions/reports/ns1-maximize/charts/delay-percentiles.png differ diff --git a/packages/zapier-decisions/reports/ns1-maximize/charts/delta-vs-baseline.png b/packages/zapier-decisions/reports/ns1-maximize/charts/delta-vs-baseline.png new file mode 100644 index 0000000..f23250c Binary files /dev/null and b/packages/zapier-decisions/reports/ns1-maximize/charts/delta-vs-baseline.png differ diff --git a/packages/zapier-decisions/reports/ns1-maximize/charts/js-throughput.png b/packages/zapier-decisions/reports/ns1-maximize/charts/js-throughput.png new file mode 100644 index 0000000..55452c7 Binary files /dev/null and b/packages/zapier-decisions/reports/ns1-maximize/charts/js-throughput.png differ diff --git a/packages/zapier-decisions/reports/ns1-maximize/charts/payload-size.png b/packages/zapier-decisions/reports/ns1-maximize/charts/payload-size.png new file mode 100644 index 0000000..1118a42 Binary files /dev/null and b/packages/zapier-decisions/reports/ns1-maximize/charts/payload-size.png differ diff --git a/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.html b/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.html new file mode 100644 index 0000000..c466b2a --- /dev/null +++ b/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.html @@ -0,0 +1,1385 @@ + + + + + + + NATS cluster message speed — NS1 host study + + + + +
Verae Time × Zapier · progress report · maximized NS1 study

NATS cluster message speed — maximized (RAM disk + 8 cores)

packages/zapier-decisions/reports/nats-cluster-bench-ns1.md
+
+

NATS cluster message speed — NS1 host study

+
+

Progress report (maximized NS1 study) · run +20260912T053120Z (UTC)

+
+

Execution provenance. Every process for this study +ran on NS1.GEORGELAMBERT.ORG +(70.88.205.138): maximize-ns1-study.sh +(cores/RAM/max_mem/tmpfs), then +study-on-ns1.sh, nats bench, +latency.mjs (LXC 510), matplotlib, pandoc, weasyprint. +Traffic stayed on vmbr1. veth/10G was not +changed. After the ladder, JetStream was put back on ZFS and product +streams were re-created; 8 cores / 16 GiB / max_mem 8G +stay.

+
+

Measured delta vs +20260912T051237Z

+

Baseline: 1 core / 1 GiB / JetStream on ZFS. This run: 8 cores / 16 +GiB / JetStream tmpfs (file r=3) plus extra +memory store rows. veth/10G unchanged.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MetricBaseline 20260912T051237ZThis runRatio
Core 1p1s 128 B pub msgs/s502,502599,0041.19×
Core 8p8s 128 B aggregate msgs/s2,065,2171,998,7330.97×
JS file r=3 1p 128 B pub msgs/s7,39317,3882.35×
JS file r=3 4p 128 B pub msgs/s17,98622,8531.27×
JS file r=3 4p 1 KiB pub msgs/s14,98518,1141.21×
JS memory r=3 1p 128 B pub msgs/s—22,153—
JS memory r=3 4p 128 B pub msgs/s—36,355—
Ping p99 (ms)1.377ms0.684ms2.01× faster
+
+ + +
+

Baseline vs maximized publish rates (log)

+
+

1. Executive summary

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ItemThis NS1-host run
Control planeNS1.GEORGELAMBERT.ORG (70.88.205.138), user +marchon
Bench clientLXC 510 verae-px-worker
BrokersLXC 511/512/513 nats-a/b/c on +10.10.10.21–23
Client URLnats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
Host load before8.77 8.39 8.26 6/3849 198471
Host load after9.02 8.82 8.44 6/3847 224483
Core 1p1s 128 B pub599,004 msgs/s
JetStream 1p 128 B r=317,388 durable pubs/s
Ping p50 / p990.299ms / 0.684ms
+

Product traffic is the JetStream row. Ping is one-message delay. +Flood is mailbox catch-up after a burst.

+
+

2. Where it ran (and where it +did not)

+
Operator laptop  ──ssh──►  NS1.GEORGELAMBERT.ORG  70.88.205.138
+                              study-on-ns1.sh
+                              python3 build-ns1-study-report.py
+                              sudo pct exec 510 ──► nats bench / latency.mjs
+                                                    │
+                                                    ▼ vmbr1
+                                              10.10.10.21-23 :4222
+
    +
  • Did run on 138: bash, python3, matplotlib, pandoc, +weasyprint, pct, nats-server (in LXC), nats CLI and Node +(in LXC 510).
    • +
  • Did not run on the laptop: no local +nats bench, no local charting, no local WeasyPrint for this +file.
    • +
+
+

3. Results (this run)

+

Host and brokers

+

Before

+ +++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NodeVMIDconnectionsin_msgsout_msgscpucoresmem (B)jetstream
nats-a51133,0133,0371814,553,088True
nats-b51231,4151,4240113,557,760True
nats-c51301,3611,3951114,028,800True
+

After

+ +++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NodeVMIDconnectionsin_msgsout_msgscpucoresmem (B)jetstream
nats-a5113644,7301,344,7682867,104,768True
nats-b5123491,295978,8001182,542,592True
nats-c5130668,5441,330,9400137,314,560True
+

nproc=40 · +uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux

+

Throughput

+ +++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunModeAggregate msgs/sPub msgs/sPub MB/sSub msgs/sSub MB/s
core-1p1s-50k-128core pub/sub810,988599,00473.12456,11655.68
core-4p4s-100k-128core pub/sub1,361,921379,34646.311,150,536140.45
core-4p4s-50k-1kcore pub/sub695,192175,725171.61584,024570.34
core-8p8s-200k-128core pub/sub1,998,733283,25934.581,780,589217.36
js-1p-20k-128-r3jetstream r=3 file—17,3882.12——
js-2p2s-20k-128-r3jetstream r=3 file19,8769,9591.229,9421.21
js-4p-20k-1k-r3jetstream r=3 file—18,11417.69——
js-4p-50k-128-r3jetstream r=3 file—22,8532.79——
js-mem-1p-20k-128-r3jetstream r=3 file—22,1532.70——
js-mem-4p-20k-1k-r3jetstream r=3 file—28,68528.01——
js-mem-4p-50k-128-r3jetstream r=3 file—36,3554.44——
+

Round-trip delay

+ +++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunKindCountPubsSizeminavgp50p90p99max
lat-ping-1k-128ping (sequential RTT)10001128 B0.250ms0.319ms0.299ms0.363ms0.684ms2.759ms
lat-1p-5k-128flood (burst queueing)50001128 B94.920ms132.026ms131.990ms156.086ms158.860ms158.965ms
lat-4p-5k-1kflood (burst queueing)500041024 B113.281ms132.425ms131.457ms140.317ms144.683ms145.365ms
lat-4p-10k-128flood (burst queueing)100004128 B153.318ms204.329ms205.865ms234.151ms237.346ms237.477ms
lat-8p-20k-128flood (burst queueing)200008128 B233.727ms310.924ms311.170ms373.437ms400.764ms402.811ms
+

Core NATS

+
+ + +
+

Core NATS throughput at four loads (NS1 host run) ### +Payload size (core)

+
+ + +
+

Core NATS 128 B vs 1 KiB (NS1 host run) ### JetStream r=3 +file

+
+ + +
+

JetStream durable publish rate (NS1 host run) ### Core vs +JetStream

+
+ + +
+

Core vs JetStream publish rate, log scale (NS1 host run) ### +Delay

+
+ + +
+

Ping vs flood delay percentiles, log scale (NS1 host +run)

+
+

4. Study methodology

+

4.1 Question

+

On the NS1 test stand, what message throughput and +delay does the three-node verae JetStream +cluster deliver at several loads, and which part of the stack is the +limiter for product traffic (jobs, events, webhooks, archive)?

+

4.2 Hypotheses (stated before +the run)

+
    +
  1. H1 — Core vs JetStream. Fire-and-forget core NATS +is at least an order of magnitude faster than JetStream file + +replicas=3, because durable publish waits for a majority disk +replica.
    2. +
  2. H2 — JetStream parallelism. Adding publishers does +not linearly increase JetStream write rate once the +replica log is saturated.
    3. +
  3. H3 — Quiet delay. Sequential pub→sub round trip on +vmbr1 is well under 1 ms p99 when the consumer is +waiting.
    4. +
  4. H4 — Burst delay. If publishers dump a batch before +the subscriber drains, observed delay is queueing time, +roughly linear in backlog, not in cluster hop count.
    5. +
  5. H5 — Payload. Moving 128 B → 1 KiB lowers message +rate and raises byte rate on core NATS; JetStream in this size band +stays replica/fsync bound.
    6. +
+

4.3 Independent variables +(what we changed)

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FactorLevels
TransportCore NATS pub/sub vs JetStream file replicas=3
Publisher count1, 2, 4, 8
Subscriber count0 (JS publish-only), 1, 2, 4, 8
Message count1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step)
Payload128 B, 1024 B
Delay modeping (publish, wait, repeat) vs +flood (publish all, then drain)
+

4.4 Dependent variables +(what we recorded)

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MetricInstrumentUnit
Publish ratenats bench 0.1.6 Pub statsmsgs/s, MB/s
Subscribe ratenats bench Sub statsmsgs/s, MB/s
Aggregatenats bench NATS Pub/Sub statsmsgs/s (fan-out counts both sides)
Publisher spreadnats min/avg/max msgs/snot delay
One-way-ish RTTlatency.mjs header timestampmin, avg, p50, p90, p99, max
Host load/proc/loadavg before and afterload average
Broker countershttp://127.0.0.1:8222/varz inside each nats LXCconnections, in/out msgs, cpu, mem
+

Important: nats CLI 0.1.6 min/avg/max are +rate spread across publishers, not microseconds of +delay. Delay is only latency.mjs.

+

4.5 Controls and constants

+
    +
  • Cluster name verae, three routes, client +:4222, cluster :6222, monitor loopback +:8222.
    • +
  • Client URL always the three-node list on vmbr1 (never +host 127.0.0.1:4222, never vmbr0).
    • +
  • Bench client is LXC 510, not a nats-* server.
    • +
  • JetStream bench stream name benchstream, +file storage, replicas=3, deleted +between JS loads (nats stream rm --force) so names do not +collide.
    • +
  • Product streams were not the bench target (no load +test on ZAPIER_* / VERAE_ARCHIVE).
    • +
  • No TLS, no nkeys, no account isolation (isolation is +vmbr1).
    • +
  • Same nats CLI version (0.1.6) and nats@2 Node client as +the first ladder.
    • +
+

4.6 Procedure

+
    +
  1. Confirm this script is executing on +NS1.GEORGELAMBERT.ORG. Refuse otherwise.
    2. +
  2. Snapshot host load, memory, LXC configs, and each nats +varz.
    3. +
  3. From NS1, pct exec 510 the core ladder (1p1s, 4p4s, +8p8s at 128 B; 4p4s at 1 KiB).
    4. +
  4. Delete benchstream; JS ladder (1p, 4p, 4p×1 KiB, 2p2s +pull) at replicas=3 file.
    5. +
  5. Copy latency.mjs into 510; ping then flood at several +batch sizes.
    6. +
  6. Snapshot host/varz again.
    7. +
  7. Parse logs on this host; draw charts; write HTML +and PDF on this host.
    8. +
+

No publish, subscribe, chart, or PDF process runs on the operator +laptop for this study.

+

4.7 Instrumentation path

+
[NS1 host 70.88.205.138]
+   study-on-ns1.sh  (bash + python3)
+        |
+        | sudo pct exec 510
+        v
+[LXC 510 verae-px-worker 10.10.10.20]
+   nats bench / node latency.mjs
+        |
+        | NATS client protocol to
+        v
+[LXC 511/512/513  10.10.10.21-23 :4222]
+   nats-server -js  cluster routes :6222
+

The hypervisor issues the guest commands. The messages themselves +never leave vmbr1.

+

4.8 Threats to validity

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ThreatEffect on numbers
One physical hostThree “replicas” share CPU, memory, and usually the same datastore. +This measures process/LXC HA, not disk HA.
Shared loadNS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. +Load average during a run is part of the result, not noise to +ignore.
Single bench clientAll publishers live in 510. Per-publisher rate spread is contention +in that guest.
Short runsSeconds of traffic. No compaction, no multi-hour page-cache +eviction, no snapshot during load.
No TLS/nkeysProduction auth will cost CPU. Do not treat these rates as +post-nkeys rates.
Fan-out aggregateCore aggregate msgs/s counts pub+sub. Do not compare that column to +JetStream unique writes.
Flood ≠ RTTMixing flood averages with ping p99 produces a fake “NATS is slow” +story.
Lab onlyNot a Zapier HTTPS bench and not live +api.veraetime.net.
+

4.9 Ethics / safety

+

Bench uses throwaway subjects (bench.core.*, +bench.js.*, bench.lat.*) and a throwaway +stream. It does not purge product streams. Zapier cloud has no NATS +socket.

+
+

5. Suggestions for fine-tuning

+

These follow from the method and from the first ladder on this stand +(JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of +ms). Apply in order of leverage. Re-run this NS1 study +after each change so the delta is measured the same way.

+

5.1 Treat JetStream as +the product limiter

+

Product jobs/events/webhooks/archive are durable. Tuning core NATS to +2M msgs/s will not move a timestamp Zap. Put effort into replica +write path and consumer lag, not core +fan-out.

+

5.2 Split storage class by +stream

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
StreamSuggested storeWhy
ZAPIER_JOBSfile, r=3Work queue; lose-a-job is bad
ZAPIER_EVENTSfile r=3, or memory r=3 if events are rebuildable from job +statusHot waiters; measure both
ZAPIER_WEBHOOKSfile, r=3, workqueueHTTPS to Zapier is the slow consumer
ZAPIER_USAGEfile, r=3, limits + max-ageTelemetry
VERAE_ARCHIVEfile, r=3, on the best diskPuts are larger and must survive
+

Try ZAPIER_EVENTS as memory store in a maintenance +window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms +and durable events still ack at a higher rate, keep it; if a CT restart +drops in-flight waiters, revert.

+

5.3 Give JetStream real disks

+

Today r=3 on three LXC guests on one Proxmox host is +three files, one failure domain.

+
    +
  • Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) +into each nats LXC store_dir.
    • +
  • Set sync: always only on archive if you need it; +default sync is often enough for jobs and is faster. Measure.
    • +
  • Do not put JetStream store_dir on the same busy rootfs +as Forgejo/Caddy if we can avoid it.
    • +
  • When moving to three metal boxes: same configs, private NIC, one +disk (or mirror) per node. That is the first change +that makes r=3 mean “two boxes can die.”
    • +
+

5.4 Isolate the nats +CTs from the rest of NS1

+

Host load on this box is often already several. Pin:

+
    +
  • nats-a/b/c: dedicated cores, no steal from keep/fleet +Node processes.
    • +
  • Memory high enough that file-backed streams stay cache-hot for the +working set.
    • +
  • cpuunits / cpuset in pct config so a +Zapier-facing Node GC pause does not stall fsync.
    • +
+

Re-run this study after pinning; H1/H2 should move more than +ping.

+

5.5 Consumer and mailbox +tuning (delay H4)

+

Flood delay is backlog / consume_rate. Fine-tune the +waiters, not the broker RTT.

+
    +
  • jobs.events and webhooks.deliver: raise +max_ack_pending so a slow HTTPS hook does not stall the +whole consumer; cap it so a poison message cannot unbounded-buffer +RAM.
    • +
  • Pull consumers: larger batch, shorter expires, more +pullers horizontally (fleet replica floors) instead of one fat +subscriber.
    • +
  • Middleware should not flood-publish then wait; it +already does per-job publish. Keep that. The flood test is the outage +profile when a consumer is stopped.
    • +
  • Alert on consumer lag (pending + ack pending) from +JetStream, not on ping RTT.
    • +
+

5.6 Publisher-side +batching in middleware

+

A timestamp job is one small JSON. 16k msgs/s is ample. Still:

+
    +
  • Avoid per-byte publishes; one message per job/event.
    • +
  • Reuse NATS connections (connection churn showed up as publisher +spread in the core 4p/8p runs).
    • +
  • Idempotent msg id / duplicate window sized to Verae +retry window, not default-only.
    • +
+

5.7 +nats-server knobs worth measuring (A/B with this script)

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
KnobWhy try it
max_payloadKeep default unless archive puts grow
write_deadlineSlow consumer protection for webhooks
max_pendingBound memory on a stuck Zapier hook
max_connectionsFleet workers + keep + middleware
JetStream max_file_store / +max_memory_storePrevent one stream from filling the CT
max_outstanding_catchupReplica restart after a nats-c blip
GOMAXPROCS = LXC coresDo not overthread a 2-core CT
+

Change one knob, re-run +study-on-ns1.sh, compare JetStream 1p 128 B and ping +p99.

+

5.8 Network

+
    +
  • Keep NATS off vmbr0. No change.
    • +
  • When on metal: dedicated NIC or VLAN for cluster :6222 +vs client :4222 if possible (replication vs client +load).
    • +
  • Check virtio queue counts on the LXC nics if core 1 KiB byte rate +plateaus.
    • +
+

5.9 Security cost (when +nkeys/mTLS flip)

+

verae-nats-accounts is still a sketch. Enabling accounts +will add CPU on publish. Budget: re-run this exact study +after creds are in every NATS_URL, and +accept a drop on both core and JS. Do not flip without that +measurement.

+

5.10 Operational +fine-tuning (lag, not peak msgs/s)

+
    +
  1. Scrape varz / jsz from the host over +vmbr1 (not public). Monitor loopback :8222 is +invisible to Prometheus on NS1 unless we add a host-side proxy on +10.10.10.21:8222 bound only to vmbr1.
    2. +
  2. Keep replica floors for webhook-deliver and job-poller — they are +the flood defense.
    3. +
  3. Backup/restore drill of JetStream during idle, then +a short JS 1p run to see catchup cost.
    4. +
  4. A 15–30 minute soak (not in this ladder) for page cache and +compaction; add that as a third study when disks are dedicated.
    5. +
+

5.11 What not to tune

+
    +
  • Do not chase core 8p8s aggregate. It is fan-out on a lab +bridge.
    • +
  • Do not treat flood 400 ms as “cluster RTT.” Fix consumers.
    • +
  • Do not load-test on ZAPIER_* streams.
    • +
  • Do not bind client NATS to 0.0.0.0 on +vmbr0.
    • +
+ +
    +
  1. CPU pin nats-a/b/c → re-run JS 1p + ping.
    2. +
  2. ZAPIER_EVENTS-shaped memory stream vs file (throwaway +stream, same flags as this JS ladder).
    3. +
  3. Distinct store_dir disks per node.
    4. +
  4. nkeys on, same ladder.
    5. +
  5. Three hardware boxes, same cluster.env IPs +updated.
    6. +
+

Each experiment should produce a new +results/<utc>/ on NS1 and a new progress-repo report +so we can diff H1–H5 instead of arguing from memory.

+
+

6. Reproducing this study

+

On NS1 only:

+
cd ~/verae-src/verae-nats-cluster
+bash scripts/study-on-ns1.sh
+

The script exits if hostname is not NS1. Outputs land in +results/<utc>/ including +nats-cluster-bench-ns1.{md,html,pdf} and +charts/. Copy those into +zapier-decisions/reports/ for the progress repo and +catalog.

+

Raw logs for this run: results/20260912T053120Z/.

+ + diff --git a/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.md b/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.md new file mode 100644 index 0000000..b0d8c69 --- /dev/null +++ b/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.md @@ -0,0 +1,355 @@ +**Progress report (maximized NS1 study)** · run `20260912T053120Z` (UTC) + +> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): `maximize-ns1-study.sh` (cores/RAM/`max_mem`/tmpfs), then `study-on-ns1.sh`, `nats bench`, `latency.mjs` (LXC 510), matplotlib, pandoc, weasyprint. Traffic stayed on `vmbr1`. veth/10G was **not** changed. After the ladder, JetStream was put back on ZFS and product streams were re-created; **8 cores / 16 GiB / max_mem 8G stay**. + +## Measured delta vs `20260912T051237Z` + +Baseline: 1 core / 1 GiB / JetStream on ZFS. This run: 8 cores / 16 GiB / JetStream **tmpfs** (file r=3) plus extra **memory** store rows. veth/10G unchanged. + +| Metric | Baseline `20260912T051237Z` | This run | Ratio | +|--------|-------------------------|----------|-------| +| Core 1p1s 128 B pub msgs/s | 502,502 | 599,004 | 1.19× | +| Core 8p8s 128 B aggregate msgs/s | 2,065,217 | 1,998,733 | 0.97× | +| JS file r=3 1p 128 B pub msgs/s | 7,393 | 17,388 | 2.35× | +| JS file r=3 4p 128 B pub msgs/s | 17,986 | 22,853 | 1.27× | +| JS file r=3 4p 1 KiB pub msgs/s | 14,985 | 18,114 | 1.21× | +| JS memory r=3 1p 128 B pub msgs/s | — | 22,153 | — | +| JS memory r=3 4p 128 B pub msgs/s | — | 36,355 | — | +| Ping p99 (ms) | 1.377ms | 0.684ms | 2.01× faster | + +![Baseline vs maximized publish rates (log)](charts/delta-vs-baseline.png) + +*Baseline vs maximized publish rates (log)* + + +--- + +## 1. Executive summary + +| Item | This NS1-host run | +|------|-------------------| +| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `marchon` | +| Bench client | LXC 510 `verae-px-worker` | +| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` | +| Client URL | `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` | +| Host load before | `8.77 8.39 8.26 6/3849 198471` | +| Host load after | `9.02 8.82 8.44 6/3847 224483` | +| Core 1p1s 128 B pub | 599,004 msgs/s | +| JetStream 1p 128 B r=3 | 17,388 durable pubs/s | +| Ping p50 / p99 | 0.299ms / 0.684ms | + +Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst. + +--- + +## 2. Where it ran (and where it did not) + +```text +Operator laptop ──ssh──► NS1.GEORGELAMBERT.ORG 70.88.205.138 + study-on-ns1.sh + python3 build-ns1-study-report.py + sudo pct exec 510 ──► nats bench / latency.mjs + │ + ▼ vmbr1 + 10.10.10.21-23 :4222 +``` + +- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510). +- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file. + +--- + +## 3. Results (this run) + +### Host and brokers + +**Before** + +| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream | +|------|------|-------------|---------|----------|-----|-------|---------|-----------| +| nats-a | 511 | 3 | 3,013 | 3,037 | 1 | 8 | 14,553,088 | True | +| nats-b | 512 | 3 | 1,415 | 1,424 | 0 | 1 | 13,557,760 | True | +| nats-c | 513 | 0 | 1,361 | 1,395 | 1 | 1 | 14,028,800 | True | + +**After** + +| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream | +|------|------|-------------|---------|----------|-----|-------|---------|-----------| +| nats-a | 511 | 3 | 644,730 | 1,344,768 | 2 | 8 | 67,104,768 | True | +| nats-b | 512 | 3 | 491,295 | 978,800 | 1 | 1 | 82,542,592 | True | +| nats-c | 513 | 0 | 668,544 | 1,330,940 | 0 | 1 | 37,314,560 | True | + +nproc=40 · uname=`Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux` + +### Throughput + +| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s | +|-----|------|------------------|------------|----------|------------|----------| +| `core-1p1s-50k-128` | core pub/sub | 810,988 | 599,004 | 73.12 | 456,116 | 55.68 | +| `core-4p4s-100k-128` | core pub/sub | 1,361,921 | 379,346 | 46.31 | 1,150,536 | 140.45 | +| `core-4p4s-50k-1k` | core pub/sub | 695,192 | 175,725 | 171.61 | 584,024 | 570.34 | +| `core-8p8s-200k-128` | core pub/sub | 1,998,733 | 283,259 | 34.58 | 1,780,589 | 217.36 | +| `js-1p-20k-128-r3` | jetstream r=3 file | — | 17,388 | 2.12 | — | — | +| `js-2p2s-20k-128-r3` | jetstream r=3 file | 19,876 | 9,959 | 1.22 | 9,942 | 1.21 | +| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 18,114 | 17.69 | — | — | +| `js-4p-50k-128-r3` | jetstream r=3 file | — | 22,853 | 2.79 | — | — | +| `js-mem-1p-20k-128-r3` | jetstream r=3 file | — | 22,153 | 2.70 | — | — | +| `js-mem-4p-20k-1k-r3` | jetstream r=3 file | — | 28,685 | 28.01 | — | — | +| `js-mem-4p-50k-128-r3` | jetstream r=3 file | — | 36,355 | 4.44 | — | — | + +### Round-trip delay + +| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max | +|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----| +| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.250ms | 0.319ms | 0.299ms | 0.363ms | 0.684ms | 2.759ms | +| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 94.920ms | 132.026ms | 131.990ms | 156.086ms | 158.860ms | 158.965ms | +| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 113.281ms | 132.425ms | 131.457ms | 140.317ms | 144.683ms | 145.365ms | +| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 153.318ms | 204.329ms | 205.865ms | 234.151ms | 237.346ms | 237.477ms | +| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 233.727ms | 310.924ms | 311.170ms | 373.437ms | 400.764ms | 402.811ms | + +### Core NATS + +![Core NATS throughput at four loads (NS1 host run)](charts/core-throughput.png) + +*Core NATS throughput at four loads (NS1 host run)* +### Payload size (core) + +![Core NATS 128 B vs 1 KiB (NS1 host run)](charts/payload-size.png) + +*Core NATS 128 B vs 1 KiB (NS1 host run)* +### JetStream r=3 file + +![JetStream durable publish rate (NS1 host run)](charts/js-throughput.png) + +*JetStream durable publish rate (NS1 host run)* +### Core vs JetStream + +![Core vs JetStream publish rate, log scale (NS1 host run)](charts/core-vs-js.png) + +*Core vs JetStream publish rate, log scale (NS1 host run)* +### Delay + +![Ping vs flood delay percentiles, log scale (NS1 host run)](charts/delay-percentiles.png) + +*Ping vs flood delay percentiles, log scale (NS1 host run)* + +--- + +## 4. Study methodology + +### 4.1 Question + +On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)? + +### 4.2 Hypotheses (stated before the run) + +1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica. +2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated. +3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting. +4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count. +5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound. + +### 4.3 Independent variables (what we changed) + +| Factor | Levels | +|--------|--------| +| Transport | Core NATS pub/sub vs JetStream file replicas=3 | +| Publisher count | 1, 2, 4, 8 | +| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 | +| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) | +| Payload | 128 B, 1024 B | +| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) | + +### 4.4 Dependent variables (what we recorded) + +| Metric | Instrument | Unit | +|--------|------------|------| +| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s | +| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s | +| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) | +| Publisher spread | nats min/avg/max **msgs/s** | not delay | +| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max | +| Host load | `/proc/loadavg` before and after | load average | +| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem | + +**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`. + +### 4.5 Controls and constants + +- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`. +- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`). +- Bench client is LXC **510**, not a nats-* server. +- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide. +- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`). +- No TLS, no nkeys, no account isolation (isolation is `vmbr1`). +- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder. + +### 4.6 Procedure + +1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise. +2. Snapshot host load, memory, LXC configs, and each nats `varz`. +3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB). +4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file. +5. Copy `latency.mjs` into 510; ping then flood at several batch sizes. +6. Snapshot host/`varz` again. +7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**. + +No publish, subscribe, chart, or PDF process runs on the operator laptop for this study. + +### 4.7 Instrumentation path + +```text +[NS1 host 70.88.205.138] + study-on-ns1.sh (bash + python3) + | + | sudo pct exec 510 + v +[LXC 510 verae-px-worker 10.10.10.20] + nats bench / node latency.mjs + | + | NATS client protocol to + v +[LXC 511/512/513 10.10.10.21-23 :4222] + nats-server -js cluster routes :6222 +``` + +The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`. + +### 4.8 Threats to validity + +| Threat | Effect on numbers | +|--------|-------------------| +| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. | +| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. | +| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. | +| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. | +| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. | +| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. | +| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. | +| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. | + +### 4.9 Ethics / safety + +Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket. + +--- + +## 5. Suggestions for fine-tuning + +These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way. + +### 5.1 Treat JetStream as the product limiter + +Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out. + +### 5.2 Split storage class by stream + +| Stream | Suggested store | Why | +|--------|-----------------|-----| +| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad | +| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both | +| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer | +| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry | +| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive | + +Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert. + +### 5.3 Give JetStream real disks + +Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain. + +- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`. +- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure. +- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it. +- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.” + +### 5.4 Isolate the nats CTs from the rest of NS1 + +Host load on this box is often already several. Pin: + +- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes. +- Memory high enough that file-backed streams stay cache-hot for the working set. +- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync. + +Re-run this study after pinning; H1/H2 should move more than ping. + +### 5.5 Consumer and mailbox tuning (delay H4) + +Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT. + +- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM. +- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber. +- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped. +- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT. + +### 5.6 Publisher-side batching in middleware + +A timestamp job is one small JSON. 16k msgs/s is ample. Still: + +- Avoid per-byte publishes; one message per job/event. +- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs). +- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only. + +### 5.7 nats-server knobs worth measuring (A/B with this script) + +| Knob | Why try it | +|------|------------| +| `max_payload` | Keep default unless archive puts grow | +| `write_deadline` | Slow consumer protection for webhooks | +| `max_pending` | Bound memory on a stuck Zapier hook | +| `max_connections` | Fleet workers + keep + middleware | +| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT | +| `max_outstanding_catchup` | Replica restart after a nats-c blip | +| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT | + +Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99. + +### 5.8 Network + +- Keep NATS off `vmbr0`. No change. +- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load). +- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus. + +### 5.9 Security cost (when nkeys/mTLS flip) + +`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement. + +### 5.10 Operational fine-tuning (lag, not peak msgs/s) + +1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`. +2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense. +3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost. +4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated. + +### 5.11 What not to tune + +- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge. +- Do not treat flood 400 ms as “cluster RTT.” Fix consumers. +- Do not load-test on `ZAPIER_*` streams. +- Do not bind client NATS to `0.0.0.0` on `vmbr0`. + +### 5.12 Recommended next experiments (same method, one change each) + +1. CPU pin nats-a/b/c → re-run JS 1p + ping. +2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder). +3. Distinct `store_dir` disks per node. +4. nkeys on, same ladder. +5. Three hardware boxes, same `cluster.env` IPs updated. + +Each experiment should produce a new `results//` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory. + + +--- + +## 6. Reproducing this study + +On **NS1 only**: + +```bash +cd ~/verae-src/verae-nats-cluster +bash scripts/study-on-ns1.sh +``` + +The script exits if `hostname` is not NS1. Outputs land in `results//` including `nats-cluster-bench-ns1.{md,html,pdf}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog. + +Raw logs for this run: `results/20260912T053120Z/`. diff --git a/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.pdf b/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.pdf new file mode 100644 index 0000000..634bd81 Binary files /dev/null and b/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.pdf differ diff --git a/scripts/build-docs-site.py b/scripts/build-docs-site.py index 0fe838f..5608c82 100755 --- a/scripts/build-docs-site.py +++ b/scripts/build-docs-site.py @@ -866,6 +866,10 @@ def main() -> None: "packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.pdf", "NATS cluster study run entirely on NS1 (methodology + tuning)", ), + ( + "packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.pdf", + "Maximized NS1 study: 8 cores, 16 GiB, tmpfs JetStream (delta)", + ), ("packages/verae-nats-cluster/BENCH.pdf", "NATS cluster throughput tables (2026-09-12)"), ("overview/README.pdf", "System overview"), ("overview/INDEX.pdf", "Documentation index"), @@ -887,6 +891,10 @@ def main() -> None: "packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.html", "NATS cluster study run entirely on NS1", ), + ( + "packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.html", + "Maximized NS1 study (8c/16G/tmpfs)", + ), ("packages/verae-nats-cluster/BENCH.html", "NATS cluster throughput tables"), ("docs-master/MESSAGE-FLOWS.html", "Numbered message flows"), ("docs-master/modules-and-nats.html", "NATS address table"),