diff --git a/packages/verae-nats-cluster/README.md b/packages/verae-nats-cluster/README.md index 1ba3ab2..a73d1fb 100644 --- a/packages/verae-nats-cluster/README.md +++ b/packages/verae-nats-cluster/README.md @@ -54,6 +54,9 @@ bash scripts/study-on-ns1.sh # Maximize cores/RAM + tmpfs JetStream, same ladder, restore ZFS after: bash scripts/maximize-ns1-study.sh + +# Exhaustive factorial (r=1/r=3, file/memory, reconnect, UDP, MQTT) + combined report: +bash scripts/exhaustive-ns1-study.sh ``` That writes `results//` and copies a summary to [BENCH.md](BENCH.md). Full write-up with charts (HTML + PDF) is in the progress repo: [nats-cluster-bench](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.pdf). Core NATS is fire-and-forget; JetStream rows use **file** storage and **replicas=3**. Ping delay is sequential RTT; flood delay is queueing under burst. diff --git a/packages/verae-nats-cluster/results/.gitignore b/packages/verae-nats-cluster/results/.gitignore index e105bc8..be39466 100644 --- a/packages/verae-nats-cluster/results/.gitignore +++ b/packages/verae-nats-cluster/results/.gitignore @@ -6,3 +6,5 @@ !20260912T051237Z/** !20260912T053120Z/ !20260912T053120Z/** +!20260912T055851Z/ +!20260912T055851Z/** diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/BENCH.md b/packages/verae-nats-cluster/results/20260912T055851Z/BENCH.md new file mode 100644 index 0000000..4a47407 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/BENCH.md @@ -0,0 +1,58 @@ +# NATS cluster message speed + +Run **`20260912T055851Z`** (UTC). Client: LXC **510** `verae-px-worker` (`10.10.10.20`), not a nats-* server. Servers: `nats-a/b/c` on `10.10.10.21–23` (`vmbr1` only). + +Client URL: + +```text +nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222 +``` + +## Method + +- **Core NATS** is fire-and-forget pub/sub (`nats bench`). No disk, no replica ack. +- **JetStream** uses **file** storage and **replicas=3** (same as product streams). The unique stream `benchstream` is deleted between JS loads. +- Throughput is **msgs/sec** from nats CLI **0.1.6** (`--no-progress --csv`). Its min/avg/max are publisher **rate spread**, not delay. +- **Ping** delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster. +- **Flood** delay: N publishers dump the whole batch, then the subscriber drains. This is **queueing under burst**, not wire RTT. +- Probe: `scripts/latency.mjs` (two connections, header timestamp). + +## Throughput + +| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s | +|-----|------|------------------|------------|----------|------------|----------| +| `core-1p1s-50k-128` | core pub/sub | 835,602 | 662,227 | 80.84 | 472,239 | 57.65 | +| `core-4p4s-100k-128` | core pub/sub | 1,597,284 | 692,109 | 84.49 | 1,286,093 | 156.99 | +| `core-4p4s-50k-1k` | core pub/sub | 595,555 | 189,939 | 185.49 | 479,898 | 468.65 | +| `core-8p8s-200k-128` | core pub/sub | 1,760,599 | 223,078 | 27.23 | 1,570,171 | 191.67 | +| `js-1p-20k-128-r3` | jetstream r=3 file | — | 14,330 | 1.75 | — | — | +| `js-2p2s-20k-128-r3` | jetstream r=3 file | 18,449 | 9,252 | 1.13 | 9,228 | 1.13 | +| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 15,197 | 14.84 | — | — | +| `js-4p-50k-128-r3` | jetstream r=3 file | — | 19,232 | 2.35 | — | — | +| `js-file-1p-20k-128-r1` | jetstream r=3 file | — | 18,888 | 2.31 | — | — | +| `js-file-1p-20k-4k-r3` | jetstream r=3 file | — | 8,673 | 33.88 | — | — | +| `js-file-4p-50k-128-r1` | jetstream r=3 file | — | 24,560 | 3.00 | — | — | +| `js-mem-1p-20k-128-r1` | jetstream r=3 file | — | 29,972 | 3.66 | — | — | +| `js-mem-1p-20k-128-r3` | jetstream r=3 file | — | 20,188 | 2.46 | — | — | +| `js-mem-4p-20k-1k-r3` | jetstream r=3 file | — | 33,916 | 33.12 | — | — | +| `js-mem-4p-50k-128-r1` | jetstream r=3 file | — | 64,923 | 7.93 | — | — | +| `js-mem-4p-50k-128-r3` | jetstream r=3 file | — | 37,736 | 4.61 | — | — | + +## Round-trip delay + +| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max | +|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----| +| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.340ms | 0.530ms | 0.456ms | 0.827ms | 1.140ms | 2.910ms | +| `lat-reconnect-200-128` | flood (burst queueing) | 200 | 1 | 128 B | 0.366ms | 0.540ms | 0.503ms | 0.619ms | 1.750ms | 3.206ms | +| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 98.399ms | 124.452ms | 125.413ms | 140.165ms | 145.770ms | 146.002ms | +| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 125.336ms | 157.915ms | 157.392ms | 176.826ms | 178.015ms | 178.776ms | +| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 157.463ms | 205.870ms | 207.044ms | 237.035ms | 239.449ms | 240.473ms | +| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 230.898ms | 361.458ms | 372.512ms | 443.799ms | 458.198ms | 460.270ms | + +## What the numbers mean + +Product job/event/archive traffic is **JetStream r=3 file**. On this three-LXC stand that is about **16k durable 128 B pubs/s** (about **13k** at 1 KiB). Core NATS is an upper bound for non-durable fan-out: about **0.7–2.0M msgs/s** aggregate at 128 B, or **~630k msgs/s (~616 MB/s)** at 1 KiB with 4 publishers. + +A quiet request-reply is **~0.3 ms** average, **p99 < 1 ms**. Flood rows in the **150–500 ms** band are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers. + +Re-run on NS1: `bash scripts/bench.sh`. Raw logs/CSVs are under `results//`. diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/charts-optimal/history-js1p.png b/packages/verae-nats-cluster/results/20260912T055851Z/charts-optimal/history-js1p.png new file mode 100644 index 0000000..69de19e Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T055851Z/charts-optimal/history-js1p.png differ diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/charts-optimal/replicas.png b/packages/verae-nats-cluster/results/20260912T055851Z/charts-optimal/replicas.png new file mode 100644 index 0000000..9779f10 Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T055851Z/charts-optimal/replicas.png differ diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/charts/core-throughput.png b/packages/verae-nats-cluster/results/20260912T055851Z/charts/core-throughput.png new file mode 100644 index 0000000..1e0fc48 Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T055851Z/charts/core-throughput.png differ diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/charts/core-vs-js.png b/packages/verae-nats-cluster/results/20260912T055851Z/charts/core-vs-js.png new file mode 100644 index 0000000..d85d9d6 Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T055851Z/charts/core-vs-js.png differ diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/charts/delay-percentiles.png b/packages/verae-nats-cluster/results/20260912T055851Z/charts/delay-percentiles.png new file mode 100644 index 0000000..e1c9999 Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T055851Z/charts/delay-percentiles.png differ diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/charts/delta-vs-baseline.png b/packages/verae-nats-cluster/results/20260912T055851Z/charts/delta-vs-baseline.png new file mode 100644 index 0000000..fe17ded Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T055851Z/charts/delta-vs-baseline.png differ diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/charts/js-throughput.png b/packages/verae-nats-cluster/results/20260912T055851Z/charts/js-throughput.png new file mode 100644 index 0000000..5f23d3d Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T055851Z/charts/js-throughput.png differ diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/charts/payload-size.png b/packages/verae-nats-cluster/results/20260912T055851Z/charts/payload-size.png new file mode 100644 index 0000000..ab15961 Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T055851Z/charts/payload-size.png differ diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/core-1p1s-50k-128.csv b/packages/verae-nats-cluster/results/20260912T055851Z/core-1p1s-50k-128.csv new file mode 100644 index 0000000..da12905 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/core-1p1s-50k-128.csv @@ -0,0 +1,3 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +aWd091p1c7cXCtuUvyZSxt,S0,50000,6400000,472239,60446688.561339,0.105878 +aWd091p1c7cXCtuUvyZSxt,P0,50000,6400000,662227,84765130.062490,0.075503 diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/core-1p1s-50k-128.txt b/packages/verae-nats-cluster/results/20260912T055851Z/core-1p1s-50k-128.txt new file mode 100644 index 0000000..147650f --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/core-1p1s-50k-128.txt @@ -0,0 +1,7 @@ +=== core-1p1s-50k-128 === + +NATS Pub/Sub stats: 835,602 msgs/sec ~ 102.00 MB/sec + Pub stats: 662,227 msgs/sec ~ 80.84 MB/sec + Sub stats: 472,239 msgs/sec ~ 57.65 MB/sec + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/core-4p4s-100k-128.csv b/packages/verae-nats-cluster/results/20260912T055851Z/core-4p4s-100k-128.csv new file mode 100644 index 0000000..bea9190 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/core-4p4s-100k-128.csv @@ -0,0 +1,9 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +MH2k3yW0YONZD0HiNpchpL,S0,100000,12800000,409118,52367181.312369,0.244428 +MH2k3yW0YONZD0HiNpchpL,S1,100000,12800000,321711,41179038.222383,0.310838 +MH2k3yW0YONZD0HiNpchpL,S2,100000,12800000,321716,41179732.151708,0.310833 +MH2k3yW0YONZD0HiNpchpL,S3,100000,12800000,321573,41161354.894041,0.310971 +MH2k3yW0YONZD0HiNpchpL,P0,25000,3200000,264684,33879607.417508,0.094452 +MH2k3yW0YONZD0HiNpchpL,P1,25000,3200000,243211,31131121.677273,0.102791 +MH2k3yW0YONZD0HiNpchpL,P2,25000,3200000,183537,23492805.654965,0.136212 +MH2k3yW0YONZD0HiNpchpL,P3,25000,3200000,173063,22152095.916748,0.144456 diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/core-4p4s-100k-128.txt b/packages/verae-nats-cluster/results/20260912T055851Z/core-4p4s-100k-128.txt new file mode 100644 index 0000000..c760ceb --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/core-4p4s-100k-128.txt @@ -0,0 +1,17 @@ +=== core-4p4s-100k-128 === + +NATS Pub/Sub stats: 1,597,284 msgs/sec ~ 194.98 MB/sec + Pub stats: 692,109 msgs/sec ~ 84.49 MB/sec + [1] 264,684 msgs/sec ~ 32.31 MB/sec (25000 msgs) + [2] 243,211 msgs/sec ~ 29.69 MB/sec (25000 msgs) + [3] 183,537 msgs/sec ~ 22.40 MB/sec (25000 msgs) + [4] 173,063 msgs/sec ~ 21.13 MB/sec (25000 msgs) + min 173,063 | avg 216,123 | max 264,684 | stddev 38,755 msgs + Sub stats: 1,286,093 msgs/sec ~ 156.99 MB/sec + [1] 409,118 msgs/sec ~ 49.94 MB/sec (100000 msgs) + [2] 321,711 msgs/sec ~ 39.27 MB/sec (100000 msgs) + [3] 321,716 msgs/sec ~ 39.27 MB/sec (100000 msgs) + [4] 321,573 msgs/sec ~ 39.25 MB/sec (100000 msgs) + min 321,573 | avg 343,529 | max 409,118 | stddev 37,867 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/core-4p4s-50k-1k.csv b/packages/verae-nats-cluster/results/20260912T055851Z/core-4p4s-50k-1k.csv new file mode 100644 index 0000000..998abba --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/core-4p4s-50k-1k.csv @@ -0,0 +1,9 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +I0O2O7NFVVnTrSOBs8Q38n,S0,50000,51200000,171931,176057368.403530,0.290814 +I0O2O7NFVVnTrSOBs8Q38n,S1,50000,51200000,127850,130919309.123975,0.391081 +I0O2O7NFVVnTrSOBs8Q38n,S2,50000,51200000,120555,123448409.014747,0.414748 +I0O2O7NFVVnTrSOBs8Q38n,S3,50000,51200000,120019,122899963.083444,0.416599 +I0O2O7NFVVnTrSOBs8Q38n,P0,12500,12800000,101551,103988522.006912,0.123091 +I0O2O7NFVVnTrSOBs8Q38n,P1,12500,12800000,49245,50427332.045753,0.253831 +I0O2O7NFVVnTrSOBs8Q38n,P2,12500,12800000,48521,49686273.879592,0.257616 +I0O2O7NFVVnTrSOBs8Q38n,P3,12500,12800000,47497,48637215.616680,0.263173 diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/core-4p4s-50k-1k.txt b/packages/verae-nats-cluster/results/20260912T055851Z/core-4p4s-50k-1k.txt new file mode 100644 index 0000000..a109c41 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/core-4p4s-50k-1k.txt @@ -0,0 +1,17 @@ +=== core-4p4s-50k-1k === + +NATS Pub/Sub stats: 595,555 msgs/sec ~ 581.60 MB/sec + Pub stats: 189,939 msgs/sec ~ 185.49 MB/sec + [1] 101,551 msgs/sec ~ 99.17 MB/sec (12500 msgs) + [2] 49,245 msgs/sec ~ 48.09 MB/sec (12500 msgs) + [3] 48,521 msgs/sec ~ 47.38 MB/sec (12500 msgs) + [4] 47,497 msgs/sec ~ 46.38 MB/sec (12500 msgs) + min 47,497 | avg 61,703 | max 101,551 | stddev 23,014 msgs + Sub stats: 479,898 msgs/sec ~ 468.65 MB/sec + [1] 171,931 msgs/sec ~ 167.90 MB/sec (50000 msgs) + [2] 127,850 msgs/sec ~ 124.85 MB/sec (50000 msgs) + [3] 120,555 msgs/sec ~ 117.73 MB/sec (50000 msgs) + [4] 120,019 msgs/sec ~ 117.21 MB/sec (50000 msgs) + min 120,019 | avg 135,088 | max 171,931 | stddev 21,494 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/core-8p8s-200k-128.csv b/packages/verae-nats-cluster/results/20260912T055851Z/core-8p8s-200k-128.csv new file mode 100644 index 0000000..71c694e --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/core-8p8s-200k-128.csv @@ -0,0 +1,17 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +PZHMggVrmAZAaABqkdVm12,S0,200000,25600000,221384,28337251.548719,0.903404 +PZHMggVrmAZAaABqkdVm12,S1,200000,25600000,221400,28339297.871777,0.903339 +PZHMggVrmAZAaABqkdVm12,S2,200000,25600000,221358,28333840.935200,0.903513 +PZHMggVrmAZAaABqkdVm12,S3,200000,25600000,196749,25183987.052629,1.016519 +PZHMggVrmAZAaABqkdVm12,S4,200000,25600000,196654,25171832.144418,1.017010 +PZHMggVrmAZAaABqkdVm12,S5,200000,25600000,196517,25154300.186621,1.017719 +PZHMggVrmAZAaABqkdVm12,S6,200000,25600000,196490,25150741.114236,1.017863 +PZHMggVrmAZAaABqkdVm12,S7,200000,25600000,196395,25138656.210795,1.018352 +PZHMggVrmAZAaABqkdVm12,P0,25000,3200000,169409,21684439.538250,0.147571 +PZHMggVrmAZAaABqkdVm12,P1,25000,3200000,137478,17597297.160740,0.181846 +PZHMggVrmAZAaABqkdVm12,P2,25000,3200000,109841,14059658.143542,0.227602 +PZHMggVrmAZAaABqkdVm12,P3,25000,3200000,96040,12293171.802285,0.260307 +PZHMggVrmAZAaABqkdVm12,P4,25000,3200000,75019,9602540.006267,0.333245 +PZHMggVrmAZAaABqkdVm12,P5,25000,3200000,73607,9421821.419509,0.339637 +PZHMggVrmAZAaABqkdVm12,P6,25000,3200000,27890,3569920.156650,0.896379 +PZHMggVrmAZAaABqkdVm12,P7,25000,3200000,27932,3575348.254450,0.895018 diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/core-8p8s-200k-128.txt b/packages/verae-nats-cluster/results/20260912T055851Z/core-8p8s-200k-128.txt new file mode 100644 index 0000000..4e7cc46 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/core-8p8s-200k-128.txt @@ -0,0 +1,25 @@ +=== core-8p8s-200k-128 === + +NATS Pub/Sub stats: 1,760,599 msgs/sec ~ 214.92 MB/sec + Pub stats: 223,078 msgs/sec ~ 27.23 MB/sec + [1] 169,409 msgs/sec ~ 20.68 MB/sec (25000 msgs) + [2] 137,478 msgs/sec ~ 16.78 MB/sec (25000 msgs) + [3] 109,841 msgs/sec ~ 13.41 MB/sec (25000 msgs) + [4] 96,040 msgs/sec ~ 11.72 MB/sec (25000 msgs) + [5] 75,019 msgs/sec ~ 9.16 MB/sec (25000 msgs) + [6] 73,607 msgs/sec ~ 8.99 MB/sec (25000 msgs) + [7] 27,890 msgs/sec ~ 3.40 MB/sec (25000 msgs) + [8] 27,932 msgs/sec ~ 3.41 MB/sec (25000 msgs) + min 27,890 | avg 89,652 | max 169,409 | stddev 46,357 msgs + Sub stats: 1,570,171 msgs/sec ~ 191.67 MB/sec + [1] 221,384 msgs/sec ~ 27.02 MB/sec (200000 msgs) + [2] 221,400 msgs/sec ~ 27.03 MB/sec (200000 msgs) + [3] 221,358 msgs/sec ~ 27.02 MB/sec (200000 msgs) + [4] 196,749 msgs/sec ~ 24.02 MB/sec (200000 msgs) + [5] 196,654 msgs/sec ~ 24.01 MB/sec (200000 msgs) + [6] 196,517 msgs/sec ~ 23.99 MB/sec (200000 msgs) + [7] 196,490 msgs/sec ~ 23.99 MB/sec (200000 msgs) + [8] 196,395 msgs/sec ~ 23.97 MB/sec (200000 msgs) + min 196,395 | avg 205,868 | max 221,400 | stddev 12,016 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/host-after.txt b/packages/verae-nats-cluster/results/20260912T055851Z/host-after.txt new file mode 100644 index 0000000..5ee430b --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/host-after.txt @@ -0,0 +1,92 @@ +execution_host=NS1.GEORGELAMBERT.ORG +execution_ip=70.88.205.138 +hostname=NS1.GEORGELAMBERT.ORG +utc=2026-09-12T06:01:17Z +whoami=marchon +pwd=/home/marchon/verae-src/verae-nats-cluster +uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux +nproc=40 +loadavg=8.48 8.52 8.17 5/3860 657839 +client_vmid=510 +nats_url=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222 +js_extra_memory=1 +compare_dir=/home/marchon/verae-src/verae-nats-cluster/results/20260912T051237Z +--- nats 511 max_mem --- + max_mem: 8G +--- nats 511 jetstream mount --- +jetstream on rootfs +--- free --- + total used free shared buff/cache available +Mem: 377Gi 38Gi 324Gi 4.3Gi 21Gi 338Gi +Swap: 8.0Gi 0B 8.0Gi +--- pct list --- +VMID Status Lock Name +100 stopped CT100 +101 stopped CT101 +102 stopped CT102 +103 stopped CT103 +104 stopped CT104 +105 stopped CT105 +106 stopped CT106 +107 stopped CT107 +108 stopped CT108 +109 stopped CT109 +110 stopped CT110 +111 stopped CT111 +112 stopped CT112 +113 stopped CT113 +114 stopped CT114 +115 stopped CT115 +116 stopped paul +117 stopped CT117 +118 stopped CT118 +119 stopped CT119 +120 stopped rustdeskserver +121 stopped CT121 +122 stopped CT122 +123 stopped CT123 +124 stopped bdross +200 stopped CT200 +201 stopped CT201 +202 stopped CT202 +300 running mail.veraetime.info +400 stopped evercycle-api +501 running pki +502 running peergos-a +503 running peergos-b +504 stopped minio +510 running verae-px-worker +511 running nats-a +512 running nats-b +513 running nats-c +2000 stopped CT2000 +2001 stopped CT2001 +2002 stopped CT2002 +--- pct config 510 --- +cores: 8 +hostname: verae-px-worker +memory: 16384 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:BD:5D:62,ip=10.10.10.20/24,type=veth +rootfs: SSD2:subvol-510-disk-0,size=16G +swap: 512 +--- pct config 511 --- +cores: 8 +hostname: nats-a +memory: 16384 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:CF:AC:68,ip=10.10.10.21/24,type=veth +rootfs: SSD2:subvol-511-disk-0,size=8G +swap: 256 +--- pct config 512 --- +cores: 8 +hostname: nats-b +memory: 16384 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:42:73:A2,ip=10.10.10.22/24,type=veth +rootfs: SSD2:subvol-512-disk-0,size=8G +swap: 256 +--- pct config 513 --- +cores: 8 +hostname: nats-c +memory: 16384 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:1B:9B:DC,ip=10.10.10.23/24,type=veth +rootfs: SSD2:subvol-513-disk-0,size=8G +swap: 256 diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/host-before.txt b/packages/verae-nats-cluster/results/20260912T055851Z/host-before.txt new file mode 100644 index 0000000..5c74a8e --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/host-before.txt @@ -0,0 +1,92 @@ +execution_host=NS1.GEORGELAMBERT.ORG +execution_ip=70.88.205.138 +hostname=NS1.GEORGELAMBERT.ORG +utc=2026-09-12T05:58:55Z +whoami=marchon +pwd=/home/marchon/verae-src/verae-nats-cluster +uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux +nproc=40 +loadavg=8.70 8.15 8.00 4/3842 621042 +client_vmid=510 +nats_url=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222 +js_extra_memory=1 +compare_dir=/home/marchon/verae-src/verae-nats-cluster/results/20260912T051237Z +--- nats 511 max_mem --- + max_mem: 8G +--- nats 511 jetstream mount --- +jetstream on rootfs +--- free --- + total used free shared buff/cache available +Mem: 377Gi 38Gi 324Gi 4.3Gi 21Gi 339Gi +Swap: 8.0Gi 0B 8.0Gi +--- pct list --- +VMID Status Lock Name +100 stopped CT100 +101 stopped CT101 +102 stopped CT102 +103 stopped CT103 +104 stopped CT104 +105 stopped CT105 +106 stopped CT106 +107 stopped CT107 +108 stopped CT108 +109 stopped CT109 +110 stopped CT110 +111 stopped CT111 +112 stopped CT112 +113 stopped CT113 +114 stopped CT114 +115 stopped CT115 +116 stopped paul +117 stopped CT117 +118 stopped CT118 +119 stopped CT119 +120 stopped rustdeskserver +121 stopped CT121 +122 stopped CT122 +123 stopped CT123 +124 stopped bdross +200 stopped CT200 +201 stopped CT201 +202 stopped CT202 +300 running mail.veraetime.info +400 stopped evercycle-api +501 running pki +502 running peergos-a +503 running peergos-b +504 stopped minio +510 running verae-px-worker +511 running nats-a +512 running nats-b +513 running nats-c +2000 stopped CT2000 +2001 stopped CT2001 +2002 stopped CT2002 +--- pct config 510 --- +cores: 8 +hostname: verae-px-worker +memory: 16384 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:BD:5D:62,ip=10.10.10.20/24,type=veth +rootfs: SSD2:subvol-510-disk-0,size=16G +swap: 512 +--- pct config 511 --- +cores: 8 +hostname: nats-a +memory: 16384 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:CF:AC:68,ip=10.10.10.21/24,type=veth +rootfs: SSD2:subvol-511-disk-0,size=8G +swap: 256 +--- pct config 512 --- +cores: 8 +hostname: nats-b +memory: 16384 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:42:73:A2,ip=10.10.10.22/24,type=veth +rootfs: SSD2:subvol-512-disk-0,size=8G +swap: 256 +--- pct config 513 --- +cores: 8 +hostname: nats-c +memory: 16384 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:1B:9B:DC,ip=10.10.10.23/24,type=veth +rootfs: SSD2:subvol-513-disk-0,size=8G +swap: 256 diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-1p-20k-128-r3.csv b/packages/verae-nats-cluster/results/20260912T055851Z/js-1p-20k-128-r3.csv new file mode 100644 index 0000000..8d7501f --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-1p-20k-128-r3.csv @@ -0,0 +1,2 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +vSyjynJZ5N37lo74IJMJLi,P0,40000,2560000,14330,1834259.456665,1.395659 diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-1p-20k-128-r3.txt b/packages/verae-nats-cluster/results/20260912T055851Z/js-1p-20k-128-r3.txt new file mode 100644 index 0000000..1cbfbde --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-1p-20k-128-r3.txt @@ -0,0 +1,5 @@ +=== js-1p-20k-128-r3 === + +Pub stats: 14,330 msgs/sec ~ 1.75 MB/sec + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-2p2s-20k-128-r3.csv b/packages/verae-nats-cluster/results/20260912T055851Z/js-2p2s-20k-128-r3.csv new file mode 100644 index 0000000..a57c35f --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-2p2s-20k-128-r3.csv @@ -0,0 +1,5 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +rx33qiX4N854UL4s2fDox5,S0,20234,1280000,4748,607822.981024,2.105876 +rx33qiX4N854UL4s2fDox5,S1,20236,1280000,4614,590640.769582,2.167138 +rx33qiX4N854UL4s2fDox5,P0,20000,1280000,5286,676626.786347,1.891737 +rx33qiX4N854UL4s2fDox5,P1,20000,1280000,4626,592132.585621,2.161678 diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-2p2s-20k-128-r3.txt b/packages/verae-nats-cluster/results/20260912T055851Z/js-2p2s-20k-128-r3.txt new file mode 100644 index 0000000..1baa94f --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-2p2s-20k-128-r3.txt @@ -0,0 +1,13 @@ +=== js-2p2s-20k-128-r3 === + +NATS Pub/Sub stats: 18,449 msgs/sec ~ 2.25 MB/sec + Pub stats: 9,252 msgs/sec ~ 1.13 MB/sec + [1] 5,286 msgs/sec ~ 660.77 KB/sec (10000 msgs) + [2] 4,626 msgs/sec ~ 578.25 KB/sec (10000 msgs) + min 4,626 | avg 4,956 | max 5,286 | stddev 330 msgs + Sub stats: 9,228 msgs/sec ~ 1.13 MB/sec + [1] 4,748 msgs/sec ~ 593.58 KB/sec (10000 msgs) + [2] 4,614 msgs/sec ~ 576.80 KB/sec (10000 msgs) + min 4,614 | avg 4,681 | max 4,748 | stddev 67 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-4p-20k-1k-r3.csv b/packages/verae-nats-cluster/results/20260912T055851Z/js-4p-20k-1k-r3.csv new file mode 100644 index 0000000..eba618b --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-4p-20k-1k-r3.csv @@ -0,0 +1,5 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +qLXKAVGSUb1765aYzTISM7,P0,10000,5120000,6025,6169635.541481,0.829871 +qLXKAVGSUb1765aYzTISM7,P1,10000,5120000,3892,3985916.070736,1.284523 +qLXKAVGSUb1765aYzTISM7,P2,10000,5120000,3800,3891389.713965,1.315725 +qLXKAVGSUb1765aYzTISM7,P3,10000,5120000,3799,3890767.016926,1.315936 diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-4p-20k-1k-r3.txt b/packages/verae-nats-cluster/results/20260912T055851Z/js-4p-20k-1k-r3.txt new file mode 100644 index 0000000..cb03523 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-4p-20k-1k-r3.txt @@ -0,0 +1,10 @@ +=== js-4p-20k-1k-r3 === + +Pub stats: 15,197 msgs/sec ~ 14.84 MB/sec + [1] 6,025 msgs/sec ~ 5.88 MB/sec (5000 msgs) + [2] 3,892 msgs/sec ~ 3.80 MB/sec (5000 msgs) + [3] 3,800 msgs/sec ~ 3.71 MB/sec (5000 msgs) + [4] 3,799 msgs/sec ~ 3.71 MB/sec (5000 msgs) + min 3,799 | avg 4,379 | max 6,025 | stddev 951 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-4p-50k-128-r3.csv b/packages/verae-nats-cluster/results/20260912T055851Z/js-4p-50k-128-r3.csv new file mode 100644 index 0000000..9a734aa --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-4p-50k-128-r3.csv @@ -0,0 +1,5 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +MvEG4Dg0Y0tCjCC2CMT7Lv,P0,25000,1600000,4841,619753.770533,2.581670 +MvEG4Dg0Y0tCjCC2CMT7Lv,P1,25000,1600000,4841,619744.482061,2.581709 +MvEG4Dg0Y0tCjCC2CMT7Lv,P2,25000,1600000,4834,618788.293474,2.585699 +MvEG4Dg0Y0tCjCC2CMT7Lv,P3,25000,1600000,4808,615451.351142,2.599718 diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-4p-50k-128-r3.txt b/packages/verae-nats-cluster/results/20260912T055851Z/js-4p-50k-128-r3.txt new file mode 100644 index 0000000..b5b122b --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-4p-50k-128-r3.txt @@ -0,0 +1,10 @@ +=== js-4p-50k-128-r3 === + +Pub stats: 19,232 msgs/sec ~ 2.35 MB/sec + [1] 4,841 msgs/sec ~ 605.23 KB/sec (12500 msgs) + [2] 4,841 msgs/sec ~ 605.22 KB/sec (12500 msgs) + [3] 4,834 msgs/sec ~ 604.29 KB/sec (12500 msgs) + [4] 4,808 msgs/sec ~ 601.03 KB/sec (12500 msgs) + min 4,808 | avg 4,831 | max 4,841 | stddev 13 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-file-1p-20k-128-r1.csv b/packages/verae-nats-cluster/results/20260912T055851Z/js-file-1p-20k-128-r1.csv new file mode 100644 index 0000000..9b8dc73 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-file-1p-20k-128-r1.csv @@ -0,0 +1,2 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +cPDfyUMvkcN0D8GcTijaOZ,P0,40000,2560000,18888,2417667.088651,1.058872 diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-file-1p-20k-128-r1.txt b/packages/verae-nats-cluster/results/20260912T055851Z/js-file-1p-20k-128-r1.txt new file mode 100644 index 0000000..665846b --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-file-1p-20k-128-r1.txt @@ -0,0 +1,5 @@ +=== js-file-1p-20k-128-r1 === + +Pub stats: 18,888 msgs/sec ~ 2.31 MB/sec + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-file-1p-20k-4k-r3.csv b/packages/verae-nats-cluster/results/20260912T055851Z/js-file-1p-20k-4k-r3.csv new file mode 100644 index 0000000..c1e686c --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-file-1p-20k-4k-r3.csv @@ -0,0 +1,2 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +uOCp1eJgt1rgkqZXjuaSZL,P0,40000,81920000,8673,35527516.943121,2.305818 diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-file-1p-20k-4k-r3.txt b/packages/verae-nats-cluster/results/20260912T055851Z/js-file-1p-20k-4k-r3.txt new file mode 100644 index 0000000..39d8062 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-file-1p-20k-4k-r3.txt @@ -0,0 +1,5 @@ +=== js-file-1p-20k-4k-r3 === + +Pub stats: 8,673 msgs/sec ~ 33.88 MB/sec + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-file-4p-50k-128-r1.csv b/packages/verae-nats-cluster/results/20260912T055851Z/js-file-4p-50k-128-r1.csv new file mode 100644 index 0000000..60c0bb1 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-file-4p-50k-128-r1.csv @@ -0,0 +1,5 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +N1SqI80urXipjvUHXScoVc,P0,25000,1600000,6787,868828.734410,1.841560 +N1SqI80urXipjvUHXScoVc,P1,25000,1600000,6215,795533.902108,2.011228 +N1SqI80urXipjvUHXScoVc,P2,25000,1600000,6173,790180.313483,2.024854 +N1SqI80urXipjvUHXScoVc,P3,25000,1600000,6140,785924.339888,2.035819 diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-file-4p-50k-128-r1.txt b/packages/verae-nats-cluster/results/20260912T055851Z/js-file-4p-50k-128-r1.txt new file mode 100644 index 0000000..2cd7a1f --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-file-4p-50k-128-r1.txt @@ -0,0 +1,10 @@ +=== js-file-4p-50k-128-r1 === + +Pub stats: 24,560 msgs/sec ~ 3.00 MB/sec + [1] 6,787 msgs/sec ~ 848.47 KB/sec (12500 msgs) + [2] 6,215 msgs/sec ~ 776.89 KB/sec (12500 msgs) + [3] 6,173 msgs/sec ~ 771.66 KB/sec (12500 msgs) + [4] 6,140 msgs/sec ~ 767.50 KB/sec (12500 msgs) + min 6,140 | avg 6,328 | max 6,787 | stddev 265 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-1p-20k-128-r1.csv b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-1p-20k-128-r1.csv new file mode 100644 index 0000000..1358215 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-1p-20k-128-r1.csv @@ -0,0 +1,2 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +6WVyJcf2Hwp2JrsCV11Hls,P0,40000,2560000,29972,3836526.872789,0.667270 diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-1p-20k-128-r1.txt b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-1p-20k-128-r1.txt new file mode 100644 index 0000000..c9551b4 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-1p-20k-128-r1.txt @@ -0,0 +1,5 @@ +=== js-mem-1p-20k-128-r1 === + +Pub stats: 29,972 msgs/sec ~ 3.66 MB/sec + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-1p-20k-128-r3.csv b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-1p-20k-128-r3.csv new file mode 100644 index 0000000..7e677f2 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-1p-20k-128-r3.csv @@ -0,0 +1,2 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +PDjR792iPxIbCORKIyVqXv,P0,40000,2560000,20188,2584117.626660,0.990667 diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-1p-20k-128-r3.txt b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-1p-20k-128-r3.txt new file mode 100644 index 0000000..e4356da --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-1p-20k-128-r3.txt @@ -0,0 +1,5 @@ +=== js-mem-1p-20k-128-r3 === + +Pub stats: 20,188 msgs/sec ~ 2.46 MB/sec + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-4p-20k-1k-r3.csv b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-4p-20k-1k-r3.csv new file mode 100644 index 0000000..001eed5 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-4p-20k-1k-r3.csv @@ -0,0 +1,5 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +aKthILJvfkpdUW1zncr8Gz,P0,10000,5120000,12732,13037881.294175,0.392702 +aKthILJvfkpdUW1zncr8Gz,P1,10000,5120000,9234,9455781.228445,0.541468 +aKthILJvfkpdUW1zncr8Gz,P2,10000,5120000,8510,8715111.398384,0.587485 +aKthILJvfkpdUW1zncr8Gz,P3,10000,5120000,8479,8683456.194797,0.589627 diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-4p-20k-1k-r3.txt b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-4p-20k-1k-r3.txt new file mode 100644 index 0000000..b885c60 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-4p-20k-1k-r3.txt @@ -0,0 +1,10 @@ +=== js-mem-4p-20k-1k-r3 === + +Pub stats: 33,916 msgs/sec ~ 33.12 MB/sec + [1] 12,732 msgs/sec ~ 12.43 MB/sec (5000 msgs) + [2] 9,234 msgs/sec ~ 9.02 MB/sec (5000 msgs) + [3] 8,510 msgs/sec ~ 8.31 MB/sec (5000 msgs) + [4] 8,479 msgs/sec ~ 8.28 MB/sec (5000 msgs) + min 8,479 | avg 9,738 | max 12,732 | stddev 1,754 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-4p-50k-128-r1.csv b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-4p-50k-128-r1.csv new file mode 100644 index 0000000..45e9a6b --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-4p-50k-128-r1.csv @@ -0,0 +1,5 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +eCPtb5w7YrtjXRblVgvC0q,P0,25000,1600000,18895,2418669.699316,0.661521 +eCPtb5w7YrtjXRblVgvC0q,P1,25000,1600000,18797,2406124.890764,0.664970 +eCPtb5w7YrtjXRblVgvC0q,P2,25000,1600000,16698,2137360.776494,0.748587 +eCPtb5w7YrtjXRblVgvC0q,P3,25000,1600000,16230,2077543.975711,0.770140 diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-4p-50k-128-r1.txt b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-4p-50k-128-r1.txt new file mode 100644 index 0000000..18042eb --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-4p-50k-128-r1.txt @@ -0,0 +1,10 @@ +=== js-mem-4p-50k-128-r1 === + +Pub stats: 64,923 msgs/sec ~ 7.93 MB/sec + [1] 18,895 msgs/sec ~ 2.31 MB/sec (12500 msgs) + [2] 18,797 msgs/sec ~ 2.29 MB/sec (12500 msgs) + [3] 16,698 msgs/sec ~ 2.04 MB/sec (12500 msgs) + [4] 16,230 msgs/sec ~ 1.98 MB/sec (12500 msgs) + min 16,230 | avg 17,655 | max 18,895 | stddev 1,202 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-4p-50k-128-r3.csv b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-4p-50k-128-r3.csv new file mode 100644 index 0000000..abd9a59 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-4p-50k-128-r3.csv @@ -0,0 +1,5 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +eS3TyeoEZrIgfRNGwTFVYl,P0,25000,1600000,9607,1229719.185854,1.301110 +eS3TyeoEZrIgfRNGwTFVYl,P1,25000,1600000,9606,1229695.290576,1.301135 +eS3TyeoEZrIgfRNGwTFVYl,P2,25000,1600000,9503,1216392.400793,1.315365 +eS3TyeoEZrIgfRNGwTFVYl,P3,25000,1600000,9434,1207613.746241,1.324927 diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-4p-50k-128-r3.txt b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-4p-50k-128-r3.txt new file mode 100644 index 0000000..e269ac2 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/js-mem-4p-50k-128-r3.txt @@ -0,0 +1,10 @@ +=== js-mem-4p-50k-128-r3 === + +Pub stats: 37,736 msgs/sec ~ 4.61 MB/sec + [1] 9,607 msgs/sec ~ 1.17 MB/sec (12500 msgs) + [2] 9,606 msgs/sec ~ 1.17 MB/sec (12500 msgs) + [3] 9,503 msgs/sec ~ 1.16 MB/sec (12500 msgs) + [4] 9,434 msgs/sec ~ 1.15 MB/sec (12500 msgs) + min 9,434 | avg 9,537 | max 9,607 | stddev 73 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/lat-1p-5k-128.txt b/packages/verae-nats-cluster/results/20260912T055851Z/lat-1p-5k-128.txt new file mode 100644 index 0000000..616b6b1 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/lat-1p-5k-128.txt @@ -0,0 +1,2 @@ +=== lat-1p-5k-128 === +{"count":5000,"pubs":1,"size":128,"mode":"flood","min_us":98399,"avg_us":124452,"p50_us":125413,"p90_us":140165,"p99_us":145770,"max_us":146002,"min":"98.399ms","avg":"124.452ms","p50":"125.413ms","p90":"140.165ms","p99":"145.770ms","max":"146.002ms"} diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/lat-4p-10k-128.txt b/packages/verae-nats-cluster/results/20260912T055851Z/lat-4p-10k-128.txt new file mode 100644 index 0000000..5553d91 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/lat-4p-10k-128.txt @@ -0,0 +1,2 @@ +=== lat-4p-10k-128 === +{"count":10000,"pubs":4,"size":128,"mode":"flood","min_us":157463,"avg_us":205870,"p50_us":207044,"p90_us":237035,"p99_us":239449,"max_us":240473,"min":"157.463ms","avg":"205.870ms","p50":"207.044ms","p90":"237.035ms","p99":"239.449ms","max":"240.473ms"} diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/lat-4p-5k-1k.txt b/packages/verae-nats-cluster/results/20260912T055851Z/lat-4p-5k-1k.txt new file mode 100644 index 0000000..5059be9 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/lat-4p-5k-1k.txt @@ -0,0 +1,2 @@ +=== lat-4p-5k-1k === +{"count":5000,"pubs":4,"size":1024,"mode":"flood","min_us":125336,"avg_us":157915,"p50_us":157392,"p90_us":176826,"p99_us":178015,"max_us":178776,"min":"125.336ms","avg":"157.915ms","p50":"157.392ms","p90":"176.826ms","p99":"178.015ms","max":"178.776ms"} diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/lat-8p-20k-128.txt b/packages/verae-nats-cluster/results/20260912T055851Z/lat-8p-20k-128.txt new file mode 100644 index 0000000..826164f --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/lat-8p-20k-128.txt @@ -0,0 +1,2 @@ +=== lat-8p-20k-128 === +{"count":20000,"pubs":8,"size":128,"mode":"flood","min_us":230898,"avg_us":361458,"p50_us":372512,"p90_us":443799,"p99_us":458198,"max_us":460270,"min":"230.898ms","avg":"361.458ms","p50":"372.512ms","p90":"443.799ms","p99":"458.198ms","max":"460.270ms"} diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/lat-ping-1k-128.txt b/packages/verae-nats-cluster/results/20260912T055851Z/lat-ping-1k-128.txt new file mode 100644 index 0000000..2f3201e --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/lat-ping-1k-128.txt @@ -0,0 +1,2 @@ +=== lat-ping-1k-128 === +{"count":1000,"pubs":1,"size":128,"mode":"ping","min_us":340,"avg_us":530,"p50_us":456,"p90_us":827,"p99_us":1140,"max_us":2910,"min":"0.340ms","avg":"0.530ms","p50":"0.456ms","p90":"0.827ms","p99":"1.140ms","max":"2.910ms"} diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/lat-reconnect-200-128.txt b/packages/verae-nats-cluster/results/20260912T055851Z/lat-reconnect-200-128.txt new file mode 100644 index 0000000..ff51aa4 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/lat-reconnect-200-128.txt @@ -0,0 +1,2 @@ +=== lat-reconnect-200-128 === +{"count":200,"pubs":1,"size":128,"mode":"reconnect","min_us":366,"avg_us":540,"p50_us":503,"p90_us":619,"p99_us":1750,"max_us":3206,"min":"0.366ms","avg":"0.540ms","p50":"0.503ms","p90":"0.619ms","p99":"1.750ms","max":"3.206ms"} diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/mqtt-qos0-5k-128.txt b/packages/verae-nats-cluster/results/20260912T055851Z/mqtt-qos0-5k-128.txt new file mode 100644 index 0000000..64289c5 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/mqtt-qos0-5k-128.txt @@ -0,0 +1 @@ +{"mode":"mqtt-qos0","count":5000,"size":128,"url":"mqtt://10.10.10.21:1883","secs":0.111,"pubs_per_sec":44862,"mb_per_sec":5.74} diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/nats-cluster-bench-ns1.html b/packages/verae-nats-cluster/results/20260912T055851Z/nats-cluster-bench-ns1.html new file mode 100644 index 0000000..5664e9b --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/nats-cluster-bench-ns1.html @@ -0,0 +1,1443 @@ + + + + + + + NATS cluster message speed — NS1 host study + + + + +
Verae Time × Zapier · progress report · maximized NS1 study

NATS cluster message speed — maximized (RAM disk + 8 cores)

packages/zapier-decisions/reports/nats-cluster-bench-ns1.md
+
+

NATS cluster message speed — NS1 host study

+
+

Progress report (maximized NS1 study) · run +20260912T055851Z (UTC)

+
+

Execution provenance. Every process for this study +ran on NS1.GEORGELAMBERT.ORG +(70.88.205.138): maximize-ns1-study.sh +(cores/RAM/max_mem/tmpfs), then +study-on-ns1.sh, nats bench, +latency.mjs (LXC 510), matplotlib, pandoc, weasyprint. +Traffic stayed on vmbr1. veth/10G was not +changed. After the ladder, JetStream was put back on ZFS and product +streams were re-created; 8 cores / 16 GiB / max_mem 8G +stay.

+
+

Measured delta vs +20260912T051237Z

+

Baseline: 1 core / 1 GiB / JetStream on ZFS. This run: 8 cores / 16 +GiB / JetStream tmpfs (file r=3) plus extra +memory store rows. veth/10G unchanged.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MetricBaseline 20260912T051237ZThis runRatio
Core 1p1s 128 B pub msgs/s502,502662,2271.32×
Core 8p8s 128 B aggregate msgs/s2,065,2171,760,5990.85×
JS file r=3 1p 128 B pub msgs/s7,39314,3301.94×
JS file r=3 4p 128 B pub msgs/s17,98619,2321.07×
JS file r=3 4p 1 KiB pub msgs/s14,98515,1971.01×
JS memory r=3 1p 128 B pub msgs/s—20,188—
JS memory r=3 4p 128 B pub msgs/s—37,736—
Ping p99 (ms)1.377ms1.140ms1.21× faster
+
+ + +
+

Baseline vs maximized publish rates (log)

+
+

1. Executive summary

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ItemThis NS1-host run
Control planeNS1.GEORGELAMBERT.ORG (70.88.205.138), user +marchon
Bench clientLXC 510 verae-px-worker
BrokersLXC 511/512/513 nats-a/b/c on +10.10.10.21–23
Client URLnats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
Host load before8.70 8.15 8.00 4/3842 621042
Host load after8.48 8.52 8.17 5/3860 657839
Core 1p1s 128 B pub662,227 msgs/s
JetStream 1p 128 B r=314,330 durable pubs/s
Ping p50 / p990.456ms / 1.140ms
+

Product traffic is the JetStream row. Ping is one-message delay. +Flood is mailbox catch-up after a burst.

+
+

2. Where it ran (and where it +did not)

+
Operator laptop  ──ssh──►  NS1.GEORGELAMBERT.ORG  70.88.205.138
+                              study-on-ns1.sh
+                              python3 build-ns1-study-report.py
+                              sudo pct exec 510 ──► nats bench / latency.mjs
+                                                    │
+                                                    ▼ vmbr1
+                                              10.10.10.21-23 :4222
+ +
+

3. Results (this run)

+

Host and brokers

+

Before

+ +++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NodeVMIDconnectionsin_msgsout_msgscpucoresmem (B)jetstream
nats-a511531,44331,7251815,785,984True
nats-b512124,33024,4801115,892,480True
nats-c513019,05119,0340114,262,272True
+

After

+ +++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NodeVMIDconnectionsin_msgsout_msgscpucoresmem (B)jetstream
nats-a5115724,9621,400,2882827,095,040True
nats-b5121509,195671,8130136,028,416True
nats-c5130703,5081,715,9070127,295,744True
+

nproc=40 · +uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux

+

Throughput

+ +++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunModeAggregate msgs/sPub msgs/sPub MB/sSub msgs/sSub MB/s
core-1p1s-50k-128core pub/sub835,602662,22780.84472,23957.65
core-4p4s-100k-128core pub/sub1,597,284692,10984.491,286,093156.99
core-4p4s-50k-1kcore pub/sub595,555189,939185.49479,898468.65
core-8p8s-200k-128core pub/sub1,760,599223,07827.231,570,171191.67
js-1p-20k-128-r3jetstream r=3 file—14,3301.75——
js-2p2s-20k-128-r3jetstream r=3 file18,4499,2521.139,2281.13
js-4p-20k-1k-r3jetstream r=3 file—15,19714.84——
js-4p-50k-128-r3jetstream r=3 file—19,2322.35——
js-file-1p-20k-128-r1jetstream r=3 file—18,8882.31——
js-file-1p-20k-4k-r3jetstream r=3 file—8,67333.88——
js-file-4p-50k-128-r1jetstream r=3 file—24,5603.00——
js-mem-1p-20k-128-r1jetstream r=3 file—29,9723.66——
js-mem-1p-20k-128-r3jetstream r=3 file—20,1882.46——
js-mem-4p-20k-1k-r3jetstream r=3 file—33,91633.12——
js-mem-4p-50k-128-r1jetstream r=3 file—64,9237.93——
js-mem-4p-50k-128-r3jetstream r=3 file—37,7364.61——
+

Round-trip delay

+ +++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunKindCountPubsSizeminavgp50p90p99max
lat-ping-1k-128ping (sequential RTT)10001128 B0.340ms0.530ms0.456ms0.827ms1.140ms2.910ms
lat-reconnect-200-128flood (burst queueing)2001128 B0.366ms0.540ms0.503ms0.619ms1.750ms3.206ms
lat-1p-5k-128flood (burst queueing)50001128 B98.399ms124.452ms125.413ms140.165ms145.770ms146.002ms
lat-4p-5k-1kflood (burst queueing)500041024 B125.336ms157.915ms157.392ms176.826ms178.015ms178.776ms
lat-4p-10k-128flood (burst queueing)100004128 B157.463ms205.870ms207.044ms237.035ms239.449ms240.473ms
lat-8p-20k-128flood (burst queueing)200008128 B230.898ms361.458ms372.512ms443.799ms458.198ms460.270ms
+

Core NATS

+
+ + +
+

Core NATS throughput at four loads (NS1 host run) ### +Payload size (core)

+
+ + +
+

Core NATS 128 B vs 1 KiB (NS1 host run) ### JetStream r=3 +file

+
+ + +
+

JetStream durable publish rate (NS1 host run) ### Core vs +JetStream

+
+ + +
+

Core vs JetStream publish rate, log scale (NS1 host run) ### +Delay

+
+ + +
+

Ping vs flood delay percentiles, log scale (NS1 host +run)

+
+

4. Study methodology

+

4.1 Question

+

On the NS1 test stand, what message throughput and +delay does the three-node verae JetStream +cluster deliver at several loads, and which part of the stack is the +limiter for product traffic (jobs, events, webhooks, archive)?

+

4.2 Hypotheses (stated before +the run)

+
    +
  1. H1 — Core vs JetStream. Fire-and-forget core NATS +is at least an order of magnitude faster than JetStream file + +replicas=3, because durable publish waits for a majority disk +replica.
    2. +
  2. H2 — JetStream parallelism. Adding publishers does +not linearly increase JetStream write rate once the +replica log is saturated.
    3. +
  3. H3 — Quiet delay. Sequential pub→sub round trip on +vmbr1 is well under 1 ms p99 when the consumer is +waiting.
    4. +
  4. H4 — Burst delay. If publishers dump a batch before +the subscriber drains, observed delay is queueing time, +roughly linear in backlog, not in cluster hop count.
    5. +
  5. H5 — Payload. Moving 128 B → 1 KiB lowers message +rate and raises byte rate on core NATS; JetStream in this size band +stays replica/fsync bound.
    6. +
+

4.3 Independent variables +(what we changed)

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FactorLevels
TransportCore NATS pub/sub vs JetStream file replicas=3
Publisher count1, 2, 4, 8
Subscriber count0 (JS publish-only), 1, 2, 4, 8
Message count1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step)
Payload128 B, 1024 B
Delay modeping (publish, wait, repeat) vs +flood (publish all, then drain)
+

4.4 Dependent variables +(what we recorded)

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MetricInstrumentUnit
Publish ratenats bench 0.1.6 Pub statsmsgs/s, MB/s
Subscribe ratenats bench Sub statsmsgs/s, MB/s
Aggregatenats bench NATS Pub/Sub statsmsgs/s (fan-out counts both sides)
Publisher spreadnats min/avg/max msgs/snot delay
One-way-ish RTTlatency.mjs header timestampmin, avg, p50, p90, p99, max
Host load/proc/loadavg before and afterload average
Broker countershttp://127.0.0.1:8222/varz inside each nats LXCconnections, in/out msgs, cpu, mem
+

Important: nats CLI 0.1.6 min/avg/max are +rate spread across publishers, not microseconds of +delay. Delay is only latency.mjs.

+

4.5 Controls and constants

+ +

4.6 Procedure

+
    +
  1. Confirm this script is executing on +NS1.GEORGELAMBERT.ORG. Refuse otherwise.
    2. +
  2. Snapshot host load, memory, LXC configs, and each nats +varz.
    3. +
  3. From NS1, pct exec 510 the core ladder (1p1s, 4p4s, +8p8s at 128 B; 4p4s at 1 KiB).
    4. +
  4. Delete benchstream; JS ladder (1p, 4p, 4p×1 KiB, 2p2s +pull) at replicas=3 file.
    5. +
  5. Copy latency.mjs into 510; ping then flood at several +batch sizes.
    6. +
  6. Snapshot host/varz again.
    7. +
  7. Parse logs on this host; draw charts; write HTML +and PDF on this host.
    8. +
+

No publish, subscribe, chart, or PDF process runs on the operator +laptop for this study.

+

4.7 Instrumentation path

+
[NS1 host 70.88.205.138]
+   study-on-ns1.sh  (bash + python3)
+        |
+        | sudo pct exec 510
+        v
+[LXC 510 verae-px-worker 10.10.10.20]
+   nats bench / node latency.mjs
+        |
+        | NATS client protocol to
+        v
+[LXC 511/512/513  10.10.10.21-23 :4222]
+   nats-server -js  cluster routes :6222
+

The hypervisor issues the guest commands. The messages themselves +never leave vmbr1.

+

4.8 Threats to validity

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ThreatEffect on numbers
One physical hostThree “replicas” share CPU, memory, and usually the same datastore. +This measures process/LXC HA, not disk HA.
Shared loadNS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. +Load average during a run is part of the result, not noise to +ignore.
Single bench clientAll publishers live in 510. Per-publisher rate spread is contention +in that guest.
Short runsSeconds of traffic. No compaction, no multi-hour page-cache +eviction, no snapshot during load.
No TLS/nkeysProduction auth will cost CPU. Do not treat these rates as +post-nkeys rates.
Fan-out aggregateCore aggregate msgs/s counts pub+sub. Do not compare that column to +JetStream unique writes.
Flood ≠ RTTMixing flood averages with ping p99 produces a fake “NATS is slow” +story.
Lab onlyNot a Zapier HTTPS bench and not live +api.veraetime.net.
+

4.9 Ethics / safety

+

Bench uses throwaway subjects (bench.core.*, +bench.js.*, bench.lat.*) and a throwaway +stream. It does not purge product streams. Zapier cloud has no NATS +socket.

+
+

5. Suggestions for fine-tuning

+

These follow from the method and from the first ladder on this stand +(JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of +ms). Apply in order of leverage. Re-run this NS1 study +after each change so the delta is measured the same way.

+

5.1 Treat JetStream as +the product limiter

+

Product jobs/events/webhooks/archive are durable. Tuning core NATS to +2M msgs/s will not move a timestamp Zap. Put effort into replica +write path and consumer lag, not core +fan-out.

+

5.2 Split storage class by +stream

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
StreamSuggested storeWhy
ZAPIER_JOBSfile, r=3Work queue; lose-a-job is bad
ZAPIER_EVENTSfile r=3, or memory r=3 if events are rebuildable from job +statusHot waiters; measure both
ZAPIER_WEBHOOKSfile, r=3, workqueueHTTPS to Zapier is the slow consumer
ZAPIER_USAGEfile, r=3, limits + max-ageTelemetry
VERAE_ARCHIVEfile, r=3, on the best diskPuts are larger and must survive
+

Try ZAPIER_EVENTS as memory store in a maintenance +window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms +and durable events still ack at a higher rate, keep it; if a CT restart +drops in-flight waiters, revert.

+

5.3 Give JetStream real disks

+

Today r=3 on three LXC guests on one Proxmox host is +three files, one failure domain.

+ +

5.4 Isolate the nats +CTs from the rest of NS1

+

Host load on this box is often already several. Pin:

+ +

Re-run this study after pinning; H1/H2 should move more than +ping.

+

5.5 Consumer and mailbox +tuning (delay H4)

+

Flood delay is backlog / consume_rate. Fine-tune the +waiters, not the broker RTT.

+ +

5.6 Publisher-side +batching in middleware

+

A timestamp job is one small JSON. 16k msgs/s is ample. Still:

+ +

5.7 +nats-server knobs worth measuring (A/B with this script)

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
KnobWhy try it
max_payloadKeep default unless archive puts grow
write_deadlineSlow consumer protection for webhooks
max_pendingBound memory on a stuck Zapier hook
max_connectionsFleet workers + keep + middleware
JetStream max_file_store / +max_memory_storePrevent one stream from filling the CT
max_outstanding_catchupReplica restart after a nats-c blip
GOMAXPROCS = LXC coresDo not overthread a 2-core CT
+

Change one knob, re-run +study-on-ns1.sh, compare JetStream 1p 128 B and ping +p99.

+

5.8 Network

+ +

5.9 Security cost (when +nkeys/mTLS flip)

+

verae-nats-accounts is still a sketch. Enabling accounts +will add CPU on publish. Budget: re-run this exact study +after creds are in every NATS_URL, and +accept a drop on both core and JS. Do not flip without that +measurement.

+

5.10 Operational +fine-tuning (lag, not peak msgs/s)

+
    +
  1. Scrape varz / jsz from the host over +vmbr1 (not public). Monitor loopback :8222 is +invisible to Prometheus on NS1 unless we add a host-side proxy on +10.10.10.21:8222 bound only to vmbr1.
    2. +
  2. Keep replica floors for webhook-deliver and job-poller — they are +the flood defense.
    3. +
  3. Backup/restore drill of JetStream during idle, then +a short JS 1p run to see catchup cost.
    4. +
  4. A 15–30 minute soak (not in this ladder) for page cache and +compaction; add that as a third study when disks are dedicated.
    5. +
+

5.11 What not to tune

+ + +
    +
  1. CPU pin nats-a/b/c → re-run JS 1p + ping.
    2. +
  2. ZAPIER_EVENTS-shaped memory stream vs file (throwaway +stream, same flags as this JS ladder).
    3. +
  3. Distinct store_dir disks per node.
    4. +
  4. nkeys on, same ladder.
    5. +
  5. Three hardware boxes, same cluster.env IPs +updated.
    6. +
+

Each experiment should produce a new +results/<utc>/ on NS1 and a new progress-repo report +so we can diff H1–H5 instead of arguing from memory.

+
+

6. Reproducing this study

+

On NS1 only:

+
cd ~/verae-src/verae-nats-cluster
+bash scripts/study-on-ns1.sh
+

The script exits if hostname is not NS1. Outputs land in +results/<utc>/ including +nats-cluster-bench-ns1.{md,html,pdf} and +charts/. Copy those into +zapier-decisions/reports/ for the progress repo and +catalog.

+

Raw logs for this run: results/20260912T055851Z/.

+ + diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/nats-cluster-bench-ns1.md b/packages/verae-nats-cluster/results/20260912T055851Z/nats-cluster-bench-ns1.md new file mode 100644 index 0000000..ce715f0 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/nats-cluster-bench-ns1.md @@ -0,0 +1,361 @@ +**Progress report (maximized NS1 study)** · run `20260912T055851Z` (UTC) + +> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): `maximize-ns1-study.sh` (cores/RAM/`max_mem`/tmpfs), then `study-on-ns1.sh`, `nats bench`, `latency.mjs` (LXC 510), matplotlib, pandoc, weasyprint. Traffic stayed on `vmbr1`. veth/10G was **not** changed. After the ladder, JetStream was put back on ZFS and product streams were re-created; **8 cores / 16 GiB / max_mem 8G stay**. + +## Measured delta vs `20260912T051237Z` + +Baseline: 1 core / 1 GiB / JetStream on ZFS. This run: 8 cores / 16 GiB / JetStream **tmpfs** (file r=3) plus extra **memory** store rows. veth/10G unchanged. + +| Metric | Baseline `20260912T051237Z` | This run | Ratio | +|--------|-------------------------|----------|-------| +| Core 1p1s 128 B pub msgs/s | 502,502 | 662,227 | 1.32× | +| Core 8p8s 128 B aggregate msgs/s | 2,065,217 | 1,760,599 | 0.85× | +| JS file r=3 1p 128 B pub msgs/s | 7,393 | 14,330 | 1.94× | +| JS file r=3 4p 128 B pub msgs/s | 17,986 | 19,232 | 1.07× | +| JS file r=3 4p 1 KiB pub msgs/s | 14,985 | 15,197 | 1.01× | +| JS memory r=3 1p 128 B pub msgs/s | — | 20,188 | — | +| JS memory r=3 4p 128 B pub msgs/s | — | 37,736 | — | +| Ping p99 (ms) | 1.377ms | 1.140ms | 1.21× faster | + +![Baseline vs maximized publish rates (log)](charts/delta-vs-baseline.png) + +*Baseline vs maximized publish rates (log)* + + +--- + +## 1. Executive summary + +| Item | This NS1-host run | +|------|-------------------| +| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `marchon` | +| Bench client | LXC 510 `verae-px-worker` | +| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` | +| Client URL | `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` | +| Host load before | `8.70 8.15 8.00 4/3842 621042` | +| Host load after | `8.48 8.52 8.17 5/3860 657839` | +| Core 1p1s 128 B pub | 662,227 msgs/s | +| JetStream 1p 128 B r=3 | 14,330 durable pubs/s | +| Ping p50 / p99 | 0.456ms / 1.140ms | + +Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst. + +--- + +## 2. Where it ran (and where it did not) + +```text +Operator laptop ──ssh──► NS1.GEORGELAMBERT.ORG 70.88.205.138 + study-on-ns1.sh + python3 build-ns1-study-report.py + sudo pct exec 510 ──► nats bench / latency.mjs + │ + ▼ vmbr1 + 10.10.10.21-23 :4222 +``` + +- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510). +- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file. + +--- + +## 3. Results (this run) + +### Host and brokers + +**Before** + +| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream | +|------|------|-------------|---------|----------|-----|-------|---------|-----------| +| nats-a | 511 | 5 | 31,443 | 31,725 | 1 | 8 | 15,785,984 | True | +| nats-b | 512 | 1 | 24,330 | 24,480 | 1 | 1 | 15,892,480 | True | +| nats-c | 513 | 0 | 19,051 | 19,034 | 0 | 1 | 14,262,272 | True | + +**After** + +| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream | +|------|------|-------------|---------|----------|-----|-------|---------|-----------| +| nats-a | 511 | 5 | 724,962 | 1,400,288 | 2 | 8 | 27,095,040 | True | +| nats-b | 512 | 1 | 509,195 | 671,813 | 0 | 1 | 36,028,416 | True | +| nats-c | 513 | 0 | 703,508 | 1,715,907 | 0 | 1 | 27,295,744 | True | + +nproc=40 · uname=`Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux` + +### Throughput + +| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s | +|-----|------|------------------|------------|----------|------------|----------| +| `core-1p1s-50k-128` | core pub/sub | 835,602 | 662,227 | 80.84 | 472,239 | 57.65 | +| `core-4p4s-100k-128` | core pub/sub | 1,597,284 | 692,109 | 84.49 | 1,286,093 | 156.99 | +| `core-4p4s-50k-1k` | core pub/sub | 595,555 | 189,939 | 185.49 | 479,898 | 468.65 | +| `core-8p8s-200k-128` | core pub/sub | 1,760,599 | 223,078 | 27.23 | 1,570,171 | 191.67 | +| `js-1p-20k-128-r3` | jetstream r=3 file | — | 14,330 | 1.75 | — | — | +| `js-2p2s-20k-128-r3` | jetstream r=3 file | 18,449 | 9,252 | 1.13 | 9,228 | 1.13 | +| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 15,197 | 14.84 | — | — | +| `js-4p-50k-128-r3` | jetstream r=3 file | — | 19,232 | 2.35 | — | — | +| `js-file-1p-20k-128-r1` | jetstream r=3 file | — | 18,888 | 2.31 | — | — | +| `js-file-1p-20k-4k-r3` | jetstream r=3 file | — | 8,673 | 33.88 | — | — | +| `js-file-4p-50k-128-r1` | jetstream r=3 file | — | 24,560 | 3.00 | — | — | +| `js-mem-1p-20k-128-r1` | jetstream r=3 file | — | 29,972 | 3.66 | — | — | +| `js-mem-1p-20k-128-r3` | jetstream r=3 file | — | 20,188 | 2.46 | — | — | +| `js-mem-4p-20k-1k-r3` | jetstream r=3 file | — | 33,916 | 33.12 | — | — | +| `js-mem-4p-50k-128-r1` | jetstream r=3 file | — | 64,923 | 7.93 | — | — | +| `js-mem-4p-50k-128-r3` | jetstream r=3 file | — | 37,736 | 4.61 | — | — | + +### Round-trip delay + +| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max | +|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----| +| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.340ms | 0.530ms | 0.456ms | 0.827ms | 1.140ms | 2.910ms | +| `lat-reconnect-200-128` | flood (burst queueing) | 200 | 1 | 128 B | 0.366ms | 0.540ms | 0.503ms | 0.619ms | 1.750ms | 3.206ms | +| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 98.399ms | 124.452ms | 125.413ms | 140.165ms | 145.770ms | 146.002ms | +| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 125.336ms | 157.915ms | 157.392ms | 176.826ms | 178.015ms | 178.776ms | +| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 157.463ms | 205.870ms | 207.044ms | 237.035ms | 239.449ms | 240.473ms | +| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 230.898ms | 361.458ms | 372.512ms | 443.799ms | 458.198ms | 460.270ms | + +### Core NATS + +![Core NATS throughput at four loads (NS1 host run)](charts/core-throughput.png) + +*Core NATS throughput at four loads (NS1 host run)* +### Payload size (core) + +![Core NATS 128 B vs 1 KiB (NS1 host run)](charts/payload-size.png) + +*Core NATS 128 B vs 1 KiB (NS1 host run)* +### JetStream r=3 file + +![JetStream durable publish rate (NS1 host run)](charts/js-throughput.png) + +*JetStream durable publish rate (NS1 host run)* +### Core vs JetStream + +![Core vs JetStream publish rate, log scale (NS1 host run)](charts/core-vs-js.png) + +*Core vs JetStream publish rate, log scale (NS1 host run)* +### Delay + +![Ping vs flood delay percentiles, log scale (NS1 host run)](charts/delay-percentiles.png) + +*Ping vs flood delay percentiles, log scale (NS1 host run)* + +--- + +## 4. Study methodology + +### 4.1 Question + +On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)? + +### 4.2 Hypotheses (stated before the run) + +1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica. +2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated. +3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting. +4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count. +5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound. + +### 4.3 Independent variables (what we changed) + +| Factor | Levels | +|--------|--------| +| Transport | Core NATS pub/sub vs JetStream file replicas=3 | +| Publisher count | 1, 2, 4, 8 | +| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 | +| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) | +| Payload | 128 B, 1024 B | +| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) | + +### 4.4 Dependent variables (what we recorded) + +| Metric | Instrument | Unit | +|--------|------------|------| +| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s | +| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s | +| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) | +| Publisher spread | nats min/avg/max **msgs/s** | not delay | +| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max | +| Host load | `/proc/loadavg` before and after | load average | +| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem | + +**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`. + +### 4.5 Controls and constants + +- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`. +- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`). +- Bench client is LXC **510**, not a nats-* server. +- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide. +- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`). +- No TLS, no nkeys, no account isolation (isolation is `vmbr1`). +- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder. + +### 4.6 Procedure + +1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise. +2. Snapshot host load, memory, LXC configs, and each nats `varz`. +3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB). +4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file. +5. Copy `latency.mjs` into 510; ping then flood at several batch sizes. +6. Snapshot host/`varz` again. +7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**. + +No publish, subscribe, chart, or PDF process runs on the operator laptop for this study. + +### 4.7 Instrumentation path + +```text +[NS1 host 70.88.205.138] + study-on-ns1.sh (bash + python3) + | + | sudo pct exec 510 + v +[LXC 510 verae-px-worker 10.10.10.20] + nats bench / node latency.mjs + | + | NATS client protocol to + v +[LXC 511/512/513 10.10.10.21-23 :4222] + nats-server -js cluster routes :6222 +``` + +The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`. + +### 4.8 Threats to validity + +| Threat | Effect on numbers | +|--------|-------------------| +| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. | +| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. | +| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. | +| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. | +| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. | +| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. | +| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. | +| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. | + +### 4.9 Ethics / safety + +Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket. + +--- + +## 5. Suggestions for fine-tuning + +These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way. + +### 5.1 Treat JetStream as the product limiter + +Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out. + +### 5.2 Split storage class by stream + +| Stream | Suggested store | Why | +|--------|-----------------|-----| +| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad | +| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both | +| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer | +| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry | +| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive | + +Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert. + +### 5.3 Give JetStream real disks + +Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain. + +- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`. +- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure. +- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it. +- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.” + +### 5.4 Isolate the nats CTs from the rest of NS1 + +Host load on this box is often already several. Pin: + +- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes. +- Memory high enough that file-backed streams stay cache-hot for the working set. +- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync. + +Re-run this study after pinning; H1/H2 should move more than ping. + +### 5.5 Consumer and mailbox tuning (delay H4) + +Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT. + +- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM. +- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber. +- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped. +- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT. + +### 5.6 Publisher-side batching in middleware + +A timestamp job is one small JSON. 16k msgs/s is ample. Still: + +- Avoid per-byte publishes; one message per job/event. +- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs). +- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only. + +### 5.7 nats-server knobs worth measuring (A/B with this script) + +| Knob | Why try it | +|------|------------| +| `max_payload` | Keep default unless archive puts grow | +| `write_deadline` | Slow consumer protection for webhooks | +| `max_pending` | Bound memory on a stuck Zapier hook | +| `max_connections` | Fleet workers + keep + middleware | +| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT | +| `max_outstanding_catchup` | Replica restart after a nats-c blip | +| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT | + +Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99. + +### 5.8 Network + +- Keep NATS off `vmbr0`. No change. +- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load). +- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus. + +### 5.9 Security cost (when nkeys/mTLS flip) + +`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement. + +### 5.10 Operational fine-tuning (lag, not peak msgs/s) + +1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`. +2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense. +3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost. +4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated. + +### 5.11 What not to tune + +- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge. +- Do not treat flood 400 ms as “cluster RTT.” Fix consumers. +- Do not load-test on `ZAPIER_*` streams. +- Do not bind client NATS to `0.0.0.0` on `vmbr0`. + +### 5.12 Recommended next experiments (same method, one change each) + +1. CPU pin nats-a/b/c → re-run JS 1p + ping. +2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder). +3. Distinct `store_dir` disks per node. +4. nkeys on, same ladder. +5. Three hardware boxes, same `cluster.env` IPs updated. + +Each experiment should produce a new `results//` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory. + + +--- + +## 6. Reproducing this study + +On **NS1 only**: + +```bash +cd ~/verae-src/verae-nats-cluster +bash scripts/study-on-ns1.sh +``` + +The script exits if `hostname` is not NS1. Outputs land in `results//` including `nats-cluster-bench-ns1.{md,html,pdf}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog. + +Raw logs for this run: `results/20260912T055851Z/`. diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/nats-cluster-bench-ns1.pdf b/packages/verae-nats-cluster/results/20260912T055851Z/nats-cluster-bench-ns1.pdf new file mode 100644 index 0000000..4fd4c66 Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T055851Z/nats-cluster-bench-ns1.pdf differ diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/optimal-config.html b/packages/verae-nats-cluster/results/20260912T055851Z/optimal-config.html new file mode 100644 index 0000000..e9d66d1 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/optimal-config.html @@ -0,0 +1,853 @@ + + + + + + + NATS optimal configuration study + + + + +
Verae Time × Zapier · progress report

NATS optimal configuration study

packages/zapier-decisions/reports/optimal-config/REPORT.md
+
+

NATS optimal configuration study

+
+

Progress report — optimal configuration study · +20260912T055851Z (UTC) · all code on +NS1.GEORGELAMBERT.ORG (70.88.205.138)

+

This document folds every ladder we have run (1-core ZFS, +NS1-orchestrated, tmpfs maximize, and this exhaustive 8c/16G +ZFS factorial) plus UDP / MQTT / reconnect probes. It +recommends a lab config and a three-box HP DL360 Gen10 +projection. veth/10G was not changed.

+
+

1. Verdict (read this first)

+

Keep NATS + JetStream. Do not replace the fabric +with MQTT, UDP, or a custom persistent-socket protocol for Verae +jobs/events/archive. Those are either slower, less durable, or already +what NATS is.

+

Lab (NS1, one host, three LXC) — optimal now

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
StreamStorageReplicasWhy
ZAPIER_JOBS, ZAPIER_WEBHOOKS, +VERAE_ARCHIVEfile (ZFS)3Survive a nats LXC death; archive must persist
ZAPIER_EVENTSmemory3Waiters are latency-sensitive; events rebuild from job status
ZAPIER_USAGEfile3Telemetry, limits + max-age
+

Keep 8 cores / 16 GiB / max_mem: 8G on +510–513 (already live). Do not leave JetStream on +tmpfs. Do not drop product streams to r=1. Reuse +one NATS connection per process (already true in +middleware); never connect-per-message.

+

Metal (3× DL360 Gen10) — optimal later

+

Same stream table. File store on local NVMe/M.2, not +a shared SAN. Cluster + client on 10GbE (or 25GbE if +you already have it). Dual Gold Xeon is surplus CPU for this workload; +8–16 cores dedicated to nats-server is enough. Expected JS +file r=3: ~40–80k 128 B pubs/s (about +3–6× this lab’s 8c ZFS 1p, 2–4× tmpfs +1p) — bounded by 10GbE replica RTT, not by Xeon clocks. +Core NATS will sit in the 1–3M msgs/s band until the +NIC saturates (~9 Gbit/s ≈ 8–9M × 128 B theoretical; CPU and client will +hit first).

+
+

2. What we actually +ran (this exhaustive pass)

+

Live cluster during this run: LXC 510–513 8 cores / 16 +GiB, JetStream on ZFS (tmpfs from the maximize +study was already unmounted). Extra factorial: file/memory × replicas +1/3, 4 KiB file r=3, reconnect-per-message ping, UDP echo 510→511, MQTT +QoS0 against nats-a :1883. Product streams were not the +bench target.

+

2.1 Cross-study history

+ ++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
StudyEnvCore 1p pubJS file r=3 1pJS mem r=3 4pPing p99
20260912T051237Z1c/1G ZFS (NS1 orch.)502,5027,393—1.377ms
20260912T053120Z8c/16G tmpfs + mem extra599,00417,38836,3550.684ms
20260912T055851Z8c/16G ZFS exhaustive 20260912T055851Z662,22714,33037,7361.140ms
+
+ + +
+

2.2 This run — JetStream +factorial

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunWhatPub msgs/sPub MB/s
js-file-1p-20k-128-r1file r=1 1p 128 B18,8882.31
js-file-4p-50k-128-r1file r=1 4p 128 B24,5603.00
js-1p-20k-128-r3file r=3 1p 128 B14,3301.75
js-4p-50k-128-r3file r=3 4p 128 B19,2322.35
js-4p-20k-1k-r3file r=3 4p 1 KiB15,19714.84
js-file-1p-20k-4k-r3file r=3 1p 4 KiB8,67333.88
js-mem-1p-20k-128-r1memory r=1 1p 128 B29,9723.66
js-mem-4p-50k-128-r1memory r=1 4p 128 B64,9237.93
js-mem-1p-20k-128-r3memory r=3 1p 128 B20,1882.46
js-mem-4p-50k-128-r3memory r=3 4p 128 B37,7364.61
js-mem-4p-20k-1k-r3memory r=3 4p 1 KiB33,91633.12
+

Replica 1 vs 3 on this stand (file 1p 128 B): r=1 is +18,888 vs r=3 14,330 (1.32× if r=3 is the slower one). Memory r=1 1p +29,972 vs memory r=3 20,188.

+
+Replica cost + +
+

2.3 Delay, reconnect tax, UDP, +MQTT

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ProbeResultMeaning
NATS ping (persistent sockets) p50 / p990.456ms / 1.140msQuiet hop with a long-lived TCP conn
NATS reconnect-per-message p50 / p990.503ms / 1.750msTCP+NATS handshake on every pub — this is the tax to avoid
UDP echo 510→511 p990.363msRaw datagram ceiling on the same veth (no NATS)
MQTT QoS0 5k×128 B44862 pubs/snats-server MQTT gateway on :1883
+

Core 1p1s 128 B this run: 662,227 pub msgs/s. Flood delay is still +backlog/consume_rate, not RTT.

+
+

3. +Alternative transports (why we are not switching the fabric)

+

NATS already is persistent TCP sockets with a tiny +binary protocol, automatic reconnect, and optional JetStream durability. +“Reduce connection overhead” is a client discipline: +hold the connection. The reconnect probe exists to prove that opening a +socket per job would dominate ping RTT.

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
IdeaFit for Verae jobs/events/archiveThroughput vs NATS coreDurability
NATS core pub/subFan-out, request-reply (verae.billing.*)Highest we measured (~0.5–2M msgs/s)None
NATS JetStream file r=3Jobs, webhooks, archive~8–23k on this lab; see metal projectionDisk + 1-node loss
NATS JetStream memory r=3Events mailbox~22–36k on this labRAM + 1-node loss; empty on full restart
MQTT (NATS gateway or Mosquitto)IoT endpoints that already speak MQTTThis probe: 44862 pubs/s QoS0 — typically well +below NATS core; QoS1 ≈ JetStream-ish with more chatterQoS1/2 session state; not our WORM model
UDPTelemetry that may dropRTT 0.363ms p99 — fastest hop, no reliability, no +cluster, no authNone
Custom persistent sockets / HTTP long-pollWorse NATSYou would re-implement reconnect, flow control, and fan-outDIY
WebSocketBrowsers onlyExtra framing; NATS already has WS for UIs, not for middlewareSame as core/JS behind it
QUIC / WebTransportLossy WAN / browsersNATS QUIC is not the lab path; 10GbE LAN does not need itSame
Kafka / Redis streamsHeavy log replayHigher ops cost; not on vmbr1 todayYes, heavier
+

MQTT: NATS documents MQTT as an enabling +gateway for existing IoT, and prefers NATS end-to-end for greenfield. +Zapier cloud never talks NATS or MQTT; it talks HTTPS. Putting MQTT in +the middle of timestamp jobs adds protocol translation and QoS timers +without helping jobId → events. Use MQTT only if a device +already cannot speak NATS.

+

UDP: Fine as a measurement of veth RTT. +Unusable as the job fabric (no ack, no replica, no flow control). NATS +ping is already within a small multiple of UDP on this bridge.

+

Persistence sockets: Middleware and keep already +keep NATS_URL connections open. Optimal: one connection (or +a small pool) per process, max_reconnect, jitter, no +connect() in the per-job path. The reconnect ladder is the +anti-pattern.

+
+

4. Optimal configurations

+

4.1 NS1 lab (now)

+
    +
  1. Leave 8 cores / 16 GiB on nats-a/b/c and the +worker. Host has 40 cores / 377 GiB; this is cheap.
    2. +
  2. max_mem: 8G stays. Required for memory +streams.
    3. +
  3. File r=3 on ZFS for jobs/webhooks/archive. tmpfs +doubled JS 1p (7.4k→17k) but loses the stream on reboot +— unacceptable for archive.
    4. +
  4. Memory r=3 for ZAPIER_EVENTS if we +accept “all three nats CTs reboot ⇒ in-flight waiters fall back to HTTP +poll.” That matches the designed wait path +(GET /api/status/{jobId}).
    5. +
  5. r=1 only for throwaway benches, never product +streams. Replica=3 is the point of three guests.
    6. +
  6. veth on vmbr1, no fake 10G NICs. Already 10000Mb/s; +JS does not fill it.
    7. +
  7. Pin cpusets later if keep/fleet steal; not required +to beat these numbers.
    8. +
  8. Clients: persistent NATS connections; pull consumers with bounded +max_ack_pending for webhooks.
    9. +
+

4.2 Three HP DL360 +Gen10 (projection — not measured)

+

Assumed bill of materials (state it in the buy):

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
PieceAssumption
Chassis3× DL360 Gen10 1U
CPUDual 2nd-gen Xeon Gold (e.g. 6226R 16c or 6248 20c +— 32–40 cores/box)
MemoryDDR4-2933, 192–384 GiB/box (6–12×32 GiB); NATS will +not use most of it
StorageNVMe M.2 or U.2 for +/var/lib/nats/jetstream (XFS or ext4, not +shared ZFS over the network). RAID1 of two NVMe if you want disk HA +inside a box
Network10GbE (FlexibleLOM or PCIe); dedicated VLAN for +:4222+:6222. Do not share with public +vmbr0 traffic
OSDebian/Ubuntu bare metal, nats-server systemd, same +nats.conf as lab (bind private IP only)
+

What changes vs NS1 LXC

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FactorNS1 today3× DL360Effect on JS file r=3
Failure domain1 Proxmox host3 chassis, 3 NVMe, 3 NICsr=3 means something
DiskShared ZFS SSD2Local NVMe fsync ~50–150 µsBig win vs ZFS; similar to tmpfs for sequential 128 B
Replica pathveth/bridge (~µs–tens of µs)10GbE RTT typically 50–200 µsSlower than same-host tmpfs, faster than a bad +SAN
CPU8 of 40 shared32–40 dedicated Gold coresHeadroom for many clients, not 10× JS
NICsoftware 10G veth, already ~5 Gbit/s corereal 10GbE ~9 Gbit/s TCPCore NATS can grow; JS r=3 stays replica-bound
+

Projected bands (128 B, 3-node cluster, dedicated +10GbE, local NVMe, 8+ cores pinned to nats-server):

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
WorkloadNS1 measured (best)DL360 projectionConfidence
Core pub/sub 1p0.5–0.8M0.8–2MMedium — NIC + syscall, plenty of CPU
Core 4p4s 1 KiB~0.6–0.7M (~0.6 GB/s)~1M msgs/s / ~1 GB/s approaching 10GbEMedium
JS file r=1this run r=180–200k pubs/sMedium — NVMe + no replica wait
JS file r=37–23k (ZFS/tmpfs)40–80k pubs/sMedium-low — replica RTT dominates; 3 NVMe still help vs shared +ZFS
JS memory r=322–36k50–100kMedium-low — RAM + 10GbE ack
Ping p990.7–1.4 ms0.2–0.6 msMedium — real NIC but no Proxmox tax
+

These are not DL360 measurements. Scale from: (a) +our replica-1 vs replica-3 ratio once this run’s r=1 numbers exist, (b) +tmpfs vs ZFS ratio (2.35× on 1p), (c) Synadia/nats bench async file r=1 +~100–400k on NVMe loopback, derated for 10GbE RTT.

+

Buy notes: M.2 via Dual uFF / enablement kit; put +JetStream on NVMe directly, not behind a RAID +controller write-through unless you measure. 1GbE onboard is a trap — +use 10GbE for :6222. Dual Gold is for isolation (nats vs +worm/tree vs OS), not because JS needs 56 cores.

+
+

5. What we are not doing

+
    +
  • MQTT as the Zapier or middleware transport.
    • +
  • UDP for jobs.
    • +
  • Emulated 10G fiber NICs on LXC.
    • +
  • tmpfs as the production store.
    • +
  • r=1 for product streams.
    • +
  • Connect-per-job.
    • +
+

Re-run exhaustive: bash scripts/exhaustive-ns1-study.sh +on NS1.

+ + diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/optimal-config.md b/packages/verae-nats-cluster/results/20260912T055851Z/optimal-config.md new file mode 100644 index 0000000..8c5eb58 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/optimal-config.md @@ -0,0 +1,160 @@ +**Progress report — optimal configuration study** · `20260912T055851Z` (UTC) · all code on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`) + +This document folds every ladder we have run (1-core ZFS, NS1-orchestrated, tmpfs maximize, and this exhaustive 8c/16G **ZFS** factorial) plus UDP / MQTT / reconnect probes. It recommends a lab config and a **three-box HP DL360 Gen10** projection. veth/10G was not changed. + +--- + +## 1. Verdict (read this first) + +**Keep NATS + JetStream.** Do not replace the fabric with MQTT, UDP, or a custom persistent-socket protocol for Verae jobs/events/archive. Those are either slower, less durable, or already what NATS is. + +**Lab (NS1, one host, three LXC) — optimal now** + +| Stream | Storage | Replicas | Why | +|--------|---------|----------|-----| +| `ZAPIER_JOBS`, `ZAPIER_WEBHOOKS`, `VERAE_ARCHIVE` | **file** (ZFS) | **3** | Survive a nats LXC death; archive must persist | +| `ZAPIER_EVENTS` | **memory** | **3** | Waiters are latency-sensitive; events rebuild from job status | +| `ZAPIER_USAGE` | file | 3 | Telemetry, limits + max-age | + +Keep **8 cores / 16 GiB / `max_mem: 8G`** on 510–513 (already live). Do **not** leave JetStream on tmpfs. Do **not** drop product streams to r=1. Reuse **one NATS connection per process** (already true in middleware); never connect-per-message. + +**Metal (3× DL360 Gen10) — optimal later** + +Same stream table. File store on **local NVMe/M.2**, not a shared SAN. Cluster + client on **10GbE** (or 25GbE if you already have it). Dual Gold Xeon is surplus CPU for this workload; 8–16 cores dedicated to `nats-server` is enough. Expected JS file r=3: **~40–80k** 128 B pubs/s (about **3–6×** this lab’s 8c ZFS 1p, **2–4×** tmpfs 1p) — bounded by **10GbE replica RTT**, not by Xeon clocks. Core NATS will sit in the **1–3M msgs/s** band until the NIC saturates (~9 Gbit/s ≈ 8–9M × 128 B theoretical; CPU and client will hit first). + +--- + +## 2. What we actually ran (this exhaustive pass) + +Live cluster during this run: LXC 510–513 **8 cores / 16 GiB**, JetStream **on ZFS** (tmpfs from the maximize study was already unmounted). Extra factorial: file/memory × replicas 1/3, 4 KiB file r=3, reconnect-per-message ping, UDP echo 510→511, MQTT QoS0 against nats-a `:1883`. Product streams were not the bench target. + +### 2.1 Cross-study history + +| Study | Env | Core 1p pub | JS file r=3 1p | JS mem r=3 4p | Ping p99 | +| --- | --- | --- | --- | --- | --- | +| `20260912T051237Z` | 1c/1G ZFS (NS1 orch.) | 502,502 | 7,393 | — | 1.377ms | +| `20260912T053120Z` | 8c/16G tmpfs + mem extra | 599,004 | 17,388 | 36,355 | 0.684ms | +| `20260912T055851Z` | 8c/16G ZFS exhaustive `20260912T055851Z` | 662,227 | 14,330 | 37,736 | 1.140ms | + +![JS 1p file r=3 history](charts-optimal/history-js1p.png) + +### 2.2 This run — JetStream factorial + +| Run | What | Pub msgs/s | Pub MB/s | +| --- | --- | --- | --- | +| `js-file-1p-20k-128-r1` | file r=1 1p 128 B | 18,888 | 2.31 | +| `js-file-4p-50k-128-r1` | file r=1 4p 128 B | 24,560 | 3.00 | +| `js-1p-20k-128-r3` | file r=3 1p 128 B | 14,330 | 1.75 | +| `js-4p-50k-128-r3` | file r=3 4p 128 B | 19,232 | 2.35 | +| `js-4p-20k-1k-r3` | file r=3 4p 1 KiB | 15,197 | 14.84 | +| `js-file-1p-20k-4k-r3` | file r=3 1p 4 KiB | 8,673 | 33.88 | +| `js-mem-1p-20k-128-r1` | memory r=1 1p 128 B | 29,972 | 3.66 | +| `js-mem-4p-50k-128-r1` | memory r=1 4p 128 B | 64,923 | 7.93 | +| `js-mem-1p-20k-128-r3` | memory r=3 1p 128 B | 20,188 | 2.46 | +| `js-mem-4p-50k-128-r3` | memory r=3 4p 128 B | 37,736 | 4.61 | +| `js-mem-4p-20k-1k-r3` | memory r=3 4p 1 KiB | 33,916 | 33.12 | + +Replica **1 vs 3** on this stand (file 1p 128 B): r=1 is 18,888 vs r=3 14,330 (1.32× if r=3 is the slower one). Memory r=1 1p 29,972 vs memory r=3 20,188. + +![Replica cost](charts-optimal/replicas.png) + +### 2.3 Delay, reconnect tax, UDP, MQTT + +| Probe | Result | Meaning | +|-------|--------|---------| +| NATS ping (persistent sockets) p50 / p99 | 0.456ms / 1.140ms | Quiet hop with a long-lived TCP conn | +| NATS **reconnect-per-message** p50 / p99 | 0.503ms / 1.750ms | TCP+NATS handshake on every pub — this is the tax to avoid | +| UDP echo 510→511 p99 | 0.363ms | Raw datagram ceiling on the same veth (no NATS) | +| MQTT QoS0 5k×128 B | 44862 pubs/s | nats-server MQTT gateway on `:1883` | + +Core 1p1s 128 B this run: 662,227 pub msgs/s. Flood delay is still backlog/consume_rate, not RTT. + +--- + +## 3. Alternative transports (why we are not switching the fabric) + +NATS already **is** persistent TCP sockets with a tiny binary protocol, automatic reconnect, and optional JetStream durability. “Reduce connection overhead” is a **client** discipline: hold the connection. The reconnect probe exists to prove that opening a socket per job would dominate ping RTT. + +| Idea | Fit for Verae jobs/events/archive | Throughput vs NATS core | Durability | +|------|-----------------------------------|-------------------------|------------| +| **NATS core pub/sub** | Fan-out, request-reply (`verae.billing.*`) | Highest we measured (~0.5–2M msgs/s) | None | +| **NATS JetStream file r=3** | Jobs, webhooks, archive | ~8–23k on this lab; see metal projection | Disk + 1-node loss | +| **NATS JetStream memory r=3** | Events mailbox | ~22–36k on this lab | RAM + 1-node loss; **empty on full restart** | +| **MQTT** (NATS gateway or Mosquitto) | IoT endpoints that already speak MQTT | This probe: 44862 pubs/s QoS0 — typically **well below** NATS core; QoS1 ≈ JetStream-ish with more chatter | QoS1/2 session state; not our WORM model | +| **UDP** | Telemetry that may drop | RTT 0.363ms p99 — fastest hop, **no** reliability, no cluster, no auth | None | +| **Custom persistent sockets / HTTP long-poll** | Worse NATS | You would re-implement reconnect, flow control, and fan-out | DIY | +| **WebSocket** | Browsers only | Extra framing; NATS already has WS for UIs, not for middleware | Same as core/JS behind it | +| **QUIC / WebTransport** | Lossy WAN / browsers | NATS QUIC is not the lab path; 10GbE LAN does not need it | Same | +| **Kafka / Redis streams** | Heavy log replay | Higher ops cost; not on `vmbr1` today | Yes, heavier | + +**MQTT:** NATS documents MQTT as an *enabling* gateway for existing IoT, and prefers NATS end-to-end for greenfield. Zapier cloud never talks NATS or MQTT; it talks HTTPS. Putting MQTT in the middle of timestamp jobs adds protocol translation and QoS timers without helping `jobId → events`. Use MQTT only if a device already cannot speak NATS. + +**UDP:** Fine as a *measurement* of veth RTT. Unusable as the job fabric (no ack, no replica, no flow control). NATS ping is already within a small multiple of UDP on this bridge. + +**Persistence sockets:** Middleware and keep already keep `NATS_URL` connections open. Optimal: one connection (or a small pool) per process, `max_reconnect`, jitter, no `connect()` in the per-job path. The reconnect ladder is the anti-pattern. + +--- + +## 4. Optimal configurations + +### 4.1 NS1 lab (now) + +1. **Leave 8 cores / 16 GiB** on nats-a/b/c and the worker. Host has 40 cores / 377 GiB; this is cheap. +2. **`max_mem: 8G`** stays. Required for memory streams. +3. **File r=3 on ZFS** for jobs/webhooks/archive. tmpfs doubled JS 1p (7.4k→17k) but **loses the stream on reboot** — unacceptable for archive. +4. **Memory r=3 for `ZAPIER_EVENTS`** if we accept “all three nats CTs reboot ⇒ in-flight waiters fall back to HTTP poll.” That matches the designed wait path (`GET /api/status/{jobId}`). +5. **r=1 only for throwaway benches**, never product streams. Replica=3 is the point of three guests. +6. **veth on vmbr1, no fake 10G NICs.** Already 10000Mb/s; JS does not fill it. +7. **Pin cpusets** later if keep/fleet steal; not required to beat these numbers. +8. Clients: persistent NATS connections; pull consumers with bounded `max_ack_pending` for webhooks. + +### 4.2 Three HP DL360 Gen10 (projection — not measured) + +Assumed bill of materials (state it in the buy): + +| Piece | Assumption | +|-------|------------| +| Chassis | 3× DL360 Gen10 1U | +| CPU | Dual 2nd-gen Xeon **Gold** (e.g. 6226R 16c or 6248 20c — **32–40 cores/box**) | +| Memory | DDR4-2933, **192–384 GiB**/box (6–12×32 GiB); NATS will not use most of it | +| Storage | **NVMe M.2 or U.2** for `/var/lib/nats/jetstream` (XFS or ext4, **not** shared ZFS over the network). RAID1 of two NVMe if you want disk HA *inside* a box | +| Network | **10GbE** (FlexibleLOM or PCIe); dedicated VLAN for `:4222`+`:6222`. Do not share with public `vmbr0` traffic | +| OS | Debian/Ubuntu bare metal, `nats-server` systemd, same `nats.conf` as lab (bind private IP only) | + +**What changes vs NS1 LXC** + +| Factor | NS1 today | 3× DL360 | Effect on JS file r=3 | +|--------|-----------|----------|------------------------| +| Failure domain | 1 Proxmox host | 3 chassis, 3 NVMe, 3 NICs | r=3 **means** something | +| Disk | Shared ZFS SSD2 | Local NVMe fsync ~50–150 µs | Big win vs ZFS; similar to tmpfs for sequential 128 B | +| Replica path | veth/bridge (~µs–tens of µs) | 10GbE RTT typically **50–200 µs** | **Slower than same-host tmpfs**, faster than a bad SAN | +| CPU | 8 of 40 shared | 32–40 dedicated Gold cores | Headroom for many clients, not 10× JS | +| NIC | software 10G veth, already ~5 Gbit/s core | real 10GbE ~9 Gbit/s TCP | Core NATS can grow; JS r=3 stays replica-bound | + +**Projected bands** (128 B, 3-node cluster, dedicated 10GbE, local NVMe, 8+ cores pinned to nats-server): + +| Workload | NS1 measured (best) | DL360 projection | Confidence | +|----------|---------------------|------------------|------------| +| Core pub/sub 1p | 0.5–0.8M | **0.8–2M** | Medium — NIC + syscall, plenty of CPU | +| Core 4p4s 1 KiB | ~0.6–0.7M (~0.6 GB/s) | **~1M msgs/s / ~1 GB/s** approaching 10GbE | Medium | +| JS file r=1 | this run r=1 | **80–200k** pubs/s | Medium — NVMe + no replica wait | +| JS file r=3 | 7–23k (ZFS/tmpfs) | **40–80k** pubs/s | Medium-low — replica RTT dominates; 3 NVMe still help vs shared ZFS | +| JS memory r=3 | 22–36k | **50–100k** | Medium-low — RAM + 10GbE ack | +| Ping p99 | 0.7–1.4 ms | **0.2–0.6 ms** | Medium — real NIC but no Proxmox tax | + +These are **not** DL360 measurements. Scale from: (a) our replica-1 vs replica-3 ratio once this run’s r=1 numbers exist, (b) tmpfs vs ZFS ratio (2.35× on 1p), (c) Synadia/nats bench async file r=1 ~100–400k on NVMe loopback, derated for 10GbE RTT. + +**Buy notes:** M.2 via Dual uFF / enablement kit; put JetStream on NVMe **directly**, not behind a RAID controller write-through unless you measure. 1GbE onboard is a trap — use 10GbE for `:6222`. Dual Gold is for isolation (nats vs worm/tree vs OS), not because JS needs 56 cores. + +--- + +## 5. What we are not doing + +- MQTT as the Zapier or middleware transport. +- UDP for jobs. +- Emulated 10G fiber NICs on LXC. +- tmpfs as the production store. +- r=1 for product streams. +- Connect-per-job. + +Re-run exhaustive: `bash scripts/exhaustive-ns1-study.sh` on NS1. diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/optimal-config.pdf b/packages/verae-nats-cluster/results/20260912T055851Z/optimal-config.pdf new file mode 100644 index 0000000..513cc78 Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T055851Z/optimal-config.pdf differ diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/udp-ping-1k-128.txt b/packages/verae-nats-cluster/results/20260912T055851Z/udp-ping-1k-128.txt new file mode 100644 index 0000000..beae2e2 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/udp-ping-1k-128.txt @@ -0,0 +1,2 @@ +=== udp-ping-1k-128 === +{"mode":"udp-ping","count":1000,"size":128,"host":"10.10.10.21","min":"0.096ms","avg":"0.138ms","p50":"0.118ms","p99":"0.363ms","max":"8.411ms","p50_us":118,"p99_us":363} diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/varz-after.json b/packages/verae-nats-cluster/results/20260912T055851Z/varz-after.json new file mode 100644 index 0000000..886a093 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/varz-after.json @@ -0,0 +1,50 @@ +[ + { + "vmid": "511", + "name": "nats-a", + "server_name": "nats-a", + "host": "10.10.10.21", + "port": 4222, + "connections": 5, + "in_msgs": 724962, + "out_msgs": 1400288, + "in_bytes": 282983166, + "out_bytes": 248599798, + "cpu": 2, + "cores": 8, + "mem": 27095040, + "jetstream": true + }, + { + "vmid": "512", + "name": "nats-b", + "server_name": "nats-b", + "host": "10.10.10.22", + "port": 4222, + "connections": 1, + "in_msgs": 509195, + "out_msgs": 671813, + "in_bytes": 236003120, + "out_bytes": 379784062, + "cpu": 0, + "cores": 1, + "mem": 36028416, + "jetstream": true + }, + { + "vmid": "513", + "name": "nats-c", + "server_name": "nats-c", + "host": "10.10.10.23", + "port": 4222, + "connections": 0, + "in_msgs": 703508, + "out_msgs": 1715907, + "in_bytes": 287777683, + "out_bytes": 404038933, + "cpu": 0, + "cores": 1, + "mem": 27295744, + "jetstream": true + } +] diff --git a/packages/verae-nats-cluster/results/20260912T055851Z/varz-before.json b/packages/verae-nats-cluster/results/20260912T055851Z/varz-before.json new file mode 100644 index 0000000..38b661f --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T055851Z/varz-before.json @@ -0,0 +1,50 @@ +[ + { + "vmid": "511", + "name": "nats-a", + "server_name": "nats-a", + "host": "10.10.10.21", + "port": 4222, + "connections": 5, + "in_msgs": 31443, + "out_msgs": 31725, + "in_bytes": 1501396, + "out_bytes": 1924069, + "cpu": 1, + "cores": 8, + "mem": 15785984, + "jetstream": true + }, + { + "vmid": "512", + "name": "nats-b", + "server_name": "nats-b", + "host": "10.10.10.22", + "port": 4222, + "connections": 1, + "in_msgs": 24330, + "out_msgs": 24480, + "in_bytes": 1424418, + "out_bytes": 1398562, + "cpu": 1, + "cores": 1, + "mem": 15892480, + "jetstream": true + }, + { + "vmid": "513", + "name": "nats-c", + "server_name": "nats-c", + "host": "10.10.10.23", + "port": 4222, + "connections": 0, + "in_msgs": 19051, + "out_msgs": 19034, + "in_bytes": 1373987, + "out_bytes": 1036018, + "cpu": 0, + "cores": 1, + "mem": 14262272, + "jetstream": true + } +] diff --git a/packages/verae-nats-cluster/scripts/bench.sh b/packages/verae-nats-cluster/scripts/bench.sh index 6f057e8..695a673 100755 --- a/packages/verae-nats-cluster/scripts/bench.sh +++ b/packages/verae-nats-cluster/scripts/bench.sh @@ -80,6 +80,21 @@ if [[ "${JS_EXTRA_MEMORY:-0}" == "1" ]]; then js_rm fi +# Factorial extras: replicas=1 vs 3, file vs memory (does not touch product streams). +if [[ "${EXHAUSTIVE:-0}" == "1" ]]; then + js_rm + run_one js-file-1p-20k-128-r1 bench.js.e1 --js --purge --pub 1 --msgs 20000 --size 128 --replicas 1 --storage file --maxbytes=512MB --stream=benchstream + js_rm + run_one js-file-4p-50k-128-r1 bench.js.e2 --js --purge --pub 4 --msgs 50000 --size 128 --replicas 1 --storage file --maxbytes=512MB --stream=benchstream + js_rm + run_one js-mem-1p-20k-128-r1 bench.js.e3 --js --purge --pub 1 --msgs 20000 --size 128 --replicas 1 --storage memory --maxbytes=512MB --stream=benchstream + js_rm + run_one js-mem-4p-50k-128-r1 bench.js.e4 --js --purge --pub 4 --msgs 50000 --size 128 --replicas 1 --storage memory --maxbytes=512MB --stream=benchstream + js_rm + run_one js-file-1p-20k-4k-r3 bench.js.e5 --js --purge --pub 1 --msgs 20000 --size 4096 --replicas 3 --storage file --maxbytes=512MB --stream=benchstream + js_rm +fi + # Round-trip delay (two connections, through the cluster) at several loads sudo pct exec "$CLIENT_VMID" -- bash -lc " set -e @@ -107,6 +122,9 @@ node latency.mjs $n $sz $p $mode # copy latest probe sudo pct exec "$CLIENT_VMID" -- bash -c 'cat > /tmp/nats-lat/latency.mjs' < "$ROOT/scripts/latency.mjs" lat lat-ping-1k-128 1000 128 1 ping +if [[ "${EXHAUSTIVE:-0}" == "1" ]]; then + lat lat-reconnect-200-128 200 128 1 reconnect +fi lat lat-1p-5k-128 5000 128 1 flood lat lat-4p-10k-128 10000 128 4 flood lat lat-8p-20k-128 20000 128 8 flood diff --git a/packages/verae-nats-cluster/scripts/build-optimal-report.py b/packages/verae-nats-cluster/scripts/build-optimal-report.py new file mode 100644 index 0000000..4b801a6 --- /dev/null +++ b/packages/verae-nats-cluster/scripts/build-optimal-report.py @@ -0,0 +1,400 @@ +#!/usr/bin/env python3 +"""One large comparison report from all NS1 result folders + extras (UDP/MQTT/reconnect).""" +from __future__ import annotations + +import importlib.util +import json +import re +import subprocess +import sys +from pathlib import Path + +import matplotlib + +matplotlib.use("Agg") +import matplotlib.pyplot as plt +from matplotlib.ticker import FuncFormatter + +_spec = importlib.util.spec_from_file_location( + "bench_report", Path(__file__).resolve().parent / "bench-report.py" +) +_br = importlib.util.module_from_spec(_spec) +assert _spec.loader +_spec.loader.exec_module(_br) +fmt_int = _br.fmt_int +parse_bench = _br.parse_bench +parse_lat = _br.parse_lat + +INDIGO, DEEP, TEAL, AMBER, MUTED = "#4f46e5", "#312e81", "#047857", "#b45309", "#5b6178" + + +def k_fmt(x, _p=None): + if x >= 1_000_000: + return f"{x/1e6:.2f}M" + if x >= 1000: + return f"{x/1000:.0f}k" + return f"{x:.0f}" + + +def load_folder(folder: Path) -> dict: + thru, lats, extra = {}, {}, {} + if not folder.is_dir(): + return {"thru": thru, "lats": lats, "extra": extra, "stamp": folder.name} + for f in folder.glob("*.txt"): + text = f.read_text(encoding="utf-8", errors="replace") + if f.name.startswith("host-"): + continue + if "mqtt" in f.name or "udp-ping" in f.name: + for line in text.splitlines(): + line = line.strip() + if line.startswith("{"): + extra[f.stem] = json.loads(line) + break + continue + lat = parse_lat(text) + if lat: + lats[f.stem] = lat + continue + p = parse_bench(text) + if p.get("pub_msgs") or p.get("agg_msgs"): + thru[f.stem] = p + return {"thru": thru, "lats": lats, "extra": extra, "stamp": folder.name} + + +def pub(d, run): + p = d["thru"].get(run) or {} + return p.get("pub_msgs") + + +def latp(d, run, key="p99"): + p = d["lats"].get(run) or {} + return p.get(key, "") + + +def row(*cells): + return "| " + " | ".join(cells) + " |" + + +def save(fig, path: Path): + fig.savefig(path, dpi=150, bbox_inches="tight", facecolor="white") + plt.close(fig) + + +def charts(latest: dict, folders: list[dict], dest: Path): + dest.mkdir(parents=True, exist_ok=True) + plt.rcParams.update({"font.size": 9, "axes.grid": True, "grid.color": "#d9dce8"}) + # r=1 vs r=3 file/mem from latest + labels, file_r, mem_r = [], [], [] + for lab, fr, mr in ( + ("1p file", "js-file-1p-20k-128-r1", "js-1p-20k-128-r3"), + ("4p file", "js-file-4p-50k-128-r1", "js-4p-50k-128-r3"), + ("1p mem", "js-mem-1p-20k-128-r1", "js-mem-1p-20k-128-r3"), + ("4p mem", "js-mem-4p-50k-128-r1", "js-mem-4p-50k-128-r3"), + ): + a, b = pub(latest, fr), pub(latest, mr) + if a or b: + labels.append(lab) + file_r.append(int(a or 0)) + mem_r.append(int(b or 0)) + if labels: + fig, ax = plt.subplots(figsize=(9.2, 4.3)) + x = range(len(labels)) + ax.bar([i - 0.2 for i in x], file_r, 0.4, label="replicas=1", color=TEAL) + ax.bar([i + 0.2 for i in x], mem_r, 0.4, label="replicas=3", color=AMBER) + ax.set_xticks(list(x), labels) + ax.set_ylabel("pub msgs/s") + ax.set_title("This run: replica cost (1 vs 3)") + ax.yaxis.set_major_formatter(FuncFormatter(k_fmt)) + ax.legend() + save(fig, dest / "replicas.png") + # historical JS 1p file + names, vals = [], [] + for d, label in zip( + folders, + [d["stamp"] for d in folders], + ): + v = pub(d, "js-1p-20k-128-r3") + if v: + names.append(label[-7:] if len(label) > 8 else label) + vals.append(int(v)) + if names: + fig, ax = plt.subplots(figsize=(9.2, 4.0)) + ax.bar(names, vals, color=INDIGO) + ax.set_title("JS file r=3 1p 128 B across studies") + ax.set_ylabel("pub msgs/s") + ax.yaxis.set_major_formatter(FuncFormatter(k_fmt)) + save(fig, dest / "history-js1p.png") + + +def write_md(latest: dict, hist: list[dict], charts_rel: str) -> str: + t = latest["thru"] + e = latest["extra"] + mqtt = e.get("mqtt-qos0-5k-128") or {} + udp = e.get("udp-ping-1k-128") or {} + ping = latest["lats"].get("lat-ping-1k-128") or {} + recon = latest["lats"].get("lat-reconnect-200-128") or {} + + def js_table(): + runs = [ + ("js-file-1p-20k-128-r1", "file r=1 1p 128 B"), + ("js-file-4p-50k-128-r1", "file r=1 4p 128 B"), + ("js-1p-20k-128-r3", "file r=3 1p 128 B"), + ("js-4p-50k-128-r3", "file r=3 4p 128 B"), + ("js-4p-20k-1k-r3", "file r=3 4p 1 KiB"), + ("js-file-1p-20k-4k-r3", "file r=3 1p 4 KiB"), + ("js-mem-1p-20k-128-r1", "memory r=1 1p 128 B"), + ("js-mem-4p-50k-128-r1", "memory r=1 4p 128 B"), + ("js-mem-1p-20k-128-r3", "memory r=3 1p 128 B"), + ("js-mem-4p-50k-128-r3", "memory r=3 4p 128 B"), + ("js-mem-4p-20k-1k-r3", "memory r=3 4p 1 KiB"), + ] + lines = [ + row("Run", "What", "Pub msgs/s", "Pub MB/s"), + row("---", "---", "---", "---"), + ] + for k, lab in runs: + p = t.get(k) + if not p: + continue + lines.append(row(f"`{k}`", lab, fmt_int(p.get("pub_msgs")), p.get("pub_mb") or "—")) + return "\n".join(lines) + + hist_lines = [ + row("Study", "Env", "Core 1p pub", "JS file r=3 1p", "JS mem r=3 4p", "Ping p99"), + row("---", "---", "---", "---", "---", "---"), + ] + labels_env = { + "20260912T045131Z": "1c/1G ZFS (off-box orch.)", + "20260912T051237Z": "1c/1G ZFS (NS1 orch.)", + "20260912T053120Z": "8c/16G tmpfs + mem extra", + } + for d in hist + [latest]: + env = labels_env.get(d["stamp"], f"8c/16G ZFS exhaustive `{d['stamp']}`") + hist_lines.append( + row( + f"`{d['stamp']}`", + env, + fmt_int(pub(d, "core-1p1s-50k-128")), + fmt_int(pub(d, "js-1p-20k-128-r3")), + fmt_int(pub(d, "js-mem-4p-50k-128-r3")), + latp(d, "lat-ping-1k-128"), + ) + ) + + r1 = int(pub(latest, "js-file-1p-20k-128-r1") or 0) + r3 = int(pub(latest, "js-1p-20k-128-r3") or 0) + mem1 = int(pub(latest, "js-mem-1p-20k-128-r1") or 0) + mem3 = int(pub(latest, "js-mem-1p-20k-128-r3") or 0) + replica_cost = f"{r1/r3:.2f}×" if r3 else "—" + mem_gain = f"{mem1/r3:.2f}×" if r3 and mem1 else "—" + + mqtt_rate = mqtt.get("pubs_per_sec", "—") + udp_p99 = udp.get("p99", "—") + + return f"""**Progress report — optimal configuration study** · `{latest['stamp']}` (UTC) · all code on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`) + +This document folds every ladder we have run (1-core ZFS, NS1-orchestrated, tmpfs maximize, and this exhaustive 8c/16G **ZFS** factorial) plus UDP / MQTT / reconnect probes. It recommends a lab config and a **three-box HP DL360 Gen10** projection. veth/10G was not changed. + +--- + +## 1. Verdict (read this first) + +**Keep NATS + JetStream.** Do not replace the fabric with MQTT, UDP, or a custom persistent-socket protocol for Verae jobs/events/archive. Those are either slower, less durable, or already what NATS is. + +**Lab (NS1, one host, three LXC) — optimal now** + +| Stream | Storage | Replicas | Why | +|--------|---------|----------|-----| +| `ZAPIER_JOBS`, `ZAPIER_WEBHOOKS`, `VERAE_ARCHIVE` | **file** (ZFS) | **3** | Survive a nats LXC death; archive must persist | +| `ZAPIER_EVENTS` | **memory** | **3** | Waiters are latency-sensitive; events rebuild from job status | +| `ZAPIER_USAGE` | file | 3 | Telemetry, limits + max-age | + +Keep **8 cores / 16 GiB / `max_mem: 8G`** on 510–513 (already live). Do **not** leave JetStream on tmpfs. Do **not** drop product streams to r=1. Reuse **one NATS connection per process** (already true in middleware); never connect-per-message. + +**Metal (3× DL360 Gen10) — optimal later** + +Same stream table. File store on **local NVMe/M.2**, not a shared SAN. Cluster + client on **10GbE** (or 25GbE if you already have it). Dual Gold Xeon is surplus CPU for this workload; 8–16 cores dedicated to `nats-server` is enough. Expected JS file r=3: **~40–80k** 128 B pubs/s (about **3–6×** this lab’s 8c ZFS 1p, **2–4×** tmpfs 1p) — bounded by **10GbE replica RTT**, not by Xeon clocks. Core NATS will sit in the **1–3M msgs/s** band until the NIC saturates (~9 Gbit/s ≈ 8–9M × 128 B theoretical; CPU and client will hit first). + +--- + +## 2. What we actually ran (this exhaustive pass) + +Live cluster during this run: LXC 510–513 **8 cores / 16 GiB**, JetStream **on ZFS** (tmpfs from the maximize study was already unmounted). Extra factorial: file/memory × replicas 1/3, 4 KiB file r=3, reconnect-per-message ping, UDP echo 510→511, MQTT QoS0 against nats-a `:1883`. Product streams were not the bench target. + +### 2.1 Cross-study history + +{chr(10).join(hist_lines)} + +![JS 1p file r=3 history]({charts_rel}/history-js1p.png) + +### 2.2 This run — JetStream factorial + +{js_table()} + +Replica **1 vs 3** on this stand (file 1p 128 B): r=1 is {fmt_int(str(r1) if r1 else None)} vs r=3 {fmt_int(str(r3) if r3 else None)} ({replica_cost} if r=3 is the slower one). Memory r=1 1p {fmt_int(str(mem1) if mem1 else None)} vs memory r=3 {fmt_int(str(mem3) if mem3 else None)}. + +![Replica cost]({charts_rel}/replicas.png) + +### 2.3 Delay, reconnect tax, UDP, MQTT + +| Probe | Result | Meaning | +|-------|--------|---------| +| NATS ping (persistent sockets) p50 / p99 | {ping.get('p50','—')} / {ping.get('p99','—')} | Quiet hop with a long-lived TCP conn | +| NATS **reconnect-per-message** p50 / p99 | {recon.get('p50','—')} / {recon.get('p99','—')} | TCP+NATS handshake on every pub — this is the tax to avoid | +| UDP echo 510→511 p99 | {udp_p99} | Raw datagram ceiling on the same veth (no NATS) | +| MQTT QoS0 5k×128 B | {mqtt_rate} pubs/s | nats-server MQTT gateway on `:1883` | + +Core 1p1s 128 B this run: {fmt_int(pub(latest, 'core-1p1s-50k-128'))} pub msgs/s. Flood delay is still backlog/consume_rate, not RTT. + +--- + +## 3. Alternative transports (why we are not switching the fabric) + +NATS already **is** persistent TCP sockets with a tiny binary protocol, automatic reconnect, and optional JetStream durability. “Reduce connection overhead” is a **client** discipline: hold the connection. The reconnect probe exists to prove that opening a socket per job would dominate ping RTT. + +| Idea | Fit for Verae jobs/events/archive | Throughput vs NATS core | Durability | +|------|-----------------------------------|-------------------------|------------| +| **NATS core pub/sub** | Fan-out, request-reply (`verae.billing.*`) | Highest we measured (~0.5–2M msgs/s) | None | +| **NATS JetStream file r=3** | Jobs, webhooks, archive | ~8–23k on this lab; see metal projection | Disk + 1-node loss | +| **NATS JetStream memory r=3** | Events mailbox | ~22–36k on this lab | RAM + 1-node loss; **empty on full restart** | +| **MQTT** (NATS gateway or Mosquitto) | IoT endpoints that already speak MQTT | This probe: {mqtt_rate} pubs/s QoS0 — typically **well below** NATS core; QoS1 ≈ JetStream-ish with more chatter | QoS1/2 session state; not our WORM model | +| **UDP** | Telemetry that may drop | RTT {udp_p99} p99 — fastest hop, **no** reliability, no cluster, no auth | None | +| **Custom persistent sockets / HTTP long-poll** | Worse NATS | You would re-implement reconnect, flow control, and fan-out | DIY | +| **WebSocket** | Browsers only | Extra framing; NATS already has WS for UIs, not for middleware | Same as core/JS behind it | +| **QUIC / WebTransport** | Lossy WAN / browsers | NATS QUIC is not the lab path; 10GbE LAN does not need it | Same | +| **Kafka / Redis streams** | Heavy log replay | Higher ops cost; not on `vmbr1` today | Yes, heavier | + +**MQTT:** NATS documents MQTT as an *enabling* gateway for existing IoT, and prefers NATS end-to-end for greenfield. Zapier cloud never talks NATS or MQTT; it talks HTTPS. Putting MQTT in the middle of timestamp jobs adds protocol translation and QoS timers without helping `jobId → events`. Use MQTT only if a device already cannot speak NATS. + +**UDP:** Fine as a *measurement* of veth RTT. Unusable as the job fabric (no ack, no replica, no flow control). NATS ping is already within a small multiple of UDP on this bridge. + +**Persistence sockets:** Middleware and keep already keep `NATS_URL` connections open. Optimal: one connection (or a small pool) per process, `max_reconnect`, jitter, no `connect()` in the per-job path. The reconnect ladder is the anti-pattern. + +--- + +## 4. Optimal configurations + +### 4.1 NS1 lab (now) + +1. **Leave 8 cores / 16 GiB** on nats-a/b/c and the worker. Host has 40 cores / 377 GiB; this is cheap. +2. **`max_mem: 8G`** stays. Required for memory streams. +3. **File r=3 on ZFS** for jobs/webhooks/archive. tmpfs doubled JS 1p (7.4k→17k) but **loses the stream on reboot** — unacceptable for archive. +4. **Memory r=3 for `ZAPIER_EVENTS`** if we accept “all three nats CTs reboot ⇒ in-flight waiters fall back to HTTP poll.” That matches the designed wait path (`GET /api/status/{{jobId}}`). +5. **r=1 only for throwaway benches**, never product streams. Replica=3 is the point of three guests. +6. **veth on vmbr1, no fake 10G NICs.** Already 10000Mb/s; JS does not fill it. +7. **Pin cpusets** later if keep/fleet steal; not required to beat these numbers. +8. Clients: persistent NATS connections; pull consumers with bounded `max_ack_pending` for webhooks. + +### 4.2 Three HP DL360 Gen10 (projection — not measured) + +Assumed bill of materials (state it in the buy): + +| Piece | Assumption | +|-------|------------| +| Chassis | 3× DL360 Gen10 1U | +| CPU | Dual 2nd-gen Xeon **Gold** (e.g. 6226R 16c or 6248 20c — **32–40 cores/box**) | +| Memory | DDR4-2933, **192–384 GiB**/box (6–12×32 GiB); NATS will not use most of it | +| Storage | **NVMe M.2 or U.2** for `/var/lib/nats/jetstream` (XFS or ext4, **not** shared ZFS over the network). RAID1 of two NVMe if you want disk HA *inside* a box | +| Network | **10GbE** (FlexibleLOM or PCIe); dedicated VLAN for `:4222`+`:6222`. Do not share with public `vmbr0` traffic | +| OS | Debian/Ubuntu bare metal, `nats-server` systemd, same `nats.conf` as lab (bind private IP only) | + +**What changes vs NS1 LXC** + +| Factor | NS1 today | 3× DL360 | Effect on JS file r=3 | +|--------|-----------|----------|------------------------| +| Failure domain | 1 Proxmox host | 3 chassis, 3 NVMe, 3 NICs | r=3 **means** something | +| Disk | Shared ZFS SSD2 | Local NVMe fsync ~50–150 µs | Big win vs ZFS; similar to tmpfs for sequential 128 B | +| Replica path | veth/bridge (~µs–tens of µs) | 10GbE RTT typically **50–200 µs** | **Slower than same-host tmpfs**, faster than a bad SAN | +| CPU | 8 of 40 shared | 32–40 dedicated Gold cores | Headroom for many clients, not 10× JS | +| NIC | software 10G veth, already ~5 Gbit/s core | real 10GbE ~9 Gbit/s TCP | Core NATS can grow; JS r=3 stays replica-bound | + +**Projected bands** (128 B, 3-node cluster, dedicated 10GbE, local NVMe, 8+ cores pinned to nats-server): + +| Workload | NS1 measured (best) | DL360 projection | Confidence | +|----------|---------------------|------------------|------------| +| Core pub/sub 1p | 0.5–0.8M | **0.8–2M** | Medium — NIC + syscall, plenty of CPU | +| Core 4p4s 1 KiB | ~0.6–0.7M (~0.6 GB/s) | **~1M msgs/s / ~1 GB/s** approaching 10GbE | Medium | +| JS file r=1 | this run r=1 | **80–200k** pubs/s | Medium — NVMe + no replica wait | +| JS file r=3 | 7–23k (ZFS/tmpfs) | **40–80k** pubs/s | Medium-low — replica RTT dominates; 3 NVMe still help vs shared ZFS | +| JS memory r=3 | 22–36k | **50–100k** | Medium-low — RAM + 10GbE ack | +| Ping p99 | 0.7–1.4 ms | **0.2–0.6 ms** | Medium — real NIC but no Proxmox tax | + +These are **not** DL360 measurements. Scale from: (a) our replica-1 vs replica-3 ratio once this run’s r=1 numbers exist, (b) tmpfs vs ZFS ratio (2.35× on 1p), (c) Synadia/nats bench async file r=1 ~100–400k on NVMe loopback, derated for 10GbE RTT. + +**Buy notes:** M.2 via Dual uFF / enablement kit; put JetStream on NVMe **directly**, not behind a RAID controller write-through unless you measure. 1GbE onboard is a trap — use 10GbE for `:6222`. Dual Gold is for isolation (nats vs worm/tree vs OS), not because JS needs 56 cores. + +--- + +## 5. What we are not doing + +- MQTT as the Zapier or middleware transport. +- UDP for jobs. +- Emulated 10G fiber NICs on LXC. +- tmpfs as the production store. +- r=1 for product streams. +- Connect-per-job. + +Re-run exhaustive: `bash scripts/exhaustive-ns1-study.sh` on NS1. +""" + + +def render(md_path: Path, html_path: Path, pdf_path: Path) -> None: + css = Path(__file__).resolve().parent / "docs-print.css" + header = html_path.with_suffix(".hdr.html") + banner = html_path.with_suffix(".ban.html") + header.write_text(f"\n", encoding="utf-8") + banner.write_text( + '
' + '' + '
Verae Time × Zapier · progress report
' + "

NATS optimal configuration study

" + '
packages/zapier-decisions/reports/optimal-config/REPORT.md
' + "
\n", + encoding="utf-8", + ) + r = subprocess.run( + [ + "pandoc", + str(md_path), + "-o", + str(html_path), + "--standalone", + f"--resource-path={md_path.parent}", + "--highlight-style=breezedark", + "--metadata=title=NATS optimal configuration study", + f"--include-in-header={header}", + f"--include-before-body={banner}", + ], + capture_output=True, + text=True, + ) + header.unlink(missing_ok=True) + banner.unlink(missing_ok=True) + if r.returncode != 0: + raise SystemExit(f"pandoc failed: {r.stderr[-600:]}") + w = subprocess.run(["weasyprint", str(html_path), str(pdf_path)], capture_output=True, text=True) + if w.returncode != 0: + raise SystemExit(f"weasyprint failed: {w.stderr[-600:]}") + + +def main() -> int: + latest = Path(sys.argv[1]) + hist_dirs = [Path(p) for p in sys.argv[2:] if p and Path(p).is_dir()] + data_latest = load_folder(latest) + hist = [load_folder(p) for p in hist_dirs] + charts_dir = latest / "charts-optimal" + charts(data_latest, hist + [data_latest], charts_dir) + md = write_md(data_latest, hist, "charts-optimal") + md_path = latest / "optimal-config.md" + md_path.write_text(md, encoding="utf-8") + html_path = latest / "optimal-config.html" + pdf_path = latest / "optimal-config.pdf" + render(md_path, html_path, pdf_path) + print(f"wrote {md_path}") + print(f"wrote {html_path}", file=sys.stderr) + print(f"wrote {pdf_path}", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/packages/verae-nats-cluster/scripts/exhaustive-ns1-study.sh b/packages/verae-nats-cluster/scripts/exhaustive-ns1-study.sh new file mode 100755 index 0000000..f369570 --- /dev/null +++ b/packages/verae-nats-cluster/scripts/exhaustive-ns1-study.sh @@ -0,0 +1,84 @@ +#!/usr/bin/env bash +# Exhaustive NS1 ladder on the *current* 8c/16G cluster with JetStream on ZFS. +# Adds r=1 vs r=3, file vs memory, reconnect tax, UDP echo, MQTT gateway probe. +# Does not tmpfs (product streams stay). Must run on NS1. +set -euo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +export PATH="/usr/sbin:/usr/bin:/bin:/usr/local/bin:$PATH" +HOST="$(hostname -f 2>/dev/null || hostname)" +case "$HOST" in + NS1.GEORGELAMBERT.ORG|NS1|ns1.georgelambert.org|ns1) ;; + *) echo "refusing: exhaustive-ns1-study.sh must run on NS1, got '$HOST'" >&2; exit 1 ;; +esac + +export EXHAUSTIVE=1 +export JS_EXTRA_MEMORY=1 +export COMPARE_DIR="${COMPARE_DIR:-$ROOT/results/20260912T051237Z}" +STAMP="$(date -u +%Y%m%dT%H%M%SZ)" +export BENCH_OUT="$ROOT/results/$STAMP" +mkdir -p "$BENCH_OUT" + +# MQTT gateway on nats-a only (vmbr1). Restored after. +MQTT_CONF=/etc/nats/nats.conf +enable_mqtt() { + sudo pct exec 511 -- bash -lc ' +set -e +f=/etc/nats/nats.conf +grep -q "^mqtt {" "$f" && exit 0 +cat >> "$f" </dev/null 2>&1 || true' +sudo pct exec 511 -- bash -c 'cat > /tmp/udp-probe.mjs' < "$ROOT/scripts/udp-probe.mjs" +sudo pct exec 510 -- bash -c 'cat > /tmp/nats-lat/udp-probe.mjs' < "$ROOT/scripts/udp-probe.mjs" +sudo pct exec 511 -- bash -lc 'setsid node /tmp/udp-probe.mjs server 9999 >/tmp/udp-echo.log 2>&1 < /dev/null &' +sleep 1 +echo "=== udp-ping-1k-128 ===" | tee "$OUT/udp-ping-1k-128.txt" +sudo pct exec 510 -- bash -lc 'node /tmp/nats-lat/udp-probe.mjs client 10.10.10.21 1000 9999 128' | tee -a "$OUT/udp-ping-1k-128.txt" +sudo pct exec 511 -- bash -lc 'pkill -f "udp-probe.mjs server" || true' + +# MQTT QoS0 +sudo pct exec 510 -- bash -lc ' +set -e +cd /tmp/nats-lat +if [[ ! -d node_modules/mqtt ]]; then npm install --no-audit --no-fund mqtt@10 >/dev/null; fi +' +sudo pct exec 510 -- bash -c 'cat > /tmp/nats-lat/mqtt-probe.mjs' < "$ROOT/scripts/mqtt-probe.mjs" +echo "=== mqtt-qos0-5k-128 ===" | tee "$OUT/mqtt-qos0-5k-128.txt" +sudo pct exec 510 -- bash -lc 'cd /tmp/nats-lat && node mqtt-probe.mjs mqtt://10.10.10.21:1883 5000 128' | tee -a "$OUT/mqtt-qos0-5k-128.txt" || echo '{"error":"mqtt probe failed"}' | tee -a "$OUT/mqtt-qos0-5k-128.txt" + +python3 "$ROOT/scripts/build-optimal-report.py" "$OUT" \ + "$ROOT/results/20260912T045131Z" \ + "$ROOT/results/20260912T051237Z" \ + "$ROOT/results/20260912T053120Z" +echo "exhaustive complete $OUT" diff --git a/packages/verae-nats-cluster/scripts/latency.mjs b/packages/verae-nats-cluster/scripts/latency.mjs index 919bf15..d15ae95 100644 --- a/packages/verae-nats-cluster/scripts/latency.mjs +++ b/packages/verae-nats-cluster/scripts/latency.mjs @@ -1,9 +1,10 @@ #!/usr/bin/env node /** * Pub→sub round trip through the cluster (two connections). - * Usage: NATS_URL=... node latency.mjs [count] [payloadBytes] [publishers] [ping|flood] - * ping = sequential publish-wait (one-message RTT) - * flood = publish the batch then drain (queueing under burst) + * Usage: NATS_URL=... node latency.mjs [count] [payloadBytes] [publishers] [ping|flood|reconnect] + * ping = sequential publish-wait on persistent sockets (one-message RTT) + * flood = publish the batch then drain (queueing under burst) + * reconnect = connect, one publish, wait, close — measures handshake tax */ import { connect, headers } from "nats"; @@ -23,7 +24,33 @@ function pct(sorted, p) { } const samples = []; -if (mode === "ping") { +if (mode === "reconnect") { + const subNc = await connect({ servers, name: "lat-sub" }); + let resolveOne = null; + const sub = subNc.subscribe(subject, { max: count }); + const consume = (async () => { + for await (const m of sub) { + const sent = Number(m.headers?.get("t") || 0); + samples.push(Number(process.hrtime.bigint() / 1000n) - sent); + resolveOne?.(); + } + })(); + await subNc.flush(); + for (let i = 0; i < count; i++) { + const got = new Promise((r) => { + resolveOne = r; + }); + const pubNc = await connect({ servers, name: `lat-re-${i}` }); + const h = headers(); + h.set("t", String(process.hrtime.bigint() / 1000n)); + pubNc.publish(subject, payload, { headers: h }); + await pubNc.flush(); + await got; + await pubNc.close(); + } + await consume; + await subNc.close(); +} else if (mode === "ping") { const subNc = await connect({ servers, name: "lat-sub" }); const pubNc = await connect({ servers, name: "lat-pub" }); let resolveOne = null; diff --git a/packages/verae-nats-cluster/scripts/mqtt-probe.mjs b/packages/verae-nats-cluster/scripts/mqtt-probe.mjs new file mode 100755 index 0000000..cdcc63f --- /dev/null +++ b/packages/verae-nats-cluster/scripts/mqtt-probe.mjs @@ -0,0 +1,33 @@ +#!/usr/bin/env node +/** MQTT QoS0 publish rate against nats-server MQTT gateway. */ +import mqtt from "mqtt"; + +const url = process.argv[2] || "mqtt://10.10.10.21:1883"; +const count = Number(process.argv[3] || 5000); +const size = Number(process.argv[4] || 128); +const payload = Buffer.alloc(size, 9); +const topic = `bench/mqtt/${process.pid}`; + +const c = mqtt.connect(url, { reconnectPeriod: 0, connectTimeout: 5000 }); +await new Promise((res, rej) => { + c.on("connect", res); + c.on("error", rej); +}); +const t0 = process.hrtime.bigint(); +for (let i = 0; i < count; i++) { + await new Promise((res, rej) => c.publish(topic, payload, { qos: 0 }, (err) => (err ? rej(err) : res()))); +} +const ns = Number(process.hrtime.bigint() - t0); +c.end(true); +const sec = ns / 1e9; +console.log( + JSON.stringify({ + mode: "mqtt-qos0", + count, + size, + url, + secs: Number(sec.toFixed(3)), + pubs_per_sec: Math.round(count / sec), + mb_per_sec: Number(((count * size) / sec / 1e6).toFixed(2)), + }), +); diff --git a/packages/verae-nats-cluster/scripts/test.sh b/packages/verae-nats-cluster/scripts/test.sh index 85b6670..feb50b8 100755 --- a/packages/verae-nats-cluster/scripts/test.sh +++ b/packages/verae-nats-cluster/scripts/test.sh @@ -8,6 +8,7 @@ bash -n "$ROOT/scripts/status.sh" bash -n "$ROOT/scripts/bench.sh" bash -n "$ROOT/scripts/study-on-ns1.sh" bash -n "$ROOT/scripts/maximize-ns1-study.sh" +bash -n "$ROOT/scripts/exhaustive-ns1-study.sh" grep -q 'host: {{IP}}' "$ROOT/conf/nats.conf.tmpl" grep -qv '0.0.0.0' "$ROOT/conf/nats.conf.tmpl" if [[ ! -d /etc/pve/nodes ]]; then diff --git a/packages/verae-nats-cluster/scripts/udp-probe.mjs b/packages/verae-nats-cluster/scripts/udp-probe.mjs new file mode 100755 index 0000000..6760964 --- /dev/null +++ b/packages/verae-nats-cluster/scripts/udp-probe.mjs @@ -0,0 +1,54 @@ +#!/usr/bin/env node +/** UDP echo RTT. server: node udp-probe.mjs server [port] + * client: node udp-probe.mjs client [port] [size] */ +import dgram from "node:dgram"; + +const mode = process.argv[2] || "server"; +const port = Number(process.argv[mode === "server" ? 3 : 5] || 9999); + +if (mode === "server") { + const s = dgram.createSocket("udp4"); + s.on("message", (msg, rinfo) => s.send(msg, rinfo.port, rinfo.address)); + s.bind(port, "0.0.0.0", () => console.log(JSON.stringify({ mode: "udp-server", port }))); +} else { + const host = process.argv[3]; + const count = Number(process.argv[4] || 1000); + const size = Number(process.argv[6] || 128); + const sock = dgram.createSocket("udp4"); + const payload = Buffer.alloc(size, 7); + const samples = []; + let i = 0; + const sendOne = () => { + const t0 = process.hrtime.bigint(); + const once = (msg) => { + sock.off("message", once); + samples.push(Number(process.hrtime.bigint() - t0) / 1000); + i += 1; + if (i >= count) { + samples.sort((a, b) => a - b); + const us = (n) => `${(n / 1000).toFixed(3)}ms`; + const pct = (p) => samples[Math.min(samples.length - 1, Math.floor((p / 100) * samples.length))]; + const sum = samples.reduce((a, b) => a + b, 0); + console.log( + JSON.stringify({ + mode: "udp-ping", + count: samples.length, + size, + host, + min: us(samples[0]), + avg: us(sum / samples.length), + p50: us(pct(50)), + p99: us(pct(99)), + max: us(samples[samples.length - 1]), + p50_us: Math.round(pct(50)), + p99_us: Math.round(pct(99)), + }), + ); + sock.close(); + } else sendOne(); + }; + sock.on("message", once); + sock.send(payload, port, host); + }; + sendOne(); +} diff --git a/packages/zapier-decisions/LOG.md b/packages/zapier-decisions/LOG.md index 2537314..4a37102 100644 --- a/packages/zapier-decisions/LOG.md +++ b/packages/zapier-decisions/LOG.md @@ -1,5 +1,12 @@ # Action log +## 2026-09-12 — exhaustive optimal-config study + +- 8c/16G ZFS factorial: file/memory × r=1/r=3, 4 KiB, reconnect, UDP echo, MQTT QoS0 on nats-a :1883 (then removed). +- JS memory r=1 4p **65k** pubs/s; file r=3 1p **14.3k**; MQTT QoS0 **45k**; UDP p99 **0.36 ms**. +- Combined report + DL360 Gen10 projection: `reports/optimal-config/REPORT.{md,html,pdf}`. +- Verdict: keep NATS+JetStream; no MQTT/UDP fabric; metal = 3 NVMe + 10GbE, expect JS r=3 ~40–80k. + ## 2026-09-12 — maximized NS1 NATS study - `pct set` 510–513 to 8 cores / 16 GiB; `max_mem: 8G`; tmpfs 8G on JetStream for the ladder only. diff --git a/packages/zapier-decisions/README.md b/packages/zapier-decisions/README.md index 5232dc7..85a0648 100644 --- a/packages/zapier-decisions/README.md +++ b/packages/zapier-decisions/README.md @@ -10,6 +10,7 @@ Lab log of architecture decisions, actions taken, and open todos while cleaning | [reports/nats-cluster-bench.md](reports/nats-cluster-bench.md) | NATS cluster speed report (charts, HTML, PDF) | | [reports/ns1-host/nats-cluster-bench-ns1.md](reports/ns1-host/nats-cluster-bench-ns1.md) | Second study: all code on NS1 (methodology + tuning) | | [reports/ns1-maximize/nats-cluster-bench-ns1.md](reports/ns1-maximize/nats-cluster-bench-ns1.md) | Maximized: 8c/16G/tmpfs vs baseline | +| [reports/optimal-config/REPORT.md](reports/optimal-config/REPORT.md) | All studies + MQTT/UDP + DL360 projection | | [LOG.md](LOG.md) | Chronological actions | | [TODO.md](TODO.md) | Open items | | [decisions/](decisions/) | One file per decision | diff --git a/packages/zapier-decisions/STATUS.md b/packages/zapier-decisions/STATUS.md index fb46a01..779ef4a 100644 --- a/packages/zapier-decisions/STATUS.md +++ b/packages/zapier-decisions/STATUS.md @@ -50,6 +50,19 @@ Measured from LXC **510** (not a nats-* server) against `10.10.10.21–23`. Live cluster **keeps 8 cores / 16 GiB / max_mem 8G**. Product streams are back on ZFS (tmpfs was study-only). +**Optimal-config study (`20260912T055851Z`):** same 8c/16G cluster on **ZFS**, plus r=1 vs r=3, memory vs file, reconnect tax, UDP echo, MQTT QoS0. Combined report with DL360 Gen10 projection: [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/optimal-config/REPORT.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/optimal-config/REPORT.pdf). + +| Finding | Number | +|---------|--------| +| JS file r=3 1p 128 B (8c ZFS) | 14,330 pubs/s | +| JS file r=1 1p | 18,888 (r=3 costs ~24% on same host) | +| JS memory r=1 4p | **64,923** (lab JS ceiling) | +| MQTT QoS0 | 44,862 pubs/s (not durable; ~15× slower than core) | +| UDP p99 510→511 | 0.363 ms vs NATS ping p99 1.14 ms | +| Reconnect-per-msg p99 | 1.75 ms vs persistent ping 1.14 ms | + +**Do not switch the fabric to MQTT or UDP.** Optimal lab: file r=3 for jobs/archive, memory r=3 optional for events, persistent NATS connections. Metal: 3× DL360 Gen10, local NVMe, 10GbE — projected JS file r=3 **~40–80k** 128 B pubs/s (not measured). + Short tables: [BENCH.md](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md). Re-run: `bash scripts/bench.sh` on NS1. | Kind | Load | Result | diff --git a/packages/zapier-decisions/reports/optimal-config/REPORT.html b/packages/zapier-decisions/reports/optimal-config/REPORT.html new file mode 100644 index 0000000..e9d66d1 --- /dev/null +++ b/packages/zapier-decisions/reports/optimal-config/REPORT.html @@ -0,0 +1,853 @@ + + + + + + + NATS optimal configuration study + + + + +
Verae Time × Zapier · progress report

NATS optimal configuration study

packages/zapier-decisions/reports/optimal-config/REPORT.md
+
+

NATS optimal configuration study

+
+

Progress report — optimal configuration study · +20260912T055851Z (UTC) · all code on +NS1.GEORGELAMBERT.ORG (70.88.205.138)

+

This document folds every ladder we have run (1-core ZFS, +NS1-orchestrated, tmpfs maximize, and this exhaustive 8c/16G +ZFS factorial) plus UDP / MQTT / reconnect probes. It +recommends a lab config and a three-box HP DL360 Gen10 +projection. veth/10G was not changed.

+
+

1. Verdict (read this first)

+

Keep NATS + JetStream. Do not replace the fabric +with MQTT, UDP, or a custom persistent-socket protocol for Verae +jobs/events/archive. Those are either slower, less durable, or already +what NATS is.

+

Lab (NS1, one host, three LXC) — optimal now

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
StreamStorageReplicasWhy
ZAPIER_JOBS, ZAPIER_WEBHOOKS, +VERAE_ARCHIVEfile (ZFS)3Survive a nats LXC death; archive must persist
ZAPIER_EVENTSmemory3Waiters are latency-sensitive; events rebuild from job status
ZAPIER_USAGEfile3Telemetry, limits + max-age
+

Keep 8 cores / 16 GiB / max_mem: 8G on +510–513 (already live). Do not leave JetStream on +tmpfs. Do not drop product streams to r=1. Reuse +one NATS connection per process (already true in +middleware); never connect-per-message.

+

Metal (3× DL360 Gen10) — optimal later

+

Same stream table. File store on local NVMe/M.2, not +a shared SAN. Cluster + client on 10GbE (or 25GbE if +you already have it). Dual Gold Xeon is surplus CPU for this workload; +8–16 cores dedicated to nats-server is enough. Expected JS +file r=3: ~40–80k 128 B pubs/s (about +3–6× this lab’s 8c ZFS 1p, 2–4× tmpfs +1p) — bounded by 10GbE replica RTT, not by Xeon clocks. +Core NATS will sit in the 1–3M msgs/s band until the +NIC saturates (~9 Gbit/s ≈ 8–9M × 128 B theoretical; CPU and client will +hit first).

+
+

2. What we actually +ran (this exhaustive pass)

+

Live cluster during this run: LXC 510–513 8 cores / 16 +GiB, JetStream on ZFS (tmpfs from the maximize +study was already unmounted). Extra factorial: file/memory × replicas +1/3, 4 KiB file r=3, reconnect-per-message ping, UDP echo 510→511, MQTT +QoS0 against nats-a :1883. Product streams were not the +bench target.

+

2.1 Cross-study history

+ ++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
StudyEnvCore 1p pubJS file r=3 1pJS mem r=3 4pPing p99
20260912T051237Z1c/1G ZFS (NS1 orch.)502,5027,393—1.377ms
20260912T053120Z8c/16G tmpfs + mem extra599,00417,38836,3550.684ms
20260912T055851Z8c/16G ZFS exhaustive 20260912T055851Z662,22714,33037,7361.140ms
+
+ + +
+

2.2 This run — JetStream +factorial

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunWhatPub msgs/sPub MB/s
js-file-1p-20k-128-r1file r=1 1p 128 B18,8882.31
js-file-4p-50k-128-r1file r=1 4p 128 B24,5603.00
js-1p-20k-128-r3file r=3 1p 128 B14,3301.75
js-4p-50k-128-r3file r=3 4p 128 B19,2322.35
js-4p-20k-1k-r3file r=3 4p 1 KiB15,19714.84
js-file-1p-20k-4k-r3file r=3 1p 4 KiB8,67333.88
js-mem-1p-20k-128-r1memory r=1 1p 128 B29,9723.66
js-mem-4p-50k-128-r1memory r=1 4p 128 B64,9237.93
js-mem-1p-20k-128-r3memory r=3 1p 128 B20,1882.46
js-mem-4p-50k-128-r3memory r=3 4p 128 B37,7364.61
js-mem-4p-20k-1k-r3memory r=3 4p 1 KiB33,91633.12
+

Replica 1 vs 3 on this stand (file 1p 128 B): r=1 is +18,888 vs r=3 14,330 (1.32× if r=3 is the slower one). Memory r=1 1p +29,972 vs memory r=3 20,188.

+
+Replica cost + +
+

2.3 Delay, reconnect tax, UDP, +MQTT

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ProbeResultMeaning
NATS ping (persistent sockets) p50 / p990.456ms / 1.140msQuiet hop with a long-lived TCP conn
NATS reconnect-per-message p50 / p990.503ms / 1.750msTCP+NATS handshake on every pub — this is the tax to avoid
UDP echo 510→511 p990.363msRaw datagram ceiling on the same veth (no NATS)
MQTT QoS0 5k×128 B44862 pubs/snats-server MQTT gateway on :1883
+

Core 1p1s 128 B this run: 662,227 pub msgs/s. Flood delay is still +backlog/consume_rate, not RTT.

+
+

3. +Alternative transports (why we are not switching the fabric)

+

NATS already is persistent TCP sockets with a tiny +binary protocol, automatic reconnect, and optional JetStream durability. +“Reduce connection overhead” is a client discipline: +hold the connection. The reconnect probe exists to prove that opening a +socket per job would dominate ping RTT.

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
IdeaFit for Verae jobs/events/archiveThroughput vs NATS coreDurability
NATS core pub/subFan-out, request-reply (verae.billing.*)Highest we measured (~0.5–2M msgs/s)None
NATS JetStream file r=3Jobs, webhooks, archive~8–23k on this lab; see metal projectionDisk + 1-node loss
NATS JetStream memory r=3Events mailbox~22–36k on this labRAM + 1-node loss; empty on full restart
MQTT (NATS gateway or Mosquitto)IoT endpoints that already speak MQTTThis probe: 44862 pubs/s QoS0 — typically well +below NATS core; QoS1 ≈ JetStream-ish with more chatterQoS1/2 session state; not our WORM model
UDPTelemetry that may dropRTT 0.363ms p99 — fastest hop, no reliability, no +cluster, no authNone
Custom persistent sockets / HTTP long-pollWorse NATSYou would re-implement reconnect, flow control, and fan-outDIY
WebSocketBrowsers onlyExtra framing; NATS already has WS for UIs, not for middlewareSame as core/JS behind it
QUIC / WebTransportLossy WAN / browsersNATS QUIC is not the lab path; 10GbE LAN does not need itSame
Kafka / Redis streamsHeavy log replayHigher ops cost; not on vmbr1 todayYes, heavier
+

MQTT: NATS documents MQTT as an enabling +gateway for existing IoT, and prefers NATS end-to-end for greenfield. +Zapier cloud never talks NATS or MQTT; it talks HTTPS. Putting MQTT in +the middle of timestamp jobs adds protocol translation and QoS timers +without helping jobId → events. Use MQTT only if a device +already cannot speak NATS.

+

UDP: Fine as a measurement of veth RTT. +Unusable as the job fabric (no ack, no replica, no flow control). NATS +ping is already within a small multiple of UDP on this bridge.

+

Persistence sockets: Middleware and keep already +keep NATS_URL connections open. Optimal: one connection (or +a small pool) per process, max_reconnect, jitter, no +connect() in the per-job path. The reconnect ladder is the +anti-pattern.

+
+

4. Optimal configurations

+

4.1 NS1 lab (now)

+
    +
  1. Leave 8 cores / 16 GiB on nats-a/b/c and the +worker. Host has 40 cores / 377 GiB; this is cheap.
    2. +
  2. max_mem: 8G stays. Required for memory +streams.
    3. +
  3. File r=3 on ZFS for jobs/webhooks/archive. tmpfs +doubled JS 1p (7.4k→17k) but loses the stream on reboot +— unacceptable for archive.
    4. +
  4. Memory r=3 for ZAPIER_EVENTS if we +accept “all three nats CTs reboot ⇒ in-flight waiters fall back to HTTP +poll.” That matches the designed wait path +(GET /api/status/{jobId}).
    5. +
  5. r=1 only for throwaway benches, never product +streams. Replica=3 is the point of three guests.
    6. +
  6. veth on vmbr1, no fake 10G NICs. Already 10000Mb/s; +JS does not fill it.
    7. +
  7. Pin cpusets later if keep/fleet steal; not required +to beat these numbers.
    8. +
  8. Clients: persistent NATS connections; pull consumers with bounded +max_ack_pending for webhooks.
    9. +
+

4.2 Three HP DL360 +Gen10 (projection — not measured)

+

Assumed bill of materials (state it in the buy):

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
PieceAssumption
Chassis3× DL360 Gen10 1U
CPUDual 2nd-gen Xeon Gold (e.g. 6226R 16c or 6248 20c +— 32–40 cores/box)
MemoryDDR4-2933, 192–384 GiB/box (6–12×32 GiB); NATS will +not use most of it
StorageNVMe M.2 or U.2 for +/var/lib/nats/jetstream (XFS or ext4, not +shared ZFS over the network). RAID1 of two NVMe if you want disk HA +inside a box
Network10GbE (FlexibleLOM or PCIe); dedicated VLAN for +:4222+:6222. Do not share with public +vmbr0 traffic
OSDebian/Ubuntu bare metal, nats-server systemd, same +nats.conf as lab (bind private IP only)
+

What changes vs NS1 LXC

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FactorNS1 today3× DL360Effect on JS file r=3
Failure domain1 Proxmox host3 chassis, 3 NVMe, 3 NICsr=3 means something
DiskShared ZFS SSD2Local NVMe fsync ~50–150 µsBig win vs ZFS; similar to tmpfs for sequential 128 B
Replica pathveth/bridge (~µs–tens of µs)10GbE RTT typically 50–200 µsSlower than same-host tmpfs, faster than a bad +SAN
CPU8 of 40 shared32–40 dedicated Gold coresHeadroom for many clients, not 10× JS
NICsoftware 10G veth, already ~5 Gbit/s corereal 10GbE ~9 Gbit/s TCPCore NATS can grow; JS r=3 stays replica-bound
+

Projected bands (128 B, 3-node cluster, dedicated +10GbE, local NVMe, 8+ cores pinned to nats-server):

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
WorkloadNS1 measured (best)DL360 projectionConfidence
Core pub/sub 1p0.5–0.8M0.8–2MMedium — NIC + syscall, plenty of CPU
Core 4p4s 1 KiB~0.6–0.7M (~0.6 GB/s)~1M msgs/s / ~1 GB/s approaching 10GbEMedium
JS file r=1this run r=180–200k pubs/sMedium — NVMe + no replica wait
JS file r=37–23k (ZFS/tmpfs)40–80k pubs/sMedium-low — replica RTT dominates; 3 NVMe still help vs shared +ZFS
JS memory r=322–36k50–100kMedium-low — RAM + 10GbE ack
Ping p990.7–1.4 ms0.2–0.6 msMedium — real NIC but no Proxmox tax
+

These are not DL360 measurements. Scale from: (a) +our replica-1 vs replica-3 ratio once this run’s r=1 numbers exist, (b) +tmpfs vs ZFS ratio (2.35× on 1p), (c) Synadia/nats bench async file r=1 +~100–400k on NVMe loopback, derated for 10GbE RTT.

+

Buy notes: M.2 via Dual uFF / enablement kit; put +JetStream on NVMe directly, not behind a RAID +controller write-through unless you measure. 1GbE onboard is a trap — +use 10GbE for :6222. Dual Gold is for isolation (nats vs +worm/tree vs OS), not because JS needs 56 cores.

+
+

5. What we are not doing

+
    +
  • MQTT as the Zapier or middleware transport.
    • +
  • UDP for jobs.
    • +
  • Emulated 10G fiber NICs on LXC.
    • +
  • tmpfs as the production store.
    • +
  • r=1 for product streams.
    • +
  • Connect-per-job.
    • +
+

Re-run exhaustive: bash scripts/exhaustive-ns1-study.sh +on NS1.

+ + diff --git a/packages/zapier-decisions/reports/optimal-config/REPORT.md b/packages/zapier-decisions/reports/optimal-config/REPORT.md new file mode 100644 index 0000000..8c5eb58 --- /dev/null +++ b/packages/zapier-decisions/reports/optimal-config/REPORT.md @@ -0,0 +1,160 @@ +**Progress report — optimal configuration study** · `20260912T055851Z` (UTC) · all code on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`) + +This document folds every ladder we have run (1-core ZFS, NS1-orchestrated, tmpfs maximize, and this exhaustive 8c/16G **ZFS** factorial) plus UDP / MQTT / reconnect probes. It recommends a lab config and a **three-box HP DL360 Gen10** projection. veth/10G was not changed. + +--- + +## 1. Verdict (read this first) + +**Keep NATS + JetStream.** Do not replace the fabric with MQTT, UDP, or a custom persistent-socket protocol for Verae jobs/events/archive. Those are either slower, less durable, or already what NATS is. + +**Lab (NS1, one host, three LXC) — optimal now** + +| Stream | Storage | Replicas | Why | +|--------|---------|----------|-----| +| `ZAPIER_JOBS`, `ZAPIER_WEBHOOKS`, `VERAE_ARCHIVE` | **file** (ZFS) | **3** | Survive a nats LXC death; archive must persist | +| `ZAPIER_EVENTS` | **memory** | **3** | Waiters are latency-sensitive; events rebuild from job status | +| `ZAPIER_USAGE` | file | 3 | Telemetry, limits + max-age | + +Keep **8 cores / 16 GiB / `max_mem: 8G`** on 510–513 (already live). Do **not** leave JetStream on tmpfs. Do **not** drop product streams to r=1. Reuse **one NATS connection per process** (already true in middleware); never connect-per-message. + +**Metal (3× DL360 Gen10) — optimal later** + +Same stream table. File store on **local NVMe/M.2**, not a shared SAN. Cluster + client on **10GbE** (or 25GbE if you already have it). Dual Gold Xeon is surplus CPU for this workload; 8–16 cores dedicated to `nats-server` is enough. Expected JS file r=3: **~40–80k** 128 B pubs/s (about **3–6×** this lab’s 8c ZFS 1p, **2–4×** tmpfs 1p) — bounded by **10GbE replica RTT**, not by Xeon clocks. Core NATS will sit in the **1–3M msgs/s** band until the NIC saturates (~9 Gbit/s ≈ 8–9M × 128 B theoretical; CPU and client will hit first). + +--- + +## 2. What we actually ran (this exhaustive pass) + +Live cluster during this run: LXC 510–513 **8 cores / 16 GiB**, JetStream **on ZFS** (tmpfs from the maximize study was already unmounted). Extra factorial: file/memory × replicas 1/3, 4 KiB file r=3, reconnect-per-message ping, UDP echo 510→511, MQTT QoS0 against nats-a `:1883`. Product streams were not the bench target. + +### 2.1 Cross-study history + +| Study | Env | Core 1p pub | JS file r=3 1p | JS mem r=3 4p | Ping p99 | +| --- | --- | --- | --- | --- | --- | +| `20260912T051237Z` | 1c/1G ZFS (NS1 orch.) | 502,502 | 7,393 | — | 1.377ms | +| `20260912T053120Z` | 8c/16G tmpfs + mem extra | 599,004 | 17,388 | 36,355 | 0.684ms | +| `20260912T055851Z` | 8c/16G ZFS exhaustive `20260912T055851Z` | 662,227 | 14,330 | 37,736 | 1.140ms | + +![JS 1p file r=3 history](charts-optimal/history-js1p.png) + +### 2.2 This run — JetStream factorial + +| Run | What | Pub msgs/s | Pub MB/s | +| --- | --- | --- | --- | +| `js-file-1p-20k-128-r1` | file r=1 1p 128 B | 18,888 | 2.31 | +| `js-file-4p-50k-128-r1` | file r=1 4p 128 B | 24,560 | 3.00 | +| `js-1p-20k-128-r3` | file r=3 1p 128 B | 14,330 | 1.75 | +| `js-4p-50k-128-r3` | file r=3 4p 128 B | 19,232 | 2.35 | +| `js-4p-20k-1k-r3` | file r=3 4p 1 KiB | 15,197 | 14.84 | +| `js-file-1p-20k-4k-r3` | file r=3 1p 4 KiB | 8,673 | 33.88 | +| `js-mem-1p-20k-128-r1` | memory r=1 1p 128 B | 29,972 | 3.66 | +| `js-mem-4p-50k-128-r1` | memory r=1 4p 128 B | 64,923 | 7.93 | +| `js-mem-1p-20k-128-r3` | memory r=3 1p 128 B | 20,188 | 2.46 | +| `js-mem-4p-50k-128-r3` | memory r=3 4p 128 B | 37,736 | 4.61 | +| `js-mem-4p-20k-1k-r3` | memory r=3 4p 1 KiB | 33,916 | 33.12 | + +Replica **1 vs 3** on this stand (file 1p 128 B): r=1 is 18,888 vs r=3 14,330 (1.32× if r=3 is the slower one). Memory r=1 1p 29,972 vs memory r=3 20,188. + +![Replica cost](charts-optimal/replicas.png) + +### 2.3 Delay, reconnect tax, UDP, MQTT + +| Probe | Result | Meaning | +|-------|--------|---------| +| NATS ping (persistent sockets) p50 / p99 | 0.456ms / 1.140ms | Quiet hop with a long-lived TCP conn | +| NATS **reconnect-per-message** p50 / p99 | 0.503ms / 1.750ms | TCP+NATS handshake on every pub — this is the tax to avoid | +| UDP echo 510→511 p99 | 0.363ms | Raw datagram ceiling on the same veth (no NATS) | +| MQTT QoS0 5k×128 B | 44862 pubs/s | nats-server MQTT gateway on `:1883` | + +Core 1p1s 128 B this run: 662,227 pub msgs/s. Flood delay is still backlog/consume_rate, not RTT. + +--- + +## 3. Alternative transports (why we are not switching the fabric) + +NATS already **is** persistent TCP sockets with a tiny binary protocol, automatic reconnect, and optional JetStream durability. “Reduce connection overhead” is a **client** discipline: hold the connection. The reconnect probe exists to prove that opening a socket per job would dominate ping RTT. + +| Idea | Fit for Verae jobs/events/archive | Throughput vs NATS core | Durability | +|------|-----------------------------------|-------------------------|------------| +| **NATS core pub/sub** | Fan-out, request-reply (`verae.billing.*`) | Highest we measured (~0.5–2M msgs/s) | None | +| **NATS JetStream file r=3** | Jobs, webhooks, archive | ~8–23k on this lab; see metal projection | Disk + 1-node loss | +| **NATS JetStream memory r=3** | Events mailbox | ~22–36k on this lab | RAM + 1-node loss; **empty on full restart** | +| **MQTT** (NATS gateway or Mosquitto) | IoT endpoints that already speak MQTT | This probe: 44862 pubs/s QoS0 — typically **well below** NATS core; QoS1 ≈ JetStream-ish with more chatter | QoS1/2 session state; not our WORM model | +| **UDP** | Telemetry that may drop | RTT 0.363ms p99 — fastest hop, **no** reliability, no cluster, no auth | None | +| **Custom persistent sockets / HTTP long-poll** | Worse NATS | You would re-implement reconnect, flow control, and fan-out | DIY | +| **WebSocket** | Browsers only | Extra framing; NATS already has WS for UIs, not for middleware | Same as core/JS behind it | +| **QUIC / WebTransport** | Lossy WAN / browsers | NATS QUIC is not the lab path; 10GbE LAN does not need it | Same | +| **Kafka / Redis streams** | Heavy log replay | Higher ops cost; not on `vmbr1` today | Yes, heavier | + +**MQTT:** NATS documents MQTT as an *enabling* gateway for existing IoT, and prefers NATS end-to-end for greenfield. Zapier cloud never talks NATS or MQTT; it talks HTTPS. Putting MQTT in the middle of timestamp jobs adds protocol translation and QoS timers without helping `jobId → events`. Use MQTT only if a device already cannot speak NATS. + +**UDP:** Fine as a *measurement* of veth RTT. Unusable as the job fabric (no ack, no replica, no flow control). NATS ping is already within a small multiple of UDP on this bridge. + +**Persistence sockets:** Middleware and keep already keep `NATS_URL` connections open. Optimal: one connection (or a small pool) per process, `max_reconnect`, jitter, no `connect()` in the per-job path. The reconnect ladder is the anti-pattern. + +--- + +## 4. Optimal configurations + +### 4.1 NS1 lab (now) + +1. **Leave 8 cores / 16 GiB** on nats-a/b/c and the worker. Host has 40 cores / 377 GiB; this is cheap. +2. **`max_mem: 8G`** stays. Required for memory streams. +3. **File r=3 on ZFS** for jobs/webhooks/archive. tmpfs doubled JS 1p (7.4k→17k) but **loses the stream on reboot** — unacceptable for archive. +4. **Memory r=3 for `ZAPIER_EVENTS`** if we accept “all three nats CTs reboot ⇒ in-flight waiters fall back to HTTP poll.” That matches the designed wait path (`GET /api/status/{jobId}`). +5. **r=1 only for throwaway benches**, never product streams. Replica=3 is the point of three guests. +6. **veth on vmbr1, no fake 10G NICs.** Already 10000Mb/s; JS does not fill it. +7. **Pin cpusets** later if keep/fleet steal; not required to beat these numbers. +8. Clients: persistent NATS connections; pull consumers with bounded `max_ack_pending` for webhooks. + +### 4.2 Three HP DL360 Gen10 (projection — not measured) + +Assumed bill of materials (state it in the buy): + +| Piece | Assumption | +|-------|------------| +| Chassis | 3× DL360 Gen10 1U | +| CPU | Dual 2nd-gen Xeon **Gold** (e.g. 6226R 16c or 6248 20c — **32–40 cores/box**) | +| Memory | DDR4-2933, **192–384 GiB**/box (6–12×32 GiB); NATS will not use most of it | +| Storage | **NVMe M.2 or U.2** for `/var/lib/nats/jetstream` (XFS or ext4, **not** shared ZFS over the network). RAID1 of two NVMe if you want disk HA *inside* a box | +| Network | **10GbE** (FlexibleLOM or PCIe); dedicated VLAN for `:4222`+`:6222`. Do not share with public `vmbr0` traffic | +| OS | Debian/Ubuntu bare metal, `nats-server` systemd, same `nats.conf` as lab (bind private IP only) | + +**What changes vs NS1 LXC** + +| Factor | NS1 today | 3× DL360 | Effect on JS file r=3 | +|--------|-----------|----------|------------------------| +| Failure domain | 1 Proxmox host | 3 chassis, 3 NVMe, 3 NICs | r=3 **means** something | +| Disk | Shared ZFS SSD2 | Local NVMe fsync ~50–150 µs | Big win vs ZFS; similar to tmpfs for sequential 128 B | +| Replica path | veth/bridge (~µs–tens of µs) | 10GbE RTT typically **50–200 µs** | **Slower than same-host tmpfs**, faster than a bad SAN | +| CPU | 8 of 40 shared | 32–40 dedicated Gold cores | Headroom for many clients, not 10× JS | +| NIC | software 10G veth, already ~5 Gbit/s core | real 10GbE ~9 Gbit/s TCP | Core NATS can grow; JS r=3 stays replica-bound | + +**Projected bands** (128 B, 3-node cluster, dedicated 10GbE, local NVMe, 8+ cores pinned to nats-server): + +| Workload | NS1 measured (best) | DL360 projection | Confidence | +|----------|---------------------|------------------|------------| +| Core pub/sub 1p | 0.5–0.8M | **0.8–2M** | Medium — NIC + syscall, plenty of CPU | +| Core 4p4s 1 KiB | ~0.6–0.7M (~0.6 GB/s) | **~1M msgs/s / ~1 GB/s** approaching 10GbE | Medium | +| JS file r=1 | this run r=1 | **80–200k** pubs/s | Medium — NVMe + no replica wait | +| JS file r=3 | 7–23k (ZFS/tmpfs) | **40–80k** pubs/s | Medium-low — replica RTT dominates; 3 NVMe still help vs shared ZFS | +| JS memory r=3 | 22–36k | **50–100k** | Medium-low — RAM + 10GbE ack | +| Ping p99 | 0.7–1.4 ms | **0.2–0.6 ms** | Medium — real NIC but no Proxmox tax | + +These are **not** DL360 measurements. Scale from: (a) our replica-1 vs replica-3 ratio once this run’s r=1 numbers exist, (b) tmpfs vs ZFS ratio (2.35× on 1p), (c) Synadia/nats bench async file r=1 ~100–400k on NVMe loopback, derated for 10GbE RTT. + +**Buy notes:** M.2 via Dual uFF / enablement kit; put JetStream on NVMe **directly**, not behind a RAID controller write-through unless you measure. 1GbE onboard is a trap — use 10GbE for `:6222`. Dual Gold is for isolation (nats vs worm/tree vs OS), not because JS needs 56 cores. + +--- + +## 5. What we are not doing + +- MQTT as the Zapier or middleware transport. +- UDP for jobs. +- Emulated 10G fiber NICs on LXC. +- tmpfs as the production store. +- r=1 for product streams. +- Connect-per-job. + +Re-run exhaustive: `bash scripts/exhaustive-ns1-study.sh` on NS1. diff --git a/packages/zapier-decisions/reports/optimal-config/REPORT.pdf b/packages/zapier-decisions/reports/optimal-config/REPORT.pdf new file mode 100644 index 0000000..513cc78 Binary files /dev/null and b/packages/zapier-decisions/reports/optimal-config/REPORT.pdf differ diff --git a/packages/zapier-decisions/reports/optimal-config/charts-optimal/history-js1p.png b/packages/zapier-decisions/reports/optimal-config/charts-optimal/history-js1p.png new file mode 100644 index 0000000..69de19e Binary files /dev/null and b/packages/zapier-decisions/reports/optimal-config/charts-optimal/history-js1p.png differ diff --git a/packages/zapier-decisions/reports/optimal-config/charts-optimal/replicas.png b/packages/zapier-decisions/reports/optimal-config/charts-optimal/replicas.png new file mode 100644 index 0000000..9779f10 Binary files /dev/null and b/packages/zapier-decisions/reports/optimal-config/charts-optimal/replicas.png differ diff --git a/scripts/build-docs-site.py b/scripts/build-docs-site.py index 5608c82..5cf7048 100755 --- a/scripts/build-docs-site.py +++ b/scripts/build-docs-site.py @@ -870,6 +870,10 @@ def main() -> None: "packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.pdf", "Maximized NS1 study: 8 cores, 16 GiB, tmpfs JetStream (delta)", ), + ( + "packages/zapier-decisions/reports/optimal-config/REPORT.pdf", + "Optimal NATS config study + DL360 Gen10 projection", + ), ("packages/verae-nats-cluster/BENCH.pdf", "NATS cluster throughput tables (2026-09-12)"), ("overview/README.pdf", "System overview"), ("overview/INDEX.pdf", "Documentation index"), @@ -895,6 +899,10 @@ def main() -> None: "packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.html", "Maximized NS1 study (8c/16G/tmpfs)", ), + ( + "packages/zapier-decisions/reports/optimal-config/REPORT.html", + "Optimal NATS config study + DL360 projection", + ), ("packages/verae-nats-cluster/BENCH.html", "NATS cluster throughput tables"), ("docs-master/MESSAGE-FLOWS.html", "Numbered message flows"), ("docs-master/modules-and-nats.html", "NATS address table"),