diff --git a/packages/verae-nats-cluster/README.md b/packages/verae-nats-cluster/README.md index 383ad04..b0500b6 100644 --- a/packages/verae-nats-cluster/README.md +++ b/packages/verae-nats-cluster/README.md @@ -46,8 +46,11 @@ nkeys/mTLS: `verae-nats-accounts` still has the INTERNAL/LEAF sketch. Do **not** From a **client that is not a nats-* server** (LXC 510 `verae-px-worker`): ```bash -# on NS1 +# on NS1 (hypervisor). Bench client is LXC 510. bash scripts/bench.sh + +# Full study that must run ON NS1 (orchestrator + charts + HTML/PDF): +bash scripts/study-on-ns1.sh ``` That writes `results//` and copies a summary to [BENCH.md](BENCH.md). Full write-up with charts (HTML + PDF) is in the progress repo: [nats-cluster-bench](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.pdf). Core NATS is fire-and-forget; JetStream rows use **file** storage and **replicas=3**. Ping delay is sequential RTT; flood delay is queueing under burst. diff --git a/packages/verae-nats-cluster/results/.gitignore b/packages/verae-nats-cluster/results/.gitignore index 1b2bd4d..d31cf0d 100644 --- a/packages/verae-nats-cluster/results/.gitignore +++ b/packages/verae-nats-cluster/results/.gitignore @@ -2,3 +2,5 @@ !.gitignore !20260912T045131Z/ !20260912T045131Z/** +!20260912T051237Z/ +!20260912T051237Z/** diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/BENCH.md b/packages/verae-nats-cluster/results/20260912T051237Z/BENCH.md new file mode 100644 index 0000000..9c58034 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/BENCH.md @@ -0,0 +1,48 @@ +# NATS cluster message speed + +Run **`20260912T051237Z`** (UTC). Client: LXC **510** `verae-px-worker` (`10.10.10.20`), not a nats-* server. Servers: `nats-a/b/c` on `10.10.10.21–23` (`vmbr1` only). + +Client URL: + +```text +nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222 +``` + +## Method + +- **Core NATS** is fire-and-forget pub/sub (`nats bench`). No disk, no replica ack. +- **JetStream** uses **file** storage and **replicas=3** (same as product streams). The unique stream `benchstream` is deleted between JS loads. +- Throughput is **msgs/sec** from nats CLI **0.1.6** (`--no-progress --csv`). Its min/avg/max are publisher **rate spread**, not delay. +- **Ping** delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster. +- **Flood** delay: N publishers dump the whole batch, then the subscriber drains. This is **queueing under burst**, not wire RTT. +- Probe: `scripts/latency.mjs` (two connections, header timestamp). + +## Throughput + +| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s | +|-----|------|------------------|------------|----------|------------|----------| +| `core-1p1s-50k-128` | core pub/sub | 776,331 | 502,502 | 61.34 | 420,976 | 51.39 | +| `core-4p4s-100k-128` | core pub/sub | 1,278,454 | 379,985 | 46.38 | 1,078,910 | 131.70 | +| `core-4p4s-50k-1k` | core pub/sub | 581,464 | 209,545 | 204.63 | 482,645 | 471.33 | +| `core-8p8s-200k-128` | core pub/sub | 2,065,217 | 333,731 | 40.74 | 1,847,242 | 225.49 | +| `js-2p2s-20k-128-r3` | jetstream r=3 file | 15,416 | — | — | — | — | +| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 14,985 | 14.63 | — | — | +| `js-4p-50k-128-r3` | jetstream r=3 file | — | 17,986 | 2.20 | — | — | + +## Round-trip delay + +| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max | +|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----| +| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.341ms | 0.470ms | 0.395ms | 0.651ms | 1.377ms | 3.243ms | +| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 132.425ms | 203.717ms | 204.458ms | 241.750ms | 249.468ms | 249.640ms | +| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 196.422ms | 221.264ms | 224.039ms | 233.180ms | 236.798ms | 238.177ms | +| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 211.108ms | 294.925ms | 300.254ms | 314.223ms | 315.551ms | 316.091ms | +| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 280.608ms | 414.972ms | 409.802ms | 513.821ms | 535.394ms | 536.084ms | + +## What the numbers mean + +Product job/event/archive traffic is **JetStream r=3 file**. On this three-LXC stand that is about **16k durable 128 B pubs/s** (about **13k** at 1 KiB). Core NATS is an upper bound for non-durable fan-out: about **0.7–2.0M msgs/s** aggregate at 128 B, or **~630k msgs/s (~616 MB/s)** at 1 KiB with 4 publishers. + +A quiet request-reply is **~0.3 ms** average, **p99 < 1 ms**. Flood rows in the **150–500 ms** band are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers. + +Re-run on NS1: `bash scripts/bench.sh`. Raw logs/CSVs are under `results//`. diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/charts/core-throughput.png b/packages/verae-nats-cluster/results/20260912T051237Z/charts/core-throughput.png new file mode 100644 index 0000000..44609c2 Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T051237Z/charts/core-throughput.png differ diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/charts/core-vs-js.png b/packages/verae-nats-cluster/results/20260912T051237Z/charts/core-vs-js.png new file mode 100644 index 0000000..dae2b58 Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T051237Z/charts/core-vs-js.png differ diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/charts/delay-percentiles.png b/packages/verae-nats-cluster/results/20260912T051237Z/charts/delay-percentiles.png new file mode 100644 index 0000000..f57cf44 Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T051237Z/charts/delay-percentiles.png differ diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/charts/js-throughput.png b/packages/verae-nats-cluster/results/20260912T051237Z/charts/js-throughput.png new file mode 100644 index 0000000..1edaebe Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T051237Z/charts/js-throughput.png differ diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/charts/payload-size.png b/packages/verae-nats-cluster/results/20260912T051237Z/charts/payload-size.png new file mode 100644 index 0000000..5c64356 Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T051237Z/charts/payload-size.png differ diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/core-1p1s-50k-128.csv b/packages/verae-nats-cluster/results/20260912T051237Z/core-1p1s-50k-128.csv new file mode 100644 index 0000000..41892fc --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/core-1p1s-50k-128.csv @@ -0,0 +1,3 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +h0r77ozVkEaFq9NntdbNXL,S0,50000,6400000,420976,53884945.437030,0.118772 +h0r77ozVkEaFq9NntdbNXL,P0,50000,6400000,502502,64320317.108811,0.099502 diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/core-1p1s-50k-128.txt b/packages/verae-nats-cluster/results/20260912T051237Z/core-1p1s-50k-128.txt new file mode 100644 index 0000000..943bab6 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/core-1p1s-50k-128.txt @@ -0,0 +1,7 @@ +=== core-1p1s-50k-128 === + +NATS Pub/Sub stats: 776,331 msgs/sec ~ 94.77 MB/sec + Pub stats: 502,502 msgs/sec ~ 61.34 MB/sec + Sub stats: 420,976 msgs/sec ~ 51.39 MB/sec + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/core-4p4s-100k-128.csv b/packages/verae-nats-cluster/results/20260912T051237Z/core-4p4s-100k-128.csv new file mode 100644 index 0000000..5938158 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/core-4p4s-100k-128.csv @@ -0,0 +1,9 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +9lnpijWl8uftGl7cjgjfDG,S0,100000,12800000,338319,43304923.645655,0.295578 +9lnpijWl8uftGl7cjgjfDG,S1,100000,12800000,328136,42001487.567998,0.304751 +9lnpijWl8uftGl7cjgjfDG,S2,100000,12800000,319449,40889502.964037,0.313039 +9lnpijWl8uftGl7cjgjfDG,S3,100000,12800000,269727,34525150.333968,0.370744 +9lnpijWl8uftGl7cjgjfDG,P0,25000,3200000,184132,23569014.120587,0.135771 +9lnpijWl8uftGl7cjgjfDG,P1,25000,3200000,183847,23532419.588601,0.135983 +9lnpijWl8uftGl7cjgjfDG,P2,25000,3200000,100400,12851250.530089,0.249003 +9lnpijWl8uftGl7cjgjfDG,P3,25000,3200000,95281,12196082.439191,0.262379 diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/core-4p4s-100k-128.txt b/packages/verae-nats-cluster/results/20260912T051237Z/core-4p4s-100k-128.txt new file mode 100644 index 0000000..773d5f7 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/core-4p4s-100k-128.txt @@ -0,0 +1,17 @@ +=== core-4p4s-100k-128 === + +NATS Pub/Sub stats: 1,278,454 msgs/sec ~ 156.06 MB/sec + Pub stats: 379,985 msgs/sec ~ 46.38 MB/sec + [1] 184,132 msgs/sec ~ 22.48 MB/sec (25000 msgs) + [2] 183,847 msgs/sec ~ 22.44 MB/sec (25000 msgs) + [3] 100,400 msgs/sec ~ 12.26 MB/sec (25000 msgs) + [4] 95,281 msgs/sec ~ 11.63 MB/sec (25000 msgs) + min 95,281 | avg 140,915 | max 184,132 | stddev 43,112 msgs + Sub stats: 1,078,910 msgs/sec ~ 131.70 MB/sec + [1] 338,319 msgs/sec ~ 41.30 MB/sec (100000 msgs) + [2] 328,136 msgs/sec ~ 40.06 MB/sec (100000 msgs) + [3] 319,449 msgs/sec ~ 39.00 MB/sec (100000 msgs) + [4] 269,727 msgs/sec ~ 32.93 MB/sec (100000 msgs) + min 269,727 | avg 313,907 | max 338,319 | stddev 26,367 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/core-4p4s-50k-1k.csv b/packages/verae-nats-cluster/results/20260912T051237Z/core-4p4s-50k-1k.csv new file mode 100644 index 0000000..ebabee0 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/core-4p4s-50k-1k.csv @@ -0,0 +1,9 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +TBolrRDh6R5ucnsIDRic4k,S0,50000,51200000,152508,156168868.728545,0.327850 +TBolrRDh6R5ucnsIDRic4k,S1,50000,51200000,143666,147114019.174715,0.348029 +TBolrRDh6R5ucnsIDRic4k,S2,50000,51200000,125714,128731399.676746,0.397727 +TBolrRDh6R5ucnsIDRic4k,S3,50000,51200000,123195,126152033.863128,0.405859 +TBolrRDh6R5ucnsIDRic4k,P0,12500,12800000,85005,87045293.746584,0.147050 +TBolrRDh6R5ucnsIDRic4k,P1,12500,12800000,84991,87030908.333567,0.147074 +TBolrRDh6R5ucnsIDRic4k,P2,12500,12800000,54739,56053364.168990,0.228354 +TBolrRDh6R5ucnsIDRic4k,P3,12500,12800000,52467,53726394.065295,0.238244 diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/core-4p4s-50k-1k.txt b/packages/verae-nats-cluster/results/20260912T051237Z/core-4p4s-50k-1k.txt new file mode 100644 index 0000000..c34c617 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/core-4p4s-50k-1k.txt @@ -0,0 +1,17 @@ +=== core-4p4s-50k-1k === + +NATS Pub/Sub stats: 581,464 msgs/sec ~ 567.84 MB/sec + Pub stats: 209,545 msgs/sec ~ 204.63 MB/sec + [1] 85,005 msgs/sec ~ 83.01 MB/sec (12500 msgs) + [2] 84,991 msgs/sec ~ 83.00 MB/sec (12500 msgs) + [3] 54,739 msgs/sec ~ 53.46 MB/sec (12500 msgs) + [4] 52,467 msgs/sec ~ 51.24 MB/sec (12500 msgs) + min 52,467 | avg 69,300 | max 85,005 | stddev 15,718 msgs + Sub stats: 482,645 msgs/sec ~ 471.33 MB/sec + [1] 152,508 msgs/sec ~ 148.93 MB/sec (50000 msgs) + [2] 143,666 msgs/sec ~ 140.30 MB/sec (50000 msgs) + [3] 125,714 msgs/sec ~ 122.77 MB/sec (50000 msgs) + [4] 123,195 msgs/sec ~ 120.31 MB/sec (50000 msgs) + min 123,195 | avg 136,270 | max 152,508 | stddev 12,255 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/core-8p8s-200k-128.csv b/packages/verae-nats-cluster/results/20260912T051237Z/core-8p8s-200k-128.csv new file mode 100644 index 0000000..4d6dc8b --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/core-8p8s-200k-128.csv @@ -0,0 +1,17 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +1wXmYbQfeZKgGIuRNRJWE9,S0,200000,25600000,294783,37732336.345174,0.678463 +1wXmYbQfeZKgGIuRNRJWE9,S1,200000,25600000,285956,36602476.046553,0.699406 +1wXmYbQfeZKgGIuRNRJWE9,S2,200000,25600000,270995,34687374.508938,0.738021 +1wXmYbQfeZKgGIuRNRJWE9,S3,200000,25600000,268481,34365602.744388,0.744931 +1wXmYbQfeZKgGIuRNRJWE9,S4,200000,25600000,242749,31071912.954289,0.823895 +1wXmYbQfeZKgGIuRNRJWE9,S5,200000,25600000,243757,31200898.690210,0.820489 +1wXmYbQfeZKgGIuRNRJWE9,S6,200000,25600000,232306,29735268.576473,0.860931 +1wXmYbQfeZKgGIuRNRJWE9,S7,200000,25600000,231301,29606645.928715,0.864671 +1wXmYbQfeZKgGIuRNRJWE9,P0,25000,3200000,161768,20706321.217329,0.154542 +1wXmYbQfeZKgGIuRNRJWE9,P1,25000,3200000,156790,20069204.386621,0.159448 +1wXmYbQfeZKgGIuRNRJWE9,P2,25000,3200000,101477,12989116.865479,0.246360 +1wXmYbQfeZKgGIuRNRJWE9,P3,25000,3200000,96840,12395608.110541,0.258156 +1wXmYbQfeZKgGIuRNRJWE9,P4,25000,3200000,48401,6195444.884723,0.516509 +1wXmYbQfeZKgGIuRNRJWE9,P5,25000,3200000,56695,7256990.291363,0.440954 +1wXmYbQfeZKgGIuRNRJWE9,P6,25000,3200000,42589,5451476.161959,0.586997 +1wXmYbQfeZKgGIuRNRJWE9,P7,25000,3200000,41987,5374439.609281,0.595411 diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/core-8p8s-200k-128.txt b/packages/verae-nats-cluster/results/20260912T051237Z/core-8p8s-200k-128.txt new file mode 100644 index 0000000..c825366 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/core-8p8s-200k-128.txt @@ -0,0 +1,25 @@ +=== core-8p8s-200k-128 === + +NATS Pub/Sub stats: 2,065,217 msgs/sec ~ 252.10 MB/sec + Pub stats: 333,731 msgs/sec ~ 40.74 MB/sec + [1] 161,768 msgs/sec ~ 19.75 MB/sec (25000 msgs) + [2] 156,790 msgs/sec ~ 19.14 MB/sec (25000 msgs) + [3] 101,477 msgs/sec ~ 12.39 MB/sec (25000 msgs) + [4] 96,840 msgs/sec ~ 11.82 MB/sec (25000 msgs) + [5] 48,401 msgs/sec ~ 5.91 MB/sec (25000 msgs) + [6] 56,695 msgs/sec ~ 6.92 MB/sec (25000 msgs) + [7] 42,589 msgs/sec ~ 5.20 MB/sec (25000 msgs) + [8] 41,987 msgs/sec ~ 5.13 MB/sec (25000 msgs) + min 41,987 | avg 88,318 | max 161,768 | stddev 46,314 msgs + Sub stats: 1,847,242 msgs/sec ~ 225.49 MB/sec + [1] 294,783 msgs/sec ~ 35.98 MB/sec (200000 msgs) + [2] 285,956 msgs/sec ~ 34.91 MB/sec (200000 msgs) + [3] 270,995 msgs/sec ~ 33.08 MB/sec (200000 msgs) + [4] 268,481 msgs/sec ~ 32.77 MB/sec (200000 msgs) + [5] 242,749 msgs/sec ~ 29.63 MB/sec (200000 msgs) + [6] 243,757 msgs/sec ~ 29.76 MB/sec (200000 msgs) + [7] 232,306 msgs/sec ~ 28.36 MB/sec (200000 msgs) + [8] 231,301 msgs/sec ~ 28.24 MB/sec (200000 msgs) + min 231,301 | avg 258,791 | max 294,783 | stddev 22,958 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/host-after.txt b/packages/verae-nats-cluster/results/20260912T051237Z/host-after.txt new file mode 100644 index 0000000..c9898c8 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/host-after.txt @@ -0,0 +1,86 @@ +execution_host=NS1.GEORGELAMBERT.ORG +execution_ip=70.88.205.138 +hostname=NS1.GEORGELAMBERT.ORG +utc=2026-09-12T05:14:00Z +whoami=marchon +pwd=/home/marchon/verae-src/verae-nats-cluster +uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux +nproc=40 +loadavg=8.67 8.33 8.15 9/3863 4114865 +client_vmid=510 +nats_url=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222 +--- free --- + total used free shared buff/cache available +Mem: 377Gi 38Gi 324Gi 4.3Gi 21Gi 339Gi +Swap: 8.0Gi 0B 8.0Gi +--- pct list --- +VMID Status Lock Name +100 stopped CT100 +101 stopped CT101 +102 stopped CT102 +103 stopped CT103 +104 stopped CT104 +105 stopped CT105 +106 stopped CT106 +107 stopped CT107 +108 stopped CT108 +109 stopped CT109 +110 stopped CT110 +111 stopped CT111 +112 stopped CT112 +113 stopped CT113 +114 stopped CT114 +115 stopped CT115 +116 stopped paul +117 stopped CT117 +118 stopped CT118 +119 stopped CT119 +120 stopped rustdeskserver +121 stopped CT121 +122 stopped CT122 +123 stopped CT123 +124 stopped bdross +200 stopped CT200 +201 stopped CT201 +202 stopped CT202 +300 running mail.veraetime.info +400 stopped evercycle-api +501 running pki +502 running peergos-a +503 running peergos-b +504 stopped minio +510 running verae-px-worker +511 running nats-a +512 running nats-b +513 running nats-c +2000 stopped CT2000 +2001 stopped CT2001 +2002 stopped CT2002 +--- pct config 510 --- +cores: 2 +hostname: verae-px-worker +memory: 2048 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:BD:5D:62,ip=10.10.10.20/24,type=veth +rootfs: SSD2:subvol-510-disk-0,size=16G +swap: 512 +--- pct config 511 --- +cores: 1 +hostname: nats-a +memory: 1024 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:CF:AC:68,ip=10.10.10.21/24,type=veth +rootfs: SSD2:subvol-511-disk-0,size=8G +swap: 256 +--- pct config 512 --- +cores: 1 +hostname: nats-b +memory: 1024 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:42:73:A2,ip=10.10.10.22/24,type=veth +rootfs: SSD2:subvol-512-disk-0,size=8G +swap: 256 +--- pct config 513 --- +cores: 1 +hostname: nats-c +memory: 1024 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:1B:9B:DC,ip=10.10.10.23/24,type=veth +rootfs: SSD2:subvol-513-disk-0,size=8G +swap: 256 diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/host-before.txt b/packages/verae-nats-cluster/results/20260912T051237Z/host-before.txt new file mode 100644 index 0000000..89ca238 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/host-before.txt @@ -0,0 +1,86 @@ +execution_host=NS1.GEORGELAMBERT.ORG +execution_ip=70.88.205.138 +hostname=NS1.GEORGELAMBERT.ORG +utc=2026-09-12T05:12:37Z +whoami=marchon +pwd=/home/marchon/verae-src/verae-nats-cluster +uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux +nproc=40 +loadavg=8.59 8.39 8.15 5/3843 4096277 +client_vmid=510 +nats_url=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222 +--- free --- + total used free shared buff/cache available +Mem: 377Gi 38Gi 324Gi 4.2Gi 21Gi 339Gi +Swap: 8.0Gi 0B 8.0Gi +--- pct list --- +VMID Status Lock Name +100 stopped CT100 +101 stopped CT101 +102 stopped CT102 +103 stopped CT103 +104 stopped CT104 +105 stopped CT105 +106 stopped CT106 +107 stopped CT107 +108 stopped CT108 +109 stopped CT109 +110 stopped CT110 +111 stopped CT111 +112 stopped CT112 +113 stopped CT113 +114 stopped CT114 +115 stopped CT115 +116 stopped paul +117 stopped CT117 +118 stopped CT118 +119 stopped CT119 +120 stopped rustdeskserver +121 stopped CT121 +122 stopped CT122 +123 stopped CT123 +124 stopped bdross +200 stopped CT200 +201 stopped CT201 +202 stopped CT202 +300 running mail.veraetime.info +400 stopped evercycle-api +501 running pki +502 running peergos-a +503 running peergos-b +504 stopped minio +510 running verae-px-worker +511 running nats-a +512 running nats-b +513 running nats-c +2000 stopped CT2000 +2001 stopped CT2001 +2002 stopped CT2002 +--- pct config 510 --- +cores: 2 +hostname: verae-px-worker +memory: 2048 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:BD:5D:62,ip=10.10.10.20/24,type=veth +rootfs: SSD2:subvol-510-disk-0,size=16G +swap: 512 +--- pct config 511 --- +cores: 1 +hostname: nats-a +memory: 1024 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:CF:AC:68,ip=10.10.10.21/24,type=veth +rootfs: SSD2:subvol-511-disk-0,size=8G +swap: 256 +--- pct config 512 --- +cores: 1 +hostname: nats-b +memory: 1024 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:42:73:A2,ip=10.10.10.22/24,type=veth +rootfs: SSD2:subvol-512-disk-0,size=8G +swap: 256 +--- pct config 513 --- +cores: 1 +hostname: nats-c +memory: 1024 +net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:1B:9B:DC,ip=10.10.10.23/24,type=veth +rootfs: SSD2:subvol-513-disk-0,size=8G +swap: 256 diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/js-1p-20k-128-r3.csv b/packages/verae-nats-cluster/results/20260912T051237Z/js-1p-20k-128-r3.csv new file mode 100644 index 0000000..9752dd6 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/js-1p-20k-128-r3.csv @@ -0,0 +1,2 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +8rEl4zHbyuPY6Hq62w9Qsw,P0,40000,2560000,7393,946342.174329,2.705153 diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/js-1p-20k-128-r3.txt b/packages/verae-nats-cluster/results/20260912T051237Z/js-1p-20k-128-r3.txt new file mode 100644 index 0000000..01e01af --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/js-1p-20k-128-r3.txt @@ -0,0 +1,5 @@ +=== js-1p-20k-128-r3 === + +Pub stats: 7,393 msgs/sec ~ 924.16 KB/sec + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/js-2p2s-20k-128-r3.csv b/packages/verae-nats-cluster/results/20260912T051237Z/js-2p2s-20k-128-r3.csv new file mode 100644 index 0000000..fa3f7fa --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/js-2p2s-20k-128-r3.csv @@ -0,0 +1,5 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +VI7cqfC8hjqPoKek3p2a06,S0,20150,1280000,4248,543810.859674,2.353760 +VI7cqfC8hjqPoKek3p2a06,S1,20159,1280000,3858,493875.492651,2.591746 +VI7cqfC8hjqPoKek3p2a06,P0,20000,1280000,11290,1445228.034916,0.885673 +VI7cqfC8hjqPoKek3p2a06,P1,20000,1280000,3864,494600.778891,2.587946 diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/js-2p2s-20k-128-r3.txt b/packages/verae-nats-cluster/results/20260912T051237Z/js-2p2s-20k-128-r3.txt new file mode 100644 index 0000000..bfcca29 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/js-2p2s-20k-128-r3.txt @@ -0,0 +1,13 @@ +=== js-2p2s-20k-128-r3 === + +NATS Pub/Sub stats: 15,416 msgs/sec ~ 1.88 MB/sec + Pub stats: 7,728 msgs/sec ~ 966.02 KB/sec + [1] 11,290 msgs/sec ~ 1.38 MB/sec (10000 msgs) + [2] 3,864 msgs/sec ~ 483.01 KB/sec (10000 msgs) + min 3,864 | avg 7,577 | max 11,290 | stddev 3,713 msgs + Sub stats: 7,711 msgs/sec ~ 963.88 KB/sec + [1] 4,248 msgs/sec ~ 531.07 KB/sec (10000 msgs) + [2] 3,858 msgs/sec ~ 482.30 KB/sec (10000 msgs) + min 3,858 | avg 4,053 | max 4,248 | stddev 195 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/js-4p-20k-1k-r3.csv b/packages/verae-nats-cluster/results/20260912T051237Z/js-4p-20k-1k-r3.csv new file mode 100644 index 0000000..4b305e7 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/js-4p-20k-1k-r3.csv @@ -0,0 +1,5 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +InppzTmEhsFOG1VYCh0Nle,P0,10000,5120000,4792,4907526.298214,1.043295 +InppzTmEhsFOG1VYCh0Nle,P1,10000,5120000,4480,4587592.638694,1.116054 +InppzTmEhsFOG1VYCh0Nle,P2,10000,5120000,3792,3883639.365215,1.318351 +InppzTmEhsFOG1VYCh0Nle,P3,10000,5120000,3750,3840425.475938,1.333186 diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/js-4p-20k-1k-r3.txt b/packages/verae-nats-cluster/results/20260912T051237Z/js-4p-20k-1k-r3.txt new file mode 100644 index 0000000..0e35ef9 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/js-4p-20k-1k-r3.txt @@ -0,0 +1,10 @@ +=== js-4p-20k-1k-r3 === + +Pub stats: 14,985 msgs/sec ~ 14.63 MB/sec + [1] 4,792 msgs/sec ~ 4.68 MB/sec (5000 msgs) + [2] 4,480 msgs/sec ~ 4.38 MB/sec (5000 msgs) + [3] 3,792 msgs/sec ~ 3.70 MB/sec (5000 msgs) + [4] 3,750 msgs/sec ~ 3.66 MB/sec (5000 msgs) + min 3,750 | avg 4,203 | max 4,792 | stddev 446 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/js-4p-50k-128-r3.csv b/packages/verae-nats-cluster/results/20260912T051237Z/js-4p-50k-128-r3.csv new file mode 100644 index 0000000..43e6f9a --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/js-4p-50k-128-r3.csv @@ -0,0 +1,5 @@ +#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs +JeTysfxECHCJccBaGH4xXX,P0,25000,1600000,5640,721943.982931,2.216238 +JeTysfxECHCJccBaGH4xXX,P1,25000,1600000,5642,722187.413742,2.215491 +JeTysfxECHCJccBaGH4xXX,P2,25000,1600000,5056,647242.012627,2.472027 +JeTysfxECHCJccBaGH4xXX,P3,25000,1600000,4497,575730.101666,2.779080 diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/js-4p-50k-128-r3.txt b/packages/verae-nats-cluster/results/20260912T051237Z/js-4p-50k-128-r3.txt new file mode 100644 index 0000000..676949a --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/js-4p-50k-128-r3.txt @@ -0,0 +1,10 @@ +=== js-4p-50k-128-r3 === + +Pub stats: 17,986 msgs/sec ~ 2.20 MB/sec + [1] 5,640 msgs/sec ~ 705.02 KB/sec (12500 msgs) + [2] 5,642 msgs/sec ~ 705.26 KB/sec (12500 msgs) + [3] 5,056 msgs/sec ~ 632.07 KB/sec (12500 msgs) + [4] 4,497 msgs/sec ~ 562.24 KB/sec (12500 msgs) + min 4,497 | avg 5,208 | max 5,642 | stddev 475 msgs + +Saved metric data in csv file /tmp/bench.csv diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/lat-1p-5k-128.txt b/packages/verae-nats-cluster/results/20260912T051237Z/lat-1p-5k-128.txt new file mode 100644 index 0000000..709e775 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/lat-1p-5k-128.txt @@ -0,0 +1,2 @@ +=== lat-1p-5k-128 === +{"count":5000,"pubs":1,"size":128,"mode":"flood","min_us":132425,"avg_us":203717,"p50_us":204458,"p90_us":241750,"p99_us":249468,"max_us":249640,"min":"132.425ms","avg":"203.717ms","p50":"204.458ms","p90":"241.750ms","p99":"249.468ms","max":"249.640ms"} diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/lat-4p-10k-128.txt b/packages/verae-nats-cluster/results/20260912T051237Z/lat-4p-10k-128.txt new file mode 100644 index 0000000..fefb67c --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/lat-4p-10k-128.txt @@ -0,0 +1,2 @@ +=== lat-4p-10k-128 === +{"count":10000,"pubs":4,"size":128,"mode":"flood","min_us":211108,"avg_us":294925,"p50_us":300254,"p90_us":314223,"p99_us":315551,"max_us":316091,"min":"211.108ms","avg":"294.925ms","p50":"300.254ms","p90":"314.223ms","p99":"315.551ms","max":"316.091ms"} diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/lat-4p-5k-1k.txt b/packages/verae-nats-cluster/results/20260912T051237Z/lat-4p-5k-1k.txt new file mode 100644 index 0000000..4464289 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/lat-4p-5k-1k.txt @@ -0,0 +1,2 @@ +=== lat-4p-5k-1k === +{"count":5000,"pubs":4,"size":1024,"mode":"flood","min_us":196422,"avg_us":221264,"p50_us":224039,"p90_us":233180,"p99_us":236798,"max_us":238177,"min":"196.422ms","avg":"221.264ms","p50":"224.039ms","p90":"233.180ms","p99":"236.798ms","max":"238.177ms"} diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/lat-8p-20k-128.txt b/packages/verae-nats-cluster/results/20260912T051237Z/lat-8p-20k-128.txt new file mode 100644 index 0000000..04dc29d --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/lat-8p-20k-128.txt @@ -0,0 +1,2 @@ +=== lat-8p-20k-128 === +{"count":20000,"pubs":8,"size":128,"mode":"flood","min_us":280608,"avg_us":414972,"p50_us":409802,"p90_us":513821,"p99_us":535394,"max_us":536084,"min":"280.608ms","avg":"414.972ms","p50":"409.802ms","p90":"513.821ms","p99":"535.394ms","max":"536.084ms"} diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/lat-ping-1k-128.txt b/packages/verae-nats-cluster/results/20260912T051237Z/lat-ping-1k-128.txt new file mode 100644 index 0000000..c9bca7f --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/lat-ping-1k-128.txt @@ -0,0 +1,2 @@ +=== lat-ping-1k-128 === +{"count":1000,"pubs":1,"size":128,"mode":"ping","min_us":341,"avg_us":470,"p50_us":395,"p90_us":651,"p99_us":1377,"max_us":3243,"min":"0.341ms","avg":"0.470ms","p50":"0.395ms","p90":"0.651ms","p99":"1.377ms","max":"3.243ms"} diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/nats-cluster-bench-ns1.html b/packages/verae-nats-cluster/results/20260912T051237Z/nats-cluster-bench-ns1.html new file mode 100644 index 0000000..59b896e --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/nats-cluster-bench-ns1.html @@ -0,0 +1,1291 @@ + + + + + + + NATS cluster message speed — NS1 host study + + + + +
Verae Time × Zapier · progress report · run on NS1.GEORGELAMBERT.ORG

NATS cluster message speed — NS1 host study

packages/zapier-decisions/reports/nats-cluster-bench-ns1.md
+
+

NATS cluster message speed — NS1 host study

+
+

Progress report (second study) · run +20260912T051237Z (UTC)

+
+

Execution provenance. Every process for this study +ran on NS1.GEORGELAMBERT.ORG +(70.88.205.138): the orchestrator +(study-on-ns1.sh), nats bench, +latency.mjs (inside LXC 510 on this hypervisor), charting +(matplotlib), and HTML/PDF (pandoc + +weasyprint). The operator laptop did not +publish, subscribe, draw charts, or render the PDF. Traffic stayed on +vmbr1 from LXC 510 to +nats-a/b/c (511–513).

+
+

This is a full methodology write-up plus the numbers from that +on-host run. The earlier report (nats-cluster-bench, run +20260912T045131Z) used the same cluster but was +orchestrated and rendered off-box. Use this document +when you need “it was all run on 138.”

+
+

1. Executive summary

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ItemThis NS1-host run
Control planeNS1.GEORGELAMBERT.ORG (70.88.205.138), user +marchon
Bench clientLXC 510 verae-px-worker
BrokersLXC 511/512/513 nats-a/b/c on +10.10.10.21–23
Client URLnats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
Host load before8.59 8.39 8.15 5/3843 4096277
Host load after8.67 8.33 8.15 9/3863 4114865
Core 1p1s 128 B pub502,502 msgs/s
JetStream 1p 128 B r=37,393 durable pubs/s
Ping p50 / p990.395ms / 1.377ms
+

Product traffic is the JetStream row. Ping is one-message delay. +Flood is mailbox catch-up after a burst.

+
+

2. Where it ran (and where it +did not)

+
Operator laptop  ──ssh──►  NS1.GEORGELAMBERT.ORG  70.88.205.138
+                              study-on-ns1.sh
+                              python3 build-ns1-study-report.py
+                              sudo pct exec 510 ──► nats bench / latency.mjs
+                                                    │
+                                                    ▼ vmbr1
+                                              10.10.10.21-23 :4222
+ +
+

3. Results (this run)

+

Host and brokers

+

Before

+ +++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NodeVMIDconnectionsin_msgsout_msgscpucoresmem (B)jetstream
nats-a51121,721,2034,933,7591126,398,720True
nats-b51221,613,2053,513,0960123,097,344True
nats-c51321,760,9734,048,5822123,867,392True
+

After

+ +++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NodeVMIDconnectionsin_msgsout_msgscpucoresmem (B)jetstream
nats-a51122,097,5965,910,1611138,273,024True
nats-b51222,121,8164,746,5391141,361,408True
nats-c51322,241,3665,053,9781138,162,432True
+

nproc=40 · +uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux

+

Throughput

+ +++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunModeAggregate msgs/sPub msgs/sPub MB/sSub msgs/sSub MB/s
core-1p1s-50k-128core pub/sub776,331502,50261.34420,97651.39
core-4p4s-100k-128core pub/sub1,278,454379,98546.381,078,910131.70
core-4p4s-50k-1kcore pub/sub581,464209,545204.63482,645471.33
core-8p8s-200k-128core pub/sub2,065,217333,73140.741,847,242225.49
js-1p-20k-128-r3jetstream r=3 file—7,3930.90——
js-2p2s-20k-128-r3jetstream r=3 file15,4167,7280.947,7110.94
js-4p-20k-1k-r3jetstream r=3 file—14,98514.63——
js-4p-50k-128-r3jetstream r=3 file—17,9862.20——
+

Round-trip delay

+ +++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunKindCountPubsSizeminavgp50p90p99max
lat-ping-1k-128ping (sequential RTT)10001128 B0.341ms0.470ms0.395ms0.651ms1.377ms3.243ms
lat-1p-5k-128flood (burst queueing)50001128 B132.425ms203.717ms204.458ms241.750ms249.468ms249.640ms
lat-4p-5k-1kflood (burst queueing)500041024 B196.422ms221.264ms224.039ms233.180ms236.798ms238.177ms
lat-4p-10k-128flood (burst queueing)100004128 B211.108ms294.925ms300.254ms314.223ms315.551ms316.091ms
lat-8p-20k-128flood (burst queueing)200008128 B280.608ms414.972ms409.802ms513.821ms535.394ms536.084ms
+

Core NATS

+
+ + +
+

Core NATS throughput at four loads (NS1 host run) ### +Payload size (core)

+
+ + +
+

Core NATS 128 B vs 1 KiB (NS1 host run) ### JetStream r=3 +file

+
+ + +
+

JetStream durable publish rate (NS1 host run) ### Core vs +JetStream

+
+ + +
+

Core vs JetStream publish rate, log scale (NS1 host run) ### +Delay

+
+ + +
+

Ping vs flood delay percentiles, log scale (NS1 host +run)

+
+

4. Study methodology

+

4.1 Question

+

On the NS1 test stand, what message throughput and +delay does the three-node verae JetStream +cluster deliver at several loads, and which part of the stack is the +limiter for product traffic (jobs, events, webhooks, archive)?

+

4.2 Hypotheses (stated before +the run)

+
    +
  1. H1 — Core vs JetStream. Fire-and-forget core NATS +is at least an order of magnitude faster than JetStream file + +replicas=3, because durable publish waits for a majority disk +replica.
    2. +
  2. H2 — JetStream parallelism. Adding publishers does +not linearly increase JetStream write rate once the +replica log is saturated.
    3. +
  3. H3 — Quiet delay. Sequential pub→sub round trip on +vmbr1 is well under 1 ms p99 when the consumer is +waiting.
    4. +
  4. H4 — Burst delay. If publishers dump a batch before +the subscriber drains, observed delay is queueing time, +roughly linear in backlog, not in cluster hop count.
    5. +
  5. H5 — Payload. Moving 128 B → 1 KiB lowers message +rate and raises byte rate on core NATS; JetStream in this size band +stays replica/fsync bound.
    6. +
+

4.3 Independent variables +(what we changed)

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FactorLevels
TransportCore NATS pub/sub vs JetStream file replicas=3
Publisher count1, 2, 4, 8
Subscriber count0 (JS publish-only), 1, 2, 4, 8
Message count1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step)
Payload128 B, 1024 B
Delay modeping (publish, wait, repeat) vs +flood (publish all, then drain)
+

4.4 Dependent variables +(what we recorded)

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MetricInstrumentUnit
Publish ratenats bench 0.1.6 Pub statsmsgs/s, MB/s
Subscribe ratenats bench Sub statsmsgs/s, MB/s
Aggregatenats bench NATS Pub/Sub statsmsgs/s (fan-out counts both sides)
Publisher spreadnats min/avg/max msgs/snot delay
One-way-ish RTTlatency.mjs header timestampmin, avg, p50, p90, p99, max
Host load/proc/loadavg before and afterload average
Broker countershttp://127.0.0.1:8222/varz inside each nats LXCconnections, in/out msgs, cpu, mem
+

Important: nats CLI 0.1.6 min/avg/max are +rate spread across publishers, not microseconds of +delay. Delay is only latency.mjs.

+

4.5 Controls and constants

+ +

4.6 Procedure

+
    +
  1. Confirm this script is executing on +NS1.GEORGELAMBERT.ORG. Refuse otherwise.
    2. +
  2. Snapshot host load, memory, LXC configs, and each nats +varz.
    3. +
  3. From NS1, pct exec 510 the core ladder (1p1s, 4p4s, +8p8s at 128 B; 4p4s at 1 KiB).
    4. +
  4. Delete benchstream; JS ladder (1p, 4p, 4p×1 KiB, 2p2s +pull) at replicas=3 file.
    5. +
  5. Copy latency.mjs into 510; ping then flood at several +batch sizes.
    6. +
  6. Snapshot host/varz again.
    7. +
  7. Parse logs on this host; draw charts; write HTML +and PDF on this host.
    8. +
+

No publish, subscribe, chart, or PDF process runs on the operator +laptop for this study.

+

4.7 Instrumentation path

+
[NS1 host 70.88.205.138]
+   study-on-ns1.sh  (bash + python3)
+        |
+        | sudo pct exec 510
+        v
+[LXC 510 verae-px-worker 10.10.10.20]
+   nats bench / node latency.mjs
+        |
+        | NATS client protocol to
+        v
+[LXC 511/512/513  10.10.10.21-23 :4222]
+   nats-server -js  cluster routes :6222
+

The hypervisor issues the guest commands. The messages themselves +never leave vmbr1.

+

4.8 Threats to validity

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ThreatEffect on numbers
One physical hostThree “replicas” share CPU, memory, and usually the same datastore. +This measures process/LXC HA, not disk HA.
Shared loadNS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. +Load average during a run is part of the result, not noise to +ignore.
Single bench clientAll publishers live in 510. Per-publisher rate spread is contention +in that guest.
Short runsSeconds of traffic. No compaction, no multi-hour page-cache +eviction, no snapshot during load.
No TLS/nkeysProduction auth will cost CPU. Do not treat these rates as +post-nkeys rates.
Fan-out aggregateCore aggregate msgs/s counts pub+sub. Do not compare that column to +JetStream unique writes.
Flood ≠ RTTMixing flood averages with ping p99 produces a fake “NATS is slow” +story.
Lab onlyNot a Zapier HTTPS bench and not live +api.veraetime.net.
+

4.9 Ethics / safety

+

Bench uses throwaway subjects (bench.core.*, +bench.js.*, bench.lat.*) and a throwaway +stream. It does not purge product streams. Zapier cloud has no NATS +socket.

+
+

5. Suggestions for fine-tuning

+

These follow from the method and from the first ladder on this stand +(JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of +ms). Apply in order of leverage. Re-run this NS1 study +after each change so the delta is measured the same way.

+

5.1 Treat JetStream as +the product limiter

+

Product jobs/events/webhooks/archive are durable. Tuning core NATS to +2M msgs/s will not move a timestamp Zap. Put effort into replica +write path and consumer lag, not core +fan-out.

+

5.2 Split storage class by +stream

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
StreamSuggested storeWhy
ZAPIER_JOBSfile, r=3Work queue; lose-a-job is bad
ZAPIER_EVENTSfile r=3, or memory r=3 if events are rebuildable from job +statusHot waiters; measure both
ZAPIER_WEBHOOKSfile, r=3, workqueueHTTPS to Zapier is the slow consumer
ZAPIER_USAGEfile, r=3, limits + max-ageTelemetry
VERAE_ARCHIVEfile, r=3, on the best diskPuts are larger and must survive
+

Try ZAPIER_EVENTS as memory store in a maintenance +window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms +and durable events still ack at a higher rate, keep it; if a CT restart +drops in-flight waiters, revert.

+

5.3 Give JetStream real disks

+

Today r=3 on three LXC guests on one Proxmox host is +three files, one failure domain.

+ +

5.4 Isolate the nats +CTs from the rest of NS1

+

Host load on this box is often already several. Pin:

+ +

Re-run this study after pinning; H1/H2 should move more than +ping.

+

5.5 Consumer and mailbox +tuning (delay H4)

+

Flood delay is backlog / consume_rate. Fine-tune the +waiters, not the broker RTT.

+ +

5.6 Publisher-side +batching in middleware

+

A timestamp job is one small JSON. 16k msgs/s is ample. Still:

+ +

5.7 +nats-server knobs worth measuring (A/B with this script)

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
KnobWhy try it
max_payloadKeep default unless archive puts grow
write_deadlineSlow consumer protection for webhooks
max_pendingBound memory on a stuck Zapier hook
max_connectionsFleet workers + keep + middleware
JetStream max_file_store / +max_memory_storePrevent one stream from filling the CT
max_outstanding_catchupReplica restart after a nats-c blip
GOMAXPROCS = LXC coresDo not overthread a 2-core CT
+

Change one knob, re-run +study-on-ns1.sh, compare JetStream 1p 128 B and ping +p99.

+

5.8 Network

+ +

5.9 Security cost (when +nkeys/mTLS flip)

+

verae-nats-accounts is still a sketch. Enabling accounts +will add CPU on publish. Budget: re-run this exact study +after creds are in every NATS_URL, and +accept a drop on both core and JS. Do not flip without that +measurement.

+

5.10 Operational +fine-tuning (lag, not peak msgs/s)

+
    +
  1. Scrape varz / jsz from the host over +vmbr1 (not public). Monitor loopback :8222 is +invisible to Prometheus on NS1 unless we add a host-side proxy on +10.10.10.21:8222 bound only to vmbr1.
    2. +
  2. Keep replica floors for webhook-deliver and job-poller — they are +the flood defense.
    3. +
  3. Backup/restore drill of JetStream during idle, then +a short JS 1p run to see catchup cost.
    4. +
  4. A 15–30 minute soak (not in this ladder) for page cache and +compaction; add that as a third study when disks are dedicated.
    5. +
+

5.11 What not to tune

+ + +
    +
  1. CPU pin nats-a/b/c → re-run JS 1p + ping.
    2. +
  2. ZAPIER_EVENTS-shaped memory stream vs file (throwaway +stream, same flags as this JS ladder).
    3. +
  3. Distinct store_dir disks per node.
    4. +
  4. nkeys on, same ladder.
    5. +
  5. Three hardware boxes, same cluster.env IPs +updated.
    6. +
+

Each experiment should produce a new +results/<utc>/ on NS1 and a new progress-repo report +so we can diff H1–H5 instead of arguing from memory.

+
+

6. Reproducing this study

+

On NS1 only:

+
cd ~/verae-src/verae-nats-cluster
+bash scripts/study-on-ns1.sh
+

The script exits if hostname is not NS1. Outputs land in +results/<utc>/ including +nats-cluster-bench-ns1.{md,html,pdf} and +charts/. Copy those into +zapier-decisions/reports/ for the progress repo and +catalog.

+

Raw logs for this run: results/20260912T051237Z/.

+ + diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/nats-cluster-bench-ns1.md b/packages/verae-nats-cluster/results/20260912T051237Z/nats-cluster-bench-ns1.md new file mode 100644 index 0000000..cf0011d --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/nats-cluster-bench-ns1.md @@ -0,0 +1,334 @@ +**Progress report (second study)** · run `20260912T051237Z` (UTC) + +> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): the orchestrator (`study-on-ns1.sh`), `nats bench`, `latency.mjs` (inside LXC 510 on this hypervisor), charting (`matplotlib`), and HTML/PDF (`pandoc` + `weasyprint`). The operator laptop did **not** publish, subscribe, draw charts, or render the PDF. Traffic stayed on `vmbr1` from LXC **510** to `nats-a/b/c` (**511–513**). + +This is a full methodology write-up plus the numbers from that on-host run. The earlier report (`nats-cluster-bench`, run `20260912T045131Z`) used the same cluster but was **orchestrated and rendered off-box**. Use this document when you need “it was all run on 138.” + +--- + +## 1. Executive summary + +| Item | This NS1-host run | +|------|-------------------| +| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `marchon` | +| Bench client | LXC 510 `verae-px-worker` | +| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` | +| Client URL | `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` | +| Host load before | `8.59 8.39 8.15 5/3843 4096277` | +| Host load after | `8.67 8.33 8.15 9/3863 4114865` | +| Core 1p1s 128 B pub | 502,502 msgs/s | +| JetStream 1p 128 B r=3 | 7,393 durable pubs/s | +| Ping p50 / p99 | 0.395ms / 1.377ms | + +Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst. + +--- + +## 2. Where it ran (and where it did not) + +```text +Operator laptop ──ssh──► NS1.GEORGELAMBERT.ORG 70.88.205.138 + study-on-ns1.sh + python3 build-ns1-study-report.py + sudo pct exec 510 ──► nats bench / latency.mjs + │ + ▼ vmbr1 + 10.10.10.21-23 :4222 +``` + +- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510). +- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file. + +--- + +## 3. Results (this run) + +### Host and brokers + +**Before** + +| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream | +|------|------|-------------|---------|----------|-----|-------|---------|-----------| +| nats-a | 511 | 2 | 1,721,203 | 4,933,759 | 1 | 1 | 26,398,720 | True | +| nats-b | 512 | 2 | 1,613,205 | 3,513,096 | 0 | 1 | 23,097,344 | True | +| nats-c | 513 | 2 | 1,760,973 | 4,048,582 | 2 | 1 | 23,867,392 | True | + +**After** + +| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream | +|------|------|-------------|---------|----------|-----|-------|---------|-----------| +| nats-a | 511 | 2 | 2,097,596 | 5,910,161 | 1 | 1 | 38,273,024 | True | +| nats-b | 512 | 2 | 2,121,816 | 4,746,539 | 1 | 1 | 41,361,408 | True | +| nats-c | 513 | 2 | 2,241,366 | 5,053,978 | 1 | 1 | 38,162,432 | True | + +nproc=40 · uname=`Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux` + +### Throughput + +| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s | +|-----|------|------------------|------------|----------|------------|----------| +| `core-1p1s-50k-128` | core pub/sub | 776,331 | 502,502 | 61.34 | 420,976 | 51.39 | +| `core-4p4s-100k-128` | core pub/sub | 1,278,454 | 379,985 | 46.38 | 1,078,910 | 131.70 | +| `core-4p4s-50k-1k` | core pub/sub | 581,464 | 209,545 | 204.63 | 482,645 | 471.33 | +| `core-8p8s-200k-128` | core pub/sub | 2,065,217 | 333,731 | 40.74 | 1,847,242 | 225.49 | +| `js-1p-20k-128-r3` | jetstream r=3 file | — | 7,393 | 0.90 | — | — | +| `js-2p2s-20k-128-r3` | jetstream r=3 file | 15,416 | 7,728 | 0.94 | 7,711 | 0.94 | +| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 14,985 | 14.63 | — | — | +| `js-4p-50k-128-r3` | jetstream r=3 file | — | 17,986 | 2.20 | — | — | + +### Round-trip delay + +| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max | +|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----| +| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.341ms | 0.470ms | 0.395ms | 0.651ms | 1.377ms | 3.243ms | +| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 132.425ms | 203.717ms | 204.458ms | 241.750ms | 249.468ms | 249.640ms | +| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 196.422ms | 221.264ms | 224.039ms | 233.180ms | 236.798ms | 238.177ms | +| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 211.108ms | 294.925ms | 300.254ms | 314.223ms | 315.551ms | 316.091ms | +| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 280.608ms | 414.972ms | 409.802ms | 513.821ms | 535.394ms | 536.084ms | + +### Core NATS + +![Core NATS throughput at four loads (NS1 host run)](charts/core-throughput.png) + +*Core NATS throughput at four loads (NS1 host run)* +### Payload size (core) + +![Core NATS 128 B vs 1 KiB (NS1 host run)](charts/payload-size.png) + +*Core NATS 128 B vs 1 KiB (NS1 host run)* +### JetStream r=3 file + +![JetStream durable publish rate (NS1 host run)](charts/js-throughput.png) + +*JetStream durable publish rate (NS1 host run)* +### Core vs JetStream + +![Core vs JetStream publish rate, log scale (NS1 host run)](charts/core-vs-js.png) + +*Core vs JetStream publish rate, log scale (NS1 host run)* +### Delay + +![Ping vs flood delay percentiles, log scale (NS1 host run)](charts/delay-percentiles.png) + +*Ping vs flood delay percentiles, log scale (NS1 host run)* + +--- + +## 4. Study methodology + +### 4.1 Question + +On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)? + +### 4.2 Hypotheses (stated before the run) + +1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica. +2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated. +3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting. +4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count. +5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound. + +### 4.3 Independent variables (what we changed) + +| Factor | Levels | +|--------|--------| +| Transport | Core NATS pub/sub vs JetStream file replicas=3 | +| Publisher count | 1, 2, 4, 8 | +| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 | +| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) | +| Payload | 128 B, 1024 B | +| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) | + +### 4.4 Dependent variables (what we recorded) + +| Metric | Instrument | Unit | +|--------|------------|------| +| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s | +| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s | +| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) | +| Publisher spread | nats min/avg/max **msgs/s** | not delay | +| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max | +| Host load | `/proc/loadavg` before and after | load average | +| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem | + +**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`. + +### 4.5 Controls and constants + +- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`. +- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`). +- Bench client is LXC **510**, not a nats-* server. +- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide. +- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`). +- No TLS, no nkeys, no account isolation (isolation is `vmbr1`). +- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder. + +### 4.6 Procedure + +1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise. +2. Snapshot host load, memory, LXC configs, and each nats `varz`. +3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB). +4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file. +5. Copy `latency.mjs` into 510; ping then flood at several batch sizes. +6. Snapshot host/`varz` again. +7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**. + +No publish, subscribe, chart, or PDF process runs on the operator laptop for this study. + +### 4.7 Instrumentation path + +```text +[NS1 host 70.88.205.138] + study-on-ns1.sh (bash + python3) + | + | sudo pct exec 510 + v +[LXC 510 verae-px-worker 10.10.10.20] + nats bench / node latency.mjs + | + | NATS client protocol to + v +[LXC 511/512/513 10.10.10.21-23 :4222] + nats-server -js cluster routes :6222 +``` + +The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`. + +### 4.8 Threats to validity + +| Threat | Effect on numbers | +|--------|-------------------| +| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. | +| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. | +| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. | +| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. | +| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. | +| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. | +| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. | +| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. | + +### 4.9 Ethics / safety + +Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket. + +--- + +## 5. Suggestions for fine-tuning + +These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way. + +### 5.1 Treat JetStream as the product limiter + +Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out. + +### 5.2 Split storage class by stream + +| Stream | Suggested store | Why | +|--------|-----------------|-----| +| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad | +| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both | +| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer | +| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry | +| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive | + +Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert. + +### 5.3 Give JetStream real disks + +Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain. + +- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`. +- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure. +- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it. +- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.” + +### 5.4 Isolate the nats CTs from the rest of NS1 + +Host load on this box is often already several. Pin: + +- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes. +- Memory high enough that file-backed streams stay cache-hot for the working set. +- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync. + +Re-run this study after pinning; H1/H2 should move more than ping. + +### 5.5 Consumer and mailbox tuning (delay H4) + +Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT. + +- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM. +- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber. +- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped. +- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT. + +### 5.6 Publisher-side batching in middleware + +A timestamp job is one small JSON. 16k msgs/s is ample. Still: + +- Avoid per-byte publishes; one message per job/event. +- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs). +- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only. + +### 5.7 nats-server knobs worth measuring (A/B with this script) + +| Knob | Why try it | +|------|------------| +| `max_payload` | Keep default unless archive puts grow | +| `write_deadline` | Slow consumer protection for webhooks | +| `max_pending` | Bound memory on a stuck Zapier hook | +| `max_connections` | Fleet workers + keep + middleware | +| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT | +| `max_outstanding_catchup` | Replica restart after a nats-c blip | +| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT | + +Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99. + +### 5.8 Network + +- Keep NATS off `vmbr0`. No change. +- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load). +- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus. + +### 5.9 Security cost (when nkeys/mTLS flip) + +`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement. + +### 5.10 Operational fine-tuning (lag, not peak msgs/s) + +1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`. +2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense. +3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost. +4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated. + +### 5.11 What not to tune + +- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge. +- Do not treat flood 400 ms as “cluster RTT.” Fix consumers. +- Do not load-test on `ZAPIER_*` streams. +- Do not bind client NATS to `0.0.0.0` on `vmbr0`. + +### 5.12 Recommended next experiments (same method, one change each) + +1. CPU pin nats-a/b/c → re-run JS 1p + ping. +2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder). +3. Distinct `store_dir` disks per node. +4. nkeys on, same ladder. +5. Three hardware boxes, same `cluster.env` IPs updated. + +Each experiment should produce a new `results//` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory. + + +--- + +## 6. Reproducing this study + +On **NS1 only**: + +```bash +cd ~/verae-src/verae-nats-cluster +bash scripts/study-on-ns1.sh +``` + +The script exits if `hostname` is not NS1. Outputs land in `results//` including `nats-cluster-bench-ns1.{md,html,pdf}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog. + +Raw logs for this run: `results/20260912T051237Z/`. diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/nats-cluster-bench-ns1.pdf b/packages/verae-nats-cluster/results/20260912T051237Z/nats-cluster-bench-ns1.pdf new file mode 100644 index 0000000..c25d95e Binary files /dev/null and b/packages/verae-nats-cluster/results/20260912T051237Z/nats-cluster-bench-ns1.pdf differ diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/varz-after.json b/packages/verae-nats-cluster/results/20260912T051237Z/varz-after.json new file mode 100644 index 0000000..c55309c --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/varz-after.json @@ -0,0 +1,50 @@ +[ + { + "vmid": "511", + "name": "nats-a", + "server_name": "nats-a", + "host": "10.10.10.21", + "port": 4222, + "connections": 2, + "in_msgs": 2097596, + "out_msgs": 5910161, + "in_bytes": 559427613, + "out_bytes": 1494241553, + "cpu": 1, + "cores": 1, + "mem": 38273024, + "jetstream": true + }, + { + "vmid": "512", + "name": "nats-b", + "server_name": "nats-b", + "host": "10.10.10.22", + "port": 4222, + "connections": 2, + "in_msgs": 2121816, + "out_msgs": 4746539, + "in_bytes": 585390797, + "out_bytes": 974695990, + "cpu": 1, + "cores": 1, + "mem": 41361408, + "jetstream": true + }, + { + "vmid": "513", + "name": "nats-c", + "server_name": "nats-c", + "host": "10.10.10.23", + "port": 4222, + "connections": 2, + "in_msgs": 2241366, + "out_msgs": 5053978, + "in_bytes": 586858467, + "out_bytes": 1040486148, + "cpu": 1, + "cores": 1, + "mem": 38162432, + "jetstream": true + } +] diff --git a/packages/verae-nats-cluster/results/20260912T051237Z/varz-before.json b/packages/verae-nats-cluster/results/20260912T051237Z/varz-before.json new file mode 100644 index 0000000..945da17 --- /dev/null +++ b/packages/verae-nats-cluster/results/20260912T051237Z/varz-before.json @@ -0,0 +1,50 @@ +[ + { + "vmid": "511", + "name": "nats-a", + "server_name": "nats-a", + "host": "10.10.10.21", + "port": 4222, + "connections": 2, + "in_msgs": 1721203, + "out_msgs": 4933759, + "in_bytes": 440427700, + "out_bytes": 1169323990, + "cpu": 1, + "cores": 1, + "mem": 26398720, + "jetstream": true + }, + { + "vmid": "512", + "name": "nats-b", + "server_name": "nats-b", + "host": "10.10.10.22", + "port": 4222, + "connections": 2, + "in_msgs": 1613205, + "out_msgs": 3513096, + "in_bytes": 428640182, + "out_bytes": 715428083, + "cpu": 0, + "cores": 1, + "mem": 23097344, + "jetstream": true + }, + { + "vmid": "513", + "name": "nats-c", + "server_name": "nats-c", + "host": "10.10.10.23", + "port": 4222, + "connections": 2, + "in_msgs": 1760973, + "out_msgs": 4048582, + "in_bytes": 447844308, + "out_bytes": 864199465, + "cpu": 2, + "cores": 1, + "mem": 23867392, + "jetstream": true + } +] diff --git a/packages/verae-nats-cluster/scripts/bench-report.py b/packages/verae-nats-cluster/scripts/bench-report.py index f3f1f5c..68446d5 100755 --- a/packages/verae-nats-cluster/scripts/bench-report.py +++ b/packages/verae-nats-cluster/scripts/bench-report.py @@ -16,18 +16,34 @@ def fmt_int(s: str | None) -> str: def parse_bench(text: str) -> dict[str, str]: out: dict[str, str] = {"kind": "throughput"} - m = re.search(r"(?m)^\s*Pub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*MB/sec", text) - if m: - out["pub_msgs"] = m.group(1).replace(",", "") - out["pub_mb"] = m.group(2) - m = re.search(r"(?m)^\s*Sub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*MB/sec", text) - if m: - out["sub_msgs"] = m.group(1).replace(",", "") - out["sub_mb"] = m.group(2) - m = re.search(r"NATS Pub/Sub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*MB/sec", text) - if m: - out["agg_msgs"] = m.group(1).replace(",", "") - out["agg_mb"] = m.group(2) + def _rate(pattern: str, msgs_key: str, mb_key: str) -> None: + m = re.search(pattern, text) + if not m: + return + out[msgs_key] = m.group(1).replace(",", "") + n = float(m.group(2)) + unit = m.group(3).upper() + if unit == "KB": + n = n / 1024.0 + elif unit == "GB": + n = n * 1024.0 + out[mb_key] = f"{n:.2f}" + + _rate( + r"(?m)^\s*Pub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*(KB|MB|GB)/sec", + "pub_msgs", + "pub_mb", + ) + _rate( + r"(?m)^\s*Sub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*(KB|MB|GB)/sec", + "sub_msgs", + "sub_mb", + ) + _rate( + r"NATS Pub/Sub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*(KB|MB|GB)/sec", + "agg_msgs", + "agg_mb", + ) if "JetStream" in text or "--js" in text or "js-" in text: out["mode"] = "jetstream r=3 file" else: diff --git a/packages/verae-nats-cluster/scripts/build-ns1-study-report.py b/packages/verae-nats-cluster/scripts/build-ns1-study-report.py new file mode 100755 index 0000000..701b6f1 --- /dev/null +++ b/packages/verae-nats-cluster/scripts/build-ns1-study-report.py @@ -0,0 +1,458 @@ +#!/usr/bin/env python3 +"""Build the NS1-host study report (charts + markdown + HTML + PDF) from a results dir. + +Must be able to run entirely on NS1.GEORGELAMBERT.ORG with python3, matplotlib, +pandoc, and weasyprint. Parses nats bench logs; does not hard-code rates. +""" +from __future__ import annotations + +import json +import re +import subprocess +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +import importlib.util + +_spec = importlib.util.spec_from_file_location( + "bench_report", Path(__file__).resolve().parent / "bench-report.py" +) +_br = importlib.util.module_from_spec(_spec) +assert _spec.loader is not None +_spec.loader.exec_module(_br) +fmt_int = _br.fmt_int +lat_mode = _br.lat_mode +lat_sort = _br.lat_sort +parse_bench = _br.parse_bench +parse_lat = _br.parse_lat +thru_sort = _br.thru_sort + +try: + import matplotlib + + matplotlib.use("Agg") + import matplotlib.pyplot as plt + from matplotlib.ticker import FuncFormatter +except ImportError as e: + raise SystemExit(f"matplotlib required on NS1: {e}") from e + +INDIGO = "#4f46e5" +DEEP = "#312e81" +TEAL = "#047857" +AMBER = "#b45309" +LILAC = "#7c74f0" +INK = "#171a26" +MUTED = "#5b6178" +GRID = "#d9dce8" + +CORE_LABELS = { + "core-1p1s-50k-128": "1p1s\n50k×128 B", + "core-4p4s-100k-128": "4p4s\n100k×128 B", + "core-8p8s-200k-128": "8p8s\n200k×128 B", + "core-4p4s-50k-1k": "4p4s\n50k×1 KiB", +} +JS_LABELS = { + "js-1p-20k-128-r3": "1p 20k×128 B", + "js-4p-50k-128-r3": "4p 50k×128 B", + "js-4p-20k-1k-r3": "4p 20k×1 KiB", + "js-2p2s-20k-128-r3": "2p2s pull 20k×128 B", +} +LAT_LABELS = { + "lat-ping-1k-128": "Ping\n1k×128 B", + "lat-1p-5k-128": "Flood 1p\n5k×128 B", + "lat-4p-5k-1k": "Flood 4p\n5k×1 KiB", + "lat-4p-10k-128": "Flood 4p\n10k×128 B", + "lat-8p-20k-128": "Flood 8p\n20k×128 B", +} + + +def ms(s: str) -> float: + return float(s.replace("ms", "").replace(",", "").strip()) + + +def k_fmt(x: float, _pos: int | None = None) -> str: + if x >= 1_000_000: + return f"{x / 1_000_000:.2f}M" + if x >= 1000: + return f"{x / 1000:.0f}k" + return f"{x:.0f}" + + +def style() -> None: + plt.rcParams.update( + { + "font.family": "sans-serif", + "font.size": 10, + "axes.titlesize": 12, + "axes.titleweight": "semibold", + "axes.edgecolor": GRID, + "axes.labelcolor": INK, + "text.color": INK, + "xtick.color": MUTED, + "ytick.color": MUTED, + "figure.facecolor": "white", + "axes.facecolor": "white", + "axes.grid": True, + "grid.color": GRID, + "grid.linewidth": 0.8, + "legend.frameon": False, + "savefig.bbox": "tight", + "savefig.dpi": 160, + "savefig.facecolor": "white", + } + ) + + +def save(fig: plt.Figure, path: Path) -> None: + fig.savefig(path, dpi=160) + plt.close(fig) + + +def load_runs(folder: Path) -> tuple[list[dict[str, str]], list[dict[str, str]]]: + thru: list[dict[str, str]] = [] + lats: list[dict[str, str]] = [] + for f in sorted(folder.glob("*.txt")): + if f.name.startswith("host-"): + continue + text = f.read_text(encoding="utf-8", errors="replace") + lat = parse_lat(text) + if lat: + lat["run"] = f.stem + lats.append(lat) + continue + p = parse_bench(text) + if p.get("pub_msgs") or p.get("agg_msgs"): + p["run"] = f.stem + thru.append(p) + return sorted(thru, key=thru_sort), sorted(lats, key=lat_sort) + + +def kv_file(path: Path) -> dict[str, str]: + out: dict[str, str] = {} + if not path.exists(): + return out + for line in path.read_text(encoding="utf-8", errors="replace").splitlines(): + if "=" in line and not line.startswith("---"): + k, _, v = line.partition("=") + if k.strip() in out: + continue + out[k.strip()] = v.strip() + return out + + +def thru_table(thru: list[dict[str, str]]) -> str: + lines = [ + "| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |", + "|-----|------|------------------|------------|----------|------------|----------|", + ] + for p in thru: + lines.append( + f"| `{p['run']}` | {p.get('mode', '')} | {fmt_int(p.get('agg_msgs'))} | " + f"{fmt_int(p.get('pub_msgs'))} | {p.get('pub_mb') or '—'} | " + f"{fmt_int(p.get('sub_msgs'))} | {p.get('sub_mb') or '—'} |" + ) + return "\n".join(lines) + + +def delay_table(lats: list[dict[str, str]]) -> str: + lines = [ + "| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |", + "|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|", + ] + for p in lats: + kind = lat_mode(p.get("run", ""), p.get("mode", "")) + label = "ping (sequential RTT)" if kind == "ping" else "flood (burst queueing)" + lines.append( + f"| `{p['run']}` | {label} | {p.get('count', '')} | {p.get('pubs', '')} | " + f"{p.get('size', '')} B | {p.get('min', '')} | {p.get('avg', '')} | " + f"{p.get('p50', '')} | {p.get('p90', '')} | {p.get('p99', '')} | {p.get('max', '')} |" + ) + return "\n".join(lines) + + +def varz_table(path: Path) -> str: + if not path.exists(): + return "_varz snapshot not captured._" + rows = json.loads(path.read_text(encoding="utf-8")) + lines = [ + "| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream |", + "|------|------|-------------|---------|----------|-----|-------|---------|-----------|", + ] + for r in rows: + if r.get("error"): + lines.append(f"| {r.get('name')} | {r.get('vmid')} | error: {r['error']} | | | | | | |") + continue + lines.append( + f"| {r.get('name')} | {r.get('vmid')} | {r.get('connections')} | " + f"{r.get('in_msgs'):,} | {r.get('out_msgs'):,} | {r.get('cpu')} | " + f"{r.get('cores')} | {r.get('mem'):,} | {r.get('jetstream')} |" + ) + return "\n".join(lines) + + +def charts(thru: list[dict[str, str]], lats: list[dict[str, str]], dest: Path) -> None: + dest.mkdir(parents=True, exist_ok=True) + style() + by = {p["run"]: p for p in thru} + core_keys = [k for k in CORE_LABELS if k in by] + if core_keys: + fig, ax = plt.subplots(figsize=(9.2, 4.4)) + x = list(range(len(core_keys))) + w = 0.25 + agg = [int(by[k].get("agg_msgs") or 0) for k in core_keys] + pub = [int(by[k].get("pub_msgs") or 0) for k in core_keys] + sub = [int(by[k].get("sub_msgs") or 0) for k in core_keys] + ax.bar([i - w for i in x], agg, w, label="Aggregate", color=DEEP) + ax.bar(x, pub, w, label="Publish", color=INDIGO) + ax.bar([i + w for i in x], sub, w, label="Subscribe", color=TEAL) + ax.set_xticks(x, [CORE_LABELS[k] for k in core_keys]) + ax.set_ylabel("messages / second") + ax.set_title("Core NATS throughput (fire-and-forget) — NS1 host run") + ax.yaxis.set_major_formatter(FuncFormatter(k_fmt)) + ax.legend(loc="upper left") + ax.set_axisbelow(True) + save(fig, dest / "core-throughput.png") + + js_keys = [k for k in JS_LABELS if k in by] + if js_keys: + fig, ax = plt.subplots(figsize=(9.2, 4.4)) + pubs = [int(by[k].get("pub_msgs") or 0) for k in js_keys] + colors = [INDIGO, INDIGO, AMBER, LILAC][: len(js_keys)] + ax.bar([JS_LABELS[k] for k in js_keys], pubs, color=colors) + ax.set_ylabel("durable publish messages / second") + ax.set_title("JetStream file store, replicas=3 — NS1 host run") + ax.yaxis.set_major_formatter(FuncFormatter(k_fmt)) + ax.set_axisbelow(True) + for i, v in enumerate(pubs): + ax.text(i, v * 1.02, f"{v:,}", ha="center", va="bottom", fontsize=9, color=MUTED) + save(fig, dest / "js-throughput.png") + + pair = [("core-1p1s-50k-128", "js-1p-20k-128-r3"), ("core-4p4s-100k-128", "js-4p-50k-128-r3"), ("core-4p4s-50k-1k", "js-4p-20k-1k-r3")] + if all(c in by and j in by for c, j in pair): + fig, ax = plt.subplots(figsize=(9.2, 4.4)) + labels = ["1 publisher\n128 B", "4 publishers\n128 B", "4 publishers\n1 KiB"] + core_pub = [int(by[c]["pub_msgs"]) for c, _ in pair] + js_pub = [int(by[j]["pub_msgs"]) for _, j in pair] + x = list(range(3)) + w = 0.35 + ax.bar([i - w / 2 for i in x], core_pub, w, label="Core NATS (no disk)", color=INDIGO) + ax.bar([i + w / 2 for i in x], js_pub, w, label="JetStream r=3 file", color=AMBER) + ax.set_xticks(x, labels) + ax.set_yscale("log") + ax.set_ylabel("publish messages / second (log)") + ax.set_title("Core vs JetStream — NS1 host run") + ax.legend(loc="upper right") + ax.set_axisbelow(True) + save(fig, dest / "core-vs-js.png") + + if "core-4p4s-100k-128" in by and "core-4p4s-50k-1k" in by: + fig, axes = plt.subplots(1, 2, figsize=(9.2, 4.2)) + labels = ["128 B\n4p4s", "1 KiB\n4p4s"] + msgs = [int(by["core-4p4s-100k-128"].get("agg_msgs") or 0), int(by["core-4p4s-50k-1k"].get("agg_msgs") or 0)] + mb = [float(by["core-4p4s-100k-128"].get("agg_mb") or 0), float(by["core-4p4s-50k-1k"].get("agg_mb") or 0)] + axes[0].bar(labels, msgs, color=[INDIGO, AMBER]) + axes[0].set_title("Aggregate messages / second") + axes[0].yaxis.set_major_formatter(FuncFormatter(k_fmt)) + axes[1].bar(labels, mb, color=[INDIGO, AMBER]) + axes[1].set_title("Aggregate MB / second") + fig.suptitle("Core NATS payload effect — NS1 host run", fontsize=12, fontweight="semibold") + fig.tight_layout() + save(fig, dest / "payload-size.png") + + if lats: + fig, ax = plt.subplots(figsize=(9.2, 4.6)) + ordered = [p for p in lats] + labels = [LAT_LABELS.get(p["run"], p["run"]) for p in ordered] + x = list(range(len(ordered))) + w = 0.25 + p50 = [ms(p["p50"]) for p in ordered] + p90 = [ms(p["p90"]) for p in ordered] + p99 = [ms(p["p99"]) for p in ordered] + ax.bar([i - w for i in x], p50, w, label="p50", color=TEAL) + ax.bar(x, p90, w, label="p90", color=INDIGO) + ax.bar([i + w for i in x], p99, w, label="p99", color=AMBER) + ax.set_xticks(x, labels) + ax.set_yscale("log") + ax.set_ylabel("milliseconds (log)") + ax.set_title("Round-trip delay — NS1 host run") + ax.axhline(1.0, color=GRID, linestyle="--", linewidth=1) + ax.legend(loc="upper left") + ax.set_axisbelow(True) + save(fig, dest / "delay-percentiles.png") + + +def figure(name: str, caption: str) -> str: + return f"![{caption}](charts/{name})\n\n*{caption}*" + + +def write_markdown(folder: Path, thru: list[dict[str, str]], lats: list[dict[str, str]]) -> str: + before = kv_file(folder / "host-before.txt") + after = kv_file(folder / "host-after.txt") + stamp = folder.name + method = (Path(__file__).resolve().parent / "ns1-study-methodology.md").read_text(encoding="utf-8") + figs = [] + charts_dir = folder / "charts" + if (charts_dir / "core-throughput.png").exists(): + figs.append("### Core NATS\n\n" + figure("core-throughput.png", "Core NATS throughput at four loads (NS1 host run)")) + if (charts_dir / "payload-size.png").exists(): + figs.append("### Payload size (core)\n\n" + figure("payload-size.png", "Core NATS 128 B vs 1 KiB (NS1 host run)")) + if (charts_dir / "js-throughput.png").exists(): + figs.append("### JetStream r=3 file\n\n" + figure("js-throughput.png", "JetStream durable publish rate (NS1 host run)")) + if (charts_dir / "core-vs-js.png").exists(): + figs.append("### Core vs JetStream\n\n" + figure("core-vs-js.png", "Core vs JetStream publish rate, log scale (NS1 host run)")) + if (charts_dir / "delay-percentiles.png").exists(): + figs.append("### Delay\n\n" + figure("delay-percentiles.png", "Ping vs flood delay percentiles, log scale (NS1 host run)")) + + ping = next((p for p in lats if "ping" in p.get("run", "")), None) + js1 = next((p for p in thru if p["run"] == "js-1p-20k-128-r3"), None) + core1 = next((p for p in thru if p["run"] == "core-1p1s-50k-128"), None) + + md = f"""**Progress report (second study)** · run `{stamp}` (UTC) + +> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): the orchestrator (`study-on-ns1.sh`), `nats bench`, `latency.mjs` (inside LXC 510 on this hypervisor), charting (`matplotlib`), and HTML/PDF (`pandoc` + `weasyprint`). The operator laptop did **not** publish, subscribe, draw charts, or render the PDF. Traffic stayed on `vmbr1` from LXC **510** to `nats-a/b/c` (**511–513**). + +This is a full methodology write-up plus the numbers from that on-host run. The earlier report (`nats-cluster-bench`, run `20260912T045131Z`) used the same cluster but was **orchestrated and rendered off-box**. Use this document when you need “it was all run on 138.” + +--- + +## 1. Executive summary + +| Item | This NS1-host run | +|------|-------------------| +| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `{before.get("whoami", "marchon")}` | +| Bench client | LXC {before.get("client_vmid", "510")} `verae-px-worker` | +| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` | +| Client URL | `{before.get("nats_url", "")}` | +| Host load before | `{before.get("loadavg", "n/a")}` | +| Host load after | `{after.get("loadavg", "n/a")}` | +| Core 1p1s 128 B pub | {fmt_int(core1.get("pub_msgs") if core1 else None)} msgs/s | +| JetStream 1p 128 B r=3 | {fmt_int(js1.get("pub_msgs") if js1 else None)} durable pubs/s | +| Ping p50 / p99 | {ping.get("p50") if ping else "—"} / {ping.get("p99") if ping else "—"} | + +Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst. + +--- + +## 2. Where it ran (and where it did not) + +```text +Operator laptop ──ssh──► NS1.GEORGELAMBERT.ORG 70.88.205.138 + study-on-ns1.sh + python3 build-ns1-study-report.py + sudo pct exec 510 ──► nats bench / latency.mjs + │ + ▼ vmbr1 + 10.10.10.21-23 :4222 +``` + +- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510). +- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file. + +--- + +## 3. Results (this run) + +### Host and brokers + +**Before** + +{varz_table(folder / "varz-before.json")} + +**After** + +{varz_table(folder / "varz-after.json")} + +nproc={before.get("nproc", "?")} · uname=`{before.get("uname", "")}` + +### Throughput + +{thru_table(thru)} + +### Round-trip delay + +{delay_table(lats)} + +{chr(10).join(figs)} + +--- + +{method} + +--- + +## 6. Reproducing this study + +On **NS1 only**: + +```bash +cd ~/verae-src/verae-nats-cluster +bash scripts/study-on-ns1.sh +``` + +The script exits if `hostname` is not NS1. Outputs land in `results//` including `nats-cluster-bench-ns1.{{md,html,pdf}}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog. + +Raw logs for this run: `results/{stamp}/`. +""" + return md + + +def render(md_path: Path, html_path: Path, pdf_path: Path) -> None: + css = Path(__file__).resolve().parent / "docs-print.css" + header = html_path.with_suffix(".hdr.html") + banner = html_path.with_suffix(".ban.html") + css_text = css.read_text(encoding="utf-8") if css.exists() else "" + header.write_text(f"\n", encoding="utf-8") + banner.write_text( + '
' + '' + '
Verae Time × Zapier · progress report · run on NS1.GEORGELAMBERT.ORG
' + "

NATS cluster message speed — NS1 host study

" + '
packages/zapier-decisions/reports/nats-cluster-bench-ns1.md
' + "
\n", + encoding="utf-8", + ) + r = subprocess.run( + [ + "pandoc", + str(md_path), + "-o", + str(html_path), + "--standalone", + f"--resource-path={md_path.parent}", + "--highlight-style=breezedark", + "--metadata=title=NATS cluster message speed — NS1 host study", + f"--include-in-header={header}", + f"--include-before-body={banner}", + ], + capture_output=True, + text=True, + ) + header.unlink(missing_ok=True) + banner.unlink(missing_ok=True) + if r.returncode != 0: + raise SystemExit(f"pandoc failed: {r.stderr[-800:]}") + w = subprocess.run(["weasyprint", str(html_path), str(pdf_path)], capture_output=True, text=True) + if w.returncode != 0: + raise SystemExit(f"weasyprint failed: {w.stderr[-800:]}") + + +def main() -> int: + folder = Path(sys.argv[1] if len(sys.argv) > 1 else ".") + thru, lats = load_runs(folder) + charts(thru, lats, folder / "charts") + md = write_markdown(folder, thru, lats) + md_path = folder / "nats-cluster-bench-ns1.md" + md_path.write_text(md, encoding="utf-8") + html_path = folder / "nats-cluster-bench-ns1.html" + pdf_path = folder / "nats-cluster-bench-ns1.pdf" + render(md_path, html_path, pdf_path) + print(f"wrote {md_path}") + print(f"wrote {html_path}") + print(f"wrote {pdf_path}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/packages/verae-nats-cluster/scripts/docs-print.css b/packages/verae-nats-cluster/scripts/docs-print.css new file mode 100644 index 0000000..99256e9 --- /dev/null +++ b/packages/verae-nats-cluster/scripts/docs-print.css @@ -0,0 +1,122 @@ +/* Colored print + screen stylesheet for zapier.georgelambert.org */ +:root { + --ink: #171a26; + --muted: #5b6178; + --line: #d9dce8; + --bg: #f4f5fb; + --paper: #ffffff; + --accent: #4f46e5; + --accent-deep: #312e81; + --accent-soft: #eef0fe; + --ok: #047857; + --warn: #8a5a00; + --code-bg: #1b1f33; + --code-fg: #e8ecff; +} +html { background: var(--bg); } +body { + margin: 0 auto; + padding: 1.5rem 1.25rem 3rem; + max-width: 48rem; + font: 15px/1.55 -apple-system, "Segoe UI", Georgia, serif; + color: var(--ink); + background: var(--paper); +} +.doc-banner { + background: linear-gradient(160deg, #312e81 0%, #4f46e5 60%, #7c74f0 100%); + color: #eef0fe; + margin: -1.5rem -1.25rem 1.5rem; + padding: 1.1rem 1.25rem 1rem; +} +.doc-banner a { color: #fff; } +.doc-banner .kicker { + letter-spacing: 0.12em; + text-transform: uppercase; + font: 700 10px system-ui, sans-serif; + opacity: 0.8; +} +.doc-banner h1 { margin: 0.25rem 0 0; font-size: 1.45rem; color: #fff; } +h1, h2, h3, h4 { color: var(--accent-deep); page-break-after: avoid; } +h1 { font-size: 1.7rem; } +h2 { + font-size: 1.2rem; + border-bottom: 2px solid var(--accent); + padding-bottom: 0.2rem; + margin-top: 1.6rem; +} +h3 { font-size: 1.05rem; color: var(--accent); } +a { color: var(--accent); } +p, li { orphans: 3; widows: 3; } +code { + font-family: ui-monospace, Menlo, Consolas, monospace; + font-size: 0.86em; + background: var(--accent-soft); + color: var(--accent-deep); + padding: 0.08em 0.28em; + border-radius: 4px; +} +pre, div.sourceCode, div.sourceCode pre { + background: var(--code-bg) !important; + color: var(--code-fg) !important; + padding: 0.85rem 1rem; + border-radius: 10px; + overflow: auto; + font-size: 0.78rem; + line-height: 1.4; + page-break-inside: avoid; +} +pre code { background: transparent; color: inherit; padding: 0; } +#title-block-header, header#title-block-header, h1.title { display: none; } +.doc-banner + h1 { display: none; } +table { + border-collapse: collapse; + width: 100%; + margin: 0.8rem 0 1.2rem; + font-size: 0.9rem; + page-break-inside: avoid; +} +th, td { border: 1px solid var(--line); padding: 0.38rem 0.55rem; text-align: left; vertical-align: top; } +th { + background: var(--accent); + color: #fff; + font: 650 12px system-ui, sans-serif; +} +tr:nth-child(even) td { background: var(--accent-soft); } +blockquote { + margin: 1rem 0; + padding: 0.4rem 0.9rem; + border-left: 4px solid var(--accent); + background: var(--accent-soft); + color: var(--accent-deep); +} +img { max-width: 100%; height: auto; border-radius: 8px; page-break-inside: avoid; } +hr { border: 0; border-top: 1px solid var(--line); } +ul, ol { padding-left: 1.25rem; } +nav.site { font: 13px system-ui, sans-serif; margin-bottom: 0.4rem; } +.source-path { font: 11px ui-monospace, Menlo, monospace; color: var(--muted); } + +@page { + size: letter; + margin: 0.65in 0.7in 0.8in 0.7in; + @top-left { + content: "Verae Time × Zapier"; + font: 700 8pt system-ui, sans-serif; + color: #4f46e5; + } + @top-right { + content: "zapier.georgelambert.org"; + font: 8pt system-ui, sans-serif; + color: #6b7186; + } + @bottom-center { + content: counter(page) " / " counter(pages); + font: 8pt system-ui, sans-serif; + color: #6b7186; + } +} +@media print { + html, body { background: #fff; max-width: none; padding: 0; } + .doc-banner { margin: 0 0 1rem; border-radius: 8px; -webkit-print-color-adjust: exact; print-color-adjust: exact; } + a { text-decoration: none; } + th, tr:nth-child(even) td, pre, blockquote, code { -webkit-print-color-adjust: exact; print-color-adjust: exact; } +} diff --git a/packages/verae-nats-cluster/scripts/ns1-study-methodology.md b/packages/verae-nats-cluster/scripts/ns1-study-methodology.md new file mode 100644 index 0000000..2d3423d --- /dev/null +++ b/packages/verae-nats-cluster/scripts/ns1-study-methodology.md @@ -0,0 +1,202 @@ +## 4. Study methodology + +### 4.1 Question + +On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)? + +### 4.2 Hypotheses (stated before the run) + +1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica. +2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated. +3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting. +4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count. +5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound. + +### 4.3 Independent variables (what we changed) + +| Factor | Levels | +|--------|--------| +| Transport | Core NATS pub/sub vs JetStream file replicas=3 | +| Publisher count | 1, 2, 4, 8 | +| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 | +| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) | +| Payload | 128 B, 1024 B | +| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) | + +### 4.4 Dependent variables (what we recorded) + +| Metric | Instrument | Unit | +|--------|------------|------| +| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s | +| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s | +| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) | +| Publisher spread | nats min/avg/max **msgs/s** | not delay | +| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max | +| Host load | `/proc/loadavg` before and after | load average | +| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem | + +**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`. + +### 4.5 Controls and constants + +- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`. +- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`). +- Bench client is LXC **510**, not a nats-* server. +- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide. +- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`). +- No TLS, no nkeys, no account isolation (isolation is `vmbr1`). +- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder. + +### 4.6 Procedure + +1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise. +2. Snapshot host load, memory, LXC configs, and each nats `varz`. +3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB). +4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file. +5. Copy `latency.mjs` into 510; ping then flood at several batch sizes. +6. Snapshot host/`varz` again. +7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**. + +No publish, subscribe, chart, or PDF process runs on the operator laptop for this study. + +### 4.7 Instrumentation path + +```text +[NS1 host 70.88.205.138] + study-on-ns1.sh (bash + python3) + | + | sudo pct exec 510 + v +[LXC 510 verae-px-worker 10.10.10.20] + nats bench / node latency.mjs + | + | NATS client protocol to + v +[LXC 511/512/513 10.10.10.21-23 :4222] + nats-server -js cluster routes :6222 +``` + +The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`. + +### 4.8 Threats to validity + +| Threat | Effect on numbers | +|--------|-------------------| +| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. | +| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. | +| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. | +| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. | +| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. | +| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. | +| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. | +| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. | + +### 4.9 Ethics / safety + +Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket. + +--- + +## 5. Suggestions for fine-tuning + +These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way. + +### 5.1 Treat JetStream as the product limiter + +Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out. + +### 5.2 Split storage class by stream + +| Stream | Suggested store | Why | +|--------|-----------------|-----| +| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad | +| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both | +| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer | +| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry | +| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive | + +Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert. + +### 5.3 Give JetStream real disks + +Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain. + +- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`. +- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure. +- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it. +- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.” + +### 5.4 Isolate the nats CTs from the rest of NS1 + +Host load on this box is often already several. Pin: + +- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes. +- Memory high enough that file-backed streams stay cache-hot for the working set. +- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync. + +Re-run this study after pinning; H1/H2 should move more than ping. + +### 5.5 Consumer and mailbox tuning (delay H4) + +Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT. + +- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM. +- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber. +- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped. +- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT. + +### 5.6 Publisher-side batching in middleware + +A timestamp job is one small JSON. 16k msgs/s is ample. Still: + +- Avoid per-byte publishes; one message per job/event. +- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs). +- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only. + +### 5.7 nats-server knobs worth measuring (A/B with this script) + +| Knob | Why try it | +|------|------------| +| `max_payload` | Keep default unless archive puts grow | +| `write_deadline` | Slow consumer protection for webhooks | +| `max_pending` | Bound memory on a stuck Zapier hook | +| `max_connections` | Fleet workers + keep + middleware | +| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT | +| `max_outstanding_catchup` | Replica restart after a nats-c blip | +| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT | + +Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99. + +### 5.8 Network + +- Keep NATS off `vmbr0`. No change. +- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load). +- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus. + +### 5.9 Security cost (when nkeys/mTLS flip) + +`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement. + +### 5.10 Operational fine-tuning (lag, not peak msgs/s) + +1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`. +2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense. +3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost. +4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated. + +### 5.11 What not to tune + +- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge. +- Do not treat flood 400 ms as “cluster RTT.” Fix consumers. +- Do not load-test on `ZAPIER_*` streams. +- Do not bind client NATS to `0.0.0.0` on `vmbr0`. + +### 5.12 Recommended next experiments (same method, one change each) + +1. CPU pin nats-a/b/c → re-run JS 1p + ping. +2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder). +3. Distinct `store_dir` disks per node. +4. nkeys on, same ladder. +5. Three hardware boxes, same `cluster.env` IPs updated. + +Each experiment should produce a new `results//` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory. diff --git a/packages/verae-nats-cluster/scripts/study-on-ns1.sh b/packages/verae-nats-cluster/scripts/study-on-ns1.sh new file mode 100755 index 0000000..4eec5cc --- /dev/null +++ b/packages/verae-nats-cluster/scripts/study-on-ns1.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +# Full message-speed study. Must run ON NS1.GEORGELAMBERT.ORG (70.88.205.138). +# Orchestration, nats bench (via pct into LXC 510), charts, HTML, and PDF all +# happen on this host. The laptop is not in the measurement path. +set -euo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +export PATH="/usr/sbin:/usr/bin:/bin:/usr/local/bin:$PATH" + +HOST="$(hostname -f 2>/dev/null || hostname)" +case "$HOST" in + NS1.GEORGELAMBERT.ORG|NS1|ns1.georgelambert.org|ns1) ;; + *) + echo "refusing: study-on-ns1.sh must run on NS1.GEORGELAMBERT.ORG (70.88.205.138), got '$HOST'" >&2 + exit 1 + ;; +esac + +# shellcheck disable=SC1091 +. "$ROOT/client.env" +CLIENT_VMID="${CLIENT_VMID:-510}" +STAMP="$(date -u +%Y%m%dT%H%M%SZ)" +OUT="${BENCH_OUT:-$ROOT/results/$STAMP}" +mkdir -p "$OUT" + +dump_env() { + local tag="$1" + local f="$OUT/host-$tag.txt" + { + echo "execution_host=NS1.GEORGELAMBERT.ORG" + echo "execution_ip=70.88.205.138" + echo "hostname=$(hostname)" + echo "utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo "whoami=$(whoami)" + echo "pwd=$(pwd)" + echo "uname=$(uname -a)" + echo "nproc=$(nproc)" + echo "loadavg=$(cat /proc/loadavg)" + echo "client_vmid=$CLIENT_VMID" + echo "nats_url=$NATS_URL" + echo "--- free ---" + free -h + echo "--- pct list ---" + sudo pct list + for v in 510 511 512 513; do + echo "--- pct config $v ---" + sudo pct config "$v" | grep -E '^(hostname|cores|memory|swap|rootfs|mp|net)' || true + done + } >"$f" + python3 - "$OUT" "$tag" <<'PY' +import json, sys, urllib.request +from pathlib import Path +out, tag = Path(sys.argv[1]), sys.argv[2] +nodes = [] +for vmid, name in (("511", "nats-a"), ("512", "nats-b"), ("513", "nats-c")): + raw = "" + try: + import subprocess + raw = subprocess.check_output( + ["sudo", "pct", "exec", vmid, "--", "curl", "-fsS", "--max-time", "3", "http://127.0.0.1:8222/varz"], + text=True, + ) + d = json.loads(raw) + nodes.append({ + "vmid": vmid, + "name": name, + "server_name": d.get("server_name"), + "host": d.get("host"), + "port": d.get("port"), + "connections": d.get("connections"), + "in_msgs": d.get("in_msgs"), + "out_msgs": d.get("out_msgs"), + "in_bytes": d.get("in_bytes"), + "out_bytes": d.get("out_bytes"), + "cpu": d.get("cpu"), + "cores": d.get("cores"), + "mem": d.get("mem"), + "jetstream": bool(d.get("jetstream")), + }) + except Exception as e: + nodes.append({"vmid": vmid, "name": name, "error": str(e)}) +(out / f"varz-{tag}.json").write_text(json.dumps(nodes, indent=2) + "\n", encoding="utf-8") +PY +} + +echo "NS1 study $STAMP out=$OUT" +dump_env before +BENCH_OUT="$OUT" CLIENT_VMID="$CLIENT_VMID" bash "$ROOT/scripts/bench.sh" +dump_env after +python3 "$ROOT/scripts/build-ns1-study-report.py" "$OUT" +echo "NS1 study complete $OUT" +ls -la "$OUT"/nats-cluster-bench-ns1.* "$OUT"/charts 2>/dev/null || ls -la "$OUT" diff --git a/packages/verae-nats-cluster/scripts/test.sh b/packages/verae-nats-cluster/scripts/test.sh index 7060cb3..9e02ae8 100755 --- a/packages/verae-nats-cluster/scripts/test.sh +++ b/packages/verae-nats-cluster/scripts/test.sh @@ -6,6 +6,7 @@ bash -n "$ROOT/scripts/lib-ct.sh" bash -n "$ROOT/scripts/create-cluster.sh" bash -n "$ROOT/scripts/status.sh" bash -n "$ROOT/scripts/bench.sh" +bash -n "$ROOT/scripts/study-on-ns1.sh" grep -q 'host: {{IP}}' "$ROOT/conf/nats.conf.tmpl" grep -qv '0.0.0.0' "$ROOT/conf/nats.conf.tmpl" if [[ ! -d /etc/pve/nodes ]]; then diff --git a/packages/zapier-decisions/LOG.md b/packages/zapier-decisions/LOG.md index 898d683..fb7372f 100644 --- a/packages/zapier-decisions/LOG.md +++ b/packages/zapier-decisions/LOG.md @@ -1,5 +1,12 @@ # Action log +## 2026-09-12 — NS1-host NATS study (second run) + +- All orchestration, nats bench, matplotlib, pandoc, and weasyprint ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`). Laptop not in the path. +- Script: `verae-nats-cluster/scripts/study-on-ns1.sh` (refuses unless hostname is NS1). +- Run `20260912T051237Z`. Report: `reports/ns1-host/nats-cluster-bench-ns1.{md,html,pdf}`. +- Host load ~8.5. JS 1p 128 B ~7.4k durable pubs/s; ping p99 1.38 ms. Methodology + fine-tuning in that report. + ## 2026-09-12 — NATS cluster speed report - Bench from LXC 510 vs nats-a/b/c (`20260912T045131Z`). diff --git a/packages/zapier-decisions/README.md b/packages/zapier-decisions/README.md index bd9e341..c556fab 100644 --- a/packages/zapier-decisions/README.md +++ b/packages/zapier-decisions/README.md @@ -8,6 +8,7 @@ Lab log of architecture decisions, actions taken, and open todos while cleaning |------|------| | [STATUS.md](STATUS.md) | **Current status** (2026-09-12) | | [reports/nats-cluster-bench.md](reports/nats-cluster-bench.md) | NATS cluster speed report (charts, HTML, PDF) | +| [reports/ns1-host/nats-cluster-bench-ns1.md](reports/ns1-host/nats-cluster-bench-ns1.md) | Second study: all code on NS1 (methodology + tuning) | | [LOG.md](LOG.md) | Chronological actions | | [TODO.md](TODO.md) | Open items | | [decisions/](decisions/) | One file per decision | diff --git a/packages/zapier-decisions/STATUS.md b/packages/zapier-decisions/STATUS.md index add8185..81abc25 100644 --- a/packages/zapier-decisions/STATUS.md +++ b/packages/zapier-decisions/STATUS.md @@ -37,6 +37,8 @@ Measured from LXC **510** (not a nats-* server) against `10.10.10.21–23`. **Full report (charts + explanations):** [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.pdf) · [markdown](reports/nats-cluster-bench.md) +**Second study (all code on NS1.GEORGELAMBERT.ORG / 70.88.205.138, run `20260912T051237Z`):** [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.pdf) · [markdown](reports/ns1-host/nats-cluster-bench-ns1.md). Orchestrator, nats bench, charts, HTML, and PDF ran on 138; traffic was LXC 510 → 511–513 on `vmbr1`. Includes full methodology and fine-tuning notes. Host load ~8.5/40 during the run; JetStream 1p 128 B was **7.4k** durable pubs/s (vs ~16k on the quieter first ladder); ping p99 **1.38 ms**. + Short tables: [BENCH.md](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md). Re-run: `bash scripts/bench.sh` on NS1. | Kind | Load | Result | diff --git a/packages/zapier-decisions/reports/ns1-host/charts/core-throughput.png b/packages/zapier-decisions/reports/ns1-host/charts/core-throughput.png new file mode 100644 index 0000000..44609c2 Binary files /dev/null and b/packages/zapier-decisions/reports/ns1-host/charts/core-throughput.png differ diff --git a/packages/zapier-decisions/reports/ns1-host/charts/core-vs-js.png b/packages/zapier-decisions/reports/ns1-host/charts/core-vs-js.png new file mode 100644 index 0000000..dae2b58 Binary files /dev/null and b/packages/zapier-decisions/reports/ns1-host/charts/core-vs-js.png differ diff --git a/packages/zapier-decisions/reports/ns1-host/charts/delay-percentiles.png b/packages/zapier-decisions/reports/ns1-host/charts/delay-percentiles.png new file mode 100644 index 0000000..f57cf44 Binary files /dev/null and b/packages/zapier-decisions/reports/ns1-host/charts/delay-percentiles.png differ diff --git a/packages/zapier-decisions/reports/ns1-host/charts/js-throughput.png b/packages/zapier-decisions/reports/ns1-host/charts/js-throughput.png new file mode 100644 index 0000000..1edaebe Binary files /dev/null and b/packages/zapier-decisions/reports/ns1-host/charts/js-throughput.png differ diff --git a/packages/zapier-decisions/reports/ns1-host/charts/payload-size.png b/packages/zapier-decisions/reports/ns1-host/charts/payload-size.png new file mode 100644 index 0000000..5c64356 Binary files /dev/null and b/packages/zapier-decisions/reports/ns1-host/charts/payload-size.png differ diff --git a/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.html b/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.html new file mode 100644 index 0000000..59b896e --- /dev/null +++ b/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.html @@ -0,0 +1,1291 @@ + + + + + + + NATS cluster message speed — NS1 host study + + + + +
Verae Time × Zapier · progress report · run on NS1.GEORGELAMBERT.ORG

NATS cluster message speed — NS1 host study

packages/zapier-decisions/reports/nats-cluster-bench-ns1.md
+
+

NATS cluster message speed — NS1 host study

+
+

Progress report (second study) · run +20260912T051237Z (UTC)

+
+

Execution provenance. Every process for this study +ran on NS1.GEORGELAMBERT.ORG +(70.88.205.138): the orchestrator +(study-on-ns1.sh), nats bench, +latency.mjs (inside LXC 510 on this hypervisor), charting +(matplotlib), and HTML/PDF (pandoc + +weasyprint). The operator laptop did not +publish, subscribe, draw charts, or render the PDF. Traffic stayed on +vmbr1 from LXC 510 to +nats-a/b/c (511–513).

+
+

This is a full methodology write-up plus the numbers from that +on-host run. The earlier report (nats-cluster-bench, run +20260912T045131Z) used the same cluster but was +orchestrated and rendered off-box. Use this document +when you need “it was all run on 138.”

+
+

1. Executive summary

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ItemThis NS1-host run
Control planeNS1.GEORGELAMBERT.ORG (70.88.205.138), user +marchon
Bench clientLXC 510 verae-px-worker
BrokersLXC 511/512/513 nats-a/b/c on +10.10.10.21–23
Client URLnats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
Host load before8.59 8.39 8.15 5/3843 4096277
Host load after8.67 8.33 8.15 9/3863 4114865
Core 1p1s 128 B pub502,502 msgs/s
JetStream 1p 128 B r=37,393 durable pubs/s
Ping p50 / p990.395ms / 1.377ms
+

Product traffic is the JetStream row. Ping is one-message delay. +Flood is mailbox catch-up after a burst.

+
+

2. Where it ran (and where it +did not)

+
Operator laptop  ──ssh──►  NS1.GEORGELAMBERT.ORG  70.88.205.138
+                              study-on-ns1.sh
+                              python3 build-ns1-study-report.py
+                              sudo pct exec 510 ──► nats bench / latency.mjs
+                                                    │
+                                                    ▼ vmbr1
+                                              10.10.10.21-23 :4222
+
    +
  • Did run on 138: bash, python3, matplotlib, pandoc, +weasyprint, pct, nats-server (in LXC), nats CLI and Node +(in LXC 510).
    • +
  • Did not run on the laptop: no local +nats bench, no local charting, no local WeasyPrint for this +file.
    • +
+
+

3. Results (this run)

+

Host and brokers

+

Before

+ +++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NodeVMIDconnectionsin_msgsout_msgscpucoresmem (B)jetstream
nats-a51121,721,2034,933,7591126,398,720True
nats-b51221,613,2053,513,0960123,097,344True
nats-c51321,760,9734,048,5822123,867,392True
+

After

+ +++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NodeVMIDconnectionsin_msgsout_msgscpucoresmem (B)jetstream
nats-a51122,097,5965,910,1611138,273,024True
nats-b51222,121,8164,746,5391141,361,408True
nats-c51322,241,3665,053,9781138,162,432True
+

nproc=40 · +uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux

+

Throughput

+ +++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunModeAggregate msgs/sPub msgs/sPub MB/sSub msgs/sSub MB/s
core-1p1s-50k-128core pub/sub776,331502,50261.34420,97651.39
core-4p4s-100k-128core pub/sub1,278,454379,98546.381,078,910131.70
core-4p4s-50k-1kcore pub/sub581,464209,545204.63482,645471.33
core-8p8s-200k-128core pub/sub2,065,217333,73140.741,847,242225.49
js-1p-20k-128-r3jetstream r=3 file—7,3930.90——
js-2p2s-20k-128-r3jetstream r=3 file15,4167,7280.947,7110.94
js-4p-20k-1k-r3jetstream r=3 file—14,98514.63——
js-4p-50k-128-r3jetstream r=3 file—17,9862.20——
+

Round-trip delay

+ +++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunKindCountPubsSizeminavgp50p90p99max
lat-ping-1k-128ping (sequential RTT)10001128 B0.341ms0.470ms0.395ms0.651ms1.377ms3.243ms
lat-1p-5k-128flood (burst queueing)50001128 B132.425ms203.717ms204.458ms241.750ms249.468ms249.640ms
lat-4p-5k-1kflood (burst queueing)500041024 B196.422ms221.264ms224.039ms233.180ms236.798ms238.177ms
lat-4p-10k-128flood (burst queueing)100004128 B211.108ms294.925ms300.254ms314.223ms315.551ms316.091ms
lat-8p-20k-128flood (burst queueing)200008128 B280.608ms414.972ms409.802ms513.821ms535.394ms536.084ms
+

Core NATS

+
+ + +
+

Core NATS throughput at four loads (NS1 host run) ### +Payload size (core)

+
+ + +
+

Core NATS 128 B vs 1 KiB (NS1 host run) ### JetStream r=3 +file

+
+ + +
+

JetStream durable publish rate (NS1 host run) ### Core vs +JetStream

+
+ + +
+

Core vs JetStream publish rate, log scale (NS1 host run) ### +Delay

+
+ + +
+

Ping vs flood delay percentiles, log scale (NS1 host +run)

+
+

4. Study methodology

+

4.1 Question

+

On the NS1 test stand, what message throughput and +delay does the three-node verae JetStream +cluster deliver at several loads, and which part of the stack is the +limiter for product traffic (jobs, events, webhooks, archive)?

+

4.2 Hypotheses (stated before +the run)

+
    +
  1. H1 — Core vs JetStream. Fire-and-forget core NATS +is at least an order of magnitude faster than JetStream file + +replicas=3, because durable publish waits for a majority disk +replica.
    2. +
  2. H2 — JetStream parallelism. Adding publishers does +not linearly increase JetStream write rate once the +replica log is saturated.
    3. +
  3. H3 — Quiet delay. Sequential pub→sub round trip on +vmbr1 is well under 1 ms p99 when the consumer is +waiting.
    4. +
  4. H4 — Burst delay. If publishers dump a batch before +the subscriber drains, observed delay is queueing time, +roughly linear in backlog, not in cluster hop count.
    5. +
  5. H5 — Payload. Moving 128 B → 1 KiB lowers message +rate and raises byte rate on core NATS; JetStream in this size band +stays replica/fsync bound.
    6. +
+

4.3 Independent variables +(what we changed)

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FactorLevels
TransportCore NATS pub/sub vs JetStream file replicas=3
Publisher count1, 2, 4, 8
Subscriber count0 (JS publish-only), 1, 2, 4, 8
Message count1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step)
Payload128 B, 1024 B
Delay modeping (publish, wait, repeat) vs +flood (publish all, then drain)
+

4.4 Dependent variables +(what we recorded)

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MetricInstrumentUnit
Publish ratenats bench 0.1.6 Pub statsmsgs/s, MB/s
Subscribe ratenats bench Sub statsmsgs/s, MB/s
Aggregatenats bench NATS Pub/Sub statsmsgs/s (fan-out counts both sides)
Publisher spreadnats min/avg/max msgs/snot delay
One-way-ish RTTlatency.mjs header timestampmin, avg, p50, p90, p99, max
Host load/proc/loadavg before and afterload average
Broker countershttp://127.0.0.1:8222/varz inside each nats LXCconnections, in/out msgs, cpu, mem
+

Important: nats CLI 0.1.6 min/avg/max are +rate spread across publishers, not microseconds of +delay. Delay is only latency.mjs.

+

4.5 Controls and constants

+
    +
  • Cluster name verae, three routes, client +:4222, cluster :6222, monitor loopback +:8222.
    • +
  • Client URL always the three-node list on vmbr1 (never +host 127.0.0.1:4222, never vmbr0).
    • +
  • Bench client is LXC 510, not a nats-* server.
    • +
  • JetStream bench stream name benchstream, +file storage, replicas=3, deleted +between JS loads (nats stream rm --force) so names do not +collide.
    • +
  • Product streams were not the bench target (no load +test on ZAPIER_* / VERAE_ARCHIVE).
    • +
  • No TLS, no nkeys, no account isolation (isolation is +vmbr1).
    • +
  • Same nats CLI version (0.1.6) and nats@2 Node client as +the first ladder.
    • +
+

4.6 Procedure

+
    +
  1. Confirm this script is executing on +NS1.GEORGELAMBERT.ORG. Refuse otherwise.
    2. +
  2. Snapshot host load, memory, LXC configs, and each nats +varz.
    3. +
  3. From NS1, pct exec 510 the core ladder (1p1s, 4p4s, +8p8s at 128 B; 4p4s at 1 KiB).
    4. +
  4. Delete benchstream; JS ladder (1p, 4p, 4p×1 KiB, 2p2s +pull) at replicas=3 file.
    5. +
  5. Copy latency.mjs into 510; ping then flood at several +batch sizes.
    6. +
  6. Snapshot host/varz again.
    7. +
  7. Parse logs on this host; draw charts; write HTML +and PDF on this host.
    8. +
+

No publish, subscribe, chart, or PDF process runs on the operator +laptop for this study.

+

4.7 Instrumentation path

+
[NS1 host 70.88.205.138]
+   study-on-ns1.sh  (bash + python3)
+        |
+        | sudo pct exec 510
+        v
+[LXC 510 verae-px-worker 10.10.10.20]
+   nats bench / node latency.mjs
+        |
+        | NATS client protocol to
+        v
+[LXC 511/512/513  10.10.10.21-23 :4222]
+   nats-server -js  cluster routes :6222
+

The hypervisor issues the guest commands. The messages themselves +never leave vmbr1.

+

4.8 Threats to validity

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ThreatEffect on numbers
One physical hostThree “replicas” share CPU, memory, and usually the same datastore. +This measures process/LXC HA, not disk HA.
Shared loadNS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. +Load average during a run is part of the result, not noise to +ignore.
Single bench clientAll publishers live in 510. Per-publisher rate spread is contention +in that guest.
Short runsSeconds of traffic. No compaction, no multi-hour page-cache +eviction, no snapshot during load.
No TLS/nkeysProduction auth will cost CPU. Do not treat these rates as +post-nkeys rates.
Fan-out aggregateCore aggregate msgs/s counts pub+sub. Do not compare that column to +JetStream unique writes.
Flood ≠ RTTMixing flood averages with ping p99 produces a fake “NATS is slow” +story.
Lab onlyNot a Zapier HTTPS bench and not live +api.veraetime.net.
+

4.9 Ethics / safety

+

Bench uses throwaway subjects (bench.core.*, +bench.js.*, bench.lat.*) and a throwaway +stream. It does not purge product streams. Zapier cloud has no NATS +socket.

+
+

5. Suggestions for fine-tuning

+

These follow from the method and from the first ladder on this stand +(JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of +ms). Apply in order of leverage. Re-run this NS1 study +after each change so the delta is measured the same way.

+

5.1 Treat JetStream as +the product limiter

+

Product jobs/events/webhooks/archive are durable. Tuning core NATS to +2M msgs/s will not move a timestamp Zap. Put effort into replica +write path and consumer lag, not core +fan-out.

+

5.2 Split storage class by +stream

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
StreamSuggested storeWhy
ZAPIER_JOBSfile, r=3Work queue; lose-a-job is bad
ZAPIER_EVENTSfile r=3, or memory r=3 if events are rebuildable from job +statusHot waiters; measure both
ZAPIER_WEBHOOKSfile, r=3, workqueueHTTPS to Zapier is the slow consumer
ZAPIER_USAGEfile, r=3, limits + max-ageTelemetry
VERAE_ARCHIVEfile, r=3, on the best diskPuts are larger and must survive
+

Try ZAPIER_EVENTS as memory store in a maintenance +window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms +and durable events still ack at a higher rate, keep it; if a CT restart +drops in-flight waiters, revert.

+

5.3 Give JetStream real disks

+

Today r=3 on three LXC guests on one Proxmox host is +three files, one failure domain.

+
    +
  • Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) +into each nats LXC store_dir.
    • +
  • Set sync: always only on archive if you need it; +default sync is often enough for jobs and is faster. Measure.
    • +
  • Do not put JetStream store_dir on the same busy rootfs +as Forgejo/Caddy if we can avoid it.
    • +
  • When moving to three metal boxes: same configs, private NIC, one +disk (or mirror) per node. That is the first change +that makes r=3 mean “two boxes can die.”
    • +
+

5.4 Isolate the nats +CTs from the rest of NS1

+

Host load on this box is often already several. Pin:

+
    +
  • nats-a/b/c: dedicated cores, no steal from keep/fleet +Node processes.
    • +
  • Memory high enough that file-backed streams stay cache-hot for the +working set.
    • +
  • cpuunits / cpuset in pct config so a +Zapier-facing Node GC pause does not stall fsync.
    • +
+

Re-run this study after pinning; H1/H2 should move more than +ping.

+

5.5 Consumer and mailbox +tuning (delay H4)

+

Flood delay is backlog / consume_rate. Fine-tune the +waiters, not the broker RTT.

+
    +
  • jobs.events and webhooks.deliver: raise +max_ack_pending so a slow HTTPS hook does not stall the +whole consumer; cap it so a poison message cannot unbounded-buffer +RAM.
    • +
  • Pull consumers: larger batch, shorter expires, more +pullers horizontally (fleet replica floors) instead of one fat +subscriber.
    • +
  • Middleware should not flood-publish then wait; it +already does per-job publish. Keep that. The flood test is the outage +profile when a consumer is stopped.
    • +
  • Alert on consumer lag (pending + ack pending) from +JetStream, not on ping RTT.
    • +
+

5.6 Publisher-side +batching in middleware

+

A timestamp job is one small JSON. 16k msgs/s is ample. Still:

+
    +
  • Avoid per-byte publishes; one message per job/event.
    • +
  • Reuse NATS connections (connection churn showed up as publisher +spread in the core 4p/8p runs).
    • +
  • Idempotent msg id / duplicate window sized to Verae +retry window, not default-only.
    • +
+

5.7 +nats-server knobs worth measuring (A/B with this script)

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
KnobWhy try it
max_payloadKeep default unless archive puts grow
write_deadlineSlow consumer protection for webhooks
max_pendingBound memory on a stuck Zapier hook
max_connectionsFleet workers + keep + middleware
JetStream max_file_store / +max_memory_storePrevent one stream from filling the CT
max_outstanding_catchupReplica restart after a nats-c blip
GOMAXPROCS = LXC coresDo not overthread a 2-core CT
+

Change one knob, re-run +study-on-ns1.sh, compare JetStream 1p 128 B and ping +p99.

+

5.8 Network

+
    +
  • Keep NATS off vmbr0. No change.
    • +
  • When on metal: dedicated NIC or VLAN for cluster :6222 +vs client :4222 if possible (replication vs client +load).
    • +
  • Check virtio queue counts on the LXC nics if core 1 KiB byte rate +plateaus.
    • +
+

5.9 Security cost (when +nkeys/mTLS flip)

+

verae-nats-accounts is still a sketch. Enabling accounts +will add CPU on publish. Budget: re-run this exact study +after creds are in every NATS_URL, and +accept a drop on both core and JS. Do not flip without that +measurement.

+

5.10 Operational +fine-tuning (lag, not peak msgs/s)

+
    +
  1. Scrape varz / jsz from the host over +vmbr1 (not public). Monitor loopback :8222 is +invisible to Prometheus on NS1 unless we add a host-side proxy on +10.10.10.21:8222 bound only to vmbr1.
    2. +
  2. Keep replica floors for webhook-deliver and job-poller — they are +the flood defense.
    3. +
  3. Backup/restore drill of JetStream during idle, then +a short JS 1p run to see catchup cost.
    4. +
  4. A 15–30 minute soak (not in this ladder) for page cache and +compaction; add that as a third study when disks are dedicated.
    5. +
+

5.11 What not to tune

+
    +
  • Do not chase core 8p8s aggregate. It is fan-out on a lab +bridge.
    • +
  • Do not treat flood 400 ms as “cluster RTT.” Fix consumers.
    • +
  • Do not load-test on ZAPIER_* streams.
    • +
  • Do not bind client NATS to 0.0.0.0 on +vmbr0.
    • +
+ +
    +
  1. CPU pin nats-a/b/c → re-run JS 1p + ping.
    2. +
  2. ZAPIER_EVENTS-shaped memory stream vs file (throwaway +stream, same flags as this JS ladder).
    3. +
  3. Distinct store_dir disks per node.
    4. +
  4. nkeys on, same ladder.
    5. +
  5. Three hardware boxes, same cluster.env IPs +updated.
    6. +
+

Each experiment should produce a new +results/<utc>/ on NS1 and a new progress-repo report +so we can diff H1–H5 instead of arguing from memory.

+
+

6. Reproducing this study

+

On NS1 only:

+
cd ~/verae-src/verae-nats-cluster
+bash scripts/study-on-ns1.sh
+

The script exits if hostname is not NS1. Outputs land in +results/<utc>/ including +nats-cluster-bench-ns1.{md,html,pdf} and +charts/. Copy those into +zapier-decisions/reports/ for the progress repo and +catalog.

+

Raw logs for this run: results/20260912T051237Z/.

+ + diff --git a/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.md b/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.md new file mode 100644 index 0000000..cf0011d --- /dev/null +++ b/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.md @@ -0,0 +1,334 @@ +**Progress report (second study)** · run `20260912T051237Z` (UTC) + +> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): the orchestrator (`study-on-ns1.sh`), `nats bench`, `latency.mjs` (inside LXC 510 on this hypervisor), charting (`matplotlib`), and HTML/PDF (`pandoc` + `weasyprint`). The operator laptop did **not** publish, subscribe, draw charts, or render the PDF. Traffic stayed on `vmbr1` from LXC **510** to `nats-a/b/c` (**511–513**). + +This is a full methodology write-up plus the numbers from that on-host run. The earlier report (`nats-cluster-bench`, run `20260912T045131Z`) used the same cluster but was **orchestrated and rendered off-box**. Use this document when you need “it was all run on 138.” + +--- + +## 1. Executive summary + +| Item | This NS1-host run | +|------|-------------------| +| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `marchon` | +| Bench client | LXC 510 `verae-px-worker` | +| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` | +| Client URL | `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` | +| Host load before | `8.59 8.39 8.15 5/3843 4096277` | +| Host load after | `8.67 8.33 8.15 9/3863 4114865` | +| Core 1p1s 128 B pub | 502,502 msgs/s | +| JetStream 1p 128 B r=3 | 7,393 durable pubs/s | +| Ping p50 / p99 | 0.395ms / 1.377ms | + +Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst. + +--- + +## 2. Where it ran (and where it did not) + +```text +Operator laptop ──ssh──► NS1.GEORGELAMBERT.ORG 70.88.205.138 + study-on-ns1.sh + python3 build-ns1-study-report.py + sudo pct exec 510 ──► nats bench / latency.mjs + │ + ▼ vmbr1 + 10.10.10.21-23 :4222 +``` + +- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510). +- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file. + +--- + +## 3. Results (this run) + +### Host and brokers + +**Before** + +| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream | +|------|------|-------------|---------|----------|-----|-------|---------|-----------| +| nats-a | 511 | 2 | 1,721,203 | 4,933,759 | 1 | 1 | 26,398,720 | True | +| nats-b | 512 | 2 | 1,613,205 | 3,513,096 | 0 | 1 | 23,097,344 | True | +| nats-c | 513 | 2 | 1,760,973 | 4,048,582 | 2 | 1 | 23,867,392 | True | + +**After** + +| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream | +|------|------|-------------|---------|----------|-----|-------|---------|-----------| +| nats-a | 511 | 2 | 2,097,596 | 5,910,161 | 1 | 1 | 38,273,024 | True | +| nats-b | 512 | 2 | 2,121,816 | 4,746,539 | 1 | 1 | 41,361,408 | True | +| nats-c | 513 | 2 | 2,241,366 | 5,053,978 | 1 | 1 | 38,162,432 | True | + +nproc=40 · uname=`Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux` + +### Throughput + +| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s | +|-----|------|------------------|------------|----------|------------|----------| +| `core-1p1s-50k-128` | core pub/sub | 776,331 | 502,502 | 61.34 | 420,976 | 51.39 | +| `core-4p4s-100k-128` | core pub/sub | 1,278,454 | 379,985 | 46.38 | 1,078,910 | 131.70 | +| `core-4p4s-50k-1k` | core pub/sub | 581,464 | 209,545 | 204.63 | 482,645 | 471.33 | +| `core-8p8s-200k-128` | core pub/sub | 2,065,217 | 333,731 | 40.74 | 1,847,242 | 225.49 | +| `js-1p-20k-128-r3` | jetstream r=3 file | — | 7,393 | 0.90 | — | — | +| `js-2p2s-20k-128-r3` | jetstream r=3 file | 15,416 | 7,728 | 0.94 | 7,711 | 0.94 | +| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 14,985 | 14.63 | — | — | +| `js-4p-50k-128-r3` | jetstream r=3 file | — | 17,986 | 2.20 | — | — | + +### Round-trip delay + +| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max | +|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----| +| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.341ms | 0.470ms | 0.395ms | 0.651ms | 1.377ms | 3.243ms | +| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 132.425ms | 203.717ms | 204.458ms | 241.750ms | 249.468ms | 249.640ms | +| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 196.422ms | 221.264ms | 224.039ms | 233.180ms | 236.798ms | 238.177ms | +| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 211.108ms | 294.925ms | 300.254ms | 314.223ms | 315.551ms | 316.091ms | +| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 280.608ms | 414.972ms | 409.802ms | 513.821ms | 535.394ms | 536.084ms | + +### Core NATS + +![Core NATS throughput at four loads (NS1 host run)](charts/core-throughput.png) + +*Core NATS throughput at four loads (NS1 host run)* +### Payload size (core) + +![Core NATS 128 B vs 1 KiB (NS1 host run)](charts/payload-size.png) + +*Core NATS 128 B vs 1 KiB (NS1 host run)* +### JetStream r=3 file + +![JetStream durable publish rate (NS1 host run)](charts/js-throughput.png) + +*JetStream durable publish rate (NS1 host run)* +### Core vs JetStream + +![Core vs JetStream publish rate, log scale (NS1 host run)](charts/core-vs-js.png) + +*Core vs JetStream publish rate, log scale (NS1 host run)* +### Delay + +![Ping vs flood delay percentiles, log scale (NS1 host run)](charts/delay-percentiles.png) + +*Ping vs flood delay percentiles, log scale (NS1 host run)* + +--- + +## 4. Study methodology + +### 4.1 Question + +On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)? + +### 4.2 Hypotheses (stated before the run) + +1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica. +2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated. +3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting. +4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count. +5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound. + +### 4.3 Independent variables (what we changed) + +| Factor | Levels | +|--------|--------| +| Transport | Core NATS pub/sub vs JetStream file replicas=3 | +| Publisher count | 1, 2, 4, 8 | +| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 | +| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) | +| Payload | 128 B, 1024 B | +| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) | + +### 4.4 Dependent variables (what we recorded) + +| Metric | Instrument | Unit | +|--------|------------|------| +| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s | +| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s | +| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) | +| Publisher spread | nats min/avg/max **msgs/s** | not delay | +| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max | +| Host load | `/proc/loadavg` before and after | load average | +| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem | + +**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`. + +### 4.5 Controls and constants + +- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`. +- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`). +- Bench client is LXC **510**, not a nats-* server. +- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide. +- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`). +- No TLS, no nkeys, no account isolation (isolation is `vmbr1`). +- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder. + +### 4.6 Procedure + +1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise. +2. Snapshot host load, memory, LXC configs, and each nats `varz`. +3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB). +4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file. +5. Copy `latency.mjs` into 510; ping then flood at several batch sizes. +6. Snapshot host/`varz` again. +7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**. + +No publish, subscribe, chart, or PDF process runs on the operator laptop for this study. + +### 4.7 Instrumentation path + +```text +[NS1 host 70.88.205.138] + study-on-ns1.sh (bash + python3) + | + | sudo pct exec 510 + v +[LXC 510 verae-px-worker 10.10.10.20] + nats bench / node latency.mjs + | + | NATS client protocol to + v +[LXC 511/512/513 10.10.10.21-23 :4222] + nats-server -js cluster routes :6222 +``` + +The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`. + +### 4.8 Threats to validity + +| Threat | Effect on numbers | +|--------|-------------------| +| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. | +| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. | +| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. | +| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. | +| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. | +| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. | +| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. | +| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. | + +### 4.9 Ethics / safety + +Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket. + +--- + +## 5. Suggestions for fine-tuning + +These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way. + +### 5.1 Treat JetStream as the product limiter + +Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out. + +### 5.2 Split storage class by stream + +| Stream | Suggested store | Why | +|--------|-----------------|-----| +| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad | +| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both | +| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer | +| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry | +| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive | + +Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert. + +### 5.3 Give JetStream real disks + +Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain. + +- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`. +- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure. +- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it. +- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.” + +### 5.4 Isolate the nats CTs from the rest of NS1 + +Host load on this box is often already several. Pin: + +- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes. +- Memory high enough that file-backed streams stay cache-hot for the working set. +- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync. + +Re-run this study after pinning; H1/H2 should move more than ping. + +### 5.5 Consumer and mailbox tuning (delay H4) + +Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT. + +- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM. +- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber. +- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped. +- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT. + +### 5.6 Publisher-side batching in middleware + +A timestamp job is one small JSON. 16k msgs/s is ample. Still: + +- Avoid per-byte publishes; one message per job/event. +- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs). +- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only. + +### 5.7 nats-server knobs worth measuring (A/B with this script) + +| Knob | Why try it | +|------|------------| +| `max_payload` | Keep default unless archive puts grow | +| `write_deadline` | Slow consumer protection for webhooks | +| `max_pending` | Bound memory on a stuck Zapier hook | +| `max_connections` | Fleet workers + keep + middleware | +| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT | +| `max_outstanding_catchup` | Replica restart after a nats-c blip | +| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT | + +Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99. + +### 5.8 Network + +- Keep NATS off `vmbr0`. No change. +- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load). +- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus. + +### 5.9 Security cost (when nkeys/mTLS flip) + +`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement. + +### 5.10 Operational fine-tuning (lag, not peak msgs/s) + +1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`. +2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense. +3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost. +4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated. + +### 5.11 What not to tune + +- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge. +- Do not treat flood 400 ms as “cluster RTT.” Fix consumers. +- Do not load-test on `ZAPIER_*` streams. +- Do not bind client NATS to `0.0.0.0` on `vmbr0`. + +### 5.12 Recommended next experiments (same method, one change each) + +1. CPU pin nats-a/b/c → re-run JS 1p + ping. +2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder). +3. Distinct `store_dir` disks per node. +4. nkeys on, same ladder. +5. Three hardware boxes, same `cluster.env` IPs updated. + +Each experiment should produce a new `results//` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory. + + +--- + +## 6. Reproducing this study + +On **NS1 only**: + +```bash +cd ~/verae-src/verae-nats-cluster +bash scripts/study-on-ns1.sh +``` + +The script exits if `hostname` is not NS1. Outputs land in `results//` including `nats-cluster-bench-ns1.{md,html,pdf}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog. + +Raw logs for this run: `results/20260912T051237Z/`. diff --git a/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.pdf b/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.pdf new file mode 100644 index 0000000..c25d95e Binary files /dev/null and b/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.pdf differ diff --git a/scripts/build-docs-site.py b/scripts/build-docs-site.py index abc5c29..0fe838f 100755 --- a/scripts/build-docs-site.py +++ b/scripts/build-docs-site.py @@ -862,6 +862,10 @@ def main() -> None: "packages/zapier-decisions/reports/nats-cluster-bench.pdf", "NATS cluster speed report (charts, 2026-09-12)", ), + ( + "packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.pdf", + "NATS cluster study run entirely on NS1 (methodology + tuning)", + ), ("packages/verae-nats-cluster/BENCH.pdf", "NATS cluster throughput tables (2026-09-12)"), ("overview/README.pdf", "System overview"), ("overview/INDEX.pdf", "Documentation index"), @@ -879,6 +883,10 @@ def main() -> None: "packages/zapier-decisions/reports/nats-cluster-bench.html", "NATS cluster speed report (charts)", ), + ( + "packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.html", + "NATS cluster study run entirely on NS1", + ), ("packages/verae-nats-cluster/BENCH.html", "NATS cluster throughput tables"), ("docs-master/MESSAGE-FLOWS.html", "Numbered message flows"), ("docs-master/modules-and-nats.html", "NATS address table"),