Add exhaustive NATS factorial, MQTT/UDP probes, and optimal-config report.
Some checks are pending
offline / test (push) Waiting to run
Some checks are pending
offline / test (push) Waiting to run
Compares r=1 vs r=3, file vs memory, reconnect tax, and projects three HP DL360 Gen10 NVMe + 10GbE boxes. Keep NATS; do not switch to MQTT/UDP.
This commit is contained in:
parent
a7a7ec86ce
commit
c83f1c6717
77 changed files with 5089 additions and 4 deletions
|
|
@ -54,6 +54,9 @@ bash scripts/study-on-ns1.sh
|
||||||
|
|
||||||
# Maximize cores/RAM + tmpfs JetStream, same ladder, restore ZFS after:
|
# Maximize cores/RAM + tmpfs JetStream, same ladder, restore ZFS after:
|
||||||
bash scripts/maximize-ns1-study.sh
|
bash scripts/maximize-ns1-study.sh
|
||||||
|
|
||||||
|
# Exhaustive factorial (r=1/r=3, file/memory, reconnect, UDP, MQTT) + combined report:
|
||||||
|
bash scripts/exhaustive-ns1-study.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
That writes `results/<utc>/` and copies a summary to [BENCH.md](BENCH.md). Full write-up with charts (HTML + PDF) is in the progress repo: [nats-cluster-bench](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.pdf). Core NATS is fire-and-forget; JetStream rows use **file** storage and **replicas=3**. Ping delay is sequential RTT; flood delay is queueing under burst.
|
That writes `results/<utc>/` and copies a summary to [BENCH.md](BENCH.md). Full write-up with charts (HTML + PDF) is in the progress repo: [nats-cluster-bench](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.pdf). Core NATS is fire-and-forget; JetStream rows use **file** storage and **replicas=3**. Ping delay is sequential RTT; flood delay is queueing under burst.
|
||||||
|
|
|
||||||
|
|
@ -6,3 +6,5 @@
|
||||||
!20260912T051237Z/**
|
!20260912T051237Z/**
|
||||||
!20260912T053120Z/
|
!20260912T053120Z/
|
||||||
!20260912T053120Z/**
|
!20260912T053120Z/**
|
||||||
|
!20260912T055851Z/
|
||||||
|
!20260912T055851Z/**
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,58 @@
|
||||||
|
# NATS cluster message speed
|
||||||
|
|
||||||
|
Run **`20260912T055851Z`** (UTC). Client: LXC **510** `verae-px-worker` (`10.10.10.20`), not a nats-* server. Servers: `nats-a/b/c` on `10.10.10.21–23` (`vmbr1` only).
|
||||||
|
|
||||||
|
Client URL:
|
||||||
|
|
||||||
|
```text
|
||||||
|
nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
|
||||||
|
```
|
||||||
|
|
||||||
|
## Method
|
||||||
|
|
||||||
|
- **Core NATS** is fire-and-forget pub/sub (`nats bench`). No disk, no replica ack.
|
||||||
|
- **JetStream** uses **file** storage and **replicas=3** (same as product streams). The unique stream `benchstream` is deleted between JS loads.
|
||||||
|
- Throughput is **msgs/sec** from nats CLI **0.1.6** (`--no-progress --csv`). Its min/avg/max are publisher **rate spread**, not delay.
|
||||||
|
- **Ping** delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster.
|
||||||
|
- **Flood** delay: N publishers dump the whole batch, then the subscriber drains. This is **queueing under burst**, not wire RTT.
|
||||||
|
- Probe: `scripts/latency.mjs` (two connections, header timestamp).
|
||||||
|
|
||||||
|
## Throughput
|
||||||
|
|
||||||
|
| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
|
||||||
|
|-----|------|------------------|------------|----------|------------|----------|
|
||||||
|
| `core-1p1s-50k-128` | core pub/sub | 835,602 | 662,227 | 80.84 | 472,239 | 57.65 |
|
||||||
|
| `core-4p4s-100k-128` | core pub/sub | 1,597,284 | 692,109 | 84.49 | 1,286,093 | 156.99 |
|
||||||
|
| `core-4p4s-50k-1k` | core pub/sub | 595,555 | 189,939 | 185.49 | 479,898 | 468.65 |
|
||||||
|
| `core-8p8s-200k-128` | core pub/sub | 1,760,599 | 223,078 | 27.23 | 1,570,171 | 191.67 |
|
||||||
|
| `js-1p-20k-128-r3` | jetstream r=3 file | — | 14,330 | 1.75 | — | — |
|
||||||
|
| `js-2p2s-20k-128-r3` | jetstream r=3 file | 18,449 | 9,252 | 1.13 | 9,228 | 1.13 |
|
||||||
|
| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 15,197 | 14.84 | — | — |
|
||||||
|
| `js-4p-50k-128-r3` | jetstream r=3 file | — | 19,232 | 2.35 | — | — |
|
||||||
|
| `js-file-1p-20k-128-r1` | jetstream r=3 file | — | 18,888 | 2.31 | — | — |
|
||||||
|
| `js-file-1p-20k-4k-r3` | jetstream r=3 file | — | 8,673 | 33.88 | — | — |
|
||||||
|
| `js-file-4p-50k-128-r1` | jetstream r=3 file | — | 24,560 | 3.00 | — | — |
|
||||||
|
| `js-mem-1p-20k-128-r1` | jetstream r=3 file | — | 29,972 | 3.66 | — | — |
|
||||||
|
| `js-mem-1p-20k-128-r3` | jetstream r=3 file | — | 20,188 | 2.46 | — | — |
|
||||||
|
| `js-mem-4p-20k-1k-r3` | jetstream r=3 file | — | 33,916 | 33.12 | — | — |
|
||||||
|
| `js-mem-4p-50k-128-r1` | jetstream r=3 file | — | 64,923 | 7.93 | — | — |
|
||||||
|
| `js-mem-4p-50k-128-r3` | jetstream r=3 file | — | 37,736 | 4.61 | — | — |
|
||||||
|
|
||||||
|
## Round-trip delay
|
||||||
|
|
||||||
|
| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
|
||||||
|
|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|
|
||||||
|
| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.340ms | 0.530ms | 0.456ms | 0.827ms | 1.140ms | 2.910ms |
|
||||||
|
| `lat-reconnect-200-128` | flood (burst queueing) | 200 | 1 | 128 B | 0.366ms | 0.540ms | 0.503ms | 0.619ms | 1.750ms | 3.206ms |
|
||||||
|
| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 98.399ms | 124.452ms | 125.413ms | 140.165ms | 145.770ms | 146.002ms |
|
||||||
|
| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 125.336ms | 157.915ms | 157.392ms | 176.826ms | 178.015ms | 178.776ms |
|
||||||
|
| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 157.463ms | 205.870ms | 207.044ms | 237.035ms | 239.449ms | 240.473ms |
|
||||||
|
| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 230.898ms | 361.458ms | 372.512ms | 443.799ms | 458.198ms | 460.270ms |
|
||||||
|
|
||||||
|
## What the numbers mean
|
||||||
|
|
||||||
|
Product job/event/archive traffic is **JetStream r=3 file**. On this three-LXC stand that is about **16k durable 128 B pubs/s** (about **13k** at 1 KiB). Core NATS is an upper bound for non-durable fan-out: about **0.7–2.0M msgs/s** aggregate at 128 B, or **~630k msgs/s (~616 MB/s)** at 1 KiB with 4 publishers.
|
||||||
|
|
||||||
|
A quiet request-reply is **~0.3 ms** average, **p99 < 1 ms**. Flood rows in the **150–500 ms** band are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers.
|
||||||
|
|
||||||
|
Re-run on NS1: `bash scripts/bench.sh`. Raw logs/CSVs are under `results/<utc>/`.
|
||||||
Binary file not shown.
|
After Width: | Height: | Size: 25 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 27 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 54 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 40 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 45 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 50 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 59 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 56 KiB |
|
|
@ -0,0 +1,3 @@
|
||||||
|
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||||
|
aWd091p1c7cXCtuUvyZSxt,S0,50000,6400000,472239,60446688.561339,0.105878
|
||||||
|
aWd091p1c7cXCtuUvyZSxt,P0,50000,6400000,662227,84765130.062490,0.075503
|
||||||
|
|
|
@ -0,0 +1,7 @@
|
||||||
|
=== core-1p1s-50k-128 ===
|
||||||
|
|
||||||
|
NATS Pub/Sub stats: 835,602 msgs/sec ~ 102.00 MB/sec
|
||||||
|
Pub stats: 662,227 msgs/sec ~ 80.84 MB/sec
|
||||||
|
Sub stats: 472,239 msgs/sec ~ 57.65 MB/sec
|
||||||
|
|
||||||
|
Saved metric data in csv file /tmp/bench.csv
|
||||||
|
|
@ -0,0 +1,9 @@
|
||||||
|
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||||
|
MH2k3yW0YONZD0HiNpchpL,S0,100000,12800000,409118,52367181.312369,0.244428
|
||||||
|
MH2k3yW0YONZD0HiNpchpL,S1,100000,12800000,321711,41179038.222383,0.310838
|
||||||
|
MH2k3yW0YONZD0HiNpchpL,S2,100000,12800000,321716,41179732.151708,0.310833
|
||||||
|
MH2k3yW0YONZD0HiNpchpL,S3,100000,12800000,321573,41161354.894041,0.310971
|
||||||
|
MH2k3yW0YONZD0HiNpchpL,P0,25000,3200000,264684,33879607.417508,0.094452
|
||||||
|
MH2k3yW0YONZD0HiNpchpL,P1,25000,3200000,243211,31131121.677273,0.102791
|
||||||
|
MH2k3yW0YONZD0HiNpchpL,P2,25000,3200000,183537,23492805.654965,0.136212
|
||||||
|
MH2k3yW0YONZD0HiNpchpL,P3,25000,3200000,173063,22152095.916748,0.144456
|
||||||
|
|
|
@ -0,0 +1,17 @@
|
||||||
|
=== core-4p4s-100k-128 ===
|
||||||
|
|
||||||
|
NATS Pub/Sub stats: 1,597,284 msgs/sec ~ 194.98 MB/sec
|
||||||
|
Pub stats: 692,109 msgs/sec ~ 84.49 MB/sec
|
||||||
|
[1] 264,684 msgs/sec ~ 32.31 MB/sec (25000 msgs)
|
||||||
|
[2] 243,211 msgs/sec ~ 29.69 MB/sec (25000 msgs)
|
||||||
|
[3] 183,537 msgs/sec ~ 22.40 MB/sec (25000 msgs)
|
||||||
|
[4] 173,063 msgs/sec ~ 21.13 MB/sec (25000 msgs)
|
||||||
|
min 173,063 | avg 216,123 | max 264,684 | stddev 38,755 msgs
|
||||||
|
Sub stats: 1,286,093 msgs/sec ~ 156.99 MB/sec
|
||||||
|
[1] 409,118 msgs/sec ~ 49.94 MB/sec (100000 msgs)
|
||||||
|
[2] 321,711 msgs/sec ~ 39.27 MB/sec (100000 msgs)
|
||||||
|
[3] 321,716 msgs/sec ~ 39.27 MB/sec (100000 msgs)
|
||||||
|
[4] 321,573 msgs/sec ~ 39.25 MB/sec (100000 msgs)
|
||||||
|
min 321,573 | avg 343,529 | max 409,118 | stddev 37,867 msgs
|
||||||
|
|
||||||
|
Saved metric data in csv file /tmp/bench.csv
|
||||||
|
|
@ -0,0 +1,9 @@
|
||||||
|
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||||
|
I0O2O7NFVVnTrSOBs8Q38n,S0,50000,51200000,171931,176057368.403530,0.290814
|
||||||
|
I0O2O7NFVVnTrSOBs8Q38n,S1,50000,51200000,127850,130919309.123975,0.391081
|
||||||
|
I0O2O7NFVVnTrSOBs8Q38n,S2,50000,51200000,120555,123448409.014747,0.414748
|
||||||
|
I0O2O7NFVVnTrSOBs8Q38n,S3,50000,51200000,120019,122899963.083444,0.416599
|
||||||
|
I0O2O7NFVVnTrSOBs8Q38n,P0,12500,12800000,101551,103988522.006912,0.123091
|
||||||
|
I0O2O7NFVVnTrSOBs8Q38n,P1,12500,12800000,49245,50427332.045753,0.253831
|
||||||
|
I0O2O7NFVVnTrSOBs8Q38n,P2,12500,12800000,48521,49686273.879592,0.257616
|
||||||
|
I0O2O7NFVVnTrSOBs8Q38n,P3,12500,12800000,47497,48637215.616680,0.263173
|
||||||
|
|
|
@ -0,0 +1,17 @@
|
||||||
|
=== core-4p4s-50k-1k ===
|
||||||
|
|
||||||
|
NATS Pub/Sub stats: 595,555 msgs/sec ~ 581.60 MB/sec
|
||||||
|
Pub stats: 189,939 msgs/sec ~ 185.49 MB/sec
|
||||||
|
[1] 101,551 msgs/sec ~ 99.17 MB/sec (12500 msgs)
|
||||||
|
[2] 49,245 msgs/sec ~ 48.09 MB/sec (12500 msgs)
|
||||||
|
[3] 48,521 msgs/sec ~ 47.38 MB/sec (12500 msgs)
|
||||||
|
[4] 47,497 msgs/sec ~ 46.38 MB/sec (12500 msgs)
|
||||||
|
min 47,497 | avg 61,703 | max 101,551 | stddev 23,014 msgs
|
||||||
|
Sub stats: 479,898 msgs/sec ~ 468.65 MB/sec
|
||||||
|
[1] 171,931 msgs/sec ~ 167.90 MB/sec (50000 msgs)
|
||||||
|
[2] 127,850 msgs/sec ~ 124.85 MB/sec (50000 msgs)
|
||||||
|
[3] 120,555 msgs/sec ~ 117.73 MB/sec (50000 msgs)
|
||||||
|
[4] 120,019 msgs/sec ~ 117.21 MB/sec (50000 msgs)
|
||||||
|
min 120,019 | avg 135,088 | max 171,931 | stddev 21,494 msgs
|
||||||
|
|
||||||
|
Saved metric data in csv file /tmp/bench.csv
|
||||||
|
|
@ -0,0 +1,17 @@
|
||||||
|
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||||
|
PZHMggVrmAZAaABqkdVm12,S0,200000,25600000,221384,28337251.548719,0.903404
|
||||||
|
PZHMggVrmAZAaABqkdVm12,S1,200000,25600000,221400,28339297.871777,0.903339
|
||||||
|
PZHMggVrmAZAaABqkdVm12,S2,200000,25600000,221358,28333840.935200,0.903513
|
||||||
|
PZHMggVrmAZAaABqkdVm12,S3,200000,25600000,196749,25183987.052629,1.016519
|
||||||
|
PZHMggVrmAZAaABqkdVm12,S4,200000,25600000,196654,25171832.144418,1.017010
|
||||||
|
PZHMggVrmAZAaABqkdVm12,S5,200000,25600000,196517,25154300.186621,1.017719
|
||||||
|
PZHMggVrmAZAaABqkdVm12,S6,200000,25600000,196490,25150741.114236,1.017863
|
||||||
|
PZHMggVrmAZAaABqkdVm12,S7,200000,25600000,196395,25138656.210795,1.018352
|
||||||
|
PZHMggVrmAZAaABqkdVm12,P0,25000,3200000,169409,21684439.538250,0.147571
|
||||||
|
PZHMggVrmAZAaABqkdVm12,P1,25000,3200000,137478,17597297.160740,0.181846
|
||||||
|
PZHMggVrmAZAaABqkdVm12,P2,25000,3200000,109841,14059658.143542,0.227602
|
||||||
|
PZHMggVrmAZAaABqkdVm12,P3,25000,3200000,96040,12293171.802285,0.260307
|
||||||
|
PZHMggVrmAZAaABqkdVm12,P4,25000,3200000,75019,9602540.006267,0.333245
|
||||||
|
PZHMggVrmAZAaABqkdVm12,P5,25000,3200000,73607,9421821.419509,0.339637
|
||||||
|
PZHMggVrmAZAaABqkdVm12,P6,25000,3200000,27890,3569920.156650,0.896379
|
||||||
|
PZHMggVrmAZAaABqkdVm12,P7,25000,3200000,27932,3575348.254450,0.895018
|
||||||
|
|
|
@ -0,0 +1,25 @@
|
||||||
|
=== core-8p8s-200k-128 ===
|
||||||
|
|
||||||
|
NATS Pub/Sub stats: 1,760,599 msgs/sec ~ 214.92 MB/sec
|
||||||
|
Pub stats: 223,078 msgs/sec ~ 27.23 MB/sec
|
||||||
|
[1] 169,409 msgs/sec ~ 20.68 MB/sec (25000 msgs)
|
||||||
|
[2] 137,478 msgs/sec ~ 16.78 MB/sec (25000 msgs)
|
||||||
|
[3] 109,841 msgs/sec ~ 13.41 MB/sec (25000 msgs)
|
||||||
|
[4] 96,040 msgs/sec ~ 11.72 MB/sec (25000 msgs)
|
||||||
|
[5] 75,019 msgs/sec ~ 9.16 MB/sec (25000 msgs)
|
||||||
|
[6] 73,607 msgs/sec ~ 8.99 MB/sec (25000 msgs)
|
||||||
|
[7] 27,890 msgs/sec ~ 3.40 MB/sec (25000 msgs)
|
||||||
|
[8] 27,932 msgs/sec ~ 3.41 MB/sec (25000 msgs)
|
||||||
|
min 27,890 | avg 89,652 | max 169,409 | stddev 46,357 msgs
|
||||||
|
Sub stats: 1,570,171 msgs/sec ~ 191.67 MB/sec
|
||||||
|
[1] 221,384 msgs/sec ~ 27.02 MB/sec (200000 msgs)
|
||||||
|
[2] 221,400 msgs/sec ~ 27.03 MB/sec (200000 msgs)
|
||||||
|
[3] 221,358 msgs/sec ~ 27.02 MB/sec (200000 msgs)
|
||||||
|
[4] 196,749 msgs/sec ~ 24.02 MB/sec (200000 msgs)
|
||||||
|
[5] 196,654 msgs/sec ~ 24.01 MB/sec (200000 msgs)
|
||||||
|
[6] 196,517 msgs/sec ~ 23.99 MB/sec (200000 msgs)
|
||||||
|
[7] 196,490 msgs/sec ~ 23.99 MB/sec (200000 msgs)
|
||||||
|
[8] 196,395 msgs/sec ~ 23.97 MB/sec (200000 msgs)
|
||||||
|
min 196,395 | avg 205,868 | max 221,400 | stddev 12,016 msgs
|
||||||
|
|
||||||
|
Saved metric data in csv file /tmp/bench.csv
|
||||||
|
|
@ -0,0 +1,92 @@
|
||||||
|
execution_host=NS1.GEORGELAMBERT.ORG
|
||||||
|
execution_ip=70.88.205.138
|
||||||
|
hostname=NS1.GEORGELAMBERT.ORG
|
||||||
|
utc=2026-09-12T06:01:17Z
|
||||||
|
whoami=marchon
|
||||||
|
pwd=/home/marchon/verae-src/verae-nats-cluster
|
||||||
|
uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux
|
||||||
|
nproc=40
|
||||||
|
loadavg=8.48 8.52 8.17 5/3860 657839
|
||||||
|
client_vmid=510
|
||||||
|
nats_url=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
|
||||||
|
js_extra_memory=1
|
||||||
|
compare_dir=/home/marchon/verae-src/verae-nats-cluster/results/20260912T051237Z
|
||||||
|
--- nats 511 max_mem ---
|
||||||
|
max_mem: 8G
|
||||||
|
--- nats 511 jetstream mount ---
|
||||||
|
jetstream on rootfs
|
||||||
|
--- free ---
|
||||||
|
total used free shared buff/cache available
|
||||||
|
Mem: 377Gi 38Gi 324Gi 4.3Gi 21Gi 338Gi
|
||||||
|
Swap: 8.0Gi 0B 8.0Gi
|
||||||
|
--- pct list ---
|
||||||
|
VMID Status Lock Name
|
||||||
|
100 stopped CT100
|
||||||
|
101 stopped CT101
|
||||||
|
102 stopped CT102
|
||||||
|
103 stopped CT103
|
||||||
|
104 stopped CT104
|
||||||
|
105 stopped CT105
|
||||||
|
106 stopped CT106
|
||||||
|
107 stopped CT107
|
||||||
|
108 stopped CT108
|
||||||
|
109 stopped CT109
|
||||||
|
110 stopped CT110
|
||||||
|
111 stopped CT111
|
||||||
|
112 stopped CT112
|
||||||
|
113 stopped CT113
|
||||||
|
114 stopped CT114
|
||||||
|
115 stopped CT115
|
||||||
|
116 stopped paul
|
||||||
|
117 stopped CT117
|
||||||
|
118 stopped CT118
|
||||||
|
119 stopped CT119
|
||||||
|
120 stopped rustdeskserver
|
||||||
|
121 stopped CT121
|
||||||
|
122 stopped CT122
|
||||||
|
123 stopped CT123
|
||||||
|
124 stopped bdross
|
||||||
|
200 stopped CT200
|
||||||
|
201 stopped CT201
|
||||||
|
202 stopped CT202
|
||||||
|
300 running mail.veraetime.info
|
||||||
|
400 stopped evercycle-api
|
||||||
|
501 running pki
|
||||||
|
502 running peergos-a
|
||||||
|
503 running peergos-b
|
||||||
|
504 stopped minio
|
||||||
|
510 running verae-px-worker
|
||||||
|
511 running nats-a
|
||||||
|
512 running nats-b
|
||||||
|
513 running nats-c
|
||||||
|
2000 stopped CT2000
|
||||||
|
2001 stopped CT2001
|
||||||
|
2002 stopped CT2002
|
||||||
|
--- pct config 510 ---
|
||||||
|
cores: 8
|
||||||
|
hostname: verae-px-worker
|
||||||
|
memory: 16384
|
||||||
|
net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:BD:5D:62,ip=10.10.10.20/24,type=veth
|
||||||
|
rootfs: SSD2:subvol-510-disk-0,size=16G
|
||||||
|
swap: 512
|
||||||
|
--- pct config 511 ---
|
||||||
|
cores: 8
|
||||||
|
hostname: nats-a
|
||||||
|
memory: 16384
|
||||||
|
net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:CF:AC:68,ip=10.10.10.21/24,type=veth
|
||||||
|
rootfs: SSD2:subvol-511-disk-0,size=8G
|
||||||
|
swap: 256
|
||||||
|
--- pct config 512 ---
|
||||||
|
cores: 8
|
||||||
|
hostname: nats-b
|
||||||
|
memory: 16384
|
||||||
|
net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:42:73:A2,ip=10.10.10.22/24,type=veth
|
||||||
|
rootfs: SSD2:subvol-512-disk-0,size=8G
|
||||||
|
swap: 256
|
||||||
|
--- pct config 513 ---
|
||||||
|
cores: 8
|
||||||
|
hostname: nats-c
|
||||||
|
memory: 16384
|
||||||
|
net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:1B:9B:DC,ip=10.10.10.23/24,type=veth
|
||||||
|
rootfs: SSD2:subvol-513-disk-0,size=8G
|
||||||
|
swap: 256
|
||||||
|
|
@ -0,0 +1,92 @@
|
||||||
|
execution_host=NS1.GEORGELAMBERT.ORG
|
||||||
|
execution_ip=70.88.205.138
|
||||||
|
hostname=NS1.GEORGELAMBERT.ORG
|
||||||
|
utc=2026-09-12T05:58:55Z
|
||||||
|
whoami=marchon
|
||||||
|
pwd=/home/marchon/verae-src/verae-nats-cluster
|
||||||
|
uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux
|
||||||
|
nproc=40
|
||||||
|
loadavg=8.70 8.15 8.00 4/3842 621042
|
||||||
|
client_vmid=510
|
||||||
|
nats_url=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
|
||||||
|
js_extra_memory=1
|
||||||
|
compare_dir=/home/marchon/verae-src/verae-nats-cluster/results/20260912T051237Z
|
||||||
|
--- nats 511 max_mem ---
|
||||||
|
max_mem: 8G
|
||||||
|
--- nats 511 jetstream mount ---
|
||||||
|
jetstream on rootfs
|
||||||
|
--- free ---
|
||||||
|
total used free shared buff/cache available
|
||||||
|
Mem: 377Gi 38Gi 324Gi 4.3Gi 21Gi 339Gi
|
||||||
|
Swap: 8.0Gi 0B 8.0Gi
|
||||||
|
--- pct list ---
|
||||||
|
VMID Status Lock Name
|
||||||
|
100 stopped CT100
|
||||||
|
101 stopped CT101
|
||||||
|
102 stopped CT102
|
||||||
|
103 stopped CT103
|
||||||
|
104 stopped CT104
|
||||||
|
105 stopped CT105
|
||||||
|
106 stopped CT106
|
||||||
|
107 stopped CT107
|
||||||
|
108 stopped CT108
|
||||||
|
109 stopped CT109
|
||||||
|
110 stopped CT110
|
||||||
|
111 stopped CT111
|
||||||
|
112 stopped CT112
|
||||||
|
113 stopped CT113
|
||||||
|
114 stopped CT114
|
||||||
|
115 stopped CT115
|
||||||
|
116 stopped paul
|
||||||
|
117 stopped CT117
|
||||||
|
118 stopped CT118
|
||||||
|
119 stopped CT119
|
||||||
|
120 stopped rustdeskserver
|
||||||
|
121 stopped CT121
|
||||||
|
122 stopped CT122
|
||||||
|
123 stopped CT123
|
||||||
|
124 stopped bdross
|
||||||
|
200 stopped CT200
|
||||||
|
201 stopped CT201
|
||||||
|
202 stopped CT202
|
||||||
|
300 running mail.veraetime.info
|
||||||
|
400 stopped evercycle-api
|
||||||
|
501 running pki
|
||||||
|
502 running peergos-a
|
||||||
|
503 running peergos-b
|
||||||
|
504 stopped minio
|
||||||
|
510 running verae-px-worker
|
||||||
|
511 running nats-a
|
||||||
|
512 running nats-b
|
||||||
|
513 running nats-c
|
||||||
|
2000 stopped CT2000
|
||||||
|
2001 stopped CT2001
|
||||||
|
2002 stopped CT2002
|
||||||
|
--- pct config 510 ---
|
||||||
|
cores: 8
|
||||||
|
hostname: verae-px-worker
|
||||||
|
memory: 16384
|
||||||
|
net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:BD:5D:62,ip=10.10.10.20/24,type=veth
|
||||||
|
rootfs: SSD2:subvol-510-disk-0,size=16G
|
||||||
|
swap: 512
|
||||||
|
--- pct config 511 ---
|
||||||
|
cores: 8
|
||||||
|
hostname: nats-a
|
||||||
|
memory: 16384
|
||||||
|
net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:CF:AC:68,ip=10.10.10.21/24,type=veth
|
||||||
|
rootfs: SSD2:subvol-511-disk-0,size=8G
|
||||||
|
swap: 256
|
||||||
|
--- pct config 512 ---
|
||||||
|
cores: 8
|
||||||
|
hostname: nats-b
|
||||||
|
memory: 16384
|
||||||
|
net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:42:73:A2,ip=10.10.10.22/24,type=veth
|
||||||
|
rootfs: SSD2:subvol-512-disk-0,size=8G
|
||||||
|
swap: 256
|
||||||
|
--- pct config 513 ---
|
||||||
|
cores: 8
|
||||||
|
hostname: nats-c
|
||||||
|
memory: 16384
|
||||||
|
net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:1B:9B:DC,ip=10.10.10.23/24,type=veth
|
||||||
|
rootfs: SSD2:subvol-513-disk-0,size=8G
|
||||||
|
swap: 256
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||||
|
vSyjynJZ5N37lo74IJMJLi,P0,40000,2560000,14330,1834259.456665,1.395659
|
||||||
|
|
|
@ -0,0 +1,5 @@
|
||||||
|
=== js-1p-20k-128-r3 ===
|
||||||
|
|
||||||
|
Pub stats: 14,330 msgs/sec ~ 1.75 MB/sec
|
||||||
|
|
||||||
|
Saved metric data in csv file /tmp/bench.csv
|
||||||
|
|
@ -0,0 +1,5 @@
|
||||||
|
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||||
|
rx33qiX4N854UL4s2fDox5,S0,20234,1280000,4748,607822.981024,2.105876
|
||||||
|
rx33qiX4N854UL4s2fDox5,S1,20236,1280000,4614,590640.769582,2.167138
|
||||||
|
rx33qiX4N854UL4s2fDox5,P0,20000,1280000,5286,676626.786347,1.891737
|
||||||
|
rx33qiX4N854UL4s2fDox5,P1,20000,1280000,4626,592132.585621,2.161678
|
||||||
|
|
|
@ -0,0 +1,13 @@
|
||||||
|
=== js-2p2s-20k-128-r3 ===
|
||||||
|
|
||||||
|
NATS Pub/Sub stats: 18,449 msgs/sec ~ 2.25 MB/sec
|
||||||
|
Pub stats: 9,252 msgs/sec ~ 1.13 MB/sec
|
||||||
|
[1] 5,286 msgs/sec ~ 660.77 KB/sec (10000 msgs)
|
||||||
|
[2] 4,626 msgs/sec ~ 578.25 KB/sec (10000 msgs)
|
||||||
|
min 4,626 | avg 4,956 | max 5,286 | stddev 330 msgs
|
||||||
|
Sub stats: 9,228 msgs/sec ~ 1.13 MB/sec
|
||||||
|
[1] 4,748 msgs/sec ~ 593.58 KB/sec (10000 msgs)
|
||||||
|
[2] 4,614 msgs/sec ~ 576.80 KB/sec (10000 msgs)
|
||||||
|
min 4,614 | avg 4,681 | max 4,748 | stddev 67 msgs
|
||||||
|
|
||||||
|
Saved metric data in csv file /tmp/bench.csv
|
||||||
|
|
@ -0,0 +1,5 @@
|
||||||
|
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||||
|
qLXKAVGSUb1765aYzTISM7,P0,10000,5120000,6025,6169635.541481,0.829871
|
||||||
|
qLXKAVGSUb1765aYzTISM7,P1,10000,5120000,3892,3985916.070736,1.284523
|
||||||
|
qLXKAVGSUb1765aYzTISM7,P2,10000,5120000,3800,3891389.713965,1.315725
|
||||||
|
qLXKAVGSUb1765aYzTISM7,P3,10000,5120000,3799,3890767.016926,1.315936
|
||||||
|
|
|
@ -0,0 +1,10 @@
|
||||||
|
=== js-4p-20k-1k-r3 ===
|
||||||
|
|
||||||
|
Pub stats: 15,197 msgs/sec ~ 14.84 MB/sec
|
||||||
|
[1] 6,025 msgs/sec ~ 5.88 MB/sec (5000 msgs)
|
||||||
|
[2] 3,892 msgs/sec ~ 3.80 MB/sec (5000 msgs)
|
||||||
|
[3] 3,800 msgs/sec ~ 3.71 MB/sec (5000 msgs)
|
||||||
|
[4] 3,799 msgs/sec ~ 3.71 MB/sec (5000 msgs)
|
||||||
|
min 3,799 | avg 4,379 | max 6,025 | stddev 951 msgs
|
||||||
|
|
||||||
|
Saved metric data in csv file /tmp/bench.csv
|
||||||
|
|
@ -0,0 +1,5 @@
|
||||||
|
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||||
|
MvEG4Dg0Y0tCjCC2CMT7Lv,P0,25000,1600000,4841,619753.770533,2.581670
|
||||||
|
MvEG4Dg0Y0tCjCC2CMT7Lv,P1,25000,1600000,4841,619744.482061,2.581709
|
||||||
|
MvEG4Dg0Y0tCjCC2CMT7Lv,P2,25000,1600000,4834,618788.293474,2.585699
|
||||||
|
MvEG4Dg0Y0tCjCC2CMT7Lv,P3,25000,1600000,4808,615451.351142,2.599718
|
||||||
|
|
|
@ -0,0 +1,10 @@
|
||||||
|
=== js-4p-50k-128-r3 ===
|
||||||
|
|
||||||
|
Pub stats: 19,232 msgs/sec ~ 2.35 MB/sec
|
||||||
|
[1] 4,841 msgs/sec ~ 605.23 KB/sec (12500 msgs)
|
||||||
|
[2] 4,841 msgs/sec ~ 605.22 KB/sec (12500 msgs)
|
||||||
|
[3] 4,834 msgs/sec ~ 604.29 KB/sec (12500 msgs)
|
||||||
|
[4] 4,808 msgs/sec ~ 601.03 KB/sec (12500 msgs)
|
||||||
|
min 4,808 | avg 4,831 | max 4,841 | stddev 13 msgs
|
||||||
|
|
||||||
|
Saved metric data in csv file /tmp/bench.csv
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||||
|
cPDfyUMvkcN0D8GcTijaOZ,P0,40000,2560000,18888,2417667.088651,1.058872
|
||||||
|
|
|
@ -0,0 +1,5 @@
|
||||||
|
=== js-file-1p-20k-128-r1 ===
|
||||||
|
|
||||||
|
Pub stats: 18,888 msgs/sec ~ 2.31 MB/sec
|
||||||
|
|
||||||
|
Saved metric data in csv file /tmp/bench.csv
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||||
|
uOCp1eJgt1rgkqZXjuaSZL,P0,40000,81920000,8673,35527516.943121,2.305818
|
||||||
|
|
|
@ -0,0 +1,5 @@
|
||||||
|
=== js-file-1p-20k-4k-r3 ===
|
||||||
|
|
||||||
|
Pub stats: 8,673 msgs/sec ~ 33.88 MB/sec
|
||||||
|
|
||||||
|
Saved metric data in csv file /tmp/bench.csv
|
||||||
|
|
@ -0,0 +1,5 @@
|
||||||
|
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||||
|
N1SqI80urXipjvUHXScoVc,P0,25000,1600000,6787,868828.734410,1.841560
|
||||||
|
N1SqI80urXipjvUHXScoVc,P1,25000,1600000,6215,795533.902108,2.011228
|
||||||
|
N1SqI80urXipjvUHXScoVc,P2,25000,1600000,6173,790180.313483,2.024854
|
||||||
|
N1SqI80urXipjvUHXScoVc,P3,25000,1600000,6140,785924.339888,2.035819
|
||||||
|
|
|
@ -0,0 +1,10 @@
|
||||||
|
=== js-file-4p-50k-128-r1 ===
|
||||||
|
|
||||||
|
Pub stats: 24,560 msgs/sec ~ 3.00 MB/sec
|
||||||
|
[1] 6,787 msgs/sec ~ 848.47 KB/sec (12500 msgs)
|
||||||
|
[2] 6,215 msgs/sec ~ 776.89 KB/sec (12500 msgs)
|
||||||
|
[3] 6,173 msgs/sec ~ 771.66 KB/sec (12500 msgs)
|
||||||
|
[4] 6,140 msgs/sec ~ 767.50 KB/sec (12500 msgs)
|
||||||
|
min 6,140 | avg 6,328 | max 6,787 | stddev 265 msgs
|
||||||
|
|
||||||
|
Saved metric data in csv file /tmp/bench.csv
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||||
|
6WVyJcf2Hwp2JrsCV11Hls,P0,40000,2560000,29972,3836526.872789,0.667270
|
||||||
|
|
|
@ -0,0 +1,5 @@
|
||||||
|
=== js-mem-1p-20k-128-r1 ===
|
||||||
|
|
||||||
|
Pub stats: 29,972 msgs/sec ~ 3.66 MB/sec
|
||||||
|
|
||||||
|
Saved metric data in csv file /tmp/bench.csv
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||||
|
PDjR792iPxIbCORKIyVqXv,P0,40000,2560000,20188,2584117.626660,0.990667
|
||||||
|
|
|
@ -0,0 +1,5 @@
|
||||||
|
=== js-mem-1p-20k-128-r3 ===
|
||||||
|
|
||||||
|
Pub stats: 20,188 msgs/sec ~ 2.46 MB/sec
|
||||||
|
|
||||||
|
Saved metric data in csv file /tmp/bench.csv
|
||||||
|
|
@ -0,0 +1,5 @@
|
||||||
|
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||||
|
aKthILJvfkpdUW1zncr8Gz,P0,10000,5120000,12732,13037881.294175,0.392702
|
||||||
|
aKthILJvfkpdUW1zncr8Gz,P1,10000,5120000,9234,9455781.228445,0.541468
|
||||||
|
aKthILJvfkpdUW1zncr8Gz,P2,10000,5120000,8510,8715111.398384,0.587485
|
||||||
|
aKthILJvfkpdUW1zncr8Gz,P3,10000,5120000,8479,8683456.194797,0.589627
|
||||||
|
|
|
@ -0,0 +1,10 @@
|
||||||
|
=== js-mem-4p-20k-1k-r3 ===
|
||||||
|
|
||||||
|
Pub stats: 33,916 msgs/sec ~ 33.12 MB/sec
|
||||||
|
[1] 12,732 msgs/sec ~ 12.43 MB/sec (5000 msgs)
|
||||||
|
[2] 9,234 msgs/sec ~ 9.02 MB/sec (5000 msgs)
|
||||||
|
[3] 8,510 msgs/sec ~ 8.31 MB/sec (5000 msgs)
|
||||||
|
[4] 8,479 msgs/sec ~ 8.28 MB/sec (5000 msgs)
|
||||||
|
min 8,479 | avg 9,738 | max 12,732 | stddev 1,754 msgs
|
||||||
|
|
||||||
|
Saved metric data in csv file /tmp/bench.csv
|
||||||
|
|
@ -0,0 +1,5 @@
|
||||||
|
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||||
|
eCPtb5w7YrtjXRblVgvC0q,P0,25000,1600000,18895,2418669.699316,0.661521
|
||||||
|
eCPtb5w7YrtjXRblVgvC0q,P1,25000,1600000,18797,2406124.890764,0.664970
|
||||||
|
eCPtb5w7YrtjXRblVgvC0q,P2,25000,1600000,16698,2137360.776494,0.748587
|
||||||
|
eCPtb5w7YrtjXRblVgvC0q,P3,25000,1600000,16230,2077543.975711,0.770140
|
||||||
|
|
|
@ -0,0 +1,10 @@
|
||||||
|
=== js-mem-4p-50k-128-r1 ===
|
||||||
|
|
||||||
|
Pub stats: 64,923 msgs/sec ~ 7.93 MB/sec
|
||||||
|
[1] 18,895 msgs/sec ~ 2.31 MB/sec (12500 msgs)
|
||||||
|
[2] 18,797 msgs/sec ~ 2.29 MB/sec (12500 msgs)
|
||||||
|
[3] 16,698 msgs/sec ~ 2.04 MB/sec (12500 msgs)
|
||||||
|
[4] 16,230 msgs/sec ~ 1.98 MB/sec (12500 msgs)
|
||||||
|
min 16,230 | avg 17,655 | max 18,895 | stddev 1,202 msgs
|
||||||
|
|
||||||
|
Saved metric data in csv file /tmp/bench.csv
|
||||||
|
|
@ -0,0 +1,5 @@
|
||||||
|
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||||
|
eS3TyeoEZrIgfRNGwTFVYl,P0,25000,1600000,9607,1229719.185854,1.301110
|
||||||
|
eS3TyeoEZrIgfRNGwTFVYl,P1,25000,1600000,9606,1229695.290576,1.301135
|
||||||
|
eS3TyeoEZrIgfRNGwTFVYl,P2,25000,1600000,9503,1216392.400793,1.315365
|
||||||
|
eS3TyeoEZrIgfRNGwTFVYl,P3,25000,1600000,9434,1207613.746241,1.324927
|
||||||
|
|
|
@ -0,0 +1,10 @@
|
||||||
|
=== js-mem-4p-50k-128-r3 ===
|
||||||
|
|
||||||
|
Pub stats: 37,736 msgs/sec ~ 4.61 MB/sec
|
||||||
|
[1] 9,607 msgs/sec ~ 1.17 MB/sec (12500 msgs)
|
||||||
|
[2] 9,606 msgs/sec ~ 1.17 MB/sec (12500 msgs)
|
||||||
|
[3] 9,503 msgs/sec ~ 1.16 MB/sec (12500 msgs)
|
||||||
|
[4] 9,434 msgs/sec ~ 1.15 MB/sec (12500 msgs)
|
||||||
|
min 9,434 | avg 9,537 | max 9,607 | stddev 73 msgs
|
||||||
|
|
||||||
|
Saved metric data in csv file /tmp/bench.csv
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
=== lat-1p-5k-128 ===
|
||||||
|
{"count":5000,"pubs":1,"size":128,"mode":"flood","min_us":98399,"avg_us":124452,"p50_us":125413,"p90_us":140165,"p99_us":145770,"max_us":146002,"min":"98.399ms","avg":"124.452ms","p50":"125.413ms","p90":"140.165ms","p99":"145.770ms","max":"146.002ms"}
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
=== lat-4p-10k-128 ===
|
||||||
|
{"count":10000,"pubs":4,"size":128,"mode":"flood","min_us":157463,"avg_us":205870,"p50_us":207044,"p90_us":237035,"p99_us":239449,"max_us":240473,"min":"157.463ms","avg":"205.870ms","p50":"207.044ms","p90":"237.035ms","p99":"239.449ms","max":"240.473ms"}
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
=== lat-4p-5k-1k ===
|
||||||
|
{"count":5000,"pubs":4,"size":1024,"mode":"flood","min_us":125336,"avg_us":157915,"p50_us":157392,"p90_us":176826,"p99_us":178015,"max_us":178776,"min":"125.336ms","avg":"157.915ms","p50":"157.392ms","p90":"176.826ms","p99":"178.015ms","max":"178.776ms"}
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
=== lat-8p-20k-128 ===
|
||||||
|
{"count":20000,"pubs":8,"size":128,"mode":"flood","min_us":230898,"avg_us":361458,"p50_us":372512,"p90_us":443799,"p99_us":458198,"max_us":460270,"min":"230.898ms","avg":"361.458ms","p50":"372.512ms","p90":"443.799ms","p99":"458.198ms","max":"460.270ms"}
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
=== lat-ping-1k-128 ===
|
||||||
|
{"count":1000,"pubs":1,"size":128,"mode":"ping","min_us":340,"avg_us":530,"p50_us":456,"p90_us":827,"p99_us":1140,"max_us":2910,"min":"0.340ms","avg":"0.530ms","p50":"0.456ms","p90":"0.827ms","p99":"1.140ms","max":"2.910ms"}
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
=== lat-reconnect-200-128 ===
|
||||||
|
{"count":200,"pubs":1,"size":128,"mode":"reconnect","min_us":366,"avg_us":540,"p50_us":503,"p90_us":619,"p99_us":1750,"max_us":3206,"min":"0.366ms","avg":"0.540ms","p50":"0.503ms","p90":"0.619ms","p99":"1.750ms","max":"3.206ms"}
|
||||||
|
|
@ -0,0 +1 @@
|
||||||
|
{"mode":"mqtt-qos0","count":5000,"size":128,"url":"mqtt://10.10.10.21:1883","secs":0.111,"pubs_per_sec":44862,"mb_per_sec":5.74}
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -0,0 +1,361 @@
|
||||||
|
**Progress report (maximized NS1 study)** · run `20260912T055851Z` (UTC)
|
||||||
|
|
||||||
|
> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): `maximize-ns1-study.sh` (cores/RAM/`max_mem`/tmpfs), then `study-on-ns1.sh`, `nats bench`, `latency.mjs` (LXC 510), matplotlib, pandoc, weasyprint. Traffic stayed on `vmbr1`. veth/10G was **not** changed. After the ladder, JetStream was put back on ZFS and product streams were re-created; **8 cores / 16 GiB / max_mem 8G stay**.
|
||||||
|
|
||||||
|
## Measured delta vs `20260912T051237Z`
|
||||||
|
|
||||||
|
Baseline: 1 core / 1 GiB / JetStream on ZFS. This run: 8 cores / 16 GiB / JetStream **tmpfs** (file r=3) plus extra **memory** store rows. veth/10G unchanged.
|
||||||
|
|
||||||
|
| Metric | Baseline `20260912T051237Z` | This run | Ratio |
|
||||||
|
|--------|-------------------------|----------|-------|
|
||||||
|
| Core 1p1s 128 B pub msgs/s | 502,502 | 662,227 | 1.32× |
|
||||||
|
| Core 8p8s 128 B aggregate msgs/s | 2,065,217 | 1,760,599 | 0.85× |
|
||||||
|
| JS file r=3 1p 128 B pub msgs/s | 7,393 | 14,330 | 1.94× |
|
||||||
|
| JS file r=3 4p 128 B pub msgs/s | 17,986 | 19,232 | 1.07× |
|
||||||
|
| JS file r=3 4p 1 KiB pub msgs/s | 14,985 | 15,197 | 1.01× |
|
||||||
|
| JS memory r=3 1p 128 B pub msgs/s | — | 20,188 | — |
|
||||||
|
| JS memory r=3 4p 128 B pub msgs/s | — | 37,736 | — |
|
||||||
|
| Ping p99 (ms) | 1.377ms | 1.140ms | 1.21× faster |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
*Baseline vs maximized publish rates (log)*
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Executive summary
|
||||||
|
|
||||||
|
| Item | This NS1-host run |
|
||||||
|
|------|-------------------|
|
||||||
|
| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `marchon` |
|
||||||
|
| Bench client | LXC 510 `verae-px-worker` |
|
||||||
|
| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` |
|
||||||
|
| Client URL | `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` |
|
||||||
|
| Host load before | `8.70 8.15 8.00 4/3842 621042` |
|
||||||
|
| Host load after | `8.48 8.52 8.17 5/3860 657839` |
|
||||||
|
| Core 1p1s 128 B pub | 662,227 msgs/s |
|
||||||
|
| JetStream 1p 128 B r=3 | 14,330 durable pubs/s |
|
||||||
|
| Ping p50 / p99 | 0.456ms / 1.140ms |
|
||||||
|
|
||||||
|
Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Where it ran (and where it did not)
|
||||||
|
|
||||||
|
```text
|
||||||
|
Operator laptop ──ssh──► NS1.GEORGELAMBERT.ORG 70.88.205.138
|
||||||
|
study-on-ns1.sh
|
||||||
|
python3 build-ns1-study-report.py
|
||||||
|
sudo pct exec 510 ──► nats bench / latency.mjs
|
||||||
|
│
|
||||||
|
▼ vmbr1
|
||||||
|
10.10.10.21-23 :4222
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510).
|
||||||
|
- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Results (this run)
|
||||||
|
|
||||||
|
### Host and brokers
|
||||||
|
|
||||||
|
**Before**
|
||||||
|
|
||||||
|
| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream |
|
||||||
|
|------|------|-------------|---------|----------|-----|-------|---------|-----------|
|
||||||
|
| nats-a | 511 | 5 | 31,443 | 31,725 | 1 | 8 | 15,785,984 | True |
|
||||||
|
| nats-b | 512 | 1 | 24,330 | 24,480 | 1 | 1 | 15,892,480 | True |
|
||||||
|
| nats-c | 513 | 0 | 19,051 | 19,034 | 0 | 1 | 14,262,272 | True |
|
||||||
|
|
||||||
|
**After**
|
||||||
|
|
||||||
|
| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream |
|
||||||
|
|------|------|-------------|---------|----------|-----|-------|---------|-----------|
|
||||||
|
| nats-a | 511 | 5 | 724,962 | 1,400,288 | 2 | 8 | 27,095,040 | True |
|
||||||
|
| nats-b | 512 | 1 | 509,195 | 671,813 | 0 | 1 | 36,028,416 | True |
|
||||||
|
| nats-c | 513 | 0 | 703,508 | 1,715,907 | 0 | 1 | 27,295,744 | True |
|
||||||
|
|
||||||
|
nproc=40 · uname=`Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux`
|
||||||
|
|
||||||
|
### Throughput
|
||||||
|
|
||||||
|
| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
|
||||||
|
|-----|------|------------------|------------|----------|------------|----------|
|
||||||
|
| `core-1p1s-50k-128` | core pub/sub | 835,602 | 662,227 | 80.84 | 472,239 | 57.65 |
|
||||||
|
| `core-4p4s-100k-128` | core pub/sub | 1,597,284 | 692,109 | 84.49 | 1,286,093 | 156.99 |
|
||||||
|
| `core-4p4s-50k-1k` | core pub/sub | 595,555 | 189,939 | 185.49 | 479,898 | 468.65 |
|
||||||
|
| `core-8p8s-200k-128` | core pub/sub | 1,760,599 | 223,078 | 27.23 | 1,570,171 | 191.67 |
|
||||||
|
| `js-1p-20k-128-r3` | jetstream r=3 file | — | 14,330 | 1.75 | — | — |
|
||||||
|
| `js-2p2s-20k-128-r3` | jetstream r=3 file | 18,449 | 9,252 | 1.13 | 9,228 | 1.13 |
|
||||||
|
| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 15,197 | 14.84 | — | — |
|
||||||
|
| `js-4p-50k-128-r3` | jetstream r=3 file | — | 19,232 | 2.35 | — | — |
|
||||||
|
| `js-file-1p-20k-128-r1` | jetstream r=3 file | — | 18,888 | 2.31 | — | — |
|
||||||
|
| `js-file-1p-20k-4k-r3` | jetstream r=3 file | — | 8,673 | 33.88 | — | — |
|
||||||
|
| `js-file-4p-50k-128-r1` | jetstream r=3 file | — | 24,560 | 3.00 | — | — |
|
||||||
|
| `js-mem-1p-20k-128-r1` | jetstream r=3 file | — | 29,972 | 3.66 | — | — |
|
||||||
|
| `js-mem-1p-20k-128-r3` | jetstream r=3 file | — | 20,188 | 2.46 | — | — |
|
||||||
|
| `js-mem-4p-20k-1k-r3` | jetstream r=3 file | — | 33,916 | 33.12 | — | — |
|
||||||
|
| `js-mem-4p-50k-128-r1` | jetstream r=3 file | — | 64,923 | 7.93 | — | — |
|
||||||
|
| `js-mem-4p-50k-128-r3` | jetstream r=3 file | — | 37,736 | 4.61 | — | — |
|
||||||
|
|
||||||
|
### Round-trip delay
|
||||||
|
|
||||||
|
| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
|
||||||
|
|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|
|
||||||
|
| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.340ms | 0.530ms | 0.456ms | 0.827ms | 1.140ms | 2.910ms |
|
||||||
|
| `lat-reconnect-200-128` | flood (burst queueing) | 200 | 1 | 128 B | 0.366ms | 0.540ms | 0.503ms | 0.619ms | 1.750ms | 3.206ms |
|
||||||
|
| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 98.399ms | 124.452ms | 125.413ms | 140.165ms | 145.770ms | 146.002ms |
|
||||||
|
| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 125.336ms | 157.915ms | 157.392ms | 176.826ms | 178.015ms | 178.776ms |
|
||||||
|
| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 157.463ms | 205.870ms | 207.044ms | 237.035ms | 239.449ms | 240.473ms |
|
||||||
|
| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 230.898ms | 361.458ms | 372.512ms | 443.799ms | 458.198ms | 460.270ms |
|
||||||
|
|
||||||
|
### Core NATS
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
*Core NATS throughput at four loads (NS1 host run)*
|
||||||
|
### Payload size (core)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
*Core NATS 128 B vs 1 KiB (NS1 host run)*
|
||||||
|
### JetStream r=3 file
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
*JetStream durable publish rate (NS1 host run)*
|
||||||
|
### Core vs JetStream
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
*Core vs JetStream publish rate, log scale (NS1 host run)*
|
||||||
|
### Delay
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
*Ping vs flood delay percentiles, log scale (NS1 host run)*
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Study methodology
|
||||||
|
|
||||||
|
### 4.1 Question
|
||||||
|
|
||||||
|
On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)?
|
||||||
|
|
||||||
|
### 4.2 Hypotheses (stated before the run)
|
||||||
|
|
||||||
|
1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica.
|
||||||
|
2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated.
|
||||||
|
3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting.
|
||||||
|
4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count.
|
||||||
|
5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound.
|
||||||
|
|
||||||
|
### 4.3 Independent variables (what we changed)
|
||||||
|
|
||||||
|
| Factor | Levels |
|
||||||
|
|--------|--------|
|
||||||
|
| Transport | Core NATS pub/sub vs JetStream file replicas=3 |
|
||||||
|
| Publisher count | 1, 2, 4, 8 |
|
||||||
|
| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 |
|
||||||
|
| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) |
|
||||||
|
| Payload | 128 B, 1024 B |
|
||||||
|
| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) |
|
||||||
|
|
||||||
|
### 4.4 Dependent variables (what we recorded)
|
||||||
|
|
||||||
|
| Metric | Instrument | Unit |
|
||||||
|
|--------|------------|------|
|
||||||
|
| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s |
|
||||||
|
| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s |
|
||||||
|
| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) |
|
||||||
|
| Publisher spread | nats min/avg/max **msgs/s** | not delay |
|
||||||
|
| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max |
|
||||||
|
| Host load | `/proc/loadavg` before and after | load average |
|
||||||
|
| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem |
|
||||||
|
|
||||||
|
**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`.
|
||||||
|
|
||||||
|
### 4.5 Controls and constants
|
||||||
|
|
||||||
|
- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`.
|
||||||
|
- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`).
|
||||||
|
- Bench client is LXC **510**, not a nats-* server.
|
||||||
|
- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide.
|
||||||
|
- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`).
|
||||||
|
- No TLS, no nkeys, no account isolation (isolation is `vmbr1`).
|
||||||
|
- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder.
|
||||||
|
|
||||||
|
### 4.6 Procedure
|
||||||
|
|
||||||
|
1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise.
|
||||||
|
2. Snapshot host load, memory, LXC configs, and each nats `varz`.
|
||||||
|
3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB).
|
||||||
|
4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file.
|
||||||
|
5. Copy `latency.mjs` into 510; ping then flood at several batch sizes.
|
||||||
|
6. Snapshot host/`varz` again.
|
||||||
|
7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**.
|
||||||
|
|
||||||
|
No publish, subscribe, chart, or PDF process runs on the operator laptop for this study.
|
||||||
|
|
||||||
|
### 4.7 Instrumentation path
|
||||||
|
|
||||||
|
```text
|
||||||
|
[NS1 host 70.88.205.138]
|
||||||
|
study-on-ns1.sh (bash + python3)
|
||||||
|
|
|
||||||
|
| sudo pct exec 510
|
||||||
|
v
|
||||||
|
[LXC 510 verae-px-worker 10.10.10.20]
|
||||||
|
nats bench / node latency.mjs
|
||||||
|
|
|
||||||
|
| NATS client protocol to
|
||||||
|
v
|
||||||
|
[LXC 511/512/513 10.10.10.21-23 :4222]
|
||||||
|
nats-server -js cluster routes :6222
|
||||||
|
```
|
||||||
|
|
||||||
|
The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`.
|
||||||
|
|
||||||
|
### 4.8 Threats to validity
|
||||||
|
|
||||||
|
| Threat | Effect on numbers |
|
||||||
|
|--------|-------------------|
|
||||||
|
| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. |
|
||||||
|
| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. |
|
||||||
|
| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. |
|
||||||
|
| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. |
|
||||||
|
| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. |
|
||||||
|
| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. |
|
||||||
|
| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. |
|
||||||
|
| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. |
|
||||||
|
|
||||||
|
### 4.9 Ethics / safety
|
||||||
|
|
||||||
|
Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Suggestions for fine-tuning
|
||||||
|
|
||||||
|
These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way.
|
||||||
|
|
||||||
|
### 5.1 Treat JetStream as the product limiter
|
||||||
|
|
||||||
|
Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out.
|
||||||
|
|
||||||
|
### 5.2 Split storage class by stream
|
||||||
|
|
||||||
|
| Stream | Suggested store | Why |
|
||||||
|
|--------|-----------------|-----|
|
||||||
|
| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad |
|
||||||
|
| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both |
|
||||||
|
| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer |
|
||||||
|
| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry |
|
||||||
|
| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive |
|
||||||
|
|
||||||
|
Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert.
|
||||||
|
|
||||||
|
### 5.3 Give JetStream real disks
|
||||||
|
|
||||||
|
Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain.
|
||||||
|
|
||||||
|
- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`.
|
||||||
|
- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure.
|
||||||
|
- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it.
|
||||||
|
- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.”
|
||||||
|
|
||||||
|
### 5.4 Isolate the nats CTs from the rest of NS1
|
||||||
|
|
||||||
|
Host load on this box is often already several. Pin:
|
||||||
|
|
||||||
|
- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes.
|
||||||
|
- Memory high enough that file-backed streams stay cache-hot for the working set.
|
||||||
|
- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync.
|
||||||
|
|
||||||
|
Re-run this study after pinning; H1/H2 should move more than ping.
|
||||||
|
|
||||||
|
### 5.5 Consumer and mailbox tuning (delay H4)
|
||||||
|
|
||||||
|
Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT.
|
||||||
|
|
||||||
|
- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM.
|
||||||
|
- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber.
|
||||||
|
- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped.
|
||||||
|
- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT.
|
||||||
|
|
||||||
|
### 5.6 Publisher-side batching in middleware
|
||||||
|
|
||||||
|
A timestamp job is one small JSON. 16k msgs/s is ample. Still:
|
||||||
|
|
||||||
|
- Avoid per-byte publishes; one message per job/event.
|
||||||
|
- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs).
|
||||||
|
- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only.
|
||||||
|
|
||||||
|
### 5.7 nats-server knobs worth measuring (A/B with this script)
|
||||||
|
|
||||||
|
| Knob | Why try it |
|
||||||
|
|------|------------|
|
||||||
|
| `max_payload` | Keep default unless archive puts grow |
|
||||||
|
| `write_deadline` | Slow consumer protection for webhooks |
|
||||||
|
| `max_pending` | Bound memory on a stuck Zapier hook |
|
||||||
|
| `max_connections` | Fleet workers + keep + middleware |
|
||||||
|
| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT |
|
||||||
|
| `max_outstanding_catchup` | Replica restart after a nats-c blip |
|
||||||
|
| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT |
|
||||||
|
|
||||||
|
Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99.
|
||||||
|
|
||||||
|
### 5.8 Network
|
||||||
|
|
||||||
|
- Keep NATS off `vmbr0`. No change.
|
||||||
|
- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load).
|
||||||
|
- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus.
|
||||||
|
|
||||||
|
### 5.9 Security cost (when nkeys/mTLS flip)
|
||||||
|
|
||||||
|
`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement.
|
||||||
|
|
||||||
|
### 5.10 Operational fine-tuning (lag, not peak msgs/s)
|
||||||
|
|
||||||
|
1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`.
|
||||||
|
2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense.
|
||||||
|
3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost.
|
||||||
|
4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated.
|
||||||
|
|
||||||
|
### 5.11 What not to tune
|
||||||
|
|
||||||
|
- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge.
|
||||||
|
- Do not treat flood 400 ms as “cluster RTT.” Fix consumers.
|
||||||
|
- Do not load-test on `ZAPIER_*` streams.
|
||||||
|
- Do not bind client NATS to `0.0.0.0` on `vmbr0`.
|
||||||
|
|
||||||
|
### 5.12 Recommended next experiments (same method, one change each)
|
||||||
|
|
||||||
|
1. CPU pin nats-a/b/c → re-run JS 1p + ping.
|
||||||
|
2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder).
|
||||||
|
3. Distinct `store_dir` disks per node.
|
||||||
|
4. nkeys on, same ladder.
|
||||||
|
5. Three hardware boxes, same `cluster.env` IPs updated.
|
||||||
|
|
||||||
|
Each experiment should produce a new `results/<utc>/` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory.
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Reproducing this study
|
||||||
|
|
||||||
|
On **NS1 only**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/verae-src/verae-nats-cluster
|
||||||
|
bash scripts/study-on-ns1.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
The script exits if `hostname` is not NS1. Outputs land in `results/<utc>/` including `nats-cluster-bench-ns1.{md,html,pdf}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog.
|
||||||
|
|
||||||
|
Raw logs for this run: `results/20260912T055851Z/`.
|
||||||
Binary file not shown.
|
|
@ -0,0 +1,853 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8" />
|
||||||
|
<meta name="generator" content="pandoc" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
|
||||||
|
<title>NATS optimal configuration study</title>
|
||||||
|
<style>
|
||||||
|
html {
|
||||||
|
color: #1a1a1a;
|
||||||
|
background-color: #fdfdfd;
|
||||||
|
}
|
||||||
|
body {
|
||||||
|
margin: 0 auto;
|
||||||
|
max-width: 36em;
|
||||||
|
padding-left: 50px;
|
||||||
|
padding-right: 50px;
|
||||||
|
padding-top: 50px;
|
||||||
|
padding-bottom: 50px;
|
||||||
|
hyphens: auto;
|
||||||
|
overflow-wrap: break-word;
|
||||||
|
text-rendering: optimizeLegibility;
|
||||||
|
font-kerning: normal;
|
||||||
|
}
|
||||||
|
@media (max-width: 600px) {
|
||||||
|
body {
|
||||||
|
font-size: 0.9em;
|
||||||
|
padding: 12px;
|
||||||
|
}
|
||||||
|
h1 {
|
||||||
|
font-size: 1.8em;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
@media print {
|
||||||
|
html {
|
||||||
|
background-color: white;
|
||||||
|
}
|
||||||
|
body {
|
||||||
|
background-color: transparent;
|
||||||
|
color: black;
|
||||||
|
font-size: 12pt;
|
||||||
|
}
|
||||||
|
p, h2, h3 {
|
||||||
|
orphans: 3;
|
||||||
|
widows: 3;
|
||||||
|
}
|
||||||
|
h2, h3, h4 {
|
||||||
|
page-break-after: avoid;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
p {
|
||||||
|
margin: 1em 0;
|
||||||
|
}
|
||||||
|
a {
|
||||||
|
color: #1a1a1a;
|
||||||
|
}
|
||||||
|
a:visited {
|
||||||
|
color: #1a1a1a;
|
||||||
|
}
|
||||||
|
img {
|
||||||
|
max-width: 100%;
|
||||||
|
}
|
||||||
|
svg {
|
||||||
|
height: auto;
|
||||||
|
max-width: 100%;
|
||||||
|
}
|
||||||
|
h1, h2, h3, h4, h5, h6 {
|
||||||
|
margin-top: 1.4em;
|
||||||
|
}
|
||||||
|
h5, h6 {
|
||||||
|
font-size: 1em;
|
||||||
|
font-style: italic;
|
||||||
|
}
|
||||||
|
h6 {
|
||||||
|
font-weight: normal;
|
||||||
|
}
|
||||||
|
ol, ul {
|
||||||
|
padding-left: 1.7em;
|
||||||
|
margin-top: 1em;
|
||||||
|
}
|
||||||
|
li > ol, li > ul {
|
||||||
|
margin-top: 0;
|
||||||
|
}
|
||||||
|
blockquote {
|
||||||
|
margin: 1em 0 1em 1.7em;
|
||||||
|
padding-left: 1em;
|
||||||
|
border-left: 2px solid #e6e6e6;
|
||||||
|
color: #606060;
|
||||||
|
}
|
||||||
|
code {
|
||||||
|
font-family: Menlo, Monaco, Consolas, 'Lucida Console', monospace;
|
||||||
|
font-size: 85%;
|
||||||
|
margin: 0;
|
||||||
|
hyphens: manual;
|
||||||
|
}
|
||||||
|
pre {
|
||||||
|
margin: 1em 0;
|
||||||
|
overflow: auto;
|
||||||
|
}
|
||||||
|
pre code {
|
||||||
|
padding: 0;
|
||||||
|
overflow: visible;
|
||||||
|
overflow-wrap: normal;
|
||||||
|
}
|
||||||
|
.sourceCode {
|
||||||
|
background-color: transparent;
|
||||||
|
overflow: visible;
|
||||||
|
}
|
||||||
|
hr {
|
||||||
|
background-color: #1a1a1a;
|
||||||
|
border: none;
|
||||||
|
height: 1px;
|
||||||
|
margin: 1em 0;
|
||||||
|
}
|
||||||
|
table {
|
||||||
|
margin: 1em 0;
|
||||||
|
border-collapse: collapse;
|
||||||
|
width: 100%;
|
||||||
|
overflow-x: auto;
|
||||||
|
display: block;
|
||||||
|
font-variant-numeric: lining-nums tabular-nums;
|
||||||
|
}
|
||||||
|
table caption {
|
||||||
|
margin-bottom: 0.75em;
|
||||||
|
}
|
||||||
|
tbody {
|
||||||
|
margin-top: 0.5em;
|
||||||
|
border-top: 1px solid #1a1a1a;
|
||||||
|
border-bottom: 1px solid #1a1a1a;
|
||||||
|
}
|
||||||
|
th {
|
||||||
|
border-top: 1px solid #1a1a1a;
|
||||||
|
padding: 0.25em 0.5em 0.25em 0.5em;
|
||||||
|
}
|
||||||
|
td {
|
||||||
|
padding: 0.125em 0.5em 0.25em 0.5em;
|
||||||
|
}
|
||||||
|
header {
|
||||||
|
margin-bottom: 4em;
|
||||||
|
text-align: center;
|
||||||
|
}
|
||||||
|
#TOC li {
|
||||||
|
list-style: none;
|
||||||
|
}
|
||||||
|
#TOC ul {
|
||||||
|
padding-left: 1.3em;
|
||||||
|
}
|
||||||
|
#TOC > ul {
|
||||||
|
padding-left: 0;
|
||||||
|
}
|
||||||
|
#TOC a:not(:hover) {
|
||||||
|
text-decoration: none;
|
||||||
|
}
|
||||||
|
code{white-space: pre-wrap;}
|
||||||
|
span.smallcaps{font-variant: small-caps;}
|
||||||
|
div.columns{display: flex; gap: min(4vw, 1.5em);}
|
||||||
|
div.column{flex: auto; overflow-x: auto;}
|
||||||
|
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
|
||||||
|
/* The extra [class] is a hack that increases specificity enough to
|
||||||
|
override a similar rule in reveal.js */
|
||||||
|
ul.task-list[class]{list-style: none;}
|
||||||
|
ul.task-list li input[type="checkbox"] {
|
||||||
|
font-size: inherit;
|
||||||
|
width: 0.8em;
|
||||||
|
margin: 0 0.8em 0.2em -1.6em;
|
||||||
|
vertical-align: middle;
|
||||||
|
}
|
||||||
|
.display.math{display: block; text-align: center; margin: 0.5rem auto;}
|
||||||
|
</style>
|
||||||
|
<style>/* Colored print + screen stylesheet for zapier.georgelambert.org */
|
||||||
|
:root {
|
||||||
|
--ink: #171a26;
|
||||||
|
--muted: #5b6178;
|
||||||
|
--line: #d9dce8;
|
||||||
|
--bg: #f4f5fb;
|
||||||
|
--paper: #ffffff;
|
||||||
|
--accent: #4f46e5;
|
||||||
|
--accent-deep: #312e81;
|
||||||
|
--accent-soft: #eef0fe;
|
||||||
|
--ok: #047857;
|
||||||
|
--warn: #8a5a00;
|
||||||
|
--code-bg: #1b1f33;
|
||||||
|
--code-fg: #e8ecff;
|
||||||
|
}
|
||||||
|
html { background: var(--bg); }
|
||||||
|
body {
|
||||||
|
margin: 0 auto;
|
||||||
|
padding: 1.5rem 1.25rem 3rem;
|
||||||
|
max-width: 48rem;
|
||||||
|
font: 15px/1.55 -apple-system, "Segoe UI", Georgia, serif;
|
||||||
|
color: var(--ink);
|
||||||
|
background: var(--paper);
|
||||||
|
}
|
||||||
|
.doc-banner {
|
||||||
|
background: linear-gradient(160deg, #312e81 0%, #4f46e5 60%, #7c74f0 100%);
|
||||||
|
color: #eef0fe;
|
||||||
|
margin: -1.5rem -1.25rem 1.5rem;
|
||||||
|
padding: 1.1rem 1.25rem 1rem;
|
||||||
|
}
|
||||||
|
.doc-banner a { color: #fff; }
|
||||||
|
.doc-banner .kicker {
|
||||||
|
letter-spacing: 0.12em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
font: 700 10px system-ui, sans-serif;
|
||||||
|
opacity: 0.8;
|
||||||
|
}
|
||||||
|
.doc-banner h1 { margin: 0.25rem 0 0; font-size: 1.45rem; color: #fff; }
|
||||||
|
h1, h2, h3, h4 { color: var(--accent-deep); page-break-after: avoid; }
|
||||||
|
h1 { font-size: 1.7rem; }
|
||||||
|
h2 {
|
||||||
|
font-size: 1.2rem;
|
||||||
|
border-bottom: 2px solid var(--accent);
|
||||||
|
padding-bottom: 0.2rem;
|
||||||
|
margin-top: 1.6rem;
|
||||||
|
}
|
||||||
|
h3 { font-size: 1.05rem; color: var(--accent); }
|
||||||
|
a { color: var(--accent); }
|
||||||
|
p, li { orphans: 3; widows: 3; }
|
||||||
|
code {
|
||||||
|
font-family: ui-monospace, Menlo, Consolas, monospace;
|
||||||
|
font-size: 0.86em;
|
||||||
|
background: var(--accent-soft);
|
||||||
|
color: var(--accent-deep);
|
||||||
|
padding: 0.08em 0.28em;
|
||||||
|
border-radius: 4px;
|
||||||
|
}
|
||||||
|
pre, div.sourceCode, div.sourceCode pre {
|
||||||
|
background: var(--code-bg) !important;
|
||||||
|
color: var(--code-fg) !important;
|
||||||
|
padding: 0.85rem 1rem;
|
||||||
|
border-radius: 10px;
|
||||||
|
overflow: auto;
|
||||||
|
font-size: 0.78rem;
|
||||||
|
line-height: 1.4;
|
||||||
|
page-break-inside: avoid;
|
||||||
|
}
|
||||||
|
pre code { background: transparent; color: inherit; padding: 0; }
|
||||||
|
#title-block-header, header#title-block-header, h1.title { display: none; }
|
||||||
|
.doc-banner + h1 { display: none; }
|
||||||
|
table {
|
||||||
|
border-collapse: collapse;
|
||||||
|
width: 100%;
|
||||||
|
margin: 0.8rem 0 1.2rem;
|
||||||
|
font-size: 0.9rem;
|
||||||
|
page-break-inside: avoid;
|
||||||
|
}
|
||||||
|
th, td { border: 1px solid var(--line); padding: 0.38rem 0.55rem; text-align: left; vertical-align: top; }
|
||||||
|
th {
|
||||||
|
background: var(--accent);
|
||||||
|
color: #fff;
|
||||||
|
font: 650 12px system-ui, sans-serif;
|
||||||
|
}
|
||||||
|
tr:nth-child(even) td { background: var(--accent-soft); }
|
||||||
|
blockquote {
|
||||||
|
margin: 1rem 0;
|
||||||
|
padding: 0.4rem 0.9rem;
|
||||||
|
border-left: 4px solid var(--accent);
|
||||||
|
background: var(--accent-soft);
|
||||||
|
color: var(--accent-deep);
|
||||||
|
}
|
||||||
|
img { max-width: 100%; height: auto; border-radius: 8px; page-break-inside: avoid; }
|
||||||
|
hr { border: 0; border-top: 1px solid var(--line); }
|
||||||
|
ul, ol { padding-left: 1.25rem; }
|
||||||
|
nav.site { font: 13px system-ui, sans-serif; margin-bottom: 0.4rem; }
|
||||||
|
.source-path { font: 11px ui-monospace, Menlo, monospace; color: var(--muted); }
|
||||||
|
|
||||||
|
@page {
|
||||||
|
size: letter;
|
||||||
|
margin: 0.65in 0.7in 0.8in 0.7in;
|
||||||
|
@top-left {
|
||||||
|
content: "Verae Time × Zapier";
|
||||||
|
font: 700 8pt system-ui, sans-serif;
|
||||||
|
color: #4f46e5;
|
||||||
|
}
|
||||||
|
@top-right {
|
||||||
|
content: "zapier.georgelambert.org";
|
||||||
|
font: 8pt system-ui, sans-serif;
|
||||||
|
color: #6b7186;
|
||||||
|
}
|
||||||
|
@bottom-center {
|
||||||
|
content: counter(page) " / " counter(pages);
|
||||||
|
font: 8pt system-ui, sans-serif;
|
||||||
|
color: #6b7186;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
@media print {
|
||||||
|
html, body { background: #fff; max-width: none; padding: 0; }
|
||||||
|
.doc-banner { margin: 0 0 1rem; border-radius: 8px; -webkit-print-color-adjust: exact; print-color-adjust: exact; }
|
||||||
|
a { text-decoration: none; }
|
||||||
|
th, tr:nth-child(even) td, pre, blockquote, code { -webkit-print-color-adjust: exact; print-color-adjust: exact; }
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="doc-banner"><nav class="site"><a href="/">zapier.georgelambert.org</a></nav><div class="kicker">Verae Time × Zapier · progress report</div><h1>NATS optimal configuration study</h1><div class="source-path">packages/zapier-decisions/reports/optimal-config/REPORT.md</div></div>
|
||||||
|
<header id="title-block-header">
|
||||||
|
<h1 class="title">NATS optimal configuration study</h1>
|
||||||
|
</header>
|
||||||
|
<p><strong>Progress report — optimal configuration study</strong> ·
|
||||||
|
<code>20260912T055851Z</code> (UTC) · all code on
|
||||||
|
<strong>NS1.GEORGELAMBERT.ORG</strong> (<code>70.88.205.138</code>)</p>
|
||||||
|
<p>This document folds every ladder we have run (1-core ZFS,
|
||||||
|
NS1-orchestrated, tmpfs maximize, and this exhaustive 8c/16G
|
||||||
|
<strong>ZFS</strong> factorial) plus UDP / MQTT / reconnect probes. It
|
||||||
|
recommends a lab config and a <strong>three-box HP DL360 Gen10</strong>
|
||||||
|
projection. veth/10G was not changed.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="verdict-read-this-first">1. Verdict (read this first)</h2>
|
||||||
|
<p><strong>Keep NATS + JetStream.</strong> Do not replace the fabric
|
||||||
|
with MQTT, UDP, or a custom persistent-socket protocol for Verae
|
||||||
|
jobs/events/archive. Those are either slower, less durable, or already
|
||||||
|
what NATS is.</p>
|
||||||
|
<p><strong>Lab (NS1, one host, three LXC) — optimal now</strong></p>
|
||||||
|
<table>
|
||||||
|
<colgroup>
|
||||||
|
<col style="width: 25%" />
|
||||||
|
<col style="width: 28%" />
|
||||||
|
<col style="width: 31%" />
|
||||||
|
<col style="width: 15%" />
|
||||||
|
</colgroup>
|
||||||
|
<thead>
|
||||||
|
<tr class="header">
|
||||||
|
<th>Stream</th>
|
||||||
|
<th>Storage</th>
|
||||||
|
<th>Replicas</th>
|
||||||
|
<th>Why</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>ZAPIER_JOBS</code>, <code>ZAPIER_WEBHOOKS</code>,
|
||||||
|
<code>VERAE_ARCHIVE</code></td>
|
||||||
|
<td><strong>file</strong> (ZFS)</td>
|
||||||
|
<td><strong>3</strong></td>
|
||||||
|
<td>Survive a nats LXC death; archive must persist</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><code>ZAPIER_EVENTS</code></td>
|
||||||
|
<td><strong>memory</strong></td>
|
||||||
|
<td><strong>3</strong></td>
|
||||||
|
<td>Waiters are latency-sensitive; events rebuild from job status</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>ZAPIER_USAGE</code></td>
|
||||||
|
<td>file</td>
|
||||||
|
<td>3</td>
|
||||||
|
<td>Telemetry, limits + max-age</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>Keep <strong>8 cores / 16 GiB / <code>max_mem: 8G</code></strong> on
|
||||||
|
510–513 (already live). Do <strong>not</strong> leave JetStream on
|
||||||
|
tmpfs. Do <strong>not</strong> drop product streams to r=1. Reuse
|
||||||
|
<strong>one NATS connection per process</strong> (already true in
|
||||||
|
middleware); never connect-per-message.</p>
|
||||||
|
<p><strong>Metal (3× DL360 Gen10) — optimal later</strong></p>
|
||||||
|
<p>Same stream table. File store on <strong>local NVMe/M.2</strong>, not
|
||||||
|
a shared SAN. Cluster + client on <strong>10GbE</strong> (or 25GbE if
|
||||||
|
you already have it). Dual Gold Xeon is surplus CPU for this workload;
|
||||||
|
8–16 cores dedicated to <code>nats-server</code> is enough. Expected JS
|
||||||
|
file r=3: <strong>~40–80k</strong> 128 B pubs/s (about
|
||||||
|
<strong>3–6×</strong> this lab’s 8c ZFS 1p, <strong>2–4×</strong> tmpfs
|
||||||
|
1p) — bounded by <strong>10GbE replica RTT</strong>, not by Xeon clocks.
|
||||||
|
Core NATS will sit in the <strong>1–3M msgs/s</strong> band until the
|
||||||
|
NIC saturates (~9 Gbit/s ≈ 8–9M × 128 B theoretical; CPU and client will
|
||||||
|
hit first).</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="what-we-actually-ran-this-exhaustive-pass">2. What we actually
|
||||||
|
ran (this exhaustive pass)</h2>
|
||||||
|
<p>Live cluster during this run: LXC 510–513 <strong>8 cores / 16
|
||||||
|
GiB</strong>, JetStream <strong>on ZFS</strong> (tmpfs from the maximize
|
||||||
|
study was already unmounted). Extra factorial: file/memory × replicas
|
||||||
|
1/3, 4 KiB file r=3, reconnect-per-message ping, UDP echo 510→511, MQTT
|
||||||
|
QoS0 against nats-a <code>:1883</code>. Product streams were not the
|
||||||
|
bench target.</p>
|
||||||
|
<h3 id="cross-study-history">2.1 Cross-study history</h3>
|
||||||
|
<table style="width:100%;">
|
||||||
|
<colgroup>
|
||||||
|
<col style="width: 16%" />
|
||||||
|
<col style="width: 16%" />
|
||||||
|
<col style="width: 16%" />
|
||||||
|
<col style="width: 16%" />
|
||||||
|
<col style="width: 16%" />
|
||||||
|
<col style="width: 16%" />
|
||||||
|
</colgroup>
|
||||||
|
<thead>
|
||||||
|
<tr class="header">
|
||||||
|
<th>Study</th>
|
||||||
|
<th>Env</th>
|
||||||
|
<th>Core 1p pub</th>
|
||||||
|
<th>JS file r=3 1p</th>
|
||||||
|
<th>JS mem r=3 4p</th>
|
||||||
|
<th>Ping p99</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>20260912T051237Z</code></td>
|
||||||
|
<td>1c/1G ZFS (NS1 orch.)</td>
|
||||||
|
<td>502,502</td>
|
||||||
|
<td>7,393</td>
|
||||||
|
<td>—</td>
|
||||||
|
<td>1.377ms</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><code>20260912T053120Z</code></td>
|
||||||
|
<td>8c/16G tmpfs + mem extra</td>
|
||||||
|
<td>599,004</td>
|
||||||
|
<td>17,388</td>
|
||||||
|
<td>36,355</td>
|
||||||
|
<td>0.684ms</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>20260912T055851Z</code></td>
|
||||||
|
<td>8c/16G ZFS exhaustive <code>20260912T055851Z</code></td>
|
||||||
|
<td>662,227</td>
|
||||||
|
<td>14,330</td>
|
||||||
|
<td>37,736</td>
|
||||||
|
<td>1.140ms</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<figure>
|
||||||
|
<img src="charts-optimal/history-js1p.png"
|
||||||
|
alt="JS 1p file r=3 history" />
|
||||||
|
<figcaption aria-hidden="true">JS 1p file r=3 history</figcaption>
|
||||||
|
</figure>
|
||||||
|
<h3 id="this-run-jetstream-factorial">2.2 This run — JetStream
|
||||||
|
factorial</h3>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr class="header">
|
||||||
|
<th>Run</th>
|
||||||
|
<th>What</th>
|
||||||
|
<th>Pub msgs/s</th>
|
||||||
|
<th>Pub MB/s</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>js-file-1p-20k-128-r1</code></td>
|
||||||
|
<td>file r=1 1p 128 B</td>
|
||||||
|
<td>18,888</td>
|
||||||
|
<td>2.31</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><code>js-file-4p-50k-128-r1</code></td>
|
||||||
|
<td>file r=1 4p 128 B</td>
|
||||||
|
<td>24,560</td>
|
||||||
|
<td>3.00</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>js-1p-20k-128-r3</code></td>
|
||||||
|
<td>file r=3 1p 128 B</td>
|
||||||
|
<td>14,330</td>
|
||||||
|
<td>1.75</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><code>js-4p-50k-128-r3</code></td>
|
||||||
|
<td>file r=3 4p 128 B</td>
|
||||||
|
<td>19,232</td>
|
||||||
|
<td>2.35</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>js-4p-20k-1k-r3</code></td>
|
||||||
|
<td>file r=3 4p 1 KiB</td>
|
||||||
|
<td>15,197</td>
|
||||||
|
<td>14.84</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><code>js-file-1p-20k-4k-r3</code></td>
|
||||||
|
<td>file r=3 1p 4 KiB</td>
|
||||||
|
<td>8,673</td>
|
||||||
|
<td>33.88</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>js-mem-1p-20k-128-r1</code></td>
|
||||||
|
<td>memory r=1 1p 128 B</td>
|
||||||
|
<td>29,972</td>
|
||||||
|
<td>3.66</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><code>js-mem-4p-50k-128-r1</code></td>
|
||||||
|
<td>memory r=1 4p 128 B</td>
|
||||||
|
<td>64,923</td>
|
||||||
|
<td>7.93</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>js-mem-1p-20k-128-r3</code></td>
|
||||||
|
<td>memory r=3 1p 128 B</td>
|
||||||
|
<td>20,188</td>
|
||||||
|
<td>2.46</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><code>js-mem-4p-50k-128-r3</code></td>
|
||||||
|
<td>memory r=3 4p 128 B</td>
|
||||||
|
<td>37,736</td>
|
||||||
|
<td>4.61</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>js-mem-4p-20k-1k-r3</code></td>
|
||||||
|
<td>memory r=3 4p 1 KiB</td>
|
||||||
|
<td>33,916</td>
|
||||||
|
<td>33.12</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>Replica <strong>1 vs 3</strong> on this stand (file 1p 128 B): r=1 is
|
||||||
|
18,888 vs r=3 14,330 (1.32× if r=3 is the slower one). Memory r=1 1p
|
||||||
|
29,972 vs memory r=3 20,188.</p>
|
||||||
|
<figure>
|
||||||
|
<img src="charts-optimal/replicas.png" alt="Replica cost" />
|
||||||
|
<figcaption aria-hidden="true">Replica cost</figcaption>
|
||||||
|
</figure>
|
||||||
|
<h3 id="delay-reconnect-tax-udp-mqtt">2.3 Delay, reconnect tax, UDP,
|
||||||
|
MQTT</h3>
|
||||||
|
<table>
|
||||||
|
<colgroup>
|
||||||
|
<col style="width: 29%" />
|
||||||
|
<col style="width: 33%" />
|
||||||
|
<col style="width: 37%" />
|
||||||
|
</colgroup>
|
||||||
|
<thead>
|
||||||
|
<tr class="header">
|
||||||
|
<th>Probe</th>
|
||||||
|
<th>Result</th>
|
||||||
|
<th>Meaning</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>NATS ping (persistent sockets) p50 / p99</td>
|
||||||
|
<td>0.456ms / 1.140ms</td>
|
||||||
|
<td>Quiet hop with a long-lived TCP conn</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>NATS <strong>reconnect-per-message</strong> p50 / p99</td>
|
||||||
|
<td>0.503ms / 1.750ms</td>
|
||||||
|
<td>TCP+NATS handshake on every pub — this is the tax to avoid</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>UDP echo 510→511 p99</td>
|
||||||
|
<td>0.363ms</td>
|
||||||
|
<td>Raw datagram ceiling on the same veth (no NATS)</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>MQTT QoS0 5k×128 B</td>
|
||||||
|
<td>44862 pubs/s</td>
|
||||||
|
<td>nats-server MQTT gateway on <code>:1883</code></td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>Core 1p1s 128 B this run: 662,227 pub msgs/s. Flood delay is still
|
||||||
|
backlog/consume_rate, not RTT.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="alternative-transports-why-we-are-not-switching-the-fabric">3.
|
||||||
|
Alternative transports (why we are not switching the fabric)</h2>
|
||||||
|
<p>NATS already <strong>is</strong> persistent TCP sockets with a tiny
|
||||||
|
binary protocol, automatic reconnect, and optional JetStream durability.
|
||||||
|
“Reduce connection overhead” is a <strong>client</strong> discipline:
|
||||||
|
hold the connection. The reconnect probe exists to prove that opening a
|
||||||
|
socket per job would dominate ping RTT.</p>
|
||||||
|
<table>
|
||||||
|
<colgroup>
|
||||||
|
<col style="width: 7%" />
|
||||||
|
<col style="width: 44%" />
|
||||||
|
<col style="width: 32%" />
|
||||||
|
<col style="width: 15%" />
|
||||||
|
</colgroup>
|
||||||
|
<thead>
|
||||||
|
<tr class="header">
|
||||||
|
<th>Idea</th>
|
||||||
|
<th>Fit for Verae jobs/events/archive</th>
|
||||||
|
<th>Throughput vs NATS core</th>
|
||||||
|
<th>Durability</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><strong>NATS core pub/sub</strong></td>
|
||||||
|
<td>Fan-out, request-reply (<code>verae.billing.*</code>)</td>
|
||||||
|
<td>Highest we measured (~0.5–2M msgs/s)</td>
|
||||||
|
<td>None</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><strong>NATS JetStream file r=3</strong></td>
|
||||||
|
<td>Jobs, webhooks, archive</td>
|
||||||
|
<td>~8–23k on this lab; see metal projection</td>
|
||||||
|
<td>Disk + 1-node loss</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><strong>NATS JetStream memory r=3</strong></td>
|
||||||
|
<td>Events mailbox</td>
|
||||||
|
<td>~22–36k on this lab</td>
|
||||||
|
<td>RAM + 1-node loss; <strong>empty on full restart</strong></td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><strong>MQTT</strong> (NATS gateway or Mosquitto)</td>
|
||||||
|
<td>IoT endpoints that already speak MQTT</td>
|
||||||
|
<td>This probe: 44862 pubs/s QoS0 — typically <strong>well
|
||||||
|
below</strong> NATS core; QoS1 ≈ JetStream-ish with more chatter</td>
|
||||||
|
<td>QoS1/2 session state; not our WORM model</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><strong>UDP</strong></td>
|
||||||
|
<td>Telemetry that may drop</td>
|
||||||
|
<td>RTT 0.363ms p99 — fastest hop, <strong>no</strong> reliability, no
|
||||||
|
cluster, no auth</td>
|
||||||
|
<td>None</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><strong>Custom persistent sockets / HTTP long-poll</strong></td>
|
||||||
|
<td>Worse NATS</td>
|
||||||
|
<td>You would re-implement reconnect, flow control, and fan-out</td>
|
||||||
|
<td>DIY</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><strong>WebSocket</strong></td>
|
||||||
|
<td>Browsers only</td>
|
||||||
|
<td>Extra framing; NATS already has WS for UIs, not for middleware</td>
|
||||||
|
<td>Same as core/JS behind it</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><strong>QUIC / WebTransport</strong></td>
|
||||||
|
<td>Lossy WAN / browsers</td>
|
||||||
|
<td>NATS QUIC is not the lab path; 10GbE LAN does not need it</td>
|
||||||
|
<td>Same</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><strong>Kafka / Redis streams</strong></td>
|
||||||
|
<td>Heavy log replay</td>
|
||||||
|
<td>Higher ops cost; not on <code>vmbr1</code> today</td>
|
||||||
|
<td>Yes, heavier</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p><strong>MQTT:</strong> NATS documents MQTT as an <em>enabling</em>
|
||||||
|
gateway for existing IoT, and prefers NATS end-to-end for greenfield.
|
||||||
|
Zapier cloud never talks NATS or MQTT; it talks HTTPS. Putting MQTT in
|
||||||
|
the middle of timestamp jobs adds protocol translation and QoS timers
|
||||||
|
without helping <code>jobId → events</code>. Use MQTT only if a device
|
||||||
|
already cannot speak NATS.</p>
|
||||||
|
<p><strong>UDP:</strong> Fine as a <em>measurement</em> of veth RTT.
|
||||||
|
Unusable as the job fabric (no ack, no replica, no flow control). NATS
|
||||||
|
ping is already within a small multiple of UDP on this bridge.</p>
|
||||||
|
<p><strong>Persistence sockets:</strong> Middleware and keep already
|
||||||
|
keep <code>NATS_URL</code> connections open. Optimal: one connection (or
|
||||||
|
a small pool) per process, <code>max_reconnect</code>, jitter, no
|
||||||
|
<code>connect()</code> in the per-job path. The reconnect ladder is the
|
||||||
|
anti-pattern.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="optimal-configurations">4. Optimal configurations</h2>
|
||||||
|
<h3 id="ns1-lab-now">4.1 NS1 lab (now)</h3>
|
||||||
|
<ol type="1">
|
||||||
|
<li><strong>Leave 8 cores / 16 GiB</strong> on nats-a/b/c and the
|
||||||
|
worker. Host has 40 cores / 377 GiB; this is cheap.</li>
|
||||||
|
<li><strong><code>max_mem: 8G</code></strong> stays. Required for memory
|
||||||
|
streams.</li>
|
||||||
|
<li><strong>File r=3 on ZFS</strong> for jobs/webhooks/archive. tmpfs
|
||||||
|
doubled JS 1p (7.4k→17k) but <strong>loses the stream on reboot</strong>
|
||||||
|
— unacceptable for archive.</li>
|
||||||
|
<li><strong>Memory r=3 for <code>ZAPIER_EVENTS</code></strong> if we
|
||||||
|
accept “all three nats CTs reboot ⇒ in-flight waiters fall back to HTTP
|
||||||
|
poll.” That matches the designed wait path
|
||||||
|
(<code>GET /api/status/{jobId}</code>).</li>
|
||||||
|
<li><strong>r=1 only for throwaway benches</strong>, never product
|
||||||
|
streams. Replica=3 is the point of three guests.</li>
|
||||||
|
<li><strong>veth on vmbr1, no fake 10G NICs.</strong> Already 10000Mb/s;
|
||||||
|
JS does not fill it.</li>
|
||||||
|
<li><strong>Pin cpusets</strong> later if keep/fleet steal; not required
|
||||||
|
to beat these numbers.</li>
|
||||||
|
<li>Clients: persistent NATS connections; pull consumers with bounded
|
||||||
|
<code>max_ack_pending</code> for webhooks.</li>
|
||||||
|
</ol>
|
||||||
|
<h3 id="three-hp-dl360-gen10-projection-not-measured">4.2 Three HP DL360
|
||||||
|
Gen10 (projection — not measured)</h3>
|
||||||
|
<p>Assumed bill of materials (state it in the buy):</p>
|
||||||
|
<table>
|
||||||
|
<colgroup>
|
||||||
|
<col style="width: 36%" />
|
||||||
|
<col style="width: 63%" />
|
||||||
|
</colgroup>
|
||||||
|
<thead>
|
||||||
|
<tr class="header">
|
||||||
|
<th>Piece</th>
|
||||||
|
<th>Assumption</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>Chassis</td>
|
||||||
|
<td>3× DL360 Gen10 1U</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>CPU</td>
|
||||||
|
<td>Dual 2nd-gen Xeon <strong>Gold</strong> (e.g. 6226R 16c or 6248 20c
|
||||||
|
— <strong>32–40 cores/box</strong>)</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>Memory</td>
|
||||||
|
<td>DDR4-2933, <strong>192–384 GiB</strong>/box (6–12×32 GiB); NATS will
|
||||||
|
not use most of it</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>Storage</td>
|
||||||
|
<td><strong>NVMe M.2 or U.2</strong> for
|
||||||
|
<code>/var/lib/nats/jetstream</code> (XFS or ext4, <strong>not</strong>
|
||||||
|
shared ZFS over the network). RAID1 of two NVMe if you want disk HA
|
||||||
|
<em>inside</em> a box</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>Network</td>
|
||||||
|
<td><strong>10GbE</strong> (FlexibleLOM or PCIe); dedicated VLAN for
|
||||||
|
<code>:4222</code>+<code>:6222</code>. Do not share with public
|
||||||
|
<code>vmbr0</code> traffic</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>OS</td>
|
||||||
|
<td>Debian/Ubuntu bare metal, <code>nats-server</code> systemd, same
|
||||||
|
<code>nats.conf</code> as lab (bind private IP only)</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p><strong>What changes vs NS1 LXC</strong></p>
|
||||||
|
<table>
|
||||||
|
<colgroup>
|
||||||
|
<col style="width: 15%" />
|
||||||
|
<col style="width: 20%" />
|
||||||
|
<col style="width: 18%" />
|
||||||
|
<col style="width: 45%" />
|
||||||
|
</colgroup>
|
||||||
|
<thead>
|
||||||
|
<tr class="header">
|
||||||
|
<th>Factor</th>
|
||||||
|
<th>NS1 today</th>
|
||||||
|
<th>3× DL360</th>
|
||||||
|
<th>Effect on JS file r=3</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>Failure domain</td>
|
||||||
|
<td>1 Proxmox host</td>
|
||||||
|
<td>3 chassis, 3 NVMe, 3 NICs</td>
|
||||||
|
<td>r=3 <strong>means</strong> something</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>Disk</td>
|
||||||
|
<td>Shared ZFS SSD2</td>
|
||||||
|
<td>Local NVMe fsync ~50–150 µs</td>
|
||||||
|
<td>Big win vs ZFS; similar to tmpfs for sequential 128 B</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>Replica path</td>
|
||||||
|
<td>veth/bridge (~µs–tens of µs)</td>
|
||||||
|
<td>10GbE RTT typically <strong>50–200 µs</strong></td>
|
||||||
|
<td><strong>Slower than same-host tmpfs</strong>, faster than a bad
|
||||||
|
SAN</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>CPU</td>
|
||||||
|
<td>8 of 40 shared</td>
|
||||||
|
<td>32–40 dedicated Gold cores</td>
|
||||||
|
<td>Headroom for many clients, not 10× JS</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>NIC</td>
|
||||||
|
<td>software 10G veth, already ~5 Gbit/s core</td>
|
||||||
|
<td>real 10GbE ~9 Gbit/s TCP</td>
|
||||||
|
<td>Core NATS can grow; JS r=3 stays replica-bound</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p><strong>Projected bands</strong> (128 B, 3-node cluster, dedicated
|
||||||
|
10GbE, local NVMe, 8+ cores pinned to nats-server):</p>
|
||||||
|
<table>
|
||||||
|
<colgroup>
|
||||||
|
<col style="width: 16%" />
|
||||||
|
<col style="width: 34%" />
|
||||||
|
<col style="width: 29%" />
|
||||||
|
<col style="width: 19%" />
|
||||||
|
</colgroup>
|
||||||
|
<thead>
|
||||||
|
<tr class="header">
|
||||||
|
<th>Workload</th>
|
||||||
|
<th>NS1 measured (best)</th>
|
||||||
|
<th>DL360 projection</th>
|
||||||
|
<th>Confidence</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>Core pub/sub 1p</td>
|
||||||
|
<td>0.5–0.8M</td>
|
||||||
|
<td><strong>0.8–2M</strong></td>
|
||||||
|
<td>Medium — NIC + syscall, plenty of CPU</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>Core 4p4s 1 KiB</td>
|
||||||
|
<td>~0.6–0.7M (~0.6 GB/s)</td>
|
||||||
|
<td><strong>~1M msgs/s / ~1 GB/s</strong> approaching 10GbE</td>
|
||||||
|
<td>Medium</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>JS file r=1</td>
|
||||||
|
<td>this run r=1</td>
|
||||||
|
<td><strong>80–200k</strong> pubs/s</td>
|
||||||
|
<td>Medium — NVMe + no replica wait</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>JS file r=3</td>
|
||||||
|
<td>7–23k (ZFS/tmpfs)</td>
|
||||||
|
<td><strong>40–80k</strong> pubs/s</td>
|
||||||
|
<td>Medium-low — replica RTT dominates; 3 NVMe still help vs shared
|
||||||
|
ZFS</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>JS memory r=3</td>
|
||||||
|
<td>22–36k</td>
|
||||||
|
<td><strong>50–100k</strong></td>
|
||||||
|
<td>Medium-low — RAM + 10GbE ack</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>Ping p99</td>
|
||||||
|
<td>0.7–1.4 ms</td>
|
||||||
|
<td><strong>0.2–0.6 ms</strong></td>
|
||||||
|
<td>Medium — real NIC but no Proxmox tax</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>These are <strong>not</strong> DL360 measurements. Scale from: (a)
|
||||||
|
our replica-1 vs replica-3 ratio once this run’s r=1 numbers exist, (b)
|
||||||
|
tmpfs vs ZFS ratio (2.35× on 1p), (c) Synadia/nats bench async file r=1
|
||||||
|
~100–400k on NVMe loopback, derated for 10GbE RTT.</p>
|
||||||
|
<p><strong>Buy notes:</strong> M.2 via Dual uFF / enablement kit; put
|
||||||
|
JetStream on NVMe <strong>directly</strong>, not behind a RAID
|
||||||
|
controller write-through unless you measure. 1GbE onboard is a trap —
|
||||||
|
use 10GbE for <code>:6222</code>. Dual Gold is for isolation (nats vs
|
||||||
|
worm/tree vs OS), not because JS needs 56 cores.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="what-we-are-not-doing">5. What we are not doing</h2>
|
||||||
|
<ul>
|
||||||
|
<li>MQTT as the Zapier or middleware transport.</li>
|
||||||
|
<li>UDP for jobs.</li>
|
||||||
|
<li>Emulated 10G fiber NICs on LXC.</li>
|
||||||
|
<li>tmpfs as the production store.</li>
|
||||||
|
<li>r=1 for product streams.</li>
|
||||||
|
<li>Connect-per-job.</li>
|
||||||
|
</ul>
|
||||||
|
<p>Re-run exhaustive: <code>bash scripts/exhaustive-ns1-study.sh</code>
|
||||||
|
on NS1.</p>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
|
|
@ -0,0 +1,160 @@
|
||||||
|
**Progress report — optimal configuration study** · `20260912T055851Z` (UTC) · all code on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`)
|
||||||
|
|
||||||
|
This document folds every ladder we have run (1-core ZFS, NS1-orchestrated, tmpfs maximize, and this exhaustive 8c/16G **ZFS** factorial) plus UDP / MQTT / reconnect probes. It recommends a lab config and a **three-box HP DL360 Gen10** projection. veth/10G was not changed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Verdict (read this first)
|
||||||
|
|
||||||
|
**Keep NATS + JetStream.** Do not replace the fabric with MQTT, UDP, or a custom persistent-socket protocol for Verae jobs/events/archive. Those are either slower, less durable, or already what NATS is.
|
||||||
|
|
||||||
|
**Lab (NS1, one host, three LXC) — optimal now**
|
||||||
|
|
||||||
|
| Stream | Storage | Replicas | Why |
|
||||||
|
|--------|---------|----------|-----|
|
||||||
|
| `ZAPIER_JOBS`, `ZAPIER_WEBHOOKS`, `VERAE_ARCHIVE` | **file** (ZFS) | **3** | Survive a nats LXC death; archive must persist |
|
||||||
|
| `ZAPIER_EVENTS` | **memory** | **3** | Waiters are latency-sensitive; events rebuild from job status |
|
||||||
|
| `ZAPIER_USAGE` | file | 3 | Telemetry, limits + max-age |
|
||||||
|
|
||||||
|
Keep **8 cores / 16 GiB / `max_mem: 8G`** on 510–513 (already live). Do **not** leave JetStream on tmpfs. Do **not** drop product streams to r=1. Reuse **one NATS connection per process** (already true in middleware); never connect-per-message.
|
||||||
|
|
||||||
|
**Metal (3× DL360 Gen10) — optimal later**
|
||||||
|
|
||||||
|
Same stream table. File store on **local NVMe/M.2**, not a shared SAN. Cluster + client on **10GbE** (or 25GbE if you already have it). Dual Gold Xeon is surplus CPU for this workload; 8–16 cores dedicated to `nats-server` is enough. Expected JS file r=3: **~40–80k** 128 B pubs/s (about **3–6×** this lab’s 8c ZFS 1p, **2–4×** tmpfs 1p) — bounded by **10GbE replica RTT**, not by Xeon clocks. Core NATS will sit in the **1–3M msgs/s** band until the NIC saturates (~9 Gbit/s ≈ 8–9M × 128 B theoretical; CPU and client will hit first).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. What we actually ran (this exhaustive pass)
|
||||||
|
|
||||||
|
Live cluster during this run: LXC 510–513 **8 cores / 16 GiB**, JetStream **on ZFS** (tmpfs from the maximize study was already unmounted). Extra factorial: file/memory × replicas 1/3, 4 KiB file r=3, reconnect-per-message ping, UDP echo 510→511, MQTT QoS0 against nats-a `:1883`. Product streams were not the bench target.
|
||||||
|
|
||||||
|
### 2.1 Cross-study history
|
||||||
|
|
||||||
|
| Study | Env | Core 1p pub | JS file r=3 1p | JS mem r=3 4p | Ping p99 |
|
||||||
|
| --- | --- | --- | --- | --- | --- |
|
||||||
|
| `20260912T051237Z` | 1c/1G ZFS (NS1 orch.) | 502,502 | 7,393 | — | 1.377ms |
|
||||||
|
| `20260912T053120Z` | 8c/16G tmpfs + mem extra | 599,004 | 17,388 | 36,355 | 0.684ms |
|
||||||
|
| `20260912T055851Z` | 8c/16G ZFS exhaustive `20260912T055851Z` | 662,227 | 14,330 | 37,736 | 1.140ms |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### 2.2 This run — JetStream factorial
|
||||||
|
|
||||||
|
| Run | What | Pub msgs/s | Pub MB/s |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `js-file-1p-20k-128-r1` | file r=1 1p 128 B | 18,888 | 2.31 |
|
||||||
|
| `js-file-4p-50k-128-r1` | file r=1 4p 128 B | 24,560 | 3.00 |
|
||||||
|
| `js-1p-20k-128-r3` | file r=3 1p 128 B | 14,330 | 1.75 |
|
||||||
|
| `js-4p-50k-128-r3` | file r=3 4p 128 B | 19,232 | 2.35 |
|
||||||
|
| `js-4p-20k-1k-r3` | file r=3 4p 1 KiB | 15,197 | 14.84 |
|
||||||
|
| `js-file-1p-20k-4k-r3` | file r=3 1p 4 KiB | 8,673 | 33.88 |
|
||||||
|
| `js-mem-1p-20k-128-r1` | memory r=1 1p 128 B | 29,972 | 3.66 |
|
||||||
|
| `js-mem-4p-50k-128-r1` | memory r=1 4p 128 B | 64,923 | 7.93 |
|
||||||
|
| `js-mem-1p-20k-128-r3` | memory r=3 1p 128 B | 20,188 | 2.46 |
|
||||||
|
| `js-mem-4p-50k-128-r3` | memory r=3 4p 128 B | 37,736 | 4.61 |
|
||||||
|
| `js-mem-4p-20k-1k-r3` | memory r=3 4p 1 KiB | 33,916 | 33.12 |
|
||||||
|
|
||||||
|
Replica **1 vs 3** on this stand (file 1p 128 B): r=1 is 18,888 vs r=3 14,330 (1.32× if r=3 is the slower one). Memory r=1 1p 29,972 vs memory r=3 20,188.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### 2.3 Delay, reconnect tax, UDP, MQTT
|
||||||
|
|
||||||
|
| Probe | Result | Meaning |
|
||||||
|
|-------|--------|---------|
|
||||||
|
| NATS ping (persistent sockets) p50 / p99 | 0.456ms / 1.140ms | Quiet hop with a long-lived TCP conn |
|
||||||
|
| NATS **reconnect-per-message** p50 / p99 | 0.503ms / 1.750ms | TCP+NATS handshake on every pub — this is the tax to avoid |
|
||||||
|
| UDP echo 510→511 p99 | 0.363ms | Raw datagram ceiling on the same veth (no NATS) |
|
||||||
|
| MQTT QoS0 5k×128 B | 44862 pubs/s | nats-server MQTT gateway on `:1883` |
|
||||||
|
|
||||||
|
Core 1p1s 128 B this run: 662,227 pub msgs/s. Flood delay is still backlog/consume_rate, not RTT.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Alternative transports (why we are not switching the fabric)
|
||||||
|
|
||||||
|
NATS already **is** persistent TCP sockets with a tiny binary protocol, automatic reconnect, and optional JetStream durability. “Reduce connection overhead” is a **client** discipline: hold the connection. The reconnect probe exists to prove that opening a socket per job would dominate ping RTT.
|
||||||
|
|
||||||
|
| Idea | Fit for Verae jobs/events/archive | Throughput vs NATS core | Durability |
|
||||||
|
|------|-----------------------------------|-------------------------|------------|
|
||||||
|
| **NATS core pub/sub** | Fan-out, request-reply (`verae.billing.*`) | Highest we measured (~0.5–2M msgs/s) | None |
|
||||||
|
| **NATS JetStream file r=3** | Jobs, webhooks, archive | ~8–23k on this lab; see metal projection | Disk + 1-node loss |
|
||||||
|
| **NATS JetStream memory r=3** | Events mailbox | ~22–36k on this lab | RAM + 1-node loss; **empty on full restart** |
|
||||||
|
| **MQTT** (NATS gateway or Mosquitto) | IoT endpoints that already speak MQTT | This probe: 44862 pubs/s QoS0 — typically **well below** NATS core; QoS1 ≈ JetStream-ish with more chatter | QoS1/2 session state; not our WORM model |
|
||||||
|
| **UDP** | Telemetry that may drop | RTT 0.363ms p99 — fastest hop, **no** reliability, no cluster, no auth | None |
|
||||||
|
| **Custom persistent sockets / HTTP long-poll** | Worse NATS | You would re-implement reconnect, flow control, and fan-out | DIY |
|
||||||
|
| **WebSocket** | Browsers only | Extra framing; NATS already has WS for UIs, not for middleware | Same as core/JS behind it |
|
||||||
|
| **QUIC / WebTransport** | Lossy WAN / browsers | NATS QUIC is not the lab path; 10GbE LAN does not need it | Same |
|
||||||
|
| **Kafka / Redis streams** | Heavy log replay | Higher ops cost; not on `vmbr1` today | Yes, heavier |
|
||||||
|
|
||||||
|
**MQTT:** NATS documents MQTT as an *enabling* gateway for existing IoT, and prefers NATS end-to-end for greenfield. Zapier cloud never talks NATS or MQTT; it talks HTTPS. Putting MQTT in the middle of timestamp jobs adds protocol translation and QoS timers without helping `jobId → events`. Use MQTT only if a device already cannot speak NATS.
|
||||||
|
|
||||||
|
**UDP:** Fine as a *measurement* of veth RTT. Unusable as the job fabric (no ack, no replica, no flow control). NATS ping is already within a small multiple of UDP on this bridge.
|
||||||
|
|
||||||
|
**Persistence sockets:** Middleware and keep already keep `NATS_URL` connections open. Optimal: one connection (or a small pool) per process, `max_reconnect`, jitter, no `connect()` in the per-job path. The reconnect ladder is the anti-pattern.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Optimal configurations
|
||||||
|
|
||||||
|
### 4.1 NS1 lab (now)
|
||||||
|
|
||||||
|
1. **Leave 8 cores / 16 GiB** on nats-a/b/c and the worker. Host has 40 cores / 377 GiB; this is cheap.
|
||||||
|
2. **`max_mem: 8G`** stays. Required for memory streams.
|
||||||
|
3. **File r=3 on ZFS** for jobs/webhooks/archive. tmpfs doubled JS 1p (7.4k→17k) but **loses the stream on reboot** — unacceptable for archive.
|
||||||
|
4. **Memory r=3 for `ZAPIER_EVENTS`** if we accept “all three nats CTs reboot ⇒ in-flight waiters fall back to HTTP poll.” That matches the designed wait path (`GET /api/status/{jobId}`).
|
||||||
|
5. **r=1 only for throwaway benches**, never product streams. Replica=3 is the point of three guests.
|
||||||
|
6. **veth on vmbr1, no fake 10G NICs.** Already 10000Mb/s; JS does not fill it.
|
||||||
|
7. **Pin cpusets** later if keep/fleet steal; not required to beat these numbers.
|
||||||
|
8. Clients: persistent NATS connections; pull consumers with bounded `max_ack_pending` for webhooks.
|
||||||
|
|
||||||
|
### 4.2 Three HP DL360 Gen10 (projection — not measured)
|
||||||
|
|
||||||
|
Assumed bill of materials (state it in the buy):
|
||||||
|
|
||||||
|
| Piece | Assumption |
|
||||||
|
|-------|------------|
|
||||||
|
| Chassis | 3× DL360 Gen10 1U |
|
||||||
|
| CPU | Dual 2nd-gen Xeon **Gold** (e.g. 6226R 16c or 6248 20c — **32–40 cores/box**) |
|
||||||
|
| Memory | DDR4-2933, **192–384 GiB**/box (6–12×32 GiB); NATS will not use most of it |
|
||||||
|
| Storage | **NVMe M.2 or U.2** for `/var/lib/nats/jetstream` (XFS or ext4, **not** shared ZFS over the network). RAID1 of two NVMe if you want disk HA *inside* a box |
|
||||||
|
| Network | **10GbE** (FlexibleLOM or PCIe); dedicated VLAN for `:4222`+`:6222`. Do not share with public `vmbr0` traffic |
|
||||||
|
| OS | Debian/Ubuntu bare metal, `nats-server` systemd, same `nats.conf` as lab (bind private IP only) |
|
||||||
|
|
||||||
|
**What changes vs NS1 LXC**
|
||||||
|
|
||||||
|
| Factor | NS1 today | 3× DL360 | Effect on JS file r=3 |
|
||||||
|
|--------|-----------|----------|------------------------|
|
||||||
|
| Failure domain | 1 Proxmox host | 3 chassis, 3 NVMe, 3 NICs | r=3 **means** something |
|
||||||
|
| Disk | Shared ZFS SSD2 | Local NVMe fsync ~50–150 µs | Big win vs ZFS; similar to tmpfs for sequential 128 B |
|
||||||
|
| Replica path | veth/bridge (~µs–tens of µs) | 10GbE RTT typically **50–200 µs** | **Slower than same-host tmpfs**, faster than a bad SAN |
|
||||||
|
| CPU | 8 of 40 shared | 32–40 dedicated Gold cores | Headroom for many clients, not 10× JS |
|
||||||
|
| NIC | software 10G veth, already ~5 Gbit/s core | real 10GbE ~9 Gbit/s TCP | Core NATS can grow; JS r=3 stays replica-bound |
|
||||||
|
|
||||||
|
**Projected bands** (128 B, 3-node cluster, dedicated 10GbE, local NVMe, 8+ cores pinned to nats-server):
|
||||||
|
|
||||||
|
| Workload | NS1 measured (best) | DL360 projection | Confidence |
|
||||||
|
|----------|---------------------|------------------|------------|
|
||||||
|
| Core pub/sub 1p | 0.5–0.8M | **0.8–2M** | Medium — NIC + syscall, plenty of CPU |
|
||||||
|
| Core 4p4s 1 KiB | ~0.6–0.7M (~0.6 GB/s) | **~1M msgs/s / ~1 GB/s** approaching 10GbE | Medium |
|
||||||
|
| JS file r=1 | this run r=1 | **80–200k** pubs/s | Medium — NVMe + no replica wait |
|
||||||
|
| JS file r=3 | 7–23k (ZFS/tmpfs) | **40–80k** pubs/s | Medium-low — replica RTT dominates; 3 NVMe still help vs shared ZFS |
|
||||||
|
| JS memory r=3 | 22–36k | **50–100k** | Medium-low — RAM + 10GbE ack |
|
||||||
|
| Ping p99 | 0.7–1.4 ms | **0.2–0.6 ms** | Medium — real NIC but no Proxmox tax |
|
||||||
|
|
||||||
|
These are **not** DL360 measurements. Scale from: (a) our replica-1 vs replica-3 ratio once this run’s r=1 numbers exist, (b) tmpfs vs ZFS ratio (2.35× on 1p), (c) Synadia/nats bench async file r=1 ~100–400k on NVMe loopback, derated for 10GbE RTT.
|
||||||
|
|
||||||
|
**Buy notes:** M.2 via Dual uFF / enablement kit; put JetStream on NVMe **directly**, not behind a RAID controller write-through unless you measure. 1GbE onboard is a trap — use 10GbE for `:6222`. Dual Gold is for isolation (nats vs worm/tree vs OS), not because JS needs 56 cores.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. What we are not doing
|
||||||
|
|
||||||
|
- MQTT as the Zapier or middleware transport.
|
||||||
|
- UDP for jobs.
|
||||||
|
- Emulated 10G fiber NICs on LXC.
|
||||||
|
- tmpfs as the production store.
|
||||||
|
- r=1 for product streams.
|
||||||
|
- Connect-per-job.
|
||||||
|
|
||||||
|
Re-run exhaustive: `bash scripts/exhaustive-ns1-study.sh` on NS1.
|
||||||
Binary file not shown.
|
|
@ -0,0 +1,2 @@
|
||||||
|
=== udp-ping-1k-128 ===
|
||||||
|
{"mode":"udp-ping","count":1000,"size":128,"host":"10.10.10.21","min":"0.096ms","avg":"0.138ms","p50":"0.118ms","p99":"0.363ms","max":"8.411ms","p50_us":118,"p99_us":363}
|
||||||
|
|
@ -0,0 +1,50 @@
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"vmid": "511",
|
||||||
|
"name": "nats-a",
|
||||||
|
"server_name": "nats-a",
|
||||||
|
"host": "10.10.10.21",
|
||||||
|
"port": 4222,
|
||||||
|
"connections": 5,
|
||||||
|
"in_msgs": 724962,
|
||||||
|
"out_msgs": 1400288,
|
||||||
|
"in_bytes": 282983166,
|
||||||
|
"out_bytes": 248599798,
|
||||||
|
"cpu": 2,
|
||||||
|
"cores": 8,
|
||||||
|
"mem": 27095040,
|
||||||
|
"jetstream": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"vmid": "512",
|
||||||
|
"name": "nats-b",
|
||||||
|
"server_name": "nats-b",
|
||||||
|
"host": "10.10.10.22",
|
||||||
|
"port": 4222,
|
||||||
|
"connections": 1,
|
||||||
|
"in_msgs": 509195,
|
||||||
|
"out_msgs": 671813,
|
||||||
|
"in_bytes": 236003120,
|
||||||
|
"out_bytes": 379784062,
|
||||||
|
"cpu": 0,
|
||||||
|
"cores": 1,
|
||||||
|
"mem": 36028416,
|
||||||
|
"jetstream": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"vmid": "513",
|
||||||
|
"name": "nats-c",
|
||||||
|
"server_name": "nats-c",
|
||||||
|
"host": "10.10.10.23",
|
||||||
|
"port": 4222,
|
||||||
|
"connections": 0,
|
||||||
|
"in_msgs": 703508,
|
||||||
|
"out_msgs": 1715907,
|
||||||
|
"in_bytes": 287777683,
|
||||||
|
"out_bytes": 404038933,
|
||||||
|
"cpu": 0,
|
||||||
|
"cores": 1,
|
||||||
|
"mem": 27295744,
|
||||||
|
"jetstream": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
@ -0,0 +1,50 @@
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"vmid": "511",
|
||||||
|
"name": "nats-a",
|
||||||
|
"server_name": "nats-a",
|
||||||
|
"host": "10.10.10.21",
|
||||||
|
"port": 4222,
|
||||||
|
"connections": 5,
|
||||||
|
"in_msgs": 31443,
|
||||||
|
"out_msgs": 31725,
|
||||||
|
"in_bytes": 1501396,
|
||||||
|
"out_bytes": 1924069,
|
||||||
|
"cpu": 1,
|
||||||
|
"cores": 8,
|
||||||
|
"mem": 15785984,
|
||||||
|
"jetstream": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"vmid": "512",
|
||||||
|
"name": "nats-b",
|
||||||
|
"server_name": "nats-b",
|
||||||
|
"host": "10.10.10.22",
|
||||||
|
"port": 4222,
|
||||||
|
"connections": 1,
|
||||||
|
"in_msgs": 24330,
|
||||||
|
"out_msgs": 24480,
|
||||||
|
"in_bytes": 1424418,
|
||||||
|
"out_bytes": 1398562,
|
||||||
|
"cpu": 1,
|
||||||
|
"cores": 1,
|
||||||
|
"mem": 15892480,
|
||||||
|
"jetstream": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"vmid": "513",
|
||||||
|
"name": "nats-c",
|
||||||
|
"server_name": "nats-c",
|
||||||
|
"host": "10.10.10.23",
|
||||||
|
"port": 4222,
|
||||||
|
"connections": 0,
|
||||||
|
"in_msgs": 19051,
|
||||||
|
"out_msgs": 19034,
|
||||||
|
"in_bytes": 1373987,
|
||||||
|
"out_bytes": 1036018,
|
||||||
|
"cpu": 0,
|
||||||
|
"cores": 1,
|
||||||
|
"mem": 14262272,
|
||||||
|
"jetstream": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
@ -80,6 +80,21 @@ if [[ "${JS_EXTRA_MEMORY:-0}" == "1" ]]; then
|
||||||
js_rm
|
js_rm
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# Factorial extras: replicas=1 vs 3, file vs memory (does not touch product streams).
|
||||||
|
if [[ "${EXHAUSTIVE:-0}" == "1" ]]; then
|
||||||
|
js_rm
|
||||||
|
run_one js-file-1p-20k-128-r1 bench.js.e1 --js --purge --pub 1 --msgs 20000 --size 128 --replicas 1 --storage file --maxbytes=512MB --stream=benchstream
|
||||||
|
js_rm
|
||||||
|
run_one js-file-4p-50k-128-r1 bench.js.e2 --js --purge --pub 4 --msgs 50000 --size 128 --replicas 1 --storage file --maxbytes=512MB --stream=benchstream
|
||||||
|
js_rm
|
||||||
|
run_one js-mem-1p-20k-128-r1 bench.js.e3 --js --purge --pub 1 --msgs 20000 --size 128 --replicas 1 --storage memory --maxbytes=512MB --stream=benchstream
|
||||||
|
js_rm
|
||||||
|
run_one js-mem-4p-50k-128-r1 bench.js.e4 --js --purge --pub 4 --msgs 50000 --size 128 --replicas 1 --storage memory --maxbytes=512MB --stream=benchstream
|
||||||
|
js_rm
|
||||||
|
run_one js-file-1p-20k-4k-r3 bench.js.e5 --js --purge --pub 1 --msgs 20000 --size 4096 --replicas 3 --storage file --maxbytes=512MB --stream=benchstream
|
||||||
|
js_rm
|
||||||
|
fi
|
||||||
|
|
||||||
# Round-trip delay (two connections, through the cluster) at several loads
|
# Round-trip delay (two connections, through the cluster) at several loads
|
||||||
sudo pct exec "$CLIENT_VMID" -- bash -lc "
|
sudo pct exec "$CLIENT_VMID" -- bash -lc "
|
||||||
set -e
|
set -e
|
||||||
|
|
@ -107,6 +122,9 @@ node latency.mjs $n $sz $p $mode
|
||||||
# copy latest probe
|
# copy latest probe
|
||||||
sudo pct exec "$CLIENT_VMID" -- bash -c 'cat > /tmp/nats-lat/latency.mjs' < "$ROOT/scripts/latency.mjs"
|
sudo pct exec "$CLIENT_VMID" -- bash -c 'cat > /tmp/nats-lat/latency.mjs' < "$ROOT/scripts/latency.mjs"
|
||||||
lat lat-ping-1k-128 1000 128 1 ping
|
lat lat-ping-1k-128 1000 128 1 ping
|
||||||
|
if [[ "${EXHAUSTIVE:-0}" == "1" ]]; then
|
||||||
|
lat lat-reconnect-200-128 200 128 1 reconnect
|
||||||
|
fi
|
||||||
lat lat-1p-5k-128 5000 128 1 flood
|
lat lat-1p-5k-128 5000 128 1 flood
|
||||||
lat lat-4p-10k-128 10000 128 4 flood
|
lat lat-4p-10k-128 10000 128 4 flood
|
||||||
lat lat-8p-20k-128 20000 128 8 flood
|
lat lat-8p-20k-128 20000 128 8 flood
|
||||||
|
|
|
||||||
400
packages/verae-nats-cluster/scripts/build-optimal-report.py
Normal file
400
packages/verae-nats-cluster/scripts/build-optimal-report.py
Normal file
|
|
@ -0,0 +1,400 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""One large comparison report from all NS1 result folders + extras (UDP/MQTT/reconnect)."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import importlib.util
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import matplotlib
|
||||||
|
|
||||||
|
matplotlib.use("Agg")
|
||||||
|
import matplotlib.pyplot as plt
|
||||||
|
from matplotlib.ticker import FuncFormatter
|
||||||
|
|
||||||
|
_spec = importlib.util.spec_from_file_location(
|
||||||
|
"bench_report", Path(__file__).resolve().parent / "bench-report.py"
|
||||||
|
)
|
||||||
|
_br = importlib.util.module_from_spec(_spec)
|
||||||
|
assert _spec.loader
|
||||||
|
_spec.loader.exec_module(_br)
|
||||||
|
fmt_int = _br.fmt_int
|
||||||
|
parse_bench = _br.parse_bench
|
||||||
|
parse_lat = _br.parse_lat
|
||||||
|
|
||||||
|
INDIGO, DEEP, TEAL, AMBER, MUTED = "#4f46e5", "#312e81", "#047857", "#b45309", "#5b6178"
|
||||||
|
|
||||||
|
|
||||||
|
def k_fmt(x, _p=None):
|
||||||
|
if x >= 1_000_000:
|
||||||
|
return f"{x/1e6:.2f}M"
|
||||||
|
if x >= 1000:
|
||||||
|
return f"{x/1000:.0f}k"
|
||||||
|
return f"{x:.0f}"
|
||||||
|
|
||||||
|
|
||||||
|
def load_folder(folder: Path) -> dict:
|
||||||
|
thru, lats, extra = {}, {}, {}
|
||||||
|
if not folder.is_dir():
|
||||||
|
return {"thru": thru, "lats": lats, "extra": extra, "stamp": folder.name}
|
||||||
|
for f in folder.glob("*.txt"):
|
||||||
|
text = f.read_text(encoding="utf-8", errors="replace")
|
||||||
|
if f.name.startswith("host-"):
|
||||||
|
continue
|
||||||
|
if "mqtt" in f.name or "udp-ping" in f.name:
|
||||||
|
for line in text.splitlines():
|
||||||
|
line = line.strip()
|
||||||
|
if line.startswith("{"):
|
||||||
|
extra[f.stem] = json.loads(line)
|
||||||
|
break
|
||||||
|
continue
|
||||||
|
lat = parse_lat(text)
|
||||||
|
if lat:
|
||||||
|
lats[f.stem] = lat
|
||||||
|
continue
|
||||||
|
p = parse_bench(text)
|
||||||
|
if p.get("pub_msgs") or p.get("agg_msgs"):
|
||||||
|
thru[f.stem] = p
|
||||||
|
return {"thru": thru, "lats": lats, "extra": extra, "stamp": folder.name}
|
||||||
|
|
||||||
|
|
||||||
|
def pub(d, run):
|
||||||
|
p = d["thru"].get(run) or {}
|
||||||
|
return p.get("pub_msgs")
|
||||||
|
|
||||||
|
|
||||||
|
def latp(d, run, key="p99"):
|
||||||
|
p = d["lats"].get(run) or {}
|
||||||
|
return p.get(key, "")
|
||||||
|
|
||||||
|
|
||||||
|
def row(*cells):
|
||||||
|
return "| " + " | ".join(cells) + " |"
|
||||||
|
|
||||||
|
|
||||||
|
def save(fig, path: Path):
|
||||||
|
fig.savefig(path, dpi=150, bbox_inches="tight", facecolor="white")
|
||||||
|
plt.close(fig)
|
||||||
|
|
||||||
|
|
||||||
|
def charts(latest: dict, folders: list[dict], dest: Path):
|
||||||
|
dest.mkdir(parents=True, exist_ok=True)
|
||||||
|
plt.rcParams.update({"font.size": 9, "axes.grid": True, "grid.color": "#d9dce8"})
|
||||||
|
# r=1 vs r=3 file/mem from latest
|
||||||
|
labels, file_r, mem_r = [], [], []
|
||||||
|
for lab, fr, mr in (
|
||||||
|
("1p file", "js-file-1p-20k-128-r1", "js-1p-20k-128-r3"),
|
||||||
|
("4p file", "js-file-4p-50k-128-r1", "js-4p-50k-128-r3"),
|
||||||
|
("1p mem", "js-mem-1p-20k-128-r1", "js-mem-1p-20k-128-r3"),
|
||||||
|
("4p mem", "js-mem-4p-50k-128-r1", "js-mem-4p-50k-128-r3"),
|
||||||
|
):
|
||||||
|
a, b = pub(latest, fr), pub(latest, mr)
|
||||||
|
if a or b:
|
||||||
|
labels.append(lab)
|
||||||
|
file_r.append(int(a or 0))
|
||||||
|
mem_r.append(int(b or 0))
|
||||||
|
if labels:
|
||||||
|
fig, ax = plt.subplots(figsize=(9.2, 4.3))
|
||||||
|
x = range(len(labels))
|
||||||
|
ax.bar([i - 0.2 for i in x], file_r, 0.4, label="replicas=1", color=TEAL)
|
||||||
|
ax.bar([i + 0.2 for i in x], mem_r, 0.4, label="replicas=3", color=AMBER)
|
||||||
|
ax.set_xticks(list(x), labels)
|
||||||
|
ax.set_ylabel("pub msgs/s")
|
||||||
|
ax.set_title("This run: replica cost (1 vs 3)")
|
||||||
|
ax.yaxis.set_major_formatter(FuncFormatter(k_fmt))
|
||||||
|
ax.legend()
|
||||||
|
save(fig, dest / "replicas.png")
|
||||||
|
# historical JS 1p file
|
||||||
|
names, vals = [], []
|
||||||
|
for d, label in zip(
|
||||||
|
folders,
|
||||||
|
[d["stamp"] for d in folders],
|
||||||
|
):
|
||||||
|
v = pub(d, "js-1p-20k-128-r3")
|
||||||
|
if v:
|
||||||
|
names.append(label[-7:] if len(label) > 8 else label)
|
||||||
|
vals.append(int(v))
|
||||||
|
if names:
|
||||||
|
fig, ax = plt.subplots(figsize=(9.2, 4.0))
|
||||||
|
ax.bar(names, vals, color=INDIGO)
|
||||||
|
ax.set_title("JS file r=3 1p 128 B across studies")
|
||||||
|
ax.set_ylabel("pub msgs/s")
|
||||||
|
ax.yaxis.set_major_formatter(FuncFormatter(k_fmt))
|
||||||
|
save(fig, dest / "history-js1p.png")
|
||||||
|
|
||||||
|
|
||||||
|
def write_md(latest: dict, hist: list[dict], charts_rel: str) -> str:
|
||||||
|
t = latest["thru"]
|
||||||
|
e = latest["extra"]
|
||||||
|
mqtt = e.get("mqtt-qos0-5k-128") or {}
|
||||||
|
udp = e.get("udp-ping-1k-128") or {}
|
||||||
|
ping = latest["lats"].get("lat-ping-1k-128") or {}
|
||||||
|
recon = latest["lats"].get("lat-reconnect-200-128") or {}
|
||||||
|
|
||||||
|
def js_table():
|
||||||
|
runs = [
|
||||||
|
("js-file-1p-20k-128-r1", "file r=1 1p 128 B"),
|
||||||
|
("js-file-4p-50k-128-r1", "file r=1 4p 128 B"),
|
||||||
|
("js-1p-20k-128-r3", "file r=3 1p 128 B"),
|
||||||
|
("js-4p-50k-128-r3", "file r=3 4p 128 B"),
|
||||||
|
("js-4p-20k-1k-r3", "file r=3 4p 1 KiB"),
|
||||||
|
("js-file-1p-20k-4k-r3", "file r=3 1p 4 KiB"),
|
||||||
|
("js-mem-1p-20k-128-r1", "memory r=1 1p 128 B"),
|
||||||
|
("js-mem-4p-50k-128-r1", "memory r=1 4p 128 B"),
|
||||||
|
("js-mem-1p-20k-128-r3", "memory r=3 1p 128 B"),
|
||||||
|
("js-mem-4p-50k-128-r3", "memory r=3 4p 128 B"),
|
||||||
|
("js-mem-4p-20k-1k-r3", "memory r=3 4p 1 KiB"),
|
||||||
|
]
|
||||||
|
lines = [
|
||||||
|
row("Run", "What", "Pub msgs/s", "Pub MB/s"),
|
||||||
|
row("---", "---", "---", "---"),
|
||||||
|
]
|
||||||
|
for k, lab in runs:
|
||||||
|
p = t.get(k)
|
||||||
|
if not p:
|
||||||
|
continue
|
||||||
|
lines.append(row(f"`{k}`", lab, fmt_int(p.get("pub_msgs")), p.get("pub_mb") or "—"))
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
hist_lines = [
|
||||||
|
row("Study", "Env", "Core 1p pub", "JS file r=3 1p", "JS mem r=3 4p", "Ping p99"),
|
||||||
|
row("---", "---", "---", "---", "---", "---"),
|
||||||
|
]
|
||||||
|
labels_env = {
|
||||||
|
"20260912T045131Z": "1c/1G ZFS (off-box orch.)",
|
||||||
|
"20260912T051237Z": "1c/1G ZFS (NS1 orch.)",
|
||||||
|
"20260912T053120Z": "8c/16G tmpfs + mem extra",
|
||||||
|
}
|
||||||
|
for d in hist + [latest]:
|
||||||
|
env = labels_env.get(d["stamp"], f"8c/16G ZFS exhaustive `{d['stamp']}`")
|
||||||
|
hist_lines.append(
|
||||||
|
row(
|
||||||
|
f"`{d['stamp']}`",
|
||||||
|
env,
|
||||||
|
fmt_int(pub(d, "core-1p1s-50k-128")),
|
||||||
|
fmt_int(pub(d, "js-1p-20k-128-r3")),
|
||||||
|
fmt_int(pub(d, "js-mem-4p-50k-128-r3")),
|
||||||
|
latp(d, "lat-ping-1k-128"),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
r1 = int(pub(latest, "js-file-1p-20k-128-r1") or 0)
|
||||||
|
r3 = int(pub(latest, "js-1p-20k-128-r3") or 0)
|
||||||
|
mem1 = int(pub(latest, "js-mem-1p-20k-128-r1") or 0)
|
||||||
|
mem3 = int(pub(latest, "js-mem-1p-20k-128-r3") or 0)
|
||||||
|
replica_cost = f"{r1/r3:.2f}×" if r3 else "—"
|
||||||
|
mem_gain = f"{mem1/r3:.2f}×" if r3 and mem1 else "—"
|
||||||
|
|
||||||
|
mqtt_rate = mqtt.get("pubs_per_sec", "—")
|
||||||
|
udp_p99 = udp.get("p99", "—")
|
||||||
|
|
||||||
|
return f"""**Progress report — optimal configuration study** · `{latest['stamp']}` (UTC) · all code on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`)
|
||||||
|
|
||||||
|
This document folds every ladder we have run (1-core ZFS, NS1-orchestrated, tmpfs maximize, and this exhaustive 8c/16G **ZFS** factorial) plus UDP / MQTT / reconnect probes. It recommends a lab config and a **three-box HP DL360 Gen10** projection. veth/10G was not changed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Verdict (read this first)
|
||||||
|
|
||||||
|
**Keep NATS + JetStream.** Do not replace the fabric with MQTT, UDP, or a custom persistent-socket protocol for Verae jobs/events/archive. Those are either slower, less durable, or already what NATS is.
|
||||||
|
|
||||||
|
**Lab (NS1, one host, three LXC) — optimal now**
|
||||||
|
|
||||||
|
| Stream | Storage | Replicas | Why |
|
||||||
|
|--------|---------|----------|-----|
|
||||||
|
| `ZAPIER_JOBS`, `ZAPIER_WEBHOOKS`, `VERAE_ARCHIVE` | **file** (ZFS) | **3** | Survive a nats LXC death; archive must persist |
|
||||||
|
| `ZAPIER_EVENTS` | **memory** | **3** | Waiters are latency-sensitive; events rebuild from job status |
|
||||||
|
| `ZAPIER_USAGE` | file | 3 | Telemetry, limits + max-age |
|
||||||
|
|
||||||
|
Keep **8 cores / 16 GiB / `max_mem: 8G`** on 510–513 (already live). Do **not** leave JetStream on tmpfs. Do **not** drop product streams to r=1. Reuse **one NATS connection per process** (already true in middleware); never connect-per-message.
|
||||||
|
|
||||||
|
**Metal (3× DL360 Gen10) — optimal later**
|
||||||
|
|
||||||
|
Same stream table. File store on **local NVMe/M.2**, not a shared SAN. Cluster + client on **10GbE** (or 25GbE if you already have it). Dual Gold Xeon is surplus CPU for this workload; 8–16 cores dedicated to `nats-server` is enough. Expected JS file r=3: **~40–80k** 128 B pubs/s (about **3–6×** this lab’s 8c ZFS 1p, **2–4×** tmpfs 1p) — bounded by **10GbE replica RTT**, not by Xeon clocks. Core NATS will sit in the **1–3M msgs/s** band until the NIC saturates (~9 Gbit/s ≈ 8–9M × 128 B theoretical; CPU and client will hit first).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. What we actually ran (this exhaustive pass)
|
||||||
|
|
||||||
|
Live cluster during this run: LXC 510–513 **8 cores / 16 GiB**, JetStream **on ZFS** (tmpfs from the maximize study was already unmounted). Extra factorial: file/memory × replicas 1/3, 4 KiB file r=3, reconnect-per-message ping, UDP echo 510→511, MQTT QoS0 against nats-a `:1883`. Product streams were not the bench target.
|
||||||
|
|
||||||
|
### 2.1 Cross-study history
|
||||||
|
|
||||||
|
{chr(10).join(hist_lines)}
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### 2.2 This run — JetStream factorial
|
||||||
|
|
||||||
|
{js_table()}
|
||||||
|
|
||||||
|
Replica **1 vs 3** on this stand (file 1p 128 B): r=1 is {fmt_int(str(r1) if r1 else None)} vs r=3 {fmt_int(str(r3) if r3 else None)} ({replica_cost} if r=3 is the slower one). Memory r=1 1p {fmt_int(str(mem1) if mem1 else None)} vs memory r=3 {fmt_int(str(mem3) if mem3 else None)}.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### 2.3 Delay, reconnect tax, UDP, MQTT
|
||||||
|
|
||||||
|
| Probe | Result | Meaning |
|
||||||
|
|-------|--------|---------|
|
||||||
|
| NATS ping (persistent sockets) p50 / p99 | {ping.get('p50','—')} / {ping.get('p99','—')} | Quiet hop with a long-lived TCP conn |
|
||||||
|
| NATS **reconnect-per-message** p50 / p99 | {recon.get('p50','—')} / {recon.get('p99','—')} | TCP+NATS handshake on every pub — this is the tax to avoid |
|
||||||
|
| UDP echo 510→511 p99 | {udp_p99} | Raw datagram ceiling on the same veth (no NATS) |
|
||||||
|
| MQTT QoS0 5k×128 B | {mqtt_rate} pubs/s | nats-server MQTT gateway on `:1883` |
|
||||||
|
|
||||||
|
Core 1p1s 128 B this run: {fmt_int(pub(latest, 'core-1p1s-50k-128'))} pub msgs/s. Flood delay is still backlog/consume_rate, not RTT.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Alternative transports (why we are not switching the fabric)
|
||||||
|
|
||||||
|
NATS already **is** persistent TCP sockets with a tiny binary protocol, automatic reconnect, and optional JetStream durability. “Reduce connection overhead” is a **client** discipline: hold the connection. The reconnect probe exists to prove that opening a socket per job would dominate ping RTT.
|
||||||
|
|
||||||
|
| Idea | Fit for Verae jobs/events/archive | Throughput vs NATS core | Durability |
|
||||||
|
|------|-----------------------------------|-------------------------|------------|
|
||||||
|
| **NATS core pub/sub** | Fan-out, request-reply (`verae.billing.*`) | Highest we measured (~0.5–2M msgs/s) | None |
|
||||||
|
| **NATS JetStream file r=3** | Jobs, webhooks, archive | ~8–23k on this lab; see metal projection | Disk + 1-node loss |
|
||||||
|
| **NATS JetStream memory r=3** | Events mailbox | ~22–36k on this lab | RAM + 1-node loss; **empty on full restart** |
|
||||||
|
| **MQTT** (NATS gateway or Mosquitto) | IoT endpoints that already speak MQTT | This probe: {mqtt_rate} pubs/s QoS0 — typically **well below** NATS core; QoS1 ≈ JetStream-ish with more chatter | QoS1/2 session state; not our WORM model |
|
||||||
|
| **UDP** | Telemetry that may drop | RTT {udp_p99} p99 — fastest hop, **no** reliability, no cluster, no auth | None |
|
||||||
|
| **Custom persistent sockets / HTTP long-poll** | Worse NATS | You would re-implement reconnect, flow control, and fan-out | DIY |
|
||||||
|
| **WebSocket** | Browsers only | Extra framing; NATS already has WS for UIs, not for middleware | Same as core/JS behind it |
|
||||||
|
| **QUIC / WebTransport** | Lossy WAN / browsers | NATS QUIC is not the lab path; 10GbE LAN does not need it | Same |
|
||||||
|
| **Kafka / Redis streams** | Heavy log replay | Higher ops cost; not on `vmbr1` today | Yes, heavier |
|
||||||
|
|
||||||
|
**MQTT:** NATS documents MQTT as an *enabling* gateway for existing IoT, and prefers NATS end-to-end for greenfield. Zapier cloud never talks NATS or MQTT; it talks HTTPS. Putting MQTT in the middle of timestamp jobs adds protocol translation and QoS timers without helping `jobId → events`. Use MQTT only if a device already cannot speak NATS.
|
||||||
|
|
||||||
|
**UDP:** Fine as a *measurement* of veth RTT. Unusable as the job fabric (no ack, no replica, no flow control). NATS ping is already within a small multiple of UDP on this bridge.
|
||||||
|
|
||||||
|
**Persistence sockets:** Middleware and keep already keep `NATS_URL` connections open. Optimal: one connection (or a small pool) per process, `max_reconnect`, jitter, no `connect()` in the per-job path. The reconnect ladder is the anti-pattern.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Optimal configurations
|
||||||
|
|
||||||
|
### 4.1 NS1 lab (now)
|
||||||
|
|
||||||
|
1. **Leave 8 cores / 16 GiB** on nats-a/b/c and the worker. Host has 40 cores / 377 GiB; this is cheap.
|
||||||
|
2. **`max_mem: 8G`** stays. Required for memory streams.
|
||||||
|
3. **File r=3 on ZFS** for jobs/webhooks/archive. tmpfs doubled JS 1p (7.4k→17k) but **loses the stream on reboot** — unacceptable for archive.
|
||||||
|
4. **Memory r=3 for `ZAPIER_EVENTS`** if we accept “all three nats CTs reboot ⇒ in-flight waiters fall back to HTTP poll.” That matches the designed wait path (`GET /api/status/{{jobId}}`).
|
||||||
|
5. **r=1 only for throwaway benches**, never product streams. Replica=3 is the point of three guests.
|
||||||
|
6. **veth on vmbr1, no fake 10G NICs.** Already 10000Mb/s; JS does not fill it.
|
||||||
|
7. **Pin cpusets** later if keep/fleet steal; not required to beat these numbers.
|
||||||
|
8. Clients: persistent NATS connections; pull consumers with bounded `max_ack_pending` for webhooks.
|
||||||
|
|
||||||
|
### 4.2 Three HP DL360 Gen10 (projection — not measured)
|
||||||
|
|
||||||
|
Assumed bill of materials (state it in the buy):
|
||||||
|
|
||||||
|
| Piece | Assumption |
|
||||||
|
|-------|------------|
|
||||||
|
| Chassis | 3× DL360 Gen10 1U |
|
||||||
|
| CPU | Dual 2nd-gen Xeon **Gold** (e.g. 6226R 16c or 6248 20c — **32–40 cores/box**) |
|
||||||
|
| Memory | DDR4-2933, **192–384 GiB**/box (6–12×32 GiB); NATS will not use most of it |
|
||||||
|
| Storage | **NVMe M.2 or U.2** for `/var/lib/nats/jetstream` (XFS or ext4, **not** shared ZFS over the network). RAID1 of two NVMe if you want disk HA *inside* a box |
|
||||||
|
| Network | **10GbE** (FlexibleLOM or PCIe); dedicated VLAN for `:4222`+`:6222`. Do not share with public `vmbr0` traffic |
|
||||||
|
| OS | Debian/Ubuntu bare metal, `nats-server` systemd, same `nats.conf` as lab (bind private IP only) |
|
||||||
|
|
||||||
|
**What changes vs NS1 LXC**
|
||||||
|
|
||||||
|
| Factor | NS1 today | 3× DL360 | Effect on JS file r=3 |
|
||||||
|
|--------|-----------|----------|------------------------|
|
||||||
|
| Failure domain | 1 Proxmox host | 3 chassis, 3 NVMe, 3 NICs | r=3 **means** something |
|
||||||
|
| Disk | Shared ZFS SSD2 | Local NVMe fsync ~50–150 µs | Big win vs ZFS; similar to tmpfs for sequential 128 B |
|
||||||
|
| Replica path | veth/bridge (~µs–tens of µs) | 10GbE RTT typically **50–200 µs** | **Slower than same-host tmpfs**, faster than a bad SAN |
|
||||||
|
| CPU | 8 of 40 shared | 32–40 dedicated Gold cores | Headroom for many clients, not 10× JS |
|
||||||
|
| NIC | software 10G veth, already ~5 Gbit/s core | real 10GbE ~9 Gbit/s TCP | Core NATS can grow; JS r=3 stays replica-bound |
|
||||||
|
|
||||||
|
**Projected bands** (128 B, 3-node cluster, dedicated 10GbE, local NVMe, 8+ cores pinned to nats-server):
|
||||||
|
|
||||||
|
| Workload | NS1 measured (best) | DL360 projection | Confidence |
|
||||||
|
|----------|---------------------|------------------|------------|
|
||||||
|
| Core pub/sub 1p | 0.5–0.8M | **0.8–2M** | Medium — NIC + syscall, plenty of CPU |
|
||||||
|
| Core 4p4s 1 KiB | ~0.6–0.7M (~0.6 GB/s) | **~1M msgs/s / ~1 GB/s** approaching 10GbE | Medium |
|
||||||
|
| JS file r=1 | this run r=1 | **80–200k** pubs/s | Medium — NVMe + no replica wait |
|
||||||
|
| JS file r=3 | 7–23k (ZFS/tmpfs) | **40–80k** pubs/s | Medium-low — replica RTT dominates; 3 NVMe still help vs shared ZFS |
|
||||||
|
| JS memory r=3 | 22–36k | **50–100k** | Medium-low — RAM + 10GbE ack |
|
||||||
|
| Ping p99 | 0.7–1.4 ms | **0.2–0.6 ms** | Medium — real NIC but no Proxmox tax |
|
||||||
|
|
||||||
|
These are **not** DL360 measurements. Scale from: (a) our replica-1 vs replica-3 ratio once this run’s r=1 numbers exist, (b) tmpfs vs ZFS ratio (2.35× on 1p), (c) Synadia/nats bench async file r=1 ~100–400k on NVMe loopback, derated for 10GbE RTT.
|
||||||
|
|
||||||
|
**Buy notes:** M.2 via Dual uFF / enablement kit; put JetStream on NVMe **directly**, not behind a RAID controller write-through unless you measure. 1GbE onboard is a trap — use 10GbE for `:6222`. Dual Gold is for isolation (nats vs worm/tree vs OS), not because JS needs 56 cores.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. What we are not doing
|
||||||
|
|
||||||
|
- MQTT as the Zapier or middleware transport.
|
||||||
|
- UDP for jobs.
|
||||||
|
- Emulated 10G fiber NICs on LXC.
|
||||||
|
- tmpfs as the production store.
|
||||||
|
- r=1 for product streams.
|
||||||
|
- Connect-per-job.
|
||||||
|
|
||||||
|
Re-run exhaustive: `bash scripts/exhaustive-ns1-study.sh` on NS1.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def render(md_path: Path, html_path: Path, pdf_path: Path) -> None:
|
||||||
|
css = Path(__file__).resolve().parent / "docs-print.css"
|
||||||
|
header = html_path.with_suffix(".hdr.html")
|
||||||
|
banner = html_path.with_suffix(".ban.html")
|
||||||
|
header.write_text(f"<style>{css.read_text() if css.exists() else ''}</style>\n", encoding="utf-8")
|
||||||
|
banner.write_text(
|
||||||
|
'<div class="doc-banner">'
|
||||||
|
'<nav class="site"><a href="/">zapier.georgelambert.org</a></nav>'
|
||||||
|
'<div class="kicker">Verae Time × Zapier · progress report</div>'
|
||||||
|
"<h1>NATS optimal configuration study</h1>"
|
||||||
|
'<div class="source-path">packages/zapier-decisions/reports/optimal-config/REPORT.md</div>'
|
||||||
|
"</div>\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
r = subprocess.run(
|
||||||
|
[
|
||||||
|
"pandoc",
|
||||||
|
str(md_path),
|
||||||
|
"-o",
|
||||||
|
str(html_path),
|
||||||
|
"--standalone",
|
||||||
|
f"--resource-path={md_path.parent}",
|
||||||
|
"--highlight-style=breezedark",
|
||||||
|
"--metadata=title=NATS optimal configuration study",
|
||||||
|
f"--include-in-header={header}",
|
||||||
|
f"--include-before-body={banner}",
|
||||||
|
],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
header.unlink(missing_ok=True)
|
||||||
|
banner.unlink(missing_ok=True)
|
||||||
|
if r.returncode != 0:
|
||||||
|
raise SystemExit(f"pandoc failed: {r.stderr[-600:]}")
|
||||||
|
w = subprocess.run(["weasyprint", str(html_path), str(pdf_path)], capture_output=True, text=True)
|
||||||
|
if w.returncode != 0:
|
||||||
|
raise SystemExit(f"weasyprint failed: {w.stderr[-600:]}")
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
latest = Path(sys.argv[1])
|
||||||
|
hist_dirs = [Path(p) for p in sys.argv[2:] if p and Path(p).is_dir()]
|
||||||
|
data_latest = load_folder(latest)
|
||||||
|
hist = [load_folder(p) for p in hist_dirs]
|
||||||
|
charts_dir = latest / "charts-optimal"
|
||||||
|
charts(data_latest, hist + [data_latest], charts_dir)
|
||||||
|
md = write_md(data_latest, hist, "charts-optimal")
|
||||||
|
md_path = latest / "optimal-config.md"
|
||||||
|
md_path.write_text(md, encoding="utf-8")
|
||||||
|
html_path = latest / "optimal-config.html"
|
||||||
|
pdf_path = latest / "optimal-config.pdf"
|
||||||
|
render(md_path, html_path, pdf_path)
|
||||||
|
print(f"wrote {md_path}")
|
||||||
|
print(f"wrote {html_path}", file=sys.stderr)
|
||||||
|
print(f"wrote {pdf_path}", file=sys.stderr)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
84
packages/verae-nats-cluster/scripts/exhaustive-ns1-study.sh
Executable file
84
packages/verae-nats-cluster/scripts/exhaustive-ns1-study.sh
Executable file
|
|
@ -0,0 +1,84 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Exhaustive NS1 ladder on the *current* 8c/16G cluster with JetStream on ZFS.
|
||||||
|
# Adds r=1 vs r=3, file vs memory, reconnect tax, UDP echo, MQTT gateway probe.
|
||||||
|
# Does not tmpfs (product streams stay). Must run on NS1.
|
||||||
|
set -euo pipefail
|
||||||
|
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||||
|
export PATH="/usr/sbin:/usr/bin:/bin:/usr/local/bin:$PATH"
|
||||||
|
HOST="$(hostname -f 2>/dev/null || hostname)"
|
||||||
|
case "$HOST" in
|
||||||
|
NS1.GEORGELAMBERT.ORG|NS1|ns1.georgelambert.org|ns1) ;;
|
||||||
|
*) echo "refusing: exhaustive-ns1-study.sh must run on NS1, got '$HOST'" >&2; exit 1 ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
export EXHAUSTIVE=1
|
||||||
|
export JS_EXTRA_MEMORY=1
|
||||||
|
export COMPARE_DIR="${COMPARE_DIR:-$ROOT/results/20260912T051237Z}"
|
||||||
|
STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||||
|
export BENCH_OUT="$ROOT/results/$STAMP"
|
||||||
|
mkdir -p "$BENCH_OUT"
|
||||||
|
|
||||||
|
# MQTT gateway on nats-a only (vmbr1). Restored after.
|
||||||
|
MQTT_CONF=/etc/nats/nats.conf
|
||||||
|
enable_mqtt() {
|
||||||
|
sudo pct exec 511 -- bash -lc '
|
||||||
|
set -e
|
||||||
|
f=/etc/nats/nats.conf
|
||||||
|
grep -q "^mqtt {" "$f" && exit 0
|
||||||
|
cat >> "$f" <<EOF
|
||||||
|
|
||||||
|
mqtt {
|
||||||
|
host: 10.10.10.21
|
||||||
|
port: 1883
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
systemctl kill -s HUP nats-server || systemctl restart nats-server
|
||||||
|
'
|
||||||
|
sleep 2
|
||||||
|
}
|
||||||
|
disable_mqtt() {
|
||||||
|
sudo pct exec 511 -- bash -lc '
|
||||||
|
f=/etc/nats/nats.conf
|
||||||
|
python3 - "$f" <<'"'"'PY'"'"'
|
||||||
|
from pathlib import Path
|
||||||
|
import re, sys
|
||||||
|
p = Path(sys.argv[1])
|
||||||
|
t = re.sub(r"\nmqtt \{[^}]*\}\n", "\n", p.read_text(), flags=re.S)
|
||||||
|
p.write_text(t)
|
||||||
|
PY
|
||||||
|
systemctl kill -s HUP nats-server || true
|
||||||
|
' || true
|
||||||
|
}
|
||||||
|
|
||||||
|
enable_mqtt
|
||||||
|
trap disable_mqtt EXIT
|
||||||
|
|
||||||
|
bash "$ROOT/scripts/study-on-ns1.sh"
|
||||||
|
OUT="$BENCH_OUT"
|
||||||
|
echo "exhaustive extras into $OUT"
|
||||||
|
|
||||||
|
# UDP echo: server in 511, client in 510
|
||||||
|
sudo pct exec 511 -- bash -lc 'pkill -f "udp-probe.mjs server" >/dev/null 2>&1 || true'
|
||||||
|
sudo pct exec 511 -- bash -c 'cat > /tmp/udp-probe.mjs' < "$ROOT/scripts/udp-probe.mjs"
|
||||||
|
sudo pct exec 510 -- bash -c 'cat > /tmp/nats-lat/udp-probe.mjs' < "$ROOT/scripts/udp-probe.mjs"
|
||||||
|
sudo pct exec 511 -- bash -lc 'setsid node /tmp/udp-probe.mjs server 9999 >/tmp/udp-echo.log 2>&1 < /dev/null &'
|
||||||
|
sleep 1
|
||||||
|
echo "=== udp-ping-1k-128 ===" | tee "$OUT/udp-ping-1k-128.txt"
|
||||||
|
sudo pct exec 510 -- bash -lc 'node /tmp/nats-lat/udp-probe.mjs client 10.10.10.21 1000 9999 128' | tee -a "$OUT/udp-ping-1k-128.txt"
|
||||||
|
sudo pct exec 511 -- bash -lc 'pkill -f "udp-probe.mjs server" || true'
|
||||||
|
|
||||||
|
# MQTT QoS0
|
||||||
|
sudo pct exec 510 -- bash -lc '
|
||||||
|
set -e
|
||||||
|
cd /tmp/nats-lat
|
||||||
|
if [[ ! -d node_modules/mqtt ]]; then npm install --no-audit --no-fund mqtt@10 >/dev/null; fi
|
||||||
|
'
|
||||||
|
sudo pct exec 510 -- bash -c 'cat > /tmp/nats-lat/mqtt-probe.mjs' < "$ROOT/scripts/mqtt-probe.mjs"
|
||||||
|
echo "=== mqtt-qos0-5k-128 ===" | tee "$OUT/mqtt-qos0-5k-128.txt"
|
||||||
|
sudo pct exec 510 -- bash -lc 'cd /tmp/nats-lat && node mqtt-probe.mjs mqtt://10.10.10.21:1883 5000 128' | tee -a "$OUT/mqtt-qos0-5k-128.txt" || echo '{"error":"mqtt probe failed"}' | tee -a "$OUT/mqtt-qos0-5k-128.txt"
|
||||||
|
|
||||||
|
python3 "$ROOT/scripts/build-optimal-report.py" "$OUT" \
|
||||||
|
"$ROOT/results/20260912T045131Z" \
|
||||||
|
"$ROOT/results/20260912T051237Z" \
|
||||||
|
"$ROOT/results/20260912T053120Z"
|
||||||
|
echo "exhaustive complete $OUT"
|
||||||
|
|
@ -1,9 +1,10 @@
|
||||||
#!/usr/bin/env node
|
#!/usr/bin/env node
|
||||||
/**
|
/**
|
||||||
* Pub→sub round trip through the cluster (two connections).
|
* Pub→sub round trip through the cluster (two connections).
|
||||||
* Usage: NATS_URL=... node latency.mjs [count] [payloadBytes] [publishers] [ping|flood]
|
* Usage: NATS_URL=... node latency.mjs [count] [payloadBytes] [publishers] [ping|flood|reconnect]
|
||||||
* ping = sequential publish-wait (one-message RTT)
|
* ping = sequential publish-wait on persistent sockets (one-message RTT)
|
||||||
* flood = publish the batch then drain (queueing under burst)
|
* flood = publish the batch then drain (queueing under burst)
|
||||||
|
* reconnect = connect, one publish, wait, close — measures handshake tax
|
||||||
*/
|
*/
|
||||||
import { connect, headers } from "nats";
|
import { connect, headers } from "nats";
|
||||||
|
|
||||||
|
|
@ -23,7 +24,33 @@ function pct(sorted, p) {
|
||||||
}
|
}
|
||||||
|
|
||||||
const samples = [];
|
const samples = [];
|
||||||
if (mode === "ping") {
|
if (mode === "reconnect") {
|
||||||
|
const subNc = await connect({ servers, name: "lat-sub" });
|
||||||
|
let resolveOne = null;
|
||||||
|
const sub = subNc.subscribe(subject, { max: count });
|
||||||
|
const consume = (async () => {
|
||||||
|
for await (const m of sub) {
|
||||||
|
const sent = Number(m.headers?.get("t") || 0);
|
||||||
|
samples.push(Number(process.hrtime.bigint() / 1000n) - sent);
|
||||||
|
resolveOne?.();
|
||||||
|
}
|
||||||
|
})();
|
||||||
|
await subNc.flush();
|
||||||
|
for (let i = 0; i < count; i++) {
|
||||||
|
const got = new Promise((r) => {
|
||||||
|
resolveOne = r;
|
||||||
|
});
|
||||||
|
const pubNc = await connect({ servers, name: `lat-re-${i}` });
|
||||||
|
const h = headers();
|
||||||
|
h.set("t", String(process.hrtime.bigint() / 1000n));
|
||||||
|
pubNc.publish(subject, payload, { headers: h });
|
||||||
|
await pubNc.flush();
|
||||||
|
await got;
|
||||||
|
await pubNc.close();
|
||||||
|
}
|
||||||
|
await consume;
|
||||||
|
await subNc.close();
|
||||||
|
} else if (mode === "ping") {
|
||||||
const subNc = await connect({ servers, name: "lat-sub" });
|
const subNc = await connect({ servers, name: "lat-sub" });
|
||||||
const pubNc = await connect({ servers, name: "lat-pub" });
|
const pubNc = await connect({ servers, name: "lat-pub" });
|
||||||
let resolveOne = null;
|
let resolveOne = null;
|
||||||
|
|
|
||||||
33
packages/verae-nats-cluster/scripts/mqtt-probe.mjs
Executable file
33
packages/verae-nats-cluster/scripts/mqtt-probe.mjs
Executable file
|
|
@ -0,0 +1,33 @@
|
||||||
|
#!/usr/bin/env node
|
||||||
|
/** MQTT QoS0 publish rate against nats-server MQTT gateway. */
|
||||||
|
import mqtt from "mqtt";
|
||||||
|
|
||||||
|
const url = process.argv[2] || "mqtt://10.10.10.21:1883";
|
||||||
|
const count = Number(process.argv[3] || 5000);
|
||||||
|
const size = Number(process.argv[4] || 128);
|
||||||
|
const payload = Buffer.alloc(size, 9);
|
||||||
|
const topic = `bench/mqtt/${process.pid}`;
|
||||||
|
|
||||||
|
const c = mqtt.connect(url, { reconnectPeriod: 0, connectTimeout: 5000 });
|
||||||
|
await new Promise((res, rej) => {
|
||||||
|
c.on("connect", res);
|
||||||
|
c.on("error", rej);
|
||||||
|
});
|
||||||
|
const t0 = process.hrtime.bigint();
|
||||||
|
for (let i = 0; i < count; i++) {
|
||||||
|
await new Promise((res, rej) => c.publish(topic, payload, { qos: 0 }, (err) => (err ? rej(err) : res())));
|
||||||
|
}
|
||||||
|
const ns = Number(process.hrtime.bigint() - t0);
|
||||||
|
c.end(true);
|
||||||
|
const sec = ns / 1e9;
|
||||||
|
console.log(
|
||||||
|
JSON.stringify({
|
||||||
|
mode: "mqtt-qos0",
|
||||||
|
count,
|
||||||
|
size,
|
||||||
|
url,
|
||||||
|
secs: Number(sec.toFixed(3)),
|
||||||
|
pubs_per_sec: Math.round(count / sec),
|
||||||
|
mb_per_sec: Number(((count * size) / sec / 1e6).toFixed(2)),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
@ -8,6 +8,7 @@ bash -n "$ROOT/scripts/status.sh"
|
||||||
bash -n "$ROOT/scripts/bench.sh"
|
bash -n "$ROOT/scripts/bench.sh"
|
||||||
bash -n "$ROOT/scripts/study-on-ns1.sh"
|
bash -n "$ROOT/scripts/study-on-ns1.sh"
|
||||||
bash -n "$ROOT/scripts/maximize-ns1-study.sh"
|
bash -n "$ROOT/scripts/maximize-ns1-study.sh"
|
||||||
|
bash -n "$ROOT/scripts/exhaustive-ns1-study.sh"
|
||||||
grep -q 'host: {{IP}}' "$ROOT/conf/nats.conf.tmpl"
|
grep -q 'host: {{IP}}' "$ROOT/conf/nats.conf.tmpl"
|
||||||
grep -qv '0.0.0.0' "$ROOT/conf/nats.conf.tmpl"
|
grep -qv '0.0.0.0' "$ROOT/conf/nats.conf.tmpl"
|
||||||
if [[ ! -d /etc/pve/nodes ]]; then
|
if [[ ! -d /etc/pve/nodes ]]; then
|
||||||
|
|
|
||||||
54
packages/verae-nats-cluster/scripts/udp-probe.mjs
Executable file
54
packages/verae-nats-cluster/scripts/udp-probe.mjs
Executable file
|
|
@ -0,0 +1,54 @@
|
||||||
|
#!/usr/bin/env node
|
||||||
|
/** UDP echo RTT. server: node udp-probe.mjs server [port]
|
||||||
|
* client: node udp-probe.mjs client <host> <count> [port] [size] */
|
||||||
|
import dgram from "node:dgram";
|
||||||
|
|
||||||
|
const mode = process.argv[2] || "server";
|
||||||
|
const port = Number(process.argv[mode === "server" ? 3 : 5] || 9999);
|
||||||
|
|
||||||
|
if (mode === "server") {
|
||||||
|
const s = dgram.createSocket("udp4");
|
||||||
|
s.on("message", (msg, rinfo) => s.send(msg, rinfo.port, rinfo.address));
|
||||||
|
s.bind(port, "0.0.0.0", () => console.log(JSON.stringify({ mode: "udp-server", port })));
|
||||||
|
} else {
|
||||||
|
const host = process.argv[3];
|
||||||
|
const count = Number(process.argv[4] || 1000);
|
||||||
|
const size = Number(process.argv[6] || 128);
|
||||||
|
const sock = dgram.createSocket("udp4");
|
||||||
|
const payload = Buffer.alloc(size, 7);
|
||||||
|
const samples = [];
|
||||||
|
let i = 0;
|
||||||
|
const sendOne = () => {
|
||||||
|
const t0 = process.hrtime.bigint();
|
||||||
|
const once = (msg) => {
|
||||||
|
sock.off("message", once);
|
||||||
|
samples.push(Number(process.hrtime.bigint() - t0) / 1000);
|
||||||
|
i += 1;
|
||||||
|
if (i >= count) {
|
||||||
|
samples.sort((a, b) => a - b);
|
||||||
|
const us = (n) => `${(n / 1000).toFixed(3)}ms`;
|
||||||
|
const pct = (p) => samples[Math.min(samples.length - 1, Math.floor((p / 100) * samples.length))];
|
||||||
|
const sum = samples.reduce((a, b) => a + b, 0);
|
||||||
|
console.log(
|
||||||
|
JSON.stringify({
|
||||||
|
mode: "udp-ping",
|
||||||
|
count: samples.length,
|
||||||
|
size,
|
||||||
|
host,
|
||||||
|
min: us(samples[0]),
|
||||||
|
avg: us(sum / samples.length),
|
||||||
|
p50: us(pct(50)),
|
||||||
|
p99: us(pct(99)),
|
||||||
|
max: us(samples[samples.length - 1]),
|
||||||
|
p50_us: Math.round(pct(50)),
|
||||||
|
p99_us: Math.round(pct(99)),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
sock.close();
|
||||||
|
} else sendOne();
|
||||||
|
};
|
||||||
|
sock.on("message", once);
|
||||||
|
sock.send(payload, port, host);
|
||||||
|
};
|
||||||
|
sendOne();
|
||||||
|
}
|
||||||
|
|
@ -1,5 +1,12 @@
|
||||||
# Action log
|
# Action log
|
||||||
|
|
||||||
|
## 2026-09-12 — exhaustive optimal-config study
|
||||||
|
|
||||||
|
- 8c/16G ZFS factorial: file/memory × r=1/r=3, 4 KiB, reconnect, UDP echo, MQTT QoS0 on nats-a :1883 (then removed).
|
||||||
|
- JS memory r=1 4p **65k** pubs/s; file r=3 1p **14.3k**; MQTT QoS0 **45k**; UDP p99 **0.36 ms**.
|
||||||
|
- Combined report + DL360 Gen10 projection: `reports/optimal-config/REPORT.{md,html,pdf}`.
|
||||||
|
- Verdict: keep NATS+JetStream; no MQTT/UDP fabric; metal = 3 NVMe + 10GbE, expect JS r=3 ~40–80k.
|
||||||
|
|
||||||
## 2026-09-12 — maximized NS1 NATS study
|
## 2026-09-12 — maximized NS1 NATS study
|
||||||
|
|
||||||
- `pct set` 510–513 to 8 cores / 16 GiB; `max_mem: 8G`; tmpfs 8G on JetStream for the ladder only.
|
- `pct set` 510–513 to 8 cores / 16 GiB; `max_mem: 8G`; tmpfs 8G on JetStream for the ladder only.
|
||||||
|
|
|
||||||
|
|
@ -10,6 +10,7 @@ Lab log of architecture decisions, actions taken, and open todos while cleaning
|
||||||
| [reports/nats-cluster-bench.md](reports/nats-cluster-bench.md) | NATS cluster speed report (charts, HTML, PDF) |
|
| [reports/nats-cluster-bench.md](reports/nats-cluster-bench.md) | NATS cluster speed report (charts, HTML, PDF) |
|
||||||
| [reports/ns1-host/nats-cluster-bench-ns1.md](reports/ns1-host/nats-cluster-bench-ns1.md) | Second study: all code on NS1 (methodology + tuning) |
|
| [reports/ns1-host/nats-cluster-bench-ns1.md](reports/ns1-host/nats-cluster-bench-ns1.md) | Second study: all code on NS1 (methodology + tuning) |
|
||||||
| [reports/ns1-maximize/nats-cluster-bench-ns1.md](reports/ns1-maximize/nats-cluster-bench-ns1.md) | Maximized: 8c/16G/tmpfs vs baseline |
|
| [reports/ns1-maximize/nats-cluster-bench-ns1.md](reports/ns1-maximize/nats-cluster-bench-ns1.md) | Maximized: 8c/16G/tmpfs vs baseline |
|
||||||
|
| [reports/optimal-config/REPORT.md](reports/optimal-config/REPORT.md) | All studies + MQTT/UDP + DL360 projection |
|
||||||
| [LOG.md](LOG.md) | Chronological actions |
|
| [LOG.md](LOG.md) | Chronological actions |
|
||||||
| [TODO.md](TODO.md) | Open items |
|
| [TODO.md](TODO.md) | Open items |
|
||||||
| [decisions/](decisions/) | One file per decision |
|
| [decisions/](decisions/) | One file per decision |
|
||||||
|
|
|
||||||
|
|
@ -50,6 +50,19 @@ Measured from LXC **510** (not a nats-* server) against `10.10.10.21–23`.
|
||||||
|
|
||||||
Live cluster **keeps 8 cores / 16 GiB / max_mem 8G**. Product streams are back on ZFS (tmpfs was study-only).
|
Live cluster **keeps 8 cores / 16 GiB / max_mem 8G**. Product streams are back on ZFS (tmpfs was study-only).
|
||||||
|
|
||||||
|
**Optimal-config study (`20260912T055851Z`):** same 8c/16G cluster on **ZFS**, plus r=1 vs r=3, memory vs file, reconnect tax, UDP echo, MQTT QoS0. Combined report with DL360 Gen10 projection: [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/optimal-config/REPORT.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/optimal-config/REPORT.pdf).
|
||||||
|
|
||||||
|
| Finding | Number |
|
||||||
|
|---------|--------|
|
||||||
|
| JS file r=3 1p 128 B (8c ZFS) | 14,330 pubs/s |
|
||||||
|
| JS file r=1 1p | 18,888 (r=3 costs ~24% on same host) |
|
||||||
|
| JS memory r=1 4p | **64,923** (lab JS ceiling) |
|
||||||
|
| MQTT QoS0 | 44,862 pubs/s (not durable; ~15× slower than core) |
|
||||||
|
| UDP p99 510→511 | 0.363 ms vs NATS ping p99 1.14 ms |
|
||||||
|
| Reconnect-per-msg p99 | 1.75 ms vs persistent ping 1.14 ms |
|
||||||
|
|
||||||
|
**Do not switch the fabric to MQTT or UDP.** Optimal lab: file r=3 for jobs/archive, memory r=3 optional for events, persistent NATS connections. Metal: 3× DL360 Gen10, local NVMe, 10GbE — projected JS file r=3 **~40–80k** 128 B pubs/s (not measured).
|
||||||
|
|
||||||
Short tables: [BENCH.md](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md). Re-run: `bash scripts/bench.sh` on NS1.
|
Short tables: [BENCH.md](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md). Re-run: `bash scripts/bench.sh` on NS1.
|
||||||
|
|
||||||
| Kind | Load | Result |
|
| Kind | Load | Result |
|
||||||
|
|
|
||||||
853
packages/zapier-decisions/reports/optimal-config/REPORT.html
Normal file
853
packages/zapier-decisions/reports/optimal-config/REPORT.html
Normal file
|
|
@ -0,0 +1,853 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8" />
|
||||||
|
<meta name="generator" content="pandoc" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
|
||||||
|
<title>NATS optimal configuration study</title>
|
||||||
|
<style>
|
||||||
|
html {
|
||||||
|
color: #1a1a1a;
|
||||||
|
background-color: #fdfdfd;
|
||||||
|
}
|
||||||
|
body {
|
||||||
|
margin: 0 auto;
|
||||||
|
max-width: 36em;
|
||||||
|
padding-left: 50px;
|
||||||
|
padding-right: 50px;
|
||||||
|
padding-top: 50px;
|
||||||
|
padding-bottom: 50px;
|
||||||
|
hyphens: auto;
|
||||||
|
overflow-wrap: break-word;
|
||||||
|
text-rendering: optimizeLegibility;
|
||||||
|
font-kerning: normal;
|
||||||
|
}
|
||||||
|
@media (max-width: 600px) {
|
||||||
|
body {
|
||||||
|
font-size: 0.9em;
|
||||||
|
padding: 12px;
|
||||||
|
}
|
||||||
|
h1 {
|
||||||
|
font-size: 1.8em;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
@media print {
|
||||||
|
html {
|
||||||
|
background-color: white;
|
||||||
|
}
|
||||||
|
body {
|
||||||
|
background-color: transparent;
|
||||||
|
color: black;
|
||||||
|
font-size: 12pt;
|
||||||
|
}
|
||||||
|
p, h2, h3 {
|
||||||
|
orphans: 3;
|
||||||
|
widows: 3;
|
||||||
|
}
|
||||||
|
h2, h3, h4 {
|
||||||
|
page-break-after: avoid;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
p {
|
||||||
|
margin: 1em 0;
|
||||||
|
}
|
||||||
|
a {
|
||||||
|
color: #1a1a1a;
|
||||||
|
}
|
||||||
|
a:visited {
|
||||||
|
color: #1a1a1a;
|
||||||
|
}
|
||||||
|
img {
|
||||||
|
max-width: 100%;
|
||||||
|
}
|
||||||
|
svg {
|
||||||
|
height: auto;
|
||||||
|
max-width: 100%;
|
||||||
|
}
|
||||||
|
h1, h2, h3, h4, h5, h6 {
|
||||||
|
margin-top: 1.4em;
|
||||||
|
}
|
||||||
|
h5, h6 {
|
||||||
|
font-size: 1em;
|
||||||
|
font-style: italic;
|
||||||
|
}
|
||||||
|
h6 {
|
||||||
|
font-weight: normal;
|
||||||
|
}
|
||||||
|
ol, ul {
|
||||||
|
padding-left: 1.7em;
|
||||||
|
margin-top: 1em;
|
||||||
|
}
|
||||||
|
li > ol, li > ul {
|
||||||
|
margin-top: 0;
|
||||||
|
}
|
||||||
|
blockquote {
|
||||||
|
margin: 1em 0 1em 1.7em;
|
||||||
|
padding-left: 1em;
|
||||||
|
border-left: 2px solid #e6e6e6;
|
||||||
|
color: #606060;
|
||||||
|
}
|
||||||
|
code {
|
||||||
|
font-family: Menlo, Monaco, Consolas, 'Lucida Console', monospace;
|
||||||
|
font-size: 85%;
|
||||||
|
margin: 0;
|
||||||
|
hyphens: manual;
|
||||||
|
}
|
||||||
|
pre {
|
||||||
|
margin: 1em 0;
|
||||||
|
overflow: auto;
|
||||||
|
}
|
||||||
|
pre code {
|
||||||
|
padding: 0;
|
||||||
|
overflow: visible;
|
||||||
|
overflow-wrap: normal;
|
||||||
|
}
|
||||||
|
.sourceCode {
|
||||||
|
background-color: transparent;
|
||||||
|
overflow: visible;
|
||||||
|
}
|
||||||
|
hr {
|
||||||
|
background-color: #1a1a1a;
|
||||||
|
border: none;
|
||||||
|
height: 1px;
|
||||||
|
margin: 1em 0;
|
||||||
|
}
|
||||||
|
table {
|
||||||
|
margin: 1em 0;
|
||||||
|
border-collapse: collapse;
|
||||||
|
width: 100%;
|
||||||
|
overflow-x: auto;
|
||||||
|
display: block;
|
||||||
|
font-variant-numeric: lining-nums tabular-nums;
|
||||||
|
}
|
||||||
|
table caption {
|
||||||
|
margin-bottom: 0.75em;
|
||||||
|
}
|
||||||
|
tbody {
|
||||||
|
margin-top: 0.5em;
|
||||||
|
border-top: 1px solid #1a1a1a;
|
||||||
|
border-bottom: 1px solid #1a1a1a;
|
||||||
|
}
|
||||||
|
th {
|
||||||
|
border-top: 1px solid #1a1a1a;
|
||||||
|
padding: 0.25em 0.5em 0.25em 0.5em;
|
||||||
|
}
|
||||||
|
td {
|
||||||
|
padding: 0.125em 0.5em 0.25em 0.5em;
|
||||||
|
}
|
||||||
|
header {
|
||||||
|
margin-bottom: 4em;
|
||||||
|
text-align: center;
|
||||||
|
}
|
||||||
|
#TOC li {
|
||||||
|
list-style: none;
|
||||||
|
}
|
||||||
|
#TOC ul {
|
||||||
|
padding-left: 1.3em;
|
||||||
|
}
|
||||||
|
#TOC > ul {
|
||||||
|
padding-left: 0;
|
||||||
|
}
|
||||||
|
#TOC a:not(:hover) {
|
||||||
|
text-decoration: none;
|
||||||
|
}
|
||||||
|
code{white-space: pre-wrap;}
|
||||||
|
span.smallcaps{font-variant: small-caps;}
|
||||||
|
div.columns{display: flex; gap: min(4vw, 1.5em);}
|
||||||
|
div.column{flex: auto; overflow-x: auto;}
|
||||||
|
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
|
||||||
|
/* The extra [class] is a hack that increases specificity enough to
|
||||||
|
override a similar rule in reveal.js */
|
||||||
|
ul.task-list[class]{list-style: none;}
|
||||||
|
ul.task-list li input[type="checkbox"] {
|
||||||
|
font-size: inherit;
|
||||||
|
width: 0.8em;
|
||||||
|
margin: 0 0.8em 0.2em -1.6em;
|
||||||
|
vertical-align: middle;
|
||||||
|
}
|
||||||
|
.display.math{display: block; text-align: center; margin: 0.5rem auto;}
|
||||||
|
</style>
|
||||||
|
<style>/* Colored print + screen stylesheet for zapier.georgelambert.org */
|
||||||
|
:root {
|
||||||
|
--ink: #171a26;
|
||||||
|
--muted: #5b6178;
|
||||||
|
--line: #d9dce8;
|
||||||
|
--bg: #f4f5fb;
|
||||||
|
--paper: #ffffff;
|
||||||
|
--accent: #4f46e5;
|
||||||
|
--accent-deep: #312e81;
|
||||||
|
--accent-soft: #eef0fe;
|
||||||
|
--ok: #047857;
|
||||||
|
--warn: #8a5a00;
|
||||||
|
--code-bg: #1b1f33;
|
||||||
|
--code-fg: #e8ecff;
|
||||||
|
}
|
||||||
|
html { background: var(--bg); }
|
||||||
|
body {
|
||||||
|
margin: 0 auto;
|
||||||
|
padding: 1.5rem 1.25rem 3rem;
|
||||||
|
max-width: 48rem;
|
||||||
|
font: 15px/1.55 -apple-system, "Segoe UI", Georgia, serif;
|
||||||
|
color: var(--ink);
|
||||||
|
background: var(--paper);
|
||||||
|
}
|
||||||
|
.doc-banner {
|
||||||
|
background: linear-gradient(160deg, #312e81 0%, #4f46e5 60%, #7c74f0 100%);
|
||||||
|
color: #eef0fe;
|
||||||
|
margin: -1.5rem -1.25rem 1.5rem;
|
||||||
|
padding: 1.1rem 1.25rem 1rem;
|
||||||
|
}
|
||||||
|
.doc-banner a { color: #fff; }
|
||||||
|
.doc-banner .kicker {
|
||||||
|
letter-spacing: 0.12em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
font: 700 10px system-ui, sans-serif;
|
||||||
|
opacity: 0.8;
|
||||||
|
}
|
||||||
|
.doc-banner h1 { margin: 0.25rem 0 0; font-size: 1.45rem; color: #fff; }
|
||||||
|
h1, h2, h3, h4 { color: var(--accent-deep); page-break-after: avoid; }
|
||||||
|
h1 { font-size: 1.7rem; }
|
||||||
|
h2 {
|
||||||
|
font-size: 1.2rem;
|
||||||
|
border-bottom: 2px solid var(--accent);
|
||||||
|
padding-bottom: 0.2rem;
|
||||||
|
margin-top: 1.6rem;
|
||||||
|
}
|
||||||
|
h3 { font-size: 1.05rem; color: var(--accent); }
|
||||||
|
a { color: var(--accent); }
|
||||||
|
p, li { orphans: 3; widows: 3; }
|
||||||
|
code {
|
||||||
|
font-family: ui-monospace, Menlo, Consolas, monospace;
|
||||||
|
font-size: 0.86em;
|
||||||
|
background: var(--accent-soft);
|
||||||
|
color: var(--accent-deep);
|
||||||
|
padding: 0.08em 0.28em;
|
||||||
|
border-radius: 4px;
|
||||||
|
}
|
||||||
|
pre, div.sourceCode, div.sourceCode pre {
|
||||||
|
background: var(--code-bg) !important;
|
||||||
|
color: var(--code-fg) !important;
|
||||||
|
padding: 0.85rem 1rem;
|
||||||
|
border-radius: 10px;
|
||||||
|
overflow: auto;
|
||||||
|
font-size: 0.78rem;
|
||||||
|
line-height: 1.4;
|
||||||
|
page-break-inside: avoid;
|
||||||
|
}
|
||||||
|
pre code { background: transparent; color: inherit; padding: 0; }
|
||||||
|
#title-block-header, header#title-block-header, h1.title { display: none; }
|
||||||
|
.doc-banner + h1 { display: none; }
|
||||||
|
table {
|
||||||
|
border-collapse: collapse;
|
||||||
|
width: 100%;
|
||||||
|
margin: 0.8rem 0 1.2rem;
|
||||||
|
font-size: 0.9rem;
|
||||||
|
page-break-inside: avoid;
|
||||||
|
}
|
||||||
|
th, td { border: 1px solid var(--line); padding: 0.38rem 0.55rem; text-align: left; vertical-align: top; }
|
||||||
|
th {
|
||||||
|
background: var(--accent);
|
||||||
|
color: #fff;
|
||||||
|
font: 650 12px system-ui, sans-serif;
|
||||||
|
}
|
||||||
|
tr:nth-child(even) td { background: var(--accent-soft); }
|
||||||
|
blockquote {
|
||||||
|
margin: 1rem 0;
|
||||||
|
padding: 0.4rem 0.9rem;
|
||||||
|
border-left: 4px solid var(--accent);
|
||||||
|
background: var(--accent-soft);
|
||||||
|
color: var(--accent-deep);
|
||||||
|
}
|
||||||
|
img { max-width: 100%; height: auto; border-radius: 8px; page-break-inside: avoid; }
|
||||||
|
hr { border: 0; border-top: 1px solid var(--line); }
|
||||||
|
ul, ol { padding-left: 1.25rem; }
|
||||||
|
nav.site { font: 13px system-ui, sans-serif; margin-bottom: 0.4rem; }
|
||||||
|
.source-path { font: 11px ui-monospace, Menlo, monospace; color: var(--muted); }
|
||||||
|
|
||||||
|
@page {
|
||||||
|
size: letter;
|
||||||
|
margin: 0.65in 0.7in 0.8in 0.7in;
|
||||||
|
@top-left {
|
||||||
|
content: "Verae Time × Zapier";
|
||||||
|
font: 700 8pt system-ui, sans-serif;
|
||||||
|
color: #4f46e5;
|
||||||
|
}
|
||||||
|
@top-right {
|
||||||
|
content: "zapier.georgelambert.org";
|
||||||
|
font: 8pt system-ui, sans-serif;
|
||||||
|
color: #6b7186;
|
||||||
|
}
|
||||||
|
@bottom-center {
|
||||||
|
content: counter(page) " / " counter(pages);
|
||||||
|
font: 8pt system-ui, sans-serif;
|
||||||
|
color: #6b7186;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
@media print {
|
||||||
|
html, body { background: #fff; max-width: none; padding: 0; }
|
||||||
|
.doc-banner { margin: 0 0 1rem; border-radius: 8px; -webkit-print-color-adjust: exact; print-color-adjust: exact; }
|
||||||
|
a { text-decoration: none; }
|
||||||
|
th, tr:nth-child(even) td, pre, blockquote, code { -webkit-print-color-adjust: exact; print-color-adjust: exact; }
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="doc-banner"><nav class="site"><a href="/">zapier.georgelambert.org</a></nav><div class="kicker">Verae Time × Zapier · progress report</div><h1>NATS optimal configuration study</h1><div class="source-path">packages/zapier-decisions/reports/optimal-config/REPORT.md</div></div>
|
||||||
|
<header id="title-block-header">
|
||||||
|
<h1 class="title">NATS optimal configuration study</h1>
|
||||||
|
</header>
|
||||||
|
<p><strong>Progress report — optimal configuration study</strong> ·
|
||||||
|
<code>20260912T055851Z</code> (UTC) · all code on
|
||||||
|
<strong>NS1.GEORGELAMBERT.ORG</strong> (<code>70.88.205.138</code>)</p>
|
||||||
|
<p>This document folds every ladder we have run (1-core ZFS,
|
||||||
|
NS1-orchestrated, tmpfs maximize, and this exhaustive 8c/16G
|
||||||
|
<strong>ZFS</strong> factorial) plus UDP / MQTT / reconnect probes. It
|
||||||
|
recommends a lab config and a <strong>three-box HP DL360 Gen10</strong>
|
||||||
|
projection. veth/10G was not changed.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="verdict-read-this-first">1. Verdict (read this first)</h2>
|
||||||
|
<p><strong>Keep NATS + JetStream.</strong> Do not replace the fabric
|
||||||
|
with MQTT, UDP, or a custom persistent-socket protocol for Verae
|
||||||
|
jobs/events/archive. Those are either slower, less durable, or already
|
||||||
|
what NATS is.</p>
|
||||||
|
<p><strong>Lab (NS1, one host, three LXC) — optimal now</strong></p>
|
||||||
|
<table>
|
||||||
|
<colgroup>
|
||||||
|
<col style="width: 25%" />
|
||||||
|
<col style="width: 28%" />
|
||||||
|
<col style="width: 31%" />
|
||||||
|
<col style="width: 15%" />
|
||||||
|
</colgroup>
|
||||||
|
<thead>
|
||||||
|
<tr class="header">
|
||||||
|
<th>Stream</th>
|
||||||
|
<th>Storage</th>
|
||||||
|
<th>Replicas</th>
|
||||||
|
<th>Why</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>ZAPIER_JOBS</code>, <code>ZAPIER_WEBHOOKS</code>,
|
||||||
|
<code>VERAE_ARCHIVE</code></td>
|
||||||
|
<td><strong>file</strong> (ZFS)</td>
|
||||||
|
<td><strong>3</strong></td>
|
||||||
|
<td>Survive a nats LXC death; archive must persist</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><code>ZAPIER_EVENTS</code></td>
|
||||||
|
<td><strong>memory</strong></td>
|
||||||
|
<td><strong>3</strong></td>
|
||||||
|
<td>Waiters are latency-sensitive; events rebuild from job status</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>ZAPIER_USAGE</code></td>
|
||||||
|
<td>file</td>
|
||||||
|
<td>3</td>
|
||||||
|
<td>Telemetry, limits + max-age</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>Keep <strong>8 cores / 16 GiB / <code>max_mem: 8G</code></strong> on
|
||||||
|
510–513 (already live). Do <strong>not</strong> leave JetStream on
|
||||||
|
tmpfs. Do <strong>not</strong> drop product streams to r=1. Reuse
|
||||||
|
<strong>one NATS connection per process</strong> (already true in
|
||||||
|
middleware); never connect-per-message.</p>
|
||||||
|
<p><strong>Metal (3× DL360 Gen10) — optimal later</strong></p>
|
||||||
|
<p>Same stream table. File store on <strong>local NVMe/M.2</strong>, not
|
||||||
|
a shared SAN. Cluster + client on <strong>10GbE</strong> (or 25GbE if
|
||||||
|
you already have it). Dual Gold Xeon is surplus CPU for this workload;
|
||||||
|
8–16 cores dedicated to <code>nats-server</code> is enough. Expected JS
|
||||||
|
file r=3: <strong>~40–80k</strong> 128 B pubs/s (about
|
||||||
|
<strong>3–6×</strong> this lab’s 8c ZFS 1p, <strong>2–4×</strong> tmpfs
|
||||||
|
1p) — bounded by <strong>10GbE replica RTT</strong>, not by Xeon clocks.
|
||||||
|
Core NATS will sit in the <strong>1–3M msgs/s</strong> band until the
|
||||||
|
NIC saturates (~9 Gbit/s ≈ 8–9M × 128 B theoretical; CPU and client will
|
||||||
|
hit first).</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="what-we-actually-ran-this-exhaustive-pass">2. What we actually
|
||||||
|
ran (this exhaustive pass)</h2>
|
||||||
|
<p>Live cluster during this run: LXC 510–513 <strong>8 cores / 16
|
||||||
|
GiB</strong>, JetStream <strong>on ZFS</strong> (tmpfs from the maximize
|
||||||
|
study was already unmounted). Extra factorial: file/memory × replicas
|
||||||
|
1/3, 4 KiB file r=3, reconnect-per-message ping, UDP echo 510→511, MQTT
|
||||||
|
QoS0 against nats-a <code>:1883</code>. Product streams were not the
|
||||||
|
bench target.</p>
|
||||||
|
<h3 id="cross-study-history">2.1 Cross-study history</h3>
|
||||||
|
<table style="width:100%;">
|
||||||
|
<colgroup>
|
||||||
|
<col style="width: 16%" />
|
||||||
|
<col style="width: 16%" />
|
||||||
|
<col style="width: 16%" />
|
||||||
|
<col style="width: 16%" />
|
||||||
|
<col style="width: 16%" />
|
||||||
|
<col style="width: 16%" />
|
||||||
|
</colgroup>
|
||||||
|
<thead>
|
||||||
|
<tr class="header">
|
||||||
|
<th>Study</th>
|
||||||
|
<th>Env</th>
|
||||||
|
<th>Core 1p pub</th>
|
||||||
|
<th>JS file r=3 1p</th>
|
||||||
|
<th>JS mem r=3 4p</th>
|
||||||
|
<th>Ping p99</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>20260912T051237Z</code></td>
|
||||||
|
<td>1c/1G ZFS (NS1 orch.)</td>
|
||||||
|
<td>502,502</td>
|
||||||
|
<td>7,393</td>
|
||||||
|
<td>—</td>
|
||||||
|
<td>1.377ms</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><code>20260912T053120Z</code></td>
|
||||||
|
<td>8c/16G tmpfs + mem extra</td>
|
||||||
|
<td>599,004</td>
|
||||||
|
<td>17,388</td>
|
||||||
|
<td>36,355</td>
|
||||||
|
<td>0.684ms</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>20260912T055851Z</code></td>
|
||||||
|
<td>8c/16G ZFS exhaustive <code>20260912T055851Z</code></td>
|
||||||
|
<td>662,227</td>
|
||||||
|
<td>14,330</td>
|
||||||
|
<td>37,736</td>
|
||||||
|
<td>1.140ms</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<figure>
|
||||||
|
<img src="charts-optimal/history-js1p.png"
|
||||||
|
alt="JS 1p file r=3 history" />
|
||||||
|
<figcaption aria-hidden="true">JS 1p file r=3 history</figcaption>
|
||||||
|
</figure>
|
||||||
|
<h3 id="this-run-jetstream-factorial">2.2 This run — JetStream
|
||||||
|
factorial</h3>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr class="header">
|
||||||
|
<th>Run</th>
|
||||||
|
<th>What</th>
|
||||||
|
<th>Pub msgs/s</th>
|
||||||
|
<th>Pub MB/s</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>js-file-1p-20k-128-r1</code></td>
|
||||||
|
<td>file r=1 1p 128 B</td>
|
||||||
|
<td>18,888</td>
|
||||||
|
<td>2.31</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><code>js-file-4p-50k-128-r1</code></td>
|
||||||
|
<td>file r=1 4p 128 B</td>
|
||||||
|
<td>24,560</td>
|
||||||
|
<td>3.00</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>js-1p-20k-128-r3</code></td>
|
||||||
|
<td>file r=3 1p 128 B</td>
|
||||||
|
<td>14,330</td>
|
||||||
|
<td>1.75</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><code>js-4p-50k-128-r3</code></td>
|
||||||
|
<td>file r=3 4p 128 B</td>
|
||||||
|
<td>19,232</td>
|
||||||
|
<td>2.35</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>js-4p-20k-1k-r3</code></td>
|
||||||
|
<td>file r=3 4p 1 KiB</td>
|
||||||
|
<td>15,197</td>
|
||||||
|
<td>14.84</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><code>js-file-1p-20k-4k-r3</code></td>
|
||||||
|
<td>file r=3 1p 4 KiB</td>
|
||||||
|
<td>8,673</td>
|
||||||
|
<td>33.88</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>js-mem-1p-20k-128-r1</code></td>
|
||||||
|
<td>memory r=1 1p 128 B</td>
|
||||||
|
<td>29,972</td>
|
||||||
|
<td>3.66</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><code>js-mem-4p-50k-128-r1</code></td>
|
||||||
|
<td>memory r=1 4p 128 B</td>
|
||||||
|
<td>64,923</td>
|
||||||
|
<td>7.93</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>js-mem-1p-20k-128-r3</code></td>
|
||||||
|
<td>memory r=3 1p 128 B</td>
|
||||||
|
<td>20,188</td>
|
||||||
|
<td>2.46</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><code>js-mem-4p-50k-128-r3</code></td>
|
||||||
|
<td>memory r=3 4p 128 B</td>
|
||||||
|
<td>37,736</td>
|
||||||
|
<td>4.61</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><code>js-mem-4p-20k-1k-r3</code></td>
|
||||||
|
<td>memory r=3 4p 1 KiB</td>
|
||||||
|
<td>33,916</td>
|
||||||
|
<td>33.12</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>Replica <strong>1 vs 3</strong> on this stand (file 1p 128 B): r=1 is
|
||||||
|
18,888 vs r=3 14,330 (1.32× if r=3 is the slower one). Memory r=1 1p
|
||||||
|
29,972 vs memory r=3 20,188.</p>
|
||||||
|
<figure>
|
||||||
|
<img src="charts-optimal/replicas.png" alt="Replica cost" />
|
||||||
|
<figcaption aria-hidden="true">Replica cost</figcaption>
|
||||||
|
</figure>
|
||||||
|
<h3 id="delay-reconnect-tax-udp-mqtt">2.3 Delay, reconnect tax, UDP,
|
||||||
|
MQTT</h3>
|
||||||
|
<table>
|
||||||
|
<colgroup>
|
||||||
|
<col style="width: 29%" />
|
||||||
|
<col style="width: 33%" />
|
||||||
|
<col style="width: 37%" />
|
||||||
|
</colgroup>
|
||||||
|
<thead>
|
||||||
|
<tr class="header">
|
||||||
|
<th>Probe</th>
|
||||||
|
<th>Result</th>
|
||||||
|
<th>Meaning</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>NATS ping (persistent sockets) p50 / p99</td>
|
||||||
|
<td>0.456ms / 1.140ms</td>
|
||||||
|
<td>Quiet hop with a long-lived TCP conn</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>NATS <strong>reconnect-per-message</strong> p50 / p99</td>
|
||||||
|
<td>0.503ms / 1.750ms</td>
|
||||||
|
<td>TCP+NATS handshake on every pub — this is the tax to avoid</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>UDP echo 510→511 p99</td>
|
||||||
|
<td>0.363ms</td>
|
||||||
|
<td>Raw datagram ceiling on the same veth (no NATS)</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>MQTT QoS0 5k×128 B</td>
|
||||||
|
<td>44862 pubs/s</td>
|
||||||
|
<td>nats-server MQTT gateway on <code>:1883</code></td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>Core 1p1s 128 B this run: 662,227 pub msgs/s. Flood delay is still
|
||||||
|
backlog/consume_rate, not RTT.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="alternative-transports-why-we-are-not-switching-the-fabric">3.
|
||||||
|
Alternative transports (why we are not switching the fabric)</h2>
|
||||||
|
<p>NATS already <strong>is</strong> persistent TCP sockets with a tiny
|
||||||
|
binary protocol, automatic reconnect, and optional JetStream durability.
|
||||||
|
“Reduce connection overhead” is a <strong>client</strong> discipline:
|
||||||
|
hold the connection. The reconnect probe exists to prove that opening a
|
||||||
|
socket per job would dominate ping RTT.</p>
|
||||||
|
<table>
|
||||||
|
<colgroup>
|
||||||
|
<col style="width: 7%" />
|
||||||
|
<col style="width: 44%" />
|
||||||
|
<col style="width: 32%" />
|
||||||
|
<col style="width: 15%" />
|
||||||
|
</colgroup>
|
||||||
|
<thead>
|
||||||
|
<tr class="header">
|
||||||
|
<th>Idea</th>
|
||||||
|
<th>Fit for Verae jobs/events/archive</th>
|
||||||
|
<th>Throughput vs NATS core</th>
|
||||||
|
<th>Durability</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><strong>NATS core pub/sub</strong></td>
|
||||||
|
<td>Fan-out, request-reply (<code>verae.billing.*</code>)</td>
|
||||||
|
<td>Highest we measured (~0.5–2M msgs/s)</td>
|
||||||
|
<td>None</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><strong>NATS JetStream file r=3</strong></td>
|
||||||
|
<td>Jobs, webhooks, archive</td>
|
||||||
|
<td>~8–23k on this lab; see metal projection</td>
|
||||||
|
<td>Disk + 1-node loss</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><strong>NATS JetStream memory r=3</strong></td>
|
||||||
|
<td>Events mailbox</td>
|
||||||
|
<td>~22–36k on this lab</td>
|
||||||
|
<td>RAM + 1-node loss; <strong>empty on full restart</strong></td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><strong>MQTT</strong> (NATS gateway or Mosquitto)</td>
|
||||||
|
<td>IoT endpoints that already speak MQTT</td>
|
||||||
|
<td>This probe: 44862 pubs/s QoS0 — typically <strong>well
|
||||||
|
below</strong> NATS core; QoS1 ≈ JetStream-ish with more chatter</td>
|
||||||
|
<td>QoS1/2 session state; not our WORM model</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><strong>UDP</strong></td>
|
||||||
|
<td>Telemetry that may drop</td>
|
||||||
|
<td>RTT 0.363ms p99 — fastest hop, <strong>no</strong> reliability, no
|
||||||
|
cluster, no auth</td>
|
||||||
|
<td>None</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><strong>Custom persistent sockets / HTTP long-poll</strong></td>
|
||||||
|
<td>Worse NATS</td>
|
||||||
|
<td>You would re-implement reconnect, flow control, and fan-out</td>
|
||||||
|
<td>DIY</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><strong>WebSocket</strong></td>
|
||||||
|
<td>Browsers only</td>
|
||||||
|
<td>Extra framing; NATS already has WS for UIs, not for middleware</td>
|
||||||
|
<td>Same as core/JS behind it</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td><strong>QUIC / WebTransport</strong></td>
|
||||||
|
<td>Lossy WAN / browsers</td>
|
||||||
|
<td>NATS QUIC is not the lab path; 10GbE LAN does not need it</td>
|
||||||
|
<td>Same</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td><strong>Kafka / Redis streams</strong></td>
|
||||||
|
<td>Heavy log replay</td>
|
||||||
|
<td>Higher ops cost; not on <code>vmbr1</code> today</td>
|
||||||
|
<td>Yes, heavier</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p><strong>MQTT:</strong> NATS documents MQTT as an <em>enabling</em>
|
||||||
|
gateway for existing IoT, and prefers NATS end-to-end for greenfield.
|
||||||
|
Zapier cloud never talks NATS or MQTT; it talks HTTPS. Putting MQTT in
|
||||||
|
the middle of timestamp jobs adds protocol translation and QoS timers
|
||||||
|
without helping <code>jobId → events</code>. Use MQTT only if a device
|
||||||
|
already cannot speak NATS.</p>
|
||||||
|
<p><strong>UDP:</strong> Fine as a <em>measurement</em> of veth RTT.
|
||||||
|
Unusable as the job fabric (no ack, no replica, no flow control). NATS
|
||||||
|
ping is already within a small multiple of UDP on this bridge.</p>
|
||||||
|
<p><strong>Persistence sockets:</strong> Middleware and keep already
|
||||||
|
keep <code>NATS_URL</code> connections open. Optimal: one connection (or
|
||||||
|
a small pool) per process, <code>max_reconnect</code>, jitter, no
|
||||||
|
<code>connect()</code> in the per-job path. The reconnect ladder is the
|
||||||
|
anti-pattern.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="optimal-configurations">4. Optimal configurations</h2>
|
||||||
|
<h3 id="ns1-lab-now">4.1 NS1 lab (now)</h3>
|
||||||
|
<ol type="1">
|
||||||
|
<li><strong>Leave 8 cores / 16 GiB</strong> on nats-a/b/c and the
|
||||||
|
worker. Host has 40 cores / 377 GiB; this is cheap.</li>
|
||||||
|
<li><strong><code>max_mem: 8G</code></strong> stays. Required for memory
|
||||||
|
streams.</li>
|
||||||
|
<li><strong>File r=3 on ZFS</strong> for jobs/webhooks/archive. tmpfs
|
||||||
|
doubled JS 1p (7.4k→17k) but <strong>loses the stream on reboot</strong>
|
||||||
|
— unacceptable for archive.</li>
|
||||||
|
<li><strong>Memory r=3 for <code>ZAPIER_EVENTS</code></strong> if we
|
||||||
|
accept “all three nats CTs reboot ⇒ in-flight waiters fall back to HTTP
|
||||||
|
poll.” That matches the designed wait path
|
||||||
|
(<code>GET /api/status/{jobId}</code>).</li>
|
||||||
|
<li><strong>r=1 only for throwaway benches</strong>, never product
|
||||||
|
streams. Replica=3 is the point of three guests.</li>
|
||||||
|
<li><strong>veth on vmbr1, no fake 10G NICs.</strong> Already 10000Mb/s;
|
||||||
|
JS does not fill it.</li>
|
||||||
|
<li><strong>Pin cpusets</strong> later if keep/fleet steal; not required
|
||||||
|
to beat these numbers.</li>
|
||||||
|
<li>Clients: persistent NATS connections; pull consumers with bounded
|
||||||
|
<code>max_ack_pending</code> for webhooks.</li>
|
||||||
|
</ol>
|
||||||
|
<h3 id="three-hp-dl360-gen10-projection-not-measured">4.2 Three HP DL360
|
||||||
|
Gen10 (projection — not measured)</h3>
|
||||||
|
<p>Assumed bill of materials (state it in the buy):</p>
|
||||||
|
<table>
|
||||||
|
<colgroup>
|
||||||
|
<col style="width: 36%" />
|
||||||
|
<col style="width: 63%" />
|
||||||
|
</colgroup>
|
||||||
|
<thead>
|
||||||
|
<tr class="header">
|
||||||
|
<th>Piece</th>
|
||||||
|
<th>Assumption</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>Chassis</td>
|
||||||
|
<td>3× DL360 Gen10 1U</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>CPU</td>
|
||||||
|
<td>Dual 2nd-gen Xeon <strong>Gold</strong> (e.g. 6226R 16c or 6248 20c
|
||||||
|
— <strong>32–40 cores/box</strong>)</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>Memory</td>
|
||||||
|
<td>DDR4-2933, <strong>192–384 GiB</strong>/box (6–12×32 GiB); NATS will
|
||||||
|
not use most of it</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>Storage</td>
|
||||||
|
<td><strong>NVMe M.2 or U.2</strong> for
|
||||||
|
<code>/var/lib/nats/jetstream</code> (XFS or ext4, <strong>not</strong>
|
||||||
|
shared ZFS over the network). RAID1 of two NVMe if you want disk HA
|
||||||
|
<em>inside</em> a box</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>Network</td>
|
||||||
|
<td><strong>10GbE</strong> (FlexibleLOM or PCIe); dedicated VLAN for
|
||||||
|
<code>:4222</code>+<code>:6222</code>. Do not share with public
|
||||||
|
<code>vmbr0</code> traffic</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>OS</td>
|
||||||
|
<td>Debian/Ubuntu bare metal, <code>nats-server</code> systemd, same
|
||||||
|
<code>nats.conf</code> as lab (bind private IP only)</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p><strong>What changes vs NS1 LXC</strong></p>
|
||||||
|
<table>
|
||||||
|
<colgroup>
|
||||||
|
<col style="width: 15%" />
|
||||||
|
<col style="width: 20%" />
|
||||||
|
<col style="width: 18%" />
|
||||||
|
<col style="width: 45%" />
|
||||||
|
</colgroup>
|
||||||
|
<thead>
|
||||||
|
<tr class="header">
|
||||||
|
<th>Factor</th>
|
||||||
|
<th>NS1 today</th>
|
||||||
|
<th>3× DL360</th>
|
||||||
|
<th>Effect on JS file r=3</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>Failure domain</td>
|
||||||
|
<td>1 Proxmox host</td>
|
||||||
|
<td>3 chassis, 3 NVMe, 3 NICs</td>
|
||||||
|
<td>r=3 <strong>means</strong> something</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>Disk</td>
|
||||||
|
<td>Shared ZFS SSD2</td>
|
||||||
|
<td>Local NVMe fsync ~50–150 µs</td>
|
||||||
|
<td>Big win vs ZFS; similar to tmpfs for sequential 128 B</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>Replica path</td>
|
||||||
|
<td>veth/bridge (~µs–tens of µs)</td>
|
||||||
|
<td>10GbE RTT typically <strong>50–200 µs</strong></td>
|
||||||
|
<td><strong>Slower than same-host tmpfs</strong>, faster than a bad
|
||||||
|
SAN</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>CPU</td>
|
||||||
|
<td>8 of 40 shared</td>
|
||||||
|
<td>32–40 dedicated Gold cores</td>
|
||||||
|
<td>Headroom for many clients, not 10× JS</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>NIC</td>
|
||||||
|
<td>software 10G veth, already ~5 Gbit/s core</td>
|
||||||
|
<td>real 10GbE ~9 Gbit/s TCP</td>
|
||||||
|
<td>Core NATS can grow; JS r=3 stays replica-bound</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p><strong>Projected bands</strong> (128 B, 3-node cluster, dedicated
|
||||||
|
10GbE, local NVMe, 8+ cores pinned to nats-server):</p>
|
||||||
|
<table>
|
||||||
|
<colgroup>
|
||||||
|
<col style="width: 16%" />
|
||||||
|
<col style="width: 34%" />
|
||||||
|
<col style="width: 29%" />
|
||||||
|
<col style="width: 19%" />
|
||||||
|
</colgroup>
|
||||||
|
<thead>
|
||||||
|
<tr class="header">
|
||||||
|
<th>Workload</th>
|
||||||
|
<th>NS1 measured (best)</th>
|
||||||
|
<th>DL360 projection</th>
|
||||||
|
<th>Confidence</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>Core pub/sub 1p</td>
|
||||||
|
<td>0.5–0.8M</td>
|
||||||
|
<td><strong>0.8–2M</strong></td>
|
||||||
|
<td>Medium — NIC + syscall, plenty of CPU</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>Core 4p4s 1 KiB</td>
|
||||||
|
<td>~0.6–0.7M (~0.6 GB/s)</td>
|
||||||
|
<td><strong>~1M msgs/s / ~1 GB/s</strong> approaching 10GbE</td>
|
||||||
|
<td>Medium</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>JS file r=1</td>
|
||||||
|
<td>this run r=1</td>
|
||||||
|
<td><strong>80–200k</strong> pubs/s</td>
|
||||||
|
<td>Medium — NVMe + no replica wait</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>JS file r=3</td>
|
||||||
|
<td>7–23k (ZFS/tmpfs)</td>
|
||||||
|
<td><strong>40–80k</strong> pubs/s</td>
|
||||||
|
<td>Medium-low — replica RTT dominates; 3 NVMe still help vs shared
|
||||||
|
ZFS</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="odd">
|
||||||
|
<td>JS memory r=3</td>
|
||||||
|
<td>22–36k</td>
|
||||||
|
<td><strong>50–100k</strong></td>
|
||||||
|
<td>Medium-low — RAM + 10GbE ack</td>
|
||||||
|
</tr>
|
||||||
|
<tr class="even">
|
||||||
|
<td>Ping p99</td>
|
||||||
|
<td>0.7–1.4 ms</td>
|
||||||
|
<td><strong>0.2–0.6 ms</strong></td>
|
||||||
|
<td>Medium — real NIC but no Proxmox tax</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>These are <strong>not</strong> DL360 measurements. Scale from: (a)
|
||||||
|
our replica-1 vs replica-3 ratio once this run’s r=1 numbers exist, (b)
|
||||||
|
tmpfs vs ZFS ratio (2.35× on 1p), (c) Synadia/nats bench async file r=1
|
||||||
|
~100–400k on NVMe loopback, derated for 10GbE RTT.</p>
|
||||||
|
<p><strong>Buy notes:</strong> M.2 via Dual uFF / enablement kit; put
|
||||||
|
JetStream on NVMe <strong>directly</strong>, not behind a RAID
|
||||||
|
controller write-through unless you measure. 1GbE onboard is a trap —
|
||||||
|
use 10GbE for <code>:6222</code>. Dual Gold is for isolation (nats vs
|
||||||
|
worm/tree vs OS), not because JS needs 56 cores.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="what-we-are-not-doing">5. What we are not doing</h2>
|
||||||
|
<ul>
|
||||||
|
<li>MQTT as the Zapier or middleware transport.</li>
|
||||||
|
<li>UDP for jobs.</li>
|
||||||
|
<li>Emulated 10G fiber NICs on LXC.</li>
|
||||||
|
<li>tmpfs as the production store.</li>
|
||||||
|
<li>r=1 for product streams.</li>
|
||||||
|
<li>Connect-per-job.</li>
|
||||||
|
</ul>
|
||||||
|
<p>Re-run exhaustive: <code>bash scripts/exhaustive-ns1-study.sh</code>
|
||||||
|
on NS1.</p>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
160
packages/zapier-decisions/reports/optimal-config/REPORT.md
Normal file
160
packages/zapier-decisions/reports/optimal-config/REPORT.md
Normal file
|
|
@ -0,0 +1,160 @@
|
||||||
|
**Progress report — optimal configuration study** · `20260912T055851Z` (UTC) · all code on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`)
|
||||||
|
|
||||||
|
This document folds every ladder we have run (1-core ZFS, NS1-orchestrated, tmpfs maximize, and this exhaustive 8c/16G **ZFS** factorial) plus UDP / MQTT / reconnect probes. It recommends a lab config and a **three-box HP DL360 Gen10** projection. veth/10G was not changed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Verdict (read this first)
|
||||||
|
|
||||||
|
**Keep NATS + JetStream.** Do not replace the fabric with MQTT, UDP, or a custom persistent-socket protocol for Verae jobs/events/archive. Those are either slower, less durable, or already what NATS is.
|
||||||
|
|
||||||
|
**Lab (NS1, one host, three LXC) — optimal now**
|
||||||
|
|
||||||
|
| Stream | Storage | Replicas | Why |
|
||||||
|
|--------|---------|----------|-----|
|
||||||
|
| `ZAPIER_JOBS`, `ZAPIER_WEBHOOKS`, `VERAE_ARCHIVE` | **file** (ZFS) | **3** | Survive a nats LXC death; archive must persist |
|
||||||
|
| `ZAPIER_EVENTS` | **memory** | **3** | Waiters are latency-sensitive; events rebuild from job status |
|
||||||
|
| `ZAPIER_USAGE` | file | 3 | Telemetry, limits + max-age |
|
||||||
|
|
||||||
|
Keep **8 cores / 16 GiB / `max_mem: 8G`** on 510–513 (already live). Do **not** leave JetStream on tmpfs. Do **not** drop product streams to r=1. Reuse **one NATS connection per process** (already true in middleware); never connect-per-message.
|
||||||
|
|
||||||
|
**Metal (3× DL360 Gen10) — optimal later**
|
||||||
|
|
||||||
|
Same stream table. File store on **local NVMe/M.2**, not a shared SAN. Cluster + client on **10GbE** (or 25GbE if you already have it). Dual Gold Xeon is surplus CPU for this workload; 8–16 cores dedicated to `nats-server` is enough. Expected JS file r=3: **~40–80k** 128 B pubs/s (about **3–6×** this lab’s 8c ZFS 1p, **2–4×** tmpfs 1p) — bounded by **10GbE replica RTT**, not by Xeon clocks. Core NATS will sit in the **1–3M msgs/s** band until the NIC saturates (~9 Gbit/s ≈ 8–9M × 128 B theoretical; CPU and client will hit first).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. What we actually ran (this exhaustive pass)
|
||||||
|
|
||||||
|
Live cluster during this run: LXC 510–513 **8 cores / 16 GiB**, JetStream **on ZFS** (tmpfs from the maximize study was already unmounted). Extra factorial: file/memory × replicas 1/3, 4 KiB file r=3, reconnect-per-message ping, UDP echo 510→511, MQTT QoS0 against nats-a `:1883`. Product streams were not the bench target.
|
||||||
|
|
||||||
|
### 2.1 Cross-study history
|
||||||
|
|
||||||
|
| Study | Env | Core 1p pub | JS file r=3 1p | JS mem r=3 4p | Ping p99 |
|
||||||
|
| --- | --- | --- | --- | --- | --- |
|
||||||
|
| `20260912T051237Z` | 1c/1G ZFS (NS1 orch.) | 502,502 | 7,393 | — | 1.377ms |
|
||||||
|
| `20260912T053120Z` | 8c/16G tmpfs + mem extra | 599,004 | 17,388 | 36,355 | 0.684ms |
|
||||||
|
| `20260912T055851Z` | 8c/16G ZFS exhaustive `20260912T055851Z` | 662,227 | 14,330 | 37,736 | 1.140ms |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### 2.2 This run — JetStream factorial
|
||||||
|
|
||||||
|
| Run | What | Pub msgs/s | Pub MB/s |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `js-file-1p-20k-128-r1` | file r=1 1p 128 B | 18,888 | 2.31 |
|
||||||
|
| `js-file-4p-50k-128-r1` | file r=1 4p 128 B | 24,560 | 3.00 |
|
||||||
|
| `js-1p-20k-128-r3` | file r=3 1p 128 B | 14,330 | 1.75 |
|
||||||
|
| `js-4p-50k-128-r3` | file r=3 4p 128 B | 19,232 | 2.35 |
|
||||||
|
| `js-4p-20k-1k-r3` | file r=3 4p 1 KiB | 15,197 | 14.84 |
|
||||||
|
| `js-file-1p-20k-4k-r3` | file r=3 1p 4 KiB | 8,673 | 33.88 |
|
||||||
|
| `js-mem-1p-20k-128-r1` | memory r=1 1p 128 B | 29,972 | 3.66 |
|
||||||
|
| `js-mem-4p-50k-128-r1` | memory r=1 4p 128 B | 64,923 | 7.93 |
|
||||||
|
| `js-mem-1p-20k-128-r3` | memory r=3 1p 128 B | 20,188 | 2.46 |
|
||||||
|
| `js-mem-4p-50k-128-r3` | memory r=3 4p 128 B | 37,736 | 4.61 |
|
||||||
|
| `js-mem-4p-20k-1k-r3` | memory r=3 4p 1 KiB | 33,916 | 33.12 |
|
||||||
|
|
||||||
|
Replica **1 vs 3** on this stand (file 1p 128 B): r=1 is 18,888 vs r=3 14,330 (1.32× if r=3 is the slower one). Memory r=1 1p 29,972 vs memory r=3 20,188.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### 2.3 Delay, reconnect tax, UDP, MQTT
|
||||||
|
|
||||||
|
| Probe | Result | Meaning |
|
||||||
|
|-------|--------|---------|
|
||||||
|
| NATS ping (persistent sockets) p50 / p99 | 0.456ms / 1.140ms | Quiet hop with a long-lived TCP conn |
|
||||||
|
| NATS **reconnect-per-message** p50 / p99 | 0.503ms / 1.750ms | TCP+NATS handshake on every pub — this is the tax to avoid |
|
||||||
|
| UDP echo 510→511 p99 | 0.363ms | Raw datagram ceiling on the same veth (no NATS) |
|
||||||
|
| MQTT QoS0 5k×128 B | 44862 pubs/s | nats-server MQTT gateway on `:1883` |
|
||||||
|
|
||||||
|
Core 1p1s 128 B this run: 662,227 pub msgs/s. Flood delay is still backlog/consume_rate, not RTT.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Alternative transports (why we are not switching the fabric)
|
||||||
|
|
||||||
|
NATS already **is** persistent TCP sockets with a tiny binary protocol, automatic reconnect, and optional JetStream durability. “Reduce connection overhead” is a **client** discipline: hold the connection. The reconnect probe exists to prove that opening a socket per job would dominate ping RTT.
|
||||||
|
|
||||||
|
| Idea | Fit for Verae jobs/events/archive | Throughput vs NATS core | Durability |
|
||||||
|
|------|-----------------------------------|-------------------------|------------|
|
||||||
|
| **NATS core pub/sub** | Fan-out, request-reply (`verae.billing.*`) | Highest we measured (~0.5–2M msgs/s) | None |
|
||||||
|
| **NATS JetStream file r=3** | Jobs, webhooks, archive | ~8–23k on this lab; see metal projection | Disk + 1-node loss |
|
||||||
|
| **NATS JetStream memory r=3** | Events mailbox | ~22–36k on this lab | RAM + 1-node loss; **empty on full restart** |
|
||||||
|
| **MQTT** (NATS gateway or Mosquitto) | IoT endpoints that already speak MQTT | This probe: 44862 pubs/s QoS0 — typically **well below** NATS core; QoS1 ≈ JetStream-ish with more chatter | QoS1/2 session state; not our WORM model |
|
||||||
|
| **UDP** | Telemetry that may drop | RTT 0.363ms p99 — fastest hop, **no** reliability, no cluster, no auth | None |
|
||||||
|
| **Custom persistent sockets / HTTP long-poll** | Worse NATS | You would re-implement reconnect, flow control, and fan-out | DIY |
|
||||||
|
| **WebSocket** | Browsers only | Extra framing; NATS already has WS for UIs, not for middleware | Same as core/JS behind it |
|
||||||
|
| **QUIC / WebTransport** | Lossy WAN / browsers | NATS QUIC is not the lab path; 10GbE LAN does not need it | Same |
|
||||||
|
| **Kafka / Redis streams** | Heavy log replay | Higher ops cost; not on `vmbr1` today | Yes, heavier |
|
||||||
|
|
||||||
|
**MQTT:** NATS documents MQTT as an *enabling* gateway for existing IoT, and prefers NATS end-to-end for greenfield. Zapier cloud never talks NATS or MQTT; it talks HTTPS. Putting MQTT in the middle of timestamp jobs adds protocol translation and QoS timers without helping `jobId → events`. Use MQTT only if a device already cannot speak NATS.
|
||||||
|
|
||||||
|
**UDP:** Fine as a *measurement* of veth RTT. Unusable as the job fabric (no ack, no replica, no flow control). NATS ping is already within a small multiple of UDP on this bridge.
|
||||||
|
|
||||||
|
**Persistence sockets:** Middleware and keep already keep `NATS_URL` connections open. Optimal: one connection (or a small pool) per process, `max_reconnect`, jitter, no `connect()` in the per-job path. The reconnect ladder is the anti-pattern.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Optimal configurations
|
||||||
|
|
||||||
|
### 4.1 NS1 lab (now)
|
||||||
|
|
||||||
|
1. **Leave 8 cores / 16 GiB** on nats-a/b/c and the worker. Host has 40 cores / 377 GiB; this is cheap.
|
||||||
|
2. **`max_mem: 8G`** stays. Required for memory streams.
|
||||||
|
3. **File r=3 on ZFS** for jobs/webhooks/archive. tmpfs doubled JS 1p (7.4k→17k) but **loses the stream on reboot** — unacceptable for archive.
|
||||||
|
4. **Memory r=3 for `ZAPIER_EVENTS`** if we accept “all three nats CTs reboot ⇒ in-flight waiters fall back to HTTP poll.” That matches the designed wait path (`GET /api/status/{jobId}`).
|
||||||
|
5. **r=1 only for throwaway benches**, never product streams. Replica=3 is the point of three guests.
|
||||||
|
6. **veth on vmbr1, no fake 10G NICs.** Already 10000Mb/s; JS does not fill it.
|
||||||
|
7. **Pin cpusets** later if keep/fleet steal; not required to beat these numbers.
|
||||||
|
8. Clients: persistent NATS connections; pull consumers with bounded `max_ack_pending` for webhooks.
|
||||||
|
|
||||||
|
### 4.2 Three HP DL360 Gen10 (projection — not measured)
|
||||||
|
|
||||||
|
Assumed bill of materials (state it in the buy):
|
||||||
|
|
||||||
|
| Piece | Assumption |
|
||||||
|
|-------|------------|
|
||||||
|
| Chassis | 3× DL360 Gen10 1U |
|
||||||
|
| CPU | Dual 2nd-gen Xeon **Gold** (e.g. 6226R 16c or 6248 20c — **32–40 cores/box**) |
|
||||||
|
| Memory | DDR4-2933, **192–384 GiB**/box (6–12×32 GiB); NATS will not use most of it |
|
||||||
|
| Storage | **NVMe M.2 or U.2** for `/var/lib/nats/jetstream` (XFS or ext4, **not** shared ZFS over the network). RAID1 of two NVMe if you want disk HA *inside* a box |
|
||||||
|
| Network | **10GbE** (FlexibleLOM or PCIe); dedicated VLAN for `:4222`+`:6222`. Do not share with public `vmbr0` traffic |
|
||||||
|
| OS | Debian/Ubuntu bare metal, `nats-server` systemd, same `nats.conf` as lab (bind private IP only) |
|
||||||
|
|
||||||
|
**What changes vs NS1 LXC**
|
||||||
|
|
||||||
|
| Factor | NS1 today | 3× DL360 | Effect on JS file r=3 |
|
||||||
|
|--------|-----------|----------|------------------------|
|
||||||
|
| Failure domain | 1 Proxmox host | 3 chassis, 3 NVMe, 3 NICs | r=3 **means** something |
|
||||||
|
| Disk | Shared ZFS SSD2 | Local NVMe fsync ~50–150 µs | Big win vs ZFS; similar to tmpfs for sequential 128 B |
|
||||||
|
| Replica path | veth/bridge (~µs–tens of µs) | 10GbE RTT typically **50–200 µs** | **Slower than same-host tmpfs**, faster than a bad SAN |
|
||||||
|
| CPU | 8 of 40 shared | 32–40 dedicated Gold cores | Headroom for many clients, not 10× JS |
|
||||||
|
| NIC | software 10G veth, already ~5 Gbit/s core | real 10GbE ~9 Gbit/s TCP | Core NATS can grow; JS r=3 stays replica-bound |
|
||||||
|
|
||||||
|
**Projected bands** (128 B, 3-node cluster, dedicated 10GbE, local NVMe, 8+ cores pinned to nats-server):
|
||||||
|
|
||||||
|
| Workload | NS1 measured (best) | DL360 projection | Confidence |
|
||||||
|
|----------|---------------------|------------------|------------|
|
||||||
|
| Core pub/sub 1p | 0.5–0.8M | **0.8–2M** | Medium — NIC + syscall, plenty of CPU |
|
||||||
|
| Core 4p4s 1 KiB | ~0.6–0.7M (~0.6 GB/s) | **~1M msgs/s / ~1 GB/s** approaching 10GbE | Medium |
|
||||||
|
| JS file r=1 | this run r=1 | **80–200k** pubs/s | Medium — NVMe + no replica wait |
|
||||||
|
| JS file r=3 | 7–23k (ZFS/tmpfs) | **40–80k** pubs/s | Medium-low — replica RTT dominates; 3 NVMe still help vs shared ZFS |
|
||||||
|
| JS memory r=3 | 22–36k | **50–100k** | Medium-low — RAM + 10GbE ack |
|
||||||
|
| Ping p99 | 0.7–1.4 ms | **0.2–0.6 ms** | Medium — real NIC but no Proxmox tax |
|
||||||
|
|
||||||
|
These are **not** DL360 measurements. Scale from: (a) our replica-1 vs replica-3 ratio once this run’s r=1 numbers exist, (b) tmpfs vs ZFS ratio (2.35× on 1p), (c) Synadia/nats bench async file r=1 ~100–400k on NVMe loopback, derated for 10GbE RTT.
|
||||||
|
|
||||||
|
**Buy notes:** M.2 via Dual uFF / enablement kit; put JetStream on NVMe **directly**, not behind a RAID controller write-through unless you measure. 1GbE onboard is a trap — use 10GbE for `:6222`. Dual Gold is for isolation (nats vs worm/tree vs OS), not because JS needs 56 cores.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. What we are not doing
|
||||||
|
|
||||||
|
- MQTT as the Zapier or middleware transport.
|
||||||
|
- UDP for jobs.
|
||||||
|
- Emulated 10G fiber NICs on LXC.
|
||||||
|
- tmpfs as the production store.
|
||||||
|
- r=1 for product streams.
|
||||||
|
- Connect-per-job.
|
||||||
|
|
||||||
|
Re-run exhaustive: `bash scripts/exhaustive-ns1-study.sh` on NS1.
|
||||||
BIN
packages/zapier-decisions/reports/optimal-config/REPORT.pdf
Normal file
BIN
packages/zapier-decisions/reports/optimal-config/REPORT.pdf
Normal file
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 25 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 27 KiB |
|
|
@ -870,6 +870,10 @@ def main() -> None:
|
||||||
"packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.pdf",
|
"packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.pdf",
|
||||||
"Maximized NS1 study: 8 cores, 16 GiB, tmpfs JetStream (delta)",
|
"Maximized NS1 study: 8 cores, 16 GiB, tmpfs JetStream (delta)",
|
||||||
),
|
),
|
||||||
|
(
|
||||||
|
"packages/zapier-decisions/reports/optimal-config/REPORT.pdf",
|
||||||
|
"Optimal NATS config study + DL360 Gen10 projection",
|
||||||
|
),
|
||||||
("packages/verae-nats-cluster/BENCH.pdf", "NATS cluster throughput tables (2026-09-12)"),
|
("packages/verae-nats-cluster/BENCH.pdf", "NATS cluster throughput tables (2026-09-12)"),
|
||||||
("overview/README.pdf", "System overview"),
|
("overview/README.pdf", "System overview"),
|
||||||
("overview/INDEX.pdf", "Documentation index"),
|
("overview/INDEX.pdf", "Documentation index"),
|
||||||
|
|
@ -895,6 +899,10 @@ def main() -> None:
|
||||||
"packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.html",
|
"packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.html",
|
||||||
"Maximized NS1 study (8c/16G/tmpfs)",
|
"Maximized NS1 study (8c/16G/tmpfs)",
|
||||||
),
|
),
|
||||||
|
(
|
||||||
|
"packages/zapier-decisions/reports/optimal-config/REPORT.html",
|
||||||
|
"Optimal NATS config study + DL360 projection",
|
||||||
|
),
|
||||||
("packages/verae-nats-cluster/BENCH.html", "NATS cluster throughput tables"),
|
("packages/verae-nats-cluster/BENCH.html", "NATS cluster throughput tables"),
|
||||||
("docs-master/MESSAGE-FLOWS.html", "Numbered message flows"),
|
("docs-master/MESSAGE-FLOWS.html", "Numbered message flows"),
|
||||||
("docs-master/modules-and-nats.html", "NATS address table"),
|
("docs-master/modules-and-nats.html", "NATS address table"),
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue