Add an NS1-host NATS study: run, charts, methodology, and tuning notes.
Some checks are pending
offline / test (push) Waiting to run
Some checks are pending
offline / test (push) Waiting to run
Second ladder executed entirely on NS1.GEORGELAMBERT.ORG (70.88.205.138) against LXC 511–513; HTML and PDF built on that host.
This commit is contained in:
parent
de20a42109
commit
c5d286dc6b
54 changed files with 4665 additions and 13 deletions
|
|
@ -46,8 +46,11 @@ nkeys/mTLS: `verae-nats-accounts` still has the INTERNAL/LEAF sketch. Do **not**
|
|||
From a **client that is not a nats-* server** (LXC 510 `verae-px-worker`):
|
||||
|
||||
```bash
|
||||
# on NS1
|
||||
# on NS1 (hypervisor). Bench client is LXC 510.
|
||||
bash scripts/bench.sh
|
||||
|
||||
# Full study that must run ON NS1 (orchestrator + charts + HTML/PDF):
|
||||
bash scripts/study-on-ns1.sh
|
||||
```
|
||||
|
||||
That writes `results/<utc>/` and copies a summary to [BENCH.md](BENCH.md). Full write-up with charts (HTML + PDF) is in the progress repo: [nats-cluster-bench](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.pdf). Core NATS is fire-and-forget; JetStream rows use **file** storage and **replicas=3**. Ping delay is sequential RTT; flood delay is queueing under burst.
|
||||
|
|
|
|||
|
|
@ -2,3 +2,5 @@
|
|||
!.gitignore
|
||||
!20260912T045131Z/
|
||||
!20260912T045131Z/**
|
||||
!20260912T051237Z/
|
||||
!20260912T051237Z/**
|
||||
|
|
|
|||
|
|
@ -0,0 +1,48 @@
|
|||
# NATS cluster message speed
|
||||
|
||||
Run **`20260912T051237Z`** (UTC). Client: LXC **510** `verae-px-worker` (`10.10.10.20`), not a nats-* server. Servers: `nats-a/b/c` on `10.10.10.21–23` (`vmbr1` only).
|
||||
|
||||
Client URL:
|
||||
|
||||
```text
|
||||
nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
|
||||
```
|
||||
|
||||
## Method
|
||||
|
||||
- **Core NATS** is fire-and-forget pub/sub (`nats bench`). No disk, no replica ack.
|
||||
- **JetStream** uses **file** storage and **replicas=3** (same as product streams). The unique stream `benchstream` is deleted between JS loads.
|
||||
- Throughput is **msgs/sec** from nats CLI **0.1.6** (`--no-progress --csv`). Its min/avg/max are publisher **rate spread**, not delay.
|
||||
- **Ping** delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster.
|
||||
- **Flood** delay: N publishers dump the whole batch, then the subscriber drains. This is **queueing under burst**, not wire RTT.
|
||||
- Probe: `scripts/latency.mjs` (two connections, header timestamp).
|
||||
|
||||
## Throughput
|
||||
|
||||
| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
|
||||
|-----|------|------------------|------------|----------|------------|----------|
|
||||
| `core-1p1s-50k-128` | core pub/sub | 776,331 | 502,502 | 61.34 | 420,976 | 51.39 |
|
||||
| `core-4p4s-100k-128` | core pub/sub | 1,278,454 | 379,985 | 46.38 | 1,078,910 | 131.70 |
|
||||
| `core-4p4s-50k-1k` | core pub/sub | 581,464 | 209,545 | 204.63 | 482,645 | 471.33 |
|
||||
| `core-8p8s-200k-128` | core pub/sub | 2,065,217 | 333,731 | 40.74 | 1,847,242 | 225.49 |
|
||||
| `js-2p2s-20k-128-r3` | jetstream r=3 file | 15,416 | — | — | — | — |
|
||||
| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 14,985 | 14.63 | — | — |
|
||||
| `js-4p-50k-128-r3` | jetstream r=3 file | — | 17,986 | 2.20 | — | — |
|
||||
|
||||
## Round-trip delay
|
||||
|
||||
| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
|
||||
|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|
|
||||
| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.341ms | 0.470ms | 0.395ms | 0.651ms | 1.377ms | 3.243ms |
|
||||
| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 132.425ms | 203.717ms | 204.458ms | 241.750ms | 249.468ms | 249.640ms |
|
||||
| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 196.422ms | 221.264ms | 224.039ms | 233.180ms | 236.798ms | 238.177ms |
|
||||
| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 211.108ms | 294.925ms | 300.254ms | 314.223ms | 315.551ms | 316.091ms |
|
||||
| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 280.608ms | 414.972ms | 409.802ms | 513.821ms | 535.394ms | 536.084ms |
|
||||
|
||||
## What the numbers mean
|
||||
|
||||
Product job/event/archive traffic is **JetStream r=3 file**. On this three-LXC stand that is about **16k durable 128 B pubs/s** (about **13k** at 1 KiB). Core NATS is an upper bound for non-durable fan-out: about **0.7–2.0M msgs/s** aggregate at 128 B, or **~630k msgs/s (~616 MB/s)** at 1 KiB with 4 publishers.
|
||||
|
||||
A quiet request-reply is **~0.3 ms** average, **p99 < 1 ms**. Flood rows in the **150–500 ms** band are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers.
|
||||
|
||||
Re-run on NS1: `bash scripts/bench.sh`. Raw logs/CSVs are under `results/<utc>/`.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 56 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 42 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 40 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 49 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 52 KiB |
|
|
@ -0,0 +1,3 @@
|
|||
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||
h0r77ozVkEaFq9NntdbNXL,S0,50000,6400000,420976,53884945.437030,0.118772
|
||||
h0r77ozVkEaFq9NntdbNXL,P0,50000,6400000,502502,64320317.108811,0.099502
|
||||
|
|
|
@ -0,0 +1,7 @@
|
|||
=== core-1p1s-50k-128 ===
|
||||
|
||||
NATS Pub/Sub stats: 776,331 msgs/sec ~ 94.77 MB/sec
|
||||
Pub stats: 502,502 msgs/sec ~ 61.34 MB/sec
|
||||
Sub stats: 420,976 msgs/sec ~ 51.39 MB/sec
|
||||
|
||||
Saved metric data in csv file /tmp/bench.csv
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||
9lnpijWl8uftGl7cjgjfDG,S0,100000,12800000,338319,43304923.645655,0.295578
|
||||
9lnpijWl8uftGl7cjgjfDG,S1,100000,12800000,328136,42001487.567998,0.304751
|
||||
9lnpijWl8uftGl7cjgjfDG,S2,100000,12800000,319449,40889502.964037,0.313039
|
||||
9lnpijWl8uftGl7cjgjfDG,S3,100000,12800000,269727,34525150.333968,0.370744
|
||||
9lnpijWl8uftGl7cjgjfDG,P0,25000,3200000,184132,23569014.120587,0.135771
|
||||
9lnpijWl8uftGl7cjgjfDG,P1,25000,3200000,183847,23532419.588601,0.135983
|
||||
9lnpijWl8uftGl7cjgjfDG,P2,25000,3200000,100400,12851250.530089,0.249003
|
||||
9lnpijWl8uftGl7cjgjfDG,P3,25000,3200000,95281,12196082.439191,0.262379
|
||||
|
|
|
@ -0,0 +1,17 @@
|
|||
=== core-4p4s-100k-128 ===
|
||||
|
||||
NATS Pub/Sub stats: 1,278,454 msgs/sec ~ 156.06 MB/sec
|
||||
Pub stats: 379,985 msgs/sec ~ 46.38 MB/sec
|
||||
[1] 184,132 msgs/sec ~ 22.48 MB/sec (25000 msgs)
|
||||
[2] 183,847 msgs/sec ~ 22.44 MB/sec (25000 msgs)
|
||||
[3] 100,400 msgs/sec ~ 12.26 MB/sec (25000 msgs)
|
||||
[4] 95,281 msgs/sec ~ 11.63 MB/sec (25000 msgs)
|
||||
min 95,281 | avg 140,915 | max 184,132 | stddev 43,112 msgs
|
||||
Sub stats: 1,078,910 msgs/sec ~ 131.70 MB/sec
|
||||
[1] 338,319 msgs/sec ~ 41.30 MB/sec (100000 msgs)
|
||||
[2] 328,136 msgs/sec ~ 40.06 MB/sec (100000 msgs)
|
||||
[3] 319,449 msgs/sec ~ 39.00 MB/sec (100000 msgs)
|
||||
[4] 269,727 msgs/sec ~ 32.93 MB/sec (100000 msgs)
|
||||
min 269,727 | avg 313,907 | max 338,319 | stddev 26,367 msgs
|
||||
|
||||
Saved metric data in csv file /tmp/bench.csv
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||
TBolrRDh6R5ucnsIDRic4k,S0,50000,51200000,152508,156168868.728545,0.327850
|
||||
TBolrRDh6R5ucnsIDRic4k,S1,50000,51200000,143666,147114019.174715,0.348029
|
||||
TBolrRDh6R5ucnsIDRic4k,S2,50000,51200000,125714,128731399.676746,0.397727
|
||||
TBolrRDh6R5ucnsIDRic4k,S3,50000,51200000,123195,126152033.863128,0.405859
|
||||
TBolrRDh6R5ucnsIDRic4k,P0,12500,12800000,85005,87045293.746584,0.147050
|
||||
TBolrRDh6R5ucnsIDRic4k,P1,12500,12800000,84991,87030908.333567,0.147074
|
||||
TBolrRDh6R5ucnsIDRic4k,P2,12500,12800000,54739,56053364.168990,0.228354
|
||||
TBolrRDh6R5ucnsIDRic4k,P3,12500,12800000,52467,53726394.065295,0.238244
|
||||
|
|
|
@ -0,0 +1,17 @@
|
|||
=== core-4p4s-50k-1k ===
|
||||
|
||||
NATS Pub/Sub stats: 581,464 msgs/sec ~ 567.84 MB/sec
|
||||
Pub stats: 209,545 msgs/sec ~ 204.63 MB/sec
|
||||
[1] 85,005 msgs/sec ~ 83.01 MB/sec (12500 msgs)
|
||||
[2] 84,991 msgs/sec ~ 83.00 MB/sec (12500 msgs)
|
||||
[3] 54,739 msgs/sec ~ 53.46 MB/sec (12500 msgs)
|
||||
[4] 52,467 msgs/sec ~ 51.24 MB/sec (12500 msgs)
|
||||
min 52,467 | avg 69,300 | max 85,005 | stddev 15,718 msgs
|
||||
Sub stats: 482,645 msgs/sec ~ 471.33 MB/sec
|
||||
[1] 152,508 msgs/sec ~ 148.93 MB/sec (50000 msgs)
|
||||
[2] 143,666 msgs/sec ~ 140.30 MB/sec (50000 msgs)
|
||||
[3] 125,714 msgs/sec ~ 122.77 MB/sec (50000 msgs)
|
||||
[4] 123,195 msgs/sec ~ 120.31 MB/sec (50000 msgs)
|
||||
min 123,195 | avg 136,270 | max 152,508 | stddev 12,255 msgs
|
||||
|
||||
Saved metric data in csv file /tmp/bench.csv
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||
1wXmYbQfeZKgGIuRNRJWE9,S0,200000,25600000,294783,37732336.345174,0.678463
|
||||
1wXmYbQfeZKgGIuRNRJWE9,S1,200000,25600000,285956,36602476.046553,0.699406
|
||||
1wXmYbQfeZKgGIuRNRJWE9,S2,200000,25600000,270995,34687374.508938,0.738021
|
||||
1wXmYbQfeZKgGIuRNRJWE9,S3,200000,25600000,268481,34365602.744388,0.744931
|
||||
1wXmYbQfeZKgGIuRNRJWE9,S4,200000,25600000,242749,31071912.954289,0.823895
|
||||
1wXmYbQfeZKgGIuRNRJWE9,S5,200000,25600000,243757,31200898.690210,0.820489
|
||||
1wXmYbQfeZKgGIuRNRJWE9,S6,200000,25600000,232306,29735268.576473,0.860931
|
||||
1wXmYbQfeZKgGIuRNRJWE9,S7,200000,25600000,231301,29606645.928715,0.864671
|
||||
1wXmYbQfeZKgGIuRNRJWE9,P0,25000,3200000,161768,20706321.217329,0.154542
|
||||
1wXmYbQfeZKgGIuRNRJWE9,P1,25000,3200000,156790,20069204.386621,0.159448
|
||||
1wXmYbQfeZKgGIuRNRJWE9,P2,25000,3200000,101477,12989116.865479,0.246360
|
||||
1wXmYbQfeZKgGIuRNRJWE9,P3,25000,3200000,96840,12395608.110541,0.258156
|
||||
1wXmYbQfeZKgGIuRNRJWE9,P4,25000,3200000,48401,6195444.884723,0.516509
|
||||
1wXmYbQfeZKgGIuRNRJWE9,P5,25000,3200000,56695,7256990.291363,0.440954
|
||||
1wXmYbQfeZKgGIuRNRJWE9,P6,25000,3200000,42589,5451476.161959,0.586997
|
||||
1wXmYbQfeZKgGIuRNRJWE9,P7,25000,3200000,41987,5374439.609281,0.595411
|
||||
|
|
|
@ -0,0 +1,25 @@
|
|||
=== core-8p8s-200k-128 ===
|
||||
|
||||
NATS Pub/Sub stats: 2,065,217 msgs/sec ~ 252.10 MB/sec
|
||||
Pub stats: 333,731 msgs/sec ~ 40.74 MB/sec
|
||||
[1] 161,768 msgs/sec ~ 19.75 MB/sec (25000 msgs)
|
||||
[2] 156,790 msgs/sec ~ 19.14 MB/sec (25000 msgs)
|
||||
[3] 101,477 msgs/sec ~ 12.39 MB/sec (25000 msgs)
|
||||
[4] 96,840 msgs/sec ~ 11.82 MB/sec (25000 msgs)
|
||||
[5] 48,401 msgs/sec ~ 5.91 MB/sec (25000 msgs)
|
||||
[6] 56,695 msgs/sec ~ 6.92 MB/sec (25000 msgs)
|
||||
[7] 42,589 msgs/sec ~ 5.20 MB/sec (25000 msgs)
|
||||
[8] 41,987 msgs/sec ~ 5.13 MB/sec (25000 msgs)
|
||||
min 41,987 | avg 88,318 | max 161,768 | stddev 46,314 msgs
|
||||
Sub stats: 1,847,242 msgs/sec ~ 225.49 MB/sec
|
||||
[1] 294,783 msgs/sec ~ 35.98 MB/sec (200000 msgs)
|
||||
[2] 285,956 msgs/sec ~ 34.91 MB/sec (200000 msgs)
|
||||
[3] 270,995 msgs/sec ~ 33.08 MB/sec (200000 msgs)
|
||||
[4] 268,481 msgs/sec ~ 32.77 MB/sec (200000 msgs)
|
||||
[5] 242,749 msgs/sec ~ 29.63 MB/sec (200000 msgs)
|
||||
[6] 243,757 msgs/sec ~ 29.76 MB/sec (200000 msgs)
|
||||
[7] 232,306 msgs/sec ~ 28.36 MB/sec (200000 msgs)
|
||||
[8] 231,301 msgs/sec ~ 28.24 MB/sec (200000 msgs)
|
||||
min 231,301 | avg 258,791 | max 294,783 | stddev 22,958 msgs
|
||||
|
||||
Saved metric data in csv file /tmp/bench.csv
|
||||
|
|
@ -0,0 +1,86 @@
|
|||
execution_host=NS1.GEORGELAMBERT.ORG
|
||||
execution_ip=70.88.205.138
|
||||
hostname=NS1.GEORGELAMBERT.ORG
|
||||
utc=2026-09-12T05:14:00Z
|
||||
whoami=marchon
|
||||
pwd=/home/marchon/verae-src/verae-nats-cluster
|
||||
uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux
|
||||
nproc=40
|
||||
loadavg=8.67 8.33 8.15 9/3863 4114865
|
||||
client_vmid=510
|
||||
nats_url=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
|
||||
--- free ---
|
||||
total used free shared buff/cache available
|
||||
Mem: 377Gi 38Gi 324Gi 4.3Gi 21Gi 339Gi
|
||||
Swap: 8.0Gi 0B 8.0Gi
|
||||
--- pct list ---
|
||||
VMID Status Lock Name
|
||||
100 stopped CT100
|
||||
101 stopped CT101
|
||||
102 stopped CT102
|
||||
103 stopped CT103
|
||||
104 stopped CT104
|
||||
105 stopped CT105
|
||||
106 stopped CT106
|
||||
107 stopped CT107
|
||||
108 stopped CT108
|
||||
109 stopped CT109
|
||||
110 stopped CT110
|
||||
111 stopped CT111
|
||||
112 stopped CT112
|
||||
113 stopped CT113
|
||||
114 stopped CT114
|
||||
115 stopped CT115
|
||||
116 stopped paul
|
||||
117 stopped CT117
|
||||
118 stopped CT118
|
||||
119 stopped CT119
|
||||
120 stopped rustdeskserver
|
||||
121 stopped CT121
|
||||
122 stopped CT122
|
||||
123 stopped CT123
|
||||
124 stopped bdross
|
||||
200 stopped CT200
|
||||
201 stopped CT201
|
||||
202 stopped CT202
|
||||
300 running mail.veraetime.info
|
||||
400 stopped evercycle-api
|
||||
501 running pki
|
||||
502 running peergos-a
|
||||
503 running peergos-b
|
||||
504 stopped minio
|
||||
510 running verae-px-worker
|
||||
511 running nats-a
|
||||
512 running nats-b
|
||||
513 running nats-c
|
||||
2000 stopped CT2000
|
||||
2001 stopped CT2001
|
||||
2002 stopped CT2002
|
||||
--- pct config 510 ---
|
||||
cores: 2
|
||||
hostname: verae-px-worker
|
||||
memory: 2048
|
||||
net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:BD:5D:62,ip=10.10.10.20/24,type=veth
|
||||
rootfs: SSD2:subvol-510-disk-0,size=16G
|
||||
swap: 512
|
||||
--- pct config 511 ---
|
||||
cores: 1
|
||||
hostname: nats-a
|
||||
memory: 1024
|
||||
net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:CF:AC:68,ip=10.10.10.21/24,type=veth
|
||||
rootfs: SSD2:subvol-511-disk-0,size=8G
|
||||
swap: 256
|
||||
--- pct config 512 ---
|
||||
cores: 1
|
||||
hostname: nats-b
|
||||
memory: 1024
|
||||
net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:42:73:A2,ip=10.10.10.22/24,type=veth
|
||||
rootfs: SSD2:subvol-512-disk-0,size=8G
|
||||
swap: 256
|
||||
--- pct config 513 ---
|
||||
cores: 1
|
||||
hostname: nats-c
|
||||
memory: 1024
|
||||
net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:1B:9B:DC,ip=10.10.10.23/24,type=veth
|
||||
rootfs: SSD2:subvol-513-disk-0,size=8G
|
||||
swap: 256
|
||||
|
|
@ -0,0 +1,86 @@
|
|||
execution_host=NS1.GEORGELAMBERT.ORG
|
||||
execution_ip=70.88.205.138
|
||||
hostname=NS1.GEORGELAMBERT.ORG
|
||||
utc=2026-09-12T05:12:37Z
|
||||
whoami=marchon
|
||||
pwd=/home/marchon/verae-src/verae-nats-cluster
|
||||
uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux
|
||||
nproc=40
|
||||
loadavg=8.59 8.39 8.15 5/3843 4096277
|
||||
client_vmid=510
|
||||
nats_url=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
|
||||
--- free ---
|
||||
total used free shared buff/cache available
|
||||
Mem: 377Gi 38Gi 324Gi 4.2Gi 21Gi 339Gi
|
||||
Swap: 8.0Gi 0B 8.0Gi
|
||||
--- pct list ---
|
||||
VMID Status Lock Name
|
||||
100 stopped CT100
|
||||
101 stopped CT101
|
||||
102 stopped CT102
|
||||
103 stopped CT103
|
||||
104 stopped CT104
|
||||
105 stopped CT105
|
||||
106 stopped CT106
|
||||
107 stopped CT107
|
||||
108 stopped CT108
|
||||
109 stopped CT109
|
||||
110 stopped CT110
|
||||
111 stopped CT111
|
||||
112 stopped CT112
|
||||
113 stopped CT113
|
||||
114 stopped CT114
|
||||
115 stopped CT115
|
||||
116 stopped paul
|
||||
117 stopped CT117
|
||||
118 stopped CT118
|
||||
119 stopped CT119
|
||||
120 stopped rustdeskserver
|
||||
121 stopped CT121
|
||||
122 stopped CT122
|
||||
123 stopped CT123
|
||||
124 stopped bdross
|
||||
200 stopped CT200
|
||||
201 stopped CT201
|
||||
202 stopped CT202
|
||||
300 running mail.veraetime.info
|
||||
400 stopped evercycle-api
|
||||
501 running pki
|
||||
502 running peergos-a
|
||||
503 running peergos-b
|
||||
504 stopped minio
|
||||
510 running verae-px-worker
|
||||
511 running nats-a
|
||||
512 running nats-b
|
||||
513 running nats-c
|
||||
2000 stopped CT2000
|
||||
2001 stopped CT2001
|
||||
2002 stopped CT2002
|
||||
--- pct config 510 ---
|
||||
cores: 2
|
||||
hostname: verae-px-worker
|
||||
memory: 2048
|
||||
net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:BD:5D:62,ip=10.10.10.20/24,type=veth
|
||||
rootfs: SSD2:subvol-510-disk-0,size=16G
|
||||
swap: 512
|
||||
--- pct config 511 ---
|
||||
cores: 1
|
||||
hostname: nats-a
|
||||
memory: 1024
|
||||
net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:CF:AC:68,ip=10.10.10.21/24,type=veth
|
||||
rootfs: SSD2:subvol-511-disk-0,size=8G
|
||||
swap: 256
|
||||
--- pct config 512 ---
|
||||
cores: 1
|
||||
hostname: nats-b
|
||||
memory: 1024
|
||||
net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:42:73:A2,ip=10.10.10.22/24,type=veth
|
||||
rootfs: SSD2:subvol-512-disk-0,size=8G
|
||||
swap: 256
|
||||
--- pct config 513 ---
|
||||
cores: 1
|
||||
hostname: nats-c
|
||||
memory: 1024
|
||||
net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:1B:9B:DC,ip=10.10.10.23/24,type=veth
|
||||
rootfs: SSD2:subvol-513-disk-0,size=8G
|
||||
swap: 256
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||
8rEl4zHbyuPY6Hq62w9Qsw,P0,40000,2560000,7393,946342.174329,2.705153
|
||||
|
|
|
@ -0,0 +1,5 @@
|
|||
=== js-1p-20k-128-r3 ===
|
||||
|
||||
Pub stats: 7,393 msgs/sec ~ 924.16 KB/sec
|
||||
|
||||
Saved metric data in csv file /tmp/bench.csv
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||
VI7cqfC8hjqPoKek3p2a06,S0,20150,1280000,4248,543810.859674,2.353760
|
||||
VI7cqfC8hjqPoKek3p2a06,S1,20159,1280000,3858,493875.492651,2.591746
|
||||
VI7cqfC8hjqPoKek3p2a06,P0,20000,1280000,11290,1445228.034916,0.885673
|
||||
VI7cqfC8hjqPoKek3p2a06,P1,20000,1280000,3864,494600.778891,2.587946
|
||||
|
|
|
@ -0,0 +1,13 @@
|
|||
=== js-2p2s-20k-128-r3 ===
|
||||
|
||||
NATS Pub/Sub stats: 15,416 msgs/sec ~ 1.88 MB/sec
|
||||
Pub stats: 7,728 msgs/sec ~ 966.02 KB/sec
|
||||
[1] 11,290 msgs/sec ~ 1.38 MB/sec (10000 msgs)
|
||||
[2] 3,864 msgs/sec ~ 483.01 KB/sec (10000 msgs)
|
||||
min 3,864 | avg 7,577 | max 11,290 | stddev 3,713 msgs
|
||||
Sub stats: 7,711 msgs/sec ~ 963.88 KB/sec
|
||||
[1] 4,248 msgs/sec ~ 531.07 KB/sec (10000 msgs)
|
||||
[2] 3,858 msgs/sec ~ 482.30 KB/sec (10000 msgs)
|
||||
min 3,858 | avg 4,053 | max 4,248 | stddev 195 msgs
|
||||
|
||||
Saved metric data in csv file /tmp/bench.csv
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||
InppzTmEhsFOG1VYCh0Nle,P0,10000,5120000,4792,4907526.298214,1.043295
|
||||
InppzTmEhsFOG1VYCh0Nle,P1,10000,5120000,4480,4587592.638694,1.116054
|
||||
InppzTmEhsFOG1VYCh0Nle,P2,10000,5120000,3792,3883639.365215,1.318351
|
||||
InppzTmEhsFOG1VYCh0Nle,P3,10000,5120000,3750,3840425.475938,1.333186
|
||||
|
|
|
@ -0,0 +1,10 @@
|
|||
=== js-4p-20k-1k-r3 ===
|
||||
|
||||
Pub stats: 14,985 msgs/sec ~ 14.63 MB/sec
|
||||
[1] 4,792 msgs/sec ~ 4.68 MB/sec (5000 msgs)
|
||||
[2] 4,480 msgs/sec ~ 4.38 MB/sec (5000 msgs)
|
||||
[3] 3,792 msgs/sec ~ 3.70 MB/sec (5000 msgs)
|
||||
[4] 3,750 msgs/sec ~ 3.66 MB/sec (5000 msgs)
|
||||
min 3,750 | avg 4,203 | max 4,792 | stddev 446 msgs
|
||||
|
||||
Saved metric data in csv file /tmp/bench.csv
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
|
||||
JeTysfxECHCJccBaGH4xXX,P0,25000,1600000,5640,721943.982931,2.216238
|
||||
JeTysfxECHCJccBaGH4xXX,P1,25000,1600000,5642,722187.413742,2.215491
|
||||
JeTysfxECHCJccBaGH4xXX,P2,25000,1600000,5056,647242.012627,2.472027
|
||||
JeTysfxECHCJccBaGH4xXX,P3,25000,1600000,4497,575730.101666,2.779080
|
||||
|
|
|
@ -0,0 +1,10 @@
|
|||
=== js-4p-50k-128-r3 ===
|
||||
|
||||
Pub stats: 17,986 msgs/sec ~ 2.20 MB/sec
|
||||
[1] 5,640 msgs/sec ~ 705.02 KB/sec (12500 msgs)
|
||||
[2] 5,642 msgs/sec ~ 705.26 KB/sec (12500 msgs)
|
||||
[3] 5,056 msgs/sec ~ 632.07 KB/sec (12500 msgs)
|
||||
[4] 4,497 msgs/sec ~ 562.24 KB/sec (12500 msgs)
|
||||
min 4,497 | avg 5,208 | max 5,642 | stddev 475 msgs
|
||||
|
||||
Saved metric data in csv file /tmp/bench.csv
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
=== lat-1p-5k-128 ===
|
||||
{"count":5000,"pubs":1,"size":128,"mode":"flood","min_us":132425,"avg_us":203717,"p50_us":204458,"p90_us":241750,"p99_us":249468,"max_us":249640,"min":"132.425ms","avg":"203.717ms","p50":"204.458ms","p90":"241.750ms","p99":"249.468ms","max":"249.640ms"}
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
=== lat-4p-10k-128 ===
|
||||
{"count":10000,"pubs":4,"size":128,"mode":"flood","min_us":211108,"avg_us":294925,"p50_us":300254,"p90_us":314223,"p99_us":315551,"max_us":316091,"min":"211.108ms","avg":"294.925ms","p50":"300.254ms","p90":"314.223ms","p99":"315.551ms","max":"316.091ms"}
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
=== lat-4p-5k-1k ===
|
||||
{"count":5000,"pubs":4,"size":1024,"mode":"flood","min_us":196422,"avg_us":221264,"p50_us":224039,"p90_us":233180,"p99_us":236798,"max_us":238177,"min":"196.422ms","avg":"221.264ms","p50":"224.039ms","p90":"233.180ms","p99":"236.798ms","max":"238.177ms"}
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
=== lat-8p-20k-128 ===
|
||||
{"count":20000,"pubs":8,"size":128,"mode":"flood","min_us":280608,"avg_us":414972,"p50_us":409802,"p90_us":513821,"p99_us":535394,"max_us":536084,"min":"280.608ms","avg":"414.972ms","p50":"409.802ms","p90":"513.821ms","p99":"535.394ms","max":"536.084ms"}
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
=== lat-ping-1k-128 ===
|
||||
{"count":1000,"pubs":1,"size":128,"mode":"ping","min_us":341,"avg_us":470,"p50_us":395,"p90_us":651,"p99_us":1377,"max_us":3243,"min":"0.341ms","avg":"0.470ms","p50":"0.395ms","p90":"0.651ms","p99":"1.377ms","max":"3.243ms"}
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -0,0 +1,334 @@
|
|||
**Progress report (second study)** · run `20260912T051237Z` (UTC)
|
||||
|
||||
> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): the orchestrator (`study-on-ns1.sh`), `nats bench`, `latency.mjs` (inside LXC 510 on this hypervisor), charting (`matplotlib`), and HTML/PDF (`pandoc` + `weasyprint`). The operator laptop did **not** publish, subscribe, draw charts, or render the PDF. Traffic stayed on `vmbr1` from LXC **510** to `nats-a/b/c` (**511–513**).
|
||||
|
||||
This is a full methodology write-up plus the numbers from that on-host run. The earlier report (`nats-cluster-bench`, run `20260912T045131Z`) used the same cluster but was **orchestrated and rendered off-box**. Use this document when you need “it was all run on 138.”
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive summary
|
||||
|
||||
| Item | This NS1-host run |
|
||||
|------|-------------------|
|
||||
| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `marchon` |
|
||||
| Bench client | LXC 510 `verae-px-worker` |
|
||||
| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` |
|
||||
| Client URL | `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` |
|
||||
| Host load before | `8.59 8.39 8.15 5/3843 4096277` |
|
||||
| Host load after | `8.67 8.33 8.15 9/3863 4114865` |
|
||||
| Core 1p1s 128 B pub | 502,502 msgs/s |
|
||||
| JetStream 1p 128 B r=3 | 7,393 durable pubs/s |
|
||||
| Ping p50 / p99 | 0.395ms / 1.377ms |
|
||||
|
||||
Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst.
|
||||
|
||||
---
|
||||
|
||||
## 2. Where it ran (and where it did not)
|
||||
|
||||
```text
|
||||
Operator laptop ──ssh──► NS1.GEORGELAMBERT.ORG 70.88.205.138
|
||||
study-on-ns1.sh
|
||||
python3 build-ns1-study-report.py
|
||||
sudo pct exec 510 ──► nats bench / latency.mjs
|
||||
│
|
||||
▼ vmbr1
|
||||
10.10.10.21-23 :4222
|
||||
```
|
||||
|
||||
- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510).
|
||||
- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file.
|
||||
|
||||
---
|
||||
|
||||
## 3. Results (this run)
|
||||
|
||||
### Host and brokers
|
||||
|
||||
**Before**
|
||||
|
||||
| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream |
|
||||
|------|------|-------------|---------|----------|-----|-------|---------|-----------|
|
||||
| nats-a | 511 | 2 | 1,721,203 | 4,933,759 | 1 | 1 | 26,398,720 | True |
|
||||
| nats-b | 512 | 2 | 1,613,205 | 3,513,096 | 0 | 1 | 23,097,344 | True |
|
||||
| nats-c | 513 | 2 | 1,760,973 | 4,048,582 | 2 | 1 | 23,867,392 | True |
|
||||
|
||||
**After**
|
||||
|
||||
| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream |
|
||||
|------|------|-------------|---------|----------|-----|-------|---------|-----------|
|
||||
| nats-a | 511 | 2 | 2,097,596 | 5,910,161 | 1 | 1 | 38,273,024 | True |
|
||||
| nats-b | 512 | 2 | 2,121,816 | 4,746,539 | 1 | 1 | 41,361,408 | True |
|
||||
| nats-c | 513 | 2 | 2,241,366 | 5,053,978 | 1 | 1 | 38,162,432 | True |
|
||||
|
||||
nproc=40 · uname=`Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux`
|
||||
|
||||
### Throughput
|
||||
|
||||
| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
|
||||
|-----|------|------------------|------------|----------|------------|----------|
|
||||
| `core-1p1s-50k-128` | core pub/sub | 776,331 | 502,502 | 61.34 | 420,976 | 51.39 |
|
||||
| `core-4p4s-100k-128` | core pub/sub | 1,278,454 | 379,985 | 46.38 | 1,078,910 | 131.70 |
|
||||
| `core-4p4s-50k-1k` | core pub/sub | 581,464 | 209,545 | 204.63 | 482,645 | 471.33 |
|
||||
| `core-8p8s-200k-128` | core pub/sub | 2,065,217 | 333,731 | 40.74 | 1,847,242 | 225.49 |
|
||||
| `js-1p-20k-128-r3` | jetstream r=3 file | — | 7,393 | 0.90 | — | — |
|
||||
| `js-2p2s-20k-128-r3` | jetstream r=3 file | 15,416 | 7,728 | 0.94 | 7,711 | 0.94 |
|
||||
| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 14,985 | 14.63 | — | — |
|
||||
| `js-4p-50k-128-r3` | jetstream r=3 file | — | 17,986 | 2.20 | — | — |
|
||||
|
||||
### Round-trip delay
|
||||
|
||||
| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
|
||||
|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|
|
||||
| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.341ms | 0.470ms | 0.395ms | 0.651ms | 1.377ms | 3.243ms |
|
||||
| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 132.425ms | 203.717ms | 204.458ms | 241.750ms | 249.468ms | 249.640ms |
|
||||
| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 196.422ms | 221.264ms | 224.039ms | 233.180ms | 236.798ms | 238.177ms |
|
||||
| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 211.108ms | 294.925ms | 300.254ms | 314.223ms | 315.551ms | 316.091ms |
|
||||
| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 280.608ms | 414.972ms | 409.802ms | 513.821ms | 535.394ms | 536.084ms |
|
||||
|
||||
### Core NATS
|
||||
|
||||
|
||||
|
||||
*Core NATS throughput at four loads (NS1 host run)*
|
||||
### Payload size (core)
|
||||
|
||||
|
||||
|
||||
*Core NATS 128 B vs 1 KiB (NS1 host run)*
|
||||
### JetStream r=3 file
|
||||
|
||||
|
||||
|
||||
*JetStream durable publish rate (NS1 host run)*
|
||||
### Core vs JetStream
|
||||
|
||||
|
||||
|
||||
*Core vs JetStream publish rate, log scale (NS1 host run)*
|
||||
### Delay
|
||||
|
||||
|
||||
|
||||
*Ping vs flood delay percentiles, log scale (NS1 host run)*
|
||||
|
||||
---
|
||||
|
||||
## 4. Study methodology
|
||||
|
||||
### 4.1 Question
|
||||
|
||||
On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)?
|
||||
|
||||
### 4.2 Hypotheses (stated before the run)
|
||||
|
||||
1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica.
|
||||
2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated.
|
||||
3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting.
|
||||
4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count.
|
||||
5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound.
|
||||
|
||||
### 4.3 Independent variables (what we changed)
|
||||
|
||||
| Factor | Levels |
|
||||
|--------|--------|
|
||||
| Transport | Core NATS pub/sub vs JetStream file replicas=3 |
|
||||
| Publisher count | 1, 2, 4, 8 |
|
||||
| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 |
|
||||
| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) |
|
||||
| Payload | 128 B, 1024 B |
|
||||
| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) |
|
||||
|
||||
### 4.4 Dependent variables (what we recorded)
|
||||
|
||||
| Metric | Instrument | Unit |
|
||||
|--------|------------|------|
|
||||
| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s |
|
||||
| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s |
|
||||
| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) |
|
||||
| Publisher spread | nats min/avg/max **msgs/s** | not delay |
|
||||
| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max |
|
||||
| Host load | `/proc/loadavg` before and after | load average |
|
||||
| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem |
|
||||
|
||||
**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`.
|
||||
|
||||
### 4.5 Controls and constants
|
||||
|
||||
- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`.
|
||||
- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`).
|
||||
- Bench client is LXC **510**, not a nats-* server.
|
||||
- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide.
|
||||
- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`).
|
||||
- No TLS, no nkeys, no account isolation (isolation is `vmbr1`).
|
||||
- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder.
|
||||
|
||||
### 4.6 Procedure
|
||||
|
||||
1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise.
|
||||
2. Snapshot host load, memory, LXC configs, and each nats `varz`.
|
||||
3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB).
|
||||
4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file.
|
||||
5. Copy `latency.mjs` into 510; ping then flood at several batch sizes.
|
||||
6. Snapshot host/`varz` again.
|
||||
7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**.
|
||||
|
||||
No publish, subscribe, chart, or PDF process runs on the operator laptop for this study.
|
||||
|
||||
### 4.7 Instrumentation path
|
||||
|
||||
```text
|
||||
[NS1 host 70.88.205.138]
|
||||
study-on-ns1.sh (bash + python3)
|
||||
|
|
||||
| sudo pct exec 510
|
||||
v
|
||||
[LXC 510 verae-px-worker 10.10.10.20]
|
||||
nats bench / node latency.mjs
|
||||
|
|
||||
| NATS client protocol to
|
||||
v
|
||||
[LXC 511/512/513 10.10.10.21-23 :4222]
|
||||
nats-server -js cluster routes :6222
|
||||
```
|
||||
|
||||
The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`.
|
||||
|
||||
### 4.8 Threats to validity
|
||||
|
||||
| Threat | Effect on numbers |
|
||||
|--------|-------------------|
|
||||
| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. |
|
||||
| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. |
|
||||
| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. |
|
||||
| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. |
|
||||
| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. |
|
||||
| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. |
|
||||
| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. |
|
||||
| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. |
|
||||
|
||||
### 4.9 Ethics / safety
|
||||
|
||||
Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket.
|
||||
|
||||
---
|
||||
|
||||
## 5. Suggestions for fine-tuning
|
||||
|
||||
These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way.
|
||||
|
||||
### 5.1 Treat JetStream as the product limiter
|
||||
|
||||
Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out.
|
||||
|
||||
### 5.2 Split storage class by stream
|
||||
|
||||
| Stream | Suggested store | Why |
|
||||
|--------|-----------------|-----|
|
||||
| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad |
|
||||
| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both |
|
||||
| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer |
|
||||
| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry |
|
||||
| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive |
|
||||
|
||||
Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert.
|
||||
|
||||
### 5.3 Give JetStream real disks
|
||||
|
||||
Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain.
|
||||
|
||||
- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`.
|
||||
- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure.
|
||||
- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it.
|
||||
- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.”
|
||||
|
||||
### 5.4 Isolate the nats CTs from the rest of NS1
|
||||
|
||||
Host load on this box is often already several. Pin:
|
||||
|
||||
- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes.
|
||||
- Memory high enough that file-backed streams stay cache-hot for the working set.
|
||||
- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync.
|
||||
|
||||
Re-run this study after pinning; H1/H2 should move more than ping.
|
||||
|
||||
### 5.5 Consumer and mailbox tuning (delay H4)
|
||||
|
||||
Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT.
|
||||
|
||||
- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM.
|
||||
- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber.
|
||||
- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped.
|
||||
- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT.
|
||||
|
||||
### 5.6 Publisher-side batching in middleware
|
||||
|
||||
A timestamp job is one small JSON. 16k msgs/s is ample. Still:
|
||||
|
||||
- Avoid per-byte publishes; one message per job/event.
|
||||
- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs).
|
||||
- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only.
|
||||
|
||||
### 5.7 nats-server knobs worth measuring (A/B with this script)
|
||||
|
||||
| Knob | Why try it |
|
||||
|------|------------|
|
||||
| `max_payload` | Keep default unless archive puts grow |
|
||||
| `write_deadline` | Slow consumer protection for webhooks |
|
||||
| `max_pending` | Bound memory on a stuck Zapier hook |
|
||||
| `max_connections` | Fleet workers + keep + middleware |
|
||||
| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT |
|
||||
| `max_outstanding_catchup` | Replica restart after a nats-c blip |
|
||||
| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT |
|
||||
|
||||
Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99.
|
||||
|
||||
### 5.8 Network
|
||||
|
||||
- Keep NATS off `vmbr0`. No change.
|
||||
- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load).
|
||||
- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus.
|
||||
|
||||
### 5.9 Security cost (when nkeys/mTLS flip)
|
||||
|
||||
`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement.
|
||||
|
||||
### 5.10 Operational fine-tuning (lag, not peak msgs/s)
|
||||
|
||||
1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`.
|
||||
2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense.
|
||||
3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost.
|
||||
4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated.
|
||||
|
||||
### 5.11 What not to tune
|
||||
|
||||
- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge.
|
||||
- Do not treat flood 400 ms as “cluster RTT.” Fix consumers.
|
||||
- Do not load-test on `ZAPIER_*` streams.
|
||||
- Do not bind client NATS to `0.0.0.0` on `vmbr0`.
|
||||
|
||||
### 5.12 Recommended next experiments (same method, one change each)
|
||||
|
||||
1. CPU pin nats-a/b/c → re-run JS 1p + ping.
|
||||
2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder).
|
||||
3. Distinct `store_dir` disks per node.
|
||||
4. nkeys on, same ladder.
|
||||
5. Three hardware boxes, same `cluster.env` IPs updated.
|
||||
|
||||
Each experiment should produce a new `results/<utc>/` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory.
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 6. Reproducing this study
|
||||
|
||||
On **NS1 only**:
|
||||
|
||||
```bash
|
||||
cd ~/verae-src/verae-nats-cluster
|
||||
bash scripts/study-on-ns1.sh
|
||||
```
|
||||
|
||||
The script exits if `hostname` is not NS1. Outputs land in `results/<utc>/` including `nats-cluster-bench-ns1.{md,html,pdf}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog.
|
||||
|
||||
Raw logs for this run: `results/20260912T051237Z/`.
|
||||
Binary file not shown.
|
|
@ -0,0 +1,50 @@
|
|||
[
|
||||
{
|
||||
"vmid": "511",
|
||||
"name": "nats-a",
|
||||
"server_name": "nats-a",
|
||||
"host": "10.10.10.21",
|
||||
"port": 4222,
|
||||
"connections": 2,
|
||||
"in_msgs": 2097596,
|
||||
"out_msgs": 5910161,
|
||||
"in_bytes": 559427613,
|
||||
"out_bytes": 1494241553,
|
||||
"cpu": 1,
|
||||
"cores": 1,
|
||||
"mem": 38273024,
|
||||
"jetstream": true
|
||||
},
|
||||
{
|
||||
"vmid": "512",
|
||||
"name": "nats-b",
|
||||
"server_name": "nats-b",
|
||||
"host": "10.10.10.22",
|
||||
"port": 4222,
|
||||
"connections": 2,
|
||||
"in_msgs": 2121816,
|
||||
"out_msgs": 4746539,
|
||||
"in_bytes": 585390797,
|
||||
"out_bytes": 974695990,
|
||||
"cpu": 1,
|
||||
"cores": 1,
|
||||
"mem": 41361408,
|
||||
"jetstream": true
|
||||
},
|
||||
{
|
||||
"vmid": "513",
|
||||
"name": "nats-c",
|
||||
"server_name": "nats-c",
|
||||
"host": "10.10.10.23",
|
||||
"port": 4222,
|
||||
"connections": 2,
|
||||
"in_msgs": 2241366,
|
||||
"out_msgs": 5053978,
|
||||
"in_bytes": 586858467,
|
||||
"out_bytes": 1040486148,
|
||||
"cpu": 1,
|
||||
"cores": 1,
|
||||
"mem": 38162432,
|
||||
"jetstream": true
|
||||
}
|
||||
]
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
[
|
||||
{
|
||||
"vmid": "511",
|
||||
"name": "nats-a",
|
||||
"server_name": "nats-a",
|
||||
"host": "10.10.10.21",
|
||||
"port": 4222,
|
||||
"connections": 2,
|
||||
"in_msgs": 1721203,
|
||||
"out_msgs": 4933759,
|
||||
"in_bytes": 440427700,
|
||||
"out_bytes": 1169323990,
|
||||
"cpu": 1,
|
||||
"cores": 1,
|
||||
"mem": 26398720,
|
||||
"jetstream": true
|
||||
},
|
||||
{
|
||||
"vmid": "512",
|
||||
"name": "nats-b",
|
||||
"server_name": "nats-b",
|
||||
"host": "10.10.10.22",
|
||||
"port": 4222,
|
||||
"connections": 2,
|
||||
"in_msgs": 1613205,
|
||||
"out_msgs": 3513096,
|
||||
"in_bytes": 428640182,
|
||||
"out_bytes": 715428083,
|
||||
"cpu": 0,
|
||||
"cores": 1,
|
||||
"mem": 23097344,
|
||||
"jetstream": true
|
||||
},
|
||||
{
|
||||
"vmid": "513",
|
||||
"name": "nats-c",
|
||||
"server_name": "nats-c",
|
||||
"host": "10.10.10.23",
|
||||
"port": 4222,
|
||||
"connections": 2,
|
||||
"in_msgs": 1760973,
|
||||
"out_msgs": 4048582,
|
||||
"in_bytes": 447844308,
|
||||
"out_bytes": 864199465,
|
||||
"cpu": 2,
|
||||
"cores": 1,
|
||||
"mem": 23867392,
|
||||
"jetstream": true
|
||||
}
|
||||
]
|
||||
|
|
@ -16,18 +16,34 @@ def fmt_int(s: str | None) -> str:
|
|||
|
||||
def parse_bench(text: str) -> dict[str, str]:
|
||||
out: dict[str, str] = {"kind": "throughput"}
|
||||
m = re.search(r"(?m)^\s*Pub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*MB/sec", text)
|
||||
if m:
|
||||
out["pub_msgs"] = m.group(1).replace(",", "")
|
||||
out["pub_mb"] = m.group(2)
|
||||
m = re.search(r"(?m)^\s*Sub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*MB/sec", text)
|
||||
if m:
|
||||
out["sub_msgs"] = m.group(1).replace(",", "")
|
||||
out["sub_mb"] = m.group(2)
|
||||
m = re.search(r"NATS Pub/Sub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*MB/sec", text)
|
||||
if m:
|
||||
out["agg_msgs"] = m.group(1).replace(",", "")
|
||||
out["agg_mb"] = m.group(2)
|
||||
def _rate(pattern: str, msgs_key: str, mb_key: str) -> None:
|
||||
m = re.search(pattern, text)
|
||||
if not m:
|
||||
return
|
||||
out[msgs_key] = m.group(1).replace(",", "")
|
||||
n = float(m.group(2))
|
||||
unit = m.group(3).upper()
|
||||
if unit == "KB":
|
||||
n = n / 1024.0
|
||||
elif unit == "GB":
|
||||
n = n * 1024.0
|
||||
out[mb_key] = f"{n:.2f}"
|
||||
|
||||
_rate(
|
||||
r"(?m)^\s*Pub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*(KB|MB|GB)/sec",
|
||||
"pub_msgs",
|
||||
"pub_mb",
|
||||
)
|
||||
_rate(
|
||||
r"(?m)^\s*Sub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*(KB|MB|GB)/sec",
|
||||
"sub_msgs",
|
||||
"sub_mb",
|
||||
)
|
||||
_rate(
|
||||
r"NATS Pub/Sub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*(KB|MB|GB)/sec",
|
||||
"agg_msgs",
|
||||
"agg_mb",
|
||||
)
|
||||
if "JetStream" in text or "--js" in text or "js-" in text:
|
||||
out["mode"] = "jetstream r=3 file"
|
||||
else:
|
||||
|
|
|
|||
458
packages/verae-nats-cluster/scripts/build-ns1-study-report.py
Executable file
458
packages/verae-nats-cluster/scripts/build-ns1-study-report.py
Executable file
|
|
@ -0,0 +1,458 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Build the NS1-host study report (charts + markdown + HTML + PDF) from a results dir.
|
||||
|
||||
Must be able to run entirely on NS1.GEORGELAMBERT.ORG with python3, matplotlib,
|
||||
pandoc, and weasyprint. Parses nats bench logs; does not hard-code rates.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
import importlib.util
|
||||
|
||||
_spec = importlib.util.spec_from_file_location(
|
||||
"bench_report", Path(__file__).resolve().parent / "bench-report.py"
|
||||
)
|
||||
_br = importlib.util.module_from_spec(_spec)
|
||||
assert _spec.loader is not None
|
||||
_spec.loader.exec_module(_br)
|
||||
fmt_int = _br.fmt_int
|
||||
lat_mode = _br.lat_mode
|
||||
lat_sort = _br.lat_sort
|
||||
parse_bench = _br.parse_bench
|
||||
parse_lat = _br.parse_lat
|
||||
thru_sort = _br.thru_sort
|
||||
|
||||
try:
|
||||
import matplotlib
|
||||
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
from matplotlib.ticker import FuncFormatter
|
||||
except ImportError as e:
|
||||
raise SystemExit(f"matplotlib required on NS1: {e}") from e
|
||||
|
||||
INDIGO = "#4f46e5"
|
||||
DEEP = "#312e81"
|
||||
TEAL = "#047857"
|
||||
AMBER = "#b45309"
|
||||
LILAC = "#7c74f0"
|
||||
INK = "#171a26"
|
||||
MUTED = "#5b6178"
|
||||
GRID = "#d9dce8"
|
||||
|
||||
CORE_LABELS = {
|
||||
"core-1p1s-50k-128": "1p1s\n50k×128 B",
|
||||
"core-4p4s-100k-128": "4p4s\n100k×128 B",
|
||||
"core-8p8s-200k-128": "8p8s\n200k×128 B",
|
||||
"core-4p4s-50k-1k": "4p4s\n50k×1 KiB",
|
||||
}
|
||||
JS_LABELS = {
|
||||
"js-1p-20k-128-r3": "1p 20k×128 B",
|
||||
"js-4p-50k-128-r3": "4p 50k×128 B",
|
||||
"js-4p-20k-1k-r3": "4p 20k×1 KiB",
|
||||
"js-2p2s-20k-128-r3": "2p2s pull 20k×128 B",
|
||||
}
|
||||
LAT_LABELS = {
|
||||
"lat-ping-1k-128": "Ping\n1k×128 B",
|
||||
"lat-1p-5k-128": "Flood 1p\n5k×128 B",
|
||||
"lat-4p-5k-1k": "Flood 4p\n5k×1 KiB",
|
||||
"lat-4p-10k-128": "Flood 4p\n10k×128 B",
|
||||
"lat-8p-20k-128": "Flood 8p\n20k×128 B",
|
||||
}
|
||||
|
||||
|
||||
def ms(s: str) -> float:
|
||||
return float(s.replace("ms", "").replace(",", "").strip())
|
||||
|
||||
|
||||
def k_fmt(x: float, _pos: int | None = None) -> str:
|
||||
if x >= 1_000_000:
|
||||
return f"{x / 1_000_000:.2f}M"
|
||||
if x >= 1000:
|
||||
return f"{x / 1000:.0f}k"
|
||||
return f"{x:.0f}"
|
||||
|
||||
|
||||
def style() -> None:
|
||||
plt.rcParams.update(
|
||||
{
|
||||
"font.family": "sans-serif",
|
||||
"font.size": 10,
|
||||
"axes.titlesize": 12,
|
||||
"axes.titleweight": "semibold",
|
||||
"axes.edgecolor": GRID,
|
||||
"axes.labelcolor": INK,
|
||||
"text.color": INK,
|
||||
"xtick.color": MUTED,
|
||||
"ytick.color": MUTED,
|
||||
"figure.facecolor": "white",
|
||||
"axes.facecolor": "white",
|
||||
"axes.grid": True,
|
||||
"grid.color": GRID,
|
||||
"grid.linewidth": 0.8,
|
||||
"legend.frameon": False,
|
||||
"savefig.bbox": "tight",
|
||||
"savefig.dpi": 160,
|
||||
"savefig.facecolor": "white",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def save(fig: plt.Figure, path: Path) -> None:
|
||||
fig.savefig(path, dpi=160)
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def load_runs(folder: Path) -> tuple[list[dict[str, str]], list[dict[str, str]]]:
|
||||
thru: list[dict[str, str]] = []
|
||||
lats: list[dict[str, str]] = []
|
||||
for f in sorted(folder.glob("*.txt")):
|
||||
if f.name.startswith("host-"):
|
||||
continue
|
||||
text = f.read_text(encoding="utf-8", errors="replace")
|
||||
lat = parse_lat(text)
|
||||
if lat:
|
||||
lat["run"] = f.stem
|
||||
lats.append(lat)
|
||||
continue
|
||||
p = parse_bench(text)
|
||||
if p.get("pub_msgs") or p.get("agg_msgs"):
|
||||
p["run"] = f.stem
|
||||
thru.append(p)
|
||||
return sorted(thru, key=thru_sort), sorted(lats, key=lat_sort)
|
||||
|
||||
|
||||
def kv_file(path: Path) -> dict[str, str]:
|
||||
out: dict[str, str] = {}
|
||||
if not path.exists():
|
||||
return out
|
||||
for line in path.read_text(encoding="utf-8", errors="replace").splitlines():
|
||||
if "=" in line and not line.startswith("---"):
|
||||
k, _, v = line.partition("=")
|
||||
if k.strip() in out:
|
||||
continue
|
||||
out[k.strip()] = v.strip()
|
||||
return out
|
||||
|
||||
|
||||
def thru_table(thru: list[dict[str, str]]) -> str:
|
||||
lines = [
|
||||
"| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |",
|
||||
"|-----|------|------------------|------------|----------|------------|----------|",
|
||||
]
|
||||
for p in thru:
|
||||
lines.append(
|
||||
f"| `{p['run']}` | {p.get('mode', '')} | {fmt_int(p.get('agg_msgs'))} | "
|
||||
f"{fmt_int(p.get('pub_msgs'))} | {p.get('pub_mb') or '—'} | "
|
||||
f"{fmt_int(p.get('sub_msgs'))} | {p.get('sub_mb') or '—'} |"
|
||||
)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def delay_table(lats: list[dict[str, str]]) -> str:
|
||||
lines = [
|
||||
"| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |",
|
||||
"|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|",
|
||||
]
|
||||
for p in lats:
|
||||
kind = lat_mode(p.get("run", ""), p.get("mode", ""))
|
||||
label = "ping (sequential RTT)" if kind == "ping" else "flood (burst queueing)"
|
||||
lines.append(
|
||||
f"| `{p['run']}` | {label} | {p.get('count', '')} | {p.get('pubs', '')} | "
|
||||
f"{p.get('size', '')} B | {p.get('min', '')} | {p.get('avg', '')} | "
|
||||
f"{p.get('p50', '')} | {p.get('p90', '')} | {p.get('p99', '')} | {p.get('max', '')} |"
|
||||
)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def varz_table(path: Path) -> str:
|
||||
if not path.exists():
|
||||
return "_varz snapshot not captured._"
|
||||
rows = json.loads(path.read_text(encoding="utf-8"))
|
||||
lines = [
|
||||
"| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream |",
|
||||
"|------|------|-------------|---------|----------|-----|-------|---------|-----------|",
|
||||
]
|
||||
for r in rows:
|
||||
if r.get("error"):
|
||||
lines.append(f"| {r.get('name')} | {r.get('vmid')} | error: {r['error']} | | | | | | |")
|
||||
continue
|
||||
lines.append(
|
||||
f"| {r.get('name')} | {r.get('vmid')} | {r.get('connections')} | "
|
||||
f"{r.get('in_msgs'):,} | {r.get('out_msgs'):,} | {r.get('cpu')} | "
|
||||
f"{r.get('cores')} | {r.get('mem'):,} | {r.get('jetstream')} |"
|
||||
)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def charts(thru: list[dict[str, str]], lats: list[dict[str, str]], dest: Path) -> None:
|
||||
dest.mkdir(parents=True, exist_ok=True)
|
||||
style()
|
||||
by = {p["run"]: p for p in thru}
|
||||
core_keys = [k for k in CORE_LABELS if k in by]
|
||||
if core_keys:
|
||||
fig, ax = plt.subplots(figsize=(9.2, 4.4))
|
||||
x = list(range(len(core_keys)))
|
||||
w = 0.25
|
||||
agg = [int(by[k].get("agg_msgs") or 0) for k in core_keys]
|
||||
pub = [int(by[k].get("pub_msgs") or 0) for k in core_keys]
|
||||
sub = [int(by[k].get("sub_msgs") or 0) for k in core_keys]
|
||||
ax.bar([i - w for i in x], agg, w, label="Aggregate", color=DEEP)
|
||||
ax.bar(x, pub, w, label="Publish", color=INDIGO)
|
||||
ax.bar([i + w for i in x], sub, w, label="Subscribe", color=TEAL)
|
||||
ax.set_xticks(x, [CORE_LABELS[k] for k in core_keys])
|
||||
ax.set_ylabel("messages / second")
|
||||
ax.set_title("Core NATS throughput (fire-and-forget) — NS1 host run")
|
||||
ax.yaxis.set_major_formatter(FuncFormatter(k_fmt))
|
||||
ax.legend(loc="upper left")
|
||||
ax.set_axisbelow(True)
|
||||
save(fig, dest / "core-throughput.png")
|
||||
|
||||
js_keys = [k for k in JS_LABELS if k in by]
|
||||
if js_keys:
|
||||
fig, ax = plt.subplots(figsize=(9.2, 4.4))
|
||||
pubs = [int(by[k].get("pub_msgs") or 0) for k in js_keys]
|
||||
colors = [INDIGO, INDIGO, AMBER, LILAC][: len(js_keys)]
|
||||
ax.bar([JS_LABELS[k] for k in js_keys], pubs, color=colors)
|
||||
ax.set_ylabel("durable publish messages / second")
|
||||
ax.set_title("JetStream file store, replicas=3 — NS1 host run")
|
||||
ax.yaxis.set_major_formatter(FuncFormatter(k_fmt))
|
||||
ax.set_axisbelow(True)
|
||||
for i, v in enumerate(pubs):
|
||||
ax.text(i, v * 1.02, f"{v:,}", ha="center", va="bottom", fontsize=9, color=MUTED)
|
||||
save(fig, dest / "js-throughput.png")
|
||||
|
||||
pair = [("core-1p1s-50k-128", "js-1p-20k-128-r3"), ("core-4p4s-100k-128", "js-4p-50k-128-r3"), ("core-4p4s-50k-1k", "js-4p-20k-1k-r3")]
|
||||
if all(c in by and j in by for c, j in pair):
|
||||
fig, ax = plt.subplots(figsize=(9.2, 4.4))
|
||||
labels = ["1 publisher\n128 B", "4 publishers\n128 B", "4 publishers\n1 KiB"]
|
||||
core_pub = [int(by[c]["pub_msgs"]) for c, _ in pair]
|
||||
js_pub = [int(by[j]["pub_msgs"]) for _, j in pair]
|
||||
x = list(range(3))
|
||||
w = 0.35
|
||||
ax.bar([i - w / 2 for i in x], core_pub, w, label="Core NATS (no disk)", color=INDIGO)
|
||||
ax.bar([i + w / 2 for i in x], js_pub, w, label="JetStream r=3 file", color=AMBER)
|
||||
ax.set_xticks(x, labels)
|
||||
ax.set_yscale("log")
|
||||
ax.set_ylabel("publish messages / second (log)")
|
||||
ax.set_title("Core vs JetStream — NS1 host run")
|
||||
ax.legend(loc="upper right")
|
||||
ax.set_axisbelow(True)
|
||||
save(fig, dest / "core-vs-js.png")
|
||||
|
||||
if "core-4p4s-100k-128" in by and "core-4p4s-50k-1k" in by:
|
||||
fig, axes = plt.subplots(1, 2, figsize=(9.2, 4.2))
|
||||
labels = ["128 B\n4p4s", "1 KiB\n4p4s"]
|
||||
msgs = [int(by["core-4p4s-100k-128"].get("agg_msgs") or 0), int(by["core-4p4s-50k-1k"].get("agg_msgs") or 0)]
|
||||
mb = [float(by["core-4p4s-100k-128"].get("agg_mb") or 0), float(by["core-4p4s-50k-1k"].get("agg_mb") or 0)]
|
||||
axes[0].bar(labels, msgs, color=[INDIGO, AMBER])
|
||||
axes[0].set_title("Aggregate messages / second")
|
||||
axes[0].yaxis.set_major_formatter(FuncFormatter(k_fmt))
|
||||
axes[1].bar(labels, mb, color=[INDIGO, AMBER])
|
||||
axes[1].set_title("Aggregate MB / second")
|
||||
fig.suptitle("Core NATS payload effect — NS1 host run", fontsize=12, fontweight="semibold")
|
||||
fig.tight_layout()
|
||||
save(fig, dest / "payload-size.png")
|
||||
|
||||
if lats:
|
||||
fig, ax = plt.subplots(figsize=(9.2, 4.6))
|
||||
ordered = [p for p in lats]
|
||||
labels = [LAT_LABELS.get(p["run"], p["run"]) for p in ordered]
|
||||
x = list(range(len(ordered)))
|
||||
w = 0.25
|
||||
p50 = [ms(p["p50"]) for p in ordered]
|
||||
p90 = [ms(p["p90"]) for p in ordered]
|
||||
p99 = [ms(p["p99"]) for p in ordered]
|
||||
ax.bar([i - w for i in x], p50, w, label="p50", color=TEAL)
|
||||
ax.bar(x, p90, w, label="p90", color=INDIGO)
|
||||
ax.bar([i + w for i in x], p99, w, label="p99", color=AMBER)
|
||||
ax.set_xticks(x, labels)
|
||||
ax.set_yscale("log")
|
||||
ax.set_ylabel("milliseconds (log)")
|
||||
ax.set_title("Round-trip delay — NS1 host run")
|
||||
ax.axhline(1.0, color=GRID, linestyle="--", linewidth=1)
|
||||
ax.legend(loc="upper left")
|
||||
ax.set_axisbelow(True)
|
||||
save(fig, dest / "delay-percentiles.png")
|
||||
|
||||
|
||||
def figure(name: str, caption: str) -> str:
|
||||
return f"\n\n*{caption}*"
|
||||
|
||||
|
||||
def write_markdown(folder: Path, thru: list[dict[str, str]], lats: list[dict[str, str]]) -> str:
|
||||
before = kv_file(folder / "host-before.txt")
|
||||
after = kv_file(folder / "host-after.txt")
|
||||
stamp = folder.name
|
||||
method = (Path(__file__).resolve().parent / "ns1-study-methodology.md").read_text(encoding="utf-8")
|
||||
figs = []
|
||||
charts_dir = folder / "charts"
|
||||
if (charts_dir / "core-throughput.png").exists():
|
||||
figs.append("### Core NATS\n\n" + figure("core-throughput.png", "Core NATS throughput at four loads (NS1 host run)"))
|
||||
if (charts_dir / "payload-size.png").exists():
|
||||
figs.append("### Payload size (core)\n\n" + figure("payload-size.png", "Core NATS 128 B vs 1 KiB (NS1 host run)"))
|
||||
if (charts_dir / "js-throughput.png").exists():
|
||||
figs.append("### JetStream r=3 file\n\n" + figure("js-throughput.png", "JetStream durable publish rate (NS1 host run)"))
|
||||
if (charts_dir / "core-vs-js.png").exists():
|
||||
figs.append("### Core vs JetStream\n\n" + figure("core-vs-js.png", "Core vs JetStream publish rate, log scale (NS1 host run)"))
|
||||
if (charts_dir / "delay-percentiles.png").exists():
|
||||
figs.append("### Delay\n\n" + figure("delay-percentiles.png", "Ping vs flood delay percentiles, log scale (NS1 host run)"))
|
||||
|
||||
ping = next((p for p in lats if "ping" in p.get("run", "")), None)
|
||||
js1 = next((p for p in thru if p["run"] == "js-1p-20k-128-r3"), None)
|
||||
core1 = next((p for p in thru if p["run"] == "core-1p1s-50k-128"), None)
|
||||
|
||||
md = f"""**Progress report (second study)** · run `{stamp}` (UTC)
|
||||
|
||||
> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): the orchestrator (`study-on-ns1.sh`), `nats bench`, `latency.mjs` (inside LXC 510 on this hypervisor), charting (`matplotlib`), and HTML/PDF (`pandoc` + `weasyprint`). The operator laptop did **not** publish, subscribe, draw charts, or render the PDF. Traffic stayed on `vmbr1` from LXC **510** to `nats-a/b/c` (**511–513**).
|
||||
|
||||
This is a full methodology write-up plus the numbers from that on-host run. The earlier report (`nats-cluster-bench`, run `20260912T045131Z`) used the same cluster but was **orchestrated and rendered off-box**. Use this document when you need “it was all run on 138.”
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive summary
|
||||
|
||||
| Item | This NS1-host run |
|
||||
|------|-------------------|
|
||||
| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `{before.get("whoami", "marchon")}` |
|
||||
| Bench client | LXC {before.get("client_vmid", "510")} `verae-px-worker` |
|
||||
| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` |
|
||||
| Client URL | `{before.get("nats_url", "")}` |
|
||||
| Host load before | `{before.get("loadavg", "n/a")}` |
|
||||
| Host load after | `{after.get("loadavg", "n/a")}` |
|
||||
| Core 1p1s 128 B pub | {fmt_int(core1.get("pub_msgs") if core1 else None)} msgs/s |
|
||||
| JetStream 1p 128 B r=3 | {fmt_int(js1.get("pub_msgs") if js1 else None)} durable pubs/s |
|
||||
| Ping p50 / p99 | {ping.get("p50") if ping else "—"} / {ping.get("p99") if ping else "—"} |
|
||||
|
||||
Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst.
|
||||
|
||||
---
|
||||
|
||||
## 2. Where it ran (and where it did not)
|
||||
|
||||
```text
|
||||
Operator laptop ──ssh──► NS1.GEORGELAMBERT.ORG 70.88.205.138
|
||||
study-on-ns1.sh
|
||||
python3 build-ns1-study-report.py
|
||||
sudo pct exec 510 ──► nats bench / latency.mjs
|
||||
│
|
||||
▼ vmbr1
|
||||
10.10.10.21-23 :4222
|
||||
```
|
||||
|
||||
- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510).
|
||||
- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file.
|
||||
|
||||
---
|
||||
|
||||
## 3. Results (this run)
|
||||
|
||||
### Host and brokers
|
||||
|
||||
**Before**
|
||||
|
||||
{varz_table(folder / "varz-before.json")}
|
||||
|
||||
**After**
|
||||
|
||||
{varz_table(folder / "varz-after.json")}
|
||||
|
||||
nproc={before.get("nproc", "?")} · uname=`{before.get("uname", "")}`
|
||||
|
||||
### Throughput
|
||||
|
||||
{thru_table(thru)}
|
||||
|
||||
### Round-trip delay
|
||||
|
||||
{delay_table(lats)}
|
||||
|
||||
{chr(10).join(figs)}
|
||||
|
||||
---
|
||||
|
||||
{method}
|
||||
|
||||
---
|
||||
|
||||
## 6. Reproducing this study
|
||||
|
||||
On **NS1 only**:
|
||||
|
||||
```bash
|
||||
cd ~/verae-src/verae-nats-cluster
|
||||
bash scripts/study-on-ns1.sh
|
||||
```
|
||||
|
||||
The script exits if `hostname` is not NS1. Outputs land in `results/<utc>/` including `nats-cluster-bench-ns1.{{md,html,pdf}}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog.
|
||||
|
||||
Raw logs for this run: `results/{stamp}/`.
|
||||
"""
|
||||
return md
|
||||
|
||||
|
||||
def render(md_path: Path, html_path: Path, pdf_path: Path) -> None:
|
||||
css = Path(__file__).resolve().parent / "docs-print.css"
|
||||
header = html_path.with_suffix(".hdr.html")
|
||||
banner = html_path.with_suffix(".ban.html")
|
||||
css_text = css.read_text(encoding="utf-8") if css.exists() else ""
|
||||
header.write_text(f"<style>{css_text}</style>\n", encoding="utf-8")
|
||||
banner.write_text(
|
||||
'<div class="doc-banner">'
|
||||
'<nav class="site"><a href="/">zapier.georgelambert.org</a>'
|
||||
' · <a href="/index-md.html">Markdown indexes</a></nav>'
|
||||
'<div class="kicker">Verae Time × Zapier · progress report · run on NS1.GEORGELAMBERT.ORG</div>'
|
||||
"<h1>NATS cluster message speed — NS1 host study</h1>"
|
||||
'<div class="source-path">packages/zapier-decisions/reports/nats-cluster-bench-ns1.md</div>'
|
||||
"</div>\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
r = subprocess.run(
|
||||
[
|
||||
"pandoc",
|
||||
str(md_path),
|
||||
"-o",
|
||||
str(html_path),
|
||||
"--standalone",
|
||||
f"--resource-path={md_path.parent}",
|
||||
"--highlight-style=breezedark",
|
||||
"--metadata=title=NATS cluster message speed — NS1 host study",
|
||||
f"--include-in-header={header}",
|
||||
f"--include-before-body={banner}",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
header.unlink(missing_ok=True)
|
||||
banner.unlink(missing_ok=True)
|
||||
if r.returncode != 0:
|
||||
raise SystemExit(f"pandoc failed: {r.stderr[-800:]}")
|
||||
w = subprocess.run(["weasyprint", str(html_path), str(pdf_path)], capture_output=True, text=True)
|
||||
if w.returncode != 0:
|
||||
raise SystemExit(f"weasyprint failed: {w.stderr[-800:]}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
folder = Path(sys.argv[1] if len(sys.argv) > 1 else ".")
|
||||
thru, lats = load_runs(folder)
|
||||
charts(thru, lats, folder / "charts")
|
||||
md = write_markdown(folder, thru, lats)
|
||||
md_path = folder / "nats-cluster-bench-ns1.md"
|
||||
md_path.write_text(md, encoding="utf-8")
|
||||
html_path = folder / "nats-cluster-bench-ns1.html"
|
||||
pdf_path = folder / "nats-cluster-bench-ns1.pdf"
|
||||
render(md_path, html_path, pdf_path)
|
||||
print(f"wrote {md_path}")
|
||||
print(f"wrote {html_path}")
|
||||
print(f"wrote {pdf_path}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
122
packages/verae-nats-cluster/scripts/docs-print.css
Normal file
122
packages/verae-nats-cluster/scripts/docs-print.css
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
/* Colored print + screen stylesheet for zapier.georgelambert.org */
|
||||
:root {
|
||||
--ink: #171a26;
|
||||
--muted: #5b6178;
|
||||
--line: #d9dce8;
|
||||
--bg: #f4f5fb;
|
||||
--paper: #ffffff;
|
||||
--accent: #4f46e5;
|
||||
--accent-deep: #312e81;
|
||||
--accent-soft: #eef0fe;
|
||||
--ok: #047857;
|
||||
--warn: #8a5a00;
|
||||
--code-bg: #1b1f33;
|
||||
--code-fg: #e8ecff;
|
||||
}
|
||||
html { background: var(--bg); }
|
||||
body {
|
||||
margin: 0 auto;
|
||||
padding: 1.5rem 1.25rem 3rem;
|
||||
max-width: 48rem;
|
||||
font: 15px/1.55 -apple-system, "Segoe UI", Georgia, serif;
|
||||
color: var(--ink);
|
||||
background: var(--paper);
|
||||
}
|
||||
.doc-banner {
|
||||
background: linear-gradient(160deg, #312e81 0%, #4f46e5 60%, #7c74f0 100%);
|
||||
color: #eef0fe;
|
||||
margin: -1.5rem -1.25rem 1.5rem;
|
||||
padding: 1.1rem 1.25rem 1rem;
|
||||
}
|
||||
.doc-banner a { color: #fff; }
|
||||
.doc-banner .kicker {
|
||||
letter-spacing: 0.12em;
|
||||
text-transform: uppercase;
|
||||
font: 700 10px system-ui, sans-serif;
|
||||
opacity: 0.8;
|
||||
}
|
||||
.doc-banner h1 { margin: 0.25rem 0 0; font-size: 1.45rem; color: #fff; }
|
||||
h1, h2, h3, h4 { color: var(--accent-deep); page-break-after: avoid; }
|
||||
h1 { font-size: 1.7rem; }
|
||||
h2 {
|
||||
font-size: 1.2rem;
|
||||
border-bottom: 2px solid var(--accent);
|
||||
padding-bottom: 0.2rem;
|
||||
margin-top: 1.6rem;
|
||||
}
|
||||
h3 { font-size: 1.05rem; color: var(--accent); }
|
||||
a { color: var(--accent); }
|
||||
p, li { orphans: 3; widows: 3; }
|
||||
code {
|
||||
font-family: ui-monospace, Menlo, Consolas, monospace;
|
||||
font-size: 0.86em;
|
||||
background: var(--accent-soft);
|
||||
color: var(--accent-deep);
|
||||
padding: 0.08em 0.28em;
|
||||
border-radius: 4px;
|
||||
}
|
||||
pre, div.sourceCode, div.sourceCode pre {
|
||||
background: var(--code-bg) !important;
|
||||
color: var(--code-fg) !important;
|
||||
padding: 0.85rem 1rem;
|
||||
border-radius: 10px;
|
||||
overflow: auto;
|
||||
font-size: 0.78rem;
|
||||
line-height: 1.4;
|
||||
page-break-inside: avoid;
|
||||
}
|
||||
pre code { background: transparent; color: inherit; padding: 0; }
|
||||
#title-block-header, header#title-block-header, h1.title { display: none; }
|
||||
.doc-banner + h1 { display: none; }
|
||||
table {
|
||||
border-collapse: collapse;
|
||||
width: 100%;
|
||||
margin: 0.8rem 0 1.2rem;
|
||||
font-size: 0.9rem;
|
||||
page-break-inside: avoid;
|
||||
}
|
||||
th, td { border: 1px solid var(--line); padding: 0.38rem 0.55rem; text-align: left; vertical-align: top; }
|
||||
th {
|
||||
background: var(--accent);
|
||||
color: #fff;
|
||||
font: 650 12px system-ui, sans-serif;
|
||||
}
|
||||
tr:nth-child(even) td { background: var(--accent-soft); }
|
||||
blockquote {
|
||||
margin: 1rem 0;
|
||||
padding: 0.4rem 0.9rem;
|
||||
border-left: 4px solid var(--accent);
|
||||
background: var(--accent-soft);
|
||||
color: var(--accent-deep);
|
||||
}
|
||||
img { max-width: 100%; height: auto; border-radius: 8px; page-break-inside: avoid; }
|
||||
hr { border: 0; border-top: 1px solid var(--line); }
|
||||
ul, ol { padding-left: 1.25rem; }
|
||||
nav.site { font: 13px system-ui, sans-serif; margin-bottom: 0.4rem; }
|
||||
.source-path { font: 11px ui-monospace, Menlo, monospace; color: var(--muted); }
|
||||
|
||||
@page {
|
||||
size: letter;
|
||||
margin: 0.65in 0.7in 0.8in 0.7in;
|
||||
@top-left {
|
||||
content: "Verae Time × Zapier";
|
||||
font: 700 8pt system-ui, sans-serif;
|
||||
color: #4f46e5;
|
||||
}
|
||||
@top-right {
|
||||
content: "zapier.georgelambert.org";
|
||||
font: 8pt system-ui, sans-serif;
|
||||
color: #6b7186;
|
||||
}
|
||||
@bottom-center {
|
||||
content: counter(page) " / " counter(pages);
|
||||
font: 8pt system-ui, sans-serif;
|
||||
color: #6b7186;
|
||||
}
|
||||
}
|
||||
@media print {
|
||||
html, body { background: #fff; max-width: none; padding: 0; }
|
||||
.doc-banner { margin: 0 0 1rem; border-radius: 8px; -webkit-print-color-adjust: exact; print-color-adjust: exact; }
|
||||
a { text-decoration: none; }
|
||||
th, tr:nth-child(even) td, pre, blockquote, code { -webkit-print-color-adjust: exact; print-color-adjust: exact; }
|
||||
}
|
||||
202
packages/verae-nats-cluster/scripts/ns1-study-methodology.md
Normal file
202
packages/verae-nats-cluster/scripts/ns1-study-methodology.md
Normal file
|
|
@ -0,0 +1,202 @@
|
|||
## 4. Study methodology
|
||||
|
||||
### 4.1 Question
|
||||
|
||||
On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)?
|
||||
|
||||
### 4.2 Hypotheses (stated before the run)
|
||||
|
||||
1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica.
|
||||
2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated.
|
||||
3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting.
|
||||
4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count.
|
||||
5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound.
|
||||
|
||||
### 4.3 Independent variables (what we changed)
|
||||
|
||||
| Factor | Levels |
|
||||
|--------|--------|
|
||||
| Transport | Core NATS pub/sub vs JetStream file replicas=3 |
|
||||
| Publisher count | 1, 2, 4, 8 |
|
||||
| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 |
|
||||
| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) |
|
||||
| Payload | 128 B, 1024 B |
|
||||
| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) |
|
||||
|
||||
### 4.4 Dependent variables (what we recorded)
|
||||
|
||||
| Metric | Instrument | Unit |
|
||||
|--------|------------|------|
|
||||
| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s |
|
||||
| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s |
|
||||
| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) |
|
||||
| Publisher spread | nats min/avg/max **msgs/s** | not delay |
|
||||
| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max |
|
||||
| Host load | `/proc/loadavg` before and after | load average |
|
||||
| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem |
|
||||
|
||||
**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`.
|
||||
|
||||
### 4.5 Controls and constants
|
||||
|
||||
- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`.
|
||||
- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`).
|
||||
- Bench client is LXC **510**, not a nats-* server.
|
||||
- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide.
|
||||
- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`).
|
||||
- No TLS, no nkeys, no account isolation (isolation is `vmbr1`).
|
||||
- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder.
|
||||
|
||||
### 4.6 Procedure
|
||||
|
||||
1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise.
|
||||
2. Snapshot host load, memory, LXC configs, and each nats `varz`.
|
||||
3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB).
|
||||
4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file.
|
||||
5. Copy `latency.mjs` into 510; ping then flood at several batch sizes.
|
||||
6. Snapshot host/`varz` again.
|
||||
7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**.
|
||||
|
||||
No publish, subscribe, chart, or PDF process runs on the operator laptop for this study.
|
||||
|
||||
### 4.7 Instrumentation path
|
||||
|
||||
```text
|
||||
[NS1 host 70.88.205.138]
|
||||
study-on-ns1.sh (bash + python3)
|
||||
|
|
||||
| sudo pct exec 510
|
||||
v
|
||||
[LXC 510 verae-px-worker 10.10.10.20]
|
||||
nats bench / node latency.mjs
|
||||
|
|
||||
| NATS client protocol to
|
||||
v
|
||||
[LXC 511/512/513 10.10.10.21-23 :4222]
|
||||
nats-server -js cluster routes :6222
|
||||
```
|
||||
|
||||
The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`.
|
||||
|
||||
### 4.8 Threats to validity
|
||||
|
||||
| Threat | Effect on numbers |
|
||||
|--------|-------------------|
|
||||
| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. |
|
||||
| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. |
|
||||
| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. |
|
||||
| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. |
|
||||
| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. |
|
||||
| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. |
|
||||
| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. |
|
||||
| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. |
|
||||
|
||||
### 4.9 Ethics / safety
|
||||
|
||||
Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket.
|
||||
|
||||
---
|
||||
|
||||
## 5. Suggestions for fine-tuning
|
||||
|
||||
These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way.
|
||||
|
||||
### 5.1 Treat JetStream as the product limiter
|
||||
|
||||
Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out.
|
||||
|
||||
### 5.2 Split storage class by stream
|
||||
|
||||
| Stream | Suggested store | Why |
|
||||
|--------|-----------------|-----|
|
||||
| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad |
|
||||
| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both |
|
||||
| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer |
|
||||
| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry |
|
||||
| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive |
|
||||
|
||||
Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert.
|
||||
|
||||
### 5.3 Give JetStream real disks
|
||||
|
||||
Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain.
|
||||
|
||||
- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`.
|
||||
- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure.
|
||||
- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it.
|
||||
- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.”
|
||||
|
||||
### 5.4 Isolate the nats CTs from the rest of NS1
|
||||
|
||||
Host load on this box is often already several. Pin:
|
||||
|
||||
- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes.
|
||||
- Memory high enough that file-backed streams stay cache-hot for the working set.
|
||||
- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync.
|
||||
|
||||
Re-run this study after pinning; H1/H2 should move more than ping.
|
||||
|
||||
### 5.5 Consumer and mailbox tuning (delay H4)
|
||||
|
||||
Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT.
|
||||
|
||||
- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM.
|
||||
- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber.
|
||||
- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped.
|
||||
- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT.
|
||||
|
||||
### 5.6 Publisher-side batching in middleware
|
||||
|
||||
A timestamp job is one small JSON. 16k msgs/s is ample. Still:
|
||||
|
||||
- Avoid per-byte publishes; one message per job/event.
|
||||
- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs).
|
||||
- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only.
|
||||
|
||||
### 5.7 nats-server knobs worth measuring (A/B with this script)
|
||||
|
||||
| Knob | Why try it |
|
||||
|------|------------|
|
||||
| `max_payload` | Keep default unless archive puts grow |
|
||||
| `write_deadline` | Slow consumer protection for webhooks |
|
||||
| `max_pending` | Bound memory on a stuck Zapier hook |
|
||||
| `max_connections` | Fleet workers + keep + middleware |
|
||||
| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT |
|
||||
| `max_outstanding_catchup` | Replica restart after a nats-c blip |
|
||||
| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT |
|
||||
|
||||
Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99.
|
||||
|
||||
### 5.8 Network
|
||||
|
||||
- Keep NATS off `vmbr0`. No change.
|
||||
- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load).
|
||||
- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus.
|
||||
|
||||
### 5.9 Security cost (when nkeys/mTLS flip)
|
||||
|
||||
`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement.
|
||||
|
||||
### 5.10 Operational fine-tuning (lag, not peak msgs/s)
|
||||
|
||||
1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`.
|
||||
2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense.
|
||||
3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost.
|
||||
4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated.
|
||||
|
||||
### 5.11 What not to tune
|
||||
|
||||
- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge.
|
||||
- Do not treat flood 400 ms as “cluster RTT.” Fix consumers.
|
||||
- Do not load-test on `ZAPIER_*` streams.
|
||||
- Do not bind client NATS to `0.0.0.0` on `vmbr0`.
|
||||
|
||||
### 5.12 Recommended next experiments (same method, one change each)
|
||||
|
||||
1. CPU pin nats-a/b/c → re-run JS 1p + ping.
|
||||
2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder).
|
||||
3. Distinct `store_dir` disks per node.
|
||||
4. nkeys on, same ladder.
|
||||
5. Three hardware boxes, same `cluster.env` IPs updated.
|
||||
|
||||
Each experiment should produce a new `results/<utc>/` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory.
|
||||
91
packages/verae-nats-cluster/scripts/study-on-ns1.sh
Executable file
91
packages/verae-nats-cluster/scripts/study-on-ns1.sh
Executable file
|
|
@ -0,0 +1,91 @@
|
|||
#!/usr/bin/env bash
|
||||
# Full message-speed study. Must run ON NS1.GEORGELAMBERT.ORG (70.88.205.138).
|
||||
# Orchestration, nats bench (via pct into LXC 510), charts, HTML, and PDF all
|
||||
# happen on this host. The laptop is not in the measurement path.
|
||||
set -euo pipefail
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
export PATH="/usr/sbin:/usr/bin:/bin:/usr/local/bin:$PATH"
|
||||
|
||||
HOST="$(hostname -f 2>/dev/null || hostname)"
|
||||
case "$HOST" in
|
||||
NS1.GEORGELAMBERT.ORG|NS1|ns1.georgelambert.org|ns1) ;;
|
||||
*)
|
||||
echo "refusing: study-on-ns1.sh must run on NS1.GEORGELAMBERT.ORG (70.88.205.138), got '$HOST'" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
# shellcheck disable=SC1091
|
||||
. "$ROOT/client.env"
|
||||
CLIENT_VMID="${CLIENT_VMID:-510}"
|
||||
STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
OUT="${BENCH_OUT:-$ROOT/results/$STAMP}"
|
||||
mkdir -p "$OUT"
|
||||
|
||||
dump_env() {
|
||||
local tag="$1"
|
||||
local f="$OUT/host-$tag.txt"
|
||||
{
|
||||
echo "execution_host=NS1.GEORGELAMBERT.ORG"
|
||||
echo "execution_ip=70.88.205.138"
|
||||
echo "hostname=$(hostname)"
|
||||
echo "utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
echo "whoami=$(whoami)"
|
||||
echo "pwd=$(pwd)"
|
||||
echo "uname=$(uname -a)"
|
||||
echo "nproc=$(nproc)"
|
||||
echo "loadavg=$(cat /proc/loadavg)"
|
||||
echo "client_vmid=$CLIENT_VMID"
|
||||
echo "nats_url=$NATS_URL"
|
||||
echo "--- free ---"
|
||||
free -h
|
||||
echo "--- pct list ---"
|
||||
sudo pct list
|
||||
for v in 510 511 512 513; do
|
||||
echo "--- pct config $v ---"
|
||||
sudo pct config "$v" | grep -E '^(hostname|cores|memory|swap|rootfs|mp|net)' || true
|
||||
done
|
||||
} >"$f"
|
||||
python3 - "$OUT" "$tag" <<'PY'
|
||||
import json, sys, urllib.request
|
||||
from pathlib import Path
|
||||
out, tag = Path(sys.argv[1]), sys.argv[2]
|
||||
nodes = []
|
||||
for vmid, name in (("511", "nats-a"), ("512", "nats-b"), ("513", "nats-c")):
|
||||
raw = ""
|
||||
try:
|
||||
import subprocess
|
||||
raw = subprocess.check_output(
|
||||
["sudo", "pct", "exec", vmid, "--", "curl", "-fsS", "--max-time", "3", "http://127.0.0.1:8222/varz"],
|
||||
text=True,
|
||||
)
|
||||
d = json.loads(raw)
|
||||
nodes.append({
|
||||
"vmid": vmid,
|
||||
"name": name,
|
||||
"server_name": d.get("server_name"),
|
||||
"host": d.get("host"),
|
||||
"port": d.get("port"),
|
||||
"connections": d.get("connections"),
|
||||
"in_msgs": d.get("in_msgs"),
|
||||
"out_msgs": d.get("out_msgs"),
|
||||
"in_bytes": d.get("in_bytes"),
|
||||
"out_bytes": d.get("out_bytes"),
|
||||
"cpu": d.get("cpu"),
|
||||
"cores": d.get("cores"),
|
||||
"mem": d.get("mem"),
|
||||
"jetstream": bool(d.get("jetstream")),
|
||||
})
|
||||
except Exception as e:
|
||||
nodes.append({"vmid": vmid, "name": name, "error": str(e)})
|
||||
(out / f"varz-{tag}.json").write_text(json.dumps(nodes, indent=2) + "\n", encoding="utf-8")
|
||||
PY
|
||||
}
|
||||
|
||||
echo "NS1 study $STAMP out=$OUT"
|
||||
dump_env before
|
||||
BENCH_OUT="$OUT" CLIENT_VMID="$CLIENT_VMID" bash "$ROOT/scripts/bench.sh"
|
||||
dump_env after
|
||||
python3 "$ROOT/scripts/build-ns1-study-report.py" "$OUT"
|
||||
echo "NS1 study complete $OUT"
|
||||
ls -la "$OUT"/nats-cluster-bench-ns1.* "$OUT"/charts 2>/dev/null || ls -la "$OUT"
|
||||
|
|
@ -6,6 +6,7 @@ bash -n "$ROOT/scripts/lib-ct.sh"
|
|||
bash -n "$ROOT/scripts/create-cluster.sh"
|
||||
bash -n "$ROOT/scripts/status.sh"
|
||||
bash -n "$ROOT/scripts/bench.sh"
|
||||
bash -n "$ROOT/scripts/study-on-ns1.sh"
|
||||
grep -q 'host: {{IP}}' "$ROOT/conf/nats.conf.tmpl"
|
||||
grep -qv '0.0.0.0' "$ROOT/conf/nats.conf.tmpl"
|
||||
if [[ ! -d /etc/pve/nodes ]]; then
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue