Snapshot of verae-nats-cluster from zapier monorepo (NS1-host NATS study)

BENCH.md

@@ -0,0 +1,49 @@
+# NATS cluster message speed
+
+Run **`20260912T045131Z`** (UTC). Client: LXC **510** `verae-px-worker` (`10.10.10.20`), not a nats-* server. Servers: `nats-a/b/c` on `10.10.10.21–23` (`vmbr1` only).
+
+Client URL:
+
+```text
+nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+```
+
+## Method
+
+- **Core NATS** is fire-and-forget pub/sub (`nats bench`). No disk, no replica ack.
+- **JetStream** uses **file** storage and **replicas=3** (same as product streams). The unique stream `benchstream` is deleted between JS loads.
+- Throughput is **msgs/sec** from nats CLI **0.1.6** (`--no-progress --csv`). Its min/avg/max are publisher **rate spread**, not delay.
+- **Ping** delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster.
+- **Flood** delay: N publishers dump the whole batch, then the subscriber drains. This is **queueing under burst**, not wire RTT.
+- Probe: `scripts/latency.mjs` (two connections, header timestamp).
+
+## Throughput
+
+| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
+|-----|------|------------------|------------|----------|------------|----------|
+| `core-1p1s-50k-128` | core pub/sub | 1,200,836 | 791,094 | 96.57 | 747,461 | 91.24 |
+| `core-4p4s-100k-128` | core pub/sub | 1,521,256 | 316,312 | 38.61 | 1,299,634 | 158.65 |
+| `core-4p4s-50k-1k` | core pub/sub | 630,460 | 247,747 | 241.94 | 510,216 | 498.26 |
+| `core-8p8s-200k-128` | core pub/sub | 2,007,937 | 333,957 | 40.77 | 1,790,736 | 218.60 |
+| `js-1p-20k-128-r3` | jetstream r=3 file | — | 16,155 | 1.97 | — | — |
+| `js-2p2s-20k-128-r3` | jetstream r=3 file | 21,861 | 10,965 | 1.34 | 10,942 | 1.34 |
+| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 13,493 | 13.18 | — | — |
+| `js-4p-50k-128-r3` | jetstream r=3 file | — | 16,607 | 2.03 | — | — |
+
+## Round-trip delay
+
+| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
+|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|
+| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.254ms | 0.307ms | 0.286ms | 0.332ms | 0.734ms | 2.763ms |
+| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 149.279ms | 238.626ms | 248.752ms | 274.314ms | 279.398ms | 279.740ms |
+| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 155.101ms | 211.706ms | 217.579ms | 223.268ms | 227.798ms | 228.370ms |
+| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 174.174ms | 263.186ms | 266.672ms | 299.073ms | 304.233ms | 304.475ms |
+| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 304.586ms | 453.749ms | 466.296ms | 499.924ms | 505.112ms | 505.620ms |
+
+## What the numbers mean
+
+Product job/event/archive traffic is **JetStream r=3 file**. On this three-LXC stand that is about **16k durable 128 B pubs/s** (about **13k** at 1 KiB). Core NATS is an upper bound for non-durable fan-out: about **0.7–2.0M msgs/s** aggregate at 128 B, or **~630k msgs/s (~616 MB/s)** at 1 KiB with 4 publishers.
+
+A quiet request-reply is **~0.3 ms** average, **p99 < 1 ms**. Flood rows in the **150–500 ms** band are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers.
+
+Re-run on NS1: `bash scripts/bench.sh`. Raw logs/CSVs are under `results/<utc>/`.

NATS.md

@@ -0,0 +1,7 @@
+# NATS — verae-nats-cluster
+
+This repo **is** the three-node JetStream cluster. Listeners are the guest private IPs on `vmbr1` (`10.10.10.21–23:4222` and `:6222`). HTTP monitor is loopback `:8222` inside each guest.
+
+Zapier cloud, browsers, and `vmbr0` never get a NATS socket.
+
+Throughput and delay at several loads: [BENCH.md](BENCH.md).

README.md

@@ -0,0 +1,70 @@
+# verae-nats-cluster
+
+Three **distinct Proxmox LXC** guests running **NATS JetStream** as a cluster, on the private bridge only. This is the test stand before the same layout moves to separate hardware.
+
+**Forgejo:** https://git.georgelambert.org/marchon/verae-nats-cluster  
+**SSH:** `ssh://git@git.georgelambert.org:2223/marchon/verae-nats-cluster.git`
+
+| Guest | VMID | `vmbr1` IP | Client | Cluster | Monitor |
+|-------|------|------------|--------|---------|---------|
+| nats-a | 511 | 10.10.10.21 | `:4222` | `:6222` | `127.0.0.1:8222` |
+| nats-b | 512 | 10.10.10.22 | `:4222` | `:6222` | `127.0.0.1:8222` |
+| nats-c | 513 | 10.10.10.23 | `:4222` | `:6222` | `127.0.0.1:8222` |
+
+Cluster name: `verae`. Client URL:
+
+```text
+nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+```
+
+**Not** on `vmbr0`. **Not** `0.0.0.0`. Host lab NATS on `127.0.0.1:4222` is left running so current keep/fleet/middleware stay up until you cut over.
+
+```bash
+# on NS1 (the Proxmox host)
+git clone ssh://git@git.georgelambert.org:2223/marchon/verae-nats-cluster.git
+cd verae-nats-cluster
+bash scripts/create-cluster.sh
+bash scripts/status.sh
+bash scripts/test.sh
+```
+
+Test-env cut-over (keep/fleet/middleware on NS1):
+
+```bash
+# 1) create product streams (replicas=3)
+bash scripts/ensure-streams.sh
+# 2) fleet overlay nats.url + services/*.json already list the three URLs
+# 3) restart fleet and keep with NATS_URL from client.env
+```
+
+`JETSTREAM_REPLICAS=3` on middleware. Host `127.0.0.1:4222` can stay up unused. Zapier cloud never talks to NATS.
+
+nkeys/mTLS: `verae-nats-accounts` still has the INTERNAL/LEAF sketch. Do **not** enable accounts on this cluster until every client passes credentials in `NATS_URL`. Private `vmbr1` is the current isolation.
+
+## Message speed (throughput + delay)
+
+From a **client that is not a nats-* server** (LXC 510 `verae-px-worker`):
+
+```bash
+# on NS1 (hypervisor). Bench client is LXC 510.
+bash scripts/bench.sh
+
+# Full study that must run ON NS1 (orchestrator + charts + HTML/PDF):
+bash scripts/study-on-ns1.sh
+```
+
+That writes `results/<utc>/` and copies a summary to [BENCH.md](BENCH.md). Full write-up with charts (HTML + PDF) is in the progress repo: [nats-cluster-bench](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.pdf). Core NATS is fire-and-forget; JetStream rows use **file** storage and **replicas=3**. Ping delay is sequential RTT; flood delay is queueing under burst.
+
+Latest run (`20260912T045131Z`):
+
+| Kind | Load | Result |
+|------|------|--------|
+| Core 1p1s 50k×128 B | pub/sub | ~1.20M msgs/s aggregate |
+| Core 8p8s 200k×128 B | pub/sub | ~2.01M msgs/s aggregate |
+| Core 4p4s 50k×1 KiB | pub/sub | ~630k msgs/s (~616 MB/s aggregate) |
+| JetStream r=3 file | 1–4 pubs, 128 B | ~16k durable pubs/s |
+| JetStream r=3 file | 4 pubs, 1 KiB | ~13.5k durable pubs/s |
+| Ping 1k×128 B | sequential RTT | avg 0.307 ms, p99 0.734 ms |
+| Flood 8p 20k×128 B | burst queueing | avg 454 ms |
+
+Hardware move: same three configs, three boxes, private NIC only — change IPs in `cluster.env` and `conf/nats.conf.tmpl`.

SUMMARY.md

@@ -0,0 +1,5 @@
+# verae-nats-cluster
+
+Proxmox LXC 511–513 (`nats-a/b/c`) form a JetStream cluster on `vmbr1` for HA testing before dedicated hardware.
+
+Message speed (client LXC 510 vs the three nodes): see [BENCH.md](BENCH.md). Product streams are JetStream **file + replicas=3** (~16k durable 128 B pubs/s on this stand). Core NATS is ~1–2M msgs/s at 128 B. Sequential RTT ~0.3 ms; flood queueing is hundreds of ms.

client.env

@@ -0,0 +1,3 @@
+# Test-environment client URL. Private vmbr1 only. Lab loopback stays until cut-over.
+export NATS_URL="${NATS_URL:-nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222}"
+export JETSTREAM_REPLICAS="${JETSTREAM_REPLICAS:-3}"

cluster.env

@@ -0,0 +1,19 @@
+# Distinct Proxmox LXC guests on NS1 vmbr1. Not the host loopback NATS.
+BRIDGE="${BRIDGE:-vmbr1}"
+GW="${GW:-10.10.10.1}"
+MEMORY="${MEMORY:-1024}"
+CORES="${CORES:-1}"
+DISK="${DISK:-8}"
+STORAGE="${STORAGE:-SSD2}"
+TEMPLATE="${TEMPLATE:-/var/lib/vz/template/cache/ubuntu-24.04-standard_24.04-2_amd64.tar.zst}"
+DNS="${DNS:-8.8.8.8}"
+USER_NAME="${USER_NAME:-marchon}"
+NATS_VER="${NATS_VER:-2.10.26}"
+CLUSTER_NAME="${CLUSTER_NAME:-verae}"
+
+# vmid hostname ipv4
+NODES="${NODES:-
+511 nats-a 10.10.10.21
+512 nats-b 10.10.10.22
+513 nats-c 10.10.10.23
+}"

conf/nats.conf.tmpl

@@ -0,0 +1,19 @@
+# JetStream cluster node. Bind the private guest IP only — never vmbr0 / 0.0.0.0.
+server_name: {{NAME}}
+host: {{IP}}
+port: 4222
+http: 127.0.0.1:8222
+
+jetstream {
+  store_dir: /var/lib/nats/jetstream
+  max_mem: 256M
+  max_file: 4G
+}
+
+cluster {
+  name: {{CLUSTER}}
+  listen: {{IP}}:6222
+  routes: [
+{{ROUTES}}
+  ]
+}

package.json

@@ -0,0 +1,12 @@
+{
+  "name": "verae-nats-cluster",
+  "version": "0.1.0",
+  "private": true,
+  "description": "Three-node NATS JetStream cluster on distinct Proxmox LXC guests (vmbr1 only)",
+  "scripts": {
+    "create": "bash scripts/create-cluster.sh",
+    "test": "bash scripts/test.sh",
+    "status": "bash scripts/status.sh",
+    "bench": "bash scripts/bench.sh"
+  }
+}

results/.gitignore

@@ -0,0 +1,6 @@
+*
+!.gitignore
+!20260912T045131Z/
+!20260912T045131Z/**
+!20260912T051237Z/
+!20260912T051237Z/**

results/20260912T045131Z/BENCH.md

@@ -0,0 +1,49 @@
+# NATS cluster message speed
+
+Run **`20260912T045131Z`** (UTC). Client: LXC **510** `verae-px-worker` (`10.10.10.20`), not a nats-* server. Servers: `nats-a/b/c` on `10.10.10.21–23` (`vmbr1` only).
+
+Client URL:
+
+```text
+nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+```
+
+## Method
+
+- **Core NATS** is fire-and-forget pub/sub (`nats bench`). No disk, no replica ack.
+- **JetStream** uses **file** storage and **replicas=3** (same as product streams). The unique stream `benchstream` is deleted between JS loads.
+- Throughput is **msgs/sec** from nats CLI **0.1.6** (`--no-progress --csv`). Its min/avg/max are publisher **rate spread**, not delay.
+- **Ping** delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster.
+- **Flood** delay: N publishers dump the whole batch, then the subscriber drains. This is **queueing under burst**, not wire RTT.
+- Probe: `scripts/latency.mjs` (two connections, header timestamp).
+
+## Throughput
+
+| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
+|-----|------|------------------|------------|----------|------------|----------|
+| `core-1p1s-50k-128` | core pub/sub | 1,200,836 | 791,094 | 96.57 | 747,461 | 91.24 |
+| `core-4p4s-100k-128` | core pub/sub | 1,521,256 | 316,312 | 38.61 | 1,299,634 | 158.65 |
+| `core-4p4s-50k-1k` | core pub/sub | 630,460 | 247,747 | 241.94 | 510,216 | 498.26 |
+| `core-8p8s-200k-128` | core pub/sub | 2,007,937 | 333,957 | 40.77 | 1,790,736 | 218.60 |
+| `js-1p-20k-128-r3` | jetstream r=3 file | — | 16,155 | 1.97 | — | — |
+| `js-2p2s-20k-128-r3` | jetstream r=3 file | 21,861 | 10,965 | 1.34 | 10,942 | 1.34 |
+| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 13,493 | 13.18 | — | — |
+| `js-4p-50k-128-r3` | jetstream r=3 file | — | 16,607 | 2.03 | — | — |
+
+## Round-trip delay
+
+| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
+|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|
+| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.254ms | 0.307ms | 0.286ms | 0.332ms | 0.734ms | 2.763ms |
+| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 149.279ms | 238.626ms | 248.752ms | 274.314ms | 279.398ms | 279.740ms |
+| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 155.101ms | 211.706ms | 217.579ms | 223.268ms | 227.798ms | 228.370ms |
+| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 174.174ms | 263.186ms | 266.672ms | 299.073ms | 304.233ms | 304.475ms |
+| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 304.586ms | 453.749ms | 466.296ms | 499.924ms | 505.112ms | 505.620ms |
+
+## What the numbers mean
+
+Product job/event/archive traffic is **JetStream r=3 file**. On this three-LXC stand that is about **16k durable 128 B pubs/s** (about **13k** at 1 KiB). Core NATS is an upper bound for non-durable fan-out: about **0.7–2.0M msgs/s** aggregate at 128 B, or **~630k msgs/s (~616 MB/s)** at 1 KiB with 4 publishers.
+
+A quiet request-reply is **~0.3 ms** average, **p99 < 1 ms**. Flood rows in the **150–500 ms** band are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers.
+
+Re-run on NS1: `bash scripts/bench.sh`. Raw logs/CSVs are under `results/<utc>/`.

results/20260912T045131Z/core-1p1s-50k-128.csv

@@ -0,0 +1,3 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+8e1ExxrZ9OtHTkwXsOfzes,S0,50000,6400000,747461,95675134.125326,0.066893
+8e1ExxrZ9OtHTkwXsOfzes,P0,50000,6400000,791094,101260059.613379,0.063204

results/20260912T045131Z/core-1p1s-50k-128.txt

@@ -0,0 +1,7 @@
+=== core-1p1s-50k-128 ===
+
+NATS Pub/Sub stats: 1,200,836 msgs/sec ~ 146.59 MB/sec
+ Pub stats: 791,094 msgs/sec ~ 96.57 MB/sec
+ Sub stats: 747,461 msgs/sec ~ 91.24 MB/sec
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T045131Z/core-4p4s-100k-128.csv

@@ -0,0 +1,9 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+d1csC8CbCBLYlUMAUVZxpb,S0,100000,12800000,377317,48296660.364795,0.265029
+d1csC8CbCBLYlUMAUVZxpb,S1,100000,12800000,351445,44985056.104940,0.284539
+d1csC8CbCBLYlUMAUVZxpb,S2,100000,12800000,372973,47740617.289389,0.268116
+d1csC8CbCBLYlUMAUVZxpb,S3,100000,12800000,324908,41588310.877432,0.307779
+d1csC8CbCBLYlUMAUVZxpb,P0,25000,3200000,524453,67129988.054428,0.047669
+d1csC8CbCBLYlUMAUVZxpb,P1,25000,3200000,324337,41515233.366208,0.077080
+d1csC8CbCBLYlUMAUVZxpb,P2,25000,3200000,103170,13205778.771206,0.242318
+d1csC8CbCBLYlUMAUVZxpb,P3,25000,3200000,79260,10145297.784464,0.315417

results/20260912T045131Z/core-4p4s-100k-128.txt

@@ -0,0 +1,17 @@
+=== core-4p4s-100k-128 ===
+
+NATS Pub/Sub stats: 1,521,256 msgs/sec ~ 185.70 MB/sec
+ Pub stats: 316,312 msgs/sec ~ 38.61 MB/sec
+  [1] 524,453 msgs/sec ~ 64.02 MB/sec (25000 msgs)
+  [2] 324,337 msgs/sec ~ 39.59 MB/sec (25000 msgs)
+  [3] 103,170 msgs/sec ~ 12.59 MB/sec (25000 msgs)
+  [4] 79,260 msgs/sec ~ 9.68 MB/sec (25000 msgs)
+  min 79,260 | avg 257,805 | max 524,453 | stddev 181,189 msgs
+ Sub stats: 1,299,634 msgs/sec ~ 158.65 MB/sec
+  [1] 377,317 msgs/sec ~ 46.06 MB/sec (100000 msgs)
+  [2] 351,445 msgs/sec ~ 42.90 MB/sec (100000 msgs)
+  [3] 372,973 msgs/sec ~ 45.53 MB/sec (100000 msgs)
+  [4] 324,908 msgs/sec ~ 39.66 MB/sec (100000 msgs)
+  min 324,908 | avg 356,660 | max 377,317 | stddev 20,785 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T045131Z/core-4p4s-50k-1k.csv

@@ -0,0 +1,9 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+LXVQ7kvd26BIwiMRRfbr7Z,S0,50000,51200000,169213,173274762.582948,0.295484
+LXVQ7kvd26BIwiMRRfbr7Z,S1,50000,51200000,150273,153880103.181538,0.332727
+LXVQ7kvd26BIwiMRRfbr7Z,S2,50000,51200000,142138,145550108.197459,0.351769
+LXVQ7kvd26BIwiMRRfbr7Z,S3,50000,51200000,132134,135306220.651698,0.378401
+LXVQ7kvd26BIwiMRRfbr7Z,P0,12500,12800000,176262,180492467.210335,0.070917
+LXVQ7kvd26BIwiMRRfbr7Z,P1,12500,12800000,132211,135384883.941361,0.094545
+LXVQ7kvd26BIwiMRRfbr7Z,P2,12500,12800000,70678,72375041.084851,0.176857
+LXVQ7kvd26BIwiMRRfbr7Z,P3,12500,12800000,61936,63423297.033934,0.201819

results/20260912T045131Z/core-4p4s-50k-1k.txt

@@ -0,0 +1,17 @@
+=== core-4p4s-50k-1k ===
+
+NATS Pub/Sub stats: 630,460 msgs/sec ~ 615.68 MB/sec
+ Pub stats: 247,747 msgs/sec ~ 241.94 MB/sec
+  [1] 176,262 msgs/sec ~ 172.13 MB/sec (12500 msgs)
+  [2] 132,211 msgs/sec ~ 129.11 MB/sec (12500 msgs)
+  [3] 70,678 msgs/sec ~ 69.02 MB/sec (12500 msgs)
+  [4] 61,936 msgs/sec ~ 60.49 MB/sec (12500 msgs)
+  min 61,936 | avg 110,271 | max 176,262 | stddev 46,744 msgs
+ Sub stats: 510,216 msgs/sec ~ 498.26 MB/sec
+  [1] 169,213 msgs/sec ~ 165.25 MB/sec (50000 msgs)
+  [2] 150,273 msgs/sec ~ 146.75 MB/sec (50000 msgs)
+  [3] 142,138 msgs/sec ~ 138.81 MB/sec (50000 msgs)
+  [4] 132,134 msgs/sec ~ 129.04 MB/sec (50000 msgs)
+  min 132,134 | avg 148,439 | max 169,213 | stddev 13,605 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T045131Z/core-8p8s-200k-128.csv

@@ -0,0 +1,17 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+HuvulruPjjaqnoHWeFCH2I,S0,200000,25600000,349350,44716843.071789,0.572491
+HuvulruPjjaqnoHWeFCH2I,S1,200000,25600000,248162,31764858.924451,0.805922
+HuvulruPjjaqnoHWeFCH2I,S2,200000,25600000,244206,31258460.045739,0.818978
+HuvulruPjjaqnoHWeFCH2I,S3,200000,25600000,243405,31155913.818220,0.821674
+HuvulruPjjaqnoHWeFCH2I,S4,200000,25600000,230550,29510438.641581,0.867490
+HuvulruPjjaqnoHWeFCH2I,S5,200000,25600000,231495,29631384.363172,0.863949
+HuvulruPjjaqnoHWeFCH2I,S6,200000,25600000,229528,29379679.692357,0.871351
+HuvulruPjjaqnoHWeFCH2I,S7,200000,25600000,223842,28651789.197519,0.893487
+HuvulruPjjaqnoHWeFCH2I,P0,25000,3200000,152536,19524693.902374,0.163895
+HuvulruPjjaqnoHWeFCH2I,P1,25000,3200000,84526,10819333.375159,0.295767
+HuvulruPjjaqnoHWeFCH2I,P2,25000,3200000,82017,10498223.303653,0.304813
+HuvulruPjjaqnoHWeFCH2I,P3,25000,3200000,57214,7323427.896669,0.436954
+HuvulruPjjaqnoHWeFCH2I,P4,25000,3200000,53879,6896566.453658,0.463999
+HuvulruPjjaqnoHWeFCH2I,P5,25000,3200000,51420,6581852.801766,0.486185
+HuvulruPjjaqnoHWeFCH2I,P6,25000,3200000,48572,6217303.262856,0.514693
+HuvulruPjjaqnoHWeFCH2I,P7,25000,3200000,41744,5343333.685114,0.598877

results/20260912T045131Z/core-8p8s-200k-128.txt

@@ -0,0 +1,25 @@
+=== core-8p8s-200k-128 ===
+
+NATS Pub/Sub stats: 2,007,937 msgs/sec ~ 245.11 MB/sec
+ Pub stats: 333,957 msgs/sec ~ 40.77 MB/sec
+  [1] 152,536 msgs/sec ~ 18.62 MB/sec (25000 msgs)
+  [2] 84,526 msgs/sec ~ 10.32 MB/sec (25000 msgs)
+  [3] 82,017 msgs/sec ~ 10.01 MB/sec (25000 msgs)
+  [4] 57,214 msgs/sec ~ 6.98 MB/sec (25000 msgs)
+  [5] 53,879 msgs/sec ~ 6.58 MB/sec (25000 msgs)
+  [6] 51,420 msgs/sec ~ 6.28 MB/sec (25000 msgs)
+  [7] 48,572 msgs/sec ~ 5.93 MB/sec (25000 msgs)
+  [8] 41,744 msgs/sec ~ 5.10 MB/sec (25000 msgs)
+  min 41,744 | avg 71,488 | max 152,536 | stddev 33,867 msgs
+ Sub stats: 1,790,736 msgs/sec ~ 218.60 MB/sec
+  [1] 349,350 msgs/sec ~ 42.65 MB/sec (200000 msgs)
+  [2] 248,162 msgs/sec ~ 30.29 MB/sec (200000 msgs)
+  [3] 244,206 msgs/sec ~ 29.81 MB/sec (200000 msgs)
+  [4] 243,405 msgs/sec ~ 29.71 MB/sec (200000 msgs)
+  [5] 230,550 msgs/sec ~ 28.14 MB/sec (200000 msgs)
+  [6] 231,495 msgs/sec ~ 28.26 MB/sec (200000 msgs)
+  [7] 229,528 msgs/sec ~ 28.02 MB/sec (200000 msgs)
+  [8] 223,842 msgs/sec ~ 27.32 MB/sec (200000 msgs)
+  min 223,842 | avg 250,067 | max 349,350 | stddev 38,364 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T045131Z/js-1p-20k-128-r3.csv

@@ -0,0 +1,2 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+S0xQOJuOM4PlwAHoh5Bj7f,P0,40000,2560000,16155,2067883.858001,1.237981

results/20260912T045131Z/js-1p-20k-128-r3.txt

@@ -0,0 +1,5 @@
+=== js-1p-20k-128-r3 ===
+
+Pub stats: 16,155 msgs/sec ~ 1.97 MB/sec
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T045131Z/js-2p2s-20k-128-r3.csv

@@ -0,0 +1,5 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+smN9zLbWAAZER4FjSmfwcO,S0,20126,1280000,5474,700679.773679,1.826797
+smN9zLbWAAZER4FjSmfwcO,S1,20130,1280000,5471,700313.845376,1.827752
+smN9zLbWAAZER4FjSmfwcO,P0,20000,1280000,8678,1110910.588857,1.152208
+smN9zLbWAAZER4FjSmfwcO,P1,20000,1280000,5485,702119.773868,1.823051

results/20260912T045131Z/js-2p2s-20k-128-r3.txt

@@ -0,0 +1,13 @@
+=== js-2p2s-20k-128-r3 ===
+
+NATS Pub/Sub stats: 21,861 msgs/sec ~ 2.67 MB/sec
+ Pub stats: 10,965 msgs/sec ~ 1.34 MB/sec
+  [1] 8,678 msgs/sec ~ 1.06 MB/sec (10000 msgs)
+  [2] 5,485 msgs/sec ~ 685.66 KB/sec (10000 msgs)
+  min 5,485 | avg 7,081 | max 8,678 | stddev 1,596 msgs
+ Sub stats: 10,942 msgs/sec ~ 1.34 MB/sec
+  [1] 5,474 msgs/sec ~ 684.26 KB/sec (10000 msgs)
+  [2] 5,471 msgs/sec ~ 683.90 KB/sec (10000 msgs)
+  min 5,471 | avg 5,472 | max 5,474 | stddev 1 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T045131Z/js-4p-20k-1k-r3.csv

@@ -0,0 +1,5 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+zOcPILhowCWfp2PGmHUgqx,P0,10000,5120000,5121,5244574.853257,0.976247
+zOcPILhowCWfp2PGmHUgqx,P1,10000,5120000,4281,4383926.080211,1.167903
+zOcPILhowCWfp2PGmHUgqx,P2,10000,5120000,3480,3564282.061583,1.436474
+zOcPILhowCWfp2PGmHUgqx,P3,10000,5120000,3373,3454449.904903,1.482146

results/20260912T045131Z/js-4p-20k-1k-r3.txt

@@ -0,0 +1,10 @@
+=== js-4p-20k-1k-r3 ===
+
+Pub stats: 13,493 msgs/sec ~ 13.18 MB/sec
+ [1] 5,121 msgs/sec ~ 5.00 MB/sec (5000 msgs)
+ [2] 4,281 msgs/sec ~ 4.18 MB/sec (5000 msgs)
+ [3] 3,480 msgs/sec ~ 3.40 MB/sec (5000 msgs)
+ [4] 3,373 msgs/sec ~ 3.29 MB/sec (5000 msgs)
+ min 3,373 | avg 4,063 | max 5,121 | stddev 704 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T045131Z/js-4p-50k-128-r3.csv

@@ -0,0 +1,5 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+vZIFpR751J0vbDTJQDYGTN,P0,25000,1600000,6628,848387.283433,1.885931
+vZIFpR751J0vbDTJQDYGTN,P1,25000,1600000,5762,737571.570534,2.169281
+vZIFpR751J0vbDTJQDYGTN,P2,25000,1600000,4162,532825.280344,3.002861
+vZIFpR751J0vbDTJQDYGTN,P3,25000,1600000,4154,531740.716359,3.008985

results/20260912T045131Z/js-4p-50k-128-r3.txt

@@ -0,0 +1,10 @@
+=== js-4p-50k-128-r3 ===
+
+Pub stats: 16,607 msgs/sec ~ 2.03 MB/sec
+ [1] 6,628 msgs/sec ~ 828.50 KB/sec (12500 msgs)
+ [2] 5,762 msgs/sec ~ 720.28 KB/sec (12500 msgs)
+ [3] 4,162 msgs/sec ~ 520.34 KB/sec (12500 msgs)
+ [4] 4,154 msgs/sec ~ 519.28 KB/sec (12500 msgs)
+ min 4,154 | avg 5,176 | max 6,628 | stddev 1,063 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T045131Z/lat-1p-5k-128.txt

@@ -0,0 +1,2 @@
+=== lat-1p-5k-128 ===
+{"count":5000,"pubs":1,"size":128,"min_us":149279,"avg_us":238626,"p50_us":248752,"p90_us":274314,"p99_us":279398,"max_us":279740,"min":"149.279ms","avg":"238.626ms","p50":"248.752ms","p90":"274.314ms","p99":"279.398ms","max":"279.740ms"}

results/20260912T045131Z/lat-4p-10k-128.txt

@@ -0,0 +1,2 @@
+=== lat-4p-10k-128 ===
+{"count":10000,"pubs":4,"size":128,"min_us":174174,"avg_us":263186,"p50_us":266672,"p90_us":299073,"p99_us":304233,"max_us":304475,"min":"174.174ms","avg":"263.186ms","p50":"266.672ms","p90":"299.073ms","p99":"304.233ms","max":"304.475ms"}

results/20260912T045131Z/lat-4p-5k-1k.txt

@@ -0,0 +1,2 @@
+=== lat-4p-5k-1k ===
+{"count":5000,"pubs":4,"size":1024,"min_us":155101,"avg_us":211706,"p50_us":217579,"p90_us":223268,"p99_us":227798,"max_us":228370,"min":"155.101ms","avg":"211.706ms","p50":"217.579ms","p90":"223.268ms","p99":"227.798ms","max":"228.370ms"}

results/20260912T045131Z/lat-8p-20k-128.txt

@@ -0,0 +1,2 @@
+=== lat-8p-20k-128 ===
+{"count":20000,"pubs":8,"size":128,"min_us":304586,"avg_us":453749,"p50_us":466296,"p90_us":499924,"p99_us":505112,"max_us":505620,"min":"304.586ms","avg":"453.749ms","p50":"466.296ms","p90":"499.924ms","p99":"505.112ms","max":"505.620ms"}

results/20260912T045131Z/lat-ping-1k-128.txt

@@ -0,0 +1,2 @@
+=== lat-ping-1k-128 ===
+{"count":1000,"pubs":1,"size":128,"min_us":254,"avg_us":307,"p50_us":286,"p90_us":332,"p99_us":734,"max_us":2763,"min":"0.254ms","avg":"0.307ms","p50":"0.286ms","p90":"0.332ms","p99":"0.734ms","max":"2.763ms"}

results/20260912T051237Z/BENCH.md

@@ -0,0 +1,48 @@
+# NATS cluster message speed
+
+Run **`20260912T051237Z`** (UTC). Client: LXC **510** `verae-px-worker` (`10.10.10.20`), not a nats-* server. Servers: `nats-a/b/c` on `10.10.10.21–23` (`vmbr1` only).
+
+Client URL:
+
+```text
+nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+```
+
+## Method
+
+- **Core NATS** is fire-and-forget pub/sub (`nats bench`). No disk, no replica ack.
+- **JetStream** uses **file** storage and **replicas=3** (same as product streams). The unique stream `benchstream` is deleted between JS loads.
+- Throughput is **msgs/sec** from nats CLI **0.1.6** (`--no-progress --csv`). Its min/avg/max are publisher **rate spread**, not delay.
+- **Ping** delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster.
+- **Flood** delay: N publishers dump the whole batch, then the subscriber drains. This is **queueing under burst**, not wire RTT.
+- Probe: `scripts/latency.mjs` (two connections, header timestamp).
+
+## Throughput
+
+| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
+|-----|------|------------------|------------|----------|------------|----------|
+| `core-1p1s-50k-128` | core pub/sub | 776,331 | 502,502 | 61.34 | 420,976 | 51.39 |
+| `core-4p4s-100k-128` | core pub/sub | 1,278,454 | 379,985 | 46.38 | 1,078,910 | 131.70 |
+| `core-4p4s-50k-1k` | core pub/sub | 581,464 | 209,545 | 204.63 | 482,645 | 471.33 |
+| `core-8p8s-200k-128` | core pub/sub | 2,065,217 | 333,731 | 40.74 | 1,847,242 | 225.49 |
+| `js-2p2s-20k-128-r3` | jetstream r=3 file | 15,416 | — | — | — | — |
+| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 14,985 | 14.63 | — | — |
+| `js-4p-50k-128-r3` | jetstream r=3 file | — | 17,986 | 2.20 | — | — |
+
+## Round-trip delay
+
+| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
+|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|
+| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.341ms | 0.470ms | 0.395ms | 0.651ms | 1.377ms | 3.243ms |
+| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 132.425ms | 203.717ms | 204.458ms | 241.750ms | 249.468ms | 249.640ms |
+| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 196.422ms | 221.264ms | 224.039ms | 233.180ms | 236.798ms | 238.177ms |
+| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 211.108ms | 294.925ms | 300.254ms | 314.223ms | 315.551ms | 316.091ms |
+| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 280.608ms | 414.972ms | 409.802ms | 513.821ms | 535.394ms | 536.084ms |
+
+## What the numbers mean
+
+Product job/event/archive traffic is **JetStream r=3 file**. On this three-LXC stand that is about **16k durable 128 B pubs/s** (about **13k** at 1 KiB). Core NATS is an upper bound for non-durable fan-out: about **0.7–2.0M msgs/s** aggregate at 128 B, or **~630k msgs/s (~616 MB/s)** at 1 KiB with 4 publishers.
+
+A quiet request-reply is **~0.3 ms** average, **p99 < 1 ms**. Flood rows in the **150–500 ms** band are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers.
+
+Re-run on NS1: `bash scripts/bench.sh`. Raw logs/CSVs are under `results/<utc>/`.

results/20260912T051237Z/core-1p1s-50k-128.csv

@@ -0,0 +1,3 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+h0r77ozVkEaFq9NntdbNXL,S0,50000,6400000,420976,53884945.437030,0.118772
+h0r77ozVkEaFq9NntdbNXL,P0,50000,6400000,502502,64320317.108811,0.099502

results/20260912T051237Z/core-1p1s-50k-128.txt

@@ -0,0 +1,7 @@
+=== core-1p1s-50k-128 ===
+
+NATS Pub/Sub stats: 776,331 msgs/sec ~ 94.77 MB/sec
+ Pub stats: 502,502 msgs/sec ~ 61.34 MB/sec
+ Sub stats: 420,976 msgs/sec ~ 51.39 MB/sec
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T051237Z/core-4p4s-100k-128.csv

@@ -0,0 +1,9 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+9lnpijWl8uftGl7cjgjfDG,S0,100000,12800000,338319,43304923.645655,0.295578
+9lnpijWl8uftGl7cjgjfDG,S1,100000,12800000,328136,42001487.567998,0.304751
+9lnpijWl8uftGl7cjgjfDG,S2,100000,12800000,319449,40889502.964037,0.313039
+9lnpijWl8uftGl7cjgjfDG,S3,100000,12800000,269727,34525150.333968,0.370744
+9lnpijWl8uftGl7cjgjfDG,P0,25000,3200000,184132,23569014.120587,0.135771
+9lnpijWl8uftGl7cjgjfDG,P1,25000,3200000,183847,23532419.588601,0.135983
+9lnpijWl8uftGl7cjgjfDG,P2,25000,3200000,100400,12851250.530089,0.249003
+9lnpijWl8uftGl7cjgjfDG,P3,25000,3200000,95281,12196082.439191,0.262379

results/20260912T051237Z/core-4p4s-100k-128.txt

@@ -0,0 +1,17 @@
+=== core-4p4s-100k-128 ===
+
+NATS Pub/Sub stats: 1,278,454 msgs/sec ~ 156.06 MB/sec
+ Pub stats: 379,985 msgs/sec ~ 46.38 MB/sec
+  [1] 184,132 msgs/sec ~ 22.48 MB/sec (25000 msgs)
+  [2] 183,847 msgs/sec ~ 22.44 MB/sec (25000 msgs)
+  [3] 100,400 msgs/sec ~ 12.26 MB/sec (25000 msgs)
+  [4] 95,281 msgs/sec ~ 11.63 MB/sec (25000 msgs)
+  min 95,281 | avg 140,915 | max 184,132 | stddev 43,112 msgs
+ Sub stats: 1,078,910 msgs/sec ~ 131.70 MB/sec
+  [1] 338,319 msgs/sec ~ 41.30 MB/sec (100000 msgs)
+  [2] 328,136 msgs/sec ~ 40.06 MB/sec (100000 msgs)
+  [3] 319,449 msgs/sec ~ 39.00 MB/sec (100000 msgs)
+  [4] 269,727 msgs/sec ~ 32.93 MB/sec (100000 msgs)
+  min 269,727 | avg 313,907 | max 338,319 | stddev 26,367 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T051237Z/core-4p4s-50k-1k.csv

@@ -0,0 +1,9 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+TBolrRDh6R5ucnsIDRic4k,S0,50000,51200000,152508,156168868.728545,0.327850
+TBolrRDh6R5ucnsIDRic4k,S1,50000,51200000,143666,147114019.174715,0.348029
+TBolrRDh6R5ucnsIDRic4k,S2,50000,51200000,125714,128731399.676746,0.397727
+TBolrRDh6R5ucnsIDRic4k,S3,50000,51200000,123195,126152033.863128,0.405859
+TBolrRDh6R5ucnsIDRic4k,P0,12500,12800000,85005,87045293.746584,0.147050
+TBolrRDh6R5ucnsIDRic4k,P1,12500,12800000,84991,87030908.333567,0.147074
+TBolrRDh6R5ucnsIDRic4k,P2,12500,12800000,54739,56053364.168990,0.228354
+TBolrRDh6R5ucnsIDRic4k,P3,12500,12800000,52467,53726394.065295,0.238244

results/20260912T051237Z/core-4p4s-50k-1k.txt

@@ -0,0 +1,17 @@
+=== core-4p4s-50k-1k ===
+
+NATS Pub/Sub stats: 581,464 msgs/sec ~ 567.84 MB/sec
+ Pub stats: 209,545 msgs/sec ~ 204.63 MB/sec
+  [1] 85,005 msgs/sec ~ 83.01 MB/sec (12500 msgs)
+  [2] 84,991 msgs/sec ~ 83.00 MB/sec (12500 msgs)
+  [3] 54,739 msgs/sec ~ 53.46 MB/sec (12500 msgs)
+  [4] 52,467 msgs/sec ~ 51.24 MB/sec (12500 msgs)
+  min 52,467 | avg 69,300 | max 85,005 | stddev 15,718 msgs
+ Sub stats: 482,645 msgs/sec ~ 471.33 MB/sec
+  [1] 152,508 msgs/sec ~ 148.93 MB/sec (50000 msgs)
+  [2] 143,666 msgs/sec ~ 140.30 MB/sec (50000 msgs)
+  [3] 125,714 msgs/sec ~ 122.77 MB/sec (50000 msgs)
+  [4] 123,195 msgs/sec ~ 120.31 MB/sec (50000 msgs)
+  min 123,195 | avg 136,270 | max 152,508 | stddev 12,255 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T051237Z/core-8p8s-200k-128.csv

@@ -0,0 +1,17 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+1wXmYbQfeZKgGIuRNRJWE9,S0,200000,25600000,294783,37732336.345174,0.678463
+1wXmYbQfeZKgGIuRNRJWE9,S1,200000,25600000,285956,36602476.046553,0.699406
+1wXmYbQfeZKgGIuRNRJWE9,S2,200000,25600000,270995,34687374.508938,0.738021
+1wXmYbQfeZKgGIuRNRJWE9,S3,200000,25600000,268481,34365602.744388,0.744931
+1wXmYbQfeZKgGIuRNRJWE9,S4,200000,25600000,242749,31071912.954289,0.823895
+1wXmYbQfeZKgGIuRNRJWE9,S5,200000,25600000,243757,31200898.690210,0.820489
+1wXmYbQfeZKgGIuRNRJWE9,S6,200000,25600000,232306,29735268.576473,0.860931
+1wXmYbQfeZKgGIuRNRJWE9,S7,200000,25600000,231301,29606645.928715,0.864671
+1wXmYbQfeZKgGIuRNRJWE9,P0,25000,3200000,161768,20706321.217329,0.154542
+1wXmYbQfeZKgGIuRNRJWE9,P1,25000,3200000,156790,20069204.386621,0.159448
+1wXmYbQfeZKgGIuRNRJWE9,P2,25000,3200000,101477,12989116.865479,0.246360
+1wXmYbQfeZKgGIuRNRJWE9,P3,25000,3200000,96840,12395608.110541,0.258156
+1wXmYbQfeZKgGIuRNRJWE9,P4,25000,3200000,48401,6195444.884723,0.516509
+1wXmYbQfeZKgGIuRNRJWE9,P5,25000,3200000,56695,7256990.291363,0.440954
+1wXmYbQfeZKgGIuRNRJWE9,P6,25000,3200000,42589,5451476.161959,0.586997
+1wXmYbQfeZKgGIuRNRJWE9,P7,25000,3200000,41987,5374439.609281,0.595411

results/20260912T051237Z/core-8p8s-200k-128.txt

@@ -0,0 +1,25 @@
+=== core-8p8s-200k-128 ===
+
+NATS Pub/Sub stats: 2,065,217 msgs/sec ~ 252.10 MB/sec
+ Pub stats: 333,731 msgs/sec ~ 40.74 MB/sec
+  [1] 161,768 msgs/sec ~ 19.75 MB/sec (25000 msgs)
+  [2] 156,790 msgs/sec ~ 19.14 MB/sec (25000 msgs)
+  [3] 101,477 msgs/sec ~ 12.39 MB/sec (25000 msgs)
+  [4] 96,840 msgs/sec ~ 11.82 MB/sec (25000 msgs)
+  [5] 48,401 msgs/sec ~ 5.91 MB/sec (25000 msgs)
+  [6] 56,695 msgs/sec ~ 6.92 MB/sec (25000 msgs)
+  [7] 42,589 msgs/sec ~ 5.20 MB/sec (25000 msgs)
+  [8] 41,987 msgs/sec ~ 5.13 MB/sec (25000 msgs)
+  min 41,987 | avg 88,318 | max 161,768 | stddev 46,314 msgs
+ Sub stats: 1,847,242 msgs/sec ~ 225.49 MB/sec
+  [1] 294,783 msgs/sec ~ 35.98 MB/sec (200000 msgs)
+  [2] 285,956 msgs/sec ~ 34.91 MB/sec (200000 msgs)
+  [3] 270,995 msgs/sec ~ 33.08 MB/sec (200000 msgs)
+  [4] 268,481 msgs/sec ~ 32.77 MB/sec (200000 msgs)
+  [5] 242,749 msgs/sec ~ 29.63 MB/sec (200000 msgs)
+  [6] 243,757 msgs/sec ~ 29.76 MB/sec (200000 msgs)
+  [7] 232,306 msgs/sec ~ 28.36 MB/sec (200000 msgs)
+  [8] 231,301 msgs/sec ~ 28.24 MB/sec (200000 msgs)
+  min 231,301 | avg 258,791 | max 294,783 | stddev 22,958 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T051237Z/host-after.txt

@@ -0,0 +1,86 @@
+execution_host=NS1.GEORGELAMBERT.ORG
+execution_ip=70.88.205.138
+hostname=NS1.GEORGELAMBERT.ORG
+utc=2026-09-12T05:14:00Z
+whoami=marchon
+pwd=/home/marchon/verae-src/verae-nats-cluster
+uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux
+nproc=40
+loadavg=8.67 8.33 8.15 9/3863 4114865
+client_vmid=510
+nats_url=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+--- free ---
+               total        used        free      shared  buff/cache   available
+Mem:           377Gi        38Gi       324Gi       4.3Gi        21Gi       339Gi
+Swap:          8.0Gi          0B       8.0Gi
+--- pct list ---
+VMID       Status     Lock         Name                
+100        stopped                 CT100               
+101        stopped                 CT101               
+102        stopped                 CT102               
+103        stopped                 CT103               
+104        stopped                 CT104               
+105        stopped                 CT105               
+106        stopped                 CT106               
+107        stopped                 CT107               
+108        stopped                 CT108               
+109        stopped                 CT109               
+110        stopped                 CT110               
+111        stopped                 CT111               
+112        stopped                 CT112               
+113        stopped                 CT113               
+114        stopped                 CT114               
+115        stopped                 CT115               
+116        stopped                 paul                
+117        stopped                 CT117               
+118        stopped                 CT118               
+119        stopped                 CT119               
+120        stopped                 rustdeskserver      
+121        stopped                 CT121               
+122        stopped                 CT122               
+123        stopped                 CT123               
+124        stopped                 bdross              
+200        stopped                 CT200               
+201        stopped                 CT201               
+202        stopped                 CT202               
+300        running                 mail.veraetime.info 
+400        stopped                 evercycle-api       
+501        running                 pki                 
+502        running                 peergos-a           
+503        running                 peergos-b           
+504        stopped                 minio               
+510        running                 verae-px-worker     
+511        running                 nats-a              
+512        running                 nats-b              
+513        running                 nats-c              
+2000       stopped                 CT2000              
+2001       stopped                 CT2001              
+2002       stopped                 CT2002              
+--- pct config 510 ---
+cores: 2
+hostname: verae-px-worker
+memory: 2048
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:BD:5D:62,ip=10.10.10.20/24,type=veth
+rootfs: SSD2:subvol-510-disk-0,size=16G
+swap: 512
+--- pct config 511 ---
+cores: 1
+hostname: nats-a
+memory: 1024
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:CF:AC:68,ip=10.10.10.21/24,type=veth
+rootfs: SSD2:subvol-511-disk-0,size=8G
+swap: 256
+--- pct config 512 ---
+cores: 1
+hostname: nats-b
+memory: 1024
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:42:73:A2,ip=10.10.10.22/24,type=veth
+rootfs: SSD2:subvol-512-disk-0,size=8G
+swap: 256
+--- pct config 513 ---
+cores: 1
+hostname: nats-c
+memory: 1024
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:1B:9B:DC,ip=10.10.10.23/24,type=veth
+rootfs: SSD2:subvol-513-disk-0,size=8G
+swap: 256

results/20260912T051237Z/host-before.txt

@@ -0,0 +1,86 @@
+execution_host=NS1.GEORGELAMBERT.ORG
+execution_ip=70.88.205.138
+hostname=NS1.GEORGELAMBERT.ORG
+utc=2026-09-12T05:12:37Z
+whoami=marchon
+pwd=/home/marchon/verae-src/verae-nats-cluster
+uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux
+nproc=40
+loadavg=8.59 8.39 8.15 5/3843 4096277
+client_vmid=510
+nats_url=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+--- free ---
+               total        used        free      shared  buff/cache   available
+Mem:           377Gi        38Gi       324Gi       4.2Gi        21Gi       339Gi
+Swap:          8.0Gi          0B       8.0Gi
+--- pct list ---
+VMID       Status     Lock         Name                
+100        stopped                 CT100               
+101        stopped                 CT101               
+102        stopped                 CT102               
+103        stopped                 CT103               
+104        stopped                 CT104               
+105        stopped                 CT105               
+106        stopped                 CT106               
+107        stopped                 CT107               
+108        stopped                 CT108               
+109        stopped                 CT109               
+110        stopped                 CT110               
+111        stopped                 CT111               
+112        stopped                 CT112               
+113        stopped                 CT113               
+114        stopped                 CT114               
+115        stopped                 CT115               
+116        stopped                 paul                
+117        stopped                 CT117               
+118        stopped                 CT118               
+119        stopped                 CT119               
+120        stopped                 rustdeskserver      
+121        stopped                 CT121               
+122        stopped                 CT122               
+123        stopped                 CT123               
+124        stopped                 bdross              
+200        stopped                 CT200               
+201        stopped                 CT201               
+202        stopped                 CT202               
+300        running                 mail.veraetime.info 
+400        stopped                 evercycle-api       
+501        running                 pki                 
+502        running                 peergos-a           
+503        running                 peergos-b           
+504        stopped                 minio               
+510        running                 verae-px-worker     
+511        running                 nats-a              
+512        running                 nats-b              
+513        running                 nats-c              
+2000       stopped                 CT2000              
+2001       stopped                 CT2001              
+2002       stopped                 CT2002              
+--- pct config 510 ---
+cores: 2
+hostname: verae-px-worker
+memory: 2048
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:BD:5D:62,ip=10.10.10.20/24,type=veth
+rootfs: SSD2:subvol-510-disk-0,size=16G
+swap: 512
+--- pct config 511 ---
+cores: 1
+hostname: nats-a
+memory: 1024
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:CF:AC:68,ip=10.10.10.21/24,type=veth
+rootfs: SSD2:subvol-511-disk-0,size=8G
+swap: 256
+--- pct config 512 ---
+cores: 1
+hostname: nats-b
+memory: 1024
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:42:73:A2,ip=10.10.10.22/24,type=veth
+rootfs: SSD2:subvol-512-disk-0,size=8G
+swap: 256
+--- pct config 513 ---
+cores: 1
+hostname: nats-c
+memory: 1024
+net0: name=eth0,bridge=vmbr1,gw=10.10.10.1,hwaddr=BC:24:11:1B:9B:DC,ip=10.10.10.23/24,type=veth
+rootfs: SSD2:subvol-513-disk-0,size=8G
+swap: 256

results/20260912T051237Z/js-1p-20k-128-r3.csv

@@ -0,0 +1,2 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+8rEl4zHbyuPY6Hq62w9Qsw,P0,40000,2560000,7393,946342.174329,2.705153

results/20260912T051237Z/js-1p-20k-128-r3.txt

@@ -0,0 +1,5 @@
+=== js-1p-20k-128-r3 ===
+
+Pub stats: 7,393 msgs/sec ~ 924.16 KB/sec
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T051237Z/js-2p2s-20k-128-r3.csv

@@ -0,0 +1,5 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+VI7cqfC8hjqPoKek3p2a06,S0,20150,1280000,4248,543810.859674,2.353760
+VI7cqfC8hjqPoKek3p2a06,S1,20159,1280000,3858,493875.492651,2.591746
+VI7cqfC8hjqPoKek3p2a06,P0,20000,1280000,11290,1445228.034916,0.885673
+VI7cqfC8hjqPoKek3p2a06,P1,20000,1280000,3864,494600.778891,2.587946

results/20260912T051237Z/js-2p2s-20k-128-r3.txt

@@ -0,0 +1,13 @@
+=== js-2p2s-20k-128-r3 ===
+
+NATS Pub/Sub stats: 15,416 msgs/sec ~ 1.88 MB/sec
+ Pub stats: 7,728 msgs/sec ~ 966.02 KB/sec
+  [1] 11,290 msgs/sec ~ 1.38 MB/sec (10000 msgs)
+  [2] 3,864 msgs/sec ~ 483.01 KB/sec (10000 msgs)
+  min 3,864 | avg 7,577 | max 11,290 | stddev 3,713 msgs
+ Sub stats: 7,711 msgs/sec ~ 963.88 KB/sec
+  [1] 4,248 msgs/sec ~ 531.07 KB/sec (10000 msgs)
+  [2] 3,858 msgs/sec ~ 482.30 KB/sec (10000 msgs)
+  min 3,858 | avg 4,053 | max 4,248 | stddev 195 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T051237Z/js-4p-20k-1k-r3.csv

@@ -0,0 +1,5 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+InppzTmEhsFOG1VYCh0Nle,P0,10000,5120000,4792,4907526.298214,1.043295
+InppzTmEhsFOG1VYCh0Nle,P1,10000,5120000,4480,4587592.638694,1.116054
+InppzTmEhsFOG1VYCh0Nle,P2,10000,5120000,3792,3883639.365215,1.318351
+InppzTmEhsFOG1VYCh0Nle,P3,10000,5120000,3750,3840425.475938,1.333186

results/20260912T051237Z/js-4p-20k-1k-r3.txt

@@ -0,0 +1,10 @@
+=== js-4p-20k-1k-r3 ===
+
+Pub stats: 14,985 msgs/sec ~ 14.63 MB/sec
+ [1] 4,792 msgs/sec ~ 4.68 MB/sec (5000 msgs)
+ [2] 4,480 msgs/sec ~ 4.38 MB/sec (5000 msgs)
+ [3] 3,792 msgs/sec ~ 3.70 MB/sec (5000 msgs)
+ [4] 3,750 msgs/sec ~ 3.66 MB/sec (5000 msgs)
+ min 3,750 | avg 4,203 | max 4,792 | stddev 446 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T051237Z/js-4p-50k-128-r3.csv

@@ -0,0 +1,5 @@
+#RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
+JeTysfxECHCJccBaGH4xXX,P0,25000,1600000,5640,721943.982931,2.216238
+JeTysfxECHCJccBaGH4xXX,P1,25000,1600000,5642,722187.413742,2.215491
+JeTysfxECHCJccBaGH4xXX,P2,25000,1600000,5056,647242.012627,2.472027
+JeTysfxECHCJccBaGH4xXX,P3,25000,1600000,4497,575730.101666,2.779080

results/20260912T051237Z/js-4p-50k-128-r3.txt

@@ -0,0 +1,10 @@
+=== js-4p-50k-128-r3 ===
+
+Pub stats: 17,986 msgs/sec ~ 2.20 MB/sec
+ [1] 5,640 msgs/sec ~ 705.02 KB/sec (12500 msgs)
+ [2] 5,642 msgs/sec ~ 705.26 KB/sec (12500 msgs)
+ [3] 5,056 msgs/sec ~ 632.07 KB/sec (12500 msgs)
+ [4] 4,497 msgs/sec ~ 562.24 KB/sec (12500 msgs)
+ min 4,497 | avg 5,208 | max 5,642 | stddev 475 msgs
+
+Saved metric data in csv file /tmp/bench.csv

results/20260912T051237Z/lat-1p-5k-128.txt

@@ -0,0 +1,2 @@
+=== lat-1p-5k-128 ===
+{"count":5000,"pubs":1,"size":128,"mode":"flood","min_us":132425,"avg_us":203717,"p50_us":204458,"p90_us":241750,"p99_us":249468,"max_us":249640,"min":"132.425ms","avg":"203.717ms","p50":"204.458ms","p90":"241.750ms","p99":"249.468ms","max":"249.640ms"}

results/20260912T051237Z/lat-4p-10k-128.txt

@@ -0,0 +1,2 @@
+=== lat-4p-10k-128 ===
+{"count":10000,"pubs":4,"size":128,"mode":"flood","min_us":211108,"avg_us":294925,"p50_us":300254,"p90_us":314223,"p99_us":315551,"max_us":316091,"min":"211.108ms","avg":"294.925ms","p50":"300.254ms","p90":"314.223ms","p99":"315.551ms","max":"316.091ms"}

results/20260912T051237Z/lat-4p-5k-1k.txt

@@ -0,0 +1,2 @@
+=== lat-4p-5k-1k ===
+{"count":5000,"pubs":4,"size":1024,"mode":"flood","min_us":196422,"avg_us":221264,"p50_us":224039,"p90_us":233180,"p99_us":236798,"max_us":238177,"min":"196.422ms","avg":"221.264ms","p50":"224.039ms","p90":"233.180ms","p99":"236.798ms","max":"238.177ms"}

results/20260912T051237Z/lat-8p-20k-128.txt

@@ -0,0 +1,2 @@
+=== lat-8p-20k-128 ===
+{"count":20000,"pubs":8,"size":128,"mode":"flood","min_us":280608,"avg_us":414972,"p50_us":409802,"p90_us":513821,"p99_us":535394,"max_us":536084,"min":"280.608ms","avg":"414.972ms","p50":"409.802ms","p90":"513.821ms","p99":"535.394ms","max":"536.084ms"}

results/20260912T051237Z/lat-ping-1k-128.txt

@@ -0,0 +1,2 @@
+=== lat-ping-1k-128 ===
+{"count":1000,"pubs":1,"size":128,"mode":"ping","min_us":341,"avg_us":470,"p50_us":395,"p90_us":651,"p99_us":1377,"max_us":3243,"min":"0.341ms","avg":"0.470ms","p50":"0.395ms","p90":"0.651ms","p99":"1.377ms","max":"3.243ms"}

results/20260912T051237Z/nats-cluster-bench-ns1.md

@@ -0,0 +1,334 @@
+**Progress report (second study)** · run `20260912T051237Z` (UTC)
+
+> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): the orchestrator (`study-on-ns1.sh`), `nats bench`, `latency.mjs` (inside LXC 510 on this hypervisor), charting (`matplotlib`), and HTML/PDF (`pandoc` + `weasyprint`). The operator laptop did **not** publish, subscribe, draw charts, or render the PDF. Traffic stayed on `vmbr1` from LXC **510** to `nats-a/b/c` (**511–513**).
+
+This is a full methodology write-up plus the numbers from that on-host run. The earlier report (`nats-cluster-bench`, run `20260912T045131Z`) used the same cluster but was **orchestrated and rendered off-box**. Use this document when you need “it was all run on 138.”
+
+---
+
+## 1. Executive summary
+
+| Item | This NS1-host run |
+|------|-------------------|
+| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `marchon` |
+| Bench client | LXC 510 `verae-px-worker` |
+| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` |
+| Client URL | `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` |
+| Host load before | `8.59 8.39 8.15 5/3843 4096277` |
+| Host load after | `8.67 8.33 8.15 9/3863 4114865` |
+| Core 1p1s 128 B pub | 502,502 msgs/s |
+| JetStream 1p 128 B r=3 | 7,393 durable pubs/s |
+| Ping p50 / p99 | 0.395ms / 1.377ms |
+
+Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst.
+
+---
+
+## 2. Where it ran (and where it did not)
+
+```text
+Operator laptop  ──ssh──►  NS1.GEORGELAMBERT.ORG  70.88.205.138
+                              study-on-ns1.sh
+                              python3 build-ns1-study-report.py
+                              sudo pct exec 510 ──► nats bench / latency.mjs
+                                                    │
+                                                    ▼ vmbr1
+                                              10.10.10.21-23 :4222
+```
+
+- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510).
+- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file.
+
+---
+
+## 3. Results (this run)
+
+### Host and brokers
+
+**Before**
+
+| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream |
+|------|------|-------------|---------|----------|-----|-------|---------|-----------|
+| nats-a | 511 | 2 | 1,721,203 | 4,933,759 | 1 | 1 | 26,398,720 | True |
+| nats-b | 512 | 2 | 1,613,205 | 3,513,096 | 0 | 1 | 23,097,344 | True |
+| nats-c | 513 | 2 | 1,760,973 | 4,048,582 | 2 | 1 | 23,867,392 | True |
+
+**After**
+
+| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream |
+|------|------|-------------|---------|----------|-----|-------|---------|-----------|
+| nats-a | 511 | 2 | 2,097,596 | 5,910,161 | 1 | 1 | 38,273,024 | True |
+| nats-b | 512 | 2 | 2,121,816 | 4,746,539 | 1 | 1 | 41,361,408 | True |
+| nats-c | 513 | 2 | 2,241,366 | 5,053,978 | 1 | 1 | 38,162,432 | True |
+
+nproc=40 · uname=`Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux`
+
+### Throughput
+
+| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
+|-----|------|------------------|------------|----------|------------|----------|
+| `core-1p1s-50k-128` | core pub/sub | 776,331 | 502,502 | 61.34 | 420,976 | 51.39 |
+| `core-4p4s-100k-128` | core pub/sub | 1,278,454 | 379,985 | 46.38 | 1,078,910 | 131.70 |
+| `core-4p4s-50k-1k` | core pub/sub | 581,464 | 209,545 | 204.63 | 482,645 | 471.33 |
+| `core-8p8s-200k-128` | core pub/sub | 2,065,217 | 333,731 | 40.74 | 1,847,242 | 225.49 |
+| `js-1p-20k-128-r3` | jetstream r=3 file | — | 7,393 | 0.90 | — | — |
+| `js-2p2s-20k-128-r3` | jetstream r=3 file | 15,416 | 7,728 | 0.94 | 7,711 | 0.94 |
+| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 14,985 | 14.63 | — | — |
+| `js-4p-50k-128-r3` | jetstream r=3 file | — | 17,986 | 2.20 | — | — |
+
+### Round-trip delay
+
+| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
+|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|
+| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.341ms | 0.470ms | 0.395ms | 0.651ms | 1.377ms | 3.243ms |
+| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 132.425ms | 203.717ms | 204.458ms | 241.750ms | 249.468ms | 249.640ms |
+| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 196.422ms | 221.264ms | 224.039ms | 233.180ms | 236.798ms | 238.177ms |
+| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 211.108ms | 294.925ms | 300.254ms | 314.223ms | 315.551ms | 316.091ms |
+| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 280.608ms | 414.972ms | 409.802ms | 513.821ms | 535.394ms | 536.084ms |
+
+### Core NATS
+
+![Core NATS throughput at four loads (NS1 host run)](charts/core-throughput.png)
+
+*Core NATS throughput at four loads (NS1 host run)*
+### Payload size (core)
+
+![Core NATS 128 B vs 1 KiB (NS1 host run)](charts/payload-size.png)
+
+*Core NATS 128 B vs 1 KiB (NS1 host run)*
+### JetStream r=3 file
+
+![JetStream durable publish rate (NS1 host run)](charts/js-throughput.png)
+
+*JetStream durable publish rate (NS1 host run)*
+### Core vs JetStream
+
+![Core vs JetStream publish rate, log scale (NS1 host run)](charts/core-vs-js.png)
+
+*Core vs JetStream publish rate, log scale (NS1 host run)*
+### Delay
+
+![Ping vs flood delay percentiles, log scale (NS1 host run)](charts/delay-percentiles.png)
+
+*Ping vs flood delay percentiles, log scale (NS1 host run)*
+
+---
+
+## 4. Study methodology
+
+### 4.1 Question
+
+On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)?
+
+### 4.2 Hypotheses (stated before the run)
+
+1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica.
+2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated.
+3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting.
+4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count.
+5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound.
+
+### 4.3 Independent variables (what we changed)
+
+| Factor | Levels |
+|--------|--------|
+| Transport | Core NATS pub/sub vs JetStream file replicas=3 |
+| Publisher count | 1, 2, 4, 8 |
+| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 |
+| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) |
+| Payload | 128 B, 1024 B |
+| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) |
+
+### 4.4 Dependent variables (what we recorded)
+
+| Metric | Instrument | Unit |
+|--------|------------|------|
+| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s |
+| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s |
+| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) |
+| Publisher spread | nats min/avg/max **msgs/s** | not delay |
+| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max |
+| Host load | `/proc/loadavg` before and after | load average |
+| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem |
+
+**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`.
+
+### 4.5 Controls and constants
+
+- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`.
+- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`).
+- Bench client is LXC **510**, not a nats-* server.
+- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide.
+- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`).
+- No TLS, no nkeys, no account isolation (isolation is `vmbr1`).
+- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder.
+
+### 4.6 Procedure
+
+1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise.
+2. Snapshot host load, memory, LXC configs, and each nats `varz`.
+3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB).
+4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file.
+5. Copy `latency.mjs` into 510; ping then flood at several batch sizes.
+6. Snapshot host/`varz` again.
+7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**.
+
+No publish, subscribe, chart, or PDF process runs on the operator laptop for this study.
+
+### 4.7 Instrumentation path
+
+```text
+[NS1 host 70.88.205.138]
+   study-on-ns1.sh  (bash + python3)
+        |
+        | sudo pct exec 510
+        v
+[LXC 510 verae-px-worker 10.10.10.20]
+   nats bench / node latency.mjs
+        |
+        | NATS client protocol to
+        v
+[LXC 511/512/513  10.10.10.21-23 :4222]
+   nats-server -js  cluster routes :6222
+```
+
+The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`.
+
+### 4.8 Threats to validity
+
+| Threat | Effect on numbers |
+|--------|-------------------|
+| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. |
+| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. |
+| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. |
+| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. |
+| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. |
+| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. |
+| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. |
+| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. |
+
+### 4.9 Ethics / safety
+
+Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket.
+
+---
+
+## 5. Suggestions for fine-tuning
+
+These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way.
+
+### 5.1 Treat JetStream as the product limiter
+
+Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out.
+
+### 5.2 Split storage class by stream
+
+| Stream | Suggested store | Why |
+|--------|-----------------|-----|
+| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad |
+| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both |
+| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer |
+| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry |
+| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive |
+
+Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert.
+
+### 5.3 Give JetStream real disks
+
+Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain.
+
+- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`.
+- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure.
+- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it.
+- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.”
+
+### 5.4 Isolate the nats CTs from the rest of NS1
+
+Host load on this box is often already several. Pin:
+
+- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes.
+- Memory high enough that file-backed streams stay cache-hot for the working set.
+- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync.
+
+Re-run this study after pinning; H1/H2 should move more than ping.
+
+### 5.5 Consumer and mailbox tuning (delay H4)
+
+Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT.
+
+- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM.
+- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber.
+- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped.
+- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT.
+
+### 5.6 Publisher-side batching in middleware
+
+A timestamp job is one small JSON. 16k msgs/s is ample. Still:
+
+- Avoid per-byte publishes; one message per job/event.
+- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs).
+- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only.
+
+### 5.7 nats-server knobs worth measuring (A/B with this script)
+
+| Knob | Why try it |
+|------|------------|
+| `max_payload` | Keep default unless archive puts grow |
+| `write_deadline` | Slow consumer protection for webhooks |
+| `max_pending` | Bound memory on a stuck Zapier hook |
+| `max_connections` | Fleet workers + keep + middleware |
+| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT |
+| `max_outstanding_catchup` | Replica restart after a nats-c blip |
+| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT |
+
+Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99.
+
+### 5.8 Network
+
+- Keep NATS off `vmbr0`. No change.
+- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load).
+- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus.
+
+### 5.9 Security cost (when nkeys/mTLS flip)
+
+`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement.
+
+### 5.10 Operational fine-tuning (lag, not peak msgs/s)
+
+1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`.
+2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense.
+3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost.
+4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated.
+
+### 5.11 What not to tune
+
+- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge.
+- Do not treat flood 400 ms as “cluster RTT.” Fix consumers.
+- Do not load-test on `ZAPIER_*` streams.
+- Do not bind client NATS to `0.0.0.0` on `vmbr0`.
+
+### 5.12 Recommended next experiments (same method, one change each)
+
+1. CPU pin nats-a/b/c → re-run JS 1p + ping.
+2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder).
+3. Distinct `store_dir` disks per node.
+4. nkeys on, same ladder.
+5. Three hardware boxes, same `cluster.env` IPs updated.
+
+Each experiment should produce a new `results/<utc>/` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory.
+
+
+---
+
+## 6. Reproducing this study
+
+On **NS1 only**:
+
+```bash
+cd ~/verae-src/verae-nats-cluster
+bash scripts/study-on-ns1.sh
+```
+
+The script exits if `hostname` is not NS1. Outputs land in `results/<utc>/` including `nats-cluster-bench-ns1.{md,html,pdf}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog.
+
+Raw logs for this run: `results/20260912T051237Z/`.

results/20260912T051237Z/varz-after.json

@@ -0,0 +1,50 @@
+[
+  {
+    "vmid": "511",
+    "name": "nats-a",
+    "server_name": "nats-a",
+    "host": "10.10.10.21",
+    "port": 4222,
+    "connections": 2,
+    "in_msgs": 2097596,
+    "out_msgs": 5910161,
+    "in_bytes": 559427613,
+    "out_bytes": 1494241553,
+    "cpu": 1,
+    "cores": 1,
+    "mem": 38273024,
+    "jetstream": true
+  },
+  {
+    "vmid": "512",
+    "name": "nats-b",
+    "server_name": "nats-b",
+    "host": "10.10.10.22",
+    "port": 4222,
+    "connections": 2,
+    "in_msgs": 2121816,
+    "out_msgs": 4746539,
+    "in_bytes": 585390797,
+    "out_bytes": 974695990,
+    "cpu": 1,
+    "cores": 1,
+    "mem": 41361408,
+    "jetstream": true
+  },
+  {
+    "vmid": "513",
+    "name": "nats-c",
+    "server_name": "nats-c",
+    "host": "10.10.10.23",
+    "port": 4222,
+    "connections": 2,
+    "in_msgs": 2241366,
+    "out_msgs": 5053978,
+    "in_bytes": 586858467,
+    "out_bytes": 1040486148,
+    "cpu": 1,
+    "cores": 1,
+    "mem": 38162432,
+    "jetstream": true
+  }
+]

results/20260912T051237Z/varz-before.json

@@ -0,0 +1,50 @@
+[
+  {
+    "vmid": "511",
+    "name": "nats-a",
+    "server_name": "nats-a",
+    "host": "10.10.10.21",
+    "port": 4222,
+    "connections": 2,
+    "in_msgs": 1721203,
+    "out_msgs": 4933759,
+    "in_bytes": 440427700,
+    "out_bytes": 1169323990,
+    "cpu": 1,
+    "cores": 1,
+    "mem": 26398720,
+    "jetstream": true
+  },
+  {
+    "vmid": "512",
+    "name": "nats-b",
+    "server_name": "nats-b",
+    "host": "10.10.10.22",
+    "port": 4222,
+    "connections": 2,
+    "in_msgs": 1613205,
+    "out_msgs": 3513096,
+    "in_bytes": 428640182,
+    "out_bytes": 715428083,
+    "cpu": 0,
+    "cores": 1,
+    "mem": 23097344,
+    "jetstream": true
+  },
+  {
+    "vmid": "513",
+    "name": "nats-c",
+    "server_name": "nats-c",
+    "host": "10.10.10.23",
+    "port": 4222,
+    "connections": 2,
+    "in_msgs": 1760973,
+    "out_msgs": 4048582,
+    "in_bytes": 447844308,
+    "out_bytes": 864199465,
+    "cpu": 2,
+    "cores": 1,
+    "mem": 23867392,
+    "jetstream": true
+  }
+]

scripts/bench-report.py

@@ -0,0 +1,192 @@
+#!/usr/bin/env python3
+"""Turn nats bench text logs + latency JSON into markdown."""
+from __future__ import annotations
+
+import json
+import re
+import sys
+from pathlib import Path
+
+
+def fmt_int(s: str | None) -> str:
+    if not s:
+        return "—"
+    return f"{int(s):,}"
+
+
+def parse_bench(text: str) -> dict[str, str]:
+    out: dict[str, str] = {"kind": "throughput"}
+    def _rate(pattern: str, msgs_key: str, mb_key: str) -> None:
+        m = re.search(pattern, text)
+        if not m:
+            return
+        out[msgs_key] = m.group(1).replace(",", "")
+        n = float(m.group(2))
+        unit = m.group(3).upper()
+        if unit == "KB":
+            n = n / 1024.0
+        elif unit == "GB":
+            n = n * 1024.0
+        out[mb_key] = f"{n:.2f}"
+
+    _rate(
+        r"(?m)^\s*Pub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*(KB|MB|GB)/sec",
+        "pub_msgs",
+        "pub_mb",
+    )
+    _rate(
+        r"(?m)^\s*Sub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*(KB|MB|GB)/sec",
+        "sub_msgs",
+        "sub_mb",
+    )
+    _rate(
+        r"NATS Pub/Sub stats:\s*([0-9,]+)\s*msgs/sec\s*~\s*([0-9.]+)\s*(KB|MB|GB)/sec",
+        "agg_msgs",
+        "agg_mb",
+    )
+    if "JetStream" in text or "--js" in text or "js-" in text:
+        out["mode"] = "jetstream r=3 file"
+    else:
+        out["mode"] = "core pub/sub"
+    # nats 0.1.6 prints min/avg/max as msgs/sec across publishers, not µs delay
+    m = re.search(
+        r"min\s+([0-9,]+)\s*\|\s*avg\s+([0-9,]+)\s*\|\s*max\s+([0-9,]+)\s*\|\s*stddev\s+([0-9,]+)\s*msgs",
+        text,
+    )
+    if m:
+        out["pub_spread"] = f"{m.group(1)}–{m.group(3)} (avg {m.group(2)})"
+    return out
+
+
+def parse_lat(text: str) -> dict[str, str] | None:
+    for line in text.splitlines():
+        line = line.strip()
+        if line.startswith("{") and "p99_us" in line:
+            d = json.loads(line)
+            mode = d.get("mode") or ""
+            return {
+                "kind": "latency",
+                "count": str(d.get("count", "")),
+                "pubs": str(d.get("pubs", "")),
+                "size": str(d.get("size", "")),
+                "mode": str(mode),
+                "min": d.get("min", ""),
+                "avg": d.get("avg", ""),
+                "p50": d.get("p50", ""),
+                "p90": d.get("p90", ""),
+                "p99": d.get("p99", ""),
+                "max": d.get("max", ""),
+            }
+    return None
+
+
+def lat_mode(run: str, recorded: str) -> str:
+    if recorded in ("ping", "flood"):
+        return recorded
+    if "ping" in run:
+        return "ping"
+    return "flood"
+
+
+def thru_sort(p: dict[str, str]) -> tuple:
+    return (0 if p.get("mode", "").startswith("core") else 1, p.get("run", ""))
+
+
+def lat_sort(p: dict[str, str]) -> tuple:
+    mode = lat_mode(p.get("run", ""), p.get("mode", ""))
+    return (0 if mode == "ping" else 1, int(p.get("count") or 0), p.get("run", ""))
+
+
+def main() -> int:
+    folder = Path(sys.argv[1] if len(sys.argv) > 1 else ".")
+    thru: list[dict[str, str]] = []
+    lats: list[dict[str, str]] = []
+    for f in sorted(folder.glob("*.txt")):
+        text = f.read_text(encoding="utf-8", errors="replace")
+        lat = parse_lat(text)
+        if lat:
+            lat["run"] = f.stem
+            lats.append(lat)
+            continue
+        p = parse_bench(text)
+        if p.get("pub_msgs") or p.get("agg_msgs"):
+            p["run"] = f.stem
+            thru.append(p)
+
+    stamp = folder.name if re.fullmatch(r"\d{8}T\d{6}Z", folder.name) else ""
+
+    print("# NATS cluster message speed")
+    print()
+    if stamp:
+        print(f"Run **`{stamp}`** (UTC). ", end="")
+    print(
+        "Client: LXC **510** `verae-px-worker` (`10.10.10.20`), not a nats-* server. "
+        "Servers: `nats-a/b/c` on `10.10.10.21–23` (`vmbr1` only)."
+    )
+    print()
+    print("Client URL:")
+    print()
+    print("```text")
+    print("nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222")
+    print("```")
+    print()
+    print("## Method")
+    print()
+    print("- **Core NATS** is fire-and-forget pub/sub (`nats bench`). No disk, no replica ack.")
+    print(
+        "- **JetStream** uses **file** storage and **replicas=3** (same as product streams). "
+        "The unique stream `benchstream` is deleted between JS loads."
+    )
+    print("- Throughput is **msgs/sec** from nats CLI **0.1.6** (`--no-progress --csv`). Its min/avg/max are publisher **rate spread**, not delay.")
+    print(
+        "- **Ping** delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster."
+    )
+    print(
+        "- **Flood** delay: N publishers dump the whole batch, then the subscriber drains. "
+        "This is **queueing under burst**, not wire RTT."
+    )
+    print("- Probe: `scripts/latency.mjs` (two connections, header timestamp).")
+    print()
+    print("## Throughput")
+    print()
+    print("| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |")
+    print("|-----|------|------------------|------------|----------|------------|----------|")
+    for p in sorted(thru, key=thru_sort):
+        print(
+            f"| `{p['run']}` | {p.get('mode', '')} | {fmt_int(p.get('agg_msgs'))} | "
+            f"{fmt_int(p.get('pub_msgs'))} | {p.get('pub_mb') or '—'} | "
+            f"{fmt_int(p.get('sub_msgs'))} | {p.get('sub_mb') or '—'} |"
+        )
+    print()
+    print("## Round-trip delay")
+    print()
+    print("| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |")
+    print("|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|")
+    for p in sorted(lats, key=lat_sort):
+        kind = lat_mode(p.get("run", ""), p.get("mode", ""))
+        label = "ping (sequential RTT)" if kind == "ping" else "flood (burst queueing)"
+        print(
+            f"| `{p['run']}` | {label} | {p.get('count', '')} | {p.get('pubs', '')} | "
+            f"{p.get('size', '')} B | {p.get('min', '')} | {p.get('avg', '')} | "
+            f"{p.get('p50', '')} | {p.get('p90', '')} | {p.get('p99', '')} | {p.get('max', '')} |"
+        )
+    print()
+    print("## What the numbers mean")
+    print()
+    print(
+        "Product job/event/archive traffic is **JetStream r=3 file**. On this three-LXC stand that is about "
+        "**16k durable 128 B pubs/s** (about **13k** at 1 KiB). Core NATS is an upper bound for "
+        "non-durable fan-out: about **0.7–2.0M msgs/s** aggregate at 128 B, or **~630k msgs/s (~616 MB/s)** at 1 KiB with 4 publishers."
+    )
+    print()
+    print(
+        "A quiet request-reply is **~0.3 ms** average, **p99 < 1 ms**. Flood rows in the **150–500 ms** band "
+        "are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers."
+    )
+    print()
+    print("Re-run on NS1: `bash scripts/bench.sh`. Raw logs/CSVs are under `results/<utc>/`.")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

scripts/bench.sh

@@ -0,0 +1,106 @@
+#!/usr/bin/env bash
+# Message throughput and delay ladder against the 3-node vmbr1 cluster.
+# Prefers a client that is not a nats-* server (px-worker LXC 510).
+set -euo pipefail
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+# shellcheck disable=SC1091
+. "$ROOT/client.env"
+export PATH="/usr/sbin:/usr/bin:/bin:/usr/local/bin:$PATH"
+STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
+OUT="${BENCH_OUT:-$ROOT/results/$STAMP}"
+CLIENT_VMID="${CLIENT_VMID:-510}"
+mkdir -p "$OUT"
+
+ensure_nats_cli() {
+  local vmid="$1"
+  sudo pct exec "$vmid" -- bash -lc '
+set -e
+export DEBIAN_FRONTEND=noninteractive
+export PATH=/usr/local/bin:/usr/bin:/bin
+if [[ ! -x /usr/local/bin/nats ]]; then
+  apt-get install -y --no-install-recommends unzip curl ca-certificates >/dev/null
+  curl -fsSL https://github.com/nats-io/natscli/releases/download/v0.1.6/nats-0.1.6-linux-amd64.zip -o /tmp/natscli.zip
+  rm -rf /tmp/natscli && mkdir -p /tmp/natscli
+  unzip -o /tmp/natscli.zip -d /tmp/natscli >/dev/null
+  BIN=$(find /tmp/natscli -type f -name nats | head -1)
+  install -m 0755 "$BIN" /usr/local/bin/nats
+fi
+nats --version
+'
+}
+
+run_one() {
+  local name="$1"
+  shift
+  echo "=== $name ===" | tee "$OUT/$name.txt"
+  # nats bench writes csv itself when --csv is a path inside the guest
+  sudo pct exec "$CLIENT_VMID" -- bash -lc "
+export PATH=/usr/local/bin:/usr/bin:/bin
+export NATS_URL='$NATS_URL'
+nats bench --no-progress --csv=/tmp/bench.csv $*
+" | tee -a "$OUT/$name.txt"
+  sudo pct exec "$CLIENT_VMID" -- cat /tmp/bench.csv >"$OUT/$name.csv" || true
+}
+
+echo "client LXC $CLIENT_VMID  NATS_URL=$NATS_URL  out=$OUT"
+ensure_nats_cli "$CLIENT_VMID"
+
+# Core NATS pub/sub — increasing publishers (same 128 B payload)
+run_one core-1p1s-50k-128   bench.core.a --pub 1 --sub 1 --msgs 50000 --size 128
+run_one core-4p4s-100k-128  bench.core.b --pub 4 --sub 4 --msgs 100000 --size 128
+run_one core-8p8s-200k-128  bench.core.c --pub 8 --sub 8 --msgs 200000 --size 128
+run_one core-4p4s-50k-1k    bench.core.d --pub 4 --sub 4 --msgs 50000 --size 1024
+
+js_rm() {
+  sudo pct exec "$CLIENT_VMID" -- bash -lc "
+export PATH=/usr/local/bin:/usr/bin:/bin
+export NATS_URL='$NATS_URL'
+nats stream rm benchstream --force >/dev/null 2>&1 || true
+"
+}
+# JetStream file store, replicas=3 (matches product streams)
+js_rm
+run_one js-1p-20k-128-r3    bench.js.a --js --purge --pub 1 --msgs 20000 --size 128 --replicas 3 --storage file --maxbytes=512MB --stream=benchstream
+js_rm
+run_one js-4p-50k-128-r3    bench.js.b --js --purge --pub 4 --msgs 50000 --size 128 --replicas 3 --storage file --maxbytes=512MB --stream=benchstream
+js_rm
+run_one js-4p-20k-1k-r3     bench.js.c --js --purge --pub 4 --msgs 20000 --size 1024 --replicas 3 --storage file --maxbytes=512MB --stream=benchstream
+js_rm
+run_one js-2p2s-20k-128-r3  bench.js.d --js --purge --pub 2 --sub 2 --msgs 20000 --size 128 --replicas 3 --storage file --maxbytes=512MB --pull --stream=benchstream
+js_rm
+
+# Round-trip delay (two connections, through the cluster) at several loads
+sudo pct exec "$CLIENT_VMID" -- bash -lc "
+set -e
+export PATH=/usr/local/bin:/usr/bin:/bin
+export NATS_URL='$NATS_URL'
+mkdir -p /tmp/nats-lat
+cd /tmp/nats-lat
+if [[ ! -d node_modules/nats ]]; then
+  npm init -y >/dev/null
+  npm install --no-audit --no-fund nats@2 >/dev/null
+fi
+" >/dev/null
+sudo pct push "$CLIENT_VMID" "$ROOT/scripts/latency.mjs" /tmp/nats-lat/latency.mjs || \
+  sudo pct exec "$CLIENT_VMID" -- bash -c 'cat > /tmp/nats-lat/latency.mjs' < "$ROOT/scripts/latency.mjs"
+lat() {
+  local name="$1" n="$2" sz="$3" p="$4" mode="${5:-flood}"
+  echo "=== $name ===" | tee "$OUT/$name.txt"
+  sudo pct exec "$CLIENT_VMID" -- bash -lc "
+export PATH=/usr/local/bin:/usr/bin:/bin
+export NATS_URL='$NATS_URL'
+cd /tmp/nats-lat
+node latency.mjs $n $sz $p $mode
+" | tee -a "$OUT/$name.txt"
+}
+# copy latest probe
+sudo pct exec "$CLIENT_VMID" -- bash -c 'cat > /tmp/nats-lat/latency.mjs' < "$ROOT/scripts/latency.mjs"
+lat lat-ping-1k-128 1000 128 1 ping
+lat lat-1p-5k-128 5000 128 1 flood
+lat lat-4p-10k-128 10000 128 4 flood
+lat lat-8p-20k-128 20000 128 8 flood
+lat lat-4p-5k-1k 5000 1024 4 flood
+
+python3 "$ROOT/scripts/bench-report.py" "$OUT" >"$OUT/BENCH.md"
+cp "$OUT/BENCH.md" "$ROOT/BENCH.md"
+echo "wrote $OUT/BENCH.md and $ROOT/BENCH.md"

scripts/build-ns1-study-report.py

@@ -0,0 +1,458 @@
+#!/usr/bin/env python3
+"""Build the NS1-host study report (charts + markdown + HTML + PDF) from a results dir.
+
+Must be able to run entirely on NS1.GEORGELAMBERT.ORG with python3, matplotlib,
+pandoc, and weasyprint. Parses nats bench logs; does not hard-code rates.
+"""
+from __future__ import annotations
+
+import json
+import re
+import subprocess
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+import importlib.util
+
+_spec = importlib.util.spec_from_file_location(
+    "bench_report", Path(__file__).resolve().parent / "bench-report.py"
+)
+_br = importlib.util.module_from_spec(_spec)
+assert _spec.loader is not None
+_spec.loader.exec_module(_br)
+fmt_int = _br.fmt_int
+lat_mode = _br.lat_mode
+lat_sort = _br.lat_sort
+parse_bench = _br.parse_bench
+parse_lat = _br.parse_lat
+thru_sort = _br.thru_sort
+
+try:
+    import matplotlib
+
+    matplotlib.use("Agg")
+    import matplotlib.pyplot as plt
+    from matplotlib.ticker import FuncFormatter
+except ImportError as e:
+    raise SystemExit(f"matplotlib required on NS1: {e}") from e
+
+INDIGO = "#4f46e5"
+DEEP = "#312e81"
+TEAL = "#047857"
+AMBER = "#b45309"
+LILAC = "#7c74f0"
+INK = "#171a26"
+MUTED = "#5b6178"
+GRID = "#d9dce8"
+
+CORE_LABELS = {
+    "core-1p1s-50k-128": "1p1s\n50k×128 B",
+    "core-4p4s-100k-128": "4p4s\n100k×128 B",
+    "core-8p8s-200k-128": "8p8s\n200k×128 B",
+    "core-4p4s-50k-1k": "4p4s\n50k×1 KiB",
+}
+JS_LABELS = {
+    "js-1p-20k-128-r3": "1p 20k×128 B",
+    "js-4p-50k-128-r3": "4p 50k×128 B",
+    "js-4p-20k-1k-r3": "4p 20k×1 KiB",
+    "js-2p2s-20k-128-r3": "2p2s pull 20k×128 B",
+}
+LAT_LABELS = {
+    "lat-ping-1k-128": "Ping\n1k×128 B",
+    "lat-1p-5k-128": "Flood 1p\n5k×128 B",
+    "lat-4p-5k-1k": "Flood 4p\n5k×1 KiB",
+    "lat-4p-10k-128": "Flood 4p\n10k×128 B",
+    "lat-8p-20k-128": "Flood 8p\n20k×128 B",
+}
+
+
+def ms(s: str) -> float:
+    return float(s.replace("ms", "").replace(",", "").strip())
+
+
+def k_fmt(x: float, _pos: int | None = None) -> str:
+    if x >= 1_000_000:
+        return f"{x / 1_000_000:.2f}M"
+    if x >= 1000:
+        return f"{x / 1000:.0f}k"
+    return f"{x:.0f}"
+
+
+def style() -> None:
+    plt.rcParams.update(
+        {
+            "font.family": "sans-serif",
+            "font.size": 10,
+            "axes.titlesize": 12,
+            "axes.titleweight": "semibold",
+            "axes.edgecolor": GRID,
+            "axes.labelcolor": INK,
+            "text.color": INK,
+            "xtick.color": MUTED,
+            "ytick.color": MUTED,
+            "figure.facecolor": "white",
+            "axes.facecolor": "white",
+            "axes.grid": True,
+            "grid.color": GRID,
+            "grid.linewidth": 0.8,
+            "legend.frameon": False,
+            "savefig.bbox": "tight",
+            "savefig.dpi": 160,
+            "savefig.facecolor": "white",
+        }
+    )
+
+
+def save(fig: plt.Figure, path: Path) -> None:
+    fig.savefig(path, dpi=160)
+    plt.close(fig)
+
+
+def load_runs(folder: Path) -> tuple[list[dict[str, str]], list[dict[str, str]]]:
+    thru: list[dict[str, str]] = []
+    lats: list[dict[str, str]] = []
+    for f in sorted(folder.glob("*.txt")):
+        if f.name.startswith("host-"):
+            continue
+        text = f.read_text(encoding="utf-8", errors="replace")
+        lat = parse_lat(text)
+        if lat:
+            lat["run"] = f.stem
+            lats.append(lat)
+            continue
+        p = parse_bench(text)
+        if p.get("pub_msgs") or p.get("agg_msgs"):
+            p["run"] = f.stem
+            thru.append(p)
+    return sorted(thru, key=thru_sort), sorted(lats, key=lat_sort)
+
+
+def kv_file(path: Path) -> dict[str, str]:
+    out: dict[str, str] = {}
+    if not path.exists():
+        return out
+    for line in path.read_text(encoding="utf-8", errors="replace").splitlines():
+        if "=" in line and not line.startswith("---"):
+            k, _, v = line.partition("=")
+            if k.strip() in out:
+                continue
+            out[k.strip()] = v.strip()
+    return out
+
+
+def thru_table(thru: list[dict[str, str]]) -> str:
+    lines = [
+        "| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |",
+        "|-----|------|------------------|------------|----------|------------|----------|",
+    ]
+    for p in thru:
+        lines.append(
+            f"| `{p['run']}` | {p.get('mode', '')} | {fmt_int(p.get('agg_msgs'))} | "
+            f"{fmt_int(p.get('pub_msgs'))} | {p.get('pub_mb') or '—'} | "
+            f"{fmt_int(p.get('sub_msgs'))} | {p.get('sub_mb') or '—'} |"
+        )
+    return "\n".join(lines)
+
+
+def delay_table(lats: list[dict[str, str]]) -> str:
+    lines = [
+        "| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |",
+        "|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|",
+    ]
+    for p in lats:
+        kind = lat_mode(p.get("run", ""), p.get("mode", ""))
+        label = "ping (sequential RTT)" if kind == "ping" else "flood (burst queueing)"
+        lines.append(
+            f"| `{p['run']}` | {label} | {p.get('count', '')} | {p.get('pubs', '')} | "
+            f"{p.get('size', '')} B | {p.get('min', '')} | {p.get('avg', '')} | "
+            f"{p.get('p50', '')} | {p.get('p90', '')} | {p.get('p99', '')} | {p.get('max', '')} |"
+        )
+    return "\n".join(lines)
+
+
+def varz_table(path: Path) -> str:
+    if not path.exists():
+        return "_varz snapshot not captured._"
+    rows = json.loads(path.read_text(encoding="utf-8"))
+    lines = [
+        "| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream |",
+        "|------|------|-------------|---------|----------|-----|-------|---------|-----------|",
+    ]
+    for r in rows:
+        if r.get("error"):
+            lines.append(f"| {r.get('name')} | {r.get('vmid')} | error: {r['error']} | | | | | | |")
+            continue
+        lines.append(
+            f"| {r.get('name')} | {r.get('vmid')} | {r.get('connections')} | "
+            f"{r.get('in_msgs'):,} | {r.get('out_msgs'):,} | {r.get('cpu')} | "
+            f"{r.get('cores')} | {r.get('mem'):,} | {r.get('jetstream')} |"
+        )
+    return "\n".join(lines)
+
+
+def charts(thru: list[dict[str, str]], lats: list[dict[str, str]], dest: Path) -> None:
+    dest.mkdir(parents=True, exist_ok=True)
+    style()
+    by = {p["run"]: p for p in thru}
+    core_keys = [k for k in CORE_LABELS if k in by]
+    if core_keys:
+        fig, ax = plt.subplots(figsize=(9.2, 4.4))
+        x = list(range(len(core_keys)))
+        w = 0.25
+        agg = [int(by[k].get("agg_msgs") or 0) for k in core_keys]
+        pub = [int(by[k].get("pub_msgs") or 0) for k in core_keys]
+        sub = [int(by[k].get("sub_msgs") or 0) for k in core_keys]
+        ax.bar([i - w for i in x], agg, w, label="Aggregate", color=DEEP)
+        ax.bar(x, pub, w, label="Publish", color=INDIGO)
+        ax.bar([i + w for i in x], sub, w, label="Subscribe", color=TEAL)
+        ax.set_xticks(x, [CORE_LABELS[k] for k in core_keys])
+        ax.set_ylabel("messages / second")
+        ax.set_title("Core NATS throughput (fire-and-forget) — NS1 host run")
+        ax.yaxis.set_major_formatter(FuncFormatter(k_fmt))
+        ax.legend(loc="upper left")
+        ax.set_axisbelow(True)
+        save(fig, dest / "core-throughput.png")
+
+    js_keys = [k for k in JS_LABELS if k in by]
+    if js_keys:
+        fig, ax = plt.subplots(figsize=(9.2, 4.4))
+        pubs = [int(by[k].get("pub_msgs") or 0) for k in js_keys]
+        colors = [INDIGO, INDIGO, AMBER, LILAC][: len(js_keys)]
+        ax.bar([JS_LABELS[k] for k in js_keys], pubs, color=colors)
+        ax.set_ylabel("durable publish messages / second")
+        ax.set_title("JetStream file store, replicas=3 — NS1 host run")
+        ax.yaxis.set_major_formatter(FuncFormatter(k_fmt))
+        ax.set_axisbelow(True)
+        for i, v in enumerate(pubs):
+            ax.text(i, v * 1.02, f"{v:,}", ha="center", va="bottom", fontsize=9, color=MUTED)
+        save(fig, dest / "js-throughput.png")
+
+    pair = [("core-1p1s-50k-128", "js-1p-20k-128-r3"), ("core-4p4s-100k-128", "js-4p-50k-128-r3"), ("core-4p4s-50k-1k", "js-4p-20k-1k-r3")]
+    if all(c in by and j in by for c, j in pair):
+        fig, ax = plt.subplots(figsize=(9.2, 4.4))
+        labels = ["1 publisher\n128 B", "4 publishers\n128 B", "4 publishers\n1 KiB"]
+        core_pub = [int(by[c]["pub_msgs"]) for c, _ in pair]
+        js_pub = [int(by[j]["pub_msgs"]) for _, j in pair]
+        x = list(range(3))
+        w = 0.35
+        ax.bar([i - w / 2 for i in x], core_pub, w, label="Core NATS (no disk)", color=INDIGO)
+        ax.bar([i + w / 2 for i in x], js_pub, w, label="JetStream r=3 file", color=AMBER)
+        ax.set_xticks(x, labels)
+        ax.set_yscale("log")
+        ax.set_ylabel("publish messages / second (log)")
+        ax.set_title("Core vs JetStream — NS1 host run")
+        ax.legend(loc="upper right")
+        ax.set_axisbelow(True)
+        save(fig, dest / "core-vs-js.png")
+
+    if "core-4p4s-100k-128" in by and "core-4p4s-50k-1k" in by:
+        fig, axes = plt.subplots(1, 2, figsize=(9.2, 4.2))
+        labels = ["128 B\n4p4s", "1 KiB\n4p4s"]
+        msgs = [int(by["core-4p4s-100k-128"].get("agg_msgs") or 0), int(by["core-4p4s-50k-1k"].get("agg_msgs") or 0)]
+        mb = [float(by["core-4p4s-100k-128"].get("agg_mb") or 0), float(by["core-4p4s-50k-1k"].get("agg_mb") or 0)]
+        axes[0].bar(labels, msgs, color=[INDIGO, AMBER])
+        axes[0].set_title("Aggregate messages / second")
+        axes[0].yaxis.set_major_formatter(FuncFormatter(k_fmt))
+        axes[1].bar(labels, mb, color=[INDIGO, AMBER])
+        axes[1].set_title("Aggregate MB / second")
+        fig.suptitle("Core NATS payload effect — NS1 host run", fontsize=12, fontweight="semibold")
+        fig.tight_layout()
+        save(fig, dest / "payload-size.png")
+
+    if lats:
+        fig, ax = plt.subplots(figsize=(9.2, 4.6))
+        ordered = [p for p in lats]
+        labels = [LAT_LABELS.get(p["run"], p["run"]) for p in ordered]
+        x = list(range(len(ordered)))
+        w = 0.25
+        p50 = [ms(p["p50"]) for p in ordered]
+        p90 = [ms(p["p90"]) for p in ordered]
+        p99 = [ms(p["p99"]) for p in ordered]
+        ax.bar([i - w for i in x], p50, w, label="p50", color=TEAL)
+        ax.bar(x, p90, w, label="p90", color=INDIGO)
+        ax.bar([i + w for i in x], p99, w, label="p99", color=AMBER)
+        ax.set_xticks(x, labels)
+        ax.set_yscale("log")
+        ax.set_ylabel("milliseconds (log)")
+        ax.set_title("Round-trip delay — NS1 host run")
+        ax.axhline(1.0, color=GRID, linestyle="--", linewidth=1)
+        ax.legend(loc="upper left")
+        ax.set_axisbelow(True)
+        save(fig, dest / "delay-percentiles.png")
+
+
+def figure(name: str, caption: str) -> str:
+    return f"![{caption}](charts/{name})\n\n*{caption}*"
+
+
+def write_markdown(folder: Path, thru: list[dict[str, str]], lats: list[dict[str, str]]) -> str:
+    before = kv_file(folder / "host-before.txt")
+    after = kv_file(folder / "host-after.txt")
+    stamp = folder.name
+    method = (Path(__file__).resolve().parent / "ns1-study-methodology.md").read_text(encoding="utf-8")
+    figs = []
+    charts_dir = folder / "charts"
+    if (charts_dir / "core-throughput.png").exists():
+        figs.append("### Core NATS\n\n" + figure("core-throughput.png", "Core NATS throughput at four loads (NS1 host run)"))
+    if (charts_dir / "payload-size.png").exists():
+        figs.append("### Payload size (core)\n\n" + figure("payload-size.png", "Core NATS 128 B vs 1 KiB (NS1 host run)"))
+    if (charts_dir / "js-throughput.png").exists():
+        figs.append("### JetStream r=3 file\n\n" + figure("js-throughput.png", "JetStream durable publish rate (NS1 host run)"))
+    if (charts_dir / "core-vs-js.png").exists():
+        figs.append("### Core vs JetStream\n\n" + figure("core-vs-js.png", "Core vs JetStream publish rate, log scale (NS1 host run)"))
+    if (charts_dir / "delay-percentiles.png").exists():
+        figs.append("### Delay\n\n" + figure("delay-percentiles.png", "Ping vs flood delay percentiles, log scale (NS1 host run)"))
+
+    ping = next((p for p in lats if "ping" in p.get("run", "")), None)
+    js1 = next((p for p in thru if p["run"] == "js-1p-20k-128-r3"), None)
+    core1 = next((p for p in thru if p["run"] == "core-1p1s-50k-128"), None)
+
+    md = f"""**Progress report (second study)** · run `{stamp}` (UTC)
+
+> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): the orchestrator (`study-on-ns1.sh`), `nats bench`, `latency.mjs` (inside LXC 510 on this hypervisor), charting (`matplotlib`), and HTML/PDF (`pandoc` + `weasyprint`). The operator laptop did **not** publish, subscribe, draw charts, or render the PDF. Traffic stayed on `vmbr1` from LXC **510** to `nats-a/b/c` (**511–513**).
+
+This is a full methodology write-up plus the numbers from that on-host run. The earlier report (`nats-cluster-bench`, run `20260912T045131Z`) used the same cluster but was **orchestrated and rendered off-box**. Use this document when you need “it was all run on 138.”
+
+---
+
+## 1. Executive summary
+
+| Item | This NS1-host run |
+|------|-------------------|
+| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `{before.get("whoami", "marchon")}` |
+| Bench client | LXC {before.get("client_vmid", "510")} `verae-px-worker` |
+| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` |
+| Client URL | `{before.get("nats_url", "")}` |
+| Host load before | `{before.get("loadavg", "n/a")}` |
+| Host load after | `{after.get("loadavg", "n/a")}` |
+| Core 1p1s 128 B pub | {fmt_int(core1.get("pub_msgs") if core1 else None)} msgs/s |
+| JetStream 1p 128 B r=3 | {fmt_int(js1.get("pub_msgs") if js1 else None)} durable pubs/s |
+| Ping p50 / p99 | {ping.get("p50") if ping else "—"} / {ping.get("p99") if ping else "—"} |
+
+Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst.
+
+---
+
+## 2. Where it ran (and where it did not)
+
+```text
+Operator laptop  ──ssh──►  NS1.GEORGELAMBERT.ORG  70.88.205.138
+                              study-on-ns1.sh
+                              python3 build-ns1-study-report.py
+                              sudo pct exec 510 ──► nats bench / latency.mjs
+                                                    │
+                                                    ▼ vmbr1
+                                              10.10.10.21-23 :4222
+```
+
+- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510).
+- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file.
+
+---
+
+## 3. Results (this run)
+
+### Host and brokers
+
+**Before**
+
+{varz_table(folder / "varz-before.json")}
+
+**After**
+
+{varz_table(folder / "varz-after.json")}
+
+nproc={before.get("nproc", "?")} · uname=`{before.get("uname", "")}`
+
+### Throughput
+
+{thru_table(thru)}
+
+### Round-trip delay
+
+{delay_table(lats)}
+
+{chr(10).join(figs)}
+
+---
+
+{method}
+
+---
+
+## 6. Reproducing this study
+
+On **NS1 only**:
+
+```bash
+cd ~/verae-src/verae-nats-cluster
+bash scripts/study-on-ns1.sh
+```
+
+The script exits if `hostname` is not NS1. Outputs land in `results/<utc>/` including `nats-cluster-bench-ns1.{{md,html,pdf}}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog.
+
+Raw logs for this run: `results/{stamp}/`.
+"""
+    return md
+
+
+def render(md_path: Path, html_path: Path, pdf_path: Path) -> None:
+    css = Path(__file__).resolve().parent / "docs-print.css"
+    header = html_path.with_suffix(".hdr.html")
+    banner = html_path.with_suffix(".ban.html")
+    css_text = css.read_text(encoding="utf-8") if css.exists() else ""
+    header.write_text(f"<style>{css_text}</style>\n", encoding="utf-8")
+    banner.write_text(
+        '<div class="doc-banner">'
+        '<nav class="site"><a href="/">zapier.georgelambert.org</a>'
+        ' · <a href="/index-md.html">Markdown indexes</a></nav>'
+        '<div class="kicker">Verae Time × Zapier · progress report · run on NS1.GEORGELAMBERT.ORG</div>'
+        "<h1>NATS cluster message speed — NS1 host study</h1>"
+        '<div class="source-path">packages/zapier-decisions/reports/nats-cluster-bench-ns1.md</div>'
+        "</div>\n",
+        encoding="utf-8",
+    )
+    r = subprocess.run(
+        [
+            "pandoc",
+            str(md_path),
+            "-o",
+            str(html_path),
+            "--standalone",
+            f"--resource-path={md_path.parent}",
+            "--highlight-style=breezedark",
+            "--metadata=title=NATS cluster message speed — NS1 host study",
+            f"--include-in-header={header}",
+            f"--include-before-body={banner}",
+        ],
+        capture_output=True,
+        text=True,
+    )
+    header.unlink(missing_ok=True)
+    banner.unlink(missing_ok=True)
+    if r.returncode != 0:
+        raise SystemExit(f"pandoc failed: {r.stderr[-800:]}")
+    w = subprocess.run(["weasyprint", str(html_path), str(pdf_path)], capture_output=True, text=True)
+    if w.returncode != 0:
+        raise SystemExit(f"weasyprint failed: {w.stderr[-800:]}")
+
+
+def main() -> int:
+    folder = Path(sys.argv[1] if len(sys.argv) > 1 else ".")
+    thru, lats = load_runs(folder)
+    charts(thru, lats, folder / "charts")
+    md = write_markdown(folder, thru, lats)
+    md_path = folder / "nats-cluster-bench-ns1.md"
+    md_path.write_text(md, encoding="utf-8")
+    html_path = folder / "nats-cluster-bench-ns1.html"
+    pdf_path = folder / "nats-cluster-bench-ns1.pdf"
+    render(md_path, html_path, pdf_path)
+    print(f"wrote {md_path}")
+    print(f"wrote {html_path}")
+    print(f"wrote {pdf_path}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

scripts/create-cluster.sh

@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+# Create three distinct Proxmox LXC guests and start a JetStream cluster on vmbr1.
+# Does not touch host loopback NATS (127.0.0.1:4222) or vmbr0.
+set -euo pipefail
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+# shellcheck disable=SC1091
+. "$ROOT/cluster.env"
+# shellcheck disable=SC1091
+. "$ROOT/scripts/lib-ct.sh"
+ct_require_proxmox
+
+mapfile -t rows < <(printf '%s\n' "$NODES" | awk 'NF==3 {print}')
+[[ ${#rows[@]} -eq 3 ]] || { echo "need exactly 3 nodes in cluster.env" >&2; exit 1; }
+
+declare -a VMIDS NAMES IPS
+for row in "${rows[@]}"; do
+  # shellcheck disable=SC2086
+  set -- $row
+  VMIDS+=("$1"); NAMES+=("$2"); IPS+=("$3")
+done
+
+i=0
+for i in 0 1 2; do
+  ct_ensure "${VMIDS[$i]}" "${NAMES[$i]}" "${IPS[$i]}"
+  ct_bootstrap_user "${VMIDS[$i]}"
+done
+
+# Install nats-server + conf + systemd on each guest
+for i in 0 1 2; do
+  routes=""
+  for j in 0 1 2; do
+    [[ $i -eq $j ]] && continue
+    routes="${routes}    nats-route://${IPS[$j]}:6222"$'\n'
+  done
+  tmpconf="$(mktemp)"
+  NAME="${NAMES[$i]}" IP="${IPS[$i]}" CLUSTER="$CLUSTER_NAME" ROUTES="$routes" \
+    python3 - "$ROOT/conf/nats.conf.tmpl" "$tmpconf" <<'PY'
+import os, pathlib, sys
+t = pathlib.Path(sys.argv[1]).read_text()
+out = t.replace("{{NAME}}", os.environ["NAME"]).replace("{{IP}}", os.environ["IP"]).replace("{{CLUSTER}}", os.environ["CLUSTER"]).replace("{{ROUTES}}", os.environ["ROUTES"])
+pathlib.Path(sys.argv[2]).write_text(out)
+PY
+  sudo pct exec "${VMIDS[$i]}" -- bash -c 'cat > /tmp/nats.conf' < "$tmpconf"
+  sudo pct exec "${VMIDS[$i]}" -- bash -c 'cat > /tmp/nats-server.service' < "$ROOT/systemd/nats-server.service"
+  rm -f "$tmpconf"
+  sudo pct exec "${VMIDS[$i]}" -- bash -lc "
+set -e
+export DEBIAN_FRONTEND=noninteractive
+id nats >/dev/null 2>&1 || useradd -r -s /usr/sbin/nologin nats
+install -d -m 755 -o nats -g nats /var/lib/nats/jetstream /etc/nats
+mv /tmp/nats.conf /etc/nats/nats.conf
+chown root:root /etc/nats/nats.conf
+chmod 644 /etc/nats/nats.conf
+if [[ ! -x /usr/local/bin/nats-server ]]; then
+  curl -fsSL https://github.com/nats-io/nats-server/releases/download/v${NATS_VER}/nats-server-v${NATS_VER}-linux-amd64.tar.gz -o /tmp/nats.tgz
+  tar -xzf /tmp/nats.tgz -C /tmp
+  install -m 0755 /tmp/nats-server-v${NATS_VER}-linux-amd64/nats-server /usr/local/bin/nats-server
+  rm -rf /tmp/nats.tgz /tmp/nats-server-v${NATS_VER}-linux-amd64
+fi
+install -m 644 /tmp/nats-server.service /etc/systemd/system/nats-server.service
+systemctl daemon-reload
+systemctl enable --now nats-server
+"
+  echo "nats-server ${NAMES[$i]} ${IPS[$i]}:4222 cluster ${IPS[$i]}:6222"
+done
+
+echo "cluster client URL: nats://${IPS[0]}:4222,nats://${IPS[1]}:4222,nats://${IPS[2]}:4222"
+echo "lab loopback NATS on the host is unchanged (127.0.0.1:4222)"
+echo "next: bash $ROOT/scripts/test.sh"

scripts/cutover-ns1.sh

@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+# Point NS1 test modules at the 3-node vmbr1 cluster. Does not change MOCK_VERAE or Zapier.
+set -euo pipefail
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+# shellcheck disable=SC1091
+. "$ROOT/client.env"
+export PATH="/usr/sbin:/usr/bin:/bin:$PATH"
+bash "$ROOT/scripts/ensure-streams.sh"
+echo "NATS_URL=$NATS_URL"
+echo "streams ensured. restart keep + fleet on the host after copying overlay/service JSON."

scripts/docs-print.css

@@ -0,0 +1,122 @@
+/* Colored print + screen stylesheet for zapier.georgelambert.org */
+:root {
+  --ink: #171a26;
+  --muted: #5b6178;
+  --line: #d9dce8;
+  --bg: #f4f5fb;
+  --paper: #ffffff;
+  --accent: #4f46e5;
+  --accent-deep: #312e81;
+  --accent-soft: #eef0fe;
+  --ok: #047857;
+  --warn: #8a5a00;
+  --code-bg: #1b1f33;
+  --code-fg: #e8ecff;
+}
+html { background: var(--bg); }
+body {
+  margin: 0 auto;
+  padding: 1.5rem 1.25rem 3rem;
+  max-width: 48rem;
+  font: 15px/1.55 -apple-system, "Segoe UI", Georgia, serif;
+  color: var(--ink);
+  background: var(--paper);
+}
+.doc-banner {
+  background: linear-gradient(160deg, #312e81 0%, #4f46e5 60%, #7c74f0 100%);
+  color: #eef0fe;
+  margin: -1.5rem -1.25rem 1.5rem;
+  padding: 1.1rem 1.25rem 1rem;
+}
+.doc-banner a { color: #fff; }
+.doc-banner .kicker {
+  letter-spacing: 0.12em;
+  text-transform: uppercase;
+  font: 700 10px system-ui, sans-serif;
+  opacity: 0.8;
+}
+.doc-banner h1 { margin: 0.25rem 0 0; font-size: 1.45rem; color: #fff; }
+h1, h2, h3, h4 { color: var(--accent-deep); page-break-after: avoid; }
+h1 { font-size: 1.7rem; }
+h2 {
+  font-size: 1.2rem;
+  border-bottom: 2px solid var(--accent);
+  padding-bottom: 0.2rem;
+  margin-top: 1.6rem;
+}
+h3 { font-size: 1.05rem; color: var(--accent); }
+a { color: var(--accent); }
+p, li { orphans: 3; widows: 3; }
+code {
+  font-family: ui-monospace, Menlo, Consolas, monospace;
+  font-size: 0.86em;
+  background: var(--accent-soft);
+  color: var(--accent-deep);
+  padding: 0.08em 0.28em;
+  border-radius: 4px;
+}
+pre, div.sourceCode, div.sourceCode pre {
+  background: var(--code-bg) !important;
+  color: var(--code-fg) !important;
+  padding: 0.85rem 1rem;
+  border-radius: 10px;
+  overflow: auto;
+  font-size: 0.78rem;
+  line-height: 1.4;
+  page-break-inside: avoid;
+}
+pre code { background: transparent; color: inherit; padding: 0; }
+#title-block-header, header#title-block-header, h1.title { display: none; }
+.doc-banner + h1 { display: none; }
+table {
+  border-collapse: collapse;
+  width: 100%;
+  margin: 0.8rem 0 1.2rem;
+  font-size: 0.9rem;
+  page-break-inside: avoid;
+}
+th, td { border: 1px solid var(--line); padding: 0.38rem 0.55rem; text-align: left; vertical-align: top; }
+th {
+  background: var(--accent);
+  color: #fff;
+  font: 650 12px system-ui, sans-serif;
+}
+tr:nth-child(even) td { background: var(--accent-soft); }
+blockquote {
+  margin: 1rem 0;
+  padding: 0.4rem 0.9rem;
+  border-left: 4px solid var(--accent);
+  background: var(--accent-soft);
+  color: var(--accent-deep);
+}
+img { max-width: 100%; height: auto; border-radius: 8px; page-break-inside: avoid; }
+hr { border: 0; border-top: 1px solid var(--line); }
+ul, ol { padding-left: 1.25rem; }
+nav.site { font: 13px system-ui, sans-serif; margin-bottom: 0.4rem; }
+.source-path { font: 11px ui-monospace, Menlo, monospace; color: var(--muted); }
+
+@page {
+  size: letter;
+  margin: 0.65in 0.7in 0.8in 0.7in;
+  @top-left {
+    content: "Verae Time × Zapier";
+    font: 700 8pt system-ui, sans-serif;
+    color: #4f46e5;
+  }
+  @top-right {
+    content: "zapier.georgelambert.org";
+    font: 8pt system-ui, sans-serif;
+    color: #6b7186;
+  }
+  @bottom-center {
+    content: counter(page) " / " counter(pages);
+    font: 8pt system-ui, sans-serif;
+    color: #6b7186;
+  }
+}
+@media print {
+  html, body { background: #fff; max-width: none; padding: 0; }
+  .doc-banner { margin: 0 0 1rem; border-radius: 8px; -webkit-print-color-adjust: exact; print-color-adjust: exact; }
+  a { text-decoration: none; }
+  th, tr:nth-child(even) td, pre, blockquote, code { -webkit-print-color-adjust: exact; print-color-adjust: exact; }
+}

scripts/ensure-streams.sh

@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+# Create product JetStream streams with replicas=3 on the Proxmox cluster.
+set -euo pipefail
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+# shellcheck disable=SC1091
+. "$ROOT/client.env"
+export PATH="/usr/sbin:/usr/bin:/bin:/usr/local/bin:$PATH"
+VMID="${1:-511}"
+sudo pct exec "$VMID" -- bash -lc "
+set -e
+export DEBIAN_FRONTEND=noninteractive
+export PATH=/usr/local/bin:/usr/bin:/bin
+export NATS_URL=nats://10.10.10.21:4222
+if [[ ! -x /usr/local/bin/nats ]]; then
+  apt-get install -y --no-install-recommends unzip >/dev/null
+  curl -fsSL https://github.com/nats-io/natscli/releases/download/v0.1.6/nats-0.1.6-linux-amd64.zip -o /tmp/natscli.zip
+  rm -rf /tmp/natscli && mkdir -p /tmp/natscli
+  unzip -o /tmp/natscli.zip -d /tmp/natscli >/dev/null
+  BIN=\$(find /tmp/natscli -type f -name nats | head -1)
+  install -m 0755 \"\$BIN\" /usr/local/bin/nats
+fi
+add() {
+  local name=\$1 subj=\$2
+  nats stream info \"\$name\" >/dev/null 2>&1 && return 0
+  nats stream add \"\$name\" --subjects=\"\$subj\" --replicas=3 --storage=file --retention=limits --discard=old --max-msgs=-1 --max-bytes=-1 --max-age=24h --dupe-window=2m --defaults
+}
+add ZAPIER_JOBS 'verae.zapier.jobs.watch'
+add ZAPIER_EVENTS 'verae.zapier.jobs.events'
+add ZAPIER_WEBHOOKS 'verae.zapier.webhooks.deliver'
+add ZAPIER_USAGE 'verae.zapier.usage'
+add VERAE_ARCHIVE 'verae.archive.>'
+nats stream ls
+"
+echo "streams ready on cluster (replicas=3)"

scripts/latency.mjs

@@ -0,0 +1,105 @@
+#!/usr/bin/env node
+/**
+ * Pub→sub round trip through the cluster (two connections).
+ * Usage: NATS_URL=... node latency.mjs [count] [payloadBytes] [publishers] [ping|flood]
+ * ping  = sequential publish-wait (one-message RTT)
+ * flood = publish the batch then drain (queueing under burst)
+ */
+import { connect, headers } from "nats";
+
+const url = process.env.NATS_URL || "nats://10.10.10.21:4222";
+const count = Number(process.argv[2] || 5000);
+const size = Number(process.argv[3] || 128);
+const pubs = Number(process.argv[4] || 1);
+const mode = process.argv[5] || "flood";
+const servers = url.split(",").map((s) => s.trim());
+const subject = `bench.lat.${process.pid}`;
+const payload = new Uint8Array(size);
+
+function pct(sorted, p) {
+  if (!sorted.length) return 0;
+  const i = Math.min(sorted.length - 1, Math.floor((p / 100) * sorted.length));
+  return sorted[i];
+}
+
+const samples = [];
+if (mode === "ping") {
+  const subNc = await connect({ servers, name: "lat-sub" });
+  const pubNc = await connect({ servers, name: "lat-pub" });
+  let resolveOne = null;
+  const sub = subNc.subscribe(subject, { max: count });
+  const consume = (async () => {
+    for await (const m of sub) {
+      const sent = Number(m.headers?.get("t") || 0);
+      samples.push(Number(process.hrtime.bigint() / 1000n) - sent);
+      resolveOne?.();
+    }
+  })();
+  await subNc.flush();
+  for (let i = 0; i < count; i++) {
+    const got = new Promise((r) => {
+      resolveOne = r;
+    });
+    const h = headers();
+    h.set("t", String(process.hrtime.bigint() / 1000n));
+    pubNc.publish(subject, payload, { headers: h });
+    await got;
+  }
+  await consume;
+  await pubNc.close();
+  await subNc.close();
+} else {
+  const subNc = await connect({ servers, name: "lat-sub" });
+  const sub = subNc.subscribe(subject, { max: count });
+  const done = (async () => {
+    for await (const m of sub) {
+      const sent = Number(m.headers?.get("t") || 0);
+      if (sent) samples.push(Number(process.hrtime.bigint() / 1000n) - sent);
+    }
+  })();
+  await subNc.flush();
+  const per = Math.ceil(count / pubs);
+  const publishers = [];
+  for (let p = 0; p < pubs; p++) {
+    publishers.push(
+      (async () => {
+        const nc = await connect({ servers, name: `lat-pub-${p}` });
+        const n = p === pubs - 1 ? count - per * (pubs - 1) : per;
+        for (let i = 0; i < n; i++) {
+          const h = headers();
+          h.set("t", String(process.hrtime.bigint() / 1000n));
+          nc.publish(subject, payload, { headers: h });
+        }
+        await nc.flush();
+        await nc.close();
+      })(),
+    );
+  }
+  await Promise.all(publishers);
+  await done;
+  await subNc.close();
+}
+
+samples.sort((a, b) => a - b);
+const sum = samples.reduce((a, b) => a + b, 0);
+const us = (n) => `${(n / 1000).toFixed(3)}ms`;
+console.log(
+  JSON.stringify({
+    count: samples.length,
+    pubs,
+    size,
+    mode,
+    min_us: samples[0],
+    avg_us: Math.round(sum / samples.length),
+    p50_us: pct(samples, 50),
+    p90_us: pct(samples, 90),
+    p99_us: pct(samples, 99),
+    max_us: samples[samples.length - 1],
+    min: us(samples[0]),
+    avg: us(sum / samples.length),
+    p50: us(pct(samples, 50)),
+    p90: us(pct(samples, 90)),
+    p99: us(pct(samples, 99)),
+    max: us(samples[samples.length - 1]),
+  }),
+);

scripts/lib-ct.sh

@@ -0,0 +1,62 @@
+# shellcheck shell=bash
+# Shared LXC bootstrap for NS1 Proxmox. Does not generate SSH keys if one exists.
+export PATH="/usr/sbin:/usr/bin:/bin:$PATH"
+
+ct_require_proxmox() {
+  if [[ ! -d /etc/pve/nodes ]]; then
+    echo "not a Proxmox host" >&2
+    return 1
+  fi
+  command -v pct >/dev/null || { echo "pct missing" >&2; return 1; }
+}
+
+ct_ensure() {
+  local vmid="$1" hostname="$2" ip="$3"
+  if [[ ! -f "$TEMPLATE" ]]; then
+    echo "missing template $TEMPLATE" >&2
+    return 1
+  fi
+  if ! sudo pct status "$vmid" >/dev/null 2>&1; then
+    echo "pct create $vmid $hostname $ip/24"
+    sudo pct create "$vmid" "$TEMPLATE" \
+      --hostname "$hostname" \
+      --memory "$MEMORY" --cores "$CORES" --swap 256 \
+      --net0 "name=eth0,bridge=${BRIDGE},ip=${ip}/24,gw=${GW},type=veth" \
+      --rootfs "${STORAGE}:${DISK}" \
+      --unprivileged 1 --onboot 1 --nameserver "$DNS" \
+      --features nesting=1 \
+      --ostype ubuntu
+  else
+    echo "CT $vmid already exists"
+  fi
+  sudo pct start "$vmid" 2>/dev/null || true
+  local i
+  for i in $(seq 1 40); do
+    sudo pct exec "$vmid" -- true 2>/dev/null && return 0
+    sleep 2
+  done
+  echo "CT $vmid did not start" >&2
+  return 1
+}
+
+ct_bootstrap_user() {
+  local vmid="$1"
+  local pub=""
+  [[ -f "$HOME/.ssh/id_ed25519.pub" ]] && pub="$(cat "$HOME/.ssh/id_ed25519.pub")"
+  [[ -z "$pub" && -f "$HOME/.ssh/authorized_keys" ]] && pub="$(head -1 "$HOME/.ssh/authorized_keys")"
+  [[ -n "$pub" ]] || { echo "no ssh public key" >&2; return 1; }
+  sudo pct exec "$vmid" -- bash -lc "
+set -e
+export DEBIAN_FRONTEND=noninteractive
+apt-get update -qq
+apt-get install -y --no-install-recommends openssh-server sudo curl ca-certificates xz-utils tar
+id $USER_NAME >/dev/null 2>&1 || useradd -m -s /bin/bash $USER_NAME
+echo '$USER_NAME ALL=(ALL) NOPASSWD:ALL' >/etc/sudoers.d/90-$USER_NAME
+chmod 440 /etc/sudoers.d/90-$USER_NAME
+install -d -m 700 -o $USER_NAME -g $USER_NAME /home/$USER_NAME/.ssh
+grep -qxF '$pub' /home/$USER_NAME/.ssh/authorized_keys 2>/dev/null || echo '$pub' >>/home/$USER_NAME/.ssh/authorized_keys
+chown $USER_NAME:$USER_NAME /home/$USER_NAME/.ssh/authorized_keys
+chmod 600 /home/$USER_NAME/.ssh/authorized_keys
+systemctl enable --now ssh
+"
+}

scripts/ns1-study-methodology.md

@@ -0,0 +1,202 @@
+## 4. Study methodology
+
+### 4.1 Question
+
+On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)?
+
+### 4.2 Hypotheses (stated before the run)
+
+1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica.
+2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated.
+3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting.
+4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count.
+5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound.
+
+### 4.3 Independent variables (what we changed)
+
+| Factor | Levels |
+|--------|--------|
+| Transport | Core NATS pub/sub vs JetStream file replicas=3 |
+| Publisher count | 1, 2, 4, 8 |
+| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 |
+| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) |
+| Payload | 128 B, 1024 B |
+| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) |
+
+### 4.4 Dependent variables (what we recorded)
+
+| Metric | Instrument | Unit |
+|--------|------------|------|
+| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s |
+| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s |
+| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) |
+| Publisher spread | nats min/avg/max **msgs/s** | not delay |
+| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max |
+| Host load | `/proc/loadavg` before and after | load average |
+| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem |
+
+**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`.
+
+### 4.5 Controls and constants
+
+- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`.
+- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`).
+- Bench client is LXC **510**, not a nats-* server.
+- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide.
+- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`).
+- No TLS, no nkeys, no account isolation (isolation is `vmbr1`).
+- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder.
+
+### 4.6 Procedure
+
+1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise.
+2. Snapshot host load, memory, LXC configs, and each nats `varz`.
+3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB).
+4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file.
+5. Copy `latency.mjs` into 510; ping then flood at several batch sizes.
+6. Snapshot host/`varz` again.
+7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**.
+
+No publish, subscribe, chart, or PDF process runs on the operator laptop for this study.
+
+### 4.7 Instrumentation path
+
+```text
+[NS1 host 70.88.205.138]
+   study-on-ns1.sh  (bash + python3)
+        |
+        | sudo pct exec 510
+        v
+[LXC 510 verae-px-worker 10.10.10.20]
+   nats bench / node latency.mjs
+        |
+        | NATS client protocol to
+        v
+[LXC 511/512/513  10.10.10.21-23 :4222]
+   nats-server -js  cluster routes :6222
+```
+
+The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`.
+
+### 4.8 Threats to validity
+
+| Threat | Effect on numbers |
+|--------|-------------------|
+| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. |
+| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. |
+| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. |
+| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. |
+| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. |
+| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. |
+| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. |
+| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. |
+
+### 4.9 Ethics / safety
+
+Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket.
+
+---
+
+## 5. Suggestions for fine-tuning
+
+These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way.
+
+### 5.1 Treat JetStream as the product limiter
+
+Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out.
+
+### 5.2 Split storage class by stream
+
+| Stream | Suggested store | Why |
+|--------|-----------------|-----|
+| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad |
+| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both |
+| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer |
+| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry |
+| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive |
+
+Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert.
+
+### 5.3 Give JetStream real disks
+
+Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain.
+
+- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`.
+- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure.
+- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it.
+- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.”
+
+### 5.4 Isolate the nats CTs from the rest of NS1
+
+Host load on this box is often already several. Pin:
+
+- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes.
+- Memory high enough that file-backed streams stay cache-hot for the working set.
+- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync.
+
+Re-run this study after pinning; H1/H2 should move more than ping.
+
+### 5.5 Consumer and mailbox tuning (delay H4)
+
+Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT.
+
+- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM.
+- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber.
+- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped.
+- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT.
+
+### 5.6 Publisher-side batching in middleware
+
+A timestamp job is one small JSON. 16k msgs/s is ample. Still:
+
+- Avoid per-byte publishes; one message per job/event.
+- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs).
+- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only.
+
+### 5.7 nats-server knobs worth measuring (A/B with this script)
+
+| Knob | Why try it |
+|------|------------|
+| `max_payload` | Keep default unless archive puts grow |
+| `write_deadline` | Slow consumer protection for webhooks |
+| `max_pending` | Bound memory on a stuck Zapier hook |
+| `max_connections` | Fleet workers + keep + middleware |
+| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT |
+| `max_outstanding_catchup` | Replica restart after a nats-c blip |
+| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT |
+
+Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99.
+
+### 5.8 Network
+
+- Keep NATS off `vmbr0`. No change.
+- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load).
+- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus.
+
+### 5.9 Security cost (when nkeys/mTLS flip)
+
+`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement.
+
+### 5.10 Operational fine-tuning (lag, not peak msgs/s)
+
+1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`.
+2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense.
+3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost.
+4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated.
+
+### 5.11 What not to tune
+
+- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge.
+- Do not treat flood 400 ms as “cluster RTT.” Fix consumers.
+- Do not load-test on `ZAPIER_*` streams.
+- Do not bind client NATS to `0.0.0.0` on `vmbr0`.
+
+### 5.12 Recommended next experiments (same method, one change each)
+
+1. CPU pin nats-a/b/c → re-run JS 1p + ping.
+2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder).
+3. Distinct `store_dir` disks per node.
+4. nkeys on, same ladder.
+5. Three hardware boxes, same `cluster.env` IPs updated.
+
+Each experiment should produce a new `results/<utc>/` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory.

scripts/status.sh

@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+set -euo pipefail
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+# shellcheck disable=SC1091
+. "$ROOT/cluster.env"
+export PATH="/usr/sbin:/usr/bin:/bin:$PATH"
+printf '%s\n' "$NODES" | awk 'NF==3 {print}' | while read -r vmid name ip; do
+  st="$(sudo pct status "$vmid" 2>/dev/null || echo missing)"
+  js="$(sudo pct exec "$vmid" -- curl -fsS --max-time 2 http://127.0.0.1:8222/varz 2>/dev/null || echo '{}')"
+  echo "$vmid $name $ip $st"
+  python3 -c "
+import json,sys
+try:
+  d=json.loads(sys.argv[1])
+except Exception:
+  print('  nats down')
+  raise SystemExit
+print('  server_name', d.get('server_name'), 'cluster', (d.get('cluster') or {}).get('name'), 'routes', len((d.get('cluster') or {}).get('urls') or d.get('connect_urls') or []))
+print('  jetstream', bool(d.get('jetstream')), 'port', d.get('port'), 'host', d.get('host'))
+" "$js" 2>/dev/null || echo "  nats down"
+done
+echo "host loopback still: $(ss -lnt | grep '127.0.0.1:4222' && echo up || echo down)"

scripts/study-on-ns1.sh

@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
+# Full message-speed study. Must run ON NS1.GEORGELAMBERT.ORG (70.88.205.138).
+# Orchestration, nats bench (via pct into LXC 510), charts, HTML, and PDF all
+# happen on this host. The laptop is not in the measurement path.
+set -euo pipefail
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+export PATH="/usr/sbin:/usr/bin:/bin:/usr/local/bin:$PATH"
+
+HOST="$(hostname -f 2>/dev/null || hostname)"
+case "$HOST" in
+  NS1.GEORGELAMBERT.ORG|NS1|ns1.georgelambert.org|ns1) ;;
+  *)
+    echo "refusing: study-on-ns1.sh must run on NS1.GEORGELAMBERT.ORG (70.88.205.138), got '$HOST'" >&2
+    exit 1
+    ;;
+esac
+
+# shellcheck disable=SC1091
+. "$ROOT/client.env"
+CLIENT_VMID="${CLIENT_VMID:-510}"
+STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
+OUT="${BENCH_OUT:-$ROOT/results/$STAMP}"
+mkdir -p "$OUT"
+
+dump_env() {
+  local tag="$1"
+  local f="$OUT/host-$tag.txt"
+  {
+    echo "execution_host=NS1.GEORGELAMBERT.ORG"
+    echo "execution_ip=70.88.205.138"
+    echo "hostname=$(hostname)"
+    echo "utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
+    echo "whoami=$(whoami)"
+    echo "pwd=$(pwd)"
+    echo "uname=$(uname -a)"
+    echo "nproc=$(nproc)"
+    echo "loadavg=$(cat /proc/loadavg)"
+    echo "client_vmid=$CLIENT_VMID"
+    echo "nats_url=$NATS_URL"
+    echo "--- free ---"
+    free -h
+    echo "--- pct list ---"
+    sudo pct list
+    for v in 510 511 512 513; do
+      echo "--- pct config $v ---"
+      sudo pct config "$v" | grep -E '^(hostname|cores|memory|swap|rootfs|mp|net)' || true
+    done
+  } >"$f"
+  python3 - "$OUT" "$tag" <<'PY'
+import json, sys, urllib.request
+from pathlib import Path
+out, tag = Path(sys.argv[1]), sys.argv[2]
+nodes = []
+for vmid, name in (("511", "nats-a"), ("512", "nats-b"), ("513", "nats-c")):
+    raw = ""
+    try:
+        import subprocess
+        raw = subprocess.check_output(
+            ["sudo", "pct", "exec", vmid, "--", "curl", "-fsS", "--max-time", "3", "http://127.0.0.1:8222/varz"],
+            text=True,
+        )
+        d = json.loads(raw)
+        nodes.append({
+            "vmid": vmid,
+            "name": name,
+            "server_name": d.get("server_name"),
+            "host": d.get("host"),
+            "port": d.get("port"),
+            "connections": d.get("connections"),
+            "in_msgs": d.get("in_msgs"),
+            "out_msgs": d.get("out_msgs"),
+            "in_bytes": d.get("in_bytes"),
+            "out_bytes": d.get("out_bytes"),
+            "cpu": d.get("cpu"),
+            "cores": d.get("cores"),
+            "mem": d.get("mem"),
+            "jetstream": bool(d.get("jetstream")),
+        })
+    except Exception as e:
+        nodes.append({"vmid": vmid, "name": name, "error": str(e)})
+(out / f"varz-{tag}.json").write_text(json.dumps(nodes, indent=2) + "\n", encoding="utf-8")
+PY
+}
+
+echo "NS1 study $STAMP  out=$OUT"
+dump_env before
+BENCH_OUT="$OUT" CLIENT_VMID="$CLIENT_VMID" bash "$ROOT/scripts/bench.sh"
+dump_env after
+python3 "$ROOT/scripts/build-ns1-study-report.py" "$OUT"
+echo "NS1 study complete $OUT"
+ls -la "$OUT"/nats-cluster-bench-ns1.* "$OUT"/charts 2>/dev/null || ls -la "$OUT"

scripts/test.sh

@@ -0,0 +1,52 @@
+#!/usr/bin/env bash
+# Local syntax check always. Live cluster check when pct is present.
+set -euo pipefail
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+bash -n "$ROOT/scripts/lib-ct.sh"
+bash -n "$ROOT/scripts/create-cluster.sh"
+bash -n "$ROOT/scripts/status.sh"
+bash -n "$ROOT/scripts/bench.sh"
+bash -n "$ROOT/scripts/study-on-ns1.sh"
+grep -q 'host: {{IP}}' "$ROOT/conf/nats.conf.tmpl"
+grep -qv '0.0.0.0' "$ROOT/conf/nats.conf.tmpl"
+if [[ ! -d /etc/pve/nodes ]]; then
+  echo "OK (syntax; not on Proxmox)"
+  exit 0
+fi
+# shellcheck disable=SC1091
+. "$ROOT/cluster.env"
+export PATH="/usr/sbin:/usr/bin:/bin:$PATH"
+mapfile -t rows < <(printf '%s\n' "$NODES" | awk 'NF==3 {print}')
+ready=0
+for row in "${rows[@]}"; do
+  # shellcheck disable=SC2086
+  set -- $row
+  vmid=$1 name=$2 ip=$3
+  js="$(sudo pct exec "$vmid" -- curl -fsS --max-time 3 http://127.0.0.1:8222/varz 2>/dev/null || true)"
+  echo "$js" | grep -q '"jetstream"' && ready=$((ready + 1)) || echo "not ready $name"
+done
+[[ $ready -eq 3 ]] || { echo "cluster not fully up ($ready/3)" >&2; exit 1; }
+
+# nats CLI on first node
+first="$(echo "${rows[0]}" | awk '{print $1}')"
+sudo pct exec "$first" -- bash -lc '
+set -e
+export DEBIAN_FRONTEND=noninteractive
+if [[ ! -x /usr/local/bin/nats ]]; then
+  apt-get install -y --no-install-recommends unzip >/dev/null
+  curl -fsSL https://github.com/nats-io/natscli/releases/download/v0.1.6/nats-0.1.6-linux-amd64.zip -o /tmp/natscli.zip
+  rm -rf /tmp/natscli && mkdir -p /tmp/natscli
+  unzip -o /tmp/natscli.zip -d /tmp/natscli >/dev/null
+  BIN=$(find /tmp/natscli /tmp -maxdepth 3 -type f -name nats | head -1)
+  test -n "$BIN"
+  install -m 0755 "$BIN" /usr/local/bin/nats
+fi
+IP=$(hostname -I | awk "{print \$1}")
+export NATS_URL=nats://$IP:4222
+export PATH=/usr/local/bin:/usr/bin:/bin
+nats stream rm VERAE_PX_TEST --force >/dev/null 2>&1 || true
+nats stream add VERAE_PX_TEST --subjects="verae.px.test" --replicas=3 --storage=file --retention=limits --discard=old --max-msgs=-1 --max-bytes=-1 --max-age=1h --dupe-window=2m --defaults
+nats pub verae.px.test cluster-ok
+nats stream info VERAE_PX_TEST
+'
+echo "OK live cluster (3/3 + replicas=3 stream)"

systemd/nats-server.service

@@ -0,0 +1,16 @@
+[Unit]
+Description=NATS JetStream (Verae cluster node)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=nats
+Group=nats
+ExecStart=/usr/local/bin/nats-server -c /etc/nats/nats.conf
+LimitNOFILE=65536
+Restart=on-failure
+RestartSec=2
+
+[Install]
+WantedBy=multi-user.target