Snapshot of zapier-decisions (maximized NS1 NATS study)

reports/nats-cluster-bench.md

@@ -0,0 +1,312 @@
+**Progress report** · 2026-09-12 · run `20260912T045131Z` (UTC)
+
+This is the full write-up of the test-environment NATS cluster bench: what was measured, how, the numbers, the charts, and what they mean for Verae Time × Zapier. Short tables also live in [`verae-nats-cluster/BENCH.md`](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md). Raw logs and CSVs are in that repo under `results/20260912T045131Z/`.
+
+---
+
+## 1. Executive summary
+
+The test cluster is three JetStream nodes on private `vmbr1` (LXC 511–513). The bench client is a **fourth** guest (LXC 510), so the numbers are cluster-plus-network, not a process talking to itself on loopback.
+
+Two different systems were measured, on purpose:
+
+| System | What it is | What we got |
+|--------|------------|-------------|
+| **Core NATS** | Fire-and-forget pub/sub. No disk, no replica ack. | About **0.75–2.0 million msgs/s** at 128 B, depending on fan-out. At 1 KiB, about **630k msgs/s** and **~616 MB/s** aggregate. |
+| **JetStream file, replicas=3** | Durable, replicated — **this is what product streams use**. | About **16k durable 128 B pubs/s**, about **13.5k** at 1 KiB. Pull consume keeps up with publish at ~11k msgs/s each side. |
+| **Ping delay** | One message at a time, publish then wait. | **avg 0.307 ms**, **p99 0.734 ms**, max 2.76 ms (1k × 128 B). |
+| **Flood delay** | Publishers dump a batch; subscriber drains. | **150–505 ms**. That is **queueing under burst**, not wire time. |
+
+**For this product:** timestamp jobs, job events, webhooks, and archive puts go through JetStream r=3. Plan capacity against **~16k durable msgs/s** on this stand, not the million-msg core numbers. A quiet job-event hop is a fraction of a millisecond. If a mailbox falls behind, delay jumps into hundreds of milliseconds — that is the flood column.
+
+Core NATS is still useful: it is the ceiling for non-durable fan-out on this host, and it shows `vmbr1` and the nats-server processes are not the JetStream bottleneck. JetStream is.
+
+---
+
+## 2. Why this test exists
+
+The lab cut the test environment over to the three-node cluster. Before treating that cluster as the message fabric for keep, fleet, middleware, billing, and archive workers, we needed:
+
+1. **Throughput at several loads** — one publisher vs many, 128 B vs 1 KiB, core vs durable.
+2. **Delay characteristics** — both the quiet path (one message RTT) and the overloaded path (burst into a mailbox).
+3. **A client that is not a nats-* server** — otherwise we would be measuring loopback on the broker.
+
+This is a **lab stand on one Proxmox host**, not three metal boxes. It answers “is this cluster in the right order of magnitude for our traffic?” It does not replace a soak test on dedicated disks.
+
+---
+
+## 3. Topology
+
+```text
+                    vmbr1  10.10.10.0/24  (not on vmbr0, not public)
+                    -----------------------------------------------
+  LXC 510           LXC 511            LXC 512            LXC 513
+  verae-px-worker   nats-a             nats-b             nats-c
+  10.10.10.20       10.10.10.21        10.10.10.22        10.10.10.23
+  bench client      :4222 client       :4222              :4222
+                    :6222 routes       :6222              :6222
+                    :8222 loopback     :8222              :8222
+```
+
+- Cluster name: `verae`. Each server has two routes to the other two.
+- Client URL: `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222`
+- HTTP monitor is **loopback :8222** inside each guest. Zapier cloud never talks to NATS.
+- Product streams already on this cluster (`ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE`) use **file** storage and **replicas=3**. The JetStream bench used the same settings on a throwaway stream `benchstream`.
+- Host `127.0.0.1:4222` is still listening on NS1; **clients no longer use it**.
+
+Credits for the stack: Scott Lindsey, George Lambert, NATS.IO, Grok-Code.
+
+---
+
+## 4. Method
+
+### 4.1 Tools
+
+| Piece | Role |
+|-------|------|
+| `nats` CLI **0.1.6** | Throughput (`nats bench --no-progress --csv`). Its min/avg/max are **publisher rate spread**, not delay. |
+| `scripts/latency.mjs` | Two connections, header timestamp `t`, delay = receive time − send time. |
+| `scripts/bench.sh` | Runs the ladder from NS1 via `pct exec` on VMID 510. |
+| `scripts/bench-report.py` | Turns logs into the short `BENCH.md` table. |
+
+Re-run on NS1, from `verae-nats-cluster`:
+
+```bash
+bash scripts/bench.sh
+```
+
+### 4.2 Load ladder
+
+**Core NATS** (subject `bench.core.*`):
+
+| Run | Publishers | Subscribers | Messages | Payload |
+|-----|------------|-------------|----------|---------|
+| `core-1p1s-50k-128` | 1 | 1 | 50,000 | 128 B |
+| `core-4p4s-100k-128` | 4 | 4 | 100,000 | 128 B |
+| `core-8p8s-200k-128` | 8 | 8 | 200,000 | 128 B |
+| `core-4p4s-50k-1k` | 4 | 4 | 50,000 | 1024 B |
+
+**JetStream** (`--js --storage file --replicas 3 --stream benchstream`). The stream is deleted between loads so the name never collides:
+
+| Run | Shape | Messages | Payload |
+|-----|-------|----------|---------|
+| `js-1p-20k-128-r3` | 1 publisher | 20,000 | 128 B |
+| `js-4p-50k-128-r3` | 4 publishers | 50,000 | 128 B |
+| `js-4p-20k-1k-r3` | 4 publishers | 20,000 | 1024 B |
+| `js-2p2s-20k-128-r3` | 2 pub + 2 pull sub | 20,000 | 128 B |
+
+**Delay** (core subjects, two connections):
+
+| Run | Mode | Count | Pubs | Payload |
+|-----|------|-------|------|---------|
+| `lat-ping-1k-128` | **ping** — publish, wait for that message, repeat | 1,000 | 1 | 128 B |
+| `lat-1p-5k-128` | **flood** — publish all, then drain | 5,000 | 1 | 128 B |
+| `lat-4p-10k-128` | flood | 10,000 | 4 | 128 B |
+| `lat-8p-20k-128` | flood | 20,000 | 8 | 128 B |
+| `lat-4p-5k-1k` | flood | 5,000 | 4 | 1024 B |
+
+Ping answers “how long does one quiet hop take?” Flood answers “what happens to the last message if we burst N messages into a mailbox?” Those are different questions. Mixing them is how 0.3 ms and 400 ms get confused.
+
+### 4.3 How to read nats bench columns
+
+- **Pub msgs/s** — rate at which publishers finished their share.
+- **Sub msgs/s** — rate at which subscribers finished. With several subscribers on the same subject, core NATS **fans out**, so sub rate can exceed pub rate.
+- **Aggregate msgs/s** — nats CLI `NATS Pub/Sub stats` line (pub+sub work in one number). Useful as a headline; do not treat it as “the network carried this many unique messages.”
+- Empty JetStream sub cells mean that run was publish-only (durable write, no consumer in the same process).
+
+---
+
+## 5. Throughput results
+
+### 5.1 Core NATS
+
+![Core NATS throughput at four loads](charts/core-throughput.png)
+
+| Run | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
+|-----|------------------|------------|----------|------------|----------|
+| `core-1p1s-50k-128` | 1,200,836 | 791,094 | 96.57 | 747,461 | 91.24 |
+| `core-4p4s-100k-128` | 1,521,256 | 316,312 | 38.61 | 1,299,634 | 158.65 |
+| `core-8p8s-200k-128` | 2,007,937 | 333,957 | 40.77 | 1,790,736 | 218.60 |
+| `core-4p4s-50k-1k` | 630,460 | 247,747 | 241.94 | 510,216 | 498.26 |
+
+**What this chart is saying.** Adding subscribers raises **aggregate** and **sub** rates because each published message is delivered to every subscriber. Publish rate does **not** climb the same way: 1 publisher at 128 B already pushes ~791k msgs/s; 4 and 8 publishers sit around 310–335k msgs/s **each process slower**, while fan-out on the sub side goes to 1.3M then 1.8M.
+
+That publisher slowdown is expected on this stand. The four/eight publisher processes and the four/eight subscribers all run **inside one LXC** (510) against three broker LXCs on the **same Proxmox CPU and `vmbr1`**. Per-publisher logs show a wide spread (example, 4p core 128 B: 79k–524k msgs/s among the four pubs). That is CPU scheduling and client-side contention, not a NATS cluster that only has one fast node.
+
+**1:1 at 128 B** is the cleanest core number: **~791k pub, ~747k sub, ~1.20M aggregate**. The cluster and the bridge can move three-quarter-million small messages per second fire-and-forget from a single client pair.
+
+### 5.2 Payload size (core)
+
+![Core NATS 128 B vs 1 KiB](charts/payload-size.png)
+
+Same 4p4s shape, two sizes:
+
+| Payload | Aggregate msgs/s | Aggregate MB/s | Pub msgs/s | Sub msgs/s |
+|---------|------------------|----------------|------------|------------|
+| 128 B | 1,521,256 | 185.70 | 316,312 | 1,299,634 |
+| 1 KiB | 630,460 | 615.68 | 247,747 | 510,216 |
+
+Message rate falls; **byte rate rises** (186 MB/s → 616 MB/s aggregate). We are leaving the “tiny message, CPU/syscall bound” region and entering “copying bytes across `vmbr1`.” Job JSON and archive metadata sit nearer 128 B–1 KiB than megabyte blobs (blobs are HTTP/WORM, not NATS payloads).
+
+### 5.3 JetStream r=3 file
+
+![JetStream durable publish rate](charts/js-throughput.png)
+
+| Run | Pub msgs/s | Pub MB/s | Sub msgs/s | Notes |
+|-----|------------|----------|------------|-------|
+| `js-1p-20k-128-r3` | **16,155** | 1.97 | — | publish-only |
+| `js-4p-50k-128-r3` | **16,607** | 2.03 | — | four pubs, same ceiling |
+| `js-4p-20k-1k-r3` | **13,493** | 13.18 | — | 1 KiB still disk/replica bound |
+| `js-2p2s-20k-128-r3` | 10,965 | 1.34 | 10,942 | pull consumers keep up |
+
+**Four publishers do not make JetStream four times faster.** 1p and 4p at 128 B are both ~16k msgs/s. The limiter is **synchronous replication to three file-backed replicas**, not client parallelism. That is the result we wanted to see: the bench stream is behaving like a replicated log, not like core fan-out.
+
+Pull consume (`js-2p2s`) is slightly slower on publish (~11k) because the same run is also reading. Pub and sub stay matched (10,965 vs 10,942): the consumer is not the straggler.
+
+1 KiB durable write is ~13.5k msgs/s (~13.2 MB/s). Bytes go up; message rate dips only a little. JetStream here is **ack/fdatasync/replica** bound, not payload-copy bound, in this size range.
+
+### 5.4 Core vs JetStream (same client, same cluster)
+
+![Core vs JetStream publish rate, log scale](charts/core-vs-js.png)
+
+The log scale is required: core publish is **~15–50×** JetStream publish on this stand.
+
+| Shape | Core pub msgs/s | JS r=3 file pub msgs/s | Ratio |
+|-------|-----------------|------------------------|-------|
+| 1 publisher, 128 B | 791,094 | 16,155 | ~49× |
+| 4 publishers, 128 B | 316,312 | 16,607 | ~19× |
+| 4 publishers, 1 KiB | 247,747 | 13,493 | ~18× |
+
+This is not JetStream “losing.” Core is allowed to forget a message the instant the server accepts it. JetStream on file with replicas=3 must **record it on a majority** before the publish acks. Our product streams (`ZAPIER_*`, `VERAE_ARCHIVE`) chose that trade on purpose: a job event that survives one LXC dying is worth ~16k msgs/s instead of ~800k.
+
+If we ever need core-like rates for a signal that may drop, that signal should not be on a replicated file stream.
+
+---
+
+## 6. Delay results
+
+![Delay percentiles, log scale](charts/delay-percentiles.png)
+
+| Run | Kind | Count | min | avg | p50 | p90 | p99 | max |
+|-----|------|-------|-----|-----|-----|-----|-----|-----|
+| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 0.254 ms | **0.307 ms** | 0.286 ms | 0.332 ms | **0.734 ms** | 2.763 ms |
+| `lat-1p-5k-128` | flood | 5000 | 149.3 ms | 238.6 ms | 248.8 ms | 274.3 ms | 279.4 ms | 279.7 ms |
+| `lat-4p-5k-1k` | flood | 5000 | 155.1 ms | 211.7 ms | 217.6 ms | 223.3 ms | 227.8 ms | 228.4 ms |
+| `lat-4p-10k-128` | flood | 10000 | 174.2 ms | 263.2 ms | 266.7 ms | 299.1 ms | 304.2 ms | 304.5 ms |
+| `lat-8p-20k-128` | flood | 20000 | 304.6 ms | 453.7 ms | 466.3 ms | 499.9 ms | 505.1 ms | 505.6 ms |
+
+The dashed line on the chart is 1 ms. Only **ping** lives there.
+
+### 6.1 Ping — the quiet hop
+
+One publisher, one subscriber, two connections, wait for each message before sending the next.
+
+- **min 0.254 ms** — guest → `vmbr1` → a nats-server → `vmbr1` → guest.
+- **p50 0.286 ms / avg 0.307 ms** — typical.
+- **p99 0.734 ms** — still under a millisecond.
+- **max 2.763 ms** — one outlier in 1,000 samples (GC, scheduler, or a slow route). Not the tail we design for.
+
+A middleware `jobs.watch` publish followed by a waiter on `jobs.events` is this shape when the poller is keeping up. Compared with HTTPS to Zapier (tens to hundreds of milliseconds) or a live Verae `GET /api/status/{jobId}`, NATS RTT is noise.
+
+### 6.2 Flood — queueing under burst
+
+Publishers write the whole batch as fast as they can, then the subscriber drains. Each message’s delay is “how long was I in the buffer before the subscriber got to me?”
+
+That is why:
+
+- **min is already ~150–300 ms** — even the first messages wait behind a burst that filled the socket/client queue.
+- **p50 ≈ p99 ≈ max** — a queue drain has a tight distribution: everyone waits for roughly the same backlog.
+- **8p × 20k is ~450 ms avg** — twice the messages of 4p × 10k, roughly twice the wait. Linear in backlog, not in cluster diameter.
+
+Flood is **not** a measurement of NATS being slow. The ping column proves the hop is ~0.3 ms. Flood is a measurement of **what operators will see if a consumer stalls** (job-events mailbox, webhook deliver, archive reply). Backlog time ≈ `queued_messages / consume_rate`.
+
+### 6.3 1 KiB flood vs 128 B flood
+
+4 publishers, 5k messages at 1 KiB: avg **212 ms**, slightly **faster** than 4p 10k × 128 B (263 ms) because the **count is half**, even though each message is 8× larger. Again: delay here tracks **how many messages are queued**, not payload size, in this range.
+
+---
+
+## 7. What this means for Verae × Zapier
+
+Product subjects on this cluster:
+
+| Address | Kind | Bench analogue |
+|---------|------|----------------|
+| `verae.zapier.jobs.watch` | work queue (JetStream) | JS durable pub ~16k/s |
+| `verae.zapier.jobs.events` | events | JS + ping if waiters keep up; flood if they do not |
+| `verae.zapier.webhooks.deliver` | work queue | JS durable |
+| `verae.zapier.usage` | optional | JS durable |
+| `verae.billing.*` | request-reply | ping (quiet RTT) |
+| `verae.archive.put` / `query` / `reply.*` | JetStream + broadcast query | JS durable; query fan-out is closer to core but still JS-backed puts |
+
+**Capacity.** 16k durable 128 B pubs/s is **~1.4×10⁹ messages/day** if you could fill the pipe. We will not. Zapier HTTPS, live `api.veraetime.net`, WORM bloom checks, and human Zap runs sit far below that. This cluster is not the product bottleneck on NS1.
+
+**Latency budget.** A timestamp wait is: HTTP in → NATS watch → poll Verae → NATS event → HTTP out (or REST Hook). The NATS pieces are **sub-millisecond** when caught up. Do not spend time “optimizing NATS RTT” until Zapier/Verae HTTP is in the same band.
+
+**Backlogs.** The failure mode that *does* show up in these numbers is flood delay. If webhook-deliver or job-events consumers pause (keep stopped, replica floor, a blocked HTTPS post to `hooks.zapier.com`), waiters will see **hundreds of milliseconds to seconds** of queue time. Fleet replica floors and keep exist to prevent that, not because 0.3 ms is too slow.
+
+**Hardware move.** Same three configs, three boxes, private NIC. Expect:
+
+- Core numbers to change with NIC and CPU (maybe up, maybe down).
+- JetStream numbers to change **more**, because they are disk + fsync + replica RTT. Distinct SSDs should help; a slow shared datastore would hurt.
+- Ping RTT to grow by whatever the real NIC and switch add (still likely low milliseconds on a LAN).
+
+---
+
+## 8. Limits of this measurement
+
+1. **One Proxmox host.** LXC 510–513 share cores, memory, and the host’s disk. Replica=3 on file is **three files on the same underlying storage**, not three failure domains. HA of “one disk dies” is **not** proven. HA of “one LXC process dies” is the actual claim.
+2. **Short runs.** Tens of thousands of messages, seconds of wall time. No compaction, no multi-hour page-cache eviction, no snapshot/restore during load.
+3. **No TLS, no nkeys.** `verae-nats-accounts` is still a sketch. Auth would add CPU; it would not turn 16k into 800k.
+4. **One bench client.** All publishers live in 510. A fleet of workers on several CTs might publish more into JetStream until disk/replicas saturate — the 1p vs 4p JS result says that saturation is already ~16k from one CT.
+5. **nats 0.1.6** does not report delay. Anyone reading `min | avg | max msgs` on a bench log as microseconds will get the wrong story. Delay is only `latency.mjs`.
+6. **Core aggregate ≠ unique messages.** Fan-out double-counts. Use pub or sub columns when comparing to JetStream.
+7. **Not a Zapier or Verae API bench.** Those are still blocked on operator login / live credentials.
+
+---
+
+## 9. How to reproduce
+
+On NS1 (Proxmox), from the `verae-nats-cluster` checkout:
+
+```bash
+bash scripts/status.sh    # 3/3 JetStream
+bash scripts/bench.sh     # writes results/<utc>/ and BENCH.md
+```
+
+The client VMID defaults to **510**. Override with `CLIENT_VMID=…`. `NATS_URL` comes from `client.env`.
+
+Rebuild this progress report (charts + HTML + PDF) from the monorepo:
+
+```bash
+python3 packages/zapier-decisions/scripts/build-nats-bench-report.py
+```
+
+---
+
+## 10. Appendix — environment and files
+
+| Item | Value |
+|------|-------|
+| Run stamp | `20260912T045131Z` |
+| Client | LXC 510 `verae-px-worker` `10.10.10.20` |
+| Servers | 511/512/513 `nats-a/b/c` `10.10.10.21–23` |
+| nats CLI | 0.1.6 linux-amd64 |
+| JS storage | file, replicas=3, stream `benchstream` (deleted between loads) |
+| Isolation | `vmbr1` only; no `0.0.0.0` client bind |
+| Short tables | [BENCH.md](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md) |
+| Raw logs | `packages/verae-nats-cluster/results/20260912T045131Z/` |
+| This report | `packages/zapier-decisions/reports/nats-cluster-bench.{md,html,pdf}` |
+
+Publisher rate spread (nats CLI, msgs/s, **not** delay):
+
+| Run | min | avg | max |
+|-----|-----|-----|-----|
+| core-4p4s-100k-128 pub | 79,260 | 257,805 | 524,453 |
+| core-8p8s-200k-128 pub | 41,744 | 71,488 | 152,536 |
+| core-4p4s-50k-1k pub | 61,936 | 110,271 | 176,262 |
+| js-4p-50k-128-r3 pub | 4,154 | 5,176 | 6,628 |
+| js-4p-20k-1k-r3 pub | 3,373 | 4,063 | 5,121 |
+| js-2p2s-20k-128-r3 pub | 5,485 | 7,081 | 8,678 |
+
+Wide core spreads are the single-client-CT effect described in §5.1. JetStream spreads are narrow and low — every publisher is waiting on the same replicated write path.

reports/ns1-host/nats-cluster-bench-ns1.md

@@ -0,0 +1,334 @@
+**Progress report (second study)** · run `20260912T051237Z` (UTC)
+
+> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): the orchestrator (`study-on-ns1.sh`), `nats bench`, `latency.mjs` (inside LXC 510 on this hypervisor), charting (`matplotlib`), and HTML/PDF (`pandoc` + `weasyprint`). The operator laptop did **not** publish, subscribe, draw charts, or render the PDF. Traffic stayed on `vmbr1` from LXC **510** to `nats-a/b/c` (**511–513**).
+
+This is a full methodology write-up plus the numbers from that on-host run. The earlier report (`nats-cluster-bench`, run `20260912T045131Z`) used the same cluster but was **orchestrated and rendered off-box**. Use this document when you need “it was all run on 138.”
+
+---
+
+## 1. Executive summary
+
+| Item | This NS1-host run |
+|------|-------------------|
+| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `marchon` |
+| Bench client | LXC 510 `verae-px-worker` |
+| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` |
+| Client URL | `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` |
+| Host load before | `8.59 8.39 8.15 5/3843 4096277` |
+| Host load after | `8.67 8.33 8.15 9/3863 4114865` |
+| Core 1p1s 128 B pub | 502,502 msgs/s |
+| JetStream 1p 128 B r=3 | 7,393 durable pubs/s |
+| Ping p50 / p99 | 0.395ms / 1.377ms |
+
+Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst.
+
+---
+
+## 2. Where it ran (and where it did not)
+
+```text
+Operator laptop  ──ssh──►  NS1.GEORGELAMBERT.ORG  70.88.205.138
+                              study-on-ns1.sh
+                              python3 build-ns1-study-report.py
+                              sudo pct exec 510 ──► nats bench / latency.mjs
+                                                    │
+                                                    ▼ vmbr1
+                                              10.10.10.21-23 :4222
+```
+
+- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510).
+- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file.
+
+---
+
+## 3. Results (this run)
+
+### Host and brokers
+
+**Before**
+
+| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream |
+|------|------|-------------|---------|----------|-----|-------|---------|-----------|
+| nats-a | 511 | 2 | 1,721,203 | 4,933,759 | 1 | 1 | 26,398,720 | True |
+| nats-b | 512 | 2 | 1,613,205 | 3,513,096 | 0 | 1 | 23,097,344 | True |
+| nats-c | 513 | 2 | 1,760,973 | 4,048,582 | 2 | 1 | 23,867,392 | True |
+
+**After**
+
+| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream |
+|------|------|-------------|---------|----------|-----|-------|---------|-----------|
+| nats-a | 511 | 2 | 2,097,596 | 5,910,161 | 1 | 1 | 38,273,024 | True |
+| nats-b | 512 | 2 | 2,121,816 | 4,746,539 | 1 | 1 | 41,361,408 | True |
+| nats-c | 513 | 2 | 2,241,366 | 5,053,978 | 1 | 1 | 38,162,432 | True |
+
+nproc=40 · uname=`Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux`
+
+### Throughput
+
+| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
+|-----|------|------------------|------------|----------|------------|----------|
+| `core-1p1s-50k-128` | core pub/sub | 776,331 | 502,502 | 61.34 | 420,976 | 51.39 |
+| `core-4p4s-100k-128` | core pub/sub | 1,278,454 | 379,985 | 46.38 | 1,078,910 | 131.70 |
+| `core-4p4s-50k-1k` | core pub/sub | 581,464 | 209,545 | 204.63 | 482,645 | 471.33 |
+| `core-8p8s-200k-128` | core pub/sub | 2,065,217 | 333,731 | 40.74 | 1,847,242 | 225.49 |
+| `js-1p-20k-128-r3` | jetstream r=3 file | — | 7,393 | 0.90 | — | — |
+| `js-2p2s-20k-128-r3` | jetstream r=3 file | 15,416 | 7,728 | 0.94 | 7,711 | 0.94 |
+| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 14,985 | 14.63 | — | — |
+| `js-4p-50k-128-r3` | jetstream r=3 file | — | 17,986 | 2.20 | — | — |
+
+### Round-trip delay
+
+| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
+|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|
+| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.341ms | 0.470ms | 0.395ms | 0.651ms | 1.377ms | 3.243ms |
+| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 132.425ms | 203.717ms | 204.458ms | 241.750ms | 249.468ms | 249.640ms |
+| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 196.422ms | 221.264ms | 224.039ms | 233.180ms | 236.798ms | 238.177ms |
+| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 211.108ms | 294.925ms | 300.254ms | 314.223ms | 315.551ms | 316.091ms |
+| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 280.608ms | 414.972ms | 409.802ms | 513.821ms | 535.394ms | 536.084ms |
+
+### Core NATS
+
+![Core NATS throughput at four loads (NS1 host run)](charts/core-throughput.png)
+
+*Core NATS throughput at four loads (NS1 host run)*
+### Payload size (core)
+
+![Core NATS 128 B vs 1 KiB (NS1 host run)](charts/payload-size.png)
+
+*Core NATS 128 B vs 1 KiB (NS1 host run)*
+### JetStream r=3 file
+
+![JetStream durable publish rate (NS1 host run)](charts/js-throughput.png)
+
+*JetStream durable publish rate (NS1 host run)*
+### Core vs JetStream
+
+![Core vs JetStream publish rate, log scale (NS1 host run)](charts/core-vs-js.png)
+
+*Core vs JetStream publish rate, log scale (NS1 host run)*
+### Delay
+
+![Ping vs flood delay percentiles, log scale (NS1 host run)](charts/delay-percentiles.png)
+
+*Ping vs flood delay percentiles, log scale (NS1 host run)*
+
+---
+
+## 4. Study methodology
+
+### 4.1 Question
+
+On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)?
+
+### 4.2 Hypotheses (stated before the run)
+
+1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica.
+2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated.
+3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting.
+4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count.
+5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound.
+
+### 4.3 Independent variables (what we changed)
+
+| Factor | Levels |
+|--------|--------|
+| Transport | Core NATS pub/sub vs JetStream file replicas=3 |
+| Publisher count | 1, 2, 4, 8 |
+| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 |
+| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) |
+| Payload | 128 B, 1024 B |
+| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) |
+
+### 4.4 Dependent variables (what we recorded)
+
+| Metric | Instrument | Unit |
+|--------|------------|------|
+| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s |
+| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s |
+| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) |
+| Publisher spread | nats min/avg/max **msgs/s** | not delay |
+| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max |
+| Host load | `/proc/loadavg` before and after | load average |
+| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem |
+
+**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`.
+
+### 4.5 Controls and constants
+
+- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`.
+- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`).
+- Bench client is LXC **510**, not a nats-* server.
+- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide.
+- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`).
+- No TLS, no nkeys, no account isolation (isolation is `vmbr1`).
+- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder.
+
+### 4.6 Procedure
+
+1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise.
+2. Snapshot host load, memory, LXC configs, and each nats `varz`.
+3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB).
+4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file.
+5. Copy `latency.mjs` into 510; ping then flood at several batch sizes.
+6. Snapshot host/`varz` again.
+7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**.
+
+No publish, subscribe, chart, or PDF process runs on the operator laptop for this study.
+
+### 4.7 Instrumentation path
+
+```text
+[NS1 host 70.88.205.138]
+   study-on-ns1.sh  (bash + python3)
+        |
+        | sudo pct exec 510
+        v
+[LXC 510 verae-px-worker 10.10.10.20]
+   nats bench / node latency.mjs
+        |
+        | NATS client protocol to
+        v
+[LXC 511/512/513  10.10.10.21-23 :4222]
+   nats-server -js  cluster routes :6222
+```
+
+The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`.
+
+### 4.8 Threats to validity
+
+| Threat | Effect on numbers |
+|--------|-------------------|
+| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. |
+| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. |
+| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. |
+| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. |
+| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. |
+| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. |
+| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. |
+| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. |
+
+### 4.9 Ethics / safety
+
+Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket.
+
+---
+
+## 5. Suggestions for fine-tuning
+
+These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way.
+
+### 5.1 Treat JetStream as the product limiter
+
+Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out.
+
+### 5.2 Split storage class by stream
+
+| Stream | Suggested store | Why |
+|--------|-----------------|-----|
+| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad |
+| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both |
+| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer |
+| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry |
+| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive |
+
+Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert.
+
+### 5.3 Give JetStream real disks
+
+Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain.
+
+- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`.
+- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure.
+- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it.
+- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.”
+
+### 5.4 Isolate the nats CTs from the rest of NS1
+
+Host load on this box is often already several. Pin:
+
+- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes.
+- Memory high enough that file-backed streams stay cache-hot for the working set.
+- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync.
+
+Re-run this study after pinning; H1/H2 should move more than ping.
+
+### 5.5 Consumer and mailbox tuning (delay H4)
+
+Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT.
+
+- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM.
+- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber.
+- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped.
+- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT.
+
+### 5.6 Publisher-side batching in middleware
+
+A timestamp job is one small JSON. 16k msgs/s is ample. Still:
+
+- Avoid per-byte publishes; one message per job/event.
+- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs).
+- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only.
+
+### 5.7 nats-server knobs worth measuring (A/B with this script)
+
+| Knob | Why try it |
+|------|------------|
+| `max_payload` | Keep default unless archive puts grow |
+| `write_deadline` | Slow consumer protection for webhooks |
+| `max_pending` | Bound memory on a stuck Zapier hook |
+| `max_connections` | Fleet workers + keep + middleware |
+| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT |
+| `max_outstanding_catchup` | Replica restart after a nats-c blip |
+| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT |
+
+Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99.
+
+### 5.8 Network
+
+- Keep NATS off `vmbr0`. No change.
+- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load).
+- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus.
+
+### 5.9 Security cost (when nkeys/mTLS flip)
+
+`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement.
+
+### 5.10 Operational fine-tuning (lag, not peak msgs/s)
+
+1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`.
+2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense.
+3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost.
+4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated.
+
+### 5.11 What not to tune
+
+- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge.
+- Do not treat flood 400 ms as “cluster RTT.” Fix consumers.
+- Do not load-test on `ZAPIER_*` streams.
+- Do not bind client NATS to `0.0.0.0` on `vmbr0`.
+
+### 5.12 Recommended next experiments (same method, one change each)
+
+1. CPU pin nats-a/b/c → re-run JS 1p + ping.
+2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder).
+3. Distinct `store_dir` disks per node.
+4. nkeys on, same ladder.
+5. Three hardware boxes, same `cluster.env` IPs updated.
+
+Each experiment should produce a new `results/<utc>/` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory.
+
+
+---
+
+## 6. Reproducing this study
+
+On **NS1 only**:
+
+```bash
+cd ~/verae-src/verae-nats-cluster
+bash scripts/study-on-ns1.sh
+```
+
+The script exits if `hostname` is not NS1. Outputs land in `results/<utc>/` including `nats-cluster-bench-ns1.{md,html,pdf}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog.
+
+Raw logs for this run: `results/20260912T051237Z/`.

reports/ns1-maximize/nats-cluster-bench-ns1.md

@@ -0,0 +1,355 @@
+**Progress report (maximized NS1 study)** · run `20260912T053120Z` (UTC)
+
+> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): `maximize-ns1-study.sh` (cores/RAM/`max_mem`/tmpfs), then `study-on-ns1.sh`, `nats bench`, `latency.mjs` (LXC 510), matplotlib, pandoc, weasyprint. Traffic stayed on `vmbr1`. veth/10G was **not** changed. After the ladder, JetStream was put back on ZFS and product streams were re-created; **8 cores / 16 GiB / max_mem 8G stay**.
+
+## Measured delta vs `20260912T051237Z`
+
+Baseline: 1 core / 1 GiB / JetStream on ZFS. This run: 8 cores / 16 GiB / JetStream **tmpfs** (file r=3) plus extra **memory** store rows. veth/10G unchanged.
+
+| Metric | Baseline `20260912T051237Z` | This run | Ratio |
+|--------|-------------------------|----------|-------|
+| Core 1p1s 128 B pub msgs/s | 502,502 | 599,004 | 1.19× |
+| Core 8p8s 128 B aggregate msgs/s | 2,065,217 | 1,998,733 | 0.97× |
+| JS file r=3 1p 128 B pub msgs/s | 7,393 | 17,388 | 2.35× |
+| JS file r=3 4p 128 B pub msgs/s | 17,986 | 22,853 | 1.27× |
+| JS file r=3 4p 1 KiB pub msgs/s | 14,985 | 18,114 | 1.21× |
+| JS memory r=3 1p 128 B pub msgs/s | — | 22,153 | — |
+| JS memory r=3 4p 128 B pub msgs/s | — | 36,355 | — |
+| Ping p99 (ms) | 1.377ms | 0.684ms | 2.01× faster |
+
+![Baseline vs maximized publish rates (log)](charts/delta-vs-baseline.png)
+
+*Baseline vs maximized publish rates (log)*
+
+
+---
+
+## 1. Executive summary
+
+| Item | This NS1-host run |
+|------|-------------------|
+| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `marchon` |
+| Bench client | LXC 510 `verae-px-worker` |
+| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` |
+| Client URL | `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` |
+| Host load before | `8.77 8.39 8.26 6/3849 198471` |
+| Host load after | `9.02 8.82 8.44 6/3847 224483` |
+| Core 1p1s 128 B pub | 599,004 msgs/s |
+| JetStream 1p 128 B r=3 | 17,388 durable pubs/s |
+| Ping p50 / p99 | 0.299ms / 0.684ms |
+
+Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst.
+
+---
+
+## 2. Where it ran (and where it did not)
+
+```text
+Operator laptop  ──ssh──►  NS1.GEORGELAMBERT.ORG  70.88.205.138
+                              study-on-ns1.sh
+                              python3 build-ns1-study-report.py
+                              sudo pct exec 510 ──► nats bench / latency.mjs
+                                                    │
+                                                    ▼ vmbr1
+                                              10.10.10.21-23 :4222
+```
+
+- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510).
+- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file.
+
+---
+
+## 3. Results (this run)
+
+### Host and brokers
+
+**Before**
+
+| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream |
+|------|------|-------------|---------|----------|-----|-------|---------|-----------|
+| nats-a | 511 | 3 | 3,013 | 3,037 | 1 | 8 | 14,553,088 | True |
+| nats-b | 512 | 3 | 1,415 | 1,424 | 0 | 1 | 13,557,760 | True |
+| nats-c | 513 | 0 | 1,361 | 1,395 | 1 | 1 | 14,028,800 | True |
+
+**After**
+
+| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream |
+|------|------|-------------|---------|----------|-----|-------|---------|-----------|
+| nats-a | 511 | 3 | 644,730 | 1,344,768 | 2 | 8 | 67,104,768 | True |
+| nats-b | 512 | 3 | 491,295 | 978,800 | 1 | 1 | 82,542,592 | True |
+| nats-c | 513 | 0 | 668,544 | 1,330,940 | 0 | 1 | 37,314,560 | True |
+
+nproc=40 · uname=`Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux`
+
+### Throughput
+
+| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
+|-----|------|------------------|------------|----------|------------|----------|
+| `core-1p1s-50k-128` | core pub/sub | 810,988 | 599,004 | 73.12 | 456,116 | 55.68 |
+| `core-4p4s-100k-128` | core pub/sub | 1,361,921 | 379,346 | 46.31 | 1,150,536 | 140.45 |
+| `core-4p4s-50k-1k` | core pub/sub | 695,192 | 175,725 | 171.61 | 584,024 | 570.34 |
+| `core-8p8s-200k-128` | core pub/sub | 1,998,733 | 283,259 | 34.58 | 1,780,589 | 217.36 |
+| `js-1p-20k-128-r3` | jetstream r=3 file | — | 17,388 | 2.12 | — | — |
+| `js-2p2s-20k-128-r3` | jetstream r=3 file | 19,876 | 9,959 | 1.22 | 9,942 | 1.21 |
+| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 18,114 | 17.69 | — | — |
+| `js-4p-50k-128-r3` | jetstream r=3 file | — | 22,853 | 2.79 | — | — |
+| `js-mem-1p-20k-128-r3` | jetstream r=3 file | — | 22,153 | 2.70 | — | — |
+| `js-mem-4p-20k-1k-r3` | jetstream r=3 file | — | 28,685 | 28.01 | — | — |
+| `js-mem-4p-50k-128-r3` | jetstream r=3 file | — | 36,355 | 4.44 | — | — |
+
+### Round-trip delay
+
+| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
+|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|
+| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.250ms | 0.319ms | 0.299ms | 0.363ms | 0.684ms | 2.759ms |
+| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 94.920ms | 132.026ms | 131.990ms | 156.086ms | 158.860ms | 158.965ms |
+| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 113.281ms | 132.425ms | 131.457ms | 140.317ms | 144.683ms | 145.365ms |
+| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 153.318ms | 204.329ms | 205.865ms | 234.151ms | 237.346ms | 237.477ms |
+| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 233.727ms | 310.924ms | 311.170ms | 373.437ms | 400.764ms | 402.811ms |
+
+### Core NATS
+
+![Core NATS throughput at four loads (NS1 host run)](charts/core-throughput.png)
+
+*Core NATS throughput at four loads (NS1 host run)*
+### Payload size (core)
+
+![Core NATS 128 B vs 1 KiB (NS1 host run)](charts/payload-size.png)
+
+*Core NATS 128 B vs 1 KiB (NS1 host run)*
+### JetStream r=3 file
+
+![JetStream durable publish rate (NS1 host run)](charts/js-throughput.png)
+
+*JetStream durable publish rate (NS1 host run)*
+### Core vs JetStream
+
+![Core vs JetStream publish rate, log scale (NS1 host run)](charts/core-vs-js.png)
+
+*Core vs JetStream publish rate, log scale (NS1 host run)*
+### Delay
+
+![Ping vs flood delay percentiles, log scale (NS1 host run)](charts/delay-percentiles.png)
+
+*Ping vs flood delay percentiles, log scale (NS1 host run)*
+
+---
+
+## 4. Study methodology
+
+### 4.1 Question
+
+On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)?
+
+### 4.2 Hypotheses (stated before the run)
+
+1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica.
+2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated.
+3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting.
+4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count.
+5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound.
+
+### 4.3 Independent variables (what we changed)
+
+| Factor | Levels |
+|--------|--------|
+| Transport | Core NATS pub/sub vs JetStream file replicas=3 |
+| Publisher count | 1, 2, 4, 8 |
+| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 |
+| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) |
+| Payload | 128 B, 1024 B |
+| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) |
+
+### 4.4 Dependent variables (what we recorded)
+
+| Metric | Instrument | Unit |
+|--------|------------|------|
+| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s |
+| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s |
+| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) |
+| Publisher spread | nats min/avg/max **msgs/s** | not delay |
+| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max |
+| Host load | `/proc/loadavg` before and after | load average |
+| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem |
+
+**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`.
+
+### 4.5 Controls and constants
+
+- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`.
+- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`).
+- Bench client is LXC **510**, not a nats-* server.
+- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide.
+- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`).
+- No TLS, no nkeys, no account isolation (isolation is `vmbr1`).
+- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder.
+
+### 4.6 Procedure
+
+1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise.
+2. Snapshot host load, memory, LXC configs, and each nats `varz`.
+3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB).
+4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file.
+5. Copy `latency.mjs` into 510; ping then flood at several batch sizes.
+6. Snapshot host/`varz` again.
+7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**.
+
+No publish, subscribe, chart, or PDF process runs on the operator laptop for this study.
+
+### 4.7 Instrumentation path
+
+```text
+[NS1 host 70.88.205.138]
+   study-on-ns1.sh  (bash + python3)
+        |
+        | sudo pct exec 510
+        v
+[LXC 510 verae-px-worker 10.10.10.20]
+   nats bench / node latency.mjs
+        |
+        | NATS client protocol to
+        v
+[LXC 511/512/513  10.10.10.21-23 :4222]
+   nats-server -js  cluster routes :6222
+```
+
+The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`.
+
+### 4.8 Threats to validity
+
+| Threat | Effect on numbers |
+|--------|-------------------|
+| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. |
+| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. |
+| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. |
+| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. |
+| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. |
+| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. |
+| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. |
+| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. |
+
+### 4.9 Ethics / safety
+
+Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket.
+
+---
+
+## 5. Suggestions for fine-tuning
+
+These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way.
+
+### 5.1 Treat JetStream as the product limiter
+
+Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out.
+
+### 5.2 Split storage class by stream
+
+| Stream | Suggested store | Why |
+|--------|-----------------|-----|
+| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad |
+| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both |
+| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer |
+| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry |
+| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive |
+
+Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert.
+
+### 5.3 Give JetStream real disks
+
+Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain.
+
+- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`.
+- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure.
+- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it.
+- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.”
+
+### 5.4 Isolate the nats CTs from the rest of NS1
+
+Host load on this box is often already several. Pin:
+
+- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes.
+- Memory high enough that file-backed streams stay cache-hot for the working set.
+- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync.
+
+Re-run this study after pinning; H1/H2 should move more than ping.
+
+### 5.5 Consumer and mailbox tuning (delay H4)
+
+Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT.
+
+- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM.
+- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber.
+- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped.
+- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT.
+
+### 5.6 Publisher-side batching in middleware
+
+A timestamp job is one small JSON. 16k msgs/s is ample. Still:
+
+- Avoid per-byte publishes; one message per job/event.
+- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs).
+- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only.
+
+### 5.7 nats-server knobs worth measuring (A/B with this script)
+
+| Knob | Why try it |
+|------|------------|
+| `max_payload` | Keep default unless archive puts grow |
+| `write_deadline` | Slow consumer protection for webhooks |
+| `max_pending` | Bound memory on a stuck Zapier hook |
+| `max_connections` | Fleet workers + keep + middleware |
+| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT |
+| `max_outstanding_catchup` | Replica restart after a nats-c blip |
+| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT |
+
+Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99.
+
+### 5.8 Network
+
+- Keep NATS off `vmbr0`. No change.
+- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load).
+- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus.
+
+### 5.9 Security cost (when nkeys/mTLS flip)
+
+`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement.
+
+### 5.10 Operational fine-tuning (lag, not peak msgs/s)
+
+1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`.
+2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense.
+3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost.
+4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated.
+
+### 5.11 What not to tune
+
+- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge.
+- Do not treat flood 400 ms as “cluster RTT.” Fix consumers.
+- Do not load-test on `ZAPIER_*` streams.
+- Do not bind client NATS to `0.0.0.0` on `vmbr0`.
+
+### 5.12 Recommended next experiments (same method, one change each)
+
+1. CPU pin nats-a/b/c → re-run JS 1p + ping.
+2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder).
+3. Distinct `store_dir` disks per node.
+4. nkeys on, same ladder.
+5. Three hardware boxes, same `cluster.env` IPs updated.
+
+Each experiment should produce a new `results/<utc>/` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory.
+
+
+---
+
+## 6. Reproducing this study
+
+On **NS1 only**:
+
+```bash
+cd ~/verae-src/verae-nats-cluster
+bash scripts/study-on-ns1.sh
+```
+
+The script exits if `hostname` is not NS1. Outputs land in `results/<utc>/` including `nats-cluster-bench-ns1.{md,html,pdf}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog.
+
+Raw logs for this run: `results/20260912T053120Z/`.