Add a full NATS cluster speed report (charts, HTML, PDF) to zapier-decisions.

Progress-repo write-up of run 20260912T045131Z with throughput and delay
charts, method, and product-capacity notes; catalog extras point at it.

packages/zapier-decisions/LOG.md

@@ -1,5 +1,11 @@
 # Action log
 
+## 2026-09-12 — NATS cluster speed report
+
+- Bench from LXC 510 vs nats-a/b/c (`20260912T045131Z`).
+- Full write-up with charts in `reports/nats-cluster-bench.{md,html,pdf}`.
+- JetStream r=3 file ~16k durable 128 B pubs/s; ping RTT ~0.3 ms; flood queueing 150–500 ms.
+
 ## 2026-09-12 — test env on the 3-node cluster
 
 - Fleet, keep, middleware, billing, and access planes `NATS_URL` → `10.10.10.21–23:4222`.

packages/zapier-decisions/README.md

@@ -7,6 +7,7 @@ Lab log of architecture decisions, actions taken, and open todos while cleaning
 | File | What |
 |------|------|
 | [STATUS.md](STATUS.md) | **Current status** (2026-09-12) |
+| [reports/nats-cluster-bench.md](reports/nats-cluster-bench.md) | NATS cluster speed report (charts, HTML, PDF) |
 | [LOG.md](LOG.md) | Chronological actions |
 | [TODO.md](TODO.md) | Open items |
 | [decisions/](decisions/) | One file per decision |

packages/zapier-decisions/STATUS.md

@@ -33,7 +33,11 @@ Repo: https://git.georgelambert.org/marchon/verae-nats-cluster
 
 ### Cluster message speed (`20260912T045131Z`)
 
-Measured from LXC **510** (not a nats-* server) against `10.10.10.21–23`. Full tables: [BENCH.md](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md) / catalog [BENCH.pdf](https://zapier.georgelambert.org/packages/verae-nats-cluster/BENCH.pdf). Re-run: `bash scripts/bench.sh` on NS1.
+Measured from LXC **510** (not a nats-* server) against `10.10.10.21–23`.
+
+**Full report (charts + explanations):** [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.pdf) · [markdown](reports/nats-cluster-bench.md)
+
+Short tables: [BENCH.md](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md). Re-run: `bash scripts/bench.sh` on NS1.
 
 | Kind | Load | Result |
 |------|------|--------|

packages/zapier-decisions/reports/nats-cluster-bench.md

@@ -0,0 +1,312 @@
+**Progress report** · 2026-09-12 · run `20260912T045131Z` (UTC)
+
+This is the full write-up of the test-environment NATS cluster bench: what was measured, how, the numbers, the charts, and what they mean for Verae Time × Zapier. Short tables also live in [`verae-nats-cluster/BENCH.md`](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md). Raw logs and CSVs are in that repo under `results/20260912T045131Z/`.
+
+---
+
+## 1. Executive summary
+
+The test cluster is three JetStream nodes on private `vmbr1` (LXC 511–513). The bench client is a **fourth** guest (LXC 510), so the numbers are cluster-plus-network, not a process talking to itself on loopback.
+
+Two different systems were measured, on purpose:
+
+| System | What it is | What we got |
+|--------|------------|-------------|
+| **Core NATS** | Fire-and-forget pub/sub. No disk, no replica ack. | About **0.75–2.0 million msgs/s** at 128 B, depending on fan-out. At 1 KiB, about **630k msgs/s** and **~616 MB/s** aggregate. |
+| **JetStream file, replicas=3** | Durable, replicated — **this is what product streams use**. | About **16k durable 128 B pubs/s**, about **13.5k** at 1 KiB. Pull consume keeps up with publish at ~11k msgs/s each side. |
+| **Ping delay** | One message at a time, publish then wait. | **avg 0.307 ms**, **p99 0.734 ms**, max 2.76 ms (1k × 128 B). |
+| **Flood delay** | Publishers dump a batch; subscriber drains. | **150–505 ms**. That is **queueing under burst**, not wire time. |
+
+**For this product:** timestamp jobs, job events, webhooks, and archive puts go through JetStream r=3. Plan capacity against **~16k durable msgs/s** on this stand, not the million-msg core numbers. A quiet job-event hop is a fraction of a millisecond. If a mailbox falls behind, delay jumps into hundreds of milliseconds — that is the flood column.
+
+Core NATS is still useful: it is the ceiling for non-durable fan-out on this host, and it shows `vmbr1` and the nats-server processes are not the JetStream bottleneck. JetStream is.
+
+---
+
+## 2. Why this test exists
+
+The lab cut the test environment over to the three-node cluster. Before treating that cluster as the message fabric for keep, fleet, middleware, billing, and archive workers, we needed:
+
+1. **Throughput at several loads** — one publisher vs many, 128 B vs 1 KiB, core vs durable.
+2. **Delay characteristics** — both the quiet path (one message RTT) and the overloaded path (burst into a mailbox).
+3. **A client that is not a nats-* server** — otherwise we would be measuring loopback on the broker.
+
+This is a **lab stand on one Proxmox host**, not three metal boxes. It answers “is this cluster in the right order of magnitude for our traffic?” It does not replace a soak test on dedicated disks.
+
+---
+
+## 3. Topology
+
+```text
+                    vmbr1  10.10.10.0/24  (not on vmbr0, not public)
+                    -----------------------------------------------
+  LXC 510           LXC 511            LXC 512            LXC 513
+  verae-px-worker   nats-a             nats-b             nats-c
+  10.10.10.20       10.10.10.21        10.10.10.22        10.10.10.23
+  bench client      :4222 client       :4222              :4222
+                    :6222 routes       :6222              :6222
+                    :8222 loopback     :8222              :8222
+```
+
+- Cluster name: `verae`. Each server has two routes to the other two.
+- Client URL: `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222`
+- HTTP monitor is **loopback :8222** inside each guest. Zapier cloud never talks to NATS.
+- Product streams already on this cluster (`ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE`) use **file** storage and **replicas=3**. The JetStream bench used the same settings on a throwaway stream `benchstream`.
+- Host `127.0.0.1:4222` is still listening on NS1; **clients no longer use it**.
+
+Credits for the stack: Scott Lindsey, George Lambert, NATS.IO, Grok-Code.
+
+---
+
+## 4. Method
+
+### 4.1 Tools
+
+| Piece | Role |
+|-------|------|
+| `nats` CLI **0.1.6** | Throughput (`nats bench --no-progress --csv`). Its min/avg/max are **publisher rate spread**, not delay. |
+| `scripts/latency.mjs` | Two connections, header timestamp `t`, delay = receive time − send time. |
+| `scripts/bench.sh` | Runs the ladder from NS1 via `pct exec` on VMID 510. |
+| `scripts/bench-report.py` | Turns logs into the short `BENCH.md` table. |
+
+Re-run on NS1, from `verae-nats-cluster`:
+
+```bash
+bash scripts/bench.sh
+```
+
+### 4.2 Load ladder
+
+**Core NATS** (subject `bench.core.*`):
+
+| Run | Publishers | Subscribers | Messages | Payload |
+|-----|------------|-------------|----------|---------|
+| `core-1p1s-50k-128` | 1 | 1 | 50,000 | 128 B |
+| `core-4p4s-100k-128` | 4 | 4 | 100,000 | 128 B |
+| `core-8p8s-200k-128` | 8 | 8 | 200,000 | 128 B |
+| `core-4p4s-50k-1k` | 4 | 4 | 50,000 | 1024 B |
+
+**JetStream** (`--js --storage file --replicas 3 --stream benchstream`). The stream is deleted between loads so the name never collides:
+
+| Run | Shape | Messages | Payload |
+|-----|-------|----------|---------|
+| `js-1p-20k-128-r3` | 1 publisher | 20,000 | 128 B |
+| `js-4p-50k-128-r3` | 4 publishers | 50,000 | 128 B |
+| `js-4p-20k-1k-r3` | 4 publishers | 20,000 | 1024 B |
+| `js-2p2s-20k-128-r3` | 2 pub + 2 pull sub | 20,000 | 128 B |
+
+**Delay** (core subjects, two connections):
+
+| Run | Mode | Count | Pubs | Payload |
+|-----|------|-------|------|---------|
+| `lat-ping-1k-128` | **ping** — publish, wait for that message, repeat | 1,000 | 1 | 128 B |
+| `lat-1p-5k-128` | **flood** — publish all, then drain | 5,000 | 1 | 128 B |
+| `lat-4p-10k-128` | flood | 10,000 | 4 | 128 B |
+| `lat-8p-20k-128` | flood | 20,000 | 8 | 128 B |
+| `lat-4p-5k-1k` | flood | 5,000 | 4 | 1024 B |
+
+Ping answers “how long does one quiet hop take?” Flood answers “what happens to the last message if we burst N messages into a mailbox?” Those are different questions. Mixing them is how 0.3 ms and 400 ms get confused.
+
+### 4.3 How to read nats bench columns
+
+- **Pub msgs/s** — rate at which publishers finished their share.
+- **Sub msgs/s** — rate at which subscribers finished. With several subscribers on the same subject, core NATS **fans out**, so sub rate can exceed pub rate.
+- **Aggregate msgs/s** — nats CLI `NATS Pub/Sub stats` line (pub+sub work in one number). Useful as a headline; do not treat it as “the network carried this many unique messages.”
+- Empty JetStream sub cells mean that run was publish-only (durable write, no consumer in the same process).
+
+---
+
+## 5. Throughput results
+
+### 5.1 Core NATS
+
+![Core NATS throughput at four loads](charts/core-throughput.png)
+
+| Run | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
+|-----|------------------|------------|----------|------------|----------|
+| `core-1p1s-50k-128` | 1,200,836 | 791,094 | 96.57 | 747,461 | 91.24 |
+| `core-4p4s-100k-128` | 1,521,256 | 316,312 | 38.61 | 1,299,634 | 158.65 |
+| `core-8p8s-200k-128` | 2,007,937 | 333,957 | 40.77 | 1,790,736 | 218.60 |
+| `core-4p4s-50k-1k` | 630,460 | 247,747 | 241.94 | 510,216 | 498.26 |
+
+**What this chart is saying.** Adding subscribers raises **aggregate** and **sub** rates because each published message is delivered to every subscriber. Publish rate does **not** climb the same way: 1 publisher at 128 B already pushes ~791k msgs/s; 4 and 8 publishers sit around 310–335k msgs/s **each process slower**, while fan-out on the sub side goes to 1.3M then 1.8M.
+
+That publisher slowdown is expected on this stand. The four/eight publisher processes and the four/eight subscribers all run **inside one LXC** (510) against three broker LXCs on the **same Proxmox CPU and `vmbr1`**. Per-publisher logs show a wide spread (example, 4p core 128 B: 79k–524k msgs/s among the four pubs). That is CPU scheduling and client-side contention, not a NATS cluster that only has one fast node.
+
+**1:1 at 128 B** is the cleanest core number: **~791k pub, ~747k sub, ~1.20M aggregate**. The cluster and the bridge can move three-quarter-million small messages per second fire-and-forget from a single client pair.
+
+### 5.2 Payload size (core)
+
+![Core NATS 128 B vs 1 KiB](charts/payload-size.png)
+
+Same 4p4s shape, two sizes:
+
+| Payload | Aggregate msgs/s | Aggregate MB/s | Pub msgs/s | Sub msgs/s |
+|---------|------------------|----------------|------------|------------|
+| 128 B | 1,521,256 | 185.70 | 316,312 | 1,299,634 |
+| 1 KiB | 630,460 | 615.68 | 247,747 | 510,216 |
+
+Message rate falls; **byte rate rises** (186 MB/s → 616 MB/s aggregate). We are leaving the “tiny message, CPU/syscall bound” region and entering “copying bytes across `vmbr1`.” Job JSON and archive metadata sit nearer 128 B–1 KiB than megabyte blobs (blobs are HTTP/WORM, not NATS payloads).
+
+### 5.3 JetStream r=3 file
+
+![JetStream durable publish rate](charts/js-throughput.png)
+
+| Run | Pub msgs/s | Pub MB/s | Sub msgs/s | Notes |
+|-----|------------|----------|------------|-------|
+| `js-1p-20k-128-r3` | **16,155** | 1.97 | — | publish-only |
+| `js-4p-50k-128-r3` | **16,607** | 2.03 | — | four pubs, same ceiling |
+| `js-4p-20k-1k-r3` | **13,493** | 13.18 | — | 1 KiB still disk/replica bound |
+| `js-2p2s-20k-128-r3` | 10,965 | 1.34 | 10,942 | pull consumers keep up |
+
+**Four publishers do not make JetStream four times faster.** 1p and 4p at 128 B are both ~16k msgs/s. The limiter is **synchronous replication to three file-backed replicas**, not client parallelism. That is the result we wanted to see: the bench stream is behaving like a replicated log, not like core fan-out.
+
+Pull consume (`js-2p2s`) is slightly slower on publish (~11k) because the same run is also reading. Pub and sub stay matched (10,965 vs 10,942): the consumer is not the straggler.
+
+1 KiB durable write is ~13.5k msgs/s (~13.2 MB/s). Bytes go up; message rate dips only a little. JetStream here is **ack/fdatasync/replica** bound, not payload-copy bound, in this size range.
+
+### 5.4 Core vs JetStream (same client, same cluster)
+
+![Core vs JetStream publish rate, log scale](charts/core-vs-js.png)
+
+The log scale is required: core publish is **~15–50×** JetStream publish on this stand.
+
+| Shape | Core pub msgs/s | JS r=3 file pub msgs/s | Ratio |
+|-------|-----------------|------------------------|-------|
+| 1 publisher, 128 B | 791,094 | 16,155 | ~49× |
+| 4 publishers, 128 B | 316,312 | 16,607 | ~19× |
+| 4 publishers, 1 KiB | 247,747 | 13,493 | ~18× |
+
+This is not JetStream “losing.” Core is allowed to forget a message the instant the server accepts it. JetStream on file with replicas=3 must **record it on a majority** before the publish acks. Our product streams (`ZAPIER_*`, `VERAE_ARCHIVE`) chose that trade on purpose: a job event that survives one LXC dying is worth ~16k msgs/s instead of ~800k.
+
+If we ever need core-like rates for a signal that may drop, that signal should not be on a replicated file stream.
+
+---
+
+## 6. Delay results
+
+![Delay percentiles, log scale](charts/delay-percentiles.png)
+
+| Run | Kind | Count | min | avg | p50 | p90 | p99 | max |
+|-----|------|-------|-----|-----|-----|-----|-----|-----|
+| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 0.254 ms | **0.307 ms** | 0.286 ms | 0.332 ms | **0.734 ms** | 2.763 ms |
+| `lat-1p-5k-128` | flood | 5000 | 149.3 ms | 238.6 ms | 248.8 ms | 274.3 ms | 279.4 ms | 279.7 ms |
+| `lat-4p-5k-1k` | flood | 5000 | 155.1 ms | 211.7 ms | 217.6 ms | 223.3 ms | 227.8 ms | 228.4 ms |
+| `lat-4p-10k-128` | flood | 10000 | 174.2 ms | 263.2 ms | 266.7 ms | 299.1 ms | 304.2 ms | 304.5 ms |
+| `lat-8p-20k-128` | flood | 20000 | 304.6 ms | 453.7 ms | 466.3 ms | 499.9 ms | 505.1 ms | 505.6 ms |
+
+The dashed line on the chart is 1 ms. Only **ping** lives there.
+
+### 6.1 Ping — the quiet hop
+
+One publisher, one subscriber, two connections, wait for each message before sending the next.
+
+- **min 0.254 ms** — guest → `vmbr1` → a nats-server → `vmbr1` → guest.
+- **p50 0.286 ms / avg 0.307 ms** — typical.
+- **p99 0.734 ms** — still under a millisecond.
+- **max 2.763 ms** — one outlier in 1,000 samples (GC, scheduler, or a slow route). Not the tail we design for.
+
+A middleware `jobs.watch` publish followed by a waiter on `jobs.events` is this shape when the poller is keeping up. Compared with HTTPS to Zapier (tens to hundreds of milliseconds) or a live Verae `GET /api/status/{jobId}`, NATS RTT is noise.
+
+### 6.2 Flood — queueing under burst
+
+Publishers write the whole batch as fast as they can, then the subscriber drains. Each message’s delay is “how long was I in the buffer before the subscriber got to me?”
+
+That is why:
+
+- **min is already ~150–300 ms** — even the first messages wait behind a burst that filled the socket/client queue.
+- **p50 ≈ p99 ≈ max** — a queue drain has a tight distribution: everyone waits for roughly the same backlog.
+- **8p × 20k is ~450 ms avg** — twice the messages of 4p × 10k, roughly twice the wait. Linear in backlog, not in cluster diameter.
+
+Flood is **not** a measurement of NATS being slow. The ping column proves the hop is ~0.3 ms. Flood is a measurement of **what operators will see if a consumer stalls** (job-events mailbox, webhook deliver, archive reply). Backlog time ≈ `queued_messages / consume_rate`.
+
+### 6.3 1 KiB flood vs 128 B flood
+
+4 publishers, 5k messages at 1 KiB: avg **212 ms**, slightly **faster** than 4p 10k × 128 B (263 ms) because the **count is half**, even though each message is 8× larger. Again: delay here tracks **how many messages are queued**, not payload size, in this range.
+
+---
+
+## 7. What this means for Verae × Zapier
+
+Product subjects on this cluster:
+
+| Address | Kind | Bench analogue |
+|---------|------|----------------|
+| `verae.zapier.jobs.watch` | work queue (JetStream) | JS durable pub ~16k/s |
+| `verae.zapier.jobs.events` | events | JS + ping if waiters keep up; flood if they do not |
+| `verae.zapier.webhooks.deliver` | work queue | JS durable |
+| `verae.zapier.usage` | optional | JS durable |
+| `verae.billing.*` | request-reply | ping (quiet RTT) |
+| `verae.archive.put` / `query` / `reply.*` | JetStream + broadcast query | JS durable; query fan-out is closer to core but still JS-backed puts |
+
+**Capacity.** 16k durable 128 B pubs/s is **~1.4×10⁹ messages/day** if you could fill the pipe. We will not. Zapier HTTPS, live `api.veraetime.net`, WORM bloom checks, and human Zap runs sit far below that. This cluster is not the product bottleneck on NS1.
+
+**Latency budget.** A timestamp wait is: HTTP in → NATS watch → poll Verae → NATS event → HTTP out (or REST Hook). The NATS pieces are **sub-millisecond** when caught up. Do not spend time “optimizing NATS RTT” until Zapier/Verae HTTP is in the same band.
+
+**Backlogs.** The failure mode that *does* show up in these numbers is flood delay. If webhook-deliver or job-events consumers pause (keep stopped, replica floor, a blocked HTTPS post to `hooks.zapier.com`), waiters will see **hundreds of milliseconds to seconds** of queue time. Fleet replica floors and keep exist to prevent that, not because 0.3 ms is too slow.
+
+**Hardware move.** Same three configs, three boxes, private NIC. Expect:
+
+- Core numbers to change with NIC and CPU (maybe up, maybe down).
+- JetStream numbers to change **more**, because they are disk + fsync + replica RTT. Distinct SSDs should help; a slow shared datastore would hurt.
+- Ping RTT to grow by whatever the real NIC and switch add (still likely low milliseconds on a LAN).
+
+---
+
+## 8. Limits of this measurement
+
+1. **One Proxmox host.** LXC 510–513 share cores, memory, and the host’s disk. Replica=3 on file is **three files on the same underlying storage**, not three failure domains. HA of “one disk dies” is **not** proven. HA of “one LXC process dies” is the actual claim.
+2. **Short runs.** Tens of thousands of messages, seconds of wall time. No compaction, no multi-hour page-cache eviction, no snapshot/restore during load.
+3. **No TLS, no nkeys.** `verae-nats-accounts` is still a sketch. Auth would add CPU; it would not turn 16k into 800k.
+4. **One bench client.** All publishers live in 510. A fleet of workers on several CTs might publish more into JetStream until disk/replicas saturate — the 1p vs 4p JS result says that saturation is already ~16k from one CT.
+5. **nats 0.1.6** does not report delay. Anyone reading `min | avg | max msgs` on a bench log as microseconds will get the wrong story. Delay is only `latency.mjs`.
+6. **Core aggregate ≠ unique messages.** Fan-out double-counts. Use pub or sub columns when comparing to JetStream.
+7. **Not a Zapier or Verae API bench.** Those are still blocked on operator login / live credentials.
+
+---
+
+## 9. How to reproduce
+
+On NS1 (Proxmox), from the `verae-nats-cluster` checkout:
+
+```bash
+bash scripts/status.sh    # 3/3 JetStream
+bash scripts/bench.sh     # writes results/<utc>/ and BENCH.md
+```
+
+The client VMID defaults to **510**. Override with `CLIENT_VMID=…`. `NATS_URL` comes from `client.env`.
+
+Rebuild this progress report (charts + HTML + PDF) from the monorepo:
+
+```bash
+python3 packages/zapier-decisions/scripts/build-nats-bench-report.py
+```
+
+---
+
+## 10. Appendix — environment and files
+
+| Item | Value |
+|------|-------|
+| Run stamp | `20260912T045131Z` |
+| Client | LXC 510 `verae-px-worker` `10.10.10.20` |
+| Servers | 511/512/513 `nats-a/b/c` `10.10.10.21–23` |
+| nats CLI | 0.1.6 linux-amd64 |
+| JS storage | file, replicas=3, stream `benchstream` (deleted between loads) |
+| Isolation | `vmbr1` only; no `0.0.0.0` client bind |
+| Short tables | [BENCH.md](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md) |
+| Raw logs | `packages/verae-nats-cluster/results/20260912T045131Z/` |
+| This report | `packages/zapier-decisions/reports/nats-cluster-bench.{md,html,pdf}` |
+
+Publisher rate spread (nats CLI, msgs/s, **not** delay):
+
+| Run | min | avg | max |
+|-----|-----|-----|-----|
+| core-4p4s-100k-128 pub | 79,260 | 257,805 | 524,453 |
+| core-8p8s-200k-128 pub | 41,744 | 71,488 | 152,536 |
+| core-4p4s-50k-1k pub | 61,936 | 110,271 | 176,262 |
+| js-4p-50k-128-r3 pub | 4,154 | 5,176 | 6,628 |
+| js-4p-20k-1k-r3 pub | 3,373 | 4,063 | 5,121 |
+| js-2p2s-20k-128-r3 pub | 5,485 | 7,081 | 8,678 |
+
+Wide core spreads are the single-client-CT effect described in §5.1. JetStream spreads are narrow and low — every publisher is waiting on the same replicated write path.

packages/zapier-decisions/scripts/build-nats-bench-report.py

@@ -0,0 +1,247 @@
+#!/usr/bin/env python3
+"""Charts + HTML + PDF for the NATS cluster speed report in zapier-decisions/reports."""
+from __future__ import annotations
+
+import subprocess
+import sys
+from pathlib import Path
+
+import matplotlib
+
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+from matplotlib.ticker import FuncFormatter
+
+ROOT = Path(__file__).resolve().parents[3]
+PKG = Path(__file__).resolve().parents[1]
+REPORT_DIR = PKG / "reports"
+CHARTS = REPORT_DIR / "charts"
+MD = REPORT_DIR / "nats-cluster-bench.md"
+HTML = REPORT_DIR / "nats-cluster-bench.html"
+PDF = REPORT_DIR / "nats-cluster-bench.pdf"
+CSS = ROOT / "scripts" / "docs-print.css"
+LUA = ROOT / "scripts" / "pdf-links.lua"
+
+INDIGO = "#4f46e5"
+DEEP = "#312e81"
+TEAL = "#047857"
+AMBER = "#b45309"
+LILAC = "#7c74f0"
+INK = "#171a26"
+MUTED = "#5b6178"
+GRID = "#d9dce8"
+
+# Run 20260912T045131Z — parsed from nats bench logs (Sub stats, not Pub/Sub aggregate).
+CORE = {
+    "labels": ["1p1s\n50k×128 B", "4p4s\n100k×128 B", "8p8s\n200k×128 B", "4p4s\n50k×1 KiB"],
+    "agg": [1_200_836, 1_521_256, 2_007_937, 630_460],
+    "pub": [791_094, 316_312, 333_957, 247_747],
+    "sub": [747_461, 1_299_634, 1_790_736, 510_216],
+    "pub_mb": [96.57, 38.61, 40.77, 241.94],
+    "sub_mb": [91.24, 158.65, 218.60, 498.26],
+}
+JS = {
+    "labels": ["1p 20k×128 B", "4p 50k×128 B", "4p 20k×1 KiB", "2p2s pull 20k×128 B"],
+    "pub": [16_155, 16_607, 13_493, 10_965],
+    "sub": [None, None, None, 10_942],
+}
+DELAY = {
+    "labels": ["Ping\n1k×128 B", "Flood 1p\n5k×128 B", "Flood 4p\n5k×1 KiB", "Flood 4p\n10k×128 B", "Flood 8p\n20k×128 B"],
+    "kind": ["ping", "flood", "flood", "flood", "flood"],
+    "p50": [0.286, 248.752, 217.579, 266.672, 466.296],
+    "p90": [0.332, 274.314, 223.268, 299.073, 499.924],
+    "p99": [0.734, 279.398, 227.798, 304.233, 505.112],
+    "avg": [0.307, 238.626, 211.706, 263.186, 453.749],
+}
+
+
+def style() -> None:
+    plt.rcParams.update(
+        {
+            "font.family": "sans-serif",
+            "font.size": 10,
+            "axes.titlesize": 12,
+            "axes.titleweight": "semibold",
+            "axes.labelsize": 10,
+            "axes.edgecolor": GRID,
+            "axes.labelcolor": INK,
+            "text.color": INK,
+            "xtick.color": MUTED,
+            "ytick.color": MUTED,
+            "figure.facecolor": "white",
+            "axes.facecolor": "white",
+            "axes.grid": True,
+            "grid.color": GRID,
+            "grid.linewidth": 0.8,
+            "legend.frameon": False,
+            "savefig.bbox": "tight",
+            "savefig.dpi": 160,
+            "savefig.facecolor": "white",
+        }
+    )
+
+
+def k_fmt(x: float, _pos: int | None = None) -> str:
+    if x >= 1_000_000:
+        return f" {x / 1_000_000:.2f}M".strip()
+    if x >= 1000:
+        return f"{x / 1000:.0f}k"
+    return f"{x:.0f}"
+
+
+def save(fig: plt.Figure, name: str) -> None:
+    CHARTS.mkdir(parents=True, exist_ok=True)
+    path = CHARTS / name
+    fig.savefig(path, dpi=160)
+    plt.close(fig)
+    print(f"wrote {path}")
+
+
+def chart_core_msgs() -> None:
+    fig, ax = plt.subplots(figsize=(9.2, 4.4))
+    x = range(len(CORE["labels"]))
+    w = 0.25
+    ax.bar([i - w for i in x], CORE["agg"], w, label="Aggregate", color=DEEP)
+    ax.bar(list(x), CORE["pub"], w, label="Publish", color=INDIGO)
+    ax.bar([i + w for i in x], CORE["sub"], w, label="Subscribe", color=TEAL)
+    ax.set_xticks(list(x), CORE["labels"])
+    ax.set_ylabel("messages / second")
+    ax.set_title("Core NATS throughput (fire-and-forget pub/sub)")
+    ax.yaxis.set_major_formatter(FuncFormatter(k_fmt))
+    ax.legend(loc="upper left")
+    ax.set_axisbelow(True)
+    save(fig, "core-throughput.png")
+
+
+def chart_js_msgs() -> None:
+    fig, ax = plt.subplots(figsize=(9.2, 4.4))
+    colors = [INDIGO, INDIGO, AMBER, LILAC]
+    ax.bar(JS["labels"], JS["pub"], color=colors)
+    ax.set_ylabel("durable publish messages / second")
+    ax.set_title("JetStream file store, replicas=3 (product-stream settings)")
+    ax.yaxis.set_major_formatter(FuncFormatter(k_fmt))
+    ax.set_axisbelow(True)
+    for i, v in enumerate(JS["pub"]):
+        ax.text(i, v + 250, f"{v:,}", ha="center", va="bottom", fontsize=9, color=MUTED)
+    save(fig, "js-throughput.png")
+
+
+def chart_core_vs_js() -> None:
+    fig, ax = plt.subplots(figsize=(9.2, 4.4))
+    labels = ["1 publisher\n128 B", "4 publishers\n128 B", "4 publishers\n1 KiB"]
+    core_pub = [791_094, 316_312, 247_747]
+    js_pub = [16_155, 16_607, 13_493]
+    x = range(len(labels))
+    w = 0.35
+    ax.bar([i - w / 2 for i in x], core_pub, w, label="Core NATS (no disk)", color=INDIGO)
+    ax.bar([i + w / 2 for i in x], js_pub, w, label="JetStream r=3 file", color=AMBER)
+    ax.set_xticks(list(x), labels)
+    ax.set_yscale("log")
+    ax.set_ylabel("publish messages / second (log)")
+    ax.set_title("Core vs JetStream: two different jobs")
+    ax.legend(loc="upper right")
+    ax.set_axisbelow(True)
+    save(fig, "core-vs-js.png")
+
+
+def chart_bytes() -> None:
+    fig, axes = plt.subplots(1, 2, figsize=(9.2, 4.2))
+    labels = ["128 B\n4p4s", "1 KiB\n4p4s"]
+    msgs = [1_521_256, 630_460]
+    mb = [185.70, 615.68]
+    axes[0].bar(labels, msgs, color=[INDIGO, AMBER])
+    axes[0].set_title("Aggregate messages / second")
+    axes[0].yaxis.set_major_formatter(FuncFormatter(k_fmt))
+    axes[0].set_axisbelow(True)
+    axes[1].bar(labels, mb, color=[INDIGO, AMBER])
+    axes[1].set_title("Aggregate MB / second")
+    axes[1].set_ylabel("MB/s")
+    axes[1].set_axisbelow(True)
+    for ax, vals, fmt in (
+        (axes[0], msgs, lambda v: f"{v/1e6:.2f}M"),
+        (axes[1], mb, lambda v: f"{v:.0f}"),
+    ):
+        for i, v in enumerate(vals):
+            ax.text(i, v * 1.02, fmt(v), ha="center", va="bottom", fontsize=9, color=MUTED)
+    fig.suptitle("Core NATS: bigger payloads move more bytes, fewer messages", fontsize=12, fontweight="semibold")
+    fig.tight_layout()
+    save(fig, "payload-size.png")
+
+
+def chart_delay() -> None:
+    fig, ax = plt.subplots(figsize=(9.2, 4.6))
+    x = range(len(DELAY["labels"]))
+    w = 0.25
+    ax.bar([i - w for i in x], DELAY["p50"], w, label="p50", color=TEAL)
+    ax.bar(list(x), DELAY["p90"], w, label="p90", color=INDIGO)
+    ax.bar([i + w for i in x], DELAY["p99"], w, label="p99", color=AMBER)
+    ax.set_xticks(list(x), DELAY["labels"])
+    ax.set_yscale("log")
+    ax.set_ylabel("milliseconds (log)")
+    ax.set_title("Round-trip delay: sequential ping vs burst flood")
+    ax.axhline(1.0, color=GRID, linestyle="--", linewidth=1)
+    ax.legend(loc="upper left")
+    ax.set_axisbelow(True)
+    save(fig, "delay-percentiles.png")
+
+
+def render() -> None:
+    if not MD.exists():
+        raise SystemExit(f"missing {MD}")
+    header = HTML.with_suffix(".hdr.html")
+    banner = HTML.with_suffix(".ban.html")
+    css_text = CSS.read_text(encoding="utf-8")
+    header.write_text(f"<style>{css_text}</style>\n", encoding="utf-8")
+    banner.write_text(
+        '<div class="doc-banner">'
+        '<nav class="site"><a href="/">zapier.georgelambert.org</a>'
+        ' · <a href="/index-md.html">Markdown indexes</a></nav>'
+        '<div class="kicker">Verae Time × Zapier · progress report</div>'
+        "<h1>NATS cluster message speed</h1>"
+        '<div class="source-path">packages/zapier-decisions/reports/nats-cluster-bench.md</div>'
+        "</div>\n",
+        encoding="utf-8",
+    )
+    r = subprocess.run(
+        [
+            "pandoc",
+            str(MD),
+            "-o",
+            str(HTML),
+            "--standalone",
+            f"--resource-path={REPORT_DIR}",
+            "--syntax-highlighting=breezedark",
+            "--metadata=title=NATS cluster message speed",
+            f"--include-in-header={header}",
+            f"--include-before-body={banner}",
+            f"--lua-filter={LUA}",
+        ],
+        capture_output=True,
+        text=True,
+    )
+    header.unlink(missing_ok=True)
+    banner.unlink(missing_ok=True)
+    if r.returncode != 0:
+        sys.stderr.write(r.stderr)
+        raise SystemExit(f"pandoc failed: {r.returncode}")
+    w = subprocess.run(["weasyprint", str(HTML), str(PDF)], capture_output=True, text=True)
+    if w.returncode != 0:
+        sys.stderr.write(w.stderr)
+        raise SystemExit(f"weasyprint failed: {w.returncode}")
+    print(f"wrote {HTML}")
+    print(f"wrote {PDF}")
+
+
+def main() -> int:
+    style()
+    chart_core_msgs()
+    chart_js_msgs()
+    chart_core_vs_js()
+    chart_bytes()
+    chart_delay()
+    render()
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())