commit 0d99c2cb6a20e453ac935a4f84e5f580c8a6d511 Author: George Lambert Date: Sat Sep 12 01:34:53 2026 -0400 Snapshot of zapier-decisions (maximized NS1 NATS study) diff --git a/LOG.md b/LOG.md new file mode 100644 index 0000000..2537314 --- /dev/null +++ b/LOG.md @@ -0,0 +1,148 @@ +# Action log + +## 2026-09-12 — maximized NS1 NATS study + +- `pct set` 510–513 to 8 cores / 16 GiB; `max_mem: 8G`; tmpfs 8G on JetStream for the ladder only. +- Run `20260912T053120Z`. JS file 1p 128 B **7.4k → 17.4k (2.35×)**; JS memory 4p **36.4k**; ping p99 **1.38 → 0.68 ms**. +- tmpfs unmounted after; product streams on ZFS again. Resources and max_mem **left raised**. +- Report: `reports/ns1-maximize/`. + +## 2026-09-12 — NS1-host NATS study (second run) + +- All orchestration, nats bench, matplotlib, pandoc, and weasyprint ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`). Laptop not in the path. +- Script: `verae-nats-cluster/scripts/study-on-ns1.sh` (refuses unless hostname is NS1). +- Run `20260912T051237Z`. Report: `reports/ns1-host/nats-cluster-bench-ns1.{md,html,pdf}`. +- Host load ~8.5. JS 1p 128 B ~7.4k durable pubs/s; ping p99 1.38 ms. Methodology + fine-tuning in that report. + +## 2026-09-12 — NATS cluster speed report + +- Bench from LXC 510 vs nats-a/b/c (`20260912T045131Z`). +- Full write-up with charts in `reports/nats-cluster-bench.{md,html,pdf}`. +- JetStream r=3 file ~16k durable 128 B pubs/s; ping RTT ~0.3 ms; flood queueing 150–500 ms. + +## 2026-09-12 — test env on the 3-node cluster + +- Fleet, keep, middleware, billing, and access planes `NATS_URL` → `10.10.10.21–23:4222`. +- Streams ZAPIER_JOBS/EVENTS/WEBHOOKS/USAGE + VERAE_ARCHIVE replicas=3. +- px-worker keep worm/tree 200 on the cluster. Backup timer enabled. Uptime loop on px-worker. +- MOCK_VERAE still true. Zapier push still waiting on login. nkeys/mTLS not flipped (would require client creds). + +## 2026-09-12 — NATS cluster on three Proxmox LXC + +- New repo **verae-nats-cluster**: LXC 511/512/513 (`nats-a/b/c`, `10.10.10.21–23`) JetStream cluster on `vmbr1`. +- Lab loopback `127.0.0.1:4222` unchanged until cut-over. +- Test: stream replicas=3; same configs later move to distinct hardware. + +## 2026-09-11 — proxmox worker + deploy/monitor/backup + +- NS1 **is** Proxmox (`vmbr0` public, `vmbr1` `10.10.10.0/24`). New repo **verae-proxmox** creates LXC **510** `verae-px-worker` at `10.10.10.20`. +- NATS for guests: socat `10.10.10.1:4222` → `127.0.0.1:4222` (never vmbr0). +- **verae-uptime** off-box HTTPS GET; **verae-backup** tar snapshots; **verae-deploy** tagged npm ci. +- Fleet `overlays/ns1/` checked in; `start.sh` no longer rewrites fleet/machines JSON. +- User systemd + linger for keep and fleet. + +## 2026-09-11 — bootstrap + +- Created this repo to record decisions while executing cleanup items 1–5 (and a second pass on 2–5). +- Assumption: single developer, lab only; breaking changes allowed. + +## Step 1 — account-balance is the prepaid source of truth + +- Persist books to `BOOKS_PATH` (atomic JSON). +- Mutations go through `balance.adjust` / `usage.recorded`; edge cache `balanceCents` is updated from the books reply only. +- Local cache mutate only when `NATS_URL` and `ACCOUNT_BALANCE_URL` are both unset (tests). + +## Step 2 — identity join + +- `zappier-identity` mailbox: `verae.identity.bind` / `verae.identity.lookup`. +- Portal + middleware tenants store `veraeUserId`; JWT never public. + +## Step 3 — access planes are the public doors + +- zappier-edge binds loopback (`127.0.0.1:3000`). +- Public: access-web `:3021`, access-api `:3022`, access-leaf `:3023`, access-zapier `:3024`, access-staff `:3025`. + +## Step 4 — leaf hardening + +- `verae-nats-accounts` holds nats.conf + subject policy. +- Leaf token required; billing subjects denied in policy file and access-leaf. + +## Step 5 — fleet spawn + +- Prefer local machines for HTTP `spawn.command` services. +- Disable lan-134 unless `FLEET_ENABLE_LAN134=1`. +- SSH spawn timeout 8s; failed hosts skipped. + +## 2026-09-11 — catalog live doors + +- Bottom of https://zapier.georgelambert.org/ lists each HTTPS door with description, screenshot, and Open link. +- UI-REVIEW.pdf includes the live-door atlas. Walkthrough §13. + +## 2026-09-11 — Caddy host names live + +- Installed `/etc/caddy/sites/zapier-services.georgelambert.org.caddy` on NS1. HTTPS: portal, api, leaf, zap, staff, iam, login, cs, sales, acct, mw, fleet. +- Not proxied: NATS, keep, workers, loopback edge. + +## 2026-09-11 — host names + +- Mapped every running service to **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`). +- Suggested `*.zapier.georgelambert.org` names for public doors; only `zapier.georgelambert.org` (docs) and `git.georgelambert.org` exist in Caddy today. + +## 2026-09-11 — docs sync, bootstrap, NS1 all-in-one + +- Catalog and ops docs: public portal is access-web `:3021/portal/`; edge is loopback; IAM + keep listed. +- New `verae-bootstrap` repo: clone + npm for `ns1-all-in-one`, `control-plane`, `ns1-archive`, `lan-worker`. +- Fleet starts `staff-iam` and `staff-session`. NS1 all-in-one uses edge `:13000` because host `:3000` is taken; keep still owns archive workers. `better-sqlite3` must be rebuilt on Linux (`npm rebuild`). + +## 2026-09-11 — verae-keep on NS1 + +- New repo `verae-keep`: keep `:3860` restarts crashed units; watch `:3861` restarts keep; `guard.sh` restarts watch. +- Honors admin pause/stop via intent files + `POST /intent`. Fleet writes `data//intent.json` (local and SSH). +- Tested on 70.88.205.138: crash-restart, pause-hold, watch respawn. webhook-deliver brought back. NATS observed only. + +## 2026-09-11 — IAM JSON doors closed + +- CS/sales/accounting/staff JSON list and sales PUT pricing now go through IAM (401 JSON, 302 HTML). +- Fleet service env sets `STAFF_AUTH=1` with `STAFF_IAM_URL`. +- Re-walk: cs credit 200 agent=`cs`; cs export 403; operator Fleet POST 200; admin `/me` `permissions:['*']`. + +## 2026-09-11 — IAM on lab, portal door, hardening + +- Restarted fleet with `STAFF_IAM_URL`. Walk: cs credits (agent=`cs`) and is 403 on accounting export; operator fleet POST 200, cs 403; admin `/me` permissions `*`. +- Portal public door `verae-access-web` `:3021/portal/` (static + `/portal/api` → loopback edge). +- IAM sessions persist in JSON; login rate-limit per IP+username (8/10min); credits stamp IAM username as authz principal. +- lan-134 remains disabled unless `FLEET_ENABLE_LAN134=1`. Receipt PDF branded (indigo header + legal footer). + +## 2026-09-11 — staff IAM + +- New `verae-staff-iam` :3028 — users, roles, permissions, sessions, audit UI. +- Admin login uses IAM when `STAFF_IAM_URL` is set; tabs hide without permission. +- CS/sales/accounting/access-staff/fleet mutating APIs check `cs.credit`, `sales.quote`, `accounting.export`, `fleet.operate`, etc. +- Seed: admin/cs/sales/accounting/operator. Shared staff key is fallback only. + +## 2026-09-11 — last three UI leftovers + +- Swagger `/docs` stays stock; banner names it OpenAPI explorer. +- Shared template `verae-staff-ui` for CS + access-staff. +- `STAFF_COOKIE_DOMAIN` + nginx example for multi-host. + +## 2026-09-11 — names, staff session, exclusive jobs.events + +- Account-balance stores display names (`customer.put` + lookup by name). Edge writes names on customer create/edit; CS/sales/accounting/staff join from edge if the ledger has no name. +- New repo `verae-staff-session` (:3027). Department HTML redirects when `STAFF_AUTH=1`. +- `JOBS_EVENTS_EXCLUSIVE=1` makes `verae-jobs-events` the durable consumer; middleware webhook router backs off. +- Catalog index is cards. Disabled fleet machines are grey, not degraded yellow. + +## 2026-09-11 — UI design-system pass + +- Restyled CS/sales/accounting/access-staff to portal indigo; dollars + names. +- Fleet ⋯ menus, role chips, Docs cards, 0.0.0.0 copy, simulator indigo (orange = faults). +- Portal masked API key; admin customer drawer; empty invoice illustrations. +- Catalog system-ui. New independent repo **UI-Docs** (walkthrough, screenshots, UI-REVIEW.pdf). + +## Second pass (2–5) + +- Identity used from portal when `IDENTITY_URL` set. +- Leaf loads allow-list from nats-accounts `policy.json`. +- Staff plane is a first-class mailbox (`verae-access-staff`). +- jobs.events mailbox process `verae-jobs-events` (passthrough + count). diff --git a/NATS.md b/NATS.md new file mode 100644 index 0000000..782a476 --- /dev/null +++ b/NATS.md @@ -0,0 +1,3 @@ +# NATS — zapier-decisions + +None. Documentation repo. diff --git a/README.md b/README.md new file mode 100644 index 0000000..5232dc7 --- /dev/null +++ b/README.md @@ -0,0 +1,17 @@ +# zapier-decisions + +Lab log of architecture decisions, actions taken, and open todos while cleaning the Verae × Zapier stack. + +**Forgejo:** https://git.georgelambert.org/marchon/zapier-decisions + +| File | What | +|------|------| +| [STATUS.md](STATUS.md) | **Current status** (2026-09-12) | +| [reports/nats-cluster-bench.md](reports/nats-cluster-bench.md) | NATS cluster speed report (charts, HTML, PDF) | +| [reports/ns1-host/nats-cluster-bench-ns1.md](reports/ns1-host/nats-cluster-bench-ns1.md) | Second study: all code on NS1 (methodology + tuning) | +| [reports/ns1-maximize/nats-cluster-bench-ns1.md](reports/ns1-maximize/nats-cluster-bench-ns1.md) | Maximized: 8c/16G/tmpfs vs baseline | +| [LOG.md](LOG.md) | Chronological actions | +| [TODO.md](TODO.md) | Open items | +| [decisions/](decisions/) | One file per decision | + +No runtime. Not a NATS mailbox. diff --git a/STATUS.md b/STATUS.md new file mode 100644 index 0000000..fb46a01 --- /dev/null +++ b/STATUS.md @@ -0,0 +1,145 @@ +# Current status — 2026-09-12 + +Verae Time × Zapier is a **working lab** on one Proxmox host (**NS1.GEORGELAMBERT.ORG**, `70.88.205.138`). Public HTTPS doors, catalog, keep/fleet, and a **test 3-node NATS cluster** on separate LXC guests are up. The test environment **points at that cluster**. Zapier Platform `push` and live `api.veraetime.net` are still waiting on operator login / bind credentials. + +**Progress repo:** this file in [zapier-decisions](https://git.georgelambert.org/marchon/zapier-decisions). +**Catalog:** https://zapier.georgelambert.org/packages/zapier-decisions/STATUS.pdf +**Monorepo:** `master-zapier-plan-draft` (`main` / `master`). + +--- + +## Live right now + +| Layer | State | +|-------|--------| +| Catalog | https://zapier.georgelambert.org/ (PDF links path-relative; module/model books work) | +| Public doors | portal, api, zap, iam, mw, fleet, git — last off-box GET 8/8 OK | +| Keep + fleet | Running. systemd + linger **enabled** for next reboot (current processes were not double-started as systemd) | +| Test NATS | **3-node JetStream on `vmbr1`:** LXC 511/512/513 `nats-a/b/c` at `10.10.10.21–23`. Cluster name `verae`, 2 routes each. | +| Product streams | `ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE` — **replicas=3** | +| Modules on that cluster | Fleet, keep, middleware (`natsConnected: true`), billing, access planes. `NATS_URL` lists all three URLs. | +| Host loopback `:4222` | Still listening; **clients no longer use it** | +| Worker CT **510** `verae-px-worker` | `10.10.10.20` on `vmbr1`. Keep worm+tree **200** on the cluster | +| Backup | Timer `verae-backup.timer` enabled; snapshots under `/SSD2/backups/verae/` | +| Uptime | Loop on px-worker (`:3870/status`), last tick `failed: 0` | + +Client URL (private `vmbr1` only): + +```text +nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222 +``` + +Repo: https://git.georgelambert.org/marchon/verae-nats-cluster + +### Cluster message speed (`20260912T045131Z`) + +Measured from LXC **510** (not a nats-* server) against `10.10.10.21–23`. + +**Full report (charts + explanations):** [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.pdf) · [markdown](reports/nats-cluster-bench.md) + +**Second study (all code on NS1.GEORGELAMBERT.ORG / 70.88.205.138, run `20260912T051237Z`):** [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.pdf) · [markdown](reports/ns1-host/nats-cluster-bench-ns1.md). Orchestrator, nats bench, charts, HTML, and PDF ran on 138; traffic was LXC 510 → 511–513 on `vmbr1`. Includes full methodology and fine-tuning notes. Host load ~8.5/40 during the run; JetStream 1p 128 B was **7.4k** durable pubs/s (vs ~16k on the quieter first ladder); ping p99 **1.38 ms**. + +**Maximized study (`20260912T053120Z`):** 8 cores + 16 GiB on 510–513, `max_mem: 8G`, JetStream **tmpfs** for the ladder (ZFS restored after). veth/10G unchanged. [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.pdf). + +| Metric | Baseline 051237Z | Maximized | Ratio | +|--------|------------------|-----------|-------| +| JS file r=3 1p 128 B | 7,393 pubs/s | **17,388** | **2.35×** | +| JS memory r=3 4p 128 B | — | **36,355** | ~4.9× vs baseline file 1p | +| Core 1p 128 B pub | 502,502 | 599,004 | 1.19× | +| Ping p99 | 1.377 ms | **0.684 ms** | **2.01× faster** | + +Live cluster **keeps 8 cores / 16 GiB / max_mem 8G**. Product streams are back on ZFS (tmpfs was study-only). + +Short tables: [BENCH.md](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md). Re-run: `bash scripts/bench.sh` on NS1. + +| Kind | Load | Result | +|------|------|--------| +| Core NATS | 1p1s 50k×128 B | ~1.20M msgs/s aggregate | +| Core NATS | 8p8s 200k×128 B | ~2.01M msgs/s aggregate | +| Core NATS | 4p4s 50k×1 KiB | ~630k msgs/s (~616 MB/s) | +| JetStream file r=3 | 1–4 pubs, 128 B | ~16k durable pubs/s | +| JetStream file r=3 | 4 pubs, 1 KiB | ~13.5k durable pubs/s | +| Ping (sequential RTT) | 1k×128 B | avg 0.307 ms, p99 0.734 ms | +| Flood (burst queueing) | 8p 20k×128 B | avg 454 ms | + +Product job/event/archive traffic is the JetStream row. Ping is one-message delay; flood is the mailbox catching up after a burst. + +--- + +## Built and checked in (Forgejo, each its own repo) + +IAM, portal on access-web, keep/watch/guard, fleet floors, host-deps, catalog host-move docs, plus: + +| Repo | Job | +|------|-----| +| verae-uptime | Off-box HTTPS GET of public doors | +| verae-backup | SQLite / keep / worm-tree tar; restore dry-run default | +| verae-deploy | Tagged checkout + npm ci on the target OS | +| verae-proxmox | LXC 510 worker provision | +| verae-nats-cluster | LXC 511–513 JetStream cluster | + +--- + +## Still open + +| Item | Notes | +|------|--------| +| Zapier Platform `push` | Needs Zapier login (see below) | +| `MOCK_VERAE=false` | Live `api.veraetime.net` + admin bind credentials (see below) | +| NATS nkeys/mTLS | Cluster is up **without** accounts so existing `NATS_URL`s work. `verae-nats-accounts` remains the sketch. Flip only after every client has credentials. Isolation today is `vmbr1`, not nkeys. | +| lan-134 | Still disabled (SSH not working) | +| Reboot test | systemd should own keep/fleet after reboot; do not also `nohup guard.sh` | +| Hardware NATS | Same three configs, three boxes, private NIC — change IPs in `cluster.env` | + +--- + +## Once we have a Zapier login + +Zapier cloud only talks **HTTPS** to our Zap plane (`https://zap.zapier.georgelambert.org`). No NATS, no `api.veraetime.net`. + +1. `zapier login` (or `zapier-platform login`). +2. In `packages/verae-zapier` (and `verae-activate` if the tiny app is still wanted): + - `zapier-platform register` (or attach an existing app id). + - `MIDDLEWARE_BASE_URL=https://zap.zapier.georgelambert.org` (or `https://mw.zapier.georgelambert.org` if skipping the Zap plane). + - `zapier-platform build` and `validate`. +3. `zapier-platform push` as a **private** app. +4. Invite the workspace; create a Zap: + - Auth = portal **API key** (`zmw_…`), not a Verae JWT. + - Test **Create Timestamp and Wait**, **Create Timestamp** (async `jobId`), **Find Job Status**, **Timestamp Completed** REST Hook. +5. Fix anything `validate` / `push` flags (auth labels, samples, 402/403 mapping). +6. When the contract is stable, bump the platform version and invite a real user. + +Do **not** put NATS URLs or Verae passwords into Zapier. + +--- + +## Once `MOCK_VERAE=false` and live `api.veraetime.net` + +Middleware already has the live HTTP client. Still needed: + +1. `VERAE_API_BASE_URL=https://api.veraetime.net` and working **login** (username/password → JWT). JWT stays in middleware (`tokenRef`); Zapier never sees it. +2. Confirm OpenAPI vs our client: + - `POST /api/timestamp` → **202** `{ jobId }` + - `GET /api/status/{jobId}` until `completed` / `failed` + - `POST /api/verify` + - Batch create/status if used +3. **Hash lookup is not on the live OpenAPI** — `veraeClient.lookupHash` returns **501** when not mocking. Central-chain “find by SHA-256” needs a Verae API gap fill or our own store. +4. Map live status JSON into the receipt shape the Zap app expects (`certificate`, `timestamp`, `blockIndex`, `sha256`). +5. Token refresh, 401 retry, and 402/403 mapping against real errors. +6. One live seal in the lab, then REST Hook delivery to `hooks.zapier.com`. + +--- + +## Submit → jobId → wait → receipt + +**Yes — that is the designed path**, and it already runs in **mock**. + +1. Zap (or curl) `POST /zapier/v1/timestamp` or `/timestamp/wait`. +2. Middleware `createTimestamp` calls Verae `POST /api/timestamp` (or the mock) and gets **`jobId`**. +3. It publishes **`verae.zapier.jobs.watch`** on NATS (the 3-node cluster). +4. **Wait:** subscribe to **`verae.zapier.jobs.events`** for that `jobId`, or poll `GET /api/status/{jobId}`. +5. Job-poller hits Verae status until `completed`. +6. Event `timestamp.completed` carries status; webhook-deliver POSTs the Zapier REST Hook. +7. Response includes **certificate / timestamp / blockIndex** (mock uses `mock-cert-{jobId}`, `blockIndex: 42`). Async create returns `{ jobId, sha256, existing }` immediately; wait returns the finished status. Same SHA-256 is idempotent (`existing: true`). + +If wait times out, middleware returns **`pending` + `jobId`** so the hook can finish the Zap. Live Verae must match that HTTP contract; we have **not** proven a real chain seal yet (`MOCK_VERAE` is still true). diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..d929275 --- /dev/null +++ b/TODO.md @@ -0,0 +1,15 @@ +# Open todos + +- [ ] Live `api.veraetime.net` with `MOCK_VERAE=false` and admin bind credentials. +- [x] Three-node NATS JetStream cluster on distinct Proxmox LXC (`verae-nats-cluster`; test env `NATS_URL` cut over; streams replicas=3). +- [ ] NATS nkeys/mTLS on that cluster (accounts file is still the lab stand-in; not flipped so clients keep working on `vmbr1`). +- [x] Exclusive JetStream consumer for `verae.zapier.jobs.events` on `verae-jobs-events` (`JOBS_EVENTS_EXCLUSIVE=1`; middleware skips the router). +- [x] Auth on CS/sales/accounting HTML via `verae-staff-session` (`STAFF_AUTH=1`). +- [x] Staff IAM: named users, roles, permissions (`verae-staff-iam` :3028). +- [ ] Zapier Platform `push` of a private app. +- [x] Move portal static files fully into `verae-access-web` (`/portal/` public door; API proxied to loopback edge). +- [x] Turn IAM on for the running lab (fleet `STAFF_IAM_URL` + restart). +- [x] IAM hardening: JSON sessions, login rate-limit, JSON 401 as well as HTML 302, credit principal = IAM username. +- [x] Host keep-alive (`verae-keep`) on NS1 with watch+guard; honors admin pause/stop. +- [x] Docs synced to IAM, public portal `:3021`, keep, bootstrap server types. +- [x] `verae-bootstrap` clones all module repos and installs deps per server type. diff --git a/decisions/D001-prepaid-sot.md b/decisions/D001-prepaid-sot.md new file mode 100644 index 0000000..9b1bace --- /dev/null +++ b/decisions/D001-prepaid-sot.md @@ -0,0 +1,6 @@ +# D001 — Account-balance is the only prepaid writer + +**Status:** accepted +**Step:** 1 + +Prepaid cents, credits, usage, and payments persist in `zappier-account-balance` (`books.json`). zappier-edge `balanceCents` is a cache filled from the books reply. Tests without NATS/HTTP books still mutate the cache so unit tests stay hermetic. diff --git a/decisions/D002-identity.md b/decisions/D002-identity.md new file mode 100644 index 0000000..7548947 --- /dev/null +++ b/decisions/D002-identity.md @@ -0,0 +1,6 @@ +# D002 — veraeUserId is the join key, not the JWT + +**Status:** accepted +**Step:** 2 + +Public token remains the zappier API key. Chain JWT stays server-side (`tokenRef`). `zappier-identity` is the mailbox for bind/lookup so portal and middleware share one map. diff --git a/decisions/D003-public-doors.md b/decisions/D003-public-doors.md new file mode 100644 index 0000000..9d32527 --- /dev/null +++ b/decisions/D003-public-doors.md @@ -0,0 +1,6 @@ +# D003 — Access planes are the only public HTTP doors + +**Status:** accepted +**Step:** 3 + +Zapier → `:3024`, browser → `:3021`, customer API → `:3022`, leaf → `:3023`, staff → `:3025`. zappier-edge listens on `127.0.0.1:3000` for identity, Stripe, admin, and as upstream for those planes. diff --git a/decisions/D004-leaf-accounts.md b/decisions/D004-leaf-accounts.md new file mode 100644 index 0000000..65de80d --- /dev/null +++ b/decisions/D004-leaf-accounts.md @@ -0,0 +1,6 @@ +# D004 — Leaf cannot touch billing, even if authz is skipped + +**Status:** accepted +**Step:** 4 + +`verae-nats-accounts/policy.json` is the allow-list. access-leaf refuses any target not in the leaf set. nats.conf documents INTERNAL vs LEAF accounts for when the lab cluster is replaced with a real accounts-enabled nats-server. diff --git a/decisions/D005-fleet-local-first.md b/decisions/D005-fleet-local-first.md new file mode 100644 index 0000000..4df6ecd --- /dev/null +++ b/decisions/D005-fleet-local-first.md @@ -0,0 +1,6 @@ +# D005 — HTTP services spawn locally; lan-134 stays off + +**Status:** accepted +**Step:** 5 + +SSH to `70.88.205.134` blocked keepFloor. HTTP apps (`spawn.command`) prefer `kind=local`. `lan-134` is disabled unless `FLEET_ENABLE_LAN134=1`. Failed SSH hosts are skipped after 8s. diff --git a/package.json b/package.json new file mode 100644 index 0000000..a12c852 --- /dev/null +++ b/package.json @@ -0,0 +1,6 @@ +{ + "name": "zapier-decisions", + "version": "0.1.0", + "private": true, + "description": "Architecture decisions and action log for the Verae × Zapier stack" +} diff --git a/reports/charts/core-throughput.png b/reports/charts/core-throughput.png new file mode 100644 index 0000000..10fbf1c Binary files /dev/null and b/reports/charts/core-throughput.png differ diff --git a/reports/charts/core-vs-js.png b/reports/charts/core-vs-js.png new file mode 100644 index 0000000..0584f9e Binary files /dev/null and b/reports/charts/core-vs-js.png differ diff --git a/reports/charts/delay-percentiles.png b/reports/charts/delay-percentiles.png new file mode 100644 index 0000000..c063bca Binary files /dev/null and b/reports/charts/delay-percentiles.png differ diff --git a/reports/charts/js-throughput.png b/reports/charts/js-throughput.png new file mode 100644 index 0000000..553af5d Binary files /dev/null and b/reports/charts/js-throughput.png differ diff --git a/reports/charts/payload-size.png b/reports/charts/payload-size.png new file mode 100644 index 0000000..c7ba0cb Binary files /dev/null and b/reports/charts/payload-size.png differ diff --git a/reports/nats-cluster-bench.html b/reports/nats-cluster-bench.html new file mode 100644 index 0000000..4201eb2 --- /dev/null +++ b/reports/nats-cluster-bench.html @@ -0,0 +1,1278 @@ + + + + + + + NATS cluster message speed + + + + +
Verae Time × Zapier · progress report

NATS cluster message speed

packages/zapier-decisions/reports/nats-cluster-bench.md
+
+

NATS cluster message speed

+
+

Progress report · 2026-09-12 · run +20260912T045131Z (UTC)

+

This is the full write-up of the test-environment NATS cluster bench: +what was measured, how, the numbers, the charts, and what they mean for +Verae Time × Zapier. Short tables also live in verae-nats-cluster/BENCH.md. +Raw logs and CSVs are in that repo under +results/20260912T045131Z/.

+
+

1. Executive summary

+

The test cluster is three JetStream nodes on private +vmbr1 (LXC 511–513). The bench client is a +fourth guest (LXC 510), so the numbers are +cluster-plus-network, not a process talking to itself on loopback.

+

Two different systems were measured, on purpose:

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
SystemWhat it isWhat we got
Core NATSFire-and-forget pub/sub. No disk, no replica ack.About 0.75–2.0 million msgs/s at 128 B, depending +on fan-out. At 1 KiB, about 630k msgs/s and +~616 MB/s aggregate.
JetStream file, replicas=3Durable, replicated — this is what product streams +use.About 16k durable 128 B pubs/s, about +13.5k at 1 KiB. Pull consume keeps up with publish at +~11k msgs/s each side.
Ping delayOne message at a time, publish then wait.avg 0.307 ms, p99 0.734 ms, max +2.76 ms (1k × 128 B).
Flood delayPublishers dump a batch; subscriber drains.150–505 ms. That is queueing under +burst, not wire time.
+

For this product: timestamp jobs, job events, +webhooks, and archive puts go through JetStream r=3. Plan capacity +against ~16k durable msgs/s on this stand, not the +million-msg core numbers. A quiet job-event hop is a fraction of a +millisecond. If a mailbox falls behind, delay jumps into hundreds of +milliseconds — that is the flood column.

+

Core NATS is still useful: it is the ceiling for non-durable fan-out +on this host, and it shows vmbr1 and the nats-server +processes are not the JetStream bottleneck. JetStream is.

+
+

2. Why this test exists

+

The lab cut the test environment over to the three-node cluster. +Before treating that cluster as the message fabric for keep, fleet, +middleware, billing, and archive workers, we needed:

+
    +
  1. Throughput at several loads — one publisher vs +many, 128 B vs 1 KiB, core vs durable.
    2. +
  2. Delay characteristics — both the quiet path (one +message RTT) and the overloaded path (burst into a mailbox).
    3. +
  3. A client that is not a nats-* server — otherwise we +would be measuring loopback on the broker.
    4. +
+

This is a lab stand on one Proxmox host, not three +metal boxes. It answers “is this cluster in the right order of magnitude +for our traffic?” It does not replace a soak test on dedicated +disks.

+
+

3. Topology

+
                    vmbr1  10.10.10.0/24  (not on vmbr0, not public)
+                    -----------------------------------------------
+  LXC 510           LXC 511            LXC 512            LXC 513
+  verae-px-worker   nats-a             nats-b             nats-c
+  10.10.10.20       10.10.10.21        10.10.10.22        10.10.10.23
+  bench client      :4222 client       :4222              :4222
+                    :6222 routes       :6222              :6222
+                    :8222 loopback     :8222              :8222
+ +

Credits for the stack: Scott Lindsey, George Lambert, NATS.IO, +Grok-Code.

+
+

4. Method

+

4.1 Tools

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + +
PieceRole
nats CLI 0.1.6Throughput (nats bench --no-progress --csv). Its +min/avg/max are publisher rate spread, not delay.
scripts/latency.mjsTwo connections, header timestamp t, delay = receive +time − send time.
scripts/bench.shRuns the ladder from NS1 via pct exec on VMID 510.
scripts/bench-report.pyTurns logs into the short BENCH.md table.
+

Re-run on NS1, from verae-nats-cluster:

+
bash scripts/bench.sh
+

4.2 Load ladder

+

Core NATS (subject bench.core.*):

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunPublishersSubscribersMessagesPayload
core-1p1s-50k-1281150,000128 B
core-4p4s-100k-12844100,000128 B
core-8p8s-200k-12888200,000128 B
core-4p4s-50k-1k4450,0001024 B
+

JetStream +(--js --storage file --replicas 3 --stream benchstream). +The stream is deleted between loads so the name never collides:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunShapeMessagesPayload
js-1p-20k-128-r31 publisher20,000128 B
js-4p-50k-128-r34 publishers50,000128 B
js-4p-20k-1k-r34 publishers20,0001024 B
js-2p2s-20k-128-r32 pub + 2 pull sub20,000128 B
+

Delay (core subjects, two connections):

+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunModeCountPubsPayload
lat-ping-1k-128ping — publish, wait for that message, repeat1,0001128 B
lat-1p-5k-128flood — publish all, then drain5,0001128 B
lat-4p-10k-128flood10,0004128 B
lat-8p-20k-128flood20,0008128 B
lat-4p-5k-1kflood5,00041024 B
+

Ping answers “how long does one quiet hop take?” Flood answers “what +happens to the last message if we burst N messages into a mailbox?” +Those are different questions. Mixing them is how 0.3 ms and 400 ms get +confused.

+

4.3 How to read nats bench +columns

+ +
+

5. Throughput results

+

5.1 Core NATS

+
+ + +
+ ++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunAggregate msgs/sPub msgs/sPub MB/sSub msgs/sSub MB/s
core-1p1s-50k-1281,200,836791,09496.57747,46191.24
core-4p4s-100k-1281,521,256316,31238.611,299,634158.65
core-8p8s-200k-1282,007,937333,95740.771,790,736218.60
core-4p4s-50k-1k630,460247,747241.94510,216498.26
+

What this chart is saying. Adding subscribers raises +aggregate and sub rates because each +published message is delivered to every subscriber. Publish rate does +not climb the same way: 1 publisher at 128 B already +pushes ~791k msgs/s; 4 and 8 publishers sit around 310–335k msgs/s +each process slower, while fan-out on the sub side goes +to 1.3M then 1.8M.

+

That publisher slowdown is expected on this stand. The four/eight +publisher processes and the four/eight subscribers all run +inside one LXC (510) against three broker LXCs on the +same Proxmox CPU and vmbr1. Per-publisher +logs show a wide spread (example, 4p core 128 B: 79k–524k msgs/s among +the four pubs). That is CPU scheduling and client-side contention, not a +NATS cluster that only has one fast node.

+

1:1 at 128 B is the cleanest core number: +~791k pub, ~747k sub, ~1.20M aggregate. The cluster and +the bridge can move three-quarter-million small messages per second +fire-and-forget from a single client pair.

+

5.2 Payload size (core)

+
+Core NATS 128 B vs 1 KiB + +
+

Same 4p4s shape, two sizes:

+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + +
PayloadAggregate msgs/sAggregate MB/sPub msgs/sSub msgs/s
128 B1,521,256185.70316,3121,299,634
1 KiB630,460615.68247,747510,216
+

Message rate falls; byte rate rises (186 MB/s → 616 +MB/s aggregate). We are leaving the “tiny message, CPU/syscall bound” +region and entering “copying bytes across vmbr1.” Job JSON +and archive metadata sit nearer 128 B–1 KiB than megabyte blobs (blobs +are HTTP/WORM, not NATS payloads).

+

5.3 JetStream r=3 file

+
+ + +
+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunPub msgs/sPub MB/sSub msgs/sNotes
js-1p-20k-128-r316,1551.97—publish-only
js-4p-50k-128-r316,6072.03—four pubs, same ceiling
js-4p-20k-1k-r313,49313.18—1 KiB still disk/replica bound
js-2p2s-20k-128-r310,9651.3410,942pull consumers keep up
+

Four publishers do not make JetStream four times +faster. 1p and 4p at 128 B are both ~16k msgs/s. The limiter is +synchronous replication to three file-backed replicas, +not client parallelism. That is the result we wanted to see: the bench +stream is behaving like a replicated log, not like core fan-out.

+

Pull consume (js-2p2s) is slightly slower on publish +(~11k) because the same run is also reading. Pub and sub stay matched +(10,965 vs 10,942): the consumer is not the straggler.

+

1 KiB durable write is ~13.5k msgs/s (~13.2 MB/s). Bytes go up; +message rate dips only a little. JetStream here is +ack/fdatasync/replica bound, not payload-copy bound, in +this size range.

+

5.4 Core vs +JetStream (same client, same cluster)

+
+ + +
+

The log scale is required: core publish is ~15–50× +JetStream publish on this stand.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ShapeCore pub msgs/sJS r=3 file pub msgs/sRatio
1 publisher, 128 B791,09416,155~49×
4 publishers, 128 B316,31216,607~19×
4 publishers, 1 KiB247,74713,493~18×
+

This is not JetStream “losing.” Core is allowed to forget a message +the instant the server accepts it. JetStream on file with replicas=3 +must record it on a majority before the publish acks. +Our product streams (ZAPIER_*, VERAE_ARCHIVE) +chose that trade on purpose: a job event that survives one LXC dying is +worth ~16k msgs/s instead of ~800k.

+

If we ever need core-like rates for a signal that may drop, that +signal should not be on a replicated file stream.

+
+

6. Delay results

+
+ + +
+ +++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunKindCountminavgp50p90p99max
lat-ping-1k-128ping (sequential RTT)10000.254 ms0.307 ms0.286 ms0.332 ms0.734 ms2.763 ms
lat-1p-5k-128flood5000149.3 ms238.6 ms248.8 ms274.3 ms279.4 ms279.7 ms
lat-4p-5k-1kflood5000155.1 ms211.7 ms217.6 ms223.3 ms227.8 ms228.4 ms
lat-4p-10k-128flood10000174.2 ms263.2 ms266.7 ms299.1 ms304.2 ms304.5 ms
lat-8p-20k-128flood20000304.6 ms453.7 ms466.3 ms499.9 ms505.1 ms505.6 ms
+

The dashed line on the chart is 1 ms. Only ping +lives there.

+

6.1 Ping — the quiet hop

+

One publisher, one subscriber, two connections, wait for each message +before sending the next.

+ +

A middleware jobs.watch publish followed by a waiter on +jobs.events is this shape when the poller is keeping up. +Compared with HTTPS to Zapier (tens to hundreds of milliseconds) or a +live Verae GET /api/status/{jobId}, NATS RTT is noise.

+

6.2 Flood — queueing under +burst

+

Publishers write the whole batch as fast as they can, then the +subscriber drains. Each message’s delay is “how long was I in the buffer +before the subscriber got to me?”

+

That is why:

+ +

Flood is not a measurement of NATS being slow. The +ping column proves the hop is ~0.3 ms. Flood is a measurement of +what operators will see if a consumer stalls +(job-events mailbox, webhook deliver, archive reply). Backlog time ≈ +queued_messages / consume_rate.

+

6.3 1 KiB flood vs 128 B flood

+

4 publishers, 5k messages at 1 KiB: avg 212 ms, +slightly faster than 4p 10k × 128 B (263 ms) because +the count is half, even though each message is 8× +larger. Again: delay here tracks how many messages are +queued, not payload size, in this range.

+
+

7. What this means for Verae × +Zapier

+

Product subjects on this cluster:

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
AddressKindBench analogue
verae.zapier.jobs.watchwork queue (JetStream)JS durable pub ~16k/s
verae.zapier.jobs.eventseventsJS + ping if waiters keep up; flood if they do not
verae.zapier.webhooks.deliverwork queueJS durable
verae.zapier.usageoptionalJS durable
verae.billing.*request-replyping (quiet RTT)
verae.archive.put / query / +reply.*JetStream + broadcast queryJS durable; query fan-out is closer to core but still JS-backed +puts
+

Capacity. 16k durable 128 B pubs/s is +~1.4×10⁹ messages/day if you could fill the pipe. We +will not. Zapier HTTPS, live api.veraetime.net, WORM bloom +checks, and human Zap runs sit far below that. This cluster is not the +product bottleneck on NS1.

+

Latency budget. A timestamp wait is: HTTP in → NATS +watch → poll Verae → NATS event → HTTP out (or REST Hook). The NATS +pieces are sub-millisecond when caught up. Do not spend +time “optimizing NATS RTT” until Zapier/Verae HTTP is in the same +band.

+

Backlogs. The failure mode that does show +up in these numbers is flood delay. If webhook-deliver or job-events +consumers pause (keep stopped, replica floor, a blocked HTTPS post to +hooks.zapier.com), waiters will see hundreds of +milliseconds to seconds of queue time. Fleet replica floors and +keep exist to prevent that, not because 0.3 ms is too slow.

+

Hardware move. Same three configs, three boxes, +private NIC. Expect:

+ +
+

8. Limits of this measurement

+
    +
  1. One Proxmox host. LXC 510–513 share cores, memory, +and the host’s disk. Replica=3 on file is three files on the +same underlying storage, not three failure domains. HA of “one +disk dies” is not proven. HA of “one LXC process dies” +is the actual claim.
    2. +
  2. Short runs. Tens of thousands of messages, seconds +of wall time. No compaction, no multi-hour page-cache eviction, no +snapshot/restore during load.
    3. +
  3. No TLS, no nkeys. verae-nats-accounts +is still a sketch. Auth would add CPU; it would not turn 16k into +800k.
    4. +
  4. One bench client. All publishers live in 510. A +fleet of workers on several CTs might publish more into JetStream until +disk/replicas saturate — the 1p vs 4p JS result says that saturation is +already ~16k from one CT.
    5. +
  5. nats 0.1.6 does not report delay. Anyone reading +min | avg | max msgs on a bench log as microseconds will +get the wrong story. Delay is only latency.mjs.
    6. +
  6. Core aggregate ≠ unique messages. Fan-out +double-counts. Use pub or sub columns when comparing to JetStream.
    7. +
  7. Not a Zapier or Verae API bench. Those are still +blocked on operator login / live credentials.
    8. +
+
+

9. How to reproduce

+

On NS1 (Proxmox), from the verae-nats-cluster +checkout:

+
bash scripts/status.sh    # 3/3 JetStream
+bash scripts/bench.sh     # writes results/<utc>/ and BENCH.md
+

The client VMID defaults to 510. Override with +CLIENT_VMID=…. NATS_URL comes from +client.env.

+

Rebuild this progress report (charts + HTML + PDF) from the +monorepo:

+
python3 packages/zapier-decisions/scripts/build-nats-bench-report.py
+
+

10. Appendix — environment and +files

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ItemValue
Run stamp20260912T045131Z
ClientLXC 510 verae-px-worker 10.10.10.20
Servers511/512/513 nats-a/b/c 10.10.10.21–23
nats CLI0.1.6 linux-amd64
JS storagefile, replicas=3, stream benchstream (deleted between +loads)
Isolationvmbr1 only; no 0.0.0.0 client bind
Short tablesBENCH.md
Raw logspackages/verae-nats-cluster/results/20260912T045131Z/
This reportpackages/zapier-decisions/reports/nats-cluster-bench.{md,html,pdf}
+

Publisher rate spread (nats CLI, msgs/s, not +delay):

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Runminavgmax
core-4p4s-100k-128 pub79,260257,805524,453
core-8p8s-200k-128 pub41,74471,488152,536
core-4p4s-50k-1k pub61,936110,271176,262
js-4p-50k-128-r3 pub4,1545,1766,628
js-4p-20k-1k-r3 pub3,3734,0635,121
js-2p2s-20k-128-r3 pub5,4857,0818,678
+

Wide core spreads are the single-client-CT effect described in §5.1. +JetStream spreads are narrow and low — every publisher is waiting on the +same replicated write path.

+ + diff --git a/reports/nats-cluster-bench.md b/reports/nats-cluster-bench.md new file mode 100644 index 0000000..8f5a70f --- /dev/null +++ b/reports/nats-cluster-bench.md @@ -0,0 +1,312 @@ +**Progress report** · 2026-09-12 · run `20260912T045131Z` (UTC) + +This is the full write-up of the test-environment NATS cluster bench: what was measured, how, the numbers, the charts, and what they mean for Verae Time × Zapier. Short tables also live in [`verae-nats-cluster/BENCH.md`](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md). Raw logs and CSVs are in that repo under `results/20260912T045131Z/`. + +--- + +## 1. Executive summary + +The test cluster is three JetStream nodes on private `vmbr1` (LXC 511–513). The bench client is a **fourth** guest (LXC 510), so the numbers are cluster-plus-network, not a process talking to itself on loopback. + +Two different systems were measured, on purpose: + +| System | What it is | What we got | +|--------|------------|-------------| +| **Core NATS** | Fire-and-forget pub/sub. No disk, no replica ack. | About **0.75–2.0 million msgs/s** at 128 B, depending on fan-out. At 1 KiB, about **630k msgs/s** and **~616 MB/s** aggregate. | +| **JetStream file, replicas=3** | Durable, replicated — **this is what product streams use**. | About **16k durable 128 B pubs/s**, about **13.5k** at 1 KiB. Pull consume keeps up with publish at ~11k msgs/s each side. | +| **Ping delay** | One message at a time, publish then wait. | **avg 0.307 ms**, **p99 0.734 ms**, max 2.76 ms (1k × 128 B). | +| **Flood delay** | Publishers dump a batch; subscriber drains. | **150–505 ms**. That is **queueing under burst**, not wire time. | + +**For this product:** timestamp jobs, job events, webhooks, and archive puts go through JetStream r=3. Plan capacity against **~16k durable msgs/s** on this stand, not the million-msg core numbers. A quiet job-event hop is a fraction of a millisecond. If a mailbox falls behind, delay jumps into hundreds of milliseconds — that is the flood column. + +Core NATS is still useful: it is the ceiling for non-durable fan-out on this host, and it shows `vmbr1` and the nats-server processes are not the JetStream bottleneck. JetStream is. + +--- + +## 2. Why this test exists + +The lab cut the test environment over to the three-node cluster. Before treating that cluster as the message fabric for keep, fleet, middleware, billing, and archive workers, we needed: + +1. **Throughput at several loads** — one publisher vs many, 128 B vs 1 KiB, core vs durable. +2. **Delay characteristics** — both the quiet path (one message RTT) and the overloaded path (burst into a mailbox). +3. **A client that is not a nats-* server** — otherwise we would be measuring loopback on the broker. + +This is a **lab stand on one Proxmox host**, not three metal boxes. It answers “is this cluster in the right order of magnitude for our traffic?” It does not replace a soak test on dedicated disks. + +--- + +## 3. Topology + +```text + vmbr1 10.10.10.0/24 (not on vmbr0, not public) + ----------------------------------------------- + LXC 510 LXC 511 LXC 512 LXC 513 + verae-px-worker nats-a nats-b nats-c + 10.10.10.20 10.10.10.21 10.10.10.22 10.10.10.23 + bench client :4222 client :4222 :4222 + :6222 routes :6222 :6222 + :8222 loopback :8222 :8222 +``` + +- Cluster name: `verae`. Each server has two routes to the other two. +- Client URL: `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` +- HTTP monitor is **loopback :8222** inside each guest. Zapier cloud never talks to NATS. +- Product streams already on this cluster (`ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE`) use **file** storage and **replicas=3**. The JetStream bench used the same settings on a throwaway stream `benchstream`. +- Host `127.0.0.1:4222` is still listening on NS1; **clients no longer use it**. + +Credits for the stack: Scott Lindsey, George Lambert, NATS.IO, Grok-Code. + +--- + +## 4. Method + +### 4.1 Tools + +| Piece | Role | +|-------|------| +| `nats` CLI **0.1.6** | Throughput (`nats bench --no-progress --csv`). Its min/avg/max are **publisher rate spread**, not delay. | +| `scripts/latency.mjs` | Two connections, header timestamp `t`, delay = receive time − send time. | +| `scripts/bench.sh` | Runs the ladder from NS1 via `pct exec` on VMID 510. | +| `scripts/bench-report.py` | Turns logs into the short `BENCH.md` table. | + +Re-run on NS1, from `verae-nats-cluster`: + +```bash +bash scripts/bench.sh +``` + +### 4.2 Load ladder + +**Core NATS** (subject `bench.core.*`): + +| Run | Publishers | Subscribers | Messages | Payload | +|-----|------------|-------------|----------|---------| +| `core-1p1s-50k-128` | 1 | 1 | 50,000 | 128 B | +| `core-4p4s-100k-128` | 4 | 4 | 100,000 | 128 B | +| `core-8p8s-200k-128` | 8 | 8 | 200,000 | 128 B | +| `core-4p4s-50k-1k` | 4 | 4 | 50,000 | 1024 B | + +**JetStream** (`--js --storage file --replicas 3 --stream benchstream`). The stream is deleted between loads so the name never collides: + +| Run | Shape | Messages | Payload | +|-----|-------|----------|---------| +| `js-1p-20k-128-r3` | 1 publisher | 20,000 | 128 B | +| `js-4p-50k-128-r3` | 4 publishers | 50,000 | 128 B | +| `js-4p-20k-1k-r3` | 4 publishers | 20,000 | 1024 B | +| `js-2p2s-20k-128-r3` | 2 pub + 2 pull sub | 20,000 | 128 B | + +**Delay** (core subjects, two connections): + +| Run | Mode | Count | Pubs | Payload | +|-----|------|-------|------|---------| +| `lat-ping-1k-128` | **ping** — publish, wait for that message, repeat | 1,000 | 1 | 128 B | +| `lat-1p-5k-128` | **flood** — publish all, then drain | 5,000 | 1 | 128 B | +| `lat-4p-10k-128` | flood | 10,000 | 4 | 128 B | +| `lat-8p-20k-128` | flood | 20,000 | 8 | 128 B | +| `lat-4p-5k-1k` | flood | 5,000 | 4 | 1024 B | + +Ping answers “how long does one quiet hop take?” Flood answers “what happens to the last message if we burst N messages into a mailbox?” Those are different questions. Mixing them is how 0.3 ms and 400 ms get confused. + +### 4.3 How to read nats bench columns + +- **Pub msgs/s** — rate at which publishers finished their share. +- **Sub msgs/s** — rate at which subscribers finished. With several subscribers on the same subject, core NATS **fans out**, so sub rate can exceed pub rate. +- **Aggregate msgs/s** — nats CLI `NATS Pub/Sub stats` line (pub+sub work in one number). Useful as a headline; do not treat it as “the network carried this many unique messages.” +- Empty JetStream sub cells mean that run was publish-only (durable write, no consumer in the same process). + +--- + +## 5. Throughput results + +### 5.1 Core NATS + +![Core NATS throughput at four loads](charts/core-throughput.png) + +| Run | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s | +|-----|------------------|------------|----------|------------|----------| +| `core-1p1s-50k-128` | 1,200,836 | 791,094 | 96.57 | 747,461 | 91.24 | +| `core-4p4s-100k-128` | 1,521,256 | 316,312 | 38.61 | 1,299,634 | 158.65 | +| `core-8p8s-200k-128` | 2,007,937 | 333,957 | 40.77 | 1,790,736 | 218.60 | +| `core-4p4s-50k-1k` | 630,460 | 247,747 | 241.94 | 510,216 | 498.26 | + +**What this chart is saying.** Adding subscribers raises **aggregate** and **sub** rates because each published message is delivered to every subscriber. Publish rate does **not** climb the same way: 1 publisher at 128 B already pushes ~791k msgs/s; 4 and 8 publishers sit around 310–335k msgs/s **each process slower**, while fan-out on the sub side goes to 1.3M then 1.8M. + +That publisher slowdown is expected on this stand. The four/eight publisher processes and the four/eight subscribers all run **inside one LXC** (510) against three broker LXCs on the **same Proxmox CPU and `vmbr1`**. Per-publisher logs show a wide spread (example, 4p core 128 B: 79k–524k msgs/s among the four pubs). That is CPU scheduling and client-side contention, not a NATS cluster that only has one fast node. + +**1:1 at 128 B** is the cleanest core number: **~791k pub, ~747k sub, ~1.20M aggregate**. The cluster and the bridge can move three-quarter-million small messages per second fire-and-forget from a single client pair. + +### 5.2 Payload size (core) + +![Core NATS 128 B vs 1 KiB](charts/payload-size.png) + +Same 4p4s shape, two sizes: + +| Payload | Aggregate msgs/s | Aggregate MB/s | Pub msgs/s | Sub msgs/s | +|---------|------------------|----------------|------------|------------| +| 128 B | 1,521,256 | 185.70 | 316,312 | 1,299,634 | +| 1 KiB | 630,460 | 615.68 | 247,747 | 510,216 | + +Message rate falls; **byte rate rises** (186 MB/s → 616 MB/s aggregate). We are leaving the “tiny message, CPU/syscall bound” region and entering “copying bytes across `vmbr1`.” Job JSON and archive metadata sit nearer 128 B–1 KiB than megabyte blobs (blobs are HTTP/WORM, not NATS payloads). + +### 5.3 JetStream r=3 file + +![JetStream durable publish rate](charts/js-throughput.png) + +| Run | Pub msgs/s | Pub MB/s | Sub msgs/s | Notes | +|-----|------------|----------|------------|-------| +| `js-1p-20k-128-r3` | **16,155** | 1.97 | — | publish-only | +| `js-4p-50k-128-r3` | **16,607** | 2.03 | — | four pubs, same ceiling | +| `js-4p-20k-1k-r3` | **13,493** | 13.18 | — | 1 KiB still disk/replica bound | +| `js-2p2s-20k-128-r3` | 10,965 | 1.34 | 10,942 | pull consumers keep up | + +**Four publishers do not make JetStream four times faster.** 1p and 4p at 128 B are both ~16k msgs/s. The limiter is **synchronous replication to three file-backed replicas**, not client parallelism. That is the result we wanted to see: the bench stream is behaving like a replicated log, not like core fan-out. + +Pull consume (`js-2p2s`) is slightly slower on publish (~11k) because the same run is also reading. Pub and sub stay matched (10,965 vs 10,942): the consumer is not the straggler. + +1 KiB durable write is ~13.5k msgs/s (~13.2 MB/s). Bytes go up; message rate dips only a little. JetStream here is **ack/fdatasync/replica** bound, not payload-copy bound, in this size range. + +### 5.4 Core vs JetStream (same client, same cluster) + +![Core vs JetStream publish rate, log scale](charts/core-vs-js.png) + +The log scale is required: core publish is **~15–50×** JetStream publish on this stand. + +| Shape | Core pub msgs/s | JS r=3 file pub msgs/s | Ratio | +|-------|-----------------|------------------------|-------| +| 1 publisher, 128 B | 791,094 | 16,155 | ~49× | +| 4 publishers, 128 B | 316,312 | 16,607 | ~19× | +| 4 publishers, 1 KiB | 247,747 | 13,493 | ~18× | + +This is not JetStream “losing.” Core is allowed to forget a message the instant the server accepts it. JetStream on file with replicas=3 must **record it on a majority** before the publish acks. Our product streams (`ZAPIER_*`, `VERAE_ARCHIVE`) chose that trade on purpose: a job event that survives one LXC dying is worth ~16k msgs/s instead of ~800k. + +If we ever need core-like rates for a signal that may drop, that signal should not be on a replicated file stream. + +--- + +## 6. Delay results + +![Delay percentiles, log scale](charts/delay-percentiles.png) + +| Run | Kind | Count | min | avg | p50 | p90 | p99 | max | +|-----|------|-------|-----|-----|-----|-----|-----|-----| +| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 0.254 ms | **0.307 ms** | 0.286 ms | 0.332 ms | **0.734 ms** | 2.763 ms | +| `lat-1p-5k-128` | flood | 5000 | 149.3 ms | 238.6 ms | 248.8 ms | 274.3 ms | 279.4 ms | 279.7 ms | +| `lat-4p-5k-1k` | flood | 5000 | 155.1 ms | 211.7 ms | 217.6 ms | 223.3 ms | 227.8 ms | 228.4 ms | +| `lat-4p-10k-128` | flood | 10000 | 174.2 ms | 263.2 ms | 266.7 ms | 299.1 ms | 304.2 ms | 304.5 ms | +| `lat-8p-20k-128` | flood | 20000 | 304.6 ms | 453.7 ms | 466.3 ms | 499.9 ms | 505.1 ms | 505.6 ms | + +The dashed line on the chart is 1 ms. Only **ping** lives there. + +### 6.1 Ping — the quiet hop + +One publisher, one subscriber, two connections, wait for each message before sending the next. + +- **min 0.254 ms** — guest → `vmbr1` → a nats-server → `vmbr1` → guest. +- **p50 0.286 ms / avg 0.307 ms** — typical. +- **p99 0.734 ms** — still under a millisecond. +- **max 2.763 ms** — one outlier in 1,000 samples (GC, scheduler, or a slow route). Not the tail we design for. + +A middleware `jobs.watch` publish followed by a waiter on `jobs.events` is this shape when the poller is keeping up. Compared with HTTPS to Zapier (tens to hundreds of milliseconds) or a live Verae `GET /api/status/{jobId}`, NATS RTT is noise. + +### 6.2 Flood — queueing under burst + +Publishers write the whole batch as fast as they can, then the subscriber drains. Each message’s delay is “how long was I in the buffer before the subscriber got to me?” + +That is why: + +- **min is already ~150–300 ms** — even the first messages wait behind a burst that filled the socket/client queue. +- **p50 ≈ p99 ≈ max** — a queue drain has a tight distribution: everyone waits for roughly the same backlog. +- **8p × 20k is ~450 ms avg** — twice the messages of 4p × 10k, roughly twice the wait. Linear in backlog, not in cluster diameter. + +Flood is **not** a measurement of NATS being slow. The ping column proves the hop is ~0.3 ms. Flood is a measurement of **what operators will see if a consumer stalls** (job-events mailbox, webhook deliver, archive reply). Backlog time ≈ `queued_messages / consume_rate`. + +### 6.3 1 KiB flood vs 128 B flood + +4 publishers, 5k messages at 1 KiB: avg **212 ms**, slightly **faster** than 4p 10k × 128 B (263 ms) because the **count is half**, even though each message is 8× larger. Again: delay here tracks **how many messages are queued**, not payload size, in this range. + +--- + +## 7. What this means for Verae × Zapier + +Product subjects on this cluster: + +| Address | Kind | Bench analogue | +|---------|------|----------------| +| `verae.zapier.jobs.watch` | work queue (JetStream) | JS durable pub ~16k/s | +| `verae.zapier.jobs.events` | events | JS + ping if waiters keep up; flood if they do not | +| `verae.zapier.webhooks.deliver` | work queue | JS durable | +| `verae.zapier.usage` | optional | JS durable | +| `verae.billing.*` | request-reply | ping (quiet RTT) | +| `verae.archive.put` / `query` / `reply.*` | JetStream + broadcast query | JS durable; query fan-out is closer to core but still JS-backed puts | + +**Capacity.** 16k durable 128 B pubs/s is **~1.4×10⁹ messages/day** if you could fill the pipe. We will not. Zapier HTTPS, live `api.veraetime.net`, WORM bloom checks, and human Zap runs sit far below that. This cluster is not the product bottleneck on NS1. + +**Latency budget.** A timestamp wait is: HTTP in → NATS watch → poll Verae → NATS event → HTTP out (or REST Hook). The NATS pieces are **sub-millisecond** when caught up. Do not spend time “optimizing NATS RTT” until Zapier/Verae HTTP is in the same band. + +**Backlogs.** The failure mode that *does* show up in these numbers is flood delay. If webhook-deliver or job-events consumers pause (keep stopped, replica floor, a blocked HTTPS post to `hooks.zapier.com`), waiters will see **hundreds of milliseconds to seconds** of queue time. Fleet replica floors and keep exist to prevent that, not because 0.3 ms is too slow. + +**Hardware move.** Same three configs, three boxes, private NIC. Expect: + +- Core numbers to change with NIC and CPU (maybe up, maybe down). +- JetStream numbers to change **more**, because they are disk + fsync + replica RTT. Distinct SSDs should help; a slow shared datastore would hurt. +- Ping RTT to grow by whatever the real NIC and switch add (still likely low milliseconds on a LAN). + +--- + +## 8. Limits of this measurement + +1. **One Proxmox host.** LXC 510–513 share cores, memory, and the host’s disk. Replica=3 on file is **three files on the same underlying storage**, not three failure domains. HA of “one disk dies” is **not** proven. HA of “one LXC process dies” is the actual claim. +2. **Short runs.** Tens of thousands of messages, seconds of wall time. No compaction, no multi-hour page-cache eviction, no snapshot/restore during load. +3. **No TLS, no nkeys.** `verae-nats-accounts` is still a sketch. Auth would add CPU; it would not turn 16k into 800k. +4. **One bench client.** All publishers live in 510. A fleet of workers on several CTs might publish more into JetStream until disk/replicas saturate — the 1p vs 4p JS result says that saturation is already ~16k from one CT. +5. **nats 0.1.6** does not report delay. Anyone reading `min | avg | max msgs` on a bench log as microseconds will get the wrong story. Delay is only `latency.mjs`. +6. **Core aggregate ≠ unique messages.** Fan-out double-counts. Use pub or sub columns when comparing to JetStream. +7. **Not a Zapier or Verae API bench.** Those are still blocked on operator login / live credentials. + +--- + +## 9. How to reproduce + +On NS1 (Proxmox), from the `verae-nats-cluster` checkout: + +```bash +bash scripts/status.sh # 3/3 JetStream +bash scripts/bench.sh # writes results// and BENCH.md +``` + +The client VMID defaults to **510**. Override with `CLIENT_VMID=…`. `NATS_URL` comes from `client.env`. + +Rebuild this progress report (charts + HTML + PDF) from the monorepo: + +```bash +python3 packages/zapier-decisions/scripts/build-nats-bench-report.py +``` + +--- + +## 10. Appendix — environment and files + +| Item | Value | +|------|-------| +| Run stamp | `20260912T045131Z` | +| Client | LXC 510 `verae-px-worker` `10.10.10.20` | +| Servers | 511/512/513 `nats-a/b/c` `10.10.10.21–23` | +| nats CLI | 0.1.6 linux-amd64 | +| JS storage | file, replicas=3, stream `benchstream` (deleted between loads) | +| Isolation | `vmbr1` only; no `0.0.0.0` client bind | +| Short tables | [BENCH.md](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md) | +| Raw logs | `packages/verae-nats-cluster/results/20260912T045131Z/` | +| This report | `packages/zapier-decisions/reports/nats-cluster-bench.{md,html,pdf}` | + +Publisher rate spread (nats CLI, msgs/s, **not** delay): + +| Run | min | avg | max | +|-----|-----|-----|-----| +| core-4p4s-100k-128 pub | 79,260 | 257,805 | 524,453 | +| core-8p8s-200k-128 pub | 41,744 | 71,488 | 152,536 | +| core-4p4s-50k-1k pub | 61,936 | 110,271 | 176,262 | +| js-4p-50k-128-r3 pub | 4,154 | 5,176 | 6,628 | +| js-4p-20k-1k-r3 pub | 3,373 | 4,063 | 5,121 | +| js-2p2s-20k-128-r3 pub | 5,485 | 7,081 | 8,678 | + +Wide core spreads are the single-client-CT effect described in §5.1. JetStream spreads are narrow and low — every publisher is waiting on the same replicated write path. diff --git a/reports/nats-cluster-bench.pdf b/reports/nats-cluster-bench.pdf new file mode 100644 index 0000000..ff11f5a Binary files /dev/null and b/reports/nats-cluster-bench.pdf differ diff --git a/reports/ns1-host/charts/core-throughput.png b/reports/ns1-host/charts/core-throughput.png new file mode 100644 index 0000000..44609c2 Binary files /dev/null and b/reports/ns1-host/charts/core-throughput.png differ diff --git a/reports/ns1-host/charts/core-vs-js.png b/reports/ns1-host/charts/core-vs-js.png new file mode 100644 index 0000000..dae2b58 Binary files /dev/null and b/reports/ns1-host/charts/core-vs-js.png differ diff --git a/reports/ns1-host/charts/delay-percentiles.png b/reports/ns1-host/charts/delay-percentiles.png new file mode 100644 index 0000000..f57cf44 Binary files /dev/null and b/reports/ns1-host/charts/delay-percentiles.png differ diff --git a/reports/ns1-host/charts/js-throughput.png b/reports/ns1-host/charts/js-throughput.png new file mode 100644 index 0000000..1edaebe Binary files /dev/null and b/reports/ns1-host/charts/js-throughput.png differ diff --git a/reports/ns1-host/charts/payload-size.png b/reports/ns1-host/charts/payload-size.png new file mode 100644 index 0000000..5c64356 Binary files /dev/null and b/reports/ns1-host/charts/payload-size.png differ diff --git a/reports/ns1-host/nats-cluster-bench-ns1.html b/reports/ns1-host/nats-cluster-bench-ns1.html new file mode 100644 index 0000000..59b896e --- /dev/null +++ b/reports/ns1-host/nats-cluster-bench-ns1.html @@ -0,0 +1,1291 @@ + + + + + + + NATS cluster message speed — NS1 host study + + + + +
Verae Time × Zapier · progress report · run on NS1.GEORGELAMBERT.ORG

NATS cluster message speed — NS1 host study

packages/zapier-decisions/reports/nats-cluster-bench-ns1.md
+
+

NATS cluster message speed — NS1 host study

+
+

Progress report (second study) · run +20260912T051237Z (UTC)

+
+

Execution provenance. Every process for this study +ran on NS1.GEORGELAMBERT.ORG +(70.88.205.138): the orchestrator +(study-on-ns1.sh), nats bench, +latency.mjs (inside LXC 510 on this hypervisor), charting +(matplotlib), and HTML/PDF (pandoc + +weasyprint). The operator laptop did not +publish, subscribe, draw charts, or render the PDF. Traffic stayed on +vmbr1 from LXC 510 to +nats-a/b/c (511–513).

+
+

This is a full methodology write-up plus the numbers from that +on-host run. The earlier report (nats-cluster-bench, run +20260912T045131Z) used the same cluster but was +orchestrated and rendered off-box. Use this document +when you need “it was all run on 138.”

+
+

1. Executive summary

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ItemThis NS1-host run
Control planeNS1.GEORGELAMBERT.ORG (70.88.205.138), user +marchon
Bench clientLXC 510 verae-px-worker
BrokersLXC 511/512/513 nats-a/b/c on +10.10.10.21–23
Client URLnats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
Host load before8.59 8.39 8.15 5/3843 4096277
Host load after8.67 8.33 8.15 9/3863 4114865
Core 1p1s 128 B pub502,502 msgs/s
JetStream 1p 128 B r=37,393 durable pubs/s
Ping p50 / p990.395ms / 1.377ms
+

Product traffic is the JetStream row. Ping is one-message delay. +Flood is mailbox catch-up after a burst.

+
+

2. Where it ran (and where it +did not)

+
Operator laptop  ──ssh──►  NS1.GEORGELAMBERT.ORG  70.88.205.138
+                              study-on-ns1.sh
+                              python3 build-ns1-study-report.py
+                              sudo pct exec 510 ──► nats bench / latency.mjs
+                                                    │
+                                                    ▼ vmbr1
+                                              10.10.10.21-23 :4222
+
    +
  • Did run on 138: bash, python3, matplotlib, pandoc, +weasyprint, pct, nats-server (in LXC), nats CLI and Node +(in LXC 510).
    • +
  • Did not run on the laptop: no local +nats bench, no local charting, no local WeasyPrint for this +file.
    • +
+
+

3. Results (this run)

+

Host and brokers

+

Before

+ +++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NodeVMIDconnectionsin_msgsout_msgscpucoresmem (B)jetstream
nats-a51121,721,2034,933,7591126,398,720True
nats-b51221,613,2053,513,0960123,097,344True
nats-c51321,760,9734,048,5822123,867,392True
+

After

+ +++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NodeVMIDconnectionsin_msgsout_msgscpucoresmem (B)jetstream
nats-a51122,097,5965,910,1611138,273,024True
nats-b51222,121,8164,746,5391141,361,408True
nats-c51322,241,3665,053,9781138,162,432True
+

nproc=40 · +uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux

+

Throughput

+ +++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunModeAggregate msgs/sPub msgs/sPub MB/sSub msgs/sSub MB/s
core-1p1s-50k-128core pub/sub776,331502,50261.34420,97651.39
core-4p4s-100k-128core pub/sub1,278,454379,98546.381,078,910131.70
core-4p4s-50k-1kcore pub/sub581,464209,545204.63482,645471.33
core-8p8s-200k-128core pub/sub2,065,217333,73140.741,847,242225.49
js-1p-20k-128-r3jetstream r=3 file—7,3930.90——
js-2p2s-20k-128-r3jetstream r=3 file15,4167,7280.947,7110.94
js-4p-20k-1k-r3jetstream r=3 file—14,98514.63——
js-4p-50k-128-r3jetstream r=3 file—17,9862.20——
+

Round-trip delay

+ +++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunKindCountPubsSizeminavgp50p90p99max
lat-ping-1k-128ping (sequential RTT)10001128 B0.341ms0.470ms0.395ms0.651ms1.377ms3.243ms
lat-1p-5k-128flood (burst queueing)50001128 B132.425ms203.717ms204.458ms241.750ms249.468ms249.640ms
lat-4p-5k-1kflood (burst queueing)500041024 B196.422ms221.264ms224.039ms233.180ms236.798ms238.177ms
lat-4p-10k-128flood (burst queueing)100004128 B211.108ms294.925ms300.254ms314.223ms315.551ms316.091ms
lat-8p-20k-128flood (burst queueing)200008128 B280.608ms414.972ms409.802ms513.821ms535.394ms536.084ms
+

Core NATS

+
+ + +
+

Core NATS throughput at four loads (NS1 host run) ### +Payload size (core)

+
+ + +
+

Core NATS 128 B vs 1 KiB (NS1 host run) ### JetStream r=3 +file

+
+ + +
+

JetStream durable publish rate (NS1 host run) ### Core vs +JetStream

+
+ + +
+

Core vs JetStream publish rate, log scale (NS1 host run) ### +Delay

+
+ + +
+

Ping vs flood delay percentiles, log scale (NS1 host +run)

+
+

4. Study methodology

+

4.1 Question

+

On the NS1 test stand, what message throughput and +delay does the three-node verae JetStream +cluster deliver at several loads, and which part of the stack is the +limiter for product traffic (jobs, events, webhooks, archive)?

+

4.2 Hypotheses (stated before +the run)

+
    +
  1. H1 — Core vs JetStream. Fire-and-forget core NATS +is at least an order of magnitude faster than JetStream file + +replicas=3, because durable publish waits for a majority disk +replica.
    2. +
  2. H2 — JetStream parallelism. Adding publishers does +not linearly increase JetStream write rate once the +replica log is saturated.
    3. +
  3. H3 — Quiet delay. Sequential pub→sub round trip on +vmbr1 is well under 1 ms p99 when the consumer is +waiting.
    4. +
  4. H4 — Burst delay. If publishers dump a batch before +the subscriber drains, observed delay is queueing time, +roughly linear in backlog, not in cluster hop count.
    5. +
  5. H5 — Payload. Moving 128 B → 1 KiB lowers message +rate and raises byte rate on core NATS; JetStream in this size band +stays replica/fsync bound.
    6. +
+

4.3 Independent variables +(what we changed)

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FactorLevels
TransportCore NATS pub/sub vs JetStream file replicas=3
Publisher count1, 2, 4, 8
Subscriber count0 (JS publish-only), 1, 2, 4, 8
Message count1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step)
Payload128 B, 1024 B
Delay modeping (publish, wait, repeat) vs +flood (publish all, then drain)
+

4.4 Dependent variables +(what we recorded)

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MetricInstrumentUnit
Publish ratenats bench 0.1.6 Pub statsmsgs/s, MB/s
Subscribe ratenats bench Sub statsmsgs/s, MB/s
Aggregatenats bench NATS Pub/Sub statsmsgs/s (fan-out counts both sides)
Publisher spreadnats min/avg/max msgs/snot delay
One-way-ish RTTlatency.mjs header timestampmin, avg, p50, p90, p99, max
Host load/proc/loadavg before and afterload average
Broker countershttp://127.0.0.1:8222/varz inside each nats LXCconnections, in/out msgs, cpu, mem
+

Important: nats CLI 0.1.6 min/avg/max are +rate spread across publishers, not microseconds of +delay. Delay is only latency.mjs.

+

4.5 Controls and constants

+
    +
  • Cluster name verae, three routes, client +:4222, cluster :6222, monitor loopback +:8222.
    • +
  • Client URL always the three-node list on vmbr1 (never +host 127.0.0.1:4222, never vmbr0).
    • +
  • Bench client is LXC 510, not a nats-* server.
    • +
  • JetStream bench stream name benchstream, +file storage, replicas=3, deleted +between JS loads (nats stream rm --force) so names do not +collide.
    • +
  • Product streams were not the bench target (no load +test on ZAPIER_* / VERAE_ARCHIVE).
    • +
  • No TLS, no nkeys, no account isolation (isolation is +vmbr1).
    • +
  • Same nats CLI version (0.1.6) and nats@2 Node client as +the first ladder.
    • +
+

4.6 Procedure

+
    +
  1. Confirm this script is executing on +NS1.GEORGELAMBERT.ORG. Refuse otherwise.
    2. +
  2. Snapshot host load, memory, LXC configs, and each nats +varz.
    3. +
  3. From NS1, pct exec 510 the core ladder (1p1s, 4p4s, +8p8s at 128 B; 4p4s at 1 KiB).
    4. +
  4. Delete benchstream; JS ladder (1p, 4p, 4p×1 KiB, 2p2s +pull) at replicas=3 file.
    5. +
  5. Copy latency.mjs into 510; ping then flood at several +batch sizes.
    6. +
  6. Snapshot host/varz again.
    7. +
  7. Parse logs on this host; draw charts; write HTML +and PDF on this host.
    8. +
+

No publish, subscribe, chart, or PDF process runs on the operator +laptop for this study.

+

4.7 Instrumentation path

+
[NS1 host 70.88.205.138]
+   study-on-ns1.sh  (bash + python3)
+        |
+        | sudo pct exec 510
+        v
+[LXC 510 verae-px-worker 10.10.10.20]
+   nats bench / node latency.mjs
+        |
+        | NATS client protocol to
+        v
+[LXC 511/512/513  10.10.10.21-23 :4222]
+   nats-server -js  cluster routes :6222
+

The hypervisor issues the guest commands. The messages themselves +never leave vmbr1.

+

4.8 Threats to validity

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ThreatEffect on numbers
One physical hostThree “replicas” share CPU, memory, and usually the same datastore. +This measures process/LXC HA, not disk HA.
Shared loadNS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. +Load average during a run is part of the result, not noise to +ignore.
Single bench clientAll publishers live in 510. Per-publisher rate spread is contention +in that guest.
Short runsSeconds of traffic. No compaction, no multi-hour page-cache +eviction, no snapshot during load.
No TLS/nkeysProduction auth will cost CPU. Do not treat these rates as +post-nkeys rates.
Fan-out aggregateCore aggregate msgs/s counts pub+sub. Do not compare that column to +JetStream unique writes.
Flood ≠ RTTMixing flood averages with ping p99 produces a fake “NATS is slow” +story.
Lab onlyNot a Zapier HTTPS bench and not live +api.veraetime.net.
+

4.9 Ethics / safety

+

Bench uses throwaway subjects (bench.core.*, +bench.js.*, bench.lat.*) and a throwaway +stream. It does not purge product streams. Zapier cloud has no NATS +socket.

+
+

5. Suggestions for fine-tuning

+

These follow from the method and from the first ladder on this stand +(JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of +ms). Apply in order of leverage. Re-run this NS1 study +after each change so the delta is measured the same way.

+

5.1 Treat JetStream as +the product limiter

+

Product jobs/events/webhooks/archive are durable. Tuning core NATS to +2M msgs/s will not move a timestamp Zap. Put effort into replica +write path and consumer lag, not core +fan-out.

+

5.2 Split storage class by +stream

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
StreamSuggested storeWhy
ZAPIER_JOBSfile, r=3Work queue; lose-a-job is bad
ZAPIER_EVENTSfile r=3, or memory r=3 if events are rebuildable from job +statusHot waiters; measure both
ZAPIER_WEBHOOKSfile, r=3, workqueueHTTPS to Zapier is the slow consumer
ZAPIER_USAGEfile, r=3, limits + max-ageTelemetry
VERAE_ARCHIVEfile, r=3, on the best diskPuts are larger and must survive
+

Try ZAPIER_EVENTS as memory store in a maintenance +window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms +and durable events still ack at a higher rate, keep it; if a CT restart +drops in-flight waiters, revert.

+

5.3 Give JetStream real disks

+

Today r=3 on three LXC guests on one Proxmox host is +three files, one failure domain.

+
    +
  • Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) +into each nats LXC store_dir.
    • +
  • Set sync: always only on archive if you need it; +default sync is often enough for jobs and is faster. Measure.
    • +
  • Do not put JetStream store_dir on the same busy rootfs +as Forgejo/Caddy if we can avoid it.
    • +
  • When moving to three metal boxes: same configs, private NIC, one +disk (or mirror) per node. That is the first change +that makes r=3 mean “two boxes can die.”
    • +
+

5.4 Isolate the nats +CTs from the rest of NS1

+

Host load on this box is often already several. Pin:

+
    +
  • nats-a/b/c: dedicated cores, no steal from keep/fleet +Node processes.
    • +
  • Memory high enough that file-backed streams stay cache-hot for the +working set.
    • +
  • cpuunits / cpuset in pct config so a +Zapier-facing Node GC pause does not stall fsync.
    • +
+

Re-run this study after pinning; H1/H2 should move more than +ping.

+

5.5 Consumer and mailbox +tuning (delay H4)

+

Flood delay is backlog / consume_rate. Fine-tune the +waiters, not the broker RTT.

+
    +
  • jobs.events and webhooks.deliver: raise +max_ack_pending so a slow HTTPS hook does not stall the +whole consumer; cap it so a poison message cannot unbounded-buffer +RAM.
    • +
  • Pull consumers: larger batch, shorter expires, more +pullers horizontally (fleet replica floors) instead of one fat +subscriber.
    • +
  • Middleware should not flood-publish then wait; it +already does per-job publish. Keep that. The flood test is the outage +profile when a consumer is stopped.
    • +
  • Alert on consumer lag (pending + ack pending) from +JetStream, not on ping RTT.
    • +
+

5.6 Publisher-side +batching in middleware

+

A timestamp job is one small JSON. 16k msgs/s is ample. Still:

+
    +
  • Avoid per-byte publishes; one message per job/event.
    • +
  • Reuse NATS connections (connection churn showed up as publisher +spread in the core 4p/8p runs).
    • +
  • Idempotent msg id / duplicate window sized to Verae +retry window, not default-only.
    • +
+

5.7 +nats-server knobs worth measuring (A/B with this script)

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
KnobWhy try it
max_payloadKeep default unless archive puts grow
write_deadlineSlow consumer protection for webhooks
max_pendingBound memory on a stuck Zapier hook
max_connectionsFleet workers + keep + middleware
JetStream max_file_store / +max_memory_storePrevent one stream from filling the CT
max_outstanding_catchupReplica restart after a nats-c blip
GOMAXPROCS = LXC coresDo not overthread a 2-core CT
+

Change one knob, re-run +study-on-ns1.sh, compare JetStream 1p 128 B and ping +p99.

+

5.8 Network

+
    +
  • Keep NATS off vmbr0. No change.
    • +
  • When on metal: dedicated NIC or VLAN for cluster :6222 +vs client :4222 if possible (replication vs client +load).
    • +
  • Check virtio queue counts on the LXC nics if core 1 KiB byte rate +plateaus.
    • +
+

5.9 Security cost (when +nkeys/mTLS flip)

+

verae-nats-accounts is still a sketch. Enabling accounts +will add CPU on publish. Budget: re-run this exact study +after creds are in every NATS_URL, and +accept a drop on both core and JS. Do not flip without that +measurement.

+

5.10 Operational +fine-tuning (lag, not peak msgs/s)

+
    +
  1. Scrape varz / jsz from the host over +vmbr1 (not public). Monitor loopback :8222 is +invisible to Prometheus on NS1 unless we add a host-side proxy on +10.10.10.21:8222 bound only to vmbr1.
    2. +
  2. Keep replica floors for webhook-deliver and job-poller — they are +the flood defense.
    3. +
  3. Backup/restore drill of JetStream during idle, then +a short JS 1p run to see catchup cost.
    4. +
  4. A 15–30 minute soak (not in this ladder) for page cache and +compaction; add that as a third study when disks are dedicated.
    5. +
+

5.11 What not to tune

+
    +
  • Do not chase core 8p8s aggregate. It is fan-out on a lab +bridge.
    • +
  • Do not treat flood 400 ms as “cluster RTT.” Fix consumers.
    • +
  • Do not load-test on ZAPIER_* streams.
    • +
  • Do not bind client NATS to 0.0.0.0 on +vmbr0.
    • +
+ +
    +
  1. CPU pin nats-a/b/c → re-run JS 1p + ping.
    2. +
  2. ZAPIER_EVENTS-shaped memory stream vs file (throwaway +stream, same flags as this JS ladder).
    3. +
  3. Distinct store_dir disks per node.
    4. +
  4. nkeys on, same ladder.
    5. +
  5. Three hardware boxes, same cluster.env IPs +updated.
    6. +
+

Each experiment should produce a new +results/<utc>/ on NS1 and a new progress-repo report +so we can diff H1–H5 instead of arguing from memory.

+
+

6. Reproducing this study

+

On NS1 only:

+
cd ~/verae-src/verae-nats-cluster
+bash scripts/study-on-ns1.sh
+

The script exits if hostname is not NS1. Outputs land in +results/<utc>/ including +nats-cluster-bench-ns1.{md,html,pdf} and +charts/. Copy those into +zapier-decisions/reports/ for the progress repo and +catalog.

+

Raw logs for this run: results/20260912T051237Z/.

+ + diff --git a/reports/ns1-host/nats-cluster-bench-ns1.md b/reports/ns1-host/nats-cluster-bench-ns1.md new file mode 100644 index 0000000..cf0011d --- /dev/null +++ b/reports/ns1-host/nats-cluster-bench-ns1.md @@ -0,0 +1,334 @@ +**Progress report (second study)** · run `20260912T051237Z` (UTC) + +> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): the orchestrator (`study-on-ns1.sh`), `nats bench`, `latency.mjs` (inside LXC 510 on this hypervisor), charting (`matplotlib`), and HTML/PDF (`pandoc` + `weasyprint`). The operator laptop did **not** publish, subscribe, draw charts, or render the PDF. Traffic stayed on `vmbr1` from LXC **510** to `nats-a/b/c` (**511–513**). + +This is a full methodology write-up plus the numbers from that on-host run. The earlier report (`nats-cluster-bench`, run `20260912T045131Z`) used the same cluster but was **orchestrated and rendered off-box**. Use this document when you need “it was all run on 138.” + +--- + +## 1. Executive summary + +| Item | This NS1-host run | +|------|-------------------| +| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `marchon` | +| Bench client | LXC 510 `verae-px-worker` | +| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` | +| Client URL | `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` | +| Host load before | `8.59 8.39 8.15 5/3843 4096277` | +| Host load after | `8.67 8.33 8.15 9/3863 4114865` | +| Core 1p1s 128 B pub | 502,502 msgs/s | +| JetStream 1p 128 B r=3 | 7,393 durable pubs/s | +| Ping p50 / p99 | 0.395ms / 1.377ms | + +Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst. + +--- + +## 2. Where it ran (and where it did not) + +```text +Operator laptop ──ssh──► NS1.GEORGELAMBERT.ORG 70.88.205.138 + study-on-ns1.sh + python3 build-ns1-study-report.py + sudo pct exec 510 ──► nats bench / latency.mjs + │ + ▼ vmbr1 + 10.10.10.21-23 :4222 +``` + +- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510). +- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file. + +--- + +## 3. Results (this run) + +### Host and brokers + +**Before** + +| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream | +|------|------|-------------|---------|----------|-----|-------|---------|-----------| +| nats-a | 511 | 2 | 1,721,203 | 4,933,759 | 1 | 1 | 26,398,720 | True | +| nats-b | 512 | 2 | 1,613,205 | 3,513,096 | 0 | 1 | 23,097,344 | True | +| nats-c | 513 | 2 | 1,760,973 | 4,048,582 | 2 | 1 | 23,867,392 | True | + +**After** + +| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream | +|------|------|-------------|---------|----------|-----|-------|---------|-----------| +| nats-a | 511 | 2 | 2,097,596 | 5,910,161 | 1 | 1 | 38,273,024 | True | +| nats-b | 512 | 2 | 2,121,816 | 4,746,539 | 1 | 1 | 41,361,408 | True | +| nats-c | 513 | 2 | 2,241,366 | 5,053,978 | 1 | 1 | 38,162,432 | True | + +nproc=40 · uname=`Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux` + +### Throughput + +| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s | +|-----|------|------------------|------------|----------|------------|----------| +| `core-1p1s-50k-128` | core pub/sub | 776,331 | 502,502 | 61.34 | 420,976 | 51.39 | +| `core-4p4s-100k-128` | core pub/sub | 1,278,454 | 379,985 | 46.38 | 1,078,910 | 131.70 | +| `core-4p4s-50k-1k` | core pub/sub | 581,464 | 209,545 | 204.63 | 482,645 | 471.33 | +| `core-8p8s-200k-128` | core pub/sub | 2,065,217 | 333,731 | 40.74 | 1,847,242 | 225.49 | +| `js-1p-20k-128-r3` | jetstream r=3 file | — | 7,393 | 0.90 | — | — | +| `js-2p2s-20k-128-r3` | jetstream r=3 file | 15,416 | 7,728 | 0.94 | 7,711 | 0.94 | +| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 14,985 | 14.63 | — | — | +| `js-4p-50k-128-r3` | jetstream r=3 file | — | 17,986 | 2.20 | — | — | + +### Round-trip delay + +| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max | +|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----| +| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.341ms | 0.470ms | 0.395ms | 0.651ms | 1.377ms | 3.243ms | +| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 132.425ms | 203.717ms | 204.458ms | 241.750ms | 249.468ms | 249.640ms | +| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 196.422ms | 221.264ms | 224.039ms | 233.180ms | 236.798ms | 238.177ms | +| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 211.108ms | 294.925ms | 300.254ms | 314.223ms | 315.551ms | 316.091ms | +| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 280.608ms | 414.972ms | 409.802ms | 513.821ms | 535.394ms | 536.084ms | + +### Core NATS + +![Core NATS throughput at four loads (NS1 host run)](charts/core-throughput.png) + +*Core NATS throughput at four loads (NS1 host run)* +### Payload size (core) + +![Core NATS 128 B vs 1 KiB (NS1 host run)](charts/payload-size.png) + +*Core NATS 128 B vs 1 KiB (NS1 host run)* +### JetStream r=3 file + +![JetStream durable publish rate (NS1 host run)](charts/js-throughput.png) + +*JetStream durable publish rate (NS1 host run)* +### Core vs JetStream + +![Core vs JetStream publish rate, log scale (NS1 host run)](charts/core-vs-js.png) + +*Core vs JetStream publish rate, log scale (NS1 host run)* +### Delay + +![Ping vs flood delay percentiles, log scale (NS1 host run)](charts/delay-percentiles.png) + +*Ping vs flood delay percentiles, log scale (NS1 host run)* + +--- + +## 4. Study methodology + +### 4.1 Question + +On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)? + +### 4.2 Hypotheses (stated before the run) + +1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica. +2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated. +3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting. +4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count. +5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound. + +### 4.3 Independent variables (what we changed) + +| Factor | Levels | +|--------|--------| +| Transport | Core NATS pub/sub vs JetStream file replicas=3 | +| Publisher count | 1, 2, 4, 8 | +| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 | +| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) | +| Payload | 128 B, 1024 B | +| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) | + +### 4.4 Dependent variables (what we recorded) + +| Metric | Instrument | Unit | +|--------|------------|------| +| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s | +| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s | +| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) | +| Publisher spread | nats min/avg/max **msgs/s** | not delay | +| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max | +| Host load | `/proc/loadavg` before and after | load average | +| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem | + +**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`. + +### 4.5 Controls and constants + +- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`. +- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`). +- Bench client is LXC **510**, not a nats-* server. +- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide. +- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`). +- No TLS, no nkeys, no account isolation (isolation is `vmbr1`). +- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder. + +### 4.6 Procedure + +1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise. +2. Snapshot host load, memory, LXC configs, and each nats `varz`. +3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB). +4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file. +5. Copy `latency.mjs` into 510; ping then flood at several batch sizes. +6. Snapshot host/`varz` again. +7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**. + +No publish, subscribe, chart, or PDF process runs on the operator laptop for this study. + +### 4.7 Instrumentation path + +```text +[NS1 host 70.88.205.138] + study-on-ns1.sh (bash + python3) + | + | sudo pct exec 510 + v +[LXC 510 verae-px-worker 10.10.10.20] + nats bench / node latency.mjs + | + | NATS client protocol to + v +[LXC 511/512/513 10.10.10.21-23 :4222] + nats-server -js cluster routes :6222 +``` + +The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`. + +### 4.8 Threats to validity + +| Threat | Effect on numbers | +|--------|-------------------| +| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. | +| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. | +| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. | +| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. | +| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. | +| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. | +| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. | +| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. | + +### 4.9 Ethics / safety + +Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket. + +--- + +## 5. Suggestions for fine-tuning + +These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way. + +### 5.1 Treat JetStream as the product limiter + +Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out. + +### 5.2 Split storage class by stream + +| Stream | Suggested store | Why | +|--------|-----------------|-----| +| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad | +| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both | +| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer | +| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry | +| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive | + +Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert. + +### 5.3 Give JetStream real disks + +Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain. + +- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`. +- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure. +- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it. +- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.” + +### 5.4 Isolate the nats CTs from the rest of NS1 + +Host load on this box is often already several. Pin: + +- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes. +- Memory high enough that file-backed streams stay cache-hot for the working set. +- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync. + +Re-run this study after pinning; H1/H2 should move more than ping. + +### 5.5 Consumer and mailbox tuning (delay H4) + +Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT. + +- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM. +- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber. +- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped. +- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT. + +### 5.6 Publisher-side batching in middleware + +A timestamp job is one small JSON. 16k msgs/s is ample. Still: + +- Avoid per-byte publishes; one message per job/event. +- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs). +- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only. + +### 5.7 nats-server knobs worth measuring (A/B with this script) + +| Knob | Why try it | +|------|------------| +| `max_payload` | Keep default unless archive puts grow | +| `write_deadline` | Slow consumer protection for webhooks | +| `max_pending` | Bound memory on a stuck Zapier hook | +| `max_connections` | Fleet workers + keep + middleware | +| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT | +| `max_outstanding_catchup` | Replica restart after a nats-c blip | +| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT | + +Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99. + +### 5.8 Network + +- Keep NATS off `vmbr0`. No change. +- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load). +- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus. + +### 5.9 Security cost (when nkeys/mTLS flip) + +`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement. + +### 5.10 Operational fine-tuning (lag, not peak msgs/s) + +1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`. +2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense. +3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost. +4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated. + +### 5.11 What not to tune + +- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge. +- Do not treat flood 400 ms as “cluster RTT.” Fix consumers. +- Do not load-test on `ZAPIER_*` streams. +- Do not bind client NATS to `0.0.0.0` on `vmbr0`. + +### 5.12 Recommended next experiments (same method, one change each) + +1. CPU pin nats-a/b/c → re-run JS 1p + ping. +2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder). +3. Distinct `store_dir` disks per node. +4. nkeys on, same ladder. +5. Three hardware boxes, same `cluster.env` IPs updated. + +Each experiment should produce a new `results//` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory. + + +--- + +## 6. Reproducing this study + +On **NS1 only**: + +```bash +cd ~/verae-src/verae-nats-cluster +bash scripts/study-on-ns1.sh +``` + +The script exits if `hostname` is not NS1. Outputs land in `results//` including `nats-cluster-bench-ns1.{md,html,pdf}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog. + +Raw logs for this run: `results/20260912T051237Z/`. diff --git a/reports/ns1-host/nats-cluster-bench-ns1.pdf b/reports/ns1-host/nats-cluster-bench-ns1.pdf new file mode 100644 index 0000000..c25d95e Binary files /dev/null and b/reports/ns1-host/nats-cluster-bench-ns1.pdf differ diff --git a/reports/ns1-maximize/charts/core-throughput.png b/reports/ns1-maximize/charts/core-throughput.png new file mode 100644 index 0000000..683f57e Binary files /dev/null and b/reports/ns1-maximize/charts/core-throughput.png differ diff --git a/reports/ns1-maximize/charts/core-vs-js.png b/reports/ns1-maximize/charts/core-vs-js.png new file mode 100644 index 0000000..b6a880c Binary files /dev/null and b/reports/ns1-maximize/charts/core-vs-js.png differ diff --git a/reports/ns1-maximize/charts/delay-percentiles.png b/reports/ns1-maximize/charts/delay-percentiles.png new file mode 100644 index 0000000..0b5f851 Binary files /dev/null and b/reports/ns1-maximize/charts/delay-percentiles.png differ diff --git a/reports/ns1-maximize/charts/delta-vs-baseline.png b/reports/ns1-maximize/charts/delta-vs-baseline.png new file mode 100644 index 0000000..f23250c Binary files /dev/null and b/reports/ns1-maximize/charts/delta-vs-baseline.png differ diff --git a/reports/ns1-maximize/charts/js-throughput.png b/reports/ns1-maximize/charts/js-throughput.png new file mode 100644 index 0000000..55452c7 Binary files /dev/null and b/reports/ns1-maximize/charts/js-throughput.png differ diff --git a/reports/ns1-maximize/charts/payload-size.png b/reports/ns1-maximize/charts/payload-size.png new file mode 100644 index 0000000..1118a42 Binary files /dev/null and b/reports/ns1-maximize/charts/payload-size.png differ diff --git a/reports/ns1-maximize/nats-cluster-bench-ns1.html b/reports/ns1-maximize/nats-cluster-bench-ns1.html new file mode 100644 index 0000000..c466b2a --- /dev/null +++ b/reports/ns1-maximize/nats-cluster-bench-ns1.html @@ -0,0 +1,1385 @@ + + + + + + + NATS cluster message speed — NS1 host study + + + + +
Verae Time × Zapier · progress report · maximized NS1 study

NATS cluster message speed — maximized (RAM disk + 8 cores)

packages/zapier-decisions/reports/nats-cluster-bench-ns1.md
+
+

NATS cluster message speed — NS1 host study

+
+

Progress report (maximized NS1 study) · run +20260912T053120Z (UTC)

+
+

Execution provenance. Every process for this study +ran on NS1.GEORGELAMBERT.ORG +(70.88.205.138): maximize-ns1-study.sh +(cores/RAM/max_mem/tmpfs), then +study-on-ns1.sh, nats bench, +latency.mjs (LXC 510), matplotlib, pandoc, weasyprint. +Traffic stayed on vmbr1. veth/10G was not +changed. After the ladder, JetStream was put back on ZFS and product +streams were re-created; 8 cores / 16 GiB / max_mem 8G +stay.

+
+

Measured delta vs +20260912T051237Z

+

Baseline: 1 core / 1 GiB / JetStream on ZFS. This run: 8 cores / 16 +GiB / JetStream tmpfs (file r=3) plus extra +memory store rows. veth/10G unchanged.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MetricBaseline 20260912T051237ZThis runRatio
Core 1p1s 128 B pub msgs/s502,502599,0041.19×
Core 8p8s 128 B aggregate msgs/s2,065,2171,998,7330.97×
JS file r=3 1p 128 B pub msgs/s7,39317,3882.35×
JS file r=3 4p 128 B pub msgs/s17,98622,8531.27×
JS file r=3 4p 1 KiB pub msgs/s14,98518,1141.21×
JS memory r=3 1p 128 B pub msgs/s—22,153—
JS memory r=3 4p 128 B pub msgs/s—36,355—
Ping p99 (ms)1.377ms0.684ms2.01× faster
+
+ + +
+

Baseline vs maximized publish rates (log)

+
+

1. Executive summary

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ItemThis NS1-host run
Control planeNS1.GEORGELAMBERT.ORG (70.88.205.138), user +marchon
Bench clientLXC 510 verae-px-worker
BrokersLXC 511/512/513 nats-a/b/c on +10.10.10.21–23
Client URLnats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
Host load before8.77 8.39 8.26 6/3849 198471
Host load after9.02 8.82 8.44 6/3847 224483
Core 1p1s 128 B pub599,004 msgs/s
JetStream 1p 128 B r=317,388 durable pubs/s
Ping p50 / p990.299ms / 0.684ms
+

Product traffic is the JetStream row. Ping is one-message delay. +Flood is mailbox catch-up after a burst.

+
+

2. Where it ran (and where it +did not)

+
Operator laptop  ──ssh──►  NS1.GEORGELAMBERT.ORG  70.88.205.138
+                              study-on-ns1.sh
+                              python3 build-ns1-study-report.py
+                              sudo pct exec 510 ──► nats bench / latency.mjs
+                                                    │
+                                                    ▼ vmbr1
+                                              10.10.10.21-23 :4222
+
    +
  • Did run on 138: bash, python3, matplotlib, pandoc, +weasyprint, pct, nats-server (in LXC), nats CLI and Node +(in LXC 510).
    • +
  • Did not run on the laptop: no local +nats bench, no local charting, no local WeasyPrint for this +file.
    • +
+
+

3. Results (this run)

+

Host and brokers

+

Before

+ +++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NodeVMIDconnectionsin_msgsout_msgscpucoresmem (B)jetstream
nats-a51133,0133,0371814,553,088True
nats-b51231,4151,4240113,557,760True
nats-c51301,3611,3951114,028,800True
+

After

+ +++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NodeVMIDconnectionsin_msgsout_msgscpucoresmem (B)jetstream
nats-a5113644,7301,344,7682867,104,768True
nats-b5123491,295978,8001182,542,592True
nats-c5130668,5441,330,9400137,314,560True
+

nproc=40 · +uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux

+

Throughput

+ +++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunModeAggregate msgs/sPub msgs/sPub MB/sSub msgs/sSub MB/s
core-1p1s-50k-128core pub/sub810,988599,00473.12456,11655.68
core-4p4s-100k-128core pub/sub1,361,921379,34646.311,150,536140.45
core-4p4s-50k-1kcore pub/sub695,192175,725171.61584,024570.34
core-8p8s-200k-128core pub/sub1,998,733283,25934.581,780,589217.36
js-1p-20k-128-r3jetstream r=3 file—17,3882.12——
js-2p2s-20k-128-r3jetstream r=3 file19,8769,9591.229,9421.21
js-4p-20k-1k-r3jetstream r=3 file—18,11417.69——
js-4p-50k-128-r3jetstream r=3 file—22,8532.79——
js-mem-1p-20k-128-r3jetstream r=3 file—22,1532.70——
js-mem-4p-20k-1k-r3jetstream r=3 file—28,68528.01——
js-mem-4p-50k-128-r3jetstream r=3 file—36,3554.44——
+

Round-trip delay

+ +++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RunKindCountPubsSizeminavgp50p90p99max
lat-ping-1k-128ping (sequential RTT)10001128 B0.250ms0.319ms0.299ms0.363ms0.684ms2.759ms
lat-1p-5k-128flood (burst queueing)50001128 B94.920ms132.026ms131.990ms156.086ms158.860ms158.965ms
lat-4p-5k-1kflood (burst queueing)500041024 B113.281ms132.425ms131.457ms140.317ms144.683ms145.365ms
lat-4p-10k-128flood (burst queueing)100004128 B153.318ms204.329ms205.865ms234.151ms237.346ms237.477ms
lat-8p-20k-128flood (burst queueing)200008128 B233.727ms310.924ms311.170ms373.437ms400.764ms402.811ms
+

Core NATS

+
+ + +
+

Core NATS throughput at four loads (NS1 host run) ### +Payload size (core)

+
+ + +
+

Core NATS 128 B vs 1 KiB (NS1 host run) ### JetStream r=3 +file

+
+ + +
+

JetStream durable publish rate (NS1 host run) ### Core vs +JetStream

+
+ + +
+

Core vs JetStream publish rate, log scale (NS1 host run) ### +Delay

+
+ + +
+

Ping vs flood delay percentiles, log scale (NS1 host +run)

+
+

4. Study methodology

+

4.1 Question

+

On the NS1 test stand, what message throughput and +delay does the three-node verae JetStream +cluster deliver at several loads, and which part of the stack is the +limiter for product traffic (jobs, events, webhooks, archive)?

+

4.2 Hypotheses (stated before +the run)

+
    +
  1. H1 — Core vs JetStream. Fire-and-forget core NATS +is at least an order of magnitude faster than JetStream file + +replicas=3, because durable publish waits for a majority disk +replica.
    2. +
  2. H2 — JetStream parallelism. Adding publishers does +not linearly increase JetStream write rate once the +replica log is saturated.
    3. +
  3. H3 — Quiet delay. Sequential pub→sub round trip on +vmbr1 is well under 1 ms p99 when the consumer is +waiting.
    4. +
  4. H4 — Burst delay. If publishers dump a batch before +the subscriber drains, observed delay is queueing time, +roughly linear in backlog, not in cluster hop count.
    5. +
  5. H5 — Payload. Moving 128 B → 1 KiB lowers message +rate and raises byte rate on core NATS; JetStream in this size band +stays replica/fsync bound.
    6. +
+

4.3 Independent variables +(what we changed)

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FactorLevels
TransportCore NATS pub/sub vs JetStream file replicas=3
Publisher count1, 2, 4, 8
Subscriber count0 (JS publish-only), 1, 2, 4, 8
Message count1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step)
Payload128 B, 1024 B
Delay modeping (publish, wait, repeat) vs +flood (publish all, then drain)
+

4.4 Dependent variables +(what we recorded)

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MetricInstrumentUnit
Publish ratenats bench 0.1.6 Pub statsmsgs/s, MB/s
Subscribe ratenats bench Sub statsmsgs/s, MB/s
Aggregatenats bench NATS Pub/Sub statsmsgs/s (fan-out counts both sides)
Publisher spreadnats min/avg/max msgs/snot delay
One-way-ish RTTlatency.mjs header timestampmin, avg, p50, p90, p99, max
Host load/proc/loadavg before and afterload average
Broker countershttp://127.0.0.1:8222/varz inside each nats LXCconnections, in/out msgs, cpu, mem
+

Important: nats CLI 0.1.6 min/avg/max are +rate spread across publishers, not microseconds of +delay. Delay is only latency.mjs.

+

4.5 Controls and constants

+
    +
  • Cluster name verae, three routes, client +:4222, cluster :6222, monitor loopback +:8222.
    • +
  • Client URL always the three-node list on vmbr1 (never +host 127.0.0.1:4222, never vmbr0).
    • +
  • Bench client is LXC 510, not a nats-* server.
    • +
  • JetStream bench stream name benchstream, +file storage, replicas=3, deleted +between JS loads (nats stream rm --force) so names do not +collide.
    • +
  • Product streams were not the bench target (no load +test on ZAPIER_* / VERAE_ARCHIVE).
    • +
  • No TLS, no nkeys, no account isolation (isolation is +vmbr1).
    • +
  • Same nats CLI version (0.1.6) and nats@2 Node client as +the first ladder.
    • +
+

4.6 Procedure

+
    +
  1. Confirm this script is executing on +NS1.GEORGELAMBERT.ORG. Refuse otherwise.
    2. +
  2. Snapshot host load, memory, LXC configs, and each nats +varz.
    3. +
  3. From NS1, pct exec 510 the core ladder (1p1s, 4p4s, +8p8s at 128 B; 4p4s at 1 KiB).
    4. +
  4. Delete benchstream; JS ladder (1p, 4p, 4p×1 KiB, 2p2s +pull) at replicas=3 file.
    5. +
  5. Copy latency.mjs into 510; ping then flood at several +batch sizes.
    6. +
  6. Snapshot host/varz again.
    7. +
  7. Parse logs on this host; draw charts; write HTML +and PDF on this host.
    8. +
+

No publish, subscribe, chart, or PDF process runs on the operator +laptop for this study.

+

4.7 Instrumentation path

+
[NS1 host 70.88.205.138]
+   study-on-ns1.sh  (bash + python3)
+        |
+        | sudo pct exec 510
+        v
+[LXC 510 verae-px-worker 10.10.10.20]
+   nats bench / node latency.mjs
+        |
+        | NATS client protocol to
+        v
+[LXC 511/512/513  10.10.10.21-23 :4222]
+   nats-server -js  cluster routes :6222
+

The hypervisor issues the guest commands. The messages themselves +never leave vmbr1.

+

4.8 Threats to validity

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ThreatEffect on numbers
One physical hostThree “replicas” share CPU, memory, and usually the same datastore. +This measures process/LXC HA, not disk HA.
Shared loadNS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. +Load average during a run is part of the result, not noise to +ignore.
Single bench clientAll publishers live in 510. Per-publisher rate spread is contention +in that guest.
Short runsSeconds of traffic. No compaction, no multi-hour page-cache +eviction, no snapshot during load.
No TLS/nkeysProduction auth will cost CPU. Do not treat these rates as +post-nkeys rates.
Fan-out aggregateCore aggregate msgs/s counts pub+sub. Do not compare that column to +JetStream unique writes.
Flood ≠ RTTMixing flood averages with ping p99 produces a fake “NATS is slow” +story.
Lab onlyNot a Zapier HTTPS bench and not live +api.veraetime.net.
+

4.9 Ethics / safety

+

Bench uses throwaway subjects (bench.core.*, +bench.js.*, bench.lat.*) and a throwaway +stream. It does not purge product streams. Zapier cloud has no NATS +socket.

+
+

5. Suggestions for fine-tuning

+

These follow from the method and from the first ladder on this stand +(JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of +ms). Apply in order of leverage. Re-run this NS1 study +after each change so the delta is measured the same way.

+

5.1 Treat JetStream as +the product limiter

+

Product jobs/events/webhooks/archive are durable. Tuning core NATS to +2M msgs/s will not move a timestamp Zap. Put effort into replica +write path and consumer lag, not core +fan-out.

+

5.2 Split storage class by +stream

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
StreamSuggested storeWhy
ZAPIER_JOBSfile, r=3Work queue; lose-a-job is bad
ZAPIER_EVENTSfile r=3, or memory r=3 if events are rebuildable from job +statusHot waiters; measure both
ZAPIER_WEBHOOKSfile, r=3, workqueueHTTPS to Zapier is the slow consumer
ZAPIER_USAGEfile, r=3, limits + max-ageTelemetry
VERAE_ARCHIVEfile, r=3, on the best diskPuts are larger and must survive
+

Try ZAPIER_EVENTS as memory store in a maintenance +window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms +and durable events still ack at a higher rate, keep it; if a CT restart +drops in-flight waiters, revert.

+

5.3 Give JetStream real disks

+

Today r=3 on three LXC guests on one Proxmox host is +three files, one failure domain.

+
    +
  • Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) +into each nats LXC store_dir.
    • +
  • Set sync: always only on archive if you need it; +default sync is often enough for jobs and is faster. Measure.
    • +
  • Do not put JetStream store_dir on the same busy rootfs +as Forgejo/Caddy if we can avoid it.
    • +
  • When moving to three metal boxes: same configs, private NIC, one +disk (or mirror) per node. That is the first change +that makes r=3 mean “two boxes can die.”
    • +
+

5.4 Isolate the nats +CTs from the rest of NS1

+

Host load on this box is often already several. Pin:

+
    +
  • nats-a/b/c: dedicated cores, no steal from keep/fleet +Node processes.
    • +
  • Memory high enough that file-backed streams stay cache-hot for the +working set.
    • +
  • cpuunits / cpuset in pct config so a +Zapier-facing Node GC pause does not stall fsync.
    • +
+

Re-run this study after pinning; H1/H2 should move more than +ping.

+

5.5 Consumer and mailbox +tuning (delay H4)

+

Flood delay is backlog / consume_rate. Fine-tune the +waiters, not the broker RTT.

+
    +
  • jobs.events and webhooks.deliver: raise +max_ack_pending so a slow HTTPS hook does not stall the +whole consumer; cap it so a poison message cannot unbounded-buffer +RAM.
    • +
  • Pull consumers: larger batch, shorter expires, more +pullers horizontally (fleet replica floors) instead of one fat +subscriber.
    • +
  • Middleware should not flood-publish then wait; it +already does per-job publish. Keep that. The flood test is the outage +profile when a consumer is stopped.
    • +
  • Alert on consumer lag (pending + ack pending) from +JetStream, not on ping RTT.
    • +
+

5.6 Publisher-side +batching in middleware

+

A timestamp job is one small JSON. 16k msgs/s is ample. Still:

+
    +
  • Avoid per-byte publishes; one message per job/event.
    • +
  • Reuse NATS connections (connection churn showed up as publisher +spread in the core 4p/8p runs).
    • +
  • Idempotent msg id / duplicate window sized to Verae +retry window, not default-only.
    • +
+

5.7 +nats-server knobs worth measuring (A/B with this script)

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
KnobWhy try it
max_payloadKeep default unless archive puts grow
write_deadlineSlow consumer protection for webhooks
max_pendingBound memory on a stuck Zapier hook
max_connectionsFleet workers + keep + middleware
JetStream max_file_store / +max_memory_storePrevent one stream from filling the CT
max_outstanding_catchupReplica restart after a nats-c blip
GOMAXPROCS = LXC coresDo not overthread a 2-core CT
+

Change one knob, re-run +study-on-ns1.sh, compare JetStream 1p 128 B and ping +p99.

+

5.8 Network

+
    +
  • Keep NATS off vmbr0. No change.
    • +
  • When on metal: dedicated NIC or VLAN for cluster :6222 +vs client :4222 if possible (replication vs client +load).
    • +
  • Check virtio queue counts on the LXC nics if core 1 KiB byte rate +plateaus.
    • +
+

5.9 Security cost (when +nkeys/mTLS flip)

+

verae-nats-accounts is still a sketch. Enabling accounts +will add CPU on publish. Budget: re-run this exact study +after creds are in every NATS_URL, and +accept a drop on both core and JS. Do not flip without that +measurement.

+

5.10 Operational +fine-tuning (lag, not peak msgs/s)

+
    +
  1. Scrape varz / jsz from the host over +vmbr1 (not public). Monitor loopback :8222 is +invisible to Prometheus on NS1 unless we add a host-side proxy on +10.10.10.21:8222 bound only to vmbr1.
    2. +
  2. Keep replica floors for webhook-deliver and job-poller — they are +the flood defense.
    3. +
  3. Backup/restore drill of JetStream during idle, then +a short JS 1p run to see catchup cost.
    4. +
  4. A 15–30 minute soak (not in this ladder) for page cache and +compaction; add that as a third study when disks are dedicated.
    5. +
+

5.11 What not to tune

+
    +
  • Do not chase core 8p8s aggregate. It is fan-out on a lab +bridge.
    • +
  • Do not treat flood 400 ms as “cluster RTT.” Fix consumers.
    • +
  • Do not load-test on ZAPIER_* streams.
    • +
  • Do not bind client NATS to 0.0.0.0 on +vmbr0.
    • +
+ +
    +
  1. CPU pin nats-a/b/c → re-run JS 1p + ping.
    2. +
  2. ZAPIER_EVENTS-shaped memory stream vs file (throwaway +stream, same flags as this JS ladder).
    3. +
  3. Distinct store_dir disks per node.
    4. +
  4. nkeys on, same ladder.
    5. +
  5. Three hardware boxes, same cluster.env IPs +updated.
    6. +
+

Each experiment should produce a new +results/<utc>/ on NS1 and a new progress-repo report +so we can diff H1–H5 instead of arguing from memory.

+
+

6. Reproducing this study

+

On NS1 only:

+
cd ~/verae-src/verae-nats-cluster
+bash scripts/study-on-ns1.sh
+

The script exits if hostname is not NS1. Outputs land in +results/<utc>/ including +nats-cluster-bench-ns1.{md,html,pdf} and +charts/. Copy those into +zapier-decisions/reports/ for the progress repo and +catalog.

+

Raw logs for this run: results/20260912T053120Z/.

+ + diff --git a/reports/ns1-maximize/nats-cluster-bench-ns1.md b/reports/ns1-maximize/nats-cluster-bench-ns1.md new file mode 100644 index 0000000..b0d8c69 --- /dev/null +++ b/reports/ns1-maximize/nats-cluster-bench-ns1.md @@ -0,0 +1,355 @@ +**Progress report (maximized NS1 study)** · run `20260912T053120Z` (UTC) + +> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): `maximize-ns1-study.sh` (cores/RAM/`max_mem`/tmpfs), then `study-on-ns1.sh`, `nats bench`, `latency.mjs` (LXC 510), matplotlib, pandoc, weasyprint. Traffic stayed on `vmbr1`. veth/10G was **not** changed. After the ladder, JetStream was put back on ZFS and product streams were re-created; **8 cores / 16 GiB / max_mem 8G stay**. + +## Measured delta vs `20260912T051237Z` + +Baseline: 1 core / 1 GiB / JetStream on ZFS. This run: 8 cores / 16 GiB / JetStream **tmpfs** (file r=3) plus extra **memory** store rows. veth/10G unchanged. + +| Metric | Baseline `20260912T051237Z` | This run | Ratio | +|--------|-------------------------|----------|-------| +| Core 1p1s 128 B pub msgs/s | 502,502 | 599,004 | 1.19× | +| Core 8p8s 128 B aggregate msgs/s | 2,065,217 | 1,998,733 | 0.97× | +| JS file r=3 1p 128 B pub msgs/s | 7,393 | 17,388 | 2.35× | +| JS file r=3 4p 128 B pub msgs/s | 17,986 | 22,853 | 1.27× | +| JS file r=3 4p 1 KiB pub msgs/s | 14,985 | 18,114 | 1.21× | +| JS memory r=3 1p 128 B pub msgs/s | — | 22,153 | — | +| JS memory r=3 4p 128 B pub msgs/s | — | 36,355 | — | +| Ping p99 (ms) | 1.377ms | 0.684ms | 2.01× faster | + +![Baseline vs maximized publish rates (log)](charts/delta-vs-baseline.png) + +*Baseline vs maximized publish rates (log)* + + +--- + +## 1. Executive summary + +| Item | This NS1-host run | +|------|-------------------| +| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `marchon` | +| Bench client | LXC 510 `verae-px-worker` | +| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` | +| Client URL | `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` | +| Host load before | `8.77 8.39 8.26 6/3849 198471` | +| Host load after | `9.02 8.82 8.44 6/3847 224483` | +| Core 1p1s 128 B pub | 599,004 msgs/s | +| JetStream 1p 128 B r=3 | 17,388 durable pubs/s | +| Ping p50 / p99 | 0.299ms / 0.684ms | + +Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst. + +--- + +## 2. Where it ran (and where it did not) + +```text +Operator laptop ──ssh──► NS1.GEORGELAMBERT.ORG 70.88.205.138 + study-on-ns1.sh + python3 build-ns1-study-report.py + sudo pct exec 510 ──► nats bench / latency.mjs + │ + ▼ vmbr1 + 10.10.10.21-23 :4222 +``` + +- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510). +- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file. + +--- + +## 3. Results (this run) + +### Host and brokers + +**Before** + +| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream | +|------|------|-------------|---------|----------|-----|-------|---------|-----------| +| nats-a | 511 | 3 | 3,013 | 3,037 | 1 | 8 | 14,553,088 | True | +| nats-b | 512 | 3 | 1,415 | 1,424 | 0 | 1 | 13,557,760 | True | +| nats-c | 513 | 0 | 1,361 | 1,395 | 1 | 1 | 14,028,800 | True | + +**After** + +| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream | +|------|------|-------------|---------|----------|-----|-------|---------|-----------| +| nats-a | 511 | 3 | 644,730 | 1,344,768 | 2 | 8 | 67,104,768 | True | +| nats-b | 512 | 3 | 491,295 | 978,800 | 1 | 1 | 82,542,592 | True | +| nats-c | 513 | 0 | 668,544 | 1,330,940 | 0 | 1 | 37,314,560 | True | + +nproc=40 · uname=`Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux` + +### Throughput + +| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s | +|-----|------|------------------|------------|----------|------------|----------| +| `core-1p1s-50k-128` | core pub/sub | 810,988 | 599,004 | 73.12 | 456,116 | 55.68 | +| `core-4p4s-100k-128` | core pub/sub | 1,361,921 | 379,346 | 46.31 | 1,150,536 | 140.45 | +| `core-4p4s-50k-1k` | core pub/sub | 695,192 | 175,725 | 171.61 | 584,024 | 570.34 | +| `core-8p8s-200k-128` | core pub/sub | 1,998,733 | 283,259 | 34.58 | 1,780,589 | 217.36 | +| `js-1p-20k-128-r3` | jetstream r=3 file | — | 17,388 | 2.12 | — | — | +| `js-2p2s-20k-128-r3` | jetstream r=3 file | 19,876 | 9,959 | 1.22 | 9,942 | 1.21 | +| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 18,114 | 17.69 | — | — | +| `js-4p-50k-128-r3` | jetstream r=3 file | — | 22,853 | 2.79 | — | — | +| `js-mem-1p-20k-128-r3` | jetstream r=3 file | — | 22,153 | 2.70 | — | — | +| `js-mem-4p-20k-1k-r3` | jetstream r=3 file | — | 28,685 | 28.01 | — | — | +| `js-mem-4p-50k-128-r3` | jetstream r=3 file | — | 36,355 | 4.44 | — | — | + +### Round-trip delay + +| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max | +|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----| +| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.250ms | 0.319ms | 0.299ms | 0.363ms | 0.684ms | 2.759ms | +| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 94.920ms | 132.026ms | 131.990ms | 156.086ms | 158.860ms | 158.965ms | +| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 113.281ms | 132.425ms | 131.457ms | 140.317ms | 144.683ms | 145.365ms | +| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 153.318ms | 204.329ms | 205.865ms | 234.151ms | 237.346ms | 237.477ms | +| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 233.727ms | 310.924ms | 311.170ms | 373.437ms | 400.764ms | 402.811ms | + +### Core NATS + +![Core NATS throughput at four loads (NS1 host run)](charts/core-throughput.png) + +*Core NATS throughput at four loads (NS1 host run)* +### Payload size (core) + +![Core NATS 128 B vs 1 KiB (NS1 host run)](charts/payload-size.png) + +*Core NATS 128 B vs 1 KiB (NS1 host run)* +### JetStream r=3 file + +![JetStream durable publish rate (NS1 host run)](charts/js-throughput.png) + +*JetStream durable publish rate (NS1 host run)* +### Core vs JetStream + +![Core vs JetStream publish rate, log scale (NS1 host run)](charts/core-vs-js.png) + +*Core vs JetStream publish rate, log scale (NS1 host run)* +### Delay + +![Ping vs flood delay percentiles, log scale (NS1 host run)](charts/delay-percentiles.png) + +*Ping vs flood delay percentiles, log scale (NS1 host run)* + +--- + +## 4. Study methodology + +### 4.1 Question + +On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)? + +### 4.2 Hypotheses (stated before the run) + +1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica. +2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated. +3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting. +4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count. +5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound. + +### 4.3 Independent variables (what we changed) + +| Factor | Levels | +|--------|--------| +| Transport | Core NATS pub/sub vs JetStream file replicas=3 | +| Publisher count | 1, 2, 4, 8 | +| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 | +| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) | +| Payload | 128 B, 1024 B | +| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) | + +### 4.4 Dependent variables (what we recorded) + +| Metric | Instrument | Unit | +|--------|------------|------| +| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s | +| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s | +| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) | +| Publisher spread | nats min/avg/max **msgs/s** | not delay | +| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max | +| Host load | `/proc/loadavg` before and after | load average | +| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem | + +**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`. + +### 4.5 Controls and constants + +- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`. +- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`). +- Bench client is LXC **510**, not a nats-* server. +- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide. +- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`). +- No TLS, no nkeys, no account isolation (isolation is `vmbr1`). +- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder. + +### 4.6 Procedure + +1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise. +2. Snapshot host load, memory, LXC configs, and each nats `varz`. +3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB). +4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file. +5. Copy `latency.mjs` into 510; ping then flood at several batch sizes. +6. Snapshot host/`varz` again. +7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**. + +No publish, subscribe, chart, or PDF process runs on the operator laptop for this study. + +### 4.7 Instrumentation path + +```text +[NS1 host 70.88.205.138] + study-on-ns1.sh (bash + python3) + | + | sudo pct exec 510 + v +[LXC 510 verae-px-worker 10.10.10.20] + nats bench / node latency.mjs + | + | NATS client protocol to + v +[LXC 511/512/513 10.10.10.21-23 :4222] + nats-server -js cluster routes :6222 +``` + +The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`. + +### 4.8 Threats to validity + +| Threat | Effect on numbers | +|--------|-------------------| +| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. | +| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. | +| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. | +| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. | +| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. | +| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. | +| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. | +| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. | + +### 4.9 Ethics / safety + +Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket. + +--- + +## 5. Suggestions for fine-tuning + +These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way. + +### 5.1 Treat JetStream as the product limiter + +Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out. + +### 5.2 Split storage class by stream + +| Stream | Suggested store | Why | +|--------|-----------------|-----| +| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad | +| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both | +| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer | +| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry | +| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive | + +Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert. + +### 5.3 Give JetStream real disks + +Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain. + +- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`. +- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure. +- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it. +- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.” + +### 5.4 Isolate the nats CTs from the rest of NS1 + +Host load on this box is often already several. Pin: + +- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes. +- Memory high enough that file-backed streams stay cache-hot for the working set. +- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync. + +Re-run this study after pinning; H1/H2 should move more than ping. + +### 5.5 Consumer and mailbox tuning (delay H4) + +Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT. + +- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM. +- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber. +- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped. +- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT. + +### 5.6 Publisher-side batching in middleware + +A timestamp job is one small JSON. 16k msgs/s is ample. Still: + +- Avoid per-byte publishes; one message per job/event. +- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs). +- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only. + +### 5.7 nats-server knobs worth measuring (A/B with this script) + +| Knob | Why try it | +|------|------------| +| `max_payload` | Keep default unless archive puts grow | +| `write_deadline` | Slow consumer protection for webhooks | +| `max_pending` | Bound memory on a stuck Zapier hook | +| `max_connections` | Fleet workers + keep + middleware | +| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT | +| `max_outstanding_catchup` | Replica restart after a nats-c blip | +| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT | + +Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99. + +### 5.8 Network + +- Keep NATS off `vmbr0`. No change. +- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load). +- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus. + +### 5.9 Security cost (when nkeys/mTLS flip) + +`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement. + +### 5.10 Operational fine-tuning (lag, not peak msgs/s) + +1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`. +2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense. +3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost. +4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated. + +### 5.11 What not to tune + +- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge. +- Do not treat flood 400 ms as “cluster RTT.” Fix consumers. +- Do not load-test on `ZAPIER_*` streams. +- Do not bind client NATS to `0.0.0.0` on `vmbr0`. + +### 5.12 Recommended next experiments (same method, one change each) + +1. CPU pin nats-a/b/c → re-run JS 1p + ping. +2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder). +3. Distinct `store_dir` disks per node. +4. nkeys on, same ladder. +5. Three hardware boxes, same `cluster.env` IPs updated. + +Each experiment should produce a new `results//` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory. + + +--- + +## 6. Reproducing this study + +On **NS1 only**: + +```bash +cd ~/verae-src/verae-nats-cluster +bash scripts/study-on-ns1.sh +``` + +The script exits if `hostname` is not NS1. Outputs land in `results//` including `nats-cluster-bench-ns1.{md,html,pdf}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog. + +Raw logs for this run: `results/20260912T053120Z/`. diff --git a/reports/ns1-maximize/nats-cluster-bench-ns1.pdf b/reports/ns1-maximize/nats-cluster-bench-ns1.pdf new file mode 100644 index 0000000..634bd81 Binary files /dev/null and b/reports/ns1-maximize/nats-cluster-bench-ns1.pdf differ diff --git a/scripts/build-nats-bench-report.py b/scripts/build-nats-bench-report.py new file mode 100644 index 0000000..8ea2701 --- /dev/null +++ b/scripts/build-nats-bench-report.py @@ -0,0 +1,247 @@ +#!/usr/bin/env python3 +"""Charts + HTML + PDF for the NATS cluster speed report in zapier-decisions/reports.""" +from __future__ import annotations + +import subprocess +import sys +from pathlib import Path + +import matplotlib + +matplotlib.use("Agg") +import matplotlib.pyplot as plt +from matplotlib.ticker import FuncFormatter + +ROOT = Path(__file__).resolve().parents[3] +PKG = Path(__file__).resolve().parents[1] +REPORT_DIR = PKG / "reports" +CHARTS = REPORT_DIR / "charts" +MD = REPORT_DIR / "nats-cluster-bench.md" +HTML = REPORT_DIR / "nats-cluster-bench.html" +PDF = REPORT_DIR / "nats-cluster-bench.pdf" +CSS = ROOT / "scripts" / "docs-print.css" +LUA = ROOT / "scripts" / "pdf-links.lua" + +INDIGO = "#4f46e5" +DEEP = "#312e81" +TEAL = "#047857" +AMBER = "#b45309" +LILAC = "#7c74f0" +INK = "#171a26" +MUTED = "#5b6178" +GRID = "#d9dce8" + +# Run 20260912T045131Z — parsed from nats bench logs (Sub stats, not Pub/Sub aggregate). +CORE = { + "labels": ["1p1s\n50k×128 B", "4p4s\n100k×128 B", "8p8s\n200k×128 B", "4p4s\n50k×1 KiB"], + "agg": [1_200_836, 1_521_256, 2_007_937, 630_460], + "pub": [791_094, 316_312, 333_957, 247_747], + "sub": [747_461, 1_299_634, 1_790_736, 510_216], + "pub_mb": [96.57, 38.61, 40.77, 241.94], + "sub_mb": [91.24, 158.65, 218.60, 498.26], +} +JS = { + "labels": ["1p 20k×128 B", "4p 50k×128 B", "4p 20k×1 KiB", "2p2s pull 20k×128 B"], + "pub": [16_155, 16_607, 13_493, 10_965], + "sub": [None, None, None, 10_942], +} +DELAY = { + "labels": ["Ping\n1k×128 B", "Flood 1p\n5k×128 B", "Flood 4p\n5k×1 KiB", "Flood 4p\n10k×128 B", "Flood 8p\n20k×128 B"], + "kind": ["ping", "flood", "flood", "flood", "flood"], + "p50": [0.286, 248.752, 217.579, 266.672, 466.296], + "p90": [0.332, 274.314, 223.268, 299.073, 499.924], + "p99": [0.734, 279.398, 227.798, 304.233, 505.112], + "avg": [0.307, 238.626, 211.706, 263.186, 453.749], +} + + +def style() -> None: + plt.rcParams.update( + { + "font.family": "sans-serif", + "font.size": 10, + "axes.titlesize": 12, + "axes.titleweight": "semibold", + "axes.labelsize": 10, + "axes.edgecolor": GRID, + "axes.labelcolor": INK, + "text.color": INK, + "xtick.color": MUTED, + "ytick.color": MUTED, + "figure.facecolor": "white", + "axes.facecolor": "white", + "axes.grid": True, + "grid.color": GRID, + "grid.linewidth": 0.8, + "legend.frameon": False, + "savefig.bbox": "tight", + "savefig.dpi": 160, + "savefig.facecolor": "white", + } + ) + + +def k_fmt(x: float, _pos: int | None = None) -> str: + if x >= 1_000_000: + return f" {x / 1_000_000:.2f}M".strip() + if x >= 1000: + return f"{x / 1000:.0f}k" + return f"{x:.0f}" + + +def save(fig: plt.Figure, name: str) -> None: + CHARTS.mkdir(parents=True, exist_ok=True) + path = CHARTS / name + fig.savefig(path, dpi=160) + plt.close(fig) + print(f"wrote {path}") + + +def chart_core_msgs() -> None: + fig, ax = plt.subplots(figsize=(9.2, 4.4)) + x = range(len(CORE["labels"])) + w = 0.25 + ax.bar([i - w for i in x], CORE["agg"], w, label="Aggregate", color=DEEP) + ax.bar(list(x), CORE["pub"], w, label="Publish", color=INDIGO) + ax.bar([i + w for i in x], CORE["sub"], w, label="Subscribe", color=TEAL) + ax.set_xticks(list(x), CORE["labels"]) + ax.set_ylabel("messages / second") + ax.set_title("Core NATS throughput (fire-and-forget pub/sub)") + ax.yaxis.set_major_formatter(FuncFormatter(k_fmt)) + ax.legend(loc="upper left") + ax.set_axisbelow(True) + save(fig, "core-throughput.png") + + +def chart_js_msgs() -> None: + fig, ax = plt.subplots(figsize=(9.2, 4.4)) + colors = [INDIGO, INDIGO, AMBER, LILAC] + ax.bar(JS["labels"], JS["pub"], color=colors) + ax.set_ylabel("durable publish messages / second") + ax.set_title("JetStream file store, replicas=3 (product-stream settings)") + ax.yaxis.set_major_formatter(FuncFormatter(k_fmt)) + ax.set_axisbelow(True) + for i, v in enumerate(JS["pub"]): + ax.text(i, v + 250, f"{v:,}", ha="center", va="bottom", fontsize=9, color=MUTED) + save(fig, "js-throughput.png") + + +def chart_core_vs_js() -> None: + fig, ax = plt.subplots(figsize=(9.2, 4.4)) + labels = ["1 publisher\n128 B", "4 publishers\n128 B", "4 publishers\n1 KiB"] + core_pub = [791_094, 316_312, 247_747] + js_pub = [16_155, 16_607, 13_493] + x = range(len(labels)) + w = 0.35 + ax.bar([i - w / 2 for i in x], core_pub, w, label="Core NATS (no disk)", color=INDIGO) + ax.bar([i + w / 2 for i in x], js_pub, w, label="JetStream r=3 file", color=AMBER) + ax.set_xticks(list(x), labels) + ax.set_yscale("log") + ax.set_ylabel("publish messages / second (log)") + ax.set_title("Core vs JetStream: two different jobs") + ax.legend(loc="upper right") + ax.set_axisbelow(True) + save(fig, "core-vs-js.png") + + +def chart_bytes() -> None: + fig, axes = plt.subplots(1, 2, figsize=(9.2, 4.2)) + labels = ["128 B\n4p4s", "1 KiB\n4p4s"] + msgs = [1_521_256, 630_460] + mb = [185.70, 615.68] + axes[0].bar(labels, msgs, color=[INDIGO, AMBER]) + axes[0].set_title("Aggregate messages / second") + axes[0].yaxis.set_major_formatter(FuncFormatter(k_fmt)) + axes[0].set_axisbelow(True) + axes[1].bar(labels, mb, color=[INDIGO, AMBER]) + axes[1].set_title("Aggregate MB / second") + axes[1].set_ylabel("MB/s") + axes[1].set_axisbelow(True) + for ax, vals, fmt in ( + (axes[0], msgs, lambda v: f"{v/1e6:.2f}M"), + (axes[1], mb, lambda v: f"{v:.0f}"), + ): + for i, v in enumerate(vals): + ax.text(i, v * 1.02, fmt(v), ha="center", va="bottom", fontsize=9, color=MUTED) + fig.suptitle("Core NATS: bigger payloads move more bytes, fewer messages", fontsize=12, fontweight="semibold") + fig.tight_layout() + save(fig, "payload-size.png") + + +def chart_delay() -> None: + fig, ax = plt.subplots(figsize=(9.2, 4.6)) + x = range(len(DELAY["labels"])) + w = 0.25 + ax.bar([i - w for i in x], DELAY["p50"], w, label="p50", color=TEAL) + ax.bar(list(x), DELAY["p90"], w, label="p90", color=INDIGO) + ax.bar([i + w for i in x], DELAY["p99"], w, label="p99", color=AMBER) + ax.set_xticks(list(x), DELAY["labels"]) + ax.set_yscale("log") + ax.set_ylabel("milliseconds (log)") + ax.set_title("Round-trip delay: sequential ping vs burst flood") + ax.axhline(1.0, color=GRID, linestyle="--", linewidth=1) + ax.legend(loc="upper left") + ax.set_axisbelow(True) + save(fig, "delay-percentiles.png") + + +def render() -> None: + if not MD.exists(): + raise SystemExit(f"missing {MD}") + header = HTML.with_suffix(".hdr.html") + banner = HTML.with_suffix(".ban.html") + css_text = CSS.read_text(encoding="utf-8") + header.write_text(f"\n", encoding="utf-8") + banner.write_text( + '
' + '' + '
Verae Time × Zapier · progress report
' + "

NATS cluster message speed

" + '
packages/zapier-decisions/reports/nats-cluster-bench.md
' + "
\n", + encoding="utf-8", + ) + r = subprocess.run( + [ + "pandoc", + str(MD), + "-o", + str(HTML), + "--standalone", + f"--resource-path={REPORT_DIR}", + "--syntax-highlighting=breezedark", + "--metadata=title=NATS cluster message speed", + f"--include-in-header={header}", + f"--include-before-body={banner}", + f"--lua-filter={LUA}", + ], + capture_output=True, + text=True, + ) + header.unlink(missing_ok=True) + banner.unlink(missing_ok=True) + if r.returncode != 0: + sys.stderr.write(r.stderr) + raise SystemExit(f"pandoc failed: {r.returncode}") + w = subprocess.run(["weasyprint", str(HTML), str(PDF)], capture_output=True, text=True) + if w.returncode != 0: + sys.stderr.write(w.stderr) + raise SystemExit(f"weasyprint failed: {w.returncode}") + print(f"wrote {HTML}") + print(f"wrote {PDF}") + + +def main() -> int: + style() + chart_core_msgs() + chart_js_msgs() + chart_core_vs_js() + chart_bytes() + chart_delay() + render() + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())