marchon/zapier-decisions
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Snapshot of zapier-decisions (optimal NATS config study)

Browse source
This commit is contained in:
George Lambert 2026-09-12 02:06:06 -04:00
commit 13f1663238
42 changed files with 6600 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

155
LOG.md Normal file
View file

@ -0,0 +1,155 @@
# Action log
## 2026-09-12 — exhaustive optimal-config study
- 8c/16G ZFS factorial: file/memory × r=1/r=3, 4 KiB, reconnect, UDP echo, MQTT QoS0 on nats-a :1883 (then removed).
- JS memory r=1 4p **65k** pubs/s; file r=3 1p **14.3k**; MQTT QoS0 **45k**; UDP p99 **0.36 ms**.
- Combined report + DL360 Gen10 projection: `reports/optimal-config/REPORT.{md,html,pdf}`.
- Verdict: keep NATS+JetStream; no MQTT/UDP fabric; metal = 3 NVMe + 10GbE, expect JS r=3 ~40–80k.
## 2026-09-12 — maximized NS1 NATS study
- `pct set` 510–513 to 8 cores / 16 GiB; `max_mem: 8G`; tmpfs 8G on JetStream for the ladder only.
- Run `20260912T053120Z`. JS file 1p 128 B **7.4k → 17.4k (2.35×)**; JS memory 4p **36.4k**; ping p99 **1.38 → 0.68 ms**.
- tmpfs unmounted after; product streams on ZFS again. Resources and max_mem **left raised**.
- Report: `reports/ns1-maximize/`.
## 2026-09-12 — NS1-host NATS study (second run)
- All orchestration, nats bench, matplotlib, pandoc, and weasyprint ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`). Laptop not in the path.
- Script: `verae-nats-cluster/scripts/study-on-ns1.sh` (refuses unless hostname is NS1).
- Run `20260912T051237Z`. Report: `reports/ns1-host/nats-cluster-bench-ns1.{md,html,pdf}`.
- Host load ~8.5. JS 1p 128 B ~7.4k durable pubs/s; ping p99 1.38 ms. Methodology + fine-tuning in that report.
## 2026-09-12 — NATS cluster speed report
- Bench from LXC 510 vs nats-a/b/c (`20260912T045131Z`).
- Full write-up with charts in `reports/nats-cluster-bench.{md,html,pdf}`.
- JetStream r=3 file ~16k durable 128 B pubs/s; ping RTT ~0.3 ms; flood queueing 150–500 ms.
## 2026-09-12 — test env on the 3-node cluster
- Fleet, keep, middleware, billing, and access planes `NATS_URL` → `10.10.10.21–23:4222`.
- Streams ZAPIER_JOBS/EVENTS/WEBHOOKS/USAGE + VERAE_ARCHIVE replicas=3.
- px-worker keep worm/tree 200 on the cluster. Backup timer enabled. Uptime loop on px-worker.
- MOCK_VERAE still true. Zapier push still waiting on login. nkeys/mTLS not flipped (would require client creds).
## 2026-09-12 — NATS cluster on three Proxmox LXC
- New repo **verae-nats-cluster**: LXC 511/512/513 (`nats-a/b/c`, `10.10.10.21–23`) JetStream cluster on `vmbr1`.
- Lab loopback `127.0.0.1:4222` unchanged until cut-over.
- Test: stream replicas=3; same configs later move to distinct hardware.
## 2026-09-11 — proxmox worker + deploy/monitor/backup
- NS1 **is** Proxmox (`vmbr0` public, `vmbr1` `10.10.10.0/24`). New repo **verae-proxmox** creates LXC **510** `verae-px-worker` at `10.10.10.20`.
- NATS for guests: socat `10.10.10.1:4222` → `127.0.0.1:4222` (never vmbr0).
- **verae-uptime** off-box HTTPS GET; **verae-backup** tar snapshots; **verae-deploy** tagged npm ci.
- Fleet `overlays/ns1/` checked in; `start.sh` no longer rewrites fleet/machines JSON.
- User systemd + linger for keep and fleet.
## 2026-09-11 — bootstrap
- Created this repo to record decisions while executing cleanup items 1–5 (and a second pass on 2–5).
- Assumption: single developer, lab only; breaking changes allowed.
## Step 1 — account-balance is the prepaid source of truth
- Persist books to `BOOKS_PATH` (atomic JSON).
- Mutations go through `balance.adjust` / `usage.recorded`; edge cache `balanceCents` is updated from the books reply only.
- Local cache mutate only when `NATS_URL` and `ACCOUNT_BALANCE_URL` are both unset (tests).
## Step 2 — identity join
- `zappier-identity` mailbox: `verae.identity.bind` / `verae.identity.lookup`.
- Portal + middleware tenants store `veraeUserId`; JWT never public.
## Step 3 — access planes are the public doors
- zappier-edge binds loopback (`127.0.0.1:3000`).
- Public: access-web `:3021`, access-api `:3022`, access-leaf `:3023`, access-zapier `:3024`, access-staff `:3025`.
## Step 4 — leaf hardening
- `verae-nats-accounts` holds nats.conf + subject policy.
- Leaf token required; billing subjects denied in policy file and access-leaf.
## Step 5 — fleet spawn
- Prefer local machines for HTTP `spawn.command` services.
- Disable lan-134 unless `FLEET_ENABLE_LAN134=1`.
- SSH spawn timeout 8s; failed hosts skipped.
## 2026-09-11 — catalog live doors
- Bottom of https://zapier.georgelambert.org/ lists each HTTPS door with description, screenshot, and Open link.
- UI-REVIEW.pdf includes the live-door atlas. Walkthrough §13.
## 2026-09-11 — Caddy host names live
- Installed `/etc/caddy/sites/zapier-services.georgelambert.org.caddy` on NS1. HTTPS: portal, api, leaf, zap, staff, iam, login, cs, sales, acct, mw, fleet.
- Not proxied: NATS, keep, workers, loopback edge.
## 2026-09-11 — host names
- Mapped every running service to **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`).
- Suggested `*.zapier.georgelambert.org` names for public doors; only `zapier.georgelambert.org` (docs) and `git.georgelambert.org` exist in Caddy today.
## 2026-09-11 — docs sync, bootstrap, NS1 all-in-one
- Catalog and ops docs: public portal is access-web `:3021/portal/`; edge is loopback; IAM + keep listed.
- New `verae-bootstrap` repo: clone + npm for `ns1-all-in-one`, `control-plane`, `ns1-archive`, `lan-worker`.
- Fleet starts `staff-iam` and `staff-session`. NS1 all-in-one uses edge `:13000` because host `:3000` is taken; keep still owns archive workers. `better-sqlite3` must be rebuilt on Linux (`npm rebuild`).
## 2026-09-11 — verae-keep on NS1
- New repo `verae-keep`: keep `:3860` restarts crashed units; watch `:3861` restarts keep; `guard.sh` restarts watch.
- Honors admin pause/stop via intent files + `POST /intent`. Fleet writes `data/<instance>/intent.json` (local and SSH).
- Tested on 70.88.205.138: crash-restart, pause-hold, watch respawn. webhook-deliver brought back. NATS observed only.
## 2026-09-11 — IAM JSON doors closed
- CS/sales/accounting/staff JSON list and sales PUT pricing now go through IAM (401 JSON, 302 HTML).
- Fleet service env sets `STAFF_AUTH=1` with `STAFF_IAM_URL`.
- Re-walk: cs credit 200 agent=`cs`; cs export 403; operator Fleet POST 200; admin `/me` `permissions:['*']`.
## 2026-09-11 — IAM on lab, portal door, hardening
- Restarted fleet with `STAFF_IAM_URL`. Walk: cs credits (agent=`cs`) and is 403 on accounting export; operator fleet POST 200, cs 403; admin `/me` permissions `*`.
- Portal public door `verae-access-web` `:3021/portal/` (static + `/portal/api` → loopback edge).
- IAM sessions persist in JSON; login rate-limit per IP+username (8/10min); credits stamp IAM username as authz principal.
- lan-134 remains disabled unless `FLEET_ENABLE_LAN134=1`. Receipt PDF branded (indigo header + legal footer).
## 2026-09-11 — staff IAM
- New `verae-staff-iam` :3028 — users, roles, permissions, sessions, audit UI.
- Admin login uses IAM when `STAFF_IAM_URL` is set; tabs hide without permission.
- CS/sales/accounting/access-staff/fleet mutating APIs check `cs.credit`, `sales.quote`, `accounting.export`, `fleet.operate`, etc.
- Seed: admin/cs/sales/accounting/operator. Shared staff key is fallback only.
## 2026-09-11 — last three UI leftovers
- Swagger `/docs` stays stock; banner names it OpenAPI explorer.
- Shared template `verae-staff-ui` for CS + access-staff.
- `STAFF_COOKIE_DOMAIN` + nginx example for multi-host.
## 2026-09-11 — names, staff session, exclusive jobs.events
- Account-balance stores display names (`customer.put` + lookup by name). Edge writes names on customer create/edit; CS/sales/accounting/staff join from edge if the ledger has no name.
- New repo `verae-staff-session` (:3027). Department HTML redirects when `STAFF_AUTH=1`.
- `JOBS_EVENTS_EXCLUSIVE=1` makes `verae-jobs-events` the durable consumer; middleware webhook router backs off.
- Catalog index is cards. Disabled fleet machines are grey, not degraded yellow.
## 2026-09-11 — UI design-system pass
- Restyled CS/sales/accounting/access-staff to portal indigo; dollars + names.
- Fleet ⋯ menus, role chips, Docs cards, 0.0.0.0 copy, simulator indigo (orange = faults).
- Portal masked API key; admin customer drawer; empty invoice illustrations.
- Catalog system-ui. New independent repo **UI-Docs** (walkthrough, screenshots, UI-REVIEW.pdf).
## Second pass (2–5)
- Identity used from portal when `IDENTITY_URL` set.
- Leaf loads allow-list from nats-accounts `policy.json`.
- Staff plane is a first-class mailbox (`verae-access-staff`).
- jobs.events mailbox process `verae-jobs-events` (passthrough + count).

3
NATS.md Normal file
View file

@ -0,0 +1,3 @@
# NATS — zapier-decisions
None. Documentation repo.

18
README.md Normal file
View file

@ -0,0 +1,18 @@
# zapier-decisions
Lab log of architecture decisions, actions taken, and open todos while cleaning the Verae × Zapier stack.
**Forgejo:** https://git.georgelambert.org/marchon/zapier-decisions
| File | What |
|------|------|
| [STATUS.md](STATUS.md) | **Current status** (2026-09-12) |
| [reports/nats-cluster-bench.md](reports/nats-cluster-bench.md) | NATS cluster speed report (charts, HTML, PDF) |
| [reports/ns1-host/nats-cluster-bench-ns1.md](reports/ns1-host/nats-cluster-bench-ns1.md) | Second study: all code on NS1 (methodology + tuning) |
| [reports/ns1-maximize/nats-cluster-bench-ns1.md](reports/ns1-maximize/nats-cluster-bench-ns1.md) | Maximized: 8c/16G/tmpfs vs baseline |
| [reports/optimal-config/REPORT.md](reports/optimal-config/REPORT.md) | All studies + MQTT/UDP + DL360 projection |
| [LOG.md](LOG.md) | Chronological actions |
| [TODO.md](TODO.md) | Open items |
| [decisions/](decisions/) | One file per decision |
No runtime. Not a NATS mailbox.

158
STATUS.md Normal file
View file

@ -0,0 +1,158 @@
# Current status — 2026-09-12
Verae Time × Zapier is a **working lab** on one Proxmox host (**NS1.GEORGELAMBERT.ORG**, `70.88.205.138`). Public HTTPS doors, catalog, keep/fleet, and a **test 3-node NATS cluster** on separate LXC guests are up. The test environment **points at that cluster**. Zapier Platform `push` and live `api.veraetime.net` are still waiting on operator login / bind credentials.
**Progress repo:** this file in [zapier-decisions](https://git.georgelambert.org/marchon/zapier-decisions).
**Catalog:** https://zapier.georgelambert.org/packages/zapier-decisions/STATUS.pdf
**Monorepo:** `master-zapier-plan-draft` (`main` / `master`).
---
## Live right now
| Layer | State |
|-------|--------|
| Catalog | https://zapier.georgelambert.org/ (PDF links path-relative; module/model books work) |
| Public doors | portal, api, zap, iam, mw, fleet, git — last off-box GET 8/8 OK |
| Keep + fleet | Running. systemd + linger **enabled** for next reboot (current processes were not double-started as systemd) |
| Test NATS | **3-node JetStream on `vmbr1`:** LXC 511/512/513 `nats-a/b/c` at `10.10.10.21–23`. Cluster name `verae`, 2 routes each. |
| Product streams | `ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE` — **replicas=3** |
| Modules on that cluster | Fleet, keep, middleware (`natsConnected: true`), billing, access planes. `NATS_URL` lists all three URLs. |
| Host loopback `:4222` | Still listening; **clients no longer use it** |
| Worker CT **510** `verae-px-worker` | `10.10.10.20` on `vmbr1`. Keep worm+tree **200** on the cluster |
| Backup | Timer `verae-backup.timer` enabled; snapshots under `/SSD2/backups/verae/` |
| Uptime | Loop on px-worker (`:3870/status`), last tick `failed: 0` |
Client URL (private `vmbr1` only):
```text
nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
```
Repo: https://git.georgelambert.org/marchon/verae-nats-cluster
### Cluster message speed (`20260912T045131Z`)
Measured from LXC **510** (not a nats-* server) against `10.10.10.21–23`.
**Full report (charts + explanations):** [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.pdf) · [markdown](reports/nats-cluster-bench.md)
**Second study (all code on NS1.GEORGELAMBERT.ORG / 70.88.205.138, run `20260912T051237Z`):** [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.pdf) · [markdown](reports/ns1-host/nats-cluster-bench-ns1.md). Orchestrator, nats bench, charts, HTML, and PDF ran on 138; traffic was LXC 510 → 511–513 on `vmbr1`. Includes full methodology and fine-tuning notes. Host load ~8.5/40 during the run; JetStream 1p 128 B was **7.4k** durable pubs/s (vs ~16k on the quieter first ladder); ping p99 **1.38 ms**.
**Maximized study (`20260912T053120Z`):** 8 cores + 16 GiB on 510–513, `max_mem: 8G`, JetStream **tmpfs** for the ladder (ZFS restored after). veth/10G unchanged. [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.pdf).
| Metric | Baseline 051237Z | Maximized | Ratio |
|--------|------------------|-----------|-------|
| JS file r=3 1p 128 B | 7,393 pubs/s | **17,388** | **2.35×** |
| JS memory r=3 4p 128 B | — | **36,355** | ~4.9× vs baseline file 1p |
| Core 1p 128 B pub | 502,502 | 599,004 | 1.19× |
| Ping p99 | 1.377 ms | **0.684 ms** | **2.01× faster** |
Live cluster **keeps 8 cores / 16 GiB / max_mem 8G**. Product streams are back on ZFS (tmpfs was study-only).
**Optimal-config study (`20260912T055851Z`):** same 8c/16G cluster on **ZFS**, plus r=1 vs r=3, memory vs file, reconnect tax, UDP echo, MQTT QoS0. Combined report with DL360 Gen10 projection: [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/optimal-config/REPORT.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/optimal-config/REPORT.pdf).
| Finding | Number |
|---------|--------|
| JS file r=3 1p 128 B (8c ZFS) | 14,330 pubs/s |
| JS file r=1 1p | 18,888 (r=3 costs ~24% on same host) |
| JS memory r=1 4p | **64,923** (lab JS ceiling) |
| MQTT QoS0 | 44,862 pubs/s (not durable; ~15× slower than core) |
| UDP p99 510→511 | 0.363 ms vs NATS ping p99 1.14 ms |
| Reconnect-per-msg p99 | 1.75 ms vs persistent ping 1.14 ms |
**Do not switch the fabric to MQTT or UDP.** Optimal lab: file r=3 for jobs/archive, memory r=3 optional for events, persistent NATS connections. Metal: 3× DL360 Gen10, local NVMe, 10GbE — projected JS file r=3 **~40–80k** 128 B pubs/s (not measured).
Short tables: [BENCH.md](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md). Re-run: `bash scripts/bench.sh` on NS1.
| Kind | Load | Result |
|------|------|--------|
| Core NATS | 1p1s 50k×128 B | ~1.20M msgs/s aggregate |
| Core NATS | 8p8s 200k×128 B | ~2.01M msgs/s aggregate |
| Core NATS | 4p4s 50k×1 KiB | ~630k msgs/s (~616 MB/s) |
| JetStream file r=3 | 1–4 pubs, 128 B | ~16k durable pubs/s |
| JetStream file r=3 | 4 pubs, 1 KiB | ~13.5k durable pubs/s |
| Ping (sequential RTT) | 1k×128 B | avg 0.307 ms, p99 0.734 ms |
| Flood (burst queueing) | 8p 20k×128 B | avg 454 ms |
Product job/event/archive traffic is the JetStream row. Ping is one-message delay; flood is the mailbox catching up after a burst.
---
## Built and checked in (Forgejo, each its own repo)
IAM, portal on access-web, keep/watch/guard, fleet floors, host-deps, catalog host-move docs, plus:
| Repo | Job |
|------|-----|
| verae-uptime | Off-box HTTPS GET of public doors |
| verae-backup | SQLite / keep / worm-tree tar; restore dry-run default |
| verae-deploy | Tagged checkout + npm ci on the target OS |
| verae-proxmox | LXC 510 worker provision |
| verae-nats-cluster | LXC 511–513 JetStream cluster |
---
## Still open
| Item | Notes |
|------|--------|
| Zapier Platform `push` | Needs Zapier login (see below) |
| `MOCK_VERAE=false` | Live `api.veraetime.net` + admin bind credentials (see below) |
| NATS nkeys/mTLS | Cluster is up **without** accounts so existing `NATS_URL`s work. `verae-nats-accounts` remains the sketch. Flip only after every client has credentials. Isolation today is `vmbr1`, not nkeys. |
| lan-134 | Still disabled (SSH not working) |
| Reboot test | systemd should own keep/fleet after reboot; do not also `nohup guard.sh` |
| Hardware NATS | Same three configs, three boxes, private NIC — change IPs in `cluster.env` |
---
## Once we have a Zapier login
Zapier cloud only talks **HTTPS** to our Zap plane (`https://zap.zapier.georgelambert.org`). No NATS, no `api.veraetime.net`.
1. `zapier login` (or `zapier-platform login`).
2. In `packages/verae-zapier` (and `verae-activate` if the tiny app is still wanted):
- `zapier-platform register` (or attach an existing app id).
- `MIDDLEWARE_BASE_URL=https://zap.zapier.georgelambert.org` (or `https://mw.zapier.georgelambert.org` if skipping the Zap plane).
- `zapier-platform build` and `validate`.
3. `zapier-platform push` as a **private** app.
4. Invite the workspace; create a Zap:
- Auth = portal **API key** (`zmw_…`), not a Verae JWT.
- Test **Create Timestamp and Wait**, **Create Timestamp** (async `jobId`), **Find Job Status**, **Timestamp Completed** REST Hook.
5. Fix anything `validate` / `push` flags (auth labels, samples, 402/403 mapping).
6. When the contract is stable, bump the platform version and invite a real user.
Do **not** put NATS URLs or Verae passwords into Zapier.
---
## Once `MOCK_VERAE=false` and live `api.veraetime.net`
Middleware already has the live HTTP client. Still needed:
1. `VERAE_API_BASE_URL=https://api.veraetime.net` and working **login** (username/password → JWT). JWT stays in middleware (`tokenRef`); Zapier never sees it.
2. Confirm OpenAPI vs our client:
- `POST /api/timestamp` → **202** `{ jobId }`
- `GET /api/status/{jobId}` until `completed` / `failed`
- `POST /api/verify`
- Batch create/status if used
3. **Hash lookup is not on the live OpenAPI** — `veraeClient.lookupHash` returns **501** when not mocking. Central-chain “find by SHA-256” needs a Verae API gap fill or our own store.
4. Map live status JSON into the receipt shape the Zap app expects (`certificate`, `timestamp`, `blockIndex`, `sha256`).
5. Token refresh, 401 retry, and 402/403 mapping against real errors.
6. One live seal in the lab, then REST Hook delivery to `hooks.zapier.com`.
---
## Submit → jobId → wait → receipt
**Yes — that is the designed path**, and it already runs in **mock**.
1. Zap (or curl) `POST /zapier/v1/timestamp` or `/timestamp/wait`.
2. Middleware `createTimestamp` calls Verae `POST /api/timestamp` (or the mock) and gets **`jobId`**.
3. It publishes **`verae.zapier.jobs.watch`** on NATS (the 3-node cluster).
4. **Wait:** subscribe to **`verae.zapier.jobs.events`** for that `jobId`, or poll `GET /api/status/{jobId}`.
5. Job-poller hits Verae status until `completed`.
6. Event `timestamp.completed` carries status; webhook-deliver POSTs the Zapier REST Hook.
7. Response includes **certificate / timestamp / blockIndex** (mock uses `mock-cert-{jobId}`, `blockIndex: 42`). Async create returns `{ jobId, sha256, existing }` immediately; wait returns the finished status. Same SHA-256 is idempotent (`existing: true`).
If wait times out, middleware returns **`pending` + `jobId`** so the hook can finish the Zap. Live Verae must match that HTTP contract; we have **not** proven a real chain seal yet (`MOCK_VERAE` is still true).

15
TODO.md Normal file
View file

@ -0,0 +1,15 @@
# Open todos
- [ ] Live `api.veraetime.net` with `MOCK_VERAE=false` and admin bind credentials.
- [x] Three-node NATS JetStream cluster on distinct Proxmox LXC (`verae-nats-cluster`; test env `NATS_URL` cut over; streams replicas=3).
- [ ] NATS nkeys/mTLS on that cluster (accounts file is still the lab stand-in; not flipped so clients keep working on `vmbr1`).
- [x] Exclusive JetStream consumer for `verae.zapier.jobs.events` on `verae-jobs-events` (`JOBS_EVENTS_EXCLUSIVE=1`; middleware skips the router).
- [x] Auth on CS/sales/accounting HTML via `verae-staff-session` (`STAFF_AUTH=1`).
- [x] Staff IAM: named users, roles, permissions (`verae-staff-iam` :3028).
- [ ] Zapier Platform `push` of a private app.
- [x] Move portal static files fully into `verae-access-web` (`/portal/` public door; API proxied to loopback edge).
- [x] Turn IAM on for the running lab (fleet `STAFF_IAM_URL` + restart).
- [x] IAM hardening: JSON sessions, login rate-limit, JSON 401 as well as HTML 302, credit principal = IAM username.
- [x] Host keep-alive (`verae-keep`) on NS1 with watch+guard; honors admin pause/stop.
- [x] Docs synced to IAM, public portal `:3021`, keep, bootstrap server types.
- [x] `verae-bootstrap` clones all module repos and installs deps per server type.

6
decisions/D001-prepaid-sot.md Normal file
View file

@ -0,0 +1,6 @@
# D001 — Account-balance is the only prepaid writer
**Status:** accepted
**Step:** 1
Prepaid cents, credits, usage, and payments persist in `zappier-account-balance` (`books.json`). zappier-edge `balanceCents` is a cache filled from the books reply. Tests without NATS/HTTP books still mutate the cache so unit tests stay hermetic.

6
decisions/D002-identity.md Normal file
View file

@ -0,0 +1,6 @@
# D002 — veraeUserId is the join key, not the JWT
**Status:** accepted
**Step:** 2
Public token remains the zappier API key. Chain JWT stays server-side (`tokenRef`). `zappier-identity` is the mailbox for bind/lookup so portal and middleware share one map.

6
decisions/D003-public-doors.md Normal file
View file

@ -0,0 +1,6 @@
# D003 — Access planes are the only public HTTP doors
**Status:** accepted
**Step:** 3
Zapier → `:3024`, browser → `:3021`, customer API → `:3022`, leaf → `:3023`, staff → `:3025`. zappier-edge listens on `127.0.0.1:3000` for identity, Stripe, admin, and as upstream for those planes.

6
decisions/D004-leaf-accounts.md Normal file
View file

@ -0,0 +1,6 @@
# D004 — Leaf cannot touch billing, even if authz is skipped
**Status:** accepted
**Step:** 4
`verae-nats-accounts/policy.json` is the allow-list. access-leaf refuses any target not in the leaf set. nats.conf documents INTERNAL vs LEAF accounts for when the lab cluster is replaced with a real accounts-enabled nats-server.

6
decisions/D005-fleet-local-first.md Normal file
View file

@ -0,0 +1,6 @@
# D005 — HTTP services spawn locally; lan-134 stays off
**Status:** accepted
**Step:** 5
SSH to `70.88.205.134` blocked keepFloor. HTTP apps (`spawn.command`) prefer `kind=local`. `lan-134` is disabled unless `FLEET_ENABLE_LAN134=1`. Failed SSH hosts are skipped after 8s.

6
package.json Normal file
View file

@ -0,0 +1,6 @@
{
"name": "zapier-decisions",
"version": "0.1.0",
"private": true,
"description": "Architecture decisions and action log for the Verae × Zapier stack"
}

BIN
reports/charts/core-throughput.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 50 KiB

BIN
reports/charts/core-vs-js.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 38 KiB

BIN
reports/charts/delay-percentiles.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 41 KiB

BIN
reports/charts/js-throughput.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 49 KiB

BIN
reports/charts/payload-size.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 58 KiB

1278
reports/nats-cluster-bench.html Normal file
View file

File diff suppressed because it is too large Load diff

312
reports/nats-cluster-bench.md Normal file
View file

@ -0,0 +1,312 @@
**Progress report** · 2026-09-12 · run `20260912T045131Z` (UTC)
This is the full write-up of the test-environment NATS cluster bench: what was measured, how, the numbers, the charts, and what they mean for Verae Time × Zapier. Short tables also live in [`verae-nats-cluster/BENCH.md`](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md). Raw logs and CSVs are in that repo under `results/20260912T045131Z/`.
---
## 1. Executive summary
The test cluster is three JetStream nodes on private `vmbr1` (LXC 511–513). The bench client is a **fourth** guest (LXC 510), so the numbers are cluster-plus-network, not a process talking to itself on loopback.
Two different systems were measured, on purpose:
| System | What it is | What we got |
|--------|------------|-------------|
| **Core NATS** | Fire-and-forget pub/sub. No disk, no replica ack. | About **0.75–2.0 million msgs/s** at 128 B, depending on fan-out. At 1 KiB, about **630k msgs/s** and **~616 MB/s** aggregate. |
| **JetStream file, replicas=3** | Durable, replicated — **this is what product streams use**. | About **16k durable 128 B pubs/s**, about **13.5k** at 1 KiB. Pull consume keeps up with publish at ~11k msgs/s each side. |
| **Ping delay** | One message at a time, publish then wait. | **avg 0.307 ms**, **p99 0.734 ms**, max 2.76 ms (1k × 128 B). |
| **Flood delay** | Publishers dump a batch; subscriber drains. | **150–505 ms**. That is **queueing under burst**, not wire time. |
**For this product:** timestamp jobs, job events, webhooks, and archive puts go through JetStream r=3. Plan capacity against **~16k durable msgs/s** on this stand, not the million-msg core numbers. A quiet job-event hop is a fraction of a millisecond. If a mailbox falls behind, delay jumps into hundreds of milliseconds — that is the flood column.
Core NATS is still useful: it is the ceiling for non-durable fan-out on this host, and it shows `vmbr1` and the nats-server processes are not the JetStream bottleneck. JetStream is.
---
## 2. Why this test exists
The lab cut the test environment over to the three-node cluster. Before treating that cluster as the message fabric for keep, fleet, middleware, billing, and archive workers, we needed:
1. **Throughput at several loads** — one publisher vs many, 128 B vs 1 KiB, core vs durable.
2. **Delay characteristics** — both the quiet path (one message RTT) and the overloaded path (burst into a mailbox).
3. **A client that is not a nats-* server** — otherwise we would be measuring loopback on the broker.
This is a **lab stand on one Proxmox host**, not three metal boxes. It answers “is this cluster in the right order of magnitude for our traffic?” It does not replace a soak test on dedicated disks.
---
## 3. Topology
```text
vmbr1 10.10.10.0/24 (not on vmbr0, not public)
-----------------------------------------------
LXC 510 LXC 511 LXC 512 LXC 513
verae-px-worker nats-a nats-b nats-c
10.10.10.20 10.10.10.21 10.10.10.22 10.10.10.23
bench client :4222 client :4222 :4222
:6222 routes :6222 :6222
:8222 loopback :8222 :8222
```
- Cluster name: `verae`. Each server has two routes to the other two.
- Client URL: `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222`
- HTTP monitor is **loopback :8222** inside each guest. Zapier cloud never talks to NATS.
- Product streams already on this cluster (`ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE`) use **file** storage and **replicas=3**. The JetStream bench used the same settings on a throwaway stream `benchstream`.
- Host `127.0.0.1:4222` is still listening on NS1; **clients no longer use it**.
Credits for the stack: Scott Lindsey, George Lambert, NATS.IO, Grok-Code.
---
## 4. Method
### 4.1 Tools
| Piece | Role |
|-------|------|
| `nats` CLI **0.1.6** | Throughput (`nats bench --no-progress --csv`). Its min/avg/max are **publisher rate spread**, not delay. |
| `scripts/latency.mjs` | Two connections, header timestamp `t`, delay = receive time − send time. |
| `scripts/bench.sh` | Runs the ladder from NS1 via `pct exec` on VMID 510. |
| `scripts/bench-report.py` | Turns logs into the short `BENCH.md` table. |
Re-run on NS1, from `verae-nats-cluster`:
```bash
bash scripts/bench.sh
```
### 4.2 Load ladder
**Core NATS** (subject `bench.core.*`):
| Run | Publishers | Subscribers | Messages | Payload |
|-----|------------|-------------|----------|---------|
| `core-1p1s-50k-128` | 1 | 1 | 50,000 | 128 B |
| `core-4p4s-100k-128` | 4 | 4 | 100,000 | 128 B |
| `core-8p8s-200k-128` | 8 | 8 | 200,000 | 128 B |
| `core-4p4s-50k-1k` | 4 | 4 | 50,000 | 1024 B |
**JetStream** (`--js --storage file --replicas 3 --stream benchstream`). The stream is deleted between loads so the name never collides:
| Run | Shape | Messages | Payload |
|-----|-------|----------|---------|
| `js-1p-20k-128-r3` | 1 publisher | 20,000 | 128 B |
| `js-4p-50k-128-r3` | 4 publishers | 50,000 | 128 B |
| `js-4p-20k-1k-r3` | 4 publishers | 20,000 | 1024 B |
| `js-2p2s-20k-128-r3` | 2 pub + 2 pull sub | 20,000 | 128 B |
**Delay** (core subjects, two connections):
| Run | Mode | Count | Pubs | Payload |
|-----|------|-------|------|---------|
| `lat-ping-1k-128` | **ping** — publish, wait for that message, repeat | 1,000 | 1 | 128 B |
| `lat-1p-5k-128` | **flood** — publish all, then drain | 5,000 | 1 | 128 B |
| `lat-4p-10k-128` | flood | 10,000 | 4 | 128 B |
| `lat-8p-20k-128` | flood | 20,000 | 8 | 128 B |
| `lat-4p-5k-1k` | flood | 5,000 | 4 | 1024 B |
Ping answers “how long does one quiet hop take?” Flood answers “what happens to the last message if we burst N messages into a mailbox?” Those are different questions. Mixing them is how 0.3 ms and 400 ms get confused.
### 4.3 How to read nats bench columns
- **Pub msgs/s** — rate at which publishers finished their share.
- **Sub msgs/s** — rate at which subscribers finished. With several subscribers on the same subject, core NATS **fans out**, so sub rate can exceed pub rate.
- **Aggregate msgs/s** — nats CLI `NATS Pub/Sub stats` line (pub+sub work in one number). Useful as a headline; do not treat it as “the network carried this many unique messages.”
- Empty JetStream sub cells mean that run was publish-only (durable write, no consumer in the same process).
---
## 5. Throughput results
### 5.1 Core NATS
![Core NATS throughput at four loads](charts/core-throughput.png)
| Run | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
|-----|------------------|------------|----------|------------|----------|
| `core-1p1s-50k-128` | 1,200,836 | 791,094 | 96.57 | 747,461 | 91.24 |
| `core-4p4s-100k-128` | 1,521,256 | 316,312 | 38.61 | 1,299,634 | 158.65 |
| `core-8p8s-200k-128` | 2,007,937 | 333,957 | 40.77 | 1,790,736 | 218.60 |
| `core-4p4s-50k-1k` | 630,460 | 247,747 | 241.94 | 510,216 | 498.26 |
**What this chart is saying.** Adding subscribers raises **aggregate** and **sub** rates because each published message is delivered to every subscriber. Publish rate does **not** climb the same way: 1 publisher at 128 B already pushes ~791k msgs/s; 4 and 8 publishers sit around 310–335k msgs/s **each process slower**, while fan-out on the sub side goes to 1.3M then 1.8M.
That publisher slowdown is expected on this stand. The four/eight publisher processes and the four/eight subscribers all run **inside one LXC** (510) against three broker LXCs on the **same Proxmox CPU and `vmbr1`**. Per-publisher logs show a wide spread (example, 4p core 128 B: 79k–524k msgs/s among the four pubs). That is CPU scheduling and client-side contention, not a NATS cluster that only has one fast node.
**1:1 at 128 B** is the cleanest core number: **~791k pub, ~747k sub, ~1.20M aggregate**. The cluster and the bridge can move three-quarter-million small messages per second fire-and-forget from a single client pair.
### 5.2 Payload size (core)
![Core NATS 128 B vs 1 KiB](charts/payload-size.png)
Same 4p4s shape, two sizes:
| Payload | Aggregate msgs/s | Aggregate MB/s | Pub msgs/s | Sub msgs/s |
|---------|------------------|----------------|------------|------------|
| 128 B | 1,521,256 | 185.70 | 316,312 | 1,299,634 |
| 1 KiB | 630,460 | 615.68 | 247,747 | 510,216 |
Message rate falls; **byte rate rises** (186 MB/s → 616 MB/s aggregate). We are leaving the “tiny message, CPU/syscall bound” region and entering “copying bytes across `vmbr1`.” Job JSON and archive metadata sit nearer 128 B–1 KiB than megabyte blobs (blobs are HTTP/WORM, not NATS payloads).
### 5.3 JetStream r=3 file
![JetStream durable publish rate](charts/js-throughput.png)
| Run | Pub msgs/s | Pub MB/s | Sub msgs/s | Notes |
|-----|------------|----------|------------|-------|
| `js-1p-20k-128-r3` | **16,155** | 1.97 | — | publish-only |
| `js-4p-50k-128-r3` | **16,607** | 2.03 | — | four pubs, same ceiling |
| `js-4p-20k-1k-r3` | **13,493** | 13.18 | — | 1 KiB still disk/replica bound |
| `js-2p2s-20k-128-r3` | 10,965 | 1.34 | 10,942 | pull consumers keep up |
**Four publishers do not make JetStream four times faster.** 1p and 4p at 128 B are both ~16k msgs/s. The limiter is **synchronous replication to three file-backed replicas**, not client parallelism. That is the result we wanted to see: the bench stream is behaving like a replicated log, not like core fan-out.
Pull consume (`js-2p2s`) is slightly slower on publish (~11k) because the same run is also reading. Pub and sub stay matched (10,965 vs 10,942): the consumer is not the straggler.
1 KiB durable write is ~13.5k msgs/s (~13.2 MB/s). Bytes go up; message rate dips only a little. JetStream here is **ack/fdatasync/replica** bound, not payload-copy bound, in this size range.
### 5.4 Core vs JetStream (same client, same cluster)
![Core vs JetStream publish rate, log scale](charts/core-vs-js.png)
The log scale is required: core publish is **~15–50×** JetStream publish on this stand.
| Shape | Core pub msgs/s | JS r=3 file pub msgs/s | Ratio |
|-------|-----------------|------------------------|-------|
| 1 publisher, 128 B | 791,094 | 16,155 | ~49× |
| 4 publishers, 128 B | 316,312 | 16,607 | ~19× |
| 4 publishers, 1 KiB | 247,747 | 13,493 | ~18× |
This is not JetStream “losing.” Core is allowed to forget a message the instant the server accepts it. JetStream on file with replicas=3 must **record it on a majority** before the publish acks. Our product streams (`ZAPIER_*`, `VERAE_ARCHIVE`) chose that trade on purpose: a job event that survives one LXC dying is worth ~16k msgs/s instead of ~800k.
If we ever need core-like rates for a signal that may drop, that signal should not be on a replicated file stream.
---
## 6. Delay results
![Delay percentiles, log scale](charts/delay-percentiles.png)
| Run | Kind | Count | min | avg | p50 | p90 | p99 | max |
|-----|------|-------|-----|-----|-----|-----|-----|-----|
| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 0.254 ms | **0.307 ms** | 0.286 ms | 0.332 ms | **0.734 ms** | 2.763 ms |
| `lat-1p-5k-128` | flood | 5000 | 149.3 ms | 238.6 ms | 248.8 ms | 274.3 ms | 279.4 ms | 279.7 ms |
| `lat-4p-5k-1k` | flood | 5000 | 155.1 ms | 211.7 ms | 217.6 ms | 223.3 ms | 227.8 ms | 228.4 ms |
| `lat-4p-10k-128` | flood | 10000 | 174.2 ms | 263.2 ms | 266.7 ms | 299.1 ms | 304.2 ms | 304.5 ms |
| `lat-8p-20k-128` | flood | 20000 | 304.6 ms | 453.7 ms | 466.3 ms | 499.9 ms | 505.1 ms | 505.6 ms |
The dashed line on the chart is 1 ms. Only **ping** lives there.
### 6.1 Ping — the quiet hop
One publisher, one subscriber, two connections, wait for each message before sending the next.
- **min 0.254 ms** — guest → `vmbr1` → a nats-server → `vmbr1` → guest.
- **p50 0.286 ms / avg 0.307 ms** — typical.
- **p99 0.734 ms** — still under a millisecond.
- **max 2.763 ms** — one outlier in 1,000 samples (GC, scheduler, or a slow route). Not the tail we design for.
A middleware `jobs.watch` publish followed by a waiter on `jobs.events` is this shape when the poller is keeping up. Compared with HTTPS to Zapier (tens to hundreds of milliseconds) or a live Verae `GET /api/status/{jobId}`, NATS RTT is noise.
### 6.2 Flood — queueing under burst
Publishers write the whole batch as fast as they can, then the subscriber drains. Each message’s delay is “how long was I in the buffer before the subscriber got to me?”
That is why:
- **min is already ~150–300 ms** — even the first messages wait behind a burst that filled the socket/client queue.
- **p50 ≈ p99 ≈ max** — a queue drain has a tight distribution: everyone waits for roughly the same backlog.
- **8p × 20k is ~450 ms avg** — twice the messages of 4p × 10k, roughly twice the wait. Linear in backlog, not in cluster diameter.
Flood is **not** a measurement of NATS being slow. The ping column proves the hop is ~0.3 ms. Flood is a measurement of **what operators will see if a consumer stalls** (job-events mailbox, webhook deliver, archive reply). Backlog time ≈ `queued_messages / consume_rate`.
### 6.3 1 KiB flood vs 128 B flood
4 publishers, 5k messages at 1 KiB: avg **212 ms**, slightly **faster** than 4p 10k × 128 B (263 ms) because the **count is half**, even though each message is 8× larger. Again: delay here tracks **how many messages are queued**, not payload size, in this range.
---
## 7. What this means for Verae × Zapier
Product subjects on this cluster:
| Address | Kind | Bench analogue |
|---------|------|----------------|
| `verae.zapier.jobs.watch` | work queue (JetStream) | JS durable pub ~16k/s |
| `verae.zapier.jobs.events` | events | JS + ping if waiters keep up; flood if they do not |
| `verae.zapier.webhooks.deliver` | work queue | JS durable |
| `verae.zapier.usage` | optional | JS durable |
| `verae.billing.*` | request-reply | ping (quiet RTT) |
| `verae.archive.put` / `query` / `reply.*` | JetStream + broadcast query | JS durable; query fan-out is closer to core but still JS-backed puts |
**Capacity.** 16k durable 128 B pubs/s is **~1.4×10⁹ messages/day** if you could fill the pipe. We will not. Zapier HTTPS, live `api.veraetime.net`, WORM bloom checks, and human Zap runs sit far below that. This cluster is not the product bottleneck on NS1.
**Latency budget.** A timestamp wait is: HTTP in → NATS watch → poll Verae → NATS event → HTTP out (or REST Hook). The NATS pieces are **sub-millisecond** when caught up. Do not spend time “optimizing NATS RTT” until Zapier/Verae HTTP is in the same band.
**Backlogs.** The failure mode that *does* show up in these numbers is flood delay. If webhook-deliver or job-events consumers pause (keep stopped, replica floor, a blocked HTTPS post to `hooks.zapier.com`), waiters will see **hundreds of milliseconds to seconds** of queue time. Fleet replica floors and keep exist to prevent that, not because 0.3 ms is too slow.
**Hardware move.** Same three configs, three boxes, private NIC. Expect:
- Core numbers to change with NIC and CPU (maybe up, maybe down).
- JetStream numbers to change **more**, because they are disk + fsync + replica RTT. Distinct SSDs should help; a slow shared datastore would hurt.
- Ping RTT to grow by whatever the real NIC and switch add (still likely low milliseconds on a LAN).
---
## 8. Limits of this measurement
1. **One Proxmox host.** LXC 510–513 share cores, memory, and the host’s disk. Replica=3 on file is **three files on the same underlying storage**, not three failure domains. HA of “one disk dies” is **not** proven. HA of “one LXC process dies” is the actual claim.
2. **Short runs.** Tens of thousands of messages, seconds of wall time. No compaction, no multi-hour page-cache eviction, no snapshot/restore during load.
3. **No TLS, no nkeys.** `verae-nats-accounts` is still a sketch. Auth would add CPU; it would not turn 16k into 800k.
4. **One bench client.** All publishers live in 510. A fleet of workers on several CTs might publish more into JetStream until disk/replicas saturate — the 1p vs 4p JS result says that saturation is already ~16k from one CT.
5. **nats 0.1.6** does not report delay. Anyone reading `min | avg | max msgs` on a bench log as microseconds will get the wrong story. Delay is only `latency.mjs`.
6. **Core aggregate ≠ unique messages.** Fan-out double-counts. Use pub or sub columns when comparing to JetStream.
7. **Not a Zapier or Verae API bench.** Those are still blocked on operator login / live credentials.
---
## 9. How to reproduce
On NS1 (Proxmox), from the `verae-nats-cluster` checkout:
```bash
bash scripts/status.sh # 3/3 JetStream
bash scripts/bench.sh # writes results/<utc>/ and BENCH.md
```
The client VMID defaults to **510**. Override with `CLIENT_VMID=…`. `NATS_URL` comes from `client.env`.
Rebuild this progress report (charts + HTML + PDF) from the monorepo:
```bash
python3 packages/zapier-decisions/scripts/build-nats-bench-report.py
```
---
## 10. Appendix — environment and files
| Item | Value |
|------|-------|
| Run stamp | `20260912T045131Z` |
| Client | LXC 510 `verae-px-worker` `10.10.10.20` |
| Servers | 511/512/513 `nats-a/b/c` `10.10.10.21–23` |
| nats CLI | 0.1.6 linux-amd64 |
| JS storage | file, replicas=3, stream `benchstream` (deleted between loads) |
| Isolation | `vmbr1` only; no `0.0.0.0` client bind |
| Short tables | [BENCH.md](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md) |
| Raw logs | `packages/verae-nats-cluster/results/20260912T045131Z/` |
| This report | `packages/zapier-decisions/reports/nats-cluster-bench.{md,html,pdf}` |
Publisher rate spread (nats CLI, msgs/s, **not** delay):
| Run | min | avg | max |
|-----|-----|-----|-----|
| core-4p4s-100k-128 pub | 79,260 | 257,805 | 524,453 |
| core-8p8s-200k-128 pub | 41,744 | 71,488 | 152,536 |
| core-4p4s-50k-1k pub | 61,936 | 110,271 | 176,262 |
| js-4p-50k-128-r3 pub | 4,154 | 5,176 | 6,628 |
| js-4p-20k-1k-r3 pub | 3,373 | 4,063 | 5,121 |
| js-2p2s-20k-128-r3 pub | 5,485 | 7,081 | 8,678 |
Wide core spreads are the single-client-CT effect described in §5.1. JetStream spreads are narrow and low — every publisher is waiting on the same replicated write path.

BIN
reports/nats-cluster-bench.pdf Normal file
View file

Binary file not shown.

BIN
reports/ns1-host/charts/core-throughput.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 56 KiB

BIN
reports/ns1-host/charts/core-vs-js.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 42 KiB

BIN
reports/ns1-host/charts/delay-percentiles.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 40 KiB

BIN
reports/ns1-host/charts/js-throughput.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 49 KiB

BIN
reports/ns1-host/charts/payload-size.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 52 KiB

1291
reports/ns1-host/nats-cluster-bench-ns1.html Normal file
View file

File diff suppressed because it is too large Load diff

334
reports/ns1-host/nats-cluster-bench-ns1.md Normal file
View file

@ -0,0 +1,334 @@
**Progress report (second study)** · run `20260912T051237Z` (UTC)
> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): the orchestrator (`study-on-ns1.sh`), `nats bench`, `latency.mjs` (inside LXC 510 on this hypervisor), charting (`matplotlib`), and HTML/PDF (`pandoc` + `weasyprint`). The operator laptop did **not** publish, subscribe, draw charts, or render the PDF. Traffic stayed on `vmbr1` from LXC **510** to `nats-a/b/c` (**511–513**).
This is a full methodology write-up plus the numbers from that on-host run. The earlier report (`nats-cluster-bench`, run `20260912T045131Z`) used the same cluster but was **orchestrated and rendered off-box**. Use this document when you need “it was all run on 138.”
---
## 1. Executive summary
| Item | This NS1-host run |
|------|-------------------|
| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `marchon` |
| Bench client | LXC 510 `verae-px-worker` |
| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` |
| Client URL | `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` |
| Host load before | `8.59 8.39 8.15 5/3843 4096277` |
| Host load after | `8.67 8.33 8.15 9/3863 4114865` |
| Core 1p1s 128 B pub | 502,502 msgs/s |
| JetStream 1p 128 B r=3 | 7,393 durable pubs/s |
| Ping p50 / p99 | 0.395ms / 1.377ms |
Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst.
---
## 2. Where it ran (and where it did not)
```text
Operator laptop ──ssh──► NS1.GEORGELAMBERT.ORG 70.88.205.138
study-on-ns1.sh
python3 build-ns1-study-report.py
sudo pct exec 510 ──► nats bench / latency.mjs
│
▼ vmbr1
10.10.10.21-23 :4222
```
- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510).
- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file.
---
## 3. Results (this run)
### Host and brokers
**Before**
| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream |
|------|------|-------------|---------|----------|-----|-------|---------|-----------|
| nats-a | 511 | 2 | 1,721,203 | 4,933,759 | 1 | 1 | 26,398,720 | True |
| nats-b | 512 | 2 | 1,613,205 | 3,513,096 | 0 | 1 | 23,097,344 | True |
| nats-c | 513 | 2 | 1,760,973 | 4,048,582 | 2 | 1 | 23,867,392 | True |
**After**
| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream |
|------|------|-------------|---------|----------|-----|-------|---------|-----------|
| nats-a | 511 | 2 | 2,097,596 | 5,910,161 | 1 | 1 | 38,273,024 | True |
| nats-b | 512 | 2 | 2,121,816 | 4,746,539 | 1 | 1 | 41,361,408 | True |
| nats-c | 513 | 2 | 2,241,366 | 5,053,978 | 1 | 1 | 38,162,432 | True |
nproc=40 · uname=`Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux`
### Throughput
| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
|-----|------|------------------|------------|----------|------------|----------|
| `core-1p1s-50k-128` | core pub/sub | 776,331 | 502,502 | 61.34 | 420,976 | 51.39 |
| `core-4p4s-100k-128` | core pub/sub | 1,278,454 | 379,985 | 46.38 | 1,078,910 | 131.70 |
| `core-4p4s-50k-1k` | core pub/sub | 581,464 | 209,545 | 204.63 | 482,645 | 471.33 |
| `core-8p8s-200k-128` | core pub/sub | 2,065,217 | 333,731 | 40.74 | 1,847,242 | 225.49 |
| `js-1p-20k-128-r3` | jetstream r=3 file | — | 7,393 | 0.90 | — | — |
| `js-2p2s-20k-128-r3` | jetstream r=3 file | 15,416 | 7,728 | 0.94 | 7,711 | 0.94 |
| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 14,985 | 14.63 | — | — |
| `js-4p-50k-128-r3` | jetstream r=3 file | — | 17,986 | 2.20 | — | — |
### Round-trip delay
| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|
| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.341ms | 0.470ms | 0.395ms | 0.651ms | 1.377ms | 3.243ms |
| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 132.425ms | 203.717ms | 204.458ms | 241.750ms | 249.468ms | 249.640ms |
| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 196.422ms | 221.264ms | 224.039ms | 233.180ms | 236.798ms | 238.177ms |
| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 211.108ms | 294.925ms | 300.254ms | 314.223ms | 315.551ms | 316.091ms |
| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 280.608ms | 414.972ms | 409.802ms | 513.821ms | 535.394ms | 536.084ms |
### Core NATS
![Core NATS throughput at four loads (NS1 host run)](charts/core-throughput.png)
*Core NATS throughput at four loads (NS1 host run)*
### Payload size (core)
![Core NATS 128 B vs 1 KiB (NS1 host run)](charts/payload-size.png)
*Core NATS 128 B vs 1 KiB (NS1 host run)*
### JetStream r=3 file
![JetStream durable publish rate (NS1 host run)](charts/js-throughput.png)
*JetStream durable publish rate (NS1 host run)*
### Core vs JetStream
![Core vs JetStream publish rate, log scale (NS1 host run)](charts/core-vs-js.png)
*Core vs JetStream publish rate, log scale (NS1 host run)*
### Delay
![Ping vs flood delay percentiles, log scale (NS1 host run)](charts/delay-percentiles.png)
*Ping vs flood delay percentiles, log scale (NS1 host run)*
---
## 4. Study methodology
### 4.1 Question
On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)?
### 4.2 Hypotheses (stated before the run)
1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica.
2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated.
3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting.
4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count.
5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound.
### 4.3 Independent variables (what we changed)
| Factor | Levels |
|--------|--------|
| Transport | Core NATS pub/sub vs JetStream file replicas=3 |
| Publisher count | 1, 2, 4, 8 |
| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 |
| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) |
| Payload | 128 B, 1024 B |
| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) |
### 4.4 Dependent variables (what we recorded)
| Metric | Instrument | Unit |
|--------|------------|------|
| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s |
| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s |
| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) |
| Publisher spread | nats min/avg/max **msgs/s** | not delay |
| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max |
| Host load | `/proc/loadavg` before and after | load average |
| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem |
**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`.
### 4.5 Controls and constants
- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`.
- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`).
- Bench client is LXC **510**, not a nats-* server.
- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide.
- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`).
- No TLS, no nkeys, no account isolation (isolation is `vmbr1`).
- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder.
### 4.6 Procedure
1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise.
2. Snapshot host load, memory, LXC configs, and each nats `varz`.
3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB).
4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file.
5. Copy `latency.mjs` into 510; ping then flood at several batch sizes.
6. Snapshot host/`varz` again.
7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**.
No publish, subscribe, chart, or PDF process runs on the operator laptop for this study.
### 4.7 Instrumentation path
```text
[NS1 host 70.88.205.138]
study-on-ns1.sh (bash + python3)
|
| sudo pct exec 510
v
[LXC 510 verae-px-worker 10.10.10.20]
nats bench / node latency.mjs
|
| NATS client protocol to
v
[LXC 511/512/513 10.10.10.21-23 :4222]
nats-server -js cluster routes :6222
```
The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`.
### 4.8 Threats to validity
| Threat | Effect on numbers |
|--------|-------------------|
| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. |
| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. |
| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. |
| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. |
| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. |
| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. |
| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. |
| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. |
### 4.9 Ethics / safety
Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket.
---
## 5. Suggestions for fine-tuning
These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way.
### 5.1 Treat JetStream as the product limiter
Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out.
### 5.2 Split storage class by stream
| Stream | Suggested store | Why |
|--------|-----------------|-----|
| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad |
| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both |
| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer |
| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry |
| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive |
Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert.
### 5.3 Give JetStream real disks
Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain.
- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`.
- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure.
- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it.
- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.”
### 5.4 Isolate the nats CTs from the rest of NS1
Host load on this box is often already several. Pin:
- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes.
- Memory high enough that file-backed streams stay cache-hot for the working set.
- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync.
Re-run this study after pinning; H1/H2 should move more than ping.
### 5.5 Consumer and mailbox tuning (delay H4)
Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT.
- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM.
- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber.
- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped.
- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT.
### 5.6 Publisher-side batching in middleware
A timestamp job is one small JSON. 16k msgs/s is ample. Still:
- Avoid per-byte publishes; one message per job/event.
- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs).
- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only.
### 5.7 nats-server knobs worth measuring (A/B with this script)
| Knob | Why try it |
|------|------------|
| `max_payload` | Keep default unless archive puts grow |
| `write_deadline` | Slow consumer protection for webhooks |
| `max_pending` | Bound memory on a stuck Zapier hook |
| `max_connections` | Fleet workers + keep + middleware |
| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT |
| `max_outstanding_catchup` | Replica restart after a nats-c blip |
| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT |
Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99.
### 5.8 Network
- Keep NATS off `vmbr0`. No change.
- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load).
- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus.
### 5.9 Security cost (when nkeys/mTLS flip)
`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement.
### 5.10 Operational fine-tuning (lag, not peak msgs/s)
1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`.
2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense.
3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost.
4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated.
### 5.11 What not to tune
- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge.
- Do not treat flood 400 ms as “cluster RTT.” Fix consumers.
- Do not load-test on `ZAPIER_*` streams.
- Do not bind client NATS to `0.0.0.0` on `vmbr0`.
### 5.12 Recommended next experiments (same method, one change each)
1. CPU pin nats-a/b/c → re-run JS 1p + ping.
2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder).
3. Distinct `store_dir` disks per node.
4. nkeys on, same ladder.
5. Three hardware boxes, same `cluster.env` IPs updated.
Each experiment should produce a new `results/<utc>/` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory.
---
## 6. Reproducing this study
On **NS1 only**:
```bash
cd ~/verae-src/verae-nats-cluster
bash scripts/study-on-ns1.sh
```
The script exits if `hostname` is not NS1. Outputs land in `results/<utc>/` including `nats-cluster-bench-ns1.{md,html,pdf}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog.
Raw logs for this run: `results/20260912T051237Z/`.

BIN
reports/ns1-host/nats-cluster-bench-ns1.pdf Normal file
View file

Binary file not shown.

BIN
reports/ns1-maximize/charts/core-throughput.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 56 KiB

BIN
reports/ns1-maximize/charts/core-vs-js.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 40 KiB

BIN
reports/ns1-maximize/charts/delay-percentiles.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 40 KiB

BIN
reports/ns1-maximize/charts/delta-vs-baseline.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 50 KiB

BIN
reports/ns1-maximize/charts/js-throughput.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 59 KiB

BIN
reports/ns1-maximize/charts/payload-size.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 51 KiB

1385
reports/ns1-maximize/nats-cluster-bench-ns1.html Normal file
View file

File diff suppressed because it is too large Load diff

355
reports/ns1-maximize/nats-cluster-bench-ns1.md Normal file
View file

@ -0,0 +1,355 @@
**Progress report (maximized NS1 study)** · run `20260912T053120Z` (UTC)
> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): `maximize-ns1-study.sh` (cores/RAM/`max_mem`/tmpfs), then `study-on-ns1.sh`, `nats bench`, `latency.mjs` (LXC 510), matplotlib, pandoc, weasyprint. Traffic stayed on `vmbr1`. veth/10G was **not** changed. After the ladder, JetStream was put back on ZFS and product streams were re-created; **8 cores / 16 GiB / max_mem 8G stay**.
## Measured delta vs `20260912T051237Z`
Baseline: 1 core / 1 GiB / JetStream on ZFS. This run: 8 cores / 16 GiB / JetStream **tmpfs** (file r=3) plus extra **memory** store rows. veth/10G unchanged.
| Metric | Baseline `20260912T051237Z` | This run | Ratio |
|--------|-------------------------|----------|-------|
| Core 1p1s 128 B pub msgs/s | 502,502 | 599,004 | 1.19× |
| Core 8p8s 128 B aggregate msgs/s | 2,065,217 | 1,998,733 | 0.97× |
| JS file r=3 1p 128 B pub msgs/s | 7,393 | 17,388 | 2.35× |
| JS file r=3 4p 128 B pub msgs/s | 17,986 | 22,853 | 1.27× |
| JS file r=3 4p 1 KiB pub msgs/s | 14,985 | 18,114 | 1.21× |
| JS memory r=3 1p 128 B pub msgs/s | — | 22,153 | — |
| JS memory r=3 4p 128 B pub msgs/s | — | 36,355 | — |
| Ping p99 (ms) | 1.377ms | 0.684ms | 2.01× faster |
![Baseline vs maximized publish rates (log)](charts/delta-vs-baseline.png)
*Baseline vs maximized publish rates (log)*
---
## 1. Executive summary
| Item | This NS1-host run |
|------|-------------------|
| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `marchon` |
| Bench client | LXC 510 `verae-px-worker` |
| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` |
| Client URL | `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` |
| Host load before | `8.77 8.39 8.26 6/3849 198471` |
| Host load after | `9.02 8.82 8.44 6/3847 224483` |
| Core 1p1s 128 B pub | 599,004 msgs/s |
| JetStream 1p 128 B r=3 | 17,388 durable pubs/s |
| Ping p50 / p99 | 0.299ms / 0.684ms |
Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst.
---
## 2. Where it ran (and where it did not)
```text
Operator laptop ──ssh──► NS1.GEORGELAMBERT.ORG 70.88.205.138
study-on-ns1.sh
python3 build-ns1-study-report.py
sudo pct exec 510 ──► nats bench / latency.mjs
│
▼ vmbr1
10.10.10.21-23 :4222
```
- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510).
- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file.
---
## 3. Results (this run)
### Host and brokers
**Before**
| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream |
|------|------|-------------|---------|----------|-----|-------|---------|-----------|
| nats-a | 511 | 3 | 3,013 | 3,037 | 1 | 8 | 14,553,088 | True |
| nats-b | 512 | 3 | 1,415 | 1,424 | 0 | 1 | 13,557,760 | True |
| nats-c | 513 | 0 | 1,361 | 1,395 | 1 | 1 | 14,028,800 | True |
**After**
| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream |
|------|------|-------------|---------|----------|-----|-------|---------|-----------|
| nats-a | 511 | 3 | 644,730 | 1,344,768 | 2 | 8 | 67,104,768 | True |
| nats-b | 512 | 3 | 491,295 | 978,800 | 1 | 1 | 82,542,592 | True |
| nats-c | 513 | 0 | 668,544 | 1,330,940 | 0 | 1 | 37,314,560 | True |
nproc=40 · uname=`Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux`
### Throughput
| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
|-----|------|------------------|------------|----------|------------|----------|
| `core-1p1s-50k-128` | core pub/sub | 810,988 | 599,004 | 73.12 | 456,116 | 55.68 |
| `core-4p4s-100k-128` | core pub/sub | 1,361,921 | 379,346 | 46.31 | 1,150,536 | 140.45 |
| `core-4p4s-50k-1k` | core pub/sub | 695,192 | 175,725 | 171.61 | 584,024 | 570.34 |
| `core-8p8s-200k-128` | core pub/sub | 1,998,733 | 283,259 | 34.58 | 1,780,589 | 217.36 |
| `js-1p-20k-128-r3` | jetstream r=3 file | — | 17,388 | 2.12 | — | — |
| `js-2p2s-20k-128-r3` | jetstream r=3 file | 19,876 | 9,959 | 1.22 | 9,942 | 1.21 |
| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 18,114 | 17.69 | — | — |
| `js-4p-50k-128-r3` | jetstream r=3 file | — | 22,853 | 2.79 | — | — |
| `js-mem-1p-20k-128-r3` | jetstream r=3 file | — | 22,153 | 2.70 | — | — |
| `js-mem-4p-20k-1k-r3` | jetstream r=3 file | — | 28,685 | 28.01 | — | — |
| `js-mem-4p-50k-128-r3` | jetstream r=3 file | — | 36,355 | 4.44 | — | — |
### Round-trip delay
| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|
| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.250ms | 0.319ms | 0.299ms | 0.363ms | 0.684ms | 2.759ms |
| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 94.920ms | 132.026ms | 131.990ms | 156.086ms | 158.860ms | 158.965ms |
| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 113.281ms | 132.425ms | 131.457ms | 140.317ms | 144.683ms | 145.365ms |
| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 153.318ms | 204.329ms | 205.865ms | 234.151ms | 237.346ms | 237.477ms |
| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 233.727ms | 310.924ms | 311.170ms | 373.437ms | 400.764ms | 402.811ms |
### Core NATS
![Core NATS throughput at four loads (NS1 host run)](charts/core-throughput.png)
*Core NATS throughput at four loads (NS1 host run)*
### Payload size (core)
![Core NATS 128 B vs 1 KiB (NS1 host run)](charts/payload-size.png)
*Core NATS 128 B vs 1 KiB (NS1 host run)*
### JetStream r=3 file
![JetStream durable publish rate (NS1 host run)](charts/js-throughput.png)
*JetStream durable publish rate (NS1 host run)*
### Core vs JetStream
![Core vs JetStream publish rate, log scale (NS1 host run)](charts/core-vs-js.png)
*Core vs JetStream publish rate, log scale (NS1 host run)*
### Delay
![Ping vs flood delay percentiles, log scale (NS1 host run)](charts/delay-percentiles.png)
*Ping vs flood delay percentiles, log scale (NS1 host run)*
---
## 4. Study methodology
### 4.1 Question
On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)?
### 4.2 Hypotheses (stated before the run)
1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica.
2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated.
3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting.
4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count.
5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound.
### 4.3 Independent variables (what we changed)
| Factor | Levels |
|--------|--------|
| Transport | Core NATS pub/sub vs JetStream file replicas=3 |
| Publisher count | 1, 2, 4, 8 |
| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 |
| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) |
| Payload | 128 B, 1024 B |
| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) |
### 4.4 Dependent variables (what we recorded)
| Metric | Instrument | Unit |
|--------|------------|------|
| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s |
| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s |
| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) |
| Publisher spread | nats min/avg/max **msgs/s** | not delay |
| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max |
| Host load | `/proc/loadavg` before and after | load average |
| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem |
**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`.
### 4.5 Controls and constants
- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`.
- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`).
- Bench client is LXC **510**, not a nats-* server.
- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide.
- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`).
- No TLS, no nkeys, no account isolation (isolation is `vmbr1`).
- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder.
### 4.6 Procedure
1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise.
2. Snapshot host load, memory, LXC configs, and each nats `varz`.
3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB).
4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file.
5. Copy `latency.mjs` into 510; ping then flood at several batch sizes.
6. Snapshot host/`varz` again.
7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**.
No publish, subscribe, chart, or PDF process runs on the operator laptop for this study.
### 4.7 Instrumentation path
```text
[NS1 host 70.88.205.138]
study-on-ns1.sh (bash + python3)
|
| sudo pct exec 510
v
[LXC 510 verae-px-worker 10.10.10.20]
nats bench / node latency.mjs
|
| NATS client protocol to
v
[LXC 511/512/513 10.10.10.21-23 :4222]
nats-server -js cluster routes :6222
```
The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`.
### 4.8 Threats to validity
| Threat | Effect on numbers |
|--------|-------------------|
| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. |
| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. |
| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. |
| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. |
| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. |
| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. |
| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. |
| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. |
### 4.9 Ethics / safety
Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket.
---
## 5. Suggestions for fine-tuning
These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way.
### 5.1 Treat JetStream as the product limiter
Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out.
### 5.2 Split storage class by stream
| Stream | Suggested store | Why |
|--------|-----------------|-----|
| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad |
| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both |
| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer |
| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry |
| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive |
Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert.
### 5.3 Give JetStream real disks
Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain.
- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`.
- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure.
- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it.
- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.”
### 5.4 Isolate the nats CTs from the rest of NS1
Host load on this box is often already several. Pin:
- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes.
- Memory high enough that file-backed streams stay cache-hot for the working set.
- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync.
Re-run this study after pinning; H1/H2 should move more than ping.
### 5.5 Consumer and mailbox tuning (delay H4)
Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT.
- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM.
- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber.
- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped.
- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT.
### 5.6 Publisher-side batching in middleware
A timestamp job is one small JSON. 16k msgs/s is ample. Still:
- Avoid per-byte publishes; one message per job/event.
- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs).
- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only.
### 5.7 nats-server knobs worth measuring (A/B with this script)
| Knob | Why try it |
|------|------------|
| `max_payload` | Keep default unless archive puts grow |
| `write_deadline` | Slow consumer protection for webhooks |
| `max_pending` | Bound memory on a stuck Zapier hook |
| `max_connections` | Fleet workers + keep + middleware |
| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT |
| `max_outstanding_catchup` | Replica restart after a nats-c blip |
| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT |
Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99.
### 5.8 Network
- Keep NATS off `vmbr0`. No change.
- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load).
- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus.
### 5.9 Security cost (when nkeys/mTLS flip)
`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement.
### 5.10 Operational fine-tuning (lag, not peak msgs/s)
1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`.
2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense.
3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost.
4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated.
### 5.11 What not to tune
- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge.
- Do not treat flood 400 ms as “cluster RTT.” Fix consumers.
- Do not load-test on `ZAPIER_*` streams.
- Do not bind client NATS to `0.0.0.0` on `vmbr0`.
### 5.12 Recommended next experiments (same method, one change each)
1. CPU pin nats-a/b/c → re-run JS 1p + ping.
2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder).
3. Distinct `store_dir` disks per node.
4. nkeys on, same ladder.
5. Three hardware boxes, same `cluster.env` IPs updated.
Each experiment should produce a new `results/<utc>/` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory.
---
## 6. Reproducing this study
On **NS1 only**:
```bash
cd ~/verae-src/verae-nats-cluster
bash scripts/study-on-ns1.sh
```
The script exits if `hostname` is not NS1. Outputs land in `results/<utc>/` including `nats-cluster-bench-ns1.{md,html,pdf}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog.
Raw logs for this run: `results/20260912T053120Z/`.

BIN
reports/ns1-maximize/nats-cluster-bench-ns1.pdf Normal file
View file

Binary file not shown.

853
reports/optimal-config/REPORT.html Normal file
View file

@ -0,0 +1,853 @@
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
<head>
<meta charset="utf-8" />
<meta name="generator" content="pandoc" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
<title>NATS optimal configuration study</title>
<style>
html {
color: #1a1a1a;
background-color: #fdfdfd;
}
body {
margin: 0 auto;
max-width: 36em;
padding-left: 50px;
padding-right: 50px;
padding-top: 50px;
padding-bottom: 50px;
hyphens: auto;
overflow-wrap: break-word;
text-rendering: optimizeLegibility;
font-kerning: normal;
}
@media (max-width: 600px) {
body {
font-size: 0.9em;
padding: 12px;
}
h1 {
font-size: 1.8em;
}
}
@media print {
html {
background-color: white;
}
body {
background-color: transparent;
color: black;
font-size: 12pt;
}
p, h2, h3 {
orphans: 3;
widows: 3;
}
h2, h3, h4 {
page-break-after: avoid;
}
}
p {
margin: 1em 0;
}
a {
color: #1a1a1a;
}
a:visited {
color: #1a1a1a;
}
img {
max-width: 100%;
}
svg {
height: auto;
max-width: 100%;
}
h1, h2, h3, h4, h5, h6 {
margin-top: 1.4em;
}
h5, h6 {
font-size: 1em;
font-style: italic;
}
h6 {
font-weight: normal;
}
ol, ul {
padding-left: 1.7em;
margin-top: 1em;
}
li > ol, li > ul {
margin-top: 0;
}
blockquote {
margin: 1em 0 1em 1.7em;
padding-left: 1em;
border-left: 2px solid #e6e6e6;
color: #606060;
}
code {
font-family: Menlo, Monaco, Consolas, 'Lucida Console', monospace;
font-size: 85%;
margin: 0;
hyphens: manual;
}
pre {
margin: 1em 0;
overflow: auto;
}
pre code {
padding: 0;
overflow: visible;
overflow-wrap: normal;
}
.sourceCode {
background-color: transparent;
overflow: visible;
}
hr {
background-color: #1a1a1a;
border: none;
height: 1px;
margin: 1em 0;
}
table {
margin: 1em 0;
border-collapse: collapse;
width: 100%;
overflow-x: auto;
display: block;
font-variant-numeric: lining-nums tabular-nums;
}
table caption {
margin-bottom: 0.75em;
}
tbody {
margin-top: 0.5em;
border-top: 1px solid #1a1a1a;
border-bottom: 1px solid #1a1a1a;
}
th {
border-top: 1px solid #1a1a1a;
padding: 0.25em 0.5em 0.25em 0.5em;
}
td {
padding: 0.125em 0.5em 0.25em 0.5em;
}
header {
margin-bottom: 4em;
text-align: center;
}
#TOC li {
list-style: none;
}
#TOC ul {
padding-left: 1.3em;
}
#TOC > ul {
padding-left: 0;
}
#TOC a:not(:hover) {
text-decoration: none;
}
code{white-space: pre-wrap;}
span.smallcaps{font-variant: small-caps;}
div.columns{display: flex; gap: min(4vw, 1.5em);}
div.column{flex: auto; overflow-x: auto;}
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
/* The extra [class] is a hack that increases specificity enough to
override a similar rule in reveal.js */
ul.task-list[class]{list-style: none;}
ul.task-list li input[type="checkbox"] {
font-size: inherit;
width: 0.8em;
margin: 0 0.8em 0.2em -1.6em;
vertical-align: middle;
}
.display.math{display: block; text-align: center; margin: 0.5rem auto;}
</style>
<style>/* Colored print + screen stylesheet for zapier.georgelambert.org */
:root {
--ink: #171a26;
--muted: #5b6178;
--line: #d9dce8;
--bg: #f4f5fb;
--paper: #ffffff;
--accent: #4f46e5;
--accent-deep: #312e81;
--accent-soft: #eef0fe;
--ok: #047857;
--warn: #8a5a00;
--code-bg: #1b1f33;
--code-fg: #e8ecff;
}
html { background: var(--bg); }
body {
margin: 0 auto;
padding: 1.5rem 1.25rem 3rem;
max-width: 48rem;
font: 15px/1.55 -apple-system, "Segoe UI", Georgia, serif;
color: var(--ink);
background: var(--paper);
}
.doc-banner {
background: linear-gradient(160deg, #312e81 0%, #4f46e5 60%, #7c74f0 100%);
color: #eef0fe;
margin: -1.5rem -1.25rem 1.5rem;
padding: 1.1rem 1.25rem 1rem;
}
.doc-banner a { color: #fff; }
.doc-banner .kicker {
letter-spacing: 0.12em;
text-transform: uppercase;
font: 700 10px system-ui, sans-serif;
opacity: 0.8;
}
.doc-banner h1 { margin: 0.25rem 0 0; font-size: 1.45rem; color: #fff; }
h1, h2, h3, h4 { color: var(--accent-deep); page-break-after: avoid; }
h1 { font-size: 1.7rem; }
h2 {
font-size: 1.2rem;
border-bottom: 2px solid var(--accent);
padding-bottom: 0.2rem;
margin-top: 1.6rem;
}
h3 { font-size: 1.05rem; color: var(--accent); }
a { color: var(--accent); }
p, li { orphans: 3; widows: 3; }
code {
font-family: ui-monospace, Menlo, Consolas, monospace;
font-size: 0.86em;
background: var(--accent-soft);
color: var(--accent-deep);
padding: 0.08em 0.28em;
border-radius: 4px;
}
pre, div.sourceCode, div.sourceCode pre {
background: var(--code-bg) !important;
color: var(--code-fg) !important;
padding: 0.85rem 1rem;
border-radius: 10px;
overflow: auto;
font-size: 0.78rem;
line-height: 1.4;
page-break-inside: avoid;
}
pre code { background: transparent; color: inherit; padding: 0; }
#title-block-header, header#title-block-header, h1.title { display: none; }
.doc-banner + h1 { display: none; }
table {
border-collapse: collapse;
width: 100%;
margin: 0.8rem 0 1.2rem;
font-size: 0.9rem;
page-break-inside: avoid;
}
th, td { border: 1px solid var(--line); padding: 0.38rem 0.55rem; text-align: left; vertical-align: top; }
th {
background: var(--accent);
color: #fff;
font: 650 12px system-ui, sans-serif;
}
tr:nth-child(even) td { background: var(--accent-soft); }
blockquote {
margin: 1rem 0;
padding: 0.4rem 0.9rem;
border-left: 4px solid var(--accent);
background: var(--accent-soft);
color: var(--accent-deep);
}
img { max-width: 100%; height: auto; border-radius: 8px; page-break-inside: avoid; }
hr { border: 0; border-top: 1px solid var(--line); }
ul, ol { padding-left: 1.25rem; }
nav.site { font: 13px system-ui, sans-serif; margin-bottom: 0.4rem; }
.source-path { font: 11px ui-monospace, Menlo, monospace; color: var(--muted); }
@page {
size: letter;
margin: 0.65in 0.7in 0.8in 0.7in;
@top-left {
content: "Verae Time × Zapier";
font: 700 8pt system-ui, sans-serif;
color: #4f46e5;
}
@top-right {
content: "zapier.georgelambert.org";
font: 8pt system-ui, sans-serif;
color: #6b7186;
}
@bottom-center {
content: counter(page) " / " counter(pages);
font: 8pt system-ui, sans-serif;
color: #6b7186;
}
}
@media print {
html, body { background: #fff; max-width: none; padding: 0; }
.doc-banner { margin: 0 0 1rem; border-radius: 8px; -webkit-print-color-adjust: exact; print-color-adjust: exact; }
a { text-decoration: none; }
th, tr:nth-child(even) td, pre, blockquote, code { -webkit-print-color-adjust: exact; print-color-adjust: exact; }
}
</style>
</head>
<body>
<div class="doc-banner"><nav class="site"><a href="/">zapier.georgelambert.org</a></nav><div class="kicker">Verae Time × Zapier · progress report</div><h1>NATS optimal configuration study</h1><div class="source-path">packages/zapier-decisions/reports/optimal-config/REPORT.md</div></div>
<header id="title-block-header">
<h1 class="title">NATS optimal configuration study</h1>
</header>
<p><strong>Progress report — optimal configuration study</strong> ·
<code>20260912T055851Z</code> (UTC) · all code on
<strong>NS1.GEORGELAMBERT.ORG</strong> (<code>70.88.205.138</code>)</p>
<p>This document folds every ladder we have run (1-core ZFS,
NS1-orchestrated, tmpfs maximize, and this exhaustive 8c/16G
<strong>ZFS</strong> factorial) plus UDP / MQTT / reconnect probes. It
recommends a lab config and a <strong>three-box HP DL360 Gen10</strong>
projection. veth/10G was not changed.</p>
<hr />
<h2 id="verdict-read-this-first">1. Verdict (read this first)</h2>
<p><strong>Keep NATS + JetStream.</strong> Do not replace the fabric
with MQTT, UDP, or a custom persistent-socket protocol for Verae
jobs/events/archive. Those are either slower, less durable, or already
what NATS is.</p>
<p><strong>Lab (NS1, one host, three LXC) — optimal now</strong></p>
<table>
<colgroup>
<col style="width: 25%" />
<col style="width: 28%" />
<col style="width: 31%" />
<col style="width: 15%" />
</colgroup>
<thead>
<tr class="header">
<th>Stream</th>
<th>Storage</th>
<th>Replicas</th>
<th>Why</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>ZAPIER_JOBS</code>, <code>ZAPIER_WEBHOOKS</code>,
<code>VERAE_ARCHIVE</code></td>
<td><strong>file</strong> (ZFS)</td>
<td><strong>3</strong></td>
<td>Survive a nats LXC death; archive must persist</td>
</tr>
<tr class="even">
<td><code>ZAPIER_EVENTS</code></td>
<td><strong>memory</strong></td>
<td><strong>3</strong></td>
<td>Waiters are latency-sensitive; events rebuild from job status</td>
</tr>
<tr class="odd">
<td><code>ZAPIER_USAGE</code></td>
<td>file</td>
<td>3</td>
<td>Telemetry, limits + max-age</td>
</tr>
</tbody>
</table>
<p>Keep <strong>8 cores / 16 GiB / <code>max_mem: 8G</code></strong> on
510–513 (already live). Do <strong>not</strong> leave JetStream on
tmpfs. Do <strong>not</strong> drop product streams to r=1. Reuse
<strong>one NATS connection per process</strong> (already true in
middleware); never connect-per-message.</p>
<p><strong>Metal (3× DL360 Gen10) — optimal later</strong></p>
<p>Same stream table. File store on <strong>local NVMe/M.2</strong>, not
a shared SAN. Cluster + client on <strong>10GbE</strong> (or 25GbE if
you already have it). Dual Gold Xeon is surplus CPU for this workload;
8–16 cores dedicated to <code>nats-server</code> is enough. Expected JS
file r=3: <strong>~40–80k</strong> 128 B pubs/s (about
<strong>3–6×</strong> this lab’s 8c ZFS 1p, <strong>2–4×</strong> tmpfs
1p) — bounded by <strong>10GbE replica RTT</strong>, not by Xeon clocks.
Core NATS will sit in the <strong>1–3M msgs/s</strong> band until the
NIC saturates (~9 Gbit/s ≈ 8–9M × 128 B theoretical; CPU and client will
hit first).</p>
<hr />
<h2 id="what-we-actually-ran-this-exhaustive-pass">2. What we actually
ran (this exhaustive pass)</h2>
<p>Live cluster during this run: LXC 510–513 <strong>8 cores / 16
GiB</strong>, JetStream <strong>on ZFS</strong> (tmpfs from the maximize
study was already unmounted). Extra factorial: file/memory × replicas
1/3, 4 KiB file r=3, reconnect-per-message ping, UDP echo 510→511, MQTT
QoS0 against nats-a <code>:1883</code>. Product streams were not the
bench target.</p>
<h3 id="cross-study-history">2.1 Cross-study history</h3>
<table style="width:100%;">
<colgroup>
<col style="width: 16%" />
<col style="width: 16%" />
<col style="width: 16%" />
<col style="width: 16%" />
<col style="width: 16%" />
<col style="width: 16%" />
</colgroup>
<thead>
<tr class="header">
<th>Study</th>
<th>Env</th>
<th>Core 1p pub</th>
<th>JS file r=3 1p</th>
<th>JS mem r=3 4p</th>
<th>Ping p99</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>20260912T051237Z</code></td>
<td>1c/1G ZFS (NS1 orch.)</td>
<td>502,502</td>
<td>7,393</td>
<td>—</td>
<td>1.377ms</td>
</tr>
<tr class="even">
<td><code>20260912T053120Z</code></td>
<td>8c/16G tmpfs + mem extra</td>
<td>599,004</td>
<td>17,388</td>
<td>36,355</td>
<td>0.684ms</td>
</tr>
<tr class="odd">
<td><code>20260912T055851Z</code></td>
<td>8c/16G ZFS exhaustive <code>20260912T055851Z</code></td>
<td>662,227</td>
<td>14,330</td>
<td>37,736</td>
<td>1.140ms</td>
</tr>
</tbody>
</table>
<figure>
<img src="charts-optimal/history-js1p.png"
alt="JS 1p file r=3 history" />
<figcaption aria-hidden="true">JS 1p file r=3 history</figcaption>
</figure>
<h3 id="this-run-jetstream-factorial">2.2 This run — JetStream
factorial</h3>
<table>
<thead>
<tr class="header">
<th>Run</th>
<th>What</th>
<th>Pub msgs/s</th>
<th>Pub MB/s</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><code>js-file-1p-20k-128-r1</code></td>
<td>file r=1 1p 128 B</td>
<td>18,888</td>
<td>2.31</td>
</tr>
<tr class="even">
<td><code>js-file-4p-50k-128-r1</code></td>
<td>file r=1 4p 128 B</td>
<td>24,560</td>
<td>3.00</td>
</tr>
<tr class="odd">
<td><code>js-1p-20k-128-r3</code></td>
<td>file r=3 1p 128 B</td>
<td>14,330</td>
<td>1.75</td>
</tr>
<tr class="even">
<td><code>js-4p-50k-128-r3</code></td>
<td>file r=3 4p 128 B</td>
<td>19,232</td>
<td>2.35</td>
</tr>
<tr class="odd">
<td><code>js-4p-20k-1k-r3</code></td>
<td>file r=3 4p 1 KiB</td>
<td>15,197</td>
<td>14.84</td>
</tr>
<tr class="even">
<td><code>js-file-1p-20k-4k-r3</code></td>
<td>file r=3 1p 4 KiB</td>
<td>8,673</td>
<td>33.88</td>
</tr>
<tr class="odd">
<td><code>js-mem-1p-20k-128-r1</code></td>
<td>memory r=1 1p 128 B</td>
<td>29,972</td>
<td>3.66</td>
</tr>
<tr class="even">
<td><code>js-mem-4p-50k-128-r1</code></td>
<td>memory r=1 4p 128 B</td>
<td>64,923</td>
<td>7.93</td>
</tr>
<tr class="odd">
<td><code>js-mem-1p-20k-128-r3</code></td>
<td>memory r=3 1p 128 B</td>
<td>20,188</td>
<td>2.46</td>
</tr>
<tr class="even">
<td><code>js-mem-4p-50k-128-r3</code></td>
<td>memory r=3 4p 128 B</td>
<td>37,736</td>
<td>4.61</td>
</tr>
<tr class="odd">
<td><code>js-mem-4p-20k-1k-r3</code></td>
<td>memory r=3 4p 1 KiB</td>
<td>33,916</td>
<td>33.12</td>
</tr>
</tbody>
</table>
<p>Replica <strong>1 vs 3</strong> on this stand (file 1p 128 B): r=1 is
18,888 vs r=3 14,330 (1.32× if r=3 is the slower one). Memory r=1 1p
29,972 vs memory r=3 20,188.</p>
<figure>
<img src="charts-optimal/replicas.png" alt="Replica cost" />
<figcaption aria-hidden="true">Replica cost</figcaption>
</figure>
<h3 id="delay-reconnect-tax-udp-mqtt">2.3 Delay, reconnect tax, UDP,
MQTT</h3>
<table>
<colgroup>
<col style="width: 29%" />
<col style="width: 33%" />
<col style="width: 37%" />
</colgroup>
<thead>
<tr class="header">
<th>Probe</th>
<th>Result</th>
<th>Meaning</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>NATS ping (persistent sockets) p50 / p99</td>
<td>0.456ms / 1.140ms</td>
<td>Quiet hop with a long-lived TCP conn</td>
</tr>
<tr class="even">
<td>NATS <strong>reconnect-per-message</strong> p50 / p99</td>
<td>0.503ms / 1.750ms</td>
<td>TCP+NATS handshake on every pub — this is the tax to avoid</td>
</tr>
<tr class="odd">
<td>UDP echo 510→511 p99</td>
<td>0.363ms</td>
<td>Raw datagram ceiling on the same veth (no NATS)</td>
</tr>
<tr class="even">
<td>MQTT QoS0 5k×128 B</td>
<td>44862 pubs/s</td>
<td>nats-server MQTT gateway on <code>:1883</code></td>
</tr>
</tbody>
</table>
<p>Core 1p1s 128 B this run: 662,227 pub msgs/s. Flood delay is still
backlog/consume_rate, not RTT.</p>
<hr />
<h2 id="alternative-transports-why-we-are-not-switching-the-fabric">3.
Alternative transports (why we are not switching the fabric)</h2>
<p>NATS already <strong>is</strong> persistent TCP sockets with a tiny
binary protocol, automatic reconnect, and optional JetStream durability.
“Reduce connection overhead” is a <strong>client</strong> discipline:
hold the connection. The reconnect probe exists to prove that opening a
socket per job would dominate ping RTT.</p>
<table>
<colgroup>
<col style="width: 7%" />
<col style="width: 44%" />
<col style="width: 32%" />
<col style="width: 15%" />
</colgroup>
<thead>
<tr class="header">
<th>Idea</th>
<th>Fit for Verae jobs/events/archive</th>
<th>Throughput vs NATS core</th>
<th>Durability</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><strong>NATS core pub/sub</strong></td>
<td>Fan-out, request-reply (<code>verae.billing.*</code>)</td>
<td>Highest we measured (~0.5–2M msgs/s)</td>
<td>None</td>
</tr>
<tr class="even">
<td><strong>NATS JetStream file r=3</strong></td>
<td>Jobs, webhooks, archive</td>
<td>~8–23k on this lab; see metal projection</td>
<td>Disk + 1-node loss</td>
</tr>
<tr class="odd">
<td><strong>NATS JetStream memory r=3</strong></td>
<td>Events mailbox</td>
<td>~22–36k on this lab</td>
<td>RAM + 1-node loss; <strong>empty on full restart</strong></td>
</tr>
<tr class="even">
<td><strong>MQTT</strong> (NATS gateway or Mosquitto)</td>
<td>IoT endpoints that already speak MQTT</td>
<td>This probe: 44862 pubs/s QoS0 — typically <strong>well
below</strong> NATS core; QoS1 ≈ JetStream-ish with more chatter</td>
<td>QoS1/2 session state; not our WORM model</td>
</tr>
<tr class="odd">
<td><strong>UDP</strong></td>
<td>Telemetry that may drop</td>
<td>RTT 0.363ms p99 — fastest hop, <strong>no</strong> reliability, no
cluster, no auth</td>
<td>None</td>
</tr>
<tr class="even">
<td><strong>Custom persistent sockets / HTTP long-poll</strong></td>
<td>Worse NATS</td>
<td>You would re-implement reconnect, flow control, and fan-out</td>
<td>DIY</td>
</tr>
<tr class="odd">
<td><strong>WebSocket</strong></td>
<td>Browsers only</td>
<td>Extra framing; NATS already has WS for UIs, not for middleware</td>
<td>Same as core/JS behind it</td>
</tr>
<tr class="even">
<td><strong>QUIC / WebTransport</strong></td>
<td>Lossy WAN / browsers</td>
<td>NATS QUIC is not the lab path; 10GbE LAN does not need it</td>
<td>Same</td>
</tr>
<tr class="odd">
<td><strong>Kafka / Redis streams</strong></td>
<td>Heavy log replay</td>
<td>Higher ops cost; not on <code>vmbr1</code> today</td>
<td>Yes, heavier</td>
</tr>
</tbody>
</table>
<p><strong>MQTT:</strong> NATS documents MQTT as an <em>enabling</em>
gateway for existing IoT, and prefers NATS end-to-end for greenfield.
Zapier cloud never talks NATS or MQTT; it talks HTTPS. Putting MQTT in
the middle of timestamp jobs adds protocol translation and QoS timers
without helping <code>jobId → events</code>. Use MQTT only if a device
already cannot speak NATS.</p>
<p><strong>UDP:</strong> Fine as a <em>measurement</em> of veth RTT.
Unusable as the job fabric (no ack, no replica, no flow control). NATS
ping is already within a small multiple of UDP on this bridge.</p>
<p><strong>Persistence sockets:</strong> Middleware and keep already
keep <code>NATS_URL</code> connections open. Optimal: one connection (or
a small pool) per process, <code>max_reconnect</code>, jitter, no
<code>connect()</code> in the per-job path. The reconnect ladder is the
anti-pattern.</p>
<hr />
<h2 id="optimal-configurations">4. Optimal configurations</h2>
<h3 id="ns1-lab-now">4.1 NS1 lab (now)</h3>
<ol type="1">
<li><strong>Leave 8 cores / 16 GiB</strong> on nats-a/b/c and the
worker. Host has 40 cores / 377 GiB; this is cheap.</li>
<li><strong><code>max_mem: 8G</code></strong> stays. Required for memory
streams.</li>
<li><strong>File r=3 on ZFS</strong> for jobs/webhooks/archive. tmpfs
doubled JS 1p (7.4k→17k) but <strong>loses the stream on reboot</strong>
— unacceptable for archive.</li>
<li><strong>Memory r=3 for <code>ZAPIER_EVENTS</code></strong> if we
accept “all three nats CTs reboot ⇒ in-flight waiters fall back to HTTP
poll.” That matches the designed wait path
(<code>GET /api/status/{jobId}</code>).</li>
<li><strong>r=1 only for throwaway benches</strong>, never product
streams. Replica=3 is the point of three guests.</li>
<li><strong>veth on vmbr1, no fake 10G NICs.</strong> Already 10000Mb/s;
JS does not fill it.</li>
<li><strong>Pin cpusets</strong> later if keep/fleet steal; not required
to beat these numbers.</li>
<li>Clients: persistent NATS connections; pull consumers with bounded
<code>max_ack_pending</code> for webhooks.</li>
</ol>
<h3 id="three-hp-dl360-gen10-projection-not-measured">4.2 Three HP DL360
Gen10 (projection — not measured)</h3>
<p>Assumed bill of materials (state it in the buy):</p>
<table>
<colgroup>
<col style="width: 36%" />
<col style="width: 63%" />
</colgroup>
<thead>
<tr class="header">
<th>Piece</th>
<th>Assumption</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Chassis</td>
<td>3× DL360 Gen10 1U</td>
</tr>
<tr class="even">
<td>CPU</td>
<td>Dual 2nd-gen Xeon <strong>Gold</strong> (e.g. 6226R 16c or 6248 20c
— <strong>32–40 cores/box</strong>)</td>
</tr>
<tr class="odd">
<td>Memory</td>
<td>DDR4-2933, <strong>192–384 GiB</strong>/box (6–12×32 GiB); NATS will
not use most of it</td>
</tr>
<tr class="even">
<td>Storage</td>
<td><strong>NVMe M.2 or U.2</strong> for
<code>/var/lib/nats/jetstream</code> (XFS or ext4, <strong>not</strong>
shared ZFS over the network). RAID1 of two NVMe if you want disk HA
<em>inside</em> a box</td>
</tr>
<tr class="odd">
<td>Network</td>
<td><strong>10GbE</strong> (FlexibleLOM or PCIe); dedicated VLAN for
<code>:4222</code>+<code>:6222</code>. Do not share with public
<code>vmbr0</code> traffic</td>
</tr>
<tr class="even">
<td>OS</td>
<td>Debian/Ubuntu bare metal, <code>nats-server</code> systemd, same
<code>nats.conf</code> as lab (bind private IP only)</td>
</tr>
</tbody>
</table>
<p><strong>What changes vs NS1 LXC</strong></p>
<table>
<colgroup>
<col style="width: 15%" />
<col style="width: 20%" />
<col style="width: 18%" />
<col style="width: 45%" />
</colgroup>
<thead>
<tr class="header">
<th>Factor</th>
<th>NS1 today</th>
<th>3× DL360</th>
<th>Effect on JS file r=3</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Failure domain</td>
<td>1 Proxmox host</td>
<td>3 chassis, 3 NVMe, 3 NICs</td>
<td>r=3 <strong>means</strong> something</td>
</tr>
<tr class="even">
<td>Disk</td>
<td>Shared ZFS SSD2</td>
<td>Local NVMe fsync ~50–150 µs</td>
<td>Big win vs ZFS; similar to tmpfs for sequential 128 B</td>
</tr>
<tr class="odd">
<td>Replica path</td>
<td>veth/bridge (~µs–tens of µs)</td>
<td>10GbE RTT typically <strong>50–200 µs</strong></td>
<td><strong>Slower than same-host tmpfs</strong>, faster than a bad
SAN</td>
</tr>
<tr class="even">
<td>CPU</td>
<td>8 of 40 shared</td>
<td>32–40 dedicated Gold cores</td>
<td>Headroom for many clients, not 10× JS</td>
</tr>
<tr class="odd">
<td>NIC</td>
<td>software 10G veth, already ~5 Gbit/s core</td>
<td>real 10GbE ~9 Gbit/s TCP</td>
<td>Core NATS can grow; JS r=3 stays replica-bound</td>
</tr>
</tbody>
</table>
<p><strong>Projected bands</strong> (128 B, 3-node cluster, dedicated
10GbE, local NVMe, 8+ cores pinned to nats-server):</p>
<table>
<colgroup>
<col style="width: 16%" />
<col style="width: 34%" />
<col style="width: 29%" />
<col style="width: 19%" />
</colgroup>
<thead>
<tr class="header">
<th>Workload</th>
<th>NS1 measured (best)</th>
<th>DL360 projection</th>
<th>Confidence</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Core pub/sub 1p</td>
<td>0.5–0.8M</td>
<td><strong>0.8–2M</strong></td>
<td>Medium — NIC + syscall, plenty of CPU</td>
</tr>
<tr class="even">
<td>Core 4p4s 1 KiB</td>
<td>~0.6–0.7M (~0.6 GB/s)</td>
<td><strong>~1M msgs/s / ~1 GB/s</strong> approaching 10GbE</td>
<td>Medium</td>
</tr>
<tr class="odd">
<td>JS file r=1</td>
<td>this run r=1</td>
<td><strong>80–200k</strong> pubs/s</td>
<td>Medium — NVMe + no replica wait</td>
</tr>
<tr class="even">
<td>JS file r=3</td>
<td>7–23k (ZFS/tmpfs)</td>
<td><strong>40–80k</strong> pubs/s</td>
<td>Medium-low — replica RTT dominates; 3 NVMe still help vs shared
ZFS</td>
</tr>
<tr class="odd">
<td>JS memory r=3</td>
<td>22–36k</td>
<td><strong>50–100k</strong></td>
<td>Medium-low — RAM + 10GbE ack</td>
</tr>
<tr class="even">
<td>Ping p99</td>
<td>0.7–1.4 ms</td>
<td><strong>0.2–0.6 ms</strong></td>
<td>Medium — real NIC but no Proxmox tax</td>
</tr>
</tbody>
</table>
<p>These are <strong>not</strong> DL360 measurements. Scale from: (a)
our replica-1 vs replica-3 ratio once this run’s r=1 numbers exist, (b)
tmpfs vs ZFS ratio (2.35× on 1p), (c) Synadia/nats bench async file r=1
~100–400k on NVMe loopback, derated for 10GbE RTT.</p>
<p><strong>Buy notes:</strong> M.2 via Dual uFF / enablement kit; put
JetStream on NVMe <strong>directly</strong>, not behind a RAID
controller write-through unless you measure. 1GbE onboard is a trap —
use 10GbE for <code>:6222</code>. Dual Gold is for isolation (nats vs
worm/tree vs OS), not because JS needs 56 cores.</p>
<hr />
<h2 id="what-we-are-not-doing">5. What we are not doing</h2>
<ul>
<li>MQTT as the Zapier or middleware transport.</li>
<li>UDP for jobs.</li>
<li>Emulated 10G fiber NICs on LXC.</li>
<li>tmpfs as the production store.</li>
<li>r=1 for product streams.</li>
<li>Connect-per-job.</li>
</ul>
<p>Re-run exhaustive: <code>bash scripts/exhaustive-ns1-study.sh</code>
on NS1.</p>
</body>
</html>

160
reports/optimal-config/REPORT.md Normal file
View file

@ -0,0 +1,160 @@
**Progress report — optimal configuration study** · `20260912T055851Z` (UTC) · all code on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`)
This document folds every ladder we have run (1-core ZFS, NS1-orchestrated, tmpfs maximize, and this exhaustive 8c/16G **ZFS** factorial) plus UDP / MQTT / reconnect probes. It recommends a lab config and a **three-box HP DL360 Gen10** projection. veth/10G was not changed.
---
## 1. Verdict (read this first)
**Keep NATS + JetStream.** Do not replace the fabric with MQTT, UDP, or a custom persistent-socket protocol for Verae jobs/events/archive. Those are either slower, less durable, or already what NATS is.
**Lab (NS1, one host, three LXC) — optimal now**
| Stream | Storage | Replicas | Why |
|--------|---------|----------|-----|
| `ZAPIER_JOBS`, `ZAPIER_WEBHOOKS`, `VERAE_ARCHIVE` | **file** (ZFS) | **3** | Survive a nats LXC death; archive must persist |
| `ZAPIER_EVENTS` | **memory** | **3** | Waiters are latency-sensitive; events rebuild from job status |
| `ZAPIER_USAGE` | file | 3 | Telemetry, limits + max-age |
Keep **8 cores / 16 GiB / `max_mem: 8G`** on 510–513 (already live). Do **not** leave JetStream on tmpfs. Do **not** drop product streams to r=1. Reuse **one NATS connection per process** (already true in middleware); never connect-per-message.
**Metal (3× DL360 Gen10) — optimal later**
Same stream table. File store on **local NVMe/M.2**, not a shared SAN. Cluster + client on **10GbE** (or 25GbE if you already have it). Dual Gold Xeon is surplus CPU for this workload; 8–16 cores dedicated to `nats-server` is enough. Expected JS file r=3: **~40–80k** 128 B pubs/s (about **3–6×** this lab’s 8c ZFS 1p, **2–4×** tmpfs 1p) — bounded by **10GbE replica RTT**, not by Xeon clocks. Core NATS will sit in the **1–3M msgs/s** band until the NIC saturates (~9 Gbit/s ≈ 8–9M × 128 B theoretical; CPU and client will hit first).
---
## 2. What we actually ran (this exhaustive pass)
Live cluster during this run: LXC 510–513 **8 cores / 16 GiB**, JetStream **on ZFS** (tmpfs from the maximize study was already unmounted). Extra factorial: file/memory × replicas 1/3, 4 KiB file r=3, reconnect-per-message ping, UDP echo 510→511, MQTT QoS0 against nats-a `:1883`. Product streams were not the bench target.
### 2.1 Cross-study history
| Study | Env | Core 1p pub | JS file r=3 1p | JS mem r=3 4p | Ping p99 |
| --- | --- | --- | --- | --- | --- |
| `20260912T051237Z` | 1c/1G ZFS (NS1 orch.) | 502,502 | 7,393 | — | 1.377ms |
| `20260912T053120Z` | 8c/16G tmpfs + mem extra | 599,004 | 17,388 | 36,355 | 0.684ms |
| `20260912T055851Z` | 8c/16G ZFS exhaustive `20260912T055851Z` | 662,227 | 14,330 | 37,736 | 1.140ms |
![JS 1p file r=3 history](charts-optimal/history-js1p.png)
### 2.2 This run — JetStream factorial
| Run | What | Pub msgs/s | Pub MB/s |
| --- | --- | --- | --- |
| `js-file-1p-20k-128-r1` | file r=1 1p 128 B | 18,888 | 2.31 |
| `js-file-4p-50k-128-r1` | file r=1 4p 128 B | 24,560 | 3.00 |
| `js-1p-20k-128-r3` | file r=3 1p 128 B | 14,330 | 1.75 |
| `js-4p-50k-128-r3` | file r=3 4p 128 B | 19,232 | 2.35 |
| `js-4p-20k-1k-r3` | file r=3 4p 1 KiB | 15,197 | 14.84 |
| `js-file-1p-20k-4k-r3` | file r=3 1p 4 KiB | 8,673 | 33.88 |
| `js-mem-1p-20k-128-r1` | memory r=1 1p 128 B | 29,972 | 3.66 |
| `js-mem-4p-50k-128-r1` | memory r=1 4p 128 B | 64,923 | 7.93 |
| `js-mem-1p-20k-128-r3` | memory r=3 1p 128 B | 20,188 | 2.46 |
| `js-mem-4p-50k-128-r3` | memory r=3 4p 128 B | 37,736 | 4.61 |
| `js-mem-4p-20k-1k-r3` | memory r=3 4p 1 KiB | 33,916 | 33.12 |
Replica **1 vs 3** on this stand (file 1p 128 B): r=1 is 18,888 vs r=3 14,330 (1.32× if r=3 is the slower one). Memory r=1 1p 29,972 vs memory r=3 20,188.
![Replica cost](charts-optimal/replicas.png)
### 2.3 Delay, reconnect tax, UDP, MQTT
| Probe | Result | Meaning |
|-------|--------|---------|
| NATS ping (persistent sockets) p50 / p99 | 0.456ms / 1.140ms | Quiet hop with a long-lived TCP conn |
| NATS **reconnect-per-message** p50 / p99 | 0.503ms / 1.750ms | TCP+NATS handshake on every pub — this is the tax to avoid |
| UDP echo 510→511 p99 | 0.363ms | Raw datagram ceiling on the same veth (no NATS) |
| MQTT QoS0 5k×128 B | 44862 pubs/s | nats-server MQTT gateway on `:1883` |
Core 1p1s 128 B this run: 662,227 pub msgs/s. Flood delay is still backlog/consume_rate, not RTT.
---
## 3. Alternative transports (why we are not switching the fabric)
NATS already **is** persistent TCP sockets with a tiny binary protocol, automatic reconnect, and optional JetStream durability. “Reduce connection overhead” is a **client** discipline: hold the connection. The reconnect probe exists to prove that opening a socket per job would dominate ping RTT.
| Idea | Fit for Verae jobs/events/archive | Throughput vs NATS core | Durability |
|------|-----------------------------------|-------------------------|------------|
| **NATS core pub/sub** | Fan-out, request-reply (`verae.billing.*`) | Highest we measured (~0.5–2M msgs/s) | None |
| **NATS JetStream file r=3** | Jobs, webhooks, archive | ~8–23k on this lab; see metal projection | Disk + 1-node loss |
| **NATS JetStream memory r=3** | Events mailbox | ~22–36k on this lab | RAM + 1-node loss; **empty on full restart** |
| **MQTT** (NATS gateway or Mosquitto) | IoT endpoints that already speak MQTT | This probe: 44862 pubs/s QoS0 — typically **well below** NATS core; QoS1 ≈ JetStream-ish with more chatter | QoS1/2 session state; not our WORM model |
| **UDP** | Telemetry that may drop | RTT 0.363ms p99 — fastest hop, **no** reliability, no cluster, no auth | None |
| **Custom persistent sockets / HTTP long-poll** | Worse NATS | You would re-implement reconnect, flow control, and fan-out | DIY |
| **WebSocket** | Browsers only | Extra framing; NATS already has WS for UIs, not for middleware | Same as core/JS behind it |
| **QUIC / WebTransport** | Lossy WAN / browsers | NATS QUIC is not the lab path; 10GbE LAN does not need it | Same |
| **Kafka / Redis streams** | Heavy log replay | Higher ops cost; not on `vmbr1` today | Yes, heavier |
**MQTT:** NATS documents MQTT as an *enabling* gateway for existing IoT, and prefers NATS end-to-end for greenfield. Zapier cloud never talks NATS or MQTT; it talks HTTPS. Putting MQTT in the middle of timestamp jobs adds protocol translation and QoS timers without helping `jobId → events`. Use MQTT only if a device already cannot speak NATS.
**UDP:** Fine as a *measurement* of veth RTT. Unusable as the job fabric (no ack, no replica, no flow control). NATS ping is already within a small multiple of UDP on this bridge.
**Persistence sockets:** Middleware and keep already keep `NATS_URL` connections open. Optimal: one connection (or a small pool) per process, `max_reconnect`, jitter, no `connect()` in the per-job path. The reconnect ladder is the anti-pattern.
---
## 4. Optimal configurations
### 4.1 NS1 lab (now)
1. **Leave 8 cores / 16 GiB** on nats-a/b/c and the worker. Host has 40 cores / 377 GiB; this is cheap.
2. **`max_mem: 8G`** stays. Required for memory streams.
3. **File r=3 on ZFS** for jobs/webhooks/archive. tmpfs doubled JS 1p (7.4k→17k) but **loses the stream on reboot** — unacceptable for archive.
4. **Memory r=3 for `ZAPIER_EVENTS`** if we accept “all three nats CTs reboot ⇒ in-flight waiters fall back to HTTP poll.” That matches the designed wait path (`GET /api/status/{jobId}`).
5. **r=1 only for throwaway benches**, never product streams. Replica=3 is the point of three guests.
6. **veth on vmbr1, no fake 10G NICs.** Already 10000Mb/s; JS does not fill it.
7. **Pin cpusets** later if keep/fleet steal; not required to beat these numbers.
8. Clients: persistent NATS connections; pull consumers with bounded `max_ack_pending` for webhooks.
### 4.2 Three HP DL360 Gen10 (projection — not measured)
Assumed bill of materials (state it in the buy):
| Piece | Assumption |
|-------|------------|
| Chassis | 3× DL360 Gen10 1U |
| CPU | Dual 2nd-gen Xeon **Gold** (e.g. 6226R 16c or 6248 20c — **32–40 cores/box**) |
| Memory | DDR4-2933, **192–384 GiB**/box (6–12×32 GiB); NATS will not use most of it |
| Storage | **NVMe M.2 or U.2** for `/var/lib/nats/jetstream` (XFS or ext4, **not** shared ZFS over the network). RAID1 of two NVMe if you want disk HA *inside* a box |
| Network | **10GbE** (FlexibleLOM or PCIe); dedicated VLAN for `:4222`+`:6222`. Do not share with public `vmbr0` traffic |
| OS | Debian/Ubuntu bare metal, `nats-server` systemd, same `nats.conf` as lab (bind private IP only) |
**What changes vs NS1 LXC**
| Factor | NS1 today | 3× DL360 | Effect on JS file r=3 |
|--------|-----------|----------|------------------------|
| Failure domain | 1 Proxmox host | 3 chassis, 3 NVMe, 3 NICs | r=3 **means** something |
| Disk | Shared ZFS SSD2 | Local NVMe fsync ~50–150 µs | Big win vs ZFS; similar to tmpfs for sequential 128 B |
| Replica path | veth/bridge (~µs–tens of µs) | 10GbE RTT typically **50–200 µs** | **Slower than same-host tmpfs**, faster than a bad SAN |
| CPU | 8 of 40 shared | 32–40 dedicated Gold cores | Headroom for many clients, not 10× JS |
| NIC | software 10G veth, already ~5 Gbit/s core | real 10GbE ~9 Gbit/s TCP | Core NATS can grow; JS r=3 stays replica-bound |
**Projected bands** (128 B, 3-node cluster, dedicated 10GbE, local NVMe, 8+ cores pinned to nats-server):
| Workload | NS1 measured (best) | DL360 projection | Confidence |
|----------|---------------------|------------------|------------|
| Core pub/sub 1p | 0.5–0.8M | **0.8–2M** | Medium — NIC + syscall, plenty of CPU |
| Core 4p4s 1 KiB | ~0.6–0.7M (~0.6 GB/s) | **~1M msgs/s / ~1 GB/s** approaching 10GbE | Medium |
| JS file r=1 | this run r=1 | **80–200k** pubs/s | Medium — NVMe + no replica wait |
| JS file r=3 | 7–23k (ZFS/tmpfs) | **40–80k** pubs/s | Medium-low — replica RTT dominates; 3 NVMe still help vs shared ZFS |
| JS memory r=3 | 22–36k | **50–100k** | Medium-low — RAM + 10GbE ack |
| Ping p99 | 0.7–1.4 ms | **0.2–0.6 ms** | Medium — real NIC but no Proxmox tax |
These are **not** DL360 measurements. Scale from: (a) our replica-1 vs replica-3 ratio once this run’s r=1 numbers exist, (b) tmpfs vs ZFS ratio (2.35× on 1p), (c) Synadia/nats bench async file r=1 ~100–400k on NVMe loopback, derated for 10GbE RTT.
**Buy notes:** M.2 via Dual uFF / enablement kit; put JetStream on NVMe **directly**, not behind a RAID controller write-through unless you measure. 1GbE onboard is a trap — use 10GbE for `:6222`. Dual Gold is for isolation (nats vs worm/tree vs OS), not because JS needs 56 cores.
---
## 5. What we are not doing
- MQTT as the Zapier or middleware transport.
- UDP for jobs.
- Emulated 10G fiber NICs on LXC.
- tmpfs as the production store.
- r=1 for product streams.
- Connect-per-job.
Re-run exhaustive: `bash scripts/exhaustive-ns1-study.sh` on NS1.

BIN
reports/optimal-config/REPORT.pdf Normal file
View file

Binary file not shown.

BIN
reports/optimal-config/charts-optimal/history-js1p.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 25 KiB

BIN
reports/optimal-config/charts-optimal/replicas.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 27 KiB

247
scripts/build-nats-bench-report.py Normal file
View file

@ -0,0 +1,247 @@
#!/usr/bin/env python3
"""Charts + HTML + PDF for the NATS cluster speed report in zapier-decisions/reports."""
from __future__ import annotations
import subprocess
import sys
from pathlib import Path
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from matplotlib.ticker import FuncFormatter
ROOT = Path(__file__).resolve().parents[3]
PKG = Path(__file__).resolve().parents[1]
REPORT_DIR = PKG / "reports"
CHARTS = REPORT_DIR / "charts"
MD = REPORT_DIR / "nats-cluster-bench.md"
HTML = REPORT_DIR / "nats-cluster-bench.html"
PDF = REPORT_DIR / "nats-cluster-bench.pdf"
CSS = ROOT / "scripts" / "docs-print.css"
LUA = ROOT / "scripts" / "pdf-links.lua"
INDIGO = "#4f46e5"
DEEP = "#312e81"
TEAL = "#047857"
AMBER = "#b45309"
LILAC = "#7c74f0"
INK = "#171a26"
MUTED = "#5b6178"
GRID = "#d9dce8"
# Run 20260912T045131Z — parsed from nats bench logs (Sub stats, not Pub/Sub aggregate).
CORE = {
"labels": ["1p1s\n50k×128 B", "4p4s\n100k×128 B", "8p8s\n200k×128 B", "4p4s\n50k×1 KiB"],
"agg": [1_200_836, 1_521_256, 2_007_937, 630_460],
"pub": [791_094, 316_312, 333_957, 247_747],
"sub": [747_461, 1_299_634, 1_790_736, 510_216],
"pub_mb": [96.57, 38.61, 40.77, 241.94],
"sub_mb": [91.24, 158.65, 218.60, 498.26],
}
JS = {
"labels": ["1p 20k×128 B", "4p 50k×128 B", "4p 20k×1 KiB", "2p2s pull 20k×128 B"],
"pub": [16_155, 16_607, 13_493, 10_965],
"sub": [None, None, None, 10_942],
}
DELAY = {
"labels": ["Ping\n1k×128 B", "Flood 1p\n5k×128 B", "Flood 4p\n5k×1 KiB", "Flood 4p\n10k×128 B", "Flood 8p\n20k×128 B"],
"kind": ["ping", "flood", "flood", "flood", "flood"],
"p50": [0.286, 248.752, 217.579, 266.672, 466.296],
"p90": [0.332, 274.314, 223.268, 299.073, 499.924],
"p99": [0.734, 279.398, 227.798, 304.233, 505.112],
"avg": [0.307, 238.626, 211.706, 263.186, 453.749],
}
def style() -> None:
plt.rcParams.update(
{
"font.family": "sans-serif",
"font.size": 10,
"axes.titlesize": 12,
"axes.titleweight": "semibold",
"axes.labelsize": 10,
"axes.edgecolor": GRID,
"axes.labelcolor": INK,
"text.color": INK,
"xtick.color": MUTED,
"ytick.color": MUTED,
"figure.facecolor": "white",
"axes.facecolor": "white",
"axes.grid": True,
"grid.color": GRID,
"grid.linewidth": 0.8,
"legend.frameon": False,
"savefig.bbox": "tight",
"savefig.dpi": 160,
"savefig.facecolor": "white",
}
)
def k_fmt(x: float, _pos: int | None = None) -> str:
if x >= 1_000_000:
return f" {x / 1_000_000:.2f}M".strip()
if x >= 1000:
return f"{x / 1000:.0f}k"
return f"{x:.0f}"
def save(fig: plt.Figure, name: str) -> None:
CHARTS.mkdir(parents=True, exist_ok=True)
path = CHARTS / name
fig.savefig(path, dpi=160)
plt.close(fig)
print(f"wrote {path}")
def chart_core_msgs() -> None:
fig, ax = plt.subplots(figsize=(9.2, 4.4))
x = range(len(CORE["labels"]))
w = 0.25
ax.bar([i - w for i in x], CORE["agg"], w, label="Aggregate", color=DEEP)
ax.bar(list(x), CORE["pub"], w, label="Publish", color=INDIGO)
ax.bar([i + w for i in x], CORE["sub"], w, label="Subscribe", color=TEAL)
ax.set_xticks(list(x), CORE["labels"])
ax.set_ylabel("messages / second")
ax.set_title("Core NATS throughput (fire-and-forget pub/sub)")
ax.yaxis.set_major_formatter(FuncFormatter(k_fmt))
ax.legend(loc="upper left")
ax.set_axisbelow(True)
save(fig, "core-throughput.png")
def chart_js_msgs() -> None:
fig, ax = plt.subplots(figsize=(9.2, 4.4))
colors = [INDIGO, INDIGO, AMBER, LILAC]
ax.bar(JS["labels"], JS["pub"], color=colors)
ax.set_ylabel("durable publish messages / second")
ax.set_title("JetStream file store, replicas=3 (product-stream settings)")
ax.yaxis.set_major_formatter(FuncFormatter(k_fmt))
ax.set_axisbelow(True)
for i, v in enumerate(JS["pub"]):
ax.text(i, v + 250, f"{v:,}", ha="center", va="bottom", fontsize=9, color=MUTED)
save(fig, "js-throughput.png")
def chart_core_vs_js() -> None:
fig, ax = plt.subplots(figsize=(9.2, 4.4))
labels = ["1 publisher\n128 B", "4 publishers\n128 B", "4 publishers\n1 KiB"]
core_pub = [791_094, 316_312, 247_747]
js_pub = [16_155, 16_607, 13_493]
x = range(len(labels))
w = 0.35
ax.bar([i - w / 2 for i in x], core_pub, w, label="Core NATS (no disk)", color=INDIGO)
ax.bar([i + w / 2 for i in x], js_pub, w, label="JetStream r=3 file", color=AMBER)
ax.set_xticks(list(x), labels)
ax.set_yscale("log")
ax.set_ylabel("publish messages / second (log)")
ax.set_title("Core vs JetStream: two different jobs")
ax.legend(loc="upper right")
ax.set_axisbelow(True)
save(fig, "core-vs-js.png")
def chart_bytes() -> None:
fig, axes = plt.subplots(1, 2, figsize=(9.2, 4.2))
labels = ["128 B\n4p4s", "1 KiB\n4p4s"]
msgs = [1_521_256, 630_460]
mb = [185.70, 615.68]
axes[0].bar(labels, msgs, color=[INDIGO, AMBER])
axes[0].set_title("Aggregate messages / second")
axes[0].yaxis.set_major_formatter(FuncFormatter(k_fmt))
axes[0].set_axisbelow(True)
axes[1].bar(labels, mb, color=[INDIGO, AMBER])
axes[1].set_title("Aggregate MB / second")
axes[1].set_ylabel("MB/s")
axes[1].set_axisbelow(True)
for ax, vals, fmt in (
(axes[0], msgs, lambda v: f"{v/1e6:.2f}M"),
(axes[1], mb, lambda v: f"{v:.0f}"),
):
for i, v in enumerate(vals):
ax.text(i, v * 1.02, fmt(v), ha="center", va="bottom", fontsize=9, color=MUTED)
fig.suptitle("Core NATS: bigger payloads move more bytes, fewer messages", fontsize=12, fontweight="semibold")
fig.tight_layout()
save(fig, "payload-size.png")
def chart_delay() -> None:
fig, ax = plt.subplots(figsize=(9.2, 4.6))
x = range(len(DELAY["labels"]))
w = 0.25
ax.bar([i - w for i in x], DELAY["p50"], w, label="p50", color=TEAL)
ax.bar(list(x), DELAY["p90"], w, label="p90", color=INDIGO)
ax.bar([i + w for i in x], DELAY["p99"], w, label="p99", color=AMBER)
ax.set_xticks(list(x), DELAY["labels"])
ax.set_yscale("log")
ax.set_ylabel("milliseconds (log)")
ax.set_title("Round-trip delay: sequential ping vs burst flood")
ax.axhline(1.0, color=GRID, linestyle="--", linewidth=1)
ax.legend(loc="upper left")
ax.set_axisbelow(True)
save(fig, "delay-percentiles.png")
def render() -> None:
if not MD.exists():
raise SystemExit(f"missing {MD}")
header = HTML.with_suffix(".hdr.html")
banner = HTML.with_suffix(".ban.html")
css_text = CSS.read_text(encoding="utf-8")
header.write_text(f"<style>{css_text}</style>\n", encoding="utf-8")
banner.write_text(
'<div class="doc-banner">'
'<nav class="site"><a href="/">zapier.georgelambert.org</a>'
' · <a href="/index-md.html">Markdown indexes</a></nav>'
'<div class="kicker">Verae Time × Zapier · progress report</div>'
"<h1>NATS cluster message speed</h1>"
'<div class="source-path">packages/zapier-decisions/reports/nats-cluster-bench.md</div>'
"</div>\n",
encoding="utf-8",
)
r = subprocess.run(
[
"pandoc",
str(MD),
"-o",
str(HTML),
"--standalone",
f"--resource-path={REPORT_DIR}",
"--syntax-highlighting=breezedark",
"--metadata=title=NATS cluster message speed",
f"--include-in-header={header}",
f"--include-before-body={banner}",
f"--lua-filter={LUA}",
],
capture_output=True,
text=True,
)
header.unlink(missing_ok=True)
banner.unlink(missing_ok=True)
if r.returncode != 0:
sys.stderr.write(r.stderr)
raise SystemExit(f"pandoc failed: {r.returncode}")
w = subprocess.run(["weasyprint", str(HTML), str(PDF)], capture_output=True, text=True)
if w.returncode != 0:
sys.stderr.write(w.stderr)
raise SystemExit(f"weasyprint failed: {w.returncode}")
print(f"wrote {HTML}")
print(f"wrote {PDF}")
def main() -> int:
style()
chart_core_msgs()
chart_js_msgs()
chart_core_vs_js()
chart_bytes()
chart_delay()
render()
return 0
if __name__ == "__main__":
raise SystemExit(main())