diff --git a/.agents/JOB_BUS.md b/.agents/JOB_BUS.md deleted file mode 120000 index e546bba..0000000 --- a/.agents/JOB_BUS.md +++ /dev/null @@ -1 +0,0 @@ -/Users/marchon/job-bus/docs/JOB_BUS_LLM_INSTRUCTIONS.md \ No newline at end of file diff --git a/.jobbus.json b/.jobbus.json deleted file mode 100644 index ab460ca..0000000 --- a/.jobbus.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "enabled": true, - "manifest": "jobs/pieces.yaml", - "farm": "scripts/jobbus-farm.sh", - "instructions": ".agents/JOB_BUS.md" -} diff --git a/AGENTS.md b/AGENTS.md deleted file mode 100644 index 2c41576..0000000 --- a/AGENTS.md +++ /dev/null @@ -1,6 +0,0 @@ -# Agent notes - -## Job-bus multi-build -This repo farms multi-build pieces via NATS job-bus. See [.agents/JOB_BUS.md](.agents/JOB_BUS.md). -Edit `jobs/pieces.yaml`, then run `./scripts/jobbus-farm.sh` (or `make farm`). -CPU pieces → Xeons (137/138); GPU pieces → Mac. Always set `requires` per piece. diff --git a/LOG.md b/LOG.md deleted file mode 100644 index 10ee026..0000000 --- a/LOG.md +++ /dev/null @@ -1,159 +0,0 @@ -# Action log - -## 2026-10-03 — D006 libSQL business SoT - -Accepted **D006**: all billing, customer, pricing, accounting, and related business data live in **libSQL**. JetStream only as needed for multi-host work queues. Core NATS only for Verae timestamp job-complete subjects. Go timestamping DBs unchanged. Single-host `billing.db` on 138 remains the lab stand-in until libSQL cutover. - -## 2026-09-12 — exhaustive optimal-config study - -- 8c/16G ZFS factorial: file/memory × r=1/r=3, 4 KiB, reconnect, UDP echo, MQTT QoS0 on nats-a :1883 (then removed). -- JS memory r=1 4p **65k** pubs/s; file r=3 1p **14.3k**; MQTT QoS0 **45k**; UDP p99 **0.36 ms**. -- Combined report + DL360 Gen10 projection: `reports/optimal-config/REPORT.{md,html,pdf}`. -- Verdict: keep NATS+JetStream; no MQTT/UDP fabric; metal = 3 NVMe + 10GbE, expect JS r=3 ~40–80k. - -## 2026-09-12 — maximized NS1 NATS study - -- `pct set` 510–513 to 8 cores / 16 GiB; `max_mem: 8G`; tmpfs 8G on JetStream for the ladder only. -- Run `20260912T053120Z`. JS file 1p 128 B **7.4k → 17.4k (2.35×)**; JS memory 4p **36.4k**; ping p99 **1.38 → 0.68 ms**. -- tmpfs unmounted after; product streams on ZFS again. Resources and max_mem **left raised**. -- Report: `reports/ns1-maximize/`. - -## 2026-09-12 — NS1-host NATS study (second run) - -- All orchestration, nats bench, matplotlib, pandoc, and weasyprint ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`). Laptop not in the path. -- Script: `verae-nats-cluster/scripts/study-on-ns1.sh` (refuses unless hostname is NS1). -- Run `20260912T051237Z`. Report: `reports/ns1-host/nats-cluster-bench-ns1.{md,html,pdf}`. -- Host load ~8.5. JS 1p 128 B ~7.4k durable pubs/s; ping p99 1.38 ms. Methodology + fine-tuning in that report. - -## 2026-09-12 — NATS cluster speed report - -- Bench from LXC 510 vs nats-a/b/c (`20260912T045131Z`). -- Full write-up with charts in `reports/nats-cluster-bench.{md,html,pdf}`. -- JetStream r=3 file ~16k durable 128 B pubs/s; ping RTT ~0.3 ms; flood queueing 150–500 ms. - -## 2026-09-12 — test env on the 3-node cluster - -- Fleet, keep, middleware, billing, and access planes `NATS_URL` → `10.10.10.21–23:4222`. -- Streams ZAPIER_JOBS/EVENTS/WEBHOOKS/USAGE + VERAE_ARCHIVE replicas=3. -- px-worker keep worm/tree 200 on the cluster. Backup timer enabled. Uptime loop on px-worker. -- MOCK_VERAE still true. Zapier push still waiting on login. nkeys/mTLS not flipped (would require client creds). - -## 2026-09-12 — NATS cluster on three Proxmox LXC - -- New repo **verae-nats-cluster**: LXC 511/512/513 (`nats-a/b/c`, `10.10.10.21–23`) JetStream cluster on `vmbr1`. -- Lab loopback `127.0.0.1:4222` unchanged until cut-over. -- Test: stream replicas=3; same configs later move to distinct hardware. - -## 2026-09-11 — proxmox worker + deploy/monitor/backup - -- NS1 **is** Proxmox (`vmbr0` public, `vmbr1` `10.10.10.0/24`). New repo **verae-proxmox** creates LXC **510** `verae-px-worker` at `10.10.10.20`. -- NATS for guests: socat `10.10.10.1:4222` → `127.0.0.1:4222` (never vmbr0). -- **verae-uptime** off-box HTTPS GET; **verae-backup** tar snapshots; **verae-deploy** tagged npm ci. -- Fleet `overlays/ns1/` checked in; `start.sh` no longer rewrites fleet/machines JSON. -- User systemd + linger for keep and fleet. - -## 2026-09-11 — bootstrap - -- Created this repo to record decisions while executing cleanup items 1–5 (and a second pass on 2–5). -- Assumption: single developer, lab only; breaking changes allowed. - -## Step 1 — account-balance is the prepaid source of truth - -- Persist books to `BOOKS_PATH` (atomic JSON). -- Mutations go through `balance.adjust` / `usage.recorded`; edge cache `balanceCents` is updated from the books reply only. -- Local cache mutate only when `NATS_URL` and `ACCOUNT_BALANCE_URL` are both unset (tests). - -## Step 2 — identity join - -- `zappier-identity` mailbox: `verae.identity.bind` / `verae.identity.lookup`. -- Portal + middleware tenants store `veraeUserId`; JWT never public. - -## Step 3 — access planes are the public doors - -- zappier-edge binds loopback (`127.0.0.1:3000`). -- Public: access-web `:3021`, access-api `:3022`, access-leaf `:3023`, access-zapier `:3024`, access-staff `:3025`. - -## Step 4 — leaf hardening - -- `verae-nats-accounts` holds nats.conf + subject policy. -- Leaf token required; billing subjects denied in policy file and access-leaf. - -## Step 5 — fleet spawn - -- Prefer local machines for HTTP `spawn.command` services. -- Disable lan-134 unless `FLEET_ENABLE_LAN134=1`. -- SSH spawn timeout 8s; failed hosts skipped. - -## 2026-09-11 — catalog live doors - -- Bottom of https://zapier.georgelambert.org/ lists each HTTPS door with description, screenshot, and Open link. -- UI-REVIEW.pdf includes the live-door atlas. Walkthrough §13. - -## 2026-09-11 — Caddy host names live - -- Installed `/etc/caddy/sites/zapier-services.georgelambert.org.caddy` on NS1. HTTPS: portal, api, leaf, zap, staff, iam, login, cs, sales, acct, mw, fleet. -- Not proxied: NATS, keep, workers, loopback edge. - -## 2026-09-11 — host names - -- Mapped every running service to **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`). -- Suggested `*.zapier.georgelambert.org` names for public doors; only `zapier.georgelambert.org` (docs) and `git.georgelambert.org` exist in Caddy today. - -## 2026-09-11 — docs sync, bootstrap, NS1 all-in-one - -- Catalog and ops docs: public portal is access-web `:3021/portal/`; edge is loopback; IAM + keep listed. -- New `verae-bootstrap` repo: clone + npm for `ns1-all-in-one`, `control-plane`, `ns1-archive`, `lan-worker`. -- Fleet starts `staff-iam` and `staff-session`. NS1 all-in-one uses edge `:13000` because host `:3000` is taken; keep still owns archive workers. `better-sqlite3` must be rebuilt on Linux (`npm rebuild`). - -## 2026-09-11 — verae-keep on NS1 - -- New repo `verae-keep`: keep `:3860` restarts crashed units; watch `:3861` restarts keep; `guard.sh` restarts watch. -- Honors admin pause/stop via intent files + `POST /intent`. Fleet writes `data//intent.json` (local and SSH). -- Tested on 70.88.205.138: crash-restart, pause-hold, watch respawn. webhook-deliver brought back. NATS observed only. - -## 2026-09-11 — IAM JSON doors closed - -- CS/sales/accounting/staff JSON list and sales PUT pricing now go through IAM (401 JSON, 302 HTML). -- Fleet service env sets `STAFF_AUTH=1` with `STAFF_IAM_URL`. -- Re-walk: cs credit 200 agent=`cs`; cs export 403; operator Fleet POST 200; admin `/me` `permissions:['*']`. - -## 2026-09-11 — IAM on lab, portal door, hardening - -- Restarted fleet with `STAFF_IAM_URL`. Walk: cs credits (agent=`cs`) and is 403 on accounting export; operator fleet POST 200, cs 403; admin `/me` permissions `*`. -- Portal public door `verae-access-web` `:3021/portal/` (static + `/portal/api` → loopback edge). -- IAM sessions persist in JSON; login rate-limit per IP+username (8/10min); credits stamp IAM username as authz principal. -- lan-134 remains disabled unless `FLEET_ENABLE_LAN134=1`. Receipt PDF branded (indigo header + legal footer). - -## 2026-09-11 — staff IAM - -- New `verae-staff-iam` :3028 — users, roles, permissions, sessions, audit UI. -- Admin login uses IAM when `STAFF_IAM_URL` is set; tabs hide without permission. -- CS/sales/accounting/access-staff/fleet mutating APIs check `cs.credit`, `sales.quote`, `accounting.export`, `fleet.operate`, etc. -- Seed: admin/cs/sales/accounting/operator. Shared staff key is fallback only. - -## 2026-09-11 — last three UI leftovers - -- Swagger `/docs` stays stock; banner names it OpenAPI explorer. -- Shared template `verae-staff-ui` for CS + access-staff. -- `STAFF_COOKIE_DOMAIN` + nginx example for multi-host. - -## 2026-09-11 — names, staff session, exclusive jobs.events - -- Account-balance stores display names (`customer.put` + lookup by name). Edge writes names on customer create/edit; CS/sales/accounting/staff join from edge if the ledger has no name. -- New repo `verae-staff-session` (:3027). Department HTML redirects when `STAFF_AUTH=1`. -- `JOBS_EVENTS_EXCLUSIVE=1` makes `verae-jobs-events` the durable consumer; middleware webhook router backs off. -- Catalog index is cards. Disabled fleet machines are grey, not degraded yellow. - -## 2026-09-11 — UI design-system pass - -- Restyled CS/sales/accounting/access-staff to portal indigo; dollars + names. -- Fleet ⋯ menus, role chips, Docs cards, 0.0.0.0 copy, simulator indigo (orange = faults). -- Portal masked API key; admin customer drawer; empty invoice illustrations. -- Catalog system-ui. New independent repo **UI-Docs** (walkthrough, screenshots, UI-REVIEW.pdf). - -## Second pass (2–5) - -- Identity used from portal when `IDENTITY_URL` set. -- Leaf loads allow-list from nats-accounts `policy.json`. -- Staff plane is a first-class mailbox (`verae-access-staff`). -- jobs.events mailbox process `verae-jobs-events` (passthrough + count). diff --git a/NATS.md b/NATS.md deleted file mode 100644 index 782a476..0000000 --- a/NATS.md +++ /dev/null @@ -1,3 +0,0 @@ -# NATS — zapier-decisions - -None. Documentation repo. diff --git a/README.md b/README.md deleted file mode 100644 index 0ed3ff4..0000000 --- a/README.md +++ /dev/null @@ -1,42 +0,0 @@ -# zapier-decisions - -[![standard-readme compliant](https://img.shields.io/badge/readme%20style-standard-brightgreen.svg?style=flat-square)](https://github.com/RichardLitt/standard-readme) - -Lab log of architecture decisions, actions taken, and open todos while cleaning the Verae × Zapier stack. - -**Forgejo:** https://git.georgelambert.org/marchon/zapier-decisions - -| File | What | -|------|------| -| [STATUS.md](STATUS.md) | **Current status** (2026-09-12) | -| [reports/nats-cluster-bench.md](reports/nats-cluster-bench.md) | NATS cluster speed report (charts, HTML, PDF) | -| [reports/ns1-host/nats-cluster-bench-ns1.md](reports/ns1-host/nats-cluster-bench-ns1.md) | Second study: all code on NS1 (methodology + tuning) | -| [reports/ns1-maximize/nats-cluster-bench-ns1.md](reports/ns1-maximize/nats-cluster-bench-ns1.md) | Maximized: 8c/16G/tmpfs vs baseline | -| [reports/optimal-config/REPORT.md](reports/optimal-config/REPORT.md) | All studies + MQTT/UDP + DL360 projection | -| [LOG.md](LOG.md) | Chronological actions | -| [TODO.md](TODO.md) | Open items | -| [decisions/](decisions/) | One file per decision | - -No runtime. Not a NATS mailbox. - -## Install - -```sh -git clone ssh://git@git.georgelambert.org:2223/marchon/zapier-decisions.git -cd zapier-decisions -``` - -## Usage - -```sh -# No single entrypoint is declared in this tree. -ls -``` - -## Contributing - -Questions: open an issue at https://git.georgelambert.org/marchon/zapier-decisions/issues. Pull requests are accepted. - -## License - -UNLICENSED © George Lambert diff --git a/STATUS.md b/STATUS.md deleted file mode 100644 index cb33d59..0000000 --- a/STATUS.md +++ /dev/null @@ -1,164 +0,0 @@ -# Current status — 2026-10-03 - -**Multi-host cutover plan in flight:** [plans/2026-10-03-libsql-multihost-architecture.md](plans/2026-10-03-libsql-multihost-architecture.md) (D006: business SoT = **libSQL**; JetStream = work only). Arch review: **approve-with-changes** (A1–A8 merged). PM board: [plans/reviews/2026-10-03-pm-board.md](plans/reviews/2026-10-03-pm-board.md). Docs map: [plans/2026-10-03-system-docs-map.md](plans/2026-10-03-system-docs-map.md). - ---- - -# Prior status — 2026-09-12 - -Verae Time × Zapier is a **working lab** on one Proxmox host (**NS1.GEORGELAMBERT.ORG**, `70.88.205.138`). Public HTTPS doors, catalog, keep/fleet, and a **test 3-node NATS cluster** on separate LXC guests are up. The test environment **points at that cluster**. Zapier Platform `push` and live `api.veraetime.net` are still waiting on operator login / bind credentials. - -**Progress repo:** this file in [zapier-decisions](https://git.georgelambert.org/marchon/zapier-decisions). -**Catalog:** https://zapier.georgelambert.org/packages/zapier-decisions/STATUS.pdf -**Monorepo:** `master-zapier-plan-draft` (`main` / `master`). - ---- - -## Live right now - -| Layer | State | -|-------|--------| -| Catalog | https://zapier.georgelambert.org/ (PDF links path-relative; module/model books work) | -| Public doors | portal, api, zap, iam, mw, fleet, git — last off-box GET 8/8 OK | -| Keep + fleet | Running. systemd + linger **enabled** for next reboot (current processes were not double-started as systemd) | -| Test NATS | **3-node JetStream on `vmbr1`:** LXC 511/512/513 `nats-a/b/c` at `10.10.10.21–23`. Cluster name `verae`, 2 routes each. | -| Product streams | `ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE` — **replicas=3** | -| Modules on that cluster | Fleet, keep, middleware (`natsConnected: true`), billing, access planes. `NATS_URL` lists all three URLs. | -| Host loopback `:4222` | Still listening; **clients no longer use it** | -| Worker CT **510** `verae-px-worker` | `10.10.10.20` on `vmbr1`. Keep worm+tree **200** on the cluster | -| Backup | Timer `verae-backup.timer` enabled; snapshots under `/SSD2/backups/verae/` | -| Uptime | Loop on px-worker (`:3870/status`), last tick `failed: 0` | - -Client URL (private `vmbr1` only): - -```text -nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222 -``` - -Repo: https://git.georgelambert.org/marchon/verae-nats-cluster - -### Cluster message speed (`20260912T045131Z`) - -Measured from LXC **510** (not a nats-* server) against `10.10.10.21–23`. - -**Full report (charts + explanations):** [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.pdf) · [markdown](reports/nats-cluster-bench.md) - -**Second study (all code on NS1.GEORGELAMBERT.ORG / 70.88.205.138, run `20260912T051237Z`):** [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.pdf) · [markdown](reports/ns1-host/nats-cluster-bench-ns1.md). Orchestrator, nats bench, charts, HTML, and PDF ran on 138; traffic was LXC 510 → 511–513 on `vmbr1`. Includes full methodology and fine-tuning notes. Host load ~8.5/40 during the run; JetStream 1p 128 B was **7.4k** durable pubs/s (vs ~16k on the quieter first ladder); ping p99 **1.38 ms**. - -**Maximized study (`20260912T053120Z`):** 8 cores + 16 GiB on 510–513, `max_mem: 8G`, JetStream **tmpfs** for the ladder (ZFS restored after). veth/10G unchanged. [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.pdf). - -| Metric | Baseline 051237Z | Maximized | Ratio | -|--------|------------------|-----------|-------| -| JS file r=3 1p 128 B | 7,393 pubs/s | **17,388** | **2.35×** | -| JS memory r=3 4p 128 B | — | **36,355** | ~4.9× vs baseline file 1p | -| Core 1p 128 B pub | 502,502 | 599,004 | 1.19× | -| Ping p99 | 1.377 ms | **0.684 ms** | **2.01× faster** | - -Live cluster **keeps 8 cores / 16 GiB / max_mem 8G**. Product streams are back on ZFS (tmpfs was study-only). - -**Optimal-config study (`20260912T055851Z`):** same 8c/16G cluster on **ZFS**, plus r=1 vs r=3, memory vs file, reconnect tax, UDP echo, MQTT QoS0. Combined report with DL360 Gen10 projection: [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/optimal-config/REPORT.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/optimal-config/REPORT.pdf). - -| Finding | Number | -|---------|--------| -| JS file r=3 1p 128 B (8c ZFS) | 14,330 pubs/s | -| JS file r=1 1p | 18,888 (r=3 costs ~24% on same host) | -| JS memory r=1 4p | **64,923** (lab JS ceiling) | -| MQTT QoS0 | 44,862 pubs/s (not durable; ~15× slower than core) | -| UDP p99 510→511 | 0.363 ms vs NATS ping p99 1.14 ms | -| Reconnect-per-msg p99 | 1.75 ms vs persistent ping 1.14 ms | - -**Do not switch the fabric to MQTT or UDP.** Optimal lab: file r=3 for jobs/archive, memory r=3 optional for events, persistent NATS connections. Metal: 3× DL360 Gen10, local NVMe, 10GbE — projected JS file r=3 **~40–80k** 128 B pubs/s (not measured). - -Short tables: [BENCH.md](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md). Re-run: `bash scripts/bench.sh` on NS1. - -| Kind | Load | Result | -|------|------|--------| -| Core NATS | 1p1s 50k×128 B | ~1.20M msgs/s aggregate | -| Core NATS | 8p8s 200k×128 B | ~2.01M msgs/s aggregate | -| Core NATS | 4p4s 50k×1 KiB | ~630k msgs/s (~616 MB/s) | -| JetStream file r=3 | 1–4 pubs, 128 B | ~16k durable pubs/s | -| JetStream file r=3 | 4 pubs, 1 KiB | ~13.5k durable pubs/s | -| Ping (sequential RTT) | 1k×128 B | avg 0.307 ms, p99 0.734 ms | -| Flood (burst queueing) | 8p 20k×128 B | avg 454 ms | - -Product job/event/archive traffic is the JetStream row. Ping is one-message delay; flood is the mailbox catching up after a burst. - ---- - -## Built and checked in (Forgejo, each its own repo) - -IAM, portal on access-web, keep/watch/guard, fleet floors, host-deps, catalog host-move docs, plus: - -| Repo | Job | -|------|-----| -| verae-uptime | Off-box HTTPS GET of public doors | -| verae-backup | SQLite / keep / worm-tree tar; restore dry-run default | -| verae-deploy | Tagged checkout + npm ci on the target OS | -| verae-proxmox | LXC 510 worker provision | -| verae-nats-cluster | LXC 511–513 JetStream cluster | - ---- - -## Still open - -| Item | Notes | -|------|--------| -| Zapier Platform `push` | Needs Zapier login (see below) | -| `MOCK_VERAE=false` | Live `api.veraetime.net` + admin bind credentials (see below) | -| NATS nkeys/mTLS | Cluster is up **without** accounts so existing `NATS_URL`s work. `verae-nats-accounts` remains the sketch. Flip only after every client has credentials. Isolation today is `vmbr1`, not nkeys. | -| lan-134 | Still disabled (SSH not working) | -| Reboot test | systemd should own keep/fleet after reboot; do not also `nohup guard.sh` | -| Hardware NATS | Same three configs, three boxes, private NIC — change IPs in `cluster.env` | - ---- - -## Once we have a Zapier login - -Zapier cloud only talks **HTTPS** to our Zap plane (`https://zap.zapier.georgelambert.org`). No NATS, no `api.veraetime.net`. - -1. `zapier login` (or `zapier-platform login`). -2. In `packages/verae-zapier` (and `verae-activate` if the tiny app is still wanted): - - `zapier-platform register` (or attach an existing app id). - - `MIDDLEWARE_BASE_URL=https://zap.zapier.georgelambert.org` (or `https://mw.zapier.georgelambert.org` if skipping the Zap plane). - - `zapier-platform build` and `validate`. -3. `zapier-platform push` as a **private** app. -4. Invite the workspace; create a Zap: - - Auth = portal **API key** (`zmw_…`), not a Verae JWT. - - Test **Create Timestamp and Wait**, **Create Timestamp** (async `jobId`), **Find Job Status**, **Timestamp Completed** REST Hook. -5. Fix anything `validate` / `push` flags (auth labels, samples, 402/403 mapping). -6. When the contract is stable, bump the platform version and invite a real user. - -Do **not** put NATS URLs or Verae passwords into Zapier. - ---- - -## Once `MOCK_VERAE=false` and live `api.veraetime.net` - -Middleware already has the live HTTP client. Still needed: - -1. `VERAE_API_BASE_URL=https://api.veraetime.net` and working **login** (username/password → JWT). JWT stays in middleware (`tokenRef`); Zapier never sees it. -2. Confirm OpenAPI vs our client: - - `POST /api/timestamp` → **202** `{ jobId }` - - `GET /api/status/{jobId}` until `completed` / `failed` - - `POST /api/verify` - - Batch create/status if used -3. **Hash lookup is not on the live OpenAPI** — `veraeClient.lookupHash` returns **501** when not mocking. Central-chain “find by SHA-256” needs a Verae API gap fill or our own store. -4. Map live status JSON into the receipt shape the Zap app expects (`certificate`, `timestamp`, `blockIndex`, `sha256`). -5. Token refresh, 401 retry, and 402/403 mapping against real errors. -6. One live seal in the lab, then REST Hook delivery to `hooks.zapier.com`. - ---- - -## Submit → jobId → wait → receipt - -**Yes — that is the designed path**, and it already runs in **mock**. - -1. Zap (or curl) `POST /zapier/v1/timestamp` or `/timestamp/wait`. -2. Middleware `createTimestamp` calls Verae `POST /api/timestamp` (or the mock) and gets **`jobId`**. -3. It publishes **`verae.zapier.jobs.watch`** on NATS (the 3-node cluster). -4. **Wait:** subscribe to **`verae.zapier.jobs.events`** for that `jobId`, or poll `GET /api/status/{jobId}`. -5. Job-poller hits Verae status until `completed`. -6. Event `timestamp.completed` carries status; webhook-deliver POSTs the Zapier REST Hook. -7. Response includes **certificate / timestamp / blockIndex** (mock uses `mock-cert-{jobId}`, `blockIndex: 42`). Async create returns `{ jobId, sha256, existing }` immediately; wait returns the finished status. Same SHA-256 is idempotent (`existing: true`). - -If wait times out, middleware returns **`pending` + `jobId`** so the hook can finish the Zap. Live Verae must match that HTTP contract; we have **not** proven a real chain seal yet (`MOCK_VERAE` is still true). diff --git a/TODO.md b/TODO.md deleted file mode 100644 index 6f50c5b..0000000 --- a/TODO.md +++ /dev/null @@ -1,16 +0,0 @@ -# Open todos - -- [ ] Live `api.veraetime.net` with `MOCK_VERAE=false` and admin bind credentials. -- [x] Three-node NATS JetStream cluster on distinct Proxmox LXC (`verae-nats-cluster`; test env `NATS_URL` cut over; streams replicas=3). -- [ ] NATS nkeys/mTLS on that cluster (accounts file is still the lab stand-in; not flipped so clients keep working on `vmbr1`). -- [x] Exclusive JetStream consumer for `verae.zapier.jobs.events` on `verae-jobs-events` (`JOBS_EVENTS_EXCLUSIVE=1`; middleware skips the router). -- [x] Auth on CS/sales/accounting HTML via `verae-staff-session` (`STAFF_AUTH=1`). -- [x] Staff IAM: named users, roles, permissions (`verae-staff-iam` :3028). -- [ ] Zapier Platform `push` of a private app. -- [x] Move portal static files fully into `verae-access-web` (`/portal/` public door; API proxied to loopback edge). -- [x] Turn IAM on for the running lab (fleet `STAFF_IAM_URL` + restart). -- [x] IAM hardening: JSON sessions, login rate-limit, JSON 401 as well as HTML 302, credit principal = IAM username. -- [x] Host keep-alive (`verae-keep`) on NS1 with watch+guard; honors admin pause/stop. -- [x] Docs synced to IAM, public portal `:3021`, keep, bootstrap server types. -- [x] `verae-bootstrap` clones all module repos and installs deps per server type. -- [ ] Cut over billing/customers/pricing/usage ledger from single-host billing.db to shared libSQL (D006); keep JetStream for jobs only. diff --git a/decisions/D001-prepaid-sot.md b/decisions/D001-prepaid-sot.md deleted file mode 100644 index e5b1d83..0000000 --- a/decisions/D001-prepaid-sot.md +++ /dev/null @@ -1,13 +0,0 @@ -# D001 — Account-balance is the only prepaid writer - -**Status:** accepted -**Step:** 1 - -Prepaid cents, credits, usage, and payments persist in `zappier-account-balance` (`books.json`). zappier-edge `balanceCents` is a cache filled from the books reply. Tests without NATS/HTTP books still mutate the cache so unit tests stay hermetic. - -## Addendum (2026-10-03, D006) - -Durable prepaid/credits/usage/payments store is shared **libSQL**, not per-process -`books.json` or per-host `billing.db`. Single writer path still required. Edge -`balanceCents` remains a cache. Lab `books.json` / `billing.db` are migration sources. - diff --git a/decisions/D002-identity.md b/decisions/D002-identity.md deleted file mode 100644 index 7548947..0000000 --- a/decisions/D002-identity.md +++ /dev/null @@ -1,6 +0,0 @@ -# D002 — veraeUserId is the join key, not the JWT - -**Status:** accepted -**Step:** 2 - -Public token remains the zappier API key. Chain JWT stays server-side (`tokenRef`). `zappier-identity` is the mailbox for bind/lookup so portal and middleware share one map. diff --git a/decisions/D003-public-doors.md b/decisions/D003-public-doors.md deleted file mode 100644 index 9d32527..0000000 --- a/decisions/D003-public-doors.md +++ /dev/null @@ -1,6 +0,0 @@ -# D003 — Access planes are the only public HTTP doors - -**Status:** accepted -**Step:** 3 - -Zapier → `:3024`, browser → `:3021`, customer API → `:3022`, leaf → `:3023`, staff → `:3025`. zappier-edge listens on `127.0.0.1:3000` for identity, Stripe, admin, and as upstream for those planes. diff --git a/decisions/D004-leaf-accounts.md b/decisions/D004-leaf-accounts.md deleted file mode 100644 index 65de80d..0000000 --- a/decisions/D004-leaf-accounts.md +++ /dev/null @@ -1,6 +0,0 @@ -# D004 — Leaf cannot touch billing, even if authz is skipped - -**Status:** accepted -**Step:** 4 - -`verae-nats-accounts/policy.json` is the allow-list. access-leaf refuses any target not in the leaf set. nats.conf documents INTERNAL vs LEAF accounts for when the lab cluster is replaced with a real accounts-enabled nats-server. diff --git a/decisions/D005-fleet-local-first.md b/decisions/D005-fleet-local-first.md deleted file mode 100644 index 4df6ecd..0000000 --- a/decisions/D005-fleet-local-first.md +++ /dev/null @@ -1,6 +0,0 @@ -# D005 — HTTP services spawn locally; lan-134 stays off - -**Status:** accepted -**Step:** 5 - -SSH to `70.88.205.134` blocked keepFloor. HTTP apps (`spawn.command`) prefer `kind=local`. `lan-134` is disabled unless `FLEET_ENABLE_LAN134=1`. Failed SSH hosts are skipped after 8s. diff --git a/decisions/D006-libsql-business-jetstream-work.md b/decisions/D006-libsql-business-jetstream-work.md deleted file mode 100644 index 11cb5a2..0000000 --- a/decisions/D006-libsql-business-jetstream-work.md +++ /dev/null @@ -1,39 +0,0 @@ -# D006 — Business data in libSQL; JetStream only for work - -**Status:** accepted -**Step:** 6 -**Date:** 2026-10-03 - -Launch topology is **3–5 machines** (gateway, bridge, delivery, and related planes). Shared business state must not live as per-host SQLite files. - -## Decision - -1. **System of record for business / accounting / customers / pricing** is **libSQL** (Turso’s SQLite fork; server mode or hosted). That includes: - - customers, segments, tiers, balances, Stripe / PO metadata - - pricing and channel×segment rate cards - - invoices and invoice lines - - metered usage ledger (`usage_entries` and its append-only hash chain) - - other accounting and CRM-style records for the Zapier/Verae edge stack - -2. **NATS JetStream** is used **as necessary** for multi-host **work distribution** (job queues, competing consumers, retries, durable event fans that are not the money ledger). Examples: `ZAPIER_JOBS`, worker pull/ack, bridge task streams. - -3. **Core NATS (ephemeral)** remains mandatory for Verae timestamp **job-complete / jobId** notifications (`verae.ts.completed`, `verae.ts.receipt`). Those must **not** move to JetStream. - -4. **Timestamping-svc Go** `blockchain.db` and other Go timestamping SQLite files stay on that service; this decision does not relocate them. - -## Relationship to earlier decisions - -- **D001** (single prepaid writer) still holds: one logical writer path for prepaid/credits. The durable store for that path becomes **libSQL**, not per-process `books.json` or a private `billing.db` on each machine. Edge `balanceCents` may remain a cache filled from the billing/libSQL service. -- Lab `billing.db` / `zappier.db` on 138 are the current single-host stand-in; migrate that schema into shared libSQL before splitting planes across hosts. - -## Non-goals - -- JetStream as the customer/balance/invoice/pricing database -- Litestream as a multi-writer fabric (Litestream remains backup/DR for a primary if used) -- Per-machine copies of customer or pricing tables - -## Implementation sketch - -- Keep repositories behind a small DB interface; swap better-sqlite3 / modernc file opens for libSQL client when the shared server is up. -- One billing writer path owns usage hash-chain appends (`BEGIN`-style sequencing). -- Gateway/bridge/delivery call billing APIs or the shared libSQL endpoint; they do not each open a local billing file for production. diff --git a/decisions/index.md b/decisions/index.md deleted file mode 100644 index 71f3d9b..0000000 --- a/decisions/index.md +++ /dev/null @@ -1,10 +0,0 @@ -# Decisions - -6 files in this book. - -- [D001-prepaid-sot.md](D001-prepaid-sot.md) -- [D002-identity.md](D002-identity.md) -- [D003-public-doors.md](D003-public-doors.md) -- [D004-leaf-accounts.md](D004-leaf-accounts.md) -- [D005-fleet-local-first.md](D005-fleet-local-first.md) -- [D006-libsql-business-jetstream-work.md](D006-libsql-business-jetstream-work.md) diff --git a/jobs/pieces.yaml b/jobs/pieces.yaml deleted file mode 100644 index 2ace19a..0000000 --- a/jobs/pieces.yaml +++ /dev/null @@ -1,23 +0,0 @@ -# Multi-build piece manifest for job-bus. -# Copy to jobs/pieces.yaml in your repo and edit. -# Each piece becomes one tracked job; kind routes cpu→Xeons, gpu→Mac. -batch: "zapier-decisions-EDITME" -defaults: - kind: cpu - requires: [] - timeout_sec: 3600 - cleanup: none # or on_success; full wipe via: job-cleanup --batch -pieces: - - id: piece-01 - kind: cpu - requires: [bin:bash] - cmd: ["make", "piece-01"] - # cwd: defaults to repo root on submitter; set worker-visible path if needed - - id: piece-02 - kind: cpu - requires: [ffmpeg, pip:numpy] - cmd: ["python3", "scripts/build_piece.py", "--piece", "2"] - - id: piece-gpu-hero - kind: gpu - requires: [pip:Pillow] - cmd: ["python3", "scripts/gpu_hero.py"] diff --git a/package.json b/package.json deleted file mode 100644 index a12c852..0000000 --- a/package.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "name": "zapier-decisions", - "version": "0.1.0", - "private": true, - "description": "Architecture decisions and action log for the Verae × Zapier stack" -} diff --git a/plans/2026-10-03-forgejo-sources.md b/plans/2026-10-03-forgejo-sources.md deleted file mode 100644 index 66e73a2..0000000 --- a/plans/2026-10-03-forgejo-sources.md +++ /dev/null @@ -1,178 +0,0 @@ -# Verae × Zapier Forgejo sources - -Research date: 2026-10-03 - -Sources: -- Forgejo API `https://git.georgelambert.org/api/v1/repos/search` (q=zapier,zappier,verae,nats,account-balance,access) plus direct `/repos/marchon/` -- Host `marchon@70.88.205.138`: catalog `/SSD2/sites/zapier.georgelambert.org/packages`, live `/opt/verae/{zappier-edge,verae-zapier-middleware}`, worktree `/home/marchon/zappier-edge-work` - -## Executive summary — what is SoT - -| Concern | Source of truth (design) | What is actually live on 138 today | -|---|---|---| -| **Billing / customers / pricing / invoices / usage ledger** | **`marchon/zappier-edge`** (+ D006: migrate durable store to **shared libSQL**). Department UIs are thin clients. | **`/opt/verae/zappier-edge`** with **`billing.db`** (ledger) + **`zappier.db`** (auth/sessions), WAL SQLite. Worktree `/home/marchon/zappier-edge-work` is ahead of Forgejo. Service `zappier-edge.service`. | -| **Prepaid balance book** | Design D001: **`marchon/zappier-account-balance`** persists **`books.json`**; edge `balanceCents` is a cache. D006: prepaid writer path stays single, store becomes libSQL. | No `books.json` found under `/opt/verae` or `/home/marchon`. Live prepaid appears folded into edge SQLite/`balanceCents` until account-balance/libSQL cutover. | -| **Public doors (access planes)** | **`verae-access-{zapier,web,api,leaf,staff}`** + **`verae-access-authz`** (D003). Edge stays loopback. | Catalog packages exist; **not** confirmed as separate live systemd units on this pass. Live public surface for Zapier metering is **`zap.verae.ai` → zappier-edge :3040** plus **`verae-zapier-middleware.service`**. | -| **NATS / JetStream fabric** | **`verae-nats-cluster`** (LXC 511–513), **`verae-nats-accounts`**, **`verae-nats-process`** template. JetStream streams designed: `ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE`. | Live: `nats-public-edge.service`, `verae-nats-private-proxy.service`, various leaf workers. **D006 / standing rule:** JetStream for **work queues only**; **core NATS** for timestamp **job-complete/jobId** (`verae.ts.completed`, `verae.ts.receipt`) — never JetStream for those. | -| **Jobs (Zapier timestamp path)** | **`verae-middleware`** owns `/zapier/v1`, job wait, NATS publish/consume; **`verae-jobs-events`** mailbox for `verae.zapier.jobs.events`; splitter/worm/tree for archive path. | **`/opt/verae/verae-zapier-middleware`** + `verae-zapier-middleware.service` active. Edge also does direct Verae/batch metering on `:3040`. | -| **Decisions / libSQL** | **`zapier-decisions`** D006 (2026-10-03): libSQL = business SoT; JetStream = multi-host work; core NATS = job-complete. | Decision file present in catalog `packages/zapier-decisions/decisions/D006-libsql-business-jetstream-work.md`. | - -**Important:** `/SSD2/sites/zapier.georgelambert.org/packages/*` are **catalog doc trees** (README/PDF), **not** git working copies (no `.git`). Live code under `/opt/verae/zappier-edge` is a **deployed tree without `.git`**. Clone from Forgejo SSH port **2223**. - -## Catalog package ↔ Forgejo repo - -| Catalog package | Forgejo `full_name` | Default branch | Forgejo updated | Size | Active? | One-line purpose | Clone (HTTPS) | Clone (SSH) | -|---|---|---|---|---|---|---|---|---| -| `zappier` | `marchon/zappier-edge` | main | 2026-09-23T14:00:59 | 187072 | LIVE on 138 (/opt/verae/zappier-edge); worktree Oct 2 ahead of Forgejo Sep 23 | Metered HTTPS edge: portal, admin, Stripe, rate card, usage ledger; proxies Verae to middleware | `https://git.georgelambert.org/marchon/zappier-edge.git` | `ssh://git@git.georgelambert.org:2223/marchon/zappier-edge.git` | -| `zappier-account-balance` | `marchon/zappier-account-balance` | main | 2026-09-23T14:00:59 | 48 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Prepaid balance SoT via NATS (books.json design); edge balanceCents is cache | `https://git.georgelambert.org/marchon/zappier-account-balance.git` | `ssh://git@git.georgelambert.org:2223/marchon/zappier-account-balance.git` | -| `zappier-customer-service` | `marchon/zappier-customer-service` | main | 2026-09-23T14:00:59 | 68 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | CS goodwill credits / statement review over verae.billing.* | `https://git.georgelambert.org/marchon/zappier-customer-service.git` | `ssh://git@git.georgelambert.org:2223/marchon/zappier-customer-service.git` | -| `zappier-sales-pricing` | `marchon/zappier-sales-pricing` | main | 2026-09-23T14:00:59 | 59 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Sales per-customer tier/multiplier; reviews via NATS statement | `https://git.georgelambert.org/marchon/zappier-sales-pricing.git` | `ssh://git@git.georgelambert.org:2223/marchon/zappier-sales-pricing.git` | -| `zappier-accounting-export` | `marchon/zappier-accounting-export` | main | 2026-09-23T14:00:59 | 55 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | QuickBooks IIF + accounting CSV from edge invoices + NATS statements | `https://git.georgelambert.org/marchon/zappier-accounting-export.git` | `ssh://git@git.georgelambert.org:2223/marchon/zappier-accounting-export.git` | -| `zappier-identity` | `marchon/zappier-identity` | main | 2026-09-23T14:00:59 | 33 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Bind/lookup veraeUserId (not JWTs) | `https://git.georgelambert.org/marchon/zappier-identity.git` | `ssh://git@git.georgelambert.org:2223/marchon/zappier-identity.git` | -| `verae-zapier-middleware` | `marchon/verae-middleware` | main | 2026-09-22T06:36:26 | 145 | LIVE on 138 (verae-zapier-middleware.service) | Zapier HTTPS /zapier/v1/* + NATS workers; Zapier never speaks NATS | `https://git.georgelambert.org/marchon/verae-middleware.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-middleware.git` | -| `verae-zapier` | `marchon/verae-zapier-app` | main | 2026-09-22T06:17:43 | 102 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Zapier Platform CLI app (timestamp/wait/batch/verify/hooks) | `https://git.georgelambert.org/marchon/verae-zapier-app.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-zapier-app.git` | -| `verae-zapier-simulator` | `marchon/verae-zapier-simulator` | main | 2026-09-23T14:00:58 | 73 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | In-process Zapier editor stand-in with hop trace console | `https://git.georgelambert.org/marchon/verae-zapier-simulator.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-zapier-simulator.git` | -| `verae-activate` | `marchon/verae-activate` | main | 2026-09-23T14:00:47 | 57 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Activate-now Zapier app (Add Numbers, Echo, SHA256, mock stamp) | `https://git.georgelambert.org/marchon/verae-activate.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-activate.git` | -| `verae-access-zapier` | `marchon/verae-access-zapier` | main | 2026-09-23T14:00:57 | 33 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Public Zapier HTTPS door (:3024) into middleware | `https://git.georgelambert.org/marchon/verae-access-zapier.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-access-zapier.git` | -| `verae-access-web` | `marchon/verae-access-web` | main | 2026-09-23T14:00:57 | 39 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Public customer web/portal door (:3021) | `https://git.georgelambert.org/marchon/verae-access-web.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-access-web.git` | -| `verae-access-api` | `marchon/verae-access-api` | main | 2026-09-23T14:00:57 | 33 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Public direct customer API door (:3022) | `https://git.georgelambert.org/marchon/verae-access-api.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-access-api.git` | -| `verae-access-leaf` | `marchon/verae-access-leaf` | main | 2026-09-23T14:00:57 | 37 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | S2S leaf plane (:3023); no billing subjects | `https://git.georgelambert.org/marchon/verae-access-leaf.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-access-leaf.git` | -| `verae-access-staff` | `marchon/verae-access-staff` | main | 2026-09-23T14:00:57 | 55 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Staff CS/sales/accounting plane (:3025) | `https://git.georgelambert.org/marchon/verae-access-staff.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-access-staff.git` | -| `verae-access-authz` | `marchon/verae-access-authz` | main | 2026-09-23T14:00:57 | 41 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | NATS authz.check gate for all access planes | `https://git.georgelambert.org/marchon/verae-access-authz.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-access-authz.git` | -| `verae-jobs-events` | `marchon/verae-jobs-events` | main | 2026-09-23T14:00:58 | 33 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Mailbox for verae.zapier.jobs.events (passthrough today) | `https://git.georgelambert.org/marchon/verae-jobs-events.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-jobs-events.git` | -| `verae-nats-accounts` | `marchon/verae-nats-accounts` | main | 2026-09-22T06:36:30 | 33 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | INTERNAL vs LEAF NATS account/subject policy | `https://git.georgelambert.org/marchon/verae-nats-accounts.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-nats-accounts.git` | -| `verae-nats-cluster` | `marchon/verae-nats-cluster` | main | 2026-09-22T06:16:14 | 3666 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | 3× Proxmox LXC JetStream cluster (10.10.10.21–23) | `https://git.georgelambert.org/marchon/verae-nats-cluster.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-nats-cluster.git` | -| `verae-nats-process` | `marchon/verae-nats-process` | main | 2026-09-23T14:00:58 | 40 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Template for a new addressed NATS process | `https://git.georgelambert.org/marchon/verae-nats-process.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-nats-process.git` | -| `verae-request-splitter` | `marchon/verae-request-splitter` | main | 2026-09-23T14:00:58 | 42 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Split chain hash vs off-chain archive.put | `https://git.georgelambert.org/marchon/verae-request-splitter.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-request-splitter.git` | -| `verae-archive-worm` | `marchon/verae-archive-worm` | main | 2026-09-23T14:00:58 | 43 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Bloom-filtered WORM archive node | `https://git.georgelambert.org/marchon/verae-archive-worm.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-archive-worm.git` | -| `verae-archive-aggregator` | `marchon/verae-archive-aggregator` | main | 2026-09-23T14:00:58 | 44 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Merge archive.query replies into job JSON | `https://git.georgelambert.org/marchon/verae-archive-aggregator.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-archive-aggregator.git` | -| `verae-tree-node` | `marchon/verae-tree-node` | main | 2026-09-23T14:00:58 | 42 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Merkle leaf proofs for bulk summaries | `https://git.georgelambert.org/marchon/verae-tree-node.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-tree-node.git` | -| `verae-fleet` | `marchon/verae-fleet` | main | 2026-09-22T06:37:31 | 3896 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Operator catalog, replica floors, pause/restart console | `https://git.georgelambert.org/marchon/verae-fleet.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-fleet.git` | -| `verae-keep` | `marchon/verae-keep` | main | 2026-09-22T06:35:54 | 43 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Host keep-alive; restarts unless operator paused/stopped | `https://git.georgelambert.org/marchon/verae-keep.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-keep.git` | -| `verae-bootstrap` | `marchon/verae-bootstrap` | main | 2026-09-22T06:17:17 | 54 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Clone all module repos + host-deps per server type | `https://git.georgelambert.org/marchon/verae-bootstrap.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-bootstrap.git` | -| `verae-ops` | `marchon/verae-ops` | main | 2026-09-11T19:00:06 | 86 | Older Forgejo (Sep 11); still the install book | Docker/Proxmox/VM/hardware install and service linking | `https://git.georgelambert.org/marchon/verae-ops.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-ops.git` | -| `verae-deploy` | `marchon/verae-deploy` | main | 2026-09-11T18:36:11 | 25 | Older Forgejo (Sep 11) | Tagged checkout + npm ci on target OS | `https://git.georgelambert.org/marchon/verae-deploy.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-deploy.git` | -| `verae-proxmox` | `marchon/verae-proxmox` | main | 2026-09-22T06:36:28 | 28 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Provision archive-worker LXC on NS1 | `https://git.georgelambert.org/marchon/verae-proxmox.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-proxmox.git` | -| `verae-backup` | `marchon/verae-backup` | main | 2026-09-22T06:31:38 | 28 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Backup/restore SQLite, JetStream, worm/tree, IAM | `https://git.georgelambert.org/marchon/verae-backup.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-backup.git` | -| `verae-uptime` | `marchon/verae-uptime` | main | 2026-09-22T06:36:26 | 28 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Off-box HTTPS watch of public doors | `https://git.georgelambert.org/marchon/verae-uptime.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-uptime.git` | -| `verae-staff-iam` | `marchon/verae-staff-iam` | main | 2026-09-23T14:00:58 | 49 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Internal staff users/roles/permissions SoT | `https://git.georgelambert.org/marchon/verae-staff-iam.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-staff-iam.git` | -| `verae-staff-session` | `marchon/verae-staff-session` | main | 2026-09-23T14:00:58 | 37 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Shared staff cookie login | `https://git.georgelambert.org/marchon/verae-staff-session.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-staff-session.git` | -| `verae-staff-ui` | `marchon/verae-staff-ui` | main | 2026-09-23T14:00:58 | 32 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Shared staff review HTML template | `https://git.georgelambert.org/marchon/verae-staff-ui.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-staff-ui.git` | -| `overview` | `marchon/overview` | main | 2026-09-22T06:41:52 | 1567 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | High-level system map and NATS/uptime docs | `https://git.georgelambert.org/marchon/overview.git` | `ssh://git@git.georgelambert.org:2223/marchon/overview.git` | -| `docs-master` | `marchon/zapier-docs-master` | main | 2026-09-22T06:32:02 | 197 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Per-module SUMMARY/NATS contracts and message flows | `https://git.georgelambert.org/marchon/zapier-docs-master.git` | `ssh://git@git.georgelambert.org:2223/marchon/zapier-docs-master.git` | -| `dev-overview` | `marchon/dev-overview` | main | 2026-09-22T06:17:14 | 327 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Designed vs running NATS routes; Zapier→Verae path | `https://git.georgelambert.org/marchon/dev-overview.git` | `ssh://git@git.georgelambert.org:2223/marchon/dev-overview.git` | -| `zapier-decisions` | `marchon/zapier-decisions` | main | 2026-09-22T06:44:58 | 1920 | Active decision log; D006 dated 2026-10-03 in catalog copy | Architecture decisions (D001–D006), logs, benches | `https://git.georgelambert.org/marchon/zapier-decisions.git` | `ssh://git@git.georgelambert.org:2223/marchon/zapier-decisions.git` | -| `zapier-user-docs` | `marchon/zapier-user-docs` | main | 2026-09-23T14:00:59 | 52 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Customer signup → register → chain/tree lookup | `https://git.georgelambert.org/marchon/zapier-user-docs.git` | `ssh://git@git.georgelambert.org:2223/marchon/zapier-user-docs.git` | -| `ui-docs` | `marchon/UI-Docs` | main | 2026-09-23T14:00:57 | 21512 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Operator/staff/portal walkthrough screenshots + review PDF | `https://git.georgelambert.org/marchon/UI-Docs.git` | `ssh://git@git.georgelambert.org:2223/marchon/UI-Docs.git` | - -## Related Forgejo repos (search hits, not catalog packages) - -| full_name | updated | purpose | clone | -|---|---|---|---| -| `marchon/master-zapier-plan-draft` | 2026-09-20T19:06:04 | Monorepo plan/snapshot workspace; not the only clone path | `https://git.georgelambert.org/marchon/master-zapier-plan-draft.git` | -| `marchon/nats-service-endpoints` | 2026-09-23 | Canonical NATS endpoint catalog | `https://git.georgelambert.org/marchon/nats-service-endpoints.git` | -| `marchon/verae-mcp-timestamping-server` | 2026-09-29 | MCP timestamping (live `/opt/verae/...`) | `https://git.georgelambert.org/marchon/verae-mcp-timestamping-server.git` | -| `marchon/verae-mcp-timestamping-server-go` | 2026-09-29 | Go MCP + tsnats sibling | `https://git.georgelambert.org/marchon/verae-mcp-timestamping-server-go.git` | - -## Notes: billing, libSQL, sqlite, JetStream, books.json, account-balance - -- **billing:** Live ledger/rate card/customers/invoices in **zappier-edge** `billing.db` (`BILLING_DB`); auth in `zappier.db`. Docs in edge `docs/BILLING-DB.md`, `CHANNEL-BILLING.md`, `PEERGOS-BILLING.md`. Department repos (CS/sales/accounting-export) call NATS + edge admin; they are not the money DB. -- **sqlite:** Lab/single-host stand-in. Edge uses better-sqlite3 + WAL. Go timestamping-svc DBs stay separate (`blockchain.db`) — do not relocate (D006). -- **libSQL:** Accepted in **D006** as shared SoT for business data across 3–5 machines. Not yet the live store on 138. -- **JetStream:** Designed streams for jobs/events/webhooks/usage/archive. Use for **work distribution** only. **Forbidden** for timestamp job-complete/jobId notify (core NATS subjects). -- **books.json:** Documented as persistence for **zappier-account-balance**. Not found on 138 in this survey; design SoT for prepaid until libSQL migration absorbs it (D001∩D006). -- **account-balance:** Forgejo `marchon/zappier-account-balance`; subjects `verae.billing.statement.get|balance.get|balance.adjust` + usage/payment/credit pubs; queue group `account-balance`; HTTP `:3010`. - -## Host inventory (138) - -### Catalog packages (`/SSD2/sites/zapier.georgelambert.org/packages`) -Doc snapshots dated ~2026-09-14. Names match table above. No `.git` remotes. - -### Live `/opt/verae` Zapier-related -- `/opt/verae/zappier-edge` — `zappier-edge.service` **active**; `billing.db`/`zappier.db` present; **no `.git`** -- `/opt/verae/verae-zapier-middleware` — `verae-zapier-middleware.service` **active**; README points at Forgejo `verae-middleware` -- Work copy: `/home/marchon/zappier-edge-work` (mtime Oct 2) — treat as practical edge SoT until Forgejo catch-up - -## Recommended clone list for implementation - -SSH base: `ssh://git@git.georgelambert.org:2223/marchon/.git` (port **2223**). - -### Must clone first (critical path) -```bash -mkdir -p ~/verae-src && cd ~/verae-src -git clone ssh://git@git.georgelambert.org:2223/marchon/zappier-edge.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-middleware.git -git clone ssh://git@git.georgelambert.org:2223/marchon/zappier-account-balance.git -git clone ssh://git@git.georgelambert.org:2223/marchon/zapier-decisions.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-access-zapier.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-access-web.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-access-api.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-access-authz.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-nats-cluster.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-nats-accounts.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-jobs-events.git -git clone ssh://git@git.georgelambert.org:2223/marchon/overview.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-bootstrap.git -``` - -### Next (jobs / archive path + Zapier app) -```bash -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-zapier-app.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-request-splitter.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-archive-worm.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-archive-aggregator.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-tree-node.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-access-leaf.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-access-staff.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-zapier-simulator.git -``` - -### Billing department satellites -```bash -git clone ssh://git@git.georgelambert.org:2223/marchon/zappier-customer-service.git -git clone ssh://git@git.georgelambert.org:2223/marchon/zappier-sales-pricing.git -git clone ssh://git@git.georgelambert.org:2223/marchon/zappier-accounting-export.git -git clone ssh://git@git.georgelambert.org:2223/marchon/zappier-identity.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-staff-iam.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-staff-session.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-staff-ui.git -``` - -### Ops / docs (as needed) -```bash -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-ops.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-fleet.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-keep.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-deploy.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-backup.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-uptime.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-proxmox.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-nats-process.git -git clone ssh://git@git.georgelambert.org:2223/marchon/verae-activate.git -git clone ssh://git@git.georgelambert.org:2223/marchon/zapier-docs-master.git -git clone ssh://git@git.georgelambert.org:2223/marchon/zapier-user-docs.git -git clone ssh://git@git.georgelambert.org:2223/marchon/UI-Docs.git -git clone ssh://git@git.georgelambert.org:2223/marchon/dev-overview.git -git clone ssh://git@git.georgelambert.org:2223/marchon/master-zapier-plan-draft.git -``` - -Or use bootstrap: clone `verae-bootstrap` then `bash scripts/fetch.sh ns1-all-in-one` (full `types/` may be incomplete in the **catalog** copy; prefer Forgejo clone of bootstrap). - -## Search hit counts (API) - -| q | results (limit 50) | -|---|---| -| zapier | 7 | -| zappier | 6 | -| verae | 32 | -| nats | 4 | -| account-balance | 1 | -| access | 7 (includes unrelated MCP mirror) | - -## Caveats - -1. Forgejo `updated_at` for most module repos clusters on **2026-09-22/23** (likely bulk push). **Live edge code on 138 is newer (Oct 2)** than Forgejo edge — sync/push before treating Forgejo as current for billing. -2. Catalog package `verae-fleet` README incorrectly linked `verae-keep` in one scrape; overview table maps fleet→`verae-fleet` correctly. -3. Catalog `docs-master` README first Forgejo link hit `verae-bootstrap` via grep noise; real docs repo is **`zapier-docs-master`**. -4. `master-zapier-plan-draft` is a large monorepo snapshot — useful archaeology, not the independent-module clone path. - diff --git a/plans/2026-10-03-libsql-multihost-architecture.md b/plans/2026-10-03-libsql-multihost-architecture.md deleted file mode 100644 index 861a07b..0000000 --- a/plans/2026-10-03-libsql-multihost-architecture.md +++ /dev/null @@ -1,295 +0,0 @@ -# Verae × Zapier Multi-Host Architecture & Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (or executing-plans) task-by-task. Steps use checkbox (`- [ ]`) syntax. - -**Goal:** Run the Verae × Zapier commercial stack across **3–5 machines** with **libSQL** as the system of record for all business/billing/customer/pricing/accounting data, **JetStream only for work queues**, and proven end-to-end userflows under continuous QA. - -**Architecture:** Public HTTPS doors → zappier-edge (meter) → middleware (stamp) → `api.veraetime.net`; private NATS for workers; shared libSQL for money & CRM; Go timestamping DBs stay local to timestamping-svc. - -**Tech Stack:** Node (edge, middleware, access planes, CS/sales/acct), NATS JetStream (3-node lab → production), libSQL / go-libsql or `@libsql/client`, better-sqlite3 only as local/dev adapter, Litestream optional for libSQL file primary backup, Jest + gate scripts + E2E userflow harness. - -**Spec / SoT docs:** -- Catalog: https://zapier.georgelambert.org/ -- Map: `/tmp/zapier-system-docs-map.md` -- Decisions: D001–D006 (`packages/zapier-decisions/decisions/`) -- Composition: `docs/02-architecture/composition.md` -- NATS: `docs/02-architecture/nats-subjects.md`, `overview/03-nats-cluster.md` -- Ops: `packages/verae-ops/GETTING-STARTED.md` -- Digests-only stamps: mcp.verae.ai + D006 note + live swagger - -**Forgejo org:** https://git.georgelambert.org/marchon/ (modules listed in §2) - -## Global Constraints - -- **D006:** Business/billing/customers/pricing/accounting/usage ledger → **libSQL only** (shared). -- **JetStream:** work queues as necessary (`ZAPIER_JOBS`, `ZAPIER_WEBHOOKS`, archive put/query). Not money SoT. -- **Core NATS only** for `verae.ts.completed` / `verae.ts.receipt` (no JetStream on that path). -- **D001 refined:** single prepaid **writer path**; durable store is libSQL (not per-host `books.json` / `billing.db`). -- **D002:** public credential = zappier `x-api-key`; `veraeUserId` join; JWT = `tokenRef` server-side. -- **D003:** only access planes are public HTTP; edge loopback (except intentional public `zap.verae.ai` edge). -- **D004:** leaf must not touch billing. -- **Digests only** for stamp submit (`POST /api/batch/timestamp` multihash; MCP shape). -- **Never** modify Go timestamping-svc SQLite (`blockchain.db`, etc.) as part of this plan. -- **WAL** (or libSQL equivalent durability) on all SQL stores. -- Credits language in product copy; ledger may keep `*_cents` column names as credit units. -- Deploy never rsyncs live `*.db*` over production data. -- **ZAPIER_USAGE** / `verae.zapier.usage` is telemetry or export only — never usage/money SoT. -- Litestream (if used) backs up a **single** libSQL primary; never multi-writer fabric. -- **Single meter/prepaid writer** process owns `usage_entries` append + balance adjust; other hosts call that API. -- Dual-run: libSQL primary for writes; file/SQLite fallback is **read-only** (or kill-switched). -- Auth (`admin_users`, `portal_sessions`): v1 remains edge-local `zappier.db`; not in `verae_business`. -- Product edge bind on 138: `127.0.0.1:3040` (`zap.verae.ai`); `ZAPPIER_UPSTREAM` defaults to `http://127.0.0.1:3130` for live lab smoke (not `:3100` mock unless intentional). -- Catalog docs that call account-balance “prepaid SoT” are stale after D006; Phase 0 patches them. - ---- - -## 1. System map (from catalog + live lab) - -```text -Internet - ├─ Zapier cloud ──HTTPS──► access-zapier :3024 ──► zappier-edge (meter) ──► middleware :3100 - ├─ Browsers ────HTTPS──► access-web :3021 /portal - ├─ Customer API ─HTTPS──► access-api :3022 - ├─ Leaf S2S ────HTTPS──► access-leaf :3023 (no billing) - └─ Staff ───────HTTPS──► access-staff :3025 + IAM :3028 - -Private - ├─ NATS JetStream 10.10.10.21–23 (jobs.watch, webhooks.deliver, archive.*) - ├─ Core NATS (verae.ts.completed, verae.ts.receipt, billing request-reply) - ├─ libSQL (TARGET) ← customers, pricing, usage chain, invoices, balances - ├─ Workers job-poller, webhook-deliver, archive-worm×N, tree-node×N - └─ api.veraetime.net / MOCK_VERAE -``` - -### Critical userflows (must pass QA) - -1. **Signup → API key → Statement** (portal via access-web). -2. **Stamp digests (single + batch)** via zap door → edge meter → middleware → Verae → 202 jobIds → credits headers. -3. **Async complete → Zapier REST Hook** (`jobs.watch` → poll → `jobs.events` → `webhooks.deliver`). -4. **CS credit adjust → balance visible** on portal/admin (single prepaid writer). -5. **Sales channel×segment quote → meter charge** matches rate card. -6. **Accounting export** sees payments/usage consistent with libSQL. -7. **Leaf plane** can stamp/authz but **cannot** adjust billing (D004). - ---- - -## 2. Forgejo source elements (implementation clones) - -| Concern | Repo | Notes | -|---------|------|-------| -| Billing edge | `marchon/zappier-edge` | Live `/opt/verae/zappier-edge` on 138; also catalog `packages/zappier` | -| Prepaid mailbox (legacy) | `marchon/zappier-account-balance` | `books.json` → migrate into libSQL | -| CS / Sales / Acct | `zappier-customer-service`, `zappier-sales-pricing`, `zappier-accounting-export` | | -| Identity | `zappier-identity` | | -| Middleware | `marchon/verae-middleware` (catalog: verae-zapier-middleware) | | -| Zapier apps | `verae-zapier-app`, `verae-activate` | | -| Access planes | `verae-access-{web,api,leaf,zapier,staff,authz}` | | -| Jobs/archive | `verae-jobs-events`, `verae-archive-*`, `verae-tree-node`, `verae-request-splitter` | | -| Ops | `verae-ops`, `verae-bootstrap`, `verae-fleet`, `verae-keep`, `verae-nats-cluster` | | -| Decisions | `zapier-decisions` | D001–D006 | -| Monorepo mirror | `master-zapier-plan-draft` | docs + packages composition | - ---- - -## 3. Conflict register (resolve before/with build) - -| ID | Conflict | Resolution (locked) | -|----|----------|---------------------| -| C1 | D001 `books.json` SoT vs live `billing.db` vs D006 libSQL | **libSQL is SoT**; migrate both books + billing.db schema; account-balance becomes libSQL-backed service or folds into billing API | -| C2 | README “imported, not yet wired” vs STATUS lab up | Treat STATUS + live doors as operational truth; update README composition status in Phase 0 docs | -| C3 | overview `zmw_` Bearer vs composition `x-api-key` | **Public = x-api-key** (D002/composition); fix overview/models | -| C4 | Edge ports `:3000` / `:13000` / live `:3040` (`zap.verae.ai`) | Document env matrix; lab doors proxy to loopback; zap.verae.ai may remain public edge | -| C5 | JetStream `jobs.events` vs ban on JetStream for job-complete | **Different subjects:** Zapier job lifecycle may use JetStream `verae.zapier.jobs.events`; Verae chain notify `verae.ts.*` stays **core NATS** | -| C6 | TimestampRequest “Hello world” data vs digests-only | Digests-only + multihash batch (already fixed on live swagger; sync catalog models) | -| C7 | Dual prepaid writers (edge SQLite + account-balance) | One writer API on libSQL; edge caches only | -| C8 | Per-host SQLite on 3–5 machines | Forbidden for business data; libSQL shared | -| C9 | Go `blockchain.db` in scope creep | Out of scope — do not touch | -| C10 | `ZAPIER_USAGE` vs libSQL usage ledger | Demote to telemetry/export only; never money SoT | -| C11 | Dual middleware `:3100` mock vs `:3130` product upstream | Env matrix; product smoke uses **3130** | -| C12 | D001 file still says books.json SoT | D001 addendum → libSQL (done with this amend) | -| C13 | Auth store edge-local vs libSQL | **v1 edge-local** `zappier.db` | -| C14 | Concurrent hash-chain append across hosts | Single meter writer / serialized append | -| C15 | GETTING-STARTED calls account-balance SoT | Phase 0 doc patch → libSQL | -| C16 | Live `:3010` + `billing.db` dual writers | Cutover runbook: one authoritative adjust path | -| C17 | Litestream multi-writer temptation | Litestream = backup of one primary only | -| C18 | Digests-only vs upload-file stamp docs | Client hasher + digests-only only | - ---- - -## 4. Target data ownership (libSQL) - -**Database:** `verae_business` (name TBD) on shared libSQL. - -| Domain | Tables / collections | Writer | -|--------|----------------------|--------| -| Customers / identity bind | `customers`, identity map | billing API + identity service | -| Pricing | `tiers`, `price_endpoints`, `channel_rates`, `channel_rate_meta` | admin / sales-pricing | -| Metered ledger | `usage_entries` (+ `prev_hash`/`row_hash` chain) | **single** meter writer | -| Prepaid | balances, credits, payments (from books.json) | billing adjust API | -| Invoices | `invoices`, `invoice_lines` | admin / Stripe jobs | -| Auth (optional split) | `admin_users`, `portal_sessions` | may stay edge-local or move to libSQL `auth` schema | - - -### 4.1 Launch topology (3–5 machines) - -- One libSQL primary (dedicated host or named primary among the 3–5). -- Replicas if any are read-only; writers always target primary URL (`LIBSQL_URL`). -- Backup: filesystem snapshot and/or Litestream of the **primary only**; tested restore before Phase 4. -- Fail closed: metered stamp paths return **503** if BusinessStore write unavailable (no silent local SQLite fallback in production). -- Network: libSQL on private fabric only (same class as NATS `10.10.10.0/24`); not public. -- Schema: `verae_business`; auth stays out for v1 (C13). - -**Interface:** `BusinessStore` (TS) with drivers: `LibsqlBusinessStore` (prod), `SqliteFileBusinessStore` (dev/test). - ---- - -## 5. NATS usage matrix - -| Subject / stream | Transport | Purpose | -|------------------|-----------|---------| -| `verae.zapier.jobs.watch` | JetStream | Poll Verae status | -| `verae.zapier.jobs.events` | JetStream | Terminal Zapier-pipeline job events | -| `verae.zapier.usage` / `ZAPIER_USAGE` | JetStream optional | **Telemetry/export only; NOT ledger SoT** | -| `verae.zapier.webhooks.deliver` | JetStream | Hook delivery retries | -| `verae.archive.*` | JetStream / fanout | WORM/tree | -| `verae.billing.*` | Core request-reply (+ optional pub) | Statement/balance; eventually thin facade over libSQL | -| `verae.ts.completed`, `verae.ts.receipt` | **Core NATS only** | Chain job-complete notify | -| `verae.access.*`, `verae.identity.*` | Core / as designed | Authz & bind | - ---- - -## 6. Constraint (TOC) - -> The system constraint is **shared business SoT cutover (libSQL + single prepaid writer)** because every multi-host userflow (meter, CS credit, sales quote, portal statement, accounting export) blocks on one consistent customer/balance/usage store; JetStream and doors already exist in the lab. - -Critical chain: **Conflict freeze → BusinessStore+libSQL → Migrate data → Rewire edge/CS/sales/acct → E2E userflow tests → QA whattochange loop**. - -Feeding chains (parallel): digests-only docs sync; NATS subject clarification; access-plane smoke; Forgejo clone inventory. - ---- - -## 7. Phased implementation (PM DAG) - -### Phase 0 — Docs & conflict freeze -- [ ] Publish this plan to catalog `docs/plans/` + `zapier-decisions/plans/` -- [ ] Patch conflict C2/C3/C5/C6 in overview + models + README status blurb -- [ ] Architecture review sign-off artifact: `plans/reviews/2026-10-03-arch-review.md` - -### Phase 1 — BusinessStore + libSQL skeleton (TDD) -- [ ] Define `BusinessStore` interface (customers, usage.record+verifyChain, rates, invoices, balances) -- [ ] Failing tests against interface -- [ ] `LibsqlBusinessStore` + docker/libsql-server or `:memory:`/file for CI -- [ ] Keep `SqliteFileBusinessStore` wrapping current repos for lab - -### Phase 2 — Migrate lab data -- [ ] One-shot migrator: `billing.db` + `books.json` → libSQL -- [ ] Preserve usage hash-chain tip integrity (verify before/after); refuse cutover if verifyChain fails -- [ ] Dual-run: writes libSQL only when `BUSINESS_STORE=libsql`; file fallback **read-only** -- [ ] Account-balance becomes libSQL-backed facade for `verae.billing.*` **or** is stopped after HTTP BusinessStore cutover (pick one in Phase 3) - -### Phase 3 — Rewire writers -- [ ] zappier-edge meter → BusinessStore only -- [ ] account-balance service reads/writes libSQL (or deprecate mailbox in favor of HTTP billing API) -- [ ] CS / sales / accounting-export point at same store -- [ ] Remove second writer paths (C7) - -### Phase 4 — Multi-host topology -- [ ] Deploy libSQL on dedicated host (or primary of 3–5) -- [ ] Edge/middleware/CS on separate hosts with `LIBSQL_URL` -- [ ] JetStream cluster already present — verify workers only -- [ ] Confirm leaf cannot call billing adjust - -### Phase 5 — Userflow E2E + gates -- [ ] Automate flows 1–7 (§1) in `harness/` or `packages/verae-zapier-simulator` extensions -- [ ] Jest unit + integration + gate scripts green -- [ ] Live smoke on 138 with MOCK_VERAE then optional live Verae - -### Phase 6 — QA loop (repeat until pass) -- [ ] QA finds bugs → `/whattochange` pack (a/b/c/d + todos) -- [ ] Engineers implement; PM reorders PreReqs -- [ ] Re-run full suite + userflows -- [ ] Exit when all critical userflows pass twice consecutively - ---- - -## 8. Test requirements (“tests for everything”) - -| Layer | Must cover | -|-------|------------| -| Unit | BusinessStore ops, hash-chain append/tamper, channel rate resolve, digests-only batch mapping, D004 leaf deny | -| Integration | edge→middleware mock stamp; billing adjust→statement; JetStream watch→event (testcontainers/nats) | -| Contract | OpenAPI digests-only examples; NATS payload schemas | -| E2E userflow | Flows 1–7 against lab compose or NS1 | -| Regression | Existing zappier-edge Jest suite must stay green during driver swap | -| Non-goals | Go timestamping-svc blockchain tests | - ---- - -## 9. Roles (agent orchestration) - -| Role | Responsibility | Artifact | -|------|----------------|----------| -| **Architect** | Review plan, expand conflict register, approve SoT | `plans/reviews/*-arch-review.md` | -| **PM** | DAG, task assignment, merge gates, buffer | `plans/reviews/*-pm-board.md` | -| **Engineers** | Implement Phases 1–5 in worktrees | PRs / commits citing this plan | -| **QA** | Break userflows; whattochange packs; retest | `plans/qa/*` + NotImplemented leftovers | -| **Orchestrator (this session)** | Launch agents, merge gates, enforce D006/JetStream rules | STATUS updates | - ---- - -## 10. Acceptance criteria (project buffer) - -1. One libSQL URL serves customers/pricing/usage/invoices/balances for all commercial services. -2. No production business writes to per-host `billing.db` / `books.json`. -3. JetStream used only for work/archive streams listed in §5. -4. `verae.ts.completed` / `verae.ts.receipt` remain core NATS. -5. Digests-only multihash batch path green with credit headers. -6. Userflows 1–7 pass twice in a row under QA. -7. Go timestamping DBs untouched. -8. Catalog docs match runtime (conflicts C2–C6, C10–C18 closed or owned). -9. `ZAPIER_USAGE` is non-SoT; no service writes authoritative balances there. -10. Production metered paths fail closed if BusinessStore writer is down. -11. Concurrent append contract test proves single-writer (or serialized) hash-chain safety. -12. Live env matrix documents `:3040` edge, `:3130` product upstream, `:3100` mock, `:3010` facade/retire. -13. D001 file contains D006 addendum; GETTING-STARTED / 01-system no longer call books.json SoT. - ---- - -## 11. Out of scope - -- Replacing api.veraetime.net / Go chain storage -- Zapier Platform public directory listing (private push may remain operator-gated) -- Peergos Seaweed fleet redesign (consume existing billing APIs only) -- lan-134 enablement (D005) -- Any JetStream binding for `verae.ts.completed` / `verae.ts.receipt` -- Editing `/var/lib/timestamping-svc/data/blockchain.db` or timestamping-svc test `results.db` -- Multi-primary libSQL or Litestream multi-writer -- Moving portal auth to libSQL in the same cutover as billing (defer) - -### Architecture review -See `plans/reviews/2026-10-03-arch-review.md` (**approve-with-changes**; A1–A8 merged herein). - ---- - -## Task checklist (engineer entry) - -### Task 1: BusinessStore interface + failing tests -**Files:** Create under zappier-edge `src/db/business-store.ts`, `tests/business-store.contract.test.ts` -**Produces:** Interface used by later tasks -- [ ] Write contract tests for recordUsage/verifyChain/findCustomer/getBalance/adjustBalance -- [ ] Run tests — expect fail -- [ ] Minimal SqliteFile adapter pass -- [ ] Commit citing this plan - -### Task 2: LibsqlBusinessStore -- [ ] Add `@libsql/client` -- [ ] Implement same contract -- [ ] CI uses file or embedded libsql -- [ ] Commit - -### Task 3: Migrator billing.db + books.json → libSQL -- [ ] Script + tests with fixtures -- [ ] Chain verify pre/post -- [ ] Commit - -*(PM expands Tasks 4+ after arch review.)* diff --git a/plans/2026-10-03-system-docs-map.md b/plans/2026-10-03-system-docs-map.md deleted file mode 100644 index 1b26567..0000000 --- a/plans/2026-10-03-system-docs-map.md +++ /dev/null @@ -1,383 +0,0 @@ -# Verae × Zapier System Documentation Map - -**Source root:** `/SSD2/sites/zapier.georgelambert.org` on `70.88.205.138` (NS1.GEORGELAMBERT.ORG) -**Public catalog:** https://zapier.georgelambert.org/ -**Mapped:** 2026-10-03 -**Scope:** Documentation tree only (not live `/opt/verae/zappier-edge` runtime audit) - ---- - -## 1. Docs structure - -### Site layout - -| Path | Role | -|------|------| -| `/SSD2/sites/zapier.georgelambert.org/` | Published catalog (Caddy) | -| `README.md` | Monorepo entry; composition of zappier + middleware + Zapier app | -| `TODO.md` | Middleware phase gates 0–15 | -| `docs/` | Numbered product/architecture books + modules/models | -| `docs-master/` | Per-module SUMMARY + NATS contracts + MESSAGE-FLOWS | -| `overview/` | System overview book (01–11) + diagrams | -| `packages/` | ~41 independent module packages (README + PDF each) | -| `live/` | Screenshots of public doors | -| `user-docs/` | Customer-facing guides | -| `simulator/` | Static simulator UI copy | -| `sphinx/` | Sphinx HTML build of modules/models | -| `research/` | Vendor Zapier research | - -### `docs/` tree - -``` -docs/ - 00-sources/ workspace-brief, provenance, getting-started-research, - FUNCTIONS/PLATFORM/MCP refs, veraetime-openapi.yaml, LOGIN/MONGO/RESTART - 01-product/ features-a-m, api-gap-analysis, billing-and-keys, zapier-billing-research - 02-architecture/ - overview, composition, modules-and-nats, nats-subjects, nats-gateway, - archive-nats, fleet, tree-nodes - 03-zapier/ auth, operations, README - 04-activate/ SETUP-ZAPIER-DEVELOPER - api/ middleware-openapi.yaml - developer/ debugging + modules/ - models/ data model sheets (+ models-pdf/) - modules/ per-source-module API sheets (+ modules-pdf/) - plans/ phase-gates.md - sphinx/ RST/MD sources for Sphinx - OPEN.md, WORK-LOG.md -``` - -### `docs-master/` - -- `README.md`, `composition.md`, `modules-and-nats.md`, `archive-nats.md` -- `MESSAGE-FLOWS.md`, `HOST-DEPS.md`, `CATALOG-HOST.md` -- `modules//{SUMMARY,NATS}.md` for each package - -### `overview/` - -| Doc | Topic | -|-----|--------| -| `01-system.md` | Planes, hard rule (Zapier never NATS) | -| `02-modules-and-repos.md` | Forgejo repos | -| `03-nats-cluster.md` | 3-server JetStream | -| `04-uptime.md` | Replica floors | -| `05-network-failures.md` | Reconnect / silence | -| `06-address-routing.md` | New functions = new subjects | -| `07-external-resources.md` | Chain / storage | -| `08-diagrams.md` | SVG + mermaid | -| `09-expansion-template.md` | verae-nats-process | -| `10-docs-sync.md` | Catalog sync | -| `11-hostnames.md` | Ports, DNS, public doors | -| `NATS.md`, `SUMMARY.md`, `INDEX.md` | Shortcuts | - -### `plans/` - -- Single plan book: `docs/plans/phase-gates.md` (gates 0–15; mirrors root `TODO.md`) - ---- - -## 2. Key document summaries - -### Architecture / overview - -**`docs/02-architecture/overview.md`** -- Connect Zapier to Verae without exposing JWTs, forcing Zapier to poll, or coupling billing to the chain API. -- Components: `verae-zapier` (Zapier cloud), middleware HTTPS, NATS JetStream, workers, `api.veraetime.net`. -- Flows: async create → `jobs.watch` → poll → `jobs.events` → webhook; wait path; auth `/auth/me`. -- Flags: `NATS_ENABLED`, `MOCK_VERAE`, `DEBUG_VERAE`. -- Still describes public auth as middleware `zmw_` Bearer (pre-composition framing). - -**`docs/02-architecture/composition.md`** -- Target path: Zapier → zappier `/v1/*` (`x-api-key`) → middleware `/zapier/v1/*`. -- Ownership split: billing/portal = zappier; stamp/NATS/hooks = middleware; nouns = verae-zapier. -- Zapier never talks to NATS or live Verae; payment invoices ≠ timestamp receipts. - -**`docs/02-architecture/modules-and-nats.md`** -- Master module table (edge, middleware, poller, webhook, splitter, aggregator, WORM, tree-node, fleet, simulator). -- Subjects: `verae.zapier.jobs.{watch,events}`, `webhooks.deliver`, `verae.archive.{put,query,reply.*}`. -- Zapier cloud never on NATS. - -**`docs/02-architecture/nats-subjects.md`** -- Streams: `ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, optional `ZAPIER_USAGE`. -- Payload schemas + ack/Nak semantics for poller and webhook deliver. -- Prefer `tokenRef` over JWT in messages; NATS private + auth. - -**`docs/02-architecture/archive-nats.md` / `fleet.md` / `tree-nodes.md`** -- Off-chain WORM + Merkle leaf proofs; bloom miss = silence. -- Fleet replica floors (worm/tree min 3); operator console `:3850`. - -**`overview/01-system.md`** -- Planes: Zapier, web, API, leaf, staff, authz, commercial edge, account-balance, CS/sales/acct, middleware, NATS, workers, archives, chain, fleet. -- One-sentence request path through meter → stamp → watch/events → billing NATS. - -**`overview/11-hostnames.md`** -- NS1 `70.88.205.138`; LXC 510 worker; NATS LXC 511–513 at `10.10.10.21–23`. -- Full service→port→DNS table (portal/api/zap/staff/iam/mw/fleet/git/catalog). -- zappier-edge on `127.0.0.1:13000` (tunnel); note that host `:3000` is “another app”. -- NATS never public; public token edge documented separately in nats-cluster package. - -**`overview/NATS.md` / `03-nats-cluster.md`** -- Central 3-server JetStream; new functions = new addresses; Zapier never connects. - -### STATUS / decisions / NATS lab - -**`packages/zapier-decisions/STATUS.md` (2026-09-12)** -- Working lab on NS1; public doors 8/8 OK; keep+fleet up; test 3-node JetStream on `vmbr1`. -- Product streams replicas=3: `ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE`. -- Clients use `nats://10.10.10.21:4222,...`; host loopback `:4222` unused by clients. -- Open: Zapier Platform push; `MOCK_VERAE=false`; nkeys/mTLS; lan-134; reboot ownership. -- Mock path submit→jobId→wait→receipt proven; live chain seal not proven. - -**Decisions (`packages/zapier-decisions/decisions/`)** -- **D001** — Account-balance (`books.json`) is sole prepaid writer; edge `balanceCents` is cache. -- **D002** — `veraeUserId` join key; public credential = zappier API key; JWT = `tokenRef` server-side. -- **D003** — Access planes are only public HTTP doors (`:3021–3025`); edge on loopback `:3000`. -- **D004** — Leaf cannot touch billing (policy allow-list). -- **D005** — HTTP spawn local-first; `lan-134` disabled. -- **D006** (2026-10-03) — Business SoT → **libSQL**; JetStream = work queues only; **core NATS** for `verae.ts.completed` / `verae.ts.receipt`; Go timestamping DBs stay local; lab `billing.db`/`zappier.db`/`books.json` are single-host stand-ins. - -**`docs-master/MESSAGE-FLOWS.md`** -- Async stamp, wait, idempotent hash, `includeAttached`, multipart, batch Merkle, simulator replay. - -**`docs/plans/phase-gates.md` / `TODO.md`** -- Gates 0–12 largely passed in source/mock; 13–15 (live Verae, private Zapier push, hardening) open. -- Root README still says “9 and 12+ open” in places — stale vs gate logs. - -**`docs/OPEN.md`** -- Needs human: Zapier login/push; live Verae gaps (hash lookup etc.); PDF artwork; LTS/share not live; NATS public bind left loopback (superseded by cluster/token edge notes elsewhere). - ---- - -## 3. Packages (`packages/`) — what each does - -| Package | Role (from README) | -|---------|-------------------| -| **overview** | High-level system book; Zapier never NATS; address routing | -| **docs-master** | Per-module SUMMARY/NATS; MESSAGE-FLOWS; catalog host move | -| **dev-overview** | Designed vs running NATS fabric map; credentials cheat sheet | -| **ui-docs** | Operator/staff/portal screenshot walkthrough + UI-REVIEW.pdf | -| **zapier-decisions** | STATUS, LOG, TODO, D001–D006, NATS bench reports | -| **zapier-user-docs** | Customer guide: signup → stamp → lookup → tree proofs | -| **zappier** (Forgejo: zappier-edge) | Metered API, portal, admin, Stripe, invoices, rate card | -| **zappier-account-balance** | Prepaid SoT (`books.json`); NATS `verae.billing.*` | -| **zappier-customer-service** | CS credits/review UI `:3011` | -| **zappier-sales-pricing** | Per-customer pricing UI `:3012` | -| **zappier-accounting-export** | QuickBooks IIF / CSV export `:3013` | -| **zappier-identity** | Bind/lookup `veraeUserId` ↔ customer (`:3026`) | -| **verae-zapier-middleware** | `/zapier/v1/*`, NATS workers, mock/live chain client `:3100` | -| **verae-zapier** | Full Zapier Platform app (creates/searches/triggers) | -| **verae-activate** | Tiny “Add Numbers” push-first Zapier app | -| **verae-zapier-simulator** | In-process Zapier+trace console `:3847` | -| **verae-access-authz** | Default-deny `verae.access.authz.check` `:3020` | -| **verae-access-web** | Browser portal plane `:3021` | -| **verae-access-api** | Direct customer API plane `:3022` | -| **verae-access-leaf** | S2S leaf plane `:3023` (no billing) | -| **verae-access-zapier** | Zapier HTTPS front door `:3024` | -| **verae-access-staff** | Staff plane UI `:3025` | -| **verae-staff-iam** | Staff users/roles/audit `:3028` | -| **verae-staff-session** | Shared staff cookie login `:3027` | -| **verae-staff-ui** | Shared review HTML template | -| **verae-request-splitter** | Hash vs archive puts (in-process today) | -| **verae-archive-worm** | Bloom WORM off-chain store | -| **verae-archive-aggregator** | Fan-out query + merge replies | -| **verae-tree-node** | Merkle leaf proof WORM (`kind=tree`) | -| **verae-jobs-events** | Mailbox for `jobs.events` `:3030` (passthrough today) | -| **verae-fleet** | Replica floors, spawn, operator console `:3850` | -| **verae-keep** | Crash restart unless paused (`:3860`/`:3861`) | -| **verae-ops** | Docker/Proxmox/metal install + linking runbooks | -| **verae-bootstrap** | Clone all Forgejo modules + host-deps | -| **verae-deploy** | Tag checkout + npm ci on target hosts | -| **verae-proxmox** | LXC worker (510) provision | -| **verae-nats-cluster** | LXC 511–513 JetStream cluster scripts | -| **verae-nats-accounts** | Leaf allow-list + future nkeys sketch | -| **verae-nats-process** | Template for new addressed workers | -| **verae-uptime** | Off-box HTTPS door probes `:3870` | -| **verae-backup** | Tar snapshots under `/SSD2/backups/verae/` | - ---- - -## 4. Extracted system facts (as documented) - -### Services & ports (NS1 lab catalog) - -| Service | Port | Bind (docs) | Public door / DNS | -|---------|------|-------------|-------------------| -| Docs catalog | 443 | Caddy | **zapier.georgelambert.org** | -| Forgejo | 443 / SSH **2223** | Caddy | **git.georgelambert.org** | -| Fleet console | 3850 | 0.0.0.0 | fleet.zapier.georgelambert.org | -| Portal (access-web) | 3021 | 0.0.0.0 | portal.zapier… | -| Customer API | 3022 | 0.0.0.0 | api.zapier… | -| Leaf | 3023 | 0.0.0.0 | leaf.zapier… | -| Zapier plane | 3024 | 0.0.0.0 | **zap.zapier.georgelambert.org** | -| Staff plane | 3025 | 0.0.0.0 | staff.zapier… | -| Authz | 3020 | 0.0.0.0 | internal | -| Identity | 3026 | 0.0.0.0 | id.zapier… | -| Staff session | 3027 | 0.0.0.0 | login.zapier… | -| Staff IAM | 3028 | 0.0.0.0 | iam.zapier… | -| Account-balance | 3010 | 0.0.0.0 | books.zapier… | -| CS / Sales / Acct | 3011 / 3012 / 3013 | 0.0.0.0 | cs / sales / acct | -| Middleware | 3100 | 0.0.0.0 | mw.zapier… | -| jobs.events mailbox | 3030 | 0.0.0.0 | events.zapier… | -| zappier-edge (admin) | **13000** (loopback) or **:3000** in older docs | 127.0.0.1 | tunnel only | -| Keep / watch | 3860 / 3861 | 127.0.0.1 | — | -| Simulator | 3847 | optional | sim.zapier… | -| Uptime status | 3870 | 127.0.0.1 | — | -| Workers | 13200–13602 | 127.0.0.1 | — | -| NATS clients | 4222 | private `10.10.10.21–23` | never public product path | -| NATS cluster routes | 6222 | vmbr1 | never public | -| NATS monitor | 8222 | loopback on nodes | — | -| Optional public NATS edge | 4222 / WSS | token-gated on 138 | nats.zapier… (not for Zapier) | - -**Also documented elsewhere (live product, not this catalog’s primary story):** `zap.verae.ai` / `zappier-edge.service` on `:3040` under `/opt/verae/zappier-edge` — see Conflicts. - -### NATS subjects & streams - -**JetStream product streams (replicas=3):** -`ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE` - -| Subject | Kind | Purpose | -|---------|------|---------| -| `verae.zapier.jobs.watch` | JS work | Poll chain status | -| `verae.zapier.jobs.events` | JS events | Terminal job outcomes → waiters/webhooks | -| `verae.zapier.webhooks.deliver` | JS work | POST Zapier REST Hook | -| `verae.zapier.usage` | optional JS | Usage export | -| `verae.archive.put` | JS | Off-chain put | -| `verae.archive.query` | broadcast | Bloom query | -| `verae.archive.reply.` | reply | Hit-only | -| `verae.billing.statement.get` | req-reply | Statement | -| `verae.billing.balance.get` | req-reply | Balance | -| `verae.billing.balance.adjust` | req-reply | Credits/reloads | -| `verae.billing.usage.recorded` | pub | Meter | -| `verae.billing.payment.recorded` | pub | Payments | -| `verae.billing.credit.applied` | pub | CS credits | -| `verae.access.authz.check` | req-reply | Allow/deny | -| `verae.access.{web,api,zapier,leaf}.*` | ingress | Plane fronts | -| `verae.identity.bind` / `.lookup` | req-reply | Identity map | -| `verae.ts.completed` / `verae.ts.receipt` | **core NATS** (D006 / standing rule) | Job-complete notify — **not** JetStream | - -Client URL (lab): `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` - -### Databases / data stores (documented) - -| Store | Owner | Contents | -|-------|-------|----------| -| **`books.json`** | zappier-account-balance (D001) | Prepaid, credits, usage, payments (lab SoT) | -| **`zappier.db`** | zappier-edge | Auth/sessions/admin (also older monolith docs) | -| **`billing.db`** | live edge / D006 note | Customers, usage_entries (+ hash chain), rates, invoices (138 stand-in) | -| **libSQL (target)** | D006 | Shared business SoT across 3–5 hosts | -| Middleware file JSON / `data/` | middleware | Tenants, webhooks, blobs (MVP; Postgres/Redis Phase 15) | -| WORM / tree blobs | archive-worm, tree-node | Off-chain metadata/files/leaf proofs | -| JetStream store | nats-a/b/c | Durable work/events/archive streams | -| Keep/fleet runtime | `~/verae-*-runtime` | Intent + spawned unit state | -| Backups | `/SSD2/backups/verae/` | Tar snapshots (not private keys) | -| Go `blockchain.db` | timestamping-svc | **Do not relocate** (D006) | - -### Public doors (D003) - -- Browser → `:3021` / portal.zapier… -- Customer API → `:3022` / api.zapier… -- Zapier → `:3024` / zap.zapier… -- Leaf → `:3023` -- Staff → `:3025` (+ iam/login) -- Catalog → zapier.georgelambert.org -- Git → git.georgelambert.org -- Edge admin/Stripe/identity stay loopback (not a public door) - -### Identity model (D002 + access planes) - -- **Public credential:** zappier API key (`x-api-key`); portal signup/regen. -- **Middleware legacy:** `zmw_` / `zmt_` Bearer (internal / pre-composition). -- **Chain auth:** Verae login → JWT held as `tokenRef` server-side; never in Zapier. -- **Join key:** `veraeUserId` (`vu_…`) via `zappier-identity` bind/lookup. -- **Staff:** `verae-staff-iam` roles (owner, cs, sales, accounting, operator, …) + session cookie. -- **Planes:** zapier / web / api / leaf / staff — each must pass `authz.check` before internal subjects; leaf cannot bill. - -### Userflows (documented) - -1. **Signup** → portal → API key (Free). -2. **Connect Zap** → HTTPS access-zapier / edge with API key. -3. **Create Timestamp (async)** → meter → middleware → chain → `jobs.watch` → 202 `{jobId}` → poll → `jobs.events` → REST Hook. -4. **Create and Wait** → same, HTTP holds until event or timeout → `pending`+`jobId`. -5. **Idempotent SHA-256** → original seal / `existing: true`. -6. **Attachments / includeAttached** → archive.put + query/merge. -7. **Batch Merkle** → root on chain; leaves on tree-nodes; lookup via `kinds=["tree"]`. -8. **CS credit / portal reload / sales pricing / accounting export** via `verae.billing.*`. -9. **Simulator** replays 1–7 in-process with fault injection. - ---- - -## 5. Conflicts / outdated claims - -| Topic | Claim A | Claim B | Notes | -|-------|---------|---------|-------| -| Wiring status | Root README: packages “imported, **not yet wired**” | STATUS: working lab, doors 8/8, NATS cluster live | README composition blurb is stale | -| Public auth | Overview / verae-zapier: Bearer **`zmw_`** | Composition / D003 / billing docs: **`x-api-key`** via zappier | Transition incomplete in docs | -| Prepaid SoT | D001: **`books.json`** via account-balance | Live edge memory / D006: **`billing.db`** + **`zappier.db`**; target **libSQL** | Three eras coexist in writing | -| Edge port | D003 / ops: **`:3000`** loopback | Hostnames: **`:13000`**; “`:3000` is another app” | Also live **`zap.verae.ai:3040`** outside this catalog | -| Public Zap host | Catalog: **zap.zapier.georgelambert.org** | Product memory: **zap.verae.ai** | Dual branding / cutover | -| NATS location | Many docs: loopback `:4222` | STATUS / nats-cluster: **10.10.10.21–23**; loopback unused by clients | Hostnames table still lists loopback NATS | -| Job-complete bus | Architecture: **`verae.zapier.jobs.events` on JetStream** | D006 + standing rule: **`verae.ts.*` on core NATS only** | Naming + persistence conflict; do not JetStream job-complete notify | -| Gate progress | README: phases **9 and 12+ open** | phase-gates gate logs: **9 and 12 pass** | Stale README sentence | -| MOCK / live | STATUS: still **MOCK_VERAE** | Some defaults printed as `MOCK_VERAE=false` | Live seal not proven | -| NATS accounts | D004 sketches accounts | STATUS: cluster **without** nkeys; isolation = vmbr1 | Expected | -| Metering owner | Middleware `PLAN_LIMITS` in older overview | Composition: metering on **zappier** only | Correct direction = composition | -| Hash lookup | Features a–m: mock done | Live OpenAPI: **no** hash route; client 501 | Gap documented | -| docs-master diagram | Still shows “NS1 loopback :4222” for JetStream | Cluster cut-over done | Update modules-and-nats diagram | - ---- - -## 6. Source-of-truth files (prefer these) - -| Concern | Prefer | -|---------|--------| -| Current lab status | `packages/zapier-decisions/STATUS.md` | -| Accepted architecture decisions | `packages/zapier-decisions/decisions/D00*.md` (esp. **D003**, **D006**) | -| System planes / hard rules | `overview/01-system.md` | -| Hostnames & ports | `overview/11-hostnames.md` | -| NATS streams (Zapier jobs/webhooks/archive) | `docs/02-architecture/nats-subjects.md` + STATUS streams list | -| Billing subjects | `docs-master/modules-and-nats.md` + `packages/zappier-account-balance/README.md` | -| Message userflows | `docs-master/MESSAGE-FLOWS.md` | -| Composition ownership | `docs/02-architecture/composition.md` | -| Product features a–m | `docs/01-product/features-a-m.md` | -| Ops bring-up | `packages/verae-ops/GETTING-STARTED.md` (+ PDF) | -| Module catalog | `docs-master/README.md` + `packages/*/README.md` | -| Phase implementation order | `docs/plans/phase-gates.md` / `TODO.md` | -| Customer narrative | `packages/zapier-user-docs/` | -| Designed vs running NATS | `packages/dev-overview/` | - -**Standing overrides outside this tree (operator rules / memory):** -- Never JetStream for timestamp job-complete (`verae.ts.completed` / `verae.ts.receipt`). -- Live commercial edge on 138: `/opt/verae/zappier-edge`, `billing.db` + `zappier.db`, service `:3040`, public `zap.verae.ai`. -- D006: migrate business data to shared **libSQL** before multi-host split. - ---- - -## 7. Quick system map (ASCII) - -```text - [Zapier cloud: verae-zapier / activate] - HTTPS only - | - +------------------+------------------+ - | | - access-zapier :3024 (alt) zappier-edge /v1 - | | - +---------> middleware :3100 <--------+ - | \ - chain HTTPS \-- NATS JetStream (jobs/webhooks/archive) - api.veraetime.net | - or MOCK_VERAE workers: poller, webhook, - aggregator, worm×N, tree×N - Browser :3021 API :3022 Leaf :3023 Staff :3025 - \ | / - authz.check :3020 - | - billing.* ---- account-balance (books → libSQL) - identity.* --- zappier-identity - fleet :3850 / keep :3860 (ops) - catalog: zapier.georgelambert.org -``` - ---- - -*End of documentation map. Absolute path of this report: `/tmp/zapier-system-docs-map.md`* diff --git a/plans/qa/2026-10-03-baseline.md b/plans/qa/2026-10-03-baseline.md deleted file mode 100644 index 731b6eb..0000000 --- a/plans/qa/2026-10-03-baseline.md +++ /dev/null @@ -1,344 +0,0 @@ -# QA Baseline — Verae × Zapier Userflows 1–7 vs D006 Multi-Host - -**Role:** QA (evaluate only; no product feature implementation) -**Date:** 2026-10-03 (UTC probe ~06:04–06:07) -**Host:** `NS1.GEORGELAMBERT.ORG` / `70.88.205.138` -**Inputs:** -- Plan: [`../2026-10-03-libsql-multihost-architecture.md`](../2026-10-03-libsql-multihost-architecture.md) -- Arch review: [`../reviews/2026-10-03-arch-review.md`](../reviews/2026-10-03-arch-review.md) (approve-with-changes; A1–A8 merged; full sign-off still open) -- PM board: [`../reviews/2026-10-03-pm-board.md`](../reviews/2026-10-03-pm-board.md) -- Decision: D006 (libSQL business SoT; JetStream = work only; core NATS for `verae.ts.*`) -**Scope:** Readiness of critical userflows **1–7** for multi-host cutover. This is a **baseline** (WP-Q precursor). Per PM §5, formal WP-Q entry requires MG3; this pack documents why that gate is not met and what must change first. - ---- - -## Summary judgment - -Lab doors and digests-only stamp submit on `:3040` → `:3130` work today, but **D006 multi-host readiness fails**: there is **no libSQL**, **no BusinessStore**, and **two live prepaid/ledger writers** (`billing.db` on edge + empty `books.json` behind `:3010`). Userflows 1, 3–6 are not E2E-proven; flow 2 is partially green; flow 7 has only weak 404 evidence for billing deny. Do **not** start Phase 4 host split or treat WP-Q as “in” until single-writer libSQL lands. - ---- - -## Live lab evidence (2026-10-03) - -### Health matrix - -| Port | Role | `/health` | Notes | -|------|------|-----------|-------| -| `:3040` | zappier-edge | **200** `{"ok":true,"role":"zappier-edge"}` | Bind `127.0.0.1`; systemd `zappier-edge.service` | -| `:3100` | middleware (lab/mock) | **200** `mockVerae:true`, `natsConnected:false` | Catalog/default; **not** product upstream | -| `:3130` | middleware (product) | **200** `mockVerae:false`, `natsConnected:true` | Edge `ZAPPIER_UPSTREAM` target | -| `:3010` | zappier-account-balance | **200** `nats:false` | Loopback; **not** a systemd unit; fleet-spawned | -| `:3021`–`:3025` | access planes | **200** each | web / api / leaf / zapier / staff | - -### Env / upstream - -- Edge secrets (`/SSD2/secrets/zap-verae/zappier-edge.env`): `PORT=3040`, **`ZAPPIER_UPSTREAM=http://127.0.0.1:3130`** (confirms arch C11 / plan A1). -- Product MW env: `PORT=3130`, `HOST=127.0.0.1`, `MOCK_VERAE=false`, `NATS_URL=nats://127.0.0.1:4222`. -- Account-balance (pid 4179): cwd under moved stack package; `FLEET_STATE_DIR=.../verae-fleet/data/runtime/account-balance-0`; `NATS_URL` lists `10.10.10.21–23` but health reports **`nats:false`**. - -### Digest-only stamp (flow 2 probe) - -- Digest: `sha256("qa-baseline")` = `edf8f682e42104442bb4637c665f50ada9873fa2fe6958ae00a6c05b2e5181cb` -- `POST /v1/timestamps` with `x-api-key: key-ada`, `x-billing-channel: api`, body `{sha256s:[…]}` → **HTTP 202**, `created:1` then idempotent `existing:1`, credit headers present (`X-Verae-Credits-*`). -- Items-shaped digest-only body also **202**. -- Single `POST /v1/timestamp` → **202** with `jobId`, but receipt `completedAt: "0001-01-01T00:00:00Z"` (placeholder / incomplete receipt shape for async complete). - -### Dual-writer / C16 (flows 1, 4–6) - -| Store | Path | Observation | -|-------|------|-------------| -| Edge ledger | `/opt/verae/zappier-edge/billing.db` | **Exists** (WAL). `usage_entries` count **3** for `cust_1` @ 10 each after probes; `customers.balance_cents` for Ada = **0** | -| Account-balance books | `FLEET_STATE_DIR/.../data/books.json` | **Exists**, empty: `prepaid:{}`, `usage:[]`, `credits:[]`, `payments:[]` | -| Statement HTTP | `GET :3010/statement/cust_1` | **200** `{prepaidCents:0, usage:[], …}` — **does not see** edge `usage_entries` | -| Edge response | batch stamp | `balanceCredits:0`, period spend headers show **30** after three 10-credit meters | - -**C16 confirmed live:** edge writes metered usage into `billing.db`; account-balance serves a separate empty `books.json`. No shared SoT. Splitting hosts now would fork money. - -### Go timestamping DB (C9) - -- `/var/lib/timestamping-svc/data/blockchain.db` **exists** (~31 MB, mtime 2026-09-25). -- **Not modified** by this QA probe (read `ls`/`stat` only). - -### libSQL / BusinessStore / harness - -- No `LIBSQL_URL` / `sqld` / libSQL container observed on 138 for business data. -- Local clone `/Users/marchon/apps/verae/zappier-edge/src/db/`: **no** `business-store.ts` (`NO_BUSINESS_STORE`). -- `verae-zapier-simulator` exists (messaging/trace) but is **not** the WP-P automated harness for flows 1–7. -- Arch checklist still open: `nats-subjects.md` `verae.ts.*` / `ZAPIER_USAGE` non-SoT; published env matrix; GETTING-STARTED SoT language. -- Host `nats stream ls` to `10.10.10.21:4222` **timed out** from this probe (MW still reports `natsConnected:true` via loopback `:4222` proxy). STATUS still lists product stream **`ZAPIER_USAGE`** (C10 risk). - -### Leaf D004 (flow 7 probe) - -- `:3023/health` **200** leaf plane up. -- `POST :3023/v1/billing/adjust` → **404** `not found` (route absent). This is **not** yet an automated policy deny of `verae.billing.balance.adjust` (PM WP-D DoD). - ---- - -## Userflow readiness (1–7) - -| # | Userflow | Today | Blocker vs D006 | -|---|----------|-------|-----------------| -| 1 | Signup → API key → Statement | Partial: seed keys exist (`key-ada`…); statement via `:3010` empty/out of band with edge ledger | Dual SoT; no portal E2E harness; account-balance `nats:false` | -| 2 | Stamp digests single + batch + credit headers | **Mostly green** on lab (`:3040`→`:3130`, 202, headers, idempotent existing) | Still local SQLite meter; zero balance still meters; receipt placeholder on single path; not libSQL | -| 3 | Async complete → Zapier REST Hook | **Not proven** | `completedAt` zero-time on single stamp; no WP-P watch→events→webhooks.deliver E2E; Platform push still open in STATUS | -| 4 | CS credit adjust → portal balance | **Broken / dual path** | Adjust would hit books/`verae.billing.*` while meter hits `billing.db`; nats false on balance service | -| 5 | Sales quote → meter match | **Unproven** | Rate tables live in edge SQLite; sales package not wired to shared SoT | -| 6 | Accounting export consistency | **Fails if run today** | Export cannot reconcile empty books vs non-empty `usage_entries` | -| 7 | Leaf stamp OK / billing deny | Health only | Need authz allow-list tests; 404 ≠ D004 proof | - -**PM QA entry (board §5):** MG3 single writer, regression green, NATS rules verified, harness skeleton, D004 automated, migrator tip report, Go DB untouched note, docs C2/C3/C5/C6 — **not met** (only Go-untouched + partial digests-only + doors up). - ---- - -## a) What to change - -| # | Component | Problem | Evidence | Severity | Type | -|---|-----------|---------|----------|----------|------| -| 1 | Prepaid / ledger SoT (C7/C16) | Two writers: edge `billing.db` meters usage; `:3010` serves empty `books.json` statement | usage_n=3 vs statement `usage:[]`; both processes up | **critical** | design flaw / dual writer | -| 2 | Missing libSQL + BusinessStore (D006 Phases 1–2) | No shared business DB or interface; multi-host impossible without per-host SQLite fork | No `business-store.ts`; no LIBSQL on 138 | **critical** | missing foundation | -| 3 | Account-balance NATS / facade | Health `nats:false` while CS/portal expect `verae.billing.*` | `:3010/health` JSON; env has NATS_URL but not connected | **high** | ops / wiring | -| 4 | Userflow E2E harness (WP-P) | Flows 1–7 not automated; simulator ≠ harness | No `plans/qa` flow reports; simulator README is messaging-only | **high** | missing gate | -| 5 | Async complete / receipts (flow 3) | Single-stamp receipt shows epoch-zero `completedAt`; REST Hook path unproven | curl body `0001-01-01T00:00:00Z` | **high** | incomplete product path | -| 6 | Zero-balance metering (flows 2/4) | Ada `balance_cents=0` still stamps and accumulates period spend | customers row + 202 responses + `X-Verae-Credits-Balance: 0` | **medium** | missing gate / prepaid policy | -| 7 | Dual middleware confusion (C11) | `:3100` mock vs `:3130` product; docs/scripts can point wrong | health JSON both ports; edge env → 3130 | **medium** | doc/ops drift | -| 8 | D004 leaf deny proof (flow 7) | Only 404 on invent-ed HTTP path; no automated NATS/HTTP deny | curl 404; WP-D tests not evidenced on lab | **medium** | missing test | -| 9 | `ZAPIER_USAGE` / nats-subjects docs (C5/C10) | STATUS still lists usage stream as product; catalog subjects omit core-only `verae.ts.*` | STATUS.md streams table; arch checklist open | **medium** | doc drift / SoT confusion | -| 10 | Env matrix / Phase 0 leftovers | Arch full sign-off blocked; GETTING-STARTED may still call books SoT | arch review checklist unchecked items | **medium** | doc freeze incomplete | -| 11 | Account-balance not systemd-managed | Process from fleet overlay / moved path; restart/ownership unclear vs edge unit | no `*account-balance*.service`; pid under `marchon` | **low** | ops hygiene | - -### What is working (preserve) - -- Digests-only batch + single stamp on product edge `:3040` with `key-ada` and credit headers. -- Product upstream correctly set to **`:3130`** (not mock `:3100`). -- Access-plane health on `:3021–3025`. -- Edge split DBs present and WAL: `billing.db` (business tables) + `zappier.db` (auth). -- Idempotent re-stamp (`created:0`, `existing:1`, spend 0). -- Standing rules respected in this probe: **no** JetStream for `verae.ts.*`; **no** writes to `blockchain.db`. -- Plan + A1–A8 + D001 addendum + D006 direction are the right target. - ---- - -## b) What to change it to - -| # | Component | Target behavior | Acceptance criteria | Constraints | -|---|-----------|-----------------|---------------------|-------------| -| 1 | Single prepaid/ledger writer | One authoritative adjust + `recordUsage` path on shared store; other path read-only or stopped | Grep: no prod writes to `books.json` / per-host `billing.db` as SoT; adjust on CS visible on portal statement and edge balance headers | D001 single writer; do not dual-append hash chain | -| 2 | libSQL BusinessStore | `LibsqlBusinessStore` (+ SqliteFile for CI) behind `BusinessStore`; lab migrator from `billing.db` + `books.json` | Contract tests green; migrator tip hash verify; `BUSINESS_STORE=libsql` writes only to libSQL | Auth stays edge-local `zappier.db` (C13); fail closed (503) if writer down | -| 3 | Billing facade | `:3010` (or successor) is libSQL-backed listener for `verae.billing.*` **or** folded into edge HTTP API — pick one (plan Phase 3) | Health `nats:true` (if NATS facade kept); statement usage matches `usage_entries` | Keep subjects stable for CS/sales | -| 4 | Harness 1–7 | Automated runner under simulator/harness producing `plans/qa/` report | Flows 1,2,4 runnable under MOCK; then 3,5,6,7; exit 0 artifact | Zapier cloud push optional for first MOCK pass | -| 5 | Async complete | Real completion times / receipts; `jobs.watch` → `jobs.events` → `webhooks.deliver` | E2E: jobId reaches terminal event + hook delivery (or recorded attempt) | `verae.ts.completed` / `verae.ts.receipt` remain **core NATS only** | -| 6 | Prepaid gate | Metering respects prepaid/credits policy (402 or configured allow) | Seed funded customer; zero-balance path returns expected product code | Do not rewrite historical ledger mid-test | -| 7 | Env matrix | Documented `:3040` / `:3130` product / `:3100` mock / `:3010` facade | STATUS + ops GETTING-STARTED + plan cite same matrix | Do not “fix” smoke by pointing edge at 3100 | -| 8 | D004 | Leaf cannot call balance adjust (HTTP door + NATS authz) | `verae-access-leaf` / authz tests green; live deny ≠ bare 404 | Leaf may still stamp/authz | -| 9 | NATS docs | `nats-subjects.md` Non-streams: `verae.ts.*` core-only; `ZAPIER_USAGE` telemetry non-SoT | Arch checklist boxes checked; STATUS wording demotes usage stream | No JetStream bind for job-complete | -| 10 | Phase 0 close | C2/C3/C5/C6 closed or owned; GETTING-STARTED SoT → libSQL | Arch full sign-off or explicit waivers | — | -| 11 | Balance process ops | systemd (or fleet-documented) unit with clear state dir | `systemctl status` shows balance facade; books path documented until migration | Deploy never rsyncs `*.db*` | - ---- - -## c) How to implement - -Aligned with PM DAG; **do not skip merge gates**. - -```mermaid -flowchart TD - P0[Phase 0 docs/env/NATS freeze] --> P1[Phase 1 BusinessStore TDD] - P1 --> P2[Phase 2 migrator + dual-read] - P2 --> MG2[MG2 writers unlock] - MG2 --> P3[Phase 3 single writer rewire] - P3 --> MG3[MG3 single writer proof] - MG3 --> P4[Phase 4 multi-host LIBSQL_URL] - MG3 --> P5[Phase 5 harness 1-7] - P4 --> P5 - P5 --> MG4[MG4 green] - MG4 --> P6[Phase 6 QA whattochange loop] -``` - -### Phase 0 — Conflict / docs freeze (feeding; ~1–2 days) - -- **Goal:** Close open arch checklist items that confuse operators and agents. -- **Changes:** Publish env matrix; patch `nats-subjects.md`; demote `ZAPIER_USAGE` in STATUS; fix overview `zmw_` / digests examples (C3/C6); GETTING-STARTED SoT → libSQL. -- **Validation:** Arch checklist; `rg` for stale SoT claims. -- **Effort:** 1–2 agent-days (WP-A/B/C). - -### Phase 1 — BusinessStore + libSQL skeleton (critical; WP-F/G/H) - -- **Goal:** Interface + SqliteFile + Libsql drivers with contract tests (including hash-chain). -- **Changes:** `zappier-edge` `src/db/business-store.ts`, contract tests, CI libSQL. -- **Validation:** Jest contract green both drivers (MG1). -- **Effort:** ~3 days serial. - -### Phase 2 — Migrate + dual-read (WP-J/K) - -- **Goal:** Import live `billing.db` + empty/legacy `books.json`; verify chain tip; reads dual-run; **writes still locked**. -- **Validation:** Migrator report archived under `plans/qa/`; MG2. -- **Effort:** ~2 days. - -### Phase 3 — Rewire writers (WP-L/M/N) — only after MG2 - -- **Goal:** Edge meter + account-balance + CS/sales/acct → one BusinessStore; remove second writer. -- **Validation:** Statement usage == ledger; CS adjust visible on portal/edge headers; MG3 grep proof. -- **Effort:** ~3–4 days. - -### Phase 4 — Multi-host (WP-O) — only after MG3 - -- **Goal:** `LIBSQL_URL` primary; edge/MW/CS on separate hosts possible. -- **Validation:** Cross-host adjust/meter/statement same balance. - -### Phase 5 — Harness (WP-P) - -- **Goal:** Automate flows 1–7; first pass MOCK_VERAE OK. -- **Validation:** Report under `plans/qa/`; MG4. - -### Phase 6 — QA loop (WP-Q) - -- **Goal:** Break → whattochange → fix → retest until **two consecutive** full greens (project buffer). - -**Explicit non-goals until gates pass:** JetStream for `verae.ts.*`; editing `blockchain.db`; Litestream multi-writer; multi-host split before MG3. - ---- - -## d) Expected results / still incomplete - -### After implementation (realistic) - -| Metric / behavior | Today (baseline) | Expected after Phases 0–5 | -|-------------------|------------------|---------------------------| -| Business SoT | Split `billing.db` + empty `books.json` | One libSQL `verae_business` | -| Flow 2 digests stamp | 202 + headers on single host SQLite | Same UX on libSQL meter writer | -| Flow 1/4 statement vs meter | Divergent (usage 3 vs `[]`) | Identical customer/usage/balance | -| Flow 3 async → hook | Unproven; zero `completedAt` | Proven under harness (MOCK then optional live) | -| Flow 5/6 quote & export | Unproven / inconsistent | Same rate tables + export = ledger | -| Flow 7 D004 | 404 only | Automated deny | -| Multi-host money | Unsafe | Safe only post-MG3 | -| `blockchain.db` | Untouched | Still untouched | - -### Still missing, incorrect, or incomplete (even after this todo list) - -| Gap | Why it persists | -|-----|-----------------| -| Zapier Platform public `push` + live workspace Zaps | Needs operator Zapier login (STATUS); out of first cutover buffer | -| Live `api.veraetime.net` seal proof beyond lab MW | Creds / OpenAPI gaps (hash lookup 501); separate from libSQL SoT | -| NATS nkeys/mTLS | Cluster isolation today is `vmbr1`, not accounts | -| Portal auth on multi-edge sticky sessions | C13: auth stays local `zappier.db` for v1 | -| libSQL HA / failover RPO | Launch = one primary + backup; not multi-primary | -| Peergos / Go timestamping redesign | Out of scope | -| Full architecture sign-off | Depends on remaining checklist (docs), not only code | - -### Success tiers - -| Tier | Criteria | -|------|----------| -| **Minimum viable (this baseline exit)** | Dual-writer named daily owner; env matrix published; harness skeleton for 1/2/4; no host split | -| **MG3 / single-writer** | libSQL SoT; one adjust path; statement == meter | -| **Production-ready multi-host** | MG3 + WP-O + flows 1–7 twice green; fail-closed meter; D004 automated | -| **Complete** | + Platform push + live Verae seal + nkeys — separate projects | - ---- - -## e) Prioritized todo list with PreReqs - -``` -Item 1 — [P0] — Freeze dual-writer authority for lab (C16 runbook) -Document which process may adjust/meter each day until MG3; forbid CS adjust against books while edge meters billing.db without a sync plan. -PreReqs: (none) - -Item 2 — [P0] — Publish live env matrix (:3040 / :3130 / :3100 / :3010) -Add to STATUS + ops docs; match edge ZAPPIER_UPSTREAM=http://127.0.0.1:3130 evidence. -PreReqs: (none) - -Item 3 — [P0] — BusinessStore interface + failing contract tests (WP-F) -Create business-store.ts + contract suite (customers, usage chain, rates, invoices, balances). -PreReqs: (Item 2) - -Item 4 — [P0] — SqliteFileBusinessStore adapter (WP-G) -Wrap existing edge repos; keep Jest green. -PreReqs: (Item 3) - -Item 5 — [P0] — LibsqlBusinessStore + CI libSQL (WP-H) -Driver + reachable LIBSQL_URL in CI/lab; MG1. -PreReqs: (Item 4) - -Item 6 — [P0] — Migrator billing.db + books.json → libSQL with chain verify (WP-K) -Refuse cutover on verifyChain failure; archive tip report under plans/qa/; MG2. -PreReqs: (Item 5) - -Item 7 — [P0] — Rewire edge meter to BusinessStore only (WP-L) -Remove production writes that treat local billing.db as SoT once flag on. -PreReqs: (Item 6) - -Item 8 — [P0] — Fold or rewire account-balance to libSQL facade (WP-M) -One verae.billing.* listener; empty books.json import-only; nats connected if facade kept. -PreReqs: (Item 7) - -Item 9 — [P1] — Rewire CS / sales / accounting-export to same store (WP-N) -Credit adjust, rates, export read libSQL; MG3 proof. -PreReqs: (Item 8) - -Item 10 — [P1] — Patch nats-subjects + demote ZAPIER_USAGE (C5/C10) -Document verae.ts.* core-only; usage stream telemetry-only in STATUS/catalog. -PreReqs: (none) - -Item 11 — [P1] — Automate D004 leaf billing deny (WP-D) -Tests prove leaf cannot balance.adjust; replace 404-only smoke. -PreReqs: (none) - -Item 12 — [P1] — Userflow harness skeleton flows 1, 2, 4 (WP-P start) -MOCK_VERAE runnable report artifact under plans/qa/. -PreReqs: (Item 9) - -Item 13 — [P1] — Extend harness to flows 3, 5, 6, 7 -Async hook path, quote match, export consistency, leaf deny E2E; MG4. -PreReqs: (Item 12 + Item 11) - -Item 14 — [P2] — Fix async receipt completedAt / completion wiring -Eliminate 0001-01-01 placeholders on product stamp path used by Zapier wait/hook. -PreReqs: (Item 12) - -Item 15 — [P2] — Prepaid zero-balance policy gate -Define and test 402 vs allow-with-debt; seed funded fixture for harness. -PreReqs: (Item 7) - -Item 16 — [P2] — Multi-host LIBSQL_URL topology (WP-O) -Only after MG3; cross-host meter/adjust/statement. -PreReqs: (Item 9) - -Item 17 — [P2] — systemd/fleet hygiene for account-balance facade -Unit or documented fleet runtime; stable state dir until books retired. -PreReqs: (Item 8) - -Item 18 — [P3] — Phase 0 leftover docs (C2/C3/C6, GETTING-STARTED) -Close arch full sign-off checklist items not covered by Item 2/10. -PreReqs: (Item 2 + Item 10) -``` - -**Dependency order (for “Do all”):** -Item 1 ∥ Item 2 ∥ Item 10 ∥ Item 11 → Item 3 → Item 4 → Item 5 → Item 6 → Item 7 → Item 8 → Item 9 → Item 12 → (Item 13 ∥ Item 14 ∥ Item 15) → Item 16 → Item 17; Item 18 can trail after Item 2+10. - -**QA note:** Formal WP-Q churn should wait for **Item 9 (MG3)** + **Item 12**. This baseline pack is the readiness report that unlocks engineering focus on Items 1–9. - ---- - -## Probe command index (reproducible) - -```bash -ssh -i ~/.ssh/id_ed25519 -o IdentitiesOnly=yes marchon@70.88.205.138 \ - 'for p in 3040 3100 3130 3010 3021 3022 3023 3024 3025; do - curl -sS -m 5 http://127.0.0.1:$p/health; echo; done' - -# Digest-only batch (do not use production customer keys outside lab seeds) -DIGEST=$(printf "qa-baseline" | sha256sum | awk "{print \$1}") -curl -sS -D - -X POST http://127.0.0.1:3040/v1/timestamps \ - -H "content-type: application/json" -H "x-api-key: key-ada" \ - -H "x-billing-channel: api" \ - -d "{\"sha256s\":[\"$DIGEST\"]}" - -# Dual-writer check (read-only) -sqlite3 /opt/verae/zappier-edge/billing.db "SELECT COUNT(*) FROM usage_entries;" -curl -sS http://127.0.0.1:3010/statement/cust_1 -# blockchain.db: ls/stat only — never migrate/edit -``` - ---- - -*End of QA baseline. No product code was changed by this review.* diff --git a/plans/qa/2026-10-03-libsql-dryrun.md b/plans/qa/2026-10-03-libsql-dryrun.md deleted file mode 100644 index ecb941e..0000000 --- a/plans/qa/2026-10-03-libsql-dryrun.md +++ /dev/null @@ -1,89 +0,0 @@ -# libSQL migrator dry-run (WP-K) — PASS - -**Date:** 2026-10-03T06:24–06:26Z -**Host:** NS1 `70.88.205.138` (`NS1.GEORGELAMBERT.ORG`) -**Role:** Engineer (safe dry-run only) -**Result:** **PASS** - -## Constraints honored - -- Did **not** modify `/var/lib/timestamping-svc/data/blockchain.db` or any Go timestamping DB (stat-only; mtime 2026-09-25). -- Did **not** set `BUSINESS_STORE=libsql` (or any `LIBSQL_*`) on `zappier-edge.service`. -- Did **not** delete/truncate live `/opt/verae/zappier-edge/billing.db`. -- All migrator I/O under `/tmp/libsql-dryrun-20261003T062452Z/` copies only. - -## Workspace - -| Item | Path | -|------|------| -| Dry-run dir | `/tmp/libsql-dryrun-20261003T062452Z/` | -| Source copy | `…/source-billing.db` (90112 bytes) | -| Dest libSQL file | `…/dest.db` (`file:/tmp/libsql-dryrun-20261003T062452Z/dest.db`, 94208 bytes) | -| Migrator | `/opt/verae/zappier-edge/dist/db/migrate-to-libsql.js` (same content/mtime as `/tmp/zappier-edge-work`) | -| Runner | `…/run-migrate.mjs` (uses opt `node_modules`; chdir to `/opt/verae/zappier-edge`) | -| JSON report | `…/migrate-report.json` | - -`/tmp/zappier-edge-work` had no `node_modules`; used live opt install for `@libsql/client` + `better-sqlite3`. Migrator source identical in both trees. - -## Commands - -```bash -# 1) Checkpoint + consistent copy (as verae-zapier → /tmp dry-run dir) -sudo -u verae-zapier sqlite3 /opt/verae/zappier-edge/billing.db "PRAGMA wal_checkpoint(TRUNCATE);" -# → 0|0|0 -sudo chmod 777 /tmp/libsql-dryrun-20261003T062452Z -sudo -u verae-zapier python3 -c "import sqlite3; s=sqlite3.connect(\"/opt/verae/zappier-edge/billing.db\"); d=sqlite3.connect(\"/tmp/libsql-dryrun-20261003T062452Z/source-billing.db\"); s.backup(d); d.close(); s.close()" -sudo chown marchon:marchon /tmp/libsql-dryrun-20261003T062452Z/source-billing.db - -# 2) Run migrator (node ESM runner → migrateBillingDbToLibsql) -cd /opt/verae/zappier-edge -node /tmp/libsql-dryrun-20261003T062452Z/run-migrate.mjs -``` - -## Row counts - -| Table | Live billing.db | Source copy | Dest libSQL | Match | -|-------|-----------------|-------------|-------------|-------| -| customers | 4 | 4 | 4 | yes | -| usage_entries | 6 | 6 | 6 | yes | -| channel_rates | 108 | 108 | 108 | yes | -| channel_rate_meta | 1 | 1 | 1 | yes | -| tiers | 3 | 3 | 3 | yes | -| price_endpoints | 11 | 11 | 11 | yes | -| invoices | 0 | 0 | 0 | yes | -| invoice_lines | 0 | 0 | 0 | yes | - -Customers on dest: `cust-platform`, `cust_1`, `cust_2`, `cust_3`. - -## Chain / tip hashes - -| Phase | ok | length | tipHash | -|-------|----|--------|---------| -| before (source copy via `verifyUsageLedgerChain`) | true | 6 | `8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5` | -| after (dest via `LibsqlBusinessStore.verifyUsageChain`) | true | 6 | `8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5` | -| live billing.db (post-run read) | — | 6 | `8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5` | - -Genesis `prev_hash` = 64 zeros; tip id = 6. Before tip === after tip; lengths match. - -## Dual-run / safety post-checks - -| Check | Result | -|-------|--------| -| `sourcePreserved: true` (migrator report) | PASS | -| Source copy still 90112 bytes after migrate | PASS | -| Live `billing.db` still 90112 bytes; counts 4/6; same tip | PASS | -| `BUSINESS_STORE` / `LIBSQL_*` absent from unit + not applied | PASS | -| zappier-edge `/health` still 200 | PASS | -| blockchain.db untouched (mtime 2026-09-25, ~31MB) | PASS | -| booksJson | `absent` (path not provided; stub skip) | - -## Pass/fail - -**PASS** — WP-K migrator dry-run on a WAL-checkpointed copy of live `billing.db` → `file:…/dest.db` succeeded with matching row counts and identical usage ledger tip hash. Live service remains on sqlite-file BusinessStore; no cutover. - -## Non-goals (explicitly not done) - -- No `BUSINESS_STORE=libsql` flip -- No rsync/deploy of edge -- No books.json prepaid import (schema stub) -- No JetStream / multi-host work diff --git a/plans/qa/2026-10-03-retest-completedAt-credits.md b/plans/qa/2026-10-03-retest-completedAt-credits.md deleted file mode 100644 index a4de42c..0000000 --- a/plans/qa/2026-10-03-retest-completedAt-credits.md +++ /dev/null @@ -1,86 +0,0 @@ -# QA Retest — completedAt sanitize + durable credit ledger - -**Role:** QA (evaluate + record; implementation already deployed this session) -**Date:** 2026-10-03 (UTC probe ~06:50) -**Host:** `NS1.GEORGELAMBERT.ORG` / `70.88.205.138` -**Prior:** [`2026-10-03-retest-mg3.md`](./2026-10-03-retest-mg3.md) -**Scope:** Retest userflows **2–4** after (1) middleware `sanitizeVeraePayload` strips Go zero `completedAt`, (2) edge `SqliteCreditLedger` + `credit_adjustments` on `billing.db`, (3) facade statement reads that table. -**Constraints honored:** No Go `blockchain.db` writes (mtime still 2026-09-25); no JetStream for `verae.ts.*`; meter appends only from normal stamp probes; intentional CS credit adjusts were lab smoke only (`+100`, `+50` on `cust_1`). - ---- - -## Summary judgment - -**Flow 2 remains PASS.** Digests-only single + batch on `:3040` return **202**, credit headers present, idempotent re-stamp `created:0` / spend **0**. Balance header now shows prepaid **100** (was 0) after durable credit smoke. - -**Flow 3 moves FAIL → PARTIAL.** Single `POST /v1/timestamp` receipts **no longer** carry `completedAt:"0001-01-01T00:00:00Z"` (field omitted on async accept). Full async complete → Zapier REST Hook E2E is still unproven. - -**Flow 4 moves PARTIAL → stronger PARTIAL.** Edge `POST /admin/api/credits` persists to `billing.db.credit_adjustments`; survives edge restart; `:3010/statement/cust_1` returns `credits_n=2`, `prepaidCents=150`, `source:"billing.db"`. CS health still `booksWrites:false`. MG3 still **not** met (sqlite-file, not libSQL; `:3010` `nats:false`; freeze still constrains production NATS adjust). - ---- - -## What shipped this session - -| Item | Where | Evidence | -|------|-------|----------| -| `sanitizeVeraeTime.js` + wire in `veraeClient.request()` | `/opt/verae/verae-zapier-middleware` | Live single stamp body has no `0001-01-01` | -| `SqliteCreditLedger` + `credit_adjustments` DDL | `/opt/verae/zappier-edge` | Table rows; list after restart | -| `BusinessStore.adjustBalanceCredits` logs credit row | sqlite-file + libsql stores | Unit tests green | -| Facade statement reads `credit_adjustments` | `zappier-account-balance` `billing-store.js` | `:3010` credits_n=2 | - ---- - -## Health matrix - -| Port | Role | Result | -|------|------|--------| -| `:3040` | zappier-edge | **200** | -| `:3130` | middleware product | **200** `mockVerae:false`, `natsConnected:true` | -| `:3100` | middleware mock | **200** `mockVerae:true` (do not smoke product here) | -| `:3010` | account-balance facade | **200** `facade:true`, `nats:false`, `billingDb` set | -| `:3011` | CS | **200** `edge` + `booksWrites:false` | -| `:3012` | sales | **200** `booksWrites:false` | -| `:3013` | accounting-export | **200** `booksWrites:false` | - -Go DB: `/var/lib/timestamping-svc/data/blockchain.db` mtime **2026-09-25**, size ~31 MB — **untouched**. - ---- - -## Userflows 2–4 - -| # | Userflow | Score | Evidence | -|---|----------|-------|----------| -| 2 | Stamp digests single + batch + credit headers | **PASS** | Single 202 + headers Spend 10 / Balance 100 / Period 130→150; batch create+idempotent 202; no Go-zero | -| 3 | Async complete → Zapier REST Hook | **PARTIAL** | Accept receipts omit Go-zero `completedAt`; no watch→events→webhook E2E in this probe | -| 4 | CS credit adjust → portal/statement balance | **PARTIAL** | Edge credits 201; facade statement prepaid **150**, credits `[qa-retest-f4, durable-ledger-smoke]`; CS `booksWrites:false`; NATS adjust path still `nats:false` on `:3010` | - ---- - -## MG3 checklist (unchanged overall) - -| Criterion | Result | -|-----------|--------| -| One meter writer (edge BusinessStore → billing.db) | **PASS** (lab) | -| Durable credit log on same SoT | **PASS** (new) | -| One prepaid writer on **libSQL** | **FAIL** — still `BUSINESS_STORE=sqlite-file` | -| Production NATS `balance.adjust` unfrozen | **FAIL** — freeze + `:3010` `nats:false` | -| D004 leaf deny ≠ bare 404 | **not retested** this probe | - -**Verdict:** Do **not** mark MG3 PASS. Lab prepaid/credit history is now durable on `billing.db`; multi-host libSQL cutover and NATS adjust unfreeze remain blockers. - ---- - -## Remaining gaps (short) - -1. Full flow 3: job watch → `verae.ts.completed` (core NATS) → webhook deliver E2E -2. Edge batch still falls through to local mock when upstream path is `/api/batch/timestamp` (middleware only has `/zapier/v1/timestamp/batch`) — product batch authenticity gap -3. MG3 / WP-O: `BUSINESS_STORE=libsql` + freeze lift for adjust -4. Automated harness WP-P for flows 1–7 - ---- - -## Probe digests (this run) - -- Single: `0d7d7500c29a5d7f5804dd2891e7b7e18a55222e59531104d1324defaa87a626` → jobId `4a6536a5-…` -- Batch: `b9a8b9842ed58de6c1cd621f4644547e19803a157f277610211bebc15b9584dd` → created then existing -- Credits: `durable-ledger-smoke` (+100), `qa-retest-f4` (+50) → prepaid **150** diff --git a/plans/qa/2026-10-03-retest-mg3.md b/plans/qa/2026-10-03-retest-mg3.md deleted file mode 100644 index 9352581..0000000 --- a/plans/qa/2026-10-03-retest-mg3.md +++ /dev/null @@ -1,255 +0,0 @@ -# QA Retest — WP-M/N + BusinessStore v2 (MG3 probe) - -**Role:** QA (evaluate only; no product feature implementation) -**Date:** 2026-10-03 (UTC probe ~06:34–06:37) -**Host:** `NS1.GEORGELAMBERT.ORG` / `70.88.205.138` -**Baseline:** [`2026-10-03-baseline.md`](./2026-10-03-baseline.md) -**Prior:** [`2026-10-03-retest-wpl.md`](./2026-10-03-retest-wpl.md) -**Scope:** Post WP-M (account-balance facade) + WP-N (CS/sales/acct `booksWrites:false`) retest against BusinessStore sqlite-file on live edge. Score userflows 1–7; judge MG3 prepaid-writer status. -**Constraints honored:** No DB clears; no intentional ledger adjusts; Go `blockchain.db` **untouched** (mtime still 2026-09-25). Meter appends only from normal stamp probes. - ---- - -## Summary judgment - -**WP-M/N lab progress is real:** `:3010` is a **billing.db facade** (`facade:true`, `source:"billing.db"`); `GET /statement/cust_1` usage **aligns** with edge `usage_entries` (n=8 after this probe); CS `:3011` / sales `:3012` / accounting-export `:3013` all advertise `edge: http://127.0.0.1:3040` and **`booksWrites: false`**; freeze doc is present at [`docs/DUAL-WRITER-FREEZE.md`](file:///opt/verae/zappier-edge/docs/DUAL-WRITER-FREEZE.md); digests-only create+idempotent on `:3040` still **202**; upstream Verae batch with admin JWT still **202**. - -**MG3 is not fully met.** There is **one live meter writer** (zappier-edge `SqliteFileBusinessStore` → `billing.db`). There is **not** yet one prepaid/adjust writer on **libSQL** (`LIBSQL_URL` absent; `BUSINESS_STORE=sqlite-file`). Freeze text still forbids production use of `verae.billing.balance.adjust` until re-pointed. `:3010` health remains `nats:false`. Flow 3 `completedAt` still epoch-zero. D004 leaf deny still **404**, not policy deny. Do **not** start WP-O host split. - ---- - -## Checklist results - -| # | Check | Result | Evidence | -|---|--------|--------|----------| -| 1 | Edge BusinessStore sqlite-file still up; freeze doc at `docs/DUAL-WRITER-FREEZE.md` | **PASS** | Journal (last start `Oct 02 21:27:46`): `BusinessStore driver: sqlite-file`, `Billing customers/usage via SqliteFileBusinessStore (BILLING_DB)`. Secrets: `BUSINESS_STORE=sqlite-file`, `PORT=3040`, `ZAPPIER_UPSTREAM=http://127.0.0.1:3130`. File present: `/opt/verae/zappier-edge/docs/DUAL-WRITER-FREEZE.md` (350 B, authoritative meter+balance writes = edge BusinessStore; `:3010` legacy facade). | -| 2 | `:3010` statement `cust_1` usage aligns with `billing.db` (facade) | **PASS** | Pre-probe: statement usage n=6, `source:"billing.db"`, prepaid 0; sqlite `usage_entries` cust_1 n=6 sum=60. Post-stamp: statement n=8, db n=8 sum=80, same source. Tip after probe `verifyChain` `{"ok":true,"length":8,"tipHash":"abf1c625…639792a"}`. Stale `books.json` still empty (mtime Sep 11) and **not** used for statement. | -| 3 | CS/sales/acct health shows edge URL and `booksWrites: false` | **PASS** | `:3011` CS `{"edge":"http://127.0.0.1:3040","booksWrites":false,"nats":true}`; `:3012` sales same edge + `booksWrites:false`; `:3013` accounting-export same. | -| 4 | Digest stamp flow on `:3040` (create + idempotent) | **PASS** | Digest `sha256("qa-retest-mg3-2026-10-03")`=`d638ddc8…d7b919`. POST1 → **202** `created:1` spend **10**; POST2 → **202** `created:0` `existing:1` spend **0**. Channel `api`. Period spend headers 70 then 80 after single follow-up. | -| 5 | Optional Verae batch with admin token | **PASS** | `POST /auth/login` admin/admin123 → JWT. `POST /api/batch/timestamp` two digests-only items → **202** `jobIds` `1d027dcd-…`, `8db4f651-…`. | -| 6 | Userflows 1–7 scored | See table below | — | -| 7 | whattochange a–e for remaining gaps | See § whattochange | — | -| 8 | MG3: one prepaid writer now? | **PARTIAL / not MG3** | See § MG3 status | - -### Health matrix (this probe) - -| Port | Role | `/health` | Notes | -|------|------|-----------|-------| -| `:3040` | zappier-edge | **200** `role:zappier-edge` | BusinessStore sqlite-file | -| `:3130` | middleware product | **200** `mockVerae:false`, `natsConnected:true` | Edge upstream | -| `:3100` | middleware mock | **200** `mockVerae:true`, `natsConnected:false` | Do not smoke here | -| `:3010` | account-balance | **200** `facade:true`, `d006:true`, `nats:false`, `billingDb:/opt/verae/zappier-edge/billing.db`, `edgeUrl:http://127.0.0.1:3040` | Read facade | -| `:3011` | customer-service | **200** edge + `booksWrites:false` | WP-N | -| `:3012` | sales-pricing | **200** edge + `booksWrites:false` | WP-N | -| `:3013` | accounting-export | **200** edge + `booksWrites:false` | WP-N | -| `:3021`–`:3025` | access planes | **200** | web/api/leaf/zapier/staff | - -### Go DB (C9) - -- `/var/lib/timestamping-svc/data/blockchain.db` — mtime **2026-09-25**, size ~31 MB. **Not modified** (stat only). - ---- - -## Dual-writer / statement vs meter (improved) - -| Store | Observation | -|-------|-------------| -| Edge `billing.db` | WAL; `usage_entries` length **8** after probe; `verifyChain` **ok**; Ada `balance_cents=0` | -| `:3010/statement/cust_1` | **200** usage n=**8**, `source:"billing.db"`, prepaid 0 — **aligns** with ledger | -| Legacy `books.json` | Exists under fleet runtime; empty counts; mtime Sep 11; **not** statement SoT | -| C16 | **Mitigated for reads/meter:** facade reads BusinessStore file; CS/sales/acct claim no books writes. **Not closed for prepaid adjust:** freeze still bans production `balance.adjust`; NATS on `:3010` still `false`; store still sqlite-file not libSQL | - ---- - -## Userflow readiness 1–7 - -| # | Userflow | Score | Evidence / remaining blocker | -|---|----------|-------|------------------------------| -| 1 | Signup → API key → Statement | **PARTIAL** | Seed `key-ada` works; statement now matches `billing.db` (was empty at baseline/WPL). No portal signup E2E harness; prepaid still 0; facade `nats:false`. | -| 2 | Stamp digests single + batch + credit headers | **PASS** | Batch create+idempotent 202; credit headers; chain ok; upstream Verae batch 202. Still sqlite-file (not libSQL); zero-balance still meters. | -| 3 | Async complete → Zapier REST Hook | **FAIL** | Single `POST /v1/timestamp` receipt `completedAt:"0001-01-01T00:00:00Z"` (jobId `a7c2842a-…`); no watch→events→webhook E2E. | -| 4 | CS credit adjust → portal balance | **PARTIAL** | Health shows edge + `booksWrites:false` (WP-N intent). Freeze still says do not use `:3010` adjust for production credits; no adjust E2E in this QA (by design: no intentional ledger writes). | -| 5 | Sales quote → meter match | **PARTIAL** | Sales `:3012` points at edge, `booksWrites:false`. Rate/quote↔meter harness not run; rates still edge SQLite. | -| 6 | Accounting export consistency | **PARTIAL** | Acct `:3013` edge + `booksWrites:false`; statement↔usage now consistent so export *can* reconcile. Export E2E not executed. | -| 7 | Leaf stamp OK / billing deny | **PARTIAL** | `:3023/health` 200; `POST :3023/v1/billing/adjust` → **404** `not found` (≠ D004 automated policy deny). | - -**Vs WPL retest:** flows 1, 4–6 moved from broken/unproven toward **PARTIAL** because facade statement aligns and staff planes advertise no books writes. Flow 2 remains green. Flows 3 and 7 unchanged in failure mode. - ---- - -## MG3 status — is there one prepaid writer now? - -**Short answer: one meter writer, yes; full MG3 prepaid/libSQL writer, no.** - -| MG3 criterion (PM board) | This probe | -|--------------------------|------------| -| One prepaid writer; second paths removed | **Partial.** Meter appends only via edge BusinessStore → `billing.db`. `:3010` is read facade over that file. CS/sales/acct `booksWrites:false`. Stale `books.json` not serving statement. | -| Runtime: one `LIBSQL_URL` serves customers/pricing/usage/invoices/balances | **FAIL.** No `LIBSQL_URL`; `BUSINESS_STORE=sqlite-file`. | -| Grep: no prod SoT via `books.json` / per-host billing.db | **Partial.** Lab still uses host-local `billing.db` as SoT (accepted freeze stand-in until libSQL). `books.json` deprecated for statement but file remains. | -| D004 leaf cannot adjust | **Weak.** 404 only; not automated deny proof. | -| Freeze / adjust path | Freeze **explicitly** still blocks production `verae.billing.balance.adjust` until re-pointed at BusinessStore/libSQL → prepaid **adjust** writer not proven unified. | - -**Verdict:** Treat WP-M/N sqlite-file facade cutover as **accepted for lab read-path + meter SoT alignment**. **Do not mark MG3 PASS.** MG3 still requires libSQL SoT + proven single adjust path (or documented fold of adjust into edge BusinessStore) + D004 tests. - ---- - -## a) What to change - -| # | Component | Problem | Evidence | Severity | Type | -|---|-----------|---------|----------|----------|------| -| 1 | Missing libSQL SoT (D006 / MG3) | Lab still sqlite-file `billing.db`; no `LIBSQL_URL` | secrets `BUSINESS_STORE=sqlite-file`; journal | **critical** | missing foundation | -| 2 | Prepaid adjust path still frozen | Freeze forbids production `balance.adjust`; `:3010` `nats:false` | freeze doc; health JSON | **critical** | incomplete WP-M | -| 3 | Async receipt / flow 3 | `completedAt` epoch-zero on single stamp | curl body `0001-01-01T00:00:00Z` | **high** | incomplete product path | -| 4 | Userflow harness (WP-P) | Flows 1–7 not automated; 4–6 only health-level | this manual retest | **high** | missing gate | -| 5 | D004 leaf deny | 404 ≠ policy deny | curl 404 | **medium** | missing test | -| 6 | Zero-balance metering | Ada `balance_cents=0` still stamps | customers row + Balance header 0 | **medium** | missing prepaid gate | -| 7 | Stale `books.json` artifact | Empty file remains in fleet runtime; confusion risk | path under `verae-fleet/.../books.json` | **low** | ops hygiene | - -### What is working (preserve) - -- Live `BusinessStore driver: sqlite-file` + explicit `BUSINESS_STORE=sqlite-file` in secrets. -- `docs/DUAL-WRITER-FREEZE.md` on live edge. -- `:3010` facade reading `billing.db`; statement == `usage_entries` for `cust_1`. -- CS/sales/acct `booksWrites:false` + edge URL. -- Digests-only batch create+idempotent on `:3040` with credit headers; `verifyChain` ok. -- Product upstream `:3130`; Verae Time `/auth/login` + `/api/batch/timestamp` digests-only. -- Access planes healthy; Go `blockchain.db` untouched; no JetStream used for job-complete in this probe. - ---- - -## b) What to change it to - -| # | Component | Target behavior | Acceptance criteria | Constraints | -|---|-----------|-----------------|---------------------|-------------| -| 1 | libSQL BusinessStore | `BUSINESS_STORE=libsql` + `LIBSQL_URL`; fail closed 503 if writer down | Migrator tip verify archived; live meter only to libSQL | Auth stays `zappier.db` (C13) | -| 2 | Single adjust writer | CS adjust / `verae.billing.balance.adjust` → same BusinessStore; unfreeze after proof | Adjust visible on statement + edge balance headers; freeze doc updated | D001; do not dual-append chain; no books.json writes | -| 3 | Async complete | Real `completedAt`; hook delivery path | E2E jobId → terminal event (+ hook attempt) | `verae.ts.*` **core NATS only** | -| 4 | Harness 1–7 | Automated report under `plans/qa/` | Flows 1,2,4 MOCK then 3,5,6,7; exit 0 | — | -| 5 | D004 | Automated leaf deny | Tests green; live deny ≠ bare 404 | Leaf may still stamp | -| 6 | Prepaid gate | Documented 402 vs allow-with-debt | Seed funded + zero-balance fixture | Do not rewrite historical ledger | -| 7 | Retire books.json | Import-only / archived; not in runtime SoT path | Ops note; facade never falls back to books for money | — | - ---- - -## c) How to implement - -Aligned with PM board; **do not skip merge gates**. - -```mermaid -flowchart TD - WPM_done[WP-M facade reads billing.db] --> WPN_done[WP-N booksWrites false] - WPN_done --> AdjustProof[Prove adjust via BusinessStore / NATS] - AdjustProof --> LibSQL[WP-K/O libSQL cutover + verifyChain] - LibSQL --> MG3[MG3 PASS: one prepaid writer on libSQL] - MG3 --> Harness[WP-P harness 1-7] - MG3 --> Async[Fix completedAt / hook path] - Harness --> MG4[MG4] -``` - -### Phase A — Close adjust writer (hours–days) - -- Point `:3010` NATS adjust (or fold into edge HTTP) at BusinessStore; connect NATS or document HTTP-only facade. -- Update freeze doc when adjust is safe; keep meter-only rule until then. -- Prove: CS adjust → statement prepaid/credits change == edge headers (lab only). - -### Phase B — MG3 libSQL - -- Migrator `billing.db` → libSQL; refuse on `verifyChain` fail; archive tip under `plans/qa/`. -- Flip `BUSINESS_STORE=libsql`; facade/`BILLING_DB` read path follows libSQL or HTTP billing API. -- Grep proof: no production money writes to `books.json` or private per-host SoT. - -### Phase C — Product paths + harness - -- Fix single-stamp `completedAt`; prove flow 3 under harness (MOCK first). -- Automate flows 1–7; D004 deny tests; prepaid policy fixture. - -**Validation for this retest exit:** checklist 1–5 green ⇒ WP-M/N sqlite-file facade **accepted**. MG3 / libSQL / adjust proof remain open — not deploy regressions. - ---- - -## d) Expected results — and what remains incomplete - -### After planned changes - -| Metric / behavior | Today (this retest) | Expected after Phases A–C / MG3 | -|-------------------|---------------------|----------------------------------| -| BusinessStore on edge | sqlite-file live | libSQL primary | -| Statement vs meter | **Aligned** (n=8) | Same on libSQL | -| CS/sales/acct booksWrites | false | false + proven adjust/export | -| Freeze doc | Present; adjust banned | Present; adjust allowed via single writer | -| Flow 3 `completedAt` | `0001-01-01…` | Real completion time | -| Flow 7 D004 | 404 only | Automated deny | -| `blockchain.db` | Untouched | Still untouched | - -### Still missing even after those todos - -| Gap | Why | -|-----|-----| -| Zapier Platform live push / workspace Zaps | Operator Zapier login; out of cutover buffer | -| NATS nkeys/mTLS | Lab isolation is network, not accounts | -| libSQL HA / multi-primary | Launch = one primary + backup | -| Peergos / Go timestamping redesign | Out of scope | - -### Success tiers - -| Tier | Status after this retest | -|------|--------------------------| -| Sqlite-file BusinessStore + freeze doc | **Met** | -| WP-M read facade (`statement` == ledger) | **Met** | -| WP-N health (`booksWrites:false` + edge URL) | **Met** | -| MG3 single prepaid writer on libSQL | **Not met** | -| WP-Q / double userflow green | **Not met** | - ---- - -## e) Prioritized todo list - -``` -Item 1 — [P0] — Prove prepaid adjust through BusinessStore (finish WP-M write path) -Wire balance.adjust (NATS and/or edge HTTP) to same store meters use; update freeze when safe; keep books.json read-only/archived. -PreReqs: (none for lab sqlite-file; libSQL preferred) - -Item 2 — [P0] — libSQL cutover with verifyChain gate (WP-K → MG3) -Refuse cutover if chain fails; archive tip under plans/qa/; set BUSINESS_STORE=libsql + LIBSQL_URL. -PreReqs: (Item 1 or explicit waiver that meter-only sqlite-file remains until cutover day) - -Item 3 — [P0] — MG3 proof artifact -Grep + runtime proof: one writer; statement/adjust/meter same tip; no books.json SoT. -PreReqs: (Item 2) - -Item 4 — [P1] — Fix single-stamp completedAt / async receipt (flow 3) -Replace 0001-01-01 placeholders on product path used by Zapier wait/hook; core NATS only for verae.ts.*. -PreReqs: (none) - -Item 5 — [P1] — Automate D004 leaf billing deny (WP-D) -Replace 404-only smoke with authz allow-list tests. -PreReqs: (none) - -Item 6 — [P1] — Userflow harness flows 1, 2, 4 then 3, 5, 6, 7 (WP-P) -MOCK_VERAE report under plans/qa/; MG4. -PreReqs: (Item 3 + Item 5) - -Item 7 — [P2] — Prepaid zero-balance policy gate -Define 402 vs allow-with-debt; seed funded fixture. -PreReqs: (Item 1) - -Item 8 — [P2] — Multi-host LIBSQL_URL topology (WP-O) -Only after MG3 PASS. -PreReqs: (Item 3) -``` - -**Dependency order (Do all):** Item 4 ∥ Item 5 → Item 1 → Item 2 → Item 3 → Item 6 → Item 7 → Item 8. - ---- - -## Probe artifacts (raw) - -- Batch digest: `d638ddc81b601d6f025db6b982f42ce5abaee5aba0b6b2d496c57d764ed7b919` (`qa-retest-mg3-2026-10-03`) -- Batch jobId: `231a1233-8a5c-49ee-8139-de8625039fe6` -- Single digest: `f94d25942a900c8ca017aa5f4a1ddfbc071a0cc5993a575c1e058d707fdeb28b` (`qa-single-mg3`) -- Single jobId: `a7c2842a-13b0-4196-a14b-4ec0c60f7013` (`completedAt` epoch-zero) -- Chain tip after probe: `abf1c6253f4ef38fd42e06ca7a167883a64bac0450d0c94454318fd0c639792a` (length 8) -- Upstream Verae jobIds: `1d027dcd-e976-4ca7-bb33-46f8275fe2b8`, `8db4f651-ad06-476c-bc8d-5356bc079710` -- Statement post-probe: `usage_n=8`, `source=billing.db`, `prepaidCents=0` - -**QA note:** WP-M/N facade + `booksWrites:false` deploy is **accepted** for lab read/meter alignment. **MG3 remains open** until libSQL + proven single prepaid adjust writer. Formal WP-Q churn still waits on MG3 + harness skeleton. diff --git a/plans/qa/2026-10-03-retest-wpl.md b/plans/qa/2026-10-03-retest-wpl.md deleted file mode 100644 index f730fa3..0000000 --- a/plans/qa/2026-10-03-retest-wpl.md +++ /dev/null @@ -1,243 +0,0 @@ -# QA Retest — BusinessStore sqlite-file deploy (WP-L precursor) - -**Role:** QA (evaluate only; no product feature implementation) -**Date:** 2026-10-03 (UTC probe ~06:21–06:24) -**Host:** `NS1.GEORGELAMBERT.ORG` / `70.88.205.138` -**Baseline:** [`2026-10-03-baseline.md`](./2026-10-03-baseline.md) -**Scope:** Retest after live edge deploy of `SqliteFileBusinessStore` (`BusinessStore driver: sqlite-file`). Confirm checklist items; compare userflows 1–7 vs baseline; emit whattochange a–e for remaining breaks. -**Constraints honored:** No writes to Go timestamping DBs; `billing.db` not cleared (read-only verify + normal meter appends from stamp probes only). - ---- - -## Summary judgment - -**WP-G / sqlite-file path on live edge is green:** journal logs `BusinessStore driver: sqlite-file`, digests-only batch on `:3040` with `key-ada` returns **202** then idempotent **existing/free**, and `usage_entries` `verifyChain` is **ok** (length 6 after this probe). Health matrix `:3040/:3130/:3010/:3100` all **200**. Upstream `api.veraetime.net` digests-only batch still **202**. - -**Still not MG3 / multi-host ready:** dedicated `DUAL-WRITER-FREEZE.md` is **missing** (freeze text only in `docs/ENV-MATRIX.md`); `:3010` statement remains empty vs non-empty edge ledger (**dual SoT / C16 still live**); account-balance `nats:false`; single-stamp `completedAt` still epoch-zero; flows 1, 3–7 unchanged vs baseline readiness. Do **not** treat WP-Q as entered or split hosts. - ---- - -## Checklist results - -| # | Check | Result | Evidence | -|---|--------|--------|----------| -| 1 | journal shows `BusinessStore driver: sqlite-file` | **PASS** | `journalctl -u zappier-edge.service`: `Oct 02 21:21:15 … BusinessStore driver: sqlite-file` and `Billing customers/usage via SqliteFileBusinessStore (BILLING_DB)`. Process env has `PORT=3040`, `ZAPPIER_UPSTREAM=http://127.0.0.1:3130`; `BUSINESS_STORE` not explicitly set (defaults to sqlite-file). | -| 2 | `POST /v1/timestamps` digests-only `key-ada` → 202; re-POST same hash → existing/free | **PASS** | Digest `sha256("qa-retest-wpl-2026-10-03")` = `5bdeb622…eacbd`. POST1 → **HTTP 202**, `created:1`, `existing:0`, `X-Verae-Credits-Spend: 10`. POST2 → **HTTP 202**, `created:0`, `existing:1`, `totalCredits:0`, `X-Verae-Credits-Spend: 0`. Channel `api`. | -| 3 | usage chain `verifyChain` ok on `billing.db` | **PASS** | Readonly `SqliteUsageRepo.verifyChain()` on `/opt/verae/zappier-edge/billing.db` → `{"ok":true,"length":6,"tipHash":"8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5"}`. WAL. Ada (`cust_1`) `balance_cents=0`; usage rollup `cust_1` n=6 sum=60 credits. | -| 4 | `DUAL-WRITER-FREEZE.md` present | **FAIL** | `ls` → no such file under `/opt/verae/zappier-edge/` or `docs/`. Freeze **policy text exists** inside [`docs/ENV-MATRIX.md`](file:///opt/verae/zappier-edge/docs/ENV-MATRIX.md) (“Dual-writer freeze (2026-10-03): Meter and balance writes go only through zappier-edge → BusinessStore…”). Checklist asked for the named file — **not present**. | -| 5 | health matrix `:3040/:3130/:3010/:3100` | **PASS** | See table below. Access planes `:3021–3025` also **200**. | -| 6 | Optional: `api.veraetime.net` login + `POST /api/batch/timestamp` two digests | **PASS** | Login `admin`/`admin123` → JWT. Batch digests-only two items → **HTTP 202** `{"jobIds":["fa1838fc-…","930de378-…"]}`. Confirms upstream still digests-only / batch path. | - -### Health matrix (this probe) - -| Port | Role | `/health` | Notes | -|------|------|-----------|-------| -| `:3040` | zappier-edge | **200** `{"ok":true,"role":"zappier-edge"}` | BusinessStore sqlite-file | -| `:3130` | middleware product | **200** `mockVerae:false`, `natsConnected:true` | Edge upstream | -| `:3010` | account-balance | **200** `nats:false` | Statement empty; dual path | -| `:3100` | middleware mock | **200** `mockVerae:true`, `natsConnected:false` | Do not smoke here | - -### Go DB (C9) - -- `/var/lib/timestamping-svc/data/blockchain.db` — mtime **2026-09-25**, size ~31 MB. **Not modified** by this QA (stat/ls only). - ---- - -## Dual-writer / statement vs meter (still broken) - -| Store | Observation | -|-------|-------------| -| Edge `billing.db` | `usage_entries` length **6**, chain tip ok; period spend headers **60** after probes | -| `:3010/statement/cust_1` | **200** `{prepaidCents:0, usage:[], credits:[], payments:[]}` — **does not see** edge usage | -| C16 | **Still live:** meter writer = edge BusinessStore/sqlite-file; facade books path still empty / out of band | - -Sqlite-file deploy moved edge metering behind `BusinessStore`, but **did not** eliminate the second empty ledger behind account-balance. - ---- - -## Userflow readiness 1–7 vs baseline - -| # | Userflow | Baseline | This retest | Remaining blocker | -|---|----------|----------|-------------|-------------------| -| 1 | Signup → API key → Statement | Partial; statement empty vs edge | **Unchanged / still blocked** | Seed keys work; `:3010` statement still `usage:[]` while edge has 6 rows; no portal E2E harness; dual SoT | -| 2 | Stamp digests single + batch + credit headers | Mostly green | **Improved evidence / still green** | sqlite-file meter + verifyChain ok; still local SQLite (not libSQL); zero balance still meters; not multi-host | -| 3 | Async complete → Zapier REST Hook | Not proven | **Still broken / unproven** | Single `POST /v1/timestamp` receipt `completedAt:"0001-01-01T00:00:00Z"` (jobId `6e925fbc-…`); no watch→events→webhook E2E | -| 4 | CS credit adjust → portal balance | Broken / dual path | **Still broken** | Adjust would hit `:3010`/books while meter hits BusinessStore/`billing.db`; `nats:false` | -| 5 | Sales quote → meter match | Unproven | **Still unproven** | Rates still edge SQLite; no shared SoT / harness | -| 6 | Accounting export consistency | Fails if run | **Still fails** | Export cannot reconcile empty statement/books vs `usage_entries` n=6 | -| 7 | Leaf stamp OK / billing deny | Health + 404 only | **Unchanged** | `:3023/health` 200; `POST :3023/v1/billing/adjust` → **404** `not found` (≠ D004 automated deny) | - -**PM QA entry (MG3):** still **not met**. Progress vs baseline: BusinessStore + SqliteFile **on live edge**, chain verify green, ENV-MATRIX published, digests-only + upstream batch reconfirmed. Missing for MG3: libSQL cutover, single writer proof (statement == meter), account-balance fold/rewire, harness, D004 tests, dedicated freeze doc filename if still required by ops. - ---- - -## a) What to change - -| # | Component | Problem | Evidence | Severity | Type | -|---|-----------|---------|----------|----------|------| -| 1 | Dual prepaid/ledger path (C16) | Edge BusinessStore meters `billing.db`; `:3010` statement still empty separate store | statement `usage:[]` vs verifyChain length 6; both services up | **critical** | design flaw / dual writer | -| 2 | Missing libSQL SoT (D006) | Lab still sqlite-file only; no `LIBSQL_URL` in edge environ | environ grep; journal `sqlite-file` | **critical** | missing foundation | -| 3 | `DUAL-WRITER-FREEZE.md` filename | Checklist/ops expect named freeze doc; only ENV-MATRIX paragraph exists | `ls` missing file; ENV-MATRIX has freeze text | **medium** | doc drift / incomplete deliverable | -| 4 | Account-balance NATS / facade | `nats:false`; not libSQL-backed; not single-writer | health JSON | **high** | ops / wiring | -| 5 | Async receipt / flow 3 | `completedAt` epoch-zero on single stamp | curl body `0001-01-01T00:00:00Z` | **high** | incomplete product path | -| 6 | Userflow harness (WP-P) | Flows 1–7 not automated | No harness report artifact beyond this manual retest | **high** | missing gate | -| 7 | Zero-balance metering | Ada `balance_cents=0` still stamps | customers row + 202 + Balance header 0 | **medium** | missing prepaid gate | -| 8 | D004 leaf deny | 404 ≠ policy deny | curl 404 on invent-ed path | **medium** | missing test | -| 9 | `BUSINESS_STORE` not explicit in secrets | Relies on code default; harder to audit live | environ lacks `BUSINESS_STORE=` | **low** | ops hygiene | - -### What is working (preserve) - -- Live `BusinessStore driver: sqlite-file` + SqliteFileBusinessStore over `BILLING_DB`. -- Digests-only batch `:3040` → `:3130` with `key-ada`, credit headers, idempotent existing/free. -- `verifyChain` ok after meter appends (hash chain intact). -- Product upstream `:3130` (not mock `:3100`); ENV-MATRIX documents the port roles + freeze policy. -- Upstream Verae Time digests-only batch authenticated path. -- Access planes healthy; Go `blockchain.db` untouched; no JetStream used for job-complete in this probe. - ---- - -## b) What to change it to - -| # | Component | Target behavior | Acceptance criteria | Constraints | -|---|-----------|-----------------|---------------------|-------------| -| 1 | Single writer | One authoritative meter + adjust path; statement usage == `usage_entries` | After CS adjust + stamp, `:3010` or successor statement matches edge/libSQL tip; no empty books SoT | D001/D006; do not dual-append chain | -| 2 | libSQL BusinessStore | `BUSINESS_STORE=libsql` + `LIBSQL_URL`; fail closed 503 if writer down | Contract tests; migrator tip verify; live meter only to libSQL | Auth stays `zappier.db` (C13) | -| 3 | Freeze doc | Publish `docs/DUAL-WRITER-FREEZE.md` (or update checklist to ENV-MATRIX as SoT) | File present on live `/opt/verae/zappier-edge/docs/` **or** QA checklist amended | Do not invent conflicting writers | -| 4 | Billing facade | Fold into edge HTTP BusinessStore **or** libSQL-backed NATS listener | Health `nats:true` if kept; statement == ledger | Stable `verae.billing.*` subjects | -| 5 | Async complete | Real `completedAt`; hook delivery path | E2E jobId → terminal event (+ hook attempt) | `verae.ts.*` **core NATS only** | -| 6 | Harness 1–7 | Automated report under `plans/qa/` | Flows 1,2,4 MOCK then 3,5,6,7; exit 0 | — | -| 7 | Prepaid gate | Documented 402 vs allow-with-debt | Seed funded + zero-balance fixture tests | Do not rewrite historical ledger | -| 8 | D004 | Automated leaf deny | Tests green; live deny ≠ bare 404 | Leaf may still stamp | -| 9 | Explicit env | `BUSINESS_STORE=sqlite-file` (then `libsql`) in secrets | `systemctl`/environ shows flag | Deploy never rsyncs `*.db*` | - ---- - -## c) How to implement - -Aligned with PM board; **do not skip merge gates**. - -```mermaid -flowchart TD - DocFix[Publish DUAL-WRITER-FREEZE.md or amend checklist] --> MG2 - EnvFlag[Explicit BUSINESS_STORE in secrets] --> MG2 - MG2[MG2 writers unlock after migrator] --> WP_L_done[Confirm WP-L meter-only BusinessStore] - WP_L_done --> WP_M[WP-M fold/rewire account-balance] - WP_M --> WP_N[WP-N CS/sales/acct] - WP_N --> MG3[MG3 single writer proof] - MG3 --> Harness[WP-P harness 1-7] - MG3 --> Async[Fix completedAt / hook path] - Harness --> MG4[MG4] -``` - -### Phase A — Doc / ops closeout (hours) - -- Add `/opt/verae/zappier-edge/docs/DUAL-WRITER-FREEZE.md` (content can mirror ENV-MATRIX freeze paragraph + owner + daily rule). -- Set `BUSINESS_STORE=sqlite-file` explicitly in `/SSD2/secrets/zap-verae/zappier-edge.env`; restart; confirm journal + environ. - -### Phase B — MG2 → MG3 (days; blocked on libSQL migrator if not done) - -- Migrator `billing.db` (+ books) → libSQL; refuse on `verifyChain` fail; archive tip under `plans/qa/`. -- Rewire/stop second writer; statement == meter proof. - -### Phase C — Product paths + harness - -- Fix single-stamp receipt `completedAt`; prove flow 3 under harness (MOCK first). -- Automate flows 1–7; D004 deny tests; prepaid policy fixture. - -**Validation for this retest exit:** checklist items 1–3,5–6 green is enough to call **sqlite-file deploy accepted**; item 4 and dual-SoT remain open work, not deploy regressions. - ---- - -## d) Expected results — and what remains incomplete - -### After planned changes - -| Metric / behavior | Today (this retest) | Expected after Phases A–C / MG3 | -|-------------------|---------------------|----------------------------------| -| BusinessStore on edge | sqlite-file live | libSQL primary | -| Flow 2 digests stamp | 202 + headers + chain ok | Same UX on libSQL | -| Statement vs meter | Divergent (`[]` vs n=6) | Identical | -| `DUAL-WRITER-FREEZE.md` | Missing file | Present (or checklist SoT = ENV-MATRIX) | -| Flow 3 `completedAt` | `0001-01-01…` | Real completion time | -| Flow 7 D004 | 404 only | Automated deny | -| `blockchain.db` | Untouched | Still untouched | - -### Still missing even after those todos - -| Gap | Why | -|-----|-----| -| Zapier Platform live push / workspace Zaps | Operator Zapier login; out of cutover buffer | -| NATS nkeys/mTLS | Lab isolation is network, not accounts | -| libSQL HA / multi-primary | Launch = one primary + backup | -| Peergos / Go timestamping redesign | Out of scope | -| Full arch sign-off leftovers (C2/C3/C6…) | Docs project parallel | - -### Success tiers - -| Tier | Status after this retest | -|------|--------------------------| -| Sqlite-file deploy smoke | **Met** (journal, 202/idempotent, verifyChain, health, upstream optional) | -| Dual-writer freeze artifact | **Partial** (policy in ENV-MATRIX; named file missing) | -| MG3 single-writer | **Not met** | -| WP-Q / double userflow green | **Not met** | - ---- - -## e) Prioritized todo list - -``` -Item 1 — [P0] — Publish docs/DUAL-WRITER-FREEZE.md on live edge (or amend QA checklist) -Ship named freeze runbook to /opt/verae/zappier-edge/docs/ mirroring ENV-MATRIX authority rule, or formally designate ENV-MATRIX as the freeze SoT. -PreReqs: (none) - -Item 2 — [P0] — Explicit BUSINESS_STORE=sqlite-file in zappier-edge.env -Add flag to secrets; restart; confirm environ + journal; keep BILLING_DB path clear. -PreReqs: (none) - -Item 3 — [P0] — Complete migrator + libSQL cutover with verifyChain gate (WP-K / MG2) -Refuse cutover if chain fails; archive tip report under plans/qa/. -PreReqs: (Item 2) - -Item 4 — [P0] — Fold or rewire account-balance to BusinessStore/libSQL (WP-M) -Eliminate empty statement SoT; nats connected if facade kept; statement usage == ledger. -PreReqs: (Item 3) - -Item 5 — [P0] — CS/sales/accounting-export same store + MG3 proof (WP-N) -Adjust visible on statement and edge balance headers. -PreReqs: (Item 4) - -Item 6 — [P1] — Fix single-stamp completedAt / async receipt wiring (flow 3) -Replace 0001-01-01 placeholders on product path used by Zapier wait/hook. -PreReqs: (none) - -Item 7 — [P1] — Automate D004 leaf billing deny (WP-D) -Replace 404-only smoke with authz allow-list tests. -PreReqs: (none) - -Item 8 — [P1] — Userflow harness skeleton flows 1, 2, 4 then 3, 5, 6, 7 (WP-P) -MOCK_VERAE report artifact under plans/qa/; MG4. -PreReqs: (Item 5 + Item 7) - -Item 9 — [P2] — Prepaid zero-balance policy gate -Define 402 vs allow-with-debt; seed funded fixture. -PreReqs: (Item 4) - -Item 10 — [P2] — Multi-host LIBSQL_URL topology (WP-O) -Only after MG3. -PreReqs: (Item 5) -``` - -**Dependency order (Do all):** Item 1 ∥ Item 2 ∥ Item 6 ∥ Item 7 → Item 3 → Item 4 → Item 5 → Item 8 → Item 9 → Item 10. - ---- - -## Probe artifacts (raw) - -- Batch digest: `5bdeb622aae9311ae2e9a5f6cea85ad615711f119a598c1ad212b6b03a4eacbd` (`qa-retest-wpl-2026-10-03`) -- Batch jobId: `d012fb8f-ed92-448d-b9d2-afe27776c11a` -- Single digest: `2a36132dd089f1231764fb37eac38999cf1531e008c28a797ff95c94f6703525` (`qa-single-wpl`) -- Single jobId: `6e925fbc-9b89-4704-8f7f-5f8793ce5b86` -- Chain tip after probe: `8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5` (length 6) -- Upstream jobIds: `fa1838fc-b6b5-40c6-a05f-93439f2c0c0c`, `930de378-994b-429b-a713-9fa27af0965f` - -**QA note:** Sqlite-file BusinessStore deploy is **accepted** for lab meter path. Dual-writer / MG3 / harness remain the critical path. Formal WP-Q churn still waits on MG3 + harness skeleton. diff --git a/plans/reviews/2026-10-03-arch-review.md b/plans/reviews/2026-10-03-arch-review.md deleted file mode 100644 index 7ee0b61..0000000 --- a/plans/reviews/2026-10-03-arch-review.md +++ /dev/null @@ -1,220 +0,0 @@ -# Architecture Review — Verae × Zapier Multi-Host / libSQL Plan - -**Plan under review:** `plans/2026-10-03-libsql-multihost-architecture.md` -**Reviewer role:** Architecture -**Date:** 2026-10-03 -**Inputs:** plan; `/tmp/zapier-system-docs-map.md` (mirrored `plans/2026-10-03-system-docs-map.md`); D001–D006; catalog `composition.md`, `nats-subjects.md`, `overview/01-system.md`, `packages/verae-ops/GETTING-STARTED.md`; live 138 `/opt/verae/zappier-edge` (`billing.db` + `zappier.db`, `zappier-edge.service` on `127.0.0.1:3040`). - ---- - -## Verdict - -**approve-with-changes** - -Direction is correct and aligned with **D006**: libSQL as business SoT, JetStream for work queues only, core NATS for `verae.ts.completed` / `verae.ts.receipt`, digests-only stamps, Go `blockchain.db` out of scope. The plan is **not** blocked on SoT choice. - -It is **not** a clean approve: multi-host money durability, single-writer sequencing, stream/subject demotions, live port matrix, and decision-doc amendments are underspecified. Build of Phases 1–2 (interface + migrator skeleton) may start **after** the amendments in §Required plan amendments are merged into the plan (and D001 text is patched). Do **not** start Phase 3 rewire or Phase 4 multi-host split until those amendments are in the plan and the conflict register rows below are closed or explicitly deferred with owner + date. - ---- - -## Expanded conflict register - -Inherited from plan §3 (C1–C9) plus new findings from catalog + live 138. - -| ID | Conflict | Severity | Resolution (required) | -|----|----------|----------|------------------------| -| C1 | D001 `books.json` SoT vs live `billing.db` vs D006 libSQL | High | **libSQL is SoT**; migrate books + billing.db; amend D001 file to point at D006 (not only the plan narrative) | -| C2 | README “imported, not yet wired” vs STATUS lab up | Med | STATUS + live doors = operational truth; patch README in Phase 0 | -| C3 | overview `zmw_` Bearer vs composition `x-api-key` | Med | Public = `x-api-key` (D002/composition); fix overview/models | -| C4 | Edge ports `:3000` / `:13000` / live `:3040` (`zap.verae.ai`) | High | Publish env matrix; lab doors proxy loopback; product edge remains `127.0.0.1:3040` until deliberately moved | -| C5 | JetStream `jobs.events` vs ban on JetStream for job-complete | High | **Different subjects:** `verae.zapier.jobs.events` (Zapier pipeline, JetStream OK) ≠ `verae.ts.*` (core NATS only). Document both in nats-subjects + standing rule | -| C6 | TimestampRequest “Hello world” data vs digests-only | Med | Digests-only + multihash batch; sync catalog models to live swagger | -| C7 | Dual prepaid writers (edge SQLite + account-balance) | High | One writer API on libSQL; edge caches only; deprecate/fold mailbox | -| C8 | Per-host SQLite on 3–5 machines | High | Forbidden for business data; libSQL shared | -| C9 | Go `blockchain.db` in scope creep | Blocker if violated | **Out of scope** — do not touch | -| **C10** | Catalog/STATUS stream **`ZAPIER_USAGE`** vs libSQL usage ledger | High | Demote `ZAPIER_USAGE` / `verae.zapier.usage` to **telemetry/export only** (or remove). Must never be treated as money/usage SoT. Plan §5 must say this explicitly | -| **C11** | Live dual middleware: catalog/ops `:3100` (MOCK, natsConnected=false) vs edge env `ZAPPIER_UPSTREAM=http://127.0.0.1:3130` (live mockVerae=false, natsConnected=true) | High | Env matrix must list both; product smoke uses **3130**; do not “fix” by pointing edge at 3100 without intent | -| **C12** | D001 decision file still says `books.json` is sole prepaid writer with no D006 amendment | High | Patch `decisions/D001-prepaid-sot.md` (addendum: durable store = libSQL; books.json lab legacy) | -| **C13** | Auth store undecided (`admin_users` / `portal_sessions` edge-local vs libSQL) | Med | **Decide before Phase 3:** v1 keep auth in edge-local `zappier.db` (or per-edge file); business tables only in libSQL. Document; revisit if edges are multi-host without sticky sessions | -| **C14** | Concurrent `usage_entries` hash-chain append across hosts | High (launch) | Single logical meter writer (one process or libSQL-serialized exclusive txn path). No competing multi-host `BEGIN IMMEDIATE` without proven libSQL semantics. Dual-run must not dual-append | -| **C15** | GETTING-STARTED / `01-system.md` still call account-balance prepaid SoT | Med | Phase 0 doc patch: SoT → libSQL; account-balance becomes facade or retired | -| **C16** | Live account-balance still up on `:3010` while edge also owns `billing.db` ledger | High | Cutover runbook must name which process is authoritative **each day** of dual-run; forbid two adjust paths | -| **C17** | Optional Litestream vs multi-writer temptation | Med | Litestream = backup/DR of **one** primary only (D006 non-goal). Plan must forbid Litestream-as-fabric | -| **C18** | Digests-only vs any remaining “upload file then stamp” Zapier creates | Med | Client hasher + digests-only batch remains the supported path; reject raw-data stamp examples in public docs | - ---- - -## Risks for 3–5 machine launch - -1. **Money split-brain (critical).** Today prepaid/ledger truth is fragmented (`books.json` via `:3010`, live `billing.db` on edge, caches). Splitting planes across hosts before one libSQL writer lands will diverge balances, statements, and CS credits. Mitigate: no Phase 4 host split until Phase 3 removes second writers and dual-run is read-only fallback. - -2. **Hash-chain integrity under concurrency (critical).** `usage_entries` depends on tip-linked `prev_hash`/`row_hash`. Multiple edge replicas opening libSQL without an exclusive append protocol will fork or fail the chain. Mitigate: one meter writer service (or proven libSQL write serialization + contract tests for concurrent append). - -3. **libSQL HA underspecified (high).** Plan says “dedicated host or primary of 3–5” but not: primary placement, replica/read role, failover RPO/RTO, backup (Litestream or native), network ACL, and what happens when libSQL is down (fail closed on meter vs stale cache). For launch, pick **one primary + backup**; do not invent multi-primary. - -4. **JetStream mistaken for ledger (high).** `ZAPIER_USAGE` already exists in STATUS with replicas=3. Operators/agents will “persist usage on JetStream” under reliability pressure. Standing rule + plan §5 must demote it; CI/docs grep for usage-as-SoT. - -5. **Job-complete subject confusion (high).** Zapier pipeline `verae.zapier.jobs.events` (JetStream) vs Verae chain notify `verae.ts.completed`/`verae.ts.receipt` (core only). A “make complete more reliable” change must not JetStream `verae.ts.*` (standing rule). Plan C5 is correct; nats-subjects.md still omits `verae.ts.*` — patch it. - -6. **Wrong upstream in smoke (medium).** Edge points at `:3130`; lab docs emphasize `:3100`. Multi-host deploy scripts that copy catalog defaults will meter against mock middleware and lie about chain connectivity. - -7. **Auth sticky-session / multi-edge (medium).** If auth stays in local `zappier.db` and two edges serve portal/admin, sessions break. Acceptable for v1 single edge; call out as launch constraint. - -8. **Migration tip verification false confidence (medium).** Lab `usage_entries` tip is tiny (observed count=1 on 138). Migrator tests need adversarial fixtures (long chain, tamper mid-chain, empty DB), not only live copy. - -9. **Deploy rsync DB wipe (medium, known).** Standing ops: never rsync `*.db*` onto `/opt/verae/zappier-edge`. libSQL cutover must use migrator + URL switch, not file copy into prod workdir via deploy. - -10. **NATS billing request-reply after mailbox retirement (medium).** `verae.billing.*` subjects remain useful as a facade; if account-balance dies without a replacement listener, CS/sales/portal statement paths break even if libSQL is healthy. Plan must keep a single billing facade process. - -11. **Credits vs cents naming (low).** Product credits; columns `*_cents`. Fine if documented; do not “fix” by rewriting historical rows mid-migration. - -12. **Go timestamping DB proximity (low but non-negotiable).** Same host may hold `/var/lib/timestamping-svc/data/blockchain.db`. Automation must path-allowlist business DBs only. - ---- - -## Required plan amendments (concrete text) - -Merge the following into `plans/2026-10-03-libsql-multihost-architecture.md` (and patch D001 as noted). - -### A1 — Amend Global Constraints (add bullets) - -```text -- ZAPIER_USAGE / verae.zapier.usage is telemetry or export only — never usage/money SoT. -- Litestream (if used) backs up a single libSQL primary; never multi-writer fabric. -- Single meter/prepaid writer process owns usage_entries append + balance adjust; - other hosts call that API or use libSQL only through that writer’s transactions. -- Dual-run: libSQL primary for writes; file/SQLite fallback is read-only (or kill-switched). -- Auth (admin_users, portal_sessions): v1 remains edge-local zappier.db; not in verae_business. -- Product edge bind on 138: 127.0.0.1:3040 (zap.verae.ai); ZAPPIER_UPSTREAM defaults to - http://127.0.0.1:3130 for live lab smoke (not :3100 mock unless MOCK path intentional). -- Catalog docs that call account-balance “prepaid SoT” are stale after D006; Phase 0 patches them. -``` - -### A2 — Expand §5 NATS matrix (replace/extend rows) - -```text -| verae.zapier.usage / ZAPIER_USAGE | JetStream optional | Telemetry/export only; NOT ledger SoT | -| verae.ts.completed, verae.ts.receipt | Core NATS only | Chain job-complete notify; never JetStream | -| verae.billing.* | Core request-reply (+ optional pub) | Facade over libSQL BusinessStore; one listener | -``` - -Also add to Phase 0: patch `docs/02-architecture/nats-subjects.md` with an explicit **Non-streams** section listing `verae.ts.*` as core-only and `ZAPIER_USAGE` as non-SoT. - -### A3 — New §4.1 libSQL topology (insert after §4) - -```text -### 4.1 Launch topology (3–5 machines) - -- One libSQL primary (dedicated host or named primary among the 3–5). -- Replicas if any are read-only; writers always target primary URL (`LIBSQL_URL`). -- Backup: filesystem snapshot and/or Litestream of the primary only; tested restore before Phase 4. -- Fail closed: metered stamp paths return 503 if BusinessStore write unavailable - (no silent local SQLite fallback in production). -- Network: libSQL listen on private fabric only (same class as NATS 10.10.10.0/24); not public. -- Schema name: `verae_business` (confirm); auth schema stays out for v1. -``` - -### A4 — Phase 2 dual-run clarification - -```text -- Writes: libSQL only once cutover flag BUSINESS_STORE=libsql is on. -- Fallback flag enables read-only SQLite/books for diagnosis, not adjust/recordUsage. -- Migrator verifies usage chain pre/post; refuses cutover if verifyChain fails. -- Account-balance either (a) becomes libSQL-backed facade for verae.billing.*, or - (b) is stopped after edge/CS/sales talk HTTP BusinessStore — pick one in Phase 3 tasking. -``` - -### A5 — Patch decision file D001 (Phase 0, same PR as plan amend) - -Append to `decisions/D001-prepaid-sot.md`: - -```text -## Addendum (2026-10-03, D006) - -Durable prepaid/credits/usage/payments store is shared **libSQL**, not per-process -`books.json` or per-host `billing.db`. Single writer path still required. Edge -`balanceCents` remains a cache. Lab `books.json` / `billing.db` are migration sources. -``` - -### A6 — Conflict register: add C10–C18 - -Copy the new rows from this review’s register into plan §3. - -### A7 — Acceptance criteria additions - -```text -9. ZAPIER_USAGE is documented/configured as non-SoT; no service writes authoritative balances there. -10. Production metered paths fail closed if libSQL/BusinessStore writer is down. -11. Concurrent append contract test proves single-writer (or serialized) hash-chain safety. -12. Live env matrix documents :3040 edge, :3130 product upstream, :3100 mock, :3010 facade/retire. -13. D001 file contains D006 addendum; GETTING-STARTED / 01-system no longer call books.json SoT. -``` - -### A8 — Out of scope (reinforce) - -```text -- Any JetStream binding for verae.ts.completed / verae.ts.receipt -- Editing /var/lib/timestamping-svc/data/blockchain.db or timestamping-svc JWT test results.db -- Multi-primary libSQL or Litestream multi-writer -- Moving portal auth to libSQL in the same cutover as billing (defer) -``` - ---- - -## What must not change (preserve list) - -| Preserve | Why | -|----------|-----| -| **libSQL = business SoT** (D006) | Launch invariant for 3–5 hosts | -| **JetStream = work/archive only** | Not money/CRM database | -| **Core NATS only for `verae.ts.completed` / `verae.ts.receipt`** | Standing rule; no “reliability” override without explicit user words | -| **Digests-only stamp submit** (batch multihash / client hasher) | Live swagger + MCP; no file-byte stamp SoT | -| **No Go timestamping-svc SQLite edits** | `blockchain.db` and related stay local to that service | -| **D001 single prepaid writer path** (store upgraded by D006) | Prevents dual adjust | -| **D002** public `x-api-key`; `veraeUserId` join; JWT as `tokenRef` | Composition ownership | -| **D003** access planes as only public HTTP doors | Edge admin/Stripe stay non-door (loopback / tunnel) | -| **D004** leaf cannot touch billing | Policy allow-list | -| **D005** lan-134 off unless explicitly enabled | Fleet safety | -| **Credits product language**; ledger may keep `*_cents` column names | Avoid migration churn | -| **WAL (or libSQL equivalent) durability** on SQL stores | Existing edge invariant | -| **Deploy never rsyncs live `*.db*`** over production data | Known SQLITE_CANTOPEN / wipe failure mode | -| **Zapier never on NATS** | Hard rule in `01-system.md` | -| **Payment invoices ≠ timestamp receipts** | Composition | -| **Composition ownership:** metering/portal on zappier; stamp/NATS/hooks on middleware | Do not re-merge Stripe into middleware | -| **Existing zappier-edge Jest suite stays green** during driver swap | Regression gate | -| **Channel×segment rate card + credit headers** on metered/batch paths | Live product behavior | - ---- - -## Sign-off checklist - -Architecture signs when all boxes are true (or explicitly waived in writing on this file). - -- [ ] Verdict understood: **approve-with-changes** (not full approve). -- [x] Plan amended with **A1–A8** (merged 2026-10-03) (or equivalent text merged). -- [x] Conflict register includes **C10–C18** with owners. -- [x] **D001** addendum committed pointing durable store at libSQL / D006. -- [ ] **nats-subjects.md** documents `verae.ts.*` core-only and `ZAPIER_USAGE` non-SoT. -- [ ] Env matrix published: `:3040` edge, `:3130` product MW, `:3100` mock MW, `:3010` balance facade. -- [ ] Single-writer / fail-closed / Litestream-backup-only rules in plan. -- [ ] Auth v1 = edge-local decision recorded (C13). -- [ ] Preserve list acknowledged by PM (no JetStream job-complete; no Go DB; digests-only). -- [ ] Phase 3/4 gated on writer consolidation + migrator chain verify. -- [ ] QA/PM may proceed to board + Task 1 **after** A1–A8 land; not before. - -**Architecture provisional sign-off:** approve-with-changes — **A1–A8 merged into plan + D001**; remaining checklist items still open (nats-subjects, env matrix, GETTING-STARTED). -**Full architecture sign-off:** withheld until checklist complete. - ---- - -## Evidence notes (live 138, 2026-10-03) - -- `/opt/verae/zappier-edge`: `billing.db` + `zappier.db` (+ WAL/SHM), owner `verae-zapier`; service **active**. -- Edge env: `PORT=3040`, `HOST=127.0.0.1`, `ZAPPIER_UPSTREAM=http://127.0.0.1:3130`. -- Health: `:3040` zappier-edge ok; `:3100` middleware `mockVerae=true`, `natsConnected=false`; `:3130` middleware `mockVerae=false`, `natsConnected=true`; `:3010` account-balance still serving. -- `billing.db` tables: customers, usage_entries, channel_rates*, invoices*, price_endpoints, tiers; `zappier.db`: admin_users, portal_sessions. -- Access planes listening on `:3021–3025` (+ staff IAM/session); NATS client ports present on private/host binds. - ---- - -*End of architecture review.* diff --git a/plans/reviews/2026-10-03-pm-board.md b/plans/reviews/2026-10-03-pm-board.md deleted file mode 100644 index c65e22f..0000000 --- a/plans/reviews/2026-10-03-pm-board.md +++ /dev/null @@ -1,511 +0,0 @@ -# PM Board — Verae × Zapier Multi-Host libSQL Cutover - -**Role:** Project Manager -**Date:** 2026-10-03 -**Plan:** [`../2026-10-03-libsql-multihost-architecture.md`](../2026-10-03-libsql-multihost-architecture.md) -**Decision:** [`../../decisions/D006-libsql-business-jetstream-work.md`](../../decisions/D006-libsql-business-jetstream-work.md) -**Docs map:** `/tmp/zapier-system-docs-map.md` (also mirrored under `plans/2026-10-03-system-docs-map.md`) -**Forgejo org:** `https://git.georgelambert.org/marchon/` -**Clone base (lab worktrees):** `/Users/marchon/apps/verae/` - -**Standing rules (non-negotiable):** -- Business SoT = **libSQL only** (D006). -- JetStream = work queues only (`ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, archive). -- `verae.ts.completed` / `verae.ts.receipt` = **core NATS only** (never JetStream). -- Do **not** touch Go timestamping-svc SQLite (`blockchain.db`, jwt_auth test DBs). -- Deploy never rsyncs live `*.db*` over production data. -- Digests-only stamp submit (multihash batch). - ---- - -## 1. Constraint statement (TOC) - -> The system constraint for the multi-host cutover is **shared business SoT cutover (libSQL + single prepaid writer path)** because every critical commercial userflow (meter charge, CS credit adjust, sales quote→meter match, portal statement, accounting export, leaf billing deny) blocks on one consistent customer / balance / usage / pricing store; JetStream workers and HTTPS access doors already exist in the NS1 lab and are not the bottleneck. - -**Critical chain (serial):** -Conflict freeze (Phase 0) → BusinessStore + libSQL skeleton (Phase 1) → Migrate lab data (Phase 2) → **MERGE GATE: writers** → Rewire edge/CS/sales/acct (Phase 3) → Multi-host topology (Phase 4) → Userflow E2E + gates (Phase 5) → QA whattochange loop (Phase 6) → **PROJECT BUFFER: double userflow pass**. - -**Feeding chains (parallel into merge gates):** -Docs/conflict patches (C2/C3/C5/C6); NATS subject clarification (`verae.zapier.jobs.events` ≠ `verae.ts.*`); digests-only catalog sync; access-plane smoke; Forgejo clone inventory; libSQL server/ops bring-up; leaf D004 deny tests. - -**Exploit the constraint:** Do not start writer rewires until BusinessStore contract + migrator + dual-run reads are green. Do not split hosts until single-writer path is proven on one machine against libSQL. - ---- - -## 2. Mermaid DAG of phases with merge gates - -```mermaid -flowchart TD - subgraph feed0["Week 0–1 feeding (parallel)"] - F0a["WP-A Docs & conflict freeze
C2/C3/C5/C6"] - F0b["WP-B NATS subject clarification
jobs.events vs verae.ts.*"] - F0c["WP-C Digests-only docs/models sync"] - F0d["WP-D Access-plane smoke + D004 leaf"] - F0e["WP-E Forgejo clone inventory + worktrees"] - end - - MG0["MERGE GATE 0
Arch review signed + conflicts frozen"] - - subgraph crit1["Critical chain — BusinessStore"] - WP_F["WP-F BusinessStore interface + contract tests"] - WP_G["WP-G SqliteFileBusinessStore adapter"] - WP_H["WP-H LibsqlBusinessStore + CI libSQL"] - end - - MG1["MERGE GATE 1
Contract green on both drivers"] - - subgraph feed1["Feeding into migrate (parallel)"] - WP_I["WP-I libSQL server/ops (verae-ops)"] - WP_J["WP-J Fixture harvest billing.db + books.json"] - end - - WP_K["WP-K Migrator + chain verify pre/post"] - MG2["MERGE GATE 2
Dual-run reads OK; NO writer rewire yet"] - - subgraph crit3["Critical chain — single writer"] - WP_L["WP-L Rewire zappier-edge meter/ledger → BusinessStore"] - WP_M["WP-M Rewire account-balance → libSQL / fold API"] - WP_N["WP-N Rewire CS / sales / accounting-export"] - end - - MG3["MERGE GATE 3
One prepaid writer; second paths removed"] - - WP_O["WP-O Multi-host topology LIBSQL_URL"] - WP_P["WP-P Userflow E2E harness 1–7"] - MG4["MERGE GATE 4
Unit+integration+gates green"] - - WP_Q["WP-Q QA whattochange loop"] - PB["PROJECT BUFFER
Userflows 1–7 pass twice consecutively"] - - F0a --> MG0 - F0b --> MG0 - F0c --> MG0 - F0d --> MG0 - F0e --> MG0 - MG0 --> WP_F --> WP_G --> WP_H --> MG1 - WP_I --> MG1 - WP_J --> MG1 - MG1 --> WP_K --> MG2 - MG2 --> WP_L --> WP_M --> WP_N --> MG3 - MG3 --> WP_O --> WP_P --> MG4 - MG4 --> WP_Q --> PB -``` - -**Artifact edges (what must exist before the next node consumes it):** - -| From → To | Required artifact | -|-----------|-------------------| -| Feed0 → MG0 | `plans/reviews/2026-10-03-arch-review.md` PASS + conflict patches merged | -| MG0 → WP-F | Frozen conflict register C1–C9; this PM board | -| WP-H → MG1 | Jest contract suite green for SqliteFile + Libsql drivers | -| WP-I → MG1 | Reachable `LIBSQL_URL` (lab docker or host) documented in ops | -| WP-K → MG2 | Migrator report: row counts, chain tip hash match, dual-read flag | -| MG2 → WP-L | Explicit PM/Orchestrator go-ahead: “writers unlocked” | -| MG3 → WP-O | Grep proof: no production writes to local `billing.db`/`books.json` SoT | -| WP-P → MG4 | Harness report for flows 1–7 (MOCK_VERAE) | -| WP-Q → PB | Two consecutive full green userflow runs | - ---- - -## 3. Engineer work packages - -Clone convention for every WP: - -```bash -ORG=https://git.georgelambert.org/marchon -BASE=/Users/marchon/apps/verae -# Prefer existing clone; else: -git clone "$ORG/.git" "$BASE/" -``` - -Commit messages must cite: `plans/2026-10-03-libsql-multihost-architecture.md` and this board (`plans/reviews/2026-10-03-pm-board.md`). - ---- - -### WP-A — Docs & conflict freeze (Phase 0) - -| Field | Content | -|-------|---------| -| **PreReqs** | Architecture plan published; D006 accepted | -| **Mode** | Parallel with WP-B/C/D/E | -| **Repos to clone** | `zapier-decisions`, `overview`, `zapier-docs-master` (catalog mirror of docs-master), optionally site tree if editing published catalog | -| **Files to touch** | `zapier-decisions/STATUS.md`, `LOG.md`, `TODO.md`; `overview/README.md` / composition status blurb; `overview` auth notes that still say `zmw_` Bearer (C3); `docs/02-architecture` equivalents in catalog if present under docs-master; ensure C5 wording: JetStream `verae.zapier.jobs.events` ≠ core `verae.ts.*` | -| **Definition of done** | Conflicts C2, C3, C5, C6 closed in writing; C1/C7/C8 restated as “libSQL SoT / single writer”; C9 “out of scope” reiterated; arch-review placeholder path ready for Architect sign-off | -| **Test commands** | Doc lint only: `rg -n 'zmw_|books\\.json is sole|JetStream.*verae\\.ts' overview zapier-docs-master zapier-decisions` — residual false claims listed and fixed or explicitly deferred with why | - ---- - -### WP-B — NATS subject clarification - -| Field | Content | -|-------|---------| -| **PreReqs** | None (feeding) | -| **Mode** | Parallel | -| **Repos** | `zapier-decisions`, `verae-middleware`, `verae-nats-cluster`, `zapier-docs-master` | -| **Files** | `verae-middleware/src/nats/subjects.js`; `verae-middleware/NATS.md`; `zapier-docs-master` / architecture `nats-subjects` copies; `verae-nats-cluster` stream list docs; `zapier-decisions/NATS.md` | -| **DoD** | Matrix matches architecture §5: JetStream for `verae.zapier.jobs.{watch,events}`, `webhooks.deliver`, archive; **core only** for `verae.ts.completed` / `verae.ts.receipt`; billing `verae.billing.*` remains core request-reply | -| **Tests** | `cd verae-middleware && npm run test:unit`; `rg -n 'verae\\.ts\\.(completed|receipt)' verae-middleware` must not show JetStream stream binds; `cd verae-nats-cluster && npm run status` (lab) — confirm product streams list does **not** include timestamp job-complete | - ---- - -### WP-C — Digests-only docs / models sync - -| Field | Content | -|-------|---------| -| **PreReqs** | None | -| **Mode** | Parallel | -| **Repos** | `zapier-docs-master`, `verae-middleware`, `verae-zapier-app`, `zapier-decisions` | -| **Files** | Catalog models / OpenAPI examples that still show “Hello world” data; `verae-middleware` route docs for batch timestamp; Zapier create field descriptions | -| **DoD** | All public stamp examples are multihash digests; no plaintext body-as-data examples in SoT docs | -| **Tests** | `cd verae-middleware && npm run test:hash` (and batch-related unit tests if present); `rg -n 'Hello world' docs models openapi` → zero in stamp request examples | - ---- - -### WP-D — Access-plane smoke + D004 leaf deny - -| Field | Content | -|-------|---------| -| **PreReqs** | Lab doors up (STATUS) preferred; else local unit tests | -| **Mode** | Parallel | -| **Repos** | `verae-access-leaf`, `verae-access-web`, `verae-access-api`, `verae-access-zapier`, `verae-access-authz`, `zappier-identity` | -| **Files** | Leaf allow-list / policy tests under each `test/`; authz check paths | -| **DoD** | Automated proof leaf cannot call `verae.billing.balance.adjust` / prepaid adjust HTTP; web/api/zapier smoke health | -| **Tests** | `cd verae-access-leaf && npm test`; `cd verae-access-authz && npm test`; `cd verae-access-web && npm test`; same for api/zapier if scripts exist | - ---- - -### WP-E — Forgejo clone inventory + worktrees - -| Field | Content | -|-------|---------| -| **PreReqs** | None | -| **Mode** | Parallel | -| **Repos** | `verae-bootstrap` (source of `repos.txt`) | -| **Files** | Produce checklist under `zapier-decisions/plans/reviews/clone-inventory-2026-10-03.md` (or append section to this board’s §7); do not invent new repos | -| **DoD** | Every WP’s clone list verified present under `/Users/marchon/apps/verae` or cloned; `repos.txt` vs WP matrix diff empty for in-scope modules | -| **Tests** | `while read r; do test -d /Users/marchon/apps/verae/$r/.git || echo MISSING $r; done < verae-bootstrap/repos.txt` | - ---- - -### WP-F — BusinessStore interface + failing contract tests (Phase 1 start) - -| Field | Content | -|-------|---------| -| **PreReqs** | **MG0 PASS** | -| **Mode** | Serial critical chain | -| **Repos** | `zappier-edge` | -| **Files** | **Create** `src/db/business-store.ts` (interface); **Create** `tests/business-store.contract.test.ts`; may re-export types from `src/usage.ts`, `src/db/customer-repo.ts`, `src/db/pricing-store.ts`, `src/db/invoice-repo.ts` | -| **Interface minimum** | `findCustomer`, `saveCustomer`, `recordUsage`, `verifyUsageChain`, `listUsage`, `getBalance`, `adjustBalance`, `getRates` / `resolveChannelRate`, `listInvoices` / `saveInvoice` | -| **DoD** | Contract tests exist and **fail** against missing impl (red); methods documented; hash-chain fields (`prev_hash`, `row_hash`) required even if current `SqliteUsageRepo` lacks them (extend schema in WP-G) | -| **Tests** | `cd zappier-edge && npm test -- tests/business-store.contract.test.ts` → expect fail | - ---- - -### WP-G — SqliteFileBusinessStore (dev/lab adapter) - -| Field | Content | -|-------|---------| -| **PreReqs** | WP-F red tests exist | -| **Mode** | Serial | -| **Repos** | `zappier-edge` | -| **Files** | **Create** `src/db/sqlite-file-business-store.ts`; adapt `src/db/usage-repo.ts` (add hash chain columns + verify); wire factory in `src/index.ts` / `src/app.ts` behind flag `BUSINESS_STORE=sqlite-file` (default keep current behavior until WP-L) | -| **DoD** | Contract tests **pass** on file/memory SQLite; existing Jest suite still green | -| **Tests** | `cd zappier-edge && npm test -- tests/business-store.contract.test.ts`; `cd zappier-edge && npm test` | - ---- - -### WP-H — LibsqlBusinessStore + CI - -| Field | Content | -|-------|---------| -| **PreReqs** | WP-G green | -| **Mode** | Serial (can overlap WP-I ops once interface stable) | -| **Repos** | `zappier-edge` | -| **Files** | **Create** `src/db/libsql-business-store.ts`; `package.json` add `@libsql/client`; `.env.example` `LIBSQL_URL`, `LIBSQL_AUTH_TOKEN`; Jest config for libsql file URL or embedded | -| **DoD** | Same contract suite passes with `BUSINESS_STORE=libsql` and `LIBSQL_URL=file:…` (or testcontainers); WAL/durable semantics documented | -| **Tests** | `cd zappier-edge && BUSINESS_STORE=libsql LIBSQL_URL=file:./tmp-business.db npm test -- tests/business-store.contract.test.ts`; full `npm test` | - ---- - -### WP-I — libSQL server / ops bring-up - -| Field | Content | -|-------|---------| -| **PreReqs** | None for draft runbook; MG1 needs live URL | -| **Mode** | Parallel feeding into MG1 | -| **Repos** | `verae-ops`, `verae-deploy`, `verae-nats-cluster` (topology reference only), `zapier-decisions` | -| **Files** | New runbook under `verae-ops` (e.g. `libsql.md` / section in `GETTING-STARTED.md`); env matrix for 3–5 hosts; **do not** relocate Go DBs | -| **DoD** | Documented `LIBSQL_URL` for lab; backup note (Litestream optional for primary file — not multi-writer); firewall private-only | -| **Tests** | Manual: `curl`/client ping create table + select; record connection string in STATUS | - ---- - -### WP-J — Fixture harvest (billing.db + books.json) - -| Field | Content | -|-------|---------| -| **PreReqs** | Read-only access to lab 138 paths (or sanitized copies) | -| **Mode** | Parallel into MG1 | -| **Repos** | `zappier-edge`, `zappier-account-balance`, `zapier-decisions` | -| **Files** | **Create** `zappier-edge/tests/fixtures/` (sanitized); `scripts/export-lab-business-fixtures.md` instructions; never commit live secrets/API keys | -| **Sources** | Live stand-ins: `/opt/verae/zappier-edge/billing.db` (BILLING_DB), `zappier.db` (auth may stay local), account-balance `books.json`; seed keys in lab only | -| **DoD** | Anonymized fixtures with known chain tip + prepaid balances for migrator tests | -| **Tests** | Fixture checksums documented; `sqlite3 fixture.db 'select count(*) from usage_entries;'` | - ---- - -### WP-K — Migrator billing.db + books.json → libSQL (Phase 2) - -| Field | Content | -|-------|---------| -| **PreReqs** | **MG1 PASS**; fixtures from WP-J; `LIBSQL_URL` from WP-I | -| **Mode** | Serial | -| **Repos** | `zappier-edge`, `zappier-account-balance` | -| **Files** | **Create** `zappier-edge/scripts/migrate-to-libsql.ts` (or `.mjs`); **Create** `tests/migrate-to-libsql.test.ts`; map tables: customers, tiers, price_endpoints, channel_rates*, usage_entries(+hashes), invoices*, balances/credits/payments from books | -| **DoD** | One-shot migrator; `verifyUsageChain` true before and after; dual-run read path flag `BUSINESS_READ_PRIMARY=libsql` with file fallback; **no production writer cutover yet** | -| **Tests** | `cd zappier-edge && npm test -- tests/migrate-to-libsql.test.ts`; dry-run against fixtures; print before/after tip hashes | - ---- - -### MERGE GATE 2 — Writers still locked - -Engineers **must not** start WP-L/M/N until Orchestrator/PM records: - -``` -MERGE GATE 2: PASS -- migrator tip hashes match -- dual-run reads validated -- writers unlocked: yes -``` - ---- - -### WP-L — Rewire zappier-edge meter / ledger → BusinessStore (Phase 3) - -| Field | Content | -|-------|---------| -| **PreReqs** | **MG2 PASS** | -| **Mode** | Serial (constraint path) | -| **Repos** | `zappier-edge` | -| **Files** | `src/meter.ts`, `src/usage.ts`, `src/ledger.ts`, `src/billing-nats.ts`, `src/credits.ts`, `src/portal.ts`, `src/pricing.ts`, `src/invoicing.ts`, `src/db/*-repo.ts`, `src/index.ts`, `src/app.ts` — inject BusinessStore; edge `balanceCents` remains cache filled from store/billing API | -| **DoD** | Meter appends usage **only** via BusinessStore; prepaid adjust goes through single writer API; no direct better-sqlite3 business writes outside adapter | -| **Tests** | `cd zappier-edge && npm test`; focus `tests/meter.test.ts`, `tests/usage.test.ts`, `tests/credit.test.ts`, `tests/portal.test.ts`, `tests/db-*.test.ts`, `tests/billing-delta.test.ts` | - ---- - -### WP-M — Rewire account-balance (books.json → libSQL) - -| Field | Content | -|-------|---------| -| **PreReqs** | WP-L in progress or done (coordinate subjects) | -| **Mode** | Serial after WP-L starts (shared `verae.billing.*` contracts) | -| **Repos** | `zappier-account-balance`, `zappier-edge` | -| **Files** | `zappier-account-balance/src/persist.js`, `books.js`, `server.js`, `subjects.js`; tests under `test/`; README SoT language → libSQL | -| **DoD** | NATS `verae.billing.balance.*` / statement backed by libSQL BusinessStore or HTTP billing API; `books.json` deprecated (import-only); D001 single writer preserved | -| **Tests** | `cd zappier-account-balance && npm test`; integration: adjust → edge statement sees same balance | - ---- - -### WP-N — Rewire CS / sales / accounting-export - -| Field | Content | -|-------|---------| -| **PreReqs** | WP-M billing subjects stable | -| **Mode** | Parallel with each other **after** WP-M contract freeze; else serial behind WP-M | -| **Repos** | `zappier-customer-service`, `zappier-sales-pricing`, `zappier-accounting-export` | -| **Files** | Each package `src/nats-billing.js`, `src/server.js`; staff UI pages as needed | -| **DoD** | CS credit adjust → libSQL balance; sales channel×segment rates → same rate tables meter uses; accounting export reads payments/usage from libSQL | -| **Tests** | `cd zappier-customer-service && npm test`; `cd zappier-sales-pricing && npm test`; `cd zappier-accounting-export && npm test` | - ---- - -### MERGE GATE 3 — Single writer proof - -| Check | Pass criteria | -|-------|---------------| -| Grep | No code path treats `books.json` or per-host `billing.db` as production SoT | -| Runtime | One `LIBSQL_URL` serves customers/pricing/usage/invoices/balances | -| D004 | Leaf still cannot adjust billing | -| Regression | `zappier-edge` full Jest green | - ---- - -### WP-O — Multi-host topology (Phase 4) - -| Field | Content | -|-------|---------| -| **PreReqs** | **MG3 PASS** | -| **Mode** | Serial | -| **Repos** | `verae-ops`, `verae-deploy`, `verae-fleet`, `verae-keep`, `verae-bootstrap`, `zappier-edge`, `verae-middleware` | -| **Files** | Host env templates (`LIBSQL_URL`, `NATS_URL=nats://10.10.10.21:4222,...`); deploy scripts; STATUS topology section | -| **DoD** | libSQL on dedicated/primary host; edge/middleware/CS on separate hosts possible; JetStream workers unchanged; leaf cannot bill | -| **Tests** | Cross-host: meter on host A, CS adjust on host B, portal statement on host C — same balance; `verae-nats-cluster` `npm run status` | - ---- - -### WP-P — Userflow E2E harness (Phase 5) - -| Field | Content | -|-------|---------| -| **PreReqs** | MG3; WP-O preferred for multi-host proof, MOCK_VERAE OK for first pass | -| **Mode** | Serial into MG4 | -| **Repos** | `verae-zapier-simulator`, `zappier-edge`, `verae-middleware`, `verae-access-web`, `verae-access-zapier`, `verae-access-leaf` | -| **Files** | Extend simulator/`harness` under simulator or `packages`-style tests; scripts for flows 1–7 from architecture §1 | -| **Flows** | (1) Signup→API key→Statement (2) Stamp digests single+batch + credit headers (3) Async complete→REST Hook via jobs.watch→events→webhooks.deliver (4) CS credit→portal balance (5) Sales quote→meter match (6) Accounting export consistency (7) Leaf stamp OK / billing deny | -| **DoD** | Automated report artifact under `zapier-decisions/plans/qa/` | -| **Tests** | `cd verae-zapier-simulator && npm test`; middleware `npm test` + selected `test:gate*`; edge `npm test`; flow runner script (to be added) exit 0 | - ---- - -### WP-Q — QA whattochange loop (Phase 6) - -| Field | Content | -|-------|---------| -| **PreReqs** | **MG4 PASS** (suite green once) | -| **Mode** | Loop until project buffer | -| **Repos** | `zapier-decisions` + whichever repos bugs land in | -| **Files** | `plans/qa/-whattochange/` packs; `NotImplemented/` leftovers per recommended-changes skill | -| **DoD** | Each bug → a/b/c/d + PreReq todos; engineers implement; re-run WP-P | -| **Tests** | Full suite + userflows after each pack | - ---- - -## 4. Parallel vs serial schedule — first 2 weeks - -Assume agent-days; calendar = 10 working days. Critical chain owns the constraint; feeding work never blocks writers early. - -| Day | Parallel (feeding) | Serial (critical chain) | Gate | -|-----|--------------------|-------------------------|------| -| **1** | WP-A, WP-B, WP-C, WP-D, WP-E kickoff | — | Aim MG0 EOD day 2 | -| **2** | Finish A–E; Architect signs `2026-10-03-arch-review.md` | — | **MG0** | -| **3** | WP-I ops draft; WP-J fixture planning | **WP-F** interface + red tests | | -| **4** | WP-I lab libSQL up; WP-J sanitize fixtures | **WP-G** SqliteFile adapter | | -| **5** | WP-I harden docs; fixture finalize | **WP-H** LibsqlBusinessStore | **MG1** target EOD day 5–6 | -| **6** | Buffer if MG1 slip | **WP-K** migrator TDD | | -| **7** | — | Finish WP-K; dual-run reads | **MG2** (writers unlock) | -| **8** | — | **WP-L** edge rewire | | -| **9** | WP-N prep (read-only) | **WP-M** account-balance | | -| **10** | **WP-N** CS/sales/acct (parallel agents) | Finish L/M integration | **MG3** stretch; else day 11 | - -**Week 2 overflow (if MG3 slips):** do **not** start WP-O multi-host. Burn down writer bugs; keep project buffer intact. - -**Week 2 if on track (days 11–14):** WP-O topology → WP-P harness → MG4 → start WP-Q. - -**Explicit non-goals in first 2 weeks:** live `MOCK_VERAE=false` seal proof; Zapier Platform public push; lan-134; Go DB moves; JetStream for `verae.ts.*`. - ---- - -## 5. QA entry criteria - -QA (WP-Q / userflow break attempts) may begin only when **all** of the following are true: - -1. **MG3 PASS** — single prepaid writer on libSQL; no dual SoT writes. -2. **Regression green:** `zappier-edge` `npm test`; `zappier-account-balance` `npm test`; CS/sales/acct `npm test`; `verae-middleware` `npm test` (unit + non-NATS integration at minimum). -3. **NATS rules verified:** product JetStream streams only for Zapier jobs/webhooks/archive; `verae.ts.completed` / `verae.ts.receipt` core-only (grep + cluster status). -4. **Harness exists (WP-P skeleton):** at least flows 1, 2, 4 runnable under `MOCK_VERAE=true`. -5. **D004 automated:** leaf billing adjust denied. -6. **Migrator evidence:** tip hash report archived under `plans/qa/`. -7. **Go timestamping DBs:** confirmation note that no PR touched them. -8. **Docs:** C2/C3/C5/C6 closed or explicitly ticketed with owner. - -**QA exit (feeds project buffer):** whattochange packs filed for every failure; no silent skips; leftovers under `NotImplemented/` with why. - ---- - -## 6. Buffer policy - -### Merge gate before rewiring writers (MG2) - -- **Hard stop:** WP-L / WP-M / WP-N are blocked until MG2 PASS. -- Rationale (Critical Chain feeding buffer): incomplete BusinessStore or bad migration hitting the meter writer destroys the only constraint we are exploiting (consistent SoT). -- Gate owner: PM / Orchestrator. Engineers may prepare branches but must not merge writer cutover. - -### Other merge gates - -| Gate | Absorbs variation from | Failure action | -|------|------------------------|----------------| -| MG0 | Docs/conflict feeding | Re-run WP-A/B/C; no Phase 1 code | -| MG1 | Driver + ops + fixtures | Fix contract/CI URL; no migrator on prod-like data | -| MG3 | Writer rewires | Roll back feature flag to SqliteFile/dual-read; do not split hosts | -| MG4 | E2E harness | Fix failures; do not declare QA loop “done” | - -### Project buffer = double userflow pass - -- After MG4 and WP-Q churn, **acceptance requires userflows 1–7 green twice consecutively** (architecture §10 / Phase 6 exit). -- First pass proves the system; second pass absorbs latent race/order bugs without consuming schedule fantasy. -- Do **not** spend project buffer on early scope (Platform push, live seal, Peergos redesign). -- If second pass fails, return to WP-Q; do not “waive” the second pass. - -### Constraint buffer - -- Before each critical-chain WP (F, G, H, K, L), feeding artifacts listed in §2 must be staged so the constraint agent never waits on clone/docs/URL discovery mid-implementation. - ---- - -## 7. Explicit Forgejo repos each WP clones - -Base URL: `https://git.georgelambert.org/marchon/.git` - -| WP | Repos to clone | -|----|----------------| -| **WP-A** | `zapier-decisions`, `overview`, `zapier-docs-master` | -| **WP-B** | `zapier-decisions`, `verae-middleware`, `verae-nats-cluster`, `zapier-docs-master` | -| **WP-C** | `zapier-docs-master`, `verae-middleware`, `verae-zapier-app`, `zapier-decisions` | -| **WP-D** | `verae-access-leaf`, `verae-access-web`, `verae-access-api`, `verae-access-zapier`, `verae-access-authz`, `zappier-identity` | -| **WP-E** | `verae-bootstrap` (plus any MISSING from `repos.txt` needed by later WPs) | -| **WP-F** | `zappier-edge` | -| **WP-G** | `zappier-edge` | -| **WP-H** | `zappier-edge` | -| **WP-I** | `verae-ops`, `verae-deploy`, `zapier-decisions` | -| **WP-J** | `zappier-edge`, `zappier-account-balance`, `zapier-decisions` | -| **WP-K** | `zappier-edge`, `zappier-account-balance` | -| **WP-L** | `zappier-edge` | -| **WP-M** | `zappier-account-balance`, `zappier-edge` | -| **WP-N** | `zappier-customer-service`, `zappier-sales-pricing`, `zappier-accounting-export` | -| **WP-O** | `verae-ops`, `verae-deploy`, `verae-fleet`, `verae-keep`, `verae-bootstrap`, `zappier-edge`, `verae-middleware` | -| **WP-P** | `verae-zapier-simulator`, `zappier-edge`, `verae-middleware`, `verae-access-web`, `verae-access-zapier`, `verae-access-leaf` | -| **WP-Q** | `zapier-decisions` + bug-target repos as discovered | - -### Union set for cutover agents (minimum clone list) - -``` -zapier-decisions -overview -zapier-docs-master -verae-bootstrap -verae-ops -verae-deploy -verae-fleet -verae-keep -verae-nats-cluster -zappier-edge -zappier-account-balance -zappier-customer-service -zappier-sales-pricing -zappier-accounting-export -zappier-identity -verae-middleware -verae-zapier-app -verae-zapier-simulator -verae-access-authz -verae-access-web -verae-access-api -verae-access-leaf -verae-access-zapier -``` - -**Out of clone scope for this cutover (do not pull for WPs above):** Go timestamping-svc repos, Peergos/Seaweed redesign trees, `verae-activate` (unless WP-C needs tiny-app copy sync). - ---- - -## Orchestrator checklist (next actions) - -- [ ] Architect produces `plans/reviews/2026-10-03-arch-review.md` → **MG0** -- [ ] Launch parallel agents: WP-A, WP-B, WP-C, WP-D, WP-E -- [ ] After MG0: engineer on WP-F → G → H; parallel WP-I, WP-J -- [ ] **Stop before writers** until MG2 -- [ ] Enforce D006 + core-NATS job-complete on every PR review - ---- - -*End of PM board. Planning artifact only — no implementation in this change.* diff --git a/reports/charts/core-throughput.png b/reports/charts/core-throughput.png deleted file mode 100644 index 10fbf1c..0000000 Binary files a/reports/charts/core-throughput.png and /dev/null differ diff --git a/reports/charts/core-vs-js.png b/reports/charts/core-vs-js.png deleted file mode 100644 index 0584f9e..0000000 Binary files a/reports/charts/core-vs-js.png and /dev/null differ diff --git a/reports/charts/delay-percentiles.png b/reports/charts/delay-percentiles.png deleted file mode 100644 index c063bca..0000000 Binary files a/reports/charts/delay-percentiles.png and /dev/null differ diff --git a/reports/charts/js-throughput.png b/reports/charts/js-throughput.png deleted file mode 100644 index 553af5d..0000000 Binary files a/reports/charts/js-throughput.png and /dev/null differ diff --git a/reports/charts/payload-size.png b/reports/charts/payload-size.png deleted file mode 100644 index c7ba0cb..0000000 Binary files a/reports/charts/payload-size.png and /dev/null differ diff --git a/reports/nats-cluster-bench.html b/reports/nats-cluster-bench.html deleted file mode 100644 index 4201eb2..0000000 --- a/reports/nats-cluster-bench.html +++ /dev/null @@ -1,1278 +0,0 @@ - - - - - - - NATS cluster message speed - - - - -
Verae Time × Zapier · progress report

NATS cluster message speed

packages/zapier-decisions/reports/nats-cluster-bench.md
-
-

NATS cluster message speed

-
-

Progress report · 2026-09-12 · run -20260912T045131Z (UTC)

-

This is the full write-up of the test-environment NATS cluster bench: -what was measured, how, the numbers, the charts, and what they mean for -Verae Time × Zapier. Short tables also live in verae-nats-cluster/BENCH.md. -Raw logs and CSVs are in that repo under -results/20260912T045131Z/.

-
-

1. Executive summary

-

The test cluster is three JetStream nodes on private -vmbr1 (LXC 511–513). The bench client is a -fourth guest (LXC 510), so the numbers are -cluster-plus-network, not a process talking to itself on loopback.

-

Two different systems were measured, on purpose:

- ----- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
SystemWhat it isWhat we got
Core NATSFire-and-forget pub/sub. No disk, no replica ack.About 0.75–2.0 million msgs/s at 128 B, depending -on fan-out. At 1 KiB, about 630k msgs/s and -~616 MB/s aggregate.
JetStream file, replicas=3Durable, replicated — this is what product streams -use.About 16k durable 128 B pubs/s, about -13.5k at 1 KiB. Pull consume keeps up with publish at -~11k msgs/s each side.
Ping delayOne message at a time, publish then wait.avg 0.307 ms, p99 0.734 ms, max -2.76 ms (1k × 128 B).
Flood delayPublishers dump a batch; subscriber drains.150–505 ms. That is queueing under -burst, not wire time.
-

For this product: timestamp jobs, job events, -webhooks, and archive puts go through JetStream r=3. Plan capacity -against ~16k durable msgs/s on this stand, not the -million-msg core numbers. A quiet job-event hop is a fraction of a -millisecond. If a mailbox falls behind, delay jumps into hundreds of -milliseconds — that is the flood column.

-

Core NATS is still useful: it is the ceiling for non-durable fan-out -on this host, and it shows vmbr1 and the nats-server -processes are not the JetStream bottleneck. JetStream is.

-
-

2. Why this test exists

-

The lab cut the test environment over to the three-node cluster. -Before treating that cluster as the message fabric for keep, fleet, -middleware, billing, and archive workers, we needed:

-
    -
  1. Throughput at several loads — one publisher vs -many, 128 B vs 1 KiB, core vs durable.
    2. -
  2. Delay characteristics — both the quiet path (one -message RTT) and the overloaded path (burst into a mailbox).
    3. -
  3. A client that is not a nats-* server — otherwise we -would be measuring loopback on the broker.
    4. -
-

This is a lab stand on one Proxmox host, not three -metal boxes. It answers “is this cluster in the right order of magnitude -for our traffic?” It does not replace a soak test on dedicated -disks.

-
-

3. Topology

-
                    vmbr1  10.10.10.0/24  (not on vmbr0, not public)
-                    -----------------------------------------------
-  LXC 510           LXC 511            LXC 512            LXC 513
-  verae-px-worker   nats-a             nats-b             nats-c
-  10.10.10.20       10.10.10.21        10.10.10.22        10.10.10.23
-  bench client      :4222 client       :4222              :4222
-                    :6222 routes       :6222              :6222
-                    :8222 loopback     :8222              :8222
-
    -
  • Cluster name: verae. Each server has two routes to the -other two.
    • -
  • Client URL: -nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
    • -
  • HTTP monitor is loopback :8222 inside each guest. -Zapier cloud never talks to NATS.
    • -
  • Product streams already on this cluster (ZAPIER_JOBS, -ZAPIER_EVENTS, ZAPIER_WEBHOOKS, -ZAPIER_USAGE, VERAE_ARCHIVE) use -file storage and replicas=3. The -JetStream bench used the same settings on a throwaway stream -benchstream.
    • -
  • Host 127.0.0.1:4222 is still listening on NS1; -clients no longer use it.
    • -
-

Credits for the stack: Scott Lindsey, George Lambert, NATS.IO, -Grok-Code.

-
-

4. Method

-

4.1 Tools

- ---- - - - - - - - - - - - - - - - - - - - - - - - - -
PieceRole
nats CLI 0.1.6Throughput (nats bench --no-progress --csv). Its -min/avg/max are publisher rate spread, not delay.
scripts/latency.mjsTwo connections, header timestamp t, delay = receive -time − send time.
scripts/bench.shRuns the ladder from NS1 via pct exec on VMID 510.
scripts/bench-report.pyTurns logs into the short BENCH.md table.
-

Re-run on NS1, from verae-nats-cluster:

-
bash scripts/bench.sh
-

4.2 Load ladder

-

Core NATS (subject bench.core.*):

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RunPublishersSubscribersMessagesPayload
core-1p1s-50k-1281150,000128 B
core-4p4s-100k-12844100,000128 B
core-8p8s-200k-12888200,000128 B
core-4p4s-50k-1k4450,0001024 B
-

JetStream -(--js --storage file --replicas 3 --stream benchstream). -The stream is deleted between loads so the name never collides:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RunShapeMessagesPayload
js-1p-20k-128-r31 publisher20,000128 B
js-4p-50k-128-r34 publishers50,000128 B
js-4p-20k-1k-r34 publishers20,0001024 B
js-2p2s-20k-128-r32 pub + 2 pull sub20,000128 B
-

Delay (core subjects, two connections):

- ------- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RunModeCountPubsPayload
lat-ping-1k-128ping — publish, wait for that message, repeat1,0001128 B
lat-1p-5k-128flood — publish all, then drain5,0001128 B
lat-4p-10k-128flood10,0004128 B
lat-8p-20k-128flood20,0008128 B
lat-4p-5k-1kflood5,00041024 B
-

Ping answers “how long does one quiet hop take?” Flood answers “what -happens to the last message if we burst N messages into a mailbox?” -Those are different questions. Mixing them is how 0.3 ms and 400 ms get -confused.

-

4.3 How to read nats bench -columns

-
    -
  • Pub msgs/s — rate at which publishers finished -their share.
    • -
  • Sub msgs/s — rate at which subscribers finished. -With several subscribers on the same subject, core NATS fans -out, so sub rate can exceed pub rate.
    • -
  • Aggregate msgs/s — nats CLI -NATS Pub/Sub stats line (pub+sub work in one number). -Useful as a headline; do not treat it as “the network carried this many -unique messages.”
    • -
  • Empty JetStream sub cells mean that run was publish-only (durable -write, no consumer in the same process).
    • -
-
-

5. Throughput results

-

5.1 Core NATS

-
- - -
- -------- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RunAggregate msgs/sPub msgs/sPub MB/sSub msgs/sSub MB/s
core-1p1s-50k-1281,200,836791,09496.57747,46191.24
core-4p4s-100k-1281,521,256316,31238.611,299,634158.65
core-8p8s-200k-1282,007,937333,95740.771,790,736218.60
core-4p4s-50k-1k630,460247,747241.94510,216498.26
-

What this chart is saying. Adding subscribers raises -aggregate and sub rates because each -published message is delivered to every subscriber. Publish rate does -not climb the same way: 1 publisher at 128 B already -pushes ~791k msgs/s; 4 and 8 publishers sit around 310–335k msgs/s -each process slower, while fan-out on the sub side goes -to 1.3M then 1.8M.

-

That publisher slowdown is expected on this stand. The four/eight -publisher processes and the four/eight subscribers all run -inside one LXC (510) against three broker LXCs on the -same Proxmox CPU and vmbr1. Per-publisher -logs show a wide spread (example, 4p core 128 B: 79k–524k msgs/s among -the four pubs). That is CPU scheduling and client-side contention, not a -NATS cluster that only has one fast node.

-

1:1 at 128 B is the cleanest core number: -~791k pub, ~747k sub, ~1.20M aggregate. The cluster and -the bridge can move three-quarter-million small messages per second -fire-and-forget from a single client pair.

-

5.2 Payload size (core)

-
-Core NATS 128 B vs 1 KiB - -
-

Same 4p4s shape, two sizes:

- ------- - - - - - - - - - - - - - - - - - - - - - - - - - -
PayloadAggregate msgs/sAggregate MB/sPub msgs/sSub msgs/s
128 B1,521,256185.70316,3121,299,634
1 KiB630,460615.68247,747510,216
-

Message rate falls; byte rate rises (186 MB/s → 616 -MB/s aggregate). We are leaving the “tiny message, CPU/syscall bound” -region and entering “copying bytes across vmbr1.” Job JSON -and archive metadata sit nearer 128 B–1 KiB than megabyte blobs (blobs -are HTTP/WORM, not NATS payloads).

-

5.3 JetStream r=3 file

-
- - -
- ------- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RunPub msgs/sPub MB/sSub msgs/sNotes
js-1p-20k-128-r316,1551.97—publish-only
js-4p-50k-128-r316,6072.03—four pubs, same ceiling
js-4p-20k-1k-r313,49313.18—1 KiB still disk/replica bound
js-2p2s-20k-128-r310,9651.3410,942pull consumers keep up
-

Four publishers do not make JetStream four times -faster. 1p and 4p at 128 B are both ~16k msgs/s. The limiter is -synchronous replication to three file-backed replicas, -not client parallelism. That is the result we wanted to see: the bench -stream is behaving like a replicated log, not like core fan-out.

-

Pull consume (js-2p2s) is slightly slower on publish -(~11k) because the same run is also reading. Pub and sub stay matched -(10,965 vs 10,942): the consumer is not the straggler.

-

1 KiB durable write is ~13.5k msgs/s (~13.2 MB/s). Bytes go up; -message rate dips only a little. JetStream here is -ack/fdatasync/replica bound, not payload-copy bound, in -this size range.

-

5.4 Core vs -JetStream (same client, same cluster)

-
- - -
-

The log scale is required: core publish is ~15–50× -JetStream publish on this stand.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ShapeCore pub msgs/sJS r=3 file pub msgs/sRatio
1 publisher, 128 B791,09416,155~49×
4 publishers, 128 B316,31216,607~19×
4 publishers, 1 KiB247,74713,493~18×
-

This is not JetStream “losing.” Core is allowed to forget a message -the instant the server accepts it. JetStream on file with replicas=3 -must record it on a majority before the publish acks. -Our product streams (ZAPIER_*, VERAE_ARCHIVE) -chose that trade on purpose: a job event that survives one LXC dying is -worth ~16k msgs/s instead of ~800k.

-

If we ever need core-like rates for a signal that may drop, that -signal should not be on a replicated file stream.

-
-

6. Delay results

-
- - -
- ----------- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RunKindCountminavgp50p90p99max
lat-ping-1k-128ping (sequential RTT)10000.254 ms0.307 ms0.286 ms0.332 ms0.734 ms2.763 ms
lat-1p-5k-128flood5000149.3 ms238.6 ms248.8 ms274.3 ms279.4 ms279.7 ms
lat-4p-5k-1kflood5000155.1 ms211.7 ms217.6 ms223.3 ms227.8 ms228.4 ms
lat-4p-10k-128flood10000174.2 ms263.2 ms266.7 ms299.1 ms304.2 ms304.5 ms
lat-8p-20k-128flood20000304.6 ms453.7 ms466.3 ms499.9 ms505.1 ms505.6 ms
-

The dashed line on the chart is 1 ms. Only ping -lives there.

-

6.1 Ping — the quiet hop

-

One publisher, one subscriber, two connections, wait for each message -before sending the next.

-
    -
  • min 0.254 ms — guest → vmbr1 → a -nats-server → vmbr1 → guest.
    • -
  • p50 0.286 ms / avg 0.307 ms — typical.
    • -
  • p99 0.734 ms — still under a millisecond.
    • -
  • max 2.763 ms — one outlier in 1,000 samples (GC, -scheduler, or a slow route). Not the tail we design for.
    • -
-

A middleware jobs.watch publish followed by a waiter on -jobs.events is this shape when the poller is keeping up. -Compared with HTTPS to Zapier (tens to hundreds of milliseconds) or a -live Verae GET /api/status/{jobId}, NATS RTT is noise.

-

6.2 Flood — queueing under -burst

-

Publishers write the whole batch as fast as they can, then the -subscriber drains. Each message’s delay is “how long was I in the buffer -before the subscriber got to me?”

-

That is why:

-
    -
  • min is already ~150–300 ms — even the first -messages wait behind a burst that filled the socket/client queue.
    • -
  • p50 ≈ p99 ≈ max — a queue drain has a tight -distribution: everyone waits for roughly the same backlog.
    • -
  • 8p × 20k is ~450 ms avg — twice the messages of 4p -× 10k, roughly twice the wait. Linear in backlog, not in cluster -diameter.
    • -
-

Flood is not a measurement of NATS being slow. The -ping column proves the hop is ~0.3 ms. Flood is a measurement of -what operators will see if a consumer stalls -(job-events mailbox, webhook deliver, archive reply). Backlog time ≈ -queued_messages / consume_rate.

-

6.3 1 KiB flood vs 128 B flood

-

4 publishers, 5k messages at 1 KiB: avg 212 ms, -slightly faster than 4p 10k × 128 B (263 ms) because -the count is half, even though each message is 8× -larger. Again: delay here tracks how many messages are -queued, not payload size, in this range.

-
-

7. What this means for Verae × -Zapier

-

Product subjects on this cluster:

- ----- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
AddressKindBench analogue
verae.zapier.jobs.watchwork queue (JetStream)JS durable pub ~16k/s
verae.zapier.jobs.eventseventsJS + ping if waiters keep up; flood if they do not
verae.zapier.webhooks.deliverwork queueJS durable
verae.zapier.usageoptionalJS durable
verae.billing.*request-replyping (quiet RTT)
verae.archive.put / query / -reply.*JetStream + broadcast queryJS durable; query fan-out is closer to core but still JS-backed -puts
-

Capacity. 16k durable 128 B pubs/s is -~1.4×10⁹ messages/day if you could fill the pipe. We -will not. Zapier HTTPS, live api.veraetime.net, WORM bloom -checks, and human Zap runs sit far below that. This cluster is not the -product bottleneck on NS1.

-

Latency budget. A timestamp wait is: HTTP in → NATS -watch → poll Verae → NATS event → HTTP out (or REST Hook). The NATS -pieces are sub-millisecond when caught up. Do not spend -time “optimizing NATS RTT” until Zapier/Verae HTTP is in the same -band.

-

Backlogs. The failure mode that does show -up in these numbers is flood delay. If webhook-deliver or job-events -consumers pause (keep stopped, replica floor, a blocked HTTPS post to -hooks.zapier.com), waiters will see hundreds of -milliseconds to seconds of queue time. Fleet replica floors and -keep exist to prevent that, not because 0.3 ms is too slow.

-

Hardware move. Same three configs, three boxes, -private NIC. Expect:

-
    -
  • Core numbers to change with NIC and CPU (maybe up, maybe down).
    • -
  • JetStream numbers to change more, because they are -disk + fsync + replica RTT. Distinct SSDs should help; a slow shared -datastore would hurt.
    • -
  • Ping RTT to grow by whatever the real NIC and switch add (still -likely low milliseconds on a LAN).
    • -
-
-

8. Limits of this measurement

-
    -
  1. One Proxmox host. LXC 510–513 share cores, memory, -and the host’s disk. Replica=3 on file is three files on the -same underlying storage, not three failure domains. HA of “one -disk dies” is not proven. HA of “one LXC process dies” -is the actual claim.
    2. -
  2. Short runs. Tens of thousands of messages, seconds -of wall time. No compaction, no multi-hour page-cache eviction, no -snapshot/restore during load.
    3. -
  3. No TLS, no nkeys. verae-nats-accounts -is still a sketch. Auth would add CPU; it would not turn 16k into -800k.
    4. -
  4. One bench client. All publishers live in 510. A -fleet of workers on several CTs might publish more into JetStream until -disk/replicas saturate — the 1p vs 4p JS result says that saturation is -already ~16k from one CT.
    5. -
  5. nats 0.1.6 does not report delay. Anyone reading -min | avg | max msgs on a bench log as microseconds will -get the wrong story. Delay is only latency.mjs.
    6. -
  6. Core aggregate ≠ unique messages. Fan-out -double-counts. Use pub or sub columns when comparing to JetStream.
    7. -
  7. Not a Zapier or Verae API bench. Those are still -blocked on operator login / live credentials.
    8. -
-
-

9. How to reproduce

-

On NS1 (Proxmox), from the verae-nats-cluster -checkout:

-
bash scripts/status.sh    # 3/3 JetStream
-bash scripts/bench.sh     # writes results/<utc>/ and BENCH.md
-

The client VMID defaults to 510. Override with -CLIENT_VMID=…. NATS_URL comes from -client.env.

-

Rebuild this progress report (charts + HTML + PDF) from the -monorepo:

-
python3 packages/zapier-decisions/scripts/build-nats-bench-report.py
-
-

10. Appendix — environment and -files

- ---- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ItemValue
Run stamp20260912T045131Z
ClientLXC 510 verae-px-worker 10.10.10.20
Servers511/512/513 nats-a/b/c 10.10.10.21–23
nats CLI0.1.6 linux-amd64
JS storagefile, replicas=3, stream benchstream (deleted between -loads)
Isolationvmbr1 only; no 0.0.0.0 client bind
Short tablesBENCH.md
Raw logspackages/verae-nats-cluster/results/20260912T045131Z/
This reportpackages/zapier-decisions/reports/nats-cluster-bench.{md,html,pdf}
-

Publisher rate spread (nats CLI, msgs/s, not -delay):

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Runminavgmax
core-4p4s-100k-128 pub79,260257,805524,453
core-8p8s-200k-128 pub41,74471,488152,536
core-4p4s-50k-1k pub61,936110,271176,262
js-4p-50k-128-r3 pub4,1545,1766,628
js-4p-20k-1k-r3 pub3,3734,0635,121
js-2p2s-20k-128-r3 pub5,4857,0818,678
-

Wide core spreads are the single-client-CT effect described in §5.1. -JetStream spreads are narrow and low — every publisher is waiting on the -same replicated write path.

- - diff --git a/reports/nats-cluster-bench.md b/reports/nats-cluster-bench.md deleted file mode 100644 index 8f5a70f..0000000 --- a/reports/nats-cluster-bench.md +++ /dev/null @@ -1,312 +0,0 @@ -**Progress report** · 2026-09-12 · run `20260912T045131Z` (UTC) - -This is the full write-up of the test-environment NATS cluster bench: what was measured, how, the numbers, the charts, and what they mean for Verae Time × Zapier. Short tables also live in [`verae-nats-cluster/BENCH.md`](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md). Raw logs and CSVs are in that repo under `results/20260912T045131Z/`. - ---- - -## 1. Executive summary - -The test cluster is three JetStream nodes on private `vmbr1` (LXC 511–513). The bench client is a **fourth** guest (LXC 510), so the numbers are cluster-plus-network, not a process talking to itself on loopback. - -Two different systems were measured, on purpose: - -| System | What it is | What we got | -|--------|------------|-------------| -| **Core NATS** | Fire-and-forget pub/sub. No disk, no replica ack. | About **0.75–2.0 million msgs/s** at 128 B, depending on fan-out. At 1 KiB, about **630k msgs/s** and **~616 MB/s** aggregate. | -| **JetStream file, replicas=3** | Durable, replicated — **this is what product streams use**. | About **16k durable 128 B pubs/s**, about **13.5k** at 1 KiB. Pull consume keeps up with publish at ~11k msgs/s each side. | -| **Ping delay** | One message at a time, publish then wait. | **avg 0.307 ms**, **p99 0.734 ms**, max 2.76 ms (1k × 128 B). | -| **Flood delay** | Publishers dump a batch; subscriber drains. | **150–505 ms**. That is **queueing under burst**, not wire time. | - -**For this product:** timestamp jobs, job events, webhooks, and archive puts go through JetStream r=3. Plan capacity against **~16k durable msgs/s** on this stand, not the million-msg core numbers. A quiet job-event hop is a fraction of a millisecond. If a mailbox falls behind, delay jumps into hundreds of milliseconds — that is the flood column. - -Core NATS is still useful: it is the ceiling for non-durable fan-out on this host, and it shows `vmbr1` and the nats-server processes are not the JetStream bottleneck. JetStream is. - ---- - -## 2. Why this test exists - -The lab cut the test environment over to the three-node cluster. Before treating that cluster as the message fabric for keep, fleet, middleware, billing, and archive workers, we needed: - -1. **Throughput at several loads** — one publisher vs many, 128 B vs 1 KiB, core vs durable. -2. **Delay characteristics** — both the quiet path (one message RTT) and the overloaded path (burst into a mailbox). -3. **A client that is not a nats-* server** — otherwise we would be measuring loopback on the broker. - -This is a **lab stand on one Proxmox host**, not three metal boxes. It answers “is this cluster in the right order of magnitude for our traffic?” It does not replace a soak test on dedicated disks. - ---- - -## 3. Topology - -```text - vmbr1 10.10.10.0/24 (not on vmbr0, not public) - ----------------------------------------------- - LXC 510 LXC 511 LXC 512 LXC 513 - verae-px-worker nats-a nats-b nats-c - 10.10.10.20 10.10.10.21 10.10.10.22 10.10.10.23 - bench client :4222 client :4222 :4222 - :6222 routes :6222 :6222 - :8222 loopback :8222 :8222 -``` - -- Cluster name: `verae`. Each server has two routes to the other two. -- Client URL: `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` -- HTTP monitor is **loopback :8222** inside each guest. Zapier cloud never talks to NATS. -- Product streams already on this cluster (`ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE`) use **file** storage and **replicas=3**. The JetStream bench used the same settings on a throwaway stream `benchstream`. -- Host `127.0.0.1:4222` is still listening on NS1; **clients no longer use it**. - -Credits for the stack: Scott Lindsey, George Lambert, NATS.IO, Grok-Code. - ---- - -## 4. Method - -### 4.1 Tools - -| Piece | Role | -|-------|------| -| `nats` CLI **0.1.6** | Throughput (`nats bench --no-progress --csv`). Its min/avg/max are **publisher rate spread**, not delay. | -| `scripts/latency.mjs` | Two connections, header timestamp `t`, delay = receive time − send time. | -| `scripts/bench.sh` | Runs the ladder from NS1 via `pct exec` on VMID 510. | -| `scripts/bench-report.py` | Turns logs into the short `BENCH.md` table. | - -Re-run on NS1, from `verae-nats-cluster`: - -```bash -bash scripts/bench.sh -``` - -### 4.2 Load ladder - -**Core NATS** (subject `bench.core.*`): - -| Run | Publishers | Subscribers | Messages | Payload | -|-----|------------|-------------|----------|---------| -| `core-1p1s-50k-128` | 1 | 1 | 50,000 | 128 B | -| `core-4p4s-100k-128` | 4 | 4 | 100,000 | 128 B | -| `core-8p8s-200k-128` | 8 | 8 | 200,000 | 128 B | -| `core-4p4s-50k-1k` | 4 | 4 | 50,000 | 1024 B | - -**JetStream** (`--js --storage file --replicas 3 --stream benchstream`). The stream is deleted between loads so the name never collides: - -| Run | Shape | Messages | Payload | -|-----|-------|----------|---------| -| `js-1p-20k-128-r3` | 1 publisher | 20,000 | 128 B | -| `js-4p-50k-128-r3` | 4 publishers | 50,000 | 128 B | -| `js-4p-20k-1k-r3` | 4 publishers | 20,000 | 1024 B | -| `js-2p2s-20k-128-r3` | 2 pub + 2 pull sub | 20,000 | 128 B | - -**Delay** (core subjects, two connections): - -| Run | Mode | Count | Pubs | Payload | -|-----|------|-------|------|---------| -| `lat-ping-1k-128` | **ping** — publish, wait for that message, repeat | 1,000 | 1 | 128 B | -| `lat-1p-5k-128` | **flood** — publish all, then drain | 5,000 | 1 | 128 B | -| `lat-4p-10k-128` | flood | 10,000 | 4 | 128 B | -| `lat-8p-20k-128` | flood | 20,000 | 8 | 128 B | -| `lat-4p-5k-1k` | flood | 5,000 | 4 | 1024 B | - -Ping answers “how long does one quiet hop take?” Flood answers “what happens to the last message if we burst N messages into a mailbox?” Those are different questions. Mixing them is how 0.3 ms and 400 ms get confused. - -### 4.3 How to read nats bench columns - -- **Pub msgs/s** — rate at which publishers finished their share. -- **Sub msgs/s** — rate at which subscribers finished. With several subscribers on the same subject, core NATS **fans out**, so sub rate can exceed pub rate. -- **Aggregate msgs/s** — nats CLI `NATS Pub/Sub stats` line (pub+sub work in one number). Useful as a headline; do not treat it as “the network carried this many unique messages.” -- Empty JetStream sub cells mean that run was publish-only (durable write, no consumer in the same process). - ---- - -## 5. Throughput results - -### 5.1 Core NATS - -![Core NATS throughput at four loads](charts/core-throughput.png) - -| Run | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s | -|-----|------------------|------------|----------|------------|----------| -| `core-1p1s-50k-128` | 1,200,836 | 791,094 | 96.57 | 747,461 | 91.24 | -| `core-4p4s-100k-128` | 1,521,256 | 316,312 | 38.61 | 1,299,634 | 158.65 | -| `core-8p8s-200k-128` | 2,007,937 | 333,957 | 40.77 | 1,790,736 | 218.60 | -| `core-4p4s-50k-1k` | 630,460 | 247,747 | 241.94 | 510,216 | 498.26 | - -**What this chart is saying.** Adding subscribers raises **aggregate** and **sub** rates because each published message is delivered to every subscriber. Publish rate does **not** climb the same way: 1 publisher at 128 B already pushes ~791k msgs/s; 4 and 8 publishers sit around 310–335k msgs/s **each process slower**, while fan-out on the sub side goes to 1.3M then 1.8M. - -That publisher slowdown is expected on this stand. The four/eight publisher processes and the four/eight subscribers all run **inside one LXC** (510) against three broker LXCs on the **same Proxmox CPU and `vmbr1`**. Per-publisher logs show a wide spread (example, 4p core 128 B: 79k–524k msgs/s among the four pubs). That is CPU scheduling and client-side contention, not a NATS cluster that only has one fast node. - -**1:1 at 128 B** is the cleanest core number: **~791k pub, ~747k sub, ~1.20M aggregate**. The cluster and the bridge can move three-quarter-million small messages per second fire-and-forget from a single client pair. - -### 5.2 Payload size (core) - -![Core NATS 128 B vs 1 KiB](charts/payload-size.png) - -Same 4p4s shape, two sizes: - -| Payload | Aggregate msgs/s | Aggregate MB/s | Pub msgs/s | Sub msgs/s | -|---------|------------------|----------------|------------|------------| -| 128 B | 1,521,256 | 185.70 | 316,312 | 1,299,634 | -| 1 KiB | 630,460 | 615.68 | 247,747 | 510,216 | - -Message rate falls; **byte rate rises** (186 MB/s → 616 MB/s aggregate). We are leaving the “tiny message, CPU/syscall bound” region and entering “copying bytes across `vmbr1`.” Job JSON and archive metadata sit nearer 128 B–1 KiB than megabyte blobs (blobs are HTTP/WORM, not NATS payloads). - -### 5.3 JetStream r=3 file - -![JetStream durable publish rate](charts/js-throughput.png) - -| Run | Pub msgs/s | Pub MB/s | Sub msgs/s | Notes | -|-----|------------|----------|------------|-------| -| `js-1p-20k-128-r3` | **16,155** | 1.97 | — | publish-only | -| `js-4p-50k-128-r3` | **16,607** | 2.03 | — | four pubs, same ceiling | -| `js-4p-20k-1k-r3` | **13,493** | 13.18 | — | 1 KiB still disk/replica bound | -| `js-2p2s-20k-128-r3` | 10,965 | 1.34 | 10,942 | pull consumers keep up | - -**Four publishers do not make JetStream four times faster.** 1p and 4p at 128 B are both ~16k msgs/s. The limiter is **synchronous replication to three file-backed replicas**, not client parallelism. That is the result we wanted to see: the bench stream is behaving like a replicated log, not like core fan-out. - -Pull consume (`js-2p2s`) is slightly slower on publish (~11k) because the same run is also reading. Pub and sub stay matched (10,965 vs 10,942): the consumer is not the straggler. - -1 KiB durable write is ~13.5k msgs/s (~13.2 MB/s). Bytes go up; message rate dips only a little. JetStream here is **ack/fdatasync/replica** bound, not payload-copy bound, in this size range. - -### 5.4 Core vs JetStream (same client, same cluster) - -![Core vs JetStream publish rate, log scale](charts/core-vs-js.png) - -The log scale is required: core publish is **~15–50×** JetStream publish on this stand. - -| Shape | Core pub msgs/s | JS r=3 file pub msgs/s | Ratio | -|-------|-----------------|------------------------|-------| -| 1 publisher, 128 B | 791,094 | 16,155 | ~49× | -| 4 publishers, 128 B | 316,312 | 16,607 | ~19× | -| 4 publishers, 1 KiB | 247,747 | 13,493 | ~18× | - -This is not JetStream “losing.” Core is allowed to forget a message the instant the server accepts it. JetStream on file with replicas=3 must **record it on a majority** before the publish acks. Our product streams (`ZAPIER_*`, `VERAE_ARCHIVE`) chose that trade on purpose: a job event that survives one LXC dying is worth ~16k msgs/s instead of ~800k. - -If we ever need core-like rates for a signal that may drop, that signal should not be on a replicated file stream. - ---- - -## 6. Delay results - -![Delay percentiles, log scale](charts/delay-percentiles.png) - -| Run | Kind | Count | min | avg | p50 | p90 | p99 | max | -|-----|------|-------|-----|-----|-----|-----|-----|-----| -| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 0.254 ms | **0.307 ms** | 0.286 ms | 0.332 ms | **0.734 ms** | 2.763 ms | -| `lat-1p-5k-128` | flood | 5000 | 149.3 ms | 238.6 ms | 248.8 ms | 274.3 ms | 279.4 ms | 279.7 ms | -| `lat-4p-5k-1k` | flood | 5000 | 155.1 ms | 211.7 ms | 217.6 ms | 223.3 ms | 227.8 ms | 228.4 ms | -| `lat-4p-10k-128` | flood | 10000 | 174.2 ms | 263.2 ms | 266.7 ms | 299.1 ms | 304.2 ms | 304.5 ms | -| `lat-8p-20k-128` | flood | 20000 | 304.6 ms | 453.7 ms | 466.3 ms | 499.9 ms | 505.1 ms | 505.6 ms | - -The dashed line on the chart is 1 ms. Only **ping** lives there. - -### 6.1 Ping — the quiet hop - -One publisher, one subscriber, two connections, wait for each message before sending the next. - -- **min 0.254 ms** — guest → `vmbr1` → a nats-server → `vmbr1` → guest. -- **p50 0.286 ms / avg 0.307 ms** — typical. -- **p99 0.734 ms** — still under a millisecond. -- **max 2.763 ms** — one outlier in 1,000 samples (GC, scheduler, or a slow route). Not the tail we design for. - -A middleware `jobs.watch` publish followed by a waiter on `jobs.events` is this shape when the poller is keeping up. Compared with HTTPS to Zapier (tens to hundreds of milliseconds) or a live Verae `GET /api/status/{jobId}`, NATS RTT is noise. - -### 6.2 Flood — queueing under burst - -Publishers write the whole batch as fast as they can, then the subscriber drains. Each message’s delay is “how long was I in the buffer before the subscriber got to me?” - -That is why: - -- **min is already ~150–300 ms** — even the first messages wait behind a burst that filled the socket/client queue. -- **p50 ≈ p99 ≈ max** — a queue drain has a tight distribution: everyone waits for roughly the same backlog. -- **8p × 20k is ~450 ms avg** — twice the messages of 4p × 10k, roughly twice the wait. Linear in backlog, not in cluster diameter. - -Flood is **not** a measurement of NATS being slow. The ping column proves the hop is ~0.3 ms. Flood is a measurement of **what operators will see if a consumer stalls** (job-events mailbox, webhook deliver, archive reply). Backlog time ≈ `queued_messages / consume_rate`. - -### 6.3 1 KiB flood vs 128 B flood - -4 publishers, 5k messages at 1 KiB: avg **212 ms**, slightly **faster** than 4p 10k × 128 B (263 ms) because the **count is half**, even though each message is 8× larger. Again: delay here tracks **how many messages are queued**, not payload size, in this range. - ---- - -## 7. What this means for Verae × Zapier - -Product subjects on this cluster: - -| Address | Kind | Bench analogue | -|---------|------|----------------| -| `verae.zapier.jobs.watch` | work queue (JetStream) | JS durable pub ~16k/s | -| `verae.zapier.jobs.events` | events | JS + ping if waiters keep up; flood if they do not | -| `verae.zapier.webhooks.deliver` | work queue | JS durable | -| `verae.zapier.usage` | optional | JS durable | -| `verae.billing.*` | request-reply | ping (quiet RTT) | -| `verae.archive.put` / `query` / `reply.*` | JetStream + broadcast query | JS durable; query fan-out is closer to core but still JS-backed puts | - -**Capacity.** 16k durable 128 B pubs/s is **~1.4×10⁹ messages/day** if you could fill the pipe. We will not. Zapier HTTPS, live `api.veraetime.net`, WORM bloom checks, and human Zap runs sit far below that. This cluster is not the product bottleneck on NS1. - -**Latency budget.** A timestamp wait is: HTTP in → NATS watch → poll Verae → NATS event → HTTP out (or REST Hook). The NATS pieces are **sub-millisecond** when caught up. Do not spend time “optimizing NATS RTT” until Zapier/Verae HTTP is in the same band. - -**Backlogs.** The failure mode that *does* show up in these numbers is flood delay. If webhook-deliver or job-events consumers pause (keep stopped, replica floor, a blocked HTTPS post to `hooks.zapier.com`), waiters will see **hundreds of milliseconds to seconds** of queue time. Fleet replica floors and keep exist to prevent that, not because 0.3 ms is too slow. - -**Hardware move.** Same three configs, three boxes, private NIC. Expect: - -- Core numbers to change with NIC and CPU (maybe up, maybe down). -- JetStream numbers to change **more**, because they are disk + fsync + replica RTT. Distinct SSDs should help; a slow shared datastore would hurt. -- Ping RTT to grow by whatever the real NIC and switch add (still likely low milliseconds on a LAN). - ---- - -## 8. Limits of this measurement - -1. **One Proxmox host.** LXC 510–513 share cores, memory, and the host’s disk. Replica=3 on file is **three files on the same underlying storage**, not three failure domains. HA of “one disk dies” is **not** proven. HA of “one LXC process dies” is the actual claim. -2. **Short runs.** Tens of thousands of messages, seconds of wall time. No compaction, no multi-hour page-cache eviction, no snapshot/restore during load. -3. **No TLS, no nkeys.** `verae-nats-accounts` is still a sketch. Auth would add CPU; it would not turn 16k into 800k. -4. **One bench client.** All publishers live in 510. A fleet of workers on several CTs might publish more into JetStream until disk/replicas saturate — the 1p vs 4p JS result says that saturation is already ~16k from one CT. -5. **nats 0.1.6** does not report delay. Anyone reading `min | avg | max msgs` on a bench log as microseconds will get the wrong story. Delay is only `latency.mjs`. -6. **Core aggregate ≠ unique messages.** Fan-out double-counts. Use pub or sub columns when comparing to JetStream. -7. **Not a Zapier or Verae API bench.** Those are still blocked on operator login / live credentials. - ---- - -## 9. How to reproduce - -On NS1 (Proxmox), from the `verae-nats-cluster` checkout: - -```bash -bash scripts/status.sh # 3/3 JetStream -bash scripts/bench.sh # writes results// and BENCH.md -``` - -The client VMID defaults to **510**. Override with `CLIENT_VMID=…`. `NATS_URL` comes from `client.env`. - -Rebuild this progress report (charts + HTML + PDF) from the monorepo: - -```bash -python3 packages/zapier-decisions/scripts/build-nats-bench-report.py -``` - ---- - -## 10. Appendix — environment and files - -| Item | Value | -|------|-------| -| Run stamp | `20260912T045131Z` | -| Client | LXC 510 `verae-px-worker` `10.10.10.20` | -| Servers | 511/512/513 `nats-a/b/c` `10.10.10.21–23` | -| nats CLI | 0.1.6 linux-amd64 | -| JS storage | file, replicas=3, stream `benchstream` (deleted between loads) | -| Isolation | `vmbr1` only; no `0.0.0.0` client bind | -| Short tables | [BENCH.md](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md) | -| Raw logs | `packages/verae-nats-cluster/results/20260912T045131Z/` | -| This report | `packages/zapier-decisions/reports/nats-cluster-bench.{md,html,pdf}` | - -Publisher rate spread (nats CLI, msgs/s, **not** delay): - -| Run | min | avg | max | -|-----|-----|-----|-----| -| core-4p4s-100k-128 pub | 79,260 | 257,805 | 524,453 | -| core-8p8s-200k-128 pub | 41,744 | 71,488 | 152,536 | -| core-4p4s-50k-1k pub | 61,936 | 110,271 | 176,262 | -| js-4p-50k-128-r3 pub | 4,154 | 5,176 | 6,628 | -| js-4p-20k-1k-r3 pub | 3,373 | 4,063 | 5,121 | -| js-2p2s-20k-128-r3 pub | 5,485 | 7,081 | 8,678 | - -Wide core spreads are the single-client-CT effect described in §5.1. JetStream spreads are narrow and low — every publisher is waiting on the same replicated write path. diff --git a/reports/nats-cluster-bench.pdf b/reports/nats-cluster-bench.pdf deleted file mode 100644 index ff11f5a..0000000 Binary files a/reports/nats-cluster-bench.pdf and /dev/null differ diff --git a/reports/ns1-host/charts/core-throughput.png b/reports/ns1-host/charts/core-throughput.png deleted file mode 100644 index 44609c2..0000000 Binary files a/reports/ns1-host/charts/core-throughput.png and /dev/null differ diff --git a/reports/ns1-host/charts/core-vs-js.png b/reports/ns1-host/charts/core-vs-js.png deleted file mode 100644 index dae2b58..0000000 Binary files a/reports/ns1-host/charts/core-vs-js.png and /dev/null differ diff --git a/reports/ns1-host/charts/delay-percentiles.png b/reports/ns1-host/charts/delay-percentiles.png deleted file mode 100644 index f57cf44..0000000 Binary files a/reports/ns1-host/charts/delay-percentiles.png and /dev/null differ diff --git a/reports/ns1-host/charts/js-throughput.png b/reports/ns1-host/charts/js-throughput.png deleted file mode 100644 index 1edaebe..0000000 Binary files a/reports/ns1-host/charts/js-throughput.png and /dev/null differ diff --git a/reports/ns1-host/charts/payload-size.png b/reports/ns1-host/charts/payload-size.png deleted file mode 100644 index 5c64356..0000000 Binary files a/reports/ns1-host/charts/payload-size.png and /dev/null differ diff --git a/reports/ns1-host/nats-cluster-bench-ns1.html b/reports/ns1-host/nats-cluster-bench-ns1.html deleted file mode 100644 index 59b896e..0000000 --- a/reports/ns1-host/nats-cluster-bench-ns1.html +++ /dev/null @@ -1,1291 +0,0 @@ - - - - - - - NATS cluster message speed — NS1 host study - - - - -
Verae Time × Zapier · progress report · run on NS1.GEORGELAMBERT.ORG

NATS cluster message speed — NS1 host study

packages/zapier-decisions/reports/nats-cluster-bench-ns1.md
-
-

NATS cluster message speed — NS1 host study

-
-

Progress report (second study) · run -20260912T051237Z (UTC)

-
-

Execution provenance. Every process for this study -ran on NS1.GEORGELAMBERT.ORG -(70.88.205.138): the orchestrator -(study-on-ns1.sh), nats bench, -latency.mjs (inside LXC 510 on this hypervisor), charting -(matplotlib), and HTML/PDF (pandoc + -weasyprint). The operator laptop did not -publish, subscribe, draw charts, or render the PDF. Traffic stayed on -vmbr1 from LXC 510 to -nats-a/b/c (511–513).

-
-

This is a full methodology write-up plus the numbers from that -on-host run. The earlier report (nats-cluster-bench, run -20260912T045131Z) used the same cluster but was -orchestrated and rendered off-box. Use this document -when you need “it was all run on 138.”

-
-

1. Executive summary

- ---- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ItemThis NS1-host run
Control planeNS1.GEORGELAMBERT.ORG (70.88.205.138), user -marchon
Bench clientLXC 510 verae-px-worker
BrokersLXC 511/512/513 nats-a/b/c on -10.10.10.21–23
Client URLnats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
Host load before8.59 8.39 8.15 5/3843 4096277
Host load after8.67 8.33 8.15 9/3863 4114865
Core 1p1s 128 B pub502,502 msgs/s
JetStream 1p 128 B r=37,393 durable pubs/s
Ping p50 / p990.395ms / 1.377ms
-

Product traffic is the JetStream row. Ping is one-message delay. -Flood is mailbox catch-up after a burst.

-
-

2. Where it ran (and where it -did not)

-
Operator laptop  ──ssh──►  NS1.GEORGELAMBERT.ORG  70.88.205.138
-                              study-on-ns1.sh
-                              python3 build-ns1-study-report.py
-                              sudo pct exec 510 ──► nats bench / latency.mjs
-                                                    │
-                                                    ▼ vmbr1
-                                              10.10.10.21-23 :4222
-
    -
  • Did run on 138: bash, python3, matplotlib, pandoc, -weasyprint, pct, nats-server (in LXC), nats CLI and Node -(in LXC 510).
    • -
  • Did not run on the laptop: no local -nats bench, no local charting, no local WeasyPrint for this -file.
    • -
-
-

3. Results (this run)

-

Host and brokers

-

Before

- ----------- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
NodeVMIDconnectionsin_msgsout_msgscpucoresmem (B)jetstream
nats-a51121,721,2034,933,7591126,398,720True
nats-b51221,613,2053,513,0960123,097,344True
nats-c51321,760,9734,048,5822123,867,392True
-

After

- ----------- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
NodeVMIDconnectionsin_msgsout_msgscpucoresmem (B)jetstream
nats-a51122,097,5965,910,1611138,273,024True
nats-b51222,121,8164,746,5391141,361,408True
nats-c51322,241,3665,053,9781138,162,432True
-

nproc=40 · -uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux

-

Throughput

- --------- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RunModeAggregate msgs/sPub msgs/sPub MB/sSub msgs/sSub MB/s
core-1p1s-50k-128core pub/sub776,331502,50261.34420,97651.39
core-4p4s-100k-128core pub/sub1,278,454379,98546.381,078,910131.70
core-4p4s-50k-1kcore pub/sub581,464209,545204.63482,645471.33
core-8p8s-200k-128core pub/sub2,065,217333,73140.741,847,242225.49
js-1p-20k-128-r3jetstream r=3 file—7,3930.90——
js-2p2s-20k-128-r3jetstream r=3 file15,4167,7280.947,7110.94
js-4p-20k-1k-r3jetstream r=3 file—14,98514.63——
js-4p-50k-128-r3jetstream r=3 file—17,9862.20——
-

Round-trip delay

- ------------- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RunKindCountPubsSizeminavgp50p90p99max
lat-ping-1k-128ping (sequential RTT)10001128 B0.341ms0.470ms0.395ms0.651ms1.377ms3.243ms
lat-1p-5k-128flood (burst queueing)50001128 B132.425ms203.717ms204.458ms241.750ms249.468ms249.640ms
lat-4p-5k-1kflood (burst queueing)500041024 B196.422ms221.264ms224.039ms233.180ms236.798ms238.177ms
lat-4p-10k-128flood (burst queueing)100004128 B211.108ms294.925ms300.254ms314.223ms315.551ms316.091ms
lat-8p-20k-128flood (burst queueing)200008128 B280.608ms414.972ms409.802ms513.821ms535.394ms536.084ms
-

Core NATS

-
- - -
-

Core NATS throughput at four loads (NS1 host run) ### -Payload size (core)

-
- - -
-

Core NATS 128 B vs 1 KiB (NS1 host run) ### JetStream r=3 -file

-
- - -
-

JetStream durable publish rate (NS1 host run) ### Core vs -JetStream

-
- - -
-

Core vs JetStream publish rate, log scale (NS1 host run) ### -Delay

-
- - -
-

Ping vs flood delay percentiles, log scale (NS1 host -run)

-
-

4. Study methodology

-

4.1 Question

-

On the NS1 test stand, what message throughput and -delay does the three-node verae JetStream -cluster deliver at several loads, and which part of the stack is the -limiter for product traffic (jobs, events, webhooks, archive)?

-

4.2 Hypotheses (stated before -the run)

-
    -
  1. H1 — Core vs JetStream. Fire-and-forget core NATS -is at least an order of magnitude faster than JetStream file + -replicas=3, because durable publish waits for a majority disk -replica.
    2. -
  2. H2 — JetStream parallelism. Adding publishers does -not linearly increase JetStream write rate once the -replica log is saturated.
    3. -
  3. H3 — Quiet delay. Sequential pub→sub round trip on -vmbr1 is well under 1 ms p99 when the consumer is -waiting.
    4. -
  4. H4 — Burst delay. If publishers dump a batch before -the subscriber drains, observed delay is queueing time, -roughly linear in backlog, not in cluster hop count.
    5. -
  5. H5 — Payload. Moving 128 B → 1 KiB lowers message -rate and raises byte rate on core NATS; JetStream in this size band -stays replica/fsync bound.
    6. -
-

4.3 Independent variables -(what we changed)

- ---- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FactorLevels
TransportCore NATS pub/sub vs JetStream file replicas=3
Publisher count1, 2, 4, 8
Subscriber count0 (JS publish-only), 1, 2, 4, 8
Message count1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step)
Payload128 B, 1024 B
Delay modeping (publish, wait, repeat) vs -flood (publish all, then drain)
-

4.4 Dependent variables -(what we recorded)

- ----- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MetricInstrumentUnit
Publish ratenats bench 0.1.6 Pub statsmsgs/s, MB/s
Subscribe ratenats bench Sub statsmsgs/s, MB/s
Aggregatenats bench NATS Pub/Sub statsmsgs/s (fan-out counts both sides)
Publisher spreadnats min/avg/max msgs/snot delay
One-way-ish RTTlatency.mjs header timestampmin, avg, p50, p90, p99, max
Host load/proc/loadavg before and afterload average
Broker countershttp://127.0.0.1:8222/varz inside each nats LXCconnections, in/out msgs, cpu, mem
-

Important: nats CLI 0.1.6 min/avg/max are -rate spread across publishers, not microseconds of -delay. Delay is only latency.mjs.

-

4.5 Controls and constants

-
    -
  • Cluster name verae, three routes, client -:4222, cluster :6222, monitor loopback -:8222.
    • -
  • Client URL always the three-node list on vmbr1 (never -host 127.0.0.1:4222, never vmbr0).
    • -
  • Bench client is LXC 510, not a nats-* server.
    • -
  • JetStream bench stream name benchstream, -file storage, replicas=3, deleted -between JS loads (nats stream rm --force) so names do not -collide.
    • -
  • Product streams were not the bench target (no load -test on ZAPIER_* / VERAE_ARCHIVE).
    • -
  • No TLS, no nkeys, no account isolation (isolation is -vmbr1).
    • -
  • Same nats CLI version (0.1.6) and nats@2 Node client as -the first ladder.
    • -
-

4.6 Procedure

-
    -
  1. Confirm this script is executing on -NS1.GEORGELAMBERT.ORG. Refuse otherwise.
    2. -
  2. Snapshot host load, memory, LXC configs, and each nats -varz.
    3. -
  3. From NS1, pct exec 510 the core ladder (1p1s, 4p4s, -8p8s at 128 B; 4p4s at 1 KiB).
    4. -
  4. Delete benchstream; JS ladder (1p, 4p, 4p×1 KiB, 2p2s -pull) at replicas=3 file.
    5. -
  5. Copy latency.mjs into 510; ping then flood at several -batch sizes.
    6. -
  6. Snapshot host/varz again.
    7. -
  7. Parse logs on this host; draw charts; write HTML -and PDF on this host.
    8. -
-

No publish, subscribe, chart, or PDF process runs on the operator -laptop for this study.

-

4.7 Instrumentation path

-
[NS1 host 70.88.205.138]
-   study-on-ns1.sh  (bash + python3)
-        |
-        | sudo pct exec 510
-        v
-[LXC 510 verae-px-worker 10.10.10.20]
-   nats bench / node latency.mjs
-        |
-        | NATS client protocol to
-        v
-[LXC 511/512/513  10.10.10.21-23 :4222]
-   nats-server -js  cluster routes :6222
-

The hypervisor issues the guest commands. The messages themselves -never leave vmbr1.

-

4.8 Threats to validity

- ---- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ThreatEffect on numbers
One physical hostThree “replicas” share CPU, memory, and usually the same datastore. -This measures process/LXC HA, not disk HA.
Shared loadNS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. -Load average during a run is part of the result, not noise to -ignore.
Single bench clientAll publishers live in 510. Per-publisher rate spread is contention -in that guest.
Short runsSeconds of traffic. No compaction, no multi-hour page-cache -eviction, no snapshot during load.
No TLS/nkeysProduction auth will cost CPU. Do not treat these rates as -post-nkeys rates.
Fan-out aggregateCore aggregate msgs/s counts pub+sub. Do not compare that column to -JetStream unique writes.
Flood ≠ RTTMixing flood averages with ping p99 produces a fake “NATS is slow” -story.
Lab onlyNot a Zapier HTTPS bench and not live -api.veraetime.net.
-

4.9 Ethics / safety

-

Bench uses throwaway subjects (bench.core.*, -bench.js.*, bench.lat.*) and a throwaway -stream. It does not purge product streams. Zapier cloud has no NATS -socket.

-
-

5. Suggestions for fine-tuning

-

These follow from the method and from the first ladder on this stand -(JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of -ms). Apply in order of leverage. Re-run this NS1 study -after each change so the delta is measured the same way.

-

5.1 Treat JetStream as -the product limiter

-

Product jobs/events/webhooks/archive are durable. Tuning core NATS to -2M msgs/s will not move a timestamp Zap. Put effort into replica -write path and consumer lag, not core -fan-out.

-

5.2 Split storage class by -stream

- ----- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
StreamSuggested storeWhy
ZAPIER_JOBSfile, r=3Work queue; lose-a-job is bad
ZAPIER_EVENTSfile r=3, or memory r=3 if events are rebuildable from job -statusHot waiters; measure both
ZAPIER_WEBHOOKSfile, r=3, workqueueHTTPS to Zapier is the slow consumer
ZAPIER_USAGEfile, r=3, limits + max-ageTelemetry
VERAE_ARCHIVEfile, r=3, on the best diskPuts are larger and must survive
-

Try ZAPIER_EVENTS as memory store in a maintenance -window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms -and durable events still ack at a higher rate, keep it; if a CT restart -drops in-flight waiters, revert.

-

5.3 Give JetStream real disks

-

Today r=3 on three LXC guests on one Proxmox host is -three files, one failure domain.

-
    -
  • Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) -into each nats LXC store_dir.
    • -
  • Set sync: always only on archive if you need it; -default sync is often enough for jobs and is faster. Measure.
    • -
  • Do not put JetStream store_dir on the same busy rootfs -as Forgejo/Caddy if we can avoid it.
    • -
  • When moving to three metal boxes: same configs, private NIC, one -disk (or mirror) per node. That is the first change -that makes r=3 mean “two boxes can die.”
    • -
-

5.4 Isolate the nats -CTs from the rest of NS1

-

Host load on this box is often already several. Pin:

-
    -
  • nats-a/b/c: dedicated cores, no steal from keep/fleet -Node processes.
    • -
  • Memory high enough that file-backed streams stay cache-hot for the -working set.
    • -
  • cpuunits / cpuset in pct config so a -Zapier-facing Node GC pause does not stall fsync.
    • -
-

Re-run this study after pinning; H1/H2 should move more than -ping.

-

5.5 Consumer and mailbox -tuning (delay H4)

-

Flood delay is backlog / consume_rate. Fine-tune the -waiters, not the broker RTT.

-
    -
  • jobs.events and webhooks.deliver: raise -max_ack_pending so a slow HTTPS hook does not stall the -whole consumer; cap it so a poison message cannot unbounded-buffer -RAM.
    • -
  • Pull consumers: larger batch, shorter expires, more -pullers horizontally (fleet replica floors) instead of one fat -subscriber.
    • -
  • Middleware should not flood-publish then wait; it -already does per-job publish. Keep that. The flood test is the outage -profile when a consumer is stopped.
    • -
  • Alert on consumer lag (pending + ack pending) from -JetStream, not on ping RTT.
    • -
-

5.6 Publisher-side -batching in middleware

-

A timestamp job is one small JSON. 16k msgs/s is ample. Still:

-
    -
  • Avoid per-byte publishes; one message per job/event.
    • -
  • Reuse NATS connections (connection churn showed up as publisher -spread in the core 4p/8p runs).
    • -
  • Idempotent msg id / duplicate window sized to Verae -retry window, not default-only.
    • -
-

5.7 -nats-server knobs worth measuring (A/B with this script)

- ---- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
KnobWhy try it
max_payloadKeep default unless archive puts grow
write_deadlineSlow consumer protection for webhooks
max_pendingBound memory on a stuck Zapier hook
max_connectionsFleet workers + keep + middleware
JetStream max_file_store / -max_memory_storePrevent one stream from filling the CT
max_outstanding_catchupReplica restart after a nats-c blip
GOMAXPROCS = LXC coresDo not overthread a 2-core CT
-

Change one knob, re-run -study-on-ns1.sh, compare JetStream 1p 128 B and ping -p99.

-

5.8 Network

-
    -
  • Keep NATS off vmbr0. No change.
    • -
  • When on metal: dedicated NIC or VLAN for cluster :6222 -vs client :4222 if possible (replication vs client -load).
    • -
  • Check virtio queue counts on the LXC nics if core 1 KiB byte rate -plateaus.
    • -
-

5.9 Security cost (when -nkeys/mTLS flip)

-

verae-nats-accounts is still a sketch. Enabling accounts -will add CPU on publish. Budget: re-run this exact study -after creds are in every NATS_URL, and -accept a drop on both core and JS. Do not flip without that -measurement.

-

5.10 Operational -fine-tuning (lag, not peak msgs/s)

-
    -
  1. Scrape varz / jsz from the host over -vmbr1 (not public). Monitor loopback :8222 is -invisible to Prometheus on NS1 unless we add a host-side proxy on -10.10.10.21:8222 bound only to vmbr1.
    2. -
  2. Keep replica floors for webhook-deliver and job-poller — they are -the flood defense.
    3. -
  3. Backup/restore drill of JetStream during idle, then -a short JS 1p run to see catchup cost.
    4. -
  4. A 15–30 minute soak (not in this ladder) for page cache and -compaction; add that as a third study when disks are dedicated.
    5. -
-

5.11 What not to tune

-
    -
  • Do not chase core 8p8s aggregate. It is fan-out on a lab -bridge.
    • -
  • Do not treat flood 400 ms as “cluster RTT.” Fix consumers.
    • -
  • Do not load-test on ZAPIER_* streams.
    • -
  • Do not bind client NATS to 0.0.0.0 on -vmbr0.
    • -
- -
    -
  1. CPU pin nats-a/b/c → re-run JS 1p + ping.
    2. -
  2. ZAPIER_EVENTS-shaped memory stream vs file (throwaway -stream, same flags as this JS ladder).
    3. -
  3. Distinct store_dir disks per node.
    4. -
  4. nkeys on, same ladder.
    5. -
  5. Three hardware boxes, same cluster.env IPs -updated.
    6. -
-

Each experiment should produce a new -results/<utc>/ on NS1 and a new progress-repo report -so we can diff H1–H5 instead of arguing from memory.

-
-

6. Reproducing this study

-

On NS1 only:

-
cd ~/verae-src/verae-nats-cluster
-bash scripts/study-on-ns1.sh
-

The script exits if hostname is not NS1. Outputs land in -results/<utc>/ including -nats-cluster-bench-ns1.{md,html,pdf} and -charts/. Copy those into -zapier-decisions/reports/ for the progress repo and -catalog.

-

Raw logs for this run: results/20260912T051237Z/.

- - diff --git a/reports/ns1-host/nats-cluster-bench-ns1.md b/reports/ns1-host/nats-cluster-bench-ns1.md deleted file mode 100644 index cf0011d..0000000 --- a/reports/ns1-host/nats-cluster-bench-ns1.md +++ /dev/null @@ -1,334 +0,0 @@ -**Progress report (second study)** · run `20260912T051237Z` (UTC) - -> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): the orchestrator (`study-on-ns1.sh`), `nats bench`, `latency.mjs` (inside LXC 510 on this hypervisor), charting (`matplotlib`), and HTML/PDF (`pandoc` + `weasyprint`). The operator laptop did **not** publish, subscribe, draw charts, or render the PDF. Traffic stayed on `vmbr1` from LXC **510** to `nats-a/b/c` (**511–513**). - -This is a full methodology write-up plus the numbers from that on-host run. The earlier report (`nats-cluster-bench`, run `20260912T045131Z`) used the same cluster but was **orchestrated and rendered off-box**. Use this document when you need “it was all run on 138.” - ---- - -## 1. Executive summary - -| Item | This NS1-host run | -|------|-------------------| -| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `marchon` | -| Bench client | LXC 510 `verae-px-worker` | -| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` | -| Client URL | `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` | -| Host load before | `8.59 8.39 8.15 5/3843 4096277` | -| Host load after | `8.67 8.33 8.15 9/3863 4114865` | -| Core 1p1s 128 B pub | 502,502 msgs/s | -| JetStream 1p 128 B r=3 | 7,393 durable pubs/s | -| Ping p50 / p99 | 0.395ms / 1.377ms | - -Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst. - ---- - -## 2. Where it ran (and where it did not) - -```text -Operator laptop ──ssh──► NS1.GEORGELAMBERT.ORG 70.88.205.138 - study-on-ns1.sh - python3 build-ns1-study-report.py - sudo pct exec 510 ──► nats bench / latency.mjs - │ - ▼ vmbr1 - 10.10.10.21-23 :4222 -``` - -- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510). -- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file. - ---- - -## 3. Results (this run) - -### Host and brokers - -**Before** - -| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream | -|------|------|-------------|---------|----------|-----|-------|---------|-----------| -| nats-a | 511 | 2 | 1,721,203 | 4,933,759 | 1 | 1 | 26,398,720 | True | -| nats-b | 512 | 2 | 1,613,205 | 3,513,096 | 0 | 1 | 23,097,344 | True | -| nats-c | 513 | 2 | 1,760,973 | 4,048,582 | 2 | 1 | 23,867,392 | True | - -**After** - -| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream | -|------|------|-------------|---------|----------|-----|-------|---------|-----------| -| nats-a | 511 | 2 | 2,097,596 | 5,910,161 | 1 | 1 | 38,273,024 | True | -| nats-b | 512 | 2 | 2,121,816 | 4,746,539 | 1 | 1 | 41,361,408 | True | -| nats-c | 513 | 2 | 2,241,366 | 5,053,978 | 1 | 1 | 38,162,432 | True | - -nproc=40 · uname=`Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux` - -### Throughput - -| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s | -|-----|------|------------------|------------|----------|------------|----------| -| `core-1p1s-50k-128` | core pub/sub | 776,331 | 502,502 | 61.34 | 420,976 | 51.39 | -| `core-4p4s-100k-128` | core pub/sub | 1,278,454 | 379,985 | 46.38 | 1,078,910 | 131.70 | -| `core-4p4s-50k-1k` | core pub/sub | 581,464 | 209,545 | 204.63 | 482,645 | 471.33 | -| `core-8p8s-200k-128` | core pub/sub | 2,065,217 | 333,731 | 40.74 | 1,847,242 | 225.49 | -| `js-1p-20k-128-r3` | jetstream r=3 file | — | 7,393 | 0.90 | — | — | -| `js-2p2s-20k-128-r3` | jetstream r=3 file | 15,416 | 7,728 | 0.94 | 7,711 | 0.94 | -| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 14,985 | 14.63 | — | — | -| `js-4p-50k-128-r3` | jetstream r=3 file | — | 17,986 | 2.20 | — | — | - -### Round-trip delay - -| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max | -|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----| -| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.341ms | 0.470ms | 0.395ms | 0.651ms | 1.377ms | 3.243ms | -| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 132.425ms | 203.717ms | 204.458ms | 241.750ms | 249.468ms | 249.640ms | -| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 196.422ms | 221.264ms | 224.039ms | 233.180ms | 236.798ms | 238.177ms | -| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 211.108ms | 294.925ms | 300.254ms | 314.223ms | 315.551ms | 316.091ms | -| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 280.608ms | 414.972ms | 409.802ms | 513.821ms | 535.394ms | 536.084ms | - -### Core NATS - -![Core NATS throughput at four loads (NS1 host run)](charts/core-throughput.png) - -*Core NATS throughput at four loads (NS1 host run)* -### Payload size (core) - -![Core NATS 128 B vs 1 KiB (NS1 host run)](charts/payload-size.png) - -*Core NATS 128 B vs 1 KiB (NS1 host run)* -### JetStream r=3 file - -![JetStream durable publish rate (NS1 host run)](charts/js-throughput.png) - -*JetStream durable publish rate (NS1 host run)* -### Core vs JetStream - -![Core vs JetStream publish rate, log scale (NS1 host run)](charts/core-vs-js.png) - -*Core vs JetStream publish rate, log scale (NS1 host run)* -### Delay - -![Ping vs flood delay percentiles, log scale (NS1 host run)](charts/delay-percentiles.png) - -*Ping vs flood delay percentiles, log scale (NS1 host run)* - ---- - -## 4. Study methodology - -### 4.1 Question - -On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)? - -### 4.2 Hypotheses (stated before the run) - -1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica. -2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated. -3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting. -4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count. -5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound. - -### 4.3 Independent variables (what we changed) - -| Factor | Levels | -|--------|--------| -| Transport | Core NATS pub/sub vs JetStream file replicas=3 | -| Publisher count | 1, 2, 4, 8 | -| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 | -| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) | -| Payload | 128 B, 1024 B | -| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) | - -### 4.4 Dependent variables (what we recorded) - -| Metric | Instrument | Unit | -|--------|------------|------| -| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s | -| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s | -| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) | -| Publisher spread | nats min/avg/max **msgs/s** | not delay | -| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max | -| Host load | `/proc/loadavg` before and after | load average | -| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem | - -**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`. - -### 4.5 Controls and constants - -- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`. -- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`). -- Bench client is LXC **510**, not a nats-* server. -- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide. -- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`). -- No TLS, no nkeys, no account isolation (isolation is `vmbr1`). -- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder. - -### 4.6 Procedure - -1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise. -2. Snapshot host load, memory, LXC configs, and each nats `varz`. -3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB). -4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file. -5. Copy `latency.mjs` into 510; ping then flood at several batch sizes. -6. Snapshot host/`varz` again. -7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**. - -No publish, subscribe, chart, or PDF process runs on the operator laptop for this study. - -### 4.7 Instrumentation path - -```text -[NS1 host 70.88.205.138] - study-on-ns1.sh (bash + python3) - | - | sudo pct exec 510 - v -[LXC 510 verae-px-worker 10.10.10.20] - nats bench / node latency.mjs - | - | NATS client protocol to - v -[LXC 511/512/513 10.10.10.21-23 :4222] - nats-server -js cluster routes :6222 -``` - -The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`. - -### 4.8 Threats to validity - -| Threat | Effect on numbers | -|--------|-------------------| -| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. | -| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. | -| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. | -| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. | -| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. | -| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. | -| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. | -| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. | - -### 4.9 Ethics / safety - -Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket. - ---- - -## 5. Suggestions for fine-tuning - -These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way. - -### 5.1 Treat JetStream as the product limiter - -Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out. - -### 5.2 Split storage class by stream - -| Stream | Suggested store | Why | -|--------|-----------------|-----| -| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad | -| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both | -| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer | -| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry | -| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive | - -Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert. - -### 5.3 Give JetStream real disks - -Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain. - -- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`. -- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure. -- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it. -- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.” - -### 5.4 Isolate the nats CTs from the rest of NS1 - -Host load on this box is often already several. Pin: - -- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes. -- Memory high enough that file-backed streams stay cache-hot for the working set. -- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync. - -Re-run this study after pinning; H1/H2 should move more than ping. - -### 5.5 Consumer and mailbox tuning (delay H4) - -Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT. - -- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM. -- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber. -- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped. -- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT. - -### 5.6 Publisher-side batching in middleware - -A timestamp job is one small JSON. 16k msgs/s is ample. Still: - -- Avoid per-byte publishes; one message per job/event. -- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs). -- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only. - -### 5.7 nats-server knobs worth measuring (A/B with this script) - -| Knob | Why try it | -|------|------------| -| `max_payload` | Keep default unless archive puts grow | -| `write_deadline` | Slow consumer protection for webhooks | -| `max_pending` | Bound memory on a stuck Zapier hook | -| `max_connections` | Fleet workers + keep + middleware | -| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT | -| `max_outstanding_catchup` | Replica restart after a nats-c blip | -| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT | - -Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99. - -### 5.8 Network - -- Keep NATS off `vmbr0`. No change. -- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load). -- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus. - -### 5.9 Security cost (when nkeys/mTLS flip) - -`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement. - -### 5.10 Operational fine-tuning (lag, not peak msgs/s) - -1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`. -2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense. -3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost. -4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated. - -### 5.11 What not to tune - -- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge. -- Do not treat flood 400 ms as “cluster RTT.” Fix consumers. -- Do not load-test on `ZAPIER_*` streams. -- Do not bind client NATS to `0.0.0.0` on `vmbr0`. - -### 5.12 Recommended next experiments (same method, one change each) - -1. CPU pin nats-a/b/c → re-run JS 1p + ping. -2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder). -3. Distinct `store_dir` disks per node. -4. nkeys on, same ladder. -5. Three hardware boxes, same `cluster.env` IPs updated. - -Each experiment should produce a new `results//` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory. - - ---- - -## 6. Reproducing this study - -On **NS1 only**: - -```bash -cd ~/verae-src/verae-nats-cluster -bash scripts/study-on-ns1.sh -``` - -The script exits if `hostname` is not NS1. Outputs land in `results//` including `nats-cluster-bench-ns1.{md,html,pdf}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog. - -Raw logs for this run: `results/20260912T051237Z/`. diff --git a/reports/ns1-host/nats-cluster-bench-ns1.pdf b/reports/ns1-host/nats-cluster-bench-ns1.pdf deleted file mode 100644 index c25d95e..0000000 Binary files a/reports/ns1-host/nats-cluster-bench-ns1.pdf and /dev/null differ diff --git a/reports/ns1-maximize/charts/core-throughput.png b/reports/ns1-maximize/charts/core-throughput.png deleted file mode 100644 index 683f57e..0000000 Binary files a/reports/ns1-maximize/charts/core-throughput.png and /dev/null differ diff --git a/reports/ns1-maximize/charts/core-vs-js.png b/reports/ns1-maximize/charts/core-vs-js.png deleted file mode 100644 index b6a880c..0000000 Binary files a/reports/ns1-maximize/charts/core-vs-js.png and /dev/null differ diff --git a/reports/ns1-maximize/charts/delay-percentiles.png b/reports/ns1-maximize/charts/delay-percentiles.png deleted file mode 100644 index 0b5f851..0000000 Binary files a/reports/ns1-maximize/charts/delay-percentiles.png and /dev/null differ diff --git a/reports/ns1-maximize/charts/delta-vs-baseline.png b/reports/ns1-maximize/charts/delta-vs-baseline.png deleted file mode 100644 index f23250c..0000000 Binary files a/reports/ns1-maximize/charts/delta-vs-baseline.png and /dev/null differ diff --git a/reports/ns1-maximize/charts/js-throughput.png b/reports/ns1-maximize/charts/js-throughput.png deleted file mode 100644 index 55452c7..0000000 Binary files a/reports/ns1-maximize/charts/js-throughput.png and /dev/null differ diff --git a/reports/ns1-maximize/charts/payload-size.png b/reports/ns1-maximize/charts/payload-size.png deleted file mode 100644 index 1118a42..0000000 Binary files a/reports/ns1-maximize/charts/payload-size.png and /dev/null differ diff --git a/reports/ns1-maximize/nats-cluster-bench-ns1.html b/reports/ns1-maximize/nats-cluster-bench-ns1.html deleted file mode 100644 index c466b2a..0000000 --- a/reports/ns1-maximize/nats-cluster-bench-ns1.html +++ /dev/null @@ -1,1385 +0,0 @@ - - - - - - - NATS cluster message speed — NS1 host study - - - - -
Verae Time × Zapier · progress report · maximized NS1 study

NATS cluster message speed — maximized (RAM disk + 8 cores)

packages/zapier-decisions/reports/nats-cluster-bench-ns1.md
-
-

NATS cluster message speed — NS1 host study

-
-

Progress report (maximized NS1 study) · run -20260912T053120Z (UTC)

-
-

Execution provenance. Every process for this study -ran on NS1.GEORGELAMBERT.ORG -(70.88.205.138): maximize-ns1-study.sh -(cores/RAM/max_mem/tmpfs), then -study-on-ns1.sh, nats bench, -latency.mjs (LXC 510), matplotlib, pandoc, weasyprint. -Traffic stayed on vmbr1. veth/10G was not -changed. After the ladder, JetStream was put back on ZFS and product -streams were re-created; 8 cores / 16 GiB / max_mem 8G -stay.

-
-

Measured delta vs -20260912T051237Z

-

Baseline: 1 core / 1 GiB / JetStream on ZFS. This run: 8 cores / 16 -GiB / JetStream tmpfs (file r=3) plus extra -memory store rows. veth/10G unchanged.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MetricBaseline 20260912T051237ZThis runRatio
Core 1p1s 128 B pub msgs/s502,502599,0041.19×
Core 8p8s 128 B aggregate msgs/s2,065,2171,998,7330.97×
JS file r=3 1p 128 B pub msgs/s7,39317,3882.35×
JS file r=3 4p 128 B pub msgs/s17,98622,8531.27×
JS file r=3 4p 1 KiB pub msgs/s14,98518,1141.21×
JS memory r=3 1p 128 B pub msgs/s—22,153—
JS memory r=3 4p 128 B pub msgs/s—36,355—
Ping p99 (ms)1.377ms0.684ms2.01× faster
-
- - -
-

Baseline vs maximized publish rates (log)

-
-

1. Executive summary

- ---- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ItemThis NS1-host run
Control planeNS1.GEORGELAMBERT.ORG (70.88.205.138), user -marchon
Bench clientLXC 510 verae-px-worker
BrokersLXC 511/512/513 nats-a/b/c on -10.10.10.21–23
Client URLnats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
Host load before8.77 8.39 8.26 6/3849 198471
Host load after9.02 8.82 8.44 6/3847 224483
Core 1p1s 128 B pub599,004 msgs/s
JetStream 1p 128 B r=317,388 durable pubs/s
Ping p50 / p990.299ms / 0.684ms
-

Product traffic is the JetStream row. Ping is one-message delay. -Flood is mailbox catch-up after a burst.

-
-

2. Where it ran (and where it -did not)

-
Operator laptop  ──ssh──►  NS1.GEORGELAMBERT.ORG  70.88.205.138
-                              study-on-ns1.sh
-                              python3 build-ns1-study-report.py
-                              sudo pct exec 510 ──► nats bench / latency.mjs
-                                                    │
-                                                    ▼ vmbr1
-                                              10.10.10.21-23 :4222
-
    -
  • Did run on 138: bash, python3, matplotlib, pandoc, -weasyprint, pct, nats-server (in LXC), nats CLI and Node -(in LXC 510).
    • -
  • Did not run on the laptop: no local -nats bench, no local charting, no local WeasyPrint for this -file.
    • -
-
-

3. Results (this run)

-

Host and brokers

-

Before

- ----------- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
NodeVMIDconnectionsin_msgsout_msgscpucoresmem (B)jetstream
nats-a51133,0133,0371814,553,088True
nats-b51231,4151,4240113,557,760True
nats-c51301,3611,3951114,028,800True
-

After

- ----------- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
NodeVMIDconnectionsin_msgsout_msgscpucoresmem (B)jetstream
nats-a5113644,7301,344,7682867,104,768True
nats-b5123491,295978,8001182,542,592True
nats-c5130668,5441,330,9400137,314,560True
-

nproc=40 · -uname=Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux

-

Throughput

- --------- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RunModeAggregate msgs/sPub msgs/sPub MB/sSub msgs/sSub MB/s
core-1p1s-50k-128core pub/sub810,988599,00473.12456,11655.68
core-4p4s-100k-128core pub/sub1,361,921379,34646.311,150,536140.45
core-4p4s-50k-1kcore pub/sub695,192175,725171.61584,024570.34
core-8p8s-200k-128core pub/sub1,998,733283,25934.581,780,589217.36
js-1p-20k-128-r3jetstream r=3 file—17,3882.12——
js-2p2s-20k-128-r3jetstream r=3 file19,8769,9591.229,9421.21
js-4p-20k-1k-r3jetstream r=3 file—18,11417.69——
js-4p-50k-128-r3jetstream r=3 file—22,8532.79——
js-mem-1p-20k-128-r3jetstream r=3 file—22,1532.70——
js-mem-4p-20k-1k-r3jetstream r=3 file—28,68528.01——
js-mem-4p-50k-128-r3jetstream r=3 file—36,3554.44——
-

Round-trip delay

- ------------- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RunKindCountPubsSizeminavgp50p90p99max
lat-ping-1k-128ping (sequential RTT)10001128 B0.250ms0.319ms0.299ms0.363ms0.684ms2.759ms
lat-1p-5k-128flood (burst queueing)50001128 B94.920ms132.026ms131.990ms156.086ms158.860ms158.965ms
lat-4p-5k-1kflood (burst queueing)500041024 B113.281ms132.425ms131.457ms140.317ms144.683ms145.365ms
lat-4p-10k-128flood (burst queueing)100004128 B153.318ms204.329ms205.865ms234.151ms237.346ms237.477ms
lat-8p-20k-128flood (burst queueing)200008128 B233.727ms310.924ms311.170ms373.437ms400.764ms402.811ms
-

Core NATS

-
- - -
-

Core NATS throughput at four loads (NS1 host run) ### -Payload size (core)

-
- - -
-

Core NATS 128 B vs 1 KiB (NS1 host run) ### JetStream r=3 -file

-
- - -
-

JetStream durable publish rate (NS1 host run) ### Core vs -JetStream

-
- - -
-

Core vs JetStream publish rate, log scale (NS1 host run) ### -Delay

-
- - -
-

Ping vs flood delay percentiles, log scale (NS1 host -run)

-
-

4. Study methodology

-

4.1 Question

-

On the NS1 test stand, what message throughput and -delay does the three-node verae JetStream -cluster deliver at several loads, and which part of the stack is the -limiter for product traffic (jobs, events, webhooks, archive)?

-

4.2 Hypotheses (stated before -the run)

-
    -
  1. H1 — Core vs JetStream. Fire-and-forget core NATS -is at least an order of magnitude faster than JetStream file + -replicas=3, because durable publish waits for a majority disk -replica.
    2. -
  2. H2 — JetStream parallelism. Adding publishers does -not linearly increase JetStream write rate once the -replica log is saturated.
    3. -
  3. H3 — Quiet delay. Sequential pub→sub round trip on -vmbr1 is well under 1 ms p99 when the consumer is -waiting.
    4. -
  4. H4 — Burst delay. If publishers dump a batch before -the subscriber drains, observed delay is queueing time, -roughly linear in backlog, not in cluster hop count.
    5. -
  5. H5 — Payload. Moving 128 B → 1 KiB lowers message -rate and raises byte rate on core NATS; JetStream in this size band -stays replica/fsync bound.
    6. -
-

4.3 Independent variables -(what we changed)

- ---- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FactorLevels
TransportCore NATS pub/sub vs JetStream file replicas=3
Publisher count1, 2, 4, 8
Subscriber count0 (JS publish-only), 1, 2, 4, 8
Message count1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step)
Payload128 B, 1024 B
Delay modeping (publish, wait, repeat) vs -flood (publish all, then drain)
-

4.4 Dependent variables -(what we recorded)

- ----- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MetricInstrumentUnit
Publish ratenats bench 0.1.6 Pub statsmsgs/s, MB/s
Subscribe ratenats bench Sub statsmsgs/s, MB/s
Aggregatenats bench NATS Pub/Sub statsmsgs/s (fan-out counts both sides)
Publisher spreadnats min/avg/max msgs/snot delay
One-way-ish RTTlatency.mjs header timestampmin, avg, p50, p90, p99, max
Host load/proc/loadavg before and afterload average
Broker countershttp://127.0.0.1:8222/varz inside each nats LXCconnections, in/out msgs, cpu, mem
-

Important: nats CLI 0.1.6 min/avg/max are -rate spread across publishers, not microseconds of -delay. Delay is only latency.mjs.

-

4.5 Controls and constants

-
    -
  • Cluster name verae, three routes, client -:4222, cluster :6222, monitor loopback -:8222.
    • -
  • Client URL always the three-node list on vmbr1 (never -host 127.0.0.1:4222, never vmbr0).
    • -
  • Bench client is LXC 510, not a nats-* server.
    • -
  • JetStream bench stream name benchstream, -file storage, replicas=3, deleted -between JS loads (nats stream rm --force) so names do not -collide.
    • -
  • Product streams were not the bench target (no load -test on ZAPIER_* / VERAE_ARCHIVE).
    • -
  • No TLS, no nkeys, no account isolation (isolation is -vmbr1).
    • -
  • Same nats CLI version (0.1.6) and nats@2 Node client as -the first ladder.
    • -
-

4.6 Procedure

-
    -
  1. Confirm this script is executing on -NS1.GEORGELAMBERT.ORG. Refuse otherwise.
    2. -
  2. Snapshot host load, memory, LXC configs, and each nats -varz.
    3. -
  3. From NS1, pct exec 510 the core ladder (1p1s, 4p4s, -8p8s at 128 B; 4p4s at 1 KiB).
    4. -
  4. Delete benchstream; JS ladder (1p, 4p, 4p×1 KiB, 2p2s -pull) at replicas=3 file.
    5. -
  5. Copy latency.mjs into 510; ping then flood at several -batch sizes.
    6. -
  6. Snapshot host/varz again.
    7. -
  7. Parse logs on this host; draw charts; write HTML -and PDF on this host.
    8. -
-

No publish, subscribe, chart, or PDF process runs on the operator -laptop for this study.

-

4.7 Instrumentation path

-
[NS1 host 70.88.205.138]
-   study-on-ns1.sh  (bash + python3)
-        |
-        | sudo pct exec 510
-        v
-[LXC 510 verae-px-worker 10.10.10.20]
-   nats bench / node latency.mjs
-        |
-        | NATS client protocol to
-        v
-[LXC 511/512/513  10.10.10.21-23 :4222]
-   nats-server -js  cluster routes :6222
-

The hypervisor issues the guest commands. The messages themselves -never leave vmbr1.

-

4.8 Threats to validity

- ---- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ThreatEffect on numbers
One physical hostThree “replicas” share CPU, memory, and usually the same datastore. -This measures process/LXC HA, not disk HA.
Shared loadNS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. -Load average during a run is part of the result, not noise to -ignore.
Single bench clientAll publishers live in 510. Per-publisher rate spread is contention -in that guest.
Short runsSeconds of traffic. No compaction, no multi-hour page-cache -eviction, no snapshot during load.
No TLS/nkeysProduction auth will cost CPU. Do not treat these rates as -post-nkeys rates.
Fan-out aggregateCore aggregate msgs/s counts pub+sub. Do not compare that column to -JetStream unique writes.
Flood ≠ RTTMixing flood averages with ping p99 produces a fake “NATS is slow” -story.
Lab onlyNot a Zapier HTTPS bench and not live -api.veraetime.net.
-

4.9 Ethics / safety

-

Bench uses throwaway subjects (bench.core.*, -bench.js.*, bench.lat.*) and a throwaway -stream. It does not purge product streams. Zapier cloud has no NATS -socket.

-
-

5. Suggestions for fine-tuning

-

These follow from the method and from the first ladder on this stand -(JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of -ms). Apply in order of leverage. Re-run this NS1 study -after each change so the delta is measured the same way.

-

5.1 Treat JetStream as -the product limiter

-

Product jobs/events/webhooks/archive are durable. Tuning core NATS to -2M msgs/s will not move a timestamp Zap. Put effort into replica -write path and consumer lag, not core -fan-out.

-

5.2 Split storage class by -stream

- ----- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
StreamSuggested storeWhy
ZAPIER_JOBSfile, r=3Work queue; lose-a-job is bad
ZAPIER_EVENTSfile r=3, or memory r=3 if events are rebuildable from job -statusHot waiters; measure both
ZAPIER_WEBHOOKSfile, r=3, workqueueHTTPS to Zapier is the slow consumer
ZAPIER_USAGEfile, r=3, limits + max-ageTelemetry
VERAE_ARCHIVEfile, r=3, on the best diskPuts are larger and must survive
-

Try ZAPIER_EVENTS as memory store in a maintenance -window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms -and durable events still ack at a higher rate, keep it; if a CT restart -drops in-flight waiters, revert.

-

5.3 Give JetStream real disks

-

Today r=3 on three LXC guests on one Proxmox host is -three files, one failure domain.

-
    -
  • Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) -into each nats LXC store_dir.
    • -
  • Set sync: always only on archive if you need it; -default sync is often enough for jobs and is faster. Measure.
    • -
  • Do not put JetStream store_dir on the same busy rootfs -as Forgejo/Caddy if we can avoid it.
    • -
  • When moving to three metal boxes: same configs, private NIC, one -disk (or mirror) per node. That is the first change -that makes r=3 mean “two boxes can die.”
    • -
-

5.4 Isolate the nats -CTs from the rest of NS1

-

Host load on this box is often already several. Pin:

-
    -
  • nats-a/b/c: dedicated cores, no steal from keep/fleet -Node processes.
    • -
  • Memory high enough that file-backed streams stay cache-hot for the -working set.
    • -
  • cpuunits / cpuset in pct config so a -Zapier-facing Node GC pause does not stall fsync.
    • -
-

Re-run this study after pinning; H1/H2 should move more than -ping.

-

5.5 Consumer and mailbox -tuning (delay H4)

-

Flood delay is backlog / consume_rate. Fine-tune the -waiters, not the broker RTT.

-
    -
  • jobs.events and webhooks.deliver: raise -max_ack_pending so a slow HTTPS hook does not stall the -whole consumer; cap it so a poison message cannot unbounded-buffer -RAM.
    • -
  • Pull consumers: larger batch, shorter expires, more -pullers horizontally (fleet replica floors) instead of one fat -subscriber.
    • -
  • Middleware should not flood-publish then wait; it -already does per-job publish. Keep that. The flood test is the outage -profile when a consumer is stopped.
    • -
  • Alert on consumer lag (pending + ack pending) from -JetStream, not on ping RTT.
    • -
-

5.6 Publisher-side -batching in middleware

-

A timestamp job is one small JSON. 16k msgs/s is ample. Still:

-
    -
  • Avoid per-byte publishes; one message per job/event.
    • -
  • Reuse NATS connections (connection churn showed up as publisher -spread in the core 4p/8p runs).
    • -
  • Idempotent msg id / duplicate window sized to Verae -retry window, not default-only.
    • -
-

5.7 -nats-server knobs worth measuring (A/B with this script)

- ---- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
KnobWhy try it
max_payloadKeep default unless archive puts grow
write_deadlineSlow consumer protection for webhooks
max_pendingBound memory on a stuck Zapier hook
max_connectionsFleet workers + keep + middleware
JetStream max_file_store / -max_memory_storePrevent one stream from filling the CT
max_outstanding_catchupReplica restart after a nats-c blip
GOMAXPROCS = LXC coresDo not overthread a 2-core CT
-

Change one knob, re-run -study-on-ns1.sh, compare JetStream 1p 128 B and ping -p99.

-

5.8 Network

-
    -
  • Keep NATS off vmbr0. No change.
    • -
  • When on metal: dedicated NIC or VLAN for cluster :6222 -vs client :4222 if possible (replication vs client -load).
    • -
  • Check virtio queue counts on the LXC nics if core 1 KiB byte rate -plateaus.
    • -
-

5.9 Security cost (when -nkeys/mTLS flip)

-

verae-nats-accounts is still a sketch. Enabling accounts -will add CPU on publish. Budget: re-run this exact study -after creds are in every NATS_URL, and -accept a drop on both core and JS. Do not flip without that -measurement.

-

5.10 Operational -fine-tuning (lag, not peak msgs/s)

-
    -
  1. Scrape varz / jsz from the host over -vmbr1 (not public). Monitor loopback :8222 is -invisible to Prometheus on NS1 unless we add a host-side proxy on -10.10.10.21:8222 bound only to vmbr1.
    2. -
  2. Keep replica floors for webhook-deliver and job-poller — they are -the flood defense.
    3. -
  3. Backup/restore drill of JetStream during idle, then -a short JS 1p run to see catchup cost.
    4. -
  4. A 15–30 minute soak (not in this ladder) for page cache and -compaction; add that as a third study when disks are dedicated.
    5. -
-

5.11 What not to tune

-
    -
  • Do not chase core 8p8s aggregate. It is fan-out on a lab -bridge.
    • -
  • Do not treat flood 400 ms as “cluster RTT.” Fix consumers.
    • -
  • Do not load-test on ZAPIER_* streams.
    • -
  • Do not bind client NATS to 0.0.0.0 on -vmbr0.
    • -
- -
    -
  1. CPU pin nats-a/b/c → re-run JS 1p + ping.
    2. -
  2. ZAPIER_EVENTS-shaped memory stream vs file (throwaway -stream, same flags as this JS ladder).
    3. -
  3. Distinct store_dir disks per node.
    4. -
  4. nkeys on, same ladder.
    5. -
  5. Three hardware boxes, same cluster.env IPs -updated.
    6. -
-

Each experiment should produce a new -results/<utc>/ on NS1 and a new progress-repo report -so we can diff H1–H5 instead of arguing from memory.

-
-

6. Reproducing this study

-

On NS1 only:

-
cd ~/verae-src/verae-nats-cluster
-bash scripts/study-on-ns1.sh
-

The script exits if hostname is not NS1. Outputs land in -results/<utc>/ including -nats-cluster-bench-ns1.{md,html,pdf} and -charts/. Copy those into -zapier-decisions/reports/ for the progress repo and -catalog.

-

Raw logs for this run: results/20260912T053120Z/.

- - diff --git a/reports/ns1-maximize/nats-cluster-bench-ns1.md b/reports/ns1-maximize/nats-cluster-bench-ns1.md deleted file mode 100644 index b0d8c69..0000000 --- a/reports/ns1-maximize/nats-cluster-bench-ns1.md +++ /dev/null @@ -1,355 +0,0 @@ -**Progress report (maximized NS1 study)** · run `20260912T053120Z` (UTC) - -> **Execution provenance.** Every process for this study ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`): `maximize-ns1-study.sh` (cores/RAM/`max_mem`/tmpfs), then `study-on-ns1.sh`, `nats bench`, `latency.mjs` (LXC 510), matplotlib, pandoc, weasyprint. Traffic stayed on `vmbr1`. veth/10G was **not** changed. After the ladder, JetStream was put back on ZFS and product streams were re-created; **8 cores / 16 GiB / max_mem 8G stay**. - -## Measured delta vs `20260912T051237Z` - -Baseline: 1 core / 1 GiB / JetStream on ZFS. This run: 8 cores / 16 GiB / JetStream **tmpfs** (file r=3) plus extra **memory** store rows. veth/10G unchanged. - -| Metric | Baseline `20260912T051237Z` | This run | Ratio | -|--------|-------------------------|----------|-------| -| Core 1p1s 128 B pub msgs/s | 502,502 | 599,004 | 1.19× | -| Core 8p8s 128 B aggregate msgs/s | 2,065,217 | 1,998,733 | 0.97× | -| JS file r=3 1p 128 B pub msgs/s | 7,393 | 17,388 | 2.35× | -| JS file r=3 4p 128 B pub msgs/s | 17,986 | 22,853 | 1.27× | -| JS file r=3 4p 1 KiB pub msgs/s | 14,985 | 18,114 | 1.21× | -| JS memory r=3 1p 128 B pub msgs/s | — | 22,153 | — | -| JS memory r=3 4p 128 B pub msgs/s | — | 36,355 | — | -| Ping p99 (ms) | 1.377ms | 0.684ms | 2.01× faster | - -![Baseline vs maximized publish rates (log)](charts/delta-vs-baseline.png) - -*Baseline vs maximized publish rates (log)* - - ---- - -## 1. Executive summary - -| Item | This NS1-host run | -|------|-------------------| -| Control plane | NS1.GEORGELAMBERT.ORG (`70.88.205.138`), user `marchon` | -| Bench client | LXC 510 `verae-px-worker` | -| Brokers | LXC 511/512/513 `nats-a/b/c` on `10.10.10.21–23` | -| Client URL | `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` | -| Host load before | `8.77 8.39 8.26 6/3849 198471` | -| Host load after | `9.02 8.82 8.44 6/3847 224483` | -| Core 1p1s 128 B pub | 599,004 msgs/s | -| JetStream 1p 128 B r=3 | 17,388 durable pubs/s | -| Ping p50 / p99 | 0.299ms / 0.684ms | - -Product traffic is the JetStream row. Ping is one-message delay. Flood is mailbox catch-up after a burst. - ---- - -## 2. Where it ran (and where it did not) - -```text -Operator laptop ──ssh──► NS1.GEORGELAMBERT.ORG 70.88.205.138 - study-on-ns1.sh - python3 build-ns1-study-report.py - sudo pct exec 510 ──► nats bench / latency.mjs - │ - ▼ vmbr1 - 10.10.10.21-23 :4222 -``` - -- **Did run on 138:** bash, python3, matplotlib, pandoc, weasyprint, `pct`, nats-server (in LXC), nats CLI and Node (in LXC 510). -- **Did not run on the laptop:** no local `nats bench`, no local charting, no local WeasyPrint for this file. - ---- - -## 3. Results (this run) - -### Host and brokers - -**Before** - -| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream | -|------|------|-------------|---------|----------|-----|-------|---------|-----------| -| nats-a | 511 | 3 | 3,013 | 3,037 | 1 | 8 | 14,553,088 | True | -| nats-b | 512 | 3 | 1,415 | 1,424 | 0 | 1 | 13,557,760 | True | -| nats-c | 513 | 0 | 1,361 | 1,395 | 1 | 1 | 14,028,800 | True | - -**After** - -| Node | VMID | connections | in_msgs | out_msgs | cpu | cores | mem (B) | jetstream | -|------|------|-------------|---------|----------|-----|-------|---------|-----------| -| nats-a | 511 | 3 | 644,730 | 1,344,768 | 2 | 8 | 67,104,768 | True | -| nats-b | 512 | 3 | 491,295 | 978,800 | 1 | 1 | 82,542,592 | True | -| nats-c | 513 | 0 | 668,544 | 1,330,940 | 0 | 1 | 37,314,560 | True | - -nproc=40 · uname=`Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux` - -### Throughput - -| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s | -|-----|------|------------------|------------|----------|------------|----------| -| `core-1p1s-50k-128` | core pub/sub | 810,988 | 599,004 | 73.12 | 456,116 | 55.68 | -| `core-4p4s-100k-128` | core pub/sub | 1,361,921 | 379,346 | 46.31 | 1,150,536 | 140.45 | -| `core-4p4s-50k-1k` | core pub/sub | 695,192 | 175,725 | 171.61 | 584,024 | 570.34 | -| `core-8p8s-200k-128` | core pub/sub | 1,998,733 | 283,259 | 34.58 | 1,780,589 | 217.36 | -| `js-1p-20k-128-r3` | jetstream r=3 file | — | 17,388 | 2.12 | — | — | -| `js-2p2s-20k-128-r3` | jetstream r=3 file | 19,876 | 9,959 | 1.22 | 9,942 | 1.21 | -| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 18,114 | 17.69 | — | — | -| `js-4p-50k-128-r3` | jetstream r=3 file | — | 22,853 | 2.79 | — | — | -| `js-mem-1p-20k-128-r3` | jetstream r=3 file | — | 22,153 | 2.70 | — | — | -| `js-mem-4p-20k-1k-r3` | jetstream r=3 file | — | 28,685 | 28.01 | — | — | -| `js-mem-4p-50k-128-r3` | jetstream r=3 file | — | 36,355 | 4.44 | — | — | - -### Round-trip delay - -| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max | -|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----| -| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.250ms | 0.319ms | 0.299ms | 0.363ms | 0.684ms | 2.759ms | -| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 94.920ms | 132.026ms | 131.990ms | 156.086ms | 158.860ms | 158.965ms | -| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 113.281ms | 132.425ms | 131.457ms | 140.317ms | 144.683ms | 145.365ms | -| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 153.318ms | 204.329ms | 205.865ms | 234.151ms | 237.346ms | 237.477ms | -| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 233.727ms | 310.924ms | 311.170ms | 373.437ms | 400.764ms | 402.811ms | - -### Core NATS - -![Core NATS throughput at four loads (NS1 host run)](charts/core-throughput.png) - -*Core NATS throughput at four loads (NS1 host run)* -### Payload size (core) - -![Core NATS 128 B vs 1 KiB (NS1 host run)](charts/payload-size.png) - -*Core NATS 128 B vs 1 KiB (NS1 host run)* -### JetStream r=3 file - -![JetStream durable publish rate (NS1 host run)](charts/js-throughput.png) - -*JetStream durable publish rate (NS1 host run)* -### Core vs JetStream - -![Core vs JetStream publish rate, log scale (NS1 host run)](charts/core-vs-js.png) - -*Core vs JetStream publish rate, log scale (NS1 host run)* -### Delay - -![Ping vs flood delay percentiles, log scale (NS1 host run)](charts/delay-percentiles.png) - -*Ping vs flood delay percentiles, log scale (NS1 host run)* - ---- - -## 4. Study methodology - -### 4.1 Question - -On the NS1 test stand, what message **throughput** and **delay** does the three-node `verae` JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)? - -### 4.2 Hypotheses (stated before the run) - -1. **H1 — Core vs JetStream.** Fire-and-forget core NATS is at least an order of magnitude faster than JetStream **file + replicas=3**, because durable publish waits for a majority disk replica. -2. **H2 — JetStream parallelism.** Adding publishers does **not** linearly increase JetStream write rate once the replica log is saturated. -3. **H3 — Quiet delay.** Sequential pub→sub round trip on `vmbr1` is well under 1 ms p99 when the consumer is waiting. -4. **H4 — Burst delay.** If publishers dump a batch before the subscriber drains, observed delay is **queueing time**, roughly linear in backlog, not in cluster hop count. -5. **H5 — Payload.** Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound. - -### 4.3 Independent variables (what we changed) - -| Factor | Levels | -|--------|--------| -| Transport | Core NATS pub/sub vs JetStream file replicas=3 | -| Publisher count | 1, 2, 4, 8 | -| Subscriber count | 0 (JS publish-only), 1, 2, 4, 8 | -| Message count | 1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step) | -| Payload | 128 B, 1024 B | -| Delay mode | **ping** (publish, wait, repeat) vs **flood** (publish all, then drain) | - -### 4.4 Dependent variables (what we recorded) - -| Metric | Instrument | Unit | -|--------|------------|------| -| Publish rate | `nats bench` 0.1.6 Pub stats | msgs/s, MB/s | -| Subscribe rate | `nats bench` Sub stats | msgs/s, MB/s | -| Aggregate | `nats bench` NATS Pub/Sub stats | msgs/s (fan-out counts both sides) | -| Publisher spread | nats min/avg/max **msgs/s** | not delay | -| One-way-ish RTT | `latency.mjs` header timestamp | min, avg, p50, p90, p99, max | -| Host load | `/proc/loadavg` before and after | load average | -| Broker counters | `http://127.0.0.1:8222/varz` inside each nats LXC | connections, in/out msgs, cpu, mem | - -**Important:** nats CLI 0.1.6 min/avg/max are **rate spread across publishers**, not microseconds of delay. Delay is only `latency.mjs`. - -### 4.5 Controls and constants - -- Cluster name `verae`, three routes, client `:4222`, cluster `:6222`, monitor loopback `:8222`. -- Client URL always the three-node list on `vmbr1` (never host `127.0.0.1:4222`, never `vmbr0`). -- Bench client is LXC **510**, not a nats-* server. -- JetStream bench stream name `benchstream`, **file** storage, **replicas=3**, deleted between JS loads (`nats stream rm --force`) so names do not collide. -- Product streams were **not** the bench target (no load test on `ZAPIER_*` / `VERAE_ARCHIVE`). -- No TLS, no nkeys, no account isolation (isolation is `vmbr1`). -- Same nats CLI version (0.1.6) and `nats@2` Node client as the first ladder. - -### 4.6 Procedure - -1. Confirm this script is executing on **NS1.GEORGELAMBERT.ORG**. Refuse otherwise. -2. Snapshot host load, memory, LXC configs, and each nats `varz`. -3. From NS1, `pct exec 510` the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB). -4. Delete `benchstream`; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file. -5. Copy `latency.mjs` into 510; ping then flood at several batch sizes. -6. Snapshot host/`varz` again. -7. Parse logs on **this host**; draw charts; write HTML and PDF on **this host**. - -No publish, subscribe, chart, or PDF process runs on the operator laptop for this study. - -### 4.7 Instrumentation path - -```text -[NS1 host 70.88.205.138] - study-on-ns1.sh (bash + python3) - | - | sudo pct exec 510 - v -[LXC 510 verae-px-worker 10.10.10.20] - nats bench / node latency.mjs - | - | NATS client protocol to - v -[LXC 511/512/513 10.10.10.21-23 :4222] - nats-server -js cluster routes :6222 -``` - -The hypervisor issues the guest commands. The messages themselves never leave `vmbr1`. - -### 4.8 Threats to validity - -| Threat | Effect on numbers | -|--------|-------------------| -| **One physical host** | Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA. | -| **Shared load** | NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore. | -| **Single bench client** | All publishers live in 510. Per-publisher rate spread is contention in that guest. | -| **Short runs** | Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load. | -| **No TLS/nkeys** | Production auth will cost CPU. Do not treat these rates as post-nkeys rates. | -| **Fan-out aggregate** | Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes. | -| **Flood ≠ RTT** | Mixing flood averages with ping p99 produces a fake “NATS is slow” story. | -| **Lab only** | Not a Zapier HTTPS bench and not live `api.veraetime.net`. | - -### 4.9 Ethics / safety - -Bench uses throwaway subjects (`bench.core.*`, `bench.js.*`, `bench.lat.*`) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket. - ---- - -## 5. Suggestions for fine-tuning - -These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run **this NS1 study** after each change so the delta is measured the same way. - -### 5.1 Treat JetStream as the product limiter - -Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into **replica write path** and **consumer lag**, not core fan-out. - -### 5.2 Split storage class by stream - -| Stream | Suggested store | Why | -|--------|-----------------|-----| -| `ZAPIER_JOBS` | file, r=3 | Work queue; lose-a-job is bad | -| `ZAPIER_EVENTS` | file r=3, or memory r=3 if events are rebuildable from job status | Hot waiters; measure both | -| `ZAPIER_WEBHOOKS` | file, r=3, workqueue | HTTPS to Zapier is the slow consumer | -| `ZAPIER_USAGE` | file, r=3, limits + max-age | Telemetry | -| `VERAE_ARCHIVE` | file, r=3, on the **best disk** | Puts are larger and must survive | - -Try `ZAPIER_EVENTS` as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert. - -### 5.3 Give JetStream real disks - -Today r=3 on three LXC guests on **one Proxmox host** is three files, one failure domain. - -- Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC `store_dir`. -- Set `sync: always` only on archive if you need it; default sync is often enough for jobs and is faster. Measure. -- Do not put JetStream `store_dir` on the same busy rootfs as Forgejo/Caddy if we can avoid it. -- When moving to three metal boxes: same configs, private NIC, one disk (or mirror) **per node**. That is the first change that makes r=3 mean “two boxes can die.” - -### 5.4 Isolate the nats CTs from the rest of NS1 - -Host load on this box is often already several. Pin: - -- `nats-a/b/c`: dedicated cores, no steal from keep/fleet Node processes. -- Memory high enough that file-backed streams stay cache-hot for the working set. -- `cpuunits` / cpuset in `pct config` so a Zapier-facing Node GC pause does not stall fsync. - -Re-run this study after pinning; H1/H2 should move more than ping. - -### 5.5 Consumer and mailbox tuning (delay H4) - -Flood delay is backlog / consume_rate. Fine-tune the **waiters**, not the broker RTT. - -- `jobs.events` and `webhooks.deliver`: raise `max_ack_pending` so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM. -- Pull consumers: larger batch, shorter `expires`, more pullers horizontally (fleet replica floors) instead of one fat subscriber. -- Middleware should **not** flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped. -- Alert on **consumer lag** (pending + ack pending) from JetStream, not on ping RTT. - -### 5.6 Publisher-side batching in middleware - -A timestamp job is one small JSON. 16k msgs/s is ample. Still: - -- Avoid per-byte publishes; one message per job/event. -- Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs). -- Idempotent `msg id` / duplicate window sized to Verae retry window, not default-only. - -### 5.7 nats-server knobs worth measuring (A/B with this script) - -| Knob | Why try it | -|------|------------| -| `max_payload` | Keep default unless archive puts grow | -| `write_deadline` | Slow consumer protection for webhooks | -| `max_pending` | Bound memory on a stuck Zapier hook | -| `max_connections` | Fleet workers + keep + middleware | -| JetStream `max_file_store` / `max_memory_store` | Prevent one stream from filling the CT | -| `max_outstanding_catchup` | Replica restart after a nats-c blip | -| GOMAXPROCS = LXC cores | Do not overthread a 2-core CT | - -Change **one** knob, re-run `study-on-ns1.sh`, compare JetStream 1p 128 B and ping p99. - -### 5.8 Network - -- Keep NATS off `vmbr0`. No change. -- When on metal: dedicated NIC or VLAN for cluster `:6222` vs client `:4222` if possible (replication vs client load). -- Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus. - -### 5.9 Security cost (when nkeys/mTLS flip) - -`verae-nats-accounts` is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study **after** creds are in every `NATS_URL`, and accept a drop on both core and JS. Do not flip without that measurement. - -### 5.10 Operational fine-tuning (lag, not peak msgs/s) - -1. Scrape `varz` / `jsz` from the host over `vmbr1` (not public). Monitor loopback `:8222` is invisible to Prometheus on NS1 unless we add a host-side proxy on `10.10.10.21:8222` bound only to `vmbr1`. -2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense. -3. Backup/restore drill of JetStream **during idle**, then a short JS 1p run to see catchup cost. -4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated. - -### 5.11 What not to tune - -- Do not chase core 8p8s aggregate. It is fan-out on a lab bridge. -- Do not treat flood 400 ms as “cluster RTT.” Fix consumers. -- Do not load-test on `ZAPIER_*` streams. -- Do not bind client NATS to `0.0.0.0` on `vmbr0`. - -### 5.12 Recommended next experiments (same method, one change each) - -1. CPU pin nats-a/b/c → re-run JS 1p + ping. -2. `ZAPIER_EVENTS`-shaped memory stream vs file (throwaway stream, same flags as this JS ladder). -3. Distinct `store_dir` disks per node. -4. nkeys on, same ladder. -5. Three hardware boxes, same `cluster.env` IPs updated. - -Each experiment should produce a new `results//` on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory. - - ---- - -## 6. Reproducing this study - -On **NS1 only**: - -```bash -cd ~/verae-src/verae-nats-cluster -bash scripts/study-on-ns1.sh -``` - -The script exits if `hostname` is not NS1. Outputs land in `results//` including `nats-cluster-bench-ns1.{md,html,pdf}` and `charts/`. Copy those into `zapier-decisions/reports/` for the progress repo and catalog. - -Raw logs for this run: `results/20260912T053120Z/`. diff --git a/reports/ns1-maximize/nats-cluster-bench-ns1.pdf b/reports/ns1-maximize/nats-cluster-bench-ns1.pdf deleted file mode 100644 index 634bd81..0000000 Binary files a/reports/ns1-maximize/nats-cluster-bench-ns1.pdf and /dev/null differ diff --git a/reports/optimal-config/REPORT.html b/reports/optimal-config/REPORT.html deleted file mode 100644 index e9d66d1..0000000 --- a/reports/optimal-config/REPORT.html +++ /dev/null @@ -1,853 +0,0 @@ - - - - - - - NATS optimal configuration study - - - - -
Verae Time × Zapier · progress report

NATS optimal configuration study

packages/zapier-decisions/reports/optimal-config/REPORT.md
-
-

NATS optimal configuration study

-
-

Progress report — optimal configuration study · -20260912T055851Z (UTC) · all code on -NS1.GEORGELAMBERT.ORG (70.88.205.138)

-

This document folds every ladder we have run (1-core ZFS, -NS1-orchestrated, tmpfs maximize, and this exhaustive 8c/16G -ZFS factorial) plus UDP / MQTT / reconnect probes. It -recommends a lab config and a three-box HP DL360 Gen10 -projection. veth/10G was not changed.

-
-

1. Verdict (read this first)

-

Keep NATS + JetStream. Do not replace the fabric -with MQTT, UDP, or a custom persistent-socket protocol for Verae -jobs/events/archive. Those are either slower, less durable, or already -what NATS is.

-

Lab (NS1, one host, three LXC) — optimal now

- ------ - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
StreamStorageReplicasWhy
ZAPIER_JOBS, ZAPIER_WEBHOOKS, -VERAE_ARCHIVEfile (ZFS)3Survive a nats LXC death; archive must persist
ZAPIER_EVENTSmemory3Waiters are latency-sensitive; events rebuild from job status
ZAPIER_USAGEfile3Telemetry, limits + max-age
-

Keep 8 cores / 16 GiB / max_mem: 8G on -510–513 (already live). Do not leave JetStream on -tmpfs. Do not drop product streams to r=1. Reuse -one NATS connection per process (already true in -middleware); never connect-per-message.

-

Metal (3× DL360 Gen10) — optimal later

-

Same stream table. File store on local NVMe/M.2, not -a shared SAN. Cluster + client on 10GbE (or 25GbE if -you already have it). Dual Gold Xeon is surplus CPU for this workload; -8–16 cores dedicated to nats-server is enough. Expected JS -file r=3: ~40–80k 128 B pubs/s (about -3–6× this lab’s 8c ZFS 1p, 2–4× tmpfs -1p) — bounded by 10GbE replica RTT, not by Xeon clocks. -Core NATS will sit in the 1–3M msgs/s band until the -NIC saturates (~9 Gbit/s ≈ 8–9M × 128 B theoretical; CPU and client will -hit first).

-
-

2. What we actually -ran (this exhaustive pass)

-

Live cluster during this run: LXC 510–513 8 cores / 16 -GiB, JetStream on ZFS (tmpfs from the maximize -study was already unmounted). Extra factorial: file/memory × replicas -1/3, 4 KiB file r=3, reconnect-per-message ping, UDP echo 510→511, MQTT -QoS0 against nats-a :1883. Product streams were not the -bench target.

-

2.1 Cross-study history

- -------- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
StudyEnvCore 1p pubJS file r=3 1pJS mem r=3 4pPing p99
20260912T051237Z1c/1G ZFS (NS1 orch.)502,5027,393—1.377ms
20260912T053120Z8c/16G tmpfs + mem extra599,00417,38836,3550.684ms
20260912T055851Z8c/16G ZFS exhaustive 20260912T055851Z662,22714,33037,7361.140ms
-
- - -
-

2.2 This run — JetStream -factorial

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RunWhatPub msgs/sPub MB/s
js-file-1p-20k-128-r1file r=1 1p 128 B18,8882.31
js-file-4p-50k-128-r1file r=1 4p 128 B24,5603.00
js-1p-20k-128-r3file r=3 1p 128 B14,3301.75
js-4p-50k-128-r3file r=3 4p 128 B19,2322.35
js-4p-20k-1k-r3file r=3 4p 1 KiB15,19714.84
js-file-1p-20k-4k-r3file r=3 1p 4 KiB8,67333.88
js-mem-1p-20k-128-r1memory r=1 1p 128 B29,9723.66
js-mem-4p-50k-128-r1memory r=1 4p 128 B64,9237.93
js-mem-1p-20k-128-r3memory r=3 1p 128 B20,1882.46
js-mem-4p-50k-128-r3memory r=3 4p 128 B37,7364.61
js-mem-4p-20k-1k-r3memory r=3 4p 1 KiB33,91633.12
-

Replica 1 vs 3 on this stand (file 1p 128 B): r=1 is -18,888 vs r=3 14,330 (1.32× if r=3 is the slower one). Memory r=1 1p -29,972 vs memory r=3 20,188.

-
-Replica cost - -
-

2.3 Delay, reconnect tax, UDP, -MQTT

- ----- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ProbeResultMeaning
NATS ping (persistent sockets) p50 / p990.456ms / 1.140msQuiet hop with a long-lived TCP conn
NATS reconnect-per-message p50 / p990.503ms / 1.750msTCP+NATS handshake on every pub — this is the tax to avoid
UDP echo 510→511 p990.363msRaw datagram ceiling on the same veth (no NATS)
MQTT QoS0 5k×128 B44862 pubs/snats-server MQTT gateway on :1883
-

Core 1p1s 128 B this run: 662,227 pub msgs/s. Flood delay is still -backlog/consume_rate, not RTT.

-
-

3. -Alternative transports (why we are not switching the fabric)

-

NATS already is persistent TCP sockets with a tiny -binary protocol, automatic reconnect, and optional JetStream durability. -“Reduce connection overhead” is a client discipline: -hold the connection. The reconnect probe exists to prove that opening a -socket per job would dominate ping RTT.

- ------ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
IdeaFit for Verae jobs/events/archiveThroughput vs NATS coreDurability
NATS core pub/subFan-out, request-reply (verae.billing.*)Highest we measured (~0.5–2M msgs/s)None
NATS JetStream file r=3Jobs, webhooks, archive~8–23k on this lab; see metal projectionDisk + 1-node loss
NATS JetStream memory r=3Events mailbox~22–36k on this labRAM + 1-node loss; empty on full restart
MQTT (NATS gateway or Mosquitto)IoT endpoints that already speak MQTTThis probe: 44862 pubs/s QoS0 — typically well -below NATS core; QoS1 ≈ JetStream-ish with more chatterQoS1/2 session state; not our WORM model
UDPTelemetry that may dropRTT 0.363ms p99 — fastest hop, no reliability, no -cluster, no authNone
Custom persistent sockets / HTTP long-pollWorse NATSYou would re-implement reconnect, flow control, and fan-outDIY
WebSocketBrowsers onlyExtra framing; NATS already has WS for UIs, not for middlewareSame as core/JS behind it
QUIC / WebTransportLossy WAN / browsersNATS QUIC is not the lab path; 10GbE LAN does not need itSame
Kafka / Redis streamsHeavy log replayHigher ops cost; not on vmbr1 todayYes, heavier
-

MQTT: NATS documents MQTT as an enabling -gateway for existing IoT, and prefers NATS end-to-end for greenfield. -Zapier cloud never talks NATS or MQTT; it talks HTTPS. Putting MQTT in -the middle of timestamp jobs adds protocol translation and QoS timers -without helping jobId → events. Use MQTT only if a device -already cannot speak NATS.

-

UDP: Fine as a measurement of veth RTT. -Unusable as the job fabric (no ack, no replica, no flow control). NATS -ping is already within a small multiple of UDP on this bridge.

-

Persistence sockets: Middleware and keep already -keep NATS_URL connections open. Optimal: one connection (or -a small pool) per process, max_reconnect, jitter, no -connect() in the per-job path. The reconnect ladder is the -anti-pattern.

-
-

4. Optimal configurations

-

4.1 NS1 lab (now)

-
    -
  1. Leave 8 cores / 16 GiB on nats-a/b/c and the -worker. Host has 40 cores / 377 GiB; this is cheap.
    2. -
  2. max_mem: 8G stays. Required for memory -streams.
    3. -
  3. File r=3 on ZFS for jobs/webhooks/archive. tmpfs -doubled JS 1p (7.4k→17k) but loses the stream on reboot -— unacceptable for archive.
    4. -
  4. Memory r=3 for ZAPIER_EVENTS if we -accept “all three nats CTs reboot ⇒ in-flight waiters fall back to HTTP -poll.” That matches the designed wait path -(GET /api/status/{jobId}).
    5. -
  5. r=1 only for throwaway benches, never product -streams. Replica=3 is the point of three guests.
    6. -
  6. veth on vmbr1, no fake 10G NICs. Already 10000Mb/s; -JS does not fill it.
    7. -
  7. Pin cpusets later if keep/fleet steal; not required -to beat these numbers.
    8. -
  8. Clients: persistent NATS connections; pull consumers with bounded -max_ack_pending for webhooks.
    9. -
-

4.2 Three HP DL360 -Gen10 (projection — not measured)

-

Assumed bill of materials (state it in the buy):

- ---- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
PieceAssumption
Chassis3× DL360 Gen10 1U
CPUDual 2nd-gen Xeon Gold (e.g. 6226R 16c or 6248 20c -— 32–40 cores/box)
MemoryDDR4-2933, 192–384 GiB/box (6–12×32 GiB); NATS will -not use most of it
StorageNVMe M.2 or U.2 for -/var/lib/nats/jetstream (XFS or ext4, not -shared ZFS over the network). RAID1 of two NVMe if you want disk HA -inside a box
Network10GbE (FlexibleLOM or PCIe); dedicated VLAN for -:4222+:6222. Do not share with public -vmbr0 traffic
OSDebian/Ubuntu bare metal, nats-server systemd, same -nats.conf as lab (bind private IP only)
-

What changes vs NS1 LXC

- ------ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FactorNS1 today3× DL360Effect on JS file r=3
Failure domain1 Proxmox host3 chassis, 3 NVMe, 3 NICsr=3 means something
DiskShared ZFS SSD2Local NVMe fsync ~50–150 µsBig win vs ZFS; similar to tmpfs for sequential 128 B
Replica pathveth/bridge (~µs–tens of µs)10GbE RTT typically 50–200 µsSlower than same-host tmpfs, faster than a bad -SAN
CPU8 of 40 shared32–40 dedicated Gold coresHeadroom for many clients, not 10× JS
NICsoftware 10G veth, already ~5 Gbit/s corereal 10GbE ~9 Gbit/s TCPCore NATS can grow; JS r=3 stays replica-bound
-

Projected bands (128 B, 3-node cluster, dedicated -10GbE, local NVMe, 8+ cores pinned to nats-server):

- ------ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
WorkloadNS1 measured (best)DL360 projectionConfidence
Core pub/sub 1p0.5–0.8M0.8–2MMedium — NIC + syscall, plenty of CPU
Core 4p4s 1 KiB~0.6–0.7M (~0.6 GB/s)~1M msgs/s / ~1 GB/s approaching 10GbEMedium
JS file r=1this run r=180–200k pubs/sMedium — NVMe + no replica wait
JS file r=37–23k (ZFS/tmpfs)40–80k pubs/sMedium-low — replica RTT dominates; 3 NVMe still help vs shared -ZFS
JS memory r=322–36k50–100kMedium-low — RAM + 10GbE ack
Ping p990.7–1.4 ms0.2–0.6 msMedium — real NIC but no Proxmox tax
-

These are not DL360 measurements. Scale from: (a) -our replica-1 vs replica-3 ratio once this run’s r=1 numbers exist, (b) -tmpfs vs ZFS ratio (2.35× on 1p), (c) Synadia/nats bench async file r=1 -~100–400k on NVMe loopback, derated for 10GbE RTT.

-

Buy notes: M.2 via Dual uFF / enablement kit; put -JetStream on NVMe directly, not behind a RAID -controller write-through unless you measure. 1GbE onboard is a trap — -use 10GbE for :6222. Dual Gold is for isolation (nats vs -worm/tree vs OS), not because JS needs 56 cores.

-
-

5. What we are not doing

-
    -
  • MQTT as the Zapier or middleware transport.
    • -
  • UDP for jobs.
    • -
  • Emulated 10G fiber NICs on LXC.
    • -
  • tmpfs as the production store.
    • -
  • r=1 for product streams.
    • -
  • Connect-per-job.
    • -
-

Re-run exhaustive: bash scripts/exhaustive-ns1-study.sh -on NS1.

- - diff --git a/reports/optimal-config/REPORT.md b/reports/optimal-config/REPORT.md deleted file mode 100644 index 8c5eb58..0000000 --- a/reports/optimal-config/REPORT.md +++ /dev/null @@ -1,160 +0,0 @@ -**Progress report — optimal configuration study** · `20260912T055851Z` (UTC) · all code on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`) - -This document folds every ladder we have run (1-core ZFS, NS1-orchestrated, tmpfs maximize, and this exhaustive 8c/16G **ZFS** factorial) plus UDP / MQTT / reconnect probes. It recommends a lab config and a **three-box HP DL360 Gen10** projection. veth/10G was not changed. - ---- - -## 1. Verdict (read this first) - -**Keep NATS + JetStream.** Do not replace the fabric with MQTT, UDP, or a custom persistent-socket protocol for Verae jobs/events/archive. Those are either slower, less durable, or already what NATS is. - -**Lab (NS1, one host, three LXC) — optimal now** - -| Stream | Storage | Replicas | Why | -|--------|---------|----------|-----| -| `ZAPIER_JOBS`, `ZAPIER_WEBHOOKS`, `VERAE_ARCHIVE` | **file** (ZFS) | **3** | Survive a nats LXC death; archive must persist | -| `ZAPIER_EVENTS` | **memory** | **3** | Waiters are latency-sensitive; events rebuild from job status | -| `ZAPIER_USAGE` | file | 3 | Telemetry, limits + max-age | - -Keep **8 cores / 16 GiB / `max_mem: 8G`** on 510–513 (already live). Do **not** leave JetStream on tmpfs. Do **not** drop product streams to r=1. Reuse **one NATS connection per process** (already true in middleware); never connect-per-message. - -**Metal (3× DL360 Gen10) — optimal later** - -Same stream table. File store on **local NVMe/M.2**, not a shared SAN. Cluster + client on **10GbE** (or 25GbE if you already have it). Dual Gold Xeon is surplus CPU for this workload; 8–16 cores dedicated to `nats-server` is enough. Expected JS file r=3: **~40–80k** 128 B pubs/s (about **3–6×** this lab’s 8c ZFS 1p, **2–4×** tmpfs 1p) — bounded by **10GbE replica RTT**, not by Xeon clocks. Core NATS will sit in the **1–3M msgs/s** band until the NIC saturates (~9 Gbit/s ≈ 8–9M × 128 B theoretical; CPU and client will hit first). - ---- - -## 2. What we actually ran (this exhaustive pass) - -Live cluster during this run: LXC 510–513 **8 cores / 16 GiB**, JetStream **on ZFS** (tmpfs from the maximize study was already unmounted). Extra factorial: file/memory × replicas 1/3, 4 KiB file r=3, reconnect-per-message ping, UDP echo 510→511, MQTT QoS0 against nats-a `:1883`. Product streams were not the bench target. - -### 2.1 Cross-study history - -| Study | Env | Core 1p pub | JS file r=3 1p | JS mem r=3 4p | Ping p99 | -| --- | --- | --- | --- | --- | --- | -| `20260912T051237Z` | 1c/1G ZFS (NS1 orch.) | 502,502 | 7,393 | — | 1.377ms | -| `20260912T053120Z` | 8c/16G tmpfs + mem extra | 599,004 | 17,388 | 36,355 | 0.684ms | -| `20260912T055851Z` | 8c/16G ZFS exhaustive `20260912T055851Z` | 662,227 | 14,330 | 37,736 | 1.140ms | - -![JS 1p file r=3 history](charts-optimal/history-js1p.png) - -### 2.2 This run — JetStream factorial - -| Run | What | Pub msgs/s | Pub MB/s | -| --- | --- | --- | --- | -| `js-file-1p-20k-128-r1` | file r=1 1p 128 B | 18,888 | 2.31 | -| `js-file-4p-50k-128-r1` | file r=1 4p 128 B | 24,560 | 3.00 | -| `js-1p-20k-128-r3` | file r=3 1p 128 B | 14,330 | 1.75 | -| `js-4p-50k-128-r3` | file r=3 4p 128 B | 19,232 | 2.35 | -| `js-4p-20k-1k-r3` | file r=3 4p 1 KiB | 15,197 | 14.84 | -| `js-file-1p-20k-4k-r3` | file r=3 1p 4 KiB | 8,673 | 33.88 | -| `js-mem-1p-20k-128-r1` | memory r=1 1p 128 B | 29,972 | 3.66 | -| `js-mem-4p-50k-128-r1` | memory r=1 4p 128 B | 64,923 | 7.93 | -| `js-mem-1p-20k-128-r3` | memory r=3 1p 128 B | 20,188 | 2.46 | -| `js-mem-4p-50k-128-r3` | memory r=3 4p 128 B | 37,736 | 4.61 | -| `js-mem-4p-20k-1k-r3` | memory r=3 4p 1 KiB | 33,916 | 33.12 | - -Replica **1 vs 3** on this stand (file 1p 128 B): r=1 is 18,888 vs r=3 14,330 (1.32× if r=3 is the slower one). Memory r=1 1p 29,972 vs memory r=3 20,188. - -![Replica cost](charts-optimal/replicas.png) - -### 2.3 Delay, reconnect tax, UDP, MQTT - -| Probe | Result | Meaning | -|-------|--------|---------| -| NATS ping (persistent sockets) p50 / p99 | 0.456ms / 1.140ms | Quiet hop with a long-lived TCP conn | -| NATS **reconnect-per-message** p50 / p99 | 0.503ms / 1.750ms | TCP+NATS handshake on every pub — this is the tax to avoid | -| UDP echo 510→511 p99 | 0.363ms | Raw datagram ceiling on the same veth (no NATS) | -| MQTT QoS0 5k×128 B | 44862 pubs/s | nats-server MQTT gateway on `:1883` | - -Core 1p1s 128 B this run: 662,227 pub msgs/s. Flood delay is still backlog/consume_rate, not RTT. - ---- - -## 3. Alternative transports (why we are not switching the fabric) - -NATS already **is** persistent TCP sockets with a tiny binary protocol, automatic reconnect, and optional JetStream durability. “Reduce connection overhead” is a **client** discipline: hold the connection. The reconnect probe exists to prove that opening a socket per job would dominate ping RTT. - -| Idea | Fit for Verae jobs/events/archive | Throughput vs NATS core | Durability | -|------|-----------------------------------|-------------------------|------------| -| **NATS core pub/sub** | Fan-out, request-reply (`verae.billing.*`) | Highest we measured (~0.5–2M msgs/s) | None | -| **NATS JetStream file r=3** | Jobs, webhooks, archive | ~8–23k on this lab; see metal projection | Disk + 1-node loss | -| **NATS JetStream memory r=3** | Events mailbox | ~22–36k on this lab | RAM + 1-node loss; **empty on full restart** | -| **MQTT** (NATS gateway or Mosquitto) | IoT endpoints that already speak MQTT | This probe: 44862 pubs/s QoS0 — typically **well below** NATS core; QoS1 ≈ JetStream-ish with more chatter | QoS1/2 session state; not our WORM model | -| **UDP** | Telemetry that may drop | RTT 0.363ms p99 — fastest hop, **no** reliability, no cluster, no auth | None | -| **Custom persistent sockets / HTTP long-poll** | Worse NATS | You would re-implement reconnect, flow control, and fan-out | DIY | -| **WebSocket** | Browsers only | Extra framing; NATS already has WS for UIs, not for middleware | Same as core/JS behind it | -| **QUIC / WebTransport** | Lossy WAN / browsers | NATS QUIC is not the lab path; 10GbE LAN does not need it | Same | -| **Kafka / Redis streams** | Heavy log replay | Higher ops cost; not on `vmbr1` today | Yes, heavier | - -**MQTT:** NATS documents MQTT as an *enabling* gateway for existing IoT, and prefers NATS end-to-end for greenfield. Zapier cloud never talks NATS or MQTT; it talks HTTPS. Putting MQTT in the middle of timestamp jobs adds protocol translation and QoS timers without helping `jobId → events`. Use MQTT only if a device already cannot speak NATS. - -**UDP:** Fine as a *measurement* of veth RTT. Unusable as the job fabric (no ack, no replica, no flow control). NATS ping is already within a small multiple of UDP on this bridge. - -**Persistence sockets:** Middleware and keep already keep `NATS_URL` connections open. Optimal: one connection (or a small pool) per process, `max_reconnect`, jitter, no `connect()` in the per-job path. The reconnect ladder is the anti-pattern. - ---- - -## 4. Optimal configurations - -### 4.1 NS1 lab (now) - -1. **Leave 8 cores / 16 GiB** on nats-a/b/c and the worker. Host has 40 cores / 377 GiB; this is cheap. -2. **`max_mem: 8G`** stays. Required for memory streams. -3. **File r=3 on ZFS** for jobs/webhooks/archive. tmpfs doubled JS 1p (7.4k→17k) but **loses the stream on reboot** — unacceptable for archive. -4. **Memory r=3 for `ZAPIER_EVENTS`** if we accept “all three nats CTs reboot ⇒ in-flight waiters fall back to HTTP poll.” That matches the designed wait path (`GET /api/status/{jobId}`). -5. **r=1 only for throwaway benches**, never product streams. Replica=3 is the point of three guests. -6. **veth on vmbr1, no fake 10G NICs.** Already 10000Mb/s; JS does not fill it. -7. **Pin cpusets** later if keep/fleet steal; not required to beat these numbers. -8. Clients: persistent NATS connections; pull consumers with bounded `max_ack_pending` for webhooks. - -### 4.2 Three HP DL360 Gen10 (projection — not measured) - -Assumed bill of materials (state it in the buy): - -| Piece | Assumption | -|-------|------------| -| Chassis | 3× DL360 Gen10 1U | -| CPU | Dual 2nd-gen Xeon **Gold** (e.g. 6226R 16c or 6248 20c — **32–40 cores/box**) | -| Memory | DDR4-2933, **192–384 GiB**/box (6–12×32 GiB); NATS will not use most of it | -| Storage | **NVMe M.2 or U.2** for `/var/lib/nats/jetstream` (XFS or ext4, **not** shared ZFS over the network). RAID1 of two NVMe if you want disk HA *inside* a box | -| Network | **10GbE** (FlexibleLOM or PCIe); dedicated VLAN for `:4222`+`:6222`. Do not share with public `vmbr0` traffic | -| OS | Debian/Ubuntu bare metal, `nats-server` systemd, same `nats.conf` as lab (bind private IP only) | - -**What changes vs NS1 LXC** - -| Factor | NS1 today | 3× DL360 | Effect on JS file r=3 | -|--------|-----------|----------|------------------------| -| Failure domain | 1 Proxmox host | 3 chassis, 3 NVMe, 3 NICs | r=3 **means** something | -| Disk | Shared ZFS SSD2 | Local NVMe fsync ~50–150 µs | Big win vs ZFS; similar to tmpfs for sequential 128 B | -| Replica path | veth/bridge (~µs–tens of µs) | 10GbE RTT typically **50–200 µs** | **Slower than same-host tmpfs**, faster than a bad SAN | -| CPU | 8 of 40 shared | 32–40 dedicated Gold cores | Headroom for many clients, not 10× JS | -| NIC | software 10G veth, already ~5 Gbit/s core | real 10GbE ~9 Gbit/s TCP | Core NATS can grow; JS r=3 stays replica-bound | - -**Projected bands** (128 B, 3-node cluster, dedicated 10GbE, local NVMe, 8+ cores pinned to nats-server): - -| Workload | NS1 measured (best) | DL360 projection | Confidence | -|----------|---------------------|------------------|------------| -| Core pub/sub 1p | 0.5–0.8M | **0.8–2M** | Medium — NIC + syscall, plenty of CPU | -| Core 4p4s 1 KiB | ~0.6–0.7M (~0.6 GB/s) | **~1M msgs/s / ~1 GB/s** approaching 10GbE | Medium | -| JS file r=1 | this run r=1 | **80–200k** pubs/s | Medium — NVMe + no replica wait | -| JS file r=3 | 7–23k (ZFS/tmpfs) | **40–80k** pubs/s | Medium-low — replica RTT dominates; 3 NVMe still help vs shared ZFS | -| JS memory r=3 | 22–36k | **50–100k** | Medium-low — RAM + 10GbE ack | -| Ping p99 | 0.7–1.4 ms | **0.2–0.6 ms** | Medium — real NIC but no Proxmox tax | - -These are **not** DL360 measurements. Scale from: (a) our replica-1 vs replica-3 ratio once this run’s r=1 numbers exist, (b) tmpfs vs ZFS ratio (2.35× on 1p), (c) Synadia/nats bench async file r=1 ~100–400k on NVMe loopback, derated for 10GbE RTT. - -**Buy notes:** M.2 via Dual uFF / enablement kit; put JetStream on NVMe **directly**, not behind a RAID controller write-through unless you measure. 1GbE onboard is a trap — use 10GbE for `:6222`. Dual Gold is for isolation (nats vs worm/tree vs OS), not because JS needs 56 cores. - ---- - -## 5. What we are not doing - -- MQTT as the Zapier or middleware transport. -- UDP for jobs. -- Emulated 10G fiber NICs on LXC. -- tmpfs as the production store. -- r=1 for product streams. -- Connect-per-job. - -Re-run exhaustive: `bash scripts/exhaustive-ns1-study.sh` on NS1. diff --git a/reports/optimal-config/REPORT.pdf b/reports/optimal-config/REPORT.pdf deleted file mode 100644 index 513cc78..0000000 Binary files a/reports/optimal-config/REPORT.pdf and /dev/null differ diff --git a/reports/optimal-config/charts-optimal/history-js1p.png b/reports/optimal-config/charts-optimal/history-js1p.png deleted file mode 100644 index 69de19e..0000000 Binary files a/reports/optimal-config/charts-optimal/history-js1p.png and /dev/null differ diff --git a/reports/optimal-config/charts-optimal/replicas.png b/reports/optimal-config/charts-optimal/replicas.png deleted file mode 100644 index 9779f10..0000000 Binary files a/reports/optimal-config/charts-optimal/replicas.png and /dev/null differ diff --git a/scripts/build-nats-bench-report.py b/scripts/build-nats-bench-report.py deleted file mode 100644 index 8ea2701..0000000 --- a/scripts/build-nats-bench-report.py +++ /dev/null @@ -1,247 +0,0 @@ -#!/usr/bin/env python3 -"""Charts + HTML + PDF for the NATS cluster speed report in zapier-decisions/reports.""" -from __future__ import annotations - -import subprocess -import sys -from pathlib import Path - -import matplotlib - -matplotlib.use("Agg") -import matplotlib.pyplot as plt -from matplotlib.ticker import FuncFormatter - -ROOT = Path(__file__).resolve().parents[3] -PKG = Path(__file__).resolve().parents[1] -REPORT_DIR = PKG / "reports" -CHARTS = REPORT_DIR / "charts" -MD = REPORT_DIR / "nats-cluster-bench.md" -HTML = REPORT_DIR / "nats-cluster-bench.html" -PDF = REPORT_DIR / "nats-cluster-bench.pdf" -CSS = ROOT / "scripts" / "docs-print.css" -LUA = ROOT / "scripts" / "pdf-links.lua" - -INDIGO = "#4f46e5" -DEEP = "#312e81" -TEAL = "#047857" -AMBER = "#b45309" -LILAC = "#7c74f0" -INK = "#171a26" -MUTED = "#5b6178" -GRID = "#d9dce8" - -# Run 20260912T045131Z — parsed from nats bench logs (Sub stats, not Pub/Sub aggregate). -CORE = { - "labels": ["1p1s\n50k×128 B", "4p4s\n100k×128 B", "8p8s\n200k×128 B", "4p4s\n50k×1 KiB"], - "agg": [1_200_836, 1_521_256, 2_007_937, 630_460], - "pub": [791_094, 316_312, 333_957, 247_747], - "sub": [747_461, 1_299_634, 1_790_736, 510_216], - "pub_mb": [96.57, 38.61, 40.77, 241.94], - "sub_mb": [91.24, 158.65, 218.60, 498.26], -} -JS = { - "labels": ["1p 20k×128 B", "4p 50k×128 B", "4p 20k×1 KiB", "2p2s pull 20k×128 B"], - "pub": [16_155, 16_607, 13_493, 10_965], - "sub": [None, None, None, 10_942], -} -DELAY = { - "labels": ["Ping\n1k×128 B", "Flood 1p\n5k×128 B", "Flood 4p\n5k×1 KiB", "Flood 4p\n10k×128 B", "Flood 8p\n20k×128 B"], - "kind": ["ping", "flood", "flood", "flood", "flood"], - "p50": [0.286, 248.752, 217.579, 266.672, 466.296], - "p90": [0.332, 274.314, 223.268, 299.073, 499.924], - "p99": [0.734, 279.398, 227.798, 304.233, 505.112], - "avg": [0.307, 238.626, 211.706, 263.186, 453.749], -} - - -def style() -> None: - plt.rcParams.update( - { - "font.family": "sans-serif", - "font.size": 10, - "axes.titlesize": 12, - "axes.titleweight": "semibold", - "axes.labelsize": 10, - "axes.edgecolor": GRID, - "axes.labelcolor": INK, - "text.color": INK, - "xtick.color": MUTED, - "ytick.color": MUTED, - "figure.facecolor": "white", - "axes.facecolor": "white", - "axes.grid": True, - "grid.color": GRID, - "grid.linewidth": 0.8, - "legend.frameon": False, - "savefig.bbox": "tight", - "savefig.dpi": 160, - "savefig.facecolor": "white", - } - ) - - -def k_fmt(x: float, _pos: int | None = None) -> str: - if x >= 1_000_000: - return f" {x / 1_000_000:.2f}M".strip() - if x >= 1000: - return f"{x / 1000:.0f}k" - return f"{x:.0f}" - - -def save(fig: plt.Figure, name: str) -> None: - CHARTS.mkdir(parents=True, exist_ok=True) - path = CHARTS / name - fig.savefig(path, dpi=160) - plt.close(fig) - print(f"wrote {path}") - - -def chart_core_msgs() -> None: - fig, ax = plt.subplots(figsize=(9.2, 4.4)) - x = range(len(CORE["labels"])) - w = 0.25 - ax.bar([i - w for i in x], CORE["agg"], w, label="Aggregate", color=DEEP) - ax.bar(list(x), CORE["pub"], w, label="Publish", color=INDIGO) - ax.bar([i + w for i in x], CORE["sub"], w, label="Subscribe", color=TEAL) - ax.set_xticks(list(x), CORE["labels"]) - ax.set_ylabel("messages / second") - ax.set_title("Core NATS throughput (fire-and-forget pub/sub)") - ax.yaxis.set_major_formatter(FuncFormatter(k_fmt)) - ax.legend(loc="upper left") - ax.set_axisbelow(True) - save(fig, "core-throughput.png") - - -def chart_js_msgs() -> None: - fig, ax = plt.subplots(figsize=(9.2, 4.4)) - colors = [INDIGO, INDIGO, AMBER, LILAC] - ax.bar(JS["labels"], JS["pub"], color=colors) - ax.set_ylabel("durable publish messages / second") - ax.set_title("JetStream file store, replicas=3 (product-stream settings)") - ax.yaxis.set_major_formatter(FuncFormatter(k_fmt)) - ax.set_axisbelow(True) - for i, v in enumerate(JS["pub"]): - ax.text(i, v + 250, f"{v:,}", ha="center", va="bottom", fontsize=9, color=MUTED) - save(fig, "js-throughput.png") - - -def chart_core_vs_js() -> None: - fig, ax = plt.subplots(figsize=(9.2, 4.4)) - labels = ["1 publisher\n128 B", "4 publishers\n128 B", "4 publishers\n1 KiB"] - core_pub = [791_094, 316_312, 247_747] - js_pub = [16_155, 16_607, 13_493] - x = range(len(labels)) - w = 0.35 - ax.bar([i - w / 2 for i in x], core_pub, w, label="Core NATS (no disk)", color=INDIGO) - ax.bar([i + w / 2 for i in x], js_pub, w, label="JetStream r=3 file", color=AMBER) - ax.set_xticks(list(x), labels) - ax.set_yscale("log") - ax.set_ylabel("publish messages / second (log)") - ax.set_title("Core vs JetStream: two different jobs") - ax.legend(loc="upper right") - ax.set_axisbelow(True) - save(fig, "core-vs-js.png") - - -def chart_bytes() -> None: - fig, axes = plt.subplots(1, 2, figsize=(9.2, 4.2)) - labels = ["128 B\n4p4s", "1 KiB\n4p4s"] - msgs = [1_521_256, 630_460] - mb = [185.70, 615.68] - axes[0].bar(labels, msgs, color=[INDIGO, AMBER]) - axes[0].set_title("Aggregate messages / second") - axes[0].yaxis.set_major_formatter(FuncFormatter(k_fmt)) - axes[0].set_axisbelow(True) - axes[1].bar(labels, mb, color=[INDIGO, AMBER]) - axes[1].set_title("Aggregate MB / second") - axes[1].set_ylabel("MB/s") - axes[1].set_axisbelow(True) - for ax, vals, fmt in ( - (axes[0], msgs, lambda v: f"{v/1e6:.2f}M"), - (axes[1], mb, lambda v: f"{v:.0f}"), - ): - for i, v in enumerate(vals): - ax.text(i, v * 1.02, fmt(v), ha="center", va="bottom", fontsize=9, color=MUTED) - fig.suptitle("Core NATS: bigger payloads move more bytes, fewer messages", fontsize=12, fontweight="semibold") - fig.tight_layout() - save(fig, "payload-size.png") - - -def chart_delay() -> None: - fig, ax = plt.subplots(figsize=(9.2, 4.6)) - x = range(len(DELAY["labels"])) - w = 0.25 - ax.bar([i - w for i in x], DELAY["p50"], w, label="p50", color=TEAL) - ax.bar(list(x), DELAY["p90"], w, label="p90", color=INDIGO) - ax.bar([i + w for i in x], DELAY["p99"], w, label="p99", color=AMBER) - ax.set_xticks(list(x), DELAY["labels"]) - ax.set_yscale("log") - ax.set_ylabel("milliseconds (log)") - ax.set_title("Round-trip delay: sequential ping vs burst flood") - ax.axhline(1.0, color=GRID, linestyle="--", linewidth=1) - ax.legend(loc="upper left") - ax.set_axisbelow(True) - save(fig, "delay-percentiles.png") - - -def render() -> None: - if not MD.exists(): - raise SystemExit(f"missing {MD}") - header = HTML.with_suffix(".hdr.html") - banner = HTML.with_suffix(".ban.html") - css_text = CSS.read_text(encoding="utf-8") - header.write_text(f"\n", encoding="utf-8") - banner.write_text( - '
' - '' - '
Verae Time × Zapier · progress report
' - "

NATS cluster message speed

" - '
packages/zapier-decisions/reports/nats-cluster-bench.md
' - "
\n", - encoding="utf-8", - ) - r = subprocess.run( - [ - "pandoc", - str(MD), - "-o", - str(HTML), - "--standalone", - f"--resource-path={REPORT_DIR}", - "--syntax-highlighting=breezedark", - "--metadata=title=NATS cluster message speed", - f"--include-in-header={header}", - f"--include-before-body={banner}", - f"--lua-filter={LUA}", - ], - capture_output=True, - text=True, - ) - header.unlink(missing_ok=True) - banner.unlink(missing_ok=True) - if r.returncode != 0: - sys.stderr.write(r.stderr) - raise SystemExit(f"pandoc failed: {r.returncode}") - w = subprocess.run(["weasyprint", str(HTML), str(PDF)], capture_output=True, text=True) - if w.returncode != 0: - sys.stderr.write(w.stderr) - raise SystemExit(f"weasyprint failed: {w.returncode}") - print(f"wrote {HTML}") - print(f"wrote {PDF}") - - -def main() -> int: - style() - chart_core_msgs() - chart_js_msgs() - chart_core_vs_js() - chart_bytes() - chart_delay() - render() - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/scripts/jobbus-farm.sh b/scripts/jobbus-farm.sh deleted file mode 100755 index 4f85abe..0000000 --- a/scripts/jobbus-farm.sh +++ /dev/null @@ -1,120 +0,0 @@ -#!/usr/bin/env bash -# Farm multi-build pieces from jobs/pieces.yaml (or PIECES_FILE) onto job-bus. -set -euo pipefail -export PATH="${HOME}/bin:/opt/homebrew/bin:/usr/local/bin:${PATH}" -ROOT="$(cd "$(dirname "$0")/.." && pwd)" -cd "$ROOT" - -if [[ -f "${HOME}/.config/nats/job-bus.env" ]]; then - set -a; # shellcheck disable=SC1090 - source "${HOME}/.config/nats/job-bus.env"; set +a -elif [[ -f /etc/nats/job-bus.env ]]; then - set -a; source /etc/nats/job-bus.env; set +a -else - echo "missing job-bus.env" >&2; exit 1 -fi - -PIECES_FILE="${PIECES_FILE:-$ROOT/jobs/pieces.yaml}" -[[ -f "$PIECES_FILE" ]] || { echo "missing $PIECES_FILE (copy from job-bus template)" >&2; exit 1; } - -PY="${JOB_BUS_PYTHON:-}" -if [[ -z "$PY" ]]; then - if [[ -x "$HOME/job-bus/venv/bin/python" ]]; then PY="$HOME/job-bus/venv/bin/python" - elif [[ -x /opt/job-bus/venv/bin/python ]]; then PY=/opt/job-bus/venv/bin/python - else PY=python3 - fi -fi - -WATCH=0 -CLEAN_AFTER=0 -PROGRESS=0 -while [[ $# -gt 0 ]]; do - case "$1" in - --watch) WATCH=1; shift ;; - --progress) PROGRESS=1; shift ;; - --cleanup-after) CLEAN_AFTER=1; shift ;; - -h|--help) - echo "usage: scripts/jobbus-farm.sh [--watch] [--progress] [--cleanup-after]" - exit 0 ;; - *) echo "unknown: $1" >&2; exit 1 ;; - esac -done - -# Ensure PyYAML available in venv lightly -"$PY" -c 'import yaml' 2>/dev/null || "$PY" -m pip install -q pyyaml - -MAP=$("$PY" - "$PIECES_FILE" "$ROOT" <<'PY' -import json, os, sys, datetime -from pathlib import Path -try: - import yaml -except ImportError: - raise SystemExit("PyYAML required") - -path, root = sys.argv[1], sys.argv[2] -doc = yaml.safe_load(open(path)) -defaults = doc.get("defaults") or {} -repo = Path(root).name -date = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") -batch = (doc.get("batch") or f"{repo}-{date}").replace("{{REPO_NAME}}", repo).replace("{{DATE}}", date) -pieces = doc.get("pieces") or [] -out_dir = Path(root) / "jobs" / "submitted" / batch -out_dir.mkdir(parents=True, exist_ok=True) -ids = [] -for i, p in enumerate(pieces, 1): - pid = str(p.get("id") or f"piece-{i:02d}") - kind = p.get("kind") or defaults.get("kind") or "cpu" - requires = p.get("requires") if "requires" in p else defaults.get("requires") or [] - if requires is None: - requires = [] - cmd = p.get("cmd") - if not cmd: - raise SystemExit(f"piece {pid} missing cmd") - job_id = f"job-{batch}-{pid}" - body = { - "id": job_id, - "kind": kind, - "requires": list(requires), - "cmd": list(cmd), - "timeout_sec": int(p.get("timeout_sec") or defaults.get("timeout_sec") or 3600), - "cleanup": p.get("cleanup") or defaults.get("cleanup") or "none", - "meta": {"batch": batch, "piece": pid, "repo": repo, **(p.get("meta") or {})}, - "ts": datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z"), - } - if p.get("cwd"): - body["cwd"] = p["cwd"] - else: - # default cwd: repo root — only works if worker can see this path (shared mount). - # Until SSHFS is set, prefer pieces that use absolute worker paths or staged inputs. - body["cwd"] = root - jf = out_dir / f"{pid}.json" - jf.write_text(json.dumps(body, indent=2) + "\n") - ids.append({"id": job_id, "kind": kind, "file": str(jf), "piece": pid}) -print(json.dumps({"batch": batch, "jobs": ids, "count": len(ids)})) -PY -) - -BATCH=$(echo "$MAP" | "$PY" -c 'import json,sys; print(json.load(sys.stdin)["batch"])') -COUNT=$(echo "$MAP" | "$PY" -c 'import json,sys; print(json.load(sys.stdin)["count"])') -echo "batch=$BATCH pieces=$COUNT" - -echo "$MAP" | "$PY" -c 'import json,sys; d=json.load(sys.stdin); print("\n".join(j["file"] for j in d["jobs"]))' | while read -r jf; do - submit-job --json "$jf" -done - -echo "submitted $COUNT jobs for batch $BATCH" -echo "track: job-progress --batch $BATCH --expect $COUNT" -echo "clean: job-cleanup --batch $BATCH" - -if [[ "$PROGRESS" -eq 1 ]]; then - job-progress --batch "$BATCH" --expect "$COUNT" || true -fi - -if [[ "$WATCH" -eq 1 ]]; then - # simple wait via status subscription count*2 (running+done) approx using progress expect - job-progress --batch "$BATCH" --expect "$COUNT" || true -fi - -if [[ "$CLEAN_AFTER" -eq 1 ]]; then - job-cleanup --batch "$BATCH" || true -fi