# Action log ## 2026-09-12 — exhaustive optimal-config study - 8c/16G ZFS factorial: file/memory × r=1/r=3, 4 KiB, reconnect, UDP echo, MQTT QoS0 on nats-a :1883 (then removed). - JS memory r=1 4p **65k** pubs/s; file r=3 1p **14.3k**; MQTT QoS0 **45k**; UDP p99 **0.36 ms**. - Combined report + DL360 Gen10 projection: `reports/optimal-config/REPORT.{md,html,pdf}`. - Verdict: keep NATS+JetStream; no MQTT/UDP fabric; metal = 3 NVMe + 10GbE, expect JS r=3 ~40–80k. ## 2026-09-12 — maximized NS1 NATS study - `pct set` 510–513 to 8 cores / 16 GiB; `max_mem: 8G`; tmpfs 8G on JetStream for the ladder only. - Run `20260912T053120Z`. JS file 1p 128 B **7.4k → 17.4k (2.35×)**; JS memory 4p **36.4k**; ping p99 **1.38 → 0.68 ms**. - tmpfs unmounted after; product streams on ZFS again. Resources and max_mem **left raised**. - Report: `reports/ns1-maximize/`. ## 2026-09-12 — NS1-host NATS study (second run) - All orchestration, nats bench, matplotlib, pandoc, and weasyprint ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`). Laptop not in the path. - Script: `verae-nats-cluster/scripts/study-on-ns1.sh` (refuses unless hostname is NS1). - Run `20260912T051237Z`. Report: `reports/ns1-host/nats-cluster-bench-ns1.{md,html,pdf}`. - Host load ~8.5. JS 1p 128 B ~7.4k durable pubs/s; ping p99 1.38 ms. Methodology + fine-tuning in that report. ## 2026-09-12 — NATS cluster speed report - Bench from LXC 510 vs nats-a/b/c (`20260912T045131Z`). - Full write-up with charts in `reports/nats-cluster-bench.{md,html,pdf}`. - JetStream r=3 file ~16k durable 128 B pubs/s; ping RTT ~0.3 ms; flood queueing 150–500 ms. ## 2026-09-12 — test env on the 3-node cluster - Fleet, keep, middleware, billing, and access planes `NATS_URL` → `10.10.10.21–23:4222`. - Streams ZAPIER_JOBS/EVENTS/WEBHOOKS/USAGE + VERAE_ARCHIVE replicas=3. - px-worker keep worm/tree 200 on the cluster. Backup timer enabled. Uptime loop on px-worker. - MOCK_VERAE still true. Zapier push still waiting on login. nkeys/mTLS not flipped (would require client creds). ## 2026-09-12 — NATS cluster on three Proxmox LXC - New repo **verae-nats-cluster**: LXC 511/512/513 (`nats-a/b/c`, `10.10.10.21–23`) JetStream cluster on `vmbr1`. - Lab loopback `127.0.0.1:4222` unchanged until cut-over. - Test: stream replicas=3; same configs later move to distinct hardware. ## 2026-09-11 — proxmox worker + deploy/monitor/backup - NS1 **is** Proxmox (`vmbr0` public, `vmbr1` `10.10.10.0/24`). New repo **verae-proxmox** creates LXC **510** `verae-px-worker` at `10.10.10.20`. - NATS for guests: socat `10.10.10.1:4222` → `127.0.0.1:4222` (never vmbr0). - **verae-uptime** off-box HTTPS GET; **verae-backup** tar snapshots; **verae-deploy** tagged npm ci. - Fleet `overlays/ns1/` checked in; `start.sh` no longer rewrites fleet/machines JSON. - User systemd + linger for keep and fleet. ## 2026-09-11 — bootstrap - Created this repo to record decisions while executing cleanup items 1–5 (and a second pass on 2–5). - Assumption: single developer, lab only; breaking changes allowed. ## Step 1 — account-balance is the prepaid source of truth - Persist books to `BOOKS_PATH` (atomic JSON). - Mutations go through `balance.adjust` / `usage.recorded`; edge cache `balanceCents` is updated from the books reply only. - Local cache mutate only when `NATS_URL` and `ACCOUNT_BALANCE_URL` are both unset (tests). ## Step 2 — identity join - `zappier-identity` mailbox: `verae.identity.bind` / `verae.identity.lookup`. - Portal + middleware tenants store `veraeUserId`; JWT never public. ## Step 3 — access planes are the public doors - zappier-edge binds loopback (`127.0.0.1:3000`). - Public: access-web `:3021`, access-api `:3022`, access-leaf `:3023`, access-zapier `:3024`, access-staff `:3025`. ## Step 4 — leaf hardening - `verae-nats-accounts` holds nats.conf + subject policy. - Leaf token required; billing subjects denied in policy file and access-leaf. ## Step 5 — fleet spawn - Prefer local machines for HTTP `spawn.command` services. - Disable lan-134 unless `FLEET_ENABLE_LAN134=1`. - SSH spawn timeout 8s; failed hosts skipped. ## 2026-09-11 — catalog live doors - Bottom of https://zapier.georgelambert.org/ lists each HTTPS door with description, screenshot, and Open link. - UI-REVIEW.pdf includes the live-door atlas. Walkthrough §13. ## 2026-09-11 — Caddy host names live - Installed `/etc/caddy/sites/zapier-services.georgelambert.org.caddy` on NS1. HTTPS: portal, api, leaf, zap, staff, iam, login, cs, sales, acct, mw, fleet. - Not proxied: NATS, keep, workers, loopback edge. ## 2026-09-11 — host names - Mapped every running service to **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`). - Suggested `*.zapier.georgelambert.org` names for public doors; only `zapier.georgelambert.org` (docs) and `git.georgelambert.org` exist in Caddy today. ## 2026-09-11 — docs sync, bootstrap, NS1 all-in-one - Catalog and ops docs: public portal is access-web `:3021/portal/`; edge is loopback; IAM + keep listed. - New `verae-bootstrap` repo: clone + npm for `ns1-all-in-one`, `control-plane`, `ns1-archive`, `lan-worker`. - Fleet starts `staff-iam` and `staff-session`. NS1 all-in-one uses edge `:13000` because host `:3000` is taken; keep still owns archive workers. `better-sqlite3` must be rebuilt on Linux (`npm rebuild`). ## 2026-09-11 — verae-keep on NS1 - New repo `verae-keep`: keep `:3860` restarts crashed units; watch `:3861` restarts keep; `guard.sh` restarts watch. - Honors admin pause/stop via intent files + `POST /intent`. Fleet writes `data//intent.json` (local and SSH). - Tested on 70.88.205.138: crash-restart, pause-hold, watch respawn. webhook-deliver brought back. NATS observed only. ## 2026-09-11 — IAM JSON doors closed - CS/sales/accounting/staff JSON list and sales PUT pricing now go through IAM (401 JSON, 302 HTML). - Fleet service env sets `STAFF_AUTH=1` with `STAFF_IAM_URL`. - Re-walk: cs credit 200 agent=`cs`; cs export 403; operator Fleet POST 200; admin `/me` `permissions:['*']`. ## 2026-09-11 — IAM on lab, portal door, hardening - Restarted fleet with `STAFF_IAM_URL`. Walk: cs credits (agent=`cs`) and is 403 on accounting export; operator fleet POST 200, cs 403; admin `/me` permissions `*`. - Portal public door `verae-access-web` `:3021/portal/` (static + `/portal/api` → loopback edge). - IAM sessions persist in JSON; login rate-limit per IP+username (8/10min); credits stamp IAM username as authz principal. - lan-134 remains disabled unless `FLEET_ENABLE_LAN134=1`. Receipt PDF branded (indigo header + legal footer). ## 2026-09-11 — staff IAM - New `verae-staff-iam` :3028 — users, roles, permissions, sessions, audit UI. - Admin login uses IAM when `STAFF_IAM_URL` is set; tabs hide without permission. - CS/sales/accounting/access-staff/fleet mutating APIs check `cs.credit`, `sales.quote`, `accounting.export`, `fleet.operate`, etc. - Seed: admin/cs/sales/accounting/operator. Shared staff key is fallback only. ## 2026-09-11 — last three UI leftovers - Swagger `/docs` stays stock; banner names it OpenAPI explorer. - Shared template `verae-staff-ui` for CS + access-staff. - `STAFF_COOKIE_DOMAIN` + nginx example for multi-host. ## 2026-09-11 — names, staff session, exclusive jobs.events - Account-balance stores display names (`customer.put` + lookup by name). Edge writes names on customer create/edit; CS/sales/accounting/staff join from edge if the ledger has no name. - New repo `verae-staff-session` (:3027). Department HTML redirects when `STAFF_AUTH=1`. - `JOBS_EVENTS_EXCLUSIVE=1` makes `verae-jobs-events` the durable consumer; middleware webhook router backs off. - Catalog index is cards. Disabled fleet machines are grey, not degraded yellow. ## 2026-09-11 — UI design-system pass - Restyled CS/sales/accounting/access-staff to portal indigo; dollars + names. - Fleet ⋯ menus, role chips, Docs cards, 0.0.0.0 copy, simulator indigo (orange = faults). - Portal masked API key; admin customer drawer; empty invoice illustrations. - Catalog system-ui. New independent repo **UI-Docs** (walkthrough, screenshots, UI-REVIEW.pdf). ## Second pass (2–5) - Identity used from portal when `IDENTITY_URL` set. - Leaf loads allow-list from nats-accounts `policy.json`. - Staff plane is a first-class mailbox (`verae-access-staff`). - jobs.events mailbox process `verae-jobs-events` (passthrough + count).