master-zapier-plan-draft/packages/zapier-decisions/LOG.md

# Action log

## 2026-09-12 — maximized NS1 NATS study

- `pct set` 510–513 to 8 cores / 16 GiB; `max_mem: 8G`; tmpfs 8G on JetStream for the ladder only.
- Run `20260912T053120Z`. JS file 1p 128 B **7.4k → 17.4k (2.35×)**; JS memory 4p **36.4k**; ping p99 **1.38 → 0.68 ms**.
- tmpfs unmounted after; product streams on ZFS again. Resources and max_mem **left raised**.
- Report: `reports/ns1-maximize/`.

## 2026-09-12 — NS1-host NATS study (second run)

- All orchestration, nats bench, matplotlib, pandoc, and weasyprint ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`). Laptop not in the path.
- Script: `verae-nats-cluster/scripts/study-on-ns1.sh` (refuses unless hostname is NS1).
- Run `20260912T051237Z`. Report: `reports/ns1-host/nats-cluster-bench-ns1.{md,html,pdf}`.
- Host load ~8.5. JS 1p 128 B ~7.4k durable pubs/s; ping p99 1.38 ms. Methodology + fine-tuning in that report.

## 2026-09-12 — NATS cluster speed report

- Bench from LXC 510 vs nats-a/b/c (`20260912T045131Z`).
- Full write-up with charts in `reports/nats-cluster-bench.{md,html,pdf}`.
- JetStream r=3 file ~16k durable 128 B pubs/s; ping RTT ~0.3 ms; flood queueing 150–500 ms.

## 2026-09-12 — test env on the 3-node cluster

- Fleet, keep, middleware, billing, and access planes `NATS_URL` → `10.10.10.21–23:4222`.
- Streams ZAPIER_JOBS/EVENTS/WEBHOOKS/USAGE + VERAE_ARCHIVE replicas=3.
- px-worker keep worm/tree 200 on the cluster. Backup timer enabled. Uptime loop on px-worker.
- MOCK_VERAE still true. Zapier push still waiting on login. nkeys/mTLS not flipped (would require client creds).

## 2026-09-12 — NATS cluster on three Proxmox LXC

- New repo **verae-nats-cluster**: LXC 511/512/513 (`nats-a/b/c`, `10.10.10.21–23`) JetStream cluster on `vmbr1`.
- Lab loopback `127.0.0.1:4222` unchanged until cut-over.
- Test: stream replicas=3; same configs later move to distinct hardware.

## 2026-09-11 — proxmox worker + deploy/monitor/backup

- NS1 **is** Proxmox (`vmbr0` public, `vmbr1` `10.10.10.0/24`). New repo **verae-proxmox** creates LXC **510** `verae-px-worker` at `10.10.10.20`.
- NATS for guests: socat `10.10.10.1:4222` → `127.0.0.1:4222` (never vmbr0).
- **verae-uptime** off-box HTTPS GET; **verae-backup** tar snapshots; **verae-deploy** tagged npm ci.
- Fleet `overlays/ns1/` checked in; `start.sh` no longer rewrites fleet/machines JSON.
- User systemd + linger for keep and fleet.

## 2026-09-11 — bootstrap

- Created this repo to record decisions while executing cleanup items 1–5 (and a second pass on 2–5).
- Assumption: single developer, lab only; breaking changes allowed.

## Step 1 — account-balance is the prepaid source of truth

- Persist books to `BOOKS_PATH` (atomic JSON).
- Mutations go through `balance.adjust` / `usage.recorded`; edge cache `balanceCents` is updated from the books reply only.
- Local cache mutate only when `NATS_URL` and `ACCOUNT_BALANCE_URL` are both unset (tests).

## Step 2 — identity join

- `zappier-identity` mailbox: `verae.identity.bind` / `verae.identity.lookup`.
- Portal + middleware tenants store `veraeUserId`; JWT never public.

## Step 3 — access planes are the public doors

- zappier-edge binds loopback (`127.0.0.1:3000`).
- Public: access-web `:3021`, access-api `:3022`, access-leaf `:3023`, access-zapier `:3024`, access-staff `:3025`.

## Step 4 — leaf hardening

- `verae-nats-accounts` holds nats.conf + subject policy.
- Leaf token required; billing subjects denied in policy file and access-leaf.

## Step 5 — fleet spawn

- Prefer local machines for HTTP `spawn.command` services.
- Disable lan-134 unless `FLEET_ENABLE_LAN134=1`.
- SSH spawn timeout 8s; failed hosts skipped.

## 2026-09-11 — catalog live doors

- Bottom of https://zapier.georgelambert.org/ lists each HTTPS door with description, screenshot, and Open link.
- UI-REVIEW.pdf includes the live-door atlas. Walkthrough §13.

## 2026-09-11 — Caddy host names live

- Installed `/etc/caddy/sites/zapier-services.georgelambert.org.caddy` on NS1. HTTPS: portal, api, leaf, zap, staff, iam, login, cs, sales, acct, mw, fleet.
- Not proxied: NATS, keep, workers, loopback edge.

## 2026-09-11 — host names

- Mapped every running service to **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`).
- Suggested `*.zapier.georgelambert.org` names for public doors; only `zapier.georgelambert.org` (docs) and `git.georgelambert.org` exist in Caddy today.

## 2026-09-11 — docs sync, bootstrap, NS1 all-in-one

- Catalog and ops docs: public portal is access-web `:3021/portal/`; edge is loopback; IAM + keep listed.
- New `verae-bootstrap` repo: clone + npm for `ns1-all-in-one`, `control-plane`, `ns1-archive`, `lan-worker`.
- Fleet starts `staff-iam` and `staff-session`. NS1 all-in-one uses edge `:13000` because host `:3000` is taken; keep still owns archive workers. `better-sqlite3` must be rebuilt on Linux (`npm rebuild`).

## 2026-09-11 — verae-keep on NS1

- New repo `verae-keep`: keep `:3860` restarts crashed units; watch `:3861` restarts keep; `guard.sh` restarts watch.
- Honors admin pause/stop via intent files + `POST /intent`. Fleet writes `data/<instance>/intent.json` (local and SSH).
- Tested on 70.88.205.138: crash-restart, pause-hold, watch respawn. webhook-deliver brought back. NATS observed only.

## 2026-09-11 — IAM JSON doors closed

- CS/sales/accounting/staff JSON list and sales PUT pricing now go through IAM (401 JSON, 302 HTML).
- Fleet service env sets `STAFF_AUTH=1` with `STAFF_IAM_URL`.
- Re-walk: cs credit 200 agent=`cs`; cs export 403; operator Fleet POST 200; admin `/me` `permissions:['*']`.

## 2026-09-11 — IAM on lab, portal door, hardening

- Restarted fleet with `STAFF_IAM_URL`. Walk: cs credits (agent=`cs`) and is 403 on accounting export; operator fleet POST 200, cs 403; admin `/me` permissions `*`.
- Portal public door `verae-access-web` `:3021/portal/` (static + `/portal/api` → loopback edge).
- IAM sessions persist in JSON; login rate-limit per IP+username (8/10min); credits stamp IAM username as authz principal.
- lan-134 remains disabled unless `FLEET_ENABLE_LAN134=1`. Receipt PDF branded (indigo header + legal footer).

## 2026-09-11 — staff IAM

- New `verae-staff-iam` :3028 — users, roles, permissions, sessions, audit UI.
- Admin login uses IAM when `STAFF_IAM_URL` is set; tabs hide without permission.
- CS/sales/accounting/access-staff/fleet mutating APIs check `cs.credit`, `sales.quote`, `accounting.export`, `fleet.operate`, etc.
- Seed: admin/cs/sales/accounting/operator. Shared staff key is fallback only.

## 2026-09-11 — last three UI leftovers

- Swagger `/docs` stays stock; banner names it OpenAPI explorer.
- Shared template `verae-staff-ui` for CS + access-staff.
- `STAFF_COOKIE_DOMAIN` + nginx example for multi-host.

## 2026-09-11 — names, staff session, exclusive jobs.events

- Account-balance stores display names (`customer.put` + lookup by name). Edge writes names on customer create/edit; CS/sales/accounting/staff join from edge if the ledger has no name.
- New repo `verae-staff-session` (:3027). Department HTML redirects when `STAFF_AUTH=1`.
- `JOBS_EVENTS_EXCLUSIVE=1` makes `verae-jobs-events` the durable consumer; middleware webhook router backs off.
- Catalog index is cards. Disabled fleet machines are grey, not degraded yellow.

## 2026-09-11 — UI design-system pass

- Restyled CS/sales/accounting/access-staff to portal indigo; dollars + names.
- Fleet ⋯ menus, role chips, Docs cards, 0.0.0.0 copy, simulator indigo (orange = faults).
- Portal masked API key; admin customer drawer; empty invoice illustrations.
- Catalog system-ui. New independent repo **UI-Docs** (walkthrough, screenshots, UI-REVIEW.pdf).

## Second pass (2–5)

- Identity used from portal when `IDENTITY_URL` set.
- Leaf loads allow-list from nats-accounts `policy.json`.
- Staff plane is a first-class mailbox (`verae-access-staff`).
- jobs.events mailbox process `verae-jobs-events` (passthrough + count).