master-zapier-plan-draft/packages/zapier-decisions/LOG.md

Action log

2026-09-12 — exhaustive optimal-config study

• 8c/16G ZFS factorial: file/memory × r=1/r=3, 4 KiB, reconnect, UDP echo, MQTT QoS0 on nats-a :1883 (then removed).
• JS memory r=1 4p 65k pubs/s; file r=3 1p 14.3k; MQTT QoS0 45k; UDP p99 0.36 ms.
• Combined report + DL360 Gen10 projection: reports/optimal-config/REPORT.{md,html,pdf}.
• Verdict: keep NATS+JetStream; no MQTT/UDP fabric; metal = 3 NVMe + 10GbE, expect JS r=3 ~40–80k.

2026-09-12 — maximized NS1 NATS study

• pct set 510–513 to 8 cores / 16 GiB; max_mem: 8G; tmpfs 8G on JetStream for the ladder only.
• Run 20260912T053120Z. JS file 1p 128 B 7.4k → 17.4k (2.35×); JS memory 4p 36.4k; ping p99 1.38 → 0.68 ms.
• tmpfs unmounted after; product streams on ZFS again. Resources and max_mem left raised.
• Report: reports/ns1-maximize/.

2026-09-12 — NS1-host NATS study (second run)

• All orchestration, nats bench, matplotlib, pandoc, and weasyprint ran on NS1.GEORGELAMBERT.ORG (70.88.205.138). Laptop not in the path.
• Script: verae-nats-cluster/scripts/study-on-ns1.sh (refuses unless hostname is NS1).
• Run 20260912T051237Z. Report: reports/ns1-host/nats-cluster-bench-ns1.{md,html,pdf}.
• Host load ~8.5. JS 1p 128 B ~7.4k durable pubs/s; ping p99 1.38 ms. Methodology + fine-tuning in that report.

2026-09-12 — NATS cluster speed report

• Bench from LXC 510 vs nats-a/b/c (20260912T045131Z).
• Full write-up with charts in reports/nats-cluster-bench.{md,html,pdf}.
• JetStream r=3 file ~16k durable 128 B pubs/s; ping RTT ~0.3 ms; flood queueing 150–500 ms.

2026-09-12 — test env on the 3-node cluster

• Fleet, keep, middleware, billing, and access planes NATS_URL → 10.10.10.21–23:4222.
• Streams ZAPIER_JOBS/EVENTS/WEBHOOKS/USAGE + VERAE_ARCHIVE replicas=3.
• px-worker keep worm/tree 200 on the cluster. Backup timer enabled. Uptime loop on px-worker.
• MOCK_VERAE still true. Zapier push still waiting on login. nkeys/mTLS not flipped (would require client creds).

2026-09-12 — NATS cluster on three Proxmox LXC

• New repo verae-nats-cluster: LXC 511/512/513 (nats-a/b/c, 10.10.10.21–23) JetStream cluster on vmbr1.
• Lab loopback 127.0.0.1:4222 unchanged until cut-over.
• Test: stream replicas=3; same configs later move to distinct hardware.

2026-09-11 — proxmox worker + deploy/monitor/backup

• NS1 is Proxmox (vmbr0 public, vmbr1 10.10.10.0/24). New repo verae-proxmox creates LXC 510 verae-px-worker at 10.10.10.20.
• NATS for guests: socat 10.10.10.1:4222 → 127.0.0.1:4222 (never vmbr0).
• verae-uptime off-box HTTPS GET; verae-backup tar snapshots; verae-deploy tagged npm ci.
• Fleet overlays/ns1/ checked in; start.sh no longer rewrites fleet/machines JSON.
• User systemd + linger for keep and fleet.

2026-09-11 — bootstrap

• Created this repo to record decisions while executing cleanup items 1–5 (and a second pass on 2–5).
• Assumption: single developer, lab only; breaking changes allowed.

Step 1 — account-balance is the prepaid source of truth

• Persist books to BOOKS_PATH (atomic JSON).
• Mutations go through balance.adjust / usage.recorded; edge cache balanceCents is updated from the books reply only.
• Local cache mutate only when NATS_URL and ACCOUNT_BALANCE_URL are both unset (tests).

Step 2 — identity join

• zappier-identity mailbox: verae.identity.bind / verae.identity.lookup.
• Portal + middleware tenants store veraeUserId; JWT never public.

Step 3 — access planes are the public doors

• zappier-edge binds loopback (127.0.0.1:3000).
• Public: access-web :3021, access-api :3022, access-leaf :3023, access-zapier :3024, access-staff :3025.

Step 4 — leaf hardening

• verae-nats-accounts holds nats.conf + subject policy.
• Leaf token required; billing subjects denied in policy file and access-leaf.

Step 5 — fleet spawn

• Prefer local machines for HTTP spawn.command services.
• Disable lan-134 unless FLEET_ENABLE_LAN134=1.
• SSH spawn timeout 8s; failed hosts skipped.

2026-09-11 — catalog live doors

• Bottom of https://zapier.georgelambert.org/ lists each HTTPS door with description, screenshot, and Open link.
• UI-REVIEW.pdf includes the live-door atlas. Walkthrough §13.

2026-09-11 — Caddy host names live

• Installed /etc/caddy/sites/zapier-services.georgelambert.org.caddy on NS1. HTTPS: portal, api, leaf, zap, staff, iam, login, cs, sales, acct, mw, fleet.
• Not proxied: NATS, keep, workers, loopback edge.

2026-09-11 — host names

• Mapped every running service to NS1.GEORGELAMBERT.ORG (70.88.205.138).
• Suggested *.zapier.georgelambert.org names for public doors; only zapier.georgelambert.org (docs) and git.georgelambert.org exist in Caddy today.

2026-09-11 — docs sync, bootstrap, NS1 all-in-one

• Catalog and ops docs: public portal is access-web :3021/portal/; edge is loopback; IAM + keep listed.
• New verae-bootstrap repo: clone + npm for ns1-all-in-one, control-plane, ns1-archive, lan-worker.
• Fleet starts staff-iam and staff-session. NS1 all-in-one uses edge :13000 because host :3000 is taken; keep still owns archive workers. better-sqlite3 must be rebuilt on Linux (npm rebuild).

2026-09-11 — verae-keep on NS1

• New repo verae-keep: keep :3860 restarts crashed units; watch :3861 restarts keep; guard.sh restarts watch.
• Honors admin pause/stop via intent files + POST /intent. Fleet writes data/<instance>/intent.json (local and SSH).
• Tested on 70.88.205.138: crash-restart, pause-hold, watch respawn. webhook-deliver brought back. NATS observed only.

2026-09-11 — IAM JSON doors closed

• CS/sales/accounting/staff JSON list and sales PUT pricing now go through IAM (401 JSON, 302 HTML).
• Fleet service env sets STAFF_AUTH=1 with STAFF_IAM_URL.
• Re-walk: cs credit 200 agent=cs; cs export 403; operator Fleet POST 200; admin /me permissions:['*'].

2026-09-11 — IAM on lab, portal door, hardening

• Restarted fleet with STAFF_IAM_URL. Walk: cs credits (agent=cs) and is 403 on accounting export; operator fleet POST 200, cs 403; admin /me permissions *.
• Portal public door verae-access-web :3021/portal/ (static + /portal/api → loopback edge).
• IAM sessions persist in JSON; login rate-limit per IP+username (8/10min); credits stamp IAM username as authz principal.
• lan-134 remains disabled unless FLEET_ENABLE_LAN134=1. Receipt PDF branded (indigo header + legal footer).

2026-09-11 — staff IAM

• New verae-staff-iam :3028 — users, roles, permissions, sessions, audit UI.
• Admin login uses IAM when STAFF_IAM_URL is set; tabs hide without permission.
• CS/sales/accounting/access-staff/fleet mutating APIs check cs.credit, sales.quote, accounting.export, fleet.operate, etc.
• Seed: admin/cs/sales/accounting/operator. Shared staff key is fallback only.

2026-09-11 — last three UI leftovers

• Swagger /docs stays stock; banner names it OpenAPI explorer.
• Shared template verae-staff-ui for CS + access-staff.
• STAFF_COOKIE_DOMAIN + nginx example for multi-host.

2026-09-11 — names, staff session, exclusive jobs.events

• Account-balance stores display names (customer.put + lookup by name). Edge writes names on customer create/edit; CS/sales/accounting/staff join from edge if the ledger has no name.
• New repo verae-staff-session (:3027). Department HTML redirects when STAFF_AUTH=1.
• JOBS_EVENTS_EXCLUSIVE=1 makes verae-jobs-events the durable consumer; middleware webhook router backs off.
• Catalog index is cards. Disabled fleet machines are grey, not degraded yellow.

2026-09-11 — UI design-system pass

• Restyled CS/sales/accounting/access-staff to portal indigo; dollars + names.
• Fleet ⋯ menus, role chips, Docs cards, 0.0.0.0 copy, simulator indigo (orange = faults).
• Portal masked API key; admin customer drawer; empty invoice illustrations.
• Catalog system-ui. New independent repo UI-Docs (walkthrough, screenshots, UI-REVIEW.pdf).

Second pass (2–5)

• Identity used from portal when IDENTITY_URL set.
• Leaf loads allow-list from nats-accounts policy.json.
• Staff plane is a first-class mailbox (verae-access-staff).
• jobs.events mailbox process verae-jobs-events (passthrough + count).