155 lines
8.3 KiB
Markdown
155 lines
8.3 KiB
Markdown
# Action log
|
||
|
||
## 2026-09-12 — exhaustive optimal-config study
|
||
|
||
- 8c/16G ZFS factorial: file/memory × r=1/r=3, 4 KiB, reconnect, UDP echo, MQTT QoS0 on nats-a :1883 (then removed).
|
||
- JS memory r=1 4p **65k** pubs/s; file r=3 1p **14.3k**; MQTT QoS0 **45k**; UDP p99 **0.36 ms**.
|
||
- Combined report + DL360 Gen10 projection: `reports/optimal-config/REPORT.{md,html,pdf}`.
|
||
- Verdict: keep NATS+JetStream; no MQTT/UDP fabric; metal = 3 NVMe + 10GbE, expect JS r=3 ~40–80k.
|
||
|
||
## 2026-09-12 — maximized NS1 NATS study
|
||
|
||
- `pct set` 510–513 to 8 cores / 16 GiB; `max_mem: 8G`; tmpfs 8G on JetStream for the ladder only.
|
||
- Run `20260912T053120Z`. JS file 1p 128 B **7.4k → 17.4k (2.35×)**; JS memory 4p **36.4k**; ping p99 **1.38 → 0.68 ms**.
|
||
- tmpfs unmounted after; product streams on ZFS again. Resources and max_mem **left raised**.
|
||
- Report: `reports/ns1-maximize/`.
|
||
|
||
## 2026-09-12 — NS1-host NATS study (second run)
|
||
|
||
- All orchestration, nats bench, matplotlib, pandoc, and weasyprint ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`). Laptop not in the path.
|
||
- Script: `verae-nats-cluster/scripts/study-on-ns1.sh` (refuses unless hostname is NS1).
|
||
- Run `20260912T051237Z`. Report: `reports/ns1-host/nats-cluster-bench-ns1.{md,html,pdf}`.
|
||
- Host load ~8.5. JS 1p 128 B ~7.4k durable pubs/s; ping p99 1.38 ms. Methodology + fine-tuning in that report.
|
||
|
||
## 2026-09-12 — NATS cluster speed report
|
||
|
||
- Bench from LXC 510 vs nats-a/b/c (`20260912T045131Z`).
|
||
- Full write-up with charts in `reports/nats-cluster-bench.{md,html,pdf}`.
|
||
- JetStream r=3 file ~16k durable 128 B pubs/s; ping RTT ~0.3 ms; flood queueing 150–500 ms.
|
||
|
||
## 2026-09-12 — test env on the 3-node cluster
|
||
|
||
- Fleet, keep, middleware, billing, and access planes `NATS_URL` → `10.10.10.21–23:4222`.
|
||
- Streams ZAPIER_JOBS/EVENTS/WEBHOOKS/USAGE + VERAE_ARCHIVE replicas=3.
|
||
- px-worker keep worm/tree 200 on the cluster. Backup timer enabled. Uptime loop on px-worker.
|
||
- MOCK_VERAE still true. Zapier push still waiting on login. nkeys/mTLS not flipped (would require client creds).
|
||
|
||
## 2026-09-12 — NATS cluster on three Proxmox LXC
|
||
|
||
- New repo **verae-nats-cluster**: LXC 511/512/513 (`nats-a/b/c`, `10.10.10.21–23`) JetStream cluster on `vmbr1`.
|
||
- Lab loopback `127.0.0.1:4222` unchanged until cut-over.
|
||
- Test: stream replicas=3; same configs later move to distinct hardware.
|
||
|
||
## 2026-09-11 — proxmox worker + deploy/monitor/backup
|
||
|
||
- NS1 **is** Proxmox (`vmbr0` public, `vmbr1` `10.10.10.0/24`). New repo **verae-proxmox** creates LXC **510** `verae-px-worker` at `10.10.10.20`.
|
||
- NATS for guests: socat `10.10.10.1:4222` → `127.0.0.1:4222` (never vmbr0).
|
||
- **verae-uptime** off-box HTTPS GET; **verae-backup** tar snapshots; **verae-deploy** tagged npm ci.
|
||
- Fleet `overlays/ns1/` checked in; `start.sh` no longer rewrites fleet/machines JSON.
|
||
- User systemd + linger for keep and fleet.
|
||
|
||
## 2026-09-11 — bootstrap
|
||
|
||
- Created this repo to record decisions while executing cleanup items 1–5 (and a second pass on 2–5).
|
||
- Assumption: single developer, lab only; breaking changes allowed.
|
||
|
||
## Step 1 — account-balance is the prepaid source of truth
|
||
|
||
- Persist books to `BOOKS_PATH` (atomic JSON).
|
||
- Mutations go through `balance.adjust` / `usage.recorded`; edge cache `balanceCents` is updated from the books reply only.
|
||
- Local cache mutate only when `NATS_URL` and `ACCOUNT_BALANCE_URL` are both unset (tests).
|
||
|
||
## Step 2 — identity join
|
||
|
||
- `zappier-identity` mailbox: `verae.identity.bind` / `verae.identity.lookup`.
|
||
- Portal + middleware tenants store `veraeUserId`; JWT never public.
|
||
|
||
## Step 3 — access planes are the public doors
|
||
|
||
- zappier-edge binds loopback (`127.0.0.1:3000`).
|
||
- Public: access-web `:3021`, access-api `:3022`, access-leaf `:3023`, access-zapier `:3024`, access-staff `:3025`.
|
||
|
||
## Step 4 — leaf hardening
|
||
|
||
- `verae-nats-accounts` holds nats.conf + subject policy.
|
||
- Leaf token required; billing subjects denied in policy file and access-leaf.
|
||
|
||
## Step 5 — fleet spawn
|
||
|
||
- Prefer local machines for HTTP `spawn.command` services.
|
||
- Disable lan-134 unless `FLEET_ENABLE_LAN134=1`.
|
||
- SSH spawn timeout 8s; failed hosts skipped.
|
||
|
||
## 2026-09-11 — catalog live doors
|
||
|
||
- Bottom of https://zapier.georgelambert.org/ lists each HTTPS door with description, screenshot, and Open link.
|
||
- UI-REVIEW.pdf includes the live-door atlas. Walkthrough §13.
|
||
|
||
## 2026-09-11 — Caddy host names live
|
||
|
||
- Installed `/etc/caddy/sites/zapier-services.georgelambert.org.caddy` on NS1. HTTPS: portal, api, leaf, zap, staff, iam, login, cs, sales, acct, mw, fleet.
|
||
- Not proxied: NATS, keep, workers, loopback edge.
|
||
|
||
## 2026-09-11 — host names
|
||
|
||
- Mapped every running service to **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`).
|
||
- Suggested `*.zapier.georgelambert.org` names for public doors; only `zapier.georgelambert.org` (docs) and `git.georgelambert.org` exist in Caddy today.
|
||
|
||
## 2026-09-11 — docs sync, bootstrap, NS1 all-in-one
|
||
|
||
- Catalog and ops docs: public portal is access-web `:3021/portal/`; edge is loopback; IAM + keep listed.
|
||
- New `verae-bootstrap` repo: clone + npm for `ns1-all-in-one`, `control-plane`, `ns1-archive`, `lan-worker`.
|
||
- Fleet starts `staff-iam` and `staff-session`. NS1 all-in-one uses edge `:13000` because host `:3000` is taken; keep still owns archive workers. `better-sqlite3` must be rebuilt on Linux (`npm rebuild`).
|
||
|
||
## 2026-09-11 — verae-keep on NS1
|
||
|
||
- New repo `verae-keep`: keep `:3860` restarts crashed units; watch `:3861` restarts keep; `guard.sh` restarts watch.
|
||
- Honors admin pause/stop via intent files + `POST /intent`. Fleet writes `data/<instance>/intent.json` (local and SSH).
|
||
- Tested on 70.88.205.138: crash-restart, pause-hold, watch respawn. webhook-deliver brought back. NATS observed only.
|
||
|
||
## 2026-09-11 — IAM JSON doors closed
|
||
|
||
- CS/sales/accounting/staff JSON list and sales PUT pricing now go through IAM (401 JSON, 302 HTML).
|
||
- Fleet service env sets `STAFF_AUTH=1` with `STAFF_IAM_URL`.
|
||
- Re-walk: cs credit 200 agent=`cs`; cs export 403; operator Fleet POST 200; admin `/me` `permissions:['*']`.
|
||
|
||
## 2026-09-11 — IAM on lab, portal door, hardening
|
||
|
||
- Restarted fleet with `STAFF_IAM_URL`. Walk: cs credits (agent=`cs`) and is 403 on accounting export; operator fleet POST 200, cs 403; admin `/me` permissions `*`.
|
||
- Portal public door `verae-access-web` `:3021/portal/` (static + `/portal/api` → loopback edge).
|
||
- IAM sessions persist in JSON; login rate-limit per IP+username (8/10min); credits stamp IAM username as authz principal.
|
||
- lan-134 remains disabled unless `FLEET_ENABLE_LAN134=1`. Receipt PDF branded (indigo header + legal footer).
|
||
|
||
## 2026-09-11 — staff IAM
|
||
|
||
- New `verae-staff-iam` :3028 — users, roles, permissions, sessions, audit UI.
|
||
- Admin login uses IAM when `STAFF_IAM_URL` is set; tabs hide without permission.
|
||
- CS/sales/accounting/access-staff/fleet mutating APIs check `cs.credit`, `sales.quote`, `accounting.export`, `fleet.operate`, etc.
|
||
- Seed: admin/cs/sales/accounting/operator. Shared staff key is fallback only.
|
||
|
||
## 2026-09-11 — last three UI leftovers
|
||
|
||
- Swagger `/docs` stays stock; banner names it OpenAPI explorer.
|
||
- Shared template `verae-staff-ui` for CS + access-staff.
|
||
- `STAFF_COOKIE_DOMAIN` + nginx example for multi-host.
|
||
|
||
## 2026-09-11 — names, staff session, exclusive jobs.events
|
||
|
||
- Account-balance stores display names (`customer.put` + lookup by name). Edge writes names on customer create/edit; CS/sales/accounting/staff join from edge if the ledger has no name.
|
||
- New repo `verae-staff-session` (:3027). Department HTML redirects when `STAFF_AUTH=1`.
|
||
- `JOBS_EVENTS_EXCLUSIVE=1` makes `verae-jobs-events` the durable consumer; middleware webhook router backs off.
|
||
- Catalog index is cards. Disabled fleet machines are grey, not degraded yellow.
|
||
|
||
## 2026-09-11 — UI design-system pass
|
||
|
||
- Restyled CS/sales/accounting/access-staff to portal indigo; dollars + names.
|
||
- Fleet ⋯ menus, role chips, Docs cards, 0.0.0.0 copy, simulator indigo (orange = faults).
|
||
- Portal masked API key; admin customer drawer; empty invoice illustrations.
|
||
- Catalog system-ui. New independent repo **UI-Docs** (walkthrough, screenshots, UI-REVIEW.pdf).
|
||
|
||
## Second pass (2–5)
|
||
|
||
- Identity used from portal when `IDENTITY_URL` set.
|
||
- Leaf loads allow-list from nats-accounts `policy.json`.
|
||
- Staff plane is a first-class mailbox (`verae-access-staff`).
|
||
- jobs.events mailbox process `verae-jobs-events` (passthrough + count).
|