Current lab status 2026-09-12

2026-09-12 00:06:24 -04:00 · 2026-09-12 00:06:24 -04:00 · 870550646d
commit 870550646d
11 changed files with 308 additions and 0 deletions
--- a/LOG.md
+++ b/LOG.md
@ -0,0 +1,128 @@
+# Action log
+
+## 2026-09-12 — test env on the 3-node cluster
+
+- Fleet, keep, middleware, billing, and access planes `NATS_URL` → `10.10.10.21–23:4222`.
+- Streams ZAPIER_JOBS/EVENTS/WEBHOOKS/USAGE + VERAE_ARCHIVE replicas=3.
+- px-worker keep worm/tree 200 on the cluster. Backup timer enabled. Uptime loop on px-worker.
+- MOCK_VERAE still true. Zapier push still waiting on login. nkeys/mTLS not flipped (would require client creds).
+
+## 2026-09-12 — NATS cluster on three Proxmox LXC
+
+- New repo **verae-nats-cluster**: LXC 511/512/513 (`nats-a/b/c`, `10.10.10.21–23`) JetStream cluster on `vmbr1`.
+- Lab loopback `127.0.0.1:4222` unchanged until cut-over.
+- Test: stream replicas=3; same configs later move to distinct hardware.
+
+## 2026-09-11 — proxmox worker + deploy/monitor/backup
+
+- NS1 **is** Proxmox (`vmbr0` public, `vmbr1` `10.10.10.0/24`). New repo **verae-proxmox** creates LXC **510** `verae-px-worker` at `10.10.10.20`.
+- NATS for guests: socat `10.10.10.1:4222` → `127.0.0.1:4222` (never vmbr0).
+- **verae-uptime** off-box HTTPS GET; **verae-backup** tar snapshots; **verae-deploy** tagged npm ci.
+- Fleet `overlays/ns1/` checked in; `start.sh` no longer rewrites fleet/machines JSON.
+- User systemd + linger for keep and fleet.
+
+## 2026-09-11 — bootstrap
+
+- Created this repo to record decisions while executing cleanup items 1–5 (and a second pass on 2–5).
+- Assumption: single developer, lab only; breaking changes allowed.
+
+## Step 1 — account-balance is the prepaid source of truth
+
+- Persist books to `BOOKS_PATH` (atomic JSON).
+- Mutations go through `balance.adjust` / `usage.recorded`; edge cache `balanceCents` is updated from the books reply only.
+- Local cache mutate only when `NATS_URL` and `ACCOUNT_BALANCE_URL` are both unset (tests).
+
+## Step 2 — identity join
+
+- `zappier-identity` mailbox: `verae.identity.bind` / `verae.identity.lookup`.
+- Portal + middleware tenants store `veraeUserId`; JWT never public.
+
+## Step 3 — access planes are the public doors
+
+- zappier-edge binds loopback (`127.0.0.1:3000`).
+- Public: access-web `:3021`, access-api `:3022`, access-leaf `:3023`, access-zapier `:3024`, access-staff `:3025`.
+
+## Step 4 — leaf hardening
+
+- `verae-nats-accounts` holds nats.conf + subject policy.
+- Leaf token required; billing subjects denied in policy file and access-leaf.
+
+## Step 5 — fleet spawn
+
+- Prefer local machines for HTTP `spawn.command` services.
+- Disable lan-134 unless `FLEET_ENABLE_LAN134=1`.
+- SSH spawn timeout 8s; failed hosts skipped.
+
+## 2026-09-11 — catalog live doors
+
+- Bottom of https://zapier.georgelambert.org/ lists each HTTPS door with description, screenshot, and Open link.
+- UI-REVIEW.pdf includes the live-door atlas. Walkthrough §13.
+
+## 2026-09-11 — Caddy host names live
+
+- Installed `/etc/caddy/sites/zapier-services.georgelambert.org.caddy` on NS1. HTTPS: portal, api, leaf, zap, staff, iam, login, cs, sales, acct, mw, fleet.
+- Not proxied: NATS, keep, workers, loopback edge.
+
+## 2026-09-11 — host names
+
+- Mapped every running service to **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`).
+- Suggested `*.zapier.georgelambert.org` names for public doors; only `zapier.georgelambert.org` (docs) and `git.georgelambert.org` exist in Caddy today.
+
+## 2026-09-11 — docs sync, bootstrap, NS1 all-in-one
+
+- Catalog and ops docs: public portal is access-web `:3021/portal/`; edge is loopback; IAM + keep listed.
+- New `verae-bootstrap` repo: clone + npm for `ns1-all-in-one`, `control-plane`, `ns1-archive`, `lan-worker`.
+- Fleet starts `staff-iam` and `staff-session`. NS1 all-in-one uses edge `:13000` because host `:3000` is taken; keep still owns archive workers. `better-sqlite3` must be rebuilt on Linux (`npm rebuild`).
+
+## 2026-09-11 — verae-keep on NS1
+
+- New repo `verae-keep`: keep `:3860` restarts crashed units; watch `:3861` restarts keep; `guard.sh` restarts watch.
+- Honors admin pause/stop via intent files + `POST /intent`. Fleet writes `data/<instance>/intent.json` (local and SSH).
+- Tested on 70.88.205.138: crash-restart, pause-hold, watch respawn. webhook-deliver brought back. NATS observed only.
+
+## 2026-09-11 — IAM JSON doors closed
+
+- CS/sales/accounting/staff JSON list and sales PUT pricing now go through IAM (401 JSON, 302 HTML).
+- Fleet service env sets `STAFF_AUTH=1` with `STAFF_IAM_URL`.
+- Re-walk: cs credit 200 agent=`cs`; cs export 403; operator Fleet POST 200; admin `/me` `permissions:['*']`.
+
+## 2026-09-11 — IAM on lab, portal door, hardening
+
+- Restarted fleet with `STAFF_IAM_URL`. Walk: cs credits (agent=`cs`) and is 403 on accounting export; operator fleet POST 200, cs 403; admin `/me` permissions `*`.
+- Portal public door `verae-access-web` `:3021/portal/` (static + `/portal/api` → loopback edge).
+- IAM sessions persist in JSON; login rate-limit per IP+username (8/10min); credits stamp IAM username as authz principal.
+- lan-134 remains disabled unless `FLEET_ENABLE_LAN134=1`. Receipt PDF branded (indigo header + legal footer).
+
+## 2026-09-11 — staff IAM
+
+- New `verae-staff-iam` :3028 — users, roles, permissions, sessions, audit UI.
+- Admin login uses IAM when `STAFF_IAM_URL` is set; tabs hide without permission.
+- CS/sales/accounting/access-staff/fleet mutating APIs check `cs.credit`, `sales.quote`, `accounting.export`, `fleet.operate`, etc.
+- Seed: admin/cs/sales/accounting/operator. Shared staff key is fallback only.
+
+## 2026-09-11 — last three UI leftovers
+
+- Swagger `/docs` stays stock; banner names it OpenAPI explorer.
+- Shared template `verae-staff-ui` for CS + access-staff.
+- `STAFF_COOKIE_DOMAIN` + nginx example for multi-host.
+
+## 2026-09-11 — names, staff session, exclusive jobs.events
+
+- Account-balance stores display names (`customer.put` + lookup by name). Edge writes names on customer create/edit; CS/sales/accounting/staff join from edge if the ledger has no name.
+- New repo `verae-staff-session` (:3027). Department HTML redirects when `STAFF_AUTH=1`.
+- `JOBS_EVENTS_EXCLUSIVE=1` makes `verae-jobs-events` the durable consumer; middleware webhook router backs off.
+- Catalog index is cards. Disabled fleet machines are grey, not degraded yellow.
+
+## 2026-09-11 — UI design-system pass
+
+- Restyled CS/sales/accounting/access-staff to portal indigo; dollars + names.
+- Fleet ⋯ menus, role chips, Docs cards, 0.0.0.0 copy, simulator indigo (orange = faults).
+- Portal masked API key; admin customer drawer; empty invoice illustrations.
+- Catalog system-ui. New independent repo **UI-Docs** (walkthrough, screenshots, UI-REVIEW.pdf).
+
+## Second pass (2–5)
+
+- Identity used from portal when `IDENTITY_URL` set.
+- Leaf loads allow-list from nats-accounts `policy.json`.
+- Staff plane is a first-class mailbox (`verae-access-staff`).
+- jobs.events mailbox process `verae-jobs-events` (passthrough + count).
--- a/NATS.md
+++ b/NATS.md
@ -0,0 +1,3 @@
+# NATS — zapier-decisions
+
+None. Documentation repo.
--- a/README.md
+++ b/README.md
@ -0,0 +1,14 @@
+# zapier-decisions
+
+Lab log of architecture decisions, actions taken, and open todos while cleaning the Verae × Zapier stack.
+
+**Forgejo:** https://git.georgelambert.org/marchon/zapier-decisions
+
+| File | What |
+|------|------|
+| [STATUS.md](STATUS.md) | **Current status** (2026-09-12) |
+| [LOG.md](LOG.md) | Chronological actions |
+| [TODO.md](TODO.md) | Open items |
+| [decisions/](decisions/) | One file per decision |
+
+No runtime. Not a NATS mailbox.
--- a/STATUS.md
+++ b/STATUS.md
@ -0,0 +1,112 @@
+# Current status — 2026-09-12
+
+Verae Time × Zapier is a **working lab** on one Proxmox host (**NS1.GEORGELAMBERT.ORG**, `70.88.205.138`). Public HTTPS doors, catalog, keep/fleet, and a **test 3-node NATS cluster** on separate LXC guests are up. The test environment **points at that cluster**. Zapier Platform `push` and live `api.veraetime.net` are still waiting on operator login / bind credentials.
+
+**Progress repo:** this file in [zapier-decisions](https://git.georgelambert.org/marchon/zapier-decisions).  
+**Catalog:** https://zapier.georgelambert.org/packages/zapier-decisions/STATUS.pdf  
+**Monorepo:** `master-zapier-plan-draft` (`main` / `master`).
+
+---
+
+## Live right now
+
+| Layer | State |
+|-------|--------|
+| Catalog | https://zapier.georgelambert.org/ (PDF links path-relative; module/model books work) |
+| Public doors | portal, api, zap, iam, mw, fleet, git — last off-box GET 8/8 OK |
+| Keep + fleet | Running. systemd + linger **enabled** for next reboot (current processes were not double-started as systemd) |
+| Test NATS | **3-node JetStream on `vmbr1`:** LXC 511/512/513 `nats-a/b/c` at `10.10.10.21–23`. Cluster name `verae`, 2 routes each. |
+| Product streams | `ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE` — **replicas=3** |
+| Modules on that cluster | Fleet, keep, middleware (`natsConnected: true`), billing, access planes. `NATS_URL` lists all three URLs. |
+| Host loopback `:4222` | Still listening; **clients no longer use it** |
+| Worker CT **510** `verae-px-worker` | `10.10.10.20` on `vmbr1`. Keep worm+tree **200** on the cluster |
+| Backup | Timer `verae-backup.timer` enabled; snapshots under `/SSD2/backups/verae/` |
+| Uptime | Loop on px-worker (`:3870/status`), last tick `failed: 0` |
+
+Client URL (private `vmbr1` only):
+
+```text
+nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+```
+
+Repo: https://git.georgelambert.org/marchon/verae-nats-cluster
+
+---
+
+## Built and checked in (Forgejo, each its own repo)
+
+IAM, portal on access-web, keep/watch/guard, fleet floors, host-deps, catalog host-move docs, plus:
+
+| Repo | Job |
+|------|-----|
+| verae-uptime | Off-box HTTPS GET of public doors |
+| verae-backup | SQLite / keep / worm-tree tar; restore dry-run default |
+| verae-deploy | Tagged checkout + npm ci on the target OS |
+| verae-proxmox | LXC 510 worker provision |
+| verae-nats-cluster | LXC 511–513 JetStream cluster |
+
+---
+
+## Still open
+
+| Item | Notes |
+|------|--------|
+| Zapier Platform `push` | Needs Zapier login (see below) |
+| `MOCK_VERAE=false` | Live `api.veraetime.net` + admin bind credentials (see below) |
+| NATS nkeys/mTLS | Cluster is up **without** accounts so existing `NATS_URL`s work. `verae-nats-accounts` remains the sketch. Flip only after every client has credentials. Isolation today is `vmbr1`, not nkeys. |
+| lan-134 | Still disabled (SSH not working) |
+| Reboot test | systemd should own keep/fleet after reboot; do not also `nohup guard.sh` |
+| Hardware NATS | Same three configs, three boxes, private NIC — change IPs in `cluster.env` |
+
+---
+
+## Once we have a Zapier login
+
+Zapier cloud only talks **HTTPS** to our Zap plane (`https://zap.zapier.georgelambert.org`). No NATS, no `api.veraetime.net`.
+
+1. `zapier login` (or `zapier-platform login`).
+2. In `packages/verae-zapier` (and `verae-activate` if the tiny app is still wanted):
+   - `zapier-platform register` (or attach an existing app id).
+   - `MIDDLEWARE_BASE_URL=https://zap.zapier.georgelambert.org` (or `https://mw.zapier.georgelambert.org` if skipping the Zap plane).
+   - `zapier-platform build` and `validate`.
+3. `zapier-platform push` as a **private** app.
+4. Invite the workspace; create a Zap:
+   - Auth = portal **API key** (`zmw_…`), not a Verae JWT.
+   - Test **Create Timestamp and Wait**, **Create Timestamp** (async `jobId`), **Find Job Status**, **Timestamp Completed** REST Hook.
+5. Fix anything `validate` / `push` flags (auth labels, samples, 402/403 mapping).
+6. When the contract is stable, bump the platform version and invite a real user.
+
+Do **not** put NATS URLs or Verae passwords into Zapier.
+
+---
+
+## Once `MOCK_VERAE=false` and live `api.veraetime.net`
+
+Middleware already has the live HTTP client. Still needed:
+
+1. `VERAE_API_BASE_URL=https://api.veraetime.net` and working **login** (username/password → JWT). JWT stays in middleware (`tokenRef`); Zapier never sees it.
+2. Confirm OpenAPI vs our client:
+   - `POST /api/timestamp` → **202** `{ jobId }`
+   - `GET /api/status/{jobId}` until `completed` / `failed`
+   - `POST /api/verify`
+   - Batch create/status if used
+3. **Hash lookup is not on the live OpenAPI** — `veraeClient.lookupHash` returns **501** when not mocking. Central-chain “find by SHA-256” needs a Verae API gap fill or our own store.
+4. Map live status JSON into the receipt shape the Zap app expects (`certificate`, `timestamp`, `blockIndex`, `sha256`).
+5. Token refresh, 401 retry, and 402/403 mapping against real errors.
+6. One live seal in the lab, then REST Hook delivery to `hooks.zapier.com`.
+
+---
+
+## Submit → jobId → wait → receipt
+
+**Yes — that is the designed path**, and it already runs in **mock**.
+
+1. Zap (or curl) `POST /zapier/v1/timestamp` or `/timestamp/wait`.
+2. Middleware `createTimestamp` calls Verae `POST /api/timestamp` (or the mock) and gets **`jobId`**.
+3. It publishes **`verae.zapier.jobs.watch`** on NATS (the 3-node cluster).
+4. **Wait:** subscribe to **`verae.zapier.jobs.events`** for that `jobId`, or poll `GET /api/status/{jobId}`.
+5. Job-poller hits Verae status until `completed`.
+6. Event `timestamp.completed` carries status; webhook-deliver POSTs the Zapier REST Hook.
+7. Response includes **certificate / timestamp / blockIndex** (mock uses `mock-cert-{jobId}`, `blockIndex: 42`). Async create returns `{ jobId, sha256, existing }` immediately; wait returns the finished status. Same SHA-256 is idempotent (`existing: true`).
+
+If wait times out, middleware returns **`pending` + `jobId`** so the hook can finish the Zap. Live Verae must match that HTTP contract; we have **not** proven a real chain seal yet (`MOCK_VERAE` is still true).
--- a/TODO.md
+++ b/TODO.md
@ -0,0 +1,15 @@
+# Open todos
+
+- [ ] Live `api.veraetime.net` with `MOCK_VERAE=false` and admin bind credentials.
+- [x] Three-node NATS JetStream cluster on distinct Proxmox LXC (`verae-nats-cluster`; test env `NATS_URL` cut over; streams replicas=3).
+- [ ] NATS nkeys/mTLS on that cluster (accounts file is still the lab stand-in; not flipped so clients keep working on `vmbr1`).
+- [x] Exclusive JetStream consumer for `verae.zapier.jobs.events` on `verae-jobs-events` (`JOBS_EVENTS_EXCLUSIVE=1`; middleware skips the router).
+- [x] Auth on CS/sales/accounting HTML via `verae-staff-session` (`STAFF_AUTH=1`).
+- [x] Staff IAM: named users, roles, permissions (`verae-staff-iam` :3028).
+- [ ] Zapier Platform `push` of a private app.
+- [x] Move portal static files fully into `verae-access-web` (`/portal/` public door; API proxied to loopback edge).
+- [x] Turn IAM on for the running lab (fleet `STAFF_IAM_URL` + restart).
+- [x] IAM hardening: JSON sessions, login rate-limit, JSON 401 as well as HTML 302, credit principal = IAM username.
+- [x] Host keep-alive (`verae-keep`) on NS1 with watch+guard; honors admin pause/stop.
+- [x] Docs synced to IAM, public portal `:3021`, keep, bootstrap server types.
+- [x] `verae-bootstrap` clones all module repos and installs deps per server type.
--- a/decisions/D001-prepaid-sot.md
+++ b/decisions/D001-prepaid-sot.md
@ -0,0 +1,6 @@
+# D001 — Account-balance is the only prepaid writer
+
+**Status:** accepted  
+**Step:** 1
+
+Prepaid cents, credits, usage, and payments persist in `zappier-account-balance` (`books.json`). zappier-edge `balanceCents` is a cache filled from the books reply. Tests without NATS/HTTP books still mutate the cache so unit tests stay hermetic.
--- a/decisions/D002-identity.md
+++ b/decisions/D002-identity.md
@ -0,0 +1,6 @@
+# D002 — veraeUserId is the join key, not the JWT
+
+**Status:** accepted  
+**Step:** 2
+
+Public token remains the zappier API key. Chain JWT stays server-side (`tokenRef`). `zappier-identity` is the mailbox for bind/lookup so portal and middleware share one map.
--- a/decisions/D003-public-doors.md
+++ b/decisions/D003-public-doors.md
@ -0,0 +1,6 @@
+# D003 — Access planes are the only public HTTP doors
+
+**Status:** accepted  
+**Step:** 3
+
+Zapier → `:3024`, browser → `:3021`, customer API → `:3022`, leaf → `:3023`, staff → `:3025`. zappier-edge listens on `127.0.0.1:3000` for identity, Stripe, admin, and as upstream for those planes.
--- a/decisions/D004-leaf-accounts.md
+++ b/decisions/D004-leaf-accounts.md
@ -0,0 +1,6 @@
+# D004 — Leaf cannot touch billing, even if authz is skipped
+
+**Status:** accepted  
+**Step:** 4
+
+`verae-nats-accounts/policy.json` is the allow-list. access-leaf refuses any target not in the leaf set. nats.conf documents INTERNAL vs LEAF accounts for when the lab cluster is replaced with a real accounts-enabled nats-server.
--- a/decisions/D005-fleet-local-first.md
+++ b/decisions/D005-fleet-local-first.md
@ -0,0 +1,6 @@
+# D005 — HTTP services spawn locally; lan-134 stays off
+
+**Status:** accepted  
+**Step:** 5
+
+SSH to `70.88.205.134` blocked keepFloor. HTTP apps (`spawn.command`) prefer `kind=local`. `lan-134` is disabled unless `FLEET_ENABLE_LAN134=1`. Failed SSH hosts are skipped after 8s.
--- a/package.json
+++ b/package.json
@ -0,0 +1,6 @@
+{
+  "name": "zapier-decisions",
+  "version": "0.1.0",
+  "private": true,
+  "description": "Architecture decisions and action log for the Verae × Zapier stack"
+}