commit 870550646d2692206850680ce5d1d87dbe9ed4eb Author: George Lambert Date: Sat Sep 12 00:06:24 2026 -0400 Current lab status 2026-09-12 diff --git a/LOG.md b/LOG.md new file mode 100644 index 0000000..7405342 --- /dev/null +++ b/LOG.md @@ -0,0 +1,128 @@ +# Action log + +## 2026-09-12 — test env on the 3-node cluster + +- Fleet, keep, middleware, billing, and access planes `NATS_URL` → `10.10.10.21–23:4222`. +- Streams ZAPIER_JOBS/EVENTS/WEBHOOKS/USAGE + VERAE_ARCHIVE replicas=3. +- px-worker keep worm/tree 200 on the cluster. Backup timer enabled. Uptime loop on px-worker. +- MOCK_VERAE still true. Zapier push still waiting on login. nkeys/mTLS not flipped (would require client creds). + +## 2026-09-12 — NATS cluster on three Proxmox LXC + +- New repo **verae-nats-cluster**: LXC 511/512/513 (`nats-a/b/c`, `10.10.10.21–23`) JetStream cluster on `vmbr1`. +- Lab loopback `127.0.0.1:4222` unchanged until cut-over. +- Test: stream replicas=3; same configs later move to distinct hardware. + +## 2026-09-11 — proxmox worker + deploy/monitor/backup + +- NS1 **is** Proxmox (`vmbr0` public, `vmbr1` `10.10.10.0/24`). New repo **verae-proxmox** creates LXC **510** `verae-px-worker` at `10.10.10.20`. +- NATS for guests: socat `10.10.10.1:4222` → `127.0.0.1:4222` (never vmbr0). +- **verae-uptime** off-box HTTPS GET; **verae-backup** tar snapshots; **verae-deploy** tagged npm ci. +- Fleet `overlays/ns1/` checked in; `start.sh` no longer rewrites fleet/machines JSON. +- User systemd + linger for keep and fleet. + +## 2026-09-11 — bootstrap + +- Created this repo to record decisions while executing cleanup items 1–5 (and a second pass on 2–5). +- Assumption: single developer, lab only; breaking changes allowed. + +## Step 1 — account-balance is the prepaid source of truth + +- Persist books to `BOOKS_PATH` (atomic JSON). +- Mutations go through `balance.adjust` / `usage.recorded`; edge cache `balanceCents` is updated from the books reply only. +- Local cache mutate only when `NATS_URL` and `ACCOUNT_BALANCE_URL` are both unset (tests). + +## Step 2 — identity join + +- `zappier-identity` mailbox: `verae.identity.bind` / `verae.identity.lookup`. +- Portal + middleware tenants store `veraeUserId`; JWT never public. + +## Step 3 — access planes are the public doors + +- zappier-edge binds loopback (`127.0.0.1:3000`). +- Public: access-web `:3021`, access-api `:3022`, access-leaf `:3023`, access-zapier `:3024`, access-staff `:3025`. + +## Step 4 — leaf hardening + +- `verae-nats-accounts` holds nats.conf + subject policy. +- Leaf token required; billing subjects denied in policy file and access-leaf. + +## Step 5 — fleet spawn + +- Prefer local machines for HTTP `spawn.command` services. +- Disable lan-134 unless `FLEET_ENABLE_LAN134=1`. +- SSH spawn timeout 8s; failed hosts skipped. + +## 2026-09-11 — catalog live doors + +- Bottom of https://zapier.georgelambert.org/ lists each HTTPS door with description, screenshot, and Open link. +- UI-REVIEW.pdf includes the live-door atlas. Walkthrough §13. + +## 2026-09-11 — Caddy host names live + +- Installed `/etc/caddy/sites/zapier-services.georgelambert.org.caddy` on NS1. HTTPS: portal, api, leaf, zap, staff, iam, login, cs, sales, acct, mw, fleet. +- Not proxied: NATS, keep, workers, loopback edge. + +## 2026-09-11 — host names + +- Mapped every running service to **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`). +- Suggested `*.zapier.georgelambert.org` names for public doors; only `zapier.georgelambert.org` (docs) and `git.georgelambert.org` exist in Caddy today. + +## 2026-09-11 — docs sync, bootstrap, NS1 all-in-one + +- Catalog and ops docs: public portal is access-web `:3021/portal/`; edge is loopback; IAM + keep listed. +- New `verae-bootstrap` repo: clone + npm for `ns1-all-in-one`, `control-plane`, `ns1-archive`, `lan-worker`. +- Fleet starts `staff-iam` and `staff-session`. NS1 all-in-one uses edge `:13000` because host `:3000` is taken; keep still owns archive workers. `better-sqlite3` must be rebuilt on Linux (`npm rebuild`). + +## 2026-09-11 — verae-keep on NS1 + +- New repo `verae-keep`: keep `:3860` restarts crashed units; watch `:3861` restarts keep; `guard.sh` restarts watch. +- Honors admin pause/stop via intent files + `POST /intent`. Fleet writes `data//intent.json` (local and SSH). +- Tested on 70.88.205.138: crash-restart, pause-hold, watch respawn. webhook-deliver brought back. NATS observed only. + +## 2026-09-11 — IAM JSON doors closed + +- CS/sales/accounting/staff JSON list and sales PUT pricing now go through IAM (401 JSON, 302 HTML). +- Fleet service env sets `STAFF_AUTH=1` with `STAFF_IAM_URL`. +- Re-walk: cs credit 200 agent=`cs`; cs export 403; operator Fleet POST 200; admin `/me` `permissions:['*']`. + +## 2026-09-11 — IAM on lab, portal door, hardening + +- Restarted fleet with `STAFF_IAM_URL`. Walk: cs credits (agent=`cs`) and is 403 on accounting export; operator fleet POST 200, cs 403; admin `/me` permissions `*`. +- Portal public door `verae-access-web` `:3021/portal/` (static + `/portal/api` → loopback edge). +- IAM sessions persist in JSON; login rate-limit per IP+username (8/10min); credits stamp IAM username as authz principal. +- lan-134 remains disabled unless `FLEET_ENABLE_LAN134=1`. Receipt PDF branded (indigo header + legal footer). + +## 2026-09-11 — staff IAM + +- New `verae-staff-iam` :3028 — users, roles, permissions, sessions, audit UI. +- Admin login uses IAM when `STAFF_IAM_URL` is set; tabs hide without permission. +- CS/sales/accounting/access-staff/fleet mutating APIs check `cs.credit`, `sales.quote`, `accounting.export`, `fleet.operate`, etc. +- Seed: admin/cs/sales/accounting/operator. Shared staff key is fallback only. + +## 2026-09-11 — last three UI leftovers + +- Swagger `/docs` stays stock; banner names it OpenAPI explorer. +- Shared template `verae-staff-ui` for CS + access-staff. +- `STAFF_COOKIE_DOMAIN` + nginx example for multi-host. + +## 2026-09-11 — names, staff session, exclusive jobs.events + +- Account-balance stores display names (`customer.put` + lookup by name). Edge writes names on customer create/edit; CS/sales/accounting/staff join from edge if the ledger has no name. +- New repo `verae-staff-session` (:3027). Department HTML redirects when `STAFF_AUTH=1`. +- `JOBS_EVENTS_EXCLUSIVE=1` makes `verae-jobs-events` the durable consumer; middleware webhook router backs off. +- Catalog index is cards. Disabled fleet machines are grey, not degraded yellow. + +## 2026-09-11 — UI design-system pass + +- Restyled CS/sales/accounting/access-staff to portal indigo; dollars + names. +- Fleet ⋯ menus, role chips, Docs cards, 0.0.0.0 copy, simulator indigo (orange = faults). +- Portal masked API key; admin customer drawer; empty invoice illustrations. +- Catalog system-ui. New independent repo **UI-Docs** (walkthrough, screenshots, UI-REVIEW.pdf). + +## Second pass (2–5) + +- Identity used from portal when `IDENTITY_URL` set. +- Leaf loads allow-list from nats-accounts `policy.json`. +- Staff plane is a first-class mailbox (`verae-access-staff`). +- jobs.events mailbox process `verae-jobs-events` (passthrough + count). diff --git a/NATS.md b/NATS.md new file mode 100644 index 0000000..782a476 --- /dev/null +++ b/NATS.md @@ -0,0 +1,3 @@ +# NATS — zapier-decisions + +None. Documentation repo. diff --git a/README.md b/README.md new file mode 100644 index 0000000..febadd9 --- /dev/null +++ b/README.md @@ -0,0 +1,14 @@ +# zapier-decisions + +Lab log of architecture decisions, actions taken, and open todos while cleaning the Verae × Zapier stack. + +**Forgejo:** https://git.georgelambert.org/marchon/zapier-decisions + +| File | What | +|------|------| +| [STATUS.md](STATUS.md) | **Current status** (2026-09-12) | +| [LOG.md](LOG.md) | Chronological actions | +| [TODO.md](TODO.md) | Open items | +| [decisions/](decisions/) | One file per decision | + +No runtime. Not a NATS mailbox. diff --git a/STATUS.md b/STATUS.md new file mode 100644 index 0000000..c0e6308 --- /dev/null +++ b/STATUS.md @@ -0,0 +1,112 @@ +# Current status — 2026-09-12 + +Verae Time × Zapier is a **working lab** on one Proxmox host (**NS1.GEORGELAMBERT.ORG**, `70.88.205.138`). Public HTTPS doors, catalog, keep/fleet, and a **test 3-node NATS cluster** on separate LXC guests are up. The test environment **points at that cluster**. Zapier Platform `push` and live `api.veraetime.net` are still waiting on operator login / bind credentials. + +**Progress repo:** this file in [zapier-decisions](https://git.georgelambert.org/marchon/zapier-decisions). +**Catalog:** https://zapier.georgelambert.org/packages/zapier-decisions/STATUS.pdf +**Monorepo:** `master-zapier-plan-draft` (`main` / `master`). + +--- + +## Live right now + +| Layer | State | +|-------|--------| +| Catalog | https://zapier.georgelambert.org/ (PDF links path-relative; module/model books work) | +| Public doors | portal, api, zap, iam, mw, fleet, git — last off-box GET 8/8 OK | +| Keep + fleet | Running. systemd + linger **enabled** for next reboot (current processes were not double-started as systemd) | +| Test NATS | **3-node JetStream on `vmbr1`:** LXC 511/512/513 `nats-a/b/c` at `10.10.10.21–23`. Cluster name `verae`, 2 routes each. | +| Product streams | `ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE` — **replicas=3** | +| Modules on that cluster | Fleet, keep, middleware (`natsConnected: true`), billing, access planes. `NATS_URL` lists all three URLs. | +| Host loopback `:4222` | Still listening; **clients no longer use it** | +| Worker CT **510** `verae-px-worker` | `10.10.10.20` on `vmbr1`. Keep worm+tree **200** on the cluster | +| Backup | Timer `verae-backup.timer` enabled; snapshots under `/SSD2/backups/verae/` | +| Uptime | Loop on px-worker (`:3870/status`), last tick `failed: 0` | + +Client URL (private `vmbr1` only): + +```text +nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222 +``` + +Repo: https://git.georgelambert.org/marchon/verae-nats-cluster + +--- + +## Built and checked in (Forgejo, each its own repo) + +IAM, portal on access-web, keep/watch/guard, fleet floors, host-deps, catalog host-move docs, plus: + +| Repo | Job | +|------|-----| +| verae-uptime | Off-box HTTPS GET of public doors | +| verae-backup | SQLite / keep / worm-tree tar; restore dry-run default | +| verae-deploy | Tagged checkout + npm ci on the target OS | +| verae-proxmox | LXC 510 worker provision | +| verae-nats-cluster | LXC 511–513 JetStream cluster | + +--- + +## Still open + +| Item | Notes | +|------|--------| +| Zapier Platform `push` | Needs Zapier login (see below) | +| `MOCK_VERAE=false` | Live `api.veraetime.net` + admin bind credentials (see below) | +| NATS nkeys/mTLS | Cluster is up **without** accounts so existing `NATS_URL`s work. `verae-nats-accounts` remains the sketch. Flip only after every client has credentials. Isolation today is `vmbr1`, not nkeys. | +| lan-134 | Still disabled (SSH not working) | +| Reboot test | systemd should own keep/fleet after reboot; do not also `nohup guard.sh` | +| Hardware NATS | Same three configs, three boxes, private NIC — change IPs in `cluster.env` | + +--- + +## Once we have a Zapier login + +Zapier cloud only talks **HTTPS** to our Zap plane (`https://zap.zapier.georgelambert.org`). No NATS, no `api.veraetime.net`. + +1. `zapier login` (or `zapier-platform login`). +2. In `packages/verae-zapier` (and `verae-activate` if the tiny app is still wanted): + - `zapier-platform register` (or attach an existing app id). + - `MIDDLEWARE_BASE_URL=https://zap.zapier.georgelambert.org` (or `https://mw.zapier.georgelambert.org` if skipping the Zap plane). + - `zapier-platform build` and `validate`. +3. `zapier-platform push` as a **private** app. +4. Invite the workspace; create a Zap: + - Auth = portal **API key** (`zmw_…`), not a Verae JWT. + - Test **Create Timestamp and Wait**, **Create Timestamp** (async `jobId`), **Find Job Status**, **Timestamp Completed** REST Hook. +5. Fix anything `validate` / `push` flags (auth labels, samples, 402/403 mapping). +6. When the contract is stable, bump the platform version and invite a real user. + +Do **not** put NATS URLs or Verae passwords into Zapier. + +--- + +## Once `MOCK_VERAE=false` and live `api.veraetime.net` + +Middleware already has the live HTTP client. Still needed: + +1. `VERAE_API_BASE_URL=https://api.veraetime.net` and working **login** (username/password → JWT). JWT stays in middleware (`tokenRef`); Zapier never sees it. +2. Confirm OpenAPI vs our client: + - `POST /api/timestamp` → **202** `{ jobId }` + - `GET /api/status/{jobId}` until `completed` / `failed` + - `POST /api/verify` + - Batch create/status if used +3. **Hash lookup is not on the live OpenAPI** — `veraeClient.lookupHash` returns **501** when not mocking. Central-chain “find by SHA-256” needs a Verae API gap fill or our own store. +4. Map live status JSON into the receipt shape the Zap app expects (`certificate`, `timestamp`, `blockIndex`, `sha256`). +5. Token refresh, 401 retry, and 402/403 mapping against real errors. +6. One live seal in the lab, then REST Hook delivery to `hooks.zapier.com`. + +--- + +## Submit → jobId → wait → receipt + +**Yes — that is the designed path**, and it already runs in **mock**. + +1. Zap (or curl) `POST /zapier/v1/timestamp` or `/timestamp/wait`. +2. Middleware `createTimestamp` calls Verae `POST /api/timestamp` (or the mock) and gets **`jobId`**. +3. It publishes **`verae.zapier.jobs.watch`** on NATS (the 3-node cluster). +4. **Wait:** subscribe to **`verae.zapier.jobs.events`** for that `jobId`, or poll `GET /api/status/{jobId}`. +5. Job-poller hits Verae status until `completed`. +6. Event `timestamp.completed` carries status; webhook-deliver POSTs the Zapier REST Hook. +7. Response includes **certificate / timestamp / blockIndex** (mock uses `mock-cert-{jobId}`, `blockIndex: 42`). Async create returns `{ jobId, sha256, existing }` immediately; wait returns the finished status. Same SHA-256 is idempotent (`existing: true`). + +If wait times out, middleware returns **`pending` + `jobId`** so the hook can finish the Zap. Live Verae must match that HTTP contract; we have **not** proven a real chain seal yet (`MOCK_VERAE` is still true). diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..d929275 --- /dev/null +++ b/TODO.md @@ -0,0 +1,15 @@ +# Open todos + +- [ ] Live `api.veraetime.net` with `MOCK_VERAE=false` and admin bind credentials. +- [x] Three-node NATS JetStream cluster on distinct Proxmox LXC (`verae-nats-cluster`; test env `NATS_URL` cut over; streams replicas=3). +- [ ] NATS nkeys/mTLS on that cluster (accounts file is still the lab stand-in; not flipped so clients keep working on `vmbr1`). +- [x] Exclusive JetStream consumer for `verae.zapier.jobs.events` on `verae-jobs-events` (`JOBS_EVENTS_EXCLUSIVE=1`; middleware skips the router). +- [x] Auth on CS/sales/accounting HTML via `verae-staff-session` (`STAFF_AUTH=1`). +- [x] Staff IAM: named users, roles, permissions (`verae-staff-iam` :3028). +- [ ] Zapier Platform `push` of a private app. +- [x] Move portal static files fully into `verae-access-web` (`/portal/` public door; API proxied to loopback edge). +- [x] Turn IAM on for the running lab (fleet `STAFF_IAM_URL` + restart). +- [x] IAM hardening: JSON sessions, login rate-limit, JSON 401 as well as HTML 302, credit principal = IAM username. +- [x] Host keep-alive (`verae-keep`) on NS1 with watch+guard; honors admin pause/stop. +- [x] Docs synced to IAM, public portal `:3021`, keep, bootstrap server types. +- [x] `verae-bootstrap` clones all module repos and installs deps per server type. diff --git a/decisions/D001-prepaid-sot.md b/decisions/D001-prepaid-sot.md new file mode 100644 index 0000000..9b1bace --- /dev/null +++ b/decisions/D001-prepaid-sot.md @@ -0,0 +1,6 @@ +# D001 — Account-balance is the only prepaid writer + +**Status:** accepted +**Step:** 1 + +Prepaid cents, credits, usage, and payments persist in `zappier-account-balance` (`books.json`). zappier-edge `balanceCents` is a cache filled from the books reply. Tests without NATS/HTTP books still mutate the cache so unit tests stay hermetic. diff --git a/decisions/D002-identity.md b/decisions/D002-identity.md new file mode 100644 index 0000000..7548947 --- /dev/null +++ b/decisions/D002-identity.md @@ -0,0 +1,6 @@ +# D002 — veraeUserId is the join key, not the JWT + +**Status:** accepted +**Step:** 2 + +Public token remains the zappier API key. Chain JWT stays server-side (`tokenRef`). `zappier-identity` is the mailbox for bind/lookup so portal and middleware share one map. diff --git a/decisions/D003-public-doors.md b/decisions/D003-public-doors.md new file mode 100644 index 0000000..9d32527 --- /dev/null +++ b/decisions/D003-public-doors.md @@ -0,0 +1,6 @@ +# D003 — Access planes are the only public HTTP doors + +**Status:** accepted +**Step:** 3 + +Zapier → `:3024`, browser → `:3021`, customer API → `:3022`, leaf → `:3023`, staff → `:3025`. zappier-edge listens on `127.0.0.1:3000` for identity, Stripe, admin, and as upstream for those planes. diff --git a/decisions/D004-leaf-accounts.md b/decisions/D004-leaf-accounts.md new file mode 100644 index 0000000..65de80d --- /dev/null +++ b/decisions/D004-leaf-accounts.md @@ -0,0 +1,6 @@ +# D004 — Leaf cannot touch billing, even if authz is skipped + +**Status:** accepted +**Step:** 4 + +`verae-nats-accounts/policy.json` is the allow-list. access-leaf refuses any target not in the leaf set. nats.conf documents INTERNAL vs LEAF accounts for when the lab cluster is replaced with a real accounts-enabled nats-server. diff --git a/decisions/D005-fleet-local-first.md b/decisions/D005-fleet-local-first.md new file mode 100644 index 0000000..4df6ecd --- /dev/null +++ b/decisions/D005-fleet-local-first.md @@ -0,0 +1,6 @@ +# D005 — HTTP services spawn locally; lan-134 stays off + +**Status:** accepted +**Step:** 5 + +SSH to `70.88.205.134` blocked keepFloor. HTTP apps (`spawn.command`) prefer `kind=local`. `lan-134` is disabled unless `FLEET_ENABLE_LAN134=1`. Failed SSH hosts are skipped after 8s. diff --git a/package.json b/package.json new file mode 100644 index 0000000..a12c852 --- /dev/null +++ b/package.json @@ -0,0 +1,6 @@ +{ + "name": "zapier-decisions", + "version": "0.1.0", + "private": true, + "description": "Architecture decisions and action log for the Verae × Zapier stack" +}