marchon/zapier-decisions
Watch
1
0
Fork
You've already forked zapier-decisions
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Compare commits

base: marchon:main
Branches Tags
marchon:main
marchon:wip/mac-20261026
marchon:master
..
compare: marchon:wip/mac-20261026
Branches Tags
marchon:wip/mac-20261026
marchon:main
marchon:master
Author SHA1 Message Date
George Lambert
3a245f2a4e wip: Mac uncommitted snapshot 2026-10-26 (auto, secrets/large files excluded) 2026-10-26 21:27:46 -04:00
59 changed files with 0 additions and 9451 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.agents/JOB_BUS.md
View file

@ -1 +0,0 @@
/Users/marchon/job-bus/docs/JOB_BUS_LLM_INSTRUCTIONS.md

6
.jobbus.json
View file

@ -1,6 +0,0 @@
{
"enabled": true,
"manifest": "jobs/pieces.yaml",
"farm": "scripts/jobbus-farm.sh",
"instructions": ".agents/JOB_BUS.md"
}

6
AGENTS.md
View file

@ -1,6 +0,0 @@
# Agent notes
## Job-bus multi-build
This repo farms multi-build pieces via NATS job-bus. See [.agents/JOB_BUS.md](.agents/JOB_BUS.md).
Edit `jobs/pieces.yaml`, then run `./scripts/jobbus-farm.sh` (or `make farm`).
CPU pieces → Xeons (137/138); GPU pieces → Mac. Always set `requires` per piece.

159
LOG.md
View file

@ -1,159 +0,0 @@
# Action log
## 2026-10-03 — D006 libSQL business SoT
Accepted **D006**: all billing, customer, pricing, accounting, and related business data live in **libSQL**. JetStream only as needed for multi-host work queues. Core NATS only for Verae timestamp job-complete subjects. Go timestamping DBs unchanged. Single-host `billing.db` on 138 remains the lab stand-in until libSQL cutover.
## 2026-09-12 — exhaustive optimal-config study
- 8c/16G ZFS factorial: file/memory × r=1/r=3, 4 KiB, reconnect, UDP echo, MQTT QoS0 on nats-a :1883 (then removed).
- JS memory r=1 4p **65k** pubs/s; file r=3 1p **14.3k**; MQTT QoS0 **45k**; UDP p99 **0.36 ms**.
- Combined report + DL360 Gen10 projection: `reports/optimal-config/REPORT.{md,html,pdf}`.
- Verdict: keep NATS+JetStream; no MQTT/UDP fabric; metal = 3 NVMe + 10GbE, expect JS r=3 ~40–80k.
## 2026-09-12 — maximized NS1 NATS study
- `pct set` 510–513 to 8 cores / 16 GiB; `max_mem: 8G`; tmpfs 8G on JetStream for the ladder only.
- Run `20260912T053120Z`. JS file 1p 128 B **7.4k → 17.4k (2.35×)**; JS memory 4p **36.4k**; ping p99 **1.38 → 0.68 ms**.
- tmpfs unmounted after; product streams on ZFS again. Resources and max_mem **left raised**.
- Report: `reports/ns1-maximize/`.
## 2026-09-12 — NS1-host NATS study (second run)
- All orchestration, nats bench, matplotlib, pandoc, and weasyprint ran on **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`). Laptop not in the path.
- Script: `verae-nats-cluster/scripts/study-on-ns1.sh` (refuses unless hostname is NS1).
- Run `20260912T051237Z`. Report: `reports/ns1-host/nats-cluster-bench-ns1.{md,html,pdf}`.
- Host load ~8.5. JS 1p 128 B ~7.4k durable pubs/s; ping p99 1.38 ms. Methodology + fine-tuning in that report.
## 2026-09-12 — NATS cluster speed report
- Bench from LXC 510 vs nats-a/b/c (`20260912T045131Z`).
- Full write-up with charts in `reports/nats-cluster-bench.{md,html,pdf}`.
- JetStream r=3 file ~16k durable 128 B pubs/s; ping RTT ~0.3 ms; flood queueing 150–500 ms.
## 2026-09-12 — test env on the 3-node cluster
- Fleet, keep, middleware, billing, and access planes `NATS_URL` → `10.10.10.21–23:4222`.
- Streams ZAPIER_JOBS/EVENTS/WEBHOOKS/USAGE + VERAE_ARCHIVE replicas=3.
- px-worker keep worm/tree 200 on the cluster. Backup timer enabled. Uptime loop on px-worker.
- MOCK_VERAE still true. Zapier push still waiting on login. nkeys/mTLS not flipped (would require client creds).
## 2026-09-12 — NATS cluster on three Proxmox LXC
- New repo **verae-nats-cluster**: LXC 511/512/513 (`nats-a/b/c`, `10.10.10.21–23`) JetStream cluster on `vmbr1`.
- Lab loopback `127.0.0.1:4222` unchanged until cut-over.
- Test: stream replicas=3; same configs later move to distinct hardware.
## 2026-09-11 — proxmox worker + deploy/monitor/backup
- NS1 **is** Proxmox (`vmbr0` public, `vmbr1` `10.10.10.0/24`). New repo **verae-proxmox** creates LXC **510** `verae-px-worker` at `10.10.10.20`.
- NATS for guests: socat `10.10.10.1:4222` → `127.0.0.1:4222` (never vmbr0).
- **verae-uptime** off-box HTTPS GET; **verae-backup** tar snapshots; **verae-deploy** tagged npm ci.
- Fleet `overlays/ns1/` checked in; `start.sh` no longer rewrites fleet/machines JSON.
- User systemd + linger for keep and fleet.
## 2026-09-11 — bootstrap
- Created this repo to record decisions while executing cleanup items 1–5 (and a second pass on 2–5).
- Assumption: single developer, lab only; breaking changes allowed.
## Step 1 — account-balance is the prepaid source of truth
- Persist books to `BOOKS_PATH` (atomic JSON).
- Mutations go through `balance.adjust` / `usage.recorded`; edge cache `balanceCents` is updated from the books reply only.
- Local cache mutate only when `NATS_URL` and `ACCOUNT_BALANCE_URL` are both unset (tests).
## Step 2 — identity join
- `zappier-identity` mailbox: `verae.identity.bind` / `verae.identity.lookup`.
- Portal + middleware tenants store `veraeUserId`; JWT never public.
## Step 3 — access planes are the public doors
- zappier-edge binds loopback (`127.0.0.1:3000`).
- Public: access-web `:3021`, access-api `:3022`, access-leaf `:3023`, access-zapier `:3024`, access-staff `:3025`.
## Step 4 — leaf hardening
- `verae-nats-accounts` holds nats.conf + subject policy.
- Leaf token required; billing subjects denied in policy file and access-leaf.
## Step 5 — fleet spawn
- Prefer local machines for HTTP `spawn.command` services.
- Disable lan-134 unless `FLEET_ENABLE_LAN134=1`.
- SSH spawn timeout 8s; failed hosts skipped.
## 2026-09-11 — catalog live doors
- Bottom of https://zapier.georgelambert.org/ lists each HTTPS door with description, screenshot, and Open link.
- UI-REVIEW.pdf includes the live-door atlas. Walkthrough §13.
## 2026-09-11 — Caddy host names live
- Installed `/etc/caddy/sites/zapier-services.georgelambert.org.caddy` on NS1. HTTPS: portal, api, leaf, zap, staff, iam, login, cs, sales, acct, mw, fleet.
- Not proxied: NATS, keep, workers, loopback edge.
## 2026-09-11 — host names
- Mapped every running service to **NS1.GEORGELAMBERT.ORG** (`70.88.205.138`).
- Suggested `*.zapier.georgelambert.org` names for public doors; only `zapier.georgelambert.org` (docs) and `git.georgelambert.org` exist in Caddy today.
## 2026-09-11 — docs sync, bootstrap, NS1 all-in-one
- Catalog and ops docs: public portal is access-web `:3021/portal/`; edge is loopback; IAM + keep listed.
- New `verae-bootstrap` repo: clone + npm for `ns1-all-in-one`, `control-plane`, `ns1-archive`, `lan-worker`.
- Fleet starts `staff-iam` and `staff-session`. NS1 all-in-one uses edge `:13000` because host `:3000` is taken; keep still owns archive workers. `better-sqlite3` must be rebuilt on Linux (`npm rebuild`).
## 2026-09-11 — verae-keep on NS1
- New repo `verae-keep`: keep `:3860` restarts crashed units; watch `:3861` restarts keep; `guard.sh` restarts watch.
- Honors admin pause/stop via intent files + `POST /intent`. Fleet writes `data/<instance>/intent.json` (local and SSH).
- Tested on 70.88.205.138: crash-restart, pause-hold, watch respawn. webhook-deliver brought back. NATS observed only.
## 2026-09-11 — IAM JSON doors closed
- CS/sales/accounting/staff JSON list and sales PUT pricing now go through IAM (401 JSON, 302 HTML).
- Fleet service env sets `STAFF_AUTH=1` with `STAFF_IAM_URL`.
- Re-walk: cs credit 200 agent=`cs`; cs export 403; operator Fleet POST 200; admin `/me` `permissions:['*']`.
## 2026-09-11 — IAM on lab, portal door, hardening
- Restarted fleet with `STAFF_IAM_URL`. Walk: cs credits (agent=`cs`) and is 403 on accounting export; operator fleet POST 200, cs 403; admin `/me` permissions `*`.
- Portal public door `verae-access-web` `:3021/portal/` (static + `/portal/api` → loopback edge).
- IAM sessions persist in JSON; login rate-limit per IP+username (8/10min); credits stamp IAM username as authz principal.
- lan-134 remains disabled unless `FLEET_ENABLE_LAN134=1`. Receipt PDF branded (indigo header + legal footer).
## 2026-09-11 — staff IAM
- New `verae-staff-iam` :3028 — users, roles, permissions, sessions, audit UI.
- Admin login uses IAM when `STAFF_IAM_URL` is set; tabs hide without permission.
- CS/sales/accounting/access-staff/fleet mutating APIs check `cs.credit`, `sales.quote`, `accounting.export`, `fleet.operate`, etc.
- Seed: admin/cs/sales/accounting/operator. Shared staff key is fallback only.
## 2026-09-11 — last three UI leftovers
- Swagger `/docs` stays stock; banner names it OpenAPI explorer.
- Shared template `verae-staff-ui` for CS + access-staff.
- `STAFF_COOKIE_DOMAIN` + nginx example for multi-host.
## 2026-09-11 — names, staff session, exclusive jobs.events
- Account-balance stores display names (`customer.put` + lookup by name). Edge writes names on customer create/edit; CS/sales/accounting/staff join from edge if the ledger has no name.
- New repo `verae-staff-session` (:3027). Department HTML redirects when `STAFF_AUTH=1`.
- `JOBS_EVENTS_EXCLUSIVE=1` makes `verae-jobs-events` the durable consumer; middleware webhook router backs off.
- Catalog index is cards. Disabled fleet machines are grey, not degraded yellow.
## 2026-09-11 — UI design-system pass
- Restyled CS/sales/accounting/access-staff to portal indigo; dollars + names.
- Fleet ⋯ menus, role chips, Docs cards, 0.0.0.0 copy, simulator indigo (orange = faults).
- Portal masked API key; admin customer drawer; empty invoice illustrations.
- Catalog system-ui. New independent repo **UI-Docs** (walkthrough, screenshots, UI-REVIEW.pdf).
## Second pass (2–5)
- Identity used from portal when `IDENTITY_URL` set.
- Leaf loads allow-list from nats-accounts `policy.json`.
- Staff plane is a first-class mailbox (`verae-access-staff`).
- jobs.events mailbox process `verae-jobs-events` (passthrough + count).

3
NATS.md
View file

@ -1,3 +0,0 @@
# NATS — zapier-decisions
None. Documentation repo.

42
README.md
View file

@ -1,42 +0,0 @@
# zapier-decisions
[![standard-readme compliant](https://img.shields.io/badge/readme%20style-standard-brightgreen.svg?style=flat-square)](https://github.com/RichardLitt/standard-readme)
Lab log of architecture decisions, actions taken, and open todos while cleaning the Verae × Zapier stack.
**Forgejo:** https://git.georgelambert.org/marchon/zapier-decisions
| File | What |
|------|------|
| [STATUS.md](STATUS.md) | **Current status** (2026-09-12) |
| [reports/nats-cluster-bench.md](reports/nats-cluster-bench.md) | NATS cluster speed report (charts, HTML, PDF) |
| [reports/ns1-host/nats-cluster-bench-ns1.md](reports/ns1-host/nats-cluster-bench-ns1.md) | Second study: all code on NS1 (methodology + tuning) |
| [reports/ns1-maximize/nats-cluster-bench-ns1.md](reports/ns1-maximize/nats-cluster-bench-ns1.md) | Maximized: 8c/16G/tmpfs vs baseline |
| [reports/optimal-config/REPORT.md](reports/optimal-config/REPORT.md) | All studies + MQTT/UDP + DL360 projection |
| [LOG.md](LOG.md) | Chronological actions |
| [TODO.md](TODO.md) | Open items |
| [decisions/](decisions/) | One file per decision |
No runtime. Not a NATS mailbox.
## Install
```sh
git clone ssh://git@git.georgelambert.org:2223/marchon/zapier-decisions.git
cd zapier-decisions
```
## Usage
```sh
# No single entrypoint is declared in this tree.
ls
```
## Contributing
Questions: open an issue at https://git.georgelambert.org/marchon/zapier-decisions/issues. Pull requests are accepted.
## License
UNLICENSED © George Lambert

164
STATUS.md
View file

@ -1,164 +0,0 @@
# Current status — 2026-10-03
**Multi-host cutover plan in flight:** [plans/2026-10-03-libsql-multihost-architecture.md](plans/2026-10-03-libsql-multihost-architecture.md) (D006: business SoT = **libSQL**; JetStream = work only). Arch review: **approve-with-changes** (A1–A8 merged). PM board: [plans/reviews/2026-10-03-pm-board.md](plans/reviews/2026-10-03-pm-board.md). Docs map: [plans/2026-10-03-system-docs-map.md](plans/2026-10-03-system-docs-map.md).
---
# Prior status — 2026-09-12
Verae Time × Zapier is a **working lab** on one Proxmox host (**NS1.GEORGELAMBERT.ORG**, `70.88.205.138`). Public HTTPS doors, catalog, keep/fleet, and a **test 3-node NATS cluster** on separate LXC guests are up. The test environment **points at that cluster**. Zapier Platform `push` and live `api.veraetime.net` are still waiting on operator login / bind credentials.
**Progress repo:** this file in [zapier-decisions](https://git.georgelambert.org/marchon/zapier-decisions).
**Catalog:** https://zapier.georgelambert.org/packages/zapier-decisions/STATUS.pdf
**Monorepo:** `master-zapier-plan-draft` (`main` / `master`).
---
## Live right now
| Layer | State |
|-------|--------|
| Catalog | https://zapier.georgelambert.org/ (PDF links path-relative; module/model books work) |
| Public doors | portal, api, zap, iam, mw, fleet, git — last off-box GET 8/8 OK |
| Keep + fleet | Running. systemd + linger **enabled** for next reboot (current processes were not double-started as systemd) |
| Test NATS | **3-node JetStream on `vmbr1`:** LXC 511/512/513 `nats-a/b/c` at `10.10.10.21–23`. Cluster name `verae`, 2 routes each. |
| Product streams | `ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE` — **replicas=3** |
| Modules on that cluster | Fleet, keep, middleware (`natsConnected: true`), billing, access planes. `NATS_URL` lists all three URLs. |
| Host loopback `:4222` | Still listening; **clients no longer use it** |
| Worker CT **510** `verae-px-worker` | `10.10.10.20` on `vmbr1`. Keep worm+tree **200** on the cluster |
| Backup | Timer `verae-backup.timer` enabled; snapshots under `/SSD2/backups/verae/` |
| Uptime | Loop on px-worker (`:3870/status`), last tick `failed: 0` |
Client URL (private `vmbr1` only):
```text
nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
```
Repo: https://git.georgelambert.org/marchon/verae-nats-cluster
### Cluster message speed (`20260912T045131Z`)
Measured from LXC **510** (not a nats-* server) against `10.10.10.21–23`.
**Full report (charts + explanations):** [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.pdf) · [markdown](reports/nats-cluster-bench.md)
**Second study (all code on NS1.GEORGELAMBERT.ORG / 70.88.205.138, run `20260912T051237Z`):** [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.pdf) · [markdown](reports/ns1-host/nats-cluster-bench-ns1.md). Orchestrator, nats bench, charts, HTML, and PDF ran on 138; traffic was LXC 510 → 511–513 on `vmbr1`. Includes full methodology and fine-tuning notes. Host load ~8.5/40 during the run; JetStream 1p 128 B was **7.4k** durable pubs/s (vs ~16k on the quieter first ladder); ping p99 **1.38 ms**.
**Maximized study (`20260912T053120Z`):** 8 cores + 16 GiB on 510–513, `max_mem: 8G`, JetStream **tmpfs** for the ladder (ZFS restored after). veth/10G unchanged. [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.pdf).
| Metric | Baseline 051237Z | Maximized | Ratio |
|--------|------------------|-----------|-------|
| JS file r=3 1p 128 B | 7,393 pubs/s | **17,388** | **2.35×** |
| JS memory r=3 4p 128 B | — | **36,355** | ~4.9× vs baseline file 1p |
| Core 1p 128 B pub | 502,502 | 599,004 | 1.19× |
| Ping p99 | 1.377 ms | **0.684 ms** | **2.01× faster** |
Live cluster **keeps 8 cores / 16 GiB / max_mem 8G**. Product streams are back on ZFS (tmpfs was study-only).
**Optimal-config study (`20260912T055851Z`):** same 8c/16G cluster on **ZFS**, plus r=1 vs r=3, memory vs file, reconnect tax, UDP echo, MQTT QoS0. Combined report with DL360 Gen10 projection: [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/optimal-config/REPORT.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/optimal-config/REPORT.pdf).
| Finding | Number |
|---------|--------|
| JS file r=3 1p 128 B (8c ZFS) | 14,330 pubs/s |
| JS file r=1 1p | 18,888 (r=3 costs ~24% on same host) |
| JS memory r=1 4p | **64,923** (lab JS ceiling) |
| MQTT QoS0 | 44,862 pubs/s (not durable; ~15× slower than core) |
| UDP p99 510→511 | 0.363 ms vs NATS ping p99 1.14 ms |
| Reconnect-per-msg p99 | 1.75 ms vs persistent ping 1.14 ms |
**Do not switch the fabric to MQTT or UDP.** Optimal lab: file r=3 for jobs/archive, memory r=3 optional for events, persistent NATS connections. Metal: 3× DL360 Gen10, local NVMe, 10GbE — projected JS file r=3 **~40–80k** 128 B pubs/s (not measured).
Short tables: [BENCH.md](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md). Re-run: `bash scripts/bench.sh` on NS1.
| Kind | Load | Result |
|------|------|--------|
| Core NATS | 1p1s 50k×128 B | ~1.20M msgs/s aggregate |
| Core NATS | 8p8s 200k×128 B | ~2.01M msgs/s aggregate |
| Core NATS | 4p4s 50k×1 KiB | ~630k msgs/s (~616 MB/s) |
| JetStream file r=3 | 1–4 pubs, 128 B | ~16k durable pubs/s |
| JetStream file r=3 | 4 pubs, 1 KiB | ~13.5k durable pubs/s |
| Ping (sequential RTT) | 1k×128 B | avg 0.307 ms, p99 0.734 ms |
| Flood (burst queueing) | 8p 20k×128 B | avg 454 ms |
Product job/event/archive traffic is the JetStream row. Ping is one-message delay; flood is the mailbox catching up after a burst.
---
## Built and checked in (Forgejo, each its own repo)
IAM, portal on access-web, keep/watch/guard, fleet floors, host-deps, catalog host-move docs, plus:
| Repo | Job |
|------|-----|
| verae-uptime | Off-box HTTPS GET of public doors |
| verae-backup | SQLite / keep / worm-tree tar; restore dry-run default |
| verae-deploy | Tagged checkout + npm ci on the target OS |
| verae-proxmox | LXC 510 worker provision |
| verae-nats-cluster | LXC 511–513 JetStream cluster |
---
## Still open
| Item | Notes |
|------|--------|
| Zapier Platform `push` | Needs Zapier login (see below) |
| `MOCK_VERAE=false` | Live `api.veraetime.net` + admin bind credentials (see below) |
| NATS nkeys/mTLS | Cluster is up **without** accounts so existing `NATS_URL`s work. `verae-nats-accounts` remains the sketch. Flip only after every client has credentials. Isolation today is `vmbr1`, not nkeys. |
| lan-134 | Still disabled (SSH not working) |
| Reboot test | systemd should own keep/fleet after reboot; do not also `nohup guard.sh` |
| Hardware NATS | Same three configs, three boxes, private NIC — change IPs in `cluster.env` |
---
## Once we have a Zapier login
Zapier cloud only talks **HTTPS** to our Zap plane (`https://zap.zapier.georgelambert.org`). No NATS, no `api.veraetime.net`.
1. `zapier login` (or `zapier-platform login`).
2. In `packages/verae-zapier` (and `verae-activate` if the tiny app is still wanted):
- `zapier-platform register` (or attach an existing app id).
- `MIDDLEWARE_BASE_URL=https://zap.zapier.georgelambert.org` (or `https://mw.zapier.georgelambert.org` if skipping the Zap plane).
- `zapier-platform build` and `validate`.
3. `zapier-platform push` as a **private** app.
4. Invite the workspace; create a Zap:
- Auth = portal **API key** (`zmw_…`), not a Verae JWT.
- Test **Create Timestamp and Wait**, **Create Timestamp** (async `jobId`), **Find Job Status**, **Timestamp Completed** REST Hook.
5. Fix anything `validate` / `push` flags (auth labels, samples, 402/403 mapping).
6. When the contract is stable, bump the platform version and invite a real user.
Do **not** put NATS URLs or Verae passwords into Zapier.
---
## Once `MOCK_VERAE=false` and live `api.veraetime.net`
Middleware already has the live HTTP client. Still needed:
1. `VERAE_API_BASE_URL=https://api.veraetime.net` and working **login** (username/password → JWT). JWT stays in middleware (`tokenRef`); Zapier never sees it.
2. Confirm OpenAPI vs our client:
- `POST /api/timestamp` → **202** `{ jobId }`
- `GET /api/status/{jobId}` until `completed` / `failed`
- `POST /api/verify`
- Batch create/status if used
3. **Hash lookup is not on the live OpenAPI** — `veraeClient.lookupHash` returns **501** when not mocking. Central-chain “find by SHA-256” needs a Verae API gap fill or our own store.
4. Map live status JSON into the receipt shape the Zap app expects (`certificate`, `timestamp`, `blockIndex`, `sha256`).
5. Token refresh, 401 retry, and 402/403 mapping against real errors.
6. One live seal in the lab, then REST Hook delivery to `hooks.zapier.com`.
---
## Submit → jobId → wait → receipt
**Yes — that is the designed path**, and it already runs in **mock**.
1. Zap (or curl) `POST /zapier/v1/timestamp` or `/timestamp/wait`.
2. Middleware `createTimestamp` calls Verae `POST /api/timestamp` (or the mock) and gets **`jobId`**.
3. It publishes **`verae.zapier.jobs.watch`** on NATS (the 3-node cluster).
4. **Wait:** subscribe to **`verae.zapier.jobs.events`** for that `jobId`, or poll `GET /api/status/{jobId}`.
5. Job-poller hits Verae status until `completed`.
6. Event `timestamp.completed` carries status; webhook-deliver POSTs the Zapier REST Hook.
7. Response includes **certificate / timestamp / blockIndex** (mock uses `mock-cert-{jobId}`, `blockIndex: 42`). Async create returns `{ jobId, sha256, existing }` immediately; wait returns the finished status. Same SHA-256 is idempotent (`existing: true`).
If wait times out, middleware returns **`pending` + `jobId`** so the hook can finish the Zap. Live Verae must match that HTTP contract; we have **not** proven a real chain seal yet (`MOCK_VERAE` is still true).

16
TODO.md
View file

@ -1,16 +0,0 @@
# Open todos
- [ ] Live `api.veraetime.net` with `MOCK_VERAE=false` and admin bind credentials.
- [x] Three-node NATS JetStream cluster on distinct Proxmox LXC (`verae-nats-cluster`; test env `NATS_URL` cut over; streams replicas=3).
- [ ] NATS nkeys/mTLS on that cluster (accounts file is still the lab stand-in; not flipped so clients keep working on `vmbr1`).
- [x] Exclusive JetStream consumer for `verae.zapier.jobs.events` on `verae-jobs-events` (`JOBS_EVENTS_EXCLUSIVE=1`; middleware skips the router).
- [x] Auth on CS/sales/accounting HTML via `verae-staff-session` (`STAFF_AUTH=1`).
- [x] Staff IAM: named users, roles, permissions (`verae-staff-iam` :3028).
- [ ] Zapier Platform `push` of a private app.
- [x] Move portal static files fully into `verae-access-web` (`/portal/` public door; API proxied to loopback edge).
- [x] Turn IAM on for the running lab (fleet `STAFF_IAM_URL` + restart).
- [x] IAM hardening: JSON sessions, login rate-limit, JSON 401 as well as HTML 302, credit principal = IAM username.
- [x] Host keep-alive (`verae-keep`) on NS1 with watch+guard; honors admin pause/stop.
- [x] Docs synced to IAM, public portal `:3021`, keep, bootstrap server types.
- [x] `verae-bootstrap` clones all module repos and installs deps per server type.
- [ ] Cut over billing/customers/pricing/usage ledger from single-host billing.db to shared libSQL (D006); keep JetStream for jobs only.

13
decisions/D001-prepaid-sot.md
View file

@ -1,13 +0,0 @@
# D001 — Account-balance is the only prepaid writer
**Status:** accepted
**Step:** 1
Prepaid cents, credits, usage, and payments persist in `zappier-account-balance` (`books.json`). zappier-edge `balanceCents` is a cache filled from the books reply. Tests without NATS/HTTP books still mutate the cache so unit tests stay hermetic.
## Addendum (2026-10-03, D006)
Durable prepaid/credits/usage/payments store is shared **libSQL**, not per-process
`books.json` or per-host `billing.db`. Single writer path still required. Edge
`balanceCents` remains a cache. Lab `books.json` / `billing.db` are migration sources.

6
decisions/D002-identity.md
View file

@ -1,6 +0,0 @@
# D002 — veraeUserId is the join key, not the JWT
**Status:** accepted
**Step:** 2
Public token remains the zappier API key. Chain JWT stays server-side (`tokenRef`). `zappier-identity` is the mailbox for bind/lookup so portal and middleware share one map.

6
decisions/D003-public-doors.md
View file

@ -1,6 +0,0 @@
# D003 — Access planes are the only public HTTP doors
**Status:** accepted
**Step:** 3
Zapier → `:3024`, browser → `:3021`, customer API → `:3022`, leaf → `:3023`, staff → `:3025`. zappier-edge listens on `127.0.0.1:3000` for identity, Stripe, admin, and as upstream for those planes.

6
decisions/D004-leaf-accounts.md
View file

@ -1,6 +0,0 @@
# D004 — Leaf cannot touch billing, even if authz is skipped
**Status:** accepted
**Step:** 4
`verae-nats-accounts/policy.json` is the allow-list. access-leaf refuses any target not in the leaf set. nats.conf documents INTERNAL vs LEAF accounts for when the lab cluster is replaced with a real accounts-enabled nats-server.

6
decisions/D005-fleet-local-first.md
View file

@ -1,6 +0,0 @@
# D005 — HTTP services spawn locally; lan-134 stays off
**Status:** accepted
**Step:** 5
SSH to `70.88.205.134` blocked keepFloor. HTTP apps (`spawn.command`) prefer `kind=local`. `lan-134` is disabled unless `FLEET_ENABLE_LAN134=1`. Failed SSH hosts are skipped after 8s.

39
decisions/D006-libsql-business-jetstream-work.md
View file

@ -1,39 +0,0 @@
# D006 — Business data in libSQL; JetStream only for work
**Status:** accepted
**Step:** 6
**Date:** 2026-10-03
Launch topology is **3–5 machines** (gateway, bridge, delivery, and related planes). Shared business state must not live as per-host SQLite files.
## Decision
1. **System of record for business / accounting / customers / pricing** is **libSQL** (Turso’s SQLite fork; server mode or hosted). That includes:
- customers, segments, tiers, balances, Stripe / PO metadata
- pricing and channel×segment rate cards
- invoices and invoice lines
- metered usage ledger (`usage_entries` and its append-only hash chain)
- other accounting and CRM-style records for the Zapier/Verae edge stack
2. **NATS JetStream** is used **as necessary** for multi-host **work distribution** (job queues, competing consumers, retries, durable event fans that are not the money ledger). Examples: `ZAPIER_JOBS`, worker pull/ack, bridge task streams.
3. **Core NATS (ephemeral)** remains mandatory for Verae timestamp **job-complete / jobId** notifications (`verae.ts.completed`, `verae.ts.receipt`). Those must **not** move to JetStream.
4. **Timestamping-svc Go** `blockchain.db` and other Go timestamping SQLite files stay on that service; this decision does not relocate them.
## Relationship to earlier decisions
- **D001** (single prepaid writer) still holds: one logical writer path for prepaid/credits. The durable store for that path becomes **libSQL**, not per-process `books.json` or a private `billing.db` on each machine. Edge `balanceCents` may remain a cache filled from the billing/libSQL service.
- Lab `billing.db` / `zappier.db` on 138 are the current single-host stand-in; migrate that schema into shared libSQL before splitting planes across hosts.
## Non-goals
- JetStream as the customer/balance/invoice/pricing database
- Litestream as a multi-writer fabric (Litestream remains backup/DR for a primary if used)
- Per-machine copies of customer or pricing tables
## Implementation sketch
- Keep repositories behind a small DB interface; swap better-sqlite3 / modernc file opens for libSQL client when the shared server is up.
- One billing writer path owns usage hash-chain appends (`BEGIN`-style sequencing).
- Gateway/bridge/delivery call billing APIs or the shared libSQL endpoint; they do not each open a local billing file for production.

10
decisions/index.md
View file

@ -1,10 +0,0 @@
# Decisions
6 files in this book.
- [D001-prepaid-sot.md](D001-prepaid-sot.md)
- [D002-identity.md](D002-identity.md)
- [D003-public-doors.md](D003-public-doors.md)
- [D004-leaf-accounts.md](D004-leaf-accounts.md)
- [D005-fleet-local-first.md](D005-fleet-local-first.md)
- [D006-libsql-business-jetstream-work.md](D006-libsql-business-jetstream-work.md)

23
jobs/pieces.yaml
View file

@ -1,23 +0,0 @@
# Multi-build piece manifest for job-bus.
# Copy to jobs/pieces.yaml in your repo and edit.
# Each piece becomes one tracked job; kind routes cpu→Xeons, gpu→Mac.
batch: "zapier-decisions-EDITME"
defaults:
kind: cpu
requires: []
timeout_sec: 3600
cleanup: none # or on_success; full wipe via: job-cleanup --batch <batch>
pieces:
- id: piece-01
kind: cpu
requires: [bin:bash]
cmd: ["make", "piece-01"]
# cwd: defaults to repo root on submitter; set worker-visible path if needed
- id: piece-02
kind: cpu
requires: [ffmpeg, pip:numpy]
cmd: ["python3", "scripts/build_piece.py", "--piece", "2"]
- id: piece-gpu-hero
kind: gpu
requires: [pip:Pillow]
cmd: ["python3", "scripts/gpu_hero.py"]

6
package.json
View file

@ -1,6 +0,0 @@
{
"name": "zapier-decisions",
"version": "0.1.0",
"private": true,
"description": "Architecture decisions and action log for the Verae × Zapier stack"
}

178
plans/2026-10-03-forgejo-sources.md
View file

@ -1,178 +0,0 @@
# Verae × Zapier Forgejo sources
Research date: 2026-10-03
Sources:
- Forgejo API `https://git.georgelambert.org/api/v1/repos/search` (q=zapier,zappier,verae,nats,account-balance,access) plus direct `/repos/marchon/<name>`
- Host `marchon@70.88.205.138`: catalog `/SSD2/sites/zapier.georgelambert.org/packages`, live `/opt/verae/{zappier-edge,verae-zapier-middleware}`, worktree `/home/marchon/zappier-edge-work`
## Executive summary — what is SoT
| Concern | Source of truth (design) | What is actually live on 138 today |
|---|---|---|
| **Billing / customers / pricing / invoices / usage ledger** | **`marchon/zappier-edge`** (+ D006: migrate durable store to **shared libSQL**). Department UIs are thin clients. | **`/opt/verae/zappier-edge`** with **`billing.db`** (ledger) + **`zappier.db`** (auth/sessions), WAL SQLite. Worktree `/home/marchon/zappier-edge-work` is ahead of Forgejo. Service `zappier-edge.service`. |
| **Prepaid balance book** | Design D001: **`marchon/zappier-account-balance`** persists **`books.json`**; edge `balanceCents` is a cache. D006: prepaid writer path stays single, store becomes libSQL. | No `books.json` found under `/opt/verae` or `/home/marchon`. Live prepaid appears folded into edge SQLite/`balanceCents` until account-balance/libSQL cutover. |
| **Public doors (access planes)** | **`verae-access-{zapier,web,api,leaf,staff}`** + **`verae-access-authz`** (D003). Edge stays loopback. | Catalog packages exist; **not** confirmed as separate live systemd units on this pass. Live public surface for Zapier metering is **`zap.verae.ai` → zappier-edge :3040** plus **`verae-zapier-middleware.service`**. |
| **NATS / JetStream fabric** | **`verae-nats-cluster`** (LXC 511–513), **`verae-nats-accounts`**, **`verae-nats-process`** template. JetStream streams designed: `ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE`. | Live: `nats-public-edge.service`, `verae-nats-private-proxy.service`, various leaf workers. **D006 / standing rule:** JetStream for **work queues only**; **core NATS** for timestamp **job-complete/jobId** (`verae.ts.completed`, `verae.ts.receipt`) — never JetStream for those. |
| **Jobs (Zapier timestamp path)** | **`verae-middleware`** owns `/zapier/v1`, job wait, NATS publish/consume; **`verae-jobs-events`** mailbox for `verae.zapier.jobs.events`; splitter/worm/tree for archive path. | **`/opt/verae/verae-zapier-middleware`** + `verae-zapier-middleware.service` active. Edge also does direct Verae/batch metering on `:3040`. |
| **Decisions / libSQL** | **`zapier-decisions`** D006 (2026-10-03): libSQL = business SoT; JetStream = multi-host work; core NATS = job-complete. | Decision file present in catalog `packages/zapier-decisions/decisions/D006-libsql-business-jetstream-work.md`. |
**Important:** `/SSD2/sites/zapier.georgelambert.org/packages/*` are **catalog doc trees** (README/PDF), **not** git working copies (no `.git`). Live code under `/opt/verae/zappier-edge` is a **deployed tree without `.git`**. Clone from Forgejo SSH port **2223**.
## Catalog package ↔ Forgejo repo
| Catalog package | Forgejo `full_name` | Default branch | Forgejo updated | Size | Active? | One-line purpose | Clone (HTTPS) | Clone (SSH) |
|---|---|---|---|---|---|---|---|---|
| `zappier` | `marchon/zappier-edge` | main | 2026-09-23T14:00:59 | 187072 | LIVE on 138 (/opt/verae/zappier-edge); worktree Oct 2 ahead of Forgejo Sep 23 | Metered HTTPS edge: portal, admin, Stripe, rate card, usage ledger; proxies Verae to middleware | `https://git.georgelambert.org/marchon/zappier-edge.git` | `ssh://git@git.georgelambert.org:2223/marchon/zappier-edge.git` |
| `zappier-account-balance` | `marchon/zappier-account-balance` | main | 2026-09-23T14:00:59 | 48 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Prepaid balance SoT via NATS (books.json design); edge balanceCents is cache | `https://git.georgelambert.org/marchon/zappier-account-balance.git` | `ssh://git@git.georgelambert.org:2223/marchon/zappier-account-balance.git` |
| `zappier-customer-service` | `marchon/zappier-customer-service` | main | 2026-09-23T14:00:59 | 68 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | CS goodwill credits / statement review over verae.billing.* | `https://git.georgelambert.org/marchon/zappier-customer-service.git` | `ssh://git@git.georgelambert.org:2223/marchon/zappier-customer-service.git` |
| `zappier-sales-pricing` | `marchon/zappier-sales-pricing` | main | 2026-09-23T14:00:59 | 59 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Sales per-customer tier/multiplier; reviews via NATS statement | `https://git.georgelambert.org/marchon/zappier-sales-pricing.git` | `ssh://git@git.georgelambert.org:2223/marchon/zappier-sales-pricing.git` |
| `zappier-accounting-export` | `marchon/zappier-accounting-export` | main | 2026-09-23T14:00:59 | 55 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | QuickBooks IIF + accounting CSV from edge invoices + NATS statements | `https://git.georgelambert.org/marchon/zappier-accounting-export.git` | `ssh://git@git.georgelambert.org:2223/marchon/zappier-accounting-export.git` |
| `zappier-identity` | `marchon/zappier-identity` | main | 2026-09-23T14:00:59 | 33 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Bind/lookup veraeUserId (not JWTs) | `https://git.georgelambert.org/marchon/zappier-identity.git` | `ssh://git@git.georgelambert.org:2223/marchon/zappier-identity.git` |
| `verae-zapier-middleware` | `marchon/verae-middleware` | main | 2026-09-22T06:36:26 | 145 | LIVE on 138 (verae-zapier-middleware.service) | Zapier HTTPS /zapier/v1/* + NATS workers; Zapier never speaks NATS | `https://git.georgelambert.org/marchon/verae-middleware.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-middleware.git` |
| `verae-zapier` | `marchon/verae-zapier-app` | main | 2026-09-22T06:17:43 | 102 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Zapier Platform CLI app (timestamp/wait/batch/verify/hooks) | `https://git.georgelambert.org/marchon/verae-zapier-app.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-zapier-app.git` |
| `verae-zapier-simulator` | `marchon/verae-zapier-simulator` | main | 2026-09-23T14:00:58 | 73 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | In-process Zapier editor stand-in with hop trace console | `https://git.georgelambert.org/marchon/verae-zapier-simulator.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-zapier-simulator.git` |
| `verae-activate` | `marchon/verae-activate` | main | 2026-09-23T14:00:47 | 57 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Activate-now Zapier app (Add Numbers, Echo, SHA256, mock stamp) | `https://git.georgelambert.org/marchon/verae-activate.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-activate.git` |
| `verae-access-zapier` | `marchon/verae-access-zapier` | main | 2026-09-23T14:00:57 | 33 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Public Zapier HTTPS door (:3024) into middleware | `https://git.georgelambert.org/marchon/verae-access-zapier.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-access-zapier.git` |
| `verae-access-web` | `marchon/verae-access-web` | main | 2026-09-23T14:00:57 | 39 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Public customer web/portal door (:3021) | `https://git.georgelambert.org/marchon/verae-access-web.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-access-web.git` |
| `verae-access-api` | `marchon/verae-access-api` | main | 2026-09-23T14:00:57 | 33 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Public direct customer API door (:3022) | `https://git.georgelambert.org/marchon/verae-access-api.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-access-api.git` |
| `verae-access-leaf` | `marchon/verae-access-leaf` | main | 2026-09-23T14:00:57 | 37 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | S2S leaf plane (:3023); no billing subjects | `https://git.georgelambert.org/marchon/verae-access-leaf.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-access-leaf.git` |
| `verae-access-staff` | `marchon/verae-access-staff` | main | 2026-09-23T14:00:57 | 55 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Staff CS/sales/accounting plane (:3025) | `https://git.georgelambert.org/marchon/verae-access-staff.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-access-staff.git` |
| `verae-access-authz` | `marchon/verae-access-authz` | main | 2026-09-23T14:00:57 | 41 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | NATS authz.check gate for all access planes | `https://git.georgelambert.org/marchon/verae-access-authz.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-access-authz.git` |
| `verae-jobs-events` | `marchon/verae-jobs-events` | main | 2026-09-23T14:00:58 | 33 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Mailbox for verae.zapier.jobs.events (passthrough today) | `https://git.georgelambert.org/marchon/verae-jobs-events.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-jobs-events.git` |
| `verae-nats-accounts` | `marchon/verae-nats-accounts` | main | 2026-09-22T06:36:30 | 33 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | INTERNAL vs LEAF NATS account/subject policy | `https://git.georgelambert.org/marchon/verae-nats-accounts.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-nats-accounts.git` |
| `verae-nats-cluster` | `marchon/verae-nats-cluster` | main | 2026-09-22T06:16:14 | 3666 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | 3× Proxmox LXC JetStream cluster (10.10.10.21–23) | `https://git.georgelambert.org/marchon/verae-nats-cluster.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-nats-cluster.git` |
| `verae-nats-process` | `marchon/verae-nats-process` | main | 2026-09-23T14:00:58 | 40 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Template for a new addressed NATS process | `https://git.georgelambert.org/marchon/verae-nats-process.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-nats-process.git` |
| `verae-request-splitter` | `marchon/verae-request-splitter` | main | 2026-09-23T14:00:58 | 42 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Split chain hash vs off-chain archive.put | `https://git.georgelambert.org/marchon/verae-request-splitter.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-request-splitter.git` |
| `verae-archive-worm` | `marchon/verae-archive-worm` | main | 2026-09-23T14:00:58 | 43 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Bloom-filtered WORM archive node | `https://git.georgelambert.org/marchon/verae-archive-worm.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-archive-worm.git` |
| `verae-archive-aggregator` | `marchon/verae-archive-aggregator` | main | 2026-09-23T14:00:58 | 44 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Merge archive.query replies into job JSON | `https://git.georgelambert.org/marchon/verae-archive-aggregator.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-archive-aggregator.git` |
| `verae-tree-node` | `marchon/verae-tree-node` | main | 2026-09-23T14:00:58 | 42 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Merkle leaf proofs for bulk summaries | `https://git.georgelambert.org/marchon/verae-tree-node.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-tree-node.git` |
| `verae-fleet` | `marchon/verae-fleet` | main | 2026-09-22T06:37:31 | 3896 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Operator catalog, replica floors, pause/restart console | `https://git.georgelambert.org/marchon/verae-fleet.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-fleet.git` |
| `verae-keep` | `marchon/verae-keep` | main | 2026-09-22T06:35:54 | 43 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Host keep-alive; restarts unless operator paused/stopped | `https://git.georgelambert.org/marchon/verae-keep.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-keep.git` |
| `verae-bootstrap` | `marchon/verae-bootstrap` | main | 2026-09-22T06:17:17 | 54 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Clone all module repos + host-deps per server type | `https://git.georgelambert.org/marchon/verae-bootstrap.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-bootstrap.git` |
| `verae-ops` | `marchon/verae-ops` | main | 2026-09-11T19:00:06 | 86 | Older Forgejo (Sep 11); still the install book | Docker/Proxmox/VM/hardware install and service linking | `https://git.georgelambert.org/marchon/verae-ops.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-ops.git` |
| `verae-deploy` | `marchon/verae-deploy` | main | 2026-09-11T18:36:11 | 25 | Older Forgejo (Sep 11) | Tagged checkout + npm ci on target OS | `https://git.georgelambert.org/marchon/verae-deploy.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-deploy.git` |
| `verae-proxmox` | `marchon/verae-proxmox` | main | 2026-09-22T06:36:28 | 28 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Provision archive-worker LXC on NS1 | `https://git.georgelambert.org/marchon/verae-proxmox.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-proxmox.git` |
| `verae-backup` | `marchon/verae-backup` | main | 2026-09-22T06:31:38 | 28 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Backup/restore SQLite, JetStream, worm/tree, IAM | `https://git.georgelambert.org/marchon/verae-backup.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-backup.git` |
| `verae-uptime` | `marchon/verae-uptime` | main | 2026-09-22T06:36:26 | 28 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Off-box HTTPS watch of public doors | `https://git.georgelambert.org/marchon/verae-uptime.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-uptime.git` |
| `verae-staff-iam` | `marchon/verae-staff-iam` | main | 2026-09-23T14:00:58 | 49 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Internal staff users/roles/permissions SoT | `https://git.georgelambert.org/marchon/verae-staff-iam.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-staff-iam.git` |
| `verae-staff-session` | `marchon/verae-staff-session` | main | 2026-09-23T14:00:58 | 37 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Shared staff cookie login | `https://git.georgelambert.org/marchon/verae-staff-session.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-staff-session.git` |
| `verae-staff-ui` | `marchon/verae-staff-ui` | main | 2026-09-23T14:00:58 | 32 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Shared staff review HTML template | `https://git.georgelambert.org/marchon/verae-staff-ui.git` | `ssh://git@git.georgelambert.org:2223/marchon/verae-staff-ui.git` |
| `overview` | `marchon/overview` | main | 2026-09-22T06:41:52 | 1567 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | High-level system map and NATS/uptime docs | `https://git.georgelambert.org/marchon/overview.git` | `ssh://git@git.georgelambert.org:2223/marchon/overview.git` |
| `docs-master` | `marchon/zapier-docs-master` | main | 2026-09-22T06:32:02 | 197 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Per-module SUMMARY/NATS contracts and message flows | `https://git.georgelambert.org/marchon/zapier-docs-master.git` | `ssh://git@git.georgelambert.org:2223/marchon/zapier-docs-master.git` |
| `dev-overview` | `marchon/dev-overview` | main | 2026-09-22T06:17:14 | 327 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Designed vs running NATS routes; Zapier→Verae path | `https://git.georgelambert.org/marchon/dev-overview.git` | `ssh://git@git.georgelambert.org:2223/marchon/dev-overview.git` |
| `zapier-decisions` | `marchon/zapier-decisions` | main | 2026-09-22T06:44:58 | 1920 | Active decision log; D006 dated 2026-10-03 in catalog copy | Architecture decisions (D001–D006), logs, benches | `https://git.georgelambert.org/marchon/zapier-decisions.git` | `ssh://git@git.georgelambert.org:2223/marchon/zapier-decisions.git` |
| `zapier-user-docs` | `marchon/zapier-user-docs` | main | 2026-09-23T14:00:59 | 52 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Customer signup → register → chain/tree lookup | `https://git.georgelambert.org/marchon/zapier-user-docs.git` | `ssh://git@git.georgelambert.org:2223/marchon/zapier-user-docs.git` |
| `ui-docs` | `marchon/UI-Docs` | main | 2026-09-23T14:00:57 | 21512 | Catalog/Forgejo batch ~Sep 22–23; not confirmed as live systemd unit | Operator/staff/portal walkthrough screenshots + review PDF | `https://git.georgelambert.org/marchon/UI-Docs.git` | `ssh://git@git.georgelambert.org:2223/marchon/UI-Docs.git` |
## Related Forgejo repos (search hits, not catalog packages)
| full_name | updated | purpose | clone |
|---|---|---|---|
| `marchon/master-zapier-plan-draft` | 2026-09-20T19:06:04 | Monorepo plan/snapshot workspace; not the only clone path | `https://git.georgelambert.org/marchon/master-zapier-plan-draft.git` |
| `marchon/nats-service-endpoints` | 2026-09-23 | Canonical NATS endpoint catalog | `https://git.georgelambert.org/marchon/nats-service-endpoints.git` |
| `marchon/verae-mcp-timestamping-server` | 2026-09-29 | MCP timestamping (live `/opt/verae/...`) | `https://git.georgelambert.org/marchon/verae-mcp-timestamping-server.git` |
| `marchon/verae-mcp-timestamping-server-go` | 2026-09-29 | Go MCP + tsnats sibling | `https://git.georgelambert.org/marchon/verae-mcp-timestamping-server-go.git` |
## Notes: billing, libSQL, sqlite, JetStream, books.json, account-balance
- **billing:** Live ledger/rate card/customers/invoices in **zappier-edge** `billing.db` (`BILLING_DB`); auth in `zappier.db`. Docs in edge `docs/BILLING-DB.md`, `CHANNEL-BILLING.md`, `PEERGOS-BILLING.md`. Department repos (CS/sales/accounting-export) call NATS + edge admin; they are not the money DB.
- **sqlite:** Lab/single-host stand-in. Edge uses better-sqlite3 + WAL. Go timestamping-svc DBs stay separate (`blockchain.db`) — do not relocate (D006).
- **libSQL:** Accepted in **D006** as shared SoT for business data across 3–5 machines. Not yet the live store on 138.
- **JetStream:** Designed streams for jobs/events/webhooks/usage/archive. Use for **work distribution** only. **Forbidden** for timestamp job-complete/jobId notify (core NATS subjects).
- **books.json:** Documented as persistence for **zappier-account-balance**. Not found on 138 in this survey; design SoT for prepaid until libSQL migration absorbs it (D001∩D006).
- **account-balance:** Forgejo `marchon/zappier-account-balance`; subjects `verae.billing.statement.get|balance.get|balance.adjust` + usage/payment/credit pubs; queue group `account-balance`; HTTP `:3010`.
## Host inventory (138)
### Catalog packages (`/SSD2/sites/zapier.georgelambert.org/packages`)
Doc snapshots dated ~2026-09-14. Names match table above. No `.git` remotes.
### Live `/opt/verae` Zapier-related
- `/opt/verae/zappier-edge` — `zappier-edge.service` **active**; `billing.db`/`zappier.db` present; **no `.git`**
- `/opt/verae/verae-zapier-middleware` — `verae-zapier-middleware.service` **active**; README points at Forgejo `verae-middleware`
- Work copy: `/home/marchon/zappier-edge-work` (mtime Oct 2) — treat as practical edge SoT until Forgejo catch-up
## Recommended clone list for implementation
SSH base: `ssh://git@git.georgelambert.org:2223/marchon/<repo>.git` (port **2223**).
### Must clone first (critical path)
```bash
mkdir -p ~/verae-src && cd ~/verae-src
git clone ssh://git@git.georgelambert.org:2223/marchon/zappier-edge.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-middleware.git
git clone ssh://git@git.georgelambert.org:2223/marchon/zappier-account-balance.git
git clone ssh://git@git.georgelambert.org:2223/marchon/zapier-decisions.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-access-zapier.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-access-web.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-access-api.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-access-authz.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-nats-cluster.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-nats-accounts.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-jobs-events.git
git clone ssh://git@git.georgelambert.org:2223/marchon/overview.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-bootstrap.git
```
### Next (jobs / archive path + Zapier app)
```bash
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-zapier-app.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-request-splitter.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-archive-worm.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-archive-aggregator.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-tree-node.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-access-leaf.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-access-staff.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-zapier-simulator.git
```
### Billing department satellites
```bash
git clone ssh://git@git.georgelambert.org:2223/marchon/zappier-customer-service.git
git clone ssh://git@git.georgelambert.org:2223/marchon/zappier-sales-pricing.git
git clone ssh://git@git.georgelambert.org:2223/marchon/zappier-accounting-export.git
git clone ssh://git@git.georgelambert.org:2223/marchon/zappier-identity.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-staff-iam.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-staff-session.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-staff-ui.git
```
### Ops / docs (as needed)
```bash
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-ops.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-fleet.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-keep.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-deploy.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-backup.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-uptime.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-proxmox.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-nats-process.git
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-activate.git
git clone ssh://git@git.georgelambert.org:2223/marchon/zapier-docs-master.git
git clone ssh://git@git.georgelambert.org:2223/marchon/zapier-user-docs.git
git clone ssh://git@git.georgelambert.org:2223/marchon/UI-Docs.git
git clone ssh://git@git.georgelambert.org:2223/marchon/dev-overview.git
git clone ssh://git@git.georgelambert.org:2223/marchon/master-zapier-plan-draft.git
```
Or use bootstrap: clone `verae-bootstrap` then `bash scripts/fetch.sh ns1-all-in-one` (full `types/` may be incomplete in the **catalog** copy; prefer Forgejo clone of bootstrap).
## Search hit counts (API)
| q | results (limit 50) |
|---|---|
| zapier | 7 |
| zappier | 6 |
| verae | 32 |
| nats | 4 |
| account-balance | 1 |
| access | 7 (includes unrelated MCP mirror) |
## Caveats
1. Forgejo `updated_at` for most module repos clusters on **2026-09-22/23** (likely bulk push). **Live edge code on 138 is newer (Oct 2)** than Forgejo edge — sync/push before treating Forgejo as current for billing.
2. Catalog package `verae-fleet` README incorrectly linked `verae-keep` in one scrape; overview table maps fleet→`verae-fleet` correctly.
3. Catalog `docs-master` README first Forgejo link hit `verae-bootstrap` via grep noise; real docs repo is **`zapier-docs-master`**.
4. `master-zapier-plan-draft` is a large monorepo snapshot — useful archaeology, not the independent-module clone path.

295
plans/2026-10-03-libsql-multihost-architecture.md
View file

@ -1,295 +0,0 @@
# Verae × Zapier Multi-Host Architecture & Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (or executing-plans) task-by-task. Steps use checkbox (`- [ ]`) syntax.
**Goal:** Run the Verae × Zapier commercial stack across **3–5 machines** with **libSQL** as the system of record for all business/billing/customer/pricing/accounting data, **JetStream only for work queues**, and proven end-to-end userflows under continuous QA.
**Architecture:** Public HTTPS doors → zappier-edge (meter) → middleware (stamp) → `api.veraetime.net`; private NATS for workers; shared libSQL for money & CRM; Go timestamping DBs stay local to timestamping-svc.
**Tech Stack:** Node (edge, middleware, access planes, CS/sales/acct), NATS JetStream (3-node lab → production), libSQL / go-libsql or `@libsql/client`, better-sqlite3 only as local/dev adapter, Litestream optional for libSQL file primary backup, Jest + gate scripts + E2E userflow harness.
**Spec / SoT docs:**
- Catalog: https://zapier.georgelambert.org/
- Map: `/tmp/zapier-system-docs-map.md`
- Decisions: D001–D006 (`packages/zapier-decisions/decisions/`)
- Composition: `docs/02-architecture/composition.md`
- NATS: `docs/02-architecture/nats-subjects.md`, `overview/03-nats-cluster.md`
- Ops: `packages/verae-ops/GETTING-STARTED.md`
- Digests-only stamps: mcp.verae.ai + D006 note + live swagger
**Forgejo org:** https://git.georgelambert.org/marchon/ (modules listed in §2)
## Global Constraints
- **D006:** Business/billing/customers/pricing/accounting/usage ledger → **libSQL only** (shared).
- **JetStream:** work queues as necessary (`ZAPIER_JOBS`, `ZAPIER_WEBHOOKS`, archive put/query). Not money SoT.
- **Core NATS only** for `verae.ts.completed` / `verae.ts.receipt` (no JetStream on that path).
- **D001 refined:** single prepaid **writer path**; durable store is libSQL (not per-host `books.json` / `billing.db`).
- **D002:** public credential = zappier `x-api-key`; `veraeUserId` join; JWT = `tokenRef` server-side.
- **D003:** only access planes are public HTTP; edge loopback (except intentional public `zap.verae.ai` edge).
- **D004:** leaf must not touch billing.
- **Digests only** for stamp submit (`POST /api/batch/timestamp` multihash; MCP shape).
- **Never** modify Go timestamping-svc SQLite (`blockchain.db`, etc.) as part of this plan.
- **WAL** (or libSQL equivalent durability) on all SQL stores.
- Credits language in product copy; ledger may keep `*_cents` column names as credit units.
- Deploy never rsyncs live `*.db*` over production data.
- **ZAPIER_USAGE** / `verae.zapier.usage` is telemetry or export only — never usage/money SoT.
- Litestream (if used) backs up a **single** libSQL primary; never multi-writer fabric.
- **Single meter/prepaid writer** process owns `usage_entries` append + balance adjust; other hosts call that API.
- Dual-run: libSQL primary for writes; file/SQLite fallback is **read-only** (or kill-switched).
- Auth (`admin_users`, `portal_sessions`): v1 remains edge-local `zappier.db`; not in `verae_business`.
- Product edge bind on 138: `127.0.0.1:3040` (`zap.verae.ai`); `ZAPPIER_UPSTREAM` defaults to `http://127.0.0.1:3130` for live lab smoke (not `:3100` mock unless intentional).
- Catalog docs that call account-balance “prepaid SoT” are stale after D006; Phase 0 patches them.
---
## 1. System map (from catalog + live lab)
```text
Internet
├─ Zapier cloud ──HTTPS──► access-zapier :3024 ──► zappier-edge (meter) ──► middleware :3100
├─ Browsers ────HTTPS──► access-web :3021 /portal
├─ Customer API ─HTTPS──► access-api :3022
├─ Leaf S2S ────HTTPS──► access-leaf :3023 (no billing)
└─ Staff ───────HTTPS──► access-staff :3025 + IAM :3028
Private
├─ NATS JetStream 10.10.10.21–23 (jobs.watch, webhooks.deliver, archive.*)
├─ Core NATS (verae.ts.completed, verae.ts.receipt, billing request-reply)
├─ libSQL (TARGET) ← customers, pricing, usage chain, invoices, balances
├─ Workers job-poller, webhook-deliver, archive-worm×N, tree-node×N
└─ api.veraetime.net / MOCK_VERAE
```
### Critical userflows (must pass QA)
1. **Signup → API key → Statement** (portal via access-web).
2. **Stamp digests (single + batch)** via zap door → edge meter → middleware → Verae → 202 jobIds → credits headers.
3. **Async complete → Zapier REST Hook** (`jobs.watch` → poll → `jobs.events` → `webhooks.deliver`).
4. **CS credit adjust → balance visible** on portal/admin (single prepaid writer).
5. **Sales channel×segment quote → meter charge** matches rate card.
6. **Accounting export** sees payments/usage consistent with libSQL.
7. **Leaf plane** can stamp/authz but **cannot** adjust billing (D004).
---
## 2. Forgejo source elements (implementation clones)
| Concern | Repo | Notes |
|---------|------|-------|
| Billing edge | `marchon/zappier-edge` | Live `/opt/verae/zappier-edge` on 138; also catalog `packages/zappier` |
| Prepaid mailbox (legacy) | `marchon/zappier-account-balance` | `books.json` → migrate into libSQL |
| CS / Sales / Acct | `zappier-customer-service`, `zappier-sales-pricing`, `zappier-accounting-export` | |
| Identity | `zappier-identity` | |
| Middleware | `marchon/verae-middleware` (catalog: verae-zapier-middleware) | |
| Zapier apps | `verae-zapier-app`, `verae-activate` | |
| Access planes | `verae-access-{web,api,leaf,zapier,staff,authz}` | |
| Jobs/archive | `verae-jobs-events`, `verae-archive-*`, `verae-tree-node`, `verae-request-splitter` | |
| Ops | `verae-ops`, `verae-bootstrap`, `verae-fleet`, `verae-keep`, `verae-nats-cluster` | |
| Decisions | `zapier-decisions` | D001–D006 |
| Monorepo mirror | `master-zapier-plan-draft` | docs + packages composition |
---
## 3. Conflict register (resolve before/with build)
| ID | Conflict | Resolution (locked) |
|----|----------|---------------------|
| C1 | D001 `books.json` SoT vs live `billing.db` vs D006 libSQL | **libSQL is SoT**; migrate both books + billing.db schema; account-balance becomes libSQL-backed service or folds into billing API |
| C2 | README “imported, not yet wired” vs STATUS lab up | Treat STATUS + live doors as operational truth; update README composition status in Phase 0 docs |
| C3 | overview `zmw_` Bearer vs composition `x-api-key` | **Public = x-api-key** (D002/composition); fix overview/models |
| C4 | Edge ports `:3000` / `:13000` / live `:3040` (`zap.verae.ai`) | Document env matrix; lab doors proxy to loopback; zap.verae.ai may remain public edge |
| C5 | JetStream `jobs.events` vs ban on JetStream for job-complete | **Different subjects:** Zapier job lifecycle may use JetStream `verae.zapier.jobs.events`; Verae chain notify `verae.ts.*` stays **core NATS** |
| C6 | TimestampRequest “Hello world” data vs digests-only | Digests-only + multihash batch (already fixed on live swagger; sync catalog models) |
| C7 | Dual prepaid writers (edge SQLite + account-balance) | One writer API on libSQL; edge caches only |
| C8 | Per-host SQLite on 3–5 machines | Forbidden for business data; libSQL shared |
| C9 | Go `blockchain.db` in scope creep | Out of scope — do not touch |
| C10 | `ZAPIER_USAGE` vs libSQL usage ledger | Demote to telemetry/export only; never money SoT |
| C11 | Dual middleware `:3100` mock vs `:3130` product upstream | Env matrix; product smoke uses **3130** |
| C12 | D001 file still says books.json SoT | D001 addendum → libSQL (done with this amend) |
| C13 | Auth store edge-local vs libSQL | **v1 edge-local** `zappier.db` |
| C14 | Concurrent hash-chain append across hosts | Single meter writer / serialized append |
| C15 | GETTING-STARTED calls account-balance SoT | Phase 0 doc patch → libSQL |
| C16 | Live `:3010` + `billing.db` dual writers | Cutover runbook: one authoritative adjust path |
| C17 | Litestream multi-writer temptation | Litestream = backup of one primary only |
| C18 | Digests-only vs upload-file stamp docs | Client hasher + digests-only only |
---
## 4. Target data ownership (libSQL)
**Database:** `verae_business` (name TBD) on shared libSQL.
| Domain | Tables / collections | Writer |
|--------|----------------------|--------|
| Customers / identity bind | `customers`, identity map | billing API + identity service |
| Pricing | `tiers`, `price_endpoints`, `channel_rates`, `channel_rate_meta` | admin / sales-pricing |
| Metered ledger | `usage_entries` (+ `prev_hash`/`row_hash` chain) | **single** meter writer |
| Prepaid | balances, credits, payments (from books.json) | billing adjust API |
| Invoices | `invoices`, `invoice_lines` | admin / Stripe jobs |
| Auth (optional split) | `admin_users`, `portal_sessions` | may stay edge-local or move to libSQL `auth` schema |
### 4.1 Launch topology (3–5 machines)
- One libSQL primary (dedicated host or named primary among the 3–5).
- Replicas if any are read-only; writers always target primary URL (`LIBSQL_URL`).
- Backup: filesystem snapshot and/or Litestream of the **primary only**; tested restore before Phase 4.
- Fail closed: metered stamp paths return **503** if BusinessStore write unavailable (no silent local SQLite fallback in production).
- Network: libSQL on private fabric only (same class as NATS `10.10.10.0/24`); not public.
- Schema: `verae_business`; auth stays out for v1 (C13).
**Interface:** `BusinessStore` (TS) with drivers: `LibsqlBusinessStore` (prod), `SqliteFileBusinessStore` (dev/test).
---
## 5. NATS usage matrix
| Subject / stream | Transport | Purpose |
|------------------|-----------|---------|
| `verae.zapier.jobs.watch` | JetStream | Poll Verae status |
| `verae.zapier.jobs.events` | JetStream | Terminal Zapier-pipeline job events |
| `verae.zapier.usage` / `ZAPIER_USAGE` | JetStream optional | **Telemetry/export only; NOT ledger SoT** |
| `verae.zapier.webhooks.deliver` | JetStream | Hook delivery retries |
| `verae.archive.*` | JetStream / fanout | WORM/tree |
| `verae.billing.*` | Core request-reply (+ optional pub) | Statement/balance; eventually thin facade over libSQL |
| `verae.ts.completed`, `verae.ts.receipt` | **Core NATS only** | Chain job-complete notify |
| `verae.access.*`, `verae.identity.*` | Core / as designed | Authz & bind |
---
## 6. Constraint (TOC)
> The system constraint is **shared business SoT cutover (libSQL + single prepaid writer)** because every multi-host userflow (meter, CS credit, sales quote, portal statement, accounting export) blocks on one consistent customer/balance/usage store; JetStream and doors already exist in the lab.
Critical chain: **Conflict freeze → BusinessStore+libSQL → Migrate data → Rewire edge/CS/sales/acct → E2E userflow tests → QA whattochange loop**.
Feeding chains (parallel): digests-only docs sync; NATS subject clarification; access-plane smoke; Forgejo clone inventory.
---
## 7. Phased implementation (PM DAG)
### Phase 0 — Docs & conflict freeze
- [ ] Publish this plan to catalog `docs/plans/` + `zapier-decisions/plans/`
- [ ] Patch conflict C2/C3/C5/C6 in overview + models + README status blurb
- [ ] Architecture review sign-off artifact: `plans/reviews/2026-10-03-arch-review.md`
### Phase 1 — BusinessStore + libSQL skeleton (TDD)
- [ ] Define `BusinessStore` interface (customers, usage.record+verifyChain, rates, invoices, balances)
- [ ] Failing tests against interface
- [ ] `LibsqlBusinessStore` + docker/libsql-server or `:memory:`/file for CI
- [ ] Keep `SqliteFileBusinessStore` wrapping current repos for lab
### Phase 2 — Migrate lab data
- [ ] One-shot migrator: `billing.db` + `books.json` → libSQL
- [ ] Preserve usage hash-chain tip integrity (verify before/after); refuse cutover if verifyChain fails
- [ ] Dual-run: writes libSQL only when `BUSINESS_STORE=libsql`; file fallback **read-only**
- [ ] Account-balance becomes libSQL-backed facade for `verae.billing.*` **or** is stopped after HTTP BusinessStore cutover (pick one in Phase 3)
### Phase 3 — Rewire writers
- [ ] zappier-edge meter → BusinessStore only
- [ ] account-balance service reads/writes libSQL (or deprecate mailbox in favor of HTTP billing API)
- [ ] CS / sales / accounting-export point at same store
- [ ] Remove second writer paths (C7)
### Phase 4 — Multi-host topology
- [ ] Deploy libSQL on dedicated host (or primary of 3–5)
- [ ] Edge/middleware/CS on separate hosts with `LIBSQL_URL`
- [ ] JetStream cluster already present — verify workers only
- [ ] Confirm leaf cannot call billing adjust
### Phase 5 — Userflow E2E + gates
- [ ] Automate flows 1–7 (§1) in `harness/` or `packages/verae-zapier-simulator` extensions
- [ ] Jest unit + integration + gate scripts green
- [ ] Live smoke on 138 with MOCK_VERAE then optional live Verae
### Phase 6 — QA loop (repeat until pass)
- [ ] QA finds bugs → `/whattochange` pack (a/b/c/d + todos)
- [ ] Engineers implement; PM reorders PreReqs
- [ ] Re-run full suite + userflows
- [ ] Exit when all critical userflows pass twice consecutively
---
## 8. Test requirements (“tests for everything”)
| Layer | Must cover |
|-------|------------|
| Unit | BusinessStore ops, hash-chain append/tamper, channel rate resolve, digests-only batch mapping, D004 leaf deny |
| Integration | edge→middleware mock stamp; billing adjust→statement; JetStream watch→event (testcontainers/nats) |
| Contract | OpenAPI digests-only examples; NATS payload schemas |
| E2E userflow | Flows 1–7 against lab compose or NS1 |
| Regression | Existing zappier-edge Jest suite must stay green during driver swap |
| Non-goals | Go timestamping-svc blockchain tests |
---
## 9. Roles (agent orchestration)
| Role | Responsibility | Artifact |
|------|----------------|----------|
| **Architect** | Review plan, expand conflict register, approve SoT | `plans/reviews/*-arch-review.md` |
| **PM** | DAG, task assignment, merge gates, buffer | `plans/reviews/*-pm-board.md` |
| **Engineers** | Implement Phases 1–5 in worktrees | PRs / commits citing this plan |
| **QA** | Break userflows; whattochange packs; retest | `plans/qa/*` + NotImplemented leftovers |
| **Orchestrator (this session)** | Launch agents, merge gates, enforce D006/JetStream rules | STATUS updates |
---
## 10. Acceptance criteria (project buffer)
1. One libSQL URL serves customers/pricing/usage/invoices/balances for all commercial services.
2. No production business writes to per-host `billing.db` / `books.json`.
3. JetStream used only for work/archive streams listed in §5.
4. `verae.ts.completed` / `verae.ts.receipt` remain core NATS.
5. Digests-only multihash batch path green with credit headers.
6. Userflows 1–7 pass twice in a row under QA.
7. Go timestamping DBs untouched.
8. Catalog docs match runtime (conflicts C2–C6, C10–C18 closed or owned).
9. `ZAPIER_USAGE` is non-SoT; no service writes authoritative balances there.
10. Production metered paths fail closed if BusinessStore writer is down.
11. Concurrent append contract test proves single-writer (or serialized) hash-chain safety.
12. Live env matrix documents `:3040` edge, `:3130` product upstream, `:3100` mock, `:3010` facade/retire.
13. D001 file contains D006 addendum; GETTING-STARTED / 01-system no longer call books.json SoT.
---
## 11. Out of scope
- Replacing api.veraetime.net / Go chain storage
- Zapier Platform public directory listing (private push may remain operator-gated)
- Peergos Seaweed fleet redesign (consume existing billing APIs only)
- lan-134 enablement (D005)
- Any JetStream binding for `verae.ts.completed` / `verae.ts.receipt`
- Editing `/var/lib/timestamping-svc/data/blockchain.db` or timestamping-svc test `results.db`
- Multi-primary libSQL or Litestream multi-writer
- Moving portal auth to libSQL in the same cutover as billing (defer)
### Architecture review
See `plans/reviews/2026-10-03-arch-review.md` (**approve-with-changes**; A1–A8 merged herein).
---
## Task checklist (engineer entry)
### Task 1: BusinessStore interface + failing tests
**Files:** Create under zappier-edge `src/db/business-store.ts`, `tests/business-store.contract.test.ts`
**Produces:** Interface used by later tasks
- [ ] Write contract tests for recordUsage/verifyChain/findCustomer/getBalance/adjustBalance
- [ ] Run tests — expect fail
- [ ] Minimal SqliteFile adapter pass
- [ ] Commit citing this plan
### Task 2: LibsqlBusinessStore
- [ ] Add `@libsql/client`
- [ ] Implement same contract
- [ ] CI uses file or embedded libsql
- [ ] Commit
### Task 3: Migrator billing.db + books.json → libSQL
- [ ] Script + tests with fixtures
- [ ] Chain verify pre/post
- [ ] Commit
*(PM expands Tasks 4+ after arch review.)*

383
plans/2026-10-03-system-docs-map.md
View file

@ -1,383 +0,0 @@
# Verae × Zapier System Documentation Map
**Source root:** `/SSD2/sites/zapier.georgelambert.org` on `70.88.205.138` (NS1.GEORGELAMBERT.ORG)
**Public catalog:** https://zapier.georgelambert.org/
**Mapped:** 2026-10-03
**Scope:** Documentation tree only (not live `/opt/verae/zappier-edge` runtime audit)
---
## 1. Docs structure
### Site layout
| Path | Role |
|------|------|
| `/SSD2/sites/zapier.georgelambert.org/` | Published catalog (Caddy) |
| `README.md` | Monorepo entry; composition of zappier + middleware + Zapier app |
| `TODO.md` | Middleware phase gates 0–15 |
| `docs/` | Numbered product/architecture books + modules/models |
| `docs-master/` | Per-module SUMMARY + NATS contracts + MESSAGE-FLOWS |
| `overview/` | System overview book (01–11) + diagrams |
| `packages/` | ~41 independent module packages (README + PDF each) |
| `live/` | Screenshots of public doors |
| `user-docs/` | Customer-facing guides |
| `simulator/` | Static simulator UI copy |
| `sphinx/` | Sphinx HTML build of modules/models |
| `research/` | Vendor Zapier research |
### `docs/` tree
```
docs/
00-sources/ workspace-brief, provenance, getting-started-research,
FUNCTIONS/PLATFORM/MCP refs, veraetime-openapi.yaml, LOGIN/MONGO/RESTART
01-product/ features-a-m, api-gap-analysis, billing-and-keys, zapier-billing-research
02-architecture/
overview, composition, modules-and-nats, nats-subjects, nats-gateway,
archive-nats, fleet, tree-nodes
03-zapier/ auth, operations, README
04-activate/ SETUP-ZAPIER-DEVELOPER
api/ middleware-openapi.yaml
developer/ debugging + modules/
models/ data model sheets (+ models-pdf/)
modules/ per-source-module API sheets (+ modules-pdf/)
plans/ phase-gates.md
sphinx/ RST/MD sources for Sphinx
OPEN.md, WORK-LOG.md
```
### `docs-master/`
- `README.md`, `composition.md`, `modules-and-nats.md`, `archive-nats.md`
- `MESSAGE-FLOWS.md`, `HOST-DEPS.md`, `CATALOG-HOST.md`
- `modules/<pkg>/{SUMMARY,NATS}.md` for each package
### `overview/`
| Doc | Topic |
|-----|--------|
| `01-system.md` | Planes, hard rule (Zapier never NATS) |
| `02-modules-and-repos.md` | Forgejo repos |
| `03-nats-cluster.md` | 3-server JetStream |
| `04-uptime.md` | Replica floors |
| `05-network-failures.md` | Reconnect / silence |
| `06-address-routing.md` | New functions = new subjects |
| `07-external-resources.md` | Chain / storage |
| `08-diagrams.md` | SVG + mermaid |
| `09-expansion-template.md` | verae-nats-process |
| `10-docs-sync.md` | Catalog sync |
| `11-hostnames.md` | Ports, DNS, public doors |
| `NATS.md`, `SUMMARY.md`, `INDEX.md` | Shortcuts |
### `plans/`
- Single plan book: `docs/plans/phase-gates.md` (gates 0–15; mirrors root `TODO.md`)
---
## 2. Key document summaries
### Architecture / overview
**`docs/02-architecture/overview.md`**
- Connect Zapier to Verae without exposing JWTs, forcing Zapier to poll, or coupling billing to the chain API.
- Components: `verae-zapier` (Zapier cloud), middleware HTTPS, NATS JetStream, workers, `api.veraetime.net`.
- Flows: async create → `jobs.watch` → poll → `jobs.events` → webhook; wait path; auth `/auth/me`.
- Flags: `NATS_ENABLED`, `MOCK_VERAE`, `DEBUG_VERAE`.
- Still describes public auth as middleware `zmw_` Bearer (pre-composition framing).
**`docs/02-architecture/composition.md`**
- Target path: Zapier → zappier `/v1/*` (`x-api-key`) → middleware `/zapier/v1/*`.
- Ownership split: billing/portal = zappier; stamp/NATS/hooks = middleware; nouns = verae-zapier.
- Zapier never talks to NATS or live Verae; payment invoices ≠ timestamp receipts.
**`docs/02-architecture/modules-and-nats.md`**
- Master module table (edge, middleware, poller, webhook, splitter, aggregator, WORM, tree-node, fleet, simulator).
- Subjects: `verae.zapier.jobs.{watch,events}`, `webhooks.deliver`, `verae.archive.{put,query,reply.*}`.
- Zapier cloud never on NATS.
**`docs/02-architecture/nats-subjects.md`**
- Streams: `ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, optional `ZAPIER_USAGE`.
- Payload schemas + ack/Nak semantics for poller and webhook deliver.
- Prefer `tokenRef` over JWT in messages; NATS private + auth.
**`docs/02-architecture/archive-nats.md` / `fleet.md` / `tree-nodes.md`**
- Off-chain WORM + Merkle leaf proofs; bloom miss = silence.
- Fleet replica floors (worm/tree min 3); operator console `:3850`.
**`overview/01-system.md`**
- Planes: Zapier, web, API, leaf, staff, authz, commercial edge, account-balance, CS/sales/acct, middleware, NATS, workers, archives, chain, fleet.
- One-sentence request path through meter → stamp → watch/events → billing NATS.
**`overview/11-hostnames.md`**
- NS1 `70.88.205.138`; LXC 510 worker; NATS LXC 511–513 at `10.10.10.21–23`.
- Full service→port→DNS table (portal/api/zap/staff/iam/mw/fleet/git/catalog).
- zappier-edge on `127.0.0.1:13000` (tunnel); note that host `:3000` is “another app”.
- NATS never public; public token edge documented separately in nats-cluster package.
**`overview/NATS.md` / `03-nats-cluster.md`**
- Central 3-server JetStream; new functions = new addresses; Zapier never connects.
### STATUS / decisions / NATS lab
**`packages/zapier-decisions/STATUS.md` (2026-09-12)**
- Working lab on NS1; public doors 8/8 OK; keep+fleet up; test 3-node JetStream on `vmbr1`.
- Product streams replicas=3: `ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE`.
- Clients use `nats://10.10.10.21:4222,...`; host loopback `:4222` unused by clients.
- Open: Zapier Platform push; `MOCK_VERAE=false`; nkeys/mTLS; lan-134; reboot ownership.
- Mock path submit→jobId→wait→receipt proven; live chain seal not proven.
**Decisions (`packages/zapier-decisions/decisions/`)**
- **D001** — Account-balance (`books.json`) is sole prepaid writer; edge `balanceCents` is cache.
- **D002** — `veraeUserId` join key; public credential = zappier API key; JWT = `tokenRef` server-side.
- **D003** — Access planes are only public HTTP doors (`:3021–3025`); edge on loopback `:3000`.
- **D004** — Leaf cannot touch billing (policy allow-list).
- **D005** — HTTP spawn local-first; `lan-134` disabled.
- **D006** (2026-10-03) — Business SoT → **libSQL**; JetStream = work queues only; **core NATS** for `verae.ts.completed` / `verae.ts.receipt`; Go timestamping DBs stay local; lab `billing.db`/`zappier.db`/`books.json` are single-host stand-ins.
**`docs-master/MESSAGE-FLOWS.md`**
- Async stamp, wait, idempotent hash, `includeAttached`, multipart, batch Merkle, simulator replay.
**`docs/plans/phase-gates.md` / `TODO.md`**
- Gates 0–12 largely passed in source/mock; 13–15 (live Verae, private Zapier push, hardening) open.
- Root README still says “9 and 12+ open” in places — stale vs gate logs.
**`docs/OPEN.md`**
- Needs human: Zapier login/push; live Verae gaps (hash lookup etc.); PDF artwork; LTS/share not live; NATS public bind left loopback (superseded by cluster/token edge notes elsewhere).
---
## 3. Packages (`packages/`) — what each does
| Package | Role (from README) |
|---------|-------------------|
| **overview** | High-level system book; Zapier never NATS; address routing |
| **docs-master** | Per-module SUMMARY/NATS; MESSAGE-FLOWS; catalog host move |
| **dev-overview** | Designed vs running NATS fabric map; credentials cheat sheet |
| **ui-docs** | Operator/staff/portal screenshot walkthrough + UI-REVIEW.pdf |
| **zapier-decisions** | STATUS, LOG, TODO, D001–D006, NATS bench reports |
| **zapier-user-docs** | Customer guide: signup → stamp → lookup → tree proofs |
| **zappier** (Forgejo: zappier-edge) | Metered API, portal, admin, Stripe, invoices, rate card |
| **zappier-account-balance** | Prepaid SoT (`books.json`); NATS `verae.billing.*` |
| **zappier-customer-service** | CS credits/review UI `:3011` |
| **zappier-sales-pricing** | Per-customer pricing UI `:3012` |
| **zappier-accounting-export** | QuickBooks IIF / CSV export `:3013` |
| **zappier-identity** | Bind/lookup `veraeUserId` ↔ customer (`:3026`) |
| **verae-zapier-middleware** | `/zapier/v1/*`, NATS workers, mock/live chain client `:3100` |
| **verae-zapier** | Full Zapier Platform app (creates/searches/triggers) |
| **verae-activate** | Tiny “Add Numbers” push-first Zapier app |
| **verae-zapier-simulator** | In-process Zapier+trace console `:3847` |
| **verae-access-authz** | Default-deny `verae.access.authz.check` `:3020` |
| **verae-access-web** | Browser portal plane `:3021` |
| **verae-access-api** | Direct customer API plane `:3022` |
| **verae-access-leaf** | S2S leaf plane `:3023` (no billing) |
| **verae-access-zapier** | Zapier HTTPS front door `:3024` |
| **verae-access-staff** | Staff plane UI `:3025` |
| **verae-staff-iam** | Staff users/roles/audit `:3028` |
| **verae-staff-session** | Shared staff cookie login `:3027` |
| **verae-staff-ui** | Shared review HTML template |
| **verae-request-splitter** | Hash vs archive puts (in-process today) |
| **verae-archive-worm** | Bloom WORM off-chain store |
| **verae-archive-aggregator** | Fan-out query + merge replies |
| **verae-tree-node** | Merkle leaf proof WORM (`kind=tree`) |
| **verae-jobs-events** | Mailbox for `jobs.events` `:3030` (passthrough today) |
| **verae-fleet** | Replica floors, spawn, operator console `:3850` |
| **verae-keep** | Crash restart unless paused (`:3860`/`:3861`) |
| **verae-ops** | Docker/Proxmox/metal install + linking runbooks |
| **verae-bootstrap** | Clone all Forgejo modules + host-deps |
| **verae-deploy** | Tag checkout + npm ci on target hosts |
| **verae-proxmox** | LXC worker (510) provision |
| **verae-nats-cluster** | LXC 511–513 JetStream cluster scripts |
| **verae-nats-accounts** | Leaf allow-list + future nkeys sketch |
| **verae-nats-process** | Template for new addressed workers |
| **verae-uptime** | Off-box HTTPS door probes `:3870` |
| **verae-backup** | Tar snapshots under `/SSD2/backups/verae/` |
---
## 4. Extracted system facts (as documented)
### Services & ports (NS1 lab catalog)
| Service | Port | Bind (docs) | Public door / DNS |
|---------|------|-------------|-------------------|
| Docs catalog | 443 | Caddy | **zapier.georgelambert.org** |
| Forgejo | 443 / SSH **2223** | Caddy | **git.georgelambert.org** |
| Fleet console | 3850 | 0.0.0.0 | fleet.zapier.georgelambert.org |
| Portal (access-web) | 3021 | 0.0.0.0 | portal.zapier… |
| Customer API | 3022 | 0.0.0.0 | api.zapier… |
| Leaf | 3023 | 0.0.0.0 | leaf.zapier… |
| Zapier plane | 3024 | 0.0.0.0 | **zap.zapier.georgelambert.org** |
| Staff plane | 3025 | 0.0.0.0 | staff.zapier… |
| Authz | 3020 | 0.0.0.0 | internal |
| Identity | 3026 | 0.0.0.0 | id.zapier… |
| Staff session | 3027 | 0.0.0.0 | login.zapier… |
| Staff IAM | 3028 | 0.0.0.0 | iam.zapier… |
| Account-balance | 3010 | 0.0.0.0 | books.zapier… |
| CS / Sales / Acct | 3011 / 3012 / 3013 | 0.0.0.0 | cs / sales / acct |
| Middleware | 3100 | 0.0.0.0 | mw.zapier… |
| jobs.events mailbox | 3030 | 0.0.0.0 | events.zapier… |
| zappier-edge (admin) | **13000** (loopback) or **:3000** in older docs | 127.0.0.1 | tunnel only |
| Keep / watch | 3860 / 3861 | 127.0.0.1 | — |
| Simulator | 3847 | optional | sim.zapier… |
| Uptime status | 3870 | 127.0.0.1 | — |
| Workers | 13200–13602 | 127.0.0.1 | — |
| NATS clients | 4222 | private `10.10.10.21–23` | never public product path |
| NATS cluster routes | 6222 | vmbr1 | never public |
| NATS monitor | 8222 | loopback on nodes | — |
| Optional public NATS edge | 4222 / WSS | token-gated on 138 | nats.zapier… (not for Zapier) |
**Also documented elsewhere (live product, not this catalog’s primary story):** `zap.verae.ai` / `zappier-edge.service` on `:3040` under `/opt/verae/zappier-edge` — see Conflicts.
### NATS subjects & streams
**JetStream product streams (replicas=3):**
`ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE`
| Subject | Kind | Purpose |
|---------|------|---------|
| `verae.zapier.jobs.watch` | JS work | Poll chain status |
| `verae.zapier.jobs.events` | JS events | Terminal job outcomes → waiters/webhooks |
| `verae.zapier.webhooks.deliver` | JS work | POST Zapier REST Hook |
| `verae.zapier.usage` | optional JS | Usage export |
| `verae.archive.put` | JS | Off-chain put |
| `verae.archive.query` | broadcast | Bloom query |
| `verae.archive.reply.<id>` | reply | Hit-only |
| `verae.billing.statement.get` | req-reply | Statement |
| `verae.billing.balance.get` | req-reply | Balance |
| `verae.billing.balance.adjust` | req-reply | Credits/reloads |
| `verae.billing.usage.recorded` | pub | Meter |
| `verae.billing.payment.recorded` | pub | Payments |
| `verae.billing.credit.applied` | pub | CS credits |
| `verae.access.authz.check` | req-reply | Allow/deny |
| `verae.access.{web,api,zapier,leaf}.*` | ingress | Plane fronts |
| `verae.identity.bind` / `.lookup` | req-reply | Identity map |
| `verae.ts.completed` / `verae.ts.receipt` | **core NATS** (D006 / standing rule) | Job-complete notify — **not** JetStream |
Client URL (lab): `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222`
### Databases / data stores (documented)
| Store | Owner | Contents |
|-------|-------|----------|
| **`books.json`** | zappier-account-balance (D001) | Prepaid, credits, usage, payments (lab SoT) |
| **`zappier.db`** | zappier-edge | Auth/sessions/admin (also older monolith docs) |
| **`billing.db`** | live edge / D006 note | Customers, usage_entries (+ hash chain), rates, invoices (138 stand-in) |
| **libSQL (target)** | D006 | Shared business SoT across 3–5 hosts |
| Middleware file JSON / `data/` | middleware | Tenants, webhooks, blobs (MVP; Postgres/Redis Phase 15) |
| WORM / tree blobs | archive-worm, tree-node | Off-chain metadata/files/leaf proofs |
| JetStream store | nats-a/b/c | Durable work/events/archive streams |
| Keep/fleet runtime | `~/verae-*-runtime` | Intent + spawned unit state |
| Backups | `/SSD2/backups/verae/` | Tar snapshots (not private keys) |
| Go `blockchain.db` | timestamping-svc | **Do not relocate** (D006) |
### Public doors (D003)
- Browser → `:3021` / portal.zapier…
- Customer API → `:3022` / api.zapier…
- Zapier → `:3024` / zap.zapier…
- Leaf → `:3023`
- Staff → `:3025` (+ iam/login)
- Catalog → zapier.georgelambert.org
- Git → git.georgelambert.org
- Edge admin/Stripe/identity stay loopback (not a public door)
### Identity model (D002 + access planes)
- **Public credential:** zappier API key (`x-api-key`); portal signup/regen.
- **Middleware legacy:** `zmw_` / `zmt_` Bearer (internal / pre-composition).
- **Chain auth:** Verae login → JWT held as `tokenRef` server-side; never in Zapier.
- **Join key:** `veraeUserId` (`vu_…`) via `zappier-identity` bind/lookup.
- **Staff:** `verae-staff-iam` roles (owner, cs, sales, accounting, operator, …) + session cookie.
- **Planes:** zapier / web / api / leaf / staff — each must pass `authz.check` before internal subjects; leaf cannot bill.
### Userflows (documented)
1. **Signup** → portal → API key (Free).
2. **Connect Zap** → HTTPS access-zapier / edge with API key.
3. **Create Timestamp (async)** → meter → middleware → chain → `jobs.watch` → 202 `{jobId}` → poll → `jobs.events` → REST Hook.
4. **Create and Wait** → same, HTTP holds until event or timeout → `pending`+`jobId`.
5. **Idempotent SHA-256** → original seal / `existing: true`.
6. **Attachments / includeAttached** → archive.put + query/merge.
7. **Batch Merkle** → root on chain; leaves on tree-nodes; lookup via `kinds=["tree"]`.
8. **CS credit / portal reload / sales pricing / accounting export** via `verae.billing.*`.
9. **Simulator** replays 1–7 in-process with fault injection.
---
## 5. Conflicts / outdated claims
| Topic | Claim A | Claim B | Notes |
|-------|---------|---------|-------|
| Wiring status | Root README: packages “imported, **not yet wired**” | STATUS: working lab, doors 8/8, NATS cluster live | README composition blurb is stale |
| Public auth | Overview / verae-zapier: Bearer **`zmw_`** | Composition / D003 / billing docs: **`x-api-key`** via zappier | Transition incomplete in docs |
| Prepaid SoT | D001: **`books.json`** via account-balance | Live edge memory / D006: **`billing.db`** + **`zappier.db`**; target **libSQL** | Three eras coexist in writing |
| Edge port | D003 / ops: **`:3000`** loopback | Hostnames: **`:13000`**; “`:3000` is another app” | Also live **`zap.verae.ai:3040`** outside this catalog |
| Public Zap host | Catalog: **zap.zapier.georgelambert.org** | Product memory: **zap.verae.ai** | Dual branding / cutover |
| NATS location | Many docs: loopback `:4222` | STATUS / nats-cluster: **10.10.10.21–23**; loopback unused by clients | Hostnames table still lists loopback NATS |
| Job-complete bus | Architecture: **`verae.zapier.jobs.events` on JetStream** | D006 + standing rule: **`verae.ts.*` on core NATS only** | Naming + persistence conflict; do not JetStream job-complete notify |
| Gate progress | README: phases **9 and 12+ open** | phase-gates gate logs: **9 and 12 pass** | Stale README sentence |
| MOCK / live | STATUS: still **MOCK_VERAE** | Some defaults printed as `MOCK_VERAE=false` | Live seal not proven |
| NATS accounts | D004 sketches accounts | STATUS: cluster **without** nkeys; isolation = vmbr1 | Expected |
| Metering owner | Middleware `PLAN_LIMITS` in older overview | Composition: metering on **zappier** only | Correct direction = composition |
| Hash lookup | Features a–m: mock done | Live OpenAPI: **no** hash route; client 501 | Gap documented |
| docs-master diagram | Still shows “NS1 loopback :4222” for JetStream | Cluster cut-over done | Update modules-and-nats diagram |
---
## 6. Source-of-truth files (prefer these)
| Concern | Prefer |
|---------|--------|
| Current lab status | `packages/zapier-decisions/STATUS.md` |
| Accepted architecture decisions | `packages/zapier-decisions/decisions/D00*.md` (esp. **D003**, **D006**) |
| System planes / hard rules | `overview/01-system.md` |
| Hostnames & ports | `overview/11-hostnames.md` |
| NATS streams (Zapier jobs/webhooks/archive) | `docs/02-architecture/nats-subjects.md` + STATUS streams list |
| Billing subjects | `docs-master/modules-and-nats.md` + `packages/zappier-account-balance/README.md` |
| Message userflows | `docs-master/MESSAGE-FLOWS.md` |
| Composition ownership | `docs/02-architecture/composition.md` |
| Product features a–m | `docs/01-product/features-a-m.md` |
| Ops bring-up | `packages/verae-ops/GETTING-STARTED.md` (+ PDF) |
| Module catalog | `docs-master/README.md` + `packages/*/README.md` |
| Phase implementation order | `docs/plans/phase-gates.md` / `TODO.md` |
| Customer narrative | `packages/zapier-user-docs/` |
| Designed vs running NATS | `packages/dev-overview/` |
**Standing overrides outside this tree (operator rules / memory):**
- Never JetStream for timestamp job-complete (`verae.ts.completed` / `verae.ts.receipt`).
- Live commercial edge on 138: `/opt/verae/zappier-edge`, `billing.db` + `zappier.db`, service `:3040`, public `zap.verae.ai`.
- D006: migrate business data to shared **libSQL** before multi-host split.
---
## 7. Quick system map (ASCII)
```text
[Zapier cloud: verae-zapier / activate]
HTTPS only
|
+------------------+------------------+
| |
access-zapier :3024 (alt) zappier-edge /v1
| |
+---------> middleware :3100 <--------+
| \
chain HTTPS \-- NATS JetStream (jobs/webhooks/archive)
api.veraetime.net |
or MOCK_VERAE workers: poller, webhook,
aggregator, worm×N, tree×N
Browser :3021 API :3022 Leaf :3023 Staff :3025
\ | /
authz.check :3020
|
billing.* ---- account-balance (books → libSQL)
identity.* --- zappier-identity
fleet :3850 / keep :3860 (ops)
catalog: zapier.georgelambert.org
```
---
*End of documentation map. Absolute path of this report: `/tmp/zapier-system-docs-map.md`*

Some files were not shown because too many files have changed in this diff Show more