zapier-decisions/plans/2026-10-03-system-docs-map.md

383 lines
20 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Verae × Zapier System Documentation Map
**Source root:** `/SSD2/sites/zapier.georgelambert.org` on `70.88.205.138` (NS1.GEORGELAMBERT.ORG)
**Public catalog:** https://zapier.georgelambert.org/
**Mapped:** 2026-10-03
**Scope:** Documentation tree only (not live `/opt/verae/zappier-edge` runtime audit)
---
## 1. Docs structure
### Site layout
| Path | Role |
|------|------|
| `/SSD2/sites/zapier.georgelambert.org/` | Published catalog (Caddy) |
| `README.md` | Monorepo entry; composition of zappier + middleware + Zapier app |
| `TODO.md` | Middleware phase gates 0–15 |
| `docs/` | Numbered product/architecture books + modules/models |
| `docs-master/` | Per-module SUMMARY + NATS contracts + MESSAGE-FLOWS |
| `overview/` | System overview book (01–11) + diagrams |
| `packages/` | ~41 independent module packages (README + PDF each) |
| `live/` | Screenshots of public doors |
| `user-docs/` | Customer-facing guides |
| `simulator/` | Static simulator UI copy |
| `sphinx/` | Sphinx HTML build of modules/models |
| `research/` | Vendor Zapier research |
### `docs/` tree
```
docs/
00-sources/ workspace-brief, provenance, getting-started-research,
FUNCTIONS/PLATFORM/MCP refs, veraetime-openapi.yaml, LOGIN/MONGO/RESTART
01-product/ features-a-m, api-gap-analysis, billing-and-keys, zapier-billing-research
02-architecture/
overview, composition, modules-and-nats, nats-subjects, nats-gateway,
archive-nats, fleet, tree-nodes
03-zapier/ auth, operations, README
04-activate/ SETUP-ZAPIER-DEVELOPER
api/ middleware-openapi.yaml
developer/ debugging + modules/
models/ data model sheets (+ models-pdf/)
modules/ per-source-module API sheets (+ modules-pdf/)
plans/ phase-gates.md
sphinx/ RST/MD sources for Sphinx
OPEN.md, WORK-LOG.md
```
### `docs-master/`
- `README.md`, `composition.md`, `modules-and-nats.md`, `archive-nats.md`
- `MESSAGE-FLOWS.md`, `HOST-DEPS.md`, `CATALOG-HOST.md`
- `modules/<pkg>/{SUMMARY,NATS}.md` for each package
### `overview/`
| Doc | Topic |
|-----|--------|
| `01-system.md` | Planes, hard rule (Zapier never NATS) |
| `02-modules-and-repos.md` | Forgejo repos |
| `03-nats-cluster.md` | 3-server JetStream |
| `04-uptime.md` | Replica floors |
| `05-network-failures.md` | Reconnect / silence |
| `06-address-routing.md` | New functions = new subjects |
| `07-external-resources.md` | Chain / storage |
| `08-diagrams.md` | SVG + mermaid |
| `09-expansion-template.md` | verae-nats-process |
| `10-docs-sync.md` | Catalog sync |
| `11-hostnames.md` | Ports, DNS, public doors |
| `NATS.md`, `SUMMARY.md`, `INDEX.md` | Shortcuts |
### `plans/`
- Single plan book: `docs/plans/phase-gates.md` (gates 0–15; mirrors root `TODO.md`)
---
## 2. Key document summaries
### Architecture / overview
**`docs/02-architecture/overview.md`**
- Connect Zapier to Verae without exposing JWTs, forcing Zapier to poll, or coupling billing to the chain API.
- Components: `verae-zapier` (Zapier cloud), middleware HTTPS, NATS JetStream, workers, `api.veraetime.net`.
- Flows: async create → `jobs.watch` → poll → `jobs.events` → webhook; wait path; auth `/auth/me`.
- Flags: `NATS_ENABLED`, `MOCK_VERAE`, `DEBUG_VERAE`.
- Still describes public auth as middleware `zmw_` Bearer (pre-composition framing).
**`docs/02-architecture/composition.md`**
- Target path: Zapier → zappier `/v1/*` (`x-api-key`) → middleware `/zapier/v1/*`.
- Ownership split: billing/portal = zappier; stamp/NATS/hooks = middleware; nouns = verae-zapier.
- Zapier never talks to NATS or live Verae; payment invoices ≠ timestamp receipts.
**`docs/02-architecture/modules-and-nats.md`**
- Master module table (edge, middleware, poller, webhook, splitter, aggregator, WORM, tree-node, fleet, simulator).
- Subjects: `verae.zapier.jobs.{watch,events}`, `webhooks.deliver`, `verae.archive.{put,query,reply.*}`.
- Zapier cloud never on NATS.
**`docs/02-architecture/nats-subjects.md`**
- Streams: `ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, optional `ZAPIER_USAGE`.
- Payload schemas + ack/Nak semantics for poller and webhook deliver.
- Prefer `tokenRef` over JWT in messages; NATS private + auth.
**`docs/02-architecture/archive-nats.md` / `fleet.md` / `tree-nodes.md`**
- Off-chain WORM + Merkle leaf proofs; bloom miss = silence.
- Fleet replica floors (worm/tree min 3); operator console `:3850`.
**`overview/01-system.md`**
- Planes: Zapier, web, API, leaf, staff, authz, commercial edge, account-balance, CS/sales/acct, middleware, NATS, workers, archives, chain, fleet.
- One-sentence request path through meter → stamp → watch/events → billing NATS.
**`overview/11-hostnames.md`**
- NS1 `70.88.205.138`; LXC 510 worker; NATS LXC 511–513 at `10.10.10.21–23`.
- Full service→port→DNS table (portal/api/zap/staff/iam/mw/fleet/git/catalog).
- zappier-edge on `127.0.0.1:13000` (tunnel); note that host `:3000` is “another app”.
- NATS never public; public token edge documented separately in nats-cluster package.
**`overview/NATS.md` / `03-nats-cluster.md`**
- Central 3-server JetStream; new functions = new addresses; Zapier never connects.
### STATUS / decisions / NATS lab
**`packages/zapier-decisions/STATUS.md` (2026-09-12)**
- Working lab on NS1; public doors 8/8 OK; keep+fleet up; test 3-node JetStream on `vmbr1`.
- Product streams replicas=3: `ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE`.
- Clients use `nats://10.10.10.21:4222,...`; host loopback `:4222` unused by clients.
- Open: Zapier Platform push; `MOCK_VERAE=false`; nkeys/mTLS; lan-134; reboot ownership.
- Mock path submit→jobId→wait→receipt proven; live chain seal not proven.
**Decisions (`packages/zapier-decisions/decisions/`)**
- **D001** — Account-balance (`books.json`) is sole prepaid writer; edge `balanceCents` is cache.
- **D002** — `veraeUserId` join key; public credential = zappier API key; JWT = `tokenRef` server-side.
- **D003** — Access planes are only public HTTP doors (`:3021–3025`); edge on loopback `:3000`.
- **D004** — Leaf cannot touch billing (policy allow-list).
- **D005** — HTTP spawn local-first; `lan-134` disabled.
- **D006** (2026-10-03) — Business SoT → **libSQL**; JetStream = work queues only; **core NATS** for `verae.ts.completed` / `verae.ts.receipt`; Go timestamping DBs stay local; lab `billing.db`/`zappier.db`/`books.json` are single-host stand-ins.
**`docs-master/MESSAGE-FLOWS.md`**
- Async stamp, wait, idempotent hash, `includeAttached`, multipart, batch Merkle, simulator replay.
**`docs/plans/phase-gates.md` / `TODO.md`**
- Gates 0–12 largely passed in source/mock; 13–15 (live Verae, private Zapier push, hardening) open.
- Root README still says “9 and 12+ open” in places — stale vs gate logs.
**`docs/OPEN.md`**
- Needs human: Zapier login/push; live Verae gaps (hash lookup etc.); PDF artwork; LTS/share not live; NATS public bind left loopback (superseded by cluster/token edge notes elsewhere).
---
## 3. Packages (`packages/`) — what each does
| Package | Role (from README) |
|---------|-------------------|
| **overview** | High-level system book; Zapier never NATS; address routing |
| **docs-master** | Per-module SUMMARY/NATS; MESSAGE-FLOWS; catalog host move |
| **dev-overview** | Designed vs running NATS fabric map; credentials cheat sheet |
| **ui-docs** | Operator/staff/portal screenshot walkthrough + UI-REVIEW.pdf |
| **zapier-decisions** | STATUS, LOG, TODO, D001–D006, NATS bench reports |
| **zapier-user-docs** | Customer guide: signup → stamp → lookup → tree proofs |
| **zappier** (Forgejo: zappier-edge) | Metered API, portal, admin, Stripe, invoices, rate card |
| **zappier-account-balance** | Prepaid SoT (`books.json`); NATS `verae.billing.*` |
| **zappier-customer-service** | CS credits/review UI `:3011` |
| **zappier-sales-pricing** | Per-customer pricing UI `:3012` |
| **zappier-accounting-export** | QuickBooks IIF / CSV export `:3013` |
| **zappier-identity** | Bind/lookup `veraeUserId` ↔ customer (`:3026`) |
| **verae-zapier-middleware** | `/zapier/v1/*`, NATS workers, mock/live chain client `:3100` |
| **verae-zapier** | Full Zapier Platform app (creates/searches/triggers) |
| **verae-activate** | Tiny “Add Numbers” push-first Zapier app |
| **verae-zapier-simulator** | In-process Zapier+trace console `:3847` |
| **verae-access-authz** | Default-deny `verae.access.authz.check` `:3020` |
| **verae-access-web** | Browser portal plane `:3021` |
| **verae-access-api** | Direct customer API plane `:3022` |
| **verae-access-leaf** | S2S leaf plane `:3023` (no billing) |
| **verae-access-zapier** | Zapier HTTPS front door `:3024` |
| **verae-access-staff** | Staff plane UI `:3025` |
| **verae-staff-iam** | Staff users/roles/audit `:3028` |
| **verae-staff-session** | Shared staff cookie login `:3027` |
| **verae-staff-ui** | Shared review HTML template |
| **verae-request-splitter** | Hash vs archive puts (in-process today) |
| **verae-archive-worm** | Bloom WORM off-chain store |
| **verae-archive-aggregator** | Fan-out query + merge replies |
| **verae-tree-node** | Merkle leaf proof WORM (`kind=tree`) |
| **verae-jobs-events** | Mailbox for `jobs.events` `:3030` (passthrough today) |
| **verae-fleet** | Replica floors, spawn, operator console `:3850` |
| **verae-keep** | Crash restart unless paused (`:3860`/`:3861`) |
| **verae-ops** | Docker/Proxmox/metal install + linking runbooks |
| **verae-bootstrap** | Clone all Forgejo modules + host-deps |
| **verae-deploy** | Tag checkout + npm ci on target hosts |
| **verae-proxmox** | LXC worker (510) provision |
| **verae-nats-cluster** | LXC 511–513 JetStream cluster scripts |
| **verae-nats-accounts** | Leaf allow-list + future nkeys sketch |
| **verae-nats-process** | Template for new addressed workers |
| **verae-uptime** | Off-box HTTPS door probes `:3870` |
| **verae-backup** | Tar snapshots under `/SSD2/backups/verae/` |
---
## 4. Extracted system facts (as documented)
### Services & ports (NS1 lab catalog)
| Service | Port | Bind (docs) | Public door / DNS |
|---------|------|-------------|-------------------|
| Docs catalog | 443 | Caddy | **zapier.georgelambert.org** |
| Forgejo | 443 / SSH **2223** | Caddy | **git.georgelambert.org** |
| Fleet console | 3850 | 0.0.0.0 | fleet.zapier.georgelambert.org |
| Portal (access-web) | 3021 | 0.0.0.0 | portal.zapier… |
| Customer API | 3022 | 0.0.0.0 | api.zapier… |
| Leaf | 3023 | 0.0.0.0 | leaf.zapier… |
| Zapier plane | 3024 | 0.0.0.0 | **zap.zapier.georgelambert.org** |
| Staff plane | 3025 | 0.0.0.0 | staff.zapier… |
| Authz | 3020 | 0.0.0.0 | internal |
| Identity | 3026 | 0.0.0.0 | id.zapier… |
| Staff session | 3027 | 0.0.0.0 | login.zapier… |
| Staff IAM | 3028 | 0.0.0.0 | iam.zapier… |
| Account-balance | 3010 | 0.0.0.0 | books.zapier… |
| CS / Sales / Acct | 3011 / 3012 / 3013 | 0.0.0.0 | cs / sales / acct |
| Middleware | 3100 | 0.0.0.0 | mw.zapier… |
| jobs.events mailbox | 3030 | 0.0.0.0 | events.zapier… |
| zappier-edge (admin) | **13000** (loopback) or **:3000** in older docs | 127.0.0.1 | tunnel only |
| Keep / watch | 3860 / 3861 | 127.0.0.1 | — |
| Simulator | 3847 | optional | sim.zapier… |
| Uptime status | 3870 | 127.0.0.1 | — |
| Workers | 13200–13602 | 127.0.0.1 | — |
| NATS clients | 4222 | private `10.10.10.21–23` | never public product path |
| NATS cluster routes | 6222 | vmbr1 | never public |
| NATS monitor | 8222 | loopback on nodes | — |
| Optional public NATS edge | 4222 / WSS | token-gated on 138 | nats.zapier… (not for Zapier) |
**Also documented elsewhere (live product, not this catalog’s primary story):** `zap.verae.ai` / `zappier-edge.service` on `:3040` under `/opt/verae/zappier-edge` — see Conflicts.
### NATS subjects & streams
**JetStream product streams (replicas=3):**
`ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE`
| Subject | Kind | Purpose |
|---------|------|---------|
| `verae.zapier.jobs.watch` | JS work | Poll chain status |
| `verae.zapier.jobs.events` | JS events | Terminal job outcomes → waiters/webhooks |
| `verae.zapier.webhooks.deliver` | JS work | POST Zapier REST Hook |
| `verae.zapier.usage` | optional JS | Usage export |
| `verae.archive.put` | JS | Off-chain put |
| `verae.archive.query` | broadcast | Bloom query |
| `verae.archive.reply.<id>` | reply | Hit-only |
| `verae.billing.statement.get` | req-reply | Statement |
| `verae.billing.balance.get` | req-reply | Balance |
| `verae.billing.balance.adjust` | req-reply | Credits/reloads |
| `verae.billing.usage.recorded` | pub | Meter |
| `verae.billing.payment.recorded` | pub | Payments |
| `verae.billing.credit.applied` | pub | CS credits |
| `verae.access.authz.check` | req-reply | Allow/deny |
| `verae.access.{web,api,zapier,leaf}.*` | ingress | Plane fronts |
| `verae.identity.bind` / `.lookup` | req-reply | Identity map |
| `verae.ts.completed` / `verae.ts.receipt` | **core NATS** (D006 / standing rule) | Job-complete notify — **not** JetStream |
Client URL (lab): `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222`
### Databases / data stores (documented)
| Store | Owner | Contents |
|-------|-------|----------|
| **`books.json`** | zappier-account-balance (D001) | Prepaid, credits, usage, payments (lab SoT) |
| **`zappier.db`** | zappier-edge | Auth/sessions/admin (also older monolith docs) |
| **`billing.db`** | live edge / D006 note | Customers, usage_entries (+ hash chain), rates, invoices (138 stand-in) |
| **libSQL (target)** | D006 | Shared business SoT across 3–5 hosts |
| Middleware file JSON / `data/` | middleware | Tenants, webhooks, blobs (MVP; Postgres/Redis Phase 15) |
| WORM / tree blobs | archive-worm, tree-node | Off-chain metadata/files/leaf proofs |
| JetStream store | nats-a/b/c | Durable work/events/archive streams |
| Keep/fleet runtime | `~/verae-*-runtime` | Intent + spawned unit state |
| Backups | `/SSD2/backups/verae/` | Tar snapshots (not private keys) |
| Go `blockchain.db` | timestamping-svc | **Do not relocate** (D006) |
### Public doors (D003)
- Browser → `:3021` / portal.zapier…
- Customer API → `:3022` / api.zapier…
- Zapier → `:3024` / zap.zapier…
- Leaf → `:3023`
- Staff → `:3025` (+ iam/login)
- Catalog → zapier.georgelambert.org
- Git → git.georgelambert.org
- Edge admin/Stripe/identity stay loopback (not a public door)
### Identity model (D002 + access planes)
- **Public credential:** zappier API key (`x-api-key`); portal signup/regen.
- **Middleware legacy:** `zmw_` / `zmt_` Bearer (internal / pre-composition).
- **Chain auth:** Verae login → JWT held as `tokenRef` server-side; never in Zapier.
- **Join key:** `veraeUserId` (`vu_…`) via `zappier-identity` bind/lookup.
- **Staff:** `verae-staff-iam` roles (owner, cs, sales, accounting, operator, …) + session cookie.
- **Planes:** zapier / web / api / leaf / staff — each must pass `authz.check` before internal subjects; leaf cannot bill.
### Userflows (documented)
1. **Signup** → portal → API key (Free).
2. **Connect Zap** → HTTPS access-zapier / edge with API key.
3. **Create Timestamp (async)** → meter → middleware → chain → `jobs.watch` → 202 `{jobId}` → poll → `jobs.events` → REST Hook.
4. **Create and Wait** → same, HTTP holds until event or timeout → `pending`+`jobId`.
5. **Idempotent SHA-256** → original seal / `existing: true`.
6. **Attachments / includeAttached** → archive.put + query/merge.
7. **Batch Merkle** → root on chain; leaves on tree-nodes; lookup via `kinds=["tree"]`.
8. **CS credit / portal reload / sales pricing / accounting export** via `verae.billing.*`.
9. **Simulator** replays 1–7 in-process with fault injection.
---
## 5. Conflicts / outdated claims
| Topic | Claim A | Claim B | Notes |
|-------|---------|---------|-------|
| Wiring status | Root README: packages “imported, **not yet wired**” | STATUS: working lab, doors 8/8, NATS cluster live | README composition blurb is stale |
| Public auth | Overview / verae-zapier: Bearer **`zmw_`** | Composition / D003 / billing docs: **`x-api-key`** via zappier | Transition incomplete in docs |
| Prepaid SoT | D001: **`books.json`** via account-balance | Live edge memory / D006: **`billing.db`** + **`zappier.db`**; target **libSQL** | Three eras coexist in writing |
| Edge port | D003 / ops: **`:3000`** loopback | Hostnames: **`:13000`**; “`:3000` is another app” | Also live **`zap.verae.ai:3040`** outside this catalog |
| Public Zap host | Catalog: **zap.zapier.georgelambert.org** | Product memory: **zap.verae.ai** | Dual branding / cutover |
| NATS location | Many docs: loopback `:4222` | STATUS / nats-cluster: **10.10.10.21–23**; loopback unused by clients | Hostnames table still lists loopback NATS |
| Job-complete bus | Architecture: **`verae.zapier.jobs.events` on JetStream** | D006 + standing rule: **`verae.ts.*` on core NATS only** | Naming + persistence conflict; do not JetStream job-complete notify |
| Gate progress | README: phases **9 and 12+ open** | phase-gates gate logs: **9 and 12 pass** | Stale README sentence |
| MOCK / live | STATUS: still **MOCK_VERAE** | Some defaults printed as `MOCK_VERAE=false` | Live seal not proven |
| NATS accounts | D004 sketches accounts | STATUS: cluster **without** nkeys; isolation = vmbr1 | Expected |
| Metering owner | Middleware `PLAN_LIMITS` in older overview | Composition: metering on **zappier** only | Correct direction = composition |
| Hash lookup | Features a–m: mock done | Live OpenAPI: **no** hash route; client 501 | Gap documented |
| docs-master diagram | Still shows “NS1 loopback :4222” for JetStream | Cluster cut-over done | Update modules-and-nats diagram |
---
## 6. Source-of-truth files (prefer these)
| Concern | Prefer |
|---------|--------|
| Current lab status | `packages/zapier-decisions/STATUS.md` |
| Accepted architecture decisions | `packages/zapier-decisions/decisions/D00*.md` (esp. **D003**, **D006**) |
| System planes / hard rules | `overview/01-system.md` |
| Hostnames & ports | `overview/11-hostnames.md` |
| NATS streams (Zapier jobs/webhooks/archive) | `docs/02-architecture/nats-subjects.md` + STATUS streams list |
| Billing subjects | `docs-master/modules-and-nats.md` + `packages/zappier-account-balance/README.md` |
| Message userflows | `docs-master/MESSAGE-FLOWS.md` |
| Composition ownership | `docs/02-architecture/composition.md` |
| Product features a–m | `docs/01-product/features-a-m.md` |
| Ops bring-up | `packages/verae-ops/GETTING-STARTED.md` (+ PDF) |
| Module catalog | `docs-master/README.md` + `packages/*/README.md` |
| Phase implementation order | `docs/plans/phase-gates.md` / `TODO.md` |
| Customer narrative | `packages/zapier-user-docs/` |
| Designed vs running NATS | `packages/dev-overview/` |
**Standing overrides outside this tree (operator rules / memory):**
- Never JetStream for timestamp job-complete (`verae.ts.completed` / `verae.ts.receipt`).
- Live commercial edge on 138: `/opt/verae/zappier-edge`, `billing.db` + `zappier.db`, service `:3040`, public `zap.verae.ai`.
- D006: migrate business data to shared **libSQL** before multi-host split.
---
## 7. Quick system map (ASCII)
```text
[Zapier cloud: verae-zapier / activate]
HTTPS only
|
+------------------+------------------+
| |
access-zapier :3024 (alt) zappier-edge /v1
| |
+---------> middleware :3100 <--------+
| \
chain HTTPS \-- NATS JetStream (jobs/webhooks/archive)
api.veraetime.net |
or MOCK_VERAE workers: poller, webhook,
aggregator, worm×N, tree×N
Browser :3021 API :3022 Leaf :3023 Staff :3025
\ | /
authz.check :3020
|
billing.* ---- account-balance (books → libSQL)
identity.* --- zappier-identity
fleet :3850 / keep :3860 (ops)
catalog: zapier.georgelambert.org
```
---
*End of documentation map. Absolute path of this report: `/tmp/zapier-system-docs-map.md`*