20 KiB
Verae × Zapier System Documentation Map
Source root: /SSD2/sites/zapier.georgelambert.org on 70.88.205.138 (NS1.GEORGELAMBERT.ORG)
Public catalog: https://zapier.georgelambert.org/
Mapped: 2026-10-03
Scope: Documentation tree only (not live /opt/verae/zappier-edge runtime audit)
1. Docs structure
Site layout
| Path | Role |
|---|---|
/SSD2/sites/zapier.georgelambert.org/ |
Published catalog (Caddy) |
README.md |
Monorepo entry; composition of zappier + middleware + Zapier app |
TODO.md |
Middleware phase gates 0–15 |
docs/ |
Numbered product/architecture books + modules/models |
docs-master/ |
Per-module SUMMARY + NATS contracts + MESSAGE-FLOWS |
overview/ |
System overview book (01–11) + diagrams |
packages/ |
~41 independent module packages (README + PDF each) |
live/ |
Screenshots of public doors |
user-docs/ |
Customer-facing guides |
simulator/ |
Static simulator UI copy |
sphinx/ |
Sphinx HTML build of modules/models |
research/ |
Vendor Zapier research |
docs/ tree
docs/
00-sources/ workspace-brief, provenance, getting-started-research,
FUNCTIONS/PLATFORM/MCP refs, veraetime-openapi.yaml, LOGIN/MONGO/RESTART
01-product/ features-a-m, api-gap-analysis, billing-and-keys, zapier-billing-research
02-architecture/
overview, composition, modules-and-nats, nats-subjects, nats-gateway,
archive-nats, fleet, tree-nodes
03-zapier/ auth, operations, README
04-activate/ SETUP-ZAPIER-DEVELOPER
api/ middleware-openapi.yaml
developer/ debugging + modules/
models/ data model sheets (+ models-pdf/)
modules/ per-source-module API sheets (+ modules-pdf/)
plans/ phase-gates.md
sphinx/ RST/MD sources for Sphinx
OPEN.md, WORK-LOG.md
docs-master/
README.md,composition.md,modules-and-nats.md,archive-nats.mdMESSAGE-FLOWS.md,HOST-DEPS.md,CATALOG-HOST.mdmodules/<pkg>/{SUMMARY,NATS}.mdfor each package
overview/
| Doc | Topic |
|---|---|
01-system.md |
Planes, hard rule (Zapier never NATS) |
02-modules-and-repos.md |
Forgejo repos |
03-nats-cluster.md |
3-server JetStream |
04-uptime.md |
Replica floors |
05-network-failures.md |
Reconnect / silence |
06-address-routing.md |
New functions = new subjects |
07-external-resources.md |
Chain / storage |
08-diagrams.md |
SVG + mermaid |
09-expansion-template.md |
verae-nats-process |
10-docs-sync.md |
Catalog sync |
11-hostnames.md |
Ports, DNS, public doors |
NATS.md, SUMMARY.md, INDEX.md |
Shortcuts |
plans/
- Single plan book:
docs/plans/phase-gates.md(gates 0–15; mirrors rootTODO.md)
2. Key document summaries
Architecture / overview
docs/02-architecture/overview.md
- Connect Zapier to Verae without exposing JWTs, forcing Zapier to poll, or coupling billing to the chain API.
- Components:
verae-zapier(Zapier cloud), middleware HTTPS, NATS JetStream, workers,api.veraetime.net. - Flows: async create →
jobs.watch→ poll →jobs.events→ webhook; wait path; auth/auth/me. - Flags:
NATS_ENABLED,MOCK_VERAE,DEBUG_VERAE. - Still describes public auth as middleware
zmw_Bearer (pre-composition framing).
docs/02-architecture/composition.md
- Target path: Zapier → zappier
/v1/*(x-api-key) → middleware/zapier/v1/*. - Ownership split: billing/portal = zappier; stamp/NATS/hooks = middleware; nouns = verae-zapier.
- Zapier never talks to NATS or live Verae; payment invoices ≠ timestamp receipts.
docs/02-architecture/modules-and-nats.md
- Master module table (edge, middleware, poller, webhook, splitter, aggregator, WORM, tree-node, fleet, simulator).
- Subjects:
verae.zapier.jobs.{watch,events},webhooks.deliver,verae.archive.{put,query,reply.*}. - Zapier cloud never on NATS.
docs/02-architecture/nats-subjects.md
- Streams:
ZAPIER_JOBS,ZAPIER_EVENTS,ZAPIER_WEBHOOKS, optionalZAPIER_USAGE. - Payload schemas + ack/Nak semantics for poller and webhook deliver.
- Prefer
tokenRefover JWT in messages; NATS private + auth.
docs/02-architecture/archive-nats.md / fleet.md / tree-nodes.md
- Off-chain WORM + Merkle leaf proofs; bloom miss = silence.
- Fleet replica floors (worm/tree min 3); operator console
:3850.
overview/01-system.md
- Planes: Zapier, web, API, leaf, staff, authz, commercial edge, account-balance, CS/sales/acct, middleware, NATS, workers, archives, chain, fleet.
- One-sentence request path through meter → stamp → watch/events → billing NATS.
overview/11-hostnames.md
- NS1
70.88.205.138; LXC 510 worker; NATS LXC 511–513 at10.10.10.21–23. - Full service→port→DNS table (portal/api/zap/staff/iam/mw/fleet/git/catalog).
- zappier-edge on
127.0.0.1:13000(tunnel); note that host:3000is “another app”. - NATS never public; public token edge documented separately in nats-cluster package.
overview/NATS.md / 03-nats-cluster.md
- Central 3-server JetStream; new functions = new addresses; Zapier never connects.
STATUS / decisions / NATS lab
packages/zapier-decisions/STATUS.md (2026-09-12)
- Working lab on NS1; public doors 8/8 OK; keep+fleet up; test 3-node JetStream on
vmbr1. - Product streams replicas=3:
ZAPIER_JOBS,ZAPIER_EVENTS,ZAPIER_WEBHOOKS,ZAPIER_USAGE,VERAE_ARCHIVE. - Clients use
nats://10.10.10.21:4222,...; host loopback:4222unused by clients. - Open: Zapier Platform push;
MOCK_VERAE=false; nkeys/mTLS; lan-134; reboot ownership. - Mock path submit→jobId→wait→receipt proven; live chain seal not proven.
Decisions (packages/zapier-decisions/decisions/)
- D001 — Account-balance (
books.json) is sole prepaid writer; edgebalanceCentsis cache. - D002 —
veraeUserIdjoin key; public credential = zappier API key; JWT =tokenRefserver-side. - D003 — Access planes are only public HTTP doors (
:3021–3025); edge on loopback:3000. - D004 — Leaf cannot touch billing (policy allow-list).
- D005 — HTTP spawn local-first;
lan-134disabled. - D006 (2026-10-03) — Business SoT → libSQL; JetStream = work queues only; core NATS for
verae.ts.completed/verae.ts.receipt; Go timestamping DBs stay local; labbilling.db/zappier.db/books.jsonare single-host stand-ins.
docs-master/MESSAGE-FLOWS.md
- Async stamp, wait, idempotent hash,
includeAttached, multipart, batch Merkle, simulator replay.
docs/plans/phase-gates.md / TODO.md
- Gates 0–12 largely passed in source/mock; 13–15 (live Verae, private Zapier push, hardening) open.
- Root README still says “9 and 12+ open” in places — stale vs gate logs.
docs/OPEN.md
- Needs human: Zapier login/push; live Verae gaps (hash lookup etc.); PDF artwork; LTS/share not live; NATS public bind left loopback (superseded by cluster/token edge notes elsewhere).
3. Packages (packages/) — what each does
| Package | Role (from README) |
|---|---|
| overview | High-level system book; Zapier never NATS; address routing |
| docs-master | Per-module SUMMARY/NATS; MESSAGE-FLOWS; catalog host move |
| dev-overview | Designed vs running NATS fabric map; credentials cheat sheet |
| ui-docs | Operator/staff/portal screenshot walkthrough + UI-REVIEW.pdf |
| zapier-decisions | STATUS, LOG, TODO, D001–D006, NATS bench reports |
| zapier-user-docs | Customer guide: signup → stamp → lookup → tree proofs |
| zappier (Forgejo: zappier-edge) | Metered API, portal, admin, Stripe, invoices, rate card |
| zappier-account-balance | Prepaid SoT (books.json); NATS verae.billing.* |
| zappier-customer-service | CS credits/review UI :3011 |
| zappier-sales-pricing | Per-customer pricing UI :3012 |
| zappier-accounting-export | QuickBooks IIF / CSV export :3013 |
| zappier-identity | Bind/lookup veraeUserId ↔ customer (:3026) |
| verae-zapier-middleware | /zapier/v1/*, NATS workers, mock/live chain client :3100 |
| verae-zapier | Full Zapier Platform app (creates/searches/triggers) |
| verae-activate | Tiny “Add Numbers” push-first Zapier app |
| verae-zapier-simulator | In-process Zapier+trace console :3847 |
| verae-access-authz | Default-deny verae.access.authz.check :3020 |
| verae-access-web | Browser portal plane :3021 |
| verae-access-api | Direct customer API plane :3022 |
| verae-access-leaf | S2S leaf plane :3023 (no billing) |
| verae-access-zapier | Zapier HTTPS front door :3024 |
| verae-access-staff | Staff plane UI :3025 |
| verae-staff-iam | Staff users/roles/audit :3028 |
| verae-staff-session | Shared staff cookie login :3027 |
| verae-staff-ui | Shared review HTML template |
| verae-request-splitter | Hash vs archive puts (in-process today) |
| verae-archive-worm | Bloom WORM off-chain store |
| verae-archive-aggregator | Fan-out query + merge replies |
| verae-tree-node | Merkle leaf proof WORM (kind=tree) |
| verae-jobs-events | Mailbox for jobs.events :3030 (passthrough today) |
| verae-fleet | Replica floors, spawn, operator console :3850 |
| verae-keep | Crash restart unless paused (:3860/:3861) |
| verae-ops | Docker/Proxmox/metal install + linking runbooks |
| verae-bootstrap | Clone all Forgejo modules + host-deps |
| verae-deploy | Tag checkout + npm ci on target hosts |
| verae-proxmox | LXC worker (510) provision |
| verae-nats-cluster | LXC 511–513 JetStream cluster scripts |
| verae-nats-accounts | Leaf allow-list + future nkeys sketch |
| verae-nats-process | Template for new addressed workers |
| verae-uptime | Off-box HTTPS door probes :3870 |
| verae-backup | Tar snapshots under /SSD2/backups/verae/ |
4. Extracted system facts (as documented)
Services & ports (NS1 lab catalog)
| Service | Port | Bind (docs) | Public door / DNS |
|---|---|---|---|
| Docs catalog | 443 | Caddy | zapier.georgelambert.org |
| Forgejo | 443 / SSH 2223 | Caddy | git.georgelambert.org |
| Fleet console | 3850 | 0.0.0.0 | fleet.zapier.georgelambert.org |
| Portal (access-web) | 3021 | 0.0.0.0 | portal.zapier… |
| Customer API | 3022 | 0.0.0.0 | api.zapier… |
| Leaf | 3023 | 0.0.0.0 | leaf.zapier… |
| Zapier plane | 3024 | 0.0.0.0 | zap.zapier.georgelambert.org |
| Staff plane | 3025 | 0.0.0.0 | staff.zapier… |
| Authz | 3020 | 0.0.0.0 | internal |
| Identity | 3026 | 0.0.0.0 | id.zapier… |
| Staff session | 3027 | 0.0.0.0 | login.zapier… |
| Staff IAM | 3028 | 0.0.0.0 | iam.zapier… |
| Account-balance | 3010 | 0.0.0.0 | books.zapier… |
| CS / Sales / Acct | 3011 / 3012 / 3013 | 0.0.0.0 | cs / sales / acct |
| Middleware | 3100 | 0.0.0.0 | mw.zapier… |
| jobs.events mailbox | 3030 | 0.0.0.0 | events.zapier… |
| zappier-edge (admin) | 13000 (loopback) or :3000 in older docs | 127.0.0.1 | tunnel only |
| Keep / watch | 3860 / 3861 | 127.0.0.1 | — |
| Simulator | 3847 | optional | sim.zapier… |
| Uptime status | 3870 | 127.0.0.1 | — |
| Workers | 13200–13602 | 127.0.0.1 | — |
| NATS clients | 4222 | private 10.10.10.21–23 |
never public product path |
| NATS cluster routes | 6222 | vmbr1 | never public |
| NATS monitor | 8222 | loopback on nodes | — |
| Optional public NATS edge | 4222 / WSS | token-gated on 138 | nats.zapier… (not for Zapier) |
Also documented elsewhere (live product, not this catalog’s primary story): zap.verae.ai / zappier-edge.service on :3040 under /opt/verae/zappier-edge — see Conflicts.
NATS subjects & streams
JetStream product streams (replicas=3):
ZAPIER_JOBS, ZAPIER_EVENTS, ZAPIER_WEBHOOKS, ZAPIER_USAGE, VERAE_ARCHIVE
| Subject | Kind | Purpose |
|---|---|---|
verae.zapier.jobs.watch |
JS work | Poll chain status |
verae.zapier.jobs.events |
JS events | Terminal job outcomes → waiters/webhooks |
verae.zapier.webhooks.deliver |
JS work | POST Zapier REST Hook |
verae.zapier.usage |
optional JS | Usage export |
verae.archive.put |
JS | Off-chain put |
verae.archive.query |
broadcast | Bloom query |
verae.archive.reply.<id> |
reply | Hit-only |
verae.billing.statement.get |
req-reply | Statement |
verae.billing.balance.get |
req-reply | Balance |
verae.billing.balance.adjust |
req-reply | Credits/reloads |
verae.billing.usage.recorded |
pub | Meter |
verae.billing.payment.recorded |
pub | Payments |
verae.billing.credit.applied |
pub | CS credits |
verae.access.authz.check |
req-reply | Allow/deny |
verae.access.{web,api,zapier,leaf}.* |
ingress | Plane fronts |
verae.identity.bind / .lookup |
req-reply | Identity map |
verae.ts.completed / verae.ts.receipt |
core NATS (D006 / standing rule) | Job-complete notify — not JetStream |
Client URL (lab): nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
Databases / data stores (documented)
| Store | Owner | Contents |
|---|---|---|
books.json |
zappier-account-balance (D001) | Prepaid, credits, usage, payments (lab SoT) |
zappier.db |
zappier-edge | Auth/sessions/admin (also older monolith docs) |
billing.db |
live edge / D006 note | Customers, usage_entries (+ hash chain), rates, invoices (138 stand-in) |
| libSQL (target) | D006 | Shared business SoT across 3–5 hosts |
Middleware file JSON / data/ |
middleware | Tenants, webhooks, blobs (MVP; Postgres/Redis Phase 15) |
| WORM / tree blobs | archive-worm, tree-node | Off-chain metadata/files/leaf proofs |
| JetStream store | nats-a/b/c | Durable work/events/archive streams |
| Keep/fleet runtime | ~/verae-*-runtime |
Intent + spawned unit state |
| Backups | /SSD2/backups/verae/ |
Tar snapshots (not private keys) |
Go blockchain.db |
timestamping-svc | Do not relocate (D006) |
Public doors (D003)
- Browser →
:3021/ portal.zapier… - Customer API →
:3022/ api.zapier… - Zapier →
:3024/ zap.zapier… - Leaf →
:3023 - Staff →
:3025(+ iam/login) - Catalog → zapier.georgelambert.org
- Git → git.georgelambert.org
- Edge admin/Stripe/identity stay loopback (not a public door)
Identity model (D002 + access planes)
- Public credential: zappier API key (
x-api-key); portal signup/regen. - Middleware legacy:
zmw_/zmt_Bearer (internal / pre-composition). - Chain auth: Verae login → JWT held as
tokenRefserver-side; never in Zapier. - Join key:
veraeUserId(vu_…) viazappier-identitybind/lookup. - Staff:
verae-staff-iamroles (owner, cs, sales, accounting, operator, …) + session cookie. - Planes: zapier / web / api / leaf / staff — each must pass
authz.checkbefore internal subjects; leaf cannot bill.
Userflows (documented)
- Signup → portal → API key (Free).
- Connect Zap → HTTPS access-zapier / edge with API key.
- Create Timestamp (async) → meter → middleware → chain →
jobs.watch→ 202{jobId}→ poll →jobs.events→ REST Hook. - Create and Wait → same, HTTP holds until event or timeout →
pending+jobId. - Idempotent SHA-256 → original seal /
existing: true. - Attachments / includeAttached → archive.put + query/merge.
- Batch Merkle → root on chain; leaves on tree-nodes; lookup via
kinds=["tree"]. - CS credit / portal reload / sales pricing / accounting export via
verae.billing.*. - Simulator replays 1–7 in-process with fault injection.
5. Conflicts / outdated claims
| Topic | Claim A | Claim B | Notes |
|---|---|---|---|
| Wiring status | Root README: packages “imported, not yet wired” | STATUS: working lab, doors 8/8, NATS cluster live | README composition blurb is stale |
| Public auth | Overview / verae-zapier: Bearer zmw_ |
Composition / D003 / billing docs: x-api-key via zappier |
Transition incomplete in docs |
| Prepaid SoT | D001: books.json via account-balance |
Live edge memory / D006: billing.db + zappier.db; target libSQL |
Three eras coexist in writing |
| Edge port | D003 / ops: :3000 loopback |
Hostnames: :13000; “:3000 is another app” |
Also live zap.verae.ai:3040 outside this catalog |
| Public Zap host | Catalog: zap.zapier.georgelambert.org | Product memory: zap.verae.ai | Dual branding / cutover |
| NATS location | Many docs: loopback :4222 |
STATUS / nats-cluster: 10.10.10.21–23; loopback unused by clients | Hostnames table still lists loopback NATS |
| Job-complete bus | Architecture: verae.zapier.jobs.events on JetStream |
D006 + standing rule: verae.ts.* on core NATS only |
Naming + persistence conflict; do not JetStream job-complete notify |
| Gate progress | README: phases 9 and 12+ open | phase-gates gate logs: 9 and 12 pass | Stale README sentence |
| MOCK / live | STATUS: still MOCK_VERAE | Some defaults printed as MOCK_VERAE=false |
Live seal not proven |
| NATS accounts | D004 sketches accounts | STATUS: cluster without nkeys; isolation = vmbr1 | Expected |
| Metering owner | Middleware PLAN_LIMITS in older overview |
Composition: metering on zappier only | Correct direction = composition |
| Hash lookup | Features a–m: mock done | Live OpenAPI: no hash route; client 501 | Gap documented |
| docs-master diagram | Still shows “NS1 loopback :4222” for JetStream | Cluster cut-over done | Update modules-and-nats diagram |
6. Source-of-truth files (prefer these)
| Concern | Prefer |
|---|---|
| Current lab status | packages/zapier-decisions/STATUS.md |
| Accepted architecture decisions | packages/zapier-decisions/decisions/D00*.md (esp. D003, D006) |
| System planes / hard rules | overview/01-system.md |
| Hostnames & ports | overview/11-hostnames.md |
| NATS streams (Zapier jobs/webhooks/archive) | docs/02-architecture/nats-subjects.md + STATUS streams list |
| Billing subjects | docs-master/modules-and-nats.md + packages/zappier-account-balance/README.md |
| Message userflows | docs-master/MESSAGE-FLOWS.md |
| Composition ownership | docs/02-architecture/composition.md |
| Product features a–m | docs/01-product/features-a-m.md |
| Ops bring-up | packages/verae-ops/GETTING-STARTED.md (+ PDF) |
| Module catalog | docs-master/README.md + packages/*/README.md |
| Phase implementation order | docs/plans/phase-gates.md / TODO.md |
| Customer narrative | packages/zapier-user-docs/ |
| Designed vs running NATS | packages/dev-overview/ |
Standing overrides outside this tree (operator rules / memory):
- Never JetStream for timestamp job-complete (
verae.ts.completed/verae.ts.receipt). - Live commercial edge on 138:
/opt/verae/zappier-edge,billing.db+zappier.db, service:3040, publiczap.verae.ai. - D006: migrate business data to shared libSQL before multi-host split.
7. Quick system map (ASCII)
[Zapier cloud: verae-zapier / activate]
HTTPS only
|
+------------------+------------------+
| |
access-zapier :3024 (alt) zappier-edge /v1
| |
+---------> middleware :3100 <--------+
| \
chain HTTPS \-- NATS JetStream (jobs/webhooks/archive)
api.veraetime.net |
or MOCK_VERAE workers: poller, webhook,
aggregator, worm×N, tree×N
Browser :3021 API :3022 Leaf :3023 Staff :3025
\ | /
authz.check :3020
|
billing.* ---- account-balance (books → libSQL)
identity.* --- zappier-identity
fleet :3850 / keep :3860 (ops)
catalog: zapier.georgelambert.org
End of documentation map. Absolute path of this report: /tmp/zapier-system-docs-map.md