# Verae × Zapier Multi-Host Architecture & Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (or executing-plans) task-by-task. Steps use checkbox (`- [ ]`) syntax. **Goal:** Run the Verae × Zapier commercial stack across **3–5 machines** with **libSQL** as the system of record for all business/billing/customer/pricing/accounting data, **JetStream only for work queues**, and proven end-to-end userflows under continuous QA. **Architecture:** Public HTTPS doors → zappier-edge (meter) → middleware (stamp) → `api.veraetime.net`; private NATS for workers; shared libSQL for money & CRM; Go timestamping DBs stay local to timestamping-svc. **Tech Stack:** Node (edge, middleware, access planes, CS/sales/acct), NATS JetStream (3-node lab → production), libSQL / go-libsql or `@libsql/client`, better-sqlite3 only as local/dev adapter, Litestream optional for libSQL file primary backup, Jest + gate scripts + E2E userflow harness. **Spec / SoT docs:** - Catalog: https://zapier.georgelambert.org/ - Map: `/tmp/zapier-system-docs-map.md` - Decisions: D001–D006 (`packages/zapier-decisions/decisions/`) - Composition: `docs/02-architecture/composition.md` - NATS: `docs/02-architecture/nats-subjects.md`, `overview/03-nats-cluster.md` - Ops: `packages/verae-ops/GETTING-STARTED.md` - Digests-only stamps: mcp.verae.ai + D006 note + live swagger **Forgejo org:** https://git.georgelambert.org/marchon/ (modules listed in §2) ## Global Constraints - **D006:** Business/billing/customers/pricing/accounting/usage ledger → **libSQL only** (shared). - **JetStream:** work queues as necessary (`ZAPIER_JOBS`, `ZAPIER_WEBHOOKS`, archive put/query). Not money SoT. - **Core NATS only** for `verae.ts.completed` / `verae.ts.receipt` (no JetStream on that path). - **D001 refined:** single prepaid **writer path**; durable store is libSQL (not per-host `books.json` / `billing.db`). - **D002:** public credential = zappier `x-api-key`; `veraeUserId` join; JWT = `tokenRef` server-side. - **D003:** only access planes are public HTTP; edge loopback (except intentional public `zap.verae.ai` edge). - **D004:** leaf must not touch billing. - **Digests only** for stamp submit (`POST /api/batch/timestamp` multihash; MCP shape). - **Never** modify Go timestamping-svc SQLite (`blockchain.db`, etc.) as part of this plan. - **WAL** (or libSQL equivalent durability) on all SQL stores. - Credits language in product copy; ledger may keep `*_cents` column names as credit units. - Deploy never rsyncs live `*.db*` over production data. - **ZAPIER_USAGE** / `verae.zapier.usage` is telemetry or export only — never usage/money SoT. - Litestream (if used) backs up a **single** libSQL primary; never multi-writer fabric. - **Single meter/prepaid writer** process owns `usage_entries` append + balance adjust; other hosts call that API. - Dual-run: libSQL primary for writes; file/SQLite fallback is **read-only** (or kill-switched). - Auth (`admin_users`, `portal_sessions`): v1 remains edge-local `zappier.db`; not in `verae_business`. - Product edge bind on 138: `127.0.0.1:3040` (`zap.verae.ai`); `ZAPPIER_UPSTREAM` defaults to `http://127.0.0.1:3130` for live lab smoke (not `:3100` mock unless intentional). - Catalog docs that call account-balance “prepaid SoT” are stale after D006; Phase 0 patches them. --- ## 1. System map (from catalog + live lab) ```text Internet ├─ Zapier cloud ──HTTPS──► access-zapier :3024 ──► zappier-edge (meter) ──► middleware :3100 ├─ Browsers ────HTTPS──► access-web :3021 /portal ├─ Customer API ─HTTPS──► access-api :3022 ├─ Leaf S2S ────HTTPS──► access-leaf :3023 (no billing) └─ Staff ───────HTTPS──► access-staff :3025 + IAM :3028 Private ├─ NATS JetStream 10.10.10.21–23 (jobs.watch, webhooks.deliver, archive.*) ├─ Core NATS (verae.ts.completed, verae.ts.receipt, billing request-reply) ├─ libSQL (TARGET) ← customers, pricing, usage chain, invoices, balances ├─ Workers job-poller, webhook-deliver, archive-worm×N, tree-node×N └─ api.veraetime.net / MOCK_VERAE ``` ### Critical userflows (must pass QA) 1. **Signup → API key → Statement** (portal via access-web). 2. **Stamp digests (single + batch)** via zap door → edge meter → middleware → Verae → 202 jobIds → credits headers. 3. **Async complete → Zapier REST Hook** (`jobs.watch` → poll → `jobs.events` → `webhooks.deliver`). 4. **CS credit adjust → balance visible** on portal/admin (single prepaid writer). 5. **Sales channel×segment quote → meter charge** matches rate card. 6. **Accounting export** sees payments/usage consistent with libSQL. 7. **Leaf plane** can stamp/authz but **cannot** adjust billing (D004). --- ## 2. Forgejo source elements (implementation clones) | Concern | Repo | Notes | |---------|------|-------| | Billing edge | `marchon/zappier-edge` | Live `/opt/verae/zappier-edge` on 138; also catalog `packages/zappier` | | Prepaid mailbox (legacy) | `marchon/zappier-account-balance` | `books.json` → migrate into libSQL | | CS / Sales / Acct | `zappier-customer-service`, `zappier-sales-pricing`, `zappier-accounting-export` | | | Identity | `zappier-identity` | | | Middleware | `marchon/verae-middleware` (catalog: verae-zapier-middleware) | | | Zapier apps | `verae-zapier-app`, `verae-activate` | | | Access planes | `verae-access-{web,api,leaf,zapier,staff,authz}` | | | Jobs/archive | `verae-jobs-events`, `verae-archive-*`, `verae-tree-node`, `verae-request-splitter` | | | Ops | `verae-ops`, `verae-bootstrap`, `verae-fleet`, `verae-keep`, `verae-nats-cluster` | | | Decisions | `zapier-decisions` | D001–D006 | | Monorepo mirror | `master-zapier-plan-draft` | docs + packages composition | --- ## 3. Conflict register (resolve before/with build) | ID | Conflict | Resolution (locked) | |----|----------|---------------------| | C1 | D001 `books.json` SoT vs live `billing.db` vs D006 libSQL | **libSQL is SoT**; migrate both books + billing.db schema; account-balance becomes libSQL-backed service or folds into billing API | | C2 | README “imported, not yet wired” vs STATUS lab up | Treat STATUS + live doors as operational truth; update README composition status in Phase 0 docs | | C3 | overview `zmw_` Bearer vs composition `x-api-key` | **Public = x-api-key** (D002/composition); fix overview/models | | C4 | Edge ports `:3000` / `:13000` / live `:3040` (`zap.verae.ai`) | Document env matrix; lab doors proxy to loopback; zap.verae.ai may remain public edge | | C5 | JetStream `jobs.events` vs ban on JetStream for job-complete | **Different subjects:** Zapier job lifecycle may use JetStream `verae.zapier.jobs.events`; Verae chain notify `verae.ts.*` stays **core NATS** | | C6 | TimestampRequest “Hello world” data vs digests-only | Digests-only + multihash batch (already fixed on live swagger; sync catalog models) | | C7 | Dual prepaid writers (edge SQLite + account-balance) | One writer API on libSQL; edge caches only | | C8 | Per-host SQLite on 3–5 machines | Forbidden for business data; libSQL shared | | C9 | Go `blockchain.db` in scope creep | Out of scope — do not touch | | C10 | `ZAPIER_USAGE` vs libSQL usage ledger | Demote to telemetry/export only; never money SoT | | C11 | Dual middleware `:3100` mock vs `:3130` product upstream | Env matrix; product smoke uses **3130** | | C12 | D001 file still says books.json SoT | D001 addendum → libSQL (done with this amend) | | C13 | Auth store edge-local vs libSQL | **v1 edge-local** `zappier.db` | | C14 | Concurrent hash-chain append across hosts | Single meter writer / serialized append | | C15 | GETTING-STARTED calls account-balance SoT | Phase 0 doc patch → libSQL | | C16 | Live `:3010` + `billing.db` dual writers | Cutover runbook: one authoritative adjust path | | C17 | Litestream multi-writer temptation | Litestream = backup of one primary only | | C18 | Digests-only vs upload-file stamp docs | Client hasher + digests-only only | --- ## 4. Target data ownership (libSQL) **Database:** `verae_business` (name TBD) on shared libSQL. | Domain | Tables / collections | Writer | |--------|----------------------|--------| | Customers / identity bind | `customers`, identity map | billing API + identity service | | Pricing | `tiers`, `price_endpoints`, `channel_rates`, `channel_rate_meta` | admin / sales-pricing | | Metered ledger | `usage_entries` (+ `prev_hash`/`row_hash` chain) | **single** meter writer | | Prepaid | balances, credits, payments (from books.json) | billing adjust API | | Invoices | `invoices`, `invoice_lines` | admin / Stripe jobs | | Auth (optional split) | `admin_users`, `portal_sessions` | may stay edge-local or move to libSQL `auth` schema | ### 4.1 Launch topology (3–5 machines) - One libSQL primary (dedicated host or named primary among the 3–5). - Replicas if any are read-only; writers always target primary URL (`LIBSQL_URL`). - Backup: filesystem snapshot and/or Litestream of the **primary only**; tested restore before Phase 4. - Fail closed: metered stamp paths return **503** if BusinessStore write unavailable (no silent local SQLite fallback in production). - Network: libSQL on private fabric only (same class as NATS `10.10.10.0/24`); not public. - Schema: `verae_business`; auth stays out for v1 (C13). **Interface:** `BusinessStore` (TS) with drivers: `LibsqlBusinessStore` (prod), `SqliteFileBusinessStore` (dev/test). --- ## 5. NATS usage matrix | Subject / stream | Transport | Purpose | |------------------|-----------|---------| | `verae.zapier.jobs.watch` | JetStream | Poll Verae status | | `verae.zapier.jobs.events` | JetStream | Terminal Zapier-pipeline job events | | `verae.zapier.usage` / `ZAPIER_USAGE` | JetStream optional | **Telemetry/export only; NOT ledger SoT** | | `verae.zapier.webhooks.deliver` | JetStream | Hook delivery retries | | `verae.archive.*` | JetStream / fanout | WORM/tree | | `verae.billing.*` | Core request-reply (+ optional pub) | Statement/balance; eventually thin facade over libSQL | | `verae.ts.completed`, `verae.ts.receipt` | **Core NATS only** | Chain job-complete notify | | `verae.access.*`, `verae.identity.*` | Core / as designed | Authz & bind | --- ## 6. Constraint (TOC) > The system constraint is **shared business SoT cutover (libSQL + single prepaid writer)** because every multi-host userflow (meter, CS credit, sales quote, portal statement, accounting export) blocks on one consistent customer/balance/usage store; JetStream and doors already exist in the lab. Critical chain: **Conflict freeze → BusinessStore+libSQL → Migrate data → Rewire edge/CS/sales/acct → E2E userflow tests → QA whattochange loop**. Feeding chains (parallel): digests-only docs sync; NATS subject clarification; access-plane smoke; Forgejo clone inventory. --- ## 7. Phased implementation (PM DAG) ### Phase 0 — Docs & conflict freeze - [ ] Publish this plan to catalog `docs/plans/` + `zapier-decisions/plans/` - [ ] Patch conflict C2/C3/C5/C6 in overview + models + README status blurb - [ ] Architecture review sign-off artifact: `plans/reviews/2026-10-03-arch-review.md` ### Phase 1 — BusinessStore + libSQL skeleton (TDD) - [ ] Define `BusinessStore` interface (customers, usage.record+verifyChain, rates, invoices, balances) - [ ] Failing tests against interface - [ ] `LibsqlBusinessStore` + docker/libsql-server or `:memory:`/file for CI - [ ] Keep `SqliteFileBusinessStore` wrapping current repos for lab ### Phase 2 — Migrate lab data - [ ] One-shot migrator: `billing.db` + `books.json` → libSQL - [ ] Preserve usage hash-chain tip integrity (verify before/after); refuse cutover if verifyChain fails - [ ] Dual-run: writes libSQL only when `BUSINESS_STORE=libsql`; file fallback **read-only** - [ ] Account-balance becomes libSQL-backed facade for `verae.billing.*` **or** is stopped after HTTP BusinessStore cutover (pick one in Phase 3) ### Phase 3 — Rewire writers - [ ] zappier-edge meter → BusinessStore only - [ ] account-balance service reads/writes libSQL (or deprecate mailbox in favor of HTTP billing API) - [ ] CS / sales / accounting-export point at same store - [ ] Remove second writer paths (C7) ### Phase 4 — Multi-host topology - [ ] Deploy libSQL on dedicated host (or primary of 3–5) - [ ] Edge/middleware/CS on separate hosts with `LIBSQL_URL` - [ ] JetStream cluster already present — verify workers only - [ ] Confirm leaf cannot call billing adjust ### Phase 5 — Userflow E2E + gates - [ ] Automate flows 1–7 (§1) in `harness/` or `packages/verae-zapier-simulator` extensions - [ ] Jest unit + integration + gate scripts green - [ ] Live smoke on 138 with MOCK_VERAE then optional live Verae ### Phase 6 — QA loop (repeat until pass) - [ ] QA finds bugs → `/whattochange` pack (a/b/c/d + todos) - [ ] Engineers implement; PM reorders PreReqs - [ ] Re-run full suite + userflows - [ ] Exit when all critical userflows pass twice consecutively --- ## 8. Test requirements (“tests for everything”) | Layer | Must cover | |-------|------------| | Unit | BusinessStore ops, hash-chain append/tamper, channel rate resolve, digests-only batch mapping, D004 leaf deny | | Integration | edge→middleware mock stamp; billing adjust→statement; JetStream watch→event (testcontainers/nats) | | Contract | OpenAPI digests-only examples; NATS payload schemas | | E2E userflow | Flows 1–7 against lab compose or NS1 | | Regression | Existing zappier-edge Jest suite must stay green during driver swap | | Non-goals | Go timestamping-svc blockchain tests | --- ## 9. Roles (agent orchestration) | Role | Responsibility | Artifact | |------|----------------|----------| | **Architect** | Review plan, expand conflict register, approve SoT | `plans/reviews/*-arch-review.md` | | **PM** | DAG, task assignment, merge gates, buffer | `plans/reviews/*-pm-board.md` | | **Engineers** | Implement Phases 1–5 in worktrees | PRs / commits citing this plan | | **QA** | Break userflows; whattochange packs; retest | `plans/qa/*` + NotImplemented leftovers | | **Orchestrator (this session)** | Launch agents, merge gates, enforce D006/JetStream rules | STATUS updates | --- ## 10. Acceptance criteria (project buffer) 1. One libSQL URL serves customers/pricing/usage/invoices/balances for all commercial services. 2. No production business writes to per-host `billing.db` / `books.json`. 3. JetStream used only for work/archive streams listed in §5. 4. `verae.ts.completed` / `verae.ts.receipt` remain core NATS. 5. Digests-only multihash batch path green with credit headers. 6. Userflows 1–7 pass twice in a row under QA. 7. Go timestamping DBs untouched. 8. Catalog docs match runtime (conflicts C2–C6, C10–C18 closed or owned). 9. `ZAPIER_USAGE` is non-SoT; no service writes authoritative balances there. 10. Production metered paths fail closed if BusinessStore writer is down. 11. Concurrent append contract test proves single-writer (or serialized) hash-chain safety. 12. Live env matrix documents `:3040` edge, `:3130` product upstream, `:3100` mock, `:3010` facade/retire. 13. D001 file contains D006 addendum; GETTING-STARTED / 01-system no longer call books.json SoT. --- ## 11. Out of scope - Replacing api.veraetime.net / Go chain storage - Zapier Platform public directory listing (private push may remain operator-gated) - Peergos Seaweed fleet redesign (consume existing billing APIs only) - lan-134 enablement (D005) - Any JetStream binding for `verae.ts.completed` / `verae.ts.receipt` - Editing `/var/lib/timestamping-svc/data/blockchain.db` or timestamping-svc test `results.db` - Multi-primary libSQL or Litestream multi-writer - Moving portal auth to libSQL in the same cutover as billing (defer) ### Architecture review See `plans/reviews/2026-10-03-arch-review.md` (**approve-with-changes**; A1–A8 merged herein). --- ## Task checklist (engineer entry) ### Task 1: BusinessStore interface + failing tests **Files:** Create under zappier-edge `src/db/business-store.ts`, `tests/business-store.contract.test.ts` **Produces:** Interface used by later tasks - [ ] Write contract tests for recordUsage/verifyChain/findCustomer/getBalance/adjustBalance - [ ] Run tests — expect fail - [ ] Minimal SqliteFile adapter pass - [ ] Commit citing this plan ### Task 2: LibsqlBusinessStore - [ ] Add `@libsql/client` - [ ] Implement same contract - [ ] CI uses file or embedded libsql - [ ] Commit ### Task 3: Migrator billing.db + books.json → libSQL - [ ] Script + tests with fixtures - [ ] Chain verify pre/post - [ ] Commit *(PM expands Tasks 4+ after arch review.)*