WIP snapshot before drive cleanup 2026-10-03

This commit is contained in:
George Lambert 2026-10-03 23:02:02 -04:00
commit e666e7bf3c

View file

@ -0,0 +1,295 @@
# Verae × Zapier Multi-Host Architecture & Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (or executing-plans) task-by-task. Steps use checkbox (`- [ ]`) syntax.
**Goal:** Run the Verae × Zapier commercial stack across **3–5 machines** with **libSQL** as the system of record for all business/billing/customer/pricing/accounting data, **JetStream only for work queues**, and proven end-to-end userflows under continuous QA.
**Architecture:** Public HTTPS doors → zappier-edge (meter) → middleware (stamp) → `api.veraetime.net`; private NATS for workers; shared libSQL for money & CRM; Go timestamping DBs stay local to timestamping-svc.
**Tech Stack:** Node (edge, middleware, access planes, CS/sales/acct), NATS JetStream (3-node lab → production), libSQL / go-libsql or `@libsql/client`, better-sqlite3 only as local/dev adapter, Litestream optional for libSQL file primary backup, Jest + gate scripts + E2E userflow harness.
**Spec / SoT docs:**
- Catalog: https://zapier.georgelambert.org/
- Map: `/tmp/zapier-system-docs-map.md`
- Decisions: D001–D006 (`packages/zapier-decisions/decisions/`)
- Composition: `docs/02-architecture/composition.md`
- NATS: `docs/02-architecture/nats-subjects.md`, `overview/03-nats-cluster.md`
- Ops: `packages/verae-ops/GETTING-STARTED.md`
- Digests-only stamps: mcp.verae.ai + D006 note + live swagger
**Forgejo org:** https://git.georgelambert.org/marchon/ (modules listed in §2)
## Global Constraints
- **D006:** Business/billing/customers/pricing/accounting/usage ledger → **libSQL only** (shared).
- **JetStream:** work queues as necessary (`ZAPIER_JOBS`, `ZAPIER_WEBHOOKS`, archive put/query). Not money SoT.
- **Core NATS only** for `verae.ts.completed` / `verae.ts.receipt` (no JetStream on that path).
- **D001 refined:** single prepaid **writer path**; durable store is libSQL (not per-host `books.json` / `billing.db`).
- **D002:** public credential = zappier `x-api-key`; `veraeUserId` join; JWT = `tokenRef` server-side.
- **D003:** only access planes are public HTTP; edge loopback (except intentional public `zap.verae.ai` edge).
- **D004:** leaf must not touch billing.
- **Digests only** for stamp submit (`POST /api/batch/timestamp` multihash; MCP shape).
- **Never** modify Go timestamping-svc SQLite (`blockchain.db`, etc.) as part of this plan.
- **WAL** (or libSQL equivalent durability) on all SQL stores.
- Credits language in product copy; ledger may keep `*_cents` column names as credit units.
- Deploy never rsyncs live `*.db*` over production data.
- **ZAPIER_USAGE** / `verae.zapier.usage` is telemetry or export only — never usage/money SoT.
- Litestream (if used) backs up a **single** libSQL primary; never multi-writer fabric.
- **Single meter/prepaid writer** process owns `usage_entries` append + balance adjust; other hosts call that API.
- Dual-run: libSQL primary for writes; file/SQLite fallback is **read-only** (or kill-switched).
- Auth (`admin_users`, `portal_sessions`): v1 remains edge-local `zappier.db`; not in `verae_business`.
- Product edge bind on 138: `127.0.0.1:3040` (`zap.verae.ai`); `ZAPPIER_UPSTREAM` defaults to `http://127.0.0.1:3130` for live lab smoke (not `:3100` mock unless intentional).
- Catalog docs that call account-balance “prepaid SoT” are stale after D006; Phase 0 patches them.
---
## 1. System map (from catalog + live lab)
```text
Internet
├─ Zapier cloud ──HTTPS──► access-zapier :3024 ──► zappier-edge (meter) ──► middleware :3100
├─ Browsers ────HTTPS──► access-web :3021 /portal
├─ Customer API ─HTTPS──► access-api :3022
├─ Leaf S2S ────HTTPS──► access-leaf :3023 (no billing)
└─ Staff ───────HTTPS──► access-staff :3025 + IAM :3028
Private
├─ NATS JetStream 10.10.10.21–23 (jobs.watch, webhooks.deliver, archive.*)
├─ Core NATS (verae.ts.completed, verae.ts.receipt, billing request-reply)
├─ libSQL (TARGET) ← customers, pricing, usage chain, invoices, balances
├─ Workers job-poller, webhook-deliver, archive-worm×N, tree-node×N
└─ api.veraetime.net / MOCK_VERAE
```
### Critical userflows (must pass QA)
1. **Signup → API key → Statement** (portal via access-web).
2. **Stamp digests (single + batch)** via zap door → edge meter → middleware → Verae → 202 jobIds → credits headers.
3. **Async complete → Zapier REST Hook** (`jobs.watch` → poll → `jobs.events` → `webhooks.deliver`).
4. **CS credit adjust → balance visible** on portal/admin (single prepaid writer).
5. **Sales channel×segment quote → meter charge** matches rate card.
6. **Accounting export** sees payments/usage consistent with libSQL.
7. **Leaf plane** can stamp/authz but **cannot** adjust billing (D004).
---
## 2. Forgejo source elements (implementation clones)
| Concern | Repo | Notes |
|---------|------|-------|
| Billing edge | `marchon/zappier-edge` | Live `/opt/verae/zappier-edge` on 138; also catalog `packages/zappier` |
| Prepaid mailbox (legacy) | `marchon/zappier-account-balance` | `books.json` → migrate into libSQL |
| CS / Sales / Acct | `zappier-customer-service`, `zappier-sales-pricing`, `zappier-accounting-export` | |
| Identity | `zappier-identity` | |
| Middleware | `marchon/verae-middleware` (catalog: verae-zapier-middleware) | |
| Zapier apps | `verae-zapier-app`, `verae-activate` | |
| Access planes | `verae-access-{web,api,leaf,zapier,staff,authz}` | |
| Jobs/archive | `verae-jobs-events`, `verae-archive-*`, `verae-tree-node`, `verae-request-splitter` | |
| Ops | `verae-ops`, `verae-bootstrap`, `verae-fleet`, `verae-keep`, `verae-nats-cluster` | |
| Decisions | `zapier-decisions` | D001–D006 |
| Monorepo mirror | `master-zapier-plan-draft` | docs + packages composition |
---
## 3. Conflict register (resolve before/with build)
| ID | Conflict | Resolution (locked) |
|----|----------|---------------------|
| C1 | D001 `books.json` SoT vs live `billing.db` vs D006 libSQL | **libSQL is SoT**; migrate both books + billing.db schema; account-balance becomes libSQL-backed service or folds into billing API |
| C2 | README “imported, not yet wired” vs STATUS lab up | Treat STATUS + live doors as operational truth; update README composition status in Phase 0 docs |
| C3 | overview `zmw_` Bearer vs composition `x-api-key` | **Public = x-api-key** (D002/composition); fix overview/models |
| C4 | Edge ports `:3000` / `:13000` / live `:3040` (`zap.verae.ai`) | Document env matrix; lab doors proxy to loopback; zap.verae.ai may remain public edge |
| C5 | JetStream `jobs.events` vs ban on JetStream for job-complete | **Different subjects:** Zapier job lifecycle may use JetStream `verae.zapier.jobs.events`; Verae chain notify `verae.ts.*` stays **core NATS** |
| C6 | TimestampRequest “Hello world” data vs digests-only | Digests-only + multihash batch (already fixed on live swagger; sync catalog models) |
| C7 | Dual prepaid writers (edge SQLite + account-balance) | One writer API on libSQL; edge caches only |
| C8 | Per-host SQLite on 3–5 machines | Forbidden for business data; libSQL shared |
| C9 | Go `blockchain.db` in scope creep | Out of scope — do not touch |
| C10 | `ZAPIER_USAGE` vs libSQL usage ledger | Demote to telemetry/export only; never money SoT |
| C11 | Dual middleware `:3100` mock vs `:3130` product upstream | Env matrix; product smoke uses **3130** |
| C12 | D001 file still says books.json SoT | D001 addendum → libSQL (done with this amend) |
| C13 | Auth store edge-local vs libSQL | **v1 edge-local** `zappier.db` |
| C14 | Concurrent hash-chain append across hosts | Single meter writer / serialized append |
| C15 | GETTING-STARTED calls account-balance SoT | Phase 0 doc patch → libSQL |
| C16 | Live `:3010` + `billing.db` dual writers | Cutover runbook: one authoritative adjust path |
| C17 | Litestream multi-writer temptation | Litestream = backup of one primary only |
| C18 | Digests-only vs upload-file stamp docs | Client hasher + digests-only only |
---
## 4. Target data ownership (libSQL)
**Database:** `verae_business` (name TBD) on shared libSQL.
| Domain | Tables / collections | Writer |
|--------|----------------------|--------|
| Customers / identity bind | `customers`, identity map | billing API + identity service |
| Pricing | `tiers`, `price_endpoints`, `channel_rates`, `channel_rate_meta` | admin / sales-pricing |
| Metered ledger | `usage_entries` (+ `prev_hash`/`row_hash` chain) | **single** meter writer |
| Prepaid | balances, credits, payments (from books.json) | billing adjust API |
| Invoices | `invoices`, `invoice_lines` | admin / Stripe jobs |
| Auth (optional split) | `admin_users`, `portal_sessions` | may stay edge-local or move to libSQL `auth` schema |
### 4.1 Launch topology (3–5 machines)
- One libSQL primary (dedicated host or named primary among the 3–5).
- Replicas if any are read-only; writers always target primary URL (`LIBSQL_URL`).
- Backup: filesystem snapshot and/or Litestream of the **primary only**; tested restore before Phase 4.
- Fail closed: metered stamp paths return **503** if BusinessStore write unavailable (no silent local SQLite fallback in production).
- Network: libSQL on private fabric only (same class as NATS `10.10.10.0/24`); not public.
- Schema: `verae_business`; auth stays out for v1 (C13).
**Interface:** `BusinessStore` (TS) with drivers: `LibsqlBusinessStore` (prod), `SqliteFileBusinessStore` (dev/test).
---
## 5. NATS usage matrix
| Subject / stream | Transport | Purpose |
|------------------|-----------|---------|
| `verae.zapier.jobs.watch` | JetStream | Poll Verae status |
| `verae.zapier.jobs.events` | JetStream | Terminal Zapier-pipeline job events |
| `verae.zapier.usage` / `ZAPIER_USAGE` | JetStream optional | **Telemetry/export only; NOT ledger SoT** |
| `verae.zapier.webhooks.deliver` | JetStream | Hook delivery retries |
| `verae.archive.*` | JetStream / fanout | WORM/tree |
| `verae.billing.*` | Core request-reply (+ optional pub) | Statement/balance; eventually thin facade over libSQL |
| `verae.ts.completed`, `verae.ts.receipt` | **Core NATS only** | Chain job-complete notify |
| `verae.access.*`, `verae.identity.*` | Core / as designed | Authz & bind |
---
## 6. Constraint (TOC)
> The system constraint is **shared business SoT cutover (libSQL + single prepaid writer)** because every multi-host userflow (meter, CS credit, sales quote, portal statement, accounting export) blocks on one consistent customer/balance/usage store; JetStream and doors already exist in the lab.
Critical chain: **Conflict freeze → BusinessStore+libSQL → Migrate data → Rewire edge/CS/sales/acct → E2E userflow tests → QA whattochange loop**.
Feeding chains (parallel): digests-only docs sync; NATS subject clarification; access-plane smoke; Forgejo clone inventory.
---
## 7. Phased implementation (PM DAG)
### Phase 0 — Docs & conflict freeze
- [ ] Publish this plan to catalog `docs/plans/` + `zapier-decisions/plans/`
- [ ] Patch conflict C2/C3/C5/C6 in overview + models + README status blurb
- [ ] Architecture review sign-off artifact: `plans/reviews/2026-10-03-arch-review.md`
### Phase 1 — BusinessStore + libSQL skeleton (TDD)
- [ ] Define `BusinessStore` interface (customers, usage.record+verifyChain, rates, invoices, balances)
- [ ] Failing tests against interface
- [ ] `LibsqlBusinessStore` + docker/libsql-server or `:memory:`/file for CI
- [ ] Keep `SqliteFileBusinessStore` wrapping current repos for lab
### Phase 2 — Migrate lab data
- [ ] One-shot migrator: `billing.db` + `books.json` → libSQL
- [ ] Preserve usage hash-chain tip integrity (verify before/after); refuse cutover if verifyChain fails
- [ ] Dual-run: writes libSQL only when `BUSINESS_STORE=libsql`; file fallback **read-only**
- [ ] Account-balance becomes libSQL-backed facade for `verae.billing.*` **or** is stopped after HTTP BusinessStore cutover (pick one in Phase 3)
### Phase 3 — Rewire writers
- [ ] zappier-edge meter → BusinessStore only
- [ ] account-balance service reads/writes libSQL (or deprecate mailbox in favor of HTTP billing API)
- [ ] CS / sales / accounting-export point at same store
- [ ] Remove second writer paths (C7)
### Phase 4 — Multi-host topology
- [ ] Deploy libSQL on dedicated host (or primary of 3–5)
- [ ] Edge/middleware/CS on separate hosts with `LIBSQL_URL`
- [ ] JetStream cluster already present — verify workers only
- [ ] Confirm leaf cannot call billing adjust
### Phase 5 — Userflow E2E + gates
- [ ] Automate flows 1–7 (§1) in `harness/` or `packages/verae-zapier-simulator` extensions
- [ ] Jest unit + integration + gate scripts green
- [ ] Live smoke on 138 with MOCK_VERAE then optional live Verae
### Phase 6 — QA loop (repeat until pass)
- [ ] QA finds bugs → `/whattochange` pack (a/b/c/d + todos)
- [ ] Engineers implement; PM reorders PreReqs
- [ ] Re-run full suite + userflows
- [ ] Exit when all critical userflows pass twice consecutively
---
## 8. Test requirements (“tests for everything”)
| Layer | Must cover |
|-------|------------|
| Unit | BusinessStore ops, hash-chain append/tamper, channel rate resolve, digests-only batch mapping, D004 leaf deny |
| Integration | edge→middleware mock stamp; billing adjust→statement; JetStream watch→event (testcontainers/nats) |
| Contract | OpenAPI digests-only examples; NATS payload schemas |
| E2E userflow | Flows 1–7 against lab compose or NS1 |
| Regression | Existing zappier-edge Jest suite must stay green during driver swap |
| Non-goals | Go timestamping-svc blockchain tests |
---
## 9. Roles (agent orchestration)
| Role | Responsibility | Artifact |
|------|----------------|----------|
| **Architect** | Review plan, expand conflict register, approve SoT | `plans/reviews/*-arch-review.md` |
| **PM** | DAG, task assignment, merge gates, buffer | `plans/reviews/*-pm-board.md` |
| **Engineers** | Implement Phases 1–5 in worktrees | PRs / commits citing this plan |
| **QA** | Break userflows; whattochange packs; retest | `plans/qa/*` + NotImplemented leftovers |
| **Orchestrator (this session)** | Launch agents, merge gates, enforce D006/JetStream rules | STATUS updates |
---
## 10. Acceptance criteria (project buffer)
1. One libSQL URL serves customers/pricing/usage/invoices/balances for all commercial services.
2. No production business writes to per-host `billing.db` / `books.json`.
3. JetStream used only for work/archive streams listed in §5.
4. `verae.ts.completed` / `verae.ts.receipt` remain core NATS.
5. Digests-only multihash batch path green with credit headers.
6. Userflows 1–7 pass twice in a row under QA.
7. Go timestamping DBs untouched.
8. Catalog docs match runtime (conflicts C2–C6, C10–C18 closed or owned).
9. `ZAPIER_USAGE` is non-SoT; no service writes authoritative balances there.
10. Production metered paths fail closed if BusinessStore writer is down.
11. Concurrent append contract test proves single-writer (or serialized) hash-chain safety.
12. Live env matrix documents `:3040` edge, `:3130` product upstream, `:3100` mock, `:3010` facade/retire.
13. D001 file contains D006 addendum; GETTING-STARTED / 01-system no longer call books.json SoT.
---
## 11. Out of scope
- Replacing api.veraetime.net / Go chain storage
- Zapier Platform public directory listing (private push may remain operator-gated)
- Peergos Seaweed fleet redesign (consume existing billing APIs only)
- lan-134 enablement (D005)
- Any JetStream binding for `verae.ts.completed` / `verae.ts.receipt`
- Editing `/var/lib/timestamping-svc/data/blockchain.db` or timestamping-svc test `results.db`
- Multi-primary libSQL or Litestream multi-writer
- Moving portal auth to libSQL in the same cutover as billing (defer)
### Architecture review
See `plans/reviews/2026-10-03-arch-review.md` (**approve-with-changes**; A1–A8 merged herein).
---
## Task checklist (engineer entry)
### Task 1: BusinessStore interface + failing tests
**Files:** Create under zappier-edge `src/db/business-store.ts`, `tests/business-store.contract.test.ts`
**Produces:** Interface used by later tasks
- [ ] Write contract tests for recordUsage/verifyChain/findCustomer/getBalance/adjustBalance
- [ ] Run tests — expect fail
- [ ] Minimal SqliteFile adapter pass
- [ ] Commit citing this plan
### Task 2: LibsqlBusinessStore
- [ ] Add `@libsql/client`
- [ ] Implement same contract
- [ ] CI uses file or embedded libsql
- [ ] Commit
### Task 3: Migrator billing.db + books.json → libSQL
- [ ] Script + tests with fixtures
- [ ] Chain verify pre/post
- [ ] Commit
*(PM expands Tasks 4+ after arch review.)*