220 lines
16 KiB
Markdown
220 lines
16 KiB
Markdown
# Architecture Review — Verae × Zapier Multi-Host / libSQL Plan
|
||
|
||
**Plan under review:** `plans/2026-10-03-libsql-multihost-architecture.md`
|
||
**Reviewer role:** Architecture
|
||
**Date:** 2026-10-03
|
||
**Inputs:** plan; `/tmp/zapier-system-docs-map.md` (mirrored `plans/2026-10-03-system-docs-map.md`); D001–D006; catalog `composition.md`, `nats-subjects.md`, `overview/01-system.md`, `packages/verae-ops/GETTING-STARTED.md`; live 138 `/opt/verae/zappier-edge` (`billing.db` + `zappier.db`, `zappier-edge.service` on `127.0.0.1:3040`).
|
||
|
||
---
|
||
|
||
## Verdict
|
||
|
||
**approve-with-changes**
|
||
|
||
Direction is correct and aligned with **D006**: libSQL as business SoT, JetStream for work queues only, core NATS for `verae.ts.completed` / `verae.ts.receipt`, digests-only stamps, Go `blockchain.db` out of scope. The plan is **not** blocked on SoT choice.
|
||
|
||
It is **not** a clean approve: multi-host money durability, single-writer sequencing, stream/subject demotions, live port matrix, and decision-doc amendments are underspecified. Build of Phases 1–2 (interface + migrator skeleton) may start **after** the amendments in §Required plan amendments are merged into the plan (and D001 text is patched). Do **not** start Phase 3 rewire or Phase 4 multi-host split until those amendments are in the plan and the conflict register rows below are closed or explicitly deferred with owner + date.
|
||
|
||
---
|
||
|
||
## Expanded conflict register
|
||
|
||
Inherited from plan §3 (C1–C9) plus new findings from catalog + live 138.
|
||
|
||
| ID | Conflict | Severity | Resolution (required) |
|
||
|----|----------|----------|------------------------|
|
||
| C1 | D001 `books.json` SoT vs live `billing.db` vs D006 libSQL | High | **libSQL is SoT**; migrate books + billing.db; amend D001 file to point at D006 (not only the plan narrative) |
|
||
| C2 | README “imported, not yet wired” vs STATUS lab up | Med | STATUS + live doors = operational truth; patch README in Phase 0 |
|
||
| C3 | overview `zmw_` Bearer vs composition `x-api-key` | Med | Public = `x-api-key` (D002/composition); fix overview/models |
|
||
| C4 | Edge ports `:3000` / `:13000` / live `:3040` (`zap.verae.ai`) | High | Publish env matrix; lab doors proxy loopback; product edge remains `127.0.0.1:3040` until deliberately moved |
|
||
| C5 | JetStream `jobs.events` vs ban on JetStream for job-complete | High | **Different subjects:** `verae.zapier.jobs.events` (Zapier pipeline, JetStream OK) ≠ `verae.ts.*` (core NATS only). Document both in nats-subjects + standing rule |
|
||
| C6 | TimestampRequest “Hello world” data vs digests-only | Med | Digests-only + multihash batch; sync catalog models to live swagger |
|
||
| C7 | Dual prepaid writers (edge SQLite + account-balance) | High | One writer API on libSQL; edge caches only; deprecate/fold mailbox |
|
||
| C8 | Per-host SQLite on 3–5 machines | High | Forbidden for business data; libSQL shared |
|
||
| C9 | Go `blockchain.db` in scope creep | Blocker if violated | **Out of scope** — do not touch |
|
||
| **C10** | Catalog/STATUS stream **`ZAPIER_USAGE`** vs libSQL usage ledger | High | Demote `ZAPIER_USAGE` / `verae.zapier.usage` to **telemetry/export only** (or remove). Must never be treated as money/usage SoT. Plan §5 must say this explicitly |
|
||
| **C11** | Live dual middleware: catalog/ops `:3100` (MOCK, natsConnected=false) vs edge env `ZAPPIER_UPSTREAM=http://127.0.0.1:3130` (live mockVerae=false, natsConnected=true) | High | Env matrix must list both; product smoke uses **3130**; do not “fix” by pointing edge at 3100 without intent |
|
||
| **C12** | D001 decision file still says `books.json` is sole prepaid writer with no D006 amendment | High | Patch `decisions/D001-prepaid-sot.md` (addendum: durable store = libSQL; books.json lab legacy) |
|
||
| **C13** | Auth store undecided (`admin_users` / `portal_sessions` edge-local vs libSQL) | Med | **Decide before Phase 3:** v1 keep auth in edge-local `zappier.db` (or per-edge file); business tables only in libSQL. Document; revisit if edges are multi-host without sticky sessions |
|
||
| **C14** | Concurrent `usage_entries` hash-chain append across hosts | High (launch) | Single logical meter writer (one process or libSQL-serialized exclusive txn path). No competing multi-host `BEGIN IMMEDIATE` without proven libSQL semantics. Dual-run must not dual-append |
|
||
| **C15** | GETTING-STARTED / `01-system.md` still call account-balance prepaid SoT | Med | Phase 0 doc patch: SoT → libSQL; account-balance becomes facade or retired |
|
||
| **C16** | Live account-balance still up on `:3010` while edge also owns `billing.db` ledger | High | Cutover runbook must name which process is authoritative **each day** of dual-run; forbid two adjust paths |
|
||
| **C17** | Optional Litestream vs multi-writer temptation | Med | Litestream = backup/DR of **one** primary only (D006 non-goal). Plan must forbid Litestream-as-fabric |
|
||
| **C18** | Digests-only vs any remaining “upload file then stamp” Zapier creates | Med | Client hasher + digests-only batch remains the supported path; reject raw-data stamp examples in public docs |
|
||
|
||
---
|
||
|
||
## Risks for 3–5 machine launch
|
||
|
||
1. **Money split-brain (critical).** Today prepaid/ledger truth is fragmented (`books.json` via `:3010`, live `billing.db` on edge, caches). Splitting planes across hosts before one libSQL writer lands will diverge balances, statements, and CS credits. Mitigate: no Phase 4 host split until Phase 3 removes second writers and dual-run is read-only fallback.
|
||
|
||
2. **Hash-chain integrity under concurrency (critical).** `usage_entries` depends on tip-linked `prev_hash`/`row_hash`. Multiple edge replicas opening libSQL without an exclusive append protocol will fork or fail the chain. Mitigate: one meter writer service (or proven libSQL write serialization + contract tests for concurrent append).
|
||
|
||
3. **libSQL HA underspecified (high).** Plan says “dedicated host or primary of 3–5” but not: primary placement, replica/read role, failover RPO/RTO, backup (Litestream or native), network ACL, and what happens when libSQL is down (fail closed on meter vs stale cache). For launch, pick **one primary + backup**; do not invent multi-primary.
|
||
|
||
4. **JetStream mistaken for ledger (high).** `ZAPIER_USAGE` already exists in STATUS with replicas=3. Operators/agents will “persist usage on JetStream” under reliability pressure. Standing rule + plan §5 must demote it; CI/docs grep for usage-as-SoT.
|
||
|
||
5. **Job-complete subject confusion (high).** Zapier pipeline `verae.zapier.jobs.events` (JetStream) vs Verae chain notify `verae.ts.completed`/`verae.ts.receipt` (core only). A “make complete more reliable” change must not JetStream `verae.ts.*` (standing rule). Plan C5 is correct; nats-subjects.md still omits `verae.ts.*` — patch it.
|
||
|
||
6. **Wrong upstream in smoke (medium).** Edge points at `:3130`; lab docs emphasize `:3100`. Multi-host deploy scripts that copy catalog defaults will meter against mock middleware and lie about chain connectivity.
|
||
|
||
7. **Auth sticky-session / multi-edge (medium).** If auth stays in local `zappier.db` and two edges serve portal/admin, sessions break. Acceptable for v1 single edge; call out as launch constraint.
|
||
|
||
8. **Migration tip verification false confidence (medium).** Lab `usage_entries` tip is tiny (observed count=1 on 138). Migrator tests need adversarial fixtures (long chain, tamper mid-chain, empty DB), not only live copy.
|
||
|
||
9. **Deploy rsync DB wipe (medium, known).** Standing ops: never rsync `*.db*` onto `/opt/verae/zappier-edge`. libSQL cutover must use migrator + URL switch, not file copy into prod workdir via deploy.
|
||
|
||
10. **NATS billing request-reply after mailbox retirement (medium).** `verae.billing.*` subjects remain useful as a facade; if account-balance dies without a replacement listener, CS/sales/portal statement paths break even if libSQL is healthy. Plan must keep a single billing facade process.
|
||
|
||
11. **Credits vs cents naming (low).** Product credits; columns `*_cents`. Fine if documented; do not “fix” by rewriting historical rows mid-migration.
|
||
|
||
12. **Go timestamping DB proximity (low but non-negotiable).** Same host may hold `/var/lib/timestamping-svc/data/blockchain.db`. Automation must path-allowlist business DBs only.
|
||
|
||
---
|
||
|
||
## Required plan amendments (concrete text)
|
||
|
||
Merge the following into `plans/2026-10-03-libsql-multihost-architecture.md` (and patch D001 as noted).
|
||
|
||
### A1 — Amend Global Constraints (add bullets)
|
||
|
||
```text
|
||
- ZAPIER_USAGE / verae.zapier.usage is telemetry or export only — never usage/money SoT.
|
||
- Litestream (if used) backs up a single libSQL primary; never multi-writer fabric.
|
||
- Single meter/prepaid writer process owns usage_entries append + balance adjust;
|
||
other hosts call that API or use libSQL only through that writer’s transactions.
|
||
- Dual-run: libSQL primary for writes; file/SQLite fallback is read-only (or kill-switched).
|
||
- Auth (admin_users, portal_sessions): v1 remains edge-local zappier.db; not in verae_business.
|
||
- Product edge bind on 138: 127.0.0.1:3040 (zap.verae.ai); ZAPPIER_UPSTREAM defaults to
|
||
http://127.0.0.1:3130 for live lab smoke (not :3100 mock unless MOCK path intentional).
|
||
- Catalog docs that call account-balance “prepaid SoT” are stale after D006; Phase 0 patches them.
|
||
```
|
||
|
||
### A2 — Expand §5 NATS matrix (replace/extend rows)
|
||
|
||
```text
|
||
| verae.zapier.usage / ZAPIER_USAGE | JetStream optional | Telemetry/export only; NOT ledger SoT |
|
||
| verae.ts.completed, verae.ts.receipt | Core NATS only | Chain job-complete notify; never JetStream |
|
||
| verae.billing.* | Core request-reply (+ optional pub) | Facade over libSQL BusinessStore; one listener |
|
||
```
|
||
|
||
Also add to Phase 0: patch `docs/02-architecture/nats-subjects.md` with an explicit **Non-streams** section listing `verae.ts.*` as core-only and `ZAPIER_USAGE` as non-SoT.
|
||
|
||
### A3 — New §4.1 libSQL topology (insert after §4)
|
||
|
||
```text
|
||
### 4.1 Launch topology (3–5 machines)
|
||
|
||
- One libSQL primary (dedicated host or named primary among the 3–5).
|
||
- Replicas if any are read-only; writers always target primary URL (`LIBSQL_URL`).
|
||
- Backup: filesystem snapshot and/or Litestream of the primary only; tested restore before Phase 4.
|
||
- Fail closed: metered stamp paths return 503 if BusinessStore write unavailable
|
||
(no silent local SQLite fallback in production).
|
||
- Network: libSQL listen on private fabric only (same class as NATS 10.10.10.0/24); not public.
|
||
- Schema name: `verae_business` (confirm); auth schema stays out for v1.
|
||
```
|
||
|
||
### A4 — Phase 2 dual-run clarification
|
||
|
||
```text
|
||
- Writes: libSQL only once cutover flag BUSINESS_STORE=libsql is on.
|
||
- Fallback flag enables read-only SQLite/books for diagnosis, not adjust/recordUsage.
|
||
- Migrator verifies usage chain pre/post; refuses cutover if verifyChain fails.
|
||
- Account-balance either (a) becomes libSQL-backed facade for verae.billing.*, or
|
||
(b) is stopped after edge/CS/sales talk HTTP BusinessStore — pick one in Phase 3 tasking.
|
||
```
|
||
|
||
### A5 — Patch decision file D001 (Phase 0, same PR as plan amend)
|
||
|
||
Append to `decisions/D001-prepaid-sot.md`:
|
||
|
||
```text
|
||
## Addendum (2026-10-03, D006)
|
||
|
||
Durable prepaid/credits/usage/payments store is shared **libSQL**, not per-process
|
||
`books.json` or per-host `billing.db`. Single writer path still required. Edge
|
||
`balanceCents` remains a cache. Lab `books.json` / `billing.db` are migration sources.
|
||
```
|
||
|
||
### A6 — Conflict register: add C10–C18
|
||
|
||
Copy the new rows from this review’s register into plan §3.
|
||
|
||
### A7 — Acceptance criteria additions
|
||
|
||
```text
|
||
9. ZAPIER_USAGE is documented/configured as non-SoT; no service writes authoritative balances there.
|
||
10. Production metered paths fail closed if libSQL/BusinessStore writer is down.
|
||
11. Concurrent append contract test proves single-writer (or serialized) hash-chain safety.
|
||
12. Live env matrix documents :3040 edge, :3130 product upstream, :3100 mock, :3010 facade/retire.
|
||
13. D001 file contains D006 addendum; GETTING-STARTED / 01-system no longer call books.json SoT.
|
||
```
|
||
|
||
### A8 — Out of scope (reinforce)
|
||
|
||
```text
|
||
- Any JetStream binding for verae.ts.completed / verae.ts.receipt
|
||
- Editing /var/lib/timestamping-svc/data/blockchain.db or timestamping-svc JWT test results.db
|
||
- Multi-primary libSQL or Litestream multi-writer
|
||
- Moving portal auth to libSQL in the same cutover as billing (defer)
|
||
```
|
||
|
||
---
|
||
|
||
## What must not change (preserve list)
|
||
|
||
| Preserve | Why |
|
||
|----------|-----|
|
||
| **libSQL = business SoT** (D006) | Launch invariant for 3–5 hosts |
|
||
| **JetStream = work/archive only** | Not money/CRM database |
|
||
| **Core NATS only for `verae.ts.completed` / `verae.ts.receipt`** | Standing rule; no “reliability” override without explicit user words |
|
||
| **Digests-only stamp submit** (batch multihash / client hasher) | Live swagger + MCP; no file-byte stamp SoT |
|
||
| **No Go timestamping-svc SQLite edits** | `blockchain.db` and related stay local to that service |
|
||
| **D001 single prepaid writer path** (store upgraded by D006) | Prevents dual adjust |
|
||
| **D002** public `x-api-key`; `veraeUserId` join; JWT as `tokenRef` | Composition ownership |
|
||
| **D003** access planes as only public HTTP doors | Edge admin/Stripe stay non-door (loopback / tunnel) |
|
||
| **D004** leaf cannot touch billing | Policy allow-list |
|
||
| **D005** lan-134 off unless explicitly enabled | Fleet safety |
|
||
| **Credits product language**; ledger may keep `*_cents` column names | Avoid migration churn |
|
||
| **WAL (or libSQL equivalent) durability** on SQL stores | Existing edge invariant |
|
||
| **Deploy never rsyncs live `*.db*`** over production data | Known SQLITE_CANTOPEN / wipe failure mode |
|
||
| **Zapier never on NATS** | Hard rule in `01-system.md` |
|
||
| **Payment invoices ≠ timestamp receipts** | Composition |
|
||
| **Composition ownership:** metering/portal on zappier; stamp/NATS/hooks on middleware | Do not re-merge Stripe into middleware |
|
||
| **Existing zappier-edge Jest suite stays green** during driver swap | Regression gate |
|
||
| **Channel×segment rate card + credit headers** on metered/batch paths | Live product behavior |
|
||
|
||
---
|
||
|
||
## Sign-off checklist
|
||
|
||
Architecture signs when all boxes are true (or explicitly waived in writing on this file).
|
||
|
||
- [ ] Verdict understood: **approve-with-changes** (not full approve).
|
||
- [x] Plan amended with **A1–A8** (merged 2026-10-03) (or equivalent text merged).
|
||
- [x] Conflict register includes **C10–C18** with owners.
|
||
- [x] **D001** addendum committed pointing durable store at libSQL / D006.
|
||
- [ ] **nats-subjects.md** documents `verae.ts.*` core-only and `ZAPIER_USAGE` non-SoT.
|
||
- [ ] Env matrix published: `:3040` edge, `:3130` product MW, `:3100` mock MW, `:3010` balance facade.
|
||
- [ ] Single-writer / fail-closed / Litestream-backup-only rules in plan.
|
||
- [ ] Auth v1 = edge-local decision recorded (C13).
|
||
- [ ] Preserve list acknowledged by PM (no JetStream job-complete; no Go DB; digests-only).
|
||
- [ ] Phase 3/4 gated on writer consolidation + migrator chain verify.
|
||
- [ ] QA/PM may proceed to board + Task 1 **after** A1–A8 land; not before.
|
||
|
||
**Architecture provisional sign-off:** approve-with-changes — **A1–A8 merged into plan + D001**; remaining checklist items still open (nats-subjects, env matrix, GETTING-STARTED).
|
||
**Full architecture sign-off:** withheld until checklist complete.
|
||
|
||
---
|
||
|
||
## Evidence notes (live 138, 2026-10-03)
|
||
|
||
- `/opt/verae/zappier-edge`: `billing.db` + `zappier.db` (+ WAL/SHM), owner `verae-zapier`; service **active**.
|
||
- Edge env: `PORT=3040`, `HOST=127.0.0.1`, `ZAPPIER_UPSTREAM=http://127.0.0.1:3130`.
|
||
- Health: `:3040` zappier-edge ok; `:3100` middleware `mockVerae=true`, `natsConnected=false`; `:3130` middleware `mockVerae=false`, `natsConnected=true`; `:3010` account-balance still serving.
|
||
- `billing.db` tables: customers, usage_entries, channel_rates*, invoices*, price_endpoints, tiers; `zappier.db`: admin_users, portal_sessions.
|
||
- Access planes listening on `:3021–3025` (+ staff IAM/session); NATS client ports present on private/host binds.
|
||
|
||
---
|
||
|
||
*End of architecture review.*
|