zapier-decisions/plans/reviews/2026-10-03-arch-review.md

220 lines
16 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Architecture Review — Verae × Zapier Multi-Host / libSQL Plan
**Plan under review:** `plans/2026-10-03-libsql-multihost-architecture.md`
**Reviewer role:** Architecture
**Date:** 2026-10-03
**Inputs:** plan; `/tmp/zapier-system-docs-map.md` (mirrored `plans/2026-10-03-system-docs-map.md`); D001–D006; catalog `composition.md`, `nats-subjects.md`, `overview/01-system.md`, `packages/verae-ops/GETTING-STARTED.md`; live 138 `/opt/verae/zappier-edge` (`billing.db` + `zappier.db`, `zappier-edge.service` on `127.0.0.1:3040`).
---
## Verdict
**approve-with-changes**
Direction is correct and aligned with **D006**: libSQL as business SoT, JetStream for work queues only, core NATS for `verae.ts.completed` / `verae.ts.receipt`, digests-only stamps, Go `blockchain.db` out of scope. The plan is **not** blocked on SoT choice.
It is **not** a clean approve: multi-host money durability, single-writer sequencing, stream/subject demotions, live port matrix, and decision-doc amendments are underspecified. Build of Phases 1–2 (interface + migrator skeleton) may start **after** the amendments in §Required plan amendments are merged into the plan (and D001 text is patched). Do **not** start Phase 3 rewire or Phase 4 multi-host split until those amendments are in the plan and the conflict register rows below are closed or explicitly deferred with owner + date.
---
## Expanded conflict register
Inherited from plan §3 (C1–C9) plus new findings from catalog + live 138.
| ID | Conflict | Severity | Resolution (required) |
|----|----------|----------|------------------------|
| C1 | D001 `books.json` SoT vs live `billing.db` vs D006 libSQL | High | **libSQL is SoT**; migrate books + billing.db; amend D001 file to point at D006 (not only the plan narrative) |
| C2 | README “imported, not yet wired” vs STATUS lab up | Med | STATUS + live doors = operational truth; patch README in Phase 0 |
| C3 | overview `zmw_` Bearer vs composition `x-api-key` | Med | Public = `x-api-key` (D002/composition); fix overview/models |
| C4 | Edge ports `:3000` / `:13000` / live `:3040` (`zap.verae.ai`) | High | Publish env matrix; lab doors proxy loopback; product edge remains `127.0.0.1:3040` until deliberately moved |
| C5 | JetStream `jobs.events` vs ban on JetStream for job-complete | High | **Different subjects:** `verae.zapier.jobs.events` (Zapier pipeline, JetStream OK) ≠ `verae.ts.*` (core NATS only). Document both in nats-subjects + standing rule |
| C6 | TimestampRequest “Hello world” data vs digests-only | Med | Digests-only + multihash batch; sync catalog models to live swagger |
| C7 | Dual prepaid writers (edge SQLite + account-balance) | High | One writer API on libSQL; edge caches only; deprecate/fold mailbox |
| C8 | Per-host SQLite on 3–5 machines | High | Forbidden for business data; libSQL shared |
| C9 | Go `blockchain.db` in scope creep | Blocker if violated | **Out of scope** — do not touch |
| **C10** | Catalog/STATUS stream **`ZAPIER_USAGE`** vs libSQL usage ledger | High | Demote `ZAPIER_USAGE` / `verae.zapier.usage` to **telemetry/export only** (or remove). Must never be treated as money/usage SoT. Plan §5 must say this explicitly |
| **C11** | Live dual middleware: catalog/ops `:3100` (MOCK, natsConnected=false) vs edge env `ZAPPIER_UPSTREAM=http://127.0.0.1:3130` (live mockVerae=false, natsConnected=true) | High | Env matrix must list both; product smoke uses **3130**; do not “fix” by pointing edge at 3100 without intent |
| **C12** | D001 decision file still says `books.json` is sole prepaid writer with no D006 amendment | High | Patch `decisions/D001-prepaid-sot.md` (addendum: durable store = libSQL; books.json lab legacy) |
| **C13** | Auth store undecided (`admin_users` / `portal_sessions` edge-local vs libSQL) | Med | **Decide before Phase 3:** v1 keep auth in edge-local `zappier.db` (or per-edge file); business tables only in libSQL. Document; revisit if edges are multi-host without sticky sessions |
| **C14** | Concurrent `usage_entries` hash-chain append across hosts | High (launch) | Single logical meter writer (one process or libSQL-serialized exclusive txn path). No competing multi-host `BEGIN IMMEDIATE` without proven libSQL semantics. Dual-run must not dual-append |
| **C15** | GETTING-STARTED / `01-system.md` still call account-balance prepaid SoT | Med | Phase 0 doc patch: SoT → libSQL; account-balance becomes facade or retired |
| **C16** | Live account-balance still up on `:3010` while edge also owns `billing.db` ledger | High | Cutover runbook must name which process is authoritative **each day** of dual-run; forbid two adjust paths |
| **C17** | Optional Litestream vs multi-writer temptation | Med | Litestream = backup/DR of **one** primary only (D006 non-goal). Plan must forbid Litestream-as-fabric |
| **C18** | Digests-only vs any remaining “upload file then stamp” Zapier creates | Med | Client hasher + digests-only batch remains the supported path; reject raw-data stamp examples in public docs |
---
## Risks for 3–5 machine launch
1. **Money split-brain (critical).** Today prepaid/ledger truth is fragmented (`books.json` via `:3010`, live `billing.db` on edge, caches). Splitting planes across hosts before one libSQL writer lands will diverge balances, statements, and CS credits. Mitigate: no Phase 4 host split until Phase 3 removes second writers and dual-run is read-only fallback.
2. **Hash-chain integrity under concurrency (critical).** `usage_entries` depends on tip-linked `prev_hash`/`row_hash`. Multiple edge replicas opening libSQL without an exclusive append protocol will fork or fail the chain. Mitigate: one meter writer service (or proven libSQL write serialization + contract tests for concurrent append).
3. **libSQL HA underspecified (high).** Plan says “dedicated host or primary of 3–5” but not: primary placement, replica/read role, failover RPO/RTO, backup (Litestream or native), network ACL, and what happens when libSQL is down (fail closed on meter vs stale cache). For launch, pick **one primary + backup**; do not invent multi-primary.
4. **JetStream mistaken for ledger (high).** `ZAPIER_USAGE` already exists in STATUS with replicas=3. Operators/agents will “persist usage on JetStream” under reliability pressure. Standing rule + plan §5 must demote it; CI/docs grep for usage-as-SoT.
5. **Job-complete subject confusion (high).** Zapier pipeline `verae.zapier.jobs.events` (JetStream) vs Verae chain notify `verae.ts.completed`/`verae.ts.receipt` (core only). A “make complete more reliable” change must not JetStream `verae.ts.*` (standing rule). Plan C5 is correct; nats-subjects.md still omits `verae.ts.*` — patch it.
6. **Wrong upstream in smoke (medium).** Edge points at `:3130`; lab docs emphasize `:3100`. Multi-host deploy scripts that copy catalog defaults will meter against mock middleware and lie about chain connectivity.
7. **Auth sticky-session / multi-edge (medium).** If auth stays in local `zappier.db` and two edges serve portal/admin, sessions break. Acceptable for v1 single edge; call out as launch constraint.
8. **Migration tip verification false confidence (medium).** Lab `usage_entries` tip is tiny (observed count=1 on 138). Migrator tests need adversarial fixtures (long chain, tamper mid-chain, empty DB), not only live copy.
9. **Deploy rsync DB wipe (medium, known).** Standing ops: never rsync `*.db*` onto `/opt/verae/zappier-edge`. libSQL cutover must use migrator + URL switch, not file copy into prod workdir via deploy.
10. **NATS billing request-reply after mailbox retirement (medium).** `verae.billing.*` subjects remain useful as a facade; if account-balance dies without a replacement listener, CS/sales/portal statement paths break even if libSQL is healthy. Plan must keep a single billing facade process.
11. **Credits vs cents naming (low).** Product credits; columns `*_cents`. Fine if documented; do not “fix” by rewriting historical rows mid-migration.
12. **Go timestamping DB proximity (low but non-negotiable).** Same host may hold `/var/lib/timestamping-svc/data/blockchain.db`. Automation must path-allowlist business DBs only.
---
## Required plan amendments (concrete text)
Merge the following into `plans/2026-10-03-libsql-multihost-architecture.md` (and patch D001 as noted).
### A1 — Amend Global Constraints (add bullets)
```text
- ZAPIER_USAGE / verae.zapier.usage is telemetry or export only — never usage/money SoT.
- Litestream (if used) backs up a single libSQL primary; never multi-writer fabric.
- Single meter/prepaid writer process owns usage_entries append + balance adjust;
other hosts call that API or use libSQL only through that writer’s transactions.
- Dual-run: libSQL primary for writes; file/SQLite fallback is read-only (or kill-switched).
- Auth (admin_users, portal_sessions): v1 remains edge-local zappier.db; not in verae_business.
- Product edge bind on 138: 127.0.0.1:3040 (zap.verae.ai); ZAPPIER_UPSTREAM defaults to
http://127.0.0.1:3130 for live lab smoke (not :3100 mock unless MOCK path intentional).
- Catalog docs that call account-balance “prepaid SoT” are stale after D006; Phase 0 patches them.
```
### A2 — Expand §5 NATS matrix (replace/extend rows)
```text
| verae.zapier.usage / ZAPIER_USAGE | JetStream optional | Telemetry/export only; NOT ledger SoT |
| verae.ts.completed, verae.ts.receipt | Core NATS only | Chain job-complete notify; never JetStream |
| verae.billing.* | Core request-reply (+ optional pub) | Facade over libSQL BusinessStore; one listener |
```
Also add to Phase 0: patch `docs/02-architecture/nats-subjects.md` with an explicit **Non-streams** section listing `verae.ts.*` as core-only and `ZAPIER_USAGE` as non-SoT.
### A3 — New §4.1 libSQL topology (insert after §4)
```text
### 4.1 Launch topology (3–5 machines)
- One libSQL primary (dedicated host or named primary among the 3–5).
- Replicas if any are read-only; writers always target primary URL (`LIBSQL_URL`).
- Backup: filesystem snapshot and/or Litestream of the primary only; tested restore before Phase 4.
- Fail closed: metered stamp paths return 503 if BusinessStore write unavailable
(no silent local SQLite fallback in production).
- Network: libSQL listen on private fabric only (same class as NATS 10.10.10.0/24); not public.
- Schema name: `verae_business` (confirm); auth schema stays out for v1.
```
### A4 — Phase 2 dual-run clarification
```text
- Writes: libSQL only once cutover flag BUSINESS_STORE=libsql is on.
- Fallback flag enables read-only SQLite/books for diagnosis, not adjust/recordUsage.
- Migrator verifies usage chain pre/post; refuses cutover if verifyChain fails.
- Account-balance either (a) becomes libSQL-backed facade for verae.billing.*, or
(b) is stopped after edge/CS/sales talk HTTP BusinessStore — pick one in Phase 3 tasking.
```
### A5 — Patch decision file D001 (Phase 0, same PR as plan amend)
Append to `decisions/D001-prepaid-sot.md`:
```text
## Addendum (2026-10-03, D006)
Durable prepaid/credits/usage/payments store is shared **libSQL**, not per-process
`books.json` or per-host `billing.db`. Single writer path still required. Edge
`balanceCents` remains a cache. Lab `books.json` / `billing.db` are migration sources.
```
### A6 — Conflict register: add C10–C18
Copy the new rows from this review’s register into plan §3.
### A7 — Acceptance criteria additions
```text
9. ZAPIER_USAGE is documented/configured as non-SoT; no service writes authoritative balances there.
10. Production metered paths fail closed if libSQL/BusinessStore writer is down.
11. Concurrent append contract test proves single-writer (or serialized) hash-chain safety.
12. Live env matrix documents :3040 edge, :3130 product upstream, :3100 mock, :3010 facade/retire.
13. D001 file contains D006 addendum; GETTING-STARTED / 01-system no longer call books.json SoT.
```
### A8 — Out of scope (reinforce)
```text
- Any JetStream binding for verae.ts.completed / verae.ts.receipt
- Editing /var/lib/timestamping-svc/data/blockchain.db or timestamping-svc JWT test results.db
- Multi-primary libSQL or Litestream multi-writer
- Moving portal auth to libSQL in the same cutover as billing (defer)
```
---
## What must not change (preserve list)
| Preserve | Why |
|----------|-----|
| **libSQL = business SoT** (D006) | Launch invariant for 3–5 hosts |
| **JetStream = work/archive only** | Not money/CRM database |
| **Core NATS only for `verae.ts.completed` / `verae.ts.receipt`** | Standing rule; no “reliability” override without explicit user words |
| **Digests-only stamp submit** (batch multihash / client hasher) | Live swagger + MCP; no file-byte stamp SoT |
| **No Go timestamping-svc SQLite edits** | `blockchain.db` and related stay local to that service |
| **D001 single prepaid writer path** (store upgraded by D006) | Prevents dual adjust |
| **D002** public `x-api-key`; `veraeUserId` join; JWT as `tokenRef` | Composition ownership |
| **D003** access planes as only public HTTP doors | Edge admin/Stripe stay non-door (loopback / tunnel) |
| **D004** leaf cannot touch billing | Policy allow-list |
| **D005** lan-134 off unless explicitly enabled | Fleet safety |
| **Credits product language**; ledger may keep `*_cents` column names | Avoid migration churn |
| **WAL (or libSQL equivalent) durability** on SQL stores | Existing edge invariant |
| **Deploy never rsyncs live `*.db*`** over production data | Known SQLITE_CANTOPEN / wipe failure mode |
| **Zapier never on NATS** | Hard rule in `01-system.md` |
| **Payment invoices ≠ timestamp receipts** | Composition |
| **Composition ownership:** metering/portal on zappier; stamp/NATS/hooks on middleware | Do not re-merge Stripe into middleware |
| **Existing zappier-edge Jest suite stays green** during driver swap | Regression gate |
| **Channel×segment rate card + credit headers** on metered/batch paths | Live product behavior |
---
## Sign-off checklist
Architecture signs when all boxes are true (or explicitly waived in writing on this file).
- [ ] Verdict understood: **approve-with-changes** (not full approve).
- [x] Plan amended with **A1–A8** (merged 2026-10-03) (or equivalent text merged).
- [x] Conflict register includes **C10–C18** with owners.
- [x] **D001** addendum committed pointing durable store at libSQL / D006.
- [ ] **nats-subjects.md** documents `verae.ts.*` core-only and `ZAPIER_USAGE` non-SoT.
- [ ] Env matrix published: `:3040` edge, `:3130` product MW, `:3100` mock MW, `:3010` balance facade.
- [ ] Single-writer / fail-closed / Litestream-backup-only rules in plan.
- [ ] Auth v1 = edge-local decision recorded (C13).
- [ ] Preserve list acknowledged by PM (no JetStream job-complete; no Go DB; digests-only).
- [ ] Phase 3/4 gated on writer consolidation + migrator chain verify.
- [ ] QA/PM may proceed to board + Task 1 **after** A1–A8 land; not before.
**Architecture provisional sign-off:** approve-with-changes — **A1–A8 merged into plan + D001**; remaining checklist items still open (nats-subjects, env matrix, GETTING-STARTED).
**Full architecture sign-off:** withheld until checklist complete.
---
## Evidence notes (live 138, 2026-10-03)
- `/opt/verae/zappier-edge`: `billing.db` + `zappier.db` (+ WAL/SHM), owner `verae-zapier`; service **active**.
- Edge env: `PORT=3040`, `HOST=127.0.0.1`, `ZAPPIER_UPSTREAM=http://127.0.0.1:3130`.
- Health: `:3040` zappier-edge ok; `:3100` middleware `mockVerae=true`, `natsConnected=false`; `:3130` middleware `mockVerae=false`, `natsConnected=true`; `:3010` account-balance still serving.
- `billing.db` tables: customers, usage_entries, channel_rates*, invoices*, price_endpoints, tiers; `zappier.db`: admin_users, portal_sessions.
- Access planes listening on `:3021–3025` (+ staff IAM/session); NATS client ports present on private/host binds.
---
*End of architecture review.*