zapier-decisions/plans/reviews/2026-10-03-arch-review.md

16 KiB
Raw Blame History

Architecture Review — Verae × Zapier Multi-Host / libSQL Plan

Plan under review: plans/2026-10-03-libsql-multihost-architecture.md
Reviewer role: Architecture
Date: 2026-10-03
Inputs: plan; /tmp/zapier-system-docs-map.md (mirrored plans/2026-10-03-system-docs-map.md); D001–D006; catalog composition.md, nats-subjects.md, overview/01-system.md, packages/verae-ops/GETTING-STARTED.md; live 138 /opt/verae/zappier-edge (billing.db + zappier.db, zappier-edge.service on 127.0.0.1:3040).


Verdict

approve-with-changes

Direction is correct and aligned with D006: libSQL as business SoT, JetStream for work queues only, core NATS for verae.ts.completed / verae.ts.receipt, digests-only stamps, Go blockchain.db out of scope. The plan is not blocked on SoT choice.

It is not a clean approve: multi-host money durability, single-writer sequencing, stream/subject demotions, live port matrix, and decision-doc amendments are underspecified. Build of Phases 1–2 (interface + migrator skeleton) may start after the amendments in §Required plan amendments are merged into the plan (and D001 text is patched). Do not start Phase 3 rewire or Phase 4 multi-host split until those amendments are in the plan and the conflict register rows below are closed or explicitly deferred with owner + date.


Expanded conflict register

Inherited from plan §3 (C1–C9) plus new findings from catalog + live 138.

ID Conflict Severity Resolution (required)
C1 D001 books.json SoT vs live billing.db vs D006 libSQL High libSQL is SoT; migrate books + billing.db; amend D001 file to point at D006 (not only the plan narrative)
C2 README “imported, not yet wired” vs STATUS lab up Med STATUS + live doors = operational truth; patch README in Phase 0
C3 overview zmw_ Bearer vs composition x-api-key Med Public = x-api-key (D002/composition); fix overview/models
C4 Edge ports :3000 / :13000 / live :3040 (zap.verae.ai) High Publish env matrix; lab doors proxy loopback; product edge remains 127.0.0.1:3040 until deliberately moved
C5 JetStream jobs.events vs ban on JetStream for job-complete High Different subjects: verae.zapier.jobs.events (Zapier pipeline, JetStream OK) ≠ verae.ts.* (core NATS only). Document both in nats-subjects + standing rule
C6 TimestampRequest “Hello world” data vs digests-only Med Digests-only + multihash batch; sync catalog models to live swagger
C7 Dual prepaid writers (edge SQLite + account-balance) High One writer API on libSQL; edge caches only; deprecate/fold mailbox
C8 Per-host SQLite on 3–5 machines High Forbidden for business data; libSQL shared
C9 Go blockchain.db in scope creep Blocker if violated Out of scope — do not touch
C10 Catalog/STATUS stream ZAPIER_USAGE vs libSQL usage ledger High Demote ZAPIER_USAGE / verae.zapier.usage to telemetry/export only (or remove). Must never be treated as money/usage SoT. Plan §5 must say this explicitly
C11 Live dual middleware: catalog/ops :3100 (MOCK, natsConnected=false) vs edge env ZAPPIER_UPSTREAM=http://127.0.0.1:3130 (live mockVerae=false, natsConnected=true) High Env matrix must list both; product smoke uses 3130; do not “fix” by pointing edge at 3100 without intent
C12 D001 decision file still says books.json is sole prepaid writer with no D006 amendment High Patch decisions/D001-prepaid-sot.md (addendum: durable store = libSQL; books.json lab legacy)
C13 Auth store undecided (admin_users / portal_sessions edge-local vs libSQL) Med Decide before Phase 3: v1 keep auth in edge-local zappier.db (or per-edge file); business tables only in libSQL. Document; revisit if edges are multi-host without sticky sessions
C14 Concurrent usage_entries hash-chain append across hosts High (launch) Single logical meter writer (one process or libSQL-serialized exclusive txn path). No competing multi-host BEGIN IMMEDIATE without proven libSQL semantics. Dual-run must not dual-append
C15 GETTING-STARTED / 01-system.md still call account-balance prepaid SoT Med Phase 0 doc patch: SoT → libSQL; account-balance becomes facade or retired
C16 Live account-balance still up on :3010 while edge also owns billing.db ledger High Cutover runbook must name which process is authoritative each day of dual-run; forbid two adjust paths
C17 Optional Litestream vs multi-writer temptation Med Litestream = backup/DR of one primary only (D006 non-goal). Plan must forbid Litestream-as-fabric
C18 Digests-only vs any remaining “upload file then stamp” Zapier creates Med Client hasher + digests-only batch remains the supported path; reject raw-data stamp examples in public docs

Risks for 3–5 machine launch

  1. Money split-brain (critical). Today prepaid/ledger truth is fragmented (books.json via :3010, live billing.db on edge, caches). Splitting planes across hosts before one libSQL writer lands will diverge balances, statements, and CS credits. Mitigate: no Phase 4 host split until Phase 3 removes second writers and dual-run is read-only fallback.

  2. Hash-chain integrity under concurrency (critical). usage_entries depends on tip-linked prev_hash/row_hash. Multiple edge replicas opening libSQL without an exclusive append protocol will fork or fail the chain. Mitigate: one meter writer service (or proven libSQL write serialization + contract tests for concurrent append).

  3. libSQL HA underspecified (high). Plan says “dedicated host or primary of 3–5” but not: primary placement, replica/read role, failover RPO/RTO, backup (Litestream or native), network ACL, and what happens when libSQL is down (fail closed on meter vs stale cache). For launch, pick one primary + backup; do not invent multi-primary.

  4. JetStream mistaken for ledger (high). ZAPIER_USAGE already exists in STATUS with replicas=3. Operators/agents will “persist usage on JetStream” under reliability pressure. Standing rule + plan §5 must demote it; CI/docs grep for usage-as-SoT.

  5. Job-complete subject confusion (high). Zapier pipeline verae.zapier.jobs.events (JetStream) vs Verae chain notify verae.ts.completed/verae.ts.receipt (core only). A “make complete more reliable” change must not JetStream verae.ts.* (standing rule). Plan C5 is correct; nats-subjects.md still omits verae.ts.* — patch it.

  6. Wrong upstream in smoke (medium). Edge points at :3130; lab docs emphasize :3100. Multi-host deploy scripts that copy catalog defaults will meter against mock middleware and lie about chain connectivity.

  7. Auth sticky-session / multi-edge (medium). If auth stays in local zappier.db and two edges serve portal/admin, sessions break. Acceptable for v1 single edge; call out as launch constraint.

  8. Migration tip verification false confidence (medium). Lab usage_entries tip is tiny (observed count=1 on 138). Migrator tests need adversarial fixtures (long chain, tamper mid-chain, empty DB), not only live copy.

  9. Deploy rsync DB wipe (medium, known). Standing ops: never rsync *.db* onto /opt/verae/zappier-edge. libSQL cutover must use migrator + URL switch, not file copy into prod workdir via deploy.

  10. NATS billing request-reply after mailbox retirement (medium). verae.billing.* subjects remain useful as a facade; if account-balance dies without a replacement listener, CS/sales/portal statement paths break even if libSQL is healthy. Plan must keep a single billing facade process.

  11. Credits vs cents naming (low). Product credits; columns *_cents. Fine if documented; do not “fix” by rewriting historical rows mid-migration.

  12. Go timestamping DB proximity (low but non-negotiable). Same host may hold /var/lib/timestamping-svc/data/blockchain.db. Automation must path-allowlist business DBs only.


Required plan amendments (concrete text)

Merge the following into plans/2026-10-03-libsql-multihost-architecture.md (and patch D001 as noted).

A1 — Amend Global Constraints (add bullets)

- ZAPIER_USAGE / verae.zapier.usage is telemetry or export only — never usage/money SoT.
- Litestream (if used) backs up a single libSQL primary; never multi-writer fabric.
- Single meter/prepaid writer process owns usage_entries append + balance adjust;
  other hosts call that API or use libSQL only through that writer’s transactions.
- Dual-run: libSQL primary for writes; file/SQLite fallback is read-only (or kill-switched).
- Auth (admin_users, portal_sessions): v1 remains edge-local zappier.db; not in verae_business.
- Product edge bind on 138: 127.0.0.1:3040 (zap.verae.ai); ZAPPIER_UPSTREAM defaults to
  http://127.0.0.1:3130 for live lab smoke (not :3100 mock unless MOCK path intentional).
- Catalog docs that call account-balance “prepaid SoT” are stale after D006; Phase 0 patches them.

A2 — Expand §5 NATS matrix (replace/extend rows)

| verae.zapier.usage / ZAPIER_USAGE | JetStream optional | Telemetry/export only; NOT ledger SoT |
| verae.ts.completed, verae.ts.receipt | Core NATS only | Chain job-complete notify; never JetStream |
| verae.billing.* | Core request-reply (+ optional pub) | Facade over libSQL BusinessStore; one listener |

Also add to Phase 0: patch docs/02-architecture/nats-subjects.md with an explicit Non-streams section listing verae.ts.* as core-only and ZAPIER_USAGE as non-SoT.

A3 — New §4.1 libSQL topology (insert after §4)

### 4.1 Launch topology (3–5 machines)

- One libSQL primary (dedicated host or named primary among the 3–5).
- Replicas if any are read-only; writers always target primary URL (`LIBSQL_URL`).
- Backup: filesystem snapshot and/or Litestream of the primary only; tested restore before Phase 4.
- Fail closed: metered stamp paths return 503 if BusinessStore write unavailable
  (no silent local SQLite fallback in production).
- Network: libSQL listen on private fabric only (same class as NATS 10.10.10.0/24); not public.
- Schema name: `verae_business` (confirm); auth schema stays out for v1.

A4 — Phase 2 dual-run clarification

- Writes: libSQL only once cutover flag BUSINESS_STORE=libsql is on.
- Fallback flag enables read-only SQLite/books for diagnosis, not adjust/recordUsage.
- Migrator verifies usage chain pre/post; refuses cutover if verifyChain fails.
- Account-balance either (a) becomes libSQL-backed facade for verae.billing.*, or
  (b) is stopped after edge/CS/sales talk HTTP BusinessStore — pick one in Phase 3 tasking.

A5 — Patch decision file D001 (Phase 0, same PR as plan amend)

Append to decisions/D001-prepaid-sot.md:

## Addendum (2026-10-03, D006)

Durable prepaid/credits/usage/payments store is shared **libSQL**, not per-process
`books.json` or per-host `billing.db`. Single writer path still required. Edge
`balanceCents` remains a cache. Lab `books.json` / `billing.db` are migration sources.

A6 — Conflict register: add C10–C18

Copy the new rows from this review’s register into plan §3.

A7 — Acceptance criteria additions

9. ZAPIER_USAGE is documented/configured as non-SoT; no service writes authoritative balances there.
10. Production metered paths fail closed if libSQL/BusinessStore writer is down.
11. Concurrent append contract test proves single-writer (or serialized) hash-chain safety.
12. Live env matrix documents :3040 edge, :3130 product upstream, :3100 mock, :3010 facade/retire.
13. D001 file contains D006 addendum; GETTING-STARTED / 01-system no longer call books.json SoT.

A8 — Out of scope (reinforce)

- Any JetStream binding for verae.ts.completed / verae.ts.receipt
- Editing /var/lib/timestamping-svc/data/blockchain.db or timestamping-svc JWT test results.db
- Multi-primary libSQL or Litestream multi-writer
- Moving portal auth to libSQL in the same cutover as billing (defer)

What must not change (preserve list)

Preserve Why
libSQL = business SoT (D006) Launch invariant for 3–5 hosts
JetStream = work/archive only Not money/CRM database
Core NATS only for verae.ts.completed / verae.ts.receipt Standing rule; no “reliability” override without explicit user words
Digests-only stamp submit (batch multihash / client hasher) Live swagger + MCP; no file-byte stamp SoT
No Go timestamping-svc SQLite edits blockchain.db and related stay local to that service
D001 single prepaid writer path (store upgraded by D006) Prevents dual adjust
D002 public x-api-key; veraeUserId join; JWT as tokenRef Composition ownership
D003 access planes as only public HTTP doors Edge admin/Stripe stay non-door (loopback / tunnel)
D004 leaf cannot touch billing Policy allow-list
D005 lan-134 off unless explicitly enabled Fleet safety
Credits product language; ledger may keep *_cents column names Avoid migration churn
WAL (or libSQL equivalent) durability on SQL stores Existing edge invariant
Deploy never rsyncs live *.db* over production data Known SQLITE_CANTOPEN / wipe failure mode
Zapier never on NATS Hard rule in 01-system.md
Payment invoices ≠ timestamp receipts Composition
Composition ownership: metering/portal on zappier; stamp/NATS/hooks on middleware Do not re-merge Stripe into middleware
Existing zappier-edge Jest suite stays green during driver swap Regression gate
Channel×segment rate card + credit headers on metered/batch paths Live product behavior

Sign-off checklist

Architecture signs when all boxes are true (or explicitly waived in writing on this file).

  • Verdict understood: approve-with-changes (not full approve).
  • Plan amended with A1–A8 (merged 2026-10-03) (or equivalent text merged).
  • Conflict register includes C10–C18 with owners.
  • D001 addendum committed pointing durable store at libSQL / D006.
  • nats-subjects.md documents verae.ts.* core-only and ZAPIER_USAGE non-SoT.
  • Env matrix published: :3040 edge, :3130 product MW, :3100 mock MW, :3010 balance facade.
  • Single-writer / fail-closed / Litestream-backup-only rules in plan.
  • Auth v1 = edge-local decision recorded (C13).
  • Preserve list acknowledged by PM (no JetStream job-complete; no Go DB; digests-only).
  • Phase 3/4 gated on writer consolidation + migrator chain verify.
  • QA/PM may proceed to board + Task 1 after A1–A8 land; not before.

Architecture provisional sign-off: approve-with-changes — A1–A8 merged into plan + D001; remaining checklist items still open (nats-subjects, env matrix, GETTING-STARTED).
Full architecture sign-off: withheld until checklist complete.


Evidence notes (live 138, 2026-10-03)

  • /opt/verae/zappier-edge: billing.db + zappier.db (+ WAL/SHM), owner verae-zapier; service active.
  • Edge env: PORT=3040, HOST=127.0.0.1, ZAPPIER_UPSTREAM=http://127.0.0.1:3130.
  • Health: :3040 zappier-edge ok; :3100 middleware mockVerae=true, natsConnected=false; :3130 middleware mockVerae=false, natsConnected=true; :3010 account-balance still serving.
  • billing.db tables: customers, usage_entries, channel_rates*, invoices*, price_endpoints, tiers; zappier.db: admin_users, portal_sessions.
  • Access planes listening on :3021–3025 (+ staff IAM/session); NATS client ports present on private/host binds.

End of architecture review.