# QA Baseline — Verae × Zapier Userflows 1–7 vs D006 Multi-Host **Role:** QA (evaluate only; no product feature implementation) **Date:** 2026-10-03 (UTC probe ~06:04–06:07) **Host:** `NS1.GEORGELAMBERT.ORG` / `70.88.205.138` **Inputs:** - Plan: [`../2026-10-03-libsql-multihost-architecture.md`](../2026-10-03-libsql-multihost-architecture.md) - Arch review: [`../reviews/2026-10-03-arch-review.md`](../reviews/2026-10-03-arch-review.md) (approve-with-changes; A1–A8 merged; full sign-off still open) - PM board: [`../reviews/2026-10-03-pm-board.md`](../reviews/2026-10-03-pm-board.md) - Decision: D006 (libSQL business SoT; JetStream = work only; core NATS for `verae.ts.*`) **Scope:** Readiness of critical userflows **1–7** for multi-host cutover. This is a **baseline** (WP-Q precursor). Per PM §5, formal WP-Q entry requires MG3; this pack documents why that gate is not met and what must change first. --- ## Summary judgment Lab doors and digests-only stamp submit on `:3040` → `:3130` work today, but **D006 multi-host readiness fails**: there is **no libSQL**, **no BusinessStore**, and **two live prepaid/ledger writers** (`billing.db` on edge + empty `books.json` behind `:3010`). Userflows 1, 3–6 are not E2E-proven; flow 2 is partially green; flow 7 has only weak 404 evidence for billing deny. Do **not** start Phase 4 host split or treat WP-Q as “in” until single-writer libSQL lands. --- ## Live lab evidence (2026-10-03) ### Health matrix | Port | Role | `/health` | Notes | |------|------|-----------|-------| | `:3040` | zappier-edge | **200** `{"ok":true,"role":"zappier-edge"}` | Bind `127.0.0.1`; systemd `zappier-edge.service` | | `:3100` | middleware (lab/mock) | **200** `mockVerae:true`, `natsConnected:false` | Catalog/default; **not** product upstream | | `:3130` | middleware (product) | **200** `mockVerae:false`, `natsConnected:true` | Edge `ZAPPIER_UPSTREAM` target | | `:3010` | zappier-account-balance | **200** `nats:false` | Loopback; **not** a systemd unit; fleet-spawned | | `:3021`–`:3025` | access planes | **200** each | web / api / leaf / zapier / staff | ### Env / upstream - Edge secrets (`/SSD2/secrets/zap-verae/zappier-edge.env`): `PORT=3040`, **`ZAPPIER_UPSTREAM=http://127.0.0.1:3130`** (confirms arch C11 / plan A1). - Product MW env: `PORT=3130`, `HOST=127.0.0.1`, `MOCK_VERAE=false`, `NATS_URL=nats://127.0.0.1:4222`. - Account-balance (pid 4179): cwd under moved stack package; `FLEET_STATE_DIR=.../verae-fleet/data/runtime/account-balance-0`; `NATS_URL` lists `10.10.10.21–23` but health reports **`nats:false`**. ### Digest-only stamp (flow 2 probe) - Digest: `sha256("qa-baseline")` = `edf8f682e42104442bb4637c665f50ada9873fa2fe6958ae00a6c05b2e5181cb` - `POST /v1/timestamps` with `x-api-key: key-ada`, `x-billing-channel: api`, body `{sha256s:[…]}` → **HTTP 202**, `created:1` then idempotent `existing:1`, credit headers present (`X-Verae-Credits-*`). - Items-shaped digest-only body also **202**. - Single `POST /v1/timestamp` → **202** with `jobId`, but receipt `completedAt: "0001-01-01T00:00:00Z"` (placeholder / incomplete receipt shape for async complete). ### Dual-writer / C16 (flows 1, 4–6) | Store | Path | Observation | |-------|------|-------------| | Edge ledger | `/opt/verae/zappier-edge/billing.db` | **Exists** (WAL). `usage_entries` count **3** for `cust_1` @ 10 each after probes; `customers.balance_cents` for Ada = **0** | | Account-balance books | `FLEET_STATE_DIR/.../data/books.json` | **Exists**, empty: `prepaid:{}`, `usage:[]`, `credits:[]`, `payments:[]` | | Statement HTTP | `GET :3010/statement/cust_1` | **200** `{prepaidCents:0, usage:[], …}` — **does not see** edge `usage_entries` | | Edge response | batch stamp | `balanceCredits:0`, period spend headers show **30** after three 10-credit meters | **C16 confirmed live:** edge writes metered usage into `billing.db`; account-balance serves a separate empty `books.json`. No shared SoT. Splitting hosts now would fork money. ### Go timestamping DB (C9) - `/var/lib/timestamping-svc/data/blockchain.db` **exists** (~31 MB, mtime 2026-09-25). - **Not modified** by this QA probe (read `ls`/`stat` only). ### libSQL / BusinessStore / harness - No `LIBSQL_URL` / `sqld` / libSQL container observed on 138 for business data. - Local clone `/Users/marchon/apps/verae/zappier-edge/src/db/`: **no** `business-store.ts` (`NO_BUSINESS_STORE`). - `verae-zapier-simulator` exists (messaging/trace) but is **not** the WP-P automated harness for flows 1–7. - Arch checklist still open: `nats-subjects.md` `verae.ts.*` / `ZAPIER_USAGE` non-SoT; published env matrix; GETTING-STARTED SoT language. - Host `nats stream ls` to `10.10.10.21:4222` **timed out** from this probe (MW still reports `natsConnected:true` via loopback `:4222` proxy). STATUS still lists product stream **`ZAPIER_USAGE`** (C10 risk). ### Leaf D004 (flow 7 probe) - `:3023/health` **200** leaf plane up. - `POST :3023/v1/billing/adjust` → **404** `not found` (route absent). This is **not** yet an automated policy deny of `verae.billing.balance.adjust` (PM WP-D DoD). --- ## Userflow readiness (1–7) | # | Userflow | Today | Blocker vs D006 | |---|----------|-------|-----------------| | 1 | Signup → API key → Statement | Partial: seed keys exist (`key-ada`…); statement via `:3010` empty/out of band with edge ledger | Dual SoT; no portal E2E harness; account-balance `nats:false` | | 2 | Stamp digests single + batch + credit headers | **Mostly green** on lab (`:3040`→`:3130`, 202, headers, idempotent existing) | Still local SQLite meter; zero balance still meters; receipt placeholder on single path; not libSQL | | 3 | Async complete → Zapier REST Hook | **Not proven** | `completedAt` zero-time on single stamp; no WP-P watch→events→webhooks.deliver E2E; Platform push still open in STATUS | | 4 | CS credit adjust → portal balance | **Broken / dual path** | Adjust would hit books/`verae.billing.*` while meter hits `billing.db`; nats false on balance service | | 5 | Sales quote → meter match | **Unproven** | Rate tables live in edge SQLite; sales package not wired to shared SoT | | 6 | Accounting export consistency | **Fails if run today** | Export cannot reconcile empty books vs non-empty `usage_entries` | | 7 | Leaf stamp OK / billing deny | Health only | Need authz allow-list tests; 404 ≠ D004 proof | **PM QA entry (board §5):** MG3 single writer, regression green, NATS rules verified, harness skeleton, D004 automated, migrator tip report, Go DB untouched note, docs C2/C3/C5/C6 — **not met** (only Go-untouched + partial digests-only + doors up). --- ## a) What to change | # | Component | Problem | Evidence | Severity | Type | |---|-----------|---------|----------|----------|------| | 1 | Prepaid / ledger SoT (C7/C16) | Two writers: edge `billing.db` meters usage; `:3010` serves empty `books.json` statement | usage_n=3 vs statement `usage:[]`; both processes up | **critical** | design flaw / dual writer | | 2 | Missing libSQL + BusinessStore (D006 Phases 1–2) | No shared business DB or interface; multi-host impossible without per-host SQLite fork | No `business-store.ts`; no LIBSQL on 138 | **critical** | missing foundation | | 3 | Account-balance NATS / facade | Health `nats:false` while CS/portal expect `verae.billing.*` | `:3010/health` JSON; env has NATS_URL but not connected | **high** | ops / wiring | | 4 | Userflow E2E harness (WP-P) | Flows 1–7 not automated; simulator ≠ harness | No `plans/qa` flow reports; simulator README is messaging-only | **high** | missing gate | | 5 | Async complete / receipts (flow 3) | Single-stamp receipt shows epoch-zero `completedAt`; REST Hook path unproven | curl body `0001-01-01T00:00:00Z` | **high** | incomplete product path | | 6 | Zero-balance metering (flows 2/4) | Ada `balance_cents=0` still stamps and accumulates period spend | customers row + 202 responses + `X-Verae-Credits-Balance: 0` | **medium** | missing gate / prepaid policy | | 7 | Dual middleware confusion (C11) | `:3100` mock vs `:3130` product; docs/scripts can point wrong | health JSON both ports; edge env → 3130 | **medium** | doc/ops drift | | 8 | D004 leaf deny proof (flow 7) | Only 404 on invent-ed HTTP path; no automated NATS/HTTP deny | curl 404; WP-D tests not evidenced on lab | **medium** | missing test | | 9 | `ZAPIER_USAGE` / nats-subjects docs (C5/C10) | STATUS still lists usage stream as product; catalog subjects omit core-only `verae.ts.*` | STATUS.md streams table; arch checklist open | **medium** | doc drift / SoT confusion | | 10 | Env matrix / Phase 0 leftovers | Arch full sign-off blocked; GETTING-STARTED may still call books SoT | arch review checklist unchecked items | **medium** | doc freeze incomplete | | 11 | Account-balance not systemd-managed | Process from fleet overlay / moved path; restart/ownership unclear vs edge unit | no `*account-balance*.service`; pid under `marchon` | **low** | ops hygiene | ### What is working (preserve) - Digests-only batch + single stamp on product edge `:3040` with `key-ada` and credit headers. - Product upstream correctly set to **`:3130`** (not mock `:3100`). - Access-plane health on `:3021–3025`. - Edge split DBs present and WAL: `billing.db` (business tables) + `zappier.db` (auth). - Idempotent re-stamp (`created:0`, `existing:1`, spend 0). - Standing rules respected in this probe: **no** JetStream for `verae.ts.*`; **no** writes to `blockchain.db`. - Plan + A1–A8 + D001 addendum + D006 direction are the right target. --- ## b) What to change it to | # | Component | Target behavior | Acceptance criteria | Constraints | |---|-----------|-----------------|---------------------|-------------| | 1 | Single prepaid/ledger writer | One authoritative adjust + `recordUsage` path on shared store; other path read-only or stopped | Grep: no prod writes to `books.json` / per-host `billing.db` as SoT; adjust on CS visible on portal statement and edge balance headers | D001 single writer; do not dual-append hash chain | | 2 | libSQL BusinessStore | `LibsqlBusinessStore` (+ SqliteFile for CI) behind `BusinessStore`; lab migrator from `billing.db` + `books.json` | Contract tests green; migrator tip hash verify; `BUSINESS_STORE=libsql` writes only to libSQL | Auth stays edge-local `zappier.db` (C13); fail closed (503) if writer down | | 3 | Billing facade | `:3010` (or successor) is libSQL-backed listener for `verae.billing.*` **or** folded into edge HTTP API — pick one (plan Phase 3) | Health `nats:true` (if NATS facade kept); statement usage matches `usage_entries` | Keep subjects stable for CS/sales | | 4 | Harness 1–7 | Automated runner under simulator/harness producing `plans/qa/` report | Flows 1,2,4 runnable under MOCK; then 3,5,6,7; exit 0 artifact | Zapier cloud push optional for first MOCK pass | | 5 | Async complete | Real completion times / receipts; `jobs.watch` → `jobs.events` → `webhooks.deliver` | E2E: jobId reaches terminal event + hook delivery (or recorded attempt) | `verae.ts.completed` / `verae.ts.receipt` remain **core NATS only** | | 6 | Prepaid gate | Metering respects prepaid/credits policy (402 or configured allow) | Seed funded customer; zero-balance path returns expected product code | Do not rewrite historical ledger mid-test | | 7 | Env matrix | Documented `:3040` / `:3130` product / `:3100` mock / `:3010` facade | STATUS + ops GETTING-STARTED + plan cite same matrix | Do not “fix” smoke by pointing edge at 3100 | | 8 | D004 | Leaf cannot call balance adjust (HTTP door + NATS authz) | `verae-access-leaf` / authz tests green; live deny ≠ bare 404 | Leaf may still stamp/authz | | 9 | NATS docs | `nats-subjects.md` Non-streams: `verae.ts.*` core-only; `ZAPIER_USAGE` telemetry non-SoT | Arch checklist boxes checked; STATUS wording demotes usage stream | No JetStream bind for job-complete | | 10 | Phase 0 close | C2/C3/C5/C6 closed or owned; GETTING-STARTED SoT → libSQL | Arch full sign-off or explicit waivers | — | | 11 | Balance process ops | systemd (or fleet-documented) unit with clear state dir | `systemctl status` shows balance facade; books path documented until migration | Deploy never rsyncs `*.db*` | --- ## c) How to implement Aligned with PM DAG; **do not skip merge gates**. ```mermaid flowchart TD P0[Phase 0 docs/env/NATS freeze] --> P1[Phase 1 BusinessStore TDD] P1 --> P2[Phase 2 migrator + dual-read] P2 --> MG2[MG2 writers unlock] MG2 --> P3[Phase 3 single writer rewire] P3 --> MG3[MG3 single writer proof] MG3 --> P4[Phase 4 multi-host LIBSQL_URL] MG3 --> P5[Phase 5 harness 1-7] P4 --> P5 P5 --> MG4[MG4 green] MG4 --> P6[Phase 6 QA whattochange loop] ``` ### Phase 0 — Conflict / docs freeze (feeding; ~1–2 days) - **Goal:** Close open arch checklist items that confuse operators and agents. - **Changes:** Publish env matrix; patch `nats-subjects.md`; demote `ZAPIER_USAGE` in STATUS; fix overview `zmw_` / digests examples (C3/C6); GETTING-STARTED SoT → libSQL. - **Validation:** Arch checklist; `rg` for stale SoT claims. - **Effort:** 1–2 agent-days (WP-A/B/C). ### Phase 1 — BusinessStore + libSQL skeleton (critical; WP-F/G/H) - **Goal:** Interface + SqliteFile + Libsql drivers with contract tests (including hash-chain). - **Changes:** `zappier-edge` `src/db/business-store.ts`, contract tests, CI libSQL. - **Validation:** Jest contract green both drivers (MG1). - **Effort:** ~3 days serial. ### Phase 2 — Migrate + dual-read (WP-J/K) - **Goal:** Import live `billing.db` + empty/legacy `books.json`; verify chain tip; reads dual-run; **writes still locked**. - **Validation:** Migrator report archived under `plans/qa/`; MG2. - **Effort:** ~2 days. ### Phase 3 — Rewire writers (WP-L/M/N) — only after MG2 - **Goal:** Edge meter + account-balance + CS/sales/acct → one BusinessStore; remove second writer. - **Validation:** Statement usage == ledger; CS adjust visible on portal/edge headers; MG3 grep proof. - **Effort:** ~3–4 days. ### Phase 4 — Multi-host (WP-O) — only after MG3 - **Goal:** `LIBSQL_URL` primary; edge/MW/CS on separate hosts possible. - **Validation:** Cross-host adjust/meter/statement same balance. ### Phase 5 — Harness (WP-P) - **Goal:** Automate flows 1–7; first pass MOCK_VERAE OK. - **Validation:** Report under `plans/qa/`; MG4. ### Phase 6 — QA loop (WP-Q) - **Goal:** Break → whattochange → fix → retest until **two consecutive** full greens (project buffer). **Explicit non-goals until gates pass:** JetStream for `verae.ts.*`; editing `blockchain.db`; Litestream multi-writer; multi-host split before MG3. --- ## d) Expected results / still incomplete ### After implementation (realistic) | Metric / behavior | Today (baseline) | Expected after Phases 0–5 | |-------------------|------------------|---------------------------| | Business SoT | Split `billing.db` + empty `books.json` | One libSQL `verae_business` | | Flow 2 digests stamp | 202 + headers on single host SQLite | Same UX on libSQL meter writer | | Flow 1/4 statement vs meter | Divergent (usage 3 vs `[]`) | Identical customer/usage/balance | | Flow 3 async → hook | Unproven; zero `completedAt` | Proven under harness (MOCK then optional live) | | Flow 5/6 quote & export | Unproven / inconsistent | Same rate tables + export = ledger | | Flow 7 D004 | 404 only | Automated deny | | Multi-host money | Unsafe | Safe only post-MG3 | | `blockchain.db` | Untouched | Still untouched | ### Still missing, incorrect, or incomplete (even after this todo list) | Gap | Why it persists | |-----|-----------------| | Zapier Platform public `push` + live workspace Zaps | Needs operator Zapier login (STATUS); out of first cutover buffer | | Live `api.veraetime.net` seal proof beyond lab MW | Creds / OpenAPI gaps (hash lookup 501); separate from libSQL SoT | | NATS nkeys/mTLS | Cluster isolation today is `vmbr1`, not accounts | | Portal auth on multi-edge sticky sessions | C13: auth stays local `zappier.db` for v1 | | libSQL HA / failover RPO | Launch = one primary + backup; not multi-primary | | Peergos / Go timestamping redesign | Out of scope | | Full architecture sign-off | Depends on remaining checklist (docs), not only code | ### Success tiers | Tier | Criteria | |------|----------| | **Minimum viable (this baseline exit)** | Dual-writer named daily owner; env matrix published; harness skeleton for 1/2/4; no host split | | **MG3 / single-writer** | libSQL SoT; one adjust path; statement == meter | | **Production-ready multi-host** | MG3 + WP-O + flows 1–7 twice green; fail-closed meter; D004 automated | | **Complete** | + Platform push + live Verae seal + nkeys — separate projects | --- ## e) Prioritized todo list with PreReqs ``` Item 1 — [P0] — Freeze dual-writer authority for lab (C16 runbook) Document which process may adjust/meter each day until MG3; forbid CS adjust against books while edge meters billing.db without a sync plan. PreReqs: (none) Item 2 — [P0] — Publish live env matrix (:3040 / :3130 / :3100 / :3010) Add to STATUS + ops docs; match edge ZAPPIER_UPSTREAM=http://127.0.0.1:3130 evidence. PreReqs: (none) Item 3 — [P0] — BusinessStore interface + failing contract tests (WP-F) Create business-store.ts + contract suite (customers, usage chain, rates, invoices, balances). PreReqs: (Item 2) Item 4 — [P0] — SqliteFileBusinessStore adapter (WP-G) Wrap existing edge repos; keep Jest green. PreReqs: (Item 3) Item 5 — [P0] — LibsqlBusinessStore + CI libSQL (WP-H) Driver + reachable LIBSQL_URL in CI/lab; MG1. PreReqs: (Item 4) Item 6 — [P0] — Migrator billing.db + books.json → libSQL with chain verify (WP-K) Refuse cutover on verifyChain failure; archive tip report under plans/qa/; MG2. PreReqs: (Item 5) Item 7 — [P0] — Rewire edge meter to BusinessStore only (WP-L) Remove production writes that treat local billing.db as SoT once flag on. PreReqs: (Item 6) Item 8 — [P0] — Fold or rewire account-balance to libSQL facade (WP-M) One verae.billing.* listener; empty books.json import-only; nats connected if facade kept. PreReqs: (Item 7) Item 9 — [P1] — Rewire CS / sales / accounting-export to same store (WP-N) Credit adjust, rates, export read libSQL; MG3 proof. PreReqs: (Item 8) Item 10 — [P1] — Patch nats-subjects + demote ZAPIER_USAGE (C5/C10) Document verae.ts.* core-only; usage stream telemetry-only in STATUS/catalog. PreReqs: (none) Item 11 — [P1] — Automate D004 leaf billing deny (WP-D) Tests prove leaf cannot balance.adjust; replace 404-only smoke. PreReqs: (none) Item 12 — [P1] — Userflow harness skeleton flows 1, 2, 4 (WP-P start) MOCK_VERAE runnable report artifact under plans/qa/. PreReqs: (Item 9) Item 13 — [P1] — Extend harness to flows 3, 5, 6, 7 Async hook path, quote match, export consistency, leaf deny E2E; MG4. PreReqs: (Item 12 + Item 11) Item 14 — [P2] — Fix async receipt completedAt / completion wiring Eliminate 0001-01-01 placeholders on product stamp path used by Zapier wait/hook. PreReqs: (Item 12) Item 15 — [P2] — Prepaid zero-balance policy gate Define and test 402 vs allow-with-debt; seed funded fixture for harness. PreReqs: (Item 7) Item 16 — [P2] — Multi-host LIBSQL_URL topology (WP-O) Only after MG3; cross-host meter/adjust/statement. PreReqs: (Item 9) Item 17 — [P2] — systemd/fleet hygiene for account-balance facade Unit or documented fleet runtime; stable state dir until books retired. PreReqs: (Item 8) Item 18 — [P3] — Phase 0 leftover docs (C2/C3/C6, GETTING-STARTED) Close arch full sign-off checklist items not covered by Item 2/10. PreReqs: (Item 2 + Item 10) ``` **Dependency order (for “Do all”):** Item 1 ∥ Item 2 ∥ Item 10 ∥ Item 11 → Item 3 → Item 4 → Item 5 → Item 6 → Item 7 → Item 8 → Item 9 → Item 12 → (Item 13 ∥ Item 14 ∥ Item 15) → Item 16 → Item 17; Item 18 can trail after Item 2+10. **QA note:** Formal WP-Q churn should wait for **Item 9 (MG3)** + **Item 12**. This baseline pack is the readiness report that unlocks engineering focus on Items 1–9. --- ## Probe command index (reproducible) ```bash ssh -i ~/.ssh/id_ed25519 -o IdentitiesOnly=yes marchon@70.88.205.138 \ 'for p in 3040 3100 3130 3010 3021 3022 3023 3024 3025; do curl -sS -m 5 http://127.0.0.1:$p/health; echo; done' # Digest-only batch (do not use production customer keys outside lab seeds) DIGEST=$(printf "qa-baseline" | sha256sum | awk "{print \$1}") curl -sS -D - -X POST http://127.0.0.1:3040/v1/timestamps \ -H "content-type: application/json" -H "x-api-key: key-ada" \ -H "x-billing-channel: api" \ -d "{\"sha256s\":[\"$DIGEST\"]}" # Dual-writer check (read-only) sqlite3 /opt/verae/zappier-edge/billing.db "SELECT COUNT(*) FROM usage_entries;" curl -sS http://127.0.0.1:3010/statement/cust_1 # blockchain.db: ls/stat only — never migrate/edit ``` --- *End of QA baseline. No product code was changed by this review.*