21 KiB
QA Baseline — Verae × Zapier Userflows 1–7 vs D006 Multi-Host
Role: QA (evaluate only; no product feature implementation)
Date: 2026-10-03 (UTC probe ~06:04–06:07)
Host: NS1.GEORGELAMBERT.ORG / 70.88.205.138
Inputs:
- Plan:
../2026-10-03-libsql-multihost-architecture.md - Arch review:
../reviews/2026-10-03-arch-review.md(approve-with-changes; A1–A8 merged; full sign-off still open) - PM board:
../reviews/2026-10-03-pm-board.md - Decision: D006 (libSQL business SoT; JetStream = work only; core NATS for
verae.ts.*)
Scope: Readiness of critical userflows 1–7 for multi-host cutover. This is a baseline (WP-Q precursor). Per PM §5, formal WP-Q entry requires MG3; this pack documents why that gate is not met and what must change first.
Summary judgment
Lab doors and digests-only stamp submit on :3040 → :3130 work today, but D006 multi-host readiness fails: there is no libSQL, no BusinessStore, and two live prepaid/ledger writers (billing.db on edge + empty books.json behind :3010). Userflows 1, 3–6 are not E2E-proven; flow 2 is partially green; flow 7 has only weak 404 evidence for billing deny. Do not start Phase 4 host split or treat WP-Q as “in” until single-writer libSQL lands.
Live lab evidence (2026-10-03)
Health matrix
| Port | Role | /health |
Notes |
|---|---|---|---|
:3040 |
zappier-edge | 200 {"ok":true,"role":"zappier-edge"} |
Bind 127.0.0.1; systemd zappier-edge.service |
:3100 |
middleware (lab/mock) | 200 mockVerae:true, natsConnected:false |
Catalog/default; not product upstream |
:3130 |
middleware (product) | 200 mockVerae:false, natsConnected:true |
Edge ZAPPIER_UPSTREAM target |
:3010 |
zappier-account-balance | 200 nats:false |
Loopback; not a systemd unit; fleet-spawned |
:3021–:3025 |
access planes | 200 each | web / api / leaf / zapier / staff |
Env / upstream
- Edge secrets (
/SSD2/secrets/zap-verae/zappier-edge.env):PORT=3040,ZAPPIER_UPSTREAM=http://127.0.0.1:3130(confirms arch C11 / plan A1). - Product MW env:
PORT=3130,HOST=127.0.0.1,MOCK_VERAE=false,NATS_URL=nats://127.0.0.1:4222. - Account-balance (pid 4179): cwd under moved stack package;
FLEET_STATE_DIR=.../verae-fleet/data/runtime/account-balance-0;NATS_URLlists10.10.10.21–23but health reportsnats:false.
Digest-only stamp (flow 2 probe)
- Digest:
sha256("qa-baseline")=edf8f682e42104442bb4637c665f50ada9873fa2fe6958ae00a6c05b2e5181cb POST /v1/timestampswithx-api-key: key-ada,x-billing-channel: api, body{sha256s:[…]}→ HTTP 202,created:1then idempotentexisting:1, credit headers present (X-Verae-Credits-*).- Items-shaped digest-only body also 202.
- Single
POST /v1/timestamp→ 202 withjobId, but receiptcompletedAt: "0001-01-01T00:00:00Z"(placeholder / incomplete receipt shape for async complete).
Dual-writer / C16 (flows 1, 4–6)
| Store | Path | Observation |
|---|---|---|
| Edge ledger | /opt/verae/zappier-edge/billing.db |
Exists (WAL). usage_entries count 3 for cust_1 @ 10 each after probes; customers.balance_cents for Ada = 0 |
| Account-balance books | FLEET_STATE_DIR/.../data/books.json |
Exists, empty: prepaid:{}, usage:[], credits:[], payments:[] |
| Statement HTTP | GET :3010/statement/cust_1 |
200 {prepaidCents:0, usage:[], …} — does not see edge usage_entries |
| Edge response | batch stamp | balanceCredits:0, period spend headers show 30 after three 10-credit meters |
C16 confirmed live: edge writes metered usage into billing.db; account-balance serves a separate empty books.json. No shared SoT. Splitting hosts now would fork money.
Go timestamping DB (C9)
/var/lib/timestamping-svc/data/blockchain.dbexists (~31 MB, mtime 2026-09-25).- Not modified by this QA probe (read
ls/statonly).
libSQL / BusinessStore / harness
- No
LIBSQL_URL/sqld/ libSQL container observed on 138 for business data. - Local clone
/Users/marchon/apps/verae/zappier-edge/src/db/: nobusiness-store.ts(NO_BUSINESS_STORE). verae-zapier-simulatorexists (messaging/trace) but is not the WP-P automated harness for flows 1–7.- Arch checklist still open:
nats-subjects.mdverae.ts.*/ZAPIER_USAGEnon-SoT; published env matrix; GETTING-STARTED SoT language. - Host
nats stream lsto10.10.10.21:4222timed out from this probe (MW still reportsnatsConnected:truevia loopback:4222proxy). STATUS still lists product streamZAPIER_USAGE(C10 risk).
Leaf D004 (flow 7 probe)
:3023/health200 leaf plane up.POST :3023/v1/billing/adjust→ 404not found(route absent). This is not yet an automated policy deny ofverae.billing.balance.adjust(PM WP-D DoD).
Userflow readiness (1–7)
| # | Userflow | Today | Blocker vs D006 |
|---|---|---|---|
| 1 | Signup → API key → Statement | Partial: seed keys exist (key-ada…); statement via :3010 empty/out of band with edge ledger |
Dual SoT; no portal E2E harness; account-balance nats:false |
| 2 | Stamp digests single + batch + credit headers | Mostly green on lab (:3040→:3130, 202, headers, idempotent existing) |
Still local SQLite meter; zero balance still meters; receipt placeholder on single path; not libSQL |
| 3 | Async complete → Zapier REST Hook | Not proven | completedAt zero-time on single stamp; no WP-P watch→events→webhooks.deliver E2E; Platform push still open in STATUS |
| 4 | CS credit adjust → portal balance | Broken / dual path | Adjust would hit books/verae.billing.* while meter hits billing.db; nats false on balance service |
| 5 | Sales quote → meter match | Unproven | Rate tables live in edge SQLite; sales package not wired to shared SoT |
| 6 | Accounting export consistency | Fails if run today | Export cannot reconcile empty books vs non-empty usage_entries |
| 7 | Leaf stamp OK / billing deny | Health only | Need authz allow-list tests; 404 ≠ D004 proof |
PM QA entry (board §5): MG3 single writer, regression green, NATS rules verified, harness skeleton, D004 automated, migrator tip report, Go DB untouched note, docs C2/C3/C5/C6 — not met (only Go-untouched + partial digests-only + doors up).
a) What to change
| # | Component | Problem | Evidence | Severity | Type |
|---|---|---|---|---|---|
| 1 | Prepaid / ledger SoT (C7/C16) | Two writers: edge billing.db meters usage; :3010 serves empty books.json statement |
usage_n=3 vs statement usage:[]; both processes up |
critical | design flaw / dual writer |
| 2 | Missing libSQL + BusinessStore (D006 Phases 1–2) | No shared business DB or interface; multi-host impossible without per-host SQLite fork | No business-store.ts; no LIBSQL on 138 |
critical | missing foundation |
| 3 | Account-balance NATS / facade | Health nats:false while CS/portal expect verae.billing.* |
:3010/health JSON; env has NATS_URL but not connected |
high | ops / wiring |
| 4 | Userflow E2E harness (WP-P) | Flows 1–7 not automated; simulator ≠ harness | No plans/qa flow reports; simulator README is messaging-only |
high | missing gate |
| 5 | Async complete / receipts (flow 3) | Single-stamp receipt shows epoch-zero completedAt; REST Hook path unproven |
curl body 0001-01-01T00:00:00Z |
high | incomplete product path |
| 6 | Zero-balance metering (flows 2/4) | Ada balance_cents=0 still stamps and accumulates period spend |
customers row + 202 responses + X-Verae-Credits-Balance: 0 |
medium | missing gate / prepaid policy |
| 7 | Dual middleware confusion (C11) | :3100 mock vs :3130 product; docs/scripts can point wrong |
health JSON both ports; edge env → 3130 | medium | doc/ops drift |
| 8 | D004 leaf deny proof (flow 7) | Only 404 on invent-ed HTTP path; no automated NATS/HTTP deny | curl 404; WP-D tests not evidenced on lab | medium | missing test |
| 9 | ZAPIER_USAGE / nats-subjects docs (C5/C10) |
STATUS still lists usage stream as product; catalog subjects omit core-only verae.ts.* |
STATUS.md streams table; arch checklist open | medium | doc drift / SoT confusion |
| 10 | Env matrix / Phase 0 leftovers | Arch full sign-off blocked; GETTING-STARTED may still call books SoT | arch review checklist unchecked items | medium | doc freeze incomplete |
| 11 | Account-balance not systemd-managed | Process from fleet overlay / moved path; restart/ownership unclear vs edge unit | no *account-balance*.service; pid under marchon |
low | ops hygiene |
What is working (preserve)
- Digests-only batch + single stamp on product edge
:3040withkey-adaand credit headers. - Product upstream correctly set to
:3130(not mock:3100). - Access-plane health on
:3021–3025. - Edge split DBs present and WAL:
billing.db(business tables) +zappier.db(auth). - Idempotent re-stamp (
created:0,existing:1, spend 0). - Standing rules respected in this probe: no JetStream for
verae.ts.*; no writes toblockchain.db. - Plan + A1–A8 + D001 addendum + D006 direction are the right target.
b) What to change it to
| # | Component | Target behavior | Acceptance criteria | Constraints |
|---|---|---|---|---|
| 1 | Single prepaid/ledger writer | One authoritative adjust + recordUsage path on shared store; other path read-only or stopped |
Grep: no prod writes to books.json / per-host billing.db as SoT; adjust on CS visible on portal statement and edge balance headers |
D001 single writer; do not dual-append hash chain |
| 2 | libSQL BusinessStore | LibsqlBusinessStore (+ SqliteFile for CI) behind BusinessStore; lab migrator from billing.db + books.json |
Contract tests green; migrator tip hash verify; BUSINESS_STORE=libsql writes only to libSQL |
Auth stays edge-local zappier.db (C13); fail closed (503) if writer down |
| 3 | Billing facade | :3010 (or successor) is libSQL-backed listener for verae.billing.* or folded into edge HTTP API — pick one (plan Phase 3) |
Health nats:true (if NATS facade kept); statement usage matches usage_entries |
Keep subjects stable for CS/sales |
| 4 | Harness 1–7 | Automated runner under simulator/harness producing plans/qa/ report |
Flows 1,2,4 runnable under MOCK; then 3,5,6,7; exit 0 artifact | Zapier cloud push optional for first MOCK pass |
| 5 | Async complete | Real completion times / receipts; jobs.watch → jobs.events → webhooks.deliver |
E2E: jobId reaches terminal event + hook delivery (or recorded attempt) | verae.ts.completed / verae.ts.receipt remain core NATS only |
| 6 | Prepaid gate | Metering respects prepaid/credits policy (402 or configured allow) | Seed funded customer; zero-balance path returns expected product code | Do not rewrite historical ledger mid-test |
| 7 | Env matrix | Documented :3040 / :3130 product / :3100 mock / :3010 facade |
STATUS + ops GETTING-STARTED + plan cite same matrix | Do not “fix” smoke by pointing edge at 3100 |
| 8 | D004 | Leaf cannot call balance adjust (HTTP door + NATS authz) | verae-access-leaf / authz tests green; live deny ≠ bare 404 |
Leaf may still stamp/authz |
| 9 | NATS docs | nats-subjects.md Non-streams: verae.ts.* core-only; ZAPIER_USAGE telemetry non-SoT |
Arch checklist boxes checked; STATUS wording demotes usage stream | No JetStream bind for job-complete |
| 10 | Phase 0 close | C2/C3/C5/C6 closed or owned; GETTING-STARTED SoT → libSQL | Arch full sign-off or explicit waivers | — |
| 11 | Balance process ops | systemd (or fleet-documented) unit with clear state dir | systemctl status shows balance facade; books path documented until migration |
Deploy never rsyncs *.db* |
c) How to implement
Aligned with PM DAG; do not skip merge gates.
flowchart TD
P0[Phase 0 docs/env/NATS freeze] --> P1[Phase 1 BusinessStore TDD]
P1 --> P2[Phase 2 migrator + dual-read]
P2 --> MG2[MG2 writers unlock]
MG2 --> P3[Phase 3 single writer rewire]
P3 --> MG3[MG3 single writer proof]
MG3 --> P4[Phase 4 multi-host LIBSQL_URL]
MG3 --> P5[Phase 5 harness 1-7]
P4 --> P5
P5 --> MG4[MG4 green]
MG4 --> P6[Phase 6 QA whattochange loop]
Phase 0 — Conflict / docs freeze (feeding; ~1–2 days)
- Goal: Close open arch checklist items that confuse operators and agents.
- Changes: Publish env matrix; patch
nats-subjects.md; demoteZAPIER_USAGEin STATUS; fix overviewzmw_/ digests examples (C3/C6); GETTING-STARTED SoT → libSQL. - Validation: Arch checklist;
rgfor stale SoT claims. - Effort: 1–2 agent-days (WP-A/B/C).
Phase 1 — BusinessStore + libSQL skeleton (critical; WP-F/G/H)
- Goal: Interface + SqliteFile + Libsql drivers with contract tests (including hash-chain).
- Changes:
zappier-edgesrc/db/business-store.ts, contract tests, CI libSQL. - Validation: Jest contract green both drivers (MG1).
- Effort: ~3 days serial.
Phase 2 — Migrate + dual-read (WP-J/K)
- Goal: Import live
billing.db+ empty/legacybooks.json; verify chain tip; reads dual-run; writes still locked. - Validation: Migrator report archived under
plans/qa/; MG2. - Effort: ~2 days.
Phase 3 — Rewire writers (WP-L/M/N) — only after MG2
- Goal: Edge meter + account-balance + CS/sales/acct → one BusinessStore; remove second writer.
- Validation: Statement usage == ledger; CS adjust visible on portal/edge headers; MG3 grep proof.
- Effort: ~3–4 days.
Phase 4 — Multi-host (WP-O) — only after MG3
- Goal:
LIBSQL_URLprimary; edge/MW/CS on separate hosts possible. - Validation: Cross-host adjust/meter/statement same balance.
Phase 5 — Harness (WP-P)
- Goal: Automate flows 1–7; first pass MOCK_VERAE OK.
- Validation: Report under
plans/qa/; MG4.
Phase 6 — QA loop (WP-Q)
- Goal: Break → whattochange → fix → retest until two consecutive full greens (project buffer).
Explicit non-goals until gates pass: JetStream for verae.ts.*; editing blockchain.db; Litestream multi-writer; multi-host split before MG3.
d) Expected results / still incomplete
After implementation (realistic)
| Metric / behavior | Today (baseline) | Expected after Phases 0–5 |
|---|---|---|
| Business SoT | Split billing.db + empty books.json |
One libSQL verae_business |
| Flow 2 digests stamp | 202 + headers on single host SQLite | Same UX on libSQL meter writer |
| Flow 1/4 statement vs meter | Divergent (usage 3 vs []) |
Identical customer/usage/balance |
| Flow 3 async → hook | Unproven; zero completedAt |
Proven under harness (MOCK then optional live) |
| Flow 5/6 quote & export | Unproven / inconsistent | Same rate tables + export = ledger |
| Flow 7 D004 | 404 only | Automated deny |
| Multi-host money | Unsafe | Safe only post-MG3 |
blockchain.db |
Untouched | Still untouched |
Still missing, incorrect, or incomplete (even after this todo list)
| Gap | Why it persists |
|---|---|
Zapier Platform public push + live workspace Zaps |
Needs operator Zapier login (STATUS); out of first cutover buffer |
Live api.veraetime.net seal proof beyond lab MW |
Creds / OpenAPI gaps (hash lookup 501); separate from libSQL SoT |
| NATS nkeys/mTLS | Cluster isolation today is vmbr1, not accounts |
| Portal auth on multi-edge sticky sessions | C13: auth stays local zappier.db for v1 |
| libSQL HA / failover RPO | Launch = one primary + backup; not multi-primary |
| Peergos / Go timestamping redesign | Out of scope |
| Full architecture sign-off | Depends on remaining checklist (docs), not only code |
Success tiers
| Tier | Criteria |
|---|---|
| Minimum viable (this baseline exit) | Dual-writer named daily owner; env matrix published; harness skeleton for 1/2/4; no host split |
| MG3 / single-writer | libSQL SoT; one adjust path; statement == meter |
| Production-ready multi-host | MG3 + WP-O + flows 1–7 twice green; fail-closed meter; D004 automated |
| Complete | + Platform push + live Verae seal + nkeys — separate projects |
e) Prioritized todo list with PreReqs
Item 1 — [P0] — Freeze dual-writer authority for lab (C16 runbook)
Document which process may adjust/meter each day until MG3; forbid CS adjust against books while edge meters billing.db without a sync plan.
PreReqs: (none)
Item 2 — [P0] — Publish live env matrix (:3040 / :3130 / :3100 / :3010)
Add to STATUS + ops docs; match edge ZAPPIER_UPSTREAM=http://127.0.0.1:3130 evidence.
PreReqs: (none)
Item 3 — [P0] — BusinessStore interface + failing contract tests (WP-F)
Create business-store.ts + contract suite (customers, usage chain, rates, invoices, balances).
PreReqs: (Item 2)
Item 4 — [P0] — SqliteFileBusinessStore adapter (WP-G)
Wrap existing edge repos; keep Jest green.
PreReqs: (Item 3)
Item 5 — [P0] — LibsqlBusinessStore + CI libSQL (WP-H)
Driver + reachable LIBSQL_URL in CI/lab; MG1.
PreReqs: (Item 4)
Item 6 — [P0] — Migrator billing.db + books.json → libSQL with chain verify (WP-K)
Refuse cutover on verifyChain failure; archive tip report under plans/qa/; MG2.
PreReqs: (Item 5)
Item 7 — [P0] — Rewire edge meter to BusinessStore only (WP-L)
Remove production writes that treat local billing.db as SoT once flag on.
PreReqs: (Item 6)
Item 8 — [P0] — Fold or rewire account-balance to libSQL facade (WP-M)
One verae.billing.* listener; empty books.json import-only; nats connected if facade kept.
PreReqs: (Item 7)
Item 9 — [P1] — Rewire CS / sales / accounting-export to same store (WP-N)
Credit adjust, rates, export read libSQL; MG3 proof.
PreReqs: (Item 8)
Item 10 — [P1] — Patch nats-subjects + demote ZAPIER_USAGE (C5/C10)
Document verae.ts.* core-only; usage stream telemetry-only in STATUS/catalog.
PreReqs: (none)
Item 11 — [P1] — Automate D004 leaf billing deny (WP-D)
Tests prove leaf cannot balance.adjust; replace 404-only smoke.
PreReqs: (none)
Item 12 — [P1] — Userflow harness skeleton flows 1, 2, 4 (WP-P start)
MOCK_VERAE runnable report artifact under plans/qa/.
PreReqs: (Item 9)
Item 13 — [P1] — Extend harness to flows 3, 5, 6, 7
Async hook path, quote match, export consistency, leaf deny E2E; MG4.
PreReqs: (Item 12 + Item 11)
Item 14 — [P2] — Fix async receipt completedAt / completion wiring
Eliminate 0001-01-01 placeholders on product stamp path used by Zapier wait/hook.
PreReqs: (Item 12)
Item 15 — [P2] — Prepaid zero-balance policy gate
Define and test 402 vs allow-with-debt; seed funded fixture for harness.
PreReqs: (Item 7)
Item 16 — [P2] — Multi-host LIBSQL_URL topology (WP-O)
Only after MG3; cross-host meter/adjust/statement.
PreReqs: (Item 9)
Item 17 — [P2] — systemd/fleet hygiene for account-balance facade
Unit or documented fleet runtime; stable state dir until books retired.
PreReqs: (Item 8)
Item 18 — [P3] — Phase 0 leftover docs (C2/C3/C6, GETTING-STARTED)
Close arch full sign-off checklist items not covered by Item 2/10.
PreReqs: (Item 2 + Item 10)
Dependency order (for “Do all”):
Item 1 ∥ Item 2 ∥ Item 10 ∥ Item 11 → Item 3 → Item 4 → Item 5 → Item 6 → Item 7 → Item 8 → Item 9 → Item 12 → (Item 13 ∥ Item 14 ∥ Item 15) → Item 16 → Item 17; Item 18 can trail after Item 2+10.
QA note: Formal WP-Q churn should wait for Item 9 (MG3) + Item 12. This baseline pack is the readiness report that unlocks engineering focus on Items 1–9.
Probe command index (reproducible)
ssh -i ~/.ssh/id_ed25519 -o IdentitiesOnly=yes marchon@70.88.205.138 \
'for p in 3040 3100 3130 3010 3021 3022 3023 3024 3025; do
curl -sS -m 5 http://127.0.0.1:$p/health; echo; done'
# Digest-only batch (do not use production customer keys outside lab seeds)
DIGEST=$(printf "qa-baseline" | sha256sum | awk "{print \$1}")
curl -sS -D - -X POST http://127.0.0.1:3040/v1/timestamps \
-H "content-type: application/json" -H "x-api-key: key-ada" \
-H "x-billing-channel: api" \
-d "{\"sha256s\":[\"$DIGEST\"]}"
# Dual-writer check (read-only)
sqlite3 /opt/verae/zappier-edge/billing.db "SELECT COUNT(*) FROM usage_entries;"
curl -sS http://127.0.0.1:3010/statement/cust_1
# blockchain.db: ls/stat only — never migrate/edit
End of QA baseline. No product code was changed by this review.