zapier-decisions/plans/qa/2026-10-03-baseline.md

21 KiB
Raw Permalink Blame History

QA Baseline — Verae × Zapier Userflows 1–7 vs D006 Multi-Host

Role: QA (evaluate only; no product feature implementation)
Date: 2026-10-03 (UTC probe ~06:04–06:07)
Host: NS1.GEORGELAMBERT.ORG / 70.88.205.138
Inputs:


Summary judgment

Lab doors and digests-only stamp submit on :3040 → :3130 work today, but D006 multi-host readiness fails: there is no libSQL, no BusinessStore, and two live prepaid/ledger writers (billing.db on edge + empty books.json behind :3010). Userflows 1, 3–6 are not E2E-proven; flow 2 is partially green; flow 7 has only weak 404 evidence for billing deny. Do not start Phase 4 host split or treat WP-Q as “in” until single-writer libSQL lands.


Live lab evidence (2026-10-03)

Health matrix

Port Role /health Notes
:3040 zappier-edge 200 {"ok":true,"role":"zappier-edge"} Bind 127.0.0.1; systemd zappier-edge.service
:3100 middleware (lab/mock) 200 mockVerae:true, natsConnected:false Catalog/default; not product upstream
:3130 middleware (product) 200 mockVerae:false, natsConnected:true Edge ZAPPIER_UPSTREAM target
:3010 zappier-account-balance 200 nats:false Loopback; not a systemd unit; fleet-spawned
:3021–:3025 access planes 200 each web / api / leaf / zapier / staff

Env / upstream

  • Edge secrets (/SSD2/secrets/zap-verae/zappier-edge.env): PORT=3040, ZAPPIER_UPSTREAM=http://127.0.0.1:3130 (confirms arch C11 / plan A1).
  • Product MW env: PORT=3130, HOST=127.0.0.1, MOCK_VERAE=false, NATS_URL=nats://127.0.0.1:4222.
  • Account-balance (pid 4179): cwd under moved stack package; FLEET_STATE_DIR=.../verae-fleet/data/runtime/account-balance-0; NATS_URL lists 10.10.10.21–23 but health reports nats:false.

Digest-only stamp (flow 2 probe)

  • Digest: sha256("qa-baseline") = edf8f682e42104442bb4637c665f50ada9873fa2fe6958ae00a6c05b2e5181cb
  • POST /v1/timestamps with x-api-key: key-ada, x-billing-channel: api, body {sha256s:[…]} → HTTP 202, created:1 then idempotent existing:1, credit headers present (X-Verae-Credits-*).
  • Items-shaped digest-only body also 202.
  • Single POST /v1/timestamp → 202 with jobId, but receipt completedAt: "0001-01-01T00:00:00Z" (placeholder / incomplete receipt shape for async complete).

Dual-writer / C16 (flows 1, 4–6)

Store Path Observation
Edge ledger /opt/verae/zappier-edge/billing.db Exists (WAL). usage_entries count 3 for cust_1 @ 10 each after probes; customers.balance_cents for Ada = 0
Account-balance books FLEET_STATE_DIR/.../data/books.json Exists, empty: prepaid:{}, usage:[], credits:[], payments:[]
Statement HTTP GET :3010/statement/cust_1 200 {prepaidCents:0, usage:[], …} — does not see edge usage_entries
Edge response batch stamp balanceCredits:0, period spend headers show 30 after three 10-credit meters

C16 confirmed live: edge writes metered usage into billing.db; account-balance serves a separate empty books.json. No shared SoT. Splitting hosts now would fork money.

Go timestamping DB (C9)

  • /var/lib/timestamping-svc/data/blockchain.db exists (~31 MB, mtime 2026-09-25).
  • Not modified by this QA probe (read ls/stat only).

libSQL / BusinessStore / harness

  • No LIBSQL_URL / sqld / libSQL container observed on 138 for business data.
  • Local clone /Users/marchon/apps/verae/zappier-edge/src/db/: no business-store.ts (NO_BUSINESS_STORE).
  • verae-zapier-simulator exists (messaging/trace) but is not the WP-P automated harness for flows 1–7.
  • Arch checklist still open: nats-subjects.md verae.ts.* / ZAPIER_USAGE non-SoT; published env matrix; GETTING-STARTED SoT language.
  • Host nats stream ls to 10.10.10.21:4222 timed out from this probe (MW still reports natsConnected:true via loopback :4222 proxy). STATUS still lists product stream ZAPIER_USAGE (C10 risk).

Leaf D004 (flow 7 probe)

  • :3023/health 200 leaf plane up.
  • POST :3023/v1/billing/adjust → 404 not found (route absent). This is not yet an automated policy deny of verae.billing.balance.adjust (PM WP-D DoD).

Userflow readiness (1–7)

# Userflow Today Blocker vs D006
1 Signup → API key → Statement Partial: seed keys exist (key-ada…); statement via :3010 empty/out of band with edge ledger Dual SoT; no portal E2E harness; account-balance nats:false
2 Stamp digests single + batch + credit headers Mostly green on lab (:3040→:3130, 202, headers, idempotent existing) Still local SQLite meter; zero balance still meters; receipt placeholder on single path; not libSQL
3 Async complete → Zapier REST Hook Not proven completedAt zero-time on single stamp; no WP-P watch→events→webhooks.deliver E2E; Platform push still open in STATUS
4 CS credit adjust → portal balance Broken / dual path Adjust would hit books/verae.billing.* while meter hits billing.db; nats false on balance service
5 Sales quote → meter match Unproven Rate tables live in edge SQLite; sales package not wired to shared SoT
6 Accounting export consistency Fails if run today Export cannot reconcile empty books vs non-empty usage_entries
7 Leaf stamp OK / billing deny Health only Need authz allow-list tests; 404 ≠ D004 proof

PM QA entry (board §5): MG3 single writer, regression green, NATS rules verified, harness skeleton, D004 automated, migrator tip report, Go DB untouched note, docs C2/C3/C5/C6 — not met (only Go-untouched + partial digests-only + doors up).


a) What to change

# Component Problem Evidence Severity Type
1 Prepaid / ledger SoT (C7/C16) Two writers: edge billing.db meters usage; :3010 serves empty books.json statement usage_n=3 vs statement usage:[]; both processes up critical design flaw / dual writer
2 Missing libSQL + BusinessStore (D006 Phases 1–2) No shared business DB or interface; multi-host impossible without per-host SQLite fork No business-store.ts; no LIBSQL on 138 critical missing foundation
3 Account-balance NATS / facade Health nats:false while CS/portal expect verae.billing.* :3010/health JSON; env has NATS_URL but not connected high ops / wiring
4 Userflow E2E harness (WP-P) Flows 1–7 not automated; simulator ≠ harness No plans/qa flow reports; simulator README is messaging-only high missing gate
5 Async complete / receipts (flow 3) Single-stamp receipt shows epoch-zero completedAt; REST Hook path unproven curl body 0001-01-01T00:00:00Z high incomplete product path
6 Zero-balance metering (flows 2/4) Ada balance_cents=0 still stamps and accumulates period spend customers row + 202 responses + X-Verae-Credits-Balance: 0 medium missing gate / prepaid policy
7 Dual middleware confusion (C11) :3100 mock vs :3130 product; docs/scripts can point wrong health JSON both ports; edge env → 3130 medium doc/ops drift
8 D004 leaf deny proof (flow 7) Only 404 on invent-ed HTTP path; no automated NATS/HTTP deny curl 404; WP-D tests not evidenced on lab medium missing test
9 ZAPIER_USAGE / nats-subjects docs (C5/C10) STATUS still lists usage stream as product; catalog subjects omit core-only verae.ts.* STATUS.md streams table; arch checklist open medium doc drift / SoT confusion
10 Env matrix / Phase 0 leftovers Arch full sign-off blocked; GETTING-STARTED may still call books SoT arch review checklist unchecked items medium doc freeze incomplete
11 Account-balance not systemd-managed Process from fleet overlay / moved path; restart/ownership unclear vs edge unit no *account-balance*.service; pid under marchon low ops hygiene

What is working (preserve)

  • Digests-only batch + single stamp on product edge :3040 with key-ada and credit headers.
  • Product upstream correctly set to :3130 (not mock :3100).
  • Access-plane health on :3021–3025.
  • Edge split DBs present and WAL: billing.db (business tables) + zappier.db (auth).
  • Idempotent re-stamp (created:0, existing:1, spend 0).
  • Standing rules respected in this probe: no JetStream for verae.ts.*; no writes to blockchain.db.
  • Plan + A1–A8 + D001 addendum + D006 direction are the right target.

b) What to change it to

# Component Target behavior Acceptance criteria Constraints
1 Single prepaid/ledger writer One authoritative adjust + recordUsage path on shared store; other path read-only or stopped Grep: no prod writes to books.json / per-host billing.db as SoT; adjust on CS visible on portal statement and edge balance headers D001 single writer; do not dual-append hash chain
2 libSQL BusinessStore LibsqlBusinessStore (+ SqliteFile for CI) behind BusinessStore; lab migrator from billing.db + books.json Contract tests green; migrator tip hash verify; BUSINESS_STORE=libsql writes only to libSQL Auth stays edge-local zappier.db (C13); fail closed (503) if writer down
3 Billing facade :3010 (or successor) is libSQL-backed listener for verae.billing.* or folded into edge HTTP API — pick one (plan Phase 3) Health nats:true (if NATS facade kept); statement usage matches usage_entries Keep subjects stable for CS/sales
4 Harness 1–7 Automated runner under simulator/harness producing plans/qa/ report Flows 1,2,4 runnable under MOCK; then 3,5,6,7; exit 0 artifact Zapier cloud push optional for first MOCK pass
5 Async complete Real completion times / receipts; jobs.watch → jobs.events → webhooks.deliver E2E: jobId reaches terminal event + hook delivery (or recorded attempt) verae.ts.completed / verae.ts.receipt remain core NATS only
6 Prepaid gate Metering respects prepaid/credits policy (402 or configured allow) Seed funded customer; zero-balance path returns expected product code Do not rewrite historical ledger mid-test
7 Env matrix Documented :3040 / :3130 product / :3100 mock / :3010 facade STATUS + ops GETTING-STARTED + plan cite same matrix Do not “fix” smoke by pointing edge at 3100
8 D004 Leaf cannot call balance adjust (HTTP door + NATS authz) verae-access-leaf / authz tests green; live deny ≠ bare 404 Leaf may still stamp/authz
9 NATS docs nats-subjects.md Non-streams: verae.ts.* core-only; ZAPIER_USAGE telemetry non-SoT Arch checklist boxes checked; STATUS wording demotes usage stream No JetStream bind for job-complete
10 Phase 0 close C2/C3/C5/C6 closed or owned; GETTING-STARTED SoT → libSQL Arch full sign-off or explicit waivers —
11 Balance process ops systemd (or fleet-documented) unit with clear state dir systemctl status shows balance facade; books path documented until migration Deploy never rsyncs *.db*

c) How to implement

Aligned with PM DAG; do not skip merge gates.

flowchart TD
  P0[Phase 0 docs/env/NATS freeze] --> P1[Phase 1 BusinessStore TDD]
  P1 --> P2[Phase 2 migrator + dual-read]
  P2 --> MG2[MG2 writers unlock]
  MG2 --> P3[Phase 3 single writer rewire]
  P3 --> MG3[MG3 single writer proof]
  MG3 --> P4[Phase 4 multi-host LIBSQL_URL]
  MG3 --> P5[Phase 5 harness 1-7]
  P4 --> P5
  P5 --> MG4[MG4 green]
  MG4 --> P6[Phase 6 QA whattochange loop]

Phase 0 — Conflict / docs freeze (feeding; ~1–2 days)

  • Goal: Close open arch checklist items that confuse operators and agents.
  • Changes: Publish env matrix; patch nats-subjects.md; demote ZAPIER_USAGE in STATUS; fix overview zmw_ / digests examples (C3/C6); GETTING-STARTED SoT → libSQL.
  • Validation: Arch checklist; rg for stale SoT claims.
  • Effort: 1–2 agent-days (WP-A/B/C).

Phase 1 — BusinessStore + libSQL skeleton (critical; WP-F/G/H)

  • Goal: Interface + SqliteFile + Libsql drivers with contract tests (including hash-chain).
  • Changes: zappier-edge src/db/business-store.ts, contract tests, CI libSQL.
  • Validation: Jest contract green both drivers (MG1).
  • Effort: ~3 days serial.

Phase 2 — Migrate + dual-read (WP-J/K)

  • Goal: Import live billing.db + empty/legacy books.json; verify chain tip; reads dual-run; writes still locked.
  • Validation: Migrator report archived under plans/qa/; MG2.
  • Effort: ~2 days.

Phase 3 — Rewire writers (WP-L/M/N) — only after MG2

  • Goal: Edge meter + account-balance + CS/sales/acct → one BusinessStore; remove second writer.
  • Validation: Statement usage == ledger; CS adjust visible on portal/edge headers; MG3 grep proof.
  • Effort: ~3–4 days.

Phase 4 — Multi-host (WP-O) — only after MG3

  • Goal: LIBSQL_URL primary; edge/MW/CS on separate hosts possible.
  • Validation: Cross-host adjust/meter/statement same balance.

Phase 5 — Harness (WP-P)

  • Goal: Automate flows 1–7; first pass MOCK_VERAE OK.
  • Validation: Report under plans/qa/; MG4.

Phase 6 — QA loop (WP-Q)

  • Goal: Break → whattochange → fix → retest until two consecutive full greens (project buffer).

Explicit non-goals until gates pass: JetStream for verae.ts.*; editing blockchain.db; Litestream multi-writer; multi-host split before MG3.


d) Expected results / still incomplete

After implementation (realistic)

Metric / behavior Today (baseline) Expected after Phases 0–5
Business SoT Split billing.db + empty books.json One libSQL verae_business
Flow 2 digests stamp 202 + headers on single host SQLite Same UX on libSQL meter writer
Flow 1/4 statement vs meter Divergent (usage 3 vs []) Identical customer/usage/balance
Flow 3 async → hook Unproven; zero completedAt Proven under harness (MOCK then optional live)
Flow 5/6 quote & export Unproven / inconsistent Same rate tables + export = ledger
Flow 7 D004 404 only Automated deny
Multi-host money Unsafe Safe only post-MG3
blockchain.db Untouched Still untouched

Still missing, incorrect, or incomplete (even after this todo list)

Gap Why it persists
Zapier Platform public push + live workspace Zaps Needs operator Zapier login (STATUS); out of first cutover buffer
Live api.veraetime.net seal proof beyond lab MW Creds / OpenAPI gaps (hash lookup 501); separate from libSQL SoT
NATS nkeys/mTLS Cluster isolation today is vmbr1, not accounts
Portal auth on multi-edge sticky sessions C13: auth stays local zappier.db for v1
libSQL HA / failover RPO Launch = one primary + backup; not multi-primary
Peergos / Go timestamping redesign Out of scope
Full architecture sign-off Depends on remaining checklist (docs), not only code

Success tiers

Tier Criteria
Minimum viable (this baseline exit) Dual-writer named daily owner; env matrix published; harness skeleton for 1/2/4; no host split
MG3 / single-writer libSQL SoT; one adjust path; statement == meter
Production-ready multi-host MG3 + WP-O + flows 1–7 twice green; fail-closed meter; D004 automated
Complete + Platform push + live Verae seal + nkeys — separate projects

e) Prioritized todo list with PreReqs

Item 1 — [P0] — Freeze dual-writer authority for lab (C16 runbook)
Document which process may adjust/meter each day until MG3; forbid CS adjust against books while edge meters billing.db without a sync plan.
PreReqs: (none)

Item 2 — [P0] — Publish live env matrix (:3040 / :3130 / :3100 / :3010)
Add to STATUS + ops docs; match edge ZAPPIER_UPSTREAM=http://127.0.0.1:3130 evidence.
PreReqs: (none)

Item 3 — [P0] — BusinessStore interface + failing contract tests (WP-F)
Create business-store.ts + contract suite (customers, usage chain, rates, invoices, balances).
PreReqs: (Item 2)

Item 4 — [P0] — SqliteFileBusinessStore adapter (WP-G)
Wrap existing edge repos; keep Jest green.
PreReqs: (Item 3)

Item 5 — [P0] — LibsqlBusinessStore + CI libSQL (WP-H)
Driver + reachable LIBSQL_URL in CI/lab; MG1.
PreReqs: (Item 4)

Item 6 — [P0] — Migrator billing.db + books.json → libSQL with chain verify (WP-K)
Refuse cutover on verifyChain failure; archive tip report under plans/qa/; MG2.
PreReqs: (Item 5)

Item 7 — [P0] — Rewire edge meter to BusinessStore only (WP-L)
Remove production writes that treat local billing.db as SoT once flag on.
PreReqs: (Item 6)

Item 8 — [P0] — Fold or rewire account-balance to libSQL facade (WP-M)
One verae.billing.* listener; empty books.json import-only; nats connected if facade kept.
PreReqs: (Item 7)

Item 9 — [P1] — Rewire CS / sales / accounting-export to same store (WP-N)
Credit adjust, rates, export read libSQL; MG3 proof.
PreReqs: (Item 8)

Item 10 — [P1] — Patch nats-subjects + demote ZAPIER_USAGE (C5/C10)
Document verae.ts.* core-only; usage stream telemetry-only in STATUS/catalog.
PreReqs: (none)

Item 11 — [P1] — Automate D004 leaf billing deny (WP-D)
Tests prove leaf cannot balance.adjust; replace 404-only smoke.
PreReqs: (none)

Item 12 — [P1] — Userflow harness skeleton flows 1, 2, 4 (WP-P start)
MOCK_VERAE runnable report artifact under plans/qa/.
PreReqs: (Item 9)

Item 13 — [P1] — Extend harness to flows 3, 5, 6, 7
Async hook path, quote match, export consistency, leaf deny E2E; MG4.
PreReqs: (Item 12 + Item 11)

Item 14 — [P2] — Fix async receipt completedAt / completion wiring
Eliminate 0001-01-01 placeholders on product stamp path used by Zapier wait/hook.
PreReqs: (Item 12)

Item 15 — [P2] — Prepaid zero-balance policy gate
Define and test 402 vs allow-with-debt; seed funded fixture for harness.
PreReqs: (Item 7)

Item 16 — [P2] — Multi-host LIBSQL_URL topology (WP-O)
Only after MG3; cross-host meter/adjust/statement.
PreReqs: (Item 9)

Item 17 — [P2] — systemd/fleet hygiene for account-balance facade
Unit or documented fleet runtime; stable state dir until books retired.
PreReqs: (Item 8)

Item 18 — [P3] — Phase 0 leftover docs (C2/C3/C6, GETTING-STARTED)
Close arch full sign-off checklist items not covered by Item 2/10.
PreReqs: (Item 2 + Item 10)

Dependency order (for “Do all”):
Item 1 ∥ Item 2 ∥ Item 10 ∥ Item 11 → Item 3 → Item 4 → Item 5 → Item 6 → Item 7 → Item 8 → Item 9 → Item 12 → (Item 13 ∥ Item 14 ∥ Item 15) → Item 16 → Item 17; Item 18 can trail after Item 2+10.

QA note: Formal WP-Q churn should wait for Item 9 (MG3) + Item 12. This baseline pack is the readiness report that unlocks engineering focus on Items 1–9.


Probe command index (reproducible)

ssh -i ~/.ssh/id_ed25519 -o IdentitiesOnly=yes marchon@70.88.205.138 \
  'for p in 3040 3100 3130 3010 3021 3022 3023 3024 3025; do
     curl -sS -m 5 http://127.0.0.1:$p/health; echo; done'

# Digest-only batch (do not use production customer keys outside lab seeds)
DIGEST=$(printf "qa-baseline" | sha256sum | awk "{print \$1}")
curl -sS -D - -X POST http://127.0.0.1:3040/v1/timestamps \
  -H "content-type: application/json" -H "x-api-key: key-ada" \
  -H "x-billing-channel: api" \
  -d "{\"sha256s\":[\"$DIGEST\"]}"

# Dual-writer check (read-only)
sqlite3 /opt/verae/zappier-edge/billing.db "SELECT COUNT(*) FROM usage_entries;"
curl -sS http://127.0.0.1:3010/statement/cust_1
# blockchain.db: ls/stat only — never migrate/edit

End of QA baseline. No product code was changed by this review.