zapier-decisions/plans/qa/2026-10-03-retest-wpl.md

15 KiB
Raw Blame History

QA Retest — BusinessStore sqlite-file deploy (WP-L precursor)

Role: QA (evaluate only; no product feature implementation)
Date: 2026-10-03 (UTC probe ~06:21–06:24)
Host: NS1.GEORGELAMBERT.ORG / 70.88.205.138
Baseline: 2026-10-03-baseline.md
Scope: Retest after live edge deploy of SqliteFileBusinessStore (BusinessStore driver: sqlite-file). Confirm checklist items; compare userflows 1–7 vs baseline; emit whattochange a–e for remaining breaks.
Constraints honored: No writes to Go timestamping DBs; billing.db not cleared (read-only verify + normal meter appends from stamp probes only).


Summary judgment

WP-G / sqlite-file path on live edge is green: journal logs BusinessStore driver: sqlite-file, digests-only batch on :3040 with key-ada returns 202 then idempotent existing/free, and usage_entries verifyChain is ok (length 6 after this probe). Health matrix :3040/:3130/:3010/:3100 all 200. Upstream api.veraetime.net digests-only batch still 202.

Still not MG3 / multi-host ready: dedicated DUAL-WRITER-FREEZE.md is missing (freeze text only in docs/ENV-MATRIX.md); :3010 statement remains empty vs non-empty edge ledger (dual SoT / C16 still live); account-balance nats:false; single-stamp completedAt still epoch-zero; flows 1, 3–7 unchanged vs baseline readiness. Do not treat WP-Q as entered or split hosts.


Checklist results

# Check Result Evidence
1 journal shows BusinessStore driver: sqlite-file PASS journalctl -u zappier-edge.service: Oct 02 21:21:15 … BusinessStore driver: sqlite-file and Billing customers/usage via SqliteFileBusinessStore (BILLING_DB). Process env has PORT=3040, ZAPPIER_UPSTREAM=http://127.0.0.1:3130; BUSINESS_STORE not explicitly set (defaults to sqlite-file).
2 POST /v1/timestamps digests-only key-ada → 202; re-POST same hash → existing/free PASS Digest sha256("qa-retest-wpl-2026-10-03") = 5bdeb622…eacbd. POST1 → HTTP 202, created:1, existing:0, X-Verae-Credits-Spend: 10. POST2 → HTTP 202, created:0, existing:1, totalCredits:0, X-Verae-Credits-Spend: 0. Channel api.
3 usage chain verifyChain ok on billing.db PASS Readonly SqliteUsageRepo.verifyChain() on /opt/verae/zappier-edge/billing.db → {"ok":true,"length":6,"tipHash":"8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5"}. WAL. Ada (cust_1) balance_cents=0; usage rollup cust_1 n=6 sum=60 credits.
4 DUAL-WRITER-FREEZE.md present FAIL ls → no such file under /opt/verae/zappier-edge/ or docs/. Freeze policy text exists inside docs/ENV-MATRIX.md (“Dual-writer freeze (2026-10-03): Meter and balance writes go only through zappier-edge → BusinessStore…”). Checklist asked for the named file — not present.
5 health matrix :3040/:3130/:3010/:3100 PASS See table below. Access planes :3021–3025 also 200.
6 Optional: api.veraetime.net login + POST /api/batch/timestamp two digests PASS Login admin/admin123 → JWT. Batch digests-only two items → HTTP 202 {"jobIds":["fa1838fc-…","930de378-…"]}. Confirms upstream still digests-only / batch path.

Health matrix (this probe)

Port Role /health Notes
:3040 zappier-edge 200 {"ok":true,"role":"zappier-edge"} BusinessStore sqlite-file
:3130 middleware product 200 mockVerae:false, natsConnected:true Edge upstream
:3010 account-balance 200 nats:false Statement empty; dual path
:3100 middleware mock 200 mockVerae:true, natsConnected:false Do not smoke here

Go DB (C9)

  • /var/lib/timestamping-svc/data/blockchain.db — mtime 2026-09-25, size ~31 MB. Not modified by this QA (stat/ls only).

Dual-writer / statement vs meter (still broken)

Store Observation
Edge billing.db usage_entries length 6, chain tip ok; period spend headers 60 after probes
:3010/statement/cust_1 200 {prepaidCents:0, usage:[], credits:[], payments:[]} — does not see edge usage
C16 Still live: meter writer = edge BusinessStore/sqlite-file; facade books path still empty / out of band

Sqlite-file deploy moved edge metering behind BusinessStore, but did not eliminate the second empty ledger behind account-balance.


Userflow readiness 1–7 vs baseline

# Userflow Baseline This retest Remaining blocker
1 Signup → API key → Statement Partial; statement empty vs edge Unchanged / still blocked Seed keys work; :3010 statement still usage:[] while edge has 6 rows; no portal E2E harness; dual SoT
2 Stamp digests single + batch + credit headers Mostly green Improved evidence / still green sqlite-file meter + verifyChain ok; still local SQLite (not libSQL); zero balance still meters; not multi-host
3 Async complete → Zapier REST Hook Not proven Still broken / unproven Single POST /v1/timestamp receipt completedAt:"0001-01-01T00:00:00Z" (jobId 6e925fbc-…); no watch→events→webhook E2E
4 CS credit adjust → portal balance Broken / dual path Still broken Adjust would hit :3010/books while meter hits BusinessStore/billing.db; nats:false
5 Sales quote → meter match Unproven Still unproven Rates still edge SQLite; no shared SoT / harness
6 Accounting export consistency Fails if run Still fails Export cannot reconcile empty statement/books vs usage_entries n=6
7 Leaf stamp OK / billing deny Health + 404 only Unchanged :3023/health 200; POST :3023/v1/billing/adjust → 404 not found (≠ D004 automated deny)

PM QA entry (MG3): still not met. Progress vs baseline: BusinessStore + SqliteFile on live edge, chain verify green, ENV-MATRIX published, digests-only + upstream batch reconfirmed. Missing for MG3: libSQL cutover, single writer proof (statement == meter), account-balance fold/rewire, harness, D004 tests, dedicated freeze doc filename if still required by ops.


a) What to change

# Component Problem Evidence Severity Type
1 Dual prepaid/ledger path (C16) Edge BusinessStore meters billing.db; :3010 statement still empty separate store statement usage:[] vs verifyChain length 6; both services up critical design flaw / dual writer
2 Missing libSQL SoT (D006) Lab still sqlite-file only; no LIBSQL_URL in edge environ environ grep; journal sqlite-file critical missing foundation
3 DUAL-WRITER-FREEZE.md filename Checklist/ops expect named freeze doc; only ENV-MATRIX paragraph exists ls missing file; ENV-MATRIX has freeze text medium doc drift / incomplete deliverable
4 Account-balance NATS / facade nats:false; not libSQL-backed; not single-writer health JSON high ops / wiring
5 Async receipt / flow 3 completedAt epoch-zero on single stamp curl body 0001-01-01T00:00:00Z high incomplete product path
6 Userflow harness (WP-P) Flows 1–7 not automated No harness report artifact beyond this manual retest high missing gate
7 Zero-balance metering Ada balance_cents=0 still stamps customers row + 202 + Balance header 0 medium missing prepaid gate
8 D004 leaf deny 404 ≠ policy deny curl 404 on invent-ed path medium missing test
9 BUSINESS_STORE not explicit in secrets Relies on code default; harder to audit live environ lacks BUSINESS_STORE= low ops hygiene

What is working (preserve)

  • Live BusinessStore driver: sqlite-file + SqliteFileBusinessStore over BILLING_DB.
  • Digests-only batch :3040 → :3130 with key-ada, credit headers, idempotent existing/free.
  • verifyChain ok after meter appends (hash chain intact).
  • Product upstream :3130 (not mock :3100); ENV-MATRIX documents the port roles + freeze policy.
  • Upstream Verae Time digests-only batch authenticated path.
  • Access planes healthy; Go blockchain.db untouched; no JetStream used for job-complete in this probe.

b) What to change it to

# Component Target behavior Acceptance criteria Constraints
1 Single writer One authoritative meter + adjust path; statement usage == usage_entries After CS adjust + stamp, :3010 or successor statement matches edge/libSQL tip; no empty books SoT D001/D006; do not dual-append chain
2 libSQL BusinessStore BUSINESS_STORE=libsql + LIBSQL_URL; fail closed 503 if writer down Contract tests; migrator tip verify; live meter only to libSQL Auth stays zappier.db (C13)
3 Freeze doc Publish docs/DUAL-WRITER-FREEZE.md (or update checklist to ENV-MATRIX as SoT) File present on live /opt/verae/zappier-edge/docs/ or QA checklist amended Do not invent conflicting writers
4 Billing facade Fold into edge HTTP BusinessStore or libSQL-backed NATS listener Health nats:true if kept; statement == ledger Stable verae.billing.* subjects
5 Async complete Real completedAt; hook delivery path E2E jobId → terminal event (+ hook attempt) verae.ts.* core NATS only
6 Harness 1–7 Automated report under plans/qa/ Flows 1,2,4 MOCK then 3,5,6,7; exit 0 —
7 Prepaid gate Documented 402 vs allow-with-debt Seed funded + zero-balance fixture tests Do not rewrite historical ledger
8 D004 Automated leaf deny Tests green; live deny ≠ bare 404 Leaf may still stamp
9 Explicit env BUSINESS_STORE=sqlite-file (then libsql) in secrets systemctl/environ shows flag Deploy never rsyncs *.db*

c) How to implement

Aligned with PM board; do not skip merge gates.

flowchart TD
  DocFix[Publish DUAL-WRITER-FREEZE.md or amend checklist] --> MG2
  EnvFlag[Explicit BUSINESS_STORE in secrets] --> MG2
  MG2[MG2 writers unlock after migrator] --> WP_L_done[Confirm WP-L meter-only BusinessStore]
  WP_L_done --> WP_M[WP-M fold/rewire account-balance]
  WP_M --> WP_N[WP-N CS/sales/acct]
  WP_N --> MG3[MG3 single writer proof]
  MG3 --> Harness[WP-P harness 1-7]
  MG3 --> Async[Fix completedAt / hook path]
  Harness --> MG4[MG4]

Phase A — Doc / ops closeout (hours)

  • Add /opt/verae/zappier-edge/docs/DUAL-WRITER-FREEZE.md (content can mirror ENV-MATRIX freeze paragraph + owner + daily rule).
  • Set BUSINESS_STORE=sqlite-file explicitly in /SSD2/secrets/zap-verae/zappier-edge.env; restart; confirm journal + environ.

Phase B — MG2 → MG3 (days; blocked on libSQL migrator if not done)

  • Migrator billing.db (+ books) → libSQL; refuse on verifyChain fail; archive tip under plans/qa/.
  • Rewire/stop second writer; statement == meter proof.

Phase C — Product paths + harness

  • Fix single-stamp receipt completedAt; prove flow 3 under harness (MOCK first).
  • Automate flows 1–7; D004 deny tests; prepaid policy fixture.

Validation for this retest exit: checklist items 1–3,5–6 green is enough to call sqlite-file deploy accepted; item 4 and dual-SoT remain open work, not deploy regressions.


d) Expected results — and what remains incomplete

After planned changes

Metric / behavior Today (this retest) Expected after Phases A–C / MG3
BusinessStore on edge sqlite-file live libSQL primary
Flow 2 digests stamp 202 + headers + chain ok Same UX on libSQL
Statement vs meter Divergent ([] vs n=6) Identical
DUAL-WRITER-FREEZE.md Missing file Present (or checklist SoT = ENV-MATRIX)
Flow 3 completedAt 0001-01-01… Real completion time
Flow 7 D004 404 only Automated deny
blockchain.db Untouched Still untouched

Still missing even after those todos

Gap Why
Zapier Platform live push / workspace Zaps Operator Zapier login; out of cutover buffer
NATS nkeys/mTLS Lab isolation is network, not accounts
libSQL HA / multi-primary Launch = one primary + backup
Peergos / Go timestamping redesign Out of scope
Full arch sign-off leftovers (C2/C3/C6…) Docs project parallel

Success tiers

Tier Status after this retest
Sqlite-file deploy smoke Met (journal, 202/idempotent, verifyChain, health, upstream optional)
Dual-writer freeze artifact Partial (policy in ENV-MATRIX; named file missing)
MG3 single-writer Not met
WP-Q / double userflow green Not met

e) Prioritized todo list

Item 1 — [P0] — Publish docs/DUAL-WRITER-FREEZE.md on live edge (or amend QA checklist)
Ship named freeze runbook to /opt/verae/zappier-edge/docs/ mirroring ENV-MATRIX authority rule, or formally designate ENV-MATRIX as the freeze SoT.
PreReqs: (none)

Item 2 — [P0] — Explicit BUSINESS_STORE=sqlite-file in zappier-edge.env
Add flag to secrets; restart; confirm environ + journal; keep BILLING_DB path clear.
PreReqs: (none)

Item 3 — [P0] — Complete migrator + libSQL cutover with verifyChain gate (WP-K / MG2)
Refuse cutover if chain fails; archive tip report under plans/qa/.
PreReqs: (Item 2)

Item 4 — [P0] — Fold or rewire account-balance to BusinessStore/libSQL (WP-M)
Eliminate empty statement SoT; nats connected if facade kept; statement usage == ledger.
PreReqs: (Item 3)

Item 5 — [P0] — CS/sales/accounting-export same store + MG3 proof (WP-N)
Adjust visible on statement and edge balance headers.
PreReqs: (Item 4)

Item 6 — [P1] — Fix single-stamp completedAt / async receipt wiring (flow 3)
Replace 0001-01-01 placeholders on product path used by Zapier wait/hook.
PreReqs: (none)

Item 7 — [P1] — Automate D004 leaf billing deny (WP-D)
Replace 404-only smoke with authz allow-list tests.
PreReqs: (none)

Item 8 — [P1] — Userflow harness skeleton flows 1, 2, 4 then 3, 5, 6, 7 (WP-P)
MOCK_VERAE report artifact under plans/qa/; MG4.
PreReqs: (Item 5 + Item 7)

Item 9 — [P2] — Prepaid zero-balance policy gate
Define 402 vs allow-with-debt; seed funded fixture.
PreReqs: (Item 4)

Item 10 — [P2] — Multi-host LIBSQL_URL topology (WP-O)
Only after MG3.
PreReqs: (Item 5)

Dependency order (Do all): Item 1 ∥ Item 2 ∥ Item 6 ∥ Item 7 → Item 3 → Item 4 → Item 5 → Item 8 → Item 9 → Item 10.


Probe artifacts (raw)

  • Batch digest: 5bdeb622aae9311ae2e9a5f6cea85ad615711f119a598c1ad212b6b03a4eacbd (qa-retest-wpl-2026-10-03)
  • Batch jobId: d012fb8f-ed92-448d-b9d2-afe27776c11a
  • Single digest: 2a36132dd089f1231764fb37eac38999cf1531e008c28a797ff95c94f6703525 (qa-single-wpl)
  • Single jobId: 6e925fbc-9b89-4704-8f7f-5f8793ce5b86
  • Chain tip after probe: 8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5 (length 6)
  • Upstream jobIds: fa1838fc-b6b5-40c6-a05f-93439f2c0c0c, 930de378-994b-429b-a713-9fa27af0965f

QA note: Sqlite-file BusinessStore deploy is accepted for lab meter path. Dual-writer / MG3 / harness remain the critical path. Formal WP-Q churn still waits on MG3 + harness skeleton.