zapier-decisions/plans/qa/2026-10-03-retest-mg3.md

16 KiB
Raw Blame History

QA Retest — WP-M/N + BusinessStore v2 (MG3 probe)

Role: QA (evaluate only; no product feature implementation)
Date: 2026-10-03 (UTC probe ~06:34–06:37)
Host: NS1.GEORGELAMBERT.ORG / 70.88.205.138
Baseline: 2026-10-03-baseline.md
Prior: 2026-10-03-retest-wpl.md
Scope: Post WP-M (account-balance facade) + WP-N (CS/sales/acct booksWrites:false) retest against BusinessStore sqlite-file on live edge. Score userflows 1–7; judge MG3 prepaid-writer status.
Constraints honored: No DB clears; no intentional ledger adjusts; Go blockchain.db untouched (mtime still 2026-09-25). Meter appends only from normal stamp probes.


Summary judgment

WP-M/N lab progress is real: :3010 is a billing.db facade (facade:true, source:"billing.db"); GET /statement/cust_1 usage aligns with edge usage_entries (n=8 after this probe); CS :3011 / sales :3012 / accounting-export :3013 all advertise edge: http://127.0.0.1:3040 and booksWrites: false; freeze doc is present at docs/DUAL-WRITER-FREEZE.md; digests-only create+idempotent on :3040 still 202; upstream Verae batch with admin JWT still 202.

MG3 is not fully met. There is one live meter writer (zappier-edge SqliteFileBusinessStore → billing.db). There is not yet one prepaid/adjust writer on libSQL (LIBSQL_URL absent; BUSINESS_STORE=sqlite-file). Freeze text still forbids production use of verae.billing.balance.adjust until re-pointed. :3010 health remains nats:false. Flow 3 completedAt still epoch-zero. D004 leaf deny still 404, not policy deny. Do not start WP-O host split.


Checklist results

# Check Result Evidence
1 Edge BusinessStore sqlite-file still up; freeze doc at docs/DUAL-WRITER-FREEZE.md PASS Journal (last start Oct 02 21:27:46): BusinessStore driver: sqlite-file, Billing customers/usage via SqliteFileBusinessStore (BILLING_DB). Secrets: BUSINESS_STORE=sqlite-file, PORT=3040, ZAPPIER_UPSTREAM=http://127.0.0.1:3130. File present: /opt/verae/zappier-edge/docs/DUAL-WRITER-FREEZE.md (350 B, authoritative meter+balance writes = edge BusinessStore; :3010 legacy facade).
2 :3010 statement cust_1 usage aligns with billing.db (facade) PASS Pre-probe: statement usage n=6, source:"billing.db", prepaid 0; sqlite usage_entries cust_1 n=6 sum=60. Post-stamp: statement n=8, db n=8 sum=80, same source. Tip after probe verifyChain {"ok":true,"length":8,"tipHash":"abf1c625…639792a"}. Stale books.json still empty (mtime Sep 11) and not used for statement.
3 CS/sales/acct health shows edge URL and booksWrites: false PASS :3011 CS {"edge":"http://127.0.0.1:3040","booksWrites":false,"nats":true}; :3012 sales same edge + booksWrites:false; :3013 accounting-export same.
4 Digest stamp flow on :3040 (create + idempotent) PASS Digest sha256("qa-retest-mg3-2026-10-03")=d638ddc8…d7b919. POST1 → 202 created:1 spend 10; POST2 → 202 created:0 existing:1 spend 0. Channel api. Period spend headers 70 then 80 after single follow-up.
5 Optional Verae batch with admin token PASS POST /auth/login admin/admin123 → JWT. POST /api/batch/timestamp two digests-only items → 202 jobIds 1d027dcd-…, 8db4f651-….
6 Userflows 1–7 scored See table below —
7 whattochange a–e for remaining gaps See § whattochange —
8 MG3: one prepaid writer now? PARTIAL / not MG3 See § MG3 status

Health matrix (this probe)

Port Role /health Notes
:3040 zappier-edge 200 role:zappier-edge BusinessStore sqlite-file
:3130 middleware product 200 mockVerae:false, natsConnected:true Edge upstream
:3100 middleware mock 200 mockVerae:true, natsConnected:false Do not smoke here
:3010 account-balance 200 facade:true, d006:true, nats:false, billingDb:/opt/verae/zappier-edge/billing.db, edgeUrl:http://127.0.0.1:3040 Read facade
:3011 customer-service 200 edge + booksWrites:false WP-N
:3012 sales-pricing 200 edge + booksWrites:false WP-N
:3013 accounting-export 200 edge + booksWrites:false WP-N
:3021–:3025 access planes 200 web/api/leaf/zapier/staff

Go DB (C9)

  • /var/lib/timestamping-svc/data/blockchain.db — mtime 2026-09-25, size ~31 MB. Not modified (stat only).

Dual-writer / statement vs meter (improved)

Store Observation
Edge billing.db WAL; usage_entries length 8 after probe; verifyChain ok; Ada balance_cents=0
:3010/statement/cust_1 200 usage n=8, source:"billing.db", prepaid 0 — aligns with ledger
Legacy books.json Exists under fleet runtime; empty counts; mtime Sep 11; not statement SoT
C16 Mitigated for reads/meter: facade reads BusinessStore file; CS/sales/acct claim no books writes. Not closed for prepaid adjust: freeze still bans production balance.adjust; NATS on :3010 still false; store still sqlite-file not libSQL

Userflow readiness 1–7

# Userflow Score Evidence / remaining blocker
1 Signup → API key → Statement PARTIAL Seed key-ada works; statement now matches billing.db (was empty at baseline/WPL). No portal signup E2E harness; prepaid still 0; facade nats:false.
2 Stamp digests single + batch + credit headers PASS Batch create+idempotent 202; credit headers; chain ok; upstream Verae batch 202. Still sqlite-file (not libSQL); zero-balance still meters.
3 Async complete → Zapier REST Hook FAIL Single POST /v1/timestamp receipt completedAt:"0001-01-01T00:00:00Z" (jobId a7c2842a-…); no watch→events→webhook E2E.
4 CS credit adjust → portal balance PARTIAL Health shows edge + booksWrites:false (WP-N intent). Freeze still says do not use :3010 adjust for production credits; no adjust E2E in this QA (by design: no intentional ledger writes).
5 Sales quote → meter match PARTIAL Sales :3012 points at edge, booksWrites:false. Rate/quote↔meter harness not run; rates still edge SQLite.
6 Accounting export consistency PARTIAL Acct :3013 edge + booksWrites:false; statement↔usage now consistent so export can reconcile. Export E2E not executed.
7 Leaf stamp OK / billing deny PARTIAL :3023/health 200; POST :3023/v1/billing/adjust → 404 not found (≠ D004 automated policy deny).

Vs WPL retest: flows 1, 4–6 moved from broken/unproven toward PARTIAL because facade statement aligns and staff planes advertise no books writes. Flow 2 remains green. Flows 3 and 7 unchanged in failure mode.


MG3 status — is there one prepaid writer now?

Short answer: one meter writer, yes; full MG3 prepaid/libSQL writer, no.

MG3 criterion (PM board) This probe
One prepaid writer; second paths removed Partial. Meter appends only via edge BusinessStore → billing.db. :3010 is read facade over that file. CS/sales/acct booksWrites:false. Stale books.json not serving statement.
Runtime: one LIBSQL_URL serves customers/pricing/usage/invoices/balances FAIL. No LIBSQL_URL; BUSINESS_STORE=sqlite-file.
Grep: no prod SoT via books.json / per-host billing.db Partial. Lab still uses host-local billing.db as SoT (accepted freeze stand-in until libSQL). books.json deprecated for statement but file remains.
D004 leaf cannot adjust Weak. 404 only; not automated deny proof.
Freeze / adjust path Freeze explicitly still blocks production verae.billing.balance.adjust until re-pointed at BusinessStore/libSQL → prepaid adjust writer not proven unified.

Verdict: Treat WP-M/N sqlite-file facade cutover as accepted for lab read-path + meter SoT alignment. Do not mark MG3 PASS. MG3 still requires libSQL SoT + proven single adjust path (or documented fold of adjust into edge BusinessStore) + D004 tests.


a) What to change

# Component Problem Evidence Severity Type
1 Missing libSQL SoT (D006 / MG3) Lab still sqlite-file billing.db; no LIBSQL_URL secrets BUSINESS_STORE=sqlite-file; journal critical missing foundation
2 Prepaid adjust path still frozen Freeze forbids production balance.adjust; :3010 nats:false freeze doc; health JSON critical incomplete WP-M
3 Async receipt / flow 3 completedAt epoch-zero on single stamp curl body 0001-01-01T00:00:00Z high incomplete product path
4 Userflow harness (WP-P) Flows 1–7 not automated; 4–6 only health-level this manual retest high missing gate
5 D004 leaf deny 404 ≠ policy deny curl 404 medium missing test
6 Zero-balance metering Ada balance_cents=0 still stamps customers row + Balance header 0 medium missing prepaid gate
7 Stale books.json artifact Empty file remains in fleet runtime; confusion risk path under verae-fleet/.../books.json low ops hygiene

What is working (preserve)

  • Live BusinessStore driver: sqlite-file + explicit BUSINESS_STORE=sqlite-file in secrets.
  • docs/DUAL-WRITER-FREEZE.md on live edge.
  • :3010 facade reading billing.db; statement == usage_entries for cust_1.
  • CS/sales/acct booksWrites:false + edge URL.
  • Digests-only batch create+idempotent on :3040 with credit headers; verifyChain ok.
  • Product upstream :3130; Verae Time /auth/login + /api/batch/timestamp digests-only.
  • Access planes healthy; Go blockchain.db untouched; no JetStream used for job-complete in this probe.

b) What to change it to

# Component Target behavior Acceptance criteria Constraints
1 libSQL BusinessStore BUSINESS_STORE=libsql + LIBSQL_URL; fail closed 503 if writer down Migrator tip verify archived; live meter only to libSQL Auth stays zappier.db (C13)
2 Single adjust writer CS adjust / verae.billing.balance.adjust → same BusinessStore; unfreeze after proof Adjust visible on statement + edge balance headers; freeze doc updated D001; do not dual-append chain; no books.json writes
3 Async complete Real completedAt; hook delivery path E2E jobId → terminal event (+ hook attempt) verae.ts.* core NATS only
4 Harness 1–7 Automated report under plans/qa/ Flows 1,2,4 MOCK then 3,5,6,7; exit 0 —
5 D004 Automated leaf deny Tests green; live deny ≠ bare 404 Leaf may still stamp
6 Prepaid gate Documented 402 vs allow-with-debt Seed funded + zero-balance fixture Do not rewrite historical ledger
7 Retire books.json Import-only / archived; not in runtime SoT path Ops note; facade never falls back to books for money —

c) How to implement

Aligned with PM board; do not skip merge gates.

flowchart TD
  WPM_done[WP-M facade reads billing.db] --> WPN_done[WP-N booksWrites false]
  WPN_done --> AdjustProof[Prove adjust via BusinessStore / NATS]
  AdjustProof --> LibSQL[WP-K/O libSQL cutover + verifyChain]
  LibSQL --> MG3[MG3 PASS: one prepaid writer on libSQL]
  MG3 --> Harness[WP-P harness 1-7]
  MG3 --> Async[Fix completedAt / hook path]
  Harness --> MG4[MG4]

Phase A — Close adjust writer (hours–days)

  • Point :3010 NATS adjust (or fold into edge HTTP) at BusinessStore; connect NATS or document HTTP-only facade.
  • Update freeze doc when adjust is safe; keep meter-only rule until then.
  • Prove: CS adjust → statement prepaid/credits change == edge headers (lab only).

Phase B — MG3 libSQL

  • Migrator billing.db → libSQL; refuse on verifyChain fail; archive tip under plans/qa/.
  • Flip BUSINESS_STORE=libsql; facade/BILLING_DB read path follows libSQL or HTTP billing API.
  • Grep proof: no production money writes to books.json or private per-host SoT.

Phase C — Product paths + harness

  • Fix single-stamp completedAt; prove flow 3 under harness (MOCK first).
  • Automate flows 1–7; D004 deny tests; prepaid policy fixture.

Validation for this retest exit: checklist 1–5 green ⇒ WP-M/N sqlite-file facade accepted. MG3 / libSQL / adjust proof remain open — not deploy regressions.


d) Expected results — and what remains incomplete

After planned changes

Metric / behavior Today (this retest) Expected after Phases A–C / MG3
BusinessStore on edge sqlite-file live libSQL primary
Statement vs meter Aligned (n=8) Same on libSQL
CS/sales/acct booksWrites false false + proven adjust/export
Freeze doc Present; adjust banned Present; adjust allowed via single writer
Flow 3 completedAt 0001-01-01… Real completion time
Flow 7 D004 404 only Automated deny
blockchain.db Untouched Still untouched

Still missing even after those todos

Gap Why
Zapier Platform live push / workspace Zaps Operator Zapier login; out of cutover buffer
NATS nkeys/mTLS Lab isolation is network, not accounts
libSQL HA / multi-primary Launch = one primary + backup
Peergos / Go timestamping redesign Out of scope

Success tiers

Tier Status after this retest
Sqlite-file BusinessStore + freeze doc Met
WP-M read facade (statement == ledger) Met
WP-N health (booksWrites:false + edge URL) Met
MG3 single prepaid writer on libSQL Not met
WP-Q / double userflow green Not met

e) Prioritized todo list

Item 1 — [P0] — Prove prepaid adjust through BusinessStore (finish WP-M write path)
Wire balance.adjust (NATS and/or edge HTTP) to same store meters use; update freeze when safe; keep books.json read-only/archived.
PreReqs: (none for lab sqlite-file; libSQL preferred)

Item 2 — [P0] — libSQL cutover with verifyChain gate (WP-K → MG3)
Refuse cutover if chain fails; archive tip under plans/qa/; set BUSINESS_STORE=libsql + LIBSQL_URL.
PreReqs: (Item 1 or explicit waiver that meter-only sqlite-file remains until cutover day)

Item 3 — [P0] — MG3 proof artifact
Grep + runtime proof: one writer; statement/adjust/meter same tip; no books.json SoT.
PreReqs: (Item 2)

Item 4 — [P1] — Fix single-stamp completedAt / async receipt (flow 3)
Replace 0001-01-01 placeholders on product path used by Zapier wait/hook; core NATS only for verae.ts.*.
PreReqs: (none)

Item 5 — [P1] — Automate D004 leaf billing deny (WP-D)
Replace 404-only smoke with authz allow-list tests.
PreReqs: (none)

Item 6 — [P1] — Userflow harness flows 1, 2, 4 then 3, 5, 6, 7 (WP-P)
MOCK_VERAE report under plans/qa/; MG4.
PreReqs: (Item 3 + Item 5)

Item 7 — [P2] — Prepaid zero-balance policy gate
Define 402 vs allow-with-debt; seed funded fixture.
PreReqs: (Item 1)

Item 8 — [P2] — Multi-host LIBSQL_URL topology (WP-O)
Only after MG3 PASS.
PreReqs: (Item 3)

Dependency order (Do all): Item 4 ∥ Item 5 → Item 1 → Item 2 → Item 3 → Item 6 → Item 7 → Item 8.


Probe artifacts (raw)

  • Batch digest: d638ddc81b601d6f025db6b982f42ce5abaee5aba0b6b2d496c57d764ed7b919 (qa-retest-mg3-2026-10-03)
  • Batch jobId: 231a1233-8a5c-49ee-8139-de8625039fe6
  • Single digest: f94d25942a900c8ca017aa5f4a1ddfbc071a0cc5993a575c1e058d707fdeb28b (qa-single-mg3)
  • Single jobId: a7c2842a-13b0-4196-a14b-4ec0c60f7013 (completedAt epoch-zero)
  • Chain tip after probe: abf1c6253f4ef38fd42e06ca7a167883a64bac0450d0c94454318fd0c639792a (length 8)
  • Upstream Verae jobIds: 1d027dcd-e976-4ca7-bb33-46f8275fe2b8, 8db4f651-ad06-476c-bc8d-5356bc079710
  • Statement post-probe: usage_n=8, source=billing.db, prepaidCents=0

QA note: WP-M/N facade + booksWrites:false deploy is accepted for lab read/meter alignment. MG3 remains open until libSQL + proven single prepaid adjust writer. Formal WP-Q churn still waits on MG3 + harness skeleton.