15 KiB
QA Retest — BusinessStore sqlite-file deploy (WP-L precursor)
Role: QA (evaluate only; no product feature implementation)
Date: 2026-10-03 (UTC probe ~06:21–06:24)
Host: NS1.GEORGELAMBERT.ORG / 70.88.205.138
Baseline: 2026-10-03-baseline.md
Scope: Retest after live edge deploy of SqliteFileBusinessStore (BusinessStore driver: sqlite-file). Confirm checklist items; compare userflows 1–7 vs baseline; emit whattochange a–e for remaining breaks.
Constraints honored: No writes to Go timestamping DBs; billing.db not cleared (read-only verify + normal meter appends from stamp probes only).
Summary judgment
WP-G / sqlite-file path on live edge is green: journal logs BusinessStore driver: sqlite-file, digests-only batch on :3040 with key-ada returns 202 then idempotent existing/free, and usage_entries verifyChain is ok (length 6 after this probe). Health matrix :3040/:3130/:3010/:3100 all 200. Upstream api.veraetime.net digests-only batch still 202.
Still not MG3 / multi-host ready: dedicated DUAL-WRITER-FREEZE.md is missing (freeze text only in docs/ENV-MATRIX.md); :3010 statement remains empty vs non-empty edge ledger (dual SoT / C16 still live); account-balance nats:false; single-stamp completedAt still epoch-zero; flows 1, 3–7 unchanged vs baseline readiness. Do not treat WP-Q as entered or split hosts.
Checklist results
| # | Check | Result | Evidence |
|---|---|---|---|
| 1 | journal shows BusinessStore driver: sqlite-file |
PASS | journalctl -u zappier-edge.service: Oct 02 21:21:15 … BusinessStore driver: sqlite-file and Billing customers/usage via SqliteFileBusinessStore (BILLING_DB). Process env has PORT=3040, ZAPPIER_UPSTREAM=http://127.0.0.1:3130; BUSINESS_STORE not explicitly set (defaults to sqlite-file). |
| 2 | POST /v1/timestamps digests-only key-ada → 202; re-POST same hash → existing/free |
PASS | Digest sha256("qa-retest-wpl-2026-10-03") = 5bdeb622…eacbd. POST1 → HTTP 202, created:1, existing:0, X-Verae-Credits-Spend: 10. POST2 → HTTP 202, created:0, existing:1, totalCredits:0, X-Verae-Credits-Spend: 0. Channel api. |
| 3 | usage chain verifyChain ok on billing.db |
PASS | Readonly SqliteUsageRepo.verifyChain() on /opt/verae/zappier-edge/billing.db → {"ok":true,"length":6,"tipHash":"8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5"}. WAL. Ada (cust_1) balance_cents=0; usage rollup cust_1 n=6 sum=60 credits. |
| 4 | DUAL-WRITER-FREEZE.md present |
FAIL | ls → no such file under /opt/verae/zappier-edge/ or docs/. Freeze policy text exists inside docs/ENV-MATRIX.md (“Dual-writer freeze (2026-10-03): Meter and balance writes go only through zappier-edge → BusinessStore…”). Checklist asked for the named file — not present. |
| 5 | health matrix :3040/:3130/:3010/:3100 |
PASS | See table below. Access planes :3021–3025 also 200. |
| 6 | Optional: api.veraetime.net login + POST /api/batch/timestamp two digests |
PASS | Login admin/admin123 → JWT. Batch digests-only two items → HTTP 202 {"jobIds":["fa1838fc-…","930de378-…"]}. Confirms upstream still digests-only / batch path. |
Health matrix (this probe)
| Port | Role | /health |
Notes |
|---|---|---|---|
:3040 |
zappier-edge | 200 {"ok":true,"role":"zappier-edge"} |
BusinessStore sqlite-file |
:3130 |
middleware product | 200 mockVerae:false, natsConnected:true |
Edge upstream |
:3010 |
account-balance | 200 nats:false |
Statement empty; dual path |
:3100 |
middleware mock | 200 mockVerae:true, natsConnected:false |
Do not smoke here |
Go DB (C9)
/var/lib/timestamping-svc/data/blockchain.db— mtime 2026-09-25, size ~31 MB. Not modified by this QA (stat/ls only).
Dual-writer / statement vs meter (still broken)
| Store | Observation |
|---|---|
Edge billing.db |
usage_entries length 6, chain tip ok; period spend headers 60 after probes |
:3010/statement/cust_1 |
200 {prepaidCents:0, usage:[], credits:[], payments:[]} — does not see edge usage |
| C16 | Still live: meter writer = edge BusinessStore/sqlite-file; facade books path still empty / out of band |
Sqlite-file deploy moved edge metering behind BusinessStore, but did not eliminate the second empty ledger behind account-balance.
Userflow readiness 1–7 vs baseline
| # | Userflow | Baseline | This retest | Remaining blocker |
|---|---|---|---|---|
| 1 | Signup → API key → Statement | Partial; statement empty vs edge | Unchanged / still blocked | Seed keys work; :3010 statement still usage:[] while edge has 6 rows; no portal E2E harness; dual SoT |
| 2 | Stamp digests single + batch + credit headers | Mostly green | Improved evidence / still green | sqlite-file meter + verifyChain ok; still local SQLite (not libSQL); zero balance still meters; not multi-host |
| 3 | Async complete → Zapier REST Hook | Not proven | Still broken / unproven | Single POST /v1/timestamp receipt completedAt:"0001-01-01T00:00:00Z" (jobId 6e925fbc-…); no watch→events→webhook E2E |
| 4 | CS credit adjust → portal balance | Broken / dual path | Still broken | Adjust would hit :3010/books while meter hits BusinessStore/billing.db; nats:false |
| 5 | Sales quote → meter match | Unproven | Still unproven | Rates still edge SQLite; no shared SoT / harness |
| 6 | Accounting export consistency | Fails if run | Still fails | Export cannot reconcile empty statement/books vs usage_entries n=6 |
| 7 | Leaf stamp OK / billing deny | Health + 404 only | Unchanged | :3023/health 200; POST :3023/v1/billing/adjust → 404 not found (≠ D004 automated deny) |
PM QA entry (MG3): still not met. Progress vs baseline: BusinessStore + SqliteFile on live edge, chain verify green, ENV-MATRIX published, digests-only + upstream batch reconfirmed. Missing for MG3: libSQL cutover, single writer proof (statement == meter), account-balance fold/rewire, harness, D004 tests, dedicated freeze doc filename if still required by ops.
a) What to change
| # | Component | Problem | Evidence | Severity | Type |
|---|---|---|---|---|---|
| 1 | Dual prepaid/ledger path (C16) | Edge BusinessStore meters billing.db; :3010 statement still empty separate store |
statement usage:[] vs verifyChain length 6; both services up |
critical | design flaw / dual writer |
| 2 | Missing libSQL SoT (D006) | Lab still sqlite-file only; no LIBSQL_URL in edge environ |
environ grep; journal sqlite-file |
critical | missing foundation |
| 3 | DUAL-WRITER-FREEZE.md filename |
Checklist/ops expect named freeze doc; only ENV-MATRIX paragraph exists | ls missing file; ENV-MATRIX has freeze text |
medium | doc drift / incomplete deliverable |
| 4 | Account-balance NATS / facade | nats:false; not libSQL-backed; not single-writer |
health JSON | high | ops / wiring |
| 5 | Async receipt / flow 3 | completedAt epoch-zero on single stamp |
curl body 0001-01-01T00:00:00Z |
high | incomplete product path |
| 6 | Userflow harness (WP-P) | Flows 1–7 not automated | No harness report artifact beyond this manual retest | high | missing gate |
| 7 | Zero-balance metering | Ada balance_cents=0 still stamps |
customers row + 202 + Balance header 0 | medium | missing prepaid gate |
| 8 | D004 leaf deny | 404 ≠ policy deny | curl 404 on invent-ed path | medium | missing test |
| 9 | BUSINESS_STORE not explicit in secrets |
Relies on code default; harder to audit live | environ lacks BUSINESS_STORE= |
low | ops hygiene |
What is working (preserve)
- Live
BusinessStore driver: sqlite-file+ SqliteFileBusinessStore overBILLING_DB. - Digests-only batch
:3040→:3130withkey-ada, credit headers, idempotent existing/free. verifyChainok after meter appends (hash chain intact).- Product upstream
:3130(not mock:3100); ENV-MATRIX documents the port roles + freeze policy. - Upstream Verae Time digests-only batch authenticated path.
- Access planes healthy; Go
blockchain.dbuntouched; no JetStream used for job-complete in this probe.
b) What to change it to
| # | Component | Target behavior | Acceptance criteria | Constraints |
|---|---|---|---|---|
| 1 | Single writer | One authoritative meter + adjust path; statement usage == usage_entries |
After CS adjust + stamp, :3010 or successor statement matches edge/libSQL tip; no empty books SoT |
D001/D006; do not dual-append chain |
| 2 | libSQL BusinessStore | BUSINESS_STORE=libsql + LIBSQL_URL; fail closed 503 if writer down |
Contract tests; migrator tip verify; live meter only to libSQL | Auth stays zappier.db (C13) |
| 3 | Freeze doc | Publish docs/DUAL-WRITER-FREEZE.md (or update checklist to ENV-MATRIX as SoT) |
File present on live /opt/verae/zappier-edge/docs/ or QA checklist amended |
Do not invent conflicting writers |
| 4 | Billing facade | Fold into edge HTTP BusinessStore or libSQL-backed NATS listener | Health nats:true if kept; statement == ledger |
Stable verae.billing.* subjects |
| 5 | Async complete | Real completedAt; hook delivery path |
E2E jobId → terminal event (+ hook attempt) | verae.ts.* core NATS only |
| 6 | Harness 1–7 | Automated report under plans/qa/ |
Flows 1,2,4 MOCK then 3,5,6,7; exit 0 | — |
| 7 | Prepaid gate | Documented 402 vs allow-with-debt | Seed funded + zero-balance fixture tests | Do not rewrite historical ledger |
| 8 | D004 | Automated leaf deny | Tests green; live deny ≠ bare 404 | Leaf may still stamp |
| 9 | Explicit env | BUSINESS_STORE=sqlite-file (then libsql) in secrets |
systemctl/environ shows flag |
Deploy never rsyncs *.db* |
c) How to implement
Aligned with PM board; do not skip merge gates.
flowchart TD
DocFix[Publish DUAL-WRITER-FREEZE.md or amend checklist] --> MG2
EnvFlag[Explicit BUSINESS_STORE in secrets] --> MG2
MG2[MG2 writers unlock after migrator] --> WP_L_done[Confirm WP-L meter-only BusinessStore]
WP_L_done --> WP_M[WP-M fold/rewire account-balance]
WP_M --> WP_N[WP-N CS/sales/acct]
WP_N --> MG3[MG3 single writer proof]
MG3 --> Harness[WP-P harness 1-7]
MG3 --> Async[Fix completedAt / hook path]
Harness --> MG4[MG4]
Phase A — Doc / ops closeout (hours)
- Add
/opt/verae/zappier-edge/docs/DUAL-WRITER-FREEZE.md(content can mirror ENV-MATRIX freeze paragraph + owner + daily rule). - Set
BUSINESS_STORE=sqlite-fileexplicitly in/SSD2/secrets/zap-verae/zappier-edge.env; restart; confirm journal + environ.
Phase B — MG2 → MG3 (days; blocked on libSQL migrator if not done)
- Migrator
billing.db(+ books) → libSQL; refuse onverifyChainfail; archive tip underplans/qa/. - Rewire/stop second writer; statement == meter proof.
Phase C — Product paths + harness
- Fix single-stamp receipt
completedAt; prove flow 3 under harness (MOCK first). - Automate flows 1–7; D004 deny tests; prepaid policy fixture.
Validation for this retest exit: checklist items 1–3,5–6 green is enough to call sqlite-file deploy accepted; item 4 and dual-SoT remain open work, not deploy regressions.
d) Expected results — and what remains incomplete
After planned changes
| Metric / behavior | Today (this retest) | Expected after Phases A–C / MG3 |
|---|---|---|
| BusinessStore on edge | sqlite-file live | libSQL primary |
| Flow 2 digests stamp | 202 + headers + chain ok | Same UX on libSQL |
| Statement vs meter | Divergent ([] vs n=6) |
Identical |
DUAL-WRITER-FREEZE.md |
Missing file | Present (or checklist SoT = ENV-MATRIX) |
Flow 3 completedAt |
0001-01-01… |
Real completion time |
| Flow 7 D004 | 404 only | Automated deny |
blockchain.db |
Untouched | Still untouched |
Still missing even after those todos
| Gap | Why |
|---|---|
| Zapier Platform live push / workspace Zaps | Operator Zapier login; out of cutover buffer |
| NATS nkeys/mTLS | Lab isolation is network, not accounts |
| libSQL HA / multi-primary | Launch = one primary + backup |
| Peergos / Go timestamping redesign | Out of scope |
| Full arch sign-off leftovers (C2/C3/C6…) | Docs project parallel |
Success tiers
| Tier | Status after this retest |
|---|---|
| Sqlite-file deploy smoke | Met (journal, 202/idempotent, verifyChain, health, upstream optional) |
| Dual-writer freeze artifact | Partial (policy in ENV-MATRIX; named file missing) |
| MG3 single-writer | Not met |
| WP-Q / double userflow green | Not met |
e) Prioritized todo list
Item 1 — [P0] — Publish docs/DUAL-WRITER-FREEZE.md on live edge (or amend QA checklist)
Ship named freeze runbook to /opt/verae/zappier-edge/docs/ mirroring ENV-MATRIX authority rule, or formally designate ENV-MATRIX as the freeze SoT.
PreReqs: (none)
Item 2 — [P0] — Explicit BUSINESS_STORE=sqlite-file in zappier-edge.env
Add flag to secrets; restart; confirm environ + journal; keep BILLING_DB path clear.
PreReqs: (none)
Item 3 — [P0] — Complete migrator + libSQL cutover with verifyChain gate (WP-K / MG2)
Refuse cutover if chain fails; archive tip report under plans/qa/.
PreReqs: (Item 2)
Item 4 — [P0] — Fold or rewire account-balance to BusinessStore/libSQL (WP-M)
Eliminate empty statement SoT; nats connected if facade kept; statement usage == ledger.
PreReqs: (Item 3)
Item 5 — [P0] — CS/sales/accounting-export same store + MG3 proof (WP-N)
Adjust visible on statement and edge balance headers.
PreReqs: (Item 4)
Item 6 — [P1] — Fix single-stamp completedAt / async receipt wiring (flow 3)
Replace 0001-01-01 placeholders on product path used by Zapier wait/hook.
PreReqs: (none)
Item 7 — [P1] — Automate D004 leaf billing deny (WP-D)
Replace 404-only smoke with authz allow-list tests.
PreReqs: (none)
Item 8 — [P1] — Userflow harness skeleton flows 1, 2, 4 then 3, 5, 6, 7 (WP-P)
MOCK_VERAE report artifact under plans/qa/; MG4.
PreReqs: (Item 5 + Item 7)
Item 9 — [P2] — Prepaid zero-balance policy gate
Define 402 vs allow-with-debt; seed funded fixture.
PreReqs: (Item 4)
Item 10 — [P2] — Multi-host LIBSQL_URL topology (WP-O)
Only after MG3.
PreReqs: (Item 5)
Dependency order (Do all): Item 1 ∥ Item 2 ∥ Item 6 ∥ Item 7 → Item 3 → Item 4 → Item 5 → Item 8 → Item 9 → Item 10.
Probe artifacts (raw)
- Batch digest:
5bdeb622aae9311ae2e9a5f6cea85ad615711f119a598c1ad212b6b03a4eacbd(qa-retest-wpl-2026-10-03) - Batch jobId:
d012fb8f-ed92-448d-b9d2-afe27776c11a - Single digest:
2a36132dd089f1231764fb37eac38999cf1531e008c28a797ff95c94f6703525(qa-single-wpl) - Single jobId:
6e925fbc-9b89-4704-8f7f-5f8793ce5b86 - Chain tip after probe:
8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5(length 6) - Upstream jobIds:
fa1838fc-b6b5-40c6-a05f-93439f2c0c0c,930de378-994b-429b-a713-9fa27af0965f
QA note: Sqlite-file BusinessStore deploy is accepted for lab meter path. Dual-writer / MG3 / harness remain the critical path. Formal WP-Q churn still waits on MG3 + harness skeleton.