# QA Retest — BusinessStore sqlite-file deploy (WP-L precursor) **Role:** QA (evaluate only; no product feature implementation) **Date:** 2026-10-03 (UTC probe ~06:21–06:24) **Host:** `NS1.GEORGELAMBERT.ORG` / `70.88.205.138` **Baseline:** [`2026-10-03-baseline.md`](./2026-10-03-baseline.md) **Scope:** Retest after live edge deploy of `SqliteFileBusinessStore` (`BusinessStore driver: sqlite-file`). Confirm checklist items; compare userflows 1–7 vs baseline; emit whattochange a–e for remaining breaks. **Constraints honored:** No writes to Go timestamping DBs; `billing.db` not cleared (read-only verify + normal meter appends from stamp probes only). --- ## Summary judgment **WP-G / sqlite-file path on live edge is green:** journal logs `BusinessStore driver: sqlite-file`, digests-only batch on `:3040` with `key-ada` returns **202** then idempotent **existing/free**, and `usage_entries` `verifyChain` is **ok** (length 6 after this probe). Health matrix `:3040/:3130/:3010/:3100` all **200**. Upstream `api.veraetime.net` digests-only batch still **202**. **Still not MG3 / multi-host ready:** dedicated `DUAL-WRITER-FREEZE.md` is **missing** (freeze text only in `docs/ENV-MATRIX.md`); `:3010` statement remains empty vs non-empty edge ledger (**dual SoT / C16 still live**); account-balance `nats:false`; single-stamp `completedAt` still epoch-zero; flows 1, 3–7 unchanged vs baseline readiness. Do **not** treat WP-Q as entered or split hosts. --- ## Checklist results | # | Check | Result | Evidence | |---|--------|--------|----------| | 1 | journal shows `BusinessStore driver: sqlite-file` | **PASS** | `journalctl -u zappier-edge.service`: `Oct 02 21:21:15 … BusinessStore driver: sqlite-file` and `Billing customers/usage via SqliteFileBusinessStore (BILLING_DB)`. Process env has `PORT=3040`, `ZAPPIER_UPSTREAM=http://127.0.0.1:3130`; `BUSINESS_STORE` not explicitly set (defaults to sqlite-file). | | 2 | `POST /v1/timestamps` digests-only `key-ada` → 202; re-POST same hash → existing/free | **PASS** | Digest `sha256("qa-retest-wpl-2026-10-03")` = `5bdeb622…eacbd`. POST1 → **HTTP 202**, `created:1`, `existing:0`, `X-Verae-Credits-Spend: 10`. POST2 → **HTTP 202**, `created:0`, `existing:1`, `totalCredits:0`, `X-Verae-Credits-Spend: 0`. Channel `api`. | | 3 | usage chain `verifyChain` ok on `billing.db` | **PASS** | Readonly `SqliteUsageRepo.verifyChain()` on `/opt/verae/zappier-edge/billing.db` → `{"ok":true,"length":6,"tipHash":"8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5"}`. WAL. Ada (`cust_1`) `balance_cents=0`; usage rollup `cust_1` n=6 sum=60 credits. | | 4 | `DUAL-WRITER-FREEZE.md` present | **FAIL** | `ls` → no such file under `/opt/verae/zappier-edge/` or `docs/`. Freeze **policy text exists** inside [`docs/ENV-MATRIX.md`](file:///opt/verae/zappier-edge/docs/ENV-MATRIX.md) (“Dual-writer freeze (2026-10-03): Meter and balance writes go only through zappier-edge → BusinessStore…”). Checklist asked for the named file — **not present**. | | 5 | health matrix `:3040/:3130/:3010/:3100` | **PASS** | See table below. Access planes `:3021–3025` also **200**. | | 6 | Optional: `api.veraetime.net` login + `POST /api/batch/timestamp` two digests | **PASS** | Login `admin`/`admin123` → JWT. Batch digests-only two items → **HTTP 202** `{"jobIds":["fa1838fc-…","930de378-…"]}`. Confirms upstream still digests-only / batch path. | ### Health matrix (this probe) | Port | Role | `/health` | Notes | |------|------|-----------|-------| | `:3040` | zappier-edge | **200** `{"ok":true,"role":"zappier-edge"}` | BusinessStore sqlite-file | | `:3130` | middleware product | **200** `mockVerae:false`, `natsConnected:true` | Edge upstream | | `:3010` | account-balance | **200** `nats:false` | Statement empty; dual path | | `:3100` | middleware mock | **200** `mockVerae:true`, `natsConnected:false` | Do not smoke here | ### Go DB (C9) - `/var/lib/timestamping-svc/data/blockchain.db` — mtime **2026-09-25**, size ~31 MB. **Not modified** by this QA (stat/ls only). --- ## Dual-writer / statement vs meter (still broken) | Store | Observation | |-------|-------------| | Edge `billing.db` | `usage_entries` length **6**, chain tip ok; period spend headers **60** after probes | | `:3010/statement/cust_1` | **200** `{prepaidCents:0, usage:[], credits:[], payments:[]}` — **does not see** edge usage | | C16 | **Still live:** meter writer = edge BusinessStore/sqlite-file; facade books path still empty / out of band | Sqlite-file deploy moved edge metering behind `BusinessStore`, but **did not** eliminate the second empty ledger behind account-balance. --- ## Userflow readiness 1–7 vs baseline | # | Userflow | Baseline | This retest | Remaining blocker | |---|----------|----------|-------------|-------------------| | 1 | Signup → API key → Statement | Partial; statement empty vs edge | **Unchanged / still blocked** | Seed keys work; `:3010` statement still `usage:[]` while edge has 6 rows; no portal E2E harness; dual SoT | | 2 | Stamp digests single + batch + credit headers | Mostly green | **Improved evidence / still green** | sqlite-file meter + verifyChain ok; still local SQLite (not libSQL); zero balance still meters; not multi-host | | 3 | Async complete → Zapier REST Hook | Not proven | **Still broken / unproven** | Single `POST /v1/timestamp` receipt `completedAt:"0001-01-01T00:00:00Z"` (jobId `6e925fbc-…`); no watch→events→webhook E2E | | 4 | CS credit adjust → portal balance | Broken / dual path | **Still broken** | Adjust would hit `:3010`/books while meter hits BusinessStore/`billing.db`; `nats:false` | | 5 | Sales quote → meter match | Unproven | **Still unproven** | Rates still edge SQLite; no shared SoT / harness | | 6 | Accounting export consistency | Fails if run | **Still fails** | Export cannot reconcile empty statement/books vs `usage_entries` n=6 | | 7 | Leaf stamp OK / billing deny | Health + 404 only | **Unchanged** | `:3023/health` 200; `POST :3023/v1/billing/adjust` → **404** `not found` (≠ D004 automated deny) | **PM QA entry (MG3):** still **not met**. Progress vs baseline: BusinessStore + SqliteFile **on live edge**, chain verify green, ENV-MATRIX published, digests-only + upstream batch reconfirmed. Missing for MG3: libSQL cutover, single writer proof (statement == meter), account-balance fold/rewire, harness, D004 tests, dedicated freeze doc filename if still required by ops. --- ## a) What to change | # | Component | Problem | Evidence | Severity | Type | |---|-----------|---------|----------|----------|------| | 1 | Dual prepaid/ledger path (C16) | Edge BusinessStore meters `billing.db`; `:3010` statement still empty separate store | statement `usage:[]` vs verifyChain length 6; both services up | **critical** | design flaw / dual writer | | 2 | Missing libSQL SoT (D006) | Lab still sqlite-file only; no `LIBSQL_URL` in edge environ | environ grep; journal `sqlite-file` | **critical** | missing foundation | | 3 | `DUAL-WRITER-FREEZE.md` filename | Checklist/ops expect named freeze doc; only ENV-MATRIX paragraph exists | `ls` missing file; ENV-MATRIX has freeze text | **medium** | doc drift / incomplete deliverable | | 4 | Account-balance NATS / facade | `nats:false`; not libSQL-backed; not single-writer | health JSON | **high** | ops / wiring | | 5 | Async receipt / flow 3 | `completedAt` epoch-zero on single stamp | curl body `0001-01-01T00:00:00Z` | **high** | incomplete product path | | 6 | Userflow harness (WP-P) | Flows 1–7 not automated | No harness report artifact beyond this manual retest | **high** | missing gate | | 7 | Zero-balance metering | Ada `balance_cents=0` still stamps | customers row + 202 + Balance header 0 | **medium** | missing prepaid gate | | 8 | D004 leaf deny | 404 ≠ policy deny | curl 404 on invent-ed path | **medium** | missing test | | 9 | `BUSINESS_STORE` not explicit in secrets | Relies on code default; harder to audit live | environ lacks `BUSINESS_STORE=` | **low** | ops hygiene | ### What is working (preserve) - Live `BusinessStore driver: sqlite-file` + SqliteFileBusinessStore over `BILLING_DB`. - Digests-only batch `:3040` → `:3130` with `key-ada`, credit headers, idempotent existing/free. - `verifyChain` ok after meter appends (hash chain intact). - Product upstream `:3130` (not mock `:3100`); ENV-MATRIX documents the port roles + freeze policy. - Upstream Verae Time digests-only batch authenticated path. - Access planes healthy; Go `blockchain.db` untouched; no JetStream used for job-complete in this probe. --- ## b) What to change it to | # | Component | Target behavior | Acceptance criteria | Constraints | |---|-----------|-----------------|---------------------|-------------| | 1 | Single writer | One authoritative meter + adjust path; statement usage == `usage_entries` | After CS adjust + stamp, `:3010` or successor statement matches edge/libSQL tip; no empty books SoT | D001/D006; do not dual-append chain | | 2 | libSQL BusinessStore | `BUSINESS_STORE=libsql` + `LIBSQL_URL`; fail closed 503 if writer down | Contract tests; migrator tip verify; live meter only to libSQL | Auth stays `zappier.db` (C13) | | 3 | Freeze doc | Publish `docs/DUAL-WRITER-FREEZE.md` (or update checklist to ENV-MATRIX as SoT) | File present on live `/opt/verae/zappier-edge/docs/` **or** QA checklist amended | Do not invent conflicting writers | | 4 | Billing facade | Fold into edge HTTP BusinessStore **or** libSQL-backed NATS listener | Health `nats:true` if kept; statement == ledger | Stable `verae.billing.*` subjects | | 5 | Async complete | Real `completedAt`; hook delivery path | E2E jobId → terminal event (+ hook attempt) | `verae.ts.*` **core NATS only** | | 6 | Harness 1–7 | Automated report under `plans/qa/` | Flows 1,2,4 MOCK then 3,5,6,7; exit 0 | — | | 7 | Prepaid gate | Documented 402 vs allow-with-debt | Seed funded + zero-balance fixture tests | Do not rewrite historical ledger | | 8 | D004 | Automated leaf deny | Tests green; live deny ≠ bare 404 | Leaf may still stamp | | 9 | Explicit env | `BUSINESS_STORE=sqlite-file` (then `libsql`) in secrets | `systemctl`/environ shows flag | Deploy never rsyncs `*.db*` | --- ## c) How to implement Aligned with PM board; **do not skip merge gates**. ```mermaid flowchart TD DocFix[Publish DUAL-WRITER-FREEZE.md or amend checklist] --> MG2 EnvFlag[Explicit BUSINESS_STORE in secrets] --> MG2 MG2[MG2 writers unlock after migrator] --> WP_L_done[Confirm WP-L meter-only BusinessStore] WP_L_done --> WP_M[WP-M fold/rewire account-balance] WP_M --> WP_N[WP-N CS/sales/acct] WP_N --> MG3[MG3 single writer proof] MG3 --> Harness[WP-P harness 1-7] MG3 --> Async[Fix completedAt / hook path] Harness --> MG4[MG4] ``` ### Phase A — Doc / ops closeout (hours) - Add `/opt/verae/zappier-edge/docs/DUAL-WRITER-FREEZE.md` (content can mirror ENV-MATRIX freeze paragraph + owner + daily rule). - Set `BUSINESS_STORE=sqlite-file` explicitly in `/SSD2/secrets/zap-verae/zappier-edge.env`; restart; confirm journal + environ. ### Phase B — MG2 → MG3 (days; blocked on libSQL migrator if not done) - Migrator `billing.db` (+ books) → libSQL; refuse on `verifyChain` fail; archive tip under `plans/qa/`. - Rewire/stop second writer; statement == meter proof. ### Phase C — Product paths + harness - Fix single-stamp receipt `completedAt`; prove flow 3 under harness (MOCK first). - Automate flows 1–7; D004 deny tests; prepaid policy fixture. **Validation for this retest exit:** checklist items 1–3,5–6 green is enough to call **sqlite-file deploy accepted**; item 4 and dual-SoT remain open work, not deploy regressions. --- ## d) Expected results — and what remains incomplete ### After planned changes | Metric / behavior | Today (this retest) | Expected after Phases A–C / MG3 | |-------------------|---------------------|----------------------------------| | BusinessStore on edge | sqlite-file live | libSQL primary | | Flow 2 digests stamp | 202 + headers + chain ok | Same UX on libSQL | | Statement vs meter | Divergent (`[]` vs n=6) | Identical | | `DUAL-WRITER-FREEZE.md` | Missing file | Present (or checklist SoT = ENV-MATRIX) | | Flow 3 `completedAt` | `0001-01-01…` | Real completion time | | Flow 7 D004 | 404 only | Automated deny | | `blockchain.db` | Untouched | Still untouched | ### Still missing even after those todos | Gap | Why | |-----|-----| | Zapier Platform live push / workspace Zaps | Operator Zapier login; out of cutover buffer | | NATS nkeys/mTLS | Lab isolation is network, not accounts | | libSQL HA / multi-primary | Launch = one primary + backup | | Peergos / Go timestamping redesign | Out of scope | | Full arch sign-off leftovers (C2/C3/C6…) | Docs project parallel | ### Success tiers | Tier | Status after this retest | |------|--------------------------| | Sqlite-file deploy smoke | **Met** (journal, 202/idempotent, verifyChain, health, upstream optional) | | Dual-writer freeze artifact | **Partial** (policy in ENV-MATRIX; named file missing) | | MG3 single-writer | **Not met** | | WP-Q / double userflow green | **Not met** | --- ## e) Prioritized todo list ``` Item 1 — [P0] — Publish docs/DUAL-WRITER-FREEZE.md on live edge (or amend QA checklist) Ship named freeze runbook to /opt/verae/zappier-edge/docs/ mirroring ENV-MATRIX authority rule, or formally designate ENV-MATRIX as the freeze SoT. PreReqs: (none) Item 2 — [P0] — Explicit BUSINESS_STORE=sqlite-file in zappier-edge.env Add flag to secrets; restart; confirm environ + journal; keep BILLING_DB path clear. PreReqs: (none) Item 3 — [P0] — Complete migrator + libSQL cutover with verifyChain gate (WP-K / MG2) Refuse cutover if chain fails; archive tip report under plans/qa/. PreReqs: (Item 2) Item 4 — [P0] — Fold or rewire account-balance to BusinessStore/libSQL (WP-M) Eliminate empty statement SoT; nats connected if facade kept; statement usage == ledger. PreReqs: (Item 3) Item 5 — [P0] — CS/sales/accounting-export same store + MG3 proof (WP-N) Adjust visible on statement and edge balance headers. PreReqs: (Item 4) Item 6 — [P1] — Fix single-stamp completedAt / async receipt wiring (flow 3) Replace 0001-01-01 placeholders on product path used by Zapier wait/hook. PreReqs: (none) Item 7 — [P1] — Automate D004 leaf billing deny (WP-D) Replace 404-only smoke with authz allow-list tests. PreReqs: (none) Item 8 — [P1] — Userflow harness skeleton flows 1, 2, 4 then 3, 5, 6, 7 (WP-P) MOCK_VERAE report artifact under plans/qa/; MG4. PreReqs: (Item 5 + Item 7) Item 9 — [P2] — Prepaid zero-balance policy gate Define 402 vs allow-with-debt; seed funded fixture. PreReqs: (Item 4) Item 10 — [P2] — Multi-host LIBSQL_URL topology (WP-O) Only after MG3. PreReqs: (Item 5) ``` **Dependency order (Do all):** Item 1 ∥ Item 2 ∥ Item 6 ∥ Item 7 → Item 3 → Item 4 → Item 5 → Item 8 → Item 9 → Item 10. --- ## Probe artifacts (raw) - Batch digest: `5bdeb622aae9311ae2e9a5f6cea85ad615711f119a598c1ad212b6b03a4eacbd` (`qa-retest-wpl-2026-10-03`) - Batch jobId: `d012fb8f-ed92-448d-b9d2-afe27776c11a` - Single digest: `2a36132dd089f1231764fb37eac38999cf1531e008c28a797ff95c94f6703525` (`qa-single-wpl`) - Single jobId: `6e925fbc-9b89-4704-8f7f-5f8793ce5b86` - Chain tip after probe: `8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5` (length 6) - Upstream jobIds: `fa1838fc-b6b5-40c6-a05f-93439f2c0c0c`, `930de378-994b-429b-a713-9fa27af0965f` **QA note:** Sqlite-file BusinessStore deploy is **accepted** for lab meter path. Dual-writer / MG3 / harness remain the critical path. Formal WP-Q churn still waits on MG3 + harness skeleton.