zapier-decisions/plans/qa/2026-10-03-retest-mg3.md

255 lines
16 KiB
Markdown
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# QA Retest — WP-M/N + BusinessStore v2 (MG3 probe)
**Role:** QA (evaluate only; no product feature implementation)
**Date:** 2026-10-03 (UTC probe ~06:34–06:37)
**Host:** `NS1.GEORGELAMBERT.ORG` / `70.88.205.138`
**Baseline:** [`2026-10-03-baseline.md`](./2026-10-03-baseline.md)
**Prior:** [`2026-10-03-retest-wpl.md`](./2026-10-03-retest-wpl.md)
**Scope:** Post WP-M (account-balance facade) + WP-N (CS/sales/acct `booksWrites:false`) retest against BusinessStore sqlite-file on live edge. Score userflows 1–7; judge MG3 prepaid-writer status.
**Constraints honored:** No DB clears; no intentional ledger adjusts; Go `blockchain.db` **untouched** (mtime still 2026-09-25). Meter appends only from normal stamp probes.
---
## Summary judgment
**WP-M/N lab progress is real:** `:3010` is a **billing.db facade** (`facade:true`, `source:"billing.db"`); `GET /statement/cust_1` usage **aligns** with edge `usage_entries` (n=8 after this probe); CS `:3011` / sales `:3012` / accounting-export `:3013` all advertise `edge: http://127.0.0.1:3040` and **`booksWrites: false`**; freeze doc is present at [`docs/DUAL-WRITER-FREEZE.md`](file:///opt/verae/zappier-edge/docs/DUAL-WRITER-FREEZE.md); digests-only create+idempotent on `:3040` still **202**; upstream Verae batch with admin JWT still **202**.
**MG3 is not fully met.** There is **one live meter writer** (zappier-edge `SqliteFileBusinessStore` → `billing.db`). There is **not** yet one prepaid/adjust writer on **libSQL** (`LIBSQL_URL` absent; `BUSINESS_STORE=sqlite-file`). Freeze text still forbids production use of `verae.billing.balance.adjust` until re-pointed. `:3010` health remains `nats:false`. Flow 3 `completedAt` still epoch-zero. D004 leaf deny still **404**, not policy deny. Do **not** start WP-O host split.
---
## Checklist results
| # | Check | Result | Evidence |
|---|--------|--------|----------|
| 1 | Edge BusinessStore sqlite-file still up; freeze doc at `docs/DUAL-WRITER-FREEZE.md` | **PASS** | Journal (last start `Oct 02 21:27:46`): `BusinessStore driver: sqlite-file`, `Billing customers/usage via SqliteFileBusinessStore (BILLING_DB)`. Secrets: `BUSINESS_STORE=sqlite-file`, `PORT=3040`, `ZAPPIER_UPSTREAM=http://127.0.0.1:3130`. File present: `/opt/verae/zappier-edge/docs/DUAL-WRITER-FREEZE.md` (350 B, authoritative meter+balance writes = edge BusinessStore; `:3010` legacy facade). |
| 2 | `:3010` statement `cust_1` usage aligns with `billing.db` (facade) | **PASS** | Pre-probe: statement usage n=6, `source:"billing.db"`, prepaid 0; sqlite `usage_entries` cust_1 n=6 sum=60. Post-stamp: statement n=8, db n=8 sum=80, same source. Tip after probe `verifyChain` `{"ok":true,"length":8,"tipHash":"abf1c625…639792a"}`. Stale `books.json` still empty (mtime Sep 11) and **not** used for statement. |
| 3 | CS/sales/acct health shows edge URL and `booksWrites: false` | **PASS** | `:3011` CS `{"edge":"http://127.0.0.1:3040","booksWrites":false,"nats":true}`; `:3012` sales same edge + `booksWrites:false`; `:3013` accounting-export same. |
| 4 | Digest stamp flow on `:3040` (create + idempotent) | **PASS** | Digest `sha256("qa-retest-mg3-2026-10-03")`=`d638ddc8…d7b919`. POST1 → **202** `created:1` spend **10**; POST2 → **202** `created:0` `existing:1` spend **0**. Channel `api`. Period spend headers 70 then 80 after single follow-up. |
| 5 | Optional Verae batch with admin token | **PASS** | `POST /auth/login` admin/admin123 → JWT. `POST /api/batch/timestamp` two digests-only items → **202** `jobIds` `1d027dcd-…`, `8db4f651-…`. |
| 6 | Userflows 1–7 scored | See table below | — |
| 7 | whattochange a–e for remaining gaps | See § whattochange | — |
| 8 | MG3: one prepaid writer now? | **PARTIAL / not MG3** | See § MG3 status |
### Health matrix (this probe)
| Port | Role | `/health` | Notes |
|------|------|-----------|-------|
| `:3040` | zappier-edge | **200** `role:zappier-edge` | BusinessStore sqlite-file |
| `:3130` | middleware product | **200** `mockVerae:false`, `natsConnected:true` | Edge upstream |
| `:3100` | middleware mock | **200** `mockVerae:true`, `natsConnected:false` | Do not smoke here |
| `:3010` | account-balance | **200** `facade:true`, `d006:true`, `nats:false`, `billingDb:/opt/verae/zappier-edge/billing.db`, `edgeUrl:http://127.0.0.1:3040` | Read facade |
| `:3011` | customer-service | **200** edge + `booksWrites:false` | WP-N |
| `:3012` | sales-pricing | **200** edge + `booksWrites:false` | WP-N |
| `:3013` | accounting-export | **200** edge + `booksWrites:false` | WP-N |
| `:3021`–`:3025` | access planes | **200** | web/api/leaf/zapier/staff |
### Go DB (C9)
- `/var/lib/timestamping-svc/data/blockchain.db` — mtime **2026-09-25**, size ~31 MB. **Not modified** (stat only).
---
## Dual-writer / statement vs meter (improved)
| Store | Observation |
|-------|-------------|
| Edge `billing.db` | WAL; `usage_entries` length **8** after probe; `verifyChain` **ok**; Ada `balance_cents=0` |
| `:3010/statement/cust_1` | **200** usage n=**8**, `source:"billing.db"`, prepaid 0 — **aligns** with ledger |
| Legacy `books.json` | Exists under fleet runtime; empty counts; mtime Sep 11; **not** statement SoT |
| C16 | **Mitigated for reads/meter:** facade reads BusinessStore file; CS/sales/acct claim no books writes. **Not closed for prepaid adjust:** freeze still bans production `balance.adjust`; NATS on `:3010` still `false`; store still sqlite-file not libSQL |
---
## Userflow readiness 1–7
| # | Userflow | Score | Evidence / remaining blocker |
|---|----------|-------|------------------------------|
| 1 | Signup → API key → Statement | **PARTIAL** | Seed `key-ada` works; statement now matches `billing.db` (was empty at baseline/WPL). No portal signup E2E harness; prepaid still 0; facade `nats:false`. |
| 2 | Stamp digests single + batch + credit headers | **PASS** | Batch create+idempotent 202; credit headers; chain ok; upstream Verae batch 202. Still sqlite-file (not libSQL); zero-balance still meters. |
| 3 | Async complete → Zapier REST Hook | **FAIL** | Single `POST /v1/timestamp` receipt `completedAt:"0001-01-01T00:00:00Z"` (jobId `a7c2842a-…`); no watch→events→webhook E2E. |
| 4 | CS credit adjust → portal balance | **PARTIAL** | Health shows edge + `booksWrites:false` (WP-N intent). Freeze still says do not use `:3010` adjust for production credits; no adjust E2E in this QA (by design: no intentional ledger writes). |
| 5 | Sales quote → meter match | **PARTIAL** | Sales `:3012` points at edge, `booksWrites:false`. Rate/quote↔meter harness not run; rates still edge SQLite. |
| 6 | Accounting export consistency | **PARTIAL** | Acct `:3013` edge + `booksWrites:false`; statement↔usage now consistent so export *can* reconcile. Export E2E not executed. |
| 7 | Leaf stamp OK / billing deny | **PARTIAL** | `:3023/health` 200; `POST :3023/v1/billing/adjust` → **404** `not found` (≠ D004 automated policy deny). |
**Vs WPL retest:** flows 1, 4–6 moved from broken/unproven toward **PARTIAL** because facade statement aligns and staff planes advertise no books writes. Flow 2 remains green. Flows 3 and 7 unchanged in failure mode.
---
## MG3 status — is there one prepaid writer now?
**Short answer: one meter writer, yes; full MG3 prepaid/libSQL writer, no.**
| MG3 criterion (PM board) | This probe |
|--------------------------|------------|
| One prepaid writer; second paths removed | **Partial.** Meter appends only via edge BusinessStore → `billing.db`. `:3010` is read facade over that file. CS/sales/acct `booksWrites:false`. Stale `books.json` not serving statement. |
| Runtime: one `LIBSQL_URL` serves customers/pricing/usage/invoices/balances | **FAIL.** No `LIBSQL_URL`; `BUSINESS_STORE=sqlite-file`. |
| Grep: no prod SoT via `books.json` / per-host billing.db | **Partial.** Lab still uses host-local `billing.db` as SoT (accepted freeze stand-in until libSQL). `books.json` deprecated for statement but file remains. |
| D004 leaf cannot adjust | **Weak.** 404 only; not automated deny proof. |
| Freeze / adjust path | Freeze **explicitly** still blocks production `verae.billing.balance.adjust` until re-pointed at BusinessStore/libSQL → prepaid **adjust** writer not proven unified. |
**Verdict:** Treat WP-M/N sqlite-file facade cutover as **accepted for lab read-path + meter SoT alignment**. **Do not mark MG3 PASS.** MG3 still requires libSQL SoT + proven single adjust path (or documented fold of adjust into edge BusinessStore) + D004 tests.
---
## a) What to change
| # | Component | Problem | Evidence | Severity | Type |
|---|-----------|---------|----------|----------|------|
| 1 | Missing libSQL SoT (D006 / MG3) | Lab still sqlite-file `billing.db`; no `LIBSQL_URL` | secrets `BUSINESS_STORE=sqlite-file`; journal | **critical** | missing foundation |
| 2 | Prepaid adjust path still frozen | Freeze forbids production `balance.adjust`; `:3010` `nats:false` | freeze doc; health JSON | **critical** | incomplete WP-M |
| 3 | Async receipt / flow 3 | `completedAt` epoch-zero on single stamp | curl body `0001-01-01T00:00:00Z` | **high** | incomplete product path |
| 4 | Userflow harness (WP-P) | Flows 1–7 not automated; 4–6 only health-level | this manual retest | **high** | missing gate |
| 5 | D004 leaf deny | 404 ≠ policy deny | curl 404 | **medium** | missing test |
| 6 | Zero-balance metering | Ada `balance_cents=0` still stamps | customers row + Balance header 0 | **medium** | missing prepaid gate |
| 7 | Stale `books.json` artifact | Empty file remains in fleet runtime; confusion risk | path under `verae-fleet/.../books.json` | **low** | ops hygiene |
### What is working (preserve)
- Live `BusinessStore driver: sqlite-file` + explicit `BUSINESS_STORE=sqlite-file` in secrets.
- `docs/DUAL-WRITER-FREEZE.md` on live edge.
- `:3010` facade reading `billing.db`; statement == `usage_entries` for `cust_1`.
- CS/sales/acct `booksWrites:false` + edge URL.
- Digests-only batch create+idempotent on `:3040` with credit headers; `verifyChain` ok.
- Product upstream `:3130`; Verae Time `/auth/login` + `/api/batch/timestamp` digests-only.
- Access planes healthy; Go `blockchain.db` untouched; no JetStream used for job-complete in this probe.
---
## b) What to change it to
| # | Component | Target behavior | Acceptance criteria | Constraints |
|---|-----------|-----------------|---------------------|-------------|
| 1 | libSQL BusinessStore | `BUSINESS_STORE=libsql` + `LIBSQL_URL`; fail closed 503 if writer down | Migrator tip verify archived; live meter only to libSQL | Auth stays `zappier.db` (C13) |
| 2 | Single adjust writer | CS adjust / `verae.billing.balance.adjust` → same BusinessStore; unfreeze after proof | Adjust visible on statement + edge balance headers; freeze doc updated | D001; do not dual-append chain; no books.json writes |
| 3 | Async complete | Real `completedAt`; hook delivery path | E2E jobId → terminal event (+ hook attempt) | `verae.ts.*` **core NATS only** |
| 4 | Harness 1–7 | Automated report under `plans/qa/` | Flows 1,2,4 MOCK then 3,5,6,7; exit 0 | — |
| 5 | D004 | Automated leaf deny | Tests green; live deny ≠ bare 404 | Leaf may still stamp |
| 6 | Prepaid gate | Documented 402 vs allow-with-debt | Seed funded + zero-balance fixture | Do not rewrite historical ledger |
| 7 | Retire books.json | Import-only / archived; not in runtime SoT path | Ops note; facade never falls back to books for money | — |
---
## c) How to implement
Aligned with PM board; **do not skip merge gates**.
```mermaid
flowchart TD
WPM_done[WP-M facade reads billing.db] --> WPN_done[WP-N booksWrites false]
WPN_done --> AdjustProof[Prove adjust via BusinessStore / NATS]
AdjustProof --> LibSQL[WP-K/O libSQL cutover + verifyChain]
LibSQL --> MG3[MG3 PASS: one prepaid writer on libSQL]
MG3 --> Harness[WP-P harness 1-7]
MG3 --> Async[Fix completedAt / hook path]
Harness --> MG4[MG4]
```
### Phase A — Close adjust writer (hours–days)
- Point `:3010` NATS adjust (or fold into edge HTTP) at BusinessStore; connect NATS or document HTTP-only facade.
- Update freeze doc when adjust is safe; keep meter-only rule until then.
- Prove: CS adjust → statement prepaid/credits change == edge headers (lab only).
### Phase B — MG3 libSQL
- Migrator `billing.db` → libSQL; refuse on `verifyChain` fail; archive tip under `plans/qa/`.
- Flip `BUSINESS_STORE=libsql`; facade/`BILLING_DB` read path follows libSQL or HTTP billing API.
- Grep proof: no production money writes to `books.json` or private per-host SoT.
### Phase C — Product paths + harness
- Fix single-stamp `completedAt`; prove flow 3 under harness (MOCK first).
- Automate flows 1–7; D004 deny tests; prepaid policy fixture.
**Validation for this retest exit:** checklist 1–5 green ⇒ WP-M/N sqlite-file facade **accepted**. MG3 / libSQL / adjust proof remain open — not deploy regressions.
---
## d) Expected results — and what remains incomplete
### After planned changes
| Metric / behavior | Today (this retest) | Expected after Phases A–C / MG3 |
|-------------------|---------------------|----------------------------------|
| BusinessStore on edge | sqlite-file live | libSQL primary |
| Statement vs meter | **Aligned** (n=8) | Same on libSQL |
| CS/sales/acct booksWrites | false | false + proven adjust/export |
| Freeze doc | Present; adjust banned | Present; adjust allowed via single writer |
| Flow 3 `completedAt` | `0001-01-01…` | Real completion time |
| Flow 7 D004 | 404 only | Automated deny |
| `blockchain.db` | Untouched | Still untouched |
### Still missing even after those todos
| Gap | Why |
|-----|-----|
| Zapier Platform live push / workspace Zaps | Operator Zapier login; out of cutover buffer |
| NATS nkeys/mTLS | Lab isolation is network, not accounts |
| libSQL HA / multi-primary | Launch = one primary + backup |
| Peergos / Go timestamping redesign | Out of scope |
### Success tiers
| Tier | Status after this retest |
|------|--------------------------|
| Sqlite-file BusinessStore + freeze doc | **Met** |
| WP-M read facade (`statement` == ledger) | **Met** |
| WP-N health (`booksWrites:false` + edge URL) | **Met** |
| MG3 single prepaid writer on libSQL | **Not met** |
| WP-Q / double userflow green | **Not met** |
---
## e) Prioritized todo list
```
Item 1 — [P0] — Prove prepaid adjust through BusinessStore (finish WP-M write path)
Wire balance.adjust (NATS and/or edge HTTP) to same store meters use; update freeze when safe; keep books.json read-only/archived.
PreReqs: (none for lab sqlite-file; libSQL preferred)
Item 2 — [P0] — libSQL cutover with verifyChain gate (WP-K → MG3)
Refuse cutover if chain fails; archive tip under plans/qa/; set BUSINESS_STORE=libsql + LIBSQL_URL.
PreReqs: (Item 1 or explicit waiver that meter-only sqlite-file remains until cutover day)
Item 3 — [P0] — MG3 proof artifact
Grep + runtime proof: one writer; statement/adjust/meter same tip; no books.json SoT.
PreReqs: (Item 2)
Item 4 — [P1] — Fix single-stamp completedAt / async receipt (flow 3)
Replace 0001-01-01 placeholders on product path used by Zapier wait/hook; core NATS only for verae.ts.*.
PreReqs: (none)
Item 5 — [P1] — Automate D004 leaf billing deny (WP-D)
Replace 404-only smoke with authz allow-list tests.
PreReqs: (none)
Item 6 — [P1] — Userflow harness flows 1, 2, 4 then 3, 5, 6, 7 (WP-P)
MOCK_VERAE report under plans/qa/; MG4.
PreReqs: (Item 3 + Item 5)
Item 7 — [P2] — Prepaid zero-balance policy gate
Define 402 vs allow-with-debt; seed funded fixture.
PreReqs: (Item 1)
Item 8 — [P2] — Multi-host LIBSQL_URL topology (WP-O)
Only after MG3 PASS.
PreReqs: (Item 3)
```
**Dependency order (Do all):** Item 4 ∥ Item 5 → Item 1 → Item 2 → Item 3 → Item 6 → Item 7 → Item 8.
---
## Probe artifacts (raw)
- Batch digest: `d638ddc81b601d6f025db6b982f42ce5abaee5aba0b6b2d496c57d764ed7b919` (`qa-retest-mg3-2026-10-03`)
- Batch jobId: `231a1233-8a5c-49ee-8139-de8625039fe6`
- Single digest: `f94d25942a900c8ca017aa5f4a1ddfbc071a0cc5993a575c1e058d707fdeb28b` (`qa-single-mg3`)
- Single jobId: `a7c2842a-13b0-4196-a14b-4ec0c60f7013` (`completedAt` epoch-zero)
- Chain tip after probe: `abf1c6253f4ef38fd42e06ca7a167883a64bac0450d0c94454318fd0c639792a` (length 8)
- Upstream Verae jobIds: `1d027dcd-e976-4ca7-bb33-46f8275fe2b8`, `8db4f651-ad06-476c-bc8d-5356bc079710`
- Statement post-probe: `usage_n=8`, `source=billing.db`, `prepaidCents=0`
**QA note:** WP-M/N facade + `booksWrites:false` deploy is **accepted** for lab read/meter alignment. **MG3 remains open** until libSQL + proven single prepaid adjust writer. Formal WP-Q churn still waits on MG3 + harness skeleton.