255 lines
16 KiB
Markdown
255 lines
16 KiB
Markdown
# QA Retest — WP-M/N + BusinessStore v2 (MG3 probe)
|
||
|
||
**Role:** QA (evaluate only; no product feature implementation)
|
||
**Date:** 2026-10-03 (UTC probe ~06:34–06:37)
|
||
**Host:** `NS1.GEORGELAMBERT.ORG` / `70.88.205.138`
|
||
**Baseline:** [`2026-10-03-baseline.md`](./2026-10-03-baseline.md)
|
||
**Prior:** [`2026-10-03-retest-wpl.md`](./2026-10-03-retest-wpl.md)
|
||
**Scope:** Post WP-M (account-balance facade) + WP-N (CS/sales/acct `booksWrites:false`) retest against BusinessStore sqlite-file on live edge. Score userflows 1–7; judge MG3 prepaid-writer status.
|
||
**Constraints honored:** No DB clears; no intentional ledger adjusts; Go `blockchain.db` **untouched** (mtime still 2026-09-25). Meter appends only from normal stamp probes.
|
||
|
||
---
|
||
|
||
## Summary judgment
|
||
|
||
**WP-M/N lab progress is real:** `:3010` is a **billing.db facade** (`facade:true`, `source:"billing.db"`); `GET /statement/cust_1` usage **aligns** with edge `usage_entries` (n=8 after this probe); CS `:3011` / sales `:3012` / accounting-export `:3013` all advertise `edge: http://127.0.0.1:3040` and **`booksWrites: false`**; freeze doc is present at [`docs/DUAL-WRITER-FREEZE.md`](file:///opt/verae/zappier-edge/docs/DUAL-WRITER-FREEZE.md); digests-only create+idempotent on `:3040` still **202**; upstream Verae batch with admin JWT still **202**.
|
||
|
||
**MG3 is not fully met.** There is **one live meter writer** (zappier-edge `SqliteFileBusinessStore` → `billing.db`). There is **not** yet one prepaid/adjust writer on **libSQL** (`LIBSQL_URL` absent; `BUSINESS_STORE=sqlite-file`). Freeze text still forbids production use of `verae.billing.balance.adjust` until re-pointed. `:3010` health remains `nats:false`. Flow 3 `completedAt` still epoch-zero. D004 leaf deny still **404**, not policy deny. Do **not** start WP-O host split.
|
||
|
||
---
|
||
|
||
## Checklist results
|
||
|
||
| # | Check | Result | Evidence |
|
||
|---|--------|--------|----------|
|
||
| 1 | Edge BusinessStore sqlite-file still up; freeze doc at `docs/DUAL-WRITER-FREEZE.md` | **PASS** | Journal (last start `Oct 02 21:27:46`): `BusinessStore driver: sqlite-file`, `Billing customers/usage via SqliteFileBusinessStore (BILLING_DB)`. Secrets: `BUSINESS_STORE=sqlite-file`, `PORT=3040`, `ZAPPIER_UPSTREAM=http://127.0.0.1:3130`. File present: `/opt/verae/zappier-edge/docs/DUAL-WRITER-FREEZE.md` (350 B, authoritative meter+balance writes = edge BusinessStore; `:3010` legacy facade). |
|
||
| 2 | `:3010` statement `cust_1` usage aligns with `billing.db` (facade) | **PASS** | Pre-probe: statement usage n=6, `source:"billing.db"`, prepaid 0; sqlite `usage_entries` cust_1 n=6 sum=60. Post-stamp: statement n=8, db n=8 sum=80, same source. Tip after probe `verifyChain` `{"ok":true,"length":8,"tipHash":"abf1c625…639792a"}`. Stale `books.json` still empty (mtime Sep 11) and **not** used for statement. |
|
||
| 3 | CS/sales/acct health shows edge URL and `booksWrites: false` | **PASS** | `:3011` CS `{"edge":"http://127.0.0.1:3040","booksWrites":false,"nats":true}`; `:3012` sales same edge + `booksWrites:false`; `:3013` accounting-export same. |
|
||
| 4 | Digest stamp flow on `:3040` (create + idempotent) | **PASS** | Digest `sha256("qa-retest-mg3-2026-10-03")`=`d638ddc8…d7b919`. POST1 → **202** `created:1` spend **10**; POST2 → **202** `created:0` `existing:1` spend **0**. Channel `api`. Period spend headers 70 then 80 after single follow-up. |
|
||
| 5 | Optional Verae batch with admin token | **PASS** | `POST /auth/login` admin/admin123 → JWT. `POST /api/batch/timestamp` two digests-only items → **202** `jobIds` `1d027dcd-…`, `8db4f651-…`. |
|
||
| 6 | Userflows 1–7 scored | See table below | — |
|
||
| 7 | whattochange a–e for remaining gaps | See § whattochange | — |
|
||
| 8 | MG3: one prepaid writer now? | **PARTIAL / not MG3** | See § MG3 status |
|
||
|
||
### Health matrix (this probe)
|
||
|
||
| Port | Role | `/health` | Notes |
|
||
|------|------|-----------|-------|
|
||
| `:3040` | zappier-edge | **200** `role:zappier-edge` | BusinessStore sqlite-file |
|
||
| `:3130` | middleware product | **200** `mockVerae:false`, `natsConnected:true` | Edge upstream |
|
||
| `:3100` | middleware mock | **200** `mockVerae:true`, `natsConnected:false` | Do not smoke here |
|
||
| `:3010` | account-balance | **200** `facade:true`, `d006:true`, `nats:false`, `billingDb:/opt/verae/zappier-edge/billing.db`, `edgeUrl:http://127.0.0.1:3040` | Read facade |
|
||
| `:3011` | customer-service | **200** edge + `booksWrites:false` | WP-N |
|
||
| `:3012` | sales-pricing | **200** edge + `booksWrites:false` | WP-N |
|
||
| `:3013` | accounting-export | **200** edge + `booksWrites:false` | WP-N |
|
||
| `:3021`–`:3025` | access planes | **200** | web/api/leaf/zapier/staff |
|
||
|
||
### Go DB (C9)
|
||
|
||
- `/var/lib/timestamping-svc/data/blockchain.db` — mtime **2026-09-25**, size ~31 MB. **Not modified** (stat only).
|
||
|
||
---
|
||
|
||
## Dual-writer / statement vs meter (improved)
|
||
|
||
| Store | Observation |
|
||
|-------|-------------|
|
||
| Edge `billing.db` | WAL; `usage_entries` length **8** after probe; `verifyChain` **ok**; Ada `balance_cents=0` |
|
||
| `:3010/statement/cust_1` | **200** usage n=**8**, `source:"billing.db"`, prepaid 0 — **aligns** with ledger |
|
||
| Legacy `books.json` | Exists under fleet runtime; empty counts; mtime Sep 11; **not** statement SoT |
|
||
| C16 | **Mitigated for reads/meter:** facade reads BusinessStore file; CS/sales/acct claim no books writes. **Not closed for prepaid adjust:** freeze still bans production `balance.adjust`; NATS on `:3010` still `false`; store still sqlite-file not libSQL |
|
||
|
||
---
|
||
|
||
## Userflow readiness 1–7
|
||
|
||
| # | Userflow | Score | Evidence / remaining blocker |
|
||
|---|----------|-------|------------------------------|
|
||
| 1 | Signup → API key → Statement | **PARTIAL** | Seed `key-ada` works; statement now matches `billing.db` (was empty at baseline/WPL). No portal signup E2E harness; prepaid still 0; facade `nats:false`. |
|
||
| 2 | Stamp digests single + batch + credit headers | **PASS** | Batch create+idempotent 202; credit headers; chain ok; upstream Verae batch 202. Still sqlite-file (not libSQL); zero-balance still meters. |
|
||
| 3 | Async complete → Zapier REST Hook | **FAIL** | Single `POST /v1/timestamp` receipt `completedAt:"0001-01-01T00:00:00Z"` (jobId `a7c2842a-…`); no watch→events→webhook E2E. |
|
||
| 4 | CS credit adjust → portal balance | **PARTIAL** | Health shows edge + `booksWrites:false` (WP-N intent). Freeze still says do not use `:3010` adjust for production credits; no adjust E2E in this QA (by design: no intentional ledger writes). |
|
||
| 5 | Sales quote → meter match | **PARTIAL** | Sales `:3012` points at edge, `booksWrites:false`. Rate/quote↔meter harness not run; rates still edge SQLite. |
|
||
| 6 | Accounting export consistency | **PARTIAL** | Acct `:3013` edge + `booksWrites:false`; statement↔usage now consistent so export *can* reconcile. Export E2E not executed. |
|
||
| 7 | Leaf stamp OK / billing deny | **PARTIAL** | `:3023/health` 200; `POST :3023/v1/billing/adjust` → **404** `not found` (≠ D004 automated policy deny). |
|
||
|
||
**Vs WPL retest:** flows 1, 4–6 moved from broken/unproven toward **PARTIAL** because facade statement aligns and staff planes advertise no books writes. Flow 2 remains green. Flows 3 and 7 unchanged in failure mode.
|
||
|
||
---
|
||
|
||
## MG3 status — is there one prepaid writer now?
|
||
|
||
**Short answer: one meter writer, yes; full MG3 prepaid/libSQL writer, no.**
|
||
|
||
| MG3 criterion (PM board) | This probe |
|
||
|--------------------------|------------|
|
||
| One prepaid writer; second paths removed | **Partial.** Meter appends only via edge BusinessStore → `billing.db`. `:3010` is read facade over that file. CS/sales/acct `booksWrites:false`. Stale `books.json` not serving statement. |
|
||
| Runtime: one `LIBSQL_URL` serves customers/pricing/usage/invoices/balances | **FAIL.** No `LIBSQL_URL`; `BUSINESS_STORE=sqlite-file`. |
|
||
| Grep: no prod SoT via `books.json` / per-host billing.db | **Partial.** Lab still uses host-local `billing.db` as SoT (accepted freeze stand-in until libSQL). `books.json` deprecated for statement but file remains. |
|
||
| D004 leaf cannot adjust | **Weak.** 404 only; not automated deny proof. |
|
||
| Freeze / adjust path | Freeze **explicitly** still blocks production `verae.billing.balance.adjust` until re-pointed at BusinessStore/libSQL → prepaid **adjust** writer not proven unified. |
|
||
|
||
**Verdict:** Treat WP-M/N sqlite-file facade cutover as **accepted for lab read-path + meter SoT alignment**. **Do not mark MG3 PASS.** MG3 still requires libSQL SoT + proven single adjust path (or documented fold of adjust into edge BusinessStore) + D004 tests.
|
||
|
||
---
|
||
|
||
## a) What to change
|
||
|
||
| # | Component | Problem | Evidence | Severity | Type |
|
||
|---|-----------|---------|----------|----------|------|
|
||
| 1 | Missing libSQL SoT (D006 / MG3) | Lab still sqlite-file `billing.db`; no `LIBSQL_URL` | secrets `BUSINESS_STORE=sqlite-file`; journal | **critical** | missing foundation |
|
||
| 2 | Prepaid adjust path still frozen | Freeze forbids production `balance.adjust`; `:3010` `nats:false` | freeze doc; health JSON | **critical** | incomplete WP-M |
|
||
| 3 | Async receipt / flow 3 | `completedAt` epoch-zero on single stamp | curl body `0001-01-01T00:00:00Z` | **high** | incomplete product path |
|
||
| 4 | Userflow harness (WP-P) | Flows 1–7 not automated; 4–6 only health-level | this manual retest | **high** | missing gate |
|
||
| 5 | D004 leaf deny | 404 ≠ policy deny | curl 404 | **medium** | missing test |
|
||
| 6 | Zero-balance metering | Ada `balance_cents=0` still stamps | customers row + Balance header 0 | **medium** | missing prepaid gate |
|
||
| 7 | Stale `books.json` artifact | Empty file remains in fleet runtime; confusion risk | path under `verae-fleet/.../books.json` | **low** | ops hygiene |
|
||
|
||
### What is working (preserve)
|
||
|
||
- Live `BusinessStore driver: sqlite-file` + explicit `BUSINESS_STORE=sqlite-file` in secrets.
|
||
- `docs/DUAL-WRITER-FREEZE.md` on live edge.
|
||
- `:3010` facade reading `billing.db`; statement == `usage_entries` for `cust_1`.
|
||
- CS/sales/acct `booksWrites:false` + edge URL.
|
||
- Digests-only batch create+idempotent on `:3040` with credit headers; `verifyChain` ok.
|
||
- Product upstream `:3130`; Verae Time `/auth/login` + `/api/batch/timestamp` digests-only.
|
||
- Access planes healthy; Go `blockchain.db` untouched; no JetStream used for job-complete in this probe.
|
||
|
||
---
|
||
|
||
## b) What to change it to
|
||
|
||
| # | Component | Target behavior | Acceptance criteria | Constraints |
|
||
|---|-----------|-----------------|---------------------|-------------|
|
||
| 1 | libSQL BusinessStore | `BUSINESS_STORE=libsql` + `LIBSQL_URL`; fail closed 503 if writer down | Migrator tip verify archived; live meter only to libSQL | Auth stays `zappier.db` (C13) |
|
||
| 2 | Single adjust writer | CS adjust / `verae.billing.balance.adjust` → same BusinessStore; unfreeze after proof | Adjust visible on statement + edge balance headers; freeze doc updated | D001; do not dual-append chain; no books.json writes |
|
||
| 3 | Async complete | Real `completedAt`; hook delivery path | E2E jobId → terminal event (+ hook attempt) | `verae.ts.*` **core NATS only** |
|
||
| 4 | Harness 1–7 | Automated report under `plans/qa/` | Flows 1,2,4 MOCK then 3,5,6,7; exit 0 | — |
|
||
| 5 | D004 | Automated leaf deny | Tests green; live deny ≠ bare 404 | Leaf may still stamp |
|
||
| 6 | Prepaid gate | Documented 402 vs allow-with-debt | Seed funded + zero-balance fixture | Do not rewrite historical ledger |
|
||
| 7 | Retire books.json | Import-only / archived; not in runtime SoT path | Ops note; facade never falls back to books for money | — |
|
||
|
||
---
|
||
|
||
## c) How to implement
|
||
|
||
Aligned with PM board; **do not skip merge gates**.
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
WPM_done[WP-M facade reads billing.db] --> WPN_done[WP-N booksWrites false]
|
||
WPN_done --> AdjustProof[Prove adjust via BusinessStore / NATS]
|
||
AdjustProof --> LibSQL[WP-K/O libSQL cutover + verifyChain]
|
||
LibSQL --> MG3[MG3 PASS: one prepaid writer on libSQL]
|
||
MG3 --> Harness[WP-P harness 1-7]
|
||
MG3 --> Async[Fix completedAt / hook path]
|
||
Harness --> MG4[MG4]
|
||
```
|
||
|
||
### Phase A — Close adjust writer (hours–days)
|
||
|
||
- Point `:3010` NATS adjust (or fold into edge HTTP) at BusinessStore; connect NATS or document HTTP-only facade.
|
||
- Update freeze doc when adjust is safe; keep meter-only rule until then.
|
||
- Prove: CS adjust → statement prepaid/credits change == edge headers (lab only).
|
||
|
||
### Phase B — MG3 libSQL
|
||
|
||
- Migrator `billing.db` → libSQL; refuse on `verifyChain` fail; archive tip under `plans/qa/`.
|
||
- Flip `BUSINESS_STORE=libsql`; facade/`BILLING_DB` read path follows libSQL or HTTP billing API.
|
||
- Grep proof: no production money writes to `books.json` or private per-host SoT.
|
||
|
||
### Phase C — Product paths + harness
|
||
|
||
- Fix single-stamp `completedAt`; prove flow 3 under harness (MOCK first).
|
||
- Automate flows 1–7; D004 deny tests; prepaid policy fixture.
|
||
|
||
**Validation for this retest exit:** checklist 1–5 green ⇒ WP-M/N sqlite-file facade **accepted**. MG3 / libSQL / adjust proof remain open — not deploy regressions.
|
||
|
||
---
|
||
|
||
## d) Expected results — and what remains incomplete
|
||
|
||
### After planned changes
|
||
|
||
| Metric / behavior | Today (this retest) | Expected after Phases A–C / MG3 |
|
||
|-------------------|---------------------|----------------------------------|
|
||
| BusinessStore on edge | sqlite-file live | libSQL primary |
|
||
| Statement vs meter | **Aligned** (n=8) | Same on libSQL |
|
||
| CS/sales/acct booksWrites | false | false + proven adjust/export |
|
||
| Freeze doc | Present; adjust banned | Present; adjust allowed via single writer |
|
||
| Flow 3 `completedAt` | `0001-01-01…` | Real completion time |
|
||
| Flow 7 D004 | 404 only | Automated deny |
|
||
| `blockchain.db` | Untouched | Still untouched |
|
||
|
||
### Still missing even after those todos
|
||
|
||
| Gap | Why |
|
||
|-----|-----|
|
||
| Zapier Platform live push / workspace Zaps | Operator Zapier login; out of cutover buffer |
|
||
| NATS nkeys/mTLS | Lab isolation is network, not accounts |
|
||
| libSQL HA / multi-primary | Launch = one primary + backup |
|
||
| Peergos / Go timestamping redesign | Out of scope |
|
||
|
||
### Success tiers
|
||
|
||
| Tier | Status after this retest |
|
||
|------|--------------------------|
|
||
| Sqlite-file BusinessStore + freeze doc | **Met** |
|
||
| WP-M read facade (`statement` == ledger) | **Met** |
|
||
| WP-N health (`booksWrites:false` + edge URL) | **Met** |
|
||
| MG3 single prepaid writer on libSQL | **Not met** |
|
||
| WP-Q / double userflow green | **Not met** |
|
||
|
||
---
|
||
|
||
## e) Prioritized todo list
|
||
|
||
```
|
||
Item 1 — [P0] — Prove prepaid adjust through BusinessStore (finish WP-M write path)
|
||
Wire balance.adjust (NATS and/or edge HTTP) to same store meters use; update freeze when safe; keep books.json read-only/archived.
|
||
PreReqs: (none for lab sqlite-file; libSQL preferred)
|
||
|
||
Item 2 — [P0] — libSQL cutover with verifyChain gate (WP-K → MG3)
|
||
Refuse cutover if chain fails; archive tip under plans/qa/; set BUSINESS_STORE=libsql + LIBSQL_URL.
|
||
PreReqs: (Item 1 or explicit waiver that meter-only sqlite-file remains until cutover day)
|
||
|
||
Item 3 — [P0] — MG3 proof artifact
|
||
Grep + runtime proof: one writer; statement/adjust/meter same tip; no books.json SoT.
|
||
PreReqs: (Item 2)
|
||
|
||
Item 4 — [P1] — Fix single-stamp completedAt / async receipt (flow 3)
|
||
Replace 0001-01-01 placeholders on product path used by Zapier wait/hook; core NATS only for verae.ts.*.
|
||
PreReqs: (none)
|
||
|
||
Item 5 — [P1] — Automate D004 leaf billing deny (WP-D)
|
||
Replace 404-only smoke with authz allow-list tests.
|
||
PreReqs: (none)
|
||
|
||
Item 6 — [P1] — Userflow harness flows 1, 2, 4 then 3, 5, 6, 7 (WP-P)
|
||
MOCK_VERAE report under plans/qa/; MG4.
|
||
PreReqs: (Item 3 + Item 5)
|
||
|
||
Item 7 — [P2] — Prepaid zero-balance policy gate
|
||
Define 402 vs allow-with-debt; seed funded fixture.
|
||
PreReqs: (Item 1)
|
||
|
||
Item 8 — [P2] — Multi-host LIBSQL_URL topology (WP-O)
|
||
Only after MG3 PASS.
|
||
PreReqs: (Item 3)
|
||
```
|
||
|
||
**Dependency order (Do all):** Item 4 ∥ Item 5 → Item 1 → Item 2 → Item 3 → Item 6 → Item 7 → Item 8.
|
||
|
||
---
|
||
|
||
## Probe artifacts (raw)
|
||
|
||
- Batch digest: `d638ddc81b601d6f025db6b982f42ce5abaee5aba0b6b2d496c57d764ed7b919` (`qa-retest-mg3-2026-10-03`)
|
||
- Batch jobId: `231a1233-8a5c-49ee-8139-de8625039fe6`
|
||
- Single digest: `f94d25942a900c8ca017aa5f4a1ddfbc071a0cc5993a575c1e058d707fdeb28b` (`qa-single-mg3`)
|
||
- Single jobId: `a7c2842a-13b0-4196-a14b-4ec0c60f7013` (`completedAt` epoch-zero)
|
||
- Chain tip after probe: `abf1c6253f4ef38fd42e06ca7a167883a64bac0450d0c94454318fd0c639792a` (length 8)
|
||
- Upstream Verae jobIds: `1d027dcd-e976-4ca7-bb33-46f8275fe2b8`, `8db4f651-ad06-476c-bc8d-5356bc079710`
|
||
- Statement post-probe: `usage_n=8`, `source=billing.db`, `prepaidCents=0`
|
||
|
||
**QA note:** WP-M/N facade + `booksWrites:false` deploy is **accepted** for lab read/meter alignment. **MG3 remains open** until libSQL + proven single prepaid adjust writer. Formal WP-Q churn still waits on MG3 + harness skeleton.
|