zapier-decisions/plans/qa/2026-10-03-retest-wpl.md

243 lines
15 KiB
Markdown
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# QA Retest — BusinessStore sqlite-file deploy (WP-L precursor)
**Role:** QA (evaluate only; no product feature implementation)
**Date:** 2026-10-03 (UTC probe ~06:21–06:24)
**Host:** `NS1.GEORGELAMBERT.ORG` / `70.88.205.138`
**Baseline:** [`2026-10-03-baseline.md`](./2026-10-03-baseline.md)
**Scope:** Retest after live edge deploy of `SqliteFileBusinessStore` (`BusinessStore driver: sqlite-file`). Confirm checklist items; compare userflows 1–7 vs baseline; emit whattochange a–e for remaining breaks.
**Constraints honored:** No writes to Go timestamping DBs; `billing.db` not cleared (read-only verify + normal meter appends from stamp probes only).
---
## Summary judgment
**WP-G / sqlite-file path on live edge is green:** journal logs `BusinessStore driver: sqlite-file`, digests-only batch on `:3040` with `key-ada` returns **202** then idempotent **existing/free**, and `usage_entries` `verifyChain` is **ok** (length 6 after this probe). Health matrix `:3040/:3130/:3010/:3100` all **200**. Upstream `api.veraetime.net` digests-only batch still **202**.
**Still not MG3 / multi-host ready:** dedicated `DUAL-WRITER-FREEZE.md` is **missing** (freeze text only in `docs/ENV-MATRIX.md`); `:3010` statement remains empty vs non-empty edge ledger (**dual SoT / C16 still live**); account-balance `nats:false`; single-stamp `completedAt` still epoch-zero; flows 1, 3–7 unchanged vs baseline readiness. Do **not** treat WP-Q as entered or split hosts.
---
## Checklist results
| # | Check | Result | Evidence |
|---|--------|--------|----------|
| 1 | journal shows `BusinessStore driver: sqlite-file` | **PASS** | `journalctl -u zappier-edge.service`: `Oct 02 21:21:15 … BusinessStore driver: sqlite-file` and `Billing customers/usage via SqliteFileBusinessStore (BILLING_DB)`. Process env has `PORT=3040`, `ZAPPIER_UPSTREAM=http://127.0.0.1:3130`; `BUSINESS_STORE` not explicitly set (defaults to sqlite-file). |
| 2 | `POST /v1/timestamps` digests-only `key-ada` → 202; re-POST same hash → existing/free | **PASS** | Digest `sha256("qa-retest-wpl-2026-10-03")` = `5bdeb622…eacbd`. POST1 → **HTTP 202**, `created:1`, `existing:0`, `X-Verae-Credits-Spend: 10`. POST2 → **HTTP 202**, `created:0`, `existing:1`, `totalCredits:0`, `X-Verae-Credits-Spend: 0`. Channel `api`. |
| 3 | usage chain `verifyChain` ok on `billing.db` | **PASS** | Readonly `SqliteUsageRepo.verifyChain()` on `/opt/verae/zappier-edge/billing.db` → `{"ok":true,"length":6,"tipHash":"8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5"}`. WAL. Ada (`cust_1`) `balance_cents=0`; usage rollup `cust_1` n=6 sum=60 credits. |
| 4 | `DUAL-WRITER-FREEZE.md` present | **FAIL** | `ls` → no such file under `/opt/verae/zappier-edge/` or `docs/`. Freeze **policy text exists** inside [`docs/ENV-MATRIX.md`](file:///opt/verae/zappier-edge/docs/ENV-MATRIX.md) (“Dual-writer freeze (2026-10-03): Meter and balance writes go only through zappier-edge → BusinessStore…”). Checklist asked for the named file — **not present**. |
| 5 | health matrix `:3040/:3130/:3010/:3100` | **PASS** | See table below. Access planes `:3021–3025` also **200**. |
| 6 | Optional: `api.veraetime.net` login + `POST /api/batch/timestamp` two digests | **PASS** | Login `admin`/`admin123` → JWT. Batch digests-only two items → **HTTP 202** `{"jobIds":["fa1838fc-…","930de378-…"]}`. Confirms upstream still digests-only / batch path. |
### Health matrix (this probe)
| Port | Role | `/health` | Notes |
|------|------|-----------|-------|
| `:3040` | zappier-edge | **200** `{"ok":true,"role":"zappier-edge"}` | BusinessStore sqlite-file |
| `:3130` | middleware product | **200** `mockVerae:false`, `natsConnected:true` | Edge upstream |
| `:3010` | account-balance | **200** `nats:false` | Statement empty; dual path |
| `:3100` | middleware mock | **200** `mockVerae:true`, `natsConnected:false` | Do not smoke here |
### Go DB (C9)
- `/var/lib/timestamping-svc/data/blockchain.db` — mtime **2026-09-25**, size ~31 MB. **Not modified** by this QA (stat/ls only).
---
## Dual-writer / statement vs meter (still broken)
| Store | Observation |
|-------|-------------|
| Edge `billing.db` | `usage_entries` length **6**, chain tip ok; period spend headers **60** after probes |
| `:3010/statement/cust_1` | **200** `{prepaidCents:0, usage:[], credits:[], payments:[]}` — **does not see** edge usage |
| C16 | **Still live:** meter writer = edge BusinessStore/sqlite-file; facade books path still empty / out of band |
Sqlite-file deploy moved edge metering behind `BusinessStore`, but **did not** eliminate the second empty ledger behind account-balance.
---
## Userflow readiness 1–7 vs baseline
| # | Userflow | Baseline | This retest | Remaining blocker |
|---|----------|----------|-------------|-------------------|
| 1 | Signup → API key → Statement | Partial; statement empty vs edge | **Unchanged / still blocked** | Seed keys work; `:3010` statement still `usage:[]` while edge has 6 rows; no portal E2E harness; dual SoT |
| 2 | Stamp digests single + batch + credit headers | Mostly green | **Improved evidence / still green** | sqlite-file meter + verifyChain ok; still local SQLite (not libSQL); zero balance still meters; not multi-host |
| 3 | Async complete → Zapier REST Hook | Not proven | **Still broken / unproven** | Single `POST /v1/timestamp` receipt `completedAt:"0001-01-01T00:00:00Z"` (jobId `6e925fbc-…`); no watch→events→webhook E2E |
| 4 | CS credit adjust → portal balance | Broken / dual path | **Still broken** | Adjust would hit `:3010`/books while meter hits BusinessStore/`billing.db`; `nats:false` |
| 5 | Sales quote → meter match | Unproven | **Still unproven** | Rates still edge SQLite; no shared SoT / harness |
| 6 | Accounting export consistency | Fails if run | **Still fails** | Export cannot reconcile empty statement/books vs `usage_entries` n=6 |
| 7 | Leaf stamp OK / billing deny | Health + 404 only | **Unchanged** | `:3023/health` 200; `POST :3023/v1/billing/adjust` → **404** `not found` (≠ D004 automated deny) |
**PM QA entry (MG3):** still **not met**. Progress vs baseline: BusinessStore + SqliteFile **on live edge**, chain verify green, ENV-MATRIX published, digests-only + upstream batch reconfirmed. Missing for MG3: libSQL cutover, single writer proof (statement == meter), account-balance fold/rewire, harness, D004 tests, dedicated freeze doc filename if still required by ops.
---
## a) What to change
| # | Component | Problem | Evidence | Severity | Type |
|---|-----------|---------|----------|----------|------|
| 1 | Dual prepaid/ledger path (C16) | Edge BusinessStore meters `billing.db`; `:3010` statement still empty separate store | statement `usage:[]` vs verifyChain length 6; both services up | **critical** | design flaw / dual writer |
| 2 | Missing libSQL SoT (D006) | Lab still sqlite-file only; no `LIBSQL_URL` in edge environ | environ grep; journal `sqlite-file` | **critical** | missing foundation |
| 3 | `DUAL-WRITER-FREEZE.md` filename | Checklist/ops expect named freeze doc; only ENV-MATRIX paragraph exists | `ls` missing file; ENV-MATRIX has freeze text | **medium** | doc drift / incomplete deliverable |
| 4 | Account-balance NATS / facade | `nats:false`; not libSQL-backed; not single-writer | health JSON | **high** | ops / wiring |
| 5 | Async receipt / flow 3 | `completedAt` epoch-zero on single stamp | curl body `0001-01-01T00:00:00Z` | **high** | incomplete product path |
| 6 | Userflow harness (WP-P) | Flows 1–7 not automated | No harness report artifact beyond this manual retest | **high** | missing gate |
| 7 | Zero-balance metering | Ada `balance_cents=0` still stamps | customers row + 202 + Balance header 0 | **medium** | missing prepaid gate |
| 8 | D004 leaf deny | 404 ≠ policy deny | curl 404 on invent-ed path | **medium** | missing test |
| 9 | `BUSINESS_STORE` not explicit in secrets | Relies on code default; harder to audit live | environ lacks `BUSINESS_STORE=` | **low** | ops hygiene |
### What is working (preserve)
- Live `BusinessStore driver: sqlite-file` + SqliteFileBusinessStore over `BILLING_DB`.
- Digests-only batch `:3040` → `:3130` with `key-ada`, credit headers, idempotent existing/free.
- `verifyChain` ok after meter appends (hash chain intact).
- Product upstream `:3130` (not mock `:3100`); ENV-MATRIX documents the port roles + freeze policy.
- Upstream Verae Time digests-only batch authenticated path.
- Access planes healthy; Go `blockchain.db` untouched; no JetStream used for job-complete in this probe.
---
## b) What to change it to
| # | Component | Target behavior | Acceptance criteria | Constraints |
|---|-----------|-----------------|---------------------|-------------|
| 1 | Single writer | One authoritative meter + adjust path; statement usage == `usage_entries` | After CS adjust + stamp, `:3010` or successor statement matches edge/libSQL tip; no empty books SoT | D001/D006; do not dual-append chain |
| 2 | libSQL BusinessStore | `BUSINESS_STORE=libsql` + `LIBSQL_URL`; fail closed 503 if writer down | Contract tests; migrator tip verify; live meter only to libSQL | Auth stays `zappier.db` (C13) |
| 3 | Freeze doc | Publish `docs/DUAL-WRITER-FREEZE.md` (or update checklist to ENV-MATRIX as SoT) | File present on live `/opt/verae/zappier-edge/docs/` **or** QA checklist amended | Do not invent conflicting writers |
| 4 | Billing facade | Fold into edge HTTP BusinessStore **or** libSQL-backed NATS listener | Health `nats:true` if kept; statement == ledger | Stable `verae.billing.*` subjects |
| 5 | Async complete | Real `completedAt`; hook delivery path | E2E jobId → terminal event (+ hook attempt) | `verae.ts.*` **core NATS only** |
| 6 | Harness 1–7 | Automated report under `plans/qa/` | Flows 1,2,4 MOCK then 3,5,6,7; exit 0 | — |
| 7 | Prepaid gate | Documented 402 vs allow-with-debt | Seed funded + zero-balance fixture tests | Do not rewrite historical ledger |
| 8 | D004 | Automated leaf deny | Tests green; live deny ≠ bare 404 | Leaf may still stamp |
| 9 | Explicit env | `BUSINESS_STORE=sqlite-file` (then `libsql`) in secrets | `systemctl`/environ shows flag | Deploy never rsyncs `*.db*` |
---
## c) How to implement
Aligned with PM board; **do not skip merge gates**.
```mermaid
flowchart TD
DocFix[Publish DUAL-WRITER-FREEZE.md or amend checklist] --> MG2
EnvFlag[Explicit BUSINESS_STORE in secrets] --> MG2
MG2[MG2 writers unlock after migrator] --> WP_L_done[Confirm WP-L meter-only BusinessStore]
WP_L_done --> WP_M[WP-M fold/rewire account-balance]
WP_M --> WP_N[WP-N CS/sales/acct]
WP_N --> MG3[MG3 single writer proof]
MG3 --> Harness[WP-P harness 1-7]
MG3 --> Async[Fix completedAt / hook path]
Harness --> MG4[MG4]
```
### Phase A — Doc / ops closeout (hours)
- Add `/opt/verae/zappier-edge/docs/DUAL-WRITER-FREEZE.md` (content can mirror ENV-MATRIX freeze paragraph + owner + daily rule).
- Set `BUSINESS_STORE=sqlite-file` explicitly in `/SSD2/secrets/zap-verae/zappier-edge.env`; restart; confirm journal + environ.
### Phase B — MG2 → MG3 (days; blocked on libSQL migrator if not done)
- Migrator `billing.db` (+ books) → libSQL; refuse on `verifyChain` fail; archive tip under `plans/qa/`.
- Rewire/stop second writer; statement == meter proof.
### Phase C — Product paths + harness
- Fix single-stamp receipt `completedAt`; prove flow 3 under harness (MOCK first).
- Automate flows 1–7; D004 deny tests; prepaid policy fixture.
**Validation for this retest exit:** checklist items 1–3,5–6 green is enough to call **sqlite-file deploy accepted**; item 4 and dual-SoT remain open work, not deploy regressions.
---
## d) Expected results — and what remains incomplete
### After planned changes
| Metric / behavior | Today (this retest) | Expected after Phases A–C / MG3 |
|-------------------|---------------------|----------------------------------|
| BusinessStore on edge | sqlite-file live | libSQL primary |
| Flow 2 digests stamp | 202 + headers + chain ok | Same UX on libSQL |
| Statement vs meter | Divergent (`[]` vs n=6) | Identical |
| `DUAL-WRITER-FREEZE.md` | Missing file | Present (or checklist SoT = ENV-MATRIX) |
| Flow 3 `completedAt` | `0001-01-01…` | Real completion time |
| Flow 7 D004 | 404 only | Automated deny |
| `blockchain.db` | Untouched | Still untouched |
### Still missing even after those todos
| Gap | Why |
|-----|-----|
| Zapier Platform live push / workspace Zaps | Operator Zapier login; out of cutover buffer |
| NATS nkeys/mTLS | Lab isolation is network, not accounts |
| libSQL HA / multi-primary | Launch = one primary + backup |
| Peergos / Go timestamping redesign | Out of scope |
| Full arch sign-off leftovers (C2/C3/C6…) | Docs project parallel |
### Success tiers
| Tier | Status after this retest |
|------|--------------------------|
| Sqlite-file deploy smoke | **Met** (journal, 202/idempotent, verifyChain, health, upstream optional) |
| Dual-writer freeze artifact | **Partial** (policy in ENV-MATRIX; named file missing) |
| MG3 single-writer | **Not met** |
| WP-Q / double userflow green | **Not met** |
---
## e) Prioritized todo list
```
Item 1 — [P0] — Publish docs/DUAL-WRITER-FREEZE.md on live edge (or amend QA checklist)
Ship named freeze runbook to /opt/verae/zappier-edge/docs/ mirroring ENV-MATRIX authority rule, or formally designate ENV-MATRIX as the freeze SoT.
PreReqs: (none)
Item 2 — [P0] — Explicit BUSINESS_STORE=sqlite-file in zappier-edge.env
Add flag to secrets; restart; confirm environ + journal; keep BILLING_DB path clear.
PreReqs: (none)
Item 3 — [P0] — Complete migrator + libSQL cutover with verifyChain gate (WP-K / MG2)
Refuse cutover if chain fails; archive tip report under plans/qa/.
PreReqs: (Item 2)
Item 4 — [P0] — Fold or rewire account-balance to BusinessStore/libSQL (WP-M)
Eliminate empty statement SoT; nats connected if facade kept; statement usage == ledger.
PreReqs: (Item 3)
Item 5 — [P0] — CS/sales/accounting-export same store + MG3 proof (WP-N)
Adjust visible on statement and edge balance headers.
PreReqs: (Item 4)
Item 6 — [P1] — Fix single-stamp completedAt / async receipt wiring (flow 3)
Replace 0001-01-01 placeholders on product path used by Zapier wait/hook.
PreReqs: (none)
Item 7 — [P1] — Automate D004 leaf billing deny (WP-D)
Replace 404-only smoke with authz allow-list tests.
PreReqs: (none)
Item 8 — [P1] — Userflow harness skeleton flows 1, 2, 4 then 3, 5, 6, 7 (WP-P)
MOCK_VERAE report artifact under plans/qa/; MG4.
PreReqs: (Item 5 + Item 7)
Item 9 — [P2] — Prepaid zero-balance policy gate
Define 402 vs allow-with-debt; seed funded fixture.
PreReqs: (Item 4)
Item 10 — [P2] — Multi-host LIBSQL_URL topology (WP-O)
Only after MG3.
PreReqs: (Item 5)
```
**Dependency order (Do all):** Item 1 ∥ Item 2 ∥ Item 6 ∥ Item 7 → Item 3 → Item 4 → Item 5 → Item 8 → Item 9 → Item 10.
---
## Probe artifacts (raw)
- Batch digest: `5bdeb622aae9311ae2e9a5f6cea85ad615711f119a598c1ad212b6b03a4eacbd` (`qa-retest-wpl-2026-10-03`)
- Batch jobId: `d012fb8f-ed92-448d-b9d2-afe27776c11a`
- Single digest: `2a36132dd089f1231764fb37eac38999cf1531e008c28a797ff95c94f6703525` (`qa-single-wpl`)
- Single jobId: `6e925fbc-9b89-4704-8f7f-5f8793ce5b86`
- Chain tip after probe: `8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5` (length 6)
- Upstream jobIds: `fa1838fc-b6b5-40c6-a05f-93439f2c0c0c`, `930de378-994b-429b-a713-9fa27af0965f`
**QA note:** Sqlite-file BusinessStore deploy is **accepted** for lab meter path. Dual-writer / MG3 / harness remain the critical path. Formal WP-Q churn still waits on MG3 + harness skeleton.