243 lines
15 KiB
Markdown
243 lines
15 KiB
Markdown
# QA Retest — BusinessStore sqlite-file deploy (WP-L precursor)
|
||
|
||
**Role:** QA (evaluate only; no product feature implementation)
|
||
**Date:** 2026-10-03 (UTC probe ~06:21–06:24)
|
||
**Host:** `NS1.GEORGELAMBERT.ORG` / `70.88.205.138`
|
||
**Baseline:** [`2026-10-03-baseline.md`](./2026-10-03-baseline.md)
|
||
**Scope:** Retest after live edge deploy of `SqliteFileBusinessStore` (`BusinessStore driver: sqlite-file`). Confirm checklist items; compare userflows 1–7 vs baseline; emit whattochange a–e for remaining breaks.
|
||
**Constraints honored:** No writes to Go timestamping DBs; `billing.db` not cleared (read-only verify + normal meter appends from stamp probes only).
|
||
|
||
---
|
||
|
||
## Summary judgment
|
||
|
||
**WP-G / sqlite-file path on live edge is green:** journal logs `BusinessStore driver: sqlite-file`, digests-only batch on `:3040` with `key-ada` returns **202** then idempotent **existing/free**, and `usage_entries` `verifyChain` is **ok** (length 6 after this probe). Health matrix `:3040/:3130/:3010/:3100` all **200**. Upstream `api.veraetime.net` digests-only batch still **202**.
|
||
|
||
**Still not MG3 / multi-host ready:** dedicated `DUAL-WRITER-FREEZE.md` is **missing** (freeze text only in `docs/ENV-MATRIX.md`); `:3010` statement remains empty vs non-empty edge ledger (**dual SoT / C16 still live**); account-balance `nats:false`; single-stamp `completedAt` still epoch-zero; flows 1, 3–7 unchanged vs baseline readiness. Do **not** treat WP-Q as entered or split hosts.
|
||
|
||
---
|
||
|
||
## Checklist results
|
||
|
||
| # | Check | Result | Evidence |
|
||
|---|--------|--------|----------|
|
||
| 1 | journal shows `BusinessStore driver: sqlite-file` | **PASS** | `journalctl -u zappier-edge.service`: `Oct 02 21:21:15 … BusinessStore driver: sqlite-file` and `Billing customers/usage via SqliteFileBusinessStore (BILLING_DB)`. Process env has `PORT=3040`, `ZAPPIER_UPSTREAM=http://127.0.0.1:3130`; `BUSINESS_STORE` not explicitly set (defaults to sqlite-file). |
|
||
| 2 | `POST /v1/timestamps` digests-only `key-ada` → 202; re-POST same hash → existing/free | **PASS** | Digest `sha256("qa-retest-wpl-2026-10-03")` = `5bdeb622…eacbd`. POST1 → **HTTP 202**, `created:1`, `existing:0`, `X-Verae-Credits-Spend: 10`. POST2 → **HTTP 202**, `created:0`, `existing:1`, `totalCredits:0`, `X-Verae-Credits-Spend: 0`. Channel `api`. |
|
||
| 3 | usage chain `verifyChain` ok on `billing.db` | **PASS** | Readonly `SqliteUsageRepo.verifyChain()` on `/opt/verae/zappier-edge/billing.db` → `{"ok":true,"length":6,"tipHash":"8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5"}`. WAL. Ada (`cust_1`) `balance_cents=0`; usage rollup `cust_1` n=6 sum=60 credits. |
|
||
| 4 | `DUAL-WRITER-FREEZE.md` present | **FAIL** | `ls` → no such file under `/opt/verae/zappier-edge/` or `docs/`. Freeze **policy text exists** inside [`docs/ENV-MATRIX.md`](file:///opt/verae/zappier-edge/docs/ENV-MATRIX.md) (“Dual-writer freeze (2026-10-03): Meter and balance writes go only through zappier-edge → BusinessStore…”). Checklist asked for the named file — **not present**. |
|
||
| 5 | health matrix `:3040/:3130/:3010/:3100` | **PASS** | See table below. Access planes `:3021–3025` also **200**. |
|
||
| 6 | Optional: `api.veraetime.net` login + `POST /api/batch/timestamp` two digests | **PASS** | Login `admin`/`admin123` → JWT. Batch digests-only two items → **HTTP 202** `{"jobIds":["fa1838fc-…","930de378-…"]}`. Confirms upstream still digests-only / batch path. |
|
||
|
||
### Health matrix (this probe)
|
||
|
||
| Port | Role | `/health` | Notes |
|
||
|------|------|-----------|-------|
|
||
| `:3040` | zappier-edge | **200** `{"ok":true,"role":"zappier-edge"}` | BusinessStore sqlite-file |
|
||
| `:3130` | middleware product | **200** `mockVerae:false`, `natsConnected:true` | Edge upstream |
|
||
| `:3010` | account-balance | **200** `nats:false` | Statement empty; dual path |
|
||
| `:3100` | middleware mock | **200** `mockVerae:true`, `natsConnected:false` | Do not smoke here |
|
||
|
||
### Go DB (C9)
|
||
|
||
- `/var/lib/timestamping-svc/data/blockchain.db` — mtime **2026-09-25**, size ~31 MB. **Not modified** by this QA (stat/ls only).
|
||
|
||
---
|
||
|
||
## Dual-writer / statement vs meter (still broken)
|
||
|
||
| Store | Observation |
|
||
|-------|-------------|
|
||
| Edge `billing.db` | `usage_entries` length **6**, chain tip ok; period spend headers **60** after probes |
|
||
| `:3010/statement/cust_1` | **200** `{prepaidCents:0, usage:[], credits:[], payments:[]}` — **does not see** edge usage |
|
||
| C16 | **Still live:** meter writer = edge BusinessStore/sqlite-file; facade books path still empty / out of band |
|
||
|
||
Sqlite-file deploy moved edge metering behind `BusinessStore`, but **did not** eliminate the second empty ledger behind account-balance.
|
||
|
||
---
|
||
|
||
## Userflow readiness 1–7 vs baseline
|
||
|
||
| # | Userflow | Baseline | This retest | Remaining blocker |
|
||
|---|----------|----------|-------------|-------------------|
|
||
| 1 | Signup → API key → Statement | Partial; statement empty vs edge | **Unchanged / still blocked** | Seed keys work; `:3010` statement still `usage:[]` while edge has 6 rows; no portal E2E harness; dual SoT |
|
||
| 2 | Stamp digests single + batch + credit headers | Mostly green | **Improved evidence / still green** | sqlite-file meter + verifyChain ok; still local SQLite (not libSQL); zero balance still meters; not multi-host |
|
||
| 3 | Async complete → Zapier REST Hook | Not proven | **Still broken / unproven** | Single `POST /v1/timestamp` receipt `completedAt:"0001-01-01T00:00:00Z"` (jobId `6e925fbc-…`); no watch→events→webhook E2E |
|
||
| 4 | CS credit adjust → portal balance | Broken / dual path | **Still broken** | Adjust would hit `:3010`/books while meter hits BusinessStore/`billing.db`; `nats:false` |
|
||
| 5 | Sales quote → meter match | Unproven | **Still unproven** | Rates still edge SQLite; no shared SoT / harness |
|
||
| 6 | Accounting export consistency | Fails if run | **Still fails** | Export cannot reconcile empty statement/books vs `usage_entries` n=6 |
|
||
| 7 | Leaf stamp OK / billing deny | Health + 404 only | **Unchanged** | `:3023/health` 200; `POST :3023/v1/billing/adjust` → **404** `not found` (≠ D004 automated deny) |
|
||
|
||
**PM QA entry (MG3):** still **not met**. Progress vs baseline: BusinessStore + SqliteFile **on live edge**, chain verify green, ENV-MATRIX published, digests-only + upstream batch reconfirmed. Missing for MG3: libSQL cutover, single writer proof (statement == meter), account-balance fold/rewire, harness, D004 tests, dedicated freeze doc filename if still required by ops.
|
||
|
||
---
|
||
|
||
## a) What to change
|
||
|
||
| # | Component | Problem | Evidence | Severity | Type |
|
||
|---|-----------|---------|----------|----------|------|
|
||
| 1 | Dual prepaid/ledger path (C16) | Edge BusinessStore meters `billing.db`; `:3010` statement still empty separate store | statement `usage:[]` vs verifyChain length 6; both services up | **critical** | design flaw / dual writer |
|
||
| 2 | Missing libSQL SoT (D006) | Lab still sqlite-file only; no `LIBSQL_URL` in edge environ | environ grep; journal `sqlite-file` | **critical** | missing foundation |
|
||
| 3 | `DUAL-WRITER-FREEZE.md` filename | Checklist/ops expect named freeze doc; only ENV-MATRIX paragraph exists | `ls` missing file; ENV-MATRIX has freeze text | **medium** | doc drift / incomplete deliverable |
|
||
| 4 | Account-balance NATS / facade | `nats:false`; not libSQL-backed; not single-writer | health JSON | **high** | ops / wiring |
|
||
| 5 | Async receipt / flow 3 | `completedAt` epoch-zero on single stamp | curl body `0001-01-01T00:00:00Z` | **high** | incomplete product path |
|
||
| 6 | Userflow harness (WP-P) | Flows 1–7 not automated | No harness report artifact beyond this manual retest | **high** | missing gate |
|
||
| 7 | Zero-balance metering | Ada `balance_cents=0` still stamps | customers row + 202 + Balance header 0 | **medium** | missing prepaid gate |
|
||
| 8 | D004 leaf deny | 404 ≠ policy deny | curl 404 on invent-ed path | **medium** | missing test |
|
||
| 9 | `BUSINESS_STORE` not explicit in secrets | Relies on code default; harder to audit live | environ lacks `BUSINESS_STORE=` | **low** | ops hygiene |
|
||
|
||
### What is working (preserve)
|
||
|
||
- Live `BusinessStore driver: sqlite-file` + SqliteFileBusinessStore over `BILLING_DB`.
|
||
- Digests-only batch `:3040` → `:3130` with `key-ada`, credit headers, idempotent existing/free.
|
||
- `verifyChain` ok after meter appends (hash chain intact).
|
||
- Product upstream `:3130` (not mock `:3100`); ENV-MATRIX documents the port roles + freeze policy.
|
||
- Upstream Verae Time digests-only batch authenticated path.
|
||
- Access planes healthy; Go `blockchain.db` untouched; no JetStream used for job-complete in this probe.
|
||
|
||
---
|
||
|
||
## b) What to change it to
|
||
|
||
| # | Component | Target behavior | Acceptance criteria | Constraints |
|
||
|---|-----------|-----------------|---------------------|-------------|
|
||
| 1 | Single writer | One authoritative meter + adjust path; statement usage == `usage_entries` | After CS adjust + stamp, `:3010` or successor statement matches edge/libSQL tip; no empty books SoT | D001/D006; do not dual-append chain |
|
||
| 2 | libSQL BusinessStore | `BUSINESS_STORE=libsql` + `LIBSQL_URL`; fail closed 503 if writer down | Contract tests; migrator tip verify; live meter only to libSQL | Auth stays `zappier.db` (C13) |
|
||
| 3 | Freeze doc | Publish `docs/DUAL-WRITER-FREEZE.md` (or update checklist to ENV-MATRIX as SoT) | File present on live `/opt/verae/zappier-edge/docs/` **or** QA checklist amended | Do not invent conflicting writers |
|
||
| 4 | Billing facade | Fold into edge HTTP BusinessStore **or** libSQL-backed NATS listener | Health `nats:true` if kept; statement == ledger | Stable `verae.billing.*` subjects |
|
||
| 5 | Async complete | Real `completedAt`; hook delivery path | E2E jobId → terminal event (+ hook attempt) | `verae.ts.*` **core NATS only** |
|
||
| 6 | Harness 1–7 | Automated report under `plans/qa/` | Flows 1,2,4 MOCK then 3,5,6,7; exit 0 | — |
|
||
| 7 | Prepaid gate | Documented 402 vs allow-with-debt | Seed funded + zero-balance fixture tests | Do not rewrite historical ledger |
|
||
| 8 | D004 | Automated leaf deny | Tests green; live deny ≠ bare 404 | Leaf may still stamp |
|
||
| 9 | Explicit env | `BUSINESS_STORE=sqlite-file` (then `libsql`) in secrets | `systemctl`/environ shows flag | Deploy never rsyncs `*.db*` |
|
||
|
||
---
|
||
|
||
## c) How to implement
|
||
|
||
Aligned with PM board; **do not skip merge gates**.
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
DocFix[Publish DUAL-WRITER-FREEZE.md or amend checklist] --> MG2
|
||
EnvFlag[Explicit BUSINESS_STORE in secrets] --> MG2
|
||
MG2[MG2 writers unlock after migrator] --> WP_L_done[Confirm WP-L meter-only BusinessStore]
|
||
WP_L_done --> WP_M[WP-M fold/rewire account-balance]
|
||
WP_M --> WP_N[WP-N CS/sales/acct]
|
||
WP_N --> MG3[MG3 single writer proof]
|
||
MG3 --> Harness[WP-P harness 1-7]
|
||
MG3 --> Async[Fix completedAt / hook path]
|
||
Harness --> MG4[MG4]
|
||
```
|
||
|
||
### Phase A — Doc / ops closeout (hours)
|
||
|
||
- Add `/opt/verae/zappier-edge/docs/DUAL-WRITER-FREEZE.md` (content can mirror ENV-MATRIX freeze paragraph + owner + daily rule).
|
||
- Set `BUSINESS_STORE=sqlite-file` explicitly in `/SSD2/secrets/zap-verae/zappier-edge.env`; restart; confirm journal + environ.
|
||
|
||
### Phase B — MG2 → MG3 (days; blocked on libSQL migrator if not done)
|
||
|
||
- Migrator `billing.db` (+ books) → libSQL; refuse on `verifyChain` fail; archive tip under `plans/qa/`.
|
||
- Rewire/stop second writer; statement == meter proof.
|
||
|
||
### Phase C — Product paths + harness
|
||
|
||
- Fix single-stamp receipt `completedAt`; prove flow 3 under harness (MOCK first).
|
||
- Automate flows 1–7; D004 deny tests; prepaid policy fixture.
|
||
|
||
**Validation for this retest exit:** checklist items 1–3,5–6 green is enough to call **sqlite-file deploy accepted**; item 4 and dual-SoT remain open work, not deploy regressions.
|
||
|
||
---
|
||
|
||
## d) Expected results — and what remains incomplete
|
||
|
||
### After planned changes
|
||
|
||
| Metric / behavior | Today (this retest) | Expected after Phases A–C / MG3 |
|
||
|-------------------|---------------------|----------------------------------|
|
||
| BusinessStore on edge | sqlite-file live | libSQL primary |
|
||
| Flow 2 digests stamp | 202 + headers + chain ok | Same UX on libSQL |
|
||
| Statement vs meter | Divergent (`[]` vs n=6) | Identical |
|
||
| `DUAL-WRITER-FREEZE.md` | Missing file | Present (or checklist SoT = ENV-MATRIX) |
|
||
| Flow 3 `completedAt` | `0001-01-01…` | Real completion time |
|
||
| Flow 7 D004 | 404 only | Automated deny |
|
||
| `blockchain.db` | Untouched | Still untouched |
|
||
|
||
### Still missing even after those todos
|
||
|
||
| Gap | Why |
|
||
|-----|-----|
|
||
| Zapier Platform live push / workspace Zaps | Operator Zapier login; out of cutover buffer |
|
||
| NATS nkeys/mTLS | Lab isolation is network, not accounts |
|
||
| libSQL HA / multi-primary | Launch = one primary + backup |
|
||
| Peergos / Go timestamping redesign | Out of scope |
|
||
| Full arch sign-off leftovers (C2/C3/C6…) | Docs project parallel |
|
||
|
||
### Success tiers
|
||
|
||
| Tier | Status after this retest |
|
||
|------|--------------------------|
|
||
| Sqlite-file deploy smoke | **Met** (journal, 202/idempotent, verifyChain, health, upstream optional) |
|
||
| Dual-writer freeze artifact | **Partial** (policy in ENV-MATRIX; named file missing) |
|
||
| MG3 single-writer | **Not met** |
|
||
| WP-Q / double userflow green | **Not met** |
|
||
|
||
---
|
||
|
||
## e) Prioritized todo list
|
||
|
||
```
|
||
Item 1 — [P0] — Publish docs/DUAL-WRITER-FREEZE.md on live edge (or amend QA checklist)
|
||
Ship named freeze runbook to /opt/verae/zappier-edge/docs/ mirroring ENV-MATRIX authority rule, or formally designate ENV-MATRIX as the freeze SoT.
|
||
PreReqs: (none)
|
||
|
||
Item 2 — [P0] — Explicit BUSINESS_STORE=sqlite-file in zappier-edge.env
|
||
Add flag to secrets; restart; confirm environ + journal; keep BILLING_DB path clear.
|
||
PreReqs: (none)
|
||
|
||
Item 3 — [P0] — Complete migrator + libSQL cutover with verifyChain gate (WP-K / MG2)
|
||
Refuse cutover if chain fails; archive tip report under plans/qa/.
|
||
PreReqs: (Item 2)
|
||
|
||
Item 4 — [P0] — Fold or rewire account-balance to BusinessStore/libSQL (WP-M)
|
||
Eliminate empty statement SoT; nats connected if facade kept; statement usage == ledger.
|
||
PreReqs: (Item 3)
|
||
|
||
Item 5 — [P0] — CS/sales/accounting-export same store + MG3 proof (WP-N)
|
||
Adjust visible on statement and edge balance headers.
|
||
PreReqs: (Item 4)
|
||
|
||
Item 6 — [P1] — Fix single-stamp completedAt / async receipt wiring (flow 3)
|
||
Replace 0001-01-01 placeholders on product path used by Zapier wait/hook.
|
||
PreReqs: (none)
|
||
|
||
Item 7 — [P1] — Automate D004 leaf billing deny (WP-D)
|
||
Replace 404-only smoke with authz allow-list tests.
|
||
PreReqs: (none)
|
||
|
||
Item 8 — [P1] — Userflow harness skeleton flows 1, 2, 4 then 3, 5, 6, 7 (WP-P)
|
||
MOCK_VERAE report artifact under plans/qa/; MG4.
|
||
PreReqs: (Item 5 + Item 7)
|
||
|
||
Item 9 — [P2] — Prepaid zero-balance policy gate
|
||
Define 402 vs allow-with-debt; seed funded fixture.
|
||
PreReqs: (Item 4)
|
||
|
||
Item 10 — [P2] — Multi-host LIBSQL_URL topology (WP-O)
|
||
Only after MG3.
|
||
PreReqs: (Item 5)
|
||
```
|
||
|
||
**Dependency order (Do all):** Item 1 ∥ Item 2 ∥ Item 6 ∥ Item 7 → Item 3 → Item 4 → Item 5 → Item 8 → Item 9 → Item 10.
|
||
|
||
---
|
||
|
||
## Probe artifacts (raw)
|
||
|
||
- Batch digest: `5bdeb622aae9311ae2e9a5f6cea85ad615711f119a598c1ad212b6b03a4eacbd` (`qa-retest-wpl-2026-10-03`)
|
||
- Batch jobId: `d012fb8f-ed92-448d-b9d2-afe27776c11a`
|
||
- Single digest: `2a36132dd089f1231764fb37eac38999cf1531e008c28a797ff95c94f6703525` (`qa-single-wpl`)
|
||
- Single jobId: `6e925fbc-9b89-4704-8f7f-5f8793ce5b86`
|
||
- Chain tip after probe: `8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5` (length 6)
|
||
- Upstream jobIds: `fa1838fc-b6b5-40c6-a05f-93439f2c0c0c`, `930de378-994b-429b-a713-9fa27af0965f`
|
||
|
||
**QA note:** Sqlite-file BusinessStore deploy is **accepted** for lab meter path. Dual-writer / MG3 / harness remain the critical path. Formal WP-Q churn still waits on MG3 + harness skeleton.
|