WIP snapshot before drive cleanup 2026-10-03

This commit is contained in:
George Lambert 2026-10-03 23:02:02 -04:00
commit e666e7bf3c
5 changed files with 2828 additions and 1 deletions

View file

@ -0,0 +1,344 @@
# QA Baseline — Verae × Zapier Userflows 1–7 vs D006 Multi-Host
**Role:** QA (evaluate only; no product feature implementation)
**Date:** 2026-10-03 (UTC probe ~06:04–06:07)
**Host:** `NS1.GEORGELAMBERT.ORG` / `70.88.205.138`
**Inputs:**
- Plan: [`../2026-10-03-libsql-multihost-architecture.md`](../2026-10-03-libsql-multihost-architecture.md)
- Arch review: [`../reviews/2026-10-03-arch-review.md`](../reviews/2026-10-03-arch-review.md) (approve-with-changes; A1–A8 merged; full sign-off still open)
- PM board: [`../reviews/2026-10-03-pm-board.md`](../reviews/2026-10-03-pm-board.md)
- Decision: D006 (libSQL business SoT; JetStream = work only; core NATS for `verae.ts.*`)
**Scope:** Readiness of critical userflows **1–7** for multi-host cutover. This is a **baseline** (WP-Q precursor). Per PM §5, formal WP-Q entry requires MG3; this pack documents why that gate is not met and what must change first.
---
## Summary judgment
Lab doors and digests-only stamp submit on `:3040` → `:3130` work today, but **D006 multi-host readiness fails**: there is **no libSQL**, **no BusinessStore**, and **two live prepaid/ledger writers** (`billing.db` on edge + empty `books.json` behind `:3010`). Userflows 1, 3–6 are not E2E-proven; flow 2 is partially green; flow 7 has only weak 404 evidence for billing deny. Do **not** start Phase 4 host split or treat WP-Q as “in” until single-writer libSQL lands.
---
## Live lab evidence (2026-10-03)
### Health matrix
| Port | Role | `/health` | Notes |
|------|------|-----------|-------|
| `:3040` | zappier-edge | **200** `{"ok":true,"role":"zappier-edge"}` | Bind `127.0.0.1`; systemd `zappier-edge.service` |
| `:3100` | middleware (lab/mock) | **200** `mockVerae:true`, `natsConnected:false` | Catalog/default; **not** product upstream |
| `:3130` | middleware (product) | **200** `mockVerae:false`, `natsConnected:true` | Edge `ZAPPIER_UPSTREAM` target |
| `:3010` | zappier-account-balance | **200** `nats:false` | Loopback; **not** a systemd unit; fleet-spawned |
| `:3021`–`:3025` | access planes | **200** each | web / api / leaf / zapier / staff |
### Env / upstream
- Edge secrets (`/SSD2/secrets/zap-verae/zappier-edge.env`): `PORT=3040`, **`ZAPPIER_UPSTREAM=http://127.0.0.1:3130`** (confirms arch C11 / plan A1).
- Product MW env: `PORT=3130`, `HOST=127.0.0.1`, `MOCK_VERAE=false`, `NATS_URL=nats://127.0.0.1:4222`.
- Account-balance (pid 4179): cwd under moved stack package; `FLEET_STATE_DIR=.../verae-fleet/data/runtime/account-balance-0`; `NATS_URL` lists `10.10.10.21–23` but health reports **`nats:false`**.
### Digest-only stamp (flow 2 probe)
- Digest: `sha256("qa-baseline")` = `edf8f682e42104442bb4637c665f50ada9873fa2fe6958ae00a6c05b2e5181cb`
- `POST /v1/timestamps` with `x-api-key: key-ada`, `x-billing-channel: api`, body `{sha256s:[…]}` → **HTTP 202**, `created:1` then idempotent `existing:1`, credit headers present (`X-Verae-Credits-*`).
- Items-shaped digest-only body also **202**.
- Single `POST /v1/timestamp` → **202** with `jobId`, but receipt `completedAt: "0001-01-01T00:00:00Z"` (placeholder / incomplete receipt shape for async complete).
### Dual-writer / C16 (flows 1, 4–6)
| Store | Path | Observation |
|-------|------|-------------|
| Edge ledger | `/opt/verae/zappier-edge/billing.db` | **Exists** (WAL). `usage_entries` count **3** for `cust_1` @ 10 each after probes; `customers.balance_cents` for Ada = **0** |
| Account-balance books | `FLEET_STATE_DIR/.../data/books.json` | **Exists**, empty: `prepaid:{}`, `usage:[]`, `credits:[]`, `payments:[]` |
| Statement HTTP | `GET :3010/statement/cust_1` | **200** `{prepaidCents:0, usage:[], …}` — **does not see** edge `usage_entries` |
| Edge response | batch stamp | `balanceCredits:0`, period spend headers show **30** after three 10-credit meters |
**C16 confirmed live:** edge writes metered usage into `billing.db`; account-balance serves a separate empty `books.json`. No shared SoT. Splitting hosts now would fork money.
### Go timestamping DB (C9)
- `/var/lib/timestamping-svc/data/blockchain.db` **exists** (~31 MB, mtime 2026-09-25).
- **Not modified** by this QA probe (read `ls`/`stat` only).
### libSQL / BusinessStore / harness
- No `LIBSQL_URL` / `sqld` / libSQL container observed on 138 for business data.
- Local clone `/Users/marchon/apps/verae/zappier-edge/src/db/`: **no** `business-store.ts` (`NO_BUSINESS_STORE`).
- `verae-zapier-simulator` exists (messaging/trace) but is **not** the WP-P automated harness for flows 1–7.
- Arch checklist still open: `nats-subjects.md` `verae.ts.*` / `ZAPIER_USAGE` non-SoT; published env matrix; GETTING-STARTED SoT language.
- Host `nats stream ls` to `10.10.10.21:4222` **timed out** from this probe (MW still reports `natsConnected:true` via loopback `:4222` proxy). STATUS still lists product stream **`ZAPIER_USAGE`** (C10 risk).
### Leaf D004 (flow 7 probe)
- `:3023/health` **200** leaf plane up.
- `POST :3023/v1/billing/adjust` → **404** `not found` (route absent). This is **not** yet an automated policy deny of `verae.billing.balance.adjust` (PM WP-D DoD).
---
## Userflow readiness (1–7)
| # | Userflow | Today | Blocker vs D006 |
|---|----------|-------|-----------------|
| 1 | Signup → API key → Statement | Partial: seed keys exist (`key-ada`…); statement via `:3010` empty/out of band with edge ledger | Dual SoT; no portal E2E harness; account-balance `nats:false` |
| 2 | Stamp digests single + batch + credit headers | **Mostly green** on lab (`:3040`→`:3130`, 202, headers, idempotent existing) | Still local SQLite meter; zero balance still meters; receipt placeholder on single path; not libSQL |
| 3 | Async complete → Zapier REST Hook | **Not proven** | `completedAt` zero-time on single stamp; no WP-P watch→events→webhooks.deliver E2E; Platform push still open in STATUS |
| 4 | CS credit adjust → portal balance | **Broken / dual path** | Adjust would hit books/`verae.billing.*` while meter hits `billing.db`; nats false on balance service |
| 5 | Sales quote → meter match | **Unproven** | Rate tables live in edge SQLite; sales package not wired to shared SoT |
| 6 | Accounting export consistency | **Fails if run today** | Export cannot reconcile empty books vs non-empty `usage_entries` |
| 7 | Leaf stamp OK / billing deny | Health only | Need authz allow-list tests; 404 ≠ D004 proof |
**PM QA entry (board §5):** MG3 single writer, regression green, NATS rules verified, harness skeleton, D004 automated, migrator tip report, Go DB untouched note, docs C2/C3/C5/C6 — **not met** (only Go-untouched + partial digests-only + doors up).
---
## a) What to change
| # | Component | Problem | Evidence | Severity | Type |
|---|-----------|---------|----------|----------|------|
| 1 | Prepaid / ledger SoT (C7/C16) | Two writers: edge `billing.db` meters usage; `:3010` serves empty `books.json` statement | usage_n=3 vs statement `usage:[]`; both processes up | **critical** | design flaw / dual writer |
| 2 | Missing libSQL + BusinessStore (D006 Phases 1–2) | No shared business DB or interface; multi-host impossible without per-host SQLite fork | No `business-store.ts`; no LIBSQL on 138 | **critical** | missing foundation |
| 3 | Account-balance NATS / facade | Health `nats:false` while CS/portal expect `verae.billing.*` | `:3010/health` JSON; env has NATS_URL but not connected | **high** | ops / wiring |
| 4 | Userflow E2E harness (WP-P) | Flows 1–7 not automated; simulator ≠ harness | No `plans/qa` flow reports; simulator README is messaging-only | **high** | missing gate |
| 5 | Async complete / receipts (flow 3) | Single-stamp receipt shows epoch-zero `completedAt`; REST Hook path unproven | curl body `0001-01-01T00:00:00Z` | **high** | incomplete product path |
| 6 | Zero-balance metering (flows 2/4) | Ada `balance_cents=0` still stamps and accumulates period spend | customers row + 202 responses + `X-Verae-Credits-Balance: 0` | **medium** | missing gate / prepaid policy |
| 7 | Dual middleware confusion (C11) | `:3100` mock vs `:3130` product; docs/scripts can point wrong | health JSON both ports; edge env → 3130 | **medium** | doc/ops drift |
| 8 | D004 leaf deny proof (flow 7) | Only 404 on invent-ed HTTP path; no automated NATS/HTTP deny | curl 404; WP-D tests not evidenced on lab | **medium** | missing test |
| 9 | `ZAPIER_USAGE` / nats-subjects docs (C5/C10) | STATUS still lists usage stream as product; catalog subjects omit core-only `verae.ts.*` | STATUS.md streams table; arch checklist open | **medium** | doc drift / SoT confusion |
| 10 | Env matrix / Phase 0 leftovers | Arch full sign-off blocked; GETTING-STARTED may still call books SoT | arch review checklist unchecked items | **medium** | doc freeze incomplete |
| 11 | Account-balance not systemd-managed | Process from fleet overlay / moved path; restart/ownership unclear vs edge unit | no `*account-balance*.service`; pid under `marchon` | **low** | ops hygiene |
### What is working (preserve)
- Digests-only batch + single stamp on product edge `:3040` with `key-ada` and credit headers.
- Product upstream correctly set to **`:3130`** (not mock `:3100`).
- Access-plane health on `:3021–3025`.
- Edge split DBs present and WAL: `billing.db` (business tables) + `zappier.db` (auth).
- Idempotent re-stamp (`created:0`, `existing:1`, spend 0).
- Standing rules respected in this probe: **no** JetStream for `verae.ts.*`; **no** writes to `blockchain.db`.
- Plan + A1–A8 + D001 addendum + D006 direction are the right target.
---
## b) What to change it to
| # | Component | Target behavior | Acceptance criteria | Constraints |
|---|-----------|-----------------|---------------------|-------------|
| 1 | Single prepaid/ledger writer | One authoritative adjust + `recordUsage` path on shared store; other path read-only or stopped | Grep: no prod writes to `books.json` / per-host `billing.db` as SoT; adjust on CS visible on portal statement and edge balance headers | D001 single writer; do not dual-append hash chain |
| 2 | libSQL BusinessStore | `LibsqlBusinessStore` (+ SqliteFile for CI) behind `BusinessStore`; lab migrator from `billing.db` + `books.json` | Contract tests green; migrator tip hash verify; `BUSINESS_STORE=libsql` writes only to libSQL | Auth stays edge-local `zappier.db` (C13); fail closed (503) if writer down |
| 3 | Billing facade | `:3010` (or successor) is libSQL-backed listener for `verae.billing.*` **or** folded into edge HTTP API — pick one (plan Phase 3) | Health `nats:true` (if NATS facade kept); statement usage matches `usage_entries` | Keep subjects stable for CS/sales |
| 4 | Harness 1–7 | Automated runner under simulator/harness producing `plans/qa/` report | Flows 1,2,4 runnable under MOCK; then 3,5,6,7; exit 0 artifact | Zapier cloud push optional for first MOCK pass |
| 5 | Async complete | Real completion times / receipts; `jobs.watch` → `jobs.events` → `webhooks.deliver` | E2E: jobId reaches terminal event + hook delivery (or recorded attempt) | `verae.ts.completed` / `verae.ts.receipt` remain **core NATS only** |
| 6 | Prepaid gate | Metering respects prepaid/credits policy (402 or configured allow) | Seed funded customer; zero-balance path returns expected product code | Do not rewrite historical ledger mid-test |
| 7 | Env matrix | Documented `:3040` / `:3130` product / `:3100` mock / `:3010` facade | STATUS + ops GETTING-STARTED + plan cite same matrix | Do not “fix” smoke by pointing edge at 3100 |
| 8 | D004 | Leaf cannot call balance adjust (HTTP door + NATS authz) | `verae-access-leaf` / authz tests green; live deny ≠ bare 404 | Leaf may still stamp/authz |
| 9 | NATS docs | `nats-subjects.md` Non-streams: `verae.ts.*` core-only; `ZAPIER_USAGE` telemetry non-SoT | Arch checklist boxes checked; STATUS wording demotes usage stream | No JetStream bind for job-complete |
| 10 | Phase 0 close | C2/C3/C5/C6 closed or owned; GETTING-STARTED SoT → libSQL | Arch full sign-off or explicit waivers | — |
| 11 | Balance process ops | systemd (or fleet-documented) unit with clear state dir | `systemctl status` shows balance facade; books path documented until migration | Deploy never rsyncs `*.db*` |
---
## c) How to implement
Aligned with PM DAG; **do not skip merge gates**.
```mermaid
flowchart TD
P0[Phase 0 docs/env/NATS freeze] --> P1[Phase 1 BusinessStore TDD]
P1 --> P2[Phase 2 migrator + dual-read]
P2 --> MG2[MG2 writers unlock]
MG2 --> P3[Phase 3 single writer rewire]
P3 --> MG3[MG3 single writer proof]
MG3 --> P4[Phase 4 multi-host LIBSQL_URL]
MG3 --> P5[Phase 5 harness 1-7]
P4 --> P5
P5 --> MG4[MG4 green]
MG4 --> P6[Phase 6 QA whattochange loop]
```
### Phase 0 — Conflict / docs freeze (feeding; ~1–2 days)
- **Goal:** Close open arch checklist items that confuse operators and agents.
- **Changes:** Publish env matrix; patch `nats-subjects.md`; demote `ZAPIER_USAGE` in STATUS; fix overview `zmw_` / digests examples (C3/C6); GETTING-STARTED SoT → libSQL.
- **Validation:** Arch checklist; `rg` for stale SoT claims.
- **Effort:** 1–2 agent-days (WP-A/B/C).
### Phase 1 — BusinessStore + libSQL skeleton (critical; WP-F/G/H)
- **Goal:** Interface + SqliteFile + Libsql drivers with contract tests (including hash-chain).
- **Changes:** `zappier-edge` `src/db/business-store.ts`, contract tests, CI libSQL.
- **Validation:** Jest contract green both drivers (MG1).
- **Effort:** ~3 days serial.
### Phase 2 — Migrate + dual-read (WP-J/K)
- **Goal:** Import live `billing.db` + empty/legacy `books.json`; verify chain tip; reads dual-run; **writes still locked**.
- **Validation:** Migrator report archived under `plans/qa/`; MG2.
- **Effort:** ~2 days.
### Phase 3 — Rewire writers (WP-L/M/N) — only after MG2
- **Goal:** Edge meter + account-balance + CS/sales/acct → one BusinessStore; remove second writer.
- **Validation:** Statement usage == ledger; CS adjust visible on portal/edge headers; MG3 grep proof.
- **Effort:** ~3–4 days.
### Phase 4 — Multi-host (WP-O) — only after MG3
- **Goal:** `LIBSQL_URL` primary; edge/MW/CS on separate hosts possible.
- **Validation:** Cross-host adjust/meter/statement same balance.
### Phase 5 — Harness (WP-P)
- **Goal:** Automate flows 1–7; first pass MOCK_VERAE OK.
- **Validation:** Report under `plans/qa/`; MG4.
### Phase 6 — QA loop (WP-Q)
- **Goal:** Break → whattochange → fix → retest until **two consecutive** full greens (project buffer).
**Explicit non-goals until gates pass:** JetStream for `verae.ts.*`; editing `blockchain.db`; Litestream multi-writer; multi-host split before MG3.
---
## d) Expected results / still incomplete
### After implementation (realistic)
| Metric / behavior | Today (baseline) | Expected after Phases 0–5 |
|-------------------|------------------|---------------------------|
| Business SoT | Split `billing.db` + empty `books.json` | One libSQL `verae_business` |
| Flow 2 digests stamp | 202 + headers on single host SQLite | Same UX on libSQL meter writer |
| Flow 1/4 statement vs meter | Divergent (usage 3 vs `[]`) | Identical customer/usage/balance |
| Flow 3 async → hook | Unproven; zero `completedAt` | Proven under harness (MOCK then optional live) |
| Flow 5/6 quote & export | Unproven / inconsistent | Same rate tables + export = ledger |
| Flow 7 D004 | 404 only | Automated deny |
| Multi-host money | Unsafe | Safe only post-MG3 |
| `blockchain.db` | Untouched | Still untouched |
### Still missing, incorrect, or incomplete (even after this todo list)
| Gap | Why it persists |
|-----|-----------------|
| Zapier Platform public `push` + live workspace Zaps | Needs operator Zapier login (STATUS); out of first cutover buffer |
| Live `api.veraetime.net` seal proof beyond lab MW | Creds / OpenAPI gaps (hash lookup 501); separate from libSQL SoT |
| NATS nkeys/mTLS | Cluster isolation today is `vmbr1`, not accounts |
| Portal auth on multi-edge sticky sessions | C13: auth stays local `zappier.db` for v1 |
| libSQL HA / failover RPO | Launch = one primary + backup; not multi-primary |
| Peergos / Go timestamping redesign | Out of scope |
| Full architecture sign-off | Depends on remaining checklist (docs), not only code |
### Success tiers
| Tier | Criteria |
|------|----------|
| **Minimum viable (this baseline exit)** | Dual-writer named daily owner; env matrix published; harness skeleton for 1/2/4; no host split |
| **MG3 / single-writer** | libSQL SoT; one adjust path; statement == meter |
| **Production-ready multi-host** | MG3 + WP-O + flows 1–7 twice green; fail-closed meter; D004 automated |
| **Complete** | + Platform push + live Verae seal + nkeys — separate projects |
---
## e) Prioritized todo list with PreReqs
```
Item 1 — [P0] — Freeze dual-writer authority for lab (C16 runbook)
Document which process may adjust/meter each day until MG3; forbid CS adjust against books while edge meters billing.db without a sync plan.
PreReqs: (none)
Item 2 — [P0] — Publish live env matrix (:3040 / :3130 / :3100 / :3010)
Add to STATUS + ops docs; match edge ZAPPIER_UPSTREAM=http://127.0.0.1:3130 evidence.
PreReqs: (none)
Item 3 — [P0] — BusinessStore interface + failing contract tests (WP-F)
Create business-store.ts + contract suite (customers, usage chain, rates, invoices, balances).
PreReqs: (Item 2)
Item 4 — [P0] — SqliteFileBusinessStore adapter (WP-G)
Wrap existing edge repos; keep Jest green.
PreReqs: (Item 3)
Item 5 — [P0] — LibsqlBusinessStore + CI libSQL (WP-H)
Driver + reachable LIBSQL_URL in CI/lab; MG1.
PreReqs: (Item 4)
Item 6 — [P0] — Migrator billing.db + books.json → libSQL with chain verify (WP-K)
Refuse cutover on verifyChain failure; archive tip report under plans/qa/; MG2.
PreReqs: (Item 5)
Item 7 — [P0] — Rewire edge meter to BusinessStore only (WP-L)
Remove production writes that treat local billing.db as SoT once flag on.
PreReqs: (Item 6)
Item 8 — [P0] — Fold or rewire account-balance to libSQL facade (WP-M)
One verae.billing.* listener; empty books.json import-only; nats connected if facade kept.
PreReqs: (Item 7)
Item 9 — [P1] — Rewire CS / sales / accounting-export to same store (WP-N)
Credit adjust, rates, export read libSQL; MG3 proof.
PreReqs: (Item 8)
Item 10 — [P1] — Patch nats-subjects + demote ZAPIER_USAGE (C5/C10)
Document verae.ts.* core-only; usage stream telemetry-only in STATUS/catalog.
PreReqs: (none)
Item 11 — [P1] — Automate D004 leaf billing deny (WP-D)
Tests prove leaf cannot balance.adjust; replace 404-only smoke.
PreReqs: (none)
Item 12 — [P1] — Userflow harness skeleton flows 1, 2, 4 (WP-P start)
MOCK_VERAE runnable report artifact under plans/qa/.
PreReqs: (Item 9)
Item 13 — [P1] — Extend harness to flows 3, 5, 6, 7
Async hook path, quote match, export consistency, leaf deny E2E; MG4.
PreReqs: (Item 12 + Item 11)
Item 14 — [P2] — Fix async receipt completedAt / completion wiring
Eliminate 0001-01-01 placeholders on product stamp path used by Zapier wait/hook.
PreReqs: (Item 12)
Item 15 — [P2] — Prepaid zero-balance policy gate
Define and test 402 vs allow-with-debt; seed funded fixture for harness.
PreReqs: (Item 7)
Item 16 — [P2] — Multi-host LIBSQL_URL topology (WP-O)
Only after MG3; cross-host meter/adjust/statement.
PreReqs: (Item 9)
Item 17 — [P2] — systemd/fleet hygiene for account-balance facade
Unit or documented fleet runtime; stable state dir until books retired.
PreReqs: (Item 8)
Item 18 — [P3] — Phase 0 leftover docs (C2/C3/C6, GETTING-STARTED)
Close arch full sign-off checklist items not covered by Item 2/10.
PreReqs: (Item 2 + Item 10)
```
**Dependency order (for “Do all”):**
Item 1 ∥ Item 2 ∥ Item 10 ∥ Item 11 → Item 3 → Item 4 → Item 5 → Item 6 → Item 7 → Item 8 → Item 9 → Item 12 → (Item 13 ∥ Item 14 ∥ Item 15) → Item 16 → Item 17; Item 18 can trail after Item 2+10.
**QA note:** Formal WP-Q churn should wait for **Item 9 (MG3)** + **Item 12**. This baseline pack is the readiness report that unlocks engineering focus on Items 1–9.
---
## Probe command index (reproducible)
```bash
ssh -i ~/.ssh/id_ed25519 -o IdentitiesOnly=yes marchon@70.88.205.138 \
'for p in 3040 3100 3130 3010 3021 3022 3023 3024 3025; do
curl -sS -m 5 http://127.0.0.1:$p/health; echo; done'
# Digest-only batch (do not use production customer keys outside lab seeds)
DIGEST=$(printf "qa-baseline" | sha256sum | awk "{print \$1}")
curl -sS -D - -X POST http://127.0.0.1:3040/v1/timestamps \
-H "content-type: application/json" -H "x-api-key: key-ada" \
-H "x-billing-channel: api" \
-d "{\"sha256s\":[\"$DIGEST\"]}"
# Dual-writer check (read-only)
sqlite3 /opt/verae/zappier-edge/billing.db "SELECT COUNT(*) FROM usage_entries;"
curl -sS http://127.0.0.1:3010/statement/cust_1
# blockchain.db: ls/stat only — never migrate/edit
```
---
*End of QA baseline. No product code was changed by this review.*

View file

@ -0,0 +1,89 @@
# libSQL migrator dry-run (WP-K) — PASS
**Date:** 2026-10-03T06:24–06:26Z
**Host:** NS1 `70.88.205.138` (`NS1.GEORGELAMBERT.ORG`)
**Role:** Engineer (safe dry-run only)
**Result:** **PASS**
## Constraints honored
- Did **not** modify `/var/lib/timestamping-svc/data/blockchain.db` or any Go timestamping DB (stat-only; mtime 2026-09-25).
- Did **not** set `BUSINESS_STORE=libsql` (or any `LIBSQL_*`) on `zappier-edge.service`.
- Did **not** delete/truncate live `/opt/verae/zappier-edge/billing.db`.
- All migrator I/O under `/tmp/libsql-dryrun-20261003T062452Z/` copies only.
## Workspace
| Item | Path |
|------|------|
| Dry-run dir | `/tmp/libsql-dryrun-20261003T062452Z/` |
| Source copy | `…/source-billing.db` (90112 bytes) |
| Dest libSQL file | `…/dest.db` (`file:/tmp/libsql-dryrun-20261003T062452Z/dest.db`, 94208 bytes) |
| Migrator | `/opt/verae/zappier-edge/dist/db/migrate-to-libsql.js` (same content/mtime as `/tmp/zappier-edge-work`) |
| Runner | `…/run-migrate.mjs` (uses opt `node_modules`; chdir to `/opt/verae/zappier-edge`) |
| JSON report | `…/migrate-report.json` |
`/tmp/zappier-edge-work` had no `node_modules`; used live opt install for `@libsql/client` + `better-sqlite3`. Migrator source identical in both trees.
## Commands
```bash
# 1) Checkpoint + consistent copy (as verae-zapier → /tmp dry-run dir)
sudo -u verae-zapier sqlite3 /opt/verae/zappier-edge/billing.db "PRAGMA wal_checkpoint(TRUNCATE);"
# → 0|0|0
sudo chmod 777 /tmp/libsql-dryrun-20261003T062452Z
sudo -u verae-zapier python3 -c "import sqlite3; s=sqlite3.connect(\"/opt/verae/zappier-edge/billing.db\"); d=sqlite3.connect(\"/tmp/libsql-dryrun-20261003T062452Z/source-billing.db\"); s.backup(d); d.close(); s.close()"
sudo chown marchon:marchon /tmp/libsql-dryrun-20261003T062452Z/source-billing.db
# 2) Run migrator (node ESM runner → migrateBillingDbToLibsql)
cd /opt/verae/zappier-edge
node /tmp/libsql-dryrun-20261003T062452Z/run-migrate.mjs
```
## Row counts
| Table | Live billing.db | Source copy | Dest libSQL | Match |
|-------|-----------------|-------------|-------------|-------|
| customers | 4 | 4 | 4 | yes |
| usage_entries | 6 | 6 | 6 | yes |
| channel_rates | 108 | 108 | 108 | yes |
| channel_rate_meta | 1 | 1 | 1 | yes |
| tiers | 3 | 3 | 3 | yes |
| price_endpoints | 11 | 11 | 11 | yes |
| invoices | 0 | 0 | 0 | yes |
| invoice_lines | 0 | 0 | 0 | yes |
Customers on dest: `cust-platform`, `cust_1`, `cust_2`, `cust_3`.
## Chain / tip hashes
| Phase | ok | length | tipHash |
|-------|----|--------|---------|
| before (source copy via `verifyUsageLedgerChain`) | true | 6 | `8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5` |
| after (dest via `LibsqlBusinessStore.verifyUsageChain`) | true | 6 | `8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5` |
| live billing.db (post-run read) | — | 6 | `8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5` |
Genesis `prev_hash` = 64 zeros; tip id = 6. Before tip === after tip; lengths match.
## Dual-run / safety post-checks
| Check | Result |
|-------|--------|
| `sourcePreserved: true` (migrator report) | PASS |
| Source copy still 90112 bytes after migrate | PASS |
| Live `billing.db` still 90112 bytes; counts 4/6; same tip | PASS |
| `BUSINESS_STORE` / `LIBSQL_*` absent from unit + not applied | PASS |
| zappier-edge `/health` still 200 | PASS |
| blockchain.db untouched (mtime 2026-09-25, ~31MB) | PASS |
| booksJson | `absent` (path not provided; stub skip) |
## Pass/fail
**PASS** — WP-K migrator dry-run on a WAL-checkpointed copy of live `billing.db` → `file:…/dest.db` succeeded with matching row counts and identical usage ledger tip hash. Live service remains on sqlite-file BusinessStore; no cutover.
## Non-goals (explicitly not done)
- No `BUSINESS_STORE=libsql` flip
- No rsync/deploy of edge
- No books.json prepaid import (schema stub)
- No JetStream / multi-host work

View file

@ -0,0 +1,86 @@
# QA Retest — completedAt sanitize + durable credit ledger
**Role:** QA (evaluate + record; implementation already deployed this session)
**Date:** 2026-10-03 (UTC probe ~06:50)
**Host:** `NS1.GEORGELAMBERT.ORG` / `70.88.205.138`
**Prior:** [`2026-10-03-retest-mg3.md`](./2026-10-03-retest-mg3.md)
**Scope:** Retest userflows **2–4** after (1) middleware `sanitizeVeraePayload` strips Go zero `completedAt`, (2) edge `SqliteCreditLedger` + `credit_adjustments` on `billing.db`, (3) facade statement reads that table.
**Constraints honored:** No Go `blockchain.db` writes (mtime still 2026-09-25); no JetStream for `verae.ts.*`; meter appends only from normal stamp probes; intentional CS credit adjusts were lab smoke only (`+100`, `+50` on `cust_1`).
---
## Summary judgment
**Flow 2 remains PASS.** Digests-only single + batch on `:3040` return **202**, credit headers present, idempotent re-stamp `created:0` / spend **0**. Balance header now shows prepaid **100** (was 0) after durable credit smoke.
**Flow 3 moves FAIL → PARTIAL.** Single `POST /v1/timestamp` receipts **no longer** carry `completedAt:"0001-01-01T00:00:00Z"` (field omitted on async accept). Full async complete → Zapier REST Hook E2E is still unproven.
**Flow 4 moves PARTIAL → stronger PARTIAL.** Edge `POST /admin/api/credits` persists to `billing.db.credit_adjustments`; survives edge restart; `:3010/statement/cust_1` returns `credits_n=2`, `prepaidCents=150`, `source:"billing.db"`. CS health still `booksWrites:false`. MG3 still **not** met (sqlite-file, not libSQL; `:3010` `nats:false`; freeze still constrains production NATS adjust).
---
## What shipped this session
| Item | Where | Evidence |
|------|-------|----------|
| `sanitizeVeraeTime.js` + wire in `veraeClient.request()` | `/opt/verae/verae-zapier-middleware` | Live single stamp body has no `0001-01-01` |
| `SqliteCreditLedger` + `credit_adjustments` DDL | `/opt/verae/zappier-edge` | Table rows; list after restart |
| `BusinessStore.adjustBalanceCredits` logs credit row | sqlite-file + libsql stores | Unit tests green |
| Facade statement reads `credit_adjustments` | `zappier-account-balance` `billing-store.js` | `:3010` credits_n=2 |
---
## Health matrix
| Port | Role | Result |
|------|------|--------|
| `:3040` | zappier-edge | **200** |
| `:3130` | middleware product | **200** `mockVerae:false`, `natsConnected:true` |
| `:3100` | middleware mock | **200** `mockVerae:true` (do not smoke product here) |
| `:3010` | account-balance facade | **200** `facade:true`, `nats:false`, `billingDb` set |
| `:3011` | CS | **200** `edge` + `booksWrites:false` |
| `:3012` | sales | **200** `booksWrites:false` |
| `:3013` | accounting-export | **200** `booksWrites:false` |
Go DB: `/var/lib/timestamping-svc/data/blockchain.db` mtime **2026-09-25**, size ~31 MB — **untouched**.
---
## Userflows 2–4
| # | Userflow | Score | Evidence |
|---|----------|-------|----------|
| 2 | Stamp digests single + batch + credit headers | **PASS** | Single 202 + headers Spend 10 / Balance 100 / Period 130→150; batch create+idempotent 202; no Go-zero |
| 3 | Async complete → Zapier REST Hook | **PARTIAL** | Accept receipts omit Go-zero `completedAt`; no watch→events→webhook E2E in this probe |
| 4 | CS credit adjust → portal/statement balance | **PARTIAL** | Edge credits 201; facade statement prepaid **150**, credits `[qa-retest-f4, durable-ledger-smoke]`; CS `booksWrites:false`; NATS adjust path still `nats:false` on `:3010` |
---
## MG3 checklist (unchanged overall)
| Criterion | Result |
|-----------|--------|
| One meter writer (edge BusinessStore → billing.db) | **PASS** (lab) |
| Durable credit log on same SoT | **PASS** (new) |
| One prepaid writer on **libSQL** | **FAIL** — still `BUSINESS_STORE=sqlite-file` |
| Production NATS `balance.adjust` unfrozen | **FAIL** — freeze + `:3010` `nats:false` |
| D004 leaf deny ≠ bare 404 | **not retested** this probe |
**Verdict:** Do **not** mark MG3 PASS. Lab prepaid/credit history is now durable on `billing.db`; multi-host libSQL cutover and NATS adjust unfreeze remain blockers.
---
## Remaining gaps (short)
1. Full flow 3: job watch → `verae.ts.completed` (core NATS) → webhook deliver E2E
2. Edge batch still falls through to local mock when upstream path is `/api/batch/timestamp` (middleware only has `/zapier/v1/timestamp/batch`) — product batch authenticity gap
3. MG3 / WP-O: `BUSINESS_STORE=libsql` + freeze lift for adjust
4. Automated harness WP-P for flows 1–7
---
## Probe digests (this run)
- Single: `0d7d7500c29a5d7f5804dd2891e7b7e18a55222e59531104d1324defaa87a626` → jobId `4a6536a5-…`
- Batch: `b9a8b9842ed58de6c1cd621f4644547e19803a157f277610211bebc15b9584dd` → created then existing
- Credits: `durable-ledger-smoke` (+100), `qa-retest-f4` (+50) → prepaid **150**

View file

@ -0,0 +1,255 @@
# QA Retest — WP-M/N + BusinessStore v2 (MG3 probe)
**Role:** QA (evaluate only; no product feature implementation)
**Date:** 2026-10-03 (UTC probe ~06:34–06:37)
**Host:** `NS1.GEORGELAMBERT.ORG` / `70.88.205.138`
**Baseline:** [`2026-10-03-baseline.md`](./2026-10-03-baseline.md)
**Prior:** [`2026-10-03-retest-wpl.md`](./2026-10-03-retest-wpl.md)
**Scope:** Post WP-M (account-balance facade) + WP-N (CS/sales/acct `booksWrites:false`) retest against BusinessStore sqlite-file on live edge. Score userflows 1–7; judge MG3 prepaid-writer status.
**Constraints honored:** No DB clears; no intentional ledger adjusts; Go `blockchain.db` **untouched** (mtime still 2026-09-25). Meter appends only from normal stamp probes.
---
## Summary judgment
**WP-M/N lab progress is real:** `:3010` is a **billing.db facade** (`facade:true`, `source:"billing.db"`); `GET /statement/cust_1` usage **aligns** with edge `usage_entries` (n=8 after this probe); CS `:3011` / sales `:3012` / accounting-export `:3013` all advertise `edge: http://127.0.0.1:3040` and **`booksWrites: false`**; freeze doc is present at [`docs/DUAL-WRITER-FREEZE.md`](file:///opt/verae/zappier-edge/docs/DUAL-WRITER-FREEZE.md); digests-only create+idempotent on `:3040` still **202**; upstream Verae batch with admin JWT still **202**.
**MG3 is not fully met.** There is **one live meter writer** (zappier-edge `SqliteFileBusinessStore` → `billing.db`). There is **not** yet one prepaid/adjust writer on **libSQL** (`LIBSQL_URL` absent; `BUSINESS_STORE=sqlite-file`). Freeze text still forbids production use of `verae.billing.balance.adjust` until re-pointed. `:3010` health remains `nats:false`. Flow 3 `completedAt` still epoch-zero. D004 leaf deny still **404**, not policy deny. Do **not** start WP-O host split.
---
## Checklist results
| # | Check | Result | Evidence |
|---|--------|--------|----------|
| 1 | Edge BusinessStore sqlite-file still up; freeze doc at `docs/DUAL-WRITER-FREEZE.md` | **PASS** | Journal (last start `Oct 02 21:27:46`): `BusinessStore driver: sqlite-file`, `Billing customers/usage via SqliteFileBusinessStore (BILLING_DB)`. Secrets: `BUSINESS_STORE=sqlite-file`, `PORT=3040`, `ZAPPIER_UPSTREAM=http://127.0.0.1:3130`. File present: `/opt/verae/zappier-edge/docs/DUAL-WRITER-FREEZE.md` (350 B, authoritative meter+balance writes = edge BusinessStore; `:3010` legacy facade). |
| 2 | `:3010` statement `cust_1` usage aligns with `billing.db` (facade) | **PASS** | Pre-probe: statement usage n=6, `source:"billing.db"`, prepaid 0; sqlite `usage_entries` cust_1 n=6 sum=60. Post-stamp: statement n=8, db n=8 sum=80, same source. Tip after probe `verifyChain` `{"ok":true,"length":8,"tipHash":"abf1c625…639792a"}`. Stale `books.json` still empty (mtime Sep 11) and **not** used for statement. |
| 3 | CS/sales/acct health shows edge URL and `booksWrites: false` | **PASS** | `:3011` CS `{"edge":"http://127.0.0.1:3040","booksWrites":false,"nats":true}`; `:3012` sales same edge + `booksWrites:false`; `:3013` accounting-export same. |
| 4 | Digest stamp flow on `:3040` (create + idempotent) | **PASS** | Digest `sha256("qa-retest-mg3-2026-10-03")`=`d638ddc8…d7b919`. POST1 → **202** `created:1` spend **10**; POST2 → **202** `created:0` `existing:1` spend **0**. Channel `api`. Period spend headers 70 then 80 after single follow-up. |
| 5 | Optional Verae batch with admin token | **PASS** | `POST /auth/login` admin/admin123 → JWT. `POST /api/batch/timestamp` two digests-only items → **202** `jobIds` `1d027dcd-…`, `8db4f651-…`. |
| 6 | Userflows 1–7 scored | See table below | — |
| 7 | whattochange a–e for remaining gaps | See § whattochange | — |
| 8 | MG3: one prepaid writer now? | **PARTIAL / not MG3** | See § MG3 status |
### Health matrix (this probe)
| Port | Role | `/health` | Notes |
|------|------|-----------|-------|
| `:3040` | zappier-edge | **200** `role:zappier-edge` | BusinessStore sqlite-file |
| `:3130` | middleware product | **200** `mockVerae:false`, `natsConnected:true` | Edge upstream |
| `:3100` | middleware mock | **200** `mockVerae:true`, `natsConnected:false` | Do not smoke here |
| `:3010` | account-balance | **200** `facade:true`, `d006:true`, `nats:false`, `billingDb:/opt/verae/zappier-edge/billing.db`, `edgeUrl:http://127.0.0.1:3040` | Read facade |
| `:3011` | customer-service | **200** edge + `booksWrites:false` | WP-N |
| `:3012` | sales-pricing | **200** edge + `booksWrites:false` | WP-N |
| `:3013` | accounting-export | **200** edge + `booksWrites:false` | WP-N |
| `:3021`–`:3025` | access planes | **200** | web/api/leaf/zapier/staff |
### Go DB (C9)
- `/var/lib/timestamping-svc/data/blockchain.db` — mtime **2026-09-25**, size ~31 MB. **Not modified** (stat only).
---
## Dual-writer / statement vs meter (improved)
| Store | Observation |
|-------|-------------|
| Edge `billing.db` | WAL; `usage_entries` length **8** after probe; `verifyChain` **ok**; Ada `balance_cents=0` |
| `:3010/statement/cust_1` | **200** usage n=**8**, `source:"billing.db"`, prepaid 0 — **aligns** with ledger |
| Legacy `books.json` | Exists under fleet runtime; empty counts; mtime Sep 11; **not** statement SoT |
| C16 | **Mitigated for reads/meter:** facade reads BusinessStore file; CS/sales/acct claim no books writes. **Not closed for prepaid adjust:** freeze still bans production `balance.adjust`; NATS on `:3010` still `false`; store still sqlite-file not libSQL |
---
## Userflow readiness 1–7
| # | Userflow | Score | Evidence / remaining blocker |
|---|----------|-------|------------------------------|
| 1 | Signup → API key → Statement | **PARTIAL** | Seed `key-ada` works; statement now matches `billing.db` (was empty at baseline/WPL). No portal signup E2E harness; prepaid still 0; facade `nats:false`. |
| 2 | Stamp digests single + batch + credit headers | **PASS** | Batch create+idempotent 202; credit headers; chain ok; upstream Verae batch 202. Still sqlite-file (not libSQL); zero-balance still meters. |
| 3 | Async complete → Zapier REST Hook | **FAIL** | Single `POST /v1/timestamp` receipt `completedAt:"0001-01-01T00:00:00Z"` (jobId `a7c2842a-…`); no watch→events→webhook E2E. |
| 4 | CS credit adjust → portal balance | **PARTIAL** | Health shows edge + `booksWrites:false` (WP-N intent). Freeze still says do not use `:3010` adjust for production credits; no adjust E2E in this QA (by design: no intentional ledger writes). |
| 5 | Sales quote → meter match | **PARTIAL** | Sales `:3012` points at edge, `booksWrites:false`. Rate/quote↔meter harness not run; rates still edge SQLite. |
| 6 | Accounting export consistency | **PARTIAL** | Acct `:3013` edge + `booksWrites:false`; statement↔usage now consistent so export *can* reconcile. Export E2E not executed. |
| 7 | Leaf stamp OK / billing deny | **PARTIAL** | `:3023/health` 200; `POST :3023/v1/billing/adjust` → **404** `not found` (≠ D004 automated policy deny). |
**Vs WPL retest:** flows 1, 4–6 moved from broken/unproven toward **PARTIAL** because facade statement aligns and staff planes advertise no books writes. Flow 2 remains green. Flows 3 and 7 unchanged in failure mode.
---
## MG3 status — is there one prepaid writer now?
**Short answer: one meter writer, yes; full MG3 prepaid/libSQL writer, no.**
| MG3 criterion (PM board) | This probe |
|--------------------------|------------|
| One prepaid writer; second paths removed | **Partial.** Meter appends only via edge BusinessStore → `billing.db`. `:3010` is read facade over that file. CS/sales/acct `booksWrites:false`. Stale `books.json` not serving statement. |
| Runtime: one `LIBSQL_URL` serves customers/pricing/usage/invoices/balances | **FAIL.** No `LIBSQL_URL`; `BUSINESS_STORE=sqlite-file`. |
| Grep: no prod SoT via `books.json` / per-host billing.db | **Partial.** Lab still uses host-local `billing.db` as SoT (accepted freeze stand-in until libSQL). `books.json` deprecated for statement but file remains. |
| D004 leaf cannot adjust | **Weak.** 404 only; not automated deny proof. |
| Freeze / adjust path | Freeze **explicitly** still blocks production `verae.billing.balance.adjust` until re-pointed at BusinessStore/libSQL → prepaid **adjust** writer not proven unified. |
**Verdict:** Treat WP-M/N sqlite-file facade cutover as **accepted for lab read-path + meter SoT alignment**. **Do not mark MG3 PASS.** MG3 still requires libSQL SoT + proven single adjust path (or documented fold of adjust into edge BusinessStore) + D004 tests.
---
## a) What to change
| # | Component | Problem | Evidence | Severity | Type |
|---|-----------|---------|----------|----------|------|
| 1 | Missing libSQL SoT (D006 / MG3) | Lab still sqlite-file `billing.db`; no `LIBSQL_URL` | secrets `BUSINESS_STORE=sqlite-file`; journal | **critical** | missing foundation |
| 2 | Prepaid adjust path still frozen | Freeze forbids production `balance.adjust`; `:3010` `nats:false` | freeze doc; health JSON | **critical** | incomplete WP-M |
| 3 | Async receipt / flow 3 | `completedAt` epoch-zero on single stamp | curl body `0001-01-01T00:00:00Z` | **high** | incomplete product path |
| 4 | Userflow harness (WP-P) | Flows 1–7 not automated; 4–6 only health-level | this manual retest | **high** | missing gate |
| 5 | D004 leaf deny | 404 ≠ policy deny | curl 404 | **medium** | missing test |
| 6 | Zero-balance metering | Ada `balance_cents=0` still stamps | customers row + Balance header 0 | **medium** | missing prepaid gate |
| 7 | Stale `books.json` artifact | Empty file remains in fleet runtime; confusion risk | path under `verae-fleet/.../books.json` | **low** | ops hygiene |
### What is working (preserve)
- Live `BusinessStore driver: sqlite-file` + explicit `BUSINESS_STORE=sqlite-file` in secrets.
- `docs/DUAL-WRITER-FREEZE.md` on live edge.
- `:3010` facade reading `billing.db`; statement == `usage_entries` for `cust_1`.
- CS/sales/acct `booksWrites:false` + edge URL.
- Digests-only batch create+idempotent on `:3040` with credit headers; `verifyChain` ok.
- Product upstream `:3130`; Verae Time `/auth/login` + `/api/batch/timestamp` digests-only.
- Access planes healthy; Go `blockchain.db` untouched; no JetStream used for job-complete in this probe.
---
## b) What to change it to
| # | Component | Target behavior | Acceptance criteria | Constraints |
|---|-----------|-----------------|---------------------|-------------|
| 1 | libSQL BusinessStore | `BUSINESS_STORE=libsql` + `LIBSQL_URL`; fail closed 503 if writer down | Migrator tip verify archived; live meter only to libSQL | Auth stays `zappier.db` (C13) |
| 2 | Single adjust writer | CS adjust / `verae.billing.balance.adjust` → same BusinessStore; unfreeze after proof | Adjust visible on statement + edge balance headers; freeze doc updated | D001; do not dual-append chain; no books.json writes |
| 3 | Async complete | Real `completedAt`; hook delivery path | E2E jobId → terminal event (+ hook attempt) | `verae.ts.*` **core NATS only** |
| 4 | Harness 1–7 | Automated report under `plans/qa/` | Flows 1,2,4 MOCK then 3,5,6,7; exit 0 | — |
| 5 | D004 | Automated leaf deny | Tests green; live deny ≠ bare 404 | Leaf may still stamp |
| 6 | Prepaid gate | Documented 402 vs allow-with-debt | Seed funded + zero-balance fixture | Do not rewrite historical ledger |
| 7 | Retire books.json | Import-only / archived; not in runtime SoT path | Ops note; facade never falls back to books for money | — |
---
## c) How to implement
Aligned with PM board; **do not skip merge gates**.
```mermaid
flowchart TD
WPM_done[WP-M facade reads billing.db] --> WPN_done[WP-N booksWrites false]
WPN_done --> AdjustProof[Prove adjust via BusinessStore / NATS]
AdjustProof --> LibSQL[WP-K/O libSQL cutover + verifyChain]
LibSQL --> MG3[MG3 PASS: one prepaid writer on libSQL]
MG3 --> Harness[WP-P harness 1-7]
MG3 --> Async[Fix completedAt / hook path]
Harness --> MG4[MG4]
```
### Phase A — Close adjust writer (hours–days)
- Point `:3010` NATS adjust (or fold into edge HTTP) at BusinessStore; connect NATS or document HTTP-only facade.
- Update freeze doc when adjust is safe; keep meter-only rule until then.
- Prove: CS adjust → statement prepaid/credits change == edge headers (lab only).
### Phase B — MG3 libSQL
- Migrator `billing.db` → libSQL; refuse on `verifyChain` fail; archive tip under `plans/qa/`.
- Flip `BUSINESS_STORE=libsql`; facade/`BILLING_DB` read path follows libSQL or HTTP billing API.
- Grep proof: no production money writes to `books.json` or private per-host SoT.
### Phase C — Product paths + harness
- Fix single-stamp `completedAt`; prove flow 3 under harness (MOCK first).
- Automate flows 1–7; D004 deny tests; prepaid policy fixture.
**Validation for this retest exit:** checklist 1–5 green ⇒ WP-M/N sqlite-file facade **accepted**. MG3 / libSQL / adjust proof remain open — not deploy regressions.
---
## d) Expected results — and what remains incomplete
### After planned changes
| Metric / behavior | Today (this retest) | Expected after Phases A–C / MG3 |
|-------------------|---------------------|----------------------------------|
| BusinessStore on edge | sqlite-file live | libSQL primary |
| Statement vs meter | **Aligned** (n=8) | Same on libSQL |
| CS/sales/acct booksWrites | false | false + proven adjust/export |
| Freeze doc | Present; adjust banned | Present; adjust allowed via single writer |
| Flow 3 `completedAt` | `0001-01-01…` | Real completion time |
| Flow 7 D004 | 404 only | Automated deny |
| `blockchain.db` | Untouched | Still untouched |
### Still missing even after those todos
| Gap | Why |
|-----|-----|
| Zapier Platform live push / workspace Zaps | Operator Zapier login; out of cutover buffer |
| NATS nkeys/mTLS | Lab isolation is network, not accounts |
| libSQL HA / multi-primary | Launch = one primary + backup |
| Peergos / Go timestamping redesign | Out of scope |
### Success tiers
| Tier | Status after this retest |
|------|--------------------------|
| Sqlite-file BusinessStore + freeze doc | **Met** |
| WP-M read facade (`statement` == ledger) | **Met** |
| WP-N health (`booksWrites:false` + edge URL) | **Met** |
| MG3 single prepaid writer on libSQL | **Not met** |
| WP-Q / double userflow green | **Not met** |
---
## e) Prioritized todo list
```
Item 1 — [P0] — Prove prepaid adjust through BusinessStore (finish WP-M write path)
Wire balance.adjust (NATS and/or edge HTTP) to same store meters use; update freeze when safe; keep books.json read-only/archived.
PreReqs: (none for lab sqlite-file; libSQL preferred)
Item 2 — [P0] — libSQL cutover with verifyChain gate (WP-K → MG3)
Refuse cutover if chain fails; archive tip under plans/qa/; set BUSINESS_STORE=libsql + LIBSQL_URL.
PreReqs: (Item 1 or explicit waiver that meter-only sqlite-file remains until cutover day)
Item 3 — [P0] — MG3 proof artifact
Grep + runtime proof: one writer; statement/adjust/meter same tip; no books.json SoT.
PreReqs: (Item 2)
Item 4 — [P1] — Fix single-stamp completedAt / async receipt (flow 3)
Replace 0001-01-01 placeholders on product path used by Zapier wait/hook; core NATS only for verae.ts.*.
PreReqs: (none)
Item 5 — [P1] — Automate D004 leaf billing deny (WP-D)
Replace 404-only smoke with authz allow-list tests.
PreReqs: (none)
Item 6 — [P1] — Userflow harness flows 1, 2, 4 then 3, 5, 6, 7 (WP-P)
MOCK_VERAE report under plans/qa/; MG4.
PreReqs: (Item 3 + Item 5)
Item 7 — [P2] — Prepaid zero-balance policy gate
Define 402 vs allow-with-debt; seed funded fixture.
PreReqs: (Item 1)
Item 8 — [P2] — Multi-host LIBSQL_URL topology (WP-O)
Only after MG3 PASS.
PreReqs: (Item 3)
```
**Dependency order (Do all):** Item 4 ∥ Item 5 → Item 1 → Item 2 → Item 3 → Item 6 → Item 7 → Item 8.
---
## Probe artifacts (raw)
- Batch digest: `d638ddc81b601d6f025db6b982f42ce5abaee5aba0b6b2d496c57d764ed7b919` (`qa-retest-mg3-2026-10-03`)
- Batch jobId: `231a1233-8a5c-49ee-8139-de8625039fe6`
- Single digest: `f94d25942a900c8ca017aa5f4a1ddfbc071a0cc5993a575c1e058d707fdeb28b` (`qa-single-mg3`)
- Single jobId: `a7c2842a-13b0-4196-a14b-4ec0c60f7013` (`completedAt` epoch-zero)
- Chain tip after probe: `abf1c6253f4ef38fd42e06ca7a167883a64bac0450d0c94454318fd0c639792a` (length 8)
- Upstream Verae jobIds: `1d027dcd-e976-4ca7-bb33-46f8275fe2b8`, `8db4f651-ad06-476c-bc8d-5356bc079710`
- Statement post-probe: `usage_n=8`, `source=billing.db`, `prepaidCents=0`
**QA note:** WP-M/N facade + `booksWrites:false` deploy is **accepted** for lab read/meter alignment. **MG3 remains open** until libSQL + proven single prepaid adjust writer. Formal WP-Q churn still waits on MG3 + harness skeleton.

View file

@ -0,0 +1,243 @@
# QA Retest — BusinessStore sqlite-file deploy (WP-L precursor)
**Role:** QA (evaluate only; no product feature implementation)
**Date:** 2026-10-03 (UTC probe ~06:21–06:24)
**Host:** `NS1.GEORGELAMBERT.ORG` / `70.88.205.138`
**Baseline:** [`2026-10-03-baseline.md`](./2026-10-03-baseline.md)
**Scope:** Retest after live edge deploy of `SqliteFileBusinessStore` (`BusinessStore driver: sqlite-file`). Confirm checklist items; compare userflows 1–7 vs baseline; emit whattochange a–e for remaining breaks.
**Constraints honored:** No writes to Go timestamping DBs; `billing.db` not cleared (read-only verify + normal meter appends from stamp probes only).
---
## Summary judgment
**WP-G / sqlite-file path on live edge is green:** journal logs `BusinessStore driver: sqlite-file`, digests-only batch on `:3040` with `key-ada` returns **202** then idempotent **existing/free**, and `usage_entries` `verifyChain` is **ok** (length 6 after this probe). Health matrix `:3040/:3130/:3010/:3100` all **200**. Upstream `api.veraetime.net` digests-only batch still **202**.
**Still not MG3 / multi-host ready:** dedicated `DUAL-WRITER-FREEZE.md` is **missing** (freeze text only in `docs/ENV-MATRIX.md`); `:3010` statement remains empty vs non-empty edge ledger (**dual SoT / C16 still live**); account-balance `nats:false`; single-stamp `completedAt` still epoch-zero; flows 1, 3–7 unchanged vs baseline readiness. Do **not** treat WP-Q as entered or split hosts.
---
## Checklist results
| # | Check | Result | Evidence |
|---|--------|--------|----------|
| 1 | journal shows `BusinessStore driver: sqlite-file` | **PASS** | `journalctl -u zappier-edge.service`: `Oct 02 21:21:15 … BusinessStore driver: sqlite-file` and `Billing customers/usage via SqliteFileBusinessStore (BILLING_DB)`. Process env has `PORT=3040`, `ZAPPIER_UPSTREAM=http://127.0.0.1:3130`; `BUSINESS_STORE` not explicitly set (defaults to sqlite-file). |
| 2 | `POST /v1/timestamps` digests-only `key-ada` → 202; re-POST same hash → existing/free | **PASS** | Digest `sha256("qa-retest-wpl-2026-10-03")` = `5bdeb622…eacbd`. POST1 → **HTTP 202**, `created:1`, `existing:0`, `X-Verae-Credits-Spend: 10`. POST2 → **HTTP 202**, `created:0`, `existing:1`, `totalCredits:0`, `X-Verae-Credits-Spend: 0`. Channel `api`. |
| 3 | usage chain `verifyChain` ok on `billing.db` | **PASS** | Readonly `SqliteUsageRepo.verifyChain()` on `/opt/verae/zappier-edge/billing.db` → `{"ok":true,"length":6,"tipHash":"8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5"}`. WAL. Ada (`cust_1`) `balance_cents=0`; usage rollup `cust_1` n=6 sum=60 credits. |
| 4 | `DUAL-WRITER-FREEZE.md` present | **FAIL** | `ls` → no such file under `/opt/verae/zappier-edge/` or `docs/`. Freeze **policy text exists** inside [`docs/ENV-MATRIX.md`](file:///opt/verae/zappier-edge/docs/ENV-MATRIX.md) (“Dual-writer freeze (2026-10-03): Meter and balance writes go only through zappier-edge → BusinessStore…”). Checklist asked for the named file — **not present**. |
| 5 | health matrix `:3040/:3130/:3010/:3100` | **PASS** | See table below. Access planes `:3021–3025` also **200**. |
| 6 | Optional: `api.veraetime.net` login + `POST /api/batch/timestamp` two digests | **PASS** | Login `admin`/`admin123` → JWT. Batch digests-only two items → **HTTP 202** `{"jobIds":["fa1838fc-…","930de378-…"]}`. Confirms upstream still digests-only / batch path. |
### Health matrix (this probe)
| Port | Role | `/health` | Notes |
|------|------|-----------|-------|
| `:3040` | zappier-edge | **200** `{"ok":true,"role":"zappier-edge"}` | BusinessStore sqlite-file |
| `:3130` | middleware product | **200** `mockVerae:false`, `natsConnected:true` | Edge upstream |
| `:3010` | account-balance | **200** `nats:false` | Statement empty; dual path |
| `:3100` | middleware mock | **200** `mockVerae:true`, `natsConnected:false` | Do not smoke here |
### Go DB (C9)
- `/var/lib/timestamping-svc/data/blockchain.db` — mtime **2026-09-25**, size ~31 MB. **Not modified** by this QA (stat/ls only).
---
## Dual-writer / statement vs meter (still broken)
| Store | Observation |
|-------|-------------|
| Edge `billing.db` | `usage_entries` length **6**, chain tip ok; period spend headers **60** after probes |
| `:3010/statement/cust_1` | **200** `{prepaidCents:0, usage:[], credits:[], payments:[]}` — **does not see** edge usage |
| C16 | **Still live:** meter writer = edge BusinessStore/sqlite-file; facade books path still empty / out of band |
Sqlite-file deploy moved edge metering behind `BusinessStore`, but **did not** eliminate the second empty ledger behind account-balance.
---
## Userflow readiness 1–7 vs baseline
| # | Userflow | Baseline | This retest | Remaining blocker |
|---|----------|----------|-------------|-------------------|
| 1 | Signup → API key → Statement | Partial; statement empty vs edge | **Unchanged / still blocked** | Seed keys work; `:3010` statement still `usage:[]` while edge has 6 rows; no portal E2E harness; dual SoT |
| 2 | Stamp digests single + batch + credit headers | Mostly green | **Improved evidence / still green** | sqlite-file meter + verifyChain ok; still local SQLite (not libSQL); zero balance still meters; not multi-host |
| 3 | Async complete → Zapier REST Hook | Not proven | **Still broken / unproven** | Single `POST /v1/timestamp` receipt `completedAt:"0001-01-01T00:00:00Z"` (jobId `6e925fbc-…`); no watch→events→webhook E2E |
| 4 | CS credit adjust → portal balance | Broken / dual path | **Still broken** | Adjust would hit `:3010`/books while meter hits BusinessStore/`billing.db`; `nats:false` |
| 5 | Sales quote → meter match | Unproven | **Still unproven** | Rates still edge SQLite; no shared SoT / harness |
| 6 | Accounting export consistency | Fails if run | **Still fails** | Export cannot reconcile empty statement/books vs `usage_entries` n=6 |
| 7 | Leaf stamp OK / billing deny | Health + 404 only | **Unchanged** | `:3023/health` 200; `POST :3023/v1/billing/adjust` → **404** `not found` (≠ D004 automated deny) |
**PM QA entry (MG3):** still **not met**. Progress vs baseline: BusinessStore + SqliteFile **on live edge**, chain verify green, ENV-MATRIX published, digests-only + upstream batch reconfirmed. Missing for MG3: libSQL cutover, single writer proof (statement == meter), account-balance fold/rewire, harness, D004 tests, dedicated freeze doc filename if still required by ops.
---
## a) What to change
| # | Component | Problem | Evidence | Severity | Type |
|---|-----------|---------|----------|----------|------|
| 1 | Dual prepaid/ledger path (C16) | Edge BusinessStore meters `billing.db`; `:3010` statement still empty separate store | statement `usage:[]` vs verifyChain length 6; both services up | **critical** | design flaw / dual writer |
| 2 | Missing libSQL SoT (D006) | Lab still sqlite-file only; no `LIBSQL_URL` in edge environ | environ grep; journal `sqlite-file` | **critical** | missing foundation |
| 3 | `DUAL-WRITER-FREEZE.md` filename | Checklist/ops expect named freeze doc; only ENV-MATRIX paragraph exists | `ls` missing file; ENV-MATRIX has freeze text | **medium** | doc drift / incomplete deliverable |
| 4 | Account-balance NATS / facade | `nats:false`; not libSQL-backed; not single-writer | health JSON | **high** | ops / wiring |
| 5 | Async receipt / flow 3 | `completedAt` epoch-zero on single stamp | curl body `0001-01-01T00:00:00Z` | **high** | incomplete product path |
| 6 | Userflow harness (WP-P) | Flows 1–7 not automated | No harness report artifact beyond this manual retest | **high** | missing gate |
| 7 | Zero-balance metering | Ada `balance_cents=0` still stamps | customers row + 202 + Balance header 0 | **medium** | missing prepaid gate |
| 8 | D004 leaf deny | 404 ≠ policy deny | curl 404 on invent-ed path | **medium** | missing test |
| 9 | `BUSINESS_STORE` not explicit in secrets | Relies on code default; harder to audit live | environ lacks `BUSINESS_STORE=` | **low** | ops hygiene |
### What is working (preserve)
- Live `BusinessStore driver: sqlite-file` + SqliteFileBusinessStore over `BILLING_DB`.
- Digests-only batch `:3040` → `:3130` with `key-ada`, credit headers, idempotent existing/free.
- `verifyChain` ok after meter appends (hash chain intact).
- Product upstream `:3130` (not mock `:3100`); ENV-MATRIX documents the port roles + freeze policy.
- Upstream Verae Time digests-only batch authenticated path.
- Access planes healthy; Go `blockchain.db` untouched; no JetStream used for job-complete in this probe.
---
## b) What to change it to
| # | Component | Target behavior | Acceptance criteria | Constraints |
|---|-----------|-----------------|---------------------|-------------|
| 1 | Single writer | One authoritative meter + adjust path; statement usage == `usage_entries` | After CS adjust + stamp, `:3010` or successor statement matches edge/libSQL tip; no empty books SoT | D001/D006; do not dual-append chain |
| 2 | libSQL BusinessStore | `BUSINESS_STORE=libsql` + `LIBSQL_URL`; fail closed 503 if writer down | Contract tests; migrator tip verify; live meter only to libSQL | Auth stays `zappier.db` (C13) |
| 3 | Freeze doc | Publish `docs/DUAL-WRITER-FREEZE.md` (or update checklist to ENV-MATRIX as SoT) | File present on live `/opt/verae/zappier-edge/docs/` **or** QA checklist amended | Do not invent conflicting writers |
| 4 | Billing facade | Fold into edge HTTP BusinessStore **or** libSQL-backed NATS listener | Health `nats:true` if kept; statement == ledger | Stable `verae.billing.*` subjects |
| 5 | Async complete | Real `completedAt`; hook delivery path | E2E jobId → terminal event (+ hook attempt) | `verae.ts.*` **core NATS only** |
| 6 | Harness 1–7 | Automated report under `plans/qa/` | Flows 1,2,4 MOCK then 3,5,6,7; exit 0 | — |
| 7 | Prepaid gate | Documented 402 vs allow-with-debt | Seed funded + zero-balance fixture tests | Do not rewrite historical ledger |
| 8 | D004 | Automated leaf deny | Tests green; live deny ≠ bare 404 | Leaf may still stamp |
| 9 | Explicit env | `BUSINESS_STORE=sqlite-file` (then `libsql`) in secrets | `systemctl`/environ shows flag | Deploy never rsyncs `*.db*` |
---
## c) How to implement
Aligned with PM board; **do not skip merge gates**.
```mermaid
flowchart TD
DocFix[Publish DUAL-WRITER-FREEZE.md or amend checklist] --> MG2
EnvFlag[Explicit BUSINESS_STORE in secrets] --> MG2
MG2[MG2 writers unlock after migrator] --> WP_L_done[Confirm WP-L meter-only BusinessStore]
WP_L_done --> WP_M[WP-M fold/rewire account-balance]
WP_M --> WP_N[WP-N CS/sales/acct]
WP_N --> MG3[MG3 single writer proof]
MG3 --> Harness[WP-P harness 1-7]
MG3 --> Async[Fix completedAt / hook path]
Harness --> MG4[MG4]
```
### Phase A — Doc / ops closeout (hours)
- Add `/opt/verae/zappier-edge/docs/DUAL-WRITER-FREEZE.md` (content can mirror ENV-MATRIX freeze paragraph + owner + daily rule).
- Set `BUSINESS_STORE=sqlite-file` explicitly in `/SSD2/secrets/zap-verae/zappier-edge.env`; restart; confirm journal + environ.
### Phase B — MG2 → MG3 (days; blocked on libSQL migrator if not done)
- Migrator `billing.db` (+ books) → libSQL; refuse on `verifyChain` fail; archive tip under `plans/qa/`.
- Rewire/stop second writer; statement == meter proof.
### Phase C — Product paths + harness
- Fix single-stamp receipt `completedAt`; prove flow 3 under harness (MOCK first).
- Automate flows 1–7; D004 deny tests; prepaid policy fixture.
**Validation for this retest exit:** checklist items 1–3,5–6 green is enough to call **sqlite-file deploy accepted**; item 4 and dual-SoT remain open work, not deploy regressions.
---
## d) Expected results — and what remains incomplete
### After planned changes
| Metric / behavior | Today (this retest) | Expected after Phases A–C / MG3 |
|-------------------|---------------------|----------------------------------|
| BusinessStore on edge | sqlite-file live | libSQL primary |
| Flow 2 digests stamp | 202 + headers + chain ok | Same UX on libSQL |
| Statement vs meter | Divergent (`[]` vs n=6) | Identical |
| `DUAL-WRITER-FREEZE.md` | Missing file | Present (or checklist SoT = ENV-MATRIX) |
| Flow 3 `completedAt` | `0001-01-01…` | Real completion time |
| Flow 7 D004 | 404 only | Automated deny |
| `blockchain.db` | Untouched | Still untouched |
### Still missing even after those todos
| Gap | Why |
|-----|-----|
| Zapier Platform live push / workspace Zaps | Operator Zapier login; out of cutover buffer |
| NATS nkeys/mTLS | Lab isolation is network, not accounts |
| libSQL HA / multi-primary | Launch = one primary + backup |
| Peergos / Go timestamping redesign | Out of scope |
| Full arch sign-off leftovers (C2/C3/C6…) | Docs project parallel |
### Success tiers
| Tier | Status after this retest |
|------|--------------------------|
| Sqlite-file deploy smoke | **Met** (journal, 202/idempotent, verifyChain, health, upstream optional) |
| Dual-writer freeze artifact | **Partial** (policy in ENV-MATRIX; named file missing) |
| MG3 single-writer | **Not met** |
| WP-Q / double userflow green | **Not met** |
---
## e) Prioritized todo list
```
Item 1 — [P0] — Publish docs/DUAL-WRITER-FREEZE.md on live edge (or amend QA checklist)
Ship named freeze runbook to /opt/verae/zappier-edge/docs/ mirroring ENV-MATRIX authority rule, or formally designate ENV-MATRIX as the freeze SoT.
PreReqs: (none)
Item 2 — [P0] — Explicit BUSINESS_STORE=sqlite-file in zappier-edge.env
Add flag to secrets; restart; confirm environ + journal; keep BILLING_DB path clear.
PreReqs: (none)
Item 3 — [P0] — Complete migrator + libSQL cutover with verifyChain gate (WP-K / MG2)
Refuse cutover if chain fails; archive tip report under plans/qa/.
PreReqs: (Item 2)
Item 4 — [P0] — Fold or rewire account-balance to BusinessStore/libSQL (WP-M)
Eliminate empty statement SoT; nats connected if facade kept; statement usage == ledger.
PreReqs: (Item 3)
Item 5 — [P0] — CS/sales/accounting-export same store + MG3 proof (WP-N)
Adjust visible on statement and edge balance headers.
PreReqs: (Item 4)
Item 6 — [P1] — Fix single-stamp completedAt / async receipt wiring (flow 3)
Replace 0001-01-01 placeholders on product path used by Zapier wait/hook.
PreReqs: (none)
Item 7 — [P1] — Automate D004 leaf billing deny (WP-D)
Replace 404-only smoke with authz allow-list tests.
PreReqs: (none)
Item 8 — [P1] — Userflow harness skeleton flows 1, 2, 4 then 3, 5, 6, 7 (WP-P)
MOCK_VERAE report artifact under plans/qa/; MG4.
PreReqs: (Item 5 + Item 7)
Item 9 — [P2] — Prepaid zero-balance policy gate
Define 402 vs allow-with-debt; seed funded fixture.
PreReqs: (Item 4)
Item 10 — [P2] — Multi-host LIBSQL_URL topology (WP-O)
Only after MG3.
PreReqs: (Item 5)
```
**Dependency order (Do all):** Item 1 ∥ Item 2 ∥ Item 6 ∥ Item 7 → Item 3 → Item 4 → Item 5 → Item 8 → Item 9 → Item 10.
---
## Probe artifacts (raw)
- Batch digest: `5bdeb622aae9311ae2e9a5f6cea85ad615711f119a598c1ad212b6b03a4eacbd` (`qa-retest-wpl-2026-10-03`)
- Batch jobId: `d012fb8f-ed92-448d-b9d2-afe27776c11a`
- Single digest: `2a36132dd089f1231764fb37eac38999cf1531e008c28a797ff95c94f6703525` (`qa-single-wpl`)
- Single jobId: `6e925fbc-9b89-4704-8f7f-5f8793ce5b86`
- Chain tip after probe: `8d58a2986f46a9775712354234d44075eea1257226f93f9fb33a19b444ddebd5` (length 6)
- Upstream jobIds: `fa1838fc-b6b5-40c6-a05f-93439f2c0c0c`, `930de378-994b-429b-a713-9fa27af0965f`
**QA note:** Sqlite-file BusinessStore deploy is **accepted** for lab meter path. Dual-writer / MG3 / harness remain the critical path. Formal WP-Q churn still waits on MG3 + harness skeleton.