zapier-decisions/plans/qa/2026-10-03-baseline.md

344 lines
21 KiB
Markdown
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# QA Baseline — Verae × Zapier Userflows 1–7 vs D006 Multi-Host
**Role:** QA (evaluate only; no product feature implementation)
**Date:** 2026-10-03 (UTC probe ~06:04–06:07)
**Host:** `NS1.GEORGELAMBERT.ORG` / `70.88.205.138`
**Inputs:**
- Plan: [`../2026-10-03-libsql-multihost-architecture.md`](../2026-10-03-libsql-multihost-architecture.md)
- Arch review: [`../reviews/2026-10-03-arch-review.md`](../reviews/2026-10-03-arch-review.md) (approve-with-changes; A1–A8 merged; full sign-off still open)
- PM board: [`../reviews/2026-10-03-pm-board.md`](../reviews/2026-10-03-pm-board.md)
- Decision: D006 (libSQL business SoT; JetStream = work only; core NATS for `verae.ts.*`)
**Scope:** Readiness of critical userflows **1–7** for multi-host cutover. This is a **baseline** (WP-Q precursor). Per PM §5, formal WP-Q entry requires MG3; this pack documents why that gate is not met and what must change first.
---
## Summary judgment
Lab doors and digests-only stamp submit on `:3040` → `:3130` work today, but **D006 multi-host readiness fails**: there is **no libSQL**, **no BusinessStore**, and **two live prepaid/ledger writers** (`billing.db` on edge + empty `books.json` behind `:3010`). Userflows 1, 3–6 are not E2E-proven; flow 2 is partially green; flow 7 has only weak 404 evidence for billing deny. Do **not** start Phase 4 host split or treat WP-Q as “in” until single-writer libSQL lands.
---
## Live lab evidence (2026-10-03)
### Health matrix
| Port | Role | `/health` | Notes |
|------|------|-----------|-------|
| `:3040` | zappier-edge | **200** `{"ok":true,"role":"zappier-edge"}` | Bind `127.0.0.1`; systemd `zappier-edge.service` |
| `:3100` | middleware (lab/mock) | **200** `mockVerae:true`, `natsConnected:false` | Catalog/default; **not** product upstream |
| `:3130` | middleware (product) | **200** `mockVerae:false`, `natsConnected:true` | Edge `ZAPPIER_UPSTREAM` target |
| `:3010` | zappier-account-balance | **200** `nats:false` | Loopback; **not** a systemd unit; fleet-spawned |
| `:3021`–`:3025` | access planes | **200** each | web / api / leaf / zapier / staff |
### Env / upstream
- Edge secrets (`/SSD2/secrets/zap-verae/zappier-edge.env`): `PORT=3040`, **`ZAPPIER_UPSTREAM=http://127.0.0.1:3130`** (confirms arch C11 / plan A1).
- Product MW env: `PORT=3130`, `HOST=127.0.0.1`, `MOCK_VERAE=false`, `NATS_URL=nats://127.0.0.1:4222`.
- Account-balance (pid 4179): cwd under moved stack package; `FLEET_STATE_DIR=.../verae-fleet/data/runtime/account-balance-0`; `NATS_URL` lists `10.10.10.21–23` but health reports **`nats:false`**.
### Digest-only stamp (flow 2 probe)
- Digest: `sha256("qa-baseline")` = `edf8f682e42104442bb4637c665f50ada9873fa2fe6958ae00a6c05b2e5181cb`
- `POST /v1/timestamps` with `x-api-key: key-ada`, `x-billing-channel: api`, body `{sha256s:[…]}` → **HTTP 202**, `created:1` then idempotent `existing:1`, credit headers present (`X-Verae-Credits-*`).
- Items-shaped digest-only body also **202**.
- Single `POST /v1/timestamp` → **202** with `jobId`, but receipt `completedAt: "0001-01-01T00:00:00Z"` (placeholder / incomplete receipt shape for async complete).
### Dual-writer / C16 (flows 1, 4–6)
| Store | Path | Observation |
|-------|------|-------------|
| Edge ledger | `/opt/verae/zappier-edge/billing.db` | **Exists** (WAL). `usage_entries` count **3** for `cust_1` @ 10 each after probes; `customers.balance_cents` for Ada = **0** |
| Account-balance books | `FLEET_STATE_DIR/.../data/books.json` | **Exists**, empty: `prepaid:{}`, `usage:[]`, `credits:[]`, `payments:[]` |
| Statement HTTP | `GET :3010/statement/cust_1` | **200** `{prepaidCents:0, usage:[], …}` — **does not see** edge `usage_entries` |
| Edge response | batch stamp | `balanceCredits:0`, period spend headers show **30** after three 10-credit meters |
**C16 confirmed live:** edge writes metered usage into `billing.db`; account-balance serves a separate empty `books.json`. No shared SoT. Splitting hosts now would fork money.
### Go timestamping DB (C9)
- `/var/lib/timestamping-svc/data/blockchain.db` **exists** (~31 MB, mtime 2026-09-25).
- **Not modified** by this QA probe (read `ls`/`stat` only).
### libSQL / BusinessStore / harness
- No `LIBSQL_URL` / `sqld` / libSQL container observed on 138 for business data.
- Local clone `/Users/marchon/apps/verae/zappier-edge/src/db/`: **no** `business-store.ts` (`NO_BUSINESS_STORE`).
- `verae-zapier-simulator` exists (messaging/trace) but is **not** the WP-P automated harness for flows 1–7.
- Arch checklist still open: `nats-subjects.md` `verae.ts.*` / `ZAPIER_USAGE` non-SoT; published env matrix; GETTING-STARTED SoT language.
- Host `nats stream ls` to `10.10.10.21:4222` **timed out** from this probe (MW still reports `natsConnected:true` via loopback `:4222` proxy). STATUS still lists product stream **`ZAPIER_USAGE`** (C10 risk).
### Leaf D004 (flow 7 probe)
- `:3023/health` **200** leaf plane up.
- `POST :3023/v1/billing/adjust` → **404** `not found` (route absent). This is **not** yet an automated policy deny of `verae.billing.balance.adjust` (PM WP-D DoD).
---
## Userflow readiness (1–7)
| # | Userflow | Today | Blocker vs D006 |
|---|----------|-------|-----------------|
| 1 | Signup → API key → Statement | Partial: seed keys exist (`key-ada`…); statement via `:3010` empty/out of band with edge ledger | Dual SoT; no portal E2E harness; account-balance `nats:false` |
| 2 | Stamp digests single + batch + credit headers | **Mostly green** on lab (`:3040`→`:3130`, 202, headers, idempotent existing) | Still local SQLite meter; zero balance still meters; receipt placeholder on single path; not libSQL |
| 3 | Async complete → Zapier REST Hook | **Not proven** | `completedAt` zero-time on single stamp; no WP-P watch→events→webhooks.deliver E2E; Platform push still open in STATUS |
| 4 | CS credit adjust → portal balance | **Broken / dual path** | Adjust would hit books/`verae.billing.*` while meter hits `billing.db`; nats false on balance service |
| 5 | Sales quote → meter match | **Unproven** | Rate tables live in edge SQLite; sales package not wired to shared SoT |
| 6 | Accounting export consistency | **Fails if run today** | Export cannot reconcile empty books vs non-empty `usage_entries` |
| 7 | Leaf stamp OK / billing deny | Health only | Need authz allow-list tests; 404 ≠ D004 proof |
**PM QA entry (board §5):** MG3 single writer, regression green, NATS rules verified, harness skeleton, D004 automated, migrator tip report, Go DB untouched note, docs C2/C3/C5/C6 — **not met** (only Go-untouched + partial digests-only + doors up).
---
## a) What to change
| # | Component | Problem | Evidence | Severity | Type |
|---|-----------|---------|----------|----------|------|
| 1 | Prepaid / ledger SoT (C7/C16) | Two writers: edge `billing.db` meters usage; `:3010` serves empty `books.json` statement | usage_n=3 vs statement `usage:[]`; both processes up | **critical** | design flaw / dual writer |
| 2 | Missing libSQL + BusinessStore (D006 Phases 1–2) | No shared business DB or interface; multi-host impossible without per-host SQLite fork | No `business-store.ts`; no LIBSQL on 138 | **critical** | missing foundation |
| 3 | Account-balance NATS / facade | Health `nats:false` while CS/portal expect `verae.billing.*` | `:3010/health` JSON; env has NATS_URL but not connected | **high** | ops / wiring |
| 4 | Userflow E2E harness (WP-P) | Flows 1–7 not automated; simulator ≠ harness | No `plans/qa` flow reports; simulator README is messaging-only | **high** | missing gate |
| 5 | Async complete / receipts (flow 3) | Single-stamp receipt shows epoch-zero `completedAt`; REST Hook path unproven | curl body `0001-01-01T00:00:00Z` | **high** | incomplete product path |
| 6 | Zero-balance metering (flows 2/4) | Ada `balance_cents=0` still stamps and accumulates period spend | customers row + 202 responses + `X-Verae-Credits-Balance: 0` | **medium** | missing gate / prepaid policy |
| 7 | Dual middleware confusion (C11) | `:3100` mock vs `:3130` product; docs/scripts can point wrong | health JSON both ports; edge env → 3130 | **medium** | doc/ops drift |
| 8 | D004 leaf deny proof (flow 7) | Only 404 on invent-ed HTTP path; no automated NATS/HTTP deny | curl 404; WP-D tests not evidenced on lab | **medium** | missing test |
| 9 | `ZAPIER_USAGE` / nats-subjects docs (C5/C10) | STATUS still lists usage stream as product; catalog subjects omit core-only `verae.ts.*` | STATUS.md streams table; arch checklist open | **medium** | doc drift / SoT confusion |
| 10 | Env matrix / Phase 0 leftovers | Arch full sign-off blocked; GETTING-STARTED may still call books SoT | arch review checklist unchecked items | **medium** | doc freeze incomplete |
| 11 | Account-balance not systemd-managed | Process from fleet overlay / moved path; restart/ownership unclear vs edge unit | no `*account-balance*.service`; pid under `marchon` | **low** | ops hygiene |
### What is working (preserve)
- Digests-only batch + single stamp on product edge `:3040` with `key-ada` and credit headers.
- Product upstream correctly set to **`:3130`** (not mock `:3100`).
- Access-plane health on `:3021–3025`.
- Edge split DBs present and WAL: `billing.db` (business tables) + `zappier.db` (auth).
- Idempotent re-stamp (`created:0`, `existing:1`, spend 0).
- Standing rules respected in this probe: **no** JetStream for `verae.ts.*`; **no** writes to `blockchain.db`.
- Plan + A1–A8 + D001 addendum + D006 direction are the right target.
---
## b) What to change it to
| # | Component | Target behavior | Acceptance criteria | Constraints |
|---|-----------|-----------------|---------------------|-------------|
| 1 | Single prepaid/ledger writer | One authoritative adjust + `recordUsage` path on shared store; other path read-only or stopped | Grep: no prod writes to `books.json` / per-host `billing.db` as SoT; adjust on CS visible on portal statement and edge balance headers | D001 single writer; do not dual-append hash chain |
| 2 | libSQL BusinessStore | `LibsqlBusinessStore` (+ SqliteFile for CI) behind `BusinessStore`; lab migrator from `billing.db` + `books.json` | Contract tests green; migrator tip hash verify; `BUSINESS_STORE=libsql` writes only to libSQL | Auth stays edge-local `zappier.db` (C13); fail closed (503) if writer down |
| 3 | Billing facade | `:3010` (or successor) is libSQL-backed listener for `verae.billing.*` **or** folded into edge HTTP API — pick one (plan Phase 3) | Health `nats:true` (if NATS facade kept); statement usage matches `usage_entries` | Keep subjects stable for CS/sales |
| 4 | Harness 1–7 | Automated runner under simulator/harness producing `plans/qa/` report | Flows 1,2,4 runnable under MOCK; then 3,5,6,7; exit 0 artifact | Zapier cloud push optional for first MOCK pass |
| 5 | Async complete | Real completion times / receipts; `jobs.watch` → `jobs.events` → `webhooks.deliver` | E2E: jobId reaches terminal event + hook delivery (or recorded attempt) | `verae.ts.completed` / `verae.ts.receipt` remain **core NATS only** |
| 6 | Prepaid gate | Metering respects prepaid/credits policy (402 or configured allow) | Seed funded customer; zero-balance path returns expected product code | Do not rewrite historical ledger mid-test |
| 7 | Env matrix | Documented `:3040` / `:3130` product / `:3100` mock / `:3010` facade | STATUS + ops GETTING-STARTED + plan cite same matrix | Do not “fix” smoke by pointing edge at 3100 |
| 8 | D004 | Leaf cannot call balance adjust (HTTP door + NATS authz) | `verae-access-leaf` / authz tests green; live deny ≠ bare 404 | Leaf may still stamp/authz |
| 9 | NATS docs | `nats-subjects.md` Non-streams: `verae.ts.*` core-only; `ZAPIER_USAGE` telemetry non-SoT | Arch checklist boxes checked; STATUS wording demotes usage stream | No JetStream bind for job-complete |
| 10 | Phase 0 close | C2/C3/C5/C6 closed or owned; GETTING-STARTED SoT → libSQL | Arch full sign-off or explicit waivers | — |
| 11 | Balance process ops | systemd (or fleet-documented) unit with clear state dir | `systemctl status` shows balance facade; books path documented until migration | Deploy never rsyncs `*.db*` |
---
## c) How to implement
Aligned with PM DAG; **do not skip merge gates**.
```mermaid
flowchart TD
P0[Phase 0 docs/env/NATS freeze] --> P1[Phase 1 BusinessStore TDD]
P1 --> P2[Phase 2 migrator + dual-read]
P2 --> MG2[MG2 writers unlock]
MG2 --> P3[Phase 3 single writer rewire]
P3 --> MG3[MG3 single writer proof]
MG3 --> P4[Phase 4 multi-host LIBSQL_URL]
MG3 --> P5[Phase 5 harness 1-7]
P4 --> P5
P5 --> MG4[MG4 green]
MG4 --> P6[Phase 6 QA whattochange loop]
```
### Phase 0 — Conflict / docs freeze (feeding; ~1–2 days)
- **Goal:** Close open arch checklist items that confuse operators and agents.
- **Changes:** Publish env matrix; patch `nats-subjects.md`; demote `ZAPIER_USAGE` in STATUS; fix overview `zmw_` / digests examples (C3/C6); GETTING-STARTED SoT → libSQL.
- **Validation:** Arch checklist; `rg` for stale SoT claims.
- **Effort:** 1–2 agent-days (WP-A/B/C).
### Phase 1 — BusinessStore + libSQL skeleton (critical; WP-F/G/H)
- **Goal:** Interface + SqliteFile + Libsql drivers with contract tests (including hash-chain).
- **Changes:** `zappier-edge` `src/db/business-store.ts`, contract tests, CI libSQL.
- **Validation:** Jest contract green both drivers (MG1).
- **Effort:** ~3 days serial.
### Phase 2 — Migrate + dual-read (WP-J/K)
- **Goal:** Import live `billing.db` + empty/legacy `books.json`; verify chain tip; reads dual-run; **writes still locked**.
- **Validation:** Migrator report archived under `plans/qa/`; MG2.
- **Effort:** ~2 days.
### Phase 3 — Rewire writers (WP-L/M/N) — only after MG2
- **Goal:** Edge meter + account-balance + CS/sales/acct → one BusinessStore; remove second writer.
- **Validation:** Statement usage == ledger; CS adjust visible on portal/edge headers; MG3 grep proof.
- **Effort:** ~3–4 days.
### Phase 4 — Multi-host (WP-O) — only after MG3
- **Goal:** `LIBSQL_URL` primary; edge/MW/CS on separate hosts possible.
- **Validation:** Cross-host adjust/meter/statement same balance.
### Phase 5 — Harness (WP-P)
- **Goal:** Automate flows 1–7; first pass MOCK_VERAE OK.
- **Validation:** Report under `plans/qa/`; MG4.
### Phase 6 — QA loop (WP-Q)
- **Goal:** Break → whattochange → fix → retest until **two consecutive** full greens (project buffer).
**Explicit non-goals until gates pass:** JetStream for `verae.ts.*`; editing `blockchain.db`; Litestream multi-writer; multi-host split before MG3.
---
## d) Expected results / still incomplete
### After implementation (realistic)
| Metric / behavior | Today (baseline) | Expected after Phases 0–5 |
|-------------------|------------------|---------------------------|
| Business SoT | Split `billing.db` + empty `books.json` | One libSQL `verae_business` |
| Flow 2 digests stamp | 202 + headers on single host SQLite | Same UX on libSQL meter writer |
| Flow 1/4 statement vs meter | Divergent (usage 3 vs `[]`) | Identical customer/usage/balance |
| Flow 3 async → hook | Unproven; zero `completedAt` | Proven under harness (MOCK then optional live) |
| Flow 5/6 quote & export | Unproven / inconsistent | Same rate tables + export = ledger |
| Flow 7 D004 | 404 only | Automated deny |
| Multi-host money | Unsafe | Safe only post-MG3 |
| `blockchain.db` | Untouched | Still untouched |
### Still missing, incorrect, or incomplete (even after this todo list)
| Gap | Why it persists |
|-----|-----------------|
| Zapier Platform public `push` + live workspace Zaps | Needs operator Zapier login (STATUS); out of first cutover buffer |
| Live `api.veraetime.net` seal proof beyond lab MW | Creds / OpenAPI gaps (hash lookup 501); separate from libSQL SoT |
| NATS nkeys/mTLS | Cluster isolation today is `vmbr1`, not accounts |
| Portal auth on multi-edge sticky sessions | C13: auth stays local `zappier.db` for v1 |
| libSQL HA / failover RPO | Launch = one primary + backup; not multi-primary |
| Peergos / Go timestamping redesign | Out of scope |
| Full architecture sign-off | Depends on remaining checklist (docs), not only code |
### Success tiers
| Tier | Criteria |
|------|----------|
| **Minimum viable (this baseline exit)** | Dual-writer named daily owner; env matrix published; harness skeleton for 1/2/4; no host split |
| **MG3 / single-writer** | libSQL SoT; one adjust path; statement == meter |
| **Production-ready multi-host** | MG3 + WP-O + flows 1–7 twice green; fail-closed meter; D004 automated |
| **Complete** | + Platform push + live Verae seal + nkeys — separate projects |
---
## e) Prioritized todo list with PreReqs
```
Item 1 — [P0] — Freeze dual-writer authority for lab (C16 runbook)
Document which process may adjust/meter each day until MG3; forbid CS adjust against books while edge meters billing.db without a sync plan.
PreReqs: (none)
Item 2 — [P0] — Publish live env matrix (:3040 / :3130 / :3100 / :3010)
Add to STATUS + ops docs; match edge ZAPPIER_UPSTREAM=http://127.0.0.1:3130 evidence.
PreReqs: (none)
Item 3 — [P0] — BusinessStore interface + failing contract tests (WP-F)
Create business-store.ts + contract suite (customers, usage chain, rates, invoices, balances).
PreReqs: (Item 2)
Item 4 — [P0] — SqliteFileBusinessStore adapter (WP-G)
Wrap existing edge repos; keep Jest green.
PreReqs: (Item 3)
Item 5 — [P0] — LibsqlBusinessStore + CI libSQL (WP-H)
Driver + reachable LIBSQL_URL in CI/lab; MG1.
PreReqs: (Item 4)
Item 6 — [P0] — Migrator billing.db + books.json → libSQL with chain verify (WP-K)
Refuse cutover on verifyChain failure; archive tip report under plans/qa/; MG2.
PreReqs: (Item 5)
Item 7 — [P0] — Rewire edge meter to BusinessStore only (WP-L)
Remove production writes that treat local billing.db as SoT once flag on.
PreReqs: (Item 6)
Item 8 — [P0] — Fold or rewire account-balance to libSQL facade (WP-M)
One verae.billing.* listener; empty books.json import-only; nats connected if facade kept.
PreReqs: (Item 7)
Item 9 — [P1] — Rewire CS / sales / accounting-export to same store (WP-N)
Credit adjust, rates, export read libSQL; MG3 proof.
PreReqs: (Item 8)
Item 10 — [P1] — Patch nats-subjects + demote ZAPIER_USAGE (C5/C10)
Document verae.ts.* core-only; usage stream telemetry-only in STATUS/catalog.
PreReqs: (none)
Item 11 — [P1] — Automate D004 leaf billing deny (WP-D)
Tests prove leaf cannot balance.adjust; replace 404-only smoke.
PreReqs: (none)
Item 12 — [P1] — Userflow harness skeleton flows 1, 2, 4 (WP-P start)
MOCK_VERAE runnable report artifact under plans/qa/.
PreReqs: (Item 9)
Item 13 — [P1] — Extend harness to flows 3, 5, 6, 7
Async hook path, quote match, export consistency, leaf deny E2E; MG4.
PreReqs: (Item 12 + Item 11)
Item 14 — [P2] — Fix async receipt completedAt / completion wiring
Eliminate 0001-01-01 placeholders on product stamp path used by Zapier wait/hook.
PreReqs: (Item 12)
Item 15 — [P2] — Prepaid zero-balance policy gate
Define and test 402 vs allow-with-debt; seed funded fixture for harness.
PreReqs: (Item 7)
Item 16 — [P2] — Multi-host LIBSQL_URL topology (WP-O)
Only after MG3; cross-host meter/adjust/statement.
PreReqs: (Item 9)
Item 17 — [P2] — systemd/fleet hygiene for account-balance facade
Unit or documented fleet runtime; stable state dir until books retired.
PreReqs: (Item 8)
Item 18 — [P3] — Phase 0 leftover docs (C2/C3/C6, GETTING-STARTED)
Close arch full sign-off checklist items not covered by Item 2/10.
PreReqs: (Item 2 + Item 10)
```
**Dependency order (for “Do all”):**
Item 1 ∥ Item 2 ∥ Item 10 ∥ Item 11 → Item 3 → Item 4 → Item 5 → Item 6 → Item 7 → Item 8 → Item 9 → Item 12 → (Item 13 ∥ Item 14 ∥ Item 15) → Item 16 → Item 17; Item 18 can trail after Item 2+10.
**QA note:** Formal WP-Q churn should wait for **Item 9 (MG3)** + **Item 12**. This baseline pack is the readiness report that unlocks engineering focus on Items 1–9.
---
## Probe command index (reproducible)
```bash
ssh -i ~/.ssh/id_ed25519 -o IdentitiesOnly=yes marchon@70.88.205.138 \
'for p in 3040 3100 3130 3010 3021 3022 3023 3024 3025; do
curl -sS -m 5 http://127.0.0.1:$p/health; echo; done'
# Digest-only batch (do not use production customer keys outside lab seeds)
DIGEST=$(printf "qa-baseline" | sha256sum | awk "{print \$1}")
curl -sS -D - -X POST http://127.0.0.1:3040/v1/timestamps \
-H "content-type: application/json" -H "x-api-key: key-ada" \
-H "x-billing-channel: api" \
-d "{\"sha256s\":[\"$DIGEST\"]}"
# Dual-writer check (read-only)
sqlite3 /opt/verae/zappier-edge/billing.db "SELECT COUNT(*) FROM usage_entries;"
curl -sS http://127.0.0.1:3010/statement/cust_1
# blockchain.db: ls/stat only — never migrate/edit
```
---
*End of QA baseline. No product code was changed by this review.*