WIP snapshot before drive cleanup 2026-10-03

This commit is contained in:
George Lambert 2026-10-03 23:02:02 -04:00
commit e666e7bf3c
2 changed files with 2828 additions and 1 deletions

View file

@ -0,0 +1,220 @@
# Architecture Review — Verae × Zapier Multi-Host / libSQL Plan
**Plan under review:** `plans/2026-10-03-libsql-multihost-architecture.md`
**Reviewer role:** Architecture
**Date:** 2026-10-03
**Inputs:** plan; `/tmp/zapier-system-docs-map.md` (mirrored `plans/2026-10-03-system-docs-map.md`); D001–D006; catalog `composition.md`, `nats-subjects.md`, `overview/01-system.md`, `packages/verae-ops/GETTING-STARTED.md`; live 138 `/opt/verae/zappier-edge` (`billing.db` + `zappier.db`, `zappier-edge.service` on `127.0.0.1:3040`).
---
## Verdict
**approve-with-changes**
Direction is correct and aligned with **D006**: libSQL as business SoT, JetStream for work queues only, core NATS for `verae.ts.completed` / `verae.ts.receipt`, digests-only stamps, Go `blockchain.db` out of scope. The plan is **not** blocked on SoT choice.
It is **not** a clean approve: multi-host money durability, single-writer sequencing, stream/subject demotions, live port matrix, and decision-doc amendments are underspecified. Build of Phases 1–2 (interface + migrator skeleton) may start **after** the amendments in §Required plan amendments are merged into the plan (and D001 text is patched). Do **not** start Phase 3 rewire or Phase 4 multi-host split until those amendments are in the plan and the conflict register rows below are closed or explicitly deferred with owner + date.
---
## Expanded conflict register
Inherited from plan §3 (C1–C9) plus new findings from catalog + live 138.
| ID | Conflict | Severity | Resolution (required) |
|----|----------|----------|------------------------|
| C1 | D001 `books.json` SoT vs live `billing.db` vs D006 libSQL | High | **libSQL is SoT**; migrate books + billing.db; amend D001 file to point at D006 (not only the plan narrative) |
| C2 | README “imported, not yet wired” vs STATUS lab up | Med | STATUS + live doors = operational truth; patch README in Phase 0 |
| C3 | overview `zmw_` Bearer vs composition `x-api-key` | Med | Public = `x-api-key` (D002/composition); fix overview/models |
| C4 | Edge ports `:3000` / `:13000` / live `:3040` (`zap.verae.ai`) | High | Publish env matrix; lab doors proxy loopback; product edge remains `127.0.0.1:3040` until deliberately moved |
| C5 | JetStream `jobs.events` vs ban on JetStream for job-complete | High | **Different subjects:** `verae.zapier.jobs.events` (Zapier pipeline, JetStream OK) ≠ `verae.ts.*` (core NATS only). Document both in nats-subjects + standing rule |
| C6 | TimestampRequest “Hello world” data vs digests-only | Med | Digests-only + multihash batch; sync catalog models to live swagger |
| C7 | Dual prepaid writers (edge SQLite + account-balance) | High | One writer API on libSQL; edge caches only; deprecate/fold mailbox |
| C8 | Per-host SQLite on 3–5 machines | High | Forbidden for business data; libSQL shared |
| C9 | Go `blockchain.db` in scope creep | Blocker if violated | **Out of scope** — do not touch |
| **C10** | Catalog/STATUS stream **`ZAPIER_USAGE`** vs libSQL usage ledger | High | Demote `ZAPIER_USAGE` / `verae.zapier.usage` to **telemetry/export only** (or remove). Must never be treated as money/usage SoT. Plan §5 must say this explicitly |
| **C11** | Live dual middleware: catalog/ops `:3100` (MOCK, natsConnected=false) vs edge env `ZAPPIER_UPSTREAM=http://127.0.0.1:3130` (live mockVerae=false, natsConnected=true) | High | Env matrix must list both; product smoke uses **3130**; do not “fix” by pointing edge at 3100 without intent |
| **C12** | D001 decision file still says `books.json` is sole prepaid writer with no D006 amendment | High | Patch `decisions/D001-prepaid-sot.md` (addendum: durable store = libSQL; books.json lab legacy) |
| **C13** | Auth store undecided (`admin_users` / `portal_sessions` edge-local vs libSQL) | Med | **Decide before Phase 3:** v1 keep auth in edge-local `zappier.db` (or per-edge file); business tables only in libSQL. Document; revisit if edges are multi-host without sticky sessions |
| **C14** | Concurrent `usage_entries` hash-chain append across hosts | High (launch) | Single logical meter writer (one process or libSQL-serialized exclusive txn path). No competing multi-host `BEGIN IMMEDIATE` without proven libSQL semantics. Dual-run must not dual-append |
| **C15** | GETTING-STARTED / `01-system.md` still call account-balance prepaid SoT | Med | Phase 0 doc patch: SoT → libSQL; account-balance becomes facade or retired |
| **C16** | Live account-balance still up on `:3010` while edge also owns `billing.db` ledger | High | Cutover runbook must name which process is authoritative **each day** of dual-run; forbid two adjust paths |
| **C17** | Optional Litestream vs multi-writer temptation | Med | Litestream = backup/DR of **one** primary only (D006 non-goal). Plan must forbid Litestream-as-fabric |
| **C18** | Digests-only vs any remaining “upload file then stamp” Zapier creates | Med | Client hasher + digests-only batch remains the supported path; reject raw-data stamp examples in public docs |
---
## Risks for 3–5 machine launch
1. **Money split-brain (critical).** Today prepaid/ledger truth is fragmented (`books.json` via `:3010`, live `billing.db` on edge, caches). Splitting planes across hosts before one libSQL writer lands will diverge balances, statements, and CS credits. Mitigate: no Phase 4 host split until Phase 3 removes second writers and dual-run is read-only fallback.
2. **Hash-chain integrity under concurrency (critical).** `usage_entries` depends on tip-linked `prev_hash`/`row_hash`. Multiple edge replicas opening libSQL without an exclusive append protocol will fork or fail the chain. Mitigate: one meter writer service (or proven libSQL write serialization + contract tests for concurrent append).
3. **libSQL HA underspecified (high).** Plan says “dedicated host or primary of 3–5” but not: primary placement, replica/read role, failover RPO/RTO, backup (Litestream or native), network ACL, and what happens when libSQL is down (fail closed on meter vs stale cache). For launch, pick **one primary + backup**; do not invent multi-primary.
4. **JetStream mistaken for ledger (high).** `ZAPIER_USAGE` already exists in STATUS with replicas=3. Operators/agents will “persist usage on JetStream” under reliability pressure. Standing rule + plan §5 must demote it; CI/docs grep for usage-as-SoT.
5. **Job-complete subject confusion (high).** Zapier pipeline `verae.zapier.jobs.events` (JetStream) vs Verae chain notify `verae.ts.completed`/`verae.ts.receipt` (core only). A “make complete more reliable” change must not JetStream `verae.ts.*` (standing rule). Plan C5 is correct; nats-subjects.md still omits `verae.ts.*` — patch it.
6. **Wrong upstream in smoke (medium).** Edge points at `:3130`; lab docs emphasize `:3100`. Multi-host deploy scripts that copy catalog defaults will meter against mock middleware and lie about chain connectivity.
7. **Auth sticky-session / multi-edge (medium).** If auth stays in local `zappier.db` and two edges serve portal/admin, sessions break. Acceptable for v1 single edge; call out as launch constraint.
8. **Migration tip verification false confidence (medium).** Lab `usage_entries` tip is tiny (observed count=1 on 138). Migrator tests need adversarial fixtures (long chain, tamper mid-chain, empty DB), not only live copy.
9. **Deploy rsync DB wipe (medium, known).** Standing ops: never rsync `*.db*` onto `/opt/verae/zappier-edge`. libSQL cutover must use migrator + URL switch, not file copy into prod workdir via deploy.
10. **NATS billing request-reply after mailbox retirement (medium).** `verae.billing.*` subjects remain useful as a facade; if account-balance dies without a replacement listener, CS/sales/portal statement paths break even if libSQL is healthy. Plan must keep a single billing facade process.
11. **Credits vs cents naming (low).** Product credits; columns `*_cents`. Fine if documented; do not “fix” by rewriting historical rows mid-migration.
12. **Go timestamping DB proximity (low but non-negotiable).** Same host may hold `/var/lib/timestamping-svc/data/blockchain.db`. Automation must path-allowlist business DBs only.
---
## Required plan amendments (concrete text)
Merge the following into `plans/2026-10-03-libsql-multihost-architecture.md` (and patch D001 as noted).
### A1 — Amend Global Constraints (add bullets)
```text
- ZAPIER_USAGE / verae.zapier.usage is telemetry or export only — never usage/money SoT.
- Litestream (if used) backs up a single libSQL primary; never multi-writer fabric.
- Single meter/prepaid writer process owns usage_entries append + balance adjust;
other hosts call that API or use libSQL only through that writer’s transactions.
- Dual-run: libSQL primary for writes; file/SQLite fallback is read-only (or kill-switched).
- Auth (admin_users, portal_sessions): v1 remains edge-local zappier.db; not in verae_business.
- Product edge bind on 138: 127.0.0.1:3040 (zap.verae.ai); ZAPPIER_UPSTREAM defaults to
http://127.0.0.1:3130 for live lab smoke (not :3100 mock unless MOCK path intentional).
- Catalog docs that call account-balance “prepaid SoT” are stale after D006; Phase 0 patches them.
```
### A2 — Expand §5 NATS matrix (replace/extend rows)
```text
| verae.zapier.usage / ZAPIER_USAGE | JetStream optional | Telemetry/export only; NOT ledger SoT |
| verae.ts.completed, verae.ts.receipt | Core NATS only | Chain job-complete notify; never JetStream |
| verae.billing.* | Core request-reply (+ optional pub) | Facade over libSQL BusinessStore; one listener |
```
Also add to Phase 0: patch `docs/02-architecture/nats-subjects.md` with an explicit **Non-streams** section listing `verae.ts.*` as core-only and `ZAPIER_USAGE` as non-SoT.
### A3 — New §4.1 libSQL topology (insert after §4)
```text
### 4.1 Launch topology (3–5 machines)
- One libSQL primary (dedicated host or named primary among the 3–5).
- Replicas if any are read-only; writers always target primary URL (`LIBSQL_URL`).
- Backup: filesystem snapshot and/or Litestream of the primary only; tested restore before Phase 4.
- Fail closed: metered stamp paths return 503 if BusinessStore write unavailable
(no silent local SQLite fallback in production).
- Network: libSQL listen on private fabric only (same class as NATS 10.10.10.0/24); not public.
- Schema name: `verae_business` (confirm); auth schema stays out for v1.
```
### A4 — Phase 2 dual-run clarification
```text
- Writes: libSQL only once cutover flag BUSINESS_STORE=libsql is on.
- Fallback flag enables read-only SQLite/books for diagnosis, not adjust/recordUsage.
- Migrator verifies usage chain pre/post; refuses cutover if verifyChain fails.
- Account-balance either (a) becomes libSQL-backed facade for verae.billing.*, or
(b) is stopped after edge/CS/sales talk HTTP BusinessStore — pick one in Phase 3 tasking.
```
### A5 — Patch decision file D001 (Phase 0, same PR as plan amend)
Append to `decisions/D001-prepaid-sot.md`:
```text
## Addendum (2026-10-03, D006)
Durable prepaid/credits/usage/payments store is shared **libSQL**, not per-process
`books.json` or per-host `billing.db`. Single writer path still required. Edge
`balanceCents` remains a cache. Lab `books.json` / `billing.db` are migration sources.
```
### A6 — Conflict register: add C10–C18
Copy the new rows from this review’s register into plan §3.
### A7 — Acceptance criteria additions
```text
9. ZAPIER_USAGE is documented/configured as non-SoT; no service writes authoritative balances there.
10. Production metered paths fail closed if libSQL/BusinessStore writer is down.
11. Concurrent append contract test proves single-writer (or serialized) hash-chain safety.
12. Live env matrix documents :3040 edge, :3130 product upstream, :3100 mock, :3010 facade/retire.
13. D001 file contains D006 addendum; GETTING-STARTED / 01-system no longer call books.json SoT.
```
### A8 — Out of scope (reinforce)
```text
- Any JetStream binding for verae.ts.completed / verae.ts.receipt
- Editing /var/lib/timestamping-svc/data/blockchain.db or timestamping-svc JWT test results.db
- Multi-primary libSQL or Litestream multi-writer
- Moving portal auth to libSQL in the same cutover as billing (defer)
```
---
## What must not change (preserve list)
| Preserve | Why |
|----------|-----|
| **libSQL = business SoT** (D006) | Launch invariant for 3–5 hosts |
| **JetStream = work/archive only** | Not money/CRM database |
| **Core NATS only for `verae.ts.completed` / `verae.ts.receipt`** | Standing rule; no “reliability” override without explicit user words |
| **Digests-only stamp submit** (batch multihash / client hasher) | Live swagger + MCP; no file-byte stamp SoT |
| **No Go timestamping-svc SQLite edits** | `blockchain.db` and related stay local to that service |
| **D001 single prepaid writer path** (store upgraded by D006) | Prevents dual adjust |
| **D002** public `x-api-key`; `veraeUserId` join; JWT as `tokenRef` | Composition ownership |
| **D003** access planes as only public HTTP doors | Edge admin/Stripe stay non-door (loopback / tunnel) |
| **D004** leaf cannot touch billing | Policy allow-list |
| **D005** lan-134 off unless explicitly enabled | Fleet safety |
| **Credits product language**; ledger may keep `*_cents` column names | Avoid migration churn |
| **WAL (or libSQL equivalent) durability** on SQL stores | Existing edge invariant |
| **Deploy never rsyncs live `*.db*`** over production data | Known SQLITE_CANTOPEN / wipe failure mode |
| **Zapier never on NATS** | Hard rule in `01-system.md` |
| **Payment invoices ≠ timestamp receipts** | Composition |
| **Composition ownership:** metering/portal on zappier; stamp/NATS/hooks on middleware | Do not re-merge Stripe into middleware |
| **Existing zappier-edge Jest suite stays green** during driver swap | Regression gate |
| **Channel×segment rate card + credit headers** on metered/batch paths | Live product behavior |
---
## Sign-off checklist
Architecture signs when all boxes are true (or explicitly waived in writing on this file).
- [ ] Verdict understood: **approve-with-changes** (not full approve).
- [x] Plan amended with **A1–A8** (merged 2026-10-03) (or equivalent text merged).
- [x] Conflict register includes **C10–C18** with owners.
- [x] **D001** addendum committed pointing durable store at libSQL / D006.
- [ ] **nats-subjects.md** documents `verae.ts.*` core-only and `ZAPIER_USAGE` non-SoT.
- [ ] Env matrix published: `:3040` edge, `:3130` product MW, `:3100` mock MW, `:3010` balance facade.
- [ ] Single-writer / fail-closed / Litestream-backup-only rules in plan.
- [ ] Auth v1 = edge-local decision recorded (C13).
- [ ] Preserve list acknowledged by PM (no JetStream job-complete; no Go DB; digests-only).
- [ ] Phase 3/4 gated on writer consolidation + migrator chain verify.
- [ ] QA/PM may proceed to board + Task 1 **after** A1–A8 land; not before.
**Architecture provisional sign-off:** approve-with-changes — **A1–A8 merged into plan + D001**; remaining checklist items still open (nats-subjects, env matrix, GETTING-STARTED).
**Full architecture sign-off:** withheld until checklist complete.
---
## Evidence notes (live 138, 2026-10-03)
- `/opt/verae/zappier-edge`: `billing.db` + `zappier.db` (+ WAL/SHM), owner `verae-zapier`; service **active**.
- Edge env: `PORT=3040`, `HOST=127.0.0.1`, `ZAPPIER_UPSTREAM=http://127.0.0.1:3130`.
- Health: `:3040` zappier-edge ok; `:3100` middleware `mockVerae=true`, `natsConnected=false`; `:3130` middleware `mockVerae=false`, `natsConnected=true`; `:3010` account-balance still serving.
- `billing.db` tables: customers, usage_entries, channel_rates*, invoices*, price_endpoints, tiers; `zappier.db`: admin_users, portal_sessions.
- Access planes listening on `:3021–3025` (+ staff IAM/session); NATS client ports present on private/host binds.
---
*End of architecture review.*

View file

@ -0,0 +1,511 @@
# PM Board — Verae × Zapier Multi-Host libSQL Cutover
**Role:** Project Manager
**Date:** 2026-10-03
**Plan:** [`../2026-10-03-libsql-multihost-architecture.md`](../2026-10-03-libsql-multihost-architecture.md)
**Decision:** [`../../decisions/D006-libsql-business-jetstream-work.md`](../../decisions/D006-libsql-business-jetstream-work.md)
**Docs map:** `/tmp/zapier-system-docs-map.md` (also mirrored under `plans/2026-10-03-system-docs-map.md`)
**Forgejo org:** `https://git.georgelambert.org/marchon/`
**Clone base (lab worktrees):** `/Users/marchon/apps/verae/<repo>`
**Standing rules (non-negotiable):**
- Business SoT = **libSQL only** (D006).
- JetStream = work queues only (`ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, archive).
- `verae.ts.completed` / `verae.ts.receipt` = **core NATS only** (never JetStream).
- Do **not** touch Go timestamping-svc SQLite (`blockchain.db`, jwt_auth test DBs).
- Deploy never rsyncs live `*.db*` over production data.
- Digests-only stamp submit (multihash batch).
---
## 1. Constraint statement (TOC)
> The system constraint for the multi-host cutover is **shared business SoT cutover (libSQL + single prepaid writer path)** because every critical commercial userflow (meter charge, CS credit adjust, sales quote→meter match, portal statement, accounting export, leaf billing deny) blocks on one consistent customer / balance / usage / pricing store; JetStream workers and HTTPS access doors already exist in the NS1 lab and are not the bottleneck.
**Critical chain (serial):**
Conflict freeze (Phase 0) → BusinessStore + libSQL skeleton (Phase 1) → Migrate lab data (Phase 2) → **MERGE GATE: writers** → Rewire edge/CS/sales/acct (Phase 3) → Multi-host topology (Phase 4) → Userflow E2E + gates (Phase 5) → QA whattochange loop (Phase 6) → **PROJECT BUFFER: double userflow pass**.
**Feeding chains (parallel into merge gates):**
Docs/conflict patches (C2/C3/C5/C6); NATS subject clarification (`verae.zapier.jobs.events` ≠ `verae.ts.*`); digests-only catalog sync; access-plane smoke; Forgejo clone inventory; libSQL server/ops bring-up; leaf D004 deny tests.
**Exploit the constraint:** Do not start writer rewires until BusinessStore contract + migrator + dual-run reads are green. Do not split hosts until single-writer path is proven on one machine against libSQL.
---
## 2. Mermaid DAG of phases with merge gates
```mermaid
flowchart TD
subgraph feed0["Week 0–1 feeding (parallel)"]
F0a["WP-A Docs & conflict freeze<br/>C2/C3/C5/C6"]
F0b["WP-B NATS subject clarification<br/>jobs.events vs verae.ts.*"]
F0c["WP-C Digests-only docs/models sync"]
F0d["WP-D Access-plane smoke + D004 leaf"]
F0e["WP-E Forgejo clone inventory + worktrees"]
end
MG0["MERGE GATE 0<br/>Arch review signed + conflicts frozen"]
subgraph crit1["Critical chain — BusinessStore"]
WP_F["WP-F BusinessStore interface + contract tests"]
WP_G["WP-G SqliteFileBusinessStore adapter"]
WP_H["WP-H LibsqlBusinessStore + CI libSQL"]
end
MG1["MERGE GATE 1<br/>Contract green on both drivers"]
subgraph feed1["Feeding into migrate (parallel)"]
WP_I["WP-I libSQL server/ops (verae-ops)"]
WP_J["WP-J Fixture harvest billing.db + books.json"]
end
WP_K["WP-K Migrator + chain verify pre/post"]
MG2["MERGE GATE 2<br/>Dual-run reads OK; NO writer rewire yet"]
subgraph crit3["Critical chain — single writer"]
WP_L["WP-L Rewire zappier-edge meter/ledger → BusinessStore"]
WP_M["WP-M Rewire account-balance → libSQL / fold API"]
WP_N["WP-N Rewire CS / sales / accounting-export"]
end
MG3["MERGE GATE 3<br/>One prepaid writer; second paths removed"]
WP_O["WP-O Multi-host topology LIBSQL_URL"]
WP_P["WP-P Userflow E2E harness 1–7"]
MG4["MERGE GATE 4<br/>Unit+integration+gates green"]
WP_Q["WP-Q QA whattochange loop"]
PB["PROJECT BUFFER<br/>Userflows 1–7 pass twice consecutively"]
F0a --> MG0
F0b --> MG0
F0c --> MG0
F0d --> MG0
F0e --> MG0
MG0 --> WP_F --> WP_G --> WP_H --> MG1
WP_I --> MG1
WP_J --> MG1
MG1 --> WP_K --> MG2
MG2 --> WP_L --> WP_M --> WP_N --> MG3
MG3 --> WP_O --> WP_P --> MG4
MG4 --> WP_Q --> PB
```
**Artifact edges (what must exist before the next node consumes it):**
| From → To | Required artifact |
|-----------|-------------------|
| Feed0 → MG0 | `plans/reviews/2026-10-03-arch-review.md` PASS + conflict patches merged |
| MG0 → WP-F | Frozen conflict register C1–C9; this PM board |
| WP-H → MG1 | Jest contract suite green for SqliteFile + Libsql drivers |
| WP-I → MG1 | Reachable `LIBSQL_URL` (lab docker or host) documented in ops |
| WP-K → MG2 | Migrator report: row counts, chain tip hash match, dual-read flag |
| MG2 → WP-L | Explicit PM/Orchestrator go-ahead: “writers unlocked” |
| MG3 → WP-O | Grep proof: no production writes to local `billing.db`/`books.json` SoT |
| WP-P → MG4 | Harness report for flows 1–7 (MOCK_VERAE) |
| WP-Q → PB | Two consecutive full green userflow runs |
---
## 3. Engineer work packages
Clone convention for every WP:
```bash
ORG=https://git.georgelambert.org/marchon
BASE=/Users/marchon/apps/verae
# Prefer existing clone; else:
git clone "$ORG/<repo>.git" "$BASE/<repo>"
```
Commit messages must cite: `plans/2026-10-03-libsql-multihost-architecture.md` and this board (`plans/reviews/2026-10-03-pm-board.md`).
---
### WP-A — Docs & conflict freeze (Phase 0)
| Field | Content |
|-------|---------|
| **PreReqs** | Architecture plan published; D006 accepted |
| **Mode** | Parallel with WP-B/C/D/E |
| **Repos to clone** | `zapier-decisions`, `overview`, `zapier-docs-master` (catalog mirror of docs-master), optionally site tree if editing published catalog |
| **Files to touch** | `zapier-decisions/STATUS.md`, `LOG.md`, `TODO.md`; `overview/README.md` / composition status blurb; `overview` auth notes that still say `zmw_` Bearer (C3); `docs/02-architecture` equivalents in catalog if present under docs-master; ensure C5 wording: JetStream `verae.zapier.jobs.events` ≠ core `verae.ts.*` |
| **Definition of done** | Conflicts C2, C3, C5, C6 closed in writing; C1/C7/C8 restated as “libSQL SoT / single writer”; C9 “out of scope” reiterated; arch-review placeholder path ready for Architect sign-off |
| **Test commands** | Doc lint only: `rg -n 'zmw_|books\\.json is sole|JetStream.*verae\\.ts' overview zapier-docs-master zapier-decisions` — residual false claims listed and fixed or explicitly deferred with why |
---
### WP-B — NATS subject clarification
| Field | Content |
|-------|---------|
| **PreReqs** | None (feeding) |
| **Mode** | Parallel |
| **Repos** | `zapier-decisions`, `verae-middleware`, `verae-nats-cluster`, `zapier-docs-master` |
| **Files** | `verae-middleware/src/nats/subjects.js`; `verae-middleware/NATS.md`; `zapier-docs-master` / architecture `nats-subjects` copies; `verae-nats-cluster` stream list docs; `zapier-decisions/NATS.md` |
| **DoD** | Matrix matches architecture §5: JetStream for `verae.zapier.jobs.{watch,events}`, `webhooks.deliver`, archive; **core only** for `verae.ts.completed` / `verae.ts.receipt`; billing `verae.billing.*` remains core request-reply |
| **Tests** | `cd verae-middleware && npm run test:unit`; `rg -n 'verae\\.ts\\.(completed|receipt)' verae-middleware` must not show JetStream stream binds; `cd verae-nats-cluster && npm run status` (lab) — confirm product streams list does **not** include timestamp job-complete |
---
### WP-C — Digests-only docs / models sync
| Field | Content |
|-------|---------|
| **PreReqs** | None |
| **Mode** | Parallel |
| **Repos** | `zapier-docs-master`, `verae-middleware`, `verae-zapier-app`, `zapier-decisions` |
| **Files** | Catalog models / OpenAPI examples that still show “Hello world” data; `verae-middleware` route docs for batch timestamp; Zapier create field descriptions |
| **DoD** | All public stamp examples are multihash digests; no plaintext body-as-data examples in SoT docs |
| **Tests** | `cd verae-middleware && npm run test:hash` (and batch-related unit tests if present); `rg -n 'Hello world' docs models openapi` → zero in stamp request examples |
---
### WP-D — Access-plane smoke + D004 leaf deny
| Field | Content |
|-------|---------|
| **PreReqs** | Lab doors up (STATUS) preferred; else local unit tests |
| **Mode** | Parallel |
| **Repos** | `verae-access-leaf`, `verae-access-web`, `verae-access-api`, `verae-access-zapier`, `verae-access-authz`, `zappier-identity` |
| **Files** | Leaf allow-list / policy tests under each `test/`; authz check paths |
| **DoD** | Automated proof leaf cannot call `verae.billing.balance.adjust` / prepaid adjust HTTP; web/api/zapier smoke health |
| **Tests** | `cd verae-access-leaf && npm test`; `cd verae-access-authz && npm test`; `cd verae-access-web && npm test`; same for api/zapier if scripts exist |
---
### WP-E — Forgejo clone inventory + worktrees
| Field | Content |
|-------|---------|
| **PreReqs** | None |
| **Mode** | Parallel |
| **Repos** | `verae-bootstrap` (source of `repos.txt`) |
| **Files** | Produce checklist under `zapier-decisions/plans/reviews/clone-inventory-2026-10-03.md` (or append section to this board’s §7); do not invent new repos |
| **DoD** | Every WP’s clone list verified present under `/Users/marchon/apps/verae` or cloned; `repos.txt` vs WP matrix diff empty for in-scope modules |
| **Tests** | `while read r; do test -d /Users/marchon/apps/verae/$r/.git || echo MISSING $r; done < verae-bootstrap/repos.txt` |
---
### WP-F — BusinessStore interface + failing contract tests (Phase 1 start)
| Field | Content |
|-------|---------|
| **PreReqs** | **MG0 PASS** |
| **Mode** | Serial critical chain |
| **Repos** | `zappier-edge` |
| **Files** | **Create** `src/db/business-store.ts` (interface); **Create** `tests/business-store.contract.test.ts`; may re-export types from `src/usage.ts`, `src/db/customer-repo.ts`, `src/db/pricing-store.ts`, `src/db/invoice-repo.ts` |
| **Interface minimum** | `findCustomer`, `saveCustomer`, `recordUsage`, `verifyUsageChain`, `listUsage`, `getBalance`, `adjustBalance`, `getRates` / `resolveChannelRate`, `listInvoices` / `saveInvoice` |
| **DoD** | Contract tests exist and **fail** against missing impl (red); methods documented; hash-chain fields (`prev_hash`, `row_hash`) required even if current `SqliteUsageRepo` lacks them (extend schema in WP-G) |
| **Tests** | `cd zappier-edge && npm test -- tests/business-store.contract.test.ts` → expect fail |
---
### WP-G — SqliteFileBusinessStore (dev/lab adapter)
| Field | Content |
|-------|---------|
| **PreReqs** | WP-F red tests exist |
| **Mode** | Serial |
| **Repos** | `zappier-edge` |
| **Files** | **Create** `src/db/sqlite-file-business-store.ts`; adapt `src/db/usage-repo.ts` (add hash chain columns + verify); wire factory in `src/index.ts` / `src/app.ts` behind flag `BUSINESS_STORE=sqlite-file` (default keep current behavior until WP-L) |
| **DoD** | Contract tests **pass** on file/memory SQLite; existing Jest suite still green |
| **Tests** | `cd zappier-edge && npm test -- tests/business-store.contract.test.ts`; `cd zappier-edge && npm test` |
---
### WP-H — LibsqlBusinessStore + CI
| Field | Content |
|-------|---------|
| **PreReqs** | WP-G green |
| **Mode** | Serial (can overlap WP-I ops once interface stable) |
| **Repos** | `zappier-edge` |
| **Files** | **Create** `src/db/libsql-business-store.ts`; `package.json` add `@libsql/client`; `.env.example` `LIBSQL_URL`, `LIBSQL_AUTH_TOKEN`; Jest config for libsql file URL or embedded |
| **DoD** | Same contract suite passes with `BUSINESS_STORE=libsql` and `LIBSQL_URL=file:…` (or testcontainers); WAL/durable semantics documented |
| **Tests** | `cd zappier-edge && BUSINESS_STORE=libsql LIBSQL_URL=file:./tmp-business.db npm test -- tests/business-store.contract.test.ts`; full `npm test` |
---
### WP-I — libSQL server / ops bring-up
| Field | Content |
|-------|---------|
| **PreReqs** | None for draft runbook; MG1 needs live URL |
| **Mode** | Parallel feeding into MG1 |
| **Repos** | `verae-ops`, `verae-deploy`, `verae-nats-cluster` (topology reference only), `zapier-decisions` |
| **Files** | New runbook under `verae-ops` (e.g. `libsql.md` / section in `GETTING-STARTED.md`); env matrix for 3–5 hosts; **do not** relocate Go DBs |
| **DoD** | Documented `LIBSQL_URL` for lab; backup note (Litestream optional for primary file — not multi-writer); firewall private-only |
| **Tests** | Manual: `curl`/client ping create table + select; record connection string in STATUS |
---
### WP-J — Fixture harvest (billing.db + books.json)
| Field | Content |
|-------|---------|
| **PreReqs** | Read-only access to lab 138 paths (or sanitized copies) |
| **Mode** | Parallel into MG1 |
| **Repos** | `zappier-edge`, `zappier-account-balance`, `zapier-decisions` |
| **Files** | **Create** `zappier-edge/tests/fixtures/` (sanitized); `scripts/export-lab-business-fixtures.md` instructions; never commit live secrets/API keys |
| **Sources** | Live stand-ins: `/opt/verae/zappier-edge/billing.db` (BILLING_DB), `zappier.db` (auth may stay local), account-balance `books.json`; seed keys in lab only |
| **DoD** | Anonymized fixtures with known chain tip + prepaid balances for migrator tests |
| **Tests** | Fixture checksums documented; `sqlite3 fixture.db 'select count(*) from usage_entries;'` |
---
### WP-K — Migrator billing.db + books.json → libSQL (Phase 2)
| Field | Content |
|-------|---------|
| **PreReqs** | **MG1 PASS**; fixtures from WP-J; `LIBSQL_URL` from WP-I |
| **Mode** | Serial |
| **Repos** | `zappier-edge`, `zappier-account-balance` |
| **Files** | **Create** `zappier-edge/scripts/migrate-to-libsql.ts` (or `.mjs`); **Create** `tests/migrate-to-libsql.test.ts`; map tables: customers, tiers, price_endpoints, channel_rates*, usage_entries(+hashes), invoices*, balances/credits/payments from books |
| **DoD** | One-shot migrator; `verifyUsageChain` true before and after; dual-run read path flag `BUSINESS_READ_PRIMARY=libsql` with file fallback; **no production writer cutover yet** |
| **Tests** | `cd zappier-edge && npm test -- tests/migrate-to-libsql.test.ts`; dry-run against fixtures; print before/after tip hashes |
---
### MERGE GATE 2 — Writers still locked
Engineers **must not** start WP-L/M/N until Orchestrator/PM records:
```
MERGE GATE 2: PASS
- migrator tip hashes match
- dual-run reads validated
- writers unlocked: yes
```
---
### WP-L — Rewire zappier-edge meter / ledger → BusinessStore (Phase 3)
| Field | Content |
|-------|---------|
| **PreReqs** | **MG2 PASS** |
| **Mode** | Serial (constraint path) |
| **Repos** | `zappier-edge` |
| **Files** | `src/meter.ts`, `src/usage.ts`, `src/ledger.ts`, `src/billing-nats.ts`, `src/credits.ts`, `src/portal.ts`, `src/pricing.ts`, `src/invoicing.ts`, `src/db/*-repo.ts`, `src/index.ts`, `src/app.ts` — inject BusinessStore; edge `balanceCents` remains cache filled from store/billing API |
| **DoD** | Meter appends usage **only** via BusinessStore; prepaid adjust goes through single writer API; no direct better-sqlite3 business writes outside adapter |
| **Tests** | `cd zappier-edge && npm test`; focus `tests/meter.test.ts`, `tests/usage.test.ts`, `tests/credit.test.ts`, `tests/portal.test.ts`, `tests/db-*.test.ts`, `tests/billing-delta.test.ts` |
---
### WP-M — Rewire account-balance (books.json → libSQL)
| Field | Content |
|-------|---------|
| **PreReqs** | WP-L in progress or done (coordinate subjects) |
| **Mode** | Serial after WP-L starts (shared `verae.billing.*` contracts) |
| **Repos** | `zappier-account-balance`, `zappier-edge` |
| **Files** | `zappier-account-balance/src/persist.js`, `books.js`, `server.js`, `subjects.js`; tests under `test/`; README SoT language → libSQL |
| **DoD** | NATS `verae.billing.balance.*` / statement backed by libSQL BusinessStore or HTTP billing API; `books.json` deprecated (import-only); D001 single writer preserved |
| **Tests** | `cd zappier-account-balance && npm test`; integration: adjust → edge statement sees same balance |
---
### WP-N — Rewire CS / sales / accounting-export
| Field | Content |
|-------|---------|
| **PreReqs** | WP-M billing subjects stable |
| **Mode** | Parallel with each other **after** WP-M contract freeze; else serial behind WP-M |
| **Repos** | `zappier-customer-service`, `zappier-sales-pricing`, `zappier-accounting-export` |
| **Files** | Each package `src/nats-billing.js`, `src/server.js`; staff UI pages as needed |
| **DoD** | CS credit adjust → libSQL balance; sales channel×segment rates → same rate tables meter uses; accounting export reads payments/usage from libSQL |
| **Tests** | `cd zappier-customer-service && npm test`; `cd zappier-sales-pricing && npm test`; `cd zappier-accounting-export && npm test` |
---
### MERGE GATE 3 — Single writer proof
| Check | Pass criteria |
|-------|---------------|
| Grep | No code path treats `books.json` or per-host `billing.db` as production SoT |
| Runtime | One `LIBSQL_URL` serves customers/pricing/usage/invoices/balances |
| D004 | Leaf still cannot adjust billing |
| Regression | `zappier-edge` full Jest green |
---
### WP-O — Multi-host topology (Phase 4)
| Field | Content |
|-------|---------|
| **PreReqs** | **MG3 PASS** |
| **Mode** | Serial |
| **Repos** | `verae-ops`, `verae-deploy`, `verae-fleet`, `verae-keep`, `verae-bootstrap`, `zappier-edge`, `verae-middleware` |
| **Files** | Host env templates (`LIBSQL_URL`, `NATS_URL=nats://10.10.10.21:4222,...`); deploy scripts; STATUS topology section |
| **DoD** | libSQL on dedicated/primary host; edge/middleware/CS on separate hosts possible; JetStream workers unchanged; leaf cannot bill |
| **Tests** | Cross-host: meter on host A, CS adjust on host B, portal statement on host C — same balance; `verae-nats-cluster` `npm run status` |
---
### WP-P — Userflow E2E harness (Phase 5)
| Field | Content |
|-------|---------|
| **PreReqs** | MG3; WP-O preferred for multi-host proof, MOCK_VERAE OK for first pass |
| **Mode** | Serial into MG4 |
| **Repos** | `verae-zapier-simulator`, `zappier-edge`, `verae-middleware`, `verae-access-web`, `verae-access-zapier`, `verae-access-leaf` |
| **Files** | Extend simulator/`harness` under simulator or `packages`-style tests; scripts for flows 1–7 from architecture §1 |
| **Flows** | (1) Signup→API key→Statement (2) Stamp digests single+batch + credit headers (3) Async complete→REST Hook via jobs.watch→events→webhooks.deliver (4) CS credit→portal balance (5) Sales quote→meter match (6) Accounting export consistency (7) Leaf stamp OK / billing deny |
| **DoD** | Automated report artifact under `zapier-decisions/plans/qa/` |
| **Tests** | `cd verae-zapier-simulator && npm test`; middleware `npm test` + selected `test:gate*`; edge `npm test`; flow runner script (to be added) exit 0 |
---
### WP-Q — QA whattochange loop (Phase 6)
| Field | Content |
|-------|---------|
| **PreReqs** | **MG4 PASS** (suite green once) |
| **Mode** | Loop until project buffer |
| **Repos** | `zapier-decisions` + whichever repos bugs land in |
| **Files** | `plans/qa/<date>-whattochange/` packs; `NotImplemented/` leftovers per recommended-changes skill |
| **DoD** | Each bug → a/b/c/d + PreReq todos; engineers implement; re-run WP-P |
| **Tests** | Full suite + userflows after each pack |
---
## 4. Parallel vs serial schedule — first 2 weeks
Assume agent-days; calendar = 10 working days. Critical chain owns the constraint; feeding work never blocks writers early.
| Day | Parallel (feeding) | Serial (critical chain) | Gate |
|-----|--------------------|-------------------------|------|
| **1** | WP-A, WP-B, WP-C, WP-D, WP-E kickoff | — | Aim MG0 EOD day 2 |
| **2** | Finish A–E; Architect signs `2026-10-03-arch-review.md` | — | **MG0** |
| **3** | WP-I ops draft; WP-J fixture planning | **WP-F** interface + red tests | |
| **4** | WP-I lab libSQL up; WP-J sanitize fixtures | **WP-G** SqliteFile adapter | |
| **5** | WP-I harden docs; fixture finalize | **WP-H** LibsqlBusinessStore | **MG1** target EOD day 5–6 |
| **6** | Buffer if MG1 slip | **WP-K** migrator TDD | |
| **7** | — | Finish WP-K; dual-run reads | **MG2** (writers unlock) |
| **8** | — | **WP-L** edge rewire | |
| **9** | WP-N prep (read-only) | **WP-M** account-balance | |
| **10** | **WP-N** CS/sales/acct (parallel agents) | Finish L/M integration | **MG3** stretch; else day 11 |
**Week 2 overflow (if MG3 slips):** do **not** start WP-O multi-host. Burn down writer bugs; keep project buffer intact.
**Week 2 if on track (days 11–14):** WP-O topology → WP-P harness → MG4 → start WP-Q.
**Explicit non-goals in first 2 weeks:** live `MOCK_VERAE=false` seal proof; Zapier Platform public push; lan-134; Go DB moves; JetStream for `verae.ts.*`.
---
## 5. QA entry criteria
QA (WP-Q / userflow break attempts) may begin only when **all** of the following are true:
1. **MG3 PASS** — single prepaid writer on libSQL; no dual SoT writes.
2. **Regression green:** `zappier-edge` `npm test`; `zappier-account-balance` `npm test`; CS/sales/acct `npm test`; `verae-middleware` `npm test` (unit + non-NATS integration at minimum).
3. **NATS rules verified:** product JetStream streams only for Zapier jobs/webhooks/archive; `verae.ts.completed` / `verae.ts.receipt` core-only (grep + cluster status).
4. **Harness exists (WP-P skeleton):** at least flows 1, 2, 4 runnable under `MOCK_VERAE=true`.
5. **D004 automated:** leaf billing adjust denied.
6. **Migrator evidence:** tip hash report archived under `plans/qa/`.
7. **Go timestamping DBs:** confirmation note that no PR touched them.
8. **Docs:** C2/C3/C5/C6 closed or explicitly ticketed with owner.
**QA exit (feeds project buffer):** whattochange packs filed for every failure; no silent skips; leftovers under `NotImplemented/` with why.
---
## 6. Buffer policy
### Merge gate before rewiring writers (MG2)
- **Hard stop:** WP-L / WP-M / WP-N are blocked until MG2 PASS.
- Rationale (Critical Chain feeding buffer): incomplete BusinessStore or bad migration hitting the meter writer destroys the only constraint we are exploiting (consistent SoT).
- Gate owner: PM / Orchestrator. Engineers may prepare branches but must not merge writer cutover.
### Other merge gates
| Gate | Absorbs variation from | Failure action |
|------|------------------------|----------------|
| MG0 | Docs/conflict feeding | Re-run WP-A/B/C; no Phase 1 code |
| MG1 | Driver + ops + fixtures | Fix contract/CI URL; no migrator on prod-like data |
| MG3 | Writer rewires | Roll back feature flag to SqliteFile/dual-read; do not split hosts |
| MG4 | E2E harness | Fix failures; do not declare QA loop “done” |
### Project buffer = double userflow pass
- After MG4 and WP-Q churn, **acceptance requires userflows 1–7 green twice consecutively** (architecture §10 / Phase 6 exit).
- First pass proves the system; second pass absorbs latent race/order bugs without consuming schedule fantasy.
- Do **not** spend project buffer on early scope (Platform push, live seal, Peergos redesign).
- If second pass fails, return to WP-Q; do not “waive” the second pass.
### Constraint buffer
- Before each critical-chain WP (F, G, H, K, L), feeding artifacts listed in §2 must be staged so the constraint agent never waits on clone/docs/URL discovery mid-implementation.
---
## 7. Explicit Forgejo repos each WP clones
Base URL: `https://git.georgelambert.org/marchon/<repo>.git`
| WP | Repos to clone |
|----|----------------|
| **WP-A** | `zapier-decisions`, `overview`, `zapier-docs-master` |
| **WP-B** | `zapier-decisions`, `verae-middleware`, `verae-nats-cluster`, `zapier-docs-master` |
| **WP-C** | `zapier-docs-master`, `verae-middleware`, `verae-zapier-app`, `zapier-decisions` |
| **WP-D** | `verae-access-leaf`, `verae-access-web`, `verae-access-api`, `verae-access-zapier`, `verae-access-authz`, `zappier-identity` |
| **WP-E** | `verae-bootstrap` (plus any MISSING from `repos.txt` needed by later WPs) |
| **WP-F** | `zappier-edge` |
| **WP-G** | `zappier-edge` |
| **WP-H** | `zappier-edge` |
| **WP-I** | `verae-ops`, `verae-deploy`, `zapier-decisions` |
| **WP-J** | `zappier-edge`, `zappier-account-balance`, `zapier-decisions` |
| **WP-K** | `zappier-edge`, `zappier-account-balance` |
| **WP-L** | `zappier-edge` |
| **WP-M** | `zappier-account-balance`, `zappier-edge` |
| **WP-N** | `zappier-customer-service`, `zappier-sales-pricing`, `zappier-accounting-export` |
| **WP-O** | `verae-ops`, `verae-deploy`, `verae-fleet`, `verae-keep`, `verae-bootstrap`, `zappier-edge`, `verae-middleware` |
| **WP-P** | `verae-zapier-simulator`, `zappier-edge`, `verae-middleware`, `verae-access-web`, `verae-access-zapier`, `verae-access-leaf` |
| **WP-Q** | `zapier-decisions` + bug-target repos as discovered |
### Union set for cutover agents (minimum clone list)
```
zapier-decisions
overview
zapier-docs-master
verae-bootstrap
verae-ops
verae-deploy
verae-fleet
verae-keep
verae-nats-cluster
zappier-edge
zappier-account-balance
zappier-customer-service
zappier-sales-pricing
zappier-accounting-export
zappier-identity
verae-middleware
verae-zapier-app
verae-zapier-simulator
verae-access-authz
verae-access-web
verae-access-api
verae-access-leaf
verae-access-zapier
```
**Out of clone scope for this cutover (do not pull for WPs above):** Go timestamping-svc repos, Peergos/Seaweed redesign trees, `verae-activate` (unless WP-C needs tiny-app copy sync).
---
## Orchestrator checklist (next actions)
- [ ] Architect produces `plans/reviews/2026-10-03-arch-review.md` → **MG0**
- [ ] Launch parallel agents: WP-A, WP-B, WP-C, WP-D, WP-E
- [ ] After MG0: engineer on WP-F → G → H; parallel WP-I, WP-J
- [ ] **Stop before writers** until MG2
- [ ] Enforce D006 + core-NATS job-complete on every PR review
---
*End of PM board. Planning artifact only — no implementation in this change.*