zapier-decisions/plans/reviews/2026-10-03-pm-board.md

25 KiB
Raw Blame History

PM Board — Verae × Zapier Multi-Host libSQL Cutover

Role: Project Manager
Date: 2026-10-03
Plan: ../2026-10-03-libsql-multihost-architecture.md
Decision: ../../decisions/D006-libsql-business-jetstream-work.md
Docs map: /tmp/zapier-system-docs-map.md (also mirrored under plans/2026-10-03-system-docs-map.md)
Forgejo org: https://git.georgelambert.org/marchon/
Clone base (lab worktrees): /Users/marchon/apps/verae/<repo>

Standing rules (non-negotiable):

  • Business SoT = libSQL only (D006).
  • JetStream = work queues only (ZAPIER_JOBS, ZAPIER_EVENTS, ZAPIER_WEBHOOKS, archive).
  • verae.ts.completed / verae.ts.receipt = core NATS only (never JetStream).
  • Do not touch Go timestamping-svc SQLite (blockchain.db, jwt_auth test DBs).
  • Deploy never rsyncs live *.db* over production data.
  • Digests-only stamp submit (multihash batch).

1. Constraint statement (TOC)

The system constraint for the multi-host cutover is shared business SoT cutover (libSQL + single prepaid writer path) because every critical commercial userflow (meter charge, CS credit adjust, sales quote→meter match, portal statement, accounting export, leaf billing deny) blocks on one consistent customer / balance / usage / pricing store; JetStream workers and HTTPS access doors already exist in the NS1 lab and are not the bottleneck.

Critical chain (serial):
Conflict freeze (Phase 0) → BusinessStore + libSQL skeleton (Phase 1) → Migrate lab data (Phase 2) → MERGE GATE: writers → Rewire edge/CS/sales/acct (Phase 3) → Multi-host topology (Phase 4) → Userflow E2E + gates (Phase 5) → QA whattochange loop (Phase 6) → PROJECT BUFFER: double userflow pass.

Feeding chains (parallel into merge gates):
Docs/conflict patches (C2/C3/C5/C6); NATS subject clarification (verae.zapier.jobs.events ≠ verae.ts.*); digests-only catalog sync; access-plane smoke; Forgejo clone inventory; libSQL server/ops bring-up; leaf D004 deny tests.

Exploit the constraint: Do not start writer rewires until BusinessStore contract + migrator + dual-run reads are green. Do not split hosts until single-writer path is proven on one machine against libSQL.


2. Mermaid DAG of phases with merge gates

flowchart TD
  subgraph feed0["Week 0–1 feeding (parallel)"]
    F0a["WP-A Docs & conflict freeze<br/>C2/C3/C5/C6"]
    F0b["WP-B NATS subject clarification<br/>jobs.events vs verae.ts.*"]
    F0c["WP-C Digests-only docs/models sync"]
    F0d["WP-D Access-plane smoke + D004 leaf"]
    F0e["WP-E Forgejo clone inventory + worktrees"]
  end

  MG0["MERGE GATE 0<br/>Arch review signed + conflicts frozen"]

  subgraph crit1["Critical chain — BusinessStore"]
    WP_F["WP-F BusinessStore interface + contract tests"]
    WP_G["WP-G SqliteFileBusinessStore adapter"]
    WP_H["WP-H LibsqlBusinessStore + CI libSQL"]
  end

  MG1["MERGE GATE 1<br/>Contract green on both drivers"]

  subgraph feed1["Feeding into migrate (parallel)"]
    WP_I["WP-I libSQL server/ops (verae-ops)"]
    WP_J["WP-J Fixture harvest billing.db + books.json"]
  end

  WP_K["WP-K Migrator + chain verify pre/post"]
  MG2["MERGE GATE 2<br/>Dual-run reads OK; NO writer rewire yet"]

  subgraph crit3["Critical chain — single writer"]
    WP_L["WP-L Rewire zappier-edge meter/ledger → BusinessStore"]
    WP_M["WP-M Rewire account-balance → libSQL / fold API"]
    WP_N["WP-N Rewire CS / sales / accounting-export"]
  end

  MG3["MERGE GATE 3<br/>One prepaid writer; second paths removed"]

  WP_O["WP-O Multi-host topology LIBSQL_URL"]
  WP_P["WP-P Userflow E2E harness 1–7"]
  MG4["MERGE GATE 4<br/>Unit+integration+gates green"]

  WP_Q["WP-Q QA whattochange loop"]
  PB["PROJECT BUFFER<br/>Userflows 1–7 pass twice consecutively"]

  F0a --> MG0
  F0b --> MG0
  F0c --> MG0
  F0d --> MG0
  F0e --> MG0
  MG0 --> WP_F --> WP_G --> WP_H --> MG1
  WP_I --> MG1
  WP_J --> MG1
  MG1 --> WP_K --> MG2
  MG2 --> WP_L --> WP_M --> WP_N --> MG3
  MG3 --> WP_O --> WP_P --> MG4
  MG4 --> WP_Q --> PB

Artifact edges (what must exist before the next node consumes it):

From → To Required artifact
Feed0 → MG0 plans/reviews/2026-10-03-arch-review.md PASS + conflict patches merged
MG0 → WP-F Frozen conflict register C1–C9; this PM board
WP-H → MG1 Jest contract suite green for SqliteFile + Libsql drivers
WP-I → MG1 Reachable LIBSQL_URL (lab docker or host) documented in ops
WP-K → MG2 Migrator report: row counts, chain tip hash match, dual-read flag
MG2 → WP-L Explicit PM/Orchestrator go-ahead: “writers unlocked”
MG3 → WP-O Grep proof: no production writes to local billing.db/books.json SoT
WP-P → MG4 Harness report for flows 1–7 (MOCK_VERAE)
WP-Q → PB Two consecutive full green userflow runs

3. Engineer work packages

Clone convention for every WP:

ORG=https://git.georgelambert.org/marchon
BASE=/Users/marchon/apps/verae
# Prefer existing clone; else:
git clone "$ORG/<repo>.git" "$BASE/<repo>"

Commit messages must cite: plans/2026-10-03-libsql-multihost-architecture.md and this board (plans/reviews/2026-10-03-pm-board.md).


WP-A — Docs & conflict freeze (Phase 0)

Field Content
PreReqs Architecture plan published; D006 accepted
Mode Parallel with WP-B/C/D/E
Repos to clone zapier-decisions, overview, zapier-docs-master (catalog mirror of docs-master), optionally site tree if editing published catalog
Files to touch zapier-decisions/STATUS.md, LOG.md, TODO.md; overview/README.md / composition status blurb; overview auth notes that still say zmw_ Bearer (C3); docs/02-architecture equivalents in catalog if present under docs-master; ensure C5 wording: JetStream verae.zapier.jobs.events ≠ core verae.ts.*
Definition of done Conflicts C2, C3, C5, C6 closed in writing; C1/C7/C8 restated as “libSQL SoT / single writer”; C9 “out of scope” reiterated; arch-review placeholder path ready for Architect sign-off
Test commands Doc lint only: `rg -n 'zmw_

WP-B — NATS subject clarification

Field Content
PreReqs None (feeding)
Mode Parallel
Repos zapier-decisions, verae-middleware, verae-nats-cluster, zapier-docs-master
Files verae-middleware/src/nats/subjects.js; verae-middleware/NATS.md; zapier-docs-master / architecture nats-subjects copies; verae-nats-cluster stream list docs; zapier-decisions/NATS.md
DoD Matrix matches architecture §5: JetStream for verae.zapier.jobs.{watch,events}, webhooks.deliver, archive; core only for verae.ts.completed / verae.ts.receipt; billing verae.billing.* remains core request-reply
Tests cd verae-middleware && npm run test:unit; `rg -n 'verae\.ts\.(completed

WP-C — Digests-only docs / models sync

Field Content
PreReqs None
Mode Parallel
Repos zapier-docs-master, verae-middleware, verae-zapier-app, zapier-decisions
Files Catalog models / OpenAPI examples that still show “Hello world” data; verae-middleware route docs for batch timestamp; Zapier create field descriptions
DoD All public stamp examples are multihash digests; no plaintext body-as-data examples in SoT docs
Tests cd verae-middleware && npm run test:hash (and batch-related unit tests if present); rg -n 'Hello world' docs models openapi → zero in stamp request examples

WP-D — Access-plane smoke + D004 leaf deny

Field Content
PreReqs Lab doors up (STATUS) preferred; else local unit tests
Mode Parallel
Repos verae-access-leaf, verae-access-web, verae-access-api, verae-access-zapier, verae-access-authz, zappier-identity
Files Leaf allow-list / policy tests under each test/; authz check paths
DoD Automated proof leaf cannot call verae.billing.balance.adjust / prepaid adjust HTTP; web/api/zapier smoke health
Tests cd verae-access-leaf && npm test; cd verae-access-authz && npm test; cd verae-access-web && npm test; same for api/zapier if scripts exist

WP-E — Forgejo clone inventory + worktrees

Field Content
PreReqs None
Mode Parallel
Repos verae-bootstrap (source of repos.txt)
Files Produce checklist under zapier-decisions/plans/reviews/clone-inventory-2026-10-03.md (or append section to this board’s §7); do not invent new repos
DoD Every WP’s clone list verified present under /Users/marchon/apps/verae or cloned; repos.txt vs WP matrix diff empty for in-scope modules
Tests `while read r; do test -d /Users/marchon/apps/verae/$r/.git

WP-F — BusinessStore interface + failing contract tests (Phase 1 start)

Field Content
PreReqs MG0 PASS
Mode Serial critical chain
Repos zappier-edge
Files Create src/db/business-store.ts (interface); Create tests/business-store.contract.test.ts; may re-export types from src/usage.ts, src/db/customer-repo.ts, src/db/pricing-store.ts, src/db/invoice-repo.ts
Interface minimum findCustomer, saveCustomer, recordUsage, verifyUsageChain, listUsage, getBalance, adjustBalance, getRates / resolveChannelRate, listInvoices / saveInvoice
DoD Contract tests exist and fail against missing impl (red); methods documented; hash-chain fields (prev_hash, row_hash) required even if current SqliteUsageRepo lacks them (extend schema in WP-G)
Tests cd zappier-edge && npm test -- tests/business-store.contract.test.ts → expect fail

WP-G — SqliteFileBusinessStore (dev/lab adapter)

Field Content
PreReqs WP-F red tests exist
Mode Serial
Repos zappier-edge
Files Create src/db/sqlite-file-business-store.ts; adapt src/db/usage-repo.ts (add hash chain columns + verify); wire factory in src/index.ts / src/app.ts behind flag BUSINESS_STORE=sqlite-file (default keep current behavior until WP-L)
DoD Contract tests pass on file/memory SQLite; existing Jest suite still green
Tests cd zappier-edge && npm test -- tests/business-store.contract.test.ts; cd zappier-edge && npm test

WP-H — LibsqlBusinessStore + CI

Field Content
PreReqs WP-G green
Mode Serial (can overlap WP-I ops once interface stable)
Repos zappier-edge
Files Create src/db/libsql-business-store.ts; package.json add @libsql/client; .env.example LIBSQL_URL, LIBSQL_AUTH_TOKEN; Jest config for libsql file URL or embedded
DoD Same contract suite passes with BUSINESS_STORE=libsql and LIBSQL_URL=file:… (or testcontainers); WAL/durable semantics documented
Tests cd zappier-edge && BUSINESS_STORE=libsql LIBSQL_URL=file:./tmp-business.db npm test -- tests/business-store.contract.test.ts; full npm test

WP-I — libSQL server / ops bring-up

Field Content
PreReqs None for draft runbook; MG1 needs live URL
Mode Parallel feeding into MG1
Repos verae-ops, verae-deploy, verae-nats-cluster (topology reference only), zapier-decisions
Files New runbook under verae-ops (e.g. libsql.md / section in GETTING-STARTED.md); env matrix for 3–5 hosts; do not relocate Go DBs
DoD Documented LIBSQL_URL for lab; backup note (Litestream optional for primary file — not multi-writer); firewall private-only
Tests Manual: curl/client ping create table + select; record connection string in STATUS

WP-J — Fixture harvest (billing.db + books.json)

Field Content
PreReqs Read-only access to lab 138 paths (or sanitized copies)
Mode Parallel into MG1
Repos zappier-edge, zappier-account-balance, zapier-decisions
Files Create zappier-edge/tests/fixtures/ (sanitized); scripts/export-lab-business-fixtures.md instructions; never commit live secrets/API keys
Sources Live stand-ins: /opt/verae/zappier-edge/billing.db (BILLING_DB), zappier.db (auth may stay local), account-balance books.json; seed keys in lab only
DoD Anonymized fixtures with known chain tip + prepaid balances for migrator tests
Tests Fixture checksums documented; sqlite3 fixture.db 'select count(*) from usage_entries;'

WP-K — Migrator billing.db + books.json → libSQL (Phase 2)

Field Content
PreReqs MG1 PASS; fixtures from WP-J; LIBSQL_URL from WP-I
Mode Serial
Repos zappier-edge, zappier-account-balance
Files Create zappier-edge/scripts/migrate-to-libsql.ts (or .mjs); Create tests/migrate-to-libsql.test.ts; map tables: customers, tiers, price_endpoints, channel_rates*, usage_entries(+hashes), invoices*, balances/credits/payments from books
DoD One-shot migrator; verifyUsageChain true before and after; dual-run read path flag BUSINESS_READ_PRIMARY=libsql with file fallback; no production writer cutover yet
Tests cd zappier-edge && npm test -- tests/migrate-to-libsql.test.ts; dry-run against fixtures; print before/after tip hashes

MERGE GATE 2 — Writers still locked

Engineers must not start WP-L/M/N until Orchestrator/PM records:

MERGE GATE 2: PASS
- migrator tip hashes match
- dual-run reads validated
- writers unlocked: yes

WP-L — Rewire zappier-edge meter / ledger → BusinessStore (Phase 3)

Field Content
PreReqs MG2 PASS
Mode Serial (constraint path)
Repos zappier-edge
Files src/meter.ts, src/usage.ts, src/ledger.ts, src/billing-nats.ts, src/credits.ts, src/portal.ts, src/pricing.ts, src/invoicing.ts, src/db/*-repo.ts, src/index.ts, src/app.ts — inject BusinessStore; edge balanceCents remains cache filled from store/billing API
DoD Meter appends usage only via BusinessStore; prepaid adjust goes through single writer API; no direct better-sqlite3 business writes outside adapter
Tests cd zappier-edge && npm test; focus tests/meter.test.ts, tests/usage.test.ts, tests/credit.test.ts, tests/portal.test.ts, tests/db-*.test.ts, tests/billing-delta.test.ts

WP-M — Rewire account-balance (books.json → libSQL)

Field Content
PreReqs WP-L in progress or done (coordinate subjects)
Mode Serial after WP-L starts (shared verae.billing.* contracts)
Repos zappier-account-balance, zappier-edge
Files zappier-account-balance/src/persist.js, books.js, server.js, subjects.js; tests under test/; README SoT language → libSQL
DoD NATS verae.billing.balance.* / statement backed by libSQL BusinessStore or HTTP billing API; books.json deprecated (import-only); D001 single writer preserved
Tests cd zappier-account-balance && npm test; integration: adjust → edge statement sees same balance

WP-N — Rewire CS / sales / accounting-export

Field Content
PreReqs WP-M billing subjects stable
Mode Parallel with each other after WP-M contract freeze; else serial behind WP-M
Repos zappier-customer-service, zappier-sales-pricing, zappier-accounting-export
Files Each package src/nats-billing.js, src/server.js; staff UI pages as needed
DoD CS credit adjust → libSQL balance; sales channel×segment rates → same rate tables meter uses; accounting export reads payments/usage from libSQL
Tests cd zappier-customer-service && npm test; cd zappier-sales-pricing && npm test; cd zappier-accounting-export && npm test

MERGE GATE 3 — Single writer proof

Check Pass criteria
Grep No code path treats books.json or per-host billing.db as production SoT
Runtime One LIBSQL_URL serves customers/pricing/usage/invoices/balances
D004 Leaf still cannot adjust billing
Regression zappier-edge full Jest green

WP-O — Multi-host topology (Phase 4)

Field Content
PreReqs MG3 PASS
Mode Serial
Repos verae-ops, verae-deploy, verae-fleet, verae-keep, verae-bootstrap, zappier-edge, verae-middleware
Files Host env templates (LIBSQL_URL, NATS_URL=nats://10.10.10.21:4222,...); deploy scripts; STATUS topology section
DoD libSQL on dedicated/primary host; edge/middleware/CS on separate hosts possible; JetStream workers unchanged; leaf cannot bill
Tests Cross-host: meter on host A, CS adjust on host B, portal statement on host C — same balance; verae-nats-cluster npm run status

WP-P — Userflow E2E harness (Phase 5)

Field Content
PreReqs MG3; WP-O preferred for multi-host proof, MOCK_VERAE OK for first pass
Mode Serial into MG4
Repos verae-zapier-simulator, zappier-edge, verae-middleware, verae-access-web, verae-access-zapier, verae-access-leaf
Files Extend simulator/harness under simulator or packages-style tests; scripts for flows 1–7 from architecture §1
Flows (1) Signup→API key→Statement (2) Stamp digests single+batch + credit headers (3) Async complete→REST Hook via jobs.watch→events→webhooks.deliver (4) CS credit→portal balance (5) Sales quote→meter match (6) Accounting export consistency (7) Leaf stamp OK / billing deny
DoD Automated report artifact under zapier-decisions/plans/qa/
Tests cd verae-zapier-simulator && npm test; middleware npm test + selected test:gate*; edge npm test; flow runner script (to be added) exit 0

WP-Q — QA whattochange loop (Phase 6)

Field Content
PreReqs MG4 PASS (suite green once)
Mode Loop until project buffer
Repos zapier-decisions + whichever repos bugs land in
Files plans/qa/<date>-whattochange/ packs; NotImplemented/ leftovers per recommended-changes skill
DoD Each bug → a/b/c/d + PreReq todos; engineers implement; re-run WP-P
Tests Full suite + userflows after each pack

4. Parallel vs serial schedule — first 2 weeks

Assume agent-days; calendar = 10 working days. Critical chain owns the constraint; feeding work never blocks writers early.

Day Parallel (feeding) Serial (critical chain) Gate
1 WP-A, WP-B, WP-C, WP-D, WP-E kickoff — Aim MG0 EOD day 2
2 Finish A–E; Architect signs 2026-10-03-arch-review.md — MG0
3 WP-I ops draft; WP-J fixture planning WP-F interface + red tests
4 WP-I lab libSQL up; WP-J sanitize fixtures WP-G SqliteFile adapter
5 WP-I harden docs; fixture finalize WP-H LibsqlBusinessStore MG1 target EOD day 5–6
6 Buffer if MG1 slip WP-K migrator TDD
7 — Finish WP-K; dual-run reads MG2 (writers unlock)
8 — WP-L edge rewire
9 WP-N prep (read-only) WP-M account-balance
10 WP-N CS/sales/acct (parallel agents) Finish L/M integration MG3 stretch; else day 11

Week 2 overflow (if MG3 slips): do not start WP-O multi-host. Burn down writer bugs; keep project buffer intact.

Week 2 if on track (days 11–14): WP-O topology → WP-P harness → MG4 → start WP-Q.

Explicit non-goals in first 2 weeks: live MOCK_VERAE=false seal proof; Zapier Platform public push; lan-134; Go DB moves; JetStream for verae.ts.*.


5. QA entry criteria

QA (WP-Q / userflow break attempts) may begin only when all of the following are true:

  1. MG3 PASS — single prepaid writer on libSQL; no dual SoT writes.
  2. Regression green: zappier-edge npm test; zappier-account-balance npm test; CS/sales/acct npm test; verae-middleware npm test (unit + non-NATS integration at minimum).
  3. NATS rules verified: product JetStream streams only for Zapier jobs/webhooks/archive; verae.ts.completed / verae.ts.receipt core-only (grep + cluster status).
  4. Harness exists (WP-P skeleton): at least flows 1, 2, 4 runnable under MOCK_VERAE=true.
  5. D004 automated: leaf billing adjust denied.
  6. Migrator evidence: tip hash report archived under plans/qa/.
  7. Go timestamping DBs: confirmation note that no PR touched them.
  8. Docs: C2/C3/C5/C6 closed or explicitly ticketed with owner.

QA exit (feeds project buffer): whattochange packs filed for every failure; no silent skips; leftovers under NotImplemented/ with why.


6. Buffer policy

Merge gate before rewiring writers (MG2)

  • Hard stop: WP-L / WP-M / WP-N are blocked until MG2 PASS.
  • Rationale (Critical Chain feeding buffer): incomplete BusinessStore or bad migration hitting the meter writer destroys the only constraint we are exploiting (consistent SoT).
  • Gate owner: PM / Orchestrator. Engineers may prepare branches but must not merge writer cutover.

Other merge gates

Gate Absorbs variation from Failure action
MG0 Docs/conflict feeding Re-run WP-A/B/C; no Phase 1 code
MG1 Driver + ops + fixtures Fix contract/CI URL; no migrator on prod-like data
MG3 Writer rewires Roll back feature flag to SqliteFile/dual-read; do not split hosts
MG4 E2E harness Fix failures; do not declare QA loop “done”

Project buffer = double userflow pass

  • After MG4 and WP-Q churn, acceptance requires userflows 1–7 green twice consecutively (architecture §10 / Phase 6 exit).
  • First pass proves the system; second pass absorbs latent race/order bugs without consuming schedule fantasy.
  • Do not spend project buffer on early scope (Platform push, live seal, Peergos redesign).
  • If second pass fails, return to WP-Q; do not “waive” the second pass.

Constraint buffer

  • Before each critical-chain WP (F, G, H, K, L), feeding artifacts listed in §2 must be staged so the constraint agent never waits on clone/docs/URL discovery mid-implementation.

7. Explicit Forgejo repos each WP clones

Base URL: https://git.georgelambert.org/marchon/<repo>.git

WP Repos to clone
WP-A zapier-decisions, overview, zapier-docs-master
WP-B zapier-decisions, verae-middleware, verae-nats-cluster, zapier-docs-master
WP-C zapier-docs-master, verae-middleware, verae-zapier-app, zapier-decisions
WP-D verae-access-leaf, verae-access-web, verae-access-api, verae-access-zapier, verae-access-authz, zappier-identity
WP-E verae-bootstrap (plus any MISSING from repos.txt needed by later WPs)
WP-F zappier-edge
WP-G zappier-edge
WP-H zappier-edge
WP-I verae-ops, verae-deploy, zapier-decisions
WP-J zappier-edge, zappier-account-balance, zapier-decisions
WP-K zappier-edge, zappier-account-balance
WP-L zappier-edge
WP-M zappier-account-balance, zappier-edge
WP-N zappier-customer-service, zappier-sales-pricing, zappier-accounting-export
WP-O verae-ops, verae-deploy, verae-fleet, verae-keep, verae-bootstrap, zappier-edge, verae-middleware
WP-P verae-zapier-simulator, zappier-edge, verae-middleware, verae-access-web, verae-access-zapier, verae-access-leaf
WP-Q zapier-decisions + bug-target repos as discovered

Union set for cutover agents (minimum clone list)

zapier-decisions
overview
zapier-docs-master
verae-bootstrap
verae-ops
verae-deploy
verae-fleet
verae-keep
verae-nats-cluster
zappier-edge
zappier-account-balance
zappier-customer-service
zappier-sales-pricing
zappier-accounting-export
zappier-identity
verae-middleware
verae-zapier-app
verae-zapier-simulator
verae-access-authz
verae-access-web
verae-access-api
verae-access-leaf
verae-access-zapier

Out of clone scope for this cutover (do not pull for WPs above): Go timestamping-svc repos, Peergos/Seaweed redesign trees, verae-activate (unless WP-C needs tiny-app copy sync).


Orchestrator checklist (next actions)

  • Architect produces plans/reviews/2026-10-03-arch-review.md → MG0
  • Launch parallel agents: WP-A, WP-B, WP-C, WP-D, WP-E
  • After MG0: engineer on WP-F → G → H; parallel WP-I, WP-J
  • Stop before writers until MG2
  • Enforce D006 + core-NATS job-complete on every PR review

End of PM board. Planning artifact only — no implementation in this change.