Milestone 0: import zappier billing, Verae middleware, and Zapier research

Compose-ready workspace: packages/zappier (rate card, portal, Stripe),
packages/verae-zapier-middleware (timestamp + NATS), packages/verae-zapier
(CLI app), vendor/zapier-platform, and research/zapier vendor corpus.

Gate 0 structure checks pass. Product code and research are not yet wired.
This commit is contained in:
George Lambert 2026-09-09 02:37:36 -04:00
commit b4150c8250
1364 changed files with 6814366 additions and 0 deletions

659
TODO.md Normal file
View file

@ -0,0 +1,659 @@
# Verae Zapier Platform — Implementation TODO
**Workspace:** `/Users/marchon/apps/zapier` (imported 2026-09-09). Packages live under `packages/`. Commercial billing is `packages/zappier` — do not reimplement it in middleware. Composition waves: see `docs/02-architecture/composition.md`.
**Product goal:** Hybrid **HTTP edge + NATS workers** middleware between Zapier servers and `https://api.veraetime.net`.
**Architecture bottom line:**
- Zapier never speaks NATS (HTTPS only).
- `verae-zapier` (CLI app) runs on Zapier infrastructure and calls middleware.
- `verae-zapier-middleware` exposes `/zapier/v1/*` (auth, billing, sync API).
- **NATS + JetStream** owns job watching, completion events, and webhook delivery.
- Workers call `api.veraetime.net` over HTTPS and POST REST Hooks to Zapier.
**How to use this file**
1. Work **top to bottom**. Phases are dependency-ordered.
2. Every phase ends with a **GATE** (tests must pass) before the next phase starts.
3. Mark items `[x]` only after the gate command succeeds and results are recorded under **Gate log**.
4. Do not skip gates. Parallel work is only allowed **within** a phase when items share no dependency.
**Runtime debug**
- Failure tracing is off by default.
- Enable at runtime: `DEBUG_VERAE=1` or `DEBUG_VERAE=auth,nats,jobs,webhooks,http,billing`.
- See [docs/developer/debugging.md](docs/developer/debugging.md).
**Primary docs**
| Doc | Purpose |
|-----|---------|
| [README.md](README.md) | Repo overview and quick start |
| [docs/architecture/overview.md](docs/architecture/overview.md) | System diagram and data flows |
| [docs/architecture/nats-subjects.md](docs/architecture/nats-subjects.md) | Subject topology and payloads |
| [docs/developer/modules/](docs/developer/modules/) | Per-module function reference |
| [docs/plans/phase-gates.md](docs/plans/phase-gates.md) | Gate commands and acceptance criteria |
| [docs/api/middleware-openapi.yaml](docs/api/middleware-openapi.yaml) | Zapier-facing OpenAPI |
---
## Legend
| Marker | Meaning |
|--------|---------|
| `[ ]` | Not started |
| `[~]` | In progress |
| `[x]` | Done and **gate passed** |
| **BLOCKED** | Waiting on listed dependency |
| **GATE** | Mandatory test validation — no next phase until green |
---
## Phase 0 — Repository baseline and documentation contract
**Depends on:** nothing
**Unblocks:** Phase 1+
### Tasks
- [x] 0.1 Create monorepo layout (`verae-zapier-middleware/`, `verae-zapier/`, `docs/`)
- [x] 0.2 Write architecture overview and NATS subject map
- [x] 0.3 Write developer module documentation skeleton for all planned modules
- [x] 0.4 Document debugging facility contract (`DEBUG_VERAE`, namespaces, redaction rules)
- [x] 0.5 Write this dependency-ordered `TODO.md` with test gates
- [x] 0.6 Add root `README.md` and package scripts for gated testing
### GATE 0 — Documentation and structure
```bash
cd /Users/marchon/datacubes/verae-zapier-api
npm run gate:0
```
**Pass criteria**
- Required paths exist (see `scripts/gate-0-structure.mjs`)
- `TODO.md`, architecture docs, debugging doc, module docs present
- No empty required doc stubs
**Gate log**
| Date | Result | Notes |
|------|--------|-------|
| 2026-08-11 | pass | structure + docs gate |
---
## Phase 1 — Debug runtime facility
**Depends on:** Phase 0 GATE
**Unblocks:** All code phases (debug used everywhere)
### Tasks
- [x] 1.1 Implement `src/debug/logger.js` — namespaced logger, levels, redact secrets
- [x] 1.2 Implement `src/debug/trace.js` — request/job correlation IDs
- [x] 1.3 Implement `src/debug/config.js` — parse `DEBUG_VERAE` / `DEBUG_VERAE_LEVEL` at runtime
- [x] 1.4 Unit tests: enable/disable namespaces, redaction of tokens/passwords/api keys
- [x] 1.5 Document every debug export in `docs/developer/modules/debug.md`
### GATE 1 — Debug facility
```bash
npm run gate:1
# equivalent: node --test verae-zapier-middleware/test/unit/debug*.test.js
```
**Pass criteria**
- Logger silent when `DEBUG_VERAE` unset
- Enabling `DEBUG_VERAE=auth` only emits auth namespace
- Redaction masks `Bearer`, `zmw_`, `zmt_`, password fields
- Correlation ID stable across nested `trace.run` calls
**Gate log**
| Date | Result | Notes |
|------|--------|-------|
| 2026-08-11 | pass | 11/11 unit tests |
---
## Phase 2 — Config, errors, and core HTTP shell
**Depends on:** Phase 1 GATE
**Unblocks:** Auth, clients, routes
### Tasks
- [x] 2.1 `src/config.js` — env load, plan limits, NATS flags, debug passthrough
- [x] 2.2 `src/errors.js``AppError`, `asyncHandler`, `sendError` with debug dumps on 5xx
- [x] 2.3 `src/app.js` / `src/index.js` — Express app, `/health`, `/zapier` mount
- [x] 2.4 Unit tests for config defaults and health endpoint
- [x] 2.5 Module docs: `config.md` (+ function-reference for errors/app)
### GATE 2 — HTTP shell
```bash
npm run gate:2
```
**Pass criteria**
- `GET /health` returns `{ status: "ok" }`
- Unknown routes return structured JSON error
- Config reads `VERAE_API_BASE_URL`, `NATS_URL`, `NATS_ENABLED`, `DEBUG_VERAE`
**Gate log**
| Date | Result | Notes |
|------|--------|-------|
| 2026-08-11 | pass | health + config unit tests; /zapier returns 501 until Phase 56 routes |
---
## Phase 3 — Persistence stores (tenants, usage, webhooks)
**Depends on:** Phase 2 GATE
**Unblocks:** Auth bridge, billing, webhook subscribe API
### Tasks
- [x] 3.1 `src/store/db.js` — load/persist store (file for MVP; interface ready for Redis/Postgres)
- [x] 3.2 `src/store/tenants.js` — CRUD, API key lookup, `resolveLimits`
- [x] 3.3 `src/store/usage.js` — monthly counters, overage
- [x] 3.4 `src/store/webhooks.js` — subscribe/unsubscribe/list by tenant+event
- [x] 3.5 Unit tests for limits, API key mapping, webhook isolation per tenant
- [x] 3.6 Module docs for each store file
### GATE 3 — Stores
```bash
npm run gate:3
```
**Pass criteria**
- Create tenant → API key resolves to same tenant
- Free plan limits applied; enterprise contract overrides
- Webhook for tenant A never returned for tenant B
- Persist + reload round-trip preserves data
**Gate log**
| Date | Result | Notes |
|------|--------|-------|
| 2026-08-11 | pass | store unit tests |
---
## Phase 4 — Tokens and Verae HTTP client
**Depends on:** Phase 3 GATE
**Unblocks:** Auth service, timestamp/verify services
### Tasks
- [x] 4.1 `src/lib/tokens.js` — API keys (`zmw_`), session tokens (`zmt_`), HMAC sign/verify
- [x] 4.2 `src/clients/veraeClient.js` — login, validate, timestamp, status, verify, waitForJob
- [x] 4.3 Mock mode (`MOCK_VERAE=true`) for offline tests
- [x] 4.4 Debug spans for outbound HTTP (url, method, status, duration; **no raw secrets**)
- [x] 4.5 Unit tests: mock client lifecycle (create → pending → completed)
- [x] 4.6 Module docs: `tokens.md`, `veraeClient.md`
### GATE 4 — Client + tokens
```bash
npm run gate:4
```
**Pass criteria**
- Session token forge fails without secret
- Mock create returns `jobId`; wait reaches `completed`
- Debug log with `DEBUG_VERAE=http` shows request metadata without Authorization header values
**Gate log**
| Date | Result | Notes |
|------|--------|-------|
| 2026-08-11 | pass | tokens + mock client |
---
## Phase 5 — Auth bridge and entitlements (HTTP, no NATS yet)
**Depends on:** Phase 4 GATE
**Unblocks:** Timestamp/verify routes, Zapier custom auth
### Tasks
- [x] 5.1 `src/services/authService.js` — login credentials, login API key, resolveAuthContext, validateSession
- [x] 5.2 `src/services/entitlementService.js` — checkEntitlement, recordUsage
- [x] 5.3 `src/middleware/authenticate.js` — Bearer / x-api-key
- [x] 5.4 `src/middleware/rateLimit.js` — per-plan RPM
- [x] 5.5 `src/routes/authRoutes.js``POST /login`, `GET /me` under `/zapier/v1/auth`
- [x] 5.6 Integration tests: API key → me; quota 402; rate limit
- [x] 5.7 Module docs for auth + entitlement + middleware
### GATE 5 — Auth + billing gates
```bash
npm run gate:5
```
**Pass criteria**
- Seeded free tenant can call `/zapier/v1/auth/me`
- Exceeding free timestamp quota returns `402` + `QUOTA_EXCEEDED`
- Invalid key returns `401`
- Debug `DEBUG_VERAE=auth,billing` traces tenantId/plan without password
**Gate log**
| Date | Result | Notes |
|------|--------|-------|
| 2026-08-11 | pass | auth me + quota |
---
## Phase 6 — Sync HTTP API path (in-process poller fallback)
**Depends on:** Phase 5 GATE
**Unblocks:** NATS swap-in (Phase 7+), Zapier app wiring
**Note:** Implements full product behavior with `NATS_ENABLED=false` so Zapier can work before NATS.
### Tasks
- [x] 6.1 `src/services/timestampService.js` — create, createAndWait, batch, getStatus
- [x] 6.2 `src/services/verifyService.js` — verify, verifyBatch
- [x] 6.3 `src/services/webhookService.js` — subscribe/unsubscribe/deliver (HTTP)
- [x] 6.4 Routes: timestamp, verify, status, webhooks
- [x] 6.5 In-process `src/workers/inProcessJobPoller.js` (legacy path, feature-flagged)
- [x] 6.6 Integration tests with `MOCK_VERAE=true`, `NATS_ENABLED=false`
- [x] 6.7 Module docs for services + routes
### GATE 6 — Full HTTP middleware (no NATS)
```bash
npm run gate:6
```
**Pass criteria**
- `POST /zapier/v1/timestamp` → 202 + jobId
- `POST /zapier/v1/timestamp/wait` → completed status with certificate fields
- `POST /zapier/v1/verify` → valid true/false
- Webhook subscribe stores targetUrl; simulated complete invokes deliver (mock)
- All tests pass with NATS disabled
**Gate log**
| Date | Result | Notes |
|------|--------|-------|
| 2026-08-11 | pass | full HTTP mock path |
---
## Phase 7 — NATS connection, streams, and publishers
**Depends on:** Phase 6 GATE
**Unblocks:** Workers, production multi-instance
### Tasks
- [x] 7.1 Add local NATS JetStream to `docker-compose.yml`
- [x] 7.2 `src/nats/subjects.js` — subject constants
- [x] 7.3 `src/nats/connection.js` — connect, reconnect, ensure streams/consumers
- [x] 7.4 `src/nats/publishers.js` — enqueueWatch, publishJobEvent, enqueueWebhook
- [x] 7.5 Unit/integration tests against test NATS (or mocked JetStream interface)
- [x] 7.6 Module docs: `nats-connection.md`, `nats-subjects.md`, `nats-publishers.md`
- [x] 7.7 Debug namespace `nats` for publish/ack/error tracing
### GATE 7 — NATS infrastructure
```bash
npm run gate:7
# starts NATS if needed, ensures streams, runs nats tests
```
**Pass criteria**
- Streams `ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS` exist
- Publish + pull consume one message end-to-end
- Connection failure surfaces structured error; debug logs connection lifecycle
- `NATS_ENABLED=false` still loads app without connecting
**Gate log**
| Date | Result | Notes |
|------|--------|-------|
| 2026-08-11 | pass | nats-server JetStream |
---
## Phase 8 — NATS workers (job poller + webhook deliver)
**Depends on:** Phase 7 GATE
**Unblocks:** Hybrid production path
### Tasks
- [x] 8.1 `src/workers/jobPollerWorker.js` — durable consumer on `verae.zapier.jobs.watch`
- [x] 8.2 `src/workers/webhookWorker.js` — durable consumer on `verae.zapier.webhooks.deliver`
- [x] 8.3 Wire `timestampService` to publish watch jobs when `NATS_ENABLED=true`
- [x] 8.4 Events consumer routes `jobs.events` → enqueue webhook deliveries for matching hooks
- [x] 8.5 **Never** put raw Verae JWT in NATS when avoidable — use `tokenRef` or re-login
- [x] 8.6 Integration tests: create → worker poll → event → webhook HTTP mock
- [x] 8.7 Module docs for workers; failure tracing via `DEBUG_VERAE=jobs,webhooks,nats`
### GATE 8 — Async path with NATS
```bash
npm run gate:8
```
**Pass criteria**
- With NATS up and `NATS_ENABLED=true`, async create eventually completes via worker
- Webhook target receives POST with `timestamp.completed` payload
- Redelivery: failing webhook is retried (JetStream max_deliver)
- Two concurrent workers do not double-complete same job (queue group)
- In-process poller **not** started when NATS enabled
**Gate log**
| Date | Result | Notes |
|------|--------|-------|
| 2026-08-11 | pass | workers watch→webhook |
---
## Phase 9 — `/timestamp/wait` via NATS events
**Depends on:** Phase 8 GATE
**Unblocks:** Efficient Zapier “Create and Wait” under multi-instance
### Tasks
- [ ] 9.1 API waits on job event (subscription or inbox) after enqueue watch
- [ ] 9.2 Hard timeout returns `{ jobId, status: "pending" }` (document Zapier fallback)
- [ ] 9.3 Tests: fast complete; timeout path; cancellation on client disconnect (if feasible)
- [ ] 9.4 Docs update: wait semantics, Zapier multi-step fallback
### GATE 9 — Wait path
```bash
npm run gate:9
```
**Pass criteria**
- Wait returns completed when worker finishes before timeout
- Wait returns pending + jobId on timeout without hanging forever
- Debug shows wait correlation id linking HTTP request → NATS event
**Gate log**
| Date | Result | Notes |
|------|--------|-------|
| | | |
---
## Phase 10 — Tenant signup / admin provision
**Depends on:** Phase 6 GATE (can parallel Phase 79 after 6)
**Unblocks:** Self-serve and enterprise keys for Zapier
### Tasks
- [x] 10.1 `src/services/tenantService.js` — selfServeSignup, provisionTenant
- [x] 10.2 Routes: public signup, admin tenants (`X-Admin-Secret`)
- [x] 10.3 Scripts: `seed-audiences.js`, `create-test-account.js`
- [x] 10.4 Tests for free signup and enterprise contract validation
- [x] 10.5 Module docs: `tenantService.md`, `tenantRoutes.md`
### GATE 10 — Tenancy
```bash
npm run gate:10
```
**Pass criteria**
- Signup returns `apiKey` + free plan
- Enterprise without contract rejected
- Admin list does not leak passwords
**Gate log**
| Date | Result | Notes |
|------|--------|-------|
| 2026-08-11 | pass | signup + admin |
---
## Phase 11 — Zapier CLI app (`verae-zapier`)
**Depends on:** Phase 6 GATE minimum; Phase 8 recommended for triggers
**Unblocks:** Private invites on Zapier
### Tasks
- [x] 11.1 `authentication.js` — custom API key → `/zapier/v1/auth/me`
- [x] 11.2 `index.js` — beforeRequest API key; afterResponse map 402/403
- [x] 11.3 Creates: timestamp_and_wait, create_timestamp, verify, batch
- [x] 11.4 Search: job_status
- [x] 11.5 Trigger: timestamp_completed REST Hook
- [x] 11.6 Unit tests with mocked middleware (nock or local server)
- [x] 11.7 `MIDDLEWARE_BASE_URL` env documentation
- [x] 11.8 Developer docs for each Zapier operation module
### GATE 11 — Zapier package
```bash
npm run gate:11
```
**Pass criteria**
- Auth test module returns tenant/plan label fields
- Each create/search perform builds correct middleware URL/body
- Trigger subscribe/unsubscribe payload shapes match middleware
- `zapier-platform validate` / local test suite green (when CLI available)
**Gate log**
| Date | Result | Notes |
|------|--------|-------|
| 2026-08-11 | pass | zapier package unit tests |
---
## Phase 12 — End-to-end validation (local stack)
**Depends on:** Gates 8, 10, 11
**Unblocks:** Remote deploy
### Tasks
- [ ] 12.1 `docker-compose.yml`: middleware + NATS (+ optional mock Verae)
- [ ] 12.2 Compose smoke script: health, auth, timestamp wait, webhook
- [ ] 12.3 Debug mode runbook: reproduce a failed job with `DEBUG_VERAE=*`
- [ ] 12.4 Record sample traces in `docs/developer/debugging.md`
### GATE 12 — E2E local
```bash
npm run gate:12
```
**Pass criteria**
- Compose stack healthy
- One full async path and one wait path succeed
- One REST Hook delivery succeeds against mock receiver
- Gate script exits 0
**Gate log**
| Date | Result | Notes |
|------|--------|-------|
| | | |
---
## Phase 13 — Production wiring to api.veraetime.net
**Depends on:** Phase 12 GATE
**Unblocks:** Real customer traffic
### Tasks
- [ ] 13.1 Set `VERAE_API_BASE_URL=https://api.veraetime.net`, `MOCK_VERAE=false`
- [ ] 13.2 Provision dedicated Zapier service user on Verae (not shared admin long-term)
- [ ] 13.3 TLS, secrets management (`TOKEN_SECRET`, `ADMIN_SECRET`, NATS auth)
- [ ] 13.4 Public HTTPS URL for middleware (required for Zapier cloud → you)
- [ ] 13.5 Live smoke tests (non-destructive verify + one timestamp)
- [ ] 13.6 Runbook: incident tracing with debug flags (no secrets in logs)
### GATE 13 — Production smoke
```bash
npm run gate:13
# requires env: PRODUCTION_SMOKE=1 and real credentials
```
**Pass criteria**
- Login + timestamp + status against live API succeed
- No secrets printed in CI logs
- Documented rollback: `NATS_ENABLED=false` still serves sync path
**Gate log**
| Date | Result | Notes |
|------|--------|-------|
| | | |
---
## Phase 14 — Private Zapier push and invite
**Depends on:** Phase 13 GATE
**Unblocks:** Beta users
### Tasks
- [ ] 14.1 `zapier-platform register` / `link`
- [ ] 14.2 `zapier-platform push` private version
- [ ] 14.3 Invite internal users; validate real Zap (Drive/Sheets → Timestamp → Slack)
- [ ] 14.4 Collect failures; enable `DEBUG_VERAE` only on middleware staging for repro
- [ ] 14.5 Update Progress / changelog
### GATE 14 — Human acceptance
**Pass criteria**
- At least one real Zap runs end-to-end with production middleware
- Auth connection label shows tenant + plan
- Async trigger or wait path verified by human sign-off
**Gate log**
| Date | Result | Notes |
|------|--------|-------|
| | | |
---
## Phase 15 — Hardening (post-beta)
**Depends on:** Phase 14
**Unblocks:** Public listing consideration
### Tasks
- [ ] 15.1 Replace file store with Postgres/Redis
- [ ] 15.2 NATS mTLS / account auth
- [ ] 15.3 Webhook SSRF protections
- [ ] 15.4 Usage stream consumer for billing export
- [ ] 15.5 Optional: Verae core publishes completion to NATS (drop polling)
- [ ] 15.6 Security review of tokenRef vs password-at-rest
### GATE 15 — Hardening checklist
**Pass criteria**
- Written security review checklist completed
- Load test: N concurrent workers, no duplicate webhook storms
- Backup/restore drill for tenant store
---
## Dependency graph (summary)
```text
0 Docs/structure
└─▶ 1 Debug facility
└─▶ 2 HTTP shell
└─▶ 3 Stores
└─▶ 4 Tokens + Verae client
└─▶ 5 Auth + entitlements
└─▶ 6 Sync HTTP API (+ in-process poller)
├─▶ 7 NATS infra ──▶ 8 Workers ──▶ 9 Wait-via-NATS
├─▶ 10 Tenancy (parallel after 6)
└─▶ 11 Zapier app (after 6; triggers prefer 8)
└─▶ 12 E2E local (needs 8,10,11)
└─▶ 13 Production Verae
└─▶ 14 Private Zapier push
└─▶ 15 Hardening
```
---
## npm gate commands (root)
| Script | Phase | What it runs |
|--------|-------|--------------|
| `npm run gate:0` | 0 | Structure + doc presence |
| `npm run gate:1` | 1 | Debug unit tests |
| `npm run gate:2` | 2 | Config/errors/app tests |
| `npm run gate:3` | 3 | Store tests |
| `npm run gate:4` | 4 | Tokens + client tests |
| `npm run gate:5` | 5 | Auth + entitlement tests |
| `npm run gate:6` | 6 | HTTP integration (NATS off) |
| `npm run gate:7` | 7 | NATS stream tests |
| `npm run gate:8` | 8 | Worker integration |
| `npm run gate:9` | 9 | Wait path tests |
| `npm run gate:10` | 10 | Tenant tests |
| `npm run gate:11` | 11 | Zapier package tests |
| `npm run gate:12` | 12 | Compose E2E |
| `npm run gate:13` | 13 | Live smoke (opt-in) |
| `npm run gate:all` | 012 | All automated gates in order; stop on first failure |
---
## Definition of done (platform MVP)
- [ ] All gates 012 green on clean machine
- [ ] `NATS_ENABLED=true` path used for job watch + webhooks
- [ ] `NATS_ENABLED=false` fallback still passes gate 6
- [ ] Developer docs cover every modules exported functions (params + returns)
- [ ] Debug can be enabled at runtime without redeploy (`DEBUG_VERAE=...`)
- [ ] No secrets in logs when debug is on (redaction tests green)
- [ ] Private Zapier version pushed and one human E2E Zap succeeds (gate 14)
---
## Working rules for implementers
1. **No phase advancement without gate.**
2. **Write tests first or with the code** for that phases modules.
3. **Document every new export** in `docs/developer/modules/<name>.md` and JSDoc in source.
4. **Instrument with `debug`** at boundaries: auth resolve, outbound Verae, NATS publish/consume, webhook HTTP.
5. **Prefer failing a gate** over merging incomplete behavior.
6. Update the **Gate log** tables when a gate passes.