zappier-edge/docs/DEVELOPER.md

406 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Zappier — Developer Documentation
**Version:** 0.2.0 · **Last updated:** 2026-07-28
Metered API platform: per-endpoint pricing, per-tier multipliers, usage
ledger, Stripe metered billing, purchase-order invoicing, a company admin
console, a customer portal with TOTP 2FA, and a Zapier integration. This
document is the full technical reference. For operations, see
[USER-MANUAL.md](USER-MANUAL.md).
---
## Table of contents
1. [Architecture](#1-architecture)
2. [Module reference](#2-module-reference)
3. [Pricing engine](#3-pricing-engine)
4. [Request lifecycle](#4-request-lifecycle)
5. [Persistence (SQLite)](#5-persistence-sqlite)
6. [Billing pipeline](#6-billing-pipeline)
7. [API reference](#7-api-reference)
8. [Zapier app](#8-zapier-app)
9. [Testing](#9-testing)
10. [Configuration](#10-configuration)
11. [Extending the system](#11-extending-the-system)
---
## 1. Architecture
**Stack:** Node 20 · TypeScript (strict) · Express 4 · better-sqlite3 ·
express-openapi-validator · swagger-ui-express · stripe SDK · Zapier Platform
(core ^19) · jest + supertest (root) and mocha (zapier-app).
```
┌────────────────────────────────────────────┐
│ Express app │
│ (src/app.ts) │
│ │
Browser ── /admin ──────┼── static admin UI (admin/) │
Browser ── /admin/api ──┼── adminAuth → adminRouter (src/admin.ts) │
│ pricing · customers · invoices · │
│ reports · zapier status │
Browser ── /portal ─────┼── static portal SPA (portal/) │
Browser ── /portal/api ─┼── portalRouter (src/portal.ts) │
│ signup/login+2FA · me · usage · │
│ invoices · reload · prefs │
Browser ── /docs ───────┼── swagger-ui (openapi.yaml) │
│ │
Client ──── /v1/* ──────┼── apiKeyAuth (src/auth.ts) │
│ └─ OpenAPI request validation │
│ └─ meter() (src/meter.ts) │
│ ├─ quoteCall() (src/pricing.ts) │
│ └─ UsageRepo.record() │
└──────────────┬─────────────────────────────┘
┌───────────────┬───────────────┼───────────────┬────────────┐
│ │ │ │ │
SqliteUsageRepo SqliteCustomerRepo SqlitePricingStore SqliteInvoiceRepo
(usage_entries) (customers) (price_endpoints, (invoices,
│ │ tiers) invoice_lines)
│ SqliteSessionRepo (portal_sessions) │
└───────────────┴───────────────┴───────────────┴────────────┘
│ zappier.db (SQLite)
Cron (daily 06:17 ET) │
"Zappier billing · report usage" │
│ │
└─► src/jobs/report-usage.ts ── reportMonthlyUsage()
├─ computeBillableCents / computeDelta (src/billing/stripe.ts)
├─ SqliteBillingReportRepo (billing_reports, job_locks)
└─ stripe.billing.meterEvents.create ──► Stripe
```
Design rules the codebase follows:
- **Ports & adapters:** `UsageRepo`, `CustomerRepo`, `PricingStore`,
`MeterEventClient` are interfaces with in-memory adapters (tests) and SQLite
/ Stripe adapters (production). Nothing outside `src/db/` and the job's
`main()` touches SQL or the Stripe SDK.
- **Live pricing reads:** the `PricingContext` getters in `buildApp` read the
store on every quote, so admin edits apply without a restart.
- **Money in integer cents everywhere.** No floats cross a boundary except
multipliers, which are applied once and rounded (`Math.round`).
## 2. Module reference
| File | Responsibility | Key exports |
|---|---|---|
| `src/index.ts` | Production entry: opens SQLite, wires repos, listens | — |
| `src/app.ts` | `buildApp(deps)` — full Express wiring; `StoredItem`; `DEFAULT_CUSTOMERS` | `buildApp`, `AppDeps` |
| `src/auth.ts` | Customer model + `x-api-key` middleware | `Customer`, `CustomerRepo`, `apiKeyAuth`, `InMemoryCustomerRepo` |
| `src/pricing.ts` | Pricing domain: rules, tiers, quote algorithm, store port | `PriceRule`, `TierConfig`, `Quote`, `quoteCall`, `PricingStore`, `DEFAULT_RATE_CARD`, `DEFAULT_TIERS` |
| `src/meter.ts` | Per-request metering middleware | `meter(endpointId, repo, pricing)` |
| `src/usage.ts` | Usage ledger domain + summaries | `UsageEntry`, `UsageSummary`, `summarize`, `UsageRepo`, `InMemoryUsageRepo` |
| `src/admin.ts` | Admin API (`/admin/api`) + admin-key guard | `adminAuth`, `adminRouter` |
| `src/billing/credit.ts` | Monthly credit application | `applyMonthlyCredit`, `BilledSummary` |
| `src/billing/stripe.ts` | Stripe-facing math + client port | `METER_EVENT_NAME`, `MeterEventClient`, `computeBillableCents`, `computeDelta`, `reportUsage` |
| `src/billing/reload.ts` | Portal reloads via Stripe PaymentIntents | `stripePaymentClient`, `hasRealStripeKey` |
| `src/accounts.ts` | Portal identity: scrypt passwords, RFC 6238 TOTP, sessions | `hashPassword`, `verifyPassword`, `totp`, `verifyTotp`, `generateTotpSecret`, `totpUri`, `SessionRepo`, `InMemorySessionRepo` |
| `src/invoicing.ts` | Invoice domain + generation | `Invoice`, `InvoiceLine`, `buildInvoice`, `InvoiceRepo`, `InMemoryInvoiceRepo` |
| `src/reports.ts` | Billing/usage aggregation + CSV | `billingRows`, `usageTrend`, `toCsv`, `BillingRow`, `TrendPoint` |
| `src/portal.ts` | Customer portal API (`/portal/api`) | `portalRouter`, `PortalDeps`, `PaymentClient`, `PaymentResult` |
| `src/paths.ts` | Installation-root resolution (cwd-independent) | `PROJECT_ROOT` |
| `src/jobs/report-usage.ts` | Billing job: delta reporting to Stripe | `reportMonthlyUsage`, `firstOfMonthUtc`, `ReportUsageDeps` |
| `src/db/usage-repo.ts` | SQLite adapter: `usage_entries` | `SqliteUsageRepo` |
| `src/db/customer-repo.ts` | SQLite adapter: `customers` (seeds when empty; idempotent column migrations) | `SqliteCustomerRepo` |
| `src/db/pricing-store.ts` | SQLite adapter: `price_endpoints`, `tiers` (seeds when empty) | `SqlitePricingStore` |
| `src/db/billing-repo.ts` | SQLite adapter: `billing_reports`, `job_locks` | `SqliteBillingReportRepo`, `BillingReportRepo`, `JobLockRepo` |
| `src/db/invoice-repo.ts` | SQLite adapter: `invoices`, `invoice_lines` | `SqliteInvoiceRepo` |
| `src/db/session-repo.ts` | SQLite adapter: `portal_sessions` | `SqliteSessionRepo` |
| `openapi.yaml` | Public API contract; drives validation and `/docs` | — |
| `admin/` | Dependency-free admin SPA (`index.html`, `app.js`) | — |
| `portal/` | Dependency-free customer portal SPA (`index.html`, `app.js`) | — |
## 3. Pricing engine
### Types (`src/pricing.ts`)
```ts
type PriceRule =
| { kind: 'free' }
| { kind: 'fixed'; fixedCents: number }
| { kind: 'variable'; baseCents: number; perKbCents: number; perMbCents: number };
interface TierConfig {
id: string; name: string;
multiplier: number; // e.g. 0.5 = 50% of list
monthlyCreditCents: number; // free included usage per month
defaultRule?: PriceRule; // fallback for endpoints with no rule
}
interface Quote {
endpointId: string;
listCents: number; // before multiplier
totalCents: number; // after multiplier — this is what is recorded
breakdown: { baseCents: number; metadataCents: number; attachmentCents: number };
}
```
### `quoteCall(pricing, tierId, endpointId, usage, multiplierOverride?)`
1. Resolve tier — throws `Unknown tier: <id>` (mapped to 403 by `meter`).
2. Resolve rule: rate-card rule for `endpointId`, else the tier's
`defaultRule`, else throw `No price rule for <tier>/<endpoint>` (403).
3. Compute breakdown:
- `free` → all zeros.
- `fixed``baseCents = fixedCents`.
- `variable``baseCents` + `perKbCents × ceil(metadataBytes/1024)` +
`perMbCents × ceil(attachmentBytes/1048576)`. Note the ceiling: 1 byte of
metadata bills a full KB unit; attachments bill per started MB.
4. `listCents` = sum of breakdown; `multiplier = multiplierOverride ?? tier.multiplier`;
`totalCents = Math.round(listCents × multiplier)`.
`totalCents` (never `listCents`) is what the usage ledger records and what the
billing pipeline sums.
## 4. Request lifecycle
For `POST /v1/storage`:
1. `express.json()` parses JSON bodies (multipart handled by the validator's
multer — 25 MB per-file cap).
2. `apiKeyAuth` (`src/auth.ts`) — `x-api-key``Customer` on `req.customer`,
else 401.
3. `express-openapi-validator` checks the request against `openapi.yaml`
(400 on violation). For `/v1/storage`, `parseMetadata` then JSON-parses the
`metadata` form field into `res.locals.parsedMetadata` (400 on bad JSON).
4. `meter('storage', usage, pricing)` (`src/meter.ts`):
- measures `metadataBytes` (UTF-8 length of the JSON-stringified metadata)
and `attachmentBytes` (sum of multer file sizes),
- calls `quoteCall` — pricing errors become 403,
- records a `UsageEntry` with `cents = quote.totalCents`,
- stashes the quote in `res.locals.quote`.
5. The route handler builds the `StoredItem` (in-memory list) and responds
`{ id, quote }`.
`GET /v1/usage` is **not** metered; it summarizes the caller's month-to-date
usage and applies the tier credit via `applyMonthlyCredit`.
## 5. Persistence (SQLite)
Single database file (`ZAPPIER_DB`, default `zappier.db`), WAL-agnostic
better-sqlite3, all tables created with `CREATE TABLE IF NOT EXISTS` in the
repo constructors. **Seeding rule:** `customers` and pricing tables seed from
`DEFAULT_CUSTOMERS` / `DEFAULT_RATE_CARD` / `DEFAULT_TIERS` only when empty.
| Table | Columns | Written by |
|---|---|---|
| `usage_entries` | `id`, `customer_id`, `endpoint_id`, `cents`, `metadata_bytes`, `attachment_bytes`, `timestamp_ms` | `meter()` on every priced call |
| `customers` | `id` PK, `name`, `tier_id`, `api_key` UNIQUE, `stripe_customer_id`, `multiplier_override`, `billing_type`, `email`, `password_hash`, `totp_secret`, `totp_enabled`, `balance_cents`, `email_invoicing` | Admin API, portal API |
| `price_endpoints` | `endpoint_id` PK, `rule_json` | Admin API |
| `tiers` | `id` PK, `name`, `multiplier`, `monthly_credit_cents`, `default_rule_json` | Admin API |
| `invoices` | `id` PK, `customer_id`, `period`, `status`, cents totals, `billing_type`, `po_number`, lifecycle timestamps | Admin API (generate/issue/paid + balance drawdown) |
| `invoice_lines` | `invoice_id`, `endpoint_id`, `calls`, `cents` | Invoice generation |
| `portal_sessions` | `token` PK, `customer_id`, `created_ms`, `expires_ms` | Portal auth |
| `billing_reports` | `customer_id` + `period` PK, `reported_cents` (cumulative), `reported_at_ms` | Billing job, after each successful meter event |
| `job_locks` | `name` PK, `acquired_at_ms` | Billing job run guard |
New customer columns are added by **idempotent migrations** (`PRAGMA
table_info` guard + `ALTER TABLE ADD COLUMN`) when the repo opens an older
database — no manual migration step.
In tests, every repo is constructed over `:memory:` databases.
## 6. Billing pipeline
### 6.1 Math (`src/billing/stripe.ts`)
```ts
computeBillableCents(entries, monthlyCreditCents)
= max(0, Σ entry.cents monthlyCreditCents)
computeDelta(billable, previouslyReported)
= max(0, billable previouslyReported)
```
`reportUsage(client, stripeCustomerId, entries, monthlyCreditCents)` is the
original whole-month reporter — **retained as public API**; the job uses the
delta path instead.
### 6.2 The job (`src/jobs/report-usage.ts`)
`reportMonthlyUsage(deps)` per run:
1. **Lock:** `locks.tryAcquireLock('report-usage', 1h TTL)` — a single atomic
`INSERT … ON CONFLICT … DO UPDATE … WHERE acquired_at_ms <= now ttl`.
Failure aborts the run; release happens in `finally`. A crashed run's lock
is taken over after the TTL.
2. **Window:** `since = firstOfMonthUtc(now)` (injectable via `deps.since` for
tests); `period = since.toISOString().slice(0, 7)` (`YYYY-MM`).
3. Per customer with a `stripeCustomerId` (others skipped silently; unknown
`tierId` warns and skips):
- `entries = usage.listFor(customer.id, since)`
- `billable = computeBillableCents(entries, tier.monthlyCreditCents)`
- `prior = billingRepo.getReportedCents(customer.id, period)` (0 for a new
month — periods are isolated by the composite PK)
- `delta = computeDelta(billable, prior)`; `delta <= 0` → log skip, continue
- `createMeterEvent({ eventName: 'zappier.api_cents',
customerId: stripeCustomerId, value: String(delta),
identifier: `${stripeCustomerId}:${period}:${billable}` })`
- **only on success:** `upsertReportedCents(customer.id, period, billable)`
— cumulative, not the delta.
**Idempotency guarantees (reviewed design):**
- *Re-run safety:* second run with same usage → delta 0 → no Stripe call.
- *Mid-month growth:* only the increase is sent; the identifier embeds the new
cumulative billable, so legitimate growth is never deduped away.
- *Crash between Stripe success and ledger write:* retry sends a byte-identical
event; Stripe drops it via the `identifier` (uniqueness enforced within a
rolling 24 h window).
- *Partial failure:* customer A's ledger write commits before customer B is
attempted; B's failure leaves A correctly recorded.
CLI: `npx ts-node src/jobs/report-usage.ts` (guarded by
`require.main === module`; loads `.env` via dotenv inside `main()`).
Scheduled by the Kimi cron job "Zappier billing · report usage to Stripe"
(`17 6 * * *`, America/New_York), which runs this command daily and reports
the outcome.
## 7. API reference
### Public API (`/v1`, auth: `x-api-key`)
Defined in `openapi.yaml`; interactive docs at `/docs`.
| Operation | Method & path | Price | Notes |
|---|---|---|---|
| `status` | GET `/v1/status` | free | Health + quote echo |
| `transform` | POST `/v1/transform` | fixed | `{text}` → `{output: TEXT, quote}` |
| `storage` | POST `/v1/storage` | variable | multipart: `metadata` (JSON string), `attachments[]` (≤25 MB/file) → `{id, quote}` |
| `storage-list` | GET `/v1/storage` | free | Caller's stored items |
| `usage` | GET `/v1/usage` | unmetered | Month-to-date summary with credit applied |
Error envelope: `{ "error": string }` with 400 (validation/metadata),
401 (bad key), 403 (unknown tier / no price rule), 413 (file over 25 MB).
### Admin API (`/admin/api`, auth: `x-admin-key` or login session)
| Route | Purpose |
|---|---|
| POST `/login` | `{username, password}` → session token (accounts: `ADMIN_USER`/`ADMIN_KEY`, `DEMO_ADMIN_USER`/`DEMO_ADMIN_PASSWORD`) |
| GET `/pricing` | `{ rateCard, tiers }` |
| PUT `/endpoints/:id` | Upsert a `PriceRule` (validated: 400 on bad shape) |
| DELETE `/endpoints/:id` | Remove a rule (endpoint becomes 403 for tiers without a default rule) |
| PUT `/tiers/:id` | Upsert a `TierConfig` |
| DELETE `/tiers/:id` | Remove a tier |
| GET `/customers` | List customers **without** API keys |
| POST `/customers` | Create `{name, tierId}` → full customer incl. generated `apiKey` (201, shown once) |
| PUT `/customers/:id` | Patch `name` / `tierId` / `multiplierOverride` / `stripeCustomerId` / `billingType` / `email` |
| POST `/invoices/generate` | `{period, customerId?, poNumber?}` — drafts per customer with usage; regenerating replaces drafts, skips issued/paid → `{generated, skipped}` |
| GET `/invoices` | Filters: `customerId`, `period`, `status` |
| GET `/invoices/:id` | JSON, or print-ready HTML with `?format=html` |
| POST `/invoices/:id/issue` | draft → issued (PO gets 30-day due date). **Prepaid drawdown:** if the customer's `balanceCents` fully covers `billableCents`, the balance is deducted and the invoice is saved as paid instead |
| POST `/invoices/:id/paid` | issued → paid |
| GET `/reports/billing` | `from`/`to`/`customerId`/`billingType` filters; JSON rows or `format=csv` |
| GET `/reports/usage-trend` | `bucket=day\|week`, same range filters |
| GET `/zapier/status` | Zapier app dir presence, version, triggers, creates |
### Portal API (`/portal/api`, auth: Bearer session)
| Route | Purpose |
|---|---|
| POST `/signup` | `{name, email, password≥8}` → 201 `{token, customer}`. New customer on `free` with instant API key; an email match on a passwordless (admin-created) customer **claims** that account; 409 when the email already has a password |
| POST `/login` | `{email, password, totpCode?}` → `{token, customer}`; 401 `totp_required` when 2FA is on and the code is missing/wrong |
| POST `/logout` | Deletes the session |
| GET `/me` | Public profile — never includes `passwordHash`/`totpSecret` |
| POST `/api-key` | Regenerates the API key (old key dies immediately) |
| GET `/usage` | Month-to-date summary with tier credit applied |
| GET `/pricing` | Live rate card + tiers for the pricing page |
| GET `/invoices` · GET `/invoices/:id` | Own invoices only (others 404); `?format=html` print view |
| POST `/2fa/setup` | Generates + stores a TOTP secret (not yet enabled) → `{secret, uri, qr}` (QR as PNG data URL via `qrcode`) |
| POST `/2fa/enable` · POST `/2fa/disable` | `{code}` verified against the stored secret |
| POST `/reload` | `{amountCents}` integer $1$10,000 via the injected `PaymentClient` — dev client credits instantly; Stripe client returns a `clientSecret` and credits on confirmation |
| PUT `/email-invoicing` | `{enabled}` preference |
Sessions live in `portal_sessions` (7-day TTL) and survive restarts.
`src/accounts.ts` implements scrypt hashing (`scrypt:N:r:p:salt:hash`,
timing-safe compare) and RFC 6238 TOTP (HMAC-SHA1, 30 s step, 6 digits,
±1 step window) with no external crypto dependency.
## 8. Zapier app
`zapier-app/` — Zapier Platform (core ^19), CommonJS, mocha tests.
| File | Purpose |
|---|---|
| `index.js` | App definition; wires auth, trigger, action |
| `authentication.js` | API-key auth; test call against `/v1/status` |
| `triggers/new_item.js` | Polling trigger: `GET /v1/storage`, newest first, dedupe by `id` |
| `creates/store_data.js` | Action: multipart `POST /v1/storage` (form-data), fields: metadata JSON + optional files |
| `test/` | mocha suite (4 tests): auth, trigger, action |
Publish flow: `zapier login` → `zapier push` → invite users / submit for
review. The app's base URL must point at a publicly reachable deployment of
the API server.
## 9. Testing
```bash
npm test # jest, repo root — 165 tests / 22 suites
npx tsc --noEmit # type gate
cd zapier-app && npm test # mocha — 4 tests
```
Conventions:
- **TDD** throughout; every module has in-memory adapters so tests never touch
disk or network.
- HTTP tests use **supertest** against `buildApp()` with in-memory repos.
- SQLite tests use `:memory:` databases.
- Stripe is faked by implementing `MeterEventClient`; the idempotency suite
(`tests/report-usage-idempotency.test.ts`) simulates growth, re-runs, month
rollover, Stripe throws, lock contention, and partial failure.
- The job is tested via the injectable `reportMonthlyUsage(deps)` — never by
executing `main()`.
## 10. Configuration
| Env var | Default | Used by |
|---|---|---|
| `PORT` | `3000` | `src/index.ts` |
| `ZAPPIER_DB` | `<root>/zappier.db` | `src/index.ts`, billing job |
| `ADMIN_KEY` | `admin-dev-key` | `adminAuth()` |
| `ADMIN_USER` | `admin` | Admin login |
| `DEMO_ADMIN_USER` / `DEMO_ADMIN_PASSWORD` | `demo` / `$$$Adm1n###` | Demo admin login |
| `STRIPE_SECRET_KEY` | — | Billing job; portal reloads when it starts with `sk_` (otherwise a dev payment client credits instantly) |
Both `src/index.ts` and the billing job load `.env` from the installation
root (`src/paths.ts` `PROJECT_ROOT`) — never from the process cwd — so the
compiled server and the job run from any working directory.
`.env` is gitignored (`chmod 600`); `.env.example` documents the shape.
Git identity is configured repo-local; `.gitignore` covers `node_modules/`,
`dist/`, `.env`, `zappier.db*`.
## 11. Extending the system
**Add an API endpoint:**
1. Add the path + `operationId` to `openapi.yaml` (validation & docs follow
automatically).
2. Add the route in `src/app.ts`, wrapping the handler with
`meter('<operationId>', usage, pricing)`.
3. Add a rate-card rule (admin UI or `PUT /admin/api/endpoints/<operationId>`)
— otherwise tiers without a `defaultRule` get 403.
4. Write the failing test first; keep `npm test` + `tsc` green.
**Add a customer type:** admin UI or `PUT /admin/api/tiers/:id`
(`{name, multiplier, monthlyCreditCents, defaultRule?}`).
**Swap the storage backend:** implement `UsageRepo` / `CustomerRepo` /
`PricingStore` against your database and pass them to `buildApp({...})` — no
other code changes. Same for `BillingReportRepo`/`JobLockRepo` in the job.
**Change the billing cadence:** the job is safe at any frequency (delta +
ledger + lock). The Kimi cron job controls scheduling; update its cron
expression to change cadence.
**Known intentional limitations:** stored items are in-memory (restart clears
them; usage ledger is unaffected); `reportUsage` is retained but superseded by
the delta path; Stripe identifier dedup covers a rolling 24 h window;
`releaseLock` is not owner-scoped (harmless at this job's runtime); admin
session tokens are in-memory (portal sessions are persisted); Stripe reloads
credit the balance only after payment confirmation (no webhook endpoint yet —
dev client credits instantly); email invoicing stores the preference but
sending requires SMTP wiring (deferred); PO invoices with partial prepaid
coverage are not partially paid by design.