marchon/zappier-edge
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Initial import of zappier-edge from zapier monorepo

Browse source
This commit is contained in:
George Lambert 2026-09-11 13:16:00 -04:00
commit 9d72cecabd
120 changed files with 19867 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

406
docs/DEVELOPER.md Normal file
View file

@ -0,0 +1,406 @@
# Zappier — Developer Documentation
**Version:** 0.2.0 · **Last updated:** 2026-07-28
Metered API platform: per-endpoint pricing, per-tier multipliers, usage
ledger, Stripe metered billing, purchase-order invoicing, a company admin
console, a customer portal with TOTP 2FA, and a Zapier integration. This
document is the full technical reference. For operations, see
[USER-MANUAL.md](USER-MANUAL.md).
---
## Table of contents
1. [Architecture](#1-architecture)
2. [Module reference](#2-module-reference)
3. [Pricing engine](#3-pricing-engine)
4. [Request lifecycle](#4-request-lifecycle)
5. [Persistence (SQLite)](#5-persistence-sqlite)
6. [Billing pipeline](#6-billing-pipeline)
7. [API reference](#7-api-reference)
8. [Zapier app](#8-zapier-app)
9. [Testing](#9-testing)
10. [Configuration](#10-configuration)
11. [Extending the system](#11-extending-the-system)
---
## 1. Architecture
**Stack:** Node 20 · TypeScript (strict) · Express 4 · better-sqlite3 ·
express-openapi-validator · swagger-ui-express · stripe SDK · Zapier Platform
(core ^19) · jest + supertest (root) and mocha (zapier-app).
```
┌────────────────────────────────────────────┐
│ Express app │
│ (src/app.ts) │
│ │
Browser ── /admin ──────┼── static admin UI (admin/) │
Browser ── /admin/api ──┼── adminAuth → adminRouter (src/admin.ts) │
│ pricing · customers · invoices · │
│ reports · zapier status │
Browser ── /portal ─────┼── static portal SPA (portal/) │
Browser ── /portal/api ─┼── portalRouter (src/portal.ts) │
│ signup/login+2FA · me · usage · │
│ invoices · reload · prefs │
Browser ── /docs ───────┼── swagger-ui (openapi.yaml) │
│ │
Client ──── /v1/* ──────┼── apiKeyAuth (src/auth.ts) │
│ └─ OpenAPI request validation │
│ └─ meter() (src/meter.ts) │
│ ├─ quoteCall() (src/pricing.ts) │
│ └─ UsageRepo.record() │
└──────────────┬─────────────────────────────┘
│
┌───────────────┬───────────────┼───────────────┬────────────┐
│ │ │ │ │
SqliteUsageRepo SqliteCustomerRepo SqlitePricingStore SqliteInvoiceRepo
(usage_entries) (customers) (price_endpoints, (invoices,
│ │ tiers) invoice_lines)
│ SqliteSessionRepo (portal_sessions) │
└───────────────┴───────────────┴───────────────┴────────────┘
│ zappier.db (SQLite)
Cron (daily 06:17 ET) │
"Zappier billing · report usage" │
│ │
└─► src/jobs/report-usage.ts ── reportMonthlyUsage()
├─ computeBillableCents / computeDelta (src/billing/stripe.ts)
├─ SqliteBillingReportRepo (billing_reports, job_locks)
└─ stripe.billing.meterEvents.create ──► Stripe
```
Design rules the codebase follows:
- **Ports & adapters:** `UsageRepo`, `CustomerRepo`, `PricingStore`,
`MeterEventClient` are interfaces with in-memory adapters (tests) and SQLite
/ Stripe adapters (production). Nothing outside `src/db/` and the job's
`main()` touches SQL or the Stripe SDK.
- **Live pricing reads:** the `PricingContext` getters in `buildApp` read the
store on every quote, so admin edits apply without a restart.
- **Money in integer cents everywhere.** No floats cross a boundary except
multipliers, which are applied once and rounded (`Math.round`).
## 2. Module reference
| File | Responsibility | Key exports |
|---|---|---|
| `src/index.ts` | Production entry: opens SQLite, wires repos, listens | — |
| `src/app.ts` | `buildApp(deps)` — full Express wiring; `StoredItem`; `DEFAULT_CUSTOMERS` | `buildApp`, `AppDeps` |
| `src/auth.ts` | Customer model + `x-api-key` middleware | `Customer`, `CustomerRepo`, `apiKeyAuth`, `InMemoryCustomerRepo` |
| `src/pricing.ts` | Pricing domain: rules, tiers, quote algorithm, store port | `PriceRule`, `TierConfig`, `Quote`, `quoteCall`, `PricingStore`, `DEFAULT_RATE_CARD`, `DEFAULT_TIERS` |
| `src/meter.ts` | Per-request metering middleware | `meter(endpointId, repo, pricing)` |
| `src/usage.ts` | Usage ledger domain + summaries | `UsageEntry`, `UsageSummary`, `summarize`, `UsageRepo`, `InMemoryUsageRepo` |
| `src/admin.ts` | Admin API (`/admin/api`) + admin-key guard | `adminAuth`, `adminRouter` |
| `src/billing/credit.ts` | Monthly credit application | `applyMonthlyCredit`, `BilledSummary` |
| `src/billing/stripe.ts` | Stripe-facing math + client port | `METER_EVENT_NAME`, `MeterEventClient`, `computeBillableCents`, `computeDelta`, `reportUsage` |
| `src/billing/reload.ts` | Portal reloads via Stripe PaymentIntents | `stripePaymentClient`, `hasRealStripeKey` |
| `src/accounts.ts` | Portal identity: scrypt passwords, RFC 6238 TOTP, sessions | `hashPassword`, `verifyPassword`, `totp`, `verifyTotp`, `generateTotpSecret`, `totpUri`, `SessionRepo`, `InMemorySessionRepo` |
| `src/invoicing.ts` | Invoice domain + generation | `Invoice`, `InvoiceLine`, `buildInvoice`, `InvoiceRepo`, `InMemoryInvoiceRepo` |
| `src/reports.ts` | Billing/usage aggregation + CSV | `billingRows`, `usageTrend`, `toCsv`, `BillingRow`, `TrendPoint` |
| `src/portal.ts` | Customer portal API (`/portal/api`) | `portalRouter`, `PortalDeps`, `PaymentClient`, `PaymentResult` |
| `src/paths.ts` | Installation-root resolution (cwd-independent) | `PROJECT_ROOT` |
| `src/jobs/report-usage.ts` | Billing job: delta reporting to Stripe | `reportMonthlyUsage`, `firstOfMonthUtc`, `ReportUsageDeps` |
| `src/db/usage-repo.ts` | SQLite adapter: `usage_entries` | `SqliteUsageRepo` |
| `src/db/customer-repo.ts` | SQLite adapter: `customers` (seeds when empty; idempotent column migrations) | `SqliteCustomerRepo` |
| `src/db/pricing-store.ts` | SQLite adapter: `price_endpoints`, `tiers` (seeds when empty) | `SqlitePricingStore` |
| `src/db/billing-repo.ts` | SQLite adapter: `billing_reports`, `job_locks` | `SqliteBillingReportRepo`, `BillingReportRepo`, `JobLockRepo` |
| `src/db/invoice-repo.ts` | SQLite adapter: `invoices`, `invoice_lines` | `SqliteInvoiceRepo` |
| `src/db/session-repo.ts` | SQLite adapter: `portal_sessions` | `SqliteSessionRepo` |
| `openapi.yaml` | Public API contract; drives validation and `/docs` | — |
| `admin/` | Dependency-free admin SPA (`index.html`, `app.js`) | — |
| `portal/` | Dependency-free customer portal SPA (`index.html`, `app.js`) | — |
## 3. Pricing engine
### Types (`src/pricing.ts`)
```ts
type PriceRule =
| { kind: 'free' }
| { kind: 'fixed'; fixedCents: number }
| { kind: 'variable'; baseCents: number; perKbCents: number; perMbCents: number };
interface TierConfig {
id: string; name: string;
multiplier: number; // e.g. 0.5 = 50% of list
monthlyCreditCents: number; // free included usage per month
defaultRule?: PriceRule; // fallback for endpoints with no rule
}
interface Quote {
endpointId: string;
listCents: number; // before multiplier
totalCents: number; // after multiplier — this is what is recorded
breakdown: { baseCents: number; metadataCents: number; attachmentCents: number };
}
```
### `quoteCall(pricing, tierId, endpointId, usage, multiplierOverride?)`
1. Resolve tier — throws `Unknown tier: <id>` (mapped to 403 by `meter`).
2. Resolve rule: rate-card rule for `endpointId`, else the tier's
`defaultRule`, else throw `No price rule for <tier>/<endpoint>` (403).
3. Compute breakdown:
- `free` → all zeros.
- `fixed` → `baseCents = fixedCents`.
- `variable` → `baseCents` + `perKbCents × ceil(metadataBytes/1024)` +
`perMbCents × ceil(attachmentBytes/1048576)`. Note the ceiling: 1 byte of
metadata bills a full KB unit; attachments bill per started MB.
4. `listCents` = sum of breakdown; `multiplier = multiplierOverride ?? tier.multiplier`;
`totalCents = Math.round(listCents × multiplier)`.
`totalCents` (never `listCents`) is what the usage ledger records and what the
billing pipeline sums.
## 4. Request lifecycle
For `POST /v1/storage`:
1. `express.json()` parses JSON bodies (multipart handled by the validator's
multer — 25 MB per-file cap).
2. `apiKeyAuth` (`src/auth.ts`) — `x-api-key` → `Customer` on `req.customer`,
else 401.
3. `express-openapi-validator` checks the request against `openapi.yaml`
(400 on violation). For `/v1/storage`, `parseMetadata` then JSON-parses the
`metadata` form field into `res.locals.parsedMetadata` (400 on bad JSON).
4. `meter('storage', usage, pricing)` (`src/meter.ts`):
- measures `metadataBytes` (UTF-8 length of the JSON-stringified metadata)
and `attachmentBytes` (sum of multer file sizes),
- calls `quoteCall` — pricing errors become 403,
- records a `UsageEntry` with `cents = quote.totalCents`,
- stashes the quote in `res.locals.quote`.
5. The route handler builds the `StoredItem` (in-memory list) and responds
`{ id, quote }`.
`GET /v1/usage` is **not** metered; it summarizes the caller's month-to-date
usage and applies the tier credit via `applyMonthlyCredit`.
## 5. Persistence (SQLite)
Single database file (`ZAPPIER_DB`, default `zappier.db`), WAL-agnostic
better-sqlite3, all tables created with `CREATE TABLE IF NOT EXISTS` in the
repo constructors. **Seeding rule:** `customers` and pricing tables seed from
`DEFAULT_CUSTOMERS` / `DEFAULT_RATE_CARD` / `DEFAULT_TIERS` only when empty.
| Table | Columns | Written by |
|---|---|---|
| `usage_entries` | `id`, `customer_id`, `endpoint_id`, `cents`, `metadata_bytes`, `attachment_bytes`, `timestamp_ms` | `meter()` on every priced call |
| `customers` | `id` PK, `name`, `tier_id`, `api_key` UNIQUE, `stripe_customer_id`, `multiplier_override`, `billing_type`, `email`, `password_hash`, `totp_secret`, `totp_enabled`, `balance_cents`, `email_invoicing` | Admin API, portal API |
| `price_endpoints` | `endpoint_id` PK, `rule_json` | Admin API |
| `tiers` | `id` PK, `name`, `multiplier`, `monthly_credit_cents`, `default_rule_json` | Admin API |
| `invoices` | `id` PK, `customer_id`, `period`, `status`, cents totals, `billing_type`, `po_number`, lifecycle timestamps | Admin API (generate/issue/paid + balance drawdown) |
| `invoice_lines` | `invoice_id`, `endpoint_id`, `calls`, `cents` | Invoice generation |
| `portal_sessions` | `token` PK, `customer_id`, `created_ms`, `expires_ms` | Portal auth |
| `billing_reports` | `customer_id` + `period` PK, `reported_cents` (cumulative), `reported_at_ms` | Billing job, after each successful meter event |
| `job_locks` | `name` PK, `acquired_at_ms` | Billing job run guard |
New customer columns are added by **idempotent migrations** (`PRAGMA
table_info` guard + `ALTER TABLE ADD COLUMN`) when the repo opens an older
database — no manual migration step.
In tests, every repo is constructed over `:memory:` databases.
## 6. Billing pipeline
### 6.1 Math (`src/billing/stripe.ts`)
```ts
computeBillableCents(entries, monthlyCreditCents)
= max(0, Σ entry.cents − monthlyCreditCents)
computeDelta(billable, previouslyReported)
= max(0, billable − previouslyReported)
```
`reportUsage(client, stripeCustomerId, entries, monthlyCreditCents)` is the
original whole-month reporter — **retained as public API**; the job uses the
delta path instead.
### 6.2 The job (`src/jobs/report-usage.ts`)
`reportMonthlyUsage(deps)` per run:
1. **Lock:** `locks.tryAcquireLock('report-usage', 1h TTL)` — a single atomic
`INSERT … ON CONFLICT … DO UPDATE … WHERE acquired_at_ms <= now − ttl`.
Failure aborts the run; release happens in `finally`. A crashed run's lock
is taken over after the TTL.
2. **Window:** `since = firstOfMonthUtc(now)` (injectable via `deps.since` for
tests); `period = since.toISOString().slice(0, 7)` (`YYYY-MM`).
3. Per customer with a `stripeCustomerId` (others skipped silently; unknown
`tierId` warns and skips):
- `entries = usage.listFor(customer.id, since)`
- `billable = computeBillableCents(entries, tier.monthlyCreditCents)`
- `prior = billingRepo.getReportedCents(customer.id, period)` (0 for a new
month — periods are isolated by the composite PK)
- `delta = computeDelta(billable, prior)`; `delta <= 0` → log skip, continue
- `createMeterEvent({ eventName: 'zappier.api_cents',
customerId: stripeCustomerId, value: String(delta),
identifier: `${stripeCustomerId}:${period}:${billable}` })`
- **only on success:** `upsertReportedCents(customer.id, period, billable)`
— cumulative, not the delta.
**Idempotency guarantees (reviewed design):**
- *Re-run safety:* second run with same usage → delta 0 → no Stripe call.
- *Mid-month growth:* only the increase is sent; the identifier embeds the new
cumulative billable, so legitimate growth is never deduped away.
- *Crash between Stripe success and ledger write:* retry sends a byte-identical
event; Stripe drops it via the `identifier` (uniqueness enforced within a
rolling 24 h window).
- *Partial failure:* customer A's ledger write commits before customer B is
attempted; B's failure leaves A correctly recorded.
CLI: `npx ts-node src/jobs/report-usage.ts` (guarded by
`require.main === module`; loads `.env` via dotenv inside `main()`).
Scheduled by the Kimi cron job "Zappier billing · report usage to Stripe"
(`17 6 * * *`, America/New_York), which runs this command daily and reports
the outcome.
## 7. API reference
### Public API (`/v1`, auth: `x-api-key`)
Defined in `openapi.yaml`; interactive docs at `/docs`.
| Operation | Method & path | Price | Notes |
|---|---|---|---|
| `status` | GET `/v1/status` | free | Health + quote echo |
| `transform` | POST `/v1/transform` | fixed | `{text}` → `{output: TEXT, quote}` |
| `storage` | POST `/v1/storage` | variable | multipart: `metadata` (JSON string), `attachments[]` (≤25 MB/file) → `{id, quote}` |
| `storage-list` | GET `/v1/storage` | free | Caller's stored items |
| `usage` | GET `/v1/usage` | unmetered | Month-to-date summary with credit applied |
Error envelope: `{ "error": string }` with 400 (validation/metadata),
401 (bad key), 403 (unknown tier / no price rule), 413 (file over 25 MB).
### Admin API (`/admin/api`, auth: `x-admin-key` or login session)
| Route | Purpose |
|---|---|
| POST `/login` | `{username, password}` → session token (accounts: `ADMIN_USER`/`ADMIN_KEY`, `DEMO_ADMIN_USER`/`DEMO_ADMIN_PASSWORD`) |
| GET `/pricing` | `{ rateCard, tiers }` |
| PUT `/endpoints/:id` | Upsert a `PriceRule` (validated: 400 on bad shape) |
| DELETE `/endpoints/:id` | Remove a rule (endpoint becomes 403 for tiers without a default rule) |
| PUT `/tiers/:id` | Upsert a `TierConfig` |
| DELETE `/tiers/:id` | Remove a tier |
| GET `/customers` | List customers **without** API keys |
| POST `/customers` | Create `{name, tierId}` → full customer incl. generated `apiKey` (201, shown once) |
| PUT `/customers/:id` | Patch `name` / `tierId` / `multiplierOverride` / `stripeCustomerId` / `billingType` / `email` |
| POST `/invoices/generate` | `{period, customerId?, poNumber?}` — drafts per customer with usage; regenerating replaces drafts, skips issued/paid → `{generated, skipped}` |
| GET `/invoices` | Filters: `customerId`, `period`, `status` |
| GET `/invoices/:id` | JSON, or print-ready HTML with `?format=html` |
| POST `/invoices/:id/issue` | draft → issued (PO gets 30-day due date). **Prepaid drawdown:** if the customer's `balanceCents` fully covers `billableCents`, the balance is deducted and the invoice is saved as paid instead |
| POST `/invoices/:id/paid` | issued → paid |
| GET `/reports/billing` | `from`/`to`/`customerId`/`billingType` filters; JSON rows or `format=csv` |
| GET `/reports/usage-trend` | `bucket=day\|week`, same range filters |
| GET `/zapier/status` | Zapier app dir presence, version, triggers, creates |
### Portal API (`/portal/api`, auth: Bearer session)
| Route | Purpose |
|---|---|
| POST `/signup` | `{name, email, password≥8}` → 201 `{token, customer}`. New customer on `free` with instant API key; an email match on a passwordless (admin-created) customer **claims** that account; 409 when the email already has a password |
| POST `/login` | `{email, password, totpCode?}` → `{token, customer}`; 401 `totp_required` when 2FA is on and the code is missing/wrong |
| POST `/logout` | Deletes the session |
| GET `/me` | Public profile — never includes `passwordHash`/`totpSecret` |
| POST `/api-key` | Regenerates the API key (old key dies immediately) |
| GET `/usage` | Month-to-date summary with tier credit applied |
| GET `/pricing` | Live rate card + tiers for the pricing page |
| GET `/invoices` · GET `/invoices/:id` | Own invoices only (others 404); `?format=html` print view |
| POST `/2fa/setup` | Generates + stores a TOTP secret (not yet enabled) → `{secret, uri, qr}` (QR as PNG data URL via `qrcode`) |
| POST `/2fa/enable` · POST `/2fa/disable` | `{code}` verified against the stored secret |
| POST `/reload` | `{amountCents}` integer $1–$10,000 via the injected `PaymentClient` — dev client credits instantly; Stripe client returns a `clientSecret` and credits on confirmation |
| PUT `/email-invoicing` | `{enabled}` preference |
Sessions live in `portal_sessions` (7-day TTL) and survive restarts.
`src/accounts.ts` implements scrypt hashing (`scrypt:N:r:p:salt:hash`,
timing-safe compare) and RFC 6238 TOTP (HMAC-SHA1, 30 s step, 6 digits,
±1 step window) with no external crypto dependency.
## 8. Zapier app
`zapier-app/` — Zapier Platform (core ^19), CommonJS, mocha tests.
| File | Purpose |
|---|---|
| `index.js` | App definition; wires auth, trigger, action |
| `authentication.js` | API-key auth; test call against `/v1/status` |
| `triggers/new_item.js` | Polling trigger: `GET /v1/storage`, newest first, dedupe by `id` |
| `creates/store_data.js` | Action: multipart `POST /v1/storage` (form-data), fields: metadata JSON + optional files |
| `test/` | mocha suite (4 tests): auth, trigger, action |
Publish flow: `zapier login` → `zapier push` → invite users / submit for
review. The app's base URL must point at a publicly reachable deployment of
the API server.
## 9. Testing
```bash
npm test # jest, repo root — 165 tests / 22 suites
npx tsc --noEmit # type gate
cd zapier-app && npm test # mocha — 4 tests
```
Conventions:
- **TDD** throughout; every module has in-memory adapters so tests never touch
disk or network.
- HTTP tests use **supertest** against `buildApp()` with in-memory repos.
- SQLite tests use `:memory:` databases.
- Stripe is faked by implementing `MeterEventClient`; the idempotency suite
(`tests/report-usage-idempotency.test.ts`) simulates growth, re-runs, month
rollover, Stripe throws, lock contention, and partial failure.
- The job is tested via the injectable `reportMonthlyUsage(deps)` — never by
executing `main()`.
## 10. Configuration
| Env var | Default | Used by |
|---|---|---|
| `PORT` | `3000` | `src/index.ts` |
| `ZAPPIER_DB` | `<root>/zappier.db` | `src/index.ts`, billing job |
| `ADMIN_KEY` | `admin-dev-key` | `adminAuth()` |
| `ADMIN_USER` | `admin` | Admin login |
| `DEMO_ADMIN_USER` / `DEMO_ADMIN_PASSWORD` | `demo` / `$$$Adm1n###` | Demo admin login |
| `STRIPE_SECRET_KEY` | — | Billing job; portal reloads when it starts with `sk_` (otherwise a dev payment client credits instantly) |
Both `src/index.ts` and the billing job load `.env` from the installation
root (`src/paths.ts` `PROJECT_ROOT`) — never from the process cwd — so the
compiled server and the job run from any working directory.
`.env` is gitignored (`chmod 600`); `.env.example` documents the shape.
Git identity is configured repo-local; `.gitignore` covers `node_modules/`,
`dist/`, `.env`, `zappier.db*`.
## 11. Extending the system
**Add an API endpoint:**
1. Add the path + `operationId` to `openapi.yaml` (validation & docs follow
automatically).
2. Add the route in `src/app.ts`, wrapping the handler with
`meter('<operationId>', usage, pricing)`.
3. Add a rate-card rule (admin UI or `PUT /admin/api/endpoints/<operationId>`)
— otherwise tiers without a `defaultRule` get 403.
4. Write the failing test first; keep `npm test` + `tsc` green.
**Add a customer type:** admin UI or `PUT /admin/api/tiers/:id`
(`{name, multiplier, monthlyCreditCents, defaultRule?}`).
**Swap the storage backend:** implement `UsageRepo` / `CustomerRepo` /
`PricingStore` against your database and pass them to `buildApp({...})` — no
other code changes. Same for `BillingReportRepo`/`JobLockRepo` in the job.
**Change the billing cadence:** the job is safe at any frequency (delta +
ledger + lock). The Kimi cron job controls scheduling; update its cron
expression to change cadence.
**Known intentional limitations:** stored items are in-memory (restart clears
them; usage ledger is unaffected); `reportUsage` is retained but superseded by
the delta path; Stripe identifier dedup covers a rolling 24 h window;
`releaseLock` is not owner-scoped (harmless at this job's runtime); admin
session tokens are in-memory (portal sessions are persisted); Stripe reloads
credit the balance only after payment confirmation (no webhook endpoint yet —
dev client credits instantly); email invoicing stores the preference but
sending requires SMTP wiring (deferred); PO invoices with partial prepaid
coverage are not partially paid by design.