Milestone 0: import zappier billing, Verae middleware, and Zapier research
Compose-ready workspace: packages/zappier (rate card, portal, Stripe), packages/verae-zapier-middleware (timestamp + NATS), packages/verae-zapier (CLI app), vendor/zapier-platform, and research/zapier vendor corpus. Gate 0 structure checks pass. Product code and research are not yet wired.
This commit is contained in:
commit
b4150c8250
1364 changed files with 6814366 additions and 0 deletions
729
docs/00-sources/getting-started-research.md
Normal file
729
docs/00-sources/getting-started-research.md
Normal file
|
|
@ -0,0 +1,729 @@
|
|||
# Getting started — Verae Time × Zapier
|
||||
|
||||
**System architecture, functionality, integration guide, and next steps**
|
||||
|
||||
This is the working document for the research workspace at `/Users/marchon/research/zapier`. It covers the full stack that connects Zapier automations to the Verae Timestamping Service, how to run and extend it locally, Zapier vs Verae billing, a client pattern for timestamped files on Peergos/IPFS with cold retrieve-on-demand, and what remains before a private Zapier listing.
|
||||
|
||||
Grok (and humans) should start here, then follow the playbook in [PLATFORM-REFERENCE.md](PLATFORM-REFERENCE.md).
|
||||
|
||||
---
|
||||
|
||||
## 1. How to start this workspace
|
||||
|
||||
```bash
|
||||
cd /Users/marchon/research/zapier
|
||||
./scripts/restart-grok.sh # Mongo tunnel + grok --resume
|
||||
```
|
||||
|
||||
Already in Grok:
|
||||
|
||||
```
|
||||
/zapier-build
|
||||
```
|
||||
|
||||
Mandatory first query (same playbook as [PLATFORM-REFERENCE.md](PLATFORM-REFERENCE.md)):
|
||||
|
||||
```js
|
||||
db.platform_reference.findOne({ kind: "guide", key: "build-new-connector" })
|
||||
```
|
||||
|
||||
Keep the Mongo tunnel up (`./scripts/ensure-mongo-tunnel.sh`). Check readiness with `./scripts/zapier-status.sh`. Load CLI PATH and env with `source scripts/dev-env.sh`.
|
||||
|
||||
Leave the TUI with `/quit` (not `/new`). Restart details: [RESTART.md](RESTART.md).
|
||||
|
||||
---
|
||||
|
||||
## 2. System architecture
|
||||
|
||||
### 2.1 Purpose
|
||||
|
||||
Connect Zapier automations to Verae blockchain timestamping **without**:
|
||||
|
||||
- exposing raw Verae JWTs to Zapier end users
|
||||
- requiring Zapier to poll async jobs on `api.veraetime.net`
|
||||
- coupling billing and plan limits to the core timestamping API
|
||||
- running a multi-instance edge with only in-memory job queues
|
||||
|
||||
### 2.2 Bottom line
|
||||
|
||||

|
||||
|
||||
```text
|
||||
Users → Zapier UI
|
||||
Zapier cloud runs the Platform CLI app (verae-zapier or scratch/veraetime)
|
||||
→ HTTPS only → verae-zapier-middleware /zapier/v1/*
|
||||
→ (sync) HTTPS → https://api.veraetime.net
|
||||
→ (async) NATS JetStream → workers
|
||||
→ HTTPS → api.veraetime.net (status poll)
|
||||
→ HTTPS → hooks.zapier.com (REST Hook delivery)
|
||||
```
|
||||
|
||||
- Zapier **never** connects to NATS.
|
||||
- Zapier **never** calls `api.veraetime.net` directly.
|
||||
- Middleware HTTP owns auth, tenancy, entitlements, metering, and the public API surface.
|
||||
- NATS owns durable job watching, completion events, and reliable webhook delivery (when `NATS_ENABLED=true`).
|
||||
- With `NATS_ENABLED=false`, an in-process poller still implements the same HTTP product path (Phase 6).
|
||||
|
||||
### 2.3 Components
|
||||
|
||||
| Component | Runs where | Role |
|
||||
|-----------|------------|------|
|
||||
| **Zapier Platform CLI app** | Zapier cloud (when a Zap step runs) | Auth fields, map operations to `/zapier/v1/*`, attach Bearer token, translate 402/403 |
|
||||
| **verae-zapier-middleware** | Your infrastructure, public HTTPS | Tenant identity, API keys, Verae login bridge, entitlements, REST Hook storage, publish work to NATS |
|
||||
| **NATS + JetStream** | Private network with middleware | Work queues for job watch and webhook delivery; event stream for terminal job states |
|
||||
| **Workers** | Same deploy or separate processes | Job poller, event router, webhook deliverer |
|
||||
| **api.veraetime.net** | Verae production | Source of truth: login, timestamp jobs, status, verification |
|
||||
|
||||
#### Zapier app (two copies in this repo)
|
||||
|
||||
| Path | Language | Auth | Status |
|
||||
|------|----------|------|--------|
|
||||
| `verae-zapier-api/verae-zapier/` | JavaScript | Custom API key (`zmw_…`) | Phase 11 package; `MIDDLEWARE_BASE_URL` env |
|
||||
| `scratch/veraetime/` | TypeScript (CLI 19.1.0) | Session: username/password **or** API key → `accessToken` | Local golden connector; `build` + `validate` |
|
||||
|
||||
Both talk **only** to middleware. The TypeScript app is the one this workspace validates day-to-day. The JavaScript app is the vendored product package from the middleware monorepo.
|
||||
|
||||
**The Zapier app does not:** call `api.veraetime.net`, speak NATS, or enforce plan quotas.
|
||||
|
||||
#### Middleware HTTP edge
|
||||
|
||||

|
||||
|
||||
Path: `verae-zapier-api/verae-zapier-middleware/`
|
||||
|
||||
- Express app: `GET /health`, mount `/zapier`
|
||||
- Public: `POST /zapier/v1/auth/login`, `GET /zapier/v1/auth/me`, tenant signup
|
||||
- Protected (Bearer or `x-api-key` + rate limit): timestamp, verify, status, webhooks
|
||||
- Admin: `/zapier/v1/admin/*` behind `X-Admin-Secret`
|
||||
- Store: file JSON MVP (`STORE_PATH`, default `./data/store.json`) — single-node only
|
||||
|
||||
#### NATS + workers
|
||||
|
||||
| Worker | Consumes | Calls |
|
||||
|--------|----------|-------|
|
||||
| Job poller | `verae.zapier.jobs.watch` | `GET /api/status/{jobId}` on Verae |
|
||||
| Event router | `verae.zapier.jobs.events` | Enqueues webhook deliveries |
|
||||
| Webhook deliver | `verae.zapier.webhooks.deliver` | `POST` Zapier `targetUrl` |
|
||||
|
||||
Streams: `ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS` (optional `ZAPIER_USAGE`).
|
||||
|
||||
#### Verae Timestamping Service
|
||||
|
||||
- Swagger UI: https://api.veraetime.net/docs/swagger/index.html
|
||||
- OpenAPI: https://api.veraetime.net/docs/swagger/openapi.yaml
|
||||
- Local copy: [scratch/our-api/openapi.yaml](scratch/our-api/openapi.yaml)
|
||||
- Auth: `POST /auth/login` → JWT in `token`; all other API routes `Authorization: Bearer <token>`
|
||||
- Async create: `POST /api/timestamp` → **202** `{ jobId }`
|
||||
|
||||
### 2.4 Security boundaries
|
||||
|
||||

|
||||
|
||||
```text
|
||||
Public Internet
|
||||
├─ Zapier cloud → Middleware HTTPS only
|
||||
└─ Middleware → Zapier REST Hook HTTPS only
|
||||
|
||||
Private
|
||||
├─ Middleware ↔ NATS (never expose NATS ports)
|
||||
└─ Middleware / workers → api.veraetime.net HTTPS
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Never put raw Verae JWTs in NATS when a `tokenRef` will do.
|
||||
- Never expose NATS (`4222`) to the public internet.
|
||||
- Debug logs must redact `Bearer`, `zmw_`, `zmt_`, passwords, and hook query secrets.
|
||||
- Treat `targetUrl` as untrusted egress (timeouts; SSRF allowlist is Phase 15).
|
||||
- Do not commit `~/.zapierrc`, `.env`, `store.json`, or `~/.mcp-env`.
|
||||
|
||||
### 2.5 Scaling model
|
||||
|
||||
- **HTTP edge:** stateless replicas behind a load balancer, **except** they share a store. File JSON is single-node. Multi-node needs Postgres/Redis (Phase 15).
|
||||
- **NATS consumers:** queue groups — more workers increase poll/deliver throughput; two workers must not double-complete the same job.
|
||||
- **Rollback:** `NATS_ENABLED=false` still serves the full HTTP API with the in-process poller.
|
||||
|
||||
---
|
||||
|
||||
## 3. Functionality
|
||||
|
||||
### 3.1 Authentication (two hops)
|
||||
|
||||

|
||||
|
||||
**End user → middleware**
|
||||
|
||||
| Mode | What the user enters | What happens |
|
||||
|------|----------------------|--------------|
|
||||
| Session (TypeScript app) | Username + password, **or** middleware API key | `POST /zapier/v1/auth/login` → `{ accessToken }` stored as `sessionKey` |
|
||||
| Custom key (JS app) | Tenant API key `zmw_…` | Sent as `Authorization: Bearer` on every request; test is `GET /zapier/v1/auth/me` |
|
||||
|
||||
Connection test: `GET /zapier/v1/auth/me` → tenant, plan, usage. Connection label: username/plan (TS) or `tenantId (plan)` (JS).
|
||||
|
||||
401 on later calls: TypeScript app throws `z.errors.RefreshAuthError` so Zapier re-runs session `perform`.
|
||||
|
||||
**Middleware → Verae**
|
||||
|
||||
Middleware logs into Verae with the tenant’s Verae credentials (or mock client when `MOCK_VERAE=true`) and holds the JWT **server-side**. Zapier never sees that JWT.
|
||||
|
||||
Token types issued by middleware:
|
||||
|
||||
| Prefix | Kind |
|
||||
|--------|------|
|
||||
| `zmw_` | Tenant API key (long-lived) |
|
||||
| `zmt_` | Middleware session token (HMAC, `TOKEN_SECRET`) |
|
||||
|
||||
### 3.2 Zapier operations ↔ middleware ↔ Verae
|
||||
|
||||

|
||||
|
||||
All Zapier routes are under `/zapier/v1`. Connector default base: `http://127.0.0.1:3100`. Production base is the **deployed middleware**, not `api.veraetime.net`.
|
||||
|
||||
| Zapier noun | Type | Middleware | Verae (what middleware wraps) | Notes |
|
||||
|-------------|------|------------|-------------------------------|-------|
|
||||
| Create Timestamp | create | `POST /timestamp` | `POST /api/timestamp` | Returns `{ jobId }` (202). Use with the hook trigger. |
|
||||
| Create Timestamp and Wait | create | `POST /timestamp/wait` | create + poll/wait | Returns terminal status, or pending + `jobId` on timeout. |
|
||||
| Create Batch Timestamps | create | `POST /timestamp/batch` | `POST /api/batch/timestamp` | `{ items: [{ data, hashAlg? }] }` |
|
||||
| Verify Certificate | create | `POST /verify` | `POST /api/verify` | `{ certificate }` → `{ valid, timestamp, blockIndex }` |
|
||||
| Find Job Status | search | `GET /status/{jobId}` | `GET /api/status/{jobId}` | Search: empty array if 404. |
|
||||
| Find Job Verification | search (TS only) | `GET /status/{jobId}/verification` | `GET /api/verify/{jobId}` | Search. |
|
||||
| Timestamp Completed | hook trigger | `POST /webhooks/subscribe` · `DELETE /webhooks/unsubscribe` | (no Verae hook) | Event `timestamp.completed`; Zapier supplies `targetUrl`. |
|
||||
|
||||
Not in v1 (admin HTML / user admin on Verae): dashboard, metrics, queue UI, user CRUD, batch verify/status, get-block-by-hash. Add later if a Zap needs them.
|
||||
|
||||
Create input (timestamp): required `data` (text), optional `hashAlg` (default SHA256).
|
||||
|
||||
### 3.3 Request flows
|
||||
|
||||
**A. Create Timestamp (async + hook)**
|
||||
|
||||

|
||||
|
||||
```text
|
||||
Zapier → POST /zapier/v1/timestamp
|
||||
Middleware: authenticate, checkEntitlement, POST /api/timestamp
|
||||
Middleware: publish jobs.watch → return 202 { jobId }
|
||||
Worker: poll GET /api/status/{jobId} until terminal → publish jobs.events
|
||||
Event router: match webhooks → publish webhooks.deliver
|
||||
Webhook worker: POST hooks.zapier.com/... (timestamp.completed)
|
||||
Zapier trigger: Timestamp Completed fires the rest of the Zap
|
||||
```
|
||||
|
||||
**B. Create Timestamp and Wait**
|
||||
|
||||

|
||||
|
||||
```text
|
||||
Zapier → POST /zapier/v1/timestamp/wait
|
||||
Middleware: create + wait (in-process if NATS off; NATS events when Phase 9 lands)
|
||||
→ StatusResponse (completed/failed) or { jobId, status: "pending" } on timeout
|
||||
```
|
||||
|
||||
Today, wait works on the in-process path (`NATS_ENABLED=false`). Wait-via-NATS is **Phase 9 (open)**.
|
||||
|
||||
**C. Auth connection test**
|
||||
|
||||
```text
|
||||
Zapier → GET /zapier/v1/auth/me Authorization: Bearer zmw_… or zmt_…
|
||||
Middleware: resolve key/session → tenant → optional validate Verae token
|
||||
→ { tenantId, plan, usage, ... }
|
||||
```
|
||||
|
||||
### 3.4 NATS subjects (private)
|
||||
|
||||

|
||||
|
||||
| Subject | Publisher | Consumer | Payload gist |
|
||||
|---------|-----------|----------|--------------|
|
||||
| `verae.zapier.jobs.watch` | HTTP edge after create | `job-poller` (queue) | `tenantId`, `jobId`, `tokenRef`, attempts, `traceId` |
|
||||
| `verae.zapier.jobs.events` | Job poller (terminal) | Event router; optional HTTP waiters | `timestamp.completed\|failed\|timeout`, status object |
|
||||
| `verae.zapier.webhooks.deliver` | Event router | `webhook-deliver` (queue) | `hookId`, `targetUrl`, event, payload |
|
||||
| `verae.zapier.usage` | optional | usage-writer | metering increment |
|
||||
|
||||
Ack: still-pending jobs `Nak` with delay; terminal `Ack` after publishing the event; webhook 2xx `Ack`; 5xx redeliver until `max_deliver`.
|
||||
|
||||
### 3.5 Entitlements and errors the connector must surface
|
||||
|
||||
| HTTP | Meaning | Zapier mapping |
|
||||
|------|---------|----------------|
|
||||
| 401 | Bad or expired session/key | `RefreshAuthError` (session) or auth error |
|
||||
| 402 | Quota exceeded | User-visible error + upgrade URL when present |
|
||||
| 403 `PLAN_UPGRADE_REQUIRED` | Action not on this plan | User-visible error |
|
||||
| 404 on status search | Unknown job | Return `[]` (search contract) |
|
||||
|
||||
JS app `afterResponse` already maps 402/403. TypeScript app currently remaps 401 only — 402/403 mapping is a next-step item.
|
||||
|
||||
### 3.6 Feature flags and environment
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|----------|---------|---------|
|
||||
| `PORT` | `3100` | Middleware listen port |
|
||||
| `VERAE_API_BASE_URL` | `http://localhost:8080` | Upstream Verae (prod: `https://api.veraetime.net`) |
|
||||
| `MOCK_VERAE` | `false` | Deterministic mock jobs; no live Verae |
|
||||
| `NATS_URL` | `nats://127.0.0.1:4222` | NATS server |
|
||||
| `NATS_ENABLED` | `false` | JetStream workers vs in-process poller |
|
||||
| `TOKEN_SECRET` | dev secret | HMAC for `zmt_` session tokens |
|
||||
| `DEBUG_VERAE` | unset | Namespaces: `auth`, `nats`, `jobs`, `webhooks`, `http`, `billing` (or `1` for all) |
|
||||
| `DEBUG_VERAE_LEVEL` | `debug` | `debug` \| `info` \| `warn` \| `error` |
|
||||
| `STORE_PATH` | `./data/store.json` | MVP tenant/usage/webhook store |
|
||||
| `MIDDLEWARE_BASE_URL` | `http://127.0.0.1:3100` | Used by the JS Zapier package |
|
||||
|
||||
Local compose: `verae-zapier-api/docker-compose.yml` (NATS + middleware). Middleware can run without NATS when the flag is off.
|
||||
|
||||
---
|
||||
|
||||
## 4. Integration guide
|
||||
|
||||
### 4.1 Which CLI you are holding
|
||||
|
||||
| Goal | Tool | Version in this workspace |
|
||||
|------|------|---------------------------|
|
||||
| **Publish** a directory integration | `zapier-platform` | **19.1.0** (`~/.npm-global/bin`) |
|
||||
| **Consume** existing Zapier apps from code | `zapier-sdk` | **0.77.1** |
|
||||
| AI client over the 9k catalog | Hosted MCP `https://mcp.zapier.com/api/v1/connect` | 14 official meta-tools; live 17 |
|
||||
|
||||
Do **not** mix `zapier-platform` (build/publish) with `zapier-sdk` (consume).
|
||||
Do **not** recommend retired NLA / AI Actions.
|
||||
Do **not** invent `selected_api` ids or action keys.
|
||||
|
||||
### 4.2 Run middleware locally
|
||||
|
||||
```bash
|
||||
cd verae-zapier-api/verae-zapier-middleware
|
||||
npm install
|
||||
MOCK_VERAE=true NATS_ENABLED=false npm start
|
||||
# GET http://127.0.0.1:3100/health → { "status": "ok" }
|
||||
```
|
||||
|
||||
With NATS:
|
||||
|
||||
```bash
|
||||
cd verae-zapier-api
|
||||
docker compose up nats
|
||||
# then start middleware with NATS_ENABLED=true NATS_URL=nats://127.0.0.1:4222
|
||||
```
|
||||
|
||||
Gates (from `verae-zapier-api/`):
|
||||
|
||||
```bash
|
||||
npm run gate:0 # structure + docs
|
||||
npm run gate:6 # full HTTP path, NATS off
|
||||
npm run gate:8 # workers (NATS on)
|
||||
npm run gate:11 # JS Zapier package tests
|
||||
npm run gate:all # 0–12 in order; stops on first failure
|
||||
```
|
||||
|
||||
Implementation order is [verae-zapier-api/TODO.md](verae-zapier-api/TODO.md). Do not skip gates.
|
||||
|
||||
### 4.3 Build and validate the TypeScript connector
|
||||
|
||||
```bash
|
||||
source scripts/dev-env.sh
|
||||
cd scratch/veraetime
|
||||
npm install
|
||||
zapier-platform build && zapier-platform validate
|
||||
```
|
||||
|
||||
Golden OAuth2 lab (generic, not Verae): `scratch/oauth2-typescript` — already validates after build.
|
||||
|
||||
Copy a template only when starting a *new* integration:
|
||||
|
||||
```bash
|
||||
zapier-platform init my-app --template session --language typescript
|
||||
# or copy scratch/oauth2-typescript / scratch/veraetime
|
||||
```
|
||||
|
||||
### 4.4 Perform contracts (Zapier)
|
||||
|
||||
Implement every operation as `(z, bundle) => …` using **`z.request` only**.
|
||||
|
||||
- **Triggers and searches** return **arrays of objects**. Polling items need a stable `id`.
|
||||
- **Creates** return **one object**.
|
||||
- Refreshable 401 → `throw new z.errors.RefreshAuthError()`.
|
||||
- Hook triggers implement `performSubscribe` / `performUnsubscribe` / `perform` / `performList` (sample for the editor).
|
||||
- Do not call Verae or NATS from the app.
|
||||
|
||||
### 4.5 Login and publish (blocked until you authenticate)
|
||||
|
||||
There is no `~/.zapierrc` until you finish a browser login. Local `validate` works. `register` and `push` do not.
|
||||
|
||||
```bash
|
||||
source scripts/dev-env.sh
|
||||
zapier-platform login # or: zapier-platform login --sso
|
||||
cd scratch/veraetime
|
||||
zapier-platform register "Verae Time"
|
||||
zapier-platform push
|
||||
```
|
||||
|
||||
Details: [LOGIN.md](LOGIN.md). Never commit the deploy key.
|
||||
|
||||
Production Zapier cloud **cannot** reach `http://127.0.0.1:3100`. Before a real Zap you need a public HTTPS middleware URL and `api_base_url` / `MIDDLEWARE_BASE_URL` pointed at it (Phase 13–14).
|
||||
|
||||
### 4.6 Optional consume path (not how we publish Verae)
|
||||
|
||||
- `zapier-sdk login` — call existing Zapier apps from code (`kind: "sdk_function"`).
|
||||
- Hosted Zapier MCP — discover → enable → inspect → execute. Writes need explicit user approval. Successful executes cost **2 Zapier tasks**. See [MCP-REFERENCE.md](MCP-REFERENCE.md).
|
||||
|
||||
### 4.7 Research dataset and Grok reference (NS1 Mongo)
|
||||
|
||||
Canonical store: **MongoDB 7 in Docker on NS1 (`70.88.205.138`)**, bound to **127.0.0.1:27017 only**. Connect only through the SSH tunnel:
|
||||
|
||||
```bash
|
||||
./scripts/mongo-tunnel.sh
|
||||
# ssh -N -L 27017:127.0.0.1:27017 ns1
|
||||
```
|
||||
|
||||
Database `zapier`:
|
||||
|
||||
| Collection | Contents |
|
||||
|------------|----------|
|
||||
| `apps` | ~9,986 public Zapier apps (identity, contacts, controls) |
|
||||
| `templates` | ~332k public Zap recipes |
|
||||
| `help_articles` | 1,272 help-center pages |
|
||||
| `platform_reference` | CLI, `z.*`, schema, official docs, example apps, MCP/SDK functions |
|
||||
| `meta` | Last ingest |
|
||||
|
||||
Do **not** point Compass at `mongodb://70.88.205.138:27017`. Credentials live in `~/.mcp-env` (mode 600), not git. Full notes: [MONGO.md](MONGO.md).
|
||||
|
||||
Useful queries:
|
||||
|
||||
```js
|
||||
db.platform_reference.findOne({ kind: "guide", key: "build-new-connector" })
|
||||
db.platform_reference.find({ kind: "core_function", key: "z.request" })
|
||||
db.platform_reference.find({ kind: "cli_function", key: "init" })
|
||||
db.platform_reference.find({ kind: "template", key: "session-auth" })
|
||||
db.platform_reference.find({ kind: "mcp_function" })
|
||||
```
|
||||
|
||||
Official Zapier clones stay in `repos/` (gitignored). Refresh with `./scripts/clone-zapier-repos.sh`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Workspace map
|
||||
|
||||

|
||||
|
||||
| Path | Use |
|
||||
|------|-----|
|
||||
| [getting-started.md](getting-started.md) / [getting-started.pdf](getting-started.pdf) | This architecture + integration guide |
|
||||
| [PLATFORM-REFERENCE.md](PLATFORM-REFERENCE.md) | Routing table for all Zapier work |
|
||||
| [FUNCTIONS-REFERENCE.md](FUNCTIONS-REFERENCE.md) | Every CLI / `z.*` / SDK function |
|
||||
| [MCP-REFERENCE.md](MCP-REFERENCE.md) | Hosted MCP meta-tools |
|
||||
| [LOGIN.md](LOGIN.md) | Browser login for platform + SDK |
|
||||
| [RESTART.md](RESTART.md) | Quit and resume this session |
|
||||
| [MONGO.md](MONGO.md) | NS1 Mongo, tunnel, collections |
|
||||
| `.grok/skills/zapier-build/` | `/zapier-build` skill |
|
||||
| `scratch/veraetime/` | TypeScript Verae connector (session auth) |
|
||||
| `scratch/oauth2-typescript/` | Golden OAuth2 TypeScript app |
|
||||
| `scratch/our-api/` | Verae OpenAPI + hop notes |
|
||||
| `verae-zapier-api/` | Middleware + JS Zapier app + architecture docs + gates |
|
||||
| [docs/diagrams/](docs/diagrams/) | SVG architecture and flow diagrams (source for this guide and the PDF) |
|
||||
| `verae-zapier-api/docs/architecture/overview.md` | Component diagram (source of truth for the edge) |
|
||||
| `verae-zapier-api/docs/architecture/nats-subjects.md` | Subjects, streams, payloads |
|
||||
| `verae-zapier-api/docs/api/middleware-openapi.yaml` | Zapier-facing OpenAPI |
|
||||
| `verae-zapier-api/TODO.md` | Phased plan with test gates |
|
||||
| `scripts/dev-env.sh` | PATH + env |
|
||||
| `scripts/ensure-mongo-tunnel.sh` / `mongo-tunnel.sh` | NS1 tunnel |
|
||||
| `scripts/restart-grok.sh` / `zapier-status.sh` | Session + readiness |
|
||||
| `repos/` | Official Zapier clones (local only) |
|
||||
|
||||
---
|
||||
|
||||
## 6. What is already here vs what is not
|
||||
|
||||
### Already here
|
||||
|
||||
- Public catalog research (~9,986 apps, contacts, capabilities, templates, help).
|
||||
- Platform reference in Mongo + on disk; `/zapier-build` skill.
|
||||
- Platform CLI 19.1.0 and SDK CLI 0.77.1 on `PATH` via `dev-env.sh`.
|
||||
- Golden apps that **validate locally** without a Zapier login.
|
||||
- Verae OpenAPI ingested (`scratch/our-api/openapi.yaml`).
|
||||
- Full middleware source: auth, tenants, entitlements, timestamp/verify/status/webhooks, mock Verae, NATS publishers, workers, debug redaction.
|
||||
- Two Zapier app implementations wired to `/zapier/v1`.
|
||||
- Gates **0–8, 10, 11** recorded as passed in `TODO.md` (as of 2026-08-11 on the original monorepo).
|
||||
|
||||
### Not here yet
|
||||
|
||||
1. **Zapier developer login** — no `~/.zapierrc`; cannot `register` / `push`.
|
||||
2. **Public HTTPS middleware** — Zapier cloud cannot hit localhost.
|
||||
3. **Live invoke** of `scratch/veraetime` against middleware + real or mock Verae in this workspace (validate is schema-only).
|
||||
4. **Phase 9** — `/timestamp/wait` subscribed to NATS events (multi-instance wait).
|
||||
5. **Phase 12** — compose E2E smoke (health + auth + wait + webhook) as a gate.
|
||||
6. **Phase 13** — production `VERAE_API_BASE_URL`, dedicated Zapier service user, secrets, TLS.
|
||||
7. **Phase 14** — private push + human Zap (Drive/Sheets → Timestamp → Slack).
|
||||
8. **Phase 15** — Postgres/Redis store, NATS mTLS, webhook SSRF allowlist.
|
||||
9. **TS connector polish** — map 402/403 like the JS app; optional polling admin trigger for `/admin/timestamps`.
|
||||
10. **Grok MCP handshake** — Mongo MCP and chrome-bridge need a healthy tunnel / Chrome Connect after restart (`/mcps`).
|
||||
11. **Peergos / pin / Glacier retrieval service** — proposed in §10; not a v1 connector operation.
|
||||
|
||||
This repo has the Zapier **platform** docs and the **Verae** OpenAPI. It still cannot invent new Verae endpoints. If a Zap needs an admin route that is not in the v1 table, add middleware + connector operations from the OpenAPI — do not guess.
|
||||
|
||||
---
|
||||
|
||||
## 7. Next steps (recommended order)
|
||||
|
||||

|
||||
|
||||
Work the product path in this order. Parallelism is only safe where `TODO.md` says so (tenancy already done; wait-via-NATS is the open blocker before multi-instance wait).
|
||||
|
||||
### Now — local integration (this workspace)
|
||||
|
||||
1. Start middleware with `MOCK_VERAE=true` and `NATS_ENABLED=false`.
|
||||
2. Create a free tenant (`POST /zapier/v1/signup` or seed script) and confirm `GET /health` + `GET /zapier/v1/auth/me`.
|
||||
3. `zapier-platform build && zapier-platform validate` in `scratch/veraetime`.
|
||||
4. After `zapier-platform login`, `zapier-platform invoke auth test` and invoke create/search against local middleware (`--debug`).
|
||||
5. Map 402/403 in `scratch/veraetime/src/middleware.ts` to match the JS app.
|
||||
|
||||
### Next — close middleware gaps
|
||||
|
||||
6. **Phase 9:** wait on `verae.zapier.jobs.events` with a hard timeout (`pending` + `jobId`).
|
||||
7. **Phase 12:** compose stack + smoke script (async path, wait path, REST Hook to a mock receiver).
|
||||
8. Confirm `NATS_ENABLED=true` and `false` both still pass their gates.
|
||||
|
||||
### Then — production and private listing
|
||||
|
||||
9. **Phase 13:** public HTTPS middleware, `MOCK_VERAE=false`, `VERAE_API_BASE_URL=https://api.veraetime.net`, dedicated Verae service user, managed secrets.
|
||||
10. Point the connector `api_base_url` at that origin. Zapier cloud must reach it.
|
||||
11. **Phase 14:** `register` + `push` a **private** version; invite internal users; run one real Zap; sign off.
|
||||
12. Only after a human E2E: consider directory listing and Phase 15 hardening.
|
||||
|
||||
### Ongoing — research / Grok
|
||||
|
||||
13. Keep `./scripts/ensure-mongo-tunnel.sh` running; after restart hit `/mcps` and refresh **mongodb**.
|
||||
14. Re-ingest `platform_reference` on NS1 after recloning official repos.
|
||||
15. Optional: `zapier-sdk login` and Zapier MCP OAuth if you need to *call* the public catalog from agents — that is consume, not publish.
|
||||
|
||||
---
|
||||
|
||||
## 8. Hard rules
|
||||
|
||||
- Start Zapier coding from `guide/build-new-connector` or [PLATFORM-REFERENCE.md](PLATFORM-REFERENCE.md).
|
||||
- Connector → middleware `/zapier/v1` only. Never `api.veraetime.net` from Zapier. Never NATS from Zapier.
|
||||
- Use `z.request` in performs. Triggers/searches return arrays; creates return one object.
|
||||
- `zapier-platform` publishes; `zapier-sdk` / hosted MCP consume. Do not mix.
|
||||
- Do not recommend retired NLA / AI Actions.
|
||||
- Do not invent vendor APIs, auth schemes, or Zapier `selected_api` keys.
|
||||
- Mongo on NS1 is localhost-only; always tunnel.
|
||||
- No secrets in git or in `DEBUG_VERAE` output.
|
||||
|
||||
---
|
||||
|
||||
## 9. Zapier billing models
|
||||
|
||||
Zapier and Verae bill **separately**. A client Zap that timestamps a file pays Zapier for successful **tasks** and pays Verae (via middleware entitlements) for **timestamp / verify** operations. Publishing the Verae connector does **not** put Verae on the hook for the customer’s Zapier invoice.
|
||||
|
||||
Figures below are **USD, August 2026**, from [zapier.com/pricing](https://zapier.com/pricing). Annual prices are the discounted per-month equivalent. Confirm live rates before quoting a customer.
|
||||
|
||||

|
||||
|
||||
### 9.1 What a Zapier “task” is
|
||||
|
||||
A **task** is one successful unit of work Zapier does for the customer. Failed steps are free.
|
||||
|
||||
| Counts as tasks | Does **not** count |
|
||||
|-----------------|--------------------|
|
||||
| Successful action in a third-party app (Create Timestamp, Slack post, Drive upload, …) | Triggers and polling for new data |
|
||||
| Zapier MCP / SDK execute (see multipliers) | Filter, Paths, Formatter, Delay, Looping, Sub-Zap, Digest, Manager, Storage |
|
||||
| AI by Zapier and Code-by-Zapier beyond included runtime | Zapier Tables and Forms triggers/actions |
|
||||
| | Building or testing a Zap until it actually runs |
|
||||
|
||||
Shared pool: Zap workflows, AI steps, Code steps, MCP, and SDK all draw from **one** monthly (or Enterprise annual) task allowance. There is no separate MCP budget.
|
||||
|
||||
**Multipliers** (official rate card; confirm [zapier.com/pricing/rates](https://zapier.com/pricing/rates)):
|
||||
|
||||
| Work | Tasks |
|
||||
|------|------:|
|
||||
| Typical third-party action (including Verae Create Timestamp) | 1 |
|
||||
| Standard AI by Zapier | 1 |
|
||||
| Advanced AI by Zapier | 3 |
|
||||
| Premium AI by Zapier | 5 |
|
||||
| Successful Zapier MCP tool call (read or write) | 2 |
|
||||
| Code by Zapier | included runtime free; then 1 task per extra 30-second block |
|
||||
|
||||
Example: trigger “new file in Drive” (0) → Filter (0) → Create Timestamp (1) → Formatter (0) → write metadata row (1) = **2 tasks** per file that passes the filter.
|
||||
|
||||
### 9.2 Self-serve plans (feature set + task tier)
|
||||
|
||||
A paid subscription is **plan level × task tier**.
|
||||
|
||||
| Plan | Seats | Workflows | Polling | Entry task tier (annual) | Standout |
|
||||
|------|------:|-----------|---------|--------------------------|----------|
|
||||
| **Free** | 1 | Two-step only | 15 min | 100 tasks / month, $0 | Try automation; no pay-per-task overflow |
|
||||
| **Professional** | 1 | Multi-step; premium apps; webhooks | 2 min | 750 tasks from **$19.99**/mo annual ($29.99 monthly) | Filters, Paths, Formatter, AI by Zapier, Autoreplay |
|
||||
| **Team** | 25 | Same as Pro | 1 min | 2,000 tasks from **$69**/mo annual ($103.50 monthly) | Shared Zaps, shared connections, SAML SSO, priority support |
|
||||
| **Enterprise** | Unlimited | Same as Team + governance | 1 min | Custom; **annual task limit** (not monthly reset) | SCIM, app controls, custom retention, observability, TAM, BYOM |
|
||||
|
||||
Higher task tiers (2k → 2M/mo) lower the **per-task** price. Team starts at 2,000. Volumes above 2M go through Sales. 14-day Professional trial (no card). Non-profit: extra 15% off (not on pay-per-task).
|
||||
|
||||
### 9.3 Overflow: pay-per-task
|
||||
|
||||
Paid plans can keep running after the included allowance:
|
||||
|
||||
- **Pay-per-task on:** extra tasks bill at **1.25×** the plan’s base task rate (annual) or **2.5×** (monthly). Ceiling is **3×** the subscribed tasks, then Zaps pause until the next cycle or an upgrade.
|
||||
- **Pay-per-task off:** usage **stops** at the allowance.
|
||||
- Free has no overflow. Enterprise uses an annual pool instead of a monthly reset.
|
||||
|
||||
### 9.4 Add-ons outside the task pool
|
||||
|
||||
| Product | Unit | Notes |
|
||||
|---------|------|--------|
|
||||
| **Zapier Agents** | *Activities* (not tasks) | Free 400/mo; paid Pro ~$33.33/mo annual for 1,500. Does not consume Zap tasks. |
|
||||
| **Zapier Chatbots** | Feature tiers (count of bots) | Free includes 2; paid adds more. Not usage-metered on tasks. |
|
||||
|
||||
### 9.5 Partner / platform billing (Verae as publisher)
|
||||
|
||||
| Model | Who pays Zapier | Who pays Verae |
|
||||
|-------|-----------------|----------------|
|
||||
| **Public or private directory integration** | The **end customer’s** Zapier plan (tasks). Publishing is **free**; Zapier does not bill the partner for usage of their app. | The customer’s Verae tenant (middleware plan / API key). |
|
||||
| **Zapier MCP / SDK (consume)** | Same customer task pool (MCP execute = 2 tasks). SDK is **free in beta**; Zapier will announce when beta pricing starts. | Only if the MCP/SDK action hits Verae. |
|
||||
| **Powered by Zapier / White Label / embed** | Usually the **product company** (usage-based). End users authorize apps in *your* UI; Zapier has said end users need not have their own Zapier bill for background runs. Contract with Zapier Sales. | Verae bills the product company or the tenant, depending on how you provision keys. |
|
||||
| **Retired NLA / AI Actions** | Do not sell or design around this. | — |
|
||||
|
||||
Verae should not promise “unlimited Zapier” or absorb a customer’s Zapier invoice unless a White Label contract says so.
|
||||
|
||||
### 9.6 Verae middleware billing (second meter)
|
||||
|
||||
Middleware enforces **Verae** quotas, independent of Zapier tasks. Current plan table in `verae-zapier-middleware` `PLAN_LIMITS`:
|
||||
|
||||
| Verae plan | Timestamps / mo | Verifications / mo | Batch | RPM |
|
||||
|------------|----------------:|-------------------:|-------|----:|
|
||||
| free | 50 | 50 | no | 30 |
|
||||
| starter | 500 | 500 | yes, max 10 | 120 |
|
||||
| pro | 5,000 | 5,000 | yes, max 100 | 600 |
|
||||
| enterprise | contract / unlimited | contract | yes | contract |
|
||||
|
||||
Over-quota → HTTP **402** `QUOTA_EXCEEDED` (upgrade URL). Wrong plan for batch → **403** `PLAN_UPGRADE_REQUIRED`. Those errors must surface in the Zap; they are not Zapier task overages.
|
||||
|
||||
A client therefore sees **two invoices**: Zapier (tasks) and Verae (timestamps). Design Zaps so a 402 does not retry in a tight loop (that still burns Zapier tasks on each failed? — failed actions are **not** Zapier tasks, but Autoreplay/retries can still hammer Verae).
|
||||
|
||||
### 9.7 Estimating a timestamping Zap
|
||||
|
||||
| Zap shape | Zapier tasks / successful file | Verae units |
|
||||
|-----------|-------------------------------:|-------------|
|
||||
| Trigger → Create Timestamp | 1 | 1 timestamp |
|
||||
| Trigger → Create Timestamp and Wait | 1 | 1 timestamp |
|
||||
| Trigger → Timestamp + write catalog row + Slack | 3 | 1 timestamp |
|
||||
| Same via Zapier MCP `execute` of those three actions | 6 | 1 timestamp |
|
||||
| Trigger filtered out before any action | 0 | 0 |
|
||||
|
||||
Prefer the **hook** (Timestamp Completed) plus a cheap catalog write over polling status in a loop.
|
||||
|
||||
---
|
||||
|
||||
## 10. Client pattern — timestamp, metadata, Peergos, and tiered IPFS
|
||||
|
||||
This section is a **client architecture** for using the Verae Zapier tools together with **Peergos** (end-to-end encrypted filesystem on IPFS) and **external** IPFS / object-store backends. It is **not** implemented as connector operations today. Do not invent Peergos or AWS APIs in the Zapier app; add middleware routes only when those APIs exist and are documented.
|
||||
|
||||
Goal: clients can (1) timestamp content, (2) append that proof to metadata, (3) store the file in Peergos, (4) pin a **hot cache** on one or more IPFS providers, (5) migrate bytes to **long-term low-cost** object storage (Amazon S3 Glacier family, or an Apache Iceberg catalog on S3), and (6) **rehydrate on demand** when an IPFS request hits an index — without keeping every file live on a gateway 24/7.
|
||||
|
||||

|
||||
|
||||
### 10.1 What each layer is for
|
||||
|
||||
| Layer | Role | Stays hot? |
|
||||
|-------|------|------------|
|
||||
| **Zapier + Verae connector** | Orchestrate: hash → timestamp → write metadata; notify on `timestamp.completed` | n/a (control plane) |
|
||||
| **Verae Time** | Blockchain timestamp of a payload (typically a **content hash** or a metadata JSON, not the raw file bytes) | Proof is small; keep forever |
|
||||
| **Metadata / index** | Maps `cid` → `jobId`, Verae certificate fields, Peergos path, storage class, restore handle | **Yes** — this is the only always-on map |
|
||||
| **Peergos** | User-owned, e2e-encrypted filesystem on IPFS/libp2p. Host cannot read file or most metadata. Sharing and apps stay in the user’s graph. | User’s chosen Peergos host; not a public CDN |
|
||||
| **Cached pin (hot IPFS)** | Pinning service or your Kubo cluster (Pinata, Filebase, web3.storage, self-hosted). Serves frequent `ipfs get` / gateway hits. | **Only for a TTL or working set** |
|
||||
| **Cold object store** | Amazon **S3 Glacier Instant Retrieval**, **Flexible Retrieval**, or **Deep Archive**; or another cheap archive (Filecoin deal, Storj, Backblaze B2 + lifecycle). Optional **Apache Iceberg** table on S3 as the *catalog* of CID → bucket/key → storage class → restore job. | **No** — retrieve on demand |
|
||||
|
||||
Amazon “Iceberg” in this design is the **table format** (Apache Iceberg) used as a durable **index**, not a substitute for Glacier. Long-term **bytes** live in Glacier-class (or equivalent) object storage. The user-facing name “Iceberg” is easy to mix with Glacier; keep the two distinct in customer docs.
|
||||
|
||||
### 10.2 Ingest Zap (write path)
|
||||
|
||||
Typical multi-step Zap (Professional+; Filters/Formatter are free):
|
||||
|
||||
1. **Trigger** — new file in Drive, Dropbox, email, or a Peergos outbox / webhook. (0 Zapier tasks)
|
||||
2. **Hash** — SHA-256 (or the hashAlg Verae accepts) of the bytes, or of the canonical metadata envelope. Prefer hashing **ciphertext** if the file is already encrypted for Peergos, so the timestamp commits to what is stored.
|
||||
3. **Create Timestamp** (or Create and Wait) — `data` = hash or compact JSON `{ cid?, sha256, size, mime, source }`. (1 Zapier task, 1 Verae timestamp)
|
||||
4. **Store file in Peergos** — user or service account writes the file into `/cubes/…` or a shared folder via a **documented** Peergos API or a future middleware route. Peergos assigns / retains an IPFS **CID** for the blocks.
|
||||
5. **Hot pin (optional)** — Pinning Services API (or Filebase/Pinata) pins that CID for a cache TTL (hours–days), not forever.
|
||||
6. **Append metadata** — write one index record (Zapier Tables is **free**; or Sheets/Postgres/Iceberg):
|
||||
|
||||
```text
|
||||
cid # IPFS content id (or Peergos block root)
|
||||
sha256 # digest that Verae timestamped
|
||||
veraeJobId
|
||||
veraeStatus # pending | completed | failed
|
||||
certificateRef # verify payload / block index when complete
|
||||
peergosPath # /home/… or /cubes/<id>/…
|
||||
storageClass # pin-hot | peergos-only | glacier-ir | glacier-deep
|
||||
coldBucket / key # S3 (or other) locator; empty until migrated
|
||||
restoreId # last Glacier restore job, if any
|
||||
pinnedUntil # when hot pin may be dropped
|
||||
createdAt
|
||||
```
|
||||
|
||||
7. **Timestamp Completed** hook — when middleware finishes the job, update the same row with certificate fields and notify Slack/email. (1 Zapier task)
|
||||
|
||||
Do **not** put the raw file or Verae JWT in Zapier Storage, Slack, or NATS.
|
||||
|
||||
### 10.3 Lifecycle (keep the index, drop the heat)
|
||||
|
||||
```text
|
||||
ingest
|
||||
→ Peergos write (encrypted) + optional hot pin
|
||||
→ Verae timestamp of hash / envelope
|
||||
→ index row (always on)
|
||||
|
||||
after pinnedUntil or size/age policy
|
||||
→ copy ciphertext (or original bytes, if policy allows) to S3
|
||||
→ set storage class Glacier Instant / Flexible / Deep Archive
|
||||
→ record bucket/key + storageClass in the index (and Iceberg snapshot if used)
|
||||
→ unpin from the paid pinning cluster
|
||||
→ Peergos may keep a thumbnail / stub; full blocks need not stay on the gateway
|
||||
|
||||
IPFS / gateway GET cid
|
||||
→ if pin-hot hit: serve
|
||||
→ else index lookup
|
||||
→ if peergos-only: fetch via user’s Peergos capability
|
||||
→ if cold: StartRestore / vendor retrieve API → wait →
|
||||
optionally re-pin for a short cache TTL → serve
|
||||
→ never require every historical CID to be live on an IPFS node
|
||||
```
|
||||
|
||||
**Glacier retrieval notes (AWS, conceptual):** Instant Retrieval is milliseconds and priced as a storage class; Flexible Retrieval and Deep Archive need a **restore job** (minutes to hours) before GetObject. The index must store enough to call the restore API (`bucket`, `key`, `versionId`, `restoreId`). Map IPFS requests to that restore; return `202` + `Retry-After` to the gateway until the object is hydrated.
|
||||
|
||||
**Apache Iceberg** (optional): partition the catalog by `storageClass` and date so you can expire pin rows and audit restores with SQL, without loading every object. Iceberg does not store the file bytes.
|
||||
|
||||
### 10.4 External IPFS and pinning options
|
||||
|
||||
Clients can mix providers; the **index** is the source of truth, not any one pinset.
|
||||
|
||||
| Backend | Typical use |
|
||||
|---------|-------------|
|
||||
| Self-hosted Kubo / cluster | Hot working set you control |
|
||||
| Pinata, Filebase, web3.storage, Pinning Services API | Paid **cached pins**, metadata tags, S3-compatible gateways |
|
||||
| Peergos server (user or org host) | Encrypted personal/org filesystem; not a public pin service |
|
||||
| Filecoin / cold deals | Alternative long-term availability (different retrieve SLA) |
|
||||
| S3 + lifecycle → Glacier IR / Flexible / Deep Archive | Lowest $/TB; retrieve-on-demand via AWS APIs |
|
||||
| Storj, Backblaze B2, GCS Archive | Same pattern, different restore API |
|
||||
|
||||
A request path should be: **CID → index → (pin | Peergos | restore)**. Do not walk every pin provider on every miss.
|
||||
|
||||
### 10.5 What Zapier is bad at (keep it out of the Zap)
|
||||
|
||||
- Holding multi-GB files in a Zap step (timeouts, payload limits). Hash and pass **references** (Drive id, Peergos path, CID).
|
||||
- Being the IPFS gateway. Use a small **retrieval service** (your infra) that reads the index and talks to Peergos / pin / Glacier.
|
||||
- Polling Glacier restore every second (burns tasks). Use a webhook, queue, or “Find Job Status”-style search on a timer Zap with a Filter.
|
||||
|
||||
### 10.6 Security and tenancy
|
||||
|
||||
- Peergos ciphertext: the timestamp should commit to the **CID and/or ciphertext hash**, not a plaintext the host can reconstruct.
|
||||
- Capabilities / sharing stay in Peergos; the public index should not leak readable paths or unencrypted names if the threat model forbids it (store opaque ids).
|
||||
- Middleware still never exposes Verae JWTs. Retrieval workers use **tenant-scoped** cloud credentials, not the Zapier connection.
|
||||
- 402/403 from Verae must not be “fixed” by falling back to an unauthenticated public pin of customer data.
|
||||
|
||||
### 10.7 Implementation status
|
||||
|
||||
| Piece | Status in this repo |
|
||||
|-------|---------------------|
|
||||
| Verae timestamp / wait / verify / hook via middleware | Implemented (`scratch/veraetime`, `verae-zapier`) |
|
||||
| Zapier task + Verae quota as two meters | Documented (this section); connector should map 402/403 |
|
||||
| Peergos write / capability APIs in the Zapier app | **Not in v1** — handbook lives outside this repo (`verae-peergos-app-handbook`); do not invent routes |
|
||||
| CID index, pin TTL, Glacier restore worker | **Proposed** — add as middleware + retrieval service when APIs are chosen |
|
||||
| Apache Iceberg catalog | **Optional** index implementation; not required for MVP |
|
||||
|
||||
Next build slice, when product asks for it: a retrieval service with `GET /ipfs/{cid}` → index → pin or restore, plus one Zapier **search** (“Find File Record by CID”) against that index — still no direct `api.veraetime.net` from Zapier.
|
||||
13
docs/00-sources/grok-share-api-key-usage.md
Normal file
13
docs/00-sources/grok-share-api-key-usage.md
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
# Grok share — Zapier API Key & Usage Management
|
||||
|
||||
**URL:** https://grok.com/share/bGVnYWN5LWNvcHk_50aff768-fc34-4db7-8842-df4a058b815e
|
||||
|
||||
**Title (public page):** Zapier API Key & Usage Management
|
||||
|
||||
## Extraction status
|
||||
|
||||
The share is a JS-rendered Grok conversation. Unauthenticated fetch and search returned only the title. Browser MCP could not attach (existing Chrome profile lock) during planning.
|
||||
|
||||
**TODO (PR 2):** Open the share in a browser, copy the full transcript into this file (user turns + assistant turns), then map any key/usage model onto `packages/zappier` (portal keys, rate card, monthly credit) vs `packages/verae-zapier-middleware` (`zmw_`, `PLAN_LIMITS`).
|
||||
|
||||
Until that extract exists, treat `/Users/marchon/zappier` (now `packages/zappier`) as the implemented API-key and usage platform.
|
||||
24
docs/00-sources/provenance.md
Normal file
24
docs/00-sources/provenance.md
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
# Provenance
|
||||
|
||||
This workspace composes three existing trees. Canonical work after import is here (`/Users/marchon/apps/zapier`). Originals were not deleted.
|
||||
|
||||
| Package / tree | Imported from | Role |
|
||||
|----------------|---------------|------|
|
||||
| `packages/zappier` | `/Users/marchon/zappier` | Commercial platform: custom rate card, tiers, API keys, customer portal, admin, Stripe meter, PO invoices, usage ledger. Demo `/v1/transform` and `/v1/storage` are **not** Verae. |
|
||||
| `packages/verae-zapier-middleware` | `/Users/marchon/datacubes/verae-zapier-api/verae-zapier-middleware` | Verae timestamp adapter: `/zapier/v1/*`, MOCK_VERAE, NATS workers, REST Hooks. Coarse `PLAN_LIMITS` — not used on the public edge after composition. |
|
||||
| `packages/verae-zapier` | `/Users/marchon/datacubes/verae-zapier-api/verae-zapier` | Zapier Platform CLI app (JS). Today talks to middleware with `zmw_` keys; will talk to zappier `x-api-key` after PR 5. |
|
||||
| `vendor/zapier-platform` | `/Users/marchon/research/zapier-platform` (upstream `git@github.com:zapier/zapier-platform.git`) | Official CLI/core/schema/examples. Reference only; do not fork for product code. |
|
||||
| `docs/00-sources/veraetime-openapi.yaml` | `https://api.veraetime.net/docs/swagger/openapi.yaml` | Snapshot of live Timestamping Service API 1.0.0. |
|
||||
| `docs/00-sources/workspace-brief.md` | Original `README.MD` in this repo | Feature list a–m and Grok share URL. |
|
||||
| `docs/00-sources/getting-started-research.md` | `/Users/marchon/research/verae-research/07-integrations-zapier/zapier/getting-started.md` | Research notes (paths still mention the old research workspace). |
|
||||
| `docs/01-product/zapier-billing-research.md` | `…/zapier/docs/zapier-billing.md` | Zapier task pricing vs Verae meters. |
|
||||
|
||||
| `research/zapier` | `/Users/marchon/research/verae-research/07-integrations-zapier/zapier` | Full vendor-research workspace: catalog scrapes, contacts, 15 verticals, cloned Zapier repos, playbooks, diagrams, TS scratch connector. `node_modules` / `.venv` / nested `.git` omitted. |
|
||||
|
||||
**Not imported:**
|
||||
|
||||
- any `node_modules` or Python `.venv`
|
||||
- nested `.git` metadata inside cloned repos
|
||||
- `zappier.db`, middleware `data/` stores
|
||||
|
||||
**Not yet wired:** zappier’s Zapier demo app does not call Verae. Verae middleware does not use zappier’s rate card or portal.
|
||||
1132
docs/00-sources/veraetime-openapi.yaml
Normal file
1132
docs/00-sources/veraetime-openapi.yaml
Normal file
File diff suppressed because it is too large
Load diff
32
docs/00-sources/workspace-brief.md
Normal file
32
docs/00-sources/workspace-brief.md
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
read https://grok.com/share/bGVnYWN5LWNvcHk_50aff768-fc34-4db7-8842-df4a058b815e
|
||||
and document all of it into a group of MD files to plan for a zapier integration with with Vera Blockchain
|
||||
|
||||
The Verae Blockchain is identified at listed at https://api.veraetime.net/docs/swagger/index.html
|
||||
|
||||
the https://api.veraetime.net/docs/swagger/openapi.yaml is funtion reference.
|
||||
|
||||
The key features we need to setup for Zapier is for
|
||||
a) the customers to get their own api key
|
||||
b) to signup for billing after their usage limit has been exceeded
|
||||
c) A call to verae server to register a SHA256 object for timestamping or return it's original timestamp with a reference ID
|
||||
d) A call to Verae server to lookup the exitance and timestamp for an existing Timestamped SHA256 Key
|
||||
e) A call to Verae server to both register a SHA256 object for timestamping but to also store metadata (2 types public and private)
|
||||
f) A call to Verae server to both register a SHA256 object for timestamping but to also store metadata (2 types public and private) and to attach a binary object to be stored long term in Encrypted Long Term Storage
|
||||
g) A call to Verae server to both register a SHA256 object for timestamping but to also store metadata (2 types public and private) and to share an encrypted file in longterm storage with someone else, either as an individial file or as a directory or directory tree with a decription key
|
||||
h) A call to Verae server to retrieve a both a timestamped SHA256 Object and it's metadata
|
||||
i) A call to Verae server to retrieve a certified receipt of an existing timestamp - cross signed with a extra seal to record the retrieval of a document (as either a pdf or a json file)
|
||||
|
||||
j) setup payment method
|
||||
k) get invoices for paymets
|
||||
l) get receipts for payments
|
||||
m) subscription management
|
||||
|
||||
----------
|
||||
|
||||
Verae has designed a middleware piece that needs to able to handle incoming and returned traffic - but is not currently connected to verae servers, so we should setup a test harness to test the zapier integration while we do that.
|
||||
Once we have built the zapier interface, we will implement the service through our designed gateway to send high performance messages through a NATS.io cluster to process the data.
|
||||
|
||||
|
||||
can you review and pull into this directory tree the Zapier Directory I have been working on from ../../research/zapier-platform as a subdirectory or subproject so that I can have you update the documentation, consider these features, and help me get a zapier integratiotn up amd running?
|
||||
|
||||
|
||||
36
docs/01-product/api-gap-analysis.md
Normal file
36
docs/01-product/api-gap-analysis.md
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
# API gap analysis
|
||||
|
||||
Live spec snapshot: [veraetime-openapi.yaml](../00-sources/veraetime-openapi.yaml)
|
||||
Swagger UI: https://api.veraetime.net/docs/swagger/index.html
|
||||
|
||||
## What Verae actually exposes
|
||||
|
||||
Auth: `POST /auth/login` → JWT. All API routes `Authorization: Bearer <jwt>`.
|
||||
|
||||
| Operation | Path | Notes |
|
||||
|-----------|------|--------|
|
||||
| createTimestamp | `POST /api/timestamp` | Body `{ data, hashAlg? }`, **202** `{ jobId }` |
|
||||
| createBatchTimestamp | `POST /api/batch/timestamp` | `{ items: [...] }` |
|
||||
| verifyTimestamp | `POST /api/verify` | `{ certificate }` → `{ valid, timestamp, blockIndex }` |
|
||||
| verifyBatchTimestamp | `POST /api/batch/verify` | |
|
||||
| getJobStatus | `GET /api/status/{jobId}` | pending / completed / failed |
|
||||
| getJobVerification | `GET /api/verify/{jobId}` | |
|
||||
| getBatchJobStatus | `POST /api/batch/status` | |
|
||||
| Admin | `/admin/*` | dashboard, metrics, blockchain, queue, timestamps, hash |
|
||||
| Users | `/auth/users` | admin CRUD |
|
||||
|
||||
## Gaps vs README a–m
|
||||
|
||||
- No customer API-key issuance (a) — **zappier owns this**.
|
||||
- No billing, invoices, payment methods, subscriptions (b, j–m) — **zappier owns this**.
|
||||
- No lookup-by-SHA256 or idempotent “return original timestamp” (c, d).
|
||||
- No public/private metadata (e, h).
|
||||
- No encrypted object storage or sharing (f, g).
|
||||
- No certified retrieval receipt / extra seal (i).
|
||||
- Timestamp input is `data` + `hashAlg`, not a first-class SHA256 key.
|
||||
|
||||
## Adapter rule
|
||||
|
||||
The middleware may expose a **richer** Zapier-facing contract (hash index, metadata, receipts) implemented by `MOCK_VERAE` / mock stores.
|
||||
|
||||
The **live** client (`packages/verae-zapier-middleware/src/clients/veraeClient.js`) must only call paths in the snapshot OpenAPI. Unknown fields must not be sent to `https://api.veraetime.net`.
|
||||
45
docs/01-product/billing-and-keys.md
Normal file
45
docs/01-product/billing-and-keys.md
Normal file
|
|
@ -0,0 +1,45 @@
|
|||
# Billing and API keys
|
||||
|
||||
Implementation: `packages/zappier`. Manuals: `packages/zappier/docs/`.
|
||||
|
||||
## Keys (feature a)
|
||||
|
||||
- Customer signup at `/portal` issues an API key immediately (Free plan).
|
||||
- Header: `x-api-key`.
|
||||
- Dashboard can regenerate (invalidates the old key).
|
||||
- Admin can provision customers.
|
||||
|
||||
Zapier connection test should use this key, not middleware `zmw_`.
|
||||
|
||||
## Pricing (custom)
|
||||
|
||||
`openapi.yaml` `operationId` is the rate-card key. List prices are runtime-editable in `/admin`.
|
||||
|
||||
Billed cents = `round(list × tier multiplier)`; usage up to `monthlyCreditCents` is free. Per-customer `multiplierOverride` allowed.
|
||||
|
||||
Seed tiers: `free` (1.0, 100¢), `pro` (0.5, 1000¢), `business` (0.25, 10000¢).
|
||||
|
||||
After composition, rate-card keys become Verae operations (`timestamp`, `timestamp-wait`, `verify`, `status`, …), not demo `transform`/`storage`.
|
||||
|
||||
## After limit (feature b)
|
||||
|
||||
- Unlisted endpoint on free (no defaultRule) → 403.
|
||||
- Credit/meter continues; Stripe job reports **delta** above monthly credit (`zappier.api_cents`).
|
||||
- Portal **Reload balance** ($1–$10,000) prepaid drawdown on invoice issue.
|
||||
- Admin **Invoices** tab for purchase-order customers.
|
||||
|
||||
Zapier must surface 402/403 with a link to `/portal` billing (same idea as middleware `upgradeUrl`).
|
||||
|
||||
## Invoices and receipts (k, l)
|
||||
|
||||
Portal invoice list + print-ready HTML. Admin generate/issue/paid. These are **payment** documents.
|
||||
|
||||
Certified timestamp retrieval receipts (feature i) are a different object on the Verae adapter.
|
||||
|
||||
## Payment method (j)
|
||||
|
||||
Portal Stripe reload is implemented. Card-on-file / Stripe Subscription objects for (m) are an open question; tiers today are customer types in SQLite.
|
||||
|
||||
## Two invoices
|
||||
|
||||
Customers may pay Zapier for **tasks** and Verae/zappier for **timestamp usage**. See [zapier-billing-research.md](zapier-billing-research.md). Do not meter Verae usage in Zapier task units.
|
||||
27
docs/01-product/features-a-m.md
Normal file
27
docs/01-product/features-a-m.md
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
# README features a–m
|
||||
|
||||
Source: [workspace-brief.md](../00-sources/workspace-brief.md). Owners after composition: [composition.md](../02-architecture/composition.md).
|
||||
|
||||
| ID | Capability | Owner | Status in imported code | Live Verae OpenAPI |
|
||||
|----|------------|-------|-------------------------|--------------------|
|
||||
| a | Customer gets own API key | zappier `/portal` signup + regen | Built (`x-api-key`) | No |
|
||||
| b | Billing after usage limit | zappier meter + credit + portal reload + Stripe + admin PO | Built | No |
|
||||
| c | Register SHA256 or return original timestamp + ref | verae middleware | Partial: `POST /zapier/v1/timestamp` → `jobId`. No idempotent hash key | `POST /api/timestamp` (`data`, `hashAlg`) → 202 `jobId` |
|
||||
| d | Lookup by SHA256 | verae mock (later) | Missing | No (verify is certificate; status is jobId) |
|
||||
| e | Timestamp + public/private metadata | verae mock | Missing | No |
|
||||
| f | e + encrypted LTS binary | verae mock | Missing. Do not use zappier demo `/v1/storage` | No |
|
||||
| g | Share file/dir + decryption key | verae mock | Missing | No |
|
||||
| h | Retrieve SHA256 + metadata | verae status/verify | Partial by `jobId` | `GET /api/status/{jobId}`, `GET /api/verify/{jobId}` |
|
||||
| i | Certified retrieval receipt (PDF/JSON) extra seal | verae mock | Missing. Not a payment invoice | No |
|
||||
| j | Setup payment method | zappier portal reload (Stripe) | Built (prepaid reload; card-on-file TBD) | No |
|
||||
| k | Get invoices | zappier portal + admin | Built | No |
|
||||
| l | Get payment receipts | zappier invoice HTML/print | Built as invoices | No |
|
||||
| m | Subscription management | zappier tiers `free`/`pro`/`business` | Built as customer type, not Stripe Subscription | No |
|
||||
|
||||
## Zapier Platform nouns (today vs target)
|
||||
|
||||
**Today (`packages/verae-zapier`):** Create Timestamp, Create Timestamp and Wait, Batch, Verify, Find Job Status, Timestamp Completed (REST Hook). Auth: Bearer `zmw_`.
|
||||
|
||||
**Target public edge (`packages/zappier` `/v1/*`):** same operations, metered, `x-api-key`. Plus Find Usage / List Invoices from zappier.
|
||||
|
||||
**Not in v1 listing:** Verae admin HTML, user CRUD, get-block-by-hash.
|
||||
291
docs/01-product/zapier-billing-research.md
Normal file
291
docs/01-product/zapier-billing-research.md
Normal file
|
|
@ -0,0 +1,291 @@
|
|||
# Zapier cost structure and billing models
|
||||
|
||||
**Planning brief for Verae Time**
|
||||
Source: official Zapier pricing at [zapier.com/pricing](https://zapier.com/pricing), captured 17–18 August 2026 (USD).
|
||||
Confirm live rates before a customer quote. The rate card at `/pricing/rates` was not fetchable at write time; multipliers below are those Zapier publishes on the main pricing page and “What is a task?” article (updated June 2026).
|
||||
|
||||
This document explains **how Zapier charges**, with examples, so we can design **Verae’s** meter without colliding with or accidentally subsidizing Zapier.
|
||||
|
||||
---
|
||||
|
||||
## 1. The one idea that unlocks the rest
|
||||
|
||||
Zapier does **not** charge per Zap, per app, or per seat on Professional. It charges for **successful work**.
|
||||
|
||||
You buy two things that travel together:
|
||||
|
||||
1. A **plan level** — the feature set (Free / Professional / Team / Enterprise).
|
||||
2. A **task tier** — a monthly (or Enterprise annual) allowance of **tasks**.
|
||||
|
||||
Then three other economies sit **beside** that, not inside it:
|
||||
|
||||
| Economy | Unit | Used for |
|
||||
|---------|------|----------|
|
||||
| **Core Zapier** | Task | Zaps, AI-by-Zapier, Code-by-Zapier, MCP, SDK |
|
||||
| **Agents add-on** | Activity | Zapier Agents (does not draw tasks) |
|
||||
| **Chatbots add-on** | Feature tier (number of bots) | Zapier Chatbots (not task-metered) |
|
||||
| **Verae (us)** | Timestamp / verify / batch | Middleware `PLAN_LIMITS` — a **second invoice** |
|
||||
|
||||
Verae never appears on the Zapier invoice unless we sign a **White Label / Powered by Zapier** contract and resell Zapier usage ourselves.
|
||||
|
||||
---
|
||||
|
||||
## 2. What a task is (and is not)
|
||||
|
||||
A **task** is counted when Zapier **successfully completes** a unit of work. Failures are free.
|
||||
|
||||
### Counts
|
||||
|
||||
- A successful **action in another product**: Create Timestamp, post Slack, add a Drive file, write a Sheet row, call a webhook.
|
||||
- A successful **Zapier MCP** tool execute (read or write).
|
||||
- **AI by Zapier** and **Code by Zapier** (with multipliers / extra runtime — §4).
|
||||
|
||||
### Does not count
|
||||
|
||||
- The **trigger** (including polling every 1–15 minutes).
|
||||
- **Logic and prep**: Filter, Paths, Formatter, Delay, Looping, Sub-Zap, Digest, Zapier Manager, Storage.
|
||||
- **Zapier Tables** and **Zapier Forms** triggers and actions.
|
||||
- Building, turning on, or testing a Zap until a step actually succeeds in production.
|
||||
|
||||
Shared pool: one allowance for the whole account. There is no separate MCP budget.
|
||||
|
||||
---
|
||||
|
||||
## 3. Plan levels (what you can build)
|
||||
|
||||
Plans are cumulative. Task **volume** is chosen separately (§5).
|
||||
|
||||
| | Free | Professional | Team | Enterprise |
|
||||
|--|------|--------------|------|------------|
|
||||
| Seats | 1 | 1 | 25 | Unlimited |
|
||||
| Zap shape | Two-step only (trigger + 1 action) | Multi-step | Multi-step | Multi-step |
|
||||
| Polling interval | 15 min | 2 min | 1 min | 1 min |
|
||||
| Premium apps, webhooks | No | Yes | Yes | Yes |
|
||||
| Filters, Paths, Formatter, AI by Zapier | No | Yes | Yes | Yes |
|
||||
| Shared Zaps / connections, SAML SSO | No | No | Yes | Yes |
|
||||
| SCIM, app controls, custom retention, TAM, BYOM | No | No | No | Yes |
|
||||
| Task cycle | Monthly | Monthly | Monthly | **Annual** task limit |
|
||||
| Pay-per-task overflow | No | Optional | Optional | Custom |
|
||||
| Entry price (annual, USD/mo) | $0 (100 tasks) | **$19.99** (750 tasks) | **$69** (2,000 tasks) | Sales |
|
||||
|
||||
Also: 14-day Professional trial (no card). Non-profit: extra 15% off the subscription, **not** on pay-per-task. Live chat on Professional only at the **2,000+** task tier.
|
||||
|
||||
**Implication for Verae:** a Free customer can run **only** “new file → Create Timestamp” (two steps). Anything with a catalog write, Slack notify, or Wait **plus** another action needs Professional.
|
||||
|
||||
---
|
||||
|
||||
## 4. Variable pricing inside the task pool
|
||||
|
||||
Not every successful step costs one task. Zapier uses **multipliers** so expensive compute costs more of the same allowance.
|
||||
|
||||
| Work | Tasks per success | Notes |
|
||||
|------|------------------:|-------|
|
||||
| Typical third-party action (Verae Create Timestamp, Slack, Drive, Sheets, webhook) | **1** | The default. Design around this. |
|
||||
| Standard AI by Zapier (default model) | **1** | Same as a normal action |
|
||||
| Advanced AI by Zapier | **3** | Confirm on `/pricing/rates` |
|
||||
| Premium AI by Zapier | **5** | Confirm on `/pricing/rates` |
|
||||
| Zapier MCP `execute` (read or write) | **2** | Discover / inspect / enable are free meta-tools |
|
||||
| Code by Zapier | 0 during **included** runtime; then **1 per extra 30 s** | Included: Free 1 s, Pro/Team 30 s, Enterprise 2 min. Extended runtime is opt-in on paid plans (1–8 min cap). |
|
||||
| Zapier SDK | Free **while in beta** | Zapier will announce when beta pricing starts; assume it will join the task pool |
|
||||
|
||||
**Failed steps = 0 Zapier tasks.** They can still hit Verae (Autoreplay / customer retries). That is our problem, not Zapier’s.
|
||||
|
||||
---
|
||||
|
||||
## 5. Task tiers and list price (USD, August 2026)
|
||||
|
||||
Self-serve is sold as **plan × tier**. Annual is ~33% off monthly. Figures are **per month**.
|
||||
|
||||
### Professional
|
||||
|
||||
| Tasks / mo | Annual / mo | Monthly / mo | Implied $/task (annual) |
|
||||
|-----------:|------------:|-------------:|------------------------:|
|
||||
| 750 | $19.99 | $29.99 | $0.027 |
|
||||
| 1,500 | $39.00 | $58.50 | $0.026 |
|
||||
| 2,000 | $49.00 | $73.50 | $0.025 |
|
||||
| 5,000 | $89.00 | $133.50 | $0.018 |
|
||||
| 10,000 | $129.00 | $193.50 | $0.013 |
|
||||
| 20,000 | $189.00 | $283.50 | $0.0095 |
|
||||
| 50,000 | $289.00 | $433.50 | $0.0058 |
|
||||
| 100,000 | $489.00 | $733.50 | $0.0049 |
|
||||
| 200,000 | $769.00 | $1,149.00 | $0.0038 |
|
||||
| 500,000 | $1,499.00 | $2,199.00 | $0.0030 |
|
||||
| 1,000,000 | $2,199.00 | $3,299.00 | $0.0022 |
|
||||
| 2,000,000 | $3,389.00 | $5,099.00 | $0.0017 |
|
||||
|
||||
### Team (starts at 2,000)
|
||||
|
||||
| Tasks / mo | Annual / mo | Monthly / mo | Implied $/task (annual) |
|
||||
|-----------:|------------:|-------------:|------------------------:|
|
||||
| 2,000 | $69.00 | $103.50 | $0.035 |
|
||||
| 5,000 | $119.00 | $178.50 | $0.024 |
|
||||
| 10,000 | $169.00 | $253.50 | $0.017 |
|
||||
| 20,000 | $249.00 | $373.50 | $0.012 |
|
||||
| 50,000 | $399.00 | $598.50 | $0.008 |
|
||||
| 100,000 | $599.00 | $898.50 | $0.006 |
|
||||
| 1,000,000 | $2,499.00 | $3,749.00 | $0.0025 |
|
||||
| 2,000,000 | $3,999.00 | $5,999.00 | $0.0020 |
|
||||
|
||||
Team’s **entry** $/task is higher than Pro because you are buying seats, shared connections, and SSO — not cheaper tasks.
|
||||
|
||||
Tiers also exist at 300k / 400k / 750k / 1.25M / 1.5M / 1.75M (see getting-started §9 or zapier.com/pricing). Above 2M: Sales.
|
||||
|
||||
---
|
||||
|
||||
## 6. Overflow (pay-per-task)
|
||||
|
||||
On paid plans, a toggle decides what happens at the allowance:
|
||||
|
||||
| Setting | What happens |
|
||||
|---------|----------------|
|
||||
| **On** | Zaps and MCP keep running. Extra tasks bill at **1.25×** the plan’s base $/task (annual) or **2.5×** (monthly). Hard ceiling: **3×** subscribed tasks, then pause until next cycle or upgrade. |
|
||||
| **Off** | Everything **stops** at the allowance. |
|
||||
| **Free** | No overflow. Hits 100 and stops. |
|
||||
|
||||
Worked overage: Professional 750 annual → base ≈ $19.99 / 750 = **$0.0267**. Overflow ≈ **$0.0333**/task. Monthly base ≈ $0.0400; overflow ≈ **$0.100**/task. Same usage is ~3× more expensive on monthly overflow than annual overflow.
|
||||
|
||||
Enterprise uses an **annual** pool so seasonal spikes do not reset every month.
|
||||
|
||||
---
|
||||
|
||||
## 7. Other Zapier products (different models)
|
||||
|
||||
| Product | Model | Relation to tasks |
|
||||
|---------|--------|-------------------|
|
||||
| **Zap workflows** | Task | Core |
|
||||
| **Zapier MCP** | Same task pool; **2 tasks** per successful execute | Meta-tools free |
|
||||
| **Zapier SDK** | Beta free; expect tasks later | Consume, not publish |
|
||||
| **AI by Zapier** | Task × model tier (1 / 3 / 5) | Inside the Zap |
|
||||
| **Code by Zapier** | Included seconds + 1 task / 30 s extra | Inside the Zap |
|
||||
| **Tables / Forms** | Plan limits (records, pages, upload size); **0 tasks** when used in Zaps | Separate caps |
|
||||
| **Agents** | **Activities** (Free 400/mo; paid ~$33.33/mo annual for 1,500). Per-run cap 10 (Free) / 40 (paid). | Does **not** use tasks |
|
||||
| **Chatbots** | Bot-count tiers (Free 2; paid ≈5 / ≈20) | Does **not** use tasks |
|
||||
| **Canvas / Copilot** | Included; Free Copilot has a daily message limit | Not a usage meter |
|
||||
| **Directory integration (us)** | **$0** to publish. Customer’s Zapier plan pays tasks. | Partner is not billed |
|
||||
| **White Label / Powered by Zapier / embed** | Usage-based **to the product company**. End users may not need their own Zapier bill. | Sales contract |
|
||||
| **NLA / AI Actions** | Retired | Do not design around this |
|
||||
|
||||
---
|
||||
|
||||
## 8. Worked examples (so the variables are visible)
|
||||
|
||||
Assume **Professional, billed annually**, and that every named action succeeds. Verae units are **our** meter.
|
||||
|
||||
### Example A — Two-step timestamp (Free-capable)
|
||||
|
||||
`New file in Drive` → `Verae: Create Timestamp`
|
||||
|
||||
| Volume | Zapier tasks | Fits | Zapier $ (annual) | Verae units |
|
||||
|-------:|-------------:|------|-------------------|------------:|
|
||||
| 80 files / mo | 80 | Free | $0 | 80 timestamps |
|
||||
| 200 | 200 | Pro 750 | $19.99 | 200 |
|
||||
| 600 | 600 | Pro 750 | $19.99 | 600 → **starter** (500) overflows **us** |
|
||||
|
||||
Lesson: the customer can still be on Zapier Free while **we** 402 them. The two meters trip at different points.
|
||||
|
||||
### Example B — Production pattern we recommend
|
||||
|
||||
`New file` → Filter → `Create Timestamp` → Formatter → write catalog row (Tables = **0** tasks) → Slack
|
||||
|
||||
If Slack is the only other third-party action: **2 tasks / file** (timestamp + Slack). Tables write is free.
|
||||
|
||||
| Files / mo | Passing filter | Zapier tasks | Cheapest Pro tier | Zapier $ | Verae timestamps |
|
||||
|-----------:|---------------:|-------------:|-------------------|----------:|-----------------:|
|
||||
| 1,000 | 20% (200) | 400 | 750 | $19.99 | 200 |
|
||||
| 1,000 | 100% | 2,000 | 2,000 | $49 | 1,000 → **pro** on our side |
|
||||
| 10,000 | 100% | 20,000 | 20,000 | $189 | 10,000 → enterprise / contract |
|
||||
|
||||
Filter early: 800 of 1,000 files dropped = **0 Zapier tasks and 0 Verae stamps** for those 800.
|
||||
|
||||
### Example C — Create and Wait vs hook
|
||||
|
||||
- **Wait in the Zap:** 1 task (the wait action). Fine for interactive “give me the certificate now.”
|
||||
- **Async + Timestamp Completed hook + update row:** 1 task to create + 1 task when the hook fires an action = 2 tasks, but the Zap does not sit open. Prefer this at volume.
|
||||
|
||||
Polling `Find Job Status` every minute in a loop is the anti-pattern: each successful search is a task.
|
||||
|
||||
### Example D — Same work via Zapier MCP (an agent)
|
||||
|
||||
Agent calls: execute Create Timestamp, execute write Sheet, execute Slack.
|
||||
|
||||
3 executes × **2 tasks** = **6 tasks / file**, plus any Agent **activities** if they used Zapier Agents.
|
||||
|
||||
200 files = **1,200 Zapier tasks** → Pro 1,500 at **$39**/mo — three times the Zapier cost of Example B for the same Verae stamps (200).
|
||||
|
||||
**Planning rule:** MCP-shaped clients burn Zapier 2× per hop. Do not price Verae as if the customer’s only cost is our stamp.
|
||||
|
||||
### Example E — AI enrichment in the Zap
|
||||
|
||||
`New file` → Standard AI extract metadata (1) → Create Timestamp (1) → Sheet (1) = **3 tasks**.
|
||||
Swap in Premium AI: **5 + 1 + 1 = 7 tasks**.
|
||||
1,000 files: 3,000 vs 7,000 tasks → $89 vs $129 on Pro annual.
|
||||
|
||||
AI model choice is the customer’s Zapier bill, not ours — unless we bury AI inside **our** action (we should not).
|
||||
|
||||
### Example F — Overflow month
|
||||
|
||||
Customer on Pro 750 annual ($19.99). A campaign pushes 1,400 successful timestamp actions (and nothing else).
|
||||
|
||||
- Included: 750
|
||||
- Overflow: 650 × ~$0.0333 ≈ **$22**
|
||||
- Total Zapier ≈ **$42** that month
|
||||
- Still under the 3× ceiling (2,250)
|
||||
|
||||
If they were on **monthly** Pro 750 ($29.99): overflow 650 × ~$0.100 ≈ **$65** + $30 = **~$95**.
|
||||
|
||||
### Example G — White Label (we are the billed party)
|
||||
|
||||
If Verae embeds Zapier and Zapier bills **us** per task, then 10,000 customer files × 2 tasks = 20,000 tasks ≈ **$189**/mo (Pro annual list) **plus** our timestamp COGS. That number has to sit in **our** COGS, not the customer’s Zapier account.
|
||||
|
||||
---
|
||||
|
||||
## 9. Verae’s meter today (so we can place it)
|
||||
|
||||
From `verae-zapier-middleware` `PLAN_LIMITS`:
|
||||
|
||||
| Verae plan | Timestamps / mo | Verifications | Batch | RPM |
|
||||
|------------|----------------:|--------------:|-------|----:|
|
||||
| free | 50 | 50 | no | 30 |
|
||||
| starter | 500 | 500 | yes, max 10 | 120 |
|
||||
| pro | 5,000 | 5,000 | yes, max 100 | 600 |
|
||||
| enterprise | unlimited / contract | contract | yes | 3,000 |
|
||||
|
||||
Over-quota → **402** `QUOTA_EXCEEDED`. Batch on free → **403** `PLAN_UPGRADE_REQUIRED`.
|
||||
|
||||
These fire **independently** of Zapier’s pause / pay-per-task.
|
||||
|
||||
---
|
||||
|
||||
## 10. How this should shape *our* billing
|
||||
|
||||
1. **Never bundle “unlimited Zapier.”** We do not control their task tier, MCP multiplier, or overflow toggle.
|
||||
2. **Meter what we uniquely do:** accepted timestamp jobs, verifies, maybe stored GB / pin-days / Glacier restores — not Zap steps.
|
||||
3. **Mirror Zapier’s shape if we want familiarity:** plan + included units + optional overage with a ceiling. Customers already understand 402 vs pause.
|
||||
4. **Do not use Zapier-like 2× MCP pricing on our API.** MCP is Zapier’s tax on agent hops. Our HTTP `/zapier/v1/timestamp` should stay **one Verae unit** whether the Zap used 1 task or an agent used 2.
|
||||
5. **Price batch as a plan gate**, not as “N Zapier tasks.” Zapier will still charge **1 task** for the batch action; we decide whether 100 items cost 1 or 100 of *our* units. Today we count batch size against `batchMaxItems` and timestamp quota.
|
||||
6. **Hook-friendly packaging:** include enough monthly stamps that the recommended “create + hook + catalog” pattern is cheaper on **our** side than polling.
|
||||
7. **Three customer motions, three packages:**
|
||||
- **Directory user** — they pay Zapier + they pay Verae (API key). Our list price must look fair next to ~$0.02–$0.03 per Zapier task at Pro entry.
|
||||
- **MCP / agent user** — they already pay 2 Zapier tasks per hop; keep Verae simple or they will skip us.
|
||||
- **Embed / White Label** — we may owe Zapier usage; our price to the end user must cover that COGS or we only sell the timestamp and let them bring their own Zapier.
|
||||
8. **Storage / Peergos / Glacier is a third meter** (see getting-started §10). Do not hide pin-days or restore fees inside “one timestamp.”
|
||||
9. **Failed Zapier steps are free for them, not for us** if we already accepted the job. Bill on **accepted jobId** (202), not on Zapier’s success callback.
|
||||
10. **Quote two lines** in every proposal: “Your Zapier plan (estimate N tasks)” and “Verae (M timestamps).” Never a single blended number.
|
||||
|
||||
### Rough juxtaposition (order-of-magnitude)
|
||||
|
||||
At Pro 750 annual, Zapier’s included work is about **2.7 cents per task**.
|
||||
If our Create Timestamp is one Zapier task + one Verae stamp, the customer’s *Zapier* share of a two-step Zap is ~$0.027. Our stamp should be priced from **our** chain/ops cost, not matched 1:1 to that 2.7¢ — but if we charge **dollars** per stamp while Zapier is **cents** per step, SMB Zaps will feel “Verae-expensive” even when Zapier is the bigger invoice at volume.
|
||||
|
||||
---
|
||||
|
||||
## 11. Planning checklist
|
||||
|
||||
- [ ] For each target Zap, count **billable Zapier actions** (not steps on the canvas).
|
||||
- [ ] Multiply MCP executes by **2**.
|
||||
- [ ] Apply AI 1 / 3 / 5 if they enrich in the Zap.
|
||||
- [ ] Pick the cheapest **Zapier** tier that covers that task count (or overflow math).
|
||||
- [ ] Count **Verae** timestamps / verifies / batch separately; check 50 / 500 / 5,000 tripwires.
|
||||
- [ ] If we embed Zapier, put Zapier list price into **our** COGS.
|
||||
- [ ] Keep Peergos pin / Glacier restore off the timestamp SKU.
|
||||
- [ ] Re-check [zapier.com/pricing](https://zapier.com/pricing) before any contract; this brief is dated **18 August 2026**.
|
||||
33
docs/02-architecture/composition.md
Normal file
33
docs/02-architecture/composition.md
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
# Composition: zappier commercial edge + Verae adapter
|
||||
|
||||
```text
|
||||
Users → Zapier UI
|
||||
Zapier cloud runs packages/verae-zapier
|
||||
--HTTPS, x-api-key--> packages/zappier /v1/*
|
||||
(meter, quote, 401/403, usage)
|
||||
--internal HTTPS--> packages/verae-zapier-middleware /zapier/v1/*
|
||||
--sync--> api.veraetime.net or MOCK_VERAE
|
||||
--NATS--> workers --> Verae + Zapier REST Hooks
|
||||
|
||||
Humans → zappier /portal signup, API key, usage, reloads, invoices
|
||||
Ops → zappier /admin rate card, tiers, customers, PO invoices
|
||||
```
|
||||
|
||||
## Ownership
|
||||
|
||||
| Concern | Package |
|
||||
|---------|---------|
|
||||
| API keys, custom pricing, Stripe, invoices, portal | `packages/zappier` |
|
||||
| Timestamp/verify/status, NATS, REST Hooks, mock Verae | `packages/verae-zapier-middleware` |
|
||||
| Zapier Platform nouns (creates/searches/triggers) | `packages/verae-zapier` |
|
||||
| Official SDK reference | `vendor/zapier-platform` |
|
||||
|
||||
Zapier never talks to NATS or `api.veraetime.net`.
|
||||
|
||||
Do not reimplement Stripe, invoices, or the rate-card UI inside the Verae middleware.
|
||||
|
||||
Public keys are zappier `x-api-key` (issued at portal signup). Middleware `zmw_` / `PLAN_LIMITS` stay internal until composition PR 4 removes them from the public path.
|
||||
|
||||
Demo zappier routes `/v1/transform` and `/v1/storage` are a metering sandbox, not Verae timestamping or encrypted LTS.
|
||||
|
||||
Payment invoices (zappier) are not certified timestamp receipts (Verae feature i).
|
||||
101
docs/02-architecture/nats-subjects.md
Normal file
101
docs/02-architecture/nats-subjects.md
Normal file
|
|
@ -0,0 +1,101 @@
|
|||
# NATS Subject Topology
|
||||
|
||||
All subjects are prefixed with `verae.zapier.` to isolate this platform from other Verae messaging.
|
||||
|
||||
## Streams
|
||||
|
||||
| Stream | Subjects | Retention | Purpose |
|
||||
|--------|----------|-----------|---------|
|
||||
| `ZAPIER_JOBS` | `verae.zapier.jobs.watch` | Work queue | Poll Verae for job status |
|
||||
| `ZAPIER_EVENTS` | `verae.zapier.jobs.events` | Limits (time) | Terminal job outcomes |
|
||||
| `ZAPIER_WEBHOOKS` | `verae.zapier.webhooks.deliver` | Work queue | POST to Zapier hook URLs |
|
||||
| `ZAPIER_USAGE` (optional) | `verae.zapier.usage` | Limits | Billing export |
|
||||
|
||||
## Subjects
|
||||
|
||||
### `verae.zapier.jobs.watch`
|
||||
|
||||
**Published by:** HTTP edge after successful `POST /api/timestamp` (or batch item).
|
||||
**Consumed by:** `job-poller` durable consumer (queue group).
|
||||
|
||||
**Payload**
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| `tenantId` | string | yes | Owning tenant |
|
||||
| `jobId` | string | yes | Verae job id |
|
||||
| `tokenRef` | string | preferred | Opaque ref to resolve Verae credentials (avoid raw JWT) |
|
||||
| `veraeToken` | string | discouraged | Only if tokenRef unavailable; redacted in logs |
|
||||
| `enqueuedAt` | ISO-8601 | yes | Enqueue time |
|
||||
| `attempt` | number | yes | Delivery attempt (0-based) |
|
||||
| `maxAttempts` | number | yes | Stop after this many polls |
|
||||
| `intervalMs` | number | yes | Suggested delay between polls |
|
||||
| `traceId` | string | yes | Correlation id for debug |
|
||||
|
||||
### `verae.zapier.jobs.events`
|
||||
|
||||
**Published by:** Job poller when status is `completed`, `failed`, or `timeout`.
|
||||
**Consumed by:** Event router → webhook enqueue; optional waiters on HTTP edge.
|
||||
|
||||
**Payload**
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| `event` | string | yes | `timestamp.completed` \| `timestamp.failed` \| `timestamp.timeout` |
|
||||
| `tenantId` | string | yes | Tenant id |
|
||||
| `jobId` | string | yes | Job id |
|
||||
| `status` | object | yes | Verae `StatusResponse` shape (or synthetic timeout) |
|
||||
| `traceId` | string | yes | Correlation id |
|
||||
| `emittedAt` | ISO-8601 | yes | Event time |
|
||||
|
||||
### `verae.zapier.webhooks.deliver`
|
||||
|
||||
**Published by:** Event router for each matching subscription.
|
||||
**Consumed by:** `webhook-deliver` queue group.
|
||||
|
||||
**Payload**
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| `hookId` | string | yes | Stored subscription id |
|
||||
| `tenantId` | string | yes | Tenant |
|
||||
| `targetUrl` | string | yes | Zapier REST Hook URL |
|
||||
| `event` | string | yes | Event name |
|
||||
| `payload` | object | yes | Body POSTed to Zapier |
|
||||
| `attempt` | number | yes | Attempt count |
|
||||
| `traceId` | string | yes | Correlation id |
|
||||
|
||||
### `verae.zapier.usage` (optional)
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `tenantId` | string | Tenant |
|
||||
| `action` | string | `timestamp` \| `verify` \| `status` \| … |
|
||||
| `amount` | number | Increment |
|
||||
| `at` | ISO-8601 | Timestamp |
|
||||
|
||||
## Consumers
|
||||
|
||||
| Name | Stream | Mode | Notes |
|
||||
|------|--------|------|-------|
|
||||
| `job-poller` | `ZAPIER_JOBS` | Pull, queue | Nak with delay when still pending |
|
||||
| `event-webhook-router` | `ZAPIER_EVENTS` | Push/pull | Fan-out to deliver subjects |
|
||||
| `webhook-deliver` | `ZAPIER_WEBHOOKS` | Pull, queue | HTTP POST with backoff |
|
||||
| `usage-writer` | `ZAPIER_USAGE` | Optional | Persist metering |
|
||||
|
||||
## Ack semantics
|
||||
|
||||
| Situation | Action |
|
||||
|-----------|--------|
|
||||
| Job still `pending` | `Nak` with delay ≈ `intervalMs` or republish with `attempt+1` |
|
||||
| Job terminal | Publish event, `Ack` watch message |
|
||||
| Webhook HTTP 2xx | `Ack` |
|
||||
| Webhook HTTP 5xx / network | `Nak` / redelivery until `max_deliver` |
|
||||
| Poison message | Term after max_deliver; write DLQ log with `DEBUG_VERAE=webhooks` |
|
||||
|
||||
## Security rules
|
||||
|
||||
1. Prefer `tokenRef` over embedding Verae JWTs in messages.
|
||||
2. NATS must use private network + auth (Phase 15: mTLS).
|
||||
3. Debug logs must redact tokens and `targetUrl` query secrets if any.
|
||||
4. Treat `targetUrl` as untrusted egress (timeouts, size limits, SSRF allowlist later).
|
||||
132
docs/02-architecture/overview.md
Normal file
132
docs/02-architecture/overview.md
Normal file
|
|
@ -0,0 +1,132 @@
|
|||
# Architecture Overview
|
||||
|
||||
## Purpose
|
||||
|
||||
Connect Zapier automations to Verae blockchain timestamping without:
|
||||
|
||||
- exposing raw Verae JWTs to end users,
|
||||
- requiring Zapier to poll async jobs,
|
||||
- coupling billing/plans to the core timestamping API,
|
||||
- running multi-instance middleware with in-memory only job queues.
|
||||
|
||||
## Components
|
||||
|
||||
### 1. `verae-zapier` (Zapier Platform CLI app)
|
||||
|
||||
**Runs on:** Zapier’s cloud when a Zap step executes.
|
||||
|
||||
**Responsibilities:**
|
||||
|
||||
- Custom auth field `api_key` (middleware-issued `zmw_…` keys).
|
||||
- Map Zapier actions/searches/triggers to middleware HTTPS routes.
|
||||
- Attach `Authorization: Bearer <api_key>` on every request.
|
||||
- Translate middleware `402` / `403` into user-visible Zapier errors.
|
||||
|
||||
**Does not:**
|
||||
|
||||
- Call `api.veraetime.net` directly.
|
||||
- Speak NATS.
|
||||
- Enforce plan quotas (middleware does).
|
||||
|
||||
### 2. `verae-zapier-middleware` HTTP edge
|
||||
|
||||
**Runs on:** Your infrastructure (public HTTPS).
|
||||
|
||||
**Responsibilities:**
|
||||
|
||||
- Tenant identity and API keys.
|
||||
- Auth bridge: API key → Verae login → JWT (server-side).
|
||||
- Entitlement checks and usage metering.
|
||||
- Synchronous API surface under `/zapier/v1/*`.
|
||||
- REST Hook subscribe/unsubscribe storage.
|
||||
- Publish async work to NATS when `NATS_ENABLED=true`.
|
||||
|
||||
### 3. NATS + JetStream
|
||||
|
||||
**Runs on:** Private network with middleware.
|
||||
|
||||
**Responsibilities:**
|
||||
|
||||
- Durable work queue for job status polling.
|
||||
- Event stream for terminal job states.
|
||||
- Work queue for Zapier webhook HTTP delivery with retries.
|
||||
|
||||
### 4. Workers
|
||||
|
||||
**Runs on:** Same deploy as middleware or separate worker processes.
|
||||
|
||||
| Worker | Consumes | Calls |
|
||||
|--------|----------|-------|
|
||||
| Job poller | `verae.zapier.jobs.watch` | `GET /api/status/{jobId}` on Verae |
|
||||
| Event router | `verae.zapier.jobs.events` | Enqueues webhook deliveries |
|
||||
| Webhook deliver | `verae.zapier.webhooks.deliver` | `POST` Zapier `targetUrl` |
|
||||
|
||||
### 5. `api.veraetime.net` (Verae Timestamping Service)
|
||||
|
||||
**Source of truth** for login, timestamp jobs, status, and verification.
|
||||
|
||||
OpenAPI: production Swagger / `Verae-Swagger.yaml`.
|
||||
|
||||
## Request flows
|
||||
|
||||
### A. Create Timestamp (async)
|
||||
|
||||
```text
|
||||
Zapier → POST /zapier/v1/timestamp
|
||||
Middleware: authenticate, checkEntitlement, POST /api/timestamp
|
||||
Middleware: publish jobs.watch → return 202 { jobId }
|
||||
Worker: poll status until terminal → publish jobs.events
|
||||
Event router: match webhooks → publish webhooks.deliver
|
||||
Webhook worker: POST hooks.zapier.com/...
|
||||
```
|
||||
|
||||
### B. Create Timestamp and Wait
|
||||
|
||||
```text
|
||||
Zapier → POST /zapier/v1/timestamp/wait
|
||||
Middleware: create + wait for jobs.events (or in-process wait if NATS off)
|
||||
→ return StatusResponse (completed/failed) or pending+jobId on timeout
|
||||
```
|
||||
|
||||
### C. Auth connection test
|
||||
|
||||
```text
|
||||
Zapier → GET /zapier/v1/auth/me Authorization: Bearer zmw_…
|
||||
Middleware: resolve API key → tenant → optional validate Verae token
|
||||
→ { tenantId, plan, usage, ... }
|
||||
```
|
||||
|
||||
## Feature flags
|
||||
|
||||
| Flag | Effect |
|
||||
|------|--------|
|
||||
| `NATS_ENABLED=false` | In-process job poller; still full HTTP API (Phase 6 path) |
|
||||
| `NATS_ENABLED=true` | JetStream workers; no in-process poller |
|
||||
| `MOCK_VERAE=true` | No live Verae; deterministic mock jobs for tests |
|
||||
| `DEBUG_VERAE=…` | Runtime failure tracing (see debugging.md) |
|
||||
|
||||
## Security boundaries
|
||||
|
||||
```text
|
||||
Public Internet
|
||||
├─ Zapier → Middleware HTTPS only
|
||||
└─ Middleware → Zapier webhook HTTPS only
|
||||
|
||||
Private
|
||||
├─ Middleware ↔ NATS
|
||||
└─ Middleware/Workers → api.veraetime.net HTTPS
|
||||
```
|
||||
|
||||
Never expose NATS ports to the public internet.
|
||||
|
||||
## Scaling model
|
||||
|
||||
- **HTTP edge:** scale replicas behind a load balancer (stateless except shared store).
|
||||
- **NATS consumers:** queue groups — adding workers increases poll/deliver throughput.
|
||||
- **Store:** file JSON is MVP single-node; Postgres/Redis required for multi-node (Phase 15).
|
||||
|
||||
## Related documents
|
||||
|
||||
- [nats-subjects.md](nats-subjects.md) — subjects, streams, payload schemas
|
||||
- [../plans/phase-gates.md](../plans/phase-gates.md) — test gates
|
||||
- [../../TODO.md](../../TODO.md) — full implementation order
|
||||
17
docs/03-zapier/auth.md
Normal file
17
docs/03-zapier/auth.md
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
# Zapier authentication
|
||||
|
||||
## Target (after PR 5)
|
||||
|
||||
- Type: custom / API key.
|
||||
- Fields: `apiKey`, `baseUrl` (default public zappier origin).
|
||||
- Header: `x-api-key`.
|
||||
- Test: `GET /v1/status` or `GET /v1/me` if added.
|
||||
- Errors: 401 invalid key; 403/402 quota or unpriced endpoint → user-visible upgrade text pointing at `/portal`.
|
||||
|
||||
## Today (imported CLI app)
|
||||
|
||||
- Bearer middleware API key `zmw_…`.
|
||||
- Test: `GET /zapier/v1/auth/me`.
|
||||
- `afterResponse` maps 402 and `PLAN_UPGRADE_REQUIRED`.
|
||||
|
||||
Do not send Verae JWTs to Zapier. Middleware (or zappier→middleware) logs into Verae server-side.
|
||||
16
docs/03-zapier/operations.md
Normal file
16
docs/03-zapier/operations.md
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
# Zapier operations
|
||||
|
||||
Current package: `packages/verae-zapier` (imported from datacubes; still aimed at middleware).
|
||||
|
||||
| Noun | Type | Middleware today | Target public path (zappier) |
|
||||
|------|------|------------------|------------------------------|
|
||||
| Create Timestamp | create | `POST /zapier/v1/timestamp` | `POST /v1/timestamp` |
|
||||
| Create Timestamp and Wait | create | `POST /zapier/v1/timestamp/wait` | `POST /v1/timestamp/wait` |
|
||||
| Create Batch Timestamps | create | `POST /zapier/v1/timestamp/batch` | `POST /v1/timestamp/batch` |
|
||||
| Verify Certificate | create | `POST /zapier/v1/verify` | `POST /v1/verify` |
|
||||
| Find Job Status | search | `GET /zapier/v1/status/{jobId}` | `GET /v1/status/{jobId}` |
|
||||
| Timestamp Completed | REST Hook | subscribe/unsubscribe webhooks | proxy through zappier or map customer id |
|
||||
|
||||
Later: Find by SHA256, retrieve metadata, certified receipt, Find Usage, List Invoices.
|
||||
|
||||
Reference examples: `vendor/zapier-platform/example-apps/` (custom-auth, files, rest-hooks, create).
|
||||
80
docs/api/middleware-openapi.yaml
Normal file
80
docs/api/middleware-openapi.yaml
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
openapi: 3.0.0
|
||||
info:
|
||||
title: Verae Zapier Middleware API
|
||||
description: |
|
||||
Zapier-facing HTTP surface. Zapier CLI app calls these routes only.
|
||||
Async job watching and webhook delivery may use NATS behind this API.
|
||||
version: 0.1.0
|
||||
servers:
|
||||
- url: http://localhost:3100
|
||||
description: Local middleware
|
||||
paths:
|
||||
/health:
|
||||
get:
|
||||
summary: Liveness probe
|
||||
operationId: health
|
||||
responses:
|
||||
'200':
|
||||
description: OK
|
||||
/zapier/v1/auth/me:
|
||||
get:
|
||||
summary: Validate API key connection (Zapier auth test)
|
||||
operationId: authMe
|
||||
security:
|
||||
- BearerAuth: []
|
||||
responses:
|
||||
'200':
|
||||
description: Tenant + plan + usage
|
||||
'401':
|
||||
description: Unauthorized
|
||||
/zapier/v1/timestamp:
|
||||
post:
|
||||
summary: Create timestamp (async jobId)
|
||||
operationId: createTimestamp
|
||||
security:
|
||||
- BearerAuth: []
|
||||
responses:
|
||||
'202':
|
||||
description: Accepted
|
||||
/zapier/v1/timestamp/wait:
|
||||
post:
|
||||
summary: Create timestamp and wait for completion
|
||||
operationId: createTimestampAndWait
|
||||
security:
|
||||
- BearerAuth: []
|
||||
responses:
|
||||
'200':
|
||||
description: Completed or failed status
|
||||
/zapier/v1/verify:
|
||||
post:
|
||||
summary: Verify certificate
|
||||
operationId: verifyTimestamp
|
||||
security:
|
||||
- BearerAuth: []
|
||||
responses:
|
||||
'200':
|
||||
description: Verification result
|
||||
/zapier/v1/webhooks/subscribe:
|
||||
post:
|
||||
summary: REST Hook subscribe
|
||||
operationId: webhookSubscribe
|
||||
security:
|
||||
- BearerAuth: []
|
||||
responses:
|
||||
'201':
|
||||
description: Subscribed
|
||||
/zapier/v1/webhooks/unsubscribe:
|
||||
delete:
|
||||
summary: REST Hook unsubscribe
|
||||
operationId: webhookUnsubscribe
|
||||
security:
|
||||
- BearerAuth: []
|
||||
responses:
|
||||
'200':
|
||||
description: Unsubscribed
|
||||
components:
|
||||
securitySchemes:
|
||||
BearerAuth:
|
||||
type: http
|
||||
scheme: bearer
|
||||
description: Middleware API key (zmw_…) or session token (zmt_…)
|
||||
147
docs/developer/debugging.md
Normal file
147
docs/developer/debugging.md
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
# Runtime Debugging and Failure Tracing
|
||||
|
||||
## Goals
|
||||
|
||||
- Trace failures across **HTTP → NATS → Verae → Zapier webhook** without redeploying.
|
||||
- Keep secrets out of logs even when debug is enabled.
|
||||
- Allow selective namespaces so production noise stays low.
|
||||
|
||||
## Enabling debug (runtime)
|
||||
|
||||
Debug is **off** when `DEBUG_VERAE` is unset or empty.
|
||||
|
||||
```bash
|
||||
# Everything
|
||||
export DEBUG_VERAE=1
|
||||
# or
|
||||
export DEBUG_VERAE=*
|
||||
|
||||
# Selected namespaces (comma-separated)
|
||||
export DEBUG_VERAE=auth,nats,jobs,webhooks,http,billing,trace
|
||||
|
||||
# Minimum level: debug | info | warn | error
|
||||
export DEBUG_VERAE_LEVEL=debug
|
||||
|
||||
# Optional: write to file as well as stderr
|
||||
export DEBUG_VERAE_FILE=/var/log/verae-zapier-debug.log
|
||||
```
|
||||
|
||||
Restart is **not** required if the process reads env only at boot — current implementation reads env at process start. To change flags:
|
||||
|
||||
```bash
|
||||
# systemd / docker: update env and restart one replica
|
||||
# or send future SIGHUP support (Phase 15)
|
||||
```
|
||||
|
||||
### Docker example
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
DEBUG_VERAE: "auth,jobs,nats,webhooks"
|
||||
DEBUG_VERAE_LEVEL: "debug"
|
||||
```
|
||||
|
||||
## Namespaces
|
||||
|
||||
| Namespace | What it traces |
|
||||
|-----------|----------------|
|
||||
| `auth` | API key resolve, session parse, login outcomes (no passwords) |
|
||||
| `billing` | Entitlement checks, quota decisions, plan limits |
|
||||
| `http` | Outbound Verae requests: method, path, status, duration |
|
||||
| `nats` | Connect, publish, consume, ack/nak, stream ensure |
|
||||
| `jobs` | Watch enqueue, poll attempts, terminal transitions |
|
||||
| `webhooks` | Subscribe, deliver attempts, HTTP status to Zapier |
|
||||
| `trace` | Correlation id enter/exit spans |
|
||||
| `app` | Boot, config summary (redacted), shutdown |
|
||||
|
||||
`DEBUG_VERAE=1` or `*` enables **all** namespaces.
|
||||
|
||||
## Correlation IDs
|
||||
|
||||
Every inbound HTTP request should get a `traceId` (generated or from `X-Trace-Id` header).
|
||||
|
||||
That id is:
|
||||
|
||||
- returned optionally as `X-Trace-Id` on responses,
|
||||
- attached to NATS payloads as `traceId`,
|
||||
- included in every debug line for that flow.
|
||||
|
||||
Example log line:
|
||||
|
||||
```text
|
||||
2026-08-11T16:00:00.000Z DEBUG jobs [trace=a1b2c3d4] poll attempt=3 jobId=550e… status=pending
|
||||
```
|
||||
|
||||
## Redaction rules
|
||||
|
||||
Always redacted (replaced with `[REDACTED]`):
|
||||
|
||||
- Headers: `authorization`, `x-api-key`, `cookie`
|
||||
- Fields named: `password`, `veraePassword`, `token`, `veraeToken`, `accessToken`, `apiKey`, `api_key`, `secret`
|
||||
- String values matching: `Bearer …`, `zmw_…`, `zmt_…`, long JWTs (`eyJ…`)
|
||||
|
||||
`targetUrl` host is kept; query string may be stripped if it contains tokens.
|
||||
|
||||
## Using debug for common failures
|
||||
|
||||
### 401 from middleware
|
||||
|
||||
```bash
|
||||
DEBUG_VERAE=auth,trace npm start
|
||||
# reproduce Zapier connection test
|
||||
# look for resolveAuthContext failures
|
||||
```
|
||||
|
||||
### Job never completes / trigger never fires
|
||||
|
||||
```bash
|
||||
DEBUG_VERAE=jobs,nats,webhooks,http
|
||||
# confirm: watch published → poll status → event emitted → deliver POST status
|
||||
```
|
||||
|
||||
### 402 quota
|
||||
|
||||
```bash
|
||||
DEBUG_VERAE=billing
|
||||
# confirm plan limits vs usage counters
|
||||
```
|
||||
|
||||
### Verae upstream errors
|
||||
|
||||
```bash
|
||||
DEBUG_VERAE=http,jobs
|
||||
# status codes and paths only; body may be summarized
|
||||
```
|
||||
|
||||
## Programmatic API
|
||||
|
||||
See [modules/debug.md](modules/debug.md) and source under `verae-zapier-middleware/src/debug/`.
|
||||
|
||||
```js
|
||||
import { createDebugger } from '../debug/logger.js';
|
||||
import { withTrace, getTraceId } from '../debug/trace.js';
|
||||
|
||||
const log = createDebugger('jobs');
|
||||
|
||||
log.debug('poll start', { jobId });
|
||||
log.error('poll failed', { jobId, err: err.message });
|
||||
```
|
||||
|
||||
## Tests
|
||||
|
||||
Phase 1 gate verifies:
|
||||
|
||||
- silence when disabled,
|
||||
- namespace filtering,
|
||||
- redaction,
|
||||
- trace id propagation.
|
||||
|
||||
```bash
|
||||
npm run gate:1
|
||||
```
|
||||
|
||||
## Production caution
|
||||
|
||||
- Prefer staging with narrowed namespaces.
|
||||
- Never commit files containing live debug output with customer data.
|
||||
- CI should run with debug off except dedicated debug unit tests that assert redaction.
|
||||
45
docs/developer/modules/README.md
Normal file
45
docs/developer/modules/README.md
Normal file
|
|
@ -0,0 +1,45 @@
|
|||
# Developer Module Index
|
||||
|
||||
Each module document lists **every exported function**, its purpose, inputs, outputs, and side effects.
|
||||
|
||||
Implementation status follows [TODO.md](../../../TODO.md) phases. Docs describe the **target contract** so implementers and reviewers share one API surface.
|
||||
|
||||
## Middleware (`verae-zapier-middleware/src`)
|
||||
|
||||
| Module doc | Source path | Phase |
|
||||
|------------|-------------|-------|
|
||||
| [debug.md](debug.md) | `debug/` | 1 |
|
||||
| [config.md](config.md) | `config.js` | 2 |
|
||||
| [errors.md](errors.md) | `errors.js` | 2 |
|
||||
| [app.md](app.md) | `app.js`, `index.js` | 2 |
|
||||
| [store-db.md](store-db.md) | `store/db.js` | 3 |
|
||||
| [store-tenants.md](store-tenants.md) | `store/tenants.js` | 3 |
|
||||
| [store-usage.md](store-usage.md) | `store/usage.js` | 3 |
|
||||
| [store-webhooks.md](store-webhooks.md) | `store/webhooks.js` | 3 |
|
||||
| [tokens.md](tokens.md) | `lib/tokens.js` | 4 |
|
||||
| [veraeClient.md](veraeClient.md) | `clients/veraeClient.js` | 4 |
|
||||
| [authService.md](authService.md) | `services/authService.js` | 5 |
|
||||
| [entitlementService.md](entitlementService.md) | `services/entitlementService.js` | 5 |
|
||||
| [middleware-http.md](middleware-http.md) | `middleware/*.js` | 5 |
|
||||
| [timestampService.md](timestampService.md) | `services/timestampService.js` | 6 |
|
||||
| [verifyService.md](verifyService.md) | `services/verifyService.js` | 6 |
|
||||
| [webhookService.md](webhookService.md) | `services/webhookService.js` | 6 |
|
||||
| [routes.md](routes.md) | `routes/*.js` | 6 |
|
||||
| [nats.md](nats.md) | `nats/*.js` | 7 |
|
||||
| [workers.md](workers.md) | `workers/*.js` | 8–9 |
|
||||
| [tenantService.md](tenantService.md) | `services/tenantService.js` | 10 |
|
||||
|
||||
## Zapier app (`verae-zapier`)
|
||||
|
||||
| Module doc | Source path | Phase |
|
||||
|------------|-------------|-------|
|
||||
| [zapier-app.md](zapier-app.md) | `index.js`, `authentication.js`, creates/searches/triggers | 11 |
|
||||
|
||||
## Documentation rules
|
||||
|
||||
When you add or change an export:
|
||||
|
||||
1. Update JSDoc in the source file (`@param`, `@returns`, purpose sentence).
|
||||
2. Update the matching module markdown table.
|
||||
3. Add/adjust unit tests in the phase that owns the module.
|
||||
4. Instrument boundaries with `createDebugger('<namespace>')`.
|
||||
27
docs/developer/modules/config.md
Normal file
27
docs/developer/modules/config.md
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
# Module: `config`
|
||||
|
||||
**Path:** `verae-zapier-middleware/src/config.js`
|
||||
**Phase:** 2
|
||||
**Debug namespace:** `app`
|
||||
|
||||
## Purpose
|
||||
|
||||
Central runtime settings for HTTP, Verae upstream, NATS flags, and plan limits.
|
||||
|
||||
## Exports
|
||||
|
||||
### `loadEnvFile()`
|
||||
|
||||
Loads `.env` without overriding existing env vars. Called at module load.
|
||||
|
||||
### `config`
|
||||
|
||||
See [function-reference.md](function-reference.md#configjs) for field table.
|
||||
|
||||
### `PLAN_LIMITS`
|
||||
|
||||
Default quotas per plan name (`free` | `starter` | `pro` | `enterprise`).
|
||||
|
||||
## Environment variables
|
||||
|
||||
Documented in `verae-zapier-middleware/.env.example` and root `README.md`.
|
||||
103
docs/developer/modules/debug.md
Normal file
103
docs/developer/modules/debug.md
Normal file
|
|
@ -0,0 +1,103 @@
|
|||
# Module: `debug`
|
||||
|
||||
**Path:** `verae-zapier-middleware/src/debug/`
|
||||
**Phase:** 1
|
||||
**Purpose:** Runtime-toggleable failure tracing with secret redaction and correlation IDs.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `config.js` | Parse env into enable/namespaces/level |
|
||||
| `redact.js` | Strip secrets from log metadata |
|
||||
| `logger.js` | Namespaced `createDebugger` |
|
||||
| `trace-context.js` | ALS store (no logger import) |
|
||||
| `trace.js` | `withTrace`, Express middleware |
|
||||
| `index.js` | Public barrel export |
|
||||
|
||||
---
|
||||
|
||||
## `parseDebugVeraeEnv(raw)`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | Interpret `DEBUG_VERAE` env string |
|
||||
| **Input** | `raw: string \| undefined` |
|
||||
| **Output** | `{ enabled: boolean, namespaces: Set<string>\|null }` |
|
||||
| **Notes** | `1`/`*` → all namespaces; `"auth,nats"` → allow-list |
|
||||
|
||||
## `loadDebugConfig(env?)`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | Build full debug config from environment |
|
||||
| **Input** | `env?: NodeJS.ProcessEnv` (default `process.env`) |
|
||||
| **Output** | `DebugConfig` `{ enabled, namespaces, level, filePath }` |
|
||||
|
||||
## `shouldLog(config, namespace, messageLevel)`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | Decide if a line should emit |
|
||||
| **Input** | config, namespace string, level |
|
||||
| **Output** | `boolean` |
|
||||
|
||||
## `redact(input)`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | Deep-redact secrets for safe logging |
|
||||
| **Input** | any JSON-like value |
|
||||
| **Output** | redacted clone |
|
||||
|
||||
## `createDebugger(namespace)`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | Get `{ debug, info, warn, error }` logger for a namespace |
|
||||
| **Input** | `namespace: string` |
|
||||
| **Output** | `Debugger` |
|
||||
| **Side effects** | Writes to stderr (or test sink) when enabled |
|
||||
|
||||
### `Debugger.debug|info|warn|error(message, meta?)`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **Input** | `message: string`, `meta?: Record<string, unknown>` |
|
||||
| **Output** | `void` |
|
||||
|
||||
## `withTrace(options, fn)`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | Run work under a correlation id |
|
||||
| **Input** | `{ traceId?, span? }`, `fn: () => T\|Promise<T>` |
|
||||
| **Output** | `T\|Promise<T>` |
|
||||
|
||||
## `getTraceId()`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | Read active correlation id |
|
||||
| **Output** | `string\|null` |
|
||||
|
||||
## `traceMiddleware(req, res, next)`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | Per-request Express trace context |
|
||||
| **Input** | Express `(req, res, next)` |
|
||||
| **Output** | `void` |
|
||||
| **Sets** | `req.traceId`, response header `X-Trace-Id` |
|
||||
|
||||
## `setDebugTestSink(sink)`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | Capture log lines in unit tests |
|
||||
| **Input** | `((line: string) => void) \| null` |
|
||||
| **Output** | `void` |
|
||||
|
||||
## Env reference
|
||||
|
||||
See [../debugging.md](../debugging.md).
|
||||
486
docs/developer/modules/function-reference.md
Normal file
486
docs/developer/modules/function-reference.md
Normal file
|
|
@ -0,0 +1,486 @@
|
|||
# Function Reference — All Modules
|
||||
|
||||
Canonical I/O contracts for the Verae Zapier middleware and Zapier app.
|
||||
Source of truth for reviewers; keep in sync with JSDoc in code.
|
||||
|
||||
Namespaces for debug: `auth`, `billing`, `http`, `nats`, `jobs`, `webhooks`, `trace`, `app`.
|
||||
|
||||
---
|
||||
|
||||
## config.js
|
||||
|
||||
### `loadEnvFile()`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | Load `.env` into `process.env` if keys are unset |
|
||||
| **Input** | none (reads file next to package root) |
|
||||
| **Output** | `void` |
|
||||
|
||||
### `config` (exported object)
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `port` | number | HTTP port |
|
||||
| `host` | string | Bind address |
|
||||
| `veraeApiBaseUrl` | string | Upstream Verae base URL (no trailing slash) |
|
||||
| `mockVerae` | boolean | Use mock client |
|
||||
| `natsEnabled` | boolean | Use JetStream workers |
|
||||
| `natsUrl` | string | NATS connection URL |
|
||||
| `tokenSecret` | string | HMAC secret for `zmt_` tokens |
|
||||
| `jobPollIntervalMs` | number | Poll delay |
|
||||
| `jobPollMaxAttempts` | number | Max poll attempts |
|
||||
| `storePath` | string | Absolute path to store file |
|
||||
| `upgradeUrl` | string | Billing upgrade link for 402 bodies |
|
||||
| `adminSecret` | string | Admin route secret |
|
||||
|
||||
### `PLAN_LIMITS`
|
||||
|
||||
Map of plan name → `{ timestamps, verifications, batch, batchMaxItems, requestsPerMinute }`.
|
||||
`null` means unlimited.
|
||||
|
||||
---
|
||||
|
||||
## errors.js
|
||||
|
||||
### `class AppError extends Error`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | Structured operational errors returned as JSON |
|
||||
| **Constructor** | `(message: string, { status?, code?, details? })` |
|
||||
| **Properties** | `status: number`, `code: string`, `details: unknown` |
|
||||
|
||||
### `asyncHandler(fn)`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | Wrap async Express handlers so rejections hit error middleware |
|
||||
| **Input** | `(req, res, next) => Promise<any>` |
|
||||
| **Output** | Express middleware function |
|
||||
|
||||
### `sendError(res, err)`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | Write JSON error response; log 5xx with debug |
|
||||
| **Input** | Express `res`, `Error\|AppError` |
|
||||
| **Output** | `void` |
|
||||
|
||||
---
|
||||
|
||||
## app.js / index.js
|
||||
|
||||
### `createApp()`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | Build Express application (no listen) |
|
||||
| **Input** | none |
|
||||
| **Output** | `express.Application` |
|
||||
| **Mounts** | `GET /health`, `/zapier/*`, error handler, `traceMiddleware` |
|
||||
|
||||
### `main()` (index.js)
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | Boot store, optional NATS, workers, listen |
|
||||
| **Input** | none |
|
||||
| **Output** | `Promise<void>` |
|
||||
|
||||
---
|
||||
|
||||
## store/db.js
|
||||
|
||||
### `loadStore()`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | Load JSON store into memory |
|
||||
| **Output** | store object |
|
||||
|
||||
### `getStore()`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | Access in-memory store |
|
||||
| **Output** | `{ tenants, apiKeys, usage, webhooks, jobWatchers }` |
|
||||
|
||||
### `persist()`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | Flush store to disk |
|
||||
| **Output** | `void` |
|
||||
|
||||
---
|
||||
|
||||
## store/tenants.js
|
||||
|
||||
### `getTenant(tenantId)`
|
||||
|
||||
| Input | `tenantId: string` |
|
||||
| Output | `Tenant\|null` |
|
||||
|
||||
### `getTenantByApiKey(apiKey)`
|
||||
|
||||
| Input | `apiKey: string` |
|
||||
| Output | `Tenant\|null` |
|
||||
|
||||
### `listTenants()`
|
||||
|
||||
| Output | `Tenant[]` |
|
||||
|
||||
### `createTenant({ id, name, plan, veraeUsername, veraePassword, contract?, apiKey?, metadata? })`
|
||||
|
||||
| Output | `{ tenant: Tenant, apiKey: string }` |
|
||||
|
||||
### `resolveLimits(tenant)`
|
||||
|
||||
| Output | Effective limits including enterprise contract overrides |
|
||||
|
||||
---
|
||||
|
||||
## store/usage.js
|
||||
|
||||
### `getUsage(tenantId)`
|
||||
|
||||
| Output | counters for current period |
|
||||
|
||||
### `getUsageSummary(tenantId)`
|
||||
|
||||
| Output | public-safe usage + limits snapshot |
|
||||
|
||||
### `incrementUsage(tenantId, metric, amount?)`
|
||||
|
||||
| For | Atomically increment a counter and persist |
|
||||
|
||||
---
|
||||
|
||||
## store/webhooks.js
|
||||
|
||||
### `createWebhook({ tenantId, targetUrl, event })`
|
||||
|
||||
| Output | `{ id, tenantId, targetUrl, event, createdAt }` |
|
||||
|
||||
### `deleteWebhook({ tenantId, hookId?, targetUrl? })`
|
||||
|
||||
| Output | `boolean` removed |
|
||||
|
||||
### `getActiveWebhooks(tenantId, event)`
|
||||
|
||||
| Output | `Webhook[]` |
|
||||
|
||||
### `listWebhooksForTenant(tenantId)`
|
||||
|
||||
| Output | `Webhook[]` |
|
||||
|
||||
---
|
||||
|
||||
## lib/tokens.js
|
||||
|
||||
### `issueSessionToken({ tenantId, veraeToken, expiresAt })`
|
||||
|
||||
| Output | `string` (`zmt_…`) |
|
||||
|
||||
### `parseSessionToken(token)`
|
||||
|
||||
| Output | `{ tenantId, veraeToken, expiresAt, nonce }\|null` |
|
||||
|
||||
### `generateApiKey()`
|
||||
|
||||
| Output | `string` (`zmw_…`) |
|
||||
|
||||
### `isApiKey(value)`
|
||||
|
||||
| Output | `boolean` |
|
||||
|
||||
### `extractBearerToken(header)`
|
||||
|
||||
| Input | `Authorization` header string |
|
||||
| Output | token string or `null` |
|
||||
|
||||
---
|
||||
|
||||
## clients/veraeClient.js
|
||||
|
||||
All methods that hit the network log under `DEBUG_VERAE=http`.
|
||||
|
||||
### `veraeClient.login(credentials)`
|
||||
|
||||
| Input | `{ username, password }` |
|
||||
| Output | `{ token, expiresAt, user }` |
|
||||
|
||||
### `veraeClient.validate(token)`
|
||||
|
||||
| Input | Verae JWT |
|
||||
| Output | validation object |
|
||||
|
||||
### `veraeClient.createTimestamp(token, body)`
|
||||
|
||||
| Input | JWT, `{ data, hashAlg? }` |
|
||||
| Output | `{ jobId }` |
|
||||
|
||||
### `veraeClient.createBatchTimestamp(token, body)`
|
||||
|
||||
| Input | JWT, `{ items: [{ data, hashAlg? }] }` |
|
||||
| Output | `{ jobIds: string[] }` |
|
||||
|
||||
### `veraeClient.getStatus(token, jobId)`
|
||||
|
||||
| Output | StatusResponse |
|
||||
|
||||
### `veraeClient.getBatchStatus(token, body)`
|
||||
|
||||
| Input | `{ jobIds: string[] }` |
|
||||
| Output | `{ results }` |
|
||||
|
||||
### `veraeClient.verify(token, body)`
|
||||
|
||||
| Input | `{ certificate }` |
|
||||
| Output | `{ valid, timestamp?, blockIndex? }` |
|
||||
|
||||
### `veraeClient.verifyBatch(token, body)`
|
||||
|
||||
| Input | `{ certificates: string[] }` |
|
||||
| Output | `{ results }` |
|
||||
|
||||
### `veraeClient.waitForJob(token, jobId, { maxAttempts, intervalMs })`
|
||||
|
||||
| Output | terminal StatusResponse or throws `GATEWAY_TIMEOUT` |
|
||||
|
||||
---
|
||||
|
||||
## services/authService.js
|
||||
|
||||
Debug namespace: `auth`.
|
||||
|
||||
### `loginWithCredentials({ username, password, tenant? })`
|
||||
|
||||
| Output | `{ accessToken, expiresAt, tenant, user }` |
|
||||
|
||||
### `loginWithApiKey(apiKey)`
|
||||
|
||||
| Output | same as loginWithCredentials after tenant lookup |
|
||||
|
||||
### `resolveAuthContext(rawToken)`
|
||||
|
||||
| Input | API key or session token |
|
||||
| Output | `{ tenantId, tenant, veraeToken, authMethod }` |
|
||||
|
||||
### `validateSession(rawToken)`
|
||||
|
||||
| Output | `{ valid, tenantId, authMethod, user }` |
|
||||
|
||||
---
|
||||
|
||||
## services/entitlementService.js
|
||||
|
||||
Debug namespace: `billing`.
|
||||
|
||||
### `checkEntitlement(tenantId, action, { amount? })`
|
||||
|
||||
| Actions | `timestamp`, `verify`, `batch_timestamp` |
|
||||
| Throws | `402 QUOTA_EXCEEDED`, `403 PLAN_UPGRADE_REQUIRED` |
|
||||
| Output | `{ tenant, limits, usage }` |
|
||||
|
||||
### `recordUsage(tenantId, action, { amount? })`
|
||||
|
||||
| Output | `void` |
|
||||
|
||||
---
|
||||
|
||||
## middleware/authenticate.js
|
||||
|
||||
### `authenticate(req, res, next)`
|
||||
|
||||
| For | Populate `req.auth` via `resolveAuthContext` |
|
||||
| Reads | `Authorization: Bearer` or `x-api-key` |
|
||||
|
||||
## middleware/rateLimit.js
|
||||
|
||||
### `rateLimit(req, res, next)`
|
||||
|
||||
| For | Enforce plan `requestsPerMinute` |
|
||||
| Throws | `429 RATE_LIMITED` |
|
||||
|
||||
---
|
||||
|
||||
## services/timestampService.js
|
||||
|
||||
Debug: `jobs`, `billing`, `http`.
|
||||
|
||||
### `createTimestamp(ctx, body)`
|
||||
|
||||
| Input | auth context, `{ data, hashAlg? }` |
|
||||
| Output | `{ jobId }` |
|
||||
| Side effects | usage++, enqueue watch (NATS or in-process) |
|
||||
|
||||
### `createTimestampAndWait(ctx, body)`
|
||||
|
||||
| Output | StatusResponse (or pending on timeout when NATS wait enabled) |
|
||||
|
||||
### `createBatchTimestamp(ctx, body)`
|
||||
|
||||
| Output | `{ jobIds }` |
|
||||
|
||||
### `getJobStatus(ctx, jobId)` / `getBatchJobStatus` / `getJobVerification`
|
||||
|
||||
| Output | status payloads from Verae |
|
||||
|
||||
---
|
||||
|
||||
## services/verifyService.js
|
||||
|
||||
### `verifyTimestamp(ctx, body)` / `verifyBatch(ctx, body)`
|
||||
|
||||
| Output | verify results; records usage |
|
||||
|
||||
---
|
||||
|
||||
## services/webhookService.js
|
||||
|
||||
Debug: `webhooks`.
|
||||
|
||||
### `subscribe(ctx, { targetUrl, event })`
|
||||
|
||||
| Output | webhook record |
|
||||
|
||||
### `unsubscribe(ctx, { hookId?, targetUrl? })`
|
||||
|
||||
| Output | `{ removed: true }` |
|
||||
|
||||
### `deliverWebhook(targetUrl, payload)`
|
||||
|
||||
| Output | `{ ok: boolean, status: number }` |
|
||||
|
||||
---
|
||||
|
||||
## services/tenantService.js
|
||||
|
||||
### `selfServeSignup({ email, name, veraeUsername, veraePassword })`
|
||||
|
||||
| Output | `{ tenant, apiKey, zapierSetup }` |
|
||||
|
||||
### `provisionTenant({ id?, name, plan, veraeUsername, veraePassword, contract?, metadata?, audience? })`
|
||||
|
||||
| Output | `{ tenant, apiKey }` |
|
||||
|
||||
### `listProvisionedTenants()`
|
||||
|
||||
| Output | public tenant summaries (no passwords) |
|
||||
|
||||
---
|
||||
|
||||
## nats/subjects.js
|
||||
|
||||
### Constants
|
||||
|
||||
`SUBJECTS.JOBS_WATCH`, `JOBS_EVENTS`, `WEBHOOKS_DELIVER`, `USAGE`
|
||||
`STREAMS.ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`
|
||||
|
||||
---
|
||||
|
||||
## nats/connection.js
|
||||
|
||||
### `connectNats(url?)`
|
||||
|
||||
| Output | `{ nc, js, jsm }` NATS connection handles |
|
||||
|
||||
### `ensureStreams(jsm)`
|
||||
|
||||
| For | Idempotent stream create |
|
||||
| Output | `Promise<void>` |
|
||||
|
||||
### `closeNats()`
|
||||
|
||||
| Output | `Promise<void>` |
|
||||
|
||||
---
|
||||
|
||||
## nats/publishers.js
|
||||
|
||||
### `enqueueWatch(jobWatchPayload)`
|
||||
|
||||
| Input | see architecture/nats-subjects.md |
|
||||
| Output | `Promise<{ seq }>` |
|
||||
|
||||
### `publishJobEvent(eventPayload)`
|
||||
|
||||
| Output | `Promise<void>` |
|
||||
|
||||
### `enqueueWebhook(deliverPayload)`
|
||||
|
||||
| Output | `Promise<void>` |
|
||||
|
||||
---
|
||||
|
||||
## workers/jobPollerWorker.js
|
||||
|
||||
### `startJobPollerWorker()`
|
||||
|
||||
| For | Pull-consume watch queue; poll Verae; emit events |
|
||||
| Output | stop handle `{ stop() }` |
|
||||
|
||||
## workers/webhookWorker.js
|
||||
|
||||
### `startWebhookWorker()`
|
||||
|
||||
| For | Deliver POSTs to Zapier with retry via JetStream |
|
||||
| Output | `{ stop() }` |
|
||||
|
||||
## workers/inProcessJobPoller.js
|
||||
|
||||
### `startInProcessJobPoller()` / `stopInProcessJobPoller()`
|
||||
|
||||
| For | Phase 6 fallback when `NATS_ENABLED=false` |
|
||||
|
||||
---
|
||||
|
||||
## Routes (all under `/zapier`)
|
||||
|
||||
| Method | Path | Handler purpose |
|
||||
|--------|------|-----------------|
|
||||
| POST | `/v1/auth/login` | Credentials or api_key → session |
|
||||
| GET | `/v1/auth/me` | Validate connection for Zapier |
|
||||
| POST | `/v1/timestamp` | Async create |
|
||||
| POST | `/v1/timestamp/wait` | Create and wait |
|
||||
| POST | `/v1/timestamp/batch` | Batch create |
|
||||
| POST | `/v1/verify` | Verify certificate |
|
||||
| GET | `/v1/status/:jobId` | Job status |
|
||||
| POST | `/v1/webhooks/subscribe` | REST Hook subscribe |
|
||||
| DELETE | `/v1/webhooks/unsubscribe` | REST Hook unsubscribe |
|
||||
| POST | `/v1/signup` | Self-serve tenant |
|
||||
| POST | `/v1/admin/tenants` | Admin provision |
|
||||
|
||||
---
|
||||
|
||||
## Zapier app modules
|
||||
|
||||
### `authentication.test` / fields
|
||||
|
||||
| Input from user | `api_key` |
|
||||
| Test URL | `GET {MIDDLEWARE_BASE_URL}/zapier/v1/auth/me` |
|
||||
|
||||
### `beforeRequest: addApiKey`
|
||||
|
||||
| Sets | `Authorization: Bearer ${bundle.authData.api_key}` |
|
||||
|
||||
### Creates
|
||||
|
||||
| Key | Middleware path |
|
||||
|-----|-----------------|
|
||||
| `timestamp_and_wait` | POST `/zapier/v1/timestamp/wait` |
|
||||
| `create_timestamp` | POST `/zapier/v1/timestamp` |
|
||||
| `verify_timestamp` | POST `/zapier/v1/verify` |
|
||||
| `batch_timestamp` | POST `/zapier/v1/timestamp/batch` |
|
||||
|
||||
### Search `job_status`
|
||||
|
||||
| Path | GET `/zapier/v1/status/{jobId}` |
|
||||
|
||||
### Trigger `timestamp_completed`
|
||||
|
||||
| Subscribe | POST `/zapier/v1/webhooks/subscribe` |
|
||||
| Unsubscribe | DELETE `/zapier/v1/webhooks/unsubscribe` |
|
||||
| Perform | return `[bundle.cleanedRequest]` |
|
||||
28
docs/developer/modules/nats.md
Normal file
28
docs/developer/modules/nats.md
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
# Module: `nats`
|
||||
|
||||
**Path:** `verae-zapier-middleware/src/nats/`
|
||||
**Phase:** 7
|
||||
**Debug namespace:** `nats`
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `subjects.js` | Subject, stream, consumer name constants |
|
||||
| `connection.js` | Connect, ensure streams, close |
|
||||
| `publishers.js` | enqueueWatch, publishJobEvent, enqueueWebhook |
|
||||
|
||||
## Function contracts
|
||||
|
||||
Full I/O tables: [function-reference.md](function-reference.md#natssubjectsjs)
|
||||
Payload schemas: [../../architecture/nats-subjects.md](../../architecture/nats-subjects.md)
|
||||
|
||||
## Implementation status
|
||||
|
||||
Stubs throw until **GATE 7**. Call sites must check `config.natsEnabled` and fall back to in-process poller (Phase 6).
|
||||
|
||||
## Security
|
||||
|
||||
- Prefer `tokenRef` over raw Verae JWT in messages.
|
||||
- All publish paths log via `createDebugger('nats')` with redaction.
|
||||
- NATS must remain private (not exposed to Zapier).
|
||||
35
docs/developer/modules/workers.md
Normal file
35
docs/developer/modules/workers.md
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
# Module: `workers`
|
||||
|
||||
**Path:** `verae-zapier-middleware/src/workers/`
|
||||
**Phases:** 6 (in-process), 8–9 (NATS)
|
||||
|
||||
## Files
|
||||
|
||||
| File | Phase | Purpose |
|
||||
|------|-------|---------|
|
||||
| `inProcessJobPoller.js` | 6 | Fallback poller when `NATS_ENABLED=false` |
|
||||
| `jobPollerWorker.js` | 8 | JetStream job watch consumer |
|
||||
| `webhookWorker.js` | 8 | JetStream webhook deliver consumer |
|
||||
|
||||
## `startJobPollerWorker()`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | Poll Verae until job terminal; publish events |
|
||||
| **Input** | none (uses NATS + config) |
|
||||
| **Output** | `Promise<{ stop: () => Promise<void> }>` |
|
||||
| **Debug** | `jobs`, `http`, `nats` |
|
||||
|
||||
## `startWebhookWorker()`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| **For** | POST completion payloads to Zapier hook URLs |
|
||||
| **Output** | `Promise<{ stop: () => Promise<void> }>` |
|
||||
| **Debug** | `webhooks`, `nats` |
|
||||
|
||||
## Rules
|
||||
|
||||
- Only one of in-process poller **or** NATS job worker should run.
|
||||
- Never log raw JWTs or full webhook secrets.
|
||||
- Queue groups allow horizontal scale without double delivery.
|
||||
189
docs/plans/phase-gates.md
Normal file
189
docs/plans/phase-gates.md
Normal file
|
|
@ -0,0 +1,189 @@
|
|||
# Phase Gates — Commands and Acceptance Criteria
|
||||
|
||||
Gates enforce **test-before-next-step**. Run from repository root:
|
||||
|
||||
```bash
|
||||
cd /Users/marchon/apps/zapier
|
||||
npm run gate:N
|
||||
```
|
||||
|
||||
`npm run gate:all` runs gates `0` … `12` sequentially and **exits non-zero** on first failure.
|
||||
|
||||
## Gate 0 — Structure and documentation
|
||||
|
||||
```bash
|
||||
npm run gate:0
|
||||
```
|
||||
|
||||
| Check | Requirement |
|
||||
|-------|-------------|
|
||||
| Layout | `TODO.md`, `README.md`, middleware + zapier dirs, docs tree |
|
||||
| Architecture | `docs/architecture/overview.md`, `nats-subjects.md` |
|
||||
| Developer | `docs/developer/debugging.md`, `docs/developer/modules/*.md` |
|
||||
| Plan | `docs/plans/phase-gates.md` |
|
||||
|
||||
## Gate 1 — Debug facility
|
||||
|
||||
```bash
|
||||
npm run gate:1
|
||||
```
|
||||
|
||||
Tests: `verae-zapier-middleware/test/unit/debug*.test.js`
|
||||
|
||||
| Check | Requirement |
|
||||
|-------|-------------|
|
||||
| Off by default | No output when `DEBUG_VERAE` unset |
|
||||
| Namespace filter | `auth` does not emit `nats` lines |
|
||||
| Redaction | Tokens/passwords not present in formatted output |
|
||||
| Trace | Child spans inherit `traceId` |
|
||||
|
||||
## Gate 2 — HTTP shell
|
||||
|
||||
```bash
|
||||
npm run gate:2
|
||||
```
|
||||
|
||||
| Check | Requirement |
|
||||
|-------|-------------|
|
||||
| Health | `GET /health` → 200 `{ status: "ok" }` |
|
||||
| Errors | Thrown `AppError` → JSON `{ error, code }` |
|
||||
| Config | Reads NATS and debug env keys |
|
||||
|
||||
## Gate 3 — Stores
|
||||
|
||||
```bash
|
||||
npm run gate:3
|
||||
```
|
||||
|
||||
| Check | Requirement |
|
||||
|-------|-------------|
|
||||
| Tenants | API key maps to tenant |
|
||||
| Limits | Plan + enterprise contract |
|
||||
| Webhooks | Tenant isolation |
|
||||
| Persist | Reload preserves state |
|
||||
|
||||
## Gate 4 — Tokens + Verae client
|
||||
|
||||
```bash
|
||||
npm run gate:4
|
||||
```
|
||||
|
||||
| Check | Requirement |
|
||||
|-------|-------------|
|
||||
| HMAC | Bad signature rejected |
|
||||
| Mock job | create → wait → completed |
|
||||
| HTTP debug | No Authorization values logged |
|
||||
|
||||
## Gate 5 — Auth + entitlements
|
||||
|
||||
```bash
|
||||
npm run gate:5
|
||||
```
|
||||
|
||||
| Check | Requirement |
|
||||
|-------|-------------|
|
||||
| Me | Valid API key |
|
||||
| 401 | Invalid key |
|
||||
| 402 | Over quota |
|
||||
|
||||
## Gate 6 — Sync HTTP API (NATS off)
|
||||
|
||||
```bash
|
||||
NATS_ENABLED=false MOCK_VERAE=true npm run gate:6
|
||||
```
|
||||
|
||||
| Check | Requirement |
|
||||
|-------|-------------|
|
||||
| Timestamp | 202 + jobId |
|
||||
| Wait | completed status |
|
||||
| Verify | boolean valid |
|
||||
| Webhooks | subscribe stored |
|
||||
|
||||
## Gate 7 — NATS infrastructure
|
||||
|
||||
```bash
|
||||
npm run gate:7
|
||||
```
|
||||
|
||||
| Check | Requirement |
|
||||
|-------|-------------|
|
||||
| Streams | Created/idempotent ensure |
|
||||
| Pub/sub | One message round-trip |
|
||||
| Flag off | App boots without NATS |
|
||||
|
||||
## Gate 8 — Workers
|
||||
|
||||
```bash
|
||||
NATS_ENABLED=true MOCK_VERAE=true npm run gate:8
|
||||
```
|
||||
|
||||
| Check | Requirement |
|
||||
|-------|-------------|
|
||||
| Watch→event | Terminal event published |
|
||||
| Webhook | Mock receiver got POST |
|
||||
| Queue group | No double complete |
|
||||
|
||||
## Gate 9 — Wait via NATS
|
||||
|
||||
```bash
|
||||
npm run gate:9
|
||||
```
|
||||
|
||||
| Check | Requirement |
|
||||
|-------|-------------|
|
||||
| Success | Wait returns completed |
|
||||
| Timeout | Returns pending + jobId |
|
||||
|
||||
## Gate 10 — Tenancy
|
||||
|
||||
```bash
|
||||
npm run gate:10
|
||||
```
|
||||
|
||||
| Check | Requirement |
|
||||
|-------|-------------|
|
||||
| Signup | free + apiKey |
|
||||
| Enterprise | requires contract |
|
||||
|
||||
## Gate 11 — Zapier package
|
||||
|
||||
```bash
|
||||
npm run gate:11
|
||||
```
|
||||
|
||||
| Check | Requirement |
|
||||
|-------|-------------|
|
||||
| Auth module | fields + test URL |
|
||||
| Creates | correct middleware paths |
|
||||
| Trigger | subscribe shapes |
|
||||
|
||||
## Gate 12 — E2E local compose
|
||||
|
||||
```bash
|
||||
npm run gate:12
|
||||
```
|
||||
|
||||
| Check | Requirement |
|
||||
|-------|-------------|
|
||||
| Stack | health green |
|
||||
| Paths | async + wait + hook |
|
||||
|
||||
## Gate 13 — Production smoke (opt-in)
|
||||
|
||||
```bash
|
||||
PRODUCTION_SMOKE=1 npm run gate:13
|
||||
```
|
||||
|
||||
Requires real secrets; skipped in default `gate:all`.
|
||||
|
||||
## Gate 14 — Human acceptance
|
||||
|
||||
Manual checklist in `TODO.md` (not automated).
|
||||
|
||||
## Gate 15 — Hardening
|
||||
|
||||
Checklist + load notes in `TODO.md`.
|
||||
|
||||
## Recording results
|
||||
|
||||
After each green gate, update the **Gate log** table in `TODO.md` with date and `pass`.
|
||||
Loading…
Add table
Add a link
Reference in a new issue