commit bcda73e3fe384c99a99088490ea98282d4c2ad78 Author: George Lambert Date: Fri Sep 11 12:33:29 2026 -0400 Initial import of zapier-docs-master from zapier monorepo diff --git a/MESSAGE-FLOWS.md b/MESSAGE-FLOWS.md new file mode 100644 index 0000000..66fe0af --- /dev/null +++ b/MESSAGE-FLOWS.md @@ -0,0 +1,29 @@ +# Message flows + +## 1. Create timestamp (async) + +1. Zapier → `POST https://zappier-edge/v1/timestamp` (`x-api-key`) +2. zappier-edge meters → `POST middleware/zapier/v1/timestamp` +3. splitter: chain sha256; optional `verae.archive.put` +4. middleware → chain create or existing lookup +5. middleware publishes `verae.zapier.jobs.watch` `{ jobId, tenantId, … }` +6. HTTP 202 `{ jobId }` back to Zapier +7. job-poller consumes watch, GET chain status +8. On terminal: `verae.zapier.jobs.events` +9. webhook-deliver POSTs Zapier REST Hook if subscribed + +## 2. Wait + +Same as (1) but HTTP holds until `jobs.events` or `WAIT_TIMEOUT_MS` → `{ status: pending, jobId }`. + +## 3. Hash already registered + +Step 4 returns original jobId + original seal. No second chain write. `receipts[0].kind = seal`. Later archive attaches become `receipts[1…]` if `includeAttached`. + +## 4. includeAttached + +After seal is known, aggregator publishes `verae.archive.query`. Each WORM: bloom miss = no packet; hit = `verae.archive.reply.`. Aggregator merges into wait JSON. + +## 5. Multipart attachments + +Splitter emits one `archive.put` per file (`kind: file`, `contentSha256`). Chain never stores bytes. diff --git a/README.md b/README.md new file mode 100644 index 0000000..0e7a029 --- /dev/null +++ b/README.md @@ -0,0 +1,21 @@ +# zapier-docs-master + +Summaries and NATS interaction map for every Verae Time × Zapier module. + +**Live HTML:** https://zapier.georgelambert.org/ + +**Monorepo:** https://git.georgelambert.org/marchon/master-zapier-plan-draft + +## Modules + +| Module | Summary | NATS in | NATS out | +|--------|---------|---------|----------| +| zappier-edge | Metered HTTPS for Zapier | — | — (HTTPS to middleware) | +| verae-middleware | Job id + wait HTTP | jobs.events | jobs.watch, archive.query via aggregator | +| verae-activate | Pushable Zapier app | — | — | +| verae-zapier-app | Full Zapier nouns | — | — | +| request-splitter | Hash vs files | — | archive.put | +| archive-worm | Bloom WORM node | archive.query, archive.put | archive.reply.* (hit only) | +| archive-aggregator | Merge archive replies | archive.reply.* | archive.query | + +Full contracts: `../docs/02-architecture/modules-and-nats.md` and `archive-nats.md` in the monorepo / on the website. diff --git a/archive-nats.md b/archive-nats.md new file mode 100644 index 0000000..a50e0b2 --- /dev/null +++ b/archive-nats.md @@ -0,0 +1,90 @@ +# Archive NATS: jobs, multipart split, hash receipts, WORM bloom fan-out + +Zapier never speaks NATS. HTTPS stops at zappier → middleware. Middleware owns jobs, splitting, chain lookup, and archive aggregation. + +## End-to-end + +```text +Zapier --HTTPS--> zappier (meter, x-api-key) + --HTTPS--> middleware /zapier/v1/timestamp[/wait] + 1. split multipart + chain: SHA256 only (+ optional public-meta digest) + archives: public JSON, encrypted private JSON, files + 2. publish jobs.watch → return jobId (202) + 3. waiters subscribe jobs.events + 4. lookup SHA256 on chain (mock or Verae) + already sealed → original receipt, no new seal + later attach records → extra receipts + 5. if includeAttached: + publish verae.archive.query + WORM nodes: bloom miss = silence + bloom hit = reply + aggregate until WAIT_ARCHIVE_MS + 6. jobs.events completed JSON → waiter / REST Hook +``` + +Blockchain stores **hash + time + block + certificate**. Public metadata, encrypted metadata, and file bytes live on **WORM archives**. + +## Splitter + +`splitRequest(body | multipart)`: + +| Field | Destination | +|-------|-------------| +| `data` / `sha256` | Chain register or lookup | +| `publicMetadata` | Archive put (clear) | +| `privateMetadata` | Archive put (ciphertext) | +| `files[]` | Archive put; chain gets content hashes + ids | +| `includeAttached` | Whether wait path queries archives | + +## Hash already registered + +Return original `jobId` and original seal. Do not write a second chain timestamp. + +If later attach jobs exist for that hash, `receipts` is an array: seal first, then attachment receipts in time order. + +## Subjects + +| Subject | Publisher | Subscriber | +|---------|-----------|------------| +| `verae.zapier.jobs.watch` | HTTP edge | job poller | +| `verae.zapier.jobs.events` | poller | waiter, webhook router | +| `verae.archive.put` | splitter | archive that owns the shard | +| `verae.archive.query` | aggregator | **every** archive (not a shared queue group) | +| `verae.archive.reply.` | archive on bloom hit | aggregator | + +Query payload: `{ correlationId, sha256, tenantId, kinds[] }`. +Reply payload: `{ archiveId, sha256, records[] }`. + +Bloom miss → no reply. Aggregator timeout → complete with whatever arrived. + +## WORM archives + +Each process holds append-only records + a bloom of SHA256 keys it stores. False positives OK; false negatives must be rare. Bloom is not an ACL — on hit, still check tenant/share. + +Harness: three mock archives with overlapping hashes. + +## Completed job JSON (wait / hook) + +```json +{ + "jobId": "…", + "status": "completed", + "sha256": "…", + "receipts": [ + { "kind": "seal", "timestamp": "…", "certificate": "…", "blockIndex": 42 }, + { "kind": "metadata-attach", "attachedAt": "…", "publicMetadata": {} } + ], + "files": [{ "id": "…", "sha256": "…", "archiveId": "archive-b" }], + "archivesQueried": true, + "archiveReplies": 2 +} +``` + +Flag off or all blooms miss → `files` empty, extra receipts omitted. + +## Security + +- Zapier never connects to NATS or archives. +- Private metadata only on authenticated archive replies. +- NS1 NATS stays loopback; use `scripts/nats-tunnel.sh`. diff --git a/composition.md b/composition.md new file mode 100644 index 0000000..e2568e5 --- /dev/null +++ b/composition.md @@ -0,0 +1,33 @@ +# Composition: zappier commercial edge + Verae adapter + +```text +Users → Zapier UI +Zapier cloud runs packages/verae-zapier + --HTTPS, x-api-key--> packages/zappier /v1/* + (meter, quote, 401/403, usage) + --internal HTTPS--> packages/verae-zapier-middleware /zapier/v1/* + --sync--> api.veraetime.net or MOCK_VERAE + --NATS--> workers --> Verae + Zapier REST Hooks + +Humans → zappier /portal signup, API key, usage, reloads, invoices +Ops → zappier /admin rate card, tiers, customers, PO invoices +``` + +## Ownership + +| Concern | Package | +|---------|---------| +| API keys, custom pricing, Stripe, invoices, portal | `packages/zappier` | +| Timestamp/verify/status, NATS, REST Hooks, mock Verae | `packages/verae-zapier-middleware` | +| Zapier Platform nouns (creates/searches/triggers) | `packages/verae-zapier` | +| Official SDK reference | `vendor/zapier-platform` | + +Zapier never talks to NATS or `api.veraetime.net`. + +Do not reimplement Stripe, invoices, or the rate-card UI inside the Verae middleware. + +Public keys are zappier `x-api-key` (issued at portal signup). Middleware `zmw_` / `PLAN_LIMITS` stay internal until composition PR 4 removes them from the public path. + +Demo zappier routes `/v1/transform` and `/v1/storage` are a metering sandbox, not Verae timestamping or encrypted LTS. + +Payment invoices (zappier) are not certified timestamp receipts (Verae feature i). diff --git a/modules-and-nats.md b/modules-and-nats.md new file mode 100644 index 0000000..cfc3bac --- /dev/null +++ b/modules-and-nats.md @@ -0,0 +1,52 @@ +# Modules and NATS message contracts + +Master map of every service in the Zapier ↔ middleware ↔ chain ↔ WORM-archive path. + +Zapier cloud **never** connects to NATS. Only middleware workers and archives do. + +## Who talks to whom (HTTPS vs NATS) + +```text +[Zapier Platform app] HTTPS [zappier-edge] HTTPS [middleware-http] + | + NATS JetStream (NS1 loopback :4222) | + +------------------+------------------+-------------+ + | | | + [job-poller] [webhook-deliver] [archive-aggregator] + | | + | HTTPS | NATS query/reply + [verae-chain-client] | + [archive-worm × N] + bloom miss = silence +``` + +## Module catalog + +| Module | Repo (Forgejo) | Runtime | Listens | Sends | +|--------|----------------|---------|---------|-------| +| zapier-platform-app | `verae-zapier-app` | Zapier cloud | User Zap steps | HTTPS to zappier-edge | +| zappier-edge | `zappier-edge` | Public HTTPS :3000 | Zapier, portal, admin | HTTPS to middleware | +| middleware-http | `verae-middleware` | Public HTTPS :3100 | zappier-edge | NATS jobs.watch; HTTP to chain; wait on jobs.events | +| job-poller | `verae-job-poller` | Worker | `verae.zapier.jobs.watch` | `verae.zapier.jobs.events`; HTTP GET chain status | +| webhook-deliver | `verae-webhook-deliver` | Worker | `verae.zapier.webhooks.deliver` + events | HTTPS POST Zapier REST Hook | +| request-splitter | `verae-request-splitter` | In middleware | HTTP body / multipart | chain hash; `verae.archive.put` | +| archive-aggregator | `verae-archive-aggregator` | Worker / in wait | `includeAttached` on wait | `verae.archive.query`; reads `verae.archive.reply.` | +| archive-worm | `verae-archive-worm` | N copies | `verae.archive.query`, `verae.archive.put` | `verae.archive.reply.` if bloom hits | +| verae-chain-client | `verae-chain-client` | Library | — | HTTPS `api.veraetime.net` or MOCK | +| docs-master | `zapier-docs-master` | Static | — | published on zapier.georgelambert.org | + +## Addresses (subjects) + +| Address | Kind | Payload (required fields) | +|---------|------|---------------------------| +| `verae.zapier.jobs.watch` | JetStream work queue | `tenantId`, `jobId`, `tokenRef`, `enqueuedAt`, `attempt`, `maxAttempts`, `intervalMs`, `traceId` | +| `verae.zapier.jobs.events` | JetStream events | `event` (`timestamp.completed\|failed\|timeout`), `tenantId`, `jobId`, `status`, `traceId`, `emittedAt` | +| `verae.zapier.webhooks.deliver` | JetStream work queue | `hookId`, `tenantId`, `targetUrl`, `event`, `payload`, `attempt`, `traceId` | +| `verae.zapier.usage` | optional | `tenantId`, `action`, `amount`, `at` | +| `verae.archive.put` | JetStream | `sha256`, `tenantId`, `kind` (`publicMeta\|privateMeta\|file\|tree`), `record`, `traceId` | +| `verae.archive.query` | pub to **all** archives | `correlationId`, `sha256`, `tenantId`, `kinds[]`, `traceId` | +| `verae.archive.reply.` | replies | `archiveId`, `sha256`, `records[]`, `traceId` | + +## Completed job JSON (back to Zapier) + +See `docs/02-architecture/archive-nats.md`. Seal receipts from chain; extra receipts and `files[]` from archive aggregation when `includeAttached` is true. diff --git a/modules/verae-activate/SUMMARY.md b/modules/verae-activate/SUMMARY.md new file mode 100644 index 0000000..975de3d --- /dev/null +++ b/modules/verae-activate/SUMMARY.md @@ -0,0 +1,11 @@ +# verae-activate (Zapier Platform app) + +**Job:** Private Zapier integration you push first. Actions run in Zapier cloud. + +**Expects:** User connection (API key optional). + +**Sends:** nothing if fields blank (Add Numbers / Echo / SHA256 / Create Timestamp local mock). If `api_base` + `api_key`, HTTPS to zappier-edge. + +**Does not** speak NATS. + +**Test:** `npm test`; `zapier-platform validate`. diff --git a/modules/verae-archive-aggregator/NATS.md b/modules/verae-archive-aggregator/NATS.md new file mode 100644 index 0000000..6a73642 --- /dev/null +++ b/modules/verae-archive-aggregator/NATS.md @@ -0,0 +1,8 @@ +# NATS — verae-archive-aggregator + +| Direction | Address | Peer | Body | +|-----------|---------|------|------| +| OUT | `verae.archive.query` | all WORM | `{ correlationId, sha256, tenantId, kinds[], traceId }` | +| IN | `verae.archive.reply.` | WORM hits | `{ archiveId, sha256, records[] }` | + +Stop collecting at `WAIT_ARCHIVE_MS`. Silent archives are listed in `silent[]`, not errors. diff --git a/modules/verae-archive-aggregator/SUMMARY.md b/modules/verae-archive-aggregator/SUMMARY.md new file mode 100644 index 0000000..d6951ab --- /dev/null +++ b/modules/verae-archive-aggregator/SUMMARY.md @@ -0,0 +1,11 @@ +# verae-archive-aggregator + +**Job:** If `includeAttached`, fan out archive query and merge replies into the job JSON. + +**Expects:** wait-path flag from middleware-http; replies on `verae.archive.reply.`. + +**Sends:** `verae.archive.query` to all WORM nodes. + +**Timeout:** missing archives omit slices; chain receipt still succeeds. + +**Test:** `npm test` — one silent bloom miss + one hit; merge seal + metadata-attach. diff --git a/modules/verae-archive-worm/NATS.md b/modules/verae-archive-worm/NATS.md new file mode 100644 index 0000000..a64f14f --- /dev/null +++ b/modules/verae-archive-worm/NATS.md @@ -0,0 +1,9 @@ +# NATS — verae-archive-worm + +| Direction | Address | From / to | Body | +|-----------|---------|-----------|------| +| IN | `verae.archive.put` | splitter | `{ sha256, tenantId, kind, record, traceId }` | +| IN | `verae.archive.query` | aggregator (broadcast) | `{ correlationId, sha256, tenantId, kinds[], traceId }` | +| OUT | `verae.archive.reply.` | aggregator | `{ archiveId, sha256, records[], traceId }` **only if bloom.mightHave(sha256)** | + +Subscribe to query **without** a shared queue group so every archive sees every query. diff --git a/modules/verae-archive-worm/SUMMARY.md b/modules/verae-archive-worm/SUMMARY.md new file mode 100644 index 0000000..fe2ac10 --- /dev/null +++ b/modules/verae-archive-worm/SUMMARY.md @@ -0,0 +1,14 @@ +# verae-archive-worm + +**Job:** Append-only WORM store of metadata and files. Bloom filter: miss → stay silent on NATS. + +**Expects:** +- `verae.archive.put` — append record, add sha256 to bloom +- `verae.archive.query` — `{ correlationId, sha256, tenantId, kinds[] }` (every node sees this; **not** a shared queue group) + +**Sends:** +- `verae.archive.reply.` **only on bloom hit** — `{ archiveId, sha256, records[] }` + +**Does not** write to the blockchain. + +**Test:** `npm test` — miss is `null`; put then query returns records. diff --git a/modules/verae-request-splitter/NATS.md b/modules/verae-request-splitter/NATS.md new file mode 100644 index 0000000..0e12c1b --- /dev/null +++ b/modules/verae-request-splitter/NATS.md @@ -0,0 +1,8 @@ +# NATS — verae-request-splitter + +| Direction | Address | To | Body | +|-----------|---------|-----|------| +| OUT | `verae.archive.put` | each WORM | one message per publicMeta / privateMeta / file | +| OUT | (in-process) chain | middleware | `{ sha256, hashAlg, data? }` | + +No inbound NATS in the in-process embedding. diff --git a/modules/verae-request-splitter/SUMMARY.md b/modules/verae-request-splitter/SUMMARY.md new file mode 100644 index 0000000..45c898f --- /dev/null +++ b/modules/verae-request-splitter/SUMMARY.md @@ -0,0 +1,13 @@ +# verae-request-splitter + +**Job:** Peel a request into (1) chain hash and (2) off-chain archive puts. + +**Expects messages from:** middleware-http (in-process call today; later NATS `verae.splitter.in` if split out). + +**Sends messages to:** +- Chain path: `{ sha256, hashAlg, data? }` +- `verae.archive.put` for each `publicMeta` / `privateMeta` / `file` + +**Does not talk to Zapier or NATS job poller.** + +**Test:** `npm test` — hash-only has zero puts; two files + public meta → 3 puts. diff --git a/modules/verae-zapier-middleware/NATS.md b/modules/verae-zapier-middleware/NATS.md new file mode 100644 index 0000000..abe42c8 --- /dev/null +++ b/modules/verae-zapier-middleware/NATS.md @@ -0,0 +1,9 @@ +# NATS — verae-middleware + +| Direction | Address | Peer | Body | +|-----------|---------|------|------| +| OUT | `verae.zapier.jobs.watch` | job-poller | job watch payload | +| IN | `verae.zapier.jobs.events` | waiter | terminal status | +| OUT | (via aggregator) `verae.archive.query` | WORM | attached lookup | + +HTTP remains the only Zapier-facing surface. diff --git a/modules/verae-zapier-middleware/SUMMARY.md b/modules/verae-zapier-middleware/SUMMARY.md new file mode 100644 index 0000000..7b275df --- /dev/null +++ b/modules/verae-zapier-middleware/SUMMARY.md @@ -0,0 +1,15 @@ +# verae-middleware (HTTP edge) + +**Job:** Public `/zapier/v1/*`. Auth, entitlements, timestamp/wait, hash lookup, objects, receipts. Publishes NATS jobs; waits on events. + +**Expects HTTPS from:** zappier-edge (or Zapier if pointed here). + +**Sends:** +- HTTPS to verae-chain-client (`/api/timestamp`, `/api/status`, `/api/verify`) +- `verae.zapier.jobs.watch` after create +- waits `verae.zapier.jobs.events` +- splitter + aggregator when attachments / `includeAttached` + +**Returns to requestor:** 202 `{ jobId }` or wait JSON (completed / pending+jobId). + +**NATS:** never exposed to Zapier. diff --git a/modules/verae-zapier/SUMMARY.md b/modules/verae-zapier/SUMMARY.md new file mode 100644 index 0000000..94b22b3 --- /dev/null +++ b/modules/verae-zapier/SUMMARY.md @@ -0,0 +1,9 @@ +# verae-zapier-app + +**Job:** Full Zapier CLI app: timestamp, wait, verify, batch, job status, hash lookup, REST Hook. + +**Expects:** `zmw_` middleware key (today) or zappier key after composition. + +**Sends:** HTTPS to middleware `/zapier/v1/*`. + +**Does not** speak NATS. diff --git a/modules/zappier/SUMMARY.md b/modules/zappier/SUMMARY.md new file mode 100644 index 0000000..fdf0ba3 --- /dev/null +++ b/modules/zappier/SUMMARY.md @@ -0,0 +1,11 @@ +# zappier-edge + +**Job:** Metered commercial API, portal, admin, Stripe. Zapier’s only public HTTPS hop. + +**Expects:** Zapier Platform app with `x-api-key`. + +**Sends:** HTTPS to middleware `/zapier/v1/*` (or local mock `/v1/timestamp`, `/v1/add`, `/v1/hashes`, `/v1/receipts`). + +**Does not** speak NATS or `api.veraetime.net`. + +**Test:** `npm test` (Jest, 177+).