diff --git a/docs/02-architecture/modules-and-nats.md b/docs/02-architecture/modules-and-nats.md new file mode 100644 index 0000000..cfc3bac --- /dev/null +++ b/docs/02-architecture/modules-and-nats.md @@ -0,0 +1,52 @@ +# Modules and NATS message contracts + +Master map of every service in the Zapier ↔ middleware ↔ chain ↔ WORM-archive path. + +Zapier cloud **never** connects to NATS. Only middleware workers and archives do. + +## Who talks to whom (HTTPS vs NATS) + +```text +[Zapier Platform app] HTTPS [zappier-edge] HTTPS [middleware-http] + | + NATS JetStream (NS1 loopback :4222) | + +------------------+------------------+-------------+ + | | | + [job-poller] [webhook-deliver] [archive-aggregator] + | | + | HTTPS | NATS query/reply + [verae-chain-client] | + [archive-worm × N] + bloom miss = silence +``` + +## Module catalog + +| Module | Repo (Forgejo) | Runtime | Listens | Sends | +|--------|----------------|---------|---------|-------| +| zapier-platform-app | `verae-zapier-app` | Zapier cloud | User Zap steps | HTTPS to zappier-edge | +| zappier-edge | `zappier-edge` | Public HTTPS :3000 | Zapier, portal, admin | HTTPS to middleware | +| middleware-http | `verae-middleware` | Public HTTPS :3100 | zappier-edge | NATS jobs.watch; HTTP to chain; wait on jobs.events | +| job-poller | `verae-job-poller` | Worker | `verae.zapier.jobs.watch` | `verae.zapier.jobs.events`; HTTP GET chain status | +| webhook-deliver | `verae-webhook-deliver` | Worker | `verae.zapier.webhooks.deliver` + events | HTTPS POST Zapier REST Hook | +| request-splitter | `verae-request-splitter` | In middleware | HTTP body / multipart | chain hash; `verae.archive.put` | +| archive-aggregator | `verae-archive-aggregator` | Worker / in wait | `includeAttached` on wait | `verae.archive.query`; reads `verae.archive.reply.` | +| archive-worm | `verae-archive-worm` | N copies | `verae.archive.query`, `verae.archive.put` | `verae.archive.reply.` if bloom hits | +| verae-chain-client | `verae-chain-client` | Library | — | HTTPS `api.veraetime.net` or MOCK | +| docs-master | `zapier-docs-master` | Static | — | published on zapier.georgelambert.org | + +## Addresses (subjects) + +| Address | Kind | Payload (required fields) | +|---------|------|---------------------------| +| `verae.zapier.jobs.watch` | JetStream work queue | `tenantId`, `jobId`, `tokenRef`, `enqueuedAt`, `attempt`, `maxAttempts`, `intervalMs`, `traceId` | +| `verae.zapier.jobs.events` | JetStream events | `event` (`timestamp.completed\|failed\|timeout`), `tenantId`, `jobId`, `status`, `traceId`, `emittedAt` | +| `verae.zapier.webhooks.deliver` | JetStream work queue | `hookId`, `tenantId`, `targetUrl`, `event`, `payload`, `attempt`, `traceId` | +| `verae.zapier.usage` | optional | `tenantId`, `action`, `amount`, `at` | +| `verae.archive.put` | JetStream | `sha256`, `tenantId`, `kind` (`publicMeta\|privateMeta\|file\|tree`), `record`, `traceId` | +| `verae.archive.query` | pub to **all** archives | `correlationId`, `sha256`, `tenantId`, `kinds[]`, `traceId` | +| `verae.archive.reply.` | replies | `archiveId`, `sha256`, `records[]`, `traceId` | + +## Completed job JSON (back to Zapier) + +See `docs/02-architecture/archive-nats.md`. Seal receipts from chain; extra receipts and `files[]` from archive aggregation when `includeAttached` is true. diff --git a/packages/docs-master/MESSAGE-FLOWS.md b/packages/docs-master/MESSAGE-FLOWS.md new file mode 100644 index 0000000..66fe0af --- /dev/null +++ b/packages/docs-master/MESSAGE-FLOWS.md @@ -0,0 +1,29 @@ +# Message flows + +## 1. Create timestamp (async) + +1. Zapier → `POST https://zappier-edge/v1/timestamp` (`x-api-key`) +2. zappier-edge meters → `POST middleware/zapier/v1/timestamp` +3. splitter: chain sha256; optional `verae.archive.put` +4. middleware → chain create or existing lookup +5. middleware publishes `verae.zapier.jobs.watch` `{ jobId, tenantId, … }` +6. HTTP 202 `{ jobId }` back to Zapier +7. job-poller consumes watch, GET chain status +8. On terminal: `verae.zapier.jobs.events` +9. webhook-deliver POSTs Zapier REST Hook if subscribed + +## 2. Wait + +Same as (1) but HTTP holds until `jobs.events` or `WAIT_TIMEOUT_MS` → `{ status: pending, jobId }`. + +## 3. Hash already registered + +Step 4 returns original jobId + original seal. No second chain write. `receipts[0].kind = seal`. Later archive attaches become `receipts[1…]` if `includeAttached`. + +## 4. includeAttached + +After seal is known, aggregator publishes `verae.archive.query`. Each WORM: bloom miss = no packet; hit = `verae.archive.reply.`. Aggregator merges into wait JSON. + +## 5. Multipart attachments + +Splitter emits one `archive.put` per file (`kind: file`, `contentSha256`). Chain never stores bytes. diff --git a/packages/docs-master/README.md b/packages/docs-master/README.md new file mode 100644 index 0000000..0e7a029 --- /dev/null +++ b/packages/docs-master/README.md @@ -0,0 +1,21 @@ +# zapier-docs-master + +Summaries and NATS interaction map for every Verae Time × Zapier module. + +**Live HTML:** https://zapier.georgelambert.org/ + +**Monorepo:** https://git.georgelambert.org/marchon/master-zapier-plan-draft + +## Modules + +| Module | Summary | NATS in | NATS out | +|--------|---------|---------|----------| +| zappier-edge | Metered HTTPS for Zapier | — | — (HTTPS to middleware) | +| verae-middleware | Job id + wait HTTP | jobs.events | jobs.watch, archive.query via aggregator | +| verae-activate | Pushable Zapier app | — | — | +| verae-zapier-app | Full Zapier nouns | — | — | +| request-splitter | Hash vs files | — | archive.put | +| archive-worm | Bloom WORM node | archive.query, archive.put | archive.reply.* (hit only) | +| archive-aggregator | Merge archive replies | archive.reply.* | archive.query | + +Full contracts: `../docs/02-architecture/modules-and-nats.md` and `archive-nats.md` in the monorepo / on the website. diff --git a/packages/docs-master/archive-nats.md b/packages/docs-master/archive-nats.md new file mode 100644 index 0000000..a50e0b2 --- /dev/null +++ b/packages/docs-master/archive-nats.md @@ -0,0 +1,90 @@ +# Archive NATS: jobs, multipart split, hash receipts, WORM bloom fan-out + +Zapier never speaks NATS. HTTPS stops at zappier → middleware. Middleware owns jobs, splitting, chain lookup, and archive aggregation. + +## End-to-end + +```text +Zapier --HTTPS--> zappier (meter, x-api-key) + --HTTPS--> middleware /zapier/v1/timestamp[/wait] + 1. split multipart + chain: SHA256 only (+ optional public-meta digest) + archives: public JSON, encrypted private JSON, files + 2. publish jobs.watch → return jobId (202) + 3. waiters subscribe jobs.events + 4. lookup SHA256 on chain (mock or Verae) + already sealed → original receipt, no new seal + later attach records → extra receipts + 5. if includeAttached: + publish verae.archive.query + WORM nodes: bloom miss = silence + bloom hit = reply + aggregate until WAIT_ARCHIVE_MS + 6. jobs.events completed JSON → waiter / REST Hook +``` + +Blockchain stores **hash + time + block + certificate**. Public metadata, encrypted metadata, and file bytes live on **WORM archives**. + +## Splitter + +`splitRequest(body | multipart)`: + +| Field | Destination | +|-------|-------------| +| `data` / `sha256` | Chain register or lookup | +| `publicMetadata` | Archive put (clear) | +| `privateMetadata` | Archive put (ciphertext) | +| `files[]` | Archive put; chain gets content hashes + ids | +| `includeAttached` | Whether wait path queries archives | + +## Hash already registered + +Return original `jobId` and original seal. Do not write a second chain timestamp. + +If later attach jobs exist for that hash, `receipts` is an array: seal first, then attachment receipts in time order. + +## Subjects + +| Subject | Publisher | Subscriber | +|---------|-----------|------------| +| `verae.zapier.jobs.watch` | HTTP edge | job poller | +| `verae.zapier.jobs.events` | poller | waiter, webhook router | +| `verae.archive.put` | splitter | archive that owns the shard | +| `verae.archive.query` | aggregator | **every** archive (not a shared queue group) | +| `verae.archive.reply.` | archive on bloom hit | aggregator | + +Query payload: `{ correlationId, sha256, tenantId, kinds[] }`. +Reply payload: `{ archiveId, sha256, records[] }`. + +Bloom miss → no reply. Aggregator timeout → complete with whatever arrived. + +## WORM archives + +Each process holds append-only records + a bloom of SHA256 keys it stores. False positives OK; false negatives must be rare. Bloom is not an ACL — on hit, still check tenant/share. + +Harness: three mock archives with overlapping hashes. + +## Completed job JSON (wait / hook) + +```json +{ + "jobId": "…", + "status": "completed", + "sha256": "…", + "receipts": [ + { "kind": "seal", "timestamp": "…", "certificate": "…", "blockIndex": 42 }, + { "kind": "metadata-attach", "attachedAt": "…", "publicMetadata": {} } + ], + "files": [{ "id": "…", "sha256": "…", "archiveId": "archive-b" }], + "archivesQueried": true, + "archiveReplies": 2 +} +``` + +Flag off or all blooms miss → `files` empty, extra receipts omitted. + +## Security + +- Zapier never connects to NATS or archives. +- Private metadata only on authenticated archive replies. +- NS1 NATS stays loopback; use `scripts/nats-tunnel.sh`. diff --git a/packages/docs-master/composition.md b/packages/docs-master/composition.md new file mode 100644 index 0000000..e2568e5 --- /dev/null +++ b/packages/docs-master/composition.md @@ -0,0 +1,33 @@ +# Composition: zappier commercial edge + Verae adapter + +```text +Users → Zapier UI +Zapier cloud runs packages/verae-zapier + --HTTPS, x-api-key--> packages/zappier /v1/* + (meter, quote, 401/403, usage) + --internal HTTPS--> packages/verae-zapier-middleware /zapier/v1/* + --sync--> api.veraetime.net or MOCK_VERAE + --NATS--> workers --> Verae + Zapier REST Hooks + +Humans → zappier /portal signup, API key, usage, reloads, invoices +Ops → zappier /admin rate card, tiers, customers, PO invoices +``` + +## Ownership + +| Concern | Package | +|---------|---------| +| API keys, custom pricing, Stripe, invoices, portal | `packages/zappier` | +| Timestamp/verify/status, NATS, REST Hooks, mock Verae | `packages/verae-zapier-middleware` | +| Zapier Platform nouns (creates/searches/triggers) | `packages/verae-zapier` | +| Official SDK reference | `vendor/zapier-platform` | + +Zapier never talks to NATS or `api.veraetime.net`. + +Do not reimplement Stripe, invoices, or the rate-card UI inside the Verae middleware. + +Public keys are zappier `x-api-key` (issued at portal signup). Middleware `zmw_` / `PLAN_LIMITS` stay internal until composition PR 4 removes them from the public path. + +Demo zappier routes `/v1/transform` and `/v1/storage` are a metering sandbox, not Verae timestamping or encrypted LTS. + +Payment invoices (zappier) are not certified timestamp receipts (Verae feature i). diff --git a/packages/docs-master/modules-and-nats.md b/packages/docs-master/modules-and-nats.md new file mode 100644 index 0000000..cfc3bac --- /dev/null +++ b/packages/docs-master/modules-and-nats.md @@ -0,0 +1,52 @@ +# Modules and NATS message contracts + +Master map of every service in the Zapier ↔ middleware ↔ chain ↔ WORM-archive path. + +Zapier cloud **never** connects to NATS. Only middleware workers and archives do. + +## Who talks to whom (HTTPS vs NATS) + +```text +[Zapier Platform app] HTTPS [zappier-edge] HTTPS [middleware-http] + | + NATS JetStream (NS1 loopback :4222) | + +------------------+------------------+-------------+ + | | | + [job-poller] [webhook-deliver] [archive-aggregator] + | | + | HTTPS | NATS query/reply + [verae-chain-client] | + [archive-worm × N] + bloom miss = silence +``` + +## Module catalog + +| Module | Repo (Forgejo) | Runtime | Listens | Sends | +|--------|----------------|---------|---------|-------| +| zapier-platform-app | `verae-zapier-app` | Zapier cloud | User Zap steps | HTTPS to zappier-edge | +| zappier-edge | `zappier-edge` | Public HTTPS :3000 | Zapier, portal, admin | HTTPS to middleware | +| middleware-http | `verae-middleware` | Public HTTPS :3100 | zappier-edge | NATS jobs.watch; HTTP to chain; wait on jobs.events | +| job-poller | `verae-job-poller` | Worker | `verae.zapier.jobs.watch` | `verae.zapier.jobs.events`; HTTP GET chain status | +| webhook-deliver | `verae-webhook-deliver` | Worker | `verae.zapier.webhooks.deliver` + events | HTTPS POST Zapier REST Hook | +| request-splitter | `verae-request-splitter` | In middleware | HTTP body / multipart | chain hash; `verae.archive.put` | +| archive-aggregator | `verae-archive-aggregator` | Worker / in wait | `includeAttached` on wait | `verae.archive.query`; reads `verae.archive.reply.` | +| archive-worm | `verae-archive-worm` | N copies | `verae.archive.query`, `verae.archive.put` | `verae.archive.reply.` if bloom hits | +| verae-chain-client | `verae-chain-client` | Library | — | HTTPS `api.veraetime.net` or MOCK | +| docs-master | `zapier-docs-master` | Static | — | published on zapier.georgelambert.org | + +## Addresses (subjects) + +| Address | Kind | Payload (required fields) | +|---------|------|---------------------------| +| `verae.zapier.jobs.watch` | JetStream work queue | `tenantId`, `jobId`, `tokenRef`, `enqueuedAt`, `attempt`, `maxAttempts`, `intervalMs`, `traceId` | +| `verae.zapier.jobs.events` | JetStream events | `event` (`timestamp.completed\|failed\|timeout`), `tenantId`, `jobId`, `status`, `traceId`, `emittedAt` | +| `verae.zapier.webhooks.deliver` | JetStream work queue | `hookId`, `tenantId`, `targetUrl`, `event`, `payload`, `attempt`, `traceId` | +| `verae.zapier.usage` | optional | `tenantId`, `action`, `amount`, `at` | +| `verae.archive.put` | JetStream | `sha256`, `tenantId`, `kind` (`publicMeta\|privateMeta\|file\|tree`), `record`, `traceId` | +| `verae.archive.query` | pub to **all** archives | `correlationId`, `sha256`, `tenantId`, `kinds[]`, `traceId` | +| `verae.archive.reply.` | replies | `archiveId`, `sha256`, `records[]`, `traceId` | + +## Completed job JSON (back to Zapier) + +See `docs/02-architecture/archive-nats.md`. Seal receipts from chain; extra receipts and `files[]` from archive aggregation when `includeAttached` is true. diff --git a/packages/docs-master/modules/verae-activate/SUMMARY.md b/packages/docs-master/modules/verae-activate/SUMMARY.md new file mode 100644 index 0000000..975de3d --- /dev/null +++ b/packages/docs-master/modules/verae-activate/SUMMARY.md @@ -0,0 +1,11 @@ +# verae-activate (Zapier Platform app) + +**Job:** Private Zapier integration you push first. Actions run in Zapier cloud. + +**Expects:** User connection (API key optional). + +**Sends:** nothing if fields blank (Add Numbers / Echo / SHA256 / Create Timestamp local mock). If `api_base` + `api_key`, HTTPS to zappier-edge. + +**Does not** speak NATS. + +**Test:** `npm test`; `zapier-platform validate`. diff --git a/packages/docs-master/modules/verae-archive-aggregator/NATS.md b/packages/docs-master/modules/verae-archive-aggregator/NATS.md new file mode 100644 index 0000000..6a73642 --- /dev/null +++ b/packages/docs-master/modules/verae-archive-aggregator/NATS.md @@ -0,0 +1,8 @@ +# NATS — verae-archive-aggregator + +| Direction | Address | Peer | Body | +|-----------|---------|------|------| +| OUT | `verae.archive.query` | all WORM | `{ correlationId, sha256, tenantId, kinds[], traceId }` | +| IN | `verae.archive.reply.` | WORM hits | `{ archiveId, sha256, records[] }` | + +Stop collecting at `WAIT_ARCHIVE_MS`. Silent archives are listed in `silent[]`, not errors. diff --git a/packages/docs-master/modules/verae-archive-aggregator/SUMMARY.md b/packages/docs-master/modules/verae-archive-aggregator/SUMMARY.md new file mode 100644 index 0000000..d6951ab --- /dev/null +++ b/packages/docs-master/modules/verae-archive-aggregator/SUMMARY.md @@ -0,0 +1,11 @@ +# verae-archive-aggregator + +**Job:** If `includeAttached`, fan out archive query and merge replies into the job JSON. + +**Expects:** wait-path flag from middleware-http; replies on `verae.archive.reply.`. + +**Sends:** `verae.archive.query` to all WORM nodes. + +**Timeout:** missing archives omit slices; chain receipt still succeeds. + +**Test:** `npm test` — one silent bloom miss + one hit; merge seal + metadata-attach. diff --git a/packages/docs-master/modules/verae-archive-worm/NATS.md b/packages/docs-master/modules/verae-archive-worm/NATS.md new file mode 100644 index 0000000..a64f14f --- /dev/null +++ b/packages/docs-master/modules/verae-archive-worm/NATS.md @@ -0,0 +1,9 @@ +# NATS — verae-archive-worm + +| Direction | Address | From / to | Body | +|-----------|---------|-----------|------| +| IN | `verae.archive.put` | splitter | `{ sha256, tenantId, kind, record, traceId }` | +| IN | `verae.archive.query` | aggregator (broadcast) | `{ correlationId, sha256, tenantId, kinds[], traceId }` | +| OUT | `verae.archive.reply.` | aggregator | `{ archiveId, sha256, records[], traceId }` **only if bloom.mightHave(sha256)** | + +Subscribe to query **without** a shared queue group so every archive sees every query. diff --git a/packages/docs-master/modules/verae-archive-worm/SUMMARY.md b/packages/docs-master/modules/verae-archive-worm/SUMMARY.md new file mode 100644 index 0000000..fe2ac10 --- /dev/null +++ b/packages/docs-master/modules/verae-archive-worm/SUMMARY.md @@ -0,0 +1,14 @@ +# verae-archive-worm + +**Job:** Append-only WORM store of metadata and files. Bloom filter: miss → stay silent on NATS. + +**Expects:** +- `verae.archive.put` — append record, add sha256 to bloom +- `verae.archive.query` — `{ correlationId, sha256, tenantId, kinds[] }` (every node sees this; **not** a shared queue group) + +**Sends:** +- `verae.archive.reply.` **only on bloom hit** — `{ archiveId, sha256, records[] }` + +**Does not** write to the blockchain. + +**Test:** `npm test` — miss is `null`; put then query returns records. diff --git a/packages/docs-master/modules/verae-request-splitter/NATS.md b/packages/docs-master/modules/verae-request-splitter/NATS.md new file mode 100644 index 0000000..0e12c1b --- /dev/null +++ b/packages/docs-master/modules/verae-request-splitter/NATS.md @@ -0,0 +1,8 @@ +# NATS — verae-request-splitter + +| Direction | Address | To | Body | +|-----------|---------|-----|------| +| OUT | `verae.archive.put` | each WORM | one message per publicMeta / privateMeta / file | +| OUT | (in-process) chain | middleware | `{ sha256, hashAlg, data? }` | + +No inbound NATS in the in-process embedding. diff --git a/packages/docs-master/modules/verae-request-splitter/SUMMARY.md b/packages/docs-master/modules/verae-request-splitter/SUMMARY.md new file mode 100644 index 0000000..45c898f --- /dev/null +++ b/packages/docs-master/modules/verae-request-splitter/SUMMARY.md @@ -0,0 +1,13 @@ +# verae-request-splitter + +**Job:** Peel a request into (1) chain hash and (2) off-chain archive puts. + +**Expects messages from:** middleware-http (in-process call today; later NATS `verae.splitter.in` if split out). + +**Sends messages to:** +- Chain path: `{ sha256, hashAlg, data? }` +- `verae.archive.put` for each `publicMeta` / `privateMeta` / `file` + +**Does not talk to Zapier or NATS job poller.** + +**Test:** `npm test` — hash-only has zero puts; two files + public meta → 3 puts. diff --git a/packages/docs-master/modules/verae-zapier-middleware/NATS.md b/packages/docs-master/modules/verae-zapier-middleware/NATS.md new file mode 100644 index 0000000..abe42c8 --- /dev/null +++ b/packages/docs-master/modules/verae-zapier-middleware/NATS.md @@ -0,0 +1,9 @@ +# NATS — verae-middleware + +| Direction | Address | Peer | Body | +|-----------|---------|------|------| +| OUT | `verae.zapier.jobs.watch` | job-poller | job watch payload | +| IN | `verae.zapier.jobs.events` | waiter | terminal status | +| OUT | (via aggregator) `verae.archive.query` | WORM | attached lookup | + +HTTP remains the only Zapier-facing surface. diff --git a/packages/docs-master/modules/verae-zapier-middleware/SUMMARY.md b/packages/docs-master/modules/verae-zapier-middleware/SUMMARY.md new file mode 100644 index 0000000..7b275df --- /dev/null +++ b/packages/docs-master/modules/verae-zapier-middleware/SUMMARY.md @@ -0,0 +1,15 @@ +# verae-middleware (HTTP edge) + +**Job:** Public `/zapier/v1/*`. Auth, entitlements, timestamp/wait, hash lookup, objects, receipts. Publishes NATS jobs; waits on events. + +**Expects HTTPS from:** zappier-edge (or Zapier if pointed here). + +**Sends:** +- HTTPS to verae-chain-client (`/api/timestamp`, `/api/status`, `/api/verify`) +- `verae.zapier.jobs.watch` after create +- waits `verae.zapier.jobs.events` +- splitter + aggregator when attachments / `includeAttached` + +**Returns to requestor:** 202 `{ jobId }` or wait JSON (completed / pending+jobId). + +**NATS:** never exposed to Zapier. diff --git a/packages/docs-master/modules/verae-zapier/SUMMARY.md b/packages/docs-master/modules/verae-zapier/SUMMARY.md new file mode 100644 index 0000000..94b22b3 --- /dev/null +++ b/packages/docs-master/modules/verae-zapier/SUMMARY.md @@ -0,0 +1,9 @@ +# verae-zapier-app + +**Job:** Full Zapier CLI app: timestamp, wait, verify, batch, job status, hash lookup, REST Hook. + +**Expects:** `zmw_` middleware key (today) or zappier key after composition. + +**Sends:** HTTPS to middleware `/zapier/v1/*`. + +**Does not** speak NATS. diff --git a/packages/docs-master/modules/zappier/SUMMARY.md b/packages/docs-master/modules/zappier/SUMMARY.md new file mode 100644 index 0000000..fdf0ba3 --- /dev/null +++ b/packages/docs-master/modules/zappier/SUMMARY.md @@ -0,0 +1,11 @@ +# zappier-edge + +**Job:** Metered commercial API, portal, admin, Stripe. Zapier’s only public HTTPS hop. + +**Expects:** Zapier Platform app with `x-api-key`. + +**Sends:** HTTPS to middleware `/zapier/v1/*` (or local mock `/v1/timestamp`, `/v1/add`, `/v1/hashes`, `/v1/receipts`). + +**Does not** speak NATS or `api.veraetime.net`. + +**Test:** `npm test` (Jest, 177+). diff --git a/packages/verae-activate/SUMMARY.md b/packages/verae-activate/SUMMARY.md new file mode 100644 index 0000000..975de3d --- /dev/null +++ b/packages/verae-activate/SUMMARY.md @@ -0,0 +1,11 @@ +# verae-activate (Zapier Platform app) + +**Job:** Private Zapier integration you push first. Actions run in Zapier cloud. + +**Expects:** User connection (API key optional). + +**Sends:** nothing if fields blank (Add Numbers / Echo / SHA256 / Create Timestamp local mock). If `api_base` + `api_key`, HTTPS to zappier-edge. + +**Does not** speak NATS. + +**Test:** `npm test`; `zapier-platform validate`. diff --git a/packages/verae-archive-aggregator/NATS.md b/packages/verae-archive-aggregator/NATS.md new file mode 100644 index 0000000..6a73642 --- /dev/null +++ b/packages/verae-archive-aggregator/NATS.md @@ -0,0 +1,8 @@ +# NATS — verae-archive-aggregator + +| Direction | Address | Peer | Body | +|-----------|---------|------|------| +| OUT | `verae.archive.query` | all WORM | `{ correlationId, sha256, tenantId, kinds[], traceId }` | +| IN | `verae.archive.reply.` | WORM hits | `{ archiveId, sha256, records[] }` | + +Stop collecting at `WAIT_ARCHIVE_MS`. Silent archives are listed in `silent[]`, not errors. diff --git a/packages/verae-archive-aggregator/SUMMARY.md b/packages/verae-archive-aggregator/SUMMARY.md new file mode 100644 index 0000000..d6951ab --- /dev/null +++ b/packages/verae-archive-aggregator/SUMMARY.md @@ -0,0 +1,11 @@ +# verae-archive-aggregator + +**Job:** If `includeAttached`, fan out archive query and merge replies into the job JSON. + +**Expects:** wait-path flag from middleware-http; replies on `verae.archive.reply.`. + +**Sends:** `verae.archive.query` to all WORM nodes. + +**Timeout:** missing archives omit slices; chain receipt still succeeds. + +**Test:** `npm test` — one silent bloom miss + one hit; merge seal + metadata-attach. diff --git a/packages/verae-archive-aggregator/package.json b/packages/verae-archive-aggregator/package.json new file mode 100644 index 0000000..56b82c5 --- /dev/null +++ b/packages/verae-archive-aggregator/package.json @@ -0,0 +1,8 @@ +{ + "name": "verae-archive-aggregator", + "version": "0.1.0", + "type": "module", + "description": "Aggregate bloom-filtered WORM archive replies into job JSON", + "scripts": { "test": "node --test test/*.test.js" }, + "engines": { "node": ">=20" } +} diff --git a/packages/verae-archive-aggregator/src/aggregate.js b/packages/verae-archive-aggregator/src/aggregate.js new file mode 100644 index 0000000..bdc308e --- /dev/null +++ b/packages/verae-archive-aggregator/src/aggregate.js @@ -0,0 +1,53 @@ +/** + * Fan-out archive query and merge replies until timeout. + * @module aggregate + */ + +/** + * @param {object} opts + * @param {string} opts.sha256 + * @param {string} opts.tenantId + * @param {string[]} [opts.kinds] + * @param {Array<{ archiveId: string, query: (sha: string) => object[]|null }>} opts.archives + * @param {number} [opts.timeoutMs=50] unused in in-process mock (archives are sync) + * @returns {{ archivesQueried: boolean, archiveReplies: number, records: object[], silent: string[] }} + */ +export function aggregateAttached({ sha256, tenantId, kinds = ['publicMeta', 'privateMeta', 'file'], archives }) { + const records = []; + const silent = []; + for (const arch of archives) { + const hit = arch.query(sha256); + if (hit == null) { + silent.push(arch.archiveId); + continue; + } + for (const rec of hit) { + if (!kinds.length || kinds.includes(rec.kind)) records.push(rec); + } + } + return { + sha256, + tenantId, + archivesQueried: true, + archiveReplies: archives.length - silent.length, + silent, + records, + }; +} + +/** + * Merge chain receipts with archive records. + * @param {object[]} chainReceipts + * @param {object[]} archiveRecords + */ +export function mergeReceipts(chainReceipts, archiveRecords) { + const extra = archiveRecords + .filter((r) => r.kind === 'publicMeta' || r.kind === 'privateMeta' || r.kind === 'file') + .map((r) => ({ + kind: r.kind === 'file' ? 'file-attach' : 'metadata-attach', + archiveId: r.archiveId, + attachedAt: r.storedAt, + ...r.record, + })); + return [...chainReceipts, ...extra]; +} diff --git a/packages/verae-archive-aggregator/test/aggregate.test.js b/packages/verae-archive-aggregator/test/aggregate.test.js new file mode 100644 index 0000000..0953521 --- /dev/null +++ b/packages/verae-archive-aggregator/test/aggregate.test.js @@ -0,0 +1,28 @@ +import { describe, it } from 'node:test'; +import assert from 'node:assert/strict'; +import { WormArchive } from '../../verae-archive-worm/src/archive.js'; +import { aggregateAttached, mergeReceipts } from '../src/aggregate.js'; + +describe('aggregateAttached', () => { + it('silent miss + one hit', () => { + const a = new WormArchive('archive-a'); + const b = new WormArchive('archive-b'); + const sha = 'c'.repeat(64); + b.put({ sha256: sha, kind: 'publicMeta', record: { publicMetadata: { k: 1 } } }); + const out = aggregateAttached({ sha256: sha, tenantId: 't1', archives: [a, b] }); + assert.equal(out.archiveReplies, 1); + assert.deepEqual(out.silent, ['archive-a']); + assert.equal(out.records[0].record.publicMetadata.k, 1); + }); + + it('merges seal + attach receipts', () => { + const seal = [{ kind: 'seal', timestamp: '2026-01-01T00:00:00Z', certificate: 'x' }]; + const recs = [ + { kind: 'publicMeta', archiveId: 'archive-b', storedAt: '2026-01-02T00:00:00Z', record: { publicMetadata: { n: 2 } } }, + ]; + const merged = mergeReceipts(seal, recs); + assert.equal(merged.length, 2); + assert.equal(merged[0].kind, 'seal'); + assert.equal(merged[1].kind, 'metadata-attach'); + }); +}); diff --git a/packages/verae-archive-worm/NATS.md b/packages/verae-archive-worm/NATS.md new file mode 100644 index 0000000..a64f14f --- /dev/null +++ b/packages/verae-archive-worm/NATS.md @@ -0,0 +1,9 @@ +# NATS — verae-archive-worm + +| Direction | Address | From / to | Body | +|-----------|---------|-----------|------| +| IN | `verae.archive.put` | splitter | `{ sha256, tenantId, kind, record, traceId }` | +| IN | `verae.archive.query` | aggregator (broadcast) | `{ correlationId, sha256, tenantId, kinds[], traceId }` | +| OUT | `verae.archive.reply.` | aggregator | `{ archiveId, sha256, records[], traceId }` **only if bloom.mightHave(sha256)** | + +Subscribe to query **without** a shared queue group so every archive sees every query. diff --git a/packages/verae-archive-worm/SUMMARY.md b/packages/verae-archive-worm/SUMMARY.md new file mode 100644 index 0000000..fe2ac10 --- /dev/null +++ b/packages/verae-archive-worm/SUMMARY.md @@ -0,0 +1,14 @@ +# verae-archive-worm + +**Job:** Append-only WORM store of metadata and files. Bloom filter: miss → stay silent on NATS. + +**Expects:** +- `verae.archive.put` — append record, add sha256 to bloom +- `verae.archive.query` — `{ correlationId, sha256, tenantId, kinds[] }` (every node sees this; **not** a shared queue group) + +**Sends:** +- `verae.archive.reply.` **only on bloom hit** — `{ archiveId, sha256, records[] }` + +**Does not** write to the blockchain. + +**Test:** `npm test` — miss is `null`; put then query returns records. diff --git a/packages/verae-archive-worm/package.json b/packages/verae-archive-worm/package.json new file mode 100644 index 0000000..b361d1e --- /dev/null +++ b/packages/verae-archive-worm/package.json @@ -0,0 +1,8 @@ +{ + "name": "verae-archive-worm", + "version": "0.1.0", + "type": "module", + "description": "Bloom-filtered WORM archive node for off-chain metadata and files", + "scripts": { "test": "node --test test/*.test.js" }, + "engines": { "node": ">=20" } +} diff --git a/packages/verae-archive-worm/src/archive.js b/packages/verae-archive-worm/src/archive.js new file mode 100644 index 0000000..1963e97 --- /dev/null +++ b/packages/verae-archive-worm/src/archive.js @@ -0,0 +1,40 @@ +/** + * In-process WORM archive: append-only records + bloom. + * @module archive + */ + +import { Bloom } from './bloom.js'; + +export class WormArchive { + /** + * @param {string} archiveId + */ + constructor(archiveId) { + this.archiveId = archiveId; + this.bloom = new Bloom(); + /** @type {Map} */ + this.byHash = new Map(); + } + + /** + * @param {{ sha256: string, kind: string, record: object }} put + */ + put(put) { + const key = String(put.sha256).toLowerCase(); + const list = this.byHash.get(key) ?? []; + list.push({ kind: put.kind, record: put.record, archiveId: this.archiveId, storedAt: new Date().toISOString() }); + this.byHash.set(key, list); + this.bloom.add(key); + } + + /** + * Bloom miss → null (caller must stay silent). Hit → records (may be empty if false positive). + * @param {string} sha256 + * @returns {object[]|null} + */ + query(sha256) { + const key = String(sha256).toLowerCase(); + if (!this.bloom.mightHave(key)) return null; + return this.byHash.get(key) ?? []; + } +} diff --git a/packages/verae-archive-worm/src/bloom.js b/packages/verae-archive-worm/src/bloom.js new file mode 100644 index 0000000..d5e23d0 --- /dev/null +++ b/packages/verae-archive-worm/src/bloom.js @@ -0,0 +1,46 @@ +/** + * Tiny Bloom filter for SHA256 membership (false positives ok). + * @module bloom + */ + +import { createHash } from 'node:crypto'; + +export class Bloom { + /** + * @param {number} [m=16384] bit count + * @param {number} [k=4] hash rounds + */ + constructor(m = 16384, k = 4) { + this.m = m; + this.k = k; + this.bits = Buffer.alloc(Math.ceil(m / 8)); + } + + /** + * @param {string} s + * @returns {number[]} + */ + hashes(s) { + const digest = createHash('sha256').update(String(s)).digest(); + const out = []; + for (let i = 0; i < this.k; i += 1) { + const n = digest.readUInt32BE((i * 4) % 28); + out.push(n % this.m); + } + return out; + } + + add(s) { + for (const i of this.hashes(s)) { + this.bits[i >> 3] |= 1 << (i & 7); + } + } + + /** + * @param {string} s + * @returns {boolean} maybe (never a guaranteed yes) + */ + mightHave(s) { + return this.hashes(s).every((i) => (this.bits[i >> 3] & (1 << (i & 7))) !== 0); + } +} diff --git a/packages/verae-archive-worm/test/archive.test.js b/packages/verae-archive-worm/test/archive.test.js new file mode 100644 index 0000000..4d9a5e4 --- /dev/null +++ b/packages/verae-archive-worm/test/archive.test.js @@ -0,0 +1,28 @@ +import { describe, it } from 'node:test'; +import assert from 'node:assert/strict'; +import { WormArchive } from '../src/archive.js'; +import { Bloom } from '../src/bloom.js'; + +describe('WormArchive + Bloom', () => { + it('miss stays silent (null)', () => { + const a = new WormArchive('archive-a'); + assert.equal(a.query('a'.repeat(64)), null); + }); + + it('put then query returns records', () => { + const a = new WormArchive('archive-b'); + const sha = 'b'.repeat(64); + a.put({ sha256: sha, kind: 'publicMeta', record: { publicMetadata: { t: 1 } } }); + const hit = a.query(sha); + assert.ok(Array.isArray(hit)); + assert.equal(hit[0].kind, 'publicMeta'); + assert.equal(hit[0].archiveId, 'archive-b'); + }); + + it('bloom reports maybe after add', () => { + const b = new Bloom(); + assert.equal(b.mightHave('hello'), false); + b.add('hello'); + assert.equal(b.mightHave('hello'), true); + }); +}); diff --git a/packages/verae-request-splitter/NATS.md b/packages/verae-request-splitter/NATS.md new file mode 100644 index 0000000..0e12c1b --- /dev/null +++ b/packages/verae-request-splitter/NATS.md @@ -0,0 +1,8 @@ +# NATS — verae-request-splitter + +| Direction | Address | To | Body | +|-----------|---------|-----|------| +| OUT | `verae.archive.put` | each WORM | one message per publicMeta / privateMeta / file | +| OUT | (in-process) chain | middleware | `{ sha256, hashAlg, data? }` | + +No inbound NATS in the in-process embedding. diff --git a/packages/verae-request-splitter/SUMMARY.md b/packages/verae-request-splitter/SUMMARY.md new file mode 100644 index 0000000..45c898f --- /dev/null +++ b/packages/verae-request-splitter/SUMMARY.md @@ -0,0 +1,13 @@ +# verae-request-splitter + +**Job:** Peel a request into (1) chain hash and (2) off-chain archive puts. + +**Expects messages from:** middleware-http (in-process call today; later NATS `verae.splitter.in` if split out). + +**Sends messages to:** +- Chain path: `{ sha256, hashAlg, data? }` +- `verae.archive.put` for each `publicMeta` / `privateMeta` / `file` + +**Does not talk to Zapier or NATS job poller.** + +**Test:** `npm test` — hash-only has zero puts; two files + public meta → 3 puts. diff --git a/packages/verae-request-splitter/package.json b/packages/verae-request-splitter/package.json new file mode 100644 index 0000000..f058e9b --- /dev/null +++ b/packages/verae-request-splitter/package.json @@ -0,0 +1,10 @@ +{ + "name": "verae-request-splitter", + "version": "0.1.0", + "type": "module", + "description": "Split hash/sign vs attachments for chain vs WORM archives", + "scripts": { + "test": "node --test test/*.test.js" + }, + "engines": { "node": ">=20" } +} diff --git a/packages/verae-request-splitter/src/splitRequest.js b/packages/verae-request-splitter/src/splitRequest.js new file mode 100644 index 0000000..7250d05 --- /dev/null +++ b/packages/verae-request-splitter/src/splitRequest.js @@ -0,0 +1,71 @@ +/** + * Split a Zapier/middleware request into chain vs archive parts. + * @module splitRequest + */ + +import { createHash } from 'node:crypto'; + +/** + * @param {string} data + * @returns {string} + */ +export function sha256Hex(data) { + return createHash('sha256').update(String(data), 'utf8').digest('hex'); +} + +/** + * @param {object} body + * @returns {{ + * chain: { sha256: string, hashAlg: string, data?: string }, + * archivePuts: Array<{ kind: string, sha256: string, record: object }>, + * includeAttached: boolean + * }} + */ +export function splitRequest(body = {}) { + const sha256 = (body.sha256 || (body.data != null ? sha256Hex(body.data) : '')).toLowerCase(); + if (!sha256) { + throw new Error('data or sha256 is required'); + } + const includeAttached = Boolean(body.includeAttached); + const archivePuts = []; + if (body.publicMetadata && Object.keys(body.publicMetadata).length) { + archivePuts.push({ + kind: 'publicMeta', + sha256, + record: { publicMetadata: body.publicMetadata }, + }); + } + if (body.privateMetadata && Object.keys(body.privateMetadata).length) { + archivePuts.push({ + kind: 'privateMeta', + sha256, + record: { privateMetadata: body.privateMetadata }, + }); + } + const files = Array.isArray(body.files) ? body.files : []; + for (const f of files) { + const bytes = f.contentBase64 + ? Buffer.from(String(f.contentBase64), 'base64') + : Buffer.from(String(f.data ?? ''), 'utf8'); + const fileHash = sha256Hex(bytes); + archivePuts.push({ + kind: 'file', + sha256, + record: { + path: f.path || f.filename || 'file.bin', + size: bytes.length, + contentSha256: fileHash, + contentBase64: bytes.toString('base64'), + }, + }); + } + return { + chain: { + sha256, + hashAlg: body.hashAlg || 'SHA256', + data: body.data, + }, + archivePuts, + includeAttached, + }; +} diff --git a/packages/verae-request-splitter/test/splitRequest.test.js b/packages/verae-request-splitter/test/splitRequest.test.js new file mode 100644 index 0000000..553904e --- /dev/null +++ b/packages/verae-request-splitter/test/splitRequest.test.js @@ -0,0 +1,32 @@ +import { describe, it } from 'node:test'; +import assert from 'node:assert/strict'; +import { splitRequest, sha256Hex } from '../src/splitRequest.js'; + +describe('splitRequest', () => { + it('hash-only has no archive puts', () => { + const out = splitRequest({ data: 'hello' }); + assert.equal(out.chain.sha256, sha256Hex('hello')); + assert.equal(out.archivePuts.length, 0); + assert.equal(out.includeAttached, false); + }); + + it('splits public metadata and two files onto archive puts', () => { + const out = splitRequest({ + data: 'hello', + publicMetadata: { title: 't' }, + files: [ + { path: 'a.txt', data: 'A' }, + { path: 'b.txt', data: 'B' }, + ], + includeAttached: true, + }); + assert.equal(out.archivePuts.length, 3); + assert.equal(out.archivePuts.filter((p) => p.kind === 'file').length, 2); + assert.equal(out.includeAttached, true); + assert.equal(out.chain.sha256, sha256Hex('hello')); + }); + + it('throws without data or sha256', () => { + assert.throws(() => splitRequest({}), /data or sha256/); + }); +}); diff --git a/packages/verae-zapier-middleware/NATS.md b/packages/verae-zapier-middleware/NATS.md new file mode 100644 index 0000000..abe42c8 --- /dev/null +++ b/packages/verae-zapier-middleware/NATS.md @@ -0,0 +1,9 @@ +# NATS — verae-middleware + +| Direction | Address | Peer | Body | +|-----------|---------|------|------| +| OUT | `verae.zapier.jobs.watch` | job-poller | job watch payload | +| IN | `verae.zapier.jobs.events` | waiter | terminal status | +| OUT | (via aggregator) `verae.archive.query` | WORM | attached lookup | + +HTTP remains the only Zapier-facing surface. diff --git a/packages/verae-zapier-middleware/SUMMARY.md b/packages/verae-zapier-middleware/SUMMARY.md new file mode 100644 index 0000000..7b275df --- /dev/null +++ b/packages/verae-zapier-middleware/SUMMARY.md @@ -0,0 +1,15 @@ +# verae-middleware (HTTP edge) + +**Job:** Public `/zapier/v1/*`. Auth, entitlements, timestamp/wait, hash lookup, objects, receipts. Publishes NATS jobs; waits on events. + +**Expects HTTPS from:** zappier-edge (or Zapier if pointed here). + +**Sends:** +- HTTPS to verae-chain-client (`/api/timestamp`, `/api/status`, `/api/verify`) +- `verae.zapier.jobs.watch` after create +- waits `verae.zapier.jobs.events` +- splitter + aggregator when attachments / `includeAttached` + +**Returns to requestor:** 202 `{ jobId }` or wait JSON (completed / pending+jobId). + +**NATS:** never exposed to Zapier. diff --git a/packages/verae-zapier/SUMMARY.md b/packages/verae-zapier/SUMMARY.md new file mode 100644 index 0000000..94b22b3 --- /dev/null +++ b/packages/verae-zapier/SUMMARY.md @@ -0,0 +1,9 @@ +# verae-zapier-app + +**Job:** Full Zapier CLI app: timestamp, wait, verify, batch, job status, hash lookup, REST Hook. + +**Expects:** `zmw_` middleware key (today) or zappier key after composition. + +**Sends:** HTTPS to middleware `/zapier/v1/*`. + +**Does not** speak NATS. diff --git a/packages/zappier/SUMMARY.md b/packages/zappier/SUMMARY.md new file mode 100644 index 0000000..fdf0ba3 --- /dev/null +++ b/packages/zappier/SUMMARY.md @@ -0,0 +1,11 @@ +# zappier-edge + +**Job:** Metered commercial API, portal, admin, Stripe. Zapier’s only public HTTPS hop. + +**Expects:** Zapier Platform app with `x-api-key`. + +**Sends:** HTTPS to middleware `/zapier/v1/*` (or local mock `/v1/timestamp`, `/v1/add`, `/v1/hashes`, `/v1/receipts`). + +**Does not** speak NATS or `api.veraetime.net`. + +**Test:** `npm test` (Jest, 177+). diff --git a/scripts/push-module-repos.sh b/scripts/push-module-repos.sh new file mode 100755 index 0000000..1381120 --- /dev/null +++ b/scripts/push-module-repos.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +# Create Forgejo repos and push each module (no node_modules). +set -euo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +TOKEN_FILE="${FORGEJO_TOKEN_FILE:-/tmp/forgejo-zapier.token}" +TOKEN="$(cat "$TOKEN_FILE")" +API="https://git.georgelambert.org/api/v1" +GITSSH="ssh://git@git.georgelambert.org:2223/marchon" +export GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new -p 2223" + +create() { + local name="$1" desc="$2" + curl -sS -X POST "$API/user/repos" \ + -H "Authorization: token $TOKEN" \ + -H "Content-Type: application/json" \ + -d "{\"name\":\"$name\",\"description\":\"$desc\",\"private\":false,\"auto_init\":false}" \ + >/tmp/forgejo-mod.json || true + echo "repo $name" +} + +push_dir() { + local src="$1" name="$2" + local tmp + tmp="$(mktemp -d)" + rsync -a --exclude node_modules --exclude dist --exclude data --exclude '*.db' --exclude .git \ + "$src/" "$tmp/" + git -C "$tmp" init -q -b main + git -C "$tmp" add -A + git -C "$tmp" -c user.email=marchon@gmail.com -c user.name="George Lambert" \ + commit -q -m "Initial import of $name from zapier monorepo" + git -C "$tmp" remote add origin "$GITSSH/${name}.git" + git -C "$tmp" push -u origin main + git -C "$tmp" push origin main:master || true + rm -rf "$tmp" + echo "pushed $name" +} + +create zappier-edge "Metered commercial HTTPS edge (portal, admin, Stripe)" +create verae-middleware "Zapier-facing HTTP middleware + NATS workers" +create verae-zapier-app "Zapier Platform CLI app (timestamp/verify/hooks)" +create verae-activate "Activate-now Zapier app (Add Numbers, Echo, SHA256)" +create verae-request-splitter "Split hash vs attachments for chain vs WORM" +create verae-archive-worm "Bloom-filtered WORM archive node" +create verae-archive-aggregator "Aggregate archive replies into job JSON" +create zapier-docs-master "Master summaries, NATS contracts, message flows" + +push_dir "$ROOT/packages/zappier" zappier-edge +push_dir "$ROOT/packages/verae-zapier-middleware" verae-middleware +push_dir "$ROOT/packages/verae-zapier" verae-zapier-app +push_dir "$ROOT/packages/verae-activate" verae-activate +push_dir "$ROOT/packages/verae-request-splitter" verae-request-splitter +push_dir "$ROOT/packages/verae-archive-worm" verae-archive-worm +push_dir "$ROOT/packages/verae-archive-aggregator" verae-archive-aggregator +push_dir "$ROOT/packages/docs-master" zapier-docs-master + +echo ALL_MODULE_REPOS_PUSHED