Split NATS modules: splitter, WORM bloom archive, aggregator, docs-master
Some checks are pending
offline / test (push) Waiting to run
Some checks are pending
offline / test (push) Waiting to run
Each module has SUMMARY.md and NATS.md (who sends/receives, subject addresses, payload). Tests: splitter 3, worm 3, aggregator 2. Forgejo repos pushed separately; this commit keeps them in the monorepo.
This commit is contained in:
parent
ca693366dc
commit
8f9707a3b7
39 changed files with 893 additions and 0 deletions
29
packages/docs-master/MESSAGE-FLOWS.md
Normal file
29
packages/docs-master/MESSAGE-FLOWS.md
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
# Message flows
|
||||
|
||||
## 1. Create timestamp (async)
|
||||
|
||||
1. Zapier → `POST https://zappier-edge/v1/timestamp` (`x-api-key`)
|
||||
2. zappier-edge meters → `POST middleware/zapier/v1/timestamp`
|
||||
3. splitter: chain sha256; optional `verae.archive.put`
|
||||
4. middleware → chain create or existing lookup
|
||||
5. middleware publishes `verae.zapier.jobs.watch` `{ jobId, tenantId, … }`
|
||||
6. HTTP 202 `{ jobId }` back to Zapier
|
||||
7. job-poller consumes watch, GET chain status
|
||||
8. On terminal: `verae.zapier.jobs.events`
|
||||
9. webhook-deliver POSTs Zapier REST Hook if subscribed
|
||||
|
||||
## 2. Wait
|
||||
|
||||
Same as (1) but HTTP holds until `jobs.events` or `WAIT_TIMEOUT_MS` → `{ status: pending, jobId }`.
|
||||
|
||||
## 3. Hash already registered
|
||||
|
||||
Step 4 returns original jobId + original seal. No second chain write. `receipts[0].kind = seal`. Later archive attaches become `receipts[1…]` if `includeAttached`.
|
||||
|
||||
## 4. includeAttached
|
||||
|
||||
After seal is known, aggregator publishes `verae.archive.query`. Each WORM: bloom miss = no packet; hit = `verae.archive.reply.<correlationId>`. Aggregator merges into wait JSON.
|
||||
|
||||
## 5. Multipart attachments
|
||||
|
||||
Splitter emits one `archive.put` per file (`kind: file`, `contentSha256`). Chain never stores bytes.
|
||||
21
packages/docs-master/README.md
Normal file
21
packages/docs-master/README.md
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
# zapier-docs-master
|
||||
|
||||
Summaries and NATS interaction map for every Verae Time × Zapier module.
|
||||
|
||||
**Live HTML:** https://zapier.georgelambert.org/
|
||||
|
||||
**Monorepo:** https://git.georgelambert.org/marchon/master-zapier-plan-draft
|
||||
|
||||
## Modules
|
||||
|
||||
| Module | Summary | NATS in | NATS out |
|
||||
|--------|---------|---------|----------|
|
||||
| zappier-edge | Metered HTTPS for Zapier | — | — (HTTPS to middleware) |
|
||||
| verae-middleware | Job id + wait HTTP | jobs.events | jobs.watch, archive.query via aggregator |
|
||||
| verae-activate | Pushable Zapier app | — | — |
|
||||
| verae-zapier-app | Full Zapier nouns | — | — |
|
||||
| request-splitter | Hash vs files | — | archive.put |
|
||||
| archive-worm | Bloom WORM node | archive.query, archive.put | archive.reply.* (hit only) |
|
||||
| archive-aggregator | Merge archive replies | archive.reply.* | archive.query |
|
||||
|
||||
Full contracts: `../docs/02-architecture/modules-and-nats.md` and `archive-nats.md` in the monorepo / on the website.
|
||||
90
packages/docs-master/archive-nats.md
Normal file
90
packages/docs-master/archive-nats.md
Normal file
|
|
@ -0,0 +1,90 @@
|
|||
# Archive NATS: jobs, multipart split, hash receipts, WORM bloom fan-out
|
||||
|
||||
Zapier never speaks NATS. HTTPS stops at zappier → middleware. Middleware owns jobs, splitting, chain lookup, and archive aggregation.
|
||||
|
||||
## End-to-end
|
||||
|
||||
```text
|
||||
Zapier --HTTPS--> zappier (meter, x-api-key)
|
||||
--HTTPS--> middleware /zapier/v1/timestamp[/wait]
|
||||
1. split multipart
|
||||
chain: SHA256 only (+ optional public-meta digest)
|
||||
archives: public JSON, encrypted private JSON, files
|
||||
2. publish jobs.watch → return jobId (202)
|
||||
3. waiters subscribe jobs.events
|
||||
4. lookup SHA256 on chain (mock or Verae)
|
||||
already sealed → original receipt, no new seal
|
||||
later attach records → extra receipts
|
||||
5. if includeAttached:
|
||||
publish verae.archive.query
|
||||
WORM nodes: bloom miss = silence
|
||||
bloom hit = reply
|
||||
aggregate until WAIT_ARCHIVE_MS
|
||||
6. jobs.events completed JSON → waiter / REST Hook
|
||||
```
|
||||
|
||||
Blockchain stores **hash + time + block + certificate**. Public metadata, encrypted metadata, and file bytes live on **WORM archives**.
|
||||
|
||||
## Splitter
|
||||
|
||||
`splitRequest(body | multipart)`:
|
||||
|
||||
| Field | Destination |
|
||||
|-------|-------------|
|
||||
| `data` / `sha256` | Chain register or lookup |
|
||||
| `publicMetadata` | Archive put (clear) |
|
||||
| `privateMetadata` | Archive put (ciphertext) |
|
||||
| `files[]` | Archive put; chain gets content hashes + ids |
|
||||
| `includeAttached` | Whether wait path queries archives |
|
||||
|
||||
## Hash already registered
|
||||
|
||||
Return original `jobId` and original seal. Do not write a second chain timestamp.
|
||||
|
||||
If later attach jobs exist for that hash, `receipts` is an array: seal first, then attachment receipts in time order.
|
||||
|
||||
## Subjects
|
||||
|
||||
| Subject | Publisher | Subscriber |
|
||||
|---------|-----------|------------|
|
||||
| `verae.zapier.jobs.watch` | HTTP edge | job poller |
|
||||
| `verae.zapier.jobs.events` | poller | waiter, webhook router |
|
||||
| `verae.archive.put` | splitter | archive that owns the shard |
|
||||
| `verae.archive.query` | aggregator | **every** archive (not a shared queue group) |
|
||||
| `verae.archive.reply.<correlationId>` | archive on bloom hit | aggregator |
|
||||
|
||||
Query payload: `{ correlationId, sha256, tenantId, kinds[] }`.
|
||||
Reply payload: `{ archiveId, sha256, records[] }`.
|
||||
|
||||
Bloom miss → no reply. Aggregator timeout → complete with whatever arrived.
|
||||
|
||||
## WORM archives
|
||||
|
||||
Each process holds append-only records + a bloom of SHA256 keys it stores. False positives OK; false negatives must be rare. Bloom is not an ACL — on hit, still check tenant/share.
|
||||
|
||||
Harness: three mock archives with overlapping hashes.
|
||||
|
||||
## Completed job JSON (wait / hook)
|
||||
|
||||
```json
|
||||
{
|
||||
"jobId": "…",
|
||||
"status": "completed",
|
||||
"sha256": "…",
|
||||
"receipts": [
|
||||
{ "kind": "seal", "timestamp": "…", "certificate": "…", "blockIndex": 42 },
|
||||
{ "kind": "metadata-attach", "attachedAt": "…", "publicMetadata": {} }
|
||||
],
|
||||
"files": [{ "id": "…", "sha256": "…", "archiveId": "archive-b" }],
|
||||
"archivesQueried": true,
|
||||
"archiveReplies": 2
|
||||
}
|
||||
```
|
||||
|
||||
Flag off or all blooms miss → `files` empty, extra receipts omitted.
|
||||
|
||||
## Security
|
||||
|
||||
- Zapier never connects to NATS or archives.
|
||||
- Private metadata only on authenticated archive replies.
|
||||
- NS1 NATS stays loopback; use `scripts/nats-tunnel.sh`.
|
||||
33
packages/docs-master/composition.md
Normal file
33
packages/docs-master/composition.md
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
# Composition: zappier commercial edge + Verae adapter
|
||||
|
||||
```text
|
||||
Users → Zapier UI
|
||||
Zapier cloud runs packages/verae-zapier
|
||||
--HTTPS, x-api-key--> packages/zappier /v1/*
|
||||
(meter, quote, 401/403, usage)
|
||||
--internal HTTPS--> packages/verae-zapier-middleware /zapier/v1/*
|
||||
--sync--> api.veraetime.net or MOCK_VERAE
|
||||
--NATS--> workers --> Verae + Zapier REST Hooks
|
||||
|
||||
Humans → zappier /portal signup, API key, usage, reloads, invoices
|
||||
Ops → zappier /admin rate card, tiers, customers, PO invoices
|
||||
```
|
||||
|
||||
## Ownership
|
||||
|
||||
| Concern | Package |
|
||||
|---------|---------|
|
||||
| API keys, custom pricing, Stripe, invoices, portal | `packages/zappier` |
|
||||
| Timestamp/verify/status, NATS, REST Hooks, mock Verae | `packages/verae-zapier-middleware` |
|
||||
| Zapier Platform nouns (creates/searches/triggers) | `packages/verae-zapier` |
|
||||
| Official SDK reference | `vendor/zapier-platform` |
|
||||
|
||||
Zapier never talks to NATS or `api.veraetime.net`.
|
||||
|
||||
Do not reimplement Stripe, invoices, or the rate-card UI inside the Verae middleware.
|
||||
|
||||
Public keys are zappier `x-api-key` (issued at portal signup). Middleware `zmw_` / `PLAN_LIMITS` stay internal until composition PR 4 removes them from the public path.
|
||||
|
||||
Demo zappier routes `/v1/transform` and `/v1/storage` are a metering sandbox, not Verae timestamping or encrypted LTS.
|
||||
|
||||
Payment invoices (zappier) are not certified timestamp receipts (Verae feature i).
|
||||
52
packages/docs-master/modules-and-nats.md
Normal file
52
packages/docs-master/modules-and-nats.md
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
# Modules and NATS message contracts
|
||||
|
||||
Master map of every service in the Zapier ↔ middleware ↔ chain ↔ WORM-archive path.
|
||||
|
||||
Zapier cloud **never** connects to NATS. Only middleware workers and archives do.
|
||||
|
||||
## Who talks to whom (HTTPS vs NATS)
|
||||
|
||||
```text
|
||||
[Zapier Platform app] HTTPS [zappier-edge] HTTPS [middleware-http]
|
||||
|
|
||||
NATS JetStream (NS1 loopback :4222) |
|
||||
+------------------+------------------+-------------+
|
||||
| | |
|
||||
[job-poller] [webhook-deliver] [archive-aggregator]
|
||||
| |
|
||||
| HTTPS | NATS query/reply
|
||||
[verae-chain-client] |
|
||||
[archive-worm × N]
|
||||
bloom miss = silence
|
||||
```
|
||||
|
||||
## Module catalog
|
||||
|
||||
| Module | Repo (Forgejo) | Runtime | Listens | Sends |
|
||||
|--------|----------------|---------|---------|-------|
|
||||
| zapier-platform-app | `verae-zapier-app` | Zapier cloud | User Zap steps | HTTPS to zappier-edge |
|
||||
| zappier-edge | `zappier-edge` | Public HTTPS :3000 | Zapier, portal, admin | HTTPS to middleware |
|
||||
| middleware-http | `verae-middleware` | Public HTTPS :3100 | zappier-edge | NATS jobs.watch; HTTP to chain; wait on jobs.events |
|
||||
| job-poller | `verae-job-poller` | Worker | `verae.zapier.jobs.watch` | `verae.zapier.jobs.events`; HTTP GET chain status |
|
||||
| webhook-deliver | `verae-webhook-deliver` | Worker | `verae.zapier.webhooks.deliver` + events | HTTPS POST Zapier REST Hook |
|
||||
| request-splitter | `verae-request-splitter` | In middleware | HTTP body / multipart | chain hash; `verae.archive.put` |
|
||||
| archive-aggregator | `verae-archive-aggregator` | Worker / in wait | `includeAttached` on wait | `verae.archive.query`; reads `verae.archive.reply.<id>` |
|
||||
| archive-worm | `verae-archive-worm` | N copies | `verae.archive.query`, `verae.archive.put` | `verae.archive.reply.<id>` if bloom hits |
|
||||
| verae-chain-client | `verae-chain-client` | Library | — | HTTPS `api.veraetime.net` or MOCK |
|
||||
| docs-master | `zapier-docs-master` | Static | — | published on zapier.georgelambert.org |
|
||||
|
||||
## Addresses (subjects)
|
||||
|
||||
| Address | Kind | Payload (required fields) |
|
||||
|---------|------|---------------------------|
|
||||
| `verae.zapier.jobs.watch` | JetStream work queue | `tenantId`, `jobId`, `tokenRef`, `enqueuedAt`, `attempt`, `maxAttempts`, `intervalMs`, `traceId` |
|
||||
| `verae.zapier.jobs.events` | JetStream events | `event` (`timestamp.completed\|failed\|timeout`), `tenantId`, `jobId`, `status`, `traceId`, `emittedAt` |
|
||||
| `verae.zapier.webhooks.deliver` | JetStream work queue | `hookId`, `tenantId`, `targetUrl`, `event`, `payload`, `attempt`, `traceId` |
|
||||
| `verae.zapier.usage` | optional | `tenantId`, `action`, `amount`, `at` |
|
||||
| `verae.archive.put` | JetStream | `sha256`, `tenantId`, `kind` (`publicMeta\|privateMeta\|file\|tree`), `record`, `traceId` |
|
||||
| `verae.archive.query` | pub to **all** archives | `correlationId`, `sha256`, `tenantId`, `kinds[]`, `traceId` |
|
||||
| `verae.archive.reply.<correlationId>` | replies | `archiveId`, `sha256`, `records[]`, `traceId` |
|
||||
|
||||
## Completed job JSON (back to Zapier)
|
||||
|
||||
See `docs/02-architecture/archive-nats.md`. Seal receipts from chain; extra receipts and `files[]` from archive aggregation when `includeAttached` is true.
|
||||
11
packages/docs-master/modules/verae-activate/SUMMARY.md
Normal file
11
packages/docs-master/modules/verae-activate/SUMMARY.md
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
# verae-activate (Zapier Platform app)
|
||||
|
||||
**Job:** Private Zapier integration you push first. Actions run in Zapier cloud.
|
||||
|
||||
**Expects:** User connection (API key optional).
|
||||
|
||||
**Sends:** nothing if fields blank (Add Numbers / Echo / SHA256 / Create Timestamp local mock). If `api_base` + `api_key`, HTTPS to zappier-edge.
|
||||
|
||||
**Does not** speak NATS.
|
||||
|
||||
**Test:** `npm test`; `zapier-platform validate`.
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
# NATS — verae-archive-aggregator
|
||||
|
||||
| Direction | Address | Peer | Body |
|
||||
|-----------|---------|------|------|
|
||||
| OUT | `verae.archive.query` | all WORM | `{ correlationId, sha256, tenantId, kinds[], traceId }` |
|
||||
| IN | `verae.archive.reply.<correlationId>` | WORM hits | `{ archiveId, sha256, records[] }` |
|
||||
|
||||
Stop collecting at `WAIT_ARCHIVE_MS`. Silent archives are listed in `silent[]`, not errors.
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
# verae-archive-aggregator
|
||||
|
||||
**Job:** If `includeAttached`, fan out archive query and merge replies into the job JSON.
|
||||
|
||||
**Expects:** wait-path flag from middleware-http; replies on `verae.archive.reply.<correlationId>`.
|
||||
|
||||
**Sends:** `verae.archive.query` to all WORM nodes.
|
||||
|
||||
**Timeout:** missing archives omit slices; chain receipt still succeeds.
|
||||
|
||||
**Test:** `npm test` — one silent bloom miss + one hit; merge seal + metadata-attach.
|
||||
9
packages/docs-master/modules/verae-archive-worm/NATS.md
Normal file
9
packages/docs-master/modules/verae-archive-worm/NATS.md
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
# NATS — verae-archive-worm
|
||||
|
||||
| Direction | Address | From / to | Body |
|
||||
|-----------|---------|-----------|------|
|
||||
| IN | `verae.archive.put` | splitter | `{ sha256, tenantId, kind, record, traceId }` |
|
||||
| IN | `verae.archive.query` | aggregator (broadcast) | `{ correlationId, sha256, tenantId, kinds[], traceId }` |
|
||||
| OUT | `verae.archive.reply.<correlationId>` | aggregator | `{ archiveId, sha256, records[], traceId }` **only if bloom.mightHave(sha256)** |
|
||||
|
||||
Subscribe to query **without** a shared queue group so every archive sees every query.
|
||||
14
packages/docs-master/modules/verae-archive-worm/SUMMARY.md
Normal file
14
packages/docs-master/modules/verae-archive-worm/SUMMARY.md
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
# verae-archive-worm
|
||||
|
||||
**Job:** Append-only WORM store of metadata and files. Bloom filter: miss → stay silent on NATS.
|
||||
|
||||
**Expects:**
|
||||
- `verae.archive.put` — append record, add sha256 to bloom
|
||||
- `verae.archive.query` — `{ correlationId, sha256, tenantId, kinds[] }` (every node sees this; **not** a shared queue group)
|
||||
|
||||
**Sends:**
|
||||
- `verae.archive.reply.<correlationId>` **only on bloom hit** — `{ archiveId, sha256, records[] }`
|
||||
|
||||
**Does not** write to the blockchain.
|
||||
|
||||
**Test:** `npm test` — miss is `null`; put then query returns records.
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
# NATS — verae-request-splitter
|
||||
|
||||
| Direction | Address | To | Body |
|
||||
|-----------|---------|-----|------|
|
||||
| OUT | `verae.archive.put` | each WORM | one message per publicMeta / privateMeta / file |
|
||||
| OUT | (in-process) chain | middleware | `{ sha256, hashAlg, data? }` |
|
||||
|
||||
No inbound NATS in the in-process embedding.
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
# verae-request-splitter
|
||||
|
||||
**Job:** Peel a request into (1) chain hash and (2) off-chain archive puts.
|
||||
|
||||
**Expects messages from:** middleware-http (in-process call today; later NATS `verae.splitter.in` if split out).
|
||||
|
||||
**Sends messages to:**
|
||||
- Chain path: `{ sha256, hashAlg, data? }`
|
||||
- `verae.archive.put` for each `publicMeta` / `privateMeta` / `file`
|
||||
|
||||
**Does not talk to Zapier or NATS job poller.**
|
||||
|
||||
**Test:** `npm test` — hash-only has zero puts; two files + public meta → 3 puts.
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
# NATS — verae-middleware
|
||||
|
||||
| Direction | Address | Peer | Body |
|
||||
|-----------|---------|------|------|
|
||||
| OUT | `verae.zapier.jobs.watch` | job-poller | job watch payload |
|
||||
| IN | `verae.zapier.jobs.events` | waiter | terminal status |
|
||||
| OUT | (via aggregator) `verae.archive.query` | WORM | attached lookup |
|
||||
|
||||
HTTP remains the only Zapier-facing surface.
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
# verae-middleware (HTTP edge)
|
||||
|
||||
**Job:** Public `/zapier/v1/*`. Auth, entitlements, timestamp/wait, hash lookup, objects, receipts. Publishes NATS jobs; waits on events.
|
||||
|
||||
**Expects HTTPS from:** zappier-edge (or Zapier if pointed here).
|
||||
|
||||
**Sends:**
|
||||
- HTTPS to verae-chain-client (`/api/timestamp`, `/api/status`, `/api/verify`)
|
||||
- `verae.zapier.jobs.watch` after create
|
||||
- waits `verae.zapier.jobs.events`
|
||||
- splitter + aggregator when attachments / `includeAttached`
|
||||
|
||||
**Returns to requestor:** 202 `{ jobId }` or wait JSON (completed / pending+jobId).
|
||||
|
||||
**NATS:** never exposed to Zapier.
|
||||
9
packages/docs-master/modules/verae-zapier/SUMMARY.md
Normal file
9
packages/docs-master/modules/verae-zapier/SUMMARY.md
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
# verae-zapier-app
|
||||
|
||||
**Job:** Full Zapier CLI app: timestamp, wait, verify, batch, job status, hash lookup, REST Hook.
|
||||
|
||||
**Expects:** `zmw_` middleware key (today) or zappier key after composition.
|
||||
|
||||
**Sends:** HTTPS to middleware `/zapier/v1/*`.
|
||||
|
||||
**Does not** speak NATS.
|
||||
11
packages/docs-master/modules/zappier/SUMMARY.md
Normal file
11
packages/docs-master/modules/zappier/SUMMARY.md
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
# zappier-edge
|
||||
|
||||
**Job:** Metered commercial API, portal, admin, Stripe. Zapier’s only public HTTPS hop.
|
||||
|
||||
**Expects:** Zapier Platform app with `x-api-key`.
|
||||
|
||||
**Sends:** HTTPS to middleware `/zapier/v1/*` (or local mock `/v1/timestamp`, `/v1/add`, `/v1/hashes`, `/v1/receipts`).
|
||||
|
||||
**Does not** speak NATS or `api.veraetime.net`.
|
||||
|
||||
**Test:** `npm test` (Jest, 177+).
|
||||
11
packages/verae-activate/SUMMARY.md
Normal file
11
packages/verae-activate/SUMMARY.md
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
# verae-activate (Zapier Platform app)
|
||||
|
||||
**Job:** Private Zapier integration you push first. Actions run in Zapier cloud.
|
||||
|
||||
**Expects:** User connection (API key optional).
|
||||
|
||||
**Sends:** nothing if fields blank (Add Numbers / Echo / SHA256 / Create Timestamp local mock). If `api_base` + `api_key`, HTTPS to zappier-edge.
|
||||
|
||||
**Does not** speak NATS.
|
||||
|
||||
**Test:** `npm test`; `zapier-platform validate`.
|
||||
8
packages/verae-archive-aggregator/NATS.md
Normal file
8
packages/verae-archive-aggregator/NATS.md
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
# NATS — verae-archive-aggregator
|
||||
|
||||
| Direction | Address | Peer | Body |
|
||||
|-----------|---------|------|------|
|
||||
| OUT | `verae.archive.query` | all WORM | `{ correlationId, sha256, tenantId, kinds[], traceId }` |
|
||||
| IN | `verae.archive.reply.<correlationId>` | WORM hits | `{ archiveId, sha256, records[] }` |
|
||||
|
||||
Stop collecting at `WAIT_ARCHIVE_MS`. Silent archives are listed in `silent[]`, not errors.
|
||||
11
packages/verae-archive-aggregator/SUMMARY.md
Normal file
11
packages/verae-archive-aggregator/SUMMARY.md
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
# verae-archive-aggregator
|
||||
|
||||
**Job:** If `includeAttached`, fan out archive query and merge replies into the job JSON.
|
||||
|
||||
**Expects:** wait-path flag from middleware-http; replies on `verae.archive.reply.<correlationId>`.
|
||||
|
||||
**Sends:** `verae.archive.query` to all WORM nodes.
|
||||
|
||||
**Timeout:** missing archives omit slices; chain receipt still succeeds.
|
||||
|
||||
**Test:** `npm test` — one silent bloom miss + one hit; merge seal + metadata-attach.
|
||||
8
packages/verae-archive-aggregator/package.json
Normal file
8
packages/verae-archive-aggregator/package.json
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
{
|
||||
"name": "verae-archive-aggregator",
|
||||
"version": "0.1.0",
|
||||
"type": "module",
|
||||
"description": "Aggregate bloom-filtered WORM archive replies into job JSON",
|
||||
"scripts": { "test": "node --test test/*.test.js" },
|
||||
"engines": { "node": ">=20" }
|
||||
}
|
||||
53
packages/verae-archive-aggregator/src/aggregate.js
Normal file
53
packages/verae-archive-aggregator/src/aggregate.js
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
/**
|
||||
* Fan-out archive query and merge replies until timeout.
|
||||
* @module aggregate
|
||||
*/
|
||||
|
||||
/**
|
||||
* @param {object} opts
|
||||
* @param {string} opts.sha256
|
||||
* @param {string} opts.tenantId
|
||||
* @param {string[]} [opts.kinds]
|
||||
* @param {Array<{ archiveId: string, query: (sha: string) => object[]|null }>} opts.archives
|
||||
* @param {number} [opts.timeoutMs=50] unused in in-process mock (archives are sync)
|
||||
* @returns {{ archivesQueried: boolean, archiveReplies: number, records: object[], silent: string[] }}
|
||||
*/
|
||||
export function aggregateAttached({ sha256, tenantId, kinds = ['publicMeta', 'privateMeta', 'file'], archives }) {
|
||||
const records = [];
|
||||
const silent = [];
|
||||
for (const arch of archives) {
|
||||
const hit = arch.query(sha256);
|
||||
if (hit == null) {
|
||||
silent.push(arch.archiveId);
|
||||
continue;
|
||||
}
|
||||
for (const rec of hit) {
|
||||
if (!kinds.length || kinds.includes(rec.kind)) records.push(rec);
|
||||
}
|
||||
}
|
||||
return {
|
||||
sha256,
|
||||
tenantId,
|
||||
archivesQueried: true,
|
||||
archiveReplies: archives.length - silent.length,
|
||||
silent,
|
||||
records,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Merge chain receipts with archive records.
|
||||
* @param {object[]} chainReceipts
|
||||
* @param {object[]} archiveRecords
|
||||
*/
|
||||
export function mergeReceipts(chainReceipts, archiveRecords) {
|
||||
const extra = archiveRecords
|
||||
.filter((r) => r.kind === 'publicMeta' || r.kind === 'privateMeta' || r.kind === 'file')
|
||||
.map((r) => ({
|
||||
kind: r.kind === 'file' ? 'file-attach' : 'metadata-attach',
|
||||
archiveId: r.archiveId,
|
||||
attachedAt: r.storedAt,
|
||||
...r.record,
|
||||
}));
|
||||
return [...chainReceipts, ...extra];
|
||||
}
|
||||
28
packages/verae-archive-aggregator/test/aggregate.test.js
Normal file
28
packages/verae-archive-aggregator/test/aggregate.test.js
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
import { describe, it } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { WormArchive } from '../../verae-archive-worm/src/archive.js';
|
||||
import { aggregateAttached, mergeReceipts } from '../src/aggregate.js';
|
||||
|
||||
describe('aggregateAttached', () => {
|
||||
it('silent miss + one hit', () => {
|
||||
const a = new WormArchive('archive-a');
|
||||
const b = new WormArchive('archive-b');
|
||||
const sha = 'c'.repeat(64);
|
||||
b.put({ sha256: sha, kind: 'publicMeta', record: { publicMetadata: { k: 1 } } });
|
||||
const out = aggregateAttached({ sha256: sha, tenantId: 't1', archives: [a, b] });
|
||||
assert.equal(out.archiveReplies, 1);
|
||||
assert.deepEqual(out.silent, ['archive-a']);
|
||||
assert.equal(out.records[0].record.publicMetadata.k, 1);
|
||||
});
|
||||
|
||||
it('merges seal + attach receipts', () => {
|
||||
const seal = [{ kind: 'seal', timestamp: '2026-01-01T00:00:00Z', certificate: 'x' }];
|
||||
const recs = [
|
||||
{ kind: 'publicMeta', archiveId: 'archive-b', storedAt: '2026-01-02T00:00:00Z', record: { publicMetadata: { n: 2 } } },
|
||||
];
|
||||
const merged = mergeReceipts(seal, recs);
|
||||
assert.equal(merged.length, 2);
|
||||
assert.equal(merged[0].kind, 'seal');
|
||||
assert.equal(merged[1].kind, 'metadata-attach');
|
||||
});
|
||||
});
|
||||
9
packages/verae-archive-worm/NATS.md
Normal file
9
packages/verae-archive-worm/NATS.md
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
# NATS — verae-archive-worm
|
||||
|
||||
| Direction | Address | From / to | Body |
|
||||
|-----------|---------|-----------|------|
|
||||
| IN | `verae.archive.put` | splitter | `{ sha256, tenantId, kind, record, traceId }` |
|
||||
| IN | `verae.archive.query` | aggregator (broadcast) | `{ correlationId, sha256, tenantId, kinds[], traceId }` |
|
||||
| OUT | `verae.archive.reply.<correlationId>` | aggregator | `{ archiveId, sha256, records[], traceId }` **only if bloom.mightHave(sha256)** |
|
||||
|
||||
Subscribe to query **without** a shared queue group so every archive sees every query.
|
||||
14
packages/verae-archive-worm/SUMMARY.md
Normal file
14
packages/verae-archive-worm/SUMMARY.md
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
# verae-archive-worm
|
||||
|
||||
**Job:** Append-only WORM store of metadata and files. Bloom filter: miss → stay silent on NATS.
|
||||
|
||||
**Expects:**
|
||||
- `verae.archive.put` — append record, add sha256 to bloom
|
||||
- `verae.archive.query` — `{ correlationId, sha256, tenantId, kinds[] }` (every node sees this; **not** a shared queue group)
|
||||
|
||||
**Sends:**
|
||||
- `verae.archive.reply.<correlationId>` **only on bloom hit** — `{ archiveId, sha256, records[] }`
|
||||
|
||||
**Does not** write to the blockchain.
|
||||
|
||||
**Test:** `npm test` — miss is `null`; put then query returns records.
|
||||
8
packages/verae-archive-worm/package.json
Normal file
8
packages/verae-archive-worm/package.json
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
{
|
||||
"name": "verae-archive-worm",
|
||||
"version": "0.1.0",
|
||||
"type": "module",
|
||||
"description": "Bloom-filtered WORM archive node for off-chain metadata and files",
|
||||
"scripts": { "test": "node --test test/*.test.js" },
|
||||
"engines": { "node": ">=20" }
|
||||
}
|
||||
40
packages/verae-archive-worm/src/archive.js
Normal file
40
packages/verae-archive-worm/src/archive.js
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
/**
|
||||
* In-process WORM archive: append-only records + bloom.
|
||||
* @module archive
|
||||
*/
|
||||
|
||||
import { Bloom } from './bloom.js';
|
||||
|
||||
export class WormArchive {
|
||||
/**
|
||||
* @param {string} archiveId
|
||||
*/
|
||||
constructor(archiveId) {
|
||||
this.archiveId = archiveId;
|
||||
this.bloom = new Bloom();
|
||||
/** @type {Map<string, object[]>} */
|
||||
this.byHash = new Map();
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {{ sha256: string, kind: string, record: object }} put
|
||||
*/
|
||||
put(put) {
|
||||
const key = String(put.sha256).toLowerCase();
|
||||
const list = this.byHash.get(key) ?? [];
|
||||
list.push({ kind: put.kind, record: put.record, archiveId: this.archiveId, storedAt: new Date().toISOString() });
|
||||
this.byHash.set(key, list);
|
||||
this.bloom.add(key);
|
||||
}
|
||||
|
||||
/**
|
||||
* Bloom miss → null (caller must stay silent). Hit → records (may be empty if false positive).
|
||||
* @param {string} sha256
|
||||
* @returns {object[]|null}
|
||||
*/
|
||||
query(sha256) {
|
||||
const key = String(sha256).toLowerCase();
|
||||
if (!this.bloom.mightHave(key)) return null;
|
||||
return this.byHash.get(key) ?? [];
|
||||
}
|
||||
}
|
||||
46
packages/verae-archive-worm/src/bloom.js
Normal file
46
packages/verae-archive-worm/src/bloom.js
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
/**
|
||||
* Tiny Bloom filter for SHA256 membership (false positives ok).
|
||||
* @module bloom
|
||||
*/
|
||||
|
||||
import { createHash } from 'node:crypto';
|
||||
|
||||
export class Bloom {
|
||||
/**
|
||||
* @param {number} [m=16384] bit count
|
||||
* @param {number} [k=4] hash rounds
|
||||
*/
|
||||
constructor(m = 16384, k = 4) {
|
||||
this.m = m;
|
||||
this.k = k;
|
||||
this.bits = Buffer.alloc(Math.ceil(m / 8));
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {string} s
|
||||
* @returns {number[]}
|
||||
*/
|
||||
hashes(s) {
|
||||
const digest = createHash('sha256').update(String(s)).digest();
|
||||
const out = [];
|
||||
for (let i = 0; i < this.k; i += 1) {
|
||||
const n = digest.readUInt32BE((i * 4) % 28);
|
||||
out.push(n % this.m);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
add(s) {
|
||||
for (const i of this.hashes(s)) {
|
||||
this.bits[i >> 3] |= 1 << (i & 7);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {string} s
|
||||
* @returns {boolean} maybe (never a guaranteed yes)
|
||||
*/
|
||||
mightHave(s) {
|
||||
return this.hashes(s).every((i) => (this.bits[i >> 3] & (1 << (i & 7))) !== 0);
|
||||
}
|
||||
}
|
||||
28
packages/verae-archive-worm/test/archive.test.js
Normal file
28
packages/verae-archive-worm/test/archive.test.js
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
import { describe, it } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { WormArchive } from '../src/archive.js';
|
||||
import { Bloom } from '../src/bloom.js';
|
||||
|
||||
describe('WormArchive + Bloom', () => {
|
||||
it('miss stays silent (null)', () => {
|
||||
const a = new WormArchive('archive-a');
|
||||
assert.equal(a.query('a'.repeat(64)), null);
|
||||
});
|
||||
|
||||
it('put then query returns records', () => {
|
||||
const a = new WormArchive('archive-b');
|
||||
const sha = 'b'.repeat(64);
|
||||
a.put({ sha256: sha, kind: 'publicMeta', record: { publicMetadata: { t: 1 } } });
|
||||
const hit = a.query(sha);
|
||||
assert.ok(Array.isArray(hit));
|
||||
assert.equal(hit[0].kind, 'publicMeta');
|
||||
assert.equal(hit[0].archiveId, 'archive-b');
|
||||
});
|
||||
|
||||
it('bloom reports maybe after add', () => {
|
||||
const b = new Bloom();
|
||||
assert.equal(b.mightHave('hello'), false);
|
||||
b.add('hello');
|
||||
assert.equal(b.mightHave('hello'), true);
|
||||
});
|
||||
});
|
||||
8
packages/verae-request-splitter/NATS.md
Normal file
8
packages/verae-request-splitter/NATS.md
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
# NATS — verae-request-splitter
|
||||
|
||||
| Direction | Address | To | Body |
|
||||
|-----------|---------|-----|------|
|
||||
| OUT | `verae.archive.put` | each WORM | one message per publicMeta / privateMeta / file |
|
||||
| OUT | (in-process) chain | middleware | `{ sha256, hashAlg, data? }` |
|
||||
|
||||
No inbound NATS in the in-process embedding.
|
||||
13
packages/verae-request-splitter/SUMMARY.md
Normal file
13
packages/verae-request-splitter/SUMMARY.md
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
# verae-request-splitter
|
||||
|
||||
**Job:** Peel a request into (1) chain hash and (2) off-chain archive puts.
|
||||
|
||||
**Expects messages from:** middleware-http (in-process call today; later NATS `verae.splitter.in` if split out).
|
||||
|
||||
**Sends messages to:**
|
||||
- Chain path: `{ sha256, hashAlg, data? }`
|
||||
- `verae.archive.put` for each `publicMeta` / `privateMeta` / `file`
|
||||
|
||||
**Does not talk to Zapier or NATS job poller.**
|
||||
|
||||
**Test:** `npm test` — hash-only has zero puts; two files + public meta → 3 puts.
|
||||
10
packages/verae-request-splitter/package.json
Normal file
10
packages/verae-request-splitter/package.json
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
{
|
||||
"name": "verae-request-splitter",
|
||||
"version": "0.1.0",
|
||||
"type": "module",
|
||||
"description": "Split hash/sign vs attachments for chain vs WORM archives",
|
||||
"scripts": {
|
||||
"test": "node --test test/*.test.js"
|
||||
},
|
||||
"engines": { "node": ">=20" }
|
||||
}
|
||||
71
packages/verae-request-splitter/src/splitRequest.js
Normal file
71
packages/verae-request-splitter/src/splitRequest.js
Normal file
|
|
@ -0,0 +1,71 @@
|
|||
/**
|
||||
* Split a Zapier/middleware request into chain vs archive parts.
|
||||
* @module splitRequest
|
||||
*/
|
||||
|
||||
import { createHash } from 'node:crypto';
|
||||
|
||||
/**
|
||||
* @param {string} data
|
||||
* @returns {string}
|
||||
*/
|
||||
export function sha256Hex(data) {
|
||||
return createHash('sha256').update(String(data), 'utf8').digest('hex');
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {object} body
|
||||
* @returns {{
|
||||
* chain: { sha256: string, hashAlg: string, data?: string },
|
||||
* archivePuts: Array<{ kind: string, sha256: string, record: object }>,
|
||||
* includeAttached: boolean
|
||||
* }}
|
||||
*/
|
||||
export function splitRequest(body = {}) {
|
||||
const sha256 = (body.sha256 || (body.data != null ? sha256Hex(body.data) : '')).toLowerCase();
|
||||
if (!sha256) {
|
||||
throw new Error('data or sha256 is required');
|
||||
}
|
||||
const includeAttached = Boolean(body.includeAttached);
|
||||
const archivePuts = [];
|
||||
if (body.publicMetadata && Object.keys(body.publicMetadata).length) {
|
||||
archivePuts.push({
|
||||
kind: 'publicMeta',
|
||||
sha256,
|
||||
record: { publicMetadata: body.publicMetadata },
|
||||
});
|
||||
}
|
||||
if (body.privateMetadata && Object.keys(body.privateMetadata).length) {
|
||||
archivePuts.push({
|
||||
kind: 'privateMeta',
|
||||
sha256,
|
||||
record: { privateMetadata: body.privateMetadata },
|
||||
});
|
||||
}
|
||||
const files = Array.isArray(body.files) ? body.files : [];
|
||||
for (const f of files) {
|
||||
const bytes = f.contentBase64
|
||||
? Buffer.from(String(f.contentBase64), 'base64')
|
||||
: Buffer.from(String(f.data ?? ''), 'utf8');
|
||||
const fileHash = sha256Hex(bytes);
|
||||
archivePuts.push({
|
||||
kind: 'file',
|
||||
sha256,
|
||||
record: {
|
||||
path: f.path || f.filename || 'file.bin',
|
||||
size: bytes.length,
|
||||
contentSha256: fileHash,
|
||||
contentBase64: bytes.toString('base64'),
|
||||
},
|
||||
});
|
||||
}
|
||||
return {
|
||||
chain: {
|
||||
sha256,
|
||||
hashAlg: body.hashAlg || 'SHA256',
|
||||
data: body.data,
|
||||
},
|
||||
archivePuts,
|
||||
includeAttached,
|
||||
};
|
||||
}
|
||||
32
packages/verae-request-splitter/test/splitRequest.test.js
Normal file
32
packages/verae-request-splitter/test/splitRequest.test.js
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
import { describe, it } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { splitRequest, sha256Hex } from '../src/splitRequest.js';
|
||||
|
||||
describe('splitRequest', () => {
|
||||
it('hash-only has no archive puts', () => {
|
||||
const out = splitRequest({ data: 'hello' });
|
||||
assert.equal(out.chain.sha256, sha256Hex('hello'));
|
||||
assert.equal(out.archivePuts.length, 0);
|
||||
assert.equal(out.includeAttached, false);
|
||||
});
|
||||
|
||||
it('splits public metadata and two files onto archive puts', () => {
|
||||
const out = splitRequest({
|
||||
data: 'hello',
|
||||
publicMetadata: { title: 't' },
|
||||
files: [
|
||||
{ path: 'a.txt', data: 'A' },
|
||||
{ path: 'b.txt', data: 'B' },
|
||||
],
|
||||
includeAttached: true,
|
||||
});
|
||||
assert.equal(out.archivePuts.length, 3);
|
||||
assert.equal(out.archivePuts.filter((p) => p.kind === 'file').length, 2);
|
||||
assert.equal(out.includeAttached, true);
|
||||
assert.equal(out.chain.sha256, sha256Hex('hello'));
|
||||
});
|
||||
|
||||
it('throws without data or sha256', () => {
|
||||
assert.throws(() => splitRequest({}), /data or sha256/);
|
||||
});
|
||||
});
|
||||
9
packages/verae-zapier-middleware/NATS.md
Normal file
9
packages/verae-zapier-middleware/NATS.md
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
# NATS — verae-middleware
|
||||
|
||||
| Direction | Address | Peer | Body |
|
||||
|-----------|---------|------|------|
|
||||
| OUT | `verae.zapier.jobs.watch` | job-poller | job watch payload |
|
||||
| IN | `verae.zapier.jobs.events` | waiter | terminal status |
|
||||
| OUT | (via aggregator) `verae.archive.query` | WORM | attached lookup |
|
||||
|
||||
HTTP remains the only Zapier-facing surface.
|
||||
15
packages/verae-zapier-middleware/SUMMARY.md
Normal file
15
packages/verae-zapier-middleware/SUMMARY.md
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
# verae-middleware (HTTP edge)
|
||||
|
||||
**Job:** Public `/zapier/v1/*`. Auth, entitlements, timestamp/wait, hash lookup, objects, receipts. Publishes NATS jobs; waits on events.
|
||||
|
||||
**Expects HTTPS from:** zappier-edge (or Zapier if pointed here).
|
||||
|
||||
**Sends:**
|
||||
- HTTPS to verae-chain-client (`/api/timestamp`, `/api/status`, `/api/verify`)
|
||||
- `verae.zapier.jobs.watch` after create
|
||||
- waits `verae.zapier.jobs.events`
|
||||
- splitter + aggregator when attachments / `includeAttached`
|
||||
|
||||
**Returns to requestor:** 202 `{ jobId }` or wait JSON (completed / pending+jobId).
|
||||
|
||||
**NATS:** never exposed to Zapier.
|
||||
9
packages/verae-zapier/SUMMARY.md
Normal file
9
packages/verae-zapier/SUMMARY.md
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
# verae-zapier-app
|
||||
|
||||
**Job:** Full Zapier CLI app: timestamp, wait, verify, batch, job status, hash lookup, REST Hook.
|
||||
|
||||
**Expects:** `zmw_` middleware key (today) or zappier key after composition.
|
||||
|
||||
**Sends:** HTTPS to middleware `/zapier/v1/*`.
|
||||
|
||||
**Does not** speak NATS.
|
||||
11
packages/zappier/SUMMARY.md
Normal file
11
packages/zappier/SUMMARY.md
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
# zappier-edge
|
||||
|
||||
**Job:** Metered commercial API, portal, admin, Stripe. Zapier’s only public HTTPS hop.
|
||||
|
||||
**Expects:** Zapier Platform app with `x-api-key`.
|
||||
|
||||
**Sends:** HTTPS to middleware `/zapier/v1/*` (or local mock `/v1/timestamp`, `/v1/add`, `/v1/hashes`, `/v1/receipts`).
|
||||
|
||||
**Does not** speak NATS or `api.veraetime.net`.
|
||||
|
||||
**Test:** `npm test` (Jest, 177+).
|
||||
Loading…
Add table
Add a link
Reference in a new issue