diff --git a/packages/dev-overview/00-sources.md b/packages/dev-overview/00-sources.md new file mode 100644 index 0000000..25e9d8d --- /dev/null +++ b/packages/dev-overview/00-sources.md @@ -0,0 +1,55 @@ +# Sources (other documentation that names routes) + +This inventory is compiled from code and from these docs. Do not invent extra `verae.*` names. + +## Master maps + +| Path | What it names | +|------|----------------| +| `packages/docs-master/modules-and-nats.md` | Module catalog, subject table, HTTPS vs NATS | +| `packages/docs-master/MESSAGE-FLOWS.md` | Create/wait/includeAttached/batch Merkle | +| `packages/docs-master/archive-nats.md` | Archive put/query/reply | +| `docs/02-architecture/nats-subjects.md` | `verae.zapier.*` payloads + streams | +| `docs/02-architecture/modules-and-nats.md` | Same catalog as docs-master | +| `docs/02-architecture/overview.md` | Poller / events / webhooks | +| `packages/overview/03-nats-cluster.md` | Cluster URLs, who may connect | +| `packages/overview/06-address-routing.md` | Address pattern + plane mapping | +| `packages/verae-ops/06-linking-services.md` | How services attach | +| `packages/verae-nats-cluster/scripts/ensure-streams.sh` | Live stream create | +| `packages/verae-nats-cluster/conf/nats.conf.tmpl` | `:4222` / `:6222` / `:7422` / monitor `:8222` | +| `packages/verae-nats-cluster/conf/public-edge.conf.tmpl` | Public leaf `:4222`, WS `:14422` | +| `packages/verae-fleet/overlays/ns1/fleet.json` | Enabled vs in-process workers | +| `packages/verae-keep` `units.px-worker.json` | Worm/tree on LXC 510 | +| `packages/verae-zapier-middleware/src/nats/subjects.js` | Product subjects + consumers | +| `packages/zappier-account-balance/src/subjects.js` | Billing subjects | +| `packages/verae-access-authz/src/subjects.js` + `policy.js` | Authz + plane allow lists | +| `packages/zappier/src/billing-nats.ts` | Edge billing client | + +## Per-package `NATS.md` (Forgejo module copies also live under `docs-master/modules/`) + +| Package | Addresses | +|---------|-----------| +| `verae-zapier-middleware` | `jobs.watch` out, `jobs.events` in, archive.query via aggregator | +| `verae-jobs-events` | `jobs.events`; exclusive → `verae.internal.jobs.events` | +| `verae-request-splitter` | `archive.put` | +| `verae-archive-worm` | `archive.put` / `query` / `reply.` | +| `verae-tree-node` | same, `kind: tree` | +| `verae-archive-aggregator` | `query` out, `reply.` in | +| `zappier-account-balance` | `verae.billing.*` | +| `zappier` (edge) | package `NATS.md` is stale (“does not publish”); **code** publishes usage/payment/credit/customer.put when `NATS_URL` is set. Use `docs-master/modules/zappier/NATS.md` | +| `zappier-customer-service` | `statement.get`, `balance.adjust` | +| `zappier-sales-pricing` | `statement.get` | +| `zappier-accounting-export` | `statement.get` | +| `zappier-identity` | `identity.bind` / `lookup` | +| `verae-access-authz` | `authz.check` / `authz.deny` | +| `verae-access-web` | `access.web.in` / `access.web.billing.*` | +| `verae-access-api` | `access.api.*` | +| `verae-access-zapier` | `access.zapier.*` then HTTPS to middleware | +| `verae-access-leaf` | `access.leaf.in` | +| `verae-access-staff` | `access.staff.*` | +| `verae-nats-process` | `verae.example.process.*` (template) | +| `verae-nats-accounts` | policy only (INTERNAL vs LEAF) | +| `verae-zapier-simulator` | in-process replay of product addresses | +| `verae-keep` / `verae-fleet` / `verae-uptime` / `verae-backup` | no product subjects | + +Simulator `src/pipeline.js` and `src/monitors.js` list the same product addresses for traces; they are not live subscribers. diff --git a/packages/dev-overview/01-nats-routes.md b/packages/dev-overview/01-nats-routes.md index 75dd29e..34a4758 100644 --- a/packages/dev-overview/01-nats-routes.md +++ b/packages/dev-overview/01-nats-routes.md @@ -1,76 +1,106 @@ # Designed vs running NATS routes -**Designed** = named in docs (`modules-and-nats.md`, `MESSAGE-FLOWS.md`, per-package `NATS.md`) or in code (`subjects.js`, fleet JSON). +**Designed** = named in docs or in code. **Running** = a process on NS1 is subscribed or publishing it **today**, and (for durable work) a JetStream stream exists. -Cluster name `verae`. Internal URL: three LXC on `vmbr1`. Public edge is a **leaf**, not a second cluster. +Cluster name `verae`. Internal clients use the three LXC on `vmbr1`. Public edge is a **leaf**, not a second cluster. Legend: **JS** = JetStream stream; **RR** = core request-reply; **pub** = core or JS fire-and-forget; **Q** = queue group. +## Transport (TCP / leaf / public) + +These are NATS **server** routes, not `verae.*` subjects. + +| Endpoint | Designed | Running | Who | +|----------|----------|---------|-----| +| `10.10.10.21–23:4222` client | yes | **yes** | middleware, access-*, billing, worm/tree, identity, CS/sales/acct | +| `10.10.10.21–23:6222` cluster routes | yes | **yes** (private `vmbr1` only) | nats-a/b/c to each other | +| `10.10.10.21–23:7422` leaf hub | yes (public-edge) | **yes** | public edge remotes | +| `127.0.0.1:8222` monitor on each LXC | yes | **yes** | operators on the guest | +| `70.88.205.138:4222` public client | yes (token) | **yes** | leaf edge; **not** Zapier | +| `127.0.0.1:14422` WS on edge | yes | **yes** | Caddy → `wss://nats.zapier.georgelambert.org` | +| `127.0.0.1:8229` edge monitor | yes | **yes** | host loopback | +| `:6222` / `:7422` / `:8222` on `vmbr0` | **no** | **must stay closed** | — | +| Host leftover `127.0.0.1:4222` (docker-proxy) + `10.10.10.1:4222` socat | lab leftover | present | **do not** point product `NATS_URL` here | +| MQTT `:1883` | study only | **removed** | — | + +Internal keep/fleet/middleware URL: + +```text +nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222 +``` + +Public (token in `/etc/nats/public-edge.token` on NS1): `nats://@70.88.205.138:4222`. Isolation on the hub is still `$G` + private LAN. nkeys/mTLS sketched in `verae-nats-accounts`, **not** flipped. + ## JetStream streams (running) -Created by `verae-nats-cluster/scripts/ensure-streams.sh` (and middleware `ensureStreams` for the first three). +Created by `verae-nats-cluster/scripts/ensure-streams.sh` (middleware `ensureStreams` also creates the first three). | Stream | Subjects | Replicas | Storage | Who consumes | |--------|----------|----------|---------|--------------| -| `ZAPIER_JOBS` | `verae.zapier.jobs.watch` | 3 | file | job-poller (in middleware process) | -| `ZAPIER_EVENTS` | `verae.zapier.jobs.events` | 3 | file | middleware wait + webhook router; optional exclusive `jobs-events` | -| `ZAPIER_WEBHOOKS` | `verae.zapier.webhooks.deliver` | 3 | file | webhook-deliver (in middleware process) | -| `ZAPIER_USAGE` | `verae.zapier.usage` | 3 | file | **no durable consumer running** (`USAGE_WRITER` is named only) | +| `ZAPIER_JOBS` | `verae.zapier.jobs.watch` | 3 | file | job-poller (in middleware process), Q `job-poller` | +| `ZAPIER_EVENTS` | `verae.zapier.jobs.events` | 3 | file | middleware wait + webhook router (`event-webhook-router`); optional exclusive `jobs-events` | +| `ZAPIER_WEBHOOKS` | `verae.zapier.webhooks.deliver` | 3 | file | webhook-deliver (in middleware), Q `webhook-deliver` | +| `ZAPIER_USAGE` | `verae.zapier.usage` | 3 | file | **no durable consumer** (`USAGE_WRITER` named only) | | `VERAE_ARCHIVE` | `verae.archive.>` | 3 | file | worm + tree on LXC 510 (keep); aggregator **not** fleet-enabled | +Billing is **core request-reply / pub**, not a JetStream stream. + ## Address inventory | Address | Kind | Designed | Running | Notes | |---------|------|----------|---------|-------| | `verae.zapier.jobs.watch` | JS work Q `job-poller` | yes | **yes** | middleware publishes; in-process poller consumes | | `verae.zapier.jobs.events` | JS events | yes | **yes** | poller publishes; wait() subscribes; webhook worker routes | -| `verae.zapier.webhooks.deliver` | JS work Q `webhook-deliver` | yes | **yes** | middleware worker in same Node process as HTTP | -| `verae.zapier.usage` | JS optional | yes | stream **yes**, consumer **no** | middleware entitlement is **local files**, not this subject | +| `verae.zapier.webhooks.deliver` | JS work Q `webhook-deliver` | yes | **yes** | same Node process as HTTP | +| `verae.zapier.usage` | JS optional | yes | stream **yes**, consumer **no** | middleware entitlement is **local files** | | `verae.internal.jobs.events` | core republish | yes (`jobs-events` exclusive) | **partial** | only if `JOBS_EVENTS_EXCLUSIVE=1` | -| `verae.billing.statement.get` | RR | yes | **yes** | account-balance | +| `verae.billing.statement.get` | RR Q `account-balance` | yes | **yes** | portal, CS, sales, accounting, access-web/api | | `verae.billing.balance.get` | RR | yes | **yes** | account-balance | | `verae.billing.balance.adjust` | RR | yes | **yes** | CS credits, portal reload | -| `verae.billing.usage.recorded` | pub | yes | **partial** | designed from zappier-edge meter; confirm publisher on live edge | -| `verae.billing.payment.recorded` | pub | yes | **partial** | portal reload / invoice | -| `verae.billing.credit.applied` | pub | yes | **partial** | CS goodwill | -| `verae.billing.customer.put` | RR/pub | yes (account-balance `subjects.js`) | **unclear** | books write | +| `verae.billing.usage.recorded` | pub | yes | **yes if edge `NATS_URL` set** | `zappier/src/app.ts` `onUsage` → `natsPublish`; account-balance subscribes | +| `verae.billing.payment.recorded` | pub | yes | **yes if `NATS_URL`** | portal reload (`portal.ts`) | +| `verae.billing.credit.applied` | pub | yes | **yes if `NATS_URL`** | admin/CS (`admin.ts`) | +| `verae.billing.customer.put` | pub | yes | **yes if `NATS_URL`** | `ledger.ts`; account-balance applies | | `verae.access.authz.check` | RR Q `access-authz` | yes | **yes** | fleet keepFloor | -| `verae.access.authz.deny` | pub audit | yes | **yes** (on deny) | | -| `verae.access.web.*` | ingress | yes | **yes** | access-web after HTTP | +| `verae.access.authz.deny` | pub audit | yes | **yes** (on deny) | no SIEM sink | +| `verae.access.web.in` / `verae.access.web.billing.*` | ingress | yes | **yes** | mapped after authz, e.g. `verae.access.web.billing.statement.get` | | `verae.access.api.*` | ingress | yes | **yes** | | -| `verae.access.zapier.*` | ingress | yes | **yes** | HTTPS plane, not Zapier NATS | +| `verae.access.zapier.*` | ingress | yes | **yes** | HTTPS plane; Zapier never speaks NATS | | `verae.access.leaf.in` / `verae.access.leaf.*` | ingress | yes | **yes** | no billing | | `verae.access.staff.*` | ingress | yes | **yes** | CS/sales/accounting | -| `verae.identity.bind` | RR Q `identity` | yes | **yes** (fleet enabled) | | +| `verae.identity.bind` | RR Q `identity` | yes | **yes** | fleet | | `verae.identity.lookup` | RR Q `identity` | yes | **yes** | | -| `verae.archive.put` | JS (`VERAE_ARCHIVE`) | yes | **partial** | splitter is **in-process** in middleware; worm/tree on 510 consume | +| `verae.archive.put` | JS (`VERAE_ARCHIVE`) | yes | **partial** | splitter **in-process** in middleware; worm/tree on 510 consume | | `verae.archive.query` | broadcast (no Q) | yes | **partial** | live aggregator hook **TBD** in middleware (simulator has it) | | `verae.archive.reply.` | replies on bloom hit | yes | **partial** | worm/tree on 510 can reply; aggregator not fleet-enabled | | `verae.splitter.in` | future | yes (docs) | **no** | splitter stays a library | -| `verae.example.process.in` / `.out` / `.reply.*` | template | yes | **no** | `verae-nats-process` is a model, not a live worker | +| `verae.example.process.in` / `.out` / `.reply.*` | template | yes | **no** | `verae-nats-process` is a model | | `bench.*` | lab only | n/a | ephemeral | speed studies; not product | +Plane mapping (`verae-access-authz`): internal `verae.billing.statement.get` + plane `web` → ingress `verae.access.web.billing.statement.get`. Planes never publish internal subjects until `authz.check` returns allow. + +## Durable consumers (named in middleware) + +| Name | Stream / subject | Running | +|------|------------------|---------| +| `job-poller` | `ZAPIER_JOBS` / `jobs.watch` | **yes** (in-process) | +| `event-webhook-router` | `ZAPIER_EVENTS` / `jobs.events` | **yes** (in-process) | +| `webhook-deliver` | `ZAPIER_WEBHOOKS` / `webhooks.deliver` | **yes** (in-process) | +| `usage-writer` | `ZAPIER_USAGE` / `verae.zapier.usage` | **no** | +| `jobs-events-exclusive` | `ZAPIER_EVENTS` | **no** unless `JOBS_EVENTS_EXCLUSIVE=1` | + ## Fleet overlay vs in-process -`overlays/ns1/fleet.json` (2026-09-14): +`packages/verae-fleet/overlays/ns1/fleet.json` (2026-09-14): | Service | fleet `enabled` | Actual lab | |---------|-----------------|------------| -| middleware-http | true | **running**; also starts job-poller + webhook workers in-process when `NATS_ENABLED` | -| job-poller | **false** | covered by middleware, not a second fleet spawn | +| middleware-http | true | **running**; starts job-poller + webhook workers in-process when `NATS_ENABLED` | +| job-poller | **false** | covered by middleware; do not spawn a second copy | | webhook-deliver | **false** | same | | archive-aggregator | **false** | **not** running as a fleet worker | -| archive-worm / tree-node | **false** on host fleet | **running on LXC 510** via `verae-keep` `units.px-worker.json` (1 worm + 1 tree) | -| account-balance, access-*, identity, jobs-events, CS/sales/acct, staff-iam/session | true | keepFloor | +| archive-worm / tree-node | **false** on host fleet | **running on LXC 510** via keep `units.px-worker.json` (1 worm + 1 tree) | +| account-balance, access-*, identity, jobs-events, CS/sales/acct, staff-iam/session, zappier-edge | true keepFloor | keepFloor | | zapier-simulator | true, keepFloor false | optional | | nats | managed false | 3-node cluster + leftover host loopback | - -## Docs that name routes (do not invent extras) - -- `packages/docs-master/modules-and-nats.md`, `MESSAGE-FLOWS.md` -- `packages/overview/03-nats-cluster.md`, `06-address-routing.md` -- `packages/verae-ops/06-linking-services.md` -- Each package `NATS.md` -- `verae-nats-cluster/scripts/ensure-streams.sh` -- Public edge: `verae-nats-cluster/conf/public-edge.conf.tmpl` (leaf, not a new subject family) diff --git a/packages/dev-overview/02-endpoint-io.md b/packages/dev-overview/02-endpoint-io.md index 3cf8b6e..38d1e5f 100644 --- a/packages/dev-overview/02-endpoint-io.md +++ b/packages/dev-overview/02-endpoint-io.md @@ -2,13 +2,13 @@ Each product NATS address: **what it is for**, **who sends**, **who receives**, **how to attach**. -Connect with a **long-lived** client: +Connect with a **long-lived** client (reconnect-per-message is the anti-pattern): ```text NATS_URL=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222 ``` -Public edge (token in `/etc/nats/public-edge.token` on NS1): `nats://@70.88.205.138:4222`. Do not put that URL in Zapier. +Public edge (token in `/etc/nats/public-edge.token` on NS1): `nats://@70.88.205.138:4222`. Do not put that URL in Zapier or in Verae. --- @@ -33,10 +33,12 @@ Payload: `tenantId`, `veraeUserId`, `jobId`, `tokenRef`, `enqueuedAt`, `attempt` |--|--| | For | Terminal outcomes: `timestamp.completed` / `failed` / `timeout` | | In from | job-poller after Verae status is not `pending` | -| Out to | (1) middleware `waitForJobEvent` (wait HTTP); (2) webhook router → `webhooks.deliver`; (3) optional `verae-jobs-events` mailbox | +| Out to | (1) middleware `waitForJobEvent` (wait HTTP); (2) webhook router → `webhooks.deliver`; (3) optional `verae-jobs-events` mailbox `:3030` | If `JOBS_EVENTS_EXCLUSIVE=1`, only `verae-jobs-events` is the durable consumer and republishes `verae.internal.jobs.events`. Default lab: middleware wait + webhook worker share the stream. +Payload: `event`, `tenantId`, `jobId`, `status`, `traceId`, `emittedAt`. + --- ## `verae.zapier.webhooks.deliver` @@ -47,7 +49,7 @@ If `JOBS_EVENTS_EXCLUSIVE=1`, only `verae-jobs-events` is the durable consumer a | In from | webhook worker on completed/failed events | | Out to | Zapier cloud **HTTPS**, never NATS | -Slow consumer: this is the flood/backlog path. Keep replica floors on the worker. +Payload: `hookId`, `tenantId`, `targetUrl`, `event`, `payload`, `attempt`, `traceId`. Slow consumer: keep replica floors on the worker if you split it out of middleware. --- @@ -55,44 +57,49 @@ Slow consumer: this is the flood/backlog path. Keep replica floors on the worker | | | |--|--| -| For | Optional meter of Zap actions on NATS | +| For | Optional meter of Zap actions on NATS (JetStream) | | In from | designed: middleware / edge | | Out to | designed: `USAGE_WRITER` → books | | Running | **stream exists; no writer; middleware uses local usage files** | -Finish: publish here from `recordUsage`, consume into `verae.billing.usage.recorded`. +Prepaid meter that **is** wired uses `verae.billing.usage.recorded` (core pub), not this stream. Finish: either publish here from `recordUsage` or drop the stream. --- -## Billing (`verae.billing.*`) — account-balance SoT +## Billing (`verae.billing.*`) — account-balance source of truth -Request-reply unless noted. Queue on account-balance. **Not** JetStream (except you could mirror later). +Request-reply unless noted. Queue `account-balance`. **Not** JetStream. | Address | For | In from | Out to | |---------|-----|---------|--------| -| `statement.get` | prepaid, credits, usage, payments | portal, CS, sales, admin, access-web/api | reply to requester | +| `statement.get` | prepaid, credits, usage, payments | portal, CS, sales, accounting, access-web/api | reply to requester | | `balance.get` | cents only | internal | reply | | `balance.adjust` | CS credit or portal reload | CS, access-web (reload kind only) | books + reply | -| `usage.recorded` | meter pub | zappier-edge (designed) | account-balance subscriber | +| `usage.recorded` | meter pub | zappier-edge `meter()` `onUsage` when `NATS_URL` set | account-balance subscriber (debits prepaid) | | `payment.recorded` | reload/invoice pub | portal | books | -| `credit.applied` | goodwill pub | CS | books | -| `customer.put` | upsert customer | edge/IAM | books | +| `credit.applied` | goodwill pub | CS/admin | books | +| `customer.put` | upsert customer | edge ledger | books | Authz **must** pass `verae.access.authz.check` before these from a public plane. Leaf plane **cannot** call billing (policy). +Middleware `recordUsage` still writes **local tenant files** and does **not** publish `usage.recorded`. Two meters until that is unified. + --- ## Access planes | Address | For | In | Out | |---------|-----|----|-----| -| `verae.access.authz.check` | allow/deny `{ plane, subject, principal }` | every plane | `{ allow, reason }`; deny also pubs `authz.deny` | -| `verae.access.web.*` | browser portal | access-web HTTP | billing statement/reload after authz | +| `verae.access.authz.check` | allow/deny `{ plane, subject, principal }` | every plane + edge `billing-nats.ts` | `{ allow, reason }`; deny also pubs `authz.deny` | +| `verae.access.authz.deny` | audit | authz on deny | **nowhere yet** (no SIEM) | +| `verae.access.web.in` / `verae.access.web.billing.*` | browser portal | access-web HTTP | `statement.get` / `balance.adjust` kind=reload after authz | | `verae.access.api.*` | customer `x-api-key` | access-api | statement; HTTPS to middleware for jobs | -| `verae.access.zapier.*` | Zapier HTTPS only | access-zapier | jobs.watch **via HTTP to middleware**, not Zapier NATS | -| `verae.access.leaf.*` | S2S | access-leaf | archive + jobs; **never billing** | +| `verae.access.zapier.*` | Zapier HTTPS only | access-zapier | jobs.watch **via HTTP to middleware**, not Zapier NATS; usage.recorded allowed by policy | +| `verae.access.leaf.in` | S2S `{ token, target, payload }` | remote leaf | archive + jobs after authz; **never billing** | | `verae.access.staff.*` | CS/sales/accounting | access-staff | billing after authz | +Ingress mapping: `accessAddress(plane, 'verae.billing.statement.get')` → `verae.access..billing.statement.get`. + --- ## Identity @@ -110,11 +117,23 @@ Queue `identity`. Fleet keepFloor. | Address | For | In | Out | |---------|-----|----|-----| -| `verae.archive.put` | WORM/tree write (hash, meta, file meta, merkle leaf) | splitter (in-process from middleware today) | every worm/tree (JS) | +| `verae.archive.put` | WORM/tree write (`kind`: publicMeta / privateMeta / file / tree) | splitter (in-process from middleware today) | every worm/tree (JS) | | `verae.archive.query` | broadcast lookup | aggregator | **all** worm/tree, **no** queue group | | `verae.archive.reply.` | bloom **hit only** | worm/tree | aggregator | -Silence = miss. Chain seal does not wait on archive. +Silence = miss. Chain seal does not wait on archive. Bytes never go to the chain. + +--- + +## Template / unused + +| Address | For | In | Out | Running | +|---------|-----|----|-----|---------| +| `verae.splitter.in` | future split-out splitter process | middleware | splitter | **no** | +| `verae.example.process.in` | template worker | any | `example-process` Q | **no** | +| `verae.example.process.out` | template fan-out | template | peers | **no** | +| `verae.example.process.reply.` | template reply | template | requester | **no** | +| `verae.internal.jobs.events` | exclusive mailbox republish | jobs-events | wait/webhook | **no** unless env | --- @@ -132,4 +151,4 @@ Silence = miss. Chain seal does not wait on archive. Do **not** point Verae at `NATS_URL`. Do **not** put NATS in Zapier env. A new worker that must see seals uses `jobs.events` or HTTP `GET /zapier/v1/status/{jobId}`. -Template for a new addressed process: clone `verae-nats-process`, rename `verae.example.process.*`. +Template for a new addressed process: clone `verae-nats-process`, rename `verae.example.process.*`, add the subject to `verae-access-authz` policy for the plane that may call it. diff --git a/packages/dev-overview/03-zapier-verae-path.md b/packages/dev-overview/03-zapier-verae-path.md index cb84fe4..a0c16de 100644 --- a/packages/dev-overview/03-zapier-verae-path.md +++ b/packages/dev-overview/03-zapier-verae-path.md @@ -30,14 +30,15 @@ Zapier never has `NATS_URL`. Verae never has `NATS_URL`. ## 1. Submit (async create) 1. Zap action **Create Timestamp** (or curl) `POST https://zap.zapier.georgelambert.org/zapier/v1/timestamp` with portal API key `zmw_…` (not a Verae JWT). -2. access-zapier / edge meters (designed: `verae.billing.usage.recorded`) and forwards to middleware. -3. Middleware `checkEntitlement` (local plan limits today). -4. `veraeClient.createTimestamp`: +2. access-zapier / edge: authz, then `meter()` records cents; if `NATS_URL` is set, `natsPublish(verae.billing.usage.recorded)`. +3. Forwards to middleware `POST /zapier/v1/timestamp`. +4. Middleware `checkEntitlement` (local plan limits today). +5. `veraeClient.createTimestamp`: - **Mock:** in-process job, fake `jobId`. - **Live:** `POST https://api.veraetime.net/api/timestamp` with JWT → **202** `{ jobId }`. -5. Splitter (in-process): hash vs attachments; optional `verae.archive.put` (bytes never go to chain). -6. `enqueueWatch` → JetStream **`verae.zapier.jobs.watch`**. -7. HTTP **202** `{ jobId, sha256, existing }` back to Zapier immediately. +6. Splitter (in-process): hash vs attachments; optional `verae.archive.put` (bytes never go to chain). +7. `enqueueWatch` → JetStream **`verae.zapier.jobs.watch`**. +8. HTTP **202** `{ jobId, sha256, existing }` back to Zapier immediately. Same SHA-256 is idempotent (`existing: true`). @@ -47,7 +48,7 @@ The poller is **in the middleware Node process** when `NATS_ENABLED=true` (`star 1. Durable consumer `job-poller` on `ZAPIER_JOBS` / `verae.zapier.jobs.watch`. 2. Resolve `tokenRef` → tenant Verae password → `POST` login → JWT (stays in middleware). -3. `GET /api/status/{jobId}` (mock or live). +3. `GET /api/status/{jobId}` (mock or live timestamp server). 4. `pending` → `nak` after `intervalMs` (retry). 5. `completed` → publish `jobs.events` `{ event: timestamp.completed, status, jobId, tenantId, traceId }`. 6. `failed` → `timestamp.failed`. Too many attempts → `timestamp.timeout`. @@ -76,6 +77,20 @@ Live OpenAPI: poll until `completed` / `failed`. Hash lookup is **501** on live `POST /zapier/v1/verify` → middleware → Verae `POST /api/verify`. No job watch. +## 6. Batch Merkle (designed; same bus) + +From `MESSAGE-FLOWS.md`: + +1. `POST /v1/timestamp/batch` — one item per line. +2. Middleware hashes leaves, builds Merkle tree. +3. Chain seals **only the root** (`POST /api/timestamp` on the root). +4. Each leaf proof → `verae.archive.put` `kind: tree`. +5. Central hash lookup of a leaf misses unless `includeTree=true`. +6. Aggregator broadcasts `verae.archive.query` `kinds=["tree"]`. +7. Tree node bloom hit → `archive.reply.` with proof. + +Live wait path does not yet call the aggregator, so tree-leaf receipts are not merged on NS1. + ## Connect live `api.veraetime.net` Still blocked on operator bind credentials. When ready: @@ -93,5 +108,18 @@ Then: working Verae login, one live seal, confirm 202/status/verify, map receipt ## Billing on this path -Designed: edge meters `usage.recorded` → account-balance; 402 if prepaid empty. -Running: middleware local usage files; account-balance NATS is up for portal/CS. **Wire edge meter → `verae.billing.usage.recorded` and 402 from books** is still a finish item. +| Hop | Designed | Running | +|-----|----------|---------| +| Edge meter | `verae.billing.usage.recorded` | **coded**; fires when edge has `NATS_URL` | +| Account-balance | subscribe + debit prepaid; 402 if empty | **subscriber running**; 402 from books not proven on every Zap | +| Middleware quota | local files via `recordUsage` | **running**; not NATS | +| `verae.zapier.usage` | optional JS meter | stream only | + +## Connect a new internal process + +```bash +export NATS_URL=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222 +# clone verae-nats-process, rename subjects, register in verae-fleet +``` + +Jobs stay: HTTP in from Zapier → HTTPS to Verae → NATS watch/events → HTTP back to Zapier. diff --git a/packages/dev-overview/04-gaps-and-finish.md b/packages/dev-overview/04-gaps-and-finish.md index b32ea72..4d3ad55 100644 --- a/packages/dev-overview/04-gaps-and-finish.md +++ b/packages/dev-overview/04-gaps-and-finish.md @@ -16,17 +16,17 @@ Modules and addresses that are designed but **not fully connected, not running a | Module | Running? | NATS | Gap | |--------|----------|------|-----| | middleware HTTP + in-process job-poller + webhook-deliver | **yes** | watch/events/deliver | live Verae + Zapier push | -| account-balance | **yes** | billing RR | edge may not publish `usage.recorded` on every Zap | +| account-balance | **yes** | billing RR + `usage.recorded` sub | prove 402 from books on a Zap when prepaid is 0 | +| zappier-edge | **yes** | `usage.recorded` / `payment.recorded` / `credit.applied` / `customer.put` **if `NATS_URL`** | confirm live process env; Stripe live | | access-* + authz | **yes** | authz + ingress | — | | identity | **yes** (fleet) | bind/lookup | confirm portal always binds `veraeUserId` | | jobs-events mailbox `:3030` | **yes** (fleet) | events counter | exclusive consumer off unless `JOBS_EVENTS_EXCLUSIVE=1` | | CS / sales / accounting | **yes** | statement/adjust | staff UX | -| zappier-edge | **yes** | billing after authz | meter → NATS; Stripe live | | archive-worm + tree-node | **yes on LXC 510** (keep) | `archive.put/query/reply` | only **1+1** copies; fleet overlay `enabled: false` for extra replicas | | archive-aggregator | **no** as fleet worker | query/reply | enable overlay **or** finish in-process hook (`includeAttached` TBD in middleware) | | request-splitter | library in middleware | `archive.put` | `verae.splitter.in` not used | | `ZAPIER_USAGE` / `USAGE_WRITER` | stream yes | `verae.zapier.usage` | no consumer; middleware usage is local files | -| fleet job-poller / webhook-deliver **extra** copies | overlay **false** | same subjects | only enable if you want SSH-spread replicas **instead of** in-process (do not double-consume without queue groups — in-process already uses queue names) | +| fleet job-poller / webhook-deliver **extra** copies | overlay **false** | same subjects | only enable if you want SSH-spread replicas **instead of** in-process (same queue names) | ## Designed, not a live worker @@ -36,32 +36,45 @@ Modules and addresses that are designed but **not fully connected, not running a | `verae-activate` | none (HTTPS Zapier) | optional tiny app; needs Zapier login | | `verae-zapier` (full app) | HTTPS | push after login | | simulator | in-process fake NATS | local `:3847`; not production | +| `verae-nats-accounts` | nkeys policy | not flipped | -## Billing triggers and event logging (aux) +## Aux (no product subjects, or HTTP only) -| Signal | Designed | Finish | -|--------|----------|--------| -| Timestamp/verify meter | `verae.billing.usage.recorded` + optional `verae.zapier.usage` | Publish from edge/middleware `recordUsage`; consume into account-balance; 402 from books not only local quota | -| Portal reload | `payment.recorded` + `balance.adjust` | confirm both fire on Stripe/test reload | -| CS credit | `balance.adjust` + `credit.applied` | running for adjust; confirm applied pub | -| Authz deny audit | `verae.access.authz.deny` | running on deny; no SIEM sink | -| Job lifecycle log | `jobs.events` | running; mailbox counts only — add structured log/export if needed | -| Archive put/query | `VERAE_ARCHIVE` | 510 worm/tree; aggregator not in wait path | -| Keep/fleet | no NATS | HTTP health only — fine | -| Uptime | HTTPS GET | no NATS | -| Backup | disk/JS snapshot | no live event subject | -| Public NATS edge | leaf, token | **do not** use from Zapier; rotate token; consider nkeys | +| Module | NATS | Notes | +|--------|------|-------| +| `verae-keep` | none | HTTP health; restarts units | +| `verae-fleet` | none | catalog + keepFloor | +| `verae-uptime` | none | off-box HTTPS GET of public doors | +| `verae-backup` | none | disk / JetStream snapshot; no live event subject | +| `verae-bootstrap` / `verae-deploy` / `verae-proxmox` / `verae-ops` | none | install/link | +| staff-iam / staff-session / staff-ui | none | HTTP cookies | +| public NATS edge | leaf, token | **do not** use from Zapier; rotate token; later nkeys LEAF vs INTERNAL | + +## Billing triggers and event logging + +| Signal | Designed | Running | Finish | +|--------|----------|---------|--------| +| Timestamp/verify meter (edge) | `verae.billing.usage.recorded` | **code path exists** (`zappier/src/app.ts` `onUsage`) | Confirm `NATS_URL` on live edge; 402 from account-balance when prepaid empty | +| Middleware quota | local files `recordUsage` | **yes** | Either publish `usage.recorded` too or drop duplicate quota | +| Optional JS meter | `verae.zapier.usage` + `USAGE_WRITER` | stream only | Wire or delete | +| Portal reload | `payment.recorded` + `balance.adjust` | coded | confirm both fire on Stripe/test reload | +| CS credit | `balance.adjust` + `credit.applied` | coded | confirm applied pub on live | +| Customer upsert | `customer.put` | coded | — | +| Authz deny audit | `verae.access.authz.deny` | running on deny | **no SIEM sink** | +| Job lifecycle log | `jobs.events` | running | mailbox counts only — add structured log/export if needed | +| Archive put/query | `VERAE_ARCHIVE` | 510 worm/tree | aggregator not in wait path | +| Public edge | leaf | running | rotate token; do not expose `:6222` | ## Ordered finish for message routing 1. **Do not double-start** job-poller: either in-process **or** fleet copies, same queue `job-poller`. -2. Wire **usage**: `recordUsage` → `verae.billing.usage.recorded` (and/or `verae.zapier.usage` + `USAGE_WRITER`). +2. Confirm edge **`NATS_URL`** so `usage.recorded` actually lands; then 402 from books. Unify or drop middleware local `recordUsage`. 3. Enable **archive-aggregator** (or in-process equivalent) so `includeAttached` and tree proofs work on wait/status. 4. Raise worm/tree **replica floors** on 510/more CTs if archive HA is required (fleet overlay min 3). 5. Operator: Zapier `push` + `MOCK_VERAE=false`. 6. Prove: Zap create → `jobs.watch` → live `GET /api/status` → `jobs.events` → wait body + REST Hook. 7. nkeys/mTLS after every client has creds; then lock down the public edge to accounts (LEAF vs INTERNAL). -8. Optional: exclusive `jobs-events` mailbox; SIEM on `authz.deny` and `jobs.events`; `verae.splitter.in` if splitter is split out. +8. Optional: exclusive `jobs-events` mailbox; SIEM on `authz.deny` and `jobs.events`; `verae.splitter.in` if splitter is split out; `USAGE_WRITER` or delete `ZAPIER_USAGE`. ## What is already enough for a mock Zap diff --git a/packages/dev-overview/NATS.md b/packages/dev-overview/NATS.md index 731740f..3af73fb 100644 --- a/packages/dev-overview/NATS.md +++ b/packages/dev-overview/NATS.md @@ -1,9 +1,9 @@ # NATS — dev-overview -This repo does not subscribe. It documents every `verae.*` address. +This repo does not subscribe. It documents every `verae.*` address and the cluster TCP/leaf ports. -Cluster (running): `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` on `vmbr1`. +Cluster (running): `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` on `vmbr1`. Cluster routes `:6222`. Leaf hubs `:7422`. Monitor `127.0.0.1:8222`. Public client edge (token): `nats://70.88.205.138:4222` and `wss://nats.zapier.georgelambert.org`. Zapier cloud still uses **HTTPS only**. -See [01-nats-routes.md](01-nats-routes.md). +See [01-nats-routes.md](01-nats-routes.md) and [02-endpoint-io.md](02-endpoint-io.md). diff --git a/packages/dev-overview/README.html b/packages/dev-overview/README.html new file mode 100644 index 0000000..2f1812e --- /dev/null +++ b/packages/dev-overview/README.html @@ -0,0 +1,1988 @@ + + + + + + + dev-overview — NATS routes + + + +
+

dev-overview — NATS routes

+
+

dev-overview

+

Developer map of the Verae Time × Zapier NATS message +fabric: what was designed, what is actually running on NS1, how +a Zapier timestamp job reaches api.veraetime.net (or the +mock) and comes back, and what is still unwired.

+

Forgejo: +https://git.georgelambert.org/marchon/dev-overview
+SSH: +ssh://git@git.georgelambert.org:2223/marchon/dev-overview.git
+Catalog: +https://zapier.georgelambert.org/packages/dev-overview/
+Date: 2026-09-14

+

Zapier cloud never connects to NATS. It talks HTTPS +to https://zap.zapier.georgelambert.org (access-zapier) +then middleware. api.veraetime.net is HTTPS +only — middleware is the NATS client.

+

Contents

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
DocWhat
00-sources.mdEvery other doc that names a NATS address or TCP route
01-nats-routes.mdTransport ports + every verae.* address: designed vs +running
02-endpoint-io.mdPer-address: purpose, inbound from, outbound to, how to connect +Verae API
03-zapier-verae-path.mdZapier → middleware → Verae timestamp server → poll → events → +Zapier
04-gaps-and-finish.mdModules not connected; billing/event/aux; ordered finish list
+

Cluster (running)

+
nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+

JetStream streams (replicas=3, file): ZAPIER_JOBS, +ZAPIER_EVENTS, ZAPIER_WEBHOOKS, +ZAPIER_USAGE, VERAE_ARCHIVE.

+

Public token-gated client edge (not cluster routes): +nats://<token>@70.88.205.138:4222, +wss://nats.zapier.georgelambert.org. Do not put this in +Zapier.

+

Sources (other +documentation that names routes)

+

This inventory is compiled from code and from these docs. Do not +invent extra verae.* names.

+

Master maps

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
PathWhat it names
packages/docs-master/modules-and-nats.mdModule catalog, subject table, HTTPS vs NATS
packages/docs-master/MESSAGE-FLOWS.mdCreate/wait/includeAttached/batch Merkle
packages/docs-master/archive-nats.mdArchive put/query/reply
docs/02-architecture/nats-subjects.mdverae.zapier.* payloads + streams
docs/02-architecture/modules-and-nats.mdSame catalog as docs-master
docs/02-architecture/overview.mdPoller / events / webhooks
packages/overview/03-nats-cluster.mdCluster URLs, who may connect
packages/overview/06-address-routing.mdAddress pattern + plane mapping
packages/verae-ops/06-linking-services.mdHow services attach
packages/verae-nats-cluster/scripts/ensure-streams.shLive stream create
packages/verae-nats-cluster/conf/nats.conf.tmpl:4222 / :6222 / :7422 / +monitor :8222
packages/verae-nats-cluster/conf/public-edge.conf.tmplPublic leaf :4222, WS :14422
packages/verae-fleet/overlays/ns1/fleet.jsonEnabled vs in-process workers
packages/verae-keep +units.px-worker.jsonWorm/tree on LXC 510
packages/verae-zapier-middleware/src/nats/subjects.jsProduct subjects + consumers
packages/zappier-account-balance/src/subjects.jsBilling subjects
packages/verae-access-authz/src/subjects.js + +policy.jsAuthz + plane allow lists
packages/zappier/src/billing-nats.tsEdge billing client
+

Per-package +NATS.md (Forgejo module copies also live under +docs-master/modules/)

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
PackageAddresses
verae-zapier-middlewarejobs.watch out, jobs.events in, +archive.query via aggregator
verae-jobs-eventsjobs.events; exclusive → +verae.internal.jobs.events
verae-request-splitterarchive.put
verae-archive-wormarchive.put / query / +reply.<id>
verae-tree-nodesame, kind: tree
verae-archive-aggregatorquery out, reply.<id> in
zappier-account-balanceverae.billing.*
zappier (edge)package NATS.md is stale (“does not publish”); +code publishes usage/payment/credit/customer.put when +NATS_URL is set. Use +docs-master/modules/zappier/NATS.md
zappier-customer-servicestatement.get, balance.adjust
zappier-sales-pricingstatement.get
zappier-accounting-exportstatement.get
zappier-identityidentity.bind / lookup
verae-access-authzauthz.check / authz.deny
verae-access-webaccess.web.in / access.web.billing.*
verae-access-apiaccess.api.*
verae-access-zapieraccess.zapier.* then HTTPS to middleware
verae-access-leafaccess.leaf.in
verae-access-staffaccess.staff.*
verae-nats-processverae.example.process.* (template)
verae-nats-accountspolicy only (INTERNAL vs LEAF)
verae-zapier-simulatorin-process replay of product addresses
verae-keep / verae-fleet / +verae-uptime / verae-backupno product subjects
+

Simulator src/pipeline.js and +src/monitors.js list the same product addresses for traces; +they are not live subscribers.

+

Designed vs running NATS +routes

+

Designed = named in docs or in code.
+Running = a process on NS1 is subscribed or publishing +it today, and (for durable work) a JetStream stream +exists.

+

Cluster name verae. Internal clients use the three LXC +on vmbr1. Public edge is a leaf, not a +second cluster.

+

Legend: JS = JetStream stream; RR = +core request-reply; pub = core or JS fire-and-forget; +Q = queue group.

+

Transport (TCP / leaf / public)

+

These are NATS server routes, not +verae.* subjects.

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
EndpointDesignedRunningWho
10.10.10.21–23:4222 clientyesyesmiddleware, access-*, billing, worm/tree, identity, +CS/sales/acct
10.10.10.21–23:6222 cluster routesyesyes (private vmbr1 only)nats-a/b/c to each other
10.10.10.21–23:7422 leaf hubyes (public-edge)yespublic edge remotes
127.0.0.1:8222 monitor on each LXCyesyesoperators on the guest
70.88.205.138:4222 public clientyes (token)yesleaf edge; not Zapier
127.0.0.1:14422 WS on edgeyesyesCaddy → wss://nats.zapier.georgelambert.org
127.0.0.1:8229 edge monitoryesyeshost loopback
:6222 / :7422 / :8222 on +vmbr0nomust stay closed—
Host leftover 127.0.0.1:4222 (docker-proxy) + +10.10.10.1:4222 socatlab leftoverpresentdo not point product NATS_URL +here
MQTT :1883study onlyremoved—
+

Internal keep/fleet/middleware URL:

+
nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+

Public (token in /etc/nats/public-edge.token on NS1): +nats://<token>@70.88.205.138:4222. Isolation on the +hub is still $G + private LAN. nkeys/mTLS sketched in +verae-nats-accounts, not flipped.

+

JetStream streams (running)

+

Created by verae-nats-cluster/scripts/ensure-streams.sh +(middleware ensureStreams also creates the first +three).

+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
StreamSubjectsReplicasStorageWho consumes
ZAPIER_JOBSverae.zapier.jobs.watch3filejob-poller (in middleware process), Q job-poller
ZAPIER_EVENTSverae.zapier.jobs.events3filemiddleware wait + webhook router +(event-webhook-router); optional exclusive +jobs-events
ZAPIER_WEBHOOKSverae.zapier.webhooks.deliver3filewebhook-deliver (in middleware), Q webhook-deliver
ZAPIER_USAGEverae.zapier.usage3fileno durable consumer (USAGE_WRITER +named only)
VERAE_ARCHIVEverae.archive.>3fileworm + tree on LXC 510 (keep); aggregator not +fleet-enabled
+

Billing is core request-reply / pub, not a JetStream +stream.

+

Address inventory

+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
AddressKindDesignedRunningNotes
verae.zapier.jobs.watchJS work Q job-polleryesyesmiddleware publishes; in-process poller consumes
verae.zapier.jobs.eventsJS eventsyesyespoller publishes; wait() subscribes; webhook worker routes
verae.zapier.webhooks.deliverJS work Q webhook-deliveryesyessame Node process as HTTP
verae.zapier.usageJS optionalyesstream yes, consumer nomiddleware entitlement is local files
verae.internal.jobs.eventscore republishyes (jobs-events exclusive)partialonly if JOBS_EVENTS_EXCLUSIVE=1
verae.billing.statement.getRR Q account-balanceyesyesportal, CS, sales, accounting, access-web/api
verae.billing.balance.getRRyesyesaccount-balance
verae.billing.balance.adjustRRyesyesCS credits, portal reload
verae.billing.usage.recordedpubyesyes if edge NATS_URL setzappier/src/app.ts onUsage → +natsPublish; account-balance subscribes
verae.billing.payment.recordedpubyesyes if NATS_URLportal reload (portal.ts)
verae.billing.credit.appliedpubyesyes if NATS_URLadmin/CS (admin.ts)
verae.billing.customer.putpubyesyes if NATS_URLledger.ts; account-balance applies
verae.access.authz.checkRR Q access-authzyesyesfleet keepFloor
verae.access.authz.denypub audityesyes (on deny)no SIEM sink
verae.access.web.in / +verae.access.web.billing.*ingressyesyesmapped after authz, +e.g. verae.access.web.billing.statement.get
verae.access.api.*ingressyesyes
verae.access.zapier.*ingressyesyesHTTPS plane; Zapier never speaks NATS
verae.access.leaf.in / +verae.access.leaf.*ingressyesyesno billing
verae.access.staff.*ingressyesyesCS/sales/accounting
verae.identity.bindRR Q identityyesyesfleet
verae.identity.lookupRR Q identityyesyes
verae.archive.putJS (VERAE_ARCHIVE)yespartialsplitter in-process in middleware; worm/tree on 510 +consume
verae.archive.querybroadcast (no Q)yespartiallive aggregator hook TBD in middleware (simulator +has it)
verae.archive.reply.<id>replies on bloom hityespartialworm/tree on 510 can reply; aggregator not fleet-enabled
verae.splitter.infutureyes (docs)nosplitter stays a library
verae.example.process.in / .out / +.reply.*templateyesnoverae-nats-process is a model
bench.*lab onlyn/aephemeralspeed studies; not product
+

Plane mapping (verae-access-authz): internal +verae.billing.statement.get + plane web → +ingress verae.access.web.billing.statement.get. Planes +never publish internal subjects until authz.check returns +allow.

+

Durable consumers (named +in middleware)

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameStream / subjectRunning
job-pollerZAPIER_JOBS / jobs.watchyes (in-process)
event-webhook-routerZAPIER_EVENTS / jobs.eventsyes (in-process)
webhook-deliverZAPIER_WEBHOOKS / webhooks.deliveryes (in-process)
usage-writerZAPIER_USAGE / verae.zapier.usageno
jobs-events-exclusiveZAPIER_EVENTSno unless JOBS_EVENTS_EXCLUSIVE=1
+

Fleet overlay vs in-process

+

packages/verae-fleet/overlays/ns1/fleet.json +(2026-09-14):

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Servicefleet enabledActual lab
middleware-httptruerunning; starts job-poller + webhook workers +in-process when NATS_ENABLED
job-pollerfalsecovered by middleware; do not spawn a second copy
webhook-deliverfalsesame
archive-aggregatorfalsenot running as a fleet worker
archive-worm / tree-nodefalse on host fleetrunning on LXC 510 via keep +units.px-worker.json (1 worm + 1 tree)
account-balance, access-*, identity, jobs-events, CS/sales/acct, +staff-iam/session, zappier-edgetrue keepFloorkeepFloor
zapier-simulatortrue, keepFloor falseoptional
natsmanaged false3-node cluster + leftover host loopback
+

Endpoint I/O

+

Each product NATS address: what it is for, +who sends, who receives, how +to attach.

+

Connect with a long-lived client +(reconnect-per-message is the anti-pattern):

+
NATS_URL=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+

Public edge (token in /etc/nats/public-edge.token on +NS1): nats://<token>@70.88.205.138:4222. Do not put +that URL in Zapier or in Verae.

+
+

verae.zapier.jobs.watch

+ ++++ + + + + + + + + + + + + + + + + + + +
ForWork queue: “poll Verae until this jobId is +terminal.”
In frommiddleware enqueueWatch after +POST /api/timestamp (or mock) returns +jobId
Out tojob-poller worker (queue job-poller, durable on +ZAPIER_JOBS)
Thenpoller calls Verae GET /api/status/{jobId} over +HTTPS, not NATS
+

Payload: tenantId, veraeUserId, +jobId, tokenRef, enqueuedAt, +attempt, maxAttempts, intervalMs, +traceId. JWT stays in middleware (tokenRef); +never on Zapier.

+

Connect Verae-side: do not +subscribe this from api.veraetime.net. The poller is the +Verae HTTP client. To add another watcher, start another +job-poller consumer in the same queue group.

+
+

verae.zapier.jobs.events

+ ++++ + + + + + + + + + + + + + + +
ForTerminal outcomes: timestamp.completed / +failed / timeout
In fromjob-poller after Verae status is not pending
Out to(1) middleware waitForJobEvent (wait HTTP); (2) webhook +router → webhooks.deliver; (3) optional +verae-jobs-events mailbox :3030
+

If JOBS_EVENTS_EXCLUSIVE=1, only +verae-jobs-events is the durable consumer and republishes +verae.internal.jobs.events. Default lab: middleware wait + +webhook worker share the stream.

+

Payload: event, tenantId, +jobId, status, traceId, +emittedAt.

+
+

verae.zapier.webhooks.deliver

+ + + + + + + + + + + + + + + +
ForHTTPS POST to hooks.zapier.com (REST Hook)
In fromwebhook worker on completed/failed events
Out toZapier cloud HTTPS, never NATS
+

Payload: hookId, tenantId, +targetUrl, event, payload, +attempt, traceId. Slow consumer: keep replica +floors on the worker if you split it out of middleware.

+
+

verae.zapier.usage

+ ++++ + + + + + + + + + + + + + + + + + + +
ForOptional meter of Zap actions on NATS (JetStream)
In fromdesigned: middleware / edge
Out todesigned: USAGE_WRITER → books
Runningstream exists; no writer; middleware uses local usage +files
+

Prepaid meter that is wired uses +verae.billing.usage.recorded (core pub), not this stream. +Finish: either publish here from recordUsage or drop the +stream.

+
+

Billing +(verae.billing.*) — account-balance source of truth

+

Request-reply unless noted. Queue account-balance. +Not JetStream.

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
AddressForIn fromOut to
statement.getprepaid, credits, usage, paymentsportal, CS, sales, accounting, access-web/apireply to requester
balance.getcents onlyinternalreply
balance.adjustCS credit or portal reloadCS, access-web (reload kind only)books + reply
usage.recordedmeter pubzappier-edge meter() onUsage when +NATS_URL setaccount-balance subscriber (debits prepaid)
payment.recordedreload/invoice pubportalbooks
credit.appliedgoodwill pubCS/adminbooks
customer.putupsert customeredge ledgerbooks
+

Authz must pass +verae.access.authz.check before these from a public plane. +Leaf plane cannot call billing (policy).

+

Middleware recordUsage still writes local tenant +files and does not publish +usage.recorded. Two meters until that is unified.

+
+

Access planes

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
AddressForInOut
verae.access.authz.checkallow/deny { plane, subject, principal }every plane + edge billing-nats.ts{ allow, reason }; deny also pubs +authz.deny
verae.access.authz.denyauditauthz on denynowhere yet (no SIEM)
verae.access.web.in / +verae.access.web.billing.*browser portalaccess-web HTTPstatement.get / balance.adjust kind=reload +after authz
verae.access.api.*customer x-api-keyaccess-apistatement; HTTPS to middleware for jobs
verae.access.zapier.*Zapier HTTPS onlyaccess-zapierjobs.watch via HTTP to middleware, not Zapier NATS; +usage.recorded allowed by policy
verae.access.leaf.inS2S { token, target, payload }remote leafarchive + jobs after authz; never billing
verae.access.staff.*CS/sales/accountingaccess-staffbilling after authz
+

Ingress mapping: +accessAddress(plane, 'verae.billing.statement.get') → +verae.access.<plane>.billing.statement.get.

+
+

Identity

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + +
AddressForInOut
verae.identity.bindbind email/customerId → veraeUserIdportal / middlewarereply
verae.identity.lookuplookup by customerId or veraeUserIdsamereply
+

Queue identity. Fleet keepFloor.

+
+

Archive / tree

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
AddressForInOut
verae.archive.putWORM/tree write (kind: publicMeta / privateMeta / file +/ tree)splitter (in-process from middleware today)every worm/tree (JS)
verae.archive.querybroadcast lookupaggregatorall worm/tree, no queue group
verae.archive.reply.<correlationId>bloom hit onlyworm/treeaggregator
+

Silence = miss. Chain seal does not wait on archive. Bytes never go +to the chain.

+
+

Template / unused

+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
AddressForInOutRunning
verae.splitter.infuture split-out splitter processmiddlewaresplitterno
verae.example.process.intemplate workeranyexample-process Qno
verae.example.process.outtemplate fan-outtemplatepeersno
verae.example.process.reply.<id>template replytemplaterequesterno
verae.internal.jobs.eventsexclusive mailbox republishjobs-eventswait/webhookno unless env
+
+

How to connect Verae +API to NATS

+

api.veraetime.net is HTTPS. It does not +subscribe. The glue is middleware:

+
    +
  1. Set VERAE_API_BASE_URL=https://api.veraetime.net and +MOCK_VERAE=false.
    2. +
  2. Middleware login (username/password) → JWT stored as +tokenRef.
    3. +
  3. POST /api/timestamp → 202 { jobId }.
    4. +
  4. Publish verae.zapier.jobs.watch.
    5. +
  5. Poller GET /api/status/{jobId} until +completed / failed.
    6. +
  6. Publish verae.zapier.jobs.events.
    7. +
  7. Wait HTTP and/or REST Hook.
    8. +
+

Do not point Verae at NATS_URL. Do +not put NATS in Zapier env. A new worker that must see +seals uses jobs.events or HTTP +GET /zapier/v1/status/{jobId}.

+

Template for a new addressed process: clone +verae-nats-process, rename +verae.example.process.*, add the subject to +verae-access-authz policy for the plane that may call +it.

+

Zapier → Verae API → Zapier

+

How a timestamp job is submitted, watched, and returned. Lab today: +MOCK_VERAE=true. Live path is the same HTTP contract with +MOCK_VERAE=false.

+

Rule

+
Zapier cloud  --HTTPS only-->  zap.zapier.georgelambert.org  (access-zapier)
+                                    |
+                                    v HTTPS
+                              middleware :3100 /zapier/v1
+                                    |
+                    +---------------+----------------+
+                    | HTTPS                         | NATS (private cluster)
+                    v                               v
+            api.veraetime.net              jobs.watch → poller
+            POST /api/timestamp            GET /api/status/{jobId}
+            GET  /api/status/{jobId}              |
+            POST /api/verify                      v
+                                           jobs.events
+                                              |
+                         +--------------------+--------------------+
+                         | wait subscriber                         | webhook-deliver
+                         v                                         v
+                    HTTP response to Zap                    POST hooks.zapier.com
+

Zapier never has NATS_URL. Verae never has +NATS_URL.

+

1. Submit (async create)

+
    +
  1. Zap action Create Timestamp (or curl) +POST https://zap.zapier.georgelambert.org/zapier/v1/timestamp +with portal API key zmw_… (not a Verae JWT).
    2. +
  2. access-zapier / edge: authz, then meter() records +cents; if NATS_URL is set, +natsPublish(verae.billing.usage.recorded).
    3. +
  3. Forwards to middleware POST /zapier/v1/timestamp.
    4. +
  4. Middleware checkEntitlement (local plan limits +today).
    5. +
  5. veraeClient.createTimestamp: +
      +
    • Mock: in-process job, fake jobId.
      • +
    • Live: +POST https://api.veraetime.net/api/timestamp with JWT → +202 { jobId }.
      • +
    6. +
  6. Splitter (in-process): hash vs attachments; optional +verae.archive.put (bytes never go to chain).
    7. +
  7. enqueueWatch → JetStream +verae.zapier.jobs.watch.
    8. +
  8. HTTP 202 { jobId, sha256, existing } +back to Zapier immediately.
    9. +
+

Same SHA-256 is idempotent (existing: true).

+

2. Monitor (job poller)

+

The poller is in the middleware Node process when +NATS_ENABLED=true (startJobPollerWorker). +Fleet does not spawn a second copy +(job-poller.enabled: false).

+
    +
  1. Durable consumer job-poller on ZAPIER_JOBS +/ verae.zapier.jobs.watch.
    2. +
  2. Resolve tokenRef → tenant Verae password → +POST login → JWT (stays in middleware).
    3. +
  3. GET /api/status/{jobId} (mock or live timestamp +server).
    4. +
  4. pending → nak after +intervalMs (retry).
    5. +
  5. completed → publish jobs.events +{ event: timestamp.completed, status, jobId, tenantId, traceId }.
    6. +
  6. failed → timestamp.failed. Too many +attempts → timestamp.timeout.
    7. +
  7. ack the watch message.
    8. +
+

Live OpenAPI: poll until completed / +failed. Hash lookup is 501 on live +(lookupHash). Map certificate, +timestamp, blockIndex, sha256 +into the receipt Zap samples expect.

+

3. Wait (sync Zap step)

+

Create Timestamp and Wait → +POST /zapier/v1/timestamp/wait.

+
    +
  1. Same as submit (watch is already on NATS).
    2. +
  2. waitForJobEvent(jobId, WAIT_TIMEOUT_MS) subscribes to +verae.zapier.jobs.events.
    3. +
  3. On matching jobId event: return status JSON +(certificate / receipt).
    4. +
  4. On timeout: { status: pending, jobId } so the REST Hook +can finish the Zap.
    5. +
+

includeAttached: after seal, aggregator should +verae.archive.query; bloom hits merge extra receipts. +Simulator does this; live aggregator hook is TBD.

+

4. Push back to Zapier

+
    +
  • Wait response: HTTP body on the same Zap step.
    • +
  • REST Hook: Zapier subscribes +POST /zapier/v1/hooks. On jobs.events, +middleware enqueues +verae.zapier.webhooks.deliver. Worker +POSTs targetUrl (hooks.zapier.com).
    • +
  • Poll status: Zap action Find Job +Status → GET /zapier/v1/status/{jobId} → +middleware GET Verae (or last event).
    • +
+

5. Verify

+

POST /zapier/v1/verify → middleware → Verae +POST /api/verify. No job watch.

+

6. Batch Merkle (designed; same +bus)

+

From MESSAGE-FLOWS.md:

+
    +
  1. POST /v1/timestamp/batch — one item per line.
    2. +
  2. Middleware hashes leaves, builds Merkle tree.
    3. +
  3. Chain seals only the root +(POST /api/timestamp on the root).
    4. +
  4. Each leaf proof → verae.archive.put +kind: tree.
    5. +
  5. Central hash lookup of a leaf misses unless +includeTree=true.
    6. +
  6. Aggregator broadcasts verae.archive.query +kinds=["tree"].
    7. +
  7. Tree node bloom hit → archive.reply.<id> with +proof.
    8. +
+

Live wait path does not yet call the aggregator, so tree-leaf +receipts are not merged on NS1.

+

Connect live +api.veraetime.net

+

Still blocked on operator bind credentials. When ready:

+
# middleware env
+MOCK_VERAE=false
+VERAE_API_BASE_URL=https://api.veraetime.net
+NATS_ENABLED=true
+NATS_URL=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+JETSTREAM_REPLICAS=3
+

Then: working Verae login, one live seal, confirm 202/status/verify, +map receipt fields, REST Hook to a private Zap. See +zapier-decisions/STATUS.md “Once MOCK_VERAE=false”.

+

Billing on this path

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
HopDesignedRunning
Edge meterverae.billing.usage.recordedcoded; fires when edge has +NATS_URL
Account-balancesubscribe + debit prepaid; 402 if emptysubscriber running; 402 from books not proven on +every Zap
Middleware quotalocal files via recordUsagerunning; not NATS
verae.zapier.usageoptional JS meterstream only
+

Connect a new internal +process

+
export NATS_URL=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
+# clone verae-nats-process, rename subjects, register in verae-fleet
+

Jobs stay: HTTP in from Zapier → HTTPS to Verae → NATS watch/events → +HTTP back to Zapier.

+

Gaps and finish list

+

Modules and addresses that are designed but not fully +connected, not running as fleet workers, or not on the live Verae/Zapier +path.

+

Blocked on the operator +(not a NATS bug)

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ItemStateTo finish
Zapier Platform pushno loginzapier login → register → +MIDDLEWARE_BASE_URL=https://zap.zapier.georgelambert.org → +push private app
Live VeraeMOCK_VERAE=truecredentials; MOCK_VERAE=false; prove one seal
nkeys / mTLScluster $G onlyverae-nats-accounts sketch; flip only after every +NATS_URL has creds
Hash lookup on live OpenAPI501Verae API gap or our store
+

Running on HTTP/NATS +but incomplete routing

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModuleRunning?NATSGap
middleware HTTP + in-process job-poller + webhook-deliveryeswatch/events/deliverlive Verae + Zapier push
account-balanceyesbilling RR + usage.recorded subprove 402 from books on a Zap when prepaid is 0
zappier-edgeyesusage.recorded / payment.recorded / +credit.applied / customer.put if +NATS_URLconfirm live process env; Stripe live
access-* + authzyesauthz + ingress—
identityyes (fleet)bind/lookupconfirm portal always binds veraeUserId
jobs-events mailbox :3030yes (fleet)events counterexclusive consumer off unless +JOBS_EVENTS_EXCLUSIVE=1
CS / sales / accountingyesstatement/adjuststaff UX
archive-worm + tree-nodeyes on LXC 510 (keep)archive.put/query/replyonly 1+1 copies; fleet overlay +enabled: false for extra replicas
archive-aggregatorno as fleet workerquery/replyenable overlay or finish in-process hook +(includeAttached TBD in middleware)
request-splitterlibrary in middlewarearchive.putverae.splitter.in not used
ZAPIER_USAGE / USAGE_WRITERstream yesverae.zapier.usageno consumer; middleware usage is local files
fleet job-poller / webhook-deliver extra +copiesoverlay falsesame subjectsonly enable if you want SSH-spread replicas instead +of in-process (same queue names)
+

Designed, not a live worker

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModuleAddressWhat to do
verae-nats-processverae.example.process.*template only; clone to add a function
verae-activatenone (HTTPS Zapier)optional tiny app; needs Zapier login
verae-zapier (full app)HTTPSpush after login
simulatorin-process fake NATSlocal :3847; not production
verae-nats-accountsnkeys policynot flipped
+

Aux (no product subjects, +or HTTP only)

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModuleNATSNotes
verae-keepnoneHTTP health; restarts units
verae-fleetnonecatalog + keepFloor
verae-uptimenoneoff-box HTTPS GET of public doors
verae-backupnonedisk / JetStream snapshot; no live event subject
verae-bootstrap / verae-deploy / +verae-proxmox / verae-opsnoneinstall/link
staff-iam / staff-session / staff-uinoneHTTP cookies
public NATS edgeleaf, tokendo not use from Zapier; rotate token; later nkeys +LEAF vs INTERNAL
+

Billing triggers and event +logging

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
SignalDesignedRunningFinish
Timestamp/verify meter (edge)verae.billing.usage.recordedcode path exists (zappier/src/app.ts +onUsage)Confirm NATS_URL on live edge; 402 from account-balance +when prepaid empty
Middleware quotalocal files recordUsageyesEither publish usage.recorded too or drop duplicate +quota
Optional JS meterverae.zapier.usage + USAGE_WRITERstream onlyWire or delete
Portal reloadpayment.recorded + balance.adjustcodedconfirm both fire on Stripe/test reload
CS creditbalance.adjust + credit.appliedcodedconfirm applied pub on live
Customer upsertcustomer.putcoded—
Authz deny auditverae.access.authz.denyrunning on denyno SIEM sink
Job lifecycle logjobs.eventsrunningmailbox counts only — add structured log/export if needed
Archive put/queryVERAE_ARCHIVE510 worm/treeaggregator not in wait path
Public edgeleafrunningrotate token; do not expose :6222
+

Ordered finish for message +routing

+
    +
  1. Do not double-start job-poller: either in-process +or fleet copies, same queue +job-poller.
    2. +
  2. Confirm edge NATS_URL so +usage.recorded actually lands; then 402 from books. Unify +or drop middleware local recordUsage.
    3. +
  3. Enable archive-aggregator (or in-process +equivalent) so includeAttached and tree proofs work on +wait/status.
    4. +
  4. Raise worm/tree replica floors on 510/more CTs if +archive HA is required (fleet overlay min 3).
    5. +
  5. Operator: Zapier push + +MOCK_VERAE=false.
    6. +
  6. Prove: Zap create → jobs.watch → live +GET /api/status → jobs.events → wait body + +REST Hook.
    7. +
  7. nkeys/mTLS after every client has creds; then lock down the public +edge to accounts (LEAF vs INTERNAL).
    8. +
  8. Optional: exclusive jobs-events mailbox; SIEM on +authz.deny and jobs.events; +verae.splitter.in if splitter is split out; +USAGE_WRITER or delete ZAPIER_USAGE.
    9. +
+

What is already enough +for a mock Zap

+

HTTPS zap/mw → mock create → jobs.watch → in-process +poller → mock status completed → jobs.events → wait returns +mock certificate. Archive extras and live chain are the missing +middle.

+ + diff --git a/packages/dev-overview/README.md b/packages/dev-overview/README.md index 9533a65..ce81f30 100644 --- a/packages/dev-overview/README.md +++ b/packages/dev-overview/README.md @@ -7,18 +7,17 @@ Developer map of the Verae Time × Zapier **NATS message fabric**: what was desi **Catalog:** https://zapier.georgelambert.org/packages/dev-overview/ **Date:** 2026-09-14 -Zapier cloud **never** connects to NATS. It talks HTTPS to `https://zap.zapier.georgelambert.org` (access-zapier) then middleware. +Zapier cloud **never** connects to NATS. It talks HTTPS to `https://zap.zapier.georgelambert.org` (access-zapier) then middleware. `api.veraetime.net` is **HTTPS only** — middleware is the NATS client. ## Contents | Doc | What | |-----|------| -| [01-nats-routes.md](01-nats-routes.md) | Every `verae.*` address: designed vs running, stream, queue | -| [02-endpoint-io.md](02-endpoint-io.md) | Per-address: purpose, inbound from, outbound to, how to connect | -| [03-zapier-verae-path.md](03-zapier-verae-path.md) | Zapier → middleware → Verae API → poll → events → Zapier | -| [04-gaps-and-finish.md](04-gaps-and-finish.md) | Modules not connected; billing/aux; finish list | - -Sources: `docs-master/modules-and-nats.md`, `MESSAGE-FLOWS.md`, fleet `services/*.json`, overlay `overlays/ns1/fleet.json`, middleware `src/nats/subjects.js`, `ensure-streams.sh`, keep `units.px-worker.json`, STATUS.md. +| [00-sources.md](00-sources.md) | Every other doc that names a NATS address or TCP route | +| [01-nats-routes.md](01-nats-routes.md) | Transport ports + every `verae.*` address: designed vs running | +| [02-endpoint-io.md](02-endpoint-io.md) | Per-address: purpose, inbound from, outbound to, how to connect Verae API | +| [03-zapier-verae-path.md](03-zapier-verae-path.md) | Zapier → middleware → Verae timestamp server → poll → events → Zapier | +| [04-gaps-and-finish.md](04-gaps-and-finish.md) | Modules not connected; billing/event/aux; ordered finish list | ## Cluster (running) @@ -28,4 +27,4 @@ nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222 JetStream streams (replicas=3, file): `ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE`. -Public token-gated **client** edge (not cluster routes): `nats://@70.88.205.138:4222`, `wss://nats.zapier.georgelambert.org`. +Public token-gated **client** edge (not cluster routes): `nats://@70.88.205.138:4222`, `wss://nats.zapier.georgelambert.org`. Do not put this in Zapier. diff --git a/packages/dev-overview/README.pdf b/packages/dev-overview/README.pdf new file mode 100644 index 0000000..2a6fecf Binary files /dev/null and b/packages/dev-overview/README.pdf differ diff --git a/packages/dev-overview/SUMMARY.md b/packages/dev-overview/SUMMARY.md index 354e711..da8e035 100644 --- a/packages/dev-overview/SUMMARY.md +++ b/packages/dev-overview/SUMMARY.md @@ -1,5 +1,8 @@ # dev-overview -Developer map of **designed vs running** NATS addresses, per-endpoint I/O, the Zapier → Verae timestamp path, and what is still unconnected. +Developer map of **designed vs running** NATS addresses, transport ports, per-endpoint I/O, the Zapier → Verae timestamp path, and what is still unconnected. -**Forgejo:** https://git.georgelambert.org/marchon/dev-overview +**Forgejo:** https://git.georgelambert.org/marchon/dev-overview +**Catalog:** https://zapier.georgelambert.org/packages/dev-overview/ + +Start: [README.md](README.md).