Add dev-overview: designed vs running NATS routes and Zapier-Verae path.
Some checks failed
offline / test (push) Has been cancelled
Some checks failed
offline / test (push) Has been cancelled
Independent Forgejo repo maps every verae.* address, endpoint I/O, job monitor loop, and remaining billing/archive wiring.
This commit is contained in:
parent
c83f1c6717
commit
86b421db05
11 changed files with 436 additions and 1 deletions
76
packages/dev-overview/01-nats-routes.md
Normal file
76
packages/dev-overview/01-nats-routes.md
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
# Designed vs running NATS routes
|
||||
|
||||
**Designed** = named in docs (`modules-and-nats.md`, `MESSAGE-FLOWS.md`, per-package `NATS.md`) or in code (`subjects.js`, fleet JSON).
|
||||
**Running** = a process on NS1 is subscribed or publishing it **today**, and (for durable work) a JetStream stream exists.
|
||||
|
||||
Cluster name `verae`. Internal URL: three LXC on `vmbr1`. Public edge is a **leaf**, not a second cluster.
|
||||
|
||||
Legend: **JS** = JetStream stream; **RR** = core request-reply; **pub** = core or JS fire-and-forget; **Q** = queue group.
|
||||
|
||||
## JetStream streams (running)
|
||||
|
||||
Created by `verae-nats-cluster/scripts/ensure-streams.sh` (and middleware `ensureStreams` for the first three).
|
||||
|
||||
| Stream | Subjects | Replicas | Storage | Who consumes |
|
||||
|--------|----------|----------|---------|--------------|
|
||||
| `ZAPIER_JOBS` | `verae.zapier.jobs.watch` | 3 | file | job-poller (in middleware process) |
|
||||
| `ZAPIER_EVENTS` | `verae.zapier.jobs.events` | 3 | file | middleware wait + webhook router; optional exclusive `jobs-events` |
|
||||
| `ZAPIER_WEBHOOKS` | `verae.zapier.webhooks.deliver` | 3 | file | webhook-deliver (in middleware process) |
|
||||
| `ZAPIER_USAGE` | `verae.zapier.usage` | 3 | file | **no durable consumer running** (`USAGE_WRITER` is named only) |
|
||||
| `VERAE_ARCHIVE` | `verae.archive.>` | 3 | file | worm + tree on LXC 510 (keep); aggregator **not** fleet-enabled |
|
||||
|
||||
## Address inventory
|
||||
|
||||
| Address | Kind | Designed | Running | Notes |
|
||||
|---------|------|----------|---------|-------|
|
||||
| `verae.zapier.jobs.watch` | JS work Q `job-poller` | yes | **yes** | middleware publishes; in-process poller consumes |
|
||||
| `verae.zapier.jobs.events` | JS events | yes | **yes** | poller publishes; wait() subscribes; webhook worker routes |
|
||||
| `verae.zapier.webhooks.deliver` | JS work Q `webhook-deliver` | yes | **yes** | middleware worker in same Node process as HTTP |
|
||||
| `verae.zapier.usage` | JS optional | yes | stream **yes**, consumer **no** | middleware entitlement is **local files**, not this subject |
|
||||
| `verae.internal.jobs.events` | core republish | yes (`jobs-events` exclusive) | **partial** | only if `JOBS_EVENTS_EXCLUSIVE=1` |
|
||||
| `verae.billing.statement.get` | RR | yes | **yes** | account-balance |
|
||||
| `verae.billing.balance.get` | RR | yes | **yes** | account-balance |
|
||||
| `verae.billing.balance.adjust` | RR | yes | **yes** | CS credits, portal reload |
|
||||
| `verae.billing.usage.recorded` | pub | yes | **partial** | designed from zappier-edge meter; confirm publisher on live edge |
|
||||
| `verae.billing.payment.recorded` | pub | yes | **partial** | portal reload / invoice |
|
||||
| `verae.billing.credit.applied` | pub | yes | **partial** | CS goodwill |
|
||||
| `verae.billing.customer.put` | RR/pub | yes (account-balance `subjects.js`) | **unclear** | books write |
|
||||
| `verae.access.authz.check` | RR Q `access-authz` | yes | **yes** | fleet keepFloor |
|
||||
| `verae.access.authz.deny` | pub audit | yes | **yes** (on deny) | |
|
||||
| `verae.access.web.*` | ingress | yes | **yes** | access-web after HTTP |
|
||||
| `verae.access.api.*` | ingress | yes | **yes** | |
|
||||
| `verae.access.zapier.*` | ingress | yes | **yes** | HTTPS plane, not Zapier NATS |
|
||||
| `verae.access.leaf.in` / `verae.access.leaf.*` | ingress | yes | **yes** | no billing |
|
||||
| `verae.access.staff.*` | ingress | yes | **yes** | CS/sales/accounting |
|
||||
| `verae.identity.bind` | RR Q `identity` | yes | **yes** (fleet enabled) | |
|
||||
| `verae.identity.lookup` | RR Q `identity` | yes | **yes** | |
|
||||
| `verae.archive.put` | JS (`VERAE_ARCHIVE`) | yes | **partial** | splitter is **in-process** in middleware; worm/tree on 510 consume |
|
||||
| `verae.archive.query` | broadcast (no Q) | yes | **partial** | live aggregator hook **TBD** in middleware (simulator has it) |
|
||||
| `verae.archive.reply.<id>` | replies on bloom hit | yes | **partial** | worm/tree on 510 can reply; aggregator not fleet-enabled |
|
||||
| `verae.splitter.in` | future | yes (docs) | **no** | splitter stays a library |
|
||||
| `verae.example.process.in` / `.out` / `.reply.*` | template | yes | **no** | `verae-nats-process` is a model, not a live worker |
|
||||
| `bench.*` | lab only | n/a | ephemeral | speed studies; not product |
|
||||
|
||||
## Fleet overlay vs in-process
|
||||
|
||||
`overlays/ns1/fleet.json` (2026-09-14):
|
||||
|
||||
| Service | fleet `enabled` | Actual lab |
|
||||
|---------|-----------------|------------|
|
||||
| middleware-http | true | **running**; also starts job-poller + webhook workers in-process when `NATS_ENABLED` |
|
||||
| job-poller | **false** | covered by middleware, not a second fleet spawn |
|
||||
| webhook-deliver | **false** | same |
|
||||
| archive-aggregator | **false** | **not** running as a fleet worker |
|
||||
| archive-worm / tree-node | **false** on host fleet | **running on LXC 510** via `verae-keep` `units.px-worker.json` (1 worm + 1 tree) |
|
||||
| account-balance, access-*, identity, jobs-events, CS/sales/acct, staff-iam/session | true | keepFloor |
|
||||
| zapier-simulator | true, keepFloor false | optional |
|
||||
| nats | managed false | 3-node cluster + leftover host loopback |
|
||||
|
||||
## Docs that name routes (do not invent extras)
|
||||
|
||||
- `packages/docs-master/modules-and-nats.md`, `MESSAGE-FLOWS.md`
|
||||
- `packages/overview/03-nats-cluster.md`, `06-address-routing.md`
|
||||
- `packages/verae-ops/06-linking-services.md`
|
||||
- Each package `NATS.md`
|
||||
- `verae-nats-cluster/scripts/ensure-streams.sh`
|
||||
- Public edge: `verae-nats-cluster/conf/public-edge.conf.tmpl` (leaf, not a new subject family)
|
||||
135
packages/dev-overview/02-endpoint-io.md
Normal file
135
packages/dev-overview/02-endpoint-io.md
Normal file
|
|
@ -0,0 +1,135 @@
|
|||
# Endpoint I/O
|
||||
|
||||
Each product NATS address: **what it is for**, **who sends**, **who receives**, **how to attach**.
|
||||
|
||||
Connect with a **long-lived** client:
|
||||
|
||||
```text
|
||||
NATS_URL=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
|
||||
```
|
||||
|
||||
Public edge (token in `/etc/nats/public-edge.token` on NS1): `nats://<token>@70.88.205.138:4222`. Do not put that URL in Zapier.
|
||||
|
||||
---
|
||||
|
||||
## `verae.zapier.jobs.watch`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| For | Work queue: “poll Verae until this `jobId` is terminal.” |
|
||||
| In from | middleware `enqueueWatch` after `POST /api/timestamp` (or mock) returns `jobId` |
|
||||
| Out to | job-poller worker (queue `job-poller`, durable on `ZAPIER_JOBS`) |
|
||||
| Then | poller calls Verae `GET /api/status/{jobId}` over **HTTPS**, not NATS |
|
||||
|
||||
Payload: `tenantId`, `veraeUserId`, `jobId`, `tokenRef`, `enqueuedAt`, `attempt`, `maxAttempts`, `intervalMs`, `traceId`. JWT stays in middleware (`tokenRef`); never on Zapier.
|
||||
|
||||
**Connect Verae-side:** do **not** subscribe this from `api.veraetime.net`. The poller is the Verae HTTP client. To add another watcher, start another `job-poller` consumer in the same queue group.
|
||||
|
||||
---
|
||||
|
||||
## `verae.zapier.jobs.events`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| For | Terminal outcomes: `timestamp.completed` / `failed` / `timeout` |
|
||||
| In from | job-poller after Verae status is not `pending` |
|
||||
| Out to | (1) middleware `waitForJobEvent` (wait HTTP); (2) webhook router → `webhooks.deliver`; (3) optional `verae-jobs-events` mailbox |
|
||||
|
||||
If `JOBS_EVENTS_EXCLUSIVE=1`, only `verae-jobs-events` is the durable consumer and republishes `verae.internal.jobs.events`. Default lab: middleware wait + webhook worker share the stream.
|
||||
|
||||
---
|
||||
|
||||
## `verae.zapier.webhooks.deliver`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| For | HTTPS POST to `hooks.zapier.com` (REST Hook) |
|
||||
| In from | webhook worker on completed/failed events |
|
||||
| Out to | Zapier cloud **HTTPS**, never NATS |
|
||||
|
||||
Slow consumer: this is the flood/backlog path. Keep replica floors on the worker.
|
||||
|
||||
---
|
||||
|
||||
## `verae.zapier.usage`
|
||||
|
||||
| | |
|
||||
|--|--|
|
||||
| For | Optional meter of Zap actions on NATS |
|
||||
| In from | designed: middleware / edge |
|
||||
| Out to | designed: `USAGE_WRITER` → books |
|
||||
| Running | **stream exists; no writer; middleware uses local usage files** |
|
||||
|
||||
Finish: publish here from `recordUsage`, consume into `verae.billing.usage.recorded`.
|
||||
|
||||
---
|
||||
|
||||
## Billing (`verae.billing.*`) — account-balance SoT
|
||||
|
||||
Request-reply unless noted. Queue on account-balance. **Not** JetStream (except you could mirror later).
|
||||
|
||||
| Address | For | In from | Out to |
|
||||
|---------|-----|---------|--------|
|
||||
| `statement.get` | prepaid, credits, usage, payments | portal, CS, sales, admin, access-web/api | reply to requester |
|
||||
| `balance.get` | cents only | internal | reply |
|
||||
| `balance.adjust` | CS credit or portal reload | CS, access-web (reload kind only) | books + reply |
|
||||
| `usage.recorded` | meter pub | zappier-edge (designed) | account-balance subscriber |
|
||||
| `payment.recorded` | reload/invoice pub | portal | books |
|
||||
| `credit.applied` | goodwill pub | CS | books |
|
||||
| `customer.put` | upsert customer | edge/IAM | books |
|
||||
|
||||
Authz **must** pass `verae.access.authz.check` before these from a public plane. Leaf plane **cannot** call billing (policy).
|
||||
|
||||
---
|
||||
|
||||
## Access planes
|
||||
|
||||
| Address | For | In | Out |
|
||||
|---------|-----|----|-----|
|
||||
| `verae.access.authz.check` | allow/deny `{ plane, subject, principal }` | every plane | `{ allow, reason }`; deny also pubs `authz.deny` |
|
||||
| `verae.access.web.*` | browser portal | access-web HTTP | billing statement/reload after authz |
|
||||
| `verae.access.api.*` | customer `x-api-key` | access-api | statement; HTTPS to middleware for jobs |
|
||||
| `verae.access.zapier.*` | Zapier HTTPS only | access-zapier | jobs.watch **via HTTP to middleware**, not Zapier NATS |
|
||||
| `verae.access.leaf.*` | S2S | access-leaf | archive + jobs; **never billing** |
|
||||
| `verae.access.staff.*` | CS/sales/accounting | access-staff | billing after authz |
|
||||
|
||||
---
|
||||
|
||||
## Identity
|
||||
|
||||
| Address | For | In | Out |
|
||||
|---------|-----|----|-----|
|
||||
| `verae.identity.bind` | bind email/customerId → `veraeUserId` | portal / middleware | reply |
|
||||
| `verae.identity.lookup` | lookup by customerId or veraeUserId | same | reply |
|
||||
|
||||
Queue `identity`. Fleet keepFloor.
|
||||
|
||||
---
|
||||
|
||||
## Archive / tree
|
||||
|
||||
| Address | For | In | Out |
|
||||
|---------|-----|----|-----|
|
||||
| `verae.archive.put` | WORM/tree write (hash, meta, file meta, merkle leaf) | splitter (in-process from middleware today) | every worm/tree (JS) |
|
||||
| `verae.archive.query` | broadcast lookup | aggregator | **all** worm/tree, **no** queue group |
|
||||
| `verae.archive.reply.<correlationId>` | bloom **hit only** | worm/tree | aggregator |
|
||||
|
||||
Silence = miss. Chain seal does not wait on archive.
|
||||
|
||||
---
|
||||
|
||||
## How to connect **Verae API** to NATS
|
||||
|
||||
`api.veraetime.net` is **HTTPS**. It does not subscribe. The glue is middleware:
|
||||
|
||||
1. Set `VERAE_API_BASE_URL=https://api.veraetime.net` and `MOCK_VERAE=false`.
|
||||
2. Middleware login (username/password) → JWT stored as `tokenRef`.
|
||||
3. `POST /api/timestamp` → 202 `{ jobId }`.
|
||||
4. Publish `verae.zapier.jobs.watch`.
|
||||
5. Poller `GET /api/status/{jobId}` until `completed` / `failed`.
|
||||
6. Publish `verae.zapier.jobs.events`.
|
||||
7. Wait HTTP and/or REST Hook.
|
||||
|
||||
Do **not** point Verae at `NATS_URL`. Do **not** put NATS in Zapier env. A new worker that must see seals uses `jobs.events` or HTTP `GET /zapier/v1/status/{jobId}`.
|
||||
|
||||
Template for a new addressed process: clone `verae-nats-process`, rename `verae.example.process.*`.
|
||||
97
packages/dev-overview/03-zapier-verae-path.md
Normal file
97
packages/dev-overview/03-zapier-verae-path.md
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
# Zapier → Verae API → Zapier
|
||||
|
||||
How a timestamp job is submitted, watched, and returned. Lab today: `MOCK_VERAE=true`. Live path is the same HTTP contract with `MOCK_VERAE=false`.
|
||||
|
||||
## Rule
|
||||
|
||||
```text
|
||||
Zapier cloud --HTTPS only--> zap.zapier.georgelambert.org (access-zapier)
|
||||
|
|
||||
v HTTPS
|
||||
middleware :3100 /zapier/v1
|
||||
|
|
||||
+---------------+----------------+
|
||||
| HTTPS | NATS (private cluster)
|
||||
v v
|
||||
api.veraetime.net jobs.watch → poller
|
||||
POST /api/timestamp GET /api/status/{jobId}
|
||||
GET /api/status/{jobId} |
|
||||
POST /api/verify v
|
||||
jobs.events
|
||||
|
|
||||
+--------------------+--------------------+
|
||||
| wait subscriber | webhook-deliver
|
||||
v v
|
||||
HTTP response to Zap POST hooks.zapier.com
|
||||
```
|
||||
|
||||
Zapier never has `NATS_URL`. Verae never has `NATS_URL`.
|
||||
|
||||
## 1. Submit (async create)
|
||||
|
||||
1. Zap action **Create Timestamp** (or curl) `POST https://zap.zapier.georgelambert.org/zapier/v1/timestamp` with portal API key `zmw_…` (not a Verae JWT).
|
||||
2. access-zapier / edge meters (designed: `verae.billing.usage.recorded`) and forwards to middleware.
|
||||
3. Middleware `checkEntitlement` (local plan limits today).
|
||||
4. `veraeClient.createTimestamp`:
|
||||
- **Mock:** in-process job, fake `jobId`.
|
||||
- **Live:** `POST https://api.veraetime.net/api/timestamp` with JWT → **202** `{ jobId }`.
|
||||
5. Splitter (in-process): hash vs attachments; optional `verae.archive.put` (bytes never go to chain).
|
||||
6. `enqueueWatch` → JetStream **`verae.zapier.jobs.watch`**.
|
||||
7. HTTP **202** `{ jobId, sha256, existing }` back to Zapier immediately.
|
||||
|
||||
Same SHA-256 is idempotent (`existing: true`).
|
||||
|
||||
## 2. Monitor (job poller)
|
||||
|
||||
The poller is **in the middleware Node process** when `NATS_ENABLED=true` (`startJobPollerWorker`). Fleet does **not** spawn a second copy (`job-poller.enabled: false`).
|
||||
|
||||
1. Durable consumer `job-poller` on `ZAPIER_JOBS` / `verae.zapier.jobs.watch`.
|
||||
2. Resolve `tokenRef` → tenant Verae password → `POST` login → JWT (stays in middleware).
|
||||
3. `GET /api/status/{jobId}` (mock or live).
|
||||
4. `pending` → `nak` after `intervalMs` (retry).
|
||||
5. `completed` → publish `jobs.events` `{ event: timestamp.completed, status, jobId, tenantId, traceId }`.
|
||||
6. `failed` → `timestamp.failed`. Too many attempts → `timestamp.timeout`.
|
||||
7. `ack` the watch message.
|
||||
|
||||
Live OpenAPI: poll until `completed` / `failed`. Hash lookup is **501** on live (`lookupHash`). Map `certificate`, `timestamp`, `blockIndex`, `sha256` into the receipt Zap samples expect.
|
||||
|
||||
## 3. Wait (sync Zap step)
|
||||
|
||||
**Create Timestamp and Wait** → `POST /zapier/v1/timestamp/wait`.
|
||||
|
||||
1. Same as submit (watch is already on NATS).
|
||||
2. `waitForJobEvent(jobId, WAIT_TIMEOUT_MS)` subscribes to **`verae.zapier.jobs.events`**.
|
||||
3. On matching `jobId` event: return status JSON (certificate / receipt).
|
||||
4. On timeout: `{ status: pending, jobId }` so the REST Hook can finish the Zap.
|
||||
|
||||
`includeAttached`: after seal, aggregator should `verae.archive.query`; bloom hits merge extra receipts. **Simulator does this; live aggregator hook is TBD.**
|
||||
|
||||
## 4. Push back to Zapier
|
||||
|
||||
- **Wait response:** HTTP body on the same Zap step.
|
||||
- **REST Hook:** Zapier subscribes `POST /zapier/v1/hooks`. On `jobs.events`, middleware enqueues **`verae.zapier.webhooks.deliver`**. Worker POSTs `targetUrl` (`hooks.zapier.com`).
|
||||
- **Poll status:** Zap action **Find Job Status** → `GET /zapier/v1/status/{jobId}` → middleware `GET` Verae (or last event).
|
||||
|
||||
## 5. Verify
|
||||
|
||||
`POST /zapier/v1/verify` → middleware → Verae `POST /api/verify`. No job watch.
|
||||
|
||||
## Connect live `api.veraetime.net`
|
||||
|
||||
Still blocked on operator bind credentials. When ready:
|
||||
|
||||
```bash
|
||||
# middleware env
|
||||
MOCK_VERAE=false
|
||||
VERAE_API_BASE_URL=https://api.veraetime.net
|
||||
NATS_ENABLED=true
|
||||
NATS_URL=nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
|
||||
JETSTREAM_REPLICAS=3
|
||||
```
|
||||
|
||||
Then: working Verae login, one live seal, confirm 202/status/verify, map receipt fields, REST Hook to a private Zap. See `zapier-decisions/STATUS.md` “Once MOCK_VERAE=false”.
|
||||
|
||||
## Billing on this path
|
||||
|
||||
Designed: edge meters `usage.recorded` → account-balance; 402 if prepaid empty.
|
||||
Running: middleware local usage files; account-balance NATS is up for portal/CS. **Wire edge meter → `verae.billing.usage.recorded` and 402 from books** is still a finish item.
|
||||
68
packages/dev-overview/04-gaps-and-finish.md
Normal file
68
packages/dev-overview/04-gaps-and-finish.md
Normal file
|
|
@ -0,0 +1,68 @@
|
|||
# Gaps and finish list
|
||||
|
||||
Modules and addresses that are designed but **not fully connected, not running as fleet workers, or not on the live Verae/Zapier path**.
|
||||
|
||||
## Blocked on the operator (not a NATS bug)
|
||||
|
||||
| Item | State | To finish |
|
||||
|------|--------|-----------|
|
||||
| Zapier Platform `push` | no login | `zapier login` → register → `MIDDLEWARE_BASE_URL=https://zap.zapier.georgelambert.org` → push private app |
|
||||
| Live Verae | `MOCK_VERAE=true` | credentials; `MOCK_VERAE=false`; prove one seal |
|
||||
| nkeys / mTLS | cluster `$G` only | `verae-nats-accounts` sketch; flip only after every `NATS_URL` has creds |
|
||||
| Hash lookup on live OpenAPI | 501 | Verae API gap or our store |
|
||||
|
||||
## Running on HTTP/NATS but incomplete routing
|
||||
|
||||
| Module | Running? | NATS | Gap |
|
||||
|--------|----------|------|-----|
|
||||
| middleware HTTP + in-process job-poller + webhook-deliver | **yes** | watch/events/deliver | live Verae + Zapier push |
|
||||
| account-balance | **yes** | billing RR | edge may not publish `usage.recorded` on every Zap |
|
||||
| access-* + authz | **yes** | authz + ingress | — |
|
||||
| identity | **yes** (fleet) | bind/lookup | confirm portal always binds `veraeUserId` |
|
||||
| jobs-events mailbox `:3030` | **yes** (fleet) | events counter | exclusive consumer off unless `JOBS_EVENTS_EXCLUSIVE=1` |
|
||||
| CS / sales / accounting | **yes** | statement/adjust | staff UX |
|
||||
| zappier-edge | **yes** | billing after authz | meter → NATS; Stripe live |
|
||||
| archive-worm + tree-node | **yes on LXC 510** (keep) | `archive.put/query/reply` | only **1+1** copies; fleet overlay `enabled: false` for extra replicas |
|
||||
| archive-aggregator | **no** as fleet worker | query/reply | enable overlay **or** finish in-process hook (`includeAttached` TBD in middleware) |
|
||||
| request-splitter | library in middleware | `archive.put` | `verae.splitter.in` not used |
|
||||
| `ZAPIER_USAGE` / `USAGE_WRITER` | stream yes | `verae.zapier.usage` | no consumer; middleware usage is local files |
|
||||
| fleet job-poller / webhook-deliver **extra** copies | overlay **false** | same subjects | only enable if you want SSH-spread replicas **instead of** in-process (do not double-consume without queue groups — in-process already uses queue names) |
|
||||
|
||||
## Designed, not a live worker
|
||||
|
||||
| Module | Address | What to do |
|
||||
|--------|---------|------------|
|
||||
| `verae-nats-process` | `verae.example.process.*` | template only; clone to add a function |
|
||||
| `verae-activate` | none (HTTPS Zapier) | optional tiny app; needs Zapier login |
|
||||
| `verae-zapier` (full app) | HTTPS | push after login |
|
||||
| simulator | in-process fake NATS | local `:3847`; not production |
|
||||
|
||||
## Billing triggers and event logging (aux)
|
||||
|
||||
| Signal | Designed | Finish |
|
||||
|--------|----------|--------|
|
||||
| Timestamp/verify meter | `verae.billing.usage.recorded` + optional `verae.zapier.usage` | Publish from edge/middleware `recordUsage`; consume into account-balance; 402 from books not only local quota |
|
||||
| Portal reload | `payment.recorded` + `balance.adjust` | confirm both fire on Stripe/test reload |
|
||||
| CS credit | `balance.adjust` + `credit.applied` | running for adjust; confirm applied pub |
|
||||
| Authz deny audit | `verae.access.authz.deny` | running on deny; no SIEM sink |
|
||||
| Job lifecycle log | `jobs.events` | running; mailbox counts only — add structured log/export if needed |
|
||||
| Archive put/query | `VERAE_ARCHIVE` | 510 worm/tree; aggregator not in wait path |
|
||||
| Keep/fleet | no NATS | HTTP health only — fine |
|
||||
| Uptime | HTTPS GET | no NATS |
|
||||
| Backup | disk/JS snapshot | no live event subject |
|
||||
| Public NATS edge | leaf, token | **do not** use from Zapier; rotate token; consider nkeys |
|
||||
|
||||
## Ordered finish for message routing
|
||||
|
||||
1. **Do not double-start** job-poller: either in-process **or** fleet copies, same queue `job-poller`.
|
||||
2. Wire **usage**: `recordUsage` → `verae.billing.usage.recorded` (and/or `verae.zapier.usage` + `USAGE_WRITER`).
|
||||
3. Enable **archive-aggregator** (or in-process equivalent) so `includeAttached` and tree proofs work on wait/status.
|
||||
4. Raise worm/tree **replica floors** on 510/more CTs if archive HA is required (fleet overlay min 3).
|
||||
5. Operator: Zapier `push` + `MOCK_VERAE=false`.
|
||||
6. Prove: Zap create → `jobs.watch` → live `GET /api/status` → `jobs.events` → wait body + REST Hook.
|
||||
7. nkeys/mTLS after every client has creds; then lock down the public edge to accounts (LEAF vs INTERNAL).
|
||||
8. Optional: exclusive `jobs-events` mailbox; SIEM on `authz.deny` and `jobs.events`; `verae.splitter.in` if splitter is split out.
|
||||
|
||||
## What is already enough for a mock Zap
|
||||
|
||||
HTTPS zap/mw → mock create → `jobs.watch` → in-process poller → mock status completed → `jobs.events` → wait returns mock certificate. Archive extras and live chain are the missing middle.
|
||||
9
packages/dev-overview/NATS.md
Normal file
9
packages/dev-overview/NATS.md
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
# NATS — dev-overview
|
||||
|
||||
This repo does not subscribe. It documents every `verae.*` address.
|
||||
|
||||
Cluster (running): `nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222` on `vmbr1`.
|
||||
|
||||
Public client edge (token): `nats://70.88.205.138:4222` and `wss://nats.zapier.georgelambert.org`. Zapier cloud still uses **HTTPS only**.
|
||||
|
||||
See [01-nats-routes.md](01-nats-routes.md).
|
||||
31
packages/dev-overview/README.md
Normal file
31
packages/dev-overview/README.md
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
# dev-overview
|
||||
|
||||
Developer map of the Verae Time × Zapier **NATS message fabric**: what was designed, what is actually running on NS1, how a Zapier timestamp job reaches `api.veraetime.net` (or the mock) and comes back, and what is still unwired.
|
||||
|
||||
**Forgejo:** https://git.georgelambert.org/marchon/dev-overview
|
||||
**SSH:** `ssh://git@git.georgelambert.org:2223/marchon/dev-overview.git`
|
||||
**Catalog:** https://zapier.georgelambert.org/packages/dev-overview/
|
||||
**Date:** 2026-09-14
|
||||
|
||||
Zapier cloud **never** connects to NATS. It talks HTTPS to `https://zap.zapier.georgelambert.org` (access-zapier) then middleware.
|
||||
|
||||
## Contents
|
||||
|
||||
| Doc | What |
|
||||
|-----|------|
|
||||
| [01-nats-routes.md](01-nats-routes.md) | Every `verae.*` address: designed vs running, stream, queue |
|
||||
| [02-endpoint-io.md](02-endpoint-io.md) | Per-address: purpose, inbound from, outbound to, how to connect |
|
||||
| [03-zapier-verae-path.md](03-zapier-verae-path.md) | Zapier → middleware → Verae API → poll → events → Zapier |
|
||||
| [04-gaps-and-finish.md](04-gaps-and-finish.md) | Modules not connected; billing/aux; finish list |
|
||||
|
||||
Sources: `docs-master/modules-and-nats.md`, `MESSAGE-FLOWS.md`, fleet `services/*.json`, overlay `overlays/ns1/fleet.json`, middleware `src/nats/subjects.js`, `ensure-streams.sh`, keep `units.px-worker.json`, STATUS.md.
|
||||
|
||||
## Cluster (running)
|
||||
|
||||
```text
|
||||
nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
|
||||
```
|
||||
|
||||
JetStream streams (replicas=3, file): `ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE`.
|
||||
|
||||
Public token-gated **client** edge (not cluster routes): `nats://<token>@70.88.205.138:4222`, `wss://nats.zapier.georgelambert.org`.
|
||||
5
packages/dev-overview/SUMMARY.md
Normal file
5
packages/dev-overview/SUMMARY.md
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
# dev-overview
|
||||
|
||||
Developer map of **designed vs running** NATS addresses, per-endpoint I/O, the Zapier → Verae timestamp path, and what is still unconnected.
|
||||
|
||||
**Forgejo:** https://git.georgelambert.org/marchon/dev-overview
|
||||
6
packages/dev-overview/package.json
Normal file
6
packages/dev-overview/package.json
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
{
|
||||
"name": "dev-overview",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"description": "Developer map of designed vs running NATS routes, Zapier↔Verae job path, and remaining wiring"
|
||||
}
|
||||
|
|
@ -37,6 +37,7 @@ Summaries, NATS contracts, and message flows for every Verae Time × Zapier modu
|
|||
| verae-deploy | https://git.georgelambert.org/marchon/verae-deploy |
|
||||
| verae-proxmox | https://git.georgelambert.org/marchon/verae-proxmox |
|
||||
| verae-nats-cluster | https://git.georgelambert.org/marchon/verae-nats-cluster |
|
||||
| dev-overview | https://git.georgelambert.org/marchon/dev-overview |
|
||||
|
||||
Clone (SSH port 2223):
|
||||
|
||||
|
|
|
|||
|
|
@ -79,6 +79,7 @@ SECTIONS = [
|
|||
("packages/verae-deploy/README.md", "Tagged deploy on the target OS"),
|
||||
("packages/verae-proxmox/README.md", "Proxmox LXC worker on NS1 vmbr1"),
|
||||
("packages/verae-nats-cluster/README.md", "Proxmox 3-node NATS JetStream cluster"),
|
||||
("packages/dev-overview/README.md", "Designed vs running NATS routes and Zapier↔Verae path"),
|
||||
],
|
||||
),
|
||||
(
|
||||
|
|
@ -600,6 +601,7 @@ def main() -> None:
|
|||
"verae-deploy",
|
||||
"verae-proxmox",
|
||||
"verae-nats-cluster",
|
||||
"dev-overview",
|
||||
):
|
||||
pkg_root = ROOT / "packages" / pkg
|
||||
if pkg in {"zapier-user-docs", "overview", "docs-master", "verae-ops"}:
|
||||
|
|
@ -619,7 +621,7 @@ def main() -> None:
|
|||
ignore=shutil.ignore_patterns("node_modules"),
|
||||
)
|
||||
copy_tree(pkg_root / "services", SITE / "packages" / pkg / "services")
|
||||
elif pkg == "verae-nats-process":
|
||||
elif pkg in {"verae-nats-process", "dev-overview"}:
|
||||
for p in pkg_root.glob("*.md"):
|
||||
copy_tree(p, SITE / "packages" / pkg / p.name)
|
||||
else:
|
||||
|
|
@ -839,6 +841,7 @@ def main() -> None:
|
|||
("verae-deploy", "Tagged deploy on the target OS"),
|
||||
("verae-proxmox", "Proxmox LXC worker (NS1 vmbr1)"),
|
||||
("verae-nats-cluster", "Proxmox 3-node NATS JetStream cluster"),
|
||||
("dev-overview", "Designed vs running NATS routes, Zapier↔Verae path"),
|
||||
]
|
||||
git_lis = "".join(
|
||||
f'<li><a href="https://git.georgelambert.org/marchon/{name}">{name}</a> '
|
||||
|
|
@ -858,6 +861,7 @@ def main() -> None:
|
|||
("packages/verae-fleet/docs/CONSOLE.pdf", "Operator console"),
|
||||
("packages/ui-docs/UI-REVIEW.pdf", "UI review (screenshots + live doors)"),
|
||||
("packages/zapier-decisions/STATUS.pdf", "Current lab status (2026-09-12)"),
|
||||
("packages/dev-overview/README.pdf", "NATS routes: designed vs running"),
|
||||
(
|
||||
"packages/zapier-decisions/reports/nats-cluster-bench.pdf",
|
||||
"NATS cluster speed report (charts, 2026-09-12)",
|
||||
|
|
@ -887,6 +891,7 @@ def main() -> None:
|
|||
("docs-master/CATALOG-HOST.html", "Move the catalog to another hostname"),
|
||||
("docs-master/HOST-DEPS.html", "Host OS / Node / NATS installer"),
|
||||
("packages/zapier-decisions/STATUS.html", "Current lab status"),
|
||||
("packages/dev-overview/README.html", "NATS routes: designed vs running"),
|
||||
(
|
||||
"packages/zapier-decisions/reports/nats-cluster-bench.html",
|
||||
"NATS cluster speed report (charts)",
|
||||
|
|
|
|||
|
|
@ -75,6 +75,7 @@ create verae-backup "Backup/restore SQLite, JetStream, worm/tree, keep, IAM"
|
|||
create verae-deploy "Tagged checkout + npm ci on the target OS"
|
||||
create verae-proxmox "Provision archive-worker LXC on NS1 Proxmox"
|
||||
create verae-nats-cluster "Three-node NATS JetStream cluster on Proxmox LXC"
|
||||
create dev-overview "Designed vs running NATS routes, Zapier↔Verae path, finish list"
|
||||
|
||||
push_dir "$ROOT/packages/zappier" zappier-edge
|
||||
push_dir "$ROOT/packages/verae-zapier-middleware" verae-middleware
|
||||
|
|
@ -116,5 +117,6 @@ push_dir "$ROOT/packages/verae-backup" verae-backup
|
|||
push_dir "$ROOT/packages/verae-deploy" verae-deploy
|
||||
push_dir "$ROOT/packages/verae-proxmox" verae-proxmox
|
||||
push_dir "$ROOT/packages/verae-nats-cluster" verae-nats-cluster
|
||||
push_dir "$ROOT/packages/dev-overview" dev-overview
|
||||
|
||||
echo ALL_MODULE_REPOS_PUSHED
|
||||
|
|
|
|||
Loading…
Reference in a new issue