106 lines
7.1 KiB
Markdown
106 lines
7.1 KiB
Markdown
# Designed vs running NATS routes
|
||
|
||
**Designed** = named in docs or in code.
|
||
**Running** = a process on NS1 is subscribed or publishing it **today**, and (for durable work) a JetStream stream exists.
|
||
|
||
Cluster name `verae`. Internal clients use the three LXC on `vmbr1`. Public edge is a **leaf**, not a second cluster.
|
||
|
||
Legend: **JS** = JetStream stream; **RR** = core request-reply; **pub** = core or JS fire-and-forget; **Q** = queue group.
|
||
|
||
## Transport (TCP / leaf / public)
|
||
|
||
These are NATS **server** routes, not `verae.*` subjects.
|
||
|
||
| Endpoint | Designed | Running | Who |
|
||
|----------|----------|---------|-----|
|
||
| `10.10.10.21–23:4222` client | yes | **yes** | middleware, access-*, billing, worm/tree, identity, CS/sales/acct |
|
||
| `10.10.10.21–23:6222` cluster routes | yes | **yes** (private `vmbr1` only) | nats-a/b/c to each other |
|
||
| `10.10.10.21–23:7422` leaf hub | yes (public-edge) | **yes** | public edge remotes |
|
||
| `127.0.0.1:8222` monitor on each LXC | yes | **yes** | operators on the guest |
|
||
| `70.88.205.138:4222` public client | yes (token) | **yes** | leaf edge; **not** Zapier |
|
||
| `127.0.0.1:14422` WS on edge | yes | **yes** | Caddy → `wss://nats.zapier.georgelambert.org` |
|
||
| `127.0.0.1:8229` edge monitor | yes | **yes** | host loopback |
|
||
| `:6222` / `:7422` / `:8222` on `vmbr0` | **no** | **must stay closed** | — |
|
||
| Host leftover `127.0.0.1:4222` (docker-proxy) + `10.10.10.1:4222` socat | lab leftover | present | **do not** point product `NATS_URL` here |
|
||
| MQTT `:1883` | study only | **removed** | — |
|
||
|
||
Internal keep/fleet/middleware URL:
|
||
|
||
```text
|
||
nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
|
||
```
|
||
|
||
Public (token in `/etc/nats/public-edge.token` on NS1): `nats://<token>@70.88.205.138:4222`. Isolation on the hub is still `$G` + private LAN. nkeys/mTLS sketched in `verae-nats-accounts`, **not** flipped.
|
||
|
||
## JetStream streams (running)
|
||
|
||
Created by `verae-nats-cluster/scripts/ensure-streams.sh` (middleware `ensureStreams` also creates the first three).
|
||
|
||
| Stream | Subjects | Replicas | Storage | Who consumes |
|
||
|--------|----------|----------|---------|--------------|
|
||
| `ZAPIER_JOBS` | `verae.zapier.jobs.watch` | 3 | file | job-poller (in middleware process), Q `job-poller` |
|
||
| `ZAPIER_EVENTS` | `verae.zapier.jobs.events` | 3 | file | middleware wait + webhook router (`event-webhook-router`); optional exclusive `jobs-events` |
|
||
| `ZAPIER_WEBHOOKS` | `verae.zapier.webhooks.deliver` | 3 | file | webhook-deliver (in middleware), Q `webhook-deliver` |
|
||
| `ZAPIER_USAGE` | `verae.zapier.usage` | 3 | file | **no durable consumer** (`USAGE_WRITER` named only) |
|
||
| `VERAE_ARCHIVE` | `verae.archive.>` | 3 | file | worm + tree on LXC 510 (keep); aggregator **not** fleet-enabled |
|
||
|
||
Billing is **core request-reply / pub**, not a JetStream stream.
|
||
|
||
## Address inventory
|
||
|
||
| Address | Kind | Designed | Running | Notes |
|
||
|---------|------|----------|---------|-------|
|
||
| `verae.zapier.jobs.watch` | JS work Q `job-poller` | yes | **yes** | middleware publishes; in-process poller consumes |
|
||
| `verae.zapier.jobs.events` | JS events | yes | **yes** | poller publishes; wait() subscribes; webhook worker routes |
|
||
| `verae.zapier.webhooks.deliver` | JS work Q `webhook-deliver` | yes | **yes** | same Node process as HTTP |
|
||
| `verae.zapier.usage` | JS optional | yes | stream **yes**, consumer **no** | middleware entitlement is **local files** |
|
||
| `verae.internal.jobs.events` | core republish | yes (`jobs-events` exclusive) | **partial** | only if `JOBS_EVENTS_EXCLUSIVE=1` |
|
||
| `verae.billing.statement.get` | RR Q `account-balance` | yes | **yes** | portal, CS, sales, accounting, access-web/api |
|
||
| `verae.billing.balance.get` | RR | yes | **yes** | account-balance |
|
||
| `verae.billing.balance.adjust` | RR | yes | **yes** | CS credits, portal reload |
|
||
| `verae.billing.usage.recorded` | pub | yes | **yes if edge `NATS_URL` set** | `zappier/src/app.ts` `onUsage` → `natsPublish`; account-balance subscribes |
|
||
| `verae.billing.payment.recorded` | pub | yes | **yes if `NATS_URL`** | portal reload (`portal.ts`) |
|
||
| `verae.billing.credit.applied` | pub | yes | **yes if `NATS_URL`** | admin/CS (`admin.ts`) |
|
||
| `verae.billing.customer.put` | pub | yes | **yes if `NATS_URL`** | `ledger.ts`; account-balance applies |
|
||
| `verae.access.authz.check` | RR Q `access-authz` | yes | **yes** | fleet keepFloor |
|
||
| `verae.access.authz.deny` | pub audit | yes | **yes** (on deny) | no SIEM sink |
|
||
| `verae.access.web.in` / `verae.access.web.billing.*` | ingress | yes | **yes** | mapped after authz, e.g. `verae.access.web.billing.statement.get` |
|
||
| `verae.access.api.*` | ingress | yes | **yes** | |
|
||
| `verae.access.zapier.*` | ingress | yes | **yes** | HTTPS plane; Zapier never speaks NATS |
|
||
| `verae.access.leaf.in` / `verae.access.leaf.*` | ingress | yes | **yes** | no billing |
|
||
| `verae.access.staff.*` | ingress | yes | **yes** | CS/sales/accounting |
|
||
| `verae.identity.bind` | RR Q `identity` | yes | **yes** | fleet |
|
||
| `verae.identity.lookup` | RR Q `identity` | yes | **yes** | |
|
||
| `verae.archive.put` | JS (`VERAE_ARCHIVE`) | yes | **partial** | splitter **in-process** in middleware; worm/tree on 510 consume |
|
||
| `verae.archive.query` | broadcast (no Q) | yes | **partial** | live aggregator hook **TBD** in middleware (simulator has it) |
|
||
| `verae.archive.reply.<id>` | replies on bloom hit | yes | **partial** | worm/tree on 510 can reply; aggregator not fleet-enabled |
|
||
| `verae.splitter.in` | future | yes (docs) | **no** | splitter stays a library |
|
||
| `verae.example.process.in` / `.out` / `.reply.*` | template | yes | **no** | `verae-nats-process` is a model |
|
||
| `bench.*` | lab only | n/a | ephemeral | speed studies; not product |
|
||
|
||
Plane mapping (`verae-access-authz`): internal `verae.billing.statement.get` + plane `web` → ingress `verae.access.web.billing.statement.get`. Planes never publish internal subjects until `authz.check` returns allow.
|
||
|
||
## Durable consumers (named in middleware)
|
||
|
||
| Name | Stream / subject | Running |
|
||
|------|------------------|---------|
|
||
| `job-poller` | `ZAPIER_JOBS` / `jobs.watch` | **yes** (in-process) |
|
||
| `event-webhook-router` | `ZAPIER_EVENTS` / `jobs.events` | **yes** (in-process) |
|
||
| `webhook-deliver` | `ZAPIER_WEBHOOKS` / `webhooks.deliver` | **yes** (in-process) |
|
||
| `usage-writer` | `ZAPIER_USAGE` / `verae.zapier.usage` | **no** |
|
||
| `jobs-events-exclusive` | `ZAPIER_EVENTS` | **no** unless `JOBS_EVENTS_EXCLUSIVE=1` |
|
||
|
||
## Fleet overlay vs in-process
|
||
|
||
`packages/verae-fleet/overlays/ns1/fleet.json` (2026-09-14):
|
||
|
||
| Service | fleet `enabled` | Actual lab |
|
||
|---------|-----------------|------------|
|
||
| middleware-http | true | **running**; starts job-poller + webhook workers in-process when `NATS_ENABLED` |
|
||
| job-poller | **false** | covered by middleware; do not spawn a second copy |
|
||
| webhook-deliver | **false** | same |
|
||
| archive-aggregator | **false** | **not** running as a fleet worker |
|
||
| archive-worm / tree-node | **false** on host fleet | **running on LXC 510** via keep `units.px-worker.json` (1 worm + 1 tree) |
|
||
| account-balance, access-*, identity, jobs-events, CS/sales/acct, staff-iam/session, zappier-edge | true keepFloor | keepFloor |
|
||
| zapier-simulator | true, keepFloor false | optional |
|
||
| nats | managed false | 3-node cluster + leftover host loopback |
|