George Lambert
a73875f4ef
Some checks are pending
offline / test (push) Waiting to run
Fleet/keep/middleware NATS_URL lists 10.10.10.21–23. Product streams are replicas=3. px-worker worm/tree run on that cluster. Backup timer and px-worker uptime watch are on. MOCK_VERAE and Zapier push unchanged.
46 lines
2.7 KiB
Markdown
46 lines
2.7 KiB
Markdown
# 3. Central Verae NATS.IO 3-server cluster
|
||
|
||
All durable messaging for this product is **NATS JetStream**, not Zapier queues and not a public TCP API.
|
||
|
||
## Cluster
|
||
|
||
Three `nats-server -js` nodes form the Verae cluster (routes between them, JetStream replication). Clients (middleware, poller, aggregator, WORM, tree nodes, account-balance, zappier-edge billing, CS/sales/accounting, and any new process cloned from `verae-nats-process`) connect with a **cluster URL list**, for example:
|
||
|
||
```text
|
||
nats://127.0.0.1:4222,nats://127.0.0.1:4223,nats://127.0.0.1:4224
|
||
```
|
||
|
||
On a given machine the listener stays on **loopback** (or a private interface). Operators reach it with `scripts/nats-tunnel.sh` / `ssh -L 14222:127.0.0.1:4222`. **Do not bind 4222 on `0.0.0.0` without auth.**
|
||
|
||
Today’s NS1 box (`NS1.GEORGELAMBERT.ORG`, `70.88.205.138`) still runs the **lab** JetStream on `127.0.0.1:4222` (keep/fleet/middleware). The **test cluster** is three distinct Proxmox LXC guests on `vmbr1` ([verae-nats-cluster](https://git.georgelambert.org/marchon/verae-nats-cluster)):
|
||
|
||
```text
|
||
nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
|
||
```
|
||
|
||
(`nats-a` VMID 511, `nats-b` 512, `nats-c` 513). Cluster routes on `:6222`. Nothing on `vmbr0`. **Test env is cut over:** fleet/keep/middleware `NATS_URL` lists all three; product streams `ZAPIER_*` and `VERAE_ARCHIVE` are `replicas=3`. Same configs later move to separate hardware. Fleet SSH hosts run **workers**, not extra public NATS listeners.
|
||
|
||
|
||
|
||
## Who may connect
|
||
|
||
| Allowed | Forbidden |
|
||
|---------|-----------|
|
||
| access-authz, access-leaf (after token), middleware, fleet workers, WORM, tree, account-balance, access planes, CS/sales | Zapier cloud, customer browsers (HTTPS to access-web / access-api / access-zapier only) |
|
||
|
||
## Address families already in use
|
||
|
||
| Address | Kind | Notes |
|
||
|---------|------|--------|
|
||
| `verae.zapier.jobs.watch` | work queue | poller |
|
||
| `verae.zapier.jobs.events` | events | wait + webhooks |
|
||
| `verae.zapier.webhooks.deliver` | work queue | HTTPS to Zapier REST Hook |
|
||
| `verae.zapier.usage` | optional | metering |
|
||
| `verae.billing.statement.get` | request-reply | portal, CS, sales, admin |
|
||
| `verae.billing.balance.adjust` | request-reply | CS credits, portal reload |
|
||
| `verae.billing.usage.recorded` | pub | zappier-edge meter |
|
||
| `verae.archive.put` | JetStream | splitter / merkle builder |
|
||
| `verae.archive.query` | **broadcast** (no queue group) | every WORM and tree node |
|
||
| `verae.archive.reply.<correlationId>` | replies | **only on bloom hit** |
|
||
|
||
New functions get new `verae.<area>.<resource>.<action>` addresses — see [06-address-routing.md](06-address-routing.md).
|