Test env NATS cluster cut-over

This commit is contained in:
George Lambert 2026-09-12 00:00:03 -04:00
commit a5b5612e84
15 changed files with 986 additions and 0 deletions

131
GETTING-STARTED.md Normal file
View file

@ -0,0 +1,131 @@
# Bring Verae Time × Zapier online
This is the **top document** for install, first-up, and monitoring. Platform-specific detail lives in the numbered ops chapters; do not start there.
**Catalog:** https://zapier.georgelambert.org/packages/verae-ops/GETTING-STARTED.pdf
**Billing / users (zappier-edge):** https://zapier.georgelambert.org/packages/zappier/docs/USER-MANUAL.pdf
**Operator console:** http://127.0.0.1:3850/ (loopback) · https://zapier.georgelambert.org/packages/verae-fleet/docs/CONSOLE.pdf
## What you are turning on
| Piece | Job | Default |
|-------|-----|---------|
| **access-authz** | NATS authorization step for every access plane | `:3020` |
| **access-web / api / leaf / zapier / staff** | Public HTTP doors | `:3021:3025` |
| **identity** | `veraeUserId` bind/lookup | `:3026` |
| **zappier-edge** | Identity cache, Stripe, admin (loopback only) | `127.0.0.1:3000` (lab; NS1 all-in-one uses `:13000` if `:3000` is taken) |
| **staff-iam** | Named staff users, roles, sessions | `:3028` |
| **staff-session** | Staff cookie (proxies IAM) | `:3027` |
| **verae-keep** | Restart crashed units unless admin paused/stopped | keep `:3860`, watch `:3861` |
| **jobs-events** | Mailbox `verae.zapier.jobs.events` | `:3030` |
| **account-balance** | NATS prepaid balances (SoT) | `:3010` |
| **customer-service** | CS credit additions + review | `:3011` |
| **sales-pricing** | Per-customer sales quotes | `:3012` |
| **accounting-export** | QuickBooks IIF / CSV | `:3013` |
| **verae-middleware** | Zapier `/zapier/v1`, chain, NATS | `:3100` |
| **NATS JetStream** | Private jobs / archive | `:4222` **not public** |
| **worm ×3, tree-node ×3** | Off-chain files and Merkle leaves | fleet floor |
| **fleet** | Replica floors, SSH hosts, Trace, Docs | `:3850` on `0.0.0.0` |
Zapier cloud talks **only** to **access-zapier** (`:3024`) over HTTPS. Browsers use access-web. Customer API keys use access-api. Staff use access-staff. zappier-edge is loopback.
The billing and user platform is the standalone **zappier** project (`/Users/marchon/zappier`, Forgejo [zappier-edge](https://git.georgelambert.org/marchon/zappier-edge)): portal signup, TOTP, API keys, rate card, Stripe meter, PO invoices, admin users. Verae middleware does **not** replace it.
## 15-minute lab (one machine)
NS1 is Proxmox. Extra worm/tree LXC: [verae-proxmox](https://git.georgelambert.org/marchon/verae-proxmox) `create-worker.sh` (VMID 510, `10.10.10.20`). Off-box watch: [verae-uptime](https://git.georgelambert.org/marchon/verae-uptime). Backup: [verae-backup](https://git.georgelambert.org/marchon/verae-backup). Tagged upgrade: [verae-deploy](https://git.georgelambert.org/marchon/verae-deploy).
Bare metal / VM (Debian or Alpine): install OS + Node first, then clone.
```bash
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-bootstrap.git
cd verae-bootstrap
bash scripts/host-deps.sh --type ns1-all-in-one
export VERAE_SRC=$HOME/verae-src
bash scripts/fetch.sh ns1-all-in-one
```
Docker lab from the monorepo:
```bash
git clone ssh://git@git.georgelambert.org:2223/marchon/master-zapier-plan-draft.git
cd master-zapier-plan-draft
export MONOREPO="$PWD"
cd packages/verae-ops
docker compose up --build
```
Check:
```bash
curl -fsS http://127.0.0.1:3000/health # zappier-edge (billing + users)
curl -fsS http://127.0.0.1:3010/health # account-balance (NATS SoT)
curl -fsS http://127.0.0.1:3011/health # customer-service
curl -fsS http://127.0.0.1:3012/health # sales-pricing
curl -fsS http://127.0.0.1:3100/health # middleware
```
Open:
- Customer portal (public door): http://127.0.0.1:3021/portal/ — sign up, copy `x-api-key`, **Statement** tab
- Loopback edge admin: http://127.0.0.1:3000/admin — **Statement** per customer (not a public door)
- Staff IAM: http://127.0.0.1:3028/
- CS review: http://127.0.0.1:3011/
- Sales review: http://127.0.0.1:3012/
- Accounting review/export: http://127.0.0.1:3013/
- API docs: http://127.0.0.1:3000/docs
Test env NATS is the **three LXC cluster** on `vmbr1`:
`nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222`
Host loopback `:4222` may still listen but modules no longer use it. Do **not** publish NATS on `vmbr0`.
## Operator console (monitor)
On the same machine (or SSH tunnel `ssh -L 3850:127.0.0.1:3850 user@host`):
```bash
cd "$MONOREPO/packages/verae-fleet"
node src/cli.js serve
```
http://0.0.0.0:3850/ — Fleet (green/yellow/red), Trace, Docs. Tree-node **min 3**; pause does not count. On NS1, `verae-keep` restarts workers if they crash unless you paused/stopped them here.
```bash
node src/cli.js status
node src/cli.js ssh-check ns1
```
## Point Zapier at the lab
1. TLS in front of `:3000` (Caddy).
2. Zapier app env: `MIDDLEWARE_BASE_URL=https://<zappier-host>` (edge meters, then proxies to middleware).
3. Connect with the portal API key.
## Production (pick one)
| Where | Read |
|-------|------|
| Docker | [02-docker.pdf](02-docker.pdf) |
| Proxmox | [03-proxmox.pdf](03-proxmox.pdf) |
| Cloud VMs | [04-virtual-servers.pdf](04-virtual-servers.pdf) |
| Dedicated / NS1 | [05-dedicated-hardware.pdf](05-dedicated-hardware.pdf) |
| Env vars & NATS | [06-linking-services.pdf](06-linking-services.pdf) |
| Backup / upgrade | [07-maintenance.pdf](07-maintenance.pdf) |
## Daily monitor
| Check | Expect |
|-------|--------|
| `curl -fsS https://zappier.example.com/health` | 200 |
| `curl -fsS https://middleware.example.com/health` | 200 |
| NATS `http://127.0.0.1:8222/healthz` on each node | ok (localhost only) |
| Fleet tree-node available ≥ 3 | green |
| Portal login + key still works | 200 `/v1/status` with `x-api-key` |
## If something is red
1. Fleet row color: yellow = paused/degraded, red = below floor — `restart` that replica.
2. 401/402 from Zapier → zappier-edge keys and rate card, not NATS.
3. Archive lookup empty → bloom miss is **silence**; check worm/tree floors, not “error replies”.