Document every repo in Sphinx, catalog PDFs, and verae-ops
Some checks are pending
offline / test (push) Waiting to run
Some checks are pending
offline / test (push) Waiting to run
Add missing package READMEs, expand Sphinx to all runtime modules, publish a LaTeX/PDF README book, and add verae-ops for Docker, Proxmox, VMs, dedicated hardware, and service linking. Catalog defaults to public PDF links for each module README and samples.
This commit is contained in:
parent
5ad3222def
commit
e401635fd6
30 changed files with 4499 additions and 124 deletions
|
|
@ -26,6 +26,7 @@ Summaries, NATS contracts, and message flows for every Verae Time × Zapier modu
|
|||
| **overview** | https://git.georgelambert.org/marchon/overview |
|
||||
| verae-nats-process | https://git.georgelambert.org/marchon/verae-nats-process |
|
||||
| **zapier-docs-master** (this repo) | https://git.georgelambert.org/marchon/zapier-docs-master |
|
||||
| **verae-ops** | https://git.georgelambert.org/marchon/verae-ops |
|
||||
|
||||
Clone (SSH port 2223):
|
||||
|
||||
|
|
@ -48,6 +49,7 @@ Clone (SSH port 2223):
|
|||
| verae-fleet | Replica floors + monitor | — | — |
|
||||
| overview | System map | — | — |
|
||||
| nats-process | Template worker | example.process.in | example.process.out / reply.* |
|
||||
| ops | Install / Docker / Proxmox / metal | — | — |
|
||||
|
||||
## Documents in this repo
|
||||
|
||||
|
|
|
|||
3
packages/docs-master/modules/verae-ops/NATS.md
Normal file
3
packages/docs-master/modules/verae-ops/NATS.md
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
# verae-ops NATS
|
||||
|
||||
Documentation only. Runtime NATS bind stays private. See https://zapier.georgelambert.org/packages/verae-ops/06-linking-services.pdf
|
||||
3
packages/docs-master/modules/verae-ops/SUMMARY.md
Normal file
3
packages/docs-master/modules/verae-ops/SUMMARY.md
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
# verae-ops
|
||||
|
||||
Install and maintenance documentation: Docker, Proxmox, VMs, dedicated hardware, linking services.
|
||||
|
|
@ -19,6 +19,7 @@ Each runtime piece is its **own git repo** on Forgejo (`git.georgelambert.org`,
|
|||
| **verae-zapier-simulator** | `packages/verae-zapier-simulator` | Trace console before `zapier-platform push` |
|
||||
| **zapier-user-docs** | `packages/zapier-user-docs` | Customer signup → register → lookup |
|
||||
| **zapier-docs-master** | `packages/docs-master` | Per-module `SUMMARY.md` + `NATS.md` |
|
||||
| **verae-ops** | `packages/verae-ops` | Docker, Proxmox, VMs, dedicated hardware, linking services |
|
||||
|
||||
Libraries that are **not** separate Forgejo apps today: `verae-chain-client` (inside middleware), job-poller and webhook-deliver (middleware workers, fleet-spawned).
|
||||
|
||||
|
|
|
|||
|
|
@ -12,7 +12,8 @@ Operator console (loopback): http://127.0.0.1:3850/ · [CONSOLE.pdf](https://zap
|
|||
| [01-system.md](01-system.md) | What runs where, in plain language |
|
||||
| [02-modules-and-repos.md](02-modules-and-repos.md) | Short names of every independent repo |
|
||||
| [zapier-docs-master](https://git.georgelambert.org/marchon/zapier-docs-master) | One-line SUMMARY + NATS per module |
|
||||
| [zapier-user-docs](https://git.georgelambert.org/marchon/zapier-user-docs) | Signup → register → lookup |
|
||||
| [zapier-user-docs](https://zapier.georgelambert.org/user-docs/README.pdf) | Signup → register → lookup |
|
||||
| [verae-ops](https://zapier.georgelambert.org/packages/verae-ops/README.pdf) | Docker / Proxmox / VM / metal install |
|
||||
|
||||
## 2. Architecture
|
||||
|
||||
|
|
@ -43,12 +44,15 @@ Operator console (loopback): http://127.0.0.1:3850/ · [CONSOLE.pdf](https://zap
|
|||
| Tree-node lookup (users) | https://zapier.georgelambert.org/packages/zapier-user-docs/09-lookup-tree-nodes.pdf |
|
||||
| Zapier developer setup | https://zapier.georgelambert.org/docs/04-activate/SETUP-ZAPIER-DEVELOPER.pdf |
|
||||
| docs-master | https://zapier.georgelambert.org/docs-master/README.pdf |
|
||||
| Sphinx HTML | https://zapier.georgelambert.org/sphinx/ |
|
||||
| Sphinx LaTeX PDF | https://zapier.georgelambert.org/sphinx/verae-zapier-modules.pdf |
|
||||
| Install (verae-ops) | https://zapier.georgelambert.org/packages/verae-ops/README.pdf |
|
||||
|
||||
## Independent git repositories
|
||||
|
||||
Prefix: `https://git.georgelambert.org/marchon/`
|
||||
|
||||
overview · verae-nats-process · master-zapier-plan-draft · zappier-edge · verae-middleware · verae-zapier-app · verae-activate · verae-request-splitter · verae-archive-worm · verae-archive-aggregator · verae-tree-node · verae-fleet · verae-zapier-simulator · zapier-user-docs · zapier-docs-master
|
||||
overview · verae-nats-process · verae-ops · master-zapier-plan-draft · zappier-edge · verae-middleware · verae-zapier-app · verae-activate · verae-request-splitter · verae-archive-worm · verae-archive-aggregator · verae-tree-node · verae-fleet · verae-zapier-simulator · zapier-user-docs · zapier-docs-master
|
||||
|
||||
## Per-module contracts
|
||||
|
||||
|
|
|
|||
|
|
@ -30,6 +30,7 @@ Zapier → HTTPS `zappier-edge` → HTTPS `verae-middleware` → **NATS cluster*
|
|||
| Repo | Role |
|
||||
|------|------|
|
||||
| [overview](https://git.georgelambert.org/marchon/overview) | This document |
|
||||
| [verae-ops](https://git.georgelambert.org/marchon/verae-ops) | Docker, Proxmox, VMs, dedicated hardware |
|
||||
| [verae-nats-process](https://git.georgelambert.org/marchon/verae-nats-process) | **Model** for a new addressed process |
|
||||
| [master-zapier-plan-draft](https://git.georgelambert.org/marchon/master-zapier-plan-draft) | Monorepo snapshot |
|
||||
| [zappier-edge](https://git.georgelambert.org/marchon/zappier-edge) | Metered public HTTPS |
|
||||
|
|
|
|||
|
|
@ -6,7 +6,11 @@ This is the integration you **push tomorrow**. It has one action:
|
|||
|
||||
It runs **inside Zapier**. You do not need `api.veraetime.net`, NATS, or a public URL.
|
||||
|
||||
Step-by-step account setup: [../../docs/04-activate/SETUP-ZAPIER-DEVELOPER.md](../../docs/04-activate/SETUP-ZAPIER-DEVELOPER.md)
|
||||
**Catalog:** https://zapier.georgelambert.org/packages/verae-activate/README.pdf
|
||||
**Setup (PDF):** https://zapier.georgelambert.org/docs/04-activate/SETUP-ZAPIER-DEVELOPER.pdf
|
||||
**Sample action:** `creates/add_numbers.js` — public sheet https://zapier.georgelambert.org/docs/modules/verae-activate/creates/add_numbers.pdf
|
||||
|
||||
Step-by-step account setup: [SETUP-ZAPIER-DEVELOPER](https://zapier.georgelambert.org/docs/04-activate/SETUP-ZAPIER-DEVELOPER.pdf)
|
||||
|
||||
```bash
|
||||
cd packages/verae-activate
|
||||
|
|
|
|||
29
packages/verae-archive-aggregator/README.md
Normal file
29
packages/verae-archive-aggregator/README.md
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
# verae-archive-aggregator
|
||||
|
||||
When a wait path sets `includeAttached`, this process broadcasts `verae.archive.query` and **merges** replies into the job JSON. Silent bloom misses omit slices; the chain receipt still succeeds.
|
||||
|
||||
**Forgejo:** https://git.georgelambert.org/marchon/verae-archive-aggregator
|
||||
**Catalog:** https://zapier.georgelambert.org/packages/verae-archive-aggregator/README.pdf
|
||||
**NATS:** https://zapier.georgelambert.org/docs-master/modules/verae-archive-aggregator/NATS.pdf
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
cd packages/verae-archive-aggregator
|
||||
npm test
|
||||
```
|
||||
|
||||
## Addresses
|
||||
|
||||
| Direction | Subject |
|
||||
|-----------|---------|
|
||||
| out | `verae.archive.query` `{ correlationId, sha256, tenantId, kinds[] }` |
|
||||
| in | `verae.archive.reply.<correlationId>` (hits only) |
|
||||
|
||||
## Depends on
|
||||
|
||||
- middleware wait path (`includeAttached` / `includeTree`)
|
||||
- one or more archive-worm and/or tree-node replicas
|
||||
- NATS (private)
|
||||
|
||||
Timeout: missing archives omit their records; do not fail the seal.
|
||||
34
packages/verae-archive-worm/README.md
Normal file
34
packages/verae-archive-worm/README.md
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
# verae-archive-worm
|
||||
|
||||
Bloom-filtered append-only WORM node. Stores metadata and files **off chain**. A bloom **miss is silence** (no NATS reply).
|
||||
|
||||
**Forgejo:** https://git.georgelambert.org/marchon/verae-archive-worm
|
||||
**Catalog:** https://zapier.georgelambert.org/packages/verae-archive-worm/README.pdf
|
||||
**NATS:** https://zapier.georgelambert.org/docs-master/modules/verae-archive-worm/NATS.pdf
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
cd packages/verae-archive-worm
|
||||
npm test
|
||||
```
|
||||
|
||||
Fleet keeps **min 3** copies (`keepFloor`). Pause does not count as available.
|
||||
|
||||
## Addresses
|
||||
|
||||
| Subject | Role |
|
||||
|---------|------|
|
||||
| `verae.archive.put` | Append `{ sha256, kind, record }` and add the hash to the bloom |
|
||||
| `verae.archive.query` | Broadcast (no queue group). Hit → `verae.archive.reply.<correlationId>` |
|
||||
| miss | **no message** |
|
||||
|
||||
Kinds: `publicMeta`, `privateMeta`, `file`. Tree leaves live on **verae-tree-node** (`kind=tree`).
|
||||
|
||||
## Depends on
|
||||
|
||||
- NATS JetStream reachable on the **private** URL (`NATS_URL`)
|
||||
- archive-aggregator (query fan-out / merge)
|
||||
- fleet replica floor
|
||||
|
||||
Does not talk to Zapier or the chain.
|
||||
|
|
@ -39,4 +39,6 @@ Operator console (Fleet · Trace · Docs) at http://127.0.0.1:3850/ — [docs/CO
|
|||
|
||||
Zapier cloud apps are listed but **not spawned**. NATS on NS1 is **monitored only** (loopback `:4222`, never a public bind).
|
||||
|
||||
Install on Docker / Proxmox / metal: https://zapier.georgelambert.org/packages/verae-ops/README.pdf
|
||||
|
||||
Clone: `ssh://git@git.georgelambert.org:2223/marchon/verae-fleet.git`
|
||||
|
|
|
|||
|
|
@ -4,7 +4,8 @@ Template for a new **addressed messaging process** on the central Verae NATS.IO
|
|||
|
||||
**Repo:** https://git.georgelambert.org/marchon/verae-nats-process
|
||||
**Clone:** `ssh://git@git.georgelambert.org:2223/marchon/verae-nats-process.git`
|
||||
**Overview:** https://git.georgelambert.org/marchon/overview
|
||||
**Overview:** https://zapier.georgelambert.org/overview/README.pdf
|
||||
**Routing:** https://zapier.georgelambert.org/packages/verae-nats-process/ROUTING.pdf
|
||||
|
||||
```bash
|
||||
npm test
|
||||
|
|
@ -17,4 +18,4 @@ npm start # /health on 127.0.0.1:13900
|
|||
4. Update `ROUTING.md`; add a fleet service with `min`/`max`.
|
||||
5. If Zapier needs the result, add HTTPS on **verae-middleware** only.
|
||||
|
||||
Zapier cloud must not subscribe. See [overview 06-address-routing](https://git.georgelambert.org/marchon/overview/src/branch/main/06-address-routing.md).
|
||||
Zapier cloud must not subscribe. See [address routing](https://zapier.georgelambert.org/overview/06-address-routing.pdf).
|
||||
|
|
|
|||
64
packages/verae-ops/01-dependencies.md
Normal file
64
packages/verae-ops/01-dependencies.md
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
# 1. Dependencies and data flow
|
||||
|
||||
```text
|
||||
Humans / Zapier cloud
|
||||
│ HTTPS x-api-key
|
||||
▼
|
||||
zappier-edge :3000 (meter, portal, admin, Stripe)
|
||||
│ HTTPS internal
|
||||
▼
|
||||
middleware-http :3100 (/zapier/v1)
|
||||
│
|
||||
├── HTTPS → api.veraetime.net (or MOCK_VERAE)
|
||||
└── NATS → private cluster
|
||||
jobs.watch / jobs.events
|
||||
webhooks.deliver
|
||||
archive.put / query / reply.*
|
||||
│
|
||||
archive-worm × N tree-node × N
|
||||
archive-aggregator job-poller
|
||||
webhook-deliver
|
||||
```
|
||||
|
||||
## Repositories (clone all that you run)
|
||||
|
||||
Prefix `ssh://git@git.georgelambert.org:2223/marchon/<name>.git` (SSH **2223**).
|
||||
|
||||
| You need | Repo | Catalog README |
|
||||
|----------|------|----------------|
|
||||
| Billing HTTPS | `zappier-edge` | https://zapier.georgelambert.org/packages/zappier/README.pdf |
|
||||
| Zapier adapter | `verae-middleware` | https://zapier.georgelambert.org/packages/verae-zapier-middleware/README.pdf |
|
||||
| Zapier app (cloud) | `verae-zapier-app` | https://zapier.georgelambert.org/packages/verae-zapier/README.pdf |
|
||||
| WORM nodes | `verae-archive-worm` | https://zapier.georgelambert.org/packages/verae-archive-worm/README.pdf |
|
||||
| Merkle leaves | `verae-tree-node` | https://zapier.georgelambert.org/packages/verae-tree-node/README.pdf |
|
||||
| Archive merge | `verae-archive-aggregator` | https://zapier.georgelambert.org/packages/verae-archive-aggregator/README.pdf |
|
||||
| Splitter | `verae-request-splitter` | https://zapier.georgelambert.org/packages/verae-request-splitter/README.pdf |
|
||||
| Replica floors | `verae-fleet` | https://zapier.georgelambert.org/packages/verae-fleet/README.pdf |
|
||||
| This install guide | `verae-ops` | https://zapier.georgelambert.org/packages/verae-ops/README.pdf |
|
||||
|
||||
Optional: `verae-activate` (tiny pushable app), `verae-zapier-simulator` (trace), `overview`, `zapier-docs-master`, `zapier-user-docs`, `verae-nats-process`.
|
||||
|
||||
## Runtime dependencies
|
||||
|
||||
| Service | Needs |
|
||||
|---------|--------|
|
||||
| zappier-edge | Node 20+, SQLite path or volume, `PORT`, optional Stripe keys |
|
||||
| middleware | Node 22+, `NATS_URL` **or** `NATS_ENABLED=false`, chain URL or `MOCK_VERAE` |
|
||||
| NATS | JetStream, **no public bind** |
|
||||
| worm / tree-node | Node, `NATS_URL`, local state dir |
|
||||
| fleet | Node, `machines.json` (SSH **paths**), loopback `:3850` |
|
||||
| Zapier cloud app | `MIDDLEWARE_BASE_URL` = **public HTTPS of zappier-edge** (or middleware if you skip metering in lab) |
|
||||
|
||||
## Ports (lab defaults)
|
||||
|
||||
| Port | Bind | Process |
|
||||
|------|------|---------|
|
||||
| 3000 | public or docker publish | zappier-edge |
|
||||
| 3100 | public or docker publish | middleware |
|
||||
| 4222 | **private only** | NATS client |
|
||||
| 6222 | **private only** | NATS cluster |
|
||||
| 8222 | private / localhost | NATS monitor |
|
||||
| 3847 | loopback | simulator |
|
||||
| 3850 | loopback | operator console |
|
||||
|
||||
Chain: `https://api.veraetime.net` (prod) or `MOCK_VERAE=true` (lab).
|
||||
54
packages/verae-ops/02-docker.md
Normal file
54
packages/verae-ops/02-docker.md
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
# 2. Docker Compose
|
||||
|
||||
`docker-compose.yml` in this repo starts:
|
||||
|
||||
- NATS JetStream **3-node cluster** (`nats1`–`nats3`) on an internal network
|
||||
- **zappier-edge** `:3000`
|
||||
- **middleware** `:3100` with `NATS_URL=nats://nats1:4222` and `MOCK_VERAE=true` by default
|
||||
- **archive-worm** ×3 and **tree-node** ×3 (fleet floors) as long-running Node workers
|
||||
- **aggregator** + **job-poller** stand-ins via middleware workers when `NATS_ENABLED=true`
|
||||
|
||||
NATS ports are **not** published to the host by default (private `verae` network). HTTPS edges are.
|
||||
|
||||
## Prereqs
|
||||
|
||||
Docker Engine 24+ and Compose v2. Clone **this repo** plus the application repos as siblings, **or** set `MONOREPO` to a checkout of `master-zapier-plan-draft`.
|
||||
|
||||
```bash
|
||||
export MONOREPO=/path/to/master-zapier-plan-draft # contains packages/
|
||||
docker compose up --build
|
||||
curl -fsS http://127.0.0.1:3000/health
|
||||
curl -fsS http://127.0.0.1:3100/health
|
||||
```
|
||||
|
||||
Without `MONOREPO`, Compose expects `../zappier`, `../verae-zapier-middleware`, … (independent clones next to `verae-ops`).
|
||||
|
||||
## Link Zapier (cloud) to the lab
|
||||
|
||||
1. Publish zappier-edge on a TLS hostname (Caddy/nginx in front of `:3000`).
|
||||
2. In the Zapier app env: `MIDDLEWARE_BASE_URL=https://<zappier-host>` (edge meters, then proxies).
|
||||
3. Sign up on `/portal`, copy `x-api-key`.
|
||||
|
||||
## Scale
|
||||
|
||||
```bash
|
||||
docker compose up --scale archive-worm=3 --scale tree-node=3
|
||||
```
|
||||
|
||||
Do not publish `4222`. To debug NATS from the host:
|
||||
|
||||
```bash
|
||||
docker compose exec nats1 wget -qO- http://127.0.0.1:8222/healthz
|
||||
```
|
||||
|
||||
## Operator console
|
||||
|
||||
Fleet is **not** inside Compose by default (it SSH-spawns). On the Docker host:
|
||||
|
||||
```bash
|
||||
cd $MONOREPO/packages/verae-fleet
|
||||
NATS_URL=nats://127.0.0.1:4222 # only if you published 4222 on localhost for lab
|
||||
node src/cli.js serve # http://127.0.0.1:3850/ loopback
|
||||
```
|
||||
|
||||
For Compose-only labs, skip fleet; replica counts come from `--scale`.
|
||||
60
packages/verae-ops/03-proxmox.md
Normal file
60
packages/verae-ops/03-proxmox.md
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
# 3. Proxmox (LXC and QEMU)
|
||||
|
||||
Target: one **NATS cluster VM/CT** (or three), one **HTTPS edge** VM, one or more **worker** VMs. Fleet SSH-spreads worm/tree-node onto worker hosts.
|
||||
|
||||
## Suggested guests
|
||||
|
||||
| Guest | OS | vCPU / RAM / disk | Role |
|
||||
|-------|----|-------------------|------|
|
||||
| `nats-a` `nats-b` `nats-c` | Debian 12 LXC unprivileged | 1 / 1G / 8G | JetStream, bind `127.0.0.1` **or** a vmbr **private** bridge only |
|
||||
| `edge` | Debian 12 QEMU | 2 / 2G / 20G | Caddy + zappier-edge + middleware |
|
||||
| `worker-1` … | Debian 12 LXC | 2 / 2G / 20G | fleet SSH target; worm + tree-node |
|
||||
|
||||
Use a **private bridge** (`vmbr1`, RFC1918) for NATS `4222`/`6222`. Public `vmbr0` only on `edge` (80/443).
|
||||
|
||||
## NATS on an LXC
|
||||
|
||||
```bash
|
||||
apt-get update && apt-get install -y curl
|
||||
# official nats-server binary
|
||||
curl -sSL https://github.com/nats-io/nats-server/releases/latest/download/nats-server-v2.10.26-linux-amd64.tar.gz | tar -xz
|
||||
install nats-server /usr/local/bin/
|
||||
```
|
||||
|
||||
`/etc/nats/nats.conf`:
|
||||
|
||||
```
|
||||
jetstream {}
|
||||
listen: 10.10.10.11:4222 # private bridge, not 0.0.0.0 on public
|
||||
cluster {
|
||||
name: verae
|
||||
listen: 10.10.10.11:6222
|
||||
routes: [ nats-route://10.10.10.12:6222, nats-route://10.10.10.13:6222 ]
|
||||
}
|
||||
http: 127.0.0.1:8222
|
||||
```
|
||||
|
||||
systemd: `ExecStart=/usr/local/bin/nats-server -c /etc/nats/nats.conf`.
|
||||
|
||||
## Edge VM
|
||||
|
||||
Install Node 22, clone `zappier-edge` and `verae-middleware`, Caddy reverse-proxy:
|
||||
|
||||
```
|
||||
zappier.example.com {
|
||||
reverse_proxy 127.0.0.1:3000
|
||||
}
|
||||
middleware.example.com {
|
||||
reverse_proxy 127.0.0.1:3100
|
||||
}
|
||||
```
|
||||
|
||||
Middleware env: `NATS_URL=nats://10.10.10.11:4222,nats://10.10.10.12:4222,nats://10.10.10.13:4222`.
|
||||
|
||||
## Fleet on Proxmox
|
||||
|
||||
On the operator workstation or a jump CT (loopback `:3850`):
|
||||
|
||||
`machines.json` entries: `kind=ssh`, `user=root` or `marchon`, `host=10.10.10.2x`, `identityFile=~/.ssh/id_ed25519` (**path only**). Roles: `tree-node,archive-worm`. Capacity ≥ 3 per worker if you want floors on one host.
|
||||
|
||||
`ssh-check` from the operator console before enabling `keepFloor`.
|
||||
28
packages/verae-ops/04-virtual-servers.md
Normal file
28
packages/verae-ops/04-virtual-servers.md
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
# 4. Virtual servers (cloud / hypervisor)
|
||||
|
||||
Same topology as Proxmox, without the Proxmox UI: three small VMs or one VM for lab.
|
||||
|
||||
## Single VM lab
|
||||
|
||||
Install Docker, clone `verae-ops`, `docker compose up`. Put Caddy/nginx TLS in front of `:3000` and `:3100`. Do **not** open 4222 on the security group.
|
||||
|
||||
## Three-VM prod-shaped
|
||||
|
||||
| VM | Security group | Software |
|
||||
|----|----------------|----------|
|
||||
| nats | private SG only 4222/6222 between the three | nats-server -js cluster |
|
||||
| edge | 80/443 public; 4222 egress to nats SG | zappier + middleware + Caddy |
|
||||
| workers | private; SSH from operator IP | Node; fleet identity file |
|
||||
|
||||
Cloud-init example (edge):
|
||||
|
||||
```yaml
|
||||
packages: [curl, git, ca-certificates]
|
||||
runcmd:
|
||||
- curl -fsSL https://deb.nodesource.com/setup_22.x | bash -
|
||||
- apt-get install -y nodejs
|
||||
```
|
||||
|
||||
Set `MIDDLEWARE_BASE_URL` in Zapier to the **public zappier hostname**.
|
||||
|
||||
Snapshots: SQLite (`ZAPPIER_DB`) and worm/tree state directories. NATS JetStream store (`/var/lib/nats`).
|
||||
59
packages/verae-ops/05-dedicated-hardware.md
Normal file
59
packages/verae-ops/05-dedicated-hardware.md
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
# 5. Dedicated hardware
|
||||
|
||||
NS1-style: a box that already runs NATS on **127.0.0.1:4222** (never a public bind), plus optional extra hosts for SSH workers.
|
||||
|
||||
## This host (control + edges)
|
||||
|
||||
1. Node 22+, git (SSH port **2223** to Forgejo).
|
||||
2. `nats-server -js` listening **127.0.0.1:4222** only (cluster later on a private NIC).
|
||||
3. Clone `zappier-edge`, `verae-middleware`, `verae-fleet`.
|
||||
4. systemd units (below) or `verae-fleet serve` which keepFloor-spawns workers.
|
||||
5. Caddy/nginx for public HTTPS to `:3000` / `:3100`.
|
||||
6. Operator console: `127.0.0.1:3850` — SSH tunnel if you are off-box:
|
||||
`ssh -L 3850:127.0.0.1:3850 -L 4222:127.0.0.1:4222 user@ns1`
|
||||
|
||||
## systemd (middleware)
|
||||
|
||||
```
|
||||
[Unit]
|
||||
Description=verae-middleware
|
||||
After=network.target nats.service
|
||||
|
||||
[Service]
|
||||
WorkingDirectory=/opt/verae/verae-middleware
|
||||
Environment=PORT=3100
|
||||
Environment=NATS_ENABLED=true
|
||||
Environment=NATS_URL=nats://127.0.0.1:4222
|
||||
Environment=VERAE_API_BASE_URL=https://api.veraetime.net
|
||||
ExecStart=/usr/bin/node src/index.js
|
||||
Restart=on-failure
|
||||
User=verae
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
zappier-edge similar with `PORT=3000` and `WorkingDirectory` of that clone.
|
||||
|
||||
## Extra machines
|
||||
|
||||
Add to fleet `machines.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "ns1",
|
||||
"kind": "ssh",
|
||||
"user": "marchon",
|
||||
"host": "70.88.205.138",
|
||||
"port": 22,
|
||||
"identityFile": "~/.ssh/id_ed25519",
|
||||
"capacity": 8,
|
||||
"roles": ["tree-node", "archive-worm"]
|
||||
}
|
||||
```
|
||||
|
||||
Never commit private key bytes. Optional `machines.secrets.json` is gitignored.
|
||||
|
||||
## NATS 3-server cluster on metal
|
||||
|
||||
Three boxes, private IPs, same `cluster_name`, routes to each other, `listen` on the private IP only. Clients (`NATS_URL`) list all three URLs. Target architecture: https://zapier.georgelambert.org/overview/03-nats-cluster.pdf
|
||||
57
packages/verae-ops/06-linking-services.md
Normal file
57
packages/verae-ops/06-linking-services.md
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
# 6. Linking dependent services
|
||||
|
||||
Every process learns peers from **environment variables** and **fleet service JSON**. There is no service mesh.
|
||||
|
||||
## Environment map
|
||||
|
||||
| Process | Variable | Points at |
|
||||
|---------|----------|-----------|
|
||||
| Zapier cloud app | `MIDDLEWARE_BASE_URL` | Public **zappier-edge** HTTPS (prod) or middleware (lab) |
|
||||
| zappier-edge | `ZAPPIER_UPSTREAM` or internal proxy target | middleware `:3100` |
|
||||
| middleware | `NATS_URL` | `nats://nats1:4222,nats://nats2:4222,nats://nats3:4222` |
|
||||
| middleware | `VERAE_API_BASE_URL` | `https://api.veraetime.net` |
|
||||
| middleware | `MOCK_VERAE` | `true` only in lab |
|
||||
| worm / tree-node / aggregator / poller | `NATS_URL` | same cluster |
|
||||
| fleet | `NATS_URL` (monitor only) | same; **publicBind=false** |
|
||||
| fleet SSH | `identityFile` | path on the operator disk |
|
||||
|
||||
## HTTPS hops (must work)
|
||||
|
||||
```text
|
||||
curl -H "x-api-key: $KEY" https://zappier.example.com/v1/status
|
||||
# edge meters, then
|
||||
curl http://127.0.0.1:3100/health
|
||||
```
|
||||
|
||||
If Zapier gets 401/402, fix keys and the rate card on zappier-edge — not NATS.
|
||||
|
||||
## NATS hops (must stay private)
|
||||
|
||||
| Address | From | To |
|
||||
|---------|------|----|
|
||||
| `verae.zapier.jobs.watch` | middleware | job-poller (queue) |
|
||||
| `verae.zapier.jobs.events` | poller | middleware wait |
|
||||
| `verae.zapier.webhooks.deliver` | middleware | webhook-deliver |
|
||||
| `verae.archive.put` | splitter | worm / tree-node |
|
||||
| `verae.archive.query` | aggregator | **all** worm/tree (broadcast) |
|
||||
| `verae.archive.reply.<id>` | worm/tree on **hit only** | aggregator |
|
||||
|
||||
Bloom **miss = silence**. Do not “fix” that with a negative reply.
|
||||
|
||||
## Fleet floors
|
||||
|
||||
https://zapier.georgelambert.org/docs/02-architecture/fleet.pdf
|
||||
|
||||
| Service | min (keepFloor) |
|
||||
|---------|-----------------|
|
||||
| archive-worm | 3 |
|
||||
| tree-node | 3 |
|
||||
| middleware-http | 1 |
|
||||
| zappier-edge | 1 |
|
||||
| aggregator / poller / webhook | 1 |
|
||||
|
||||
Link a new host: operator console → Add machine → `ssh-check` → reconcile.
|
||||
|
||||
## Adding an unplanned function
|
||||
|
||||
Copy `verae-nats-process`, pick `verae.<area>.<resource>.*`, add a fleet service, and if Zapier must see the result, add **one HTTPS route on middleware**. Do not teach Zapier NATS. https://zapier.georgelambert.org/overview/06-address-routing.pdf
|
||||
45
packages/verae-ops/07-maintenance.md
Normal file
45
packages/verae-ops/07-maintenance.md
Normal file
|
|
@ -0,0 +1,45 @@
|
|||
# 7. Maintenance
|
||||
|
||||
## Health
|
||||
|
||||
| Check | Command |
|
||||
|-------|---------|
|
||||
| Edges | `curl -fsS https://zappier.example.com/health` and middleware `/health` |
|
||||
| NATS | `curl -fsS http://127.0.0.1:8222/healthz` on each nats node (localhost) |
|
||||
| Fleet | `node src/cli.js status` or http://127.0.0.1:3850/ (loopback) |
|
||||
| Tree floor | status: tree-node available ≥ min, none of those paused |
|
||||
|
||||
Green / yellow / red on the operator console: https://zapier.georgelambert.org/packages/verae-fleet/docs/CONSOLE.pdf
|
||||
|
||||
## Restart
|
||||
|
||||
- systemd: `systemctl restart verae-middleware zappier-edge`
|
||||
- Compose: `docker compose up -d --no-deps middleware`
|
||||
- One replica: fleet `restart tree-node-1` (floor spawns a replacement if needed)
|
||||
- Pause is **not** off — paused copies do not count toward `keepFloor`
|
||||
|
||||
## Backup
|
||||
|
||||
| Data | Where |
|
||||
|------|--------|
|
||||
| zappier SQLite | `ZAPPIER_DB` path / volume |
|
||||
| worm / tree blobs | instance `data/` dirs |
|
||||
| NATS JetStream | nats store dir |
|
||||
| fleet overlay | `packages/verae-fleet/data/` (gitignored) |
|
||||
|
||||
Do not back up private keys into git. Restore `identityFile` paths on the operator host.
|
||||
|
||||
## Upgrade
|
||||
|
||||
1. `git pull` each independent repo (or refresh the monorepo snapshot).
|
||||
2. `npm test` in that package.
|
||||
3. Restart that unit only; NATS cluster stays up.
|
||||
4. Zapier Platform app: `zapier-platform push` from `verae-zapier-app` when HTTPS contracts change.
|
||||
|
||||
## Logs
|
||||
|
||||
journald for systemd; `docker compose logs -f middleware nats1`. Redact API keys (middleware debug redaction).
|
||||
|
||||
## Docs
|
||||
|
||||
After doc edits: from the monorepo `npm run docs:site && npm run docs:deploy` so https://zapier.georgelambert.org/ stays current (PDF default, Markdown indexes at `/index-md.html`).
|
||||
10
packages/verae-ops/NATS.md
Normal file
10
packages/verae-ops/NATS.md
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
# NATS (ops)
|
||||
|
||||
Ops does not subscribe. Runtime processes use:
|
||||
|
||||
- `NATS_URL` — comma-separated client URLs on a **private** network
|
||||
- JetStream required (`-js`)
|
||||
- Cluster routes on port 6222
|
||||
- **No public bind** of 4222
|
||||
|
||||
See [06-linking-services.md](06-linking-services.md) and https://zapier.georgelambert.org/overview/03-nats-cluster.pdf
|
||||
44
packages/verae-ops/README.md
Normal file
44
packages/verae-ops/README.md
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
# verae-ops — install, run, maintain
|
||||
|
||||
How to stand up Verae Time × Zapier on **Docker**, **Proxmox**, **generic VMs**, or **dedicated hardware**, and how to **link dependent services**.
|
||||
|
||||
**Forgejo:** https://git.georgelambert.org/marchon/verae-ops
|
||||
**Clone:** `ssh://git@git.georgelambert.org:2223/marchon/verae-ops.git`
|
||||
**Catalog:** https://zapier.georgelambert.org/packages/verae-ops/README.pdf
|
||||
**System map:** https://zapier.georgelambert.org/overview/README.pdf
|
||||
|
||||
This is the operations repo. Application code lives in the other independent git repositories.
|
||||
|
||||
## Reading order
|
||||
|
||||
1. [01-dependencies.md](01-dependencies.md) — what must exist, who talks to whom
|
||||
2. [02-docker.md](02-docker.md) — Compose (NATS 3-node cluster + HTTPS edges)
|
||||
3. [03-proxmox.md](03-proxmox.md) — LXC / QEMU VMs
|
||||
4. [04-virtual-servers.md](04-virtual-servers.md) — cloud or hypervisor VMs
|
||||
5. [05-dedicated-hardware.md](05-dedicated-hardware.md) — bare metal (NS1-style)
|
||||
6. [06-linking-services.md](06-linking-services.md) — env vars, URLs, replica floors
|
||||
7. [07-maintenance.md](07-maintenance.md) — upgrade, backup, fleet, NATS
|
||||
|
||||
Public PDFs of the same files: `https://zapier.georgelambert.org/packages/verae-ops/<name>.pdf`.
|
||||
|
||||
## Non-negotiables
|
||||
|
||||
| Rule | Why |
|
||||
|------|-----|
|
||||
| Zapier cloud → **HTTPS only** (zappier-edge) | Never NATS, never `api.veraetime.net` |
|
||||
| NATS binds **loopback or a private docker/VM net** | Not on the public NIC |
|
||||
| Tree-node **min 3**, pause does not count | Bulk Merkle lookups |
|
||||
| Private keys stay on disk; git stores **paths** | `machines.json` `identityFile` |
|
||||
| Operator console is **loopback** `:3850` | Not a public site |
|
||||
|
||||
## Quick lab (one machine)
|
||||
|
||||
```bash
|
||||
git clone ssh://git@git.georgelambert.org:2223/marchon/verae-ops.git
|
||||
cd verae-ops
|
||||
docker compose up --build
|
||||
```
|
||||
|
||||
Then: zappier http://127.0.0.1:3000/ middleware http://127.0.0.1:3100/health NATS monitoring http://127.0.0.1:8222/
|
||||
|
||||
Production layout is three NATS nodes + fleet-spread workers; see [02-docker.md](02-docker.md) and [05-dedicated-hardware.md](05-dedicated-hardware.md).
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue