George Lambert
cf4ef9e3af
Some checks are pending
offline / test (push) Waiting to run
Unify loopback ops, hop testing, and reading order. Overview lists high-level docs first, then architecture, then the rest. Credits: Scott Lindsey, George Lambert, NATS.IO, Grok-Code by Grok.com.
41 lines
2.3 KiB
Markdown
41 lines
2.3 KiB
Markdown
# Fleet: replica floors, monitor, pause, restart
|
|
|
|
Runtime copies of middleware workers, WORM archives, and **tree nodes** are declared in `packages/verae-fleet/fleet.json` (how many) and `packages/verae-fleet/services/*.json` (how each one runs).
|
|
|
|
```text
|
|
operator --HTTP 127.0.0.1:3850--> fleet control
|
|
| spawn workers with /health
|
|
tree-node min=3 keepFloor |
|
|
archive-worm min=3 |
|
|
paused ≠ available |
|
|
unhealthy → restart v
|
|
replicas on loopback health ports
|
|
```
|
|
|
|
Operator console (Fleet · Trace · Docs): http://127.0.0.1:3850/ — see `packages/verae-fleet/docs/CONSOLE.md`.
|
|
|
|
- **list** — `node src/cli.js list`
|
|
- **monitor** — `serve` probes `/health` and reconciles
|
|
- **restart** — crash or 503 → same instance id respawned
|
|
- **pause / off** — pause does not count toward `min`; keepFloor starts another tree-node. `stop` on a service disables it.
|
|
- **tree-node floor** — `fleet.json` `tree-node.min` (default 3). Do not drop this without changing the spec; bulk-summary leaf queries need several bloom-filtered nodes.
|
|
|
|
## Extra machines
|
|
|
|
Define hosts in `packages/verae-fleet/machines.json` (or **Add machine** on the monitor). Each machine has `capacity` and `roles` it may run. The supervisor places new replicas on the **least-loaded** eligible host.
|
|
|
|
- `kind: local` — spawn on the control plane.
|
|
- `kind: ssh` — `user@host` with `identityFile` (path only, e.g. `~/.ssh/id_ed25519`). Control plane `scp`s the worker and `ssh`es spawn/kill/health. Workers stay on `127.0.0.1` on the remote host; NATS is not published.
|
|
- `kind: agent` — HTTP to `http://host:agentPort` (`node src/agent.js` on that box).
|
|
|
|
Override credentials in gitignored `machines.secrets.json` (see `machines.secrets.json.example`). Never put private-key bytes in git.
|
|
|
|
`node src/cli.js ssh-check ns1` verifies login as marchon@70.88.205.138.
|
|
|
|
Raising `tree-node.max` and adding machines increases bulk-summary lookup capacity.
|
|
|
|
## Message RTT (planning)
|
|
|
|
The monitor samples `POST /message` on each healthy replica and shows **min / avg / p50 / p90** milliseconds of message processing (service row, instance, and machine). Use p90 when sizing extra tree nodes.
|
|
|
|
Zapier cloud is not spawned. NATS is monitored, not bound publicly.
|