marchon/master-zapier-plan-draft
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
master-zapier-plan-draft/docs/02-architecture/fleet.md
George Lambert cf4ef9e3af
Some checks are pending
offline / test (push) Waiting to run
Details
Add operator console (Fleet, Trace, Docs) with screenshots 
Unify loopback ops, hop testing, and reading order. Overview lists
high-level docs first, then architecture, then the rest. Credits:
Scott Lindsey, George Lambert, NATS.IO, Grok-Code by Grok.com.
2026-09-11 14:06:14 -04:00

2.3 KiB
Raw Blame History

Fleet: replica floors, monitor, pause, restart

Runtime copies of middleware workers, WORM archives, and tree nodes are declared in packages/verae-fleet/fleet.json (how many) and packages/verae-fleet/services/*.json (how each one runs).

operator  --HTTP 127.0.0.1:3850-->  fleet control
                                      | spawn workers with /health
          tree-node min=3 keepFloor   |
          archive-worm min=3          |
          paused ≠ available          |
          unhealthy → restart         v
                               replicas on loopback health ports

Operator console (Fleet · Trace · Docs): http://127.0.0.1:3850/ — see packages/verae-fleet/docs/CONSOLE.md.

  • list — node src/cli.js list
  • monitor — serve probes /health and reconciles
  • restart — crash or 503 → same instance id respawned
  • pause / off — pause does not count toward min; keepFloor starts another tree-node. stop on a service disables it.
  • tree-node floor — fleet.json tree-node.min (default 3). Do not drop this without changing the spec; bulk-summary leaf queries need several bloom-filtered nodes.

Extra machines

Define hosts in packages/verae-fleet/machines.json (or Add machine on the monitor). Each machine has capacity and roles it may run. The supervisor places new replicas on the least-loaded eligible host.

  • kind: local — spawn on the control plane.
  • kind: ssh — user@host with identityFile (path only, e.g. ~/.ssh/id_ed25519). Control plane scps the worker and sshes spawn/kill/health. Workers stay on 127.0.0.1 on the remote host; NATS is not published.
  • kind: agent — HTTP to http://host:agentPort (node src/agent.js on that box).

Override credentials in gitignored machines.secrets.json (see machines.secrets.json.example). Never put private-key bytes in git.

node src/cli.js ssh-check ns1 verifies login as marchon@70.88.205.138.

Raising tree-node.max and adding machines increases bulk-summary lookup capacity.

Message RTT (planning)

The monitor samples POST /message on each healthy replica and shows min / avg / p50 / p90 milliseconds of message processing (service row, instance, and machine). Use p90 when sizing extra tree nodes.

Zapier cloud is not spawned. NATS is monitored, not bound publicly.