Add fleet control: service configs, replica floors, monitor, pause/restart

Central fleet.json sets min/max copies. Tree-node keepFloor respawns
until three healthy unpaused replicas remain. CLI and loopback UI
pause, resume, stop, and restart instances that fail health checks.

docs/02-architecture/fleet.md

@@ -0,0 +1,21 @@
+# Fleet: replica floors, monitor, pause, restart
+
+Runtime copies of middleware workers, WORM archives, and **tree nodes** are declared in `packages/verae-fleet/fleet.json` (how many) and `packages/verae-fleet/services/*.json` (how each one runs).
+
+```text
+operator  --HTTP 127.0.0.1:3850-->  fleet control
+                                      | spawn workers with /health
+          tree-node min=3 keepFloor   |
+          archive-worm min=3          |
+          paused ≠ available          |
+          unhealthy → restart         v
+                               replicas on loopback health ports
+```
+
+- **list** — `node src/cli.js list`
+- **monitor** — `serve` probes `/health` and reconciles
+- **restart** — crash or 503 → same instance id respawned
+- **pause / off** — pause does not count toward `min`; keepFloor starts another tree-node. `stop` on a service disables it.
+- **tree-node floor** — `fleet.json` `tree-node.min` (default 3). Do not drop this without changing the spec; bulk-summary leaf queries need several bloom-filtered nodes.
+
+Zapier cloud is not spawned. NATS is monitored, not bound publicly.

docs/02-architecture/modules-and-nats.md

@@ -34,6 +34,7 @@ Zapier cloud **never** connects to NATS. Only middleware workers and archives do
 | archive-worm | `verae-archive-worm` | N copies | `verae.archive.query`, `verae.archive.put` | `verae.archive.reply.<id>` if bloom hits |
 | tree-node | `verae-tree-node` | N copies (WORM role) | `verae.archive.query`, `verae.archive.put` kind `tree` | `verae.archive.reply.<id>` if bloom hits |
 | zapier-simulator | `verae-zapier-simulator` | Local HTTP :3847 | operator browser | in-process replay of all addresses |
+| fleet | `verae-fleet` | Local HTTP :3850 | operator | spawns workers; keepFloor on tree-node |
 | zapier-user-docs | `zapier-user-docs` | Static | — | catalog `/user-docs/` |
 | verae-chain-client | `verae-chain-client` | Library | — | HTTPS `api.veraetime.net` or MOCK |
 | docs-master | `zapier-docs-master` | Static | — | published on zapier.georgelambert.org |

packages/docs-master/README.md

@@ -20,6 +20,7 @@ Summaries, NATS contracts, and message flows for every Verae Time × Zapier modu
 | verae-tree-node | https://git.georgelambert.org/marchon/verae-tree-node |
 | verae-zapier-simulator | https://git.georgelambert.org/marchon/verae-zapier-simulator |
 | zapier-user-docs | https://git.georgelambert.org/marchon/zapier-user-docs |
+| verae-fleet | https://git.georgelambert.org/marchon/verae-fleet |
 | **zapier-docs-master** (this repo) | https://git.georgelambert.org/marchon/zapier-docs-master |
 
 Clone (SSH port 2223):
@@ -40,6 +41,7 @@ Clone (SSH port 2223):
 | tree-node | Merkle leaf proofs | archive.query, archive.put | archive.reply.* (hit only) |
 | zapier-simulator | Trace console (in-process) | — | — |
 | zapier-user-docs | Signup → lookup guide | — | — |
+| verae-fleet | Replica floors + monitor | — | — |
 
 ## Documents in this repo
 

packages/docs-master/modules/verae-fleet/NATS.md

@@ -0,0 +1,3 @@
+# NATS — verae-fleet
+
+Does not subscribe. Records which workers should. NATS remains loopback.

packages/docs-master/modules/verae-fleet/SUMMARY.md

@@ -0,0 +1,5 @@
+# verae-fleet
+
+**Job:** Service catalog, per-service configs, monitor, restart, pause/on/off, keep tree-node replica floor.
+
+**Config:** `fleet.json` (min/max) + `services/<id>.json`.

packages/verae-fleet/.gitignore

@@ -0,0 +1 @@
+data/

packages/verae-fleet/NATS.md

@@ -0,0 +1,15 @@
+# NATS — verae-fleet
+
+Fleet control **does not subscribe** to JetStream. It only records which workers should.
+
+| Service | NATS in | NATS out |
+|---------|---------|----------|
+| nats (external) | — | — (the broker) |
+| middleware-http | jobs.events | jobs.watch |
+| job-poller | jobs.watch | jobs.events |
+| webhook-deliver | webhooks.deliver, jobs.events | HTTPS to Zapier |
+| archive-aggregator | archive.reply.* | archive.query (broadcast) |
+| archive-worm | archive.put, archive.query | archive.reply.* on bloom hit |
+| tree-node | archive.put, archive.query | archive.reply.* on bloom hit |
+
+NS1 nats-server stays on `127.0.0.1:4222`. `fleet.json` `nats.publicBind` is false.

packages/verae-fleet/README.md

@@ -0,0 +1,32 @@
+# verae-fleet
+
+Operator control plane for Verae Time × Zapier **runtime** services:
+
+- a **list** of every service and its config file
+- a **central replica spec** (`fleet.json`) — how many copies must be up
+- a **monitor** of active / paused / unhealthy replicas
+- **restart** when a copy is offline or fails `/health`
+- **on / off / pause / resume** per service or per instance
+- **keepFloor** on tree nodes so at least `min` copies stay available (paused copies do not count)
+
+```bash
+cd packages/verae-fleet
+npm test
+node src/cli.js list
+node src/cli.js serve    # http://127.0.0.1:3850/
+```
+
+Against a running daemon:
+
+```bash
+node src/cli.js status
+node src/cli.js pause tree-node-0      # floor starts another tree-node
+node src/cli.js resume tree-node-0
+node src/cli.js restart tree-node-1
+node src/cli.js stop webhook-deliver   # disable that service
+node src/cli.js start webhook-deliver
+```
+
+Zapier cloud apps are listed but **not spawned**. NATS on NS1 is **monitored only** (loopback `:4222`, never a public bind).
+
+Clone: `ssh://git@git.georgelambert.org:2223/marchon/verae-fleet.git`

packages/verae-fleet/SERVICES.md

@@ -0,0 +1,22 @@
+# Service list
+
+Central replica spec: [`fleet.json`](fleet.json).  
+Per-service files: [`services/`](services/).
+
+| Service | Config | min | max | keepFloor | Spawned by fleet |
+|---------|--------|-----|-----|-----------|------------------|
+| nats | `services/nats.json` | 1 | 1 | no | no (external monitor) |
+| zappier-edge | `services/zappier-edge.json` | 1 | 1 | yes | yes |
+| middleware-http | `services/middleware-http.json` | 1 | 1 | yes | yes |
+| job-poller | `services/job-poller.json` | 1 | 2 | yes | yes |
+| webhook-deliver | `services/webhook-deliver.json` | 1 | 2 | yes | yes |
+| archive-aggregator | `services/archive-aggregator.json` | 1 | 2 | yes | yes |
+| archive-worm | `services/archive-worm.json` | 3 | 6 | yes | yes |
+| **tree-node** | `services/tree-node.json` | **3** | 9 | **yes** | yes |
+| zapier-simulator | `services/zapier-simulator.json` | 0 | 1 | no | optional |
+| zapier-platform-app | `services/unmanaged.json` | — | — | — | no (Zapier cloud) |
+| verae-activate | `services/unmanaged.json` | — | — | — | no |
+| request-splitter | `services/unmanaged.json` | — | — | — | no (library) |
+| verae-chain-client | `services/unmanaged.json` | — | — | — | no (library) |
+
+`keepFloor`: monitor starts replacements when **available** (running, healthy, not paused) drops below `min`. That is how tree nodes stay at three live copies if one is paused, crashed, or not responding.

packages/verae-fleet/SUMMARY.md

@@ -0,0 +1,9 @@
+# verae-fleet
+
+**Job:** Catalog services, hold per-service configs, monitor health, restart bad replicas, pause/resume/on/off, and keep `fleet.json` replica floors — especially tree-node `min`.
+
+**Expects:** operator CLI/HTTP on `127.0.0.1:3850`. Spawns `src/worker.js` children with `/health`.
+
+**Sends:** nothing to Zapier or public NATS.
+
+**Test:** `npm test` — list configs; start 3 tree-nodes; pause one still ≥3 available; unhealthy restart; stop+replace.

packages/verae-fleet/fleet.json

@@ -0,0 +1,26 @@
+{
+  "apiVersion": "verae.fleet/v1",
+  "control": {
+    "bind": "127.0.0.1",
+    "port": 3850,
+    "probeIntervalMs": 750,
+    "unhealthyAfterMs": 2500,
+    "restartBackoffMs": [200, 800, 2000],
+    "maxRestartsPerHour": 40
+  },
+  "nats": {
+    "url": "nats://127.0.0.1:4222",
+    "publicBind": false
+  },
+  "services": {
+    "nats": { "min": 1, "max": 1, "keepFloor": false, "enabled": true, "managed": false },
+    "zappier-edge": { "min": 1, "max": 1, "keepFloor": true, "enabled": true },
+    "middleware-http": { "min": 1, "max": 1, "keepFloor": true, "enabled": true },
+    "job-poller": { "min": 1, "max": 2, "keepFloor": true, "enabled": true },
+    "webhook-deliver": { "min": 1, "max": 2, "keepFloor": true, "enabled": true },
+    "archive-aggregator": { "min": 1, "max": 2, "keepFloor": true, "enabled": true },
+    "archive-worm": { "min": 3, "max": 6, "keepFloor": true, "enabled": true },
+    "tree-node": { "min": 3, "max": 9, "keepFloor": true, "enabled": true },
+    "zapier-simulator": { "min": 0, "max": 1, "keepFloor": false, "enabled": true }
+  }
+}

packages/verae-fleet/package.json

@@ -0,0 +1,13 @@
+{
+  "name": "verae-fleet",
+  "version": "0.1.0",
+  "type": "module",
+  "description": "Service catalog, replica floors, monitor, restart, pause/resume for Verae Zapier workers and tree nodes",
+  "bin": { "verae-fleet": "./src/cli.js" },
+  "scripts": {
+    "start": "node src/cli.js serve",
+    "list": "node src/cli.js list",
+    "test": "node --test test/*.test.js"
+  },
+  "engines": { "node": ">=20" }
+}

packages/verae-fleet/public/index.html

@@ -0,0 +1,76 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="utf-8"/>
+  <meta name="viewport" content="width=device-width, initial-scale=1"/>
+  <title>Verae fleet monitor</title>
+  <style>
+    :root { --ink:#12202c; --muted:#5b6d78; --line:#d5dee4; --bg:#f3efe8; --ok:#0f6e56; --err:#a32020; --warn:#8a5a00; }
+    * { box-sizing: border-box; }
+    body { margin:0; font:14px/1.45 system-ui,sans-serif; color:var(--ink); background:var(--bg); }
+    header { background:var(--ink); color:#f6f3ee; padding:.9rem 1.1rem; }
+    header h1 { margin:0; font-size:1.1rem; }
+    header p { margin:.35rem 0 0; color:#c5d0d8; font-size:13px; }
+    main { padding:1rem; max-width:1100px; margin:0 auto; }
+    table { width:100%; border-collapse:collapse; background:#fff; }
+    th, td { text-align:left; padding:.4rem .5rem; border-bottom:1px solid var(--line); font-size:13px; }
+    th { font-size:11px; letter-spacing:.06em; text-transform:uppercase; color:var(--muted); }
+    .pill { font:700 10px system-ui; letter-spacing:.06em; text-transform:uppercase; padding:.12rem .35rem; border-radius:4px; }
+    .good { background:#d4efe6; color:var(--ok); }
+    .bad { background:#f8d4d4; color:var(--err); }
+    .warn { background:#f5e6c8; color:var(--warn); }
+    button { margin-right:.25rem; font:650 12px system-ui; border:1px solid var(--line); background:#fff; border-radius:5px; padding:.25rem .45rem; cursor:pointer; }
+    code { font:12px ui-monospace,Menlo,monospace; }
+    .inst { font:12px ui-monospace,Menlo,monospace; margin:.15rem 0; }
+  </style>
+</head>
+<body>
+<header>
+  <h1>Verae fleet monitor</h1>
+  <p>Active replicas, replica floors, pause / restart. Tree-node <code>keepFloor</code> respawns until <code>min</code> copies are healthy and unpaused. Loopback only.</p>
+</header>
+<main>
+  <p id="meta"></p>
+  <table>
+    <thead><tr><th>Service</th><th>Config</th><th>min/max</th><th>available</th><th>instances</th><th>actions</th></tr></thead>
+    <tbody id="rows"></tbody>
+  </table>
+</main>
+<script>
+async function j(url, opts) { const r = await fetch(url, opts); return r.json(); }
+async function act(path) { await j(path, { method:'POST' }); await draw(); }
+async function draw() {
+  const s = await j('/api/status');
+  document.getElementById('meta').textContent = 'probe ' + (s.monitor?.t || '—') + ' · NATS ' + (s.nats?.url || '');
+  const tb = document.getElementById('rows');
+  tb.innerHTML = Object.values(s.services).map((sv) => {
+    const floor = sv.belowFloor ? '<span class="pill bad">below floor</span>' : (sv.keepFloor ? '<span class="pill good">floor ok</span>' : '');
+    const inst = (sv.instances || []).map((i) => {
+      const st = !i.pid ? 'dead' : i.paused ? 'paused' : i.healthy === false ? 'unhealthy' : 'up';
+      const cls = st === 'up' ? 'good' : st === 'paused' ? 'warn' : 'bad';
+      return `<div class="inst"><span class="pill ${cls}">${st}</span> ${i.id} pid=${i.pid || '—'} :${i.healthPort}
+        <button onclick="act('/api/instances/${i.id}/pause')">pause</button>
+        <button onclick="act('/api/instances/${i.id}/resume')">resume</button>
+        <button onclick="act('/api/instances/${i.id}/restart')">restart</button>
+      </div>`;
+    }).join('') || '<span style="color:#5b6d78">none</span>';
+    return `<tr>
+      <td><strong>${sv.id}</strong><br><span style="color:#5b6d78">${sv.title || ''}</span></td>
+      <td><code>${sv.configPath || ''}</code></td>
+      <td>${sv.min} / ${sv.max} ${floor}</td>
+      <td>${sv.available} running=${sv.running} paused=${sv.paused}</td>
+      <td>${inst}</td>
+      <td>
+        <button onclick="act('/api/services/${sv.id}/start')">on</button>
+        <button onclick="act('/api/services/${sv.id}/pause')">pause</button>
+        <button onclick="act('/api/services/${sv.id}/resume')">resume</button>
+        <button onclick="act('/api/services/${sv.id}/stop')">off</button>
+      </td>
+    </tr>`;
+  }).join('');
+}
+draw();
+setInterval(draw, 1500);
+</script>
+</body>
+</html>

packages/verae-fleet/services/archive-aggregator.json

@@ -0,0 +1,16 @@
+{
+  "id": "archive-aggregator",
+  "title": "Archive reply aggregator",
+  "kind": "nats-worker",
+  "package": "verae-archive-aggregator",
+  "role": "archive-aggregator",
+  "managed": true,
+  "runtime": "query fan-out + merge",
+  "health": { "type": "http", "path": "/health", "timeoutMs": 800 },
+  "ports": { "healthBase": 13400 },
+  "nats": {
+    "in": ["verae.archive.reply.*"],
+    "out": ["verae.archive.query"]
+  },
+  "notes": "Broadcast archive.query (no queue group). kinds may include tree."
+}

packages/verae-fleet/services/archive-worm.json

@@ -0,0 +1,16 @@
+{
+  "id": "archive-worm",
+  "title": "Bloom-filtered WORM archive",
+  "kind": "nats-archive",
+  "package": "verae-archive-worm",
+  "role": "archive-worm",
+  "managed": true,
+  "runtime": "N copies, bloom miss = silence",
+  "health": { "type": "http", "path": "/health", "timeoutMs": 800 },
+  "ports": { "healthBase": 13500 },
+  "nats": {
+    "in": ["verae.archive.put", "verae.archive.query"],
+    "out": ["verae.archive.reply.<correlationId>"]
+  },
+  "notes": "Subscribe to query without a shared queue group."
+}

packages/verae-fleet/services/job-poller.json

@@ -0,0 +1,16 @@
+{
+  "id": "job-poller",
+  "title": "Job poller worker",
+  "kind": "nats-worker",
+  "package": "verae-zapier-middleware",
+  "role": "job-poller",
+  "managed": true,
+  "runtime": "JetStream consumer",
+  "health": { "type": "http", "path": "/health", "timeoutMs": 800 },
+  "ports": { "healthBase": 13200 },
+  "nats": {
+    "in": ["verae.zapier.jobs.watch"],
+    "out": ["verae.zapier.jobs.events"]
+  },
+  "notes": "GET chain status; emit terminal events. Queue group job-poller."
+}

packages/verae-fleet/services/middleware-http.json

@@ -0,0 +1,17 @@
+{
+  "id": "middleware-http",
+  "title": "Verae Zapier HTTP middleware",
+  "kind": "http",
+  "package": "verae-zapier-middleware",
+  "role": "middleware-http",
+  "managed": true,
+  "runtime": "HTTPS :3100 /zapier/v1",
+  "health": { "type": "http", "path": "/health", "timeoutMs": 800 },
+  "ports": { "healthBase": 13100 },
+  "env": { "PORT": "3100", "NATS_URL": "nats://127.0.0.1:4222" },
+  "nats": {
+    "in": ["verae.zapier.jobs.events"],
+    "out": ["verae.zapier.jobs.watch", "verae.zapier.webhooks.deliver"]
+  },
+  "notes": "Wait path subscribes jobs.events. Never expose NATS to Zapier."
+}

packages/verae-fleet/services/nats.json

@@ -0,0 +1,11 @@
+{
+  "id": "nats",
+  "title": "NATS JetStream (NS1 loopback)",
+  "kind": "external",
+  "package": null,
+  "managed": false,
+  "runtime": "nats-server -js on 127.0.0.1:4222",
+  "health": { "type": "tcp", "host": "127.0.0.1", "port": 4222, "timeoutMs": 400 },
+  "nats": { "in": [], "out": [] },
+  "notes": "Do not bind 0.0.0.0. Zapier never connects here. Fleet monitors only; does not spawn nats-server."
+}

packages/verae-fleet/services/tree-node.json

@@ -0,0 +1,16 @@
+{
+  "id": "tree-node",
+  "title": "Merkle tree-node archive",
+  "kind": "nats-archive",
+  "package": "verae-tree-node",
+  "role": "tree-node",
+  "managed": true,
+  "runtime": "N copies holding leaf proofs for bulk summaries",
+  "health": { "type": "http", "path": "/health", "timeoutMs": 800 },
+  "ports": { "healthBase": 13600 },
+  "nats": {
+    "in": ["verae.archive.put", "verae.archive.query"],
+    "out": ["verae.archive.reply.<correlationId>"]
+  },
+  "notes": "keepFloor in fleet.json must stay >= 3 so a leaf query still has a node that might hold the shard. Bloom miss = no packet."
+}

packages/verae-fleet/services/unmanaged.json

@@ -0,0 +1,37 @@
+{
+  "id": "_unmanaged",
+  "title": "Not fleet-spawned (listed for operators)",
+  "entries": [
+    {
+      "id": "zapier-platform-app",
+      "package": "verae-zapier",
+      "runtime": "Zapier cloud",
+      "notes": "HTTPS only to zappier-edge. Never NATS."
+    },
+    {
+      "id": "verae-activate",
+      "package": "verae-activate",
+      "runtime": "Zapier cloud (activate-now app)"
+    },
+    {
+      "id": "request-splitter",
+      "package": "verae-request-splitter",
+      "runtime": "library in middleware"
+    },
+    {
+      "id": "verae-chain-client",
+      "package": "verae-zapier-middleware",
+      "runtime": "library → api.veraetime.net or MOCK"
+    },
+    {
+      "id": "zapier-user-docs",
+      "package": "zapier-user-docs",
+      "runtime": "static"
+    },
+    {
+      "id": "docs-master",
+      "package": "docs-master",
+      "runtime": "static"
+    }
+  ]
+}