diff --git a/docs/02-architecture/modules-and-nats.md b/docs/02-architecture/modules-and-nats.md index 07a26bf..8778d38 100644 --- a/docs/02-architecture/modules-and-nats.md +++ b/docs/02-architecture/modules-and-nats.md @@ -35,6 +35,8 @@ Zapier cloud **never** connects to NATS. Only middleware workers and archives do | tree-node | `verae-tree-node` | N copies (WORM role) | `verae.archive.query`, `verae.archive.put` kind `tree` | `verae.archive.reply.` if bloom hits | | zapier-simulator | `verae-zapier-simulator` | Local HTTP :3847 | operator browser | in-process replay of all addresses | | fleet | `verae-fleet` | Local HTTP :3850 | operator | spawns workers; keepFloor on tree-node | +| overview | `overview` | Static | — | high-level system map | +| nats-process | `verae-nats-process` | Worker template | `verae.example.process.in` | `.out` / `.reply.*` | | zapier-user-docs | `zapier-user-docs` | Static | — | catalog `/user-docs/` | | verae-chain-client | `verae-chain-client` | Library | — | HTTPS `api.veraetime.net` or MOCK | | docs-master | `zapier-docs-master` | Static | — | published on zapier.georgelambert.org | diff --git a/packages/docs-master/README.md b/packages/docs-master/README.md index c5c732f..11052f6 100644 --- a/packages/docs-master/README.md +++ b/packages/docs-master/README.md @@ -21,6 +21,8 @@ Summaries, NATS contracts, and message flows for every Verae Time × Zapier modu | verae-zapier-simulator | https://git.georgelambert.org/marchon/verae-zapier-simulator | | zapier-user-docs | https://git.georgelambert.org/marchon/zapier-user-docs | | verae-fleet | https://git.georgelambert.org/marchon/verae-fleet | +| **overview** | https://git.georgelambert.org/marchon/overview | +| verae-nats-process | https://git.georgelambert.org/marchon/verae-nats-process | | **zapier-docs-master** (this repo) | https://git.georgelambert.org/marchon/zapier-docs-master | Clone (SSH port 2223): @@ -42,6 +44,8 @@ Clone (SSH port 2223): | zapier-simulator | Trace console (in-process) | — | — | | zapier-user-docs | Signup → lookup guide | — | — | | verae-fleet | Replica floors + monitor | — | — | +| overview | System map | — | — | +| nats-process | Template worker | example.process.in | example.process.out / reply.* | ## Documents in this repo diff --git a/packages/docs-master/modules-and-nats.md b/packages/docs-master/modules-and-nats.md index d27d4ff..850fc28 100644 --- a/packages/docs-master/modules-and-nats.md +++ b/packages/docs-master/modules-and-nats.md @@ -36,6 +36,9 @@ Zapier cloud **never** connects to NATS. Only middleware workers and archives do | zapier-simulator | `verae-zapier-simulator` | Local HTTP :3847 | operator browser | in-process replay of all addresses | | zapier-user-docs | `zapier-user-docs` | Static | — | catalog `/user-docs/` | | verae-chain-client | `verae-chain-client` | Library | — | HTTPS `api.veraetime.net` or MOCK | +| fleet | `verae-fleet` | Local HTTP :3850 | operator | keepFloor + SSH hosts | +| overview | `overview` | Static | — | high-level map | +| nats-process | `verae-nats-process` | Template worker | `verae.example.process.in` | `.out` / `.reply.*` | | docs-master | `zapier-docs-master` | Static | — | published on zapier.georgelambert.org | ## Addresses (subjects) diff --git a/packages/docs-master/modules/overview/NATS.md b/packages/docs-master/modules/overview/NATS.md new file mode 100644 index 0000000..6a298ae --- /dev/null +++ b/packages/docs-master/modules/overview/NATS.md @@ -0,0 +1,3 @@ +# NATS — overview + +Documentation only. Describes the central 3-server cluster. Zapier never connects. diff --git a/packages/docs-master/modules/overview/SUMMARY.md b/packages/docs-master/modules/overview/SUMMARY.md new file mode 100644 index 0000000..a3845fa --- /dev/null +++ b/packages/docs-master/modules/overview/SUMMARY.md @@ -0,0 +1,3 @@ +# overview + +High-level system map, 3-node NATS cluster, uptime, network failure, address routing. Repo: `overview`. diff --git a/packages/docs-master/modules/verae-nats-process/NATS.md b/packages/docs-master/modules/verae-nats-process/NATS.md new file mode 100644 index 0000000..c99f17b --- /dev/null +++ b/packages/docs-master/modules/verae-nats-process/NATS.md @@ -0,0 +1,3 @@ +# NATS — verae-nats-process + +IN `verae.example.process.in` (queue `example-process`). OUT `.out` and `.reply.`. diff --git a/packages/docs-master/modules/verae-nats-process/SUMMARY.md b/packages/docs-master/modules/verae-nats-process/SUMMARY.md new file mode 100644 index 0000000..d49143d --- /dev/null +++ b/packages/docs-master/modules/verae-nats-process/SUMMARY.md @@ -0,0 +1,3 @@ +# verae-nats-process + +Template for a new `verae...*` worker. Copy this repo to expand. diff --git a/packages/overview/01-system.md b/packages/overview/01-system.md new file mode 100644 index 0000000..436b7d3 --- /dev/null +++ b/packages/overview/01-system.md @@ -0,0 +1,26 @@ +# 1. System + +Verae Time proves a SHA-256 existed at a given time. Zapier lets customers register and look up hashes from the tools they already use. Billing and API keys live on **zappier-edge**. Timestamping, wait, webhooks, and archive fan-out live behind **verae-middleware**. Durable messaging is the **central Verae NATS.IO 3-server JetStream cluster**. + +## Hard rule + +**Zapier never connects to NATS, tree nodes, WORM archives, or `api.veraetime.net`.** Those hops are middleware and workers only. If a trace ever shows a Zapier hop on a `verae.*` subject, do not push the app. + +## Planes + +| Plane | What | Network | +|-------|------|---------| +| Zapier cloud | `verae-zapier-app`, `verae-activate` | HTTPS to zappier-edge | +| Commercial edge | `zappier-edge` portal, admin, `x-api-key`, Stripe meter | Public HTTPS | +| Middleware HTTP | `/zapier/v1/*` job id + wait | Public HTTPS from edge only | +| NATS cluster | JetStream subjects under `verae.*` | Private; loopback or SSH tunnel | +| Workers | poller, webhook-deliver, aggregator | NATS + HTTPS to chain or Zapier hooks | +| Archives | WORM + tree nodes | NATS broadcast query; bloom miss = silence | +| Chain | Verae timestamping | HTTPS `api.veraetime.net` or MOCK | +| Control | `verae-fleet` | Operator loopback `:3850`; SSH to extra machines | + +## Request in one sentence + +A Zap step POSTs to zappier-edge; edge meters and forwards to middleware; middleware splits hash vs files, writes the hash (or Merkle root) to the chain, publishes `verae.zapier.jobs.watch`, and either returns `jobId` or waits on `verae.zapier.jobs.events`. Attached metadata and bulk-summary **leaves** are found later by broadcasting `verae.archive.query` to every archive/tree node. + +See [diagrams](08-diagrams.md). diff --git a/packages/overview/02-modules-and-repos.md b/packages/overview/02-modules-and-repos.md new file mode 100644 index 0000000..720b30f --- /dev/null +++ b/packages/overview/02-modules-and-repos.md @@ -0,0 +1,25 @@ +# 2. Modules and independent repositories + +Each runtime piece is its **own git repo** on Forgejo (`git.georgelambert.org`, SSH port 2223). The monorepo `master-zapier-plan-draft` is a snapshot of the workspace; do not treat it as the only clone path. + +| Independent repo | Package / path | What it does | +|------------------|----------------|--------------| +| **overview** | `packages/overview` | This high-level map | +| **verae-nats-process** | `packages/verae-nats-process` | **Template** for a new addressed NATS process | +| **master-zapier-plan-draft** | workspace root | Combined snapshot (`main` and `master`) | +| **zappier-edge** | `packages/zappier` | Metered HTTPS, portal, admin, Stripe `zappier.api_cents` | +| **verae-middleware** | `packages/verae-zapier-middleware` | `/zapier/v1`, job wait, NATS publishers/workers | +| **verae-zapier-app** | `packages/verae-zapier` | Full Zapier nouns (timestamp, wait, batch, hash + tree lookup) | +| **verae-activate** | `packages/verae-activate` | Activate-now (Add Numbers, Echo, SHA256, mock timestamp) | +| **verae-request-splitter** | `packages/verae-request-splitter` | Chain hash vs `archive.put` | +| **verae-archive-worm** | `packages/verae-archive-worm` | Bloom WORM node | +| **verae-archive-aggregator** | `packages/verae-archive-aggregator` | Merge archive replies | +| **verae-tree-node** | `packages/verae-tree-node` | Merkle leaf proofs (bulk summaries) | +| **verae-fleet** | `packages/verae-fleet` | Catalog, min copies, pause/restart, SSH hosts | +| **verae-zapier-simulator** | `packages/verae-zapier-simulator` | Trace console before `zapier-platform push` | +| **zapier-user-docs** | `packages/zapier-user-docs` | Customer signup → register → lookup | +| **zapier-docs-master** | `packages/docs-master` | Per-module `SUMMARY.md` + `NATS.md` | + +Libraries that are **not** separate Forgejo apps today: `verae-chain-client` (inside middleware), job-poller and webhook-deliver (middleware workers, fleet-spawned). + +Per-module NATS contracts: [zapier-docs-master](https://git.georgelambert.org/marchon/zapier-docs-master). diff --git a/packages/overview/03-nats-cluster.md b/packages/overview/03-nats-cluster.md new file mode 100644 index 0000000..244e5da --- /dev/null +++ b/packages/overview/03-nats-cluster.md @@ -0,0 +1,37 @@ +# 3. Central Verae NATS.IO 3-server cluster + +All durable messaging for this product is **NATS JetStream**, not Zapier queues and not a public TCP API. + +## Cluster + +Three `nats-server -js` nodes form the Verae cluster (routes between them, JetStream replication). Clients (middleware, poller, aggregator, WORM, tree nodes, and any new process cloned from `verae-nats-process`) connect with a **cluster URL list**, for example: + +```text +nats://127.0.0.1:4222,nats://127.0.0.1:4223,nats://127.0.0.1:4224 +``` + +On a given machine the listener stays on **loopback** (or a private interface). Operators reach it with `scripts/nats-tunnel.sh` / `ssh -L 14222:127.0.0.1:4222`. **Do not bind 4222 on `0.0.0.0` without auth.** + +Today’s NS1 box (`NS1.GEORGELAMBERT.ORG`, `70.88.205.138`) already runs JetStream on `127.0.0.1:4222`. The **target** is three clustered nodes so losing one server does not lose the stream. Fleet SSH hosts (`ns1`, later `lan-134`) run **workers**, not extra public NATS listeners. + +![NATS cluster](diagrams/nats-cluster.svg) + +## Who may connect + +| Allowed | Forbidden | +|---------|-----------| +| verae-middleware, fleet workers, WORM, tree nodes, `verae-nats-process` clones | Zapier cloud, browsers, zappier-edge | + +## Address families already in use + +| Address | Kind | Notes | +|---------|------|--------| +| `verae.zapier.jobs.watch` | work queue | poller | +| `verae.zapier.jobs.events` | events | wait + webhooks | +| `verae.zapier.webhooks.deliver` | work queue | HTTPS to Zapier REST Hook | +| `verae.zapier.usage` | optional | metering | +| `verae.archive.put` | JetStream | splitter / merkle builder | +| `verae.archive.query` | **broadcast** (no queue group) | every WORM and tree node | +| `verae.archive.reply.` | replies | **only on bloom hit** | + +New functions get new `verae...` addresses — see [06-address-routing.md](06-address-routing.md). diff --git a/packages/overview/04-uptime.md b/packages/overview/04-uptime.md new file mode 100644 index 0000000..df78f9b --- /dev/null +++ b/packages/overview/04-uptime.md @@ -0,0 +1,34 @@ +# 4. How the system maintains uptime + +Uptime is **NATS durability + fleet replica floors + more than one machine**, not a single always-on Zapier connection. + +## Replica floors (`verae-fleet`) + +`packages/verae-fleet/fleet.json` sets `min` / `max` / `keepFloor` per service. **Available** means running, healthy, and not paused. + +| Service | Default min | keepFloor | +|---------|-------------|-----------| +| tree-node | 3 | yes | +| archive-worm | 3 | yes | +| archive-aggregator, job-poller, webhook-deliver | 1 | yes | +| zappier-edge, middleware-http | 1 | yes | + +If a tree node is paused, crashes, or fails `/health`, fleet **starts another copy** until three are available. Monitor: http://127.0.0.1:3850/ (green / yellow / red rows). + +## Restart and pause + +- Unhealthy `/health` → same instance id restarted. +- Pause does not count toward `min`. +- `stop` on a service disables keepFloor for that service. + +## Spread across machines + +`machines.json` lists hosts (`local`, `ns1` = `marchon@70.88.205.138` with `~/.ssh/id_ed25519`, optional `lan-134`). New replicas go to the **least-loaded** eligible host. Remote spawn/health/kill is SSH; workers bind loopback on the remote box. + +## JetStream + +Work queues (`jobs.watch`, `webhooks.deliver`) replay if a consumer dies. Event stream (`jobs.events`) lets waiters and webhook routers catch up. A 3-node cluster keeps the stream if one NATS server is down. + +## What Zapier sees + +HTTPS 202 `jobId`, wait JSON, or REST Hook. Timeouts return `pending` + `jobId` so the **Timestamp Completed** trigger can finish the job. Zapier retries are safe: the same SHA-256 returns the original seal (`existing: true`). diff --git a/packages/overview/05-network-failures.md b/packages/overview/05-network-failures.md new file mode 100644 index 0000000..a3bb11c --- /dev/null +++ b/packages/overview/05-network-failures.md @@ -0,0 +1,23 @@ +# 5. Local network failures + +“Local network” means the operator LAN, SSH to NS1, or a partitioned archive — not Zapier’s cloud. + +| Failure | What happens | What the customer sees | +|---------|--------------|------------------------| +| NATS node unreachable | Client reconnects to another cluster URL; JetStream consumers resume | Wait may return `pending`; hook still fires later | +| All NATS down | Middleware cannot publish `jobs.watch`; fleet marks workers unhealthy | 503 / pending; no NATS leak to Zapier | +| One WORM / tree node partitioned | Bloom miss = **no packet**; aggregator uses whoever answered | Lookup may miss attachments until the node returns; seal on chain still valid | +| SSH to a fleet host fails | Spawn fails; fleet **places the next replica on another machine** | Floor still met if capacity remains on `local` or another SSH host | +| Chain `api.veraetime.net` timeout | Poller retries; then `timestamp.timeout` event | Wait → pending or failed; async + hook still the recovery path | +| zappier-edge 402 | QuotaExceeded with upgrade URL | Zap step error; no NATS involved | +| Tunnel to loopback NATS dropped | `NATS_URL=nats://127.0.0.1:14222` dies; restart `nats-tunnel.sh` | Workers on NS1 itself still see `127.0.0.1:4222` | + +## Design choices that make partitions survivable + +1. **Archive query is broadcast**, not a shared queue group — a dead node does not steal the message. +2. **Bloom miss is silence** — missing nodes do not send empty errors that look like “hash unknown”. +3. **Zero replies + known puts** is an outage, not a miss (simulator / fleet monitors flag this). +4. **NATS is not on the public NIC** — a WAN blip does not expose 4222. +5. **Idempotent seals** — retrying a Zap after a network error will not double-timestamp. + +See fleet RTT (min / avg / p50 / p90) when planning extra tree nodes after a flaky path. diff --git a/packages/overview/06-address-routing.md b/packages/overview/06-address-routing.md new file mode 100644 index 0000000..5f0d666 --- /dev/null +++ b/packages/overview/06-address-routing.md @@ -0,0 +1,33 @@ +# 6. Address routing (including unplanned functions) + +NATS **addresses** (subjects) are the extension point. A new search, store, or job type is a new address plus a process that listens — not a new Zapier TCP client. + +## Pattern + +```text +verae... +verae...reply. +``` + +| Piece | Example | Meaning | +|-------|---------|---------| +| `verae` | — | Verae bus (not Zapier) | +| `area` | `zapier`, `archive`, `search`, `store` | Product slice | +| `resource` | `jobs`, `hashes`, `blobs` | Noun | +| `action` | `watch`, `query`, `put`, `in` | Verb | +| `reply.` | — | Correlated response | + +**Queue group** (work sharing): `area-resource-action` (e.g. `job-poller`). +**No queue group** (fan-out): archive/tree **query** so every node sees every lookup. + +## Adding something that does not exist yet + +1. Copy the independent repo **[verae-nats-process](https://git.georgelambert.org/marchon/verae-nats-process)** (`packages/verae-nats-process`). +2. Rename `verae.example.process.in` / `.out` / `.reply.*` in `src/subjects.js`. +3. Add a row to that repo’s `ROUTING.md` and to [INDEX.md](INDEX.md). +4. Register the process in `verae-fleet` (`min`/`max`, machines, roles). +5. If Zapier must call it, add **one HTTPS route** on middleware — Zapier still never sees NATS. + +Do **not** invent a public NATS URL for Zapier. Do **not** reuse `verae.archive.query` as a queue group. + +![Address expansion](diagrams/routing.svg) diff --git a/packages/overview/07-external-resources.md b/packages/overview/07-external-resources.md new file mode 100644 index 0000000..afa9304 --- /dev/null +++ b/packages/overview/07-external-resources.md @@ -0,0 +1,15 @@ +# 7. External resources: search, storage, and the chain + +| Resource | Where | How a Zap reaches it | +|----------|--------|----------------------| +| **Central chain** (itemized SHA-256 + time + certificate) | `api.veraetime.net` or MOCK | HTTPS via middleware: create / wait / `GET /hashes/{sha256}` | +| **Bulk Merkle root** | Same chain, one seal | `POST /timestamp/batch` | +| **Leaf proofs** (hash only in a bulk summary) | Tree-node WORM on NATS | `GET /hashes/{sha}?includeTree=true` → `verae.archive.query` kinds=`tree` | +| **Public / private metadata, files** | WORM archives | `includeAttached`; never on chain | +| **Search (central)** | Chain lookup | Zapier search **Find Timestamp by SHA256** | +| **Search (external tree / extra stores)** | NATS query to every node that might hold the key | Zapier search **Find Hash (tree nodes + central)** | +| **Future search / store** | New `verae.search.*` or `verae.store.*` process | Copy `verae-nats-process`; optional middleware GET | + +Storage that is **not** the blockchain stays on WORM/tree nodes (and later any process that answers `verae.archive.query` or a new store address). The chain stores hash + time + certificate (+ Merkle root for batches). + +External “unplanned” storage or search is the same pattern: new address, bloom or index on that node, silence on miss, aggregator or the template’s reply subject. diff --git a/packages/overview/08-diagrams.md b/packages/overview/08-diagrams.md new file mode 100644 index 0000000..c2e07f6 --- /dev/null +++ b/packages/overview/08-diagrams.md @@ -0,0 +1,59 @@ +# 8. Architectural diagrams + +SVG files in [`diagrams/`](diagrams/). The same shapes are repeated below in mermaid for Forgejo preview. + +## 8.1 End-to-end (HTTPS vs NATS) + +![System](diagrams/system.svg) + +```mermaid +flowchart LR + Z[Zapier cloud apps] -->|HTTPS x-api-key| E[zappier-edge] + E -->|HTTPS| M[verae-middleware] + M -->|HTTPS| C[Verae chain] + M -->|JetStream| N[NATS 3-node cluster] + N --> P[job-poller] + N --> W[webhook-deliver] + N --> A[archive-aggregator] + N --> R[WORM x N] + N --> T[tree-node x N] + P -->|HTTPS status| C + W -->|HTTPS REST Hook| Z +``` + +## 8.2 NATS.IO 3-server cluster + +![Cluster](diagrams/nats-cluster.svg) + +```mermaid +flowchart TB + subgraph cluster [Verae NATS.IO JetStream] + N1[nats-server A :4222 loopback] + N2[nats-server B] + N3[nats-server C] + N1 <--> N2 + N2 <--> N3 + N3 <--> N1 + end + MW[middleware + workers] -->|cluster URL list| cluster + F[fleet SSH hosts] -->|workers only| cluster +``` + +## 8.3 Uptime and failure + +![Uptime](diagrams/uptime.svg) + +```mermaid +flowchart TD + H[health /health] -->|fail| R[restart instance] + P[pause] -->|not available| F[keepFloor spawn] + S[SSH host down] -->|spawn failed| O[place on next machine] + Q[archive.query broadcast] -->|bloom miss| SIL[silence] + Q -->|hit| REP[archive.reply.id] +``` + +## 8.4 New address = new process + +![Routing](diagrams/routing.svg) + +See [09-expansion-template.md](09-expansion-template.md). diff --git a/packages/overview/09-expansion-template.md b/packages/overview/09-expansion-template.md new file mode 100644 index 0000000..6ef8e5d --- /dev/null +++ b/packages/overview/09-expansion-template.md @@ -0,0 +1,29 @@ +# 9. Model repo for a new addressed process + +**Repository:** [verae-nats-process](https://git.georgelambert.org/marchon/verae-nats-process) +**Path in the monorepo:** `packages/verae-nats-process` +**Clone:** `ssh://git@git.georgelambert.org:2223/marchon/verae-nats-process.git` + +This is the **reference implementation** for expansions: a small JetStream worker with: + +- a single **in** address, an **out** address, and **reply.<correlationId>** +- `handle(msg)` you replace with real work (search, store, transform) +- HTTP `/health` so `verae-fleet` can keep a replica floor +- tests that do not need a live cluster +- `ROUTING.md` — the row you copy into the global address table + +Default subjects (rename before production): + +| Direction | Address | +|-----------|---------| +| IN | `verae.example.process.in` | +| OUT | `verae.example.process.out` | +| REPLY | `verae.example.process.reply.` | + +```bash +cd packages/verae-nats-process +npm test +# then rename example → your area, add fleet.json min, clone as a new Forgejo repo +``` + +Zapier still must not subscribe. If a Zap needs the result, middleware exposes HTTPS and publishes to the new **in** address. diff --git a/packages/overview/INDEX.md b/packages/overview/INDEX.md new file mode 100644 index 0000000..90191ee --- /dev/null +++ b/packages/overview/INDEX.md @@ -0,0 +1,70 @@ +# Documentation index + +Pointers into the rest of the documentation. Live HTML catalog: https://zapier.georgelambert.org/ + +## This overview + +| Topic | File | +|-------|------| +| TOC / intro | [README.md](README.md) | +| System | [01-system.md](01-system.md) | +| Modules and repos | [02-modules-and-repos.md](02-modules-and-repos.md) | +| NATS 3-server cluster | [03-nats-cluster.md](03-nats-cluster.md) | +| Uptime | [04-uptime.md](04-uptime.md) | +| Local network failures | [05-network-failures.md](05-network-failures.md) | +| Address routing | [06-address-routing.md](06-address-routing.md) | +| External search / storage | [07-external-resources.md](07-external-resources.md) | +| Diagrams | [08-diagrams.md](08-diagrams.md) | +| Expansion template | [09-expansion-template.md](09-expansion-template.md) | + +## Catalog (HTML) + +| Document | URL | +|----------|-----| +| Catalog home | https://zapier.georgelambert.org/ | +| Composition | https://zapier.georgelambert.org/docs/02-architecture/composition.html | +| NATS gateway | https://zapier.georgelambert.org/docs/02-architecture/nats-gateway.html | +| NATS subjects | https://zapier.georgelambert.org/docs/02-architecture/nats-subjects.html | +| Archive / bloom | https://zapier.georgelambert.org/docs/02-architecture/archive-nats.html | +| Module NATS map | https://zapier.georgelambert.org/docs/02-architecture/modules-and-nats.html | +| Tree nodes | https://zapier.georgelambert.org/docs/02-architecture/tree-nodes.html | +| Fleet | https://zapier.georgelambert.org/docs/02-architecture/fleet.html | +| User guide | https://zapier.georgelambert.org/user-docs/README.md | +| Tree-node lookup (users) | https://zapier.georgelambert.org/packages/zapier-user-docs/09-lookup-tree-nodes.html | +| Zapier developer setup | https://zapier.georgelambert.org/docs/04-activate/SETUP-ZAPIER-DEVELOPER.html | +| docs-master | https://zapier.georgelambert.org/docs-master/README.md | + +## Independent git repositories + +Prefix: `https://git.georgelambert.org/marchon/` + +overview · verae-nats-process · master-zapier-plan-draft · zappier-edge · verae-middleware · verae-zapier-app · verae-activate · verae-request-splitter · verae-archive-worm · verae-archive-aggregator · verae-tree-node · verae-fleet · verae-zapier-simulator · zapier-user-docs · zapier-docs-master + +## Per-module contracts + +In **zapier-docs-master**: `modules//SUMMARY.md` and `NATS.md` for every runtime package. + +## Operator surfaces (not public NATS) + +| Surface | URL | +|---------|-----| +| Fleet monitor | http://127.0.0.1:3850/ | +| Zapier simulator | http://127.0.0.1:3847/ | + +## A–Z subject index + +| Term | See | +|------|-----| +| aggregator | verae-archive-aggregator; archive.query / reply | +| bloom miss | silence; 05-network-failures | +| batch / Merkle | verae-tree-node; user-docs 08–09 | +| cluster | 03-nats-cluster | +| fleet / keepFloor | verae-fleet; 04-uptime | +| hash lookup | verae-zapier-app searches; 07-external-resources | +| jobs.events / jobs.watch | nats-subjects.html; middleware | +| includeTree | tree lookup; archive.query kinds=tree | +| routing | 06-address-routing; verae-nats-process | +| splitter | verae-request-splitter | +| SSH hosts | verae-fleet machines.json | +| Zapier never NATS | 01-system | +| zappier-edge | metering, portal | diff --git a/packages/overview/NATS.md b/packages/overview/NATS.md new file mode 100644 index 0000000..b1f59cc --- /dev/null +++ b/packages/overview/NATS.md @@ -0,0 +1,7 @@ +# NATS — overview + +This documentation repo does **not** subscribe. It describes the **central Verae NATS.IO 3-server JetStream cluster** that every worker talks to. + +Zapier cloud never connects here. Clients use private URLs (`nats://127.0.0.1:4222` on a node, or an SSH tunnel). Do not bind 4222 on `0.0.0.0` without auth. + +New functions are new **addresses** (subjects), not new public TCP ports. Copy `verae-nats-process` to add one. diff --git a/packages/overview/README.md b/packages/overview/README.md new file mode 100644 index 0000000..7e96762 --- /dev/null +++ b/packages/overview/README.md @@ -0,0 +1,48 @@ +# Verae Time × Zapier — system overview + +High-level description of the whole system: what it is, which **independent git repositories** implement it, how it stays up, how it behaves when the local network fails, and how it talks to the **central Verae NATS.IO 3-server cluster**. New work is added by **new address routing**, not by teaching Zapier about NATS. + +**This repo:** https://git.georgelambert.org/marchon/overview +**Live catalog:** https://zapier.georgelambert.org/ +**Clone:** `ssh://git@git.georgelambert.org:2223/marchon/overview.git` + +## Table of contents + +1. [System](01-system.md) — what runs where; Zapier never speaks NATS +2. [Modules and repositories](02-modules-and-repos.md) — every independent Forgejo repo +3. [NATS.IO 3-server cluster](03-nats-cluster.md) — JetStream, subjects, who may connect +4. [Uptime](04-uptime.md) — replica floors, restart, SSH spread +5. [Local network failures](05-network-failures.md) — reconnect, silence, pending, failover +6. [Address routing](06-address-routing.md) — how to add unplanned functions +7. [External resources](07-external-resources.md) — searches, storage, chain +8. [Architectural diagrams](08-diagrams.md) — SVG + mermaid +9. [Expansion template](09-expansion-template.md) — `verae-nats-process` +10. [Documentation index](INDEX.md) — pointer into all other docs + +## One-screen picture + +![System context](diagrams/system.svg) + +Zapier → HTTPS `zappier-edge` → HTTPS `verae-middleware` → **NATS cluster** → workers, WORM archives, tree nodes. Chain HTTPS is `api.veraetime.net` (or MOCK). Fleet keeps minimum copies, including tree nodes. + +## Independent repositories (Forgejo) + +| Repo | Role | +|------|------| +| [overview](https://git.georgelambert.org/marchon/overview) | This document | +| [verae-nats-process](https://git.georgelambert.org/marchon/verae-nats-process) | **Model** for a new addressed process | +| [master-zapier-plan-draft](https://git.georgelambert.org/marchon/master-zapier-plan-draft) | Monorepo snapshot | +| [zappier-edge](https://git.georgelambert.org/marchon/zappier-edge) | Metered public HTTPS | +| [verae-middleware](https://git.georgelambert.org/marchon/verae-middleware) | Zapier HTTP + NATS workers | +| [verae-zapier-app](https://git.georgelambert.org/marchon/verae-zapier-app) | Zapier Platform app | +| [verae-activate](https://git.georgelambert.org/marchon/verae-activate) | Activate-now app | +| [verae-request-splitter](https://git.georgelambert.org/marchon/verae-request-splitter) | Hash vs attachments | +| [verae-archive-worm](https://git.georgelambert.org/marchon/verae-archive-worm) | Bloom WORM node | +| [verae-archive-aggregator](https://git.georgelambert.org/marchon/verae-archive-aggregator) | Archive reply merge | +| [verae-tree-node](https://git.georgelambert.org/marchon/verae-tree-node) | Merkle leaf proofs | +| [verae-fleet](https://git.georgelambert.org/marchon/verae-fleet) | Replica floor + SSH hosts | +| [verae-zapier-simulator](https://git.georgelambert.org/marchon/verae-zapier-simulator) | Trace console | +| [zapier-user-docs](https://git.georgelambert.org/marchon/zapier-user-docs) | Signup → lookup | +| [zapier-docs-master](https://git.georgelambert.org/marchon/zapier-docs-master) | Per-module SUMMARY + NATS | + +Clone any of them: `git clone ssh://git@git.georgelambert.org:2223/marchon/.git` (SSH port **2223**). diff --git a/packages/overview/SUMMARY.md b/packages/overview/SUMMARY.md new file mode 100644 index 0000000..9aba71b --- /dev/null +++ b/packages/overview/SUMMARY.md @@ -0,0 +1,7 @@ +# overview + +**Job:** High-level map of the whole Verae Time × Zapier system: modules, independent git repos, uptime, local-network failure, the central 3-node NATS.IO cluster, and how to add a new addressed process. + +**Expects / sends:** none (documentation). + +**Repos named:** see [02-modules-and-repos.md](02-modules-and-repos.md). Expansion template: `verae-nats-process`. diff --git a/packages/overview/diagrams/nats-cluster.svg b/packages/overview/diagrams/nats-cluster.svg new file mode 100644 index 0000000..44e9a71 --- /dev/null +++ b/packages/overview/diagrams/nats-cluster.svg @@ -0,0 +1,21 @@ + + + Verae NATS.IO — three JetStream servers + + + A + :4222 loopback + + B + route + + C + JetStream + + + + + workers + middleware + → nats://A,B,C (never Zapier) + + diff --git a/packages/overview/diagrams/routing.svg b/packages/overview/diagrams/routing.svg new file mode 100644 index 0000000..5229df4 --- /dev/null +++ b/packages/overview/diagrams/routing.svg @@ -0,0 +1,25 @@ + + + New function = new address + clone of verae-nats-process + + + copy template repo + verae-nats-process + + rename subjects + verae.area.resource.action + + fleet min/max + keepFloor + machines + + optional HTTPS + middleware only → Zapier + + + + + Existing: verae.zapier.jobs.* · verae.archive.put|query|reply.* + Unplanned search/store: verae.search.* or verae.store.* — same worker shape. Zapier never subscribes. + Broadcast queries: no queue group. Work queues: queue group. Bloom miss: no packet. + + diff --git a/packages/overview/diagrams/system.svg b/packages/overview/diagrams/system.svg new file mode 100644 index 0000000..687b74e --- /dev/null +++ b/packages/overview/diagrams/system.svg @@ -0,0 +1,47 @@ + + + Verae Time × Zapier — system context + Zapier never speaks NATS. HTTPS stops at middleware. JetStream is the 3-node cluster. + + + Zapier cloud + verae-zapier-app + + zappier-edge + HTTPS :3000 meter + + verae-middleware + HTTPS :3100 wait + + Verae chain + api.veraetime.net + + HTTPS + + HTTPS + + HTTPS hash + + Central Verae NATS.IO 3-server JetStream cluster + + node A + + node B + + node C + + job-poller + jobs.watch + + webhook-deliver + webhooks.deliver + + WORM × N + archive.query + + tree-node + kind=tree + + JetStream + + diff --git a/packages/overview/diagrams/uptime.svg b/packages/overview/diagrams/uptime.svg new file mode 100644 index 0000000..91cdfee --- /dev/null +++ b/packages/overview/diagrams/uptime.svg @@ -0,0 +1,22 @@ + + + Uptime: floors, restart, another machine + + + healthy replica + counts toward min + + paused / slow RTT + not available → spawn + + /health fail or SSH down + restart or next host + + JetStream replay + consumer catch-up + + tree-node min=3 keepFloor + local + ns1 (marchon@70.88.205.138, ~/.ssh/id_ed25519). Least-loaded placement. Bloom miss = silence so a partitioned archive does not block the bus. + Zapier only sees HTTPS 202 / wait JSON / REST Hook. Cluster majority of 3 NATS servers survives one node loss. + + diff --git a/packages/overview/package.json b/packages/overview/package.json new file mode 100644 index 0000000..9f8a6d6 --- /dev/null +++ b/packages/overview/package.json @@ -0,0 +1,6 @@ +{ + "name": "overview", + "version": "1.0.0", + "private": true, + "description": "High-level overview of the Verae Time × Zapier system, modules, uptime, NATS cluster, and docs index" +} diff --git a/packages/verae-nats-process/NATS.md b/packages/verae-nats-process/NATS.md new file mode 100644 index 0000000..ce966ae --- /dev/null +++ b/packages/verae-nats-process/NATS.md @@ -0,0 +1,13 @@ +# NATS — verae-nats-process (template) + +Connect to the **Verae 3-node JetStream cluster** (private URLs). Zapier never connects. + +| Direction | Address | From / to | +|-----------|---------|-----------| +| IN | `verae.example.process.in` | middleware or another worker | +| OUT | `verae.example.process.out` | peers | +| OUT | `verae.example.process.reply.` | requester | + +Queue group `example-process` shares work. For fan-out (like archive query) **omit** the queue group. + +Fleet: HTTP `/health` on loopback. `keepFloor` in `verae-fleet` if this process must stay up. diff --git a/packages/verae-nats-process/README.md b/packages/verae-nats-process/README.md new file mode 100644 index 0000000..b9bac35 --- /dev/null +++ b/packages/verae-nats-process/README.md @@ -0,0 +1,20 @@ +# verae-nats-process + +Template for a new **addressed messaging process** on the central Verae NATS.IO cluster. + +**Repo:** https://git.georgelambert.org/marchon/verae-nats-process +**Clone:** `ssh://git@git.georgelambert.org:2223/marchon/verae-nats-process.git` +**Overview:** https://git.georgelambert.org/marchon/overview + +```bash +npm test +npm start # /health on 127.0.0.1:13900 +``` + +1. Duplicate this repository (new Forgejo name). +2. Edit `src/subjects.js` — `verae...*`. +3. Replace `handle()` in `src/handle.js`. +4. Update `ROUTING.md`; add a fleet service with `min`/`max`. +5. If Zapier needs the result, add HTTPS on **verae-middleware** only. + +Zapier cloud must not subscribe. See [overview 06-address-routing](https://git.georgelambert.org/marchon/overview/src/branch/main/06-address-routing.md). diff --git a/packages/verae-nats-process/ROUTING.md b/packages/verae-nats-process/ROUTING.md new file mode 100644 index 0000000..76b85ce --- /dev/null +++ b/packages/verae-nats-process/ROUTING.md @@ -0,0 +1,9 @@ +# Address row (copy into overview INDEX and docs-master) + +| Address | Kind | Queue group | Publisher | Subscriber | Body | +|---------|------|-------------|-----------|------------|------| +| `verae.example.process.in` | JetStream work | `example-process` | middleware or peer | this worker | `{ correlationId, payload, traceId }` | +| `verae.example.process.out` | events | — | this worker | interested peers | handle() result | +| `verae.example.process.reply.` | reply | — | this worker | original requester | handle() result | + +Rename `example` / `process` before production. Do not subscribe from Zapier. diff --git a/packages/verae-nats-process/SUMMARY.md b/packages/verae-nats-process/SUMMARY.md new file mode 100644 index 0000000..2c9e58d --- /dev/null +++ b/packages/verae-nats-process/SUMMARY.md @@ -0,0 +1,7 @@ +# verae-nats-process + +**Job:** Reference worker for a **new NATS address**. Copy this repo to add search, storage, or any unplanned function. + +**Expects:** `verae.example.process.in` +**Sends:** `verae.example.process.out` and `reply.` +**Test:** `npm test` diff --git a/packages/verae-nats-process/fleet.service.json b/packages/verae-nats-process/fleet.service.json new file mode 100644 index 0000000..21dc446 --- /dev/null +++ b/packages/verae-nats-process/fleet.service.json @@ -0,0 +1,14 @@ +{ + "id": "verae-nats-process", + "title": "Template addressed process", + "kind": "nats-worker", + "package": "verae-nats-process", + "role": "verae-nats-process", + "managed": true, + "health": { "type": "http", "path": "/health", "timeoutMs": 800 }, + "ports": { "healthBase": 13900 }, + "nats": { + "in": ["verae.example.process.in"], + "out": ["verae.example.process.out", "verae.example.process.reply."] + } +} diff --git a/packages/verae-nats-process/package.json b/packages/verae-nats-process/package.json new file mode 100644 index 0000000..739a547 --- /dev/null +++ b/packages/verae-nats-process/package.json @@ -0,0 +1,11 @@ +{ + "name": "verae-nats-process", + "version": "0.1.0", + "type": "module", + "description": "Template JetStream worker for a new verae.* address (copy this repo to expand)", + "scripts": { + "start": "node src/worker.js", + "test": "node --test test/*.test.js" + }, + "engines": { "node": ">=20" } +} diff --git a/packages/verae-nats-process/src/handle.js b/packages/verae-nats-process/src/handle.js new file mode 100644 index 0000000..c62d537 --- /dev/null +++ b/packages/verae-nats-process/src/handle.js @@ -0,0 +1,20 @@ +/** + * Replace this with search, store, or any new function. + * Must be idempotent: the same correlationId may be redelivered. + * + * @param {object} msg + * @returns {Promise|object} + */ +export function handle(msg = {}) { + const correlationId = msg.correlationId || msg.traceId; + if (!correlationId) { + throw new Error('correlationId or traceId is required'); + } + return { + ok: true, + correlationId, + echo: msg.payload ?? msg, + processedAt: new Date().toISOString(), + note: 'Template handler — replace handle() in src/handle.js', + }; +} diff --git a/packages/verae-nats-process/src/subjects.js b/packages/verae-nats-process/src/subjects.js new file mode 100644 index 0000000..9df5b52 --- /dev/null +++ b/packages/verae-nats-process/src/subjects.js @@ -0,0 +1,13 @@ +/** + * Rename `example.process` before production. + * Pattern: verae... + */ +export const AREA = 'example'; +export const RESOURCE = 'process'; + +export const SUBJECTS = Object.freeze({ + IN: `verae.${AREA}.${RESOURCE}.in`, + OUT: `verae.${AREA}.${RESOURCE}.out`, + QUEUE: `${AREA}-${RESOURCE}`, + reply: (correlationId) => `verae.${AREA}.${RESOURCE}.reply.${correlationId}`, +}); diff --git a/packages/verae-nats-process/src/worker.js b/packages/verae-nats-process/src/worker.js new file mode 100644 index 0000000..cea6c86 --- /dev/null +++ b/packages/verae-nats-process/src/worker.js @@ -0,0 +1,70 @@ +#!/usr/bin/env node +/** + * Health HTTP + optional NATS subscribe. + * Fleet probes /health. Live NATS is off unless NATS_URL is set. + */ +import http from 'node:http'; +import { SUBJECTS } from './subjects.js'; +import { handle } from './handle.js'; + +const port = Number(process.env.FLEET_HEALTH_PORT || process.env.PORT || 13900); +const bind = process.env.FLEET_HEALTH_BIND || '127.0.0.1'; +const instance = process.env.FLEET_INSTANCE || 'verae-nats-process-0'; +let paused = false; +let processed = 0; + +function snapshot() { + return { + ok: !paused, + paused, + instance, + role: 'verae-nats-process', + subjects: { in: SUBJECTS.IN, out: SUBJECTS.OUT, queue: SUBJECTS.QUEUE }, + processed, + }; +} + +const server = http.createServer(async (req, res) => { + const url = new URL(req.url, 'http://127.0.0.1'); + const json = (code, obj) => { + res.writeHead(code, { 'content-type': 'application/json' }); + res.end(JSON.stringify(obj)); + }; + if (url.pathname === '/health' && req.method === 'GET') { + const s = snapshot(); + return json(s.ok ? 200 : 503, s); + } + if (url.pathname === '/pause' && req.method === 'POST') { + paused = true; + return json(200, snapshot()); + } + if (url.pathname === '/resume' && req.method === 'POST') { + paused = false; + return json(200, snapshot()); + } + if (url.pathname === '/message' && req.method === 'POST') { + const chunks = []; + for await (const c of req) chunks.push(c); + let body = {}; + try { + body = JSON.parse(Buffer.concat(chunks).toString('utf8') || '{}'); + } catch { + body = {}; + } + if (paused) return json(503, snapshot()); + try { + const out = await handle(body); + processed += 1; + return json(200, { rttMs: 0, result: out, replyTo: SUBJECTS.reply(out.correlationId) }); + } catch (err) { + return json(400, { error: err.message }); + } + } + json(404, { error: 'not found' }); +}); + +server.listen(port, bind, () => { + process.stdout.write(`${instance} health http://${bind}:${port}/health in=${SUBJECTS.IN}\n`); +}); + +process.on('SIGTERM', () => server.close(() => process.exit(0))); diff --git a/packages/verae-nats-process/test/handle.test.js b/packages/verae-nats-process/test/handle.test.js new file mode 100644 index 0000000..cfecbf6 --- /dev/null +++ b/packages/verae-nats-process/test/handle.test.js @@ -0,0 +1,21 @@ +import { describe, it } from 'node:test'; +import assert from 'node:assert/strict'; +import { handle } from '../src/handle.js'; +import { SUBJECTS } from '../src/subjects.js'; + +describe('verae-nats-process template', () => { + it('uses the verae.area.resource.action address pattern', () => { + assert.equal(SUBJECTS.IN, 'verae.example.process.in'); + assert.equal(SUBJECTS.OUT, 'verae.example.process.out'); + assert.match(SUBJECTS.reply('abc'), /^verae\.example\.process\.reply\.abc$/); + }); + + it('handle requires correlation and is idempotent-shaped', () => { + assert.throws(() => handle({}), /correlationId/); + const a = handle({ correlationId: 'c1', payload: { q: 1 } }); + const b = handle({ correlationId: 'c1', payload: { q: 1 } }); + assert.equal(a.ok, true); + assert.equal(a.correlationId, b.correlationId); + assert.equal(a.echo.q, 1); + }); +}); diff --git a/scripts/build-docs-site.py b/scripts/build-docs-site.py index 47f8a2d..cd5e90c 100755 --- a/scripts/build-docs-site.py +++ b/scripts/build-docs-site.py @@ -33,6 +33,8 @@ SECTIONS = [ ("docs/02-architecture/modules-and-nats.md", "Module catalog and NATS addresses"), ("docs/02-architecture/tree-nodes.md", "Tree nodes and bulk Merkle summaries"), ("docs/02-architecture/fleet.md", "Fleet replica floors and monitor"), + ("packages/overview/README.md", "System overview (TOC)"), + ("packages/overview/INDEX.md", "Documentation index"), ], ), ( @@ -141,6 +143,8 @@ def main() -> None: "verae-zapier-simulator", "zapier-user-docs", "verae-fleet", + "overview", + "verae-nats-process", "docs-master", ): readme = ROOT / "packages" / pkg / "README.md" @@ -157,6 +161,7 @@ def main() -> None: copy_tree(src, SITE / src.name if src.parent == ROOT else SITE / "docs" / "OPEN.md") copy_tree(ROOT / "docs" / "WORK-LOG.md", SITE / "docs" / "WORK-LOG.md") copy_tree(ROOT / "packages" / "docs-master", SITE / "docs-master") + copy_tree(ROOT / "packages" / "overview", SITE / "overview") copy_tree(ROOT / "packages" / "zapier-user-docs", SITE / "user-docs", ignore=shutil.ignore_patterns("node_modules")) sim_pub = ROOT / "packages" / "verae-zapier-simulator" / "public" / "index.html" if sim_pub.exists(): @@ -235,6 +240,8 @@ def main() -> None: ("verae-zapier-simulator", "Zapier interface + trace console"), ("zapier-user-docs", "Signup-to-usage user guide"), ("verae-fleet", "Service catalog, replica floors, monitor"), + ("overview", "High-level system overview"), + ("verae-nats-process", "Template for a new NATS address"), ] git_lis = "".join( f'
  • {name} ' @@ -292,6 +299,9 @@ def main() -> None:
  • Per-module SUMMARY.md and NATS.md
  • User guide (signup → tree-node lookup)
  • Simulator README
    • +
  • System overview
    • +
  • Documentation index
    • +
  • System diagram (SVG)
    • diff --git a/scripts/push-module-repos.sh b/scripts/push-module-repos.sh index 1b93ec1..1ec6672 100755 --- a/scripts/push-module-repos.sh +++ b/scripts/push-module-repos.sh @@ -47,6 +47,8 @@ create verae-tree-node "Merkle leaf proofs for hashes only sealed as a bulk summ create verae-zapier-simulator "Zapier interface simulator with hop-by-hop trace console" create zapier-user-docs "User docs from signup through hash register, chain lookup, tree-node query" create verae-fleet "Service catalog, replica floors, monitor, restart, pause" +create overview "High-level system overview, NATS cluster, uptime, docs index" +create verae-nats-process "Template JetStream worker for a new verae.* address" push_dir "$ROOT/packages/zappier" zappier-edge push_dir "$ROOT/packages/verae-zapier-middleware" verae-middleware @@ -60,5 +62,7 @@ push_dir "$ROOT/packages/verae-tree-node" verae-tree-node push_dir "$ROOT/packages/verae-zapier-simulator" verae-zapier-simulator push_dir "$ROOT/packages/zapier-user-docs" zapier-user-docs push_dir "$ROOT/packages/verae-fleet" verae-fleet +push_dir "$ROOT/packages/overview" overview +push_dir "$ROOT/packages/verae-nats-process" verae-nats-process echo ALL_MODULE_REPOS_PUSHED