Add overview repo and verae-nats-process expansion template
Some checks are pending
offline / test (push) Waiting to run

High-level system map with diagrams, TOC, and a docs index. Template
worker shows how to add a new verae.* address for search, storage, or
unplanned functions without teaching Zapier NATS.
This commit is contained in:
George Lambert 2026-09-11 13:50:37 -04:00
parent a694f246fb
commit a32c91475c
37 changed files with 767 additions and 0 deletions

View file

@ -35,6 +35,8 @@ Zapier cloud **never** connects to NATS. Only middleware workers and archives do
| tree-node | `verae-tree-node` | N copies (WORM role) | `verae.archive.query`, `verae.archive.put` kind `tree` | `verae.archive.reply.<id>` if bloom hits |
| zapier-simulator | `verae-zapier-simulator` | Local HTTP :3847 | operator browser | in-process replay of all addresses |
| fleet | `verae-fleet` | Local HTTP :3850 | operator | spawns workers; keepFloor on tree-node |
| overview | `overview` | Static | — | high-level system map |
| nats-process | `verae-nats-process` | Worker template | `verae.example.process.in` | `.out` / `.reply.*` |
| zapier-user-docs | `zapier-user-docs` | Static | — | catalog `/user-docs/` |
| verae-chain-client | `verae-chain-client` | Library | — | HTTPS `api.veraetime.net` or MOCK |
| docs-master | `zapier-docs-master` | Static | — | published on zapier.georgelambert.org |

View file

@ -21,6 +21,8 @@ Summaries, NATS contracts, and message flows for every Verae Time × Zapier modu
| verae-zapier-simulator | https://git.georgelambert.org/marchon/verae-zapier-simulator |
| zapier-user-docs | https://git.georgelambert.org/marchon/zapier-user-docs |
| verae-fleet | https://git.georgelambert.org/marchon/verae-fleet |
| **overview** | https://git.georgelambert.org/marchon/overview |
| verae-nats-process | https://git.georgelambert.org/marchon/verae-nats-process |
| **zapier-docs-master** (this repo) | https://git.georgelambert.org/marchon/zapier-docs-master |
Clone (SSH port 2223):
@ -42,6 +44,8 @@ Clone (SSH port 2223):
| zapier-simulator | Trace console (in-process) | — | — |
| zapier-user-docs | Signup → lookup guide | — | — |
| verae-fleet | Replica floors + monitor | — | — |
| overview | System map | — | — |
| nats-process | Template worker | example.process.in | example.process.out / reply.* |
## Documents in this repo

View file

@ -36,6 +36,9 @@ Zapier cloud **never** connects to NATS. Only middleware workers and archives do
| zapier-simulator | `verae-zapier-simulator` | Local HTTP :3847 | operator browser | in-process replay of all addresses |
| zapier-user-docs | `zapier-user-docs` | Static | — | catalog `/user-docs/` |
| verae-chain-client | `verae-chain-client` | Library | — | HTTPS `api.veraetime.net` or MOCK |
| fleet | `verae-fleet` | Local HTTP :3850 | operator | keepFloor + SSH hosts |
| overview | `overview` | Static | — | high-level map |
| nats-process | `verae-nats-process` | Template worker | `verae.example.process.in` | `.out` / `.reply.*` |
| docs-master | `zapier-docs-master` | Static | — | published on zapier.georgelambert.org |
## Addresses (subjects)

View file

@ -0,0 +1,3 @@
# NATS — overview
Documentation only. Describes the central 3-server cluster. Zapier never connects.

View file

@ -0,0 +1,3 @@
# overview
High-level system map, 3-node NATS cluster, uptime, network failure, address routing. Repo: `overview`.

View file

@ -0,0 +1,3 @@
# NATS — verae-nats-process
IN `verae.example.process.in` (queue `example-process`). OUT `.out` and `.reply.<id>`.

View file

@ -0,0 +1,3 @@
# verae-nats-process
Template for a new `verae.<area>.<resource>.*` worker. Copy this repo to expand.

View file

@ -0,0 +1,26 @@
# 1. System
Verae Time proves a SHA-256 existed at a given time. Zapier lets customers register and look up hashes from the tools they already use. Billing and API keys live on **zappier-edge**. Timestamping, wait, webhooks, and archive fan-out live behind **verae-middleware**. Durable messaging is the **central Verae NATS.IO 3-server JetStream cluster**.
## Hard rule
**Zapier never connects to NATS, tree nodes, WORM archives, or `api.veraetime.net`.** Those hops are middleware and workers only. If a trace ever shows a Zapier hop on a `verae.*` subject, do not push the app.
## Planes
| Plane | What | Network |
|-------|------|---------|
| Zapier cloud | `verae-zapier-app`, `verae-activate` | HTTPS to zappier-edge |
| Commercial edge | `zappier-edge` portal, admin, `x-api-key`, Stripe meter | Public HTTPS |
| Middleware HTTP | `/zapier/v1/*` job id + wait | Public HTTPS from edge only |
| NATS cluster | JetStream subjects under `verae.*` | Private; loopback or SSH tunnel |
| Workers | poller, webhook-deliver, aggregator | NATS + HTTPS to chain or Zapier hooks |
| Archives | WORM + tree nodes | NATS broadcast query; bloom miss = silence |
| Chain | Verae timestamping | HTTPS `api.veraetime.net` or MOCK |
| Control | `verae-fleet` | Operator loopback `:3850`; SSH to extra machines |
## Request in one sentence
A Zap step POSTs to zappier-edge; edge meters and forwards to middleware; middleware splits hash vs files, writes the hash (or Merkle root) to the chain, publishes `verae.zapier.jobs.watch`, and either returns `jobId` or waits on `verae.zapier.jobs.events`. Attached metadata and bulk-summary **leaves** are found later by broadcasting `verae.archive.query` to every archive/tree node.
See [diagrams](08-diagrams.md).

View file

@ -0,0 +1,25 @@
# 2. Modules and independent repositories
Each runtime piece is its **own git repo** on Forgejo (`git.georgelambert.org`, SSH port 2223). The monorepo `master-zapier-plan-draft` is a snapshot of the workspace; do not treat it as the only clone path.
| Independent repo | Package / path | What it does |
|------------------|----------------|--------------|
| **overview** | `packages/overview` | This high-level map |
| **verae-nats-process** | `packages/verae-nats-process` | **Template** for a new addressed NATS process |
| **master-zapier-plan-draft** | workspace root | Combined snapshot (`main` and `master`) |
| **zappier-edge** | `packages/zappier` | Metered HTTPS, portal, admin, Stripe `zappier.api_cents` |
| **verae-middleware** | `packages/verae-zapier-middleware` | `/zapier/v1`, job wait, NATS publishers/workers |
| **verae-zapier-app** | `packages/verae-zapier` | Full Zapier nouns (timestamp, wait, batch, hash + tree lookup) |
| **verae-activate** | `packages/verae-activate` | Activate-now (Add Numbers, Echo, SHA256, mock timestamp) |
| **verae-request-splitter** | `packages/verae-request-splitter` | Chain hash vs `archive.put` |
| **verae-archive-worm** | `packages/verae-archive-worm` | Bloom WORM node |
| **verae-archive-aggregator** | `packages/verae-archive-aggregator` | Merge archive replies |
| **verae-tree-node** | `packages/verae-tree-node` | Merkle leaf proofs (bulk summaries) |
| **verae-fleet** | `packages/verae-fleet` | Catalog, min copies, pause/restart, SSH hosts |
| **verae-zapier-simulator** | `packages/verae-zapier-simulator` | Trace console before `zapier-platform push` |
| **zapier-user-docs** | `packages/zapier-user-docs` | Customer signup → register → lookup |
| **zapier-docs-master** | `packages/docs-master` | Per-module `SUMMARY.md` + `NATS.md` |
Libraries that are **not** separate Forgejo apps today: `verae-chain-client` (inside middleware), job-poller and webhook-deliver (middleware workers, fleet-spawned).
Per-module NATS contracts: [zapier-docs-master](https://git.georgelambert.org/marchon/zapier-docs-master).

View file

@ -0,0 +1,37 @@
# 3. Central Verae NATS.IO 3-server cluster
All durable messaging for this product is **NATS JetStream**, not Zapier queues and not a public TCP API.
## Cluster
Three `nats-server -js` nodes form the Verae cluster (routes between them, JetStream replication). Clients (middleware, poller, aggregator, WORM, tree nodes, and any new process cloned from `verae-nats-process`) connect with a **cluster URL list**, for example:
```text
nats://127.0.0.1:4222,nats://127.0.0.1:4223,nats://127.0.0.1:4224
```
On a given machine the listener stays on **loopback** (or a private interface). Operators reach it with `scripts/nats-tunnel.sh` / `ssh -L 14222:127.0.0.1:4222`. **Do not bind 4222 on `0.0.0.0` without auth.**
Todays NS1 box (`NS1.GEORGELAMBERT.ORG`, `70.88.205.138`) already runs JetStream on `127.0.0.1:4222`. The **target** is three clustered nodes so losing one server does not lose the stream. Fleet SSH hosts (`ns1`, later `lan-134`) run **workers**, not extra public NATS listeners.
![NATS cluster](diagrams/nats-cluster.svg)
## Who may connect
| Allowed | Forbidden |
|---------|-----------|
| verae-middleware, fleet workers, WORM, tree nodes, `verae-nats-process` clones | Zapier cloud, browsers, zappier-edge |
## Address families already in use
| Address | Kind | Notes |
|---------|------|--------|
| `verae.zapier.jobs.watch` | work queue | poller |
| `verae.zapier.jobs.events` | events | wait + webhooks |
| `verae.zapier.webhooks.deliver` | work queue | HTTPS to Zapier REST Hook |
| `verae.zapier.usage` | optional | metering |
| `verae.archive.put` | JetStream | splitter / merkle builder |
| `verae.archive.query` | **broadcast** (no queue group) | every WORM and tree node |
| `verae.archive.reply.<correlationId>` | replies | **only on bloom hit** |
New functions get new `verae.<area>.<resource>.<action>` addresses — see [06-address-routing.md](06-address-routing.md).

View file

@ -0,0 +1,34 @@
# 4. How the system maintains uptime
Uptime is **NATS durability + fleet replica floors + more than one machine**, not a single always-on Zapier connection.
## Replica floors (`verae-fleet`)
`packages/verae-fleet/fleet.json` sets `min` / `max` / `keepFloor` per service. **Available** means running, healthy, and not paused.
| Service | Default min | keepFloor |
|---------|-------------|-----------|
| tree-node | 3 | yes |
| archive-worm | 3 | yes |
| archive-aggregator, job-poller, webhook-deliver | 1 | yes |
| zappier-edge, middleware-http | 1 | yes |
If a tree node is paused, crashes, or fails `/health`, fleet **starts another copy** until three are available. Monitor: http://127.0.0.1:3850/ (green / yellow / red rows).
## Restart and pause
- Unhealthy `/health` → same instance id restarted.
- Pause does not count toward `min`.
- `stop` on a service disables keepFloor for that service.
## Spread across machines
`machines.json` lists hosts (`local`, `ns1` = `marchon@70.88.205.138` with `~/.ssh/id_ed25519`, optional `lan-134`). New replicas go to the **least-loaded** eligible host. Remote spawn/health/kill is SSH; workers bind loopback on the remote box.
## JetStream
Work queues (`jobs.watch`, `webhooks.deliver`) replay if a consumer dies. Event stream (`jobs.events`) lets waiters and webhook routers catch up. A 3-node cluster keeps the stream if one NATS server is down.
## What Zapier sees
HTTPS 202 `jobId`, wait JSON, or REST Hook. Timeouts return `pending` + `jobId` so the **Timestamp Completed** trigger can finish the job. Zapier retries are safe: the same SHA-256 returns the original seal (`existing: true`).

View file

@ -0,0 +1,23 @@
# 5. Local network failures
“Local network” means the operator LAN, SSH to NS1, or a partitioned archive — not Zapiers cloud.
| Failure | What happens | What the customer sees |
|---------|--------------|------------------------|
| NATS node unreachable | Client reconnects to another cluster URL; JetStream consumers resume | Wait may return `pending`; hook still fires later |
| All NATS down | Middleware cannot publish `jobs.watch`; fleet marks workers unhealthy | 503 / pending; no NATS leak to Zapier |
| One WORM / tree node partitioned | Bloom miss = **no packet**; aggregator uses whoever answered | Lookup may miss attachments until the node returns; seal on chain still valid |
| SSH to a fleet host fails | Spawn fails; fleet **places the next replica on another machine** | Floor still met if capacity remains on `local` or another SSH host |
| Chain `api.veraetime.net` timeout | Poller retries; then `timestamp.timeout` event | Wait → pending or failed; async + hook still the recovery path |
| zappier-edge 402 | QuotaExceeded with upgrade URL | Zap step error; no NATS involved |
| Tunnel to loopback NATS dropped | `NATS_URL=nats://127.0.0.1:14222` dies; restart `nats-tunnel.sh` | Workers on NS1 itself still see `127.0.0.1:4222` |
## Design choices that make partitions survivable
1. **Archive query is broadcast**, not a shared queue group — a dead node does not steal the message.
2. **Bloom miss is silence** — missing nodes do not send empty errors that look like “hash unknown”.
3. **Zero replies + known puts** is an outage, not a miss (simulator / fleet monitors flag this).
4. **NATS is not on the public NIC** — a WAN blip does not expose 4222.
5. **Idempotent seals** — retrying a Zap after a network error will not double-timestamp.
See fleet RTT (min / avg / p50 / p90) when planning extra tree nodes after a flaky path.

View file

@ -0,0 +1,33 @@
# 6. Address routing (including unplanned functions)
NATS **addresses** (subjects) are the extension point. A new search, store, or job type is a new address plus a process that listens — not a new Zapier TCP client.
## Pattern
```text
verae.<area>.<resource>.<action>
verae.<area>.<resource>.reply.<correlationId>
```
| Piece | Example | Meaning |
|-------|---------|---------|
| `verae` | — | Verae bus (not Zapier) |
| `area` | `zapier`, `archive`, `search`, `store` | Product slice |
| `resource` | `jobs`, `hashes`, `blobs` | Noun |
| `action` | `watch`, `query`, `put`, `in` | Verb |
| `reply.<id>` | — | Correlated response |
**Queue group** (work sharing): `area-resource-action` (e.g. `job-poller`).
**No queue group** (fan-out): archive/tree **query** so every node sees every lookup.
## Adding something that does not exist yet
1. Copy the independent repo **[verae-nats-process](https://git.georgelambert.org/marchon/verae-nats-process)** (`packages/verae-nats-process`).
2. Rename `verae.example.process.in` / `.out` / `.reply.*` in `src/subjects.js`.
3. Add a row to that repos `ROUTING.md` and to [INDEX.md](INDEX.md).
4. Register the process in `verae-fleet` (`min`/`max`, machines, roles).
5. If Zapier must call it, add **one HTTPS route** on middleware — Zapier still never sees NATS.
Do **not** invent a public NATS URL for Zapier. Do **not** reuse `verae.archive.query` as a queue group.
![Address expansion](diagrams/routing.svg)

View file

@ -0,0 +1,15 @@
# 7. External resources: search, storage, and the chain
| Resource | Where | How a Zap reaches it |
|----------|--------|----------------------|
| **Central chain** (itemized SHA-256 + time + certificate) | `api.veraetime.net` or MOCK | HTTPS via middleware: create / wait / `GET /hashes/{sha256}` |
| **Bulk Merkle root** | Same chain, one seal | `POST /timestamp/batch` |
| **Leaf proofs** (hash only in a bulk summary) | Tree-node WORM on NATS | `GET /hashes/{sha}?includeTree=true``verae.archive.query` kinds=`tree` |
| **Public / private metadata, files** | WORM archives | `includeAttached`; never on chain |
| **Search (central)** | Chain lookup | Zapier search **Find Timestamp by SHA256** |
| **Search (external tree / extra stores)** | NATS query to every node that might hold the key | Zapier search **Find Hash (tree nodes + central)** |
| **Future search / store** | New `verae.search.*` or `verae.store.*` process | Copy `verae-nats-process`; optional middleware GET |
Storage that is **not** the blockchain stays on WORM/tree nodes (and later any process that answers `verae.archive.query` or a new store address). The chain stores hash + time + certificate (+ Merkle root for batches).
External “unplanned” storage or search is the same pattern: new address, bloom or index on that node, silence on miss, aggregator or the templates reply subject.

View file

@ -0,0 +1,59 @@
# 8. Architectural diagrams
SVG files in [`diagrams/`](diagrams/). The same shapes are repeated below in mermaid for Forgejo preview.
## 8.1 End-to-end (HTTPS vs NATS)
![System](diagrams/system.svg)
```mermaid
flowchart LR
Z[Zapier cloud apps] -->|HTTPS x-api-key| E[zappier-edge]
E -->|HTTPS| M[verae-middleware]
M -->|HTTPS| C[Verae chain]
M -->|JetStream| N[NATS 3-node cluster]
N --> P[job-poller]
N --> W[webhook-deliver]
N --> A[archive-aggregator]
N --> R[WORM x N]
N --> T[tree-node x N]
P -->|HTTPS status| C
W -->|HTTPS REST Hook| Z
```
## 8.2 NATS.IO 3-server cluster
![Cluster](diagrams/nats-cluster.svg)
```mermaid
flowchart TB
subgraph cluster [Verae NATS.IO JetStream]
N1[nats-server A :4222 loopback]
N2[nats-server B]
N3[nats-server C]
N1 <--> N2
N2 <--> N3
N3 <--> N1
end
MW[middleware + workers] -->|cluster URL list| cluster
F[fleet SSH hosts] -->|workers only| cluster
```
## 8.3 Uptime and failure
![Uptime](diagrams/uptime.svg)
```mermaid
flowchart TD
H[health /health] -->|fail| R[restart instance]
P[pause] -->|not available| F[keepFloor spawn]
S[SSH host down] -->|spawn failed| O[place on next machine]
Q[archive.query broadcast] -->|bloom miss| SIL[silence]
Q -->|hit| REP[archive.reply.id]
```
## 8.4 New address = new process
![Routing](diagrams/routing.svg)
See [09-expansion-template.md](09-expansion-template.md).

View file

@ -0,0 +1,29 @@
# 9. Model repo for a new addressed process
**Repository:** [verae-nats-process](https://git.georgelambert.org/marchon/verae-nats-process)
**Path in the monorepo:** `packages/verae-nats-process`
**Clone:** `ssh://git@git.georgelambert.org:2223/marchon/verae-nats-process.git`
This is the **reference implementation** for expansions: a small JetStream worker with:
- a single **in** address, an **out** address, and **reply.&lt;correlationId&gt;**
- `handle(msg)` you replace with real work (search, store, transform)
- HTTP `/health` so `verae-fleet` can keep a replica floor
- tests that do not need a live cluster
- `ROUTING.md` — the row you copy into the global address table
Default subjects (rename before production):
| Direction | Address |
|-----------|---------|
| IN | `verae.example.process.in` |
| OUT | `verae.example.process.out` |
| REPLY | `verae.example.process.reply.<correlationId>` |
```bash
cd packages/verae-nats-process
npm test
# then rename example → your area, add fleet.json min, clone as a new Forgejo repo
```
Zapier still must not subscribe. If a Zap needs the result, middleware exposes HTTPS and publishes to the new **in** address.

View file

@ -0,0 +1,70 @@
# Documentation index
Pointers into the rest of the documentation. Live HTML catalog: https://zapier.georgelambert.org/
## This overview
| Topic | File |
|-------|------|
| TOC / intro | [README.md](README.md) |
| System | [01-system.md](01-system.md) |
| Modules and repos | [02-modules-and-repos.md](02-modules-and-repos.md) |
| NATS 3-server cluster | [03-nats-cluster.md](03-nats-cluster.md) |
| Uptime | [04-uptime.md](04-uptime.md) |
| Local network failures | [05-network-failures.md](05-network-failures.md) |
| Address routing | [06-address-routing.md](06-address-routing.md) |
| External search / storage | [07-external-resources.md](07-external-resources.md) |
| Diagrams | [08-diagrams.md](08-diagrams.md) |
| Expansion template | [09-expansion-template.md](09-expansion-template.md) |
## Catalog (HTML)
| Document | URL |
|----------|-----|
| Catalog home | https://zapier.georgelambert.org/ |
| Composition | https://zapier.georgelambert.org/docs/02-architecture/composition.html |
| NATS gateway | https://zapier.georgelambert.org/docs/02-architecture/nats-gateway.html |
| NATS subjects | https://zapier.georgelambert.org/docs/02-architecture/nats-subjects.html |
| Archive / bloom | https://zapier.georgelambert.org/docs/02-architecture/archive-nats.html |
| Module NATS map | https://zapier.georgelambert.org/docs/02-architecture/modules-and-nats.html |
| Tree nodes | https://zapier.georgelambert.org/docs/02-architecture/tree-nodes.html |
| Fleet | https://zapier.georgelambert.org/docs/02-architecture/fleet.html |
| User guide | https://zapier.georgelambert.org/user-docs/README.md |
| Tree-node lookup (users) | https://zapier.georgelambert.org/packages/zapier-user-docs/09-lookup-tree-nodes.html |
| Zapier developer setup | https://zapier.georgelambert.org/docs/04-activate/SETUP-ZAPIER-DEVELOPER.html |
| docs-master | https://zapier.georgelambert.org/docs-master/README.md |
## Independent git repositories
Prefix: `https://git.georgelambert.org/marchon/`
overview · verae-nats-process · master-zapier-plan-draft · zappier-edge · verae-middleware · verae-zapier-app · verae-activate · verae-request-splitter · verae-archive-worm · verae-archive-aggregator · verae-tree-node · verae-fleet · verae-zapier-simulator · zapier-user-docs · zapier-docs-master
## Per-module contracts
In **zapier-docs-master**: `modules/<package>/SUMMARY.md` and `NATS.md` for every runtime package.
## Operator surfaces (not public NATS)
| Surface | URL |
|---------|-----|
| Fleet monitor | http://127.0.0.1:3850/ |
| Zapier simulator | http://127.0.0.1:3847/ |
## AZ subject index
| Term | See |
|------|-----|
| aggregator | verae-archive-aggregator; archive.query / reply |
| bloom miss | silence; 05-network-failures |
| batch / Merkle | verae-tree-node; user-docs 0809 |
| cluster | 03-nats-cluster |
| fleet / keepFloor | verae-fleet; 04-uptime |
| hash lookup | verae-zapier-app searches; 07-external-resources |
| jobs.events / jobs.watch | nats-subjects.html; middleware |
| includeTree | tree lookup; archive.query kinds=tree |
| routing | 06-address-routing; verae-nats-process |
| splitter | verae-request-splitter |
| SSH hosts | verae-fleet machines.json |
| Zapier never NATS | 01-system |
| zappier-edge | metering, portal |

View file

@ -0,0 +1,7 @@
# NATS — overview
This documentation repo does **not** subscribe. It describes the **central Verae NATS.IO 3-server JetStream cluster** that every worker talks to.
Zapier cloud never connects here. Clients use private URLs (`nats://127.0.0.1:4222` on a node, or an SSH tunnel). Do not bind 4222 on `0.0.0.0` without auth.
New functions are new **addresses** (subjects), not new public TCP ports. Copy `verae-nats-process` to add one.

View file

@ -0,0 +1,48 @@
# Verae Time × Zapier — system overview
High-level description of the whole system: what it is, which **independent git repositories** implement it, how it stays up, how it behaves when the local network fails, and how it talks to the **central Verae NATS.IO 3-server cluster**. New work is added by **new address routing**, not by teaching Zapier about NATS.
**This repo:** https://git.georgelambert.org/marchon/overview
**Live catalog:** https://zapier.georgelambert.org/
**Clone:** `ssh://git@git.georgelambert.org:2223/marchon/overview.git`
## Table of contents
1. [System](01-system.md) — what runs where; Zapier never speaks NATS
2. [Modules and repositories](02-modules-and-repos.md) — every independent Forgejo repo
3. [NATS.IO 3-server cluster](03-nats-cluster.md) — JetStream, subjects, who may connect
4. [Uptime](04-uptime.md) — replica floors, restart, SSH spread
5. [Local network failures](05-network-failures.md) — reconnect, silence, pending, failover
6. [Address routing](06-address-routing.md) — how to add unplanned functions
7. [External resources](07-external-resources.md) — searches, storage, chain
8. [Architectural diagrams](08-diagrams.md) — SVG + mermaid
9. [Expansion template](09-expansion-template.md) — `verae-nats-process`
10. [Documentation index](INDEX.md) — pointer into all other docs
## One-screen picture
![System context](diagrams/system.svg)
Zapier → HTTPS `zappier-edge` → HTTPS `verae-middleware`**NATS cluster** → workers, WORM archives, tree nodes. Chain HTTPS is `api.veraetime.net` (or MOCK). Fleet keeps minimum copies, including tree nodes.
## Independent repositories (Forgejo)
| Repo | Role |
|------|------|
| [overview](https://git.georgelambert.org/marchon/overview) | This document |
| [verae-nats-process](https://git.georgelambert.org/marchon/verae-nats-process) | **Model** for a new addressed process |
| [master-zapier-plan-draft](https://git.georgelambert.org/marchon/master-zapier-plan-draft) | Monorepo snapshot |
| [zappier-edge](https://git.georgelambert.org/marchon/zappier-edge) | Metered public HTTPS |
| [verae-middleware](https://git.georgelambert.org/marchon/verae-middleware) | Zapier HTTP + NATS workers |
| [verae-zapier-app](https://git.georgelambert.org/marchon/verae-zapier-app) | Zapier Platform app |
| [verae-activate](https://git.georgelambert.org/marchon/verae-activate) | Activate-now app |
| [verae-request-splitter](https://git.georgelambert.org/marchon/verae-request-splitter) | Hash vs attachments |
| [verae-archive-worm](https://git.georgelambert.org/marchon/verae-archive-worm) | Bloom WORM node |
| [verae-archive-aggregator](https://git.georgelambert.org/marchon/verae-archive-aggregator) | Archive reply merge |
| [verae-tree-node](https://git.georgelambert.org/marchon/verae-tree-node) | Merkle leaf proofs |
| [verae-fleet](https://git.georgelambert.org/marchon/verae-fleet) | Replica floor + SSH hosts |
| [verae-zapier-simulator](https://git.georgelambert.org/marchon/verae-zapier-simulator) | Trace console |
| [zapier-user-docs](https://git.georgelambert.org/marchon/zapier-user-docs) | Signup → lookup |
| [zapier-docs-master](https://git.georgelambert.org/marchon/zapier-docs-master) | Per-module SUMMARY + NATS |
Clone any of them: `git clone ssh://git@git.georgelambert.org:2223/marchon/<name>.git` (SSH port **2223**).

View file

@ -0,0 +1,7 @@
# overview
**Job:** High-level map of the whole Verae Time × Zapier system: modules, independent git repos, uptime, local-network failure, the central 3-node NATS.IO cluster, and how to add a new addressed process.
**Expects / sends:** none (documentation).
**Repos named:** see [02-modules-and-repos.md](02-modules-and-repos.md). Expansion template: `verae-nats-process`.

View file

@ -0,0 +1,21 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 720 300" width="720" height="300" role="img" aria-label="Three nats-server nodes clustered, clients on private URLs">
<rect width="720" height="300" fill="#f6f3ee"/>
<text x="20" y="28" font-family="Georgia,serif" font-size="16" fill="#12202c">Verae NATS.IO — three JetStream servers</text>
<g font-family="system-ui,sans-serif" font-size="12" text-anchor="middle">
<circle cx="160" cy="150" r="54" fill="#1d4f91"/>
<text x="160" y="146" fill="#fff">A</text>
<text x="160" y="164" fill="#d9e2e8" font-size="10">:4222 loopback</text>
<circle cx="360" cy="150" r="54" fill="#1d4f91"/>
<text x="360" y="146" fill="#fff">B</text>
<text x="360" y="164" fill="#d9e2e8" font-size="10">route</text>
<circle cx="560" cy="150" r="54" fill="#1d4f91"/>
<text x="560" y="146" fill="#fff">C</text>
<text x="560" y="164" fill="#d9e2e8" font-size="10">JetStream</text>
<line x1="214" y1="150" x2="306" y2="150" stroke="#12202c" stroke-width="2"/>
<line x1="414" y1="150" x2="506" y2="150" stroke="#12202c" stroke-width="2"/>
<path d="M160 96 C 360 20 360 20 560 96" fill="none" stroke="#12202c" stroke-width="2"/>
<rect x="40" y="230" width="200" height="44" rx="8" fill="#12202c"/>
<text x="140" y="256" fill="#f6f3ee">workers + middleware</text>
<text x="250" y="256" fill="#52646f" font-size="11" text-anchor="start">→ nats://A,B,C (never Zapier)</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 1.4 KiB

View file

@ -0,0 +1,25 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 840 260" width="840" height="260" role="img" aria-label="Copy verae-nats-process to add a new subject">
<rect width="840" height="260" fill="#f6f3ee"/>
<text x="20" y="28" font-family="Georgia,serif" font-size="16" fill="#12202c">New function = new address + clone of verae-nats-process</text>
<g font-family="system-ui,sans-serif" font-size="12" text-anchor="middle">
<rect x="20" y="60" width="160" height="64" rx="8" fill="#12202c"/>
<text x="100" y="88" fill="#f6f3ee">copy template repo</text>
<text x="100" y="106" fill="#c5d0d8" font-size="10">verae-nats-process</text>
<rect x="220" y="60" width="180" height="64" rx="8" fill="#0b4f8a"/>
<text x="310" y="88" fill="#fff">rename subjects</text>
<text x="310" y="106" fill="#d9e2e8" font-size="10">verae.area.resource.action</text>
<rect x="440" y="60" width="160" height="64" rx="8" fill="#0f6e56"/>
<text x="520" y="88" fill="#fff">fleet min/max</text>
<text x="520" y="106" fill="#d4efe6" font-size="10">keepFloor + machines</text>
<rect x="640" y="60" width="180" height="64" rx="8" fill="#8a5a00"/>
<text x="730" y="88" fill="#fff">optional HTTPS</text>
<text x="730" y="106" fill="#fff3bf" font-size="10">middleware only → Zapier</text>
<polygon points="180,92 220,92 220,88 232,92 220,96 220,92" fill="#52646f"/>
<polygon points="400,92 440,92 440,88 452,92 440,96 440,92" fill="#52646f"/>
<polygon points="600,92 640,92 640,88 652,92 640,96 640,92" fill="#52646f"/>
<rect x="20" y="150" width="800" height="84" rx="8" fill="#fff" stroke="#d5dee4"/>
<text x="420" y="178" fill="#12202c">Existing: verae.zapier.jobs.* · verae.archive.put|query|reply.*</text>
<text x="420" y="200" fill="#52646f" font-size="12">Unplanned search/store: verae.search.* or verae.store.* — same worker shape. Zapier never subscribes.</text>
<text x="420" y="218" fill="#52646f" font-size="12">Broadcast queries: no queue group. Work queues: queue group. Bloom miss: no packet.</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 2 KiB

View file

@ -0,0 +1,47 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 920 420" width="920" height="420" role="img" aria-label="System context: Zapier HTTPS to edge to middleware to NATS cluster">
<rect width="920" height="420" fill="#f6f3ee"/>
<text x="24" y="28" font-family="Georgia,serif" font-size="16" fill="#12202c">Verae Time × Zapier — system context</text>
<text x="24" y="48" font-family="system-ui,sans-serif" font-size="11" fill="#52646f">Zapier never speaks NATS. HTTPS stops at middleware. JetStream is the 3-node cluster.</text>
<g font-family="system-ui,sans-serif" font-size="12" text-anchor="middle">
<rect x="20" y="80" width="140" height="56" rx="8" fill="#12202c"/>
<text x="90" y="104" fill="#f6f3ee">Zapier cloud</text>
<text x="90" y="120" fill="#c5d0d8" font-size="10">verae-zapier-app</text>
<rect x="200" y="80" width="140" height="56" rx="8" fill="#0f6e56"/>
<text x="270" y="104" fill="#fff">zappier-edge</text>
<text x="270" y="120" fill="#d4efe6" font-size="10">HTTPS :3000 meter</text>
<rect x="380" y="80" width="160" height="56" rx="8" fill="#0b4f8a"/>
<text x="460" y="104" fill="#fff">verae-middleware</text>
<text x="460" y="120" fill="#d9e2e8" font-size="10">HTTPS :3100 wait</text>
<rect x="600" y="80" width="140" height="56" rx="8" fill="#5b2d8a"/>
<text x="670" y="104" fill="#fff">Verae chain</text>
<text x="670" y="120" fill="#e8d9f5" font-size="10">api.veraetime.net</text>
<polygon points="160,108 200,108 200,104 212,108 200,112 200,108" fill="#52646f"/>
<text x="180" y="98" font-size="9" fill="#52646f">HTTPS</text>
<polygon points="340,108 380,108 380,104 392,108 380,112 380,108" fill="#52646f"/>
<text x="360" y="98" font-size="9" fill="#52646f">HTTPS</text>
<polygon points="540,108 600,108 600,104 612,108 600,112 600,108" fill="#52646f"/>
<text x="568" y="98" font-size="9" fill="#52646f">HTTPS hash</text>
<rect x="200" y="200" width="520" height="180" rx="10" fill="#e8eef2" stroke="#1d4f91" stroke-width="1.5"/>
<text x="460" y="222" fill="#1d4f91" font-size="13">Central Verae NATS.IO 3-server JetStream cluster</text>
<rect x="230" y="240" width="100" height="40" rx="6" fill="#1d4f91"/>
<text x="280" y="264" fill="#fff">node A</text>
<rect x="410" y="240" width="100" height="40" rx="6" fill="#1d4f91"/>
<text x="460" y="264" fill="#fff">node B</text>
<rect x="590" y="240" width="100" height="40" rx="6" fill="#1d4f91"/>
<text x="640" y="264" fill="#fff">node C</text>
<rect x="220" y="310" width="110" height="48" rx="6" fill="#fff" stroke="#0f6e56"/>
<text x="275" y="330" fill="#12202c">job-poller</text>
<text x="275" y="346" fill="#52646f" font-size="10">jobs.watch</text>
<rect x="345" y="310" width="120" height="48" rx="6" fill="#fff" stroke="#0f6e56"/>
<text x="405" y="330" fill="#12202c">webhook-deliver</text>
<text x="405" y="346" fill="#52646f" font-size="10">webhooks.deliver</text>
<rect x="480" y="310" width="110" height="48" rx="6" fill="#fff" stroke="#8a5a00"/>
<text x="535" y="330" fill="#12202c">WORM × N</text>
<text x="535" y="346" fill="#52646f" font-size="10">archive.query</text>
<rect x="605" y="310" width="100" height="48" rx="6" fill="#fff" stroke="#8a5a00"/>
<text x="655" y="330" fill="#12202c">tree-node</text>
<text x="655" y="346" fill="#52646f" font-size="10">kind=tree</text>
<line x1="460" y1="136" x2="460" y2="200" stroke="#1d4f91" stroke-width="1.5"/>
<text x="478" y="168" font-size="9" fill="#1d4f91" text-anchor="start">JetStream</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 3.5 KiB

View file

@ -0,0 +1,22 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 800 280" width="800" height="280" role="img" aria-label="Fleet keepFloor, restart, SSH failover">
<rect width="800" height="280" fill="#f6f3ee"/>
<text x="20" y="28" font-family="Georgia,serif" font-size="16" fill="#12202c">Uptime: floors, restart, another machine</text>
<g font-family="system-ui,sans-serif" font-size="12">
<rect x="20" y="50" width="180" height="70" rx="8" fill="#e3f6e8" stroke="#0f6e56"/>
<text x="110" y="80" text-anchor="middle" fill="#0f6e56">healthy replica</text>
<text x="110" y="100" text-anchor="middle" fill="#52646f" font-size="10">counts toward min</text>
<rect x="220" y="50" width="180" height="70" rx="8" fill="#fff3bf" stroke="#8a5a00"/>
<text x="310" y="80" text-anchor="middle" fill="#8a5a00">paused / slow RTT</text>
<text x="310" y="100" text-anchor="middle" fill="#52646f" font-size="10">not available → spawn</text>
<rect x="420" y="50" width="180" height="70" rx="8" fill="#f9d4d4" stroke="#a32020"/>
<text x="510" y="80" text-anchor="middle" fill="#a32020">/health fail or SSH down</text>
<text x="510" y="100" text-anchor="middle" fill="#52646f" font-size="10">restart or next host</text>
<rect x="620" y="50" width="160" height="70" rx="8" fill="#1d4f91"/>
<text x="700" y="80" text-anchor="middle" fill="#fff">JetStream replay</text>
<text x="700" y="100" text-anchor="middle" fill="#d9e2e8" font-size="10">consumer catch-up</text>
<rect x="20" y="150" width="760" height="100" rx="8" fill="#fff" stroke="#d5dee4"/>
<text x="40" y="178" fill="#12202c">tree-node min=3 keepFloor</text>
<text x="40" y="200" fill="#52646f" font-size="12">local + ns1 (marchon@70.88.205.138, ~/.ssh/id_ed25519). Least-loaded placement. Bloom miss = silence so a partitioned archive does not block the bus.</text>
<text x="40" y="222" fill="#52646f" font-size="12">Zapier only sees HTTPS 202 / wait JSON / REST Hook. Cluster majority of 3 NATS servers survives one node loss.</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 2 KiB

View file

@ -0,0 +1,6 @@
{
"name": "overview",
"version": "1.0.0",
"private": true,
"description": "High-level overview of the Verae Time × Zapier system, modules, uptime, NATS cluster, and docs index"
}

View file

@ -0,0 +1,13 @@
# NATS — verae-nats-process (template)
Connect to the **Verae 3-node JetStream cluster** (private URLs). Zapier never connects.
| Direction | Address | From / to |
|-----------|---------|-----------|
| IN | `verae.example.process.in` | middleware or another worker |
| OUT | `verae.example.process.out` | peers |
| OUT | `verae.example.process.reply.<correlationId>` | requester |
Queue group `example-process` shares work. For fan-out (like archive query) **omit** the queue group.
Fleet: HTTP `/health` on loopback. `keepFloor` in `verae-fleet` if this process must stay up.

View file

@ -0,0 +1,20 @@
# verae-nats-process
Template for a new **addressed messaging process** on the central Verae NATS.IO cluster.
**Repo:** https://git.georgelambert.org/marchon/verae-nats-process
**Clone:** `ssh://git@git.georgelambert.org:2223/marchon/verae-nats-process.git`
**Overview:** https://git.georgelambert.org/marchon/overview
```bash
npm test
npm start # /health on 127.0.0.1:13900
```
1. Duplicate this repository (new Forgejo name).
2. Edit `src/subjects.js``verae.<area>.<resource>.*`.
3. Replace `handle()` in `src/handle.js`.
4. Update `ROUTING.md`; add a fleet service with `min`/`max`.
5. If Zapier needs the result, add HTTPS on **verae-middleware** only.
Zapier cloud must not subscribe. See [overview 06-address-routing](https://git.georgelambert.org/marchon/overview/src/branch/main/06-address-routing.md).

View file

@ -0,0 +1,9 @@
# Address row (copy into overview INDEX and docs-master)
| Address | Kind | Queue group | Publisher | Subscriber | Body |
|---------|------|-------------|-----------|------------|------|
| `verae.example.process.in` | JetStream work | `example-process` | middleware or peer | this worker | `{ correlationId, payload, traceId }` |
| `verae.example.process.out` | events | — | this worker | interested peers | handle() result |
| `verae.example.process.reply.<id>` | reply | — | this worker | original requester | handle() result |
Rename `example` / `process` before production. Do not subscribe from Zapier.

View file

@ -0,0 +1,7 @@
# verae-nats-process
**Job:** Reference worker for a **new NATS address**. Copy this repo to add search, storage, or any unplanned function.
**Expects:** `verae.example.process.in`
**Sends:** `verae.example.process.out` and `reply.<id>`
**Test:** `npm test`

View file

@ -0,0 +1,14 @@
{
"id": "verae-nats-process",
"title": "Template addressed process",
"kind": "nats-worker",
"package": "verae-nats-process",
"role": "verae-nats-process",
"managed": true,
"health": { "type": "http", "path": "/health", "timeoutMs": 800 },
"ports": { "healthBase": 13900 },
"nats": {
"in": ["verae.example.process.in"],
"out": ["verae.example.process.out", "verae.example.process.reply.<correlationId>"]
}
}

View file

@ -0,0 +1,11 @@
{
"name": "verae-nats-process",
"version": "0.1.0",
"type": "module",
"description": "Template JetStream worker for a new verae.* address (copy this repo to expand)",
"scripts": {
"start": "node src/worker.js",
"test": "node --test test/*.test.js"
},
"engines": { "node": ">=20" }
}

View file

@ -0,0 +1,20 @@
/**
* Replace this with search, store, or any new function.
* Must be idempotent: the same correlationId may be redelivered.
*
* @param {object} msg
* @returns {Promise<object>|object}
*/
export function handle(msg = {}) {
const correlationId = msg.correlationId || msg.traceId;
if (!correlationId) {
throw new Error('correlationId or traceId is required');
}
return {
ok: true,
correlationId,
echo: msg.payload ?? msg,
processedAt: new Date().toISOString(),
note: 'Template handler — replace handle() in src/handle.js',
};
}

View file

@ -0,0 +1,13 @@
/**
* Rename `example.process` before production.
* Pattern: verae.<area>.<resource>.<action>
*/
export const AREA = 'example';
export const RESOURCE = 'process';
export const SUBJECTS = Object.freeze({
IN: `verae.${AREA}.${RESOURCE}.in`,
OUT: `verae.${AREA}.${RESOURCE}.out`,
QUEUE: `${AREA}-${RESOURCE}`,
reply: (correlationId) => `verae.${AREA}.${RESOURCE}.reply.${correlationId}`,
});

View file

@ -0,0 +1,70 @@
#!/usr/bin/env node
/**
* Health HTTP + optional NATS subscribe.
* Fleet probes /health. Live NATS is off unless NATS_URL is set.
*/
import http from 'node:http';
import { SUBJECTS } from './subjects.js';
import { handle } from './handle.js';
const port = Number(process.env.FLEET_HEALTH_PORT || process.env.PORT || 13900);
const bind = process.env.FLEET_HEALTH_BIND || '127.0.0.1';
const instance = process.env.FLEET_INSTANCE || 'verae-nats-process-0';
let paused = false;
let processed = 0;
function snapshot() {
return {
ok: !paused,
paused,
instance,
role: 'verae-nats-process',
subjects: { in: SUBJECTS.IN, out: SUBJECTS.OUT, queue: SUBJECTS.QUEUE },
processed,
};
}
const server = http.createServer(async (req, res) => {
const url = new URL(req.url, 'http://127.0.0.1');
const json = (code, obj) => {
res.writeHead(code, { 'content-type': 'application/json' });
res.end(JSON.stringify(obj));
};
if (url.pathname === '/health' && req.method === 'GET') {
const s = snapshot();
return json(s.ok ? 200 : 503, s);
}
if (url.pathname === '/pause' && req.method === 'POST') {
paused = true;
return json(200, snapshot());
}
if (url.pathname === '/resume' && req.method === 'POST') {
paused = false;
return json(200, snapshot());
}
if (url.pathname === '/message' && req.method === 'POST') {
const chunks = [];
for await (const c of req) chunks.push(c);
let body = {};
try {
body = JSON.parse(Buffer.concat(chunks).toString('utf8') || '{}');
} catch {
body = {};
}
if (paused) return json(503, snapshot());
try {
const out = await handle(body);
processed += 1;
return json(200, { rttMs: 0, result: out, replyTo: SUBJECTS.reply(out.correlationId) });
} catch (err) {
return json(400, { error: err.message });
}
}
json(404, { error: 'not found' });
});
server.listen(port, bind, () => {
process.stdout.write(`${instance} health http://${bind}:${port}/health in=${SUBJECTS.IN}\n`);
});
process.on('SIGTERM', () => server.close(() => process.exit(0)));

View file

@ -0,0 +1,21 @@
import { describe, it } from 'node:test';
import assert from 'node:assert/strict';
import { handle } from '../src/handle.js';
import { SUBJECTS } from '../src/subjects.js';
describe('verae-nats-process template', () => {
it('uses the verae.area.resource.action address pattern', () => {
assert.equal(SUBJECTS.IN, 'verae.example.process.in');
assert.equal(SUBJECTS.OUT, 'verae.example.process.out');
assert.match(SUBJECTS.reply('abc'), /^verae\.example\.process\.reply\.abc$/);
});
it('handle requires correlation and is idempotent-shaped', () => {
assert.throws(() => handle({}), /correlationId/);
const a = handle({ correlationId: 'c1', payload: { q: 1 } });
const b = handle({ correlationId: 'c1', payload: { q: 1 } });
assert.equal(a.ok, true);
assert.equal(a.correlationId, b.correlationId);
assert.equal(a.echo.q, 1);
});
});

View file

@ -33,6 +33,8 @@ SECTIONS = [
("docs/02-architecture/modules-and-nats.md", "Module catalog and NATS addresses"),
("docs/02-architecture/tree-nodes.md", "Tree nodes and bulk Merkle summaries"),
("docs/02-architecture/fleet.md", "Fleet replica floors and monitor"),
("packages/overview/README.md", "System overview (TOC)"),
("packages/overview/INDEX.md", "Documentation index"),
],
),
(
@ -141,6 +143,8 @@ def main() -> None:
"verae-zapier-simulator",
"zapier-user-docs",
"verae-fleet",
"overview",
"verae-nats-process",
"docs-master",
):
readme = ROOT / "packages" / pkg / "README.md"
@ -157,6 +161,7 @@ def main() -> None:
copy_tree(src, SITE / src.name if src.parent == ROOT else SITE / "docs" / "OPEN.md")
copy_tree(ROOT / "docs" / "WORK-LOG.md", SITE / "docs" / "WORK-LOG.md")
copy_tree(ROOT / "packages" / "docs-master", SITE / "docs-master")
copy_tree(ROOT / "packages" / "overview", SITE / "overview")
copy_tree(ROOT / "packages" / "zapier-user-docs", SITE / "user-docs", ignore=shutil.ignore_patterns("node_modules"))
sim_pub = ROOT / "packages" / "verae-zapier-simulator" / "public" / "index.html"
if sim_pub.exists():
@ -235,6 +240,8 @@ def main() -> None:
("verae-zapier-simulator", "Zapier interface + trace console"),
("zapier-user-docs", "Signup-to-usage user guide"),
("verae-fleet", "Service catalog, replica floors, monitor"),
("overview", "High-level system overview"),
("verae-nats-process", "Template for a new NATS address"),
]
git_lis = "".join(
f'<li><a href="https://git.georgelambert.org/marchon/{name}">{name}</a> '
@ -292,6 +299,9 @@ def main() -> None:
<li><a href="docs-master/modules/">Per-module SUMMARY.md and NATS.md</a></li>
<li><a href="user-docs/README.md">User guide (signup tree-node lookup)</a></li>
<li><a href="packages/verae-zapier-simulator/README.md">Simulator README</a></li>
<li><a href="overview/README.md">System overview</a></li>
<li><a href="overview/INDEX.md">Documentation index</a></li>
<li><a href="overview/diagrams/system.svg">System diagram (SVG)</a></li>
</ul>
</section>
<section>

View file

@ -47,6 +47,8 @@ create verae-tree-node "Merkle leaf proofs for hashes only sealed as a bulk summ
create verae-zapier-simulator "Zapier interface simulator with hop-by-hop trace console"
create zapier-user-docs "User docs from signup through hash register, chain lookup, tree-node query"
create verae-fleet "Service catalog, replica floors, monitor, restart, pause"
create overview "High-level system overview, NATS cluster, uptime, docs index"
create verae-nats-process "Template JetStream worker for a new verae.* address"
push_dir "$ROOT/packages/zappier" zappier-edge
push_dir "$ROOT/packages/verae-zapier-middleware" verae-middleware
@ -60,5 +62,7 @@ push_dir "$ROOT/packages/verae-tree-node" verae-tree-node
push_dir "$ROOT/packages/verae-zapier-simulator" verae-zapier-simulator
push_dir "$ROOT/packages/zapier-user-docs" zapier-user-docs
push_dir "$ROOT/packages/verae-fleet" verae-fleet
push_dir "$ROOT/packages/overview" overview
push_dir "$ROOT/packages/verae-nats-process" verae-nats-process
echo ALL_MODULE_REPOS_PUSHED