marchon/overview
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Initial import of overview from zapier monorepo

Browse source
This commit is contained in:
George Lambert 2026-09-11 14:37:18 -04:00
commit b64e3bf072
21 changed files with 611 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

26
01-system.md Normal file
View file

@ -0,0 +1,26 @@
# 1. System
Verae Time proves a SHA-256 existed at a given time. Zapier lets customers register and look up hashes from the tools they already use. Billing and API keys live on **zappier-edge**. Timestamping, wait, webhooks, and archive fan-out live behind **verae-middleware**. Durable messaging is the **central Verae NATS.IO 3-server JetStream cluster**.
## Hard rule
**Zapier never connects to NATS, tree nodes, WORM archives, or `api.veraetime.net`.** Those hops are middleware and workers only. If a trace ever shows a Zapier hop on a `verae.*` subject, do not push the app.
## Planes
| Plane | What | Network |
|-------|------|---------|
| Zapier cloud | `verae-zapier-app`, `verae-activate` | HTTPS to zappier-edge |
| Commercial edge | `zappier-edge` portal, admin, `x-api-key`, Stripe meter | Public HTTPS |
| Middleware HTTP | `/zapier/v1/*` job id + wait | Public HTTPS from edge only |
| NATS cluster | JetStream subjects under `verae.*` | Private; loopback or SSH tunnel |
| Workers | poller, webhook-deliver, aggregator | NATS + HTTPS to chain or Zapier hooks |
| Archives | WORM + tree nodes | NATS broadcast query; bloom miss = silence |
| Chain | Verae timestamping | HTTPS `api.veraetime.net` or MOCK |
| Control | `verae-fleet` | Operator loopback `:3850`; SSH to extra machines |
## Request in one sentence
A Zap step POSTs to zappier-edge; edge meters and forwards to middleware; middleware splits hash vs files, writes the hash (or Merkle root) to the chain, publishes `verae.zapier.jobs.watch`, and either returns `jobId` or waits on `verae.zapier.jobs.events`. Attached metadata and bulk-summary **leaves** are found later by broadcasting `verae.archive.query` to every archive/tree node.
See [diagrams](08-diagrams.md).

26
02-modules-and-repos.md Normal file
View file

@ -0,0 +1,26 @@
# 2. Modules and independent repositories
Each runtime piece is its **own git repo** on Forgejo (`git.georgelambert.org`, SSH port 2223). The monorepo `master-zapier-plan-draft` is a snapshot of the workspace; do not treat it as the only clone path.
| Independent repo | Package / path | What it does |
|------------------|----------------|--------------|
| **overview** | `packages/overview` | This high-level map |
| **verae-nats-process** | `packages/verae-nats-process` | **Template** for a new addressed NATS process |
| **master-zapier-plan-draft** | workspace root | Combined snapshot (`main` and `master`) |
| **zappier-edge** | `packages/zappier` | Metered HTTPS, portal, admin, Stripe `zappier.api_cents` |
| **verae-middleware** | `packages/verae-zapier-middleware` | `/zapier/v1`, job wait, NATS publishers/workers |
| **verae-zapier-app** | `packages/verae-zapier` | Full Zapier nouns (timestamp, wait, batch, hash + tree lookup) |
| **verae-activate** | `packages/verae-activate` | Activate-now (Add Numbers, Echo, SHA256, mock timestamp) |
| **verae-request-splitter** | `packages/verae-request-splitter` | Chain hash vs `archive.put` |
| **verae-archive-worm** | `packages/verae-archive-worm` | Bloom WORM node |
| **verae-archive-aggregator** | `packages/verae-archive-aggregator` | Merge archive replies |
| **verae-tree-node** | `packages/verae-tree-node` | Merkle leaf proofs (bulk summaries) |
| **verae-fleet** | `packages/verae-fleet` | Catalog, min copies, pause/restart, SSH hosts, operator console |
| **verae-zapier-simulator** | `packages/verae-zapier-simulator` | Trace console before `zapier-platform push` |
| **zapier-user-docs** | `packages/zapier-user-docs` | Customer signup → register → lookup |
| **zapier-docs-master** | `packages/docs-master` | Per-module `SUMMARY.md` + `NATS.md` |
| **verae-ops** | `packages/verae-ops` | Docker, Proxmox, VMs, dedicated hardware, linking services |
Libraries that are **not** separate Forgejo apps today: `verae-chain-client` (inside middleware), job-poller and webhook-deliver (middleware workers, fleet-spawned).
Per-module NATS contracts: [zapier-docs-master](https://git.georgelambert.org/marchon/zapier-docs-master).

37
03-nats-cluster.md Normal file
View file

@ -0,0 +1,37 @@
# 3. Central Verae NATS.IO 3-server cluster
All durable messaging for this product is **NATS JetStream**, not Zapier queues and not a public TCP API.
## Cluster
Three `nats-server -js` nodes form the Verae cluster (routes between them, JetStream replication). Clients (middleware, poller, aggregator, WORM, tree nodes, and any new process cloned from `verae-nats-process`) connect with a **cluster URL list**, for example:
```text
nats://127.0.0.1:4222,nats://127.0.0.1:4223,nats://127.0.0.1:4224
```
On a given machine the listener stays on **loopback** (or a private interface). Operators reach it with `scripts/nats-tunnel.sh` / `ssh -L 14222:127.0.0.1:4222`. **Do not bind 4222 on `0.0.0.0` without auth.**
Today’s NS1 box (`NS1.GEORGELAMBERT.ORG`, `70.88.205.138`) already runs JetStream on `127.0.0.1:4222`. The **target** is three clustered nodes so losing one server does not lose the stream. Fleet SSH hosts (`ns1`, later `lan-134`) run **workers**, not extra public NATS listeners.
![NATS cluster](diagrams/nats-cluster.svg)
## Who may connect
| Allowed | Forbidden |
|---------|-----------|
| verae-middleware, fleet workers, WORM, tree nodes, `verae-nats-process` clones | Zapier cloud, browsers, zappier-edge |
## Address families already in use
| Address | Kind | Notes |
|---------|------|--------|
| `verae.zapier.jobs.watch` | work queue | poller |
| `verae.zapier.jobs.events` | events | wait + webhooks |
| `verae.zapier.webhooks.deliver` | work queue | HTTPS to Zapier REST Hook |
| `verae.zapier.usage` | optional | metering |
| `verae.archive.put` | JetStream | splitter / merkle builder |
| `verae.archive.query` | **broadcast** (no queue group) | every WORM and tree node |
| `verae.archive.reply.<correlationId>` | replies | **only on bloom hit** |
New functions get new `verae.<area>.<resource>.<action>` addresses — see [06-address-routing.md](06-address-routing.md).

34
04-uptime.md Normal file
View file

@ -0,0 +1,34 @@
# 4. How the system maintains uptime
Uptime is **NATS durability + fleet replica floors + more than one machine**, not a single always-on Zapier connection.
## Replica floors (`verae-fleet`)
`packages/verae-fleet/fleet.json` sets `min` / `max` / `keepFloor` per service. **Available** means running, healthy, and not paused.
| Service | Default min | keepFloor |
|---------|-------------|-----------|
| tree-node | 3 | yes |
| archive-worm | 3 | yes |
| archive-aggregator, job-poller, webhook-deliver | 1 | yes |
| zappier-edge, middleware-http | 1 | yes |
If a tree node is paused, crashes, or fails `/health`, fleet **starts another copy** until three are available. Operator console: http://127.0.0.1:3850/ (Fleet tab — green / yellow / red; Trace tab for hop tests; Docs tab for reading order).
## Restart and pause
- Unhealthy `/health` → same instance id restarted.
- Pause does not count toward `min`.
- `stop` on a service disables keepFloor for that service.
## Spread across machines
`machines.json` lists hosts (`local`, `ns1` = `marchon@70.88.205.138` with `~/.ssh/id_ed25519`, optional `lan-134`). New replicas go to the **least-loaded** eligible host. Remote spawn/health/kill is SSH; workers bind loopback on the remote box.
## JetStream
Work queues (`jobs.watch`, `webhooks.deliver`) replay if a consumer dies. Event stream (`jobs.events`) lets waiters and webhook routers catch up. A 3-node cluster keeps the stream if one NATS server is down.
## What Zapier sees
HTTPS 202 `jobId`, wait JSON, or REST Hook. Timeouts return `pending` + `jobId` so the **Timestamp Completed** trigger can finish the job. Zapier retries are safe: the same SHA-256 returns the original seal (`existing: true`).

23
05-network-failures.md Normal file
View file

@ -0,0 +1,23 @@
# 5. Local network failures
“Local network” means the operator LAN, SSH to NS1, or a partitioned archive — not Zapier’s cloud.
| Failure | What happens | What the customer sees |
|---------|--------------|------------------------|
| NATS node unreachable | Client reconnects to another cluster URL; JetStream consumers resume | Wait may return `pending`; hook still fires later |
| All NATS down | Middleware cannot publish `jobs.watch`; fleet marks workers unhealthy | 503 / pending; no NATS leak to Zapier |
| One WORM / tree node partitioned | Bloom miss = **no packet**; aggregator uses whoever answered | Lookup may miss attachments until the node returns; seal on chain still valid |
| SSH to a fleet host fails | Spawn fails; fleet **places the next replica on another machine** | Floor still met if capacity remains on `local` or another SSH host |
| Chain `api.veraetime.net` timeout | Poller retries; then `timestamp.timeout` event | Wait → pending or failed; async + hook still the recovery path |
| zappier-edge 402 | QuotaExceeded with upgrade URL | Zap step error; no NATS involved |
| Tunnel to loopback NATS dropped | `NATS_URL=nats://127.0.0.1:14222` dies; restart `nats-tunnel.sh` | Workers on NS1 itself still see `127.0.0.1:4222` |
## Design choices that make partitions survivable
1. **Archive query is broadcast**, not a shared queue group — a dead node does not steal the message.
2. **Bloom miss is silence** — missing nodes do not send empty errors that look like “hash unknown”.
3. **Zero replies + known puts** is an outage, not a miss (simulator / fleet monitors flag this).
4. **NATS is not on the public NIC** — a WAN blip does not expose 4222.
5. **Idempotent seals** — retrying a Zap after a network error will not double-timestamp.
See fleet RTT (min / avg / p50 / p90) when planning extra tree nodes after a flaky path.

33
06-address-routing.md Normal file
View file

@ -0,0 +1,33 @@
# 6. Address routing (including unplanned functions)
NATS **addresses** (subjects) are the extension point. A new search, store, or job type is a new address plus a process that listens — not a new Zapier TCP client.
## Pattern
```text
verae.<area>.<resource>.<action>
verae.<area>.<resource>.reply.<correlationId>
```
| Piece | Example | Meaning |
|-------|---------|---------|
| `verae` | — | Verae bus (not Zapier) |
| `area` | `zapier`, `archive`, `search`, `store` | Product slice |
| `resource` | `jobs`, `hashes`, `blobs` | Noun |
| `action` | `watch`, `query`, `put`, `in` | Verb |
| `reply.<id>` | — | Correlated response |
**Queue group** (work sharing): `area-resource-action` (e.g. `job-poller`).
**No queue group** (fan-out): archive/tree **query** so every node sees every lookup.
## Adding something that does not exist yet
1. Copy the independent repo **[verae-nats-process](https://git.georgelambert.org/marchon/verae-nats-process)** (`packages/verae-nats-process`).
2. Rename `verae.example.process.in` / `.out` / `.reply.*` in `src/subjects.js`.
3. Add a row to that repo’s `ROUTING.md` and to [INDEX.md](INDEX.md).
4. Register the process in `verae-fleet` (`min`/`max`, machines, roles).
5. If Zapier must call it, add **one HTTPS route** on middleware — Zapier still never sees NATS.
Do **not** invent a public NATS URL for Zapier. Do **not** reuse `verae.archive.query` as a queue group.
![Address expansion](diagrams/routing.svg)

15
07-external-resources.md Normal file
View file

@ -0,0 +1,15 @@
# 7. External resources: search, storage, and the chain
| Resource | Where | How a Zap reaches it |
|----------|--------|----------------------|
| **Central chain** (itemized SHA-256 + time + certificate) | `api.veraetime.net` or MOCK | HTTPS via middleware: create / wait / `GET /hashes/{sha256}` |
| **Bulk Merkle root** | Same chain, one seal | `POST /timestamp/batch` |
| **Leaf proofs** (hash only in a bulk summary) | Tree-node WORM on NATS | `GET /hashes/{sha}?includeTree=true` → `verae.archive.query` kinds=`tree` |
| **Public / private metadata, files** | WORM archives | `includeAttached`; never on chain |
| **Search (central)** | Chain lookup | Zapier search **Find Timestamp by SHA256** |
| **Search (external tree / extra stores)** | NATS query to every node that might hold the key | Zapier search **Find Hash (tree nodes + central)** |
| **Future search / store** | New `verae.search.*` or `verae.store.*` process | Copy `verae-nats-process`; optional middleware GET |
Storage that is **not** the blockchain stays on WORM/tree nodes (and later any process that answers `verae.archive.query` or a new store address). The chain stores hash + time + certificate (+ Merkle root for batches).
External “unplanned” storage or search is the same pattern: new address, bloom or index on that node, silence on miss, aggregator or the template’s reply subject.

59
08-diagrams.md Normal file
View file

@ -0,0 +1,59 @@
# 8. Architectural diagrams
SVG files in [`diagrams/`](diagrams/). The same shapes are repeated below in mermaid for Forgejo preview.
## 8.1 End-to-end (HTTPS vs NATS)
![System](diagrams/system.svg)
```mermaid
flowchart LR
Z[Zapier cloud apps] -->|HTTPS x-api-key| E[zappier-edge]
E -->|HTTPS| M[verae-middleware]
M -->|HTTPS| C[Verae chain]
M -->|JetStream| N[NATS 3-node cluster]
N --> P[job-poller]
N --> W[webhook-deliver]
N --> A[archive-aggregator]
N --> R[WORM x N]
N --> T[tree-node x N]
P -->|HTTPS status| C
W -->|HTTPS REST Hook| Z
```
## 8.2 NATS.IO 3-server cluster
![Cluster](diagrams/nats-cluster.svg)
```mermaid
flowchart TB
subgraph cluster [Verae NATS.IO JetStream]
N1[nats-server A :4222 loopback]
N2[nats-server B]
N3[nats-server C]
N1 <--> N2
N2 <--> N3
N3 <--> N1
end
MW[middleware + workers] -->|cluster URL list| cluster
F[fleet SSH hosts] -->|workers only| cluster
```
## 8.3 Uptime and failure
![Uptime](diagrams/uptime.svg)
```mermaid
flowchart TD
H[health /health] -->|fail| R[restart instance]
P[pause] -->|not available| F[keepFloor spawn]
S[SSH host down] -->|spawn failed| O[place on next machine]
Q[archive.query broadcast] -->|bloom miss| SIL[silence]
Q -->|hit| REP[archive.reply.id]
```
## 8.4 New address = new process
![Routing](diagrams/routing.svg)
See [09-expansion-template.md](09-expansion-template.md).

29
09-expansion-template.md Normal file
View file

@ -0,0 +1,29 @@
# 9. Model repo for a new addressed process
**Repository:** [verae-nats-process](https://git.georgelambert.org/marchon/verae-nats-process)
**Path in the monorepo:** `packages/verae-nats-process`
**Clone:** `ssh://git@git.georgelambert.org:2223/marchon/verae-nats-process.git`
This is the **reference implementation** for expansions: a small JetStream worker with:
- a single **in** address, an **out** address, and **reply.&lt;correlationId&gt;**
- `handle(msg)` you replace with real work (search, store, transform)
- HTTP `/health` so `verae-fleet` can keep a replica floor
- tests that do not need a live cluster
- `ROUTING.md` — the row you copy into the global address table
Default subjects (rename before production):
| Direction | Address |
|-----------|---------|
| IN | `verae.example.process.in` |
| OUT | `verae.example.process.out` |
| REPLY | `verae.example.process.reply.<correlationId>` |
```bash
cd packages/verae-nats-process
npm test
# then rename example → your area, add fleet.json min, clone as a new Forgejo repo
```
Zapier still must not subscribe. If a Zap needs the result, middleware exposes HTTPS and publishes to the new **in** address.

89
INDEX.md Normal file
View file

@ -0,0 +1,89 @@
# Documentation index
Read **high-level first**, then **architecture**, then the rest. Live catalog (PDF by default): https://zapier.georgelambert.org/ — [Markdown indexes](https://zapier.georgelambert.org/index-md.html)
Operator console (loopback): http://127.0.0.1:3850/ · [CONSOLE.pdf](https://zapier.georgelambert.org/packages/verae-fleet/docs/CONSOLE.pdf)
## 1. High-level (non-technical)
| Document | What it is |
|----------|------------|
| [README.md](README.md) | Overview (start here) |
| [01-system.md](01-system.md) | What runs where, in plain language |
| [02-modules-and-repos.md](02-modules-and-repos.md) | Short names of every independent repo |
| [zapier-docs-master](https://git.georgelambert.org/marchon/zapier-docs-master) | One-line SUMMARY + NATS per module |
| [zapier-user-docs](https://zapier.georgelambert.org/user-docs/README.pdf) | Signup → register → lookup |
| [verae-ops](https://zapier.georgelambert.org/packages/verae-ops/README.pdf) | Docker / Proxmox / VM / metal install |
## 2. Architecture
| Document | URL |
|----------|-----|
| NATS cluster | [03-nats-cluster.md](03-nats-cluster.md) |
| Uptime | [04-uptime.md](04-uptime.md) |
| Network failures | [05-network-failures.md](05-network-failures.md) |
| Address routing | [06-address-routing.md](06-address-routing.md) |
| External resources | [07-external-resources.md](07-external-resources.md) |
| Diagrams | [08-diagrams.md](08-diagrams.md) |
| Catalog home | https://zapier.georgelambert.org/ (PDF) · [Markdown indexes](https://zapier.georgelambert.org/index-md.html) |
| Composition | https://zapier.georgelambert.org/docs/02-architecture/composition.pdf |
| NATS gateway | https://zapier.georgelambert.org/docs/02-architecture/nats-gateway.pdf |
| NATS subjects | https://zapier.georgelambert.org/docs/02-architecture/nats-subjects.pdf |
| Archive / bloom | https://zapier.georgelambert.org/docs/02-architecture/archive-nats.pdf |
| Module NATS map | https://zapier.georgelambert.org/docs/02-architecture/modules-and-nats.pdf |
| Tree nodes | https://zapier.georgelambert.org/docs/02-architecture/tree-nodes.pdf |
| Fleet | https://zapier.georgelambert.org/docs/02-architecture/fleet.pdf |
## 3. Remaining documentation
| Document | URL |
|----------|-----|
| Expansion template | [09-expansion-template.md](09-expansion-template.md) |
| Operator console | https://zapier.georgelambert.org/packages/verae-fleet/docs/CONSOLE.pdf |
| User guide | https://zapier.georgelambert.org/user-docs/README.pdf |
| Tree-node lookup (users) | https://zapier.georgelambert.org/packages/zapier-user-docs/09-lookup-tree-nodes.pdf |
| Zapier developer setup | https://zapier.georgelambert.org/docs/04-activate/SETUP-ZAPIER-DEVELOPER.pdf |
| docs-master | https://zapier.georgelambert.org/docs-master/README.pdf |
| Sphinx HTML | https://zapier.georgelambert.org/sphinx/ |
| Sphinx LaTeX PDF | https://zapier.georgelambert.org/sphinx/verae-zapier-modules.pdf |
| Install (verae-ops) | https://zapier.georgelambert.org/packages/verae-ops/README.pdf |
## Independent git repositories
Prefix: `https://git.georgelambert.org/marchon/`
overview · verae-nats-process · verae-ops · master-zapier-plan-draft · zappier-edge · verae-middleware · verae-zapier-app · verae-activate · verae-request-splitter · verae-archive-worm · verae-archive-aggregator · verae-tree-node · verae-fleet · verae-zapier-simulator · zapier-user-docs · zapier-docs-master
## Per-module contracts
In **zapier-docs-master**: `modules/<package>/SUMMARY.md` and `NATS.md` for every runtime package.
## Operator surfaces (not public NATS)
| Surface | URL |
|---------|-----|
| Operator console (Fleet · Trace · Docs) | http://127.0.0.1:3850/ |
| Standalone simulator | http://127.0.0.1:3847/ |
## A–Z subject index
| Term | See |
|------|-----|
| aggregator | verae-archive-aggregator; archive.query / reply |
| bloom miss | silence; 05-network-failures |
| batch / Merkle | verae-tree-node; user-docs 08–09 |
| cluster | 03-nats-cluster |
| fleet / keepFloor | verae-fleet; 04-uptime |
| hash lookup | verae-zapier-app searches; 07-external-resources |
| jobs.events / jobs.watch | nats-subjects.html; middleware |
| includeTree | tree lookup; archive.query kinds=tree |
| routing | 06-address-routing; verae-nats-process |
| splitter | verae-request-splitter |
| SSH hosts | verae-fleet machines.json |
| Zapier never NATS | 01-system |
| zappier-edge | metering, portal |
| operator console | http://127.0.0.1:3850/ ; verae-fleet docs/CONSOLE.md |
---
Design: **Scott Lindsey**, **George Lambert**, **NATS.IO**, and **Grok-Code** by [Grok.com](https://grok.com).

7
NATS.md Normal file
View file

@ -0,0 +1,7 @@
# NATS — overview
This documentation repo does **not** subscribe. It describes the **central Verae NATS.IO 3-server JetStream cluster** that every worker talks to.
Zapier cloud never connects here. Clients use private URLs (`nats://127.0.0.1:4222` on a node, or an SSH tunnel). Do not bind 4222 on `0.0.0.0` without auth.
New functions are new **addresses** (subjects), not new public TCP ports. Copy `verae-nats-process` to add one.

105
README.md Normal file
View file

@ -0,0 +1,105 @@
# Verae Time × Zapier — system overview
High-level description of the whole system: what it is, which **independent git repositories** implement it, how it stays up, how it behaves when the local network fails, and how it talks to the **central Verae NATS.IO 3-server cluster**. New work is added by **new address routing**, not by teaching Zapier about NATS.
**This repo:** https://git.georgelambert.org/marchon/overview
**Live catalog (PDF by default):** https://zapier.georgelambert.org/ · [Markdown indexes](https://zapier.georgelambert.org/index-md.html)
**Clone:** `ssh://git@git.georgelambert.org:2223/marchon/overview.git`
## Table of contents
1. [System](01-system.md) — what runs where; Zapier never speaks NATS
2. [Modules and repositories](02-modules-and-repos.md) — every independent Forgejo repo
3. [NATS.IO 3-server cluster](03-nats-cluster.md) — JetStream, subjects, who may connect
4. [Uptime](04-uptime.md) — replica floors, restart, SSH spread
5. [Local network failures](05-network-failures.md) — reconnect, silence, pending, failover
6. [Address routing](06-address-routing.md) — how to add unplanned functions
7. [External resources](07-external-resources.md) — searches, storage, chain
8. [Architectural diagrams](08-diagrams.md) — SVG + mermaid
9. [Expansion template](09-expansion-template.md) — `verae-nats-process`
10. [Documentation index](INDEX.md) — pointer into all other docs
## One-screen picture
![System context](diagrams/system.svg)
Zapier → HTTPS `zappier-edge` → HTTPS `verae-middleware` → **NATS cluster** → workers, WORM archives, tree nodes. Chain HTTPS is `api.veraetime.net` (or MOCK). Fleet keeps minimum copies, including tree nodes.
## Independent repositories (Forgejo)
| Repo | Role |
|------|------|
| [overview](https://git.georgelambert.org/marchon/overview) | This document |
| [verae-ops](https://git.georgelambert.org/marchon/verae-ops) | Docker, Proxmox, VMs, dedicated hardware |
| [verae-nats-process](https://git.georgelambert.org/marchon/verae-nats-process) | **Model** for a new addressed process |
| [master-zapier-plan-draft](https://git.georgelambert.org/marchon/master-zapier-plan-draft) | Monorepo snapshot |
| [zappier-edge](https://git.georgelambert.org/marchon/zappier-edge) | Metered public HTTPS |
| [verae-middleware](https://git.georgelambert.org/marchon/verae-middleware) | Zapier HTTP + NATS workers |
| [verae-zapier-app](https://git.georgelambert.org/marchon/verae-zapier-app) | Zapier Platform app |
| [verae-activate](https://git.georgelambert.org/marchon/verae-activate) | Activate-now app |
| [verae-request-splitter](https://git.georgelambert.org/marchon/verae-request-splitter) | Hash vs attachments |
| [verae-archive-worm](https://git.georgelambert.org/marchon/verae-archive-worm) | Bloom WORM node |
| [verae-archive-aggregator](https://git.georgelambert.org/marchon/verae-archive-aggregator) | Archive reply merge |
| [verae-tree-node](https://git.georgelambert.org/marchon/verae-tree-node) | Merkle leaf proofs |
| [verae-fleet](https://git.georgelambert.org/marchon/verae-fleet) | Replica floor + SSH hosts |
| [verae-zapier-simulator](https://git.georgelambert.org/marchon/verae-zapier-simulator) | Trace console |
| [zapier-user-docs](https://git.georgelambert.org/marchon/zapier-user-docs) | Signup → lookup |
| [zapier-docs-master](https://git.georgelambert.org/marchon/zapier-docs-master) | Per-module SUMMARY + NATS |
Clone any of them: `git clone ssh://git@git.georgelambert.org:2223/marchon/<name>.git` (SSH port **2223**).
## Operator console (loopback)
Fleet, message **Trace**, and **Docs** share one shell at http://127.0.0.1:3850/ (`verae-fleet`). Not public. See [CONSOLE.pdf](https://zapier.georgelambert.org/packages/verae-fleet/docs/CONSOLE.pdf).
![Fleet tab](screenshots/console-fleet.png)
![Trace tab](screenshots/console-trace.png)
![Docs tab](screenshots/console-docs.png)
## Where to read next
Start at the top. Architecture is second. Everything else is reference.
### 1. High-level (non-technical)
| Document | What it is |
|----------|------------|
| **This overview** | Whole system in one place |
| [Modules and repositories](02-modules-and-repos.md) | Short names of every independent repo |
| [zapier-docs-master](https://git.georgelambert.org/marchon/zapier-docs-master) | One-line SUMMARY + NATS per module |
| [zapier-user-docs](https://git.georgelambert.org/marchon/zapier-user-docs) | Signup → register a hash → lookup |
### 2. Architecture
| Document | What it is |
|----------|------------|
| [NATS 3-server cluster](03-nats-cluster.md) | Who may connect; private URLs |
| [Uptime](04-uptime.md) | Replica floors, restart, SSH hosts |
| [Local network failures](05-network-failures.md) | Reconnect, bloom silence, failover |
| [Address routing](06-address-routing.md) | Unplanned functions as new subjects |
| [External resources](07-external-resources.md) | Chain, WORM, tree-node search |
| [Diagrams](08-diagrams.md) | SVG + mermaid |
| [Fleet](https://zapier.georgelambert.org/docs/02-architecture/fleet.pdf) | Operator replica control |
| [Module NATS map](https://zapier.georgelambert.org/docs/02-architecture/modules-and-nats.pdf) | Address table |
| [Archive / bloom](https://zapier.georgelambert.org/docs/02-architecture/archive-nats.pdf) | Multipart and multi-receipt |
| [Tree nodes](https://zapier.georgelambert.org/docs/02-architecture/tree-nodes.pdf) | Bulk Merkle leaves |
| [Composition](https://zapier.georgelambert.org/docs/02-architecture/composition.pdf) | zappier + middleware HTTPS |
### 3. Remaining documentation
| Document | What it is |
|----------|------------|
| [Documentation index](INDEX.md) | A–Z and catalog URLs |
| [Expansion template](09-expansion-template.md) / [verae-nats-process](https://git.georgelambert.org/marchon/verae-nats-process) | Copy this to add an address |
| [Operator console](https://zapier.georgelambert.org/packages/verae-fleet/docs/CONSOLE.pdf) | Screenshots of Fleet / Trace / Docs |
| [Simulator](https://git.georgelambert.org/marchon/verae-zapier-simulator) | Standalone trace (also inside the console) |
| [Zapier developer setup](https://zapier.georgelambert.org/docs/04-activate/SETUP-ZAPIER-DEVELOPER.pdf) | CLI / login / push |
| Live catalog | https://zapier.georgelambert.org/ (PDF) · [Markdown indexes](https://zapier.georgelambert.org/index-md.html) |
---
## Credits
Design: **Scott Lindsey**, **George Lambert**, **NATS.IO**, and **Grok-Code** by [Grok.com](https://grok.com).

7
SUMMARY.md Normal file
View file

@ -0,0 +1,7 @@
# overview
**Job:** High-level map of the whole Verae Time × Zapier system: modules, independent git repos, uptime, local-network failure, the central 3-node NATS.IO cluster, and how to add a new addressed process.
**Expects / sends:** none (documentation).
**Repos named:** see [02-modules-and-repos.md](02-modules-and-repos.md). Expansion template: `verae-nats-process`.

21
diagrams/nats-cluster.svg Normal file
View file

@ -0,0 +1,21 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 720 300" width="720" height="300" role="img" aria-label="Three nats-server nodes clustered, clients on private URLs">
<rect width="720" height="300" fill="#f6f3ee"/>
<text x="20" y="28" font-family="Georgia,serif" font-size="16" fill="#12202c">Verae NATS.IO — three JetStream servers</text>
<g font-family="system-ui,sans-serif" font-size="12" text-anchor="middle">
<circle cx="160" cy="150" r="54" fill="#1d4f91"/>
<text x="160" y="146" fill="#fff">A</text>
<text x="160" y="164" fill="#d9e2e8" font-size="10">:4222 loopback</text>
<circle cx="360" cy="150" r="54" fill="#1d4f91"/>
<text x="360" y="146" fill="#fff">B</text>
<text x="360" y="164" fill="#d9e2e8" font-size="10">route</text>
<circle cx="560" cy="150" r="54" fill="#1d4f91"/>
<text x="560" y="146" fill="#fff">C</text>
<text x="560" y="164" fill="#d9e2e8" font-size="10">JetStream</text>
<line x1="214" y1="150" x2="306" y2="150" stroke="#12202c" stroke-width="2"/>
<line x1="414" y1="150" x2="506" y2="150" stroke="#12202c" stroke-width="2"/>
<path d="M160 96 C 360 20 360 20 560 96" fill="none" stroke="#12202c" stroke-width="2"/>
<rect x="40" y="230" width="200" height="44" rx="8" fill="#12202c"/>
<text x="140" y="256" fill="#f6f3ee">workers + middleware</text>
<text x="250" y="256" fill="#52646f" font-size="11" text-anchor="start">→ nats://A,B,C (never Zapier)</text>
</g>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 1.4 KiB

25
diagrams/routing.svg Normal file
View file

@ -0,0 +1,25 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 840 260" width="840" height="260" role="img" aria-label="Copy verae-nats-process to add a new subject">
<rect width="840" height="260" fill="#f6f3ee"/>
<text x="20" y="28" font-family="Georgia,serif" font-size="16" fill="#12202c">New function = new address + clone of verae-nats-process</text>
<g font-family="system-ui,sans-serif" font-size="12" text-anchor="middle">
<rect x="20" y="60" width="160" height="64" rx="8" fill="#12202c"/>
<text x="100" y="88" fill="#f6f3ee">copy template repo</text>
<text x="100" y="106" fill="#c5d0d8" font-size="10">verae-nats-process</text>
<rect x="220" y="60" width="180" height="64" rx="8" fill="#0b4f8a"/>
<text x="310" y="88" fill="#fff">rename subjects</text>
<text x="310" y="106" fill="#d9e2e8" font-size="10">verae.area.resource.action</text>
<rect x="440" y="60" width="160" height="64" rx="8" fill="#0f6e56"/>
<text x="520" y="88" fill="#fff">fleet min/max</text>
<text x="520" y="106" fill="#d4efe6" font-size="10">keepFloor + machines</text>
<rect x="640" y="60" width="180" height="64" rx="8" fill="#8a5a00"/>
<text x="730" y="88" fill="#fff">optional HTTPS</text>
<text x="730" y="106" fill="#fff3bf" font-size="10">middleware only → Zapier</text>
<polygon points="180,92 220,92 220,88 232,92 220,96 220,92" fill="#52646f"/>
<polygon points="400,92 440,92 440,88 452,92 440,96 440,92" fill="#52646f"/>
<polygon points="600,92 640,92 640,88 652,92 640,96 640,92" fill="#52646f"/>
<rect x="20" y="150" width="800" height="84" rx="8" fill="#fff" stroke="#d5dee4"/>
<text x="420" y="178" fill="#12202c">Existing: verae.zapier.jobs.* · verae.archive.put|query|reply.*</text>
<text x="420" y="200" fill="#52646f" font-size="12">Unplanned search/store: verae.search.* or verae.store.* — same worker shape. Zapier never subscribes.</text>
<text x="420" y="218" fill="#52646f" font-size="12">Broadcast queries: no queue group. Work queues: queue group. Bloom miss: no packet.</text>
</g>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 2 KiB

47
diagrams/system.svg Normal file
View file

@ -0,0 +1,47 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 920 420" width="920" height="420" role="img" aria-label="System context: Zapier HTTPS to edge to middleware to NATS cluster">
<rect width="920" height="420" fill="#f6f3ee"/>
<text x="24" y="28" font-family="Georgia,serif" font-size="16" fill="#12202c">Verae Time × Zapier — system context</text>
<text x="24" y="48" font-family="system-ui,sans-serif" font-size="11" fill="#52646f">Zapier never speaks NATS. HTTPS stops at middleware. JetStream is the 3-node cluster.</text>
<g font-family="system-ui,sans-serif" font-size="12" text-anchor="middle">
<rect x="20" y="80" width="140" height="56" rx="8" fill="#12202c"/>
<text x="90" y="104" fill="#f6f3ee">Zapier cloud</text>
<text x="90" y="120" fill="#c5d0d8" font-size="10">verae-zapier-app</text>
<rect x="200" y="80" width="140" height="56" rx="8" fill="#0f6e56"/>
<text x="270" y="104" fill="#fff">zappier-edge</text>
<text x="270" y="120" fill="#d4efe6" font-size="10">HTTPS :3000 meter</text>
<rect x="380" y="80" width="160" height="56" rx="8" fill="#0b4f8a"/>
<text x="460" y="104" fill="#fff">verae-middleware</text>
<text x="460" y="120" fill="#d9e2e8" font-size="10">HTTPS :3100 wait</text>
<rect x="600" y="80" width="140" height="56" rx="8" fill="#5b2d8a"/>
<text x="670" y="104" fill="#fff">Verae chain</text>
<text x="670" y="120" fill="#e8d9f5" font-size="10">api.veraetime.net</text>
<polygon points="160,108 200,108 200,104 212,108 200,112 200,108" fill="#52646f"/>
<text x="180" y="98" font-size="9" fill="#52646f">HTTPS</text>
<polygon points="340,108 380,108 380,104 392,108 380,112 380,108" fill="#52646f"/>
<text x="360" y="98" font-size="9" fill="#52646f">HTTPS</text>
<polygon points="540,108 600,108 600,104 612,108 600,112 600,108" fill="#52646f"/>
<text x="568" y="98" font-size="9" fill="#52646f">HTTPS hash</text>
<rect x="200" y="200" width="520" height="180" rx="10" fill="#e8eef2" stroke="#1d4f91" stroke-width="1.5"/>
<text x="460" y="222" fill="#1d4f91" font-size="13">Central Verae NATS.IO 3-server JetStream cluster</text>
<rect x="230" y="240" width="100" height="40" rx="6" fill="#1d4f91"/>
<text x="280" y="264" fill="#fff">node A</text>
<rect x="410" y="240" width="100" height="40" rx="6" fill="#1d4f91"/>
<text x="460" y="264" fill="#fff">node B</text>
<rect x="590" y="240" width="100" height="40" rx="6" fill="#1d4f91"/>
<text x="640" y="264" fill="#fff">node C</text>
<rect x="220" y="310" width="110" height="48" rx="6" fill="#fff" stroke="#0f6e56"/>
<text x="275" y="330" fill="#12202c">job-poller</text>
<text x="275" y="346" fill="#52646f" font-size="10">jobs.watch</text>
<rect x="345" y="310" width="120" height="48" rx="6" fill="#fff" stroke="#0f6e56"/>
<text x="405" y="330" fill="#12202c">webhook-deliver</text>
<text x="405" y="346" fill="#52646f" font-size="10">webhooks.deliver</text>
<rect x="480" y="310" width="110" height="48" rx="6" fill="#fff" stroke="#8a5a00"/>
<text x="535" y="330" fill="#12202c">WORM × N</text>
<text x="535" y="346" fill="#52646f" font-size="10">archive.query</text>
<rect x="605" y="310" width="100" height="48" rx="6" fill="#fff" stroke="#8a5a00"/>
<text x="655" y="330" fill="#12202c">tree-node</text>
<text x="655" y="346" fill="#52646f" font-size="10">kind=tree</text>
<line x1="460" y1="136" x2="460" y2="200" stroke="#1d4f91" stroke-width="1.5"/>
<text x="478" y="168" font-size="9" fill="#1d4f91" text-anchor="start">JetStream</text>
</g>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 3.5 KiB

22
diagrams/uptime.svg Normal file
View file

@ -0,0 +1,22 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 800 280" width="800" height="280" role="img" aria-label="Fleet keepFloor, restart, SSH failover">
<rect width="800" height="280" fill="#f6f3ee"/>
<text x="20" y="28" font-family="Georgia,serif" font-size="16" fill="#12202c">Uptime: floors, restart, another machine</text>
<g font-family="system-ui,sans-serif" font-size="12">
<rect x="20" y="50" width="180" height="70" rx="8" fill="#e3f6e8" stroke="#0f6e56"/>
<text x="110" y="80" text-anchor="middle" fill="#0f6e56">healthy replica</text>
<text x="110" y="100" text-anchor="middle" fill="#52646f" font-size="10">counts toward min</text>
<rect x="220" y="50" width="180" height="70" rx="8" fill="#fff3bf" stroke="#8a5a00"/>
<text x="310" y="80" text-anchor="middle" fill="#8a5a00">paused / slow RTT</text>
<text x="310" y="100" text-anchor="middle" fill="#52646f" font-size="10">not available → spawn</text>
<rect x="420" y="50" width="180" height="70" rx="8" fill="#f9d4d4" stroke="#a32020"/>
<text x="510" y="80" text-anchor="middle" fill="#a32020">/health fail or SSH down</text>
<text x="510" y="100" text-anchor="middle" fill="#52646f" font-size="10">restart or next host</text>
<rect x="620" y="50" width="160" height="70" rx="8" fill="#1d4f91"/>
<text x="700" y="80" text-anchor="middle" fill="#fff">JetStream replay</text>
<text x="700" y="100" text-anchor="middle" fill="#d9e2e8" font-size="10">consumer catch-up</text>
<rect x="20" y="150" width="760" height="100" rx="8" fill="#fff" stroke="#d5dee4"/>
<text x="40" y="178" fill="#12202c">tree-node min=3 keepFloor</text>
<text x="40" y="200" fill="#52646f" font-size="12">local + ns1 (marchon@70.88.205.138, ~/.ssh/id_ed25519). Least-loaded placement. Bloom miss = silence so a partitioned archive does not block the bus.</text>
<text x="40" y="222" fill="#52646f" font-size="12">Zapier only sees HTTPS 202 / wait JSON / REST Hook. Cluster majority of 3 NATS servers survives one node loss.</text>
</g>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 2 KiB

6
package.json Normal file
View file

@ -0,0 +1,6 @@
{
"name": "overview",
"version": "1.0.0",
"private": true,
"description": "High-level overview of the Verae Time × Zapier system, modules, uptime, NATS cluster, and docs index"
}

BIN
screenshots/console-docs.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 142 KiB

BIN
screenshots/console-fleet.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 211 KiB

BIN
screenshots/console-trace.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 254 KiB