commit 340720724bd248a0ebde78f10b8d9e556ee996ba Author: George Lambert Date: Fri Sep 11 15:36:41 2026 -0400 Initial import of overview from zapier monorepo diff --git a/01-system.md b/01-system.md new file mode 100644 index 0000000..35089e8 --- /dev/null +++ b/01-system.md @@ -0,0 +1,28 @@ +# 1. System + +Verae Time proves a SHA-256 existed at a given time. Zapier lets customers register and look up hashes from the tools they already use. Billing and API keys live on **zappier-edge**. Timestamping, wait, webhooks, and archive fan-out live behind **verae-middleware**. Durable messaging is the **central Verae NATS.IO 3-server JetStream cluster**. + +## Hard rule + +**Zapier never connects to NATS, tree nodes, WORM archives, or `api.veraetime.net`.** Those hops are middleware and workers only. If a trace ever shows a Zapier hop on a `verae.*` subject, do not push the app. + +## Planes + +| Plane | What | Network | +|-------|------|---------| +| Zapier cloud | `verae-zapier-app`, `verae-activate` | HTTPS to zappier-edge | +| Commercial edge | `zappier-edge` portal, admin, `x-api-key`, Stripe meter | Public HTTPS; NATS billing pubs/requests | +| Account balance | `zappier-account-balance` prepaid SoT | NATS `verae.billing.*` + HTTP `:3010` | +| CS / sales / accounting | `zappier-customer-service`, `zappier-sales-pricing`, `zappier-accounting-export` | Private HTTP; NATS statement/adjust | +| Middleware HTTP | `/zapier/v1/*` job id + wait | Public HTTPS from edge only | +| NATS cluster | JetStream subjects under `verae.*` | Private; loopback or SSH tunnel | +| Workers | poller, webhook-deliver, aggregator | NATS + HTTPS to chain or Zapier hooks | +| Archives | WORM + tree nodes | NATS broadcast query; bloom miss = silence | +| Chain | Verae timestamping | HTTPS `api.veraetime.net` or MOCK | +| Control | `verae-fleet` | Operator loopback `:3850`; SSH to extra machines | + +## Request in one sentence + +A Zap step POSTs to zappier-edge (`x-api-key`); edge meters the call and, when `ZAPPIER_UPSTREAM` is set, forwards timestamp/receipt/hash to middleware `/zapier/v1`. Middleware splits hash vs files, writes the hash (or Merkle root) to the chain, publishes `verae.zapier.jobs.watch`, and either returns `jobId` or waits on `verae.zapier.jobs.events`. Meter events, CS credits, and portal reloads go over NATS `verae.billing.*` to **account-balance** (source of truth). Customers review the same statement in the portal; CS (`:3011`) and sales (`:3012`) review it on their department UIs. Attached metadata and bulk-summary **leaves** are found later by broadcasting `verae.archive.query` to every archive/tree node. + +See [diagrams](08-diagrams.md). diff --git a/02-modules-and-repos.md b/02-modules-and-repos.md new file mode 100644 index 0000000..4be3f26 --- /dev/null +++ b/02-modules-and-repos.md @@ -0,0 +1,30 @@ +# 2. Modules and independent repositories + +Each runtime piece is its **own git repo** on Forgejo (`git.georgelambert.org`, SSH port 2223). The monorepo `master-zapier-plan-draft` is a snapshot of the workspace; do not treat it as the only clone path. + +| Independent repo | Package / path | What it does | +|------------------|----------------|--------------| +| **overview** | `packages/overview` | This high-level map | +| **verae-nats-process** | `packages/verae-nats-process` | **Template** for a new addressed NATS process | +| **master-zapier-plan-draft** | workspace root | Combined snapshot (`main` and `master`) | +| **zappier-edge** | `packages/zappier` | Metered HTTPS, portal, admin, Stripe; **proxies Verae calls to middleware** | +| **zappier-account-balance** | `packages/zappier-account-balance` | NATS source of truth for prepaid balances | +| **zappier-customer-service** | `packages/zappier-customer-service` | CS goodwill credits onto prepaid balances | +| **zappier-sales-pricing** | `packages/zappier-sales-pricing` | Sales per-customer tier / multiplier | +| **zappier-accounting-export** | `packages/zappier-accounting-export` | QuickBooks IIF + accounting CSV | +| **verae-middleware** | `packages/verae-zapier-middleware` | `/zapier/v1`, job wait, NATS publishers/workers | +| **verae-zapier-app** | `packages/verae-zapier` | Full Zapier nouns (timestamp, wait, batch, hash + tree lookup) | +| **verae-activate** | `packages/verae-activate` | Activate-now (Add Numbers, Echo, SHA256, mock timestamp) | +| **verae-request-splitter** | `packages/verae-request-splitter` | Chain hash vs `archive.put` | +| **verae-archive-worm** | `packages/verae-archive-worm` | Bloom WORM node | +| **verae-archive-aggregator** | `packages/verae-archive-aggregator` | Merge archive replies | +| **verae-tree-node** | `packages/verae-tree-node` | Merkle leaf proofs (bulk summaries) | +| **verae-fleet** | `packages/verae-fleet` | Catalog, min copies, pause/restart, SSH hosts, operator console | +| **verae-zapier-simulator** | `packages/verae-zapier-simulator` | Trace console before `zapier-platform push` | +| **zapier-user-docs** | `packages/zapier-user-docs` | Customer signup → register → lookup | +| **zapier-docs-master** | `packages/docs-master` | Per-module `SUMMARY.md` + `NATS.md` | +| **verae-ops** | `packages/verae-ops` | Docker, Proxmox, VMs, dedicated hardware, linking services | + +Libraries that are **not** separate Forgejo apps today: `verae-chain-client` (inside middleware), job-poller and webhook-deliver (middleware workers, fleet-spawned). + +Per-module NATS contracts: [zapier-docs-master](https://git.georgelambert.org/marchon/zapier-docs-master). diff --git a/03-nats-cluster.md b/03-nats-cluster.md new file mode 100644 index 0000000..fc3042b --- /dev/null +++ b/03-nats-cluster.md @@ -0,0 +1,40 @@ +# 3. Central Verae NATS.IO 3-server cluster + +All durable messaging for this product is **NATS JetStream**, not Zapier queues and not a public TCP API. + +## Cluster + +Three `nats-server -js` nodes form the Verae cluster (routes between them, JetStream replication). Clients (middleware, poller, aggregator, WORM, tree nodes, account-balance, zappier-edge billing, CS/sales/accounting, and any new process cloned from `verae-nats-process`) connect with a **cluster URL list**, for example: + +```text +nats://127.0.0.1:4222,nats://127.0.0.1:4223,nats://127.0.0.1:4224 +``` + +On a given machine the listener stays on **loopback** (or a private interface). Operators reach it with `scripts/nats-tunnel.sh` / `ssh -L 14222:127.0.0.1:4222`. **Do not bind 4222 on `0.0.0.0` without auth.** + +Today’s NS1 box (`NS1.GEORGELAMBERT.ORG`, `70.88.205.138`) already runs JetStream on `127.0.0.1:4222`. The **target** is three clustered nodes so losing one server does not lose the stream. Fleet SSH hosts (`ns1`, later `lan-134`) run **workers**, not extra public NATS listeners. + +![NATS cluster](diagrams/nats-cluster.svg) + +## Who may connect + +| Allowed | Forbidden | +|---------|-----------| +| verae-middleware, fleet workers, WORM, tree nodes, account-balance, zappier-edge billing, CS/sales/accounting, `verae-nats-process` clones | Zapier cloud, customer browsers (they use HTTPS portal/CS/sales UIs) | + +## Address families already in use + +| Address | Kind | Notes | +|---------|------|--------| +| `verae.zapier.jobs.watch` | work queue | poller | +| `verae.zapier.jobs.events` | events | wait + webhooks | +| `verae.zapier.webhooks.deliver` | work queue | HTTPS to Zapier REST Hook | +| `verae.zapier.usage` | optional | metering | +| `verae.billing.statement.get` | request-reply | portal, CS, sales, admin | +| `verae.billing.balance.adjust` | request-reply | CS credits, portal reload | +| `verae.billing.usage.recorded` | pub | zappier-edge meter | +| `verae.archive.put` | JetStream | splitter / merkle builder | +| `verae.archive.query` | **broadcast** (no queue group) | every WORM and tree node | +| `verae.archive.reply.` | replies | **only on bloom hit** | + +New functions get new `verae...` addresses — see [06-address-routing.md](06-address-routing.md). diff --git a/04-uptime.md b/04-uptime.md new file mode 100644 index 0000000..1b3ea27 --- /dev/null +++ b/04-uptime.md @@ -0,0 +1,34 @@ +# 4. How the system maintains uptime + +Uptime is **NATS durability + fleet replica floors + more than one machine**, not a single always-on Zapier connection. + +## Replica floors (`verae-fleet`) + +`packages/verae-fleet/fleet.json` sets `min` / `max` / `keepFloor` per service. **Available** means running, healthy, and not paused. + +| Service | Default min | keepFloor | +|---------|-------------|-----------| +| tree-node | 3 | yes | +| archive-worm | 3 | yes | +| archive-aggregator, job-poller, webhook-deliver | 1 | yes | +| zappier-edge, middleware-http | 1 | yes | + +If a tree node is paused, crashes, or fails `/health`, fleet **starts another copy** until three are available. Operator console: http://127.0.0.1:3850/ (Fleet tab — green / yellow / red; Trace tab for hop tests; Docs tab for reading order). + +## Restart and pause + +- Unhealthy `/health` → same instance id restarted. +- Pause does not count toward `min`. +- `stop` on a service disables keepFloor for that service. + +## Spread across machines + +`machines.json` lists hosts (`local`, `ns1` = `marchon@70.88.205.138` with `~/.ssh/id_ed25519`, optional `lan-134`). New replicas go to the **least-loaded** eligible host. Remote spawn/health/kill is SSH; workers bind loopback on the remote box. + +## JetStream + +Work queues (`jobs.watch`, `webhooks.deliver`) replay if a consumer dies. Event stream (`jobs.events`) lets waiters and webhook routers catch up. A 3-node cluster keeps the stream if one NATS server is down. + +## What Zapier sees + +HTTPS 202 `jobId`, wait JSON, or REST Hook. Timeouts return `pending` + `jobId` so the **Timestamp Completed** trigger can finish the job. Zapier retries are safe: the same SHA-256 returns the original seal (`existing: true`). diff --git a/05-network-failures.md b/05-network-failures.md new file mode 100644 index 0000000..a3bb11c --- /dev/null +++ b/05-network-failures.md @@ -0,0 +1,23 @@ +# 5. Local network failures + +“Local network” means the operator LAN, SSH to NS1, or a partitioned archive — not Zapier’s cloud. + +| Failure | What happens | What the customer sees | +|---------|--------------|------------------------| +| NATS node unreachable | Client reconnects to another cluster URL; JetStream consumers resume | Wait may return `pending`; hook still fires later | +| All NATS down | Middleware cannot publish `jobs.watch`; fleet marks workers unhealthy | 503 / pending; no NATS leak to Zapier | +| One WORM / tree node partitioned | Bloom miss = **no packet**; aggregator uses whoever answered | Lookup may miss attachments until the node returns; seal on chain still valid | +| SSH to a fleet host fails | Spawn fails; fleet **places the next replica on another machine** | Floor still met if capacity remains on `local` or another SSH host | +| Chain `api.veraetime.net` timeout | Poller retries; then `timestamp.timeout` event | Wait → pending or failed; async + hook still the recovery path | +| zappier-edge 402 | QuotaExceeded with upgrade URL | Zap step error; no NATS involved | +| Tunnel to loopback NATS dropped | `NATS_URL=nats://127.0.0.1:14222` dies; restart `nats-tunnel.sh` | Workers on NS1 itself still see `127.0.0.1:4222` | + +## Design choices that make partitions survivable + +1. **Archive query is broadcast**, not a shared queue group — a dead node does not steal the message. +2. **Bloom miss is silence** — missing nodes do not send empty errors that look like “hash unknown”. +3. **Zero replies + known puts** is an outage, not a miss (simulator / fleet monitors flag this). +4. **NATS is not on the public NIC** — a WAN blip does not expose 4222. +5. **Idempotent seals** — retrying a Zap after a network error will not double-timestamp. + +See fleet RTT (min / avg / p50 / p90) when planning extra tree nodes after a flaky path. diff --git a/06-address-routing.md b/06-address-routing.md new file mode 100644 index 0000000..5f0d666 --- /dev/null +++ b/06-address-routing.md @@ -0,0 +1,33 @@ +# 6. Address routing (including unplanned functions) + +NATS **addresses** (subjects) are the extension point. A new search, store, or job type is a new address plus a process that listens — not a new Zapier TCP client. + +## Pattern + +```text +verae... +verae...reply. +``` + +| Piece | Example | Meaning | +|-------|---------|---------| +| `verae` | — | Verae bus (not Zapier) | +| `area` | `zapier`, `archive`, `search`, `store` | Product slice | +| `resource` | `jobs`, `hashes`, `blobs` | Noun | +| `action` | `watch`, `query`, `put`, `in` | Verb | +| `reply.` | — | Correlated response | + +**Queue group** (work sharing): `area-resource-action` (e.g. `job-poller`). +**No queue group** (fan-out): archive/tree **query** so every node sees every lookup. + +## Adding something that does not exist yet + +1. Copy the independent repo **[verae-nats-process](https://git.georgelambert.org/marchon/verae-nats-process)** (`packages/verae-nats-process`). +2. Rename `verae.example.process.in` / `.out` / `.reply.*` in `src/subjects.js`. +3. Add a row to that repo’s `ROUTING.md` and to [INDEX.md](INDEX.md). +4. Register the process in `verae-fleet` (`min`/`max`, machines, roles). +5. If Zapier must call it, add **one HTTPS route** on middleware — Zapier still never sees NATS. + +Do **not** invent a public NATS URL for Zapier. Do **not** reuse `verae.archive.query` as a queue group. + +![Address expansion](diagrams/routing.svg) diff --git a/07-external-resources.md b/07-external-resources.md new file mode 100644 index 0000000..afa9304 --- /dev/null +++ b/07-external-resources.md @@ -0,0 +1,15 @@ +# 7. External resources: search, storage, and the chain + +| Resource | Where | How a Zap reaches it | +|----------|--------|----------------------| +| **Central chain** (itemized SHA-256 + time + certificate) | `api.veraetime.net` or MOCK | HTTPS via middleware: create / wait / `GET /hashes/{sha256}` | +| **Bulk Merkle root** | Same chain, one seal | `POST /timestamp/batch` | +| **Leaf proofs** (hash only in a bulk summary) | Tree-node WORM on NATS | `GET /hashes/{sha}?includeTree=true` → `verae.archive.query` kinds=`tree` | +| **Public / private metadata, files** | WORM archives | `includeAttached`; never on chain | +| **Search (central)** | Chain lookup | Zapier search **Find Timestamp by SHA256** | +| **Search (external tree / extra stores)** | NATS query to every node that might hold the key | Zapier search **Find Hash (tree nodes + central)** | +| **Future search / store** | New `verae.search.*` or `verae.store.*` process | Copy `verae-nats-process`; optional middleware GET | + +Storage that is **not** the blockchain stays on WORM/tree nodes (and later any process that answers `verae.archive.query` or a new store address). The chain stores hash + time + certificate (+ Merkle root for batches). + +External “unplanned” storage or search is the same pattern: new address, bloom or index on that node, silence on miss, aggregator or the template’s reply subject. diff --git a/08-diagrams.md b/08-diagrams.md new file mode 100644 index 0000000..d78e79e --- /dev/null +++ b/08-diagrams.md @@ -0,0 +1,65 @@ +# 8. Architectural diagrams + +SVG files in [`diagrams/`](diagrams/). The same shapes are repeated below in mermaid for Forgejo preview. + +## 8.1 End-to-end (HTTPS vs NATS) + +![System](diagrams/system.svg) + +```mermaid +flowchart LR + Z[Zapier cloud apps] -->|HTTPS x-api-key| E[zappier-edge] + Cust[customer portal] -->|HTTPS statement| E + CS[customer-service] -->|NATS statement/adjust| N[NATS 3-node cluster] + SA[sales-pricing] -->|NATS statement| N + AC[accounting-export] -->|NATS statement| N + E -->|NATS billing| N + BAL[account-balance] -->|reply verae.billing.*| N + E -->|HTTPS metered| M[verae-middleware] + M -->|HTTPS| C[Verae chain] + M -->|JetStream| N + N --> P[job-poller] + N --> W[webhook-deliver] + N --> A[archive-aggregator] + N --> R[WORM x N] + N --> T[tree-node x N] + P -->|HTTPS status| C + W -->|HTTPS REST Hook| Z +``` + +## 8.2 NATS.IO 3-server cluster + +![Cluster](diagrams/nats-cluster.svg) + +```mermaid +flowchart TB + subgraph cluster [Verae NATS.IO JetStream] + N1[nats-server A :4222 loopback] + N2[nats-server B] + N3[nats-server C] + N1 <--> N2 + N2 <--> N3 + N3 <--> N1 + end + MW[middleware + workers] -->|cluster URL list| cluster + F[fleet SSH hosts] -->|workers only| cluster +``` + +## 8.3 Uptime and failure + +![Uptime](diagrams/uptime.svg) + +```mermaid +flowchart TD + H[health /health] -->|fail| R[restart instance] + P[pause] -->|not available| F[keepFloor spawn] + S[SSH host down] -->|spawn failed| O[place on next machine] + Q[archive.query broadcast] -->|bloom miss| SIL[silence] + Q -->|hit| REP[archive.reply.id] +``` + +## 8.4 New address = new process + +![Routing](diagrams/routing.svg) + +See [09-expansion-template.md](09-expansion-template.md). diff --git a/09-expansion-template.md b/09-expansion-template.md new file mode 100644 index 0000000..6ef8e5d --- /dev/null +++ b/09-expansion-template.md @@ -0,0 +1,29 @@ +# 9. Model repo for a new addressed process + +**Repository:** [verae-nats-process](https://git.georgelambert.org/marchon/verae-nats-process) +**Path in the monorepo:** `packages/verae-nats-process` +**Clone:** `ssh://git@git.georgelambert.org:2223/marchon/verae-nats-process.git` + +This is the **reference implementation** for expansions: a small JetStream worker with: + +- a single **in** address, an **out** address, and **reply.<correlationId>** +- `handle(msg)` you replace with real work (search, store, transform) +- HTTP `/health` so `verae-fleet` can keep a replica floor +- tests that do not need a live cluster +- `ROUTING.md` — the row you copy into the global address table + +Default subjects (rename before production): + +| Direction | Address | +|-----------|---------| +| IN | `verae.example.process.in` | +| OUT | `verae.example.process.out` | +| REPLY | `verae.example.process.reply.` | + +```bash +cd packages/verae-nats-process +npm test +# then rename example → your area, add fleet.json min, clone as a new Forgejo repo +``` + +Zapier still must not subscribe. If a Zap needs the result, middleware exposes HTTPS and publishes to the new **in** address. diff --git a/INDEX.md b/INDEX.md new file mode 100644 index 0000000..ce83659 --- /dev/null +++ b/INDEX.md @@ -0,0 +1,91 @@ +# Documentation index + +**Bring the system online:** https://zapier.georgelambert.org/packages/verae-ops/GETTING-STARTED.pdf + +Live catalog (PDF by default): https://zapier.georgelambert.org/ — [Markdown indexes](https://zapier.georgelambert.org/index-md.html) + +Operator console (loopback): http://127.0.0.1:3850/ · [CONSOLE.pdf](https://zapier.georgelambert.org/packages/verae-fleet/docs/CONSOLE.pdf) + +## 1. High-level (non-technical) + +| Document | What it is | +|----------|------------| +| [README.md](README.md) | Overview (start here) | +| [01-system.md](01-system.md) | What runs where, in plain language | +| [02-modules-and-repos.md](02-modules-and-repos.md) | Short names of every independent repo | +| [zapier-docs-master](https://git.georgelambert.org/marchon/zapier-docs-master) | One-line SUMMARY + NATS per module | +| [zapier-user-docs](https://zapier.georgelambert.org/user-docs/README.pdf) | Signup → register → lookup | +| [verae-ops](https://zapier.georgelambert.org/packages/verae-ops/README.pdf) | Docker / Proxmox / VM / metal install | + +## 2. Architecture + +| Document | URL | +|----------|-----| +| NATS cluster | [03-nats-cluster.md](03-nats-cluster.md) | +| Uptime | [04-uptime.md](04-uptime.md) | +| Network failures | [05-network-failures.md](05-network-failures.md) | +| Address routing | [06-address-routing.md](06-address-routing.md) | +| External resources | [07-external-resources.md](07-external-resources.md) | +| Diagrams | [08-diagrams.md](08-diagrams.md) | +| Catalog home | https://zapier.georgelambert.org/ (PDF) · [Markdown indexes](https://zapier.georgelambert.org/index-md.html) | +| Composition | https://zapier.georgelambert.org/docs/02-architecture/composition.pdf | +| NATS gateway | https://zapier.georgelambert.org/docs/02-architecture/nats-gateway.pdf | +| NATS subjects | https://zapier.georgelambert.org/docs/02-architecture/nats-subjects.pdf | +| Archive / bloom | https://zapier.georgelambert.org/docs/02-architecture/archive-nats.pdf | +| Module NATS map | https://zapier.georgelambert.org/docs/02-architecture/modules-and-nats.pdf | +| Tree nodes | https://zapier.georgelambert.org/docs/02-architecture/tree-nodes.pdf | +| Fleet | https://zapier.georgelambert.org/docs/02-architecture/fleet.pdf | + +## 3. Remaining documentation + +| Document | URL | +|----------|-----| +| Expansion template | [09-expansion-template.md](09-expansion-template.md) | +| Operator console | https://zapier.georgelambert.org/packages/verae-fleet/docs/CONSOLE.pdf | +| User guide | https://zapier.georgelambert.org/user-docs/README.pdf | +| Tree-node lookup (users) | https://zapier.georgelambert.org/packages/zapier-user-docs/09-lookup-tree-nodes.pdf | +| Zapier developer setup | https://zapier.georgelambert.org/docs/04-activate/SETUP-ZAPIER-DEVELOPER.pdf | +| docs-master | https://zapier.georgelambert.org/docs-master/README.pdf | +| Sphinx HTML | https://zapier.georgelambert.org/sphinx/ | +| Sphinx LaTeX PDF | https://zapier.georgelambert.org/sphinx/verae-zapier-modules.pdf | +| Install (verae-ops) | https://zapier.georgelambert.org/packages/verae-ops/README.pdf | + +## Independent git repositories + +Prefix: `https://git.georgelambert.org/marchon/` + +overview · verae-nats-process · verae-ops · master-zapier-plan-draft · zappier-edge · verae-middleware · verae-zapier-app · verae-activate · verae-request-splitter · verae-archive-worm · verae-archive-aggregator · verae-tree-node · verae-fleet · verae-zapier-simulator · zapier-user-docs · zapier-docs-master + +## Per-module contracts + +In **zapier-docs-master**: `modules//SUMMARY.md` and `NATS.md` for every runtime package. + +## Operator surfaces (not public NATS) + +| Surface | URL | +|---------|-----| +| Operator console (Fleet · Trace · Docs) | http://127.0.0.1:3850/ | +| Standalone simulator | http://127.0.0.1:3847/ | + +## A–Z subject index + +| Term | See | +|------|-----| +| aggregator | verae-archive-aggregator; archive.query / reply | +| bloom miss | silence; 05-network-failures | +| batch / Merkle | verae-tree-node; user-docs 08–09 | +| cluster | 03-nats-cluster | +| fleet / keepFloor | verae-fleet; 04-uptime | +| hash lookup | verae-zapier-app searches; 07-external-resources | +| jobs.events / jobs.watch | nats-subjects.html; middleware | +| includeTree | tree lookup; archive.query kinds=tree | +| routing | 06-address-routing; verae-nats-process | +| splitter | verae-request-splitter | +| SSH hosts | verae-fleet machines.json | +| Zapier never NATS | 01-system | +| zappier-edge | metering, portal | +| operator console | http://127.0.0.1:3850/ ; verae-fleet docs/CONSOLE.md | + +--- + +Design: **Scott Lindsey**, **George Lambert**, **NATS.IO**, and **Grok-Code** by [Grok.com](https://grok.com). diff --git a/NATS.md b/NATS.md new file mode 100644 index 0000000..b1f59cc --- /dev/null +++ b/NATS.md @@ -0,0 +1,7 @@ +# NATS — overview + +This documentation repo does **not** subscribe. It describes the **central Verae NATS.IO 3-server JetStream cluster** that every worker talks to. + +Zapier cloud never connects here. Clients use private URLs (`nats://127.0.0.1:4222` on a node, or an SSH tunnel). Do not bind 4222 on `0.0.0.0` without auth. + +New functions are new **addresses** (subjects), not new public TCP ports. Copy `verae-nats-process` to add one. diff --git a/README.md b/README.md new file mode 100644 index 0000000..0f53d3f --- /dev/null +++ b/README.md @@ -0,0 +1,106 @@ +# Verae Time × Zapier — system overview + +High-level description of the whole system: what it is, which **independent git repositories** implement it, how it stays up, how it behaves when the local network fails, and how it talks to the **central Verae NATS.IO 3-server cluster**. New work is added by **new address routing**, not by teaching Zapier about NATS. + +**This repo:** https://git.georgelambert.org/marchon/overview +**Bring online:** https://zapier.georgelambert.org/packages/verae-ops/GETTING-STARTED.pdf +**Live catalog (PDF by default):** https://zapier.georgelambert.org/ · [Markdown indexes](https://zapier.georgelambert.org/index-md.html) +**Clone:** `ssh://git@git.georgelambert.org:2223/marchon/overview.git` + +## Table of contents + +1. [System](01-system.md) — what runs where; Zapier never speaks NATS +2. [Modules and repositories](02-modules-and-repos.md) — every independent Forgejo repo +3. [NATS.IO 3-server cluster](03-nats-cluster.md) — JetStream, subjects, who may connect +4. [Uptime](04-uptime.md) — replica floors, restart, SSH spread +5. [Local network failures](05-network-failures.md) — reconnect, silence, pending, failover +6. [Address routing](06-address-routing.md) — how to add unplanned functions +7. [External resources](07-external-resources.md) — searches, storage, chain +8. [Architectural diagrams](08-diagrams.md) — SVG + mermaid +9. [Expansion template](09-expansion-template.md) — `verae-nats-process` +10. [Documentation index](INDEX.md) — pointer into all other docs + +## One-screen picture + +![System context](diagrams/system.svg) + +Zapier → HTTPS `zappier-edge` → HTTPS `verae-middleware` → **NATS cluster** → workers, WORM archives, tree nodes. Chain HTTPS is `api.veraetime.net` (or MOCK). Fleet keeps minimum copies, including tree nodes. + +## Independent repositories (Forgejo) + +| Repo | Role | +|------|------| +| [overview](https://git.georgelambert.org/marchon/overview) | This document | +| [verae-ops](https://git.georgelambert.org/marchon/verae-ops) | Docker, Proxmox, VMs, dedicated hardware | +| [verae-nats-process](https://git.georgelambert.org/marchon/verae-nats-process) | **Model** for a new addressed process | +| [master-zapier-plan-draft](https://git.georgelambert.org/marchon/master-zapier-plan-draft) | Monorepo snapshot | +| [zappier-edge](https://git.georgelambert.org/marchon/zappier-edge) | Metered public HTTPS | +| [verae-middleware](https://git.georgelambert.org/marchon/verae-middleware) | Zapier HTTP + NATS workers | +| [verae-zapier-app](https://git.georgelambert.org/marchon/verae-zapier-app) | Zapier Platform app | +| [verae-activate](https://git.georgelambert.org/marchon/verae-activate) | Activate-now app | +| [verae-request-splitter](https://git.georgelambert.org/marchon/verae-request-splitter) | Hash vs attachments | +| [verae-archive-worm](https://git.georgelambert.org/marchon/verae-archive-worm) | Bloom WORM node | +| [verae-archive-aggregator](https://git.georgelambert.org/marchon/verae-archive-aggregator) | Archive reply merge | +| [verae-tree-node](https://git.georgelambert.org/marchon/verae-tree-node) | Merkle leaf proofs | +| [verae-fleet](https://git.georgelambert.org/marchon/verae-fleet) | Replica floor + SSH hosts | +| [verae-zapier-simulator](https://git.georgelambert.org/marchon/verae-zapier-simulator) | Trace console | +| [zapier-user-docs](https://git.georgelambert.org/marchon/zapier-user-docs) | Signup → lookup | +| [zapier-docs-master](https://git.georgelambert.org/marchon/zapier-docs-master) | Per-module SUMMARY + NATS | + +Clone any of them: `git clone ssh://git@git.georgelambert.org:2223/marchon/.git` (SSH port **2223**). + +## Operator console (loopback) + +Fleet, message **Trace**, and **Docs** share one shell at http://127.0.0.1:3850/ (`verae-fleet`). Not public. See [CONSOLE.pdf](https://zapier.georgelambert.org/packages/verae-fleet/docs/CONSOLE.pdf). + +![Fleet tab](screenshots/console-fleet.png) + +![Trace tab](screenshots/console-trace.png) + +![Docs tab](screenshots/console-docs.png) + +## Where to read next + +Start at the top. Architecture is second. Everything else is reference. + +### 1. High-level (non-technical) + +| Document | What it is | +|----------|------------| +| **This overview** | Whole system in one place | +| [Modules and repositories](02-modules-and-repos.md) | Short names of every independent repo | +| [zapier-docs-master](https://git.georgelambert.org/marchon/zapier-docs-master) | One-line SUMMARY + NATS per module | +| [zapier-user-docs](https://git.georgelambert.org/marchon/zapier-user-docs) | Signup → register a hash → lookup | + +### 2. Architecture + +| Document | What it is | +|----------|------------| +| [NATS 3-server cluster](03-nats-cluster.md) | Who may connect; private URLs | +| [Uptime](04-uptime.md) | Replica floors, restart, SSH hosts | +| [Local network failures](05-network-failures.md) | Reconnect, bloom silence, failover | +| [Address routing](06-address-routing.md) | Unplanned functions as new subjects | +| [External resources](07-external-resources.md) | Chain, WORM, tree-node search | +| [Diagrams](08-diagrams.md) | SVG + mermaid | +| [Fleet](https://zapier.georgelambert.org/docs/02-architecture/fleet.pdf) | Operator replica control | +| [Module NATS map](https://zapier.georgelambert.org/docs/02-architecture/modules-and-nats.pdf) | Address table | +| [Archive / bloom](https://zapier.georgelambert.org/docs/02-architecture/archive-nats.pdf) | Multipart and multi-receipt | +| [Tree nodes](https://zapier.georgelambert.org/docs/02-architecture/tree-nodes.pdf) | Bulk Merkle leaves | +| [Composition](https://zapier.georgelambert.org/docs/02-architecture/composition.pdf) | zappier + middleware HTTPS | + +### 3. Remaining documentation + +| Document | What it is | +|----------|------------| +| [Documentation index](INDEX.md) | A–Z and catalog URLs | +| [Expansion template](09-expansion-template.md) / [verae-nats-process](https://git.georgelambert.org/marchon/verae-nats-process) | Copy this to add an address | +| [Operator console](https://zapier.georgelambert.org/packages/verae-fleet/docs/CONSOLE.pdf) | Screenshots of Fleet / Trace / Docs | +| [Simulator](https://git.georgelambert.org/marchon/verae-zapier-simulator) | Standalone trace (also inside the console) | +| [Zapier developer setup](https://zapier.georgelambert.org/docs/04-activate/SETUP-ZAPIER-DEVELOPER.pdf) | CLI / login / push | +| Live catalog | https://zapier.georgelambert.org/ (PDF) · [Markdown indexes](https://zapier.georgelambert.org/index-md.html) | + +--- + +## Credits + +Design: **Scott Lindsey**, **George Lambert**, **NATS.IO**, and **Grok-Code** by [Grok.com](https://grok.com). diff --git a/SUMMARY.md b/SUMMARY.md new file mode 100644 index 0000000..9aba71b --- /dev/null +++ b/SUMMARY.md @@ -0,0 +1,7 @@ +# overview + +**Job:** High-level map of the whole Verae Time × Zapier system: modules, independent git repos, uptime, local-network failure, the central 3-node NATS.IO cluster, and how to add a new addressed process. + +**Expects / sends:** none (documentation). + +**Repos named:** see [02-modules-and-repos.md](02-modules-and-repos.md). Expansion template: `verae-nats-process`. diff --git a/diagrams/nats-cluster.svg b/diagrams/nats-cluster.svg new file mode 100644 index 0000000..44e9a71 --- /dev/null +++ b/diagrams/nats-cluster.svg @@ -0,0 +1,21 @@ + + + Verae NATS.IO — three JetStream servers + + + A + :4222 loopback + + B + route + + C + JetStream + + + + + workers + middleware + → nats://A,B,C (never Zapier) + + diff --git a/diagrams/routing.svg b/diagrams/routing.svg new file mode 100644 index 0000000..5229df4 --- /dev/null +++ b/diagrams/routing.svg @@ -0,0 +1,25 @@ + + + New function = new address + clone of verae-nats-process + + + copy template repo + verae-nats-process + + rename subjects + verae.area.resource.action + + fleet min/max + keepFloor + machines + + optional HTTPS + middleware only → Zapier + + + + + Existing: verae.zapier.jobs.* · verae.archive.put|query|reply.* + Unplanned search/store: verae.search.* or verae.store.* — same worker shape. Zapier never subscribes. + Broadcast queries: no queue group. Work queues: queue group. Bloom miss: no packet. + + diff --git a/diagrams/system.svg b/diagrams/system.svg new file mode 100644 index 0000000..bf48d34 --- /dev/null +++ b/diagrams/system.svg @@ -0,0 +1,53 @@ + + + Verae Time × Zapier — system context + Zapier never speaks NATS. HTTPS stops at middleware. JetStream is the 3-node cluster. + + + Zapier cloud + verae-zapier-app + + zappier-edge + HTTPS :3000 meter + + CS credits + + Sales $ + + QBooks + + verae-middleware + HTTPS :3100 wait + + Verae chain + api.veraetime.net + + HTTPS + + HTTPS + + HTTPS hash + + Central Verae NATS.IO 3-server JetStream cluster + + node A + + node B + + node C + + job-poller + jobs.watch + + webhook-deliver + webhooks.deliver + + WORM × N + archive.query + + tree-node + kind=tree + + JetStream + + diff --git a/diagrams/uptime.svg b/diagrams/uptime.svg new file mode 100644 index 0000000..91cdfee --- /dev/null +++ b/diagrams/uptime.svg @@ -0,0 +1,22 @@ + + + Uptime: floors, restart, another machine + + + healthy replica + counts toward min + + paused / slow RTT + not available → spawn + + /health fail or SSH down + restart or next host + + JetStream replay + consumer catch-up + + tree-node min=3 keepFloor + local + ns1 (marchon@70.88.205.138, ~/.ssh/id_ed25519). Least-loaded placement. Bloom miss = silence so a partitioned archive does not block the bus. + Zapier only sees HTTPS 202 / wait JSON / REST Hook. Cluster majority of 3 NATS servers survives one node loss. + + diff --git a/package.json b/package.json new file mode 100644 index 0000000..9f8a6d6 --- /dev/null +++ b/package.json @@ -0,0 +1,6 @@ +{ + "name": "overview", + "version": "1.0.0", + "private": true, + "description": "High-level overview of the Verae Time × Zapier system, modules, uptime, NATS cluster, and docs index" +} diff --git a/screenshots/console-docs.png b/screenshots/console-docs.png new file mode 100644 index 0000000..975c6a1 Binary files /dev/null and b/screenshots/console-docs.png differ diff --git a/screenshots/console-fleet.png b/screenshots/console-fleet.png new file mode 100644 index 0000000..a599cb8 Binary files /dev/null and b/screenshots/console-fleet.png differ diff --git a/screenshots/console-trace.png b/screenshots/console-trace.png new file mode 100644 index 0000000..9915079 Binary files /dev/null and b/screenshots/console-trace.png differ