George Lambert
a7a7ec86ce
Some checks are pending
offline / test (push) Waiting to run
Same NS1 ladder as 20260912T051237Z. Product streams restored to ZFS; raised cores, RAM, and max_mem stay on 510–513.
145 lines
8.7 KiB
Markdown
145 lines
8.7 KiB
Markdown
# Current status — 2026-09-12
|
||
|
||
Verae Time × Zapier is a **working lab** on one Proxmox host (**NS1.GEORGELAMBERT.ORG**, `70.88.205.138`). Public HTTPS doors, catalog, keep/fleet, and a **test 3-node NATS cluster** on separate LXC guests are up. The test environment **points at that cluster**. Zapier Platform `push` and live `api.veraetime.net` are still waiting on operator login / bind credentials.
|
||
|
||
**Progress repo:** this file in [zapier-decisions](https://git.georgelambert.org/marchon/zapier-decisions).
|
||
**Catalog:** https://zapier.georgelambert.org/packages/zapier-decisions/STATUS.pdf
|
||
**Monorepo:** `master-zapier-plan-draft` (`main` / `master`).
|
||
|
||
---
|
||
|
||
## Live right now
|
||
|
||
| Layer | State |
|
||
|-------|--------|
|
||
| Catalog | https://zapier.georgelambert.org/ (PDF links path-relative; module/model books work) |
|
||
| Public doors | portal, api, zap, iam, mw, fleet, git — last off-box GET 8/8 OK |
|
||
| Keep + fleet | Running. systemd + linger **enabled** for next reboot (current processes were not double-started as systemd) |
|
||
| Test NATS | **3-node JetStream on `vmbr1`:** LXC 511/512/513 `nats-a/b/c` at `10.10.10.21–23`. Cluster name `verae`, 2 routes each. |
|
||
| Product streams | `ZAPIER_JOBS`, `ZAPIER_EVENTS`, `ZAPIER_WEBHOOKS`, `ZAPIER_USAGE`, `VERAE_ARCHIVE` — **replicas=3** |
|
||
| Modules on that cluster | Fleet, keep, middleware (`natsConnected: true`), billing, access planes. `NATS_URL` lists all three URLs. |
|
||
| Host loopback `:4222` | Still listening; **clients no longer use it** |
|
||
| Worker CT **510** `verae-px-worker` | `10.10.10.20` on `vmbr1`. Keep worm+tree **200** on the cluster |
|
||
| Backup | Timer `verae-backup.timer` enabled; snapshots under `/SSD2/backups/verae/` |
|
||
| Uptime | Loop on px-worker (`:3870/status`), last tick `failed: 0` |
|
||
|
||
Client URL (private `vmbr1` only):
|
||
|
||
```text
|
||
nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
|
||
```
|
||
|
||
Repo: https://git.georgelambert.org/marchon/verae-nats-cluster
|
||
|
||
### Cluster message speed (`20260912T045131Z`)
|
||
|
||
Measured from LXC **510** (not a nats-* server) against `10.10.10.21–23`.
|
||
|
||
**Full report (charts + explanations):** [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/nats-cluster-bench.pdf) · [markdown](reports/nats-cluster-bench.md)
|
||
|
||
**Second study (all code on NS1.GEORGELAMBERT.ORG / 70.88.205.138, run `20260912T051237Z`):** [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-host/nats-cluster-bench-ns1.pdf) · [markdown](reports/ns1-host/nats-cluster-bench-ns1.md). Orchestrator, nats bench, charts, HTML, and PDF ran on 138; traffic was LXC 510 → 511–513 on `vmbr1`. Includes full methodology and fine-tuning notes. Host load ~8.5/40 during the run; JetStream 1p 128 B was **7.4k** durable pubs/s (vs ~16k on the quieter first ladder); ping p99 **1.38 ms**.
|
||
|
||
**Maximized study (`20260912T053120Z`):** 8 cores + 16 GiB on 510–513, `max_mem: 8G`, JetStream **tmpfs** for the ladder (ZFS restored after). veth/10G unchanged. [HTML](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.html) · [PDF](https://zapier.georgelambert.org/packages/zapier-decisions/reports/ns1-maximize/nats-cluster-bench-ns1.pdf).
|
||
|
||
| Metric | Baseline 051237Z | Maximized | Ratio |
|
||
|--------|------------------|-----------|-------|
|
||
| JS file r=3 1p 128 B | 7,393 pubs/s | **17,388** | **2.35×** |
|
||
| JS memory r=3 4p 128 B | — | **36,355** | ~4.9× vs baseline file 1p |
|
||
| Core 1p 128 B pub | 502,502 | 599,004 | 1.19× |
|
||
| Ping p99 | 1.377 ms | **0.684 ms** | **2.01× faster** |
|
||
|
||
Live cluster **keeps 8 cores / 16 GiB / max_mem 8G**. Product streams are back on ZFS (tmpfs was study-only).
|
||
|
||
Short tables: [BENCH.md](https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md). Re-run: `bash scripts/bench.sh` on NS1.
|
||
|
||
| Kind | Load | Result |
|
||
|------|------|--------|
|
||
| Core NATS | 1p1s 50k×128 B | ~1.20M msgs/s aggregate |
|
||
| Core NATS | 8p8s 200k×128 B | ~2.01M msgs/s aggregate |
|
||
| Core NATS | 4p4s 50k×1 KiB | ~630k msgs/s (~616 MB/s) |
|
||
| JetStream file r=3 | 1–4 pubs, 128 B | ~16k durable pubs/s |
|
||
| JetStream file r=3 | 4 pubs, 1 KiB | ~13.5k durable pubs/s |
|
||
| Ping (sequential RTT) | 1k×128 B | avg 0.307 ms, p99 0.734 ms |
|
||
| Flood (burst queueing) | 8p 20k×128 B | avg 454 ms |
|
||
|
||
Product job/event/archive traffic is the JetStream row. Ping is one-message delay; flood is the mailbox catching up after a burst.
|
||
|
||
---
|
||
|
||
## Built and checked in (Forgejo, each its own repo)
|
||
|
||
IAM, portal on access-web, keep/watch/guard, fleet floors, host-deps, catalog host-move docs, plus:
|
||
|
||
| Repo | Job |
|
||
|------|-----|
|
||
| verae-uptime | Off-box HTTPS GET of public doors |
|
||
| verae-backup | SQLite / keep / worm-tree tar; restore dry-run default |
|
||
| verae-deploy | Tagged checkout + npm ci on the target OS |
|
||
| verae-proxmox | LXC 510 worker provision |
|
||
| verae-nats-cluster | LXC 511–513 JetStream cluster |
|
||
|
||
---
|
||
|
||
## Still open
|
||
|
||
| Item | Notes |
|
||
|------|--------|
|
||
| Zapier Platform `push` | Needs Zapier login (see below) |
|
||
| `MOCK_VERAE=false` | Live `api.veraetime.net` + admin bind credentials (see below) |
|
||
| NATS nkeys/mTLS | Cluster is up **without** accounts so existing `NATS_URL`s work. `verae-nats-accounts` remains the sketch. Flip only after every client has credentials. Isolation today is `vmbr1`, not nkeys. |
|
||
| lan-134 | Still disabled (SSH not working) |
|
||
| Reboot test | systemd should own keep/fleet after reboot; do not also `nohup guard.sh` |
|
||
| Hardware NATS | Same three configs, three boxes, private NIC — change IPs in `cluster.env` |
|
||
|
||
---
|
||
|
||
## Once we have a Zapier login
|
||
|
||
Zapier cloud only talks **HTTPS** to our Zap plane (`https://zap.zapier.georgelambert.org`). No NATS, no `api.veraetime.net`.
|
||
|
||
1. `zapier login` (or `zapier-platform login`).
|
||
2. In `packages/verae-zapier` (and `verae-activate` if the tiny app is still wanted):
|
||
- `zapier-platform register` (or attach an existing app id).
|
||
- `MIDDLEWARE_BASE_URL=https://zap.zapier.georgelambert.org` (or `https://mw.zapier.georgelambert.org` if skipping the Zap plane).
|
||
- `zapier-platform build` and `validate`.
|
||
3. `zapier-platform push` as a **private** app.
|
||
4. Invite the workspace; create a Zap:
|
||
- Auth = portal **API key** (`zmw_…`), not a Verae JWT.
|
||
- Test **Create Timestamp and Wait**, **Create Timestamp** (async `jobId`), **Find Job Status**, **Timestamp Completed** REST Hook.
|
||
5. Fix anything `validate` / `push` flags (auth labels, samples, 402/403 mapping).
|
||
6. When the contract is stable, bump the platform version and invite a real user.
|
||
|
||
Do **not** put NATS URLs or Verae passwords into Zapier.
|
||
|
||
---
|
||
|
||
## Once `MOCK_VERAE=false` and live `api.veraetime.net`
|
||
|
||
Middleware already has the live HTTP client. Still needed:
|
||
|
||
1. `VERAE_API_BASE_URL=https://api.veraetime.net` and working **login** (username/password → JWT). JWT stays in middleware (`tokenRef`); Zapier never sees it.
|
||
2. Confirm OpenAPI vs our client:
|
||
- `POST /api/timestamp` → **202** `{ jobId }`
|
||
- `GET /api/status/{jobId}` until `completed` / `failed`
|
||
- `POST /api/verify`
|
||
- Batch create/status if used
|
||
3. **Hash lookup is not on the live OpenAPI** — `veraeClient.lookupHash` returns **501** when not mocking. Central-chain “find by SHA-256” needs a Verae API gap fill or our own store.
|
||
4. Map live status JSON into the receipt shape the Zap app expects (`certificate`, `timestamp`, `blockIndex`, `sha256`).
|
||
5. Token refresh, 401 retry, and 402/403 mapping against real errors.
|
||
6. One live seal in the lab, then REST Hook delivery to `hooks.zapier.com`.
|
||
|
||
---
|
||
|
||
## Submit → jobId → wait → receipt
|
||
|
||
**Yes — that is the designed path**, and it already runs in **mock**.
|
||
|
||
1. Zap (or curl) `POST /zapier/v1/timestamp` or `/timestamp/wait`.
|
||
2. Middleware `createTimestamp` calls Verae `POST /api/timestamp` (or the mock) and gets **`jobId`**.
|
||
3. It publishes **`verae.zapier.jobs.watch`** on NATS (the 3-node cluster).
|
||
4. **Wait:** subscribe to **`verae.zapier.jobs.events`** for that `jobId`, or poll `GET /api/status/{jobId}`.
|
||
5. Job-poller hits Verae status until `completed`.
|
||
6. Event `timestamp.completed` carries status; webhook-deliver POSTs the Zapier REST Hook.
|
||
7. Response includes **certificate / timestamp / blockIndex** (mock uses `mock-cert-{jobId}`, `blockIndex: 42`). Async create returns `{ jobId, sha256, existing }` immediately; wait returns the finished status. Same SHA-256 is idempotent (`existing: true`).
|
||
|
||
If wait times out, middleware returns **`pending` + `jobId`** so the hook can finish the Zap. Live Verae must match that HTTP contract; we have **not** proven a real chain seal yet (`MOCK_VERAE` is still true).
|