George Lambert
a32c91475c
Some checks are pending
offline / test (push) Waiting to run
High-level system map with diagrams, TOC, and a docs index. Template worker shows how to add a new verae.* address for search, storage, or unplanned functions without teaching Zapier NATS.
23 lines
1.8 KiB
Markdown
23 lines
1.8 KiB
Markdown
# 5. Local network failures
|
||
|
||
“Local network” means the operator LAN, SSH to NS1, or a partitioned archive — not Zapier’s cloud.
|
||
|
||
| Failure | What happens | What the customer sees |
|
||
|---------|--------------|------------------------|
|
||
| NATS node unreachable | Client reconnects to another cluster URL; JetStream consumers resume | Wait may return `pending`; hook still fires later |
|
||
| All NATS down | Middleware cannot publish `jobs.watch`; fleet marks workers unhealthy | 503 / pending; no NATS leak to Zapier |
|
||
| One WORM / tree node partitioned | Bloom miss = **no packet**; aggregator uses whoever answered | Lookup may miss attachments until the node returns; seal on chain still valid |
|
||
| SSH to a fleet host fails | Spawn fails; fleet **places the next replica on another machine** | Floor still met if capacity remains on `local` or another SSH host |
|
||
| Chain `api.veraetime.net` timeout | Poller retries; then `timestamp.timeout` event | Wait → pending or failed; async + hook still the recovery path |
|
||
| zappier-edge 402 | QuotaExceeded with upgrade URL | Zap step error; no NATS involved |
|
||
| Tunnel to loopback NATS dropped | `NATS_URL=nats://127.0.0.1:14222` dies; restart `nats-tunnel.sh` | Workers on NS1 itself still see `127.0.0.1:4222` |
|
||
|
||
## Design choices that make partitions survivable
|
||
|
||
1. **Archive query is broadcast**, not a shared queue group — a dead node does not steal the message.
|
||
2. **Bloom miss is silence** — missing nodes do not send empty errors that look like “hash unknown”.
|
||
3. **Zero replies + known puts** is an outage, not a miss (simulator / fleet monitors flag this).
|
||
4. **NATS is not on the public NIC** — a WAN blip does not expose 4222.
|
||
5. **Idempotent seals** — retrying a Zap after a network error will not double-timestamp.
|
||
|
||
See fleet RTT (min / avg / p50 / p90) when planning extra tree nodes after a flaky path.
|