George Lambert
a32c91475c
Some checks are pending
offline / test (push) Waiting to run
High-level system map with diagrams, TOC, and a docs index. Template worker shows how to add a new verae.* address for search, storage, or unplanned functions without teaching Zapier NATS.
1.8 KiB
1.8 KiB
5. Local network failures
“Local network” means the operator LAN, SSH to NS1, or a partitioned archive — not Zapier’s cloud.
| Failure | What happens | What the customer sees |
|---|---|---|
| NATS node unreachable | Client reconnects to another cluster URL; JetStream consumers resume | Wait may return pending; hook still fires later |
| All NATS down | Middleware cannot publish jobs.watch; fleet marks workers unhealthy |
503 / pending; no NATS leak to Zapier |
| One WORM / tree node partitioned | Bloom miss = no packet; aggregator uses whoever answered | Lookup may miss attachments until the node returns; seal on chain still valid |
| SSH to a fleet host fails | Spawn fails; fleet places the next replica on another machine | Floor still met if capacity remains on local or another SSH host |
Chain api.veraetime.net timeout |
Poller retries; then timestamp.timeout event |
Wait → pending or failed; async + hook still the recovery path |
| zappier-edge 402 | QuotaExceeded with upgrade URL | Zap step error; no NATS involved |
| Tunnel to loopback NATS dropped | NATS_URL=nats://127.0.0.1:14222 dies; restart nats-tunnel.sh |
Workers on NS1 itself still see 127.0.0.1:4222 |
Design choices that make partitions survivable
- Archive query is broadcast, not a shared queue group — a dead node does not steal the message.
- Bloom miss is silence — missing nodes do not send empty errors that look like “hash unknown”.
- Zero replies + known puts is an outage, not a miss (simulator / fleet monitors flag this).
- NATS is not on the public NIC — a WAN blip does not expose 4222.
- Idempotent seals — retrying a Zap after a network error will not double-timestamp.
See fleet RTT (min / avg / p50 / p90) when planning extra tree nodes after a flaky path.