overview/05-network-failures.md

5. Local network failures

“Local network” means the operator LAN, SSH to NS1, or a partitioned archive — not Zapier’s cloud.

Failure	What happens	What the customer sees
NATS node unreachable	Client reconnects to another cluster URL; JetStream consumers resume	Wait may return pending; hook still fires later
All NATS down	Middleware cannot publish jobs.watch; fleet marks workers unhealthy	503 / pending; no NATS leak to Zapier
One WORM / tree node partitioned	Bloom miss = no packet; aggregator uses whoever answered	Lookup may miss attachments until the node returns; seal on chain still valid
SSH to a fleet host fails	Spawn fails; fleet places the next replica on another machine	Floor still met if capacity remains on local or another SSH host
Chain api.veraetime.net timeout	Poller retries; then timestamp.timeout event	Wait → pending or failed; async + hook still the recovery path
zappier-edge 402	QuotaExceeded with upgrade URL	Zap step error; no NATS involved
Tunnel to loopback NATS dropped	NATS_URL=nats://127.0.0.1:14222 dies; restart nats-tunnel.sh	Workers on NS1 itself still see 127.0.0.1:4222

Design choices that make partitions survivable

1. Archive query is broadcast, not a shared queue group — a dead node does not steal the message.
2. Bloom miss is silence — missing nodes do not send empty errors that look like “hash unknown”.
3. Zero replies + known puts is an outage, not a miss (simulator / fleet monitors flag this).
4. NATS is not on the public NIC — a WAN blip does not expose 4222.
5. Idempotent seals — retrying a Zap after a network error will not double-timestamp.

See fleet RTT (min / avg / p50 / p90) when planning extra tree nodes after a flaky path.