Initial import of overview from zapier monorepo

05-network-failures.md

@@ -0,0 +1,23 @@
+# 5. Local network failures
+
+“Local network” means the operator LAN, SSH to NS1, or a partitioned archive — not Zapier’s cloud.
+
+| Failure | What happens | What the customer sees |
+|---------|--------------|------------------------|
+| NATS node unreachable | Client reconnects to another cluster URL; JetStream consumers resume | Wait may return `pending`; hook still fires later |
+| All NATS down | Middleware cannot publish `jobs.watch`; fleet marks workers unhealthy | 503 / pending; no NATS leak to Zapier |
+| One WORM / tree node partitioned | Bloom miss = **no packet**; aggregator uses whoever answered | Lookup may miss attachments until the node returns; seal on chain still valid |
+| SSH to a fleet host fails | Spawn fails; fleet **places the next replica on another machine** | Floor still met if capacity remains on `local` or another SSH host |
+| Chain `api.veraetime.net` timeout | Poller retries; then `timestamp.timeout` event | Wait → pending or failed; async + hook still the recovery path |
+| zappier-edge 402 | QuotaExceeded with upgrade URL | Zap step error; no NATS involved |
+| Tunnel to loopback NATS dropped | `NATS_URL=nats://127.0.0.1:14222` dies; restart `nats-tunnel.sh` | Workers on NS1 itself still see `127.0.0.1:4222` |
+
+## Design choices that make partitions survivable
+
+1. **Archive query is broadcast**, not a shared queue group — a dead node does not steal the message.
+2. **Bloom miss is silence** — missing nodes do not send empty errors that look like “hash unknown”.
+3. **Zero replies + known puts** is an outage, not a miss (simulator / fleet monitors flag this).
+4. **NATS is not on the public NIC** — a WAN blip does not expose 4222.
+5. **Idempotent seals** — retrying a Zap after a network error will not double-timestamp.
+
+See fleet RTT (min / avg / p50 / p90) when planning extra tree nodes after a flaky path.