verae-nats-cluster/scripts/ns1-study-methodology.md

4. Study methodology

4.1 Question

On the NS1 test stand, what message throughput and delay does the three-node verae JetStream cluster deliver at several loads, and which part of the stack is the limiter for product traffic (jobs, events, webhooks, archive)?

4.2 Hypotheses (stated before the run)

1. H1 — Core vs JetStream. Fire-and-forget core NATS is at least an order of magnitude faster than JetStream file + replicas=3, because durable publish waits for a majority disk replica.
2. H2 — JetStream parallelism. Adding publishers does not linearly increase JetStream write rate once the replica log is saturated.
3. H3 — Quiet delay. Sequential pub→sub round trip on vmbr1 is well under 1 ms p99 when the consumer is waiting.
4. H4 — Burst delay. If publishers dump a batch before the subscriber drains, observed delay is queueing time, roughly linear in backlog, not in cluster hop count.
5. H5 — Payload. Moving 128 B → 1 KiB lowers message rate and raises byte rate on core NATS; JetStream in this size band stays replica/fsync bound.

4.3 Independent variables (what we changed)

Factor	Levels
Transport	Core NATS pub/sub vs JetStream file replicas=3
Publisher count	1, 2, 4, 8
Subscriber count	0 (JS publish-only), 1, 2, 4, 8
Message count	1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step)
Payload	128 B, 1024 B
Delay mode	ping (publish, wait, repeat) vs flood (publish all, then drain)

4.4 Dependent variables (what we recorded)

Metric	Instrument	Unit
Publish rate	nats bench 0.1.6 Pub stats	msgs/s, MB/s
Subscribe rate	nats bench Sub stats	msgs/s, MB/s
Aggregate	nats bench NATS Pub/Sub stats	msgs/s (fan-out counts both sides)
Publisher spread	nats min/avg/max msgs/s	not delay
One-way-ish RTT	latency.mjs header timestamp	min, avg, p50, p90, p99, max
Host load	/proc/loadavg before and after	load average
Broker counters	http://127.0.0.1:8222/varz inside each nats LXC	connections, in/out msgs, cpu, mem

Important: nats CLI 0.1.6 min/avg/max are rate spread across publishers, not microseconds of delay. Delay is only latency.mjs.

4.5 Controls and constants

• Cluster name verae, three routes, client :4222, cluster :6222, monitor loopback :8222.
• Client URL always the three-node list on vmbr1 (never host 127.0.0.1:4222, never vmbr0).
• Bench client is LXC 510, not a nats-* server.
• JetStream bench stream name benchstream, file storage, replicas=3, deleted between JS loads (nats stream rm --force) so names do not collide.
• Product streams were not the bench target (no load test on ZAPIER_* / VERAE_ARCHIVE).
• No TLS, no nkeys, no account isolation (isolation is vmbr1).
• Same nats CLI version (0.1.6) and nats@2 Node client as the first ladder.

4.6 Procedure

1. Confirm this script is executing on NS1.GEORGELAMBERT.ORG. Refuse otherwise.
2. Snapshot host load, memory, LXC configs, and each nats varz.
3. From NS1, pct exec 510 the core ladder (1p1s, 4p4s, 8p8s at 128 B; 4p4s at 1 KiB).
4. Delete benchstream; JS ladder (1p, 4p, 4p×1 KiB, 2p2s pull) at replicas=3 file.
5. Copy latency.mjs into 510; ping then flood at several batch sizes.
6. Snapshot host/varz again.
7. Parse logs on this host; draw charts; write HTML and PDF on this host.

No publish, subscribe, chart, or PDF process runs on the operator laptop for this study.

4.7 Instrumentation path

[NS1 host 70.88.205.138]
   study-on-ns1.sh  (bash + python3)
        |
        | sudo pct exec 510
        v
[LXC 510 verae-px-worker 10.10.10.20]
   nats bench / node latency.mjs
        |
        | NATS client protocol to
        v
[LXC 511/512/513  10.10.10.21-23 :4222]
   nats-server -js  cluster routes :6222

The hypervisor issues the guest commands. The messages themselves never leave vmbr1.

4.8 Threats to validity

Threat	Effect on numbers
One physical host	Three “replicas” share CPU, memory, and usually the same datastore. This measures process/LXC HA, not disk HA.
Shared load	NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs. Load average during a run is part of the result, not noise to ignore.
Single bench client	All publishers live in 510. Per-publisher rate spread is contention in that guest.
Short runs	Seconds of traffic. No compaction, no multi-hour page-cache eviction, no snapshot during load.
No TLS/nkeys	Production auth will cost CPU. Do not treat these rates as post-nkeys rates.
Fan-out aggregate	Core aggregate msgs/s counts pub+sub. Do not compare that column to JetStream unique writes.
Flood ≠ RTT	Mixing flood averages with ping p99 produces a fake “NATS is slow” story.
Lab only	Not a Zapier HTTPS bench and not live api.veraetime.net.

4.9 Ethics / safety

Bench uses throwaway subjects (bench.core.*, bench.js.*, bench.lat.*) and a throwaway stream. It does not purge product streams. Zapier cloud has no NATS socket.

---

5. Suggestions for fine-tuning

These follow from the method and from the first ladder on this stand (JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of ms). Apply in order of leverage. Re-run this NS1 study after each change so the delta is measured the same way.

5.1 Treat JetStream as the product limiter

Product jobs/events/webhooks/archive are durable. Tuning core NATS to 2M msgs/s will not move a timestamp Zap. Put effort into replica write path and consumer lag, not core fan-out.

5.2 Split storage class by stream

Stream	Suggested store	Why
ZAPIER_JOBS	file, r=3	Work queue; lose-a-job is bad
ZAPIER_EVENTS	file r=3, or memory r=3 if events are rebuildable from job status	Hot waiters; measure both
ZAPIER_WEBHOOKS	file, r=3, workqueue	HTTPS to Zapier is the slow consumer
ZAPIER_USAGE	file, r=3, limits + max-age	Telemetry
VERAE_ARCHIVE	file, r=3, on the best disk	Puts are larger and must survive

Try ZAPIER_EVENTS as memory store in a maintenance window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms and durable events still ack at a higher rate, keep it; if a CT restart drops in-flight waiters, revert.

5.3 Give JetStream real disks

Today r=3 on three LXC guests on one Proxmox host is three files, one failure domain.

• Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev) into each nats LXC store_dir.
• Set sync: always only on archive if you need it; default sync is often enough for jobs and is faster. Measure.
• Do not put JetStream store_dir on the same busy rootfs as Forgejo/Caddy if we can avoid it.
• When moving to three metal boxes: same configs, private NIC, one disk (or mirror) per node. That is the first change that makes r=3 mean “two boxes can die.”

5.4 Isolate the nats CTs from the rest of NS1

Host load on this box is often already several. Pin:

• nats-a/b/c: dedicated cores, no steal from keep/fleet Node processes.
• Memory high enough that file-backed streams stay cache-hot for the working set.
• cpuunits / cpuset in pct config so a Zapier-facing Node GC pause does not stall fsync.

Re-run this study after pinning; H1/H2 should move more than ping.

5.5 Consumer and mailbox tuning (delay H4)

Flood delay is backlog / consume_rate. Fine-tune the waiters, not the broker RTT.

• jobs.events and webhooks.deliver: raise max_ack_pending so a slow HTTPS hook does not stall the whole consumer; cap it so a poison message cannot unbounded-buffer RAM.
• Pull consumers: larger batch, shorter expires, more pullers horizontally (fleet replica floors) instead of one fat subscriber.
• Middleware should not flood-publish then wait; it already does per-job publish. Keep that. The flood test is the outage profile when a consumer is stopped.
• Alert on consumer lag (pending + ack pending) from JetStream, not on ping RTT.

5.6 Publisher-side batching in middleware

A timestamp job is one small JSON. 16k msgs/s is ample. Still:

• Avoid per-byte publishes; one message per job/event.
• Reuse NATS connections (connection churn showed up as publisher spread in the core 4p/8p runs).
• Idempotent msg id / duplicate window sized to Verae retry window, not default-only.

5.7 nats-server knobs worth measuring (A/B with this script)

Knob	Why try it
max_payload	Keep default unless archive puts grow
write_deadline	Slow consumer protection for webhooks
max_pending	Bound memory on a stuck Zapier hook
max_connections	Fleet workers + keep + middleware
JetStream max_file_store / max_memory_store	Prevent one stream from filling the CT
max_outstanding_catchup	Replica restart after a nats-c blip
GOMAXPROCS = LXC cores	Do not overthread a 2-core CT

Change one knob, re-run study-on-ns1.sh, compare JetStream 1p 128 B and ping p99.

5.8 Network

• Keep NATS off vmbr0. No change.
• When on metal: dedicated NIC or VLAN for cluster :6222 vs client :4222 if possible (replication vs client load).
• Check virtio queue counts on the LXC nics if core 1 KiB byte rate plateaus.

5.9 Security cost (when nkeys/mTLS flip)

verae-nats-accounts is still a sketch. Enabling accounts will add CPU on publish. Budget: re-run this exact study after creds are in every NATS_URL, and accept a drop on both core and JS. Do not flip without that measurement.

5.10 Operational fine-tuning (lag, not peak msgs/s)

1. Scrape varz / jsz from the host over vmbr1 (not public). Monitor loopback :8222 is invisible to Prometheus on NS1 unless we add a host-side proxy on 10.10.10.21:8222 bound only to vmbr1.
2. Keep replica floors for webhook-deliver and job-poller — they are the flood defense.
3. Backup/restore drill of JetStream during idle, then a short JS 1p run to see catchup cost.
4. A 15–30 minute soak (not in this ladder) for page cache and compaction; add that as a third study when disks are dedicated.

5.11 What not to tune

• Do not chase core 8p8s aggregate. It is fan-out on a lab bridge.
• Do not treat flood 400 ms as “cluster RTT.” Fix consumers.
• Do not load-test on ZAPIER_* streams.
• Do not bind client NATS to 0.0.0.0 on vmbr0.

5.12 Recommended next experiments (same method, one change each)

1. CPU pin nats-a/b/c → re-run JS 1p + ping.
2. ZAPIER_EVENTS-shaped memory stream vs file (throwaway stream, same flags as this JS ladder).
3. Distinct store_dir disks per node.
4. nkeys on, same ladder.
5. Three hardware boxes, same cluster.env IPs updated.

Each experiment should produce a new results/<utc>/ on NS1 and a new progress-repo report so we can diff H1–H5 instead of arguing from memory.