verae-nats-cluster/results/20260912T053120Z/BENCH.md

NATS cluster message speed

Run 20260912T053120Z (UTC). Client: LXC 510 verae-px-worker (10.10.10.20), not a nats-* server. Servers: nats-a/b/c on 10.10.10.21–23 (vmbr1 only).

Client URL:

nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222

Method

• Core NATS is fire-and-forget pub/sub (nats bench). No disk, no replica ack.
• JetStream uses file storage and replicas=3 (same as product streams). The unique stream benchstream is deleted between JS loads.
• Throughput is msgs/sec from nats CLI 0.1.6 (--no-progress --csv). Its min/avg/max are publisher rate spread, not delay.
• Ping delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster.
• Flood delay: N publishers dump the whole batch, then the subscriber drains. This is queueing under burst, not wire RTT.
• Probe: scripts/latency.mjs (two connections, header timestamp).

Throughput

Run	Mode	Aggregate msgs/s	Pub msgs/s	Pub MB/s	Sub msgs/s	Sub MB/s
core-1p1s-50k-128	core pub/sub	810,988	599,004	73.12	456,116	55.68
core-4p4s-100k-128	core pub/sub	1,361,921	379,346	46.31	1,150,536	140.45
core-4p4s-50k-1k	core pub/sub	695,192	175,725	171.61	584,024	570.34
core-8p8s-200k-128	core pub/sub	1,998,733	283,259	34.58	1,780,589	217.36
js-1p-20k-128-r3	jetstream r=3 file	—	17,388	2.12	—	—
js-2p2s-20k-128-r3	jetstream r=3 file	19,876	9,959	1.22	9,942	1.21
js-4p-20k-1k-r3	jetstream r=3 file	—	18,114	17.69	—	—
js-4p-50k-128-r3	jetstream r=3 file	—	22,853	2.79	—	—
js-mem-1p-20k-128-r3	jetstream r=3 file	—	22,153	2.70	—	—
js-mem-4p-20k-1k-r3	jetstream r=3 file	—	28,685	28.01	—	—
js-mem-4p-50k-128-r3	jetstream r=3 file	—	36,355	4.44	—	—

Round-trip delay

Run	Kind	Count	Pubs	Size	min	avg	p50	p90	p99	max
lat-ping-1k-128	ping (sequential RTT)	1000	1	128 B	0.250ms	0.319ms	0.299ms	0.363ms	0.684ms	2.759ms
lat-1p-5k-128	flood (burst queueing)	5000	1	128 B	94.920ms	132.026ms	131.990ms	156.086ms	158.860ms	158.965ms
lat-4p-5k-1k	flood (burst queueing)	5000	4	1024 B	113.281ms	132.425ms	131.457ms	140.317ms	144.683ms	145.365ms
lat-4p-10k-128	flood (burst queueing)	10000	4	128 B	153.318ms	204.329ms	205.865ms	234.151ms	237.346ms	237.477ms
lat-8p-20k-128	flood (burst queueing)	20000	8	128 B	233.727ms	310.924ms	311.170ms	373.437ms	400.764ms	402.811ms

What the numbers mean

Product job/event/archive traffic is JetStream r=3 file. On this three-LXC stand that is about 16k durable 128 B pubs/s (about 13k at 1 KiB). Core NATS is an upper bound for non-durable fan-out: about 0.7–2.0M msgs/s aggregate at 128 B, or ~630k msgs/s (~616 MB/s) at 1 KiB with 4 publishers.

A quiet request-reply is ~0.3 ms average, p99 < 1 ms. Flood rows in the 150–500 ms band are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers.

Re-run on NS1: bash scripts/bench.sh. Raw logs/CSVs are under results/<utc>/.