verae-nats-cluster/results/20260912T055851Z/BENCH.md

4 KiB
Raw Blame History

NATS cluster message speed

Run 20260912T055851Z (UTC). Client: LXC 510 verae-px-worker (10.10.10.20), not a nats-* server. Servers: nats-a/b/c on 10.10.10.2123 (vmbr1 only).

Client URL:

nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222

Method

  • Core NATS is fire-and-forget pub/sub (nats bench). No disk, no replica ack.
  • JetStream uses file storage and replicas=3 (same as product streams). The unique stream benchstream is deleted between JS loads.
  • Throughput is msgs/sec from nats CLI 0.1.6 (--no-progress --csv). Its min/avg/max are publisher rate spread, not delay.
  • Ping delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster.
  • Flood delay: N publishers dump the whole batch, then the subscriber drains. This is queueing under burst, not wire RTT.
  • Probe: scripts/latency.mjs (two connections, header timestamp).

Throughput

Run Mode Aggregate msgs/s Pub msgs/s Pub MB/s Sub msgs/s Sub MB/s
core-1p1s-50k-128 core pub/sub 835,602 662,227 80.84 472,239 57.65
core-4p4s-100k-128 core pub/sub 1,597,284 692,109 84.49 1,286,093 156.99
core-4p4s-50k-1k core pub/sub 595,555 189,939 185.49 479,898 468.65
core-8p8s-200k-128 core pub/sub 1,760,599 223,078 27.23 1,570,171 191.67
js-1p-20k-128-r3 jetstream r=3 file 14,330 1.75
js-2p2s-20k-128-r3 jetstream r=3 file 18,449 9,252 1.13 9,228 1.13
js-4p-20k-1k-r3 jetstream r=3 file 15,197 14.84
js-4p-50k-128-r3 jetstream r=3 file 19,232 2.35
js-file-1p-20k-128-r1 jetstream r=3 file 18,888 2.31
js-file-1p-20k-4k-r3 jetstream r=3 file 8,673 33.88
js-file-4p-50k-128-r1 jetstream r=3 file 24,560 3.00
js-mem-1p-20k-128-r1 jetstream r=3 file 29,972 3.66
js-mem-1p-20k-128-r3 jetstream r=3 file 20,188 2.46
js-mem-4p-20k-1k-r3 jetstream r=3 file 33,916 33.12
js-mem-4p-50k-128-r1 jetstream r=3 file 64,923 7.93
js-mem-4p-50k-128-r3 jetstream r=3 file 37,736 4.61

Round-trip delay

Run Kind Count Pubs Size min avg p50 p90 p99 max
lat-ping-1k-128 ping (sequential RTT) 1000 1 128 B 0.340ms 0.530ms 0.456ms 0.827ms 1.140ms 2.910ms
lat-reconnect-200-128 flood (burst queueing) 200 1 128 B 0.366ms 0.540ms 0.503ms 0.619ms 1.750ms 3.206ms
lat-1p-5k-128 flood (burst queueing) 5000 1 128 B 98.399ms 124.452ms 125.413ms 140.165ms 145.770ms 146.002ms
lat-4p-5k-1k flood (burst queueing) 5000 4 1024 B 125.336ms 157.915ms 157.392ms 176.826ms 178.015ms 178.776ms
lat-4p-10k-128 flood (burst queueing) 10000 4 128 B 157.463ms 205.870ms 207.044ms 237.035ms 239.449ms 240.473ms
lat-8p-20k-128 flood (burst queueing) 20000 8 128 B 230.898ms 361.458ms 372.512ms 443.799ms 458.198ms 460.270ms

What the numbers mean

Product job/event/archive traffic is JetStream r=3 file. On this three-LXC stand that is about 16k durable 128 B pubs/s (about 13k at 1 KiB). Core NATS is an upper bound for non-durable fan-out: about 0.72.0M msgs/s aggregate at 128 B, or ~630k msgs/s (~616 MB/s) at 1 KiB with 4 publishers.

A quiet request-reply is ~0.3 ms average, p99 < 1 ms. Flood rows in the 150500 ms band are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers.

Re-run on NS1: bash scripts/bench.sh. Raw logs/CSVs are under results/<utc>/.