George Lambert
fef7590c27
Some checks are pending
offline / test (push) Waiting to run
Client LXC 510 against nats-a/b/c at several core and JetStream r=3 loads.
3.3 KiB
3.3 KiB
NATS cluster message speed
Run 20260912T045131Z (UTC). Client: LXC 510 verae-px-worker (10.10.10.20), not a nats-* server. Servers: nats-a/b/c on 10.10.10.21–23 (vmbr1 only).
Client URL:
nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
Method
- Core NATS is fire-and-forget pub/sub (
nats bench). No disk, no replica ack. - JetStream uses file storage and replicas=3 (same as product streams). The unique stream
benchstreamis deleted between JS loads. - Throughput is msgs/sec from nats CLI 0.1.6 (
--no-progress --csv). Its min/avg/max are publisher rate spread, not delay. - Ping delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster.
- Flood delay: N publishers dump the whole batch, then the subscriber drains. This is queueing under burst, not wire RTT.
- Probe:
scripts/latency.mjs(two connections, header timestamp).
Throughput
| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
|---|---|---|---|---|---|---|
core-1p1s-50k-128 |
core pub/sub | 1,200,836 | 791,094 | 96.57 | 747,461 | 91.24 |
core-4p4s-100k-128 |
core pub/sub | 1,521,256 | 316,312 | 38.61 | 1,299,634 | 158.65 |
core-4p4s-50k-1k |
core pub/sub | 630,460 | 247,747 | 241.94 | 510,216 | 498.26 |
core-8p8s-200k-128 |
core pub/sub | 2,007,937 | 333,957 | 40.77 | 1,790,736 | 218.60 |
js-1p-20k-128-r3 |
jetstream r=3 file | — | 16,155 | 1.97 | — | — |
js-2p2s-20k-128-r3 |
jetstream r=3 file | 21,861 | 10,965 | 1.34 | 10,942 | 1.34 |
js-4p-20k-1k-r3 |
jetstream r=3 file | — | 13,493 | 13.18 | — | — |
js-4p-50k-128-r3 |
jetstream r=3 file | — | 16,607 | 2.03 | — | — |
Round-trip delay
| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
|---|---|---|---|---|---|---|---|---|---|---|
lat-ping-1k-128 |
ping (sequential RTT) | 1000 | 1 | 128 B | 0.254ms | 0.307ms | 0.286ms | 0.332ms | 0.734ms | 2.763ms |
lat-1p-5k-128 |
flood (burst queueing) | 5000 | 1 | 128 B | 149.279ms | 238.626ms | 248.752ms | 274.314ms | 279.398ms | 279.740ms |
lat-4p-5k-1k |
flood (burst queueing) | 5000 | 4 | 1024 B | 155.101ms | 211.706ms | 217.579ms | 223.268ms | 227.798ms | 228.370ms |
lat-4p-10k-128 |
flood (burst queueing) | 10000 | 4 | 128 B | 174.174ms | 263.186ms | 266.672ms | 299.073ms | 304.233ms | 304.475ms |
lat-8p-20k-128 |
flood (burst queueing) | 20000 | 8 | 128 B | 304.586ms | 453.749ms | 466.296ms | 499.924ms | 505.112ms | 505.620ms |
What the numbers mean
Product job/event/archive traffic is JetStream r=3 file. On this three-LXC stand that is about 16k durable 128 B pubs/s (about 13k at 1 KiB). Core NATS is an upper bound for non-durable fan-out: about 0.7–2.0M msgs/s aggregate at 128 B, or ~630k msgs/s (~616 MB/s) at 1 KiB with 4 publishers.
A quiet request-reply is ~0.3 ms average, p99 < 1 ms. Flood rows in the 150–500 ms band are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers.
Re-run on NS1: bash scripts/bench.sh. Raw logs/CSVs are under results/<utc>/.