49 lines
3.3 KiB
Markdown
49 lines
3.3 KiB
Markdown
# NATS cluster message speed
|
||
|
||
Run **`20260912T045131Z`** (UTC). Client: LXC **510** `verae-px-worker` (`10.10.10.20`), not a nats-* server. Servers: `nats-a/b/c` on `10.10.10.21–23` (`vmbr1` only).
|
||
|
||
Client URL:
|
||
|
||
```text
|
||
nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222
|
||
```
|
||
|
||
## Method
|
||
|
||
- **Core NATS** is fire-and-forget pub/sub (`nats bench`). No disk, no replica ack.
|
||
- **JetStream** uses **file** storage and **replicas=3** (same as product streams). The unique stream `benchstream` is deleted between JS loads.
|
||
- Throughput is **msgs/sec** from nats CLI **0.1.6** (`--no-progress --csv`). Its min/avg/max are publisher **rate spread**, not delay.
|
||
- **Ping** delay: one publisher, sequential publish-then-wait. This is one-message round-trip through the cluster.
|
||
- **Flood** delay: N publishers dump the whole batch, then the subscriber drains. This is **queueing under burst**, not wire RTT.
|
||
- Probe: `scripts/latency.mjs` (two connections, header timestamp).
|
||
|
||
## Throughput
|
||
|
||
| Run | Mode | Aggregate msgs/s | Pub msgs/s | Pub MB/s | Sub msgs/s | Sub MB/s |
|
||
|-----|------|------------------|------------|----------|------------|----------|
|
||
| `core-1p1s-50k-128` | core pub/sub | 1,200,836 | 791,094 | 96.57 | 747,461 | 91.24 |
|
||
| `core-4p4s-100k-128` | core pub/sub | 1,521,256 | 316,312 | 38.61 | 1,299,634 | 158.65 |
|
||
| `core-4p4s-50k-1k` | core pub/sub | 630,460 | 247,747 | 241.94 | 510,216 | 498.26 |
|
||
| `core-8p8s-200k-128` | core pub/sub | 2,007,937 | 333,957 | 40.77 | 1,790,736 | 218.60 |
|
||
| `js-1p-20k-128-r3` | jetstream r=3 file | — | 16,155 | 1.97 | — | — |
|
||
| `js-2p2s-20k-128-r3` | jetstream r=3 file | 21,861 | 10,965 | 1.34 | 10,942 | 1.34 |
|
||
| `js-4p-20k-1k-r3` | jetstream r=3 file | — | 13,493 | 13.18 | — | — |
|
||
| `js-4p-50k-128-r3` | jetstream r=3 file | — | 16,607 | 2.03 | — | — |
|
||
|
||
## Round-trip delay
|
||
|
||
| Run | Kind | Count | Pubs | Size | min | avg | p50 | p90 | p99 | max |
|
||
|-----|------|-------|------|------|-----|-----|-----|-----|-----|-----|
|
||
| `lat-ping-1k-128` | ping (sequential RTT) | 1000 | 1 | 128 B | 0.254ms | 0.307ms | 0.286ms | 0.332ms | 0.734ms | 2.763ms |
|
||
| `lat-1p-5k-128` | flood (burst queueing) | 5000 | 1 | 128 B | 149.279ms | 238.626ms | 248.752ms | 274.314ms | 279.398ms | 279.740ms |
|
||
| `lat-4p-5k-1k` | flood (burst queueing) | 5000 | 4 | 1024 B | 155.101ms | 211.706ms | 217.579ms | 223.268ms | 227.798ms | 228.370ms |
|
||
| `lat-4p-10k-128` | flood (burst queueing) | 10000 | 4 | 128 B | 174.174ms | 263.186ms | 266.672ms | 299.073ms | 304.233ms | 304.475ms |
|
||
| `lat-8p-20k-128` | flood (burst queueing) | 20000 | 8 | 128 B | 304.586ms | 453.749ms | 466.296ms | 499.924ms | 505.112ms | 505.620ms |
|
||
|
||
## What the numbers mean
|
||
|
||
Product job/event/archive traffic is **JetStream r=3 file**. On this three-LXC stand that is about **16k durable 128 B pubs/s** (about **13k** at 1 KiB). Core NATS is an upper bound for non-durable fan-out: about **0.7–2.0M msgs/s** aggregate at 128 B, or **~630k msgs/s (~616 MB/s)** at 1 KiB with 4 publishers.
|
||
|
||
A quiet request-reply is **~0.3 ms** average, **p99 < 1 ms**. Flood rows in the **150–500 ms** band are the subscriber catching up after a burst, which is what a job-events mailbox sees if publishers outrun consumers.
|
||
|
||
Re-run on NS1: `bash scripts/bench.sh`. Raw logs/CSVs are under `results/<utc>/`.
|