George Lambert
de20a42109
Some checks are pending
offline / test (push) Waiting to run
Progress-repo write-up of run 20260912T045131Z with throughput and delay charts, method, and product-capacity notes; catalog extras point at it.
1278 lines
38 KiB
HTML
1278 lines
38 KiB
HTML
<!DOCTYPE html>
|
||
<html xmlns="http://www.w3.org/1999/xhtml">
|
||
<head>
|
||
<meta charset="utf-8" />
|
||
<meta name="generator" content="pandoc" />
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
|
||
<title>NATS cluster message speed</title>
|
||
<style>
|
||
/* Default styles provided by pandoc.
|
||
** See https://pandoc.org/MANUAL.html#variables-for-html for config info.
|
||
*/
|
||
html {
|
||
color: #1a1a1a;
|
||
background-color: #fdfdfd;
|
||
}
|
||
body {
|
||
margin: 0 auto;
|
||
max-width: 36em;
|
||
padding-left: 50px;
|
||
padding-right: 50px;
|
||
padding-top: 50px;
|
||
padding-bottom: 50px;
|
||
hyphens: auto;
|
||
overflow-wrap: break-word;
|
||
text-rendering: optimizeLegibility;
|
||
font-kerning: normal;
|
||
}
|
||
@media (max-width: 600px) {
|
||
body {
|
||
font-size: 0.9em;
|
||
padding: 12px;
|
||
}
|
||
h1 {
|
||
font-size: 1.8em;
|
||
}
|
||
}
|
||
@media print {
|
||
html {
|
||
background-color: white;
|
||
}
|
||
body {
|
||
background-color: transparent;
|
||
color: black;
|
||
font-size: 12pt;
|
||
}
|
||
p, h2, h3 {
|
||
orphans: 3;
|
||
widows: 3;
|
||
}
|
||
h2, h3, h4 {
|
||
page-break-after: avoid;
|
||
}
|
||
}
|
||
p {
|
||
margin: 1em 0;
|
||
}
|
||
a {
|
||
color: #1a1a1a;
|
||
}
|
||
a:visited {
|
||
color: #1a1a1a;
|
||
}
|
||
img {
|
||
max-width: 100%;
|
||
}
|
||
svg {
|
||
height: auto;
|
||
max-width: 100%;
|
||
}
|
||
h1, h2, h3, h4, h5, h6 {
|
||
margin-top: 1.4em;
|
||
}
|
||
h5, h6 {
|
||
font-size: 1em;
|
||
font-style: italic;
|
||
}
|
||
h6 {
|
||
font-weight: normal;
|
||
}
|
||
ol, ul {
|
||
padding-left: 1.7em;
|
||
margin-top: 1em;
|
||
}
|
||
li > ol, li > ul {
|
||
margin-top: 0;
|
||
}
|
||
blockquote {
|
||
margin: 1em 0 1em 1.7em;
|
||
padding-left: 1em;
|
||
border-left: 2px solid #e6e6e6;
|
||
color: #606060;
|
||
}
|
||
code {
|
||
font-family: Menlo, Monaco, Consolas, 'Lucida Console', monospace;
|
||
font-size: 85%;
|
||
margin: 0;
|
||
hyphens: manual;
|
||
}
|
||
pre {
|
||
margin: 1em 0;
|
||
overflow: auto;
|
||
}
|
||
pre code {
|
||
padding: 0;
|
||
overflow: visible;
|
||
overflow-wrap: normal;
|
||
}
|
||
.sourceCode {
|
||
background-color: transparent;
|
||
overflow: visible;
|
||
}
|
||
hr {
|
||
border: none;
|
||
border-top: 1px solid #1a1a1a;
|
||
height: 1px;
|
||
margin: 1em 0;
|
||
}
|
||
table {
|
||
margin: 1em 0;
|
||
border-collapse: collapse;
|
||
width: 100%;
|
||
overflow-x: auto;
|
||
display: block;
|
||
font-variant-numeric: lining-nums tabular-nums;
|
||
}
|
||
table caption {
|
||
margin-bottom: 0.75em;
|
||
}
|
||
tbody {
|
||
margin-top: 0.5em;
|
||
border-top: 1px solid #1a1a1a;
|
||
border-bottom: 1px solid #1a1a1a;
|
||
}
|
||
th {
|
||
border-top: 1px solid #1a1a1a;
|
||
padding: 0.25em 0.5em 0.25em 0.5em;
|
||
}
|
||
td {
|
||
padding: 0.125em 0.5em 0.25em 0.5em;
|
||
}
|
||
header {
|
||
margin-bottom: 4em;
|
||
text-align: center;
|
||
}
|
||
#TOC li {
|
||
list-style: none;
|
||
}
|
||
#TOC ul {
|
||
padding-left: 1.3em;
|
||
}
|
||
#TOC > ul {
|
||
padding-left: 0;
|
||
}
|
||
#TOC a:not(:hover) {
|
||
text-decoration: none;
|
||
}
|
||
code{white-space: pre-wrap;}
|
||
span.smallcaps{font-variant: small-caps;}
|
||
div.columns{display: flex; gap: min(4vw, 1.5em);}
|
||
div.column{flex: auto; overflow-x: auto;}
|
||
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
|
||
/* The extra [class] is a hack that increases specificity enough to
|
||
override a similar rule in reveal.js */
|
||
ul.task-list[class]{list-style: none;}
|
||
ul.task-list li input[type="checkbox"] {
|
||
font-size: inherit;
|
||
width: 0.8em;
|
||
margin: 0 0.8em 0.2em -1.6em;
|
||
vertical-align: middle;
|
||
}
|
||
.display.math{display: block; text-align: center; margin: 0.5rem auto;}
|
||
/* CSS for syntax highlighting */
|
||
html { -webkit-text-size-adjust: 100%; }
|
||
pre > code.sourceCode { white-space: pre; position: relative; }
|
||
pre > code.sourceCode > span { display: inline-block; line-height: 1.25; }
|
||
pre > code.sourceCode > span:empty { height: 1.2em; }
|
||
.sourceCode { overflow: visible; }
|
||
code.sourceCode > span { color: inherit; text-decoration: inherit; }
|
||
div.sourceCode { margin: 1em 0; }
|
||
pre.sourceCode { margin: 0; }
|
||
@media screen {
|
||
div.sourceCode { overflow: auto; }
|
||
}
|
||
@media print {
|
||
pre > code.sourceCode { white-space: pre-wrap; }
|
||
pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; }
|
||
}
|
||
pre.numberSource code
|
||
{ counter-reset: source-line 0; }
|
||
pre.numberSource code > span
|
||
{ position: relative; left: -4em; counter-increment: source-line; }
|
||
pre.numberSource code > span > a:first-child::before
|
||
{ content: counter(source-line);
|
||
position: relative; left: -1em; text-align: right; vertical-align: baseline;
|
||
border: none; display: inline-block;
|
||
-webkit-touch-callout: none; -webkit-user-select: none;
|
||
-khtml-user-select: none; -moz-user-select: none;
|
||
-ms-user-select: none; user-select: none;
|
||
padding: 0 4px; width: 4em;
|
||
background-color: #232629;
|
||
color: #7a7c7d;
|
||
}
|
||
pre.numberSource { margin-left: 3em; border-left: 1px solid #7a7c7d; padding-left: 4px; }
|
||
div.sourceCode
|
||
{ color: #cfcfc2; background-color: #232629; }
|
||
@media screen {
|
||
pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; }
|
||
}
|
||
code span { color: #cfcfc2; } /* Normal */
|
||
code span.al { color: #95da4c; background-color: #4d1f24; font-weight: bold; } /* Alert */
|
||
code span.an { color: #3f8058; } /* Annotation */
|
||
code span.at { color: #2980b9; } /* Attribute */
|
||
code span.bn { color: #f67400; } /* BaseN */
|
||
code span.bu { color: #7f8c8d; } /* BuiltIn */
|
||
code span.cf { color: #fdbc4b; font-weight: bold; } /* ControlFlow */
|
||
code span.ch { color: #3daee9; } /* Char */
|
||
code span.cn { color: #27aeae; font-weight: bold; } /* Constant */
|
||
code span.co { color: #7a7c7d; } /* Comment */
|
||
code span.cv { color: #7f8c8d; } /* CommentVar */
|
||
code span.do { color: #a43340; } /* Documentation */
|
||
code span.dt { color: #2980b9; } /* DataType */
|
||
code span.dv { color: #f67400; } /* DecVal */
|
||
code span.er { color: #da4453; text-decoration: underline; } /* Error */
|
||
code span.ex { color: #0099ff; font-weight: bold; } /* Extension */
|
||
code span.fl { color: #f67400; } /* Float */
|
||
code span.fu { color: #8e44ad; } /* Function */
|
||
code span.im { color: #27ae60; } /* Import */
|
||
code span.in { color: #c45b00; } /* Information */
|
||
code span.kw { color: #cfcfc2; font-weight: bold; } /* Keyword */
|
||
code span.op { color: #cfcfc2; } /* Operator */
|
||
code span.ot { color: #27ae60; } /* Other */
|
||
code span.pp { color: #27ae60; } /* Preprocessor */
|
||
code span.re { color: #2980b9; background-color: #153042; } /* RegionMarker */
|
||
code span.sc { color: #3daee9; } /* SpecialChar */
|
||
code span.ss { color: #da4453; } /* SpecialString */
|
||
code span.st { color: #f44f4f; } /* String */
|
||
code span.va { color: #27aeae; } /* Variable */
|
||
code span.vs { color: #da4453; } /* VerbatimString */
|
||
code span.wa { color: #da4453; } /* Warning */
|
||
</style>
|
||
<style>/* Colored print + screen stylesheet for zapier.georgelambert.org */
|
||
:root {
|
||
--ink: #171a26;
|
||
--muted: #5b6178;
|
||
--line: #d9dce8;
|
||
--bg: #f4f5fb;
|
||
--paper: #ffffff;
|
||
--accent: #4f46e5;
|
||
--accent-deep: #312e81;
|
||
--accent-soft: #eef0fe;
|
||
--ok: #047857;
|
||
--warn: #8a5a00;
|
||
--code-bg: #1b1f33;
|
||
--code-fg: #e8ecff;
|
||
}
|
||
html { background: var(--bg); }
|
||
body {
|
||
margin: 0 auto;
|
||
padding: 1.5rem 1.25rem 3rem;
|
||
max-width: 48rem;
|
||
font: 15px/1.55 -apple-system, "Segoe UI", Georgia, serif;
|
||
color: var(--ink);
|
||
background: var(--paper);
|
||
}
|
||
.doc-banner {
|
||
background: linear-gradient(160deg, #312e81 0%, #4f46e5 60%, #7c74f0 100%);
|
||
color: #eef0fe;
|
||
margin: -1.5rem -1.25rem 1.5rem;
|
||
padding: 1.1rem 1.25rem 1rem;
|
||
}
|
||
.doc-banner a { color: #fff; }
|
||
.doc-banner .kicker {
|
||
letter-spacing: 0.12em;
|
||
text-transform: uppercase;
|
||
font: 700 10px system-ui, sans-serif;
|
||
opacity: 0.8;
|
||
}
|
||
.doc-banner h1 { margin: 0.25rem 0 0; font-size: 1.45rem; color: #fff; }
|
||
h1, h2, h3, h4 { color: var(--accent-deep); page-break-after: avoid; }
|
||
h1 { font-size: 1.7rem; }
|
||
h2 {
|
||
font-size: 1.2rem;
|
||
border-bottom: 2px solid var(--accent);
|
||
padding-bottom: 0.2rem;
|
||
margin-top: 1.6rem;
|
||
}
|
||
h3 { font-size: 1.05rem; color: var(--accent); }
|
||
a { color: var(--accent); }
|
||
p, li { orphans: 3; widows: 3; }
|
||
code {
|
||
font-family: ui-monospace, Menlo, Consolas, monospace;
|
||
font-size: 0.86em;
|
||
background: var(--accent-soft);
|
||
color: var(--accent-deep);
|
||
padding: 0.08em 0.28em;
|
||
border-radius: 4px;
|
||
}
|
||
pre, div.sourceCode, div.sourceCode pre {
|
||
background: var(--code-bg) !important;
|
||
color: var(--code-fg) !important;
|
||
padding: 0.85rem 1rem;
|
||
border-radius: 10px;
|
||
overflow: auto;
|
||
font-size: 0.78rem;
|
||
line-height: 1.4;
|
||
page-break-inside: avoid;
|
||
}
|
||
pre code { background: transparent; color: inherit; padding: 0; }
|
||
#title-block-header, header#title-block-header, h1.title { display: none; }
|
||
.doc-banner + h1 { display: none; }
|
||
table {
|
||
border-collapse: collapse;
|
||
width: 100%;
|
||
margin: 0.8rem 0 1.2rem;
|
||
font-size: 0.9rem;
|
||
page-break-inside: avoid;
|
||
}
|
||
th, td { border: 1px solid var(--line); padding: 0.38rem 0.55rem; text-align: left; vertical-align: top; }
|
||
th {
|
||
background: var(--accent);
|
||
color: #fff;
|
||
font: 650 12px system-ui, sans-serif;
|
||
}
|
||
tr:nth-child(even) td { background: var(--accent-soft); }
|
||
blockquote {
|
||
margin: 1rem 0;
|
||
padding: 0.4rem 0.9rem;
|
||
border-left: 4px solid var(--accent);
|
||
background: var(--accent-soft);
|
||
color: var(--accent-deep);
|
||
}
|
||
img { max-width: 100%; height: auto; border-radius: 8px; page-break-inside: avoid; }
|
||
hr { border: 0; border-top: 1px solid var(--line); }
|
||
ul, ol { padding-left: 1.25rem; }
|
||
nav.site { font: 13px system-ui, sans-serif; margin-bottom: 0.4rem; }
|
||
.source-path { font: 11px ui-monospace, Menlo, monospace; color: var(--muted); }
|
||
|
||
@page {
|
||
size: letter;
|
||
margin: 0.65in 0.7in 0.8in 0.7in;
|
||
@top-left {
|
||
content: "Verae Time × Zapier";
|
||
font: 700 8pt system-ui, sans-serif;
|
||
color: #4f46e5;
|
||
}
|
||
@top-right {
|
||
content: "zapier.georgelambert.org";
|
||
font: 8pt system-ui, sans-serif;
|
||
color: #6b7186;
|
||
}
|
||
@bottom-center {
|
||
content: counter(page) " / " counter(pages);
|
||
font: 8pt system-ui, sans-serif;
|
||
color: #6b7186;
|
||
}
|
||
}
|
||
@media print {
|
||
html, body { background: #fff; max-width: none; padding: 0; }
|
||
.doc-banner { margin: 0 0 1rem; border-radius: 8px; -webkit-print-color-adjust: exact; print-color-adjust: exact; }
|
||
a { text-decoration: none; }
|
||
th, tr:nth-child(even) td, pre, blockquote, code { -webkit-print-color-adjust: exact; print-color-adjust: exact; }
|
||
}
|
||
</style>
|
||
</head>
|
||
<body>
|
||
<div class="doc-banner"><nav class="site"><a href="/">zapier.georgelambert.org</a> · <a href="/index-md.html">Markdown indexes</a></nav><div class="kicker">Verae Time × Zapier · progress report</div><h1>NATS cluster message speed</h1><div class="source-path">packages/zapier-decisions/reports/nats-cluster-bench.md</div></div>
|
||
<header id="title-block-header">
|
||
<h1 class="title">NATS cluster message speed</h1>
|
||
</header>
|
||
<p><strong>Progress report</strong> · 2026-09-12 · run
|
||
<code>20260912T045131Z</code> (UTC)</p>
|
||
<p>This is the full write-up of the test-environment NATS cluster bench:
|
||
what was measured, how, the numbers, the charts, and what they mean for
|
||
Verae Time × Zapier. Short tables also live in <a
|
||
href="https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md"><code>verae-nats-cluster/BENCH.md</code></a>.
|
||
Raw logs and CSVs are in that repo under
|
||
<code>results/20260912T045131Z/</code>.</p>
|
||
<hr />
|
||
<h2 id="executive-summary">1. Executive summary</h2>
|
||
<p>The test cluster is three JetStream nodes on private
|
||
<code>vmbr1</code> (LXC 511–513). The bench client is a
|
||
<strong>fourth</strong> guest (LXC 510), so the numbers are
|
||
cluster-plus-network, not a process talking to itself on loopback.</p>
|
||
<p>Two different systems were measured, on purpose:</p>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 24%" />
|
||
<col style="width: 36%" />
|
||
<col style="width: 39%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr>
|
||
<th>System</th>
|
||
<th>What it is</th>
|
||
<th>What we got</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr>
|
||
<td><strong>Core NATS</strong></td>
|
||
<td>Fire-and-forget pub/sub. No disk, no replica ack.</td>
|
||
<td>About <strong>0.75–2.0 million msgs/s</strong> at 128 B, depending
|
||
on fan-out. At 1 KiB, about <strong>630k msgs/s</strong> and
|
||
<strong>~616 MB/s</strong> aggregate.</td>
|
||
</tr>
|
||
<tr>
|
||
<td><strong>JetStream file, replicas=3</strong></td>
|
||
<td>Durable, replicated — <strong>this is what product streams
|
||
use</strong>.</td>
|
||
<td>About <strong>16k durable 128 B pubs/s</strong>, about
|
||
<strong>13.5k</strong> at 1 KiB. Pull consume keeps up with publish at
|
||
~11k msgs/s each side.</td>
|
||
</tr>
|
||
<tr>
|
||
<td><strong>Ping delay</strong></td>
|
||
<td>One message at a time, publish then wait.</td>
|
||
<td><strong>avg 0.307 ms</strong>, <strong>p99 0.734 ms</strong>, max
|
||
2.76 ms (1k × 128 B).</td>
|
||
</tr>
|
||
<tr>
|
||
<td><strong>Flood delay</strong></td>
|
||
<td>Publishers dump a batch; subscriber drains.</td>
|
||
<td><strong>150–505 ms</strong>. That is <strong>queueing under
|
||
burst</strong>, not wire time.</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p><strong>For this product:</strong> timestamp jobs, job events,
|
||
webhooks, and archive puts go through JetStream r=3. Plan capacity
|
||
against <strong>~16k durable msgs/s</strong> on this stand, not the
|
||
million-msg core numbers. A quiet job-event hop is a fraction of a
|
||
millisecond. If a mailbox falls behind, delay jumps into hundreds of
|
||
milliseconds — that is the flood column.</p>
|
||
<p>Core NATS is still useful: it is the ceiling for non-durable fan-out
|
||
on this host, and it shows <code>vmbr1</code> and the nats-server
|
||
processes are not the JetStream bottleneck. JetStream is.</p>
|
||
<hr />
|
||
<h2 id="why-this-test-exists">2. Why this test exists</h2>
|
||
<p>The lab cut the test environment over to the three-node cluster.
|
||
Before treating that cluster as the message fabric for keep, fleet,
|
||
middleware, billing, and archive workers, we needed:</p>
|
||
<ol type="1">
|
||
<li><strong>Throughput at several loads</strong> — one publisher vs
|
||
many, 128 B vs 1 KiB, core vs durable.</li>
|
||
<li><strong>Delay characteristics</strong> — both the quiet path (one
|
||
message RTT) and the overloaded path (burst into a mailbox).</li>
|
||
<li><strong>A client that is not a nats-* server</strong> — otherwise we
|
||
would be measuring loopback on the broker.</li>
|
||
</ol>
|
||
<p>This is a <strong>lab stand on one Proxmox host</strong>, not three
|
||
metal boxes. It answers “is this cluster in the right order of magnitude
|
||
for our traffic?” It does not replace a soak test on dedicated
|
||
disks.</p>
|
||
<hr />
|
||
<h2 id="topology">3. Topology</h2>
|
||
<pre class="text"><code> vmbr1 10.10.10.0/24 (not on vmbr0, not public)
|
||
-----------------------------------------------
|
||
LXC 510 LXC 511 LXC 512 LXC 513
|
||
verae-px-worker nats-a nats-b nats-c
|
||
10.10.10.20 10.10.10.21 10.10.10.22 10.10.10.23
|
||
bench client :4222 client :4222 :4222
|
||
:6222 routes :6222 :6222
|
||
:8222 loopback :8222 :8222</code></pre>
|
||
<ul>
|
||
<li>Cluster name: <code>verae</code>. Each server has two routes to the
|
||
other two.</li>
|
||
<li>Client URL:
|
||
<code>nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222</code></li>
|
||
<li>HTTP monitor is <strong>loopback :8222</strong> inside each guest.
|
||
Zapier cloud never talks to NATS.</li>
|
||
<li>Product streams already on this cluster (<code>ZAPIER_JOBS</code>,
|
||
<code>ZAPIER_EVENTS</code>, <code>ZAPIER_WEBHOOKS</code>,
|
||
<code>ZAPIER_USAGE</code>, <code>VERAE_ARCHIVE</code>) use
|
||
<strong>file</strong> storage and <strong>replicas=3</strong>. The
|
||
JetStream bench used the same settings on a throwaway stream
|
||
<code>benchstream</code>.</li>
|
||
<li>Host <code>127.0.0.1:4222</code> is still listening on NS1;
|
||
<strong>clients no longer use it</strong>.</li>
|
||
</ul>
|
||
<p>Credits for the stack: Scott Lindsey, George Lambert, NATS.IO,
|
||
Grok-Code.</p>
|
||
<hr />
|
||
<h2 id="method">4. Method</h2>
|
||
<h3 id="tools">4.1 Tools</h3>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 53%" />
|
||
<col style="width: 46%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr>
|
||
<th>Piece</th>
|
||
<th>Role</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr>
|
||
<td><code>nats</code> CLI <strong>0.1.6</strong></td>
|
||
<td>Throughput (<code>nats bench --no-progress --csv</code>). Its
|
||
min/avg/max are <strong>publisher rate spread</strong>, not delay.</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>scripts/latency.mjs</code></td>
|
||
<td>Two connections, header timestamp <code>t</code>, delay = receive
|
||
time − send time.</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>scripts/bench.sh</code></td>
|
||
<td>Runs the ladder from NS1 via <code>pct exec</code> on VMID 510.</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>scripts/bench-report.py</code></td>
|
||
<td>Turns logs into the short <code>BENCH.md</code> table.</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p>Re-run on NS1, from <code>verae-nats-cluster</code>:</p>
|
||
<div class="sourceCode" id="cb2"><pre
|
||
class="sourceCode bash"><code class="sourceCode bash"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="fu">bash</span> scripts/bench.sh</span></code></pre></div>
|
||
<h3 id="load-ladder">4.2 Load ladder</h3>
|
||
<p><strong>Core NATS</strong> (subject <code>bench.core.*</code>):</p>
|
||
<table>
|
||
<thead>
|
||
<tr>
|
||
<th>Run</th>
|
||
<th>Publishers</th>
|
||
<th>Subscribers</th>
|
||
<th>Messages</th>
|
||
<th>Payload</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr>
|
||
<td><code>core-1p1s-50k-128</code></td>
|
||
<td>1</td>
|
||
<td>1</td>
|
||
<td>50,000</td>
|
||
<td>128 B</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>core-4p4s-100k-128</code></td>
|
||
<td>4</td>
|
||
<td>4</td>
|
||
<td>100,000</td>
|
||
<td>128 B</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>core-8p8s-200k-128</code></td>
|
||
<td>8</td>
|
||
<td>8</td>
|
||
<td>200,000</td>
|
||
<td>128 B</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>core-4p4s-50k-1k</code></td>
|
||
<td>4</td>
|
||
<td>4</td>
|
||
<td>50,000</td>
|
||
<td>1024 B</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p><strong>JetStream</strong>
|
||
(<code>--js --storage file --replicas 3 --stream benchstream</code>).
|
||
The stream is deleted between loads so the name never collides:</p>
|
||
<table>
|
||
<thead>
|
||
<tr>
|
||
<th>Run</th>
|
||
<th>Shape</th>
|
||
<th>Messages</th>
|
||
<th>Payload</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr>
|
||
<td><code>js-1p-20k-128-r3</code></td>
|
||
<td>1 publisher</td>
|
||
<td>20,000</td>
|
||
<td>128 B</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>js-4p-50k-128-r3</code></td>
|
||
<td>4 publishers</td>
|
||
<td>50,000</td>
|
||
<td>128 B</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>js-4p-20k-1k-r3</code></td>
|
||
<td>4 publishers</td>
|
||
<td>20,000</td>
|
||
<td>1024 B</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>js-2p2s-20k-128-r3</code></td>
|
||
<td>2 pub + 2 pull sub</td>
|
||
<td>20,000</td>
|
||
<td>128 B</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p><strong>Delay</strong> (core subjects, two connections):</p>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 15%" />
|
||
<col style="width: 18%" />
|
||
<col style="width: 21%" />
|
||
<col style="width: 18%" />
|
||
<col style="width: 27%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr>
|
||
<th>Run</th>
|
||
<th>Mode</th>
|
||
<th>Count</th>
|
||
<th>Pubs</th>
|
||
<th>Payload</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr>
|
||
<td><code>lat-ping-1k-128</code></td>
|
||
<td><strong>ping</strong> — publish, wait for that message, repeat</td>
|
||
<td>1,000</td>
|
||
<td>1</td>
|
||
<td>128 B</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>lat-1p-5k-128</code></td>
|
||
<td><strong>flood</strong> — publish all, then drain</td>
|
||
<td>5,000</td>
|
||
<td>1</td>
|
||
<td>128 B</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>lat-4p-10k-128</code></td>
|
||
<td>flood</td>
|
||
<td>10,000</td>
|
||
<td>4</td>
|
||
<td>128 B</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>lat-8p-20k-128</code></td>
|
||
<td>flood</td>
|
||
<td>20,000</td>
|
||
<td>8</td>
|
||
<td>128 B</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>lat-4p-5k-1k</code></td>
|
||
<td>flood</td>
|
||
<td>5,000</td>
|
||
<td>4</td>
|
||
<td>1024 B</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p>Ping answers “how long does one quiet hop take?” Flood answers “what
|
||
happens to the last message if we burst N messages into a mailbox?”
|
||
Those are different questions. Mixing them is how 0.3 ms and 400 ms get
|
||
confused.</p>
|
||
<h3 id="how-to-read-nats-bench-columns">4.3 How to read nats bench
|
||
columns</h3>
|
||
<ul>
|
||
<li><strong>Pub msgs/s</strong> — rate at which publishers finished
|
||
their share.</li>
|
||
<li><strong>Sub msgs/s</strong> — rate at which subscribers finished.
|
||
With several subscribers on the same subject, core NATS <strong>fans
|
||
out</strong>, so sub rate can exceed pub rate.</li>
|
||
<li><strong>Aggregate msgs/s</strong> — nats CLI
|
||
<code>NATS Pub/Sub stats</code> line (pub+sub work in one number).
|
||
Useful as a headline; do not treat it as “the network carried this many
|
||
unique messages.”</li>
|
||
<li>Empty JetStream sub cells mean that run was publish-only (durable
|
||
write, no consumer in the same process).</li>
|
||
</ul>
|
||
<hr />
|
||
<h2 id="throughput-results">5. Throughput results</h2>
|
||
<h3 id="core-nats">5.1 Core NATS</h3>
|
||
<figure>
|
||
<img src="charts/core-throughput.png"
|
||
alt="Core NATS throughput at four loads" />
|
||
<figcaption aria-hidden="true">Core NATS throughput at four
|
||
loads</figcaption>
|
||
</figure>
|
||
<table style="width:100%;">
|
||
<colgroup>
|
||
<col style="width: 7%" />
|
||
<col style="width: 26%" />
|
||
<col style="width: 17%" />
|
||
<col style="width: 14%" />
|
||
<col style="width: 17%" />
|
||
<col style="width: 14%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr>
|
||
<th>Run</th>
|
||
<th>Aggregate msgs/s</th>
|
||
<th>Pub msgs/s</th>
|
||
<th>Pub MB/s</th>
|
||
<th>Sub msgs/s</th>
|
||
<th>Sub MB/s</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr>
|
||
<td><code>core-1p1s-50k-128</code></td>
|
||
<td>1,200,836</td>
|
||
<td>791,094</td>
|
||
<td>96.57</td>
|
||
<td>747,461</td>
|
||
<td>91.24</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>core-4p4s-100k-128</code></td>
|
||
<td>1,521,256</td>
|
||
<td>316,312</td>
|
||
<td>38.61</td>
|
||
<td>1,299,634</td>
|
||
<td>158.65</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>core-8p8s-200k-128</code></td>
|
||
<td>2,007,937</td>
|
||
<td>333,957</td>
|
||
<td>40.77</td>
|
||
<td>1,790,736</td>
|
||
<td>218.60</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>core-4p4s-50k-1k</code></td>
|
||
<td>630,460</td>
|
||
<td>247,747</td>
|
||
<td>241.94</td>
|
||
<td>510,216</td>
|
||
<td>498.26</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p><strong>What this chart is saying.</strong> Adding subscribers raises
|
||
<strong>aggregate</strong> and <strong>sub</strong> rates because each
|
||
published message is delivered to every subscriber. Publish rate does
|
||
<strong>not</strong> climb the same way: 1 publisher at 128 B already
|
||
pushes ~791k msgs/s; 4 and 8 publishers sit around 310–335k msgs/s
|
||
<strong>each process slower</strong>, while fan-out on the sub side goes
|
||
to 1.3M then 1.8M.</p>
|
||
<p>That publisher slowdown is expected on this stand. The four/eight
|
||
publisher processes and the four/eight subscribers all run
|
||
<strong>inside one LXC</strong> (510) against three broker LXCs on the
|
||
<strong>same Proxmox CPU and <code>vmbr1</code></strong>. Per-publisher
|
||
logs show a wide spread (example, 4p core 128 B: 79k–524k msgs/s among
|
||
the four pubs). That is CPU scheduling and client-side contention, not a
|
||
NATS cluster that only has one fast node.</p>
|
||
<p><strong>1:1 at 128 B</strong> is the cleanest core number:
|
||
<strong>~791k pub, ~747k sub, ~1.20M aggregate</strong>. The cluster and
|
||
the bridge can move three-quarter-million small messages per second
|
||
fire-and-forget from a single client pair.</p>
|
||
<h3 id="payload-size-core">5.2 Payload size (core)</h3>
|
||
<figure>
|
||
<img src="charts/payload-size.png" alt="Core NATS 128 B vs 1 KiB" />
|
||
<figcaption aria-hidden="true">Core NATS 128 B vs 1 KiB</figcaption>
|
||
</figure>
|
||
<p>Same 4p4s shape, two sizes:</p>
|
||
<table style="width:100%;">
|
||
<colgroup>
|
||
<col style="width: 13%" />
|
||
<col style="width: 26%" />
|
||
<col style="width: 23%" />
|
||
<col style="width: 17%" />
|
||
<col style="width: 17%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr>
|
||
<th>Payload</th>
|
||
<th>Aggregate msgs/s</th>
|
||
<th>Aggregate MB/s</th>
|
||
<th>Pub msgs/s</th>
|
||
<th>Sub msgs/s</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr>
|
||
<td>128 B</td>
|
||
<td>1,521,256</td>
|
||
<td>185.70</td>
|
||
<td>316,312</td>
|
||
<td>1,299,634</td>
|
||
</tr>
|
||
<tr>
|
||
<td>1 KiB</td>
|
||
<td>630,460</td>
|
||
<td>615.68</td>
|
||
<td>247,747</td>
|
||
<td>510,216</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p>Message rate falls; <strong>byte rate rises</strong> (186 MB/s → 616
|
||
MB/s aggregate). We are leaving the “tiny message, CPU/syscall bound”
|
||
region and entering “copying bytes across <code>vmbr1</code>.” Job JSON
|
||
and archive metadata sit nearer 128 B–1 KiB than megabyte blobs (blobs
|
||
are HTTP/WORM, not NATS payloads).</p>
|
||
<h3 id="jetstream-r3-file">5.3 JetStream r=3 file</h3>
|
||
<figure>
|
||
<img src="charts/js-throughput.png"
|
||
alt="JetStream durable publish rate" />
|
||
<figcaption aria-hidden="true">JetStream durable publish
|
||
rate</figcaption>
|
||
</figure>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 10%" />
|
||
<col style="width: 26%" />
|
||
<col style="width: 21%" />
|
||
<col style="width: 26%" />
|
||
<col style="width: 15%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr>
|
||
<th>Run</th>
|
||
<th>Pub msgs/s</th>
|
||
<th>Pub MB/s</th>
|
||
<th>Sub msgs/s</th>
|
||
<th>Notes</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr>
|
||
<td><code>js-1p-20k-128-r3</code></td>
|
||
<td><strong>16,155</strong></td>
|
||
<td>1.97</td>
|
||
<td>—</td>
|
||
<td>publish-only</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>js-4p-50k-128-r3</code></td>
|
||
<td><strong>16,607</strong></td>
|
||
<td>2.03</td>
|
||
<td>—</td>
|
||
<td>four pubs, same ceiling</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>js-4p-20k-1k-r3</code></td>
|
||
<td><strong>13,493</strong></td>
|
||
<td>13.18</td>
|
||
<td>—</td>
|
||
<td>1 KiB still disk/replica bound</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>js-2p2s-20k-128-r3</code></td>
|
||
<td>10,965</td>
|
||
<td>1.34</td>
|
||
<td>10,942</td>
|
||
<td>pull consumers keep up</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p><strong>Four publishers do not make JetStream four times
|
||
faster.</strong> 1p and 4p at 128 B are both ~16k msgs/s. The limiter is
|
||
<strong>synchronous replication to three file-backed replicas</strong>,
|
||
not client parallelism. That is the result we wanted to see: the bench
|
||
stream is behaving like a replicated log, not like core fan-out.</p>
|
||
<p>Pull consume (<code>js-2p2s</code>) is slightly slower on publish
|
||
(~11k) because the same run is also reading. Pub and sub stay matched
|
||
(10,965 vs 10,942): the consumer is not the straggler.</p>
|
||
<p>1 KiB durable write is ~13.5k msgs/s (~13.2 MB/s). Bytes go up;
|
||
message rate dips only a little. JetStream here is
|
||
<strong>ack/fdatasync/replica</strong> bound, not payload-copy bound, in
|
||
this size range.</p>
|
||
<h3 id="core-vs-jetstream-same-client-same-cluster">5.4 Core vs
|
||
JetStream (same client, same cluster)</h3>
|
||
<figure>
|
||
<img src="charts/core-vs-js.png"
|
||
alt="Core vs JetStream publish rate, log scale" />
|
||
<figcaption aria-hidden="true">Core vs JetStream publish rate, log
|
||
scale</figcaption>
|
||
</figure>
|
||
<p>The log scale is required: core publish is <strong>~15–50×</strong>
|
||
JetStream publish on this stand.</p>
|
||
<table>
|
||
<thead>
|
||
<tr>
|
||
<th>Shape</th>
|
||
<th>Core pub msgs/s</th>
|
||
<th>JS r=3 file pub msgs/s</th>
|
||
<th>Ratio</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr>
|
||
<td>1 publisher, 128 B</td>
|
||
<td>791,094</td>
|
||
<td>16,155</td>
|
||
<td>~49×</td>
|
||
</tr>
|
||
<tr>
|
||
<td>4 publishers, 128 B</td>
|
||
<td>316,312</td>
|
||
<td>16,607</td>
|
||
<td>~19×</td>
|
||
</tr>
|
||
<tr>
|
||
<td>4 publishers, 1 KiB</td>
|
||
<td>247,747</td>
|
||
<td>13,493</td>
|
||
<td>~18×</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p>This is not JetStream “losing.” Core is allowed to forget a message
|
||
the instant the server accepts it. JetStream on file with replicas=3
|
||
must <strong>record it on a majority</strong> before the publish acks.
|
||
Our product streams (<code>ZAPIER_*</code>, <code>VERAE_ARCHIVE</code>)
|
||
chose that trade on purpose: a job event that survives one LXC dying is
|
||
worth ~16k msgs/s instead of ~800k.</p>
|
||
<p>If we ever need core-like rates for a signal that may drop, that
|
||
signal should not be on a replicated file stream.</p>
|
||
<hr />
|
||
<h2 id="delay-results">6. Delay results</h2>
|
||
<figure>
|
||
<img src="charts/delay-percentiles.png"
|
||
alt="Delay percentiles, log scale" />
|
||
<figcaption aria-hidden="true">Delay percentiles, log scale</figcaption>
|
||
</figure>
|
||
<table style="width:100%;">
|
||
<colgroup>
|
||
<col style="width: 10%" />
|
||
<col style="width: 12%" />
|
||
<col style="width: 14%" />
|
||
<col style="width: 10%" />
|
||
<col style="width: 10%" />
|
||
<col style="width: 10%" />
|
||
<col style="width: 10%" />
|
||
<col style="width: 10%" />
|
||
<col style="width: 10%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr>
|
||
<th>Run</th>
|
||
<th>Kind</th>
|
||
<th>Count</th>
|
||
<th>min</th>
|
||
<th>avg</th>
|
||
<th>p50</th>
|
||
<th>p90</th>
|
||
<th>p99</th>
|
||
<th>max</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr>
|
||
<td><code>lat-ping-1k-128</code></td>
|
||
<td>ping (sequential RTT)</td>
|
||
<td>1000</td>
|
||
<td>0.254 ms</td>
|
||
<td><strong>0.307 ms</strong></td>
|
||
<td>0.286 ms</td>
|
||
<td>0.332 ms</td>
|
||
<td><strong>0.734 ms</strong></td>
|
||
<td>2.763 ms</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>lat-1p-5k-128</code></td>
|
||
<td>flood</td>
|
||
<td>5000</td>
|
||
<td>149.3 ms</td>
|
||
<td>238.6 ms</td>
|
||
<td>248.8 ms</td>
|
||
<td>274.3 ms</td>
|
||
<td>279.4 ms</td>
|
||
<td>279.7 ms</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>lat-4p-5k-1k</code></td>
|
||
<td>flood</td>
|
||
<td>5000</td>
|
||
<td>155.1 ms</td>
|
||
<td>211.7 ms</td>
|
||
<td>217.6 ms</td>
|
||
<td>223.3 ms</td>
|
||
<td>227.8 ms</td>
|
||
<td>228.4 ms</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>lat-4p-10k-128</code></td>
|
||
<td>flood</td>
|
||
<td>10000</td>
|
||
<td>174.2 ms</td>
|
||
<td>263.2 ms</td>
|
||
<td>266.7 ms</td>
|
||
<td>299.1 ms</td>
|
||
<td>304.2 ms</td>
|
||
<td>304.5 ms</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>lat-8p-20k-128</code></td>
|
||
<td>flood</td>
|
||
<td>20000</td>
|
||
<td>304.6 ms</td>
|
||
<td>453.7 ms</td>
|
||
<td>466.3 ms</td>
|
||
<td>499.9 ms</td>
|
||
<td>505.1 ms</td>
|
||
<td>505.6 ms</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p>The dashed line on the chart is 1 ms. Only <strong>ping</strong>
|
||
lives there.</p>
|
||
<h3 id="ping-the-quiet-hop">6.1 Ping — the quiet hop</h3>
|
||
<p>One publisher, one subscriber, two connections, wait for each message
|
||
before sending the next.</p>
|
||
<ul>
|
||
<li><strong>min 0.254 ms</strong> — guest → <code>vmbr1</code> → a
|
||
nats-server → <code>vmbr1</code> → guest.</li>
|
||
<li><strong>p50 0.286 ms / avg 0.307 ms</strong> — typical.</li>
|
||
<li><strong>p99 0.734 ms</strong> — still under a millisecond.</li>
|
||
<li><strong>max 2.763 ms</strong> — one outlier in 1,000 samples (GC,
|
||
scheduler, or a slow route). Not the tail we design for.</li>
|
||
</ul>
|
||
<p>A middleware <code>jobs.watch</code> publish followed by a waiter on
|
||
<code>jobs.events</code> is this shape when the poller is keeping up.
|
||
Compared with HTTPS to Zapier (tens to hundreds of milliseconds) or a
|
||
live Verae <code>GET /api/status/{jobId}</code>, NATS RTT is noise.</p>
|
||
<h3 id="flood-queueing-under-burst">6.2 Flood — queueing under
|
||
burst</h3>
|
||
<p>Publishers write the whole batch as fast as they can, then the
|
||
subscriber drains. Each message’s delay is “how long was I in the buffer
|
||
before the subscriber got to me?”</p>
|
||
<p>That is why:</p>
|
||
<ul>
|
||
<li><strong>min is already ~150–300 ms</strong> — even the first
|
||
messages wait behind a burst that filled the socket/client queue.</li>
|
||
<li><strong>p50 ≈ p99 ≈ max</strong> — a queue drain has a tight
|
||
distribution: everyone waits for roughly the same backlog.</li>
|
||
<li><strong>8p × 20k is ~450 ms avg</strong> — twice the messages of 4p
|
||
× 10k, roughly twice the wait. Linear in backlog, not in cluster
|
||
diameter.</li>
|
||
</ul>
|
||
<p>Flood is <strong>not</strong> a measurement of NATS being slow. The
|
||
ping column proves the hop is ~0.3 ms. Flood is a measurement of
|
||
<strong>what operators will see if a consumer stalls</strong>
|
||
(job-events mailbox, webhook deliver, archive reply). Backlog time ≈
|
||
<code>queued_messages / consume_rate</code>.</p>
|
||
<h3 id="kib-flood-vs-128-b-flood">6.3 1 KiB flood vs 128 B flood</h3>
|
||
<p>4 publishers, 5k messages at 1 KiB: avg <strong>212 ms</strong>,
|
||
slightly <strong>faster</strong> than 4p 10k × 128 B (263 ms) because
|
||
the <strong>count is half</strong>, even though each message is 8×
|
||
larger. Again: delay here tracks <strong>how many messages are
|
||
queued</strong>, not payload size, in this range.</p>
|
||
<hr />
|
||
<h2 id="what-this-means-for-verae-zapier">7. What this means for Verae ×
|
||
Zapier</h2>
|
||
<p>Product subjects on this cluster:</p>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 29%" />
|
||
<col style="width: 19%" />
|
||
<col style="width: 51%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr>
|
||
<th>Address</th>
|
||
<th>Kind</th>
|
||
<th>Bench analogue</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr>
|
||
<td><code>verae.zapier.jobs.watch</code></td>
|
||
<td>work queue (JetStream)</td>
|
||
<td>JS durable pub ~16k/s</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>verae.zapier.jobs.events</code></td>
|
||
<td>events</td>
|
||
<td>JS + ping if waiters keep up; flood if they do not</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>verae.zapier.webhooks.deliver</code></td>
|
||
<td>work queue</td>
|
||
<td>JS durable</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>verae.zapier.usage</code></td>
|
||
<td>optional</td>
|
||
<td>JS durable</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>verae.billing.*</code></td>
|
||
<td>request-reply</td>
|
||
<td>ping (quiet RTT)</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>verae.archive.put</code> / <code>query</code> /
|
||
<code>reply.*</code></td>
|
||
<td>JetStream + broadcast query</td>
|
||
<td>JS durable; query fan-out is closer to core but still JS-backed
|
||
puts</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p><strong>Capacity.</strong> 16k durable 128 B pubs/s is
|
||
<strong>~1.4×10⁹ messages/day</strong> if you could fill the pipe. We
|
||
will not. Zapier HTTPS, live <code>api.veraetime.net</code>, WORM bloom
|
||
checks, and human Zap runs sit far below that. This cluster is not the
|
||
product bottleneck on NS1.</p>
|
||
<p><strong>Latency budget.</strong> A timestamp wait is: HTTP in → NATS
|
||
watch → poll Verae → NATS event → HTTP out (or REST Hook). The NATS
|
||
pieces are <strong>sub-millisecond</strong> when caught up. Do not spend
|
||
time “optimizing NATS RTT” until Zapier/Verae HTTP is in the same
|
||
band.</p>
|
||
<p><strong>Backlogs.</strong> The failure mode that <em>does</em> show
|
||
up in these numbers is flood delay. If webhook-deliver or job-events
|
||
consumers pause (keep stopped, replica floor, a blocked HTTPS post to
|
||
<code>hooks.zapier.com</code>), waiters will see <strong>hundreds of
|
||
milliseconds to seconds</strong> of queue time. Fleet replica floors and
|
||
keep exist to prevent that, not because 0.3 ms is too slow.</p>
|
||
<p><strong>Hardware move.</strong> Same three configs, three boxes,
|
||
private NIC. Expect:</p>
|
||
<ul>
|
||
<li>Core numbers to change with NIC and CPU (maybe up, maybe down).</li>
|
||
<li>JetStream numbers to change <strong>more</strong>, because they are
|
||
disk + fsync + replica RTT. Distinct SSDs should help; a slow shared
|
||
datastore would hurt.</li>
|
||
<li>Ping RTT to grow by whatever the real NIC and switch add (still
|
||
likely low milliseconds on a LAN).</li>
|
||
</ul>
|
||
<hr />
|
||
<h2 id="limits-of-this-measurement">8. Limits of this measurement</h2>
|
||
<ol type="1">
|
||
<li><strong>One Proxmox host.</strong> LXC 510–513 share cores, memory,
|
||
and the host’s disk. Replica=3 on file is <strong>three files on the
|
||
same underlying storage</strong>, not three failure domains. HA of “one
|
||
disk dies” is <strong>not</strong> proven. HA of “one LXC process dies”
|
||
is the actual claim.</li>
|
||
<li><strong>Short runs.</strong> Tens of thousands of messages, seconds
|
||
of wall time. No compaction, no multi-hour page-cache eviction, no
|
||
snapshot/restore during load.</li>
|
||
<li><strong>No TLS, no nkeys.</strong> <code>verae-nats-accounts</code>
|
||
is still a sketch. Auth would add CPU; it would not turn 16k into
|
||
800k.</li>
|
||
<li><strong>One bench client.</strong> All publishers live in 510. A
|
||
fleet of workers on several CTs might publish more into JetStream until
|
||
disk/replicas saturate — the 1p vs 4p JS result says that saturation is
|
||
already ~16k from one CT.</li>
|
||
<li><strong>nats 0.1.6</strong> does not report delay. Anyone reading
|
||
<code>min | avg | max msgs</code> on a bench log as microseconds will
|
||
get the wrong story. Delay is only <code>latency.mjs</code>.</li>
|
||
<li><strong>Core aggregate ≠ unique messages.</strong> Fan-out
|
||
double-counts. Use pub or sub columns when comparing to JetStream.</li>
|
||
<li><strong>Not a Zapier or Verae API bench.</strong> Those are still
|
||
blocked on operator login / live credentials.</li>
|
||
</ol>
|
||
<hr />
|
||
<h2 id="how-to-reproduce">9. How to reproduce</h2>
|
||
<p>On NS1 (Proxmox), from the <code>verae-nats-cluster</code>
|
||
checkout:</p>
|
||
<div class="sourceCode" id="cb3"><pre
|
||
class="sourceCode bash"><code class="sourceCode bash"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="fu">bash</span> scripts/status.sh <span class="co"># 3/3 JetStream</span></span>
|
||
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a><span class="fu">bash</span> scripts/bench.sh <span class="co"># writes results/<utc>/ and BENCH.md</span></span></code></pre></div>
|
||
<p>The client VMID defaults to <strong>510</strong>. Override with
|
||
<code>CLIENT_VMID=…</code>. <code>NATS_URL</code> comes from
|
||
<code>client.env</code>.</p>
|
||
<p>Rebuild this progress report (charts + HTML + PDF) from the
|
||
monorepo:</p>
|
||
<div class="sourceCode" id="cb4"><pre
|
||
class="sourceCode bash"><code class="sourceCode bash"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="ex">python3</span> packages/zapier-decisions/scripts/build-nats-bench-report.py</span></code></pre></div>
|
||
<hr />
|
||
<h2 id="appendix-environment-and-files">10. Appendix — environment and
|
||
files</h2>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 46%" />
|
||
<col style="width: 53%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr>
|
||
<th>Item</th>
|
||
<th>Value</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr>
|
||
<td>Run stamp</td>
|
||
<td><code>20260912T045131Z</code></td>
|
||
</tr>
|
||
<tr>
|
||
<td>Client</td>
|
||
<td>LXC 510 <code>verae-px-worker</code> <code>10.10.10.20</code></td>
|
||
</tr>
|
||
<tr>
|
||
<td>Servers</td>
|
||
<td>511/512/513 <code>nats-a/b/c</code> <code>10.10.10.21–23</code></td>
|
||
</tr>
|
||
<tr>
|
||
<td>nats CLI</td>
|
||
<td>0.1.6 linux-amd64</td>
|
||
</tr>
|
||
<tr>
|
||
<td>JS storage</td>
|
||
<td>file, replicas=3, stream <code>benchstream</code> (deleted between
|
||
loads)</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Isolation</td>
|
||
<td><code>vmbr1</code> only; no <code>0.0.0.0</code> client bind</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Short tables</td>
|
||
<td><a
|
||
href="https://git.georgelambert.org/marchon/verae-nats-cluster/src/branch/main/BENCH.md">BENCH.md</a></td>
|
||
</tr>
|
||
<tr>
|
||
<td>Raw logs</td>
|
||
<td><code>packages/verae-nats-cluster/results/20260912T045131Z/</code></td>
|
||
</tr>
|
||
<tr>
|
||
<td>This report</td>
|
||
<td><code>packages/zapier-decisions/reports/nats-cluster-bench.{md,html,pdf}</code></td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p>Publisher rate spread (nats CLI, msgs/s, <strong>not</strong>
|
||
delay):</p>
|
||
<table>
|
||
<thead>
|
||
<tr>
|
||
<th>Run</th>
|
||
<th>min</th>
|
||
<th>avg</th>
|
||
<th>max</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr>
|
||
<td>core-4p4s-100k-128 pub</td>
|
||
<td>79,260</td>
|
||
<td>257,805</td>
|
||
<td>524,453</td>
|
||
</tr>
|
||
<tr>
|
||
<td>core-8p8s-200k-128 pub</td>
|
||
<td>41,744</td>
|
||
<td>71,488</td>
|
||
<td>152,536</td>
|
||
</tr>
|
||
<tr>
|
||
<td>core-4p4s-50k-1k pub</td>
|
||
<td>61,936</td>
|
||
<td>110,271</td>
|
||
<td>176,262</td>
|
||
</tr>
|
||
<tr>
|
||
<td>js-4p-50k-128-r3 pub</td>
|
||
<td>4,154</td>
|
||
<td>5,176</td>
|
||
<td>6,628</td>
|
||
</tr>
|
||
<tr>
|
||
<td>js-4p-20k-1k-r3 pub</td>
|
||
<td>3,373</td>
|
||
<td>4,063</td>
|
||
<td>5,121</td>
|
||
</tr>
|
||
<tr>
|
||
<td>js-2p2s-20k-128-r3 pub</td>
|
||
<td>5,485</td>
|
||
<td>7,081</td>
|
||
<td>8,678</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p>Wide core spreads are the single-client-CT effect described in §5.1.
|
||
JetStream spreads are narrow and low — every publisher is waiting on the
|
||
same replicated write path.</p>
|
||
</body>
|
||
</html>
|