George Lambert
c83f1c6717
Some checks are pending
offline / test (push) Waiting to run
Compares r=1 vs r=3, file vs memory, reconnect tax, and projects three HP DL360 Gen10 NVMe + 10GbE boxes. Keep NATS; do not switch to MQTT/UDP.
1443 lines
41 KiB
HTML
1443 lines
41 KiB
HTML
<!DOCTYPE html>
|
||
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
|
||
<head>
|
||
<meta charset="utf-8" />
|
||
<meta name="generator" content="pandoc" />
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
|
||
<title>NATS cluster message speed — NS1 host study</title>
|
||
<style>
|
||
html {
|
||
color: #1a1a1a;
|
||
background-color: #fdfdfd;
|
||
}
|
||
body {
|
||
margin: 0 auto;
|
||
max-width: 36em;
|
||
padding-left: 50px;
|
||
padding-right: 50px;
|
||
padding-top: 50px;
|
||
padding-bottom: 50px;
|
||
hyphens: auto;
|
||
overflow-wrap: break-word;
|
||
text-rendering: optimizeLegibility;
|
||
font-kerning: normal;
|
||
}
|
||
@media (max-width: 600px) {
|
||
body {
|
||
font-size: 0.9em;
|
||
padding: 12px;
|
||
}
|
||
h1 {
|
||
font-size: 1.8em;
|
||
}
|
||
}
|
||
@media print {
|
||
html {
|
||
background-color: white;
|
||
}
|
||
body {
|
||
background-color: transparent;
|
||
color: black;
|
||
font-size: 12pt;
|
||
}
|
||
p, h2, h3 {
|
||
orphans: 3;
|
||
widows: 3;
|
||
}
|
||
h2, h3, h4 {
|
||
page-break-after: avoid;
|
||
}
|
||
}
|
||
p {
|
||
margin: 1em 0;
|
||
}
|
||
a {
|
||
color: #1a1a1a;
|
||
}
|
||
a:visited {
|
||
color: #1a1a1a;
|
||
}
|
||
img {
|
||
max-width: 100%;
|
||
}
|
||
svg {
|
||
height: auto;
|
||
max-width: 100%;
|
||
}
|
||
h1, h2, h3, h4, h5, h6 {
|
||
margin-top: 1.4em;
|
||
}
|
||
h5, h6 {
|
||
font-size: 1em;
|
||
font-style: italic;
|
||
}
|
||
h6 {
|
||
font-weight: normal;
|
||
}
|
||
ol, ul {
|
||
padding-left: 1.7em;
|
||
margin-top: 1em;
|
||
}
|
||
li > ol, li > ul {
|
||
margin-top: 0;
|
||
}
|
||
blockquote {
|
||
margin: 1em 0 1em 1.7em;
|
||
padding-left: 1em;
|
||
border-left: 2px solid #e6e6e6;
|
||
color: #606060;
|
||
}
|
||
code {
|
||
font-family: Menlo, Monaco, Consolas, 'Lucida Console', monospace;
|
||
font-size: 85%;
|
||
margin: 0;
|
||
hyphens: manual;
|
||
}
|
||
pre {
|
||
margin: 1em 0;
|
||
overflow: auto;
|
||
}
|
||
pre code {
|
||
padding: 0;
|
||
overflow: visible;
|
||
overflow-wrap: normal;
|
||
}
|
||
.sourceCode {
|
||
background-color: transparent;
|
||
overflow: visible;
|
||
}
|
||
hr {
|
||
background-color: #1a1a1a;
|
||
border: none;
|
||
height: 1px;
|
||
margin: 1em 0;
|
||
}
|
||
table {
|
||
margin: 1em 0;
|
||
border-collapse: collapse;
|
||
width: 100%;
|
||
overflow-x: auto;
|
||
display: block;
|
||
font-variant-numeric: lining-nums tabular-nums;
|
||
}
|
||
table caption {
|
||
margin-bottom: 0.75em;
|
||
}
|
||
tbody {
|
||
margin-top: 0.5em;
|
||
border-top: 1px solid #1a1a1a;
|
||
border-bottom: 1px solid #1a1a1a;
|
||
}
|
||
th {
|
||
border-top: 1px solid #1a1a1a;
|
||
padding: 0.25em 0.5em 0.25em 0.5em;
|
||
}
|
||
td {
|
||
padding: 0.125em 0.5em 0.25em 0.5em;
|
||
}
|
||
header {
|
||
margin-bottom: 4em;
|
||
text-align: center;
|
||
}
|
||
#TOC li {
|
||
list-style: none;
|
||
}
|
||
#TOC ul {
|
||
padding-left: 1.3em;
|
||
}
|
||
#TOC > ul {
|
||
padding-left: 0;
|
||
}
|
||
#TOC a:not(:hover) {
|
||
text-decoration: none;
|
||
}
|
||
code{white-space: pre-wrap;}
|
||
span.smallcaps{font-variant: small-caps;}
|
||
div.columns{display: flex; gap: min(4vw, 1.5em);}
|
||
div.column{flex: auto; overflow-x: auto;}
|
||
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
|
||
/* The extra [class] is a hack that increases specificity enough to
|
||
override a similar rule in reveal.js */
|
||
ul.task-list[class]{list-style: none;}
|
||
ul.task-list li input[type="checkbox"] {
|
||
font-size: inherit;
|
||
width: 0.8em;
|
||
margin: 0 0.8em 0.2em -1.6em;
|
||
vertical-align: middle;
|
||
}
|
||
.display.math{display: block; text-align: center; margin: 0.5rem auto;}
|
||
/* CSS for syntax highlighting */
|
||
pre > code.sourceCode { white-space: pre; position: relative; }
|
||
pre > code.sourceCode > span { line-height: 1.25; }
|
||
pre > code.sourceCode > span:empty { height: 1.2em; }
|
||
.sourceCode { overflow: visible; }
|
||
code.sourceCode > span { color: inherit; text-decoration: inherit; }
|
||
div.sourceCode { margin: 1em 0; }
|
||
pre.sourceCode { margin: 0; }
|
||
@media screen {
|
||
div.sourceCode { overflow: auto; }
|
||
}
|
||
@media print {
|
||
pre > code.sourceCode { white-space: pre-wrap; }
|
||
pre > code.sourceCode > span { display: inline-block; text-indent: -5em; padding-left: 5em; }
|
||
}
|
||
pre.numberSource code
|
||
{ counter-reset: source-line 0; }
|
||
pre.numberSource code > span
|
||
{ position: relative; left: -4em; counter-increment: source-line; }
|
||
pre.numberSource code > span > a:first-child::before
|
||
{ content: counter(source-line);
|
||
position: relative; left: -1em; text-align: right; vertical-align: baseline;
|
||
border: none; display: inline-block;
|
||
-webkit-touch-callout: none; -webkit-user-select: none;
|
||
-khtml-user-select: none; -moz-user-select: none;
|
||
-ms-user-select: none; user-select: none;
|
||
padding: 0 4px; width: 4em;
|
||
background-color: #232629;
|
||
color: #7a7c7d;
|
||
}
|
||
pre.numberSource { margin-left: 3em; border-left: 1px solid #7a7c7d; padding-left: 4px; }
|
||
div.sourceCode
|
||
{ color: #cfcfc2; background-color: #232629; }
|
||
@media screen {
|
||
pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; }
|
||
}
|
||
code span { color: #cfcfc2; } /* Normal */
|
||
code span.al { color: #95da4c; background-color: #4d1f24; font-weight: bold; } /* Alert */
|
||
code span.an { color: #3f8058; } /* Annotation */
|
||
code span.at { color: #2980b9; } /* Attribute */
|
||
code span.bn { color: #f67400; } /* BaseN */
|
||
code span.bu { color: #7f8c8d; } /* BuiltIn */
|
||
code span.cf { color: #fdbc4b; font-weight: bold; } /* ControlFlow */
|
||
code span.ch { color: #3daee9; } /* Char */
|
||
code span.cn { color: #27aeae; font-weight: bold; } /* Constant */
|
||
code span.co { color: #7a7c7d; } /* Comment */
|
||
code span.cv { color: #7f8c8d; } /* CommentVar */
|
||
code span.do { color: #a43340; } /* Documentation */
|
||
code span.dt { color: #2980b9; } /* DataType */
|
||
code span.dv { color: #f67400; } /* DecVal */
|
||
code span.er { color: #da4453; text-decoration: underline; } /* Error */
|
||
code span.ex { color: #0099ff; font-weight: bold; } /* Extension */
|
||
code span.fl { color: #f67400; } /* Float */
|
||
code span.fu { color: #8e44ad; } /* Function */
|
||
code span.im { color: #27ae60; } /* Import */
|
||
code span.in { color: #c45b00; } /* Information */
|
||
code span.kw { color: #cfcfc2; font-weight: bold; } /* Keyword */
|
||
code span.op { color: #cfcfc2; } /* Operator */
|
||
code span.ot { color: #27ae60; } /* Other */
|
||
code span.pp { color: #27ae60; } /* Preprocessor */
|
||
code span.re { color: #2980b9; background-color: #153042; } /* RegionMarker */
|
||
code span.sc { color: #3daee9; } /* SpecialChar */
|
||
code span.ss { color: #da4453; } /* SpecialString */
|
||
code span.st { color: #f44f4f; } /* String */
|
||
code span.va { color: #27aeae; } /* Variable */
|
||
code span.vs { color: #da4453; } /* VerbatimString */
|
||
code span.wa { color: #da4453; } /* Warning */
|
||
</style>
|
||
<style>/* Colored print + screen stylesheet for zapier.georgelambert.org */
|
||
:root {
|
||
--ink: #171a26;
|
||
--muted: #5b6178;
|
||
--line: #d9dce8;
|
||
--bg: #f4f5fb;
|
||
--paper: #ffffff;
|
||
--accent: #4f46e5;
|
||
--accent-deep: #312e81;
|
||
--accent-soft: #eef0fe;
|
||
--ok: #047857;
|
||
--warn: #8a5a00;
|
||
--code-bg: #1b1f33;
|
||
--code-fg: #e8ecff;
|
||
}
|
||
html { background: var(--bg); }
|
||
body {
|
||
margin: 0 auto;
|
||
padding: 1.5rem 1.25rem 3rem;
|
||
max-width: 48rem;
|
||
font: 15px/1.55 -apple-system, "Segoe UI", Georgia, serif;
|
||
color: var(--ink);
|
||
background: var(--paper);
|
||
}
|
||
.doc-banner {
|
||
background: linear-gradient(160deg, #312e81 0%, #4f46e5 60%, #7c74f0 100%);
|
||
color: #eef0fe;
|
||
margin: -1.5rem -1.25rem 1.5rem;
|
||
padding: 1.1rem 1.25rem 1rem;
|
||
}
|
||
.doc-banner a { color: #fff; }
|
||
.doc-banner .kicker {
|
||
letter-spacing: 0.12em;
|
||
text-transform: uppercase;
|
||
font: 700 10px system-ui, sans-serif;
|
||
opacity: 0.8;
|
||
}
|
||
.doc-banner h1 { margin: 0.25rem 0 0; font-size: 1.45rem; color: #fff; }
|
||
h1, h2, h3, h4 { color: var(--accent-deep); page-break-after: avoid; }
|
||
h1 { font-size: 1.7rem; }
|
||
h2 {
|
||
font-size: 1.2rem;
|
||
border-bottom: 2px solid var(--accent);
|
||
padding-bottom: 0.2rem;
|
||
margin-top: 1.6rem;
|
||
}
|
||
h3 { font-size: 1.05rem; color: var(--accent); }
|
||
a { color: var(--accent); }
|
||
p, li { orphans: 3; widows: 3; }
|
||
code {
|
||
font-family: ui-monospace, Menlo, Consolas, monospace;
|
||
font-size: 0.86em;
|
||
background: var(--accent-soft);
|
||
color: var(--accent-deep);
|
||
padding: 0.08em 0.28em;
|
||
border-radius: 4px;
|
||
}
|
||
pre, div.sourceCode, div.sourceCode pre {
|
||
background: var(--code-bg) !important;
|
||
color: var(--code-fg) !important;
|
||
padding: 0.85rem 1rem;
|
||
border-radius: 10px;
|
||
overflow: auto;
|
||
font-size: 0.78rem;
|
||
line-height: 1.4;
|
||
page-break-inside: avoid;
|
||
}
|
||
pre code { background: transparent; color: inherit; padding: 0; }
|
||
#title-block-header, header#title-block-header, h1.title { display: none; }
|
||
.doc-banner + h1 { display: none; }
|
||
table {
|
||
border-collapse: collapse;
|
||
width: 100%;
|
||
margin: 0.8rem 0 1.2rem;
|
||
font-size: 0.9rem;
|
||
page-break-inside: avoid;
|
||
}
|
||
th, td { border: 1px solid var(--line); padding: 0.38rem 0.55rem; text-align: left; vertical-align: top; }
|
||
th {
|
||
background: var(--accent);
|
||
color: #fff;
|
||
font: 650 12px system-ui, sans-serif;
|
||
}
|
||
tr:nth-child(even) td { background: var(--accent-soft); }
|
||
blockquote {
|
||
margin: 1rem 0;
|
||
padding: 0.4rem 0.9rem;
|
||
border-left: 4px solid var(--accent);
|
||
background: var(--accent-soft);
|
||
color: var(--accent-deep);
|
||
}
|
||
img { max-width: 100%; height: auto; border-radius: 8px; page-break-inside: avoid; }
|
||
hr { border: 0; border-top: 1px solid var(--line); }
|
||
ul, ol { padding-left: 1.25rem; }
|
||
nav.site { font: 13px system-ui, sans-serif; margin-bottom: 0.4rem; }
|
||
.source-path { font: 11px ui-monospace, Menlo, monospace; color: var(--muted); }
|
||
|
||
@page {
|
||
size: letter;
|
||
margin: 0.65in 0.7in 0.8in 0.7in;
|
||
@top-left {
|
||
content: "Verae Time × Zapier";
|
||
font: 700 8pt system-ui, sans-serif;
|
||
color: #4f46e5;
|
||
}
|
||
@top-right {
|
||
content: "zapier.georgelambert.org";
|
||
font: 8pt system-ui, sans-serif;
|
||
color: #6b7186;
|
||
}
|
||
@bottom-center {
|
||
content: counter(page) " / " counter(pages);
|
||
font: 8pt system-ui, sans-serif;
|
||
color: #6b7186;
|
||
}
|
||
}
|
||
@media print {
|
||
html, body { background: #fff; max-width: none; padding: 0; }
|
||
.doc-banner { margin: 0 0 1rem; border-radius: 8px; -webkit-print-color-adjust: exact; print-color-adjust: exact; }
|
||
a { text-decoration: none; }
|
||
th, tr:nth-child(even) td, pre, blockquote, code { -webkit-print-color-adjust: exact; print-color-adjust: exact; }
|
||
}
|
||
</style>
|
||
</head>
|
||
<body>
|
||
<div class="doc-banner"><nav class="site"><a href="/">zapier.georgelambert.org</a> · <a href="/index-md.html">Markdown indexes</a></nav><div class="kicker">Verae Time × Zapier · progress report · maximized NS1 study</div><h1>NATS cluster message speed — maximized (RAM disk + 8 cores)</h1><div class="source-path">packages/zapier-decisions/reports/nats-cluster-bench-ns1.md</div></div>
|
||
<header id="title-block-header">
|
||
<h1 class="title">NATS cluster message speed — NS1 host study</h1>
|
||
</header>
|
||
<p><strong>Progress report (maximized NS1 study)</strong> · run
|
||
<code>20260912T055851Z</code> (UTC)</p>
|
||
<blockquote>
|
||
<p><strong>Execution provenance.</strong> Every process for this study
|
||
ran on <strong>NS1.GEORGELAMBERT.ORG</strong>
|
||
(<code>70.88.205.138</code>): <code>maximize-ns1-study.sh</code>
|
||
(cores/RAM/<code>max_mem</code>/tmpfs), then
|
||
<code>study-on-ns1.sh</code>, <code>nats bench</code>,
|
||
<code>latency.mjs</code> (LXC 510), matplotlib, pandoc, weasyprint.
|
||
Traffic stayed on <code>vmbr1</code>. veth/10G was <strong>not</strong>
|
||
changed. After the ladder, JetStream was put back on ZFS and product
|
||
streams were re-created; <strong>8 cores / 16 GiB / max_mem 8G
|
||
stay</strong>.</p>
|
||
</blockquote>
|
||
<h2 id="measured-delta-vs-20260912t051237z">Measured delta vs
|
||
<code>20260912T051237Z</code></h2>
|
||
<p>Baseline: 1 core / 1 GiB / JetStream on ZFS. This run: 8 cores / 16
|
||
GiB / JetStream <strong>tmpfs</strong> (file r=3) plus extra
|
||
<strong>memory</strong> store rows. veth/10G unchanged.</p>
|
||
<table>
|
||
<thead>
|
||
<tr class="header">
|
||
<th>Metric</th>
|
||
<th>Baseline <code>20260912T051237Z</code></th>
|
||
<th>This run</th>
|
||
<th>Ratio</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="odd">
|
||
<td>Core 1p1s 128 B pub msgs/s</td>
|
||
<td>502,502</td>
|
||
<td>662,227</td>
|
||
<td>1.32×</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>Core 8p8s 128 B aggregate msgs/s</td>
|
||
<td>2,065,217</td>
|
||
<td>1,760,599</td>
|
||
<td>0.85×</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>JS file r=3 1p 128 B pub msgs/s</td>
|
||
<td>7,393</td>
|
||
<td>14,330</td>
|
||
<td>1.94×</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>JS file r=3 4p 128 B pub msgs/s</td>
|
||
<td>17,986</td>
|
||
<td>19,232</td>
|
||
<td>1.07×</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>JS file r=3 4p 1 KiB pub msgs/s</td>
|
||
<td>14,985</td>
|
||
<td>15,197</td>
|
||
<td>1.01×</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>JS memory r=3 1p 128 B pub msgs/s</td>
|
||
<td>—</td>
|
||
<td>20,188</td>
|
||
<td>—</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>JS memory r=3 4p 128 B pub msgs/s</td>
|
||
<td>—</td>
|
||
<td>37,736</td>
|
||
<td>—</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>Ping p99 (ms)</td>
|
||
<td>1.377ms</td>
|
||
<td>1.140ms</td>
|
||
<td>1.21× faster</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<figure>
|
||
<img src="charts/delta-vs-baseline.png"
|
||
alt="Baseline vs maximized publish rates (log)" />
|
||
<figcaption aria-hidden="true">Baseline vs maximized publish rates
|
||
(log)</figcaption>
|
||
</figure>
|
||
<p><em>Baseline vs maximized publish rates (log)</em></p>
|
||
<hr />
|
||
<h2 id="executive-summary">1. Executive summary</h2>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 24%" />
|
||
<col style="width: 76%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr class="header">
|
||
<th>Item</th>
|
||
<th>This NS1-host run</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="odd">
|
||
<td>Control plane</td>
|
||
<td>NS1.GEORGELAMBERT.ORG (<code>70.88.205.138</code>), user
|
||
<code>marchon</code></td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>Bench client</td>
|
||
<td>LXC 510 <code>verae-px-worker</code></td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>Brokers</td>
|
||
<td>LXC 511/512/513 <code>nats-a/b/c</code> on
|
||
<code>10.10.10.21–23</code></td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>Client URL</td>
|
||
<td><code>nats://10.10.10.21:4222,nats://10.10.10.22:4222,nats://10.10.10.23:4222</code></td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>Host load before</td>
|
||
<td><code>8.70 8.15 8.00 4/3842 621042</code></td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>Host load after</td>
|
||
<td><code>8.48 8.52 8.17 5/3860 657839</code></td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>Core 1p1s 128 B pub</td>
|
||
<td>662,227 msgs/s</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>JetStream 1p 128 B r=3</td>
|
||
<td>14,330 durable pubs/s</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>Ping p50 / p99</td>
|
||
<td>0.456ms / 1.140ms</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p>Product traffic is the JetStream row. Ping is one-message delay.
|
||
Flood is mailbox catch-up after a burst.</p>
|
||
<hr />
|
||
<h2 id="where-it-ran-and-where-it-did-not">2. Where it ran (and where it
|
||
did not)</h2>
|
||
<pre class="text"><code>Operator laptop ──ssh──► NS1.GEORGELAMBERT.ORG 70.88.205.138
|
||
study-on-ns1.sh
|
||
python3 build-ns1-study-report.py
|
||
sudo pct exec 510 ──► nats bench / latency.mjs
|
||
│
|
||
▼ vmbr1
|
||
10.10.10.21-23 :4222</code></pre>
|
||
<ul>
|
||
<li><strong>Did run on 138:</strong> bash, python3, matplotlib, pandoc,
|
||
weasyprint, <code>pct</code>, nats-server (in LXC), nats CLI and Node
|
||
(in LXC 510).</li>
|
||
<li><strong>Did not run on the laptop:</strong> no local
|
||
<code>nats bench</code>, no local charting, no local WeasyPrint for this
|
||
file.</li>
|
||
</ul>
|
||
<hr />
|
||
<h2 id="results-this-run">3. Results (this run)</h2>
|
||
<h3 id="host-and-brokers">Host and brokers</h3>
|
||
<p><strong>Before</strong></p>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 7%" />
|
||
<col style="width: 7%" />
|
||
<col style="width: 17%" />
|
||
<col style="width: 11%" />
|
||
<col style="width: 13%" />
|
||
<col style="width: 6%" />
|
||
<col style="width: 9%" />
|
||
<col style="width: 11%" />
|
||
<col style="width: 14%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr class="header">
|
||
<th>Node</th>
|
||
<th>VMID</th>
|
||
<th>connections</th>
|
||
<th>in_msgs</th>
|
||
<th>out_msgs</th>
|
||
<th>cpu</th>
|
||
<th>cores</th>
|
||
<th>mem (B)</th>
|
||
<th>jetstream</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="odd">
|
||
<td>nats-a</td>
|
||
<td>511</td>
|
||
<td>5</td>
|
||
<td>31,443</td>
|
||
<td>31,725</td>
|
||
<td>1</td>
|
||
<td>8</td>
|
||
<td>15,785,984</td>
|
||
<td>True</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>nats-b</td>
|
||
<td>512</td>
|
||
<td>1</td>
|
||
<td>24,330</td>
|
||
<td>24,480</td>
|
||
<td>1</td>
|
||
<td>1</td>
|
||
<td>15,892,480</td>
|
||
<td>True</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>nats-c</td>
|
||
<td>513</td>
|
||
<td>0</td>
|
||
<td>19,051</td>
|
||
<td>19,034</td>
|
||
<td>0</td>
|
||
<td>1</td>
|
||
<td>14,262,272</td>
|
||
<td>True</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p><strong>After</strong></p>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 7%" />
|
||
<col style="width: 7%" />
|
||
<col style="width: 17%" />
|
||
<col style="width: 11%" />
|
||
<col style="width: 13%" />
|
||
<col style="width: 6%" />
|
||
<col style="width: 9%" />
|
||
<col style="width: 11%" />
|
||
<col style="width: 14%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr class="header">
|
||
<th>Node</th>
|
||
<th>VMID</th>
|
||
<th>connections</th>
|
||
<th>in_msgs</th>
|
||
<th>out_msgs</th>
|
||
<th>cpu</th>
|
||
<th>cores</th>
|
||
<th>mem (B)</th>
|
||
<th>jetstream</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="odd">
|
||
<td>nats-a</td>
|
||
<td>511</td>
|
||
<td>5</td>
|
||
<td>724,962</td>
|
||
<td>1,400,288</td>
|
||
<td>2</td>
|
||
<td>8</td>
|
||
<td>27,095,040</td>
|
||
<td>True</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>nats-b</td>
|
||
<td>512</td>
|
||
<td>1</td>
|
||
<td>509,195</td>
|
||
<td>671,813</td>
|
||
<td>0</td>
|
||
<td>1</td>
|
||
<td>36,028,416</td>
|
||
<td>True</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>nats-c</td>
|
||
<td>513</td>
|
||
<td>0</td>
|
||
<td>703,508</td>
|
||
<td>1,715,907</td>
|
||
<td>0</td>
|
||
<td>1</td>
|
||
<td>27,295,744</td>
|
||
<td>True</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p>nproc=40 ·
|
||
uname=<code>Linux NS1.GEORGELAMBERT.ORG 6.17.2-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-1 (2025-10-21T11:55Z) x86_64 GNU/Linux</code></p>
|
||
<h3 id="throughput">Throughput</h3>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 6%" />
|
||
<col style="width: 8%" />
|
||
<col style="width: 24%" />
|
||
<col style="width: 16%" />
|
||
<col style="width: 13%" />
|
||
<col style="width: 16%" />
|
||
<col style="width: 13%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr class="header">
|
||
<th>Run</th>
|
||
<th>Mode</th>
|
||
<th>Aggregate msgs/s</th>
|
||
<th>Pub msgs/s</th>
|
||
<th>Pub MB/s</th>
|
||
<th>Sub msgs/s</th>
|
||
<th>Sub MB/s</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="odd">
|
||
<td><code>core-1p1s-50k-128</code></td>
|
||
<td>core pub/sub</td>
|
||
<td>835,602</td>
|
||
<td>662,227</td>
|
||
<td>80.84</td>
|
||
<td>472,239</td>
|
||
<td>57.65</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>core-4p4s-100k-128</code></td>
|
||
<td>core pub/sub</td>
|
||
<td>1,597,284</td>
|
||
<td>692,109</td>
|
||
<td>84.49</td>
|
||
<td>1,286,093</td>
|
||
<td>156.99</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><code>core-4p4s-50k-1k</code></td>
|
||
<td>core pub/sub</td>
|
||
<td>595,555</td>
|
||
<td>189,939</td>
|
||
<td>185.49</td>
|
||
<td>479,898</td>
|
||
<td>468.65</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>core-8p8s-200k-128</code></td>
|
||
<td>core pub/sub</td>
|
||
<td>1,760,599</td>
|
||
<td>223,078</td>
|
||
<td>27.23</td>
|
||
<td>1,570,171</td>
|
||
<td>191.67</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><code>js-1p-20k-128-r3</code></td>
|
||
<td>jetstream r=3 file</td>
|
||
<td>—</td>
|
||
<td>14,330</td>
|
||
<td>1.75</td>
|
||
<td>—</td>
|
||
<td>—</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>js-2p2s-20k-128-r3</code></td>
|
||
<td>jetstream r=3 file</td>
|
||
<td>18,449</td>
|
||
<td>9,252</td>
|
||
<td>1.13</td>
|
||
<td>9,228</td>
|
||
<td>1.13</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><code>js-4p-20k-1k-r3</code></td>
|
||
<td>jetstream r=3 file</td>
|
||
<td>—</td>
|
||
<td>15,197</td>
|
||
<td>14.84</td>
|
||
<td>—</td>
|
||
<td>—</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>js-4p-50k-128-r3</code></td>
|
||
<td>jetstream r=3 file</td>
|
||
<td>—</td>
|
||
<td>19,232</td>
|
||
<td>2.35</td>
|
||
<td>—</td>
|
||
<td>—</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><code>js-file-1p-20k-128-r1</code></td>
|
||
<td>jetstream r=3 file</td>
|
||
<td>—</td>
|
||
<td>18,888</td>
|
||
<td>2.31</td>
|
||
<td>—</td>
|
||
<td>—</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>js-file-1p-20k-4k-r3</code></td>
|
||
<td>jetstream r=3 file</td>
|
||
<td>—</td>
|
||
<td>8,673</td>
|
||
<td>33.88</td>
|
||
<td>—</td>
|
||
<td>—</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><code>js-file-4p-50k-128-r1</code></td>
|
||
<td>jetstream r=3 file</td>
|
||
<td>—</td>
|
||
<td>24,560</td>
|
||
<td>3.00</td>
|
||
<td>—</td>
|
||
<td>—</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>js-mem-1p-20k-128-r1</code></td>
|
||
<td>jetstream r=3 file</td>
|
||
<td>—</td>
|
||
<td>29,972</td>
|
||
<td>3.66</td>
|
||
<td>—</td>
|
||
<td>—</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><code>js-mem-1p-20k-128-r3</code></td>
|
||
<td>jetstream r=3 file</td>
|
||
<td>—</td>
|
||
<td>20,188</td>
|
||
<td>2.46</td>
|
||
<td>—</td>
|
||
<td>—</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>js-mem-4p-20k-1k-r3</code></td>
|
||
<td>jetstream r=3 file</td>
|
||
<td>—</td>
|
||
<td>33,916</td>
|
||
<td>33.12</td>
|
||
<td>—</td>
|
||
<td>—</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><code>js-mem-4p-50k-128-r1</code></td>
|
||
<td>jetstream r=3 file</td>
|
||
<td>—</td>
|
||
<td>64,923</td>
|
||
<td>7.93</td>
|
||
<td>—</td>
|
||
<td>—</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>js-mem-4p-50k-128-r3</code></td>
|
||
<td>jetstream r=3 file</td>
|
||
<td>—</td>
|
||
<td>37,736</td>
|
||
<td>4.61</td>
|
||
<td>—</td>
|
||
<td>—</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<h3 id="round-trip-delay">Round-trip delay</h3>
|
||
<table style="width:100%;">
|
||
<colgroup>
|
||
<col style="width: 8%" />
|
||
<col style="width: 9%" />
|
||
<col style="width: 11%" />
|
||
<col style="width: 9%" />
|
||
<col style="width: 9%" />
|
||
<col style="width: 8%" />
|
||
<col style="width: 8%" />
|
||
<col style="width: 8%" />
|
||
<col style="width: 8%" />
|
||
<col style="width: 8%" />
|
||
<col style="width: 8%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr class="header">
|
||
<th>Run</th>
|
||
<th>Kind</th>
|
||
<th>Count</th>
|
||
<th>Pubs</th>
|
||
<th>Size</th>
|
||
<th>min</th>
|
||
<th>avg</th>
|
||
<th>p50</th>
|
||
<th>p90</th>
|
||
<th>p99</th>
|
||
<th>max</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="odd">
|
||
<td><code>lat-ping-1k-128</code></td>
|
||
<td>ping (sequential RTT)</td>
|
||
<td>1000</td>
|
||
<td>1</td>
|
||
<td>128 B</td>
|
||
<td>0.340ms</td>
|
||
<td>0.530ms</td>
|
||
<td>0.456ms</td>
|
||
<td>0.827ms</td>
|
||
<td>1.140ms</td>
|
||
<td>2.910ms</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>lat-reconnect-200-128</code></td>
|
||
<td>flood (burst queueing)</td>
|
||
<td>200</td>
|
||
<td>1</td>
|
||
<td>128 B</td>
|
||
<td>0.366ms</td>
|
||
<td>0.540ms</td>
|
||
<td>0.503ms</td>
|
||
<td>0.619ms</td>
|
||
<td>1.750ms</td>
|
||
<td>3.206ms</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><code>lat-1p-5k-128</code></td>
|
||
<td>flood (burst queueing)</td>
|
||
<td>5000</td>
|
||
<td>1</td>
|
||
<td>128 B</td>
|
||
<td>98.399ms</td>
|
||
<td>124.452ms</td>
|
||
<td>125.413ms</td>
|
||
<td>140.165ms</td>
|
||
<td>145.770ms</td>
|
||
<td>146.002ms</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>lat-4p-5k-1k</code></td>
|
||
<td>flood (burst queueing)</td>
|
||
<td>5000</td>
|
||
<td>4</td>
|
||
<td>1024 B</td>
|
||
<td>125.336ms</td>
|
||
<td>157.915ms</td>
|
||
<td>157.392ms</td>
|
||
<td>176.826ms</td>
|
||
<td>178.015ms</td>
|
||
<td>178.776ms</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><code>lat-4p-10k-128</code></td>
|
||
<td>flood (burst queueing)</td>
|
||
<td>10000</td>
|
||
<td>4</td>
|
||
<td>128 B</td>
|
||
<td>157.463ms</td>
|
||
<td>205.870ms</td>
|
||
<td>207.044ms</td>
|
||
<td>237.035ms</td>
|
||
<td>239.449ms</td>
|
||
<td>240.473ms</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>lat-8p-20k-128</code></td>
|
||
<td>flood (burst queueing)</td>
|
||
<td>20000</td>
|
||
<td>8</td>
|
||
<td>128 B</td>
|
||
<td>230.898ms</td>
|
||
<td>361.458ms</td>
|
||
<td>372.512ms</td>
|
||
<td>443.799ms</td>
|
||
<td>458.198ms</td>
|
||
<td>460.270ms</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<h3 id="core-nats">Core NATS</h3>
|
||
<figure>
|
||
<img src="charts/core-throughput.png"
|
||
alt="Core NATS throughput at four loads (NS1 host run)" />
|
||
<figcaption aria-hidden="true">Core NATS throughput at four loads (NS1
|
||
host run)</figcaption>
|
||
</figure>
|
||
<p><em>Core NATS throughput at four loads (NS1 host run)</em> ###
|
||
Payload size (core)</p>
|
||
<figure>
|
||
<img src="charts/payload-size.png"
|
||
alt="Core NATS 128 B vs 1 KiB (NS1 host run)" />
|
||
<figcaption aria-hidden="true">Core NATS 128 B vs 1 KiB (NS1 host
|
||
run)</figcaption>
|
||
</figure>
|
||
<p><em>Core NATS 128 B vs 1 KiB (NS1 host run)</em> ### JetStream r=3
|
||
file</p>
|
||
<figure>
|
||
<img src="charts/js-throughput.png"
|
||
alt="JetStream durable publish rate (NS1 host run)" />
|
||
<figcaption aria-hidden="true">JetStream durable publish rate (NS1 host
|
||
run)</figcaption>
|
||
</figure>
|
||
<p><em>JetStream durable publish rate (NS1 host run)</em> ### Core vs
|
||
JetStream</p>
|
||
<figure>
|
||
<img src="charts/core-vs-js.png"
|
||
alt="Core vs JetStream publish rate, log scale (NS1 host run)" />
|
||
<figcaption aria-hidden="true">Core vs JetStream publish rate, log scale
|
||
(NS1 host run)</figcaption>
|
||
</figure>
|
||
<p><em>Core vs JetStream publish rate, log scale (NS1 host run)</em> ###
|
||
Delay</p>
|
||
<figure>
|
||
<img src="charts/delay-percentiles.png"
|
||
alt="Ping vs flood delay percentiles, log scale (NS1 host run)" />
|
||
<figcaption aria-hidden="true">Ping vs flood delay percentiles, log
|
||
scale (NS1 host run)</figcaption>
|
||
</figure>
|
||
<p><em>Ping vs flood delay percentiles, log scale (NS1 host
|
||
run)</em></p>
|
||
<hr />
|
||
<h2 id="study-methodology">4. Study methodology</h2>
|
||
<h3 id="question">4.1 Question</h3>
|
||
<p>On the NS1 test stand, what message <strong>throughput</strong> and
|
||
<strong>delay</strong> does the three-node <code>verae</code> JetStream
|
||
cluster deliver at several loads, and which part of the stack is the
|
||
limiter for product traffic (jobs, events, webhooks, archive)?</p>
|
||
<h3 id="hypotheses-stated-before-the-run">4.2 Hypotheses (stated before
|
||
the run)</h3>
|
||
<ol type="1">
|
||
<li><strong>H1 — Core vs JetStream.</strong> Fire-and-forget core NATS
|
||
is at least an order of magnitude faster than JetStream <strong>file +
|
||
replicas=3</strong>, because durable publish waits for a majority disk
|
||
replica.</li>
|
||
<li><strong>H2 — JetStream parallelism.</strong> Adding publishers does
|
||
<strong>not</strong> linearly increase JetStream write rate once the
|
||
replica log is saturated.</li>
|
||
<li><strong>H3 — Quiet delay.</strong> Sequential pub→sub round trip on
|
||
<code>vmbr1</code> is well under 1 ms p99 when the consumer is
|
||
waiting.</li>
|
||
<li><strong>H4 — Burst delay.</strong> If publishers dump a batch before
|
||
the subscriber drains, observed delay is <strong>queueing time</strong>,
|
||
roughly linear in backlog, not in cluster hop count.</li>
|
||
<li><strong>H5 — Payload.</strong> Moving 128 B → 1 KiB lowers message
|
||
rate and raises byte rate on core NATS; JetStream in this size band
|
||
stays replica/fsync bound.</li>
|
||
</ol>
|
||
<h3 id="independent-variables-what-we-changed">4.3 Independent variables
|
||
(what we changed)</h3>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 50%" />
|
||
<col style="width: 50%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr class="header">
|
||
<th>Factor</th>
|
||
<th>Levels</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="odd">
|
||
<td>Transport</td>
|
||
<td>Core NATS pub/sub vs JetStream file replicas=3</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>Publisher count</td>
|
||
<td>1, 2, 4, 8</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>Subscriber count</td>
|
||
<td>0 (JS publish-only), 1, 2, 4, 8</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>Message count</td>
|
||
<td>1k, 5k, 10k, 20k, 50k, 100k, 200k (by ladder step)</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>Payload</td>
|
||
<td>128 B, 1024 B</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>Delay mode</td>
|
||
<td><strong>ping</strong> (publish, wait, repeat) vs
|
||
<strong>flood</strong> (publish all, then drain)</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<h3 id="dependent-variables-what-we-recorded">4.4 Dependent variables
|
||
(what we recorded)</h3>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 30%" />
|
||
<col style="width: 46%" />
|
||
<col style="width: 23%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr class="header">
|
||
<th>Metric</th>
|
||
<th>Instrument</th>
|
||
<th>Unit</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="odd">
|
||
<td>Publish rate</td>
|
||
<td><code>nats bench</code> 0.1.6 Pub stats</td>
|
||
<td>msgs/s, MB/s</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>Subscribe rate</td>
|
||
<td><code>nats bench</code> Sub stats</td>
|
||
<td>msgs/s, MB/s</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>Aggregate</td>
|
||
<td><code>nats bench</code> NATS Pub/Sub stats</td>
|
||
<td>msgs/s (fan-out counts both sides)</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>Publisher spread</td>
|
||
<td>nats min/avg/max <strong>msgs/s</strong></td>
|
||
<td>not delay</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>One-way-ish RTT</td>
|
||
<td><code>latency.mjs</code> header timestamp</td>
|
||
<td>min, avg, p50, p90, p99, max</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>Host load</td>
|
||
<td><code>/proc/loadavg</code> before and after</td>
|
||
<td>load average</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>Broker counters</td>
|
||
<td><code>http://127.0.0.1:8222/varz</code> inside each nats LXC</td>
|
||
<td>connections, in/out msgs, cpu, mem</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p><strong>Important:</strong> nats CLI 0.1.6 min/avg/max are
|
||
<strong>rate spread across publishers</strong>, not microseconds of
|
||
delay. Delay is only <code>latency.mjs</code>.</p>
|
||
<h3 id="controls-and-constants">4.5 Controls and constants</h3>
|
||
<ul>
|
||
<li>Cluster name <code>verae</code>, three routes, client
|
||
<code>:4222</code>, cluster <code>:6222</code>, monitor loopback
|
||
<code>:8222</code>.</li>
|
||
<li>Client URL always the three-node list on <code>vmbr1</code> (never
|
||
host <code>127.0.0.1:4222</code>, never <code>vmbr0</code>).</li>
|
||
<li>Bench client is LXC <strong>510</strong>, not a nats-* server.</li>
|
||
<li>JetStream bench stream name <code>benchstream</code>,
|
||
<strong>file</strong> storage, <strong>replicas=3</strong>, deleted
|
||
between JS loads (<code>nats stream rm --force</code>) so names do not
|
||
collide.</li>
|
||
<li>Product streams were <strong>not</strong> the bench target (no load
|
||
test on <code>ZAPIER_*</code> / <code>VERAE_ARCHIVE</code>).</li>
|
||
<li>No TLS, no nkeys, no account isolation (isolation is
|
||
<code>vmbr1</code>).</li>
|
||
<li>Same nats CLI version (0.1.6) and <code>nats@2</code> Node client as
|
||
the first ladder.</li>
|
||
</ul>
|
||
<h3 id="procedure">4.6 Procedure</h3>
|
||
<ol type="1">
|
||
<li>Confirm this script is executing on
|
||
<strong>NS1.GEORGELAMBERT.ORG</strong>. Refuse otherwise.</li>
|
||
<li>Snapshot host load, memory, LXC configs, and each nats
|
||
<code>varz</code>.</li>
|
||
<li>From NS1, <code>pct exec 510</code> the core ladder (1p1s, 4p4s,
|
||
8p8s at 128 B; 4p4s at 1 KiB).</li>
|
||
<li>Delete <code>benchstream</code>; JS ladder (1p, 4p, 4p×1 KiB, 2p2s
|
||
pull) at replicas=3 file.</li>
|
||
<li>Copy <code>latency.mjs</code> into 510; ping then flood at several
|
||
batch sizes.</li>
|
||
<li>Snapshot host/<code>varz</code> again.</li>
|
||
<li>Parse logs on <strong>this host</strong>; draw charts; write HTML
|
||
and PDF on <strong>this host</strong>.</li>
|
||
</ol>
|
||
<p>No publish, subscribe, chart, or PDF process runs on the operator
|
||
laptop for this study.</p>
|
||
<h3 id="instrumentation-path">4.7 Instrumentation path</h3>
|
||
<pre class="text"><code>[NS1 host 70.88.205.138]
|
||
study-on-ns1.sh (bash + python3)
|
||
|
|
||
| sudo pct exec 510
|
||
v
|
||
[LXC 510 verae-px-worker 10.10.10.20]
|
||
nats bench / node latency.mjs
|
||
|
|
||
| NATS client protocol to
|
||
v
|
||
[LXC 511/512/513 10.10.10.21-23 :4222]
|
||
nats-server -js cluster routes :6222</code></pre>
|
||
<p>The hypervisor issues the guest commands. The messages themselves
|
||
never leave <code>vmbr1</code>.</p>
|
||
<h3 id="threats-to-validity">4.8 Threats to validity</h3>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 29%" />
|
||
<col style="width: 70%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr class="header">
|
||
<th>Threat</th>
|
||
<th>Effect on numbers</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="odd">
|
||
<td><strong>One physical host</strong></td>
|
||
<td>Three “replicas” share CPU, memory, and usually the same datastore.
|
||
This measures process/LXC HA, not disk HA.</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><strong>Shared load</strong></td>
|
||
<td>NS1 also runs Caddy, Forgejo, keep, fleet, portal, and other CTs.
|
||
Load average during a run is part of the result, not noise to
|
||
ignore.</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><strong>Single bench client</strong></td>
|
||
<td>All publishers live in 510. Per-publisher rate spread is contention
|
||
in that guest.</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><strong>Short runs</strong></td>
|
||
<td>Seconds of traffic. No compaction, no multi-hour page-cache
|
||
eviction, no snapshot during load.</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><strong>No TLS/nkeys</strong></td>
|
||
<td>Production auth will cost CPU. Do not treat these rates as
|
||
post-nkeys rates.</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><strong>Fan-out aggregate</strong></td>
|
||
<td>Core aggregate msgs/s counts pub+sub. Do not compare that column to
|
||
JetStream unique writes.</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><strong>Flood ≠ RTT</strong></td>
|
||
<td>Mixing flood averages with ping p99 produces a fake “NATS is slow”
|
||
story.</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><strong>Lab only</strong></td>
|
||
<td>Not a Zapier HTTPS bench and not live
|
||
<code>api.veraetime.net</code>.</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<h3 id="ethics-safety">4.9 Ethics / safety</h3>
|
||
<p>Bench uses throwaway subjects (<code>bench.core.*</code>,
|
||
<code>bench.js.*</code>, <code>bench.lat.*</code>) and a throwaway
|
||
stream. It does not purge product streams. Zapier cloud has no NATS
|
||
socket.</p>
|
||
<hr />
|
||
<h2 id="suggestions-for-fine-tuning">5. Suggestions for fine-tuning</h2>
|
||
<p>These follow from the method and from the first ladder on this stand
|
||
(JetStream ~16k durable 128 B pubs/s; ping ~0.3 ms; flood hundreds of
|
||
ms). Apply in order of leverage. Re-run <strong>this NS1 study</strong>
|
||
after each change so the delta is measured the same way.</p>
|
||
<h3 id="treat-jetstream-as-the-product-limiter">5.1 Treat JetStream as
|
||
the product limiter</h3>
|
||
<p>Product jobs/events/webhooks/archive are durable. Tuning core NATS to
|
||
2M msgs/s will not move a timestamp Zap. Put effort into <strong>replica
|
||
write path</strong> and <strong>consumer lag</strong>, not core
|
||
fan-out.</p>
|
||
<h3 id="split-storage-class-by-stream">5.2 Split storage class by
|
||
stream</h3>
|
||
<table style="width:100%;">
|
||
<colgroup>
|
||
<col style="width: 26%" />
|
||
<col style="width: 56%" />
|
||
<col style="width: 16%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr class="header">
|
||
<th>Stream</th>
|
||
<th>Suggested store</th>
|
||
<th>Why</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="odd">
|
||
<td><code>ZAPIER_JOBS</code></td>
|
||
<td>file, r=3</td>
|
||
<td>Work queue; lose-a-job is bad</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>ZAPIER_EVENTS</code></td>
|
||
<td>file r=3, or memory r=3 if events are rebuildable from job
|
||
status</td>
|
||
<td>Hot waiters; measure both</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><code>ZAPIER_WEBHOOKS</code></td>
|
||
<td>file, r=3, workqueue</td>
|
||
<td>HTTPS to Zapier is the slow consumer</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>ZAPIER_USAGE</code></td>
|
||
<td>file, r=3, limits + max-age</td>
|
||
<td>Telemetry</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><code>VERAE_ARCHIVE</code></td>
|
||
<td>file, r=3, on the <strong>best disk</strong></td>
|
||
<td>Puts are larger and must survive</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p>Try <code>ZAPIER_EVENTS</code> as memory store in a maintenance
|
||
window and re-run only the JS + ping/flood steps. If ping stays ~0.3 ms
|
||
and durable events still ack at a higher rate, keep it; if a CT restart
|
||
drops in-flight waiters, revert.</p>
|
||
<h3 id="give-jetstream-real-disks">5.3 Give JetStream real disks</h3>
|
||
<p>Today r=3 on three LXC guests on <strong>one Proxmox host</strong> is
|
||
three files, one failure domain.</p>
|
||
<ul>
|
||
<li>Bind-mount a distinct SSD/NVMe (or ZFS dataset with its own vdev)
|
||
into each nats LXC <code>store_dir</code>.</li>
|
||
<li>Set <code>sync: always</code> only on archive if you need it;
|
||
default sync is often enough for jobs and is faster. Measure.</li>
|
||
<li>Do not put JetStream <code>store_dir</code> on the same busy rootfs
|
||
as Forgejo/Caddy if we can avoid it.</li>
|
||
<li>When moving to three metal boxes: same configs, private NIC, one
|
||
disk (or mirror) <strong>per node</strong>. That is the first change
|
||
that makes r=3 mean “two boxes can die.”</li>
|
||
</ul>
|
||
<h3 id="isolate-the-nats-cts-from-the-rest-of-ns1">5.4 Isolate the nats
|
||
CTs from the rest of NS1</h3>
|
||
<p>Host load on this box is often already several. Pin:</p>
|
||
<ul>
|
||
<li><code>nats-a/b/c</code>: dedicated cores, no steal from keep/fleet
|
||
Node processes.</li>
|
||
<li>Memory high enough that file-backed streams stay cache-hot for the
|
||
working set.</li>
|
||
<li><code>cpuunits</code> / cpuset in <code>pct config</code> so a
|
||
Zapier-facing Node GC pause does not stall fsync.</li>
|
||
</ul>
|
||
<p>Re-run this study after pinning; H1/H2 should move more than
|
||
ping.</p>
|
||
<h3 id="consumer-and-mailbox-tuning-delay-h4">5.5 Consumer and mailbox
|
||
tuning (delay H4)</h3>
|
||
<p>Flood delay is backlog / consume_rate. Fine-tune the
|
||
<strong>waiters</strong>, not the broker RTT.</p>
|
||
<ul>
|
||
<li><code>jobs.events</code> and <code>webhooks.deliver</code>: raise
|
||
<code>max_ack_pending</code> so a slow HTTPS hook does not stall the
|
||
whole consumer; cap it so a poison message cannot unbounded-buffer
|
||
RAM.</li>
|
||
<li>Pull consumers: larger batch, shorter <code>expires</code>, more
|
||
pullers horizontally (fleet replica floors) instead of one fat
|
||
subscriber.</li>
|
||
<li>Middleware should <strong>not</strong> flood-publish then wait; it
|
||
already does per-job publish. Keep that. The flood test is the outage
|
||
profile when a consumer is stopped.</li>
|
||
<li>Alert on <strong>consumer lag</strong> (pending + ack pending) from
|
||
JetStream, not on ping RTT.</li>
|
||
</ul>
|
||
<h3 id="publisher-side-batching-in-middleware">5.6 Publisher-side
|
||
batching in middleware</h3>
|
||
<p>A timestamp job is one small JSON. 16k msgs/s is ample. Still:</p>
|
||
<ul>
|
||
<li>Avoid per-byte publishes; one message per job/event.</li>
|
||
<li>Reuse NATS connections (connection churn showed up as publisher
|
||
spread in the core 4p/8p runs).</li>
|
||
<li>Idempotent <code>msg id</code> / duplicate window sized to Verae
|
||
retry window, not default-only.</li>
|
||
</ul>
|
||
<h3 id="nats-server-knobs-worth-measuring-ab-with-this-script">5.7
|
||
nats-server knobs worth measuring (A/B with this script)</h3>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 33%" />
|
||
<col style="width: 66%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr class="header">
|
||
<th>Knob</th>
|
||
<th>Why try it</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="odd">
|
||
<td><code>max_payload</code></td>
|
||
<td>Keep default unless archive puts grow</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>write_deadline</code></td>
|
||
<td>Slow consumer protection for webhooks</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><code>max_pending</code></td>
|
||
<td>Bound memory on a stuck Zapier hook</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>max_connections</code></td>
|
||
<td>Fleet workers + keep + middleware</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>JetStream <code>max_file_store</code> /
|
||
<code>max_memory_store</code></td>
|
||
<td>Prevent one stream from filling the CT</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>max_outstanding_catchup</code></td>
|
||
<td>Replica restart after a nats-c blip</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>GOMAXPROCS = LXC cores</td>
|
||
<td>Do not overthread a 2-core CT</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p>Change <strong>one</strong> knob, re-run
|
||
<code>study-on-ns1.sh</code>, compare JetStream 1p 128 B and ping
|
||
p99.</p>
|
||
<h3 id="network">5.8 Network</h3>
|
||
<ul>
|
||
<li>Keep NATS off <code>vmbr0</code>. No change.</li>
|
||
<li>When on metal: dedicated NIC or VLAN for cluster <code>:6222</code>
|
||
vs client <code>:4222</code> if possible (replication vs client
|
||
load).</li>
|
||
<li>Check virtio queue counts on the LXC nics if core 1 KiB byte rate
|
||
plateaus.</li>
|
||
</ul>
|
||
<h3 id="security-cost-when-nkeysmtls-flip">5.9 Security cost (when
|
||
nkeys/mTLS flip)</h3>
|
||
<p><code>verae-nats-accounts</code> is still a sketch. Enabling accounts
|
||
will add CPU on publish. Budget: re-run this exact study
|
||
<strong>after</strong> creds are in every <code>NATS_URL</code>, and
|
||
accept a drop on both core and JS. Do not flip without that
|
||
measurement.</p>
|
||
<h3 id="operational-fine-tuning-lag-not-peak-msgss">5.10 Operational
|
||
fine-tuning (lag, not peak msgs/s)</h3>
|
||
<ol type="1">
|
||
<li>Scrape <code>varz</code> / <code>jsz</code> from the host over
|
||
<code>vmbr1</code> (not public). Monitor loopback <code>:8222</code> is
|
||
invisible to Prometheus on NS1 unless we add a host-side proxy on
|
||
<code>10.10.10.21:8222</code> bound only to <code>vmbr1</code>.</li>
|
||
<li>Keep replica floors for webhook-deliver and job-poller — they are
|
||
the flood defense.</li>
|
||
<li>Backup/restore drill of JetStream <strong>during idle</strong>, then
|
||
a short JS 1p run to see catchup cost.</li>
|
||
<li>A 15–30 minute soak (not in this ladder) for page cache and
|
||
compaction; add that as a third study when disks are dedicated.</li>
|
||
</ol>
|
||
<h3 id="what-not-to-tune">5.11 What not to tune</h3>
|
||
<ul>
|
||
<li>Do not chase core 8p8s aggregate. It is fan-out on a lab
|
||
bridge.</li>
|
||
<li>Do not treat flood 400 ms as “cluster RTT.” Fix consumers.</li>
|
||
<li>Do not load-test on <code>ZAPIER_*</code> streams.</li>
|
||
<li>Do not bind client NATS to <code>0.0.0.0</code> on
|
||
<code>vmbr0</code>.</li>
|
||
</ul>
|
||
<h3 id="recommended-next-experiments-same-method-one-change-each">5.12
|
||
Recommended next experiments (same method, one change each)</h3>
|
||
<ol type="1">
|
||
<li>CPU pin nats-a/b/c → re-run JS 1p + ping.</li>
|
||
<li><code>ZAPIER_EVENTS</code>-shaped memory stream vs file (throwaway
|
||
stream, same flags as this JS ladder).</li>
|
||
<li>Distinct <code>store_dir</code> disks per node.</li>
|
||
<li>nkeys on, same ladder.</li>
|
||
<li>Three hardware boxes, same <code>cluster.env</code> IPs
|
||
updated.</li>
|
||
</ol>
|
||
<p>Each experiment should produce a new
|
||
<code>results/<utc>/</code> on NS1 and a new progress-repo report
|
||
so we can diff H1–H5 instead of arguing from memory.</p>
|
||
<hr />
|
||
<h2 id="reproducing-this-study">6. Reproducing this study</h2>
|
||
<p>On <strong>NS1 only</strong>:</p>
|
||
<div class="sourceCode" id="cb3"><pre
|
||
class="sourceCode bash"><code class="sourceCode bash"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="bu">cd</span> ~/verae-src/verae-nats-cluster</span>
|
||
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a><span class="fu">bash</span> scripts/study-on-ns1.sh</span></code></pre></div>
|
||
<p>The script exits if <code>hostname</code> is not NS1. Outputs land in
|
||
<code>results/<utc>/</code> including
|
||
<code>nats-cluster-bench-ns1.{md,html,pdf}</code> and
|
||
<code>charts/</code>. Copy those into
|
||
<code>zapier-decisions/reports/</code> for the progress repo and
|
||
catalog.</p>
|
||
<p>Raw logs for this run: <code>results/20260912T055851Z/</code>.</p>
|
||
</body>
|
||
</html>
|