George Lambert
c83f1c6717
Some checks are pending
offline / test (push) Waiting to run
Compares r=1 vs r=3, file vs memory, reconnect tax, and projects three HP DL360 Gen10 NVMe + 10GbE boxes. Keep NATS; do not switch to MQTT/UDP.
853 lines
24 KiB
HTML
853 lines
24 KiB
HTML
<!DOCTYPE html>
|
||
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
|
||
<head>
|
||
<meta charset="utf-8" />
|
||
<meta name="generator" content="pandoc" />
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
|
||
<title>NATS optimal configuration study</title>
|
||
<style>
|
||
html {
|
||
color: #1a1a1a;
|
||
background-color: #fdfdfd;
|
||
}
|
||
body {
|
||
margin: 0 auto;
|
||
max-width: 36em;
|
||
padding-left: 50px;
|
||
padding-right: 50px;
|
||
padding-top: 50px;
|
||
padding-bottom: 50px;
|
||
hyphens: auto;
|
||
overflow-wrap: break-word;
|
||
text-rendering: optimizeLegibility;
|
||
font-kerning: normal;
|
||
}
|
||
@media (max-width: 600px) {
|
||
body {
|
||
font-size: 0.9em;
|
||
padding: 12px;
|
||
}
|
||
h1 {
|
||
font-size: 1.8em;
|
||
}
|
||
}
|
||
@media print {
|
||
html {
|
||
background-color: white;
|
||
}
|
||
body {
|
||
background-color: transparent;
|
||
color: black;
|
||
font-size: 12pt;
|
||
}
|
||
p, h2, h3 {
|
||
orphans: 3;
|
||
widows: 3;
|
||
}
|
||
h2, h3, h4 {
|
||
page-break-after: avoid;
|
||
}
|
||
}
|
||
p {
|
||
margin: 1em 0;
|
||
}
|
||
a {
|
||
color: #1a1a1a;
|
||
}
|
||
a:visited {
|
||
color: #1a1a1a;
|
||
}
|
||
img {
|
||
max-width: 100%;
|
||
}
|
||
svg {
|
||
height: auto;
|
||
max-width: 100%;
|
||
}
|
||
h1, h2, h3, h4, h5, h6 {
|
||
margin-top: 1.4em;
|
||
}
|
||
h5, h6 {
|
||
font-size: 1em;
|
||
font-style: italic;
|
||
}
|
||
h6 {
|
||
font-weight: normal;
|
||
}
|
||
ol, ul {
|
||
padding-left: 1.7em;
|
||
margin-top: 1em;
|
||
}
|
||
li > ol, li > ul {
|
||
margin-top: 0;
|
||
}
|
||
blockquote {
|
||
margin: 1em 0 1em 1.7em;
|
||
padding-left: 1em;
|
||
border-left: 2px solid #e6e6e6;
|
||
color: #606060;
|
||
}
|
||
code {
|
||
font-family: Menlo, Monaco, Consolas, 'Lucida Console', monospace;
|
||
font-size: 85%;
|
||
margin: 0;
|
||
hyphens: manual;
|
||
}
|
||
pre {
|
||
margin: 1em 0;
|
||
overflow: auto;
|
||
}
|
||
pre code {
|
||
padding: 0;
|
||
overflow: visible;
|
||
overflow-wrap: normal;
|
||
}
|
||
.sourceCode {
|
||
background-color: transparent;
|
||
overflow: visible;
|
||
}
|
||
hr {
|
||
background-color: #1a1a1a;
|
||
border: none;
|
||
height: 1px;
|
||
margin: 1em 0;
|
||
}
|
||
table {
|
||
margin: 1em 0;
|
||
border-collapse: collapse;
|
||
width: 100%;
|
||
overflow-x: auto;
|
||
display: block;
|
||
font-variant-numeric: lining-nums tabular-nums;
|
||
}
|
||
table caption {
|
||
margin-bottom: 0.75em;
|
||
}
|
||
tbody {
|
||
margin-top: 0.5em;
|
||
border-top: 1px solid #1a1a1a;
|
||
border-bottom: 1px solid #1a1a1a;
|
||
}
|
||
th {
|
||
border-top: 1px solid #1a1a1a;
|
||
padding: 0.25em 0.5em 0.25em 0.5em;
|
||
}
|
||
td {
|
||
padding: 0.125em 0.5em 0.25em 0.5em;
|
||
}
|
||
header {
|
||
margin-bottom: 4em;
|
||
text-align: center;
|
||
}
|
||
#TOC li {
|
||
list-style: none;
|
||
}
|
||
#TOC ul {
|
||
padding-left: 1.3em;
|
||
}
|
||
#TOC > ul {
|
||
padding-left: 0;
|
||
}
|
||
#TOC a:not(:hover) {
|
||
text-decoration: none;
|
||
}
|
||
code{white-space: pre-wrap;}
|
||
span.smallcaps{font-variant: small-caps;}
|
||
div.columns{display: flex; gap: min(4vw, 1.5em);}
|
||
div.column{flex: auto; overflow-x: auto;}
|
||
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
|
||
/* The extra [class] is a hack that increases specificity enough to
|
||
override a similar rule in reveal.js */
|
||
ul.task-list[class]{list-style: none;}
|
||
ul.task-list li input[type="checkbox"] {
|
||
font-size: inherit;
|
||
width: 0.8em;
|
||
margin: 0 0.8em 0.2em -1.6em;
|
||
vertical-align: middle;
|
||
}
|
||
.display.math{display: block; text-align: center; margin: 0.5rem auto;}
|
||
</style>
|
||
<style>/* Colored print + screen stylesheet for zapier.georgelambert.org */
|
||
:root {
|
||
--ink: #171a26;
|
||
--muted: #5b6178;
|
||
--line: #d9dce8;
|
||
--bg: #f4f5fb;
|
||
--paper: #ffffff;
|
||
--accent: #4f46e5;
|
||
--accent-deep: #312e81;
|
||
--accent-soft: #eef0fe;
|
||
--ok: #047857;
|
||
--warn: #8a5a00;
|
||
--code-bg: #1b1f33;
|
||
--code-fg: #e8ecff;
|
||
}
|
||
html { background: var(--bg); }
|
||
body {
|
||
margin: 0 auto;
|
||
padding: 1.5rem 1.25rem 3rem;
|
||
max-width: 48rem;
|
||
font: 15px/1.55 -apple-system, "Segoe UI", Georgia, serif;
|
||
color: var(--ink);
|
||
background: var(--paper);
|
||
}
|
||
.doc-banner {
|
||
background: linear-gradient(160deg, #312e81 0%, #4f46e5 60%, #7c74f0 100%);
|
||
color: #eef0fe;
|
||
margin: -1.5rem -1.25rem 1.5rem;
|
||
padding: 1.1rem 1.25rem 1rem;
|
||
}
|
||
.doc-banner a { color: #fff; }
|
||
.doc-banner .kicker {
|
||
letter-spacing: 0.12em;
|
||
text-transform: uppercase;
|
||
font: 700 10px system-ui, sans-serif;
|
||
opacity: 0.8;
|
||
}
|
||
.doc-banner h1 { margin: 0.25rem 0 0; font-size: 1.45rem; color: #fff; }
|
||
h1, h2, h3, h4 { color: var(--accent-deep); page-break-after: avoid; }
|
||
h1 { font-size: 1.7rem; }
|
||
h2 {
|
||
font-size: 1.2rem;
|
||
border-bottom: 2px solid var(--accent);
|
||
padding-bottom: 0.2rem;
|
||
margin-top: 1.6rem;
|
||
}
|
||
h3 { font-size: 1.05rem; color: var(--accent); }
|
||
a { color: var(--accent); }
|
||
p, li { orphans: 3; widows: 3; }
|
||
code {
|
||
font-family: ui-monospace, Menlo, Consolas, monospace;
|
||
font-size: 0.86em;
|
||
background: var(--accent-soft);
|
||
color: var(--accent-deep);
|
||
padding: 0.08em 0.28em;
|
||
border-radius: 4px;
|
||
}
|
||
pre, div.sourceCode, div.sourceCode pre {
|
||
background: var(--code-bg) !important;
|
||
color: var(--code-fg) !important;
|
||
padding: 0.85rem 1rem;
|
||
border-radius: 10px;
|
||
overflow: auto;
|
||
font-size: 0.78rem;
|
||
line-height: 1.4;
|
||
page-break-inside: avoid;
|
||
}
|
||
pre code { background: transparent; color: inherit; padding: 0; }
|
||
#title-block-header, header#title-block-header, h1.title { display: none; }
|
||
.doc-banner + h1 { display: none; }
|
||
table {
|
||
border-collapse: collapse;
|
||
width: 100%;
|
||
margin: 0.8rem 0 1.2rem;
|
||
font-size: 0.9rem;
|
||
page-break-inside: avoid;
|
||
}
|
||
th, td { border: 1px solid var(--line); padding: 0.38rem 0.55rem; text-align: left; vertical-align: top; }
|
||
th {
|
||
background: var(--accent);
|
||
color: #fff;
|
||
font: 650 12px system-ui, sans-serif;
|
||
}
|
||
tr:nth-child(even) td { background: var(--accent-soft); }
|
||
blockquote {
|
||
margin: 1rem 0;
|
||
padding: 0.4rem 0.9rem;
|
||
border-left: 4px solid var(--accent);
|
||
background: var(--accent-soft);
|
||
color: var(--accent-deep);
|
||
}
|
||
img { max-width: 100%; height: auto; border-radius: 8px; page-break-inside: avoid; }
|
||
hr { border: 0; border-top: 1px solid var(--line); }
|
||
ul, ol { padding-left: 1.25rem; }
|
||
nav.site { font: 13px system-ui, sans-serif; margin-bottom: 0.4rem; }
|
||
.source-path { font: 11px ui-monospace, Menlo, monospace; color: var(--muted); }
|
||
|
||
@page {
|
||
size: letter;
|
||
margin: 0.65in 0.7in 0.8in 0.7in;
|
||
@top-left {
|
||
content: "Verae Time × Zapier";
|
||
font: 700 8pt system-ui, sans-serif;
|
||
color: #4f46e5;
|
||
}
|
||
@top-right {
|
||
content: "zapier.georgelambert.org";
|
||
font: 8pt system-ui, sans-serif;
|
||
color: #6b7186;
|
||
}
|
||
@bottom-center {
|
||
content: counter(page) " / " counter(pages);
|
||
font: 8pt system-ui, sans-serif;
|
||
color: #6b7186;
|
||
}
|
||
}
|
||
@media print {
|
||
html, body { background: #fff; max-width: none; padding: 0; }
|
||
.doc-banner { margin: 0 0 1rem; border-radius: 8px; -webkit-print-color-adjust: exact; print-color-adjust: exact; }
|
||
a { text-decoration: none; }
|
||
th, tr:nth-child(even) td, pre, blockquote, code { -webkit-print-color-adjust: exact; print-color-adjust: exact; }
|
||
}
|
||
</style>
|
||
</head>
|
||
<body>
|
||
<div class="doc-banner"><nav class="site"><a href="/">zapier.georgelambert.org</a></nav><div class="kicker">Verae Time × Zapier · progress report</div><h1>NATS optimal configuration study</h1><div class="source-path">packages/zapier-decisions/reports/optimal-config/REPORT.md</div></div>
|
||
<header id="title-block-header">
|
||
<h1 class="title">NATS optimal configuration study</h1>
|
||
</header>
|
||
<p><strong>Progress report — optimal configuration study</strong> ·
|
||
<code>20260912T055851Z</code> (UTC) · all code on
|
||
<strong>NS1.GEORGELAMBERT.ORG</strong> (<code>70.88.205.138</code>)</p>
|
||
<p>This document folds every ladder we have run (1-core ZFS,
|
||
NS1-orchestrated, tmpfs maximize, and this exhaustive 8c/16G
|
||
<strong>ZFS</strong> factorial) plus UDP / MQTT / reconnect probes. It
|
||
recommends a lab config and a <strong>three-box HP DL360 Gen10</strong>
|
||
projection. veth/10G was not changed.</p>
|
||
<hr />
|
||
<h2 id="verdict-read-this-first">1. Verdict (read this first)</h2>
|
||
<p><strong>Keep NATS + JetStream.</strong> Do not replace the fabric
|
||
with MQTT, UDP, or a custom persistent-socket protocol for Verae
|
||
jobs/events/archive. Those are either slower, less durable, or already
|
||
what NATS is.</p>
|
||
<p><strong>Lab (NS1, one host, three LXC) — optimal now</strong></p>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 25%" />
|
||
<col style="width: 28%" />
|
||
<col style="width: 31%" />
|
||
<col style="width: 15%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr class="header">
|
||
<th>Stream</th>
|
||
<th>Storage</th>
|
||
<th>Replicas</th>
|
||
<th>Why</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="odd">
|
||
<td><code>ZAPIER_JOBS</code>, <code>ZAPIER_WEBHOOKS</code>,
|
||
<code>VERAE_ARCHIVE</code></td>
|
||
<td><strong>file</strong> (ZFS)</td>
|
||
<td><strong>3</strong></td>
|
||
<td>Survive a nats LXC death; archive must persist</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>ZAPIER_EVENTS</code></td>
|
||
<td><strong>memory</strong></td>
|
||
<td><strong>3</strong></td>
|
||
<td>Waiters are latency-sensitive; events rebuild from job status</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><code>ZAPIER_USAGE</code></td>
|
||
<td>file</td>
|
||
<td>3</td>
|
||
<td>Telemetry, limits + max-age</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p>Keep <strong>8 cores / 16 GiB / <code>max_mem: 8G</code></strong> on
|
||
510–513 (already live). Do <strong>not</strong> leave JetStream on
|
||
tmpfs. Do <strong>not</strong> drop product streams to r=1. Reuse
|
||
<strong>one NATS connection per process</strong> (already true in
|
||
middleware); never connect-per-message.</p>
|
||
<p><strong>Metal (3× DL360 Gen10) — optimal later</strong></p>
|
||
<p>Same stream table. File store on <strong>local NVMe/M.2</strong>, not
|
||
a shared SAN. Cluster + client on <strong>10GbE</strong> (or 25GbE if
|
||
you already have it). Dual Gold Xeon is surplus CPU for this workload;
|
||
8–16 cores dedicated to <code>nats-server</code> is enough. Expected JS
|
||
file r=3: <strong>~40–80k</strong> 128 B pubs/s (about
|
||
<strong>3–6×</strong> this lab’s 8c ZFS 1p, <strong>2–4×</strong> tmpfs
|
||
1p) — bounded by <strong>10GbE replica RTT</strong>, not by Xeon clocks.
|
||
Core NATS will sit in the <strong>1–3M msgs/s</strong> band until the
|
||
NIC saturates (~9 Gbit/s ≈ 8–9M × 128 B theoretical; CPU and client will
|
||
hit first).</p>
|
||
<hr />
|
||
<h2 id="what-we-actually-ran-this-exhaustive-pass">2. What we actually
|
||
ran (this exhaustive pass)</h2>
|
||
<p>Live cluster during this run: LXC 510–513 <strong>8 cores / 16
|
||
GiB</strong>, JetStream <strong>on ZFS</strong> (tmpfs from the maximize
|
||
study was already unmounted). Extra factorial: file/memory × replicas
|
||
1/3, 4 KiB file r=3, reconnect-per-message ping, UDP echo 510→511, MQTT
|
||
QoS0 against nats-a <code>:1883</code>. Product streams were not the
|
||
bench target.</p>
|
||
<h3 id="cross-study-history">2.1 Cross-study history</h3>
|
||
<table style="width:100%;">
|
||
<colgroup>
|
||
<col style="width: 16%" />
|
||
<col style="width: 16%" />
|
||
<col style="width: 16%" />
|
||
<col style="width: 16%" />
|
||
<col style="width: 16%" />
|
||
<col style="width: 16%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr class="header">
|
||
<th>Study</th>
|
||
<th>Env</th>
|
||
<th>Core 1p pub</th>
|
||
<th>JS file r=3 1p</th>
|
||
<th>JS mem r=3 4p</th>
|
||
<th>Ping p99</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="odd">
|
||
<td><code>20260912T051237Z</code></td>
|
||
<td>1c/1G ZFS (NS1 orch.)</td>
|
||
<td>502,502</td>
|
||
<td>7,393</td>
|
||
<td>—</td>
|
||
<td>1.377ms</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>20260912T053120Z</code></td>
|
||
<td>8c/16G tmpfs + mem extra</td>
|
||
<td>599,004</td>
|
||
<td>17,388</td>
|
||
<td>36,355</td>
|
||
<td>0.684ms</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><code>20260912T055851Z</code></td>
|
||
<td>8c/16G ZFS exhaustive <code>20260912T055851Z</code></td>
|
||
<td>662,227</td>
|
||
<td>14,330</td>
|
||
<td>37,736</td>
|
||
<td>1.140ms</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<figure>
|
||
<img src="charts-optimal/history-js1p.png"
|
||
alt="JS 1p file r=3 history" />
|
||
<figcaption aria-hidden="true">JS 1p file r=3 history</figcaption>
|
||
</figure>
|
||
<h3 id="this-run-jetstream-factorial">2.2 This run — JetStream
|
||
factorial</h3>
|
||
<table>
|
||
<thead>
|
||
<tr class="header">
|
||
<th>Run</th>
|
||
<th>What</th>
|
||
<th>Pub msgs/s</th>
|
||
<th>Pub MB/s</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="odd">
|
||
<td><code>js-file-1p-20k-128-r1</code></td>
|
||
<td>file r=1 1p 128 B</td>
|
||
<td>18,888</td>
|
||
<td>2.31</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>js-file-4p-50k-128-r1</code></td>
|
||
<td>file r=1 4p 128 B</td>
|
||
<td>24,560</td>
|
||
<td>3.00</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><code>js-1p-20k-128-r3</code></td>
|
||
<td>file r=3 1p 128 B</td>
|
||
<td>14,330</td>
|
||
<td>1.75</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>js-4p-50k-128-r3</code></td>
|
||
<td>file r=3 4p 128 B</td>
|
||
<td>19,232</td>
|
||
<td>2.35</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><code>js-4p-20k-1k-r3</code></td>
|
||
<td>file r=3 4p 1 KiB</td>
|
||
<td>15,197</td>
|
||
<td>14.84</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>js-file-1p-20k-4k-r3</code></td>
|
||
<td>file r=3 1p 4 KiB</td>
|
||
<td>8,673</td>
|
||
<td>33.88</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><code>js-mem-1p-20k-128-r1</code></td>
|
||
<td>memory r=1 1p 128 B</td>
|
||
<td>29,972</td>
|
||
<td>3.66</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>js-mem-4p-50k-128-r1</code></td>
|
||
<td>memory r=1 4p 128 B</td>
|
||
<td>64,923</td>
|
||
<td>7.93</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><code>js-mem-1p-20k-128-r3</code></td>
|
||
<td>memory r=3 1p 128 B</td>
|
||
<td>20,188</td>
|
||
<td>2.46</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><code>js-mem-4p-50k-128-r3</code></td>
|
||
<td>memory r=3 4p 128 B</td>
|
||
<td>37,736</td>
|
||
<td>4.61</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><code>js-mem-4p-20k-1k-r3</code></td>
|
||
<td>memory r=3 4p 1 KiB</td>
|
||
<td>33,916</td>
|
||
<td>33.12</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p>Replica <strong>1 vs 3</strong> on this stand (file 1p 128 B): r=1 is
|
||
18,888 vs r=3 14,330 (1.32× if r=3 is the slower one). Memory r=1 1p
|
||
29,972 vs memory r=3 20,188.</p>
|
||
<figure>
|
||
<img src="charts-optimal/replicas.png" alt="Replica cost" />
|
||
<figcaption aria-hidden="true">Replica cost</figcaption>
|
||
</figure>
|
||
<h3 id="delay-reconnect-tax-udp-mqtt">2.3 Delay, reconnect tax, UDP,
|
||
MQTT</h3>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 29%" />
|
||
<col style="width: 33%" />
|
||
<col style="width: 37%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr class="header">
|
||
<th>Probe</th>
|
||
<th>Result</th>
|
||
<th>Meaning</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="odd">
|
||
<td>NATS ping (persistent sockets) p50 / p99</td>
|
||
<td>0.456ms / 1.140ms</td>
|
||
<td>Quiet hop with a long-lived TCP conn</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>NATS <strong>reconnect-per-message</strong> p50 / p99</td>
|
||
<td>0.503ms / 1.750ms</td>
|
||
<td>TCP+NATS handshake on every pub — this is the tax to avoid</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>UDP echo 510→511 p99</td>
|
||
<td>0.363ms</td>
|
||
<td>Raw datagram ceiling on the same veth (no NATS)</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>MQTT QoS0 5k×128 B</td>
|
||
<td>44862 pubs/s</td>
|
||
<td>nats-server MQTT gateway on <code>:1883</code></td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p>Core 1p1s 128 B this run: 662,227 pub msgs/s. Flood delay is still
|
||
backlog/consume_rate, not RTT.</p>
|
||
<hr />
|
||
<h2 id="alternative-transports-why-we-are-not-switching-the-fabric">3.
|
||
Alternative transports (why we are not switching the fabric)</h2>
|
||
<p>NATS already <strong>is</strong> persistent TCP sockets with a tiny
|
||
binary protocol, automatic reconnect, and optional JetStream durability.
|
||
“Reduce connection overhead” is a <strong>client</strong> discipline:
|
||
hold the connection. The reconnect probe exists to prove that opening a
|
||
socket per job would dominate ping RTT.</p>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 7%" />
|
||
<col style="width: 44%" />
|
||
<col style="width: 32%" />
|
||
<col style="width: 15%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr class="header">
|
||
<th>Idea</th>
|
||
<th>Fit for Verae jobs/events/archive</th>
|
||
<th>Throughput vs NATS core</th>
|
||
<th>Durability</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="odd">
|
||
<td><strong>NATS core pub/sub</strong></td>
|
||
<td>Fan-out, request-reply (<code>verae.billing.*</code>)</td>
|
||
<td>Highest we measured (~0.5–2M msgs/s)</td>
|
||
<td>None</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><strong>NATS JetStream file r=3</strong></td>
|
||
<td>Jobs, webhooks, archive</td>
|
||
<td>~8–23k on this lab; see metal projection</td>
|
||
<td>Disk + 1-node loss</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><strong>NATS JetStream memory r=3</strong></td>
|
||
<td>Events mailbox</td>
|
||
<td>~22–36k on this lab</td>
|
||
<td>RAM + 1-node loss; <strong>empty on full restart</strong></td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><strong>MQTT</strong> (NATS gateway or Mosquitto)</td>
|
||
<td>IoT endpoints that already speak MQTT</td>
|
||
<td>This probe: 44862 pubs/s QoS0 — typically <strong>well
|
||
below</strong> NATS core; QoS1 ≈ JetStream-ish with more chatter</td>
|
||
<td>QoS1/2 session state; not our WORM model</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><strong>UDP</strong></td>
|
||
<td>Telemetry that may drop</td>
|
||
<td>RTT 0.363ms p99 — fastest hop, <strong>no</strong> reliability, no
|
||
cluster, no auth</td>
|
||
<td>None</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><strong>Custom persistent sockets / HTTP long-poll</strong></td>
|
||
<td>Worse NATS</td>
|
||
<td>You would re-implement reconnect, flow control, and fan-out</td>
|
||
<td>DIY</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><strong>WebSocket</strong></td>
|
||
<td>Browsers only</td>
|
||
<td>Extra framing; NATS already has WS for UIs, not for middleware</td>
|
||
<td>Same as core/JS behind it</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td><strong>QUIC / WebTransport</strong></td>
|
||
<td>Lossy WAN / browsers</td>
|
||
<td>NATS QUIC is not the lab path; 10GbE LAN does not need it</td>
|
||
<td>Same</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td><strong>Kafka / Redis streams</strong></td>
|
||
<td>Heavy log replay</td>
|
||
<td>Higher ops cost; not on <code>vmbr1</code> today</td>
|
||
<td>Yes, heavier</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p><strong>MQTT:</strong> NATS documents MQTT as an <em>enabling</em>
|
||
gateway for existing IoT, and prefers NATS end-to-end for greenfield.
|
||
Zapier cloud never talks NATS or MQTT; it talks HTTPS. Putting MQTT in
|
||
the middle of timestamp jobs adds protocol translation and QoS timers
|
||
without helping <code>jobId → events</code>. Use MQTT only if a device
|
||
already cannot speak NATS.</p>
|
||
<p><strong>UDP:</strong> Fine as a <em>measurement</em> of veth RTT.
|
||
Unusable as the job fabric (no ack, no replica, no flow control). NATS
|
||
ping is already within a small multiple of UDP on this bridge.</p>
|
||
<p><strong>Persistence sockets:</strong> Middleware and keep already
|
||
keep <code>NATS_URL</code> connections open. Optimal: one connection (or
|
||
a small pool) per process, <code>max_reconnect</code>, jitter, no
|
||
<code>connect()</code> in the per-job path. The reconnect ladder is the
|
||
anti-pattern.</p>
|
||
<hr />
|
||
<h2 id="optimal-configurations">4. Optimal configurations</h2>
|
||
<h3 id="ns1-lab-now">4.1 NS1 lab (now)</h3>
|
||
<ol type="1">
|
||
<li><strong>Leave 8 cores / 16 GiB</strong> on nats-a/b/c and the
|
||
worker. Host has 40 cores / 377 GiB; this is cheap.</li>
|
||
<li><strong><code>max_mem: 8G</code></strong> stays. Required for memory
|
||
streams.</li>
|
||
<li><strong>File r=3 on ZFS</strong> for jobs/webhooks/archive. tmpfs
|
||
doubled JS 1p (7.4k→17k) but <strong>loses the stream on reboot</strong>
|
||
— unacceptable for archive.</li>
|
||
<li><strong>Memory r=3 for <code>ZAPIER_EVENTS</code></strong> if we
|
||
accept “all three nats CTs reboot ⇒ in-flight waiters fall back to HTTP
|
||
poll.” That matches the designed wait path
|
||
(<code>GET /api/status/{jobId}</code>).</li>
|
||
<li><strong>r=1 only for throwaway benches</strong>, never product
|
||
streams. Replica=3 is the point of three guests.</li>
|
||
<li><strong>veth on vmbr1, no fake 10G NICs.</strong> Already 10000Mb/s;
|
||
JS does not fill it.</li>
|
||
<li><strong>Pin cpusets</strong> later if keep/fleet steal; not required
|
||
to beat these numbers.</li>
|
||
<li>Clients: persistent NATS connections; pull consumers with bounded
|
||
<code>max_ack_pending</code> for webhooks.</li>
|
||
</ol>
|
||
<h3 id="three-hp-dl360-gen10-projection-not-measured">4.2 Three HP DL360
|
||
Gen10 (projection — not measured)</h3>
|
||
<p>Assumed bill of materials (state it in the buy):</p>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 36%" />
|
||
<col style="width: 63%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr class="header">
|
||
<th>Piece</th>
|
||
<th>Assumption</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="odd">
|
||
<td>Chassis</td>
|
||
<td>3× DL360 Gen10 1U</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>CPU</td>
|
||
<td>Dual 2nd-gen Xeon <strong>Gold</strong> (e.g. 6226R 16c or 6248 20c
|
||
— <strong>32–40 cores/box</strong>)</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>Memory</td>
|
||
<td>DDR4-2933, <strong>192–384 GiB</strong>/box (6–12×32 GiB); NATS will
|
||
not use most of it</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>Storage</td>
|
||
<td><strong>NVMe M.2 or U.2</strong> for
|
||
<code>/var/lib/nats/jetstream</code> (XFS or ext4, <strong>not</strong>
|
||
shared ZFS over the network). RAID1 of two NVMe if you want disk HA
|
||
<em>inside</em> a box</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>Network</td>
|
||
<td><strong>10GbE</strong> (FlexibleLOM or PCIe); dedicated VLAN for
|
||
<code>:4222</code>+<code>:6222</code>. Do not share with public
|
||
<code>vmbr0</code> traffic</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>OS</td>
|
||
<td>Debian/Ubuntu bare metal, <code>nats-server</code> systemd, same
|
||
<code>nats.conf</code> as lab (bind private IP only)</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p><strong>What changes vs NS1 LXC</strong></p>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 15%" />
|
||
<col style="width: 20%" />
|
||
<col style="width: 18%" />
|
||
<col style="width: 45%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr class="header">
|
||
<th>Factor</th>
|
||
<th>NS1 today</th>
|
||
<th>3× DL360</th>
|
||
<th>Effect on JS file r=3</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="odd">
|
||
<td>Failure domain</td>
|
||
<td>1 Proxmox host</td>
|
||
<td>3 chassis, 3 NVMe, 3 NICs</td>
|
||
<td>r=3 <strong>means</strong> something</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>Disk</td>
|
||
<td>Shared ZFS SSD2</td>
|
||
<td>Local NVMe fsync ~50–150 µs</td>
|
||
<td>Big win vs ZFS; similar to tmpfs for sequential 128 B</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>Replica path</td>
|
||
<td>veth/bridge (~µs–tens of µs)</td>
|
||
<td>10GbE RTT typically <strong>50–200 µs</strong></td>
|
||
<td><strong>Slower than same-host tmpfs</strong>, faster than a bad
|
||
SAN</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>CPU</td>
|
||
<td>8 of 40 shared</td>
|
||
<td>32–40 dedicated Gold cores</td>
|
||
<td>Headroom for many clients, not 10× JS</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>NIC</td>
|
||
<td>software 10G veth, already ~5 Gbit/s core</td>
|
||
<td>real 10GbE ~9 Gbit/s TCP</td>
|
||
<td>Core NATS can grow; JS r=3 stays replica-bound</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p><strong>Projected bands</strong> (128 B, 3-node cluster, dedicated
|
||
10GbE, local NVMe, 8+ cores pinned to nats-server):</p>
|
||
<table>
|
||
<colgroup>
|
||
<col style="width: 16%" />
|
||
<col style="width: 34%" />
|
||
<col style="width: 29%" />
|
||
<col style="width: 19%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr class="header">
|
||
<th>Workload</th>
|
||
<th>NS1 measured (best)</th>
|
||
<th>DL360 projection</th>
|
||
<th>Confidence</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="odd">
|
||
<td>Core pub/sub 1p</td>
|
||
<td>0.5–0.8M</td>
|
||
<td><strong>0.8–2M</strong></td>
|
||
<td>Medium — NIC + syscall, plenty of CPU</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>Core 4p4s 1 KiB</td>
|
||
<td>~0.6–0.7M (~0.6 GB/s)</td>
|
||
<td><strong>~1M msgs/s / ~1 GB/s</strong> approaching 10GbE</td>
|
||
<td>Medium</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>JS file r=1</td>
|
||
<td>this run r=1</td>
|
||
<td><strong>80–200k</strong> pubs/s</td>
|
||
<td>Medium — NVMe + no replica wait</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>JS file r=3</td>
|
||
<td>7–23k (ZFS/tmpfs)</td>
|
||
<td><strong>40–80k</strong> pubs/s</td>
|
||
<td>Medium-low — replica RTT dominates; 3 NVMe still help vs shared
|
||
ZFS</td>
|
||
</tr>
|
||
<tr class="odd">
|
||
<td>JS memory r=3</td>
|
||
<td>22–36k</td>
|
||
<td><strong>50–100k</strong></td>
|
||
<td>Medium-low — RAM + 10GbE ack</td>
|
||
</tr>
|
||
<tr class="even">
|
||
<td>Ping p99</td>
|
||
<td>0.7–1.4 ms</td>
|
||
<td><strong>0.2–0.6 ms</strong></td>
|
||
<td>Medium — real NIC but no Proxmox tax</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p>These are <strong>not</strong> DL360 measurements. Scale from: (a)
|
||
our replica-1 vs replica-3 ratio once this run’s r=1 numbers exist, (b)
|
||
tmpfs vs ZFS ratio (2.35× on 1p), (c) Synadia/nats bench async file r=1
|
||
~100–400k on NVMe loopback, derated for 10GbE RTT.</p>
|
||
<p><strong>Buy notes:</strong> M.2 via Dual uFF / enablement kit; put
|
||
JetStream on NVMe <strong>directly</strong>, not behind a RAID
|
||
controller write-through unless you measure. 1GbE onboard is a trap —
|
||
use 10GbE for <code>:6222</code>. Dual Gold is for isolation (nats vs
|
||
worm/tree vs OS), not because JS needs 56 cores.</p>
|
||
<hr />
|
||
<h2 id="what-we-are-not-doing">5. What we are not doing</h2>
|
||
<ul>
|
||
<li>MQTT as the Zapier or middleware transport.</li>
|
||
<li>UDP for jobs.</li>
|
||
<li>Emulated 10G fiber NICs on LXC.</li>
|
||
<li>tmpfs as the production store.</li>
|
||
<li>r=1 for product streams.</li>
|
||
<li>Connect-per-job.</li>
|
||
</ul>
|
||
<p>Re-run exhaustive: <code>bash scripts/exhaustive-ns1-study.sh</code>
|
||
on NS1.</p>
|
||
</body>
|
||
</html>
|