fix(auth): address Copilot review round 4 on the secret-store PR

Four findings (two inline, two suppressed), all taken. Three are the lock,
which is where the remaining sharp edges were.

- **A lock could be created and then never stamped.** If `mkdir` succeeded
  but writing the owner token failed, the catch treated it as "cannot lock
  here", returned the no-op release, and left the directory behind. The
  mutation then ran unlocked *and* every later writer queued behind an
  orphan nobody could deliberately release, until it aged out. The
  acquisition now tracks whether it created the directory, removes it on a
  failed stamp, and refuses rather than entering the section unlocked.

- **An old mtime was being treated as proof the owner was dead.** It is
  not: a holder that is SIGSTOPed, starved by a blocked event loop, or on a
  machine that suspended is alive and not running, and expiring it on
  elapsed time alone takes the lock out from under a live critical section
  — the lost update the lock exists to prevent, arrived at through the
  machinery meant to prevent it. The heartbeat narrowed the window; it
  could not close it, because the holder that most needs protecting is
  precisely the one not running its timers.

  The owner stamp now carries `pid` and `host`, and staleness only *opens*
  the question — on the same host a waiter asks the OS whether that pid
  still exists (`kill(pid, 0)`, which sends no signal; `EPERM` counts as
  alive, so a holder running as another user is never stolen from). Only a
  genuinely absent process is stolen from. Where the stamp cannot be
  trusted — another machine over a shared filesystem, or a missing/corrupt
  owner file — the mtime remains the answer, because the alternative is a
  lock nothing can ever break, turning one dead process into a permanent
  outage.

- **Podman was classified as a host.** `/.dockerenv` is Docker's marker and
  rootless Podman has no equivalent of it, while its cgroup paths say
  `libpod-<id>.scope` rather than the literal "podman" the pattern looked
  for. Such a box fell back to a file in the ephemeral writable layer and
  reported it as durable — the wrong way round to be wrong.
  `/run/.containerenv` and `libpod` are both recognized now.

- **Two tests floated a second `acquireLock`**, so the test could finish and
  `afterEach` could remove the temp directory while the acquisition was
  still running. Both hold and settle it now.

Separately, a defect of my own found while re-reading round 3:
`describeFileStore` called `tightenSecretFilePermissions`, so the function
whose job is to *describe* the store issued a `chmod` on every
`GET /api/config` — once per page load, on a file another process may be
mid-write on. Split out a read-only `readSecretFilePermissions`; the repair
stays at startup, where it happens once and its outcome is announced.

Tests: a stale lock whose owner is alive is not stolen (and survives the
attempt), one whose pid is gone is, one stamped by another host is; a
failed ownership stamp reports and leaves nothing behind; Podman via both
signals; and the read-only permission check leaves a 0644 file at 0644.
Full `npm run ci` green — 5,711 tests under the coverage gate.

Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-08-22 13:40:32 -04:00
commit 050556e1a7
2 changed files with 324 additions and 28 deletions

View file

@ -60,6 +60,7 @@
import * as crypto from "node:crypto";
import * as fs from "node:fs/promises";
import * as os from "node:os";
import * as path from "node:path";
import { readStoreFile, writeStoreFile } from "../../storage/store-io.js";
import {
@ -300,6 +301,10 @@ export async function acquireLock(
const lockDir = `${filePath}.lock`;
const ownerFile = `${lockDir}/owner`;
const token = crypto.randomBytes(16).toString("hex");
// The stamp a would-be thief consults before taking this lock: the token
// proves ownership on release, the pid and host let a waiter ask the OS
// whether we are still alive rather than inferring it from elapsed time.
const owner: LockOwner = { token, pid: process.pid, host: os.hostname() };
const noop = async (): Promise<void> => {};
const deadline = Date.now() + timeoutMs;
@ -317,9 +322,16 @@ export async function acquireLock(
}
for (;;) {
let created = false;
try {
await fs.mkdir(lockDir, { recursive: false });
await fs.writeFile(ownerFile, token, "utf-8");
created = true;
// Stamp ownership before entering the section. If this fails the lock
// exists but says nothing about who holds it, so nobody could ever
// release it deliberately and every later writer would queue behind an
// orphan until it aged out. Clean up and report, rather than proceeding
// unlocked *and* leaving the obstruction behind.
await fs.writeFile(ownerFile, JSON.stringify(owner), "utf-8");
// Prove liveness for as long as we hold it. `unref` so a held lock
// never keeps the process alive on its own — the interval is a
// side-channel, not work anyone is waiting for.
@ -336,8 +348,8 @@ export async function acquireLock(
// Release only what we still own. A lock judged stale and stolen
// now belongs to someone else, and removing it would drop them
// into a race with whoever comes next.
const held = await fs.readFile(ownerFile, "utf-8");
if (held !== token) return;
const held = parseOwner(await fs.readFile(ownerFile, "utf-8"));
if (held?.token !== token) return;
await fs.rm(lockDir, { recursive: true, force: true });
} catch {
// Already gone, or unreadable. Either way there is nothing to
@ -346,15 +358,17 @@ export async function acquireLock(
};
} catch (err) {
const code = (err as NodeJS.ErrnoException).code;
if (created) {
// We made the directory and then failed to stamp it. Ours to remove.
await fs.rm(lockDir, { recursive: true, force: true }).catch(() => {});
throw new SecretStoreUnavailableError(
`Could not initialize the lock for the secrets file at ${filePath}: ${err instanceof Error ? err.message : String(err)}`,
);
}
if (code !== "EEXIST") return noop; // cannot lock here at all — see above
try {
const held = (await fs.stat(lockDir)).mtimeMs;
if (Date.now() - held > staleMs) {
await fs.rm(lockDir, { recursive: true, force: true });
continue;
}
} catch {
// The holder released it between our mkdir and our stat. Retry.
if (await canSteal(lockDir, ownerFile, staleMs)) {
await fs.rm(lockDir, { recursive: true, force: true }).catch(() => {});
continue;
}
if (Date.now() >= deadline)
throw new SecretFileLockTimeoutError(filePath);
@ -363,6 +377,89 @@ export async function acquireLock(
}
}
/**
* May this waiter take a lock it found already held?
*
* Elapsed wall time alone does not answer that. An old mtime means the
* holder has not run its heartbeat lately, and there are two very different
* reasons for that: it died, or it is *alive but not running* — SIGSTOPed,
* starved by a blocked event loop, or on a machine that suspended. Expiring
* on time alone treats those identically and takes the lock out from under a
* live critical section, which is the lost update the lock exists to
* prevent, now arrived at through the machinery meant to prevent it.
*
* So the mtime only opens the question, and the owner stamp answers it: on
* the same host, ask the operating system whether that pid is still there.
* `kill(pid, 0)` sends no signal — it is the standard existence probe.
* `EPERM` counts as alive (the process exists; it just isn't ours), so a
* holder running as another user is never stolen from.
*
* The stamp is not always usable — a lock written by a different machine
* over a shared filesystem, or one whose owner file is missing or corrupt.
* There the mtime is all there is, and stealing on it is still right: the
* alternative is a lock nothing can ever break, which turns one dead process
* into a permanent outage.
*/
async function canSteal(
lockDir: string,
ownerFile: string,
staleMs: number,
): Promise<boolean> {
let held: number;
try {
held = (await fs.stat(lockDir)).mtimeMs;
} catch {
// Released between our mkdir and our stat — retry rather than steal.
return false;
}
if (Date.now() - held <= staleMs) return false;
let owner: LockOwner | null = null;
try {
owner = parseOwner(await fs.readFile(ownerFile, "utf-8"));
} catch {
// No stamp to consult; fall through to the time-only answer.
}
if (!owner || owner.host !== os.hostname() || typeof owner.pid !== "number") {
return true;
}
return !processIsAlive(owner.pid);
}
/** Does this pid exist? `kill(pid, 0)` is the probe; it sends no signal. */
function processIsAlive(pid: number): boolean {
try {
process.kill(pid, 0);
return true;
} catch (err) {
// EPERM means it exists and belongs to someone else — still alive.
return (err as NodeJS.ErrnoException).code === "EPERM";
}
}
interface LockOwner {
token: string;
pid: number;
host: string;
}
/** Parse an owner stamp, tolerating anything that isn't one. */
function parseOwner(raw: string): LockOwner | null {
try {
const parsed: unknown = JSON.parse(raw);
if (typeof parsed !== "object" || parsed === null) return null;
const { token, pid, host } = parsed as Partial<LockOwner>;
if (typeof token !== "string") return null;
return {
token,
pid: typeof pid === "number" ? pid : -1,
host: String(host),
};
} catch {
return null;
}
}
export interface FileSecretStoreOptions {
/** Absolute path of the secrets file. */
filePath: string;
@ -696,9 +793,24 @@ export async function tightenSecretFilePermissions(
}
// Verify rather than assume. The chmod above is best-effort, but the UI and
// the README both state the file is 0600 — so a file we failed to tighten
// (owned by another user, on a read-only mount) must be *reported*, not
// quietly described as protected. This is the one place that can tell the
// difference, because it is the only one that looks at the file.
// (owned by another user, on a read-only mount) must be reported, not
// quietly described as protected.
return readSecretFilePermissions(filePath);
}
/**
* Read the file's permission state **without touching it**.
*
* The read-only half of {@link tightenSecretFilePermissions}, split out
* because the descriptor is rebuilt on every `GET /api/config` and a
* function whose job is to *describe* must not chmod as a side effect —
* once per page load, on a file another process may be mid-write on. The
* repair belongs at startup, where it happens once and its outcome is
* announced; from then on the question is only "what is the mode now".
*/
export async function readSecretFilePermissions(
filePath: string,
): Promise<SecretFilePermissionState> {
try {
const mode = (await fs.stat(filePath)).mode & 0o777;
if (mode === 0o600) return { state: "ok" };
@ -711,7 +823,8 @@ export async function tightenSecretFilePermissions(
}
/**
* Outcome of {@link tightenSecretFilePermissions}: the file is `0600`
* Outcome of {@link tightenSecretFilePermissions} and
* {@link readSecretFilePermissions}: the file is `0600`
* (`ok`), does not exist yet (`absent`), or exists with a wider mode we
* could not fix (`loose`) — the only case anyone needs to hear about.
*/

View file

@ -49,6 +49,7 @@ import {
} from "../secret-storage-info.js";
import {
FileSecretStore,
readSecretFilePermissions,
tightenSecretFilePermissions,
} from "./file-secret-store.js";
import {
@ -132,13 +133,26 @@ export function defaultSecretFilePath(): string {
export function isContainer(): boolean {
if (process.env.KUBERNETES_SERVICE_HOST) return true;
try {
if (fsSync.existsSync("/.dockerenv")) return true;
// `/.dockerenv` is Docker's marker; `/run/.containerenv` is Podman's
// equivalent and is the only one a rootless Podman container reliably
// has. Missing it classified such a container as a host, which is the
// wrong way round to be wrong: the fallback then writes a file into the
// ephemeral writable layer and calls it durable.
if (
fsSync.existsSync("/.dockerenv") ||
fsSync.existsSync("/run/.containerenv")
) {
return true;
}
} catch {
// A sandboxed / restricted filesystem. Fall through to the cgroup read.
}
try {
const cgroup = fsSync.readFileSync("/proc/1/cgroup", "utf-8");
return /\b(docker|containerd|kubepods|podman|lxc)\b/.test(cgroup);
// `libpod` is what Podman actually writes into cgroup paths
// (`/machine.slice/libpod-<id>.scope`); the literal string "podman"
// often does not appear at all.
return /\b(docker|containerd|kubepods|podman|libpod|lxc)\b/.test(cgroup);
} catch {
// Not Linux, or /proc is not mounted. Not a container as far as we know.
return false;
@ -300,8 +314,11 @@ async function describeFileStore(
const plaintext = onDisk === null ? !store.encrypted : onDisk === "none";
// Re-checked here rather than captured at selection, for the same reason
// the encryption state is: the mode can change under a running process,
// and this descriptor is what the browser renders as fact.
const perms = await tightenSecretFilePermissions(filePath);
// and this descriptor is what the browser renders as fact. The *read-only*
// half deliberately — describing must not chmod, least of all once per
// page load on a file another process may be mid-write on. The repair
// happens once, at selection.
const perms = await readSecretFilePermissions(filePath);
return {
kind,
reason,