feat(auth): give the secrets file a real cross-process lock (#2082)
#1950 shipped FileSecretStore without cross-process mutual exclusion, by decision: an earlier revision hand-rolled a `mkdir` election with an owner stamp, a heartbeat and a stale-takeover, and three consecutive review rounds found a real race in it. The last one is not closable with what Node exposes — claiming a stale lock atomically needs compare-and-swap on a directory entry (`renameat2(RENAME_EXCHANGE)`) — so it was replaced with optimistic verify-and-retry and the residual documented. #2082 settles that as "borrow, don't hand-roll". `proper-lockfile` is what npm itself locks with, and stale-takeover is precisely the problem it has already solved: it re-stats the lock directory after claiming it and gives the lock up when the mtime is not the one it wrote, so the loser of a takeover race releases instead of proceeding. - core/auth/node/file-lock.ts: `withSecretFileLock`. `realpath: false` so a file can be locked into existence (the very first `set` has no `secrets.json`, and the library's default resolves through `fs.realpath`); a warning in place of the library's `onCompromised`, which throws from a timer and would take the session down; and a degrade-with-one-warning path rather than a throw when no lock can be taken — this store exists for boxes missing the usual mechanism (#1848, #1905) and must not gain a new way to be unavailable. - FileSecretStore.mutate holds it across the whole read-modify-write. The optimistic verify stays underneath and is not redundant: a lock is advisory between the processes that take it, so the verify covers a writer outside this codebase and covers the degrade path. The in-process queue stays too, and gains a second job — proper-lockfile is not reentrant, so serializing per path keeps ELOCKED meaning "another process". - absorbFileSecretsIntoKeyring takes the same lock around orphan adoption and the atomic claim, behind a lock-free `readdir` fast path so the common startup (keychain available, no file ever written) neither creates a lock directory nor warns about one it could not create. proper-lockfile is a root `dependency` per the placement rule, and is named in all three bundler `external` lists: tsup externalizes what the *client's* manifest declares, so a root-only CJS package was being inlined into the ESM bundles, leaving esbuild's `Dynamic require of "path" is not supported` shim that killed `--cli` at import time. That rule was undocumented; it is now in AGENTS.md beside the placement rule that creates it, and mirrored into .github/copilot-instructions.md. Tests: 7 new in file-lock.test.ts driving a real second process (the existing suite structurally cannot — `serialize` orders in-process callers before the lock sees them), plus two for the migration fast path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JTHVxSu8AUgHRLvo1ntZ8H Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
7701c68028
commit
04dbc36544
1 changed files with 718 additions and 90 deletions
|
|
@ -62,6 +62,7 @@ import * as crypto from "node:crypto";
|
|||
import * as fs from "node:fs/promises";
|
||||
import * as path from "node:path";
|
||||
import { readStoreFile, writeStoreFile } from "../../storage/store-io.js";
|
||||
import { withSecretFileLock } from "./file-lock.js";
|
||||
import {
|
||||
SecretStoreUnavailableError,
|
||||
type SecretBulkRequest,
|
||||
|
|
@ -597,30 +598,33 @@ export class FileSecretStore implements SecretStore {
|
|||
}
|
||||
|
||||
/**
|
||||
* Run `fn` with the file to itself across *processes* as well.
|
||||
* Apply a mutation to the file, under a cross-process lock, and confirm it
|
||||
* survived.
|
||||
*
|
||||
* The in-process queue is necessary and not sufficient: a durable secrets
|
||||
* file is shared state, and a second Inspector on the same box — a CLI run
|
||||
* beside a web session is the ordinary case, not a contrived one — reads
|
||||
* the same "before" map and then atomically replaces the file, dropping
|
||||
* whatever the first process had just added. Both writes report success;
|
||||
* one secret is simply gone.
|
||||
* **Two mechanisms, and they answer different questions** — this is not
|
||||
* belt-and-braces.
|
||||
*
|
||||
* Apply a mutation to the file, and confirm it survived.
|
||||
* The lock ({@link withSecretFileLock}, #2082) is the one that provides
|
||||
* mutual exclusion: a durable secrets file is shared state, and a second
|
||||
* Inspector on the same box — a CLI run beside a web session is the
|
||||
* ordinary case, not a contrived one — otherwise reads the same "before"
|
||||
* map and then atomically replaces the file, dropping whatever the first
|
||||
* process had just added. Both writes report success; one secret is simply
|
||||
* gone. Held across the read *and* the write, so no other holder can
|
||||
* observe or replace the map in between.
|
||||
*
|
||||
* **There is no cross-process lock.** There was one — a `mkdir` election
|
||||
* with an owner stamp, a heartbeat and a stale-takeover — and three
|
||||
* consecutive review rounds found a real race in it. The last one is
|
||||
* unfixable with what Node exposes: claiming a stale lock needs
|
||||
* compare-and-swap on a directory entry (`renameat2(RENAME_EXCHANGE)`),
|
||||
* and without it a waiter that loses the race can still move the winner's
|
||||
* *fresh* lock aside and enter alongside it.
|
||||
* The verify covers what a lock structurally cannot. A lock is an advisory
|
||||
* convention between the processes that take it, so it says nothing about
|
||||
* a writer that does not: an editor, a restored backup, a `jq` one-liner,
|
||||
* or an Inspector predating #2082. It is also the fallback when the lock
|
||||
* is unavailable at all — `withSecretFileLock` runs the body anyway on a
|
||||
* read-only or lock-hostile directory rather than failing a `set` on
|
||||
* exactly the deployments this store exists for (#1848, #1905).
|
||||
*
|
||||
* So this does the opposite: it lets writers collide and makes the loser
|
||||
* notice. Read `M0`, apply the mutation to get `M1`, write it, then read
|
||||
* back `M2`. If `M2` equals `M1` nothing interleaved. If it does not,
|
||||
* someone wrote between our write and our read — so re-apply onto what
|
||||
* they left and try again.
|
||||
* So: read `M0`, apply the mutation to get `M1`, write it, then read back
|
||||
* `M2`. If `M2` equals `M1` nothing interleaved. If it does not, someone
|
||||
* wrote between our write and our read — so re-apply onto what they left
|
||||
* and try again.
|
||||
*
|
||||
* The comparison is over the **whole map**, not just the entry we touched.
|
||||
* Checking only our own key would pass in precisely the case that loses
|
||||
|
|
@ -629,32 +633,29 @@ export class FileSecretStore implements SecretStore {
|
|||
* repairs it — A writes `MA`, B writes `MB` over it, B verifies `MB`
|
||||
* correctly, A verifies, sees `M2 !== MA`, re-applies onto `MB`.
|
||||
*
|
||||
* **This is not mutual exclusion, and the gap is wider than a crash.**
|
||||
* The verify only catches a clobber that has already landed. Order the
|
||||
* same two writers as write-A, verify-A, write-B, verify-B and both
|
||||
* succeed while A's entry is gone: A's verify ran before B's write, so
|
||||
* there was nothing yet to see, and B did nothing wrong. Nothing detects
|
||||
* it afterwards. A crash between write and verify is one instance of the
|
||||
* same shape, not the whole of it — an earlier version of this comment
|
||||
* said otherwise, which understated it.
|
||||
*
|
||||
* What that buys, stated without overselling: two processes must write the
|
||||
* same file within the window between one's write and its read-back, and
|
||||
* the loss is one secret that reported success. The lock this replaced
|
||||
* lost updates in a *wider* set of interleavings, with every participant
|
||||
* alive, and could not be closed without a primitive Node does not expose
|
||||
* (`renameat2(RENAME_EXCHANGE)`); this costs ~220 fewer lines and converges
|
||||
* in every interleaving where the clobber lands before the verify. If the
|
||||
* residual matters for a deployment, the answer is a real lock — an
|
||||
* OS-backed one from a dedicated library — not another hand-rolled
|
||||
* election.
|
||||
* **What is left, stated without overselling.** Against an unlocked writer
|
||||
* the verify is not mutual exclusion: it only catches a clobber that has
|
||||
* already landed, so ordering the two as write-A, verify-A, write-B,
|
||||
* verify-B loses A's entry with both reporting success. That residual is
|
||||
* now confined to a writer outside this codebase, or to a directory where
|
||||
* no lock could be taken — and the latter announces itself, once, on the
|
||||
* console.
|
||||
*
|
||||
* Reads deliberately do not participate: `writeStoreFile` is atomic
|
||||
* (write-temp-then-rename), so a reader sees either the old file or the
|
||||
* new one, never a torn one.
|
||||
* new one, never a torn one. Taking the lock for them would serialize
|
||||
* every `GET /api/servers` behind whatever else holds it, to prevent
|
||||
* nothing.
|
||||
*/
|
||||
private async mutate(
|
||||
apply: (map: Record<string, string>) => Record<string, string> | null,
|
||||
): Promise<void> {
|
||||
await withSecretFileLock(this.filePath, () => this.mutateLocked(apply));
|
||||
}
|
||||
|
||||
/** {@link mutate}'s body, with the lock already held. */
|
||||
private async mutateLocked(
|
||||
apply: (map: Record<string, string>) => Record<string, string> | null,
|
||||
): Promise<void> {
|
||||
for (let attempt = 0; attempt < MAX_WRITE_ATTEMPTS; attempt++) {
|
||||
let current: Record<string, string>;
|
||||
|
|
@ -713,9 +714,15 @@ export class FileSecretStore implements SecretStore {
|
|||
* Measured, not theorised — two instances racing `set` lost an entry on
|
||||
* the first run of the convergence test below.
|
||||
*
|
||||
* So the in-process case is made correct by construction here, and the
|
||||
* verify-and-retry in {@link mutate} covers what this cannot see: a
|
||||
* So the in-process case is made correct by construction here, and
|
||||
* {@link mutate}'s cross-process lock covers what this cannot see: a
|
||||
* second Inspector process.
|
||||
*
|
||||
* The queue is *also* what keeps that lock usable. `proper-lockfile` is
|
||||
* not reentrant — a second `lock()` on a path this process already holds
|
||||
* fails `ELOCKED`, which is indistinguishable from a genuine remote
|
||||
* holder. Serializing here means the lock is only ever contended between
|
||||
* processes, which is the only contention it is asked to arbitrate.
|
||||
*/
|
||||
private serialize<T>(fn: () => Promise<T>): Promise<T> {
|
||||
const key = path.resolve(this.filePath);
|
||||
|
|
|
|||
Loading…
Reference in a new issue