The patient reconnect gives up on permanent failures (issue 058 review)

The retry loop treated everything but a cert-pin mismatch as transient,
so a refused login (revoked/mis-sealed credential) or a malformed broker
URL retried for ever logging 'not reachable yet' — the silent
non-progress the fix set out to remove, and a contradiction of its own
docstring. fatalBrokerReason now classifies those three as fatal and
everything else (connection refused, timeout, DNS) as retryable, with a
unit test covering the split — the honest proof the bed cannot give,
since it only ever starts the consumer after the broker is up.

The pin case is now a typed PinMismatchError caught by instanceof, not a
prose substring a reword could silently downgrade to an infinite retry
against an impostor. Added a little jitter so modules do not stampede a
recovering broker in lockstep.
This commit is contained in:
2026-09-20 13:30:52 +02:00
parent 0ea3db3b20
commit 5b111da4a9
3 changed files with 97 additions and 8 deletions
+15 -7
View File
@@ -22,7 +22,7 @@
import { readFileSync } from "node:fs";
import { pathToFileURL } from "node:url";
import { connectAmqp } from "./broker-amqp.js";
import { connectAmqp, fatalBrokerReason } from "./broker-amqp.js";
import type { Credential } from "./broker-amqp.js";
import { runTools } from "./runtime.js";
import { invokeTool } from "@novox/mesh-sdk/tools";
@@ -73,19 +73,27 @@ async function connectBroker(): Promise<Broker> {
* runtime, which read as a crash-loop to every restart-counting health check and every person
* watching. Retried indefinitely, aloud: the dependency appears or somebody reads why not.
*
* Only reachability retries. A pinned-certificate mismatch is a refusal, not a wait — an
* impostor does not become the broker by being asked again — and configuration errors already
* exit inside connectBroker before anything is thrown here.
* A failure that waiting cannot fix (see fatalBrokerReason) is thrown at once rather than retried —
* a permanent fault masquerading as "not reachable yet" is the silent non-progress this whole
* change exists to remove. Missing-file and empty-URL configuration errors exit inside
* connectBroker before they reach here; a malformed URL and a refused login are caught here.
*/
async function connectBrokerPatiently(): Promise<Broker> {
for (let delay = 2_000; ; delay = Math.min(delay * 2, 30_000)) {
try {
return await connectBroker();
} catch (err) {
const fatal = fatalBrokerReason(err);
if (fatal !== null) {
console.error(`mesh-tools: ${fatal} — waiting will not fix this; giving up`);
throw err;
}
const why = err instanceof Error ? err.message : String(err);
if (why.includes("does not match the pinned")) throw err;
console.error(`mesh-tools: the broker is not reachable yet (${why}); retrying in ${delay / 1000}s`);
await new Promise((r) => setTimeout(r, delay));
// A little jitter so every module that was up when the broker bounced does not retry in
// lockstep and stampede it as it recovers.
const wait = delay + Math.floor(Math.random() * 1_000);
console.error(`mesh-tools: the broker is not reachable yet (${why}); retrying in ${Math.round(wait / 1000)}s`);
await new Promise((r) => setTimeout(r, wait));
}
}
}