The patient reconnect gives up on permanent failures (issue 058 review)
The retry loop treated everything but a cert-pin mismatch as transient, so a refused login (revoked/mis-sealed credential) or a malformed broker URL retried for ever logging 'not reachable yet' — the silent non-progress the fix set out to remove, and a contradiction of its own docstring. fatalBrokerReason now classifies those three as fatal and everything else (connection refused, timeout, DNS) as retryable, with a unit test covering the split — the honest proof the bed cannot give, since it only ever starts the consumer after the broker is up. The pin case is now a typed PinMismatchError caught by instanceof, not a prose substring a reword could silently downgrade to an infinite retry against an impostor. Added a little jitter so modules do not stampede a recovering broker in lockstep.
This commit is contained in:
+15
-7
@@ -22,7 +22,7 @@
|
||||
|
||||
import { readFileSync } from "node:fs";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { connectAmqp } from "./broker-amqp.js";
|
||||
import { connectAmqp, fatalBrokerReason } from "./broker-amqp.js";
|
||||
import type { Credential } from "./broker-amqp.js";
|
||||
import { runTools } from "./runtime.js";
|
||||
import { invokeTool } from "@novox/mesh-sdk/tools";
|
||||
@@ -73,19 +73,27 @@ async function connectBroker(): Promise<Broker> {
|
||||
* runtime, which read as a crash-loop to every restart-counting health check and every person
|
||||
* watching. Retried indefinitely, aloud: the dependency appears or somebody reads why not.
|
||||
*
|
||||
* Only reachability retries. A pinned-certificate mismatch is a refusal, not a wait — an
|
||||
* impostor does not become the broker by being asked again — and configuration errors already
|
||||
* exit inside connectBroker before anything is thrown here.
|
||||
* A failure that waiting cannot fix (see fatalBrokerReason) is thrown at once rather than retried —
|
||||
* a permanent fault masquerading as "not reachable yet" is the silent non-progress this whole
|
||||
* change exists to remove. Missing-file and empty-URL configuration errors exit inside
|
||||
* connectBroker before they reach here; a malformed URL and a refused login are caught here.
|
||||
*/
|
||||
async function connectBrokerPatiently(): Promise<Broker> {
|
||||
for (let delay = 2_000; ; delay = Math.min(delay * 2, 30_000)) {
|
||||
try {
|
||||
return await connectBroker();
|
||||
} catch (err) {
|
||||
const fatal = fatalBrokerReason(err);
|
||||
if (fatal !== null) {
|
||||
console.error(`mesh-tools: ${fatal} — waiting will not fix this; giving up`);
|
||||
throw err;
|
||||
}
|
||||
const why = err instanceof Error ? err.message : String(err);
|
||||
if (why.includes("does not match the pinned")) throw err;
|
||||
console.error(`mesh-tools: the broker is not reachable yet (${why}); retrying in ${delay / 1000}s`);
|
||||
await new Promise((r) => setTimeout(r, delay));
|
||||
// A little jitter so every module that was up when the broker bounced does not retry in
|
||||
// lockstep and stampede it as it recovers.
|
||||
const wait = delay + Math.floor(Math.random() * 1_000);
|
||||
console.error(`mesh-tools: the broker is not reachable yet (${why}); retrying in ${Math.round(wait / 1000)}s`);
|
||||
await new Promise((r) => setTimeout(r, wait));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user