The patient reconnect gives up on permanent failures (issue 058 review)

The retry loop treated everything but a cert-pin mismatch as transient,
so a refused login (revoked/mis-sealed credential) or a malformed broker
URL retried for ever logging 'not reachable yet' — the silent
non-progress the fix set out to remove, and a contradiction of its own
docstring. fatalBrokerReason now classifies those three as fatal and
everything else (connection refused, timeout, DNS) as retryable, with a
unit test covering the split — the honest proof the bed cannot give,
since it only ever starts the consumer after the broker is up.

The pin case is now a typed PinMismatchError caught by instanceof, not a
prose substring a reword could silently downgrade to an infinite retry
against an impostor. Added a little jitter so modules do not stampede a
recovering broker in lockstep.

src/broker-amqp.ts

@@ -24,6 +24,41 @@ interface Reply {
   error?: string;
 }
 
+/** The broker presented a certificate whose fingerprint is not the one the mesh pinned. A distinct
+ *  type rather than a message to grep, so a caller deciding "wait or refuse" (serve mode's patient
+ *  reconnect, novox/hq issue 058) tells this apart from an absent broker by `instanceof`, not by a
+ *  prose string that a later reword would silently turn back into an infinite retry against an
+ *  impostor. */
+export class PinMismatchError extends Error {}
+
+/**
+ * Why a broker connection failed in a way no amount of waiting will fix — or null when it is worth
+ * retrying. Serve mode's patient reconnect (novox/hq issue 058) uses this to tell a permanent
+ * fault from a broker that is merely not up yet. Three failures are permanent:
+ *
+ *   - the certificate does not match the pin — an impostor does not become the broker by being
+ *     asked again (typed, so a reworded message cannot silently turn this back into a retry);
+ *   - the broker URL is not a URL — a malformed address never parses on the next try;
+ *   - the broker answered and refused the login — a wrong or revoked credential, not an absent
+ *     broker, and it will refuse the next attempt identically.
+ *
+ * Everything else — connection refused, timeout, DNS not resolving yet — is the overlay still
+ * coming up, and is retried.
+ */
+export function fatalBrokerReason(err: unknown): string | null {
+  if (err instanceof PinMismatchError) return "the broker's certificate does not match the pin";
+  const e = err as { code?: unknown; message?: unknown };
+  const code = typeof e?.code === "string" ? e.code : "";
+  const message = typeof e?.message === "string" ? e.message : String(err);
+  if (code === "ERR_INVALID_URL" || /invalid url/i.test(message)) {
+    return `the broker URL is not a URL (${message})`;
+  }
+  if (/access[-_ ]?refused|login was refused|handshake terminated|\b403\b/i.test(message)) {
+    return `the broker refused the login (${message})`;
+  }
+  return null;
+}
+
 /** A broker credential as the mesh delivers it (novox/hq ADR 0043): an amqps URL, the fingerprint
  *  of the certificate the broker must present, and the node and module the account is scoped to (so
  *  the runtime names its queue as the mesh did). A plain string is a bootstrap URL. */
@@ -299,7 +334,7 @@ async function pinnedOptions(rawUrl: string, fingerprint: string): Promise<tls.C
 
   const seen = createHash("sha256").update(certificate.raw).digest("hex");
   if (seen !== normalizeFingerprint(fingerprint)) {
-    throw new Error(
+    throw new PinMismatchError(
       `the broker's certificate (sha256:${seen}) does not match the pinned ${fingerprint} — refusing`,
     );
   }

src/main.ts

@@ -22,7 +22,7 @@
 
 import { readFileSync } from "node:fs";
 import { pathToFileURL } from "node:url";
-import { connectAmqp } from "./broker-amqp.js";
+import { connectAmqp, fatalBrokerReason } from "./broker-amqp.js";
 import type { Credential } from "./broker-amqp.js";
 import { runTools } from "./runtime.js";
 import { invokeTool } from "@novox/mesh-sdk/tools";
@@ -73,19 +73,27 @@ async function connectBroker(): Promise<Broker> {
  * runtime, which read as a crash-loop to every restart-counting health check and every person
  * watching. Retried indefinitely, aloud: the dependency appears or somebody reads why not.
  *
- * Only reachability retries. A pinned-certificate mismatch is a refusal, not a wait — an
- * impostor does not become the broker by being asked again — and configuration errors already
- * exit inside connectBroker before anything is thrown here.
+ * A failure that waiting cannot fix (see fatalBrokerReason) is thrown at once rather than retried —
+ * a permanent fault masquerading as "not reachable yet" is the silent non-progress this whole
+ * change exists to remove. Missing-file and empty-URL configuration errors exit inside
+ * connectBroker before they reach here; a malformed URL and a refused login are caught here.
  */
 async function connectBrokerPatiently(): Promise<Broker> {
   for (let delay = 2_000; ; delay = Math.min(delay * 2, 30_000)) {
     try {
       return await connectBroker();
     } catch (err) {
+      const fatal = fatalBrokerReason(err);
+      if (fatal !== null) {
+        console.error(`mesh-tools: ${fatal} — waiting will not fix this; giving up`);
+        throw err;
+      }
       const why = err instanceof Error ? err.message : String(err);
-      if (why.includes("does not match the pinned")) throw err;
-      console.error(`mesh-tools: the broker is not reachable yet (${why}); retrying in ${delay / 1000}s`);
-      await new Promise((r) => setTimeout(r, delay));
+      // A little jitter so every module that was up when the broker bounced does not retry in
+      // lockstep and stampede it as it recovers.
+      const wait = delay + Math.floor(Math.random() * 1_000);
+      console.error(`mesh-tools: the broker is not reachable yet (${why}); retrying in ${Math.round(wait / 1000)}s`);
+      await new Promise((r) => setTimeout(r, wait));
     }
   }
 }