A secret that will never be readable stops calling itself a race (hq issue 225)

The harness says 'secret not readable yet' when it cannot open a consumer's
sealed credential, which is true and ordinary on the first pass. It said it
4330 times in three hours on the control machine while two consumers were
never provisioned, because a permission the host will never grant reads
exactly like a file that has not arrived.

After about a minute it says so plainly instead, once and then rarely, and
names what to look at: who owns the file and who this process runs as. The
ownership itself is the controller's to fix.

0.1.9.

src/provisioner/index.ts

@@ -77,6 +77,15 @@ export interface ProvisionerOptions {
 /** The longest a consumer whose `create` keeps failing to satisfy `holds` waits between checks. */
 const MAX_BACKOFF_MS = 60 * 60_000;
 
+/** How many passes a secret may be unreadable before the harness stops calling it a race. At the
+ *  default cadence this is about a minute, which is longer than the host takes to write a file
+ *  and far shorter than the three hours issue 225 went unread. */
+const patiently = 12;
+
+/** Once it is saying so loudly, how often to repeat it. Rare enough to stay readable, often
+ *  enough that somebody arriving at the log finds it. */
+const loudlyEvery = 240;
+
 /** One entry in the mesh's contributions file: a consumer the provider must serve. */
 interface Contribution {
   readonly as: string;
@@ -107,6 +116,9 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
   // login (`as`) -> what was last applied: the hash that says whether it changed, and the
   // derivation to hand back on withdrawal, which by then is no longer in the file.
   const applied = new Map<string, { hash: string; derived: Readonly<Record<string, unknown>> }>();
+  // How many passes each consumer's secret has been unreadable for, so the wait can stop sounding
+  // like a wait.
+  const waiting = new Map<string, number>();
   let stopped = false;
 
   async function reconcile(): Promise<void> {
@@ -127,9 +139,29 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
       } catch (err) {
         // The contribution names a secret the host has not written yet — a normal race on the
         // first pass. Skipped, retried on the next tick, never fatal.
-        console.error(`[provisioner:${resource}] ${g.as}: secret not readable yet (${g.secret}): ${err}`);
+        //
+        // **"Yet" stops being true, and then the word is the fault** (novox/hq issue 225). A
+        // permission the host will never grant reads exactly like a file that has not arrived,
+        // and the two want opposite responses: one is waiting, the other is a person. Said the
+        // patient way while it is plausibly a race, and then said plainly, once, and from then
+        // on only occasionally — because a line repeated every five seconds is a line nobody
+        // reads. Three hours of "not readable yet" is how a provider that served nobody looked
+        // healthy.
+        const n = (waiting.get(g.as) ?? 0) + 1;
+        waiting.set(g.as, n);
+        if (n <= patiently) {
+          console.error(`[provisioner:${resource}] ${g.as}: secret not readable yet (${g.secret}): ${err}`);
+        } else if (n === patiently + 1 || n % loudlyEvery === 0) {
+          console.error(
+            `[provisioner:${resource}] ${g.as}: CANNOT READ the secret after ${n} attempts ` +
+              `(${g.secret}): ${err}. This is not a race any more — nothing has been provisioned ` +
+              `for this consumer and nothing will be until somebody looks. Check who owns the ` +
+              `file and who this process runs as (novox/hq issue 225)`,
+          );
+        }
         continue;
       }
+      waiting.delete(g.as);
       const h = hash(g.as, password, g.values ?? {}, g.derived ?? {});
       const p: Provision = {
         as: g.as, password, values: g.values ?? {}, derived: g.derived ?? {}, at: g.at, consumer: g.node,