A poll that could not ask has not been answered

`settled` used `must`, so a failed exec ended the wait as though the
machine had reported a failure. It had reported nothing: the control
plane is a container on the node being polled, and while that node
applies a declaration an exec into it can lose its stdout fifo to
containerd. The run then blamed the mesh for a question that missed.

Could not ask and asked, and the answer was bad are different facts, and
only the second is the machine's. A failed poll now keeps the reason and
tries again; the timeout reports whichever came last, so a control plane
that is genuinely unreachable still fails the test — with the reason
rather than with a stack trace.

Every five seconds rather than every two. Each poll is an exec into a
container on a machine that is busy applying, and thirty times a minute
was competing with the apply rather than observing it.
This commit is contained in:
2026-09-01 20:25:54 +02:00
parent 2bf5846bf2
commit 144362be37
+17 -3
View File
@@ -124,7 +124,17 @@ async function settled(node: string, withinMs = 240_000): Promise<void> {
const until = Date.now() + withinMs; const until = Date.now() + withinMs;
let last = ""; let last = "";
while (Date.now() < until) { while (Date.now() < until) {
const said = await mesh("status --json"); // **Could not ask** and **asked, and the answer was bad** are different facts, and only the
// second is this machine's fault. The control plane is a container on the node being polled:
// while it applies a declaration, an exec into it can lose its fifo to containerd, and a poll
// loop that treats that as a verdict reports the mesh broken because the question missed.
const { out: said, ok } = await on("anchor",
`docker exec mesh-control /mesh-control status --json`);
if (!ok) {
last = said;
await new Promise((r) => setTimeout(r, 5000));
continue;
}
const state = JSON.parse(said) as { const state = JSON.parse(said) as {
wrong: { node: string; outcome: string; refused?: string; wrong: { node: string; outcome: string; refused?: string;
failed?: { id: string; error: string }[] }[]; failed?: { id: string; error: string }[] }[];
@@ -138,9 +148,13 @@ async function settled(node: string, withinMs = 240_000): Promise<void> {
} }
if (!state.waiting.some((w) => w.node === node)) return; if (!state.waiting.some((w) => w.node === node)) return;
last = said; last = said;
await new Promise((r) => setTimeout(r, 2000)); // Unhurried on purpose: each poll is an exec into a container on the machine that is busy
// applying, and asking four times a minute rather than thirty is the difference between
// observing the apply and competing with it.
await new Promise((r) => setTimeout(r, 5000));
} }
throw new Error(`${node} never caught up with what it was sent:\n${last}`); throw new Error(`${node} never caught up with what it was sent within ` +
`${Math.round(withinMs / 1000)}s. Last answer, or the reason there was none:\n${last}`);
} }
/** /**