A poll that could not ask has not been answered
`settled` used `must`, so a failed exec ended the wait as though the machine had reported a failure. It had reported nothing: the control plane is a container on the node being polled, and while that node applies a declaration an exec into it can lose its stdout fifo to containerd. The run then blamed the mesh for a question that missed. Could not ask and asked, and the answer was bad are different facts, and only the second is the machine's. A failed poll now keeps the reason and tries again; the timeout reports whichever came last, so a control plane that is genuinely unreachable still fails the test — with the reason rather than with a stack trace. Every five seconds rather than every two. Each poll is an exec into a container on a machine that is busy applying, and thirty times a minute was competing with the apply rather than observing it.
This commit is contained in:
@@ -124,7 +124,17 @@ async function settled(node: string, withinMs = 240_000): Promise<void> {
|
|||||||
const until = Date.now() + withinMs;
|
const until = Date.now() + withinMs;
|
||||||
let last = "";
|
let last = "";
|
||||||
while (Date.now() < until) {
|
while (Date.now() < until) {
|
||||||
const said = await mesh("status --json");
|
// **Could not ask** and **asked, and the answer was bad** are different facts, and only the
|
||||||
|
// second is this machine's fault. The control plane is a container on the node being polled:
|
||||||
|
// while it applies a declaration, an exec into it can lose its fifo to containerd, and a poll
|
||||||
|
// loop that treats that as a verdict reports the mesh broken because the question missed.
|
||||||
|
const { out: said, ok } = await on("anchor",
|
||||||
|
`docker exec mesh-control /mesh-control status --json`);
|
||||||
|
if (!ok) {
|
||||||
|
last = said;
|
||||||
|
await new Promise((r) => setTimeout(r, 5000));
|
||||||
|
continue;
|
||||||
|
}
|
||||||
const state = JSON.parse(said) as {
|
const state = JSON.parse(said) as {
|
||||||
wrong: { node: string; outcome: string; refused?: string;
|
wrong: { node: string; outcome: string; refused?: string;
|
||||||
failed?: { id: string; error: string }[] }[];
|
failed?: { id: string; error: string }[] }[];
|
||||||
@@ -138,9 +148,13 @@ async function settled(node: string, withinMs = 240_000): Promise<void> {
|
|||||||
}
|
}
|
||||||
if (!state.waiting.some((w) => w.node === node)) return;
|
if (!state.waiting.some((w) => w.node === node)) return;
|
||||||
last = said;
|
last = said;
|
||||||
await new Promise((r) => setTimeout(r, 2000));
|
// Unhurried on purpose: each poll is an exec into a container on the machine that is busy
|
||||||
|
// applying, and asking four times a minute rather than thirty is the difference between
|
||||||
|
// observing the apply and competing with it.
|
||||||
|
await new Promise((r) => setTimeout(r, 5000));
|
||||||
}
|
}
|
||||||
throw new Error(`${node} never caught up with what it was sent:\n${last}`);
|
throw new Error(`${node} never caught up with what it was sent within ` +
|
||||||
|
`${Math.round(withinMs / 1000)}s. Last answer, or the reason there was none:\n${last}`);
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|||||||
Reference in New Issue
Block a user