From 187442ec7bcee29b21194d4e4ef253f3bd0ad9f4 Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 1 Oct 2026 18:10:57 +0200 Subject: [PATCH 1/3] Issue 188: a refusal inside on-the-network drops a machine silently --- .../00-report.md | 48 +++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 04-ISSUES/188-a-refusal-inside-on-the-network-drops-a-machine-silently/00-report.md diff --git a/04-ISSUES/188-a-refusal-inside-on-the-network-drops-a-machine-silently/00-report.md b/04-ISSUES/188-a-refusal-inside-on-the-network-drops-a-machine-silently/00-report.md new file mode 100644 index 0000000..7efa488 --- /dev/null +++ b/04-ISSUES/188-a-refusal-inside-on-the-network-drops-a-machine-silently/00-report.md @@ -0,0 +1,48 @@ +--- +status: located +opened: 2026-10-01 +located-in: [mesh-controller cmd/mesh-controller/network.go (onTheNetwork resolves every machine unchecked and skips one that refuses, saying nothing)] +fixed-by: +amended-design: [] +--- + +# 188 — A refusal inside "who is on the network" drops a machine silently, and every symptom points elsewhere + +## What was observed + +At 15:28Z on 2026-10-01 the controller rolled to a build that refuses a machine with two modules +answering one provision and no pin naming which (mesh-controller 195). The control node had two +issuers of `acme-ca`. From that moment every plan of the control node failed with *step-ca has a +content that says `${machine:at}`, and this machine says mesh-range or name*; `seats` listed every +seat as unheld; the build machine refused the builder's and the proxy's builds with *no clone base +for that seat — nothing holds it*; and the roll-out of the next controller was refused with the +`${machine:at}` words. Not one of those names the cause. It was found by running the previous image +as a one-shot beside the current one and reading the difference, forty minutes later. + +## Why this is here + +`onTheNetwork` decides which machines have an address by resolving each one, unchecked, and +*skipping* any whose resolution errs. A machine skipped there has no `at`, so its own plan fails on +the first placeholder that needs one, in another module's words; everything held on it reads as +unheld; everything built from it cannot be built. The design lets one refusal become four unrelated +symptoms and no sentence about the refusal itself. It is the same shape as +[issue 187](../187-the-mesh-tells-nobody-when-it-stops-working/00-report.md): a fault that is swallowed +where it happens and discovered where it hurts. + +## What a fix needs + +- A machine whose resolution refuses is said, by `onTheNetwork`'s caller or in `status`: *the + control node does not resolve: more than one module provides acme-ca; pin one* — the resolver's + own words, which exist and were dropped. +- A refusal that a release introduces for a machine already converged — a new rule the stored + state does not meet — must not be silent at the roll either; the controller's prepare or first + resolution after a roll should name every machine it now refuses. + +*How this would be checked:* a controller test where one machine's unchecked resolution refuses: +`status` names the machine and the refusal, and the other machines keep their addresses. + +## Resolved in the live mesh, 2026-10-01 + +By hand: `pin novox acme-ca novox step-ca`, the provider the previous controller had in fact bound +the proxy to (read from its plan), after which the control node resolved, every seat read as held +and the roll-out proceeded. The design fault above stands. -- 2.54.0 From a92e4bf121ee48a7553ce1bdc9ef952018ad44ce Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 1 Oct 2026 18:14:37 +0200 Subject: [PATCH 2/3] Issue 188: what the live fault was and how it was resolved --- .../00-report.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/04-ISSUES/188-a-refusal-inside-on-the-network-drops-a-machine-silently/00-report.md b/04-ISSUES/188-a-refusal-inside-on-the-network-drops-a-machine-silently/00-report.md index 7efa488..d85abc9 100644 --- a/04-ISSUES/188-a-refusal-inside-on-the-network-drops-a-machine-silently/00-report.md +++ b/04-ISSUES/188-a-refusal-inside-on-the-network-drops-a-machine-silently/00-report.md @@ -43,6 +43,11 @@ where it happens and discovered where it hurts. ## Resolved in the live mesh, 2026-10-01 -By hand: `pin novox acme-ca novox step-ca`, the provider the previous controller had in fact bound -the proxy to (read from its plan), after which the control node resolved, every seat read as held -and the roll-out proceeded. The design fault above stands. +The refusal itself was a fault of mesh-controller 195, already corrected on main by its author's +hotfix (196) when the control node was found refusing; the running controller was the one build in +between. Found by running the previous image and main's image as one-shots beside the running one +and reading which resolved. Resolved by pushing the control node from a one-shot of main's image, as +the recipe for a controller that cannot roll itself says. A pin naming the proxy's issuer +(`step-ca`, the one the previous plan had bound) was made first and kept; it changes nothing. The +design fault above stands and is fixed by mesh-controller PR `fix/a-machine-not-on-the-network-is-said`: +the dropped machine and the resolver's words are said where the drop happens. -- 2.54.0 From 606fbb7add85be232db9e68a36785b97510d9d46 Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 1 Oct 2026 18:22:19 +0200 Subject: [PATCH 3/3] Issue 188: the second fault beneath the first, and its fix --- .../00-report.md | 20 ++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/04-ISSUES/188-a-refusal-inside-on-the-network-drops-a-machine-silently/00-report.md b/04-ISSUES/188-a-refusal-inside-on-the-network-drops-a-machine-silently/00-report.md index d85abc9..1f6cc1d 100644 --- a/04-ISSUES/188-a-refusal-inside-on-the-network-drops-a-machine-silently/00-report.md +++ b/04-ISSUES/188-a-refusal-inside-on-the-network-drops-a-machine-silently/00-report.md @@ -1,7 +1,7 @@ --- status: located opened: 2026-10-01 -located-in: [mesh-controller cmd/mesh-controller/network.go (onTheNetwork resolves every machine unchecked and skips one that refuses, saying nothing)] +located-in: [mesh-controller cmd/mesh-controller/plan.go (theRestOfTheMesh resolves every other machine without its pins and skips one that refuses, saying nothing), mesh-controller cmd/mesh-controller/network.go (onTheNetwork, the same)] fixed-by: amended-design: [] --- @@ -43,11 +43,13 @@ where it happens and discovered where it hurts. ## Resolved in the live mesh, 2026-10-01 -The refusal itself was a fault of mesh-controller 195, already corrected on main by its author's -hotfix (196) when the control node was found refusing; the running controller was the one build in -between. Found by running the previous image and main's image as one-shots beside the running one -and reading which resolved. Resolved by pushing the control node from a one-shot of main's image, as -the recipe for a controller that cannot roll itself says. A pin naming the proxy's issuer -(`step-ca`, the one the previous plan had bound) was made first and kept; it changes nothing. The -design fault above stands and is fixed by mesh-controller PR `fix/a-machine-not-on-the-network-is-said`: -the dropped machine and the resolver's words are said where the drop happens. +Two faults, one on top of the other. The refusal was mesh-controller 195's new rule — two modules +answering one provision on one machine need a pin — which its author hotfixed for the first pass +(196). The second pass of "the rest of the mesh" resolves every machine *without its pins*, so the +control node, pinned or not, was refused there and vanished: every seat it holds read as unheld, +the builder's and the proxy's builds were refused for want of the git seat's clone base, the +roll-out of the next controller was refused, and the first tiered plan failed at its first tier. +Found by a diagnostic build counting what each machine yielded. Fixed by mesh-controller PR +`fix/a-machine-not-on-the-network-is-said`: each machine is resolved with its own pins, and a machine +left out is named with the resolver's words in both places. The pin itself (`step-ca`, the issuer +the proxy already had) was made by hand and stands. -- 2.54.0