From 5042ffd8d3db4339248866ebedf1ae8e3ec893b5 Mon Sep 17 00:00:00 2001 From: jochen Date: Wed, 30 Sep 2026 12:28:01 +0200 Subject: [PATCH 1/2] Self-update works, and a delivered host is one fact short of usable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The loop closed on the workstation: the version landed, the launcher was replaced, the running host stood aside, and after one restart the launcher started a binary the mesh had compiled, published and delivered. The launcher goes as a file resource rather than inside the archive, and that is the safety rather than a preference. A file is written atomically, so the running launcher keeps the inode it started from; an archive writes in place with truncate and would cut a script a shell is reading. The manifest carries a second copy and a test refuses any drift from the one in packaging. Then it would have refused the first declaration it was asked to apply. The Makefile links in two facts the mesh's toolchain does not, on purpose, and one of them is the system the host was built for — read before anything is applied, so the failure is safe and total. Nothing reports it: the unit is active, the bus link is up, and the log says it is hearing what the node should be. Worse, the declaration that would fix it is the declaration it cannot apply, so the mesh cannot repair such a machine. Restored by moving the delivered versions aside and letting the launcher fall back, which is the fallback working as designed. 0142 already settles the version — it comes from where the component sits, not from its linker — and that is unimplemented. The system pin has no answer, and the candidates are a decision rather than a fix: put it in the path too, carry it in a file beside the binary, or stop pinning at link time at all, which is 0005's to change. --- .../01-progress.md | 23 +++++ .../00-report.md | 95 +++++++++++++++++++ 2 files changed, 118 insertions(+) create mode 100644 04-ISSUES/161-a-delivered-host-carries-none-of-its-link-time-facts/00-report.md diff --git a/04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/01-progress.md b/04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/01-progress.md index cf8c83e..a501f6c 100644 --- a/04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/01-progress.md +++ b/04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/01-progress.md @@ -95,3 +95,26 @@ and then read by nothing: it does not reach the compiler, no machine is matched chooses between two artifacts by it. The host built here is x86-64 because the build machine is, not because anything in the declaration said so — correct for this mesh by coincidence. That is [issue 159](../159-an-artifacts-system-is-checked-and-then-ignored/00-report.md). + +## Delivered, started, and one fact short (2026-09-30, later) + +The loop closed. The launcher is delivered as a **file** resource rather than inside the archive, and +that difference is the safety: a file is written atomically — temp file, then rename — so the running +launcher keeps the inode it was started from, where an archive writes in place with truncate and would +cut the script a running shell is reading. The manifest carries a second copy of the launcher and a +test refuses any difference from `packaging/nox-mesh-host-launch`. + +On the workstation, in order: the version landed, the launcher was replaced, the running host saw a +delivered version and stood aside, and after one restart of the unit the launcher started +`/usr/lib/nox-mesh-host/versions/637f65559d16/nox-mesh-host`. **A host the mesh compiled, published, +delivered and started.** + +It would then have refused the first declaration it was asked to apply. The host's Makefile links in +two facts the mesh's toolchain does not, and one of them — the system it was built for — is read before +anything is applied. That is +[issue 161](../161-a-delivered-host-carries-none-of-its-link-time-facts/00-report.md), and the machine +is back on its hand-placed binary until it is answered. + +**The fallback is what made that safe**, and it was not luck: the launcher runs the pinned version, or +the newest delivered one, or the one placed by hand — so moving the delivered versions aside restored +the machine in one step. diff --git a/04-ISSUES/161-a-delivered-host-carries-none-of-its-link-time-facts/00-report.md b/04-ISSUES/161-a-delivered-host-carries-none-of-its-link-time-facts/00-report.md new file mode 100644 index 0000000..d09a197 --- /dev/null +++ b/04-ISSUES/161-a-delivered-host-carries-none-of-its-link-time-facts/00-report.md @@ -0,0 +1,95 @@ +--- +status: located +opened: 2026-09-30 +located-in: + - mesh-host cmd/mesh-host/main.go (version and builtFor, both set at link time) + - mesh-controller internal/builder (the toolchain, which deliberately takes nothing from the module) +fixed-by: +amended-design: +--- + +# 161 — A host the mesh built carries none of the facts its Makefile stamps in + +## What was observed + +*2026-09-30, on the workstation, having just made the host self-updating.* + +The mesh compiled the host, published it, delivered it and the launcher started it. It ran, read the +machine correctly, and **would have refused the first declaration it was asked to apply.** + +The host's own Makefile links in two facts: + +``` +LDFLAGS := -s -w -X main.builtFor=$(SYSTEM) -X main.version=$(VERSION) +``` + +The mesh's Go toolchain links in neither, on purpose: a toolchain accepts nothing from the module, +because anything a module could override there it would be writing a Dockerfile to override +([ADR 0142](../../02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md)). So a +delivered host has `builtFor = ""` and `version = "development build"`. + +**`builtFor` empty is the one that bites.** Before applying anything, the host asks which system it +was built for: + +```go +sys, err := system.For(builtFor) +``` + +and that answers, for an empty name: + +``` +this host was built for "", which is not a system it knows. Built hosts are: … +``` + +It is called before any resource is applied, so the failure is in the safe direction — the machine is +not half-configured. It is still a host that cannot do its job, and nothing about it looks wrong: the +unit is active, the link to the bus is up, and the log says it is hearing what the node should be. + +Measured: after the crossover the machine logged nothing further, where the previous host had written +a reconcile line every five minutes. + +## Why this was found rather than reported + +Nothing reports it. The host does not check its own stamps at start, the mesh does not ask, and the +declaration that would fail is the same declaration that would deliver a fix — so **a machine in this +state cannot be repaired by the mesh.** It was restored by moving the delivered versions aside and +letting the launcher fall back to the hand-placed binary, which is the fallback working exactly as +designed. + +## What the records already say about half of it + +[ADR 0142](../../02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md) settles the +version and its answer is not implemented: + +> A component's version comes from where it sits, not from its linker. It is unpacked into a directory +> named for its version, so it can read its own version from its path. The stamp goes, and with it the +> need for a build to know what it will be called. + +That is exactly right and would also fix what the mesh reports: a delivered host would say +`637f65559d16` rather than `development build`, and +[issue 087](../087-the-controller-cannot-tell-a-host-is-too-old/00-report.md)'s host comparison would +mean something for delivered hosts. + +**The system pin has no answer yet**, and it needs one before any mesh-built host can apply anything. +The tension is real: the target is a property of the artifact and 0142 says so, but a toolchain that +passed it would be linking a value into a variable whose name belongs to the module — which is the +coupling the toolchain exists to avoid. Candidates, none decided: + +- the path carries it as well as the version, so the host reads both from where it sits, as 0142 does + for the version; +- the bundle carries a small file beside the binary saying what it was built for, written by the + builder from the artifact's declaration; +- the host stops being pinned at link time and refuses on a fact it reads from the machine instead — + which changes what [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) decided and is the biggest of + the three. + +## What is true in the meantime + +Self-update works end to end and is one fact short of usable: the mesh builds the host, publishes it, +delivers it to a machine, the running host stands aside, and the launcher starts the delivered one. The +machine is left on its hand-placed binary until this is answered, which is one command to undo. + +## How a fix is checked + +A host the mesh built and delivered applies a declaration on a machine, shown by the machine's own +reconcile line; and it reports a version that names the build it came from rather than a placeholder. From 3c535ead3132ebe98776721e16b5ee08fc8347c9 Mon Sep 17 00:00:00 2001 From: jochen Date: Wed, 30 Sep 2026 13:16:37 +0200 Subject: [PATCH 2/2] The host self-updates, and an archive cannot be undeclared MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 161 resolved and verified on a machine: the workstation runs a host the mesh compiled, published, delivered and started, applying declarations and reporting the version it was delivered as. The system it was built for comes from the artifact — the one thing a toolchain takes from a module, which 0142 already allowed because the target is a property of the artifact. The version comes from where the binary sits, which 0142 decided and nothing had implemented. Two mistakes on the way, both caught by reading the output rather than the line that claimed success. A second -ldflags does not merge with the first: the binary gained its system and lost -s -w, 12.2MB against 8.5MB. And the delivered binary was named after its package, so the first delivery was correct, reported success and was invisible to the launcher. A delivered host that cannot apply is a machine the mesh cannot repair, because the declaration that would fix it is the one it cannot apply. The launcher's fallback is what made that an inconvenience instead of an expedition. 162 is new and not about the host: an archive has no removal, so a module using one can never be unassigned, and the attempt takes the whole apply with it — the machine applies nothing else either. It is how undoing the first delivery froze the workstation. --- .../01-progress.md | 20 +++++++ .../00-report.md | 4 +- .../01-resolution.md | 59 +++++++++++++++++++ .../00-report.md | 56 ++++++++++++++++++ 4 files changed, 137 insertions(+), 2 deletions(-) create mode 100644 04-ISSUES/161-a-delivered-host-carries-none-of-its-link-time-facts/01-resolution.md create mode 100644 04-ISSUES/162-an-archive-cannot-be-undeclared/00-report.md diff --git a/04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/01-progress.md b/04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/01-progress.md index a501f6c..3114ee0 100644 --- a/04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/01-progress.md +++ b/04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/01-progress.md @@ -118,3 +118,23 @@ is back on its hand-placed binary until it is answered. **The fallback is what made that safe**, and it was not luck: the launcher runs the pinned version, or the newest delivered one, or the one placed by hand — so moving the delivered versions aside restored the machine in one step. + +## Self-update works (2026-09-30, end of the day) + +``` +running: /usr/lib/nox-mesh-host/versions/093231796eb0/nox-mesh-host +mesh-controller node show shanks + host 093231796eb0 +``` + +One machine runs a host the mesh compiled, published, delivered and started, applying declarations and +reporting the version it was delivered as. The two facts a delivered binary was missing are +[issue 161](../161-a-delivered-host-carries-none-of-its-link-time-facts/01-resolution.md) and resolved: +the system comes from the artifact, the version from where the binary sits. + +**Three machines still run a hand-placed host**, and rolling each forward is one assignment and one +push. The control node is worth last. + +One thing this found on the way out: an archive cannot be undeclared, and the attempt stops the machine +applying anything at all — [issue 162](../162-an-archive-cannot-be-undeclared/00-report.md). It is how +undoing the first delivery froze the workstation, and it is not specific to the host. diff --git a/04-ISSUES/161-a-delivered-host-carries-none-of-its-link-time-facts/00-report.md b/04-ISSUES/161-a-delivered-host-carries-none-of-its-link-time-facts/00-report.md index d09a197..8b856a4 100644 --- a/04-ISSUES/161-a-delivered-host-carries-none-of-its-link-time-facts/00-report.md +++ b/04-ISSUES/161-a-delivered-host-carries-none-of-its-link-time-facts/00-report.md @@ -1,10 +1,10 @@ --- -status: located +status: resolved opened: 2026-09-30 located-in: - mesh-host cmd/mesh-host/main.go (version and builtFor, both set at link time) - mesh-controller internal/builder (the toolchain, which deliberately takes nothing from the module) -fixed-by: +fixed-by: mesh-controller (the system stamp, and one linker flag rather than two), mesh-host (the version read from the path) — verified on a machine 2026-09-30, 01-resolution.md amended-design: --- diff --git a/04-ISSUES/161-a-delivered-host-carries-none-of-its-link-time-facts/01-resolution.md b/04-ISSUES/161-a-delivered-host-carries-none-of-its-link-time-facts/01-resolution.md new file mode 100644 index 0000000..c3c0cf7 --- /dev/null +++ b/04-ISSUES/161-a-delivered-host-carries-none-of-its-link-time-facts/01-resolution.md @@ -0,0 +1,59 @@ +# 161 — resolved: a host the mesh built runs a machine + +*2026-09-30. Measured on the workstation.* + +``` +running: /usr/lib/nox-mesh-host/versions/093231796eb0/nox-mesh-host +agent: active +reconciles in the last six minutes: 1 + +mesh-controller node show shanks + host 093231796eb0 +``` + +A binary the mesh compiled, published to its own registry, delivered over the bus, started by the +launcher, applying declarations, and reporting a version that names the build it came from. + +## The two facts, and where each now comes from + +**The system it was built for comes from the artifact.** ADR 0142 already made the target a property +of the artifact rather than of the recipe, so the toolchain names the variable it fills and the +artifact supplies the value. It is the one thing a toolchain takes from a module, and it is stated +rather than inferred. + +**The version comes from where the binary sits**, which is what +[ADR 0142](../../02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md) decided and +nothing had implemented: a delivered host reads the directory it was unpacked into. A host placed by +hand keeps its link-time stamp, which is the honest answer for one the mesh did not deliver — and is +every other machine today. + +## Two mistakes on the way, both found by reading the output + +**A repeated flag is not a merged one.** The stamp was appended as a second `-ldflags`, and the Go +command takes the last and drops the first. The binary gained its system and lost `-s -w`: 12.2MB +against 8.5MB, with its debug info. The comment I had written said the linker "accepts and merges" +them. It does not. Linker flags are the toolchain's own list now, composed into one flag, and a test +refuses a compile line that carries `-ldflags` itself. + +**The delivered binary was named after its package.** `cmd/mesh-host` builds `mesh-host`; every +machine runs `nox-mesh-host`, which is what the launcher looks for inside a version. The first +delivery landed, reported `created … 1 file(s)`, and was invisible. An artifact says what its +executable is called now. + +Both were caught by listing the directory and reading the binary rather than believing the line that +said it worked. + +## What this cost while it was wrong, and what saved it + +A delivered host that cannot apply is a machine the mesh cannot repair, because the declaration that +would fix it is the declaration it cannot apply. The workstation was restored by moving the delivered +versions aside so the launcher fell back to the hand-placed binary — **the fallback in the launcher, +working exactly as designed**, and the reason this was an inconvenience rather than an expedition. + +It also loops if you are not careful: the working binary applies, delivers a version, stands aside, +and the broken one starts. Stopping the unit while the fix was built was the way through. + +## What is left + +**Three machines still run a hand-placed host.** Rolling them forward is one assignment and one push +each, and the control node is worth doing last and watching. diff --git a/04-ISSUES/162-an-archive-cannot-be-undeclared/00-report.md b/04-ISSUES/162-an-archive-cannot-be-undeclared/00-report.md new file mode 100644 index 0000000..7b28013 --- /dev/null +++ b/04-ISSUES/162-an-archive-cannot-be-undeclared/00-report.md @@ -0,0 +1,56 @@ +--- +status: located +opened: 2026-09-30 +located-in: [mesh-host internal/apply (no removal for an archive)] +fixed-by: +amended-design: +--- + +# 162 — An archive cannot be undeclared, and trying stops the machine applying anything + +## What was observed + +*2026-09-30, unassigning the host module from the workstation to undo a delivery.* + +``` +holding this machine: 0 applied, and map[apply:applying "mesh-host.next": no way to remove a "archive" +0 resource(s) were applied and remain; everything was attempted, so what is not listed as failed was done. +``` + +**Nothing was applied at all** — not the archive, not the other forty resources that had nothing to do +with it. The machine stopped reconciling and stayed that way until the module was assigned again. + +## Why it matters + +Every other resource kind can be taken away. A file is removed and what was found under it is put +back; a container is stopped and removed; a unit is given back the state it was found in +([ADR 0118](../../02-DECISIONS/0118-undeclaring-gives-a-unit-back-the-state-it-was-found-in.md)). An +archive has no removal at all, so: + +- **a module with an archive can never be unassigned** — the attempt fails for ever; +- **the failure takes the whole apply with it**, so the machine applies nothing else either, and one + unassignable resource is a machine frozen against every other change; +- it is silent in the mesh's terms: the push reported sent, and only the machine's own journal says + what happened. + +The host module is the obvious case and not the only one. An archive is for what inlining cannot +serve — a theme, an icon set, a tree of configuration — and any module using one is in the same +position. + +## What the right answer probably is, and the question in it + +The other kinds answer this by remembering what they found. An archive unpacks many files into a +directory the mesh did not necessarily create, so removal has a real question in it: **remove what the +archive put there, or remove the directory?** The first needs the applier to have recorded the file +list; the second would delete whatever else lives there — and for the host's own versions directory, +that is every other delivered version. + +Recording what was unpacked is the answer that matches how the rest of the host behaves, and it is +what [issue 126](../126-a-volume-path-is-not-in-the-spec-comparison/00-report.md) and ADR 0118 already +argue for elsewhere: the mesh gives back what it found. + +## How a fix is checked + +A module with an archive is assigned, pushed, unassigned and pushed again; what the archive put on the +machine is gone, anything that was in the directory beforehand is still there, and the apply that +removed it applied everything else in the same declaration.