2026-10-03 23:14:50 +00:00
7 changed files with 143 additions and 6 deletions
@@ -2,7 +2,7 @@
 layer: to-be
 status: in-progress
 code: [mesh-tools, mesh-controller, mesh-host, mesh-catalog]
-updated: 2026-10-03
+updated: 2026-10-04
 decisions:
  - 02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md
  - 02-DECISIONS/0173-the-operators-machine-is-the-meshs-and-a-module-is-what-it-declares.md
@@ -309,6 +309,20 @@ nextcloud, minio), the two whose clients exist on no system (mongodb, mssql: a d
 and last the mesh's own (mesh-catalog, mesh-vault, records, gitea, mailu, audit-logger, lab, and the
 three mains).

+*Built 2026-10-04.* Thirty-four modules no longer run their own code in a container: the first wave
+(mesh-catalog#245), the mesh's own and the media modules (mesh-catalog#248, mesh-media-catalog#13),
+with a step run where and as it is declared (mesh-host#85) and a process's words filled like a
+container's (mesh-controller#250). **Proven live** on every machine that runs them: each moved
+module's tools answer from the node's runtime, the steps run as their oneshot units, and the forge's
+merge events reach the build pipeline from the runtime — the merge after the move started its own
+plan. **Two corrections the machines taught:** a tool that called a broker's command-line client now
+runs it inside the broker's own container, because a host package may be uninstallable on a machine
+whose package index is stale (mesh-catalog#249); and a module reading its application's own key reads
+it through the application's container, because that directory belongs to the account the application
+runs as there, which is not the runtime's (mesh-media-catalog#14). **Still in a container:**
+mesh-catalog, mongodb and mssql, whose code imports npm packages of its own, which the builder cannot
+yet install into a bundle; that builder change is in progress.
+
 ## WP5 — The shell, on a server first

 *mesh-catalog #224, already written. Half a day to assign and prove.*
@@ -50,3 +50,14 @@ of a plan today.

 Located only by owner; the move is a change of the controller's module and its deployment, not of
 its code.
+
+## Fix prepared (2026-10-04)
+
+Three changes. mesh-host#86, merged: a process may name the container it `replaces`, and the host
+removes that container only once the process has stayed up across two checks. mesh-controller#252,
+awaiting the operator's merge: the controller's composition for a service process, and two
+controllers safe together for the handover — the second stands by on the controller's consumers
+until the first lets go, and all plan work holds one advisory lock. mesh-controller#253, held: the
+controller's manifest as a Go bundle and a process. It waits on
+[issue 223](../223-a-new-mesh-installs-its-controller-as-a-container/00-report.md), because with it a
+new mesh cannot be installed.
@@ -1,9 +1,10 @@
 ---
-status: diagnosing
+status: resolved
 opened: 2026-10-04
 located-in:
  - mesh-controller
 fixed-by:
+  - mesh-controller#249
 amended-design:
 ---

@@ -35,3 +36,13 @@ build that started before it existed. Every passing check still passes.
 How a finished build is recorded and how a module's current artifact is chosen. **How it is checked:**
 a test in which an older request completes after a newer one for the same artifact, and the newer
 stays current.
+
+## Resolved
+
+A build is ordered by when it was requested, read from the id the controller gives it, and a
+module's registered manifest is replaced only by a build requested at or after the one it came from.
+An older request finishing later is recorded and changes nothing; a plan settles only from builds it
+asked for itself. **How it is checked:** store-backed tests replay the incident — the newer request
+stays what the module is — and fail without the fix. Live since 2026-10-04: the rebuilds of every
+runtime-image module and the three waves of module code moves since then each registered the build
+they asked for.
@@ -1,8 +1,10 @@
 ---
-status: open
+status: resolved
 opened: 2026-10-04
-located-in: []
+located-in:
+  - mesh-host
 fixed-by:
+  - mesh-host#84
 amended-design:
 ---

@@ -28,3 +30,12 @@ read from the artifact.

 How the host unpacks a bundle into its directory. **How it is checked:** deliver a bundle, then a
 version without one of its files, and the file is gone.
+
+## Resolved
+
+An archive is unpacked into a fresh directory beside the old one and swapped in by rename; a refused
+or failed unpack leaves the old tree whole. **How it is checked:** a second delivery without a file
+removes it, and nothing is left beside the directory; both tests fail without the fix. Proven
+2026-10-04: a bundle rebuilt and delivered after the fix holds exactly the new build and nothing
+beside it. A bundle that has not changed since keeps its old leftovers until its next version, by
+design: an unchanged archive is not unpacked again.
@@ -1,8 +1,10 @@
 ---
-status: open
+status: located
 opened: 2026-10-04
-located-in: []
+located-in:
+  - mesh-controller
 fixed-by:
+  - policy: `upgrade build-agent roll-out` (2026-10-04)
 amended-design:
 ---

@@ -31,3 +33,14 @@ Whether a controller plan should end by delivering itself to the build machines
 machine, or whether a build should refuse a builder older than the controller that asked for it.
 **How it is checked:** after a controller merge, a build asked right after its plan finishes runs
 the new builder.
+
+## Located
+
+The mechanism existed: a module whose upgrade policy is `roll-out` is sent to its machines when its
+tier is built, and the plan's next tier waits until it is applied. The build agent's policy was
+`record` — built, never sent — as was that of 76 other modules, which is why every rollout on
+2026-10-03 needed a push by hand. The build agent was set to `roll-out`, one machine at a time,
+stopping at the first failure. **How it is checked:** the next controller merge's plan sends the
+build agent to the build machines before its last tier, and a build asked right after the plan
+finishes runs the new builder; then this moves to resolved. Whether the other modules roll out is
+the operator's policy, not this issue's.
@@ -0,0 +1,35 @@
+---
+status: open
+opened: 2026-10-04
+located-in: []
+fixed-by:
+amended-design:
+---
+
+# 222 — An assignment is refused on the bus until the bus's machine is pushed
+
+## What was observed
+
+2026-10-04. A module was assigned to the laptop, and the laptop alone was pushed. The module's two
+bundles arrived and the node's runtime launched both, and then the bus refused every one of the
+module's subjects:
+
+```
+nats: permissions violation: Permissions Violation for Subscription to "mesh.mod.<module>.tool.<tool>.<node>"
+```
+
+The tools were unreachable until a later push that included the machine running the bus. Then the
+runtime served them without a restart, because a new membership arrived and it re-subscribed.
+
+## Why it matters beyond this instance
+
+What an account may answer lives in the bus's user list, and the controller writes that list only
+into the declaration of the machine that runs the bus. Assigning anything to any machine changes
+that list, so `push <machine>` after `assign <machine> <module>`, which is what the controller itself
+tells the operator to run, leaves the module running and unreachable, with nothing reporting a fault.
+
+## Where to look
+
+Whether a push to one machine should also send the bus's machine when the user list it would
+compose differs from the one that machine holds. **How it is checked:** assign a module with tools
+to a machine that does not run the bus, push only that machine, and its tools answer.
@@ -0,0 +1,42 @@
+---
+status: open
+opened: 2026-10-04
+located-in:
+  - mesh-host
+fixed-by:
+amended-design:
+---
+
+# 223 — A new mesh installs its controller as a container
+
+## What was observed
+
+2026-10-04. Preparing [issue 213](../213-the-controller-is-a-go-program-run-in-a-container/00-report.md),
+the controller's own manifest was changed to a Go bundle run as a process. The installer that raises a
+new mesh assumes the controller is an image and a container at every step from its third on:
+
+- it requires the controller's build to produce exactly one image;
+- it starts a temporary controller from that image, and publishes the image to the registry;
+- at the pivot, it finds the controller's container in the declaration, reads its environment and
+  volumes, waits for it, and from then on talks to the controller only through the container.
+
+## Why it matters beyond this instance
+
+With the controller's manifest changed, a new mesh cannot be installed: the pivot fails. The deeper
+constraint is ordering. A process's bundle is fetched from the artifact store, and the installer
+raises the artifact store only after the pivot, so the controller's first declaration names a bundle
+nothing can serve yet.
+
+## What a fix has to settle
+
+One of two shapes, and it is a decision, not a repair:
+
+1. raise the artifact store before the pivot, publish the controller's bundle to it, and talk to the
+   controller from the host's side rather than through a container; or
+2. pivot to the image form as today, and let the first push hand over to the process, which
+   requires the controller's manifest to carry both forms.
+
+Until it is settled, the change of the controller's manifest (mesh-controller#253) is held. The
+handover itself is built and merged (mesh-host#86); the controller's half (mesh-controller#252) waits
+on the operator. **How it is checked:** the installer's test raises a mesh whose controller manifest
+is the process form, and the controller answers its seat's verbs at the end.