Issues 219, 220 resolved, 221 located, 222 and 223 opened; WP4c built and proven

This commit is contained in:
jochen
2026-10-04 01:14:44 +02:00
parent c2fc829822
commit df503d1cff
7 changed files with 143 additions and 6 deletions
@@ -2,7 +2,7 @@
layer: to-be layer: to-be
status: in-progress status: in-progress
code: [mesh-tools, mesh-controller, mesh-host, mesh-catalog] code: [mesh-tools, mesh-controller, mesh-host, mesh-catalog]
updated: 2026-10-03 updated: 2026-10-04
decisions: decisions:
- 02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md - 02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md
- 02-DECISIONS/0173-the-operators-machine-is-the-meshs-and-a-module-is-what-it-declares.md - 02-DECISIONS/0173-the-operators-machine-is-the-meshs-and-a-module-is-what-it-declares.md
@@ -309,6 +309,20 @@ nextcloud, minio), the two whose clients exist on no system (mongodb, mssql: a d
and last the mesh's own (mesh-catalog, mesh-vault, records, gitea, mailu, audit-logger, lab, and the and last the mesh's own (mesh-catalog, mesh-vault, records, gitea, mailu, audit-logger, lab, and the
three mains). three mains).
*Built 2026-10-04.* Thirty-four modules no longer run their own code in a container: the first wave
(mesh-catalog#245), the mesh's own and the media modules (mesh-catalog#248, mesh-media-catalog#13),
with a step run where and as it is declared (mesh-host#85) and a process's words filled like a
container's (mesh-controller#250). **Proven live** on every machine that runs them: each moved
module's tools answer from the node's runtime, the steps run as their oneshot units, and the forge's
merge events reach the build pipeline from the runtime — the merge after the move started its own
plan. **Two corrections the machines taught:** a tool that called a broker's command-line client now
runs it inside the broker's own container, because a host package may be uninstallable on a machine
whose package index is stale (mesh-catalog#249); and a module reading its application's own key reads
it through the application's container, because that directory belongs to the account the application
runs as there, which is not the runtime's (mesh-media-catalog#14). **Still in a container:**
mesh-catalog, mongodb and mssql, whose code imports npm packages of its own, which the builder cannot
yet install into a bundle; that builder change is in progress.
## WP5 — The shell, on a server first ## WP5 — The shell, on a server first
*mesh-catalog #224, already written. Half a day to assign and prove.* *mesh-catalog #224, already written. Half a day to assign and prove.*
@@ -50,3 +50,14 @@ of a plan today.
Located only by owner; the move is a change of the controller's module and its deployment, not of Located only by owner; the move is a change of the controller's module and its deployment, not of
its code. its code.
## Fix prepared (2026-10-04)
Three changes. mesh-host#86, merged: a process may name the container it `replaces`, and the host
removes that container only once the process has stayed up across two checks. mesh-controller#252,
awaiting the operator's merge: the controller's composition for a service process, and two
controllers safe together for the handover — the second stands by on the controller's consumers
until the first lets go, and all plan work holds one advisory lock. mesh-controller#253, held: the
controller's manifest as a Go bundle and a process. It waits on
[issue 223](../223-a-new-mesh-installs-its-controller-as-a-container/00-report.md), because with it a
new mesh cannot be installed.
@@ -1,9 +1,10 @@
--- ---
status: diagnosing status: resolved
opened: 2026-10-04 opened: 2026-10-04
located-in: located-in:
- mesh-controller - mesh-controller
fixed-by: fixed-by:
- mesh-controller#249
amended-design: amended-design:
--- ---
@@ -35,3 +36,13 @@ build that started before it existed. Every passing check still passes.
How a finished build is recorded and how a module's current artifact is chosen. **How it is checked:** How a finished build is recorded and how a module's current artifact is chosen. **How it is checked:**
a test in which an older request completes after a newer one for the same artifact, and the newer a test in which an older request completes after a newer one for the same artifact, and the newer
stays current. stays current.
## Resolved
A build is ordered by when it was requested, read from the id the controller gives it, and a
module's registered manifest is replaced only by a build requested at or after the one it came from.
An older request finishing later is recorded and changes nothing; a plan settles only from builds it
asked for itself. **How it is checked:** store-backed tests replay the incident — the newer request
stays what the module is — and fail without the fix. Live since 2026-10-04: the rebuilds of every
runtime-image module and the three waves of module code moves since then each registered the build
they asked for.
@@ -1,8 +1,10 @@
--- ---
status: open status: resolved
opened: 2026-10-04 opened: 2026-10-04
located-in: [] located-in:
- mesh-host
fixed-by: fixed-by:
- mesh-host#84
amended-design: amended-design:
--- ---
@@ -28,3 +30,12 @@ read from the artifact.
How the host unpacks a bundle into its directory. **How it is checked:** deliver a bundle, then a How the host unpacks a bundle into its directory. **How it is checked:** deliver a bundle, then a
version without one of its files, and the file is gone. version without one of its files, and the file is gone.
## Resolved
An archive is unpacked into a fresh directory beside the old one and swapped in by rename; a refused
or failed unpack leaves the old tree whole. **How it is checked:** a second delivery without a file
removes it, and nothing is left beside the directory; both tests fail without the fix. Proven
2026-10-04: a bundle rebuilt and delivered after the fix holds exactly the new build and nothing
beside it. A bundle that has not changed since keeps its old leftovers until its next version, by
design: an unchanged archive is not unpacked again.
@@ -1,8 +1,10 @@
--- ---
status: open status: located
opened: 2026-10-04 opened: 2026-10-04
located-in: [] located-in:
- mesh-controller
fixed-by: fixed-by:
- policy: `upgrade build-agent roll-out` (2026-10-04)
amended-design: amended-design:
--- ---
@@ -31,3 +33,14 @@ Whether a controller plan should end by delivering itself to the build machines
machine, or whether a build should refuse a builder older than the controller that asked for it. machine, or whether a build should refuse a builder older than the controller that asked for it.
**How it is checked:** after a controller merge, a build asked right after its plan finishes runs **How it is checked:** after a controller merge, a build asked right after its plan finishes runs
the new builder. the new builder.
## Located
The mechanism existed: a module whose upgrade policy is `roll-out` is sent to its machines when its
tier is built, and the plan's next tier waits until it is applied. The build agent's policy was
`record` — built, never sent — as was that of 76 other modules, which is why every rollout on
2026-10-03 needed a push by hand. The build agent was set to `roll-out`, one machine at a time,
stopping at the first failure. **How it is checked:** the next controller merge's plan sends the
build agent to the build machines before its last tier, and a build asked right after the plan
finishes runs the new builder; then this moves to resolved. Whether the other modules roll out is
the operator's policy, not this issue's.
@@ -0,0 +1,35 @@
---
status: open
opened: 2026-10-04
located-in: []
fixed-by:
amended-design:
---
# 222 — An assignment is refused on the bus until the bus's machine is pushed
## What was observed
2026-10-04. A module was assigned to the laptop, and the laptop alone was pushed. The module's two
bundles arrived and the node's runtime launched both, and then the bus refused every one of the
module's subjects:
```
nats: permissions violation: Permissions Violation for Subscription to "mesh.mod.<module>.tool.<tool>.<node>"
```
The tools were unreachable until a later push that included the machine running the bus. Then the
runtime served them without a restart, because a new membership arrived and it re-subscribed.
## Why it matters beyond this instance
What an account may answer lives in the bus's user list, and the controller writes that list only
into the declaration of the machine that runs the bus. Assigning anything to any machine changes
that list, so `push <machine>` after `assign <machine> <module>`, which is what the controller itself
tells the operator to run, leaves the module running and unreachable, with nothing reporting a fault.
## Where to look
Whether a push to one machine should also send the bus's machine when the user list it would
compose differs from the one that machine holds. **How it is checked:** assign a module with tools
to a machine that does not run the bus, push only that machine, and its tools answer.
@@ -0,0 +1,42 @@
---
status: open
opened: 2026-10-04
located-in:
- mesh-host
fixed-by:
amended-design:
---
# 223 — A new mesh installs its controller as a container
## What was observed
2026-10-04. Preparing [issue 213](../213-the-controller-is-a-go-program-run-in-a-container/00-report.md),
the controller's own manifest was changed to a Go bundle run as a process. The installer that raises a
new mesh assumes the controller is an image and a container at every step from its third on:
- it requires the controller's build to produce exactly one image;
- it starts a temporary controller from that image, and publishes the image to the registry;
- at the pivot, it finds the controller's container in the declaration, reads its environment and
volumes, waits for it, and from then on talks to the controller only through the container.
## Why it matters beyond this instance
With the controller's manifest changed, a new mesh cannot be installed: the pivot fails. The deeper
constraint is ordering. A process's bundle is fetched from the artifact store, and the installer
raises the artifact store only after the pivot, so the controller's first declaration names a bundle
nothing can serve yet.
## What a fix has to settle
One of two shapes, and it is a decision, not a repair:
1. raise the artifact store before the pivot, publish the controller's bundle to it, and talk to the
controller from the host's side rather than through a container; or
2. pivot to the image form as today, and let the first push hand over to the process, which
requires the controller's manifest to carry both forms.
Until it is settled, the change of the controller's manifest (mesh-controller#253) is held. The
handover itself is built and merged (mesh-host#86); the controller's half (mesh-controller#252) waits
on the operator. **How it is checked:** the installer's test raises a mesh whose controller manifest
is the process form, and the controller answers its seat's verbs at the end.