From f1941304cc17459ad96da377d45f8363ea281049 Mon Sep 17 00:00:00 2001 From: jochens Date: Thu, 1 Oct 2026 23:13:18 +0200 Subject: [PATCH 1/3] ADRs 0164-0166 and issue 190: the container runtime gets a module, a seat and declared settings Proposed for the operator's review: settings declared with defaults and cost (0164), container-runtime as a kernel capability (0165), node-container-runtime seat with the host creating containers through its holder (0166), and the runtime's file written by modules that are not its own (190). --- ...-its-meaning-and-what-changing-it-costs.md | 131 +++++++++++++++ ...a-running-runtime-is-its-holders-health.md | 105 ++++++++++++ ...t-creates-containers-through-its-holder.md | 151 ++++++++++++++++++ 02-DECISIONS/README.md | 3 + .../00-report.md | 76 +++++++++ 5 files changed, 466 insertions(+) create mode 100644 02-DECISIONS/0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md create mode 100644 02-DECISIONS/0165-container-runtime-is-what-a-machine-can-run-and-a-running-runtime-is-its-holders-health.md create mode 100644 02-DECISIONS/0166-the-container-runtime-is-a-node-seat-and-the-host-creates-containers-through-its-holder.md create mode 100644 04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md diff --git a/02-DECISIONS/0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md b/02-DECISIONS/0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md new file mode 100644 index 0000000..44a096e --- /dev/null +++ b/02-DECISIONS/0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md @@ -0,0 +1,131 @@ +--- +topic: what runs on it +status: proposed +date: 2026-10-01 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0046-a-module-configuration-is-its-assignments-not-its-manifest.md +--- + +# 164. A setting is declared with its default, its meaning and what changing it costs + +## Context + +The operator asked for one thing for every module, with the container runtime as the first case: **one +consistent default configuration for every machine, overridable per assignment, and easy to change +later.** The four machines' runtime configurations were each written by hand and differ — one keeps +running containers through a daemon restart and one does not, their log rotation differs, and each +names its resolver and its trusted registries in its own words. + +Most of this was already decided. +[ADR 0046](0046-a-module-configuration-is-its-assignments-not-its-manifest.md) said the definition is +identity and **defaults**, the assignment's settings are the configuration, *unset is the default*, and +*an unknown setting is refused*. [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md) made +a setting an operator requirement whose contract is "a type and, optionally, a default", answered by +"the assignment's, or the requirement's default, or unresolved". An assignment is a module on a node, +so the node layer of a module's settings already *is* the per-assignment override, and the mesh-wide +layer is the one consistent default a person changes once. + +What was built is narrower than what was decided, measured in the controller on the day of deciding: + +- **Nothing declares which keys are settable.** A mergeable file's content is its defaults, and every + key of it — and every key not in it — is accepted. Nothing tells a person, or the console, what can be + set, of what type, or what it means. +- **The refusal of an unknown setting is not there for most modules.** The stray-setting report returns + nothing at all for a module with any mergeable file, because such a file "takes any key" + ([issue 173](../04-ISSUES/173-a-modules-settings-reach-every-fact-it-contributes/00-report.md) left + files that way on purpose). It reports rather than refuses where it does run. +- **A value in a file that is not JSON can have no default.** `${setting:}` + ([ADR 0155](0155-a-definition-names-no-installation-and-how-that-is-checked.md)) is refused when no + layer sets it — right for a mail domain, where a default is the very literal 0155 removes, and wrong + for a tunable like the resolver's upstreams, which the resolver module therefore carries as literals + in its file. +- **What a change costs is said per file, not per key.** A service names the files it is reloaded or + restarted on. The runtime re-reads its trusted registries on a reload and its `dns` key only when it + starts; the resolver module declared a reload, so on two machines the key was written, reloaded, + and never read, and every container got a public resolver for weeks while everything read as + current ([issue 110](../04-ISSUES/110-a-container-on-the-runtimes-own-network-cannot-reach-the-resolver/01-resolution.md)). + +## Considered Options + +1. **Leave settings implicit; document each module's keys in its README.** Rejected: a key the mesh + does not know cannot be refused, typed, listed by the console or costed, and a README is a rule + enforced by nothing. +2. **A second mechanism for tunables beside settings** — defaults in a new block, settings untouched. + Rejected: two ways to state one person's value, and design 27 already retires six mechanisms + that grew that way. +3. **Settings declared in the definition, as 0112's operator requirement: a key, a type, a meaning, + optionally a default, and what a change costs.** Adopted. + +## Decision + +**A module declares every setting it takes.** Each declared setting has a name, a type, one sentence +of meaning, optionally a default, and what a change to it costs. The spelling is design 27's to settle +with the rest of the requirement form; this record decides the content. + +**A setting with a default is a tunable; a setting without one is the operator's.** A tunable resolves +to its default when no layer sets it — wherever it is read, a mergeable file or `${setting:}` in a +file of any format. A setting with no default is refused by name when nothing sets it, as 0155 decided; +0155's refusal is narrowed to exactly that case, not changed for it. Whether a value has a default is a +fact about the software (a log size does, a mail domain does not), and the definition states it once. + +**The layers stay as they are, and every value says where it came from.** The definition's default, +then the mesh-wide layer, then the node's — later wins, objects merge, lists replace. One consistent +configuration for every machine is the default plus the mesh-wide layer; one machine that differs says +so in its own layer and nothing else. Asked for a module's configuration on a machine, the mesh lists +every declared setting with its effective value and its source: *default*, *mesh*, or *node*. + +**Changing later is changing one of three places, and the plan shows its reach before anything moves.** +A new default ships with the module's next version and reaches every assignment that does not override +it; a mesh-wide setting reaches every assignment of the module; a node's reaches one. The plan of a +change names each assignment whose effective value moves. + +**A declared setting is the only kind accepted.** Setting a key the module does not declare is refused +when it is set, naming the declared keys, rather than reported when the machine is planned. A module +that declares no settings keeps today's behaviour until it does; a catalogue test lists those modules, +and the list shrinks to empty before the implicit form is removed — design 27's rule for every retired +mechanism. + +**A setting says what it costs: nothing, a reload, or a restart.** When a file changes, the host +applies the strongest cost among the settings whose values moved in it, so a key the software reads +only at start can no longer be written and never read. A service's `reload-on` and `restart-on` keep +naming the files that are not settings — a generated roster, a credential. + +**The container runtime is the first module to declare its settings** and the model for the rest: +its log rotation, keeping containers through a daemon restart, and its resolver are tunables, and +its trusted registries are what the mesh tells it. + +## Consequences + +- The console can show a module's settings as a form: what can be set, of what type, its default, + and where the current value came from. That is the surface the operator wants for changing a + default later. +- `settings set` can refuse an unknown key, so ADR 0046's rule is enforced where it was only stated. +- The resolver's upstreams, the runtime's log rotation, and other literals a definition carries + because it could not give them a default become declared tunables. +- **What got harder:** every module that takes settings must list them, and a mergeable file no + longer silently accepts a key its author did not foresee. A person who needs one adds it to the + definition, which is a new module version, not a setting. +- Issue 173's open question — a consumer checks nothing against a contract — is unchanged; this record + is the operator half of design 27's contract, not the provider half. +- Not decided here: the spelling (design 27); whether a node's layer may be narrowed to a single key + rather than replaced whole, as `settings set` does today. + +## How this is checked + +| Rule | Checked by | +|---|---| +| Every setting a module takes is declared | A parser test refusing a setting declaration without a type or meaning; a catalogue test listing modules with mergeable files or `${setting:}` and no declarations, which must be empty before the implicit form is removed | +| A tunable resolves to its default; an operator value without one is refused | Resolution tests: an unset tunable in a JSON file and in a text file both take the default; an unset setting with no default is refused naming it (0155's existing test) | +| An undeclared key is refused when set | A controller test: `settings set` with an undeclared key fails naming the declared keys, and nothing is stored | +| Every effective value names its source | A test listing a module's configuration on a node with one key from each of default, mesh and node | +| A change's reach is shown before it moves | A plan test: changing a mesh-wide setting names every assignment whose effective value moves and no other | +| The strongest cost applies | A host test: a file where a reload-cost key and a restart-cost key both moved restarts; a file where only reload-cost keys moved reloads | +| Live | The container runtime's module lists its settings with their sources on every machine, and a mesh-wide change to its log rotation reaches all four at the next push | + +## References + +- [ADR 0046](0046-a-module-configuration-is-its-assignments-not-its-manifest.md), [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md), [ADR 0155](0155-a-definition-names-no-installation-and-how-that-is-checked.md), [ADR 0102](0102-the-mesh-writes-into-a-shared-file-never-over-it.md) +- [Design 27 — a module requires, the mesh resolves](../03-DESIGN/01-to-be/27-a-module-requires-the-mesh-resolves.md) +- Issues [110](../04-ISSUES/110-a-container-on-the-runtimes-own-network-cannot-reach-the-resolver/00-report.md), [173](../04-ISSUES/173-a-modules-settings-reach-every-fact-it-contributes/00-report.md) +- mesh-controller `internal/catalogue/settings.go` (`settle`, `UnusedSettings`), `internal/catalogue/setting_into.go` diff --git a/02-DECISIONS/0165-container-runtime-is-what-a-machine-can-run-and-a-running-runtime-is-its-holders-health.md b/02-DECISIONS/0165-container-runtime-is-what-a-machine-can-run-and-a-running-runtime-is-its-holders-health.md new file mode 100644 index 0000000..64f693e --- /dev/null +++ b/02-DECISIONS/0165-container-runtime-is-what-a-machine-can-run-and-a-running-runtime-is-its-holders-health.md @@ -0,0 +1,105 @@ +--- +topic: what runs on it +status: proposed +date: 2026-10-01 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0161-what-deserves-a-seat.md +--- + +# 165. `container-runtime` is what a machine can run; that a runtime is running is its holder's health + +## Context + +A capability is a requirement a module places on a machine, detected by the host and renewed with +every report ([ADR 0161](0161-what-deserves-a-seat.md)). The host's `container-runtime` asks the +daemon for its version: *a running daemon, not an installed client*. It was made that way by +[issue 007](../04-ISSUES/007-an-installed-package-is-not-a-capability/00-report.md), where an +installed package was believed to be a working service, and +[design 05](../03-DESIGN/01-to-be/05-the-node-host.md)'s table says the same: *a runtime is +running*. The installer's preflight borrows the same detector to wait for the runtime the +foundation bundle installs, so there is one answer to "is there a runtime here". + +The mesh is now to have a module for the runtime itself — its packages, its configuration, its +service — on every machine ([ADR 0166](0166-the-container-runtime-is-a-node-seat-and-the-host-creates-containers-through-its-holder.md)). +That module cannot declare `container-runtime` as defined: it would require the very thing it +installs, the cycle [research 011](../01-RESEARCH/011-the-module-graph/cases.md)'s case 12 names +("something the mesh installs that then becomes a node capability"). The operator defined the word +for it: **`container-runtime` means the machine is able, at the kernel level, to install a runtime and +execute containers** — not that one is installed, and not that one is running. + +The host already draws this line once. `seat` is hardware, a display server *could* run here; +`graphical-session` is state, one *is* running; the detector's own comment says "assignment needs the +first". A machine without a display has no seat however much software is installed, and a machine +with one has a seat before anything is. + +Fifty-four catalogue modules declare `container-runtime` today, counted on the catalogue's main +branch on the day of deciding: every module that delivers a container. Each relies on the current +meaning to keep it off a machine with no running runtime. + +## Considered Options + +1. **Keep the meaning; let the runtime's module declare nothing.** Rejected: a module that installs + the runtime has requirements on the machine — the kernel features without which installing it is + pointless — and would state none of them. The cycle stays, only hidden. +2. **Two capabilities, "can run" and "is running".** Rejected: the second is made true by assigning a + module, so it is the module's state, not a fact of the machine; a capability the mesh itself + flips by its own assignment is case 12's cycle with an extra name. +3. **The capability is the kernel's; whether a runtime runs is the runtime module's health, and a + module that delivers a container needs the runtime's seat held.** Adopted. + +## Decision + +**`container-runtime` is detected from what the kernel offers**, as `seat` is: the namespaces a +container needs, a control-group hierarchy the runtime can manage, and an overlay filesystem the +running kernel has or can load. Present when all three are; absent naming the missing one. Nothing is +run and no runtime is asked. The verdict's detail names what was found, not a runtime's version. + +**"A runtime is running and answers" is one probe, owned by the host and used twice:** by the +installer's preflight, which waits for the runtime the foundation installs, and as the runtime +module's health. It asks the daemon, as issue 007 requires. The preflight stops borrowing the +capability's detector, and there is still one answer to "is a runtime running here". + +**The runtime's module declares `container-runtime`**, with `package-manager`, `service-manager` and +`privileged`, like any module that manages machine software. + +**A module that delivers a container needs the runtime seat held on its machine**, and is refused +otherwise, naming the seat and the modules that could hold it — the refusal design 27 already lists +for an unheld seat. That requirement is derived from the container resource and needs no manifest +field ([ADR 0166](0166-the-container-runtime-is-a-node-seat-and-the-host-creates-containers-through-its-holder.md)). +The fifty-four existing declarations of the capability stay valid and become redundant; a catalogue +test lists them, and they retire when the list is empty. + +**The order is fixed, not preferred.** The detector changes only once the seat requirement is +enforced. In between, a machine with the kernel and no running runtime would read as able to run +every containerised module, which is issue 007 again. + +## Consequences + +- Design 05's capability table changes its `container-runtime` row from *a runtime is running* to + *the kernel can run containers*, and names the runtime module's health as where "running" is now + asked. +- The node listing stops showing the runtime's version beside the capability. The version moves to + the runtime module's health and its seat's verbs. +- A fresh machine with no runtime reads as able to run one, so it can be assigned the runtime's + module, which is what makes the mesh able to install the runtime instead of the bootstrap alone. +- **What got harder:** "is this machine running containers" is no longer one glance at the profile; + it is the runtime seat's holder and its health. The node's listing should show both side by side. + +## How this is checked + +| Rule | Checked by | +|---|---| +| The capability is the kernel's | Host detector tests over a fixture `/proc` and `/sys`: all three present → present; each one missing → absent naming it; no runtime binary on the fixture machine changes nothing | +| One probe asks whether a runtime runs | A host test that the preflight and the runtime module's health call the same probe, and that the probe fails against a stopped daemon with an installed client (issue 007's shape) | +| A containerised module needs the runtime seat held | A resolution test: a module with a container resource on a machine whose runtime seat is unheld is refused, naming the seat and its candidate holders | +| The order holds | The host release that changes the detector is gated on the controller release that enforces the seat requirement — stated in both changes' descriptions and checked at review | +| Live | Every machine's profile shows `container-runtime` present with the kernel's features as its detail; a machine with no runtime installed reads present too | + +## References + +- [ADR 0161](0161-what-deserves-a-seat.md) — the profile renewed by every report; a capability that names a dialect +- [ADR 0166](0166-the-container-runtime-is-a-node-seat-and-the-host-creates-containers-through-its-holder.md) — the seat and its holder +- [Issue 007](../04-ISSUES/007-an-installed-package-is-not-a-capability/00-report.md), [research 011](../01-RESEARCH/011-the-module-graph/cases.md) cases 12–13 +- [Design 05 — the node host](../03-DESIGN/01-to-be/05-the-node-host.md) +- mesh-host `internal/profile/detectors.go` (the runtime detector), `internal/profile/seat.go` (the hardware/state split), `internal/bootstrap/preflight.go` (the preflight that borrows it) diff --git a/02-DECISIONS/0166-the-container-runtime-is-a-node-seat-and-the-host-creates-containers-through-its-holder.md b/02-DECISIONS/0166-the-container-runtime-is-a-node-seat-and-the-host-creates-containers-through-its-holder.md new file mode 100644 index 0000000..396c27b --- /dev/null +++ b/02-DECISIONS/0166-the-container-runtime-is-a-node-seat-and-the-host-creates-containers-through-its-holder.md @@ -0,0 +1,151 @@ +--- +topic: what runs on it +status: proposed +date: 2026-10-01 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0161-what-deserves-a-seat.md +--- + +# 166. The container runtime is a node seat, and the host creates containers through its holder + +## Context + +Every container the mesh runs on a machine is created by the host, which looks for a runtime +(`docker info`, then `podman info`) and drives that runtime's command line itself: run, inspect, +remove, exec. Research 012 called this "detected rather than declared": the host takes over whatever +runtime it finds. Nothing in the mesh owns the runtime. Its package came from the foundation bundle +or was already on the machine. Its configuration file was written by hand, differs on each of the +four machines, and is also written into by two modules that are not the runtime's +([issue 190](../04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md)). +Its service is declared by those same two. + +The operator set the direction: + +- a module for the runtime, on every machine, owning "what is needed to run containers here": its + packages, its configuration and its service; +- that module holds a node seat for the runtime, so a second runtime (podman) can later compete + for the seat; +- the host stops speaking to the runtime directly and uses the seat's holder. The host stays the one + that decides, and the holder becomes the one that executes; +- every container on the machine is in scope, not only the mesh's. A development environment started + by hand, or a test database a tool runs, is legitimate. The host already calls these *strays*: 3, + 8 and 25 on three of the machines on the day of deciding; +- the runtime's events and verbs are subjects on the bus, and the mesh's own interface is built on + them. The third-party interface run until now was removed by hand. + +[ADR 0161](0161-what-deserves-a-seat.md)'s test for a seat is whether the mesh's own code finds it by +name. Here it does: the host would look up the holder on its own machine. A singular role of a module +held once per machine is a `node-*` seat ([ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md)), +in the controller's seed. + +The constraint that decides most of this record is a cycle. The bus runs in containers. On the +broker's machine, the broker's own container is created by the host. A holder's code served from a +container cannot create the container that runs it. On a first machine, before the controller exists, +nothing holds anything. + +## Considered Options + +1. **The host calls the holder's verbs over the bus.** Rejected: with the broker down, no machine can + create any container, including the broker's. The mesh would be unable to restart its own + transport. +2. **The holder picks a dialect that the host speaks itself, as with the uplink.** Rejected: the + host would still drive the runtime, and the module would drive it too for every other caller. + That is two programs speaking to one daemon, and they come to disagree about the same machine + (the installer's preflight already exists to avoid this). +3. **The holder's code runs as a supervised process on the machine and serves the seat's verbs + twice: locally to the host, on the bus to everyone else.** Adopted. + +## Decision + +**`node-container-runtime` is a seat of the mesh's own, node-scoped,** in the controller's seed under +this record. It delivers no provision; what it carries is its role's protocol: verbs its holder must +serve ([ADR 0132](0132-a-seat-carries-the-tools-its-holder-must-serve.md)) and events its holder emits +([ADR 0129](0129-a-seat-carries-the-protocol-of-its-role.md)). The runtime's module, `docker`, claims +it and is assigned to every machine. A podman module may claim it later; one machine runs one. + +**The seat's verbs cover every container on the machine:** list, inspect, logs, stats, start, stop, +restart, create and remove. A mesh-held container is marked by the host's label and says which +assignment holds it. **Creating or removing a mesh-held container is the host's alone.** Any other +caller is refused naming the assignment, because the host would undo it at its next apply. Starting, +stopping or restarting one is allowed, and the answer says the host will restore what its +declaration says. A container the mesh does not hold is the caller's to do anything with. + +**The seat's events are the runtime's own** — a container created, started, stopped, died, removed, +its health changed. They are emitted on the seat's subjects, so every holder emits the same events and +no reader depends on which runtime holds the seat. As +[ADR 0160](0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md) +decides, the subjects are issued by the controller, not composed by the module. + +**The holder's code is a supervised process, not a container** ([ADR 0150](0150-a-modules-own-code-runs-as-supervised-processes-under-one-account.md)). +A runtime cannot be run by the thing it runs. The process serves the seat's verbs on the bus to the +console, to tools and to the mesh's interface. The same verbs are served on a local socket on the +machine, which only the host may use. **The host creates, inspects and removes its containers +through that socket and nothing else.** If the holder does not answer, the host creates nothing. It +says so in its report, naming the seat. It never falls back to the command line. + +**A container needs the seat held on its machine.** An assignment that delivers a container on a +machine whose runtime seat is unheld is refused, naming the seat and its candidates +([ADR 0165](0165-container-runtime-is-what-a-machine-can-run-and-a-running-runtime-is-its-holders-health.md)). +Mounting the runtime's socket into a container is granted by the seat, not by the capability. The +socket's path is the holder's to state, because podman's is not docker's. + +**The runtime module owns the runtime's configuration.** Its settings are declared with defaults +([ADR 0164](0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md)): +the resolver containers use, the registries it trusts, log rotation, and keeping containers through a +daemon restart. The module is given the resolver's address and the mesh's registry as values; no other +module writes the runtime's file or declares its service. + +**The first machine is bootstrapped with the holder, and adopted afterwards.** The foundation bundle +already installs the runtime's package and service. It also carries the holder's process, delivered as +a binary the way the host is ([ADR 0142](0142-the-mesh-delivers-its-own-components-as-binaries.md)). +When the runtime module is assigned, it adopts what the bundle made, as the store and broker modules +adopt theirs ([ADR 0078](0078-the-store-and-broker-are-modules.md)). + +## Consequences + +- **The migration on the running mesh has a fixed order:** + 1. The resolver module and the private network stop writing the runtime's file + ([issue 190](../04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md)). + The runtime module takes the same file in the same push. Otherwise the controller refuses two + modules declaring one path. + 2. The runtime module is assigned to every machine and adopts the runtime there. Each machine's + hand-written configuration is read before the first push, because the module's defaults + replace what differs. + 3. The controller seeds the seat and enforces the container requirement. + 4. The host releases the version that uses the holder. + 5. The host's command-line path is removed in the release after every machine's holder answers. + Until then, the host reports per machine which path it used. +- **Every container the host makes depends on the holder's process.** A crash-looping holder stops + new containers on its machine. Running containers are unaffected. The host's report names the cause. +- The process form of a module's own code must serve tools on the live mesh before this ships. Only the + showcase declares it, and [issue 117](../04-ISSUES/117-a-modules-own-code-is-a-container-and-a-process/01-diagnosis.md) + found the showcase's tools declared in a form nothing runs. The runtime module is the first whose + tools cannot fall back to a container. +- A user interface subscribing to events directly does not exist. Today a reader of events is a module + that consumes them. The mesh's container view is a module, or waits for that path. +- The operator's decision to remove the third-party interface by hand needs no mechanism. No + module-retires-module rule is introduced. +- **What got harder:** the host gains a dependency it did not have, and a first machine's bundle gains + a component. The direct path was simpler and is what makes a runtime a black box to the rest of the + mesh. + +## How this is checked + +| Rule | Checked by | +|---|---| +| The seat is the mesh's own, node-scoped, with its verbs and events | A catalogue test on the default seats; registration refuses a claimant that does not serve every verb (design 33's existing check) | +| Creating or removing a mesh-held container is the host's alone | A test of the runtime module's verbs: create or remove of a container carrying the host's label, from any caller but the host's socket, is refused naming the assignment; the same verbs on an unlabelled container succeed | +| The host uses the holder and never the command line | A host test with a fake holder on the local socket: every container operation goes to it, and with the holder absent the apply creates nothing and reports the seat; after step 5, the host carries no command-line runtime code (checked by build: the package is gone) | +| A container needs the seat held | A resolution test refusing a containerised assignment on a machine with the seat unheld, naming the seat | +| Socket mounts are granted by the seat | A catalogue test: a module mounting the runtime's socket on a machine whose holder states a different path is refused | +| No other module writes the runtime's file | The existing collision check, once the private network's computed resources are inside it ([issue 190](../04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md)) | +| Live | `seats` lists `node-container-runtime` held on every machine; the node listing shows each machine's containers, strays included, from the seat's `list` verb; a container started by hand appears as an event on the bus | + +## References + +- [ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md), [ADR 0129](0129-a-seat-carries-the-protocol-of-its-role.md), [ADR 0132](0132-a-seat-carries-the-tools-its-holder-must-serve.md), [ADR 0159](0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md), [ADR 0160](0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md), [ADR 0161](0161-what-deserves-a-seat.md) +- [ADR 0150](0150-a-modules-own-code-runs-as-supervised-processes-under-one-account.md), [ADR 0142](0142-the-mesh-delivers-its-own-components-as-binaries.md), [ADR 0078](0078-the-store-and-broker-are-modules.md), [ADR 0005](0005-the-node-host.md) +- [ADR 0164](0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md), [ADR 0165](0165-container-runtime-is-what-a-machine-can-run-and-a-running-runtime-is-its-holders-health.md), [issue 190](../04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md) +- [Design 26 — the seats](../03-DESIGN/01-to-be/26-the-seats.md), [design 33 — the tools the mesh answers](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md) +- mesh-host `internal/apply/apply.go` (the runtime lookup and the command line it drives) diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index da05750..5426d63 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -265,6 +265,9 @@ python3 00-META/checks/index.py fail if stale - **0150** — [A module's own code runs as supervised processes under the module's one account](0150-a-modules-own-code-runs-as-supervised-processes-under-one-account.md) - **0152** — [The operator's surface is a module the mesh assigns: the console](0152-the-operators-surface-is-a-module-the-console.md) - **0155** — [A definition names no installation: how that is checked, and the three ways a value that did gets out](0155-a-definition-names-no-installation-and-how-that-is-checked.md) +- **0164** — [A setting is declared with its default, its meaning and what changing it costs](0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md) *(proposed)* +- **0165** — [`container-runtime` is what a machine can run; that a runtime is running is its holder's health](0165-container-runtime-is-what-a-machine-can-run-and-a-running-runtime-is-its-holders-health.md) *(proposed)* +- **0166** — [The container runtime is a node seat, and the host creates containers through its holder](0166-the-container-runtime-is-a-node-seat-and-the-host-creates-containers-through-its-holder.md) *(proposed)* ### How it is built diff --git a/04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md b/04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md new file mode 100644 index 0000000..aafa3c4 --- /dev/null +++ b/04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md @@ -0,0 +1,76 @@ +--- +status: located +opened: 2026-10-01 +located-in: [mesh-catalog modules/dnsmasq, mesh-controller internal/overlay/generator.go, mesh-controller internal/catalogue/resolve.go (checkResources)] +fixed-by: +amended-design: +--- + +# 190 — The container runtime's configuration is written by modules that are not the runtime's + +## What was observed + +The runtime's configuration file and its service are declared by two parties, neither of which is +the runtime: + +- **The resolver module** writes the runtime's `dns` key (the machine's private address) and + `live-restore` into the runtime's file, written into rather than over + ([ADR 0102](../../02-DECISIONS/0102-the-mesh-writes-into-a-shared-file-never-over-it.md)). It also + declares the runtime's service, reloaded when that file changes. It was added on 2026-09-30 to + fix [issue 110](../110-a-container-on-the-runtimes-own-network-cannot-reach-the-resolver/00-report.md), + where containers silently resolved through a public resolver. +- **The private network** writes the runtime's `insecure-registries` into the same file, and declares + the same service reloaded on it, as [ADR 0082](../../02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md) + and ADR 0102 decided. The controller generates both resources per machine. + +On the three machines that run the resolver module, both declare one path and one unit. Nothing refuses +it. The collision check compares the resources of catalogue modules. The private network is computed, +so its resources are produced when a machine's declaration is composed, and the check never sees them. + +The machine without the resolver module shows the other half. Its runtime still has the predecessor's +resolver and `live-restore` off, because the only module that sets them is a DNS server. A machine +gets a correct container runtime only as a side effect of being given a resolver. + +## Why this is here + +The operator ruled it a defect, not a design: **a module does not write another software's +configuration.** The need behind each write is real. Containers must resolve the mesh's names +([ADR 0148](../../02-DECISIONS/0148-the-meshs-names-are-resolved-not-copied-into-containers.md) step 2). +A daemon restart must not stop every container. Every machine on the network must trust the mesh's +registry. But each of these is a fact the runtime must be *given*, and the module that gives it is the +runtime's own. With three writers, nobody can say what the file should contain. Two of the facts are +reloaded when one of them needs a restart (issue 110's first fault). And the moment a module for the +runtime exists, it is refused on every machine with the resolver, or, through the private network's +path, accepted without anyone noticing a collision. + +## What resolves it + +[ADR 0166](../../02-DECISIONS/0166-the-container-runtime-is-a-node-seat-and-the-host-creates-containers-through-its-holder.md) +gives the runtime a module that holds its seat and owns its file and service. +[ADR 0164](../../02-DECISIONS/0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md) +gives that module declared settings with defaults. The fix, once both are accepted: + +1. The resolver module drops its runtime file and runtime service. It knows nothing of the runtime. +2. The private network stops generating either resource. ADR 0082's decision stands — being on the + network is what grants the trust, and no module author is involved — and only *who writes it* + moves. The mesh gives the registry to the runtime module as a value. ADR 0082 and ADR 0102 each + get a dated note saying where their mechanism now lives. +3. The runtime module writes `dns`, `live-restore` and `insecure-registries`, each a declared + setting with its cost: `dns` costs a restart, which `live-restore` makes harmless. +4. Steps 1–3 land in one push. A runtime module declaring the file beside a resolver module still + declaring it is refused. +5. The collision check sees a computed module's resources as well, so a second writer cannot come + back through generated code. + +## Open questions + +- **How the resolver's address reaches the runtime.** Either the resolver seat (`node-dns-resolver`) + delivers an address its holder serves, or the runtime module reads a machine fact and the seat + being held is only a precondition. The first tracks a resolver moving off the private address. The + second needs nothing new. +- **What `dns` defaults to on a machine with no resolver seat held.** Nothing, leaving the runtime's + own behaviour, is the honest default. A public resolver hides exactly the failure issue 110 took a + day to find. +- **The adopted machine's predecessor values.** The runtime module adopting a file with a + hand-written `dns` and `live-restore: false` replaces both. That is intended, and is the one + restart the operator must make on that machine. -- 2.54.0 From 27c1db8a863edcf76456fbe140a0e47e98ac2be5 Mon Sep 17 00:00:00 2001 From: jochens Date: Fri, 2 Oct 2026 00:48:06 +0200 Subject: [PATCH 2/3] Review of 0164-0166 and 190: the mesh's own setting words stay settable; changing runtime verbs are not the console's wildcard; migration steps 1-2 are one push; dnsmasq's dns key dates from 09-23 --- ...-its-meaning-and-what-changing-it-costs.md | 8 ++++-- ...t-creates-containers-through-its-holder.md | 26 +++++++++++++------ .../00-report.md | 6 +++-- 3 files changed, 28 insertions(+), 12 deletions(-) diff --git a/02-DECISIONS/0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md b/02-DECISIONS/0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md index 44a096e..6673420 100644 --- a/02-DECISIONS/0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md +++ b/02-DECISIONS/0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md @@ -81,14 +81,18 @@ it; a mesh-wide setting reaches every assignment of the module; a node's reaches change names each assignment whose effective value moves. **A declared setting is the only kind accepted.** Setting a key the module does not declare is refused -when it is set, naming the declared keys, rather than reported when the machine is planned. A module +when it is set, naming the declared keys, rather than reported when the machine is planned. The mesh's +own words — where a port, a directory or an operator's data is placed, how far an endpoint reaches — +are the mesh's to validate as they are today, and no module declares them. A module that declares no settings keeps today's behaviour until it does; a catalogue test lists those modules, and the list shrinks to empty before the implicit form is removed — design 27's rule for every retired mechanism. **A setting says what it costs: nothing, a reload, or a restart.** When a file changes, the host applies the strongest cost among the settings whose values moved in it, so a key the software reads -only at start can no longer be written and never read. A service's `reload-on` and `restart-on` keep +only at start can no longer be written and never read. A setting that reaches a container's environment +costs that container being recreated, which the host already does when a container's specification +changes; it needs no declaration. A service's `reload-on` and `restart-on` keep naming the files that are not settings — a generated roster, a credential. **The container runtime is the first module to declare its settings** and the model for the rest: diff --git a/02-DECISIONS/0166-the-container-runtime-is-a-node-seat-and-the-host-creates-containers-through-its-holder.md b/02-DECISIONS/0166-the-container-runtime-is-a-node-seat-and-the-host-creates-containers-through-its-holder.md index 396c27b..891962b 100644 --- a/02-DECISIONS/0166-the-container-runtime-is-a-node-seat-and-the-host-creates-containers-through-its-holder.md +++ b/02-DECISIONS/0166-the-container-runtime-is-a-node-seat-and-the-host-creates-containers-through-its-holder.md @@ -66,7 +66,12 @@ it and is assigned to every machine. A podman module may claim it later; one mac **The seat's verbs cover every container on the machine:** list, inspect, logs, stats, start, stop, restart, create and remove. A mesh-held container is marked by the host's label and says which -assignment holds it. **Creating or removing a mesh-held container is the host's alone.** Any other +assignment holds it. **A container the runtime runs can be root on the machine** — privileged, a host +path mounted, the host's network or process namespace, the runtime's own socket — so a caller other +than the host may not create one that is any of these; only a declaration the mesh composed may ask +for them. And the verbs that change anything are granted by name, never by a wildcard: a grant of +every tool (the console's today) reaches the reading verbs only. Issue 193 is what a verb that trusts +its caller costs. **Creating or removing a mesh-held container is the host's alone.** Any other caller is refused naming the assignment, because the host would undo it at its next apply. Starting, stopping or restarting one is allowed, and the answer says the host will restore what its declaration says. A container the mesh does not hold is the caller's to do anything with. @@ -105,13 +110,13 @@ adopt theirs ([ADR 0078](0078-the-store-and-broker-are-modules.md)). ## Consequences - **The migration on the running mesh has a fixed order:** - 1. The resolver module and the private network stop writing the runtime's file - ([issue 190](../04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md)). - The runtime module takes the same file in the same push. Otherwise the controller refuses two - modules declaring one path. - 2. The runtime module is assigned to every machine and adopts the runtime there. Each machine's - hand-written configuration is read before the first push, because the module's defaults - replace what differs. + 1. Each machine's hand-written configuration is read, because the module's defaults replace what + differs. + 2. In one push per machine: the resolver module and the private network stop writing the + runtime's file ([issue 190](../04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md)), + and the runtime module is assigned and adopts the runtime, its file and its service. Split in + two, either the controller refuses two modules declaring one path, or a machine is left with + nothing setting `dns` and `live-restore`. 3. The controller seeds the seat and enforces the container requirement. 4. The host releases the version that uses the holder. 5. The host's command-line path is removed in the release after every machine's holder answers. @@ -124,6 +129,10 @@ adopt theirs ([ADR 0078](0078-the-store-and-broker-are-modules.md)). tools cannot fall back to a container. - A user interface subscribing to events directly does not exist. Today a reader of events is a module that consumes them. The mesh's container view is a module, or waits for that path. +- [ADR 0005](0005-the-node-host.md) ("a container runtime is detected, not chosen") and + [ADR 0006](0006-the-substrate-and-the-control-plane.md)'s matching line describe the mechanism this replaces: on + acceptance, each gets a dated note saying the runtime is now a seat's holder, as the decision + records' rule for a moved mechanism requires. Design 05 and design 26 are amended after acceptance. - The operator's decision to remove the third-party interface by hand needs no mechanism. No module-retires-module rule is introduced. - **What got harder:** the host gains a dependency it did not have, and a first machine's bundle gains @@ -136,6 +145,7 @@ adopt theirs ([ADR 0078](0078-the-store-and-broker-are-modules.md)). |---|---| | The seat is the mesh's own, node-scoped, with its verbs and events | A catalogue test on the default seats; registration refuses a claimant that does not serve every verb (design 33's existing check) | | Creating or removing a mesh-held container is the host's alone | A test of the runtime module's verbs: create or remove of a container carrying the host's label, from any caller but the host's socket, is refused naming the assignment; the same verbs on an unlabelled container succeed | +| No caller but the host creates a container that is root on the machine | A test of `create` from the bus: privileged, a host path, the host's namespaces and the runtime's socket are each refused; the same request on the host's socket is accepted. A broker test: a grant of every tool does not reach a changing verb | | The host uses the holder and never the command line | A host test with a fake holder on the local socket: every container operation goes to it, and with the holder absent the apply creates nothing and reports the seat; after step 5, the host carries no command-line runtime code (checked by build: the package is gone) | | A container needs the seat held | A resolution test refusing a containerised assignment on a machine with the seat unheld, naming the seat | | Socket mounts are granted by the seat | A catalogue test: a module mounting the runtime's socket on a machine whose holder states a different path is refused | diff --git a/04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md b/04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md index aafa3c4..aef74d9 100644 --- a/04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md +++ b/04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md @@ -16,8 +16,10 @@ the runtime: - **The resolver module** writes the runtime's `dns` key (the machine's private address) and `live-restore` into the runtime's file, written into rather than over ([ADR 0102](../../02-DECISIONS/0102-the-mesh-writes-into-a-shared-file-never-over-it.md)). It also - declares the runtime's service, reloaded when that file changes. It was added on 2026-09-30 to - fix [issue 110](../110-a-container-on-the-runtimes-own-network-cannot-reach-the-resolver/00-report.md), + declares the runtime's service, reloaded when that file changes. The `dns` key has been written + since the resolver module was converted from its predecessor on 2026-09-23; `live-restore` and the + service were added on 2026-09-30 while fixing + [issue 110](../110-a-container-on-the-runtimes-own-network-cannot-reach-the-resolver/00-report.md), where containers silently resolved through a public resolver. - **The private network** writes the runtime's `insecure-registries` into the same file, and declares the same service reloaded on it, as [ADR 0082](../../02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md) -- 2.54.0 From 17ca9a262b1b2bb0b17e74a2b1a12cb02659c326 Mon Sep 17 00:00:00 2001 From: jochens Date: Fri, 2 Oct 2026 12:23:28 +0200 Subject: [PATCH 3/3] 0164: a setting names the file it lands in (issue 198's leak between one module's files); 190 notes the fourth machine now has the resolver --- ...-default-its-meaning-and-what-changing-it-costs.md | 11 +++++++++++ .../00-report.md | 7 +++++++ 2 files changed, 18 insertions(+) diff --git a/02-DECISIONS/0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md b/02-DECISIONS/0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md index 6673420..f1caeda 100644 --- a/02-DECISIONS/0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md +++ b/02-DECISIONS/0164-a-setting-is-declared-with-its-default-its-meaning-and-what-changing-it-costs.md @@ -40,6 +40,12 @@ What was built is narrower than what was decided, measured in the controller on layer sets it — right for a mail domain, where a default is the very literal 0155 removes, and wrong for a tunable like the resolver's upstreams, which the resolver module therefore carries as literals in its file. +- **A setting reaches every mergeable file its module owns.** The layers are one flat map per module, + laid over each such file. Adding a setting to the resolver module for its own configuration put the + key into the container runtime's file as well — the resolver writes into that file too — and the + runtime refuses keys it does not know. The plan showed it before any push; the runtime's file was + then made to take no settings at all ([issue 198](../04-ISSUES/198-the-lans-dns-server-ran-outside-the-mesh-and-its-filter-closed-it/00-report.md)). Issue 173 stopped settings leaking into + contributions and served facts; between one module's own files the leak remains. - **What a change costs is said per file, not per key.** A service names the files it is reloaded or restarted on. The runtime re-reads its trusted registries on a reload and its `dns` key only when it starts; the resolver module declared a reload, so on two machines the key was written, reloaded, @@ -80,6 +86,10 @@ A new default ships with the module's next version and reaches every assignment it; a mesh-wide setting reaches every assignment of the module; a node's reaches one. The plan of a change names each assignment whose effective value moves. +**A declared setting says where it lands.** Each names the file or files of its module that read it, +and reaches no other: a module that owns two mergeable files no longer has one flat map laid over both. +A file that names no setting takes none. + **A declared setting is the only kind accepted.** Setting a key the module does not declare is refused when it is set, naming the declared keys, rather than reported when the machine is planned. The mesh's own words — where a port, a directory or an operator's data is placed, how far an endpoint reaches — @@ -121,6 +131,7 @@ its trusted registries are what the mesh tells it. |---|---| | Every setting a module takes is declared | A parser test refusing a setting declaration without a type or meaning; a catalogue test listing modules with mergeable files or `${setting:}` and no declarations, which must be empty before the implicit form is removed | | A tunable resolves to its default; an operator value without one is refused | Resolution tests: an unset tunable in a JSON file and in a text file both take the default; an unset setting with no default is refused naming it (0155's existing test) | +| A setting reaches only the files it names | A resolution test: a module with two mergeable files and a setting declared for one; the other file's content is unchanged by it (the case of issue 198) | | An undeclared key is refused when set | A controller test: `settings set` with an undeclared key fails naming the declared keys, and nothing is stored | | Every effective value names its source | A test listing a module's configuration on a node with one key from each of default, mesh and node | | A change's reach is shown before it moves | A plan test: changing a mesh-wide setting names every assignment whose effective value moves and no other | diff --git a/04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md b/04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md index aef74d9..c8769e4 100644 --- a/04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md +++ b/04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md @@ -33,6 +33,13 @@ The machine without the resolver module shows the other half. Its runtime still resolver and `live-restore` off, because the only module that sets them is a DNS server. A machine gets a correct container runtime only as a side effect of being given a resolver. +> **Later the same day, 2026-10-02.** The resolver module and its sibling for the resolver file were +> assigned to the fourth machine ([issue 198](../198-the-lans-dns-server-ran-outside-the-mesh-and-its-filter-closed-it/00-report.md)), +> so all four now have the resolver writing into the runtime's file, and the predecessor's +> `live-restore: false` there is gone. The same work made the runtime's file, as the resolver declares +> it, take no settings: a setting meant for the resolver's own configuration had reached it. The +> collision and the ownership question above are unchanged. + ## Why this is here The operator ruled it a defect, not a design: **a module does not write another software's -- 2.54.0