From 25d599094f75cd430d7ba4d52d9d22c7d677bfaa Mon Sep 17 00:00:00 2001 From: jochen Date: Sat, 26 Sep 2026 20:13:37 +0200 Subject: [PATCH 01/27] 112 is fixed: a carried peer is nameable, and the resolver answers for all four machines --- .../00-report.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/04-ISSUES/112-adopting-a-tunnel-gives-the-mesh-addresses-without-names/00-report.md b/04-ISSUES/112-adopting-a-tunnel-gives-the-mesh-addresses-without-names/00-report.md index 246fbf4..0a9a223 100644 --- a/04-ISSUES/112-adopting-a-tunnel-gives-the-mesh-addresses-without-names/00-report.md +++ b/04-ISSUES/112-adopting-a-tunnel-gives-the-mesh-addresses-without-names/00-report.md @@ -1,8 +1,8 @@ --- -status: located +status: fixed opened: 2026-09-24 located-in: [mesh-controller internal/inventory, mesh-controller internal/catalogue, mesh-controller cmd/mesh-controller, mesh-catalog modules/dnsmasq] -fixed-by: +fixed-by: [mesh-controller#73 overlay-name + namesInTheMesh, mesh-catalog#106 dnsmasq daemon.json merge] amended-design: --- From bb334e138b0b1f73db4cab290a34029a2251fa17 Mon Sep 17 00:00:00 2001 From: jochen Date: Sat, 26 Sep 2026 22:24:58 +0200 Subject: [PATCH 02/27] issue 127: a declaration that shrinks to empty is skipped, so the node keeps what it should drop --- .../00-report.md | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 04-ISSUES/127-a-declaration-that-shrinks-to-empty-is-skipped-not-sent/00-report.md diff --git a/04-ISSUES/127-a-declaration-that-shrinks-to-empty-is-skipped-not-sent/00-report.md b/04-ISSUES/127-a-declaration-that-shrinks-to-empty-is-skipped-not-sent/00-report.md new file mode 100644 index 0000000..9a57cb6 --- /dev/null +++ b/04-ISSUES/127-a-declaration-that-shrinks-to-empty-is-skipped-not-sent/00-report.md @@ -0,0 +1,39 @@ +--- +status: located +opened: 2026-09-27 +located-in: [mesh-controller cmd/mesh-controller/push.go] +--- + +# A declaration that shrinks to empty is skipped, so the node keeps what it should drop + +## What was observed + +Fixing the broker-opening leak (the foundation port scoped to the broker's host) made ace's +declaration compose to **zero resources** — ace is adopted with nothing assigned, and the +stray opening was its only resource. `push ace` then printed `ace is assigned nothing — +skipped` and sent nothing. ace goes on holding `adoption.opening-tcp-5671-incoming` in its +ufw, because it was never told the resource is gone. + +`composeEach` (push.go) skips any node whose composed declaration has no resources. That is +right for a node that never had anything. It is wrong for a node that **had** resources and +now composes to none: the empty declaration is the correction, and skipping it leaves the last +non-empty one in force forever. + +## Why it matters + +Any adopted node whose openings (or other baseline resources) are all removed keeps the stale +ones until something else pushes a non-empty declaration to it. Converge is unaffected — it +composes the full ruleset fresh — so this is an incremental-push gap, not a firewall-safety +one. But "the mesh cannot tell a node to drop its last resource" is a real hole in reconcile. + +## The fix, roughly + +Send the empty declaration when the node's last-sent declaration was non-empty — i.e. skip +only when empty-and-was-already-empty. Requires push to know (or the host to be told) that the +node held something. Simplest: always send to a placed, enrolled node; let an empty declaration +mean "own nothing", which the host already applies correctly when it receives one. + +## Workaround used + +On ace, one command drops it permanently (the corrected controller never re-composes it): +`sudo ufw delete allow 5671`. At ace's converge it would clear on its own. From 64bbdc30c187a25373e0978fce243f8b4a3971ea Mon Sep 17 00:00:00 2001 From: jochen Date: Sat, 26 Sep 2026 23:53:13 +0200 Subject: [PATCH 03/27] to-be 29: a node has operator accounts, and the mesh owns what lives under a home MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The mesh models machines but not the people on them — a node record holds no username, and no module places anything under a home. So who you are on each node (jochens/ace/jochen) is unknown to the mesh, and nothing owns ~/.ssh, dotfiles or ~/.config. HAL knew it; the nox mesh dropped it. Proposes the account as a node fact and a home-scoped resource class (the ~/ mirror of ADR 0112's /var/lib placement), with the login key staying the operator's (ADR 0051). Not urgent — HAL's generators still run — load-bearing at node-by-node retirement. Found generating ~/.ssh/config from HAL's registry, which nox has no equivalent for. --- .../29-a-node-has-operator-accounts.md | 84 +++++++++++++++++++ 03-DESIGN/01-to-be/README.md | 1 + 2 files changed, 85 insertions(+) create mode 100644 03-DESIGN/01-to-be/29-a-node-has-operator-accounts.md diff --git a/03-DESIGN/01-to-be/29-a-node-has-operator-accounts.md b/03-DESIGN/01-to-be/29-a-node-has-operator-accounts.md new file mode 100644 index 0000000..f038a5d --- /dev/null +++ b/03-DESIGN/01-to-be/29-a-node-has-operator-accounts.md @@ -0,0 +1,84 @@ +--- +layer: to-be +status: proposed +code: [] +updated: 2026-09-27 +decisions: + - 02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md + - 02-DECISIONS/0051-shared-data-is-the-operators.md +--- + +# 29 — A node has operator accounts, and the mesh owns what lives under a home + +**The mesh models machines but not the people on them.** A node record holds its name, its +address, its mode — and nothing about *who a person is* on it: `jochens` on novox, `ace` on ace, +`jochen` on shanks and g14. That username is not incidental. It decides who a file under `~` is +owned by, who a user service runs as, and — the case that surfaced this — which account `ssh +` logs in as. The predecessor knew it (its per-node `user:`, and the modules that wrote a +person's `~/.ssh/config`, `~/.zshrc`, `~/.config`); the mesh, taking those over, kept the machine +facts and dropped the human one. + +Two things are missing, and they are one idea: + +## 1. The account is a node fact + +A node has one or more **operator accounts**: the human logins on it. At minimum a name; the +mesh already knows the node and its address, so `@` is then a complete answer to +"who am I, where." It is the mesh's to hold because everything below is derived from it, and +because it is exactly the fact that was silently lost — `ssh ace` failed to `ace` because nothing +in the mesh said ace's account is `ace`. + +It is **not** a credential. The account names a login; the key that authorises it is the +operator's, placed as a secret or an operator-owned file, never minted by the mesh (ADR 0051). + +## 2. A resource may live under a home, owned by its account + +[ADR 0112](../../02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md) placed a +module's *system* data — `/`, owned by the module. It has no analog for the other +half of the filesystem: the things that belong under a person's home and are owned by that +person. `~/.ssh/config`, `~/.ssh/config.d/mesh`, `~/.zshrc`, `~/.config/hal` — every one of these +is a resource the mesh should be able to place and own, resolved against **the account's home** +rather than a system root, and chowned to **the account** rather than to root or a module uid. + +This is the same move as `${dir:…}`, one level over: a resource says `home: ` (or names +an account requirement), and the mesh resolves the home directory and the owning uid on the node +that account lives on. A module that writes operator config — the eventual replacements for +`hal/terminal`, `hal/claude-code`, `hal/secrets` — declares its files this way and names no +`/home/...` path, exactly as a system module now names no `/var/lib` path. + +## Why now, and why not yet + +**Why it matters:** when HAL retires, the generators that keep `~/.ssh/config`, shell config and +the operator's `~/.config/hal` current retire with it. Without this, adding a node stops adding +its ssh alias, and a fresh machine has no operator dotfiles at all — the mesh would run every +service and leave the human unable to work on the box. The account is also load-bearing for +correctness already: `ssh ` (issue 122's cousin), user-scoped systemd units, and any file a +person rather than a daemon must own. + +**Why not build it reflexively:** it is a real addition to the node model and the resource model, +and it must be gotten right, not smuggled in beside a firewall fix. Open questions to settle +first: + +- **One account or several per node?** A workstation has one human; a shared box might have more. + The model should allow more than one without forcing the common case to name it. +- **Where the login key lives.** An operator-owned file (ADR 0051) or an accepted secret — never + minted. The account fact and the key that authorises it are separate, and only the first is the + mesh's to generate. +- **The boundary with `sshd`.** The `sshd` module (server side) already exists. This is the + *client* and *identity* side: the account a node offers, and the home-scoped files an operator + needs. They meet at the account but are not the same module. +- **Multi-operator.** Today there is one human. The model should not assume it, but the first + cut may serve one and leave the shape open. + +**Not urgent, not blocking.** ssh and dotfiles work today because HAL's generators still run as +the substrate. This becomes load-bearing in the node-by-node retirement phase, not before — which +is the right time to build it, once the account model is decided here. + +## References + +- The gap was found generating `~/.ssh/config` from the *HAL* registry (`hal/terminal`'s + postConfigure hook), which the nox mesh has no equivalent for. +- [ADR 0112](../../02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md) — the + system-path placement this mirrors for home paths. +- [ADR 0051](../../02-DECISIONS/0051-shared-data-is-the-operators.md) — why the login key stays + the operator's, never minted. diff --git a/03-DESIGN/01-to-be/README.md b/03-DESIGN/01-to-be/README.md index 584c25b..bae51da 100644 --- a/03-DESIGN/01-to-be/README.md +++ b/03-DESIGN/01-to-be/README.md @@ -38,6 +38,7 @@ document is written and this one's status becomes `implemented`. | [`26-the-seats.md`](26-the-seats.md) | **Proposed.** What a mesh can have one of, who fills each, and a seat's holder answering for the provision it delivers — including the `git` seat a build's source can live on | [ADR 0110](../../02-DECISIONS/0110-a-seat-is-a-module-assignment-from-a-closed-set.md), [ADR 0111](../../02-DECISIONS/0111-a-build-source-is-on-the-git-seat-or-external.md), [ADR 0109](../../02-DECISIONS/0109-a-package-registry-seat-is-one-per-ecosystem.md) | | [`27-a-module-requires-the-mesh-resolves.md`](27-a-module-requires-the-mesh-resolves.md) | **Proposed.** One concept for everything a module needs: a requirement with a contract, answered by one of four kinds of provider, resolved at assignment or refused. Retires settings, placeholders, facts and paths in definitions | [ADR 0112](../../02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md), [ADR 0113](../../02-DECISIONS/0113-the-vault-makes-every-secret.md), [ADR 0114](../../02-DECISIONS/0114-a-shared-credential-rotates-over-two-credentials.md), [ADR 0110](../../02-DECISIONS/0110-a-seat-is-a-module-assignment-from-a-closed-set.md) | | [`28-building-the-bus.md`](28-building-the-bus.md) | **Proposed.** The five steps of the bus work in the order their dependencies allow, each ending at a bed — with the surface measured, so no step's size is a guess | [ADR 0116](../../02-DECISIONS/0116-the-bus-is-built-in-five-steps.md), [ADR 0106](../../02-DECISIONS/0106-the-bus-is-nats.md), [ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md) | +| [`29-a-node-has-operator-accounts.md`](29-a-node-has-operator-accounts.md) | **Proposed.** The mesh models machines but not the humans on them: a node gains operator accounts, and a resource may live under a home owned by its account — what would own ~/.ssh, dotfiles and ~/.config when HAL retires | [ADR 0112](../../02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md), [ADR 0051](../../02-DECISIONS/0051-shared-data-is-the-operators.md) | ## Not yet written From 504adef22158e2e8d7e1341a299b693a59be502a Mon Sep 17 00:00:00 2001 From: jochen Date: Sat, 26 Sep 2026 23:37:48 +0200 Subject: [PATCH 04/27] =?UTF-8?q?ADR=200117:=20a=20machine's=20uplink=20is?= =?UTF-8?q?=20a=20seat=20=E2=80=94=20the=20mesh=20configures=20the=20manag?= =?UTF-8?q?er,=20never=20the=20link?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../0117-a-machines-uplink-is-a-seat.md | 108 ++++++++++++++++++ 02-DECISIONS/README.md | 1 + 2 files changed, 109 insertions(+) create mode 100644 02-DECISIONS/0117-a-machines-uplink-is-a-seat.md diff --git a/02-DECISIONS/0117-a-machines-uplink-is-a-seat.md b/02-DECISIONS/0117-a-machines-uplink-is-a-seat.md new file mode 100644 index 0000000..0103e8c --- /dev/null +++ b/02-DECISIONS/0117-a-machines-uplink-is-a-seat.md @@ -0,0 +1,108 @@ +--- +topic: what runs on it +status: proposed +date: 2026-09-26 +deciders: jochen +reconstructed: false +extends: 0110-a-seat-is-a-module-assignment-from-a-closed-set.md +--- + +# 117. A machine's uplink is a seat: the mesh configures the manager, never the link + +## Context + +The mesh installs on top of a machine's own networking. The private network's generator says +so in as many words: a machine has an address and a route to the broker *before* the mesh +exists, the broker's address travels in the enrolment token rather than being resolved, and the +private network is something the mesh installs on top, like anything else. Nothing in the mesh +says who manages that uplink, or what the mesh needs from whoever does. + +Adopting the first workstations showed that the mesh does need something from it, and gets it +by accident: + +- **The resolver the mesh owns depends on a file the mesh does not.** `resolv-conf` writes + `/etc/resolv.conf` and names the mesh's resolver. On a machine running NetworkManager, the + manager rewrites that file on every connectivity change unless it is told `dns=none`; on one + running dhcpcd, every lease renewal rewrites it unless it is told `nohook resolv.conf`. On + the adopted machines both settings exist only because the predecessor wrote them. No module + declares them. Remove the predecessor's file and the mesh's resolver is silently replaced the + next time a laptop changes network, while every surface of the mesh still reads green. +- **`resolv-conf` cannot declare them itself.** Which setting is needed depends on which + manager runs, and a `service` resource for a manager that is not installed fails the + declaration. A resolver module that knew about network managers would be the wrong module + knowing the wrong thing. +- **The private network's interface is exposed to the manager.** A manager that considers + every interface its own may try to configure `mesh0`, or tear it down on a profile change. + Nothing tells it not to. +- **Two managers on one machine go unnoticed.** Of the four machines adopted so far, one runs + systemd-networkd, two run NetworkManager, and one runs NetworkManager *and* dhcpcd at once: + two programs that each believe they own the machine's addresses and its resolver file. + Nothing detected it, because nothing in the mesh knows the role exists. + +The machines differ in a way that matters: servers are wired and never move, while +workstations join wireless networks, captive portals and phone hotspots wherever they are. + +## Considered Options + +**1. The mesh manages the uplink: links, addressing, wireless networks and their +credentials.** Rejected. The mesh reaches a machine only over that link. A declaration that +gets it wrong — a mistyped network, a stale credential, a manager that fails to start — takes +the machine off the network, and with it the only channel a fix could arrive on. That is the +one failure the sshd module's `listens` rule forbids the firewall to arrange; a mesh that owned +the link could arrange it with any push. And a wireless network is joined at the machine, by +the person using it, in the moment. A declaration composed elsewhere cannot answer a captive +portal. + +**2. Leave the uplink unmanaged; accept the implicit dependency.** Rejected. It keeps the +resolver working only for as long as a predecessor's file survives, and it leaves two managers +on one machine undetectable. + +**3. The uplink is a seat. The module holding it configures the manager's relationship to the +mesh, and never the link.** Chosen. + +## Decision + +**`the-uplink` is a node-scoped seat** in the closed set ([ADR 0110](0110-a-seat-is-a-module-assignment-from-a-closed-set.md)). +It delivers no provision. It is held by the module for the program that manages the machine's +own network, one per manager: `networkmanager`, `systemd-networkd`, and `dhcpcd` for a machine +with nothing more. Assigning a second is refused, naming the first. + +**What a holder declares** — only what keeps the manager and the mesh from contradicting each +other: + +- the manager's package, and its service running and enabled at boot; +- the manager's own configuration that leaves the resolver file to the mesh (`dns=none` for + NetworkManager, `nohook resolv.conf` for dhcpcd, and for systemd-networkd the equivalent + that stops it handing DNS to a resolver the machine does not use); +- the manager's own configuration that leaves the private network's interface alone + (NetworkManager's `unmanaged-devices` naming `mesh0`; for systemd-networkd, no network file + of the module's matches it); +- each as a drop-in beside the manager's main file where the manager reads one, and written + *into* a shared file otherwise ([ADR 0102](0102-the-mesh-writes-into-a-shared-file-never-over-it.md)); +- the service **reloaded** when a drop-in changes, never restarted — a restart drops the link, + and the link is the mesh's own channel to the machine. Whether each manager applies these + settings on a reload is measured in the lab before its module is written, not assumed. + +**What a holder never declares:** a link, an address, a route, a connection profile, a +wireless network or its credentials. Those are the operator's, in the sense of +[ADR 0051](0051-shared-data-is-the-operators.md): the mesh does not create, change or delete +them, and the module's `access`, if it needs one, is read-only. + +## Consequences + +- `resolv-conf` stays generic. The condition it could not express — "only if NetworkManager + runs" — is expressed by assigning the module for the manager that does. +- The dependency on the predecessor's `dns=none` file becomes a declared resource. On an + adopted node the holder's drop-in arrives beside the predecessor's; both say the same thing, + and the predecessor's is retired by hand after the take, like any other file the mesh + replaced under another name. +- A machine running two managers is found at assignment: the second holder is refused, and the + operator decides which manager the machine keeps before either module is taken. +- Workstations keep joining networks the way they always have. The host already cooperates + with the manager — its dispatcher hook wakes it on every connectivity change — and nothing + here changes that. +- The seat table gains one entry: `the-uplink`, node scope, delivering nothing, decided here. +- **Not decided here:** whether the mesh should ever *offer* known networks to a machine — a + sealed, add-only list the operator curates once for all workstations. That is a different + question (the mesh holding credentials for links it must never be able to break) and gets its + own record if it is wanted. diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 67eb0fe..1f26d55 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -199,6 +199,7 @@ python3 00-META/checks/index.py fail if stale - **0113** — [The vault makes every shared secret, a provider makes resources and data, and the mesh carries both](0113-the-vault-makes-every-secret.md) *(proposed)* - **0114** — [A credential two parties hold rotates over two credentials; one a single party holds rotates in place, staged; and retiring a credential never removes what it reached](0114-a-shared-credential-rotates-over-two-credentials.md) *(proposed)* - **0115** — [One assignment of a module per node: the module's name is the assignment's identity](0115-one-assignment-of-a-module-per-node.md) *(proposed)* +- **0117** — [A machine's uplink is a seat: the mesh configures the manager, never the link](0117-a-machines-uplink-is-a-seat.md) *(proposed)* ### How it is built From 0e066473b37eb1523e2f60e0b0705553f7f1be8c Mon Sep 17 00:00:00 2001 From: jochen Date: Sat, 26 Sep 2026 23:44:50 +0200 Subject: [PATCH 05/27] ADR 0117 accepted; a manager that cannot reload takes the setting at its next start (dhcpcd, measured) --- 02-DECISIONS/0117-a-machines-uplink-is-a-seat.md | 10 +++++++--- 02-DECISIONS/README.md | 2 +- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/02-DECISIONS/0117-a-machines-uplink-is-a-seat.md b/02-DECISIONS/0117-a-machines-uplink-is-a-seat.md index 0103e8c..3cbbbbb 100644 --- a/02-DECISIONS/0117-a-machines-uplink-is-a-seat.md +++ b/02-DECISIONS/0117-a-machines-uplink-is-a-seat.md @@ -1,6 +1,6 @@ --- topic: what runs on it -status: proposed +status: accepted date: 2026-09-26 deciders: jochen reconstructed: false @@ -80,8 +80,12 @@ other: - each as a drop-in beside the manager's main file where the manager reads one, and written *into* a shared file otherwise ([ADR 0102](0102-the-mesh-writes-into-a-shared-file-never-over-it.md)); - the service **reloaded** when a drop-in changes, never restarted — a restart drops the link, - and the link is the mesh's own channel to the machine. Whether each manager applies these - settings on a reload is measured in the lab before its module is written, not assumed. + and the link is the mesh's own channel to the machine. **A manager that cannot reload is not + restarted instead:** its setting takes effect at the manager's next start. Measured on the + adopted machines: NetworkManager (1.58) and systemd-networkd (systemd 261) both report + `CanReload=yes`; dhcpcd (10.3) reports `CanReload=no`, so its module declares no trigger at + all. Whether each setting is actually *applied* by a reload is confirmed on a machine before + the module is taken there, not assumed. **What a holder never declares:** a link, an address, a route, a connection profile, a wireless network or its credentials. Those are the operator's, in the sense of diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 1f26d55..de2b056 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -199,7 +199,7 @@ python3 00-META/checks/index.py fail if stale - **0113** — [The vault makes every shared secret, a provider makes resources and data, and the mesh carries both](0113-the-vault-makes-every-secret.md) *(proposed)* - **0114** — [A credential two parties hold rotates over two credentials; one a single party holds rotates in place, staged; and retiring a credential never removes what it reached](0114-a-shared-credential-rotates-over-two-credentials.md) *(proposed)* - **0115** — [One assignment of a module per node: the module's name is the assignment's identity](0115-one-assignment-of-a-module-per-node.md) *(proposed)* -- **0117** — [A machine's uplink is a seat: the mesh configures the manager, never the link](0117-a-machines-uplink-is-a-seat.md) *(proposed)* +- **0117** — [A machine's uplink is a seat: the mesh configures the manager, never the link](0117-a-machines-uplink-is-a-seat.md) ### How it is built From ee31f9f7618d82a90ae638b58dc106a640fe5f00 Mon Sep 17 00:00:00 2001 From: jochen Date: Sat, 26 Sep 2026 23:38:08 +0200 Subject: [PATCH 06/27] issue 128: the machine's hosts file is written whole, and on a workstation it is shared --- .../00-report.md | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 04-ISSUES/128-the-hosts-file-is-written-whole/00-report.md diff --git a/04-ISSUES/128-the-hosts-file-is-written-whole/00-report.md b/04-ISSUES/128-the-hosts-file-is-written-whole/00-report.md new file mode 100644 index 0000000..8e5b9f5 --- /dev/null +++ b/04-ISSUES/128-the-hosts-file-is-written-whole/00-report.md @@ -0,0 +1,59 @@ +--- +status: open +opened: 2026-09-26 +located-in: [mesh-controller internal/catalogue/facts.go, mesh-host internal/apply] +--- + +# 128 — the machine's hosts file is written whole, and on a workstation it is shared + +## What was observed + +The private network asks for the `node-names` fact, and the mesh delivers it as +`/etc/hosts`. `nodeNames` composes a **complete** file — its own header, `localhost`, the +machine's own name, and every name in the mesh — and the host writes it over whatever is there. + +On an adopted workstation the file the mesh holds contains, besides the predecessor's block of +mesh names: + +- the distribution's own lines (`localhost`, the machine's `.localdomain` name); +- two marked blocks (`# BEGIN … # END …`) maintained by a local-development tool, pointing a + dozen development hostnames at `127.0.0.1` — rewritten by that tool whenever its project + list changes; +- hand-added entries of the operator's. + +Today the file is only **held** ([ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md)): +the private network was assigned, not yet taken, so nothing was lost. Taking it — or +converging the node, which takes everything — replaces the file. The development tool's +entries disappear, its projects stop resolving, and every later write it makes is overwritten +at the next change to the mesh's names (a machine joins, a route is contributed), silently and +without a failure anywhere: the development tool thinks it wrote its block, and the mesh thinks +it owns the file. + +This is [ADR 0102](../../02-DECISIONS/0102-the-mesh-writes-into-a-shared-file-never-over-it.md)'s +failure exactly — a file the mesh shares with software it did not install, written over — in a +file 0102 did not name, because its merge verb is structured (`into: json`) and a hosts file is +not JSON. + +A second, smaller finding from the same reading: the fact's contents depend on which machines +hold the private network. A machine that is enrolled but not yet assigned the private network +is in neither `node-names` nor `node-zones`; its name resolves on the others only for as long +as a predecessor's hosts block survives. Taking the hosts file before every machine is on the +private network loses that name too. + +## What would have prevented it + +- A **marked-region** merge in the host's vocabulary: `into: "block"` (or similar) — the host + owns only the lines between its own begin and end markers, keeps everything outside them + byte for byte, records what the region held before, and on undeclare removes the region and + nothing else. The shape local tools already use for this very file. +- The `node-names` fact written as that region — no header of its own, no `localhost`, no + machine name — so the distribution's lines and every other tool's stay where they are. +- A converge preview that names a held file the take would replace *whole*, with its line + count before and after, so a person sees "hosts: 31 lines → 12" before the flip. + +## Evidence to carry into diagnosis + +- `internal/catalogue/facts.go`, `nodeNames`: the complete file is built here. +- The host's file resource supports `into: "json"` only; anything else is a whole write. +- `node show ` on the adopted workstation: `holds file /etc/hosts + mesh-wireguard.fact-node-names`, original kept. From 60a53f9b1862277fb6b2f67a1801237acac38d32 Mon Sep 17 00:00:00 2001 From: jochen Date: Sat, 26 Sep 2026 23:38:30 +0200 Subject: [PATCH 07/27] issue 129: nothing makes a machine trust the mesh's own certificate authority --- .../00-report.md | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md diff --git a/04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md b/04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md new file mode 100644 index 0000000..47ce7a4 --- /dev/null +++ b/04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md @@ -0,0 +1,62 @@ +--- +status: open +opened: 2026-09-26 +located-in: [mesh-controller, mesh-catalog step-ca] +--- + +# 129 — nothing makes a machine trust the mesh's own certificate authority + +## What was observed + +On an enrolled, adopted workstation — on the private network, resolving the mesh's names +through the mesh's resolver — every HTTPS name the mesh serves internally fails verification: + +``` +curl https://git..internal/ +curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20) +``` + +The route proxy presents a certificate issued by the mesh's internal authority (step-ca, the +`internal-acme-ca` provision). The machine's trust store holds the **predecessor's** authority +and a developer tool's local root, and nothing of the mesh's. No module installs the mesh's +root, and no fact carries it: step-ca's only consumers are proxies, which obtain certificates +over ACME and never need the root on the machine they run on. + +[Issue 048](../048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md) found the same +shape for the mesh's registry and resolved it by treating the private network as the transport +security ([ADR 0082](../../02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md)): +the runtime pulls in the clear, over the tunnel. That answer does not carry over. A browser, git +over HTTPS, a package manager and every TLS client a person or a module uses verify the +certificate chain, and there is no "insecure registries" for them — nor should there be. + +Consequences today, all silent until someone tries: + +- a person on a workstation cannot open any internal HTTPS name without a warning; +- git over HTTPS to the mesh's forge fails, so the working clone URL is ssh-only; +- a module on a non-hub machine that calls another module's internal HTTPS name fails + verification unless its image happens to carry the root; +- the predecessor's authority cannot be retired from any machine while anything there still + speaks TLS to a mesh name, because it is the only authority those machines trust. + +## What would have prevented it + +- A **mesh fact carrying the internal authority's root** (public material; the controller or + the step-ca module is its source), written onto every machine on the private network — the + same reasoning that has the private network write the registry trust and the names: being on + the network is what makes a machine one that speaks to the mesh's names. +- A resource that puts it where the machine's TLS clients look — on Arch, + `/etc/ca-certificates/trust-source/anchors/` — and **refreshes the extracted bundles** + (`update-ca-trust`). The refresh is the open design question: it is a command, and the link + may not carry an action ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)). A declared + one-shot unit, or a host primitive for "trust this anchor", are the obvious candidates. +- Removal symmetric to arrival: undeclared, the anchor goes and the bundles are refreshed again, + so a machine leaving the mesh stops trusting it. + +## Evidence to carry into diagnosis + +- `step-ca` module: provides `acme-ca` / `internal-acme-ca`, listens on 9000 for proxies; no + resource writes its root anywhere but its own state directory. +- The private network's generator writes `/etc/hosts` and the registry trust, and nothing + about certificates. +- On the workstation, the trust anchors present are the predecessor's authority and a local + development root; `trust list` shows no entry for the mesh. From c3313f6e1779f641165776a2e7baa504d20c9570 Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 00:03:15 +0200 Subject: [PATCH 08/27] 0117 review: seat table row + decision cited, networkd/dhcpcd lines match the modules, block placement, dispatcher scope, references --- .../0117-a-machines-uplink-is-a-seat.md | 37 +++++++++++++------ 03-DESIGN/01-to-be/26-the-seats.md | 4 +- 2 files changed, 29 insertions(+), 12 deletions(-) diff --git a/02-DECISIONS/0117-a-machines-uplink-is-a-seat.md b/02-DECISIONS/0117-a-machines-uplink-is-a-seat.md index 3cbbbbb..79a2f57 100644 --- a/02-DECISIONS/0117-a-machines-uplink-is-a-seat.md +++ b/02-DECISIONS/0117-a-machines-uplink-is-a-seat.md @@ -34,9 +34,9 @@ by accident: - **The private network's interface is exposed to the manager.** A manager that considers every interface its own may try to configure `mesh0`, or tear it down on a profile change. Nothing tells it not to. -- **Two managers on one machine go unnoticed.** Of the four machines adopted so far, one runs - systemd-networkd, two run NetworkManager, and one runs NetworkManager *and* dhcpcd at once: - two programs that each believe they own the machine's addresses and its resolver file. +- **Two managers on one machine go unnoticed.** Among the machines adopted so far, one runs + NetworkManager *and* dhcpcd at once: two programs that each believe they own the machine's + addresses and its resolver file. Nothing detected it, because nothing in the mesh knows the role exists. The machines differ in a way that matters: servers are wired and never move, while @@ -72,13 +72,16 @@ other: - the manager's package, and its service running and enabled at boot; - the manager's own configuration that leaves the resolver file to the mesh (`dns=none` for - NetworkManager, `nohook resolv.conf` for dhcpcd, and for systemd-networkd the equivalent - that stops it handing DNS to a resolver the machine does not use); + NetworkManager, `nohook resolv.conf` for dhcpcd, and nothing for systemd-networkd, which + never writes the resolver file); - the manager's own configuration that leaves the private network's interface alone - (NetworkManager's `unmanaged-devices` naming `mesh0`; for systemd-networkd, no network file - of the module's matches it); + (NetworkManager's `unmanaged-devices` naming `mesh0`; dhcpcd's `denyinterfaces mesh0`; for + systemd-networkd a network file of the module's matching `mesh0` as `Unmanaged=yes`); - each as a drop-in beside the manager's main file where the manager reads one, and written - *into* a shared file otherwise ([ADR 0102](0102-the-mesh-writes-into-a-shared-file-never-over-it.md)); + *into* a shared file otherwise, as a marked region the host owns + ([ADR 0102](0102-the-mesh-writes-into-a-shared-file-never-over-it.md)'s idea for text files), + placed where the manager reads it as global — at the start of `dhcpcd.conf`, above any + `interface` line, because every line after one belongs to that interface; - the service **reloaded** when a drop-in changes, never restarted — a restart drops the link, and the link is the mesh's own channel to the machine. **A manager that cannot reload is not restarted instead:** its setting takes effect at the manager's next start. Measured on the @@ -102,11 +105,23 @@ them, and the module's `access`, if it needs one, is read-only. replaced under another name. - A machine running two managers is found at assignment: the second holder is refused, and the operator decides which manager the machine keeps before either module is taken. -- Workstations keep joining networks the way they always have. The host already cooperates - with the manager — its dispatcher hook wakes it on every connectivity change — and nothing - here changes that. +- Workstations keep joining networks the way they always have. Under NetworkManager and + systemd-networkd the host already cooperates with the manager — its dispatcher hook wakes it + on every connectivity change — and nothing here changes that. A dhcpcd-only machine has no + such hook, and nothing here adds one. - The seat table gains one entry: `the-uplink`, node scope, delivering nothing, decided here. - **Not decided here:** whether the mesh should ever *offer* known networks to a machine — a sealed, add-only list the operator curates once for all workstations. That is a different question (the mesh holding credentials for links it must never be able to break) and gets its own record if it is wanted. + +## References + +- [ADR 0110](0110-a-seat-is-a-module-assignment-from-a-closed-set.md): the closed set this seat joins; + [to-be 26](../03-DESIGN/01-to-be/26-the-seats.md): the seat table +- [ADR 0102](0102-the-mesh-writes-into-a-shared-file-never-over-it.md): written into, never over +- [ADR 0051](0051-shared-data-is-the-operators.md): what is the operator's stays the operator's +- mesh-controller `internal/catalogue/seats.go` (the seat), `internal/overlay/generator.go` (the + mesh installs on top of the machine's own networking) +- mesh-catalog `modules/networkmanager`, `modules/systemd-networkd`, `modules/dhcpcd` +- mesh-host `internal/apply/block.go` (a file written into a marked region, `at` start or end) diff --git a/03-DESIGN/01-to-be/26-the-seats.md b/03-DESIGN/01-to-be/26-the-seats.md index 65dc1b9..bf2317d 100644 --- a/03-DESIGN/01-to-be/26-the-seats.md +++ b/03-DESIGN/01-to-be/26-the-seats.md @@ -8,11 +8,12 @@ code: - mesh-controller cmd/mesh-controller/source.go - mesh-controller internal/inventory/migrations/0032-a-source-may-live-on-a-seat.sql - mesh-catalog modules/gitea/module.json -updated: 2026-09-26 +updated: 2026-09-27 decisions: - 02-DECISIONS/0110-a-seat-is-a-module-assignment-from-a-closed-set.md - 02-DECISIONS/0111-a-build-source-is-on-the-git-seat-or-external.md - 02-DECISIONS/0109-a-package-registry-seat-is-one-per-ecosystem.md + - 02-DECISIONS/0117-a-machines-uplink-is-a-seat.md --- # 26 — The seats @@ -64,6 +65,7 @@ nobody argued for is an entry nobody can explain. | `the-private-network` | node | — | the private network the mesh runs over | | `the-resolver-configuration` | node | — | whichever of the alternative resolver configurations is chosen | | `the-showcase` | node | — | the showcase module | +| `the-uplink` | node | — | the program that manages the machine's own network ([ADR 0117](../../02-DECISIONS/0117-a-machines-uplink-is-a-seat.md)) | The controller holds this set in code, and a test asserts both its size and that every entry names the record that made it a seat. **This table and [ADR 0110](../../02-DECISIONS/0110-a-seat-is-a-module-assignment-from-a-closed-set.md) From a865fc7d7913f7c55f26c4571eca39e25437cf67 Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 00:03:30 +0200 Subject: [PATCH 09/27] 128 review: located; the fix as built (block, at, never held, order) --- .../00-report.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/04-ISSUES/128-the-hosts-file-is-written-whole/00-report.md b/04-ISSUES/128-the-hosts-file-is-written-whole/00-report.md index 8e5b9f5..fa02a00 100644 --- a/04-ISSUES/128-the-hosts-file-is-written-whole/00-report.md +++ b/04-ISSUES/128-the-hosts-file-is-written-whole/00-report.md @@ -1,5 +1,5 @@ --- -status: open +status: located opened: 2026-09-26 located-in: [mesh-controller internal/catalogue/facts.go, mesh-host internal/apply] --- @@ -51,6 +51,20 @@ private network loses that name too. - A converge preview that names a held file the take would replace *whole*, with its line count before and after, so a person sees "hosts: 31 lines → 12" before the flip. +## The fix, as built (in review) + +- **Host:** a file resource may say `"into": "block"`. The host owns only the lines between + `# BEGIN mesh ` and `# END mesh ` and keeps everything outside them byte for byte. A + new region goes at the `end` by default, or at the `start` (`"at": "start"`) for files where a + line's meaning depends on what stands above it; a region already present is never moved. + Undeclared, what the region held before is put back, or the region is removed and nothing + else. Replacing nothing, it is written on an adopted node without being held — so a machine + gets the mesh's names before its private network is taken. +- **Controller:** `node-names` is a fact written into a shared file, emitted as that region: the + mesh's names only, no header, no `localhost`, no `127.0.1.1` line. +- **Order:** a host older than the block mode refuses the whole declaration on an unknown + `into`, so hosts are upgraded before the controller that emits it. + ## Evidence to carry into diagnosis - `internal/catalogue/facts.go`, `nodeNames`: the complete file is built here. From 90b8eeff6b92e20bdbc26538d584bca314959843 Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 00:03:36 +0200 Subject: [PATCH 10/27] 129 review: the example name is a routed name, not a doubled suffix --- .../00-report.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md b/04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md index 47ce7a4..322cb2d 100644 --- a/04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md +++ b/04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md @@ -12,7 +12,7 @@ On an enrolled, adopted workstation — on the private network, resolving the me through the mesh's resolver — every HTTPS name the mesh serves internally fails verification: ``` -curl https://git..internal/ +curl https:/// curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20) ``` From df4a3538c3decbc7024fca9ddad4ae1db1dc1f85 Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 00:07:42 +0200 Subject: [PATCH 11/27] =?UTF-8?q?0117=20review:=20a=20holder's=20service?= =?UTF-8?q?=20declares=20no=20state=20=E2=80=94=20the=20manager's=20lifecy?= =?UTF-8?q?cle=20is=20the=20machine's;=20a=20start-only=20setting's=20gap?= =?UTF-8?q?=20on=20a=20fresh=20machine?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- 02-DECISIONS/0117-a-machines-uplink-is-a-seat.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/02-DECISIONS/0117-a-machines-uplink-is-a-seat.md b/02-DECISIONS/0117-a-machines-uplink-is-a-seat.md index 79a2f57..7fbc5da 100644 --- a/02-DECISIONS/0117-a-machines-uplink-is-a-seat.md +++ b/02-DECISIONS/0117-a-machines-uplink-is-a-seat.md @@ -70,7 +70,11 @@ with nothing more. Assigning a second is refused, naming the first. **What a holder declares** — only what keeps the manager and the mesh from contradicting each other: -- the manager's package, and its service running and enabled at boot; +- the manager's package, present — and its service **with no state**: the manager's lifecycle is + the machine's. The mesh never starts, stops, enables or disables it, because stopping it takes + the link down, and a holder unassigned by mistake — or the wrong holder assigned — must not be + able to do that, nor start a second manager beside the one the machine runs. The service is + declared only so a change to the holder's settings reaches a *running* manager; - the manager's own configuration that leaves the resolver file to the mesh (`dns=none` for NetworkManager, `nohook resolv.conf` for dhcpcd, and nothing for systemd-networkd, which never writes the resolver file); @@ -103,6 +107,10 @@ them, and the module's `access`, if it needs one, is read-only. adopted node the holder's drop-in arrives beside the predecessor's; both say the same thing, and the predecessor's is retired by hand after the take, like any other file the mesh replaced under another name. +- A setting a manager reads only at its start is not in force until then. On an adopted machine + the predecessor's identical line normally already is; on a machine that was not adopted, + dhcpcd's resolver hook keeps rewriting the resolver file until dhcpcd next starts, and the + operator restarts it once, in a window of their choosing. - A machine running two managers is found at assignment: the second holder is refused, and the operator decides which manager the machine keeps before either module is taken. - Workstations keep joining networks the way they always have. Under NetworkManager and From 5b76a09da678eefb42c1f53106203e562ad3f091 Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 00:08:09 +0200 Subject: [PATCH 12/27] issue 130: undeclaring a service stops it, even one the mesh only reloads or keeps running --- .../00-report.md | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md diff --git a/04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md b/04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md new file mode 100644 index 0000000..1b6177d --- /dev/null +++ b/04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md @@ -0,0 +1,43 @@ +--- +status: located +opened: 2026-09-27 +located-in: [mesh-host internal/apply/apply.go (remove), mesh-controller internal/overlay, mesh-catalog modules/sshd] +--- + +# 130 — undeclaring a service stops it, even one the mesh only reloads or only keeps running + +## What was observed + +Reviewing the uplink modules ([ADR 0117](../../02-DECISIONS/0117-a-machines-uplink-is-a-seat.md)) +found that the host's `remove` path stops every `service` resource that is no longer declared: +`SetServiceState(..., "stopped")`, reported as "stopped; the unit file is not the host's to +delete". `store.Orphans` matches by id alone. So any of these stops the unit: + +- the module is unassigned — by mistake, or to switch it for another; +- the node is sent a deliberately-empty declaration ([issue 127](../127-a-declaration-that-shrinks-to-empty-is-skipped-not-sent/00-report.md)); +- a later catalogue version renames the resource's `id`. + +That is right for a service the mesh brought into being. It is wrong for a unit the mesh +declares only to act on — and the catalogue already has two: + +- **The private network declares `docker.service`** (`registry-trust-reload`, state `running`) + so that a change to the registry trust reloads the runtime ([ADR 0102](../../02-DECISIONS/0102-the-mesh-writes-into-a-shared-file-never-over-it.md)). + Unassigning the private network stops the container runtime, and every container on the + machine with it — including ones the mesh does not manage. +- **The sshd module declares `sshd.service`.** Unassigning it stops the machine's ssh daemon: + the lockout the same module's `listens` rule says a firewall must never arrange. + +The uplink modules would have added a third and a fourth: unassigning the network manager's +module would have stopped the network manager, taking the machine off the only link the mesh +reaches it by. + +## What would have prevented it + +- A service resource that says the unit's **lifecycle is the machine's**: declared with no + `state`, the mesh never starts, stops, enables or disables it; it only reloads or restarts a + *running* unit when a trigger changes; undeclared, it is left exactly as it is. (Being built + on mesh-host `feat/a-file-written-into-a-marked-block` for the uplink modules.) +- Then: `registry-trust-reload` declared that way (the runtime is the machine's), and the sshd + module's service too — a machine's ssh daemon outlives any module that configures it. +- A plan or unassign preview that names every unit an undeclare will stop, so the consequence + is read before it happens. From dd4cbabffbc08f1e345c45dc1a7e8d456377e20f Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 00:08:23 +0200 Subject: [PATCH 13/27] 130: ADR 0117 named, not linked, until it is on main --- 04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md b/04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md index 1b6177d..4c0a4cd 100644 --- a/04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md +++ b/04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md @@ -8,7 +8,7 @@ located-in: [mesh-host internal/apply/apply.go (remove), mesh-controller interna ## What was observed -Reviewing the uplink modules ([ADR 0117](../../02-DECISIONS/0117-a-machines-uplink-is-a-seat.md)) +Reviewing the uplink modules (ADR 0117, in review) found that the host's `remove` path stops every `service` resource that is no longer declared: `SetServiceState(..., "stopped")`, reported as "stopped; the unit file is not the host's to delete". `store.Orphans` matches by id alone. So any of these stops the unit: From ebd19c4c6c9434e535887cfe7a6526af514e6912 Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 00:16:01 +0200 Subject: [PATCH 14/27] =?UTF-8?q?ADR=200118:=20undeclaring=20removes=20wha?= =?UTF-8?q?t=20the=20mesh=20made,=20gives=20back=20what=20it=20changed,=20?= =?UTF-8?q?leaves=20the=20machine's=20units=20as=20they=20are=20=E2=80=94?= =?UTF-8?q?=20resolves=20issue=20130?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...aring-leaves-the-machines-units-running.md | 100 ++++++++++++++++++ 02-DECISIONS/README.md | 1 + .../00-report.md | 13 ++- 3 files changed, 113 insertions(+), 1 deletion(-) create mode 100644 02-DECISIONS/0118-undeclaring-leaves-the-machines-units-running.md diff --git a/02-DECISIONS/0118-undeclaring-leaves-the-machines-units-running.md b/02-DECISIONS/0118-undeclaring-leaves-the-machines-units-running.md new file mode 100644 index 0000000..6d01fe8 --- /dev/null +++ b/02-DECISIONS/0118-undeclaring-leaves-the-machines-units-running.md @@ -0,0 +1,100 @@ +--- +topic: what runs on it +status: accepted +date: 2026-09-27 +deciders: jochen +reconstructed: false +extends: 0102-the-mesh-writes-into-a-shared-file-never-over-it.md +--- + +# 118. Undeclaring removes what the mesh made, gives back what it changed, and leaves the machine's units as they are + +## Context + +When a resource stops being declared — its module unassigned, the node sent a +deliberately-empty declaration ([issue 127](../04-ISSUES/127-a-declaration-that-shrinks-to-empty-is-skipped-not-sent/00-report.md)), +or a new catalogue version renaming its id — the host undoes it. The host's own code states +the rule it means to follow: **it removes what it made and leaves what it merely configured.** +For almost every resource it does exactly that: + +- a container, a network, a process's unit, a directory it created: removed; +- a file it created: removed; a file it replaced: its kept original put back + ([ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md)); +- keys and list members it wrote into a shared file: given back as they were + ([ADR 0102](0102-the-mesh-writes-into-a-shared-file-never-over-it.md)); +- a package: left installed — the host cannot know it is unused; +- an operator's path it was given access to: never touched + ([ADR 0051](0051-shared-data-is-the-operators.md)). + +**A service is the exception.** A `service` resource never installs a unit: it puts one that +already exists — the distribution's, the operator's — into a state. Undeclared, the host stops +it. That contradicts the rule above, and in practice it is the most dangerous thing an +undeclare can do. Found reviewing the uplink modules +([issue 130](../04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md)): + +- the private network declares the container runtime's unit only so a change to the registry + trust reloads it — unassigning the private network stops the runtime, and every container on + the machine, the mesh's and not; +- the sshd module declares the ssh daemon — unassigning it stops ssh, the lockout that module's + own `listens` rule forbids; +- the uplink modules would have stopped the network manager, taking the machine off the only + link the mesh reaches it by. + +ADR 0117 (in review) answered that for its own modules with a +service declared with no `state`. Every other module that declares a unit it did not make is +exposed in the same way, and relying on each author to remember an opt-out is how the next one +is missed. + +## Considered Options + +**1. Undeclaring touches nothing on the machine.** Rejected. What the mesh made would outlive +the module that made it: a container nobody manages keeps serving and stops being patched; a +unit the mesh wrote keeps running a bundle nothing updates; a name collides when the module +is assigned again. An undeclare that leaves the mesh's own work behind is an orphan factory. + +**2. Keep stopping services; make "leave it running" an opt-in per resource.** Rejected. It +keeps the dangerous behaviour as the default for exactly the units that matter most — the +runtime, the ssh daemon, the network — and each new module is one forgotten field away from a +machine that goes dark when it is unassigned. + +**3. The line is ownership.** Chosen. + +## Decision + +**Undeclaring removes what the mesh made, gives back what it changed, and leaves what was the +machine's as it is.** For a unit, that means: **the host never stops, starts, disables or +enables a unit it did not create when that unit stops being declared.** An undeclared `service` +is forgotten — reported as such — and the unit keeps whatever state it is in. + +- A unit the mesh *did* create — a `process` resource's unit, which the host writes — is still + stopped and removed with it. That is the mesh's own code. +- The service's settings the mesh wrote are given back by their own resources (a kept original + restored, a region or keys removed). A running service keeps running on what it read until it + next reads its configuration; the mesh does not restart it to make it notice. +- A service declared with no `state` (ADR 0117) remains the way to say the mesh must not + **start** a unit either; this record is about what happens when a declaration goes away, and + covers every service. +- An operator who wants a unit stopped when its module goes says so first: declare it + `stopped`, push, then unassign. Stopping a machine's unit is a decision, made visibly — never + a side effect of removing a module. + +## Consequences + +- Unassigning the private network no longer stops the container runtime; unassigning sshd no + longer stops ssh; no uplink module can take a machine's network down on its way out. +- The host's removal report says "forgotten; the unit is the machine's" where it used to say + "stopped". A module that relied on its daemon stopping when unassigned — none in the catalogue + today does on purpose — needs the explicit step above. +- A daemon can keep running after its module is gone, on configuration that was taken back from + under it. That is a visible, running process the operator can see and stop; the alternative + was an invisible outage. +- **Not decided here:** an unassign preview that lists what an undeclare will remove and what it + will leave running. Issue 130 asks for it; it is the controller's to build. + +## References + +- [issue 130](../04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md): the finding +- ADR 0117 (in review): the uplink modules, and a service with no state +- [ADR 0102](0102-the-mesh-writes-into-a-shared-file-never-over-it.md), [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md): + what is given back, and how +- mesh-host `internal/apply/apply.go` (`remove`, the service case) diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index de2b056..ba76b62 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -200,6 +200,7 @@ python3 00-META/checks/index.py fail if stale - **0114** — [A credential two parties hold rotates over two credentials; one a single party holds rotates in place, staged; and retiring a credential never removes what it reached](0114-a-shared-credential-rotates-over-two-credentials.md) *(proposed)* - **0115** — [One assignment of a module per node: the module's name is the assignment's identity](0115-one-assignment-of-a-module-per-node.md) *(proposed)* - **0117** — [A machine's uplink is a seat: the mesh configures the manager, never the link](0117-a-machines-uplink-is-a-seat.md) +- **0118** — [Undeclaring removes what the mesh made, gives back what it changed, and leaves the machine's units as they are](0118-undeclaring-leaves-the-machines-units-running.md) ### How it is built diff --git a/04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md b/04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md index 4c0a4cd..2bfbd3e 100644 --- a/04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md +++ b/04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md @@ -1,7 +1,8 @@ --- status: located opened: 2026-09-27 -located-in: [mesh-host internal/apply/apply.go (remove), mesh-controller internal/overlay, mesh-catalog modules/sshd] +located-in: [mesh-host internal/apply/apply.go (remove)] +amended-design: 02-DECISIONS/0118-undeclaring-leaves-the-machines-units-running.md --- # 130 — undeclaring a service stops it, even one the mesh only reloads or only keeps running @@ -41,3 +42,13 @@ reaches it by. module's service too — a machine's ssh daemon outlives any module that configures it. - A plan or unassign preview that names every unit an undeclare will stop, so the consequence is read before it happens. + +## Resolution + +[ADR 0118](../../02-DECISIONS/0118-undeclaring-leaves-the-machines-units-running.md): undeclaring +removes what the mesh made, gives back what it changed, and leaves the machine's units as they +are. An undeclared `service` is forgotten, never stopped — the host did not create the unit. +That covers the runtime, sshd and the uplink modules at once, without each module opting out; +the private network and the sshd module need no change. A `process`'s unit, which the host does +write, is still stopped and removed. The unassign preview asked for above is left open for the +controller. From 13208f0f4825962038f454b682ebd5f08d4eeef7 Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 00:21:18 +0200 Subject: [PATCH 15/27] =?UTF-8?q?0118:=20give=20a=20unit=20back=20the=20st?= =?UTF-8?q?ate=20it=20was=20found=20in=20=E2=80=94=20never-stop=20broke=20?= =?UTF-8?q?the=20converge=20rollback;=20process=20removal=20found=20and=20?= =?UTF-8?q?fixed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...-a-unit-back-the-state-it-was-found-in.md} | 51 +++++++++++++------ 02-DECISIONS/README.md | 2 +- .../00-report.md | 28 +++++++--- 3 files changed, 56 insertions(+), 25 deletions(-) rename 02-DECISIONS/{0118-undeclaring-leaves-the-machines-units-running.md => 0118-undeclaring-gives-a-unit-back-the-state-it-was-found-in.md} (60%) diff --git a/02-DECISIONS/0118-undeclaring-leaves-the-machines-units-running.md b/02-DECISIONS/0118-undeclaring-gives-a-unit-back-the-state-it-was-found-in.md similarity index 60% rename from 02-DECISIONS/0118-undeclaring-leaves-the-machines-units-running.md rename to 02-DECISIONS/0118-undeclaring-gives-a-unit-back-the-state-it-was-found-in.md index 6d01fe8..27ad6f7 100644 --- a/02-DECISIONS/0118-undeclaring-leaves-the-machines-units-running.md +++ b/02-DECISIONS/0118-undeclaring-gives-a-unit-back-the-state-it-was-found-in.md @@ -7,7 +7,7 @@ reconstructed: false extends: 0102-the-mesh-writes-into-a-shared-file-never-over-it.md --- -# 118. Undeclaring removes what the mesh made, gives back what it changed, and leaves the machine's units as they are +# 118. Undeclaring removes what the mesh made, and gives a unit back the state it was found in ## Context @@ -57,34 +57,53 @@ keeps the dangerous behaviour as the default for exactly the units that matter m runtime, the ssh daemon, the network — and each new module is one forgotten field away from a machine that goes dark when it is unassigned. -**3. The line is ownership.** Chosen. +**3. Never stop a unit the mesh did not create.** Rejected, found while implementing it. The +mesh's packet filter is a unit the distribution installed and the mesh started at converge; +returning a node to adopted ([ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md)) +unloads it by undeclaring it. Never stopping it would leave the mesh's filter loaded beside the +predecessor's firewall re-enabled — the one rollback a converge promises, broken. Who wrote the +unit file is not the line; what the mesh *did* to the unit is. + +**4. Give the unit back the state it was found in.** Chosen. ## Decision -**Undeclaring removes what the mesh made, gives back what it changed, and leaves what was the -machine's as it is.** For a unit, that means: **the host never stops, starts, disables or -enables a unit it did not create when that unit stops being declared.** An undeclared `service` -is forgotten — reported as such — and the unit keeps whatever state it is in. +**Undeclaring removes what the mesh made and gives back what it changed.** For a unit the mesh +did not create, what it changed is the unit's state, so that is what is given back: **the host +records the state it first found the unit in, and undeclaring returns the unit to it.** -- A unit the mesh *did* create — a `process` resource's unit, which the host writes — is still - stopped and removed with it. That is the mesh's own code. +- **Recorded once**, the first time the host applies the service — whether it was running, and, + where the declaration sets it, whether it was enabled at boot — and carried in the host's + record from then on. Later applies never overwrite it: by then the unit's state is the mesh's + doing. +- **A unit found running is left running.** The container runtime, the ssh daemon, a network + manager: running before the mesh arrived, running after it leaves. +- **A unit the mesh started is stopped again**, and one it enabled is disabled again — the packet + filter a converge loaded, which returning to adopted unloads. +- **Never started on the way out.** A unit the mesh stopped is not started again when its + declaration goes; starting something is a decision, and the operator makes it. +- **Unknown is left alone.** A record written before the host kept what it found says nothing + about the unit before the mesh; the unit is left exactly as it is. A unit left running can be + stopped by the operator; one stopped by mistake may be the link the operator needed to do it. +- A unit the mesh *did* create — a `process` resource's unit and bundle — is stopped and removed + with its declaration. That is the mesh's own code. (Before this record there was no way to + remove one at all: an undeclared process failed every apply on its node.) - The service's settings the mesh wrote are given back by their own resources (a kept original restored, a region or keys removed). A running service keeps running on what it read until it next reads its configuration; the mesh does not restart it to make it notice. - A service declared with no `state` (ADR 0117) remains the way to say the mesh must not - **start** a unit either; this record is about what happens when a declaration goes away, and - covers every service. -- An operator who wants a unit stopped when its module goes says so first: declare it - `stopped`, push, then unassign. Stopping a machine's unit is a decision, made visibly — never - a side effect of removing a module. + **start** a unit either; undeclared, it is forgotten. ## Consequences - Unassigning the private network no longer stops the container runtime; unassigning sshd no longer stops ssh; no uplink module can take a machine's network down on its way out. -- The host's removal report says "forgotten; the unit is the machine's" where it used to say - "stopped". A module that relied on its daemon stopping when unassigned — none in the catalogue - today does on purpose — needs the explicit step above. +- The host's removal report says what it gave back — "restored: stopped again, as the host + found it" — or "forgotten: it was running before the mesh; left as it is" where it used to say + "stopped". Its plan names each unit an undeclare will stop, before it does. +- On a fresh machine where the mesh installed and started a service, unassigning its module + stops it again — the mesh gave, the mesh takes back. An operator who wants it kept declares it + in a module of their own, or starts it themselves after. - A daemon can keep running after its module is gone, on configuration that was taken back from under it. That is a visible, running process the operator can see and stop; the alternative was an invisible outage. diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index ba76b62..350db19 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -200,7 +200,7 @@ python3 00-META/checks/index.py fail if stale - **0114** — [A credential two parties hold rotates over two credentials; one a single party holds rotates in place, staged; and retiring a credential never removes what it reached](0114-a-shared-credential-rotates-over-two-credentials.md) *(proposed)* - **0115** — [One assignment of a module per node: the module's name is the assignment's identity](0115-one-assignment-of-a-module-per-node.md) *(proposed)* - **0117** — [A machine's uplink is a seat: the mesh configures the manager, never the link](0117-a-machines-uplink-is-a-seat.md) -- **0118** — [Undeclaring removes what the mesh made, gives back what it changed, and leaves the machine's units as they are](0118-undeclaring-leaves-the-machines-units-running.md) +- **0118** — [Undeclaring removes what the mesh made, and gives a unit back the state it was found in](0118-undeclaring-gives-a-unit-back-the-state-it-was-found-in.md) ### How it is built diff --git a/04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md b/04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md index 2bfbd3e..23e7288 100644 --- a/04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md +++ b/04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md @@ -2,7 +2,7 @@ status: located opened: 2026-09-27 located-in: [mesh-host internal/apply/apply.go (remove)] -amended-design: 02-DECISIONS/0118-undeclaring-leaves-the-machines-units-running.md +amended-design: 02-DECISIONS/0118-undeclaring-gives-a-unit-back-the-state-it-was-found-in.md --- # 130 — undeclaring a service stops it, even one the mesh only reloads or only keeps running @@ -45,10 +45,22 @@ reaches it by. ## Resolution -[ADR 0118](../../02-DECISIONS/0118-undeclaring-leaves-the-machines-units-running.md): undeclaring -removes what the mesh made, gives back what it changed, and leaves the machine's units as they -are. An undeclared `service` is forgotten, never stopped — the host did not create the unit. -That covers the runtime, sshd and the uplink modules at once, without each module opting out; -the private network and the sshd module need no change. A `process`'s unit, which the host does -write, is still stopped and removed. The unassign preview asked for above is left open for the -controller. +[ADR 0118](../../02-DECISIONS/0118-undeclaring-gives-a-unit-back-the-state-it-was-found-in.md): +undeclaring removes what the mesh made and gives back what it changed. The host records the state +it first found a unit in, and undeclaring returns the unit to it — a unit found running (the +container runtime, sshd, a network manager) is left running; one the mesh started (the packet +filter a converge loaded) is stopped again; nothing is started on the way out; a record from +before the host kept what it found leaves the unit alone. That covers the runtime, sshd and the +uplink modules at once, without each module opting out; the private network and the sshd module +need no change. + +A first draft — never stop a unit the mesh did not create — was rejected while implementing it: +returning a converged node to adopted unloads the mesh's filter by exactly this path. + +Found on the way: an undeclared `process` failed every apply on its node (`remove` had no case +for it). Now removed with its unit, timer and bundle — the mesh's own code. `user` and `archive` +have the same gap and are left for their own decisions: removing a login or unpacked files is not +something to settle in passing. + +The unassign preview is partly answered — the host's plan names each unit it will stop — and the +controller's side is left open. From 248c99ca6c307310e0cdad9ec1ed6876a99c53e3 Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 00:58:17 +0200 Subject: [PATCH 16/27] 0118/130: link ADR 0117 now that it is on main --- ...undeclaring-gives-a-unit-back-the-state-it-was-found-in.md | 4 ++-- 04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/02-DECISIONS/0118-undeclaring-gives-a-unit-back-the-state-it-was-found-in.md b/02-DECISIONS/0118-undeclaring-gives-a-unit-back-the-state-it-was-found-in.md index 27ad6f7..cb925b0 100644 --- a/02-DECISIONS/0118-undeclaring-gives-a-unit-back-the-state-it-was-found-in.md +++ b/02-DECISIONS/0118-undeclaring-gives-a-unit-back-the-state-it-was-found-in.md @@ -40,7 +40,7 @@ undeclare can do. Found reviewing the uplink modules - the uplink modules would have stopped the network manager, taking the machine off the only link the mesh reaches it by. -ADR 0117 (in review) answered that for its own modules with a +[ADR 0117](0117-a-machines-uplink-is-a-seat.md) answered that for its own modules with a service declared with no `state`. Every other module that declares a unit it did not make is exposed in the same way, and relying on each author to remember an opt-out is how the next one is missed. @@ -113,7 +113,7 @@ records the state it first found the unit in, and undeclaring returns the unit t ## References - [issue 130](../04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md): the finding -- ADR 0117 (in review): the uplink modules, and a service with no state +- [ADR 0117](0117-a-machines-uplink-is-a-seat.md): the uplink modules, and a service with no state - [ADR 0102](0102-the-mesh-writes-into-a-shared-file-never-over-it.md), [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md): what is given back, and how - mesh-host `internal/apply/apply.go` (`remove`, the service case) diff --git a/04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md b/04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md index 23e7288..4824dd2 100644 --- a/04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md +++ b/04-ISSUES/130-undeclaring-a-service-stops-it/00-report.md @@ -9,7 +9,7 @@ amended-design: 02-DECISIONS/0118-undeclaring-gives-a-unit-back-the-state-it-was ## What was observed -Reviewing the uplink modules (ADR 0117, in review) +Reviewing the uplink modules ([ADR 0117](../../02-DECISIONS/0117-a-machines-uplink-is-a-seat.md)) found that the host's `remove` path stops every `service` resource that is no longer declared: `SetServiceState(..., "stopped")`, reported as "stopped; the unit file is not the host's to delete". `store.Orphans` matches by id alone. So any of these stops the unit: From 8a78ff4efef3448a1f2e27b81f6151baff8c9cd3 Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 00:40:31 +0200 Subject: [PATCH 17/27] ADR 0119: a taken tunnel's predecessor is retired once the take is proven --- ...-a-taken-tunnels-predecessor-is-retired.md | 78 +++++++++++++++++++ 02-DECISIONS/README.md | 1 + 2 files changed, 79 insertions(+) create mode 100644 02-DECISIONS/0119-a-taken-tunnels-predecessor-is-retired.md diff --git a/02-DECISIONS/0119-a-taken-tunnels-predecessor-is-retired.md b/02-DECISIONS/0119-a-taken-tunnels-predecessor-is-retired.md new file mode 100644 index 0000000..89db5b1 --- /dev/null +++ b/02-DECISIONS/0119-a-taken-tunnels-predecessor-is-retired.md @@ -0,0 +1,78 @@ +--- +topic: the mesh +status: accepted +date: 2026-09-27 +deciders: jochen +reconstructed: false +extends: 0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md +--- + +# 119. A taken tunnel's predecessor is retired once the take is proven + +## Context + +[ADR 0105](0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md) has the private network take +over the tunnel it finds: the found unit stopped and disabled, never flushed, and **its +configuration left on disk, kept like any held file.** That was the right caution for the take +itself — if the mesh's interface failed to come up, the host starts the found unit again and the +peers never notice — and every apply since stops the found unit again should anyone start it. + +What it leaves is a predecessor that never finishes leaving. On every machine that has enrolled, +the tunnel is the mesh's and has been proven so — its interface up with the found key, the peers +handshaking, the machines resolving and reaching each other over it — and still the predecessor's +configuration sits where its unit reads it, held for a module that has long since replaced it. +The predecessor itself is being deprecated. A tunnel that can be started again by one command, with +a configuration nothing maintains any more, is not a rollback path; it is a second way onto the +network that nobody is watching. And the hold never ends, so every node report keeps listing it. + +## Considered Options + +**1. Keep it, as 0105 says.** Rejected: the caution it bought is spent once the take is proven, and +what remains is a live, unmaintained way back onto the network. + +**2. Delete it at the take.** Rejected: the take is exactly the moment the fallback is needed. If +the mesh's interface does not come up, the host must still be able to raise the found one. + +**3. Retire it once the take is proven.** Chosen. + +## Decision + +**Once the mesh's interface has proven it carries the tunnel, the found interface's configuration +is removed from where its unit reads it.** + +- **Proven means:** the tunnel's state is *taken* — the found unit down and disabled, the mesh's + interface up with the found key — and the mesh's interface has completed a handshake with at + least one peer. Not before: until then, a failed take still falls back to the found unit. +- **Retired means:** the configuration file the found unit reads is removed. Its original was + already kept, before anything happened to it + ([ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md)), and stays kept; that copy + is the record of what the predecessor was, and a person's way back if one is ever wanted. +- The found unit stays disabled. Without its configuration it cannot raise the interface, so the + every-apply stop that guarded against it becomes a check that finds nothing to do. +- **The hold ends.** What was held for the private network has been replaced; the node stops + reporting it. +- **The mesh never brings it back.** Undeclaring the private network does not restore the found + tunnel: the mesh stopped it, and nothing is started on the way out + (ADR 0118, in review). A machine whose + private network is unassigned has no tunnel until it is assigned again — which is what + unassigning it means. + +## Consequences + +- On every machine that took a tunnel, the predecessor's tunnel configuration disappears at the + first apply after the take is proven. Nothing a peer sees changes; the mesh's interface already + carries the same key, port, address and peers. +- A take that is never proven — no peer ever handshakes — keeps the found configuration, and the + node says so, so a broken take is visible rather than silently retired. +- Rolling back to the predecessor's tunnel becomes a deliberate act: copy the kept original back + and start its unit. The mesh does neither. +- 0105's "its configuration stays on disk, kept like any held file" holds until the take is proven, + and not after. + +## References + +- [ADR 0105](0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md): the take, and why it keeps + the found configuration during it +- [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md): kept originals +- ADR 0118 (in review): nothing is started on the way out +- mesh-host `internal/apply/takeover.go` diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 350db19..aae6bfc 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -133,6 +133,7 @@ python3 00-META/checks/index.py fail if stale - **0105** — [The mesh adopts the predecessor's tunnel in place](0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md) - **0106** — [The bus is NATS](0106-the-bus-is-nats.md) - **0116** — [The bus is built in five steps, and the protocol moves with it](0116-the-bus-is-built-in-five-steps.md) +- **0119** — [A taken tunnel's predecessor is retired once the take is proven](0119-a-taken-tunnels-predecessor-is-retired.md) ### Its tiers, from the bottom up From 2f195d501e0a9d498818d738d46ce9ef9e5419ac Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 00:40:57 +0200 Subject: [PATCH 18/27] to-be 08: the found tunnel's configuration is retired once the take is proven (ADR 0119) --- 03-DESIGN/01-to-be/08-connectivity.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/03-DESIGN/01-to-be/08-connectivity.md b/03-DESIGN/01-to-be/08-connectivity.md index ae7801f..2337889 100644 --- a/03-DESIGN/01-to-be/08-connectivity.md +++ b/03-DESIGN/01-to-be/08-connectivity.md @@ -7,11 +7,12 @@ code: - mesh-controller internal/identity/authority.go - mesh-host internal/identity/serving.go - mesh-host internal/apply (the service that reflects a rule set) -updated: 2026-09-25 +updated: 2026-09-27 decisions: - 02-DECISIONS/0104-a-provision-may-be-answered-by-an-adapter-to-the-predecessor.md - 02-DECISIONS/0106-the-bus-is-nats.md - 02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md + - 02-DECISIONS/0119-a-taken-tunnels-predecessor-is-retired.md - 02-DECISIONS/0108-a-route-carries-the-policy-applied-to-a-request.md - 02-DECISIONS/0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md - 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md @@ -768,6 +769,12 @@ where a found tunnel is left running beside the mesh's; where it is adopted ther The guard admits the mesh's ports from that one interface, and the predecessor's peers arrive on it. +*2026-09-27, [ADR 0119](../../02-DECISIONS/0119-a-taken-tunnels-predecessor-is-retired.md).* The +found configuration is kept only until the take is proven — the found unit down, the mesh's +interface up and handshaking with a peer. Then it is removed from where the found unit reads it +(its original stays kept), the hold ends, and the predecessor's tunnel cannot be raised again by +anything but a person restoring it by hand. Undeclaring the private network does not bring it back. + ## The bus is NATS *2026-09-23, [ADR 0106](../../02-DECISIONS/0106-the-bus-is-nats.md). Architecture to be written From 63c19456b4b1d9d8779ddd39972b66a4fdd4fc76 Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 00:53:13 +0200 Subject: [PATCH 19/27] 0119 review: rollback needs the private network unassigned first; a configuration written back is retired again with the first original kept; only the interface's own file, never a link --- .../0119-a-taken-tunnels-predecessor-is-retired.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/02-DECISIONS/0119-a-taken-tunnels-predecessor-is-retired.md b/02-DECISIONS/0119-a-taken-tunnels-predecessor-is-retired.md index 89db5b1..40f8f33 100644 --- a/02-DECISIONS/0119-a-taken-tunnels-predecessor-is-retired.md +++ b/02-DECISIONS/0119-a-taken-tunnels-predecessor-is-retired.md @@ -64,8 +64,17 @@ is removed from where its unit reads it.** carries the same key, port, address and peers. - A take that is never proven — no peer ever handshakes — keeps the found configuration, and the node says so, so a broken take is visible rather than silently retired. -- Rolling back to the predecessor's tunnel becomes a deliberate act: copy the kept original back - and start its unit. The mesh does neither. +- Rolling back to the predecessor's tunnel becomes a deliberate act, in this order: **unassign the + private network first**, then copy the kept original back and start its unit. The mesh does + neither. While the private network is still assigned, the tunnel is the mesh's: a restored + configuration is held and retired again at the next proven apply, and the found unit cannot + bind the port the mesh's interface holds. The node says so when it happens. +- A configuration something keeps writing back — the predecessor's own tooling, say — is retired + again each time it appears, but the first original stays the one kept; a different content is + kept once beside it, and the node reports that the configuration came back. +- The host retires only the found interface's own configuration file (`/etc/wireguard/.conf`), + never a path the mesh writes, and never a link: a configuration that is a link to somewhere else + is left, with its target, for a person to retire. - 0105's "its configuration stays on disk, kept like any held file" holds until the take is proven, and not after. From 0042ca9258d9a544a53fb3c11aef9c61b5adf34c Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 00:58:55 +0200 Subject: [PATCH 20/27] 0119: link ADR 0118 now that it is on main --- 02-DECISIONS/0119-a-taken-tunnels-predecessor-is-retired.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/02-DECISIONS/0119-a-taken-tunnels-predecessor-is-retired.md b/02-DECISIONS/0119-a-taken-tunnels-predecessor-is-retired.md index 40f8f33..d9bdade 100644 --- a/02-DECISIONS/0119-a-taken-tunnels-predecessor-is-retired.md +++ b/02-DECISIONS/0119-a-taken-tunnels-predecessor-is-retired.md @@ -53,7 +53,7 @@ is removed from where its unit reads it.** reporting it. - **The mesh never brings it back.** Undeclaring the private network does not restore the found tunnel: the mesh stopped it, and nothing is started on the way out - (ADR 0118, in review). A machine whose + ([ADR 0118](0118-undeclaring-gives-a-unit-back-the-state-it-was-found-in.md)). A machine whose private network is unassigned has no tunnel until it is assigned again — which is what unassigning it means. @@ -83,5 +83,5 @@ is removed from where its unit reads it.** - [ADR 0105](0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md): the take, and why it keeps the found configuration during it - [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md): kept originals -- ADR 0118 (in review): nothing is started on the way out +- [ADR 0118](0118-undeclaring-gives-a-unit-back-the-state-it-was-found-in.md): nothing is started on the way out - mesh-host `internal/apply/takeover.go` From 4d4012cdf6d2b9218f1a6259597555d5a265fdda Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 01:33:12 +0200 Subject: [PATCH 21/27] ADR 0120: a roster fact carries its format as a template; rewrite to-be 29 around it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The facts mechanism formatted the roster in Go in the control plane — one formatter per fact, in the consumer's own configuration language. ADR 0120 makes a fact a path and a template: the mesh owns the data, the module owns the format, and the control plane holds no format at all. to-be 29 (operator accounts + what lives under a home) is rewritten to ride it: the ssh files become roster templates, the whole ~/.ssh is owned with a found/owned boundary that cannot lock the operator out, keys are mesh-owned through an SSH CA (existing keys adopted not regenerated, the operator's personal key signed not minted), and the ssh-agent is a user-scoped service. --- ...r-fact-carries-its-format-as-a-template.md | 127 +++++++++++++++ .../29-a-node-has-operator-accounts.md | 154 ++++++++++++++---- 2 files changed, 251 insertions(+), 30 deletions(-) create mode 100644 02-DECISIONS/0120-a-roster-fact-carries-its-format-as-a-template.md diff --git a/02-DECISIONS/0120-a-roster-fact-carries-its-format-as-a-template.md b/02-DECISIONS/0120-a-roster-fact-carries-its-format-as-a-template.md new file mode 100644 index 0000000..8b28a97 --- /dev/null +++ b/02-DECISIONS/0120-a-roster-fact-carries-its-format-as-a-template.md @@ -0,0 +1,127 @@ +--- +topic: what runs on it +status: accepted +date: 2026-09-27 +deciders: jochen +reconstructed: false +extends: 0112-a-module-definition-names-no-node-mesh-or-path.md +--- + +# 120. A roster fact carries its format as a template: the mesh owns the data, the module owns the format + +## Context + +A **fact** is a thing only the mesh knows — which machines exist, what they are called, where they +are — written into a file where a module asks for it. The mesh computes it from the graph; a module +loads it, restarts on it, does what its software does with it. Facts replaced three modules that +existed only because computed output needed somewhere to live and ran no software of their own +([ADR 0040](0040-what-a-module-is.md)). + +But the *format* lived in the control plane. A fact was a name from a closed list, and each name had +a formatter written in Go beside the others: `node-names` wrote the roster as an `/etc/hosts` file, +`node-zones` wrote it as a dnsmasq resolver's `local=`/`address=` lines. Adding a consumer meant +adding a formatter — in the consumer's own configuration language — to the mesh. + +The ssh work made the cost plain. An operator's `~/.ssh` wants three roster projections — a +`known_hosts`, an ssh `config` of `Host` blocks, an `authorized_keys` — each in ssh's syntax. Under +the closed list that is three more formatters in the control plane, teaching it ssh's configuration +language. And it does not stop at ssh: every daemon that reads the roster in its own file format +would put its grammar here. The control plane was accreting the configuration languages of software +it does not run — the exact thing [ADR 0040](0040-what-a-module-is.md) says is a +module's and not the mesh's. + +The shape underneath is one shape. WireGuard's `[Peer]` blocks, `/etc/hosts`, dnsmasq's zones, an +ssh `known_hosts` — all of them are *the roster, projected into a file*. Only the projection differs, +and the projection belongs to whoever runs the software that reads it. + +## Considered Options + +**1. Keep the closed list; add a formatter per consumer.** Rejected. The control plane learns the +configuration language of every daemon any module might run, without bound, and each format lives in +the mesh rather than in the module that owns the file. A module cannot change how its own file is +written without a control-plane change. + +**2. A general placeholder vocabulary over `content`, like `${machine:address}` but for the +roster.** Rejected. The mesh's other substitutions each resolve to *one* scalar — this machine's +address, one provider's port. The roster is inherently a *repetition*: one block per machine. A flat +`${…}` vocabulary cannot iterate, and a mechanism that could would be a template in all but name. + +**3. The module gives a path and a template over the roster; the mesh renders it.** Chosen. The mesh +owns the data — who exists, their names and addresses — and hands it to a Go `text/template` the +module wrote. The mesh renders and reads neither the template's intent nor the file's meaning. + +## Decision + +**A fact is a path and a template.** In a module's manifest, `facts` maps a name the module chooses +to a `{ path, template }`. The template is a Go `text/template` over a fixed **roster view**: + +- `.Node` — this machine's bare name. +- `.Suffix` — what a mesh name ends in (`internal`, or the operator's choice), as composed. +- `.Names` — every name the mesh serves: the machines *and* the names it was told to route. +- `.Machines` — only the machines that are nodes of this mesh. + +Each of `.Names` and `.Machines` is a list of `{ Name, FQDN, Address }`. A machine the mesh has a +record for but cannot yet place has no address and is left out of both — a name that resolves to +nothing is a connection that hangs, so it is omitted rather than written (the same rule as before). + +**The mesh owns the data; the module owns the format.** The control plane holds **no** formatter. +The two built-in projections render through the same path any module uses: + +- **`/etc/hosts`** is a template on the mesh's own network module. The mesh writes `/etc/hosts` + because being on the private network is what gives a machine a name — but the *layout* is a + template like any other, shipped with the control plane because that module ships with it, not + because the control plane knows the hosts-file format. +- **dnsmasq's zones** move into dnsmasq. The `local=`/`address=` grammar is dnsmasq's configuration + language, and it now lives in dnsmasq's manifest, where the module that runs dnsmasq owns it. + +**The names-vs-machines distinction is the template's choice** ([04-ISSUES/111](../04-ISSUES/111-the-resolver-is-told-names-the-mesh-serves-not-only-machines/00-report.md)): +a container's hosts ranges `.Names`, so a routed name resolves to the machine serving it; a resolver +told the mesh's suffix is its own ranges `.Machines`, or a routed name written there with the suffix +appended is a name nobody will ever ask for. + +**A template that will not render is refused at composition, not on a machine.** A template that does +not parse, or reads a field the roster does not have, fails where the manifest is — the closed-list +safety, moved from the fact's *name* to the roster's *shape*. A daemon that starts, reads a file the +mesh could not render, and answers nothing is a much worse way to find out. + +**WireGuard stays a computed generator, and that is the line.** Its `mesh0.conf` is not a pure roster +projection — it carries topology the control plane decides: which peers are reachable, endpoints, hub +forwarding, keepalive for a NAT'd node. And it is *foundational*: the overlay must be up before any +module can be delivered, so the thing that writes it cannot itself be a delivered module. The line +this draws: **the substrate that delivery rides on is the control plane's; everything layered on a +working overlay is a roster template.** DNS, hosts, and ssh are layered; the overlay is the floor. + +## Consequences + +- **ssh is two templates and no control-plane change.** Once the roster view carries a machine's ssh + host key and its operator account ([to-be 29](../03-DESIGN/01-to-be/29-a-node-has-operator-accounts.md)), + `known_hosts`, the ssh `config`, and `authorized_keys` are templates on the ssh modules — the mesh + gains no knowledge of ssh's syntax. This ADR is what makes that work land without touching the + controller. +- **A new roster projection never touches the control plane.** Any module that reads the roster in + its own format ships its own template. +- **A module can change how its own file is written** without a control-plane change — it is editing + its own manifest. +- **The schema changed and is not backward compatible.** A fact was a string (a path); it is now + `{ path, template }`. The old string form has no template and cannot be auto-upgraded, because the + format it implied was the formatter this ADR deletes. The controller and every catalogue module + using facts — only dnsmasq — land together. A controller and a catalogue that disagree cannot + compose the module: the running daemon on a machine is unaffected, but the mesh will not send it a + new declaration until both sides agree. +- **The output did not change.** The `/etc/hosts` and dnsmasq zones a machine receives are + byte-for-byte what the deleted formatters wrote, pinned by tests that render the built-in template + and compose the real dnsmasq manifest. + +## References + +- [ADR 0040](0040-what-a-module-is.md): a module is software the mesh runs — a + format the mesh knows for software it does not run was the accretion this stops +- [ADR 0112](0112-a-module-definition-names-no-node-mesh-or-path.md): a module definition names no + path; this is its sibling for content — a module definition names no format the mesh must know +- [to-be 29](../03-DESIGN/01-to-be/29-a-node-has-operator-accounts.md): the ssh consumer this + unblocks, and the roster fields it will add +- [04-ISSUES/111](../04-ISSUES/111-the-resolver-is-told-names-the-mesh-serves-not-only-machines/00-report.md): every served name is not a + machine — now the template's choice of `.Names` or `.Machines` +- mesh-controller `internal/catalogue/roster.go` (the mechanism), `internal/overlay/generator.go` + (the built-in `/etc/hosts` template), `internal/catalogue/manifest.go` (`RosterFile`) +- mesh-catalog `modules/dnsmasq/module.json` (the zones template, dnsmasq's own) diff --git a/03-DESIGN/01-to-be/29-a-node-has-operator-accounts.md b/03-DESIGN/01-to-be/29-a-node-has-operator-accounts.md index f038a5d..ce62923 100644 --- a/03-DESIGN/01-to-be/29-a-node-has-operator-accounts.md +++ b/03-DESIGN/01-to-be/29-a-node-has-operator-accounts.md @@ -5,7 +5,10 @@ code: [] updated: 2026-09-27 decisions: - 02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md + - 02-DECISIONS/0120-a-roster-fact-carries-its-format-as-a-template.md - 02-DECISIONS/0051-shared-data-is-the-operators.md + - 02-DECISIONS/0113-the-vault-makes-every-secret.md + - 02-DECISIONS/0117-a-machines-uplink-is-a-seat.md --- # 29 — A node has operator accounts, and the mesh owns what lives under a home @@ -18,7 +21,7 @@ owned by, who a user service runs as, and — the case that surfaced this — wh person's `~/.ssh/config`, `~/.zshrc`, `~/.config`); the mesh, taking those over, kept the machine facts and dropped the human one. -Two things are missing, and they are one idea: +Several things are missing, and they are one idea. ## 1. The account is a node fact @@ -28,17 +31,14 @@ mesh already knows the node and its address, so `@` is then a com because it is exactly the fact that was silently lost — `ssh ace` failed to `ace` because nothing in the mesh said ace's account is `ace`. -It is **not** a credential. The account names a login; the key that authorises it is the -operator's, placed as a secret or an operator-owned file, never minted by the mesh (ADR 0051). - ## 2. A resource may live under a home, owned by its account [ADR 0112](../../02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md) placed a module's *system* data — `/`, owned by the module. It has no analog for the other half of the filesystem: the things that belong under a person's home and are owned by that -person. `~/.ssh/config`, `~/.ssh/config.d/mesh`, `~/.zshrc`, `~/.config/hal` — every one of these -is a resource the mesh should be able to place and own, resolved against **the account's home** -rather than a system root, and chowned to **the account** rather than to root or a module uid. +person. `~/.ssh/config`, `~/.zshrc`, `~/.config/hal` — every one of these is a resource the mesh +should be able to place and own, resolved against **the account's home** rather than a system +root, and chowned to **the account** rather than to root or a module uid. This is the same move as `${dir:…}`, one level over: a resource says `home: ` (or names an account requirement), and the mesh resolves the home directory and the owning uid on the node @@ -46,39 +46,133 @@ that account lives on. A module that writes operator config — the eventual rep `hal/terminal`, `hal/claude-code`, `hal/secrets` — declares its files this way and names no `/home/...` path, exactly as a system module now names no `/var/lib` path. +These are a **family**, not one module: an `ssh-client` module, a shell module, a `~/.config` +module, each a *universal-tier* consumer of the account fact — assigned wherever a person logs in, +which is every node, unlike the graphical stack that a capability gates. + +## 3. The whole of `~/.ssh` is the mesh's — with one boundary drawn inside it + +The predecessor owned a single file (`~/.ssh/config`) and left the rest alone; it drifted, because +owning one file beside foreign ones is not owning anything. The mesh should own **the directory**: +create `~/.ssh` at `0700`, chown it to the account, and own the files it places there — + +- **`config`** (or the mesh's region of it): the `Host` blocks for every other node, composed + from the roster; +- **`known_hosts`**: authoritative, so the "Host key verification failed / accept-new" dance that + cost real time during enrolment simply ends; +- **`authorized_keys`**: who may log into this account, governed centrally rather than by whichever + key happened to be pasted where. + +**The boundary — and it is the reason this is safe:** `~/.ssh` is the one directory where a wrong +declaration locks a person out of their own machine. So the mesh's *found-vs-owned* semantics +([ADR 0118](../../02-DECISIONS/0118-undeclaring-gives-a-unit-back-the-state-it-was-found-in.md), +adoption) apply *inside* the home directory. The mesh **owns** the directory and the files above; it +**holds as found — never rewrites, never removes** — the operator's own contents: their **private +keys** and their **personal drop-ins** (`config.d/personal`, the personal `Host` aliases a +workstation carries, exactly as `hosts.local` is the home the mesh never rewrites for `/etc/hosts`). +Reconcile removing an unassigned `config.d/mesh` is fine; the same logic aimed at `id_ed25519` or an +operator's own `authorized_keys` entry is a lockout. This is the login-channel cousin of the rule +[ADR 0117](../../02-DECISIONS/0117-a-machines-uplink-is-a-seat.md) draws for the uplink and the sshd +module draws for the firewall: **the mesh must never be able to arrange the one failure that severs +its own way back in.** The carve-out is not a convenience; it is that rule, in `~/.ssh`. + +## 4. Keys are the mesh's to generate — through a CA, and existing keys are adopted, not replaced + +Key *generation* is the mesh's, not each node's improvising its own. The clean form is an **SSH +certificate authority as a seat**, the sibling of the TLS internal CA the mesh already runs: + +- **Host certs.** The mesh signs each node's host key. Every node's `known_hosts` becomes one line + — `@cert-authority *. ` — and nothing is distributed per node; a new node is + trusted the instant its host key is signed. +- **User certs.** The mesh signs a cert naming the principals (accounts) allowed. Every node's + `authorized_keys` / sshd `TrustedUserCAKeys` becomes one trust line — no N×N key spraying — and + short-lived certs give rotation for free + ([ADR 0114](../../02-DECISIONS/0114-a-shared-credential-rotates-over-two-credentials.md)). +- The **CA private key is the mesh's**, a secret the vault makes + ([ADR 0113](../../02-DECISIONS/0113-the-vault-makes-every-secret.md)). + +**Three kinds of key, and only one is never minted.** Host keys (server identity) and pure +machine-to-machine keys the mesh may generate end to end. The operator's **personal** private key — +possibly on a hardware token, possibly used from an off-mesh laptop — the mesh **signs into a cert +but never generates**; that, and only that, is the residue of +[ADR 0051](../../02-DECISIONS/0051-shared-data-is-the-operators.md). So "keys are mesh-owned" and +"the operator's login key is the operator's" reconcile: the mesh owns the CA and the signing; it +holds the human's private half, never mints it. + +**Existing keys are not lost.** Taking ownership is *adoption*, not regeneration: a key already on a +machine is recorded and signed, not overwritten. The mesh gains authority over `~/.ssh` — it does +not clear it. An enrolling node's host key and the operator's existing key are carried forward; the +found-vs-owned boundary of §3 is exactly what guarantees nothing already there is destroyed. + +## 5. How it is distributed: the controller composes, the node applies + +None of this needs a node to discover the mesh, and none of it needs a control-plane module of its +own. The ssh files are **roster facts** +([ADR 0120](../../02-DECISIONS/0120-a-roster-fact-carries-its-format-as-a-template.md)): once the +roster view carries a node's **host key** and its **account** beside its name and address, the +`ssh-client` module ships a template for `known_hosts`, `config` and `authorized_keys`, and the +controller renders each node's copy from the full roster and pushes it. The mesh owns the data; the +module owns ssh's format; the control plane gains no ssh syntax. It is the same act as composing a +peer list or `/etc/hosts` — which is why there is **no novox-only "mesh-ssh" module**: the +centralization is the controller's composition, not a module that runs somewhere. Only non-secret +facts travel (names, addresses, accounts, host keys, the CA public key); the private key stays the +operator's, placed as an operator-owned file, referenced by path. + +## 6. The two modules, and the seat between them + +- **`sshd`** (server, every node) — manages sshd, owns and **reports** its host key so the roster + carries it, and trusts the user CA. +- **`ssh-client`** (client, every node) — owns `~/.ssh` per §3, consumes the roster and the CA + public key. +- **`the-ssh-ca`** (a seat, held on the control node) — signs host and user certs. + +They meet at the account and the CA, not at a bespoke module. The `sshd` server side already exists; +the client/identity side and the CA are the open pieces. + ## Why now, and why not yet -**Why it matters:** when HAL retires, the generators that keep `~/.ssh/config`, shell config and -the operator's `~/.config/hal` current retire with it. Without this, adding a node stops adding -its ssh alias, and a fresh machine has no operator dotfiles at all — the mesh would run every -service and leave the human unable to work on the box. The account is also load-bearing for -correctness already: `ssh ` (issue 122's cousin), user-scoped systemd units, and any file a -person rather than a daemon must own. +**Why it matters:** when HAL retires, the generators that keep `~/.ssh`, shell config and the +operator's `~/.config/hal` current retire with it. Without this, adding a node stops adding its ssh +alias and its trust, and a fresh machine has no operator dotfiles at all — the mesh would run every +service and leave the human unable to work on the box. -**Why not build it reflexively:** it is a real addition to the node model and the resource model, -and it must be gotten right, not smuggled in beside a firewall fix. Open questions to settle -first: +**Why not build it reflexively:** it is a real addition to the node model, the resource model, and +the seat set, and must be gotten right. The mechanism half is now settled — ADR 0120 is what lets +the ssh files be templates with no control-plane format — so what remains to decide here is the +model: - **One account or several per node?** A workstation has one human; a shared box might have more. - The model should allow more than one without forcing the common case to name it. -- **Where the login key lives.** An operator-owned file (ADR 0051) or an accepted secret — never - minted. The account fact and the key that authorises it are separate, and only the first is the - mesh's to generate. -- **The boundary with `sshd`.** The `sshd` module (server side) already exists. This is the - *client* and *identity* side: the account a node offers, and the home-scoped files an operator - needs. They meet at the account but are not the same module. -- **Multi-operator.** Today there is one human. The model should not assume it, but the first - cut may serve one and leave the shape open. + Allow more than one without forcing the common case to name it. +- **The CA's shape.** Host-cert and user-cert principals, cert lifetime and renewal, where the CA + runs (a seat on the control node). The one thing fixed: the operator's personal key is signed, + never minted. +- **Adoption of existing keys.** How an enrolling node's host key and an operator's existing key are + recorded and signed rather than replaced — the found-vs-owned boundary, made concrete for keys. +- **The `sshd` boundary.** Server side exists; this is the client, the identity, and the CA. +- **The ssh-agent.** An agent is a *user-scoped service running as the account* — the first concrete + case of the user services §2 anticipates. It holds the operator's private key in memory; the mesh + declares the unit and sets `AddKeysToAgent`/`IdentityAgent` in `config`, and still never sees the + private half. Agent *forwarding* wants a policy, not a default: with user certs it is largely + unnecessary, and forwarding an agent into a node exposes the operator's keys to that node's root — + so prefer certificates and `ProxyJump` over forwarding. -**Not urgent, not blocking.** ssh and dotfiles work today because HAL's generators still run as -the substrate. This becomes load-bearing in the node-by-node retirement phase, not before — which -is the right time to build it, once the account model is decided here. +**Not urgent, not blocking.** ssh and dotfiles work today because HAL's generators still run as the +substrate. This becomes load-bearing in the node-by-node retirement phase, not before — which is the +right time to build it, once the account and CA model are decided here. ## References - The gap was found generating `~/.ssh/config` from the *HAL* registry (`hal/terminal`'s postConfigure hook), which the nox mesh has no equivalent for. +- [ADR 0120](../../02-DECISIONS/0120-a-roster-fact-carries-its-format-as-a-template.md) — the roster + fact mechanism that renders the ssh files, format owned by the module. - [ADR 0112](../../02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md) — the system-path placement this mirrors for home paths. -- [ADR 0051](../../02-DECISIONS/0051-shared-data-is-the-operators.md) — why the login key stays - the operator's, never minted. +- [ADR 0051](../../02-DECISIONS/0051-shared-data-is-the-operators.md) — why the operator's personal + key is signed, never minted. +- [ADR 0113](../../02-DECISIONS/0113-the-vault-makes-every-secret.md) — the CA key is a secret the + vault makes; [ADR 0114](../../02-DECISIONS/0114-a-shared-credential-rotates-over-two-credentials.md) + — short-lived certs as rotation. +- [ADR 0117](../../02-DECISIONS/0117-a-machines-uplink-is-a-seat.md), + [ADR 0118](../../02-DECISIONS/0118-undeclaring-gives-a-unit-back-the-state-it-was-found-in.md) — + the never-sever-the-channel rule and the found-vs-owned semantics, applied here to `~/.ssh`. From 0f7f628730477a850e17341ef11582d34adc82ae Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 01:42:00 +0200 Subject: [PATCH 22/27] ADR 0120: note the shared/region interaction with hq 128 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A roster fact may be shared — written into a marked region of the machine's file (into: block, hq 128) rather than as the whole file. The template renders the content; shared decides how the host lays it down. Composes with hq 128: the region mechanism is the host's, the format is the module's. --- ...120-a-roster-fact-carries-its-format-as-a-template.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/02-DECISIONS/0120-a-roster-fact-carries-its-format-as-a-template.md b/02-DECISIONS/0120-a-roster-fact-carries-its-format-as-a-template.md index 8b28a97..c142cda 100644 --- a/02-DECISIONS/0120-a-roster-fact-carries-its-format-as-a-template.md +++ b/02-DECISIONS/0120-a-roster-fact-carries-its-format-as-a-template.md @@ -74,6 +74,15 @@ The two built-in projections render through the same path any module uses: - **dnsmasq's zones** move into dnsmasq. The `local=`/`address=` grammar is dnsmasq's configuration language, and it now lives in dnsmasq's manifest, where the module that runs dnsmasq owns it. +**A fact also says whether its file is the mesh's whole or a region of the machine's.** A hosts file +is the machine's — its `localhost`, the operator's lines, another tool's marked blocks — so +`node-names` is `shared`: the mesh owns only its region and keeps the rest byte for byte, the host +laying it down `into: block` +([ADR 0102](0102-the-mesh-writes-into-a-shared-file-never-over-it.md), hq issue 128). A resolver's +zones file is the mesh's whole, and is not shared. The template renders the content either way; +`shared` decides how the host writes it. This composes with hq 128 rather than replacing it: the +region *mechanism* is the host's, the region's *format* is the module's template. + **The names-vs-machines distinction is the template's choice** ([04-ISSUES/111](../04-ISSUES/111-the-resolver-is-told-names-the-mesh-serves-not-only-machines/00-report.md)): a container's hosts ranges `.Names`, so a routed name resolves to the machine serving it; a resolver told the mesh's suffix is its own ranges `.Machines`, or a routed name written there with the suffix From 8a6ee9177ce47a071947aa95f319b0c73b48dbaa Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 14:15:05 +0200 Subject: [PATCH 23/27] ADR 0121: a system seat is named for its scope; a module may define its own MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The control plane's seats grew a second naming style (the-*) beside mesh-*, and the closed set was the only place any seat could be defined. This settles both: system seats are mesh-* (one, mesh-wide) or node-* (one per node), named for scope; a module may define its own seat outside the closed set. Folds in the seat review: mesh-build-machine (scope fix), mesh-private-network (one server + client modules, dropping per-node VPN choice), showcase becomes the first module-defined seat, node-uplink, and the node-* renames — plus the registry consolidation onto gitea, which reshapes the registry seats and gates retiring distribution/verdaccio. Records why the renames are a coordinated migration and why distribution cannot be removed until gitea serves images. --- ...-its-scope-and-modules-define-their-own.md | 128 ++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 02-DECISIONS/0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md diff --git a/02-DECISIONS/0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md b/02-DECISIONS/0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md new file mode 100644 index 0000000..555db40 --- /dev/null +++ b/02-DECISIONS/0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md @@ -0,0 +1,128 @@ +--- +topic: what runs on it +status: accepted +date: 2026-09-27 +deciders: jochen +reconstructed: false +extends: 0110-a-seat-is-a-module-assignment-from-a-closed-set.md +--- + +# 121. A system seat is named for its scope, and a module may define its own + +## Context + +[ADR 0110](0110-a-seat-is-a-module-assignment-from-a-closed-set.md) made seats a closed set the +control plane defines: a well-formed name no longer becomes a seat by being claimed, so a person can +read what a mesh can have and who fills each role. It left two things unsettled that the growing set +now exposes: + +- **The names carry no rule.** `mesh-controller`, `mesh-store`, `mesh-broker` are named for the mesh; + beside them sit `the-artifact-store`, `the-build-machine`, `the-dns-port`, `the-showcase`, + `the-uplink` — a second naming style with no principle behind it. A reader cannot tell a seat's + scope from its name, and the mesh's own roles do not look like the mesh's. +- **The set is the *only* place a seat may be defined.** A module claiming any name not in the + control plane's set is refused. That is right for *system* roles — one broker, one packet filter + per node — but it means a module can never define a role of its own: a demo module's + `the-showcase`, a future application's coordination role, must be smuggled into the control plane's + set or not exist. The control plane ends up holding roles that are not the mesh's to define. + +Reviewing the set against these also found seats whose *scope* or *membership* is wrong, not just +their name — the review is the occasion to fix those too. + +## Decision + +**A system seat — one the control plane defines — is named for its scope:** + +- **`mesh-*`** for a mesh-scoped seat: one holder in the whole mesh, a role the mesh has once + (`mesh-controller`, `mesh-store`, `mesh-broker`, `mesh-git`, …). A `mesh-*` seat is always held by + a module **on a named node** — `mesh-git` is gitea *on novox*, not "gitea"; another node running + gitea does not hold `mesh-git` unless it is the holder. The seat is the mesh's single answer for + the role, and which node answers is part of what the seat records. +- **`node-*`** for a node-scoped seat: one holder per node, a role each machine has at most once + (`node-packet-filter`, `node-intrusion-prevention`, `node-uplink`, …). + +The three already-`mesh-*` seats keep their names; the rest are renamed by this rule. The scope a +name declares must match the seat's actual scope — a `mesh-*` seat at node scope, or the reverse, is +a contradiction the reader is entitled to trust is impossible. + +**The control plane defines only system seats. A module may define its own.** A seat named `mesh-*` +or `node-*` is the control plane's, and claiming one the control plane does not define is refused as +before. Any *other* name is a **module-defined seat**: valid when the module declaring the claim also +declares the seat (its name, scope, and — if any — the protocol its holder speaks). The control plane +enforces one-holder-per-scope for it exactly as for its own, but does not otherwise know what it +means. So an application can coordinate its own instances through a seat of its own, and the mesh's +closed set stays what its name says it is: the *system's* roles, not everyone's. + +**Specific seats this settles:** + +- **`the-build-machine` → `mesh-build-machine`, and its scope becomes mesh.** There is one build + machine in the mesh (the builder on novox), not one per node. Node scope said the opposite. It + delivers no provision; it is the mesh's single build machine. +- **`the-private-network` → `mesh-private-network`, held by the network *server* on one node.** Today + it is node-scoped and held on every node, with a stated (untested) story that a different VPN could + hold it per machine — which would force every provider module to independently implement receiving + and applying the controller-composed configuration. The mesh does not work that way and should not + pretend to: **one mesh decides one private network.** The seat is mesh-scoped, held by the server + module (WireGuard on the hub, novox). A machine that joins is given a **client module** that + receives the composed configuration and applies it; when a node joins, the mesh emits each node's + configuration so all of them know each other at once. This drops per-node VPN choice deliberately — + the private network is nox-mesh's own, and it defines the nodes' configuration rather than being + assembled from each node's opinion. (Implementation: the overlay generator's per-node computation + is unchanged; what changes is the seat's scope and the server/client split of the module.) +- **`the-showcase` → removed from the set; it becomes a module-defined seat.** It is a demo module's + own coordination role, claimed by nothing else and held nowhere. It is the first module-defined + seat, and the reason the rule above is needed rather than hypothetical. +- **`the-dns-port` → `node-dns-resolver`** (the daemon that binds `:53`), kept distinct from + **`the-resolver-configuration` → `node-resolver-config`** (what writes `resolv.conf`). Two roles, + two seats; the rename must not blur them. +- **`the-packet-filter` → `node-packet-filter`** and **`the-intrusion-prevention` → + `node-intrusion-prevention`** — names kept as-is but for the prefix. "Packet filter" stays distinct + from "firewall", which would swallow intrusion-prevention too. +- **`the-uplink` → `node-uplink`** ([ADR 0117](0117-a-machines-uplink-is-a-seat.md)). Unheld, so it + renames with no migration. +- **The registry seats — `the-artifact-store`, `npm-package-registry` (→ `mesh-artifact-store`, + `mesh-npm-package-registry`) — and `git` (→ `mesh-git`) — are decided but gated.** They are + entangled with consolidating every registry onto gitea (below), so their final shape is settled + when that lands, not renamed in isolation first. + +**The registry consolidates onto gitea.** The mesh should have **one** registry: gitea serving the +container/OCI images, the npm packages, crates, and trivial-tarball artifacts. `distribution` (the +standalone OCI registry) and `verdaccio` (a second npm registry) are retired once gitea serves what +each did. This is recorded here because it reshapes the registry seats; it is **not** a rename and +**not** surgical — see Consequences. + +## Consequences + +- **A reader learns a seat's scope from its name.** `mesh-*` is mesh-wide and one; `node-*` is + per-machine. The mesh's own roles finally look like the mesh's. +- **Applications get their own seats** without the control plane learning their meaning. The closed + set shrinks to what it should be — the system's roles — and stops being where unrelated roles hide. +- **The renames are a coordinated migration, not a rename.** A held seat's name lives in three places + that must move together: the control plane's set (`seats.go`), every claiming manifest, and what + each node reports it holds (re-derived by re-registering the manifest and re-pushing). A seat + renamed in one place and not the others stops resolving to its holder — and for a *delivering* seat + (`mesh-store`→postgres, `mesh-broker`→amqp, the registry seats) that is a mesh-wide provision + outage, the same failure mode as a schema change hitting an old manifest. So: the non-delivering + `node-*` seats and `mesh-build-machine` migrate as one tested controller+catalogue change; + `node-uplink` is free (unheld); the delivering registry seats wait for the gitea consolidation. +- **Retiring `distribution` is blocked until gitea serves images, and is the mesh's highest-risk + operation.** Every image — the control plane's own, the builder's, every module's — is + `artifact-store://…@sha256` served by `distribution`. gitea today provides only `npm-package-registry` + and `git`; it has no OCI registry. Removing `distribution` before gitea serves images strands every + image: nothing pulls, nothing reconciles, and the control plane cannot recover itself. The order is + fixed: stand up gitea's container registry → gitea `provides artifact-store` and holds the seat → + repoint the builder to push there → migrate or re-push existing images → **only then** retire + `distribution` and `verdaccio`. Recorded here so the sequence is not skipped. +- **The private network stops pretending to be swappable per node.** The gain is a coherent + server/client model matching how the controller already composes configuration; the cost is that + choosing a different VPN is now a mesh-wide change, not a per-node one — accepted. + +## References + +- [ADR 0110](0110-a-seat-is-a-module-assignment-from-a-closed-set.md) — the closed set this refines +- [ADR 0117](0117-a-machines-uplink-is-a-seat.md) — `the-uplink`, renamed here to `node-uplink` +- [ADR 0079](0079-the-foundation-seats-are-named-after-their-servers.md) — the original `mesh-*` seats + whose naming this generalises +- [to-be 26](../03-DESIGN/01-to-be/26-the-seats.md) — the seat table, updated by this +- mesh-controller `internal/catalogue/seats.go` (the set and claim validation), + `internal/overlay/generator.go` (the private network as server + client) From 1bb0ef5658a763a281f740f08b7ed1f2a68e7844 Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 14:36:33 +0200 Subject: [PATCH 24/27] ADR 0121: keep distribution, retire only verdaccio; node-* seats migrated Records the reversal: distribution stays as the mesh's OCI registry (it serves every artifact-store:// image); only verdaccio, a redundant second npm registry, is removed. The 'consolidate onto gitea / retire distribution' direction was dropped. Also records that the node-* rename was executed as one controlled migration with a brief compose freeze, and why the delivering registry seats are deferred rather than folded in. --- ...-its-scope-and-modules-define-their-own.md | 38 ++++++++++--------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/02-DECISIONS/0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md b/02-DECISIONS/0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md index 555db40..9a3ace4 100644 --- a/02-DECISIONS/0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md +++ b/02-DECISIONS/0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md @@ -81,15 +81,18 @@ closed set stays what its name says it is: the *system's* roles, not everyone's. - **`the-uplink` → `node-uplink`** ([ADR 0117](0117-a-machines-uplink-is-a-seat.md)). Unheld, so it renames with no migration. - **The registry seats — `the-artifact-store`, `npm-package-registry` (→ `mesh-artifact-store`, - `mesh-npm-package-registry`) — and `git` (→ `mesh-git`) — are decided but gated.** They are - entangled with consolidating every registry onto gitea (below), so their final shape is settled - when that lands, not renamed in isolation first. + `mesh-npm-package-registry`) — and `git` (→ `mesh-git`) — are decided but deferred.** They each + *deliver* a provision, so renaming them is a delivering-seat migration: a holder that stops + resolving mid-flight takes a provision away from every consumer. That risk is not worth carrying in + the same pass as the node-* renames, so they keep their names until done deliberately. -**The registry consolidates onto gitea.** The mesh should have **one** registry: gitea serving the -container/OCI images, the npm packages, crates, and trivial-tarball artifacts. `distribution` (the -standalone OCI registry) and `verdaccio` (a second npm registry) are retired once gitea serves what -each did. This is recorded here because it reshapes the registry seats; it is **not** a rename and -**not** surgical — see Consequences. +**`distribution` stays the mesh's registry; only `verdaccio` is retired.** An earlier draft of this +record had the registry consolidating onto gitea and `distribution` retired — that was reversed: +`distribution` is the standalone OCI registry serving every `artifact-store://…@sha256` image (the +control plane's own included), and the mesh keeps it. `verdaccio` was a *second* npm registry; +gitea already provides `npm-package-registry`, so verdaccio is redundant and is removed. It is only +in the catalogue (never registered in the running mesh), so removing it is deleting the module — no +migration, nothing to strand. ## Consequences @@ -104,15 +107,16 @@ each did. This is recorded here because it reshapes the registry seats; it is ** (`mesh-store`→postgres, `mesh-broker`→amqp, the registry seats) that is a mesh-wide provision outage, the same failure mode as a schema change hitting an old manifest. So: the non-delivering `node-*` seats and `mesh-build-machine` migrate as one tested controller+catalogue change; - `node-uplink` is free (unheld); the delivering registry seats wait for the gitea consolidation. -- **Retiring `distribution` is blocked until gitea serves images, and is the mesh's highest-risk - operation.** Every image — the control plane's own, the builder's, every module's — is - `artifact-store://…@sha256` served by `distribution`. gitea today provides only `npm-package-registry` - and `git`; it has no OCI registry. Removing `distribution` before gitea serves images strands every - image: nothing pulls, nothing reconciles, and the control plane cannot recover itself. The order is - fixed: stand up gitea's container registry → gitea `provides artifact-store` and holds the seat → - repoint the builder to push there → migrate or re-push existing images → **only then** retire - `distribution` and `verdaccio`. Recorded here so the sequence is not skipped. + `node-uplink` is free (unheld); the delivering registry seats are deferred to their own pass. +- **The node-* migration was done as one controlled step, and it froze briefly.** Deploying the new + controller made it reject the still-old-named claims in the stored manifests, so composition stopped + for the affected nodes until each manifest was re-registered under its new name; running services + were untouched, and the window was seconds. This is the coordinated-migration cost named above, + paid once — and the reason the *delivering* registry seats, whose freeze would be a provision + outage rather than a compose pause, are not folded into the same pass. +- **`distribution` is not retired.** It stays as the registry; only `verdaccio` (a redundant second + npm registry) is removed. The mesh keeps one OCI registry (`distribution`) and gitea for npm/git — + the "one registry, on gitea" idea was considered and dropped. - **The private network stops pretending to be swappable per node.** The gain is a coherent server/client model matching how the controller already composes configuration; the cost is that choosing a different VPN is now a mesh-wide change, not a per-node one — accepted. From b8cdfce16da5b3dc3b14fd891bb6631b852584c6 Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 14:49:02 +0200 Subject: [PATCH 25/27] to-be 30: the mesh updates itself on a push Records the manual update process (module moved -> build -> reconcile; and the breaking-change freeze/re-register recovery), and the two things that make self-update more than a webhook: the build-on-push trigger is currently HAL's (hal-gitea-tools on :9877), a retirement gap the mesh must replace with its own forge-webhook trigger wired to every repo including mesh-controller; and the builder validates manifests too, so a breaking change couples controller + builder + manifests + hosts, and renaming the builder's own seat deadlocks its rebuild. Names the transition discipline (accept old+new for one release) that self-update needs so a push does not auto-freeze. --- .../30-the-mesh-updates-itself-on-a-push.md | 137 ++++++++++++++++++ 1 file changed, 137 insertions(+) create mode 100644 03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md diff --git a/03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md b/03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md new file mode 100644 index 0000000..76d96d3 --- /dev/null +++ b/03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md @@ -0,0 +1,137 @@ +--- +layer: to-be +status: proposed +code: [] +updated: 2026-09-27 +decisions: + - 02-DECISIONS/0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md + - 02-DECISIONS/0120-a-roster-fact-carries-its-format-as-a-template.md +--- + +# 30 — The mesh updates itself on a push + +**Today the mesh does not update itself; a person drives the pipeline by hand, and one class of +change freezes it.** A code change lands in `mesh-controller` or `mesh-catalog`, and getting it onto +the machines is a sequence somebody types. The predecessor's pipelines rebuilt and redeployed on a +push without anyone watching; the successor should too. This records the process as it is done by +hand now — so it can be read, and then coded — and the two things that make it more than "add a +webhook". + +## The process, as done by hand + +**An ordinary (non-breaking) change** — new module code, a bug fix, a manifest tweak that changes no +seat or schema: + +1. `module moved ` — tell the mesh its source advanced (the controller repo has no + trigger, so this is manual; the catalogue's webhook does it automatically — see below). +2. `build --behind` (or `build [--ref] [--path ]`) — the build machine rebuilds and + records the new image. +3. The mesh **reconciles on its own**: the module's declaration now names the new image, the next + push/heartbeat sends it, and the host swaps the container. For the control plane this is a + self-upgrade — the running controller composes its own new image and the host replaces it. No + restart is typed. + +**A breaking change** — a manifest schema the controller parses differently (a fact's shape, a +seat's name), where the new control plane cannot read the manifests the old one stored: + +4. Land the code (controller + catalogue together — they are one change). +5. Rebuild + deploy the new controller (steps 1–3). **The moment it is live it refuses the + still-old-shape stored manifests, and composition freezes for every node that runs an affected + module.** Running services are untouched; only new declarations stop. +6. **Re-register each affected manifest under the new shape**, which the *new* controller accepts — + `module add -source -ref -commit `. This writes the manifest to the + store without a build, so it is the fast way to lift the freeze. (The controller container is + distroless: `docker cp` the file to the container root `/x.json`; `/tmp` does not exist; the + root filesystem is writable. The file is lost when the container is recreated on the next image + swap, so copy it *after* the swap.) +7. `push --behind`, then verify `status` is clean and `seats` (or the relevant surface) shows the + new shape held by the right holders. + +The freeze in a breaking change has been paid three times in one session (a fact-shape change, the +`/etc/hosts` region, a seat rename); each time it lasted seconds and no service dropped. It is +recoverable, but it is not something a push should trigger unwatched — which is the crux of what +automating this must solve. + +## Why it is more than "add a webhook" + +### 1. The trigger today is HAL's, not the mesh's + +Build-on-push works for the catalogue because its repository has a Gitea webhook pointing at +`http://host.docker.internal:9877/webhook/gitea` — and **that receiver is `hal-gitea-tools.service`** +(`~/.hal/modules/hal/gitea/tools/server.js`), a *predecessor* component. The nox builder consumes +build work; it does not receive Git events. So the mesh's own build pipeline currently rides on a +HAL service, and: + +- the `mesh-controller` repository was never wired to it, which is why the control plane is the one + thing that does **not** self-update — every controller deploy this session was `module moved` + + `build` by hand; +- when HAL is retired, build-on-push stops for the whole mesh. + +**The mesh needs its own forge-webhook→build trigger**, a nox component (a module, and likely a +seat — `mesh-forge-trigger` or folded into the git seat's holder) that receives Git events and turns +them into build work over the broker, for **every** repository including `mesh-controller`. Replacing +`hal-gitea-tools` is the concrete first build. Its logic already exists to copy: match the pushed +repository (and changed paths, for a monorepo like the catalogue) against the build-context +repository of every registered module, and rebuild the matches. + +### 2. The builder validates too — and a breaking change deadlocks it + +The build machine embeds the same catalogue package the controller does, so **it validates a +manifest against its own compiled-in seat/schema set**. A breaking change therefore couples *four* +things, not two: the controller, the **builder**, every affected manifest, and every node's host. +This session's seat rename rebuilt the controller but not the builder, and the stale builder then +refused every manifest claiming a renamed seat. + +Worse, one rename **deadlocked** the builder: the build machine's own seat was renamed +(`the-build-machine` → `mesh-build-machine`). To refresh the builder you must build it; to build it +the *running* (old) builder must accept the new builder's manifest — which claims the new name it +does not know. The old builder cannot build the new builder. Escapes: + +- **Never rename a seat whose holder validates manifests** in an ordinary pass — the build machine's + seat belongs with the deferred delivering seats (ADR 0121). Reverting `mesh-build-machine` to + `the-build-machine` (deferred) lets the old builder build the new builder, which then knows the + new names. +- Or bootstrap a new builder image **out of band** (build locally, publish to the registry, register + the module at that digest), the way genesis loads the first builder — bypassing the old builder's + validation once. + +Either way, self-update for breaking changes needs a **transition discipline** so a push does not +auto-freeze: the new control plane (and builder) should accept the *old and new* shape together for +one release — deprecated aliases in the seat set, a schema that reads both — then a later release +drops the old. With that, a breaking change rolls out on a push like any other: everything reads +both, the manifests migrate, the compatibility is removed. Without it, self-update would simply +automate the freeze. + +## What to build + +- **A nox forge-webhook trigger** (replaces `hal-gitea-tools`): receives Git events for every mesh + repository, dispatches build work to the builder over the broker, and records `module moved` + automatically. Wire `mesh-controller` to it so the control plane self-updates like everything else. +- **A transition discipline for breaking changes**: the control plane and builder accept old+new for + one release; the tooling that lands a schema/seat change emits the compatibility shim and the + follow-up that removes it. This is what makes step 4–7 above safe to trigger unwatched. +- **Config/package modules need no builder** — `module add` registers their manifest directly + (this is how the uplink managers and the re-registrations above were done). Only image-bearing + modules need the build machine, which narrows what the deadlock above can block. + +## Why now, and why not yet + +**Why it matters:** self-update is the difference between a mesh a person maintains by typing +pipeline steps and one that maintains itself, and it is a stated goal (parity with the predecessor's +pipelines). The HAL trigger dependency also makes it a retirement blocker: build-on-push dies with +HAL. + +**Why not reflexively:** the trigger is a new component with the broker and forge in its blast +radius, and the transition discipline changes how every breaking change is written. Both should be +designed, not bolted on beside a freeze. The manual process above is the interim, and it works. + +## References + +- [ADR 0121](../../02-DECISIONS/0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md) + — the seat rename whose migration and builder deadlock this record is drawn from +- [ADR 0120](../../02-DECISIONS/0120-a-roster-fact-carries-its-format-as-a-template.md) — the + fact-shape change that first showed the breaking-change freeze +- `hal-gitea-tools.service` (`~/.hal/modules/hal/gitea/tools/server.js`) — the predecessor webhook + receiver on `:9877` the mesh currently rides on +- mesh-controller `cmd/mesh-builder` (the build machine), `internal/catalogue` (the seat/schema + validation the builder shares with the controller) From a8921fe737a0a5d07555cf03f7ef0be2afe34518 Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 15:40:55 +0200 Subject: [PATCH 26/27] ADR 0122: a seat is data the controller owns; a rename is a database update MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reviews 0110/0121 after a session where renaming seats cost three freezes, a builder deadlock, and hand-resolved manifests. The seat rules were right; the set being a compiled Go slice referenced by name-string everywhere was the mistake. Seats become a table keyed by a stable id; claims/held/production code reference the id; a rename is one UPDATE, no rebuild, no re-registration, no freeze. The build machine reads the set from the mesh instead of embedding it, removing the controller/builder seat coupling. Closed set and scope naming unchanged; only storage and reference change. Outstanding renames (registry seats, private-network scope) wait for this — as data each is a write. --- ...t-is-data-a-rename-is-a-database-update.md | 107 ++++++++++++++++++ 1 file changed, 107 insertions(+) create mode 100644 02-DECISIONS/0122-a-seat-is-data-a-rename-is-a-database-update.md diff --git a/02-DECISIONS/0122-a-seat-is-data-a-rename-is-a-database-update.md b/02-DECISIONS/0122-a-seat-is-data-a-rename-is-a-database-update.md new file mode 100644 index 0000000..4d3eeb7 --- /dev/null +++ b/02-DECISIONS/0122-a-seat-is-data-a-rename-is-a-database-update.md @@ -0,0 +1,107 @@ +--- +topic: what runs on it +status: accepted +date: 2026-09-27 +deciders: jochen +reconstructed: false +supersedes-in-part: + - 0110-a-seat-is-a-module-assignment-from-a-closed-set.md + - 0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md +--- + +# 122. A seat is data the controller owns, and a rename is a database update + +## Context + +[ADR 0110](0110-a-seat-is-a-module-assignment-from-a-closed-set.md) made the seats a closed set the +control plane defines, and [ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md) +named them by scope. Both were right about *what* a seat is. Both left it defined the wrong *way*: +**the set is a hardcoded Go slice compiled into the controller, and everything references a seat by +its name as a string literal.** Renaming `the-packet-filter` to `node-packet-filter` this session +took, in one pass: + +- an edit to the Go slice in `internal/catalogue/seats.go`, recompiled into a new controller image; +- an edit to a `const gitSeat = "git"` in *production* control-plane code (`source.go`), because a + seat's name was hardcoded where a repository's home is resolved; +- edits to every claiming manifest in the catalogue, each re-registered; +- a controller **rebuild and redeploy**, which — because the running controller then refused the + still-old-named claims in stored manifests — **froze composition** for the affected nodes until + each manifest was re-registered under its new name; +- the same coupling in the **build machine**, which embeds the same seat set and refused to build + anything claiming a name it did not yet know; +- a **deadlock** when the build machine's own seat was renamed, since the old builder could not + build the new builder whose manifest claimed a name it rejected. + +None of that is what a rename should cost. A rename is the operator changing a label. It should be a +single write, and nothing should have to be rebuilt, refused, or unfrozen. The set being *closed* +(0110) and *named by scope* (0121) are good rules; **the set being code is the mistake.** When +adhering to the design means twenty steps and a `const` in the resolver, the design is what to fix. + +## Decision + +**The seat set is data the control plane owns, not code it is compiled from.** The seats live in a +table in the controller's store — one row per seat: a **stable id**, a `name`, a `scope`, what it +`delivers` (a provision, or nothing), and the record that decided it. The rows are seeded by a +migration (the closed set 0110 defines still ships with the mesh), and thereafter they are ordinary +data the control plane reads and writes. + +**A seat is referenced by its stable id, never by its name.** A claim, a held-seat record, and any +control-plane code that must name a seat (the git-seat resolver, the artifact-store guard) hold the +**id**. The `name` is a label for people and for what a manifest writes; it is resolved to an id +once, when a claim is registered. So: + +- **A rename is one `UPDATE seats set name = … where id = …`.** Nothing is recompiled, nothing is + re-registered, nothing is refused, nothing freezes. Held records and claims already point at the + id, so they follow the rename for free. The build machine is not involved, because the build + machine validates a claim against the set it reads from the mesh, not one baked into its image. +- **Adding or removing a seat is an `INSERT`/`DELETE`** (within the closed-set discipline: a change + to the set is still a decision with a record — the record is now a row's `decided` column and an + ADR, not a line of Go). No controller release is needed to change the roster of roles. +- **Production code stops hardcoding names.** `const gitSeat = "git"` becomes a lookup of the seat + that delivers the `git` provision (or a well-known id), so renaming its label cannot break the + code that finds a repository's forge. + +**What does not change** (0110 and 0121 still hold): a seat is still a module assignment from a +closed set; there is still one holder per scope; a delivering seat is still the single answer for +its provision; system seats are still `mesh-*`/`node-*` and a module may still define its own. Only +their *storage and reference* change — from a compiled slice keyed by name to a table keyed by id. + +**A manifest still claims by name, and that is fine.** A manifest is written by a person and names +the seat in words; the mesh resolves the name to an id at registration and stores the id. If a +seat's name changes, manifests written against the old name are updated in the catalogue like any +other edit (and the mesh can keep the old name as an alias row during a transition so nothing breaks +in the window) — but the *control plane* never has to change or redeploy for it, which is the whole +point. The heavy, mesh-wide, freeze-prone half of a rename disappears; only the ordinary catalogue +edit remains. + +## Consequences + +- **A rename, and a set change, become operations, not releases.** The pain this session paid — + three freezes, a builder deadlock, hand-resolved manifests — is designed out. The seat migrations + still outstanding (the delivering registry seats, and the private network's scope change) should + wait for this: done as data, each is a write, not a coupled multi-repo deploy. +- **The controller gains a small table and a seed migration**, and its seat lookups change from + slice scans to id-keyed reads. `SeatNamed`, `SeatDelivering`, `claimProblems` read the table. +- **The build machine reads the set from the mesh** (it already talks to the control plane), rather + than embedding it — which removes the controller/builder seat coupling that made every breaking + seat change a two-sided deadlock (see [to-be 30](../03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md)). +- **The closed set is still closed.** Data being editable is not the set being open: changing it is + still a decision, still recorded. What changes is that recording it no longer means shipping a + binary. +- **This is a real refactor**, touching the store schema, the seat lookups, claim registration + (name→id resolution), and the held-seat records. It is worth its own build; until it lands, the + current compiled set stands and further renames are held rather than forced through the heavy path. +- **Config on a seat is still the module's** (the question that surfaced this): a seat row carries + the seat's own metadata (scope, delivers, protocol), not a module's configuration — that stays in + the holding module's manifest ([ADR 0046](0046-a-module-configuration-is-its-assignments-not-its-manifest.md)). Making + seats data does not make them a config store. + +## References + +- [ADR 0110](0110-a-seat-is-a-module-assignment-from-a-closed-set.md), + [ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md) — the seat + rules this keeps, whose *storage* it changes +- [to-be 30](../03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md) — the controller/builder + seat coupling and the breaking-change freeze this removes for seat changes +- mesh-controller `internal/catalogue/seats.go` (the compiled slice this replaces), + `cmd/mesh-controller/source.go` (`const gitSeat`, the hardcoded name this removes) From 225dfa9451ae236cdfc511fcac10cb80c27ceefa Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 27 Sep 2026 16:59:56 +0200 Subject: [PATCH 27/27] to-be 31: a module declares its fail2ban jail, mesh composes them per node A node's intrusion filter should be composed from its assigned modules, like its firewall (the Filtering mechanism): a service module (postgres, mssql, mailu) declares its jail in its manifest (filter + stanza, no node/path per ADR 0112), and the mesh writes the jails of a node's modules into the fail2ban holder's jail.d. The base (sshd, recidive, ignoreip=mesh-range) stays the fail2ban module's. Records the model after novox's HAL per-module jails were lost as dangling symlinks; the ignoreip is now safe on disk, the service jails need this to be restored. --- .../31-a-module-declares-its-fail2ban-jail.md | 65 +++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100644 03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md diff --git a/03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md b/03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md new file mode 100644 index 0000000..e477693 --- /dev/null +++ b/03-DESIGN/01-to-be/31-a-module-declares-its-fail2ban-jail.md @@ -0,0 +1,65 @@ +--- +layer: to-be +status: proposed +code: [] +updated: 2026-09-27 +decisions: + - 02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md +--- + +# 31 — A module declares its fail2ban jail, and the mesh composes them per node + +**A node's intrusion filter should be composed from the modules it runs, the same way its firewall +is.** The mesh already derives a node's nftables ruleset from every assigned module's `listens` and +`guards` (the `Filtering` mechanism). fail2ban is the same shape and is not modelled: a module that +runs an authenticating service — postgres, mssql, mailu — has a jail (a filter that reads its log +and a jail stanza that bans on it), and which jails a node's fail2ban runs should be exactly the +jails of the modules assigned to that node. + +The predecessor did this with per-module files: `postgres` shipped `postgres-auth.conf`, `mssql` +shipped `mssql-auth.conf`, `mailu` shipped `mailu.conf`, and the node's fail2ban read whichever were +present. When HAL retired on novox those became dangling symlinks — fail2ban ran the jails only from +memory, and a restart would have dropped them. The base was salvaged (the fail2ban module now ships +`sshd`, `recidive`, and the `ignoreip` that spares the mesh's own range), but the **service jails +are gone**, because no nox module declares one yet. + +## The shape + +- **A module declares its jail in its manifest**, naming no node and no path (ADR 0112): the filter + (the failregex, or a stock filter it uses) and the jail stanza (port, logpath, maxretry, bantime). + The `postgres` module says what a postgres brute-force looks like and how to ban it; it does not + say on which machine, because it does not know. +- **The mesh composes them per node.** For each node, the jails of its assigned modules are gathered + and written into the fail2ban holder's `jail.d/` (and filters into `filter.d/`), exactly as + `listens`/`guards` are gathered into the node's firewall. So a node running postgres gets the + postgres jail; a node not running it does not. The `node-intrusion-prevention` holder receives + them the way a provider receives its consumers' contributions. +- **The base stays the fail2ban module's**: `sshd`, `recidive`, and the `ignoreip` naming + `${machine:mesh-range}` so a tunnel peer is never banned. + +## Why this, and not the module writing the file itself + +A module could declare a `file` resource at `/etc/fail2ban/jail.d/.conf` directly. Rejected: the +path is the fail2ban holder's to own (one module owns `jail.d`, as one module owns the firewall +table), the jail's logpath and defaults want the mesh's composition (the `ignoreip`, the ban action +the node uses), and two modules writing into one directory is the collision the seat/holder model +exists to prevent. The module declares *what its jail is*; the holder's composition decides *how it +lands* — the same split as `listens` (the module says the port; the mesh says the rule). + +## Why now + +fail2ban on novox currently runs the service jails from memory only; the next restart drops them +(the `ignoreip` is safe on disk, so the mesh-partition risk is closed, but postgres/mssql/mailu +auth-banning would be lost). This is the mechanism that restores them properly, and it is needed as +each of those modules migrates to the other nodes — ace running postgres should get the postgres +jail, composed from the postgres module's manifest, without anyone editing a node. + +## References + +- [ADR 0112](../../02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md) — a module + names no node or path; its jail is declared the same way its `listens` are +- mesh-controller `internal/catalogue/adoption.go` (`Filtering` — the firewall composition this + mirrors), `internal/catalogue/manifest.go` (`Listens`/`Guards`, the fields a jail field sits + beside) +- mesh-catalog `modules/fail2ban` (the base: sshd, recidive, ignoreip); the service modules + (`postgres`, `mssql`, `mailu`) that will declare jails