to-be 44: in progress — the three guards in the controller

Two incidents in two days had one shape: a change took effect that nobody saw first. Three guards
where the change is made — a settings layer read before it is replaced, a push that says what it
will change, a running module's data move acknowledged — each silent when nothing is at stake.

02-DECISIONS/0217-no-change-to-a-machine-takes-effect-unseen.md

@@ -0,0 +1,81 @@
+---
+topic: the mesh
+status: accepted
+date: 2026-10-05
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0030-data-outlives-the-mesh-that-declared-it.md
+---
+
+# 217. No change to a machine takes effect unseen
+
+## Context
+
+Two incidents in two days had one shape: **a change took effect that nobody saw before it did.**
+
+- Issue 241: a provisioner read an unreadable grants file as "nobody asks" and withdrew every
+  consumer; the postgres provider dropped seven databases. Nothing announced the withdrawal before it
+  happened.
+- Issue 246: one placement was set for one module on one machine with `settings set`. The layer is
+  replaced whole, so the module's other settings on that machine — three directory placements, eight
+  media accesses, an exposure, four endpoints, the account's ids — were dropped, and the command
+  answered only "places". The machine's plan then ran the module on an empty configuration
+  directory. It was caught by reading the plan before a push, and the lost layer was read back from a
+  database backup a few hours old.
+
+Since issue 241 the host no longer destroys what it did not make (ADR 0030 for directories, issue
+241's fix for files) and no provider drops a store. What remains is the class above: the mesh doing
+exactly what it was told, when what it was told was not what the person meant, and nothing showing
+the difference until a machine had changed.
+
+## Considered Options
+
+1. **Confirm every push.** Rejected: every push would ask, and a confirmation asked every time is
+   answered without reading. The guard must speak only when something is at stake.
+2. **Make `settings set` merge.** Rejected: removing a key would then need a second command, and the
+   layer stops being a statement of the whole (ADR 0046, ADR 0164). The fault is that the removal was silent,
+   not that it was possible.
+3. **Three guards, each where the change is made, each silent when nothing is at stake.** Adopted.
+
+## Decision
+
+**A change that removes, moves or replaces something a machine is running is shown before it takes
+effect; one that only adds is not interrupted.**
+
+1. **A settings layer is read before it is replaced, and what a replacement removes is said.**
+   `settings show` (and the verb) reads a layer. `settings set` answers with the keys it adds, changes
+   and removes, and **refuses to remove a key unless told `--replace`**. Every change keeps the layer
+   it replaced, so the previous value is one command away, not in a backup.
+2. **A push says what it will change.** `plan <node> --diff` lists what differs from what the machine
+   was last sent: resources added, changed and removed, and every container that will be recreated
+   with the reason. **`push` with no machine named is refused** unless `--all` says so.
+3. **Moving a running module's data is acknowledged.** When a push would give a running container a
+   different host directory at a mount it already has — its data moving, or not following — the push
+   to that machine is held, naming the module, the mount and both directories; the other machines in
+   the same push go ahead. It is sent when the operator says so for that module (`--move <module>`).
+
+None of the three adds a step to an ordinary change: a new module, a rebuilt image, a setting that
+only adds keys and a push to a named machine behave exactly as before.
+
+## Consequences
+
+- **The controller keeps what it last sent each machine**, not only its digest, so a plan can be
+  compared with it. Migration 0014 declined to keep it — "a second account of what a machine should
+  be" — and that reason stands for what it was about: the copy is never read as what a machine
+  *should* be, only as what it *was told*, which is the one thing a comparison needs and the digest
+  cannot give.
+- Settings carry a history; `settings show` reads it.
+- A script that ran `push` with no machine must say `--all`.
+- A changed placement for a running module needs one acknowledgement, once.
+
+## How it is checked
+
+Tests in the controller: `settings set` that drops a key without `--replace` is refused and changes
+nothing; with it, the answer names the removed key and the history holds the previous layer. `plan
+--diff` of an unchanged node is empty. A push changing a running container's mount source is held,
+and goes ahead with `--move` for that module. `push` with no machine and no `--all` is refused.
+
+## References
+
+- [04-ISSUES/241](../04-ISSUES/241-one-unreadable-grants-file-dropped-every-database-on-the-control-node/00-report.md), [04-ISSUES/246](../04-ISSUES/246-setting-a-modules-settings-replaces-the-whole-layer-and-nothing-shows-it-first/00-report.md)
+- ADR 0030 (data outlives its declaration), ADR 0046 (a module's configuration is its assignments), ADR 0164 (a setting is declared with its default and its cost), ADR 0214 (backups)

02-DECISIONS/README.md

@@ -194,6 +194,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0207** — [A module depends on the node seats that apply its resources](0207-a-module-depends-on-the-node-seats-that-apply-its-resources.md)
 - **0210** — [A tool's configuration is its seat holder's, and every other module extends it through the seat](0210-a-tools-configuration-is-its-seat-holders-and-every-other-module-extends-it-through-the-seat.md)
 - **0212** — [A seat says what it receives, and the machine's hotkeys are a seat](0212-a-seat-says-what-it-receives-and-the-machines-hotkeys-are-a-seat.md)
+- **0217** — [No change to a machine takes effect unseen](0217-no-change-to-a-machine-takes-effect-unseen.md)
 
 ### Its tiers, from the bottom up
 

03-DESIGN/01-to-be/44-no-change-takes-effect-unseen.md

@@ -0,0 +1,60 @@
+---
+layer: to-be
+status: in-progress
+code:
+  - mesh-controller: cmd/mesh-controller/unseen.go, cmd/mesh-controller/push.go (--all, --move, the hold), cmd/mesh-controller/plan.go (--diff), cmd/mesh-controller/modules.go (settings show, --replace), internal/inventory/unseen.go, internal/inventory/migrations/0057-what-a-change-replaces.sql
+updated: 2026-10-05
+decisions:
+  - 02-DECISIONS/0217-no-change-to-a-machine-takes-effect-unseen.md
+  - 02-DECISIONS/0030-data-outlives-the-mesh-that-declared-it.md
+---
+
+# 44 — No change to a machine takes effect unseen
+
+**A change that removes, moves or replaces something a machine runs is shown before it takes effect;
+a change that only adds goes through as before** (ADR 0217). Three guards, each at the place the
+change is made.
+
+## 1. A settings layer is read before it is replaced
+
+A module's settings are layers — the whole mesh, and one per machine — each replaced whole when set.
+That stays. Around it:
+
+- **Reading.** `settings show <module> [--node]` prints the layers as they are, and the console verb
+  answers the same.
+- **Saying what changed.** Setting a layer answers with the keys it adds, changes and removes,
+  compared with the layer it replaces.
+- **Refusing a silent removal.** A set that would remove a key is refused, naming the keys, unless
+  `--replace` says the removal is meant. A set that only adds or changes keys needs nothing.
+- **Keeping the previous layer.** Each set and each clear records the layer it replaced and when, so
+  the previous value is read back with `settings show --history`, not from a backup.
+
+## 2. A push says what it will change
+
+- **What was sent is kept.** Each send records the declaration it sent the machine, beside the digest
+  already kept. It is read only to compare; what a machine *should* be is still composed from the
+  mesh's records every time.
+- **`plan <node> --diff`** compares what would be sent now with what was sent last: resources added,
+  removed and changed by id, and for a changed container the fields that differ — which is what
+  recreates it. A machine with nothing to change shows nothing.
+- **`push` with no machine is refused** unless `--all` names that intent. A push to one machine, or
+  `--behind`, is unchanged; the console verb never pushed every machine and still does not.
+
+## 3. Moving a running module's data is acknowledged
+
+Before sending, each machine's declaration is compared with what it was last sent. **A container
+that keeps a mount at the same place inside it, with a different directory on the machine behind
+it**, is a module's data moving — or, as on 2026-10-05, a module about to run on an empty directory
+because a placement was lost. That machine's push is held, saying the module, the mount and both
+directories; the other machines in the same push go ahead. `push … --move <module>` sends it.
+
+A first send to a machine, a new container, a mount added or removed, and a changed image are not
+moves and are not held.
+
+## How it is checked
+
+Controller tests: a set dropping a key without `--replace` is refused and changes nothing, and with
+it the answer names the removed key and the history holds the layer it replaced; `plan --diff` of an
+unchanged machine is empty and names a changed container's changed field; a push changing a running
+container's mount source is held for that machine only and goes with `--move`; `push` with no machine
+and no `--all` is refused.

04-ISSUES/240-a-dry-run-build-is-recorded-and-rolled-out/00-report.md

@@ -1,8 +1,8 @@
 ---
-status: open
+status: resolved
 opened: 2026-10-04
-located-in: []
-fixed-by:
+located-in: [mesh-controller cmd/mesh-controller/main.go, mesh-controller cmd/mesh-builder, mesh-controller internal/link/build.go]
+fixed-by: mesh-controller#47
 amended-design:
 ---
 
@@ -35,3 +35,11 @@ acts on it, review becomes a formality: the unreviewed definition reaches a mach
 1. Where does the dry run's outcome enter the record — the build machine's `built` event, consumed as
    any other build's?
 2. Did the controller roll the dry run out, or did a later push compose from it?
+
+## Resolution (2026-10-05)
+
+A dry run is marked on the request (`DryRun`), the builder echoes the mark on its outcome, and the
+controller's daemon sets a marked outcome aside: no record, no registration, no plan, nothing a push
+could send (mesh-controller#47, with a test that the daemon takes a dry run in with no store at all).
+Left open as a follow-up: the catalogue module also hears build outcomes and records their edges; it
+should skip a dry run too.

04-ISSUES/242-the-mesh-has-no-backups/00-report.md

@@ -1,8 +1,8 @@
 ---
-status: open
+status: located
 opened: 2026-10-05
-located-in: []
-fixed-by:
+located-in: [mesh-controller internal/catalogue/seats.go, mesh-catalog modules/restic]
+fixed-by: mesh-controller#49, mesh-catalog#49, mesh-catalog#54, mesh-catalog#55, mesh-catalog#56, mesh-media-catalog#1
 amended-design: 03-DESIGN/01-to-be/43-backups-against-mistakes.md
 ---
 
@@ -47,3 +47,27 @@ research 030; proposed as ADR 0214.
 
 Issue 241's recovery cost a night and lost the forge's records of twelve days. With a nightly backup
 held on another machine, it would have been a ten-minute restore of yesterday.
+
+## Where it stands (2026-10-05)
+
+Backups run (ADR 0214, to-be 43): the control node keeps nightly restore points of its eight stores
+and services, the home server of its databases and its media apps' libraries and cover art; each on
+its own machine, on its larger filesystem. The first runs were tried one module, then two, then a
+whole node, each proven by a restore beside the live data. Not yet built, which is why this stays
+open: the weekly test restore into a throwaway instance, the 48-hour status line, and failures
+reaching the operator rather than the holder's log.
+
+What the rollout taught, for the next module that takes contributions:
+
+- **A contribution makes the contributing module depend on the seat.** Merging the stores'
+  contributions before a holder was assigned left every node's plan unresolvable — the whole mesh,
+  not only the machines running a store — until the holder was assigned. Assign the holder in the
+  same step as the merge.
+- **A brand-new module is not built by the push that adds it**; its first build is asked for by hand.
+- **The holder must look as the account that can see.** Its first run called a store's dumps missing
+  because it checked as the runtime's account, which cannot see inside the store's own directory.
+- **A kept single file is not a directory.** restic restores a snapshot's subfolder, not a file;
+  the first restore of a module keeping its settings file refused it.
+- **An update kills a running backup** — the holder's restic is its child. A hand-off to the new
+  version, the run living on as its own unit under the machine's service manager, is the idea to
+  take forward.

04-ISSUES/245-a-media-servers-previews-were-reached-through-a-link-its-container-never-mounted/00-report.md

@@ -0,0 +1,39 @@
+---
+status: resolved
+opened: 2026-10-05
+located-in: [mesh-media-catalog modules/plex]
+fixed-by: mesh-media-catalog#2
+amended-design:
+---
+
+# 245 — A media server's previews were reached through a link its container never mounted
+
+## What was observed
+
+On the home server the media server's scrub previews — 423 GB, about 57 000 preview files, one per
+video — had not grown in seven months: the newest was from the day its preview folder was moved off
+the server's own disk onto the large storage pool. The server's log repeated, live, that it could not
+create a directory under its preview folder.
+
+## Why
+
+The move left the preview folder as a **symbolic link** inside the server's configuration directory,
+pointing at the pool. The container mounts the configuration directory and the media libraries, and
+not the pool's path, so inside the container the link pointed at nothing: the server could neither
+show the previews it had nor make new ones, and said so only in its own log. Nothing the mesh reports
+showed it — the container ran, and answered.
+
+It is the failure the mesh's rule against symbolic links exists for: a link resolves differently in
+every place that reads it, and a container is such a place.
+
+## Resolution
+
+The previews are a directory of the module, mounted at the server's preview path; where that
+directory lives on a machine is the machine's placement setting, here the pool path the files were
+already in. The link was removed at the cutover, the container recreated, the preview folders visible
+inside it again and the log's errors gone. The previews are not backed up, by the operator's choice.
+
+## How it is checked
+
+The catalogue check that every mount is declared passes over the module. On the machine: the preview
+folder seen from inside the container lists the same folders as the pool path.

04-ISSUES/246-setting-a-modules-settings-replaces-the-whole-layer-and-nothing-shows-it-first/00-report.md

@@ -0,0 +1,45 @@
+---
+status: open
+opened: 2026-10-05
+located-in: [mesh-controller cmd/mesh-controller (settings), mesh-controller internal/inventory (SetSettings)]
+fixed-by:
+amended-design:
+---
+
+# 246 — Setting a module's settings replaces the whole layer, and nothing shows it first
+
+## What was observed
+
+An operator's agent set one placement — where a media server's preview folder lives on one machine —
+with `settings set <module> {"places": {…}} --node <machine>`. The command answered that the setting
+was recorded. The machine's plan then mounted the server's configuration from an empty default
+directory: the node's layer had held the placements of three other directories, eight media
+accesses, a public exposure, four endpoints and the account's ids, and every one of them was gone.
+
+Caught before any push, by reading the plan. The previous layer was read back from the controller
+database's nightly dump — the backups that issue 242 asked for, a few hours old.
+
+## Why
+
+The layer is a statement of the whole, by design (the inventory's `SetSettings`: "replacing rather
+than merging … removing a key is done by leaving it out"). That design is sound; what is missing
+around it is everything that makes it safe to use:
+
+- **There is no way to read a layer.** `settings` has `set` and `clear`, no `show`; the console verb
+  likewise. To change one key, a person must already know every other key in the layer.
+- **There is no history.** The row is updated in place; the previous values exist nowhere but a
+  database backup.
+- **The answer does not say what was dropped.** "places" was reported as set; the six keys removed
+  were not mentioned.
+
+## What would fix it
+
+1. A way to read a layer — `settings show <module> [--node]` and the same on the verb.
+2. `set` answers with what changed: keys added, changed and **removed**, so dropping one is never
+   silent. A removal could even require saying so.
+3. The previous value kept: a settings history row per change, so an undo needs no backup.
+
+## Status
+
+Open. Until it is fixed: read the layer (from the store, read-only) before setting it, and compare
+the node's plan before and after.