diff --git a/02-DECISIONS/0217-no-change-to-a-machine-takes-effect-unseen.md b/02-DECISIONS/0217-no-change-to-a-machine-takes-effect-unseen.md new file mode 100644 index 00000000..810941c7 --- /dev/null +++ b/02-DECISIONS/0217-no-change-to-a-machine-takes-effect-unseen.md @@ -0,0 +1,81 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-08 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0030-data-outlives-the-mesh-that-declared-it.md +--- + +# 217. No change to a machine takes effect unseen + +## Context + +Two incidents in two days had one shape: **a change took effect that nobody saw before it did.** + +- Issue 241: a provisioner read an unreadable grants file as "nobody asks" and withdrew every + consumer; the postgres provider dropped seven databases. Nothing announced the withdrawal before it + happened. +- Issue 304: one placement was set for one module on one machine with `settings set`. The layer is + replaced whole, so the module's other settings on that machine — three directory placements, eight + media accesses, an exposure, four endpoints, the account's ids — were dropped, and the command + answered only "places". The machine's plan then ran the module on an empty configuration + directory. It was caught by reading the plan before a push, and the lost layer was read back from a + database backup a few hours old. + +Since issue 241 the host no longer destroys what it did not make (ADR 0030 for directories, issue +241's fix for files) and no provider drops a store. What remains is the class above: the mesh doing +exactly what it was told, when what it was told was not what the person meant, and nothing showing +the difference until a machine had changed. + +## Considered Options + +1. **Confirm every push.** Rejected: every push would ask, and a confirmation asked every time is + answered without reading. The guard must speak only when something is at stake. +2. **Make `settings set` merge.** Rejected: removing a key would then need a second command, and the + layer stops being a statement of the whole (ADR 0046, ADR 0164). The fault is that the removal was silent, + not that it was possible. +3. **Three guards, each where the change is made, each silent when nothing is at stake.** Adopted. + +## Decision + +**A change that removes, moves or replaces something a machine is running is shown before it takes +effect; one that only adds is not interrupted.** + +1. **A settings layer is read before it is replaced, and what a replacement removes is said.** + `settings show` (and the verb) reads a layer. `settings set` answers with the keys it adds, changes + and removes, and **refuses to remove a key unless told `--replace`**. Every change keeps the layer + it replaced, so the previous value is one command away, not in a backup. +2. **A push says what it will change.** `plan --diff` lists what differs from what the machine + was last sent: resources added, changed and removed, and every container that will be recreated + with the reason. **`push` with no machine named is refused** unless `--all` says so. +3. **Moving a running module's data is acknowledged.** When a push would give a running container a + different host directory at a mount it already has — its data moving, or not following — the push + to that machine is held, naming the module, the mount and both directories; the other machines in + the same push go ahead. It is sent when the operator says so for that module (`--move `). + +None of the three adds a step to an ordinary change: a new module, a rebuilt image, a setting that +only adds keys and a push to a named machine behave exactly as before. + +## Consequences + +- **The controller keeps what it last sent each machine**, not only its digest, so a plan can be + compared with it. Migration 0014 declined to keep it — "a second account of what a machine should + be" — and that reason stands for what it was about: the copy is never read as what a machine + *should* be, only as what it *was told*, which is the one thing a comparison needs and the digest + cannot give. +- Settings carry a history; `settings show` reads it. +- A script that ran `push` with no machine must say `--all`. +- A changed placement for a running module needs one acknowledgement, once. + +## How it is checked + +Tests in the controller: `settings set` that drops a key without `--replace` is refused and changes +nothing; with it, the answer names the removed key and the history holds the previous layer. `plan +--diff` of an unchanged node is empty. A push changing a running container's mount source is held, +and goes ahead with `--move` for that module. `push` with no machine and no `--all` is refused. + +## References + +- [04-ISSUES/241](../04-ISSUES/241-one-unreadable-grants-file-dropped-every-database-on-the-control-node/00-report.md), [04-ISSUES/304](../04-ISSUES/304-setting-a-modules-settings-replaces-the-whole-layer-and-nothing-shows-it-first/00-report.md) +- ADR 0030 (data outlives its declaration), ADR 0046 (a module's configuration is its assignments), ADR 0164 (a setting is declared with its default and its cost), ADR 0214 (backups) diff --git a/03-DESIGN/01-to-be/44-no-change-takes-effect-unseen.md b/03-DESIGN/01-to-be/44-no-change-takes-effect-unseen.md new file mode 100644 index 00000000..7ab102fc --- /dev/null +++ b/03-DESIGN/01-to-be/44-no-change-takes-effect-unseen.md @@ -0,0 +1,71 @@ +--- +layer: to-be +status: in-progress +code: + - mesh-controller: cmd/mesh-controller/unseen.go, cmd/mesh-controller/push.go (--all, --move, the hold), cmd/mesh-controller/held_back.go (the hold in a named push's cascade), cmd/mesh-controller/plan.go (--diff), cmd/mesh-controller/modules.go (settings show, --replace), cmd/mesh-controller/seatverbs.go and internal/catalogue/verbs.go (the verbs' arguments), internal/inventory/unseen.go, internal/inventory/catalogue.go (the history kept on set and clear), internal/inventory/migrations/0078-what-a-change-replaces.sql +updated: 2026-10-08 +decisions: + - 02-DECISIONS/0217-no-change-to-a-machine-takes-effect-unseen.md + - 02-DECISIONS/0030-data-outlives-the-mesh-that-declared-it.md +--- + +# 44 — No change to a machine takes effect unseen + +**A change that removes, moves or replaces something a machine runs is shown before it takes effect; +a change that only adds goes through as before** (ADR 0217). Three guards, each at the place the +change is made. + +## 1. A settings layer is read before it is replaced + +A module's settings are layers — the whole mesh, and one per machine — each replaced whole when set. +That stays. Around it: + +- **Reading.** `settings show [--node]` prints the layers as they are, and the controller's `settings` verb + answers the same. +- **Saying what changed.** Setting a layer answers with the keys it adds, changes and removes, + compared with the layer it replaces. +- **Refusing a silent removal.** A set that would remove a key is refused, naming the keys, unless + `--replace` says the removal is meant. A set that only adds or changes keys needs nothing. +- **Keeping the previous layer.** Each set and each clear records the layer it replaced and when, so + the previous value is read back with `settings show --history`, not from a backup. + +## 2. A push says what it will change + +- **What was sent is kept.** Each send records a summary of the declaration it sent the machine, beside + the digest already kept: per resource its id, its type, a digest of it and of each field, and a + container's mounts — never a file's content, which may carry a secret and would land in the store's + backups. It is read only to compare; what a machine *should* be is still composed from the mesh's + records every time. +- **`plan --diff`** compares what would be sent now with what was sent last: resources added, + removed and changed by id, and for a changed container the fields that differ — which is what + recreates it. A machine with nothing to change shows nothing. +- **`push` with no machine is refused** unless `--all` names that intent. A push to one machine, or + `--behind`, is unchanged, and a push of the whole mesh still says so first and still leaves a machine + where a build waits for a gate. The controller's `push` verb with no machine is `--behind`, as before. +- **Beside what a push recreates.** A person's push already says, per machine, each module whose build + moves and what that recreates (ADR 0242); the comparison here is of the declaration, so it also shows + what a setting, an assignment or a placement changes, which no build does. +- **On the verbs.** The controller's `plan` verb takes `diff`, `settings` takes `replace` and `history`, + and `push` takes `move` beside a machine; given where it cannot take effect, each is refused rather + than passed over. + +## 3. Moving a running module's data is acknowledged + +Before sending, each machine's declaration is compared with what it was last sent. **A container +that keeps a mount at the same place inside it, with a different directory on the machine behind +it**, is a module's data moving — or, as on 2026-10-05, a module about to run on an empty directory +because a placement was lost. That machine's push is held, saying the module, the mount and both +directories; the other machines in the same push go ahead. The machines a named push sends after it, +because they are behind as its consequence, are held the same way, one by one. `push … --move ` +sends it. + +A first send to a machine, a new container, a mount added or removed, and a changed image are not +moves and are not held. + +## How it is checked + +Controller tests: a set dropping a key without `--replace` is refused and changes nothing, and with +it the answer names the removed key and the history holds the layer it replaced; `plan --diff` of an +unchanged machine is empty and names a changed container's changed field; a push changing a running +container's mount source is held for that machine only and goes with `--move`; `push` with no machine +and no `--all` is refused. diff --git a/04-ISSUES/304-setting-a-modules-settings-replaces-the-whole-layer-and-nothing-shows-it-first/00-report.md b/04-ISSUES/304-setting-a-modules-settings-replaces-the-whole-layer-and-nothing-shows-it-first/00-report.md index 8350bf57..e16d0867 100644 --- a/04-ISSUES/304-setting-a-modules-settings-replaces-the-whole-layer-and-nothing-shows-it-first/00-report.md +++ b/04-ISSUES/304-setting-a-modules-settings-replaces-the-whole-layer-and-nothing-shows-it-first/00-report.md @@ -1,9 +1,9 @@ --- -status: open +status: located opened: 2026-10-05 located-in: [mesh-controller cmd/mesh-controller (settings), mesh-controller internal/inventory (SetSettings)] -fixed-by: -amended-design: +fixed-by: novox/mesh-controller PR #50 +amended-design: 03-DESIGN/01-to-be/44-no-change-takes-effect-unseen.md --- # 304 — Setting a module's settings replaces the whole layer, and nothing shows it first @@ -41,5 +41,7 @@ around it is everything that makes it safe to use: ## Status -Open. Until it is fixed: read the layer (from the store, read-only) before setting it, and compare -the node's plan before and after. +Located: ADR 0217 decides the fix and to-be 44 designs it — `settings show` and its history, a set +that names what it adds, changes and removes and refuses a removal unless told `--replace`. Built in +mesh-controller PR #50; resolved when it runs. Until then: read the layer (from the store, +read-only) before setting it, and compare the node's plan before and after.