ADR 0189: the store keeps what the records name, and a maintenance step holds its writers still

Issue 108: the artifact store has never collected anything. Fifty-three repositories on the machine that serves everything else, and the only outcome of leaving it is a full disk reported as somebody else's failure. The mesh decides what may go — from its own build records, so it never names a digest it did not put there — and the store reclaims the bytes in a nightly window with its server held still. Deletion on the one door takes nothing a push did not already have. Designs 18 and 20 amended; issue 108 resolved. Also issue 202, found running the controller's suite: a module whose required setting nobody set is left out of the machine in silence, and dnsmasq became that module this morning.
2026-10-02 21:48:48 +02:00
6 changed files with 298 additions and 6 deletions
@@ -0,0 +1,131 @@
+---
+topic: the mesh
+status: accepted
+date: 2026-10-02
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md
+---
+
+# 189. The store keeps what the records name, and a maintenance step holds its writers still
+
+## Context
+
+The mesh's artifact store has never collected anything
+([issue 108](../04-ISSUES/108-the-registry-has-no-garbage-collection-once-it-has-two-doors/00-report.md)).
+Every build pushes another layer set; nothing has ever removed one. The predecessor ran a routine
+on a timer — stop the registry, collect, start it — and the conversion carried the settings that
+routine depends on without the routine, because the routine was a script beside the module and not
+a resource in it. The store now holds fifty-three repositories on the machine that serves
+everything else, and the only outcome of leaving it is a full disk reported as somebody else's
+failure.
+
+Three things stood in the way, and the issue names all three.
+
+**Nothing in the mesh's vocabulary expresses a maintenance window.** The collector requires every
+writer stopped while it runs. A `run-once` step runs *beside* containers, not instead of them, and
+a scheduled step is the same container on a cadence. There is no way for a module to say *hold this
+container of mine still while this runs*.
+
+**Deletion is not enabled, and the door it would be enabled on has no accounts.** The store is
+internal, reached by name over the overlay, trusted because being on that network is the permission
+([ADR 0082](0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md)). The predecessor
+kept deletion behind an authenticated door, which it could, having one.
+
+**Nothing says what may be removed.** The registry's own answer — collect everything no tag names —
+is wrong here. The mesh pushes each artifact under one moving tag and pins machines by digest, so
+every build but the newest is untagged and some machine may still be running it.
+
+## Decision
+
+**1. Deletion is enabled on the store's one door, and the overlay stays the permission.** The
+objection dissolves on inspection: that door **already accepts a push**, and a writer who can push
+can replace any tag in the store with anything it likes. Delete takes nothing a push did not
+already have, and the machines that can reach the door are the ones the mesh's own filter admits
+([ADR 0168](0168-a-converged-machine-is-filtered-by-the-mesh-alone.md)). Putting an authenticated
+door in front of deletion while leaving push open would be a lock on the window beside an open
+door, and it would cost the thing ADR 0082 bought: a store every machine can reach without a
+credential to distribute first.
+
+**2. The mesh deletes what it made and no longer keeps; the store reclaims the bytes.** Two halves,
+each doing what only it can.
+
+The **mesh** decides. It does not need to enumerate the store to do it — it has never put anything
+there it did not record, so **every digest it could remove is already in its own build records**.
+It deletes those manifests through the store's door, by digest, and remembers that it did.
+
+The **store** reclaims. A deleted manifest frees no bytes until the registry's own collector walks
+the storage with nothing writing to it, so the module declares that collector as a scheduled step
+with the server held still for its duration. Plain collection, not `--delete-untagged`: what the
+mesh keeps is still a manifest in the store, so it is still referenced, so its blobs stay — the
+dangerous flag is not needed at all once the mesh is the one deciding.
+
+**3. What the mesh keeps, stated as three reasons rather than a number.** A digest is kept because:
+
+- **a definition names it** — every artifact reference in any module's current recorded manifest,
+  which is what the mesh would hand a machine now. No age limit: this is the floor;
+- **the mesh can still go back to it** — every artifact of the **five most recent successful
+  builds** of each module, so a release that turns out wrong has somewhere to return to;
+- **nothing else.** An artifact older than that, which no definition names, is what the store is
+  carrying for no stated reason.
+
+A digest the mesh did not record making is never touched. That is not a safety margin, it is the
+whole rule restated: the mesh removes what it put there and can account for, and the images genesis
+pushed before any record existed are exactly what this must not reach
+([04-ISSUES/102](../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md), F4).
+
+**4. A scheduled step may hold its module's own containers still while it runs** —
+`while-stopped`, naming resource ids in the same module. The host stops each, runs the step, and
+starts them again **whatever the step did**, including when it failed or the host was interrupted.
+Three boundaries:
+
+- **Its own module's containers only.** A module that could quiesce a neighbour could stop the
+  mesh; a maintenance window is a statement about one service's own insides.
+- **Scheduled steps only, not `run-once`.** At apply time the host already has a window: the
+  declaration is applied in order and a step gates what follows, so a one-time offline migration
+  says *before* rather than *instead of*. A recurring window is the case order cannot express.
+- **Restoring is not conditional.** A step that fails must leave the service running; the whole
+  risk of this field is a window that never closes.
+
+**5. The sweep runs where the records change — after a build the mesh recorded.** That is the
+moment new bytes landed and the moment the keep set moved, and it needs no new timer. The
+store's collection runs nightly, because reclaiming is slow and the thing it reclaims is already
+unreferenced.
+
+## Consequences
+
+- Disk stops growing without bound on the machine that serves the mesh. That is the whole point
+  and it has no other way to be true.
+- A machine behind by more than five builds of a module, which recreates a container, cannot pull
+  what it was running. It is already a machine the mesh reports as behind, and the answer is the
+  one the mesh already gives it: the current declaration. Stated here rather than discovered.
+- The store is a little less of a museum. A digest in an old build record may no longer be
+  fetchable, and the record still says what that build made — the record is history, not an
+  index of what is on disk. The collected mark is kept beside it so the two can be told apart.
+- `while-stopped` is a second thing the host does to a container it did not start this pass. It is
+  deliberately the narrowest form: the module's own, by id, restored unconditionally.
+- The store is briefly unavailable each night, for as long as collection takes. Everything that
+  pulls from it retries; nothing in the mesh treats a momentary store as a failure
+  ([ADR 0185](0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md)).
+
+## How this is checked
+
+- The host: a scheduled step with `while-stopped` stops the named containers before the run and
+  starts them after; it starts them again **when the step fails**; it refuses an id that is not a
+  container of the same module, its own id, and `while-stopped` on a `run-once` step. Each refusal
+  is tested for what it says, not only that it says something.
+- The controller: given build records and current manifests, the keep set holds every reference a
+  manifest names and every reference of the five most recent builds per module, and nothing else;
+  a reference the mesh never recorded is never in the delete set; a delete that answers 404 is
+  recorded as collected rather than retried forever.
+- The sweep is tested against a fake store that records what it was asked to delete, so what is
+  asserted is the decision and not the registry's behaviour.
+- Live: the store's size before and after the first nightly collection, read from the machine.
+
+## References
+
+- [issue 108 — the registry has no garbage collection](../04-ISSUES/108-the-registry-has-no-garbage-collection-once-it-has-two-doors/00-report.md)
+- [ADR 0082 — the registry is reached by name and trusted by the overlay](0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md)
+- [ADR 0053 — a step that runs on a schedule](0053-a-step-that-runs-on-a-schedule.md)
+- [ADR 0156 — an artifact is what a build produces, and the store is named for its scope](0156-an-artifact-is-what-a-build-produces-and-the-store-is-named-for-its-scope.md)
+- [design 32 — what a module declares](../03-DESIGN/01-to-be/32-what-a-module-declares.md)
@@ -189,6 +189,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0186** — [A ban list never holds a neighbour, and the mesh's own bans are its own wherever they hang](0186-a-ban-list-never-holds-a-neighbour.md)
 - **0187** — [A dead tracker is not the machine's failure](0187-a-dead-tracker-is-not-the-machines-failure.md)
 - **0188** — [A provider declares what it derives for each consumer, and the mesh tells both ends](0188-a-provider-declares-what-it-derives-for-each-consumer.md)
+- **0189** — [The store keeps what the records name, and a maintenance step holds its writers still](0189-the-store-keeps-what-the-records-name.md)

 ### Its tiers, from the bottom up

@@ -5,9 +5,10 @@ code:
  - mesh-controller cmd/mesh-builder
  - mesh-controller internal/builder
  - mesh-catalog modules/builder
-updated: 2026-10-01
+updated: 2026-10-02
 decisions:
  - 02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md
+  - 02-DECISIONS/0189-the-store-keeps-what-the-records-name.md
  - 02-DECISIONS/0150-a-modules-own-code-runs-as-supervised-processes-under-one-account.md
  - 02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md
  - 02-DECISIONS/0111-a-build-source-is-on-the-git-seat-or-external.md
@@ -293,6 +294,44 @@ build's lines reach a reader of its subject in order and the stream holds them a
 against a real server); the seat verb with an id reads the log (controller test); and, live, a build
 after the roll-out read line by line through the console.

+## The store keeps what the records name
+
+*2026-10-02 — [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md),
+[issue 108](../../04-ISSUES/108-the-registry-has-no-garbage-collection-once-it-has-two-doors/00-report.md).*
+
+Every build pushes another layer set and, until this, nothing ever removed one. The registry's own
+answer — collect what no tag names — is wrong for this mesh: each artifact is pushed under one
+moving tag and machines are pinned by digest, so every build but the newest is untagged and some
+machine may still be running it.
+
+**The mesh decides and the store reclaims.** Deletion is enabled on the store's one door — that
+door already accepts a push, and a writer who can push can replace any tag, so delete takes
+nothing a push did not already have, and ADR 0082's bargain (a store every machine reaches with no
+credential to distribute first) is kept. The mesh then removes what it put there and no longer
+keeps, **naming it from its own build records** rather than enumerating the store: it has never
+put anything there it did not record, so a digest it did not record making is never named, which
+is what keeps the sweep away from the images genesis pushed before any record existed.
+
+An artifact stays for one of two reasons and otherwise goes: a definition the mesh holds names it
+(no age limit — this is the floor), or it belongs to one of the five most recent successful builds
+of its module (somewhere for a wrong release to return to). The sweep runs after a build the mesh
+recorded, which is the moment new bytes landed and the moment the keep set moved; it needs no
+timer. Deleting a manifest frees no bytes, so the store's own collector runs nightly as a
+scheduled step with the server held still — which is what `while-stopped` exists for
+([design 20](20-writing-a-module.md)). Plain collection, not `--delete-untagged`: what the mesh
+keeps is still a manifest and so still referenced, and the dangerous flag is not needed once the
+mesh is the one deciding.
+
+A machine behind by more than five builds of a module, recreating a container, cannot pull what it
+was running. It is already a machine the mesh reports as behind, and the answer is the current
+declaration.
+
+*How it is checked:* the keep set, against records, holds what a manifest names and the five most
+recent builds and nothing else; a reference the mesh never recorded is never in the delete set; an
+image and an archive are asked for at their own endpoints; a store with deletion off names the
+remedy rather than the status code; a store that does not have it is recorded collected rather
+than retried for ever. Live: the store's size before and after the first nightly collection.
+
 ## The builder compiles the languages the mesh is written in

 *2026-09-29 —
@@ -5,11 +5,12 @@ code:
  - mesh-catalog modules/showcase
  - mesh-controller internal/builder
  - mesh-sdk src
-updated: 2026-09-30
+updated: 2026-10-02
 decisions:
  - 02-DECISIONS/0150-a-modules-own-code-runs-as-supervised-processes-under-one-account.md
  - 02-DECISIONS/0099-a-step-that-runs-once-names-what-it-reads.md
  - 02-DECISIONS/0053-a-step-that-runs-on-a-schedule.md
+  - 02-DECISIONS/0189-the-store-keeps-what-the-records-name.md
  - 02-DECISIONS/0074-the-wire-is-specified-not-the-types.md
  - 02-DECISIONS/0040-what-a-module-is.md
  - 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md
@@ -208,3 +209,27 @@ is recreated with the new fact
 ([ADR 0099](../../02-DECISIONS/0099-a-step-that-runs-once-names-what-it-reads.md)). *How it is
 checked:* the host's unit tests run a step again when its named file changed and not otherwise,
 and recreate a container naming a step after the step ran.
+
+## A recurring step may hold its own module's containers still
+
+*Written 2026-10-02, from [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md)
+and [issue 108](../../04-ISSUES/108-the-registry-has-no-garbage-collection-once-it-has-two-doors/00-report.md).*
+
+Some work cannot be done underneath a running service: an artifact store's collector walks the
+storage and requires every writer stopped. A `run-once` step runs *beside* containers and a
+scheduled one is the same container again, so until this a module had no way to say it — and the
+mesh inherited a store that has never collected anything, because the predecessor said it with a
+shell script and a script beside a module is not a resource in it.
+
+A scheduled step may name `while-stopped`: resource ids of **its own module's** containers, which
+the host stops before the run and starts again after it, in the reverse order, **whatever the step
+did**. Three boundaries, each refused where it can be seen earliest — its own module's containers
+only, because a module that could quiesce a neighbour could stop the mesh; scheduled steps only,
+because at apply the declaration is applied in order and a step already gates what follows, so a
+one-time offline job says *before* rather than *instead of*; and restoring that is not conditional
+on anything, because the only real risk of the field is a window that never closes.
+
+*How it is checked:* the host's unit tests assert stop–run–start in that order, the restart after a
+step that **failed**, the reverse order for several containers, and a service left down said
+loudly. The controller refuses, from the definition alone, a window with no schedule, one on a
+run-once step, one naming a container the module does not declare, and one naming itself.
@@ -1,9 +1,9 @@
 ---
-status: open
+status: resolved
 opened: 2026-09-23
-located-in: []
-fixed-by:
-amended-design:
+located-in: [mesh-controller internal/inventory, mesh-controller internal/artifacts, mesh-host internal/apply, mesh-catalog modules/distribution]
+fixed-by: 02-DECISIONS/0189-the-store-keeps-what-the-records-name.md
+amended-design: 03-DESIGN/01-to-be/18-building-a-module.md
 ---

 # 108 — The registry has no garbage collection, and two doors make it harder to add
@@ -75,3 +75,32 @@ real thing services need, and the mesh cannot express one.
  images by digest and moves by version — is retention "the digests no recorded build names"?
 - Who owns the routine when the store and its public door are two modules — the store, since the
  volume is its?
+
+## Answered, 2026-10-02 — [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md)
+
+The three open questions, answered:
+
+- **A maintenance step, or a backend that does not need its writers stopped?** The step. A
+  scheduled container may name `while-stopped` — resource ids of **its own module's** containers,
+  which the host stops before the run and starts again after it whatever the step did. A storage
+  backend the mesh does not run would be a bigger thing to own than the mechanism it avoids, and
+  the mechanism is wanted anyway: a service that cannot have work done underneath it is a real
+  shape and the mesh could not express it at all.
+- **Is retention "the digests no recorded build names"?** Nearly. An artifact stays because a
+  definition the mesh holds names it (no age limit), or because it belongs to one of the five most
+  recent successful builds of its module. Last-N-tags was the predecessor's rule for a registry
+  that knew nothing else; this mesh knows what each digest is for.
+- **Who owns the routine now the second door is gone?** Both halves, each where it can be. The
+  **mesh** decides what may go — only it holds the records — and asks the store to drop it. The
+  **store** reclaims the bytes, because only it can stop its own server. Neither half can be done
+  by the other.
+
+And the sharpened point — enabling deletion on a door with no accounts — dissolved on inspection:
+**that door already accepts a push**, so a writer who can reach it can already replace any tag.
+Delete takes nothing a push did not have. What it does not do is undo ADR 0082's bargain, which
+putting an authenticated door in front of deletion would have.
+
+The second registry process is not built, as the 2026-09-26 note says, so the shared blob cache
+and the deletion-cached-by-the-other-door problem never arise. Plain `garbage-collect` is enough:
+what the mesh keeps is still a manifest in the store, so `--delete-untagged` — the flag that would
+delete images machines are running — is not needed at all.
@@ -0,0 +1,67 @@
+---
+status: open
+opened: 2026-10-02
+located-in: [mesh-catalog modules/dnsmasq, mesh-controller cmd/mesh-controller]
+fixed-by:
+amended-design:
+---
+
+# 202 — A module whose required setting nobody set is left out of the machine, and the resolver is the module it happened to
+
+## What was observed
+
+Running the controller's own test suite against the catalogue beside it, 2026-10-02.
+`TestTheResolverIsToldEveryMachineOnTheNetworkAndToldAgainWhenOneLeaves` fails with *"the resolver
+was not handed the machines"*. Composing the same machine by hand and listing what it receives
+shows why: **dnsmasq contributes nothing at all.** Four resources are composed for that node, all
+of them the overlay's. The resolver's package, its configuration, its service and the fact that
+carries every machine's name are simply not there.
+
+The cause is one line added to `dnsmasq`'s configuration earlier the same day: the addresses it
+listens on beside the machine's own became an operator setting,
+`listen-address=${setting:listen-addresses}`, with no default. A `${setting:…}` nothing sets is
+refused, a module that cannot be composed is **left out** rather than failing the whole machine
+([ADR 0163](../../02-DECISIONS/0163-taking-a-module-over-is-a-comparison.md)), and so a node
+assigned the resolver is handed a declaration with no resolver in it.
+
+The failing test is the symptom that surfaced it. The test is not what is wrong.
+
+**Proven rather than inferred.** Composing the same machine a second time with
+`listen-addresses` set to `127.0.0.1` and nothing else changed, every one of dnsmasq's eight
+resources appears — `needs-broker`, `mesh-state`, `package`, `config`, `runtime-dns`, `runtime`,
+`service` and `fact-node-zones`. The only difference between a machine with a resolver and a
+machine without one is whether somebody set a value that did not exist yesterday.
+
+## Why it matters beyond this instance
+
+**Leaving a module out is right, and being quiet about it is not.** The rule exists so one
+module's broken setting cannot stop a machine converging — a good rule. But the outcome here is a
+machine that applies cleanly, reports current, and is missing its DNS resolver. Every name on that
+machine then resolves through whatever was there before, or not at all, and nothing in the mesh
+says the resolver was dropped. That is the shape
+[issue 152](../152-a-nodes-plan-failure-silently-drops-its-routed-names/00-report.md) records for
+routed names, here for a whole module.
+
+**And a setting with no default is a definition that cannot be assigned.** Every other
+`${setting:…}` in the catalogue names something that is genuinely particular to one installation —
+a public domain, an issuer. "Which addresses besides my own do I answer on" has an obvious correct
+default for every machine that is not a LAN gateway: none beside loopback. A definition that
+refuses to compose until somebody sets a value most machines do not need is a definition that
+breaks the next node to be assigned it, and genesis with it.
+
+## What this does not claim
+
+Whether the live machines are affected was not checked — those four have had the setting set, or
+their resolvers would already be gone. The claim is about a machine assigned the resolver *from
+now on*, and about the silence.
+
+## Open questions
+
+- Should the declaration say which modules it left out, where a person or the console can see it?
+  `left_out` already travels to the host ([ADR 0163](../../02-DECISIONS/0163-taking-a-module-over-is-a-comparison.md));
+  what is missing is anything that reads it back and says so.
+- Should a `${setting:…}` be allowed a default in the definition — making "unset" mean "the
+  default" rather than "refuse" — or is a setting with a default no longer the operator's value?
+- Is leaving a module out ever right for a module a node is **assigned**, as opposed to one it
+  merely pulls in? An assignment is somebody saying *this machine runs this*; silently not running
+  it is the one answer nobody asked for.