novox/hq
6
0
Fork 0
Code Issues Pull Requests 9 Releases Wiki Activity

Issue 295 and ADR 0242: a recorded build was carried by a plan's send, and mail went down with a health check

Browse Source 
The plan delivering the first health declarations recreated the database and
the document store (record) and nine of mail's containers at once. Record
held only against a person's push to another machine; it now holds against
every send but a person's, and a send says what it recreates.
This commit is contained in:
jochen
2026-10-07 20:24:24 +02:00
parent 4556d37370
commit be8ea16d28
3 changed files with 224 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
02-DECISIONS/0242-a-recorded-build-moves-only-by-a-persons-push-and-a-send-says-what-it-recreates.md
+131
View File
@@ -0,0 +1,131 @@
---
topic: what runs on it
status: proposed
date: 2026-10-07
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md
---
# 242. A recorded build moves only by a person's push, and a send says what it recreates
## Context
[ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md)
§4 keeps `record` for three kinds of module: those whose new build a rollback could not undo, the
network path, and the providers whose restart costs. A person takes each of their builds. Its §4a
says that "a gated send carries everything waiting on its machine, and its gate judges all of it".
A send carries the machine's whole declaration
([ADR 0221](0221-a-push-sends-no-build-a-policy-or-a-plan-holds-back-except-to-the-machine-it-names.md)),
and that declaration is composed from the builds the mesh holds. A recorded module's build is
registered at its merge, so every send to its machine composes the new build. "Waiting" and
"judged" were only ever computed for modules that roll out.
On 2026-10-07 ([issue 295](../04-ISSUES/295-a-recorded-build-was-carried-by-a-plans-send/00-report.md)),
a catalogue merge adopted the images' own health checks in twelve modules. Its plan's gated sends to
its two first machines carried the new builds of the database (on both machines) and of the
document store (on the control node). Both modules are `record`. No gate judged them, and nobody
pushed. The same send recreated nine of mail's eleven containers at once for a change that brought
no new image, and the operator's mail was down until they were up again. The change plan had said
only that mail "receives" a build.
Measured that evening:
- Fourteen modules record: the bus, two holding irreplaceable data, five on the network path, five
providers and keycloak.
- Each of them, waiting on a machine, was carried by the next send there for any other module that
rolled out. That happened twice that evening.
- One was still waiting afterwards: the media library, rebuilt at 19:01.
## Considered Options
1. **Refuse every send but a person's to a machine where a recorded build waits**, as the bus is
held today. Rejected. One recorded build would stop every plan on its machine until a person
pushed it. On the control node that is the controller's own path, and [issue 280](../04-ISSUES/280-a-rebuild-of-an-unchanged-source-was-read-as-a-new-bus/00-report.md)
is the evening it cost.
2. **Do not register a recorded build until a person pushes.** Rejected. The mesh would no longer
hold what was built. `status` could not say a machine is behind, and a person's push would have
nothing to send.
3. **Leave the recorded module out of the declaration** ([ADR 0163](0163-taking-a-module-over-is-a-comparison.md) rule 6: its containers
untouched). Rejected. A left-out module contributes nothing, so its consumers on the machine would
lose what it provides to them in the same send.
4. **Compose the recorded module at the build its machine runs.** Chosen. The machine runs what it
ran, everything else in the send moves, and the person's push remains the one way the new build
arrives.
For the interruption:
5. **The node-engine recreates a module's containers one at a time where the module allows it.**
Not decided here. It is the node-engine's apply order, and mail cannot tolerate even one of its
front, smtp and imap containers going down unseen. It stays open as the better answer for
modules with replicas.
6. **Schedule a recreation-only change into a quiet window a module declares.** Not decided here.
It needs a window per module and a clock in the walk, and a person's choice of moment does the
same today.
7. **A module that people use directly declares `record` and says why, and every send says what it
recreates.** Chosen. It uses the policy that exists, which 1–4 make hold, and it marks the
interruption where people read the send.
## Decision
1. **A recorded build moves only by a person's push.** A person's push names a machine. Every other
send composes each recorded module at the build that machine was last sent: the manifest of that
build, from the build records, standing for the one the mesh holds. It also records that it still
carries that build. This covers a plan's send to its first machine and to the rest, a release
plan's, a rollback's, a healer's and a rotation's.
- `status` still says the machine is behind, and `push <node>` sends the new build.
- **The bus step is a person's word for the bus alone.** It moves the bus, and every other
recorded module on that machine stays.
- A module the machine was never sent is composed as the mesh holds it, since there is nothing
running to keep.
- If the build to keep is no longer in the records, the send is refused and the refusal names
`push <node>`. Composing the new build there would be the very move this rule stops.
§4a's "a gated send carries everything waiting on its machine" now reads: everything *that rolls
out*.
2. **A send says what it recreates.** For every module a gated send moves, it compares the build
the machine ran with the build it is sent, container by container. It says how many containers
it recreates and of how many, which ones, and whether any image is new or only the declaration
changed. It warns when it recreates more than one container at once, because the module's service
is interrupted until they are up again. This goes in the walk's record (`plans <id>`, the
delivery's walk), the plan's note and the controller's log.
3. **A module people use directly declares `record` and says why.** Mail is the first. A change to
how its containers are declared recreates them together, so a person chooses the moment. Its
image updates wait for a person too. A module that can be interrupted unseen keeps rolling.
## Consequences
- **Plans no longer restart providers behind a person's back.** In exchange, a recorded module's
machines stay behind until somebody pushes, as `record` always promised. `status` and `upgrade
backlog` already say so.
- **A declaration can now hold a build older than the one the mesh holds.** The resolution sees the
kept manifest for that machine only. Other machines' consumers still resolve against the
provider's registered manifest.
- **Pruning the build records can make a kept build disappear.** [ADR 0189](0189-the-store-keeps-what-the-records-name.md)
keeps five builds per module. If a recorded module falls more than five builds behind, sends to its
machine are refused, said with the remedy, until a person pushes. That is loud rather than silent.
- **The marking comes from the build, not before it.** The change plan at merge time cannot yet
know which containers a build changes. The walk says it at the send. Saying it before the merge
needs the planner to read the merged manifests, which is a later step.
- **The time interrupted is not yet a number.** The mesh measures a send-to-report time per
machine (`durations`), not how long one container takes to restart. The send says
"interrupted until they are up again". A per-container start time in the node-engine's report
would let it name a number.
How it is checked: the controller's `TestARecordedBuildIsCarriedOnlyByAPersonsPush` replays
issue 295 (a recorded provider and a rolled module on one machine, both rebuilt, the gated send). It
fails on the commit before the fix. A mesh-lab replay is owed with the issue.
## References
- [Issue 295](../04-ISSUES/295-a-recorded-build-was-carried-by-a-plans-send/00-report.md), the
evidence.
- mesh-controller PR #115: `recordedKept`, `keepRecorded`, `sendKeeps`, `inventory.ManifestAt`,
`catalogue.Recreates`.
- mesh-catalog PR #106: mail's policy.
- [ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md)
§4, §4a; [ADR 0221](0221-a-push-sends-no-build-a-policy-or-a-plan-holds-back-except-to-the-machine-it-names.md);
[ADR 0240](0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md), whose first
declarations were the change that recreated mail.
02-DECISIONS/README.md
+1
View File
@@ -339,6 +339,7 @@ python3 00-META/checks/index.py fail if stale
- **0233** — [A module declares the data it holds, and the mesh protects and watches it from that declaration](0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md)
- **0235** — [The bus is backed up by its own snapshot of each stream, taken under the bus module's account](0235-the-bus-is-backed-up-by-its-own-snapshot-of-each-stream.md)
- **0240** — [A module says how it is healthy, and the node-engine judges it](0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md)
- **0242** — [A recorded build moves only by a person's push, and a send says what it recreates](0242-a-recorded-build-moves-only-by-a-persons-push-and-a-send-says-what-it-recreates.md) *(proposed)*
- **0243** — [The agent module removes a home item it did not place only on the person's word, and keeps a copy](0243-the-agent-module-removes-a-home-item-it-did-not-place-only-on-the-persons-word-and-keeps-a-copy.md)
### How it is built