Issue 295 and ADR 0242: a recorded build was carried by a plan's send, and mail went down with a health check
The plan delivering the first health declarations recreated the database and the document store (record) and nine of mail's containers at once. Record held only against a person's push to another machine; it now holds against every send but a person's, and a send says what it recreates.
This commit is contained in:
jochen
@@ -0,0 +1,92 @@
|
||||
---
|
||||
status: located
|
||||
opened: 2026-10-07
|
||||
located-in: [mesh-controller (cmd/mesh-controller release.go, plan.go, push.go), mesh-catalog modules/mailu]
|
||||
fixed-by: mesh-controller PR #115; mesh-catalog PR #106
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 295. A recorded build was carried by a plan's send, and mail went down with a health check
|
||||
|
||||
## Symptom
|
||||
|
||||
At 18:54 on 2026-10-07 a catalogue merge was delivered. It added the first health declarations of
|
||||
[ADR 0240](../../02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md)
|
||||
to twelve modules: the images' own health checks, adopted by name. Nothing else changed in them. Its
|
||||
plan rebuilt all twelve. Two minutes later two things happened:
|
||||
|
||||
- **On the control node, every mail container but two was recreated in the same minute.** Nine
|
||||
were recreated; the antivirus and the cache were not, because neither was given a check. The
|
||||
operator's phone reported mail errors, and a `mailu … smtp not running` condition was raised
|
||||
and then cleared.
|
||||
- **The database and the document store were recreated on the control node, and the database on
|
||||
the anchor as well.** Their upgrade policy is `record` ([ADR 0236](../../02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md)
|
||||
§4): they are providers whose restart drops every consumer on the machine, so a person takes
|
||||
each build. Nobody pushed.
|
||||
|
||||
The mesh's own record shows it:
|
||||
|
||||
- **The plan.** Ten of its modules carry a gate line. postgres and mongodb read only "built from"
|
||||
the merge's commit: no gate judged them, because the gate judges only what rolls out.
|
||||
- **The gate verdicts.** Ten were kept for the merge's commit: seven on the anchor, three on the
|
||||
control node. There are none for postgres or mongodb.
|
||||
- **What each machine was last sent.** After the sends, both machines are recorded as carrying
|
||||
postgres at the merge's commit, and the control node mongodb too. Before the merge they carried
|
||||
the previous commit.
|
||||
- **The controller's journal**, read through the service manager seat's `journal` verb:
|
||||
1. 18:55:52: postgres registered.
|
||||
2. 18:56:05: "sent 7 module(s) to [the anchor] first in one send (baserow, grafana, matrix,
|
||||
mosquitto, nodered, redis, supabase)".
|
||||
3. 18:56:17: "sent [the control node] 480 resource(s)", then "sent 3 module(s) to [the control
|
||||
node] first in one send (mailu, step-ca, website)".
|
||||
4. 18:57:26: the control node applied.
|
||||
- **What each machine reports.** `postgres.server`, `mongodb.server` and nine `mailu.*` containers
|
||||
have run only since 18:57 on the control node, and `postgres.server` since 18:56 on the anchor.
|
||||
|
||||
No release plan was involved. The sends were the plan's own sends to its first machines.
|
||||
|
||||
## Why it happened
|
||||
|
||||
1. **A send carries the machine's whole declaration, composed from the builds the mesh holds**
|
||||
([ADR 0221](../../02-DECISIONS/0221-a-push-sends-no-build-a-policy-or-a-plan-holds-back-except-to-the-machine-it-names.md)). The controller registers a recorded module's new build at
|
||||
the merge. From that moment, any send to a machine running the module composes the new build,
|
||||
whoever the send is for.
|
||||
2. **ADR 0236 §4a guards that only for modules that roll out.** "A gated send carries everything
|
||||
waiting on its machine, and its gate judges all of it." The controller's list of moves waiting
|
||||
for a gate leaves out a module whose policy records, because "a recorded one is a person's
|
||||
push". So the gate did not judge the recorded builds, the refusal of ungated sends did not see
|
||||
them, and the gated send carried them. `record` held only against the end of a person's push to
|
||||
another machine (issue 259's `heldBack`) and against the bus (the bus step). It did not hold
|
||||
against a plan, a release plan, a rollback, a healer or a rotation.
|
||||
3. **Nothing tells a person, before or during a send, that it will recreate every container of a
|
||||
module at once.** A change that only alters how containers are declared reads as harmless. A
|
||||
health check, an environment variable or a label each recreates the container all the same,
|
||||
and the node-engine applies a module's containers together. The change plan said mail
|
||||
"receives" a build. Nothing said its service would be interrupted, and mail's policy let a plan
|
||||
decide when.
|
||||
|
||||
## What is at stake
|
||||
|
||||
`record` is how the mesh keeps the providers, the network path and the irreplaceable data out of
|
||||
unattended rollout. Tonight showed that it holds only until some other module on the same machine
|
||||
rolls out. This was still true after the incident: plex (`record`, irreplaceable data) was rebuilt
|
||||
by a media-catalogue merge at 19:01 and sent nowhere. The next gated send to the anchor, for any
|
||||
module, would recreate it.
|
||||
|
||||
## Fix
|
||||
|
||||
Located and fixed in pull requests, not merged. Decision: [ADR 0242](../../02-DECISIONS/0242-a-recorded-build-moves-only-by-a-persons-push-and-a-send-says-what-it-recreates.md).
|
||||
|
||||
- **mesh-controller #115.** Every send except a person's push composes a recorded module from the
|
||||
manifest of the build the machine was last sent, and records that it still carries that build.
|
||||
The bus step lets the bus alone move. A kept build that is gone from the records refuses the send
|
||||
and names `push <node>` as the remedy. Every move a gated send carries now says how many of the
|
||||
module's containers it recreates, and whether with a new image or only a changed declaration, in
|
||||
the plan's note, the controller's log and `plans <id>`. The test
|
||||
`TestARecordedBuildIsCarriedOnlyByAPersonsPush` replays this case and fails without the fix.
|
||||
- **mesh-catalog #106.** Mail declares `record`: a module people use directly moves at a moment a
|
||||
person chooses. It holds only once #115 is live, so it merges second.
|
||||
|
||||
A replay in mesh-lab's register is still owed before this resolves (ADR 0237). The controller test
|
||||
above shows the replay's shape: a recorded provider and a rolled module on one machine, both
|
||||
rebuilt, and the plan's gated send.
|
||||
Reference in New Issue
Block a user