ADR 0236, to-be 45: a build is judged on its first machine and put back by something other than itself

Phase 4 of to-be 45, started by the operator: the core's health as probes, a gate on
every plan's first machine, the rollback there once per build, the witness's contract
with the host, the bus as a step a person starts, and — with those in place — rolling
out as the default, keeping record where a module says why. 47 pushes by hand in one
day are what it ends.
This commit is contained in:
jochen
2026-10-06 18:39:14 +02:00
parent 3f32462434
commit ffc3a55f8c
5 changed files with 294 additions and 3 deletions
@@ -4,6 +4,7 @@ status: in-progress
code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-sdk, mesh-lab]
updated: 2026-10-06
decisions:
- 02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md
- 02-DECISIONS/0234-the-mesh-holds-a-conversation-with-its-operator.md
- 02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md
- 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md
@@ -224,7 +225,9 @@ itself — an unanswered probe is never a pass.
| D12 | every consumer of a provision that keeps its data is bound where it was last sent, or moves by a pin (`binding-kept`, `binding-moving`, `binding-moved`) | ADR 0232 |
| D13 | every item of data a machine declares is measured, is there, holds what it held, is written where it says it is, is backed up within its bound or sits on healthy redundant storage, and is no empty replacement of a copy kept elsewhere; an irreplaceable or valuable item a machine no longer declares is retired, not forgotten. Each machine's `node-backup` holder is asked `backed-up`; every provider of kept consumer data `provisioner_retirement` (with each held consumer's size) — see §2 for the kinds | ADR 0233, issue 273 |
| DW | the watchdogs of §3 ran within three of their intervals: the watchers are watched, and raise S10 when the self-check stops | rule 6 |
| H-* | the health probes of §8, run for every core component on every machine | rule 8 |
| H-* | the health probes of §8, run for every core component on every machine: H-controller, H-engine, H-tools, H-bus (as built, [ADR 0236](../../02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md)) | rule 8 |
| DG | every build that failed its gate, and every core build a witness put back, is said — `build.<module>.<machine>.rolled-back`, `core.<component>.<machine>.<outcome>`, `rollback-failed` — until a newer build passes or the reports stop saying it (ADR 0236) | rule 8 |
| DB | a bus upgrade a person started is said as `bus-maintenance` while it runs, and ends healthy within its bound or is said `bus-upgrade-failed`, with its snapshot (ADR 0236) | rule 8 |
- **`doctor`** answers the last run's verdict at once: per probe, pass, fail or failed-to-run, and age.
**`doctor run`** runs now under a call id. **`doctor probes`** lists the registry; **`doctor
@@ -403,12 +406,44 @@ the other machines follow. "Reported applied" is not enough.
verdict, time to verdict, rolled back or not. Read through `plans`.
- **A rolled-back build is not retried** by the same plan. A newer merge makes a new plan.
**As built** ([ADR 0236](../../02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md)):
- **Every release plan is gated, not only the core's.** A module is judged by its own health: it reported
applied, no witness put it back, no condition was raised since its send about the machine or about the
module there, and its tools are served on that machine where it has tools. Three healthy judgings at
least forty seconds apart and two minutes after the send, within ten minutes of it. A policy of
*together* is not gated.
- **The rollback is the ordinary path once per build.** The build is marked failed at its gate (never
registered again, `plans retry` refuses it); the module's registered build goes back to the build the
first machine ran before, still kept by ADR 0189; that machine is sent it. The verdict is written before
the send. Said as `build.<module>.<machine>.rolled-back` (warning) or `core.<component>.<machine>.
rolled-back` (urgent), `rollback-failed` (urgent, the operator's) when nothing could be put back, and as
the event `rolled-back`.
- **The witness, as built on the host** (mesh-host `internal/witness/contract.go`, the controller's half
`internal/lease/witness.go`): the controller's is the lease alone — the holder on this machine, taken
since the start, renewed within fifteen seconds, within sixty seconds of the start — and the node tools'
is their answer to the services protocol's ping within five seconds, within sixty. The controller's
"status in bound, doctor ran once" is the controller's own word in the lease's value, which the gate
reads and the host does not: a controller that holds the lease and never becomes ready is put back by
the gate sending the previous build. A witness says its verdict in `rollbacks` on every report while it
stands; the controller raises `core.<component>.<machine>.<outcome>` from it — urgent for rolled-back,
not-reversible, restore-failed and halted — and clears it with the first report without it.
- **A new controller that passes its gate sends the bus's machine the user list it composes**, when that
changed and nothing held back would go with it.
- **The default policy is to roll** (ADR 0236 §4); `record` stays where a module says why, keeps
irreplaceable data, or is the bus.
**The bus is a planned step.** A bus upgrade is a maintenance step a person starts through the
controller: the streams are snapshotted; a `bus-maintenance` condition is open for the step's duration;
the bus is replaced; afterwards D6, D7 and the round trip must pass, or the step is reported failed and
the snapshot is the way back. A step whose new version cannot be reverted (the bus's 2.10 → 2.11 is one)
says so before it starts and runs only on a person's explicit word, recorded as a hand act. Whether the
bus becomes a cluster that can be upgraded live is left to its own effort. *2026-10-06:* the snapshot
bus becomes a cluster that can be upgraded live is left to its own effort.
As built (ADR 0236): the verb is `bus upgrade --why … --reversible|--irreversible`; the snapshot is the
bus machine's backup holder backing the bus module up now ([ADR 0235](../../02-DECISIONS/0235-the-bus-is-backed-up-by-its-own-snapshot-of-each-stream.md)),
or one a person took and names with `--snapshot-taken`; the step's bound is fifteen minutes; a push
naming the bus's machine is refused while a new bus build waits for it. *2026-10-06:* the snapshot
and the way back from it exist — the bus image's own snapshot program, the same one the night's backup
runs, and a restore that builds a new store beside the live one for a person to swap in
([ADR 0235](../../02-DECISIONS/0235-the-bus-is-backed-up-by-its-own-snapshot-of-each-stream.md),
@@ -547,7 +582,7 @@ warning, not compared), and SMART read under an array.
| Repository | Delivers |
|---|---|
| mesh-controller | the health probes of §8; the gate on a core component's first machine; the rollout record in the plan; the bus maintenance step as a verb |
| mesh-controller | the health probes of §8; the gate on a core component's first machine — as built, on every plan's first machine (ADR 0236); the rollout record in the plan; the bus maintenance step as a verb; the default upgrade policy `roll` (ADR 0236) |
| mesh-host | keeping the previous controller and node tools builds; restoring one when its health is not met in bound, watching the lease bucket for the controller |
| mesh-tools | answering the health ping |
| mesh-lab | R3, R7, R8 |
@@ -557,6 +592,21 @@ one that starts and does nothing, one that crashes, one that cannot reach the bu
back with no hand, the mesh ends on the previous build, and a condition and a message say so. Live: the
next three core rollouts each record a health verdict.
**Built 2026-10-06** ([ADR 0236](../../02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md),
the operator's "start phase 4"): in mesh-controller (migration 0073), the probes H-controller, H-engine,
H-tools, H-bus, DG and DB; the gate on every plan's first machine, its record in the plan (`plans <id>`)
and in the store; the rollback by the ordinary path, once per build; the witnesses' verdicts as
conditions; the grants the witness reads; `bus` and `bus upgrade`; the default policy `roll`, `upgrade`
listing every module's policy and where it comes from; the user list carried after a controller passes;
a module deleted at its source forgotten, not built. In mesh-host, the witness (keeping the previous
controller and node tools, restoring one not healthy in bound, the launcher's for the node-engine) and the
installer's grants. In mesh-catalog, the modules that keep `record` say why, and the forge's announcer
says which files a merge deleted. On branches, not yet merged. **Not yet:** R3, R7, R8 on a lab mesh, and
so the *done when*; container state in the node-engine's report, without which a container that
crash-loops after its compose applied is seen only through what it breaks; composing `not-reversible` for
a build whose migration cannot be undone; the gate for a build asked by hand outside a plan; the next
three core rollouts' verdicts.
### Phase 5 — Checks before merge, and the replays
| Repository | Delivers |