ADR 0236, to-be 45: a build is judged on its first machine and put back by something other than itself

Phase 4 of to-be 45, started by the operator: the core's health as probes, a gate on
every plan's first machine, the rollback there once per build, the witness's contract
with the host, the bus as a step a person starts, and — with those in place — rolling
out as the default, keeping record where a module says why. 47 pushes by hand in one
day are what it ends.

02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md

@@ -61,6 +61,13 @@ is tried again. It is never reported as done "until the next push".
 The plan records which machine went first, so a controller replaced mid-rollout resumes from there. A
 policy of *together* keeps today's behaviour.
 
+> **The mechanism changed — 2026-10-06, by [ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md).** What still stands: one machine
+> first, the first by name, the rest only after it, a failed first machine stopping the module and the
+> plan, *together* as it was. What moved: "reported applied and current" is no longer enough — the first
+> machine is judged at a gate by the component's health, three times over at least two minutes within ten,
+> before the rest are sent; and a first machine that fails, by its report or at the gate, is not left on
+> the build that failed it: the previous build is registered again and sent to it, once per build.
+
 **3. A newer merge takes over an older plan.** When a merge into a repository's branch makes a plan,
 every open plan for the same repository and branch made before it is superseded, ordered by when each
 plan was made, never by commit:

02-DECISIONS/0221-a-push-sends-no-build-a-policy-or-a-plan-holds-back-except-to-the-machine-it-names.md

@@ -84,6 +84,13 @@ behind, held or not. It is the remedy `record` names when an upgrade is announce
 when you want them"), and the one command for taking a recorded upgrade everywhere. Narrowing it would
 leave `record` with no way to say "now".
 
+> **The mechanism changed — 2026-10-06, by [ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md).** What still stands: a send records
+> the builds it carried, a held machine is left and named, the named machine is sent everything, and
+> `push --behind` takes a recorded upgrade everywhere. What moved: `record` is no longer the default — a
+> build rolls out one machine first, gated, unless a person, the module, its irreplaceable data or the bus
+> says otherwise — and one case of "the machine a push names is sent everything" is refused: a machine
+> whose bus would move, which is replaced only as the planned step `bus upgrade`.
+
 ## Consequences
 
 - `record` and "one machine first" hold a change back from every push that does not name the machine.

02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md

@@ -0,0 +1,226 @@
+---
+topic: the mesh
+status: accepted
+date: 2026-10-06
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md
+---
+
+# 236. A build is judged on its first machine and put back by something other than itself, and so it rolls out unattended
+
+## Context
+
+**The operator said "start phase 4"** of [to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md)
+on 2026-10-06: a health definition per core component, the gate on a plan's first machine, rollback by
+a witness that is not the new build, the bus as a planned step ([ADR 0227](0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md)
+rule 8). The same day's hand-act log says what the missing phase costs:
+
+- **47 pushes by hand in one day**, the cause the log records most and the one that raised
+  `healer-wanted` (S15). By what they did: **21 walked a new node-engine build through the four machines
+  one at a time**, each pushed only after the one before was looked at; **8 carried a new controller's
+  grant into the bus's user list**, which the controller's own rollout does not send; **2 were `push
+  --behind`** to deliver catalogue builds that their policy held back; 16 were steps of other work a person
+  was walking through.
+- **Every module but five had the upgrade policy `record`** — built on a merge, sent to no machine
+  until a person pushed. `record` was the column's default since [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md)
+  §3, and nothing told a `record` somebody chose from the default it always was. The five that rolled out
+  were set by hand: the controller, the catalogue, the build agent, the intrusion filter and the records
+  module.
+- **The one-machine-first rollout ([ADR 0218](0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md)
+  §2) judged the first machine by "reported applied".** A build that applies and then serves no tools,
+  crashes after the report, or makes the machine's own conditions fire passes that test. Nothing put a
+  failed build back: a first machine that failed stopped the plan, and the machine stayed on the build
+  that failed it.
+- **The witness for the core is being built beside this.** mesh-host's node-engine keeps the previous
+  controller and node tools bundles beside the new ones, judges the new controller by the lease and the
+  node tools by their answer to the services protocol's ping, puts the previous build back when either is
+  not healthy in its bound, and says so in every report while it stands (mesh-host PR #40). The controller
+  must grant it what it reads, raise what it says, and never send what it put back again.
+- **A merge that deleted a module's directory made the controller ask the build seat to build it.**
+  The merge of 2026-10-06 that folded `public-acme` into the proxy failed its plan: "has no module.json at
+  modules/public-acme"; the four other modules of its tier were built and never sent.
+- **The bus's planned step had a snapshot to take since [ADR 0235](0235-the-bus-is-backed-up-by-its-own-snapshot-of-each-stream.md)**:
+  the bus machine's backup holder snapshots every stream through the bus module's dump.
+
+**Checked against GENESIS.** *Anything requiring a human to notice it will be noticed late*: a person
+walking a build through four machines is that sentence, every day. *Failure must be loud*: a build put
+back without saying it, or never put back, is the silent failure the core exists to end. Nothing here
+conflicts with GENESIS.
+
+## Considered Options
+
+1. **Keep `record` as the default and heal the pushes** — a healer that pushes what is behind. Rejected:
+   it is a push with no judgement, the cascade of [ADR 0221](0221-a-push-sends-no-build-a-policy-or-a-plan-holds-back-except-to-the-machine-it-names.md)
+   with the hand taken out; a broken build would reach every machine as fast as before, and nothing would
+   put it back.
+2. **Roll everything out, gate nothing, and rely on the conditions to say what broke.** Rejected: rule 8
+   says the core is judged by health, not by "applied"; a condition after a build is everywhere says what
+   broke after it broke everywhere.
+3. **Gate only the core, and leave modules to `record`.** Rejected: the pushes are mostly the core's, but
+   the catalogue's are the ones that stay behind the longest, and a module has health the controller can
+   read as well as the core has.
+4. **A health definition per core component, a gate on every plan's first machine that judges a module by
+   its own health, a rollback there by the ordinary path once per build, a witness on the machine for what
+   cannot put itself back, and then rolling out as the default.** Chosen.
+
+## Decision
+
+**1. A core component's health is a definition, written as probes.** In the self-check's registry
+(to-be 45 §4), run every five minutes on every machine and by the gate on a first machine:
+
+| Probe | Component | Healthy when |
+|---|---|---|
+| H-controller | controller | the lease is held, renewed within its age, by a controller that says it is ready: its self-check ran, and `status` answered in full within ten seconds in that run (D9) |
+| H-engine | node-engine | every machine heard from has reported its current declaration, under a node-engine build it names; on its first machine, under the new build |
+| H-tools | node tools | every machine heard from that runs them has them answering the bus's discovery |
+| H-bus | bus | every stream and durable consumer the mesh defines is on the bus, and a request crosses it to the machines' node tools and back |
+
+**2. Every release plan's first machine passes a gate before the rest are sent.** ADR 0218's first
+machine is judged from the moment it was sent the build: it reported the build applied; no witness on it
+put the build back; no condition was raised since about the machine, or about the module on it; and the
+component's health holds — a core component's definition above, or a module's own: its tools are served
+on that machine where it has tools and the machine runs the node tools that serve them. **Healthy three
+times, at least forty seconds apart and two minutes after the send, within ten minutes of it.** A module
+on one machine is judged the same way; only then does the plan go on. A policy of *together* is not
+gated: it is the module saying it must change everywhere at once.
+
+**3. A build that fails its gate is put back there, once, and never sent again on its own.** The plan
+stops; the build is marked failed at its gate; the module's registered build goes back to the build the
+first machine ran before — the newest successful build of that commit still kept ([ADR 0189](0189-the-store-keeps-what-the-records-name.md)
+keeps five) — and that machine is sent it, by the ordinary send. The verdict is written before the send
+and is never written over, so a controller replaced between the two does not send it twice. A build
+marked failed is refused registration if its outcome is heard again, and `plans retry` refuses the plan;
+a newer merge or a `rebuild` makes a new build, judged again. It is said as the condition
+`build.<module>.<machine>.rolled-back` (a warning: the machine runs what it ran before) or
+`core.<component>.<machine>.rolled-back` (urgent), `rollback-failed` (urgent, the operator's) when
+nothing could be put back — no earlier build kept, the machine had never had the module, the send refused
+— and as the event `rolled-back`. The probe DG keeps each until a newer build of the module passes.
+
+**4. A build rolls out unattended by default.** With the gate and the rollback in place, a module's
+build is sent one machine first, judged, then the rest, unless something says otherwise:
+
+- **a person**, through `upgrade <module> roll-out|record|default`, `record` with why; kept, said with
+  who and why, and taken back by `default`;
+- **the module**, in its manifest: `upgrade` with policy `roll`, `together` or `record`, the last two
+  with why;
+- **its data**: a module that declares irreplaceable data ([ADR 0233](0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md)),
+  its own or kept with a provider, records — a rollback cannot undo what a new build does to data that
+  cannot be had again;
+- **the bus** records whatever anyone says (rule 6 below).
+
+The store's `record` rows were the old default and become no choice; a person's `roll-out` is kept.
+The catalogue keeps `record` where a module says why — the network path a rollback could not cross, and
+the providers every consumer on a machine drops with, among them the two holding the photos. The
+resulting policy of every module the mesh held on 2026-10-06:
+
+| Policy | Modules | Why |
+|---|---|---|
+| record — the bus | nats | a planned step a person starts (rule 6) |
+| record — irreplaceable data | plex, photos | the media library; the photos and their albums (the operator's ranking, ADR 0233) |
+| record — the network path | dnsmasq, nftables, networkmanager, systemd-networkd, sshd | a build that cuts a machine off from the bus cannot be put back from outside it; sshd is the way in when the mesh cannot reach it, which no gate sees |
+| record — a provider whose restart costs | postgres, mssql, mongodb, minio, keycloak | every consumer on the machine drops with it, a new version may change its data's format in place, and minio and mongodb hold the photos |
+| roll — a person's choice, kept | mesh-controller, mesh-catalog, build-agent, fail2ban, records | — |
+| roll — the default | every other module: 111 of the 129, ten of them on no machine, the node-engine and the node tools among them | gated on the first machine; the node-engine and node tools also witnessed on the machine |
+
+The private network itself is provided by the controller and moves only with it; the modules that run on
+no machine move nothing.
+
+**5. The controller's rollback is the node-engine's on its machine; the contract is written once on
+each side.** mesh-controller `internal/lease/witness.go` and mesh-host `internal/witness/contract.go`,
+held field for field:
+
+- **What the host reads.** A direct get of the lease bucket's one key, `holder`: the lease's holder as
+  the controller writes it (instance, host, epoch, taken, renewed; anything else ignored). The node
+  principal of a machine assigned the controller is granted that one subject; every machine's node
+  principal is granted the ping of its own node tools. Reads only: the writers table still refuses any
+  write.
+- **When it is healthy.** The holder's host is the machine, it took the key at or after the moment the
+  host started the new build (two seconds of skew), and renewed it within the key's fifteen seconds:
+  asked every five seconds, within sixty of the start. The node tools: they answer the ping within five
+  seconds, within sixty of the start.
+- **What it starts.** The previous bundle it kept, as the previous declaration ran it.
+- **What it says.** `rollbacks` on every report while the verdict stands — component, from, to, outcome,
+  why, when — and `witness` with the contract's version. The controller raises
+  `core.<component>.<machine>.<outcome>`: urgent for rolled-back, not-reversible, restore-failed and
+  halted; a warning for nothing-to-restore and unwitnessed; cleared by the first report without it. On a
+  first machine, a verdict made since the send fails the gate, so the build is marked and the registered
+  build put back.
+- **What the controller adds.** It writes in the lease's value whether it is ready (rule 1), which the
+  host does not read and the gate does: the host rolls back a controller that never holds the lease, the
+  gate one that holds it and never becomes ready, by sending the previous build, which the host applies
+  as any declaration. A process's `witness` and `not-reversible` are the host's to read; the controller
+  sends neither yet — the host's defaults by name are the two witnesses above — and sends them only to a
+  machine whose report carries `witness`.
+- **And the grant that follows the controller.** Once a new controller passes its gate, it sends the
+  machine holding the bus the user list it composes, when that differs from the one last sent and nothing
+  held back would go with it: the eight pushes of the day that carried a controller's grant by hand.
+
+**6. The bus is never rolled out; its upgrade is a step a person starts.** Its policy is `record`
+whatever is said, `upgrade` refuses it a roll-out, a plan builds it and sends nothing, a cascade holds its
+machine (ADR 0221), and **a push naming its machine is refused while a new bus build waits for it**, with
+the remedy. The step is `bus upgrade --why …`: refused unless the person says whether the new version can
+be reverted (`--reversible`, or `--irreversible` as their explicit word that it runs anyway); the bus
+machine's backup holder snapshots the streams first (ADR 0235) — a person who took one by hand says where
+— and only then is the machine sent; recorded as a hand act; `bus-maintenance` (the probe DB) is open
+while it runs, and the step ends done when the machine reported the new bus applied and H-bus passes, or
+failed after fifteen minutes, said urgent with its snapshot as the way back while the bus is not healthy.
+
+**7. A module deleted at its source is not built.** The forge's announcer says which of a merge's files
+it deleted; a module whose manifest is among them is forgotten where nothing holds it and said otherwise
+— never asked to build. A build that finds no manifest at a module's path, from an announcer that does
+not say which files went, marks the module deleted in its plan, which goes on.
+
+## Consequences
+
+- **A merge reaches every machine with no hand**, one machine first, judged for at least two minutes —
+  and a build that breaks its first machine is put back there and goes nowhere else. The 21 pushes that
+  walked a node-engine build, and the 2 that delivered held catalogue builds, are the plan's.
+- **A rollout is slower by the gate**: two minutes at the least per tier that rolls out, ten at the most
+  before a verdict. A plan with a module on one machine waits on that machine's gate before its next
+  tier.
+- **The controller's grant gains one verb that acts**, `node-backup now`, used only by the bus step, and
+  an event, `rolled-back`; the installer's first user list carries both (mesh-host).
+- **Thirteen modules still wait for a person**, each saying why in the catalogue or by its data; `status`
+  lists the machines behind them and `push <machine>` walks them, as before. A person who wants another
+  held says `upgrade <module> record --why …`.
+- **What the gate cannot see.** A container that crash-loops after its compose applied is seen only
+  through what it breaks — its tools, a provider's failing word, the machine's own conditions — until the
+  node-engine reports container state, which is mesh-host's to add. A build whose migration cannot be
+  undone is put back all the same; marking such a build `not-reversible` for the witness is not composed
+  yet.
+- **A build asked by hand that rolls out** (a `rebuild` outside a plan) is still sent one machine at a
+  time by the upgrade handler, without the gate; only plans are gated.
+- **Rollout order**: mesh-host first (its genesis user list, and the witness, which reads nothing the
+  controller does not yet grant until then); the controller second, whose migration turns the store's
+  default `record` rows into no choice; the catalogue third — its manifests carry `upgrade`, which a
+  builder older than the controller refuses.
+
+## How it is checked
+
+| Rule | Checked by |
+|---|---|
+| the gate holds back the rest | the controller's test against a real store: a build that fails its gate on the first machine is put back there, registered at the previous build, marked, said as a condition and an event, never sent to the second machine, never registered or rolled back again, and `plans retry` refuses it |
+| a passing build rolls everywhere | the controller's test: three healthy judgings, then the rest sent and the plan done, its verdict kept, no condition |
+| the bound | the controller's test: a first machine never healthy fails at the bound and is put back |
+| the health definitions | the controller's test per component: tools not served, a condition since the send, a witness's verdict, node tools not answering, a lease held by an older controller or one not ready |
+| the policy | the catalogue's test: default roll, a module's word, irreplaceable data, the bus whatever it says; a record without why refused; the store's test: the current builds read the derived policy |
+| the bus is never rolled out | the controller's test: the bus records whatever it says, a person's roll-out is refused, a plan sends nothing, a push may not send its machine, the step refuses without its word on reversibility and without its snapshot, and starts with both |
+| a deleted module is not built | the controller's test: a merge deleting a module's manifest asks no build and forgets it; a build finding no manifest leaves the plan going |
+| the witness's contract | the lease package's test of the host's rule; the broker's test of the grants (the ping on every machine, the lease's key where the controller runs, no write); the controller's test that a witness's verdict is its condition while reports carry it and cleared after |
+| live | the next merge to the catalogue sends one machine first and the rest after its gate, with no push; `plans <id>` shows the gate's record; the next week's hand-act log has no push for a build that rolled out |
+
+## References
+
+- [To-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §8 and Phase 4 — the design
+  this builds and amends.
+- [ADR 0218](0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md),
+  [ADR 0221](0221-a-push-sends-no-build-a-policy-or-a-plan-holds-back-except-to-the-machine-it-names.md),
+  [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md) — the rollout, the hold, the policy,
+  whose mechanisms move here.
+- [ADR 0233](0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md),
+  [ADR 0232](0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md) — the data a rollout never
+  moves; [ADR 0235](0235-the-bus-is-backed-up-by-its-own-snapshot-of-each-stream.md) — the snapshot the
+  bus step takes.
+- mesh-controller, mesh-host, mesh-catalog: the branches `feat/core-upgrades-that-roll-back`;
+  mesh-host's witness `feat/core-upgrades-roll-back`.

02-DECISIONS/README.md

@@ -204,6 +204,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0230** — [A consumer the mesh stops asking for is retired, and deleted only by a person](0230-a-consumer-the-mesh-stops-asking-for-is-retired-and-deleted-only-by-a-person.md)
 - **0231** — [A healer acts on what observation raised, and only observation says it worked](0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md)
 - **0234** — [The mesh holds a conversation with its operator, over channels that are seats, and an answer that performs an action is authorised by the controller](0234-the-mesh-holds-a-conversation-with-its-operator.md)
+- **0236** — [A build is judged on its first machine and put back by something other than itself, and so it rolls out unattended](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md)
 
 ### Its tiers, from the bottom up