Issues 301 and 302: a push of recorded builds is no repair, and the lab's Go is checked in Go

301: S15 counted the operator's pushes of recorded builds, the word ADR 0242 asks for, as repairs and wanted a healer for three causes. 302: the lab's replays register was never compiled before a merge, because its check ran in the TypeScript toolchain. To-be 45 §7 and §9 say how each is now decided and checked.
2026-10-08 00:17:14 +02:00
parent 69ab2fd7f7
commit 7fde8f4bbe
5 changed files with 275 additions and 2 deletions
@@ -2,7 +2,7 @@
 layer: to-be
 status: in-progress
 code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-sdk, mesh-lab]
-updated: 2026-10-07
+updated: 2026-10-08
 decisions:
  - 02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md
  - 02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md
@@ -424,6 +424,16 @@ row no longer sees.
 `hand-act record` refuses that cause, so a repair cannot pass for a drill by the word it gives (issue
 292; checked by the controller's S15 tests).
 **A push of recorded builds is no repair.** A build of a module whose upgrade policy records moves only
 by a person's push ([ADR 0242](../../02-DECISIONS/0242-a-recorded-build-moves-only-by-a-persons-push-and-a-send-says-what-it-recreates.md)),
 and that push is the word the policy asks for. So a push says what it was, read by the controller from
 what it carries and never from a word the person gives. It is a push of recorded builds when it names one
 machine, that machine's last send is known and its last report is no failure, and every build it moves
 records rather than rolls out: a held new build, or a recorded module's first build on a machine newly
 assigned it. The act is written with its kind and what it carried; the table lists a push of that kind as
 a person's decision, and S15 never counts it. Any other push by hand still counts (issue 301; checked by
 the controller's replay R301 and its tests of the rule).
 > **Progressive insight — 2026-10-06.** This paragraph said before: "except `retire-waiting` and
 > `cleanup-waiting`: approving a retirement and deleting what was retired are a person's decision by
 > design (ADR 0230), and no healer may take them over." The exception was keyed on two causes, and the
@@ -536,7 +546,10 @@ A changed file touches exactly the modules whose build reads it; a file no build
 change reaching a module, or adding one, runs **the gate** on the build seat (`mesh/merge-gate`): the
 touched manifests through `module check`, failing only what the change brings; every machine composed with
 the definitions of the modules the plan moves or adds; the replays. Every repository of the mesh runs **its
-own** `merge-check.sh` (`mesh/repo-check`) in the toolchain it declares; one with none has "no repository
+own** `merge-check.sh` (`mesh/repo-check`) in the toolchain it declares. A repository in two languages
 declares each further part of its check and the toolchain that part runs in, and the build seat runs each
 part in that toolchain's own container: the layer passes only when every part does (issue 302; checked by
 the builder's `TestARepositoryInTwoLanguagesIsCheckedInBoth`). A repository with none has "no repository
 check defined", a success where the branch does not require `mesh/repo-check` and a failure for a person
 where it does. **A note is never a warning on a required status**: the forge combines `warning` as a
 failure, so a note — a wide rebuild, a problem already so on the base — is a success that says it, and only
@@ -0,0 +1,106 @@
 ---
 status: resolved
 opened: 2026-10-07
 located-in: [mesh-controller cmd/mesh-controller (handacts.go, recorded_push.go, signals.go S15)]
 fixed-by: novox/mesh-controller PR #126
 replay: R301
 amended-design: 03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md
 ---
 # 301. A person's push of a recorded build was counted as a repair
 ## Symptom
 On the evening of 2026-10-07 the controller's `conditions` held three warnings from S15:
 - `mesh.hand-acts.split-dns.healer-wanted`
 - `mesh.hand-acts.words.healer-wanted`
 - `mesh.hand-acts.uplink-verbs.healer-wanted`
 The first one read:
 > "split-dns" was repaired by hand 4 times in 14 days … a healer is wanted for it
 None of the ten acts behind them was a repair. Each was `mesh-controller.push {node, why, cause}`
 through the seat, with the operator's yes as its why. Machine by machine, they rolled out new builds of
 the resolver (`systemd-resolved`, its first build on the laptop and the workstation), the FortiClient
 adapter, the two network managers, the packet filter and mail. Every one of those modules declares the
 upgrade policy `record`. [ADR 0242](../../02-DECISIONS/0242-a-recorded-build-moves-only-by-a-persons-push-and-a-send-says-what-it-recreates.md)
 says a recorded build moves only by a person's push. So each push was the person's word that the policy
 asks for: the mesh working as decided.
 ## Why it is a design issue
 To-be 45 §7 says a push by hand counts toward `healer-wanted`, because roll-out by default
 ([ADR 0236](../../02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md))
 exists to end pushes by hand. ADR 0242 then made one kind of push the only way a recorded build moves.
 The two rules were never put side by side. A module the mesh may never roll out by itself is exactly one
 no healer may move, so a healer for it is never wanted. Yet every time a person rolled out such a module
 the condition was raised. And the more carefully the person did it, one machine at a time, the sooner it
 was raised.
 The table of verbs that write the log ([issue 292](../292-a-drill-was-counted-as-a-repair/00-report.md),
 to-be 45 §7) reads whether an act is a repair from its verb. Here that is not enough: the push the policy
 asks for and a push that repairs are the same verb. What tells them apart is what the push carried, and
 the log did not keep that.
 ## Fix
 **A push says what it was, read from what it carried.** It is never read from a word the person gives,
 which issue 292 ruled out. Before a push that names one machine is recorded, the controller reads what
 that machine was last sent against what the mesh holds. The push is a **push of recorded builds** when
 all of these hold:
 - what the machine was last sent is known;
 - its last report is no failure or refusal, so the push is not re-sending to mend one;
 - it moves at least one module's build; and
 - every module it moves declares `record`: either a held new build of a module the machine runs, or the
  first build of one newly assigned there.
 Anything else counts as a push by hand, as before: a build that rolls out, a module taken off, or a push
 that moves nothing.
 The act is written with `kind: recorded-builds` and, in `carried`, each module's move from one build to
 the next. The table of verbs reads a push of that kind as a person's decision, and S15 never counts it.
 `hand-acts` prints it as such. The push prints which kind it is, and when it is not a push of recorded
 builds, why.
 **The three open warnings clear by observation.** The ten acts behind them were recorded before a push
 said its kind, and the log kept no record of what they carried. So the controller names them by their
 log ids, each with what it carried (`recordedBefore`). That list is checked in the pull request and
 covers no act recorded after the fix. S15 then reads those ten as pushes of recorded builds. On its next
 tick after the build goes out it no longer sees the three causes repeated, so the conditions close.
 Nobody closes them by hand.
 A fourth warning, `mesh.hand-acts.push.healer-wanted`, stays open. It comes from four pushes that set a
 setting on each machine and one that sent a new holder's assignment. Those pushes carried a settings
 change and a module that rolls out, not a recorded build. Whether either is a repair is a separate
 question.
 ## How it is checked
 - **The replay, R301**, in the controller (`TestReplay301APersonsPushOfAHeldRecordedBuildIsNoRepair`).
  It sets up two machines running a recorded module, then registers the module's new build. Each
  machine gets a push, recorded the way the seat's push records it, with the same cause. S15 then reads
  the log as the watchdog does. The test fails on the commit before the fix with "split-dns was repaired
  by hand 2 times in 14 days", and passes on the fix. It is registered in mesh-lab's replays register
  (novox/mesh-lab PR #59).
 - `TestAPushIsOfRecordedBuildsOnlyWhenEveryBuildItMovesWasHeldForAPersonsWord` covers the rule case by
  case. One held build, two, and a recorded module's first build are pushes of recorded builds. A build
  that rolls out, a push that moves nothing, a module taken off, an unknown last send and a failed last
  report are not.
 - `TestThePushesOfRecordedBuildsOf20261007WantNoHealer` takes the live log's ten acts as recorded and
  checks that the three warnings do not open. It also checks that the settings pushes still count, a push
  with no kind still counts, and another verb under a listed id is not read as such a push.
 Live: once the build goes out, the three `healer-wanted` warnings close within a watchdog tick, and
 `hand-acts` marks the ten pushes as pushes of recorded builds.
 The trail is in [01-diagnosis.md](01-diagnosis.md).
 ## Resolved — 2026-10-08
 Fixed by novox/mesh-controller PR #126. The replay is R301 in mesh-lab's replays register: run against
 the fix's branch, it fails on the commit before the fix and passes on the fix. Once the pull requests
 merge, the build goes out like any controller build. Watching the three warnings close then is the live
 check.
@@ -0,0 +1,42 @@
 # 301 — diagnosis
 ## 2026-10-07: where a push becomes a repair
 `mesh-controller.push` writes its entry in the hand-act log (`handActFlags.record`) before anything
 else, before it reads the store. The entry holds the verb `push`, the machine, the why and the cause.
 S15 (`watchHandActs`) counts every entry that the table of verbs (`handActVerbs`) does not mark as a
 person's decision. `push` was listed there with no decision, and the comment beside it said what the
 design says: "a push by hand is exactly what roll-out by default exists to end".
 So the controller could not have told these pushes apart from repairs. It recorded each one before
 knowing what it would send, and kept nothing of what it sent.
 `hand-acts` showed all ten acts with the causes the person gave (`split-dns`, `words`, `uplink-verbs`).
 Each why names the modules moved. All of them (`systemd-resolved`, `forticlient`, `networkmanager`,
 `systemd-networkd`, `nftables`, `mailu`) declare `upgrade.policy: record` in the catalogue, each with
 the reason ADR 0236 §4 or ADR 0242 gives.
 ## Ruled out
 - **Keying the exemption on the cause.** Issue 292 ruled this out: any repair could then avoid S15 by
  using that word.
 - **A separate verb for such a push**, as issue 292 gave drills `hand-act drill`. For a drill, the person
  is the only one who knows that something was broken on purpose. For a push of recorded builds, the controller can
  read the answer from its own store: what the machine was last sent, what the mesh holds, and each
  module's policy. A second verb would make the person pick the right one, and a recorded build pushed with
  the wrong verb would still raise the condition. It would also leave the push itself unable to say what it
  carried. Reading the kind from the send is what the mesh observes, so it is the answer that needs no
  one's word.
 - **Reading the kind at S15's tick instead of at the push.** By the time the watchdog reads the log, the
  machine's last send is the push itself, and what it moved from is gone. The kind has to be read before
  the push sends, and kept in the act.
 - **Reconstructing the ten acts from the store.** The store keeps only each machine's last send
  (`sent_builds`), not a history of sends. Nothing in it says what those pushes moved. Their whys and the
  catalogue's policies do say it, so the controller names them, and the pull request is where that list
  is checked.
 ## What is not compared
 A push carries the machine's whole declaration (ADR 0221). A push of a recorded build also carries any
 settings or grants that changed with it. Those are not compared: such a push is judged by the builds it
 moves. A push that moves no build is never one, so a settings change pushed alone still counts.
@@ -0,0 +1,77 @@
 ---
 status: resolved
 opened: 2026-10-07
 located-in: [mesh-controller internal/builder (check.go, the build agent's merge check), mesh-lab merge-check.sh]
 fixed-by: novox/mesh-controller PR #125, novox/mesh-lab PR #59
 replay: R302
 amended-design: 03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md
 ---
 # 302. The lab's Go replays were never checked before merge
 ## Symptom
 The verdict report of novox/mesh-lab PR #58, which registered the replays of issues 296, 299 and 300,
 read via `mesh-delivery.checks`:
 > NOT CHECKED HERE: replays/ (Go) — the TypeScript toolchain holds no Go compiler
 `mesh/repo-check` passed. The pull request changed only `replays/register.go`, Go code that every core
 change's gate runs, and nothing on the build seat had compiled it, vetted it, or run its own test,
 `TestEveryReplayIsWhole`. A register that did not compile, or that registered a replay with no fix, would
 have merged green. The first gate of a core change after it would then have failed for a reason that
 had nothing to do with that change.
 ## Why it is a design issue
 To-be 45 §9 and ADR 0237 (as amended) say every repository of the mesh runs its own `merge-check.sh` in
 the toolchain it declares. The build agent read one toolchain per repository from the script's header
 and ran the script in that toolchain's container. The lab is in two languages: a TypeScript lab, and the
 Go replays register that the gate itself runs. Its script could declare only one toolchain. The script
 did say the gap, and the rule that nothing is passed silently was kept. But no step of any check could
 close it, so the replays were reviewed only by eye and proven by hand.
 ## Fix
 **A repository's own check has parts, each in its own toolchain.** Among its first twenty lines,
 `merge-check.sh` declares its toolchain as before (`# mesh-check-toolchain: <toolchain>`). It may now also
 declare further parts, one line each: `# mesh-check-also: <toolchain> <script>`. The build seat runs the
 script first, then each part in the order declared, from the repository's root and in the named
 toolchain's own container, with the same environment and none of the container runtime's socket.
 `mesh/repo-check` passes only when every part passes. A part fails the layer when it fails, when the
 repository does not hold its script, or when its path leads outside the repository. A part whose
 toolchain the mesh does not hold is an error, never a pass. The summary names each part that ran and its
 toolchain.
 **The lab declares its Go part.** `replays/merge-check.sh` runs in the Go toolchain. It checks that the
 register, the replays and the prover are gofmt'd, runs `go vet` over them (which compiles every test
 file), and runs the tests that need only Go and git: `TestEveryReplayIsWhole` and the bed's
 `TestTheBedProvesWhatTheChangeTouches`. The replays themselves still do not run there, because they raise
 containers and the check's container holds no container runtime. The script says so on every run, and
 the gate of every core change still runs the replays from the lab's main. By hand with Go at hand, the
 lab's `merge-check.sh` runs the Go part itself.
 ## How it is checked
 **The check:** `mesh/repo-check` on every mesh-lab pull request. Its report shows a
 `--- its replays/merge-check.sh (go toolchain)` section, and its summary lists
 `replays/merge-check.sh (go)` among the parts that passed. A register that does not compile or is not
 whole fails the status by name.
 - `TestARepositoryInTwoLanguagesIsCheckedInBoth` (mesh-controller `internal/builder`) reads the parts from
  the script's header. It checks that each part runs in its own toolchain's image, in order. A failing
  second part fails the layer, naming the part and what failed. A failing first part stops the check
  before the second runs. A declared script that is missing, or outside the repository, fails. A part in
  a toolchain the mesh does not hold is an error.
 - **The replay, R302**, in mesh-lab's register, is a `gate` replay of that test: on the commit before the
  fix the check it replays did not exist, and on the fix it passes.
 The trail is in [01-diagnosis.md](01-diagnosis.md).
 ## Resolved — 2026-10-08
 Fixed by novox/mesh-controller PR #125, which runs a declared part in its own toolchain, and
 novox/mesh-lab PR #59, which declares the replays as the lab's Go part. Until the controller's build
 reaches the build seat, the lab's new header line is read as a comment, and its pull requests are checked
 as before. The live check is the first lab pull request checked after the controller's build goes out, whose report must show
 the Go part.
@@ -0,0 +1,35 @@
 # 302 — diagnosis
 ## 2026-10-07: how a repository's check picks its toolchain
 The build agent's merge check (mesh-controller `internal/builder`, `Check`) runs two layers. The second,
 `mesh/repo-check`, is the repository's own `merge-check.sh`. The agent reads the script's toolchain from
 a `# mesh-check-toolchain: <toolchain>` line among its first twenty lines, defaulting to go
 (`ScriptToolchain`). It takes that toolchain's image from the toolchains the mesh holds (`ToolchainsOf`:
 the Go toolchain, and the TypeScript build image of the mesh's tools), and runs `sh merge-check.sh` in
 one container of it. The controller's script declares `go`, which is the "go toolchain" its reports name.
 The lab's declares `typescript`, because the lab is a TypeScript program.
 The TypeScript toolchain holds Node and the compiler, and no Go. The lab's script knew this. It ran the
 Go part only `if command -v go`, and otherwise printed "NOT CHECKED HERE". On the build seat it always
 took the second branch.
 ## Ruled out
 - **Go in the TypeScript toolchain's image.** That image is what every TypeScript module of the mesh is
  built on, and it is pinned and fingerprinted as their base. Adding a compiler to it for one
  repository's check would rebuild every module that stands on it, and it would hide the fact that the
  check needs two toolchains.
 - **Declaring the lab `go`.** Then its TypeScript sources and unit suite would go unchecked instead.
 - **Running the replays' Go code inside the gate only.** The gate does run them, from the lab's main, for
  core changes, and nowhere for a change to the lab itself, which is where a broken register comes in.
  Checking a lab pull request is the repository check's job, not the gate's.
 - **A second status per toolchain.** The forge's branch protection names one `mesh/repo-check`. One layer
  whose parts must all pass keeps the protection, the merge gate's reading and the delivery's checks as
  they are.
 ## The fix
 Covered in the report. The order is script first, then parts as declared. A script that fails stops the
 layer before the parts run, as `set -e` would. A part's path must be inside the repository and must
 exist, so a mistyped header line fails the check instead of being skipped.