Issues 219-221: the builder's ordering gaps #341

Merged
mesh-admin merged 1 commits from issues/219-221-the-builder into main 2026-10-03 22:16:13 +00:00
3 changed files with 100 additions and 0 deletions
@@ -0,0 +1,37 @@
---
status: diagnosing
opened: 2026-10-04
located-in:
- mesh-controller
fixed-by:
amended-design:
---
# 219 — An older build that finishes later replaces a newer one
## What was observed
2026-10-03. Two merges to the runtime module came minutes apart. Each plan asked for every module
built on the runtime image to be rebuilt. One module's two builds were of the same source and
differed only in the runtime image they stood on:
| Build | Requested | Finished | Stood on |
|---|---|---|---|
| asked by the first plan | 21:33 | 22:04 | the runtime image before the fix |
| asked by the second plan | 21:49 | 21:56 | the runtime image with the fix |
The older request finished last, and its image became the module's current artifact. The next push
deployed it, and the module's runtime crash-looped on a fault the newer image had already fixed
([issue 218](../218-a-mesh-seat-is-answered-by-a-module-that-does-not-hold-it/00-report.md)).
## Why it matters beyond this instance
Which build is current should follow what it was built from, not which build machine was slowest.
Whenever two plans overlap, which happens on any busy evening, a fix can be silently reverted by a
build that started before it existed. Every passing check still passes.
## Where to look
How a finished build is recorded and how a module's current artifact is chosen. **How it is checked:**
a test in which an older request completes after a newer one for the same artifact, and the newer
stays current.
@@ -0,0 +1,30 @@
---
status: open
opened: 2026-10-04
located-in: []
fixed-by:
amended-design:
---
# 220 — A delivered bundle keeps the files of the one before
## What was observed
2026-10-04. A tools bundle was rebuilt as one self-contained file per entrypoint
([ADR 0193](../../02-DECISIONS/0193-every-bundle-the-runtime-serves-is-launched-and-the-runtime-knows-no-language.md)),
so it no longer carries a package directory. On the machine it was delivered to, its directory still
held the package directory and a compiled file from the earlier delivery, dated hours before the
new files. The new files were written over the old directory, and nothing removed what the new
bundle no longer contains.
## Why it matters beyond this instance
A bundle on disk should be exactly the artifact that was built. Leftover files can be imported by
code that should no longer find them. A fix that removes a file then works on a fresh machine and
fails on every machine that ran an earlier version. It also makes "what runs here" impossible to
read from the artifact.
## Where to look
How the host unpacks a bundle into its directory. **How it is checked:** deliver a bundle, then a
version without one of its files, and the file is gone.
@@ -0,0 +1,33 @@
---
status: open
opened: 2026-10-04
located-in: []
fixed-by:
amended-design:
---
# 221 — A build machine learns a new builder only from a push
## What was observed
2026-10-04. A controller merge changed how TypeScript bundles are built: one file per entrypoint
instead of a package directory. Its plan finished, and six bundles were rebuilt right after. They
came out in the old shape, because the build machines still ran the previous builder. They got the
new one only from the next push. Rebuilt after that push, the same six came out right.
The same order showed in the bus grants the same night. A push sent while the controller was still
the previous build composed grants with the previous code. A second push was needed after the new
controller had started.
## Why it matters beyond this instance
A merge to the controller is not in effect when its plan says done. Anything built or pushed in the
gap uses the old code and looks current. Today only the operator knows to push first and build
second, and even the operator forgot.
## Where to look
Whether a controller plan should end by delivering itself to the build machines and the control
machine, or whether a build should refuse a builder older than the controller that asked for it.
**How it is checked:** after a controller merge, a build asked right after its plan finishes runs
the new builder.