2.1 KiB
status: located opened: 2026-10-04 located-in:
- mesh-controller fixed-by:
- policy:
upgrade build-agent roll-out(2026-10-04) amended-design:
221 — A build machine learns a new builder only from a push
What was observed
2026-10-04. A controller merge changed how TypeScript bundles are built: one file per entrypoint instead of a package directory. Its plan finished, and six bundles were rebuilt right after. They came out in the old shape, because the build machines still ran the previous builder. They got the new one only from the next push. Rebuilt after that push, the same six came out right.
The same order showed in the bus grants the same night. A push sent while the controller was still the previous build composed grants with the previous code. A second push was needed after the new controller had started.
Why it matters beyond this instance
A merge to the controller is not in effect when its plan says done. Anything built or pushed in the gap uses the old code and looks current. Today only the operator knows to push first and build second, and even the operator forgot.
Where to look
Whether a controller plan should end by delivering itself to the build machines and the control machine, or whether a build should refuse a builder older than the controller that asked for it. How it is checked: after a controller merge, a build asked right after its plan finishes runs the new builder.
Located
The mechanism existed: a module whose upgrade policy is roll-out is sent to its machines when its
tier is built, and the plan's next tier waits until it is applied. The build agent's policy was
record — built, never sent — as was that of 76 other modules, which is why every rollout on
2026-10-03 needed a push by hand. The build agent was set to roll-out, one machine at a time,
stopping at the first failure. How it is checked: the next controller merge's plan sends the
build agent to the build machines before its last tier, and a build asked right after the plan
finishes runs the new builder; then this moves to resolved. Whether the other modules roll out is
the operator's policy, not this issue's.