53 lines
2.4 KiB
Markdown
53 lines
2.4 KiB
Markdown
---
|
|
status: resolved
|
|
opened: 2026-10-04
|
|
located-in:
|
|
- mesh-controller
|
|
fixed-by:
|
|
- policy: `upgrade build-agent roll-out` (2026-10-04)
|
|
amended-design:
|
|
---
|
|
|
|
# 221 — A build machine learns a new builder only from a push
|
|
|
|
## What was observed
|
|
|
|
2026-10-04. A controller merge changed how TypeScript bundles are built: one file per entrypoint
|
|
instead of a package directory. Its plan finished, and six bundles were rebuilt right after. They
|
|
came out in the old shape, because the build machines still ran the previous builder. They got the
|
|
new one only from the next push. Rebuilt after that push, the same six came out right.
|
|
|
|
The same order showed in the bus grants the same night. A push sent while the controller was still
|
|
the previous build composed grants with the previous code. A second push was needed after the new
|
|
controller had started.
|
|
|
|
## Why it matters beyond this instance
|
|
|
|
A merge to the controller is not in effect when its plan says done. Anything built or pushed in the
|
|
gap uses the old code and looks current. Today only the operator knows to push first and build
|
|
second, and even the operator forgot.
|
|
|
|
## Where to look
|
|
|
|
Whether a controller plan should end by delivering itself to the build machines and the control
|
|
machine, or whether a build should refuse a builder older than the controller that asked for it.
|
|
**How it is checked:** after a controller merge, a build asked right after its plan finishes runs
|
|
the new builder.
|
|
|
|
## Located
|
|
|
|
The mechanism existed: a module whose upgrade policy is `roll-out` is sent to its machines when its
|
|
tier is built, and the plan's next tier waits until it is applied. The build agent's policy was
|
|
`record` — built, never sent — as was that of 76 other modules, which is why every rollout on
|
|
2026-10-03 needed a push by hand. The build agent was set to `roll-out`, one machine at a time,
|
|
stopping at the first failure. **How it is checked:** the next controller merge's plan sends the
|
|
build agent to the build machines before its last tier, and a build asked right after the plan
|
|
finishes runs the new builder; then this moves to resolved. Whether the other modules roll out is
|
|
the operator's policy, not this issue's.
|
|
|
|
## Resolved
|
|
|
|
Proven 2026-10-04: the next controller merge's plan logged that its first tier was built and the
|
|
build agent sent to all four build machines, and only then asked its next tier. The builder change
|
|
in that merge reached the build machines without a hand push.
|