Issue 219: an older build request never replaces a newer one's artifact #249

2026-10-03T22:22:30Z

mesh-admin commented

2026-10-03 22:22:30 +00:00

novox/hq issue 219.

Root cause

Builds of one module in flight together finish in any order, and the controller took whatever it heard last as what the module is:

takeIn → RegisterModule (internal/inventory/catalogue.go) overwrote module.manifest unconditionally. The resolved manifest carries the image digests the composer deploys, so the last outcome heard won.
Held / BuiltAgainst / ReadRepositories / Announceable (internal/inventory/builds.go) ordered by build.at, which is when the outcome was recorded (that is, when the build finished).
Plans had the same problem: planBuilt marked a module built in every open plan by module name, and settleFromRecords took any build recorded after the ask. So an earlier plan's leftover build could settle a later plan.

Fix

A build is ordered by when it was asked. The controller already writes that time into the correlation id (build-<unix nanos>), and every outcome echoes the id, so no wire change is needed.

link.NewBuildID and link.BuildAskedAt are the only places that write and read that shape. Any other id has no request time and keeps the old order.
Migration 0055 adds build.asked, backfilled from existing ids, and module.built_asked.
RegisterModule only replaces a module whose built_asked is null or no newer. Otherwise it returns ErrSuperseded, and the build is still recorded. A manifest handed over by hand counts as asked now.
Held and the other queries use coalesce(asked, at) desc, at desc.
planBuilt and settleFromRecords ignore a build that was asked before the plan's own ask. The daemon treats a superseded outcome as not a failure.

Tests

inventory: TestAnOlderRequestFinishingLaterIsNotWhatTheModuleHolds, TestABuildWithNoKnownRequestTimeIsOrderedByWhenItWasRecorded, TestARegistrationFromAnOlderRequestDoesNotReplaceANewerOne
cmd: TestAnOlderBuildHeardLaterDoesNotReplaceTheNewer, TestABuildIDSaysWhenItWasAsked, plus extended TestAPlanSettlesAnAskedBuildFromTheRecords

With the ordering, the registration guard and the settle filter reverted, all four behavioural tests fail. Full suite passes against postgres 16, apart from the known unrelated TestTheResolverIsToldEveryMachineOnTheNetworkAndToldAgainWhenOneLeaves.

Note: the repo has no consolidated schema, only migrations, so 0055 is the only schema change.

novox/hq issue 219. ## Root cause Builds of one module in flight together finish in any order, and the controller took whatever it heard last as what the module is: - `takeIn` → `RegisterModule` (internal/inventory/catalogue.go) overwrote `module.manifest` unconditionally. The resolved manifest carries the image digests the composer deploys, so the last outcome heard won. - `Held` / `BuiltAgainst` / `ReadRepositories` / `Announceable` (internal/inventory/builds.go) ordered by `build.at`, which is when the outcome was recorded (that is, when the build finished). - Plans had the same problem: `planBuilt` marked a module built in every open plan by module name, and `settleFromRecords` took any build recorded after the ask. So an earlier plan's leftover build could settle a later plan. ## Fix A build is ordered by when it was **asked**. The controller already writes that time into the correlation id (`build-<unix nanos>`), and every outcome echoes the id, so no wire change is needed. - `link.NewBuildID` and `link.BuildAskedAt` are the only places that write and read that shape. Any other id has no request time and keeps the old order. - Migration 0055 adds `build.asked`, backfilled from existing ids, and `module.built_asked`. - `RegisterModule` only replaces a module whose `built_asked` is null or no newer. Otherwise it returns `ErrSuperseded`, and the build is still recorded. A manifest handed over by hand counts as asked now. - `Held` and the other queries use `coalesce(asked, at) desc, at desc`. - `planBuilt` and `settleFromRecords` ignore a build that was asked before the plan's own ask. The daemon treats a superseded outcome as not a failure. ## Tests - inventory: `TestAnOlderRequestFinishingLaterIsNotWhatTheModuleHolds`, `TestABuildWithNoKnownRequestTimeIsOrderedByWhenItWasRecorded`, `TestARegistrationFromAnOlderRequestDoesNotReplaceANewerOne` - cmd: `TestAnOlderBuildHeardLaterDoesNotReplaceTheNewer`, `TestABuildIDSaysWhenItWasAsked`, plus extended `TestAPlanSettlesAnAskedBuildFromTheRecords` With the ordering, the registration guard and the settle filter reverted, all four behavioural tests fail. Full suite passes against postgres 16, apart from the known unrelated `TestTheResolverIsToldEveryMachineOnTheNetworkAndToldAgainWhenOneLeaves`. Note: the repo has no consolidated schema, only migrations, so 0055 is the only schema change.

mesh-admin added 1 commit 2026-10-03 22:22:31 +00:00

An older build request never replaces a newer one's artifact 9745c1ab31

Builds of one module in flight together finish in any order, and the mesh
took whatever it heard last as what the module is: RegisterModule overwrote
the module's manifest unconditionally, and Held/BuiltAgainst/ReadRepositories
ordered builds by when they were recorded. A postgres build asked before the
mesh-tools runtime fix finished after the one asked after it, and the next
push deployed the stale image (novox/hq issue 219).

A build is now ordered by when it was asked, read from the build-<nanos> id
the controller writes: build.asked and module.built_asked (migration 0055).
A registration from an earlier request than the module's current one is
recorded and refused as superseded. A plan takes as its outcome only a build
asked at or after its own ask, so an earlier plan's leftover build cannot
settle a later plan. Ids of any other shape keep the old order.

mesh-admin merged commit 6a803ea5b3 into main

2026-10-03 22:23:31 +00:00

mesh-admin deleted branch fix/issue-219-an-older-build-never-replaces-a-newer

2026-10-03 22:23:32 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-controller#249