Issue 099: a module's image pin ages into a downgrade, and taking it over is where that is discovered #85

2026-09-23T00:47:58Z

jschoubben commented

2026-09-23 00:47:58 +00:00

Three modules in a row on the control-node pinned an older image than the machine was running: the forge (1.22.6 against 1.27.3), the package registry (6.10.1 against 6.10.3), the analytics service (a month behind). The first was found by taking it — the service came up on the older build, would not start against a database its newer self had migrated, and was down about three minutes.

The pin is not wrong when written. It goes wrong by sitting still while the world moves, and the moment it is discovered is the moment a running service is replaced by it.

The host is holding the very container it is about to replace. It knows the image that container runs and replaces it without a word about what changes. The runbook now says to compare versions before every cutover — a rule stated where a person will read it and enforced by nothing, which is the shape this repository says not to settle for.

Three modules in a row on the control-node pinned an older image than the machine was running: the forge (1.22.6 against 1.27.3), the package registry (6.10.1 against 6.10.3), the analytics service (a month behind). The first was found by taking it — the service came up on the older build, would not start against a database its newer self had migrated, and was down about three minutes. The pin is not wrong when written. It goes wrong by sitting still while the world moves, and the moment it is discovered is the moment a running service is replaced by it. The host is holding the very container it is about to replace. It knows the image that container runs and replaces it without a word about what changes. The runbook now says to compare versions before every cutover — a rule stated where a person will read it and enforced by nothing, which is the shape this repository says not to settle for.

jschoubben added 1 commit 2026-09-23 00:47:59 +00:00

Issue 099: a module's image pin ages into a downgrade 9faa985be3

Three modules in a row on one machine; the first was found by taking it and cost a
three-minute outage. The runbook's answer is a rule a person must remember, which is
the shape this repository says not to settle for.

jschoubben merged commit e9e5df36bd into main

2026-09-23 00:48:02 +00:00

jschoubben deleted branch issues/099-a-pin-ages-into-a-downgrade

2026-09-23 00:48:03 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/hq#85