From 9faa985be3cfa9f50fa6555f461495775eea7a3f Mon Sep 17 00:00:00 2001 From: jochen Date: Wed, 23 Sep 2026 02:47:50 +0200 Subject: [PATCH] Issue 099: a module's image pin ages into a downgrade Three modules in a row on one machine; the first was found by taking it and cost a three-minute outage. The runbook's answer is a rule a person must remember, which is the shape this repository says not to settle for. --- .../00-report.md | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 04-ISSUES/099-a-modules-image-pin-ages-into-a-downgrade/00-report.md diff --git a/04-ISSUES/099-a-modules-image-pin-ages-into-a-downgrade/00-report.md b/04-ISSUES/099-a-modules-image-pin-ages-into-a-downgrade/00-report.md new file mode 100644 index 0000000..88a2b09 --- /dev/null +++ b/04-ISSUES/099-a-modules-image-pin-ages-into-a-downgrade/00-report.md @@ -0,0 +1,62 @@ +--- +status: open +opened: 2026-09-23 +located-in: [] +fixed-by: +amended-design: +--- + +# 099 — A module's image pin ages into a downgrade, and taking it over is where that is discovered + +## What was observed + +Three modules in a row, on the same machine, 2026-09-22 and 2026-09-23. + +A catalogue module pins its image by digest, which is right: a module must run the same software +everywhere, and a floating tag is not a version. The digest is chosen when the module is written. +The machine whose service it takes over runs whatever *its* floating tag resolved to, and that has +moved on since. + +| | the module pinned | the machine ran | +|---|---|---| +| the forge | 1.22.6 | 1.27.3 | +| the package registry | 6.10.1 | 6.10.3 | +| the analytics service | built 2026-08-20 | built 2026-09-17 | + +The first was found by taking it. The service came up on the older build, refused to start against a +database its newer self had migrated, and **was down about three minutes** before the cutover was +rolled back. The other two were found by hand afterwards, only because the first had made it a +thing to check. + +Nothing in the mesh compares the two. The host is holding the very container it is about to +replace — it knows the image that container runs, it reports it as held, and it replaces it without +a word about what changes. `take` says a container was replaced. It does not say *with what*. + +## Why it matters beyond this instance + +The pin is not wrong when it is written and it is not wrong for a fresh install; it goes wrong by +**sitting still while the world moves**, and the moment it is discovered is the moment a running +service is replaced by it. Every module in the catalogue has this property, and the ones most +likely to have drifted are the ones written earliest — which, in a migration, is most of them. + +A downgrade is not an ordinary failure. A service that has already migrated its own data forward +does not start on an older build; it may also start and quietly write the older format. Rolling +back is a restore, not an undo. + +The runbook now says to compare versions before every cutover, and that is the shape of rule this +repository says it should not settle for: a rule stated where a person will read it, enforced by +nothing, checked by remembering. It was checked by remembering twice and missed once, which is the +expected rate. + +## Open questions + +- Should the mesh refuse to replace a held container with an **older** image and make the operator + say so explicitly — and what orders two digests, given neither a digest nor a registry tells you + which came first? An image's own creation date is in its config, available before pulling. +- Should `take` preview at all? It is the step that replaces a running service and it previews + nothing; the whole-node flip previews more than the per-service cutover it is made of. +- Should the catalogue know when a module's pin is behind its upstream, the way the mesh already + knows when a module is behind its **source**? `build --behind` answers the second question and + nothing answers the first. +- Is a digest pin the right thing for a module that takes over an existing service at all, or + should a cutover be able to say *keep what is running* and record what that was? -- 2.54.0