jschoubben
ba14e2b629
Two things changed at once: a fourth image in the scenario, and scenario machines raised from 1 GiB to 2 GiB. The bootstrap then failed every time, and the image was blamed. Removing the image did not fix it. Removing the memory increase did — nine assertions pass again on three images with the machines back at 1 GiB. Three machines at 2 GiB on a host doing other work contend enough that the store container does not come up at all. The ordinary lesson, and it still caught me: two changes together, the failure attributed to the plausible one, and an issue written recording the wrong cause. What found it was reverting to the exact last-known-good state rather than reverting the suspicious change. What remains untested is whether a fourth image alone is fine. Probably. Nothing has measured it, and the honest state of this issue is that what it was opened about was never demonstrated.
71 lines
3.2 KiB
Markdown
71 lines
3.2 KiB
Markdown
---
|
|
status: open
|
|
opened: 2026-08-30
|
|
located-in: [mesh-lab]
|
|
fixed-by:
|
|
amended-design:
|
|
---
|
|
|
|
# 012 — Raising a scenario machine's memory stops the substrate coming up
|
|
|
|
*Renamed after the first diagnosis turned out to be wrong. What that was, and how it was wrong, is
|
|
below — it is the more useful half of this report.*
|
|
|
|
## Symptom
|
|
|
|
Adding a fourth image to the two-machine scenario made the bootstrap fail every time. The store
|
|
container was created, and its readiness check then failed for the full three minutes with
|
|
**no output at all**:
|
|
|
|
```
|
|
failed store-ready (in mesh-store: ... pg_isready ... exit 1):
|
|
the action ran without error and its own verify still fails: docker exited 1:
|
|
```
|
|
|
|
Empty after the colon. The check runs `pg_isready` inside the container and prints the store's own
|
|
last lines when it gives up; producing nothing means **the container was not running**, which is a
|
|
different fault from a database that is slow to start.
|
|
|
|
Three images: the scenario raises, both machines join, and nine assertions pass. Four: it never
|
|
gets past the store. Reverting the fourth image restores it.
|
|
|
|
## The first diagnosis was wrong, and this is why it is worth writing down
|
|
|
|
**Two things changed at once.** A fourth image was added to the scenario, and — reasoning that a
|
|
machine running a database, a broker and the control plane at once is genuinely small — scenario
|
|
machines were raised from 1 GiB to 2 GiB. The bootstrap then failed every time, and the fourth
|
|
image was blamed.
|
|
|
|
**Removing the image did not fix it. Removing the memory increase did.** Nine assertions pass again
|
|
with the extra memory reverted, on a scenario with three images. So the cause is the memory change:
|
|
three machines at 2 GiB, on a host also running other work, contend enough that the store container
|
|
does not come up at all.
|
|
|
|
**The lesson is the ordinary one and it still caught me:** two changes went in together, the failure
|
|
was attributed to the plausible one, and an issue was written recording the wrong cause. What found
|
|
it was reverting to the exact last-known-good state rather than reverting the suspicious change.
|
|
|
|
**What remains untested** is whether a fourth image alone is fine. It probably is. Nothing has
|
|
measured it, and the honest state of this issue is that the thing it was opened about was never
|
|
demonstrated.
|
|
|
|
## What was worth keeping
|
|
|
|
**A better diagnostic.** The readiness check now prints what it saw before giving up. That is what
|
|
showed the output was empty, which is what said *the container is not running* rather than *the
|
|
database is slow* — and which will make the next occurrence of this a diagnosis instead of a
|
|
retry.
|
|
|
|
## What this blocks
|
|
|
|
The mesh running its **own artifact store** — a registry as a module — needs a registry image on
|
|
the machine so the module can mirror one, which is the fourth image. The module is written and its
|
|
manifest is accepted; what has not been proven is a machine assigned it serving artifacts to
|
|
another machine.
|
|
|
|
## How this will be checked
|
|
|
|
A scenario raised with four images comes up and passes the assertions that three do — which is
|
|
what was never actually established. Until then the artifact-store test is not in the shared
|
|
scenario, with a note saying where it went and why.
|