When the store does not come up, say what it said

The bootstrap's readiness wait failed on a loaded machine and reported
`docker exited 1:` with nothing after the colon. The file's own comment
already records this failing three times before and being fixed by
running it again — "the worst kind, because it teaches people to run
things twice".

Raising the timeout a second time would treat the symptom. What makes a
retry the only available response is a timeout that reports nothing, so
the wait now prints what pg_isready says and the store's own last lines
before giving up.
This commit is contained in:
2026-08-30 04:06:10 +02:00
parent c57087d75d
commit b2ecf25594
+6 -1
View File
@@ -21,6 +21,11 @@
// failed that way three times in the lab -- a flaky bootstrap that a second run always fixed, // failed that way three times in the lab -- a flaky bootstrap that a second run always fixed,
// which is the worst kind because it teaches people to run things twice. // which is the worst kind because it teaches people to run things twice.
// //
// It failed a fourth time on 2026-08-30, on a loaded machine, and the report was `docker exited
// 1:` with nothing after the colon. Raising the timeout again would treat the symptom; what makes
// a retry the only available response is a timeout that reports nothing. So the wait now says
// what it saw before giving up.
//
// The store's data is a NAMED VOLUME, not a directory on the machine. A directory the host // The store's data is a NAMED VOLUME, not a directory on the machine. A directory the host
// creates is owned by root, and the database runs as somebody else inside the container — so it // creates is owned by root, and the database runs as somebody else inside the container — so it
// could not write, and the container crash-looped. A named volume lets the image set up its own // could not write, and the container crash-looped. A named volume lets the image set up its own
@@ -57,7 +62,7 @@
"id": "store-ready", "id": "store-ready",
"type": "action", "type": "action",
"in": "mesh-store", "in": "mesh-store",
"command": ["sh", "-c", "for i in $(seq 1 180); do pg_isready -U postgres >/dev/null 2>&1 && exit 0; sleep 1; done; exit 1"], "command": ["sh", "-c", "for i in $(seq 1 180); do pg_isready -U postgres >/dev/null 2>&1 && exit 0; sleep 1; done; echo 'the store did not answer within 180s; its own last words follow'; pg_isready -U postgres; tail -n 20 /var/lib/postgresql/data/log/*.log 2>/dev/null; exit 1"],
"verify": ["pg_isready", "-U", "postgres"] "verify": ["pg_isready", "-U", "postgres"]
}, },
{ {