When the store does not come up, say what it said
The bootstrap's readiness wait failed on a loaded machine and reported `docker exited 1:` with nothing after the colon. The file's own comment already records this failing three times before and being fixed by running it again — "the worst kind, because it teaches people to run things twice". Raising the timeout a second time would treat the symptom. What makes a retry the only available response is a timeout that reports nothing, so the wait now prints what pg_isready says and the store's own last lines before giving up.
This commit is contained in:
@@ -21,6 +21,11 @@
|
||||
// failed that way three times in the lab -- a flaky bootstrap that a second run always fixed,
|
||||
// which is the worst kind because it teaches people to run things twice.
|
||||
//
|
||||
// It failed a fourth time on 2026-08-30, on a loaded machine, and the report was `docker exited
|
||||
// 1:` with nothing after the colon. Raising the timeout again would treat the symptom; what makes
|
||||
// a retry the only available response is a timeout that reports nothing. So the wait now says
|
||||
// what it saw before giving up.
|
||||
//
|
||||
// The store's data is a NAMED VOLUME, not a directory on the machine. A directory the host
|
||||
// creates is owned by root, and the database runs as somebody else inside the container — so it
|
||||
// could not write, and the container crash-looped. A named volume lets the image set up its own
|
||||
@@ -57,7 +62,7 @@
|
||||
"id": "store-ready",
|
||||
"type": "action",
|
||||
"in": "mesh-store",
|
||||
"command": ["sh", "-c", "for i in $(seq 1 180); do pg_isready -U postgres >/dev/null 2>&1 && exit 0; sleep 1; done; exit 1"],
|
||||
"command": ["sh", "-c", "for i in $(seq 1 180); do pg_isready -U postgres >/dev/null 2>&1 && exit 0; sleep 1; done; echo 'the store did not answer within 180s; its own last words follow'; pg_isready -U postgres; tail -n 20 /var/lib/postgresql/data/log/*.log 2>/dev/null; exit 1"],
|
||||
"verify": ["pg_isready", "-U", "postgres"]
|
||||
},
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user