4 modules: persistent data is a directory bind, never a named volume #54

2026-09-24T14:20:42Z

jschoubben commented

2026-09-24 14:20:42 +00:00

hq ADR 0107 / issue 115. HAL's own postgres and lavinmq both used a directory bind (./db-data, ./data) for exactly this data — the mesh's adoption of them three weeks ago switched to a named Docker volume instead, and distribution/searxng followed the same pattern since.

Already live on novox, not just proposed — this was urgent enough (operator instruction, mid-session) to build and roll out before opening this PR. Full account of what happened:

Data copied and verified before any manifest change applied: mesh-registry-data → /var/lib/mesh-registry (11G, diff -rq clean), mesh-broker-data/mesh-broker-tls → /var/lib/mesh-broker(-tls) (37M + 12K), mesh-store-data → /var/lib/mesh-store (1.7G) — mesh-store stopped cleanly first so the final copy is crash-consistent, not a live-file copy of a running postgres; diff -rq clean after.
Incident during rollout, self-inflicted and resolved: the new host directories were created via sudo, so mesh-store crash-looped on mkdir: cannot create directory '/var/lib/postgresql/data': Permission denied — a host bind mount doesn't get auto-chowned by Docker the way a named volume does. Fixed by matching the original volume's ownership (70:70, mode 1777) onto the new directory. mesh-broker and mesh-registry didn't hit this — their copied files' ownership was already compatible. Total mesh-store downtime: a few minutes, during which mesh-controller itself couldn't reach its own inventory (expected — it's also on mesh-store) and briefly restart-looped; recovered on its own once mesh-store came back.
Verified after recovery: all four post-migration databases present and correct (mesh_novox_gitea, mesh_novox_umami, mesh_novox_catalog, mesh_novox_keycloak), keycloak's realm/user counts unchanged (2 realms, 13 users, matching the count from the restore earlier tonight), mesh-broker healthy with active connections, mesh-registry serving, gitea's sidecar self-healed.
Old named volumes deliberately left in place, not deleted — the rollback path, per standing instruction to always be able to roll back.

searxng/valkey isn't assigned anywhere yet — manifest-only fix, nothing to copy.

hq ADR 0107 / issue 115. HAL's own `postgres` and `lavinmq` both used a directory bind (`./db-data`, `./data`) for exactly this data — the mesh's adoption of them three weeks ago switched to a named Docker volume instead, and `distribution`/`searxng` followed the same pattern since. **Already live on `novox`, not just proposed** — this was urgent enough (operator instruction, mid-session) to build and roll out before opening this PR. Full account of what happened: - Data copied and verified before any manifest change applied: `mesh-registry-data` → `/var/lib/mesh-registry` (11G, `diff -rq` clean), `mesh-broker-data`/`mesh-broker-tls` → `/var/lib/mesh-broker(-tls)` (37M + 12K), `mesh-store-data` → `/var/lib/mesh-store` (1.7G) — `mesh-store` stopped cleanly first so the final copy is crash-consistent, not a live-file copy of a running postgres; `diff -rq` clean after. - **Incident during rollout, self-inflicted and resolved:** the new host directories were created via `sudo`, so `mesh-store` crash-looped on `mkdir: cannot create directory '/var/lib/postgresql/data': Permission denied` — a host bind mount doesn't get auto-chowned by Docker the way a named volume does. Fixed by matching the original volume's ownership (`70:70`, mode `1777`) onto the new directory. `mesh-broker` and `mesh-registry` didn't hit this — their copied files' ownership was already compatible. Total `mesh-store` downtime: a few minutes, during which `mesh-controller` itself couldn't reach its own inventory (expected — it's also on `mesh-store`) and briefly restart-looped; recovered on its own once `mesh-store` came back. - Verified after recovery: all four post-migration databases present and correct (`mesh_novox_gitea`, `mesh_novox_umami`, `mesh_novox_catalog`, `mesh_novox_keycloak`), `keycloak`'s realm/user counts unchanged (2 realms, 13 users, matching the count from the restore earlier tonight), `mesh-broker` healthy with active connections, `mesh-registry` serving, `gitea`'s sidecar self-healed. - Old named volumes deliberately left in place, not deleted — the rollback path, per standing instruction to always be able to roll back. `searxng`/`valkey` isn't assigned anywhere yet — manifest-only fix, nothing to copy.

jschoubben added 1 commit 2026-09-24 14:20:42 +00:00

4 modules: persistent data is a directory bind, never a named volume 6a3e0ab9d3

hq ADR 0107 / issue 115. HAL's own postgres and lavinmq both used a
directory bind (./db-data, ./data) for exactly this data -- the mesh's
adoption of them, three weeks ago, switched to a named Docker volume
instead, and searxng/distribution followed the same pattern since.

A named volume survives ordinary container recreation, same as a
directory bind -- that was never the problem. The problem is everything
else: docker rm -fv, docker volume rm, and docker system prune --volumes
all target it (one flag away from the docker rm -f this migration already
uses routinely); it is invisible to every tool this migration has used all
night (ls, find, grep across /var/lib, /services); and nothing outside
Docker's own volume machinery can back it up or notice it growing.

mesh-store carries the sharpest version: every database migrated tonight,
including keycloak's, live inside it.

Data already copied and verified on novox before this merges:
- mesh-registry-data -> /var/lib/mesh-registry (11G, diff -rq clean)
- mesh-broker-data -> /var/lib/mesh-broker (37M, cp -a)
- mesh-broker-tls -> /var/lib/mesh-broker-tls (12K, cp -a)
- mesh-store-data -> /var/lib/mesh-store (1.7G) -- mesh-store stopped
  cleanly first, so the final copy is crash-consistent, not a live-file
  copy of a running postgres; diff -rq clean after.
- searxng/valkey: not yet assigned anywhere, manifest-only fix, nothing
  to copy.

Old named volumes left in place, not deleted, as the rollback path.

jschoubben merged commit 5c3781d8c2 into main

2026-09-24 14:24:15 +00:00

jschoubben deleted branch fix/persistent-data-is-a-directory-bind-not-a-volume

2026-09-24 14:24:16 +00:00

jschoubben referenced this pull request

2026-09-24 14:41:50 +00:00

postgres: declare the data directory's real owner; keycloak: use the port template #55

jschoubben referenced this issue from a commit

2026-09-26 14:12:30 +00:00

lavinmq: the broker TLS directory is the operator's, read by whoever needs it

jschoubben referenced this pull request

2026-09-26 14:12:38 +00:00