mosquitto: run-once dynsec bootstrap + fix the exit-0-on-failure provisioner #8

2026-09-05T23:03:26Z

jschoubben commented

2026-09-05 23:03:26 +00:00

Two things, both lab-verified green end-to-end:

Run-once bootstrap (ADR 0052): an init container seeds mosquitto's dynsec admin client into dynamic-security.json (offline, via mesh-tools run + mosquitto_ctrl) and chowns it to the broker's uid 1883, placed before the broker server container. The broker now comes up seeded instead of crash-looping.
Provisioner fix: mosquitto_ctrl dynsec exits 0 even on failure (verified live — "Client not found"/"Not authorized"/"Unable to connect" all exit 0). The client trusted the exit code, so the existence-probe always returned true, createClient never ran, and the provisioner logged success for a consumer client that never landed. Now ctl() scans stderr and throws on the real error shapes; idempotent re-provision and rotation still pass; a real auth/connection failure now errors instead of silently "succeeding."

Proven by assigned-catalogue-mqtt (green): the broker comes up seeded and a consumer's scoped MQTT client actually lands in the store.

https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF

Two things, both lab-verified green end-to-end: 1. **Run-once bootstrap (ADR 0052):** an init container seeds mosquitto's dynsec admin client into `dynamic-security.json` (offline, via `mesh-tools run` + `mosquitto_ctrl`) and chowns it to the broker's uid 1883, placed before the broker `server` container. The broker now comes up seeded instead of crash-looping. 2. **Provisioner fix:** `mosquitto_ctrl dynsec` **exits 0 even on failure** (verified live — "Client not found"/"Not authorized"/"Unable to connect" all exit 0). The client trusted the exit code, so the existence-probe always returned true, `createClient` never ran, and the provisioner logged success for a consumer client that never landed. Now `ctl()` scans stderr and throws on the real error shapes; idempotent re-provision and rotation still pass; a real auth/connection failure now errors instead of silently "succeeding." Proven by `assigned-catalogue-mqtt` (green): the broker comes up seeded and a consumer's scoped MQTT client actually lands in the store. https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF

jschoubben added 2 commits 2026-09-05 23:03:26 +00:00

mosquitto: seed dynsec with a run-once bootstrap before the broker ec1e7189a3

The Dynamic Security plugin refuses to start the broker unless
dynamic-security.json already holds an admin client, and no reconcile loop
seeds it (novox/hq ADR 0052). Add a run-once init container, declared before
the server container, that runs mosquitto's own bootstrap entrypoint in the
runtime image: it seeds the store offline via mosquitto_ctrl and exits, and
the host gates the broker on its completion.

The bootstrap hands the seeded file to the broker's user (uid 1883, chown +
0600): the broker must read the seed at startup AND persist to it as clients
come and go, but the init container runs as root and would otherwise leave a
file the broker can neither read nor rewrite. This is the ownership question
ADR 0052 left for the lab to settle. It seeds only when the file is absent, so
what the running plugin grows is never clobbered (issue 035).

Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF

mosquitto: don't read a rejected dynsec command as success e58cbc1c46

mosquitto_ctrl's dynsec subcommands exit 0 even when they fail — a
"Client not found", an "already exists", a "Connection error: Not
authorized", an "Unable to connect" all return status 0 and report the
failure only as a line of text (verified live against 2.0.11). ctl()
trusted the exit code, so clientExists()'s getClient probe never threw
and always returned true; createScopedClient therefore took the
setClientPassword branch, never ran createClient, and the consumer's
client never landed in the dynsec store — while the provisioner logged
it as provisioned. That is the assigned-catalogue-mqtt failure.

ctl() now scans the combined stdout/stderr for the tool's error markers
and raises a match as the failure it is. Surfacing those errors exposed
two calls that only "worked" by being swallowed: addRoleACL re-run
reports "already exists" (now ignored like createRole), and addClientRole
re-run reports a bare "Internal error" that cannot be told from a real
fault — so the binding is checked with clientHasRole and only added when
absent. Fresh provision, idempotent re-provision, password rotation, bad
admin auth and unreachable broker all verified against a live broker.

Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF

jschoubben merged commit 3b5f214f3d into main

2026-09-05 23:06:08 +00:00

jschoubben deleted branch feat/mosquitto-bootstrap

2026-09-05 23:06:08 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-catalog#8