jochen
2c151e7c21
Issue 241 withdrew seven consumers in one pass on one misread file; its fix refuses a file it cannot read, and a file read whole that names nobody still withdraws everybody. A pass that would withdraw more than one consumer at once, or more than half of those it holds, now withdraws nothing: each consumer it kept is announced provisioner.failing with the class withdrawal-braked, so the controller raises it as a condition and the operator is told, and one is let go each hour while the mesh goes on not asking for them. A consumer asked for again is kept and said recovered. Postgres and keycloak carry the harness identically.
keycloak
The mesh's identity provider: one Keycloak server that provides the oidc-client provision to every
module that logs a person in. Each consumer is given one confidential OpenID Connect client in the
realm named by the assignment's issuer setting, under the client id the mesh derived for it and
the secret the mesh minted (ADR 0048); its redirect is its contributed callback under the names the
mesh composed for its endpoint. A client the mesh did not make — no mesh.provisioned attribute — is
never adopted, changed or deleted.
The admin keeps the mesh's password
The manifest mints the admin own-secret, and the server takes it from its environment only when
it creates its master realm. A database that was adopted, restored or moved already has one, and
its admin keeps the password it had. Every admin call then fails with 401 invalid_grant, and
with it every consumer's client. On 2026-10-05 that went on for a day, about 31,000 failures, seen only
in the journal (hq issue 179). Both times the fix was the same, done by hand.
The module now does that fix itself. The guard in the provider:
- checks that
adminlogs in with the mesh's secret: at start (every 15 s until the server answers), then every 5 minutes, and at once when the admin API refuses the credentials; - on a refusal —
invalid_grant, including a missing or disabled admin — and only then, repairs it inside thekeycloakcontainer with Keycloak's own recovery:kc.sh bootstrap-admin usermakes a temporary admin (on a free management port; the server holds 9000),kcadm.shcreates or re-enablesadminif it must and sets its password to the mesh's, and the temporary admin is deleted. Both passwords go in on the exec's standard input. Neither is on a command line or printed; - checks again, and says
REPAIRED the admin …in the log and emitsadmin.repaired; - when it cannot, logs
COULD NOT REPAIR …with the step that failed, emitsadmin.unrepaired, and brakes: the next automatic attempt comes 10 minutes later and the wait doubles each time, up to 6 hours. If the temporary admin may be left behind, the log and the event say so.
While the admin is refused, the provisioner does not call Keycloak. Each attempt would be one more
failed admin login, and enough of those lock the account. It still counts each attempt as a failure
of the consumer, so the provider's standing (below) reports credentials-rejected.
keycloak_admin_check reports the state, the last repair and the brake. With repair: true it
repairs a refused admin straight away, ignoring the brake, because a person asked.
A consumer failing for minutes is announced
The provisioner loop (harness.go) is shared, byte for byte, with postgres. A consumer whose create,
check or secret keeps failing for 5 minutes with no success in between is announced as
provisioner.failing, with the consumer, its machine and the error's class. The announcement repeats
every 15 minutes while the failure lasts. provisioner.recovered follows the first success
(ADR 0224), and the controller shows the latest one in status.
Tools
The realm, user, client, group and role tools (keycloak_list_realms, keycloak_create_user,
keycloak_list_clients, keycloak_assign_user_role, …), and keycloak_admin_check. A tool that
writes something announces it: user.created, user.deleted, password.reset, client.created,
group.created, role.created.
Where the code lives
One Go bundle, cmd/keycloak-provider, launched by the node's runtime. It speaks MCP over stdio
through the Go SDK, and was ported from TypeScript in 2026-10. It reaches the server on
MESH_KEYCLOAK_URL, reads the mesh's admin secret from MESH_KEYCLOAK_PASSWORD_FILE at every check,
and reaches the container named by MESH_KEYCLOAK_CONTAINER through the container-runtime
capability.
Tests
go test ./... runs against a fake Keycloak and a fake container. It covers:
- repair on refusal, with no password in argv;
- no repair while the server is unreachable;
- the brake, and the operator overriding it;
- the provisioner going quiet while the admin is refused;
- the OIDC client rules;
- the harness and its standing;
harness_same_test.go, which fails when this module'sharness.goand postgres's differ.
live_test.go runs the repair against a real, throwaway Keycloak 26 container whose admin keeps an
older password. The file's comment has the commands.