Files

T

jochen 77fb1ecfb2 keycloak: port to Go and repair an admin that refuses the mesh's secret

Twice the identity provider's admin kept an older password than the one the
mesh minted (an adopted, then a moved database), and the provisioner failed
every consumer until it was repaired by hand (hq issue 179). The module now
checks the admin's login and repairs a refusal itself through the server's
bootstrap command, verifies, brakes a failed repair and announces it, and
stops asking the server while refused. Ported to Go to change it.

2026-10-06 00:13:42 +02:00

4.4 KiB

Raw Blame History

keycloak

The mesh's identity provider: one Keycloak server that provides the oidc-client provision to every module that logs a person in. Each consumer is given one confidential OpenID Connect client in the realm named by the assignment's issuer setting, under the client id the mesh derived for it and the secret the mesh minted (ADR 0048); its redirect is its contributed callback under the names the mesh composed for its endpoint. A client the mesh did not make — no mesh.provisioned attribute — is never adopted, changed or deleted.

The admin keeps the mesh's password

The manifest mints the admin own-secret, and the server takes it from its environment only when it creates its master realm. A database that was adopted, restored or moved already has one, and its admin keeps the password it had. Every admin call then fails with 401 invalid_grant, and with it every consumer's client. On 2026-10-05 that went on for a day, about 31,000 failures, seen only in the journal (hq issue 179). Both times the fix was the same, done by hand.

The module now does that fix itself. The guard in the provider:

checks that admin logs in with the mesh's secret: at start (every 15 s until the server answers), then every 5 minutes, and at once when the admin API refuses the credentials;
on a refusal — invalid_grant, including a missing or disabled admin — and only then, repairs it inside the keycloak container with Keycloak's own recovery: kc.sh bootstrap-admin user makes a temporary admin (on a free management port; the server holds 9000), kcadm.sh creates or re-enables admin if it must and sets its password to the mesh's, and the temporary admin is deleted. Both passwords go in on the exec's standard input. Neither is on a command line or printed;
checks again, and says REPAIRED the admin … in the log and emits admin.repaired;
when it cannot, logs COULD NOT REPAIR … with the step that failed, emits admin.unrepaired, and brakes: the next automatic attempt comes 10 minutes later and the wait doubles each time, up to 6 hours. If the temporary admin may be left behind, the log and the event say so.

While the admin is refused, the provisioner does not call Keycloak. Each attempt would be one more failed admin login, and enough of those lock the account. It still counts each attempt as a failure of the consumer, so the provider's standing (below) reports credentials-rejected.

keycloak_admin_check reports the state, the last repair and the brake. With repair: true it repairs a refused admin straight away, ignoring the brake, because a person asked.

A consumer failing for minutes is announced

The provisioner loop (harness.go) is shared, byte for byte, with postgres. A consumer whose create, check or secret keeps failing for 5 minutes with no success in between is announced as provisioner.failing, with the consumer, its machine and the error's class. The announcement repeats every 15 minutes while the failure lasts. provisioner.recovered follows the first success (ADR 0224), and the controller shows the latest one in status.

Tools

The realm, user, client, group and role tools (keycloak_list_realms, keycloak_create_user, keycloak_list_clients, keycloak_assign_user_role, …), and keycloak_admin_check. A tool that writes something announces it: user.created, user.deleted, password.reset, client.created, group.created, role.created.

Where the code lives

One Go bundle, cmd/keycloak-provider, launched by the node's runtime. It speaks MCP over stdio through the Go SDK, and was ported from TypeScript in 2026-10. It reaches the server on MESH_KEYCLOAK_URL, reads the mesh's admin secret from MESH_KEYCLOAK_PASSWORD_FILE at every check, and reaches the container named by MESH_KEYCLOAK_CONTAINER through the container-runtime capability.

Tests

go test ./... runs against a fake Keycloak and a fake container. It covers:

repair on refusal, with no password in argv;
no repair while the server is unreachable;
the brake, and the operator overriding it;
the provisioner going quiet while the admin is refused;
the OIDC client rules;
the harness and its standing;
harness_same_test.go, which fails when this module's harness.go and postgres's differ.

live_test.go runs the repair against a real, throwaway Keycloak 26 container whose admin keeps an older password. The file's comment has the commands.

4.4 KiB Raw Blame History