keycloak: port to Go and repair a refused admin; providers announce a failing consumer #80

2026-10-05T22:14:21Z

mesh-admin commented

2026-10-05 22:14:21 +00:00

hq issue 179 recurred: the keycloak database moved on 2026-10-05, the realm's admin kept an older password, and the provisioner failed every consumer ~31,000 times until fixed by hand.

postgres (provisioner loop, shared): announces provisioner.failing after 5 min without a success (create, holds check, or unreadable secret), repeats every 15 min, provisioner.recovered on success / withdrawal / first success after a restart. Error class: credentials-rejected, unreachable, secret-unreadable, refused.

keycloak: ported to Go (cmd/keycloak-provider: client, oidc, provisioner, all tools, events) with the identical loop (harness_same_test.go in both modules fails on drift). New admin guard: checks the admin's login at start, every 5 min, and on any 401; on invalid_grant runs kc.sh bootstrap-admin + kcadm.sh inside the container via docker exec -i (secrets on stdin only), creates/enables admin if needed, removes the temporary admin, verifies, emits admin.repaired. A failed repair emits admin.unrepaired, logs loudly, and is braked 10m doubling to 6h; while refused the provisioner does not call Keycloak (no lockout) but still counts failures (credentials-rejected). Tool keycloak_admin_check {repair}. Resource ids unchanged; build artifact now Go.

Tests: go test -race green (keycloak 35, postgres 33); module check ok for both. Live test (gated, throwaway Postgres + Keycloak 26.0.8): old password, disabled admin, deleted admin — all repaired in 8–12 s, temp admin gone, second pass no-op.

Merge after mesh-controller #70 (derives the emit permission) and mesh-host.

hq issue 179 recurred: the keycloak database moved on 2026-10-05, the realm's admin kept an older password, and the provisioner failed every consumer ~31,000 times until fixed by hand. **postgres (provisioner loop, shared):** announces `provisioner.failing` after 5 min without a success (create, holds check, or unreadable secret), repeats every 15 min, `provisioner.recovered` on success / withdrawal / first success after a restart. Error class: credentials-rejected, unreachable, secret-unreadable, refused. **keycloak:** ported to Go (`cmd/keycloak-provider`: client, oidc, provisioner, all tools, events) with the identical loop (`harness_same_test.go` in both modules fails on drift). New admin guard: checks the admin's login at start, every 5 min, and on any 401; on invalid_grant runs `kc.sh bootstrap-admin` + `kcadm.sh` inside the container via `docker exec -i` (secrets on stdin only), creates/enables admin if needed, removes the temporary admin, verifies, emits `admin.repaired`. A failed repair emits `admin.unrepaired`, logs loudly, and is braked 10m doubling to 6h; while refused the provisioner does not call Keycloak (no lockout) but still counts failures (credentials-rejected). Tool `keycloak_admin_check {repair}`. Resource ids unchanged; build artifact now Go. Tests: go test -race green (keycloak 35, postgres 33); `module check` ok for both. Live test (gated, throwaway Postgres + Keycloak 26.0.8): old password, disabled admin, deleted admin — all repaired in 8–12 s, temp admin gone, second pass no-op. Merge after mesh-controller #70 (derives the emit permission) and mesh-host.

mesh-admin added 2 commits 2026-10-05 22:14:21 +00:00

postgres: announce a consumer failed for minutes, and its recovery 6f1e2f5a0d

A provider failed every consumer for a day and said so only in its journal
(hq issue 179). The provisioner loop now emits provisioner.failing after five
minutes without a success — create, check or secret — and repeats it every
fifteen; provisioner.recovered on the next success, on withdrawal, and on the
first success after a restart, so the controller can name it in status
(hq ADR 0224).

keycloak: port to Go and repair an admin that refuses the mesh's secret 77fb1ecfb2

Twice the identity provider's admin kept an older password than the one the
mesh minted (an adopted, then a moved database), and the provisioner failed
every consumer until it was repaired by hand (hq issue 179). The module now
checks the admin's login and repairs a refusal itself through the server's
bootstrap command, verifies, brakes a failed repair and announces it, and
stops asking the server while refused. Ported to Go to change it.

jschoubben added 1 commit 2026-10-05 22:17:37 +00:00

keycloak repair: remove the temporary admin even when the bootstrap failed after making it 48d4188927

The bootstrap once created the temporary admin and then failed on a held port; marked only after
it succeeded, the cleanup did not know the admin existed and left it.

mesh-admin merged commit 78328d4ab2 into main

2026-10-05 22:39:02 +00:00

mesh-admin deleted branch feat/identity-provider-admin-safety-nets

2026-10-05 22:39:03 +00:00

mesh-admin commented

2026-10-06 23:08:46 +00:00

Delivery novox/mesh-catalog@78328d4ab26d — held since 2026-10-06T23:05:28Z

Held for a person: merged without a passing check: only a person decides that it goes on — mesh-delivery.release with why lets it go on.

Delivery plan — builds keycloak, postgres, sent nowhere; no bus step; 3 wait(s) for a person

tier 0: keycloak, postgres
ace: waits for a person: postgres
novox: waits for a person: keycloak, postgres
keycloak waits for a person: its policy records (module)
keycloak provides oidc-client: its consumers are sent again after it
keycloak keeps data (ADR 0233): what it holds is backed up before it moves
postgres waits for a person: its policy records (module)
postgres provides postgres-database: its consumers are sent again after it
postgres keeps data (ADR 0233): what it holds is backed up before it moves

Transitions

2026-10-06 23:05 (new) → proposed (announced): novox/mesh-catalog#80 merged; its head was not heard by this owner, so it is made from the forge's word
2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-catalog@78328d4ab26d` — **held** since 2026-10-06T23:05:28Z Held for a person: merged without a passing check: only a person decides that it goes on — `mesh-delivery.release` with why lets it go on. **Delivery plan** — builds keycloak, postgres, sent nowhere; no bus step; 3 wait(s) for a person - tier 0: keycloak, postgres - ace: waits for a person: postgres - novox: waits for a person: keycloak, postgres - keycloak waits for a person: its policy records (module) - keycloak provides oidc-client: its consumers are sent again after it - keycloak keeps data (ADR 0233): what it holds is backed up before it moves - postgres waits for a person: its policy records (module) - postgres provides postgres-database: its consumers are sent again after it - postgres keeps data (ADR 0233): what it holds is backed up before it moves **Transitions** - 2026-10-06 23:05 (new) → proposed (announced): novox/mesh-catalog#80 merged; its head was not heard by this owner, so it is made from the forge's word - 2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

mesh-tools referenced this pull request

2026-10-25 17:40:12 +00:00

Gate: hand every member of an approved group in pin order, settle pull requests closed on the forge, and say a stopped pass as a condition (issue 1275) #480

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-catalog#80