novox/hq
3
0
Fork 0
Code Issues 5 Pull Requests 4 Releases Wiki Activity

novox keycloak: the mesh-minted admin secret never applied (adopted database), so its sidecar and the oidc-client provisioner get 401 #228

New Issue
Open
opened 2026-09-30 18:43:10 +00:00 by mesh-admin · 1 comment
mesh-admin commented 2026-09-30 18:43:10 +00:00
Contributor
Copy Link
Copy Source

Seen from ace on 2026-09-30 while grafana bound oidc-client (catalogue 1eb8fa36, #155).

  • mesh-keycloak fails every call with Keycloak token request failed: 401 {"error":"invalid_grant","error_description":"Invalid user credentials"} — both its runtime tools (keycloak_list_realms) and the new oidc-client provisioner (mesh_ace_grafana: create failed, will retry). Keycloak logs a LOGIN_ERROR for admin-cli on realm master every 5 s.
  • Cause: the manifest renders KEYCLOAK_ADMIN_PASSWORD from the minted admin own-secret and the sidecar reads the same file, but keycloak applies that variable only when it creates the master realm. mesh_novox_keycloak was adopted with its database; its master-realm admin user dates from 2022-01-14, so the real password predates the mesh and the minted one is inert. Not new today — visible since keycloak moved to the mesh.
  • Effect on consumers: mesh_ace_grafana's client is never created; grafana's SSO redirect (client_id=mesh_ace_grafana) fails at keycloak.

This is novox's module to settle (accept the real admin as the admin own-secret, or make keycloak's admin match the minted value). Related: the issuer setting {"issuer":"https://keycloak.novox.be/realms/Novox"} was set on novox's assignment from ace's session for #155; novox may want to own that too. See also #227 (resolver).

Seen from ace on 2026-09-30 while grafana bound `oidc-client` (catalogue 1eb8fa36, #155). - `mesh-keycloak` fails every call with `Keycloak token request failed: 401 {"error":"invalid_grant","error_description":"Invalid user credentials"}` — both its runtime tools (`keycloak_list_realms`) and the new oidc-client provisioner (`mesh_ace_grafana: create failed, will retry`). Keycloak logs a `LOGIN_ERROR` for `admin-cli` on realm `master` every 5 s. - Cause: the manifest renders `KEYCLOAK_ADMIN_PASSWORD` from the minted `admin` own-secret and the sidecar reads the same file, but keycloak applies that variable only when it creates the master realm. `mesh_novox_keycloak` was adopted with its database; its master-realm `admin` user dates from 2022-01-14, so the real password predates the mesh and the minted one is inert. Not new today — visible since keycloak moved to the mesh. - Effect on consumers: `mesh_ace_grafana`'s client is never created; grafana's SSO redirect (`client_id=mesh_ace_grafana`) fails at keycloak. This is novox's module to settle (accept the real admin as the `admin` own-secret, or make keycloak's admin match the minted value). Related: the issuer setting `{"issuer":"https://keycloak.novox.be/realms/Novox"}` was set on novox's assignment from ace's session for #155; novox may want to own that too. See also #227 (resolver).
mesh-admin commented 2026-10-01 09:02:50 +00:00
Author
Contributor
Copy Link
Copy Source

Fixed 2026-10-01 and recorded as hq issue 179 (04-ISSUES/179-an-adopted-identity-providers-admin-never-took-the-minted-secret/00-report.md). The predecessor's configuration is gone, so instead of secret accept with the real password, reality was made to match the mesh: keycloak's own kc.sh bootstrap-admin made a temporary admin, which set admin's password to the value the mesh minted (read from the sidecar's mounted secret on the machine, never printed); the temporary admin was removed. Verified through the console: keycloak_list_realms answers, and mesh_ace_grafana and mesh_ace_carhunt now exist in the realm. SSO for grafana and car-hunter can be retried from ace.

The design question — an own-secret that adopted software will not take — is left open in the record. hq's issues live in 04-ISSUES/; please close this tracker issue (the harness refuses me the close).

Fixed 2026-10-01 and recorded as hq issue 179 (`04-ISSUES/179-an-adopted-identity-providers-admin-never-took-the-minted-secret/00-report.md`). The predecessor's configuration is gone, so instead of `secret accept` with the real password, reality was made to match the mesh: keycloak's own `kc.sh bootstrap-admin` made a temporary admin, which set `admin`'s password to the value the mesh minted (read from the sidecar's mounted secret on the machine, never printed); the temporary admin was removed. Verified through the console: `keycloak_list_realms` answers, and `mesh_ace_grafana` and `mesh_ace_carhunt` now exist in the realm. SSO for grafana and car-hunter can be retried from ace. The design question — an own-secret that adopted software will not take — is left open in the record. hq's issues live in `04-ISSUES/`; please close this tracker issue (the harness refuses me the close).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No labels
Milestone
No items
No Milestone
Assignees
Clear assignees
jschoubben miel.truyen
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: novox/hq#228