novox/mesh-catalog
6
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Merge pull request 'keycloak: port to Go and repair a refused admin; providers announce a failing consumer' (#80) from feat/identity-provider-admin-safety-nets into main

Browse Source
This commit was merged in pull request #80.
This commit is contained in:
mesh-admin
2026-10-05 22:39:02 +00:00
parent 5c2157b81c 48d4188927
commit 78328d4ab2
31 changed files with 4002 additions and 1283 deletions
Download Patch File Download Diff File Expand all files Collapse all files
modules/keycloak/README.md
+75
View File
@@ -0,0 +1,75 @@
# keycloak
The mesh's identity provider: one Keycloak server that provides the `oidc-client` provision to every
module that logs a person in. Each consumer is given one confidential OpenID Connect client in the
realm named by the assignment's `issuer` setting, under the client id the mesh derived for it and
the secret the mesh minted (ADR 0048); its redirect is its contributed `callback` under the names the
mesh composed for its endpoint. A client the mesh did not make — no `mesh.provisioned` attribute — is
never adopted, changed or deleted.
## The admin keeps the mesh's password
The manifest mints the `admin` own-secret, and the server takes it from its environment **only when
it creates its master realm**. A database that was adopted, restored or moved already has one, and
its `admin` keeps the password it had. Every admin call then fails with `401 invalid_grant`, and
with it every consumer's client. On 2026-10-05 that went on for a day, about 31,000 failures, seen only
in the journal (hq issue 179). Both times the fix was the same, done by hand.
The module now does that fix itself. The **guard** in the provider:
- checks that `admin` logs in with the mesh's secret: at start (every 15 s until the server answers),
then every 5 minutes, and at once when the admin API refuses the credentials;
- on a refusal — `invalid_grant`, including a missing or disabled admin — and only then, repairs it
inside the `keycloak` container with Keycloak's own recovery: `kc.sh bootstrap-admin user` makes a
temporary admin (on a free management port; the server holds 9000), `kcadm.sh` creates or
re-enables `admin` if it must and sets its password to the mesh's, and the temporary admin is
deleted. Both passwords go in on the exec's standard input. Neither is on a command line or printed;
- checks again, and says `REPAIRED the admin …` in the log and emits `admin.repaired`;
- when it cannot, logs `COULD NOT REPAIR …` with the step that failed, emits `admin.unrepaired`, and
**brakes**: the next automatic attempt comes 10 minutes later and the wait doubles each time, up to
6 hours. If the temporary admin may be left behind, the log and the event say so.
While the admin is refused, the provisioner does not call Keycloak. Each attempt would be one more
failed admin login, and enough of those lock the account. It still counts each attempt as a failure
of the consumer, so the provider's standing (below) reports `credentials-rejected`.
`keycloak_admin_check` reports the state, the last repair and the brake. With `repair: true` it
repairs a refused admin straight away, ignoring the brake, because a person asked.
## A consumer failing for minutes is announced
The provisioner loop (`harness.go`) is shared, byte for byte, with postgres. A consumer whose create,
check or secret keeps failing for 5 minutes with no success in between is announced as
`provisioner.failing`, with the consumer, its machine and the error's class. The announcement repeats
every 15 minutes while the failure lasts. `provisioner.recovered` follows the first success
(ADR 0224), and the controller shows the latest one in `status`.
## Tools
The realm, user, client, group and role tools (`keycloak_list_realms`, `keycloak_create_user`,
`keycloak_list_clients`, `keycloak_assign_user_role`, …), and `keycloak_admin_check`. A tool that
writes something announces it: `user.created`, `user.deleted`, `password.reset`, `client.created`,
`group.created`, `role.created`.
## Where the code lives
One Go bundle, `cmd/keycloak-provider`, launched by the node's runtime. It speaks MCP over stdio
through the Go SDK, and was ported from TypeScript in 2026-10. It reaches the server on
`MESH_KEYCLOAK_URL`, reads the mesh's admin secret from `MESH_KEYCLOAK_PASSWORD_FILE` at every check,
and reaches the container named by `MESH_KEYCLOAK_CONTAINER` through the `container-runtime`
capability.
## Tests
`go test ./...` runs against a fake Keycloak and a fake container. It covers:
- repair on refusal, with no password in argv;
- no repair while the server is unreachable;
- the brake, and the operator overriding it;
- the provisioner going quiet while the admin is refused;
- the OIDC client rules;
- the harness and its standing;
- `harness_same_test.go`, which fails when this module's `harness.go` and postgres's differ.
`live_test.go` runs the repair against a real, throwaway Keycloak 26 container whose admin keeps an
older password. The file's comment has the commands.
modules/keycloak/client.ts
-317
View File
@@ -1,317 +0,0 @@
// The Keycloak admin API client — keycloak's own code, living in the module (novox/hq ADR 0039).
// Moved out of the shared hal sdk, where a change to Keycloak's admin API rebuilt everything; here
// it rebuilds only keycloak. Both this module's tools and its events entrypoint import it, and
// nothing outside keycloak does.
import { readFileSync } from "node:fs";
/** The settings-merged config the mesh delivers (novox/hq ADR 0046): { url, apiKey, token, password, user, ... }. */
function meshConfig(file?: string): Record<string, string> {
if (!file) return {};
try { return JSON.parse(readFileSync(file, "utf8")) as Record<string, string>; }
catch { return {}; }
}
/** A secret file's value, trailing newline trimmed; undefined when unset or unreadable. */
function secretFile(file?: string): string | undefined {
if (!file) return undefined;
try { return readFileSync(file, "utf8").replace(/\n$/, "") || undefined; }
catch { return undefined; }
}
/** A client as the admin API represents it — only the fields this module reads or writes are typed;
* the rest travel through untouched, so an update never drops what somebody else set. */
export interface ClientRepresentation {
id?: string;
clientId: string;
name?: string;
enabled?: boolean;
protocol?: string;
publicClient?: boolean;
clientAuthenticatorType?: string;
secret?: string;
rootUrl?: string;
baseUrl?: string;
redirectUris?: string[];
webOrigins?: string[];
standardFlowEnabled?: boolean;
implicitFlowEnabled?: boolean;
directAccessGrantsEnabled?: boolean;
serviceAccountsEnabled?: boolean;
attributes?: Record<string, string>;
protocolMappers?: ProtocolMapperRepresentation[];
[other: string]: unknown;
}
export interface ProtocolMapperRepresentation {
id?: string;
name: string;
protocol: string;
protocolMapper: string;
config: Record<string, string>;
}
export class KeycloakClient {
readonly baseUrl: string;
readonly defaultRealm: string;
// The admin token is short-lived; caching it (minus a safety margin) spares every call a fresh
// password grant, and a 401 mid-flight refreshes it once rather than failing the request.
private tokenCache: { token: string; expiresAt: number } | null = null;
constructor(
url: string,
private readonly adminUser: string,
private readonly adminPass: string,
defaultRealm = "master",
) {
this.baseUrl = url.replace(/\/+$/, "");
this.defaultRealm = defaultRealm;
}
/**
* Build from the module's resolved environment. Admin URL, credentials and the fallback realm are
* read from MESH_KEYCLOAK_* — the names the mesh sets — falling back to the container's own
* KEYCLOAK_ADMIN/KEYCLOAK_ADMIN_PASSWORD so a co-located server needs nothing configured twice.
* Throws when no admin password can be found: without it the client can do nothing, so failing
* here lets the tool runtime expose no keycloak tools rather than tools that always error.
*/
static fromEnv(env: NodeJS.ProcessEnv = process.env): KeycloakClient {
const cfg = meshConfig(env.MESH_KEYCLOAK_CONFIG_FILE);
const url = cfg.url ?? env.MESH_KEYCLOAK_URL ?? `http://127.0.0.1:${env.KEYCLOAK_PORT ?? "8080"}`;
const adminUser = cfg.user ?? env.MESH_KEYCLOAK_ADMIN ?? env.KEYCLOAK_ADMIN ?? "admin";
// The admin password reaches the runtime as a file (novox/hq ADR 0086): the module's own `admin`
// secret, mounted read-only. The environment forms stay for a co-located server that has them.
const adminPass = cfg.password ?? secretFile(env.MESH_KEYCLOAK_PASSWORD_FILE)
?? env.MESH_KEYCLOAK_PASSWORD ?? env.KEYCLOAK_ADMIN_PASSWORD;
if (!adminPass) {
throw new Error("no Keycloak admin password — set MESH_KEYCLOAK_PASSWORD_FILE (or MESH_KEYCLOAK_PASSWORD)");
}
const realm = cfg.realm ?? env.MESH_KEYCLOAK_REALM ?? "master";
return new KeycloakClient(url, adminUser, adminPass, realm);
}
private async getToken(): Promise<string> {
if (this.tokenCache && Date.now() < this.tokenCache.expiresAt) return this.tokenCache.token;
const res = await fetch(`${this.baseUrl}/realms/master/protocol/openid-connect/token`, {
method: "POST",
headers: { "Content-Type": "application/x-www-form-urlencoded" },
body: new URLSearchParams({
grant_type: "password",
client_id: "admin-cli",
username: this.adminUser,
password: this.adminPass,
}),
});
if (!res.ok) throw new Error(`Keycloak token request failed: ${res.status} ${await res.text()}`);
const data = (await res.json()) as { access_token: string; expires_in: number };
this.tokenCache = { token: data.access_token, expiresAt: Date.now() + (data.expires_in - 30) * 1000 };
return data.access_token;
}
private async request<T = unknown>(path: string, options: RequestInit = {}): Promise<T> {
const doRequest = async (token: string): Promise<Response> =>
fetch(`${this.baseUrl}/admin/realms${path}`, {
...options,
headers: {
"Content-Type": "application/json",
Authorization: `Bearer ${token}`,
...(options.headers as Record<string, string>),
},
});
let res = await doRequest(await this.getToken());
// A cached token that expired against the server's clock reads as 401; drop it and retry once.
if (res.status === 401) {
this.tokenCache = null;
res = await doRequest(await this.getToken());
}
if (!res.ok) throw new Error(`Keycloak API error ${res.status}: ${await res.text()}`);
// 201/204 carry no body — the admin API's create/update/delete answer with an empty response.
if (res.status === 201 || res.status === 204) return null as T;
return res.json() as Promise<T>;
}
// Realms
async listRealms(): Promise<Array<{ id: string; realm: string; displayName?: string; enabled: boolean }>> {
return this.request("/");
}
// Users
async listUsers(realm: string, params: { search?: string; max?: number } = {}): Promise<unknown[]> {
const qs = new URLSearchParams();
if (params.search) qs.set("search", params.search);
if (params.max) qs.set("max", String(params.max));
const query = qs.toString();
return this.request(`/${realm}/users${query ? `?${query}` : ""}`);
}
async createUser(realm: string, data: {
username: string;
email?: string;
enabled?: boolean;
credentials?: Array<{ type: string; value: string; temporary: boolean }>;
}): Promise<void> {
await this.request(`/${realm}/users`, { method: "POST", body: JSON.stringify({ enabled: true, ...data }) });
}
async updateUser(realm: string, userId: string, data: Record<string, unknown>): Promise<void> {
await this.request(`/${realm}/users/${userId}`, { method: "PUT", body: JSON.stringify(data) });
}
async deleteUser(realm: string, userId: string): Promise<void> {
await this.request(`/${realm}/users/${userId}`, { method: "DELETE" });
}
async resetPassword(realm: string, userId: string, password: string, temporary = false): Promise<void> {
await this.request(`/${realm}/users/${userId}/reset-password`, {
method: "PUT",
body: JSON.stringify({ type: "password", value: password, temporary }),
});
}
async getUserSessions(realm: string, userId: string): Promise<unknown[]> {
return this.request(`/${realm}/users/${userId}/sessions`);
}
// Clients
async listClients(realm: string): Promise<unknown[]> {
return this.request(`/${realm}/clients`);
}
async createClient(realm: string, data: {
clientId: string;
name?: string;
rootUrl?: string;
redirectUris?: string[];
publicClient?: boolean;
protocol?: string;
}): Promise<void> {
await this.request(`/${realm}/clients`, {
method: "POST",
body: JSON.stringify({ protocol: "openid-connect", enabled: true, ...data }),
});
}
// The admin API addresses a client by its internal UUID, not the human clientId a caller knows;
// every client-scoped call resolves the one to the other first.
private async resolveClientId(realm: string, clientId: string): Promise<string> {
const clients = (await this.listClients(realm)) as Array<Record<string, unknown>>;
const client = clients.find((c) => c.clientId === clientId);
if (!client) throw new Error(`Client '${clientId}' not found in realm '${realm}'`);
return client.id as string;
}
/** The one client with exactly this clientId, or undefined. The admin API's `clientId` filter is an
* exact match unless `search=true` is asked for. */
async findClient(realm: string, clientId: string): Promise<ClientRepresentation | undefined> {
const found = await this.request<ClientRepresentation[]>(
`/${realm}/clients?clientId=${encodeURIComponent(clientId)}`);
return found.find((c) => c.clientId === clientId);
}
async createClientFrom(realm: string, rep: ClientRepresentation): Promise<void> {
await this.request(`/${realm}/clients`, { method: "POST", body: JSON.stringify(rep) });
}
/** Replace a client's representation, addressed by its internal id. */
async updateClient(realm: string, id: string, rep: ClientRepresentation): Promise<void> {
await this.request(`/${realm}/clients/${id}`, { method: "PUT", body: JSON.stringify(rep) });
}
async deleteClientById(realm: string, id: string): Promise<void> {
await this.request(`/${realm}/clients/${id}`, { method: "DELETE" });
}
async clientSecretById(realm: string, id: string): Promise<string | undefined> {
const result = await this.request<{ value?: string }>(`/${realm}/clients/${id}/client-secret`);
return result.value;
}
async listClientMappers(realm: string, id: string): Promise<ProtocolMapperRepresentation[]> {
return this.request(`/${realm}/clients/${id}/protocol-mappers/models`);
}
async addClientMapper(realm: string, id: string, mapper: ProtocolMapperRepresentation): Promise<void> {
await this.request(`/${realm}/clients/${id}/protocol-mappers/models`, {
method: "POST",
body: JSON.stringify(mapper),
});
}
async updateClientMapper(realm: string, id: string, mapper: ProtocolMapperRepresentation): Promise<void> {
await this.request(`/${realm}/clients/${id}/protocol-mappers/models/${mapper.id}`, {
method: "PUT",
body: JSON.stringify(mapper),
});
}
async deleteClient(realm: string, clientId: string): Promise<void> {
await this.request(`/${realm}/clients/${await this.resolveClientId(realm, clientId)}`, { method: "DELETE" });
}
async getClientSecret(realm: string, clientId: string): Promise<string> {
const id = await this.resolveClientId(realm, clientId);
const result = await this.request<{ value: string }>(`/${realm}/clients/${id}/client-secret`);
return result.value;
}
async addProtocolMapper(realm: string, clientId: string, mapper: {
name: string;
protocolMapper: string;
config: Record<string, string>;
}): Promise<void> {
const id = await this.resolveClientId(realm, clientId);
await this.request(`/${realm}/clients/${id}/protocol-mappers/models`, {
method: "POST",
body: JSON.stringify({ protocol: "openid-connect", ...mapper }),
});
}
// Roles
async listRealmRoles(realm: string): Promise<Array<{ id: string; name: string; description?: string; composite: boolean }>> {
return this.request(`/${realm}/roles`);
}
async createRealmRole(realm: string, data: { name: string; description?: string }): Promise<void> {
await this.request(`/${realm}/roles`, { method: "POST", body: JSON.stringify(data) });
}
async getUserRealmRoles(realm: string, userId: string): Promise<Array<{ id: string; name: string; description?: string }>> {
return this.request(`/${realm}/users/${userId}/role-mappings/realm`);
}
async getAvailableRealmRoles(realm: string, userId: string): Promise<Array<{ id: string; name: string; description?: string }>> {
return this.request(`/${realm}/users/${userId}/role-mappings/realm/available`);
}
async assignRealmRoles(realm: string, userId: string, roles: Array<{ id: string; name: string }>): Promise<void> {
await this.request(`/${realm}/users/${userId}/role-mappings/realm`, { method: "POST", body: JSON.stringify(roles) });
}
async removeRealmRoles(realm: string, userId: string, roles: Array<{ id: string; name: string }>): Promise<void> {
await this.request(`/${realm}/users/${userId}/role-mappings/realm`, { method: "DELETE", body: JSON.stringify(roles) });
}
// Groups
async listGroups(realm: string): Promise<Array<{ id: string; name: string; path: string; subGroupCount?: number }>> {
return this.request(`/${realm}/groups`);
}
async createGroup(realm: string, name: string): Promise<void> {
await this.request(`/${realm}/groups`, { method: "POST", body: JSON.stringify({ name }) });
}
async getUserGroups(realm: string, userId: string): Promise<Array<{ id: string; name: string; path: string }>> {
return this.request(`/${realm}/users/${userId}/groups`);
}
async addUserToGroup(realm: string, userId: string, groupId: string): Promise<void> {
await this.request(`/${realm}/users/${userId}/groups/${groupId}`, { method: "PUT" });
}
async removeUserFromGroup(realm: string, userId: string, groupId: string): Promise<void> {
await this.request(`/${realm}/users/${userId}/groups/${groupId}`, { method: "DELETE" });
}
}
modules/keycloak/cmd/keycloak-provider/admin.go
+485
View File
@@ -0,0 +1,485 @@
package main
// The admin guard: Keycloak's admin must log in with the password the mesh minted, and when it does
// not, the module makes it — itself, inside the container, and says so (novox/hq issue 179).
//
// **Why it is needed.** The manifest mints an `admin` own-secret and renders it into the server's
// environment, and Keycloak applies that environment only when it creates its master realm. A
// database that was adopted, restored or moved already has a master realm, whose `admin` keeps the
// password it had. Every admin call then fails with *401 invalid_grant*. It happened twice: an
// adopted database on 2026-10-01, and a moved one on 2026-10-05, when the provisioner failed every
// five seconds for twenty-three hours — about 31,000 times — and nothing but the journal said so.
// Both times the fix was the same by hand. This is that fix, run by the module.
//
// **Where it runs, and why here.** In the module's own bundle, which already holds the two things the
// repair needs: the mesh's admin secret (the file the provisioner reads) and the container runtime
// (the `container-runtime` capability; the nats and nextcloud bundles reach their containers the
// same way). A declared host step would have to be handed the secret a second time and could not tell
// the provisioner to stop; the bundle can, and it is the process that sees the 401 first.
//
// **What it does.** Checks the admin's login at start, every five minutes, and at once when the
// admin API refuses the credentials. On a refusal — and only a refusal: an unreachable server is
// waited for, never repaired — it runs Keycloak's own recovery inside the container: a temporary
// admin through `kc.sh bootstrap-admin`, which sets the mesh's admin password (creating or enabling
// that admin if it must), and is removed again. Then it checks again. Both passwords travel on the
// exec's standard input, never on a command line, and neither is ever printed.
//
// **When it cannot.** It says so loudly (`admin.unrepaired`, and the provisioner's standing names the
// consumers it fails), and it brakes: the next automatic attempt is ten minutes on, doubling to six
// hours. While the admin is refused, the provisioner does not call Keycloak at all — each attempt
// would be one more failed login against the admin, and enough of those lock it out.
import (
"bufio"
"bytes"
"context"
"crypto/rand"
"encoding/base64"
"encoding/hex"
"errors"
"fmt"
"math/big"
"net"
"os/exec"
"strings"
"sync"
"sync/atomic"
"time"
)
// AdminState is what the last check found.
type AdminState string
const (
AdminUnknown AdminState = "unknown"
AdminOK AdminState = "ok"
AdminRejected AdminState = "rejected"
AdminUnreachable AdminState = "unreachable"
// AdminUnchecked is a check that could not be made for a reason of the module's own — the
// mesh's secret unreadable, say. Nothing to repair in Keycloak.
AdminUnchecked AdminState = "unchecked"
)
// Events the guard emits.
const (
EventRepaired = "admin.repaired"
EventUnrepaired = "admin.unrepaired"
)
// ErrAdminRejected is what the provisioner is answered while the admin is refused: fast, and without
// asking Keycloak.
var ErrAdminRejected = fmt.Errorf("the admin is refused by Keycloak and not yet repaired (%w); not asking it again until it is", ErrRejected)
// Executor runs one command with something on its standard input, answering its combined output.
type Executor interface {
Run(ctx context.Context, argv []string, stdin []byte) ([]byte, error)
}
// DockerExec runs commands on this machine.
type DockerExec struct{}
func (DockerExec) Run(ctx context.Context, argv []string, stdin []byte) ([]byte, error) {
cmd := exec.CommandContext(ctx, argv[0], argv[1:]...)
cmd.Stdin = bytes.NewReader(stdin)
return cmd.CombinedOutput()
}
// Repair is what one repair did.
type Repair struct {
At time.Time `json:"at"`
Outcome string `json:"outcome"` // "repaired" | "unrepaired"
Step string `json:"step,omitempty"`
Error string `json:"error,omitempty"`
TempUser string `json:"temporaryAdmin,omitempty"`
// TempLeft says the temporary admin may still be in the master realm, for a person to delete.
TempLeft bool `json:"temporaryAdminLeft,omitempty"`
}
// Guard keeps the admin's login true.
type Guard struct {
KC *Client
Exec Executor
Container string
Every time.Duration // 5m
Waiting time.Duration // 15s: how often to look for a server not yet seen
BrakeFrom time.Duration // 10m
BrakeMax time.Duration // 6h
Timeout time.Duration // 5m: the whole repair
Now func() time.Time
Log func(format string, args ...any)
Announce func(event string, body map[string]any)
once sync.Once
mu sync.Mutex // one check-and-repair at a time: the background's or an operator's
state atomic.Value
last *Repair
brakeTill time.Time
brakeWait time.Duration
nudge chan struct{}
}
func (g *Guard) init() {
g.once.Do(func() {
if g.Every == 0 {
g.Every = 5 * time.Minute
}
if g.Waiting == 0 {
g.Waiting = 15 * time.Second
}
if g.BrakeFrom == 0 {
g.BrakeFrom = 10 * time.Minute
}
if g.BrakeMax == 0 {
g.BrakeMax = 6 * time.Hour
}
if g.Timeout == 0 {
g.Timeout = 5 * time.Minute
}
if g.Now == nil {
g.Now = time.Now
}
if g.Log == nil {
g.Log = func(string, ...any) {}
}
if g.Container == "" {
g.Container = "keycloak"
}
if g.Exec == nil {
g.Exec = DockerExec{}
}
g.nudge = make(chan struct{}, 1)
g.state.Store(AdminUnknown)
})
}
// State is what the last check found.
func (g *Guard) State() AdminState {
g.init()
return g.state.Load().(AdminState)
}
// Refused says the admin was refused at the last check and has not been repaired since: what the
// provisioner asks before calling Keycloak.
func (g *Guard) Refused() bool { return g.State() == AdminRejected }
// Nudge asks for a check now; never blocks.
func (g *Guard) Nudge() {
g.init()
select {
case g.nudge <- struct{}{}:
default:
}
}
// Check asks Keycloak for a token as the admin, with the mesh's secret as it is now.
func (g *Guard) Check(ctx context.Context) (AdminState, error) {
g.init()
cctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
_, _, err := g.KC.Login(cctx)
state := classifyLogin(err)
g.state.Store(state)
return state, err
}
func classifyLogin(err error) AdminState {
if err == nil {
return AdminOK
}
if errors.Is(err, ErrRejected) {
return AdminRejected
}
var terr *TokenError
if errors.As(err, &terr) {
if terr.Status >= 500 || terr.Status == 404 {
return AdminUnreachable // starting, or not Keycloak yet
}
return AdminUnchecked
}
var nerr net.Error
if errors.As(err, &nerr) || errors.Is(err, context.DeadlineExceeded) ||
strings.Contains(err.Error(), "connection refused") || strings.Contains(err.Error(), "EOF") {
return AdminUnreachable
}
return AdminUnchecked
}
// Report is the guard's account of itself, for the tool.
type Report struct {
State AdminState `json:"state"`
Error string `json:"error,omitempty"`
Repaired bool `json:"repaired,omitempty"`
LastRepair *Repair `json:"lastRepair,omitempty"`
BrakeUntil string `json:"brakeUntil,omitempty"`
Note string `json:"note,omitempty"`
}
// Ensure checks the admin and repairs a refused one. operator is a person asking: the brake is
// theirs to override. Without repair, it only checks.
func (g *Guard) Ensure(ctx context.Context, repair, operator bool) Report {
g.init()
g.mu.Lock()
defer g.mu.Unlock()
state, err := g.Check(ctx)
r := g.report(state, err)
if state != AdminRejected || !repair {
if state == AdminOK {
g.brakeTill, g.brakeWait = time.Time{}, 0
r.BrakeUntil = ""
}
return r
}
if !operator && g.Now().Before(g.brakeTill) {
r.Note = "the last repair failed; the next automatic attempt is at brakeUntil (keycloak_admin_check with repair: true tries now)"
return r
}
g.Log("[keycloak] the admin is refused by Keycloak (invalid_grant) with the mesh's password; repairing it inside %s", g.Container)
done := g.repair(ctx)
state, err = g.Check(ctx)
if state == AdminOK && done.Outcome == "repaired" {
g.brakeTill, g.brakeWait = time.Time{}, 0
g.last = &done
g.Log("[keycloak] REPAIRED the admin: the realm's admin did not take the mesh's password (invalid_grant) — " +
"the database was adopted or moved and kept an older one; set to the mesh's through a temporary " +
"bootstrap admin, which was removed (novox/hq issue 179)")
if done.TempLeft {
g.Log("[keycloak] the temporary admin %s could not be removed: delete it from the master realm", done.TempUser)
}
g.announce(EventRepaired, map[string]any{
"cause": "the master realm's admin kept a password older than the mesh's — an adopted, restored or moved database",
"at": done.At.UTC().Format(time.RFC3339), "temporaryAdminLeft": done.TempLeft,
})
r = g.report(state, err)
r.Repaired = true
return r
}
if done.Outcome == "repaired" {
// The script finished and the login still fails: say what the check found.
done.Outcome, done.Step = "unrepaired", "verify"
done.Error = fmt.Sprintf("after the repair the admin still cannot log in: %s", errText(err))
}
g.last = &done
if g.brakeWait == 0 {
g.brakeWait = g.BrakeFrom
} else if g.brakeWait *= 2; g.brakeWait > g.BrakeMax {
g.brakeWait = g.BrakeMax
}
g.brakeTill = g.Now().Add(g.brakeWait)
left := ""
if done.TempLeft {
left = fmt.Sprintf(" A temporary admin %s may be left in the master realm: delete it.", done.TempUser)
}
g.Log("[keycloak] COULD NOT REPAIR the admin at step %s: %s. Every consumer's client is unmanaged until it is; "+
"the next automatic attempt is in %s. By hand: novox/hq issue 179.%s",
done.Step, done.Error, g.brakeWait, left)
g.announce(EventUnrepaired, map[string]any{
"step": done.Step, "error": done.Error, "temporaryAdminLeft": done.TempLeft,
"next": g.brakeTill.UTC().Format(time.RFC3339),
})
r = g.report(state, err)
return r
}
func (g *Guard) report(state AdminState, err error) Report {
r := Report{State: state, LastRepair: g.last}
if err != nil {
r.Error = errText(err)
}
if g.Now().Before(g.brakeTill) {
r.BrakeUntil = g.brakeTill.UTC().Format(time.RFC3339)
}
return r
}
func errText(err error) string {
if err == nil {
return ""
}
return err.Error()
}
func (g *Guard) announce(event string, body map[string]any) {
if g.Announce != nil {
g.Announce(event, body)
}
}
// Run checks until ctx ends: often until the server has been seen, then every Every, and at once
// when nudged.
func (g *Guard) Run(ctx context.Context) {
g.init()
seen := false
for {
r := g.Ensure(ctx, true, false)
if r.State != AdminUnreachable && r.State != AdminUnknown {
seen = true
}
wait := g.Every
if !seen {
wait = g.Waiting
}
select {
case <-ctx.Done():
return
case <-g.nudge:
case <-time.After(wait):
}
}
}
// repairScript is Keycloak's own recovery, run inside its container (Keycloak 26). It reads the
// temporary admin's password and then the mesh's admin password from standard input; nothing secret
// is on its command line or in its environment as docker sees it. Every step announces itself on
// stderr, so a failure names the step it failed at.
const repairScript = `set -eu
umask 077
IFS= read -r TMP_PW
IFS= read -r NEW_PW
export TMP_PW
bin=/opt/keycloak/bin
cfg=/tmp/mesh-kcadm.$$.config
bootstrapped=
logged_in=
step() { echo "mesh-repair-step: $1" >&2; }
user_id() {
"$bin/kcadm.sh" get users -r master --config "$cfg" -q username="$1" -q exact=true --fields id --format csv --noquotes
}
cleanup() {
rc=$?
set +e
if [ -z "$logged_in" ] && [ -n "$bootstrapped" ]; then
# The bootstrap may have made the temporary admin and failed after (it did once, on a held port):
# log in as it anyway, so it is removed rather than left behind.
"$bin/kcadm.sh" config credentials --config "$cfg" --server http://localhost:8080 --realm master \
--user "$TMP_USER" --password "$TMP_PW" >/dev/null 2>&1 && logged_in=1
fi
if [ -n "$logged_in" ]; then
tid=$(user_id "$TMP_USER" 2>/dev/null)
if [ -n "$tid" ] && "$bin/kcadm.sh" delete "users/$tid" -r master --config "$cfg" >&2; then
echo "mesh-repair-removed: $TMP_USER" >&2
else
echo "mesh-repair-left: $TMP_USER" >&2
fi
elif [ -n "$bootstrapped" ]; then
echo "mesh-repair-left: $TMP_USER" >&2
fi
rm -f "$cfg"
exit $rc
}
trap cleanup EXIT
step bootstrap-admin
bootstrapped=1
"$bin/kc.sh" bootstrap-admin user --username "$TMP_USER" --password:env TMP_PW --http-management-port="$MGMT_PORT" >&2
step login
"$bin/kcadm.sh" config credentials --config "$cfg" --server http://localhost:8080 --realm master --user "$TMP_USER" --password "$TMP_PW" >&2
logged_in=1
step find-admin
id=$(user_id "$ADMIN_USER")
if [ -z "$id" ]; then
step create-admin
"$bin/kcadm.sh" create users -r master --config "$cfg" -s username="$ADMIN_USER" -s enabled=true >&2
"$bin/kcadm.sh" add-roles -r master --config "$cfg" --uusername "$ADMIN_USER" --rolename admin >&2
id=$(user_id "$ADMIN_USER")
fi
step enable-admin
"$bin/kcadm.sh" update "users/$id" -r master --config "$cfg" -s enabled=true >&2
step set-password
"$bin/kcadm.sh" set-password -r master --config "$cfg" --username "$ADMIN_USER" --new-password "$NEW_PW" >&2
step remove-temporary-admin
echo "mesh-repair-done" >&2
`
// repair runs the script once.
func (g *Guard) repair(ctx context.Context) Repair {
done := Repair{At: g.Now(), Outcome: "unrepaired", Step: "prepare"}
meshPW, err := g.KC.Password()
if err != nil {
done.Error = "the mesh's admin password cannot be read: " + err.Error()
return done
}
if strings.ContainsAny(meshPW, "\n\r") {
done.Error = "the mesh's admin password spans lines and cannot be handed over on one"
return done
}
tmpPW, user, port, err := temporaries()
if err != nil {
done.Error = err.Error()
return done
}
done.TempUser = user
argv := []string{"docker", "exec", "-i",
"-e", "TMP_USER=" + user, "-e", "MGMT_PORT=" + port, "-e", "ADMIN_USER=" + g.KC.AdminUser,
g.Container, "bash", "-c", repairScript}
rctx, cancel := context.WithTimeout(ctx, g.Timeout)
defer cancel()
out, runErr := g.Exec.Run(rctx, argv, []byte(tmpPW+"\n"+meshPW+"\n"))
text := scrubAll(string(out), tmpPW, meshPW)
sc := bufio.NewScanner(strings.NewReader(text))
finished := false
for sc.Scan() {
line := sc.Text()
switch {
case strings.HasPrefix(line, "mesh-repair-step: "):
done.Step = strings.TrimPrefix(line, "mesh-repair-step: ")
case strings.HasPrefix(line, "mesh-repair-left: "):
done.TempLeft = true
case line == "mesh-repair-done":
finished = true
}
}
if runErr == nil && finished {
done.Outcome, done.Step = "repaired", ""
return done
}
why := "the script did not finish"
if runErr != nil {
why = scrubAll(runErr.Error(), tmpPW, meshPW)
}
done.Error = why + ": " + tail(text, 20)
return done
}
// temporaries are the temporary admin's password and name, and a management port for the second
// server bootstrap-admin starts (the running server holds the default one).
func temporaries() (pw, user, port string, err error) {
raw := make([]byte, 32)
if _, err = rand.Read(raw); err != nil {
return "", "", "", fmt.Errorf("no randomness for a temporary password: %w", err)
}
id := make([]byte, 4)
if _, err = rand.Read(id); err != nil {
return "", "", "", err
}
n, err := rand.Int(rand.Reader, big.NewInt(1000))
if err != nil {
return "", "", "", err
}
return base64.RawURLEncoding.EncodeToString(raw), "mesh-repair-" + hex.EncodeToString(id),
fmt.Sprint(19000 + n.Int64()), nil
}
func scrubAll(text string, secrets ...string) string {
for _, s := range secrets {
if s != "" {
text = strings.ReplaceAll(text, s, "***")
}
}
return text
}
// tail is the last n non-empty lines of text, on one line each joined by " | ".
func tail(text string, n int) string {
var lines []string
for _, l := range strings.Split(text, "\n") {
if l = strings.TrimSpace(l); l != "" {
lines = append(lines, l)
}
}
if len(lines) > n {
lines = lines[len(lines)-n:]
}
return strings.Join(lines, " | ")
}
modules/keycloak/cmd/keycloak-provider/admin_test.go
+269
View File
@@ -0,0 +1,269 @@
package main
// The guard (novox/hq issue 179): an admin refused with the mesh's password is repaired inside the
// container, without a secret on any command line, verified, and said; one it cannot repair is said
// loudly and braked; one it cannot reach is waited for; and while it is refused the provisioner does
// not ask Keycloak.
import (
"context"
"errors"
"strings"
"testing"
"time"
)
// fakeExec is the container: it records what it was asked, and — when it works — does what the
// script does, setting the fake server's admin password to the second line of its input.
type fakeExec struct {
f *fakeKeycloak
runs []execRun
fails bool
output string
noEffect bool
}
type execRun struct {
argv []string
stdin string
}
func (x *fakeExec) Run(_ context.Context, argv []string, stdin []byte) ([]byte, error) {
x.runs = append(x.runs, execRun{argv, string(stdin)})
if x.fails {
lines := strings.Split(string(stdin), "\n")
return []byte("mesh-repair-step: bootstrap-admin\nERROR: boom " + lines[0] + " " + lines[1] + "\nmesh-repair-left: x\n"), errors.New("exit status 1")
}
if !x.noEffect {
x.f.set(func() { x.f.password = strings.Split(string(stdin), "\n")[1] })
}
return []byte("mesh-repair-step: set-password\nmesh-repair-step: remove-temporary-admin\nmesh-repair-removed: x\nmesh-repair-done\n"), nil
}
type guardWorld struct {
f *fakeKeycloak
x *fakeExec
g *Guard
now time.Time
events []string
said []string
}
func newGuardWorld(t *testing.T) *guardWorld {
w := &guardWorld{now: time.Date(2026, 10, 5, 23, 55, 0, 0, time.UTC)}
w.f = newFakeKeycloak(t, "Novox", "old-password-from-2022")
w.x = &fakeExec{f: w.f}
w.g = &Guard{KC: w.f.client("the-mesh-minted-this"), Exec: w.x,
Now: func() time.Time { return w.now },
Log: func(f string, a ...any) { w.said = append(w.said, f) },
Announce: func(e string, _ map[string]any) { w.events = append(w.events, e) }}
return w
}
func TestARefusedAdminIsRepairedInsideTheContainerAndSaid(t *testing.T) {
w := newGuardWorld(t)
r := w.g.Ensure(ctx, true, false)
if !r.Repaired || r.State != AdminOK || w.f.password != "the-mesh-minted-this" {
t.Fatalf("%+v, server password %q", r, w.f.password)
}
if strings.Join(w.events, ",") != EventRepaired {
t.Fatal(w.events)
}
if !strings.Contains(strings.Join(w.said, "\n"), "REPAIRED the admin") {
t.Fatal(w.said)
}
run := w.x.runs[0]
if run.argv[0] != "docker" || run.argv[1] != "exec" || run.argv[2] != "-i" || !contains(run.argv, "keycloak") ||
!contains(run.argv, "ADMIN_USER=admin") {
t.Fatal(run.argv)
}
// Both passwords on standard input — the temporary one, then the mesh's — and on no command line.
lines := strings.Split(run.stdin, "\n")
if len(lines) != 3 || lines[1] != "the-mesh-minted-this" || len(lines[0]) < 40 {
t.Fatalf("stdin has %d lines", len(lines))
}
for _, a := range run.argv {
if strings.Contains(a, "the-mesh-minted-this") || strings.Contains(a, lines[0]) {
t.Fatalf("a password is on the command line: %q", a)
}
}
if w.g.Refused() {
t.Fatal("still refused after a repair")
}
}
func TestTheScriptIsKeycloaksOwnRecovery(t *testing.T) {
for _, want := range []string{
`kc.sh" bootstrap-admin user --username "$TMP_USER" --password:env TMP_PW --http-management-port="$MGMT_PORT"`,
`set-password -r master --config "$cfg" --username "$ADMIN_USER" --new-password "$NEW_PW"`,
`umask 077`, `trap cleanup EXIT`, `rm -f "$cfg"`, `delete "users/$tid"`,
} {
if !strings.Contains(repairScript, want) {
t.Errorf("the script lacks %s", want)
}
}
if strings.Contains(repairScript, "--cache") {
t.Error("bootstrap-admin takes no --cache")
}
}
func TestAnAdminThatLogsInIsLeftAlone(t *testing.T) {
w := newGuardWorld(t)
w.f.password = "the-mesh-minted-this"
if r := w.g.Ensure(ctx, true, false); r.State != AdminOK || r.Repaired || len(w.x.runs) != 0 {
t.Fatalf("%+v", r)
}
}
func TestAServerNotAnsweringIsWaitedForNeverRepaired(t *testing.T) {
w := newGuardWorld(t)
w.f.down = true
if r := w.g.Ensure(ctx, true, false); r.State != AdminUnreachable || len(w.x.runs) != 0 {
t.Fatalf("%+v", r)
}
w.f.srv.Close()
if r := w.g.Ensure(ctx, true, false); r.State != AdminUnreachable || len(w.x.runs) != 0 {
t.Fatalf("%+v", r)
}
}
func TestARepairThatFailsIsSaidLoudlyAndBraked(t *testing.T) {
w := newGuardWorld(t)
w.x.fails = true
r := w.g.Ensure(ctx, true, false)
if r.State != AdminRejected || r.LastRepair == nil || r.LastRepair.Step != "bootstrap-admin" || !r.LastRepair.TempLeft || r.BrakeUntil == "" {
t.Fatalf("%+v %+v", r, r.LastRepair)
}
// Neither password survives into what is said.
if strings.Contains(r.LastRepair.Error, "the-mesh-minted-this") || strings.Contains(r.LastRepair.Error, strings.Split(w.x.runs[0].stdin, "\n")[0]) {
t.Fatal(r.LastRepair.Error)
}
if strings.Join(w.events, ",") != EventUnrepaired || !strings.Contains(strings.Join(w.said, "\n"), "COULD NOT REPAIR") {
t.Fatal(w.events, w.said)
}
if !w.g.Refused() {
t.Fatal("not refused")
}
// Inside the brake: checked, not repaired.
w.now = w.now.Add(9 * time.Minute)
w.g.Ensure(ctx, true, false)
if len(w.x.runs) != 1 {
t.Fatalf("repaired inside the brake: %d runs", len(w.x.runs))
}
// Past it: tried again, and the next brake is twice as long.
w.now = w.now.Add(2 * time.Minute)
r = w.g.Ensure(ctx, true, false)
if len(w.x.runs) != 2 {
t.Fatalf("%d runs", len(w.x.runs))
}
if until, _ := time.Parse(time.RFC3339, r.BrakeUntil); until.Sub(w.now) != 20*time.Minute {
t.Fatal(r.BrakeUntil)
}
// An operator asking repairs now, brake or not.
w.x.fails = false
if r := w.g.Ensure(ctx, true, true); !r.Repaired || len(w.x.runs) != 3 || r.BrakeUntil != "" {
t.Fatalf("%+v", r)
}
}
func TestAScriptThatFinishesButChangesNothingIsNotARepair(t *testing.T) {
w := newGuardWorld(t)
w.x.noEffect = true
r := w.g.Ensure(ctx, true, false)
if r.Repaired || r.LastRepair.Step != "verify" || strings.Join(w.events, ",") != EventUnrepaired {
t.Fatalf("%+v %+v %v", r, r.LastRepair, w.events)
}
}
func TestOnlyCheckingRepairsNothing(t *testing.T) {
w := newGuardWorld(t)
if r := w.g.Ensure(ctx, false, true); r.State != AdminRejected || len(w.x.runs) != 0 {
t.Fatalf("%+v", r)
}
}
func TestWhileTheAdminIsRefusedTheProvisionerDoesNotAskKeycloak(t *testing.T) {
w := newGuardWorld(t)
w.x.fails = true
w.g.Ensure(ctx, true, false)
a := provisioner{clients: OidcClients{KC: w.g.KC, Realm: "Novox"}, guard: w.g,
announce: func(string, map[string]any) {}, log: func(string, ...any) {}}
logins := w.f.logins
err := a.Create(ctx, grafana("s", nil))
if !errors.Is(err, ErrRejected) || a.Class(err) != ClassCredentials {
t.Fatal(err)
}
if _, err := a.Holds(ctx, grafana("s", nil)); !errors.Is(err, ErrRejected) {
t.Fatal(err)
}
if w.f.logins != logins {
t.Fatal("Keycloak was asked while the admin is refused")
}
}
func TestARefusalSeenByTheAdminAPINudgesTheGuard(t *testing.T) {
w := newGuardWorld(t)
w.g.init()
w.g.KC.onRejected = w.g.Nudge
if _, err := w.g.KC.ListRealms(ctx); !errors.Is(err, ErrRejected) {
t.Fatal(err)
}
select {
case <-w.g.nudge:
default:
t.Fatal("not nudged")
}
// The guard's own check does not nudge it: that would be a guard checking in a loop.
w.g.Check(ctx)
select {
case <-w.g.nudge:
t.Fatal("the guard's own check nudged it")
default:
}
}
func TestTheGuardReadsTheMeshsSecretEachTime(t *testing.T) {
w := newGuardWorld(t)
secret := "first"
w.g.KC.password = func() (string, error) { return secret, nil }
w.f.password = "second"
if s, _ := w.g.Check(ctx); s != AdminRejected {
t.Fatal(s)
}
secret = "second"
if s, _ := w.g.Check(ctx); s != AdminOK {
t.Fatal(s)
}
}
func TestRunRepairsWhenNudged(t *testing.T) {
w := newGuardWorld(t)
w.f.password = "the-mesh-minted-this"
w.g.Every, w.g.Waiting = time.Hour, time.Hour
c, cancel := context.WithCancel(ctx)
defer cancel()
go w.g.Run(c)
deadline := time.Now().Add(5 * time.Second)
for w.g.State() != AdminOK {
if time.Now().After(deadline) {
t.Fatal("no first check")
}
time.Sleep(10 * time.Millisecond)
}
w.f.set(func() { w.f.password = "moved-database" })
w.g.Nudge()
for {
w.f.mu.Lock()
done := w.f.password == "the-mesh-minted-this"
w.f.mu.Unlock()
if done {
break
}
if time.Now().After(deadline) {
t.Fatal("not repaired when nudged")
}
time.Sleep(10 * time.Millisecond)
}
}
modules/keycloak/cmd/keycloak-provider/client.go
+486
View File
@@ -0,0 +1,486 @@
package main
// The Keycloak admin API client — keycloak's own code, living in the module (novox/hq ADR 0039).
// Ported from client.ts: the same environment, the same token cache, the same one retry on a 401.
//
// A representation is a map rather than a struct, so an update carries back every field the server
// sent — including ones this module does not know — and never drops what somebody else set.
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strings"
"sync"
"time"
)
// Rep is a representation as the admin API sends and takes it.
type Rep = map[string]any
// ErrRejected marks a token request the server refused for the credentials: 401, or an
// `invalid_grant` — wrong password, missing or disabled user. The guard repairs exactly this.
var ErrRejected = errors.New("credentials rejected: invalid_grant")
// Client speaks to one Keycloak server's admin API as one admin.
type Client struct {
BaseURL string
AdminUser string
DefaultRealm string
// password is read each time a token is needed, so a rotated secret is used without a restart.
password func() (string, error)
http *http.Client
// onRejected is told when the server refuses the admin's credentials (the guard's nudge).
onRejected func()
mu sync.Mutex
token string
expiresAt time.Time
}
// meshConfig is the settings-merged config the mesh delivers (novox/hq ADR 0046).
func meshConfig(file string) map[string]any {
out := map[string]any{}
if file == "" {
return out
}
raw, err := os.ReadFile(file)
if err != nil {
return out
}
_ = json.Unmarshal(raw, &out)
return out
}
func cfgString(cfg map[string]any, key string) string {
if s, ok := cfg[key].(string); ok {
return s
}
return ""
}
func firstOf(values ...string) string {
for _, v := range values {
if v != "" {
return v
}
}
return ""
}
// secretFile is a secret file's value with its trailing newline trimmed.
func secretFile(file string) (string, error) {
raw, err := os.ReadFile(file)
if err != nil {
return "", err
}
s := strings.TrimSuffix(string(raw), "\n")
if s == "" {
return "", fmt.Errorf("%s is empty", file)
}
return s, nil
}
// ClientFromEnv builds the client from the module's resolved environment, as client.ts did: the
// config file first, then MESH_KEYCLOAK_*, then the container's own KEYCLOAK_ADMIN* names.
// Refused when no admin password can be found at all: without one there is nothing to serve.
func ClientFromEnv(getenv func(string) string) (*Client, error) {
cfg := meshConfig(getenv("MESH_KEYCLOAK_CONFIG_FILE"))
port := firstOf(getenv("KEYCLOAK_PORT"), "8080")
base := firstOf(cfgString(cfg, "url"), getenv("MESH_KEYCLOAK_URL"), "http://127.0.0.1:"+port)
user := firstOf(cfgString(cfg, "user"), getenv("MESH_KEYCLOAK_ADMIN"), getenv("KEYCLOAK_ADMIN"), "admin")
realm := firstOf(cfgString(cfg, "realm"), getenv("MESH_KEYCLOAK_REALM"), "master")
// The admin password reaches the runtime as a file (novox/hq ADR 0086): the module's own `admin`
// secret. Read on every token request, so the guard and the provisioner always use the mesh's
// current one.
fixed := firstOf(cfgString(cfg, "password"))
file := getenv("MESH_KEYCLOAK_PASSWORD_FILE")
env := firstOf(getenv("MESH_KEYCLOAK_PASSWORD"), getenv("KEYCLOAK_ADMIN_PASSWORD"))
password := func() (string, error) {
if fixed != "" {
return fixed, nil
}
if file != "" {
if s, err := secretFile(file); err == nil {
return s, nil
} else if env == "" {
return "", fmt.Errorf("the admin password cannot be read: %w", err)
}
}
if env != "" {
return env, nil
}
return "", errors.New("no Keycloak admin password — set MESH_KEYCLOAK_PASSWORD_FILE (or MESH_KEYCLOAK_PASSWORD)")
}
if fixed == "" && file == "" && env == "" {
return nil, errors.New("no Keycloak admin password — set MESH_KEYCLOAK_PASSWORD_FILE (or MESH_KEYCLOAK_PASSWORD)")
}
return NewClient(base, user, password, realm), nil
}
// NewClient is a client for one server and admin.
func NewClient(base, user string, password func() (string, error), realm string) *Client {
return &Client{
BaseURL: strings.TrimRight(base, "/"), AdminUser: user, DefaultRealm: realm,
password: password, http: &http.Client{Timeout: 30 * time.Second},
}
}
// Password is the admin password the mesh holds now.
func (c *Client) Password() (string, error) { return c.password() }
// TokenError is a token request the server answered with something other than a token.
type TokenError struct {
Status int
Body string
}
func (e *TokenError) Error() string {
return fmt.Sprintf("Keycloak token request failed: %d %s", e.Status, e.Body)
}
// Unwrap makes a refusal of the credentials an ErrRejected.
func (e *TokenError) Unwrap() error {
if e.Status == http.StatusUnauthorized || strings.Contains(e.Body, "invalid_grant") {
return ErrRejected
}
return nil
}
// Login asks for a fresh token with the admin's credentials, bypassing the cache: what the guard
// checks with. It tells nobody about a refusal; getToken does.
func (c *Client) Login(ctx context.Context) (string, time.Duration, error) {
pw, err := c.password()
if err != nil {
return "", 0, err
}
form := url.Values{"grant_type": {"password"}, "client_id": {"admin-cli"},
"username": {c.AdminUser}, "password": {pw}}
req, err := http.NewRequestWithContext(ctx, http.MethodPost,
c.BaseURL+"/realms/master/protocol/openid-connect/token", strings.NewReader(form.Encode()))
if err != nil {
return "", 0, err
}
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
res, err := c.http.Do(req)
if err != nil {
return "", 0, err
}
defer res.Body.Close()
body, _ := io.ReadAll(io.LimitReader(res.Body, 1<<20))
if res.StatusCode != http.StatusOK {
return "", 0, &TokenError{Status: res.StatusCode, Body: strings.TrimSpace(string(body))}
}
var data struct {
AccessToken string `json:"access_token"`
ExpiresIn int `json:"expires_in"`
}
if err := json.Unmarshal(body, &data); err != nil || data.AccessToken == "" {
return "", 0, fmt.Errorf("Keycloak token response unreadable: %v", err)
}
return data.AccessToken, time.Duration(data.ExpiresIn) * time.Second, nil
}
// getToken is a cached token, valid for at least thirty seconds more.
func (c *Client) getToken(ctx context.Context) (string, error) {
c.mu.Lock()
if c.token != "" && time.Now().Before(c.expiresAt) {
t := c.token
c.mu.Unlock()
return t, nil
}
c.mu.Unlock()
token, life, err := c.Login(ctx)
if err != nil {
// The guard is told, so a refused admin is repaired now rather than at its next check. Only
// here, never in Login: the guard checks with Login, and a check that nudged the guard
// would be a guard checking in a loop.
if errors.Is(err, ErrRejected) && c.onRejected != nil {
c.onRejected()
}
return "", err
}
c.mu.Lock()
c.token, c.expiresAt = token, time.Now().Add(life-30*time.Second)
c.mu.Unlock()
return token, nil
}
func (c *Client) dropToken() {
c.mu.Lock()
c.token = ""
c.mu.Unlock()
}
// APIError is an admin API answer that was not a success.
type APIError struct {
Status int
Body string
}
func (e *APIError) Error() string { return fmt.Sprintf("Keycloak API error %d: %s", e.Status, e.Body) }
// request calls the admin API under /admin/realms, decoding the answer into out when there is one.
func (c *Client) request(ctx context.Context, method, path string, in, out any) error {
var payload []byte
if in != nil {
var err error
if payload, err = json.Marshal(in); err != nil {
return err
}
}
do := func(token string) (*http.Response, error) {
var body io.Reader
if payload != nil {
body = bytes.NewReader(payload)
}
req, err := http.NewRequestWithContext(ctx, method, c.BaseURL+"/admin/realms"+path, body)
if err != nil {
return nil, err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer "+token)
return c.http.Do(req)
}
token, err := c.getToken(ctx)
if err != nil {
return err
}
res, err := do(token)
if err != nil {
return err
}
// A cached token that expired against the server's clock reads as 401; drop it and retry once.
if res.StatusCode == http.StatusUnauthorized {
res.Body.Close()
c.dropToken()
if token, err = c.getToken(ctx); err != nil {
return err
}
if res, err = do(token); err != nil {
return err
}
}
defer res.Body.Close()
raw, _ := io.ReadAll(io.LimitReader(res.Body, 16<<20))
if res.StatusCode < 200 || res.StatusCode > 299 {
return &APIError{Status: res.StatusCode, Body: strings.TrimSpace(string(raw))}
}
// 201/204 carry no body — the admin API's create/update/delete answer with an empty response.
if out == nil || res.StatusCode == http.StatusCreated || res.StatusCode == http.StatusNoContent || len(raw) == 0 {
return nil
}
return json.Unmarshal(raw, out)
}
func esc(s string) string { return url.PathEscape(s) }
// Realms
func (c *Client) ListRealms(ctx context.Context) ([]Rep, error) {
var out []Rep
return out, c.request(ctx, http.MethodGet, "/", nil, &out)
}
// Users
func (c *Client) ListUsers(ctx context.Context, realm, search string, max int) ([]Rep, error) {
q := url.Values{}
if search != "" {
q.Set("search", search)
}
if max > 0 {
q.Set("max", fmt.Sprint(max))
}
path := "/" + esc(realm) + "/users"
if len(q) > 0 {
path += "?" + q.Encode()
}
var out []Rep
return out, c.request(ctx, http.MethodGet, path, nil, &out)
}
func (c *Client) CreateUser(ctx context.Context, realm string, rep Rep) error {
body := Rep{"enabled": true}
for k, v := range rep {
body[k] = v
}
return c.request(ctx, http.MethodPost, "/"+esc(realm)+"/users", body, nil)
}
func (c *Client) UpdateUser(ctx context.Context, realm, id string, rep Rep) error {
return c.request(ctx, http.MethodPut, "/"+esc(realm)+"/users/"+esc(id), rep, nil)
}
func (c *Client) DeleteUser(ctx context.Context, realm, id string) error {
return c.request(ctx, http.MethodDelete, "/"+esc(realm)+"/users/"+esc(id), nil, nil)
}
func (c *Client) ResetPassword(ctx context.Context, realm, id, password string, temporary bool) error {
return c.request(ctx, http.MethodPut, "/"+esc(realm)+"/users/"+esc(id)+"/reset-password",
Rep{"type": "password", "value": password, "temporary": temporary}, nil)
}
func (c *Client) UserSessions(ctx context.Context, realm, id string) ([]Rep, error) {
var out []Rep
return out, c.request(ctx, http.MethodGet, "/"+esc(realm)+"/users/"+esc(id)+"/sessions", nil, &out)
}
// Clients
func (c *Client) ListClients(ctx context.Context, realm string) ([]Rep, error) {
var out []Rep
return out, c.request(ctx, http.MethodGet, "/"+esc(realm)+"/clients", nil, &out)
}
func (c *Client) CreateClient(ctx context.Context, realm string, rep Rep) error {
return c.request(ctx, http.MethodPost, "/"+esc(realm)+"/clients", rep, nil)
}
// FindClient is the one client with exactly this clientId, or nil. The admin API's `clientId`
// filter is an exact match unless `search=true` is asked for.
func (c *Client) FindClient(ctx context.Context, realm, clientID string) (Rep, error) {
var found []Rep
if err := c.request(ctx, http.MethodGet, "/"+esc(realm)+"/clients?clientId="+url.QueryEscape(clientID), nil, &found); err != nil {
return nil, err
}
for _, f := range found {
if f["clientId"] == clientID {
return f, nil
}
}
return nil, nil
}
// resolveClientID is a client's internal id from the clientId a caller knows.
func (c *Client) resolveClientID(ctx context.Context, realm, clientID string) (string, error) {
clients, err := c.ListClients(ctx, realm)
if err != nil {
return "", err
}
for _, cl := range clients {
if cl["clientId"] == clientID {
id, _ := cl["id"].(string)
return id, nil
}
}
return "", fmt.Errorf("Client '%s' not found in realm '%s'", clientID, realm)
}
func (c *Client) UpdateClient(ctx context.Context, realm, id string, rep Rep) error {
return c.request(ctx, http.MethodPut, "/"+esc(realm)+"/clients/"+esc(id), rep, nil)
}
func (c *Client) DeleteClientByID(ctx context.Context, realm, id string) error {
return c.request(ctx, http.MethodDelete, "/"+esc(realm)+"/clients/"+esc(id), nil, nil)
}
func (c *Client) ClientSecretByID(ctx context.Context, realm, id string) (string, error) {
var out struct {
Value string `json:"value"`
}
err := c.request(ctx, http.MethodGet, "/"+esc(realm)+"/clients/"+esc(id)+"/client-secret", nil, &out)
return out.Value, err
}
func (c *Client) ListClientMappers(ctx context.Context, realm, id string) ([]Rep, error) {
var out []Rep
return out, c.request(ctx, http.MethodGet, "/"+esc(realm)+"/clients/"+esc(id)+"/protocol-mappers/models", nil, &out)
}
func (c *Client) AddClientMapper(ctx context.Context, realm, id string, mapper Rep) error {
return c.request(ctx, http.MethodPost, "/"+esc(realm)+"/clients/"+esc(id)+"/protocol-mappers/models", mapper, nil)
}
func (c *Client) UpdateClientMapper(ctx context.Context, realm, id string, mapper Rep) error {
mid, _ := mapper["id"].(string)
return c.request(ctx, http.MethodPut, "/"+esc(realm)+"/clients/"+esc(id)+"/protocol-mappers/models/"+esc(mid), mapper, nil)
}
func (c *Client) DeleteClient(ctx context.Context, realm, clientID string) error {
id, err := c.resolveClientID(ctx, realm, clientID)
if err != nil {
return err
}
return c.DeleteClientByID(ctx, realm, id)
}
func (c *Client) GetClientSecret(ctx context.Context, realm, clientID string) (string, error) {
id, err := c.resolveClientID(ctx, realm, clientID)
if err != nil {
return "", err
}
return c.ClientSecretByID(ctx, realm, id)
}
func (c *Client) AddProtocolMapper(ctx context.Context, realm, clientID string, mapper Rep) error {
id, err := c.resolveClientID(ctx, realm, clientID)
if err != nil {
return err
}
body := Rep{"protocol": "openid-connect"}
for k, v := range mapper {
body[k] = v
}
return c.AddClientMapper(ctx, realm, id, body)
}
// Roles
func (c *Client) ListRealmRoles(ctx context.Context, realm string) ([]Rep, error) {
var out []Rep
return out, c.request(ctx, http.MethodGet, "/"+esc(realm)+"/roles", nil, &out)
}
func (c *Client) CreateRealmRole(ctx context.Context, realm string, rep Rep) error {
return c.request(ctx, http.MethodPost, "/"+esc(realm)+"/roles", rep, nil)
}
func (c *Client) UserRealmRoles(ctx context.Context, realm, id string) ([]Rep, error) {
var out []Rep
return out, c.request(ctx, http.MethodGet, "/"+esc(realm)+"/users/"+esc(id)+"/role-mappings/realm", nil, &out)
}
func (c *Client) AvailableRealmRoles(ctx context.Context, realm, id string) ([]Rep, error) {
var out []Rep
return out, c.request(ctx, http.MethodGet, "/"+esc(realm)+"/users/"+esc(id)+"/role-mappings/realm/available", nil, &out)
}
func (c *Client) AssignRealmRoles(ctx context.Context, realm, id string, roles []Rep) error {
return c.request(ctx, http.MethodPost, "/"+esc(realm)+"/users/"+esc(id)+"/role-mappings/realm", roles, nil)
}
func (c *Client) RemoveRealmRoles(ctx context.Context, realm, id string, roles []Rep) error {
return c.request(ctx, http.MethodDelete, "/"+esc(realm)+"/users/"+esc(id)+"/role-mappings/realm", roles, nil)
}
// Groups
func (c *Client) ListGroups(ctx context.Context, realm string) ([]Rep, error) {
var out []Rep
return out, c.request(ctx, http.MethodGet, "/"+esc(realm)+"/groups", nil, &out)
}
func (c *Client) CreateGroup(ctx context.Context, realm, name string) error {
return c.request(ctx, http.MethodPost, "/"+esc(realm)+"/groups", Rep{"name": name}, nil)
}
func (c *Client) UserGroups(ctx context.Context, realm, id string) ([]Rep, error) {
var out []Rep
return out, c.request(ctx, http.MethodGet, "/"+esc(realm)+"/users/"+esc(id)+"/groups", nil, &out)
}
func (c *Client) AddUserToGroup(ctx context.Context, realm, id, group string) error {
return c.request(ctx, http.MethodPut, "/"+esc(realm)+"/users/"+esc(id)+"/groups/"+esc(group), nil, nil)
}
func (c *Client) RemoveUserFromGroup(ctx context.Context, realm, id, group string) error {
return c.request(ctx, http.MethodDelete, "/"+esc(realm)+"/users/"+esc(id)+"/groups/"+esc(group), nil, nil)
}
modules/keycloak/cmd/keycloak-provider/fake_test.go
+169
View File
@@ -0,0 +1,169 @@
package main
// A fake Keycloak: the token endpoint, accepting one password that a test may change, and the admin
// routes this module touches on one realm's clients, answering with the status codes and shapes
// Keycloak gives.
import (
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
)
var ctx = context.Background()
type fakeKeycloak struct {
mu sync.Mutex
srv *httptest.Server
realm string
password string // what the admin's login accepts
down bool // answer 503, as a starting server does
logins int
clients map[string]Rep
calls []string
}
func newFakeKeycloak(t *testing.T, realm, password string) *fakeKeycloak {
f := &fakeKeycloak{realm: realm, password: password, clients: map[string]Rep{}}
f.srv = httptest.NewServer(http.HandlerFunc(f.serve))
t.Cleanup(f.srv.Close)
return f
}
func uuid() string {
b := make([]byte, 16)
_, _ = rand.Read(b)
return hex.EncodeToString(b)
}
func send(w http.ResponseWriter, status int, v any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
if v != nil {
_ = json.NewEncoder(w).Encode(v)
}
}
func (f *fakeKeycloak) set(fn func()) {
f.mu.Lock()
defer f.mu.Unlock()
fn()
}
func (f *fakeKeycloak) serve(w http.ResponseWriter, r *http.Request) {
f.mu.Lock()
defer f.mu.Unlock()
f.calls = append(f.calls, r.Method+" "+r.URL.Path)
if f.down {
send(w, 503, nil)
return
}
if r.URL.Path == "/realms/master/protocol/openid-connect/token" {
_ = r.ParseForm()
f.logins++
if r.PostForm.Get("password") != f.password {
send(w, 401, map[string]string{"error": "invalid_grant", "error_description": "Invalid user credentials"})
return
}
send(w, 200, map[string]any{"access_token": "t", "expires_in": 300})
return
}
base := "/admin/realms/" + f.realm + "/clients"
if !strings.HasPrefix(r.URL.Path, base) {
send(w, 404, map[string]string{"error": "Realm not found."})
return
}
var rest []string
for _, p := range strings.Split(strings.TrimPrefix(r.URL.Path, base), "/") {
if p != "" {
rest = append(rest, p)
}
}
body := func() Rep {
raw, _ := io.ReadAll(r.Body)
var v Rep
_ = json.Unmarshal(raw, &v)
return v
}
if len(rest) == 0 && r.Method == "GET" {
want := r.URL.Query().Get("clientId")
out := []Rep{}
for _, c := range f.clients {
if want == "" || c["clientId"] == want {
out = append(out, c)
}
}
send(w, 200, out)
return
}
if len(rest) == 0 && r.Method == "POST" {
rep := body()
for _, c := range f.clients {
if c["clientId"] == rep["clientId"] {
send(w, 409, map[string]string{"errorMessage": "exists"})
return
}
}
id := uuid()
mappers := []any{}
if list, ok := rep["protocolMappers"].([]any); ok {
for _, m := range list {
mm := m.(map[string]any)
mm["id"] = uuid()
mappers = append(mappers, mm)
}
}
rep["id"], rep["protocolMappers"] = id, mappers
f.clients[id] = rep
send(w, 201, nil)
return
}
c := f.clients[rest[0]]
if c == nil {
send(w, 404, map[string]string{"error": "Could not find client"})
return
}
switch {
case len(rest) == 1 && r.Method == "PUT":
// Keycloak ignores protocolMappers on a client update: they have their own endpoints.
rep := body()
rep["id"], rep["protocolMappers"] = c["id"], c["protocolMappers"]
f.clients[rest[0]] = rep
send(w, 204, nil)
case len(rest) == 1 && r.Method == "DELETE":
delete(f.clients, rest[0])
send(w, 204, nil)
case rest[1] == "client-secret" && r.Method == "GET":
send(w, 200, map[string]any{"type": "secret", "value": c["secret"]})
case rest[1] == "protocol-mappers" && r.Method == "GET":
send(w, 200, c["protocolMappers"])
case rest[1] == "protocol-mappers" && r.Method == "POST":
m := body()
m["id"] = uuid()
list, _ := c["protocolMappers"].([]any)
c["protocolMappers"] = append(list, m)
send(w, 201, nil)
case rest[1] == "protocol-mappers" && r.Method == "PUT":
m := body()
list, _ := c["protocolMappers"].([]any)
for i, x := range list {
if x.(map[string]any)["id"] == rest[4] {
list[i] = m
}
}
send(w, 204, nil)
default:
send(w, 405, nil)
}
}
func (f *fakeKeycloak) client(password string) *Client {
return NewClient(f.srv.URL, "admin", func() (string, error) { return password, nil }, "master")
}
modules/keycloak/cmd/keycloak-provider/harness.go
+527
View File
@@ -0,0 +1,527 @@
package main
// The reconcile loop every provider shares, as the TypeScript SDK's runProvisioner runs it
// (@novox/mesh-sdk/provisioner, 0.1.10). The Go SDK has no provisioner yet, so this module carries
// the loop itself, line for line in behaviour; when the Go SDK grows one, this file is what moves
// there (novox/hq ADR 0039: the loop is the SDK's, the adapter is the module's).
//
// Read the contributions the mesh delivered; bring each consumer's resource into being through the
// adapter, under the login and password the mesh minted; withdraw what the mesh no longer asks for.
// **A provider creates the credential the mesh minted, and seals nothing (novox/hq ADR 0048).**
//
// **A provider that keeps failing a consumer says so on the bus (novox/hq ADR 0224).** A consumer
// whose create, check or secret has failed without one success in between for FailingAfter is
// announced as `provisioner.failing` — naming the consumer, its machine and the class of error — and
// again every SayAgainEvery while it lasts; the first success after that is `provisioner.recovered`.
// The controller keeps the newest per provider and consumer and `status` names it. On 2026-10-05 the
// identity provider failed every consumer 31,000 times in a day and said so only in its journal
// (novox/hq issue 179).
//
// Carried, identical, by every Go provider until the Go SDK has the loop: postgres and keycloak.
// Each module's `harness_same_test.go` fails when its copy and the other's differ.
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/url"
"os"
"strings"
"time"
)
// Provision is one consumer's resource to bring into being — everything the mesh derived and delivered.
type Provision struct {
// As is the login the mesh derived and gave the consumer to present.
As string
// Password is the one the mesh minted, read from the file the host unsealed.
Password string
// Values are what the consumer contributed (e.g. {"name": "letta", "extensions": ["vector"]}).
Values map[string]any
// Derived is what this provider's own definition derives for the consumer (novox/hq ADR 0201).
Derived map[string]any
// At is where the consumer is; Consumer is its node.
At string
Consumer string
}
// Adapter is the per-service half.
type Adapter interface {
Create(ctx context.Context, p Provision) error
// Remove withdraws what Create made; derived is what the mesh last derived, remembered here.
Remove(ctx context.Context, as string, derived map[string]any) error
// Holds says whether the backend still holds the consumer exactly as p says. Read-only.
Holds(ctx context.Context, p Provision) (bool, error)
}
// Harness is the loop's settings and memory.
type Harness struct {
Resource string
Receives string
Adapter Adapter
Every time.Duration // 5s
VerifyEvery time.Duration // 60s
HoldsTimeout time.Duration // 30s
Log func(format string, args ...any)
Now func() time.Time
// Announce publishes one of the provider's standing events; nil announces nothing. Node is the
// machine this provider runs on, said in each.
Announce func(event string, body map[string]any)
Node string
// FailingAfter is how long a consumer fails without a success before it is announced (5m);
// SayAgainEvery is how often it is announced again while it lasts (15m), so a controller that
// missed the first hears the next, and a standing nobody repeats can be told from one that holds.
FailingAfter time.Duration
SayAgainEvery time.Duration
verifiedAt time.Time
applied map[string]appliedEntry
lost map[string]brake
waiting map[string]int
failing map[string]failure
trouble map[string]*standing
cleared map[string]bool
lastWarning string
}
// standing is one consumer's unbroken run of failures: since when, how often, and the last error.
type standing struct {
node string
since time.Time
attempts int
class string
text string
saidAt time.Time
}
// The events a provider's standing is announced as (novox/hq ADR 0224). The controller derives the
// permission to emit them for every module that receives contributions; no manifest lists them.
const (
EventFailing = "provisioner.failing"
EventRecovered = "provisioner.recovered"
)
// The classes of error a standing is announced with: what a person reading `status` needs to know
// before reading the journal. An adapter may say better (Classifier).
const (
ClassCredentials = "credentials-rejected"
ClassUnreachable = "unreachable"
ClassSecret = "secret-unreadable"
ClassRefused = "refused"
)
// Classifier is an adapter that can say what class an error of its own is.
type Classifier interface {
Class(err error) string
}
type appliedEntry struct {
hash string
derived map[string]any
}
type brake struct {
times int
nextAt time.Time
}
type failure struct {
text string
times int
}
// The longest a consumer whose create keeps failing to satisfy holds waits between checks.
const maxBackoff = time.Hour
// How many passes a secret may be unreadable before it stops being called a race (issue 225), and
// once said loudly, how often it is repeated. The same cadence quiets a create that keeps failing
// the same way.
const (
patiently = 12
loudlyEvery = 240
)
type contribution struct {
As string `json:"as"`
Secret string `json:"secret"`
Node string `json:"node"`
At string `json:"at"`
Values map[string]any `json:"values"`
Derived map[string]any `json:"derived"`
}
func (h *Harness) init() {
if h.Every == 0 {
h.Every = 5 * time.Second
}
if h.VerifyEvery == 0 {
h.VerifyEvery = time.Minute
}
if h.HoldsTimeout == 0 {
h.HoldsTimeout = 30 * time.Second
}
if h.Now == nil {
h.Now = time.Now
}
if h.FailingAfter == 0 {
h.FailingAfter = 5 * time.Minute
}
if h.SayAgainEvery == 0 {
h.SayAgainEvery = 15 * time.Minute
}
if h.Log == nil {
h.Log = func(format string, args ...any) { fmt.Fprintf(os.Stderr, format+"\n", args...) }
}
if h.applied == nil {
h.applied = map[string]appliedEntry{}
h.lost = map[string]brake{}
h.waiting = map[string]int{}
h.failing = map[string]failure{}
h.trouble = map[string]*standing{}
h.cleared = map[string]bool{}
}
}
// Run reconciles until ctx ends. One consumer's failure never stops the others'.
func (h *Harness) Run(ctx context.Context) {
h.init()
for {
h.Reconcile(ctx)
select {
case <-ctx.Done():
return
case <-time.After(h.Every):
}
}
}
func (h *Harness) say(format string, args ...any) {
h.Log("[provisioner:"+h.Resource+"] "+format, args...)
}
func (h *Harness) warn(why string) {
if why == h.lastWarning {
return
}
if why != "" {
h.say("%s; nothing applied or removed until it can be read", why)
} else {
h.say("contributions file readable again")
}
h.lastWarning = why
}
// readContributions answers the consumers asked for, or nil when the file says nothing usable.
// **Nothing read is not nobody asking** (novox/hq issue 241): only a file that was read can withdraw.
func (h *Harness) readContributions() []contribution {
raw, err := os.ReadFile(h.Receives)
if err != nil {
h.warn(fmt.Sprintf("contributions file unreadable (%s): %v", h.Receives, err))
return nil
}
var doc struct {
Requirement string `json:"requirement"`
Given json.RawMessage `json:"given"`
}
if err := json.Unmarshal(raw, &doc); err != nil {
h.warn(fmt.Sprintf("contributions file is not JSON (%s): %v", h.Receives, err))
return nil
}
if doc.Requirement != "" && doc.Requirement != h.Resource {
h.warn(fmt.Sprintf("%s is for %s, not %s", h.Receives, doc.Requirement, h.Resource))
return nil
}
var given []contribution
if len(doc.Given) == 0 || string(doc.Given) == "null" || json.Unmarshal(doc.Given, &given) != nil {
h.warn(fmt.Sprintf("%s has no given list", h.Receives))
return nil
}
h.warn("")
out := []contribution{}
for _, g := range given {
// No `as` is not a credential grant: nothing to create for it.
if g.As != "" && g.Secret != "" {
out = append(out, g)
}
}
return out
}
func hashOf(as, password string, values, derived map[string]any) string {
// Derived is in the hash: a provider that renames what it derives gave a different resource.
b, _ := json.Marshal([]any{as, password, orEmpty(values), orEmpty(derived)})
return string(b)
}
func orEmpty(m map[string]any) map[string]any {
if m == nil {
return map[string]any{}
}
return m
}
// Reconcile is one pass.
func (h *Harness) Reconcile(ctx context.Context) {
h.init()
given := h.readContributions()
if given == nil {
return
}
want := map[string]bool{}
for _, g := range given {
want[g.As] = true
}
verifying := h.Now().Sub(h.verifiedAt) >= h.VerifyEvery
if verifying {
h.verifiedAt = h.Now()
}
for _, g := range given {
raw, err := os.ReadFile(g.Secret)
if err != nil {
// A secret the host has not written yet is a race on the first pass; past a minute it is
// a person's to look at, and said so (novox/hq issue 225).
n := h.waiting[g.As] + 1
h.waiting[g.As] = n
if n <= patiently {
h.say("%s: secret not readable yet (%s): %v", g.As, g.Secret, err)
} else if n == patiently+1 || n%loudlyEvery == 0 {
h.say("%s: CANNOT READ the secret after %d attempts (%s): %v. This is not a race any more — "+
"nothing has been provisioned for this consumer and nothing will be until somebody looks. "+
"Check who owns the file and who this process runs as (novox/hq issue 225)", g.As, n, g.Secret, err)
}
h.failed(g.As, g.Node, ClassSecret, fmt.Sprintf("secret not readable (%s): %v", g.Secret, err))
continue
}
delete(h.waiting, g.As)
password := strings.TrimSuffix(string(raw), "\n")
p := Provision{As: g.As, Password: password, Values: orEmpty(g.Values), Derived: orEmpty(g.Derived), At: g.At, Consumer: g.Node}
hash := hashOf(g.As, password, g.Values, g.Derived)
reapplying := 0
if was, ok := h.applied[g.As]; ok && was.hash == hash {
if !verifying {
continue
}
b, braked := h.lost[g.As]
if braked && h.Now().Before(b.nextAt) {
continue
}
hctx, cancel := context.WithTimeout(ctx, h.HoldsTimeout)
held, err := h.Adapter.Holds(hctx, p)
timedOut := errors.Is(hctx.Err(), context.DeadlineExceeded)
cancel()
if err != nil {
// Unable to ask is not evidence of loss. A backend that timed out will time out for
// the next consumer too, so the rest of this pass is not asked.
text := scrub(err, password)
h.say("%s: could not check the backend, will ask again: %s", g.As, text)
h.failed(g.As, g.Node, h.classOf(err, text), text)
if timedOut {
verifying = false
}
continue
}
if held {
delete(h.lost, g.As)
h.succeeded(g.As)
continue
}
reapplying = b.times + 1
if reapplying == 1 {
h.say("%s: the backend no longer holds it; applying again", g.As)
} else {
h.say("%s: still not held after being applied again (%d times in a row) — create does not "+
"produce what holds checks; applying again", g.As, reapplying)
}
}
if err := h.Adapter.Create(ctx, p); err != nil {
text := scrub(err, password)
f := h.failing[g.As]
if f.text != text {
f = failure{text: text}
}
f.times++
h.failing[g.As] = f
// Said each time it changes, and while it stays the same, as rarely as a lost secret.
if f.times == 1 || f.times%loudlyEvery == 0 {
h.say("%s: create failed, will retry: %s", g.As, text)
}
h.failed(g.As, g.Node, h.classOf(err, text), text)
if reapplying > 0 {
h.lost[g.As] = brake{times: reapplying - 1}
}
continue
}
if f, was := h.failing[g.As]; was {
h.say("%s: created, after %d failed attempt(s)", g.As, f.times)
delete(h.failing, g.As)
}
h.succeeded(g.As)
h.applied[g.As] = appliedEntry{hash: hash, derived: p.Derived}
if reapplying == 0 {
delete(h.lost, g.As)
} else {
wait := h.VerifyEvery << (reapplying - 1)
if wait > maxBackoff || wait <= 0 {
wait = maxBackoff
}
h.lost[g.As] = brake{times: reapplying, nextAt: h.Now().Add(wait)}
if reapplying > 1 {
h.say("%s: next check in %s", g.As, wait.Round(time.Second))
}
}
}
// Withdraw every login this process made that the mesh no longer asks for.
for as, was := range h.applied {
if want[as] {
continue
}
h.say("%s: no longer in %s; withdrawing it from the backend", as, h.Receives)
if err := h.Adapter.Remove(ctx, as, was.derived); err != nil {
h.say("%s: remove failed, will retry: %v", as, err)
continue
}
delete(h.applied, as)
delete(h.lost, as)
}
for as := range h.failing {
if !want[as] {
delete(h.failing, as)
}
}
// A consumer the mesh stopped asking for is no longer failed by anyone: said, so a standing
// the controller keeps for it is cleared rather than left naming a consumer that is gone.
for as := range h.trouble {
if !want[as] {
h.recovered(as, "withdrawn")
}
}
}
// failed counts one more failure in a consumer's unbroken run, and announces the run once it has
// lasted FailingAfter — then again every SayAgainEvery while it lasts.
func (h *Harness) failed(as, node, class, text string) {
now := h.Now()
s := h.trouble[as]
if s == nil {
s = &standing{since: now}
h.trouble[as] = s
}
s.node, s.class, s.text = node, class, text
s.attempts++
if now.Sub(s.since) < h.FailingAfter {
return
}
if !s.saidAt.IsZero() && now.Sub(s.saidAt) < h.SayAgainEvery {
return
}
first := s.saidAt.IsZero()
s.saidAt = now
if first {
h.say("%s: FAILING for %s (%d attempts, %s): %s. Announced as %s; `status` names it until it "+
"succeeds (novox/hq ADR 0224)", as, now.Sub(s.since).Round(time.Second), s.attempts, class, text, EventFailing)
}
h.announce(EventFailing, map[string]any{
"provider": h.Resource, "provider-node": h.Node,
"consumer": as, "node": node,
"class": class, "error": clip(text),
"since": s.since.UTC().Format(time.RFC3339), "attempts": s.attempts,
})
}
// succeeded ends a consumer's run of failures; one that was announced is announced recovered.
//
// **And the first success for a consumer since this process started is announced too**, failing or
// not: a provider that announced a failure and was restarted has forgotten it, and without this the
// controller would name the consumer failing for ever after it recovered unheard.
func (h *Harness) succeeded(as string) {
if h.trouble[as] == nil && !h.cleared[as] {
h.cleared[as] = true
h.announce(EventRecovered, map[string]any{
"provider": h.Resource, "provider-node": h.Node, "consumer": as, "why": "first-success",
})
return
}
h.cleared[as] = true
h.recovered(as, "")
}
func (h *Harness) recovered(as, why string) {
s := h.trouble[as]
if s == nil {
return
}
delete(h.trouble, as)
if s.saidAt.IsZero() {
return // never announced, so there is nothing to take back
}
if why == "" {
h.say("%s: recovered after %s and %d failed attempt(s)", as, h.Now().Sub(s.since).Round(time.Second), s.attempts)
}
body := map[string]any{
"provider": h.Resource, "provider-node": h.Node, "consumer": as, "node": s.node,
"since": s.since.UTC().Format(time.RFC3339), "attempts": s.attempts,
}
if why != "" {
body["why"] = why
}
h.announce(EventRecovered, body)
}
func (h *Harness) announce(event string, body map[string]any) {
if h.Announce != nil {
h.Announce(event, body)
}
}
// classOf is an error's class: the adapter's word when it has one, else read from the text.
func (h *Harness) classOf(err error, text string) string {
if c, ok := h.Adapter.(Classifier); ok {
if class := c.Class(err); class != "" {
return class
}
}
return ClassOf(text)
}
// ClassOf reads an error's class from its text — the words the backends the mesh runs use.
func ClassOf(text string) string {
t := strings.ToLower(text)
for _, w := range []string{"invalid_grant", "invalid user credentials", "password authentication failed",
"authentication failed", "unauthorized", " 401"} {
if strings.Contains(t, w) {
return ClassCredentials
}
}
for _, w := range []string{"connection refused", "no such host", "i/o timeout", "deadline exceeded",
"connection reset", "network is unreachable", "no route to host", "eof"} {
if strings.Contains(t, w) {
return ClassUnreachable
}
}
return ClassRefused
}
// clip keeps an announced error to what belongs in a status line.
func clip(text string) string {
const most = 300
if len(text) <= most {
return text
}
return text[:most] + "…"
}
// scrub is an error's text with the consumer's password removed, raw and URL-encoded.
func scrub(err error, password string) string {
text := err.Error()
if password == "" {
return text
}
for _, form := range []string{password, url.QueryEscape(password), url.PathEscape(password)} {
text = strings.ReplaceAll(text, form, "***")
}
return text
}
modules/keycloak/cmd/keycloak-provider/harness_same_test.go
+31
View File
@@ -0,0 +1,31 @@
package main
// The provisioner loop is carried, identical, by every Go provider until the Go SDK has it
// (harness.go). Two copies drift the moment one is fixed and the other is not — and the one left
// behind is the provider that fails a consumer without saying so (novox/hq ADR 0224). This holds
// them to one text. Skipped where postgres is not beside this module, as in a build of this one alone.
import (
"bytes"
"errors"
"io/fs"
"os"
"testing"
)
func TestTheHarnessIsTheSameAsPostgress(t *testing.T) {
theirs, err := os.ReadFile("../../../postgres/cmd/postgres-provider/harness.go")
if errors.Is(err, fs.ErrNotExist) {
t.Skip("postgres is not beside this module")
}
if err != nil {
t.Fatal(err)
}
ours, err := os.ReadFile("harness.go")
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(ours, theirs) {
t.Fatal("harness.go differs from postgres/cmd/postgres-provider/harness.go: change both, identically")
}
}
modules/keycloak/cmd/keycloak-provider/harness_test.go
+185
View File
@@ -0,0 +1,185 @@
package main
// The shared harness, as postgres tests it (harness.go is the same file in both modules).
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
type recorder struct {
created []Provision
removed []string
held bool
failing error
holdsErr error
}
func (r *recorder) Create(_ context.Context, p Provision) error {
if r.failing != nil {
return r.failing
}
r.created = append(r.created, p)
return nil
}
func (r *recorder) Remove(_ context.Context, as string, _ map[string]any) error {
r.removed = append(r.removed, as)
return nil
}
func (r *recorder) Holds(context.Context, Provision) (bool, error) {
if r.holdsErr != nil {
return false, r.holdsErr
}
return r.held, nil
}
type world struct {
t *testing.T
dir string
receives string
now time.Time
h *Harness
a *recorder
said []string
}
func newWorld(t *testing.T) *world {
w := &world{t: t, dir: t.TempDir(), now: time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC), a: &recorder{held: true}}
w.receives = filepath.Join(w.dir, "mesh.json")
w.h = &Harness{Resource: "oidc-client", Receives: w.receives, Adapter: w.a,
Now: func() time.Time { return w.now },
Log: func(f string, a ...any) { w.said = append(w.said, f) }}
return w
}
func (w *world) give(given ...map[string]any) {
for _, g := range given {
secret := filepath.Join(w.dir, g["as"].(string)+".secret")
if err := os.WriteFile(secret, []byte("pw-"+g["as"].(string)+"\n"), 0o600); err != nil {
w.t.Fatal(err)
}
g["secret"] = secret
}
if given == nil {
given = []map[string]any{}
}
raw, _ := json.Marshal(map[string]any{"requirement": "oidc-client", "given": given})
if err := os.WriteFile(w.receives, raw, 0o600); err != nil {
w.t.Fatal(err)
}
}
func TestAConsumerIsCreatedOnceUnderTheMeshsLoginAndPassword(t *testing.T) {
w := newWorld(t)
w.give(map[string]any{"as": "mesh_ace_letta", "node": "ace", "values": map[string]any{"name": "letta"}})
w.h.Reconcile(ctx)
w.h.Reconcile(ctx)
if len(w.a.created) != 1 {
t.Fatalf("created %d times", len(w.a.created))
}
p := w.a.created[0]
if p.As != "mesh_ace_letta" || p.Password != "pw-mesh_ace_letta" || p.Consumer != "ace" {
t.Fatalf("%+v", p)
}
}
func TestAConsumerNoLongerAskedForIsWithdrawn(t *testing.T) {
w := newWorld(t)
w.give(map[string]any{"as": "a"}, map[string]any{"as": "b"})
w.h.Reconcile(ctx)
w.give(map[string]any{"as": "a"})
w.h.Reconcile(ctx)
if strings.Join(w.a.removed, ",") != "b" {
t.Fatal(w.a.removed)
}
// Only a file that says nobody asks withdraws everybody.
w.give()
w.h.Reconcile(ctx)
if strings.Join(w.a.removed, ",") != "b,a" {
t.Fatal(w.a.removed)
}
}
func TestNothingReadIsNotNobodyAsking(t *testing.T) {
for name, content := range map[string]string{
"unreadable": "",
"not JSON": "{",
"no given": `{"requirement": "oidc-client"}`,
"another": `{"requirement": "mssql-database", "given": []}`,
} {
t.Run(name, func(t *testing.T) {
w := newWorld(t)
w.give(map[string]any{"as": "a"})
w.h.Reconcile(ctx)
if content == "" {
os.Remove(w.receives)
} else {
os.WriteFile(w.receives, []byte(content), 0o600)
}
w.h.Reconcile(ctx)
if len(w.a.removed) != 0 {
t.Fatalf("withdrew %v on a file it could not use", w.a.removed)
}
})
}
}
func TestALostConsumerIsAppliedAgainAndBraked(t *testing.T) {
w := newWorld(t)
w.give(map[string]any{"as": "a"})
w.h.Reconcile(ctx)
w.a.held = false
w.now = w.now.Add(2 * time.Minute)
w.h.Reconcile(ctx)
if len(w.a.created) != 2 {
t.Fatalf("created %d times", len(w.a.created))
}
// Still not held a minute later: applied again, then braked for two minutes.
w.now = w.now.Add(61 * time.Second)
w.h.Reconcile(ctx)
w.now = w.now.Add(61 * time.Second)
w.h.Reconcile(ctx)
if len(w.a.created) != 3 {
t.Fatalf("not braked: created %d times", len(w.a.created))
}
}
func TestAFailingCreateIsRetriedAndSaidOnce(t *testing.T) {
w := newWorld(t)
w.a.failing = &pgErr{"realm refused, password pw-a"}
w.give(map[string]any{"as": "a"})
for i := 0; i < 5; i++ {
w.h.Reconcile(ctx)
}
n := 0
for _, s := range w.said {
if strings.Contains(s, "create failed") {
n++
}
}
if n != 1 {
t.Fatalf("said %d times", n)
}
w.a.failing = nil
w.h.Reconcile(ctx)
if len(w.a.created) != 1 {
t.Fatal("not retried")
}
}
type pgErr struct{ s string }
func (e *pgErr) Error() string { return e.s }
func TestScrubRemovesThePassword(t *testing.T) {
if got := scrub(&pgErr{"bad pw a/b c and a%2Fb+c"}, "a/b c"); strings.Contains(got, "a/b c") || strings.Contains(got, "a%2Fb+c") {
t.Fatal(got)
}
}
modules/keycloak/cmd/keycloak-provider/live_test.go
+60
View File
@@ -0,0 +1,60 @@
package main
// The repair against a real Keycloak (issue 179's procedure, run by the guard). Skipped unless
// MESH_KEYCLOAK_LIVE_CONTAINER names a throwaway Keycloak 26 container whose realm was created with
// another admin password than "new", and MESH_KEYCLOAK_LIVE_URL reaches it, e.g.:
//
// docker network create kc-live
// docker run -d --name kc-live-db --network kc-live -e POSTGRES_PASSWORD=pg -e POSTGRES_DB=keycloak postgres:17-alpine
// docker run -d --name kc-live --network kc-live -p 127.0.0.1:18080:8080 \
// -e KC_DB=postgres -e KC_DB_URL=jdbc:postgresql://kc-live-db/keycloak -e KC_DB_USERNAME=postgres \
// -e KC_DB_PASSWORD=pg -e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=old \
// quay.io/keycloak/keycloak:26.0.8 start-dev
// MESH_KEYCLOAK_LIVE_CONTAINER=kc-live MESH_KEYCLOAK_LIVE_URL=http://127.0.0.1:18080 go test -run Live ./...
//
// A database whose admin kept an older password than the mesh's is exactly that container.
import (
"os"
"strings"
"testing"
"time"
)
func TestLiveRepair(t *testing.T) {
container, base := os.Getenv("MESH_KEYCLOAK_LIVE_CONTAINER"), os.Getenv("MESH_KEYCLOAK_LIVE_URL")
if container == "" || base == "" {
t.Skip("no MESH_KEYCLOAK_LIVE_CONTAINER / MESH_KEYCLOAK_LIVE_URL")
}
kc := NewClient(base, "admin", func() (string, error) { return "new", nil }, "master")
var said, events []string
g := &Guard{KC: kc, Container: container,
Log: func(f string, a ...any) { said = append(said, f) },
Announce: func(e string, _ map[string]any) { events = append(events, e) }}
if s, err := g.Check(ctx); s != AdminRejected {
t.Fatalf("the container's admin should refuse the mesh's password first: %s %v", s, err)
}
start := time.Now()
r := g.Ensure(ctx, true, false)
if !r.Repaired {
t.Fatalf("not repaired after %s: %+v %+v", time.Since(start), r, r.LastRepair)
}
t.Logf("repaired in %s", time.Since(start).Round(time.Second))
users, err := kc.ListUsers(ctx, "master", "", 100)
if err != nil {
t.Fatal(err)
}
for _, u := range users {
if name, _ := u["username"].(string); strings.HasPrefix(name, "mesh-repair-") {
t.Fatalf("the temporary admin %s is still there", name)
}
}
if strings.Join(events, ",") != EventRepaired {
t.Fatal(events)
}
// And a second pass changes nothing.
if r := g.Ensure(ctx, true, false); r.Repaired || r.State != AdminOK {
t.Fatalf("%+v", r)
}
}
modules/keycloak/cmd/keycloak-provider/main.go
+68
View File
@@ -0,0 +1,68 @@
// keycloak-provider: keycloak's code, one binary the node's runtime launches and speaks MCP to over
// stdio through the Go SDK (novox/hq ADR 0188, 0193). It serves keycloak's tools and, beside them,
// runs long: the provisioner that makes keycloak the provider of the mesh `oidc-client` interface,
// and the guard that keeps the admin logging in with the mesh's secret (novox/hq issue 179).
//
// stdout is the MCP channel; everything this module says, it says on stderr.
package main
import (
"context"
"os"
stdio "git.novox.be/novox/mesh-sdk/go"
)
func say(format string, args ...any) { logStderr("[keycloak] "+format, args...) }
// announce emits an event without letting a broker hiccup fail what it announces: the change already
// happened in Keycloak.
func announce(event string, body map[string]any) {
if err := stdio.Emit(event, body); err != nil {
say("emit %s failed: %v", event, err)
}
}
func main() {
kc, err := ClientFromEnv(os.Getenv)
if err != nil {
// Without the admin password there is nothing to serve, provision or guard; said, not fatal,
// so the runtime does not restart a process that cannot do better.
say("%v; serving no tools and provisioning nothing", err)
if err := stdio.Serve("", nil); err != nil {
say("%v", err)
os.Exit(1)
}
return
}
guard := &Guard{
KC: kc, Container: os.Getenv("MESH_KEYCLOAK_CONTAINER"),
Log: logStderr, Announce: announce,
}
kc.onRejected = guard.Nudge
go guard.Run(context.Background())
if receives := os.Getenv("MESH_RECEIVES"); receives == "" {
say("MESH_RECEIVES is not set — the provisioner cannot run without it")
} else if issuer, err := Issuer(os.Getenv); err != nil {
say("%v — the provisioner cannot run without it", err)
} else if realm, err := RealmOf(issuer); err != nil {
say("%v — the provisioner cannot run without it", err)
} else {
h := &Harness{
Resource: "oidc-client",
Receives: receives,
Adapter: provisioner{clients: OidcClients{KC: kc, Realm: realm}, guard: guard, announce: announce, log: logStderr},
Log: logStderr,
// A consumer failed for minutes is said on the bus, where the controller hears it and
// `status` names it (novox/hq ADR 0224).
Announce: announce,
Node: os.Getenv("MESH_NODE"),
}
go h.Run(context.Background())
}
if err := stdio.Serve("", Tools(kc, guard)); err != nil {
say("%v", err)
os.Exit(1)
}
}
modules/keycloak/cmd/keycloak-provider/oidc.go
+307
View File
@@ -0,0 +1,307 @@
package main
// What the `oidc-client` provision means in Keycloak: one confidential OpenID Connect client per
// consumer, in the realm this module serves, under the name and secret the mesh gave both ends.
// Ported from oidc.ts, behaviour for behaviour.
//
// **The client id and the secret are the mesh's, not Keycloak's (novox/hq ADR 0048).** The mesh
// derives the consumer's identity (`as`) and hands it to both ends, and mints the secret, which this
// sets as the client's secret. Keycloak generates neither.
//
// **Where the consumer's browser comes back to is the consumer's to say.** Its contribution carries
// `callback` (a path) and the mesh composes its endpoint's names into `name` (public) and
// `internal-name` (private network) exactly as it does for a route (novox/hq ADR 0056, 0138).
//
// **Only what the mesh made is touched.** A client this module creates carries the attribute
// `mesh.provisioned=true`. A client with the same id that lacks the mark is somebody else's: it is
// refused, never adopted, never updated, never deleted.
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/url"
"regexp"
"sort"
"strings"
)
// Mark is the attribute marking a client as the mesh's own work.
const Mark = "mesh.provisioned"
// RolesMapper is the mapper every mesh client carries: realm roles as a flat `roles` claim in the id
// token, the access token and userinfo — what a consumer maps its own roles from.
func RolesMapper() Rep {
return Rep{
"name": "realm roles",
"protocol": "openid-connect",
"protocolMapper": "oidc-usermodel-realm-role-mapper",
"config": map[string]any{
"claim.name": "roles",
"jsonType.label": "String",
"multivalued": "true",
"id.token.claim": "true",
"access.token.claim": "true",
"userinfo.token.claim": "true",
},
}
}
var realmPath = regexp.MustCompile(`/realms/([^/]+)/?$`)
// RealmOf is the realm named by an issuer URL — `https://id.example/realms/Novox` is realm `Novox`.
// The issuer is the one value an assignment sets, so the realm is read out of it rather than set a
// second time where the two could disagree.
func RealmOf(issuer string) (string, error) {
u, err := url.Parse(issuer)
if err != nil || u.Scheme == "" || u.Host == "" {
return "", fmt.Errorf("the issuer %q is not a URL", issuer)
}
m := realmPath.FindStringSubmatch(u.EscapedPath())
if m == nil {
return "", fmt.Errorf("the issuer %q does not end in /realms/<realm>", issuer)
}
realm, err := url.PathUnescape(m[1])
if err != nil {
return "", err
}
return realm, nil
}
// RedirectsOf is the redirect URIs a consumer's contribution asks for: its callback under each name
// the mesh composed for its endpoint. Refused when there is nothing to register.
func RedirectsOf(values map[string]any) (root string, redirects []string, err error) {
callback, _ := values["callback"].(string)
if !strings.HasPrefix(callback, "/") {
raw, _ := json.Marshal(values["callback"])
return "", nil, fmt.Errorf("contributes no callback path (`callback`, starting with \"/\"): %s", raw)
}
var names []string
for _, key := range []string{"name", "internal-name"} {
n, _ := values[key].(string)
n = strings.TrimSpace(n)
if n != "" && !contains(names, n) {
names = append(names, n)
}
}
if len(names) == 0 {
return "", nil, errors.New("has no name the mesh composed (`name` / `internal-name`) — contribute a `label` and the `endpoint` it is reached on")
}
for _, n := range names {
redirects = append(redirects, "https://"+n+callback)
}
return "https://" + names[0], redirects, nil
}
func contains(list []string, s string) bool {
for _, x := range list {
if x == s {
return true
}
}
return false
}
// wanted is the fields the mesh owns on a client it made. Everything else is left as found.
func wanted(p Provision) (Rep, []string, error) {
root, redirects, err := RedirectsOf(p.Values)
if err != nil {
return nil, nil, err
}
whose := "a consumer"
if p.Consumer != "" {
whose = "a module on " + p.Consumer
}
uris := make([]any, len(redirects))
for i, r := range redirects {
uris[i] = r
}
return Rep{
"clientId": p.As,
"name": p.As,
"description": "made by the mesh for " + whose + " — do not edit; it is reset",
"enabled": true,
"protocol": "openid-connect",
"publicClient": false,
"clientAuthenticatorType": "client-secret",
"secret": p.Password,
"rootUrl": root,
"baseUrl": root,
"redirectUris": uris,
"standardFlowEnabled": true,
"implicitFlowEnabled": false,
"directAccessGrantsEnabled": false,
"serviceAccountsEnabled": false,
}, redirects, nil
}
func stringList(v any) []string {
list, _ := v.([]any)
out := make([]string, 0, len(list))
for _, x := range list {
if s, ok := x.(string); ok {
out = append(out, s)
}
}
return out
}
func sameSet(a, b []string) bool {
x, y := append([]string(nil), a...), append([]string(nil), b...)
sort.Strings(x)
sort.Strings(y)
if len(x) != len(y) {
return false
}
for i := range x {
if x[i] != y[i] {
return false
}
}
return true
}
func attributes(c Rep) map[string]any {
if a, ok := c["attributes"].(map[string]any); ok {
return a
}
return map[string]any{}
}
func marked(c Rep) bool { return attributes(c)[Mark] == "true" }
// OidcClients is the provision's meaning in one realm.
type OidcClients struct {
KC *Client
Realm string
}
// Ensure creates the consumer's client, or brings the mesh's existing one back to what the grant
// says. Answers "created" or "updated". Idempotent.
func (o OidcClients) Ensure(ctx context.Context, p Provision) (string, error) {
want, _, err := wanted(p)
if err != nil {
return "", err
}
found, err := o.KC.FindClient(ctx, o.Realm, p.As)
if err != nil {
return "", err
}
if found != nil && !marked(found) {
return "", fmt.Errorf("realm %s already has a client %s the mesh did not make — left alone; "+
"delete or rename it if the mesh should own that id", o.Realm, p.As)
}
if found == nil {
want["attributes"] = map[string]any{Mark: "true"}
want["protocolMappers"] = []any{RolesMapper()}
if err := o.KC.CreateClient(ctx, o.Realm, want); err != nil {
return "", err
}
return "created", nil
}
// Overlay what the mesh owns on what is there, so a field Keycloak added or an operator set on a
// field the mesh does not own survives the update.
merged := Rep{}
for k, v := range found {
merged[k] = v
}
for k, v := range want {
merged[k] = v
}
attrs := map[string]any{}
for k, v := range attributes(found) {
attrs[k] = v
}
attrs[Mark] = "true"
merged["attributes"] = attrs
id, _ := found["id"].(string)
if err := o.KC.UpdateClient(ctx, o.Realm, id, merged); err != nil {
return "", err
}
return "updated", o.ensureMapper(ctx, id)
}
func (o OidcClients) ensureMapper(ctx context.Context, id string) error {
want := RolesMapper()
mappers, err := o.KC.ListClientMappers(ctx, o.Realm, id)
if err != nil {
return err
}
var have Rep
for _, m := range mappers {
if m["name"] == want["name"] {
have = m
break
}
}
if have == nil {
return o.KC.AddClientMapper(ctx, o.Realm, id, want)
}
drifted := have["protocolMapper"] != want["protocolMapper"]
hc, _ := have["config"].(map[string]any)
for k, v := range want["config"].(map[string]any) {
if hc[k] != v {
drifted = true
}
}
if drifted {
want["id"] = have["id"]
return o.KC.UpdateClientMapper(ctx, o.Realm, id, want)
}
return nil
}
// Holds says whether Keycloak still holds this consumer's client exactly as the grant says: present,
// the mesh's, enabled, confidential, with the mesh's secret and the redirects asked for. Reads only.
func (o OidcClients) Holds(ctx context.Context, p Provision) (bool, error) {
_, redirects, err := wanted(p)
if err != nil {
return false, err
}
found, err := o.KC.FindClient(ctx, o.Realm, p.As)
if err != nil {
return false, err
}
if found == nil || !marked(found) || found["enabled"] == false || found["publicClient"] == true {
return false, nil
}
if !sameSet(stringList(found["redirectUris"]), redirects) {
return false, nil
}
id, _ := found["id"].(string)
mappers, err := o.KC.ListClientMappers(ctx, o.Realm, id)
if err != nil {
return false, err
}
hasMapper := false
for _, m := range mappers {
if m["name"] == RolesMapper()["name"] {
hasMapper = true
}
}
if !hasMapper {
return false, nil
}
secret, err := o.KC.ClientSecretByID(ctx, o.Realm, id)
if err != nil {
return false, err
}
return secret == p.Password, nil
}
// Remove withdraws a consumer's client — only one the mesh made. Answers what happened.
func (o OidcClients) Remove(ctx context.Context, as string) (string, error) {
found, err := o.KC.FindClient(ctx, o.Realm, as)
if err != nil {
return "", err
}
if found == nil {
return "absent", nil
}
if !marked(found) {
return "not ours", nil
}
id, _ := found["id"].(string)
return "removed", o.KC.DeleteClientByID(ctx, o.Realm, id)
}
modules/keycloak/cmd/keycloak-provider/oidc_test.go
+199
View File
@@ -0,0 +1,199 @@
package main
// What holds keycloak to the `oidc-client` provision: one confidential client per consumer, under
// the id and secret the mesh gave, redirecting only to the consumer's own callback under the names
// the mesh composed; made once and brought back on every apply; and a client the mesh did not make —
// same id or not — never adopted, changed or deleted. Ported from test/oidc.test.ts.
import (
"encoding/json"
"reflect"
"strings"
"testing"
)
func oidcWorld(t *testing.T) (*fakeKeycloak, OidcClients) {
f := newFakeKeycloak(t, "Novox", "pw")
return f, OidcClients{KC: f.client("pw"), Realm: "Novox"}
}
// grafana is a dashboard on the home server, as the mesh hands it to the provisioner.
func grafana(secret string, extra map[string]any) Provision {
values := map[string]any{"label": "grafana", "endpoint": "web", "port": 20010.0, "callback": "/login/generic_oauth",
"name": "grafana.example.org", "internal-name": "grafana.home.internal"}
for k, v := range extra {
values[k] = v
}
return Provision{As: "mesh_home_grafana", Password: secret, Consumer: "home", Values: values}
}
func only(t *testing.T, f *fakeKeycloak, clientID string) Rep {
t.Helper()
var found []Rep
for _, c := range f.clients {
if c["clientId"] == clientID {
found = append(found, c)
}
}
if len(found) != 1 {
t.Fatalf("exactly one client %s, found %d", clientID, len(found))
}
return found[0]
}
func mustHold(t *testing.T, o OidcClients, p Provision, want bool) {
t.Helper()
held, err := o.Holds(ctx, p)
if err != nil || held != want {
t.Fatalf("holds = %v, %v; want %v", held, err, want)
}
}
func TestTheRealmIsReadOutOfTheIssuer(t *testing.T) {
for issuer, want := range map[string]string{
"https://id.example.org/realms/Novox": "Novox",
"https://id.example.org/realms/Novox/": "Novox",
"http://127.0.0.1:18500/realms/master": "master",
} {
if got, err := RealmOf(issuer); err != nil || got != want {
t.Errorf("%s: %q %v", issuer, got, err)
}
}
if _, err := RealmOf("https://id.example.org"); err == nil || !strings.Contains(err.Error(), "realms") {
t.Error(err)
}
if _, err := RealmOf("keycloak"); err == nil || !strings.Contains(err.Error(), "not a URL") {
t.Error(err)
}
}
func TestTheRedirectIsTheCallbackUnderEveryComposedName(t *testing.T) {
root, redirects, err := RedirectsOf(grafana("s", nil).Values)
if err != nil || root != "https://grafana.example.org" || !reflect.DeepEqual(redirects,
[]string{"https://grafana.example.org/login/generic_oauth", "https://grafana.home.internal/login/generic_oauth"}) {
t.Fatal(root, redirects, err)
}
if _, r, _ := RedirectsOf(map[string]any{"callback": "/cb", "internal-name": "x.home.internal"}); !reflect.DeepEqual(r, []string{"https://x.home.internal/cb"}) {
t.Fatal(r)
}
for _, values := range []map[string]any{{"name": "g"}, {"name": "g", "callback": "login"}} {
if _, _, err := RedirectsOf(values); err == nil || !strings.Contains(err.Error(), "callback") {
t.Error(err)
}
}
if _, _, err := RedirectsOf(map[string]any{"callback": "/cb"}); err == nil || !strings.Contains(err.Error(), "label") {
t.Error(err)
}
}
func TestAConsumerIsGivenOneConfidentialClient(t *testing.T) {
f, o := oidcWorld(t)
if done, err := o.Ensure(ctx, grafana("s3cret", nil)); err != nil || done != "created" {
t.Fatal(done, err)
}
c := only(t, f, "mesh_home_grafana")
for k, v := range map[string]any{"publicClient": false, "clientAuthenticatorType": "client-secret", "secret": "s3cret",
"enabled": true, "standardFlowEnabled": true, "directAccessGrantsEnabled": false, "implicitFlowEnabled": false} {
if c[k] != v {
t.Errorf("%s = %v, want %v", k, c[k], v)
}
}
if attributes(c)[Mark] != "true" || len(c["protocolMappers"].([]any)) != 1 {
t.Fatal(c)
}
mustHold(t, o, grafana("s3cret", nil), true)
}
func TestApplyingTheSameGrantAgainMakesNoSecondClient(t *testing.T) {
f, o := oidcWorld(t)
o.Ensure(ctx, grafana("s", nil))
for i := 0; i < 2; i++ {
if done, err := o.Ensure(ctx, grafana("s", nil)); err != nil || done != "updated" {
t.Fatal(done, err)
}
}
if n := len(only(t, f, "mesh_home_grafana")["protocolMappers"].([]any)); n != 1 {
t.Fatalf("the roles mapper was added %d times", n)
}
}
func TestANewSecretIsAppliedInPlaceAndWhatTheMeshDoesNotOwnSurvives(t *testing.T) {
f, o := oidcWorld(t)
o.Ensure(ctx, grafana("s", nil))
c := only(t, f, "mesh_home_grafana")
id := c["id"]
c["consentRequired"] = true
attributes(c)["post.logout.redirect.uris"] = "+"
mustHold(t, o, grafana("rotated", nil), false)
if _, err := o.Ensure(ctx, grafana("rotated", map[string]any{"name": "dash.example.org"})); err != nil {
t.Fatal(err)
}
c = only(t, f, "mesh_home_grafana")
if c["id"] != id || c["secret"] != "rotated" || c["rootUrl"] != "https://dash.example.org" ||
c["consentRequired"] != true || attributes(c)["post.logout.redirect.uris"] != "+" || attributes(c)[Mark] != "true" {
raw, _ := json.Marshal(c)
t.Fatal(string(raw))
}
mustHold(t, o, grafana("rotated", map[string]any{"name": "dash.example.org"}), true)
}
func TestAClientEditedBehindTheMeshsBackIsNotHeldAndIsMadeWhole(t *testing.T) {
f, o := oidcWorld(t)
o.Ensure(ctx, grafana("s", nil))
only(t, f, "mesh_home_grafana")["redirectUris"] = []any{"*"}
mustHold(t, o, grafana("s", nil), false)
o.Ensure(ctx, grafana("s", nil))
mustHold(t, o, grafana("s", nil), true)
only(t, f, "mesh_home_grafana")["protocolMappers"] = []any{}
mustHold(t, o, grafana("s", nil), false)
o.Ensure(ctx, grafana("s", nil))
mustHold(t, o, grafana("s", nil), true)
f.clients = map[string]Rep{}
mustHold(t, o, grafana("s", nil), false)
}
func TestAClientTheMeshDidNotMakeIsRefusedAndLeftAlone(t *testing.T) {
f, o := oidcWorld(t)
f.clients["theirs"] = Rep{"id": "theirs", "clientId": "mesh_home_grafana", "secret": "their-secret", "redirectUris": []any{"*"}}
before, _ := json.Marshal(f.clients["theirs"])
from := len(f.calls)
if _, err := o.Ensure(ctx, grafana("s", nil)); err == nil || !strings.Contains(err.Error(), "did not make") {
t.Fatal(err)
}
after, _ := json.Marshal(f.clients["theirs"])
if string(before) != string(after) {
t.Fatal("changed")
}
for _, c := range f.calls[from:] {
if !strings.HasPrefix(c, "GET") && !strings.HasPrefix(c, "POST /realms/master") {
t.Fatalf("wrote: %v", f.calls[from:])
}
}
mustHold(t, o, grafana("s", nil), false)
if done, _ := o.Remove(ctx, "mesh_home_grafana"); done != "not ours" || f.clients["theirs"] == nil {
t.Fatal(done)
}
}
func TestAWithdrawnClientIsRemovedAndAnAbsentOneIsNoError(t *testing.T) {
f, o := oidcWorld(t)
o.Ensure(ctx, grafana("s", nil))
if done, err := o.Remove(ctx, "mesh_home_grafana"); done != "removed" || err != nil || len(f.clients) != 0 {
t.Fatal(done, err)
}
if done, err := o.Remove(ctx, "mesh_home_grafana"); done != "absent" || err != nil {
t.Fatal(done, err)
}
}
func TestAContributionWithNoCallbackMakesNoClient(t *testing.T) {
f, o := oidcWorld(t)
p := grafana("s", nil)
p.Values = map[string]any{"name": "grafana.example.org"}
if _, err := o.Ensure(ctx, p); err == nil || !strings.Contains(err.Error(), "callback") || len(f.clients) != 0 {
t.Fatal(err)
}
}
modules/keycloak/cmd/keycloak-provider/provisioner.go
+98
View File
@@ -0,0 +1,98 @@
package main
// keycloak's provisioner — the adapter that makes keycloak a provider of the mesh `oidc-client`
// interface (novox/hq ADR 0039/0040/0048). The reconcile loop is harness.go's; this writes only how
// Keycloak creates, checks and removes a consumer's client. What a client is, is oidc.go's.
//
// **The realm is read out of the issuer**, the one value an assignment sets (settings reach both the
// served facts and this module's config.json): a realm set in one place and an issuer in another
// would let the consumer be told one realm while its client is made in another.
//
// **While the admin is refused, Keycloak is not asked** (admin.go): every attempt would be one more
// failed login against the admin. The harness still counts each pass as a failure of the consumer,
// classed credentials-rejected, so the standing it announces says what is wrong.
import (
"context"
"errors"
"fmt"
"os"
)
// Issuer is the issuer this assignment serves, from the settings-merged config the mesh delivers.
func Issuer(getenv func(string) string) (string, error) {
if said := cfgString(meshConfig(getenv("MESH_KEYCLOAK_CONFIG_FILE")), "issuer"); said != "" {
return said, nil
}
if said := getenv("MESH_KEYCLOAK_ISSUER"); said != "" {
return said, nil
}
return "", errors.New("no issuer — the module's config.json carries none and MESH_KEYCLOAK_ISSUER is unset")
}
// provisioner is the adapter.
type provisioner struct {
clients OidcClients
guard *Guard
announce func(event string, body map[string]any)
log func(format string, args ...any)
}
func (a provisioner) refused() error {
if a.guard != nil && a.guard.Refused() {
return ErrAdminRejected
}
return nil
}
func (a provisioner) Create(ctx context.Context, p Provision) error {
if err := a.refused(); err != nil {
return err
}
done, err := a.clients.Ensure(ctx, p)
if err != nil {
return err
}
if done == "created" {
a.log("[provisioner:oidc-client] created client %s in realm %s", p.As, a.clients.Realm)
a.announce("client.created", map[string]any{"realm": a.clients.Realm, "clientId": p.As, "consumer": p.Consumer})
}
return nil
}
func (a provisioner) Remove(ctx context.Context, as string, _ map[string]any) error {
if err := a.refused(); err != nil {
return err
}
done, err := a.clients.Remove(ctx, as)
if err != nil {
return err
}
switch done {
case "not ours":
a.log("[provisioner:oidc-client] %s: a client of that id exists that the mesh did not make — left alone", as)
case "removed":
a.log("[provisioner:oidc-client] removed client %s from realm %s", as, a.clients.Realm)
}
return nil
}
// Holds is asked every minute by the harness: whether Keycloak still holds this consumer's client
// exactly as the mesh gave it, so one deleted or edited behind the mesh's back is made again
// (novox/hq issue 120).
func (a provisioner) Holds(ctx context.Context, p Provision) (bool, error) {
if err := a.refused(); err != nil {
return false, err
}
return a.clients.Holds(ctx, p)
}
// Class says a refused admin is a credentials problem, whatever the words around it.
func (provisioner) Class(err error) string {
if errors.Is(err, ErrRejected) {
return ClassCredentials
}
return ""
}
func logStderr(format string, args ...any) { fmt.Fprintf(os.Stderr, format+"\n", args...) }
modules/keycloak/cmd/keycloak-provider/standing_test.go
+187
View File
@@ -0,0 +1,187 @@
package main
// A provider that keeps failing a consumer says so on the bus (novox/hq ADR 0224): not on the first
// failure, which may be a restart; after FailingAfter of failures with no success between; again
// every SayAgainEvery while it lasts; and recovered on the first success, or when the consumer goes.
import (
"errors"
"os"
"strings"
"testing"
"time"
)
type announced struct {
event string
body map[string]any
}
func standingWorld(t *testing.T) (*world, *[]announced) {
w := newWorld(t)
var said []announced
w.h.Announce = func(e string, b map[string]any) {
// The first success since start is its own test's; every other test reads past it.
if b["why"] != "first-success" {
said = append(said, announced{e, b})
}
}
w.h.Node = "anchor"
return w, &said
}
// passes reconciles every five seconds for d, as Run would.
func (w *world) passes(d time.Duration) {
for end := w.now.Add(d); w.now.Before(end); w.now = w.now.Add(5 * time.Second) {
w.h.Reconcile(ctx)
}
}
func events(said []announced) string {
var out []string
for _, a := range said {
out = append(out, a.event)
}
return strings.Join(out, ",")
}
func TestAConsumerFailedForMinutesIsAnnouncedNamingItAndTheClass(t *testing.T) {
w, said := standingWorld(t)
w.a.failing = errors.New(`token request failed: 401 {"error":"invalid_grant","error_description":"Invalid user credentials"}`)
w.give(map[string]any{"as": "mesh_home_grafana", "node": "home-server"})
w.passes(4 * time.Minute)
if len(*said) != 0 {
t.Fatalf("announced before FailingAfter: %v", events(*said))
}
w.passes(2 * time.Minute)
if events(*said) != EventFailing {
t.Fatalf("want one %s, got %q", EventFailing, events(*said))
}
b := (*said)[0].body
if b["consumer"] != "mesh_home_grafana" || b["node"] != "home-server" || b["class"] != ClassCredentials ||
b["provider"] != "oidc-client" || b["provider-node"] != "anchor" || b["attempts"].(int) < 60 {
t.Fatalf("%v", b)
}
// Said again while it lasts, not every pass.
w.passes(14 * time.Minute)
if events(*said) != EventFailing {
t.Fatalf("repeated too soon: %q", events(*said))
}
w.passes(2 * time.Minute)
if events(*said) != EventFailing+","+EventFailing {
t.Fatalf("not repeated: %q", events(*said))
}
// The first success takes it back.
w.a.failing = nil
w.passes(5 * time.Second)
if events(*said) != EventFailing+","+EventFailing+","+EventRecovered {
t.Fatalf("no recovery: %q", events(*said))
}
if (*said)[2].body["consumer"] != "mesh_home_grafana" {
t.Fatal((*said)[2].body)
}
}
func TestOneSuccessBetweenFailuresStartsTheRunAgain(t *testing.T) {
w, said := standingWorld(t)
w.a.failing = errors.New("connection refused")
w.give(map[string]any{"as": "a"})
w.passes(4 * time.Minute)
w.a.failing = nil
w.passes(5 * time.Second)
w.give(map[string]any{"as": "a", "values": map[string]any{"name": "changed"}})
w.a.failing = errors.New("connection refused")
w.passes(4 * time.Minute)
if len(*said) != 0 {
t.Fatalf("two runs of four minutes are not one of eight: %q", events(*said))
}
}
// The check that failed for a day on 2026-10-05: clients already made, every minute's check refused
// at the token. A check that cannot be asked is a failure too.
func TestACheckThatKeepsFailingIsAFailureToo(t *testing.T) {
w, said := standingWorld(t)
w.give(map[string]any{"as": "a"})
w.h.Reconcile(ctx)
w.a.holdsErr = errors.New("401 invalid_grant")
w.passes(7 * time.Minute)
if events(*said) != EventFailing || (*said)[0].body["class"] != ClassCredentials {
t.Fatalf("%q %v", events(*said), *said)
}
w.a.holdsErr = nil
w.passes(time.Minute + 5*time.Second)
if events(*said) != EventFailing+","+EventRecovered {
t.Fatalf("%q", events(*said))
}
}
func TestAnUnreadableSecretIsAnnouncedAsSuch(t *testing.T) {
w, said := standingWorld(t)
w.give(map[string]any{"as": "a"})
os.Remove(w.dir + "/a.secret")
w.passes(6 * time.Minute)
if events(*said) != EventFailing || (*said)[0].body["class"] != ClassSecret {
t.Fatalf("%q %v", events(*said), *said)
}
}
func TestAWithdrawnConsumerIsNoLongerFailing(t *testing.T) {
w, said := standingWorld(t)
w.a.failing = errors.New("boom")
w.give(map[string]any{"as": "a"}, map[string]any{"as": "b"})
w.passes(6 * time.Minute)
if events(*said) != EventFailing+","+EventFailing {
t.Fatalf("%q", events(*said))
}
w.give(map[string]any{"as": "a"})
w.passes(5 * time.Second)
last := (*said)[len(*said)-1]
if last.event != EventRecovered || last.body["consumer"] != "b" || last.body["why"] != "withdrawn" {
t.Fatalf("%v", *said)
}
}
func TestAnErrorIsClassedByItsWords(t *testing.T) {
for text, want := range map[string]string{
`Keycloak token request failed: 401 {"error":"invalid_grant"}`: ClassCredentials,
`FATAL: password authentication failed for user "postgres"`: ClassCredentials,
`dial tcp 127.0.0.1:5432: connect: connection refused`: ClassUnreachable,
`context deadline exceeded`: ClassUnreachable,
`extension "nope" is not available`: ClassRefused,
} {
if got := ClassOf(text); got != want {
t.Errorf("%s: %s, want %s", text, got, want)
}
}
}
func TestAnAdapterThatClassesItsOwnErrorsIsBelieved(t *testing.T) {
w, said := standingWorld(t)
w.h.Adapter = classing{w.a}
w.a.failing = errors.New("anything")
w.give(map[string]any{"as": "a"})
w.passes(6 * time.Minute)
if (*said)[0].body["class"] != "its-own" {
t.Fatal((*said)[0].body)
}
}
type classing struct{ *recorder }
func (classing) Class(error) string { return "its-own" }
// A provider restarted after announcing a failure has forgotten it; its first success for each
// consumer is announced, so the controller clears what it kept rather than naming it for ever.
func TestTheFirstSuccessSinceStartIsAnnouncedOnce(t *testing.T) {
w := newWorld(t)
var said []announced
w.h.Announce = func(e string, b map[string]any) { said = append(said, announced{e, b}) }
w.give(map[string]any{"as": "a"})
w.passes(3 * time.Minute)
if events(said) != EventRecovered || said[0].body["why"] != "first-success" || said[0].body["consumer"] != "a" {
t.Fatalf("%v", said)
}
}
modules/keycloak/cmd/keycloak-provider/tools.go
+414
View File
@@ -0,0 +1,414 @@
package main
// keycloak's tools — ported from tools/index.ts with the same names, arguments and answers. Write
// actions announce themselves at the point they succeed, in the module's single event vocabulary:
//
// user.created / user.deleted an identity appeared or was removed
// password.reset a user's credential was reset (no secret in the body)
// client.created an OIDC client was registered
// group.created / role.created a group or a realm role was created
// admin.repaired / .unrepaired the guard set the admin to the mesh's password, or could not
//
// Keycloak consumes nothing: it is upstream of everything that authenticates against it.
import (
"context"
"fmt"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
func prop(kind, description string) map[string]any {
return map[string]any{"type": kind, "description": description}
}
var realmProp = prop("string", "realm name (defaults to the module's realm)")
func str(args map[string]any, key string) string {
switch v := args[key].(type) {
case string:
return v
case nil:
return ""
default:
return fmt.Sprint(v)
}
}
func flag(args map[string]any, key string) (value, set bool) {
v, ok := args[key].(bool)
return v, ok
}
func number(args map[string]any, key string) int {
switch v := args[key].(type) {
case float64:
return int(v)
case int:
return v
}
return 0
}
func pick(r Rep, keys ...string) Rep {
out := Rep{}
for _, k := range keys {
if v, ok := r[k]; ok {
out[k] = v
}
}
return out
}
func bg() (context.Context, context.CancelFunc) {
return context.WithTimeout(context.Background(), time.Minute)
}
// Tools are keycloak's own; the guard answers keycloak_admin_check.
func Tools(kc *Client, guard *Guard) []stdio.Tool {
realmOf := func(args map[string]any) string {
if r := str(args, "realm"); r != "" {
return r
}
return kc.DefaultRealm
}
tool := func(name, description string, input map[string]any, run func(ctx context.Context, args map[string]any) (any, error)) stdio.Tool {
return stdio.Tool{Name: name, Description: description, Input: input, Run: func(args map[string]any) (any, error) {
ctx, cancel := bg()
defer cancel()
return run(ctx, args)
}}
}
userID := prop("string", "user ID (UUID)")
return []stdio.Tool{
// The guard
{
Name: "keycloak_admin_check",
Description: "Check that Keycloak's admin logs in with the password the mesh minted. With repair: true, a " +
"refused admin is repaired now — its password set to the mesh's through a temporary bootstrap admin " +
"inside the container, which is removed again — even inside the brake a failed automatic repair set.",
Input: map[string]any{"repair": prop("boolean", "repair a refused admin now (default false: only check)")},
Run: func(args map[string]any) (any, error) {
repair, _ := flag(args, "repair")
ctx, cancel := context.WithTimeout(context.Background(), 6*time.Minute)
defer cancel()
return guard.Ensure(ctx, repair, true), nil
},
},
// Realms & sessions
tool("keycloak_list_realms", "List all Keycloak realms.", map[string]any{},
func(ctx context.Context, _ map[string]any) (any, error) {
realms, err := kc.ListRealms(ctx)
if err != nil {
return nil, err
}
out := []Rep{}
for _, r := range realms {
out = append(out, pick(r, "id", "realm", "displayName", "enabled"))
}
return map[string]any{"realms": out}, nil
}),
tool("keycloak_list_sessions", "List active sessions for a user in a Keycloak realm.",
map[string]any{"realm": realmProp, "user_id": userID},
func(ctx context.Context, a map[string]any) (any, error) {
s, err := kc.UserSessions(ctx, realmOf(a), str(a, "user_id"))
return map[string]any{"sessions": s}, err
}),
// Users
tool("keycloak_list_users", "List users in a Keycloak realm.",
map[string]any{"realm": realmProp, "search": prop("string", "search by username, email, first/last name"),
"max": prop("number", "maximum number of results")},
func(ctx context.Context, a map[string]any) (any, error) {
u, err := kc.ListUsers(ctx, realmOf(a), str(a, "search"), number(a, "max"))
return map[string]any{"users": u}, err
}),
tool("keycloak_create_user", "Create a user in a Keycloak realm.",
map[string]any{"realm": realmProp, "username": prop("string", "username"), "email": prop("string", "email address"),
"password": prop("string", "initial password"),
"temporary_password": prop("boolean", "require a password change on first login (default true)")},
func(ctx context.Context, a map[string]any) (any, error) {
realm, username, email := realmOf(a), str(a, "username"), str(a, "email")
rep := Rep{"username": username}
if email != "" {
rep["email"] = email
}
if pw := str(a, "password"); pw != "" {
temporary, set := flag(a, "temporary_password")
rep["credentials"] = []any{Rep{"type": "password", "value": pw, "temporary": temporary || !set}}
}
if err := kc.CreateUser(ctx, realm, rep); err != nil {
return nil, err
}
body := map[string]any{"realm": realm, "username": username}
if email != "" {
body["email"] = email
}
announce("user.created", body)
return map[string]any{"created": body}, nil
}),
tool("keycloak_delete_user", "Delete a user from a Keycloak realm (requires confirm).",
map[string]any{"realm": realmProp, "user_id": userID, "confirm": prop("boolean", "must be true to confirm deletion")},
func(ctx context.Context, a map[string]any) (any, error) {
realm, id := realmOf(a), str(a, "user_id")
if ok, _ := flag(a, "confirm"); !ok {
return map[string]any{"aborted": "confirm must be true to delete a user"}, nil
}
if err := kc.DeleteUser(ctx, realm, id); err != nil {
return nil, err
}
announce("user.deleted", map[string]any{"realm": realm, "userId": id})
return map[string]any{"deleted": map[string]any{"realm": realm, "userId": id}}, nil
}),
tool("keycloak_update_user", "Update a user's attributes in a Keycloak realm (enable/disable, change email, name).",
map[string]any{"realm": realmProp, "user_id": userID, "enabled": prop("boolean", "enable or disable the user"),
"email": prop("string", "new email address"), "firstName": prop("string", "new first name"),
"lastName": prop("string", "new last name")},
func(ctx context.Context, a map[string]any) (any, error) {
realm, id := realmOf(a), str(a, "user_id")
updates := Rep{}
var fields []string
if v, set := flag(a, "enabled"); set {
updates["enabled"], fields = v, append(fields, "enabled")
}
for _, k := range []string{"email", "firstName", "lastName"} {
if _, set := a[k]; set {
updates[k], fields = str(a, k), append(fields, k)
}
}
if len(updates) == 0 {
return map[string]any{"aborted": "no updates provided"}, nil
}
if err := kc.UpdateUser(ctx, realm, id, updates); err != nil {
return nil, err
}
return map[string]any{"updated": map[string]any{"realm": realm, "userId": id, "fields": fields}}, nil
}),
tool("keycloak_reset_password", "Reset a user's password in a Keycloak realm.",
map[string]any{"realm": realmProp, "user_id": userID, "password": prop("string", "new password"),
"temporary": prop("boolean", "require a password change on next login (default false)")},
func(ctx context.Context, a map[string]any) (any, error) {
realm, id := realmOf(a), str(a, "user_id")
temporary, _ := flag(a, "temporary")
if err := kc.ResetPassword(ctx, realm, id, str(a, "password"), temporary); err != nil {
return nil, err
}
announce("password.reset", map[string]any{"realm": realm, "userId": id})
return map[string]any{"reset": map[string]any{"realm": realm, "userId": id}}, nil
}),
// Clients
tool("keycloak_list_clients", "List OIDC clients in a Keycloak realm.", map[string]any{"realm": realmProp},
func(ctx context.Context, a map[string]any) (any, error) {
clients, err := kc.ListClients(ctx, realmOf(a))
if err != nil {
return nil, err
}
out := []Rep{}
for _, c := range clients {
out = append(out, pick(c, "id", "clientId", "name", "enabled", "protocol", "publicClient", "rootUrl"))
}
return map[string]any{"clients": out}, nil
}),
tool("keycloak_create_client", "Create an OIDC client in a Keycloak realm.",
map[string]any{"realm": realmProp, "client_id": prop("string", "client ID (e.g. 'my-app')"),
"name": prop("string", "display name"), "root_url": prop("string", "root URL of the application"),
"redirect_uris": prop("array", "allowed redirect URIs"),
"public_client": prop("boolean", "public client, no client secret (default true)")},
func(ctx context.Context, a map[string]any) (any, error) {
realm, clientID, name := realmOf(a), str(a, "client_id"), str(a, "name")
public, set := flag(a, "public_client")
rep := Rep{"protocol": "openid-connect", "enabled": true, "clientId": clientID, "publicClient": public || !set}
if name != "" {
rep["name"] = name
}
if u := str(a, "root_url"); u != "" {
rep["rootUrl"] = u
}
if list, ok := a["redirect_uris"].([]any); ok {
uris := []any{}
for _, u := range list {
uris = append(uris, fmt.Sprint(u))
}
rep["redirectUris"] = uris
}
if err := kc.CreateClient(ctx, realm, rep); err != nil {
return nil, err
}
body := map[string]any{"realm": realm, "clientId": clientID}
if name != "" {
body["name"] = name
}
announce("client.created", body)
return map[string]any{"created": body}, nil
}),
tool("keycloak_delete_client", "Delete an OIDC client from a Keycloak realm (requires confirm).",
map[string]any{"realm": realmProp, "client_id": prop("string", "client ID (e.g. 'my-app')"),
"confirm": prop("boolean", "must be true to confirm deletion")},
func(ctx context.Context, a map[string]any) (any, error) {
realm, clientID := realmOf(a), str(a, "client_id")
if ok, _ := flag(a, "confirm"); !ok {
return map[string]any{"aborted": "confirm must be true to delete a client"}, nil
}
if err := kc.DeleteClient(ctx, realm, clientID); err != nil {
return nil, err
}
return map[string]any{"deleted": map[string]any{"realm": realm, "clientId": clientID}}, nil
}),
tool("keycloak_get_client_secret", "Get the client secret for a confidential OIDC client.",
map[string]any{"realm": realmProp, "client_id": prop("string", "client ID")},
func(ctx context.Context, a map[string]any) (any, error) {
s, err := kc.GetClientSecret(ctx, realmOf(a), str(a, "client_id"))
return map[string]any{"secret": s}, err
}),
tool("keycloak_add_protocol_mapper",
"Add a protocol mapper to an OIDC client. Common types: oidc-usermodel-realm-role-mapper "+
"(realm roles), oidc-usermodel-attribute-mapper (user attributes), oidc-audience-mapper.",
map[string]any{"realm": realmProp, "client_id": prop("string", "client ID (e.g. 'grafana')"),
"name": prop("string", "mapper name (e.g. 'realm roles')"),
"mapper_type": prop("string", "protocol mapper type (e.g. 'oidc-usermodel-realm-role-mapper')"),
"claim_name": prop("string", "token claim name (e.g. 'realm_access.roles')"),
"claim_type": prop("string", "JSON type: String, long, int, boolean (default String)"),
"multivalued": prop("boolean", "whether the claim has multiple values (default false)"),
"id_token": prop("boolean", "include in ID token (default true)"),
"access_token": prop("boolean", "include in access token (default true)"),
"userinfo": prop("boolean", "include in userinfo response (default true)")},
func(ctx context.Context, a map[string]any) (any, error) {
realm, clientID, name := realmOf(a), str(a, "client_id"), str(a, "name")
claimType := str(a, "claim_type")
if claimType == "" {
claimType = "String"
}
on := func(key string) string {
v, set := flag(a, key)
return fmt.Sprint(v || !set)
}
multi, _ := flag(a, "multivalued")
err := kc.AddProtocolMapper(ctx, realm, clientID, Rep{
"name": name, "protocolMapper": str(a, "mapper_type"),
"config": map[string]any{
"claim.name": str(a, "claim_name"), "jsonType.label": claimType,
"multivalued": fmt.Sprint(multi), "id.token.claim": on("id_token"),
"access.token.claim": on("access_token"), "userinfo.token.claim": on("userinfo"),
},
})
if err != nil {
return nil, err
}
return map[string]any{"added": map[string]any{"realm": realm, "clientId": clientID, "mapper": name}}, nil
}),
// Groups
tool("keycloak_list_groups", "List groups in a Keycloak realm.", map[string]any{"realm": realmProp},
func(ctx context.Context, a map[string]any) (any, error) {
g, err := kc.ListGroups(ctx, realmOf(a))
return map[string]any{"groups": g}, err
}),
tool("keycloak_create_group", "Create a group in a Keycloak realm.",
map[string]any{"realm": realmProp, "name": prop("string", "group name")},
func(ctx context.Context, a map[string]any) (any, error) {
realm, name := realmOf(a), str(a, "name")
if err := kc.CreateGroup(ctx, realm, name); err != nil {
return nil, err
}
announce("group.created", map[string]any{"realm": realm, "name": name})
return map[string]any{"created": map[string]any{"realm": realm, "group": name}}, nil
}),
tool("keycloak_get_user_groups", "List the groups a user belongs to in a Keycloak realm.",
map[string]any{"realm": realmProp, "user_id": userID},
func(ctx context.Context, a map[string]any) (any, error) {
g, err := kc.UserGroups(ctx, realmOf(a), str(a, "user_id"))
return map[string]any{"groups": g}, err
}),
tool("keycloak_add_user_to_group", "Add a user to a group in a Keycloak realm.",
map[string]any{"realm": realmProp, "user_id": userID, "group_id": prop("string", "group ID (UUID)")},
func(ctx context.Context, a map[string]any) (any, error) {
realm := realmOf(a)
if err := kc.AddUserToGroup(ctx, realm, str(a, "user_id"), str(a, "group_id")); err != nil {
return nil, err
}
return map[string]any{"added": map[string]any{"realm": realm, "userId": str(a, "user_id"), "groupId": str(a, "group_id")}}, nil
}),
tool("keycloak_remove_user_from_group", "Remove a user from a group in a Keycloak realm.",
map[string]any{"realm": realmProp, "user_id": userID, "group_id": prop("string", "group ID (UUID)")},
func(ctx context.Context, a map[string]any) (any, error) {
realm := realmOf(a)
if err := kc.RemoveUserFromGroup(ctx, realm, str(a, "user_id"), str(a, "group_id")); err != nil {
return nil, err
}
return map[string]any{"removed": map[string]any{"realm": realm, "userId": str(a, "user_id"), "groupId": str(a, "group_id")}}, nil
}),
// Roles
tool("keycloak_get_user_roles", "List the realm roles assigned to a user in a Keycloak realm.",
map[string]any{"realm": realmProp, "user_id": userID},
func(ctx context.Context, a map[string]any) (any, error) {
r, err := kc.UserRealmRoles(ctx, realmOf(a), str(a, "user_id"))
return map[string]any{"roles": r}, err
}),
tool("keycloak_create_role", "Create a realm role in a Keycloak realm.",
map[string]any{"realm": realmProp, "role_name": prop("string", "role name"), "description": prop("string", "role description")},
func(ctx context.Context, a map[string]any) (any, error) {
realm, name := realmOf(a), str(a, "role_name")
rep := Rep{"name": name}
if d := str(a, "description"); d != "" {
rep["description"] = d
}
if err := kc.CreateRealmRole(ctx, realm, rep); err != nil {
return nil, err
}
announce("role.created", map[string]any{"realm": realm, "name": name})
return map[string]any{"created": map[string]any{"realm": realm, "role": name}}, nil
}),
tool("keycloak_assign_user_role", "Assign an existing realm role to a user. Create it first with keycloak_create_role if needed.",
map[string]any{"realm": realmProp, "user_id": userID, "role_name": prop("string", "role name to assign")},
func(ctx context.Context, a map[string]any) (any, error) {
realm, id, role := realmOf(a), str(a, "user_id"), str(a, "role_name")
// The mapping API needs the role's UUID, which only the "available" list carries; a role
// neither available nor assigned does not exist in this realm.
available, err := kc.AvailableRealmRoles(ctx, realm, id)
if err != nil {
return nil, err
}
for _, r := range available {
if r["name"] == role {
if err := kc.AssignRealmRoles(ctx, realm, id, []Rep{pick(r, "id", "name")}); err != nil {
return nil, err
}
return map[string]any{"assigned": map[string]any{"realm": realm, "userId": id, "role": role}}, nil
}
}
assigned, err := kc.UserRealmRoles(ctx, realm, id)
if err != nil {
return nil, err
}
for _, r := range assigned {
if r["name"] == role {
return map[string]any{"alreadyAssigned": map[string]any{"realm": realm, "userId": id, "role": role}}, nil
}
}
return map[string]any{"notFound": map[string]any{"realm": realm, "role": role}}, nil
}),
tool("keycloak_remove_user_role", "Remove a realm role from a user in a Keycloak realm.",
map[string]any{"realm": realmProp, "user_id": userID, "role_name": prop("string", "role name to remove")},
func(ctx context.Context, a map[string]any) (any, error) {
realm, id, role := realmOf(a), str(a, "user_id"), str(a, "role_name")
assigned, err := kc.UserRealmRoles(ctx, realm, id)
if err != nil {
return nil, err
}
for _, r := range assigned {
if r["name"] == role {
if err := kc.RemoveRealmRoles(ctx, realm, id, []Rep{pick(r, "id", "name")}); err != nil {
return nil, err
}
return map[string]any{"removed": map[string]any{"realm": realm, "userId": id, "role": role}}, nil
}
}
return map[string]any{"notAssigned": map[string]any{"realm": realm, "userId": id, "role": role}}, nil
}),
}
}
modules/keycloak/go.mod
+5
View File
@@ -0,0 +1,5 @@
module keycloak
go 1.25.0
require git.novox.be/novox/mesh-sdk/go v0.1.7
modules/keycloak/go.sum
+2
View File
@@ -0,0 +1,2 @@
git.novox.be/novox/mesh-sdk/go v0.1.7 h1:C0sTQmtTiyYH7bnqZb7PusXnqA37gKuT7Nqjn9gG47w=
git.novox.be/novox/mesh-sdk/go v0.1.7/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
modules/keycloak/index.ts
-44
View File
@@ -1,44 +0,0 @@
// keycloak's events. Keycloak's worth to the mesh is in what it changes — an identity created, a
// client registered, a password reset — so its events are emitted from the admin actions themselves
// (novox/hq ADR 0041/0042), not scraped back by polling. This module is the single vocabulary for
// them: every keycloak event goes through one of the helpers here, and the tools call them at the
// point the change succeeds.
//
// Emits:
// module.keycloak.user.created / .deleted — an identity appeared or was removed
// module.keycloak.password.reset — a user's credential was reset (no secret in the body)
// module.keycloak.client.created — an OIDC client was registered
// module.keycloak.group.created — a group was created
// module.keycloak.role.created — a realm role was created
// Consumes:
// nothing — Keycloak is upstream of the things that authenticate against it; it reacts to none of
// their events. There is no honest `on(...)` to write, so there is none.
import { emit } from "@novox/mesh-sdk/events";
// A completed admin action must not be undone by a flaky broker: the change already happened in
// Keycloak, so a failed emit is logged and swallowed rather than thrown back through the tool.
async function announce(type: string, body: Record<string, unknown>): Promise<void> {
try {
await emit(type, body);
} catch (err) {
console.error(`[keycloak] emit ${type} failed: ${err}`);
}
}
export const events = {
userCreated: (realm: string, username: string, email?: string) =>
announce("user.created", { realm, username, ...(email ? { email } : {}) }),
userDeleted: (realm: string, userId: string) =>
announce("user.deleted", { realm, userId }),
passwordReset: (realm: string, userId: string) =>
announce("password.reset", { realm, userId }),
clientCreated: (realm: string, clientId: string, name?: string) =>
announce("client.created", { realm, clientId, ...(name ? { name } : {}) }),
groupCreated: (realm: string, name: string) =>
announce("group.created", { realm, name }),
roleCreated: (realm: string, name: string) =>
announce("role.created", { realm, name }),
};
console.log("[keycloak] event surface ready — identity, client, group and role changes are announced");
modules/keycloak/module.json
+10 -11
View File
@@ -36,7 +36,9 @@
"password.reset",
"client.created",
"group.created",
"role.created"
"role.created",
"admin.repaired",
"admin.unrepaired"
],
"listens": [
{
@@ -150,22 +152,19 @@
{
"name": "code",
"kind": "bundle",
"language": "typescript",
"entrypoints": [
"index.js",
"tools/index.js",
"provisioner/index.js"
],
"language": "go",
"system": "arch",
"from": "cmd/keycloak-provider",
"binary": "keycloak-provider",
"loads": [
"index.js",
"tools/index.js",
"provisioner/index.js"
"keycloak-provider"
],
"env": {
"MESH_KEYCLOAK_URL": "http://127.0.0.1:${port:8080}",
"MESH_KEYCLOAK_CONFIG_FILE": "${dir:mesh-state}/config.json",
"MESH_KEYCLOAK_PASSWORD_FILE": "${dir:state}/admin.secret",
"MESH_RECEIVES": "${dir:grants}/mesh.json"
"MESH_RECEIVES": "${dir:grants}/mesh.json",
"MESH_KEYCLOAK_CONTAINER": "keycloak"
}
}
]
modules/keycloak/oidc.ts
-185
View File
@@ -1,185 +0,0 @@
// What the `oidc-client` provision means in Keycloak: one confidential OpenID Connect client per
// consumer, in the realm this module serves, under the name and secret the mesh gave both ends.
// The provisioner (provisioner/index.ts) is the sdk harness calling these; they are here, apart from
// it, so they can be exercised against a fake admin API without a broker or a contributions file.
//
// **The client id and the secret are the mesh's, not Keycloak's (novox/hq ADR 0048).** The mesh
// derives the consumer's identity (`as`, e.g. `mesh_ace_grafana`) and hands it to both ends — the
// consumer names it as its client id through `${bound:oidc-client:as}` — and mints the secret, which
// this sets as the client's secret. Keycloak generates neither.
//
// **Where the consumer's browser comes back to is the consumer's to say.** Its contribution carries
// `callback` (a path, e.g. `/login/generic_oauth`) and the `label`/`endpoint` of the endpoint it is
// reached on; the mesh composes that endpoint's names into `name` (public) and `internal-name`
// (private network) exactly as it does for a route (novox/hq ADR 0056, 0138), so the redirect URI
// registered here is built from the same names the proxy serves the consumer under.
//
// **Only what the mesh made is touched.** A client this module creates carries the attribute
// `mesh.provisioned=true`, and its id starts with the mesh's own prefix. A client with the same id
// that lacks the mark is somebody else's: it is refused, never adopted, never updated, never deleted.
import type { ClientRepresentation, KeycloakClient, ProtocolMapperRepresentation } from "./client.js";
/** The attribute marking a client as the mesh's own work. */
export const MARK = "mesh.provisioned";
/** The mapper every mesh client carries: realm roles as a flat `roles` claim in the id token, the
* access token and userinfo — what a consumer maps its own roles from (grafana's role path reads
* `roles[*]`), and what the predecessor added to its hand-made clients by hand. */
export const ROLES_MAPPER: ProtocolMapperRepresentation = {
name: "realm roles",
protocol: "openid-connect",
protocolMapper: "oidc-usermodel-realm-role-mapper",
config: {
"claim.name": "roles",
"jsonType.label": "String",
multivalued: "true",
"id.token.claim": "true",
"access.token.claim": "true",
"userinfo.token.claim": "true",
},
};
/** One consumer, as the harness hands it over. */
export interface OidcGrant {
readonly as: string;
readonly password: string;
readonly values: Readonly<Record<string, unknown>>;
readonly consumer?: string;
}
/** The realm named by an issuer URL — `https://id.example/realms/Novox` is realm `Novox`. The issuer is
* the one value an assignment sets (it is also what consumers are served), so the realm is read
* out of it rather than set a second time where the two could disagree. */
export function realmOf(issuer: string): string {
let path: string;
try {
path = new URL(issuer).pathname;
} catch {
throw new Error(`the issuer ${JSON.stringify(issuer)} is not a URL`);
}
const m = /\/realms\/([^/]+)\/?$/.exec(path);
if (!m) throw new Error(`the issuer ${JSON.stringify(issuer)} does not end in /realms/<realm>`);
return decodeURIComponent(m[1]);
}
/** The redirect URIs a consumer's contribution asks for: its callback under each name the mesh
* composed for its endpoint. Refused when there is nothing to register — a client that accepts no
* redirect is a client nobody can log in through, and one that accepts any is worse. */
export function redirectsOf(values: Readonly<Record<string, unknown>>): { root: string; redirects: string[] } {
const callback = values.callback;
if (typeof callback !== "string" || !callback.startsWith("/")) {
throw new Error(`contributes no callback path (\`callback\`, starting with "/"): ${JSON.stringify(callback)}`);
}
const names: string[] = [];
for (const key of ["name", "internal-name"]) {
const n = values[key];
if (typeof n === "string" && n.trim() !== "" && !names.includes(n.trim())) names.push(n.trim());
}
if (names.length === 0) {
throw new Error("has no name the mesh composed (`name` / `internal-name`) — contribute a `label` and the `endpoint` it is reached on");
}
return { root: `https://${names[0]}`, redirects: names.map((n) => `https://${n}${callback}`) };
}
/** The fields the mesh owns on a client it made. Everything else on the client is left as found. */
function wanted(g: OidcGrant): ClientRepresentation {
const { root, redirects } = redirectsOf(g.values);
return {
clientId: g.as,
name: g.as,
description: `made by the mesh for ${g.consumer ? `a module on ${g.consumer}` : "a consumer"} — do not edit; it is reset`,
enabled: true,
protocol: "openid-connect",
publicClient: false,
clientAuthenticatorType: "client-secret",
secret: g.password,
rootUrl: root,
baseUrl: root,
redirectUris: redirects,
standardFlowEnabled: true,
implicitFlowEnabled: false,
directAccessGrantsEnabled: false,
serviceAccountsEnabled: false,
};
}
function sameSet(a: readonly string[] | undefined, b: readonly string[]): boolean {
const x = [...(a ?? [])].sort();
const y = [...b].sort();
return x.length === y.length && x.every((v, i) => v === y[i]);
}
function marked(c: ClientRepresentation): boolean {
return c.attributes?.[MARK] === "true";
}
export class OidcClients {
constructor(private readonly kc: KeycloakClient, readonly realm: string) {}
/** Create the consumer's client, or bring the mesh's existing one back to what the grant says.
* Returns whether it was newly created. Idempotent: applying the same grant twice changes nothing
* the second time beyond re-asserting it. */
async ensure(g: OidcGrant): Promise<"created" | "updated"> {
const want = wanted(g);
const found = await this.kc.findClient(this.realm, g.as);
if (found && !marked(found)) {
throw new Error(
`realm ${this.realm} already has a client ${g.as} the mesh did not make — left alone; ` +
`delete or rename it if the mesh should own that id`);
}
if (!found) {
await this.kc.createClientFrom(this.realm, {
...want,
attributes: { [MARK]: "true" },
protocolMappers: [ROLES_MAPPER],
});
return "created";
}
// Overlay what the mesh owns on what is there, so a field Keycloak added or an operator set on a
// field the mesh does not own survives the update.
await this.kc.updateClient(this.realm, found.id!, {
...found,
...want,
attributes: { ...(found.attributes ?? {}), [MARK]: "true" },
});
await this.ensureMapper(found.id!);
return "updated";
}
private async ensureMapper(id: string): Promise<void> {
const mappers = await this.kc.listClientMappers(this.realm, id);
const have = mappers.find((m) => m.name === ROLES_MAPPER.name);
if (!have) {
await this.kc.addClientMapper(this.realm, id, ROLES_MAPPER);
return;
}
const drifted =
have.protocolMapper !== ROLES_MAPPER.protocolMapper ||
Object.entries(ROLES_MAPPER.config).some(([k, v]) => have.config?.[k] !== v);
if (drifted) {
await this.kc.updateClientMapper(this.realm, id, { ...ROLES_MAPPER, id: have.id });
}
}
/** Whether Keycloak still holds this consumer's client exactly as the grant says: present, the
* mesh's, enabled, confidential, with the mesh's secret and the redirects asked for. Reads only. */
async holds(g: OidcGrant): Promise<boolean> {
const want = wanted(g);
const found = await this.kc.findClient(this.realm, g.as);
if (!found || !marked(found) || found.enabled === false || found.publicClient) return false;
if (!sameSet(found.redirectUris, want.redirectUris!)) return false;
const mappers = await this.kc.listClientMappers(this.realm, found.id!);
if (!mappers.some((m) => m.name === ROLES_MAPPER.name)) return false;
return (await this.kc.clientSecretById(this.realm, found.id!)) === g.password;
}
/** Withdraw a consumer's client — only one the mesh made. Returns what happened, for the log. */
async remove(as: string): Promise<"removed" | "absent" | "not ours"> {
const found = await this.kc.findClient(this.realm, as);
if (!found) return "absent";
if (!marked(found)) return "not ours";
await this.kc.deleteClientById(this.realm, found.id!);
return "removed";
}
}
modules/keycloak/package.json
-18
View File
@@ -1,18 +0,0 @@
{
"name": "@novox/module-keycloak",
"version": "0.1.0",
"description": "keycloak — identity and access; provides the mesh oidc-client interface. Its admin API client, provisioner, tools and events live here (novox/hq ADR 0039).",
"type": "module",
"private": true,
"scripts": {
"build": "tsc client.ts oidc.ts index.ts provisioner/index.ts tools/index.ts --module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist",
"test": "npm run build && node --test --experimental-strip-types 'test/*.test.ts'"
},
"dependencies": {
"@novox/mesh-sdk": "^0.1.1"
},
"devDependencies": {
"@types/node": "^22.0.0",
"typescript": "^5.6.0"
}
}
modules/keycloak/provisioner/index.ts
-73
View File
@@ -1,73 +0,0 @@
// keycloak's provisioner — the adapter that makes keycloak a provider of the mesh `oidc-client`
// interface. The reconcile loop, the contributions file and reading the mesh's minted secret are the
// sdk harness's; this writes only the per-service half: how Keycloak creates, checks and removes a
// consumer's client (novox/hq ADR 0039/0040/0048). What a client is, and which ones are the mesh's,
// is in ../oidc.ts.
//
// The `oidc-client` interface: a consumer logs people in through the realm this module serves, as
// the confidential client `as` with the secret the mesh minted, and is redirected back to the
// callback it contributed under the names the mesh composed for its endpoint. What it is served —
// the issuer and the endpoint paths under it — is in the manifest's `serves`, settled with the
// assignment's settings.
//
// **The realm is read out of the issuer**, the one value an assignment sets (settings reach both the
// served facts and this module's config.json): a realm set in one place and an issuer in another
// would let the consumer be told one realm while its client is made in another.
import { runProvisioner, type Provision } from "@novox/mesh-sdk/provisioner";
import { emit } from "@novox/mesh-sdk/events";
import { readFileSync } from "node:fs";
import { KeycloakClient } from "../client.js";
import { OidcClients, realmOf } from "../oidc.js";
/** The issuer this assignment serves, from the settings-merged config the mesh delivers. */
function issuer(): string {
const file = process.env.MESH_KEYCLOAK_CONFIG_FILE;
let cfg: Record<string, unknown> = {};
if (file) {
try {
cfg = JSON.parse(readFileSync(file, "utf8")) as Record<string, unknown>;
} catch {
// Absent or unreadable: fall through to the environment, and refuse below if that is empty too.
}
}
const said = typeof cfg.issuer === "string" ? cfg.issuer : process.env.MESH_KEYCLOAK_ISSUER;
if (!said) throw new Error("no issuer — the module's config.json carries none and MESH_KEYCLOAK_ISSUER is unset");
return said;
}
const clients = new OidcClients(KeycloakClient.fromEnv(), realmOf(issuer()));
/** Emit a lifecycle event without letting a broker hiccup fail the provisioning itself. */
async function announce(type: string, body: Record<string, string>): Promise<void> {
try {
await emit(type, body);
} catch (err) {
console.error(`[provisioner:oidc-client] emit ${type} failed: ${err}`);
}
}
runProvisioner("oidc-client", {
async create(p: Provision): Promise<void> {
const done = await clients.ensure(p);
if (done === "created") {
console.log(`[provisioner:oidc-client] created client ${p.as} in realm ${clients.realm}`);
await announce("client.created", { realm: clients.realm, clientId: p.as, consumer: p.consumer ?? "" });
}
},
async remove(p: { as: string }): Promise<void> {
const done = await clients.remove(p.as);
if (done === "not ours") {
console.error(`[provisioner:oidc-client] ${p.as}: a client of that id exists that the mesh did not make — left alone`);
} else if (done === "removed") {
console.log(`[provisioner:oidc-client] removed client ${p.as} from realm ${clients.realm}`);
}
},
// Asked every minute by the harness: whether Keycloak still holds this consumer's client exactly as
// the mesh gave it, so a client deleted or edited behind the mesh's back is made again (hq issue 120).
async holds(p: Provision): Promise<boolean> {
return clients.holds(p);
},
});
modules/keycloak/test/oidc.test.ts
-239
View File
@@ -1,239 +0,0 @@
// What holds keycloak to the `oidc-client` provision (oidc.ts): one confidential client per consumer,
// under the id and secret the mesh gave, redirecting only to the consumer's own callback under the
// names the mesh composed; made once and brought back on every apply; and a client the mesh did not
// make — same id or not — never adopted, changed or deleted.
//
// Keycloak is a fake: the admin routes the module touches, answering with the status codes and the
// shapes Keycloak gives. Run against the compiled module (npm test builds first), the way the runtime
// loads it.
import { test, after } from "node:test";
import assert from "node:assert/strict";
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
import { randomUUID } from "node:crypto";
import { KeycloakClient } from "../dist/client.js";
import { MARK, OidcClients, ROLES_MAPPER, realmOf, redirectsOf } from "../dist/oidc.js";
type Client = Record<string, any>;
/** The realm's clients, by internal id, and what the fake was asked. */
const realm = "Novox";
const clients = new Map<string, Client>();
const calls: string[] = [];
function body(req: IncomingMessage): Promise<any> {
return new Promise((resolve) => {
let raw = "";
req.on("data", (c) => (raw += c));
req.on("end", () => resolve(raw ? JSON.parse(raw) : undefined));
});
}
function send(res: ServerResponse, status: number, value?: unknown): void {
res.writeHead(status, { "Content-Type": "application/json" });
res.end(value === undefined ? "" : JSON.stringify(value));
}
const server = createServer(async (req, res) => {
const url = new URL(req.url!, "http://fake");
calls.push(`${req.method} ${url.pathname}`);
if (url.pathname === "/realms/master/protocol/openid-connect/token") {
return send(res, 200, { access_token: "t", expires_in: 300 });
}
const base = `/admin/realms/${realm}/clients`;
if (!url.pathname.startsWith(base)) return send(res, 404, { error: "Realm not found." });
const rest = url.pathname.slice(base.length).split("/").filter(Boolean);
if (rest.length === 0 && req.method === "GET") {
const want = url.searchParams.get("clientId");
return send(res, 200, [...clients.values()].filter((c) => !want || c.clientId === want));
}
if (rest.length === 0 && req.method === "POST") {
const rep = await body(req);
if ([...clients.values()].some((c) => c.clientId === rep.clientId)) {
return send(res, 409, { errorMessage: `Client ${rep.clientId} already exists` });
}
const id = randomUUID();
const mappers = (rep.protocolMappers ?? []).map((m: Client) => ({ ...m, id: randomUUID() }));
clients.set(id, { ...rep, id, protocolMappers: mappers });
return send(res, 201);
}
const c = clients.get(rest[0]);
if (!c) return send(res, 404, { error: "Could not find client" });
if (rest.length === 1 && req.method === "PUT") {
// Keycloak ignores protocolMappers on a client update: they have their own endpoints.
const rep = await body(req);
clients.set(c.id, { ...rep, id: c.id, protocolMappers: c.protocolMappers });
return send(res, 204);
}
if (rest.length === 1 && req.method === "DELETE") {
clients.delete(c.id);
return send(res, 204);
}
if (rest[1] === "client-secret" && req.method === "GET") {
return send(res, 200, { type: "secret", value: c.secret });
}
if (rest[1] === "protocol-mappers") {
if (req.method === "GET") return send(res, 200, c.protocolMappers ?? []);
if (req.method === "POST") {
c.protocolMappers = [...(c.protocolMappers ?? []), { ...(await body(req)), id: randomUUID() }];
return send(res, 201);
}
if (req.method === "PUT") {
const m = await body(req);
c.protocolMappers = c.protocolMappers.map((x: Client) => (x.id === rest[4] ? m : x));
return send(res, 204);
}
}
send(res, 405);
});
await new Promise<void>((r) => server.listen(0, "127.0.0.1", r));
after(() => server.close());
const port = (server.address() as { port: number }).port;
const oidc = new OidcClients(new KeycloakClient(`http://127.0.0.1:${port}`, "admin", "pw"), realm);
/** Grafana on ace, as the mesh hands it to the provisioner. */
function grafana(secret = "s3cret", values: Record<string, unknown> = {}) {
return {
as: "mesh_ace_grafana",
password: secret,
consumer: "ace",
values: {
label: "grafana", endpoint: "web", port: 20010, callback: "/login/generic_oauth",
name: "grafana.zurag.be", "internal-name": "grafana.ace.internal", ...values,
},
};
}
function only(clientId: string): Client {
const found = [...clients.values()].filter((c) => c.clientId === clientId);
assert.equal(found.length, 1, `exactly one client ${clientId}, found ${found.length}`);
return found[0];
}
test("the realm is read out of the issuer, and an issuer that names none is refused", () => {
assert.equal(realmOf("https://keycloak.novox.be/realms/Novox"), "Novox");
assert.equal(realmOf("https://keycloak.novox.be/realms/Novox/"), "Novox");
assert.equal(realmOf("http://127.0.0.1:18500/realms/master"), "master");
assert.throws(() => realmOf("https://keycloak.novox.be"), /realms/);
assert.throws(() => realmOf("keycloak"), /not a URL/);
});
test("the redirect is the consumer's callback under every name the mesh composed for it", () => {
assert.deepEqual(redirectsOf(grafana().values), {
root: "https://grafana.zurag.be",
redirects: ["https://grafana.zurag.be/login/generic_oauth", "https://grafana.ace.internal/login/generic_oauth"],
});
// A route reaching only the private network has only the internal name, and that is enough.
assert.deepEqual(redirectsOf({ callback: "/cb", "internal-name": "x.ace.internal" }).redirects,
["https://x.ace.internal/cb"]);
assert.throws(() => redirectsOf({ name: "grafana.zurag.be" }), /callback/);
assert.throws(() => redirectsOf({ name: "grafana.zurag.be", callback: "login" }), /callback/);
assert.throws(() => redirectsOf({ callback: "/cb" }), /label/);
});
test("a consumer is given one confidential client, under its id and the mesh's secret", async () => {
clients.clear();
assert.equal(await oidc.ensure(grafana()), "created");
const c = only("mesh_ace_grafana");
assert.equal(c.publicClient, false);
assert.equal(c.clientAuthenticatorType, "client-secret");
assert.equal(c.secret, "s3cret");
assert.equal(c.enabled, true);
assert.equal(c.standardFlowEnabled, true);
assert.equal(c.directAccessGrantsEnabled, false);
assert.equal(c.implicitFlowEnabled, false);
assert.deepEqual(c.redirectUris, [
"https://grafana.zurag.be/login/generic_oauth", "https://grafana.ace.internal/login/generic_oauth"]);
assert.equal(c.attributes[MARK], "true");
assert.deepEqual(c.protocolMappers.map((m: Client) => m.name), [ROLES_MAPPER.name]);
assert.equal(await oidc.holds(grafana()), true);
});
test("applying the same grant again makes no second client", async () => {
clients.clear();
await oidc.ensure(grafana());
assert.equal(await oidc.ensure(grafana()), "updated");
assert.equal(await oidc.ensure(grafana()), "updated");
only("mesh_ace_grafana");
assert.equal(only("mesh_ace_grafana").protocolMappers.length, 1, "the roles mapper is not added twice");
});
test("a new secret or a moved name is applied in place, and what the mesh does not own survives", async () => {
clients.clear();
await oidc.ensure(grafana());
const id = only("mesh_ace_grafana").id;
// Something the mesh does not own, set on the client after it was made.
clients.get(id)!.consentRequired = true;
clients.get(id)!.attributes["post.logout.redirect.uris"] = "+";
assert.equal(await oidc.holds(grafana("rotated")), false, "a rotated secret is not held until applied");
await oidc.ensure(grafana("rotated", { name: "dash.zurag.be" }));
const c = only("mesh_ace_grafana");
assert.equal(c.id, id, "updated, not replaced");
assert.equal(c.secret, "rotated");
assert.deepEqual(c.redirectUris, [
"https://dash.zurag.be/login/generic_oauth", "https://grafana.ace.internal/login/generic_oauth"]);
assert.equal(c.rootUrl, "https://dash.zurag.be");
assert.equal(c.consentRequired, true);
assert.equal(c.attributes["post.logout.redirect.uris"], "+");
assert.equal(c.attributes[MARK], "true");
assert.equal(await oidc.holds(grafana("rotated", { name: "dash.zurag.be" })), true);
});
test("a client lost or edited behind the mesh's back is not held, and is made whole again", async () => {
clients.clear();
await oidc.ensure(grafana());
const c = only("mesh_ace_grafana");
c.redirectUris = ["*"];
assert.equal(await oidc.holds(grafana()), false, "a widened redirect is not what the mesh gave");
await oidc.ensure(grafana());
assert.equal(await oidc.holds(grafana()), true);
only("mesh_ace_grafana").protocolMappers = [];
assert.equal(await oidc.holds(grafana()), false, "a client without its roles mapper is not held");
await oidc.ensure(grafana());
assert.equal(await oidc.holds(grafana()), true);
clients.clear();
assert.equal(await oidc.holds(grafana()), false);
});
test("a client of the same id the mesh did not make is refused, and left exactly as it was", async () => {
clients.clear();
clients.set("theirs", { id: "theirs", clientId: "mesh_ace_grafana", secret: "their-secret", redirectUris: ["*"] });
const before = JSON.stringify(clients.get("theirs"));
const writes = calls.length;
await assert.rejects(oidc.ensure(grafana()), /did not make/);
assert.equal(JSON.stringify(clients.get("theirs")), before);
assert.ok(calls.slice(writes).every((c) => c.startsWith("GET") || c.startsWith("POST /realms/master")),
`only reads were made: ${calls.slice(writes).join(", ")}`);
assert.equal(await oidc.holds(grafana()), false);
assert.equal(await oidc.remove("mesh_ace_grafana"), "not ours");
assert.ok(clients.has("theirs"), "a client the mesh did not make is never deleted");
});
test("the predecessor's hand-made client is never touched: the mesh's has its own id", async () => {
clients.clear();
clients.set("hal", { id: "hal", clientId: "grafana", secret: "old", redirectUris: ["https://grafana.zurag.be/*"] });
await oidc.ensure(grafana());
assert.equal(clients.get("hal")!.secret, "old");
only("mesh_ace_grafana");
assert.equal(await oidc.remove("grafana"), "not ours");
assert.ok(clients.has("hal"));
});
test("a withdrawn consumer's client is removed, and an absent one is not an error", async () => {
clients.clear();
await oidc.ensure(grafana());
assert.equal(await oidc.remove("mesh_ace_grafana"), "removed");
assert.equal([...clients.values()].length, 0);
assert.equal(await oidc.remove("mesh_ace_grafana"), "absent");
});
test("a contribution with no callback makes no client at all", async () => {
clients.clear();
await assert.rejects(oidc.ensure({ ...grafana(), values: { name: "grafana.zurag.be" } }), /callback/);
assert.equal(clients.size, 0);
});
modules/keycloak/tools/index.ts
-378
View File
@@ -1,378 +0,0 @@
// keycloak's tools — moved here from the shared sdk (novox/hq ADR 0039), importing keycloak's own
// client. They return structured data (not the hal MCP `{content:[...]}` shape); the mesh serves
// them through the sdk's tool harness. Write actions announce themselves through the module's event
// surface at the point they succeed.
import { registerModuleTools, type ToolDefinition } from "@novox/mesh-sdk/tools";
import { KeycloakClient } from "../client.js";
import { events } from "../index.js";
export function getKeycloakTools(kc: KeycloakClient): ToolDefinition[] {
// Almost every tool is realm-scoped; an omitted realm falls back to the one the module resolved
// from its environment, so the common single-realm case needs no argument.
const realmOf = (args: Readonly<Record<string, unknown>>): string =>
args.realm ? String(args.realm) : kc.defaultRealm;
return [
// Realms & sessions
{
name: "keycloak_list_realms",
description: "List all Keycloak realms.",
input: {},
run: async () => {
const realms = await kc.listRealms();
return { realms: realms.map((r) => ({ id: r.id, realm: r.realm, displayName: r.displayName, enabled: r.enabled })) };
},
},
{
name: "keycloak_list_sessions",
description: "List active sessions for a user in a Keycloak realm.",
input: {
realm: { type: "string", description: "realm name (defaults to the module's realm)" },
user_id: { type: "string", description: "user ID (UUID)" },
},
run: async (args) => ({ sessions: await kc.getUserSessions(realmOf(args), String(args.user_id)) }),
},
// Users
{
name: "keycloak_list_users",
description: "List users in a Keycloak realm.",
input: {
realm: { type: "string", description: "realm name (defaults to the module's realm)" },
search: { type: "string", description: "search by username, email, first/last name" },
max: { type: "number", description: "maximum number of results" },
},
run: async (args) => ({
users: await kc.listUsers(realmOf(args), {
search: args.search ? String(args.search) : undefined,
max: args.max ? Number(args.max) : undefined,
}),
}),
},
{
name: "keycloak_create_user",
description: "Create a user in a Keycloak realm.",
input: {
realm: { type: "string", description: "realm name (defaults to the module's realm)" },
username: { type: "string", description: "username" },
email: { type: "string", description: "email address" },
password: { type: "string", description: "initial password" },
temporary_password: { type: "boolean", description: "require a password change on first login (default true)" },
},
run: async (args) => {
const realm = realmOf(args);
const username = String(args.username);
const email = args.email ? String(args.email) : undefined;
const credentials = args.password
? [{ type: "password", value: String(args.password), temporary: args.temporary_password !== false }]
: undefined;
await kc.createUser(realm, { username, email, credentials });
await events.userCreated(realm, username, email);
return { created: { realm, username, email } };
},
},
{
name: "keycloak_delete_user",
description: "Delete a user from a Keycloak realm (requires confirm).",
input: {
realm: { type: "string", description: "realm name (defaults to the module's realm)" },
user_id: { type: "string", description: "user ID (UUID)" },
confirm: { type: "boolean", description: "must be true to confirm deletion" },
},
run: async (args) => {
const realm = realmOf(args);
const userId = String(args.user_id);
if (args.confirm !== true) return { aborted: "confirm must be true to delete a user" };
await kc.deleteUser(realm, userId);
await events.userDeleted(realm, userId);
return { deleted: { realm, userId } };
},
},
{
name: "keycloak_update_user",
description: "Update a user's attributes in a Keycloak realm (enable/disable, change email, name).",
input: {
realm: { type: "string", description: "realm name (defaults to the module's realm)" },
user_id: { type: "string", description: "user ID (UUID)" },
enabled: { type: "boolean", description: "enable or disable the user" },
email: { type: "string", description: "new email address" },
firstName: { type: "string", description: "new first name" },
lastName: { type: "string", description: "new last name" },
},
run: async (args) => {
const realm = realmOf(args);
const userId = String(args.user_id);
const updates: Record<string, unknown> = {};
if (args.enabled !== undefined) updates.enabled = args.enabled === true;
if (args.email !== undefined) updates.email = String(args.email);
if (args.firstName !== undefined) updates.firstName = String(args.firstName);
if (args.lastName !== undefined) updates.lastName = String(args.lastName);
if (Object.keys(updates).length === 0) return { aborted: "no updates provided" };
await kc.updateUser(realm, userId, updates);
return { updated: { realm, userId, fields: Object.keys(updates) } };
},
},
{
name: "keycloak_reset_password",
description: "Reset a user's password in a Keycloak realm.",
input: {
realm: { type: "string", description: "realm name (defaults to the module's realm)" },
user_id: { type: "string", description: "user ID (UUID)" },
password: { type: "string", description: "new password" },
temporary: { type: "boolean", description: "require a password change on next login (default false)" },
},
run: async (args) => {
const realm = realmOf(args);
const userId = String(args.user_id);
await kc.resetPassword(realm, userId, String(args.password), args.temporary === true);
await events.passwordReset(realm, userId);
return { reset: { realm, userId } };
},
},
// Clients
{
name: "keycloak_list_clients",
description: "List OIDC clients in a Keycloak realm.",
input: { realm: { type: "string", description: "realm name (defaults to the module's realm)" } },
run: async (args) => {
const clients = (await kc.listClients(realmOf(args))) as Array<Record<string, unknown>>;
return {
clients: clients.map((c) => ({
id: c.id, clientId: c.clientId, name: c.name, enabled: c.enabled,
protocol: c.protocol, publicClient: c.publicClient, rootUrl: c.rootUrl,
})),
};
},
},
{
name: "keycloak_create_client",
description: "Create an OIDC client in a Keycloak realm.",
input: {
realm: { type: "string", description: "realm name (defaults to the module's realm)" },
client_id: { type: "string", description: "client ID (e.g. 'my-app')" },
name: { type: "string", description: "display name" },
root_url: { type: "string", description: "root URL of the application" },
redirect_uris: { type: "array", description: "allowed redirect URIs" },
public_client: { type: "boolean", description: "public client, no client secret (default true)" },
},
run: async (args) => {
const realm = realmOf(args);
const clientId = String(args.client_id);
const name = args.name ? String(args.name) : undefined;
await kc.createClient(realm, {
clientId,
name,
rootUrl: args.root_url ? String(args.root_url) : undefined,
redirectUris: Array.isArray(args.redirect_uris) ? args.redirect_uris.map(String) : undefined,
publicClient: args.public_client !== false,
});
await events.clientCreated(realm, clientId, name);
return { created: { realm, clientId, name } };
},
},
{
name: "keycloak_delete_client",
description: "Delete an OIDC client from a Keycloak realm (requires confirm).",
input: {
realm: { type: "string", description: "realm name (defaults to the module's realm)" },
client_id: { type: "string", description: "client ID (e.g. 'my-app')" },
confirm: { type: "boolean", description: "must be true to confirm deletion" },
},
run: async (args) => {
const realm = realmOf(args);
const clientId = String(args.client_id);
if (args.confirm !== true) return { aborted: "confirm must be true to delete a client" };
await kc.deleteClient(realm, clientId);
return { deleted: { realm, clientId } };
},
},
{
name: "keycloak_get_client_secret",
description: "Get the client secret for a confidential OIDC client.",
input: {
realm: { type: "string", description: "realm name (defaults to the module's realm)" },
client_id: { type: "string", description: "client ID" },
},
run: async (args) => ({ secret: await kc.getClientSecret(realmOf(args), String(args.client_id)) }),
},
{
name: "keycloak_add_protocol_mapper",
description:
"Add a protocol mapper to an OIDC client. Common types: oidc-usermodel-realm-role-mapper " +
"(realm roles), oidc-usermodel-attribute-mapper (user attributes), oidc-audience-mapper.",
input: {
realm: { type: "string", description: "realm name (defaults to the module's realm)" },
client_id: { type: "string", description: "client ID (e.g. 'grafana')" },
name: { type: "string", description: "mapper name (e.g. 'realm roles')" },
mapper_type: { type: "string", description: "protocol mapper type (e.g. 'oidc-usermodel-realm-role-mapper')" },
claim_name: { type: "string", description: "token claim name (e.g. 'realm_access.roles')" },
claim_type: { type: "string", description: "JSON type: String, long, int, boolean (default String)" },
multivalued: { type: "boolean", description: "whether the claim has multiple values (default false)" },
id_token: { type: "boolean", description: "include in ID token (default true)" },
access_token: { type: "boolean", description: "include in access token (default true)" },
userinfo: { type: "boolean", description: "include in userinfo response (default true)" },
},
run: async (args) => {
const realm = realmOf(args);
const clientId = String(args.client_id);
const name = String(args.name);
await kc.addProtocolMapper(realm, clientId, {
name,
protocolMapper: String(args.mapper_type),
config: {
"claim.name": String(args.claim_name),
"jsonType.label": args.claim_type ? String(args.claim_type) : "String",
"multivalued": String(args.multivalued === true),
"id.token.claim": String(args.id_token !== false),
"access.token.claim": String(args.access_token !== false),
"userinfo.token.claim": String(args.userinfo !== false),
},
});
return { added: { realm, clientId, mapper: name } };
},
},
// Groups
{
name: "keycloak_list_groups",
description: "List groups in a Keycloak realm.",
input: { realm: { type: "string", description: "realm name (defaults to the module's realm)" } },
run: async (args) => ({ groups: await kc.listGroups(realmOf(args)) }),
},
{
name: "keycloak_create_group",
description: "Create a group in a Keycloak realm.",
input: {
realm: { type: "string", description: "realm name (defaults to the module's realm)" },
name: { type: "string", description: "group name" },
},
run: async (args) => {
const realm = realmOf(args);
const name = String(args.name);
await kc.createGroup(realm, name);
await events.groupCreated(realm, name);
return { created: { realm, group: name } };
},
},
{
name: "keycloak_get_user_groups",
description: "List the groups a user belongs to in a Keycloak realm.",
input: {
realm: { type: "string", description: "realm name (defaults to the module's realm)" },
user_id: { type: "string", description: "user ID (UUID)" },
},
run: async (args) => ({ groups: await kc.getUserGroups(realmOf(args), String(args.user_id)) }),
},
{
name: "keycloak_add_user_to_group",
description: "Add a user to a group in a Keycloak realm.",
input: {
realm: { type: "string", description: "realm name (defaults to the module's realm)" },
user_id: { type: "string", description: "user ID (UUID)" },
group_id: { type: "string", description: "group ID (UUID)" },
},
run: async (args) => {
const realm = realmOf(args);
await kc.addUserToGroup(realm, String(args.user_id), String(args.group_id));
return { added: { realm, userId: String(args.user_id), groupId: String(args.group_id) } };
},
},
{
name: "keycloak_remove_user_from_group",
description: "Remove a user from a group in a Keycloak realm.",
input: {
realm: { type: "string", description: "realm name (defaults to the module's realm)" },
user_id: { type: "string", description: "user ID (UUID)" },
group_id: { type: "string", description: "group ID (UUID)" },
},
run: async (args) => {
const realm = realmOf(args);
await kc.removeUserFromGroup(realm, String(args.user_id), String(args.group_id));
return { removed: { realm, userId: String(args.user_id), groupId: String(args.group_id) } };
},
},
// Roles
{
name: "keycloak_get_user_roles",
description: "List the realm roles assigned to a user in a Keycloak realm.",
input: {
realm: { type: "string", description: "realm name (defaults to the module's realm)" },
user_id: { type: "string", description: "user ID (UUID)" },
},
run: async (args) => ({ roles: await kc.getUserRealmRoles(realmOf(args), String(args.user_id)) }),
},
{
name: "keycloak_create_role",
description: "Create a realm role in a Keycloak realm.",
input: {
realm: { type: "string", description: "realm name (defaults to the module's realm)" },
role_name: { type: "string", description: "role name" },
description: { type: "string", description: "role description" },
},
run: async (args) => {
const realm = realmOf(args);
const name = String(args.role_name);
await kc.createRealmRole(realm, { name, description: args.description ? String(args.description) : undefined });
await events.roleCreated(realm, name);
return { created: { realm, role: name } };
},
},
{
name: "keycloak_assign_user_role",
description: "Assign an existing realm role to a user. Create it first with keycloak_create_role if needed.",
input: {
realm: { type: "string", description: "realm name (defaults to the module's realm)" },
user_id: { type: "string", description: "user ID (UUID)" },
role_name: { type: "string", description: "role name to assign" },
},
run: async (args) => {
const realm = realmOf(args);
const userId = String(args.user_id);
const roleName = String(args.role_name);
// The mapping API needs the role's UUID, which only the "available" list carries; if the
// role is neither available nor already assigned it does not exist in this realm.
const available = await kc.getAvailableRealmRoles(realm, userId);
const role = available.find((r) => r.name === roleName);
if (!role) {
const assigned = await kc.getUserRealmRoles(realm, userId);
if (assigned.find((r) => r.name === roleName)) return { alreadyAssigned: { realm, userId, role: roleName } };
return { notFound: { realm, role: roleName } };
}
await kc.assignRealmRoles(realm, userId, [{ id: role.id, name: role.name }]);
return { assigned: { realm, userId, role: roleName } };
},
},
{
name: "keycloak_remove_user_role",
description: "Remove a realm role from a user in a Keycloak realm.",
input: {
realm: { type: "string", description: "realm name (defaults to the module's realm)" },
user_id: { type: "string", description: "user ID (UUID)" },
role_name: { type: "string", description: "role name to remove" },
},
run: async (args) => {
const realm = realmOf(args);
const userId = String(args.user_id);
const roleName = String(args.role_name);
const assigned = await kc.getUserRealmRoles(realm, userId);
const role = assigned.find((r) => r.name === roleName);
if (!role) return { notAssigned: { realm, userId, role: roleName } };
await kc.removeRealmRoles(realm, userId, [{ id: role.id, name: role.name }]);
return { removed: { realm, userId, role: roleName } };
},
},
];
}
// The tools exist only when the client can be configured; without an admin password, keycloak
// contributes none rather than failing the whole runtime.
registerModuleTools("keycloak", (env) => {
try {
return getKeycloakTools(KeycloakClient.fromEnv(env));
} catch {
return [];
}
});
modules/keycloak/tsconfig.json
-12
View File
@@ -1,12 +0,0 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "NodeNext",
"moduleResolution": "NodeNext",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"noEmit": true
},
"include": ["client.ts", "oidc.ts", "index.ts", "provisioner/index.ts", "tools/index.ts"]
}
modules/postgres/cmd/postgres-provider/harness.go
+188 -1
View File
@@ -8,6 +8,17 @@ package main
// Read the contributions the mesh delivered; bring each consumer's resource into being through the
// adapter, under the login and password the mesh minted; withdraw what the mesh no longer asks for.
// **A provider creates the credential the mesh minted, and seals nothing (novox/hq ADR 0048).**
//
// **A provider that keeps failing a consumer says so on the bus (novox/hq ADR 0224).** A consumer
// whose create, check or secret has failed without one success in between for FailingAfter is
// announced as `provisioner.failing` — naming the consumer, its machine and the class of error — and
// again every SayAgainEvery while it lasts; the first success after that is `provisioner.recovered`.
// The controller keeps the newest per provider and consumer and `status` names it. On 2026-10-05 the
// identity provider failed every consumer 31,000 times in a day and said so only in its journal
// (novox/hq issue 179).
//
// Carried, identical, by every Go provider until the Go SDK has the loop: postgres and keycloak.
// Each module's `harness_same_test.go` fails when its copy and the other's differ.
import (
"context"
@@ -54,15 +65,57 @@ type Harness struct {
HoldsTimeout time.Duration // 30s
Log func(format string, args ...any)
Now func() time.Time
// Announce publishes one of the provider's standing events; nil announces nothing. Node is the
// machine this provider runs on, said in each.
Announce func(event string, body map[string]any)
Node string
// FailingAfter is how long a consumer fails without a success before it is announced (5m);
// SayAgainEvery is how often it is announced again while it lasts (15m), so a controller that
// missed the first hears the next, and a standing nobody repeats can be told from one that holds.
FailingAfter time.Duration
SayAgainEvery time.Duration
verifiedAt time.Time
applied map[string]appliedEntry
lost map[string]brake
waiting map[string]int
failing map[string]failure
trouble map[string]*standing
cleared map[string]bool
lastWarning string
}
// standing is one consumer's unbroken run of failures: since when, how often, and the last error.
type standing struct {
node string
since time.Time
attempts int
class string
text string
saidAt time.Time
}
// The events a provider's standing is announced as (novox/hq ADR 0224). The controller derives the
// permission to emit them for every module that receives contributions; no manifest lists them.
const (
EventFailing = "provisioner.failing"
EventRecovered = "provisioner.recovered"
)
// The classes of error a standing is announced with: what a person reading `status` needs to know
// before reading the journal. An adapter may say better (Classifier).
const (
ClassCredentials = "credentials-rejected"
ClassUnreachable = "unreachable"
ClassSecret = "secret-unreadable"
ClassRefused = "refused"
)
// Classifier is an adapter that can say what class an error of its own is.
type Classifier interface {
Class(err error) string
}
type appliedEntry struct {
hash string
derived map[string]any
@@ -111,6 +164,12 @@ func (h *Harness) init() {
if h.Now == nil {
h.Now = time.Now
}
if h.FailingAfter == 0 {
h.FailingAfter = 5 * time.Minute
}
if h.SayAgainEvery == 0 {
h.SayAgainEvery = 15 * time.Minute
}
if h.Log == nil {
h.Log = func(format string, args ...any) { fmt.Fprintf(os.Stderr, format+"\n", args...) }
}
@@ -119,6 +178,8 @@ func (h *Harness) init() {
h.lost = map[string]brake{}
h.waiting = map[string]int{}
h.failing = map[string]failure{}
h.trouble = map[string]*standing{}
h.cleared = map[string]bool{}
}
}
@@ -230,6 +291,7 @@ func (h *Harness) Reconcile(ctx context.Context) {
"nothing has been provisioned for this consumer and nothing will be until somebody looks. "+
"Check who owns the file and who this process runs as (novox/hq issue 225)", g.As, n, g.Secret, err)
}
h.failed(g.As, g.Node, ClassSecret, fmt.Sprintf("secret not readable (%s): %v", g.Secret, err))
continue
}
delete(h.waiting, g.As)
@@ -253,7 +315,9 @@ func (h *Harness) Reconcile(ctx context.Context) {
if err != nil {
// Unable to ask is not evidence of loss. A backend that timed out will time out for
// the next consumer too, so the rest of this pass is not asked.
h.say("%s: could not check the backend, will ask again: %s", g.As, scrub(err, password))
text := scrub(err, password)
h.say("%s: could not check the backend, will ask again: %s", g.As, text)
h.failed(g.As, g.Node, h.classOf(err, text), text)
if timedOut {
verifying = false
}
@@ -261,6 +325,7 @@ func (h *Harness) Reconcile(ctx context.Context) {
}
if held {
delete(h.lost, g.As)
h.succeeded(g.As)
continue
}
reapplying = b.times + 1
@@ -283,6 +348,7 @@ func (h *Harness) Reconcile(ctx context.Context) {
if f.times == 1 || f.times%loudlyEvery == 0 {
h.say("%s: create failed, will retry: %s", g.As, text)
}
h.failed(g.As, g.Node, h.classOf(err, text), text)
if reapplying > 0 {
h.lost[g.As] = brake{times: reapplying - 1}
}
@@ -292,6 +358,7 @@ func (h *Harness) Reconcile(ctx context.Context) {
h.say("%s: created, after %d failed attempt(s)", g.As, f.times)
delete(h.failing, g.As)
}
h.succeeded(g.As)
h.applied[g.As] = appliedEntry{hash: hash, derived: p.Derived}
if reapplying == 0 {
delete(h.lost, g.As)
@@ -325,6 +392,126 @@ func (h *Harness) Reconcile(ctx context.Context) {
delete(h.failing, as)
}
}
// A consumer the mesh stopped asking for is no longer failed by anyone: said, so a standing
// the controller keeps for it is cleared rather than left naming a consumer that is gone.
for as := range h.trouble {
if !want[as] {
h.recovered(as, "withdrawn")
}
}
}
// failed counts one more failure in a consumer's unbroken run, and announces the run once it has
// lasted FailingAfter — then again every SayAgainEvery while it lasts.
func (h *Harness) failed(as, node, class, text string) {
now := h.Now()
s := h.trouble[as]
if s == nil {
s = &standing{since: now}
h.trouble[as] = s
}
s.node, s.class, s.text = node, class, text
s.attempts++
if now.Sub(s.since) < h.FailingAfter {
return
}
if !s.saidAt.IsZero() && now.Sub(s.saidAt) < h.SayAgainEvery {
return
}
first := s.saidAt.IsZero()
s.saidAt = now
if first {
h.say("%s: FAILING for %s (%d attempts, %s): %s. Announced as %s; `status` names it until it "+
"succeeds (novox/hq ADR 0224)", as, now.Sub(s.since).Round(time.Second), s.attempts, class, text, EventFailing)
}
h.announce(EventFailing, map[string]any{
"provider": h.Resource, "provider-node": h.Node,
"consumer": as, "node": node,
"class": class, "error": clip(text),
"since": s.since.UTC().Format(time.RFC3339), "attempts": s.attempts,
})
}
// succeeded ends a consumer's run of failures; one that was announced is announced recovered.
//
// **And the first success for a consumer since this process started is announced too**, failing or
// not: a provider that announced a failure and was restarted has forgotten it, and without this the
// controller would name the consumer failing for ever after it recovered unheard.
func (h *Harness) succeeded(as string) {
if h.trouble[as] == nil && !h.cleared[as] {
h.cleared[as] = true
h.announce(EventRecovered, map[string]any{
"provider": h.Resource, "provider-node": h.Node, "consumer": as, "why": "first-success",
})
return
}
h.cleared[as] = true
h.recovered(as, "")
}
func (h *Harness) recovered(as, why string) {
s := h.trouble[as]
if s == nil {
return
}
delete(h.trouble, as)
if s.saidAt.IsZero() {
return // never announced, so there is nothing to take back
}
if why == "" {
h.say("%s: recovered after %s and %d failed attempt(s)", as, h.Now().Sub(s.since).Round(time.Second), s.attempts)
}
body := map[string]any{
"provider": h.Resource, "provider-node": h.Node, "consumer": as, "node": s.node,
"since": s.since.UTC().Format(time.RFC3339), "attempts": s.attempts,
}
if why != "" {
body["why"] = why
}
h.announce(EventRecovered, body)
}
func (h *Harness) announce(event string, body map[string]any) {
if h.Announce != nil {
h.Announce(event, body)
}
}
// classOf is an error's class: the adapter's word when it has one, else read from the text.
func (h *Harness) classOf(err error, text string) string {
if c, ok := h.Adapter.(Classifier); ok {
if class := c.Class(err); class != "" {
return class
}
}
return ClassOf(text)
}
// ClassOf reads an error's class from its text — the words the backends the mesh runs use.
func ClassOf(text string) string {
t := strings.ToLower(text)
for _, w := range []string{"invalid_grant", "invalid user credentials", "password authentication failed",
"authentication failed", "unauthorized", " 401"} {
if strings.Contains(t, w) {
return ClassCredentials
}
}
for _, w := range []string{"connection refused", "no such host", "i/o timeout", "deadline exceeded",
"connection reset", "network is unreachable", "no route to host", "eof"} {
if strings.Contains(t, w) {
return ClassUnreachable
}
}
return ClassRefused
}
// clip keeps an announced error to what belongs in a status line.
func clip(text string) string {
const most = 300
if len(text) <= most {
return text
}
return text[:most] + "…"
}
// scrub is an error's text with the consumer's password removed, raw and URL-encoded.
modules/postgres/cmd/postgres-provider/harness_same_test.go
+31
View File
@@ -0,0 +1,31 @@
package main
// The provisioner loop is carried, identical, by every Go provider until the Go SDK has it
// (harness.go). Two copies drift the moment one is fixed and the other is not — and the one left
// behind is the provider that fails a consumer without saying so (novox/hq ADR 0224). This holds
// them to one text. Skipped where keycloak is not beside this module, as in a build of this one alone.
import (
"bytes"
"errors"
"io/fs"
"os"
"testing"
)
func TestTheHarnessIsTheSameAsKeycloaks(t *testing.T) {
theirs, err := os.ReadFile("../../../keycloak/cmd/keycloak-provider/harness.go")
if errors.Is(err, fs.ErrNotExist) {
t.Skip("keycloak is not beside this module")
}
if err != nil {
t.Fatal(err)
}
ours, err := os.ReadFile("harness.go")
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(ours, theirs) {
t.Fatal("harness.go differs from keycloak/cmd/keycloak-provider/harness.go: change both, identically")
}
}
modules/postgres/cmd/postgres-provider/harness_test.go
+11 -5
View File
@@ -11,10 +11,11 @@ import (
)
type recorder struct {
created []Provision
removed []string
held bool
failing error
created []Provision
removed []string
held bool
failing error
holdsErr error
}
func (r *recorder) Create(_ context.Context, p Provision) error {
@@ -30,7 +31,12 @@ func (r *recorder) Remove(_ context.Context, as string, _ map[string]any) error
return nil
}
func (r *recorder) Holds(context.Context, Provision) (bool, error) { return r.held, nil }
func (r *recorder) Holds(context.Context, Provision) (bool, error) {
if r.holdsErr != nil {
return false, r.holdsErr
}
return r.held, nil
}
type world struct {
t *testing.T
modules/postgres/cmd/postgres-provider/main.go
+8
View File
@@ -40,6 +40,14 @@ func main() {
Receives: receives,
Adapter: provisioner{pg: pg, announce: announce},
Log: func(format string, args ...any) { fmt.Fprintf(os.Stderr, format+"\n", args...) },
// A consumer failed for minutes is said on the bus, where the controller hears it and
// `status` names it (novox/hq ADR 0224).
Announce: func(event string, body map[string]any) {
if err := stdio.Emit(event, body); err != nil {
say("emit %s failed: %v", event, err)
}
},
Node: os.Getenv("MESH_NODE"),
}
go h.Run(context.Background())
}
modules/postgres/cmd/postgres-provider/standing_test.go
+187
View File
@@ -0,0 +1,187 @@
package main
// A provider that keeps failing a consumer says so on the bus (novox/hq ADR 0224): not on the first
// failure, which may be a restart; after FailingAfter of failures with no success between; again
// every SayAgainEvery while it lasts; and recovered on the first success, or when the consumer goes.
import (
"errors"
"os"
"strings"
"testing"
"time"
)
type announced struct {
event string
body map[string]any
}
func standingWorld(t *testing.T) (*world, *[]announced) {
w := newWorld(t)
var said []announced
w.h.Announce = func(e string, b map[string]any) {
// The first success since start is its own test's; every other test reads past it.
if b["why"] != "first-success" {
said = append(said, announced{e, b})
}
}
w.h.Node = "anchor"
return w, &said
}
// passes reconciles every five seconds for d, as Run would.
func (w *world) passes(d time.Duration) {
for end := w.now.Add(d); w.now.Before(end); w.now = w.now.Add(5 * time.Second) {
w.h.Reconcile(ctx)
}
}
func events(said []announced) string {
var out []string
for _, a := range said {
out = append(out, a.event)
}
return strings.Join(out, ",")
}
func TestAConsumerFailedForMinutesIsAnnouncedNamingItAndTheClass(t *testing.T) {
w, said := standingWorld(t)
w.a.failing = errors.New(`token request failed: 401 {"error":"invalid_grant","error_description":"Invalid user credentials"}`)
w.give(map[string]any{"as": "mesh_home_grafana", "node": "home-server"})
w.passes(4 * time.Minute)
if len(*said) != 0 {
t.Fatalf("announced before FailingAfter: %v", events(*said))
}
w.passes(2 * time.Minute)
if events(*said) != EventFailing {
t.Fatalf("want one %s, got %q", EventFailing, events(*said))
}
b := (*said)[0].body
if b["consumer"] != "mesh_home_grafana" || b["node"] != "home-server" || b["class"] != ClassCredentials ||
b["provider"] != "postgres-database" || b["provider-node"] != "anchor" || b["attempts"].(int) < 60 {
t.Fatalf("%v", b)
}
// Said again while it lasts, not every pass.
w.passes(14 * time.Minute)
if events(*said) != EventFailing {
t.Fatalf("repeated too soon: %q", events(*said))
}
w.passes(2 * time.Minute)
if events(*said) != EventFailing+","+EventFailing {
t.Fatalf("not repeated: %q", events(*said))
}
// The first success takes it back.
w.a.failing = nil
w.passes(5 * time.Second)
if events(*said) != EventFailing+","+EventFailing+","+EventRecovered {
t.Fatalf("no recovery: %q", events(*said))
}
if (*said)[2].body["consumer"] != "mesh_home_grafana" {
t.Fatal((*said)[2].body)
}
}
func TestOneSuccessBetweenFailuresStartsTheRunAgain(t *testing.T) {
w, said := standingWorld(t)
w.a.failing = errors.New("connection refused")
w.give(map[string]any{"as": "a"})
w.passes(4 * time.Minute)
w.a.failing = nil
w.passes(5 * time.Second)
w.give(map[string]any{"as": "a", "values": map[string]any{"name": "changed"}})
w.a.failing = errors.New("connection refused")
w.passes(4 * time.Minute)
if len(*said) != 0 {
t.Fatalf("two runs of four minutes are not one of eight: %q", events(*said))
}
}
// The check that failed for a day on 2026-10-05: clients already made, every minute's check refused
// at the token. A check that cannot be asked is a failure too.
func TestACheckThatKeepsFailingIsAFailureToo(t *testing.T) {
w, said := standingWorld(t)
w.give(map[string]any{"as": "a"})
w.h.Reconcile(ctx)
w.a.holdsErr = errors.New("401 invalid_grant")
w.passes(7 * time.Minute)
if events(*said) != EventFailing || (*said)[0].body["class"] != ClassCredentials {
t.Fatalf("%q %v", events(*said), *said)
}
w.a.holdsErr = nil
w.passes(time.Minute + 5*time.Second)
if events(*said) != EventFailing+","+EventRecovered {
t.Fatalf("%q", events(*said))
}
}
func TestAnUnreadableSecretIsAnnouncedAsSuch(t *testing.T) {
w, said := standingWorld(t)
w.give(map[string]any{"as": "a"})
os.Remove(w.dir + "/a.secret")
w.passes(6 * time.Minute)
if events(*said) != EventFailing || (*said)[0].body["class"] != ClassSecret {
t.Fatalf("%q %v", events(*said), *said)
}
}
func TestAWithdrawnConsumerIsNoLongerFailing(t *testing.T) {
w, said := standingWorld(t)
w.a.failing = errors.New("boom")
w.give(map[string]any{"as": "a"}, map[string]any{"as": "b"})
w.passes(6 * time.Minute)
if events(*said) != EventFailing+","+EventFailing {
t.Fatalf("%q", events(*said))
}
w.give(map[string]any{"as": "a"})
w.passes(5 * time.Second)
last := (*said)[len(*said)-1]
if last.event != EventRecovered || last.body["consumer"] != "b" || last.body["why"] != "withdrawn" {
t.Fatalf("%v", *said)
}
}
func TestAnErrorIsClassedByItsWords(t *testing.T) {
for text, want := range map[string]string{
`Keycloak token request failed: 401 {"error":"invalid_grant"}`: ClassCredentials,
`FATAL: password authentication failed for user "postgres"`: ClassCredentials,
`dial tcp 127.0.0.1:5432: connect: connection refused`: ClassUnreachable,
`context deadline exceeded`: ClassUnreachable,
`extension "nope" is not available`: ClassRefused,
} {
if got := ClassOf(text); got != want {
t.Errorf("%s: %s, want %s", text, got, want)
}
}
}
func TestAnAdapterThatClassesItsOwnErrorsIsBelieved(t *testing.T) {
w, said := standingWorld(t)
w.h.Adapter = classing{w.a}
w.a.failing = errors.New("anything")
w.give(map[string]any{"as": "a"})
w.passes(6 * time.Minute)
if (*said)[0].body["class"] != "its-own" {
t.Fatal((*said)[0].body)
}
}
type classing struct{ *recorder }
func (classing) Class(error) string { return "its-own" }
// A provider restarted after announcing a failure has forgotten it; its first success for each
// consumer is announced, so the controller clears what it kept rather than naming it for ever.
func TestTheFirstSuccessSinceStartIsAnnouncedOnce(t *testing.T) {
w := newWorld(t)
var said []announced
w.h.Announce = func(e string, b map[string]any) { said = append(said, announced{e, b}) }
w.give(map[string]any{"as": "a"})
w.passes(3 * time.Minute)
if events(said) != EventRecovered || said[0].body["why"] != "first-success" || said[0].body["consumer"] != "a" {
t.Fatalf("%v", said)
}
}