hq/02-DECISIONS/0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md

topic, status, date, deciders, reconstructed, extends

topic	status	date	deciders	reconstructed	extends
the mesh	accepted	2026-10-06	jochen	false	02-DECISIONS/0040-what-a-module-is.md

224. A provider that keeps failing a consumer is a problem the controller reports

Context

A provider's provisioner is the only thing in the mesh that knows whether a provision was made. The controller composes a grant, delivers the contributions file and the minted secret, and the node reports that it applied every resource. Whether the provider then made the consumer's database, client or bucket is known to the provisioner loop alone (ADR 0040, ADR 0048), and until now it said so only in its own journal.

On 2026-10-05 that cost a day. The identity provider's database was moved that morning. Its realm's administrator kept the password it had before, the module's minted one was inert, and its provisioner failed every consumer every five seconds — about 31,000 refused logins from shortly after midnight until it was repaired by hand that night. Every surface the mesh has said the mesh was well: the machines applied what they were sent, the modules were current, status printed its all-well sentence. The same fault had been found and repaired by hand four days earlier (issue 179).

Issue 145 already taught that "the mesh and the machines agree" is not "it works", and status says so under its all-well line. This is the narrower, checkable half of that gap: not whether a consumer can reach its provider, which only dialling answers, but whether the provider has told the mesh it cannot do its job.

Considered Options

1. Leave it to the journal, and to a person reading it. Rejected: that is what happened, twice.
2. The controller dials every provision. Rejected for now: it is issue 145's open question, it needs the controller to hold or borrow every consumer's credential, and it would still not say why a provider fails.
3. The provider reports its standing through the node's report. Rejected: the node-engine applies resources and knows nothing of what a module's code does after it starts; the provisioner runs in the node's tool runtime, which speaks to the bus, not to the host.
4. The provider announces, as an event, a consumer it keeps failing; the controller keeps the newest word and status names it. Chosen.

Decision

1. A provider announces a consumer it keeps failing. When a provisioner has failed one consumer — its create, its periodic check, or reading the consumer's minted secret — for five minutes without a single success in between, it emits provisioner.failing, naming the consumer, the consumer's machine, the provision, the class of error and the error's first words, since when, and how many attempts. It says it again every fifteen minutes while it lasts. The first success after that is provisioner.recovered; so is a consumer the mesh stopped asking for, and so is the first success for each consumer after the provider starts, because a provider restarted after announcing a failure has forgotten it.

• The classes are credentials-rejected, unreachable, secret-unreadable and refused (any other refusal), read from the error's words unless the provider's own code classes it better. They are what a person reading status needs before opening the journal.
• No secret travels. The error is the provider's own text with the consumer's password removed, as its log line already is.

The mechanism changed — 2026-10-06, by ADR 0230. What stands: a consumer the mesh stopped asking for is announced recovered. What moved: a consumer the provider still holds is no longer withdrawn on the first pass that misses it; it is said recovered when it is retired, after five passes and ten minutes, or a person's approval. A provider says what it retires, approves, re-enables and deletes on an event of its own, provisioner.retirement, permitted the same way as the two events here.

2. Every provider may say it, whatever its manifest lists. The permission to publish the two events is derived for every module that receives contributions; no manifest declares them. A provider whose manifest forgot them would otherwise have its announcement refused by the bus and fail as silently as before.

3. The controller follows both events from every module, keeps the newest failing word per provider module, its machine and the consumer, and removes it on recovery. Who said it is read from the subject the bus let the provider publish on, never from the body. A recovery that arrives while the store is away is held and delivered again, because it is said once. The controller's subscription names the two events with a wildcard for the emitter — the one pattern on its list — rather than a list of providers somebody would have to extend.

4. status names every consumer a provider still assigned where it ran says it keeps failing, and such a consumer breaks the all-well sentence. Its JSON carries them as failing; node show lists those whose provider or consumer is on that machine. A provider no longer assigned there has nothing running to fail anybody, and is not asked about. A standing not said again for thirty minutes is shown with how long it has been silent: the provider stopped saying anything, and its last word is all the mesh has.

5. Where a provider's failure has one known cause it can repair safely, it repairs it. The identity provider's administrator refusing the mesh's secret is the first: the module now checks the administrator's login on start, every five minutes and whenever its provisioner is refused, and repairs a refusal through the server's own bootstrap command — a temporary administrator, the real one's password set to the mesh's, the temporary one removed, nothing printed — then checks again and says what it did. A repair that fails is braked, from ten minutes doubling to six hours, and announced; while the administrator is refused the provisioner stops asking the server, so a lockout policy is not provoked, and its consumers are still announced failing. The mechanism is the module's (issue 179), the rule is this record's: detected automatically, repaired where safe, loud where not.

Consequences

• The loop is carried by every Go provider until the Go SDK has one. The provisioner loop is the SDK's (ADR 0039); the Go SDK has none yet, so the two Go providers carry identical copies and a test in each fails when they differ. A TypeScript provider announces nothing until the TypeScript SDK's loop does the same; until then its consumers fail as silently as before, and the next provider to be ported to Go closes that gap for itself.
• The controller's event consumer takes one message at a time, so a standing can wait behind a build being acted on for some minutes. Fifteen-minute repetition makes that harmless for a failure; a recovery is held, never dropped.
• The store keeps one row per provider module, its machine and consumer, through a numbered migration.
• The rollout order matters. The controller that derives the permission and follows the events first; the catalogue's providers second — an older controller refuses nothing that matters, but every announcement is then refused by the bus and logged by the provider.

How it is checked

Rule	Checked by
A consumer failed for five minutes without a success is announced, again every fifteen, and recovered on the first success	provider loop tests in mesh-catalog, postgres and keycloak (standing_test.go)
A failing periodic check and an unreadable secret count, not only a failing create	the same tests
A withdrawn consumer and the first success after a start are announced recovered	the same tests
The two Go providers carry the same loop	harness_same_test.go in each, comparing the files
Every module that receives contributions is granted the two events, and no other module is	mesh-controller inventory test over the derived declaration and the bus permissions
The controller may hear the two events from any module, and not every event	the same test over the controller's permissions
The emitter is read from the subject; a failing word is kept, a recovery cleared, and a recovery held while the store is away	mesh-controller link tests with a fake store
One row per provider, machine and consumer; recovered removes only its own	an inventory test against a live database, through the migration
status names it and is not all-well; the JSON carries it; node show names it on both machines; an unassigned provider's word is not a problem	mesh-controller command tests against a live database
The identity provider repairs a refused administrator, verifies, brakes a failed repair, never puts a secret in a command line, and stops asking the server while refused	keycloak module tests with a fake server and a fake container runtime; a live test against a throwaway server when asked for
Live: after rollout, status shows nothing failing on a healthy mesh; a provider made to fail a consumer for five minutes appears, and disappears on its next success	read through the console after rollout

References

• Issue 179 — the failure, twice, and the repair this record makes automatic.
• Issue 145 — the scope of the all-well sentence, which this narrows and does not close.
• ADR 0040, ADR 0048, ADR 0039, ADR 0042.
• The module protocol, provisioning, amended alongside.
• mesh-controller internal/link/standing.go, internal/inventory/standing.go and its migration, cmd/mesh-controller/standing.go; mesh-catalog modules/postgres and modules/keycloak.