hq/01-RESEARCH/032-a-module-says-how-it-is-healthy/02-options.md

02 — Options

Each question of the overview, the options for it, and what the evidence says about each. The recommendation is 03.

A. Who runs the checks

A1 — The container runtime's own HEALTHCHECK, read by the node-engine

The image's check, or one the module sets, run by the runtime inside the container; the node-engine reads the health state it keeps.

• For: 19 images already ship one; it runs inside the container's namespace, so a check of localhost sees what the program sees; the runtime handles interval, timeout, retries and start period.
• Against: containers only — nothing for the 19 service-only modules, the one long-running process, or anything a module's own tool knows. Every check is an exec, and on the home server the 77 a minute already push every lifecycle event out of the runtime's history (01 §2). Two of 19 shipped checks were wrong in the mesh's configuration; an image check the module never stated is a check nobody owns. The runtime does nothing when a container turns unhealthy (it restarts only on exit), so the state is only useful to whoever reads it.

A2 — The node-engine runs every check itself

HTTP and TCP from the machine, exec through the runtime, a unit's state through the service manager, a tool through the node tools.

• For: one runner and one reader per machine, for every hosting form; it keeps what the runtime forgets (restarts across recreates, 01 §2); a check from outside the container tests the path a caller takes, which issue 145 says matters; HTTP and TCP cost no exec.
• Against: the engine grows a scheduler; a check that only makes sense inside the container (a CLI, a pid file) still needs an exec, which A1 does better.

A3 — The module's own tool answers "healthy"

The bundle exposes a health tool; something calls it.

• For: the only kind that catches a failure of function — the identity provider's refused admin (issue 179) — and the module knows what "working" means for it.
• Against: a bundle is hosted by the node tools, not by the service; a tool that answers proves the bundle is up, not the server. One tool in the catalogue is a health tool today. And a module that judges itself is the thing ADR 0227 rule 8 refuses for the core: the component being replaced is never the judge. For readiness of function it is the only option; for liveness it must not be the only one.

A4 — A mix, by kind (the shape the evidence points at)

The engine owns every check and every verdict. It runs HTTP, TCP and unit checks itself; it delegates an in-container command to the runtime by setting the container's HEALTHCHECK from the declaration (so the runtime's retries and start period do the timing) and reads the state; it asks a module's health tool through the node tools. Liveness — running and not restarting — it observes for every long-running resource with no declaration at all.

B. Where the result goes

Option	For	Against
B1 — A field of the report: per module, per long-running resource, a state (healthy, unhealthy, starting, unknown), since when, the failing streak and the restarts the engine counted	the report is how a machine states facts about itself; the gate already reads the last report; one place to read	a report is sent after an apply, not on a change — a container that goes bad at 03:00 waits for the next report
B2 — An event on each transition (module.<m>.<machine>.health changed)	immediate; the controller can keep the state	events are lost or replayed; an event alone is a sample, not a state
B3 — A condition, raised by the controller	the gate already fails a module on a new condition naming it on that machine (01 §5, point 3); the operator's conversation (ADR 0234) and the healers (ADR 0231) already act on conditions; the two-look rule and the content rule already apply	a condition is the controller's word, raised from something — it needs B1 or B2 under it
B4 — A seat verb the controller calls (health <module>)	always current	a pull per module per judging; a machine that is slow to answer reads as unhealthy

B1 + B2 + B3 together is how the core already works: the engine states its state in the report and emits the change; the controller keeps the last state per machine, raises the condition on the second look, and clears it on the look that no longer sees it. That makes the gate need no new rule: an unhealthy module raises a condition naming it, and point 3 already holds the gate on it.

C. Liveness and readiness

• Liveness — it runs: a container running and not restarted within a window; a unit active and not failed, NRestarts not climbing; a process up. Observable for every long-running resource with no declaration. It alone catches the worst incident of the window (the crash loop). It needs a start period and a settle rule, or it reads issue 058's churn-that-stops as a crash loop.
• Readiness — it serves: the declared check passes. Only the module can say what serving means.
• Function — it does its job: the identity provider's admin logs in. A module's own tool, and only for what no endpoint shows.

Options: judge liveness only (cheap, no declarations, misses issue 145 and 179); judge readiness only (misses nothing that readiness sees, but every module must declare before anything is judged); judge liveness everywhere at once and readiness where declared, making the declaration required over a migration. The last keeps the gate meaningful from the first day.

What the mesh does with each is a separate choice. Restarting a container that is unhealthy is what an orchestrator's liveness probe does; the runtime here does not, and a restart hides the failure the gate is meant to see. Options: the healers restart what stays unhealthy (ADR 0231's shape: act on what observation raised, say whether it worked), or nothing restarts on health and the condition reaches a person. The evidence has no case where a restart would have fixed anything — the crash loop was restarting already.

D. Dependency-aware health

A module requiring a provision fails when its provider fails. With 12 consumers of the database provision and 36 of a route, a provider down would be a dozen conditions said once each, and a dozen gates failed for something none of them did.

Option	What it costs
D1 — Ignore it	twelve conditions for one fault; the operator learns which one matters by reading all of them; the gate blames the consumers
D2 — A consumer's check names the provision it exercises; when that provider is unhealthy on the record, the consumer's finding is held under the provider's — said as waiting on the provider, not raised on its own — and its gate waits rather than fails	one condition, at the provider; needs the controller to know which provider answers which consumer — it does: it composes the grants (issue 181 is about recording it)
D3 — Order the judging: providers first, consumers only when their providers are healthy	simple, but a consumer broken on its own is not said while a provider is down, which is the moment it is most needed

D2 is how issue 281 already treats a machine-level condition: what is about the machine is the machine's, and is never pinned on the module the gate is kept on.

E. The manifest field

Research may sketch a shape; a design doc may only name the field. Where the declaration lives:

• E1 — On the resource (health on a container, a process, a service): the check belongs to the thing it checks; a module with three containers states three. Matches how restart-on and witness already sit on the resource.
• E2 — One per module (a top-level health): one verdict, but a module of eleven containers (the mail module) cannot say which one is wrong.

The sketch for E1 — a field named health, holding:

Part	Meaning	Default
kind	runtime (the image's own check, adopted as is), http (an endpoint from listens, a path, the status expected), tcp (an endpoint from listens), exec (a command in the container), unit (the unit's own readiness), tool (one of the module's tools answering)	—
every	interval	30 s; not under 10 s
timeout	how long one look may take	5 s; under every
after	looks failing in a row before it is unhealthy	3; not under 2 (issue 277)
grace	after a start, how long failure does not count	60 s
needs	the provision whose provider it exercises, for D2	none

An HTTP or TCP check names an endpoint by its listens name, never a port or an address, so a check follows the machine's ports the way the endpoint does. A runtime kind is an explicit adoption of the image's check, so a module that ships one says it does, and module check can prove it on a lab.

F. Migration

• F1 — Required at once — every long-running resource declares, or module check refuses. 68 modules to touch before the next merge; nothing is judged until all are done.
• F2 — Liveness at once, declaration required by a date — every long-running resource is judged by liveness from the first build; module check warns, then refuses after a stated date; a catalogue-wide test counts the modules still without one, and the count only goes down.
• F3 — Optional for ever — modules without one are judged by liveness and tools served. The 38 of 45 container modules with nothing today would stay at liveness.

What a module without a declaration is judged by in F2 and F3: liveness of every long-running resource plus today's five points (01 §5). A bundle-only module (50) and a files-only module (7) need no declaration: they run nothing long-lived of their own, and the node tools serving their tools is their liveness.