Research 017: a mesh that heals itself

The operator's wish written as intended behaviour for the NATS bus: every loop compares against what is, repairs by the ordinary path, never destroys, and raises a condition for what it cannot fix. What is done before NATS is limited to what survives the move.
2026-09-26 00:54:27 +02:00
parent 56669ee23b
commit b5512bed32
3 changed files with 178 additions and 0 deletions
@@ -0,0 +1,47 @@
+---
+status: active
+initiated: 2026-09-26
+touches:
+  - 00-META/mission.md
+  - 02-DECISIONS/0106-the-bus-is-nats.md
+  - 02-DECISIONS/0010-delivery.md
+  - 02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md
+  - 03-DESIGN/01-to-be/06-the-controller.md
+  - 03-DESIGN/01-to-be/09-the-node-lifecycle.md
+  - 03-DESIGN/00-as-is/09-interfaces-and-observability.md
+---
+
+# 017 — A mesh that heals itself
+
+**What.** The behaviour the operator wants: a mesh that runs itself. It notices what is wrong,
+repairs what it can, and hands what it cannot repair to someone who can, with the reason. This effort
+writes that wish down as intended behaviour, designed for the bus the mesh is moving to
+([ADR 0106](../../02-DECISIONS/0106-the-bus-is-nats.md): NATS). It also records what can be done
+pragmatically before that move.
+
+**Why.** The mission is *a mesh that controls itself* ([mission](../../00-META/mission.md)). The
+mesh can tell whether it is up. It cannot tell whether it is right. The as-is page on observability says so
+([as-is 09](../../03-DESIGN/00-as-is/09-interfaces-and-observability.md)). To-be 06 names an
+`observability` context in the controller and leaves its store undecided. Nothing routes a condition
+the mesh cannot fix to anyone. The cost is measurable: **46 of the 116 issue reports in this
+repository describe a failure that was silent.** A mesh that heals itself is, first, a mesh that stops
+failing silently.
+
+**What it touches.** The controller's observability context, the node lifecycle's liveness, delivery
+([ADR 0010](../../02-DECISIONS/0010-delivery.md), [ADR 0083](../../02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md)),
+the provisioner harness, and rotation, which is proposed alongside to-be 27 as ADR 0114.
+
+**Documents.**
+
+- [01 — The intended behaviour](01-the-intended-behaviour.md): the wish, as principles and as how
+  the mesh behaves once the bus is NATS.
+- [02 — Now, pragmatically](02-now-pragmatically.md): what is done before NATS, why it does not
+  build anything the move would throw away, and what has been done already.
+
+**Next.** Two measurements this effort owes before it can graduate:
+
+1. **Every loop in the mesh**: what it converges, and whether it compares against observed state or
+   against its own memory. Issue 120 found the provisioner harness trusting memory. The same pattern is
+   expected elsewhere.
+2. **The 46 silent failures, classified**: a missing observation, a loop trusting memory, or a missing
+   escalation. That shows which mechanism removes the most of them.
@@ -0,0 +1,85 @@
+# 01 — The intended behaviour
+
+The operator's wish, written as behaviour: what a person or an agent sees the mesh doing. This is a
+target to design toward, not a design. Every part of it is to be decided through a record before it
+is built.
+
+## Principles
+
+**1. Every loop compares what should be with what is, never with what it did.** Desired state is the
+mesh's: assignments, requirements, seats. Observed state is read from the thing itself: the container,
+the backend, the node. A loop that compares against its own memory of what it applied is blind to
+anything that changed behind its back. That is issue 120, and it is the pattern this whole effort is
+written against.
+
+**2. Healing is the ordinary path run again, never a second path.** Repairing a lost login is
+provisioning it. Repairing a dead container is converging the node. Repairing a stale declaration is
+delivering it. A repair that needs its own code is a second way of doing something, which is exactly
+what the mesh is removing everywhere else.
+
+**3. A repair never destroys.** Healing may recreate, re-provision, re-deliver and restart. It may
+never delete a consumer's data, retire a credential someone still uses, or pick a winner between two
+contradictory states. Where the only repair is destructive, it is escalated.
+
+**4. Nothing fails silently.** Every condition the mesh cannot repair within its budget becomes
+visible. It is named, it says since when, why, and who can resolve it. It is visible until it is
+resolved, and resolved by observation, not by someone clicking it away.
+
+**5. What the mesh cannot fix goes to an agent.** Per the mission, an agent may be human or not. A
+condition that needs judgement is handed to one, as work, with what the mesh knows. It is not handed
+over as a notification that someone may or may not read.
+
+**6. Correctness, not only liveness.** A running process that authenticates with a dead credential,
+serves an old version, or routes nowhere is not healthy. What a provision's contract promises is what
+is checked: the credential authenticates, the route answers, the version is the declared one.
+
+## The loop, everywhere
+
+Every part of the mesh that owns something runs the same loop:
+
+1. **know** what should be true: from assignments, requirements and seats;
+2. **observe** what is true: from the thing itself, on its own cadence;
+3. **repair** the difference by running the ordinary path again, within a budget of attempts and
+   time;
+4. **raise** a *condition* when the budget is spent or the only repair is destructive;
+5. **clear** the condition when observation shows it resolved.
+
+A **condition** is a durable fact about something the mesh owns, such as a node, an assignment, a
+provision, a seat or a rotation: what is wrong, since when, the evidence, what was tried, and who can
+resolve it. Conditions are the one thing a person or an agent looks at to know whether the mesh is
+right. `status` is the list of open conditions. When it is empty, the mesh is right, not just up.
+
+## On NATS
+
+[ADR 0106](../../02-DECISIONS/0106-the-bus-is-nats.md) moves the bus to NATS, and NATS makes most of
+this cheaper, because observation becomes something every component publishes rather than something
+a central process polls.
+
+| the wish needs | on NATS |
+|---|---|
+| every component says it is alive | a heartbeat on a subject per node and assignment; silence past its interval is a condition, and nobody polls |
+| every component says what it observed | observations published on subjects (`mesh.observed.<node>.<assignment>`, for instance), consumed by whoever owns the comparison |
+| the last known state survives restarts | a JetStream key-value bucket of observed state per owner; the provisioner's "what I applied" and a rotation's step live there, not in memory |
+| conditions are durable and watchable | conditions as entries in a key-value bucket, watched by anyone who cares: a surface, an agent, the controller |
+| the bus itself is observed | the server's advisories (a consumer exceeding its deliveries, a slow consumer, a client disconnecting) and its monitoring endpoint become observations like any other |
+| a repair is retried, not lost | JetStream redelivery with delay, which is the same mechanism [ADR 0083](../../02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md)'s guarantee moves to |
+| work handed to an agent | a condition that needs judgement published as a task on a subject an agent's queue group consumes |
+
+**Who compares.** Each owner compares its own: the host for its node's containers and files, a
+provisioner for its backend, the vault for rotations, the controller for delivery and seats. The
+controller's observability context does not repair anything. It holds conditions, their history,
+and the view across the mesh. It notices what no owner can see about itself: an owner gone silent.
+
+## What stays human
+
+Some repairs need the operator's key, and the mesh says so rather than pretending otherwise:
+re-raising the vault or the broker, and recovering a node's identity. These are conditions too, with
+the procedure named, and they are the only ones that can never clear themselves.
+
+## Open
+
+- The budgets: how many attempts, over how long, per kind of repair.
+- How a condition that needs judgement reaches an agent, and how the agent's action is recorded.
+- Where the observability context stores history (to-be 06 left it open; volume argues against the
+  relational store).
+- Which correctness probe each provision's contract offers, and how often it runs.
@@ -0,0 +1,46 @@
+# 02 — Now, pragmatically
+
+The intended behaviour lands on NATS. The bus moves after the migration's core
+([ADR 0106](../../02-DECISIONS/0106-the-bus-is-nats.md)). Until then, work toward it is chosen by one
+test:
+
+**Does it survive the move?** A change to what a loop compares against, or to what an adapter can
+tell about its backend, survives, because it is independent of the bus. A new AMQP queue for health
+reports, a poller written against the broker's management API, or a condition store built on the
+current broker does not survive, and is not built.
+
+## Done
+
+**The provisioner asks the backend, not memory** (issue 120). The SDK's harness gained an optional
+`holds` on the adapter, asked for every applied consumer every minute. A consumer the backend no
+longer holds is provisioned again. Being unable to ask is not treated as loss. The cache module
+implements it first, because its server keeps its users in memory and forgets them all on a restart.
+That was verified against a real server: a restart erases every consumer's user, and `holds` answers
+correctly for absent, present, wrong-password, disabled and deleted.
+Changes: mesh-sdk PR #7 (0.1.1) and mesh-catalog PR #84.
+
+This is principle 1 applied to one loop. It survives the move unchanged. On NATS, the harness's
+record of what it applied moves from memory into a key-value bucket, and `holds` stays as it is.
+
+## Next, in order of silent failures removed
+
+1. **`holds` for the other credential providers.** Each backend can answer whether a login exists
+   with the mesh's password without changing anything. Where a backend cannot check a password without
+   logging in, logging in is the check.
+2. **The harness's other blind spot.** A consumer that goes away while its provisioner is down is never
+   removed. The fix is the same principle in reverse: list what the backend holds, and compare it with
+   what the mesh asks for. Removal stays subject to ADR 0114's rule that it never follows from a login
+   changing.
+3. **`status` reports what owners already know.** The host knows which containers it recreated and
+   why. A rotation knows who it waits on. Delivery knows what is outstanding. Surfacing those as
+   conditions in the existing `status` needs no new transport. It is the shape the NATS condition store
+   will hold.
+4. **The loop inventory and the classification** in [00](00-overview.md). They decide what comes after
+   these three.
+
+## Not now
+
+- Heartbeats, observation subjects, key-value state, advisories: all NATS, all after the move.
+- Handing conditions to agents: designed with NATS, where a task on a subject is native.
+- Choosing the observability store: decided when there is something to store, which is after the
+  move.