Research 017: a mesh that heals itself #115

2026-09-25T22:54:37Z

jschoubben commented

2026-09-25 22:54:37 +00:00

The operator's wish, written as intended behaviour. This is a target to design toward. Nothing here is decided.

Principles

Every loop compares what should be against what is, never against what it did.
Healing is the ordinary path run again, never a second path.
A repair never destroys.
Nothing fails silently: a condition the mesh can't repair is named, dated and visible until observation clears it.
What the mesh can't fix goes to an agent (human or not) as work.
The mesh checks correctness, not only liveness.

The loop, everywhere: know, observe, repair (within a budget), raise a condition, clear it on observation. status becomes the list of open conditions, so an empty list means the mesh is right, not just up.

On NATS (ADR 0106)

Heartbeats and observations are published on subjects.
Observed state and conditions live in JetStream key-value buckets. The provisioner's applied record and a rotation's step move there too.
Server advisories become observations.
Redelivery with delay handles retries.
Work for an agent is a task on a subject.

Now, pragmatically. The test is whether a change survives the bus move. Loop and adapter changes do; anything built on AMQP doesn't. One step is already done: issue 120's harness check, in mesh-sdk #7 and mesh-catalog #84.

In order of silent failures removed, the next steps are:

holds for the other providers.
Removing consumers that went away while their provisioner was down.
Surfacing what owners already know as conditions in status.

Owed before graduation:

the loop inventory;
a classification of the 46 silent-failure issues (of 116).

records.py, index.py and cycle.py pass. The leak scan is clean.

**The operator's wish, written as intended behaviour.** This is a target to design toward. Nothing here is decided. **Principles** 1. Every loop compares what should be against what is, never against what it did. 2. Healing is the ordinary path run again, never a second path. 3. A repair never destroys. 4. Nothing fails silently: a condition the mesh can't repair is named, dated and visible until observation clears it. 5. What the mesh can't fix goes to an agent (human or not) as work. 6. The mesh checks correctness, not only liveness. **The loop, everywhere:** know, observe, repair (within a budget), raise a condition, clear it on observation. `status` becomes the list of open conditions, so an empty list means the mesh is *right*, not just up. **On NATS (ADR 0106)** - Heartbeats and observations are published on subjects. - Observed state and conditions live in JetStream key-value buckets. The provisioner's applied record and a rotation's step move there too. - Server advisories become observations. - Redelivery with delay handles retries. - Work for an agent is a task on a subject. **Now, pragmatically.** The test is whether a change survives the bus move. Loop and adapter changes do; anything built on AMQP doesn't. One step is already done: issue 120's harness check, in mesh-sdk #7 and mesh-catalog #84. In order of silent failures removed, the next steps are: 1. `holds` for the other providers. 2. Removing consumers that went away while their provisioner was down. 3. Surfacing what owners already know as conditions in `status`. **Owed before graduation:** - the loop inventory; - a classification of the 46 silent-failure issues (of 116). `records.py`, `index.py` and `cycle.py` pass. The leak scan is clean.

jschoubben added 1 commit 2026-09-25 22:54:37 +00:00

Research 017: a mesh that heals itself b5512bed32

The operator's wish written as intended behaviour for the NATS bus:
every loop compares against what is, repairs by the ordinary path, never
destroys, and raises a condition for what it cannot fix. What is done
before NATS is limited to what survives the move.

jschoubben merged commit 1272a97fd6 into main

2026-09-26 12:14:24 +00:00

jschoubben deleted branch research/017-a-mesh-that-heals-itself

2026-09-26 12:14:24 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/hq#115