02-DECISIONS/0145-a-module-checks-what-the-mesh-claims-is-reachable.md

@@ -0,0 +1,119 @@
+---
+topic: what runs on it
+status: accepted
+date: 2026-09-29
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md
+---
+
+# 145. A module checks what the mesh claims is reachable, and it checks itself
+
+## Context
+
+The mesh asserts three things are callable ([ADR 0144](0144-anything-on-a-machine-may-call-anything-on-it.md)):
+what runs on the same machine, another machine's service exposed to the private network, and another
+machine's service exposed publicly. It has never checked any of them.
+
+[Issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md):
+the first of the three was broken for eleven hours and the mesh answered *all heard from, every module
+current with its source* throughout. Every check it makes is about the relationship between the mesh and
+a machine — applied, current, containers running — and none about whether anything can reach anything.
+
+**A first answer was drafted and withdrawn.** [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md)
+put the check inside the host, verifying each grant from the consumer's network position. It was
+superseded because the difference it worked so hard to reproduce — whether a caller sat in a container —
+was the bug itself. What survives from it is the part that was right: a check run from the wrong place
+proves nothing, and the mesh's own reports are not evidence about the network.
+
+**The mesh already has the shape for this and it is a module.** A module can declare a container that
+runs on a cadence ([ADR 0053](0053-a-step-that-runs-on-a-schedule.md), and three modules already use
+`*/5 * * * *`), can be given the mesh's roster as a rendered fact — every machine's name, address and
+this node's own identity, the same mechanism the resolver and the operator's ssh configuration use — and
+can emit what it found on the bus. Nothing new is needed to build this except the module.
+
+**What it must not check is the trap.** The obvious probe target is ssh: present on every machine, never
+closed by design. Dialling it would have passed throughout the outage, because ssh is admitted
+unconditionally and the thing that broke was a service exposed to the private network. A checker whose
+probe is unconditionally open measures the one path that cannot fail, which is the failure this whole
+sequence keeps producing — a check that reads as verification and verifies nothing.
+
+## Considered Options
+
+1. **The host verifies each grant** ([ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md)).
+   Superseded. It needed the host to act from another network position, which is machinery that exists
+   only while local calls are filtered wrongly.
+2. **The control plane dials every node.** Rejected: it sits on one machine and reaches the others by a
+   path no ordinary caller uses. It would have passed throughout the outage.
+3. **Probe an existing service.** Rejected for the target problem above: the services guaranteed on every
+   machine are the ones that are never closed, so they cannot fail the way the mesh fails.
+4. **A module on every machine that serves its own probe and dials the others'.** Adopted.
+
+## Decision
+
+**A module runs on every machine, serves an endpoint of its own, and dials every other machine's.** The
+probe is the module's own endpoint, declared reachable over the private network — so the thing being
+dialled is admitted by exactly the rule that governs every other internally-exposed service, and fails
+when that rule is wrong. A second endpoint, declared public, does the same for the public path where a
+machine has one.
+
+**It checks the three cases the mesh claims, by name:**
+
+- its **own machine**, by dialling its own machine's address — the case that broke, and the only one that
+  distinguishes a caller on the machine from a caller in one of its containers;
+- **each other machine over the private network**;
+- **each machine's public path**, where one is recorded.
+
+**It resolves before it dials, and says which failed.** A name that does not resolve and a port that does
+not answer are different faults with different owners, and a checker that reports one sentence for both
+sends a reader to the wrong place.
+
+**It runs where the callers run.** The module's own code in its own container, on the cadence the mesh
+already has, from the same position as every other module on that machine. It is not the host and not the
+control plane, and that is the whole point.
+
+**It says what it found and nothing else.** It emits results; it repairs nothing, opens nothing and holds
+no credential beyond its own. A checker that fixes things is a second control plane.
+
+**One failure is not a fault.** A machine rebooting is ordinary. A path is reported broken after it has
+failed on consecutive runs, and the count travels with the result so a reader can tell "briefly away"
+from "never worked" — the one thing [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) got
+right and worth keeping.
+
+## Consequences
+
+- **The mesh gains the ability to be wrong out loud about the network.** Eleven hours becomes two runs.
+- **It is a module, so it is assigned, built, pushed and reported on like everything else** — no new host
+  capability, no new vocabulary, nothing in the control plane that has to know about checking.
+- **Its own endpoint is the instrument.** That is what makes it able to fail; it also means the checker
+  must be assigned to a machine before that machine can be checked, and a machine without it is
+  unchecked rather than healthy.
+- **It cannot check what it cannot be told.** The roster gives it machines; it does not give it every
+  module's endpoints, so this checks the paths the mesh claims and not every grant in the mesh. That is
+  the honest scope of a first one, and the difference is worth saying rather than growing quietly.
+- **What got harder:** one more module on every machine, and a module whose whole purpose is to fail
+  visibly when something else is wrong. Its own failures will be read as the mesh's, which is the cost of
+  an instrument.
+
+## How it is checked
+
+- **It catches the measured outage.** A bed closes the path from a container to a service exposed to the
+  private network on its own machine — issue 145's shape — and the checker reports its own machine
+  unreachable while every other path still reads reachable. This fails against a probe on a port that is
+  never closed, which is the wrong target this record exists to name.
+- **A machine rebooting is not a fault**: one failed run reports nothing, the count rises and falls.
+- **A name that does not resolve is reported as that**, not as a port that did not answer.
+- **It reports and does not act**: asserted by giving it a broken path and checking nothing on the machine
+  changed.
+- **A machine without the module reads unchecked**, never healthy — asserted on what the mesh says about
+  a machine it is not assigned to.
+
+## References
+
+- [ADR 0144](0144-anything-on-a-machine-may-call-anything-on-it.md) — the three things that must be callable
+- [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) — superseded; what survives is that the
+  position matters
+- [ADR 0053](0053-a-step-that-runs-on-a-schedule.md) — the cadence
+- [ADR 0138](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md) — internal and public,
+  which the probe endpoints declare
+- [issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)

02-DECISIONS/README.md

@@ -225,6 +225,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0141** — [The host delivers its own successor, and versions live side by side](0141-the-host-delivers-its-own-successor.md)
 - **0143** — [A consumer verifies the grant it is given](0143-a-consumer-verifies-the-grant-it-is-given.md)  *(superseded)*
 - **0144** — [Anything on a machine may call anything on it, and that is the whole of "local"](0144-anything-on-a-machine-may-call-anything-on-it.md)
+- **0145** — [A module checks what the mesh claims is reachable, and it checks itself](0145-a-module-checks-what-the-mesh-claims-is-reachable.md)
 
 ### How it is built
 

04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md

@@ -90,9 +90,17 @@ same machine, another machine's service exposed to the private network, and anot
 exposed publicly. The filter had the second and third and expressed the first as a list of addresses that
 no container could match.
 
-**The question this issue is actually about is still open.** Nothing in the mesh would have said a grant
-had stopped working, and nothing does now. That is not answered by a configuration fix, and it should not
-be answered by a mechanism adopted as a remedy for one.
+**And then the question this issue is actually about was answered on its own terms.**
+[ADR 0145](../../02-DECISIONS/0145-a-module-checks-what-the-mesh-claims-is-reachable.md): a module on
+every machine serves an endpoint of its own and dials every other machine's, from the position the
+callers are in. Its probe is its own endpoint declared reachable over the private network, so it is
+admitted by exactly the rule that governs every internally-exposed service and fails when that rule is
+wrong — where a probe on a service every machine has would have passed for all eleven hours, because the
+services every machine has are the ones never closed.
+
+Adopted on its merits rather than as the remedy for a configuration error, which is what 0143 was and
+why it went. The module is written and merged; it is not yet assigned, so every machine currently reads
+unchecked.
 
 ## Open questions