diff --git a/02-DECISIONS/0145-a-module-checks-what-the-mesh-claims-is-reachable.md b/02-DECISIONS/0145-a-module-checks-what-the-mesh-claims-is-reachable.md new file mode 100644 index 0000000..bb4e51b --- /dev/null +++ b/02-DECISIONS/0145-a-module-checks-what-the-mesh-claims-is-reachable.md @@ -0,0 +1,119 @@ +--- +topic: what runs on it +status: accepted +date: 2026-09-29 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md +--- + +# 145. A module checks what the mesh claims is reachable, and it checks itself + +## Context + +The mesh asserts three things are callable ([ADR 0144](0144-anything-on-a-machine-may-call-anything-on-it.md)): +what runs on the same machine, another machine's service exposed to the private network, and another +machine's service exposed publicly. It has never checked any of them. + +[Issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md): +the first of the three was broken for eleven hours and the mesh answered *all heard from, every module +current with its source* throughout. Every check it makes is about the relationship between the mesh and +a machine — applied, current, containers running — and none about whether anything can reach anything. + +**A first answer was drafted and withdrawn.** [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) +put the check inside the host, verifying each grant from the consumer's network position. It was +superseded because the difference it worked so hard to reproduce — whether a caller sat in a container — +was the bug itself. What survives from it is the part that was right: a check run from the wrong place +proves nothing, and the mesh's own reports are not evidence about the network. + +**The mesh already has the shape for this and it is a module.** A module can declare a container that +runs on a cadence ([ADR 0053](0053-a-step-that-runs-on-a-schedule.md), and three modules already use +`*/5 * * * *`), can be given the mesh's roster as a rendered fact — every machine's name, address and +this node's own identity, the same mechanism the resolver and the operator's ssh configuration use — and +can emit what it found on the bus. Nothing new is needed to build this except the module. + +**What it must not check is the trap.** The obvious probe target is ssh: present on every machine, never +closed by design. Dialling it would have passed throughout the outage, because ssh is admitted +unconditionally and the thing that broke was a service exposed to the private network. A checker whose +probe is unconditionally open measures the one path that cannot fail, which is the failure this whole +sequence keeps producing — a check that reads as verification and verifies nothing. + +## Considered Options + +1. **The host verifies each grant** ([ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md)). + Superseded. It needed the host to act from another network position, which is machinery that exists + only while local calls are filtered wrongly. +2. **The control plane dials every node.** Rejected: it sits on one machine and reaches the others by a + path no ordinary caller uses. It would have passed throughout the outage. +3. **Probe an existing service.** Rejected for the target problem above: the services guaranteed on every + machine are the ones that are never closed, so they cannot fail the way the mesh fails. +4. **A module on every machine that serves its own probe and dials the others'.** Adopted. + +## Decision + +**A module runs on every machine, serves an endpoint of its own, and dials every other machine's.** The +probe is the module's own endpoint, declared reachable over the private network — so the thing being +dialled is admitted by exactly the rule that governs every other internally-exposed service, and fails +when that rule is wrong. A second endpoint, declared public, does the same for the public path where a +machine has one. + +**It checks the three cases the mesh claims, by name:** + +- its **own machine**, by dialling its own machine's address — the case that broke, and the only one that + distinguishes a caller on the machine from a caller in one of its containers; +- **each other machine over the private network**; +- **each machine's public path**, where one is recorded. + +**It resolves before it dials, and says which failed.** A name that does not resolve and a port that does +not answer are different faults with different owners, and a checker that reports one sentence for both +sends a reader to the wrong place. + +**It runs where the callers run.** The module's own code in its own container, on the cadence the mesh +already has, from the same position as every other module on that machine. It is not the host and not the +control plane, and that is the whole point. + +**It says what it found and nothing else.** It emits results; it repairs nothing, opens nothing and holds +no credential beyond its own. A checker that fixes things is a second control plane. + +**One failure is not a fault.** A machine rebooting is ordinary. A path is reported broken after it has +failed on consecutive runs, and the count travels with the result so a reader can tell "briefly away" +from "never worked" — the one thing [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) got +right and worth keeping. + +## Consequences + +- **The mesh gains the ability to be wrong out loud about the network.** Eleven hours becomes two runs. +- **It is a module, so it is assigned, built, pushed and reported on like everything else** — no new host + capability, no new vocabulary, nothing in the control plane that has to know about checking. +- **Its own endpoint is the instrument.** That is what makes it able to fail; it also means the checker + must be assigned to a machine before that machine can be checked, and a machine without it is + unchecked rather than healthy. +- **It cannot check what it cannot be told.** The roster gives it machines; it does not give it every + module's endpoints, so this checks the paths the mesh claims and not every grant in the mesh. That is + the honest scope of a first one, and the difference is worth saying rather than growing quietly. +- **What got harder:** one more module on every machine, and a module whose whole purpose is to fail + visibly when something else is wrong. Its own failures will be read as the mesh's, which is the cost of + an instrument. + +## How it is checked + +- **It catches the measured outage.** A bed closes the path from a container to a service exposed to the + private network on its own machine — issue 145's shape — and the checker reports its own machine + unreachable while every other path still reads reachable. This fails against a probe on a port that is + never closed, which is the wrong target this record exists to name. +- **A machine rebooting is not a fault**: one failed run reports nothing, the count rises and falls. +- **A name that does not resolve is reported as that**, not as a port that did not answer. +- **It reports and does not act**: asserted by giving it a broken path and checking nothing on the machine + changed. +- **A machine without the module reads unchecked**, never healthy — asserted on what the mesh says about + a machine it is not assigned to. + +## References + +- [ADR 0144](0144-anything-on-a-machine-may-call-anything-on-it.md) — the three things that must be callable +- [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) — superseded; what survives is that the + position matters +- [ADR 0053](0053-a-step-that-runs-on-a-schedule.md) — the cadence +- [ADR 0138](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md) — internal and public, + which the probe endpoints declare +- [issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md) diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index f732795..3ce3459 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -225,6 +225,7 @@ python3 00-META/checks/index.py fail if stale - **0141** — [The host delivers its own successor, and versions live side by side](0141-the-host-delivers-its-own-successor.md) - **0143** — [A consumer verifies the grant it is given](0143-a-consumer-verifies-the-grant-it-is-given.md) *(superseded)* - **0144** — [Anything on a machine may call anything on it, and that is the whole of "local"](0144-anything-on-a-machine-may-call-anything-on-it.md) +- **0145** — [A module checks what the mesh claims is reachable, and it checks itself](0145-a-module-checks-what-the-mesh-claims-is-reachable.md) ### How it is built diff --git a/04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md b/04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md index 2a1e122..ee8ac6b 100644 --- a/04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md +++ b/04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md @@ -90,9 +90,17 @@ same machine, another machine's service exposed to the private network, and anot exposed publicly. The filter had the second and third and expressed the first as a list of addresses that no container could match. -**The question this issue is actually about is still open.** Nothing in the mesh would have said a grant -had stopped working, and nothing does now. That is not answered by a configuration fix, and it should not -be answered by a mechanism adopted as a remedy for one. +**And then the question this issue is actually about was answered on its own terms.** +[ADR 0145](../../02-DECISIONS/0145-a-module-checks-what-the-mesh-claims-is-reachable.md): a module on +every machine serves an endpoint of its own and dials every other machine's, from the position the +callers are in. Its probe is its own endpoint declared reachable over the private network, so it is +admitted by exactly the rule that governs every internally-exposed service and fails when that rule is +wrong — where a probe on a service every machine has would have passed for all eleven hours, because the +services every machine has are the ones never closed. + +Adopted on its merits rather than as the remedy for a configuration error, which is what 0143 was and +why it went. The module is written and merged; it is not yet assigned, so every machine currently reads +unchecked. ## Open questions