76 lines
4.2 KiB
Markdown
76 lines
4.2 KiB
Markdown
---
|
|
topic: the mesh
|
|
status: accepted
|
|
date: 2026-10-02
|
|
deciders: jochen
|
|
reconstructed: false
|
|
extends: 02-DECISIONS/0005-the-node-host.md
|
|
---
|
|
|
|
# 184. A service the mesh asked to run is still running a moment later
|
|
|
|
## Context
|
|
|
|
The host already refuses to take a service manager's word for it. Three places in one function read
|
|
a unit back after acting on it, each with a comment saying why: *a service manager accepting a
|
|
command says the transaction was accepted, not that the unit is running — one that starts and
|
|
immediately dies satisfies it.* The intent was right and the implementation did not reach it.
|
|
|
|
On 2026-10-02 the mesh composed a fail2ban jail whose pattern the daemon refused. The host wrote the
|
|
files, restarted the service, read the unit back and reported *restarted*. The unit was `active` at
|
|
that instant and `failed` 221 milliseconds later, which the unit's own record states. Both public
|
|
machines then kept no bans at all — every jail, not the one at fault — and nothing in the mesh said
|
|
so. The fault was found by calling a tool that needed the daemon, not by the mesh noticing.
|
|
|
|
The read-back races the failure. A service manager returns when it has started the process; a daemon
|
|
that reads its configuration, refuses it and exits does so a fraction of a second afterwards. One
|
|
look sees `activating` or `active` whatever the process is about to do, and *the host reports success
|
|
for a machine that is already wrong* — the one shape of failure this host exists to refuse
|
|
([ADR 0005](0005-the-node-host.md)).
|
|
|
|
A command the module declares — *test the configuration before restarting* — was considered and
|
|
rejected. The link carries no actions ([ADR 0005](0005-the-node-host.md)), and a verification
|
|
command is a command: a declaration that carried one would be remote execution over the bus,
|
|
arriving as root on every machine, which is a far larger door than the fault it closes. The host
|
|
does not need one. It already knows what it asked for.
|
|
|
|
## Decision
|
|
|
|
**1. A unit the host has just asked to run is read twice**, with a pause between the reads long
|
|
enough for a daemon that refuses its configuration to have exited. Not running at the second look is
|
|
a failure of that resource, named with the unit and the state it is in — the same failure the single
|
|
read was always meant to catch.
|
|
|
|
**2. It is never a wait for a unit to come up.** A unit still starting reads as running at both
|
|
looks and is accepted, exactly as before. What the second look catches is a unit that *was* running
|
|
and is not any more. A service asked to be stopped is not waited on at all.
|
|
|
|
**3. The host tests nothing and runs nothing of a module's.** The second look is the host checking
|
|
the state it was told to establish, which is its whole job; the declaration gains no vocabulary, and
|
|
no command reaches a machine that did not already come from a built artifact.
|
|
|
|
## Consequences
|
|
|
|
- Every apply that starts, restarts or reloads a service spends a moment confirming it. The cost is
|
|
bounded by the number of services that changed in that apply, which is usually none.
|
|
- A module whose configuration the mesh composes — the packet filter, the intrusion prevention, the
|
|
resolver — now fails its apply when the composition is bad, instead of reporting success onto a
|
|
dead daemon. `status` names the machine, which is how the operator finds out.
|
|
- It does not prevent the bad composition. [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)'s
|
|
manifest check is what refuses the one that caused this, at merge time; this record is what makes
|
|
the *next* one visible within a minute rather than invisible until something asks the daemon a
|
|
question.
|
|
|
|
## How this is checked
|
|
|
|
| Rule | Checked by |
|
|
|---|---|
|
|
| A unit that is running at the first look and dead at the second fails the apply, naming the unit and its state | a host test over a service manager that answers as systemd does |
|
|
| A unit still starting is accepted at both looks | a host test |
|
|
| A service asked to be stopped is not waited on | a host test |
|
|
|
|
## References
|
|
|
|
- [ADR 0005](0005-the-node-host.md), [ADR 0179](0179-the-intrusion-seat-serves-its-verbs-and-every-door-declares-its-jail.md)
|
|
- [Design 05 — The node host](../03-DESIGN/01-to-be/05-the-node-host.md)
|