Compare commits

...
Author SHA1 Message Date
jschoubben ddb3980f09 Issue 143: correct the diagnosis — the step exists and did not fire
The first account said the declaration carries no resource that would disable the
found firewall and that nothing implemented the sentence the flip prints. Wrong:
the mechanism is a step in the host's own apply, retireFirewall, and it is careful
— it reads back that the mesh's table is loaded before retiring anything, records
the forward policies first, and verifies ufw reports inactive afterwards.

What is established: ufw was active and enabled two minutes after a flip that
reported the node converged with nothing failed; and the machine's record now
reads disabled_by_mesh: true, written by a reconcile fifty minutes later that
found ufw already inactive because an operator had disabled it by hand.

So the step did not take effect and the record says it did. The candidates are
named rather than chosen — the step is skipped silently when the apply has any
failure, the mesh's table is loaded by a service in the same apply so ordering is
open, and the host's detail lines do not reach the journal, which is why this has
candidates instead of a cause.
2026-09-29 02:33:10 +02:00
mesh-admin 75a8f6abc7 Merge pull request 'Issues 143 and 144: the found firewall is neither retired nor all of it' (#176) from issue/143-and-144-the-found-firewall into main 2026-09-28 23:36:10 +00:00
jschoubben 862253518f Issues 143 and 144: the found firewall is neither retired nor all of it
Both found by converging the control node — the first machine with a firewall to
flip, since the two before it had none.

143: the preview and the flip both say the found firewall is disabled. The node
reported converged, 372 resources applied, nothing failed, and ufw is still
enabled and active. A converged node's declaration carries no resource that would
disable it; the sentence is printed by the command and nothing implements it.

144: ufw was never what filtered the traffic that mattered there. Fifty forwarded
openings converged through it had matched zero packets, while a chain the
predecessor installed in the container runtime's pre-accept hook did the work —
in memory only, recreated by nothing. The mesh's filter now covers that path, so
the machine no longer depends on it, but the chain remains and is the only thing
refusing the bus and the registry, which the design requires reachable from
anywhere so a machine can enrol before it has a private address.
2026-09-29 01:36:08 +02:00
mesh-admin 51ef3eb7e2 Merge pull request 'ADR 0142: the mesh delivers its own components as binaries' (#175) from decision/0142-mesh-delivers-its-own-components into main 2026-09-28 22:51:44 +00:00
jschoubben 346e613995 ADR 0142: the mesh delivers its own components as binaries, not container images
Measured: the host is a binary somebody copied to four machines, owned by no
package and built by nothing, while the controller, catalogue, builder and vault
are container images publishing no ports at all. Same language, same project,
same kind of work, delivered two ways — and the difference is not a judgement
about either, it is that images are the only delivery that works.

What it costs: genesis must raise a container runtime before the control plane
can exist; updating the control plane goes through a registry the control plane
runs; a host change cannot be rolled out at all; and compiling the language the
mesh is written in is not a capability of the builder, so the controller is built
from a hand-written Dockerfile — the incantation the bundle toolchain exists to
abolish.

Third-party software stays a container: the store, the registry, the broker are
somebody else's build. The container runtime stays on the machine for modules.
What changes is that the control plane no longer needs it to exist.

The receiving half is already built and tested (ADR 0141). Staged: compile Go, an
artifact names its target, deliver a binary, the host first, then the rest, genesis
last.
2026-09-29 00:51:42 +02:00
mesh-admin 25ca9898d5 Merge pull request 'ADR 0141: a progressive insight on what delivery costs' (#174) from decision/0141-progressive-insight-on-delivery into main 2026-09-28 22:32:17 +00:00
jschoubben 709c095387 ADR 0141: a progressive insight — the delivery is not 'nothing new'
The record claimed a version reaches a machine as an ordinary archive with
nothing new needed. Two things it needs do not exist: no toolchain can compile
the host (the list is typescript and python, and the control plane, also Go, is
built as an image from a Dockerfile instead), and nothing interpolates a built
version into a resource path, so nothing can ask for .../versions/<version>/.

The decision, the options weighed and every consequence stand — the host half is
merged and tested. What was understated was the cost, so it is corrected in place
and dated rather than superseded.
2026-09-29 00:32:14 +02:00
mesh-admin c262de3833 Merge pull request 'ADR 0141: the host delivers its own successor' (#173) from decision/0141-the-host-delivers-its-own-successor into main 2026-09-28 22:22:28 +00:00
jschoubben a815433214 ADR 0141: the host delivers its own successor, and versions live side by side
The supervision was already right — a clean exit means the host stood aside and
the launcher runs what is on disk, failures are counted, and a rollback happens
at the limit. Two things made it dead code: nothing told the running host a
successor was waiting, and the rollback resolved its known-good version through
pacman, which no machine here uses and which two of three operating systems do
not have.

Keeping a version rather than a path was the clue. Versions live side by side in
directories named for them; the newest runs; the running one stands aside between
reconciles; a reconcile that completes records itself and retires what is older
than its predecessor; rollback starts that predecessor. No new resource kind and
nothing new on the bus — an archive already fetches by digest, and the path
written is never the path executing.

Answers issue 142.
2026-09-29 00:22:04 +02:00
mesh-admin 61dc90e508 Merge pull request 'Issue 142: the host is the one thing the mesh does not deliver' (#172) from issue/142-the-host-is-not-delivered into main 2026-09-28 22:03:58 +00:00
jschoubben a7cf5c0c1b Issue 142: the host is the one thing the mesh does not deliver
A host change merged yesterday reached no machine without a person copying a
file. The host is not a build target, no declaration delivers it, and the half
that recovers from a bad host — noticing the executable changed, a known-good
record, a launcher that rolls back — is written, tested and called by nothing.
All four machines run a byte-identical hand-copied binary that no package owns
and no record names, so nothing can say a machine is behind.

Found because ADR 0140 needs the machine to report a new fact, and merging that
could not roll it out.
2026-09-29 00:03:41 +02:00
mesh-admin c6f86ae935 Merge pull request 'ADR 0140: the filter constrains what arrives from outside' (#171) from decision/0140-filter-constrains-what-arrives-from-outside into main 2026-09-28 21:32:10 +00:00
jschoubben 1aeb4fe8d8 ADR 0140: the filter constrains what arrives from outside, and says nothing about a machine's own guests
Reading a converged machine's rendered rules showed the cause: the chain blocks
everything passing through and then allows the machine's own containers back by
listing their address ranges. 0137 made that list typeable and 0139 tried to
generate it; both refined a list that should not exist, because the mesh has no
position on a container reaching outward. Constrain what arrives from outside,
allow what did not, and let the machine report which links face outside — one
fact instead of a list. Ports keep following the modules unchanged.

The records check now allows one record to supersede several, and stops
requiring a withdrawn record's own citations to be live.
2026-09-28 23:31:50 +02:00
mesh-admin 08108b569d Merge pull request 'ADRs 0138 and 0139: an endpoint's reach, and networks forwarded because a module declared them' (#170) from decision/0138-endpoint-reach-and-0139-declared-networks into main 2026-09-28 21:03:49 +00:00
15 changed files with 871 additions and 38 deletions
+16 -3
View File
@@ -164,6 +164,12 @@ def check_rests_on(failures, records):
# decision is exactly what as-is is for."
if rel(path).startswith("03-DESIGN/00-as-is/"):
continue
# A withdrawn record's citations are history. It instructs nobody -- every reader
# is sent to its superseder -- so what it was built on may itself be withdrawn.
# Refusing that would mean rewriting the lineage of a record whose reasoning is
# the thing the immutability rule protects.
if frontmatter(read(path)).get("status") == "superseded":
continue
# An extension that supersedes legitimately names what it replaced.
this = ADR_FILE.match(os.path.basename(path))
supersedes = records[number]["front"].get("superseded-by", "")
@@ -241,13 +247,20 @@ def check_supersession_symmetry(failures, records):
failures.add("supersession", rel(record["path"]), f"superseder does not exist: {by}")
continue
other = records[match.group(1)]
claims = os.path.basename(str(other["front"].get("supersedes", "")))
if claims != record["name"]:
# `supersedes:` may name one record or several. One decision replacing two is a real
# situation -- two records that built and refined the same wrong mechanism are withdrawn
# by the one record that removes it -- and a check that allows only one would force
# either a chain of pro-forma records or an unmarked supersession.
claimed = other["front"].get("supersedes", "")
if isinstance(claimed, str):
claimed = [claimed] if claimed else []
claims = [os.path.basename(str(entry)) for entry in claimed]
if record["name"] not in claims:
failures.add(
"supersession",
rel(other["path"]),
f"ADR {number} says this supersedes it; this record does not say so "
f"(supersedes: {claims or 'absent'})",
f"(supersedes: {', '.join(claims) or 'absent'})",
)
@@ -1,10 +1,11 @@
---
topic: what runs on it
status: accepted
status: superseded
date: 2026-09-28
deciders: jochen
extends: 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
reconstructed: false
superseded-by: 02-DECISIONS/0140-the-filter-constrains-what-arrives-from-outside.md
---
# 137. A machine says which networks it routes
@@ -1,10 +1,11 @@
---
topic: what runs on it
status: accepted
status: superseded
date: 2026-09-28
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0137-a-machine-says-which-networks-it-routes.md
superseded-by: 02-DECISIONS/0140-the-filter-constrains-what-arrives-from-outside.md
---
# 139. A network is forwarded because a module declared it
@@ -0,0 +1,146 @@
---
topic: what runs on it
status: accepted
date: 2026-09-28
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md
supersedes:
- 02-DECISIONS/0137-a-machine-says-which-networks-it-routes.md
- 02-DECISIONS/0139-a-network-is-forwarded-because-a-module-declared-it.md
---
# 140. The filter constrains what arrives from outside, and says nothing about a machine's own guests
## Context
The filter the mesh derives blocks traffic passing *through* a machine unless something allows it,
because a container's published port is traffic passing through rather than traffic arriving at the
machine itself ([ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md),
[issue 047](../04-ISSUES/047-the-firewall-does-not-cover-published-container-ports/00-report.md)).
Having blocked all of it, the filter then had to let the machine's own containers reach outward again.
It does that by listing the address ranges those containers sit on.
As rendered on a converged workstation today:
```
policy drop
ct state established,related accept
ip saddr 172.16.0.0/12 accept
ip saddr 192.168.128.0/17 accept
ip saddr 10.0.0.0/8 accept
ip saddr 192.168.16.0/20 accept
... four more
```
Two of those ranges were constants in the control plane's source. The rest were typed by the operator
after [ADR 0137](0137-a-machine-says-which-networks-it-routes.md), which existed to make the typing
possible, because converging that workstation had cut every one of its containers off from the
internet and nothing reported a fault
([issue 137](../04-ISSUES/137-converging-a-machine-cut-off-its-own-guests/00-report.md)).
**The list is the mistake, not its contents.** Every attempt to make it correct fails the same way.
A constant describes one machine. A typed range goes stale, and cannot tell a network the mesh made
from one a predecessor left behind — measured on the control-node, where six such ranges fall outside
the constants and two of the six belong to services the mesh does not run
([issue 141](../04-ISSUES/141-the-forward-chain-does-not-follow-the-modules/00-report.md)).
[ADR 0139](0139-a-network-is-forwarded-because-a-module-declared-it.md) tried to generate the same
list from the modules and put half the rule set on the machine to do it. Three records, one list, and
the list should not exist.
**Because the mesh has no policy about a container reaching outward.** What the filter is for is
stated in [ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md): which port is open,
and to whom. That is about what arrives. A container of this machine's own opening a connection to
something else is not a port being opened to anybody, and enumerating the addresses it might do so
from is bookkeeping about the machine's internal plumbing, which the mesh neither owns nor can know.
**The system being replaced never had this fault, and its rule says why.** The chain still protecting
the control-node applies only to traffic arriving on that machine's outward link, and leaves
everything else alone. The mesh's filter dropped that distinction and replaced it with a list of
addresses.
## Considered Options
1. **Keep the list and generate it better** — from the modules' declared networks, or from what the
machine reports. Rejected: [ADR 0139](0139-a-network-is-forwarded-because-a-module-declared-it.md)
is that, and it puts part of the rule set on the machine, which makes the rule set partly the
machine's and the derivation advisory.
2. **Name the guest links instead of their addresses, and allow only those.** Rejected as more than is
needed: it fails in the safe direction, but it is still a list that has to keep up with the
machine, and the thing it protects against — a container reaching outward — is not a thing the mesh
has a position on.
3. **Do not block traffic passing through at all.** Rejected: that is
[issue 047](../04-ISSUES/047-the-firewall-does-not-cover-published-container-ports/00-report.md),
where a published port was reachable from anywhere because no rule mentioned it.
4. **Constrain what arrives from outside, and nothing else.** Adopted.
## Decision
**The filter constrains traffic arriving from outside the machine, and says nothing about traffic that
did not.** Traffic passing through the machine is allowed unless it arrived on one of the machine's
outward links, in which case it is allowed only where a declared endpoint's reach admits it
([ADR 0138](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md)). A container of this
machine's own reaching anywhere is not filtered, because the mesh has no position on it.
**A machine says which of its links face outside.** One node-level fact, reported by the machine the
way it already reports the kind of firewall it found and the tunnel it carries — not a setting, not a
list of addresses, and not something anybody types. It does not change when a module is added or
removed, which is what separates it from the list it replaces.
**A machine that has reported no outward link is sent no filter.** Rendering a rule around a link
whose name is not known produces a rule set that does not load, which is a machine filtering nothing
while its unit reports success. The refusal happens in the control plane, where a person reads it, and
the machine keeps the filter it already has.
**No addresses of the machine's own networks appear in the filter.** The two constants are removed and
`node networks` is removed with them, along with everything any machine was told to say through it.
Ports continue to follow the modules exactly as before: a module assigned to a machine opens the port
its assignment says it reaches on, and nothing about a network is said anywhere.
## Consequences
- **Three records collapse into one rule.** 0137 and 0139 are superseded. What 0137 was right about —
that converging a machine had silently cut off its own containers, and that nothing previewed it — is
answered by removing the cause rather than by giving the operator a way to compensate for it.
- **Every machine already converged loses its declared ranges and keeps working**, because the traffic
those ranges allowed is now allowed by not having arrived from outside. The workstation's five ranges
and the laptop's one are deleted rather than migrated.
- **A machine's test beds stop being a special case.** A bed's network is created while the machine
runs and was the case no list could cover; it is now covered by not being mentioned.
- **A new fact travels in the report**, and the control plane refuses to compose a filter without it,
so the order of the roll-out matters: the machines report before the control plane depends on it.
- **A machine with more than one outward link says so**, and a machine that acquires one while the mesh
is not looking is treated as internal until its next report. That window is the cost of this shape;
it is bounded by the report interval, and it exists on machines whose outward link changes, which
are the machines with nothing published to the outside.
- **What got harder:** nothing in the declaration, and one more thing a machine must be able to work
out about itself. A machine that cannot say which link faces outside cannot be given a filter.
## How it is checked
- **A machine's own container reaches outward with no network named anywhere.** A bed converges a
machine carrying containers on several networks, none of them mentioned in any setting, and each
reaches out afterwards. This fails against the previous behaviour, where the same flip cut them off,
and that is how it is written.
- **A port declared reachable from outside is reachable; one that is not, is not.** Probed from off the
machine's private network, for a published port and for an undeclared one, before and after the flip.
- **A network created after the filter was composed needs no new filter.** A network is made on the
machine after its last declaration and a container on it reaches out, with nothing re-sent.
- **No address of a machine's own networks appears in a rendered filter**, asserted on the text so a
range cannot creep back in.
- **A machine that reports no outward link is sent no filter, and the refusal names it** — asserted in
the control plane, and that the machine's existing filter is left alone.
- **A machine reporting two outward links has both constrained**, asserted per chain body so a rule
covering one and not the other cannot pass.
## References
- [ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md) — what the filter is for
- [ADR 0138](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md) — what admits traffic
arriving from outside
- [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md) — why traffic passing through is
filtered at all
- [ADR 0137](0137-a-machine-says-which-networks-it-routes.md),
[ADR 0139](0139-a-network-is-forwarded-because-a-module-declared-it.md) — superseded here
- [issue 137](../04-ISSUES/137-converging-a-machine-cut-off-its-own-guests/00-report.md),
[issue 141](../04-ISSUES/141-the-forward-chain-does-not-follow-the-modules/00-report.md)
@@ -0,0 +1,154 @@
---
topic: what runs on it
status: accepted
date: 2026-09-29
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0005-the-node-host.md
---
# 141. The host delivers its own successor, and versions live side by side
## Context
[Issue 142](../04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/00-report.md). A
merge builds every changed module and the control plane — which is itself a module — and the result
reaches the machines running it with nobody asking. The host is the exception: it is not a build
target, no declaration delivers it, and every machine in this mesh runs a byte-identical binary that
somebody built on a workstation and copied out.
The half that *recovers* from a bad host exists. `internal/upgrade` can tell that the executable this
process started from was replaced on disk, and it records which version last completed a reconcile.
The launcher counts consecutive failed starts, calls a rollback at the limit, and treats a clean exit
as the host standing aside so that the next loop runs whatever is on disk now. That supervision is
complete and correct.
Two things make it dead code:
- **`Replaced()` is called by nothing but its own tests.** Nothing tells the running host that a
successor is waiting.
- **The rollback resolves a version through the machine's package manager** — `pacman -U` from the
package cache. No machine here has the host installed as a package, so the recovery cannot run on
any of them; and being written in one package manager's terms, it cannot run on two of the three
operating systems the host is built for — [ADR 0005](0005-the-node-host.md) builds one binary per
operating system, pinned at link time.
**The record already points at the answer.** What is kept is a *version*, not a path. Keeping a
version is only useful to something that can choose between versions present on the machine, which is
what the package manager was being asked to do. The versions can simply be on disk.
## Considered Options
1. **Deliver the host as a package, as the rollback assumes.** Rejected: it needs a package built and
a repository trusted per operating system, three of each, and the existing `package` resource
asserts presence and deliberately never a version — "version is the package manager's business and
the mesh does not hold a second opinion about it" — so it cannot ask for a particular host anyway.
Heaviest of the three and the only one that is different on every machine.
2. **Write the new binary over the running one.** Rejected on a fact: a running executable cannot be
truncated, and `archive` opens what it unpacks with `O_TRUNC`. It could be made to write and
rename, which is better hygiene and worth doing for its own sake, but it buys nothing here that
option 3 does not, and it leaves rollback with nowhere to go back to.
3. **Versions side by side; the newest retires the old.** Adopted.
## Decision
**A host version is delivered as an archive into a directory named for it, and never over a running
one.** The declaration names it like any other archive — fetched by digest, the digest checked before
anything is unpacked. Nothing new travels, no new resource kind, and no change to how archives are
applied, because the path being written is not the path being executed.
**The launcher starts the most recently delivered version.** That is what "the newest" means: the
version whose directory arrived last. It reads no pointer and follows no link — the mesh creates no
links ([ADR 0012](0012-the-mesh-creates-no-symlinks.md)) — and the version is in the path, so nothing
has to be told what is running.
**The running host stands aside for a successor, and only between reconciles.** Finding a newer
version delivered, it finishes the reconcile it is in and exits cleanly. The launcher already reads a
clean exit as exactly this and starts what is on disk now. A host that stood aside mid-apply is the
half-configured machine this project exists to prevent, so the check happens at the boundary and
nowhere else.
**A version that completes a reconcile records itself, and retires what came before it.** The
known-good record is written as it is today. Then versions older than the one before the running one
are removed: the running version and its predecessor are kept, which is exactly what a rollback
needs, and nothing else accumulates.
**Rollback starts the previous version instead of reinstalling a package.** At the failure limit the
launcher pins the known-good version and starts that, once. The second failure is still a different
diagnosis — the previously working version does not run either, so it is the machine and not the
binary — and the halt is unchanged. No package manager, no package cache, and the same script on every
operating system.
**A machine says which host version it is running,** on the report it already sends, beside the other
facts it states about itself. Without it nothing can say a machine is behind, so "every machine
current with its source" cannot include the host.
## Progressive insight — 2026-09-29, the same day
**The delivery is not "nothing new", and this record said it was.** The decision above stands and is
built: versions side by side, the newest runs, the running host stands aside between reconciles, a
completed reconcile retires what is older than the predecessor, rollback picks a directory. What was
wrong was a claim about how a version reaches a machine. The paragraph on delivery said the
declaration "names it like any other archive… nothing new travels, no new resource kind"; the second
half is true and the first is not, because two things the delivery needs do not exist:
- **Nothing can compile it.** A `bundle` artifact is compiled by a closed list of toolchains —
typescript and python — whose own comment says adding a language is a decision, because a language
used by *modules* needs an SDK carrying the broker client, the event envelope and tool serving. The
host uses none of that: it is what applies modules, not one of them. So the obligation that list
warns about attaches to a module written in a language, not to the language being buildable, and
the control plane — also written in Go — is built as an image from a Dockerfile rather than through
a toolchain at all.
- **A version cannot reach the path.** An `archive` resource names a fixed path in the manifest, and
nothing interpolates the built version into it, so nothing can ask for
`…/versions/<version>/`.
Neither changes what was decided, which options were weighed, or any consequence: the shape is
unaffected and the host half is merged and tested. What it changes is the cost, which this record
understated as none. The remaining work is a way to build the host and a way to name a version in a
path, and until both exist nothing delivers a version and every machine takes the fallback — which is
what every machine does today.
## Consequences
- **The host becomes a build target and a module** — a module whose resource is the next host, applied
by the host that is running. The bootstrap is not circular because the two are different versions in
different directories.
- **Rollback becomes usable on every machine**, having been usable on none. It also stops being
written in one operating system's terms.
- **One copy by hand remains, once.** The first host that understands versioned directories cannot be
fetched by a host that does not. That copy is the last, and it is the honest cost of the change
rather than a step in the design.
- **Two versions occupy disk instead of one.** About nine megabytes. The predecessor is the price of a
rollback that does not depend on a cache somebody else may clean.
- **What got harder:** a host must now be able to find its own successor and to judge when it is safe
to stand aside. Both are between reconciles, which is the only moment the host is not mid-change.
- **A machine that is never told a newer version keeps running what it has**, indefinitely and
visibly, because its report says which version that is.
## How it is checked
- **A delivered version is run, and the old one is not.** A bed delivers a second version to a machine
running the first; the host exits between reconciles, the launcher starts the new one, and the
machine reports the new version. This fails against the previous behaviour, where nothing notices a
delivered version at all.
- **It stands aside between reconciles and never inside one.** Asserted by delivering a version while
an apply is in flight: the apply completes, and the exit follows it.
- **A version that will not start is rolled back to its predecessor, once**, and the second failure
halts with the machine named rather than the binary — asserted with no package manager involved.
- **A completed reconcile retires what is older than the predecessor**, and never the predecessor
itself, because that is what a rollback needs. Asserted on the directory afterwards.
- **The report names the running version**, asserted end to end rather than on the function that reads
it, since the point is that the control plane can tell a machine is behind.
- **The launcher picks the newest delivered version** with no pointer file and no link, asserted by
delivering two and checking which runs.
## References
- [ADR 0005](0005-the-node-host.md) — the host, and what its supervision is for
- [ADR 0010](0010-delivery.md) — a declaration is owned resources; this adds no kind to it
- [ADR 0012](0012-the-mesh-creates-no-symlinks.md) — why the version is in the path
- [ADR 0005](0005-the-node-host.md), *it is built per operating system* — why a rollback written in
one package manager's terms was wrong for two of three
- [issue 142](../04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/00-report.md) — the
measurement
@@ -0,0 +1,158 @@
---
topic: the mesh
status: accepted
date: 2026-09-29
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0141-the-host-delivers-its-own-successor.md
---
# 142. The mesh delivers its own components as binaries, not as container images
## Context
Measured on the control-node, 2026-09-29:
| what | how it runs | publishes |
|---|---|---|
| host | a binary on the machine | — |
| controller, catalogue, builder, vault | containers | nothing |
| store, registry, broker | containers | ports |
**The mesh's own software is delivered two ways, and the difference is not a property of the
software.** The host and the controller are both written in the same language, both the mesh's own,
both doing the mesh's own work. One is an image fetched from a registry. The other is a file somebody
copied to four machines, owned by no package, built by nothing
([issue 142](../04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/00-report.md)).
**The reason is not a judgement about either, it is that images are the only delivery that works.**
There is no way to put a binary on a machine. The host is hand-copied because of that, and the
controller is an image because of that. Neither was chosen on its merits.
What it costs, all of it measured rather than argued:
- **Genesis must raise a container runtime before the control plane can exist.** The bundle carries
three images and one of them is the controller, *"in the bundle for the same reason they are: there
is nothing to fetch it with yet"*
([design 07](../03-DESIGN/01-to-be/07-the-foundation.md)). So the hardest moment in the mesh's life
has a prerequisite that the thing being started does not need.
- **Updating the control plane depends on the control plane.** Its image is fetched from the registry,
which is a container the controller manages.
- **A change to the host cannot be rolled out at all.** Every machine here runs a byte-identical
hand-copied binary. A change merged yesterday reached none of them.
- **Compiling the language the mesh is written in is not a capability of the builder.** The bundle
toolchains are typescript — real, with a registered base module — and python, which is named in the
list and absent from the catalogue. The controller is built as an image from a Dockerfile, which is
the per-repository incantation the bundle toolchain exists to abolish
([design 18](../03-DESIGN/01-to-be/18-building-a-module.md)).
The half that *receives* a binary safely is already built and tested
([ADR 0141](0141-the-host-delivers-its-own-successor.md)): versions side by side in directories named
for them, the newest run, the running one standing aside between reconciles, retirement keeping the
predecessor, and a rollback that chooses a directory. What is missing is everything that puts one
there.
## Considered Options
1. **Leave it as it is.** Rejected: it is not a design, it is the reach of one mechanism. And it is
what makes a host change undeliverable.
2. **Containerise the host too**, so everything is delivered one way. Rejected: the host is what
starts the container runtime and what applies containers. A host in a container is the bootstrap
problem made total, and the machine would have no way back from a bad one.
3. **Deliver the mesh's components as operating-system packages.** Rejected for the reason
[ADR 0141](0141-the-host-delivers-its-own-successor.md) rejected it for the host: a package and a
trusted repository per operating system, three of each, and the `package` resource asserts presence
and deliberately never a version.
4. **Binaries for the mesh's own components, containers for third-party software.** Adopted.
## Decision
**The mesh's own components are delivered as binaries on the machine.** The host, the controller, the
catalogue, the builder, the vault — the software this project writes. They are delivered by the
mechanism [ADR 0141](0141-the-host-delivers-its-own-successor.md) built: an archive, fetched by
digest, unpacked into a directory named for its version, with the running one standing aside between
reconciles and a rollback that chooses the predecessor.
**Third-party software stays a container.** The store, the registry, the broker. They are somebody
else's build, they are already adopted as modules
([ADR 0078](0078-the-store-and-broker-are-modules.md)), and an image is the right way to carry
somebody else's software. **The container runtime remains required** — modules use it — so this
removes a dependency from the control plane, not from the machine.
**The builder compiles the languages the mesh is written in.** A toolchain for Go, with a base module
providing the compiler, exactly as typescript has. The obligation the toolchain list warns about — an
SDK carrying the broker client, the envelope and tool serving — attaches to a *module* written in a
language, not to the language being compilable. None of these components is a module in that sense;
the host is what applies modules.
**An artifact says what it targets.** A compiled binary is per operating system, pinned at link time
([ADR 0005](0005-the-node-host.md)), and a toolchain deliberately takes nothing from the module,
because anything a module could override there it would be writing a Dockerfile to override. So the
target is a property of the artifact rather than of the recipe, and one artifact declared per target
is one build each.
**A component's version comes from where it sits, not from its linker.** It is unpacked into a
directory named for its version, so it can read its own version from its path. The stamp goes, and
with it the need for a build to know what it will be called.
**Genesis carries a binary reference where it carried an image reference.** The principle does not
change — the bundle names a thing by digest and the host fetches it, pinned because nothing can
resolve a version when no mesh exists — and the container runtime stops being a prerequisite for the
control plane. It stays a prerequisite for the store and the broker, which is where it belongs.
**The order is staged, and each step stands alone.** Compiling Go; an artifact naming its target;
delivering a binary; the host as the first component delivered; the controller, catalogue, builder and
vault out of their containers; genesis last. Genesis is last for the reason it is always last: it
matters for a machine nobody has yet, and every earlier step is provable on a mesh that exists.
## Consequences
- **One delivery for the mesh's own software**, so a change to the host ships the way a change to the
controller does, and neither is copied by hand.
- **The control plane stops depending on a container runtime and on its own registry.** Both remain on
the machine for other reasons; neither gates the control plane's own life any more.
- **`Replaced()`, the known-good record and the launcher's rollback stop being dead code.** They were
written for this and have been called by nothing but their tests.
- **Four more components gain a rollback they do not have.** Today a bad controller image is recovered
by an operator; under this it is recovered the way a bad host is.
- **Two versions of each component occupy disk.** Around nine megabytes each. The predecessor is what a
rollback needs.
- **Genesis gets smaller, not larger.** One fewer image to carry and one fewer runtime to raise before
the control plane.
- **This does not make the components smaller or simpler.** They are the same programs; what changes is
how they arrive. A reader expecting the containers to have been hiding complexity will not find any.
- **What got harder:** the builder gains a language, artifacts gain a target, and the mesh gains a
second kind of thing it must deliver correctly — one where getting it wrong takes the control plane
down rather than a module. That is why the host is first: it is the component whose recovery is
already built and tested.
## How it is checked
- **A component is delivered and runs, with nothing copied by hand.** A bed builds the host from its
repository, delivers it to a machine running an older one, and the machine reports the new version.
This fails today at the first step, because nothing builds it.
- **Each target is built once and only the matching one is delivered.** Asserted by declaring an
artifact per operating system and checking that a machine is offered the one it can run — a host
built for another is what ADR 0005's link-time pin exists to refuse.
- **A component reads its version from its path**, asserted by unpacking the same bytes into two
differently named directories and seeing each report its own.
- **A bad component is rolled back without an operator**, for the host first: a version that will not
start is replaced by its predecessor once, and the second failure halts naming the machine.
- **The control plane comes up with no registry reachable**, which is the dependency this removes —
asserted by raising it with the registry stopped.
- **Genesis raises a control plane with no container runtime running**, and raises the store and the
broker afterwards. Last, and on a machine with nothing on it.
- **A published port count that does not change.** The mesh's own components publish nothing today, so
moving them out of containers must not open anything — asserted on the machine's reachable set before
and after, which the converge preview already reads.
## References
- [ADR 0141](0141-the-host-delivers-its-own-successor.md) — the receiving half, already built
- [ADR 0005](0005-the-node-host.md) — the host, its supervision, and one binary per operating system
- [ADR 0078](0078-the-store-and-broker-are-modules.md) — why third-party software stays a container
- [ADR 0006](0006-the-substrate-and-the-control-plane.md) — what genesis must raise, and in what order
- [issue 142](../04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/00-report.md) — the
measurement that started this
- [design 07](../03-DESIGN/01-to-be/07-the-foundation.md) — the bundle's three images, one of them the
controller
+5 -2
View File
@@ -142,6 +142,7 @@ python3 00-META/checks/index.py fail if stale
- **0131** — [Everything on the mesh speaks to the broker seat, and AMQP is not a provision](0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)
- **0132** — [A seat carries the tools its holder must serve](0132-a-seat-carries-the-tools-its-holder-must-serve.md)
- **0134** — [The mesh says what it applied](0134-the-mesh-says-what-it-applied.md)
- **0142** — [The mesh delivers its own components as binaries, not as container images](0142-the-mesh-delivers-its-own-components-as-binaries.md)
### Its tiers, from the bottom up
@@ -217,9 +218,11 @@ python3 00-META/checks/index.py fail if stale
- **0133** — [A module owns its migrations, and the mesh owns when they run](0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md) *(superseded)*
- **0135** — [A module version prepares its state before it runs](0135-a-module-version-prepares-its-state-before-it-runs.md)
- **0136** — [A step gates its module, not the machine](0136-a-step-gates-its-module-not-the-machine.md)
- **0137** — [A machine says which networks it routes](0137-a-machine-says-which-networks-it-routes.md)
- **0137** — [A machine says which networks it routes](0137-a-machine-says-which-networks-it-routes.md) *(superseded)*
- **0138** — [An assignment binds an endpoint and says how far it reaches](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md)
- **0139** — [A network is forwarded because a module declared it](0139-a-network-is-forwarded-because-a-module-declared-it.md)
- **0139** — [A network is forwarded because a module declared it](0139-a-network-is-forwarded-because-a-module-declared-it.md) *(superseded)*
- **0140** — [The filter constrains what arrives from outside, and says nothing about a machine's own guests](0140-the-filter-constrains-what-arrives-from-outside.md)
- **0141** — [The host delivers its own successor, and versions live side by side](0141-the-host-delivers-its-own-successor.md)
### How it is built
+44 -1
View File
@@ -2,8 +2,9 @@
layer: to-be
status: in-progress
code: [mesh-host]
updated: 2026-09-22
updated: 2026-09-29
decisions:
- 02-DECISIONS/0141-the-host-delivers-its-own-successor.md
- 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
- 02-DECISIONS/0102-the-mesh-writes-into-a-shared-file-never-over-it.md
- 02-DECISIONS/0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md
@@ -423,3 +424,45 @@ ignored an instruction and "applied" would be a lie. Applying stays one at a tim
is not. **Checked** by the link's unit tests on the drain, and by the genesis bed's settle wait,
which counts on a node catching up to the newest declaration rather than the oldest.
## The host delivers its own successor
*2026-09-29, from a change to the host that could reach no machine —
[issue 142](../../04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/00-report.md),
settled by [ADR 0141](../../02-DECISIONS/0141-the-host-delivers-its-own-successor.md).*
A merge builds every changed module and the control plane, and the result reaches the machines running
it with nobody asking. The host was the exception: not a build target, named by no declaration, and
identical on every machine because somebody had copied it there.
The supervision needed for this was already right. A clean exit from the host means it has stood aside,
and the launcher's next turn runs whatever is on disk. Consecutive failed starts are counted, a
rollback happens at the limit, and a second failure halts with the machine named rather than the binary.
What was missing was smaller than it looked: nothing told the running host a successor was waiting, and
the rollback resolved its known-good *version* through one operating system's package manager, which no
machine here used.
Keeping a version rather than a path was the clue. That is only useful to something that can choose
between versions present on the machine — so the versions live side by side:
- **A version arrives as an archive, in a directory named for it.** The ordinary resource, fetched by
digest and checked before anything is unpacked. The path written is never the path being executed, so
replacing a running binary — which the kernel refuses — never comes up.
- **The launcher starts the most recently delivered version**, reading no pointer and following no
link, because the version is in the path.
- **The running host stands aside between reconciles and never inside one.** Standing aside mid-apply is
the half-configured machine this document exists to prevent.
- **A version that completes a reconcile records itself and retires what is older than its
predecessor.** The predecessor stays, because that is what a rollback needs.
- **Rollback starts that predecessor** instead of reinstalling a package: no package manager, no cache
somebody else may clean, and the same script on every operating system.
- **A machine says which host version it runs**, on the report it already sends, so being behind is
answerable at all.
One copy by hand remains, once: the first host that understands versioned directories cannot be fetched
by a host that does not.
*How it is checked* is stated with the decision — a second version delivered to a running machine is
run and reported; the exit follows an in-flight apply rather than interrupting it; a version that will
not start is rolled back once and the second failure halts; a completed reconcile retires what is older
than the predecessor and never the predecessor; and the newest of two delivered versions is the one
that runs.
+23 -2
View File
@@ -11,8 +11,9 @@ code:
- mesh-catalog modules/postgres
- mesh-catalog modules/lavinmq
- mesh-lab test/integration/mesh.test.ts (a bare machine becomes a mesh)
updated: 2026-09-22
updated: 2026-09-29
decisions:
- 02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md
- 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
- 02-DECISIONS/0088-the-foundation-filters-before-anything-listens.md
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
@@ -314,4 +315,24 @@ of a database and pushed to over the broker. What arrived and what did not is th
**One fault, and it was in the joining.** The token did not say what the mesh calls the machine,
so enrolment needed a flag its own help said it did not — and failed at the broker with an empty
username. Recorded in ADR 0004 as the fifth thing a token carries.
username. Recorded in ADR 0004 as the fifth thing a token carries.
## The mesh's own components arrive as binaries
*2026-09-29 —
[ADR 0142](../../02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md).*
The bundle carries three images and one of them is the controller, *because there is nothing to fetch
it with yet*. That reasoning holds and its conclusion changes: the controller is carried as a **binary**
reference rather than an image reference, pinned by digest exactly as before. Nothing about the bundle's
shape moves — it names a thing and the host fetches it — and the container runtime stops being something
genesis must raise before the control plane can exist. It still raises one, for the store and the broker,
which is where somebody else's software belongs.
The mesh's own components — the host, the controller, the catalogue, the builder, the vault — are
delivered as binaries into directories named for their versions, by the mechanism
[ADR 0141](../../02-DECISIONS/0141-the-host-delivers-its-own-successor.md) describes. Third-party
software stays a container. The split is not about isolation; it is about who built the thing.
Measured before deciding it: the mesh's own components publish no ports at all, so this opens nothing.
Only the store, the registry and the broker publish, and they are staying as they are.
+35 -27
View File
@@ -10,7 +10,7 @@ code:
updated: 2026-09-28
decisions:
- 02-DECISIONS/0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md
- 02-DECISIONS/0139-a-network-is-forwarded-because-a-module-declared-it.md
- 02-DECISIONS/0140-the-filter-constrains-what-arrives-from-outside.md
- 02-DECISIONS/0104-a-provision-may-be-answered-by-an-adapter-to-the-predecessor.md
- 02-DECISIONS/0106-the-bus-is-nats.md
- 02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md
@@ -627,41 +627,49 @@ the found firewall reloads and reachable from a container on the node, that a ma
enrols through the openings before and after a reload and a reboot, and that after the flip the
declared port is open and the undeclared one closed.
### The networks it forwards are the ones its modules declared
### It filters what arrives from outside, and not what the machine's own guests send
*2026-09-28, preparing the control-node's convergence —
[issue 141](../../04-ISSUES/141-the-forward-chain-does-not-follow-the-modules/00-report.md), settled by
[ADR 0140](../../02-DECISIONS/0140-the-filter-constrains-what-arrives-from-outside.md), which replaces
[ADR 0137](../../02-DECISIONS/0137-a-machine-says-which-networks-it-routes.md) and
[ADR 0139](../../02-DECISIONS/0139-a-network-is-forwarded-because-a-module-declared-it.md).*
The forward chain denies by default, so a machine's own guests have to be allowed back in. Until now
that was two ranges named in the control plane and, since
[ADR 0137](../../02-DECISIONS/0137-a-machine-says-which-networks-it-routes.md), a list a machine could
add to. On the machine measured here, six of its container networks fell outside those ranges — and
four of the six were networks the mesh's own modules had declared and the host had created, while two
were the predecessor's leftovers. A range wide enough to keep the four keeps the two: a filter widened
by hand to protect what should not be there.
Traffic passing through a machine is filtered, because a container's published port is traffic passing
through rather than traffic arriving at the machine itself. Having blocked it, the filter then had to
let the machine's own containers reach outward again — and it did that by listing the address ranges
they sit on. Two of those ranges were constants in this repository's code, and the rest were typed by an
operator after the flip had already cut a workstation's containers off from everything.
**A network is forwarded because a module declared it.** The forward chain forwards the networks of the
modules assigned to that node and, by default, nothing else; a module unassigned stops being forwarded
at the next reconcile. The control plane says which networks, by name, and the host — which created
them — renders their addresses, because the runtime allocates the subnet and the host is what knows it.
That keeps §4's shape and this document's: the mesh decides, the host applies.
**The list was the mistake, not its contents.** A constant describes one machine. A typed range goes
stale and cannot tell a network the mesh made from one a predecessor left behind — on the control-node,
six ranges fall outside the constants and two of the six belong to services the mesh does not run. The
attempt to generate the list from the modules put half the rule set on the machine and made the
derivation advisory. Three records, one list.
**This is derivation from the declaration, not from the machine.** 0137 rejected reading the machine,
for two reasons that do not apply here: a network created between two declarations is already named in
the one that asked for it, and a network nobody declared is never forwarded however it appeared. What
0137's mechanism keeps is the case it was right for — guests no module declares, a test bed's range —
added to the derived set and never replacing it.
**And the mesh has no position on a container reaching outward.** §4 exists to say which port is open
and to whom, which is about what arrives. A container of this machine's own opening a connection
somewhere is not a port opened to anybody, and the addresses it might do that from are the machine's
internal plumbing, which the mesh neither owns nor can know.
The runtime's own default bridge, which a container attaches to when it names no module network, is the
runtime's and not a module's, so the host renders it from what the runtime reports. With that, no range
is named in the control plane at all.
So the filter constrains what arrives from **outside** the machine and says nothing about what did not.
Traffic passing through is allowed unless it came in on one of the machine's outward links, and then
only where a declared endpoint's reach admits it (§6). The machine says which of its links face
outside — one fact it reports, like the kind of firewall it found and the tunnel it carries, not a
setting and not a list of addresses. It does not change when a module is added or removed, which is the
whole difference from what it replaces. A machine that has reported no outward link is sent no filter
at all, and keeps the one it has, because a rule written around a link with no name is a rule set that
does not load — a machine filtering nothing while its unit reports success.
*How it is checked:* a node with two modules declaring networks renders rules for exactly those two and
none for a third present on the machine that nothing declared — which fails against forwarding by
range, and is how it was written; unassigning one removes its rule at the next reconcile; the default
bridge is asserted for a runtime whose bridge is somewhere other than the old constant named; and the
guests of a declared network keep address and name service, per chain body.
Ports go on following the modules exactly as before: assign a module to a machine and the port its
assignment says it reaches on opens. Nothing about a network is said anywhere, by anybody.
*How it is checked:* a bed converges a machine carrying containers on several networks, none of them
named in any setting, and each reaches outward afterwards — which fails against the previous behaviour,
where the same flip cut them off, and is how it was written; a network made *after* the last declaration
needs no new filter; a declared port is reachable from off the private network and an undeclared one is
not; no address of a machine's own networks appears in a rendered filter, asserted on the text; and a
machine reporting no outward link is refused in the control plane with its existing filter left alone.
## 5 — Certificates
+26 -1
View File
@@ -5,8 +5,9 @@ code:
- mesh-controller cmd/mesh-builder
- mesh-controller internal/builder
- mesh-catalog modules/builder
updated: 2026-09-25
updated: 2026-09-29
decisions:
- 02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md
- 02-DECISIONS/0111-a-build-source-is-on-the-git-seat-or-external.md
- 02-DECISIONS/0097-a-vendor-image-is-a-declared-build-input.md
- 02-DECISIONS/0096-an-upstream-image-is-copied-between-registries.md
@@ -263,3 +264,27 @@ ships one and wrong for code the mesh built, which has no unit until the mesh wr
**Tools, hooks and consumers are not further modes**, which is the test of whether three is the
right number: they are loaded by a tool host, and a tool host is a process that stays up.
## The builder compiles the languages the mesh is written in
*2026-09-29 —
[ADR 0142](../../02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md).*
The toolchain list was typescript and python, and only typescript had a base module in the catalogue.
Meanwhile the control plane — written in the language this project is mostly written in — was built as
an image from a hand-written Dockerfile, which is the per-repository incantation this whole mechanism
exists to abolish.
So the list gains Go, with a base module providing the compiler exactly as typescript has one. The
obligation the list's own comment warns about — an SDK carrying the broker client, the event envelope
and tool serving — attaches to a **module** written in a language, not to the language being
compilable. The mesh's own components are not modules in that sense; the host is what applies modules.
**And an artifact says what it targets.** A compiled binary is per operating system, pinned at link
time, and a toolchain deliberately accepts nothing from the module — anything a module could override
there it would be writing a Dockerfile to override. The target is therefore a property of the artifact,
not of the recipe: one artifact declared per target, one build each.
A component's version stops being stamped in at link time. It is unpacked into a directory named for
its version, so it reads its version from its own path, and a build no longer has to know what it will
be called.
@@ -61,6 +61,18 @@ module declares — a test bed's pool — which is a much smaller residue than t
has to know the runtime's allocations to tell whether that line is sufficient. On the machine
measured here it read as though nothing needed saying.
## What was decided
*2026-09-28, later the same day.* The answer is not a better list. The question in the first open
item below — should the chain be derived from the networks the modules declare — was answered *no*,
after a converged machine's rendered rules were read: the chain blocks everything passing through the
machine and then allows its own guests back by listing their addresses. Every route to a correct list
fails, because the mesh has no position on a container reaching outward in the first place. The filter
now constrains what arrives from **outside** the machine and says nothing about what did not, and a
machine says which of its links face outside — one reported fact instead of a list. See
[ADR 0140](../../02-DECISIONS/0140-the-filter-constrains-what-arrives-from-outside.md), which
supersedes both 0137 and the first attempt at answering this.
## Open questions
- Should the forward chain be derived from the network resources the node's modules declare, with the
@@ -0,0 +1,76 @@
---
status: located
opened: 2026-09-29
located-in:
- mesh-host internal/upgrade
- mesh-host cmd/mesh-host
- mesh-controller (no build source for the host; no resource delivers it)
fixed-by:
amended-design: 03-DESIGN/01-to-be/05-the-node-host.md
---
# 142 — The host is the one thing the mesh does not deliver
## What was observed
A change to the host was merged and could not reach any machine without a person copying a file.
Checked on the mesh of four machines, 2026-09-29:
- **The host is not a build target.** Asked what had been built for it, the control plane answered
`nothing has been built for mesh-host`. A merge on the forge builds every changed module and the
control plane itself, because the control plane is a module. The host is not one, and nothing
builds it.
- **No declaration delivers it.** No resource kind names an executable to place on a machine, and
nothing on a machine fetches one.
- **The half that recovers from a bad host exists and is unused.** `internal/upgrade` can report that
the executable this process started from has been replaced on disk, and records which version last
completed a reconcile so a shell script can roll back a host that will not start. The launcher reads
that record and rolls back. But `Replaced()` is called by nothing except its own tests — the
recovery is wired and the delivery was never built.
- **Every machine runs a byte-identical binary, stamped by hand.** All four carry the same size and
the same timestamp, from the last time somebody built it on a workstation and copied it out. No
package owns the file.
## Why it matters beyond this instance
**The component that implements updating is the one thing not updated.** The mesh's stated shape is
that a push produces the right builds and they reach the machines running them with nobody asking. It
is true of every module and of the control plane. It is false for the host, which is what applies all
of them.
**It is a bootstrap problem being answered by a person.** The host cannot be an ordinary module
because the host is what applies modules; a module that replaces the thing applying it has to survive
its own replacement. That is a real difficulty, and the work already done — noticing that the
executable changed, recording a known-good version, a launcher that rolls back — is the hard half of
solving it. What is missing is the easy half, and its absence makes the hard half dead code.
**A hand-copied binary has no record anywhere.** Nothing says which version a machine runs, so
nothing can say a machine is behind, and the mesh's own account of itself — every machine current with
its source — cannot include the host. Four machines agreeing today is luck, not a property.
**And it silently gates any change that starts in the host.** A change that needs the host to report
something new cannot be rolled out by merging it: the control plane must wait for a person, and until
then it either refuses what depends on the new report or renders something wrong. That cost is paid by
every future change of this shape, and it was paid today.
## Open questions
- How is the host delivered without being applied by itself? A candidate shape: the host is built like
anything else, published as an artifact, and the *running* host fetches and stages the next one, then
stands aside — which is what `Replaced()` was written for and what the launcher's rollback already
covers.
- **Should this ride the bus, rather than becoming a mechanism of its own?** Everything else that
reaches a machine already does: a declaration is sent over it, a report comes back over it, and a
build announces what it produced on it, which is how a module's new version reaches the machines
running it. A host build announcing itself the same way, consumed by the host already running,
would make this the existing mechanism pointed at one more artifact rather than a second way of
delivering things. It would also give the machine somewhere to say which host it is running, on the
report it already sends.
- What records which version of the host a machine runs, so "behind" is answerable? Nothing does now.
- Does the host's version belong in its report, beside the other facts a machine states about itself?
- Who decides when a machine takes a new host — the mesh, on a build, or an operator per machine as
with converging? The rollback path means a bad host costs a reconcile rather than a machine, which
argues for the former.
- Does the same gap apply to the launcher and the units beside the binary, which are also files no
declaration names?
@@ -0,0 +1,102 @@
---
status: located
opened: 2026-09-29
located-in:
- mesh-host internal/apply/opening.go (retireFirewall)
- mesh-host internal/apply/apply.go (the condition it is called under)
fixed-by:
amended-design:
---
# 143 — Converging a machine does not retire the firewall it found, and says it does
## What was observed
The control-node was converged on 2026-09-29, the first machine with a found firewall to be flipped —
the two converged before it had none.
The preview said, and the flip repeated:
```
the found firewall (ufw) is disabled, never flushed: its configuration stays on disk
...
sent: the host loads the mesh's filter and disables the firewall it found
```
The mesh then reported the node `converged`, 372 resources applied, nothing failed. Afterwards, on the
machine:
```
systemctl is-enabled ufw -> enabled
systemctl is-active ufw -> active
```
*Corrected 2026-09-29, an hour later, from reading the host rather than the declaration.* **The first
account of this was wrong.** It said the declaration carries no resource that would disable the found
firewall, and that the sentence was printed by the command with nothing implementing it. The
declaration indeed carries no such resource — but the mechanism was never meant to be one. It is a
step in the host's own apply, `retireFirewall`, and it exists, is careful, and is strict: it refuses to
retire anything until it has read back from the machine that the mesh's own table is loaded, it records
the forward policies first so a half-done retirement can be retried, and it verifies ufw reports
inactive afterwards.
What is established is narrower and stranger than "nothing implements it":
- ufw was **active and enabled two minutes after the flip**, and the flip had reported the node
converged with 372 resources applied and nothing failed.
- The machine's own record now reads `disabled_by_mesh: true` — but it was written by a reconcile
*after* an operator disabled ufw by hand, roughly fifty minutes later. A reconcile found ufw already
inactive, asked it to be inactive, read that back, and recorded that the mesh had done it.
- So the step did not take effect at the flip, and the machine's record now says it did.
The candidates are named rather than chosen, because the evidence does not separate them: the step is
called only when the apply had no failures, and a skipped step is silent; the mesh's table is loaded by
a service in the same apply, so whether it was loaded *at the moment the step asked* is an ordering
question; and the host's own detail lines do not reach the journal, so what it decided is not
recoverable after the fact.
## Why it matters beyond this instance
**It is a stated behaviour that does not happen, reported as success** — the fault this repository
exists to catch, and
[ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md) states it as
part of what the flip *is*: "loads the mesh's derived filter in place of its refusal-only table, and
retires the found firewall by disabling it, never by flushing".
**It could only be found on the first machine that had one.** The two machines converged before this
had no firewall to retire, so the step had never run, and nothing reported that it had not. That is
the same shape as [issue 136](../136-a-module-may-name-a-program-the-machine-does-not-have/00-report.md):
a step that is silent when it does nothing.
**The machine is left doubly filtered, which is not what either firewall describes.** Every base chain
at a hook runs and a drop in any is final, so the machine now enforces the *intersection* of the mesh's
derived filter and a rule set left by the system being replaced. Nothing is broken by that today —
measured from outside, mail, the proxy and git-over-ssh answer and the databases and admin interfaces
are refused — but the machine's behaviour is described by neither of the two things claiming to
describe it, and the stale set includes a rule for a broker that no longer exists.
**And returning the node to adopted would be wrong in the other direction.** ADR 0100 says that
restores the found firewall by enabling it again; enabling something that was never disabled is
harmless, but the mesh's belief about which firewall is in force has been wrong in both modes.
## Open questions
- Which side owns retiring it — a resource in the declaration, so it is applied and reported like
everything else, or the flip as an act? A resource seems right: the flip is otherwise entirely
expressed as one, and an act that only the command performs cannot be re-checked on a later
reconcile.
- What should a reconcile do if the found firewall is enabled again by hand, or by a package update?
Convergence is a state, so presumably re-disable it and say so.
- Should the preview say what it *will* do rather than what it does, until a step exists that does it?
The wording was read as evidence twice in one session.
- Is there a check that a sentence the mesh prints corresponds to something that happened? This is the
second time in one session that a printed claim and the machine disagreed.
- **Why did the step not take effect?** It is called only when the apply had no failures, and being
skipped is silent. The mesh's table is loaded by a service in the same apply, so whether it was
loaded when the step asked is an ordering question — and ADR 0100 makes loading it first a
precondition rather than an expectation.
- **A step that records the mesh as having done what an operator did is worse than the omission.** The
record now says the mesh disabled ufw. Nothing distinguishes "we did this" from "we found it already
so". Should it?
- Why do the host's own detail lines not reach the journal? Everything it decided during the flip is
unrecoverable, which is why this account has candidates instead of a cause.
@@ -0,0 +1,70 @@
---
status: located
opened: 2026-09-29
located-in:
- mesh-host internal/apply/opening.go
- mesh-controller cmd/mesh-controller (the converge preview)
fixed-by:
amended-design:
---
# 144 — A predecessor's rules outlive the firewall the mesh found, and the mesh cannot see them
## What was observed
The mesh reports one thing about a machine's existing filtering: `firewall found: ufw`. On the
control-node, ufw was never what filtered the traffic that mattered.
Measured on 2026-09-29, before the machine was converged:
- ufw filters connections *to the machine*. It does not filter connections to a container's published
port, which arrive on the forwarded path where the container runtime accepts them before ufw's
forward chains are reached. Around thirty ports were published that way.
- Every one of the mesh's own forwarded openings, converged through ufw, had matched **zero packets** —
fifty rules in that chain, none ever matched, while the chain itself had passed 1.6 million
established packets. The restrictions read as applied and were inert.
- What actually kept those ports off the internet was a chain the predecessor installed in the
container runtime's own pre-accept hook, allowing the deliberately public ports and the private
ranges and dropping the rest on the outward link. Confirmed from outside: the proxy answered, the
container manager did not.
- That chain exists only in the running kernel. The persisted rule file is the distribution's empty
default, and nothing on disk recreates the chain.
After the flip, the mesh's own filter is loaded and does cover the forwarded path, so the machine no
longer depends on that chain. But **the chain is still there**, and it is now the only thing refusing
two ports the mesh believes are open: the bus and the registry, which
[ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md) requires be
reachable from anywhere so a machine can enrol and pull before it has a private-network address. The
mesh's rendered filter accepts both from anywhere. From outside, both are refused.
## Why it matters beyond this instance
**"The firewall found" is a kind, and filtering is not all in one place.** The host identifies one
front-end and reports it. A machine can carry rules from several sources — the front-end's own, the
container runtime's, an intrusion-prevention chain, and whatever a predecessor installed directly —
and the mesh's account of what filters the machine names exactly one of them.
**So adoption's central promise was half-true in both directions.** What the mesh converged through
the found firewall on the forwarded path did nothing at all, and what did the work was invisible to it.
A machine was reported as filtered by a mechanism that was not filtering.
**And convergence cannot retire what it cannot see.** Even once
[issue 143](../143-converging-does-not-retire-the-firewall-it-found/00-report.md) is fixed and the found
firewall is disabled, this chain remains, silently narrowing the machine below what the mesh's own
filter says. A rule the mesh did not write, cannot list, and will not remove — which today breaks the
enrolment path the design guarantees.
**The safe direction is not the same as the correct one.** Being more closed than intended broke nothing
visible, which is exactly why it went unnoticed for as long as the mesh has been on this machine.
## Open questions
- Should the host report every place the machine filters from, rather than one kind — the front-end,
the runtime's hooks, and any chain it does not recognise, named so a person can look?
- What should the mesh do about rules it did not write and does not understand? Reporting them seems
right; removing them cannot be, and leaving them silent is what produced this.
- Does an opening converged through a found firewall need a check that it can actually take effect?
Fifty rules matching nothing would have been visible from the counters at any point.
- Is the bus and the registry being reachable from anywhere still what the mesh wants on a machine that
faces the internet? The design says yes, for enrolment. It deserves asking on its own rather than
being answered by a leftover.