Issue 146: what made it slow, and what was changed so it is not

Issue 146: a first node now enrols, and is enrolled twice
Two more faults behind the three already fixed. The account a token is the password of was never recorded, and the comment above the issuing code said it was; issuing now records it. Placing the composed list at genesis is the other half — the control plane says what it composed and whoever raises the machine writes it beside the bus, because no declaration can reach a machine that has not enrolled. With that a first node enrols. It is then enrolled twice from one attempt, each minting a credential, and it keeps the answer to the first while the mesh keeps the second. The trail for that one stops at a duplicate that survived message-id deduplication.
2026-09-29 17:45:25 +02:00 · 2026-09-29 17:36:40 +02:00 · 2026-09-29 14:06:41 +00:00 · 2026-09-29 15:42:32 +02:00 · 2026-09-29 15:25:58 +02:00 · 2026-09-29 15:02:37 +02:00
19 changed files with 1501 additions and 7 deletions
@@ -99,6 +99,31 @@ composed, so it is not certified.
 binding. The per-node source override becomes its reach, widened from the filter alone to the names
 and the certificate as well.

+## Progressive insight — 2026-09-29, from building it
+
+**Reach does not mean the same thing to the filter for an endpoint the proxy serves.** The decision
+above says `internal` means "the filter opens the machine port to the private network" and `public`
+means "the filter opens it to anywhere". For a routed endpoint the second half is wrong, and
+[ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md) already said so before this
+record was written: *a public service is exposed through the proxy, not by opening its own port* — it
+listens `from: mesh`, only the proxy reaches it, and it is exposed by name.
+
+Found by trying to express one real module, not by review. Its routed name must be public, because
+browsers post to it; its machine-side port must not be, because that port serves the dashboard in
+cleartext. Under one value driving both, saying "public" would have reopened a port an operator had
+just closed. Measured the same evening: that module's routed name answered from the internet over TLS
+while its machine-side port was refused from the same place. The port is not the path.
+
+So the reach of a **routed** endpoint asks for names, and its port keeps what the manifest said. The
+reach of an **unrouted** endpoint — git over ssh, a mail port, the bus — governs the port, because
+there is no name and the port is the only way in. That is the same split this record already draws in
+*an endpoint that is not routed is reached but never named*; what it got wrong was carrying the filter
+across it.
+
+This corrects a fact, not the decision: one statement per endpoint, three things derived from it and
+none of them deciding on its own, all stand. The table in the decision should be read with the filter
+column applying to an unrouted endpoint.
+
 ## Consequences

 - **A manifest gains endpoint names, and a route contribution names an endpoint instead of a port.**
@@ -0,0 +1,158 @@
+---
+topic: the mesh
+status: accepted
+date: 2026-09-29
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0141-the-host-delivers-its-own-successor.md
+---
+
+# 142. The mesh delivers its own components as binaries, not as container images
+
+## Context
+
+Measured on the control-node, 2026-09-29:
+
+| what | how it runs | publishes |
+|---|---|---|
+| host | a binary on the machine | — |
+| controller, catalogue, builder, vault | containers | nothing |
+| store, registry, broker | containers | ports |
+
+**The mesh's own software is delivered two ways, and the difference is not a property of the
+software.** The host and the controller are both written in the same language, both the mesh's own,
+both doing the mesh's own work. One is an image fetched from a registry. The other is a file somebody
+copied to four machines, owned by no package, built by nothing
+([issue 142](../04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/00-report.md)).
+
+**The reason is not a judgement about either, it is that images are the only delivery that works.**
+There is no way to put a binary on a machine. The host is hand-copied because of that, and the
+controller is an image because of that. Neither was chosen on its merits.
+
+What it costs, all of it measured rather than argued:
+
+- **Genesis must raise a container runtime before the control plane can exist.** The bundle carries
+  three images and one of them is the controller, *"in the bundle for the same reason they are: there
+  is nothing to fetch it with yet"*
+  ([design 07](../03-DESIGN/01-to-be/07-the-foundation.md)). So the hardest moment in the mesh's life
+  has a prerequisite that the thing being started does not need.
+- **Updating the control plane depends on the control plane.** Its image is fetched from the registry,
+  which is a container the controller manages.
+- **A change to the host cannot be rolled out at all.** Every machine here runs a byte-identical
+  hand-copied binary. A change merged yesterday reached none of them.
+- **Compiling the language the mesh is written in is not a capability of the builder.** The bundle
+  toolchains are typescript — real, with a registered base module — and python, which is named in the
+  list and absent from the catalogue. The controller is built as an image from a Dockerfile, which is
+  the per-repository incantation the bundle toolchain exists to abolish
+  ([design 18](../03-DESIGN/01-to-be/18-building-a-module.md)).
+
+The half that *receives* a binary safely is already built and tested
+([ADR 0141](0141-the-host-delivers-its-own-successor.md)): versions side by side in directories named
+for them, the newest run, the running one standing aside between reconciles, retirement keeping the
+predecessor, and a rollback that chooses a directory. What is missing is everything that puts one
+there.
+
+## Considered Options
+
+1. **Leave it as it is.** Rejected: it is not a design, it is the reach of one mechanism. And it is
+   what makes a host change undeliverable.
+2. **Containerise the host too**, so everything is delivered one way. Rejected: the host is what
+   starts the container runtime and what applies containers. A host in a container is the bootstrap
+   problem made total, and the machine would have no way back from a bad one.
+3. **Deliver the mesh's components as operating-system packages.** Rejected for the reason
+   [ADR 0141](0141-the-host-delivers-its-own-successor.md) rejected it for the host: a package and a
+   trusted repository per operating system, three of each, and the `package` resource asserts presence
+   and deliberately never a version.
+4. **Binaries for the mesh's own components, containers for third-party software.** Adopted.
+
+## Decision
+
+**The mesh's own components are delivered as binaries on the machine.** The host, the controller, the
+catalogue, the builder, the vault — the software this project writes. They are delivered by the
+mechanism [ADR 0141](0141-the-host-delivers-its-own-successor.md) built: an archive, fetched by
+digest, unpacked into a directory named for its version, with the running one standing aside between
+reconciles and a rollback that chooses the predecessor.
+
+**Third-party software stays a container.** The store, the registry, the broker. They are somebody
+else's build, they are already adopted as modules
+([ADR 0078](0078-the-store-and-broker-are-modules.md)), and an image is the right way to carry
+somebody else's software. **The container runtime remains required** — modules use it — so this
+removes a dependency from the control plane, not from the machine.
+
+**The builder compiles the languages the mesh is written in.** A toolchain for Go, with a base module
+providing the compiler, exactly as typescript has. The obligation the toolchain list warns about — an
+SDK carrying the broker client, the envelope and tool serving — attaches to a *module* written in a
+language, not to the language being compilable. None of these components is a module in that sense;
+the host is what applies modules.
+
+**An artifact says what it targets.** A compiled binary is per operating system, pinned at link time
+([ADR 0005](0005-the-node-host.md)), and a toolchain deliberately takes nothing from the module,
+because anything a module could override there it would be writing a Dockerfile to override. So the
+target is a property of the artifact rather than of the recipe, and one artifact declared per target
+is one build each.
+
+**A component's version comes from where it sits, not from its linker.** It is unpacked into a
+directory named for its version, so it can read its own version from its path. The stamp goes, and
+with it the need for a build to know what it will be called.
+
+**Genesis carries a binary reference where it carried an image reference.** The principle does not
+change — the bundle names a thing by digest and the host fetches it, pinned because nothing can
+resolve a version when no mesh exists — and the container runtime stops being a prerequisite for the
+control plane. It stays a prerequisite for the store and the broker, which is where it belongs.
+
+**The order is staged, and each step stands alone.** Compiling Go; an artifact naming its target;
+delivering a binary; the host as the first component delivered; the controller, catalogue, builder and
+vault out of their containers; genesis last. Genesis is last for the reason it is always last: it
+matters for a machine nobody has yet, and every earlier step is provable on a mesh that exists.
+
+## Consequences
+
+- **One delivery for the mesh's own software**, so a change to the host ships the way a change to the
+  controller does, and neither is copied by hand.
+- **The control plane stops depending on a container runtime and on its own registry.** Both remain on
+  the machine for other reasons; neither gates the control plane's own life any more.
+- **`Replaced()`, the known-good record and the launcher's rollback stop being dead code.** They were
+  written for this and have been called by nothing but their tests.
+- **Four more components gain a rollback they do not have.** Today a bad controller image is recovered
+  by an operator; under this it is recovered the way a bad host is.
+- **Two versions of each component occupy disk.** Around nine megabytes each. The predecessor is what a
+  rollback needs.
+- **Genesis gets smaller, not larger.** One fewer image to carry and one fewer runtime to raise before
+  the control plane.
+- **This does not make the components smaller or simpler.** They are the same programs; what changes is
+  how they arrive. A reader expecting the containers to have been hiding complexity will not find any.
+- **What got harder:** the builder gains a language, artifacts gain a target, and the mesh gains a
+  second kind of thing it must deliver correctly — one where getting it wrong takes the control plane
+  down rather than a module. That is why the host is first: it is the component whose recovery is
+  already built and tested.
+
+## How it is checked
+
+- **A component is delivered and runs, with nothing copied by hand.** A bed builds the host from its
+  repository, delivers it to a machine running an older one, and the machine reports the new version.
+  This fails today at the first step, because nothing builds it.
+- **Each target is built once and only the matching one is delivered.** Asserted by declaring an
+  artifact per operating system and checking that a machine is offered the one it can run — a host
+  built for another is what ADR 0005's link-time pin exists to refuse.
+- **A component reads its version from its path**, asserted by unpacking the same bytes into two
+  differently named directories and seeing each report its own.
+- **A bad component is rolled back without an operator**, for the host first: a version that will not
+  start is replaced by its predecessor once, and the second failure halts naming the machine.
+- **The control plane comes up with no registry reachable**, which is the dependency this removes —
+  asserted by raising it with the registry stopped.
+- **Genesis raises a control plane with no container runtime running**, and raises the store and the
+  broker afterwards. Last, and on a machine with nothing on it.
+- **A published port count that does not change.** The mesh's own components publish nothing today, so
+  moving them out of containers must not open anything — asserted on the machine's reachable set before
+  and after, which the converge preview already reads.
+
+## References
+
+- [ADR 0141](0141-the-host-delivers-its-own-successor.md) — the receiving half, already built
+- [ADR 0005](0005-the-node-host.md) — the host, its supervision, and one binary per operating system
+- [ADR 0078](0078-the-store-and-broker-are-modules.md) — why third-party software stays a container
+- [ADR 0006](0006-the-substrate-and-the-control-plane.md) — what genesis must raise, and in what order
+- [issue 142](../04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/00-report.md) — the
+  measurement that started this
+- [design 07](../03-DESIGN/01-to-be/07-the-foundation.md) — the bundle's three images, one of them the
+  controller
@@ -0,0 +1,135 @@
+---
+topic: what runs on it
+status: superseded
+date: 2026-09-29
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0010-delivery.md
+superseded-by: 02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md
+---
+
+# 143. A consumer verifies the grant it is given
+
+## Context
+
+A **grant** is what the mesh writes on a consumer's machine so it can reach a provider. The real one
+the forge receives for its database, as it arrives:
+
+```
+provision  postgres-database
+at         <the provider's machine, by name>
+port       the machine port the provider is published on
+as         the role the provider created for this consumer
+```
+
+with the credential sealed in a separate file. Four facts and a password, and they are the whole
+mechanism by which anything in the mesh reaches anything else.
+
+**The mesh asserts that claim and never finds out whether it is true.**
+[Issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md):
+converging a machine dropped the path from a container to a port on its own machine, and for eleven
+hours the mesh answered *all doing what they were told, all heard from, every module current with its
+source* while a web application logged, six thousand times:
+
+```
+connection to server at "<the machine>" (10.10.0.1), port 6852 failed: timeout expired
+```
+
+Every check the mesh makes passed, because every check it makes is about the relationship between the
+mesh and a machine: the declaration was applied, the digest matched, every container named was running.
+None of them asks whether a consumer can reach what it requires — though the mesh composed the grant
+and therefore knows the consumer, the machine, the address, the port and the credential.
+
+**And where the check runs decides whether it catches anything.** The rule in force admitted the
+machines' own addresses on the private network. A dial from the *machine* to its own address carries
+exactly such a source address, so a check run by the host on its own behalf would have matched that rule
+and passed — while every container on the machine was refused. This is inference from the rule that was
+loaded, not a measurement: the fault was found and fixed before anyone thought to dial from the host.
+It is enough to decide the question, because a check whose position differs from the consumer's is
+testing something nobody asked about.
+
+## Considered Options
+
+1. **The control plane dials each provision.** Rejected, and it is the tempting one because the control
+   plane holds every fact. It sits on the provider's machine for most provisions here and reaches the
+   address by a path no consumer uses; in the measured outage it would have passed throughout.
+2. **The host dials on the consumer's behalf, from the machine.** Rejected for the reason above: the
+   machine's network position is not the consumer's, and the one outage this exists to catch is exactly
+   a difference between them.
+3. **Ask the module.** Rejected: a module is arbitrary software that the mesh does not write. Some could
+   report on their provisions and most cannot, and a check that covers the modules that opted in tells
+   nobody anything about the rest.
+4. **Read the module's logs.** Rejected: the failure was in a log the whole time, and reading a module's
+   logs makes the mesh depend on the wording of software it does not control.
+5. **The consumer verifies it, from its own network position.** Adopted.
+
+## Decision
+
+**A consumer verifies each grant it is given, from its own network position.** After a reconcile has
+applied a grant, the machine opens a connection to the address and port that grant names, from inside
+the consumer's own network namespace — the same position the consumer's software dials from, which is
+the only position that answers the question the grant asks.
+
+**It is a connection, not a conversation.** Whether the port accepts a connection is what a grant
+claims; whether the credential is right, the role exists or the schema is current is the provider's to
+answer and the consumer's to discover. A check that spoke each provision's protocol would be a second
+implementation of every provision, and would fail for reasons that are not the mesh's.
+
+**One failure is not news.** A provider restarting is ordinary, and so is a consumer between containers.
+A grant is reported unreachable only after it has failed on **consecutive** reconciles, and the count is
+what the machine reports rather than the last attempt — so a reader can tell "it was briefly away" from
+"it has never worked".
+
+**A grant that cannot be checked is said to be unchecked, never assumed good.** A consumer that is not
+running has no network position to dial from; that is not a broken grant and must not read as one. It is
+also not a verified grant, and the two are different sentences.
+
+**What it costs to be wrong is the constraint on all of it.** A check that reports a working provision
+broken trains a reader to ignore the report, which is worse than having none — the fault this
+repository keeps finding, one level up. So the threshold is consecutive failures, the check is the
+cheapest thing that answers the question, and an unknown is reported as unknown.
+
+**The mesh says it where it says everything else.** A machine's report carries its unreachable grants,
+and `status` names them beside what is out of date — so "every module current with its source" stops
+being the whole of what the mesh will tell you about a machine whose modules cannot reach each other.
+
+## Consequences
+
+- **The mesh can be wrong out loud.** It has been able to assert a grant and not check it; now a grant
+  that does not work is a thing the mesh says, and the eleven hours of issue 145 become minutes.
+- **The host gains the ability to act from a container's network position**, which it has not needed
+  before. That is a real capability and the only one this needs.
+- **A machine reports something that is not about the declaration.** Everything it reports today is
+  what it applied and what it holds; this is the first thing it says about whether what it applied
+  works.
+- **A provision with no port is not checked**, because there is nothing to dial. Several are files and
+  secrets, and saying "checked" about those would be the appearance of verification that this record
+  exists to remove.
+- **What got harder:** a reconcile does more than apply. Every grant adds a connection attempt on a
+  cadence, which is cheap individually and worth naming: a machine with many consumers dials once per
+  grant per reconcile.
+
+## How it is checked
+
+- **The outage is caught.** A bed drops the path from a consumer's network position to a provider's
+  port while leaving the machine's own path to it open — the exact shape of issue 145 — and the grant
+  reads unreachable. This fails against the previous behaviour, where nothing reported anything, and
+  against a check run from the machine, which passes while the consumer cannot reach it.
+- **A restarting provider is not an outage.** One failed reconcile reports nothing; the count rises and
+  falls, and the grant reads reachable again without anybody acting.
+- **A consumer that is not running reads unchecked, not broken**, asserted separately from the
+  unreachable case because they are different sentences.
+- **A provision with no port is not claimed to be checked.**
+- **The report carries the count, not the last attempt**, so "briefly away" and "never worked" are
+  distinguishable by a reader who sees only the report.
+- **`status` names an unreachable grant**, asserted on the output, since a check nothing surfaces is
+  the same as no check.
+
+## References
+
+- [ADR 0010](0010-delivery.md) — the declaration is owned resources; a grant is one of them
+- [ADR 0009](0009-modules-and-the-graph.md) — what a provision and a consumer are
+- [issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)
+  — the eleven hours
+- [issue 136](../04-ISSUES/136-a-module-may-name-a-program-the-machine-does-not-have/00-report.md) — the
+  same distance between a declaration and a machine, one level down
@@ -0,0 +1,121 @@
+---
+topic: what runs on it
+status: accepted
+date: 2026-09-29
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md
+supersedes: 02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md
+---
+
+# 144. Anything on a machine may call anything on it, and that is the whole of "local"
+
+## Context
+
+Everything in the mesh should be able to call:
+
+- what runs on the same machine;
+- another machine's service over the private network, if that service is exposed there;
+- another machine's service over the public network, if it is exposed there.
+
+Three cases. The filter had two of them.
+
+**The first was broken and the break was invisible.** A service exposed to the private network rendered
+as the machines' own addresses on it. A caller on the machine carries such an address; a caller inside
+one of that machine's containers carries a bridge address and matched nothing. Measured:
+
+```
+the machine:   local 10.10.0.1 dev lo   src 10.10.0.1
+a container:   10.10.0.1 via 172.17.0.1 dev eth0  src 172.17.0.8
+```
+
+Same destination, same machine, two source addresses. The rule named the first and silently refused the
+second, so a module reaching its database on its own machine's name timed out for eleven hours
+([issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)).
+
+**The second case works, and by accident.** A caller on another machine reaches the private network over
+the tunnel, and arrives carrying that machine's own address — so the rule matches. It would not have
+matched the caller's own address either; the tunnel rewrites it. That two of three cases worked is why
+this looked correct.
+
+**[ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) answered the wrong question.** Written
+hours earlier, it proposed that a consumer verify each grant it is given by opening a connection from
+its own network position — and it went to some length about *which* position, because whether a caller
+sat in a container changed the answer. That difference was the bug. A verification mechanism would have
+reported this outage sooner and would not have prevented it, and the machinery it needed existed only
+because the rule was wrong. The remedy for a configuration error is the correct configuration.
+
+**And a module is not a container.** A module is software that delivers one or more services, and it may
+do that as a container, an installed package with a unit, a binary, or files something else reads. Of 72
+modules in the catalogue, 61 happen to use a container and 11 do not — among them the resolver, the ssh
+daemon and the intrusion-prevention module. A rule that reasons about containers describes most of the
+mesh and not the mesh.
+
+## Considered Options
+
+1. **A line per service admitting the machine's own callers.** Rejected: it is what was written first,
+   and it only ever covers the services somebody remembered to think about. It also states, service by
+   service, a thing that is true of the machine.
+2. **Verify each grant from the consumer's position** ([ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md)).
+   Rejected as a remedy: it observes the fault rather than removing it, and the question it agonised over
+   — which network position — exists only while the fault does.
+3. **Enumerate the addresses a machine's callers may have.** Rejected for the reason no address is named
+   anywhere in this filter any more ([ADR 0140](0140-the-filter-constrains-what-arrives-from-outside.md)):
+   a range describes one machine and goes stale in silence.
+4. **Local is not filtered, stated once.** Adopted.
+
+## Decision
+
+**Anything on a machine may call anything on that machine, and the filter says so once.** Not per
+service, not per port, and not by naming who the callers are: traffic that did not arrive from outside
+the machine and did not arrive over the private network is the machine's own, and is admitted. It is
+asked by the link the traffic arrived on, because that is a fact about the machine rather than a list
+that describes one.
+
+**Local is not a boundary this mesh draws.** Whether a caller is a container, a unit, or the operator's
+shell changes nothing, because the thing being decided is "is this the same machine" and the answer does
+not depend on the form the caller takes.
+
+**The other two cases are unchanged and are now legible beside it.** A service exposed to the private
+network admits the machines on it; a service exposed publicly admits anything. Three cases, three lines,
+and a reader can see all three at once.
+
+**[ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) is superseded and nothing replaces it.**
+Whether the mesh should check that a grant works is a real question — it reported this machine healthy
+for eleven hours — but it is a question about what the mesh can say, not about what it should do, and it
+must stand on its own rather than as the remedy for a rule that was wrong. It is not built.
+
+## Consequences
+
+- **The three things everything should be able to call are three lines**, and the first is one line
+  rather than one per service, so a service added tomorrow is reachable locally without anybody
+  remembering to say so.
+- **A form of module stops mattering to the filter.** The 11 modules that are not containers were never
+  affected by this bug and were never the reason it was hard to see; they are the reason the rule should
+  never have mentioned containers.
+- **The mesh still cannot say when a grant stops working.** That is the live gap, recorded in issue 145
+  and no longer pretending to have an answer.
+- **What got harder:** nothing. This removes a line per service and replaces it with one.
+
+## How it is checked
+
+- **A caller on the machine reaches a service on it, in the input chain**, asserted on that chain's own
+  body — because the forward chain carries the same line in the same words, and an assertion on the
+  whole rendered file passed with the input chain's copy deleted. That is what
+  [ADR 0137](0137-a-machine-says-which-networks-it-routes.md)'s tests already say to do.
+- **It is one rule, not one per service.** Asserted by rendering two services of different reach and
+  refusing a per-port local line.
+- **The three reaches render as three lines**, asserted together, so the whole of what the filter says
+  about who may call what is one test.
+- **The measured case:** from a container on the machine, a service exposed to the private network on
+  that machine answers. This is the outage, and it fails against the rule this replaces.
+
+## References
+
+- [ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md) — the filter is the sum
+  of what its modules listen on
+- [ADR 0140](0140-the-filter-constrains-what-arrives-from-outside.md) — why no address is named
+- [ADR 0138](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md) — internal and public,
+  the other two cases
+- [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) — superseded here
+- [issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)
@@ -0,0 +1,120 @@
+---
+topic: what runs on it
+status: superseded
+date: 2026-09-29
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md
+superseded-by: 02-DECISIONS/0146-connectivity-is-checked-by-name-per-hosting-form.md
+---
+
+# 145. A module checks what the mesh claims is reachable, and it checks itself
+
+## Context
+
+The mesh asserts three things are callable ([ADR 0144](0144-anything-on-a-machine-may-call-anything-on-it.md)):
+what runs on the same machine, another machine's service exposed to the private network, and another
+machine's service exposed publicly. It has never checked any of them.
+
+[Issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md):
+the first of the three was broken for eleven hours and the mesh answered *all heard from, every module
+current with its source* throughout. Every check it makes is about the relationship between the mesh and
+a machine — applied, current, containers running — and none about whether anything can reach anything.
+
+**A first answer was drafted and withdrawn.** [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md)
+put the check inside the host, verifying each grant from the consumer's network position. It was
+superseded because the difference it worked so hard to reproduce — whether a caller sat in a container —
+was the bug itself. What survives from it is the part that was right: a check run from the wrong place
+proves nothing, and the mesh's own reports are not evidence about the network.
+
+**The mesh already has the shape for this and it is a module.** A module can declare a container that
+runs on a cadence ([ADR 0053](0053-a-step-that-runs-on-a-schedule.md), and three modules already use
+`*/5 * * * *`), can be given the mesh's roster as a rendered fact — every machine's name, address and
+this node's own identity, the same mechanism the resolver and the operator's ssh configuration use — and
+can emit what it found on the bus. Nothing new is needed to build this except the module.
+
+**What it must not check is the trap.** The obvious probe target is ssh: present on every machine, never
+closed by design. Dialling it would have passed throughout the outage, because ssh is admitted
+unconditionally and the thing that broke was a service exposed to the private network. A checker whose
+probe is unconditionally open measures the one path that cannot fail, which is the failure this whole
+sequence keeps producing — a check that reads as verification and verifies nothing.
+
+## Considered Options
+
+1. **The host verifies each grant** ([ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md)).
+   Superseded. It needed the host to act from another network position, which is machinery that exists
+   only while local calls are filtered wrongly.
+2. **The control plane dials every node.** Rejected: it sits on one machine and reaches the others by a
+   path no ordinary caller uses. It would have passed throughout the outage.
+3. **Probe an existing service.** Rejected for the target problem above: the services guaranteed on every
+   machine are the ones that are never closed, so they cannot fail the way the mesh fails.
+4. **A module on every machine that serves its own probe and dials the others'.** Adopted.
+
+## Decision
+
+**A module runs on every machine, serves an endpoint of its own, and dials every other machine's.** The
+probe is the module's own endpoint, declared reachable over the private network — so the thing being
+dialled is admitted by exactly the rule that governs every other internally-exposed service, and fails
+when that rule is wrong. A second endpoint, declared public, does the same for the public path where a
+machine has one.
+
+**It checks the three cases the mesh claims, by name:**
+
+- its **own machine**, by dialling its own machine's address — the case that broke, and the only one that
+  distinguishes a caller on the machine from a caller in one of its containers;
+- **each other machine over the private network**;
+- **each machine's public path**, where one is recorded.
+
+**It resolves before it dials, and says which failed.** A name that does not resolve and a port that does
+not answer are different faults with different owners, and a checker that reports one sentence for both
+sends a reader to the wrong place.
+
+**It runs where the callers run.** The module's own code in its own container, on the cadence the mesh
+already has, from the same position as every other module on that machine. It is not the host and not the
+control plane, and that is the whole point.
+
+**It says what it found and nothing else.** It emits results; it repairs nothing, opens nothing and holds
+no credential beyond its own. A checker that fixes things is a second control plane.
+
+**One failure is not a fault.** A machine rebooting is ordinary. A path is reported broken after it has
+failed on consecutive runs, and the count travels with the result so a reader can tell "briefly away"
+from "never worked" — the one thing [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) got
+right and worth keeping.
+
+## Consequences
+
+- **The mesh gains the ability to be wrong out loud about the network.** Eleven hours becomes two runs.
+- **It is a module, so it is assigned, built, pushed and reported on like everything else** — no new host
+  capability, no new vocabulary, nothing in the control plane that has to know about checking.
+- **Its own endpoint is the instrument.** That is what makes it able to fail; it also means the checker
+  must be assigned to a machine before that machine can be checked, and a machine without it is
+  unchecked rather than healthy.
+- **It cannot check what it cannot be told.** The roster gives it machines; it does not give it every
+  module's endpoints, so this checks the paths the mesh claims and not every grant in the mesh. That is
+  the honest scope of a first one, and the difference is worth saying rather than growing quietly.
+- **What got harder:** one more module on every machine, and a module whose whole purpose is to fail
+  visibly when something else is wrong. Its own failures will be read as the mesh's, which is the cost of
+  an instrument.
+
+## How it is checked
+
+- **It catches the measured outage.** A bed closes the path from a container to a service exposed to the
+  private network on its own machine — issue 145's shape — and the checker reports its own machine
+  unreachable while every other path still reads reachable. This fails against a probe on a port that is
+  never closed, which is the wrong target this record exists to name.
+- **A machine rebooting is not a fault**: one failed run reports nothing, the count rises and falls.
+- **A name that does not resolve is reported as that**, not as a port that did not answer.
+- **It reports and does not act**: asserted by giving it a broken path and checking nothing on the machine
+  changed.
+- **A machine without the module reads unchecked**, never healthy — asserted on what the mesh says about
+  a machine it is not assigned to.
+
+## References
+
+- [ADR 0144](0144-anything-on-a-machine-may-call-anything-on-it.md) — the three things that must be callable
+- [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) — superseded; what survives is that the
+  position matters
+- [ADR 0053](0053-a-step-that-runs-on-a-schedule.md) — the cadence
+- [ADR 0138](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md) — internal and public,
+  which the probe endpoints declare
+- [issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)
@@ -0,0 +1,125 @@
+---
+topic: what runs on it
+status: accepted
+date: 2026-09-29
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0145-a-module-checks-what-the-mesh-claims-is-reachable.md
+supersedes: 02-DECISIONS/0145-a-module-checks-what-the-mesh-claims-is-reachable.md
+---
+
+# 146. Connectivity is checked by name, per hosting form, with a valid certificate
+
+## Context
+
+[ADR 0145](0145-a-module-checks-what-the-mesh-claims-is-reachable.md) decided that a module checks what
+the mesh claims is reachable, from where the callers are, because the mesh reported four machines healthy
+for eleven hours while a module could not reach its database
+([issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)).
+That decision stands. What it got wrong is everything about *what* is dialled.
+
+It dialled a raw port on each machine's address. Three things are wrong with that:
+
+- **A raw port is not how anything in this mesh is reached.** A real caller resolves a name, the proxy
+  answers it, and the proxy reaches the service. A check that dials a port tests the last hop of a path
+  with four hops in it, and the three it skips — resolution, the proxy, the certificate — are where most
+  of the mesh's connectivity actually lives.
+- **It tested one hosting form.** A module is software that delivers services, and it may deliver them
+  from a container, from a unit the mesh writes for its own code, or from a unit a package ships. Those
+  are three different paths to the same machine, and the outage that produced this was two of them
+  disagreeing. A probe served one way measures one way.
+- **It said nothing about certificates.** An internal name that resolves, routes and answers over TLS
+  that nothing can verify is not a working path; it is a working path for whoever holds the proxy's
+  trust and nobody else.
+
+## Decision
+
+**Each hosting form gets its own endpoint, its own route and therefore its own name.** On every machine:
+
+| name | what serves it |
+|---|---|
+| `connect-docker.<node>.internal` | a container |
+| `connect-process.<node>.internal` | the mesh's own code, in a unit the mesh writes |
+| `connect-unit.<node>.internal` | a unit a package ships |
+
+and the same set under each machine's public domain where it has one — `connect-docker.<domain>` and its
+siblings. The names are the instrument: a failure reads as *`connect-docker.g14.internal` did not answer*,
+which says which machine and which hosting form without anybody interpreting anything.
+
+**Every machine checks every machine, by name, over TLS, verifying the certificate.** Not a port, not an
+address: resolve the name, connect, complete the handshake, check the certificate against the authority
+that should have issued it — the mesh's own for an internal name, a public one for a public name. That is
+the whole path a real caller takes, and each step failing is reported as itself.
+
+**No name is written anywhere.** The machines come from the roster the mesh already renders as a fact, and
+the labels are the module's. A machine that joins appears in every other machine's roster on the next
+push, and they begin checking it without an edit.
+
+**And the module arrives on a machine because the machine exists, not because somebody assigned it.** A
+machine that joins and does not have it is worse than unchecked: every other machine is already dialling
+its names, so it reads as broken everywhere until someone notices. This is the part the mesh cannot
+currently express — see below — and it is the part that makes the rest safe.
+
+**What survives from 0145**, unchanged: it reports and repairs nothing; one failure is not a fault and a
+path is broken after consecutive runs with the count travelling with the result; findings are said on the
+bus, because a finding in a file on the machine is what this exists to end; and the bus is the one path
+that cannot report its own failure, so an emit that does not land is written locally and nowhere else.
+
+## What this needs that the mesh does not have
+
+Named here rather than assumed, because each is a decision of its own and this record is not the place to
+make them:
+
+1. **A module that every machine has.** `ScopeNode` means *at most one holder per node* — an exclusivity
+   rule, not an obligation — and nothing assigns a module at enrolment. Today the resolver, the packet
+   filter, ssh and intrusion prevention are each assigned per machine by hand, which is the same gap
+   wearing different clothes.
+2. **A container running a module's own bundle.** A `process` runs the mesh's own compiled code with no
+   image; a `container` needs an image of the module's own, which means a Dockerfile — the thing the
+   `bundle` artifact exists to abolish. Nothing in the catalogue runs a bundle in a container, so
+   `connect-docker` has no shape yet.
+3. **A unit a package ships, for `connect-unit`.** The `service` resource puts an existing unit into a
+   state and deliberately installs none, so this form needs a package that serves a port — and naming a
+   program the machine may not have is
+   [issue 136](../04-ISSUES/136-a-module-may-name-a-program-the-machine-does-not-have/00-report.md).
+4. **A machine's public domain in the roster fact.** The fact carries each machine's name, mesh name,
+   address and operator account. The public names cannot be composed without the domain.
+5. **Something that installs the mesh's own root on a machine.** This is
+   [issue 129](../04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md), open
+   since before any of this. Until it is closed, every internal name will fail certificate verification
+   from every machine — correctly, because nothing can verify it. That is the checker working, and it is
+   worth saying in advance so the first run is not read as the checker being broken.
+
+## Consequences
+
+- **A failure names the machine and the hosting form.** That is the whole gain over a port: eleven hours
+  became two runs under 0145, and under this it also becomes one line that says where to look.
+- **The checker surfaces issue 129 immediately**, and will report every internal name unverifiable until
+  it is fixed. A reader must be told that before the first run rather than after.
+- **Five things must be built before this is what it says it is**, and until they are, what exists is a
+  port dial from one position — useful, and not this.
+- **What got harder:** a module with three hosting forms of the same trivial service is a strange thing to
+  read. It is justified only because those three forms are how the mesh actually runs software, and a
+  checker that tested one of them would keep the class of outage it exists to catch.
+
+## How it is checked
+
+- **A name per hosting form answers from every machine**, asserted by name and not by port.
+- **A certificate that does not verify is reported as that**, distinctly from a name that does not resolve
+  and a port that does not answer — three faults, three owners.
+- **A machine that joins is checked by every other machine without an edit**, asserted by adding one to a
+  bed and looking at what the others dial on their next run.
+- **A machine that joins has the module**, which is gap 1 above and is the assertion that cannot be
+  written yet.
+- **The measured outage is still caught**: the path from a container to a service on its own machine is
+  closed and `connect-docker.<that node>.internal` fails from that machine while the others still pass.
+
+## References
+
+- [ADR 0145](0145-a-module-checks-what-the-mesh-claims-is-reachable.md) — superseded; its core stands
+- [ADR 0138](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md) — the two reaches these
+  names come from
+- [ADR 0066](0066-public-routing-is-name-agnostic.md) — a label plus a domain, which is why no name is written
+- [issue 129](../04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md) — what the
+  internal names will fail on until it is closed
+- [issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)
@@ -0,0 +1,139 @@
+---
+topic: what runs on it
+status: accepted
+date: 2026-09-29
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md
+---
+
+# 147. A module anchors the mesh's authority on a machine, and takes it away again
+
+## Context
+
+The mesh runs its own certificate authority and every internal name is served with a certificate
+from it. No machine trusts it. On an enrolled, adopted workstation — on the private network,
+resolving through the mesh's resolver — every internal HTTPS name fails verification with
+*unable to get local issuer certificate*
+([issue 129](../04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md)).
+The certificates are genuine; nothing on the machine has ever been told what issued them.
+
+The authority's only consumer today is a proxy, which fetches the root into a directory of its own
+and hands it to one program ([ADR 0098](0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md)).
+That is enough for the proxy and for nothing else: a browser, `git` over HTTPS, `curl`, a package
+manager and every module that calls another module by an internal name read the machine's trust
+store, which holds the predecessor's authority and a developer tool's local root, and nothing of
+the mesh's.
+
+The predecessor wrote its root into every machine it set up. Removing it was deliberate — an
+honest failure beats a name that verifies for the wrong reason — and it leaves the mesh with no
+answer at all until this one lands. It is also what keeps the predecessor alive on the machines
+that still speak TLS to a mesh name.
+
+**What makes this a decision rather than a patch** is where the knowledge goes. Two mechanisms in
+the mesh already write things onto a machine because it is on the private network: `/etc/hosts`
+and the registry's plaintext trust ([ADR 0082](0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md)).
+Following that precedent, the controller would inject an anchor into every such machine's
+declaration, and issue 129 proposed exactly that. It would work. It would also put *where this
+operating system keeps trust anchors* and *which command refreshes its bundles* into the control
+plane, for a fact the control plane does not have (the root does not exist until the authority has
+run) and a machine that may have no reason to verify a mesh name at all.
+
+## Considered Options
+
+1. **The controller injects the anchor into every machine on the private network**, the
+   `/etc/hosts` and insecure-registry shape. Rejected: being on the network is what makes the
+   registry reachable, and that is why network presence is the right trigger *there* — the trust
+   and the reachability are the same fact. Trusting an authority is not the same fact as being
+   able to reach it, and the anchor's path and the bundle refresh are a property of the machine's
+   operating system, which is the host's half of the mesh, not the controller's.
+2. **A new host primitive — a `trust-anchor` resource type.** Rejected for now, not on principle.
+   The host's vocabulary should grow when a shape cannot be said with what exists, and this one
+   can: a file and a service already express it, as the packet filter proves
+   ([ADR 0140](0140-the-filter-constrains-what-arrives-from-outside.md), whose module writes a
+   unit file and a service and nothing else). The primitive becomes right the moment a second
+   operating system is in play, because the anchor directory and the refresh command are exactly
+   the difference `internal/system` exists to hold. Until then it would be a vocabulary word with
+   one speaker.
+3. **A module that requires the authority, fetches its root, installs it as an anchor and
+   refreshes the machine's bundles — and removes both when it is no longer assigned.** Adopted.
+
+## Decision
+
+**A machine trusts the mesh's authority because a module put its root there, and stops trusting it
+when that module is taken away.**
+
+1. **The module requires `internal-acme-ca`** and reads the provider's bound address and the path
+   it serves its root at. It requires nothing else and provides nothing: it is a consumer of the
+   authority like any other.
+2. **It fetches the root over the mesh's own network, without prior trust**, because there is no
+   prior trust to have — this is the module that establishes it — and the network is what
+   authenticates the fetch ([ADR 0098](0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md),
+   the same reasoning that lets the proxy fetch it). What it accepts is checked: a body that is
+   not a certificate fails, and the failure is the module's, not a later handshake's.
+3. **It installs the root where this machine's TLS clients look, and refreshes the extracted
+   bundles** — the command that does the refresh is an ordinary part of the unit that places the
+   anchor, not a new thing the mesh can be asked to do.
+4. **Removal is symmetric and is the same unit's business.** Undeclared, the host stops the unit;
+   stopping it removes the anchor and refreshes the bundles again. A machine that leaves the mesh
+   stops trusting the mesh, without anybody remembering to go and look.
+5. **It is an ordinary assignment.** No machine is given it automatically. A machine that verifies
+   a mesh name is assigned it, and a machine that does not is not — which is the same statement
+   the mesh already makes about every other module, and is why this is not the controller's
+   business.
+
+**One operating system, said out loud.** The anchor directory and the refresh command in the
+module today are Arch's. On a machine that is not Arch the unit fails, visibly, rather than
+writing a file nothing reads. That is the accurate failure, and it is the signal that option 2
+above has become right.
+
+## How this is checked
+
+- **The verification that could not succeed before.** On a machine holding the module, a plain
+  client fetches an internal HTTPS name with no `-k` and no bundle argument and verifies. On a
+  machine without it, the same fetch fails with *unable to get local issuer certificate*. Both
+  halves, because only the pair distinguishes "the anchor works" from "something else already
+  trusted it".
+- **The removal half, in the same bed:** unassign the module, refetch, and the failure returns.
+  Checking only the arrival is how a trust store fills up with authorities nobody can account for.
+- **What is deliberately not checked here:** that the authority issues, that a name resolves, that
+  the proxy serves. Those have their own beds, and this module's bed passing for those reasons is
+  the failure mode this record is most exposed to — which is why the negative half is not optional.
+
+**What this bed is dialled at, and why it is the authority itself.** The authority serves its own
+API with a certificate it issued, so the handshake under test needs nothing else in the mesh to be
+right. A trust bed that reached for a routed name through the proxy would be passing or failing for
+the proxy's reasons and the resolver's.
+
+**Written, and not yet run** *(2026-09-29)*. The bed is `trust-anchor` in the lab, and it cannot
+execute: raising a foundation fails before any module is reached, in both bundles that exist
+([issue 146](../04-ISSUES/146-the-foundation-cannot-be-raised-on-the-bus-the-mesh-runs-on/00-report.md)).
+So what stands behind this record today is the rendering — the script the machine would run names
+the authority it was bound to, checked in the control plane's own test suite — and **not** a machine
+that verified anything. That is a weaker thing than the paragraph above describes, and it stays
+written this way until the bed runs.
+
+## Consequences
+
+The predecessor's authority can be retired from a machine once this module is assigned to it,
+which is the first time that has been true. `git` over HTTPS to the mesh's forge starts working,
+so the ssh-only clone URL stops being a rule. A module on any machine can call another module's
+internal name and verify it.
+
+What got harder: one more module to assign to a machine that needs it, and the machine's trust
+store now changes when an assignment changes — which is the point, and is also a thing an operator
+can be surprised by. The fetch without prior trust is the same exposure ADR 0098 accepted, now on
+every machine that holds the module rather than only where a proxy runs: anything that can stand
+in the middle of the mesh's own network at the moment of the fetch can be believed. The mesh
+already treats that network as the thing it authenticates.
+
+## References
+
+- [issue 129](../04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md) —
+  the symptom and the evidence.
+- [ADR 0098](0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md) — a fact made at
+  first start is fetched from its provider; this extends it from one program to the machine.
+- [ADR 0082](0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md) — the precedent
+  this deliberately does not follow, and why it is right where it is.
+- [ADR 0005](0005-the-node-host.md) — the host is where one operating system's difference lives.
+- [`03-DESIGN/01-to-be/08-connectivity.md`](../03-DESIGN/01-to-be/08-connectivity.md).
@@ -142,6 +142,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0131** — [Everything on the mesh speaks to the broker seat, and AMQP is not a provision](0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)
 - **0132** — [A seat carries the tools its holder must serve](0132-a-seat-carries-the-tools-its-holder-must-serve.md)
 - **0134** — [The mesh says what it applied](0134-the-mesh-says-what-it-applied.md)
+- **0142** — [The mesh delivers its own components as binaries, not as container images](0142-the-mesh-delivers-its-own-components-as-binaries.md)

 ### Its tiers, from the bottom up

@@ -222,6 +223,11 @@ python3 00-META/checks/index.py            fail if stale
 - **0139** — [A network is forwarded because a module declared it](0139-a-network-is-forwarded-because-a-module-declared-it.md)  *(superseded)*
 - **0140** — [The filter constrains what arrives from outside, and says nothing about a machine's own guests](0140-the-filter-constrains-what-arrives-from-outside.md)
 - **0141** — [The host delivers its own successor, and versions live side by side](0141-the-host-delivers-its-own-successor.md)
+- **0143** — [A consumer verifies the grant it is given](0143-a-consumer-verifies-the-grant-it-is-given.md)  *(superseded)*
+- **0144** — [Anything on a machine may call anything on it, and that is the whole of "local"](0144-anything-on-a-machine-may-call-anything-on-it.md)
+- **0145** — [A module checks what the mesh claims is reachable, and it checks itself](0145-a-module-checks-what-the-mesh-claims-is-reachable.md)  *(superseded)*
+- **0146** — [Connectivity is checked by name, per hosting form, with a valid certificate](0146-connectivity-is-checked-by-name-per-hosting-form.md)
+- **0147** — [A module anchors the mesh's authority on a machine, and takes it away again](0147-a-module-anchors-the-meshs-authority.md)

 ### How it is built

@@ -11,8 +11,9 @@ code:
  - mesh-catalog modules/postgres
  - mesh-catalog modules/lavinmq
  - mesh-lab test/integration/mesh.test.ts (a bare machine becomes a mesh)
-updated: 2026-09-22
+updated: 2026-09-29
 decisions:
+  - 02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md
  - 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
  - 02-DECISIONS/0088-the-foundation-filters-before-anything-listens.md
  - 02-DECISIONS/0004-a-node-and-how-it-joins.md
@@ -314,4 +315,24 @@ of a database and pushed to over the broker. What arrived and what did not is th

 **One fault, and it was in the joining.** The token did not say what the mesh calls the machine,
 so enrolment needed a flag its own help said it did not — and failed at the broker with an empty
-username. Recorded in ADR 0004 as the fifth thing a token carries.
+username. Recorded in ADR 0004 as the fifth thing a token carries.
+
+## The mesh's own components arrive as binaries
+
+*2026-09-29 —
+[ADR 0142](../../02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md).*
+
+The bundle carries three images and one of them is the controller, *because there is nothing to fetch
+it with yet*. That reasoning holds and its conclusion changes: the controller is carried as a **binary**
+reference rather than an image reference, pinned by digest exactly as before. Nothing about the bundle's
+shape moves — it names a thing and the host fetches it — and the container runtime stops being something
+genesis must raise before the control plane can exist. It still raises one, for the store and the broker,
+which is where somebody else's software belongs.
+
+The mesh's own components — the host, the controller, the catalogue, the builder, the vault — are
+delivered as binaries into directories named for their versions, by the mechanism
+[ADR 0141](../../02-DECISIONS/0141-the-host-delivers-its-own-successor.md) describes. Third-party
+software stays a container. The split is not about isolation; it is about who built the thing.
+
+Measured before deciding it: the mesh's own components publish no ports at all, so this opens nothing.
+Only the store, the registry and the broker publish, and they are staying as they are.
@@ -7,8 +7,9 @@ code:
  - mesh-controller internal/identity/authority.go
  - mesh-host internal/identity/serving.go
  - mesh-host internal/apply (the service that reflects a rule set)
-updated: 2026-09-28
+updated: 2026-09-29
 decisions:
+  - 02-DECISIONS/0147-a-module-anchors-the-meshs-authority.md
  - 02-DECISIONS/0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md
  - 02-DECISIONS/0140-the-filter-constrains-what-arrives-from-outside.md
  - 02-DECISIONS/0104-a-provision-may-be-answered-by-an-adapter-to-the-predecessor.md
@@ -710,6 +711,22 @@ step, so when the authority moves the root is fetched again and the proxy is rec
 *How it is checked:* the route-forwarding bed installs the authority, the proxy and a consumer
 from the catalogue and asserts the routed name is served.

+**And a machine trusts that authority because a module put its root in its trust store**
+([ADR 0147](../../02-DECISIONS/0147-a-module-anchors-the-meshs-authority.md)). The proxy's fetch
+answers for the proxy and for nothing else: a browser, `git` over HTTPS, a package manager and
+every module calling another by an internal name read the machine's own trust store, and the mesh
+had never written anything there. A module requiring the authority does the whole of it — fetch
+the root over the mesh network, place it where this machine's TLS clients look, refresh the
+extracted bundles — and stopping it, which is what being unassigned does, takes the anchor away
+and refreshes them again. Not the controller's business, because being on the private network is
+what makes the authority *reachable* and is not the same fact as having a reason to *verify* a
+mesh name; and because where anchors live and which command refreshes them is one operating
+system's difference, which is the host's half of the mesh
+([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)).
+*How it is checked:* on a machine holding the module a plain client verifies an internal HTTPS
+name with no bundle argument, and on one without it the same fetch fails to find an issuer — both
+halves, because only the pair tells the anchor apart from something that already trusted it.
+
 ### What was built

 *2026-08-31.*
@@ -5,7 +5,7 @@ code:
  - mesh-controller internal/builder
  - mesh-controller cmd/mesh-controller (build, build --behind, push, status)
  - mesh-controller internal/inventory/builds.go
-updated: 2026-09-21
+updated: 2026-09-29
 decisions:
  - 02-DECISIONS/0090-a-failure-that-repeats-is-said-to-be-stuck.md
  - 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md
@@ -226,3 +226,34 @@ when the current failure began and how many reports in a row have said it — th
 id, whatever the words; three make the machine stuck, and `status` says so beside the failure. The host keeps trying — stuck is what the mesh
 knows, not what the machine is told. *How it is checked:* an inventory test counts three identical
 reports, a different one, and a clean apply; the status test asserts the word appears.
+
+## Everything may call what is exposed to it, and local is not a boundary
+
+*2026-09-29, from an outage that ran eleven hours —
+[issue 145](../../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md),
+settled by [ADR 0144](../../02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md).*
+
+A grant is four facts and a credential: the provision, the machine, the port, and who the consumer is
+when it connects. It is the whole mechanism by which anything in the mesh reaches anything else, and it
+rests on three things being callable — what runs on the same machine, another machine's service over the
+private network where it is exposed there, and another machine's service over the public network where it
+is exposed there.
+
+The filter had two of those. A service exposed to the private network admitted the machines' own addresses
+on it; a caller on the machine carries such an address, and a caller inside one of that machine's
+containers carries a bridge address and matched nothing. Measured, same destination and same machine:
+`src 10.10.0.1` from the machine, `src 172.17.0.8` from a container on it. So a module reaching its
+database on its own machine's name timed out for eleven hours while the mesh called the machine healthy.
+
+The second case worked by accident: a caller on another machine arrives over the tunnel carrying that
+machine's address, which the rule matched. Two of three working is why this read as correct.
+
+**So local is not a boundary this mesh draws, and the filter says so once.** Traffic that did not arrive
+from outside the machine and did not arrive over the private network is the machine's own, and is
+admitted — for every service there, not per service. Whether the caller is a container, a unit or a shell
+decides nothing, because the question is "is this the same machine".
+
+A verification mechanism was drafted for this and withdrawn. It would have reported the outage sooner and
+would not have prevented it, and the part of it that was hard — deciding which network position to check
+from — existed only because the rule was wrong. Whether the mesh should check that a grant works is still
+open, in issue 145; it is not the remedy for a configuration error.
@@ -5,8 +5,9 @@ code:
  - mesh-controller cmd/mesh-builder
  - mesh-controller internal/builder
  - mesh-catalog modules/builder
-updated: 2026-09-25
+updated: 2026-09-29
 decisions:
+  - 02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md
  - 02-DECISIONS/0111-a-build-source-is-on-the-git-seat-or-external.md
  - 02-DECISIONS/0097-a-vendor-image-is-a-declared-build-input.md
  - 02-DECISIONS/0096-an-upstream-image-is-copied-between-registries.md
@@ -263,3 +264,27 @@ ships one and wrong for code the mesh built, which has no unit until the mesh wr

 **Tools, hooks and consumers are not further modes**, which is the test of whether three is the
 right number: they are loaded by a tool host, and a tool host is a process that stays up.
+
+## The builder compiles the languages the mesh is written in
+
+*2026-09-29 —
+[ADR 0142](../../02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md).*
+
+The toolchain list was typescript and python, and only typescript had a base module in the catalogue.
+Meanwhile the control plane — written in the language this project is mostly written in — was built as
+an image from a hand-written Dockerfile, which is the per-repository incantation this whole mechanism
+exists to abolish.
+
+So the list gains Go, with a base module providing the compiler exactly as typescript has one. The
+obligation the list's own comment warns about — an SDK carrying the broker client, the event envelope
+and tool serving — attaches to a **module** written in a language, not to the language being
+compilable. The mesh's own components are not modules in that sense; the host is what applies modules.
+
+**And an artifact says what it targets.** A compiled binary is per operating system, pinned at link
+time, and a toolchain deliberately accepts nothing from the module — anything a module could override
+there it would be writing a Dockerfile to override. The target is therefore a property of the artifact,
+not of the recipe: one artifact declared per target, one build each.
+
+A component's version stops being stamped in at link time. It is unpacked into a directory named for
+its version, so it reads its version from its own path, and a build no longer has to know what it will
+be called.
@@ -1,7 +1,7 @@
 ---
-status: open
+status: located
 opened: 2026-09-26
-located-in: [mesh-controller, mesh-catalog step-ca]
+located-in: [mesh-catalog ca-trust]
 ---

 # 129 — nothing makes a machine trust the mesh's own certificate authority
@@ -0,0 +1,32 @@
+# Diagnosis
+
+*2026-09-29.*
+
+## What was ruled out
+
+**That something already carries the root and it is only misplaced.** It does not. The authority
+serves its root at a path beside its ACME directory, and the one thing that fetches it — the route
+proxy — puts it in a directory of its own and hands it to one program. Nothing has ever written
+into a machine's trust store. Measured on three converged machines: the anchors present are the
+predecessor's authority and a developer tool's local root, and on the machines where the
+predecessor's was deliberately removed, every internal name fails verification.
+
+**That the private network could carry it, the way it carries the registry's trust.** That is what
+the report proposed, and it was rejected on consideration rather than on difficulty
+([ADR 0147](../../02-DECISIONS/0147-a-module-anchors-the-meshs-authority.md), option 1): being on
+the network is what makes the registry *reachable* and is therefore the right trigger there, while
+trusting an authority is a separate fact from being able to reach it. The anchor's directory and
+the command that refreshes the extracted bundles are also one operating system's difference, which
+is the host's half of the mesh and not the controller's.
+
+**That it needs a new host resource type.** It does not, today. A file and a service say the whole
+of it, which the packet filter already proves. The primitive becomes the right answer when a second
+operating system is in play, and not before.
+
+## Where it belongs
+
+A module in the catalogue: it requires `internal-acme-ca`, fetches the root over the mesh's own
+network, installs it as a trust anchor, refreshes the machine's bundles, and — because being
+unassigned stops its unit, and stopping the unit is what undoes it — takes both away again.
+
+The owner is therefore `mesh-catalog`, module `ca-trust`, and nothing in the control plane.
@@ -0,0 +1,102 @@
+---
+status: located
+opened: 2026-09-29
+located-in:
+  - mesh-host internal/apply/opening.go (retireFirewall)
+  - mesh-host internal/apply/apply.go (the condition it is called under)
+fixed-by:
+amended-design:
+---
+
+# 143 — Converging a machine does not retire the firewall it found, and says it does
+
+## What was observed
+
+The control-node was converged on 2026-09-29, the first machine with a found firewall to be flipped —
+the two converged before it had none.
+
+The preview said, and the flip repeated:
+
+```
+the found firewall (ufw) is disabled, never flushed: its configuration stays on disk
+...
+sent: the host loads the mesh's filter and disables the firewall it found
+```
+
+The mesh then reported the node `converged`, 372 resources applied, nothing failed. Afterwards, on the
+machine:
+
+```
+systemctl is-enabled ufw   -> enabled
+systemctl is-active  ufw   -> active
+```
+
+*Corrected 2026-09-29, an hour later, from reading the host rather than the declaration.* **The first
+account of this was wrong.** It said the declaration carries no resource that would disable the found
+firewall, and that the sentence was printed by the command with nothing implementing it. The
+declaration indeed carries no such resource — but the mechanism was never meant to be one. It is a
+step in the host's own apply, `retireFirewall`, and it exists, is careful, and is strict: it refuses to
+retire anything until it has read back from the machine that the mesh's own table is loaded, it records
+the forward policies first so a half-done retirement can be retried, and it verifies ufw reports
+inactive afterwards.
+
+What is established is narrower and stranger than "nothing implements it":
+
+- ufw was **active and enabled two minutes after the flip**, and the flip had reported the node
+  converged with 372 resources applied and nothing failed.
+- The machine's own record now reads `disabled_by_mesh: true` — but it was written by a reconcile
+  *after* an operator disabled ufw by hand, roughly fifty minutes later. A reconcile found ufw already
+  inactive, asked it to be inactive, read that back, and recorded that the mesh had done it.
+- So the step did not take effect at the flip, and the machine's record now says it did.
+
+The candidates are named rather than chosen, because the evidence does not separate them: the step is
+called only when the apply had no failures, and a skipped step is silent; the mesh's table is loaded by
+a service in the same apply, so whether it was loaded *at the moment the step asked* is an ordering
+question; and the host's own detail lines do not reach the journal, so what it decided is not
+recoverable after the fact.
+
+## Why it matters beyond this instance
+
+**It is a stated behaviour that does not happen, reported as success** — the fault this repository
+exists to catch, and
+[ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md) states it as
+part of what the flip *is*: "loads the mesh's derived filter in place of its refusal-only table, and
+retires the found firewall by disabling it, never by flushing".
+
+**It could only be found on the first machine that had one.** The two machines converged before this
+had no firewall to retire, so the step had never run, and nothing reported that it had not. That is
+the same shape as [issue 136](../136-a-module-may-name-a-program-the-machine-does-not-have/00-report.md):
+a step that is silent when it does nothing.
+
+**The machine is left doubly filtered, which is not what either firewall describes.** Every base chain
+at a hook runs and a drop in any is final, so the machine now enforces the *intersection* of the mesh's
+derived filter and a rule set left by the system being replaced. Nothing is broken by that today —
+measured from outside, mail, the proxy and git-over-ssh answer and the databases and admin interfaces
+are refused — but the machine's behaviour is described by neither of the two things claiming to
+describe it, and the stale set includes a rule for a broker that no longer exists.
+
+**And returning the node to adopted would be wrong in the other direction.** ADR 0100 says that
+restores the found firewall by enabling it again; enabling something that was never disabled is
+harmless, but the mesh's belief about which firewall is in force has been wrong in both modes.
+
+## Open questions
+
+- Which side owns retiring it — a resource in the declaration, so it is applied and reported like
+  everything else, or the flip as an act? A resource seems right: the flip is otherwise entirely
+  expressed as one, and an act that only the command performs cannot be re-checked on a later
+  reconcile.
+- What should a reconcile do if the found firewall is enabled again by hand, or by a package update?
+  Convergence is a state, so presumably re-disable it and say so.
+- Should the preview say what it *will* do rather than what it does, until a step exists that does it?
+  The wording was read as evidence twice in one session.
+- Is there a check that a sentence the mesh prints corresponds to something that happened? This is the
+  second time in one session that a printed claim and the machine disagreed.
+- **Why did the step not take effect?** It is called only when the apply had no failures, and being
+  skipped is silent. The mesh's table is loaded by a service in the same apply, so whether it was
+  loaded when the step asked is an ordering question — and ADR 0100 makes loading it first a
+  precondition rather than an expectation.
+- **A step that records the mesh as having done what an operator did is worse than the omission.** The
+  record now says the mesh disabled ufw. Nothing distinguishes "we did this" from "we found it already
+  so". Should it?
+- Why do the host's own detail lines not reach the journal? Everything it decided during the flip is
+  unrecoverable, which is why this account has candidates instead of a cause.
@@ -0,0 +1,84 @@
+---
+status: located
+opened: 2026-09-29
+located-in:
+  - mesh-host internal/apply/opening.go
+  - mesh-controller cmd/mesh-controller (the converge preview)
+fixed-by:
+amended-design:
+---
+
+# 144 — A predecessor's rules outlive the firewall the mesh found, and the mesh cannot see them
+
+## What was observed
+
+The mesh reports one thing about a machine's existing filtering: `firewall found: ufw`. On the
+control-node, ufw was never what filtered the traffic that mattered.
+
+Measured on 2026-09-29, before the machine was converged:
+
+- ufw filters connections *to the machine*. It does not filter connections to a container's published
+  port, which arrive on the forwarded path where the container runtime accepts them before ufw's
+  forward chains are reached. Around thirty ports were published that way.
+- Every one of the mesh's own forwarded openings, converged through ufw, had matched **zero packets** —
+  fifty rules in that chain, none ever matched, while the chain itself had passed 1.6 million
+  established packets. The restrictions read as applied and were inert.
+- What actually kept those ports off the internet was a chain the predecessor installed in the
+  container runtime's own pre-accept hook, allowing the deliberately public ports and the private
+  ranges and dropping the rest on the outward link. Confirmed from outside: the proxy answered, the
+  container manager did not.
+- That chain exists only in the running kernel. The persisted rule file is the distribution's empty
+  default, and nothing on disk recreates the chain.
+
+After the flip, the mesh's own filter is loaded and does cover the forwarded path, so the machine no
+longer depends on that chain. But **the chain is still there**, and it is now the only thing refusing
+two ports the mesh believes are open: the bus and the registry, which
+[ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md) requires be
+reachable from anywhere so a machine can enrol and pull before it has a private-network address. The
+mesh's rendered filter accepts both from anywhere. From outside, both are refused.
+
+## Why it matters beyond this instance
+
+**"The firewall found" is a kind, and filtering is not all in one place.** The host identifies one
+front-end and reports it. A machine can carry rules from several sources — the front-end's own, the
+container runtime's, an intrusion-prevention chain, and whatever a predecessor installed directly —
+and the mesh's account of what filters the machine names exactly one of them.
+
+**So adoption's central promise was half-true in both directions.** What the mesh converged through
+the found firewall on the forwarded path did nothing at all, and what did the work was invisible to it.
+A machine was reported as filtered by a mechanism that was not filtering.
+
+**And convergence cannot retire what it cannot see.** Even once
+[issue 143](../143-converging-does-not-retire-the-firewall-it-found/00-report.md) is fixed and the found
+firewall is disabled, this chain remains, silently narrowing the machine below what the mesh's own
+filter says. A rule the mesh did not write, cannot list, and will not remove — which today breaks the
+enrolment path the design guarantees.
+
+**The safe direction is not the same as the correct one.** Being more closed than intended broke nothing
+visible, which is exactly why it went unnoticed for as long as the mesh has been on this machine.
+
+## What it cost, measured later the same day
+
+*2026-09-29.* The predecessor's chain was removed, and something it had been carrying went with it. It
+admitted the private ranges wholesale, which is how a container on the machine reached a port declared
+for the private network — the mesh's own filter admits the machines' overlay addresses, and a container
+comes from a bridge. Every module that reached another by the machine's own name had been relying on the
+predecessor's rule without anybody knowing.
+
+That is [issue 145](../145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md),
+and it ran for eleven hours while the mesh reported the machine healthy. The filter is fixed. What this
+adds to the account here is that "the machine is more closed than the mesh believes" was not the
+harmless direction after all — it was harmless for everything reached from outside, and an outage for
+everything reached from within.
+
+## Open questions
+
+- Should the host report every place the machine filters from, rather than one kind — the front-end,
+  the runtime's hooks, and any chain it does not recognise, named so a person can look?
+- What should the mesh do about rules it did not write and does not understand? Reporting them seems
+  right; removing them cannot be, and leaving them silent is what produced this.
+- Does an opening converged through a found firewall need a check that it can actually take effect?
+  Fifty rules matching nothing would have been visible from the counters at any point.
+- Is the bus and the registry being reachable from anywhere still what the mesh wants on a machine that
+  faces the internet? The design says yes, for enrolment. It deserves asking on its own rather than
+  being answered by a leftover.
@@ -0,0 +1,119 @@
+---
+status: located
+opened: 2026-09-29
+located-in:
+  - mesh-controller internal/catalogue/filtering.go (fixed for this instance)
+  - mesh-controller (what status reports, and what it does not ask)
+fixed-by:
+amended-design: 03-DESIGN/01-to-be/10-delivery.md
+---
+
+# 145 — A machine reads healthy while its modules cannot reach each other
+
+## What was observed
+
+Converging the control-node closed every path by which a module on that machine reached another module
+by the machine's own name. It ran for **eleven hours**. Throughout, the mesh answered:
+
+```
+4 machine(s), all doing what they were told, all heard from,
+running what the mesh would send them, and every module current with its source
+```
+
+What was actually happening, from one affected module's own log:
+
+```
+Doctrine\DBAL\Exception: Failed to connect to the database:
+SQLSTATE[08006] connection to server at "novox.internal" (10.10.0.1), port 6852 failed: timeout expired
+```
+
+6,154 of them, beginning at the minute of the flip. The web application accepted TCP connections and
+never answered an HTTP request; a client waited 35 seconds and gave up. Confirmed from a throwaway
+container on the machine: neither the store nor the forge was reachable on the machine's own address.
+
+The cause is [issue 144](../144-the-predecessors-rules-outlive-the-firewall-it-was-found-as/00-report.md)'s
+sibling and is fixed: a port declared reachable from the private network admitted the machines' own
+overlay addresses, and a container on the machine comes from a bridge address, matching none of them.
+What this issue is about is the eleven hours.
+
+**Nothing the mesh reports would have shown it.** Every check the mesh makes passed, because every
+check the mesh makes is about the relationship between the mesh and a machine:
+
+- the machine applied what it was sent, and said so;
+- its declaration digest matches what the mesh would send;
+- every module's source commit matches what the mesh holds;
+- every container the declaration names is running.
+
+None of those asks whether a module can reach what it requires. The mesh knows precisely who requires
+what — it composes the grants — and never checks that the grant works.
+
+**Nor would an operator's usual look.** The ports were probed from outside and behaved correctly; the
+routed services answered; a container's egress to the internet worked. Those are the paths a person
+checks after changing a firewall, and all three were fine. The broken path was module-to-module over
+the machine's own name, which nothing routine exercises.
+
+## Why it matters beyond this instance
+
+**A mesh that composes a dependency and never tests it can only report on itself.** Every provision the
+mesh grants is a claim that a consumer can reach a provider. The mesh asserts that claim, delivers
+credentials for it, and has no mechanism that ever finds out. "Every module current with its source"
+is a statement about bytes, not about whether anything works.
+
+**The failure was silent in the direction that hides longest.** A service that will not start is
+noticed. A service that starts, accepts connections and then cannot reach its database serves errors
+under a healthy-looking process, and the machine's own report says the container is running — which it
+is.
+
+**It is the same shape as [issue 136](../136-a-module-may-name-a-program-the-machine-does-not-have/00-report.md),
+one level up.** There, a module named a program the machine lacked and everything reported success.
+Here, the mesh granted a provision the filter refused and everything reported success. Both are the
+distance between a declaration and the machine, and in both cases the report was about the declaration.
+
+**And the eleven hours are the measurement, not the bug.** The filter fault was one line and is fixed.
+What is not fixed is that nothing in the mesh would have told anybody.
+
+## What was decided
+
+*2026-09-29, the same day, in two steps and the first was wrong.*
+
+The first answer was [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md):
+the consumer verifies each grant from its own network position, because whether a caller sat in a
+container changed whether it could reach the provider. **That difference was the fault**, and the record
+is superseded. A verification mechanism would have reported this sooner and would not have prevented it,
+and the part of it that was difficult — deciding which network position to check from — existed only
+while the rule was wrong.
+
+The remedy is [ADR 0144](../../02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md):
+anything on a machine may call anything on it, said once rather than per service, and asked by the link
+traffic arrives on rather than the address it carries. Everything should be able to call what runs on the
+same machine, another machine's service exposed to the private network, and another machine's service
+exposed publicly. The filter had the second and third and expressed the first as a list of addresses that
+no container could match.
+
+**And then the question this issue is actually about was answered on its own terms.**
+[ADR 0145](../../02-DECISIONS/0145-a-module-checks-what-the-mesh-claims-is-reachable.md): a module on
+every machine serves an endpoint of its own and dials every other machine's, from the position the
+callers are in. Its probe is its own endpoint declared reachable over the private network, so it is
+admitted by exactly the rule that governs every internally-exposed service and fails when that rule is
+wrong — where a probe on a service every machine has would have passed for all eleven hours, because the
+services every machine has are the ones never closed.
+
+Adopted on its merits rather than as the remedy for a configuration error, which is what 0143 was and
+why it went. The module is written and merged; it is not yet assigned, so every machine currently reads
+unchecked.
+
+## Open questions
+
+- Should a grant be checked? The mesh knows the consumer, the provider, the address and the port, so a
+  reachability check is expressible — but from where: the consumer's machine, as part of a reconcile,
+  or the provider's?
+- What would it cost to be wrong in the other direction? A check that reports a provision broken while
+  it works is worse than none, because it trains a reader to ignore the report. A provider restarting is
+  ordinary; a consumer between containers is ordinary.
+- What should `status` say about a machine whose modules cannot reach each other? It currently has one
+  vocabulary for "heard from and current", and that sentence was true the whole time.
+- Is there a cheaper signal than a probe? The affected module was logging the failure 6,154 times. The
+  mesh reads no module's logs and arguably should not — but something a module could *say* about its
+  own provisions would have surfaced this in minutes.
+- Does the same blindness apply to the other direction — a provider that lost a consumer's grant and
+  is refusing it? Nothing checks that either.
@@ -0,0 +1,73 @@
+---
+status: located
+opened: 2026-09-29
+located-in: [mesh-host examples + internal/link, mesh-controller internal/broker]
+---
+
+# 146 — the foundation cannot be raised on the bus the mesh runs on
+
+## What was observed
+
+Raising a first node in the lab, to check a module against a real mesh, fails before any module is
+reached. Two separate faults, in the two bundles that exist:
+
+**The older bundle raises a control plane that cannot start.** It brings up the previous broker,
+and the control plane it then starts says, once every few seconds, for ever:
+
+```
+mesh-controller: this control plane has no MESH_BUS_NATS, so it cannot reach the mesh's bus
+```
+
+That is the control plane being right. The mesh moved to one bus
+([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)) and the
+bundle did not. Every bed that raises a foundation raises this one, so every bed is in this state.
+
+**The newer bundle, written for the new bus, stops one step earlier.** Its certificate step asks a
+container to make the broker's certificate:
+
+```
+docker run --rm --entrypoint sh -v <the broker's tls volume>:/tls <the bus image> \
+  -c "test -f /tls/tls.crt || (openssl req -x509 ... )"
+...
+failed bus-certificate: running the action: docker exited 127
+```
+
+127 is *command not found*. The bus's image has a shell and no `openssl`; the previous broker's
+image had both, which is why the step worked when it was written against that one. Substituting the
+store's image — the only other image the bundle carries — does not help: it has no `openssl`
+either. So the step as written cannot succeed with anything the bundle names, and the fault is not
+one image's: **the bundle asks for a certificate to be made by a tool it never says must be there.**
+
+Measured 2026-09-29 on a fresh lab machine, both bundles, from bare.
+
+## Why this is here and not a note in the knowledge base
+
+The mesh's own foundation is the one thing it cannot raise. Nothing reports that: the bundles are
+files in a repository, nothing applies them but a person raising a node, and the last thing that
+did was the hand-driven cut-over
+([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md), whose
+work was done on the machines rather than from a bundle). So the state where the mesh cannot make
+another one of itself is reachable, and was reached, without anything saying so.
+
+It is also load-bearing for everything else: a lab bed proves a claim by raising a mesh, so while
+this holds, **no bed can run**, and every "checked in the lab" written from now on is a promise
+against a suite nobody can execute.
+
+## What would have prevented it
+
+- **Something raising the foundation on a schedule, from the bundle, as it is written** — the
+  bundle is the mesh's own installer and nothing installs from it. A bed that raises a first node
+  is exactly that check, and it is the bed that cannot run.
+- **A step naming what it needs.** The certificate step names an image and assumes a program inside
+  it. An action that said which tool it requires would have failed at the declaration rather than
+  at 127 on a machine.
+
+## Evidence to carry into diagnosis
+
+- `mesh-host examples/foundation-first-node.lock` — the previous broker, no `MESH_BUS_NATS`.
+- `mesh-host examples/foundation-first-node-nats.lock` — the new bus; `bus-certificate` and its
+  `verify` both run `openssl` in the bus's image.
+- The bus image the bundle pins has `sh` and no `openssl`; the store's image likewise.
+- The lab rewrites a bundle's registry-prefixed third-party references to upstream ones for a
+  machine with an uplink (`test/integration/harness.ts`); the new bundle's bus reference needed
+  that rule added, which is done and is not this issue.
@@ -0,0 +1,161 @@
+# Diagnosis
+
+*2026-09-29, by raising a first node in the lab over and over and writing down each thing it hit.*
+
+Not one fault. **Four, stacked**, each hidden behind the one before it, and every one of them the
+same shape: a step that was right while the mesh ran on the previous broker and was never asked a
+question again after the bus changed. Nothing had raised a foundation since, so nothing said so.
+
+## 1 — the bundle's bus image is named for a registry that is gone *(fixed)*
+
+The newer bundle pins `<a lab registry>/nats@…`, which resolves nowhere outside the lab that
+raised that registry. The lab already rewrites the store's and the previous broker's references to
+upstream ones for a machine with an uplink; the bus had no such rule because no bed had ever tried
+to raise this bundle. Added (`mesh-lab test/integration/harness.ts`). The digest is the bundle's
+own — what the registry served was a copy, so the same digest resolves upstream, and this is a
+prefix being removed rather than a reference being replaced.
+
+## 2 — the bus's certificate was made by a tool the bus does not have *(fixed)*
+
+```
+failed bus-certificate … docker exited 127
+```
+
+The step ran `openssl` inside the broker's image. The previous broker's image carried it; the bus's
+does not — it is Alpine with a shell and no `openssl` — and neither does any other image the bundle
+names, so there was nothing to substitute. **The program that needs the certificate now makes it**:
+`mesh-controller broker certificate --into <dir>`, with `--check` as the step's verify. The
+controller is already on the machine at that point (the schema step ran it) and needs nothing from
+the image it writes into. Self-signed, as before and on purpose — a host pins this server's exact
+certificate ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)) and at that moment
+there is no authority to ask. Idempotent, because a second certificate is one every host that
+pinned the first no longer believes. It runs `--user 0:0`: the volume is root's, and the control
+plane's image runs as nobody, which is right for the long-lived server and wrong for a one-shot
+writing into a fresh volume.
+
+## 3 — enrolment dialled TLS at a bus that speaks first *(fixed)*
+
+```
+mesh-host: cannot reach the broker at …:5671: tls: first record does not look like a TLS handshake
+```
+
+Enrolment opened a raw TLS connection to check the pinned certificate before saying anything. NATS
+speaks its own protocol and upgrades afterwards, so the handshake met a plaintext greeting. The pin
+was never the problem: the same pinned configuration is handed to the client that presents the
+token, and the verification runs inside *that* handshake — so what
+[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) requires still holds, and holds
+better, because the one-time secret is sent only after the certificate has been checked. The raw
+dial is gone from the enrolment path and kept only as what its tests always proved: that a wrong
+certificate is refused before a byte of application data is sent.
+
+**Then, immediately behind it:**
+
+```
+mesh-host: this token is for the "" bus, and the mesh's bus is nats
+```
+
+The enrolment left the transport empty and meant *whatever the mesh runs today*, which was true
+while two buses existed and became a refusal the moment one did. The host knows which bus the mesh
+runs; it says so now.
+
+## 4 — a first node cannot be let onto its own bus *(open, and this is the real one)*
+
+```
+mesh-host: cannot reach the bus at …:5671 as anchor: nats: Authorization Violation
+```
+
+The bus's user list is a file beside its configuration. The installer carries the first one — the
+controller's own account at a bootstrap password — and **the controller composes every user after
+that** (design 25 §6; the controller's own test asserts the carried list matches what it would
+derive). On the running mesh that composition reaches the bus because the bus is a *module*, with
+the list delivered to it the way anything is delivered to a module.
+
+At genesis there is no module. The foundation's bus is raised by the installer, the control plane is
+given no way to write beside its configuration — it mounts the certificate and nothing else — and so
+the account a joining node needs cannot come into existence. **The first node cannot join the mesh
+it just raised.**
+
+That is not a line to fix in a bundle. It is the open half of the mesh delivering its own components
+([ADR 0142](../../02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md)) and of the
+bus becoming a module: either the installer's bus is raised as the module the mesh will go on
+managing, or genesis carries a user list that includes the first node's enrolment and the controller
+takes over from there. Both are decisions, not patches, and both belong to the genesis step that was
+deliberately left until last.
+
+## 5 — the composed user list has to be placed by hand at genesis *(fixed)*
+
+The account a token is the password of is **not recorded at all**: the composer names an enrolment
+user for every machine with a live token, nothing minted a credential for it, and the composition
+left it out as a user with no password. The comment above the issuing code already claimed
+otherwise — *"the account is created before the token is handed over"* — which is how it went
+unnoticed. Issuing a token now records that account, with the token's own secret as its password,
+because that is the string the machine will present.
+
+Placing it is the other half. The list reaches the machine running the bus in that machine's
+declaration, which a machine that has not enrolled does not get, so at genesis it cannot arrive
+that way. **The control plane composes and says what it composed** — `broker accounts`, to standard
+output — and whoever is raising the machine writes it beside the bus's configuration and makes the
+server re-read it. Twice, because two accounts come into existence at different moments: the
+enrolment when the token is issued, and the machine's own when it enrols. A control plane that
+wrote the file itself would have to know where the bus keeps its configuration and how to make it
+reload, which is the module's knowledge and is what the module takes over on the first push.
+
+With that, **a first node enrols against the bus it just raised** — measured, from bare, in the
+lab.
+
+## 6 — and is enrolled twice, keeping a credential the mesh has replaced *(open)*
+
+```
+mesh-controller: enrolled anchor
+mesh-controller: enrolled anchor      (the same second)
+```
+
+One `enrol` on the machine, two enrolments in the control plane. Each mints the node a fresh bus
+password and returns it; the machine keeps the answer to the first, and the mesh keeps the hash of
+the second. The machine then reconnects for ever as a user whose password the mesh rotated out from
+under it — *authentication error - User "node.anchor"* on the bus, `Authorization Violation` in the
+host's log, and a node that never reports.
+
+What is ruled out: the host asking twice — it asks again only when the mesh says *try again*, and
+a refused attempt is not logged as an enrolment. Redelivery by the consumer — there is one
+consumer, its acknowledgement window is thirty seconds, and the handler is quick.
+
+What is left: the client re-publishing when an acknowledgement is slow, which is what its defaults
+do. That was addressed by giving the publish a message id derived from its own bytes, so the stream
+discards the copy — **and the duplicate survived it**, so either the id is not reaching the stream
+or the second copy is not a copy. This is where the trail stops.
+
+Worth saying plainly: **the mint is the fragile part, not the delivery.** An enrolment answered
+twice is survivable if the answer is the same both times, and it cannot be — the mesh keeps only
+the hash, so a second answer is necessarily a different credential. Whatever closes this either
+makes the enrolment arrive once, or stops the second arrival from rotating anything.
+
+## Where it belongs
+
+`mesh-host` (the bundle and the enrolment path) and `mesh-controller` (the certificate command, and
+the composition that cannot reach the bus at genesis). Three of the four are fixed on branches; the
+fourth is the genesis work.
+
+## What made it slow, and what was changed so it is not
+
+Six faults behind one another, each found by raising a machine and reading what it said. What cost
+the most was not the faults:
+
+- **Every bed's own instructions named the bundle that cannot work**, so the first three attempts
+  ended in a control plane crash-looping on a missing bus. They name the working one now.
+- **A host binary built without its system** refuses everything it is given with *this host was
+  built for ""*, which reads like a broken bundle. The lab's README says so.
+- **`make image` in the control plane had been broken for as long as its base was pinned**: the
+  Dockerfile's fallback is a Go older than the module asks for, and the pipeline never saw it
+  because the pipeline passes the declared base in. It reads the base from the manifest now.
+- **Leaving the machine standing is what answers the question.** Every finding above came from
+  shelling in afterwards — the host's log, the bus's log, the file the bus was actually handed —
+  and none from the test's own output, which says only that nothing converged. The bed takes
+  `MESH_LAB_KEEP`, and the README says to reach for it first.
+
+## What it cost, for the next person
+
+Every lab bed still names `foundation-first-node.lock` in its own instructions, and that bundle
+raises the previous broker with a control plane that refuses to start without `MESH_BUS_NATS`. Until
+the fourth fault is answered and the two bundles become one, a bed runs with `MESH_LAB_BUNDLE`
+pointing at the NATS bundle by hand, and stops at the enrolment.
Author	SHA1	Message	Date
jschoubben	e9b1010bc0	Issue 146: what made it slow, and what was changed so it is not	2026-09-29 17:45:25 +02:00
jschoubben	f6ed3545b7	Issue 146: a first node now enrols, and is enrolled twice Two more faults behind the three already fixed. The account a token is the password of was never recorded, and the comment above the issuing code said it was; issuing now records it. Placing the composed list at genesis is the other half — the control plane says what it composed and whoever raises the machine writes it beside the bus, because no declaration can reach a machine that has not enrolled. With that a first node enrols. It is then enrolled twice from one attempt, each minting a credential, and it keeps the answer to the first while the mesh keeps the second. The trail for that one stops at a duplicate that survived message-id deduplication.	2026-09-29 17:36:40 +02:00
jschoubben	0b08cdfce1	Merge pull request 'ADR 0147: a module anchors the mesh's authority, and issue 146: the foundation cannot be raised' (#184 ) from decision/0147-a-module-anchors-the-meshs-authority into main	2026-09-29 14:06:41 +00:00
jschoubben	bffd2af40c	Issue 146 diagnosed: four faults stacked, three fixed, the fourth is genesis Raising a first node hits them in order: the bundle's bus image named for a registry that is gone; the certificate made by openssl in an image that has none; enrolment dialling TLS at a bus that speaks first, then refusing its own token for the empty bus. Each was right until the bus changed and nothing has raised a foundation since. The fourth is not a patch: the installer carries the bus's first user list and the controller composes the rest through the bus being a module, and at genesis there is no module — so the first node cannot be let onto the bus it just raised.	2026-09-29 15:42:32 +02:00
jschoubben	4a51ea4b3a	Issue 146: the foundation cannot be raised on the bus the mesh runs on Found trying to run ADR 0147's bed. The older bundle raises the previous broker and a control plane that refuses to start without MESH_BUS_NATS; the newer one stops a step earlier, asking for a certificate from openssl in an image that has none. No bed can run while this holds, so 0147's check section now says what actually stands behind it — the rendering, not a machine.	2026-09-29 15:25:58 +02:00
jschoubben	6c14d313b8	ADR 0147: a module anchors the mesh's authority on a machine, and takes it away again Issue 129: every internal HTTPS name fails verification on every machine, because nothing has ever written the mesh's root into a trust store. The report proposed the controller inject it the way the private network writes the registry's trust; this record rejects that — reachability and trust are not the same fact, and where anchors live is the host's difference, not the controller's. A module requiring internal-acme-ca does the whole of it, and being unassigned undoes it.	2026-09-29 15:02:37 +02:00
mesh-admin	ced547dae9	Merge pull request 'ADR 0146: connectivity is checked by name, per hosting form' (#183 ) from decision/0146-connectivity-by-name into main	2026-09-29 12:01:57 +00:00
jschoubben	38482435af	ADR 0146: connectivity is checked by name, per hosting form, with a valid certificate 0145's core stands — a module checks what the mesh claims, from where the callers are — and everything it said about what to dial was wrong. A raw port is not how anything in this mesh is reached. A real caller resolves a name, the proxy answers, the proxy reaches the service; dialling a port tests the last hop of a four-hop path and skips the three where most connectivity lives. So each hosting form gets its own endpoint, route and name: connect-docker.<node>.internal for a container, connect-process for the mesh's own code in a unit it writes, connect-unit for a unit a package ships — and the same under each public domain. Every machine checks every machine, by name, over TLS, verifying the certificate against the authority that should have issued it. The names are the instrument: a failure reads as connect-docker.g14.internal did not answer. No name is written anywhere: the machines come from the roster fact, the labels are the module's, and a machine that joins is checked by the others on the next push. Five things this needs that the mesh does not have, named rather than assumed — a module every machine has (ScopeNode is exclusivity, not obligation), a container running a bundle, a unit a package ships, the public domain in the roster fact, and issue 129, until which every internal name will correctly fail verification.	2026-09-29 14:01:55 +02:00
mesh-admin	cf8a8d78c9	Merge pull request 'ADR 0145: a module checks what the mesh claims is reachable' (#182 ) from decision/0145-a-module-checks-what-the-mesh-claims into main	2026-09-29 11:44:58 +00:00
jschoubben	9de25994e9	ADR 0145: a module checks what the mesh claims is reachable, and it checks itself The mesh asserts three things are callable and has never checked any of them. A module on every machine serves an endpoint of its own and dials every other machine's, from the position the callers are in — not the host and not the control plane, both of which reach those addresses by paths no ordinary caller uses and would have passed throughout the outage. Its probe is its own endpoint, declared reachable over the private network, so it is admitted by exactly the rule that governs every internally-exposed service. The tempting target is a service every machine has, and those are the ones never closed — ssh above all — which would have passed for all eleven hours. Resolution and connection reported separately, because they have different owners. One failure is not a fault, and the count travels with the result. It reports and repairs nothing. Built on a scheduled container, a rendered roster fact and a bundle: no new vocabulary. Adopted on its own merits, which is what 0143 was not.	2026-09-29 13:44:56 +02:00
mesh-admin	64ea47b11d	Merge pull request 'ADR 0144: anything on a machine may call anything on it' (#181 ) from decision/0144-local-is-not-a-boundary into main	2026-09-29 11:32:03 +00:00
jschoubben	eba24a72af	ADR 0144: anything on a machine may call anything on it, superseding 0143 Everything should be able to call what runs on the same machine, another machine's service exposed to the private network, and another machine's service exposed publicly. Three cases; the filter had two. The first was expressed as the machines' own addresses on the private network. A caller on the machine carries such an address; a caller in one of its containers carries a bridge address and matched nothing — measured, same destination and same machine: src 10.10.0.1 against src 172.17.0.8. The second case worked by accident, because the tunnel rewrites a caller's address to the sending machine's. Two of three working is why it read as correct. 0143 answered the wrong question. It proposed verifying each grant from the consumer's own network position and went to length about which position, because whether a caller sat in a container changed the answer — and that difference was the bug. Observing a configuration error is not its remedy. Superseded, and nothing replaces it; whether the mesh should check a grant is still open in issue 145 and must stand on its own. And a module is not a container: 61 of 72 happen to use one, 11 do not, and a rule reasoning about containers describes most of the mesh rather than the mesh.	2026-09-29 13:32:01 +02:00
mesh-admin	78d4873f4f	Merge pull request 'ADR 0143: a consumer verifies the grant it is given' (#180 ) from decision/0143-a-consumer-verifies-its-grant into main	2026-09-29 11:07:20 +00:00
jschoubben	ddfd62edf6	ADR 0143: a consumer verifies the grant it is given A grant is four facts and a credential — the provision, the machine, the port, and who the consumer is when it connects — and it is the whole mechanism by which anything in the mesh reaches anything else. The mesh asserted it and never found out whether it was true. Issue 145 is what that cost: eleven hours of 'every module current with its source' while a module could not reach its database. The consumer verifies it, from its own network position. Not the control plane, which reaches the address by a path no consumer uses. Not the machine, whose own packets carried a source address the filter admitted while every container's was refused — so a host-side check would have passed throughout the outage it exists to catch. That is inference from the rule that was loaded, stated as such. A connection and nothing more; speaking each provision's protocol would be a second implementation of every provision. One failure is not news, only consecutive ones, and the count is reported rather than the last attempt. A consumer that is not running reads unchecked, which is a different sentence from broken. What it costs to be wrong is the constraint on all of it: a check that calls a working provision broken trains a reader to ignore the report.	2026-09-29 13:07:18 +02:00
mesh-admin	f65664640a	Merge pull request 'Issue 145: a machine reads healthy while its modules cannot reach each other' (#179 ) from issue/145-healthy-while-broken into main	2026-09-29 10:37:10 +00:00
jschoubben	492ac7be18	Issue 145: a machine reads healthy while its modules cannot reach each other Converging the control node closed every path by which a module reached another by the machine's own name, and it ran for eleven hours while the mesh answered 'all doing what they were told, all heard from, running what the mesh would send them, and every module current with its source'. 6,154 database failures in one module's log, beginning at the minute of the flip. The service accepted TCP and never answered HTTP. Every check the mesh makes passed, because every check the mesh makes is about the relationship between the mesh and a machine — applied, current, containers running. None asks whether a module can reach what it requires, though the mesh composes every grant and so knows exactly who requires what. The filter fault is fixed. The eleven hours are the measurement, not the bug. Also records on 144 that 'more closed than the mesh believes' was harmless only for what is reached from outside, and an outage for what is reached from within.	2026-09-29 12:37:08 +02:00
mesh-admin	5ac77e3cef	Merge pull request 'ADR 0138: reach asks for names on a routed endpoint' (#178 ) from decision/0138-insight-reach-and-the-proxy into main	2026-09-29 00:50:28 +00:00
jschoubben	a619022c35	ADR 0138: a progressive insight — reach asks for names on a routed endpoint The record says internal and public each mean something to the filter. For an endpoint the proxy serves, the second half is wrong, and ADR 0045 said so first: a public service is exposed through the proxy, listening from the mesh, not by opening its own port. Found by trying to express one real module, not by review — routed name public because browsers post to it, machine port private because it serves a dashboard in cleartext. Under one value for both, saying public would have reopened a port an operator had just closed. Measured the same evening: the routed name answered from the internet over TLS while the port was refused from the same place. Corrects a fact. One statement per endpoint with three things derived from it stands; the filter column applies to an unrouted endpoint.	2026-09-29 02:50:25 +02:00
mesh-admin	49c065c204	Merge pull request 'Issue 143: correct the diagnosis' (#177 ) from issue/143-corrected-diagnosis into main	2026-09-29 00:33:13 +00:00
jschoubben	ddb3980f09	Issue 143: correct the diagnosis — the step exists and did not fire The first account said the declaration carries no resource that would disable the found firewall and that nothing implemented the sentence the flip prints. Wrong: the mechanism is a step in the host's own apply, retireFirewall, and it is careful — it reads back that the mesh's table is loaded before retiring anything, records the forward policies first, and verifies ufw reports inactive afterwards. What is established: ufw was active and enabled two minutes after a flip that reported the node converged with nothing failed; and the machine's record now reads disabled_by_mesh: true, written by a reconcile fifty minutes later that found ufw already inactive because an operator had disabled it by hand. So the step did not take effect and the record says it did. The candidates are named rather than chosen — the step is skipped silently when the apply has any failure, the mesh's table is loaded by a service in the same apply so ordering is open, and the host's detail lines do not reach the journal, which is why this has candidates instead of a cause.	2026-09-29 02:33:10 +02:00
mesh-admin	75a8f6abc7	Merge pull request 'Issues 143 and 144: the found firewall is neither retired nor all of it' (#176 ) from issue/143-and-144-the-found-firewall into main	2026-09-28 23:36:10 +00:00
jschoubben	862253518f	Issues 143 and 144: the found firewall is neither retired nor all of it Both found by converging the control node — the first machine with a firewall to flip, since the two before it had none. 143: the preview and the flip both say the found firewall is disabled. The node reported converged, 372 resources applied, nothing failed, and ufw is still enabled and active. A converged node's declaration carries no resource that would disable it; the sentence is printed by the command and nothing implements it. 144: ufw was never what filtered the traffic that mattered there. Fifty forwarded openings converged through it had matched zero packets, while a chain the predecessor installed in the container runtime's pre-accept hook did the work — in memory only, recreated by nothing. The mesh's filter now covers that path, so the machine no longer depends on it, but the chain remains and is the only thing refusing the bus and the registry, which the design requires reachable from anywhere so a machine can enrol before it has a private address.	2026-09-29 01:36:08 +02:00
mesh-admin	51ef3eb7e2	Merge pull request 'ADR 0142: the mesh delivers its own components as binaries' (#175 ) from decision/0142-mesh-delivers-its-own-components into main	2026-09-28 22:51:44 +00:00
jschoubben	346e613995	ADR 0142: the mesh delivers its own components as binaries, not container images Measured: the host is a binary somebody copied to four machines, owned by no package and built by nothing, while the controller, catalogue, builder and vault are container images publishing no ports at all. Same language, same project, same kind of work, delivered two ways — and the difference is not a judgement about either, it is that images are the only delivery that works. What it costs: genesis must raise a container runtime before the control plane can exist; updating the control plane goes through a registry the control plane runs; a host change cannot be rolled out at all; and compiling the language the mesh is written in is not a capability of the builder, so the controller is built from a hand-written Dockerfile — the incantation the bundle toolchain exists to abolish. Third-party software stays a container: the store, the registry, the broker are somebody else's build. The container runtime stays on the machine for modules. What changes is that the control plane no longer needs it to exist. The receiving half is already built and tested (ADR 0141). Staged: compile Go, an artifact names its target, deliver a binary, the host first, then the rest, genesis last.	2026-09-29 00:51:42 +02:00
mesh-admin	25ca9898d5	Merge pull request 'ADR 0141: a progressive insight on what delivery costs' (#174 ) from decision/0141-progressive-insight-on-delivery into main	2026-09-28 22:32:17 +00:00