Compare commits
53
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f89aef992d | ||
|
|
b967ef7be3 | ||
|
|
e417906241 | ||
|
|
b1bf895688 | ||
|
|
0d64677c70 | ||
|
|
04205c1dc8 | ||
|
|
9dc49cd831 | ||
|
|
66b413a076 | ||
|
|
fcba05fed9 | ||
|
|
18f37c25b2 | ||
|
|
3c2b4fc6b6 | ||
|
|
72eaf52867 | ||
|
|
741625e725 | ||
|
|
c3730b9a23 | ||
|
|
3a59099c81 | ||
|
|
3bd6f34de3 | ||
|
|
2b5119ecd2 | ||
|
|
96bdffa9bc | ||
|
|
4eb16f1028 | ||
|
|
d199de40db | ||
|
|
9a1dc4665c | ||
|
|
14be8576f8 | ||
|
|
ec8676c225 | ||
|
|
3c0f7082e6 | ||
|
|
0dd00e88b6 | ||
|
|
eef54917ec | ||
|
|
e9b1010bc0 | ||
|
|
f6ed3545b7 | ||
|
|
0b08cdfce1 | ||
|
|
bffd2af40c | ||
|
|
4a51ea4b3a | ||
|
|
6c14d313b8 | ||
|
|
ced547dae9 | ||
|
|
38482435af | ||
|
|
cf8a8d78c9 | ||
|
|
9de25994e9 | ||
|
|
64ea47b11d | ||
|
|
eba24a72af | ||
|
|
78d4873f4f | ||
|
|
ddfd62edf6 | ||
|
|
f65664640a | ||
|
|
492ac7be18 | ||
|
|
5ac77e3cef | ||
|
|
a619022c35 | ||
|
|
49c065c204 | ||
|
|
ddb3980f09 | ||
|
|
75a8f6abc7 | ||
|
|
862253518f | ||
|
|
51ef3eb7e2 | ||
|
|
346e613995 | ||
|
|
25ca9898d5 | ||
|
|
709c095387 | ||
|
|
c262de3833 |
+23
-1
@@ -14,7 +14,8 @@ What is enforced:
|
||||
its owning code (`code:`) -- no development without a design that says where.
|
||||
issues a known `status:`; once `located`, `located-in:` names the owner;
|
||||
once `resolved`, `fixed-by:` says what fixed it (prose counts --
|
||||
"nothing, the capability existed" is an answer).
|
||||
"nothing, the capability existed" is an answer). And no two records share a
|
||||
number -- the number is how a record is cited.
|
||||
research a known `status:`; a `graduated` overview says what it `became:`, and every
|
||||
target it names exists.
|
||||
decisions every accepted record is REACHABLE from the cycle: cited by a design doc's
|
||||
@@ -109,6 +110,27 @@ def main():
|
||||
"without a design that says where" % status)
|
||||
|
||||
# ---- issues ------------------------------------------------------------------------
|
||||
# Two records may not share a number. Numbers are taken as "next free after main", and work
|
||||
# sits on unmerged branches for days -- so two people reading the same main allocate the same
|
||||
# number, and nothing said so. It happened twice in one evening between two machines, and the
|
||||
# second collision landed on main with all three checks passing (issue 155). An issue number is
|
||||
# how every other record cites this one; two records answering to it means a pointer that
|
||||
# resolves to whichever the reader happened to open.
|
||||
seen = {}
|
||||
for folder in sorted(glob.glob(os.path.join(ROOT, "04-ISSUES", "*", ""))):
|
||||
name = os.path.basename(os.path.normpath(folder))
|
||||
number = name.split("-", 1)[0]
|
||||
if not number.isdigit():
|
||||
continue
|
||||
if number in seen:
|
||||
bad(os.path.join("04-ISSUES", name),
|
||||
"is numbered %s, and so is %s -- an issue number is how it is cited, and two "
|
||||
"records answering to one means a citation that resolves to whichever the reader "
|
||||
"opened. Take the next free number across main AND every open pull request"
|
||||
% (number, seen[number]))
|
||||
else:
|
||||
seen[number] = name
|
||||
|
||||
for path in sorted(glob.glob(os.path.join(ROOT, "04-ISSUES", "*", "00-report.md"))):
|
||||
front = frontmatter(path)
|
||||
if front is None:
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
topic: building it
|
||||
status: proposed
|
||||
status: accepted
|
||||
date: 2026-09-01
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
@@ -101,3 +101,19 @@ the digest down after building.
|
||||
**Whether kind 4 deserves a module at all.** Thirty-five descriptions that say *install this and
|
||||
write these files* may be better as one module with settings than as thirty-five modules. Left
|
||||
open deliberately; it is a question about the shape of the catalogue, not about whether to have one.
|
||||
|
||||
## Accepted, 2026-09-29, against what was built
|
||||
|
||||
*Marked in a grooming pass: the mesh was built to this record and the record still said `proposed`.*
|
||||
|
||||
The proposal is the arrangement that exists. `mesh-catalog` holds descriptions of software we did
|
||||
not write and the programs that provision it, and holds neither the mesh's own components nor an
|
||||
application's own module. The mesh's list of modules is a table in the control plane, filled by
|
||||
`module add`, and every module records the source it came from with the commit it was read at.
|
||||
|
||||
**One half is not built: `module check` as a command on the control plane's binary.** A manifest is
|
||||
still validated by a test that reaches into the control plane's internals — which works for this
|
||||
catalogue and gives nothing at all to somebody describing their own application in their own
|
||||
repository, which this record says is the case that matters most. That is
|
||||
[issue 148](../04-ISSUES/148-a-manifest-outside-this-catalogue-has-no-check/00-report.md).
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: proposed
|
||||
status: accepted
|
||||
date: 2026-09-25
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
@@ -275,3 +275,11 @@ On acceptance, each of these is superseded or amended by this record, not edited
|
||||
everything a module needs is a requirement
|
||||
- [Issue 095](../04-ISSUES/095-a-module-assigned-after-genesis-has-no-broker-account/00-report.md),
|
||||
[issue 103](../04-ISSUES/103-a-container-is-not-recreated-when-a-file-it-reads-changes/00-report.md): what fails today
|
||||
|
||||
## Accepted, 2026-09-29, against what was built
|
||||
|
||||
*Marked in a grooming pass.* The vault is a module providing `secret` at mesh scope, and six
|
||||
modules in the catalogue require it — so a shared secret is a requirement answered by the vault,
|
||||
which is what this record asks for. Private keys are still made where they are used and never
|
||||
travel, which is the other half and was never in question.
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: proposed
|
||||
status: accepted
|
||||
date: 2026-09-26
|
||||
deciders: jochen
|
||||
extends: 0112-a-module-definition-names-no-node-mesh-or-path.md
|
||||
@@ -47,3 +47,10 @@ other boundary already is: the module name.
|
||||
node runs one of each (ADR 0115)" — instead of failing on whichever name collides first.
|
||||
- Multi-tenant asks are answered in the catalogue (a second module definition), not in the
|
||||
control plane.
|
||||
|
||||
## Accepted, 2026-09-29, against what was built
|
||||
|
||||
*Marked in a grooming pass.* The rule is enforced where it cannot be forgotten: `assignment`'s
|
||||
primary key is `(node, module)`, so a second assignment of one module to one machine is not a thing
|
||||
the mesh can hold. The record read `proposed` while the schema had already settled it.
|
||||
|
||||
|
||||
@@ -12,7 +12,7 @@ extends: 0102-the-mesh-writes-into-a-shared-file-never-over-it.md
|
||||
## Context
|
||||
|
||||
When a resource stops being declared — its module unassigned, the node sent a
|
||||
deliberately-empty declaration ([issue 127](../04-ISSUES/127-a-declaration-that-shrinks-to-empty-is-skipped-not-sent/00-report.md)),
|
||||
deliberately-empty declaration ([issue 149](../04-ISSUES/149-a-declaration-that-shrinks-to-empty-is-skipped-not-sent/00-report.md)),
|
||||
or a new catalogue version renaming its id — the host undoes it. The host's own code states
|
||||
the rule it means to follow: **it removes what it made and leaves what it merely configured.**
|
||||
For almost every resource it does exactly that:
|
||||
|
||||
@@ -99,6 +99,31 @@ composed, so it is not certified.
|
||||
binding. The per-node source override becomes its reach, widened from the filter alone to the names
|
||||
and the certificate as well.
|
||||
|
||||
## Progressive insight — 2026-09-29, from building it
|
||||
|
||||
**Reach does not mean the same thing to the filter for an endpoint the proxy serves.** The decision
|
||||
above says `internal` means "the filter opens the machine port to the private network" and `public`
|
||||
means "the filter opens it to anywhere". For a routed endpoint the second half is wrong, and
|
||||
[ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md) already said so before this
|
||||
record was written: *a public service is exposed through the proxy, not by opening its own port* — it
|
||||
listens `from: mesh`, only the proxy reaches it, and it is exposed by name.
|
||||
|
||||
Found by trying to express one real module, not by review. Its routed name must be public, because
|
||||
browsers post to it; its machine-side port must not be, because that port serves the dashboard in
|
||||
cleartext. Under one value driving both, saying "public" would have reopened a port an operator had
|
||||
just closed. Measured the same evening: that module's routed name answered from the internet over TLS
|
||||
while its machine-side port was refused from the same place. The port is not the path.
|
||||
|
||||
So the reach of a **routed** endpoint asks for names, and its port keeps what the manifest said. The
|
||||
reach of an **unrouted** endpoint — git over ssh, a mail port, the bus — governs the port, because
|
||||
there is no name and the port is the only way in. That is the same split this record already draws in
|
||||
*an endpoint that is not routed is reached but never named*; what it got wrong was carrying the filter
|
||||
across it.
|
||||
|
||||
This corrects a fact, not the decision: one statement per endpoint, three things derived from it and
|
||||
none of them deciding on its own, all stand. The table in the decision should be read with the filter
|
||||
column applying to an unrouted endpoint.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **A manifest gains endpoint names, and a route contribution names an endpoint instead of a port.**
|
||||
|
||||
@@ -83,6 +83,32 @@ operating system.
|
||||
facts it states about itself. Without it nothing can say a machine is behind, so "every machine
|
||||
current with its source" cannot include the host.
|
||||
|
||||
## Progressive insight — 2026-09-29, the same day
|
||||
|
||||
**The delivery is not "nothing new", and this record said it was.** The decision above stands and is
|
||||
built: versions side by side, the newest runs, the running host stands aside between reconciles, a
|
||||
completed reconcile retires what is older than the predecessor, rollback picks a directory. What was
|
||||
wrong was a claim about how a version reaches a machine. The paragraph on delivery said the
|
||||
declaration "names it like any other archive… nothing new travels, no new resource kind"; the second
|
||||
half is true and the first is not, because two things the delivery needs do not exist:
|
||||
|
||||
- **Nothing can compile it.** A `bundle` artifact is compiled by a closed list of toolchains —
|
||||
typescript and python — whose own comment says adding a language is a decision, because a language
|
||||
used by *modules* needs an SDK carrying the broker client, the event envelope and tool serving. The
|
||||
host uses none of that: it is what applies modules, not one of them. So the obligation that list
|
||||
warns about attaches to a module written in a language, not to the language being buildable, and
|
||||
the control plane — also written in Go — is built as an image from a Dockerfile rather than through
|
||||
a toolchain at all.
|
||||
- **A version cannot reach the path.** An `archive` resource names a fixed path in the manifest, and
|
||||
nothing interpolates the built version into it, so nothing can ask for
|
||||
`…/versions/<version>/`.
|
||||
|
||||
Neither changes what was decided, which options were weighed, or any consequence: the shape is
|
||||
unaffected and the host half is merged and tested. What it changes is the cost, which this record
|
||||
understated as none. The remaining work is a way to build the host and a way to name a version in a
|
||||
path, and until both exist nothing delivers a version and every machine takes the fallback — which is
|
||||
what every machine does today.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The host becomes a build target and a module** — a module whose resource is the next host, applied
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-09-29
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0141-the-host-delivers-its-own-successor.md
|
||||
---
|
||||
|
||||
# 142. The mesh delivers its own components as binaries, not as container images
|
||||
|
||||
## Context
|
||||
|
||||
Measured on the control-node, 2026-09-29:
|
||||
|
||||
| what | how it runs | publishes |
|
||||
|---|---|---|
|
||||
| host | a binary on the machine | — |
|
||||
| controller, catalogue, builder, vault | containers | nothing |
|
||||
| store, registry, broker | containers | ports |
|
||||
|
||||
**The mesh's own software is delivered two ways, and the difference is not a property of the
|
||||
software.** The host and the controller are both written in the same language, both the mesh's own,
|
||||
both doing the mesh's own work. One is an image fetched from a registry. The other is a file somebody
|
||||
copied to four machines, owned by no package, built by nothing
|
||||
([issue 142](../04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/00-report.md)).
|
||||
|
||||
**The reason is not a judgement about either, it is that images are the only delivery that works.**
|
||||
There is no way to put a binary on a machine. The host is hand-copied because of that, and the
|
||||
controller is an image because of that. Neither was chosen on its merits.
|
||||
|
||||
What it costs, all of it measured rather than argued:
|
||||
|
||||
- **Genesis must raise a container runtime before the control plane can exist.** The bundle carries
|
||||
three images and one of them is the controller, *"in the bundle for the same reason they are: there
|
||||
is nothing to fetch it with yet"*
|
||||
([design 07](../03-DESIGN/01-to-be/07-the-foundation.md)). So the hardest moment in the mesh's life
|
||||
has a prerequisite that the thing being started does not need.
|
||||
- **Updating the control plane depends on the control plane.** Its image is fetched from the registry,
|
||||
which is a container the controller manages.
|
||||
- **A change to the host cannot be rolled out at all.** Every machine here runs a byte-identical
|
||||
hand-copied binary. A change merged yesterday reached none of them.
|
||||
- **Compiling the language the mesh is written in is not a capability of the builder.** The bundle
|
||||
toolchains are typescript — real, with a registered base module — and python, which is named in the
|
||||
list and absent from the catalogue. The controller is built as an image from a Dockerfile, which is
|
||||
the per-repository incantation the bundle toolchain exists to abolish
|
||||
([design 18](../03-DESIGN/01-to-be/18-building-a-module.md)).
|
||||
|
||||
The half that *receives* a binary safely is already built and tested
|
||||
([ADR 0141](0141-the-host-delivers-its-own-successor.md)): versions side by side in directories named
|
||||
for them, the newest run, the running one standing aside between reconciles, retirement keeping the
|
||||
predecessor, and a rollback that chooses a directory. What is missing is everything that puts one
|
||||
there.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Leave it as it is.** Rejected: it is not a design, it is the reach of one mechanism. And it is
|
||||
what makes a host change undeliverable.
|
||||
2. **Containerise the host too**, so everything is delivered one way. Rejected: the host is what
|
||||
starts the container runtime and what applies containers. A host in a container is the bootstrap
|
||||
problem made total, and the machine would have no way back from a bad one.
|
||||
3. **Deliver the mesh's components as operating-system packages.** Rejected for the reason
|
||||
[ADR 0141](0141-the-host-delivers-its-own-successor.md) rejected it for the host: a package and a
|
||||
trusted repository per operating system, three of each, and the `package` resource asserts presence
|
||||
and deliberately never a version.
|
||||
4. **Binaries for the mesh's own components, containers for third-party software.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**The mesh's own components are delivered as binaries on the machine.** The host, the controller, the
|
||||
catalogue, the builder, the vault — the software this project writes. They are delivered by the
|
||||
mechanism [ADR 0141](0141-the-host-delivers-its-own-successor.md) built: an archive, fetched by
|
||||
digest, unpacked into a directory named for its version, with the running one standing aside between
|
||||
reconciles and a rollback that chooses the predecessor.
|
||||
|
||||
**Third-party software stays a container.** The store, the registry, the broker. They are somebody
|
||||
else's build, they are already adopted as modules
|
||||
([ADR 0078](0078-the-store-and-broker-are-modules.md)), and an image is the right way to carry
|
||||
somebody else's software. **The container runtime remains required** — modules use it — so this
|
||||
removes a dependency from the control plane, not from the machine.
|
||||
|
||||
**The builder compiles the languages the mesh is written in.** A toolchain for Go, with a base module
|
||||
providing the compiler, exactly as typescript has. The obligation the toolchain list warns about — an
|
||||
SDK carrying the broker client, the envelope and tool serving — attaches to a *module* written in a
|
||||
language, not to the language being compilable. None of these components is a module in that sense;
|
||||
the host is what applies modules.
|
||||
|
||||
**An artifact says what it targets.** A compiled binary is per operating system, pinned at link time
|
||||
([ADR 0005](0005-the-node-host.md)), and a toolchain deliberately takes nothing from the module,
|
||||
because anything a module could override there it would be writing a Dockerfile to override. So the
|
||||
target is a property of the artifact rather than of the recipe, and one artifact declared per target
|
||||
is one build each.
|
||||
|
||||
**A component's version comes from where it sits, not from its linker.** It is unpacked into a
|
||||
directory named for its version, so it can read its own version from its path. The stamp goes, and
|
||||
with it the need for a build to know what it will be called.
|
||||
|
||||
**Genesis carries a binary reference where it carried an image reference.** The principle does not
|
||||
change — the bundle names a thing by digest and the host fetches it, pinned because nothing can
|
||||
resolve a version when no mesh exists — and the container runtime stops being a prerequisite for the
|
||||
control plane. It stays a prerequisite for the store and the broker, which is where it belongs.
|
||||
|
||||
**The order is staged, and each step stands alone.** Compiling Go; an artifact naming its target;
|
||||
delivering a binary; the host as the first component delivered; the controller, catalogue, builder and
|
||||
vault out of their containers; genesis last. Genesis is last for the reason it is always last: it
|
||||
matters for a machine nobody has yet, and every earlier step is provable on a mesh that exists.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **One delivery for the mesh's own software**, so a change to the host ships the way a change to the
|
||||
controller does, and neither is copied by hand.
|
||||
- **The control plane stops depending on a container runtime and on its own registry.** Both remain on
|
||||
the machine for other reasons; neither gates the control plane's own life any more.
|
||||
- **`Replaced()`, the known-good record and the launcher's rollback stop being dead code.** They were
|
||||
written for this and have been called by nothing but their tests.
|
||||
- **Four more components gain a rollback they do not have.** Today a bad controller image is recovered
|
||||
by an operator; under this it is recovered the way a bad host is.
|
||||
- **Two versions of each component occupy disk.** Around nine megabytes each. The predecessor is what a
|
||||
rollback needs.
|
||||
- **Genesis gets smaller, not larger.** One fewer image to carry and one fewer runtime to raise before
|
||||
the control plane.
|
||||
- **This does not make the components smaller or simpler.** They are the same programs; what changes is
|
||||
how they arrive. A reader expecting the containers to have been hiding complexity will not find any.
|
||||
- **What got harder:** the builder gains a language, artifacts gain a target, and the mesh gains a
|
||||
second kind of thing it must deliver correctly — one where getting it wrong takes the control plane
|
||||
down rather than a module. That is why the host is first: it is the component whose recovery is
|
||||
already built and tested.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- **A component is delivered and runs, with nothing copied by hand.** A bed builds the host from its
|
||||
repository, delivers it to a machine running an older one, and the machine reports the new version.
|
||||
This fails today at the first step, because nothing builds it.
|
||||
- **Each target is built once and only the matching one is delivered.** Asserted by declaring an
|
||||
artifact per operating system and checking that a machine is offered the one it can run — a host
|
||||
built for another is what ADR 0005's link-time pin exists to refuse.
|
||||
- **A component reads its version from its path**, asserted by unpacking the same bytes into two
|
||||
differently named directories and seeing each report its own.
|
||||
- **A bad component is rolled back without an operator**, for the host first: a version that will not
|
||||
start is replaced by its predecessor once, and the second failure halts naming the machine.
|
||||
- **The control plane comes up with no registry reachable**, which is the dependency this removes —
|
||||
asserted by raising it with the registry stopped.
|
||||
- **Genesis raises a control plane with no container runtime running**, and raises the store and the
|
||||
broker afterwards. Last, and on a machine with nothing on it.
|
||||
- **A published port count that does not change.** The mesh's own components publish nothing today, so
|
||||
moving them out of containers must not open anything — asserted on the machine's reachable set before
|
||||
and after, which the converge preview already reads.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0141](0141-the-host-delivers-its-own-successor.md) — the receiving half, already built
|
||||
- [ADR 0005](0005-the-node-host.md) — the host, its supervision, and one binary per operating system
|
||||
- [ADR 0078](0078-the-store-and-broker-are-modules.md) — why third-party software stays a container
|
||||
- [ADR 0006](0006-the-substrate-and-the-control-plane.md) — what genesis must raise, and in what order
|
||||
- [issue 142](../04-ISSUES/142-the-host-is-the-one-thing-the-mesh-does-not-deliver/00-report.md) — the
|
||||
measurement that started this
|
||||
- [design 07](../03-DESIGN/01-to-be/07-the-foundation.md) — the bundle's three images, one of them the
|
||||
controller
|
||||
@@ -0,0 +1,135 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: superseded
|
||||
date: 2026-09-29
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0010-delivery.md
|
||||
superseded-by: 02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md
|
||||
---
|
||||
|
||||
# 143. A consumer verifies the grant it is given
|
||||
|
||||
## Context
|
||||
|
||||
A **grant** is what the mesh writes on a consumer's machine so it can reach a provider. The real one
|
||||
the forge receives for its database, as it arrives:
|
||||
|
||||
```
|
||||
provision postgres-database
|
||||
at <the provider's machine, by name>
|
||||
port the machine port the provider is published on
|
||||
as the role the provider created for this consumer
|
||||
```
|
||||
|
||||
with the credential sealed in a separate file. Four facts and a password, and they are the whole
|
||||
mechanism by which anything in the mesh reaches anything else.
|
||||
|
||||
**The mesh asserts that claim and never finds out whether it is true.**
|
||||
[Issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md):
|
||||
converging a machine dropped the path from a container to a port on its own machine, and for eleven
|
||||
hours the mesh answered *all doing what they were told, all heard from, every module current with its
|
||||
source* while a web application logged, six thousand times:
|
||||
|
||||
```
|
||||
connection to server at "<the machine>" (10.10.0.1), port 6852 failed: timeout expired
|
||||
```
|
||||
|
||||
Every check the mesh makes passed, because every check it makes is about the relationship between the
|
||||
mesh and a machine: the declaration was applied, the digest matched, every container named was running.
|
||||
None of them asks whether a consumer can reach what it requires — though the mesh composed the grant
|
||||
and therefore knows the consumer, the machine, the address, the port and the credential.
|
||||
|
||||
**And where the check runs decides whether it catches anything.** The rule in force admitted the
|
||||
machines' own addresses on the private network. A dial from the *machine* to its own address carries
|
||||
exactly such a source address, so a check run by the host on its own behalf would have matched that rule
|
||||
and passed — while every container on the machine was refused. This is inference from the rule that was
|
||||
loaded, not a measurement: the fault was found and fixed before anyone thought to dial from the host.
|
||||
It is enough to decide the question, because a check whose position differs from the consumer's is
|
||||
testing something nobody asked about.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **The control plane dials each provision.** Rejected, and it is the tempting one because the control
|
||||
plane holds every fact. It sits on the provider's machine for most provisions here and reaches the
|
||||
address by a path no consumer uses; in the measured outage it would have passed throughout.
|
||||
2. **The host dials on the consumer's behalf, from the machine.** Rejected for the reason above: the
|
||||
machine's network position is not the consumer's, and the one outage this exists to catch is exactly
|
||||
a difference between them.
|
||||
3. **Ask the module.** Rejected: a module is arbitrary software that the mesh does not write. Some could
|
||||
report on their provisions and most cannot, and a check that covers the modules that opted in tells
|
||||
nobody anything about the rest.
|
||||
4. **Read the module's logs.** Rejected: the failure was in a log the whole time, and reading a module's
|
||||
logs makes the mesh depend on the wording of software it does not control.
|
||||
5. **The consumer verifies it, from its own network position.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**A consumer verifies each grant it is given, from its own network position.** After a reconcile has
|
||||
applied a grant, the machine opens a connection to the address and port that grant names, from inside
|
||||
the consumer's own network namespace — the same position the consumer's software dials from, which is
|
||||
the only position that answers the question the grant asks.
|
||||
|
||||
**It is a connection, not a conversation.** Whether the port accepts a connection is what a grant
|
||||
claims; whether the credential is right, the role exists or the schema is current is the provider's to
|
||||
answer and the consumer's to discover. A check that spoke each provision's protocol would be a second
|
||||
implementation of every provision, and would fail for reasons that are not the mesh's.
|
||||
|
||||
**One failure is not news.** A provider restarting is ordinary, and so is a consumer between containers.
|
||||
A grant is reported unreachable only after it has failed on **consecutive** reconciles, and the count is
|
||||
what the machine reports rather than the last attempt — so a reader can tell "it was briefly away" from
|
||||
"it has never worked".
|
||||
|
||||
**A grant that cannot be checked is said to be unchecked, never assumed good.** A consumer that is not
|
||||
running has no network position to dial from; that is not a broken grant and must not read as one. It is
|
||||
also not a verified grant, and the two are different sentences.
|
||||
|
||||
**What it costs to be wrong is the constraint on all of it.** A check that reports a working provision
|
||||
broken trains a reader to ignore the report, which is worse than having none — the fault this
|
||||
repository keeps finding, one level up. So the threshold is consecutive failures, the check is the
|
||||
cheapest thing that answers the question, and an unknown is reported as unknown.
|
||||
|
||||
**The mesh says it where it says everything else.** A machine's report carries its unreachable grants,
|
||||
and `status` names them beside what is out of date — so "every module current with its source" stops
|
||||
being the whole of what the mesh will tell you about a machine whose modules cannot reach each other.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The mesh can be wrong out loud.** It has been able to assert a grant and not check it; now a grant
|
||||
that does not work is a thing the mesh says, and the eleven hours of issue 145 become minutes.
|
||||
- **The host gains the ability to act from a container's network position**, which it has not needed
|
||||
before. That is a real capability and the only one this needs.
|
||||
- **A machine reports something that is not about the declaration.** Everything it reports today is
|
||||
what it applied and what it holds; this is the first thing it says about whether what it applied
|
||||
works.
|
||||
- **A provision with no port is not checked**, because there is nothing to dial. Several are files and
|
||||
secrets, and saying "checked" about those would be the appearance of verification that this record
|
||||
exists to remove.
|
||||
- **What got harder:** a reconcile does more than apply. Every grant adds a connection attempt on a
|
||||
cadence, which is cheap individually and worth naming: a machine with many consumers dials once per
|
||||
grant per reconcile.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- **The outage is caught.** A bed drops the path from a consumer's network position to a provider's
|
||||
port while leaving the machine's own path to it open — the exact shape of issue 145 — and the grant
|
||||
reads unreachable. This fails against the previous behaviour, where nothing reported anything, and
|
||||
against a check run from the machine, which passes while the consumer cannot reach it.
|
||||
- **A restarting provider is not an outage.** One failed reconcile reports nothing; the count rises and
|
||||
falls, and the grant reads reachable again without anybody acting.
|
||||
- **A consumer that is not running reads unchecked, not broken**, asserted separately from the
|
||||
unreachable case because they are different sentences.
|
||||
- **A provision with no port is not claimed to be checked.**
|
||||
- **The report carries the count, not the last attempt**, so "briefly away" and "never worked" are
|
||||
distinguishable by a reader who sees only the report.
|
||||
- **`status` names an unreachable grant**, asserted on the output, since a check nothing surfaces is
|
||||
the same as no check.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0010](0010-delivery.md) — the declaration is owned resources; a grant is one of them
|
||||
- [ADR 0009](0009-modules-and-the-graph.md) — what a provision and a consumer are
|
||||
- [issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)
|
||||
— the eleven hours
|
||||
- [issue 136](../04-ISSUES/136-a-module-may-name-a-program-the-machine-does-not-have/00-report.md) — the
|
||||
same distance between a declaration and a machine, one level down
|
||||
@@ -0,0 +1,121 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-29
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md
|
||||
supersedes: 02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md
|
||||
---
|
||||
|
||||
# 144. Anything on a machine may call anything on it, and that is the whole of "local"
|
||||
|
||||
## Context
|
||||
|
||||
Everything in the mesh should be able to call:
|
||||
|
||||
- what runs on the same machine;
|
||||
- another machine's service over the private network, if that service is exposed there;
|
||||
- another machine's service over the public network, if it is exposed there.
|
||||
|
||||
Three cases. The filter had two of them.
|
||||
|
||||
**The first was broken and the break was invisible.** A service exposed to the private network rendered
|
||||
as the machines' own addresses on it. A caller on the machine carries such an address; a caller inside
|
||||
one of that machine's containers carries a bridge address and matched nothing. Measured:
|
||||
|
||||
```
|
||||
the machine: local 10.10.0.1 dev lo src 10.10.0.1
|
||||
a container: 10.10.0.1 via 172.17.0.1 dev eth0 src 172.17.0.8
|
||||
```
|
||||
|
||||
Same destination, same machine, two source addresses. The rule named the first and silently refused the
|
||||
second, so a module reaching its database on its own machine's name timed out for eleven hours
|
||||
([issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)).
|
||||
|
||||
**The second case works, and by accident.** A caller on another machine reaches the private network over
|
||||
the tunnel, and arrives carrying that machine's own address — so the rule matches. It would not have
|
||||
matched the caller's own address either; the tunnel rewrites it. That two of three cases worked is why
|
||||
this looked correct.
|
||||
|
||||
**[ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) answered the wrong question.** Written
|
||||
hours earlier, it proposed that a consumer verify each grant it is given by opening a connection from
|
||||
its own network position — and it went to some length about *which* position, because whether a caller
|
||||
sat in a container changed the answer. That difference was the bug. A verification mechanism would have
|
||||
reported this outage sooner and would not have prevented it, and the machinery it needed existed only
|
||||
because the rule was wrong. The remedy for a configuration error is the correct configuration.
|
||||
|
||||
**And a module is not a container.** A module is software that delivers one or more services, and it may
|
||||
do that as a container, an installed package with a unit, a binary, or files something else reads. Of 72
|
||||
modules in the catalogue, 61 happen to use a container and 11 do not — among them the resolver, the ssh
|
||||
daemon and the intrusion-prevention module. A rule that reasons about containers describes most of the
|
||||
mesh and not the mesh.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **A line per service admitting the machine's own callers.** Rejected: it is what was written first,
|
||||
and it only ever covers the services somebody remembered to think about. It also states, service by
|
||||
service, a thing that is true of the machine.
|
||||
2. **Verify each grant from the consumer's position** ([ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md)).
|
||||
Rejected as a remedy: it observes the fault rather than removing it, and the question it agonised over
|
||||
— which network position — exists only while the fault does.
|
||||
3. **Enumerate the addresses a machine's callers may have.** Rejected for the reason no address is named
|
||||
anywhere in this filter any more ([ADR 0140](0140-the-filter-constrains-what-arrives-from-outside.md)):
|
||||
a range describes one machine and goes stale in silence.
|
||||
4. **Local is not filtered, stated once.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**Anything on a machine may call anything on that machine, and the filter says so once.** Not per
|
||||
service, not per port, and not by naming who the callers are: traffic that did not arrive from outside
|
||||
the machine and did not arrive over the private network is the machine's own, and is admitted. It is
|
||||
asked by the link the traffic arrived on, because that is a fact about the machine rather than a list
|
||||
that describes one.
|
||||
|
||||
**Local is not a boundary this mesh draws.** Whether a caller is a container, a unit, or the operator's
|
||||
shell changes nothing, because the thing being decided is "is this the same machine" and the answer does
|
||||
not depend on the form the caller takes.
|
||||
|
||||
**The other two cases are unchanged and are now legible beside it.** A service exposed to the private
|
||||
network admits the machines on it; a service exposed publicly admits anything. Three cases, three lines,
|
||||
and a reader can see all three at once.
|
||||
|
||||
**[ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) is superseded and nothing replaces it.**
|
||||
Whether the mesh should check that a grant works is a real question — it reported this machine healthy
|
||||
for eleven hours — but it is a question about what the mesh can say, not about what it should do, and it
|
||||
must stand on its own rather than as the remedy for a rule that was wrong. It is not built.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The three things everything should be able to call are three lines**, and the first is one line
|
||||
rather than one per service, so a service added tomorrow is reachable locally without anybody
|
||||
remembering to say so.
|
||||
- **A form of module stops mattering to the filter.** The 11 modules that are not containers were never
|
||||
affected by this bug and were never the reason it was hard to see; they are the reason the rule should
|
||||
never have mentioned containers.
|
||||
- **The mesh still cannot say when a grant stops working.** That is the live gap, recorded in issue 145
|
||||
and no longer pretending to have an answer.
|
||||
- **What got harder:** nothing. This removes a line per service and replaces it with one.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- **A caller on the machine reaches a service on it, in the input chain**, asserted on that chain's own
|
||||
body — because the forward chain carries the same line in the same words, and an assertion on the
|
||||
whole rendered file passed with the input chain's copy deleted. That is what
|
||||
[ADR 0137](0137-a-machine-says-which-networks-it-routes.md)'s tests already say to do.
|
||||
- **It is one rule, not one per service.** Asserted by rendering two services of different reach and
|
||||
refusing a per-port local line.
|
||||
- **The three reaches render as three lines**, asserted together, so the whole of what the filter says
|
||||
about who may call what is one test.
|
||||
- **The measured case:** from a container on the machine, a service exposed to the private network on
|
||||
that machine answers. This is the outage, and it fails against the rule this replaces.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md) — the filter is the sum
|
||||
of what its modules listen on
|
||||
- [ADR 0140](0140-the-filter-constrains-what-arrives-from-outside.md) — why no address is named
|
||||
- [ADR 0138](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md) — internal and public,
|
||||
the other two cases
|
||||
- [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) — superseded here
|
||||
- [issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)
|
||||
@@ -0,0 +1,120 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: superseded
|
||||
date: 2026-09-29
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md
|
||||
superseded-by: 02-DECISIONS/0146-connectivity-is-checked-by-name-per-hosting-form.md
|
||||
---
|
||||
|
||||
# 145. A module checks what the mesh claims is reachable, and it checks itself
|
||||
|
||||
## Context
|
||||
|
||||
The mesh asserts three things are callable ([ADR 0144](0144-anything-on-a-machine-may-call-anything-on-it.md)):
|
||||
what runs on the same machine, another machine's service exposed to the private network, and another
|
||||
machine's service exposed publicly. It has never checked any of them.
|
||||
|
||||
[Issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md):
|
||||
the first of the three was broken for eleven hours and the mesh answered *all heard from, every module
|
||||
current with its source* throughout. Every check it makes is about the relationship between the mesh and
|
||||
a machine — applied, current, containers running — and none about whether anything can reach anything.
|
||||
|
||||
**A first answer was drafted and withdrawn.** [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md)
|
||||
put the check inside the host, verifying each grant from the consumer's network position. It was
|
||||
superseded because the difference it worked so hard to reproduce — whether a caller sat in a container —
|
||||
was the bug itself. What survives from it is the part that was right: a check run from the wrong place
|
||||
proves nothing, and the mesh's own reports are not evidence about the network.
|
||||
|
||||
**The mesh already has the shape for this and it is a module.** A module can declare a container that
|
||||
runs on a cadence ([ADR 0053](0053-a-step-that-runs-on-a-schedule.md), and three modules already use
|
||||
`*/5 * * * *`), can be given the mesh's roster as a rendered fact — every machine's name, address and
|
||||
this node's own identity, the same mechanism the resolver and the operator's ssh configuration use — and
|
||||
can emit what it found on the bus. Nothing new is needed to build this except the module.
|
||||
|
||||
**What it must not check is the trap.** The obvious probe target is ssh: present on every machine, never
|
||||
closed by design. Dialling it would have passed throughout the outage, because ssh is admitted
|
||||
unconditionally and the thing that broke was a service exposed to the private network. A checker whose
|
||||
probe is unconditionally open measures the one path that cannot fail, which is the failure this whole
|
||||
sequence keeps producing — a check that reads as verification and verifies nothing.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **The host verifies each grant** ([ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md)).
|
||||
Superseded. It needed the host to act from another network position, which is machinery that exists
|
||||
only while local calls are filtered wrongly.
|
||||
2. **The control plane dials every node.** Rejected: it sits on one machine and reaches the others by a
|
||||
path no ordinary caller uses. It would have passed throughout the outage.
|
||||
3. **Probe an existing service.** Rejected for the target problem above: the services guaranteed on every
|
||||
machine are the ones that are never closed, so they cannot fail the way the mesh fails.
|
||||
4. **A module on every machine that serves its own probe and dials the others'.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**A module runs on every machine, serves an endpoint of its own, and dials every other machine's.** The
|
||||
probe is the module's own endpoint, declared reachable over the private network — so the thing being
|
||||
dialled is admitted by exactly the rule that governs every other internally-exposed service, and fails
|
||||
when that rule is wrong. A second endpoint, declared public, does the same for the public path where a
|
||||
machine has one.
|
||||
|
||||
**It checks the three cases the mesh claims, by name:**
|
||||
|
||||
- its **own machine**, by dialling its own machine's address — the case that broke, and the only one that
|
||||
distinguishes a caller on the machine from a caller in one of its containers;
|
||||
- **each other machine over the private network**;
|
||||
- **each machine's public path**, where one is recorded.
|
||||
|
||||
**It resolves before it dials, and says which failed.** A name that does not resolve and a port that does
|
||||
not answer are different faults with different owners, and a checker that reports one sentence for both
|
||||
sends a reader to the wrong place.
|
||||
|
||||
**It runs where the callers run.** The module's own code in its own container, on the cadence the mesh
|
||||
already has, from the same position as every other module on that machine. It is not the host and not the
|
||||
control plane, and that is the whole point.
|
||||
|
||||
**It says what it found and nothing else.** It emits results; it repairs nothing, opens nothing and holds
|
||||
no credential beyond its own. A checker that fixes things is a second control plane.
|
||||
|
||||
**One failure is not a fault.** A machine rebooting is ordinary. A path is reported broken after it has
|
||||
failed on consecutive runs, and the count travels with the result so a reader can tell "briefly away"
|
||||
from "never worked" — the one thing [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) got
|
||||
right and worth keeping.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The mesh gains the ability to be wrong out loud about the network.** Eleven hours becomes two runs.
|
||||
- **It is a module, so it is assigned, built, pushed and reported on like everything else** — no new host
|
||||
capability, no new vocabulary, nothing in the control plane that has to know about checking.
|
||||
- **Its own endpoint is the instrument.** That is what makes it able to fail; it also means the checker
|
||||
must be assigned to a machine before that machine can be checked, and a machine without it is
|
||||
unchecked rather than healthy.
|
||||
- **It cannot check what it cannot be told.** The roster gives it machines; it does not give it every
|
||||
module's endpoints, so this checks the paths the mesh claims and not every grant in the mesh. That is
|
||||
the honest scope of a first one, and the difference is worth saying rather than growing quietly.
|
||||
- **What got harder:** one more module on every machine, and a module whose whole purpose is to fail
|
||||
visibly when something else is wrong. Its own failures will be read as the mesh's, which is the cost of
|
||||
an instrument.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- **It catches the measured outage.** A bed closes the path from a container to a service exposed to the
|
||||
private network on its own machine — issue 145's shape — and the checker reports its own machine
|
||||
unreachable while every other path still reads reachable. This fails against a probe on a port that is
|
||||
never closed, which is the wrong target this record exists to name.
|
||||
- **A machine rebooting is not a fault**: one failed run reports nothing, the count rises and falls.
|
||||
- **A name that does not resolve is reported as that**, not as a port that did not answer.
|
||||
- **It reports and does not act**: asserted by giving it a broken path and checking nothing on the machine
|
||||
changed.
|
||||
- **A machine without the module reads unchecked**, never healthy — asserted on what the mesh says about
|
||||
a machine it is not assigned to.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0144](0144-anything-on-a-machine-may-call-anything-on-it.md) — the three things that must be callable
|
||||
- [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) — superseded; what survives is that the
|
||||
position matters
|
||||
- [ADR 0053](0053-a-step-that-runs-on-a-schedule.md) — the cadence
|
||||
- [ADR 0138](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md) — internal and public,
|
||||
which the probe endpoints declare
|
||||
- [issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)
|
||||
@@ -0,0 +1,125 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-29
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0145-a-module-checks-what-the-mesh-claims-is-reachable.md
|
||||
supersedes: 02-DECISIONS/0145-a-module-checks-what-the-mesh-claims-is-reachable.md
|
||||
---
|
||||
|
||||
# 146. Connectivity is checked by name, per hosting form, with a valid certificate
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0145](0145-a-module-checks-what-the-mesh-claims-is-reachable.md) decided that a module checks what
|
||||
the mesh claims is reachable, from where the callers are, because the mesh reported four machines healthy
|
||||
for eleven hours while a module could not reach its database
|
||||
([issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)).
|
||||
That decision stands. What it got wrong is everything about *what* is dialled.
|
||||
|
||||
It dialled a raw port on each machine's address. Three things are wrong with that:
|
||||
|
||||
- **A raw port is not how anything in this mesh is reached.** A real caller resolves a name, the proxy
|
||||
answers it, and the proxy reaches the service. A check that dials a port tests the last hop of a path
|
||||
with four hops in it, and the three it skips — resolution, the proxy, the certificate — are where most
|
||||
of the mesh's connectivity actually lives.
|
||||
- **It tested one hosting form.** A module is software that delivers services, and it may deliver them
|
||||
from a container, from a unit the mesh writes for its own code, or from a unit a package ships. Those
|
||||
are three different paths to the same machine, and the outage that produced this was two of them
|
||||
disagreeing. A probe served one way measures one way.
|
||||
- **It said nothing about certificates.** An internal name that resolves, routes and answers over TLS
|
||||
that nothing can verify is not a working path; it is a working path for whoever holds the proxy's
|
||||
trust and nobody else.
|
||||
|
||||
## Decision
|
||||
|
||||
**Each hosting form gets its own endpoint, its own route and therefore its own name.** On every machine:
|
||||
|
||||
| name | what serves it |
|
||||
|---|---|
|
||||
| `connect-docker.<node>.internal` | a container |
|
||||
| `connect-process.<node>.internal` | the mesh's own code, in a unit the mesh writes |
|
||||
| `connect-unit.<node>.internal` | a unit a package ships |
|
||||
|
||||
and the same set under each machine's public domain where it has one — `connect-docker.<domain>` and its
|
||||
siblings. The names are the instrument: a failure reads as *`connect-docker.g14.internal` did not answer*,
|
||||
which says which machine and which hosting form without anybody interpreting anything.
|
||||
|
||||
**Every machine checks every machine, by name, over TLS, verifying the certificate.** Not a port, not an
|
||||
address: resolve the name, connect, complete the handshake, check the certificate against the authority
|
||||
that should have issued it — the mesh's own for an internal name, a public one for a public name. That is
|
||||
the whole path a real caller takes, and each step failing is reported as itself.
|
||||
|
||||
**No name is written anywhere.** The machines come from the roster the mesh already renders as a fact, and
|
||||
the labels are the module's. A machine that joins appears in every other machine's roster on the next
|
||||
push, and they begin checking it without an edit.
|
||||
|
||||
**And the module arrives on a machine because the machine exists, not because somebody assigned it.** A
|
||||
machine that joins and does not have it is worse than unchecked: every other machine is already dialling
|
||||
its names, so it reads as broken everywhere until someone notices. This is the part the mesh cannot
|
||||
currently express — see below — and it is the part that makes the rest safe.
|
||||
|
||||
**What survives from 0145**, unchanged: it reports and repairs nothing; one failure is not a fault and a
|
||||
path is broken after consecutive runs with the count travelling with the result; findings are said on the
|
||||
bus, because a finding in a file on the machine is what this exists to end; and the bus is the one path
|
||||
that cannot report its own failure, so an emit that does not land is written locally and nowhere else.
|
||||
|
||||
## What this needs that the mesh does not have
|
||||
|
||||
Named here rather than assumed, because each is a decision of its own and this record is not the place to
|
||||
make them:
|
||||
|
||||
1. **A module that every machine has.** `ScopeNode` means *at most one holder per node* — an exclusivity
|
||||
rule, not an obligation — and nothing assigns a module at enrolment. Today the resolver, the packet
|
||||
filter, ssh and intrusion prevention are each assigned per machine by hand, which is the same gap
|
||||
wearing different clothes.
|
||||
2. **A container running a module's own bundle.** A `process` runs the mesh's own compiled code with no
|
||||
image; a `container` needs an image of the module's own, which means a Dockerfile — the thing the
|
||||
`bundle` artifact exists to abolish. Nothing in the catalogue runs a bundle in a container, so
|
||||
`connect-docker` has no shape yet.
|
||||
3. **A unit a package ships, for `connect-unit`.** The `service` resource puts an existing unit into a
|
||||
state and deliberately installs none, so this form needs a package that serves a port — and naming a
|
||||
program the machine may not have is
|
||||
[issue 136](../04-ISSUES/136-a-module-may-name-a-program-the-machine-does-not-have/00-report.md).
|
||||
4. **A machine's public domain in the roster fact.** The fact carries each machine's name, mesh name,
|
||||
address and operator account. The public names cannot be composed without the domain.
|
||||
5. **Something that installs the mesh's own root on a machine.** This is
|
||||
[issue 129](../04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md), open
|
||||
since before any of this. Until it is closed, every internal name will fail certificate verification
|
||||
from every machine — correctly, because nothing can verify it. That is the checker working, and it is
|
||||
worth saying in advance so the first run is not read as the checker being broken.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **A failure names the machine and the hosting form.** That is the whole gain over a port: eleven hours
|
||||
became two runs under 0145, and under this it also becomes one line that says where to look.
|
||||
- **The checker surfaces issue 129 immediately**, and will report every internal name unverifiable until
|
||||
it is fixed. A reader must be told that before the first run rather than after.
|
||||
- **Five things must be built before this is what it says it is**, and until they are, what exists is a
|
||||
port dial from one position — useful, and not this.
|
||||
- **What got harder:** a module with three hosting forms of the same trivial service is a strange thing to
|
||||
read. It is justified only because those three forms are how the mesh actually runs software, and a
|
||||
checker that tested one of them would keep the class of outage it exists to catch.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- **A name per hosting form answers from every machine**, asserted by name and not by port.
|
||||
- **A certificate that does not verify is reported as that**, distinctly from a name that does not resolve
|
||||
and a port that does not answer — three faults, three owners.
|
||||
- **A machine that joins is checked by every other machine without an edit**, asserted by adding one to a
|
||||
bed and looking at what the others dial on their next run.
|
||||
- **A machine that joins has the module**, which is gap 1 above and is the assertion that cannot be
|
||||
written yet.
|
||||
- **The measured outage is still caught**: the path from a container to a service on its own machine is
|
||||
closed and `connect-docker.<that node>.internal` fails from that machine while the others still pass.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0145](0145-a-module-checks-what-the-mesh-claims-is-reachable.md) — superseded; its core stands
|
||||
- [ADR 0138](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md) — the two reaches these
|
||||
names come from
|
||||
- [ADR 0066](0066-public-routing-is-name-agnostic.md) — a label plus a domain, which is why no name is written
|
||||
- [issue 129](../04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md) — what the
|
||||
internal names will fail on until it is closed
|
||||
- [issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)
|
||||
@@ -0,0 +1,139 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-29
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md
|
||||
---
|
||||
|
||||
# 147. A module anchors the mesh's authority on a machine, and takes it away again
|
||||
|
||||
## Context
|
||||
|
||||
The mesh runs its own certificate authority and every internal name is served with a certificate
|
||||
from it. No machine trusts it. On an enrolled, adopted workstation — on the private network,
|
||||
resolving through the mesh's resolver — every internal HTTPS name fails verification with
|
||||
*unable to get local issuer certificate*
|
||||
([issue 129](../04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md)).
|
||||
The certificates are genuine; nothing on the machine has ever been told what issued them.
|
||||
|
||||
The authority's only consumer today is a proxy, which fetches the root into a directory of its own
|
||||
and hands it to one program ([ADR 0098](0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md)).
|
||||
That is enough for the proxy and for nothing else: a browser, `git` over HTTPS, `curl`, a package
|
||||
manager and every module that calls another module by an internal name read the machine's trust
|
||||
store, which holds the predecessor's authority and a developer tool's local root, and nothing of
|
||||
the mesh's.
|
||||
|
||||
The predecessor wrote its root into every machine it set up. Removing it was deliberate — an
|
||||
honest failure beats a name that verifies for the wrong reason — and it leaves the mesh with no
|
||||
answer at all until this one lands. It is also what keeps the predecessor alive on the machines
|
||||
that still speak TLS to a mesh name.
|
||||
|
||||
**What makes this a decision rather than a patch** is where the knowledge goes. Two mechanisms in
|
||||
the mesh already write things onto a machine because it is on the private network: `/etc/hosts`
|
||||
and the registry's plaintext trust ([ADR 0082](0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md)).
|
||||
Following that precedent, the controller would inject an anchor into every such machine's
|
||||
declaration, and issue 129 proposed exactly that. It would work. It would also put *where this
|
||||
operating system keeps trust anchors* and *which command refreshes its bundles* into the control
|
||||
plane, for a fact the control plane does not have (the root does not exist until the authority has
|
||||
run) and a machine that may have no reason to verify a mesh name at all.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **The controller injects the anchor into every machine on the private network**, the
|
||||
`/etc/hosts` and insecure-registry shape. Rejected: being on the network is what makes the
|
||||
registry reachable, and that is why network presence is the right trigger *there* — the trust
|
||||
and the reachability are the same fact. Trusting an authority is not the same fact as being
|
||||
able to reach it, and the anchor's path and the bundle refresh are a property of the machine's
|
||||
operating system, which is the host's half of the mesh, not the controller's.
|
||||
2. **A new host primitive — a `trust-anchor` resource type.** Rejected for now, not on principle.
|
||||
The host's vocabulary should grow when a shape cannot be said with what exists, and this one
|
||||
can: a file and a service already express it, as the packet filter proves
|
||||
([ADR 0140](0140-the-filter-constrains-what-arrives-from-outside.md), whose module writes a
|
||||
unit file and a service and nothing else). The primitive becomes right the moment a second
|
||||
operating system is in play, because the anchor directory and the refresh command are exactly
|
||||
the difference `internal/system` exists to hold. Until then it would be a vocabulary word with
|
||||
one speaker.
|
||||
3. **A module that requires the authority, fetches its root, installs it as an anchor and
|
||||
refreshes the machine's bundles — and removes both when it is no longer assigned.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**A machine trusts the mesh's authority because a module put its root there, and stops trusting it
|
||||
when that module is taken away.**
|
||||
|
||||
1. **The module requires `internal-acme-ca`** and reads the provider's bound address and the path
|
||||
it serves its root at. It requires nothing else and provides nothing: it is a consumer of the
|
||||
authority like any other.
|
||||
2. **It fetches the root over the mesh's own network, without prior trust**, because there is no
|
||||
prior trust to have — this is the module that establishes it — and the network is what
|
||||
authenticates the fetch ([ADR 0098](0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md),
|
||||
the same reasoning that lets the proxy fetch it). What it accepts is checked: a body that is
|
||||
not a certificate fails, and the failure is the module's, not a later handshake's.
|
||||
3. **It installs the root where this machine's TLS clients look, and refreshes the extracted
|
||||
bundles** — the command that does the refresh is an ordinary part of the unit that places the
|
||||
anchor, not a new thing the mesh can be asked to do.
|
||||
4. **Removal is symmetric and is the same unit's business.** Undeclared, the host stops the unit;
|
||||
stopping it removes the anchor and refreshes the bundles again. A machine that leaves the mesh
|
||||
stops trusting the mesh, without anybody remembering to go and look.
|
||||
5. **It is an ordinary assignment.** No machine is given it automatically. A machine that verifies
|
||||
a mesh name is assigned it, and a machine that does not is not — which is the same statement
|
||||
the mesh already makes about every other module, and is why this is not the controller's
|
||||
business.
|
||||
|
||||
**One operating system, said out loud.** The anchor directory and the refresh command in the
|
||||
module today are Arch's. On a machine that is not Arch the unit fails, visibly, rather than
|
||||
writing a file nothing reads. That is the accurate failure, and it is the signal that option 2
|
||||
above has become right.
|
||||
|
||||
## How this is checked
|
||||
|
||||
- **The verification that could not succeed before.** On a machine holding the module, a plain
|
||||
client fetches an internal HTTPS name with no `-k` and no bundle argument and verifies. On a
|
||||
machine without it, the same fetch fails with *unable to get local issuer certificate*. Both
|
||||
halves, because only the pair distinguishes "the anchor works" from "something else already
|
||||
trusted it".
|
||||
- **The removal half, in the same bed:** unassign the module, refetch, and the failure returns.
|
||||
Checking only the arrival is how a trust store fills up with authorities nobody can account for.
|
||||
- **What is deliberately not checked here:** that the authority issues, that a name resolves, that
|
||||
the proxy serves. Those have their own beds, and this module's bed passing for those reasons is
|
||||
the failure mode this record is most exposed to — which is why the negative half is not optional.
|
||||
|
||||
**What this bed is dialled at, and why it is the authority itself.** The authority serves its own
|
||||
API with a certificate it issued, so the handshake under test needs nothing else in the mesh to be
|
||||
right. A trust bed that reached for a routed name through the proxy would be passing or failing for
|
||||
the proxy's reasons and the resolver's.
|
||||
|
||||
**Written, and not yet run** *(2026-09-29)*. The bed is `trust-anchor` in the lab, and it cannot
|
||||
execute: raising a foundation fails before any module is reached, in both bundles that exist
|
||||
([issue 146](../04-ISSUES/146-the-foundation-cannot-be-raised-on-the-bus-the-mesh-runs-on/00-report.md)).
|
||||
So what stands behind this record today is the rendering — the script the machine would run names
|
||||
the authority it was bound to, checked in the control plane's own test suite — and **not** a machine
|
||||
that verified anything. That is a weaker thing than the paragraph above describes, and it stays
|
||||
written this way until the bed runs.
|
||||
|
||||
## Consequences
|
||||
|
||||
The predecessor's authority can be retired from a machine once this module is assigned to it,
|
||||
which is the first time that has been true. `git` over HTTPS to the mesh's forge starts working,
|
||||
so the ssh-only clone URL stops being a rule. A module on any machine can call another module's
|
||||
internal name and verify it.
|
||||
|
||||
What got harder: one more module to assign to a machine that needs it, and the machine's trust
|
||||
store now changes when an assignment changes — which is the point, and is also a thing an operator
|
||||
can be surprised by. The fetch without prior trust is the same exposure ADR 0098 accepted, now on
|
||||
every machine that holds the module rather than only where a proxy runs: anything that can stand
|
||||
in the middle of the mesh's own network at the moment of the fetch can be believed. The mesh
|
||||
already treats that network as the thing it authenticates.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 129](../04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md) —
|
||||
the symptom and the evidence.
|
||||
- [ADR 0098](0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md) — a fact made at
|
||||
first start is fetched from its provider; this extends it from one program to the machine.
|
||||
- [ADR 0082](0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md) — the precedent
|
||||
this deliberately does not follow, and why it is right where it is.
|
||||
- [ADR 0005](0005-the-node-host.md) — the host is where one operating system's difference lives.
|
||||
- [`03-DESIGN/01-to-be/08-connectivity.md`](../03-DESIGN/01-to-be/08-connectivity.md).
|
||||
@@ -142,6 +142,7 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0131** — [Everything on the mesh speaks to the broker seat, and AMQP is not a provision](0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)
|
||||
- **0132** — [A seat carries the tools its holder must serve](0132-a-seat-carries-the-tools-its-holder-must-serve.md)
|
||||
- **0134** — [The mesh says what it applied](0134-the-mesh-says-what-it-applied.md)
|
||||
- **0142** — [The mesh delivers its own components as binaries, not as container images](0142-the-mesh-delivers-its-own-components-as-binaries.md)
|
||||
|
||||
### Its tiers, from the bottom up
|
||||
|
||||
@@ -206,9 +207,9 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0099** — [A step that runs once names what it reads, and runs again when it changed](0099-a-step-that-runs-once-names-what-it-reads.md)
|
||||
- **0110** — [A seat is held by one assignment, from a closed set, and it may deliver a provision](0110-a-seat-is-a-module-assignment-from-a-closed-set.md)
|
||||
- **0112** — [A module definition names no node, no mesh and no path: everything it needs is a requirement the mesh resolves](0112-a-module-definition-names-no-node-mesh-or-path.md)
|
||||
- **0113** — [The vault makes every shared secret, a provider makes resources and data, and the mesh carries both](0113-the-vault-makes-every-secret.md) *(proposed)*
|
||||
- **0113** — [The vault makes every shared secret, a provider makes resources and data, and the mesh carries both](0113-the-vault-makes-every-secret.md)
|
||||
- **0114** — [A credential two parties hold rotates over two credentials; one a single party holds rotates in place, staged; and retiring a credential never removes what it reached](0114-a-shared-credential-rotates-over-two-credentials.md) *(proposed)*
|
||||
- **0115** — [One assignment of a module per node: the module's name is the assignment's identity](0115-one-assignment-of-a-module-per-node.md) *(proposed)*
|
||||
- **0115** — [One assignment of a module per node: the module's name is the assignment's identity](0115-one-assignment-of-a-module-per-node.md)
|
||||
- **0117** — [A machine's uplink is a seat: the mesh configures the manager, never the link](0117-a-machines-uplink-is-a-seat.md)
|
||||
- **0118** — [Undeclaring removes what the mesh made, and gives a unit back the state it was found in](0118-undeclaring-gives-a-unit-back-the-state-it-was-found-in.md)
|
||||
- **0120** — [A roster fact carries its format as a template: the mesh owns the data, the module owns the format](0120-a-roster-fact-carries-its-format-as-a-template.md)
|
||||
@@ -222,6 +223,11 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0139** — [A network is forwarded because a module declared it](0139-a-network-is-forwarded-because-a-module-declared-it.md) *(superseded)*
|
||||
- **0140** — [The filter constrains what arrives from outside, and says nothing about a machine's own guests](0140-the-filter-constrains-what-arrives-from-outside.md)
|
||||
- **0141** — [The host delivers its own successor, and versions live side by side](0141-the-host-delivers-its-own-successor.md)
|
||||
- **0143** — [A consumer verifies the grant it is given](0143-a-consumer-verifies-the-grant-it-is-given.md) *(superseded)*
|
||||
- **0144** — [Anything on a machine may call anything on it, and that is the whole of "local"](0144-anything-on-a-machine-may-call-anything-on-it.md)
|
||||
- **0145** — [A module checks what the mesh claims is reachable, and it checks itself](0145-a-module-checks-what-the-mesh-claims-is-reachable.md) *(superseded)*
|
||||
- **0146** — [Connectivity is checked by name, per hosting form, with a valid certificate](0146-connectivity-is-checked-by-name-per-hosting-form.md)
|
||||
- **0147** — [A module anchors the mesh's authority on a machine, and takes it away again](0147-a-module-anchors-the-meshs-authority.md)
|
||||
|
||||
### How it is built
|
||||
|
||||
@@ -231,7 +237,7 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0014** — [No workspace — each module is a standalone package consuming published dependencies](0014-no-npm-workspace.md)
|
||||
- **0015** — [Applications live in their own repository; the monorepo is for the mesh](0015-applications-live-in-their-own-repository.md)
|
||||
- **0016** — [The lab](0016-the-lab.md)
|
||||
- **0037** — [Where a module lives](0037-where-a-module-lives.md) *(proposed)*
|
||||
- **0037** — [Where a module lives](0037-where-a-module-lives.md)
|
||||
- **0039** — [What the SDK holds, and what it refuses](0039-what-the-sdk-holds-and-refuses.md)
|
||||
- **0068** — [The lab takes requests, one at a time, and runs each from its own copy](0068-the-lab-takes-requests.md) *(proposed)*
|
||||
- **0069** — [A module is a repository and a path within it](0069-a-module-is-a-repository-and-a-path.md)
|
||||
|
||||
@@ -11,8 +11,9 @@ code:
|
||||
- mesh-catalog modules/postgres
|
||||
- mesh-catalog modules/lavinmq
|
||||
- mesh-lab test/integration/mesh.test.ts (a bare machine becomes a mesh)
|
||||
updated: 2026-09-22
|
||||
updated: 2026-09-29
|
||||
decisions:
|
||||
- 02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md
|
||||
- 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
|
||||
- 02-DECISIONS/0088-the-foundation-filters-before-anything-listens.md
|
||||
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
|
||||
@@ -314,4 +315,24 @@ of a database and pushed to over the broker. What arrived and what did not is th
|
||||
|
||||
**One fault, and it was in the joining.** The token did not say what the mesh calls the machine,
|
||||
so enrolment needed a flag its own help said it did not — and failed at the broker with an empty
|
||||
username. Recorded in ADR 0004 as the fifth thing a token carries.
|
||||
username. Recorded in ADR 0004 as the fifth thing a token carries.
|
||||
|
||||
## The mesh's own components arrive as binaries
|
||||
|
||||
*2026-09-29 —
|
||||
[ADR 0142](../../02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md).*
|
||||
|
||||
The bundle carries three images and one of them is the controller, *because there is nothing to fetch
|
||||
it with yet*. That reasoning holds and its conclusion changes: the controller is carried as a **binary**
|
||||
reference rather than an image reference, pinned by digest exactly as before. Nothing about the bundle's
|
||||
shape moves — it names a thing and the host fetches it — and the container runtime stops being something
|
||||
genesis must raise before the control plane can exist. It still raises one, for the store and the broker,
|
||||
which is where somebody else's software belongs.
|
||||
|
||||
The mesh's own components — the host, the controller, the catalogue, the builder, the vault — are
|
||||
delivered as binaries into directories named for their versions, by the mechanism
|
||||
[ADR 0141](../../02-DECISIONS/0141-the-host-delivers-its-own-successor.md) describes. Third-party
|
||||
software stays a container. The split is not about isolation; it is about who built the thing.
|
||||
|
||||
Measured before deciding it: the mesh's own components publish no ports at all, so this opens nothing.
|
||||
Only the store, the registry and the broker publish, and they are staying as they are.
|
||||
|
||||
@@ -7,8 +7,9 @@ code:
|
||||
- mesh-controller internal/identity/authority.go
|
||||
- mesh-host internal/identity/serving.go
|
||||
- mesh-host internal/apply (the service that reflects a rule set)
|
||||
updated: 2026-09-28
|
||||
updated: 2026-09-29
|
||||
decisions:
|
||||
- 02-DECISIONS/0147-a-module-anchors-the-meshs-authority.md
|
||||
- 02-DECISIONS/0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md
|
||||
- 02-DECISIONS/0140-the-filter-constrains-what-arrives-from-outside.md
|
||||
- 02-DECISIONS/0104-a-provision-may-be-answered-by-an-adapter-to-the-predecessor.md
|
||||
@@ -710,6 +711,22 @@ step, so when the authority moves the root is fetched again and the proxy is rec
|
||||
*How it is checked:* the route-forwarding bed installs the authority, the proxy and a consumer
|
||||
from the catalogue and asserts the routed name is served.
|
||||
|
||||
**And a machine trusts that authority because a module put its root in its trust store**
|
||||
([ADR 0147](../../02-DECISIONS/0147-a-module-anchors-the-meshs-authority.md)). The proxy's fetch
|
||||
answers for the proxy and for nothing else: a browser, `git` over HTTPS, a package manager and
|
||||
every module calling another by an internal name read the machine's own trust store, and the mesh
|
||||
had never written anything there. A module requiring the authority does the whole of it — fetch
|
||||
the root over the mesh network, place it where this machine's TLS clients look, refresh the
|
||||
extracted bundles — and stopping it, which is what being unassigned does, takes the anchor away
|
||||
and refreshes them again. Not the controller's business, because being on the private network is
|
||||
what makes the authority *reachable* and is not the same fact as having a reason to *verify* a
|
||||
mesh name; and because where anchors live and which command refreshes them is one operating
|
||||
system's difference, which is the host's half of the mesh
|
||||
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)).
|
||||
*How it is checked:* on a machine holding the module a plain client verifies an internal HTTPS
|
||||
name with no bundle argument, and on one without it the same fetch fails to find an issuer — both
|
||||
halves, because only the pair tells the anchor apart from something that already trusted it.
|
||||
|
||||
### What was built
|
||||
|
||||
*2026-08-31.*
|
||||
|
||||
@@ -5,7 +5,7 @@ code:
|
||||
- mesh-controller internal/builder
|
||||
- mesh-controller cmd/mesh-controller (build, build --behind, push, status)
|
||||
- mesh-controller internal/inventory/builds.go
|
||||
updated: 2026-09-21
|
||||
updated: 2026-09-29
|
||||
decisions:
|
||||
- 02-DECISIONS/0090-a-failure-that-repeats-is-said-to-be-stuck.md
|
||||
- 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md
|
||||
@@ -226,3 +226,34 @@ when the current failure began and how many reports in a row have said it — th
|
||||
id, whatever the words; three make the machine stuck, and `status` says so beside the failure. The host keeps trying — stuck is what the mesh
|
||||
knows, not what the machine is told. *How it is checked:* an inventory test counts three identical
|
||||
reports, a different one, and a clean apply; the status test asserts the word appears.
|
||||
|
||||
## Everything may call what is exposed to it, and local is not a boundary
|
||||
|
||||
*2026-09-29, from an outage that ran eleven hours —
|
||||
[issue 145](../../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md),
|
||||
settled by [ADR 0144](../../02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md).*
|
||||
|
||||
A grant is four facts and a credential: the provision, the machine, the port, and who the consumer is
|
||||
when it connects. It is the whole mechanism by which anything in the mesh reaches anything else, and it
|
||||
rests on three things being callable — what runs on the same machine, another machine's service over the
|
||||
private network where it is exposed there, and another machine's service over the public network where it
|
||||
is exposed there.
|
||||
|
||||
The filter had two of those. A service exposed to the private network admitted the machines' own addresses
|
||||
on it; a caller on the machine carries such an address, and a caller inside one of that machine's
|
||||
containers carries a bridge address and matched nothing. Measured, same destination and same machine:
|
||||
`src 10.10.0.1` from the machine, `src 172.17.0.8` from a container on it. So a module reaching its
|
||||
database on its own machine's name timed out for eleven hours while the mesh called the machine healthy.
|
||||
|
||||
The second case worked by accident: a caller on another machine arrives over the tunnel carrying that
|
||||
machine's address, which the rule matched. Two of three working is why this read as correct.
|
||||
|
||||
**So local is not a boundary this mesh draws, and the filter says so once.** Traffic that did not arrive
|
||||
from outside the machine and did not arrive over the private network is the machine's own, and is
|
||||
admitted — for every service there, not per service. Whether the caller is a container, a unit or a shell
|
||||
decides nothing, because the question is "is this the same machine".
|
||||
|
||||
A verification mechanism was drafted for this and withdrawn. It would have reported the outage sooner and
|
||||
would not have prevented it, and the part of it that was hard — deciding which network position to check
|
||||
from — existed only because the rule was wrong. Whether the mesh should check that a grant works is still
|
||||
open, in issue 145; it is not the remedy for a configuration error.
|
||||
|
||||
@@ -5,8 +5,9 @@ code:
|
||||
- mesh-controller cmd/mesh-builder
|
||||
- mesh-controller internal/builder
|
||||
- mesh-catalog modules/builder
|
||||
updated: 2026-09-25
|
||||
updated: 2026-09-29
|
||||
decisions:
|
||||
- 02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md
|
||||
- 02-DECISIONS/0111-a-build-source-is-on-the-git-seat-or-external.md
|
||||
- 02-DECISIONS/0097-a-vendor-image-is-a-declared-build-input.md
|
||||
- 02-DECISIONS/0096-an-upstream-image-is-copied-between-registries.md
|
||||
@@ -263,3 +264,27 @@ ships one and wrong for code the mesh built, which has no unit until the mesh wr
|
||||
|
||||
**Tools, hooks and consumers are not further modes**, which is the test of whether three is the
|
||||
right number: they are loaded by a tool host, and a tool host is a process that stays up.
|
||||
|
||||
## The builder compiles the languages the mesh is written in
|
||||
|
||||
*2026-09-29 —
|
||||
[ADR 0142](../../02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md).*
|
||||
|
||||
The toolchain list was typescript and python, and only typescript had a base module in the catalogue.
|
||||
Meanwhile the control plane — written in the language this project is mostly written in — was built as
|
||||
an image from a hand-written Dockerfile, which is the per-repository incantation this whole mechanism
|
||||
exists to abolish.
|
||||
|
||||
So the list gains Go, with a base module providing the compiler exactly as typescript has one. The
|
||||
obligation the list's own comment warns about — an SDK carrying the broker client, the event envelope
|
||||
and tool serving — attaches to a **module** written in a language, not to the language being
|
||||
compilable. The mesh's own components are not modules in that sense; the host is what applies modules.
|
||||
|
||||
**And an artifact says what it targets.** A compiled binary is per operating system, pinned at link
|
||||
time, and a toolchain deliberately accepts nothing from the module — anything a module could override
|
||||
there it would be writing a Dockerfile to override. The target is therefore a property of the artifact,
|
||||
not of the recipe: one artifact declared per target, one build each.
|
||||
|
||||
A component's version stops being stamped in at link time. It is unpacked into a directory named for
|
||||
its version, so it reads its version from its own path, and a build no longer has to know what it will
|
||||
be called.
|
||||
|
||||
@@ -155,3 +155,17 @@ design document here, and get it back. That check fails today by design.
|
||||
|
||||
**What stands until then** is the signpost, and the honest description of it: reachable, not
|
||||
surfacing.
|
||||
|
||||
## Where this stands, 2026-09-29
|
||||
|
||||
*Added in a grooming pass.* The knowledge base this record is about is the **predecessor's**, and it
|
||||
is no longer reachable from anything: the surface that answered `recall_search` speaks the transport
|
||||
the mesh removed at the cut-over
|
||||
([issue 147](../147-the-operators-tools-still-dial-the-bus-that-was-removed/00-report.md)).
|
||||
|
||||
So the sentence in `README.md` that this record catches — *these documents are still indexed into
|
||||
the knowledge base* — is now wrong twice over: nothing indexed them, and there is nothing to index
|
||||
them into. The record stays open, and its answer is no longer "index this repository somewhere"; it
|
||||
is whatever the mesh grows as its own knowledge surface, if it grows one. Until then the honest fix
|
||||
is the README, which should stop claiming a property nothing provides.
|
||||
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
---
|
||||
status: open
|
||||
status: resolved
|
||||
opened: 2026-09-22
|
||||
located-in: []
|
||||
fixed-by:
|
||||
located-in: [mesh-catalog modules/gitea, mesh-controller internal/catalogue/declaration.go]
|
||||
fixed-by: mesh-controller 7352c84, merged in #46 — a module is told its port in a container's environment too, as a file already was
|
||||
amended-design:
|
||||
---
|
||||
|
||||
@@ -39,3 +39,9 @@ the assignment happens to differ.
|
||||
- Should composition refuse an environment value that names a port the module does not fix, the
|
||||
way it refuses other claims a module cannot make?
|
||||
- Which other modules write their own address, with a port, into their environment?
|
||||
|
||||
## Closed
|
||||
|
||||
*2026-09-29, in a grooming pass rather than by whoever fixed it.* The forge's address follows a moved port the same way every other reader does. Found by
|
||||
reading what the code repositories' commits cite: the fix names this issue and is on `main`. It was
|
||||
not re-verified on a machine, and this record says so rather than implying a run that did not happen.
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
---
|
||||
status: open
|
||||
status: resolved
|
||||
opened: 2026-09-22
|
||||
located-in: []
|
||||
fixed-by:
|
||||
located-in: [mesh-controller internal/catalogue/declaration.go, mesh-catalog (every routed module)]
|
||||
fixed-by: mesh-controller bdf965d (a route names the endpoint it serves) with `portOfEndpoint` and `AtPublishedPort` — the contribution carries the endpoint's declared port and the machine-side redirection is applied to it like any other
|
||||
amended-design:
|
||||
---
|
||||
|
||||
@@ -45,3 +45,9 @@ precisely because the predecessor holds the usual one.
|
||||
keeping the mapping out of rendered configuration?
|
||||
- What should refuse a declaration whose contributed route names a port nothing on that node
|
||||
listens on?
|
||||
|
||||
## Closed
|
||||
|
||||
*2026-09-29, in a grooming pass rather than by whoever fixed it.* A route names an endpoint rather than a port, and the redirection that turns a declared port into the published one is applied to contributions too. Found by
|
||||
reading what the code repositories' commits cite: the fix names this issue and is on `main`. It was
|
||||
not re-verified on a machine, and this record says so rather than implying a run that did not happen.
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
---
|
||||
status: open
|
||||
status: resolved
|
||||
opened: 2026-09-24
|
||||
located-in: [mesh-controller module.json, mesh-host internal/apply]
|
||||
fixed-by:
|
||||
fixed-by: ADR 0142 — the mesh's own components are delivered as binaries on the machine, so the controller is a process; the delivery itself is issue 142
|
||||
amended-design:
|
||||
---
|
||||
|
||||
@@ -84,3 +84,21 @@ restart and run-to-completion semantics — so this would not need host-side wor
|
||||
|
||||
The two do not collapse into one. The controller is not a code-carrying sidecar, and `network: host`
|
||||
is what makes the asymmetry visible here and nowhere else.
|
||||
|
||||
## Answered
|
||||
|
||||
*2026-09-29, in a grooming pass.* This asked a question rather than reporting a defect, and the
|
||||
question was taken: [ADR 0142](../../02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md)
|
||||
decides that the mesh's own components — the host, the controller, the catalogue, the builder, the
|
||||
vault — are **binaries on the machine**, delivered by the mechanism
|
||||
[ADR 0141](../../02-DECISIONS/0141-the-host-delivers-its-own-successor.md) built, and that
|
||||
third-party software (the store, the registry, the broker) stays a container because an image is the
|
||||
right way to carry somebody else's build.
|
||||
|
||||
So the operating experience this record was written from — every mutating command reached through
|
||||
`docker exec mesh-controller` — is answered, and answered against the container.
|
||||
|
||||
**The delivery is a separate matter and is not this record's.** Step 1 of it is built and no
|
||||
component travels yet; that is
|
||||
[issue 142](../142-the-host-is-the-one-thing-the-mesh-does-not-deliver/00-report.md).
|
||||
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
---
|
||||
status: located
|
||||
status: resolved
|
||||
opened: 2026-09-26
|
||||
located-in: [mesh-sdk src/provisioner, mesh-catalog modules/redis]
|
||||
fixed-by:
|
||||
fixed-by: mesh-catalog bbda88c, merged in #84 — every credential provider says whether it still holds a consumer
|
||||
amended-design:
|
||||
---
|
||||
|
||||
@@ -60,3 +60,9 @@ checks it after the first pass.
|
||||
instance and leaves the gap for the others.
|
||||
- Where does the record of what was applied live, if not in memory? ADR 0114, still
|
||||
proposed, puts rotation state with the vault. The same place may answer this.
|
||||
|
||||
## Closed
|
||||
|
||||
*2026-09-29, in a grooming pass rather than by whoever fixed it.* A provisioner asks the backend what is there rather than trusting what it remembers doing. Found by
|
||||
reading what the code repositories' commits cite: the fix names this issue and is on `main`. It was
|
||||
not re-verified on a machine, and this record says so rather than implying a run that did not happen.
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
status: located
|
||||
status: resolved
|
||||
fixed-by: mesh-host 1cb8953 and fdc768c — the mesh writes into a marked block of a text file instead of over it
|
||||
opened: 2026-09-26
|
||||
located-in: [mesh-controller internal/catalogue/facts.go, mesh-host internal/apply]
|
||||
---
|
||||
@@ -71,3 +72,9 @@ private network loses that name too.
|
||||
- The host's file resource supports `into: "json"` only; anything else is a whole write.
|
||||
- `node show <node>` on the adopted workstation: `holds file /etc/hosts
|
||||
mesh-wireguard.fact-node-names`, original kept.
|
||||
|
||||
## Closed
|
||||
|
||||
*2026-09-29, in a grooming pass rather than by whoever fixed it.* A shared hosts file keeps every line that is not the mesh's. Found by
|
||||
reading what the code repositories' commits cite: the fix names this issue and is on `main`. It was
|
||||
not re-verified on a machine, and this record says so rather than implying a run that did not happen.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
status: open
|
||||
status: located
|
||||
opened: 2026-09-26
|
||||
located-in: [mesh-controller, mesh-catalog step-ca]
|
||||
located-in: [mesh-catalog ca-trust]
|
||||
---
|
||||
|
||||
# 129 — nothing makes a machine trust the mesh's own certificate authority
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# Diagnosis
|
||||
|
||||
*2026-09-29.*
|
||||
|
||||
## What was ruled out
|
||||
|
||||
**That something already carries the root and it is only misplaced.** It does not. The authority
|
||||
serves its root at a path beside its ACME directory, and the one thing that fetches it — the route
|
||||
proxy — puts it in a directory of its own and hands it to one program. Nothing has ever written
|
||||
into a machine's trust store. Measured on three converged machines: the anchors present are the
|
||||
predecessor's authority and a developer tool's local root, and on the machines where the
|
||||
predecessor's was deliberately removed, every internal name fails verification.
|
||||
|
||||
**That the private network could carry it, the way it carries the registry's trust.** That is what
|
||||
the report proposed, and it was rejected on consideration rather than on difficulty
|
||||
([ADR 0147](../../02-DECISIONS/0147-a-module-anchors-the-meshs-authority.md), option 1): being on
|
||||
the network is what makes the registry *reachable* and is therefore the right trigger there, while
|
||||
trusting an authority is a separate fact from being able to reach it. The anchor's directory and
|
||||
the command that refreshes the extracted bundles are also one operating system's difference, which
|
||||
is the host's half of the mesh and not the controller's.
|
||||
|
||||
**That it needs a new host resource type.** It does not, today. A file and a service say the whole
|
||||
of it, which the packet filter already proves. The primitive becomes the right answer when a second
|
||||
operating system is in play, and not before.
|
||||
|
||||
## Where it belongs
|
||||
|
||||
A module in the catalogue: it requires `internal-acme-ca`, fetches the root over the mesh's own
|
||||
network, installs it as a trust anchor, refreshes the machine's bundles, and — because being
|
||||
unassigned stops its unit, and stopping the unit is what undoes it — takes both away again.
|
||||
|
||||
The owner is therefore `mesh-catalog`, module `ca-trust`, and nothing in the control plane.
|
||||
|
||||
## The module exists, and this stays open until a machine holds it
|
||||
|
||||
*2026-09-29.* `ca-trust` is in the catalogue and merged
|
||||
([ADR 0147](../../02-DECISIONS/0147-a-module-anchors-the-meshs-authority.md)), and what it renders
|
||||
is checked in the control plane's own suite: the script fetches from the authority it was bound to,
|
||||
and the unit runs it both ways.
|
||||
|
||||
**No machine has been assigned it, and nothing has verified a name because of it.** The bed written
|
||||
for that cannot run ([issue 146](../146-the-foundation-cannot-be-raised-on-the-bus-the-mesh-runs-on/00-report.md)),
|
||||
and the live mesh has not been given the module. So the symptom this record opened on — every
|
||||
internal name failing verification on every machine — is still true everywhere, and the record stays
|
||||
`located` until it is not. Closing it on a module that exists would be closing it on an intention.
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
status: located
|
||||
status: resolved
|
||||
fixed-by: mesh-host 3112c88 — undeclaring gives a unit back the state it was found in, and removes only a process the mesh made
|
||||
opened: 2026-09-27
|
||||
located-in: [mesh-host internal/apply/apply.go (remove)]
|
||||
amended-design: 02-DECISIONS/0118-undeclaring-gives-a-unit-back-the-state-it-was-found-in.md
|
||||
@@ -15,7 +16,7 @@ found that the host's `remove` path stops every `service` resource that is no lo
|
||||
delete". `store.Orphans` matches by id alone. So any of these stops the unit:
|
||||
|
||||
- the module is unassigned — by mistake, or to switch it for another;
|
||||
- the node is sent a deliberately-empty declaration ([issue 127](../127-a-declaration-that-shrinks-to-empty-is-skipped-not-sent/00-report.md));
|
||||
- the node is sent a deliberately-empty declaration ([issue 149](../149-a-declaration-that-shrinks-to-empty-is-skipped-not-sent/00-report.md));
|
||||
- a later catalogue version renames the resource's `id`.
|
||||
|
||||
That is right for a service the mesh brought into being. It is wrong for a unit the mesh
|
||||
@@ -64,3 +65,9 @@ something to settle in passing.
|
||||
|
||||
The unassign preview is partly answered — the host's plan names each unit it will stop — and the
|
||||
controller's side is left open.
|
||||
|
||||
## Closed
|
||||
|
||||
*2026-09-29, in a grooming pass rather than by whoever fixed it.* Undeclaring no longer stops a unit the mesh only reloaded or only kept running. Found by
|
||||
reading what the code repositories' commits cite: the fix names this issue and is on `main`. It was
|
||||
not re-verified on a machine, and this record says so rather than implying a run that did not happen.
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
status: located
|
||||
status: resolved
|
||||
opened: 2026-09-28
|
||||
located-in:
|
||||
- mesh-controller internal/catalogue/manifest.go
|
||||
@@ -7,7 +7,7 @@ located-in:
|
||||
- mesh-controller internal/catalogue/declaration.go
|
||||
- mesh-controller examples/route-proxy
|
||||
- mesh-catalog (every routed module manifest)
|
||||
fixed-by:
|
||||
fixed-by: mesh-controller bdf965d (a module names its endpoints) and c68d3a7 (an assignment configures an endpoint as one thing) — the filter, the proxy's names and both authorities now read one statement
|
||||
amended-design: 03-DESIGN/01-to-be/08-connectivity.md
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
# Resolution
|
||||
|
||||
*2026-09-29.*
|
||||
|
||||
**Built, and this record did not say so.** The issue was written on 2026-09-28 and answered the same
|
||||
week by two commits in `mesh-controller`; nothing came back to close it, so the mesh's own account of
|
||||
itself said for a day that reach was declared nowhere while the code read it in three places.
|
||||
|
||||
- `bdf965d` — *a module names its endpoints, and a route names the one it serves*. `listens[].name`
|
||||
is the endpoint; a route contribution names the endpoint rather than repeating a port.
|
||||
- `c68d3a7` — *an assignment configures an endpoint as one thing*. The `endpoints` settings key, per
|
||||
node, by endpoint name: `{"endpoints": {"ssh": {"port": 20134, "reach": "public"}}}` — port, label
|
||||
and reach in one block, which is what [ADR 0138](../../02-DECISIONS/0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md)
|
||||
asked for and what [ADR 0046](../../02-DECISIONS/0046-a-module-configuration-is-its-assignments-not-its-manifest.md)
|
||||
said configuration is.
|
||||
|
||||
## The three readers, which is what the issue was about
|
||||
|
||||
The complaint was that the per-node source override had exactly one caller. It now has three, and
|
||||
they are the three mechanisms reach was decided to settle at once:
|
||||
|
||||
| reader | what it does with it |
|
||||
|---|---|
|
||||
| the filter | `Reaches` turns each endpoint's reach into the rule for its machine port |
|
||||
| the proxy's names | `composeName` composes the public name, the internal name, or both — and a name nobody asked for is not composed |
|
||||
| the authorities | the proxy certifies only names it was actually given, each from its own authority, through two host policies rather than one |
|
||||
|
||||
**A routed endpoint keeps the manifest's port**, which is ADR 0138's own insight and older than it
|
||||
([ADR 0045](../../02-DECISIONS/0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md)): the
|
||||
proxy is how it is reached, so `public` there asks for a public *name*, not an open port.
|
||||
|
||||
**An endpoint that is not routed is reached and never named.** Git over ssh is that case — the one
|
||||
the issue said the model could not express — and it is now the ordinary one.
|
||||
|
||||
## How it is checked
|
||||
|
||||
`internal/catalogue/endpoints_setting_test.go`: a block says port, label and reach; a block may say
|
||||
only a reach; a name the module does not declare is refused; a reach outside the four values is
|
||||
refused; and saying the same thing twice — once in the block, once through the older per-port keys —
|
||||
is refused rather than resolved by whichever is read last. The proxy's half is `policy_test.go` and
|
||||
`authority_test.go`: a name the mesh did not send is not certified, by either authority.
|
||||
|
||||
## What is left, and it is not this
|
||||
|
||||
The older keys (`ports`, `expose`, and reach keyed by port) still work beside the block. They are
|
||||
what the block replaces, and retiring them is its own small change — not a gap in what reach can
|
||||
say.
|
||||
@@ -0,0 +1,102 @@
|
||||
---
|
||||
status: located
|
||||
opened: 2026-09-29
|
||||
located-in:
|
||||
- mesh-host internal/apply/opening.go (retireFirewall)
|
||||
- mesh-host internal/apply/apply.go (the condition it is called under)
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 143 — Converging a machine does not retire the firewall it found, and says it does
|
||||
|
||||
## What was observed
|
||||
|
||||
The control-node was converged on 2026-09-29, the first machine with a found firewall to be flipped —
|
||||
the two converged before it had none.
|
||||
|
||||
The preview said, and the flip repeated:
|
||||
|
||||
```
|
||||
the found firewall (ufw) is disabled, never flushed: its configuration stays on disk
|
||||
...
|
||||
sent: the host loads the mesh's filter and disables the firewall it found
|
||||
```
|
||||
|
||||
The mesh then reported the node `converged`, 372 resources applied, nothing failed. Afterwards, on the
|
||||
machine:
|
||||
|
||||
```
|
||||
systemctl is-enabled ufw -> enabled
|
||||
systemctl is-active ufw -> active
|
||||
```
|
||||
|
||||
*Corrected 2026-09-29, an hour later, from reading the host rather than the declaration.* **The first
|
||||
account of this was wrong.** It said the declaration carries no resource that would disable the found
|
||||
firewall, and that the sentence was printed by the command with nothing implementing it. The
|
||||
declaration indeed carries no such resource — but the mechanism was never meant to be one. It is a
|
||||
step in the host's own apply, `retireFirewall`, and it exists, is careful, and is strict: it refuses to
|
||||
retire anything until it has read back from the machine that the mesh's own table is loaded, it records
|
||||
the forward policies first so a half-done retirement can be retried, and it verifies ufw reports
|
||||
inactive afterwards.
|
||||
|
||||
What is established is narrower and stranger than "nothing implements it":
|
||||
|
||||
- ufw was **active and enabled two minutes after the flip**, and the flip had reported the node
|
||||
converged with 372 resources applied and nothing failed.
|
||||
- The machine's own record now reads `disabled_by_mesh: true` — but it was written by a reconcile
|
||||
*after* an operator disabled ufw by hand, roughly fifty minutes later. A reconcile found ufw already
|
||||
inactive, asked it to be inactive, read that back, and recorded that the mesh had done it.
|
||||
- So the step did not take effect at the flip, and the machine's record now says it did.
|
||||
|
||||
The candidates are named rather than chosen, because the evidence does not separate them: the step is
|
||||
called only when the apply had no failures, and a skipped step is silent; the mesh's table is loaded by
|
||||
a service in the same apply, so whether it was loaded *at the moment the step asked* is an ordering
|
||||
question; and the host's own detail lines do not reach the journal, so what it decided is not
|
||||
recoverable after the fact.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
**It is a stated behaviour that does not happen, reported as success** — the fault this repository
|
||||
exists to catch, and
|
||||
[ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md) states it as
|
||||
part of what the flip *is*: "loads the mesh's derived filter in place of its refusal-only table, and
|
||||
retires the found firewall by disabling it, never by flushing".
|
||||
|
||||
**It could only be found on the first machine that had one.** The two machines converged before this
|
||||
had no firewall to retire, so the step had never run, and nothing reported that it had not. That is
|
||||
the same shape as [issue 136](../136-a-module-may-name-a-program-the-machine-does-not-have/00-report.md):
|
||||
a step that is silent when it does nothing.
|
||||
|
||||
**The machine is left doubly filtered, which is not what either firewall describes.** Every base chain
|
||||
at a hook runs and a drop in any is final, so the machine now enforces the *intersection* of the mesh's
|
||||
derived filter and a rule set left by the system being replaced. Nothing is broken by that today —
|
||||
measured from outside, mail, the proxy and git-over-ssh answer and the databases and admin interfaces
|
||||
are refused — but the machine's behaviour is described by neither of the two things claiming to
|
||||
describe it, and the stale set includes a rule for a broker that no longer exists.
|
||||
|
||||
**And returning the node to adopted would be wrong in the other direction.** ADR 0100 says that
|
||||
restores the found firewall by enabling it again; enabling something that was never disabled is
|
||||
harmless, but the mesh's belief about which firewall is in force has been wrong in both modes.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Which side owns retiring it — a resource in the declaration, so it is applied and reported like
|
||||
everything else, or the flip as an act? A resource seems right: the flip is otherwise entirely
|
||||
expressed as one, and an act that only the command performs cannot be re-checked on a later
|
||||
reconcile.
|
||||
- What should a reconcile do if the found firewall is enabled again by hand, or by a package update?
|
||||
Convergence is a state, so presumably re-disable it and say so.
|
||||
- Should the preview say what it *will* do rather than what it does, until a step exists that does it?
|
||||
The wording was read as evidence twice in one session.
|
||||
- Is there a check that a sentence the mesh prints corresponds to something that happened? This is the
|
||||
second time in one session that a printed claim and the machine disagreed.
|
||||
- **Why did the step not take effect?** It is called only when the apply had no failures, and being
|
||||
skipped is silent. The mesh's table is loaded by a service in the same apply, so whether it was
|
||||
loaded when the step asked is an ordering question — and ADR 0100 makes loading it first a
|
||||
precondition rather than an expectation.
|
||||
- **A step that records the mesh as having done what an operator did is worse than the omission.** The
|
||||
record now says the mesh disabled ufw. Nothing distinguishes "we did this" from "we found it already
|
||||
so". Should it?
|
||||
- Why do the host's own detail lines not reach the journal? Everything it decided during the flip is
|
||||
unrecoverable, which is why this account has candidates instead of a cause.
|
||||
+84
@@ -0,0 +1,84 @@
|
||||
---
|
||||
status: located
|
||||
opened: 2026-09-29
|
||||
located-in:
|
||||
- mesh-host internal/apply/opening.go
|
||||
- mesh-controller cmd/mesh-controller (the converge preview)
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 144 — A predecessor's rules outlive the firewall the mesh found, and the mesh cannot see them
|
||||
|
||||
## What was observed
|
||||
|
||||
The mesh reports one thing about a machine's existing filtering: `firewall found: ufw`. On the
|
||||
control-node, ufw was never what filtered the traffic that mattered.
|
||||
|
||||
Measured on 2026-09-29, before the machine was converged:
|
||||
|
||||
- ufw filters connections *to the machine*. It does not filter connections to a container's published
|
||||
port, which arrive on the forwarded path where the container runtime accepts them before ufw's
|
||||
forward chains are reached. Around thirty ports were published that way.
|
||||
- Every one of the mesh's own forwarded openings, converged through ufw, had matched **zero packets** —
|
||||
fifty rules in that chain, none ever matched, while the chain itself had passed 1.6 million
|
||||
established packets. The restrictions read as applied and were inert.
|
||||
- What actually kept those ports off the internet was a chain the predecessor installed in the
|
||||
container runtime's own pre-accept hook, allowing the deliberately public ports and the private
|
||||
ranges and dropping the rest on the outward link. Confirmed from outside: the proxy answered, the
|
||||
container manager did not.
|
||||
- That chain exists only in the running kernel. The persisted rule file is the distribution's empty
|
||||
default, and nothing on disk recreates the chain.
|
||||
|
||||
After the flip, the mesh's own filter is loaded and does cover the forwarded path, so the machine no
|
||||
longer depends on that chain. But **the chain is still there**, and it is now the only thing refusing
|
||||
two ports the mesh believes are open: the bus and the registry, which
|
||||
[ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md) requires be
|
||||
reachable from anywhere so a machine can enrol and pull before it has a private-network address. The
|
||||
mesh's rendered filter accepts both from anywhere. From outside, both are refused.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
**"The firewall found" is a kind, and filtering is not all in one place.** The host identifies one
|
||||
front-end and reports it. A machine can carry rules from several sources — the front-end's own, the
|
||||
container runtime's, an intrusion-prevention chain, and whatever a predecessor installed directly —
|
||||
and the mesh's account of what filters the machine names exactly one of them.
|
||||
|
||||
**So adoption's central promise was half-true in both directions.** What the mesh converged through
|
||||
the found firewall on the forwarded path did nothing at all, and what did the work was invisible to it.
|
||||
A machine was reported as filtered by a mechanism that was not filtering.
|
||||
|
||||
**And convergence cannot retire what it cannot see.** Even once
|
||||
[issue 143](../143-converging-does-not-retire-the-firewall-it-found/00-report.md) is fixed and the found
|
||||
firewall is disabled, this chain remains, silently narrowing the machine below what the mesh's own
|
||||
filter says. A rule the mesh did not write, cannot list, and will not remove — which today breaks the
|
||||
enrolment path the design guarantees.
|
||||
|
||||
**The safe direction is not the same as the correct one.** Being more closed than intended broke nothing
|
||||
visible, which is exactly why it went unnoticed for as long as the mesh has been on this machine.
|
||||
|
||||
## What it cost, measured later the same day
|
||||
|
||||
*2026-09-29.* The predecessor's chain was removed, and something it had been carrying went with it. It
|
||||
admitted the private ranges wholesale, which is how a container on the machine reached a port declared
|
||||
for the private network — the mesh's own filter admits the machines' overlay addresses, and a container
|
||||
comes from a bridge. Every module that reached another by the machine's own name had been relying on the
|
||||
predecessor's rule without anybody knowing.
|
||||
|
||||
That is [issue 145](../145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md),
|
||||
and it ran for eleven hours while the mesh reported the machine healthy. The filter is fixed. What this
|
||||
adds to the account here is that "the machine is more closed than the mesh believes" was not the
|
||||
harmless direction after all — it was harmless for everything reached from outside, and an outage for
|
||||
everything reached from within.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Should the host report every place the machine filters from, rather than one kind — the front-end,
|
||||
the runtime's hooks, and any chain it does not recognise, named so a person can look?
|
||||
- What should the mesh do about rules it did not write and does not understand? Reporting them seems
|
||||
right; removing them cannot be, and leaving them silent is what produced this.
|
||||
- Does an opening converged through a found firewall need a check that it can actually take effect?
|
||||
Fifty rules matching nothing would have been visible from the counters at any point.
|
||||
- Is the bus and the registry being reachable from anywhere still what the mesh wants on a machine that
|
||||
faces the internet? The design says yes, for enrolment. It deserves asking on its own rather than
|
||||
being answered by a leftover.
|
||||
+119
@@ -0,0 +1,119 @@
|
||||
---
|
||||
status: located
|
||||
opened: 2026-09-29
|
||||
located-in:
|
||||
- mesh-controller internal/catalogue/filtering.go (fixed for this instance)
|
||||
- mesh-controller (what status reports, and what it does not ask)
|
||||
fixed-by:
|
||||
amended-design: 03-DESIGN/01-to-be/10-delivery.md
|
||||
---
|
||||
|
||||
# 145 — A machine reads healthy while its modules cannot reach each other
|
||||
|
||||
## What was observed
|
||||
|
||||
Converging the control-node closed every path by which a module on that machine reached another module
|
||||
by the machine's own name. It ran for **eleven hours**. Throughout, the mesh answered:
|
||||
|
||||
```
|
||||
4 machine(s), all doing what they were told, all heard from,
|
||||
running what the mesh would send them, and every module current with its source
|
||||
```
|
||||
|
||||
What was actually happening, from one affected module's own log:
|
||||
|
||||
```
|
||||
Doctrine\DBAL\Exception: Failed to connect to the database:
|
||||
SQLSTATE[08006] connection to server at "novox.internal" (10.10.0.1), port 6852 failed: timeout expired
|
||||
```
|
||||
|
||||
6,154 of them, beginning at the minute of the flip. The web application accepted TCP connections and
|
||||
never answered an HTTP request; a client waited 35 seconds and gave up. Confirmed from a throwaway
|
||||
container on the machine: neither the store nor the forge was reachable on the machine's own address.
|
||||
|
||||
The cause is [issue 144](../144-the-predecessors-rules-outlive-the-firewall-it-was-found-as/00-report.md)'s
|
||||
sibling and is fixed: a port declared reachable from the private network admitted the machines' own
|
||||
overlay addresses, and a container on the machine comes from a bridge address, matching none of them.
|
||||
What this issue is about is the eleven hours.
|
||||
|
||||
**Nothing the mesh reports would have shown it.** Every check the mesh makes passed, because every
|
||||
check the mesh makes is about the relationship between the mesh and a machine:
|
||||
|
||||
- the machine applied what it was sent, and said so;
|
||||
- its declaration digest matches what the mesh would send;
|
||||
- every module's source commit matches what the mesh holds;
|
||||
- every container the declaration names is running.
|
||||
|
||||
None of those asks whether a module can reach what it requires. The mesh knows precisely who requires
|
||||
what — it composes the grants — and never checks that the grant works.
|
||||
|
||||
**Nor would an operator's usual look.** The ports were probed from outside and behaved correctly; the
|
||||
routed services answered; a container's egress to the internet worked. Those are the paths a person
|
||||
checks after changing a firewall, and all three were fine. The broken path was module-to-module over
|
||||
the machine's own name, which nothing routine exercises.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
**A mesh that composes a dependency and never tests it can only report on itself.** Every provision the
|
||||
mesh grants is a claim that a consumer can reach a provider. The mesh asserts that claim, delivers
|
||||
credentials for it, and has no mechanism that ever finds out. "Every module current with its source"
|
||||
is a statement about bytes, not about whether anything works.
|
||||
|
||||
**The failure was silent in the direction that hides longest.** A service that will not start is
|
||||
noticed. A service that starts, accepts connections and then cannot reach its database serves errors
|
||||
under a healthy-looking process, and the machine's own report says the container is running — which it
|
||||
is.
|
||||
|
||||
**It is the same shape as [issue 136](../136-a-module-may-name-a-program-the-machine-does-not-have/00-report.md),
|
||||
one level up.** There, a module named a program the machine lacked and everything reported success.
|
||||
Here, the mesh granted a provision the filter refused and everything reported success. Both are the
|
||||
distance between a declaration and the machine, and in both cases the report was about the declaration.
|
||||
|
||||
**And the eleven hours are the measurement, not the bug.** The filter fault was one line and is fixed.
|
||||
What is not fixed is that nothing in the mesh would have told anybody.
|
||||
|
||||
## What was decided
|
||||
|
||||
*2026-09-29, the same day, in two steps and the first was wrong.*
|
||||
|
||||
The first answer was [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md):
|
||||
the consumer verifies each grant from its own network position, because whether a caller sat in a
|
||||
container changed whether it could reach the provider. **That difference was the fault**, and the record
|
||||
is superseded. A verification mechanism would have reported this sooner and would not have prevented it,
|
||||
and the part of it that was difficult — deciding which network position to check from — existed only
|
||||
while the rule was wrong.
|
||||
|
||||
The remedy is [ADR 0144](../../02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md):
|
||||
anything on a machine may call anything on it, said once rather than per service, and asked by the link
|
||||
traffic arrives on rather than the address it carries. Everything should be able to call what runs on the
|
||||
same machine, another machine's service exposed to the private network, and another machine's service
|
||||
exposed publicly. The filter had the second and third and expressed the first as a list of addresses that
|
||||
no container could match.
|
||||
|
||||
**And then the question this issue is actually about was answered on its own terms.**
|
||||
[ADR 0145](../../02-DECISIONS/0145-a-module-checks-what-the-mesh-claims-is-reachable.md): a module on
|
||||
every machine serves an endpoint of its own and dials every other machine's, from the position the
|
||||
callers are in. Its probe is its own endpoint declared reachable over the private network, so it is
|
||||
admitted by exactly the rule that governs every internally-exposed service and fails when that rule is
|
||||
wrong — where a probe on a service every machine has would have passed for all eleven hours, because the
|
||||
services every machine has are the ones never closed.
|
||||
|
||||
Adopted on its merits rather than as the remedy for a configuration error, which is what 0143 was and
|
||||
why it went. The module is written and merged; it is not yet assigned, so every machine currently reads
|
||||
unchecked.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Should a grant be checked? The mesh knows the consumer, the provider, the address and the port, so a
|
||||
reachability check is expressible — but from where: the consumer's machine, as part of a reconcile,
|
||||
or the provider's?
|
||||
- What would it cost to be wrong in the other direction? A check that reports a provision broken while
|
||||
it works is worse than none, because it trains a reader to ignore the report. A provider restarting is
|
||||
ordinary; a consumer between containers is ordinary.
|
||||
- What should `status` say about a machine whose modules cannot reach each other? It currently has one
|
||||
vocabulary for "heard from and current", and that sentence was true the whole time.
|
||||
- Is there a cheaper signal than a probe? The affected module was logging the failure 6,154 times. The
|
||||
mesh reads no module's logs and arguably should not — but something a module could *say* about its
|
||||
own provisions would have surfaced this in minutes.
|
||||
- Does the same blindness apply to the other direction — a provider that lost a consumer's grant and
|
||||
is refusing it? Nothing checks that either.
|
||||
+73
@@ -0,0 +1,73 @@
|
||||
---
|
||||
status: located
|
||||
opened: 2026-09-29
|
||||
located-in: [mesh-host examples + internal/link, mesh-controller internal/broker]
|
||||
---
|
||||
|
||||
# 146 — the foundation cannot be raised on the bus the mesh runs on
|
||||
|
||||
## What was observed
|
||||
|
||||
Raising a first node in the lab, to check a module against a real mesh, fails before any module is
|
||||
reached. Two separate faults, in the two bundles that exist:
|
||||
|
||||
**The older bundle raises a control plane that cannot start.** It brings up the previous broker,
|
||||
and the control plane it then starts says, once every few seconds, for ever:
|
||||
|
||||
```
|
||||
mesh-controller: this control plane has no MESH_BUS_NATS, so it cannot reach the mesh's bus
|
||||
```
|
||||
|
||||
That is the control plane being right. The mesh moved to one bus
|
||||
([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)) and the
|
||||
bundle did not. Every bed that raises a foundation raises this one, so every bed is in this state.
|
||||
|
||||
**The newer bundle, written for the new bus, stops one step earlier.** Its certificate step asks a
|
||||
container to make the broker's certificate:
|
||||
|
||||
```
|
||||
docker run --rm --entrypoint sh -v <the broker's tls volume>:/tls <the bus image> \
|
||||
-c "test -f /tls/tls.crt || (openssl req -x509 ... )"
|
||||
...
|
||||
failed bus-certificate: running the action: docker exited 127
|
||||
```
|
||||
|
||||
127 is *command not found*. The bus's image has a shell and no `openssl`; the previous broker's
|
||||
image had both, which is why the step worked when it was written against that one. Substituting the
|
||||
store's image — the only other image the bundle carries — does not help: it has no `openssl`
|
||||
either. So the step as written cannot succeed with anything the bundle names, and the fault is not
|
||||
one image's: **the bundle asks for a certificate to be made by a tool it never says must be there.**
|
||||
|
||||
Measured 2026-09-29 on a fresh lab machine, both bundles, from bare.
|
||||
|
||||
## Why this is here and not a note in the knowledge base
|
||||
|
||||
The mesh's own foundation is the one thing it cannot raise. Nothing reports that: the bundles are
|
||||
files in a repository, nothing applies them but a person raising a node, and the last thing that
|
||||
did was the hand-driven cut-over
|
||||
([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md), whose
|
||||
work was done on the machines rather than from a bundle). So the state where the mesh cannot make
|
||||
another one of itself is reachable, and was reached, without anything saying so.
|
||||
|
||||
It is also load-bearing for everything else: a lab bed proves a claim by raising a mesh, so while
|
||||
this holds, **no bed can run**, and every "checked in the lab" written from now on is a promise
|
||||
against a suite nobody can execute.
|
||||
|
||||
## What would have prevented it
|
||||
|
||||
- **Something raising the foundation on a schedule, from the bundle, as it is written** — the
|
||||
bundle is the mesh's own installer and nothing installs from it. A bed that raises a first node
|
||||
is exactly that check, and it is the bed that cannot run.
|
||||
- **A step naming what it needs.** The certificate step names an image and assumes a program inside
|
||||
it. An action that said which tool it requires would have failed at the declaration rather than
|
||||
at 127 on a machine.
|
||||
|
||||
## Evidence to carry into diagnosis
|
||||
|
||||
- `mesh-host examples/foundation-first-node.lock` — the previous broker, no `MESH_BUS_NATS`.
|
||||
- `mesh-host examples/foundation-first-node-nats.lock` — the new bus; `bus-certificate` and its
|
||||
`verify` both run `openssl` in the bus's image.
|
||||
- The bus image the bundle pins has `sh` and no `openssl`; the store's image likewise.
|
||||
- The lab rewrites a bundle's registry-prefixed third-party references to upstream ones for a
|
||||
machine with an uplink (`test/integration/harness.ts`); the new bundle's bus reference needed
|
||||
that rule added, which is done and is not this issue.
|
||||
+184
@@ -0,0 +1,184 @@
|
||||
# Diagnosis
|
||||
|
||||
*2026-09-29, by raising a first node in the lab over and over and writing down each thing it hit.*
|
||||
|
||||
Not one fault. **Four, stacked**, each hidden behind the one before it, and every one of them the
|
||||
same shape: a step that was right while the mesh ran on the previous broker and was never asked a
|
||||
question again after the bus changed. Nothing had raised a foundation since, so nothing said so.
|
||||
|
||||
## 1 — the bundle's bus image is named for a registry that is gone *(fixed)*
|
||||
|
||||
The newer bundle pins `<a lab registry>/nats@…`, which resolves nowhere outside the lab that
|
||||
raised that registry. The lab already rewrites the store's and the previous broker's references to
|
||||
upstream ones for a machine with an uplink; the bus had no such rule because no bed had ever tried
|
||||
to raise this bundle. Added (`mesh-lab test/integration/harness.ts`). The digest is the bundle's
|
||||
own — what the registry served was a copy, so the same digest resolves upstream, and this is a
|
||||
prefix being removed rather than a reference being replaced.
|
||||
|
||||
## 2 — the bus's certificate was made by a tool the bus does not have *(fixed)*
|
||||
|
||||
```
|
||||
failed bus-certificate … docker exited 127
|
||||
```
|
||||
|
||||
The step ran `openssl` inside the broker's image. The previous broker's image carried it; the bus's
|
||||
does not — it is Alpine with a shell and no `openssl` — and neither does any other image the bundle
|
||||
names, so there was nothing to substitute. **The program that needs the certificate now makes it**:
|
||||
`mesh-controller broker certificate --into <dir>`, with `--check` as the step's verify. The
|
||||
controller is already on the machine at that point (the schema step ran it) and needs nothing from
|
||||
the image it writes into. Self-signed, as before and on purpose — a host pins this server's exact
|
||||
certificate ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)) and at that moment
|
||||
there is no authority to ask. Idempotent, because a second certificate is one every host that
|
||||
pinned the first no longer believes. It runs `--user 0:0`: the volume is root's, and the control
|
||||
plane's image runs as nobody, which is right for the long-lived server and wrong for a one-shot
|
||||
writing into a fresh volume.
|
||||
|
||||
## 3 — enrolment dialled TLS at a bus that speaks first *(fixed)*
|
||||
|
||||
```
|
||||
mesh-host: cannot reach the broker at …:5671: tls: first record does not look like a TLS handshake
|
||||
```
|
||||
|
||||
Enrolment opened a raw TLS connection to check the pinned certificate before saying anything. NATS
|
||||
speaks its own protocol and upgrades afterwards, so the handshake met a plaintext greeting. The pin
|
||||
was never the problem: the same pinned configuration is handed to the client that presents the
|
||||
token, and the verification runs inside *that* handshake — so what
|
||||
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) requires still holds, and holds
|
||||
better, because the one-time secret is sent only after the certificate has been checked. The raw
|
||||
dial is gone from the enrolment path and kept only as what its tests always proved: that a wrong
|
||||
certificate is refused before a byte of application data is sent.
|
||||
|
||||
**Then, immediately behind it:**
|
||||
|
||||
```
|
||||
mesh-host: this token is for the "" bus, and the mesh's bus is nats
|
||||
```
|
||||
|
||||
The enrolment left the transport empty and meant *whatever the mesh runs today*, which was true
|
||||
while two buses existed and became a refusal the moment one did. The host knows which bus the mesh
|
||||
runs; it says so now.
|
||||
|
||||
## 4 — a first node cannot be let onto its own bus *(open, and this is the real one)*
|
||||
|
||||
```
|
||||
mesh-host: cannot reach the bus at …:5671 as anchor: nats: Authorization Violation
|
||||
```
|
||||
|
||||
The bus's user list is a file beside its configuration. The installer carries the first one — the
|
||||
controller's own account at a bootstrap password — and **the controller composes every user after
|
||||
that** (design 25 §6; the controller's own test asserts the carried list matches what it would
|
||||
derive). On the running mesh that composition reaches the bus because the bus is a *module*, with
|
||||
the list delivered to it the way anything is delivered to a module.
|
||||
|
||||
At genesis there is no module. The foundation's bus is raised by the installer, the control plane is
|
||||
given no way to write beside its configuration — it mounts the certificate and nothing else — and so
|
||||
the account a joining node needs cannot come into existence. **The first node cannot join the mesh
|
||||
it just raised.**
|
||||
|
||||
That is not a line to fix in a bundle. It is the open half of the mesh delivering its own components
|
||||
([ADR 0142](../../02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md)) and of the
|
||||
bus becoming a module: either the installer's bus is raised as the module the mesh will go on
|
||||
managing, or genesis carries a user list that includes the first node's enrolment and the controller
|
||||
takes over from there. Both are decisions, not patches, and both belong to the genesis step that was
|
||||
deliberately left until last.
|
||||
|
||||
## 5 — the composed user list has to be placed by hand at genesis *(fixed)*
|
||||
|
||||
The account a token is the password of is **not recorded at all**: the composer names an enrolment
|
||||
user for every machine with a live token, nothing minted a credential for it, and the composition
|
||||
left it out as a user with no password. The comment above the issuing code already claimed
|
||||
otherwise — *"the account is created before the token is handed over"* — which is how it went
|
||||
unnoticed. Issuing a token now records that account, with the token's own secret as its password,
|
||||
because that is the string the machine will present.
|
||||
|
||||
Placing it is the other half. The list reaches the machine running the bus in that machine's
|
||||
declaration, which a machine that has not enrolled does not get, so at genesis it cannot arrive
|
||||
that way. **The control plane composes and says what it composed** — `broker accounts`, to standard
|
||||
output — and whoever is raising the machine writes it beside the bus's configuration and makes the
|
||||
server re-read it. Twice, because two accounts come into existence at different moments: the
|
||||
enrolment when the token is issued, and the machine's own when it enrols. A control plane that
|
||||
wrote the file itself would have to know where the bus keeps its configuration and how to make it
|
||||
reload, which is the module's knowledge and is what the module takes over on the first push.
|
||||
|
||||
With that, **a first node enrols against the bus it just raised** — measured, from bare, in the
|
||||
lab.
|
||||
|
||||
## 6 — and is enrolled twice, keeping a credential the mesh has replaced *(fixed)*
|
||||
|
||||
```
|
||||
mesh-controller: enrolled anchor
|
||||
mesh-controller: enrolled anchor (the same second)
|
||||
```
|
||||
|
||||
One `enrol` on the machine, two enrolments in the control plane. Each mints the node a fresh bus
|
||||
password and returns it; the machine keeps the answer to the first, and the mesh keeps the hash of
|
||||
the second. The machine then reconnects for ever as a user whose password the mesh rotated out from
|
||||
under it — *authentication error - User "node.anchor"* on the bus, `Authorization Violation` in the
|
||||
host's log, and a node that never reports.
|
||||
|
||||
What is ruled out: the host asking twice — it asks again only when the mesh says *try again*, and
|
||||
a refused attempt is not logged as an enrolment. Redelivery by the consumer — there is one
|
||||
consumer, its acknowledgement window is thirty seconds, and the handler is quick.
|
||||
|
||||
What is left: the client re-publishing when an acknowledgement is slow, which is what its defaults
|
||||
do. That was addressed by giving the publish a message id derived from its own bytes, so the stream
|
||||
discards the copy — **and the duplicate survived it**, so either the id is not reaching the stream
|
||||
or the second copy is not a copy. This is where the trail stops.
|
||||
|
||||
Worth saying plainly: **the mint is the fragile part, not the delivery.** An enrolment answered
|
||||
twice is survivable if the answer is the same both times, and it cannot be — the mesh keeps only
|
||||
the hash, so a second answer is necessarily a different credential.
|
||||
|
||||
**Found, and it is not about enrolment at all** *(fixed)*. The bus's own counters settled it: one
|
||||
message published, one held in the stream, one delivery, nothing redelivered — and the controller
|
||||
enrolled the machine twice. So the handler ran twice on one delivery.
|
||||
|
||||
A push consumer delivers onto an ordinary subject, and **everything subscribed to that subject gets
|
||||
a copy**. The controller holds a consumer called `controller` on CONTROL and another called
|
||||
`controller` on EVENTS, and the delivery subject was derived from the consumer's name alone — so
|
||||
both were `_DELIVER.controller`, the one process held both subscriptions, and every message from
|
||||
either stream was acted on twice.
|
||||
|
||||
Enrolment is where it drew blood, because enrolling twice mints twice and the second credential
|
||||
replaces the first. But it applied to **every report and every event the controller follows**, and
|
||||
it is the kind of fault that leaves no trace: nothing is redelivered, no counter is wrong, the work
|
||||
simply happens twice. The comment in the receiving code about a merge that ran the whole catalogue
|
||||
five times over on 2026-09-28 is the same shape seen from the other end.
|
||||
|
||||
The stream is in the delivery subject now, because the pair is what identifies a consumer — the
|
||||
server scopes a durable's name to its stream, and this subject was the one place that scoping was
|
||||
dropped. A subscriber's permission gains the same shape, keeping the bare name so an existing
|
||||
consumer keeps working until the controller's next assertion moves it.
|
||||
|
||||
*How it is checked:* the consumers the mesh asks for are asserted to deliver onto distinct subjects,
|
||||
in the controller's own suite. Against a server it would be invisible, which is the point.
|
||||
|
||||
## Where it belongs
|
||||
|
||||
`mesh-host` (the bundle and the enrolment path) and `mesh-controller` (the certificate command, and
|
||||
the composition that cannot reach the bus at genesis). Three of the four are fixed on branches; the
|
||||
fourth is the genesis work.
|
||||
|
||||
## What made it slow, and what was changed so it is not
|
||||
|
||||
Six faults behind one another, each found by raising a machine and reading what it said. What cost
|
||||
the most was not the faults:
|
||||
|
||||
- **Every bed's own instructions named the bundle that cannot work**, so the first three attempts
|
||||
ended in a control plane crash-looping on a missing bus. They name the working one now.
|
||||
- **A host binary built without its system** refuses everything it is given with *this host was
|
||||
built for ""*, which reads like a broken bundle. The lab's README says so.
|
||||
- **`make image` in the control plane had been broken for as long as its base was pinned**: the
|
||||
Dockerfile's fallback is a Go older than the module asks for, and the pipeline never saw it
|
||||
because the pipeline passes the declared base in. It reads the base from the manifest now.
|
||||
- **Leaving the machine standing is what answers the question.** Every finding above came from
|
||||
shelling in afterwards — the host's log, the bus's log, the file the bus was actually handed —
|
||||
and none from the test's own output, which says only that nothing converged. The bed takes
|
||||
`MESH_LAB_KEEP`, and the README says to reach for it first.
|
||||
|
||||
## What it cost, for the next person
|
||||
|
||||
Every lab bed still names `foundation-first-node.lock` in its own instructions, and that bundle
|
||||
raises the previous broker with a control plane that refuses to start without `MESH_BUS_NATS`. Until
|
||||
the fourth fault is answered and the two bundles become one, a bed runs with `MESH_LAB_BUNDLE`
|
||||
pointing at the NATS bundle by hand, and stops at the enrolment.
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
status: located
|
||||
opened: 2026-09-29
|
||||
located-in: [nothing in the mesh — the surface in use is the predecessor's, installed on the workstation]
|
||||
---
|
||||
|
||||
# 147 — the operator's tools still dial the bus that was removed
|
||||
|
||||
## What was observed
|
||||
|
||||
Every tool call an operator makes against the mesh fails, on every machine, with the same answer:
|
||||
|
||||
```
|
||||
AMQP not connected — cannot reach hal/mesh@novox
|
||||
AMQP not connected — cannot reach hal/mesh@shanks
|
||||
```
|
||||
|
||||
The mesh moved to one bus and the previous transport was deleted
|
||||
([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md), cut over
|
||||
2026-09-28). The surface an operator drives the mesh through — the tool bridge their assistant
|
||||
speaks to — still opens an AMQP connection, so it cannot reach anything. Not one node answers,
|
||||
including the machine the operator is sitting at.
|
||||
|
||||
**What that leaves.** The mesh itself is healthy: nodes are current, modules run, the bus carries
|
||||
the mesh's own traffic. What is gone is the way a person asks it anything. Every question — what a
|
||||
node runs, what is assigned, what a module's settings are — has to be asked by opening a shell on
|
||||
the machine and running the control plane's binary inside its container, which is precisely the
|
||||
path the tool surface exists to remove, and which nothing checks, records or permits.
|
||||
|
||||
It also silently changes how work gets done: an assistant told to use the mesh's tools finds them
|
||||
dead, falls back to `ssh` and `docker exec`, and the operator discovers the fallback rather than
|
||||
the fault.
|
||||
|
||||
## Why this is here and not a note in the knowledge base
|
||||
|
||||
The rule is that everything happens over the mesh's bus. A surface that cannot reach that bus is
|
||||
that rule enforced by nothing — and the mesh reports itself healthy throughout, because what broke
|
||||
is not a module, a node or a provision but the thing standing outside asking them questions.
|
||||
|
||||
The same cut-over removed the assistant's long-term memory (`recall`, `memorize`) for the same
|
||||
reason, which is why lessons from the last two days were written into this repository by hand.
|
||||
|
||||
## What would have prevented it
|
||||
|
||||
- **The tool surface as a consumer of the bus like any other**, so moving the bus moves it — rather
|
||||
than a separate bridge with its own connection settings that nothing resolves.
|
||||
- **A check that a tool call reaches a node**, run where the mesh's other checks run. Every check
|
||||
the mesh makes today is about the relationship between the mesh and a machine; none asks whether
|
||||
a person can ask it anything ([issue 145](../145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md),
|
||||
the same shape one level out).
|
||||
|
||||
## Diagnosed at once, because the answer was in the configuration
|
||||
|
||||
**The surface is not the mesh's.** The tool server the operator's assistant speaks to is the
|
||||
predecessor's brain, installed on the workstation and started as a local process, with the
|
||||
predecessor's broker URL — `amqp://…@<the control node>` — written into the assistant's own
|
||||
configuration. Nothing about it is a module: it has no manifest, no assignment, no seat, no account
|
||||
on the mesh's bus, and the mesh has never known it exists.
|
||||
|
||||
So nothing regressed. The mesh removed a transport that this program still dials, and the program
|
||||
was never part of the mesh to be moved. **The mesh has a tool model** — `tools` in a manifest, the
|
||||
calls a seat accepts, the subjects a module answers on, and a control-plane command that asks one —
|
||||
and **nothing publishes an operator-facing surface onto it.** What an operator uses is the thing
|
||||
that came before, kept alive by a URL in a file.
|
||||
|
||||
That is the issue, and it is larger than a broken connection: the way a person drives this mesh is
|
||||
outside the mesh.
|
||||
|
||||
## Evidence to carry into diagnosis
|
||||
|
||||
- The failure text names `hal/mesh@<node>`, so the bridge is resolving a node and then dialling the
|
||||
old transport.
|
||||
- It fails identically for the local machine, which rules out reachability and points at the
|
||||
transport alone.
|
||||
- The mesh's own traffic over the new bus is unaffected: nodes report, declarations apply.
|
||||
@@ -0,0 +1,37 @@
|
||||
---
|
||||
status: open
|
||||
opened: 2026-09-29
|
||||
located-in: [mesh-controller cmd/mesh-controller]
|
||||
---
|
||||
|
||||
# 148 — a manifest outside this catalogue has no check
|
||||
|
||||
## What was observed
|
||||
|
||||
A module's manifest is validated by a **test** — `internal/catalogue`'s suite parses every manifest
|
||||
in the catalogue checkout beside it and fails on one it cannot resolve. That works, and it is how
|
||||
several real faults were caught before a machine saw them.
|
||||
|
||||
It is available to exactly one repository: this one. Somebody describing their own application in
|
||||
their own repository — the case
|
||||
[ADR 0037](../../02-DECISIONS/0037-where-a-module-lives.md) calls *the case that matters most* —
|
||||
has no check at all. They write a manifest, register it with a running mesh, and find out whether
|
||||
it is valid when the mesh refuses it, or later, when a machine applies something that resolved and
|
||||
should not have.
|
||||
|
||||
The same record asks for the answer: **a `module check` command on the control plane's binary**, so
|
||||
a manifest is checked by the tool rather than by a test that imports the tool's internals.
|
||||
|
||||
## What would have prevented it
|
||||
|
||||
Nothing prevents this; it was noticed and left. ADR 0037 named it on 2026-09-01 and the record sat
|
||||
`proposed` until 2026-09-29, so the missing half was never anybody's task.
|
||||
|
||||
## Evidence to carry into diagnosis
|
||||
|
||||
- `mesh-controller/internal/catalogue` — `ParseManifest` and `CatalogueProblems` are the check, and
|
||||
both are internal.
|
||||
- The catalogue-wide test is `TestEveryCatalogueManifestDeclaresWhatItMounts` and its siblings; they
|
||||
take a path from `MESH_CATALOG`, so the mechanism is already path-driven and not repository-bound.
|
||||
- `mesh-controller module add` refuses a bad manifest at registration, which is the same check far
|
||||
too late: by then it is in a running mesh's records.
|
||||
+22
-3
@@ -1,10 +1,15 @@
|
||||
---
|
||||
status: located
|
||||
status: resolved
|
||||
opened: 2026-09-27
|
||||
located-in: [mesh-controller cmd/mesh-controller/push.go]
|
||||
located-in: [mesh-controller cmd/mesh-controller/push.go, mesh-controller cmd/mesh-controller/sendable.go]
|
||||
fixed-by: mesh-controller sendable.go and push.go — an empty declaration is sent carrying `owns_nothing`, and the host refuses an empty body that does not carry it
|
||||
---
|
||||
|
||||
# A declaration that shrinks to empty is skipped, so the node keeps what it should drop
|
||||
# 149 — a declaration that shrinks to empty is skipped, so the node keeps what it should drop
|
||||
|
||||
*Opened as 127 and renumbered on 2026-09-29: two records were given that number on the same day.*
|
||||
*The other kept it, because three documents and three source files cite it by number and nothing
|
||||
cited this one but a decision and a sibling issue, both corrected with this move.*
|
||||
|
||||
## What was observed
|
||||
|
||||
@@ -37,3 +42,17 @@ mean "own nothing", which the host already applies correctly when it receives on
|
||||
|
||||
On ace, one command drops it permanently (the corrected controller never re-composes it):
|
||||
`sudo ufw delete allow 5671`. At ace's converge it would clear on its own.
|
||||
|
||||
## Closed
|
||||
|
||||
*2026-09-29, in a grooming pass.* Both halves are on `main` and both name this issue.
|
||||
|
||||
- The control plane **sends** it: a declaration that composes to no resources goes out with
|
||||
`owns_nothing`, and `push` says *sent, not skipped*.
|
||||
- The host **refuses an empty body that does not carry it**, so a truncated or mis-composed
|
||||
declaration can never be read as "own nothing" — which is the failure the fix had to avoid while
|
||||
making the empty case expressible.
|
||||
|
||||
Closed by reading the code rather than by watching a machine let go of a stray resource; the record
|
||||
says so rather than implying a run.
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
---
|
||||
status: open
|
||||
opened: 2026-09-29
|
||||
located-in:
|
||||
- mesh-controller internal/catalogue/declaration.go (contributions are emitted for an assigned module, taken or not)
|
||||
fixed-by:
|
||||
---
|
||||
|
||||
# 150 — A route is contributed before its module is taken
|
||||
|
||||
## What was observed
|
||||
|
||||
ace is adopted and runs `route-adapter` beside the predecessor's traefik (ADR 0104). The first web
|
||||
module migrated there was searxng. `assign ace searxng` + `push ace` — the step that is supposed to
|
||||
change nothing on an adopted machine (ADR 0100, hq 125: assign holds, take replaces) — took
|
||||
`searxng.zurag.be` down:
|
||||
|
||||
```
|
||||
https://searxng.zurag.be/ → 502
|
||||
```
|
||||
|
||||
for about five minutes, until the module was unassigned again.
|
||||
|
||||
## Why
|
||||
|
||||
Assign held everything it found on the machine — the predecessor's `searxng` container, its
|
||||
directories — exactly as designed. But the module's **route contribution** is not a resource on the
|
||||
machine, so nothing held it: it reached `route-adapter` at once, which wrote
|
||||
`mesh-searxng.zurag.be.yml` into traefik's file provider pointing at the mesh's assigned machine port
|
||||
(`http://ace.internal:20000`) — where nothing listened, because the container that would was held.
|
||||
traefik's file router for the name then won over the predecessor's docker-label router for the same
|
||||
name, and the name served a dead backend.
|
||||
|
||||
On this occasion the window was lengthened by an unrelated failure (the machine's docker address
|
||||
pools were exhausted, so the module's network could not be created), but the fault does not depend on
|
||||
it: **between assign and take, every routed module's public name points at a backend the mesh has
|
||||
deliberately not started.** The runbook's §6 step 2 ("check the preparation — HAL's service still
|
||||
serves") is false for every routed module on a node running route-adapter.
|
||||
|
||||
## What the operator did
|
||||
|
||||
Rolled back per runbook (unassign before take), then re-ran as assign → push → wait for the node to
|
||||
report its holds → take → push, keeping the window to the ~10 s between the two pushes. A workaround,
|
||||
not a fix: a module whose data must move between assign and take (runbook §6 step 3) cannot shrink
|
||||
its window this way.
|
||||
|
||||
## What would be right
|
||||
|
||||
A contribution from a module that is assigned but **not taken** on an adopted node should be held like
|
||||
the module's resources are — withheld from the provider until take — or the provider should be told
|
||||
the contributor is held and leave the predecessor's route alone. Either keeps "assign changes nothing"
|
||||
true for routed modules.
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
status: open
|
||||
opened: 2026-09-29
|
||||
located-in:
|
||||
- mesh-host internal/apply/apply.go (containerSpecReading hashes every `host` entry)
|
||||
- mesh-controller internal/catalogue/declaration.go (withMeshNames gives every container the mesh's names)
|
||||
fixed-by:
|
||||
---
|
||||
|
||||
# 151 — A new name recreates every container in the mesh
|
||||
|
||||
## What was observed
|
||||
|
||||
Migrating one small module on ace (searxng) took four routine controller actions: `node public-domain
|
||||
ace zurag.be`, `assign ace searxng` + push, `unassign ace searxng` + push (a rollback, see
|
||||
[issue 150](../150-a-route-is-contributed-before-its-module-is-taken/00-report.md)), and assign + take
|
||||
+ push again. Each push to ace also pushed novox ("this push left g14, novox, shanks behind … sending it
|
||||
too"). novox's host then **replaced every container it runs, twice**:
|
||||
|
||||
```
|
||||
22:46 … updated postgres.server (mesh-store): replaced; a container's configuration is fixed when it is created
|
||||
22:46 … updated distribution.store (mesh-registry): replaced; …
|
||||
22:46 … updated route-proxy.server (route-proxy): recreated …
|
||||
22:51 … updated postgres.server (mesh-store): replaced; …
|
||||
22:52 … updated gitea.server (gitea): replaced; …
|
||||
22:52–22:55 mesh-vault, builder, all of mailu, keycloak, minio, mongodb, invoicing, photos, umami, …
|
||||
```
|
||||
|
||||
Replacements per minute on novox, 22:46–22:55: 12, 8, 13, 5, 6, 8, 8, 8, 8, 4. The control plane was
|
||||
unreachable twice while its own store came back through crash recovery
|
||||
(`FATAL: the database system is starting up`), and dependants (umami) crash-looped until it did. None
|
||||
of the replaced containers belonged to the module being migrated, or to ace.
|
||||
|
||||
## Why (confirmed part)
|
||||
|
||||
Every container the mesh runs is given the mesh's names as `--add-host` entries
|
||||
(`withMeshNames`), and the host puts every entry into the container's spec digest — deliberately,
|
||||
since [issue 135](../135-a-containers-mesh-names-are-not-compared/00-report.md): a container left alone when the roster moved kept a five-day-old
|
||||
address and restarted 2286 times. A running container cannot have its hosts changed, so a changed
|
||||
digest means a replace.
|
||||
|
||||
The consequence is that **the roster is part of every container everywhere**: anything that adds,
|
||||
removes or moves one name — a node's public domain, a routed name, an unassign — replaces every
|
||||
container on every machine that carries the list. On the hub that includes the control plane's store,
|
||||
the registry, the edge and mail.
|
||||
|
||||
## Not yet established
|
||||
|
||||
Which name moved in each pass. gitea's hosts after the fact list the `.internal` names and novox's
|
||||
internally routed `*.novox.be` names, and **not** `searxng.zurag.be` — so it is not simply "a routed
|
||||
name was added". Two passes suggest the set changed at assign and changed back at unassign; the
|
||||
declarations before and after would say, and nothing on the machine records the previous one.
|
||||
|
||||
## Why it matters now
|
||||
|
||||
The node-by-node migration adds names one module at a time. On ace alone that is ~25 web modules; if
|
||||
each assignment moves the roster, each is a full restart of every hub service, and a rollback is
|
||||
another. The migration is paused on this.
|
||||
|
||||
## What would be right (for diagnosis)
|
||||
|
||||
Keep 135's guarantee (no container runs with a stale address) without making the roster part of every
|
||||
container's identity — e.g. resolve mesh names through a resolver the container asks at lookup time
|
||||
rather than baked entries, or scope each container's entries to the names it actually binds.
|
||||
@@ -0,0 +1,124 @@
|
||||
---
|
||||
status: resolved
|
||||
opened: 2026-09-29
|
||||
located-in:
|
||||
- mesh-controller cmd/mesh-controller/plan.go (routeNamesInTheMesh skips a node whose plan will not compose)
|
||||
fixed-by: mesh-control fix/152-a-lookup-failure-is-not-an-absence
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 152 — A node whose plan will not compose silently removes its names from every machine
|
||||
|
||||
## What was observed
|
||||
|
||||
For twenty-five minutes, and for seventeen of them after the last operator action, the control-node's
|
||||
host applied all 327 of its resources every ~6.5 minutes without pause, replacing every container on
|
||||
the machine each time — the control plane's own store and registry, the edge proxy, the forge, the directory, mail,
|
||||
and the bus the mesh runs on. The forge's web surface answered `502` throughout; load on the machine
|
||||
sat near 8. Nothing was converging: each pass ended and the next began four seconds later.
|
||||
|
||||
The two machines carrying no containers were not churning. They were only knocked off the bus each
|
||||
time the control node re-created it, reconnected, re-heard the same declaration and applied it again
|
||||
as a no-op.
|
||||
|
||||
## Why: the roster alternates between two values, and it is part of every container
|
||||
|
||||
Two consecutive declarations were compared by reading the `--add-host` entries of four containers
|
||||
the moment each pass created them:
|
||||
|
||||
```
|
||||
23:02 ace.internal drive.novox.be g14.internal keycloak.novox.be novox.internal
|
||||
office.novox.be portainer.novox.be shanks.internal umami.novox.be (9 names)
|
||||
23:07 … the same nine, and searxng.zurag.be (10 names)
|
||||
```
|
||||
|
||||
One routed name — belonging to a module on another machine entirely — leaves the roster and comes
|
||||
back. Because the roster is part of every container's spec digest ([issue 151](../151-a-new-name-recreates-every-container-in-the-mesh/00-report.md)),
|
||||
each flip is a different identity for every container on the machine, and a running container cannot
|
||||
have its hosts changed. So every flip replaces all of them.
|
||||
|
||||
**What makes it flip is a swallowed error.** `routeNamesInTheMesh` composes every node's plan to
|
||||
find the names it serves, and when one will not compose it moves on:
|
||||
|
||||
```
|
||||
plan, settings, err := planFor(ctx, open, n.Name)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
```
|
||||
|
||||
A node whose plan cannot be composed *right now* therefore contributes no names — not "the mesh does
|
||||
not know", but "the mesh states these names do not exist", to every machine at once.
|
||||
|
||||
## Why it sustains itself
|
||||
|
||||
The loop closes through the control plane's own database:
|
||||
|
||||
1. An apply replaces `mesh-store` — the store the control plane reads — by removing the container,
|
||||
so postgres comes back through crash recovery.
|
||||
2. While it recovers it refuses connections: `FATAL: the database system is not yet accepting
|
||||
connections / Consistent recovery state has not been yet reached` (observed, 21:08:39 UTC, every
|
||||
pass).
|
||||
3. `planFor` for the other machine fails against that store. `routeNamesInTheMesh` swallows it and
|
||||
drops its routed name.
|
||||
4. The roster changed, so all 327 resources differ, so all are replaced — including `mesh-store`,
|
||||
and including the bus, which is why the host also cannot report: `applied, and could not tell the
|
||||
mesh: reporting: nats: connection closed`.
|
||||
5. Back to 1.
|
||||
|
||||
Every pass destroys the evidence the next pass needs to decide it has nothing to do, and nothing
|
||||
outside the machine has to be wrong for it to continue.
|
||||
|
||||
**It is metastable, not permanent.** It ran from 22:46 to 23:11 — five full replacements of every
|
||||
container on the machine — and then stopped on its own, when one pass happened to read the store
|
||||
during a window it was up, composed the same roster twice running, and found nothing to do. Load fell
|
||||
from 7.8 to 1.7 and the machine returned to its five-minute idle tick.
|
||||
|
||||
That it ends by luck is the point, not a mitigation. The exit condition is a race the mesh does not
|
||||
control, the operator cannot see, and nothing reports; the same four actions on a slower machine, or
|
||||
a larger store, would not have found it. An outage that clears itself after twenty-five minutes and
|
||||
five restarts of the forge, the directory and mail is not a smaller fault than one that does not — it
|
||||
is the same fault, harder to catch.
|
||||
|
||||
## What it is not
|
||||
|
||||
- Not the operator's four actions on the other machine. Those explain the first passes
|
||||
([issue 151](../151-a-new-name-recreates-every-container-in-the-mesh/00-report.md)); they were
|
||||
finished seventeen minutes and three full passes before these measurements.
|
||||
- Not a file that keeps changing. `/etc/hosts`, the bus's account list and the vault's export were
|
||||
hashed across passes and are byte-identical, and the directory reported `mode 755 to 700` every
|
||||
pass while already being `700`. Those resources are **misreported as changed** and are worth their
|
||||
own question, but they are not what moves a container's identity.
|
||||
- Not the lost report alone. A report that cannot be delivered explains a re-apply; it does not
|
||||
explain a re-apply that finds 327 differences.
|
||||
|
||||
## Why it matters beyond this outage
|
||||
|
||||
The same `continue` makes every routed name in the mesh conditional on every node's plan composing at
|
||||
the moment any machine is pushed to. One unreachable or half-migrated machine is enough to withdraw
|
||||
its names from everywhere — and the withdrawal is indistinguishable, on the receiving machine, from
|
||||
the operator having removed them.
|
||||
|
||||
The codebase already states the rule this breaks, forty lines away, about the same kind of lookup:
|
||||
|
||||
> A lookup failure is an error, never "not found": collapsing the two composed a declaration without
|
||||
> the trust whenever the inventory hiccuped, delivered by a push that reported success.
|
||||
|
||||
## How it was fixed, and how the fix is checked
|
||||
|
||||
`planFor` now marks the two failures that really are a statement about the node — its set not
|
||||
composing, and a setting that reaches nothing — and the three gatherers pass over those and only
|
||||
those. Every other failure is raised, naming the machine and the read.
|
||||
|
||||
Five tests hold it: a set that cannot compose is marked as the node's own; a store that cannot be
|
||||
read is *not*; one incoherent node still does not cost the rest their names; a roster is never
|
||||
returned beside an error; and the raised failure names what could not be read.
|
||||
|
||||
The two sibling gatherers were audited and fixed the same way — the grant composer, which would have
|
||||
withheld a consumer's credential, and the private-network membership, which would have taken a
|
||||
machine off the overlay. Three other `planFor` callers were audited and left alone: they refuse or
|
||||
report rather than silently withdraw, which is the safe direction.
|
||||
|
||||
**This does not close [issue 151](../151-a-new-name-recreates-every-container-in-the-mesh/00-report.md).**
|
||||
A roster that changes for a real reason still replaces every container in the mesh. This removes the
|
||||
false reasons; whether the roster belongs in a container's identity at all is that record's question.
|
||||
@@ -0,0 +1,57 @@
|
||||
---
|
||||
status: open
|
||||
opened: 2026-09-29
|
||||
located-in:
|
||||
- mesh-controller internal/catalogue/dir_into.go (dirsFor: a stated path or <data root>/<module>/<id>, nothing else)
|
||||
- mesh-controller (accesses: the path is the manifest's literal)
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 153 — An adopted machine's data cannot be placed where it is
|
||||
|
||||
## What was observed
|
||||
|
||||
Preparing ace's media modules (plex, sonarr, radarr, lidarr, bazarr, nzbget, qbittorrent, bookshelf)
|
||||
for migration. ace is adopted; its data is where the predecessor put it and **must stay there**:
|
||||
|
||||
- the library and download spool: `/storage/media/*`, `/storage/downloads` — a separate ZFS pool,
|
||||
~40 TB, the operator's shared data (ADR 0051);
|
||||
- plex's own state: `/mnt/plex/{config,data,temp}` — 133 GB on a second disk;
|
||||
- large configuration directories held in place: lidarr 46 GB, radarr 17 GB, sonarr 3.2 GB.
|
||||
|
||||
The catalogue's manifests name `/services/media/*` (as `accesses`) and `/services/<m>/config` (as
|
||||
owned directories), which is novox's layout, not ace's, and not a value a definition may carry.
|
||||
|
||||
## What was decided, and what exists
|
||||
|
||||
[ADR 0112](../../02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md) (accepted) says
|
||||
exactly what is needed:
|
||||
|
||||
> *Where* it is on the machine is the assignment's. A node has a default layout, and an assignment may
|
||||
> place a directory elsewhere: on a second disk, or where an adopted machine's data already is.
|
||||
|
||||
> [ADR 0051]: an access keeps its shape and its semantics; its path moves from the definition to the
|
||||
> assignment.
|
||||
|
||||
What the control plane implements (`dirsFor`): a directory is either a path the manifest states, or
|
||||
`<data root>/<module>/<id>` under the node's one data root. There is **no per-assignment placement of
|
||||
one directory**, and an `access` path is the manifest's literal — no setting reaches either.
|
||||
|
||||
## Consequence
|
||||
|
||||
Every module whose data an adopted machine already holds somewhere other than the default layout can
|
||||
only be migrated by (a) writing the machine's path into the manifest — which 0112 forbids and which
|
||||
is wrong on the next machine — or (b) moving the data into the placed layout in a window. (b) is
|
||||
acceptable for a 40 MB configuration and impossible for a 40 TB library the operator has ruled must
|
||||
never be moved, copied or re-owned.
|
||||
|
||||
The same gap covers ownership: the predecessor runs ace's media stack as `1001:2000`; a manifest's
|
||||
`owner` is one value for every machine.
|
||||
|
||||
## What would be right
|
||||
|
||||
The two assignment halves 0112 decided: a setting that places a declared directory (by id) at a given
|
||||
path on this node, and a setting that says where an access's data is — both validated like
|
||||
`endpoints` (unknown ids refused), and an access placed by the assignment still never created,
|
||||
chowned or removed.
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
status: open
|
||||
opened: 2026-09-29
|
||||
located-in:
|
||||
- hq 02-DECISIONS/0138 (reach: internal | public | both)
|
||||
- mesh-controller internal/catalogue/filtering.go (Reaches)
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 154 — A machine's own network is not a reach
|
||||
|
||||
## What was observed
|
||||
|
||||
Preparing ace's modules. ace sits on a home network (192.168.1.0/24) behind a router, and several of
|
||||
its services are reached **from that network by devices that will never be mesh machines**:
|
||||
|
||||
- mosquitto `1883` — an IoT light switch (`sonoff-office-light-switch`) and home-assistant;
|
||||
- unifi `8080`/`3478 udp`/`10001 udp` — the access points' inform, STUN and discovery;
|
||||
- plex `32400` — LAN streaming clients (three connected at survey time);
|
||||
- home-assistant `8123`, and the resolver on the LAN address.
|
||||
|
||||
ADR 0138 gives an endpoint's reach as `internal` (the private overlay), `public` (anywhere) or `both`.
|
||||
None of them says *this machine's own network*. The predecessor could: its unifi manifest opened
|
||||
inform/STUN/discovery `from: 192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12`.
|
||||
|
||||
## Consequence
|
||||
|
||||
The only reach that includes a LAN device is `public`. While ace is adopted that is harmless — its
|
||||
own firewall stays and admits the LAN — and behind NAT "anywhere" happens to mean the LAN. But:
|
||||
|
||||
- it states the wrong thing: an operator reading `reach: public` on an IoT broker believes it is on
|
||||
the internet, and a router port-forward added later for something else makes it so;
|
||||
- at `converge ace`, the mesh's filter is the sum of what it listens on (ADR 0045). An endpoint left
|
||||
`internal` cuts every LAN device off at the flip; one set `public` opens it to the internet on any
|
||||
machine with a public address.
|
||||
|
||||
## What would be right (for diagnosis)
|
||||
|
||||
A reach — or a source — that means the networks the machine is directly attached to (its uplink's
|
||||
subnets, as the machine reports them), so a LAN-only service is declared as exactly that and the
|
||||
filter can admit it without admitting the internet.
|
||||
@@ -0,0 +1,53 @@
|
||||
---
|
||||
status: resolved
|
||||
opened: 2026-09-29
|
||||
located-in: [hq 00-META/checks/cycle.py]
|
||||
fixed-by: hq issue/two-records-share-a-number-and-nothing-says-so
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 155 — Two records may share a number, and every check passes
|
||||
|
||||
## What was observed
|
||||
|
||||
On 2026-09-29 two machines opened issues against this repository within the same hour. Both read
|
||||
`main` correctly and both took "the next free number", and they collided twice:
|
||||
|
||||
| | one machine opened | the other had already used |
|
||||
|---|---|---|
|
||||
| first | 147, 148 | 147, 148 on an unmerged branch |
|
||||
| second | 149, 150 | 149 on an unmerged branch, 150 from renumbering the first collision |
|
||||
|
||||
The first collision was reconciled by hand before merging. The second was **merged into `main`**, and
|
||||
`records.py`, `cycle.py` and `index.py` all reported success over a tree holding
|
||||
`149-a-declaration-that-shrinks-to-empty` beside `149-an-adopted-machines-data-cannot-be-placed-where-it-is`,
|
||||
and two folders numbered 150.
|
||||
|
||||
## Why
|
||||
|
||||
The number is allocated as `max(main) + 1`, and `main` lags every open pull request — seven of them
|
||||
that evening. Two readers of the same `main` therefore compute the same next number, and neither is
|
||||
doing anything wrong. The existing reconciliation precedent (a second record numbered 127 became 149)
|
||||
assumed a single writer, which stopped being true when a second machine began filing its own findings.
|
||||
|
||||
## Why it matters
|
||||
|
||||
An issue number is how every other record cites this one — `fixed-by:`, `located-in:`, a decision
|
||||
record's consequence, a commit message. Two records answering to one number is a citation that
|
||||
resolves to whichever folder the reader happened to open, and the failure is silent on both sides:
|
||||
the citer is not wrong, and the cited record exists.
|
||||
|
||||
It is also exactly the class this repository says it does not permit — a rule (`00-META/process/03-issues.md`:
|
||||
"take the next free number") enforced by nothing.
|
||||
|
||||
## How it was fixed, and how the fix is checked
|
||||
|
||||
`cycle.py` now refuses a tree in which two issue folders share a leading number, and names both.
|
||||
Proven by adding a duplicate and watching it fail, then removing it and watching it pass.
|
||||
|
||||
The colliding records were renumbered 153 and 154, in the branch that landed last — renumbering a
|
||||
branch whose author is still pushing only moves the race.
|
||||
|
||||
**The check catches the collision; it does not prevent it.** Allocating a number still needs the open
|
||||
pull requests read as well as `main`. That is a habit the check now backstops rather than one it
|
||||
replaces.
|
||||
Reference in New Issue
Block a user