Compare commits

..
Author SHA1 Message Date
jschoubben 0cf1ad5dad Merge pull request 'Issue 172: the ssh client block matches one spelling of a machine's name' (#221) from issue/172-the-ssh-client-block-matches-one-spelling-of-a-machine into main 2026-09-30 13:25:34 +00:00
jschoubben 69a002fce3 Issue 172: the ssh client block matches one spelling of a machine's name
Reported by the operator: ssh by the bare name logs in, by the mesh name
is refused. The predecessor's generator writes the bare name only; the
mesh's ssh-client roster already matches both and is not yet shipped.
2026-09-30 15:25:30 +02:00
jschoubben 16a1a52cd8 Merge pull request 'Issue 171: a module that names its own resolver knows no mesh name' (#220) from issue/171-a-modules-own-resolver-knows-no-mesh-name into main 2026-09-30 13:20:30 +00:00
jschoubben af170e3a67 Issue 171: a module that names its own resolver knows no mesh name
Found and fixed the afternoon ADR 0148 landed: mailu-admin lost its
database behind Mailu's own resolver. Two catalogue PRs; an insight on
0148 that a container's dns is a decision, not a preference.
2026-09-30 15:20:26 +02:00
jschoubben 6c2d5f5913 Merge pull request 'Group 2 is resolved: containers resolve, nothing is copied, a route's name says where it arrives' (#219) from issue/110-resolved into main 2026-09-30 13:07:14 +00:00
jschoubben 04c9500b5b Group 2 is resolved: containers resolve, nothing is copied, a route's name says where it arrives
Issue 110's cause was not the filter: the runtime had never been told,
and the resolver dropped a query arriving on a bridge. ADR 0148 step 3
landed once it did (109, 151 resolved). ADR 0151 composes a route's
internal name under the serving node and drops the suffixed alias
(139, 157 resolved). Design 08 amended; a fact in 0148 corrected.
2026-09-30 14:56:43 +02:00
13 changed files with 390 additions and 82 deletions
@@ -103,6 +103,13 @@ reintroduces 109 and 135 — silently, and on a live mesh, which is exactly how
([issue 110](../04-ISSUES/110-a-container-on-the-runtimes-own-network-cannot-reach-the-resolver/00-report.md));
and on two of four machines the resolver binds loopback only, so the runtime hands containers a
public resolver instead. Both are prerequisites, not related work.
> **Progressive insight — 2026-09-30, later the same day. The loopback claim was wrong.** The
> resolver bound the private address on all four machines; on two the runtime had never been told
> to use it, and on all four the resolver discarded a query that arrived on the runtime's bridge.
> The step stands; the facts under it were those. Both fixed the same day
> ([issue 110's resolution](../04-ISSUES/110-a-container-on-the-runtimes-own-network-cannot-reach-the-resolver/01-resolution.md)),
> and step 3 landed after them.
2. **The runtime is told which resolver to use, per machine, as a file** — not per container as a
creation-time argument, or the resolver's address is back in every container's identity and the
problem has only got smaller.
@@ -143,6 +150,16 @@ closed by this record, only answered by it.
resolvable inside the mesh — holds unchanged and by the same means the machine already uses.
- **Issue 110 stops being a container-DNS inconvenience and becomes a prerequisite** for the mesh not
restarting itself whenever it learns a name.
- **A container that names a resolver of its own has opted out of the machine's**, and the copy this
record removes was the only reason such a container could reach anything by a mesh name.
> **Progressive insight — 2026-09-30, the afternoon this landed. Found the hard way.** The mail
> system's admin, behind Mailu's own resolver, lost its database the moment the copy went
> ([issue 171](../04-ISSUES/171-a-modules-own-resolver-knows-no-mesh-name/00-report.md)). A `dns` on
> a container is a decision about whether mesh names exist inside it, not a preference; the module
> was corrected, and whether the controller should refuse the contradiction is that issue's open
> question.
- **A container started by hand gets the mesh's names too**, where before only declared containers did.
Design 08 drew that boundary deliberately, on the grounds that reaching into every container is what
a nameserver would be for. This record accepts that consequence rather than working around it: a
@@ -0,0 +1,99 @@
---
topic: the tiers
status: accepted
date: 2026-09-30
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0066-public-routing-is-name-agnostic.md
---
# 151. A route's internal name is composed under the node that serves it
## Context
A module that requires a route is given two names from one label: a public one, `<label>.<public
domain>`, and an internal one, `<label>.<node>.internal`
([ADR 0138](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md)). Both were composed
from the node the module runs on.
The two are answered differently. The public name is published into every machine's roster at the
address of the node whose proxy serves it ([ADR 0066](0066-public-routing-is-name-agnostic.md)), so
it reaches the proxy from anywhere in the mesh. The internal name is answered by every machine's
resolver as *anything under a node's name goes to that node*
([design 08 §2](../03-DESIGN/01-to-be/08-connectivity.md)) — the node it was composed from, which is
the consumer's. Where the proxy runs on another machine, that name sends a client to a machine with
nothing listening, while the public name works
([issue 139](../04-ISSUES/139-an-internal-route-name-resolves-to-the-consumers-node/00-report.md)).
Every route on this mesh today is served beside its module, so it has not been seen; `route` is
provided mesh-wide precisely so that stops being true.
Beside it, the roster gave every routed name a second entry with the mesh's suffix appended —
`<name>.<public domain>.internal` — because it composed a full name for every entry as it does for a
machine. That name resolved on every machine, was served by nothing, and was refused by the proxy at
the handshake; the first three names tried while reproducing an unrelated issue were those, and the
evidence pointed at a regression that had not happened
([issue 157](../04-ISSUES/157-a-routed-names-internal-alias-is-served-by-nothing/00-report.md)).
## Considered Options
**1. Keep the consumer's name and publish it at the serving node's address**, as the public name is.
The name stays `<label>.<consumer>.internal` and an exact roster entry overrides the wildcard.
Rejected: it makes `<x>.<node>.internal` mean *goes to that node* except when it does not, which is
the one rule the resolver design states; it needs an entry per route where the wildcard needed none;
and which of an exact entry and a wildcard a resolver answers first is the resolver's business, which
the mesh deliberately does not know.
**2. A proxy on every machine, so the serving node is always the consumer's.** Rejected for this
question: it is a different decision about what `route` is — a node-scoped seat with a mesh-wide
fallback — and this mesh runs one proxy on the hub today. Whatever is decided there, a route served
from another machine must have a name that reaches it.
**3. Compose the internal name under the node that serves the route.** Chosen.
## Decision
**A route's internal name is `<label>.<serving node>.internal` — composed under the node whose proxy
answers the route, which is the machine the request arrives at.** The public name is unchanged:
`<label>.<public domain>` of the node the module runs on, which is where the operator put it.
Where the proxy runs beside the module — every route on this mesh today — the two nodes are one and
nothing changes. Where it does not, the name says where the request goes, which is what a name under
a node's name has always meant.
**A routed name has no mesh form.** The roster publishes it as itself, once, at the serving node's
address. Only a machine has a bare name beside its full one.
What certifies the internal name is unchanged by this: the proxy that terminates it obtains a
certificate from the mesh's authority for the names it is given, and it is given this one.
Taken on the operator's standing instruction to answer the open design questions in the work order.
## How this is checked
- **Composition.** A controller test contributes a route from a module on one node to a proxy offered
from another, gathered the way the controller gathers a consumer's contribution for a provider on
another machine, and asserts the internal name carries the serving node.
- **Publication.** A controller test renders a roster with a machine and a routed name and asserts
the routed name appears as itself, once, and never with the suffix appended.
- **On the mesh.** After the change no machine's roster carries a `<domain>.internal` entry, and a
route's internal name still answers from a container with a certificate from the mesh's authority.
## Consequences
- **A route served from another machine now has a usable internal name.** The first module assigned
that way will resolve, where before it would have resolved to the wrong machine with no error.
- **The internal name of a route can change when its proxy moves.** A route re-homed from one proxy
to another gets a new internal name, as the design's rule implies; clients that dialled the old one
reach the old machine. The public name does not move with the proxy and is the stable one.
- **The roster is one line shorter per routed name**, and a person reading a hosts file no longer
finds names that resolve to a refusal.
- **Issue 139's second question — a per-node route holder — is left open**, and is a decision about
what a seat is rather than about a name.
## References
- [issue 139](../04-ISSUES/139-an-internal-route-name-resolves-to-the-consumers-node/00-report.md) — the question
- [issue 157](../04-ISSUES/157-a-routed-names-internal-alias-is-served-by-nothing/00-report.md) — the alias
- [ADR 0066](0066-public-routing-is-name-agnostic.md) — routed names propagate mesh-wide; extended here
- [ADR 0138](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md) — how the two names are composed and how far each reaches
- [design 08 §2](../03-DESIGN/01-to-be/08-connectivity.md) — anything under a node's name goes to that node
+1
View File
@@ -200,6 +200,7 @@ python3 00-META/checks/index.py fail if stale
- **0109** — [A package registry seat is one per ecosystem, not one for all of them](0109-a-package-registry-seat-is-one-per-ecosystem.md)
- **0126** — [A module declares its own seats; the mesh reserves its own](0126-a-module-declares-its-own-seats.md)
- **0148** — [The mesh's names are resolved, not copied into every container](0148-the-meshs-names-are-resolved-not-copied-into-containers.md)
- **0151** — [A route's internal name is composed under the node that serves it](0151-a-routes-internal-name-is-composed-under-the-node-that-serves-it.md)
### What runs on them, and how it gets there
+17 -5
View File
@@ -10,6 +10,7 @@ code:
updated: 2026-09-30
decisions:
- 02-DECISIONS/0148-the-meshs-names-are-resolved-not-copied-into-containers.md
- 02-DECISIONS/0151-a-routes-internal-name-is-composed-under-the-node-that-serves-it.md
- 02-DECISIONS/0147-a-module-anchors-the-meshs-authority.md
- 02-DECISIONS/0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md
- 02-DECISIONS/0140-the-filter-constrains-what-arrives-from-outside.md
@@ -305,10 +306,11 @@ and [135](../../04-ISSUES/135-a-containers-mesh-names-are-not-compared/00-report
**A container resolves the mesh's names through its machine's resolver, at the moment it asks, and
nothing is copied.** The resolver is a machine-level process rather than a container, so nothing
circular is being asked for. This is gated on a container being able to reach the resolver from any of
the runtime's networks, which it cannot today
([issue 110](../../04-ISSUES/110-a-container-on-the-runtimes-own-network-cannot-reach-the-resolver/00-report.md)) —
until that lands the mesh keeps copying and keeps comparing, and the order is stated in the record.
circular is being asked for. It was gated on a container being able to reach the resolver from any of
the runtime's networks
([issue 110](../../04-ISSUES/110-a-container-on-the-runtimes-own-network-cannot-reach-the-resolver/00-report.md)),
and landed the day that did, 2026-09-30: the controller writes no mesh name into a container and the
host's digest carries only what the module declared for itself.
The paragraph below states the old boundary, and 0148 deliberately gives it up: a container somebody
started by hand resolves the same names as everything else, because the resolver answers the machine,
@@ -324,6 +326,14 @@ machine — declared or not — is what a nameserver in `resolv.conf` would be f
service, the rest is the node — so what resolves is *anything under a node's name*, going to that
node. What routes it once it arrives is a proxy's, and stays separate.
*2026-09-30.* **So the node in a route's internal name is the one whose proxy answers it**
([ADR 0151](../../02-DECISIONS/0151-a-routes-internal-name-is-composed-under-the-node-that-serves-it.md)).
Composed from the node the module ran on, the name sent a client to a machine with nothing listening
whenever the proxy ran elsewhere
([issue 139](../../04-ISSUES/139-an-internal-route-name-resolves-to-the-consumers-node/00-report.md));
composed from the serving node, the rule above holds without exception. The public name stays the
module's node's, which is where the operator put it.
**The mesh writes the data and runs no daemon.** One wildcard per machine, from the same set that
writes the hosts file. A resolver is third-party software and runs *on* the mesh rather than being
*of* it: the mesh has no business shipping one, choosing which one, or knowing its configuration
@@ -399,7 +409,9 @@ can reach from the outside but cannot resolve from the inside is a name it canno
authority of its own.
**So a granted route is published into internal resolution as well** — the routed name to the node
that serves it, mesh-wide, by the same mechanism that writes the node names. It is *given by the
that serves it, mesh-wide, by the same mechanism that writes the node names — and as itself: a routed
name has no mesh form, and the suffixed alias the roster once added beside it resolved to a refusal
([issue 157](../../04-ISSUES/157-a-routed-names-internal-alias-is-served-by-nothing/00-report.md)). It is *given by the
mesh, not chosen by a module*, for the same reason the node names are: a module listing the routes
would go stale the day one changes. The mesh propagates the names it was told to serve and still
knows nothing about what they mean
@@ -1,8 +1,8 @@
---
status: located
status: resolved
opened: 2026-09-24
located-in: [mesh-host internal/apply]
fixed-by:
located-in: [mesh-controller internal/catalogue/declaration.go (every container was given the roster at creation)]
fixed-by: mesh-controller PR 161 — no container is given a mesh name; it resolves through its machine's resolver (ADR 0148, landed 2026-09-30 once issue 110 did)
amended-design:
---
@@ -73,3 +73,13 @@ copying: a container resolves through its machine's resolver at the moment it as
record reports then has nowhere to occur. It is gated on
[issue 110](../110-a-container-on-the-runtimes-own-network-cannot-reach-the-resolver/00-report.md), so
until that lands the mesh still copies and still compares.
## Resolved (2026-09-30)
110 landed the same day ([its resolution](../110-a-container-on-the-runtimes-own-network-cannot-reach-the-resolver/01-resolution.md)),
and mesh-controller PR 161 then removed the copy: no container is given a mesh name or a mesh address,
and a module's own declared entries are the only `host` lines it carries. Verified on the control-node
after its containers were recreated once — the last time a name will do that: the forge's container
carries no extra hosts and resolves another machine and a routed name through the machine's resolver,
so the shape this record describes has nowhere to occur. Checked in the controller's tests: a
container's declaration is byte-for-byte the same under a roster of one machine and a roster of three.
@@ -1,8 +1,9 @@
---
status: open
status: resolved
opened: 2026-09-24
located-in: []
fixed-by:
located-in:
- mesh-catalog modules/dnsmasq (the runtime was never told; the resolver answered by interface)
fixed-by: mesh-catalog PR 175 (the runtime is reloaded and keeps its containers over a restart) and PR 176 (the resolver answers by address, so a query from a bridge is admitted) — measured 2026-09-30, 01-resolution.md
amended-design:
---
@@ -67,3 +68,6 @@ and [135](../135-a-containers-mesh-names-are-not-compared/00-report.md)).
network is the case with no DNS at all, and it is the case the mesh's own forge runs in. Two of four
machines also bind the resolver to loopback only, so the runtime hands their containers a public
resolver. Both halves are this issue.
*Later the same day: the second half was wrong, and the first had a different cause than the one above.
[01-resolution.md](01-resolution.md) has what was actually found.*
@@ -0,0 +1,64 @@
# 110 — resolved: a container on any network reaches the resolver, and is answered
*2026-09-30. Measured on the three converged machines; the adopted one holds its resolver module until it
is taken and is not covered.*
## What was actually wrong
Not what the report predicted. The report named the filter: a container on the runtime's default
network asks from a bridge address, and the converged filter admitted queries by source address only.
That was true when it was written and was fixed before this issue was ever tested — the filter admits
by the link a packet arrives on ([ADR 0144](../../02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md)),
and a container's bridge is admitted whole. Tested on every machine: the query arrives, the filter
passes it.
Three other things were wrong, each hiding the next.
**The runtime had never been told.** The resolver module writes the runtime's `dns` key into the
runtime's own configuration file. The runtime reads that key when it starts and not on a reload, and on
two machines the runtime predated the file — so every container they started got a public resolver, and
`novox.internal` came back as not existing. Nothing reported this: the file was present and current,
the resolver ran, and a name not existing is a valid answer. Fixed in mesh-catalog PR 175: the module
also sets `live-restore` and reloads the runtime when its file changes, so the one restart the `dns` key
needs no longer stops every container. The restart is then the operator's, once per machine; done on
both today, with every running container kept.
**The resolver dropped the query.** With the runtime corrected, a container's query reached the resolver
— and got no answer, on every machine, including the one whose runtime had been right all along. The
socket was bound to the private address; the filter admitted the packet; dnsmasq received it and
discarded it without a line of log. Its configuration said `interface=mesh0`, and dnsmasq admits a
query by the interface it arrives on when told an interface: a container's query is addressed to the
private address but arrives on the runtime's bridge, and the bridge is not `mesh0`. Fixed in mesh-catalog
PR 176: the resolver is told the address to answer on, not the interface that carries it, and a query to
that address is admitted whatever bridge brings it. The bridges are the runtime's to name.
**The report's second half was wrong.** "Two of four machines bind the resolver to loopback only" was
an inference from the containers' behaviour, and the behaviour had the cause above. The resolver bound
the private address on all four; nothing had asked it there.
## What is verified
From a container on the runtime's default network, started by hand and given nothing, on each of the
three converged machines: `novox.internal` answers with the hub's private address, through the machine's
own resolver. That is the fourth check of
[ADR 0148](../../02-DECISIONS/0148-the-meshs-names-are-resolved-not-copied-into-containers.md) — "on every
network the runtime offers" — and its first step; the record's step 2 (the runtime told per machine, as
a file) was already how the module works. Step 3 may now begin.
## What checks it
By hand, today. Nothing in the mesh asserts that a container can resolve a mesh name: the resolver's
own tests cover what it answers, not who can ask. The check that would have caught all three faults is
the one the report asked for and 0148 lists — a container on the default network resolving a mesh name
— and it is not built. It belongs with the reachability check of
[issue 145](../145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md),
which is parked; until then this is a thing a person verifies after touching the resolver, the filter,
or the runtime's configuration.
## What this cost to find
The three faults produced one symptom — a container that cannot resolve — and each fix revealed the
next. The first was found by reading the runtime's own view of its configuration rather than the file;
the second by capturing the query on the bridge and finding it arrive and go unanswered; the third only
by admitting the first belief was wrong. A machine that had been believed to work all day had never
worked either.
@@ -1,9 +1,9 @@
---
status: open
status: resolved
opened: 2026-09-28
located-in: [mesh-controller internal/catalogue]
fixed-by:
amended-design:
located-in: [mesh-controller internal/catalogue/declaration.go (composeName took the consumer's own name as the internal domain)]
fixed-by: mesh-controller PR 163 — the internal name composes under the node whose proxy serves the route (ADR 0151, 2026-09-30)
amended-design: 03-DESIGN/01-to-be/08-connectivity.md
---
# 139 — An internal route name resolves to the consumer's node, not the one that serves it
@@ -49,3 +49,15 @@ resolve whether or not anything answers.
and if so, is `route` still one mesh-wide provision or a node-scoped seat with a mesh-wide fallback?
- What certifies the name in either case? The certificate is obtained by whoever terminates TLS, and
that is the question above in another form.
## Answered (2026-09-30)
[ADR 0151](../../02-DECISIONS/0151-a-routes-internal-name-is-composed-under-the-node-that-serves-it.md):
the internal name is composed under the node that serves the route — the machine the request arrives
at — because `<x>.<node>.internal` means *goes to that node* and nothing else. The public name stays
the consumer node's, which is where the operator put it. The first question is answered that way; the
second, a per-node route holder, is a decision about seats and is left where it is; the third is
unchanged, since the proxy that terminates the name is given it and certifies it.
mesh-controller PR 163 carries it. On this mesh every route is served beside its module, so no name
changed; the controller's tests hold the case where it would.
@@ -1,10 +1,10 @@
---
status: open
status: resolved
opened: 2026-09-29
located-in:
- mesh-host internal/apply/apply.go (containerSpecReading hashes every `host` entry)
- mesh-controller internal/catalogue/declaration.go (withMeshNames gives every container the mesh's names)
fixed-by:
fixed-by: mesh-controller PR 161 — the roster left every container's declaration and so its digest (ADR 0148 step 3, 2026-09-30)
---
# 151 — A new name recreates every container in the mesh
@@ -92,3 +92,15 @@ copying names until a container can reach the resolver from any of the runtime's
([issue 110](../110-a-container-on-the-runtimes-own-network-cannot-reach-the-resolver/00-report.md)),
which it cannot on two of four machines today. Removing the copy first reintroduces 109 and 135
silently, on a live mesh, which is how both were found. The order is in the record.
## Resolved (2026-09-30)
Step 3 landed the day 110 did. mesh-controller PR 161 stops writing the roster into any container, so
a container's digest no longer carries a name that is not its own. The controller's tests hold the
record's check — a container's declaration does not move when the mesh's roster does, and does move
when the module's own declared entries do.
The first push after the change recreated every container once, because every digest lost its host
entries at the same moment. That was the last such event: from here a name added or moved on one
machine changes no container anywhere, and the record's second check — add a routed name, watch every
other machine's apply report say nothing changed — is what the next module assignment will show.
@@ -1,9 +1,9 @@
---
status: located
status: resolved
opened: 2026-09-30
located-in:
- mesh-controller internal/catalogue/roster.go (the roster's entries for a routed name)
fixed-by:
fixed-by: mesh-controller PR 163 — a routed name is published as itself, once, with no suffixed alias (ADR 0151, 2026-09-30)
amended-design:
---
@@ -61,3 +61,10 @@ composition produces `keycloak.novox.be.internal`, which is not a name anything
The fix is a judgement about what a routed name's internal form is, and 139 is the record that asks it;
this one is the evidence that the current answer publishes a third thing that is neither.
## Resolved (2026-09-30)
The judgement 139 asked for is [ADR 0151](../../02-DECISIONS/0151-a-routes-internal-name-is-composed-under-the-node-that-serves-it.md):
a routed name has no mesh form. The roster now publishes it as itself, once, at the serving node's
address; the `<domain>.internal` line is gone from every machine's hosts file, and a controller test
refuses it coming back.
@@ -1,63 +0,0 @@
---
status: open
opened: 2026-09-30
located-in:
- mesh-controller internal/catalogue/resolve.go (holdings are derived from every resolved assignment's manifest `claims`)
- mesh-controller cmd/mesh-controller/seats.go (the deliberate act exists — HoldSeat, "recording … as its standing holder" — beside it)
fixed-by:
amended-design:
---
# 170 — Assigning a module claims every seat it could hold
## What was observed
ace's migration needs a postgres of its own: the operator's decision is that a `postgres` module
assigned on ace provides `postgres-database` to ace's modules and has **nothing to do with the
`mesh-store` seat**, which novox's assignment holds by a deliberate act already taken ("make
novox's postgres the mesh-store").
`assign ace postgres` (2026-09-30):
```
ace is assigned postgres
AND 1 other machine(s) cannot be worked out as things stand, so nothing will be sent to them:
novox
- postgres on novox claims "mesh-store", which postgres on ace already holds — one per mesh
mesh-controller: these assignments cannot be applied:
- postgres on ace claims "mesh-store", which postgres on novox already holds — one per mesh
```
The second assignment did not merely fail: it made **the control plane's own store's
assignment unresolvable** until unassigned. Nothing was pushed; the state is restored.
## Why
`resolve.go` derives what a node holds from the manifest's `claims` of every module resolved on
it, so a claim in a definition is a claim by every assignment of that module. The deliberate
act ADR 0110 describes exists beside it — `seat …` records "X on Y as its standing holder"
(`HoldSeat`) — but resolution does not consult that record; it consults the manifests.
## What was decided
[ADR 0110](../../02-DECISIONS/0110-a-seat-is-a-module-assignment-from-a-closed-set.md):
> **A definition says which seats a module *can* hold. An assignment says which it *does*
> hold.** The store module can hold `mesh-store`, and it may be assigned to every node. Exactly
> one of those assignments holds the seat, because that assignment said so.
The manifest's `claims` is being read as *does hold*.
## What would be right
Resolution takes the holder of a seat from the recorded holding (the seat's standing holder),
not from the manifests: a module whose definition can hold a seat is assignable anywhere, and only
the assignment recorded as holder claims it — with the refusal reserved for a second *recorded*
holder at the seat's scope. Assigning postgres to ace is then exactly what the operator said it
is: a database provider on ace, and no more.
## Until then
`postgres` cannot be assigned on any second node; ace's database windows (baserow, letta, n8n,
car-hunter, txt-game) wait on this.
@@ -0,0 +1,78 @@
---
status: resolved
opened: 2026-09-30
located-in:
- mesh-catalog modules/mailu (eight containers name Mailu's own resolver, and one of them binds a mesh name)
- mesh-catalog modules/dnsmasq (dropped the DNSSEC bit its upstreams set)
fixed-by: mesh-catalog PR 178 (mailu-admin uses the machine's resolver) and PR 179 (the machine's resolver passes the DNSSEC bit down) — 2026-09-30, the same afternoon
amended-design:
---
# 171 — A module that names its own resolver knows no mesh name
## What was observed
The afternoon [ADR 0148](../../02-DECISIONS/0148-the-meshs-names-are-resolved-not-copied-into-containers.md)
landed — no container is given the mesh's names any more; it asks the machine's resolver — the mail
system's admin container began logging, 523 times in three minutes:
```
psycopg2.OperationalError: could not translate host name "novox.internal" to address: Name does not resolve
```
Mail was accepted on every port and the web front answered; the admin and the spam filter beside it
were unhealthy, and anything that needed the database — a mailbox change through the API, the spam
filter's domain list — failed. Found by the operator asking whether mail was back, forty minutes in.
Mailu ships its own resolver, an unbound in a container, and every other Mailu container is told to
use it — the module carries `dns: [192.168.203.254]` on eight containers. That resolver recurses from the
root and knows nothing under `.internal`. Until that afternoon the admin container had the database's
name anyway, because the mesh wrote every name into every container at creation; the copy was the only
reason a container behind its own resolver could reach anything by a mesh name, and nobody knew it was
load-bearing.
**Removing the override was not enough.** Given the machine's resolver instead, the admin refused to
start: `Your DNS resolver at 127.0.0.11 isn't doing DNSSEC validation`. Mailu checks, at start, that
its resolver returns the Authenticated Data bit for a signed name. The mesh's resolver forwards to two
upstreams that validate and set the bit, and dropped it on the way down — dnsmasq does unless told
otherwise. Mailu's own unbound has no hook to forward a zone elsewhere, so it could not be taught the
mesh's names either.
## Why it matters beyond this instance
**A container with a resolver of its own has opted out of the machine's, and nothing says so.** 0148
made the machine's resolver load-bearing for every container; a `dns` on a container is a quiet
exception to that, and the exception used to be papered over by the copy the record removed. The
manifest field reads like a preference and is a decision about whether mesh names exist inside the
container.
**A resolver that forwards to validating upstreams and hides the fact is less useful than it could
be**, and the first program to check found out.
**The mesh reported nothing.** Every container ran; the failing one accepted connections; the report
was about bytes. It is [issue 145](../145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)
again, and the check that would have caught it is the same unbuilt one.
## What was done
- The one Mailu container that binds a mesh name — the admin, through the database it is granted —
no longer names Mailu's resolver and uses the machine's, like every container without a `dns` of
its own (mesh-catalog PR 178). The other seven keep unbound: the spam filter needs a validating
resolver for its blocklist lookups, and none of them asks for a mesh name.
- The machine's resolver passes the DNSSEC bit down from its upstreams, `proxy-dnssec` (PR 179). It
does not validate itself; the trust is the upstream's and the path to it, as a forwarding resolver's
always was, and the configuration says so.
## What checks it
The admin container's own start-up check, which is what failed, and the mesh's status once it reads
healthy. A container-level check that a mesh name resolves from inside every declared container is
the one 110 and 145 both ask for and is not built.
## Open questions
- Should a container's `dns` be refused, or made to say what it gives up? A module that names its
own resolver and binds a mesh name is a contradiction the controller can see at composition — the
grant hands it a name its resolver will not answer.
- Should the machine's resolver validate rather than proxy? It would cost a trust anchor on every
machine and make the resolver slower to start; proxying was enough for the one program that asked.
@@ -0,0 +1,55 @@
---
status: located
opened: 2026-09-30
located-in:
- the predecessor's terminal module (still generating the operator's ssh client blocks on every workstation)
- mesh-controller internal/catalogue (the ssh-client roster, tested and not yet a catalogue module)
fixed-by:
amended-design:
---
# 172 — The ssh client block for a machine matches one spelling of its name, and the other gets the wrong user
## What was observed
On a workstation, 2026-09-30, reported by the operator. `ssh home-server` logs in; `ssh
home-server.internal` is refused with `Permission denied (publickey)`. The operator expected the
opposite, if either: the full mesh name is the one the resolver serves.
The name is not the fault. Both spellings resolve to the machine's private address — the mesh's roster
region in the hosts file carries `<node>.internal <node>` on one line, and the resolver answers
anything under the node's name. What differs is the login: the generated client configuration has a
`Host home-server` block naming the account to log in as, and `home-server.internal` matches no block,
so ssh falls back to the operator's local username, which has no account on that machine. Spelled
`account@home-server.internal` it works.
The file is the predecessor's. `~/.ssh/config.d/mesh` says in its own header that it is generated by
the predecessor's terminal module, which only ever wrote the bare name. The mesh's own ssh-client
roster — every other machine's Host block, written as a marked region of the operator's `~/.ssh/config`
with the account the mesh knows for that machine (to-be 29) — already matches both spellings, and a
controller test holds `Host marge marge.internal`. It is composed and tested in the controller and is
not a module in the catalogue, so no machine receives it; every workstation still runs the
predecessor's generator.
## Why it matters beyond this instance
**A name the mesh serves and a name a person can use are not the same set**, and the difference is
silent. The resolver, the hosts file and the certificate authority all treat `<node>.internal` as the
machine's name; the one file that decides who you log in as does not know it. A person who learns the
mesh's name from `status` or from a certificate and types it is refused with an error that says
nothing about a missing Host block.
**It is the migration story for the operator's own tooling, arriving as a symptom.** The mesh has the
right file and does not ship it. Until the ssh-client roster is a module and is assigned to the
workstations, the predecessor's generator keeps writing a file the mesh has already superseded, and
every such file is one the mesh cannot correct.
## Open questions
- Should the ssh-client roster become a catalogue module now, assigned to every workstation, and take
the predecessor's `config.d/mesh` out of the operator's `Include`? Its content is settled; what is
not is the takeover of a file in a person's home that another generator still writes.
- Should the block match a third spelling — the machine's public name, where it has one — or is that
a different key and a different account?
- What checks it? A controller test holds the two spellings; nothing checks that the file a workstation
actually has is the mesh's rather than the predecessor's.