Files
hq/02-DECISIONS/0009-modules-and-the-graph.md
T
jschoubben 0f7e4ab597 The provisioner, which is where the mesh stops
A password nothing was told to create authenticates nowhere. The mesh
generates one, seals it to both ends and cannot read it — so it cannot
tell the software to accept it either. Something on the providing machine
reads what arrived and makes it true.

That something belongs to the module, not to the mesh. The control plane
decides and never touches a machine; a provisioner runs on the machine
and touches it. What the mesh owns is the contract: a manifest of who
asked and where each credential is, and one file per consumer holding it.

It reconciles and is never told what changed, which forces three things
that are each a fault somebody has shipped: set the password every time
or a rotation changes nothing; remove what nobody asks for or a departed
consumer keeps a login for ever; leave alone what it did not make or it
cannot be run on anything that predates it.

Saying where the mesh stops is the point. It decides, delivers, and can
prove what it delivered; the last inch belongs to whoever knows what
`create role` means.
2026-08-30 01:31:58 +02:00

442 lines
26 KiB
Markdown

---
topic: what runs on it
status: accepted
date: 2026-08-28
deciders: jochen
reconstructed: false
---
# 9. Modules and the graph
*Consolidated 2026-08-28 from six records.*
## Everything is a module
One kind of thing, one manifest describing all of them. A database, a web application, a window
manager and a firewall rule set are all modules — not because they are alike, but because
**anything else means a second kind of thing with its own rules, and then a third.**
**A module is the unit of delivery**: assignable to a node, versionable, replaceable on its own.
## There are no domain modules
An earlier decision grouped modules by domain — four things constituting *how a node is
reachable* becoming one `networking` module. **That was wrong, and the correction is worth
keeping** because the observation behind it was right.
The measurement holds: reachability is the **only** place in the catalogue where modules
genuinely change together under one intent. What did not hold is the conclusion. Tight coupling
means they share an **authority** — one place that decides for all of them — and not that they
should be one artifact. `wireguard` and the proxy are deployed to different sets of nodes, so a
module containing both would be assigned where half of it is unwanted.
> **Coherence is a context. Delivery is a module.**
**Folders assert relationships; edges record them.** What grouping was for — finding things,
seeing what belongs together — is a tag and a query, neither of which anybody has to keep true by
hand.
### What a domain module turns out to be, and why it is not the one refused above
*Written 2026-08-29, from building it. The heading above reads as a contradiction of what now
exists and is not one — but only if the difference is stated, so it is stated here.*
**What was refused contains things. What exists contains nothing.**
| | `networking` as refused | `networking` as built |
|---|---|---|
| what is in it | WireGuard, a proxy, a firewall — artifacts | nothing at all |
| what it says | *these ship together* | *I want a private network and names* |
| what is assigned | one module, half of it unwanted | whatever answers each requirement, each on its own |
The objection above is untouched by this and still correct: a module holding WireGuard and a
proxy is assigned where half of it is unwanted. **A module holding nothing cannot be, because
there is no half.** It is requirements and a name, and every artifact it leads to is still an
ordinary module assigned on its own terms.
**Why it is worth having.** Most people want the network working and do not want to choose a VPN.
`assign networking` finds one answer to each requirement and takes it without asking, because
with one candidate there was never a question — the rule below about refusing does the work.
Somebody who does care assigns the VPN they want, and *that is the whole of choosing*: there is no
flavor field, no variant syntax, and no second verb. **Picking an implementation is assigning a
module.**
**What it costs, stated because it is real.** Adding a second implementation to the catalogue
turns a settled question into an open one for **everyone using the bundle**, not only for whoever
wanted the alternative. Every node assigned `networking` refuses until somebody says which. That
is [the refusing rule](#a-requirement-with-several-answers-is-refused-never-guessed) applied
consistently, and the alternative is a default — which is the flavor field returning under a
better name. The cost is one assignment per node, and the message names the candidates.
**A consequence that had to be found by running it.** A bundle can drag an implementation in
through a requirement nobody looked at. Choosing a different VPN still installed WireGuard,
because the names module needed addresses only WireGuard hands out, and nobody was told. Two VPNs
on one machine is not always wrong — a machine may run one for another purpose — but being **the**
network the mesh runs over is singular, so that is a claim, and the collision is refused by name.
**The general rule: what a bundle pulls in is only as safe as the claims on what it pulls in
from.**
## Three edges
| edge | means | declared? | satisfied |
|---|---|---|---|
| **presence** | that thing must exist and be reachable here | yes | at provisioning |
| **instantiation** | that thing makes something for me and hands back credentials — a database, a bucket, a route | yes | at provisioning, and again whenever it must be |
| **build** | I was compiled against that artifact | **no — read from imports** | **at build, once** |
**Instantiation implies presence; presence does not imply instantiation.**
**A route is an instantiation edge**, and it is worth noticing because the direction is the mirror
of a database: the consumer supplies a target and receives a *name*, rather than supplying nothing
and receiving credentials. Same edge.
**Provider stops being a category.** Any hosted thing can be a factory — an identity provider
grants clients, a mail server grants mailboxes. It is a facet, not a kind.
**A module may also declare what it claims**, because some things cannot coexist and that is a
fact about the module rather than about a particular node. What that means precisely is below.
### Where the answer to a requirement is allowed to live
*Written 2026-08-29, from building it. The table above distinguishes **presence** from
**instantiation** and this is the half of that distinction nobody had noticed was missing: not
what the edge hands over, but **where the thing on the other end is.***
Two different things were both being written as a requirement:
| | *a shell*, *a display server*, *a private network* | *a database*, *an object store*, *an identity provider* |
|---|---|---|
| where the answer lives | **this machine** | **somewhere in the mesh** |
| how it is answered | install another module here | find the node already running it |
| what is missing if absent | a module to assign here | **a decision about where**, which is nobody's to make silently |
Answering the second like the first installs a database on every machine that uses one, which is
what it did.
**So a provided name carries a scope**, the same idea a claim already has, and written short in
the ordinary case so the few that are not node-scoped stand out rather than drowning. Scope is a
property of **the name, not of each provider**: two modules disagreeing about whether a database
is local would make one requirement mean two things depending on which happened to answer it, so
that is refused.
**A requirement answered from the mesh is never satisfied by installing it here.** Nothing, and
the mesh refuses and says which module to assign somewhere. Two, and it refuses and says how to
choose — the same rule as everywhere else, for the same reason: picking is guessing, and the wrong
guess puts somebody's data on a machine they did not choose.
**Choosing is recorded per node**, because that is the granularity the choice actually has — two
machines may reasonably use two different databases and a mesh-wide answer could not say so. A
choice pointing at a machine that does not provide the thing is refused rather than quietly
replaced by one that does, and a single available provider does not override a choice either.
**Both are the same rule: the mesh does not overrule a person, and it does not move data without
being told to.**
**What this is a prerequisite for.** Knowing *which node* answers is the first half of handing a
credential back — you cannot be given a database's password before it is settled whose database it
is. So a node's resolution now records what it takes from elsewhere, which is both the only part
of its set that stops working when a *different* machine goes away, and the place a credential
will hang.
### An edge has two directions, and only one of them is built
*Written 2026-08-29, from building it. The row above already says a consumer **supplies a target
and receives a name**; what it did not say is that those are two separate mechanisms, and that
having one without the other is what forced two modules outside the system entirely.*
| direction | the consumer says | who needs it |
|---|---|---|
| **contribution** | *publish me at this name, on this port* | the proxy, the DNS server, a firewall |
| **binding** | *and give me back a credential to it* | the database, the object store, the identity provider |
**Contribution is built.** A module declares what it contributes to a requirement; the control
plane collects every contribution on a node and writes them to a path the provider named, as a
file, in the mesh's own shape. **Contributing to something is requiring it** — asking to be
published means a publisher must exist, and a module that had to say both would eventually say
one, with the failure appearing as a machine where nothing serves the route.
**The control plane does not know what a reverse proxy is**, and does not write one's
configuration. It delivers the facts; the module turns them into whatever it runs. That boundary
is what makes swapping the proxy cost nothing in any module that publishes through it, and it is
[the same separation](0001-mesh-brokers-nodes-host-agents-think.md) that keeps third-party
software *on* the mesh rather than *of* it. It also costs the host nothing: a received file is a
file, which was checked by putting the control plane's output through the host's own parser rather
than by asserting it.
**Binding is built except for the secret**, and that turned out to be the useful way to cut it.
A provider says what a consumer needs in order to use it — a port, a driver, a realm — and a
consumer says where it wants to be told. The mesh adds the half only it has: **which machine, and
what that machine is called on the private network.** So an application on one node is handed the
address of its database on another, as a file, and reaches it by a name the mesh also created.
**The file states that it carries no credential, and why.** A missing field looks like a bug; a
stated absence looks like a boundary, and somebody wiring this up should not spend an afternoon
looking for a password that was never going to be there.
### And the secret, which is delivered without ever being held
*Written 2026-08-30, after looking at how the existing mesh does it. The design here is a reaction
to a measurement, not a preference.*
**The obvious arrangement is a credentials column, encrypted at rest.** It exists, and its own
tooling records what it bought:
| | |
|---|---|
| the tool for finding a secret matches **by value**, not by name | because one password is in the provisions table, the environment table, each node's environment file in plain text, and **inside every connection string composed from it** — copies its documentation calls *"often the only copies actually in use"* |
| a query against the encrypted column **returns zero rows and proves nothing** | so auditing moved to the decrypted copies on the machines |
**Two faults, and encryption at rest addresses neither.** The control plane can read what it
stores, so a copy of its database is a copy of every credential in the mesh. And one secret has
many homes with nothing tracking them — **composition is what mints the untracked ones**, because
building a connection string centrally creates a new secret-bearing value no rotation path knows
about.
**So the value is sealed to the node that will use it before it is stored.** With a key that node
generated and whose private half the mesh has never seen — a third key beside the identity and the
overlay, for the same reason those are two rather than one. What is stored is unusable by whoever
holds it, the mesh included, and the broker relays a blob it cannot read. This is what makes
[ADR 0004](0004-a-node-and-how-it-joins.md)'s *compromise of a node is compromise of that node*
true of secrets rather than true of identity and quietly false of everything that matters.
**And nothing is composed centrally.** A connection string is assembled on the machine that needs
one, if at all. The mesh delivers parts.
**What it costs, stated because it is real:** the mesh cannot audit by value. That is the right
trade rather than an oversight — a query over an encrypted column could not either, so the audit
was never real. What *is* answerable is which node holds what, which is the question rotation
actually asks.
**A consequence that shapes the mechanism.** The mesh discarded the plaintext, so it cannot
compose a file containing it. The credential is therefore **its own file**, holding the value and
nothing else, beside the readable one. That is better than the alternative it was forced into:
the readable half stays readable in the declaration, and the secret half changes only when the
secret does, so a service reloading on it reloads for a real reason.
**Rotation is generating a new one**, because reading the old one back is not possible. Both ends
are re-sealed and reach their machines in the same push — which removes the window where half the
mesh holds a dead credential, the failure
[recorded in ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md) as consumers on three nodes
holding one for two days.
### The provisioner, which is where the mesh stops
*Written 2026-08-30, from building one and running it against a real database.*
**A password nothing was told to create authenticates nowhere.** The mesh generates one, seals it
to both ends and cannot read it — so it cannot tell the software to start accepting it either.
Something on the providing machine reads what arrived and makes it true. That is a provisioner.
**It belongs to the module, not to the mesh**, and the boundary is the same one that keeps
third-party software running *on* the mesh rather than being *of* it
([ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md)). The control plane decides and never
touches a machine. **What the mesh owns is the contract**, which is two files the host writes from
an ordinary declaration:
| | |
|---|---|
| the manifest | every consumer, what it asked for, and **where** its credential is |
| one file per consumer | that credential, alone in it |
Two files because the mesh discarded the value and cannot compose a document containing it. As
before, the constraint produces the better shape: the readable half stays readable and auditable
in the declaration, and the secret half changes only when the secret does.
**It reconciles; it is never told what changed.** It runs after every declaration and must reach
the same state from wherever it starts. Three consequences, and each of them is a fault that has
been shipped somewhere:
- **the password is set every time, not only on creation** — otherwise the role already exists,
nothing happens, and a rotation reports success while changing nothing
- **what it made and nobody asks for any more is removed** — otherwise a consumer that left keeps
a working login for ever and nothing ever says so. This is the same rule the host follows about
[removing what it declared and no longer declares](../04-ISSUES/010-the-first-declaration-destroys-the-substrate/00-report.md)
- **what it did not make is left alone** — otherwise it cannot be run on a system that predates
it, which is every system anybody would want to adopt
**A missing credential is refused rather than worked around.** A role created without one is a
login nothing can use, and nothing would report it until something tried to connect.
**This is where the mesh stops**, and saying so is the point of the section. It decides, delivers
and can prove what it delivered; the last inch belongs to whoever knows what `create role` means.
**One check that only became possible now.** Two machines wired together across no private network
is a mesh that reports itself configured and does not work, and the failure surfaces as a
connection timing out — the slowest place to find anything. It is refused, and it is only
*checkable* because the network became [something a machine is
given](#what-a-domain-module-turns-out-to-be-and-why-it-is-not-the-one-refused-above) rather than
something it has by virtue of holding an address.
**What the absence cost, measured.** Exactly two modules opened a direct connection to the control
plane's database — the proxy and the VPN — and they are the reason every node permanently holds a
credential to it. Both were doing by hand what this edge is for. The VPN's half is closed by being
[a module whose files are computed](../03-DESIGN/01-to-be/08-connectivity.md); the proxy's is
closed by contribution. **Neither needed a new kind of thing, and both had been outside the model
for as long as there was one.**
### Why the build edge is a different kind
It is fixed inside an artifact rather than negotiated when something runs, and **its only remedy
is a rebuild** — nothing can re-provision it.
It is also **derived rather than declared**, and the asymmetry is deliberate: a runtime edge is an
*intention* somebody has about how the mesh should be wired, and only a person can state it. A
build edge is a *fact about code that already exists*, and a declared list of dependencies drifts
from the imports it describes.
**An artifact is out of date when its source moved, or when anything it was built against moved.**
So what is recorded is a commit *and the identity of every artifact it was built against*, which
is what makes the rebuild set computable and *is this current?* answerable without building.
**The graph measures design quality, not just build order.** A module with many inbound build
edges is one whose every change is expensive — and that is readable before anything is built. The
current shared library is exactly that, and nobody could see it because nothing drew the edges.
## Provisioning is declared, never configured by hand
A module declares what it **provides** and what it **requires**. The mesh satisfies it: a
provisioner belonging to the provider creates the resource and its credential, records the grant,
and the values are derived onto the consumer. **Neither the credential nor the topology is ever
written by hand.** A requirement may name a provider on another node, so cross-node wiring is the
same declaration.
## What a module claims, and why it is not a list of rivals
*Written 2026-08-29, replacing pairwise exclusion.*
**Exclusivity is not a property of a module. It is a property of a singular resource the module
takes over.** Two shells do not compete for anything and any number may be installed. Two display
servers both want the seat, and only one may have it.
> **A module declares what it *claims*. Two modules claiming the same thing cannot both be
> assigned within that claim's scope.**
**Not "xorg conflicts with wayland".** Pairwise exclusion has a property that only shows up later:
adding a third display server means **editing xorg and wayland to know about it**. Every new
module requires changing modules nobody who wrote it owns, and the edits grow as the square of
the count. With a claim, the third one says `claims: the seat` and nothing else changes anywhere.
**The new module is the only thing that has to know anything** — which is the difference between
a catalogue that grows and one that calcifies.
The pattern is common enough to be worth listing, because seeing it is most of understanding it:
| these coexist | these claim one thing |
|---|---|
| shells — bash, zsh, fish | display servers — xorg, wayland (*the seat*) |
| editors — vim, emacs, helix | init — systemd, openrc (*pid 1*) |
| language runtimes | container runtime — docker, podman |
| terminal emulators | reverse proxies — nginx, caddy, traefik (*ports 80/443*) |
| browsers | time — chrony, timesyncd, ntpd (*the clock*) |
| | resolvers — resolved, dnsmasq, unbound (*`/etc/resolv.conf`*) |
| | network management — NetworkManager, networkd, netctl |
| | mail — postfix, exim, msmtp (*port 25*) |
| | audio — pipewire, pulseaudio (*the device*) |
**A claim has a scope**, because not everything singular is singular per machine:
| scope | example |
|---|---|
| **node** | the seat, pid 1, port 443 |
| **site** | a DHCP server on a segment |
| **mesh** | the hub, the control plane |
The last is not new — the mesh already enforces exactly one hub with a unique index
([ADR 0007](0007-connectivity.md)). Scope is that idea, said once rather than hard-coded per case.
**Some conflicts need no claim at all.** Two modules declaring the same file, or binding the same
port, are visible from *what they declare* — the mesh already holds every resource of every
declaration. So a claim is only written for the abstract ones, where nothing in the declaration
reveals the clash. That keeps the manifest small, which is worth protecting.
## A requirement with several answers is refused, never guessed
A module requiring *a shell* may be satisfied by three. The mesh does not pick.
| candidates | what happens |
|---|---|
| exactly one | assigned, silently — there was no choice to make |
| none | refused, naming what is missing |
| several | **refused, naming them**, and a person chooses |
**This is what makes a solver unnecessary.** Counting candidates is a few lines and has no
surprising behaviour; a solver that picks has to be understood before its answer can be trusted,
and it is understood by whoever is debugging it at the time. Nothing here is lost by waiting —
a solver can be added later without changing a single manifest, and the reverse is not true.
**Requiring a module and requiring a capability are different fields**, because the remedies
differ and the message should say which:
- *i3 needs xorg, which is not assigned here* — assign it.
- *this machine has no seat* — wrong machine; nothing can be installed to fix it.
### A capability may carry a value, and that is not a new idea
A capability is a named fact about a machine, **detected and never assumed**. Its presence gates
an assignment; its detail can also carry a value — `seat: card1-DP-1`, `panel: oled`, an
architecture, an amount of memory. Nothing new is needed for that: a verdict has always had a
detail beside its yes or no.
So *can this run here* and *what should it be configured as* are answered by the same fact, read
two ways. A module that must not be assigned without an OLED panel and one that dims itself
differently on one are reading the same line.
**What keeps the set from sprawling is the cost of adding one.** A capability must be detected,
and the detector must say how it knows — so nobody can add one they cannot check, which is the
whole of [04-ISSUES/007](../04-ISSUES/007-an-installed-package-is-not-a-capability/00-report.md):
an installed package was treated as a capability and a node was assigned work it could not do.
**And detectors ship inside the host**, which is one statically linked binary. Adding a capability
means shipping a new host to every node that needs it. That is a real cost and it argues for
keeping the vocabulary small and general — `seat`, not `has-nvidia-with-two-outputs`.
## "Flavor" is retired
It was carrying three unrelated meanings — variants of a thing, a subset of one module a node
installs, and whatever the current system does, which earned two knowledge-base entries about
going wrong. **A word with three meanings cannot be reasoned about**, and every attempt to design
around it produced a rule that was right for one meaning and wrong for the others.
What it was reaching for is two ordinary things:
- **Different modules that provide the same thing.** `zsh` and `fish` both provide *a shell*. They
are two modules, not one module with a switch: they share a name and nothing else — different
packages, different configuration, different everything.
- **One module with a setting.** A monitoring module that is an agent here and a server there is
one module, configured. Nothing varies but a value.
If something is neither, it is probably two modules.
**A third thing it was reaching for, added 2026-08-29:** *I want this working and I do not care
which one.* That is a module with requirements and no files —
[a domain module](#what-a-domain-module-turns-out-to-be-and-why-it-is-not-the-one-refused-above) —
and it is what makes "different modules that provide the same thing" bearable for somebody who
does not want to know there is a choice.
## The core library is the mesh's domain
One module everything may depend on. It holds **what is true of the mesh regardless of which
context you are in**: a module, a node, an assignment.
The test: *would this still mean the same thing in a context that had never heard of the one it
came from?* A node would. A pipeline stage would not — that is delivery's.
**Types ship with the module that owns them**, not here. A consumer needing `inventory`'s types
depends on `inventory` — one narrow, visible edge — rather than everything depending on a hub
where the relationship cannot be seen. **A library everything depends on is expensive to change
whether it holds types or code; the fan-in is what makes it expensive**, which is why *types, not
behaviour* was the wrong guard.
**It stays small on its own.** A domain model changes when what the mesh *is* changes, which is
rare. A drawer labelled *shared* changes whenever anybody writes something reusable, which is
constantly — and *who else might want this* always answers yes, which is how the current one grew.
## Consequences
- **Fewer things will be shared, and some code will be written twice.** That is the trade: the
current library exists because sharing felt free. Two similar functions in two modules is often
the better answer.
- **The check is a measurement rather than a prohibition.** Inbound build edges say when something
is becoming a hub, while it is happening rather than after.
- **Reading build edges needs a language-aware tool per language**, which is the real cost and the
reason declaring them looks tempting. It is still wrong.