Three guards, each silent when nothing is at stake:
- settings: show reads a layer; set says what it adds, changes and removes, refuses a removal
without --replace, and keeps the layer it replaced (settings_history, migration 0057).
- push: plan --diff compares with what the machine was last sent, now kept as a summary that holds
no file content; push with no machine needs --all.
- a running container whose mount would point at another directory holds that machine's push
until --move names the module; the other machines go ahead.
A dry run of an unreviewed branch was heard by the daemon like any build, registered, and its
definition reached a machine (novox/hq issue 240). The mark now travels with the build and the
daemon records, registers and plans nothing for it.
Every assigned module was composed as a bus user, though only one declaring
a broker secret can ever be issued an account; the rest were named on every
status, plan and push as credentials never minted (137 now), burying the
real gaps. Their durable consumers are now derived from what the runtime
carries, so nothing they hear changes. The composed file is unchanged:
those users had no password and were already left out. Fixes hq issue 195.
- mesh-dns-resolver: a mesh seat delivering wildcard-resolution, so every node's resolver
configuration resolves to its one holder; node-dns-resolver kept until nothing claims it.
- ${bound:<provision>:address}: the providing machine's private address, for the one consumer
that cannot use a name — a machine's resolver configuration.
- zone: a module declares the zone it answers and the listen that answers it; the controller
settles it per node, refuses duplicates and shadowing, and hands the resolver .Zones to forward.
- node-hosts-file: a node seat whose holder owns /etc/hosts, with entries/add/remove.
The resolver tests follow the catalogue: no runtime dns (containers copy the machine's resolvers),
live-restore held by resolv-conf, resolv.conf naming the resolver by address then a public one.
The environment and shell contributions were written nowhere on a node without their holder;
they now derive a dependency on node-environment, node-login-shell or node-display-server, met
and refused as ADR 0207's are. Two modules declaring one package, path or unit made the node
unresolvable after the assignment was recorded; that is refused first now, because no later
assignment can complete it.
status reported no unmet dependency on any node once systemd, pacman and docker
were assigned to all four (to-be 42), which is the condition ADR 0207 set for the
switch. A dependency no catalogue module could meet stays a report before and
after the switch, as assign already said it: there is no remedy to name.
assign and unassign say only what they changed on their node; push <node>
lists that node's, push to many counts each and points at status. The
once-per-change log is the serving controller's alone: a one-shot command
starts with no memory, so it logged every node on every call.
Seed node-package-manager and node-container-runtime. Derive each module's
dependencies from its declared service, package and container resources;
judge them over the node's whole set, exempting the foundation. Refuse at
assign (several modules may go on as one act) and at unassign of the last
holder; report at composition in status, behind one switch.
Issue 225. The mesh seals one credential per consumer beside the provider's
contributions file, and wrote it root-owned. That was right while a module's
own code ran in a container as root; ADR 0198 moved that code under the node's
runtime, as the node's account, and the secret stayed root's. On the control
machine two consumers went unprovisioned for three hours and the only sign
was a line reading 'secret not readable yet', 4330 times.
The same sentence is already written for a module's own secrets a few hundred
lines above — 'a root-owned 0600 file is one that process cannot read'. This
is that rule reaching the other kind of secret the mesh writes for a module.
Issue 226. The sweep met a reference recorded with the store's old address,
read 'I will not address this' as 'the store refuses everything', and
collected none of the 1681 it had found. Two changes: references from build
records are read through Recorded, where the provenance is known — not in
LetGo, which cannot tell one registry host from another and must stay strict
— and a reference the sweep will not address is now ErrNotOurs, skipped,
never a reason to stop. Only the store refusing ends a sweep.
make check: the two failures both fail on main as well — the resolver test
(hq 202/203) and the service-manager test, which reads this machine's own
shell environment.
The raise at start was the only place buckets were asserted, so a module
registered and assigned since had none until the control plane restarted —
found on the first module to declare state.
Three things found reading this back, each of which would have been quiet.
A consumer that keeps several holders of one provision (ADR 0094) gets a
login per holder, and a provider derives from the login — so it would make a
resource per holder while the consumer is told one value for the requirement.
That is issue 124's own failure one case to the side: authenticate, then be
refused on every object. Refused now, naming both ends.
The sweep runs inside somebody's build and was unbounded. At most two hundred
artifacts and sixty seconds, stopping at the first refusal because a store
that refuses one refuses all; the rest is offered again next build.
The citation and migration renumbers are in the commit before this one.
A manifest names the state it keeps (state) and reads (reads); the controller
asserts a key-value bucket per name on every raise, grants owners write and
readers read (measured against a running server), issues each assignment its
buckets in the membership, and reports buckets nothing declares without
removing them.
The mesh names what may go from its own build records — a digest it did not
record making is never named, which is what keeps the sweep away from the
images genesis pushed. An artifact stays because a definition the mesh holds
names it, or because it belongs to one of the five most recent successful
builds of its module.
internal/artifacts asks the store to let go of one; internal/inventory
decides and remembers (migration 0055); the sweep runs after a build the mesh
recorded, which is when both the bytes and the keep set moved. Never fatal to
a build.
And the manifest side of while-stopped, refused from the definition alone:
no schedule, run-once, a container the module does not declare, itself.
The controller is a Go program and was the one piece of the mesh's own Go
code still shipped and run as an image (novox/hq issue 213; ADR 0188 §1:
a module's own code is bundles; §3: a service bundle is a process).
The manifest now builds one Go bundle, `controller`, and runs it as the
process `mesh-controller` (`./mesh-controller serve`) under an account
the module declares. What the container gave it, replaced:
- host network: a process is on the host's network; nothing it reads
names a container network
- user 65534: the account `mesh-controller`, which owns its secrets and
its state directory
- the eight mounts: the env names the host paths the mesh already places
(the store, broker and bus files under the state directory, the
broker's certificate under /var/lib/mesh-broker-tls); the `broker`
mount was read by nothing and is gone with the others
- `container-runtime` is no longer required on its machine
Its preparation is the same binary with `prepare`, as a run-once process,
and the process `replaces` the container `server`: the host keeps the
container answering until the process is running (mesh-host). Needs the
previous commit live in the running controller, and the host's
`replaces` on the controller's machine, before it is registered.
No image is built by the mesh any more. The Dockerfile stays for genesis
and the lab (`make image`, its Go base now pinned in the Makefile).
The controller's machine moves it from the container to a process by
starting the process first and removing the container once the process
is up (mesh-host's `replaces`). For that moment two controllers share the
store and the bus. Checked what each does:
- the seat's verbs: a queue group per seat, each call answered once. Safe.
- the controller's consumers on CONTROL and EVENTS: push consumers with
no delivery group, so the second bind is refused with "consumer is
already bound" and serve exited. The process would restart for ever,
the host would never see it up, and the container would never go. The
second controller now stands by and binds when the first lets go
(tested on a real bus; fails without the change).
- plans: read, changed and saved whole by the 30s timer, by build
outcomes, by a merge and by `plans stop`. Two timers would each ask a
tier the other had just asked. Working the plans now takes a
session-level advisory lock on the inventory: the timer skips while
another holds it, the other paths wait for it. Build asks happen only
inside plan work and are covered by the same lock.
Builds of one module in flight together finish in any order, and the mesh
took whatever it heard last as what the module is: RegisterModule overwrote
the module's manifest unconditionally, and Held/BuiltAgainst/ReadRepositories
ordered builds by when they were recorded. A postgres build asked before the
mesh-tools runtime fix finished after the one asked after it, and the next
push deployed the stale image (novox/hq issue 219).
A build is now ordered by when it was asked, read from the build-<nanos> id
the controller writes: build.asked and module.built_asked (migration 0055).
A registration from an earlier request than the module's current one is
recorded and refused as superseded. A plan takes as its outcome only a build
asked at or after its own ask, so an earlier plan's leftover build cannot
settle a later plan. Ids of any other shape keep the old order.
A build asked at a commit recorded that commit as the module's ref. Every merge after it failed to
match the module and its plan left it out without a word, and every plan that rebuilt it asked for
the same old commit again. Registration now keeps the branch the module followed (the default
branch for a new one); matching and re-asking read a recorded commit as the default branch, which
heals records already pinned this way; and a merge says which modules of its repository it leaves
out because they follow another branch.
A merge to the controller's own repository replaces the controller in its first tier; the build
that produced the new one was recorded, the plan never heard it, and it waited for ever with every
later plan behind it. The record is the fact: a build recorded after the ask is the tier's outcome,
whoever was listening when it came.
A manifest names its toolchain by language, not in build.on, so the planner did not know a bundle
depends on the module that publishes its toolchain and built the two in one tier: the bundle
against the old toolchain, recorded as built from the new commit. The edge is read from the
manifest, so it holds before any build recorded it, and a toolchain that moves rebuilds every
bundle compiled in it.
Grants: a principal that serves tools subscribes $SRV.PING/$SRV.INFO and those questions under
each name it serves — its own and no other's; the tool runtime and people may ask. The controller
answers discovery for the mesh-controller seat in NATS's services format, one endpoint per verb it
serves, with the seat's description and schema. module list --json says which modules declare tools,
so the console expects an announcement only from those.
The console's discovery reads the machines and the modules; parsing a printed column breaks when it
is reworded. Both now answer JSON on --json, as status and seats do, and the seat verbs ask for it.
The roster published routed names — public ones first, then (in this PR's first take) internal ones
told apart by suffix. Neither is needed: a node has one internal domain and every route on it is a
name under it, answered by the resolver's per-node wildcard; a node's public domains are public
DNS's. routeNamesInTheMesh and NamesServed are removed, and a test pins .Names to the machines.
NamesServed read a route's public `name` and plan.go then filtered by suffix — telling the mesh's
names from public ones by their spelling, when the mesh composed both itself. It now publishes the
`internal-name` it composed under the serving node (ADR 0151); the suffix filter is gone.
Every routed public name was published into each machine's hosts region at its serving node's
private address. ace's resolver also answers its LAN, so a phone there got the control-node's
tunnel address for the mail server and could not connect. Routes have internal names under the
serving node (ADR 0151), so only names under the mesh suffix are published now.
A plan is ordered by artifacts and says nothing about what must be running before what (ADR 0162);
on 2026-10-03 that put the build machine in tier 0 and the controller in tier 1, and the new build
machine could not bind the worker the old controller had defined. One running order enters the
graph, named as its own edge: a module claiming the build seat follows the control plane, and the
built-by edge from the control plane to that holder yields to it — the controller is built by
whichever build machine is running, as the runtime image always was. The edge orders a plan and
never widens it, like built-by.
On 2026-10-02 a runtime assigned and applied on two machines was undone two seconds later by a
declaration that had the assignments of a minute earlier. Every path composes from the records at
compose time and holds the machines it sends — but the number went on at SEND time, after
composing, so a declaration composed before an assignment changed and sent after a newer one
carried the higher number, and the host, which rightly refuses a lower number, took the older
content as the mesh's newest word. The record of that send was never written either: it is written
after the declaration is away, on the sender's context, and the controller sending it was being
replaced in that very second — status read "applied, current" over a machine just told otherwise.
Now the number is taken before the composition reads anything, in every path, so what was composed
earlier is numbered lower however late it goes out and the host's refusal does what it is for; and
what was sent is written down on a context that outlives the sender, bounded, so a dying controller
still records what it told a machine. The `declare` command — a declaration a person sends by hand —
records its send too. Proven: compositions in one order and sends in the other keep the numbers in
composition order; a send is recorded after the sender's context is cancelled.
`assign` recorded a module and `push` sealed a random own secret where its bus credential belongs;
the process crash-looped until a person ran `module issue` and pushed again, and the only warning was
one line in a list printed on every push. Now assigning a module that declares a broker secret issues
the credential in the same act — kept when one exists, so re-assigning rotates nothing — and when the
bus cannot be reached from here the assignment says which verb to run. A push never seals a
placeholder in a credential's place: a module whose bus user is unminted is refused by name, with the
verb. The control plane's own user is the installer's, seeded at genesis, which the test now says.
And what reads one of a module's own secrets is restarted when it changes — composed for a container
or daemon that names the secret's path in its volumes, environment or env-files, so a manifest need
not say it: the build machine ran on an hour-old credential because its manifest restarted it on its
environment file alone (issue 206). A scheduled or run-once process is left alone; it reads afresh.
A controller that asked node-build-agent from its first run would queue every build where nothing
pulls, and the build that registers build-agent — the first holder — would be among them. So the
role is chosen at ask time from the catalogue: the current role when any assigned module claims it,
the retired one while only the builder does, the current one when neither. Outcomes are followed on
both seats, the controller may publish to both, and a build's log is read under whichever role did
it; a machine on the retired role is proven on the bus to take that role's asks. The switch order
is written where the role is named, and the retired half is marked for removal with the seat row.
After the build role moved to node-build-agent, nothing would hold it until build-agent is
registered — and registering build-agent needs a build outcome that only the running builder
could produce, bound as it was to the old seat by name. One binary, two roles: the seat a machine
serves is the first its credential claims, as the mesh writes the claims beside the credential it
issues (ADR 0159); the old builder keeps draining mesh-build-machine, a build-agent takes
node-build-agent, and what each says about a build goes out as that seat's events, so an outcome
is heard where the asker of that seat listens. A credential naming no claim serves the current role.
The controller widened the bus's from-mesh port to from-anywhere on the
broker's host so a machine could enrol before it had a tunnel. ADR 0169
has machines join through the tunnel and decides the bus is never public;
every live bus connection already comes from the mesh.
A module declaring tools, a container, and a build on mesh-tools' runtime image is a container whose
purpose is serving tools — the pattern the node's tool runtime retires. Once node-tools is in the
catalogue, registering one is refused by name, with the record that says why; before, it is accepted
as it always was, so a mesh converts in the design's order and nothing is refused before there is
anything to move to. This is the mechanism that keeps the old pattern from returning by habit.
Judged from a repository manifest's own build.on, and for a built manifest — which carries no build
— from what its build stood on, now recorded beside the commit as part of a module's provenance.
One verb in the row that this binary cannot run aborted the start, and a stale push that put an
older control plane back took the whole mesh off the bus for ten minutes — recoverable only by a
person running the binary outside its service, because the push that repairs it is one of the verbs
that had stopped being served. Now the verbs it knows are served, the ones it does not answer the
reason, and the start names them once.
Where the node-tools module is assigned, the machine's bus user list gains one principal of kind
node-tools in place of that module's own: it may subscribe every carried module's tool namespace
and every held seat's verbs on its node, read and follow every membership on its node, call any
tool anywhere, answer what it is asked — and consume nothing, because tools are what it runs.
Every other module keeps its own principal, so a module still serving tools from its container
holds its own credential until it moves.
Named exactly as the module it stands for, so `module issue` and `rollout mint` deliver its
credential through the path a module's already takes, into node-tools' own `broker` secret. The
runtime module's name is one constant in each of the broker and catalogue packages, held to one
string by the agreement test, because a rule turns on it.
Beside the named verbs, `command` takes a command line as the controller's
own shell would — `node account g14 jochen`, `node show ace`, `module list` —
splits it as a shell does (quotes group, backslash escapes, nothing expanded)
and runs it in this binary like every other verb. The named verbs keep their
schemas; this is the whole binary, added because the operator decided any
node may call any tool (hq ADR 0175) and a verb per command was the only
thing keeping the rest behind a shell on the control node. ADR 0154 carries
the dated note. Tests: a plain line, quoted words, an empty line and an
unclosed quote refused.
A host reports every table and chain that refuses traffic with its owner,
and a converged machine's found firewall's state. The controller keeps both
on the node's record (migration 0054), shows them on node show, names every
converged machine something other than the mesh filters in status — text
and JSON, and such a machine is not well — and the converge preview lists
what filters the machine with the fate of each: retired with the front end,
left as the runtime's, left as a ban, or left in force and not the mesh's.
What was invisible for eleven hours (issues 144, 145) is said by name.
jschoubben
jochen
mesh-admin