Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
bff32e3370 | ||
|
|
d6b62387f2 | ||
|
|
4789624857 | ||
|
|
e00862e317 | ||
|
|
3a92e4b80c | ||
|
|
bfddf78bf3 | ||
|
|
0b291d89f3 | ||
|
|
a924efcc28 | ||
|
|
b421c73a5a | ||
|
|
017b1401e4 | ||
|
|
476cda417d | ||
|
|
74ba3ff1d4 | ||
|
|
891c7a945e | ||
|
|
83cbeb8db8 | ||
|
|
ab6db9369b | ||
|
|
84571b4825 | ||
|
|
d57196102d | ||
|
|
e6402cf777 | ||
|
|
4f0d144833 | ||
|
|
98d94ef71e | ||
|
|
4e13280604 | ||
|
|
8783a13448 | ||
|
|
a31cfcf461 | ||
|
|
75b3861911 | ||
|
|
694555214a | ||
|
|
a7249541df | ||
|
|
95d8253f71 | ||
|
|
784b487bf9 | ||
|
|
7ae711ba0b |
+1
-1
@@ -40,7 +40,7 @@ another — and a mesh you cannot name precisely is a mesh two people describe d
|
||||
- **the deprecated broker** — the lavinmq module. It was the mesh's bus and is not any more. It
|
||||
keeps running as an **ordinary provider** of the `amqp` provision, for modules that need a
|
||||
message broker of their own the way something needs a database
|
||||
([ADR 0127](../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md)) — no seat, not foundation,
|
||||
([ADR 0127](../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md) (superseded by [ADR 0131](../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md))) — no seat, not foundation,
|
||||
never raised at genesis, and a mesh that never installs it is complete.
|
||||
|
||||
Say *the deprecated broker*, not "the compatibility broker" (it serves the mesh's own modules,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: proposed
|
||||
status: accepted
|
||||
date: 2026-09-25
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
status: superseded
|
||||
superseded-by: 0131-everything-on-the-mesh-speaks-to-the-broker-seat.md
|
||||
date: 2026-09-26
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
|
||||
@@ -4,11 +4,18 @@ status: accepted
|
||||
date: 2026-09-26
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md
|
||||
extends: 02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md
|
||||
---
|
||||
|
||||
# 128. The mesh bus is required, not ambient
|
||||
|
||||
|
||||
> **Pointer repointed, 2026-09-27.** This record was written extending
|
||||
> [ADR 0127](0127-amqp-is-a-provision-not-the-bus.md) (superseded by [ADR 0131](0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)), which
|
||||
> [ADR 0131](0131-everything-on-the-mesh-speaks-to-the-broker-seat.md) has since superseded — AMQP is
|
||||
> not a provision at all. Nothing decided here changes; the frontmatter now rests on the live record,
|
||||
> and the citations below are read with that in mind.
|
||||
|
||||
## Context
|
||||
|
||||
[Design 29](../03-DESIGN/01-to-be/32-what-a-module-declares.md) opened by saying the bus is
|
||||
@@ -72,7 +79,7 @@ process in the path and nothing waiting on a bus account to create bus accounts.
|
||||
whose provider is the mesh itself.
|
||||
|
||||
**A module may also provide a NATS server of its own, and that is a different interface.** Exactly
|
||||
as the AMQP broker provides `amqp` ([ADR 0127](0127-amqp-is-a-provision-not-the-bus.md)), a module
|
||||
as the AMQP broker provides `amqp` ([ADR 0127](0127-amqp-is-a-provision-not-the-bus.md) (superseded by [ADR 0131](0131-everything-on-the-mesh-speaks-to-the-broker-seat.md))), a module
|
||||
may run its own NATS and offer it as a backing service. That interface is **`nats`**; the mesh's
|
||||
own bus is **`mesh-bus`**; the two are never the same name, because a manifest that said `nats`
|
||||
could mean either and the difference is the whole architecture. The rule from 0119 decides which
|
||||
@@ -109,7 +116,7 @@ is legitimate: a private bus is a backing service, never a channel to another mo
|
||||
|
||||
- [ADR 0125](0125-the-bus-is-the-only-broker.md) — superseded by 0119; its bootstrap argument is
|
||||
narrowed here to the case it supports.
|
||||
- [ADR 0127](0127-amqp-is-a-provision-not-the-bus.md) — a broker as a backing service; this
|
||||
- [ADR 0127](0127-amqp-is-a-provision-not-the-bus.md) (superseded by [ADR 0131](0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)) — a broker as a backing service; this
|
||||
applies the same shape to the mesh's own bus and separates the two names.
|
||||
- [ADR 0043](0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md) — authority from
|
||||
declarations, which this leaves untouched.
|
||||
|
||||
@@ -4,14 +4,21 @@ status: accepted
|
||||
date: 2026-09-27
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md
|
||||
extends: 02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md
|
||||
---
|
||||
|
||||
# 130. The predecessor is ending, and its broker goes with it
|
||||
|
||||
|
||||
> **Pointer repointed, 2026-09-27.** This record was written extending
|
||||
> [ADR 0127](0127-amqp-is-a-provision-not-the-bus.md) (superseded by [ADR 0131](0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)), which
|
||||
> [ADR 0131](0131-everything-on-the-mesh-speaks-to-the-broker-seat.md) has since superseded — AMQP is
|
||||
> not a provision at all. Nothing decided here changes; the frontmatter now rests on the live record,
|
||||
> and the citations below are read with that in mind.
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0127](0127-amqp-is-a-provision-not-the-bus.md) settled that the old broker is an ordinary
|
||||
[ADR 0127](0127-amqp-is-a-provision-not-the-bus.md) (superseded by [ADR 0131](0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)) settled that the old broker is an ordinary
|
||||
provider of the `amqp` provision rather than a compatibility module with an end date. It rejected
|
||||
giving it a retirement condition, and said why: *"its clients are not only the predecessor's, so the
|
||||
retirement condition describes a day that will not come."*
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-09-27
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
supersedes: 0127-amqp-is-a-provision-not-the-bus.md
|
||||
---
|
||||
|
||||
# 131. Everything on the mesh speaks to the broker seat, and AMQP is not a provision
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0127](0127-amqp-is-a-provision-not-the-bus.md) settled the old broker as an ordinary provider
|
||||
of an ordinary provision, `amqp`, kept for whatever wanted a message broker of its own. The day the
|
||||
bus moved was the day that framing was tested, and it failed in a way that took the control plane
|
||||
down for an evening.
|
||||
|
||||
Three things came out of the wreckage. **The protocol had leaked into the seat's contract**: for a
|
||||
module to hold `mesh-broker`, it had to provide what the seat delivers, and what it delivered was
|
||||
`amqp` — so the module that will carry the bus on NATS could not hold the seat that names the bus,
|
||||
while the module the mesh was leaving could. **A consumer of `amqp` is not asking for AMQP.** The two
|
||||
modules requiring it wanted the mesh's messaging — to emit an event, to hear a topic — and named the
|
||||
wire protocol only because that was the word available. **And AMQP and NATS are not interchangeable
|
||||
at the wire.** A provision named after a protocol can only ever be answered by that protocol, so once
|
||||
the bus is NATS an `amqp` provision has one possible provider, and it is the thing being retired.
|
||||
|
||||
The operator's position, stated during the outage: modules depend on the broker *seat*, not on a
|
||||
protocol; AMQP is obsolete as anything the mesh's core knows about; a module that depends on `amqp`
|
||||
is wrong; and everything should reach the mesh's bus and be able to emit events and consume topics
|
||||
through it.
|
||||
|
||||
## Decision
|
||||
|
||||
**A module that needs messaging uses the mesh's bus, and the mesh's bus is whatever holds
|
||||
`mesh-broker`.** Emitting an event and consuming a topic go through the sdk, which is handed the
|
||||
bus by the mesh with the module's own credential. No manifest names a wire protocol to get it.
|
||||
|
||||
**`amqp` is neither a provision nor a requirement.** Registration refuses a manifest that provides
|
||||
it or requires it. The `mesh-broker` seat delivers `mesh-bus`, and its holder is the module that
|
||||
provides `mesh-bus` — today the nats module, and only it.
|
||||
|
||||
**The old broker's module and the two modules that required it leave the catalogue.** They are
|
||||
removed, not converted: one was a proof that a grant worked end to end, the other forwards mail off a
|
||||
queue, and both are re-done against the bus if wanted, as new modules under this record.
|
||||
|
||||
**The controller's AMQP transport is deleted once every node reports on the new bus**, and the
|
||||
switch that selects a transport goes with it — one bus, so nothing to select.
|
||||
|
||||
The predecessor's own broker is outside the mesh and not this record's concern
|
||||
([ADR 0130](0130-the-predecessor-is-ending-and-its-broker-goes-with-it.md)): what the predecessor's
|
||||
tooling loses when it stops is accepted there.
|
||||
|
||||
## Options considered
|
||||
|
||||
1. **Keep 0127: AMQP stays an ordinary provision with the old broker as its provider.** Rejected. It
|
||||
is what put the protocol into the seat's contract, it is why the seat could be left with no valid
|
||||
holder mid-change, and it keeps two transports in the control plane indefinitely for the benefit of
|
||||
two modules that did not want AMQP in the first place.
|
||||
2. **Bridge it: the old broker's module also provides `mesh-bus`, so both can hold the seat during the
|
||||
change.** Rejected. It makes the retiring broker a legitimate mesh bus for exactly as long as
|
||||
nobody removes the line, which in practice is forever, and it leaves `amqp` as a thing the core
|
||||
still knows the name of.
|
||||
3. **The seat is the dependency; the protocol is nobody's business but the holder's.** Adopted.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The change of holder is a handover, and it needs a command.** Nothing today moves a seat from
|
||||
one assignment to another as one act, and a seat the control plane dereferences cannot be empty
|
||||
in between — that emptiness is the outage this record comes from. The command takes a seat and the
|
||||
assignment taking it over. Designed and built before the cutover, under
|
||||
[28 — Building the bus](../03-DESIGN/01-to-be/28-building-the-bus.md).
|
||||
- **The seat's row moves to `mesh-bus` before the new holder registers, and that is safe.** The
|
||||
control plane composes its own bus address through the seat *by name*
|
||||
(`${seat:mesh-broker:…}`), and the overview derives holders by name; only registration and the
|
||||
provision-to-seat resolution read what a seat delivers. So the row can change under the current
|
||||
holder without unseating it, the new holder can then register its claim, and the handover happens
|
||||
when both are running. Verified in the code during the outage, not assumed.
|
||||
- **Registration gains two refusals**: a manifest providing `amqp`, and one requiring it.
|
||||
- **The `rollout check` stops saying the old broker stays.** It said so under 0127; it now lists
|
||||
unassigning it as the last step of the move.
|
||||
- **What got harder**: a third party that genuinely wants an AMQP broker on a mesh node runs one as
|
||||
any application module, with no provision and no seat, and nothing on the mesh routes to it. That
|
||||
is the cost of the mesh not knowing the word.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| No manifest provides or requires `amqp` | a registration test refusing each, naming this record; and a whole-catalogue test asserting no registered manifest names it |
|
||||
| `mesh-broker` delivers `mesh-bus`, and only a `mesh-bus` provider may hold it | the existing registration test for a delivering seat, with the row's value read from the store (mesh-controller#89) |
|
||||
| The seat's row can change without unseating the holder | a test composing the control plane's own address and the overview under a row that the current holder does not satisfy |
|
||||
| The rollout does not leave the old broker running | `rollout check` output, asserted in its test |
|
||||
| The AMQP transport is gone | the package does not compile with it referenced; the switch variable is refused as unknown at start |
|
||||
@@ -0,0 +1,150 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-09-28
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0129-a-seat-carries-the-protocol-of-its-role.md
|
||||
---
|
||||
|
||||
# 132. A seat carries the tools its holder must serve
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0129](0129-a-seat-carries-the-protocol-of-its-role.md) gave a seat the protocol of its role in
|
||||
three parts: the work it accepts, the events it emits, and the verbs it **serves** — request and
|
||||
reply, awaited. The bus already derives authority from all three: a holder subscribes
|
||||
`mesh.seat.<seat>.tool.<verb>`, and a module that uses the seat may publish it and nothing else.
|
||||
|
||||
**The serving third has never been used.** The mesh defines 14 seats, 8 mesh-scoped and 6
|
||||
node-scoped. Exactly one carries a protocol at all — the build machine, which accepts `build` and
|
||||
emits `built`. Not one seat declares a single verb it serves. The mechanism is built, enforced, and
|
||||
empty.
|
||||
|
||||
Meanwhile every tool on the mesh is addressed to a module. Of 72 modules in the catalogue, 45 serve
|
||||
tools, about 203 of them, each on `mesh.mod.<module>.tool.<name>`. So a caller binds to the module
|
||||
that happens to hold a role rather than to the role, and replacing that module breaks every caller —
|
||||
which is the thing seats exist to prevent everywhere else.
|
||||
|
||||
**Nothing can say what tools exist.** Measured on 2026-09-28, with the bus carrying the whole mesh: a
|
||||
workstation client holding an operator credential connected, the bus accepted the account, and
|
||||
`mesh call gitea.gitea_list_repos` answered with real repositories. The same client's `mesh tools`
|
||||
found nothing, because it asks `mesh-catalog.catalog_tools` and no module serves that: the catalogue
|
||||
serves `catalog_modules`, `catalog_module`, `catalog_provides`, `catalog_dependents` and
|
||||
`catalog_stale`. An agent can therefore call any tool it already knows the name of and discover none.
|
||||
MCP's `tools/list` is that same question, so the MCP surface is a working transport over an empty
|
||||
catalogue.
|
||||
|
||||
**And there is nowhere for a tool's definition to live.** A manifest has a `tools` field: 0 of the 45
|
||||
modules that serve tools fill it. That is not neglect, it is the arrangement failing — the field was
|
||||
the bus grant's source for what a module may subscribe, and because nothing filled it every module
|
||||
that served a tool was refused its own subscription on the new bus, live, until the grant was changed
|
||||
to the module's own namespace. Today a tool's name, description and argument schema exist only in the
|
||||
module's code.
|
||||
|
||||
Two facts about the machinery matter for what follows. A seat's protocol is not in the store: the seat
|
||||
rows lack the ADR 0129 columns, so the protocol comes from compiled defaults and is merged in when a
|
||||
row is read. And `seatSubject` is flat — `mesh.seat.<seat>.<kind>.<verb>` with no node in it — so a
|
||||
node-scoped seat's tool call would reach every node's holder at once, and the holders' queue group
|
||||
would hand it to whichever answered first.
|
||||
|
||||
## Decision
|
||||
|
||||
**A seat's protocol carries its tools in full**: the verb, what it does, and the schema of its
|
||||
arguments and of its answer. The seat is the definition of the role's interface; the holder is an
|
||||
implementation of it.
|
||||
|
||||
**Serving the seat's tools is a condition of holding the seat.** A module that does not serve every
|
||||
verb the seat declares may not occupy it. This is checked where the other conditions of holding are
|
||||
checked — registration and handover — and refused by naming the verbs that are missing.
|
||||
|
||||
**A role's tools are addressed to the role.** `mesh.seat.<seat>.tool.<verb>` mesh-wide. A node-scoped
|
||||
seat carries the node in the address, because one subject reaching six machines' holders is not an
|
||||
address, and the queue group that made it look like one would silently pick a winner.
|
||||
|
||||
**A module keeps its own tools, and both exist.** `gitea_list_repos` stays, because gitea can run
|
||||
without holding the `git` seat — a second forge, an instance kept for one purpose. The module's name
|
||||
answers *this gitea*; the seat's verb answers *whoever is the forge*. Which of the two a caller wants
|
||||
is a decision in the running session, not one the mesh makes for it.
|
||||
|
||||
**What answers "what tools exist" follows where the definition lives.** A seat's tools are read from
|
||||
the mesh's own records. A module's own tools are answered by the module, from the code that defines
|
||||
them. Discovery is therefore a read for the durable half and a question to the running mesh for the
|
||||
free half.
|
||||
|
||||
**A seat's tools are an interface, and change like one.** Additive within a version; a change that
|
||||
would break a caller takes the version token the subject already has room for (design 29 §8), and the
|
||||
two run side by side until nothing is bound to the old one.
|
||||
|
||||
**The mesh's own verbs are the `mesh-controller` seat's tools.** `status`, `push`, `build`, `assign`
|
||||
and the rest are a role's interface, not a container's, and the audit point [ADR 0095](0095-the-control-plane-is-the-way-to-ask-a-module.md)
|
||||
asks for is the seat's holder.
|
||||
|
||||
## Options considered
|
||||
|
||||
1. **The manifest declares each module's tools.** Rejected. The list is then written twice — in the
|
||||
manifest and in the code — and a schema in a manifest goes stale silently, which is the worst kind
|
||||
of wrong for something an agent reads to decide what to call. It is also the arrangement that has
|
||||
already failed once: the field exists, 0 of 45 modules fill it, and the grant that depended on it
|
||||
refused every tool subscription on the mesh.
|
||||
2. **Every runtime answers an introspection call, and something aggregates them.** Rejected as the
|
||||
shape for a role's tools, kept for a module's own. An aggregator needs permission to publish into
|
||||
every module's namespace, which is a widening the mesh otherwise gives only to the control plane;
|
||||
and the answer is only as available as the modules are, so a mesh whose catalogue cannot say what a
|
||||
role answers while its holder is down cannot plan against it.
|
||||
3. **The control plane answers everything.** Rejected. It puts a tool surface on the control plane for
|
||||
tools it does not implement, and makes discovery depend on the one component that must stay
|
||||
answerable while it is itself being replaced. The mesh's own verbs are its to answer, and it answers
|
||||
them as the holder of a seat.
|
||||
4. **Seats only; no module tools.** Rejected. Most modules hold no seat, and inventing a seat per
|
||||
module to give its tools a home would dilute what a seat is: one holder of a role the mesh needs
|
||||
exactly one of.
|
||||
|
||||
## Consequences
|
||||
|
||||
**One capability can have two names, deliberately.** A forge that holds the `git` seat answers both
|
||||
`mesh.seat.git.tool.list_repos` and `mesh.mod.gitea.tool.gitea_list_repos`. This is the one place the
|
||||
mesh accepts two names for one thing, because they are answers to different questions and the second
|
||||
one survives the module not holding the seat. The glossary rule stands everywhere else.
|
||||
|
||||
**A seat becomes a contract to implement.** Adding a verb to a seat is a change every holder must
|
||||
make, and a claim that was valid becomes invalid until it does. That is the point, and it is also the
|
||||
reason a seat's tools should be few and durable while a module's own stay free.
|
||||
|
||||
**Three prerequisites, none of them in place.** The seat's protocol must be in the store rather than in
|
||||
compiled defaults, or discovery reads a binary rather than the mesh. The protocol must become richer
|
||||
than a list of verbs, because a verb without a schema is not something an agent can call. And a
|
||||
node-scoped seat needs the node in its subject before any of its tools can exist.
|
||||
|
||||
**Discovery becomes cheap for the half that matters.** What roles the mesh has and what each answers is
|
||||
a query, with no fan-out and nothing to be up. An agent's authority can then be role-shaped — *the
|
||||
forge's tools* — rather than a list of module-specific names that changes when a module is replaced.
|
||||
|
||||
**The MCP surface belongs inside the mesh.** Once the tools are the mesh's own records, the thing that
|
||||
serves them to an agent is a module the mesh assigns to the machine where the agent sits, with a
|
||||
credential the mesh minted and authority derived from what it may call — not a program started by hand
|
||||
with a credential printed to a terminal.
|
||||
|
||||
## How this is checked
|
||||
|
||||
- **Holding is refused without the verbs.** The condition sits with the other conditions of holding a
|
||||
seat, so registration and a handover both refuse a module that does not serve what the seat declares,
|
||||
and the refusal names the missing verbs. A test per condition, as the other seat conditions have.
|
||||
- **The grant is derived from the seat, and already is.** A holder's subscription and a user's publish
|
||||
come from the seat's protocol, so a verb nobody declared is a subject nobody may use, and a verb the
|
||||
seat declares reaches exactly its holder. The golden composition of the bus's user list is the test
|
||||
that keeps it honest.
|
||||
- **Discovery is a read, and is tested as one.** What the mesh answers for a seat's tools equals what
|
||||
the seat's records declare — no call to a module in the path, so the test needs no running module.
|
||||
- **A node-scoped seat's subject carries its node**, checked by the same test that checks the subject
|
||||
table: two nodes holding one node-scoped seat derive two addresses.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0129](0129-a-seat-carries-the-protocol-of-its-role.md) — the protocol this widens
|
||||
- [ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md) — the role, its work and its events
|
||||
- [ADR 0095](0095-the-control-plane-is-the-way-to-ask-a-module.md) — a tool call passes one process where an audit belongs
|
||||
- [ADR 0126](0126-a-module-declares-its-own-seats.md) — an event is addressed to its emitter, for the same reason a role's verb is addressed to its role
|
||||
- [`03-DESIGN/01-to-be/26-the-seats.md`](../03-DESIGN/01-to-be/26-the-seats.md) — how a seat is held and handed over
|
||||
- mesh-controller #116, #117, #118 — the grants as they now stand: a module serves its own namespace, the control plane may ask any tool
|
||||
- Measured 2026-09-28 on the live mesh: an operator credential calling a module's tool over the bus answers; `tools/list` finds nothing
|
||||
@@ -0,0 +1,159 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: superseded
|
||||
date: 2026-09-28
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0052-a-step-that-runs-once-before-a-container.md
|
||||
superseded-by: 02-DECISIONS/0135-a-module-version-prepares-its-state-before-it-runs.md
|
||||
---
|
||||
|
||||
# 133. A module owns its migrations, and the mesh owns when they run
|
||||
|
||||
## Context
|
||||
|
||||
On 2026-09-28 the mesh replaced its own control plane, through its own upgrade path, with a build
|
||||
carrying a migration. Nothing applied it. For the next three quarters of an hour every build the mesh
|
||||
made was refused by the store with one line — *column "built_contexts" does not exist* — which reached
|
||||
only whoever happened to be waiting on that build's reply. The images were built and published, so the
|
||||
registry filled with artifacts the mesh has no record of, and the overview went on reporting that every
|
||||
module was current ([issue 133](../04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md)).
|
||||
|
||||
The schema had been created once, at genesis, by an action in the foundation bundle. Nothing ran it
|
||||
again, through many updates of the control plane since.
|
||||
|
||||
**The mechanism to do this right already existed and one module used it wrong.**
|
||||
[ADR 0052](0052-a-step-that-runs-once-before-a-container.md) makes a run-once container a step the host
|
||||
runs to completion before whatever the declaration places after it, and names migrating a schema as the
|
||||
case it exists for. Three facts about how it is used today:
|
||||
|
||||
- The control plane's manifest had no step at all. The immediate fix was to write one by hand, and that
|
||||
hand-written step repeats three environment variables and three volume mounts from the server
|
||||
resource it precedes — six chances to drift from the thing it prepares.
|
||||
- Two other modules hand-write the same shape for the same reason: gitea's admin bootstrap and
|
||||
mosquitto's dynsec seed, each repeating its sibling's image, environment and mounts. One of them
|
||||
ends in `|| true`, which is a lock implemented as a shrug.
|
||||
- The catalogue module takes the other road: it migrates its own schema in its own code when it starts.
|
||||
That failure mode is a crash loop rather than a stop — the catalogue restarted 338 times this
|
||||
morning on an unrelated start-time failure, and nothing anywhere said the mesh's graph had a gap.
|
||||
|
||||
**What the mesh already has, and what HAL needed stages for.** Ordering a provider before its consumer
|
||||
is `providersFirst`, which topologically orders a node's modules. Ordering within a module is
|
||||
declaration order, and a run-once container gates everything after it. Remembering that a step has
|
||||
already run is the digest of its declaration, recorded only after it exits 0
|
||||
([ADR 0018](0018-a-picture-is-read-from-what-runs.md)) — and the image is part of that digest, so a new
|
||||
build re-runs it. Three of the four things a stage system provides are therefore already here. The
|
||||
fourth — that a module has a schema at all — is the only thing missing.
|
||||
|
||||
**Nothing in the catalogue ships a migrations directory.** Of 72 modules, none has one; the modules that
|
||||
migrate do it in their own code. So this is not a decision about where SQL files live. It is a decision
|
||||
about who runs them and when.
|
||||
|
||||
**Two facts bound what is safely expressible.** A node converges toward its own declaration without
|
||||
waiting on any other node. And of the five modules that run on more than one machine today — dnsmasq,
|
||||
fail2ban, networking, networkmanager, sshd — not one wants a store; every module with a database is on
|
||||
exactly one machine.
|
||||
|
||||
## Decision
|
||||
|
||||
**A container may declare steps to run before it.** The same container, run to completion, with
|
||||
different arguments, in order, before it starts. The mesh derives the run-once resources from that
|
||||
declaration, so the image, the environment, the volumes, the network and the credentials come from the
|
||||
one place they are already described and cannot drift from it.
|
||||
|
||||
**A module's migrations are the first user of this, and the module owns them entirely.** The SQL, the
|
||||
order, the idempotence, the lock, and which dialect it speaks. The mesh never learns that postgres and
|
||||
mssql differ, because it runs the module's own image with the module's own arguments against the
|
||||
module's own binding and requires exit 0. A module needing both stores runs one step that does both.
|
||||
|
||||
**The mesh owns the moment, and the gate is the guarantee.** Whether a version may serve when its
|
||||
schema is not there yet is a deployment question, and the mesh is the only thing that can answer it,
|
||||
because the mesh is what starts the container. A step that fails stops the container it precedes, so
|
||||
a failed migration is a version that does not serve rather than a version serving against a store it
|
||||
does not match.
|
||||
|
||||
**Per node, and there is no level.** The step runs wherever the module runs. A step that ran "once,
|
||||
somewhere" would leave every other machine with no gate at all, and additive migrations protect old
|
||||
code against a new schema, never new code against an old one. The cost is an obligation a migration
|
||||
runner already carries: a version table and a lock.
|
||||
|
||||
**"Once, mesh-wide" is what holding a seat means.** A step that is not idempotent — seeding an
|
||||
account, sending a notice, taking a backup — belongs to a module that holds a seat, where the mesh
|
||||
already guarantees one holder, on record, handed over deliberately. That is the answer to the level
|
||||
question rather than a field that has to invent an election and keep it somewhere.
|
||||
|
||||
**Migrations are forward-only and additive.** The step runs before the *new* container starts, so the
|
||||
old one is still running against the new schema for the length of the apply.
|
||||
|
||||
**Declared, never inferred.** The control plane cannot see inside an image, so a module that ships
|
||||
migrations and declares no step is not refusable at registration; it breaks on its first upgrade. This
|
||||
record says so rather than implying a check that cannot exist.
|
||||
|
||||
## Options considered
|
||||
|
||||
1. **Each module migrates itself when it starts** — what the catalogue does today. Rejected: it turns a
|
||||
schema failure into a crash loop instead of a stop, it is invisible in the declaration so nothing can
|
||||
say the module even has a schema, and two machines running the module both migrate at start with
|
||||
nothing sequencing them.
|
||||
2. **The mesh applies migrations itself**, with a driver and a version table per store — HAL's shape.
|
||||
Rejected: the mesh would have to know one store type from another, hold another module's store
|
||||
credentials, and reach a machine to use them, which [ADR 0005](0005-the-node-host.md) forbids. It is
|
||||
also the reason that shape needs levels: something central has to decide where the once happens.
|
||||
3. **A hook lifecycle** — pre-build, post-build, pre-deploy, post-deploy. Rejected: there is no deploy
|
||||
event here to hook. A declaration is a desired state applied in order and reconciled forever, so
|
||||
"pre-deploy" is exactly "a step before this container", pre- and post-build are what a Dockerfile and
|
||||
the artifact list already are, and "post-deploy" has no moment to name.
|
||||
4. **A hook level** — once per module, or once per module-node assignment. Rejected as a field, kept as
|
||||
a property: see the decision. A once-per-module step needs cross-node ordering underneath it to be
|
||||
safe, and a node converging without waiting on its neighbours is worth losing on purpose rather than
|
||||
by accident.
|
||||
5. **Every module hand-writes its own run-once step** — the immediate fix for the control plane.
|
||||
Rejected as the general answer: it duplicates the resource it precedes, in three places already, and
|
||||
a hand-written step is one the next module forgets. Forgetting it is the fault this record exists
|
||||
for.
|
||||
6. **Record a schema level per module in the store.** Rejected: gating makes the invariant true by
|
||||
construction, so a level is a second account of the same fact and the first one to go stale.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Three hand-written steps collapse into one line each**, and the control plane's own migrate step stops
|
||||
repeating its server's environment and mounts.
|
||||
|
||||
**The catalogue's self-migration becomes the exception to remove.** One shape, and the mesh's own
|
||||
control plane is not an exception to it either.
|
||||
|
||||
**A module on two machines with one shared store must lock.** Today none is, so this is an obligation
|
||||
stated before it is needed rather than discovered by two concurrent migrations.
|
||||
|
||||
**There is still no readiness-gated step.** Only an action carries `verify`; a container has no health
|
||||
notion, so "run this once the service answers" remains unexpressible and seeding through a running
|
||||
service's API has no home. That is its own decision about a container's readiness, and this record does
|
||||
not make it.
|
||||
|
||||
**Genesis keeps its own action.** At birth there is no control plane to derive anything from, which is
|
||||
what [ADR 0067](0067-genesis-is-a-pivot.md) already says about that moment.
|
||||
|
||||
## How this is checked
|
||||
|
||||
- **The composition carries the step.** A test on a node's composed declaration: every container that
|
||||
declares steps before it is preceded by them, and the derived step's image, environment, volumes and
|
||||
network equal the container's — so the two cannot drift, which is the failure the hand-written kind
|
||||
has.
|
||||
- **A failed step stops what follows.** The host already refuses to go on past a run-once step that did
|
||||
not exit 0; the test for that is extended to a derived one, so the gate is checked rather than
|
||||
assumed.
|
||||
- **The mesh's own schema is covered by the same mechanism as everything else.** The control plane
|
||||
declares its step in its own manifest, so the case that failed on 2026-09-28 is the case the test
|
||||
covers.
|
||||
- **A module claiming a seat for a once-only step is checked where seats are checked** — the conditions
|
||||
of holding, not a new mechanism.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0052](0052-a-step-that-runs-once-before-a-container.md) — the step this extends
|
||||
- [ADR 0018](0018-a-picture-is-read-from-what-runs.md) — a digest is the record that something happened
|
||||
- [ADR 0005](0005-the-node-host.md) — the control plane decides and never touches a machine
|
||||
- [ADR 0067](0067-genesis-is-a-pivot.md) — why genesis does it differently, once
|
||||
- [issue 133](../04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md) — the failure that produced this record
|
||||
- [`03-DESIGN/01-to-be/32-what-a-module-declares.md`](../03-DESIGN/01-to-be/32-what-a-module-declares.md) §6 — the lifecycle this sits in
|
||||
- Measured 2026-09-28: three hand-written run-once steps repeating their sibling's resource; 0 of 72 modules with a migrations directory; 5 modules on more than one machine, none of them wanting a store
|
||||
@@ -0,0 +1,126 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-09-28
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 134. The mesh says what it applied
|
||||
|
||||
## Context
|
||||
|
||||
The pipeline is observable on the bus from a merge to an artifact, and modules already plug into it:
|
||||
the forge emits `pull.merged`, the build machine's seat emits `built`, the catalogue emits `registered`,
|
||||
`upgraded` and `rebuild-needed`, providers emit `postgres.database.provisioned` and its siblings. Things
|
||||
consume them today — the catalogue consumes `built`, each provider consumes its own provisioning events,
|
||||
model-usage consumes `*.usage.*`, the audit logger consumes `**`. Nothing had to be invented for any of
|
||||
that; subscribing *is* plugging in.
|
||||
|
||||
**It goes dark at the moment it touches a machine.** A host applies a declaration and reports to the
|
||||
control plane on the control branch, which only the control plane may read — correctly, because a report
|
||||
carries what a machine is and enrolment travels the same way. So nothing on the mesh says *this machine
|
||||
now runs version Y of module Z*, or that it refused to, or why.
|
||||
|
||||
What that cost on 2026-09-28, in one morning:
|
||||
|
||||
- A build result the store refused was visible only to whoever was waiting on that build's reply. For
|
||||
three quarters of an hour the mesh built things and recorded none of them, while the overview said
|
||||
every module was current ([issue 133](../04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md)).
|
||||
- A module crash-looping at start — 338 restarts — was found by reading a container's logs by hand.
|
||||
Nothing on the bus said the mesh's graph had stopped learning.
|
||||
- A run-once step that fails now stops an upgrade by design ([ADR 0133](0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md)),
|
||||
and the same silence would cover it: the version simply would not appear.
|
||||
|
||||
**And the one thing the control plane does emit is refused by its own account.** Answering a catalogue
|
||||
that asks to catch up, it publishes each recorded build under `mesh.mod.control-plane.event.…` — a
|
||||
module namespace for a module that does not exist. Its own permissions refuse it, so a catalogue that
|
||||
restarts gets nothing and keeps its gap. The control plane has facts to state and nowhere to state
|
||||
them.
|
||||
|
||||
## Decision
|
||||
|
||||
**The mesh emits the deploy half of the pipeline as facts on the bus.** What a machine now runs, and
|
||||
what it refused to run and why. Both are facts about the mesh doing its work, in the same form as every
|
||||
other fact on the bus, so anything that wants them subscribes the way the catalogue subscribes to
|
||||
`built`.
|
||||
|
||||
**The control plane states them, as the holder of the `mesh-controller` seat.** Its facts live under the
|
||||
seat's own namespace, which is where a role's events belong
|
||||
([ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md),
|
||||
[ADR 0129](0129-a-seat-carries-the-protocol-of-its-role.md)) and which survives the control plane being
|
||||
replaced. That is also what gives the catch-up replay a subject it may publish instead of an invented
|
||||
module namespace.
|
||||
|
||||
**Emitted when what a machine runs changes, not on every convergence pass.** A host reconciles
|
||||
continuously and reports each time; a fact per pass would be a fact per minute per machine that says
|
||||
nothing. The report carries the declaration it applied and what changed, so the control plane has what
|
||||
it needs to speak only when there is something to say.
|
||||
|
||||
**A refusal is a fact with a subject in it** — which machine, which resource, and the reason as the host
|
||||
gave it. A refusal that names only the machine is the silence this record is about, one level up.
|
||||
|
||||
**Reports stay where they are.** A node's report remains control traffic that only the control plane
|
||||
reads. The deploy facts are derived from it, which makes them second-hand on purpose: one emitter, one
|
||||
ordering, and no widening of the narrowest account in the mesh.
|
||||
|
||||
## Options considered
|
||||
|
||||
1. **Leave it as it is, and let whatever cares ask the control plane.** Rejected: asking for a fact that
|
||||
already arrives is the shape the mesh removed everywhere else, and nothing can react at the moment a
|
||||
machine changes — which is exactly when a graph, an audit or an operator wants to know.
|
||||
2. **Each node emits its own facts.** Rejected: it widens every host's account to an event namespace, and
|
||||
a host's authority is deliberately the narrowest in the mesh. Its report already reaches the one thing
|
||||
that can speak for it.
|
||||
3. **Widen who may read the control branch.** Rejected: that branch carries what machines say *to* the
|
||||
control plane, enrolment included. Widening its readers widens that too, for an unrelated reason.
|
||||
4. **A registry of deploy hooks** — something registers interest and is called. Rejected: an event is
|
||||
already the mechanism; there is nothing to register, and a callback is an address the mesh spent
|
||||
[issue 102](../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md)
|
||||
learning not to keep.
|
||||
5. **Put the facts on the node's own declaration stream.** Rejected: that stream is last-per-subject by
|
||||
design, so a machine away for an hour gets exactly the current declaration and nothing older. A
|
||||
history of what happened cannot live in a stream built to forget.
|
||||
|
||||
## Consequences
|
||||
|
||||
**The audit logger gets the deploy half for nothing**, because it consumes everything.
|
||||
|
||||
**A failure becomes visible where the mesh is watched** rather than where someone happened to be
|
||||
looking. That answers the open question [issue 133](../04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md)
|
||||
left about a record the store refused.
|
||||
|
||||
**The catch-up replay stops being a burst of events.** With the control plane able to state its own
|
||||
facts, replaying history as if it were happening now is a choice rather than the only option — and the
|
||||
better shape is the question the catalogue is actually asking, answered once
|
||||
([design 33](../03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md)).
|
||||
|
||||
**The facts are second-hand.** The control plane says what a machine reported, so a machine that cannot
|
||||
reach the bus produces no fact at all. Absence is not health, and what a machine was last heard from
|
||||
stays the place that says so.
|
||||
|
||||
**The events stream carries more.** Bounded by emitting on change rather than on every pass, and each
|
||||
fact is small; the stream's own limits remain what keeps it finite.
|
||||
|
||||
## How this is checked
|
||||
|
||||
- **The control plane's grant names exactly the subjects it emits**, derived from its seat like every
|
||||
other principal's, and the composed user list is compared against a golden file — so a fact it cannot
|
||||
publish fails a test rather than a catalogue's replay.
|
||||
- **A convergence that changed nothing emits nothing.** A test with two identical reports and one
|
||||
expected fact, because the failure this guards against is a fact per minute per machine.
|
||||
- **A refusal names its resource.** A test where a host reports a failed resource and the emitted fact
|
||||
carries which one and why, not merely that something went wrong.
|
||||
- **What the mesh emits is what something consumes.** The subject a module declares it consumes derives
|
||||
to the subject the control plane publishes — the same agreement test that already keeps the
|
||||
controller's own subscriptions honest.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0041](0041-events-are-a-relationship.md) — an event is a relationship, not a call
|
||||
- [ADR 0126](0126-a-module-declares-its-own-seats.md) — an event is addressed to its emitter, because the emitter's identity is the meaning
|
||||
- [ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md), [ADR 0129](0129-a-seat-carries-the-protocol-of-its-role.md) — a role's events belong to the role
|
||||
- [ADR 0083](0083-one-push-leaves-the-mesh-consistent.md) — a report is held for the store rather than lost
|
||||
- [ADR 0133](0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md) — the gate whose failure this makes visible
|
||||
- [`03-DESIGN/01-to-be/32-what-a-module-declares.md`](../03-DESIGN/01-to-be/32-what-a-module-declares.md) §6 — the lifecycle, which ends today at a report nobody else may read
|
||||
- Measured 2026-09-28: 45 minutes of builds recorded nowhere with the overview reporting health; a module at 338 restarts found by hand; the control plane's only emitted event refused by its own permissions
|
||||
@@ -0,0 +1,153 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-28
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
supersedes: 0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md
|
||||
---
|
||||
|
||||
# 135. A module version prepares its state before it runs
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0133](0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md) settled who runs a
|
||||
module's migrations and when, and it said so in the wrong vocabulary. It put the declaration on a
|
||||
*container* — "a container may declare steps to run before it" — and derived the scope of the work from
|
||||
the *machine*. Both are wrong at the level a module author works at, and the second is wrong on the
|
||||
facts.
|
||||
|
||||
**A container is one resource kind the host applies.** A module has code, state and a version; whether
|
||||
its artifact is an image, a bundle or something later is the mesh's business. The module-facing
|
||||
vocabulary for a module's own code already exists and has nothing to do with a container runtime: a
|
||||
module declares **entrypoints** — this file is my tools, this file is my provisioner — and the mesh runs
|
||||
them. A manifest that says "run this container with these arguments, and here are the volumes and
|
||||
environment again" has an author writing down the machine's business twice.
|
||||
|
||||
**And the scope is not the machine's to decide, because the mesh already decided what a state is.** A
|
||||
consumer is a module *on a machine* (migration 0015, from
|
||||
[issue 022](../04-ISSUES/022-one-credential-per-node-per-provision-not-per-module/00-report.md)):
|
||||
the mesh derives a login per consumer and the provider creates a database owned by exactly that login
|
||||
([ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)). So a module on three machines is three
|
||||
consumers, three credentials and three databases. There is no shared state for two machines to race over,
|
||||
and ADR 0133's central caveat — that a module's migrations must take a lock because two machines might
|
||||
migrate at once — describes a situation the mesh does not currently produce.
|
||||
|
||||
That correction makes the whole "level" question HAL answered with stages disappear: the scope of
|
||||
preparation is the scope of the state, and the mesh knows it.
|
||||
|
||||
What the earlier record got right and this one keeps: the module owns the work, the mesh owns the moment,
|
||||
the gate is the guarantee, migrations stay forward-only, and none of it can be inferred from inside an
|
||||
artifact. What produced it also stands — the control plane was replaced with a build carrying a migration,
|
||||
nothing applied it, and for three quarters of an hour every build was refused by the store with one line
|
||||
that reached only whoever was waiting on a reply
|
||||
([issue 133](../04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md)).
|
||||
|
||||
## Decision
|
||||
|
||||
**A module version declares an entrypoint that prepares its state.** One name in the manifest, in the
|
||||
same vocabulary as the entrypoints it already declares for its tools and its provisioner. No container,
|
||||
no command line, no environment, no mounts — those are how a machine runs the module's code, and the
|
||||
module already said that once.
|
||||
|
||||
**The mesh runs it as it runs that module's own code, to completion, in the module's own context.** Every
|
||||
binding, credential and setting the module's code would receive, because it *is* the module's code. How a
|
||||
machine does that is the host's business and stays there: for an image artifact it is the step
|
||||
[ADR 0052](0052-a-step-that-runs-once-before-a-container.md) already defines, and a later kind of artifact
|
||||
changes the host, not the manifest.
|
||||
|
||||
**Preparation gates the version.** A version whose preparation did not succeed does not run — anywhere.
|
||||
Since the rollout already sends machines one at a time and stops at the first that does not take a
|
||||
version, a preparation that fails stops the rollout there, leaving every other machine on the version
|
||||
that works.
|
||||
|
||||
**Preparation is scoped to the state, and the mesh derives that scope.** State the mesh provisions is per
|
||||
consumer — a module on a machine — so preparation happens once per consumer. State the module keeps on
|
||||
the machine is per machine, which is the same answer. A module that holds an exclusive seat has one of
|
||||
itself, so its preparation happens once by definition. No level, no election, no cross-node ordering, and
|
||||
no lock obligation invented for a race the mesh does not create.
|
||||
|
||||
**Once per version per state.** A version bump attempts preparation once against each state it has; the
|
||||
module's own runner decides there is nothing to do, which is what a runner with a version table does
|
||||
anyway. A retry after a partial failure runs it again, so the work is the module's to make safe against
|
||||
that — the one obligation no design can remove.
|
||||
|
||||
**Forward-only and additive.** Preparation runs while the previous version is still serving, so a
|
||||
migration that removes or renames what the old code reads breaks the mesh in the window between the two.
|
||||
|
||||
**Declared, never inferred.** The control plane cannot see inside an artifact, so a module that ships
|
||||
migrations and declares no entrypoint is not refusable at registration. It breaks on its first upgrade,
|
||||
and this record says so rather than implying a check that cannot exist.
|
||||
|
||||
## Options considered
|
||||
|
||||
1. **A container declares steps before it** — [ADR 0133](0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md).
|
||||
Superseded, not because the mechanism is wrong but because the *declaration* is in the wrong place: it
|
||||
makes every module author restate the machine's arrangement, and it ties a module's own lifecycle to
|
||||
one resource kind. The host-side mechanism it named is retained and is now an implementation detail.
|
||||
2. **Each module prepares itself when it starts** — what the catalogue does today. Rejected: a schema
|
||||
failure becomes a crash loop rather than a stop, nothing in the declaration says the module has a
|
||||
state to prepare, and the version serves the moment it starts rather than after the state is right.
|
||||
3. **The mesh applies migrations itself**, with a driver and a version table per store type. Rejected:
|
||||
the mesh would have to know one store from another, hold another module's credentials and reach a
|
||||
machine with them, which [ADR 0005](0005-the-node-host.md) forbids. It is also what forces a stage
|
||||
system: something central has to decide where the work happens.
|
||||
4. **A hook lifecycle** — pre-build, post-build, pre-deploy, post-deploy. Rejected: a declaration is a
|
||||
desired state reconciled forever, so there is no deploy moment to hook. "Pre-deploy" is exactly this
|
||||
record; pre- and post-build are what a recipe and the artifact list already are; "post-deploy" names
|
||||
nothing that happens.
|
||||
5. **A declared level** — once per module, or once per assignment. Rejected: the mesh already knows what a
|
||||
state is, so asking an author to choose is asking them to restate a fact the mesh holds, with a chance
|
||||
of contradicting it.
|
||||
6. **Record a preparation level per module in the store.** Rejected for the reason ADR 0133 gave and this
|
||||
record keeps: gating makes the invariant true by construction, and a level is a second account of the
|
||||
same fact.
|
||||
|
||||
## Consequences
|
||||
|
||||
**An author's whole contract is one line, once.** Write the migration in the module's code, name the
|
||||
entrypoint that runs it, and every later version rolls out as: build, prepare, run — with nothing
|
||||
per-version to remember and nothing about the machine to restate. That is the property this exists for.
|
||||
|
||||
**Three hand-written steps in the catalogue collapse**, and the control plane's own migrate step stops
|
||||
repeating its server's environment and mounts.
|
||||
|
||||
**The catalogue's self-preparation becomes the exception to remove.** One shape, and the mesh's own
|
||||
control plane is not an exception either.
|
||||
|
||||
**A module scaled across machines with one shared state is not expressible**, and this record does not
|
||||
make it so. The mesh gives each consumer its own state; a deliberately shared one is a different
|
||||
provision model, and the place the "once, mesh-wide" question would genuinely return. Named here so it is
|
||||
a decision when it happens rather than a surprise.
|
||||
|
||||
**There is still no readiness-gated step.** Only an action carries `verify`; nothing declares that a
|
||||
service answers, so preparation that must happen *after* something is serving — seeding through its own
|
||||
API — remains unexpressible.
|
||||
|
||||
**Genesis keeps its own action.** At birth there is no control plane to derive anything, which is what
|
||||
[ADR 0067](0067-genesis-is-a-pivot.md) says about that moment.
|
||||
|
||||
## How this is checked
|
||||
|
||||
- **The composition carries the preparation, in the module's own context.** A test on a node's composed
|
||||
declaration: a version declaring a preparation entrypoint is preceded by it, and what it is given
|
||||
equals what the module's own code is given — asserted equal rather than written twice, which is the
|
||||
drift the superseded shape invited.
|
||||
- **A preparation that fails stops the version.** The host does not go past a step that did not complete,
|
||||
and the rollout stops at the first machine that did not take a version. Both are existing behaviours
|
||||
with existing tests; the test for preparation asserts the two together — the machine does not run it,
|
||||
and the machines after it are left alone.
|
||||
- **Once per version per state.** A test that a second convergence of the same version prepares nothing,
|
||||
and that a new version prepares again.
|
||||
- **The mesh's own control plane declares one.** The case that failed on 2026-09-28 is the case the tests
|
||||
cover, rather than a case a comment says is covered.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0133](0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md) — what this supersedes, and why
|
||||
- [ADR 0052](0052-a-step-that-runs-once-before-a-container.md) — the host-side step that implements it for an image artifact
|
||||
- [ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md), [issue 022](../04-ISSUES/022-one-credential-per-node-per-provision-not-per-module/00-report.md) — a consumer is a module on a machine, which is what makes the scope derivable
|
||||
- [ADR 0018](0018-a-picture-is-read-from-what-runs.md) — a digest is the record that something happened
|
||||
- [ADR 0005](0005-the-node-host.md) — the control plane decides and never touches a machine
|
||||
- [ADR 0134](0134-the-mesh-says-what-it-applied.md) — what makes a failed preparation visible
|
||||
- [issue 133](../04-ISSUES/133-the-control-planes-schema-is-migrated-at-birth-and-never-again/00-report.md) — the failure that produced both records
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-28
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0052-a-step-that-runs-once-before-a-container.md
|
||||
---
|
||||
|
||||
# 136. A step gates its module, not the machine
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0052](0052-a-step-that-runs-once-before-a-container.md) made a run-once container a step the host
|
||||
runs to completion, and gave it the same reach a failed action has: it stops everything the declaration
|
||||
places after it. When the only steps on the mesh were a broker's seed and a forge's admin account, that
|
||||
reach was invisible — the thing after the step was the container the step existed for, in the same
|
||||
module.
|
||||
|
||||
[ADR 0135](0135-a-module-version-prepares-its-state-before-it-runs.md) made a step something the mesh
|
||||
derives for **any** module that prepares its state, and that turns the reach into a fault. A module
|
||||
whose database is briefly unreachable now stops every module declared after it on that machine, for as
|
||||
long as it is unreachable.
|
||||
|
||||
**The host already rejected this for every other shape, and says why in its own loop.** From
|
||||
[issue 011](../04-ISSUES/011-one-broken-module-blocks-every-other/00-report.md):
|
||||
|
||||
> It used to stop at the first one, and that made one broken resource hold the whole machine hostage: a
|
||||
> module declaring a package that does not exist meant every module ordered after it was never applied,
|
||||
> for ever, and the mesh reported "failed" without saying that the rest had not been tried. A machine
|
||||
> with one bad module and nine good ones ran none of the nine.
|
||||
|
||||
Everything is attempted and every failure reported — except an action and a run-once step, kept as the
|
||||
deliberate exceptions. So the mesh has two rules about the same question and the wider one is now
|
||||
reachable by any module that declares a schema.
|
||||
|
||||
**And it deadlocks a case the catalogue already named.** The catalogue migrates its own schema when it
|
||||
starts rather than in a step, and says why in its code: *a schema step that had to reach the provider
|
||||
over the overlay would block the very apply that brings the overlay up*. With a machine-wide gate that
|
||||
is exactly right — the step fails, the apply stops, the overlay module after it is never applied, and
|
||||
the next reconcile is blocked the same way. The module that most obviously wants a step could not have
|
||||
one.
|
||||
|
||||
## Decision
|
||||
|
||||
**A step gates its own module.** A run-once container that does not complete stops the rest of *that
|
||||
module's* resources and nothing else. Every other module on the machine is attempted, as every other
|
||||
shape already is.
|
||||
|
||||
**An action still gates the machine.** Genesis is a row of actions, each making the next possible, and
|
||||
they belong to no module — there is nothing narrower for their reach to be.
|
||||
|
||||
**What was not attempted is reported, not inferred from silence.** A skipped resource appears in the
|
||||
machine's account of the apply as skipped, with the reason, because "not attempted" and "nothing to do"
|
||||
are different answers and only one of them is somebody's to fix.
|
||||
|
||||
**A module is the part of a resource's identity before the first dot**, which is how the mesh composes
|
||||
them. What the mesh declares in its own right — a guard, an opening, the adoption's own resources —
|
||||
belongs to no module, and its gate is therefore the machine's.
|
||||
|
||||
## Options considered
|
||||
|
||||
1. **Leave the reach as it is.** Rejected: it reintroduces, through a mechanism now derived for every
|
||||
module, exactly the fault issue 011 removed. A mesh where one module's unreachable database stops a
|
||||
machine converging is worse than one where that module alone is behind.
|
||||
2. **Make preparation not a gate at all** — run it and carry on. Rejected: then a version serves against
|
||||
a state nobody shaped, which is the whole of what ADR 0135 exists to prevent.
|
||||
3. **Order every module's step before everything else on the machine**, so a gate stops nothing that
|
||||
matters. Rejected: it inverts the order a module needs — its files and directories are declared before
|
||||
its step because the step reads them — and it would still stop later modules.
|
||||
4. **Let a module declare how far its step reaches.** Rejected: the answer is the same for every module,
|
||||
and a field would let one be wrong about it.
|
||||
|
||||
## Consequences
|
||||
|
||||
**The catalogue can move to a step.** The reason it migrates at start — that a step blocks the apply
|
||||
that would make its provider reachable — stops being true: the step fails, that module waits, the
|
||||
overlay comes up, and the next reconcile prepares it. One shape for the whole mesh, which is what
|
||||
ADR 0135 asked for and could not have had.
|
||||
|
||||
**A module can sit behind while the machine is otherwise current.** That is the honest state and it is
|
||||
what the report now says. It also means a preparation that never succeeds is a module that never
|
||||
upgrades, quietly, until somebody reads the report — which is an argument for
|
||||
[ADR 0134](0134-the-mesh-says-what-it-applied.md) rather than against this.
|
||||
|
||||
**A module's resources must be ordered within the module for the gate to mean anything.** They already
|
||||
are: the mesh composes a module's resources in the order its manifest declares them, and its own
|
||||
workload comes after the files it reads.
|
||||
|
||||
## How this is checked
|
||||
|
||||
- **A failed step stops its module and nothing else.** A test with two modules: the one whose step
|
||||
failed does not start its workload, the other starts, and the error still says the failure gated
|
||||
something. It fails against the previous behaviour, which is how it was written.
|
||||
- **An action still stops the machine.** The existing test for a failed action is unchanged, and a step
|
||||
with no module in its identity — which is what genesis carries — takes the same path.
|
||||
- **The report names what was skipped.** Asserted in the same test, because a gate nobody can see is
|
||||
indistinguishable from a module that had nothing to do.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0052](0052-a-step-that-runs-once-before-a-container.md) — the step this narrows
|
||||
- [ADR 0135](0135-a-module-version-prepares-its-state-before-it-runs.md) — what made the reach reachable
|
||||
- [issue 011](../04-ISSUES/011-one-broken-module-blocks-every-other/00-report.md) — the same fault, removed once already
|
||||
- [ADR 0134](0134-the-mesh-says-what-it-applied.md) — how a module left behind becomes visible
|
||||
- mesh-host `internal/apply` — the loop whose own comment argued this case for every other shape
|
||||
@@ -135,10 +135,13 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0116** — [The bus is built in five steps, and the protocol moves with it](0116-the-bus-is-built-in-five-steps.md)
|
||||
- **0119** — [A taken tunnel's predecessor is retired once the take is proven](0119-a-taken-tunnels-predecessor-is-retired.md)
|
||||
- **0125** — [The bus is the only broker](0125-the-bus-is-the-only-broker.md) *(superseded)*
|
||||
- **0127** — [AMQP is a provision, not the bus](0127-amqp-is-a-provision-not-the-bus.md)
|
||||
- **0127** — [AMQP is a provision, not the bus](0127-amqp-is-a-provision-not-the-bus.md) *(superseded)*
|
||||
- **0128** — [The mesh bus is required, not ambient](0128-the-mesh-bus-is-required-not-ambient.md)
|
||||
- **0129** — [A seat carries the protocol of its role](0129-a-seat-carries-the-protocol-of-its-role.md)
|
||||
- **0130** — [The predecessor is ending, and its broker goes with it](0130-the-predecessor-is-ending-and-its-broker-goes-with-it.md)
|
||||
- **0131** — [Everything on the mesh speaks to the broker seat, and AMQP is not a provision](0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)
|
||||
- **0132** — [A seat carries the tools its holder must serve](0132-a-seat-carries-the-tools-its-holder-must-serve.md)
|
||||
- **0134** — [The mesh says what it applied](0134-the-mesh-says-what-it-applied.md)
|
||||
|
||||
### Its tiers, from the bottom up
|
||||
|
||||
@@ -202,7 +205,7 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0091** — [A mount is declared, and there are three things it can be](0091-a-mount-is-declared-three-ways.md)
|
||||
- **0099** — [A step that runs once names what it reads, and runs again when it changed](0099-a-step-that-runs-once-names-what-it-reads.md)
|
||||
- **0110** — [A seat is held by one assignment, from a closed set, and it may deliver a provision](0110-a-seat-is-a-module-assignment-from-a-closed-set.md)
|
||||
- **0112** — [A module definition names no node, no mesh and no path: everything it needs is a requirement the mesh resolves](0112-a-module-definition-names-no-node-mesh-or-path.md) *(proposed)*
|
||||
- **0112** — [A module definition names no node, no mesh and no path: everything it needs is a requirement the mesh resolves](0112-a-module-definition-names-no-node-mesh-or-path.md)
|
||||
- **0113** — [The vault makes every shared secret, a provider makes resources and data, and the mesh carries both](0113-the-vault-makes-every-secret.md) *(proposed)*
|
||||
- **0114** — [A credential two parties hold rotates over two credentials; one a single party holds rotates in place, staged; and retiring a credential never removes what it reached](0114-a-shared-credential-rotates-over-two-credentials.md) *(proposed)*
|
||||
- **0115** — [One assignment of a module per node: the module's name is the assignment's identity](0115-one-assignment-of-a-module-per-node.md) *(proposed)*
|
||||
@@ -211,6 +214,9 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0120** — [A roster fact carries its format as a template: the mesh owns the data, the module owns the format](0120-a-roster-fact-carries-its-format-as-a-template.md)
|
||||
- **0121** — [A system seat is named for its scope, and a module may define its own](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md)
|
||||
- **0122** — [A seat is data the controller owns, and a rename is a database update](0122-a-seat-is-data-a-rename-is-a-database-update.md)
|
||||
- **0133** — [A module owns its migrations, and the mesh owns when they run](0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md) *(superseded)*
|
||||
- **0135** — [A module version prepares its state before it runs](0135-a-module-version-prepares-its-state-before-it-runs.md)
|
||||
- **0136** — [A step gates its module, not the machine](0136-a-step-gates-its-module-not-the-machine.md)
|
||||
|
||||
### How it is built
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ code:
|
||||
updated: 2026-09-27
|
||||
decisions:
|
||||
- 02-DECISIONS/0106-the-bus-is-nats.md
|
||||
- 02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md
|
||||
- 02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md
|
||||
- 02-DECISIONS/0116-the-bus-is-built-in-five-steps.md
|
||||
- 02-DECISIONS/0074-the-wire-is-specified-not-the-types.md
|
||||
- 02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md
|
||||
@@ -303,7 +303,7 @@ the private network. It is raised at genesis like the store, adopted as a module
|
||||
phase.
|
||||
|
||||
**The deprecated broker is an ordinary module, not a compatibility layer.** Revision, second review
|
||||
([ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md)): earlier text here, and
|
||||
([ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md) (superseded by [ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md): AMQP is not a provision, and everything speaks to the `mesh-broker` seat)): earlier text here, and
|
||||
ADR 0106 before it, called it `lavinmq-compat` — one purpose, the predecessor's clients, and a
|
||||
retirement condition of no client connected for a period the operator sets. It is none of those.
|
||||
A module may legitimately need an AMQP broker as a **backing service**, the way it needs a
|
||||
@@ -534,7 +534,7 @@ find what changed and why.
|
||||
**Still open:**
|
||||
|
||||
- ~~Whether EVENTS should be one stream or one per emitting module.~~ **Closed**
|
||||
([ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md) (superseding [ADR 0125](../../02-DECISIONS/0125-the-bus-is-the-only-broker.md))): one stream, and not as a
|
||||
([ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md) (superseded by [ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md): AMQP is not a provision, and everything speaks to the `mesh-broker` seat) (superseding [ADR 0125](../../02-DECISIONS/0125-the-bus-is-the-only-broker.md))): one stream, and not as a
|
||||
preference — the streams the bus is made of are composed as configuration before any module
|
||||
runs, because a provisioner is itself a module that needs a bus account to start. Bootstrapping
|
||||
decides it.
|
||||
|
||||
@@ -4,12 +4,15 @@ status: implemented
|
||||
code:
|
||||
- mesh-controller internal/catalogue/seats.go
|
||||
- mesh-controller internal/catalogue/resolve.go
|
||||
- mesh-controller internal/inventory/seats.go
|
||||
- mesh-controller internal/inventory/migrations/0039-a-seat-is-held-by-one-assignment-on-record.sql
|
||||
- mesh-controller cmd/mesh-controller/seats.go
|
||||
- mesh-controller cmd/mesh-controller/source.go
|
||||
- mesh-controller internal/inventory/migrations/0032-a-source-may-live-on-a-seat.sql
|
||||
- mesh-catalog modules/gitea/module.json
|
||||
updated: 2026-09-27
|
||||
decisions:
|
||||
- 02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md
|
||||
- 02-DECISIONS/0126-a-module-declares-its-own-seats.md
|
||||
- 02-DECISIONS/0128-the-mesh-bus-is-required-not-ambient.md
|
||||
- 02-DECISIONS/0111-a-build-source-is-on-the-git-seat-or-external.md
|
||||
@@ -42,6 +45,31 @@ is refused. A seat makes a role singular, never a module.
|
||||
**The seat points at the assignment.** Everything the mesh knows about the holder is what it knows
|
||||
about that assignment: the node, the node's settings for the module, and what the module serves.
|
||||
|
||||
**Which assignment holds a seat is a fact on record, and changes as one act.** Revision, 2026-09-27
|
||||
([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)). Until
|
||||
then the holder was derived — the module that is assigned and claims the seat holds it, and a second
|
||||
eligible assignment was refused. That has no way to pass a seat from one holder to the next without a
|
||||
moment in which nobody holds it, and the controller finds its own bus through one of these seats: that
|
||||
moment took the control plane down for an evening. So the holder is now one row the controller keeps,
|
||||
written by a handover — `seat <name> --to <node>/<module>` — that names the seat and the assignment
|
||||
taking it over and replaces the previous holder in the same write. Between two handovers the seat has
|
||||
exactly one holder, and it is never none.
|
||||
|
||||
Three consequences follow. **A seat with no row is held as it always was**: the sole eligible
|
||||
assignment holds it, and two eligible ones are refused — so a mesh that has never handed a seat over
|
||||
behaves exactly as before, and the row appears the first time somebody does. **With a row, any other
|
||||
assignment whose module could hold the seat is eligible and silent**: neither refused nor holding.
|
||||
That is what lets the next holder run beside the current one until the handover, which the bus's move
|
||||
needs ([28](28-building-the-bus.md), task 5.3). **And a holding is the assignment's**: unassigning the
|
||||
holder takes the row with it, so a seat never points at something that is not running anywhere, and
|
||||
the seat falls back to derivation rather than to nothing.
|
||||
|
||||
The handover refuses what would make the new holder wrong before anything is written: the seat must
|
||||
exist, the assignment must exist, and the module must be able to hold the seat — claim it at its scope
|
||||
and provide what it delivers, judged against the store's row and not against anything compiled into a
|
||||
binary. It does not check that the module is running yet; `push` confirms that afterwards, and a
|
||||
handover that could only be recorded after the new holder was up could not be the switch.
|
||||
|
||||
**The set is closed.** A seat the mesh does not define is refused wherever it is named, and so is one
|
||||
named at the wrong scope. Adding a seat is a decision, recorded, for the reason every addition to the
|
||||
host's vocabulary is one: the set is what a person reads to learn what a mesh can have, and an entry
|
||||
@@ -207,7 +235,9 @@ checked as their tables say:
|
||||
| `mesh-*` is the mesh's, and a module may not declare one | 0118: a registration test refusing a manifest that declares any `mesh-*` seat, naming the prefix. |
|
||||
| Two modules cannot declare the same seat | 0118: a registration test; the second is refused and the first untouched. |
|
||||
| A holder satisfies the seat's protocol | 0118: a claim whose module does not serve what the seat declares is refused at assignment. |
|
||||
| A seat is held by one assignment, and only by one whose module can hold it | 0118: resolution tests for a second holder and for a seat the definition does not name. |
|
||||
| A seat is held by one assignment, and only by one whose module can hold it | 0118: resolution tests for a second holder and for a seat the definition does not name. 0131: `CanHold` is the one judgement, shared by registration and the handover, and its test follows the store's row. |
|
||||
| A holder on record settles the seat; another eligible assignment is silent, not refused | 0131: resolution tests with a recorded holder on the same machine, on another machine, and under a seat's former name; without a record, the old rule's tests still pass unchanged. |
|
||||
| A handover replaces the holder as one write, needs an assignment to point at, and goes with it | 0131: store tests — a second handover leaves one row; a handover to a module not assigned where named is refused; unassigning the holder removes the row. |
|
||||
| A requirement naming a seat is answered by its holder; a foundation seat cannot be named | 0118: resolution tests with a second provider on the consumer's node, with the seat unheld, and naming `mesh-store`. |
|
||||
| Several providers and none local is a person's choice | 0118: an assignment test listing candidates with the seat's holder first and recording the pin. |
|
||||
| `secret` is reserved | 0118: the parser and resolution refusals for another provider and a pin. |
|
||||
|
||||
@@ -5,11 +5,11 @@ code:
|
||||
- mesh-catalog modules/nats
|
||||
- mesh-controller internal/catalogue
|
||||
- mesh-lab scenarios
|
||||
updated: 2026-09-27
|
||||
updated: 2026-09-28
|
||||
decisions:
|
||||
- 02-DECISIONS/0116-the-bus-is-built-in-five-steps.md
|
||||
- 02-DECISIONS/0106-the-bus-is-nats.md
|
||||
- 02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md
|
||||
- 02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md
|
||||
- 02-DECISIONS/0126-a-module-declares-its-own-seats.md
|
||||
- 02-DECISIONS/0074-the-wire-is-specified-not-the-types.md
|
||||
- 02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md
|
||||
@@ -535,7 +535,7 @@ it, and the beds that need a mesh living on NATS can finally run.
|
||||
The outcome carries the module name, because only the manifest says what was built and one
|
||||
message now has three readers. A failed build names none: it produced no module version, and
|
||||
the catalogue would otherwise place something that was never made.
|
||||
- [~] 4.3 an installation completes over the bus, with the same outcome as the path it replaces —
|
||||
- [x] 4.3 an installation completes over the bus, with the same outcome as the path it replaces —
|
||||
**the installer can raise it**: a foundation template that stands up the server, writes the
|
||||
server's own settings and the mesh's first user list beside them, and starts a controller
|
||||
reaching the new bus. What remains is running it, which is 4.1's bed.
|
||||
@@ -655,55 +655,72 @@ healthy while reacting to nothing.
|
||||
- [ ] 5.1 the cutover bed: a mesh on AMQP with a predecessor stand-in on the deprecated broker
|
||||
moves its bus in one rollout, every node reporting on NATS afterwards, the stand-in's own
|
||||
client still connected throughout
|
||||
- [~] 5.2 the rollout: accounts composed, then the controller, every host and every runtime
|
||||
- [x] 5.2 the rollout: accounts composed, then the controller, every host and every runtime
|
||||
together; every node confirmed heard before AMQP stops.
|
||||
|
||||
**The readiness half is in and is the half worth having.** The move takes every node at once, so
|
||||
there is nothing to inspect afterwards and no half to roll back — either the mesh was ready or it
|
||||
was not. `rollout check` answers that from records, with one dial: is a bus answering, does a
|
||||
machine hold the seat, has it been sent the composed user list, does every machine and every
|
||||
module that speaks have a credential. Each missing thing names its own next step, because "not
|
||||
ready" that cannot be acted on is not an answer at the point where the next step is irreversible.
|
||||
|
||||
**A machine with no credential is what must stop it.** It keeps running, cannot come back, and
|
||||
afterwards there is no bus to tell it anything over.
|
||||
|
||||
The move itself is deliberately not written yet, and the command says so rather than pretending:
|
||||
it waits on the check having been run against a real mesh. Writing the irreversible half before
|
||||
the question it depends on has ever been asked of something real is how the plan's own rule about
|
||||
beds gets broken by another route.
|
||||
|
||||
> **What this costs if it goes wrong, measured rather than assumed.** On the installation this is
|
||||
> for, the old broker is also what a whole automation layer outside the mesh connects to — so it
|
||||
> stays, as an ordinary provider of `amqp` ([ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md)),
|
||||
> and this step is not its retirement. Nothing in a served request's path goes over the mesh's own
|
||||
> bus: modules serve from their own containers. What a failed move costs is the mesh's ability to
|
||||
> *change* anything — pushes, tool calls, new provisioning — until it is finished or undone. That
|
||||
> is worth knowing before rather than after, and it is why the operator's "as long as my services
|
||||
> keep running" is a reasonable position rather than a gamble.
|
||||
- [ ] 5.3 the mesh's own accounts removed from the deprecated broker, and then the broker itself:
|
||||
after the rollout nothing of the mesh speaks to it, and an account nothing uses is one nobody
|
||||
rotates. **It finishes now** ([ADR 0130](../../02-DECISIONS/0130-the-predecessor-is-ending-and-its-broker-goes-with-it.md)):
|
||||
the predecessor is deprecated rather than kept, so once its remnants have stopped the module is
|
||||
unassigned and the port is free. No retirement machinery — a provision with no consumers has its
|
||||
provider unassigned, which is ADR 0127 being paid off rather than revised.
|
||||
**Done 2026-09-28, 02:25.** Every machine reports on the new bus, the seat is held by the
|
||||
module that provides it, the old broker is unassigned and forgotten, and every credential was
|
||||
minted afresh at the end because two had been printed on the way. What it took, in the order
|
||||
it was found, each fixed on the trunk before the next step: the control plane's `serve` and
|
||||
`push` never selected the new transport (task 4.3, open until then); a machine's user was
|
||||
granted neither the asking nor the delivery of its own consumer; the account had no JetStream
|
||||
of its own; the control plane's client verified the bus's certificate by name instead of
|
||||
pinning it; the seat table's rows carried no protocol, so no role's work queue was raised; the
|
||||
build machine decided its bus from a variable its container never received; and a rotation
|
||||
put new hashes on the bus before three machines had received their new memberships — which
|
||||
is why there is now `rollout hand <node>` and a host adopts a delivered membership at start.
|
||||
The bootstrap loop — a bus that can only be raised by a declaration that can only arrive
|
||||
over that bus — was broken once, by hand: the mesh's own composed configuration started the
|
||||
server, and the controller binary was run on the node directly until the managed container
|
||||
could be rebuilt over the bus it was on.
|
||||
|
||||
- [x] 5.3 **the seat changes hands as one act.** A command takes a seat and the assignment taking it
|
||||
over, and the seat is never empty in between — the emptiness is the outage of 2026-09-27, when
|
||||
the control plane, which finds its own bus through this seat, lost the address and looped.
|
||||
**Built 2026-09-27** (`seat_holding`, migration 0039; design 26 says how it is checked), and used
|
||||
live the next night to hand `mesh-broker` from the old broker's assignment to the new one's. This
|
||||
is what 5.2 uses to move `mesh-broker` from the old
|
||||
broker's assignment to the new one's, and it is built first ([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)).
|
||||
- [x] 5.4 **the old broker and everything that named AMQP leave the mesh** ([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md),
|
||||
superseding [ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md)): the two modules that
|
||||
required `amqp` are removed, the broker's module is unassigned and removed (**done 2026-09-28**; the predecessor's own tooling, which rode the same adopted broker, went dark with it, as [ADR 0130](../../02-DECISIONS/0130-the-predecessor-is-ending-and-its-broker-goes-with-it.md) accepted), registration refuses
|
||||
a manifest that provides or requires `amqp`, and a whole-catalogue check asserts none does. Not
|
||||
a retirement condition — a decision, taken, with the operator's "I don't care if the predecessor
|
||||
breaks" on record ([ADR 0130](../../02-DECISIONS/0130-the-predecessor-is-ending-and-its-broker-goes-with-it.md)).
|
||||
Retiring with it: the build outcome's second announcement under the module's own name, which
|
||||
exists only so a catalogue deployed before the rename and one deployed after both hear it.
|
||||
existed only so a catalogue deployed before the rename and one after both heard it.
|
||||
|
||||
> **The remote tooling goes with it too.** The predecessor's own mesh talks over that broker, so
|
||||
> shutting it down ends the path that reaches this installation's machines from a workstation.
|
||||
> The rollout has to be driven from the node, or driven before the broker stops — which is a
|
||||
> sequencing constraint on 5.2 and not an afterthought.
|
||||
> The rollout is driven from the node, or before the broker stops — a sequencing constraint on
|
||||
> 5.2, not an afterthought.
|
||||
- [x] 5.5 **the AMQP transport is deleted from the control plane and the hosts**. One bus, nothing
|
||||
to select ([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)).
|
||||
|
||||
> **5.4 is gone, and was wrong from ADR 0127 onward.** It read "the deprecated broker retires
|
||||
> when its condition holds — no client connected for the period the operator sets", which is
|
||||
> ADR 0106's framing of it as a compatibility module with an end date.
|
||||
> [ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md) settled that it is an
|
||||
> **ordinary provider** of the `amqp` provision, like any module answering a backing service:
|
||||
> no seat, not foundation, and **no retirement condition**, because the day its last client
|
||||
> disappears is not a day anything is waiting for. Removed rather than reworded — a step that
|
||||
> waits for a condition nobody set would sit open forever.
|
||||
**Done 2026-09-28.** The control plane's old consume loop, build request, tool call, management
|
||||
API and account scoping went, and the host's old dialling and enrolment paths with them; a
|
||||
membership or token naming any other bus is refused before anything is sent. Nothing selects a
|
||||
transport any more: the variable that once did (`MESH_BUS_NATS`) now only names where the
|
||||
control plane reads its own bus credential, the way any module reads a secret. **Checked by the
|
||||
build**: neither repository's module file names the AMQP client library, so a line that still
|
||||
used it would not compile. The store-window guarantee ([issue 083](../../04-ISSUES/083-other-control-messages-are-lost-while-the-store-restarts/00-report.md))
|
||||
is tested against a bus-less fake rather than the old transport's memory, which is what let
|
||||
that memory go — the one thing it did that the stream does not (superseding a held report) is
|
||||
the staleness check on the message itself (design 25 §3).
|
||||
|
||||
Found on the way: **no build had ever recorded what it stood on.** A recipe reads its base from
|
||||
a build argument, so the digest was never in the file the builder derived edges from, and every
|
||||
order that says *bases first* — `build --on`, `build --behind`, the merge follow-up of
|
||||
[issue 131](../../04-ISSUES/131-nothing-tells-the-mesh-a-source-moved/00-report.md) — walked a
|
||||
graph with no edges. The builder now reports the bases it was handed, the control plane records
|
||||
them by artifact path, and the graph is read from the newest build of each module — a recorded
|
||||
manifest carries no `build.on`, so the edge is derived from the build or it does not exist.
|
||||
|
||||
> **The old 5.4 note is history.** It recorded that a retirement *condition* was wrong from
|
||||
> [ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md) onward, which framed the old broker
|
||||
> as an ordinary provider with no end. [ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md) ends that
|
||||
> framing in turn: the broker is not kept as a provider either, because AMQP is not a provision. Both
|
||||
> readings are kept here so the two reversals can be read in order.
|
||||
|
||||
**Done when.** Every node reports on NATS, and nothing of the mesh's own is left connected to the
|
||||
deprecated broker.
|
||||
|
||||
@@ -1,17 +1,29 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: proposed
|
||||
code: []
|
||||
updated: 2026-09-27
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-controller internal/catalogue/declaration.go
|
||||
- mesh-controller internal/catalogue/manifest.go
|
||||
- mesh-controller internal/link/serve.go
|
||||
- mesh-controller internal/link/bus.go
|
||||
- mesh-controller internal/broker/nats.go
|
||||
- mesh-controller internal/inventory/nodes.go
|
||||
- mesh-host internal/apply/apply.go
|
||||
- mesh-tools src/main.ts
|
||||
- mesh-catalog modules/mesh-catalog
|
||||
updated: 2026-09-28
|
||||
decisions:
|
||||
- 02-DECISIONS/0126-a-module-declares-its-own-seats.md
|
||||
- 02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md
|
||||
- 02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md
|
||||
- 02-DECISIONS/0128-the-mesh-bus-is-required-not-ambient.md
|
||||
- 02-DECISIONS/0106-the-bus-is-nats.md
|
||||
- 02-DECISIONS/0041-events-are-a-relationship.md
|
||||
- 02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md
|
||||
- 02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md
|
||||
- 02-DECISIONS/0129-a-seat-carries-the-protocol-of-its-role.md
|
||||
- 02-DECISIONS/0135-a-module-version-prepares-its-state-before-it-runs.md
|
||||
- 02-DECISIONS/0136-a-step-gates-its-module-not-the-machine.md
|
||||
- 02-DECISIONS/0134-the-mesh-says-what-it-applied.md
|
||||
---
|
||||
|
||||
# 32. What a module declares, and what the bus makes of it
|
||||
@@ -258,10 +270,30 @@ queue.
|
||||
and publishes it last-per-subject. A node that was away gets exactly the current one, never a
|
||||
queue of superseded ones, and a replayed older one is refused by sequence.
|
||||
|
||||
**A version prepares its state before it runs.** *Built 2026-09-28.* A module version may declare an
|
||||
entrypoint that brings its state to the shape that version needs — the same vocabulary as the entrypoints it declares for its
|
||||
tools and its provisioner, and nothing about how a machine runs it. The mesh runs that entrypoint as it
|
||||
runs the module's own code, to completion, in the module's own context, and a version whose preparation
|
||||
did not succeed does not run: the step gates that module and nothing else on the machine
|
||||
([ADR 0136](../../02-DECISIONS/0136-a-step-gates-its-module-not-the-machine.md)), and the rollout stops
|
||||
at the first machine that did not take it
|
||||
([ADR 0135](../../02-DECISIONS/0135-a-module-version-prepares-its-state-before-it-runs.md), superseding
|
||||
[ADR 0133](../../02-DECISIONS/0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md)).
|
||||
Once per state, and the mesh derives what a state is: a consumer is a module on a machine, so what the
|
||||
mesh provisions is per consumer and preparation is too. No level to choose, and no race to lock against.
|
||||
|
||||
**Applying is reported to a role.** The host applies and reports to the `mesh-controller` seat —
|
||||
not to an address it was given at genesis. Held and retried while the store restarts
|
||||
([ADR 0083](../../02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md)).
|
||||
|
||||
**And the mesh says what it applied.** *Built 2026-09-28.* A report is control traffic only the control plane reads, so the
|
||||
chain above went dark at the moment it touched a machine: nothing said which version a machine now runs,
|
||||
or that it refused to. The control plane states those as facts under its own seat's namespace, when what
|
||||
a machine runs changes rather than on every convergence pass, and anything that cares subscribes the way
|
||||
the catalogue subscribes to `built` ([ADR 0134](../../02-DECISIONS/0134-the-mesh-says-what-it-applied.md)).
|
||||
The facts are second-hand by design — one emitter, one ordering — and a machine that cannot reach the bus
|
||||
produces none, so absence is not health.
|
||||
|
||||
What disappears across that chain is every address. No webhook URL, no registered callback, no
|
||||
"which node is the builder on", no controller endpoint baked into a joining node. That is the
|
||||
class of bug
|
||||
@@ -357,7 +389,7 @@ controller — because the bus's accounts are configuration rather than somethin
|
||||
creates, so there is no provisioner process in the path and nothing waiting on a bus account in
|
||||
order to make bus accounts. A module that runs a NATS server of its own and offers it as a
|
||||
backing service provides **`nats`**, exactly as the deprecated broker provides `amqp`
|
||||
([ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md)).
|
||||
([ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md) (superseded by [ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md): AMQP is not a provision, and everything speaks to the `mesh-broker` seat)).
|
||||
|
||||
They are never the same name. A manifest saying `nats` could otherwise mean either the mesh's
|
||||
nervous system or a private queue, and the difference between those is the whole architecture.
|
||||
@@ -427,7 +459,7 @@ it as an ordinary module once the registry exists.
|
||||
|
||||
So there are exactly two things the normal path cannot make, both at genesis, both ending the
|
||||
moment the mesh can mint for itself: **the bus's own accounts** (§the bootstrap argument in
|
||||
[ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md) (superseding [ADR 0125](../../02-DECISIONS/0125-the-bus-is-the-only-broker.md)) — a provisioner is a module and
|
||||
[ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md) (superseded by [ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md): AMQP is not a provision, and everything speaks to the `mesh-broker` seat) (superseding [ADR 0125](../../02-DECISIONS/0125-the-bus-is-the-only-broker.md)) — a provisioner is a module and
|
||||
needs an account before it can run) and **the vault's own credential**. Any third exception is a
|
||||
design failure, and naming these two is what makes a third one visible.
|
||||
|
||||
@@ -439,6 +471,10 @@ it is the residue of a question the rest of §8 answers and the part a fingerpri
|
||||
**Whether a module may declare a seat it does not itself claim** — the contract as one thing, the
|
||||
implementation as another, which is how two competing implementations would ever exist.
|
||||
|
||||
**Whether a container should have a readiness notion.** Only an action carries `verify`, so a step that
|
||||
must run once a service *answers* — seeding through its own API — cannot be declared at all. Named here
|
||||
because the steps above make the gap obvious, not because they caused it.
|
||||
|
||||
**Whether `consumes` naming another module couples too tightly.** It is kept here deliberately —
|
||||
an event's provenance is its meaning — but a consumer of `billing.order.placed` does depend on
|
||||
billing existing under that name.
|
||||
@@ -447,6 +483,11 @@ billing existing under that name.
|
||||
|
||||
- **A manifest holds no subject.** A catalogue test: no manifest contains a string matching the
|
||||
subject grammar. The rule is worthless if it is followed by convention.
|
||||
- **A preparation is given what the module is given.** A composition test: what the preparation
|
||||
entrypoint receives equals what the module's own code receives, asserted rather than written twice —
|
||||
which is the drift a hand-written step invites, three times over in the catalogue today.
|
||||
- **A convergence that changed nothing says nothing.** Two identical reports, one emitted fact: what is
|
||||
guarded against is a fact per minute per machine, which is a stream nobody reads.
|
||||
- **Permissions are exactly the three namespaces.** A composition test per module: the derived
|
||||
permission set equals what its declaration implies, and a hand-written addition to it fails.
|
||||
- **A sender cannot read the queue it writes to.** A bed: a module declaring `uses` is refused
|
||||
|
||||
@@ -0,0 +1,137 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: []
|
||||
updated: 2026-09-28
|
||||
decisions:
|
||||
- 02-DECISIONS/0132-a-seat-carries-the-tools-its-holder-must-serve.md
|
||||
- 02-DECISIONS/0129-a-seat-carries-the-protocol-of-its-role.md
|
||||
- 02-DECISIONS/0095-the-control-plane-is-the-way-to-ask-a-module.md
|
||||
---
|
||||
|
||||
# 33 — The tools the mesh answers
|
||||
|
||||
**An agent can call the mesh's tools and cannot find out what they are.** Both halves were measured
|
||||
on the live mesh on 2026-09-28: a client holding an operator credential connected to the bus, asked a
|
||||
module for its repositories and got them; the same client's request for the tool list found nothing
|
||||
serving it. The transport works, the account model works, the adapter that speaks the agent protocol
|
||||
works. What is missing is the mesh being able to say what it can do.
|
||||
|
||||
This design is the answer to that question, and it has three families in it, because a tool belongs to
|
||||
whoever is accountable for answering it.
|
||||
|
||||
## 1. Three families, and why the split is not arbitrary
|
||||
|
||||
| Family | Addressed to | Where the definition lives | Example |
|
||||
|---|---|---|---|
|
||||
| A **role's** tools | the seat: `mesh.seat.<seat>.tool.<verb>` | the seat's protocol, in the mesh's records | ask *the forge* to list its repositories |
|
||||
| A **module's** tools | the module: `mesh.mod.<module>.tool.<name>` | that module's code | ask *this gitea* for `gitea_list_repos` |
|
||||
| The **mesh's** own verbs | the `mesh-controller` seat | the seat's protocol, as above | `status`, `push`, `build`, `assign` |
|
||||
|
||||
The split follows accountability. A role is something the mesh guarantees exactly one holder of, so
|
||||
what the role answers is the mesh's to define and a holder's to implement
|
||||
([ADR 0132](../../02-DECISIONS/0132-a-seat-carries-the-tools-its-holder-must-serve.md)). A module's
|
||||
own tools are nobody's business but the module's, and their definitions live where they are
|
||||
implemented, because a copy kept anywhere else drifts from the code that answers.
|
||||
|
||||
The mesh's own verbs are the third family only in where they come from, not in kind: the control plane
|
||||
holds a seat like anything else, and its tools are that seat's. This is what keeps them addressable
|
||||
while the control plane is being replaced, which is the moment they are most needed.
|
||||
|
||||
**Both names for one capability is deliberate and bounded to this.** A forge holding the `git` seat
|
||||
answers the role's `list_repos` and its own `gitea_list_repos`, because the same module may run
|
||||
without the seat — a second instance, kept for one purpose — and then only the second name is true.
|
||||
The caller chooses which question it is asking. Nothing else in the mesh gets two names.
|
||||
|
||||
## 2. What a seat's tool is
|
||||
|
||||
A verb, what it does, and the schema of its arguments and its answer. A name alone is not callable by
|
||||
something that has never seen the mesh before, which is the whole population this surface exists for.
|
||||
|
||||
The protocol a seat carries today is three lists of bare verbs
|
||||
([ADR 0129](../../02-DECISIONS/0129-a-seat-carries-the-protocol-of-its-role.md)), and it must widen to
|
||||
carry the rest. Two constraints on that widening:
|
||||
|
||||
- **It lives in the mesh's records, not in the control plane's binary.** Today a seat's protocol comes
|
||||
from compiled defaults, merged in as a row is read, because the seat rows never gained the columns.
|
||||
Discovery that reads a binary is discovery that disagrees with the mesh the moment the two are on
|
||||
different versions.
|
||||
- **The schema is stated in the form an agent protocol already uses**, so nothing translates between a
|
||||
seat's idea of an argument and the caller's. A translation layer would be a second definition of
|
||||
what a tool is.
|
||||
|
||||
## 3. Holding a seat means serving its tools
|
||||
|
||||
A module may not occupy a seat unless it serves every verb that seat declares. This joins the
|
||||
conditions of holding that already exist — providing what the seat delivers, being assigned at the
|
||||
seat's scope — and is refused the same way: at registration and at handover, naming the verbs that are
|
||||
missing rather than the fact that something is.
|
||||
|
||||
A module knows which seats it claims, so knowing which tools it must serve is not a discovery problem
|
||||
for the module: the seat says, the module implements, and anything beyond that is its own.
|
||||
|
||||
## 4. Addressing a node-scoped seat
|
||||
|
||||
A seat's subject is flat today — `mesh.seat.<seat>.<kind>.<verb>` — which is correct for a seat the
|
||||
mesh has one holder of and wrong for the six node-scoped seats, where one subject would reach every
|
||||
machine's holder and the holders' queue group would hand the call to whichever answered first. A
|
||||
node-scoped seat's tool therefore carries the node it is asked of. Nothing about a mesh-scoped seat
|
||||
changes.
|
||||
|
||||
## 5. Discovery
|
||||
|
||||
**What a role answers is a read.** The seats and their protocols are records, so the list is a query
|
||||
against the mesh's own store: no call to a module in the path, nothing that has to be running, and an
|
||||
answer that stays true while a holder is restarting or being replaced.
|
||||
|
||||
**What a module answers comes from the module.** Its definitions live in its code, so it is asked, and
|
||||
the answer is as available as the module is — which is the right coupling for a tool that only exists
|
||||
while that module does.
|
||||
|
||||
A caller therefore gets one list assembled from two sources, and the difference is visible in it: a
|
||||
role's tool names a seat, a module's names a module. An agent that wants to survive a holder being
|
||||
replaced binds to the first.
|
||||
|
||||
## 6. What serves this to an agent
|
||||
|
||||
A module the mesh assigns to the machine where the agent runs, holding a credential the mesh minted,
|
||||
with authority derived from what it may call — not a program started by hand with a credential printed
|
||||
to a terminal. The adapter itself already exists and is thin by design; what changes is that it stops
|
||||
being something a person carries and becomes something the mesh runs, on a node, like everything else.
|
||||
|
||||
An agent's authority can then be role-shaped: *the forge's tools*, rather than a list of
|
||||
module-specific names that changes the day the forge is replaced.
|
||||
|
||||
## 7. Versioning
|
||||
|
||||
A seat's tools are an interface and change like one. Additive within a version. A change that would
|
||||
break a caller takes the version token the subject already has room for, and the two versions run side
|
||||
by side until nothing is bound to the old one.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- **A holder missing a verb cannot take the seat.** One test per condition of holding, as the existing
|
||||
conditions have, and the live refusal names the verbs.
|
||||
- **A verb nobody declared is a subject nobody may use.** The bus grants are derived from the seat's
|
||||
protocol already, and the golden composition of the user list is what keeps that honest: a holder is
|
||||
granted exactly the seat's verbs, a user of the seat exactly the publish side.
|
||||
- **Discovery needs no running module.** The test for a role's tools reads records and asserts the
|
||||
answer equals what the seats declare — if it needed a module up, it would not be a read.
|
||||
- **Two nodes holding one node-scoped seat derive two addresses.** Checked by the same test as the rest
|
||||
of the subject table.
|
||||
|
||||
## What this does not settle
|
||||
|
||||
- Which verbs each seat should serve. That is a decision per seat, and the reason to do it slowly: a
|
||||
seat's tools bind every future holder.
|
||||
- Whether a module's own tool definitions should also be recorded when a build resolves its manifest.
|
||||
There is an argument for it — the mesh could then answer for a module that is down — and an argument
|
||||
against, which is that a recorded copy of a live definition is a copy that can be wrong.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0132](../../02-DECISIONS/0132-a-seat-carries-the-tools-its-holder-must-serve.md) — the decision this designs
|
||||
- [ADR 0129](../../02-DECISIONS/0129-a-seat-carries-the-protocol-of-its-role.md) — a seat carries the protocol of its role
|
||||
- [ADR 0095](../../02-DECISIONS/0095-the-control-plane-is-the-way-to-ask-a-module.md) — a tool call passes one process where an audit belongs
|
||||
- [`26-the-seats.md`](26-the-seats.md) — what a seat is, how it is held and handed over
|
||||
- [`25-the-bus-on-nats.md`](25-the-bus-on-nats.md) §7 — a person's account, their inbox, and the adapter
|
||||
@@ -40,7 +40,7 @@ document is written and this one's status becomes `implemented`.
|
||||
| [`28-building-the-bus.md`](28-building-the-bus.md) | **Proposed.** The five steps of the bus work in the order their dependencies allow, each ending at a bed — with the surface measured, so no step's size is a guess | [ADR 0116](../../02-DECISIONS/0116-the-bus-is-built-in-five-steps.md), [ADR 0106](../../02-DECISIONS/0106-the-bus-is-nats.md), [ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md) |
|
||||
| [`29-a-node-has-operator-accounts.md`](29-a-node-has-operator-accounts.md) | **Proposed.** The mesh models machines but not the humans on them: a node gains operator accounts, and a resource may live under a home owned by its account — what would own ~/.ssh, dotfiles and ~/.config when HAL retires | [ADR 0112](../../02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md), [ADR 0051](../../02-DECISIONS/0051-shared-data-is-the-operators.md) |
|
||||
|
||||
| [`32-what-a-module-declares.md`](32-what-a-module-declares.md) | **Proposed.** What a module declares and what the bus derives from it: three namespaces, subjects from local names, queues never declared, the five relationships, and the build-publish-deploy lifecycle on one bus | [ADR 0126](../../02-DECISIONS/0126-a-module-declares-its-own-seats.md), [ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md) (superseding [ADR 0125](../../02-DECISIONS/0125-the-bus-is-the-only-broker.md)), [ADR 0041](../../02-DECISIONS/0041-events-are-a-relationship.md) |
|
||||
| [`32-what-a-module-declares.md`](32-what-a-module-declares.md) | **Proposed.** What a module declares and what the bus derives from it: three namespaces, subjects from local names, queues never declared, the five relationships, and the build-publish-deploy lifecycle on one bus | [ADR 0126](../../02-DECISIONS/0126-a-module-declares-its-own-seats.md), [ADR 0127](../../02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md), superseded by [ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md) (superseding [ADR 0125](../../02-DECISIONS/0125-the-bus-is-the-only-broker.md)), [ADR 0041](../../02-DECISIONS/0041-events-are-a-relationship.md) |
|
||||
|
||||
## Not yet written
|
||||
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
---
|
||||
status: resolved
|
||||
opened: 2026-09-27
|
||||
located-in: [mesh-catalog modules/gitea, mesh-controller cmd/mesh-controller, mesh-controller internal/builder]
|
||||
fixed-by: mesh-catalog #124 — the forge module watches for merged pull requests and emits `pull.merged` with the merge commit and the clone address; mesh-controller #110/#111 — the control plane follows that event on the bus, marks every module built from that repository and branch as moved, and builds them bases first, stopping when a base fails; mesh-controller #113/#114 — a build records the bases it was handed and the graph is read from builds, without which "bases first" had no edges to order by.
|
||||
amended-design: 03-DESIGN/01-to-be/28-building-the-bus.md
|
||||
---
|
||||
|
||||
# 131 — Nothing tells the mesh a source moved, and it reports itself current anyway
|
||||
|
||||
## What was observed
|
||||
|
||||
Six changes were merged to the trunk of six repositories in one sitting. The build machine built
|
||||
nothing. Its last build, minutes before the first merge, was still the one it reported; no build was
|
||||
requested, refused or failed, because none was ever asked for.
|
||||
|
||||
Asked afterwards what was wrong, the mesh said:
|
||||
|
||||
> 4 machine(s), all doing what they were told, all heard from, running what the mesh would send them,
|
||||
> and every module current with its source
|
||||
|
||||
Every one of the six had moved. The last clause was false for all of them, and it is the clause a
|
||||
person reads to decide whether there is anything to do.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
**The mesh learns a source moved by being told, and there is no longer anything to tell it.** The
|
||||
command exists — a person names the module and the commit — and so does the question the overview
|
||||
answers. What is missing is whatever used to connect the two. One repository still carries a forge
|
||||
webhook aimed at a port; the rest carry none, and the port belongs to a different service than the
|
||||
one the arrangement implies. So the state is not "the trigger is broken" but "there is no trigger,
|
||||
and nothing says so".
|
||||
|
||||
**A wrong answer is worse here than no answer.** "Every module current with its source" is
|
||||
indistinguishable, to a reader, from a mesh that has genuinely caught up. The overview is built to be
|
||||
the thing you check instead of checking by hand, so a confident false negative removes the habit that
|
||||
would otherwise have caught it. Nothing in the mesh is at fault for being out of date — it is at
|
||||
fault for saying it is not.
|
||||
|
||||
**It is also why "current with its source" cannot be a stored fact.** The mesh compares what it built
|
||||
against what it was last told the source was, and calls that agreement. Two facts agreeing tells you
|
||||
nothing when both come from the same place.
|
||||
|
||||
## The intended shape, which is decided and not built
|
||||
|
||||
The forge emits what happened to it — a pull request merged — and the build machine reacts by
|
||||
building what that commit affects. That keeps the forge ignorant of the build system and the build
|
||||
machine ignorant of the forge's internals, which is the same argument
|
||||
[ADR 0126](../../02-DECISIONS/0126-a-module-declares-its-own-seats.md) makes for addressing an event
|
||||
to its emitter: the merge is a fact about the forge, and what should be rebuilt because of it is not
|
||||
the forge's business to know.
|
||||
|
||||
The forge's module already declares the event. The build machine declares that it consumes nothing.
|
||||
|
||||
## What the trigger cannot be
|
||||
|
||||
**Not one build per changed module.** The modules form a graph: several are built from one
|
||||
repository, and some are the base another is compiled on — a runtime image, a compiler base, a
|
||||
repository whose context a second module builds from. Firing a build for each changed module
|
||||
independently would start work that cannot succeed yet and produce a failure per dependent, for one
|
||||
cause.
|
||||
|
||||
Observed while catching the mesh up by hand on 2026-09-27: a compiler base had to move before
|
||||
anything compiled against it could build, and when it failed, the right behaviour was for its
|
||||
dependents to wait rather than each fail the same way. Fifteen modules shared the cause. A trigger
|
||||
that reports it fifteen times has buried it.
|
||||
|
||||
So whatever reacts to the forge's event resolves what changed into an order, builds the bases first,
|
||||
and holds a dependent while its base is unbuilt or failed. That is a larger thing than "rebuild what
|
||||
the commit touched", and knowing it now is cheaper than discovering it from fifteen identical
|
||||
failures.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Is "the source moved" still a thing a person can assert by hand once the event path exists, or does
|
||||
the hand-operated form become the thing that made this failure possible?
|
||||
- Which commit does the build machine act on — the merge, or each commit it brought — and what does
|
||||
it do when several arrive for one module at once?
|
||||
- How does the overview stop being able to lie? Comparing what was built against what was recorded
|
||||
will always agree. Whether the trunk has moved is a question only the forge can answer, so either
|
||||
the overview asks it, or it stops claiming to know.
|
||||
- Does this want to be the same mechanism as the build request on the bus
|
||||
([ADR 0129](../../02-DECISIONS/0129-a-seat-carries-the-protocol-of-its-role.md)), or does it sit in
|
||||
front of it?
|
||||
|
||||
## What was done (2026-09-28)
|
||||
|
||||
The shape above was built as described: the forge's module emits the merge, the control plane
|
||||
consumes it, and nothing on either side knows the other's internals. The build is asked for the
|
||||
merge commit, not each commit the merge brought — the trunk moved once, to one place. Several merges
|
||||
for one module arriving in a row are followed in turn, each moving the recorded source to its own
|
||||
commit, so the last one to arrive is the one the mesh ends up built from.
|
||||
|
||||
The hand-operated form stays. `module moved` is how a source is recorded without a forge — a module
|
||||
built from a repository elsewhere, or a mesh whose forge module is down — and it is the same act the
|
||||
event performs, so the two cannot disagree about what "moved" means.
|
||||
|
||||
**Bases first needed edges, and there were none.** The order this report asked for was written and
|
||||
walked a graph that no build had ever recorded: a recipe reads its base from a build argument, so the
|
||||
digest was never in the file the builder read edges from. A build now reports what it was handed, the
|
||||
control plane records it by artifact path, and the order is read from each module's newest build.
|
||||
|
||||
**What still can lie.** The overview compares what was built against where it was last told the
|
||||
source is; the forge's event is now what moves that mark, so it is right for as long as the forge
|
||||
module was listening. A merge made while that module was down is a merge the mesh does not know of
|
||||
until the module next polls — it announces what merged since it last looked, so the gap closes when
|
||||
it comes back, and not before. The overview does not say so.
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
status: resolved
|
||||
opened: 2026-09-28
|
||||
located-in: [mesh-controller cmd/mesh-controller]
|
||||
fixed-by: mesh-controller — `module add` takes `--path` and `--self`, so a module handed over by hand records the whole location it came from; a record naming a repository and no directory says so in the reply; the rule is one function with a test beside it. The nine records already wrong were corrected by rebuilding each with its real directory, which is the same act through the same door.
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 132 — A module can be recorded without the directory it lives in
|
||||
|
||||
## What was observed
|
||||
|
||||
Nine modules on one mesh could not be rebuilt. Each attempt failed the same way:
|
||||
|
||||
> has no module.json at its root, so there is nothing saying what it is
|
||||
|
||||
All nine were recorded as coming from a repository that holds many modules, each in its own
|
||||
directory — and each record named the repository and no directory. So every build cloned the
|
||||
repository and looked for a manifest where there has never been one.
|
||||
|
||||
The failure only surfaced when something asked for all of them at once. Before that, the overview
|
||||
said every module was current with its source, because what it compares is what was built against
|
||||
what the mesh was last told the source has, and neither half knows whether the source can be found
|
||||
at all.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
**A module is a repository and a directory inside it** ([ADR 0069](../../02-DECISIONS/0069-a-module-is-a-repository-and-a-path.md)),
|
||||
and one of the two doors into the catalogue could record only the first half. A build records the
|
||||
directory it was given, so a module that arrived by being built is always whole; a module handed over
|
||||
by hand had no way to say where it lived, and the flag to say it did not exist. The rule was decided
|
||||
and enforced on one path out of two.
|
||||
|
||||
**Half a location reads exactly like a whole one.** Nothing in the record is empty in a way a person
|
||||
would notice: the repository is there, the branch is there, the commit is there. The mesh only finds
|
||||
out at the moment it needs the manifest, which is the moment it is trying to rebuild — and the module
|
||||
stays on whatever it last built, indefinitely, with nothing saying why.
|
||||
|
||||
**It is the same shape as [131](../131-nothing-tells-the-mesh-a-source-moved/00-report.md).** A
|
||||
comparison between two facts the mesh holds about itself will agree with itself. Whether the source
|
||||
can be found is a question only an attempt to read it answers, and the answer had nowhere to go.
|
||||
|
||||
## What was done
|
||||
|
||||
`module add` takes the directory and which forge holds the repository, so a hand-registered module
|
||||
records the same whole location a built one does. What a record must say to be worth anything is one
|
||||
function with a test beside it, rather than a paragraph in a help string: provenance together or not
|
||||
at all, a directory needs a repository to be inside, a path on the mesh's own forge is not an address.
|
||||
And a record that names a repository but no directory says so when it is made — not refused, because a
|
||||
module really at a repository's root is ordinary, but said, because the person adding it is the one
|
||||
who knows which it is.
|
||||
|
||||
The nine wrong records were corrected by building each with its real directory, which re-records it.
|
||||
No row was written by hand.
|
||||
|
||||
## What is still true
|
||||
|
||||
A directory that does not exist in the repository cannot be refused when the module is added: the
|
||||
control plane does not clone, and inventing a check there would mean it did. The first build says so
|
||||
plainly, which is one build rather than nine, and the record it leaves behind is right from then on.
|
||||
+78
@@ -0,0 +1,78 @@
|
||||
---
|
||||
status: resolved
|
||||
opened: 2026-09-28
|
||||
located-in: [mesh-controller module.json]
|
||||
fixed-by: mesh-controller — the control plane's module declares a run-once `migrate` step before its server, which is the shape ADR 0052 prescribes for exactly this. A step's record of having run is the digest of its declaration and the image is part of that digest, so a new build of the control plane re-runs it; and because a run-once step gates what the declaration places after it, a migration that fails stops the new server from starting at all rather than letting it run against a schema it does not have.
|
||||
amended-design: 03-DESIGN/01-to-be/32-what-a-module-declares.md
|
||||
---
|
||||
|
||||
# 133 — The control plane's schema is migrated at birth and never again
|
||||
|
||||
## What was observed
|
||||
|
||||
On 2026-09-28 at 08:17 the control plane was replaced, by the mesh's own upgrade path, with a build
|
||||
whose code writes a column that a migration **in that same build** creates. Nothing ran the migration.
|
||||
|
||||
For the next three quarters of an hour the mesh built things and recorded none of them. Every build
|
||||
answered:
|
||||
|
||||
> ERROR: column "built_contexts" of relation "build" does not exist (SQLSTATE 42703)
|
||||
|
||||
and that sentence went only to whoever happened to be waiting on a build's reply. The overview kept
|
||||
saying the mesh was fine. The builds themselves worked — images were built and published — so the
|
||||
registry filled up with artifacts the mesh has no record of, and the graph stopped learning without
|
||||
anything saying so.
|
||||
|
||||
The schema was created once, at genesis, by an action in the foundation bundle that runs the same
|
||||
binary's `migrate`. Nothing runs it again. The mesh has updated its own control plane many times since
|
||||
that bundle, and every one of those updates carried whatever migrations the new build brought and
|
||||
applied none of them. This is the first time a build needed one.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
**The schema and the code that needs it ship as one artifact and are applied by two mechanisms, only
|
||||
one of which is automatic.** A module's version is atomic everywhere else in the mesh — the manifest,
|
||||
the image and what the machine runs move together. Its schema did not, so "the mesh updates itself on
|
||||
a push" was true of the code and false of what the code needs.
|
||||
|
||||
**The failure is quiet exactly where quiet is worst.** A build that cannot be recorded is a build that
|
||||
happened and left no trace, which is the fault [issue 050](../050-the-catalogue-knows-nothing-built-before-it/00-report.md)
|
||||
and [issue 131](../131-nothing-tells-the-mesh-a-source-moved/00-report.md) are both about. The mesh
|
||||
has three mechanisms for noticing a module is behind its source and none for noticing that what it
|
||||
recorded was refused.
|
||||
|
||||
**The shape was already decided, and the control plane was the one module that did not use it.**
|
||||
[ADR 0052](../../02-DECISIONS/0052-a-step-that-runs-once-before-a-container.md) says a run-once container is a
|
||||
step the host runs to completion before whatever the declaration places after it, and names migrating
|
||||
a schema as the case it exists for. The genesis code's own comment says a manifest may name its image
|
||||
in more than one resource — "a migrate step beside the server". The control plane's manifest had no
|
||||
such step; it went straight from a state directory to the server.
|
||||
|
||||
## What is still true
|
||||
|
||||
**Additive migrations are load-bearing, not a style preference.** The step runs before the *new*
|
||||
server starts, which means the old binary briefly runs against the new schema. A migration that
|
||||
removes or renames something would break the running control plane in the window between the two.
|
||||
|
||||
**A hand-written step is one the next module forgets**, which is why this fix is not where the matter
|
||||
ends: [ADR 0135](../../02-DECISIONS/0135-a-module-version-prepares-its-state-before-it-runs.md) makes it
|
||||
derived and puts it where an author works: a module version declares an entrypoint that prepares its
|
||||
state, and the mesh composes the gated work from it, so the control plane stops being the only module
|
||||
that had to remember. That record also settles the level question HAL answered with stages — a consumer
|
||||
is a module on a machine, so the scope of preparation is the scope of the state — and
|
||||
[ADR 0134](../../02-DECISIONS/0134-the-mesh-says-what-it-applied.md) answers the second open question
|
||||
below: what a machine applied, and what it refused, become facts on the bus rather than a line in a log.
|
||||
|
||||
**The mesh now has two shapes for one problem.** The catalogue module migrates its own schema in its
|
||||
own code when it starts; the control plane migrates in a step the host gates on. Both work and the
|
||||
reasons differ — a module that owns its store entirely can do it at start, while a step is visible in
|
||||
the declaration and refuses to let a broken upgrade serve. Which one the mesh should standardise on is
|
||||
a decision, not a fix, and it is not made here.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Should a module be refusable at registration when it ships migrations and declares no step and no
|
||||
other way to apply them? The mesh can see both halves.
|
||||
- Should a record the store refuses reach the overview? Today the only reader of that failure is
|
||||
whoever asked for the thing that failed, and for an event arriving on the bus there is no such
|
||||
person.
|
||||
@@ -0,0 +1,66 @@
|
||||
---
|
||||
status: open
|
||||
opened: 2026-09-28
|
||||
located-in: [mesh-catalog, mesh-controller internal/catalogue]
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 134 — A definition may still name the mesh, and the check that would say so does not exist
|
||||
|
||||
## What was observed
|
||||
|
||||
[ADR 0112](../../02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md) says a module
|
||||
definition names no node, no mesh and no host path, and states how that is checked:
|
||||
|
||||
> A catalogue test finds no domain name in any definition value.
|
||||
|
||||
There is no such test. Run by hand on 2026-09-28, across the 72 manifests in the catalogue, the
|
||||
question it asks has 15 answers. They are not all the same kind of thing, and the difference matters
|
||||
more than the count:
|
||||
|
||||
**Values the mesh acts on** — seven:
|
||||
|
||||
| module | where | what it names |
|
||||
|---|---|---|
|
||||
| keycloak | `env.KC_HOSTNAME` | this installation's public name for itself |
|
||||
| minio | `env.MINIO_BROWSER_REDIRECT_URL` | the same, for its console |
|
||||
| invoicing | a resource's `image` | a named registry rather than the mesh's artifact store |
|
||||
| builder | `build.artifacts[].context.repository` | the forge, by URL |
|
||||
| route-proxy | `build.artifacts[].context.repository` | the forge, by URL |
|
||||
| route-adapter | a resource's `content` | a proxy's dynamic configuration |
|
||||
| novox.be | `module` | the module is named after the domain it serves |
|
||||
|
||||
**Prose** — eight, in `listens[].why`: de-spiegel, mailu, n8n, only-office, photos, photos-eef,
|
||||
photos-filip, portainer. Each explains what a port is for and mentions the public name it is reached
|
||||
by. Nothing reads these; a check written as a string search would report them, and reporting them as
|
||||
violations of the same rule would be wrong.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
**An unenforced rule is indistinguishable from a wrong one, and costs more, because people believe
|
||||
it.** The record says the mesh is name-agnostic, four design documents rest on that, and a reader
|
||||
checking whether it holds finds that it does not — in the places that matter most. The two forge URLs
|
||||
are what a build reaches into for its source; the two hostnames are what a service tells a browser
|
||||
about itself.
|
||||
|
||||
**It is the difference between a mesh and this mesh.** A definition carrying `novox.be` is a
|
||||
definition that can only be installed here. The whole point of the rule is that the same catalogue
|
||||
raises a different mesh with a different name, and today seven modules would need editing to do it.
|
||||
|
||||
**And the shape of the fix is not the same for each.** A public name is an operator's choice about an
|
||||
assignment, which ADR 0112 already provides for; a forge URL should be a path on the git seat
|
||||
([ADR 0111](../../02-DECISIONS/0111-a-build-source-is-on-the-git-seat-or-external.md)); an image from a named
|
||||
registry is a question about the artifact store, not about naming. Counting them together would hide
|
||||
that.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Does a domain in a `why` string break the rule? It is documentation the mesh never reads, and a
|
||||
check that cannot tell the two apart will either pass things it should catch or fail things nobody
|
||||
should change.
|
||||
- Where does a service's public name live, concretely — a setting on the assignment, or a fact the
|
||||
mesh composes from the node's domain? ADR 0112 says a requirement the mesh resolves; the two
|
||||
hostnames above are the first real cases.
|
||||
- Should a build context name a repository on the git seat rather than by URL, and if so, what does
|
||||
that mean for a context in *another* mesh's forge?
|
||||
@@ -0,0 +1,70 @@
|
||||
---
|
||||
status: resolved
|
||||
opened: 2026-09-28
|
||||
located-in: [mesh-host internal/apply]
|
||||
fixed-by: mesh-host — a container's mesh names are part of the spec digest the host compares, sorted so the digest does not move for a reordering. A container whose names moved is now recreated like a container whose image moved, and the test fails against the previous behaviour.
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 135 — A container's mesh names are not compared, so a moved address is never noticed
|
||||
|
||||
## What was observed
|
||||
|
||||
One container on this mesh had been restarting every thirty seconds for five days — 2286 times — and
|
||||
the mesh reported the machine as doing what it was told.
|
||||
|
||||
Its logs said its database connected and then a query timed out. The database was reachable: the same
|
||||
query from the same network, with the same credential, answered in milliseconds. What differed was the
|
||||
name. Inside that container, `novox.internal` resolved to `10.42.0.1`; in every other container on the
|
||||
machine it resolved to `10.10.0.1`. The mesh's overlay range had moved, and this container still held
|
||||
the old one:
|
||||
|
||||
```
|
||||
umami created 2026-09-23 novox.internal:10.42.0.1
|
||||
mesh-catalog created today novox.internal:10.10.0.1
|
||||
```
|
||||
|
||||
A container resolves other machines and public names through the entries the mesh gives it when it is
|
||||
created, and nothing re-reads them afterwards. The host compares a container against what was declared
|
||||
by a digest of its spec — image, name, environment, ports, volumes, arguments, resolver, address, and
|
||||
what it reads — and **the mesh's names were not in it**. So this container matched what was declared,
|
||||
was left alone, and kept an address that had not existed for five days.
|
||||
|
||||
Forty-eight other containers had current names. Not because anything corrected them: each had been
|
||||
recreated for some other reason — a new image, a changed file — and picked up the current roster on the
|
||||
way. This one's image is an upstream release that had not moved, and nothing else about it changed, so
|
||||
nothing ever recreated it.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
**It is the exact fault [issue 045](../045-a-container-keeps-the-values-it-started-with/00-report.md)
|
||||
named, in the one field that was left out.** That issue is why the digest carries what a container
|
||||
reads: "a container whose configuration had since been rewritten compared equal and was left alone —
|
||||
running values the machine no longer holds, while every check reported success." The same sentence
|
||||
describes this, with *names* in place of *files*.
|
||||
|
||||
**The failure is invisible in exactly the way that matters.** The container runs, so the machine
|
||||
reports it applied. It restarts, but a restarting container is a normal sight during an upgrade. The
|
||||
only account of the fault is inside the container's own log, in the words of the application rather
|
||||
than of the mesh — and what it says is that a query timed out, which points at the database.
|
||||
|
||||
**And it is most likely to bite what changes least.** Every container that is rebuilt often repairs
|
||||
itself by accident. The victim is the module whose image is stable — which is to say, the module that
|
||||
was working fine.
|
||||
|
||||
## What was done
|
||||
|
||||
The mesh's names are part of the digest, sorted so the digest does not move for a reordering nobody
|
||||
made. A container whose names moved is now recreated exactly as one whose image moved.
|
||||
|
||||
The first apply after this recreates every container that carries mesh names — one restart each,
|
||||
already the price the mesh pays for any image update — because their recorded digests predate the
|
||||
field.
|
||||
|
||||
## What is still true
|
||||
|
||||
The mesh gives a container its names at creation and has no way to change them in place. That is the
|
||||
container runtime's shape, not a choice; the answer is to recreate, which is what this does. A module
|
||||
that would rather re-read a roster from a file can already ask for one as a fact
|
||||
([ADR 0120](../../02-DECISIONS/0120-a-roster-fact-carries-its-format-as-a-template.md)) and restart on
|
||||
it.
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
status: resolved
|
||||
opened: 2026-09-28
|
||||
located-in: [mesh-catalog modules/fail2ban]
|
||||
fixed-by: mesh-catalog — the intrusion-prevention module bans through an action it ships itself, already in use on every machine, instead of naming a firewall front-end two of them do not have. The instance is closed; the class in "What is still true" is not.
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 136 — A module may name a program the machine does not have, and everything reports success
|
||||
|
||||
## What was observed
|
||||
|
||||
Two machines were given the intrusion-prevention module on 2026-09-28. Both refused to start it:
|
||||
|
||||
```
|
||||
ERROR Failed during configuration: Have not found any log file for 'recidive' jail.
|
||||
ERROR Async configuration of server failed
|
||||
fail2ban.service: Main process exited, code=exited, status=255/EXCEPTION
|
||||
```
|
||||
|
||||
The jail that bans whoever keeps coming back reads the service's *own* log, and the service checks
|
||||
every jail's log file while it configures itself — before it has created that log. The module
|
||||
declared the jail and shipped the rotation for that log, and never declared the log. On the two
|
||||
machines where it had run for years the file was simply there, so nothing had ever noticed.
|
||||
|
||||
That failure was loud. Fixing it uncovered a second one in the same module that is not.
|
||||
|
||||
The module's defaults named `ufw` as the way to ban an address. Two of these four machines have no
|
||||
`ufw` — they filter with nftables — and nothing checks that until an address is banned. Asked to ban
|
||||
a documentation address on such a machine, the service accepted the instruction, counted it, ran the
|
||||
command, and wrote this to a log nobody reads:
|
||||
|
||||
```
|
||||
ERROR ... -- stderr: '/bin/sh: line 5: ufw: command not found'
|
||||
ERROR ... -- returned 127
|
||||
ERROR Failed to execute ban jail 'sshd' action 'ufw' ... Error banning 192.0.2.99
|
||||
```
|
||||
|
||||
No rule existed afterwards. Throughout, the unit was `active`, the module was applied, and the
|
||||
machine's report said so. **A machine had been added to the mesh's intrusion prevention, reported as
|
||||
protected, and was banning nobody.**
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
**The two faults are the same mistake with opposite symptoms.** Both are the module assuming
|
||||
something about the machine — a file that happens to exist, a program that happens to be installed.
|
||||
One stopped the service, which anybody notices. The other left it running and empty, which nobody
|
||||
does. A mesh that only catches the loud one is a mesh whose coverage is unknown.
|
||||
|
||||
**"The unit is running" was taken for "the module is doing its job".** That is the only health a
|
||||
service resource has. It is the right answer for most modules and it is silent for any module whose
|
||||
work happens later, on an event — a ban, a renewal, a backup, a notification. The report cannot
|
||||
distinguish "protecting this machine" from "installed and inert".
|
||||
|
||||
**And it is exactly the naming rule, one level down.**
|
||||
[ADR 0112](../../02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md) says a
|
||||
definition names no node, no mesh and no host path, because the same definition has to raise a
|
||||
different mesh. `ufw` is not a node name, but it is the same class of assumption: a value the module
|
||||
cannot know, true on some machines and false on others, written as though it were a constant. The
|
||||
module already knew how to do better a few lines away — the mesh's own address range is named there
|
||||
as something the machine fills in.
|
||||
|
||||
## What was done
|
||||
|
||||
The module declares the log its own jail reads, created once and never touched again, since what
|
||||
grows in it is the service's and the rotation the module already ships is what keeps it small. And
|
||||
it bans through the action it ships itself, which every machine here can run, which was already in
|
||||
use by the other jail on all four, and which covers a container's published port as well as the
|
||||
host's own.
|
||||
|
||||
All four machines now run it, with both jails, and a ban lands on each — verified by banning and
|
||||
unbanning a documentation address on every one.
|
||||
|
||||
## What is still true
|
||||
|
||||
**Nothing would have caught either fault before it shipped.** The control plane reads a manifest, not
|
||||
a machine; `ufw` and `/var/log/…` are strings in a file it has no way to evaluate. The host could in
|
||||
principle be asked whether a declared program exists, but no resource says "this file names a command
|
||||
that must be there", so there is nothing to check.
|
||||
|
||||
**Two machines' bans from before this are stale rules in the old front-end**, which the service no
|
||||
longer knows about and will never lift. They reject two addresses for ever. Harmless, and a reminder
|
||||
that changing how a module enforces something leaves what it already enforced behind.
|
||||
|
||||
## Open questions
|
||||
|
||||
- What does a service resource's health mean for a module whose work is event-driven? A unit being
|
||||
active is the weakest claim available, and four of this mesh's modules are of that kind.
|
||||
- Should a declaration be able to say that a resource depends on a program, so the machine can refuse
|
||||
what it cannot carry out rather than reporting success?
|
||||
- Where should the packet filter a module bans through come from — the module's own choice, as now,
|
||||
or the seat that owns the machine's filtering?
|
||||
Reference in New Issue
Block a user