The broker precedes the control plane, and it is written in Go
Two things found by trying to build tier 2. The substrate design asked whether the message broker has to be running before the control plane, and framed it as depending on whether the control plane's own parts talk to each other over it. They do not -- it is one process -- so under that framing the broker stays out of the bundle. The framing cannot answer the question. What decides it is how the control plane reaches a node, and the answer was already decided: only ever over the link, and the link is the broker. So provisioning the broker would require the broker. The first node does not escape this by being local, because it enrols the ordinary way, by dialling the broker at the address in its token -- which was deliberate, and worth keeping. The bundle is two images now. The record says what that costs, including a certificate the broker needs at a moment when there is no mesh to issue one. The language had never been decided for tier 2. Go, for the same reason the host is: the bundle pins this image by digest and runs it where nothing can check it, so the image should hold the program and nothing else. Also corrects something already built: the bootstrap created one database and called it 'mesh'. ADR 0008 grants a context only what it exclusively owns and ADR 0006 says the mesh database names a thing that will not exist. One database per context, so one today, called inventory.
This commit is contained in:
@@ -0,0 +1,140 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: proposed
|
||||
date: 2026-08-29
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0006-the-substrate-and-the-control-plane.md
|
||||
---
|
||||
|
||||
# 24. Running the control plane
|
||||
|
||||
[ADR 0006](0006-the-substrate-and-the-control-plane.md) settles what the control plane *is* — seven
|
||||
contexts, one deployable, one node runs it. This settles what it takes to actually run one, which
|
||||
is the half you discover by trying to build it.
|
||||
|
||||
Two decisions. Neither is large; both were blocking, and one of them had been asked in the design
|
||||
in a form that could not answer it.
|
||||
|
||||
## It is written in Go
|
||||
|
||||
The same language as the host, so tiers 0 and 2 are one language and not two.
|
||||
|
||||
The reason that decides it is not familiarity. **The control plane's image is pinned by digest in
|
||||
the bundle the host carries** ([ADR 0006](0006-the-substrate-and-the-control-plane.md)), which
|
||||
means it is fetched and run on a machine where no mesh exists yet — nothing to check it against,
|
||||
nothing watching, and a person expected to have read the bundle and believed it. A statically
|
||||
linked binary makes that image the program and nothing else: no interpreter, no package tree, no
|
||||
transitive dependency that arrived because something needed a date library.
|
||||
|
||||
Everything under that line is a thing somebody would have to audit, on the one image the whole
|
||||
mesh is raised from.
|
||||
|
||||
A second reason, smaller and still real: the control plane runs a reconcile loop of its own
|
||||
([ADR 0010](0010-delivery.md) — artifacts against source, the way the host reconciles machine state
|
||||
against declarations). Two loops with the same shape are cheaper to hold in one head when they are
|
||||
also the same language.
|
||||
|
||||
### The option rejected, and why the obvious argument for it does not hold
|
||||
|
||||
**TypeScript**, matching the lab and the surfaces that will speak to this. The argument is that
|
||||
tier 3 is web and CLI, so tier 3 is TypeScript, so a TypeScript control plane shares its types with
|
||||
its callers directly rather than through a generated contract.
|
||||
|
||||
That argument is true and does not reach far enough. `mesh-sdk` is *contracts shared across tiers*
|
||||
and **tier 0 is Go** — so the contracts have to survive a language boundary no matter what tier 2
|
||||
is written in. The choice is not between sharing types and generating them; it is between
|
||||
generating them for one consumer or for two. That is a much smaller difference than it first looks,
|
||||
and it does not outweigh what is in the image.
|
||||
|
||||
**What this costs, stated plainly:** the control plane cannot import from the lab or from anything
|
||||
that exists today, and a person moving between tier 2 and tier 3 changes language. Neither is
|
||||
recovered later — a language is the most expensive thing in this record to reverse.
|
||||
|
||||
## The message broker is in the bundle
|
||||
|
||||
**LavinMQ is raised from the bundle, alongside PostgreSQL, before the control plane starts.**
|
||||
|
||||
[ADR 0006](0006-the-substrate-and-the-control-plane.md) left this not established, and
|
||||
[`07-the-substrate.md`](../03-DESIGN/01-to-be/07-the-substrate.md) recorded the open question as
|
||||
turning on *whether the control plane's own contexts talk to each other over the bus*.
|
||||
|
||||
**They do not** — they are one process and dispatch internally
|
||||
([ADR 0006](0006-the-substrate-and-the-control-plane.md)). Under the question as asked, that answer
|
||||
means the broker is provisioned like anything else, and the bundle stays at one image.
|
||||
|
||||
**The question was asked on the wrong axis.** What decides it is not how the contexts reach each
|
||||
other. It is how the control plane reaches a *node* — and the answer, already decided, is that it
|
||||
never reaches one except over the link, and the link is AMQP
|
||||
([ADR 0002](0002-nodes-communicate-over-a-broker.md),
|
||||
[ADR 0004](0004-a-node-and-how-it-joins.md)).
|
||||
|
||||
The first node then closes a circle that has no way out:
|
||||
|
||||
```
|
||||
the bundle raises the control plane
|
||||
the control plane provisions the broker ← by telling a host to run it
|
||||
telling a host happens over the link
|
||||
the link is the broker
|
||||
```
|
||||
|
||||
And it is not avoided by the first node being local. `enrol` **dials the broker at the address in
|
||||
the token** ([`09-the-node-lifecycle.md`](../03-DESIGN/01-to-be/09-the-node-lifecycle.md)), which is
|
||||
step 3 of the first node's own path. A machine that raised the mesh still joins it the ordinary
|
||||
way, and that was deliberate — *its specialness lasted two commands*. Making the first node join by
|
||||
some other route would buy the smaller bundle by giving up the property the design was built to
|
||||
have.
|
||||
|
||||
So the broker precedes the control plane, and it precedes it for the same reason PostgreSQL does:
|
||||
**the control plane cannot grant itself the thing it would need in order to grant it.**
|
||||
|
||||
### What this costs
|
||||
|
||||
**The bundle is two images rather than one**, against
|
||||
[ADR 0006](0006-the-substrate-and-the-control-plane.md)'s wish to keep it at roughly one so that a
|
||||
person can read it. Two is still readable. Four would not have been, which is why the object store
|
||||
and the registry stay out — nothing needs them to reach a node.
|
||||
|
||||
**And it grows three things that are not an image**, each an action the bundle declares and the
|
||||
host runs, the way the database already is:
|
||||
|
||||
- a virtual host, because the control plane cannot grant itself one
|
||||
([ADR 0006](0006-the-substrate-and-the-control-plane.md));
|
||||
- a credential on it for the control plane;
|
||||
- **a certificate, and this is the awkward one.** A token pins the fingerprint the host must expect
|
||||
*before it sends anything*
|
||||
([`09-the-node-lifecycle.md`](../03-DESIGN/01-to-be/09-the-node-lifecycle.md)), so the broker
|
||||
needs a certificate at bootstrap, when there is no mesh to issue one and no public name to get
|
||||
one for. Self-signed and pinned is the shape that fits, and how it is later replaced by the
|
||||
connectivity design's certificates is **not decided here**.
|
||||
|
||||
## What follows from ADR 0008, and was being built wrong
|
||||
|
||||
Not a decision — a correction, recorded because the mistake was already in a file that runs.
|
||||
|
||||
[ADR 0008](0008-a-context-owns-its-store.md) grants a context only what it **exclusively owns**, and
|
||||
rejects a schema per consumer inside a shared database on the grounds that *a boundary that is
|
||||
merely inconvenient to cross gets crossed*.
|
||||
[ADR 0006](0006-the-substrate-and-the-control-plane.md) says there is no single mesh database and
|
||||
that *the mesh database* names a thing that will not exist.
|
||||
|
||||
The bootstrap created one database and called it `mesh`.
|
||||
|
||||
**One database per context, named for the context.** Contexts that do not exist yet do not get one,
|
||||
so the bundle today creates exactly one, called `inventory`. A separate database rather than a
|
||||
separate schema is the point: in PostgreSQL a cross-schema join is a qualified name away, and a
|
||||
cross-database join needs a foreign data wrapper somebody has to install and explain.
|
||||
|
||||
**How a context added later gets its database is not answered here**, and it is a real question —
|
||||
by then there is a control plane, but a control plane holding a credential that can create
|
||||
databases is holding rather more than the thing it exclusively owns.
|
||||
|
||||
## Consequences
|
||||
|
||||
- `novox/mesh-control` exists, in Go, and tier 2 stops being the tier where nothing is built.
|
||||
- The bundle carries two images and four actions, and the substrate bootstrap grows a step.
|
||||
- Nothing in the first node's path is special-cased. Enrolment is walked on node one, as designed.
|
||||
- The certificate at bootstrap is named as unfinished rather than assumed. It is the first thing
|
||||
that will be wanted when the link is built, and it has no answer yet.
|
||||
- The language cannot be revisited cheaply. It is the one line in this record that is close to
|
||||
irreversible.
|
||||
@@ -92,6 +92,7 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0006** — [The substrate and the control plane](0006-the-substrate-and-the-control-plane.md)
|
||||
- **0007** — [Connectivity](0007-connectivity.md)
|
||||
- **0008** — [A context owns its store, exclusively](0008-a-context-owns-its-store.md)
|
||||
- **0024** — [Running the control plane](0024-running-the-control-plane.md) *(proposed)*
|
||||
|
||||
### What runs on them, and how it gets there
|
||||
|
||||
|
||||
@@ -4,14 +4,12 @@ status: designed
|
||||
code: []
|
||||
updated: 2026-08-27
|
||||
decisions:
|
||||
- 02-DECISIONS/0019-how-this-repository-works.md
|
||||
- 02-DECISIONS/0005-the-node-host.md
|
||||
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
|
||||
- 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
|
||||
- 02-DECISIONS/0005-the-node-host.md
|
||||
- 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
|
||||
- 02-DECISIONS/0007-connectivity.md
|
||||
- 02-DECISIONS/0005-the-node-host.md
|
||||
- 02-DECISIONS/0008-a-context-owns-its-store.md
|
||||
- 02-DECISIONS/0019-how-this-repository-works.md
|
||||
---
|
||||
|
||||
# The substrate
|
||||
@@ -94,15 +92,16 @@ Being substrate and being in the bundle are two different questions:
|
||||
| | is it substrate? | must it precede the control plane? |
|
||||
|---|---|---|
|
||||
| PostgreSQL | yes — the control plane's own state lives in it | **yes** — there is nowhere to put that state otherwise |
|
||||
| LavinMQ | yes — it cannot grant itself a virtual host | **not established** — see below |
|
||||
| LavinMQ | yes — it cannot grant itself a virtual host | **yes** — the control plane reaches a node only over the link, and the link is the broker ([ADR 0024](../../02-DECISIONS/0024-running-the-control-plane.md)) |
|
||||
| MinIO | yes — it cannot grant itself a bucket | no — nothing is delivered before the mesh exists |
|
||||
| the OCI registry | yes — it cannot grant itself a repository | no — the first node fetches upstream ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) |
|
||||
|
||||
The three on the bottom rows are **substrate by role and ordinary by delivery**: by the time
|
||||
they are wanted there is a control plane, and it provisions them the way it provisions anything.
|
||||
That keeps the bundle to roughly one image rather than four, which is what makes it small enough
|
||||
for the review [ADR 0005](../../02-DECISIONS/0005-the-node-host.md)
|
||||
requires.
|
||||
The two on the bottom rows are **substrate by role and ordinary by delivery**: by the time they
|
||||
are wanted there is a control plane, and it provisions them the way it provisions anything.
|
||||
That keeps the bundle to two images rather than four, which is what makes it small enough for the
|
||||
review [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) requires. It was one until
|
||||
[ADR 0024](../../02-DECISIONS/0024-running-the-control-plane.md) established that the broker has
|
||||
to precede the control plane.
|
||||
|
||||
**Why pinned:** the bundle is applied when no mesh exists, so nothing can resolve a version, ask
|
||||
a registry, or check a constraint. What the host carries must already be exact.
|
||||
@@ -121,13 +120,28 @@ The order, from [research 011](../../01-RESEARCH/011-the-module-graph/worked-pro
|
||||
```
|
||||
0 a container runtime exists detected — docker or podman — or installed
|
||||
1 PostgreSQL runs pulled by digest, from the bundle
|
||||
2 a database is created in it an action, run locally
|
||||
3 the control plane's schema applied an action, against that database
|
||||
4 the control plane starts and only now is there a mesh
|
||||
5 LavinMQ, MinIO, the registry, and the ordinary path
|
||||
everything else are provisioned
|
||||
2 a database per context is created an action, run locally — one today, `inventory`
|
||||
3 each context's schema is applied an action, against its own database
|
||||
4 LavinMQ runs pulled by digest, from the bundle
|
||||
5 a virtual host, a credential, and actions, run locally
|
||||
a self-signed certificate
|
||||
6 the control plane starts and only now is there a mesh
|
||||
7 MinIO, the registry, and everything the ordinary path
|
||||
else are provisioned
|
||||
```
|
||||
|
||||
**Steps 4 and 5 are why the bundle is not one image**
|
||||
([ADR 0024](../../02-DECISIONS/0024-running-the-control-plane.md)). The control plane cannot
|
||||
provision the broker, because provisioning means telling a host, and telling a host happens over
|
||||
the broker. The first node does not escape this by being local: it enrols the ordinary way, by
|
||||
dialling the broker at the address in its token.
|
||||
|
||||
**Step 2 is one database per context and not one called `mesh`.** A context is granted only what it
|
||||
exclusively owns ([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)), *the mesh
|
||||
database* names a thing that will not exist
|
||||
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)), and a separate
|
||||
database is a boundary a cross-context join cannot casually cross where a separate schema is not.
|
||||
|
||||
Only PostgreSQL is raised from the bundle, for the reason in *The pinned bundle* above — the
|
||||
rest of the substrate is wanted only once there is a control plane to provision it.
|
||||
|
||||
@@ -161,14 +175,21 @@ host's vocabulary grows by one shape rather than by one resource type per substr
|
||||
## Open
|
||||
|
||||
- **Whether identity is the fifth.** Above; it follows from a decision not yet taken.
|
||||
- **Whether the bus must precede the control plane.** The bundle table marks this *not
|
||||
established*, and it is the one row that could still move. The control plane reaches nodes over
|
||||
AMQP, but at step 4 there is exactly one node and it is the local machine — so whether LavinMQ
|
||||
is needed to *start* or only to *reach a second node* depends on whether the control plane's own
|
||||
contexts talk to each other over the bus. If they do, LavinMQ joins PostgreSQL in the bundle and
|
||||
the bootstrap grows a step; if they do not, it is provisioned like anything else. **This is a
|
||||
question about the control plane's internal shape, not about the substrate**, which is why it is
|
||||
not answered here.
|
||||
- ~~**Whether the bus must precede the control plane.**~~ **Resolved** by
|
||||
[ADR 0024](../../02-DECISIONS/0024-running-the-control-plane.md) — it must, and the question as
|
||||
posed here could not have answered it. This asked whether the control plane's contexts talk to
|
||||
each other over the bus; they do not, being one process, which under this framing would have
|
||||
kept LavinMQ out of the bundle. What decides it is how the control plane reaches a *node*, which
|
||||
is only ever over the link.
|
||||
- **What issues the broker's certificate at bootstrap.** New, and created by the row above. A
|
||||
token pins the fingerprint a host must expect before it sends anything
|
||||
([`09-the-node-lifecycle.md`](09-the-node-lifecycle.md)), so the broker needs a certificate at a
|
||||
moment when there is no mesh to issue one and no public name to obtain one for. Self-signed and
|
||||
pinned is the shape that fits; how it is later replaced by the certificates in
|
||||
[`08-connectivity.md`](08-connectivity.md) is not decided.
|
||||
- **How a context added later gets its database.** By then there is a control plane — but one
|
||||
holding a credential that can create databases holds more than what it exclusively owns
|
||||
([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)).
|
||||
- ~~**Whether the host can do step 2.**~~ **Resolved** by
|
||||
[ADR 0005](../../02-DECISIONS/0005-the-node-host.md). A service
|
||||
running on this machine is part of this machine, so the scope was never in question — the real
|
||||
|
||||
Reference in New Issue
Block a user