hq/03-DESIGN/01-to-be/06-the-controller.md

---
layer: to-be
status: in-progress
code:
  - mesh-controller
updated: 2026-08-31
decisions:
  - 02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md
  - 02-DECISIONS/0005-the-node-host.md
  - 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
  - 02-DECISIONS/0008-a-context-owns-its-store.md
  - 02-DECISIONS/0019-how-this-repository-works.md
  - 02-DECISIONS/0046-a-module-configuration-is-its-assignments-not-its-manifest.md
---

# The controller

Tier 2. The term appears seventy-nine times across this repository and was defined nowhere,
which is `how-we-build` §5 failing on this repository's own vocabulary.

This document defines it. It does **not** design the contexts inside it; those are open in
[research 006](../../01-RESEARCH/006-mesh-from-scratch/00-overview.md).

## The definition

> **The controller is everything that needs to know about more than one node.**

That is the whole test, and it is not arbitrary — it follows from
[ADR 0005](../../02-DECISIONS/0005-the-node-host.md). The host applies and
does not decide *because deciding needs knowledge the machine does not have*. So the line falls
exactly there:

| Question | Whose |
|---|---|
| write this file, with this content, with this mode | the **host** — one machine |
| which nodes should run the store | the **controller** — needs every node |
| is this unit running | the **host** — one machine |
| which peers belong in this node's overlay | the **controller** — needs every node |
| what does this machine have installed | the **host** reports; the controller **records** |
| has this node been unreachable for a week | the **controller** — nobody else is watching |

A useful consequence: **anything a single machine could answer alone is not the control
plane's.** If it needs no second node, putting it here is a mistake, and the tier rule will not
catch it because the dependency direction is still correct.

## What is inside it

**Seven contexts and one interface**
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) —
each one earning its place by the test above rather than by being ours:

| | | needs to know about more than one node because |
|---|---|---|
| **inventory** | nodes, modules, assignments, versions | that *is* the mesh-wide fact |
| **config** | settings, secrets, and deriving them onto nodes | it derives **onto nodes** |
| **connectivity** | overlay, resolution, exposure, filtering, certificates — **specified in full in [`08-connectivity.md`](08-connectivity.md)** | who peers with whom; which node is reachable |
| **provisioning** | resource grants between modules | consumer and provider may be on different nodes |
| **delivery** | source to artifact to node | it targets nodes |
| **observability** | health, logs, metrics, alerts | *unreachable for a week* is nobody else's to notice |
| **identity** | agents, humans, services, authorisation | credentials follow an agent's node bindings and modality |
| **api** | the one interface every surface speaks to | — it is an interface, not a context |

**What is deliberately not here.** `work`, `knowledge` and `stream` are **mesh-hosted
applications** — first-party, shipped with everything else, and running on the mesh the way
anything else does. A task does not need to know a node exists, and *being ours does not make
something infrastructure*. `ai` is folded into `config`: a provider licence is an ordinary grant.

**`record` is an open question rather than an eighth entry.** Contexts integrate through it
([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)), which makes it load-bearing,
and [research 006](../../01-RESEARCH/006-mesh-from-scratch/skeleton.md) leaves *where it lives*
unresolved — putting it in the foundation risks recreating the circularity the tier design just
removed. Listing it here would settle by naming what has not been settled by arguing.

**One of the seven is built.** `inventory` owns a database of that name and holds the node records;
the rest do not exist. What it takes to run any of them — the language, and what must already be
running before it starts — is
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md), which also records where the
build stopped and why: at **identity**, because what a node presents to prove who it is is not
decided anywhere, and a migration is the most expensive place in this system to guess.

**These are contexts, not services.** They are separate in the sense that matters — each owns
its own store, and they integrate through the record rather than by reading one another
([`how-we-build`](../../00-META/how-we-build.md) §4). They are not separate deployables, and
[research 011](../../01-RESEARCH/011-the-module-graph/00-overview.md) records why that
constraint is load-bearing: a single surface can compose them only while there is one interface
in front of them.

## Nothing outside a context touches its store

The question this answers: **can a node write to the registry database?** No — and not "only
through one node", which is the weaker arrangement it might be mistaken for.

> **No node holds a credential to any controller store, for writing or for reading.**

That is not a new rule here; it is four already taken, and it is worth seeing them together
because each one alone reads like a detail:

| | |
|---|---|
| [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) | the host never queries the mesh database |
| [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) | a node holds its own identity **and nothing else** — the shared database credential every node carries today is the exposure this exists to remove |
| [ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md) | a context is granted only what it **exclusively** owns: no shared writes, no read-only roles |
| [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) | there is no single mesh database, and nothing reads one |

### So how does anything get in

**Over the broker, as a message; the owning context writes.**

```
node ──event/report──► broker ──► the context that owns that data ──► its own store
```

A node reports what it applied, what it holds, and that it is alive. It **states**; it does not
**write**. The difference is the whole security boundary: a node that can write cannot be
prevented from writing anything, and a node that can only state has its blast radius bounded by
what the message vocabulary can say.

Reads work the same way in reverse — a node is *told*, in declarations. It never asks.

### Who actually consumes, and who writes

**The controller is the consumer. There is one of it, and the context that owns the data does
the write.**

```
node ──► broker ──► the controller, consuming
                          ├─ a node reported what it applied   ─► inventory   writes the registry
                          ├─ a node reported health            ─► observability  writes its own store
                          └─ a grant was requested             ─► provisioning   writes its own store
```

Seven contexts, **one deployable** — they are not separate services, so this is one process
consuming and dispatching internally, not seven consumers racing. Each context then writes only
the store it exclusively owns
([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)).

**One consumer is a property worth having**, not just a consequence of
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md). The as-is records that
*two consumers accidentally sharing one queue silently split the traffic between them, each
receiving half of what it expects* — which has happened, between a module's daemon and its
capability server. With one consumer that class of fault cannot arise.

**And the broker is the buffer while the controller is down.** Nodes go on publishing;
messages queue; the controller drains them when it returns. That is what makes
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)'s single controller
tolerable — an outage delays the mesh's *knowledge* rather than losing it.

**With one consequence that must be bounded before it is discovered:** a queue with no limit
grows until the broker's disk is full, and the broker is the one component every node depends
on. Queues carrying node reports need a maximum length or a message lifetime, and losing the
oldest health report is obviously right where losing the oldest declaration acknowledgement is
not — so the bound is per queue and is not decided here.

### On volume, which is the real worry underneath

**Most high-frequency writes are not registry writes, and that is the first thing to check
before designing for throughput.** The registry is `inventory`'s store: nodes, modules,
assignments, versions. Those change when somebody changes something.

**Logs, metrics and health checks belong to `observability`**, which owns a different store
([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)). Sending them to the registry
would be exactly the shared-schema mistake 0045 exists to stop, arriving through the back door
marked *performance*.

That leaves one genuine funnel: every context's writes go through the process that owns it, and
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) says there is one of it.
For a mesh of a handful of machines this is not a scaling problem, and **it should not be solved
by giving nodes database credentials** — that trades a bounded problem for an unbounded one. If
it ever binds, the answers are at the consumer: batch, apply backpressure, or move the highest
volume stream out of a relational store entirely.

**What observability actually stores its data in is not decided**, and it is the one place where
volume genuinely argues against a relational store.

## What it is not

- **Not the thing that changes machines.** It decides; the host applies. It never reaches into a
  node except through the host.
- **Not a surface.** Tier 3 is how people and agents reach it. It has one interface; the
  surfaces are what speak to that interface.
- **Not the foundation.** It *runs on* tier 1 — PostgreSQL, LavinMQ, an OCI registry
  ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) — and cannot start without
  them, which is what makes them a lower tier.
- **Not privileged on a node.** It has no more access to a machine than the declaration
  vocabulary allows ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)).

## It is also a consumer

The property that makes tier 2 unlike the others: **the controller has requirements of its
own.** It needs a PostgreSQL database, an AMQP virtual host, and a bucket — the same things any
module needs, granted the same way.

That is the circularity the tiers exist to resolve rather than hide: the controller cannot
provision its own database, because it is not running yet. So its **store** is raised from the
bundle the host carries, before there is a controller to ask
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md),
[research 011](../../01-RESEARCH/011-the-module-graph/worked-provider.md)).

Its virtual host and its bucket are **not** in the bundle — by the time they are wanted there is
a controller to grant them. Whether the bus must come first is
[open](07-the-foundation.md#open), and it turns on whether these contexts talk to each other over
it.

## Where it runs

**On nodes, like anything else.** It is not a place outside the mesh; it is modules the mesh
hosts, assigned to nodes by the same mechanism as everything else.

**One node runs it, and nothing takes over**
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)). The node is assigned,
never elected — no promotion, no quorum, no split brain.

That is sound rather than merely cheap, because the design already tolerates the controller
being absent by construction: a node reconciles from **its own** store
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)) and
never needed to ask anybody to hold the state it was last given. So the controller being down
is not a new failure mode — it is
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s ordinary disconnected
situation, happening to every node at once. **What is lost is change, not operation.**

The honest half: this node is a single point of failure, recovery is restore rather than
failover, and **certificate renewal is the clock** — an outage outlasting a renewal window expires
every public name.

## Open

- ~~**The contexts themselves.**~~ **Decided** — seven, by
  [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md).
  What remains open is narrower and named there: **where the record lives**, which research 006
  leaves unresolved because the foundation is the one place it must not go.
- **How far it may be split.** One deployable today. Splitting a context out costs the single
  interface a surface depends on
  ([research 011](../../01-RESEARCH/011-the-module-graph/00-overview.md)).
- ~~**How many run, and what a node does without one.**~~ **Resolved** by
  [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md). What remains is
  measurement: nothing reports how long the controller has been unreachable, or how close a
  certificate is to expiry — both needed for restore-not-failover to be a plan rather than a
  hope.
- **What the interface is.** One interface is stated; its shape, and whether it is request,
  subscription or both, is not
  ([research 011](../../01-RESEARCH/011-the-module-graph/worked-provider.md)).