mesh-controller/README.md

# mesh-controller

**Tier 2 of Novox Mesh — the control plane.** Everything that needs to know about more than one
node.

That is the whole test, and it draws the line the host cannot: the host applies and does not
decide, *because deciding needs knowledge one machine does not have*. Which nodes should run the
store, which peers belong in an overlay, whether a node has been unreachable for a week — nobody
on a single machine can answer any of them.

The reasoning lives in [novox/hq](https://git.novox.be/novox/hq); this repository carries no
argument that is not settled there.

## What it is not

- **Not the thing that changes machines.** It decides; the host applies. It never reaches into a
  node except through the host, over the link, in a bounded vocabulary.
- **Not a database.** There is no mesh database. Each context owns its own store and nothing
  outside a context touches it — including nodes, which hold no credential to any of them.
- **Not privileged.** It has no more access to a machine than a declaration can express.

## What exists today

**One context of seven, and one of the things it will do.**

| | |
|---|---|
| `inventory` | node records and enrolment tokens — **built, as far as identity** |
| `identity` | the control plane's signing key, and the keys nodes are known by — **built** |
| `config`, `connectivity`, `provisioning`, `delivery`, `observability` | not built |
| the interface every surface speaks to | not built; its shape is not decided |

```
mesh-controller migrate                    bring each context's schema up to date
mesh-controller node add <name>            create a node record
mesh-controller node list                  the nodes this mesh knows about
mesh-controller token issue --node <name>  a one-time right to join, for an existing record
mesh-controller token issue --new <name>   create the record and issue for it
mesh-controller identity show              this control plane's signing key
mesh-controller broker show                where the broker is, and what to expect there
mesh-controller version                    what this binary is
```

`migrate` is **step 3 of the foundation bootstrap** — the step the first node cannot get past, run
against a database raised moments earlier from the bundle the host carries.

### Tokens, and what they are missing

A token is **a one-time right to join, issued for a node record** — which is where re-enrolment is
decided, since what an identity binds to is settled when the token is made rather than when it is
presented.

What is built: the secret is 256 bits from the system's random source, shown once, and **stored
only as a hash**, so a copy of this database is not a set of working credentials. It is usable
exactly once and only before it expires, and both are read from the row rather than from a status
something would have had to write. Issuing again for the same node invalidates the outstanding
one — two live tokens are two machines able to join as the same node.

Redemption is a single statement that both finds a live token and spends it, so eight concurrent
attempts on one secret produce exactly one winner. There is a test that runs them.

**A token now carries three of its four parts**, and is one line of base64 a person can copy. The
signing key is real: an Ed25519 key this control plane generates once and keeps, whose public half
travels in every token. A node believes a declaration because it carries a signature that key made
— and pinning only the broker would not do, because it would make the control plane's authority
transitive, so a compromised broker could forge declarations, and since the host applies whatever
the link delivers that is the whole machine.

**All four parts are built.** Given `MESH_BROKER_ADDRESS` and `MESH_BROKER_CERTIFICATE`, a token
carries everything ADR 0004 requires. Without them it carries two, and `token issue` prints it
**and names what is missing** rather than producing something that looks complete and cannot be
used.

**The fingerprint is derived from the certificate, never configured.** A configured one can drift
from the certificate it describes, and a drifted pin is worse than none: every node issued a token
during the drift refuses to connect, and the failure looks like an attack. It is computed over the
DER bytes — what a client actually sees on the wire — so the same certificate re-wrapped with
different line endings still produces the same pin.

### Two contexts, and the rule between them is real

`identity` is the second context and it exists partly to test a claim this repository had made and
never checked: that a context reaches only its own store. It holds `MESH_STORE_IDENTITY`;
`inventory` holds `MESH_STORE_INVENTORY`; there is no setting that reaches both and no way to ask
for one. Run `migrate` with only one and it stops, naming the grant it does not have.

A token needs a node record from one and a signing key from the other. Neither reads the other's
store — the process holding both grants asks each for its part.

### How a node is known

**The node generates a keypair; the mesh records the public half.** The same principle as SSH, and
the same rule `08-connectivity` already applies to the overlay keys — which is where this was
settled all along, though it took being asked directly to notice.

Only the public half is ever stored, and that is the property worth having: **a copy of this
database grants nothing.** It is a list of who to believe, not a set of credentials, which is what
makes *compromise of a node is compromise of that node* literally true.

Exactly one key is live per node. Re-enrolment revokes the one it replaced, in the same
transaction — two live identities for one node record is the stolen-laptop case, with the replaced
machine still believed. The database enforces it as well as the code, and there is a test running
six concurrent enrolments that fails when the constraint is removed.

**Not the machine's SSH host key**, though that was the obvious economy. Host keys are regenerated
by reinstalls and image clones, which would silently un-enrol a node; their lifecycle belongs to
sshd rather than the mesh; and a partial host has no SSH daemon at all.

## Reaching a store

**A context is granted only what it exclusively owns.** No shared writes, no read-only role on
another context's store, and no connection string that reaches more than one.

That is a rule about credentials, so it is built as one. There is no mesh-wide connection setting
and no way to ask for one:

```
MESH_STORE_INVENTORY=postgres://…/inventory
```

A process granted `inventory` holds that variable and no other. Reaching another context's store
is not a matter of restraint — it has no address for it and no credential to present. And it is
how the rule is *checked*: what a context can reach is visible in the declaration that runs it,
as the list of variables it was given.

Each context's database is named after the context. There is deliberately no database named for
the mesh as a whole.

## Migrations

Numbered, embedded in the binary, applied in order, each in a transaction with the row recording
it. The applying is four lines; the rest is refusals, and the refusals are the point:

| it stops when | because |
|---|---|
| a migration that ran has since been edited | the database holds the old version, the repository holds the new one, and nothing holds the difference |
| a migration is numbered below one that already ran | usually two branches taking the same next number — applying it now runs the schema in an order nobody tested |
| two migrations share a number | order is the entire guarantee, and two files with one number have none |
| a file in the directory is not a valid migration name | a misnamed migration would otherwise never run and nothing would say so |
| a migration is empty | it records that something happened and changes nothing, which cannot be told from a mistake |

All of them stop rather than warn. At the moment any of them is true, nobody knows what the
database contains, and there is no correct guess about a schema.

Running it again does nothing. Two copies running at once take an advisory lock, so a restart
during a slow migration does not become two runners racing.

## Building

```
make build      the binary
make image      the container image
make check      gofmt, vet, and every test against a real PostgreSQL
```

`make check` raises a throwaway PostgreSQL in a container and takes it down afterwards, including
when the tests fail. Without one the tests that need a database **skip and say so** rather than
passing quietly — `make test` is the honest subset, not the gate.

Tests run against a real database rather than a fake because what is being tested *is* the
database's behaviour: that DDL is transactional, that an advisory lock serialises, that a
checksum mismatch is caught against a record PostgreSQL actually kept. A fake would assert that
the fake behaves as expected.

Every test here has been confirmed to fail when the behaviour it defends is removed. Two did not,
when first written, and both are now commented with what they were missing.

## Checks that cross into the host's repository

Two things are agreed between this repository and `novox/mesh-host`, and each is a separate struct
on each side. A field renamed on one of them fails **silently** — the crossing succeeds and
something is simply absent — so both are checked by handing one side's real output to the other's
real parser. Neither runs by default; each skips with a reason, because a repository that fails
without its neighbour checked out is a repository nobody can build.

**What this mesh sends, read by the host that receives it:**

```
mesh-controller:  ./build/mesh-controller plan <node> --json > /tmp/d.json
mesh-host:     MESH_EMITTED=/tmp/d.json go test ./internal/declaration/ -v
```

**What a node says when it joins, read by this mesh:**

```
mesh-host:     MESH_ENROL_OUT=/tmp/enrol.json go test ./internal/link/
mesh-controller:  MESH_ENROL=/tmp/enrol.json make check
```

The second does more than compare shapes: it seals something to the key that arrived and opens it
with the private half the host kept. A key that is present, correctly named and simply *wrong*
passes every check that only looks at the message.

## The image

`FROM scratch`, holding one statically linked binary and nothing else — no shell, no package
manager, no libc, no CA certificates.

Not a size optimisation. The bundle a host carries pins this image by digest, and it is fetched
and run on a machine where no mesh exists to check anything and a person is expected to have read
the bundle and believed it. Everything in the image is something that person would have to audit.

## Where the reasoning lives

| | |
|---|---|
| what the control plane is | novox/hq ADR 0006 |
| what it takes to run one, and why Go | novox/hq ADR 0006 |
| a context owns its store, exclusively | novox/hq ADR 0008 |
| schema changes are numbered migrations | novox/hq ADR 0013 |
| a test defends a decision | novox/hq ADR 0017 |