Files
hq/02-DECISIONS/0004-a-node-and-how-it-joins.md
T
jschoubben a73014dcd5 A bare machine became a mesh, and something joined it
First end-to-end raise. A machine with a container runtime applied the
bundle its host carries and ended with a store, databases, schemas, a
broker holding a certificate it generated itself, and the control plane
serving. Then it took a token, checked the broker against the pinned
fingerprint, generated three keypairs and enrolled — the first node being
a node whose mesh is not up yet, observed rather than argued.

And a credential crossed. Declared the provider of a database for a
second node and pushed to over the broker, the machine ended with the
password in one file at mode 0600, and that password appears nowhere in
the declaration that crossed the broker, nowhere in the control plane's
database, and nowhere in what the node reported back. That is the whole
secrets argument, measured.

One fault, in the joining: the token did not say what the mesh calls the
machine, so enrolment needed a flag its own help said it did not, and
failed at the broker with an empty username. It is the fifth thing a
token carries now — the node cannot work its own name out, because the
broker account it authenticates as is named after it and exists before
the mesh has told it anything.
2026-08-30 02:37:37 +02:00

264 lines
15 KiB
Markdown

---
topic: the tiers
status: accepted
date: 2026-08-28
deciders: jochen
reconstructed: false
---
# 4. A node, and how it joins
*Consolidated 2026-08-28 from four records.*
## What a node is
**A managed machine inside the mesh.** Not a device that is merely known about, not an
unprivileged something. If the mesh does not manage it, it is not a node — it is a client, a
peer, or a thing on the network, and those want their own names rather than a weakened version of
this one.
**A disconnected node is still a node, in a different situation.** Reachability is **state, not
class**. A machine switched off, roaming, or behind a connection that has dropped has not become
a lesser kind of thing; it has a last-known state and a pending set of declarations.
The distinction people reach for is real, but it is **capability** — what this machine can be
asked to do — and that belongs in the host's profile rather than in the definition of a node.
**This is the rule that does the most work elsewhere.** A single control plane is tolerable
because its absence is every node in the ordinary disconnected situation at once. An episodic
host on a phone is that situation more often. Neither needed a new mechanism.
### A node runs one agent session
*Written 2026-08-29. It runs on every node today and appeared in no record, which is how something
deliberate comes to look accidental.*
**A node is a machine. The session is a feature of it** — one of the things running there, like the
host, like any workload. The node does not think; something on the node does. Which is why
[ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md)'s *a node does not authenticate to a model
provider, agents do* holds unchanged: the session authenticates, and it is not the machine.
**It is permanent, and it remembers.** Anything in the mesh can send it a message; it replies; and
what it was asked ten minutes ago is still there next week, alongside what everything else asked in
between — the same way both sides of any conversation remember it.
Its system prompt is the node's **engram** — what makes one node's replies recognisably its own
rather than generic.
**It has its own tools**, and fewer than a session a person is driving directly. So a question can
be answered by going and looking: *what is in our forge*, not only *what is your battery*.
**Messages travel the broker like everything else** ([ADR 0002](0002-nodes-communicate-over-a-broker.md)).
There is no second transport and nothing is dialled.
**Any node can message any node, and this is the one part of the system that is genuinely a mesh**
— symmetric, with no centre. A node that is asked something it does not have can ask another, and
how it passes the question on is its own business: it may say who wants to know, or simply ask. A
person relaying a question makes the same choice, and it follows from the engram rather than from a
message format.
**There is no authorisation between nodes.** Every node is the operator's own, so a message from
one is a message from them, and asking a node something is asking a colleague rather than
presenting credentials. Stated once so it is not discovered later: **the mesh boundary is therefore
the security boundary** — anything inside can reach what any node can reach, which is what puts the
whole perimeter on the token and the overlay
([ADR 0007](0007-connectivity.md)).
**It can be switched off, and switched off it still answers.** A node whose session is disabled
replies saying so, at once, with no model involved — the queue is still read, and the state is the
reply. That is deliberate and it is the same rule the host follows about a service that does not
exist: **absence must never be indistinguishable from a failure to answer.** A node with nothing
there is a silence somebody has to go and diagnose; a node that says *I am switched off* is not.
**One per node, always, and it cannot be moved to another machine.** Two and nothing decides which
replies; none and the node is mute; moved, and one machine is answering as another.
**It is not an employee** ([ADR 0003](0003-agents-are-persistent-employees.md)). Nobody hires it,
it holds no tasks, it drains nothing and it is never reassigned — that vocabulary was written for
workers and does not describe this. It exists because the node does, and it is gone when the node
leaves.
## How it joins
**The host has one behaviour and two sources of declaration.** What differs between the first
node and the fiftieth is not what the host does but where the declaration comes from — and, as
above, that is a situation rather than a class.
| | declaration comes from |
|---|---|
| no mesh reachable | the pinned bundle the host carries |
| mesh reachable | the control plane, over the link |
**The first node is not a different kind of node.** It is a node whose mesh is not up yet. It
applies the bundle it carries, the control plane comes up on top of it, and from that moment it
takes declarations like everything else. **Its specialness is temporary and self-erasing**, which
is what the hand-run bootstrap scripts never were.
**A joining node does the minimum to be reachable and nothing else** — an identity, an address,
and one peer to reach. It does **not** compute the overlay: the whole peer set is derived
centrally and pushed down.
That is also why the migration is smaller than it looked. The hard part of the overlay — every
node's key, address, site and reachability — is only needed to compute the *whole* mesh, and a
joining node needs one peer.
## The link is the security boundary
**Everything reaching a node arrives one way**, and four properties make that a boundary rather
than a pipe.
**It is outbound and node-initiated.** The node dials the control plane; nothing dials a node. Not
only defensive — most nodes sit behind a household connection with no forwarded port, so an
inbound control channel would work for one node and not the rest, and the difference would be
invisible until it mattered. **A node has no listening control surface at all.**
**A node holds its own identity and nothing else.** No shared secret, no credential to anything it
does not own. **Compromise of a node is compromise of that node** — which the current arrangement
does not have, because every node permanently holds the same database and object-store
credentials, and there is no mechanism that rotates one and informs everything holding it.
### What that identity is: a keypair the node generates
*Written 2026-08-29. This is the same rule as the sentence above, and it had been treated as an
open question for weeks because of a word.*
**The node generates a keypair. The private half never leaves the machine. The mesh records the
public half.** Ed25519, the same as the control plane's signing key, in the other direction:
the mesh proves itself to a node by signing, and a node proves itself to the mesh by signing.
**This was never open.** [`08-connectivity.md`](../03-DESIGN/01-to-be/08-connectivity.md) already
says it of the overlay keys, in these words: *each node generates its own keypair, the private key
never leaves the machine, the public key is published to the mesh* — and adds that this **is**
ADR 0004's *a node holds its own identity*, applied. What was missing was applying it to the thing
this record is about.
**The word that caused it:** the lifecycle says a joining node *receives* its own durable identity,
which reads as the mesh issuing something, and then the question becomes *issuing what*. It does
not issue anything. The node arrives holding its identity; what it receives is **being known**.
Enrolment is the moment the mesh writes down a public key it will believe, and the one-time secret
is what buys the right to have it written down.
**Everything above then holds literally.** Nothing is stored that could be stolen and replayed: the
mesh's copy is a public key, so a copy of the mesh's database grants nothing. *Compromise of a node
is compromise of that node* becomes true rather than aspirational, because the only secret on a
machine is the one that identifies it.
### What "connecting to the mesh" is, concretely
*Written 2026-08-29, because it was asked and this record had never said it.*
**One outbound AMQP connection from the node to the broker, held open.** That is all of it. There
is no second connection and nothing is ever dialled *at* a node. Being in the mesh, operationally,
means that connection is up; being disconnected means it is not
([`09-the-node-lifecycle.md`](../03-DESIGN/01-to-be/09-the-node-lifecycle.md)).
**Two different things ride on it, and conflating them is what made this confusing:**
| | what it answers | who issues it |
|---|---|---|
| **an AMQP account** | may this connection be accepted at all | **the mesh, at enrolment** |
| **the node's keypair** | which node is speaking, on every message | **the node**, above |
**The account is the mesh's to issue**, and per node. The broker has to authenticate somebody
before a connection exists, and a shared account would let any node consume another's queue —
which is the shared-credential fault this record exists to remove, reappearing at the transport.
So enrolment creates that node's account and hands it over, and it is rotatable without touching
the node's identity.
**The keypair is not made redundant by it.** With only an account, the control plane would know
which node is speaking *because the broker says so* — and that is the same transitive authority
this record refuses in the other direction. A compromised broker could then attribute reports to
whichever node it liked, and the control plane would act on them. Signing is what removes the
broker from the question in both directions.
**So a node holds two things after enrolment**: a credential the mesh issued for reaching the
broker, and a key it generated itself that the mesh only ever sees the public half of. Both are
its own, neither reaches anything else, and *compromise of a node is compromise of that node*
still holds.
### Its own key, not the machine's SSH host key
Reusing the host key is the obvious economy and it is refused, for reasons that are operational
rather than fastidious:
- **It is regenerated by ordinary events.** A reinstall, an image cloned, `ssh-keygen -A` on a
rebuild — each silently un-enrols the node, and the failure appears as an authentication problem
with no cause anybody changed.
- **It is managed by something else.** Its lifecycle belongs to the machine's SSH daemon, and an
identity the mesh depends on should not rotate on a schedule the mesh does not know about.
- **Not every node has one.** A partial host has no SSH daemon
([ADR 0005](0005-the-node-host.md)), and an identity scheme that excludes a supported kind of
node is not one.
**The mesh should still know the host key** — it knows every node, so it can distribute host keys
the same way it distributes authorised keys
([ADR 0006](0006-the-substrate-and-the-control-plane.md)), and node-to-node SSH stops depending on
trust-on-first-use. That is the good half of the idea, kept.
**Authority is mutual.** The node proves it may join, and the control plane proves it is the
mesh. One-way is not enough: the host applies whatever the link delivers, so a node that cannot
tell the mesh from something impersonating it will apply that something's declarations.
**What may be pushed is bounded by form, not by trust.** Declarations of known shape, never a
command to run. Stated honestly, **this bounds form and not impact**: a compromised control plane
can declare harmful state and the host will apply it faithfully, because that is what it is for.
What the property buys is that the blast radius is *describable* — exactly what the declaration
language can express, which can be reviewed. An arbitrary command channel has no such bound.
## The enrolment token carries the mesh
Mutual authority needs the node to verify something before it trusts anything, and that is a
circle: verifying the mesh needs the mesh's certificate authority, and obtaining one means
trusting whoever hands it over. There is a second circle beside it — a node must reach the mesh
before the mesh has configured it, so it can resolve no mesh name.
**Both are the same shape: a node needs a fact about the mesh before it has any trustworthy way
to obtain one.** So that fact arrives by a path other than the mesh.
**The token carries five things**, and it is the only thing a joining node needs:
| | |
|---|---|
| **who it is** | the name the mesh calls this machine |
| **where** | the broker's **address**, not a name — there is no resolution yet, and this is why none is needed |
| **what it is connecting to** | the fingerprint of the broker's certificate |
| **who it will believe** | the control plane's signing identity |
| **the right to join** | a one-time secret, useless once used and useless after it expires |
*The first row was added 2026-08-30, from raising a mesh end to end for the first time.* It reads
like an oversight and is not: **the node cannot work its own name out.** The name is the mesh's,
chosen when the record was created, and the broker account the node authenticates as is named
after it — so it must be known *before* the mesh can tell the node anything. It is not a secret,
and whoever issues the token already has it.
Without it, enrolment fails at the broker with an empty username and a message about credentials,
which points at everything except the cause. **A missing fact that surfaces as an authentication
error is worse than one that surfaces as a missing fact.**
**Carried out of band**, by the person adopting the machine. That is what breaks both circles:
its authenticity comes from the channel it travelled, not from anything the node can check
afterwards. **Trust on first use, with the first use moved out of band** — the difference between
a pin and a guess.
**The endpoint and the authority are two identities.** A node connects to the broker and takes
instruction from the control plane behind it. Pinning only the broker would make the control
plane's authority *transitive*, and a compromised broker could then forge declarations — which,
since the host applies whatever the link delivers, is the whole machine. So the transport is
verified once at connect, and **each declaration is verified by its signature, every time**.
**What this settles:** the mesh's certificate authority is not a bootstrap concern — it certifies
internal names once a node is a member. Nothing needs name resolution before the link. And
nothing is placed on disk beforehand except the token, which is the first moment *a node holds
only its own identity* becomes true rather than aspirational.
## Consequences
- **Declarations must be signed**, and the host must tell *this is not from the mesh I joined*
apart from *this is malformed*. Rotating the signing identity is a fleet-wide operation with an
overlapping rollover, and that is the cost of not trusting the broker.
- **The token becomes security-critical**, because it carries the pin. Tampering substitutes the
mesh — which is strictly better than the alternative, where there is nothing to tamper with and
the node trusts the first answer unconditionally.
- **A rejoining node is ordinary.** There is no long-lived secret to recover, so a node that lost
its identity gets a new token.