novox/hq 09-the-node-lifecycle asks for this and it was missing: the host
reports what it owns and the mesh keeps the last report. A backup, never a
source -- nothing decides anything from it, and a node that disagrees with it
wins, because the node is the one that can see the machine.
Its point is the orphans. A node that loses its state file currently strands
whatever it applied: nothing on the machine knows those resources were the
mesh's doing, so nothing removes them. With this, a rebuilt node receives both
the declaration and the record of what it previously owned.
Never reported and reported nothing are kept apart, and that is the whole care
in it. A node that applied nothing holds nothing; a node that has never spoken
is unknown -- and handing back an empty list for the second would tell a
rebuilding node it owns nothing and have it remove whatever it found.
The age comes back with the answer rather than being left for the caller to go
and find. An answer about a machine is worth much less without one, and this
repository has already been bitten by a cache with no age on it.
A refusal or a partial failure moves last_seen and nothing else: neither is an
account of what the machine holds, and recording one as though it were would
tell a rebuilding node to remove what it still has.
The control queue was bound to enrol and not to report, so every report a node
sent was accepted by the broker, matched no binding, and dropped. The publisher
saw success and the consumer saw nothing, for an afternoon.
The refactor that was meant to bind both never applied -- it left behind a
helper nothing called, which compiled and passed vet. The loop is now where the
bind is, so there is one place to forget rather than two.
`declare` sends a node a signed declaration; `serve` now also consumes reports.
Signed over the exact bytes published, which is what the node verifies. Anything
re-encoding in between would sign one thing and check another, and a difference
in key order alone would have a node refuse a declaration that was genuinely
the mesh's.
Sent to the node's queue directly rather than through the exchange: a
declaration is for one node, and routing by name through a shared exchange
means a binding per node that nothing removes when a node is retired.
Enrolment now issues the node its own broker password, replacing the token's
secret, and tells it the broker address, the fingerprint and the signing key --
so a node can reconnect after a restart without a person and a new token, which
is what makes disconnection ordinary rather than a crisis.
A report is a statement, not a write. What a node says it applied is its own
account of its own machine, kept as a copy for recovery rather than as a source.
There was no chicken-and-egg to solve. The mesh runs the broker, so it creates
the node's account when it issues the token, and the one-time secret is that
account's password. A joining node's first connection is already authenticated;
enrolment is what it says once it is in. I had been treating this as a decision
that needed taking, and it did not.
The account is per node and scoped: it may read its own queue, write to the one
exchange, and configure nothing else. The patterns are anchored and the node
name is constrained to characters that cannot widen them, because a name
carrying a dot or a star would silently let that node read everybody's queues.
`serve` is the control plane running: one connection, one queue, one consumer.
One deliberately -- two consumers on a queue get round-robined and each receives
half of what it expects, which has happened on this project before, between a
module's daemon and its capability server.
Enrolment spends the token first, in the single statement that both finds and
marks it, and only then records the key. That order is the order things become
irreversible: recording a key for a node whose token turned out to be spent
would leave the mesh believing a machine that never had the right to join.
Refusals are one message for every reason. The log says which, where an
operator can see it; the node is told only that the token cannot be used.
Verified in the lab, on a sealed machine, through the whole first-node path.
The host defines the same wire format separately, because it requires nothing
present and does not import this. A test on each side asserts the exact field
names, so renaming one breaks both immediately rather than at enrolment on a
real machine.
The thing I had been calling blocked for weeks, built in an afternoon once it
was pointed out that it was already decided. 08-connectivity says of the
overlay keys: each node generates its own keypair, the private half never
leaves the machine, the public half is published -- and says outright this IS
ADR 0004's "a node holds its own identity". Nobody had applied it to node
identity itself.
identity now holds the public half of each node's key. Only the public half,
which is the property worth having: a copy of this database is a list of who to
believe, not a set of credentials, so compromise of a node really is compromise
of only that node.
Exactly one key is live per node, and re-enrolment revokes the one it replaced
in the same transaction -- two live identities is the stolen-laptop case with
the replaced machine still believed.
Fault injection was worth the time here. Three findings. The unique index was
defended by no test at all: sequential enrolment is already safe because the
code revokes before inserting, so removing the constraint changed nothing. The
constraint only matters when two enrolments race, and there is now a test that
runs six at once and fails without it.
My injection harness also lied to me. One injection matched nothing, changed no
file, and reported NO BITE identically to a real one -- so a test that defends
nothing and an injection that does nothing look the same. The harness now
checksums the files and says NO-OP when they did not change.
And one honest NO BITE left standing: making the key lookup return a zero key
for an unknown node does not fail the test, because the signature check refuses
it a line later. Two independent mechanisms, not a placebo.
61 tests, none skipped.
Given the broker's address and its certificate, mesh-control now issues a
token carrying everything ADR 0004 asks for: where to connect, what to expect
there, whose signature to believe afterwards, and a one-time right to join.
Verified by decoding one and checking the fingerprint against `openssl x509 |
sha256sum` -- they match.
The fingerprint is derived from the certificate on disk and never configured.
A configured pin can drift from the certificate it describes, and a drifted pin
is worse than none: every node issued a token during the drift refuses to
connect, and the failure looks like an attack rather than a mistake.
Computed over DER, which is what a client sees on the wire. Hashing the PEM
text instead would mean the same certificate, re-wrapped with different line
endings, produced a different pin -- there is a test for exactly that, and one
for pointing this at tls.key by mistake, which would otherwise produce a
confident pin over the wrong file.
Having no broker stays a state rather than a failure: a control plane holds
records and a signing key without one. Having half a broker is refused, because
a token with an address and nothing to check it against invites a node to trust
whatever answers.
Fault injection caught the same weak test I wrote earlier in the day -- asking
whether something failed rather than why, so deleting the guard changed nothing
because it failed one line later anyway. Both are now asserted on the reason.
Everything is blocked on what a node presents to prove which node it is. This
builds the other direction, which is not blocked: what a node believes.
identity is the second of the seven contexts. It holds an Ed25519 signing key
the control plane generates once, whose public half now travels in every
enrolment token. A node believes a declaration because it carries a signature
that key made -- pinning only the broker would make the control plane's
authority transitive, and since the host applies whatever the link delivers, a
compromised broker forging declarations is the whole machine.
Establishing the key is idempotent, and it has to be: a second key generated by
a restart is a mesh where every node holds the wrong public half, so every
declaration is refused by every node with nothing visibly wrong. The guarantee
is a partial unique index plus a read-back, not the check before the insert --
six processes racing to establish all agree on one key, and there is a test
that runs them.
Tokens are now one line of base64 carrying three of their four parts. The
missing two are the broker's address and its certificate fingerprint, both step
5 of the bootstrap. The command prints the token and names what is missing
rather than emitting something that looks usable.
The second context also tests a claim this repository had made and never
checked: that a context reaches only its own store. Two databases, two
credentials, no setting that reaches both. Running migrate with one stops and
names the grant it lacks -- verified, not asserted. Assembling a token needs a
node record from one and a key from the other, and neither reads the other's
store; the process holding both grants asks each for its part.
45 tests, none skipped. Fault injection found one test whose property is
enforced somewhere other than where I injected -- idempotency comes from the
database constraint, not from the early return, which is what the code comment
already said.
The next step after the schema: inventory now holds node records and enrolment
tokens, and mesh-control has the commands to work with them.
A token is issued for a node record, which is where re-enrolment gets decided
-- what an identity binds to is settled when the token is made, not when it is
presented, so the machine presenting one does not need to know whether it is
joining or returning.
What the token guarantees, each with a test confirmed to fail when the
behaviour is removed: the secret is 256 random bits, shown once and stored only
as a hash; it works exactly once; it stops working when it expires; issuing
again for a node invalidates the outstanding one, because two live tokens are
two machines able to join as the same node. Redemption is a single statement
that finds and spends together, so eight concurrent attempts on one secret
produce exactly one winner rather than a race between a check and a write.
Refusals are deliberately identical for unknown, spent and expired. Somebody
guessing must not learn which guess was a real token that had merely aged out.
SHA-256 rather than a password hash, and that is a choice not a shortcut: the
secret is high-entropy random, so there is nothing to guess and a slow hash
would buy nothing while making every redemption expensive.
It stops before what a node receives in exchange. What a machine presents
afterwards to prove it is that node is not decided anywhere, and a migration is
the most expensive place here to guess.
So a token carries one of the four things ADR 0004 requires. The command prints
the secret and then says exactly that -- the broker's address, its certificate
fingerprint and the control plane's signing identity do not exist yet. Better
than emitting something that looks complete and silently cannot be used.
The language and the bundle ordering went into ADR 0006, where the substrate
and the control plane already live, and the store-per-context mechanics into
0008, which already decided the rule. Nothing changed but where the reasoning
is kept.
Tier 2 exists now. It holds one context of seven, inventory, and does one
thing with it: brings its schema up to date. That is step 3 of the substrate
bootstrap -- the step the first node cannot get past.
Verified against a real PostgreSQL, with the built binary: applied 0001-nodes,
reported 'already up to date' on the second run, and the node table is there
with the index and the unique constraint the migration asks for.
Written in Go, and the image is FROM scratch holding one file. Confirmed by
unpacking it. That is the whole argument of ADR 0024: the bundle pins this
image by digest and runs it where nothing can check it, so everything in it is
something a person has to audit before trusting a first node.
Exclusive store ownership is built as a rule about credentials rather than
about intentions. There is no mesh-wide connection setting and no way to ask
for one -- a context reads MESH_STORE_<ITS OWN NAME> and holds nothing else, so
reaching another context's store needs a new variable, which is visible in the
declaration that runs it.
The migration runner is mostly refusals: an edited migration that already ran,
a migration numbered below one that has run, duplicate numbers, misnamed files,
empty files. All stop rather than warn, because at the moment any of them is
true nobody knows what the database holds.
It stops before identity, deliberately. What a node presents to prove who it is
has not been decided anywhere, and a migration is the most expensive place in
this system to guess.
Two tests did not defend what they claimed, and both are fixed rather than
removed. One asked only whether Open returned an error, which it did either way
-- a bad context name and a missing credential both fail, so deleting the name
check changed nothing. The other claimed to prove the migration runs in a
transaction, but PostgreSQL already wraps a multi-statement query in one of its
own, so it passed with the transaction taken out. What the transaction actually
buys is that the schema change and the row recording it commit together, and
there is now a test for that which fails when they are split.
jschoubben