0e9035d4792b850a866a8aefe74b9850199192c8
216
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jschoubben
|
409cd16a09 |
The mesh decides what a node runs
The gap that has been named at the end of every report for a week. Until now a
declaration came from a person handing over a file; now it comes from what was
assigned, resolved against the catalogue, and the control plane is deciding
rather than relaying.
Everything from the module conversation, built and run on real machines:
assign laptop i3 -> accepted, brings xorg, because nothing else provides
it and there was no choice to make
assign laptop sway -> refused: xorg and wayland both claim the-seat
assign laptop editor -> refused: three modules provide a shell -- bash,
fish, zsh -- choose one
assign laptop zsh -> accepted, and the editor's requirement is answered
bash, fish beside it -> fine, nothing is claimed
Claims rather than pairwise exclusion, so a third display server would say what
it claims and need no edit to xorg or wayland. Scoped to node, site or mesh:
two DHCP servers at one site collide and at two sites do not, and the mesh-wide
one is the hub said as a claim instead of hard-coded.
Some conflicts cost no manifest field at all. The refusal above names the seat
AND the two files, because the mesh already holds every resource of every
module -- neither i3 nor sway knows the other exists.
Resource identities carry their module, so two modules may both call something
"config" without the second silently replacing the first. What a service
reflects is qualified the same way, or it would name a resource that no longer
exists and stop being restarted when its own configuration changes.
Nothing is sent until every node resolves. A push that configured three and
refused on the fourth would leave the mesh in a state nobody asked for, and the
fourth is exactly where a claim collision appears.
One real flaw found by using it rather than by testing it: assigning zsh did
not satisfy a requirement for a shell. Requirements were counted against the
catalogue without first asking what the set already offers, so "choose one and
assign it" named three modules and then ignored the one you chose. The remedy
was useless and every test passed.
|
||
|
jschoubben
|
f0cff88172 |
The mesh knows who is out of touch
09-the-node-lifecycle asks for this in as many words -- *how long it has been disconnected is a fact the mesh must hold, and nothing holds it today. Without it, a node running last month's assignments looks exactly like one that is current.* Now it holds it. `node list` says "here", "out of touch 4m", or "never spoken", and the third is kept distinct from the second on purpose: a node that has never spoken did not finish joining, and a node last heard from a month ago is running a month-old picture of the mesh. Those need different responses from a person. A bare word that a node is there moves last_seen and touches nothing else. It is not an account of what the machine holds, and recording it as one would replace the recovery copy with an empty list every minute -- so a rebuilding node would then be told it owns nothing and remove whatever it found. There is a test for exactly that. Heard is silent in the log. A node saying it is there every minute would fill the log with the ordinary case, and a log where the ordinary case is loud is a log nobody reads. Verified in the lab across the threshold, both directions. |
||
|
jschoubben
|
fc1417be72 |
Names, from the same graph as the network
Step 5 of the connectivity order. Every node's internal name resolves to its overlay address, on every node, computed centrally because it needs every node at once. Under `.internal`, which IANA reserved for exactly this in 2024 -- a name there can never collide with a public one, so an internal name that leaks into a public resolver fails rather than reaching a stranger's machine. The suffix is settable for a mesh that wants its own. Delivered in the same declaration as the peer list rather than a second one. A node holding the peers and not the names, or the reverse, is half on the network for as long as that lasts. This is not the /etc/hosts floor the design removes. That floor existed because a node had to reach the mesh's database before its own DNS worked -- a fallback for a circularity that is now gone. This is the mechanism: the complete set of names, generated whole and owned by the mesh, rather than a patch written underneath something else. A resolver daemon becomes necessary when names are wanted that are not one-per-node, and that is not yet true. A node resolves its own name to its overlay address rather than a loopback, because a service binding to the name it was given would otherwise listen somewhere nothing else can reach -- and the failure would appear on every other machine rather than that one. A node with no address gets no name. A name resolving to nothing is worse than no name: connecting to an address that does not answer hangs, where a name that does not resolve fails at once and says which name it was. Found while writing it: a test asserting every file in the declaration is mode 0600 would have forced /etc/hosts to 0600 and broken every lookup on the machine, to protect a file that is not secret. Verified in the lab: three machines, nine name lookups, each resolving to the right overlay address and reaching it. |
||
|
jschoubben
|
8b974deb42 |
A working private network, and four reasons it did not work
Three machines across two sites, two of them behind no reachable address, all nine paths open. The mesh computes the graph, delivers it as a declaration, and the nodes bring it up. Every fault below looked like success from inside the mesh: the graph was right, the files were right, the services were up, every node reported it had applied. None was reachable by reasoning. A running interface does not re-read its configuration. A node joins, every existing node's peer list changes, the file is replaced -- and the service is already running, so nothing reloads it. Fixed as declared state rather than a command: the service must reflect the file. A command to restart would be an action, and the link may not carry one. The host refused exactly that, which is how this shape was arrived at. A hub sharing a site with a spoke appeared twice in that spoke's peer list -- once as a direct peer, once as the route of last resort. WireGuard takes one entry per key and refuses the file. The ordinary shape of a small mesh, and in none of the tests written before it ran. Two nodes at one site that neither can be dialled were peered directly. Nobody opens the path, and the direct route is more specific than the hub's, so it wins and blackholes -- this design's own warning arriving in its implementation. They now route through the hub unless one end can be dialled. And Docker sets the FORWARD policy to DROP, so a hub with ip_forward enabled carried nothing between its spokes. The substrate at tier 1 silently breaks the network at tier 2, and nothing in either tier's state says so. The hub inserts its own rule above those chains and removes it on the way down. Two weak tests found by injection along the way: one asserted the keepalive rule only against the hub, whose peer entries happen not to set that field at all, so it tested an absence; the other checked the firewall rules by looking for FORWARD anywhere, which the PostDown line satisfies on its own. |
||
|
jschoubben
|
f44e73d286 |
The mesh computes a private network it cannot impersonate
The first thing the control plane decides rather than relays. Every node's peer list is derived from every node at once, which is what makes this control-plane work by definition: no node has that view. A hub, with direct peering between nodes at the same site. Not a full mesh, and the reason is a property of WireGuard rather than a preference -- there is no failover, so a more specific route to a dead endpoint blackholes instead of falling back. A node gets exactly one path to any peer, because two would mean one of them silently swallowing traffic. A roaming node is hub-only for the same reason. Reachability and the hub are declared, never inferred from an address. The address is evidence and is not the fact: carrier-grade NAT looks public and is not, a routable address behind a closed firewall looks public and is not, and the regular expression that used to decide it got the lab wrong too. Hub election by address prefix failed silently when nobody knew the convention. No private key travels, and that is the whole design. The node generated its own keypair and kept the private half; the configuration points at a file the node wrote, using WireGuard's own PostUp. So the control plane composes a complete configuration for a node it cannot pretend to be -- it knows every public key and holds none of the private ones. Delivered as an ordinary declaration: a package, a file and a service. The host does not know what a private network is and does not learn one. There is a test holding that line, because the moment connectivity needs a new shape in tier 0 is the moment the host stops being small enough to trust. The generated file is written to be read: each peer says why it is there, a peer with no endpoint says why it has none, and the header says not to edit it -- an edit survives until the graph next changes and then vanishes, which is worse than never being applied, because the machine works and then stops and nothing changed that anybody remembers. Fault injection found one weak test. The keepalive rule was asserted only against the hub, whose peer entries happen not to set the field at all, so it was testing an absence rather than the rule. It now checks two direct peers where one is reachable and one is not. |
||
|
jschoubben
|
f563ababa1 |
The mesh keeps a copy of what each node owns
novox/hq 09-the-node-lifecycle asks for this and it was missing: the host reports what it owns and the mesh keeps the last report. A backup, never a source -- nothing decides anything from it, and a node that disagrees with it wins, because the node is the one that can see the machine. Its point is the orphans. A node that loses its state file currently strands whatever it applied: nothing on the machine knows those resources were the mesh's doing, so nothing removes them. With this, a rebuilt node receives both the declaration and the record of what it previously owned. Never reported and reported nothing are kept apart, and that is the whole care in it. A node that applied nothing holds nothing; a node that has never spoken is unknown -- and handing back an empty list for the second would tell a rebuilding node it owns nothing and have it remove whatever it found. The age comes back with the answer rather than being left for the caller to go and find. An answer about a machine is worth much less without one, and this repository has already been bitten by a cache with no age on it. A refusal or a partial failure moves last_seen and nothing else: neither is an account of what the machine holds, and recording one as though it were would tell a rebuilding node to remove what it still has. |
||
|
jschoubben
|
0e116d2d65 |
Bind every key a node may publish
The control queue was bound to enrol and not to report, so every report a node sent was accepted by the broker, matched no binding, and dropped. The publisher saw success and the consumer saw nothing, for an afternoon. The refactor that was meant to bind both never applied -- it left behind a helper nothing called, which compiled and passed vet. The loop is now where the bind is, so there is one place to forget rather than two. |
||
|
jschoubben
|
bbbc860188 |
The control plane declares, and hears back
`declare` sends a node a signed declaration; `serve` now also consumes reports. Signed over the exact bytes published, which is what the node verifies. Anything re-encoding in between would sign one thing and check another, and a difference in key order alone would have a node refuse a declaration that was genuinely the mesh's. Sent to the node's queue directly rather than through the exchange: a declaration is for one node, and routing by name through a shared exchange means a binding per node that nothing removes when a node is retired. Enrolment now issues the node its own broker password, replacing the token's secret, and tells it the broker address, the fingerprint and the signing key -- so a node can reconnect after a restart without a person and a new token, which is what makes disconnection ordinary rather than a crisis. A report is a statement, not a write. What a node says it applied is its own account of its own machine, kept as a copy for recovery rather than as a source. |
||
|
jschoubben
|
46e760fc94 |
The control plane serves, and a node can join
There was no chicken-and-egg to solve. The mesh runs the broker, so it creates the node's account when it issues the token, and the one-time secret is that account's password. A joining node's first connection is already authenticated; enrolment is what it says once it is in. I had been treating this as a decision that needed taking, and it did not. The account is per node and scoped: it may read its own queue, write to the one exchange, and configure nothing else. The patterns are anchored and the node name is constrained to characters that cannot widen them, because a name carrying a dot or a star would silently let that node read everybody's queues. `serve` is the control plane running: one connection, one queue, one consumer. One deliberately -- two consumers on a queue get round-robined and each receives half of what it expects, which has happened on this project before, between a module's daemon and its capability server. Enrolment spends the token first, in the single statement that both finds and marks it, and only then records the key. That order is the order things become irreversible: recording a key for a node whose token turned out to be spent would leave the mesh believing a machine that never had the right to join. Refusals are one message for every reason. The log says which, where an operator can see it; the node is told only that the token cannot be used. Verified in the lab, on a sealed machine, through the whole first-node path. |
||
|
jschoubben
|
afb65c2201 |
A contract test for the token's field names
The host defines the same wire format separately, because it requires nothing present and does not import this. A test on each side asserts the exact field names, so renaming one breaks both immediately rather than at enrolment on a real machine. |
||
|
jschoubben
|
6d3bb18546 |
A node is known by a key it generated
The thing I had been calling blocked for weeks, built in an afternoon once it was pointed out that it was already decided. 08-connectivity says of the overlay keys: each node generates its own keypair, the private half never leaves the machine, the public half is published -- and says outright this IS ADR 0004's "a node holds its own identity". Nobody had applied it to node identity itself. identity now holds the public half of each node's key. Only the public half, which is the property worth having: a copy of this database is a list of who to believe, not a set of credentials, so compromise of a node really is compromise of only that node. Exactly one key is live per node, and re-enrolment revokes the one it replaced in the same transaction -- two live identities is the stolen-laptop case with the replaced machine still believed. Fault injection was worth the time here. Three findings. The unique index was defended by no test at all: sequential enrolment is already safe because the code revokes before inserting, so removing the constraint changed nothing. The constraint only matters when two enrolments race, and there is now a test that runs six at once and fails without it. My injection harness also lied to me. One injection matched nothing, changed no file, and reported NO BITE identically to a real one -- so a test that defends nothing and an injection that does nothing look the same. The harness now checksums the files and says NO-OP when they did not change. And one honest NO BITE left standing: making the key lookup return a zero key for an unknown node does not fail the test, because the signature check refuses it a line later. Two independent mechanisms, not a placebo. 61 tests, none skipped. |
||
|
jschoubben
|
ea6569277d |
A token with all four parts
Given the broker's address and its certificate, mesh-control now issues a token carrying everything ADR 0004 asks for: where to connect, what to expect there, whose signature to believe afterwards, and a one-time right to join. Verified by decoding one and checking the fingerprint against `openssl x509 | sha256sum` -- they match. The fingerprint is derived from the certificate on disk and never configured. A configured pin can drift from the certificate it describes, and a drifted pin is worse than none: every node issued a token during the drift refuses to connect, and the failure looks like an attack rather than a mistake. Computed over DER, which is what a client sees on the wire. Hashing the PEM text instead would mean the same certificate, re-wrapped with different line endings, produced a different pin -- there is a test for exactly that, and one for pointing this at tls.key by mistake, which would otherwise produce a confident pin over the wrong file. Having no broker stays a state rather than a failure: a control plane holds records and a signing key without one. Having half a broker is refused, because a token with an address and nothing to check it against invites a node to trust whatever answers. Fault injection caught the same weak test I wrote earlier in the day -- asking whether something failed rather than why, so deleting the guard changed nothing because it failed one line later anyway. Both are now asserted on the reason. |
||
|
jschoubben
|
7553af6c5a |
The control plane's signing key, and a second context to hold it
Everything is blocked on what a node presents to prove which node it is. This builds the other direction, which is not blocked: what a node believes. identity is the second of the seven contexts. It holds an Ed25519 signing key the control plane generates once, whose public half now travels in every enrolment token. A node believes a declaration because it carries a signature that key made -- pinning only the broker would make the control plane's authority transitive, and since the host applies whatever the link delivers, a compromised broker forging declarations is the whole machine. Establishing the key is idempotent, and it has to be: a second key generated by a restart is a mesh where every node holds the wrong public half, so every declaration is refused by every node with nothing visibly wrong. The guarantee is a partial unique index plus a read-back, not the check before the insert -- six processes racing to establish all agree on one key, and there is a test that runs them. Tokens are now one line of base64 carrying three of their four parts. The missing two are the broker's address and its certificate fingerprint, both step 5 of the bootstrap. The command prints the token and names what is missing rather than emitting something that looks usable. The second context also tests a claim this repository had made and never checked: that a context reaches only its own store. Two databases, two credentials, no setting that reaches both. Running migrate with one stops and names the grant it lacks -- verified, not asserted. Assembling a token needs a node record from one and a key from the other, and neither reads the other's store; the process holding both grants asks each for its part. 45 tests, none skipped. Fault injection found one test whose property is enforced somewhere other than where I injected -- idempotency comes from the database constraint, not from the early return, which is what the code comment already said. |
||
|
jschoubben
|
66768208d2 |
Node records, and the right to join once
The next step after the schema: inventory now holds node records and enrolment tokens, and mesh-control has the commands to work with them. A token is issued for a node record, which is where re-enrolment gets decided -- what an identity binds to is settled when the token is made, not when it is presented, so the machine presenting one does not need to know whether it is joining or returning. What the token guarantees, each with a test confirmed to fail when the behaviour is removed: the secret is 256 random bits, shown once and stored only as a hash; it works exactly once; it stops working when it expires; issuing again for a node invalidates the outstanding one, because two live tokens are two machines able to join as the same node. Redemption is a single statement that finds and spends together, so eight concurrent attempts on one secret produce exactly one winner rather than a race between a check and a write. Refusals are deliberately identical for unknown, spent and expired. Somebody guessing must not learn which guess was a real token that had merely aged out. SHA-256 rather than a password hash, and that is a choice not a shortcut: the secret is high-entropy random, so there is nothing to guess and a slow hash would buy nothing while making every redemption expensive. It stops before what a node receives in exchange. What a machine presents afterwards to prove it is that node is not decided anywhere, and a migration is the most expensive place here to guess. So a token carries one of the four things ADR 0004 requires. The command prints the secret and then says exactly that -- the broker's address, its certificate fingerprint and the control plane's signing identity do not exist yet. Better than emitting something that looks complete and silently cannot be used. |
||
|
jschoubben
|
3b5861282c |
Point at 0006 and 0008 rather than a record of their own
The language and the bundle ordering went into ADR 0006, where the substrate and the control plane already live, and the store-per-context mechanics into 0008, which already decided the rule. Nothing changed but where the reasoning is kept. |
||
|
jschoubben
|
306c4ca13b |
The control plane, as far as identity
Tier 2 exists now. It holds one context of seven, inventory, and does one thing with it: brings its schema up to date. That is step 3 of the substrate bootstrap -- the step the first node cannot get past. Verified against a real PostgreSQL, with the built binary: applied 0001-nodes, reported 'already up to date' on the second run, and the node table is there with the index and the unique constraint the migration asks for. Written in Go, and the image is FROM scratch holding one file. Confirmed by unpacking it. That is the whole argument of ADR 0024: the bundle pins this image by digest and runs it where nothing can check it, so everything in it is something a person has to audit before trusting a first node. Exclusive store ownership is built as a rule about credentials rather than about intentions. There is no mesh-wide connection setting and no way to ask for one -- a context reads MESH_STORE_<ITS OWN NAME> and holds nothing else, so reaching another context's store needs a new variable, which is visible in the declaration that runs it. The migration runner is mostly refusals: an edited migration that already ran, a migration numbered below one that has run, duplicate numbers, misnamed files, empty files. All stop rather than warn, because at the moment any of them is true nobody knows what the database holds. It stops before identity, deliberately. What a node presents to prove who it is has not been decided anywhere, and a migration is the most expensive place in this system to guess. Two tests did not defend what they claimed, and both are fixed rather than removed. One asked only whether Open returned an error, which it did either way -- a bad context name and a missing credential both fail, so deleting the name check changed nothing. The other claimed to prove the migration runs in a transaction, but PostgreSQL already wraps a multi-statement query in one of its own, so it passed with the transaction taken out. What the transaction actually buys is that the schema change and the row recording it commit together, and there is now a test for that which fails when they are split. |