jschoubben
8b974deb42
Three machines across two sites, two of them behind no reachable address, all nine paths open. The mesh computes the graph, delivers it as a declaration, and the nodes bring it up. Every fault below looked like success from inside the mesh: the graph was right, the files were right, the services were up, every node reported it had applied. None was reachable by reasoning. A running interface does not re-read its configuration. A node joins, every existing node's peer list changes, the file is replaced -- and the service is already running, so nothing reloads it. Fixed as declared state rather than a command: the service must reflect the file. A command to restart would be an action, and the link may not carry one. The host refused exactly that, which is how this shape was arrived at. A hub sharing a site with a spoke appeared twice in that spoke's peer list -- once as a direct peer, once as the route of last resort. WireGuard takes one entry per key and refuses the file. The ordinary shape of a small mesh, and in none of the tests written before it ran. Two nodes at one site that neither can be dialled were peered directly. Nobody opens the path, and the direct route is more specific than the hub's, so it wins and blackholes -- this design's own warning arriving in its implementation. They now route through the hub unless one end can be dialled. And Docker sets the FORWARD policy to DROP, so a hub with ip_forward enabled carried nothing between its spokes. The substrate at tier 1 silently breaks the network at tier 2, and nothing in either tier's state says so. The hub inserts its own rule above those chains and removes it on the way down. Two weak tests found by injection along the way: one asserted the keepalive rule only against the hub, whose peer entries happen not to set that field at all, so it tested an absence; the other checked the firewall rules by looking for FORWARD anywhere, which the PostDown line satisfies on its own.
98 lines
4.3 KiB
Go
98 lines
4.3 KiB
Go
// Package link is the control plane's side of the connection nodes hold open.
|
|
//
|
|
// novox/hq ADR 0002: nodes communicate over a message broker, not over HTTP. One exchange, and
|
|
// the control plane is the single consumer behind it — ADR 0006 makes that a property worth
|
|
// having rather than an accident, because two consumers sharing a queue silently split the
|
|
// traffic between them, each receiving half of what it expects. That has happened here before.
|
|
package link
|
|
|
|
// Exchange is where nodes publish everything they have to say.
|
|
const Exchange = "mesh"
|
|
|
|
// ControlQueue is what the control plane consumes. One queue, one consumer.
|
|
const ControlQueue = "control"
|
|
|
|
// Routing keys. A node may publish these; it may not publish anything else, because its broker
|
|
// account is scoped to this exchange and its own queue.
|
|
const (
|
|
KeyEnrol = "enrol"
|
|
KeyReport = "report"
|
|
)
|
|
|
|
// QueueFor is the queue a node consumes from — the only one it may read.
|
|
func QueueFor(node string) string { return "node." + node }
|
|
|
|
// EnrolRequest is what a joining node says.
|
|
//
|
|
// It arrives on a connection the broker has already authenticated, because the account was
|
|
// created when the token was issued and the token's secret is its password. So this message is
|
|
// not how a node gets in — it is what it says once it is in.
|
|
type EnrolRequest struct {
|
|
// Node is what this machine believes it is called. Checked against the token, never trusted.
|
|
Node string `json:"node"`
|
|
|
|
// Secret is the one-time right to join. The account password and this are the same string,
|
|
// which is deliberate: the broker proves somebody holds the token, and this proves the same
|
|
// thing to the control plane without the control plane having to ask the broker who connected.
|
|
Secret string `json:"secret"`
|
|
|
|
// PublicKey is what the mesh will believe from now on. The node generated it; the private
|
|
// half has never left that machine (novox/hq ADR 0004).
|
|
PublicKey []byte `json:"public_key"`
|
|
|
|
// OverlayKey is the public half of this node's key on the private network — a different key
|
|
// from PublicKey, and the mesh only ever sees this half.
|
|
OverlayKey string `json:"overlay_key,omitempty"`
|
|
|
|
// Profile is what this machine can be asked to do. The control plane cannot decide what a
|
|
// node should run without it, so it arrives with enrolment rather than being asked for after.
|
|
Profile map[string]any `json:"profile,omitempty"`
|
|
}
|
|
|
|
// Signed is a declaration and the signature over it.
|
|
//
|
|
// The signature is over Declaration exactly as it will arrive, bytes unchanged — a node verifies
|
|
// what it received rather than what it re-encoded, because any difference in key order or spacing
|
|
// would break a signature over the same meaning.
|
|
type Signed struct {
|
|
Declaration []byte `json:"declaration"`
|
|
Signature []byte `json:"signature"`
|
|
}
|
|
|
|
// Report is what a node states after applying. It states; the owning context writes.
|
|
type Report struct {
|
|
Node string `json:"node"`
|
|
Applied []string `json:"applied,omitempty"`
|
|
Failed map[string]string `json:"failed,omitempty"`
|
|
Refused string `json:"refused,omitempty"`
|
|
}
|
|
|
|
// EnrolReply is what the mesh says back.
|
|
type EnrolReply struct {
|
|
// Accepted says whether the node is now known.
|
|
Accepted bool `json:"accepted"`
|
|
|
|
// Node is the name the mesh has for this machine, which settles any disagreement: the token
|
|
// was issued for a node record, and that record's name wins over what the machine called
|
|
// itself.
|
|
Node string `json:"node,omitempty"`
|
|
|
|
// Queue is where this node listens from now on.
|
|
Queue string `json:"queue,omitempty"`
|
|
|
|
// Password is this node's own broker account from now on, replacing the token's secret.
|
|
// A credential that lives for as long as the node should not be the same string as one that
|
|
// was meant to be used once.
|
|
Password string `json:"password,omitempty"`
|
|
|
|
// Fingerprint and Signer are what the node keeps so it can reconnect and keep verifying
|
|
// without a person and a new token.
|
|
Fingerprint string `json:"fingerprint,omitempty"`
|
|
Signer []byte `json:"signer,omitempty"`
|
|
Broker string `json:"broker,omitempty"`
|
|
|
|
// Refusal says why not, in words for a person. Deliberately the same for every reason a
|
|
// token can fail — unknown, spent, expired — so that guessing learns nothing.
|
|
Refusal string `json:"refusal,omitempty"`
|
|
}
|