jschoubben
133e10a738
The addresses follow: the control plane holds the ports the node gave, a recorded build holds no address at all, and the two forwarders that had been holding the control plane together are removed. 097's stranded container turned out to be listening on every interface and connected to the mesh's store — by its own old database, which is the only reason nothing was at risk.
3.3 KiB
Diagnosis and resolution — 2026-09-24
One fault in three readers
Each was the same mistake: an address was written down when a port was decided, rather than read where it is used.
- The control plane's own store and broker addresses were whole connection strings, sealed at genesis. Nothing could rewrite them, so moving the store and the broker onto the predecessor's ports left the control plane dialling numbers nothing listened on — twice, and each time the mesh was headless until a forwarder was put in front of the old address by hand.
- Every recorded build carried the registry's address in the image reference. Found by reading rather than by breaking, because the registry had not moved yet.
What it is now
A port that is a node's setting is answered where it is asked. The control plane's own manifest asks for it — one value per context, per address — and the controller fills it from the same place every consumer's binding comes from: what the node was given. An address the mesh does not need to add anything to comes back empty, and the value sealed at genesis stands; there is no second source of truth to disagree.
A recorded build no longer holds an address at all. It names the artifact — its module, its name and its digest — and the address is composed when a declaration is built, from the node that holds the artifact store and the port that node was given. A reference is assembled where it is used.
Verified on the machine, not only in tests
Rolled out in two steps, deliberately: the code first, with the manifest unchanged, so the running control plane composed a declaration it fully understood; then the manifest, composed by the new binary, which filled it in. The control plane came back holding 6852 for all three store contexts, 5679 for the bus it consumes, the broker's management port as the node gave it, and an empty value for the one address the node never moved — exactly the rule. Its own image reference, recorded address-free, was routed back to the node's registry port at compose.
Then both forwarders were stopped, and with neither in place: the mesh reports healthy, a push round-trips, a broker account is minted through the management port, and a build runs. The addresses follow.
Two things the review caught before they could happen
- A control plane older than the manifest it is handed would have passed the unfilled request through as a literal, and the reader would have refused it as "not a port" — headless, by exactly the mechanism this issue is about. The reader now treats an unfilled request as nothing said and says so on the way past; the split rollout above is the belt to that brace.
- On a machine with nothing on the private network yet — a first node, mid-genesis — there is no address for the artifact store at all, and the composed reference would have been refused. The store is now reached by loopback in that one case, which is the only case where the machine composing and the machine holding are the same.
What this does not close
Two references pinned at genesis — the control plane's own image and the builder's — name a single-segment repository with no build behind them, so nothing re-routes them until each is rebuilt through the mesh. The registry has not moved and is not going to; when it does, rebuild both first.