novox/mesh-controller
3
0
Fork 0
Code Issues Pull Requests 3 Releases Wiki Activity

No container is given the mesh's names; it resolves them

Browse Source 
novox/hq ADR 0148, step 3. Every container got the whole roster as
--add-host entries at creation and nothing re-read them (issues 109,
135); once the roster was in the digest so that could be caught, one
name moving anywhere replaced every container in the mesh (issue 151).
A container resolves through its machine's resolver, which the resolver
module tells the runtime about once per machine. A module's own hosts
entries stay exactly as declared.

Also brings the resolver tests up to the catalogue as it now is: the
runtime is reloaded (never restarted) and given live-restore, and the
resolver answers by address, not by interface (issue 110).
This commit is contained in:
jschoubben
2026-09-30 14:38:07 +02:00
parent 0c7f42a18a
commit c978aa7d64
5 changed files with 92 additions and 159 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/catalogue/declaration.go
+9 -48
View File
@@ -618,17 +618,15 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
// merging earlier would throw away the files it still needs. // merging earlier would throw away the files it still needs.
resources = append(append([]map[string]any{}, first...), resources...) resources = append(append([]map[string]any{}, first...), resources...)
// Every container is given the mesh's names. Not a choice a module makes: a module that // No container is given the mesh's names (novox/hq ADR 0148). It used to be: every
// listed them would go stale the day a machine joins, and one that did not would be a // container got the whole roster as `--add-host` entries at creation, and a name that
// module whose containers cannot reach anything by name. // moved afterwards was wrong inside it for as long as it ran (issues 109, 135) — and once
// // the roster was made part of a container's identity so that could be caught, one name
// A container that was given names of its own keeps them and gets the mesh's beside them: // moving anywhere replaced every container in the mesh (issue 151). A container resolves a
// the mesh does not know what else a workload needs to reach, and taking something away // mesh name through its machine's resolver at the moment it asks, which the runtime is
// to add something is not what "also" means. // told once per machine, as a file, by the resolver's own module. The names a module
if len(with.Names) > 0 { // declares for itself are its own and stay exactly as written: they are part of what the
resources = withMeshNames(resources, with.Names) // module is, and the mesh does not know what they mean.
}
// What this module may name from inside one of its own files. Gathered once per module // What this module may name from inside one of its own files. Gathered once per module
// rather than per file, because it is a fact about the module. // rather than per file, because it is a fact about the module.
sealed, err := sealedFor(m, r.Needs, with) sealed, err := sealedFor(m, r.Needs, with)
@@ -1482,43 +1480,6 @@ func (r Resolution) servedOnThisMachine(provision string, with Rendering) (map[s
return nil, false, nil return nil, false, nil
} }
// withMeshNames gives every container in a set the mesh's names.
//
// Copied rather than edited in place: these maps come from a module's manifest, and mutating one
// would change what the catalogue holds for every other machine running that module.
//
// A host-network container gets the names too. It was once skipped, on the belief that it "shares
// the machine's hosts file already" — but it does not: `docker run --network host` still gives the
// container its own /etc/hosts (localhost and its own id only), so every `<node>.internal` name the
// mesh wrote for the machine is invisible inside it, and a client that dials one gets EAI_AGAIN. The
// remedy is the same `--add-host` every other container gets — the runtime accepts it with
// `--network host` (verified), and without it a host-network consumer cannot reach a provider by the
// `.internal` address the mesh hands it as `${bound:...:at}`.
func withMeshNames(resources []map[string]any, names map[string]string) []map[string]any {
out := make([]map[string]any, 0, len(resources))
for _, r := range resources {
if r["type"] != "container" {
out = append(out, r)
continue
}
copied := map[string]any{}
for k, v := range r {
copied[k] = v
}
var given []any
if already, ok := copied["hosts"].([]any); ok {
given = append(given, already...)
}
for _, name := range sortedKeys(names) {
given = append(given, name+":"+names[name])
}
copied["hosts"] = given
out = append(out, copied)
}
return out
}
// pinned refuses an image that is not really pinned, on its way to a machine. // pinned refuses an image that is not really pinned, on its way to a machine.
// //
// **Here and not at parse** (novox/hq 04-ISSUES/025). A manifest in a repository names artifacts // **Here and not at parse** (novox/hq 04-ISSUES/025). A manifest in a repository names artifacts
internal/catalogue/names_test.go
+34 -81
View File
@@ -1,6 +1,7 @@
package catalogue package catalogue
import ( import (
"reflect"
"strings" "strings"
"testing" "testing"
) )
@@ -30,36 +31,38 @@ func namesOf(r map[string]any) []string {
return out return out
} }
// A container does not inherit the machine's names, so the mesh gives them to it. // No mesh name is written into a container (novox/hq ADR 0148). It resolves them through its
// machine's resolver at the moment it asks, so a name that moves is answered differently by the
// next lookup, in every container, with nothing recreated.
// //
// It gets its own hosts file holding only its own hostname — every internal name the mesh wrote // Checked the way the record says: the declaration a container gets does not move when the mesh's
// for the machine is invisible to what the machine runs. A database client on one node could not // roster does. A roster with one machine and a roster with three produce the same container, byte
// resolve another node, on a mesh where both names were correct and present on both machines. // for byte, so the digest a host computes from it cannot move either — which is what stopped one
func TestEveryContainerIsGivenTheMeshsNames(t *testing.T) { // name moving from replacing every container in the mesh (issue 151).
got := containersOf(t, Resolution{Node: "laptop", Modules: []Manifest{{ func TestAContainerIsTheSameWhateverTheMeshsRosterSays(t *testing.T) {
Module: "app", module := Manifest{Module: "app", Resources: []map[string]any{{"id": "web", "type": "container",
Resources: []map[string]any{{"id": "web", "type": "container", "name": "web", "name": "web", "image": "registry.example/web@sha256:" + strings.Repeat("a", 64)}}}
"image": "registry.example/web@sha256:" + strings.Repeat("a", 64)}}, one := containersOf(t, Resolution{Node: "laptop", Modules: []Manifest{module}},
}}}, Rendering{Names: map[string]string{ Rendering{Names: map[string]string{"laptop.internal": "10.42.0.2"}})
"anchor.internal": "10.42.0.1", "laptop.internal": "10.42.0.2", three := containersOf(t, Resolution{Node: "laptop", Modules: []Manifest{module}},
}}) Rendering{Names: map[string]string{
if len(got) != 1 { "anchor.internal": "10.42.0.1", "laptop.internal": "10.42.0.2", "git.example.tld": "10.42.0.1",
t.Fatalf("expected one container, got %d", len(got)) }})
if len(one) != 1 || len(three) != 1 {
t.Fatalf("expected one container each, got %d and %d", len(one), len(three))
} }
given := namesOf(got[0]) if given := namesOf(three[0]); len(given) != 0 {
if len(given) != 2 { t.Fatalf("the mesh's names were copied into the container: %v", given)
t.Fatalf("the container was given %d name(s): %v", len(given), given)
} }
if given[0] != "anchor.internal:10.42.0.1" { if !reflect.DeepEqual(one[0], three[0]) {
t.Fatalf("the name is not in the form a runtime writes: %v", given) t.Fatalf("the container moved with the roster:\n%v\n%v", one[0], three[0])
} }
} }
// A container that named its own keeps them and gets the mesh's beside them. // The names a module declares for itself are its own: part of what the module is, kept exactly as
// // written, and the mesh does not know what they mean. They are the one thing in a container's
// The mesh does not know what else a workload needs to reach, and taking something away in order // hosts that does move its identity, because they do not move when the mesh's roster does.
// to add something is not what "also" means. func TestAContainersOwnNamesAreKeptAsWritten(t *testing.T) {
func TestAContainersOwnNamesAreKept(t *testing.T) {
got := containersOf(t, Resolution{Node: "laptop", Modules: []Manifest{{ got := containersOf(t, Resolution{Node: "laptop", Modules: []Manifest{{
Module: "app", Module: "app",
Resources: []map[string]any{{"id": "web", "type": "container", "name": "web", Resources: []map[string]any{{"id": "web", "type": "container", "name": "web",
@@ -68,62 +71,15 @@ func TestAContainersOwnNamesAreKept(t *testing.T) {
}}}, Rendering{Names: map[string]string{"anchor.internal": "10.42.0.1"}}) }}}, Rendering{Names: map[string]string{"anchor.internal": "10.42.0.1"}})
given := namesOf(got[0]) given := namesOf(got[0])
if len(given) != 2 || given[0] != "something.else:203.0.113.9" { if len(given) != 1 || given[0] != "something.else:203.0.113.9" {
t.Fatalf("the container's own names were lost: %v", given) t.Fatalf("the container's own names were not kept as written: %v", given)
} }
} }
// A container on the machine's own network gets the names too — it does NOT share the machine's // No resource is given a `hosts` key it did not declare. A file or a service carrying one is a
// hosts file. `docker run --network host` still gives the container its own /etc/hosts (localhost // declaration the host refuses outright — it takes no unknown field — so an invented key breaks
// and its own id only), so every `<node>.internal` name the mesh wrote is invisible inside it, and a // the whole machine rather than one resource.
// client that dials one gets EAI_AGAIN. It gets the same `--add-host` entries every other container func TestNothingIsGivenNamesItDidNotDeclare(t *testing.T) {
// gets (the runtime accepts them with `--network host`), so a host-network consumer can reach a
// provider by the `.internal` address the mesh hands it.
func TestAContainerOnTheMachinesNetworkIsGivenTheNamesToo(t *testing.T) {
got := containersOf(t, Resolution{Node: "anchor", Modules: []Manifest{{
Module: "control",
Resources: []map[string]any{{"id": "c", "type": "container", "name": "c",
"image": "registry.example/c@sha256:" + strings.Repeat("a", 64), "network": "host"}},
}}}, Rendering{Names: map[string]string{"anchor.internal": "10.42.0.1"}})
given := namesOf(got[0])
if len(given) != 1 || given[0] != "anchor.internal:10.42.0.1" {
t.Fatalf("a host-networked container was not given the mesh's names: %v", got[0])
}
}
// A mesh with no private network gives nothing, rather than a name with no address behind it.
func TestAMeshWithNoNamesGivesNone(t *testing.T) {
got := containersOf(t, Resolution{Node: "alone", Modules: []Manifest{{
Module: "app",
Resources: []map[string]any{{"id": "web", "type": "container", "name": "web",
"image": "registry.example/web@sha256:" + strings.Repeat("a", 64)}},
}}}, Rendering{})
if len(namesOf(got[0])) != 0 {
t.Fatalf("names were invented for a mesh that has none: %v", got[0])
}
}
// The catalogue's copy is not edited: these maps come from a manifest, and mutating one would
// change what every other machine running that module is given.
func TestGivingNamesDoesNotChangeTheCatalogue(t *testing.T) {
held := map[string]any{"id": "web", "type": "container", "name": "web",
"image": "registry.example/web@sha256:" + strings.Repeat("a", 64)}
module := Manifest{Module: "app", Resources: []map[string]any{held}}
for _, node := range []string{"one", "two"} {
containersOf(t, Resolution{Node: node, Modules: []Manifest{module}},
Rendering{Names: map[string]string{"anchor.internal": "10.42.0.1"}})
}
if _, changed := held["hosts"]; changed {
t.Fatal("the manifest the catalogue holds was edited, so every machine now carries this")
}
}
// Only containers. A file or a service given a `hosts` key is a declaration the host refuses
// outright — it takes no unknown field — so getting this wrong breaks the whole machine rather
// than one resource, and breaks it for something that was never about names.
func TestNothingButAContainerIsGivenNames(t *testing.T) {
out, err := Resolution{Node: "laptop", Modules: []Manifest{{ out, err := Resolution{Node: "laptop", Modules: []Manifest{{
Module: "app", Module: "app",
Resources: []map[string]any{ Resources: []map[string]any{
@@ -137,11 +93,8 @@ func TestNothingButAContainerIsGivenNames(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
for _, r := range out { for _, r := range out {
if r["type"] == "container" {
continue
}
if _, given := r["hosts"]; given { if _, given := r["hosts"]; given {
t.Fatalf("a %v was given names, which the host will refuse: %v", r["type"], r) t.Fatalf("a %v was given names it never declared: %v", r["type"], r)
} }
} }
} }
internal/catalogue/port_into.go
+1 -2
View File
@@ -98,8 +98,7 @@ func portInto(resource map[string]any, module string, listens []Listening, with
// **A fresh map, and only when something changes.** This map came out of the module's // **A fresh map, and only when something changes.** This map came out of the module's
// manifest and the resource around it is a shallow copy, so filling a value in place would // manifest and the resource around it is a shallow copy, so filling a value in place would
// change what the catalogue holds for every other machine running the module — the trap // change what the catalogue holds for every other machine running the module.
// withMeshNames is written to avoid, one field along.
var filled map[string]any var filled map[string]any
for _, key := range named { for _, key := range named {
written, ok := env[key].(string) written, ok := env[key].(string)
internal/catalogue/resolver_manifests_test.go
+32 -8
View File
@@ -48,7 +48,7 @@ func TestTheResolverForwardsToFixedUpstreamsAndNeverReadsResolvConf(t *testing.T
} }
for _, want := range []string{ for _, want := range []string{
"\nno-resolv\n", "\nserver=1.1.1.1\n", "\nserver=8.8.8.8\n", "\nno-resolv\n", "\nserver=1.1.1.1\n", "\nserver=8.8.8.8\n",
"\nlisten-address=127.0.0.1\n", "\ninterface=mesh0\n", "\nbind-dynamic\n", "\nlisten-address=127.0.0.1\n", "\nlisten-address=${machine:address}\n", "\nbind-dynamic\n",
"\ndomain-needed\n", "\nbogus-priv\n", "\ndomain-needed\n", "\nbogus-priv\n",
"\nconf-file=" + m.Facts["node-zones"].Path + "\n", "\nconf-file=" + m.Facts["node-zones"].Path + "\n",
} { } {
@@ -56,6 +56,14 @@ func TestTheResolverForwardsToFixedUpstreamsAndNeverReadsResolvConf(t *testing.T
t.Errorf("the resolver's configuration lacks %q:\n%s", strings.TrimSpace(want), config) t.Errorf("the resolver's configuration lacks %q:\n%s", strings.TrimSpace(want), config)
} }
} }
// By address and never by interface: dnsmasq admits a query by the interface it arrives on
// when told one, and a container's query to the private address arrives on the runtime's
// bridge — `interface=mesh0` dropped every such query, silently (novox/hq issue 110).
for _, line := range strings.Split(config, "\n") {
if strings.HasPrefix(line, "interface=") {
t.Errorf("the resolver answers by interface, so a container's query on a bridge is dropped: %s", line)
}
}
// Not .53 or .54, which systemd-resolved holds; and not .55 any more, which was a convention // Not .53 or .54, which systemd-resolved holds; and not .55 any more, which was a convention
// beside the one every machine already followed — the predecessor's resolv.conf says .1. // beside the one every machine already followed — the predecessor's resolv.conf says .1.
for _, taken := range []string{"127.0.0.53", "127.0.0.54", "127.0.0.55"} { for _, taken := range []string{"127.0.0.53", "127.0.0.54", "127.0.0.55"} {
@@ -137,23 +145,39 @@ func TestTheResolverAndWhatAsksItComposeOnOneMachine(t *testing.T) {
t.Errorf("the daemon does not restart on its configuration and the machines file both: %v", service["restart-on"]) t.Errorf("the daemon does not restart on its configuration and the machines file both: %v", service["restart-on"])
} }
// The runtime's own file, written into (novox/hq ADR 0102) with the one key this module states. // The runtime's own file, written into (novox/hq ADR 0102) with the keys this module states:
// where containers resolve, and that a restart keeps them running — because the runtime reads
// `dns` only when it starts, and the one restart that needs is the operator's (issue 110).
runtime := ids["dnsmasq.runtime-dns"] runtime := ids["dnsmasq.runtime-dns"]
if runtime == nil || runtime["path"] != "/etc/docker/daemon.json" || runtime["into"] != "json" { if runtime == nil || runtime["path"] != "/etc/docker/daemon.json" || runtime["into"] != "json" {
t.Fatalf("the runtime's dns is not written into its file: %v", runtime) t.Fatalf("the runtime's dns is not written into its file: %v", runtime)
} }
var keys map[string][]string var keys map[string]any
if err := json.Unmarshal([]byte(runtime["content"].(string)), &keys); err != nil { if err := json.Unmarshal([]byte(runtime["content"].(string)), &keys); err != nil {
t.Fatalf("the runtime's keys are not JSON: %v", err) t.Fatalf("the runtime's keys are not JSON: %v", err)
} }
if len(keys) != 1 || len(keys["dns"]) != 1 || keys["dns"][0] != "10.42.0.1" { dns, _ := keys["dns"].([]any)
t.Errorf("the runtime is pointed at %v; containers resolve at this machine's own private-network address, and nothing else is written", keys) if len(keys) != 2 || len(dns) != 1 || dns[0] != "10.42.0.1" || keys["live-restore"] != true {
t.Errorf("the runtime is given %v; containers resolve at this machine's own private-network address, a restart keeps them, and nothing else is written", keys)
} }
// The runtime is reloaded when that file changes, and never restarted: a restart stops every
// container on the machine (ADR 0102), and a reload is what turns live-restore on.
var reloaded bool
for _, r := range out { for _, r := range out {
if r["type"] == "service" && r["unit"] == "docker.service" && r["id"] != "" && if r["type"] != "service" || r["unit"] != "docker.service" {
strings.HasPrefix(r["id"].(string), "dnsmasq.") { continue
t.Errorf("the resolver orders the runtime restarted or reloaded, which stops every container (ADR 0102) or does nothing for dns: %v", r)
} }
if _, restarts := r["restart-on"]; restarts {
t.Errorf("the resolver orders the runtime restarted, which stops every container (ADR 0102): %v", r)
}
for _, on := range asStrings(r["reload-on"]) {
if on == "dnsmasq.runtime-dns" {
reloaded = true
}
}
}
if !reloaded {
t.Errorf("the runtime is not reloaded when its file changes, so live-restore never takes effect")
} }
resolv := ids["resolv-conf.resolv"] resolv := ids["resolv-conf.resolv"]
internal/catalogue/routing_test.go
+16 -20
View File
@@ -198,37 +198,33 @@ func TestTheApexLabelComposesToTheBarePrivateAddress(t *testing.T) {
} }
} }
// novox/hq ADR 0066 propagate, by ADR 0148's means: a granted route name is published into the
// machine's roster mapped to the node that serves it, beside the `<node>.internal` names, and the
// machine's resolver answers it to every container. Nothing is written into the container itself —
// a routed name that moved would otherwise be wrong inside every container until each was recreated.
func TestARoutedNameResolvesToTheServingNode(t *testing.T) { func TestARoutedNameResolvesToTheServingNode(t *testing.T) {
// novox/hq ADR 0066 propagate: a granted route name is published into internal resolution, names := map[string]string{
// mapped to the node that serves it, alongside the `<node>.internal` names — so every "anchor.internal": "10.42.0.1",
// container, and an in-mesh ACME validator, resolves a routed name to the proxy that serves it. "git.example.tld": "10.42.0.1",
// The names map is what withMeshNames writes into every container as `--add-host`; a route name }
// mapped to the serving node's address rides the same mechanism.
got := containersOf(t, Resolution{Node: "anchor", Modules: []Manifest{{ got := containersOf(t, Resolution{Node: "anchor", Modules: []Manifest{{
Module: "app", Module: "app",
Resources: []map[string]any{{"id": "web", "type": "container", "name": "web", Resources: []map[string]any{{"id": "web", "type": "container", "name": "web",
"image": "registry.example/web@sha256:" + strings.Repeat("a", 64)}}, "image": "registry.example/web@sha256:" + strings.Repeat("a", 64)}},
}}}, Rendering{Names: map[string]string{ }}}, Rendering{Names: names})
"anchor.internal": "10.42.0.1",
"git.example.tld": "10.42.0.1",
}})
if len(got) != 1 { if len(got) != 1 {
t.Fatalf("expected one container, got %d", len(got)) t.Fatalf("expected one container, got %d", len(got))
} }
given := namesOf(got[0]) if given := namesOf(got[0]); len(given) != 0 {
var sawNode, sawRoute bool t.Fatalf("the routed name was copied into the container, where it would go stale: %v", given)
for _, h := range given { }
if h == "anchor.internal:10.42.0.1" { var sawRoute bool
sawNode = true for _, e := range entriesFrom(names, nil, "internal") {
} if e.Name == "git.example.tld" && e.Address == "10.42.0.1" {
if h == "git.example.tld:10.42.0.1" {
sawRoute = true sawRoute = true
} }
} }
if !sawNode {
t.Fatalf("the container lost the mesh's node names: %v", given)
}
if !sawRoute { if !sawRoute {
t.Fatalf("the routed name was not published to the serving node: %v", given) t.Fatalf("the routed name is not in the roster the machine's resolver answers from")
} }
} }