Networking is a module, and a domain module is how you avoid choosing

Connectivity was code beside the module system doing the module system's
job: every machine with an address was on the private network and there
was no way to keep one off.

A manifest can now say its resources are computed by the control plane,
which is what a peer list needs — it is derived from every machine at
once, so nothing could be written in advance. The network is a module
from there on: assigned, resolved, settled, and absent from a machine
nobody gave it to.

Three modules rather than one, because WireGuard is one VPN of several:

  mesh-wireguard   provides private-network, mesh-addressing
                   claims the-private-network, one per node
  mesh-names       provides name-resolution, requires mesh-addressing
  networking       requires both, and ships no files of its own

The last is the point. Most people want the network up and do not want
to choose a VPN, so `assign networking` takes the only answer to each
requirement silently. The day the catalogue holds a second one there are
two answers, the resolver refuses and names them, and choosing is
assigning the one you want. No flavor field, nothing to configure.

Names left the WireGuard declaration for their own module. They would be
identical over a different private network, and bundling them made one
module out of two things.

Three faults the walk found:

- choosing tailscale still installed WireGuard, dragged back in by the
  names needing the mesh's own addresses. Caught now by a claim: running
  two VPNs is fine, being *the* mesh network is singular.
- a requirement wanted by two modules was reported twice, identically.
- "this mesh has no hub" was reported when the real cause was that a
  node could not be resolved at all. It now names the node and the why.

And a test that asserts the manifests actually shipped, after the claim
went missing from the real one while every test stayed green.
This commit is contained in:
2026-08-29 23:19:32 +02:00
parent 65ade756f2
commit 44d134ba25
12 changed files with 940 additions and 53 deletions
+173 -31
View File
@@ -181,9 +181,48 @@ func migrate(ctx context.Context) error {
fmt.Printf("%s: already up to date — %d migration(s)\n", c.name, len(applied)) fmt.Printf("%s: already up to date — %d migration(s)\n", c.name, len(applied))
} }
} }
// The modules the control plane ships with itself. Recorded here rather than by hand, because
// a mesh whose own private network is missing from the catalogue would have nothing to assign
// and no way to say why.
inv, err := openInventory(ctx)
if err != nil {
return err
}
defer inv.Close()
for _, m := range provided {
if err := inv.Provide(ctx, m); err != nil {
return err
}
fmt.Printf("provided %s\n", m.Module)
}
return nil return nil
} }
// provided is what comes with the control plane rather than from a repository.
//
// WireGuard, the names, and the domain module over both. The first two are here because the code
// that works out their files is here:
// a peer list is derived from every machine at once, so it cannot be written in a manifest, and
// whatever computes it has to live wherever the whole picture is.
//
// **It is a module in every other respect** — assigned, unassigned, resolved, settled, and absent
// from a machine nobody gave it to.
func providedModules() []catalogue.Manifest {
var out []catalogue.Manifest
for _, raw := range []map[string]any{
overlay.Manifest(), overlay.NamesManifest(), overlay.DomainManifest(),
} {
var m catalogue.Manifest
b, _ := json.Marshal(raw)
_ = json.Unmarshal(b, &m)
out = append(out, m)
}
return out
}
var provided = providedModules()
// openInventory connects and waits, the way every command that touches it needs to. // openInventory connects and waits, the way every command that touches it needs to.
func openInventory(ctx context.Context) (*inventory.Inventory, error) { func openInventory(ctx context.Context) (*inventory.Inventory, error) {
inv, err := inventory.Open(ctx) inv, err := inventory.Open(ctx)
@@ -572,22 +611,120 @@ func overlayPlace(ctx context.Context, inv *inventory.Inventory, args []string)
return nil return nil
} }
// graph reads every node's place and computes the network. Every node at once, which is the whole // network builds the private network over the machines that resolved the module for it.
// reason this is the control plane's work. //
func graph(ctx context.Context, inv *inventory.Inventory) ([]overlay.Node, overlay.Graph, error) { // Not over every node the mesh knows. **A machine is on the private network because it was given
// the module**, and one that was not is absent from every peer list and from the names — which is
// the only thing "not on the network" can mean. Until this, having an address was enough, and
// there was no way to keep a machine off.
//
// Every node at once, which is the whole reason this is the control plane's work: a peer list is
// derived from all the others, so no node could compute its own.
func network(ctx context.Context, inv *inventory.Inventory, on map[string]bool,
refused map[string]string) (*overlay.Generator, error) {
places, err := inv.Overlays(ctx) places, err := inv.Overlays(ctx)
if err != nil { if err != nil {
return nil, nil, err return nil, err
} }
nodes := make([]overlay.Node, 0, len(places)) nodes := make([]overlay.Node, 0, len(places))
for _, p := range places { for _, p := range places {
if !on[p.Name] {
continue
}
nodes = append(nodes, overlay.Node{ nodes = append(nodes, overlay.Node{
Name: p.Name, Key: p.Key, Endpoint: p.Endpoint, Name: p.Name, Key: p.Key, Endpoint: p.Endpoint,
Site: p.Site, Hub: p.Hub, Address: p.Address, Site: p.Site, Hub: p.Hub, Address: p.Address,
}) })
} }
computed, err := overlay.Compute(nodes, overlayCIDR()) if len(nodes) == 0 {
return nodes, computed, err // Nobody was given it. An empty network is a legitimate mesh, not a broken one, so this
// answers rather than refusing -- Compute would refuse for want of a hub, and reporting
// "no hub" to somebody who never asked for a network would be a lie about the cause.
return overlay.Empty(), nil
}
g, err := overlay.From(nodes, overlayCIDR(), "")
if err != nil && len(refused) > 0 {
// The network is missing something, and some machines could not be resolved at all. Those
// are almost always the same fact: a node that does not resolve contributes nothing, so
// reporting "no hub" would name a consequence and hide the cause.
var who []string
for name, why := range refused {
who = append(who, fmt.Sprintf(" %s: %s", name, why))
}
sort.Strings(who)
return nil, fmt.Errorf("%w\n\nand %d node(s) could not be resolved at all, which is "+
"probably why:\n%s", err, len(refused), strings.Join(who, "\n"))
}
return g, err
}
// graph is the whole mesh's network, for showing it.
func graph(ctx context.Context, inv *inventory.Inventory) ([]overlay.Node, overlay.Graph, error) {
on, refused, err := whoResolves(ctx, inv, overlay.Requirement)
if err != nil {
return nil, nil, err
}
g, err := network(ctx, inv, on, refused)
if err != nil {
return nil, nil, err
}
return g.Nodes(), g.Graph(), nil
}
// whoResolves is the machines whose resolution answers a requirement, and why the others did not.
//
// By what a module **provides**, not by its name. WireGuard is one way to have a private network
// and there could be others, so a machine is on the network because something it runs provides
// one — asking for a particular module by name would be the mistake this whole mechanism exists
// to avoid.
//
// Resolved rather than read from the assignment table, because a module can arrive by being
// required by something else, and a machine that needs the private network to do its job is on it
// for the same reason as one that was handed it directly.
func whoResolves(ctx context.Context, inv *inventory.Inventory, requirement string) (
map[string]bool, map[string]string, error) {
nodes, err := inv.Nodes(ctx)
if err != nil {
return nil, nil, err
}
on := map[string]bool{}
// Why a node could not be resolved, kept rather than raised: one broken node must not stop
// the rest being described, and whoever is rendering that node will raise it themselves.
refused := map[string]string{}
for _, n := range nodes {
plan, _, err := planFor(ctx, inv, n.Name)
if err != nil {
refused[n.Name] = err.Error()
continue
}
for _, m := range plan.Modules {
for _, offered := range m.Offers() {
if offered == requirement {
on[n.Name] = true
}
}
}
}
return on, refused, nil
}
// rendering is everything a declaration needs, computed over the whole mesh.
func generators(ctx context.Context, inv *inventory.Inventory) (
map[string]catalogue.Generator, error) {
on, refused, err := whoResolves(ctx, inv, overlay.Addressing)
if err != nil {
return nil, err
}
net, err := network(ctx, inv, on, refused)
if err != nil {
return nil, err
}
// Both generators see the same machines: the ones on the private network. Names for a machine
// that is not on it would resolve to addresses it cannot reach, which is worse than no names.
return map[string]catalogue.Generator{
overlay.Name: net,
overlay.Names: overlay.NamesFor(net.Nodes()),
}, nil
} }
func overlayShow(ctx context.Context, inv *inventory.Inventory) error { func overlayShow(ctx context.Context, inv *inventory.Inventory) error {
@@ -596,7 +733,11 @@ func overlayShow(ctx context.Context, inv *inventory.Inventory) error {
return err return err
} }
if len(nodes) == 0 { if len(nodes) == 0 {
fmt.Println("this mesh has no nodes") // Not "this mesh has no nodes", which it said until the network became a module and was
// then a lie about the cause: a mesh can have every node it will ever have and nobody on
// the private network, because nobody asked for one.
fmt.Printf("nobody is on the private network — assign %s to put a machine on it\n",
overlay.Name)
return nil return nil
} }
@@ -926,7 +1067,11 @@ func planCommand(ctx context.Context, args []string) error {
for _, c := range plan.Claims { for _, c := range plan.Claims {
fmt.Printf(" holds %s, one per %s\n", c.Claim, c.Scope) fmt.Printf(" holds %s, one per %s\n", c.Claim, c.Scope)
} }
resources, err := plan.Declaration(settings) gens, err := generators(ctx, inv)
if err != nil {
return err
}
resources, err := plan.Declaration(catalogue.Rendering{Settings: settings, Generators: gens})
if err != nil { if err != nil {
return err return err
} }
@@ -971,7 +1116,14 @@ func pushCommand(ctx context.Context, args []string) error {
} }
defer ident.Close() defer ident.Close()
nodes, computed, err := graph(ctx, inv) // Every node, not only the ones on the private network. A machine that was never given the
// network module still takes modules, and iterating the network here is what used to make
// "on the network" and "managed" the same thing.
nodes, err := inv.Nodes(ctx)
if err != nil {
return err
}
gens, err := generators(ctx, inv)
if err != nil { if err != nil {
return err return err
} }
@@ -986,7 +1138,7 @@ func pushCommand(ctx context.Context, args []string) error {
// refused on the fourth would leave the mesh in a state nobody asked for, and the fourth is // refused on the fourth would leave the mesh in a state nobody asked for, and the fourth is
// exactly where a claim collision shows up. // exactly where a claim collision shows up.
type ready struct { type ready struct {
node overlay.Node node string
resources []map[string]any resources []map[string]any
} }
var sending []ready var sending []ready
@@ -996,34 +1148,24 @@ func pushCommand(ctx context.Context, args []string) error {
if len(args) == 1 && n.Name != args[0] { if len(args) == 1 && n.Name != args[0] {
continue continue
} }
peers, onOverlay := computed[n.Name]
if !onOverlay {
fmt.Printf("%s is not on the overlay yet — skipped\n", n.Name)
continue
}
declaration, err := overlay.Declaration(n, peers, nodes, "")
if err != nil {
return err
}
var resources struct {
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(declaration, &resources); err != nil {
return err
}
plan, settings, err := planFor(ctx, inv, n.Name) plan, settings, err := planFor(ctx, inv, n.Name)
if err != nil { if err != nil {
refusals = append(refusals, fmt.Sprintf("%s:\n%v", n.Name, err)) refusals = append(refusals, fmt.Sprintf("%s:\n%v", n.Name, err))
continue continue
} }
fromModules, err := plan.Declaration(settings) // The private network is in here with everything else. It used to be composed separately
// and prepended, which meant every machine with an address was on it and no machine could
// be kept off. It is a module now, so it arrives the way a module does.
resources, err := plan.Declaration(catalogue.Rendering{Settings: settings, Generators: gens})
if err != nil { if err != nil {
refusals = append(refusals, fmt.Sprintf("%s:\n%v", n.Name, err)) refusals = append(refusals, fmt.Sprintf("%s:\n%v", n.Name, err))
continue continue
} }
sending = append(sending, ready{n, append(resources.Resources, fromModules...)}) if len(resources) == 0 {
fmt.Printf("%s is assigned nothing — skipped\n", n.Name)
continue
}
sending = append(sending, ready{n.Name, resources})
} }
if len(refusals) > 0 { if len(refusals) > 0 {
@@ -1036,10 +1178,10 @@ func pushCommand(ctx context.Context, args []string) error {
if err != nil { if err != nil {
return err return err
} }
if err := link.Declare(ctx, server.Channel(), ident, s.node.Name, body, 15*time.Second); err != nil { if err := link.Declare(ctx, server.Channel(), ident, s.node, body, 15*time.Second); err != nil {
return err return err
} }
fmt.Printf("sent %s %d resource(s)\n", s.node.Name, len(s.resources)) fmt.Printf("sent %s %d resource(s)\n", s.node, len(s.resources))
} }
fmt.Printf("\n%d node(s) told\n", len(sending)) fmt.Printf("\n%d node(s) told\n", len(sending))
return nil return nil
+158
View File
@@ -0,0 +1,158 @@
package catalogue
import (
"strings"
"testing"
)
// A module whose files cannot be written in advance.
//
// The mesh's private network is the case: a machine's peer list is derived from every other
// machine, so it differs on each one and changes when any of them changes. What matters in these
// tests is not that it works, but that being computed changes *nothing else* about being a
// module — it is assigned, resolved, settled and absent when nobody asked for it.
type fake struct {
on map[string]bool
asked []string
err error
}
func (f *fake) Resources(node string) ([]map[string]any, bool, error) {
f.asked = append(f.asked, node)
if f.err != nil {
return nil, false, f.err
}
if !f.on[node] {
return nil, false, nil
}
return []map[string]any{{"id": "peers", "type": "file", "path": "/etc/x.conf",
"merge": MergeJSON, "content": `{"peer":"` + node + `"}`}}, true, nil
}
func computedShelf() map[string]Manifest {
return shelf(Manifest{Module: "mesh-network", Computed: "mesh-network",
Provides: []string{"private-network"}})
}
func TestAComputedModuleIsAskedAboutTheNodeItIsFor(t *testing.T) {
// The generator gets a node name, not a plan. Everything it needs is the whole mesh, which
// it was built with — this is the one thing a node could never work out for itself.
got, err := Resolve(computedShelf(), []string{"mesh-network"}, workstation(), nil)
if err != nil {
t.Fatal(err)
}
gen := &fake{on: map[string]bool{"workstation": true}}
out, err := got.Declaration(Rendering{Generators: map[string]Generator{"mesh-network": gen}})
if err != nil {
t.Fatal(err)
}
if len(gen.asked) != 1 || gen.asked[0] != "workstation" {
t.Fatalf("asked about %v, wanted workstation once", gen.asked)
}
if len(out) != 1 || !strings.Contains(out[0]["content"].(string), `"peer": "workstation"`) {
t.Fatalf("got %v", out)
}
}
func TestAMachineNobodyGaveItGetsNothing(t *testing.T) {
// The whole point of making the network a module. Before this, every machine with an address
// was on the private network and there was no way to say one should stay off.
got, err := Resolve(computedShelf(), nil, workstation(), nil)
if err != nil {
t.Fatal(err)
}
gen := &fake{on: map[string]bool{"workstation": true}}
out, err := got.Declaration(Rendering{Generators: map[string]Generator{"mesh-network": gen}})
if err != nil {
t.Fatal(err)
}
if len(out) != 0 {
t.Fatalf("a machine that was assigned nothing got %d resource(s)", len(out))
}
if len(gen.asked) != 0 {
t.Fatalf("the generator was asked about %v, and nobody had asked for it", gen.asked)
}
}
func TestAssignedAndNotYetPartOfItIsNotAFailure(t *testing.T) {
// A machine given the network module before it has a place on it. Brief and ordinary — the
// answer is "nothing yet", and treating it as an error would make an ordering a fault.
got, _ := Resolve(computedShelf(), []string{"mesh-network"}, workstation(), nil)
gen := &fake{on: map[string]bool{}}
out, err := got.Declaration(Rendering{Generators: map[string]Generator{"mesh-network": gen}})
if err != nil {
t.Fatalf("refused a node that is not on the network yet: %v", err)
}
if len(out) != 0 {
t.Fatalf("got %v", out)
}
}
func TestAGeneratorThisControlPlaneDoesNotHaveIsRefused(t *testing.T) {
// Sending a machine a module with no files would look like it worked. Named in the message,
// because the only fix is a control plane that has it.
got, _ := Resolve(shelf(Manifest{Module: "weather", Computed: "the-weather"}),
[]string{"weather"}, workstation(), nil)
_, err := got.Declaration(Rendering{Generators: map[string]Generator{"mesh-network": &fake{}}})
if err == nil {
t.Fatal("a module computed by nothing was accepted")
}
if !strings.Contains(err.Error(), "the-weather") {
t.Fatalf("the refusal does not name what is missing: %v", err)
}
}
func TestAModuleIsEitherWrittenOrComputedNotBoth(t *testing.T) {
// Otherwise nobody could say where a given file on a machine came from.
_, err := ParseManifest([]byte(`{"module":"mesh-network","version":"1",
"computed":"mesh-network",
"resources":[{"id":"a","type":"package","package":"wireguard-tools"}]}`))
if err == nil {
t.Fatal("a module that both ships files and has them computed was accepted")
}
if !strings.Contains(err.Error(), "one or the other") {
t.Fatalf("unhelpful refusal: %v", err)
}
}
func TestSettingsApplyToAComputedModuleToo(t *testing.T) {
// Being computed is about where the files come from, not about whether they are configurable.
// A line drawn there would be arbitrary and nobody could predict it.
got, _ := Resolve(computedShelf(), []string{"mesh-network"}, workstation(), nil)
gen := &fake{on: map[string]bool{"workstation": true}}
out, err := got.Declaration(Rendering{
Generators: map[string]Generator{"mesh-network": gen},
Settings: SettingsBy{"mesh-network": {{From: "the mesh",
Values: map[string]any{"keepalive": 25}}}},
})
if err != nil {
t.Fatal(err)
}
content := out[0]["content"].(string)
if !strings.Contains(content, `"keepalive": 25`) {
t.Fatalf("the setting did not reach a computed file: %s", content)
}
if !strings.Contains(content, `"peer": "workstation"`) {
t.Fatalf("the setting replaced what the generator computed: %s", content)
}
}
func TestTwoWaysToBeOnAPrivateNetworkRefuseAndNameBoth(t *testing.T) {
// WireGuard is one way. The refusing rule is what makes a second one safe to add: assigning
// both is caught rather than producing a machine on two networks that each half-work.
_, err := Resolve(shelf(
Manifest{Module: "mesh-network", Computed: "mesh-network",
Provides: []string{"private-network"}},
Manifest{Module: "tailscale", Provides: []string{"private-network"}},
Manifest{Module: "backups", Requires: []string{"private-network"}},
), []string{"backups"}, workstation(), nil)
if err == nil {
t.Fatal("two answers to one requirement were taken silently")
}
for _, want := range []string{"mesh-network", "tailscale", "private-network"} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("the refusal does not name %s: %v", want, err)
}
}
}
+129
View File
@@ -0,0 +1,129 @@
package catalogue
import (
"strings"
"testing"
)
// A domain module is a module with requirements and no files of its own.
//
// Most people want the network working and do not want to choose a VPN. Some want a particular
// one. Both are the same mechanism: assigning `networking` takes the only answer to each of its
// requirements silently, and the day there are two answers the resolver refuses and names them,
// so choosing is assigning the one you want. There is no flavor field and nothing to configure.
func networkingShelf(extra ...Manifest) map[string]Manifest {
base := []Manifest{
{Module: "networking", Requires: []string{"private-network", "name-resolution"}},
{Module: "mesh-wireguard", Computed: "mesh-wireguard",
Provides: []string{"private-network", "mesh-addressing"},
Claims: []Claim{{Name: "the-private-network", Scope: ScopeNode}}},
{Module: "mesh-names", Computed: "mesh-names",
Provides: []string{"name-resolution"}, Requires: []string{"mesh-addressing"}},
}
return shelf(append(base, extra...)...)
}
func TestOneWordBringsUpTheNetwork(t *testing.T) {
// The case that has to stay easy. Nothing is asked, because with one answer to each
// requirement there was never a question.
got, err := Resolve(networkingShelf(), []string{"networking"}, workstation(), nil)
if err != nil {
t.Fatal(err)
}
have := strings.Join(names(got), " ")
for _, want := range []string{"networking", "mesh-wireguard", "mesh-names"} {
if !strings.Contains(have, want) {
t.Fatalf("assigning networking did not bring in %s: %s", want, have)
}
}
}
func TestASecondVPNTurnsItIntoAChoice(t *testing.T) {
// And the choice is offered rather than made. A default here would be the flavor field coming
// back under another name.
_, err := Resolve(networkingShelf(
Manifest{Module: "tailscale", Provides: []string{"private-network"},
Claims: []Claim{{Name: "the-private-network", Scope: ScopeNode}}},
), []string{"networking"}, workstation(), nil)
if err == nil {
t.Fatal("two VPNs and one was picked silently")
}
for _, want := range []string{"mesh-wireguard", "tailscale"} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("the refusal does not name %s: %v", want, err)
}
}
}
func TestChoosingIsAssigning(t *testing.T) {
// No second verb. Assigning the one you want answers the requirement, and the bundle takes it.
got, err := Resolve(networkingShelf(
Manifest{Module: "tailscale",
Provides: []string{"private-network", "name-resolution"},
Claims: []Claim{{Name: "the-private-network", Scope: ScopeNode}}},
), []string{"networking", "tailscale"}, workstation(), nil)
if err != nil {
t.Fatal(err)
}
have := strings.Join(names(got), " ")
if !strings.Contains(have, "tailscale") {
t.Fatalf("the chosen VPN is not in the set: %s", have)
}
if strings.Contains(have, "mesh-wireguard") {
t.Fatalf("choosing tailscale still installed WireGuard: %s", have)
}
}
func TestChoosingOneVPNCannotDragTheOtherBackIn(t *testing.T) {
// This happened. The person chose tailscale; the names module required the mesh's own
// addressing; only WireGuard provides that; so both were installed and nobody was told.
//
// The claim is what catches it. Providing a private network is not the singular part — a
// machine could run two VPNs for two purposes — but being *the* one the mesh runs over is.
_, err := Resolve(networkingShelf(
Manifest{Module: "tailscale", Provides: []string{"private-network"},
Claims: []Claim{{Name: "the-private-network", Scope: ScopeNode}}},
), []string{"networking", "tailscale"}, workstation(), nil)
if err == nil {
t.Fatal("a machine was given two private networks without being told")
}
if !strings.Contains(err.Error(), "the-private-network") {
t.Fatalf("the refusal does not say what collided: %v", err)
}
}
func TestNamesNeedTheMeshsOwnAddresses(t *testing.T) {
// Names are computed from addresses the mesh handed out. Over a VPN that hands out its own,
// the mesh has nothing to write, so the names module requires the addressing rather than a
// private network in general — otherwise a machine gets a hosts file full of addresses that
// mean nothing on it.
_, err := Resolve(shelf(
Manifest{Module: "mesh-names", Computed: "mesh-names",
Provides: []string{"name-resolution"}, Requires: []string{"mesh-addressing"}},
Manifest{Module: "tailscale", Provides: []string{"private-network"}},
), []string{"mesh-names", "tailscale"}, workstation(), nil)
if err == nil {
t.Fatal("the mesh's names were installed over a VPN whose addresses it does not hand out")
}
if !strings.Contains(err.Error(), "mesh-addressing") {
t.Fatalf("the refusal does not say what is missing: %v", err)
}
}
func TestARequirementWantedTwiceIsReportedOnce(t *testing.T) {
// Two identical lines make a person hunt for the difference between them before realising
// there is none.
_, err := Resolve(shelf(
Manifest{Module: "one", Requires: []string{"shell"}},
Manifest{Module: "two", Requires: []string{"shell"}},
Manifest{Module: "bash", Provides: []string{"shell"}},
Manifest{Module: "zsh", Provides: []string{"shell"}},
), []string{"one", "two"}, workstation(), nil)
if err == nil {
t.Fatal("two shells and one was picked silently")
}
if n := strings.Count(err.Error(), `"shell" is wanted by`); n != 1 {
t.Fatalf("the same requirement was reported %d times:\n%v", n, err)
}
}
+21
View File
@@ -69,6 +69,20 @@ type Manifest struct {
// Resources are what this module puts on a node, in the host's own vocabulary. // Resources are what this module puts on a node, in the host's own vocabulary.
Resources []map[string]any `json:"resources,omitempty"` Resources []map[string]any `json:"resources,omitempty"`
// Computed names something in the control plane that works this module's resources out per
// node, instead of them being fixed here.
//
// Because some files cannot be written in advance. A machine's peer list on the private
// network is derived from every other machine, so it differs on each one and changes when any
// of them changes — there is nothing to put in a manifest.
//
// Being a module anyway is the point: it is assigned like anything else, so a machine that
// should not be on the private network simply is not given it, and the network is worked out
// over the machines that have it. Before this, connectivity was code beside the module system
// doing the same job, and every machine with an address was on the network whether or not
// anybody wanted it there.
Computed string `json:"computed,omitempty"`
} }
// ParseManifest reads a module manifest, refusing anything it cannot act on. // ParseManifest reads a module manifest, refusing anything it cannot act on.
@@ -117,6 +131,13 @@ func ParseManifest(raw []byte) (Manifest, error) {
m.Module, c.Name, c.Scope)) m.Module, c.Name, c.Scope))
} }
} }
if m.Computed != "" && len(m.Resources) > 0 {
// One or the other. A module that both ships files and has them computed would leave
// nobody able to say where a given file came from.
problems = append(problems, fmt.Sprintf(
"%s has resources of its own and says they are computed by %q; it is one or the other",
m.Module, m.Computed))
}
for i, r := range m.Resources { for i, r := range m.Resources {
id, _ := r["id"].(string) id, _ := r["id"].(string)
if id == "" { if id == "" {
+98
View File
@@ -0,0 +1,98 @@
package catalogue_test
import (
"encoding/json"
"strings"
"testing"
"github.com/novox/mesh-control/internal/catalogue"
"github.com/novox/mesh-control/internal/overlay"
)
// The manifests the control plane actually ships, resolved.
//
// Written because the earlier tests built their own manifests and passed while the real one was
// missing a claim — a whole mechanism could have been absent from what ships and every test would
// still have been green.
func provided(t *testing.T) map[string]catalogue.Manifest {
t.Helper()
out := map[string]catalogue.Manifest{}
for _, raw := range []map[string]any{
overlay.Manifest(), overlay.NamesManifest(), overlay.DomainManifest(),
} {
b, err := json.Marshal(raw)
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(b)
if err != nil {
t.Fatalf("a manifest this control plane ships is not valid: %v", err)
}
out[m.Module] = m
}
return out
}
func TestTheShippedNetworkingModulesResolveOnTheirOwn(t *testing.T) {
got, err := catalogue.Resolve(provided(t), []string{overlay.Domain},
catalogue.Node{Name: "workstation", Site: "house"}, nil)
if err != nil {
t.Fatalf("assigning %s does not work out of the box: %v", overlay.Domain, err)
}
var have []string
for _, m := range got.Modules {
have = append(have, m.Module)
}
for _, want := range []string{overlay.Domain, overlay.Name, overlay.Names} {
if !strings.Contains(strings.Join(have, " "), want) {
t.Fatalf("%s did not bring in %s: %v", overlay.Domain, want, have)
}
}
}
func TestTheShippedWireGuardModuleClaimsBeingTheNetwork(t *testing.T) {
// Without this, a person who chose another VPN gets WireGuard as well, dragged in by the
// names, and is not told. The claim is the only thing that catches it.
shipped := provided(t)
_, err := catalogue.Resolve(
withTailscale(shipped),
[]string{overlay.Domain, "tailscale"},
catalogue.Node{Name: "workstation", Site: "house"}, nil)
if err == nil {
t.Fatal("a machine was given two private networks and nobody was told")
}
if !strings.Contains(err.Error(), overlay.TheNetwork) {
t.Fatalf("the refusal does not say what collided: %v", err)
}
}
func TestTheShippedNamesModuleNeedsTheMeshsOwnAddresses(t *testing.T) {
// Over a VPN whose addresses the mesh does not hand out, it has no names to write. Refusing
// is what stops a machine getting a hosts file that means nothing on it.
shipped := provided(t)
delete(shipped, overlay.Name)
_, err := catalogue.Resolve(shipped, []string{overlay.Names},
catalogue.Node{Name: "workstation", Site: "house"}, nil)
if err == nil {
t.Fatal("the mesh's names resolved with nothing handing out the mesh's addresses")
}
if !strings.Contains(err.Error(), overlay.Addressing) {
t.Fatalf("the refusal does not name what is missing: %v", err)
}
}
func withTailscale(shelf map[string]catalogue.Manifest) map[string]catalogue.Manifest {
out := map[string]catalogue.Manifest{}
for k, v := range shelf {
out[k] = v
}
// Deliberately without name-resolution of its own, which is the case that used to install
// both VPNs: the names then needed the mesh's addressing, and only WireGuard has it.
out["tailscale"] = catalogue.Manifest{
Module: "tailscale", Version: "1",
Provides: []string{overlay.Requirement},
Claims: []catalogue.Claim{{Name: overlay.TheNetwork, Scope: catalogue.ScopeNode}},
}
return out
}
+49 -5
View File
@@ -34,6 +34,9 @@ type Held struct {
// Resolution is what a node should run, and why. // Resolution is what a node should run, and why.
type Resolution struct { type Resolution struct {
// Node is which machine this was resolved for, so a generator can be asked about it.
Node string
// Modules in the order they were resolved: assigned first, then what they pulled in. // Modules in the order they were resolved: assigned first, then what they pulled in.
Modules []Manifest Modules []Manifest
// Because says why each module is here — assigned, or required by something. // Because says why each module is here — assigned, or required by something.
@@ -99,10 +102,15 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, elsewh
} }
} }
// What has already been complained about. A requirement can be wanted by several modules at
// once, and saying the same thing twice makes a person hunt for the difference between two
// identical lines before realising there is none.
reported := map[string]bool{}
for len(queue) > 0 { for len(queue) > 0 {
want := queue[0] want := queue[0]
queue = queue[1:] queue = queue[1:]
if chosen[want] { if chosen[want] || reported[want] {
continue continue
} }
// Already answered by something in the set. This is the case that makes assigning zsh do // Already answered by something in the set. This is the case that makes assigning zsh do
@@ -114,6 +122,7 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, elsewh
candidates := offers[want] candidates := offers[want]
switch len(candidates) { switch len(candidates) {
case 0: case 0:
reported[want] = true
problems = append(problems, fmt.Sprintf( problems = append(problems, fmt.Sprintf(
"nothing provides %q, wanted by %s", want, because[want])) "nothing provides %q, wanted by %s", want, because[want]))
continue continue
@@ -121,6 +130,7 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, elsewh
// No choice to make, so none is made. This is the case that lets `install i3` bring // No choice to make, so none is made. This is the case that lets `install i3` bring
// in xorg without anybody being asked anything. // in xorg without anybody being asked anything.
default: default:
reported[want] = true
problems = append(problems, fmt.Sprintf( problems = append(problems, fmt.Sprintf(
"%q is wanted by %s and %d modules provide it — choose one and assign it: %s", "%q is wanted by %s and %d modules provide it — choose one and assign it: %s",
want, because[want], len(candidates), strings.Join(candidates, ", "))) want, because[want], len(candidates), strings.Join(candidates, ", ")))
@@ -148,7 +158,7 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, elsewh
} }
} }
resolution := Resolution{Because: because} resolution := Resolution{Node: node.Name, Because: because}
for _, n := range order { for _, n := range order {
resolution.Modules = append(resolution.Modules, catalogue[n]) resolution.Modules = append(resolution.Modules, catalogue[n])
} }
@@ -275,16 +285,50 @@ func checkResources(modules []Manifest) []string {
// SettingsBy is the layers that apply to each module, keyed by module name. // SettingsBy is the layers that apply to each module, keyed by module name.
type SettingsBy map[string][]Layer type SettingsBy map[string][]Layer
// Generator works out a module's resources for one node, where they cannot be written in advance.
type Generator interface {
// Resources for this node. Absent means the node is not part of whatever this generates,
// which is an ordinary answer rather than a failure — a machine assigned the module before it
// has an address on the network is in exactly that state.
Resources(node string) ([]map[string]any, bool, error)
}
// Rendering is everything needed to turn a resolution into the declaration a node is sent.
type Rendering struct {
Settings SettingsBy
Generators map[string]Generator
}
// Declaration is everything the resolved modules put on the node, with settings applied. // Declaration is everything the resolved modules put on the node, with settings applied.
// //
// Resource identities are prefixed with the module they came from. Two modules may reasonably // Resource identities are prefixed with the module they came from. Two modules may reasonably
// both call something "config", and without this the second would silently replace the first — // both call something "config", and without this the second would silently replace the first —
// the node applying one of them and reporting success. // the node applying one of them and reporting success.
func (r Resolution) Declaration(settings SettingsBy) ([]map[string]any, error) { func (r Resolution) Declaration(with Rendering) ([]map[string]any, error) {
var out []map[string]any var out []map[string]any
for _, m := range r.Modules { for _, m := range r.Modules {
for _, unsettled := range m.Resources { resources := m.Resources
resource, err := ApplySettings(unsettled, settings[m.Module]) if m.Computed != "" {
generator, known := with.Generators[m.Computed]
if !known {
return nil, fmt.Errorf(
"%s says its resources are computed by %q, and this control plane has no %q",
m.Module, m.Computed, m.Computed)
}
generated, part, err := generator.Resources(r.Node)
if err != nil {
return nil, err
}
if !part {
// Assigned, and not yet part of what this generates. Nothing to put on the
// machine, which is different from an error: a node given the network module
// before it has an address is in exactly that state, briefly.
continue
}
resources = generated
}
for _, unsettled := range resources {
resource, err := ApplySettings(unsettled, with.Settings[m.Module])
if err != nil { if err != nil {
return nil, err return nil, err
} }
+1 -1
View File
@@ -320,7 +320,7 @@ func TestARequirementNamingAModuleMeansThatModule(t *testing.T) {
func mustDeclare(t *testing.T, r Resolution) []map[string]any { func mustDeclare(t *testing.T, r Resolution) []map[string]any {
t.Helper() t.Helper()
out, err := r.Declaration(nil) out, err := r.Declaration(Rendering{})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
+45 -1
View File
@@ -173,8 +173,52 @@ func (i *Inventory) Catalogue(ctx context.Context) (map[string]catalogue.Manifes
return out, rows.Err() return out, rows.Err()
} }
// ForgetModule removes a module, unless a machine is running it. // Provide records a module the control plane ships with itself.
//
// The mesh's own private network is one: what computes its files is in this binary, so the
// manifest is too. Marked so it can be told apart from a module somebody wrote — not because it
// behaves differently, but because "where did this come from" must have an answer for everything
// in the catalogue, and "it came with the control plane" is that answer.
func (i *Inventory) Provide(ctx context.Context, m catalogue.Manifest) error {
raw, err := json.Marshal(m)
if err != nil {
return err
}
_, err = i.store.Pool().Exec(ctx,
`insert into module (name, manifest, version, source)
values ($1, $2, $3, 'the control plane')
on conflict (name) do update set manifest = excluded.manifest,
version = excluded.version, source = 'the control plane'`,
m.Module, raw, m.Version)
return err
}
// Provided reports whether a module came with the control plane rather than from a repository.
func (i *Inventory) Provided(ctx context.Context, name string) (bool, error) {
var source *string
err := i.store.Pool().QueryRow(ctx,
`select source from module where name = $1`, name).Scan(&source)
if err != nil {
return false, err
}
return source != nil && *source == "the control plane", nil
}
// ForgetModule removes a module, unless a machine is running it, and never one the control plane
// provides.
func (i *Inventory) ForgetModule(ctx context.Context, name string) error { func (i *Inventory) ForgetModule(ctx context.Context, name string) error {
provided, err := i.Provided(ctx, name)
if err != nil {
return err
}
if provided {
// Refused rather than removed and re-created on the next migrate, which would look like
// it worked and quietly come back. Taking it off a machine is what "I do not want this"
// means, and that is what unassign is for.
return fmt.Errorf(
"%s comes with the control plane and cannot be forgotten; unassign it instead", name)
}
var on []string var on []string
rows, err := i.store.Pool().Query(ctx, rows, err := i.store.Pool().Query(ctx,
`select n.name from assignment a join node n on n.id = a.node where a.module = $1 `select n.name from assignment a join node n on n.id = a.node where a.module = $1
+6 -13
View File
@@ -34,7 +34,7 @@ type Resource map[string]any
// Three resources and nothing clever: the tools, the configuration, and the interface running. A // Three resources and nothing clever: the tools, the configuration, and the interface running. A
// person can read it, which is the point — this is the first thing a node is ever told, and if it // person can read it, which is the point — this is the first thing a node is ever told, and if it
// is wrong the node is unreachable and the mistake has to be findable by eye. // is wrong the node is unreachable and the mistake has to be findable by eye.
func Declaration(node Node, peers []Peer, everyone []Node, keyPath string) ([]byte, error) { func Declaration(node Node, peers []Peer, keyPath string) ([]byte, error) {
if node.Address == "" { if node.Address == "" {
return nil, fmt.Errorf("%s has no address on the overlay, so there is nothing to configure", return nil, fmt.Errorf("%s has no address on the overlay, so there is nothing to configure",
node.Name) node.Name)
@@ -78,18 +78,11 @@ func Declaration(node Node, peers []Peer, everyone []Node, keyPath string) ([]by
}, },
} }
// And the names, which come from the same graph and arrive in the same declaration. Separate // The names used to be appended here, on the argument that a node with peers and no names is
// steps in the design and one delivery in practice: a node that had the peers and not the // half on the network. True, and the wrong place to fix it: names would be identical over a
// names, or the reverse, would be half on the network for as long as that lasted. // different private network, so bundling them with WireGuard made one module out of two
names, err := Hosts(everyone, node.Name) // things. They are their own module now, requiring this one — which is what keeps them
if err != nil { // arriving together without pretending they are the same concern.
return nil, err
}
resources = append(resources, Resource{
"id": "mesh-names", "type": "file", "path": HostsPath,
"mode": "0644", "content": names,
})
return json.Marshal(map[string]any{"declaration": 1, "resources": resources}) return json.Marshal(map[string]any{"declaration": 1, "resources": resources})
} }
+2 -2
View File
@@ -9,7 +9,7 @@ import (
func declarationFor(t *testing.T, node Node, peers []Peer) (string, []map[string]any) { func declarationFor(t *testing.T, node Node, peers []Peer) (string, []map[string]any) {
t.Helper() t.Helper()
raw, err := Declaration(node, peers, nil, "") raw, err := Declaration(node, peers, "")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -127,7 +127,7 @@ func TestTheFileSaysNotToEditIt(t *testing.T) {
func TestANodeWithNoAddressIsRefused(t *testing.T) { func TestANodeWithNoAddressIsRefused(t *testing.T) {
// Rather than a configuration with a blank address, which wg-quick would reject on the // Rather than a configuration with a blank address, which wg-quick would reject on the
// machine, at boot, where the failure is much harder to see. // machine, at boot, where the failure is much harder to see.
if _, err := Declaration(Node{Name: "laptop", Key: "PUB"}, nil, nil, ""); err == nil { if _, err := Declaration(Node{Name: "laptop", Key: "PUB"}, nil, ""); err == nil {
t.Fatal("a node with no overlay address was given a configuration") t.Fatal("a node with no overlay address was given a configuration")
} }
} }
+197
View File
@@ -0,0 +1,197 @@
package overlay
import "encoding/json"
// The private network as a module rather than as code beside the module system.
//
// A machine's peer list is derived from every other machine, so it differs on each one and
// changes when any of them changes — there is nothing that could be written in a manifest. So the
// module says its resources are computed, and this computes them.
//
// What that buys, beyond one mechanism instead of two: **a machine is on the private network
// because it was assigned the module.** Before this, every machine with a key and an address was
// on it, and there was no way to say a machine should stay off.
//
// It is worth saying what is *not* here, because networking looks like it should be foundational.
// The host needs none of this. It has an address and a route to the broker before the mesh exists
// — that is the machine's own networking — and the broker's address is carried in the enrolment
// token rather than resolved. So the private network is something the mesh installs on top, like
// anything else, and a node without it is an ordinary node that nothing reaches directly.
// What a module asks for when it needs machines to reach each other. **This is the name that
// matters** — WireGuard is one way to answer it, and naming the requirement after the answer is
// how a mesh ends up unable to have a second one.
const Requirement = "private-network"
// Name is the module that answers it with WireGuard, and Names is the one that gives the machines
// names. Two modules rather than one, because they are two different things: names would be the
// same over any private network, and they are only bundled here by an accident of both being
// computed.
const (
Name = "mesh-wireguard"
Names = "mesh-names"
)
// Resolution is what a module asks for when it needs to reach other machines by name. Separate
// from Requirement because they are separate jobs: one is whether packets arrive, the other is
// whether a name means anything. A machine can want the first without the second.
const Resolution = "name-resolution"
// Addressing is the mesh handing out addresses on the private network itself.
//
// Names are computed from it, which is why they require this rather than a private network in
// general. A different VPN that hands out its own addresses would come with its own names — the
// mesh has nothing to write about a machine whose address it did not choose. Saying so here is
// what keeps a machine from being given a hosts file full of addresses that mean nothing.
const Addressing = "mesh-addressing"
// TheNetwork is what a machine can only have one of.
//
// Providing a private network is not the singular part — a machine could reasonably run two VPNs
// for two different purposes. Being **the** one the mesh runs over is singular, and without
// saying so a person who chose a different VPN can still end up with this one dragged back in by
// something that needed the mesh's own addresses. Which is exactly what happened, once.
const TheNetwork = "the-private-network"
// Domain is the module for people who want a network and do not want to choose one.
//
// It has no files of its own — it is requirements and nothing else. Assigning it finds one
// answer to each and takes them silently, so getting a mesh onto a private network is one word.
// The day the catalogue holds a second VPN there are two answers, the resolver refuses and names
// both, and choosing is assigning the one you want. **That is the whole mechanism**: picking an
// implementation is assigning a module, and there is no flavor field, no configuration language,
// and nothing to learn.
const Domain = "networking"
// Generator answers what one node's network configuration is.
type Generator struct {
nodes []Node
graph Graph
// keyPath is where each node keeps the private half it generated. Named rather than carried:
// the mesh has never seen it and never will.
keyPath string
}
// From builds a generator over the machines that are part of the network.
//
// The nodes given are the ones assigned the module — not every node the mesh knows. A machine
// that was never given it is absent from everybody's peer list and from the names, which is what
// "not on the network" has to mean.
func From(nodes []Node, cidr, keyPath string) (*Generator, error) {
graph, err := Compute(nodes, cidr)
if err != nil {
return nil, err
}
return &Generator{nodes: nodes, graph: graph, keyPath: keyPath}, nil
}
// Resources is one node's interface, peers and names.
func (g *Generator) Resources(node string) ([]map[string]any, bool, error) {
peers, part := g.graph[node]
if !part {
// Assigned and not yet placed on the network. Ordinary and brief, so it is an answer
// rather than an error.
return nil, false, nil
}
var self Node
for _, n := range g.nodes {
if n.Name == node {
self = n
}
}
raw, err := Declaration(self, peers, g.keyPath)
if err != nil {
return nil, false, err
}
var parsed struct {
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(raw, &parsed); err != nil {
return nil, false, err
}
return parsed.Resources, true, nil
}
// NameGenerator answers what one node's hosts file is.
//
// Separate from the interface and the peers because it is a separate concern. A machine's names
// come from the mesh knowing every machine, not from how the packets travel — over a different
// private network the peers would be written by something else and this would be unchanged.
type NameGenerator struct{ nodes []Node }
// NamesFor builds the name generator over the machines on the private network.
func NamesFor(nodes []Node) *NameGenerator { return &NameGenerator{nodes: nodes} }
// Resources is the one file.
func (g *NameGenerator) Resources(node string) ([]map[string]any, bool, error) {
var found bool
for _, n := range g.nodes {
if n.Name == node {
found = true
}
}
if !found {
return nil, false, nil
}
hosts, err := Hosts(g.nodes, node)
if err != nil {
return nil, false, err
}
return []map[string]any{{
"id": "mesh-names", "type": "file", "path": HostsPath,
"mode": "0644", "content": hosts,
}}, true, nil
}
// Nodes are the machines this generator was built over, so a caller can say who is on the network.
func (g *Generator) Nodes() []Node { return g.nodes }
// Graph is the peer list per node, for showing.
func (g *Generator) Graph() Graph { return g.graph }
// Manifest is the module the mesh provides for itself.
//
// It ships with the control plane rather than coming from a repository, because the thing that
// computes it ships with the control plane. That is the only way it is unusual: it is assigned,
// unassigned, resolved and settled exactly like a module somebody wrote.
func Manifest() map[string]any {
return map[string]any{
"module": Name,
"version": "1",
"computed": Name,
"provides": []string{Requirement, Addressing},
"claims": []map[string]any{{"name": TheNetwork, "scope": "node"}},
}
}
// NamesManifest is the module that gives machines names on the private network.
//
// It requires the network rather than providing it, which is the whole reason it is separate: a
// name resolves to an address on the private wire, so having names without being on it would
// point every machine at somewhere it cannot reach.
func NamesManifest() map[string]any {
return map[string]any{
"module": Names,
"version": "1",
"computed": Names,
"provides": []string{Resolution},
"requires": []string{Addressing},
}
}
// DomainManifest is the module that means "get the network working".
func DomainManifest() map[string]any {
return map[string]any{
"module": Domain,
"version": "1",
"requires": []string{Requirement, Resolution},
}
}
// Empty is a network nobody is on.
//
// A mesh where no machine was given the module. Legitimate rather than broken — every node still
// reaches the broker, which is what being in the mesh is — so it answers "not part of this" for
// everyone instead of refusing for want of a hub.
func Empty() *Generator { return &Generator{graph: Graph{}} }
+61
View File
@@ -0,0 +1,61 @@
package overlay
import (
"strings"
"testing"
)
// Names are their own module.
//
// They used to arrive inside the WireGuard declaration, on the argument that a machine with peers
// and no names is half on the network. True, and the wrong place to fix it: names would be
// identical over a different private network, so bundling them made one module out of two things.
func TestAMachineNotOnTheNetworkGetsNoNames(t *testing.T) {
// Names resolve to addresses on the private wire. Giving them to a machine that is not on it
// would point every lookup somewhere it cannot reach — worse than having no names at all.
g := NamesFor([]Node{at("anchor", "dc", "10.42.0.1", "198.51.100.10:51820", true)})
_, part, err := g.Resources("laptop")
if err != nil {
t.Fatal(err)
}
if part {
t.Fatal("a machine that is not on the private network was given the mesh's names")
}
}
func TestTheNamesAreOneFileAndSayWhoIsAsking(t *testing.T) {
g := NamesFor([]Node{
at("anchor", "dc", "10.42.0.1", "198.51.100.10:51820", true),
at("workstation", "house", "10.42.0.2", "", false),
})
out, part, err := g.Resources("workstation")
if err != nil || !part {
t.Fatalf("part=%v err=%v", part, err)
}
if len(out) != 1 || out[0]["path"] != HostsPath {
t.Fatalf("got %v", out)
}
content := out[0]["content"].(string)
if !strings.Contains(content, "anchor.internal") {
t.Fatalf("another machine on the network has no name here:\n%s", content)
}
if !strings.Contains(content, "this machine") {
t.Fatalf("the file does not say which machine it is on:\n%s", content)
}
}
func TestTheWireGuardDeclarationNoLongerCarriesTheNames(t *testing.T) {
// The split, asserted. Two modules, so a machine can have the peers from one and the names
// from another — which is what makes a second VPN possible at all.
raw, err := Declaration(
at("workstation", "house", "10.42.0.2", "", false),
[]Peer{{Name: "anchor", Key: "PUB", Allowed: "10.42.0.0/16",
Endpoint: "198.51.100.10:51820"}}, "")
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(raw), HostsPath) {
t.Fatalf("the WireGuard declaration still writes %s", HostsPath)
}
}