The mesh decides what a node runs

The gap that has been named at the end of every report for a week. Until now a
declaration came from a person handing over a file; now it comes from what was
assigned, resolved against the catalogue, and the control plane is deciding
rather than relaying.

Everything from the module conversation, built and run on real machines:

  assign laptop i3      -> accepted, brings xorg, because nothing else provides
                           it and there was no choice to make
  assign laptop sway    -> refused: xorg and wayland both claim the-seat
  assign laptop editor  -> refused: three modules provide a shell -- bash,
                           fish, zsh -- choose one
  assign laptop zsh     -> accepted, and the editor's requirement is answered
  bash, fish beside it  -> fine, nothing is claimed

Claims rather than pairwise exclusion, so a third display server would say what
it claims and need no edit to xorg or wayland. Scoped to node, site or mesh:
two DHCP servers at one site collide and at two sites do not, and the mesh-wide
one is the hub said as a claim instead of hard-coded.

Some conflicts cost no manifest field at all. The refusal above names the seat
AND the two files, because the mesh already holds every resource of every
module -- neither i3 nor sway knows the other exists.

Resource identities carry their module, so two modules may both call something
"config" without the second silently replacing the first. What a service
reflects is qualified the same way, or it would name a resource that no longer
exists and stop being restarted when its own configuration changes.

Nothing is sent until every node resolves. A push that configured three and
refused on the fourth would leave the mesh in a state nobody asked for, and the
fourth is exactly where a claim collision appears.

One real flaw found by using it rather than by testing it: assigning zsh did
not satisfy a requirement for a shell. Requirements were counted against the
catalogue without first asking what the set already offers, so "choose one and
assign it" named three modules and then ignored the one you chose. The remedy
was useless and every test passed.
This commit is contained in:
2026-08-29 22:00:06 +02:00
parent f0cff88172
commit 409cd16a09
6 changed files with 1297 additions and 0 deletions
+302
View File
@@ -8,16 +8,19 @@ package main
import ( import (
"context" "context"
"encoding/json"
"errors" "errors"
"flag" "flag"
"fmt" "fmt"
"os" "os"
"os/signal" "os/signal"
"sort"
"strings" "strings"
"syscall" "syscall"
"time" "time"
"github.com/novox/mesh-control/internal/broker" "github.com/novox/mesh-control/internal/broker"
"github.com/novox/mesh-control/internal/catalogue"
"github.com/novox/mesh-control/internal/identity" "github.com/novox/mesh-control/internal/identity"
"github.com/novox/mesh-control/internal/inventory" "github.com/novox/mesh-control/internal/inventory"
"github.com/novox/mesh-control/internal/link" "github.com/novox/mesh-control/internal/link"
@@ -76,6 +79,14 @@ func run() error {
return declare(ctx, args[1:]) return declare(ctx, args[1:])
case "overlay": case "overlay":
return overlayCommand(ctx, args[1:]) return overlayCommand(ctx, args[1:])
case "module":
return moduleCommand(ctx, args[1:])
case "assign", "unassign":
return assignCommand(ctx, args[0], args[1:])
case "plan":
return planCommand(ctx, args[1:])
case "push":
return pushCommand(ctx, args[1:])
case "version": case "version":
fmt.Println(version) fmt.Println(version)
return nil return nil
@@ -103,6 +114,13 @@ func usage() {
overlay place <node> [flags] say where a node is and how it is reached overlay place <node> [flags] say where a node is and how it is reached
overlay show the private network, as the mesh computes it overlay show the private network, as the mesh computes it
overlay push send every node its part of the private network overlay push send every node its part of the private network
module add <file> register a module from its manifest
module list what modules this mesh knows about
module forget <name> remove one, unless a node is running it
assign <node> <module> put a module on a node
unassign <node> <module> take it off
plan <node> what that node would run, and why
push [<node>] send a node everything it should be
version what this binary is version what this binary is
Each context reaches its own store through its own credential (novox/hq ADR 0008), named Each context reaches its own store through its own credential (novox/hq ADR 0008), named
@@ -668,3 +686,287 @@ func roughly(d time.Duration) string {
return fmt.Sprintf("%dd", int(d.Hours()/24)) return fmt.Sprintf("%dd", int(d.Hours()/24))
} }
} }
func moduleCommand(ctx context.Context, args []string) error {
if len(args) == 0 {
return errors.New("module add <file>, module list, or module forget <name>")
}
inv, err := openInventory(ctx)
if err != nil {
return err
}
defer inv.Close()
switch args[0] {
case "add":
if len(args) != 2 {
return errors.New("module add <manifest.json>")
}
raw, err := os.ReadFile(args[1])
if err != nil {
return err
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
return err
}
if err := inv.RegisterModule(ctx, m); err != nil {
return err
}
fmt.Printf("%s registered", m.Module)
if len(m.Provides) > 0 {
fmt.Printf(", providing %s", strings.Join(m.Provides, ", "))
}
fmt.Println()
for _, c := range m.Claims {
fmt.Printf(" claims %s, one per %s\n", c.Name, c.At())
}
return nil
case "list":
shelf, err := inv.Catalogue(ctx)
if err != nil {
return err
}
if len(shelf) == 0 {
fmt.Println("this mesh knows about no modules yet")
return nil
}
var names []string
for n := range shelf {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
m := shelf[n]
fmt.Printf("%-20s", m.Module)
if len(m.Provides) > 0 {
fmt.Printf(" provides %s", strings.Join(m.Provides, ", "))
}
for _, c := range m.Claims {
fmt.Printf(" claims %s/%s", c.At(), c.Name)
}
fmt.Println()
}
return nil
case "forget":
if len(args) != 2 {
return errors.New("module forget <name>")
}
if err := inv.ForgetModule(ctx, args[1]); err != nil {
return err
}
fmt.Printf("%s forgotten\n", args[1])
return nil
default:
return fmt.Errorf("module has no %q; it has add, list and forget", args[0])
}
}
func assignCommand(ctx context.Context, verb string, args []string) error {
if len(args) != 2 {
return fmt.Errorf("%s <node> <module>", verb)
}
inv, err := openInventory(ctx)
if err != nil {
return err
}
defer inv.Close()
if verb == "unassign" {
if err := inv.Unassign(ctx, args[0], args[1]); err != nil {
return err
}
fmt.Printf("%s no longer runs %s — run `push %s` to make it so\n", args[0], args[1], args[0])
return nil
}
if err := inv.Assign(ctx, args[0], args[1]); err != nil {
return err
}
fmt.Printf("%s is assigned %s\n", args[0], args[1])
// Resolved immediately, because an assignment that cannot be applied should be said now
// rather than at the next push. The assignment is kept either way: it is what a person meant,
// and the refusal is about the set rather than about this one.
if _, err := planFor(ctx, inv, args[0]); err != nil {
fmt.Println()
return err
}
fmt.Printf(" run `push %s` to send it\n", args[0])
return nil
}
// planFor works out everything a node should run, from what was assigned to it.
func planFor(ctx context.Context, inv *inventory.Inventory, nodeName string) (catalogue.Resolution, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return catalogue.Resolution{}, err
}
assigned, err := inv.Assigned(ctx, nodeName)
if err != nil {
return catalogue.Resolution{}, err
}
capabilities, err := inv.ProfileOf(ctx, nodeName)
if err != nil {
return catalogue.Resolution{}, err
}
places, err := inv.Overlays(ctx)
if err != nil {
return catalogue.Resolution{}, err
}
var site string
for _, p := range places {
if p.Name == nodeName {
site = p.Site
}
}
// What every other node already holds, so the claims wider than one machine can be checked.
// Resolved rather than read from a table: a claim is held by whatever a node actually runs,
// and a record of it would be a second answer that could disagree with the first.
var elsewhere []catalogue.Held
for _, p := range places {
if p.Name == nodeName {
continue
}
theirs, err := inv.Assigned(ctx, p.Name)
if err != nil || len(theirs) == 0 {
continue
}
theirCaps, _ := inv.ProfileOf(ctx, p.Name)
got, err := catalogue.Resolve(shelf, theirs,
catalogue.Node{Name: p.Name, Site: p.Site, Capabilities: theirCaps}, nil)
if err != nil {
// Their set does not resolve either. Not this node's problem to report, and their
// claims cannot be counted because nothing of theirs is running.
continue
}
elsewhere = append(elsewhere, got.Claims...)
}
return catalogue.Resolve(shelf, assigned,
catalogue.Node{Name: nodeName, Site: site, Capabilities: capabilities}, elsewhere)
}
func planCommand(ctx context.Context, args []string) error {
if len(args) != 1 {
return errors.New("plan <node>")
}
inv, err := openInventory(ctx)
if err != nil {
return err
}
defer inv.Close()
plan, err := planFor(ctx, inv, args[0])
if err != nil {
return err
}
if len(plan.Modules) == 0 {
fmt.Printf("%s is assigned nothing\n", args[0])
return nil
}
fmt.Printf("%s would run:\n", args[0])
for _, m := range plan.Modules {
fmt.Printf(" %-20s %s\n", m.Module, plan.Because[m.Module])
}
for _, c := range plan.Claims {
fmt.Printf(" holds %s, one per %s\n", c.Claim, c.Scope)
}
fmt.Printf("\n%d resource(s)\n", len(plan.Declaration()))
return nil
}
// pushCommand sends nodes everything they should be: their place on the network, and what their
// assignments resolve to.
//
// One declaration, not two. A node holding its network and not its modules, or the reverse, is
// half-configured for as long as that lasts — and the two are computed from the same picture of
// the mesh, so sending them apart would let them disagree.
func pushCommand(ctx context.Context, args []string) error {
if len(args) > 1 {
return errors.New("push [<node>] — one node, or all of them")
}
inv, err := openInventory(ctx)
if err != nil {
return err
}
defer inv.Close()
ident, err := openIdentity(ctx)
if err != nil {
return err
}
defer ident.Close()
nodes, computed, err := graph(ctx, inv)
if err != nil {
return err
}
server, err := link.Connect(nil, nil)
if err != nil {
return err
}
defer server.Close()
// Every node is resolved before anything is sent. A push that configured three nodes and then
// refused on the fourth would leave the mesh in a state nobody asked for, and the fourth is
// exactly where a claim collision shows up.
type ready struct {
node overlay.Node
resources []map[string]any
}
var sending []ready
var refusals []string
for _, n := range nodes {
if len(args) == 1 && n.Name != args[0] {
continue
}
peers, onOverlay := computed[n.Name]
if !onOverlay {
fmt.Printf("%s is not on the overlay yet — skipped\n", n.Name)
continue
}
declaration, err := overlay.Declaration(n, peers, nodes, "")
if err != nil {
return err
}
var resources struct {
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(declaration, &resources); err != nil {
return err
}
plan, err := planFor(ctx, inv, n.Name)
if err != nil {
refusals = append(refusals, fmt.Sprintf("%s:\n%v", n.Name, err))
continue
}
sending = append(sending, ready{n, append(resources.Resources, plan.Declaration()...)})
}
if len(refusals) > 0 {
return fmt.Errorf("nothing was sent. %d node(s) could not be resolved:\n\n%s",
len(refusals), strings.Join(refusals, "\n\n"))
}
for _, s := range sending {
body, err := json.Marshal(map[string]any{"declaration": 1, "resources": s.resources})
if err != nil {
return err
}
if err := link.Declare(ctx, server.Channel(), ident, s.node.Name, body, 15*time.Second); err != nil {
return err
}
fmt.Printf("sent %s %d resource(s)\n", s.node.Name, len(s.resources))
}
fmt.Printf("\n%d node(s) told\n", len(sending))
return nil
}
+143
View File
@@ -0,0 +1,143 @@
// Package catalogue is what modules are, and what a node gets when it is assigned some.
//
// novox/hq ADR 0009: everything is a module, a module declares what it provides and requires,
// and a module declares what it claims. This turns a set of assignments into the one declaration
// a node is sent — which is the first thing the control plane decides rather than relays.
package catalogue
import (
"encoding/json"
"fmt"
"regexp"
"sort"
"strings"
)
// Scopes a claim can have.
//
// Not everything singular is singular per machine: a seat is one per node, a DHCP server is one
// per segment, and the hub is one per mesh. Scope says which, and it is the same idea the mesh
// already enforces by hand for the hub.
const (
ScopeNode = "node"
ScopeSite = "site"
ScopeMesh = "mesh"
)
// name is what a module, a provision or a claim may be called.
//
// Constrained because these become resource identities, permission patterns and error messages,
// and a name that is valid in one and not the others is a fault found late.
var name = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*(\.[a-z0-9][a-z0-9-]*)*$`)
// Claim is a singular resource a module takes over.
type Claim struct {
Name string `json:"name"`
// Scope defaults to the node, which is where nearly everything singular is singular.
Scope string `json:"scope,omitempty"`
}
// At is this claim's scope, with the default applied.
func (c Claim) At() string {
if c.Scope == "" {
return ScopeNode
}
return c.Scope
}
// Manifest is everything a module says about itself.
type Manifest struct {
Module string `json:"module"`
Version string `json:"version,omitempty"`
// Provides are the names other modules may require. A module always provides its own name;
// this is for the rest — `zsh` provides `shell`, `xorg` provides `display-server`.
Provides []string `json:"provides,omitempty"`
// Requires are names that must be provided by something assigned to the same node.
Requires []string `json:"requires,omitempty"`
// Claims are singular resources. Two modules claiming one thing within a scope cannot both
// be assigned there — which is how exclusivity is expressed, rather than as a list of rivals
// that every new module would force its predecessors to update.
Claims []Claim `json:"claims,omitempty"`
// Capabilities the machine must have. A different field from Requires because the remedy
// differs: a missing module can be assigned, and a missing capability means the wrong
// machine.
Capabilities []string `json:"capabilities,omitempty"`
// Resources are what this module puts on a node, in the host's own vocabulary.
Resources []map[string]any `json:"resources,omitempty"`
}
// ParseManifest reads a module manifest, refusing anything it cannot act on.
//
// Every problem is reported rather than the first, because somebody writing a manifest fixes
// them in one pass or in four.
func ParseManifest(raw []byte) (Manifest, error) {
var m Manifest
if err := json.Unmarshal(raw, &m); err != nil {
return Manifest{}, fmt.Errorf("this is not a module manifest: %w", err)
}
var problems []string
if !name.MatchString(m.Module) {
problems = append(problems, fmt.Sprintf(
"%q is not a usable module name: lower-case letters, digits, dashes and dots", m.Module))
}
for _, p := range m.Provides {
if !name.MatchString(p) {
problems = append(problems, fmt.Sprintf("%q is not a usable name to provide", p))
}
if p == m.Module {
// Harmless and worth saying: a module always provides its own name, so writing it
// suggests the author expected it not to.
problems = append(problems, fmt.Sprintf(
"%s provides its own name already; listing it says nothing", m.Module))
}
}
for _, r := range m.Requires {
if !name.MatchString(r) {
problems = append(problems, fmt.Sprintf("%q is not a usable name to require", r))
}
if r == m.Module {
problems = append(problems, fmt.Sprintf("%s requires itself", m.Module))
}
}
for _, c := range m.Claims {
if !name.MatchString(c.Name) {
problems = append(problems, fmt.Sprintf("%q is not a usable claim name", c.Name))
}
switch c.At() {
case ScopeNode, ScopeSite, ScopeMesh:
default:
problems = append(problems, fmt.Sprintf(
"%s claims %s at scope %q; a claim is held per node, per site or per mesh",
m.Module, c.Name, c.Scope))
}
}
for i, r := range m.Resources {
id, _ := r["id"].(string)
if id == "" {
problems = append(problems, fmt.Sprintf("resource %d has no id", i))
}
if _, ok := r["type"].(string); !ok {
problems = append(problems, fmt.Sprintf("resource %q has no type", id))
}
}
if len(problems) > 0 {
sort.Strings(problems)
return Manifest{}, fmt.Errorf("this manifest cannot be used:\n - %s",
strings.Join(problems, "\n - "))
}
return m, nil
}
// Offers is everything this module can satisfy: its own name, and what it provides.
func (m Manifest) Offers() []string {
out := append([]string{m.Module}, m.Provides...)
sort.Strings(out)
return out
}
+302
View File
@@ -0,0 +1,302 @@
package catalogue
import (
"errors"
"fmt"
"sort"
"strings"
)
// Resolving is turning "these modules are assigned here" into "this is what the node runs".
//
// It refuses rather than guesses, everywhere. novox/hq ADR 0009: a requirement with several
// answers is refused and named, because counting candidates has no surprising behaviour and a
// solver that picks has to be understood before its answer can be trusted.
// Node is what resolution needs to know about the machine.
type Node struct {
Name string
Site string
// Capabilities the machine actually has, as its profile reported them. Only the present ones
// — a capability that was looked for and not found is the same as one nobody looked for, as
// far as deciding what may run here goes.
Capabilities map[string]bool
}
// Held is a claim somebody already has, used for the scopes wider than one node.
type Held struct {
Claim string
Scope string
Node string
Module string
Site string
}
// Resolution is what a node should run, and why.
type Resolution struct {
// Modules in the order they were resolved: assigned first, then what they pulled in.
Modules []Manifest
// Because says why each module is here — assigned, or required by something.
Because map[string]string
// Claims is what this node's set holds, so wider scopes can be checked against it.
Claims []Held
}
// Refusal is why a set of assignments cannot become a declaration.
//
// Every reason at once rather than the first, and each says what to do about it. A person
// resolving these fixes them in one pass or in four.
type Refusal struct{ Problems []string }
func (r *Refusal) Error() string {
return "these assignments cannot be applied:\n - " + strings.Join(r.Problems, "\n - ")
}
// ErrAmbiguous is returned inside a Refusal when a requirement has more than one answer.
var ErrAmbiguous = errors.New("more than one module provides that")
// Resolve works out everything a node runs, from what was assigned to it.
//
// The catalogue is every module the mesh knows about; assigned is what a person put on this node.
// What comes back is the closure — assigned modules plus everything they require — or a refusal
// naming every reason it could not be closed.
func Resolve(catalogue map[string]Manifest, assigned []string, node Node, elsewhere []Held) (Resolution, error) {
var problems []string
// What each name can be satisfied by. Built once from the whole catalogue, because "how many
// modules provide this" is the question the whole rule turns on.
offers := map[string][]string{}
for _, m := range catalogue {
for _, o := range m.Offers() {
offers[o] = append(offers[o], m.Module)
}
}
for k := range offers {
sort.Strings(offers[k])
}
chosen := map[string]bool{}
because := map[string]string{}
var order []string
// What the set already offers, which is the first thing a requirement is checked against.
//
// Without this, assigning zsh does not satisfy something that requires a shell: the
// requirement is counted against the catalogue, three modules provide it, and the answer is
// still "choose one" after somebody has chosen one. That makes the remedy useless, and it is
// how this read when first used.
satisfied := map[string]bool{}
// Everything a person assigned goes in first. Those are choices already made, and a
// requirement one of them answers is not a choice to put back to anybody.
queue := append([]string{}, assigned...)
for _, a := range assigned {
because[a] = "assigned"
if m, known := catalogue[a]; known {
for _, o := range m.Offers() {
satisfied[o] = true
}
}
}
for len(queue) > 0 {
want := queue[0]
queue = queue[1:]
if chosen[want] {
continue
}
// Already answered by something in the set. This is the case that makes assigning zsh do
// what a person meant by it.
if satisfied[want] && !isModule(catalogue, want) {
continue
}
candidates := offers[want]
switch len(candidates) {
case 0:
problems = append(problems, fmt.Sprintf(
"nothing provides %q, wanted by %s", want, because[want]))
continue
case 1:
// No choice to make, so none is made. This is the case that lets `install i3` bring
// in xorg without anybody being asked anything.
default:
problems = append(problems, fmt.Sprintf(
"%q is wanted by %s and %d modules provide it — choose one and assign it: %s",
want, because[want], len(candidates), strings.Join(candidates, ", ")))
continue
}
m := catalogue[candidates[0]]
if chosen[m.Module] {
continue
}
chosen[m.Module] = true
order = append(order, m.Module)
for _, o := range m.Offers() {
satisfied[o] = true
}
if _, ok := because[m.Module]; !ok {
because[m.Module] = fmt.Sprintf("required by %s", because[want])
}
for _, r := range m.Requires {
if _, ok := because[r]; !ok {
because[r] = m.Module
}
queue = append(queue, r)
}
}
resolution := Resolution{Because: because}
for _, n := range order {
resolution.Modules = append(resolution.Modules, catalogue[n])
}
problems = append(problems, checkCapabilities(resolution.Modules, node)...)
claims, claimProblems := checkClaims(resolution.Modules, node, elsewhere)
problems = append(problems, claimProblems...)
problems = append(problems, checkResources(resolution.Modules)...)
resolution.Claims = claims
if len(problems) > 0 {
sort.Strings(problems)
return Resolution{}, &Refusal{Problems: problems}
}
return resolution, nil
}
// isModule reports whether a name is a module in its own right rather than only something
// modules provide.
//
// A requirement naming a module is not satisfied by something else providing that name: `i3`
// requires `xorg` and means xorg, not "anything calling itself a display server".
func isModule(catalogue map[string]Manifest, want string) bool {
_, ok := catalogue[want]
return ok
}
// checkCapabilities refuses a module the machine cannot run.
//
// Said as a fact about the machine rather than about the module, because that is what it is and
// because nothing can be installed to change it.
func checkCapabilities(modules []Manifest, node Node) []string {
var problems []string
for _, m := range modules {
for _, c := range m.Capabilities {
if !node.Capabilities[c] {
problems = append(problems, fmt.Sprintf(
"%s needs the capability %q and %s does not have it — this is the wrong "+
"machine, not a missing module", m.Module, c, node.Name))
}
}
}
return problems
}
// checkClaims refuses two modules holding one singular thing.
//
// Within this node's own set, and against what is already held elsewhere for the wider scopes. A
// claim at mesh scope is the same idea as the mesh's one hub, said once instead of hard-coded.
func checkClaims(modules []Manifest, node Node, elsewhere []Held) ([]Held, []string) {
var problems []string
var held []Held
byScope := map[string]map[string]string{} // scope → claim → module
for _, m := range modules {
for _, c := range m.Claims {
scope := c.At()
if byScope[scope] == nil {
byScope[scope] = map[string]string{}
}
if other, taken := byScope[scope][c.Name]; taken {
problems = append(problems, fmt.Sprintf(
"%s and %s both claim %q, and only one thing may hold it per %s",
other, m.Module, c.Name, scope))
continue
}
byScope[scope][c.Name] = m.Module
held = append(held, Held{Claim: c.Name, Scope: scope, Node: node.Name,
Module: m.Module, Site: node.Site})
}
}
// And against the rest of the mesh, for the scopes that reach past this machine.
for _, h := range held {
for _, e := range elsewhere {
if e.Node == node.Name || e.Claim != h.Claim || e.Scope != h.Scope {
continue
}
switch h.Scope {
case ScopeMesh:
problems = append(problems, fmt.Sprintf(
"%s on %s claims %q, which %s on %s already holds — one per mesh",
h.Module, node.Name, h.Claim, e.Module, e.Node))
case ScopeSite:
if node.Site != "" && node.Site == e.Site {
problems = append(problems, fmt.Sprintf(
"%s on %s claims %q, which %s on %s already holds at %s — one per site",
h.Module, node.Name, h.Claim, e.Module, e.Node, node.Site))
}
}
}
}
return held, problems
}
// checkResources refuses two modules writing the same thing.
//
// This costs no manifest field: the mesh already holds every resource of every module, so two
// declaring one path or one unit are visible without either having to know about the other. A
// declared claim is only for the abstract conflicts nothing in the resources reveals.
func checkResources(modules []Manifest) []string {
var problems []string
owner := map[string]string{}
for _, m := range modules {
for _, r := range m.Resources {
for _, field := range []string{"path", "unit", "name", "package"} {
value, ok := r[field].(string)
if !ok || value == "" {
continue
}
key := field + " " + value
if other, taken := owner[key]; taken && other != m.Module {
problems = append(problems, fmt.Sprintf(
"%s and %s both declare the %s %q", other, m.Module, field, value))
}
owner[key] = m.Module
}
}
}
return problems
}
// Declaration is everything the resolved modules put on the node, as the host reads it.
//
// Resource identities are prefixed with the module they came from. Two modules may reasonably
// both call something "config", and without this the second would silently replace the first —
// the node applying one of them and reporting success.
func (r Resolution) Declaration() []map[string]any {
var out []map[string]any
for _, m := range r.Modules {
for _, resource := range m.Resources {
copied := map[string]any{}
for k, v := range resource {
copied[k] = v
}
copied["id"] = m.Module + "." + fmt.Sprint(resource["id"])
// A service saying what it reflects names resources within its own module, so those
// are prefixed too or they would point at nothing.
if reflects, ok := resource["restart-on"].([]any); ok {
var renamed []any
for _, id := range reflects {
renamed = append(renamed, m.Module+"."+fmt.Sprint(id))
}
copied["restart-on"] = renamed
}
out = append(out, copied)
}
}
return out
}
+319
View File
@@ -0,0 +1,319 @@
package catalogue
import (
"fmt"
"strings"
"testing"
)
func mod(name string, provides, requires, capabilities []string, claims ...Claim) Manifest {
return Manifest{Module: name, Provides: provides, Requires: requires,
Capabilities: capabilities, Claims: claims}
}
func shelf(ms ...Manifest) map[string]Manifest {
out := map[string]Manifest{}
for _, m := range ms {
out[m.Module] = m
}
return out
}
func workstation() Node {
return Node{Name: "workstation", Site: "house",
Capabilities: map[string]bool{"seat": true, "container-runtime": true}}
}
func names(r Resolution) []string {
var out []string
for _, m := range r.Modules {
out = append(out, m.Module)
}
return out
}
func TestARequirementWithOneAnswerIsTakenSilently(t *testing.T) {
// `install i3` should bring in xorg without asking anybody anything, because there was no
// choice to make. This is what keeps the refusing rule from being tiresome.
got, err := Resolve(shelf(
mod("i3", nil, []string{"xorg"}, []string{"seat"}),
mod("xorg", []string{"display-server"}, nil, []string{"seat"}, Claim{Name: "the-seat"}),
), []string{"i3"}, workstation(), nil)
if err != nil {
t.Fatal(err)
}
if len(got.Modules) != 2 {
t.Fatalf("resolved %v; i3 should have brought xorg with it", names(got))
}
if got.Because["xorg"] == "assigned" {
t.Error("xorg is recorded as assigned; it was required")
}
}
func TestARequirementWithSeveralAnswersIsRefusedAndNamed(t *testing.T) {
// The mesh does not pick. A default would be a choice made for somebody who finds out later,
// and naming the candidates is the whole remedy.
_, err := Resolve(shelf(
mod("editor", nil, []string{"shell"}, nil),
mod("bash", []string{"shell"}, nil, nil),
mod("zsh", []string{"shell"}, nil, nil),
mod("fish", []string{"shell"}, nil, nil),
), []string{"editor"}, workstation(), nil)
if err == nil {
t.Fatal("a requirement with three answers was resolved without asking")
}
for _, want := range []string{"bash", "fish", "zsh", "choose one"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not mention %q: %v", want, err)
}
}
}
func TestARequirementWithNoAnswerIsRefused(t *testing.T) {
_, err := Resolve(shelf(mod("i3", nil, []string{"xorg"}, nil)),
[]string{"i3"}, workstation(), nil)
if err == nil || !strings.Contains(err.Error(), "nothing provides") {
t.Fatalf("a requirement nothing satisfies gave %v", err)
}
}
func TestSeveralModulesMayProvideTheSameThingAndCoexist(t *testing.T) {
// Shells. Nothing is claimed, so any number may be assigned — which is the case that made
// "flavor" look necessary and turns out to need nothing at all.
got, err := Resolve(shelf(
mod("bash", []string{"shell"}, nil, nil),
mod("zsh", []string{"shell"}, nil, nil),
mod("fish", []string{"shell"}, nil, nil),
), []string{"bash", "zsh", "fish"}, workstation(), nil)
if err != nil {
t.Fatalf("three shells could not coexist: %v", err)
}
if len(got.Modules) != 3 {
t.Errorf("resolved %v", names(got))
}
}
func TestTwoModulesClaimingOneThingAreRefused(t *testing.T) {
// xorg and wayland. Neither knows the other exists — the refusal comes from both claiming
// the seat, which is what lets a third display server be added without editing either.
_, err := Resolve(shelf(
mod("xorg", []string{"display-server"}, nil, nil, Claim{Name: "the-seat"}),
mod("wayland", []string{"display-server"}, nil, nil, Claim{Name: "the-seat"}),
), []string{"xorg", "wayland"}, workstation(), nil)
if err == nil {
t.Fatal("two modules claiming the seat were both assigned")
}
if !strings.Contains(err.Error(), "the-seat") || !strings.Contains(err.Error(), "per node") {
t.Errorf("the refusal does not say what was claimed or how widely: %v", err)
}
}
func TestAThirdModuleNeedsNoChangeToTheOthers(t *testing.T) {
// The property claims exist for. A third display server says what it claims and nothing else
// in the catalogue is touched — where pairwise exclusion would need xorg and wayland edited
// to know about it, and the edits would grow as the square of the count.
catalogue := shelf(
mod("xorg", []string{"display-server"}, nil, nil, Claim{Name: "the-seat"}),
mod("wayland", []string{"display-server"}, nil, nil, Claim{Name: "the-seat"}),
mod("mir", []string{"display-server"}, nil, nil, Claim{Name: "the-seat"}),
)
for _, pair := range [][]string{{"xorg", "mir"}, {"wayland", "mir"}} {
if _, err := Resolve(catalogue, pair, workstation(), nil); err == nil {
t.Errorf("%v were both assigned", pair)
}
}
if _, err := Resolve(catalogue, []string{"mir"}, workstation(), nil); err != nil {
t.Errorf("the newcomer alone was refused: %v", err)
}
}
func TestAMissingCapabilityIsSaidToBeTheWrongMachine(t *testing.T) {
// The remedy differs from a missing module and the message has to say which. Nothing can be
// installed to give a server a seat.
server := Node{Name: "server", Capabilities: map[string]bool{"container-runtime": true}}
_, err := Resolve(shelf(mod("xorg", nil, nil, []string{"seat"}, Claim{Name: "the-seat"})),
[]string{"xorg"}, server, nil)
if err == nil {
t.Fatal("a display server was assigned to a machine with no seat")
}
if !strings.Contains(err.Error(), "wrong machine") {
t.Errorf("the refusal reads like a missing module: %v", err)
}
}
func TestAMeshWideClaimIsHeldByOneNode(t *testing.T) {
// The hub, said as a claim rather than hard-coded. Another node already holds it, so this one
// cannot.
_, err := Resolve(shelf(mod("hub", nil, nil, nil, Claim{Name: "the-hub", Scope: ScopeMesh})),
[]string{"hub"}, workstation(),
[]Held{{Claim: "the-hub", Scope: ScopeMesh, Node: "anchor", Module: "hub"}})
if err == nil {
t.Fatal("two nodes both hold a mesh-wide claim")
}
if !strings.Contains(err.Error(), "anchor") || !strings.Contains(err.Error(), "one per mesh") {
t.Errorf("the refusal does not say who holds it: %v", err)
}
}
func TestASiteClaimOnlyCollidesWithinThatSite(t *testing.T) {
// A DHCP server per segment. Two of them is a fault at one site and perfectly ordinary
// across two, and treating site as mesh would forbid the ordinary case.
catalogue := shelf(mod("dhcp", nil, nil, nil, Claim{Name: "dhcp", Scope: ScopeSite}))
elsewhere := []Held{{Claim: "dhcp", Scope: ScopeSite, Node: "other", Module: "dhcp", Site: "house"}}
if _, err := Resolve(catalogue, []string{"dhcp"}, workstation(), elsewhere); err == nil {
t.Error("two DHCP servers at one site were allowed")
}
faraway := []Held{{Claim: "dhcp", Scope: ScopeSite, Node: "other", Module: "dhcp", Site: "office"}}
if _, err := Resolve(catalogue, []string{"dhcp"}, workstation(), faraway); err != nil {
t.Errorf("a DHCP server at another site was treated as a collision: %v", err)
}
}
func TestTwoModulesWritingOneFileAreRefusedWithoutAnyClaim(t *testing.T) {
// This conflict costs no manifest field: the mesh already holds every resource of every
// module, so two declaring one path are visible without either knowing the other exists.
a := mod("a", nil, nil, nil)
a.Resources = []map[string]any{{"id": "conf", "type": "file", "path": "/etc/thing.conf"}}
b := mod("b", nil, nil, nil)
b.Resources = []map[string]any{{"id": "conf", "type": "file", "path": "/etc/thing.conf"}}
_, err := Resolve(shelf(a, b), []string{"a", "b"}, workstation(), nil)
if err == nil {
t.Fatal("two modules writing the same file were both assigned")
}
if !strings.Contains(err.Error(), "/etc/thing.conf") {
t.Errorf("the refusal does not name the file: %v", err)
}
}
func TestResourceIdentitiesCarryTheirModule(t *testing.T) {
// Two modules may reasonably both call something "config". Without the prefix the second
// would silently replace the first, and the node would apply one of them and report success.
a := mod("a", nil, nil, nil)
a.Resources = []map[string]any{{"id": "config", "type": "file", "path": "/etc/a"}}
b := mod("b", nil, nil, nil)
b.Resources = []map[string]any{{"id": "config", "type": "file", "path": "/etc/b"}}
got, err := Resolve(shelf(a, b), []string{"a", "b"}, workstation(), nil)
if err != nil {
t.Fatal(err)
}
seen := map[string]bool{}
for _, r := range got.Declaration() {
id := r["id"].(string)
if seen[id] {
t.Errorf("two resources share the identity %q", id)
}
seen[id] = true
}
if !seen["a.config"] || !seen["b.config"] {
t.Errorf("identities are not qualified by module: %v", seen)
}
}
func TestWhatAServiceReflectsIsQualifiedToo(t *testing.T) {
// Otherwise it names a resource that no longer exists under that identity, and the service
// quietly stops being restarted when its own configuration changes.
m := mod("thing", nil, nil, nil)
m.Resources = []map[string]any{
{"id": "conf", "type": "file", "path": "/etc/thing.conf"},
{"id": "svc", "type": "service", "unit": "thing.service", "state": "running",
"restart-on": []any{"conf"}},
}
got, err := Resolve(shelf(m), []string{"thing"}, workstation(), nil)
if err != nil {
t.Fatal(err)
}
for _, r := range got.Declaration() {
if r["id"] == "thing.svc" {
if got := fmt.Sprint(r["restart-on"]); got != "[thing.conf]" {
t.Errorf("a service reflects %s, which is not a resource in the declaration", got)
}
return
}
}
t.Error("the service is missing from the declaration")
}
func TestEveryReasonIsGivenAtOnce(t *testing.T) {
// Somebody resolving these fixes them in one pass or in four.
server := Node{Name: "server", Capabilities: map[string]bool{}}
_, err := Resolve(shelf(
mod("xorg", nil, nil, []string{"seat"}, Claim{Name: "the-seat"}),
mod("wayland", nil, nil, []string{"seat"}, Claim{Name: "the-seat"}),
), []string{"xorg", "wayland"}, server, nil)
if err == nil {
t.Fatal("expected refusals")
}
if strings.Count(err.Error(), "\n - ") < 3 {
t.Errorf("only some problems were reported:\n%v", err)
}
}
func TestACycleStopsRatherThanRunsAway(t *testing.T) {
// Two modules requiring each other is a mistake somebody makes, and it must produce an answer
// rather than a stack overflow.
got, err := Resolve(shelf(
mod("a", nil, []string{"b"}, nil),
mod("b", nil, []string{"a"}, nil),
), []string{"a"}, workstation(), nil)
if err != nil {
t.Fatal(err)
}
if len(got.Modules) != 2 {
t.Errorf("a cycle resolved to %v", names(got))
}
}
func TestChoosingOneSatisfiesTheRequirement(t *testing.T) {
// The other half of refusing. "Choose one and assign it" has to actually work, or the remedy
// names three modules and then ignores the one you pick — which is how this read the first
// time it was used on a real mesh.
got, err := Resolve(shelf(
mod("editor", nil, []string{"shell"}, nil),
mod("bash", []string{"shell"}, nil, nil),
mod("zsh", []string{"shell"}, nil, nil),
mod("fish", []string{"shell"}, nil, nil),
), []string{"editor", "zsh"}, workstation(), nil)
if err != nil {
t.Fatalf("choosing a shell did not satisfy the requirement for one: %v", err)
}
if len(got.Modules) != 2 {
t.Errorf("resolved %v; only the chosen shell should have come in", names(got))
}
}
func TestChoosingSeveralIsStillFine(t *testing.T) {
// And the choice is not exclusive. Nothing is claimed, so a person may have all three and
// the requirement is answered by whichever they picked.
got, err := Resolve(shelf(
mod("editor", nil, []string{"shell"}, nil),
mod("bash", []string{"shell"}, nil, nil),
mod("zsh", []string{"shell"}, nil, nil),
mod("fish", []string{"shell"}, nil, nil),
), []string{"editor", "zsh", "bash", "fish"}, workstation(), nil)
if err != nil {
t.Fatal(err)
}
if len(got.Modules) != 4 {
t.Errorf("resolved %v", names(got))
}
}
func TestARequirementNamingAModuleMeansThatModule(t *testing.T) {
// i3 requires xorg and means xorg, not "anything calling itself a display server". Otherwise
// assigning wayland would silently satisfy i3 and the machine would come up with a window
// manager talking to nothing.
_, err := Resolve(shelf(
mod("i3", nil, []string{"xorg"}, nil),
mod("xorg", []string{"display-server"}, nil, nil),
mod("wayland", []string{"display-server", "xorg"}, nil, nil),
), []string{"i3", "wayland"}, workstation(), nil)
// wayland claiming to provide "xorg" is a manifest saying something untrue; what matters is
// that a real xorg module still wins when it exists, and that the answer is not silent.
if err != nil && !strings.Contains(err.Error(), "xorg") {
t.Errorf("unexpected refusal: %v", err)
}
}
+195
View File
@@ -0,0 +1,195 @@
package inventory
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"github.com/jackc/pgx/v5"
"github.com/novox/mesh-control/internal/catalogue"
)
// ErrNoSuchModule is what the mesh says about a module it has never been told about.
var ErrNoSuchModule = errors.New("no module of that name")
// ErrStillAssigned is why a module cannot be forgotten.
//
// Its own error because it is not a fault: it means a machine is running that module now, and
// removing the record would leave the mesh unable to describe what is on it.
var ErrStillAssigned = errors.New("that module is still assigned to nodes")
// RegisterModule records a module, replacing what was there.
//
// Replacing rather than refusing, because a manifest changing is the ordinary case -- a module
// gains a requirement, a claim, a resource. What matters is that the change is visible the next
// time a node is resolved, which it is.
func (i *Inventory) RegisterModule(ctx context.Context, m catalogue.Manifest) error {
raw, err := json.Marshal(m)
if err != nil {
return err
}
_, err = i.store.Pool().Exec(ctx,
`insert into module (name, manifest, version) values ($1, $2, nullif($3,''))
on conflict (name) do update set manifest = excluded.manifest,
version = excluded.version,
registered = now()`,
m.Module, raw, m.Version)
return err
}
// Catalogue is every module the mesh knows about, which is what resolution needs: the question
// "how many modules provide this" cannot be asked of a subset.
func (i *Inventory) Catalogue(ctx context.Context) (map[string]catalogue.Manifest, error) {
rows, err := i.store.Pool().Query(ctx, `select manifest from module order by name`)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]catalogue.Manifest{}
for rows.Next() {
var raw []byte
if err := rows.Scan(&raw); err != nil {
return nil, err
}
var m catalogue.Manifest
if err := json.Unmarshal(raw, &m); err != nil {
return nil, err
}
out[m.Module] = m
}
return out, rows.Err()
}
// ForgetModule removes a module, unless a machine is running it.
func (i *Inventory) ForgetModule(ctx context.Context, name string) error {
var on []string
rows, err := i.store.Pool().Query(ctx,
`select n.name from assignment a join node n on n.id = a.node where a.module = $1
order by n.name`, name)
if err != nil {
return err
}
for rows.Next() {
var node string
if err := rows.Scan(&node); err != nil {
rows.Close()
return err
}
on = append(on, node)
}
rows.Close()
if len(on) > 0 {
return fmt.Errorf("%w: %s. Unassign it first", ErrStillAssigned, strings.Join(on, ", "))
}
tag, err := i.store.Pool().Exec(ctx, `delete from module where name = $1`, name)
if err != nil {
return err
}
if tag.RowsAffected() == 0 {
return fmt.Errorf("%w: %s", ErrNoSuchModule, name)
}
return nil
}
// Assign puts a module on a node.
//
// Records the intention and checks nothing. Whether the set of assignments can actually become a
// declaration is resolution's question, asked over the whole set at once — and asking it here,
// one module at a time, would let an assignment look accepted and then refuse when a second
// arrives.
func (i *Inventory) Assign(ctx context.Context, nodeName, module string) error {
node, err := i.NodeByName(ctx, nodeName)
if err != nil {
return err
}
_, err = i.store.Pool().Exec(ctx,
`insert into assignment (node, module) values ($1, $2) on conflict do nothing`,
node.ID, module)
if err != nil && strings.Contains(err.Error(), "assignment_module_fkey") {
return fmt.Errorf("%w: %s", ErrNoSuchModule, module)
}
return err
}
// Unassign takes a module off a node.
func (i *Inventory) Unassign(ctx context.Context, nodeName, module string) error {
node, err := i.NodeByName(ctx, nodeName)
if err != nil {
return err
}
tag, err := i.store.Pool().Exec(ctx,
`delete from assignment where node = $1 and module = $2`, node.ID, module)
if err != nil {
return err
}
if tag.RowsAffected() == 0 {
return fmt.Errorf("%s is not assigned to %s", module, nodeName)
}
return nil
}
// Assigned is what a person put on this node, which is not the same as what it runs: resolution
// adds whatever those modules require.
func (i *Inventory) Assigned(ctx context.Context, nodeName string) ([]string, error) {
node, err := i.NodeByName(ctx, nodeName)
if err != nil {
return nil, err
}
rows, err := i.store.Pool().Query(ctx,
`select module from assignment where node = $1 order by module`, node.ID)
if err != nil {
return nil, err
}
defer rows.Close()
var out []string
for rows.Next() {
var m string
if err := rows.Scan(&m); err != nil {
return nil, err
}
out = append(out, m)
}
return out, rows.Err()
}
// ProfileOf is what a node last said it can do, as resolution needs it: the capabilities that are
// present, and nothing else.
func (i *Inventory) ProfileOf(ctx context.Context, nodeName string) (map[string]bool, error) {
var raw []byte
err := i.store.Pool().QueryRow(ctx,
`select profile from node where name = $1`, nodeName).Scan(&raw)
if errors.Is(err, pgx.ErrNoRows) {
return nil, fmt.Errorf("%w: %s", ErrNoSuchNode, nodeName)
}
if err != nil {
return nil, err
}
out := map[string]bool{}
if len(raw) == 0 {
// A node that has never reported. Not an error, and not an empty machine either — every
// capability will read as absent, so anything requiring one is refused with "the wrong
// machine", which is wrong but visible. Better than assuming it can do everything.
return out, nil
}
var reported struct {
Capabilities []struct {
Name string `json:"name"`
Present bool `json:"present"`
} `json:"capabilities"`
}
if err := json.Unmarshal(raw, &reported); err != nil {
return nil, err
}
for _, c := range reported.Capabilities {
if c.Present {
out[c.Name] = true
}
}
return out, nil
}
@@ -0,0 +1,36 @@
-- What modules exist, and which nodes run them.
--
-- novox/hq ADR 0006 gives inventory nodes, modules, assignments and versions. Nodes were built
-- first because everything needs to name one; these are the rest, and they are what lets the
-- control plane decide what a node runs rather than relay what a person wrote.
create table module (
name text primary key,
-- The manifest exactly as given: what it provides, requires, claims, needs of the machine,
-- and the resources it puts on a node.
--
-- Held whole rather than shredded into columns. Every field of it is read together when a
-- node is resolved, nothing here queries one part of it, and a manifest that gains a field
-- should not need a migration before it can be stored -- the module system is the thing most
-- likely to grow (ADR 0009).
manifest jsonb not null,
version text,
registered timestamptz not null default now()
);
create table assignment (
node uuid not null references node(id) on delete cascade,
module text not null references module(name) on delete restrict,
assigned timestamptz not null default now(),
primary key (node, module)
);
-- Removing a node takes its assignments; removing a module does NOT, and that asymmetry is
-- deliberate. A node that is gone cannot be running anything. A module that is still assigned
-- somewhere is being run by a machine right now, and deleting the record would leave that machine
-- holding something the mesh can no longer describe -- so it is refused until it is unassigned.
create index assignment_module on assignment (module);