README.md

@@ -22,13 +22,17 @@ the host never queries the mesh database. It receives declarations and applies t
 
 ## What exists today
 
-**Stage 1 only: it reports.** It applies nothing, connects to nothing, and listens on nothing.
+**Stages 1 and 2.** It reports what a machine is, and it applies a declaration to one. It
+connects to nothing and listens on nothing — what it applies comes from a file.
 
 ```
-mesh-host profile      what this machine can be asked to do
-mesh-host inventory    what this machine is, and what it holds
-  --json               machine-readable
-  --timeout            how long any single probe may take (default 10s)
+mesh-host profile        what this machine can be asked to do
+mesh-host inventory      what this machine is, and what it holds
+mesh-host apply FILE     make this machine match a declaration
+mesh-host owned          what this host has applied and still owns
+  --json                 machine-readable
+  --state                where this node keeps what it knows
+  --dry-run              read and check the declaration, change nothing
 ```
 
 ```
@@ -46,8 +50,35 @@ linux/amd64
 cannot be asked to: [firewall privileged]
 ```
 
-Stages 2 to 4 — applying from a pinned bundle, the link and the local store, and enrolment —
-are designed and not built.
+## Applying
+
+A declaration is JSON, versioned, and an **ordered list** of resources — the order is stated
+rather than derived, because deriving it would be the host deciding
+([`novox/hq` ADR 0043](https://git.novox.be/novox/hq)). The vocabulary is `directory`, `file`
+and `service`, and **anything outside it refuses the whole declaration**: a host that skipped
+what it did not understand would apply most of a declaration and report success.
+
+```json
+{"declaration":1,"resources":[
+  {"id":"mesh-etc","type":"directory","path":"/etc/mesh","mode":"0755"},
+  {"id":"node-conf","type":"file","path":"/etc/mesh/node.conf","content":"role = anchor\n","mode":"0640"},
+  {"id":"journal","type":"service","unit":"systemd-journald.service","state":"running"}
+]}
+```
+
+**It converges rather than executes.** Applying twice changes nothing the second time; applying
+to a drifted machine returns it. A mode is *maintained*, not merely set — a permission applied
+at creation is not a permission held.
+
+**It owns a footprint, and only that.** What it applied and is no longer declared is removed;
+what it did not create is never touched. It knows which is which because it recorded what it
+did, after each thing worked.
+
+**A failed step fails the apply.** No step runs after a failure, and the error carries what had
+already been done — the machine is in whatever state that left it, and pretending otherwise is
+the fault this exists to prevent.
+
+Stages 3 and 4 — the link, and enrolment — are designed and not built.
 
 ## A capability is detected, never assumed
 
@@ -63,6 +94,12 @@ firewall is asked to list a ruleset, which needs the privilege as well as the to
 nobody can act on. The reason is what a person reads when a node will not take work they
 expected it to take.
 
+**A unit that does not exist is not a unit that is stopped.** `systemctl is-active` says
+`inactive` for both, so declaring a unit stopped reported success for a unit the host cannot
+manage at all. `LoadState` separates them. Found by applying inside a raised machine, not by
+reasoning — and its sibling: removing an orphaned service whose unit has since been uninstalled
+used to fail the whole apply, which left a node able to apply *nothing*, ever.
+
 **Exit codes are not the whole answer.** Found by running against a real machine rather than by
 reasoning: `systemctl is-system-running` exits non-zero for every state except `running` —
 including `degraded`, which means some units failed and the init is emphatically there. Reading

cmd/mesh-host/main.go

@@ -9,6 +9,7 @@ package main
 import (
 	"context"
 	"encoding/json"
+	"errors"
 	"flag"
 	"fmt"
 	"os"
@@ -17,8 +18,11 @@ import (
 	"text/tabwriter"
 	"time"
 
+	"github.com/novox/mesh-host/internal/apply"
+	"github.com/novox/mesh-host/internal/declaration"
 	"github.com/novox/mesh-host/internal/inventory"
 	"github.com/novox/mesh-host/internal/profile"
+	"github.com/novox/mesh-host/internal/store"
 )
 
 // version is stamped at build time. Unset in a development build, and said so rather than
@@ -29,12 +33,16 @@ const usage = `mesh-host — the node host
 
   profile     what this machine can be asked to do
   inventory   what this machine is, and what it holds
+  apply FILE  make this machine match a declaration
+  owned       what this host has applied and still owns
   version
 
   --json      machine-readable output
   --timeout   how long any single probe may take (default 10s)
+  --state     where this node keeps what it knows (default /var/lib/mesh-host/state.json)
+  --dry-run   read the declaration and refuse it if wrong, but change nothing
 
-Stage 1: reports only. It applies nothing, connects to nothing, listens on nothing.
+It connects to nothing and listens on nothing. What it applies comes from a file.
 `
 
 func main() {
@@ -45,7 +53,7 @@ func main() {
 
 	command, opts, err := parseArgs(os.Args[1:])
 	if err == nil {
-		err = run(ctx, command, opts.json, opts.timeout)
+		err = run(ctx, command, opts)
 	}
 	if err != nil {
 		fmt.Fprintf(os.Stderr, "mesh-host: %v\n", err)
@@ -56,6 +64,9 @@ func main() {
 type options struct {
 	json    bool
 	timeout time.Duration
+	state   string
+	dryRun  bool
+	file    string
 }
 
 // parseArgs takes the subcommand first, then its flags.
@@ -65,7 +76,7 @@ type options struct {
 // passed, silently ignored, with a successful exit. That is the fault this whole project keeps
 // naming, so the parser takes the subcommand off the front and parses what follows.
 func parseArgs(args []string) (string, options, error) {
-	opts := options{timeout: 10 * time.Second}
+	opts := options{timeout: 10 * time.Second, state: store.DefaultPath}
 
 	command := ""
 	if len(args) > 0 {
@@ -78,19 +89,45 @@ func parseArgs(args []string) (string, options, error) {
 	set.Usage = func() { fmt.Fprint(os.Stderr, usage) }
 	set.BoolVar(&opts.json, "json", false, "machine-readable output")
 	set.DurationVar(&opts.timeout, "timeout", opts.timeout, "how long any single probe may take")
+	set.StringVar(&opts.state, "state", opts.state, "where this node keeps what it knows")
+	set.BoolVar(&opts.dryRun, "dry-run", false, "read and check the declaration, change nothing")
 
-	if err := set.Parse(args); err != nil {
-		return "", opts, err
+	// Parsed in a loop, because the standard library stops at the FIRST non-flag argument.
+	// `mesh-host inventory --json` hit that once, and taking the subcommand off the front
+	// fixed only half of it: `mesh-host apply decl.json --dry-run` left --dry-run unread in
+	// exactly the same way. A flag may sit before, after or between positionals, and one that
+	// is silently dropped is the fault this whole project keeps naming.
+	var positionals []string
+	rest := args
+	for {
+		if err := set.Parse(rest); err != nil {
+			return "", opts, err
+		}
+		rest = set.Args()
+		if len(rest) == 0 {
+			break
+		}
+		positionals = append(positionals, rest[0])
+		rest = rest[1:]
+	}
+
+	if command == "apply" {
+		if len(positionals) != 1 {
+			return "", opts, errors.New("apply needs exactly one declaration file")
+		}
+		opts.file = positionals[0]
+		return command, opts, nil
 	}
 	// Anything left over was neither the command nor a flag. Refused rather than ignored: a
 	// mistyped argument that changes nothing and reports success is worse than an error.
-	if rest := set.Args(); len(rest) > 0 {
-		return "", opts, fmt.Errorf("unexpected argument %q — try `mesh-host help`", rest[0])
+	if len(positionals) > 0 {
+		return "", opts, fmt.Errorf("unexpected argument %q — try `mesh-host help`", positionals[0])
 	}
 	return command, opts, nil
 }
 
-func run(ctx context.Context, command string, jsonOut bool, timeout time.Duration) error {
+func run(ctx context.Context, command string, opts options) error {
+	jsonOut, timeout := opts.json, opts.timeout
 	switch command {
 	case "profile":
 		p := profile.Detect(ctx, profile.Default(nil), timeout)
@@ -108,6 +145,27 @@ func run(ctx context.Context, command string, jsonOut bool, timeout time.Duratio
 		writeInventory(inv)
 		return nil
 
+	case "apply":
+		return runApply(ctx, opts)
+
+	case "owned":
+		known, err := store.Load(opts.state)
+		if err != nil {
+			return err
+		}
+		if jsonOut {
+			return writeJSON(known)
+		}
+		if len(known.Resources) == 0 {
+			fmt.Println("this host has applied nothing on this machine")
+			return nil
+		}
+		w := tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', 0)
+		for _, r := range known.Resources {
+			fmt.Fprintf(w, "  %s\t%s\t%s\n", r.Type, r.ID, r.Target)
+		}
+		return w.Flush()
+
 	case "version":
 		fmt.Println(version)
 		return nil
@@ -178,3 +236,59 @@ func writeInventory(inv inventory.Inventory) {
 		}
 	}
 }
+
+// runApply reads a declaration and makes the machine match it.
+//
+// The state is loaded before anything is touched and saved after, including when the apply
+// fails part-way: what was applied before the failure is on the machine, and a host that did
+// not record it would believe it owns less than it does and leave that behind forever.
+func runApply(ctx context.Context, opts options) error {
+	raw, err := os.ReadFile(opts.file)
+	if err != nil {
+		return fmt.Errorf("reading the declaration: %w", err)
+	}
+
+	d, err := declaration.Parse(raw)
+	if err != nil {
+		return err
+	}
+
+	known, err := store.Load(opts.state)
+	if err != nil {
+		return err
+	}
+
+	if opts.dryRun {
+		fmt.Printf("%s: %d resource(s), version %d — accepted, nothing applied\n",
+			opts.file, len(d.Resources), d.Version)
+		return nil
+	}
+
+	report, updated, applyErr := apply.Apply(ctx, d, known, apply.ExecRunner, func(line string) {
+		if !opts.json {
+			fmt.Println(line)
+		}
+	})
+
+	// Saved whichever way it went. Recording only on success would lose the footprint of a
+	// failed apply, and that footprint is on the machine either way.
+	if saveErr := store.Save(opts.state, updated); saveErr != nil {
+		if applyErr != nil {
+			return fmt.Errorf("%w\n\nand the node's state could not be saved: %v", applyErr, saveErr)
+		}
+		return saveErr
+	}
+	if applyErr != nil {
+		return applyErr
+	}
+
+	if opts.json {
+		return writeJSON(report)
+	}
+	if !report.Changed() {
+		fmt.Printf("%s: already matches — %d resource(s) checked\n", opts.file, len(report.Outcomes))
+		return nil
+	}
+	fmt.Printf("%s: applied — %d resource(s)\n", opts.file, len(report.Outcomes))
+	return nil
+}

cmd/mesh-host/main_test.go

@@ -1,6 +1,7 @@
 package main
 
 import (
+	"github.com/novox/mesh-host/internal/store"
 	"testing"
 	"time"
 )
@@ -81,3 +82,56 @@ func TestNoCommandIsNotAnError(t *testing.T) {
 		t.Errorf("command = %q, want empty", command)
 	}
 }
+
+func TestApplyNeedsExactlyOneDeclaration(t *testing.T) {
+	// `apply` takes a file where every other command takes nothing, so the leftover-argument
+	// rule has an exception — and an exception is where a parser stops refusing things it
+	// should. Both directions are checked.
+	if _, _, err := parseArgs([]string{"apply"}); err == nil {
+		t.Error("apply with no file was accepted")
+	}
+	if _, _, err := parseArgs([]string{"apply", "a.json", "b.json"}); err == nil {
+		t.Error("apply with two files was accepted")
+	}
+
+	command, opts, err := parseArgs([]string{"apply", "decl.json", "--dry-run"})
+	if err != nil {
+		t.Fatalf("unexpected error: %v", err)
+	}
+	if command != "apply" || opts.file != "decl.json" || !opts.dryRun {
+		t.Errorf("parsed as command=%q file=%q dry-run=%v", command, opts.file, opts.dryRun)
+	}
+}
+
+func TestTheStateHasADocumentedDefault(t *testing.T) {
+	// A host that wrote its state somewhere unexpected would forget what it owns on the next
+	// run, and then leave everything it had applied behind forever.
+	_, opts, err := parseArgs([]string{"owned"})
+	if err != nil {
+		t.Fatalf("unexpected error: %v", err)
+	}
+	if opts.state != store.DefaultPath {
+		t.Errorf("default state path is %q, not the documented %q", opts.state, store.DefaultPath)
+	}
+}
+
+func TestAFlagAfterAPositionalIsRead(t *testing.T) {
+	// The same fault as TestAFlagAfterTheCommandIsRead, one level down. Taking the subcommand
+	// off the front fixed the flag after the COMMAND and not the flag after its ARGUMENT: the
+	// standard library stops at the first non-flag argument wherever that argument is.
+	for _, args := range [][]string{
+		{"apply", "decl.json", "--dry-run", "--json"},
+		{"apply", "--dry-run", "decl.json", "--json"},
+		{"apply", "--dry-run", "--json", "decl.json"},
+	} {
+		command, opts, err := parseArgs(args)
+		if err != nil {
+			t.Errorf("%v: unexpected error: %v", args, err)
+			continue
+		}
+		if command != "apply" || opts.file != "decl.json" || !opts.dryRun || !opts.json {
+			t.Errorf("%v parsed as file=%q dry-run=%v json=%v",
+				args, opts.file, opts.dryRun, opts.json)
+		}
+	}
+}

internal/apply/apply.go

@@ -0,0 +1,447 @@
+// Package apply makes a machine match a declaration.
+//
+// Three properties, each following a recorded decision, and each of them the difference
+// between this and a script that writes files:
+//
+//   - A failed step fails the apply (novox/hq ADR 0008). Not "logs and continues": a partial
+//     apply that reports success is the mesh's most expensive shape.
+//   - Every applier READS BACK. Setting a value is not evidence the value took.
+//   - What was applied is recorded after it works, never before (ADR 0035). A failed apply
+//     leaves the machine in whatever state it reached, and nothing must claim otherwise.
+package apply
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/novox/mesh-host/internal/declaration"
+	"github.com/novox/mesh-host/internal/store"
+)
+
+// Runner executes a command. The real one is used everywhere outside unit tests; behaviour
+// against a real system is tested alongside rather than mocked (novox/hq ADR 0034).
+type Runner func(ctx context.Context, name string, args ...string) (string, error)
+
+// Outcome is what happened to one resource.
+type Outcome struct {
+	ID     string `json:"id"`
+	Type   string `json:"type"`
+	Target string `json:"target"`
+	Action string `json:"action"` // created · updated · unchanged · removed
+	Detail string `json:"detail,omitempty"`
+}
+
+// Report is what an apply did, in the order it did it.
+type Report struct {
+	Outcomes []Outcome `json:"outcomes"`
+}
+
+// Changed reports whether anything about the machine actually moved. An apply that changed
+// nothing is the ordinary steady state, and saying so is not the same as saying it failed.
+func (r Report) Changed() bool {
+	for _, o := range r.Outcomes {
+		if o.Action != "unchanged" {
+			return true
+		}
+	}
+	return false
+}
+
+// Error is a failure part-way through, carrying what had already been done.
+//
+// The outcomes matter as much as the message: the machine is in whatever state the apply
+// reached, and the only honest thing to hand back is the list of what did happen.
+type Error struct {
+	Resource string
+	Err      error
+	Done     Report
+}
+
+func (e *Error) Error() string {
+	return fmt.Sprintf("applying %q: %v\n\n%d resource(s) were applied before this and remain; "+
+		"the machine is in whatever state that left it.", e.Resource, e.Err, len(e.Done.Outcomes))
+}
+
+func (e *Error) Unwrap() error { return e.Err }
+
+// Apply makes the machine match the declaration, and returns what it did.
+//
+// Removal happens FIRST, and the order is not arbitrary. A resource that leaves a declaration
+// while another arrives at the same path is an ordinary rename: removing afterwards would
+// delete the file that had just been written. Removing first risks losing the old state if the
+// apply then fails — a recovery concern, where the other is a correctness one.
+func Apply(
+	ctx context.Context,
+	d *declaration.Declaration,
+	known store.State,
+	run Runner,
+	log func(string),
+) (Report, store.State, error) {
+	if log == nil {
+		log = func(string) {}
+	}
+	report := Report{}
+
+	declared := map[string]bool{}
+	for _, r := range d.Resources {
+		declared[r.ID] = true
+	}
+
+	for _, orphan := range known.Orphans(declared) {
+		if err := remove(ctx, orphan, run); err != nil {
+			return report, known, &Error{Resource: orphan.ID, Err: err, Done: report}
+		}
+		known.Forget(orphan.ID)
+		report.Outcomes = append(report.Outcomes, Outcome{
+			ID: orphan.ID, Type: orphan.Type, Target: orphan.Target, Action: "removed",
+			Detail: "no longer declared",
+		})
+		log(fmt.Sprintf("  removed %s (%s)", orphan.ID, orphan.Target))
+	}
+
+	for _, resource := range d.Resources {
+		outcome, err := applyOne(ctx, resource, run)
+		if err != nil {
+			return report, known, &Error{Resource: resource.ID, Err: err, Done: report}
+		}
+
+		// Only now. The record follows the fact, never leads it.
+		known.Record(store.Applied{
+			ID: resource.ID, Type: string(resource.Type),
+			Target: outcome.Target, AppliedAt: time.Now().UTC(),
+		})
+		report.Outcomes = append(report.Outcomes, outcome)
+		if outcome.Action != "unchanged" {
+			log(fmt.Sprintf("  %s %s (%s)", outcome.Action, outcome.ID, outcome.Target))
+		}
+	}
+	return report, known, nil
+}
+
+func applyOne(ctx context.Context, r declaration.Resource, run Runner) (Outcome, error) {
+	switch r.Type {
+	case declaration.TypeDirectory:
+		return applyDirectory(r)
+	case declaration.TypeFile:
+		return applyFile(r)
+	case declaration.TypeService:
+		return applyService(ctx, r, run)
+	default:
+		// Unreachable: the declaration refused this already. Present because "unreachable"
+		// stops being true the moment someone adds a type and forgets this switch.
+		return Outcome{}, fmt.Errorf("no applier for type %q", r.Type)
+	}
+}
+
+func modeOf(spec string, fallback os.FileMode) (os.FileMode, error) {
+	if spec == "" {
+		return fallback, nil
+	}
+	parsed, err := strconv.ParseUint(spec, 8, 32)
+	if err != nil {
+		return 0, fmt.Errorf("mode %q: %w", spec, err)
+	}
+	return os.FileMode(parsed), nil
+}
+
+func applyDirectory(r declaration.Resource) (Outcome, error) {
+	out := Outcome{ID: r.ID, Type: string(r.Type), Target: r.Path}
+	mode, err := modeOf(r.Mode, 0o755)
+	if err != nil {
+		return out, err
+	}
+
+	before, err := os.Stat(r.Path)
+	existed := err == nil
+	if err != nil && !errors.Is(err, os.ErrNotExist) {
+		return out, err
+	}
+	if existed && !before.IsDir() {
+		return out, fmt.Errorf("%s exists and is not a directory", r.Path)
+	}
+
+	if !existed {
+		if err := os.MkdirAll(r.Path, mode); err != nil {
+			return out, err
+		}
+	}
+	// Set explicitly even when it existed: MkdirAll applies the mode only on creation, and a
+	// permission set at creation is not a permission maintained — a lesson this repository
+	// already paid for once, with world-readable environment files.
+	if err := os.Chmod(r.Path, mode); err != nil {
+		return out, err
+	}
+
+	// Read back.
+	after, err := os.Stat(r.Path)
+	if err != nil {
+		return out, fmt.Errorf("made %s and cannot stat it: %w", r.Path, err)
+	}
+	if !after.IsDir() {
+		return out, fmt.Errorf("%s is not a directory after applying", r.Path)
+	}
+	if after.Mode().Perm() != mode.Perm() {
+		return out, fmt.Errorf("%s is mode %o after setting %o", r.Path, after.Mode().Perm(), mode.Perm())
+	}
+
+	out.Action = "unchanged"
+	if !existed {
+		out.Action = "created"
+	} else if before.Mode().Perm() != mode.Perm() {
+		out.Action = "updated"
+		out.Detail = fmt.Sprintf("mode %o to %o", before.Mode().Perm(), mode.Perm())
+	}
+	return out, nil
+}
+
+func applyFile(r declaration.Resource) (Outcome, error) {
+	out := Outcome{ID: r.ID, Type: string(r.Type), Target: r.Path}
+	mode, err := modeOf(r.Mode, 0o644)
+	if err != nil {
+		return out, err
+	}
+
+	existing, readErr := os.ReadFile(r.Path)
+	existed := readErr == nil
+	if readErr != nil && !errors.Is(readErr, os.ErrNotExist) {
+		return out, readErr
+	}
+
+	var beforeMode os.FileMode
+	if existed {
+		if info, err := os.Stat(r.Path); err == nil {
+			beforeMode = info.Mode().Perm()
+		}
+	}
+
+	contentSame := existed && string(existing) == r.Content
+	modeSame := existed && beforeMode == mode.Perm()
+
+	if !contentSame {
+		if err := os.MkdirAll(filepath.Dir(r.Path), 0o755); err != nil {
+			return out, err
+		}
+		if err := writeAtomically(r.Path, []byte(r.Content), mode); err != nil {
+			return out, err
+		}
+	} else if !modeSame {
+		if err := os.Chmod(r.Path, mode); err != nil {
+			return out, err
+		}
+	}
+
+	// Read back — the file, not the call that wrote it.
+	written, err := os.ReadFile(r.Path)
+	if err != nil {
+		return out, fmt.Errorf("wrote %s and cannot read it back: %w", r.Path, err)
+	}
+	if string(written) != r.Content {
+		return out, fmt.Errorf("%s does not contain what was declared after writing it", r.Path)
+	}
+	info, err := os.Stat(r.Path)
+	if err != nil {
+		return out, err
+	}
+	if info.Mode().Perm() != mode.Perm() {
+		return out, fmt.Errorf("%s is mode %o after setting %o", r.Path, info.Mode().Perm(), mode.Perm())
+	}
+
+	switch {
+	case !existed:
+		out.Action = "created"
+	case !contentSame && !modeSame:
+		out.Action = "updated"
+		out.Detail = "content and mode"
+	case !contentSame:
+		out.Action = "updated"
+		out.Detail = "content"
+	case !modeSame:
+		out.Action = "updated"
+		out.Detail = fmt.Sprintf("mode %o to %o", beforeMode, mode.Perm())
+	default:
+		out.Action = "unchanged"
+	}
+	return out, nil
+}
+
+// writeAtomically writes through a temporary file in the same directory.
+//
+// A reader of a managed file must never see half of one. The mesh's own configuration is read
+// by daemons that reload on change, so a torn write is a service reading a truncated config.
+func writeAtomically(path string, content []byte, mode os.FileMode) error {
+	tmp, err := os.CreateTemp(filepath.Dir(path), ".mesh-host-*")
+	if err != nil {
+		return err
+	}
+	defer os.Remove(tmp.Name())
+
+	if _, err := tmp.Write(content); err != nil {
+		tmp.Close()
+		return err
+	}
+	if err := tmp.Sync(); err != nil {
+		tmp.Close()
+		return err
+	}
+	if err := tmp.Close(); err != nil {
+		return err
+	}
+	if err := os.Chmod(tmp.Name(), mode); err != nil {
+		return err
+	}
+	return os.Rename(tmp.Name(), path)
+}
+
+func applyService(ctx context.Context, r declaration.Resource, run Runner) (Outcome, error) {
+	out := Outcome{ID: r.ID, Type: string(r.Type), Target: r.Unit}
+
+	before, err := serviceState(ctx, r.Unit, run)
+	if err != nil {
+		return out, err
+	}
+	if before == r.State {
+		out.Action = "unchanged"
+		out.Detail = before
+		return out, nil
+	}
+
+	verb := "start"
+	if r.State == "stopped" {
+		verb = "stop"
+	}
+	if _, err := run(ctx, "systemctl", verb, r.Unit); err != nil {
+		return out, fmt.Errorf("%s %s: %w", verb, r.Unit, err)
+	}
+
+	// Read back. `systemctl start` returning zero says the transaction was accepted, not that
+	// the unit is running — a unit that starts and immediately dies satisfies the command.
+	after, err := serviceState(ctx, r.Unit, run)
+	if err != nil {
+		return out, err
+	}
+	if after != r.State {
+		return out, fmt.Errorf("%s was asked to be %s and is %s", r.Unit, r.State, after)
+	}
+
+	out.Action = "updated"
+	out.Detail = before + " to " + after
+	return out, nil
+}
+
+// serviceState reads what the service manager says about a unit.
+//
+// Two traps here, and both were hit before this read what it now reads.
+//
+// The exit code is not the answer: `is-active` exits non-zero for every state except active —
+// the same shape as the capability detector reading a degraded init as no init at all.
+//
+// And "inactive" does not mean stopped. `systemctl is-active` says "inactive" for a unit that
+// DOES NOT EXIST exactly as it does for one that is installed and stopped. Declaring a unit
+// stopped therefore reported success for a unit the host cannot manage at all — absence read
+// as satisfaction, which is 04-ISSUES/007 wearing a different hat. LoadState is what separates
+// them, so LoadState is what is read.
+func serviceState(ctx context.Context, unit string, run Runner) (string, error) {
+	out, _ := run(ctx, "systemctl", "show", unit,
+		"--property=LoadState", "--property=ActiveState")
+
+	var load, active string
+	for _, line := range strings.Split(out, "\n") {
+		key, value, found := strings.Cut(strings.TrimSpace(line), "=")
+		if !found {
+			continue
+		}
+		switch key {
+		case "LoadState":
+			load = value
+		case "ActiveState":
+			active = value
+		}
+	}
+
+	switch load {
+	case "":
+		return "", fmt.Errorf("the service manager said nothing about %s", unit)
+	case "not-found":
+		return "", fmt.Errorf(
+			"%s does not exist on this machine. A declaration naming a unit that is not "+
+				"installed cannot be satisfied, and reporting it stopped would be reporting "+
+				"absence as success", unit)
+	case "masked":
+		return "", fmt.Errorf("%s is masked, so its state cannot be declared", unit)
+	case "error", "bad-setting":
+		return "", fmt.Errorf("%s is installed but its unit file cannot be loaded (%s)", unit, load)
+	}
+
+	switch active {
+	case "active", "activating", "reloading":
+		return "running", nil
+	case "inactive", "failed", "deactivating":
+		return "stopped", nil
+	default:
+		return "", fmt.Errorf(
+			"the service manager reports %s as %q, which is neither running nor stopped", unit, active)
+	}
+}
+
+// remove undoes one resource the host applied and the declaration no longer names.
+//
+// Only ever called for something in the store, which is what bounds it: the host is
+// authoritative over its own footprint and inert everywhere else (novox/hq ADR 0043).
+func remove(ctx context.Context, a store.Applied, run Runner) error {
+	switch declaration.Type(a.Type) {
+	case declaration.TypeFile, declaration.TypeDirectory:
+		if err := os.RemoveAll(a.Target); err != nil {
+			return err
+		}
+		if _, err := os.Stat(a.Target); !errors.Is(err, os.ErrNotExist) {
+			return fmt.Errorf("%s is still there after removing it", a.Target)
+		}
+		return nil
+
+	case declaration.TypeService:
+		// A unit that is no longer declared is stopped, not deleted. The host did not install
+		// it and does not own the unit file — only the state it put the unit into.
+		//
+		// A unit that no longer EXISTS is already in the state removal is trying to reach, and
+		// saying so matters: stopping it fails, and a failure here fails the whole apply. A
+		// host holding a record of an uninstalled unit would then be unable to apply anything,
+		// ever, with no way out but editing its state by hand. Removal is idempotent for the
+		// same reason `os.RemoveAll` is.
+		if _, err := serviceState(ctx, a.Target, run); err != nil {
+			if strings.Contains(err.Error(), "does not exist on this machine") {
+				return nil
+			}
+			return err
+		}
+		if _, err := run(ctx, "systemctl", "stop", a.Target); err != nil {
+			return fmt.Errorf("stopping %s: %w", a.Target, err)
+		}
+		return nil
+
+	default:
+		return fmt.Errorf("no way to remove a %q", a.Type)
+	}
+}
+
+// ExecRunner runs a real command, with stdin closed and output captured.
+func ExecRunner(ctx context.Context, name string, args ...string) (string, error) {
+	cmd := exec.CommandContext(ctx, name, args...)
+	cmd.Stdin = nil
+	out, err := cmd.Output()
+	if err != nil {
+		var exit *exec.ExitError
+		if errors.As(err, &exit) {
+			return string(out), fmt.Errorf("%s exited %d: %s",
+				name, exit.ExitCode(), strings.TrimSpace(string(exit.Stderr)))
+		}
+		return string(out), fmt.Errorf("%s: %w", name, err)
+	}
+	return string(out), nil
+}

internal/apply/apply_test.go

@@ -0,0 +1,413 @@
+package apply
+
+import (
+	"context"
+	"errors"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"github.com/novox/mesh-host/internal/declaration"
+	"github.com/novox/mesh-host/internal/store"
+)
+
+// Each test names the decision it defends (novox/hq ADR 0034).
+
+func parse(t *testing.T, raw string) *declaration.Declaration {
+	t.Helper()
+	d, err := declaration.Parse([]byte(raw))
+	if err != nil {
+		t.Fatalf("fixture is not a valid declaration: %v", err)
+	}
+	return d
+}
+
+// noServices refuses to run anything. Used where a test declares no services, so that a test
+// which accidentally reaches the service manager fails loudly instead of passing quietly.
+func noServices(context.Context, string, ...string) (string, error) {
+	return "", errors.New("this test declares no services and should not have run a command")
+}
+
+func TestApplyingTwiceChangesNothingTheSecondTime(t *testing.T) {
+	// Idempotence is what makes an apply safe to run on a schedule. Without it, a host that
+	// reconciles every few minutes rewrites files forever and every reader sees churn.
+	dir := t.TempDir()
+	d := parse(t, `{"declaration":1,"resources":[
+	  {"id":"d","type":"directory","path":"`+dir+`/etc","mode":"0755"},
+	  {"id":"f","type":"file","path":"`+dir+`/etc/a.conf","content":"hello\n","mode":"0640"}
+	]}`)
+
+	first, state, err := Apply(context.Background(), d, store.State{}, noServices, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !first.Changed() {
+		t.Fatal("the first apply on an empty machine changed nothing")
+	}
+
+	second, _, err := Apply(context.Background(), d, state, noServices, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if second.Changed() {
+		t.Errorf("the second apply changed something: %+v", second.Outcomes)
+	}
+}
+
+func TestADriftedMachineIsReturned(t *testing.T) {
+	// The other half of idempotence, and the half that matters: converging is not "do nothing
+	// if the state file says it was done". The machine is read, not the record.
+	dir := t.TempDir()
+	path := filepath.Join(dir, "a.conf")
+	d := parse(t, `{"declaration":1,"resources":[
+	  {"id":"f","type":"file","path":"`+path+`","content":"correct\n","mode":"0644"}
+	]}`)
+
+	_, state, err := Apply(context.Background(), d, store.State{}, noServices, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(path, []byte("someone edited this\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+
+	report, _, err := Apply(context.Background(), d, state, noServices, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !report.Changed() {
+		t.Fatal("a drifted file was left drifted")
+	}
+	got, _ := os.ReadFile(path)
+	if string(got) != "correct\n" {
+		t.Errorf("the file was not returned: %q", got)
+	}
+}
+
+func TestADroppedResourceIsRemoved(t *testing.T) {
+	// novox/hq ADR 0043: the host removes what it previously applied and is no longer
+	// declared. Removing a line from a declaration is an act with an effect.
+	dir := t.TempDir()
+	keep := filepath.Join(dir, "keep.conf")
+	drop := filepath.Join(dir, "drop.conf")
+
+	both := parse(t, `{"declaration":1,"resources":[
+	  {"id":"keep","type":"file","path":"`+keep+`","content":"a\n"},
+	  {"id":"drop","type":"file","path":"`+drop+`","content":"b\n"}
+	]}`)
+	_, state, err := Apply(context.Background(), both, store.State{}, noServices, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	one := parse(t, `{"declaration":1,"resources":[
+	  {"id":"keep","type":"file","path":"`+keep+`","content":"a\n"}
+	]}`)
+	report, state, err := Apply(context.Background(), one, state, noServices, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if _, err := os.Stat(drop); !errors.Is(err, os.ErrNotExist) {
+		t.Error("a resource dropped from the declaration was left on the machine")
+	}
+	if _, err := os.Stat(keep); err != nil {
+		t.Error("a declared resource was removed")
+	}
+	if _, still := state.Find("drop"); still {
+		t.Error("the host still believes it owns what it removed")
+	}
+	if report.Outcomes[0].Action != "removed" {
+		t.Errorf("removal is not reported first: %+v", report.Outcomes)
+	}
+}
+
+func TestNothingTheHostDidNotCreateIsTouched(t *testing.T) {
+	// The boundary the whole removal rule turns on. A machine has things on it the mesh did
+	// not put there, and a converger that treats "not declared" as "must not exist" deletes
+	// them. Authoritative over its own footprint; inert everywhere else.
+	dir := t.TempDir()
+	stranger := filepath.Join(dir, "not-ours.conf")
+	if err := os.WriteFile(stranger, []byte("someone else's\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+
+	d := parse(t, `{"declaration":1,"resources":[
+	  {"id":"ours","type":"file","path":"`+filepath.Join(dir, "ours.conf")+`","content":"a\n"}
+	]}`)
+	if _, _, err := Apply(context.Background(), d, store.State{}, noServices, nil); err != nil {
+		t.Fatal(err)
+	}
+
+	got, err := os.ReadFile(stranger)
+	if err != nil || string(got) != "someone else's\n" {
+		t.Error("a file the host did not create was removed or changed")
+	}
+}
+
+func TestARenameToTheSamePathDoesNotDeleteTheNewFile(t *testing.T) {
+	// Why removal happens FIRST. A resource leaving a declaration while another arrives at the
+	// same path is an ordinary rename; removing afterwards would delete the file just written.
+	dir := t.TempDir()
+	path := filepath.Join(dir, "shared.conf")
+
+	before := parse(t, `{"declaration":1,"resources":[
+	  {"id":"old","type":"file","path":"`+path+`","content":"old\n"}
+	]}`)
+	_, state, err := Apply(context.Background(), before, store.State{}, noServices, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	after := parse(t, `{"declaration":1,"resources":[
+	  {"id":"new","type":"file","path":"`+path+`","content":"new\n"}
+	]}`)
+	if _, _, err := Apply(context.Background(), after, state, noServices, nil); err != nil {
+		t.Fatal(err)
+	}
+
+	got, err := os.ReadFile(path)
+	if err != nil {
+		t.Fatalf("the renamed resource is gone: %v", err)
+	}
+	if string(got) != "new\n" {
+		t.Errorf("content is %q, want the new one", got)
+	}
+}
+
+func TestAFailedStepFailsTheApply(t *testing.T) {
+	// novox/hq ADR 0008. And the error carries what HAD been done, because the machine is in
+	// whatever state the apply reached and the only honest thing to hand back is that list.
+	dir := t.TempDir()
+	blocker := filepath.Join(dir, "blocker")
+	if err := os.WriteFile(blocker, []byte("i am a file\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+
+	d := parse(t, `{"declaration":1,"resources":[
+	  {"id":"fine","type":"file","path":"`+filepath.Join(dir, "fine.conf")+`","content":"a\n"},
+	  {"id":"doomed","type":"directory","path":"`+blocker+`"},
+	  {"id":"never","type":"file","path":"`+filepath.Join(dir, "never.conf")+`","content":"b\n"}
+	]}`)
+
+	_, _, err := Apply(context.Background(), d, store.State{}, noServices, nil)
+	if err == nil {
+		t.Fatal("an impossible resource did not fail the apply")
+	}
+
+	var applyErr *Error
+	if !errors.As(err, &applyErr) {
+		t.Fatalf("expected an apply error, got %T", err)
+	}
+	if applyErr.Resource != "doomed" {
+		t.Errorf("the failure names %q, not the resource that failed", applyErr.Resource)
+	}
+	if len(applyErr.Done.Outcomes) != 1 {
+		t.Errorf("the error does not carry what was already applied: %+v", applyErr.Done.Outcomes)
+	}
+	// And nothing after the failure ran.
+	if _, err := os.Stat(filepath.Join(dir, "never.conf")); !errors.Is(err, os.ErrNotExist) {
+		t.Error("the apply continued past a failure")
+	}
+}
+
+func TestNothingIsRecordedUntilItWorked(t *testing.T) {
+	// novox/hq ADR 0035. A record written before the fact restates the request in a new place
+	// and inherits none of the authority of having happened.
+	dir := t.TempDir()
+	blocker := filepath.Join(dir, "blocker")
+	if err := os.WriteFile(blocker, []byte("x\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	d := parse(t, `{"declaration":1,"resources":[
+	  {"id":"doomed","type":"directory","path":"`+blocker+`"}
+	]}`)
+
+	_, state, err := Apply(context.Background(), d, store.State{}, noServices, nil)
+	if err == nil {
+		t.Fatal("expected a failure")
+	}
+	if _, claimed := state.Find("doomed"); claimed {
+		t.Error("the host recorded owning something it failed to apply")
+	}
+}
+
+func TestAModeIsMaintainedNotJustSet(t *testing.T) {
+	// A permission set at creation is not a permission maintained — this repository has
+	// already paid for that once, with generated files left world-readable because the mode
+	// applied only when the file was first written.
+	dir := t.TempDir()
+	path := filepath.Join(dir, "secret.conf")
+	d := parse(t, `{"declaration":1,"resources":[
+	  {"id":"f","type":"file","path":"`+path+`","content":"s\n","mode":"0600"}
+	]}`)
+
+	_, state, err := Apply(context.Background(), d, store.State{}, noServices, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := os.Chmod(path, 0o666); err != nil {
+		t.Fatal(err)
+	}
+
+	report, _, err := Apply(context.Background(), d, state, noServices, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	info, _ := os.Stat(path)
+	if info.Mode().Perm() != 0o600 {
+		t.Errorf("mode is %o after reconciling, want 0600", info.Mode().Perm())
+	}
+	if !report.Changed() {
+		t.Error("a mode that had drifted was reported as unchanged")
+	}
+}
+
+func TestAServiceIsReadBackNotAssumed(t *testing.T) {
+	// `systemctl start` returning zero says the transaction was accepted, not that the unit is
+	// running. A unit that starts and immediately dies satisfies the command.
+	started := false
+	run := func(ctx context.Context, name string, args ...string) (string, error) {
+		if args[0] == "show" {
+			if started {
+				return "LoadState=loaded\nActiveState=failed\n", nil // started, then died
+			}
+			return "LoadState=loaded\nActiveState=inactive\n", nil
+		}
+		started = true
+		return "", nil // `systemctl start` succeeds
+	}
+
+	d := parse(t, `{"declaration":1,"resources":[
+	  {"id":"s","type":"service","unit":"doomed.service","state":"running"}
+	]}`)
+	_, _, err := Apply(context.Background(), d, store.State{}, run, nil)
+	if err == nil {
+		t.Fatal("a service that died immediately was reported as running")
+	}
+	if !strings.Contains(err.Error(), "asked to be running and is stopped") {
+		t.Errorf("the failure does not say what was observed: %v", err)
+	}
+}
+
+func TestAnUnknownServiceStateIsRefusedNotGuessed(t *testing.T) {
+	run := func(ctx context.Context, name string, args ...string) (string, error) {
+		return "LoadState=loaded\nActiveState=reticent\n", nil
+	}
+	d := parse(t, `{"declaration":1,"resources":[
+	  {"id":"s","type":"service","unit":"odd.service","state":"running"}
+	]}`)
+	_, _, err := Apply(context.Background(), d, store.State{}, run, nil)
+	if err == nil || !strings.Contains(err.Error(), "neither running nor stopped") {
+		t.Errorf("an unrecognised service state was not refused: %v", err)
+	}
+}
+
+func TestADroppedServiceIsStoppedNotDeleted(t *testing.T) {
+	// The host did not install the unit and does not own the unit file — only the state it put
+	// the unit into.
+	var commands []string
+	run := func(ctx context.Context, name string, args ...string) (string, error) {
+		commands = append(commands, strings.Join(args, " "))
+		if args[0] == "show" {
+			return "LoadState=loaded\nActiveState=active\n", nil
+		}
+		return "", nil
+	}
+	state := store.State{Resources: []store.Applied{
+		{ID: "s", Type: "service", Target: "gone.service"},
+	}}
+	d := parse(t, `{"declaration":1,"resources":[
+	  {"id":"other","type":"file","path":"`+filepath.Join(t.TempDir(), "a")+`","content":"a\n"}
+	]}`)
+
+	if _, _, err := Apply(context.Background(), d, state, run, nil); err != nil {
+		t.Fatal(err)
+	}
+	joined := strings.Join(commands, "; ")
+	if !strings.Contains(joined, "stop gone.service") {
+		t.Errorf("the dropped service was not stopped: %s", joined)
+	}
+	if strings.Contains(joined, "disable") || strings.Contains(joined, "mask") {
+		t.Errorf("the host did more than stop a unit it does not own: %s", joined)
+	}
+}
+
+func TestAUnitThatDoesNotExistIsNotStopped(t *testing.T) {
+	// Found by applying inside a raised machine. `systemctl is-active` says "inactive" for a
+	// unit that DOES NOT EXIST exactly as it does for one that is installed and stopped, so
+	// declaring a unit stopped reported success for a unit the host cannot manage at all.
+	//
+	// Absence read as satisfaction — 04-ISSUES/007 wearing a different hat, and the mirror of
+	// the degraded-init bug the capability detector had.
+	absent := func(ctx context.Context, name string, args ...string) (string, error) {
+		return "LoadState=not-found\nActiveState=inactive\n", nil
+	}
+	d := parse(t, `{"declaration":1,"resources":[
+	  {"id":"s","type":"service","unit":"never-installed.service","state":"stopped"}
+	]}`)
+
+	_, state, err := Apply(context.Background(), d, store.State{}, absent, nil)
+	if err == nil {
+		t.Fatal("a unit that does not exist was reported as satisfactorily stopped")
+	}
+	if !strings.Contains(err.Error(), "does not exist on this machine") {
+		t.Errorf("the failure does not say the unit is absent: %v", err)
+	}
+	if _, claimed := state.Find("s"); claimed {
+		t.Error("the host recorded owning a unit that is not installed")
+	}
+}
+
+func TestAMaskedUnitIsRefused(t *testing.T) {
+	// Masked means someone deliberately made it unstartable. Applying over that would undo a
+	// decision the host did not make and cannot see the reason for.
+	masked := func(ctx context.Context, name string, args ...string) (string, error) {
+		return "LoadState=masked\nActiveState=inactive\n", nil
+	}
+	d := parse(t, `{"declaration":1,"resources":[
+	  {"id":"s","type":"service","unit":"masked.service","state":"running"}
+	]}`)
+	if _, _, err := Apply(context.Background(), d, store.State{}, masked, nil); err == nil {
+		t.Fatal("a masked unit was accepted")
+	}
+}
+
+func TestForgettingAUnitThatIsGoneDoesNotStrandTheNode(t *testing.T) {
+	// Found on a real machine. Removing an orphaned service runs `systemctl stop`, which fails
+	// when the unit no longer exists — and a failure there fails the whole apply. A host
+	// holding a record of an uninstalled unit could then apply NOTHING, ever, with no way out
+	// but editing its state by hand.
+	//
+	// Removal is idempotent for the same reason os.RemoveAll is: the desired end state is
+	// already true.
+	var stopped bool
+	run := func(ctx context.Context, name string, args ...string) (string, error) {
+		if args[0] == "show" {
+			return "LoadState=not-found\nActiveState=inactive\n", nil
+		}
+		stopped = true
+		return "", errors.New("systemctl exited 5: Unit not loaded")
+	}
+	known := store.State{Resources: []store.Applied{
+		{ID: "gone", Type: "service", Target: "uninstalled.service"},
+	}}
+	d := parse(t, `{"declaration":1,"resources":[
+	  {"id":"f","type":"file","path":"`+filepath.Join(t.TempDir(), "a")+`","content":"a\n"}
+	]}`)
+
+	report, state, err := Apply(context.Background(), d, known, run, nil)
+	if err != nil {
+		t.Fatalf("a vanished unit stranded the apply: %v", err)
+	}
+	if stopped {
+		t.Error("the host tried to stop a unit that does not exist")
+	}
+	if _, still := state.Find("gone"); still {
+		t.Error("the host still believes it owns a unit that is gone")
+	}
+	if report.Outcomes[0].Action != "removed" {
+		t.Errorf("the vanished unit was not reported as removed: %+v", report.Outcomes)
+	}
+}

internal/declaration/declaration.go

@@ -0,0 +1,221 @@
+// Package declaration is what the host is told a machine should be.
+//
+// Data, never instructions. The vocabulary is finite, versioned, and anything outside it
+// refuses the whole declaration rather than being skipped — a host that applied most of what
+// it was sent and reported success is a node that looks configured and is not
+// (novox/hq ADR 0043).
+package declaration
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"sort"
+	"strings"
+)
+
+// Version is the vocabulary this host speaks. A declaration naming any other version is
+// refused: an older host handed a newer vocabulary must not quietly do half of it.
+const Version = 1
+
+// Type names a kind of resource. Every addition widens what a compromised control plane can
+// express, so the list is a security artefact and grows deliberately.
+type Type string
+
+const (
+	TypeDirectory Type = "directory"
+	TypeFile      Type = "file"
+	TypeService   Type = "service"
+)
+
+// known is the whole vocabulary. Anything else is refused.
+var known = map[Type]bool{
+	TypeDirectory: true,
+	TypeFile:      true,
+	TypeService:   true,
+}
+
+// Resource is one thing that should be true of the machine.
+//
+// Identity is a name the control plane keeps stable across declarations, not a position and
+// not a hash of the content. It is what lets the store say *this is the same resource I
+// applied last time*, which is what makes removal possible at all.
+type Resource struct {
+	ID   string `json:"id"`
+	Type Type   `json:"type"`
+
+	// Path, for a file or directory.
+	Path string `json:"path,omitempty"`
+	// Content, for a file. Literal; the host renders nothing.
+	Content string `json:"content,omitempty"`
+	// Mode, for a file or directory, as an octal string such as "0644".
+	Mode string `json:"mode,omitempty"`
+
+	// Unit and State, for a service. State is "running" or "stopped".
+	Unit  string `json:"unit,omitempty"`
+	State string `json:"state,omitempty"`
+}
+
+// Declaration is what a machine should be, in the order it should be made so.
+type Declaration struct {
+	Version int `json:"declaration"`
+	// For names the node this is meant for. A host with an identity refuses one addressed
+	// elsewhere; a host without one — the first node, applying the bundle it carries — has
+	// nothing to check against.
+	For string `json:"for,omitempty"`
+	// Resources, in the order they are applied. The host does not sort them: ordering is a
+	// decision, and deciding is not what the host does (novox/hq ADR 0037).
+	Resources []Resource `json:"resources"`
+}
+
+// RefusalError refuses a whole declaration, naming every problem at once.
+//
+// Every problem rather than the first: a caller fixing one at a time learns the next only by
+// running again, and a declaration is generated, so a person reading this is debugging the
+// generator.
+type RefusalError struct {
+	Problems []string
+}
+
+func (e *RefusalError) Error() string {
+	return fmt.Sprintf(
+		"this declaration is refused, and none of it was applied:\n  - %s\n\n"+
+			"A host that applied the parts it understood would leave a machine that looks "+
+			"configured and is not.",
+		strings.Join(e.Problems, "\n  - "))
+}
+
+// Parse reads a declaration and refuses anything it does not fully understand.
+func Parse(raw []byte) (*Declaration, error) {
+	// DisallowUnknownFields is the whole point rather than strictness for its own sake: a
+	// field the host does not know is a thing the control plane believes it asked for.
+	dec := json.NewDecoder(bytes.NewReader(raw))
+	dec.DisallowUnknownFields()
+
+	var d Declaration
+	if err := dec.Decode(&d); err != nil {
+		return nil, &RefusalError{Problems: []string{"not a declaration: " + err.Error()}}
+	}
+
+	if problems := validate(&d); len(problems) > 0 {
+		return nil, &RefusalError{Problems: problems}
+	}
+	return &d, nil
+}
+
+func validate(d *Declaration) []string {
+	var problems []string
+
+	if d.Version != Version {
+		problems = append(problems, fmt.Sprintf(
+			"declaration version %d; this host speaks version %d. Refused whole rather than "+
+				"partly, so a newer vocabulary is never half-applied by an older host",
+			d.Version, Version))
+		// Everything below assumes the vocabulary, so there is nothing further to say.
+		return problems
+	}
+
+	if len(d.Resources) == 0 {
+		problems = append(problems, "no resources. An empty declaration is a mistake, not a "+
+			"machine with nothing on it — say so with an explicit empty list if that is meant")
+	}
+
+	seen := map[string]int{}
+	for i, r := range d.Resources {
+		where := fmt.Sprintf("resource %d", i)
+		if r.ID != "" {
+			where = fmt.Sprintf("resource %q", r.ID)
+		}
+
+		if r.ID == "" {
+			problems = append(problems, where+": no id. Identity is what lets the host know "+
+				"this is the same resource it applied last time")
+		} else if first, ok := seen[r.ID]; ok {
+			problems = append(problems, fmt.Sprintf(
+				"%s: id already used by resource %d. Two resources with one identity cannot "+
+					"both be tracked", where, first))
+		} else {
+			seen[r.ID] = i
+		}
+
+		if !known[r.Type] {
+			problems = append(problems, fmt.Sprintf(
+				"%s: unknown type %q. This host understands %s", where, r.Type, vocabulary()))
+			continue
+		}
+		problems = append(problems, validateResource(where, r)...)
+	}
+	return problems
+}
+
+func validateResource(where string, r Resource) []string {
+	var problems []string
+	switch r.Type {
+	case TypeDirectory:
+		if r.Path == "" {
+			problems = append(problems, where+": a directory needs a path")
+		}
+		problems = append(problems, checkMode(where, r.Mode)...)
+		problems = append(problems, unusedBy(where, r, "unit", r.Unit, "state", r.State, "content", r.Content)...)
+
+	case TypeFile:
+		if r.Path == "" {
+			problems = append(problems, where+": a file needs a path")
+		}
+		problems = append(problems, checkMode(where, r.Mode)...)
+		problems = append(problems, unusedBy(where, r, "unit", r.Unit, "state", r.State)...)
+
+	case TypeService:
+		if r.Unit == "" {
+			problems = append(problems, where+": a service needs a unit")
+		}
+		if r.State != "running" && r.State != "stopped" {
+			problems = append(problems, fmt.Sprintf(
+				"%s: state %q; a service is \"running\" or \"stopped\"", where, r.State))
+		}
+		problems = append(problems, unusedBy(where, r, "path", r.Path, "content", r.Content, "mode", r.Mode)...)
+	}
+	return problems
+}
+
+// unusedBy refuses a field this type does not use.
+//
+// A field set and ignored is the fault this package exists to prevent, in miniature: the
+// control plane believes it asked for something the host will never do.
+func unusedBy(where string, r Resource, pairs ...string) []string {
+	var problems []string
+	for i := 0; i+1 < len(pairs); i += 2 {
+		if pairs[i+1] != "" {
+			problems = append(problems, fmt.Sprintf(
+				"%s: a %s does not use %q, and it is set. Refused rather than ignored",
+				where, r.Type, pairs[i]))
+		}
+	}
+	return problems
+}
+
+func checkMode(where, mode string) []string {
+	if mode == "" {
+		return nil
+	}
+	if len(mode) != 4 || mode[0] != '0' {
+		return []string{fmt.Sprintf(
+			"%s: mode %q; write it as four octal digits such as \"0644\", so it means the "+
+				"same thing here as it does in the manifest it came from", where, mode)}
+	}
+	for _, c := range mode[1:] {
+		if c < '0' || c > '7' {
+			return []string{fmt.Sprintf("%s: mode %q is not octal", where, mode)}
+		}
+	}
+	return nil
+}
+
+func vocabulary() string {
+	var names []string
+	for t := range known {
+		names = append(names, string(t))
+	}
+	sort.Strings(names)
+	return strings.Join(names, ", ")
+}

internal/declaration/declaration_test.go

@@ -0,0 +1,158 @@
+package declaration
+
+import (
+	"errors"
+	"strings"
+	"testing"
+)
+
+// Each test names the decision it defends (novox/hq ADR 0034). The decision here is ADR 0043,
+// and the property it turns on is that unknown is REFUSED, never skipped.
+
+func valid() string {
+	return `{"declaration":1,"resources":[
+	  {"id":"etc","type":"directory","path":"/etc/mesh","mode":"0755"},
+	  {"id":"conf","type":"file","path":"/etc/mesh/host.conf","content":"a\n","mode":"0640"},
+	  {"id":"svc","type":"service","unit":"mesh-host.service","state":"running"}
+	]}`
+}
+
+func refusalFor(t *testing.T, raw string) *RefusalError {
+	t.Helper()
+	_, err := Parse([]byte(raw))
+	if err == nil {
+		t.Fatal("expected a refusal")
+	}
+	var refusal *RefusalError
+	if !errors.As(err, &refusal) {
+		t.Fatalf("expected a RefusalError, got %T: %v", err, err)
+	}
+	return refusal
+}
+
+func TestAValidDeclarationParsesInOrder(t *testing.T) {
+	d, err := Parse([]byte(valid()))
+	if err != nil {
+		t.Fatalf("unexpected refusal: %v", err)
+	}
+	// Order is stated, not derived. The host must not sort.
+	got := []string{d.Resources[0].ID, d.Resources[1].ID, d.Resources[2].ID}
+	want := []string{"etc", "conf", "svc"}
+	for i := range want {
+		if got[i] != want[i] {
+			t.Fatalf("resources reordered: %v, want %v", got, want)
+		}
+	}
+}
+
+func TestAnUnknownTypeRefusesTheWholeDeclaration(t *testing.T) {
+	// The property everything else rests on. A host that skipped what it did not understand
+	// would apply most of a declaration and report success — a node that looks configured and
+	// is not, which is 04-ISSUES/003 with the declaration on the other side of the wire.
+	refusal := refusalFor(t, `{"declaration":1,"resources":[
+	  {"id":"ok","type":"directory","path":"/etc/mesh"},
+	  {"id":"what","type":"blockchain","path":"/etc/mesh"}
+	]}`)
+
+	joined := strings.Join(refusal.Problems, "\n")
+	if !strings.Contains(joined, "blockchain") {
+		t.Errorf("the unknown type was not named: %v", refusal.Problems)
+	}
+	// And it must say what IS understood, or the reader goes to the source to find out.
+	if !strings.Contains(joined, "directory") || !strings.Contains(joined, "service") {
+		t.Errorf("the refusal does not say what this host understands: %v", refusal.Problems)
+	}
+}
+
+func TestAnUnknownFieldIsRefused(t *testing.T) {
+	// A field the host does not know is a thing the control plane believes it asked for.
+	refusal := refusalFor(t, `{"declaration":1,"resources":[
+	  {"id":"conf","type":"file","path":"/etc/x","content":"a","owner":"root"}
+	]}`)
+	if !strings.Contains(strings.Join(refusal.Problems, "\n"), "owner") {
+		t.Errorf("the unknown field was not named: %v", refusal.Problems)
+	}
+}
+
+func TestAFieldTheTypeDoesNotUseIsRefusedNotIgnored(t *testing.T) {
+	// The same fault in miniature: set and ignored means the control plane believes it asked
+	// for something the host will never do.
+	refusal := refusalFor(t, `{"declaration":1,"resources":[
+	  {"id":"svc","type":"service","unit":"a.service","state":"running","path":"/etc/x"}
+	]}`)
+	joined := strings.Join(refusal.Problems, "\n")
+	if !strings.Contains(joined, "path") || !strings.Contains(joined, "Refused rather than ignored") {
+		t.Errorf("a field a service does not use was accepted: %v", refusal.Problems)
+	}
+}
+
+func TestAnUnknownVersionIsRefusedWhole(t *testing.T) {
+	// An older host handed a newer vocabulary must not quietly do half of it.
+	refusal := refusalFor(t, `{"declaration":99,"resources":[
+	  {"id":"a","type":"directory","path":"/etc/mesh"}
+	]}`)
+	joined := strings.Join(refusal.Problems, "\n")
+	if !strings.Contains(joined, "99") || !strings.Contains(joined, "version 1") {
+		t.Errorf("the version mismatch was not stated plainly: %v", refusal.Problems)
+	}
+	// Nothing else is reported, because everything else assumes a vocabulary this host does
+	// not have — a list of complaints derived from the wrong grammar is noise.
+	if len(refusal.Problems) != 1 {
+		t.Errorf("expected only the version problem, got: %v", refusal.Problems)
+	}
+}
+
+func TestEveryProblemIsReportedAtOnce(t *testing.T) {
+	// A declaration is generated, so a person reading a refusal is debugging the generator.
+	// Fixing one problem at a time and re-running to find the next wastes their afternoon.
+	refusal := refusalFor(t, `{"declaration":1,"resources":[
+	  {"id":"","type":"directory","path":"/a"},
+	  {"id":"b","type":"file"},
+	  {"id":"c","type":"service","unit":"x.service","state":"dancing"}
+	]}`)
+	if len(refusal.Problems) < 3 {
+		t.Errorf("expected every problem at once, got: %v", refusal.Problems)
+	}
+}
+
+func TestIdentityIsRequiredAndUnique(t *testing.T) {
+	// Identity is what lets the store know this is the same resource it applied last time,
+	// which is what makes removal possible at all.
+	refusal := refusalFor(t, `{"declaration":1,"resources":[
+	  {"id":"same","type":"directory","path":"/a"},
+	  {"id":"same","type":"directory","path":"/b"}
+	]}`)
+	if !strings.Contains(strings.Join(refusal.Problems, "\n"), "already used") {
+		t.Errorf("a duplicate identity was accepted: %v", refusal.Problems)
+	}
+}
+
+func TestModeIsRefusedUnlessItMeansWhatItLooksLike(t *testing.T) {
+	// "644" and "0644" differ, and the one that looks right in a manifest is the four-digit
+	// form. Accepting both would make a mode mean two things.
+	for _, mode := range []string{"644", "0999", "rwxr-xr-x", "07777777"} {
+		refusal := refusalFor(t, `{"declaration":1,"resources":[
+		  {"id":"f","type":"file","path":"/a","mode":"`+mode+`"}
+		]}`)
+		if !strings.Contains(strings.Join(refusal.Problems, "\n"), "mode") {
+			t.Errorf("mode %q was accepted: %v", mode, refusal.Problems)
+		}
+	}
+	if _, err := Parse([]byte(`{"declaration":1,"resources":[
+	  {"id":"f","type":"file","path":"/a","mode":"0644"}
+	]}`)); err != nil {
+		t.Errorf("a well-formed mode was refused: %v", err)
+	}
+}
+
+func TestARefusalSaysNothingWasApplied(t *testing.T) {
+	// The reader's first question is whether the machine was left half-changed.
+	refusal := refusalFor(t, `{"declaration":1,"resources":[{"id":"x","type":"nope"}]}`)
+	if !strings.Contains(refusal.Error(), "none of it was applied") {
+		t.Errorf("the refusal does not say the machine is untouched: %s", refusal.Error())
+	}
+}
+
+func TestAnEmptyDeclarationIsAMistake(t *testing.T) {
+	refusalFor(t, `{"declaration":1,"resources":[]}`)
+}

internal/store/store.go

@@ -0,0 +1,176 @@
+// Package store is what this node knows about itself, and it is authoritative while
+// disconnected.
+//
+// Not a cache of the control plane. novox/hq ADR 0036 makes disconnection an ordinary
+// situation rather than an exception, and this is what makes it ordinary: a machine shut for a
+// week comes back and reconciles, it does not come back and ask what it is.
+//
+// Its first job arrives with the first apply rather than with the link (ADR 0043): the host
+// removes what it previously applied and is no longer declared, and it can only know that
+// because it wrote it down.
+package store
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"sort"
+	"time"
+)
+
+// DefaultPath is where a node keeps what it knows. Under /var/lib because it survives a
+// reboot and is not configuration — nothing generates this, the host writes it.
+const DefaultPath = "/var/lib/mesh-host/state.json"
+
+// Applied is one resource the host put on this machine, and what it did.
+//
+// Recorded AFTER the resource was applied and read back, never before (novox/hq ADR 0035).
+// A record written up front restates the request in a new place and inherits none of the
+// authority of having happened.
+type Applied struct {
+	ID   string `json:"id"`
+	Type string `json:"type"`
+	// Target is what was changed — a path, a unit — so removal knows what to undo without
+	// re-reading a declaration that may no longer exist.
+	Target    string    `json:"target"`
+	AppliedAt time.Time `json:"applied_at"`
+}
+
+// State is the whole of what a node knows about what it has done.
+type State struct {
+	// Resources, keyed by identity, in the order they were applied. Order matters for removal:
+	// undoing in reverse is the only ordering the host can derive without deciding anything.
+	Resources []Applied `json:"resources"`
+	UpdatedAt time.Time `json:"updated_at"`
+}
+
+// Find returns what was applied under an identity.
+func (s State) Find(id string) (Applied, bool) {
+	for _, r := range s.Resources {
+		if r.ID == id {
+			return r, true
+		}
+	}
+	return Applied{}, false
+}
+
+// IDs returns every identity the host has applied, sorted.
+func (s State) IDs() []string {
+	out := make([]string, 0, len(s.Resources))
+	for _, r := range s.Resources {
+		out = append(out, r.ID)
+	}
+	sort.Strings(out)
+	return out
+}
+
+// Load reads the state. A node that has never applied anything has an empty state, which is a
+// fact rather than an error — the first apply on a fresh machine is the ordinary case.
+//
+// A state file that exists and cannot be read IS an error, and a loud one: continuing with an
+// empty state would make the host believe it owns nothing, and it would then remove nothing it
+// should and re-apply everything it need not.
+func Load(path string) (State, error) {
+	raw, err := os.ReadFile(path)
+	if errors.Is(err, fs.ErrNotExist) {
+		return State{}, nil
+	}
+	if err != nil {
+		return State{}, fmt.Errorf("reading what this node knows about itself (%s): %w", path, err)
+	}
+
+	var s State
+	if err := json.Unmarshal(raw, &s); err != nil {
+		return State{}, fmt.Errorf(
+			"what this node knows about itself is unreadable (%s): %w\n"+
+				"Refusing rather than starting empty: an empty state would mean the host "+
+				"believes it owns nothing, so it would remove nothing it should and re-apply "+
+				"everything it need not", path, err)
+	}
+	return s, nil
+}
+
+// Save writes the state, atomically.
+//
+// Atomic because the alternative has a failure mode with no floor: a host interrupted while
+// writing loses the record of everything it owns, and then owns nothing it can clean up.
+func Save(path string, s State) error {
+	s.UpdatedAt = time.Now().UTC()
+
+	if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
+		return fmt.Errorf("making room for the node's state: %w", err)
+	}
+
+	raw, err := json.MarshalIndent(s, "", "  ")
+	if err != nil {
+		return fmt.Errorf("encoding the node's state: %w", err)
+	}
+	raw = append(raw, '\n')
+
+	tmp, err := os.CreateTemp(filepath.Dir(path), ".state-*.json")
+	if err != nil {
+		return fmt.Errorf("writing the node's state: %w", err)
+	}
+	defer os.Remove(tmp.Name())
+
+	if _, err := tmp.Write(raw); err != nil {
+		tmp.Close()
+		return fmt.Errorf("writing the node's state: %w", err)
+	}
+	// Flushed before the rename: a rename is atomic, and a rename of a file whose contents are
+	// still in the page cache is atomically the wrong thing.
+	if err := tmp.Sync(); err != nil {
+		tmp.Close()
+		return fmt.Errorf("flushing the node's state: %w", err)
+	}
+	if err := tmp.Close(); err != nil {
+		return fmt.Errorf("closing the node's state: %w", err)
+	}
+	if err := os.Chmod(tmp.Name(), 0o600); err != nil {
+		return fmt.Errorf("securing the node's state: %w", err)
+	}
+	if err := os.Rename(tmp.Name(), path); err != nil {
+		return fmt.Errorf("replacing the node's state: %w", err)
+	}
+	return nil
+}
+
+// Record adds or replaces what is known about one resource, preserving order.
+func (s *State) Record(a Applied) {
+	for i, existing := range s.Resources {
+		if existing.ID == a.ID {
+			s.Resources[i] = a
+			return
+		}
+	}
+	s.Resources = append(s.Resources, a)
+}
+
+// Forget drops a resource from what the node owns.
+func (s *State) Forget(id string) {
+	kept := s.Resources[:0]
+	for _, r := range s.Resources {
+		if r.ID != id {
+			kept = append(kept, r)
+		}
+	}
+	s.Resources = kept
+}
+
+// Orphans returns what the host applied and the declaration no longer names, newest first.
+//
+// Reverse order because undoing in the order things were made undoes a directory before the
+// file inside it. Reversing is the only ordering the host can derive without deciding
+// anything, which is the line novox/hq ADR 0037 draws.
+func (s State) Orphans(declared map[string]bool) []Applied {
+	var out []Applied
+	for i := len(s.Resources) - 1; i >= 0; i-- {
+		if !declared[s.Resources[i].ID] {
+			out = append(out, s.Resources[i])
+		}
+	}
+	return out
+}

internal/store/store_test.go

@@ -0,0 +1,136 @@
+package store
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestAFreshMachineHasAnEmptyStateNotAnError(t *testing.T) {
+	// The first apply on a machine that has never been touched is the ordinary case, not a
+	// failure. A host that errored here could never bootstrap anything.
+	s, err := Load(filepath.Join(t.TempDir(), "nothing-here.json"))
+	if err != nil {
+		t.Fatalf("a fresh machine produced an error: %v", err)
+	}
+	if len(s.Resources) != 0 {
+		t.Errorf("a fresh machine claims to own %d resources", len(s.Resources))
+	}
+}
+
+func TestAnUnreadableStateIsRefusedNotIgnored(t *testing.T) {
+	// The dangerous one. Starting empty would make the host believe it owns nothing, so it
+	// would remove nothing it should and re-apply everything it need not — silently.
+	path := filepath.Join(t.TempDir(), "state.json")
+	if err := os.WriteFile(path, []byte("{this is not json"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	_, err := Load(path)
+	if err == nil {
+		t.Fatal("a corrupt state was read as an empty one")
+	}
+	if !strings.Contains(err.Error(), "believes it owns nothing") {
+		t.Errorf("the error does not say why this matters: %v", err)
+	}
+}
+
+func TestWhatIsSavedIsWhatIsLoaded(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "state.json")
+	want := State{Resources: []Applied{
+		{ID: "etc", Type: "directory", Target: "/etc/mesh", AppliedAt: time.Now().UTC().Truncate(time.Second)},
+		{ID: "conf", Type: "file", Target: "/etc/mesh/host.conf", AppliedAt: time.Now().UTC().Truncate(time.Second)},
+	}}
+	if err := Save(path, want); err != nil {
+		t.Fatal(err)
+	}
+	got, err := Load(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(got.Resources) != 2 || got.Resources[0].ID != "etc" || got.Resources[1].ID != "conf" {
+		t.Fatalf("order or content was lost: %+v", got.Resources)
+	}
+	if got.UpdatedAt.IsZero() {
+		t.Error("the state does not say when it was written")
+	}
+}
+
+func TestTheStateIsNotWorldReadable(t *testing.T) {
+	// It records what is on the machine and where. Not secret, and not everyone's business.
+	path := filepath.Join(t.TempDir(), "state.json")
+	if err := Save(path, State{Resources: []Applied{{ID: "a", Type: "file", Target: "/a"}}}); err != nil {
+		t.Fatal(err)
+	}
+	info, err := os.Stat(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if mode := info.Mode().Perm(); mode&0o077 != 0 {
+		t.Errorf("the state is readable by others: %o", mode)
+	}
+}
+
+func TestSavingLeavesNoDebrisBehind(t *testing.T) {
+	// The write is atomic through a temporary file. A run that left those behind would fill a
+	// directory with near-copies of the truth, and the next reader would have to guess.
+	dir := t.TempDir()
+	path := filepath.Join(dir, "state.json")
+	for i := 0; i < 3; i++ {
+		if err := Save(path, State{Resources: []Applied{{ID: "a", Type: "file", Target: "/a"}}}); err != nil {
+			t.Fatal(err)
+		}
+	}
+	entries, err := os.ReadDir(dir)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(entries) != 1 {
+		names := []string{}
+		for _, e := range entries {
+			names = append(names, e.Name())
+		}
+		t.Errorf("expected only the state file, found: %v", names)
+	}
+}
+
+func TestRecordReplacesRatherThanDuplicating(t *testing.T) {
+	s := State{}
+	s.Record(Applied{ID: "a", Type: "file", Target: "/old"})
+	s.Record(Applied{ID: "a", Type: "file", Target: "/new"})
+
+	if len(s.Resources) != 1 {
+		t.Fatalf("one identity produced %d records", len(s.Resources))
+	}
+	if s.Resources[0].Target != "/new" {
+		t.Errorf("the record was not updated: %+v", s.Resources[0])
+	}
+}
+
+func TestOrphansAreWhatWasAppliedAndIsNoLongerDeclared(t *testing.T) {
+	// The whole reason the store arrives at stage 2 rather than stage 3: removal is impossible
+	// without knowing what was applied.
+	s := State{Resources: []Applied{
+		{ID: "dir", Type: "directory", Target: "/etc/mesh"},
+		{ID: "file", Type: "file", Target: "/etc/mesh/a.conf"},
+		{ID: "kept", Type: "file", Target: "/etc/mesh/b.conf"},
+	}}
+	orphans := s.Orphans(map[string]bool{"kept": true})
+
+	if len(orphans) != 2 {
+		t.Fatalf("expected two orphans, got %d: %+v", len(orphans), orphans)
+	}
+	// Reverse order: undoing in the order things were made would remove a directory before the
+	// file inside it.
+	if orphans[0].ID != "file" || orphans[1].ID != "dir" {
+		t.Errorf("orphans are not in reverse order: %s then %s", orphans[0].ID, orphans[1].ID)
+	}
+}
+
+func TestNothingIsAnOrphanWhenEverythingIsDeclared(t *testing.T) {
+	s := State{Resources: []Applied{{ID: "a", Type: "file", Target: "/a"}}}
+	if got := s.Orphans(map[string]bool{"a": true}); len(got) != 0 {
+		t.Errorf("a declared resource was treated as an orphan: %+v", got)
+	}
+}