diff --git a/modules/nfs-server/README.md b/modules/nfs-server/README.md new file mode 100644 index 00000000..f6d0b0c5 --- /dev/null +++ b/modules/nfs-server/README.md @@ -0,0 +1,79 @@ +# nfs-server + +A node shares a folder with the mesh's machines, as a module (novox/hq ADR 0263, to-be 52). It holds +the node seat `node-nfs-server` and provides the provision `nfs-share`, which the `mounts` module on +another node requires. + +## The rules it keeps + +- **One share per folder.** The shares are the assignment's `shares` setting: `name=folder`, or + `name=folder:ro`, separated by spaces. The setting has no default: which folders a machine shares is + the operator's value (ADR 0112, ADR 0262). +- **To the mesh's machines only.** Every export line names the private network's range + (`${machine:mesh-range}`) and nothing else, and the module opens port 2049 to the mesh only. The home + network is never named. +- **Every client is the folder's owner.** NFS with `sec=sys` believes the user id a client sends, and one + id names different accounts on different machines. Every line is `all_squash` with `anonuid` and + `anongid` read from the folder's owner on this machine, so a file written from any node lands owned by + the folder's owner, and root is squashed with everyone else. +- **Never the backups.** A folder that is the machine's root, is not there, is not a folder, or holds a + backup (restic) repository at its path or one level below is refused, said by the tools, and makes the + module unhealthy. +- **The folders are the operator's** (ADR 0051). The module reads a folder's owner and exports it. It + never creates, chowns, chmods or removes anything under a shared path; its code writes two files of its + own and nothing else, and a test holds it to that. +- **NFS version 4 only**, so 2049 is the only port a client needs: rpcbind and mountd stay closed. + +## What it owns + +- the `nfs-utils` package and `nfs-server.service`, running and enabled; +- `/etc/nfs.conf.d/50-mesh.conf`: versions 2, 3 and 4.0 off, 4.1 and 4.2 on; +- `/etc/nfs-server/shares.conf`: the setting and the range, rendered by the mesh; +- `/etc/exports.d/mesh.exports`: written by its process, with a marker header; +- `/run/nfs-server/status.json`: the process's last look, for the tools; +- the state `exports` on the bus, one key per machine: what `exports` answers, for a consumer's module + and for a view (ADR 0201). + +## How it runs + +- **Its process** (`nfs-server exports`, as root) looks every 30 seconds. It reads the shares, judges each + folder, writes the export file when it changed, and runs `exportfs -ra` when the file changed or a share + is missing from the kernel's table. Then it writes its status. +- **Its tools** run as the operator account. They read the status and the kernel's table, and run what + only root may do (`exportfs -ra`, `zfs set`) through `sudo -n`, like the packet filter's tools. +- **Health** is the tool `nfs_health`, asked by the node-engine every minute: unhealthy when the NFS + server is not active, the process has not looked for five minutes, a share is refused or missing from + the kernel's table, or the kernel could not read the files again. + +## Tools + +| tool | | answers | +|---|---|---| +| `node-nfs-server.exports` | r | every share: name, folder, rw or ro, the owner every client is mapped to, the clients, whether it is live; refused shares and why; every export that is not the mesh's (`/etc/exports`, other files in `/etc/exports.d`, ZFS `sharenfs`) | +| `node-nfs-server.clients` | r | the NFS clients the kernel knows (address, name), or the connections to 2049 | +| `node-nfs-server.test` | r | whether one share is exported to the range now, the server is active, and an address is inside the range | +| `node-nfs-server.reload` | a | `exportfs -ra`, then the shares as the kernel holds them | +| `node-nfs-server.adopt` | a | take over a ZFS dataset's `sharenfs` export of a folder the mesh exports: `sharenfs=off` on that one dataset, then `exportfs -ra`; only when the mesh's export of the folder is live; a dry run without `confirm`; a person's act, with why | +| `nfs_health` | r | the module's health, one sentence per finding | + +`exports` answers, for the `mounts` module: + +``` +{"shares": [{"name", "path", "access", "uid", "gid", "clients", "line", "live"}], + "refused": [{"name", "path", "why"}], "foreign": [{"path", "client", "options", "from"}], + "service": "active", "range", "looked", "said", "notes"} +``` + +## What it leaves + +- **The export file after unassignment.** The process writes `/etc/exports.d/mesh.exports`, not the + node-engine, because each line carries the folder's owner as this machine reads it. When the module is + unassigned, the process stops and the file stays until removed. `exports` on any later holder, and the + `mounts` module's `mounts_exports`, show it. +- **A subfolder export shares its filesystem.** A share that is a folder inside a larger filesystem (not + a filesystem of its own) is exported with `no_subtree_check`, so a client that forges file handles could + reach other folders of that filesystem. Every client is a node and is squashed to the share's owner, so + a folder that owner cannot read stays closed. A share that must be strictly fenced is better given its + own dataset. +- **Exports that are not the mesh's** (`/etc/exports`, a ZFS dataset's `sharenfs`) are left as found and + shown, until `adopt` takes one over on a person's word. diff --git a/modules/nfs-server/cmd/nfs-server/machine.go b/modules/nfs-server/cmd/nfs-server/machine.go new file mode 100644 index 00000000..eb96cc60 --- /dev/null +++ b/modules/nfs-server/cmd/nfs-server/machine.go @@ -0,0 +1,210 @@ +package main + +import ( + "bytes" + "context" + "errors" + "fmt" + "io/fs" + "os" + "os/exec" + "path/filepath" + "strings" + "syscall" + "time" +) + +// The machine as the module sees it (novox/hq ADR 0263). +// +// **Two processes, two accounts.** The node's tool runtime runs the tools as the operator account, not +// root (novox/hq ADR 0175 §4): they read, and what only root may do — `exportfs -ra`, `zfs set` — goes +// through `sudo -n`, as the packet filter's and the service manager's tools do. The module's long-running +// process runs as root: it renders the export file and writes the status the tools read. It cannot reach +// the bus, so the tools' bundle puts that status in the module's state. +// +// **It writes two files and nothing else.** The export file and the status file are the module's own. +// Nothing here can create, chown, chmod or remove anything under a shared path: the machine offers no +// such call, and Write refuses every path but the module's two (novox/hq ADR 0051). + +// The module's files on the machine. +const ( + // ConfigFile is what the mesh renders from the assignment: the shares and the private network's range. + ConfigFile = "/etc/nfs-server/shares.conf" + // ExportFile is the module's own export file, which the distribution's exportfs reads with the rest of + // /etc/exports.d. + ExportFile = "/etc/exports.d/mesh.exports" + // StatusFile is the process's last look, for the tools. + StatusFile = "/run/nfs-server/status.json" + // Etab is the kernel's export table as nfs-utils keeps it. + Etab = "/var/lib/nfs/etab" + // Unit is the NFS server's unit. + Unit = "nfs-server.service" +) + +// CommandTimeout bounds every command. None touches a shared path's contents. +const CommandTimeout = 20 * time.Second + +// Ran is what one command did. +type Ran struct { + Out string + Err string + Code int +} + +// Runner runs one command; injected, so every tool is tested over recorded answers. +type Runner func(ctx context.Context, name string, args ...string) Ran + +// ExecRunner runs a command on the machine, bounded by CommandTimeout. +func ExecRunner(ctx context.Context, name string, args ...string) Ran { + ctx, cancel := context.WithTimeout(ctx, CommandTimeout) + defer cancel() + cmd := exec.CommandContext(ctx, name, args...) + var stdout, stderr bytes.Buffer + cmd.Stdout, cmd.Stderr = &stdout, &stderr + err := cmd.Run() + r := Ran{Out: stdout.String(), Err: strings.TrimSpace(stderr.String())} + if err != nil { + var exit *exec.ExitError + if errors.As(err, &exit) { + r.Code = exit.ExitCode() + } else { + r.Code = -1 + if r.Err == "" { + r.Err = err.Error() + } + } + } + return r +} + +// Owner is who owns a path, and whether it is a directory. +type Owner struct { + UID, GID int + Dir bool +} + +// Machine is a filesystem root (the real one, or a test's tree), a runner, who this process is, and a +// clock. +type Machine struct { + Root string + Run Runner + UID int + Now func() time.Time +} + +// Here is the machine this process runs on. +func Here() *Machine { + return &Machine{Root: "/", Run: ExecRunner, UID: os.Getuid(), Now: time.Now} +} + +func (m *Machine) at(path string) string { return filepath.Join(m.Root, path) } + +// Read is one file's text. +func (m *Machine) Read(path string) (string, error) { + b, err := os.ReadFile(m.at(path)) + return string(b), err +} + +// Exists says whether a path is there. +func (m *Machine) Exists(path string) bool { + _, err := os.Lstat(m.at(path)) + return err == nil +} + +// Stat is a path's owner. It reads the inode and nothing else. +func (m *Machine) Stat(path string) (Owner, error) { + info, err := os.Stat(m.at(path)) + if err != nil { + return Owner{}, err + } + o := Owner{Dir: info.IsDir()} + if st, ok := info.Sys().(*syscall.Stat_t); ok { + o.UID, o.GID = int(st.Uid), int(st.Gid) + } + return o, nil +} + +// Children are the names of a directory's subdirectories. Used only to look for a backup repository one +// level below a share; it lists names and opens nothing. +func (m *Machine) Children(path string) []string { + entries, err := os.ReadDir(m.at(path)) + if err != nil { + return nil + } + var out []string + for _, e := range entries { + if e.IsDir() { + out = append(out, e.Name()) + } + } + return out +} + +// Glob is the paths under the root matching a pattern, as the machine names them. +func (m *Machine) Glob(pattern string) []string { + found, _ := filepath.Glob(m.at(pattern)) + out := make([]string, 0, len(found)) + for _, f := range found { + rel, err := filepath.Rel(m.Root, f) + if err != nil { + continue + } + out = append(out, "/"+rel) + } + return out +} + +// written are the only paths Write may touch. +var written = map[string]bool{ExportFile: true, StatusFile: true} + +// Write replaces one of the module's own files, atomically. Every other path is refused: the module owns +// these two files and nothing else on the machine. +func (m *Machine) Write(path, content string) error { + if !written[path] { + return fmt.Errorf("the module writes only %s and %s, not %s", ExportFile, StatusFile, path) + } + full := m.at(path) + if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil && !errors.Is(err, fs.ErrExist) { + return err + } + tmp := full + ".tmp" + if err := os.WriteFile(tmp, []byte(content), 0o644); err != nil { + return err + } + return os.Rename(tmp, full) +} + +// Escalated is a command as it is run: as given when this process is root, else through sudo without a +// prompt. +func (m *Machine) Escalated(name string, args ...string) (string, []string) { + if m.UID == 0 { + return name, args + } + return "sudo", append([]string{"-n", name}, args...) +} + +// AsRoot runs what only root may do, and names a refusal by how it failed. +func (m *Machine) AsRoot(name string, args ...string) (Ran, error) { + program, argv := m.Escalated(name, args...) + r := m.Run(context.Background(), program, argv...) + if r.Code != 0 { + why := r.Err + if why == "" { + why = strings.TrimSpace(r.Out) + } + if program == "sudo" && (strings.Contains(why, "a password is required") || strings.Contains(why, "not allowed")) { + return r, fmt.Errorf("%s needs root, and sudo refused the operator account without a prompt: %s", name, why) + } + return r, fmt.Errorf("%s %s: exit %d: %s", name, strings.Join(args, " "), r.Code, why) + } + return r, nil +} + +// ServiceState is what the service manager says of the NFS server: active, inactive, failed, … +func (m *Machine) ServiceState() string { + r := m.Run(context.Background(), "systemctl", "is-active", Unit) + if s := strings.TrimSpace(r.Out); s != "" { + return s + } + return "unknown" +} diff --git a/modules/nfs-server/cmd/nfs-server/main.go b/modules/nfs-server/cmd/nfs-server/main.go new file mode 100644 index 00000000..7dab8055 --- /dev/null +++ b/modules/nfs-server/cmd/nfs-server/main.go @@ -0,0 +1,74 @@ +// The nfs-server module's Go bundle (novox/hq ADR 0263): a node shares a folder with the mesh's machines +// only, one share per folder, every client mapped to the folder's owner. +// +// Started with no arguments it is served by the node's runtime as the operator account, over MCP on stdio +// (ADR 0188, ADR 0193): the node-nfs-server seat's verbs and the module's health, and beside them it puts +// the process's last look in the module's state `exports`, keyed by this machine (ADR 0201). Started as +// `nfs-server exports` it is the module's long-running process, as root: it renders the export file from +// the assignment's shares and has the kernel read it (process.go). stdout is the MCP channel; everything +// else is said on stderr. +package main + +import ( + "context" + "fmt" + "os" + "os/signal" + "syscall" + "time" + + stdio "git.novox.be/novox/mesh-sdk/go" +) + +// ExportsState is the state the exports are kept in on the bus. +const ExportsState = "exports" + +func main() { + if len(os.Args) > 1 { + if os.Args[1] != "exports" || len(os.Args) != 2 { + fmt.Fprintln(os.Stderr, "usage: nfs-server [exports]") + os.Exit(2) + } + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt) + defer stop() + if err := Here().RunExports(ctx); err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } + return + } + m := Here() + go ShareExports(m) + if err := stdio.Serve("", Tools(m)); err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } +} + +// ShareExports puts the exports in the module's state every Every, for as long as the bundle runs, so a +// consumer's module and a view read them from the bus rather than from this machine. +func ShareExports(m *Machine) { + machine := os.Getenv("MESH_NODE") + if machine == "" { + fmt.Fprintln(os.Stderr, "nfs-server: the runtime gave no machine name (MESH_NODE); the exports are not shared") + return + } + last := "" + put := func() { + _, err := stdio.State(ExportsState).Put(machine, m.Exports()) + said := "put" + if err != nil { + said = err.Error() + } + if said != last { + fmt.Fprintln(os.Stderr, "nfs-server: exports state:", said) + } + last = said + } + // The SDK reaches the bus once Serve runs. + time.Sleep(2 * time.Second) + put() + for range time.Tick(Every) { + put() + } +} diff --git a/modules/nfs-server/cmd/nfs-server/manifest_test.go b/modules/nfs-server/cmd/nfs-server/manifest_test.go new file mode 100644 index 00000000..7df914fa --- /dev/null +++ b/modules/nfs-server/cmd/nfs-server/manifest_test.go @@ -0,0 +1,118 @@ +package main + +import ( + "encoding/json" + "os" + "regexp" + "sort" + "strings" + "testing" +) + +type manifest struct { + Tools []string `json:"tools"` + Claims []struct { + Name string `json:"name"` + Scope string `json:"scope"` + Serves []string `json:"serves"` + } `json:"claims"` + Provides []map[string]any `json:"provides"` + Listens []struct { + Port int `json:"port"` + Protocol string `json:"protocol"` + From string `json:"from"` + } `json:"listens"` + Settings map[string]any `json:"settings"` + Resources []map[string]any `json:"resources"` +} + +func readManifest(t *testing.T) (manifest, string) { + t.Helper() + raw, err := os.ReadFile("../../module.json") + if err != nil { + t.Fatal(err) + } + var m manifest + if err := json.Unmarshal(raw, &m); err != nil { + t.Fatal(err) + } + return m, string(raw) +} + +// The seat's verbs are served as `node-nfs-server.`, the health tool is listed, and every tool says +// whether it reads or acts. +func TestTheSeatsVerbsAreAllServedAndTheHealthToolListed(t *testing.T) { + m, _ := readManifest(t) + if len(m.Claims) != 1 || m.Claims[0].Name != Seat || m.Claims[0].Scope != "node" { + t.Fatalf("claims: %+v", m.Claims) + } + var verbs, tools []string + for _, tl := range Tools(newFake(t).machine()) { + if name, ok := strings.CutPrefix(tl.Name, Seat+"."); ok { + verbs = append(verbs, name) + } else { + tools = append(tools, tl.Name) + } + if !strings.HasSuffix(tl.Description, "(r)") && !strings.HasSuffix(tl.Description, "(a)") { + t.Errorf("%s does not say whether it reads or acts", tl.Name) + } + } + sort.Strings(verbs) + serves := append([]string(nil), m.Claims[0].Serves...) + sort.Strings(serves) + if strings.Join(verbs, ",") != strings.Join(serves, ",") { + t.Errorf("served %v, the claim lists %v", verbs, serves) + } + if strings.Join(tools, ",") != strings.Join(m.Tools, ",") { + t.Errorf("tools %v, listed %v", tools, m.Tools) + } +} + +// Port 2049 to the mesh's machines and nothing else; the provision is the mesh's, keeping no name of its +// consumers. +func TestItListensOnTwoThousandFortyNineFromTheMeshOnlyAndProvidesTheShare(t *testing.T) { + m, raw := readManifest(t) + if len(m.Listens) != 1 || m.Listens[0].Port != 2049 || m.Listens[0].Protocol != "tcp" || m.Listens[0].From != "mesh" { + t.Errorf("listens: %+v", m.Listens) + } + if len(m.Provides) != 1 || m.Provides[0]["name"] != "nfs-share" || m.Provides[0]["scope"] != "mesh" || m.Provides[0]["identity"] != false { + t.Errorf("provides: %+v", m.Provides) + } + for _, port := range []string{"111", "20048", "\"port\": 445"} { + if strings.Contains(raw, `"port": `+port) { + t.Errorf("the module opens %s", port) + } + } + if !strings.Contains(raw, "vers3=n") { + t.Errorf("NFS version 3 is not switched off, so 2049 alone would not do") + } +} + +// The definition names no folder, no machine and no person (novox/hq ADR 0112): the shares are the +// assignment's setting, and the range the mesh's. +func TestTheDefinitionNamesNoFolderNoMachineAndGivesTheSharesNoDefault(t *testing.T) { + m, raw := readManifest(t) + if hit := regexp.MustCompile(`\b(g14|novox|ace|shanks|jochen)\b|/home/|/storage|/srv/`).FindString(strings.ReplaceAll(raw, "novox/hq", "")); hit != "" { + t.Errorf("the manifest says %q", hit) + } + if _, declared := m.Settings["shares"]; declared { + t.Errorf("the shares are the operator's value and have no default (ADR 0262)") + } + var config string + for _, r := range m.Resources { + if r["id"] == "config" { + config, _ = r["content"].(string) + } + } + if !strings.Contains(config, "shares=${setting:shares}") || !strings.Contains(config, "range=${machine:mesh-range}") { + t.Errorf("the config file: %q", config) + } + // Every long-running resource says how it is ready (ADR 0240 rule 8). + for _, r := range m.Resources { + if r["type"] == "service" || r["type"] == "process" { + if r["health"] == nil { + t.Errorf("%v has no health", r["id"]) + } + } + } +} diff --git a/modules/nfs-server/cmd/nfs-server/nfs_test.go b/modules/nfs-server/cmd/nfs-server/nfs_test.go new file mode 100644 index 00000000..b2f32a87 --- /dev/null +++ b/modules/nfs-server/cmd/nfs-server/nfs_test.go @@ -0,0 +1,389 @@ +package main + +import ( + "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +// fake is a machine under a temporary root, with recorded command answers. +type fake struct { + t *testing.T + root string + answers map[string]Ran + ran []string + now time.Time + uid int + // onExportfs, when set, writes the kernel's table as exportfs -ra would. + onExportfs func() +} + +func newFake(t *testing.T) *fake { + f := &fake{t: t, root: t.TempDir(), answers: map[string]Ran{}, now: time.Date(2026, 10, 8, 19, 21, 0, 0, time.UTC), uid: 0} + f.answers["systemctl is-active nfs-server.service"] = Ran{Out: "active\n"} + f.answers["zfs list -H -o name,mountpoint,sharenfs -t filesystem"] = Ran{Code: -1, Err: `exec: "zfs": executable file not found in $PATH`} + return f +} + +func (f *fake) machine() *Machine { + return &Machine{Root: f.root, UID: f.uid, Now: func() time.Time { return f.now }, Run: func(_ context.Context, name string, args ...string) Ran { + line := strings.Join(append([]string{name}, args...), " ") + f.ran = append(f.ran, line) + if (line == "exportfs -ra" || line == "sudo -n exportfs -ra") && f.onExportfs != nil { + f.onExportfs() + } + if r, ok := f.answers[line]; ok { + return r + } + return Ran{} + }} +} + +func (f *fake) dir(p string) { + if err := os.MkdirAll(filepath.Join(f.root, p), 0o755); err != nil { + f.t.Fatal(err) + } +} + +func (f *fake) file(p, content string) { + f.dir(filepath.Dir(p)) + if err := os.WriteFile(filepath.Join(f.root, p), []byte(content), 0o644); err != nil { + f.t.Fatal(err) + } +} + +func (f *fake) read(p string) string { + b, _ := os.ReadFile(filepath.Join(f.root, p)) + return string(b) +} + +func (f *fake) restic(p string) { + f.file(p+"/config", "x") + for _, d := range []string{"keys", "snapshots", "data"} { + f.dir(p + "/" + d) + } +} + +// kernelReads makes exportfs -ra copy the mesh's export file into the kernel's table, as it would. +func (f *fake) kernelReads() { + f.onExportfs = func() { + var b strings.Builder + for _, e := range ParseExports(f.read(ExportFile), ExportFile) { + fmt.Fprintf(&b, "%s\t%s(%s)\n", e.Path, e.Client, e.Options) + } + f.file(Etab, b.String()) + } +} + +const meshRange = "10.0.0.0/24" + +func config(shares string) string { + return "# test\nshares=" + shares + "\nrange=" + meshRange + "\n" +} + +func TestAnExportLineSquashesEveryClientToTheFoldersOwnerToTheMeshRangeOnly(t *testing.T) { + f := newFake(t) + f.dir("/srv/library") + f.dir("/srv/downloads") + f.file(ConfigFile, config("library=/srv/library downloads=/srv/downloads:ro")) + f.kernelReads() + s := f.machine().Pass(context.Background(), nil) + if len(s.Shares) != 2 || len(s.Refused) != 0 { + t.Fatalf("status: %+v", s) + } + uid, gid := os.Getuid(), os.Getgid() + text := f.read(ExportFile) + for _, want := range []string{ + fmt.Sprintf("/srv/library %s(rw,sync,secure,no_subtree_check,all_squash,anonuid=%d,anongid=%d)", meshRange, uid, gid), + fmt.Sprintf("/srv/downloads %s(ro,sync,secure,no_subtree_check,all_squash,anonuid=%d,anongid=%d)", meshRange, uid, gid), + } { + if !strings.Contains(text, want+"\n") { + t.Errorf("the export file lacks %q:\n%s", want, text) + } + } + for _, e := range ParseExports(text, ExportFile) { + if e.Client != meshRange { + t.Errorf("a client other than the private network's range: %+v", e) + } + if !strings.Contains(e.Options, "all_squash") || strings.Contains(e.Options, "no_root_squash") { + t.Errorf("every client is squashed: %+v", e) + } + } + if !s.Shares[0].Live || !s.Shares[1].Live { + t.Errorf("both shares are live after exportfs -ra: %+v", s.Shares) + } + if s.Reloaded == nil || s.ReloadWhy != "the export file changed" { + t.Errorf("the kernel read the file again because it changed: %+v", s) + } + // Rendering is stable: a second pass changes nothing and asks the kernel nothing. + f.ran = nil + f.machine().Pass(context.Background(), &s) + for _, r := range f.ran { + if strings.Contains(r, "exportfs") { + t.Errorf("a pass with nothing changed reloaded: %v", f.ran) + } + } +} + +func TestTheRootABackupRepositoryAndAMissingFolderAreRefusedAndNothingIsCreated(t *testing.T) { + f := newFake(t) + f.restic("/srv/pool/restic") + f.dir("/srv/pool/media") + f.restic("/srv/backups") + f.file("/srv/a-file", "x") + f.file(ConfigFile, config("root=/ pool=/srv/pool backups=/srv/backups missing=/srv/nowhere file=/srv/a-file rel=srv/x media=/srv/pool/media")) + f.kernelReads() + s := f.machine().Pass(context.Background(), nil) + refused := map[string]string{} + for _, r := range s.Refused { + refused[r.Name] = r.Why + } + for name, word := range map[string]string{ + "root": "root", "pool": "holds a backup repository", "backups": "is a backup repository", + "missing": "not there", "file": "not a folder", "rel": "not an absolute path", + } { + if !strings.Contains(refused[name], word) { + t.Errorf("%s refused with %q, want %q", name, refused[name], word) + } + } + if len(s.Shares) != 1 || s.Shares[0].Name != "media" { + t.Errorf("only the folder beside the backups is exported: %+v", s.Shares) + } + if strings.Contains(f.read(ExportFile), "/srv/pool ") || strings.Contains(f.read(ExportFile), "/srv/backups") { + t.Errorf("a refused folder is in the export file:\n%s", f.read(ExportFile)) + } + if _, err := os.Stat(filepath.Join(f.root, "/srv/nowhere")); err == nil { + t.Errorf("the module created a missing folder") + } + h := f.machine().Health() + if h.Healthy || !strings.Contains(h.Why, "backups are never exported") { + t.Errorf("a refused share makes the module unhealthy, said: %+v", h) + } +} + +func TestTheModuleWritesOnlyItsTwoFilesAndHasNoWayToChangeAnOwnerOrMode(t *testing.T) { + m := newFake(t).machine() + for _, p := range []string{"/srv/library/x", "/etc/exports", "/etc/exports.d/zfs.exports", ConfigFile} { + if err := m.Write(p, "x"); err == nil { + t.Errorf("Write accepted %s", p) + } + } + // No code path changes an owner or a mode, or removes anything: the words are not in the bundle. + files, _ := filepath.Glob("*.go") + for _, file := range files { + if strings.HasSuffix(file, "_test.go") { + continue + } + raw, _ := os.ReadFile(file) + for _, word := range []string{"Chown", "Lchown", "Chmod", "os.Remove", "RemoveAll", `"chown"`, `"chmod"`, `"rm"`, "Mkdir(", "MkdirAll(filepath.Join(m.Root"} { + if strings.Contains(string(raw), word) { + t.Errorf("%s says %s", file, word) + } + } + } +} + +func TestParseSharesAndConfig(t *testing.T) { + shares, problems := ParseShares("library=/srv/library, downloads=/srv/downloads:ro Bad=/x noequals library=/again") + if len(shares) != 2 || shares[0].Name != "downloads" || !shares[0].ReadOnly || shares[1].Access != "rw" { + t.Errorf("shares: %+v", shares) + } + if len(problems) != 3 { + t.Errorf("problems: %v", problems) + } + c, problems := ParseConfig("shares=\nrange=not-a-range\n") + if c.Range != "" || len(problems) != 1 { + t.Errorf("a range that is not one is said: %+v %v", c, problems) + } + _, problems = ParseConfig("shares=a=/x\n") + if len(problems) != 1 || !strings.Contains(problems[0], "no private network range") { + t.Errorf("no range is said: %v", problems) + } +} + +func TestParseExportsReadsTheKernelsTableAndZFSFile(t *testing.T) { + etab := "/srv/library\t192.0.2.0/24(rw,sync,wdelay,root_squash,sec=sys)\n/srv/x\n# comment\n/srv/y a(ro) b(rw)\n" + got := ParseExports(etab, Etab) + if len(got) != 4 || got[0].Client != "192.0.2.0/24" || got[1].Client != "*" || got[3].Client != "b" || got[3].Options != "rw" { + t.Errorf("%+v", got) + } +} + +func zfsWith(f *fake, sharenfs string) { + f.answers["zfs list -H -o name,mountpoint,sharenfs -t filesystem"] = Ran{Out: "pool\t/srv/pool\toff\npool/library\t/srv/library\t" + sharenfs + "\n"} +} + +func TestAdoptIsADryRunWithoutConfirmAndRefusesWhileTheMeshExportIsNotLive(t *testing.T) { + f := newFake(t) + f.uid = 1000 + f.dir("/srv/library") + zfsWith(f, "rw=@192.0.2.0/24") + f.file(ConfigFile, config("library=/srv/library")) + // The kernel has not read the mesh's file: no exportfs effect. + f.machine().Pass(context.Background(), nil) + if _, err := f.machine().Adopt("/srv/library", true, "take it over"); err == nil || !strings.Contains(err.Error(), "not live") { + t.Fatalf("adopt while not live: %v", err) + } + f.kernelReads() + f.machine().Pass(context.Background(), nil) + f.ran = nil + dry, err := f.machine().Adopt("/srv/library", false, "") + if err != nil || dry.Done || !dry.DryRun || !strings.Contains(dry.Said, "would set sharenfs=off on the dataset pool/library") { + t.Fatalf("dry run: %+v %v", dry, err) + } + for _, r := range f.ran { + if strings.Contains(r, "zfs set") { + t.Fatalf("a dry run changed something: %v", f.ran) + } + } + if _, err := f.machine().Adopt("/srv/library", true, ""); err == nil { + t.Errorf("adopt with confirm and no why was done") + } + done, err := f.machine().Adopt("/srv/library", true, "the mesh exports it now") + if err != nil || !done.Done { + t.Fatalf("adopt: %+v %v", done, err) + } + var set []string + for _, r := range f.ran { + if strings.Contains(r, "zfs set") { + set = append(set, r) + } + } + if len(set) != 1 || set[0] != "sudo -n zfs set sharenfs=off pool/library" { + t.Errorf("exactly one property of exactly one dataset, through sudo: %v", set) + } + zfsWith(f, "off") + if _, err := f.machine().Adopt("/srv/library", true, "again"); err == nil || !strings.Contains(err.Error(), "already has sharenfs off") { + t.Errorf("adopt of a dataset already off: %v", err) + } + if _, err := f.machine().Adopt("/srv/elsewhere", false, ""); err == nil { + t.Errorf("adopt of a folder no dataset is mounted at") + } +} + +func TestAdoptRefusesAFolderTheMeshDoesNotExport(t *testing.T) { + f := newFake(t) + f.dir("/srv/library") + zfsWith(f, "on") + f.file(ConfigFile, config("")) + f.kernelReads() + f.machine().Pass(context.Background(), nil) + if _, err := f.machine().Adopt("/srv/library", false, ""); err == nil || !strings.Contains(err.Error(), "does not export") { + t.Errorf("%v", err) + } +} + +func TestHealthSaysAStoppedServerAStaleProcessAndAShareNotInTheTable(t *testing.T) { + f := newFake(t) + f.dir("/srv/library") + f.file(ConfigFile, config("library=/srv/library")) + f.kernelReads() + f.machine().Pass(context.Background(), nil) + if h := f.machine().Health(); !h.Healthy || !strings.Contains(h.Why, "exports library to 10.0.0.0/24") { + t.Errorf("healthy: %+v", h) + } + f.file(Etab, "") + f.answers["systemctl is-active nfs-server.service"] = Ran{Out: "failed\n", Code: 3} + f.now = f.now.Add(10 * time.Minute) + h := f.machine().Health() + for _, want := range []string{"is failed", "last looked 10m0s ago", "The share library (/srv/library) is not in the kernel's export table."} { + if h.Healthy || !strings.Contains(h.Why, want) { + t.Errorf("health lacks %q: %+v", want, h) + } + } + if h := newFake(t).machine().Health(); h.Healthy || !strings.Contains(h.Why, "written no status") { + t.Errorf("no status is unhealthy: %+v", h) + } +} + +func TestExportsAnswersTheSharesAndWhatIsNotTheMeshs(t *testing.T) { + f := newFake(t) + f.dir("/srv/library") + f.file(ConfigFile, config("library=/srv/library")) + f.file("/etc/exports.d/zfs.exports", "# !!! DO NOT EDIT\n/srv/library 192.0.2.0/24(sec=sys,rw,no_subtree_check)\n") + f.file("/etc/exports", "# only comments\n") + zfsWith(f, "rw=@192.0.2.0/24") + f.kernelReads() + f.machine().Pass(context.Background(), nil) + a := f.machine().Exports() + raw, _ := json.Marshal(a) + var shape struct { + Shares []struct { + Name, Path, Access, Clients string + UID, GID int + Live bool + } + Foreign []map[string]any + Service string + } + if err := json.Unmarshal(raw, &shape); err != nil { + t.Fatal(err) + } + if len(shape.Shares) != 1 || shape.Shares[0].Name != "library" || !shape.Shares[0].Live || shape.Shares[0].Clients != meshRange || shape.Service != "active" { + t.Errorf("%s", raw) + } + if len(shape.Foreign) != 2 { + t.Errorf("the ZFS file's line and the dataset's property are not the mesh's: %s", raw) + } + for _, key := range []string{`"shares"`, `"foreign"`, `"service"`, `"uid"`, `"gid"`, `"live"`, `"clients"`, `"access"`} { + if !strings.Contains(string(raw), key) { + t.Errorf("the answer lacks %s, which the mounts module reads", key) + } + } +} + +func TestTestSaysWhetherAShareIsServedAndWhetherAnAddressIsInRange(t *testing.T) { + f := newFake(t) + f.dir("/srv/library") + f.file(ConfigFile, config("library=/srv/library")) + f.kernelReads() + f.machine().Pass(context.Background(), nil) + ok, _ := f.machine().Test("library", "10.0.0.3") + if !ok.OK || ok.InRange == nil || !*ok.InRange { + t.Errorf("%+v", ok) + } + out, _ := f.machine().Test("library", "192.0.2.7") + if out.OK || !strings.Contains(out.Why, "not inside the private network's range") { + t.Errorf("%+v", out) + } + none, _ := f.machine().Test("photos", "") + if none.OK || !strings.Contains(none.Why, "no share named photos") { + t.Errorf("%+v", none) + } +} + +func TestClientsReadTheKernelsListOrFallBackToConnections(t *testing.T) { + f := newFake(t) + f.file("/proc/fs/nfsd/clients/7/info", "clientid: 0xabc\naddress: \"10.0.0.3:785\"\nname: \"Linux NFSv4.2 workstation\"\nminor version: 2\n") + a := f.machine().Clients() + if len(a.Clients) != 1 || a.Clients[0].Address != "10.0.0.3:785" || a.Clients[0].Minor != "2" { + t.Errorf("%+v", a) + } + g := newFake(t) + g.answers["ss -tnH state established ( sport = :2049 )"] = Ran{Out: "0 0 10.0.0.2:2049 10.0.0.3:785\n"} + b := g.machine().Clients() + if len(b.Clients) != 1 || b.Clients[0].Address != "10.0.0.3:785" || b.From != "connections to port 2049" { + t.Errorf("%+v", b) + } +} + +func TestAnActNeedingRootGoesThroughSudoOnlyWhenNotRoot(t *testing.T) { + f := newFake(t) + f.uid = 1000 + f.machine().AsRoot("exportfs", "-ra") + f.uid = 0 + f.machine().AsRoot("exportfs", "-ra") + if len(f.ran) != 2 || f.ran[0] != "sudo -n exportfs -ra" || f.ran[1] != "exportfs -ra" { + t.Errorf("%v", f.ran) + } + f.answers["sudo -n zfs set sharenfs=off x"] = Ran{Code: 1, Err: "sudo: a password is required"} + f.uid = 1000 + if _, err := f.machine().AsRoot("zfs", "set", "sharenfs=off", "x"); err == nil || !strings.Contains(err.Error(), "sudo refused") { + t.Errorf("%v", err) + } +} diff --git a/modules/nfs-server/cmd/nfs-server/process.go b/modules/nfs-server/cmd/nfs-server/process.go new file mode 100644 index 00000000..d416029b --- /dev/null +++ b/modules/nfs-server/cmd/nfs-server/process.go @@ -0,0 +1,114 @@ +package main + +import ( + "context" + "encoding/json" + "fmt" + "time" +) + +// The module's process, as root: every Every it reads the assignment's shares, renders the export file, +// has the kernel read it again when anything changed or a share is not live, and writes what it saw for +// the tools (novox/hq ADR 0263). + +// Every is how often the process looks. +const Every = 30 * time.Second + +// Status is the process's last look, as the tools read it. +type Status struct { + At time.Time `json:"at"` + Range string `json:"range"` + Shares []Exported `json:"shares"` + Refused []Refused `json:"refused,omitempty"` + Problems []string `json:"problems,omitempty"` + // Reloaded is when the process last had the kernel read the export files again, and why. + Reloaded *time.Time `json:"reloaded,omitempty"` + ReloadWhy string `json:"reload_why,omitempty"` + ReloadError string `json:"reload_error,omitempty"` +} + +// Pass is one look: what it did is in the status it answers, which it has written. +func (m *Machine) Pass(ctx context.Context, last *Status) Status { + s := Status{At: m.Now().UTC()} + if last != nil { + s.Reloaded, s.ReloadWhy = last.Reloaded, last.ReloadWhy + } + text, err := m.Read(ConfigFile) + if err != nil { + s.Problems = append(s.Problems, fmt.Sprintf("the assignment's shares (%s) could not be read: %v; nothing is exported", ConfigFile, err)) + } + c, problems := ParseConfig(text) + s.Problems = append(s.Problems, problems...) + s.Range = c.Range + exports, refused := m.Judge(c) + s.Refused = refused + + want := ExportFileText(exports) + have, _ := m.Read(ExportFile) + reload := "" + if have != want { + if err := m.Write(ExportFile, want); err != nil { + s.Problems = append(s.Problems, fmt.Sprintf("the export file could not be written: %v", err)) + } else { + reload = "the export file changed" + } + } + live, note := m.Live() + if reload == "" { + for _, e := range exports { + if !IsLive(live, e.Path, e.Clients) { + reload = fmt.Sprintf("the share %s was not in the kernel's table", e.Name) + break + } + } + } + if reload != "" { + if _, err := m.AsRoot("exportfs", "-ra"); err != nil { + s.ReloadError = err.Error() + } else { + at := m.Now().UTC() + s.Reloaded, s.ReloadWhy = &at, reload + } + live, note = m.Live() + } + if note != "" && len(exports) > 0 { + s.Problems = append(s.Problems, note) + } + for i := range exports { + exports[i].Live = IsLive(live, exports[i].Path, exports[i].Clients) + } + s.Shares = exports + raw, _ := json.MarshalIndent(s, "", " ") + if err := m.Write(StatusFile, string(raw)+"\n"); err != nil { + s.Problems = append(s.Problems, fmt.Sprintf("the status file could not be written: %v", err)) + } + return s +} + +// RunExports is the process: a pass now and every Every until the context ends. +func (m *Machine) RunExports(ctx context.Context) error { + last := m.Pass(ctx, nil) + t := time.NewTicker(Every) + defer t.Stop() + for { + select { + case <-ctx.Done(): + return nil + case <-t.C: + last = m.Pass(ctx, &last) + } + } +} + +// LastStatus is the process's last look, or why there is none. +func (m *Machine) LastStatus() (*Status, error) { + text, err := m.Read(StatusFile) + if err != nil { + return nil, fmt.Errorf("the module's process has written no status (%s): %v", StatusFile, err) + } + var s Status + if err := json.Unmarshal([]byte(text), &s); err != nil { + return nil, fmt.Errorf("the module's status file is not one: %v", err) + } + return &s, nil +} diff --git a/modules/nfs-server/cmd/nfs-server/shares.go b/modules/nfs-server/cmd/nfs-server/shares.go new file mode 100644 index 00000000..59016fc4 --- /dev/null +++ b/modules/nfs-server/cmd/nfs-server/shares.go @@ -0,0 +1,225 @@ +package main + +import ( + "fmt" + "net/netip" + "path" + "sort" + "strings" +) + +// The shares, from the assignment, and the export lines made from them (novox/hq ADR 0263). +// +// **One share per folder.** A share is a name and a path the operator gives in the `shares` setting: +// `library=/srv/library downloads=/srv/downloads`, `:ro` after a path for read-only. The path is the +// operator's data (novox/hq ADR 0051): the module reads its owner and exports it, and never creates, +// chowns, chmods or removes it. +// +// **Every client is the folder's owner.** NFS with sec=sys believes the user id the client sends, and the +// same id names different accounts on different machines. So every line squashes every client +// (`all_squash`) to the owner of the shared folder as this machine reads it (`anonuid`, `anongid`): a file +// written from any node lands owned by the folder's owner, and no client acts as any other account here. +// Root is squashed with everyone else. +// +// **To the mesh's machines only.** The clients are the private network's range and nothing else: every +// address there is a node. The home network is never named. +// +// **Never the backups.** A share that is the machine's root, that does not exist, that is not a +// directory, or that holds a backup repository at its path or one level below is refused, and said. + +// Share is one share the operator asked for. +type Share struct { + Name string `json:"name"` + Path string `json:"path"` + ReadOnly bool `json:"-"` + Access string `json:"access"` +} + +// Exported is a share as this machine exports it. +type Exported struct { + Name string `json:"name"` + Path string `json:"path"` + Access string `json:"access"` + UID int `json:"uid"` + GID int `json:"gid"` + Clients string `json:"clients"` + Line string `json:"line"` + Live bool `json:"live"` +} + +// Refused is a share that is not exported, and why. +type Refused struct { + Name string `json:"name"` + Path string `json:"path"` + Why string `json:"why"` +} + +// Config is what the mesh rendered from the assignment. +type Config struct { + Shares []Share + Range string +} + +// ParseConfig reads the config file: `shares=` and `range=` lines, comments and blanks ignored. +func ParseConfig(text string) (Config, []string) { + var c Config + var problems []string + for _, line := range strings.Split(text, "\n") { + line = strings.TrimSpace(line) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + key, value, ok := strings.Cut(line, "=") + if !ok { + problems = append(problems, fmt.Sprintf("the line %q of %s says nothing this module reads", line, ConfigFile)) + continue + } + switch strings.TrimSpace(key) { + case "shares": + shares, bad := ParseShares(value) + c.Shares = shares + problems = append(problems, bad...) + case "range": + c.Range = strings.TrimSpace(value) + default: + problems = append(problems, fmt.Sprintf("%s names %q, which this module does not read", ConfigFile, key)) + } + } + if c.Range == "" { + problems = append(problems, fmt.Sprintf("%s gives no private network range, so nothing can be exported", ConfigFile)) + } else if _, err := netip.ParsePrefix(c.Range); err != nil { + problems = append(problems, fmt.Sprintf("the private network's range %q is not a range: %v", c.Range, err)) + c.Range = "" + } + return c, problems +} + +// ParseShares reads the `shares` setting: `name=path` or `name=path:ro`, separated by spaces or commas. +func ParseShares(value string) ([]Share, []string) { + var shares []Share + var problems []string + seen := map[string]bool{} + for _, item := range strings.FieldsFunc(value, func(r rune) bool { return r == ' ' || r == ',' || r == '\t' }) { + name, p, ok := strings.Cut(item, "=") + if !ok || name == "" || p == "" { + problems = append(problems, fmt.Sprintf("the share %q is not name=path", item)) + continue + } + if !validName(name) { + problems = append(problems, fmt.Sprintf("the share name %q is not lower-case letters, digits and dashes", name)) + continue + } + if seen[name] { + problems = append(problems, fmt.Sprintf("the share %q is named twice", name)) + continue + } + seen[name] = true + s := Share{Name: name, Path: p, Access: "rw"} + if strings.HasSuffix(p, ":ro") { + s.Path, s.ReadOnly, s.Access = strings.TrimSuffix(p, ":ro"), true, "ro" + } else if strings.HasSuffix(p, ":rw") { + s.Path = strings.TrimSuffix(p, ":rw") + } + shares = append(shares, s) + } + sort.Slice(shares, func(i, j int) bool { return shares[i].Name < shares[j].Name }) + return shares, problems +} + +func validName(n string) bool { + if n == "" || n[0] == '-' { + return false + } + for _, r := range n { + if !(r >= 'a' && r <= 'z' || r >= '0' && r <= '9' || r == '-') { + return false + } + } + return true +} + +// IsBackupRepository says whether a directory is a restic repository: a `config` file beside `keys`, +// `snapshots` and `data`. It looks at names only. +func (m *Machine) IsBackupRepository(dir string) bool { + if !m.Exists(path.Join(dir, "config")) { + return false + } + for _, sub := range []string{"keys", "snapshots", "data"} { + if o, err := m.Stat(path.Join(dir, sub)); err != nil || !o.Dir { + return false + } + } + return true +} + +// Judge turns the asked-for shares into export lines, or refusals. +func (m *Machine) Judge(c Config) ([]Exported, []Refused) { + var out []Exported + var refused []Refused + for _, s := range c.Shares { + no := func(why string) { refused = append(refused, Refused{Name: s.Name, Path: s.Path, Why: why}) } + clean := path.Clean(s.Path) + switch { + case !path.IsAbs(s.Path): + no(fmt.Sprintf("The share %s names %q, which is not an absolute path.", s.Name, s.Path)) + continue + case clean == "/": + no(fmt.Sprintf("The share %s is the machine's root, which is never exported.", s.Name)) + continue + case strings.ContainsAny(s.Path, " \t\"\\#"): + no(fmt.Sprintf("The share %s's path %q has a character an export line cannot carry.", s.Name, s.Path)) + continue + } + o, err := m.Stat(clean) + if err != nil { + no(fmt.Sprintf("The share %s's folder %s is not there (%v); the module never creates it.", s.Name, clean, err)) + continue + } + if !o.Dir { + no(fmt.Sprintf("The share %s's path %s is not a folder.", s.Name, clean)) + continue + } + if m.IsBackupRepository(clean) { + no(fmt.Sprintf("The share %s's folder %s is a backup repository, and backups are never exported.", s.Name, clean)) + continue + } + holds := "" + for _, child := range m.Children(clean) { + if m.IsBackupRepository(path.Join(clean, child)) { + holds = path.Join(clean, child) + break + } + } + if holds != "" { + no(fmt.Sprintf("The share %s's folder %s holds a backup repository (%s), and backups are never exported: share the folders beside it instead.", s.Name, clean, holds)) + continue + } + if c.Range == "" { + no(fmt.Sprintf("The share %s is not exported: the private network's range is not known.", s.Name)) + continue + } + e := Exported{Name: s.Name, Path: clean, Access: s.Access, UID: o.UID, GID: o.GID, Clients: c.Range} + e.Line = ExportLine(e) + out = append(out, e) + } + return out, refused +} + +// ExportLine is one share's line in the export file. +func ExportLine(e Exported) string { + return fmt.Sprintf("%s %s(%s,sync,secure,no_subtree_check,all_squash,anonuid=%d,anongid=%d)", + e.Path, e.Clients, e.Access, e.UID, e.GID) +} + +// ExportFileText is the whole export file. +func ExportFileText(exports []Exported) string { + var b strings.Builder + b.WriteString("# Written by the mesh's module nfs-server (novox/hq ADR 0263). Rendered from its assignment's\n") + b.WriteString("# `shares` setting by the module's own process; edit the setting, never this file.\n") + b.WriteString("# One share per folder, to the private network's range only, every client squashed to the\n") + b.WriteString("# folder's owner.\n") + for _, e := range exports { + fmt.Fprintf(&b, "# share %s\n%s\n", e.Name, e.Line) + } + return b.String() +} diff --git a/modules/nfs-server/cmd/nfs-server/table.go b/modules/nfs-server/cmd/nfs-server/table.go new file mode 100644 index 00000000..b2014ebf --- /dev/null +++ b/modules/nfs-server/cmd/nfs-server/table.go @@ -0,0 +1,133 @@ +package main + +import ( + "context" + "path" + "strings" +) + +// What the machine exports, read: the kernel's table, the other export files, and ZFS's property. + +// Entry is one export: a path, one client, its options, and where it was found. +type Entry struct { + Path string `json:"path"` + Client string `json:"client"` + Options string `json:"options,omitempty"` + From string `json:"from"` +} + +// ParseExports reads an exports(5) file or the kernel's etab: a path, then clients each with options. +func ParseExports(text, from string) []Entry { + var out []Entry + for _, line := range strings.Split(text, "\n") { + if i := strings.Index(line, "#"); i >= 0 { + line = line[:i] + } + fields := strings.Fields(line) + if len(fields) < 1 { + continue + } + p := fields[0] + if len(fields) == 1 { + out = append(out, Entry{Path: p, Client: "*", From: from}) + continue + } + for _, c := range fields[1:] { + client, opts := c, "" + if i := strings.Index(c, "("); i >= 0 { + client, opts = c[:i], strings.TrimSuffix(c[i+1:], ")") + } + if client == "" { + client = "*" + } + out = append(out, Entry{Path: p, Client: client, Options: opts, From: from}) + } + } + return out +} + +// Live is the kernel's export table, or nil and why when it cannot be read. +func (m *Machine) Live() ([]Entry, string) { + text, err := m.Read(Etab) + if err != nil { + return nil, "the kernel's export table (" + Etab + ") could not be read: " + err.Error() + } + return ParseExports(text, Etab), "" +} + +// IsLive says whether the kernel exports a path to a client. +func IsLive(live []Entry, p, client string) bool { + for _, e := range live { + if path.Clean(e.Path) == path.Clean(p) && e.Client == client { + return true + } + } + return false +} + +// Dataset is one ZFS filesystem as `zfs list` answers it. +type Dataset struct { + Name string `json:"name"` + Mountpoint string `json:"mountpoint"` + ShareNFS string `json:"sharenfs"` +} + +// Datasets are the ZFS filesystems, when zfs is there; nil without it. +func (m *Machine) Datasets() ([]Dataset, string) { + r := m.Run(context.Background(), "zfs", "list", "-H", "-o", "name,mountpoint,sharenfs", "-t", "filesystem") + if r.Code != 0 { + if r.Code == -1 && strings.Contains(r.Err, "executable file not found") { + return nil, "" + } + return nil, "zfs list did not answer: " + firstLine(r.Err) + } + var out []Dataset + for _, line := range strings.Split(r.Out, "\n") { + f := strings.Split(line, "\t") + if len(f) != 3 { + continue + } + out = append(out, Dataset{Name: f[0], Mountpoint: f[1], ShareNFS: f[2]}) + } + return out, "" +} + +// Foreign is every export on the machine that is not the mesh's: /etc/exports, the other files in +// /etc/exports.d (ZFS writes one), and every dataset whose sharenfs is on. +func (m *Machine) Foreign() ([]Entry, []Dataset, []string) { + var entries []Entry + var notes []string + files := append([]string{"/etc/exports"}, m.Glob("/etc/exports.d/*.exports")...) + for _, f := range files { + if f == ExportFile { + continue + } + text, err := m.Read(f) + if err != nil { + if m.Exists(f) { + notes = append(notes, f+" could not be read: "+err.Error()) + } + continue + } + entries = append(entries, ParseExports(text, f)...) + } + sets, note := m.Datasets() + if note != "" { + notes = append(notes, note) + } + var shared []Dataset + for _, d := range sets { + if d.ShareNFS != "off" && d.ShareNFS != "-" && d.ShareNFS != "" { + shared = append(shared, d) + } + } + return entries, shared, notes +} + +func firstLine(s string) string { + s = strings.TrimSpace(s) + if i := strings.IndexByte(s, '\n'); i >= 0 { + return s[:i] + } + return s +} diff --git a/modules/nfs-server/cmd/nfs-server/tools.go b/modules/nfs-server/cmd/nfs-server/tools.go new file mode 100644 index 00000000..ba6e33f9 --- /dev/null +++ b/modules/nfs-server/cmd/nfs-server/tools.go @@ -0,0 +1,427 @@ +package main + +import ( + "context" + "fmt" + "net/netip" + "path" + "sort" + "strings" + "time" + + stdio "git.novox.be/novox/mesh-sdk/go" +) + +// Seat is the role this module holds on its machine (novox/hq ADR 0263). +const Seat = "node-nfs-server" + +// Stale is how old the process's last look may be before the tools say it stopped looking. +const Stale = 5 * time.Minute + +// verb is one of the seat's verbs, listed as `.` so the runtime serves it on the seat's +// subject, as /node-nfs-server.. +func verb(name, description string, input map[string]any, run func(a map[string]any) (any, error)) stdio.Tool { + return stdio.Tool{Name: Seat + "." + name, Description: description, Input: input, Run: run} +} + +func str(description string) map[string]any { + return map[string]any{"type": "string", "description": description} +} + +func arg(a map[string]any, k string) string { + v, _ := a[k].(string) + return strings.TrimSpace(v) +} + +func flag(a map[string]any, k string) bool { + v, _ := a[k].(bool) + return v +} + +// Tools are the seat's verbs and the module's health. +func Tools(m *Machine) []stdio.Tool { + return []stdio.Tool{ + verb("exports", "Every share this machine exports for the mesh: its name, folder, rw or ro, the owner every "+ + "client is mapped to (uid, gid), the clients (the private network's range, nothing else) and whether the "+ + "kernel holds it now; the shares the assignment asked for that were refused, and why; and every export "+ + "on the machine that is not the mesh's (/etc/exports, the other files in /etc/exports.d, a ZFS dataset's "+ + "sharenfs). Replaces exportfs -v, cat /etc/exports and zfs get sharenfs. (r)", + map[string]any{}, func(map[string]any) (any, error) { return m.Exports(), nil }), + verb("clients", "Which machines have a share of this machine mounted now: each NFS client the kernel knows, "+ + "with its address and the name it gave, or — when the kernel's list cannot be read — the connections "+ + "to port 2049. Replaces cat /proc/fs/nfsd/clients/*/info and ss -tn sport = :2049. (r)", + map[string]any{}, func(map[string]any) (any, error) { return m.Clients(), nil }), + verb("test", "Whether this machine exports one share for the mesh now: the share is asked for and not "+ + "refused, the kernel holds it for the private network's range, and the NFS server is active; with an "+ + "address, also whether that address is inside the range. Replaces showmount -e. (r)", + map[string]any{"share": str("the share's name"), "address": str("a client's address on the private network (optional)")}, + func(a map[string]any) (any, error) { return m.Test(arg(a, "share"), arg(a, "address")) }), + verb("reload", "Have the kernel read the export files again now (exportfs -ra), and answer the mesh's shares "+ + "as it then holds them. The module's process does this itself whenever its file changes or a share is "+ + "missing; this is for after a change made outside it. Replaces exportfs -ra. (a)", + map[string]any{}, func(map[string]any) (any, error) { return m.Reload() }), + verb("adopt", "Take over the export a ZFS dataset's sharenfs property makes for a folder the mesh now "+ + "exports: sets sharenfs=off on that one dataset and nothing else, then exportfs -ra. Only when the "+ + "mesh's own export of the same folder is live in the kernel, so the folder is never unexported; "+ + "without confirm, a dry run that says what it would do. A person's act, with why. Replaces zfs set "+ + "sharenfs=off. (a)", + map[string]any{ + "path": str("the shared folder, as the share names it"), + "confirm": map[string]any{"type": "boolean", "description": "do it; without, a dry run"}, + "why": str("why, in a sentence (required with confirm)"), + }, + func(a map[string]any) (any, error) { return m.Adopt(arg(a, "path"), flag(a, "confirm"), arg(a, "why")) }), + { + Name: "nfs_health", + Description: "Whether this machine's shares are served, as the module's health check (novox/hq ADR 0240, " + + "ADR 0263): unhealthy when the NFS server is not active, the module's process has not looked for five " + + "minutes, a share the assignment asks for is refused or missing from the kernel's table, or the " + + "export file could not be written or read again; with why, one sentence per finding. (r)", + Run: func(map[string]any) (any, error) { return m.Health(), nil }, + }, + } +} + +// ExportsAnswer is what `exports` answers. +type ExportsAnswer struct { + Shares []Exported `json:"shares"` + Refused []Refused `json:"refused,omitempty"` + Foreign []Entry `json:"foreign"` + Service string `json:"service"` + Range string `json:"range,omitempty"` + Looked string `json:"looked,omitempty"` + Said []string `json:"said"` + Notes []string `json:"notes,omitempty"` +} + +// Exports is the machine's exports now. +func (m *Machine) Exports() ExportsAnswer { + a := ExportsAnswer{Shares: []Exported{}, Foreign: []Entry{}, Service: m.ServiceState()} + st, err := m.LastStatus() + if err != nil { + a.Notes = append(a.Notes, err.Error()) + } else { + a.Range, a.Refused = st.Range, st.Refused + a.Looked = st.At.Format(time.RFC3339) + live, note := m.Live() + if note != "" { + a.Notes = append(a.Notes, note) + } + for _, e := range st.Shares { + if note == "" { + e.Live = IsLive(live, e.Path, e.Clients) + } + a.Shares = append(a.Shares, e) + } + a.Notes = append(a.Notes, st.Problems...) + } + entries, sets, notes := m.Foreign() + a.Notes = append(a.Notes, notes...) + a.Foreign = append(a.Foreign, entries...) + for _, d := range sets { + a.Foreign = append(a.Foreign, Entry{Path: d.Mountpoint, Client: d.ShareNFS, From: "zfs sharenfs on " + d.Name}) + } + for _, e := range a.Shares { + state := "live" + if !e.Live { + state = "NOT in the kernel's table" + } + a.Said = append(a.Said, fmt.Sprintf("%s: %s %s to %s, every client as %d:%d — %s", e.Name, e.Path, e.Access, e.Clients, e.UID, e.GID, state)) + } + for _, r := range a.Refused { + a.Said = append(a.Said, "refused: "+r.Why) + } + for _, f := range a.Foreign { + a.Said = append(a.Said, fmt.Sprintf("not the mesh's: %s to %s (%s)", f.Path, f.Client, f.From)) + } + if len(a.Shares) == 0 && len(a.Refused) == 0 { + a.Said = append(a.Said, "the mesh exports nothing here") + } + a.Said = append(a.Said, "the NFS server is "+a.Service) + return a +} + +// Client is one machine that has a share mounted. +type Client struct { + Address string `json:"address"` + Name string `json:"name,omitempty"` + Minor string `json:"minor_version,omitempty"` +} + +// ClientsAnswer is what `clients` answers. +type ClientsAnswer struct { + Clients []Client `json:"clients"` + From string `json:"from"` + Notes []string `json:"notes,omitempty"` +} + +// ParseClientInfo reads one /proc/fs/nfsd/clients//info. +func ParseClientInfo(text string) Client { + var c Client + for _, line := range strings.Split(text, "\n") { + k, v, ok := strings.Cut(line, ":") + if !ok { + continue + } + v = strings.Trim(strings.TrimSpace(v), `"`) + switch strings.TrimSpace(k) { + case "address": + c.Address = v + case "name": + c.Name = v + case "minor version": + c.Minor = v + } + } + return c +} + +// Clients are the NFS clients the kernel knows, or the connections to 2049. +func (m *Machine) Clients() ClientsAnswer { + a := ClientsAnswer{Clients: []Client{}} + const dir = "/proc/fs/nfsd/clients" + infos := m.Glob(dir + "/*/info") + if len(infos) == 0 && m.UID != 0 { + if r, err := m.AsRoot("find", dir, "-maxdepth", "2", "-name", "info"); err == nil { + for _, l := range strings.Fields(r.Out) { + infos = append(infos, l) + } + } else { + a.Notes = append(a.Notes, err.Error()) + } + } + for _, f := range infos { + text, err := m.Read(f) + if err != nil && m.UID != 0 { + r, e := m.AsRoot("cat", f) + text, err = r.Out, e + } + if err != nil { + a.Notes = append(a.Notes, f+": "+err.Error()) + continue + } + a.Clients = append(a.Clients, ParseClientInfo(text)) + } + if len(infos) > 0 { + a.From = dir + return a + } + r := m.Run(context.Background(), "ss", "-tnH", "state", "established", "( sport = :2049 )") + a.From = "connections to port 2049" + for _, line := range strings.Split(r.Out, "\n") { + f := strings.Fields(line) + if len(f) >= 4 { + a.Clients = append(a.Clients, Client{Address: f[len(f)-1]}) + } + } + return a +} + +// TestAnswer is what `test` answers. +type TestAnswer struct { + Share string `json:"share"` + Path string `json:"path,omitempty"` + Access string `json:"access,omitempty"` + Live bool `json:"live"` + Service string `json:"service"` + InRange *bool `json:"in_range,omitempty"` + OK bool `json:"ok"` + Why string `json:"why"` +} + +// Test says whether one share is served now. +func (m *Machine) Test(share, address string) (TestAnswer, error) { + if share == "" { + return TestAnswer{}, fmt.Errorf("name the share") + } + a := TestAnswer{Share: share, Service: m.ServiceState()} + st, err := m.LastStatus() + if err != nil { + a.Why = err.Error() + return a, nil + } + for _, r := range st.Refused { + if r.Name == share { + a.Path, a.Why = r.Path, r.Why + return a, nil + } + } + var found *Exported + for i := range st.Shares { + if st.Shares[i].Name == share { + found = &st.Shares[i] + } + } + if found == nil { + a.Why = fmt.Sprintf("This machine's assignment asks for no share named %s.", share) + return a, nil + } + a.Path, a.Access = found.Path, found.Access + live, note := m.Live() + a.Live = note == "" && IsLive(live, found.Path, found.Clients) + var why []string + if address != "" { + addr, err1 := netip.ParseAddr(address) + prefix, err2 := netip.ParsePrefix(found.Clients) + in := err1 == nil && err2 == nil && prefix.Contains(addr) + a.InRange = &in + if !in { + why = append(why, fmt.Sprintf("%s is not inside the private network's range %s.", address, found.Clients)) + } + } + if !a.Live { + why = append(why, fmt.Sprintf("The kernel does not export %s to %s now.", found.Path, found.Clients)) + } + if a.Service != "active" { + why = append(why, "The NFS server is "+a.Service+".") + } + a.OK = len(why) == 0 + if a.OK { + a.Why = fmt.Sprintf("The share %s (%s, %s) is exported to %s and the NFS server is active.", share, found.Path, found.Access, found.Clients) + } else { + a.Why = strings.Join(why, " ") + } + return a, nil +} + +// Reload has the kernel read the export files again. +func (m *Machine) Reload() (any, error) { + if _, err := m.AsRoot("exportfs", "-ra"); err != nil { + return nil, err + } + return m.Exports(), nil +} + +// AdoptAnswer is what `adopt` answers. +type AdoptAnswer struct { + Path string `json:"path"` + Dataset string `json:"dataset,omitempty"` + Was string `json:"sharenfs_was,omitempty"` + Done bool `json:"done"` + DryRun bool `json:"dry_run"` + Said string `json:"said"` +} + +// Adopt clears one dataset's sharenfs once the mesh's own export of its folder is live. +func (m *Machine) Adopt(p string, confirm bool, why string) (AdoptAnswer, error) { + if p == "" { + return AdoptAnswer{}, fmt.Errorf("name the shared folder (path)") + } + p = path.Clean(p) + a := AdoptAnswer{Path: p, DryRun: !confirm} + sets, note := m.Datasets() + if note != "" { + return a, fmt.Errorf("%s", note) + } + var ds *Dataset + for i := range sets { + if path.Clean(sets[i].Mountpoint) == p { + ds = &sets[i] + } + } + if ds == nil { + return a, fmt.Errorf("no ZFS dataset is mounted at %s, so there is no sharenfs to take over", p) + } + a.Dataset, a.Was = ds.Name, ds.ShareNFS + if ds.ShareNFS == "off" || ds.ShareNFS == "-" || ds.ShareNFS == "" { + return a, fmt.Errorf("the dataset %s already has sharenfs off; nothing to take over", ds.Name) + } + st, err := m.LastStatus() + if err != nil { + return a, err + } + var mine *Exported + for i := range st.Shares { + if st.Shares[i].Path == p { + mine = &st.Shares[i] + } + } + if mine == nil { + return a, fmt.Errorf("the mesh does not export %s: add it to this machine's nfs-server `shares` setting first, so the folder is never unexported", p) + } + live, lnote := m.Live() + if lnote != "" || !IsLive(live, p, mine.Clients) { + return a, fmt.Errorf("the mesh's export of %s to %s is not live in the kernel yet, so taking over would unexport it; wait for the module's process, or call reload", p, mine.Clients) + } + plan := fmt.Sprintf("set sharenfs=off on the dataset %s (now %q) and run exportfs -ra; %s stays exported as the share %s to %s, every client as %d:%d", + ds.Name, ds.ShareNFS, p, mine.Name, mine.Clients, mine.UID, mine.GID) + if !confirm { + a.Said = "Dry run: would " + plan + "." + return a, nil + } + if why == "" { + return a, fmt.Errorf("say why: adopt is a person's act, and the why is kept with it") + } + if _, err := m.AsRoot("zfs", "set", "sharenfs=off", ds.Name); err != nil { + return a, err + } + if _, err := m.AsRoot("exportfs", "-ra"); err != nil { + return a, fmt.Errorf("sharenfs is off on %s, and exportfs -ra failed: %v", ds.Name, err) + } + a.Done = true + a.Said = fmt.Sprintf("Did: %s. Why: %s", plan, why) + return a, nil +} + +// Health is the module's health, as the node-engine asks it. +type Health struct { + Healthy bool `json:"healthy"` + Why string `json:"why"` + Found []string `json:"found,omitempty"` +} + +// Health judges the shares. +func (m *Machine) Health() Health { + var found []string + service := m.ServiceState() + if service != "active" { + found = append(found, fmt.Sprintf("The NFS server (%s) is %s.", Unit, service)) + } + st, err := m.LastStatus() + if err != nil { + found = append(found, err.Error()+".") + return Health{Healthy: false, Why: strings.Join(found, " "), Found: found} + } + if age := m.Now().Sub(st.At); age > Stale { + found = append(found, fmt.Sprintf("The module's process last looked %s ago, at %s.", age.Round(time.Second), st.At.Local().Format("15:04"))) + } + for _, p := range st.Problems { + found = append(found, sentence(p)) + } + for _, r := range st.Refused { + found = append(found, r.Why) + } + if st.ReloadError != "" { + found = append(found, sentence("the kernel could not read the export files again: "+st.ReloadError)) + } + live, note := m.Live() + for _, e := range st.Shares { + if note == "" && !IsLive(live, e.Path, e.Clients) { + found = append(found, fmt.Sprintf("The share %s (%s) is not in the kernel's export table.", e.Name, e.Path)) + } + } + sort.Strings(found) + if len(found) > 0 { + return Health{Healthy: false, Why: strings.Join(found, " "), Found: found} + } + names := make([]string, 0, len(st.Shares)) + for _, e := range st.Shares { + names = append(names, e.Name) + } + if len(names) == 0 { + return Health{Healthy: true, Why: "The NFS server is active, and the mesh exports nothing here."} + } + return Health{Healthy: true, Why: fmt.Sprintf("The NFS server is active and exports %s to %s.", strings.Join(names, ", "), st.Range)} +} + +func sentence(s string) string { + s = strings.TrimSpace(s) + if s == "" { + return s + } + s = strings.ToUpper(s[:1]) + s[1:] + if !strings.HasSuffix(s, ".") { + s += "." + } + return s +} diff --git a/modules/nfs-server/go.mod b/modules/nfs-server/go.mod new file mode 100644 index 00000000..6e122025 --- /dev/null +++ b/modules/nfs-server/go.mod @@ -0,0 +1,5 @@ +module nfsserver + +go 1.22 + +require git.novox.be/novox/mesh-sdk/go v0.1.7 diff --git a/modules/nfs-server/go.sum b/modules/nfs-server/go.sum new file mode 100644 index 00000000..b474419e --- /dev/null +++ b/modules/nfs-server/go.sum @@ -0,0 +1,2 @@ +git.novox.be/novox/mesh-sdk/go v0.1.7 h1:C0sTQmtTiyYH7bnqZb7PusXnqA37gKuT7Nqjn9gG47w= +git.novox.be/novox/mesh-sdk/go v0.1.7/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY= diff --git a/modules/nfs-server/module.json b/modules/nfs-server/module.json new file mode 100644 index 00000000..780e09ba --- /dev/null +++ b/modules/nfs-server/module.json @@ -0,0 +1,142 @@ +{ + "module": "nfs-server", + "version": "1", + "upgrade": { + "policy": "record", + "why": "the folders other machines mount: a build that breaks the exports leaves every client's mount hanging or refused, and the gate on this one machine does not see the clients (hq ADR 0236, ADR 0263)" + }, + "capabilities": [ + "package-manager", + "service-manager" + ], + "provides": [ + { + "name": "nfs-share", + "scope": "mesh", + "identity": false + } + ], + "data": { + "consumers": { + "nfs-share": { + "class": "none", + "why": "the shared folders are the operator's data (hq ADR 0051): what a client writes lands in them, and they are protected where the operator declares them, never by this module, which keeps nothing of a consumer's" + } + } + }, + "claims": [ + { + "name": "node-nfs-server", + "scope": "node", + "serves": [ + "exports", + "clients", + "test", + "reload", + "adopt" + ] + } + ], + "state": [ + { + "name": "exports", + "ttl-seconds": 120, + "per-machine": true + } + ], + "tools": [ + "nfs_health" + ], + "listens": [ + { + "name": "nfs", + "port": 2049, + "protocol": "tcp", + "from": "mesh", + "fixed": true, + "why": "the shares, to the mesh's machines only (hq ADR 0263): NFS version 4 alone, which needs no other port, and never the home network, where a device that is not a node could claim any user id" + } + ], + "resources": [ + { + "id": "package", + "type": "package", + "package": "nfs-utils" + }, + { + "id": "nfs-conf", + "type": "file", + "path": "/etc/nfs.conf.d/50-mesh.conf", + "mode": "0644", + "content": "# Written by the mesh (module nfs-server, novox/hq ADR 0263). Replaced on every push; a drop-in of\n# the operator's that sorts after this one overrides it, and is theirs.\n#\n# NFS version 4 only: a client needs port 2049 and nothing else, so the module opens nothing more\n# than that, to the private network. Version 3 needs rpcbind and mountd, on ports the mesh does not open.\n[nfsd]\nvers2=n\nvers3=n\nvers4=y\nvers4.0=n\nvers4.1=y\nvers4.2=y\n" + }, + { + "id": "config-dir", + "type": "directory", + "path": "/etc/nfs-server", + "mode": "0755" + }, + { + "id": "config", + "type": "file", + "path": "/etc/nfs-server/shares.conf", + "mode": "0644", + "content": "# Written by the mesh (module nfs-server, novox/hq ADR 0263) from this machine's assignment.\n# Replaced on every push; change the `shares` setting, never this file.\n#\n# The shares: name=folder, or name=folder:ro, one share per folder. The module's process exports each\n# to the private network's range below, every client mapped to the folder's owner.\nshares=${setting:shares}\nrange=${machine:mesh-range}\n" + }, + { + "id": "run-dir", + "type": "directory", + "path": "/run/nfs-server", + "mode": "0755" + }, + { + "id": "server", + "type": "service", + "unit": "nfs-server.service", + "state": "running", + "boot": "enabled", + "restart-on": [ + "nfs-conf" + ], + "health": { + "kind": "unit" + } + }, + { + "id": "exports", + "type": "process", + "name": "nfs-server-exports", + "artifact": "tools", + "run": [ + "./nfs-server", + "exports" + ], + "restart-on": [ + "config" + ], + "health": { + "kind": "tool", + "tool": "nfs_health", + "interval": "60s", + "timeout": "10s", + "looks": 2, + "grace": "90s" + } + } + ], + "build": { + "artifacts": [ + { + "name": "tools", + "kind": "bundle", + "language": "go", + "system": "arch", + "from": "cmd/nfs-server", + "binary": "nfs-server", + "loads": [ + "nfs-server" + ] + } + ] + } +}