jochen
26f9c6e10d
mesh/merge-gate pass: builds nfs-server, sent nowhere; no bus step; 1 wait(s) for a person; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery superseded: a newer delivery to the same trunk took over its walk
156 lines
12 KiB
Markdown
156 lines
12 KiB
Markdown
# nfs-server
|
|
|
|
A node shares a folder with the mesh's machines, as a module (novox/hq ADR 0263, to-be 52). It holds
|
|
the node seat `node-nfs-server` and provides the provision `nfs-share`, which the `mounts` module on
|
|
another node requires.
|
|
|
|
## The rules it keeps
|
|
|
|
- **One share per folder.** The shares are the assignment's `shares` setting: `name=folder`, or
|
|
`name=folder:ro`, separated by spaces. The setting has no default: which folders a machine shares is
|
|
the operator's value (ADR 0112, ADR 0262).
|
|
- **To the nodes that ask, as far as this machine allows.** A share is exported to a node only when
|
|
this machine's `grants` setting names that node for it (`share=node:rw|ro[,node:rw|ro]`, separated by
|
|
spaces; `none`, the default, grants nothing) **and** that node's `mounts` module wants it from this
|
|
machine (its state `shares` on the bus). The access is the lower of the grant and the want, and of the
|
|
share's own `:ro`. The address is the node's private address from the mesh's roster
|
|
(`/etc/nfs-server/machines`, rendered by the mesh), never one a consumer gives, and it must lie inside
|
|
the private network's range. The control node, or any node, gets a share only when both name it. A
|
|
want that is not granted, and a grant nobody wants, are said by `exports`. Port 2049 is opened to the
|
|
mesh only; the home network is never named.
|
|
- **Every client is the folder's owner.** NFS with `sec=sys` believes the user id a client sends, and one
|
|
id names different accounts on different machines. Every line is `all_squash` with `anonuid` and
|
|
`anongid` read from the folder's owner on this machine, so a file written from any node lands owned by
|
|
the folder's owner, and root is squashed with everyone else.
|
|
- **Judged where it really is.** A share's path is resolved through every symlink, and every check runs
|
|
on the real folder, which is what the export line names.
|
|
- **Never root's.** A folder owned by root, or by root's group, is refused: squashing every client to
|
|
root would make every client root there.
|
|
- **Never the backups.** A folder that is the machine's root, is not there, is not a folder, is inside a
|
|
backup (restic) repository (any parent up to `/`), or holds one within four levels below it is refused,
|
|
said by the tools, and makes the module unhealthy. The look below is bounded (200 000 folders); when it
|
|
is cut short, the share is refused rather than exported unchecked. A clean look is believed for an hour.
|
|
- **Never where the backup seat keeps its data, checked on every pass.** So that the hour cannot hide a
|
|
repository started inside it, the tools ask the controller over the bus, every five minutes, which module
|
|
holds `node-backup` on this machine (`seat:mesh-controller.seats`) and every path that module keeps data
|
|
in here (`seat:mesh-controller.data`; on the home server the restic repository and its state). They
|
|
leave it in `/var/lib/nfs-server/from-bus/backup.json`, and every pass refuses a share at, inside or
|
|
holding one of those paths, compared after resolving symbolic links, with nothing kept. A failed ask
|
|
keeps the last paths, and the status and the health warn that they are the last answer, with its age
|
|
and why. Until a list has been learnt, what is exported stays exported and the health warns that the
|
|
list is not known: a slow bus is no reason to take the library away, and the look below still guards.
|
|
A warning is said in the health without making the module unhealthy. **This is a deliberate choice**:
|
|
while the backup seat's paths are unknown the module stays healthy with a warning, rather than
|
|
unhealthy or refusing, because the look below every share still refuses one that holds a repository,
|
|
and taking the library away from every node because the bus was slow would be the larger harm.
|
|
- **Only while its filesystem is mounted.** A share that is a filesystem of its own is exported with `mp`,
|
|
a folder inside one with `mp=<its mount point>`, so an unmounted dataset never exports the empty folder
|
|
under it. A folder of the root filesystem carries neither.
|
|
- **The folders are the operator's** (ADR 0051). The module reads a folder's owner and exports it. It
|
|
never creates, chowns, chmods or removes anything under a shared path; its code writes two files of its
|
|
own and nothing else, and a test holds it to that.
|
|
- **NFS version 4 only**, so 2049 is the only port a client needs: rpcbind and mountd stay closed.
|
|
|
|
## What it owns
|
|
|
|
- the `nfs-utils` package and `nfs-server.service`, running and enabled;
|
|
- `/etc/nfs.conf.d/50-mesh.conf`: versions 2, 3 and 4.0 off, 4.1 and 4.2 on;
|
|
- `/etc/nfs-server/shares.conf`: the settings `shares` and `grants` and the range, rendered by the mesh;
|
|
- `/etc/nfs-server/machines`: every machine of the mesh and its private address, rendered by the mesh;
|
|
- `/var/lib/nfs-server/from-bus/wants.json`: what the nodes want of this machine, written by the tools
|
|
from the bus (the directory belongs to the operator account the tools run as);
|
|
- `/etc/exports.d/mesh.exports`: written by its process, with a marker header;
|
|
- `/run/nfs-server/status.json`: the process's last look, for the tools;
|
|
- the state `exports` on the bus, one key per machine: what `exports` answers, for a consumer's module
|
|
and for a view (ADR 0201).
|
|
|
|
## How it runs
|
|
|
|
- **Its process** (`nfs-server exports`, as root) looks every 30 seconds. It reads the shares, the grants,
|
|
the roster and the wants, judges each folder, writes the export file when it changed, and runs
|
|
`exportfs -ra` when the file changed or a share is missing from the kernel's table. Then it writes its
|
|
status. With no wants taken from the bus yet, it exports nothing.
|
|
- **Its tools** also take every node's `mounts` state from the bus every 30 seconds and write what is
|
|
asked of this machine to the wants file; when the bus does not answer, the last wants stay.
|
|
- **Its tools** run as the operator account. They read the status and the kernel's table, and run what
|
|
only root may do (`exportfs -ra`, `zfs set`) through `sudo -n`, like the packet filter's tools.
|
|
- **Health** is the tool `nfs_health`, asked by the node-engine every minute: unhealthy when the NFS
|
|
server is not active, the process has not looked for five minutes, a share is refused or missing from
|
|
the kernel's table, or the kernel could not read the files again.
|
|
|
|
## Tools
|
|
|
|
| tool | | answers |
|
|
|---|---|---|
|
|
| `node-nfs-server.exports` | r | every share: name, folder, rw or ro, the owner every client is mapped to, the clients, whether it is live; refused shares and why; every export that is not the mesh's (`/etc/exports`, other files in `/etc/exports.d`, ZFS `sharenfs`) |
|
|
| `node-nfs-server.clients` | r | the NFS clients the kernel knows (address, name), or the connections to 2049 |
|
|
| `node-nfs-server.test` | r | whether one share is exported now and the server is active; with an address, whether it is inside the range and the share is exported to it |
|
|
| `node-nfs-server.reload` | a | `exportfs -ra`, then the shares as the kernel holds them |
|
|
| `node-nfs-server.adopt` | a | take over a ZFS dataset's `sharenfs` export of a folder the mesh exports: `sharenfs=off` on that one dataset, then `exportfs -ra`; only when the mesh's export of the folder is live to at least one node; a dry run unless `confirm` is `true` (or "true"); a person's act, with why |
|
|
| `nfs_health` | r | the module's health, one sentence per finding |
|
|
|
|
`exports` answers, for the `mounts` module:
|
|
|
|
```
|
|
{"shares": [{"name", "path", "access", "uid", "gid", "clients", "to": [{"node", "address", "access"}],
|
|
"mp", "line", "live"}], "unmatched": [...],
|
|
"refused": [{"name", "path", "why"}], "foreign": [{"path", "client", "options", "from"}],
|
|
"service": "active", "range", "looked", "said", "notes"}
|
|
```
|
|
|
|
## When it is unassigned
|
|
|
|
The process writes `/etc/exports.d/mesh.exports`, not the node-engine, because each line carries the
|
|
folder's owner and the nodes' addresses as this machine reads them. The node-engine removes a module's
|
|
resources in reverse order of declaration (mesh-host `internal/apply`), and the process is declared before
|
|
its config file, so on unassignment the config file is gone when the process is stopped. The process then
|
|
removes its export file — the one removal its code makes — and runs `exportfs -ra`, so the kernel lets the
|
|
shares go. A plain restart, a push or a shutdown leaves the config file in place, and the exports stay.
|
|
|
|
**Two signs of being taken away.** The process takes a stop as an unassignment when its config file is
|
|
gone, or when its own unit (`nfs-server-exports.service`) is disabled: the node-engine stops a process it
|
|
removes with `systemctl disable --now`, which disables the unit first, while a restart, a push or a
|
|
shutdown leaves it enabled. The second sign covers a first apply that failed and recorded the process
|
|
after its config (mesh-host #62), when the config is still there at the stop.
|
|
|
|
**Within the stop timeout.** The service manager kills a process that has not stopped 90 s after SIGTERM
|
|
(mesh-host #62), and a killed process cleans up nothing. So SIGTERM ends a running pass at once (its
|
|
commands and its look below a share are cancelled, and it writes nothing), and the clean-up has its own
|
|
bound of 10 s: the export file is removed first, which needs no command, then `exportfs -ra` runs within
|
|
that bound. Whatever it could not finish is logged as `nfs-server: UNASSIGNED, NOT CLEAN: …`, with what a
|
|
person must do.
|
|
|
|
**If it is killed anyway** (it did not stop within the service manager's 90 s), nothing cleans up and the
|
|
module never starts again. Then, on the home server as root: remove `/etc/exports.d/mesh.exports` and run
|
|
`exportfs -ra`; `exportfs -v` should list no client of the private network's range for the module's folders.
|
|
|
|
## The kernel's table follows the file
|
|
|
|
The process has the kernel read the export files again (`exportfs -ra`) on its first pass, after any
|
|
pass whose reload failed, when the file changed, when a share it exports is missing from the kernel's
|
|
table, and when the kernel exports one of the module's folders to a client the file no longer has — a
|
|
client inside the private network's range that no other export file names. While the kernel still
|
|
exports such an entry, the module is unhealthy and says which. An export another file makes (ZFS's own,
|
|
`/etc/exports`) is not the module's and is left alone.
|
|
|
|
**A hand-made export does not survive.** `exportfs -ra` makes the kernel's table exactly what the export
|
|
files say, so an export a person added by hand with `exportfs -o …` (in no file) is dropped by the first
|
|
pass of every start of the process, and by any later reload. An export meant to last belongs in a file:
|
|
the module's settings for a share of the mesh, or `/etc/exports` for anything else.
|
|
|
|
## What it leaves
|
|
- **A subfolder export shares its filesystem.** A share that is a folder inside a larger filesystem (not
|
|
a filesystem of its own) is exported with `no_subtree_check`, so a client that forges file handles could
|
|
reach other folders of that filesystem. Every client is a node and is squashed to the share's owner, so
|
|
a folder that owner cannot read stays closed. A share that must be strictly fenced is better given its
|
|
own dataset.
|
|
- **Exports that are not the mesh's** (`/etc/exports`, a ZFS dataset's `sharenfs`) are left as found and
|
|
shown, until `adopt` takes one over on a person's word.
|
|
|
|
## Root acts and sudo
|
|
|
|
The tools run as the operator account and act as root through `sudo -n`. The `sudo` module grants that
|
|
account every command, so **this module's code is the only line** between a tool call and root: each act
|
|
checks again what it may touch before it runs a command, and a test holds each check. Narrowing what sudo
|
|
grants the account to the commands these modules need is a separate decision, not made here.
|