mesh-catalog/modules/nfs-server

nfs-server

A node shares a folder with the mesh's machines, as a module (novox/hq ADR 0263, to-be 52). It holds the node seat node-nfs-server and provides the provision nfs-share, which the mounts module on another node requires.

The rules it keeps

• One share per folder. The shares are the assignment's shares setting: name=folder, or name=folder:ro, separated by spaces. The setting has no default: which folders a machine shares is the operator's value (ADR 0112, ADR 0262).
• To the nodes that ask, as far as this machine allows. A share is exported to a node only when this machine's grants setting names that node for it (share=node:rw|ro[,node:rw|ro], separated by spaces; none, the default, grants nothing) and that node's mounts module wants it from this machine (its state shares on the bus). The access is the lower of the grant and the want, and of the share's own :ro. The address is the node's private address from the mesh's roster (/etc/nfs-server/machines, rendered by the mesh), never one a consumer gives, and it must lie inside the private network's range. The control node, or any node, gets a share only when both name it. A want that is not granted, and a grant nobody wants, are said by exports. Port 2049 is opened to the mesh only; the home network is never named.
• Every client is the folder's owner. NFS with sec=sys believes the user id a client sends, and one id names different accounts on different machines. Every line is all_squash with anonuid and anongid read from the folder's owner on this machine, so a file written from any node lands owned by the folder's owner, and root is squashed with everyone else.
• Judged where it really is. A share's path is resolved through every symlink, and every check runs on the real folder, which is what the export line names.
• Never root's. A folder owned by root, or by root's group, is refused: squashing every client to root would make every client root there.
• Never the backups. A folder that is the machine's root, is not there, is not a folder, is inside a backup (restic) repository (any parent up to /), or holds one within four levels below it is refused, said by the tools, and makes the module unhealthy. The look below is bounded (200 000 folders); when it is cut short, the share is refused rather than exported unchecked. A clean look is believed for an hour.
• Never where the backup seat keeps its data, checked on every pass. So that the hour cannot hide a repository started inside it, the tools ask the controller over the bus, every five minutes, which module holds node-backup on this machine (seat:mesh-controller.seats) and every path that module keeps data in here (seat:mesh-controller.data; on the home server the restic repository and its state). They leave it in /var/lib/nfs-server/from-bus/backup.json, and every pass refuses a share at, inside or holding one of those paths, compared after resolving symbolic links, with nothing kept. A failed ask keeps the last paths, and the status and the health warn that they are the last answer, with its age and why. Until a list has been learnt, what is exported stays exported and the health warns that the list is not known: a slow bus is no reason to take the library away, and the look below still guards. A warning is said in the health without making the module unhealthy. This is a deliberate choice: while the backup seat's paths are unknown the module stays healthy with a warning, rather than unhealthy or refusing, because the look below every share still refuses one that holds a repository, and taking the library away from every node because the bus was slow would be the larger harm.
• Only while its filesystem is mounted. A share that is a filesystem of its own is exported with mp, a folder inside one with mp=<its mount point>, so an unmounted dataset never exports the empty folder under it. A folder of the root filesystem carries neither.
• The folders are the operator's (ADR 0051). The module reads a folder's owner and exports it. It never creates, chowns, chmods or removes anything under a shared path; its code writes two files of its own and nothing else, and a test holds it to that.
• NFS version 4 only, so 2049 is the only port a client needs: rpcbind and mountd stay closed.

What it owns

• the nfs-utils package and nfs-server.service, running and enabled;
• /etc/nfs.conf.d/50-mesh.conf: versions 2, 3 and 4.0 off, 4.1 and 4.2 on;
• /etc/nfs-server/shares.conf: the settings shares and grants and the range, rendered by the mesh;
• /etc/nfs-server/machines: every machine of the mesh and its private address, rendered by the mesh;
• /var/lib/nfs-server/from-bus/wants.json: what the nodes want of this machine, written by the tools from the bus (the directory belongs to the operator account the tools run as);
• /etc/exports.d/mesh.exports: written by its process, with a marker header;
• /run/nfs-server/status.json: the process's last look, for the tools;
• the state exports on the bus, one key per machine: what exports answers, for a consumer's module and for a view (ADR 0201).

How it runs

• Its process (nfs-server exports, as root) looks every 30 seconds. It reads the shares, the grants, the roster and the wants, judges each folder, writes the export file when it changed, and runs exportfs -ra when the file changed or a share is missing from the kernel's table. Then it writes its status. With no wants taken from the bus yet, it exports nothing.
• Its tools also take every node's mounts state from the bus every 30 seconds and write what is asked of this machine to the wants file; when the bus does not answer, the last wants stay.
• Its tools run as the operator account. They read the status and the kernel's table, and run what only root may do (exportfs -ra, zfs set) through sudo -n, like the packet filter's tools.
• Health is the tool nfs_health, asked by the node-engine every minute: unhealthy when the NFS server is not active, the process has not looked for five minutes, a share is refused or missing from the kernel's table, or the kernel could not read the files again.

Tools

tool		answers
node-nfs-server.exports	r	every share: name, folder, rw or ro, the owner every client is mapped to, the clients, whether it is live; refused shares and why; every export that is not the mesh's (/etc/exports, other files in /etc/exports.d, ZFS sharenfs)
node-nfs-server.clients	r	the NFS clients the kernel knows (address, name), or the connections to 2049
node-nfs-server.test	r	whether one share is exported now and the server is active; with an address, whether it is inside the range and the share is exported to it
node-nfs-server.reload	a	exportfs -ra, then the shares as the kernel holds them
node-nfs-server.adopt	a	take over a ZFS dataset's sharenfs export of a folder the mesh exports: sharenfs=off on that one dataset, then exportfs -ra; only when the mesh's export of the folder is live to at least one node; a dry run unless confirm is true (or "true"); a person's act, with why
nfs_health	r	the module's health, one sentence per finding

exports answers, for the mounts module:

{"shares": [{"name", "path", "access", "uid", "gid", "clients", "to": [{"node", "address", "access"}],
             "mp", "line", "live"}], "unmatched": [...],
 "refused": [{"name", "path", "why"}], "foreign": [{"path", "client", "options", "from"}],
 "service": "active", "range", "looked", "said", "notes"}

When it is unassigned

The process writes /etc/exports.d/mesh.exports, not the node-engine, because each line carries the folder's owner and the nodes' addresses as this machine reads them. The node-engine removes a module's resources in reverse order of declaration (mesh-host internal/apply), and the process is declared before its config file, so on unassignment the config file is gone when the process is stopped. The process then removes its export file — the one removal its code makes — and runs exportfs -ra, so the kernel lets the shares go. A plain restart, a push or a shutdown leaves the config file in place, and the exports stay.

Two signs of being taken away. The process takes a stop as an unassignment when its config file is gone, or when its own unit (nfs-server-exports.service) is disabled: the node-engine stops a process it removes with systemctl disable --now, which disables the unit first, while a restart, a push or a shutdown leaves it enabled. The second sign covers a first apply that failed and recorded the process after its config (mesh-host #62), when the config is still there at the stop.

Within the stop timeout. The service manager kills a process that has not stopped 90 s after SIGTERM (mesh-host #62), and a killed process cleans up nothing. So SIGTERM ends a running pass at once (its commands and its look below a share are cancelled, and it writes nothing), and the clean-up has its own bound of 10 s: the export file is removed first, which needs no command, then exportfs -ra runs within that bound. Whatever it could not finish is logged as nfs-server: UNASSIGNED, NOT CLEAN: …, with what a person must do.

If it is killed anyway (it did not stop within the service manager's 90 s), nothing cleans up and the module never starts again. Then, on the home server as root: remove /etc/exports.d/mesh.exports and run exportfs -ra; exportfs -v should list no client of the private network's range for the module's folders.

The kernel's table follows the file

The process has the kernel read the export files again (exportfs -ra) on its first pass, after any pass whose reload failed, when the file changed, when a share it exports is missing from the kernel's table, and when the kernel exports one of the module's folders to a client the file no longer has — a client inside the private network's range that no other export file names. While the kernel still exports such an entry, the module is unhealthy and says which. An export another file makes (ZFS's own, /etc/exports) is not the module's and is left alone.

A hand-made export does not survive. exportfs -ra makes the kernel's table exactly what the export files say, so an export a person added by hand with exportfs -o … (in no file) is dropped by the first pass of every start of the process, and by any later reload. An export meant to last belongs in a file: the module's settings for a share of the mesh, or /etc/exports for anything else.

What it leaves

• A subfolder export shares its filesystem. A share that is a folder inside a larger filesystem (not a filesystem of its own) is exported with no_subtree_check, so a client that forges file handles could reach other folders of that filesystem. Every client is a node and is squashed to the share's owner, so a folder that owner cannot read stays closed. A share that must be strictly fenced is better given its own dataset.
• Exports that are not the mesh's (/etc/exports, a ZFS dataset's sharenfs) are left as found and shown, until adopt takes one over on a person's word.

Root acts and sudo

The tools run as the operator account and act as root through sudo -n. The sudo module grants that account every command, so this module's code is the only line between a tool call and root: each act checks again what it may touch before it runs a command, and a test holds each check. Narrowing what sudo grants the account to the commands these modules need is a separate decision, not made here.