ADR 0263 and design 52 after review: what each export promises, per-node grants, refusals, not-built checks, and a read-only-first rollout

This commit is contained in:
jochen
2026-10-08 20:21:48 +02:00
parent f4b79ba7c2
commit bffd0f90d0
2 changed files with 91 additions and 33 deletions
@@ -72,22 +72,32 @@ Rejected:
path, and read-only or read-write. The path is the operator's data (ADR 0051): the module never
creates, chowns, chmods or removes anything under it. It writes its own file in `/etc/exports.d`, never `/etc/exports`, and listens on 2049/tcp
from the private network. It serves NFS version 4 only, so no other port opens.
2. **Every export squashes every client to the share's owner.** `all_squash`, with `anonuid` and
`anongid` read by the module from the exported path's owner on the server, never written as
numbers. The clients are `${machine:mesh-range}` and nothing else. A file written from any node lands
owned by the library's owner, and no client acts as any other account on the server.
2. **Every export maps every client to the share's owner.** `all_squash`, with `anonuid` and `anongid`
read by the module from the exported folder's owner on the server (after following any symbolic link
to the real folder), never written as numbers. A share owned by root is refused, so no client is ever
mapped to root. A file written from any node lands owned by the folder's owner. **What this does not
promise:** a client acts as that owner, so it can do whatever the owner may do with what it reaches.
Where a share is a folder inside a larger filesystem, NFS cannot fully confine a client that forges
file handles to that folder; it can reach the rest of that filesystem as the owner. That is why
rule 3 refuses a share at or under a backup repository, or with one below it, and why the export
carries `mp`, so it is served only while its filesystem is mounted.
3. **One share per folder, never a pool's root** (the operator's choice, 2026-10-08): on the home server,
the library and the downloads. The backup repository and the archive are not exported. The module
refuses a share that is `/` or that holds a backup repository at its path or one level below, so a
setting cannot export one by mistake.
refuses a share that is `/`, is owned by root, is inside a backup repository, or holds one within four
levels below (refusing when it cannot finish looking), all judged on the real folder after any
symbolic link, so a setting cannot export one by mistake.
4. **A node mounts through one module.** The module `mounts` holds the node seat `node-mounts`. For each
share it writes a `.mount` unit and an `.automount` unit, never an fstab line. Only the automount is
enabled, so nothing mounts at boot and nothing can fail the boot. Each mount has an idle timeout
(default 10 minutes) and a mount timeout (default 30 s; 10 s for an occasional source). A read-write
NFS share is mounted `hard`. A mount point is always an autofs mount, so nothing writes into the
empty directory under it (issue 169's data-loss case).
5. **Only the nodes that ask get a share.** A node mounts what its `mounts` assignment's setting `shares`
lists. The list may be set once at the mesh-wide settings layer, which is how "every node" is said,
5. **Only the nodes that ask, and that the server grants, get a share.** A share is exported to one
node's address only when the server's setting `grants` names that node for that share and the node's
`mounts` assignment asks for it; the access is the lower of the two, decided on the server. Addresses
come from the controller's list of machines, never from what a client says. A share nobody is granted
is exported to nobody: the control-node, like any node, gets nothing unless both name it. A node mounts
what its `mounts` assignment's setting `shares` lists. The list may be set once at the mesh-wide settings layer, which is how "every node" is said,
and a node's own layer can leave a share out. Which nodes are assigned, and whether the laptop mounts
read-only, is decided at assignment, which is the operator's; the recommendation is the workstation
and the laptop (read-only), not the control-node.
@@ -137,11 +147,13 @@ Rejected:
- The `mounts` module's `mounts_list` shows every mount point's source, its unit pair, its state and
which settings layer asked for it. `mounts_exports` shows the server's export table. A test in the
module renders the unit pair from a binding and asserts the options above.
- The lab composes a server and two clients and asserts: the export names only the private network's
range with `all_squash`; a file written from a client is owned by the share's owner; the client's
boot completes with the server stopped; and an occasional source that is absent leaves no failed unit.
- The controller's self-check counts fstab lines of type `nfs` or `cifs` on every node through
`mounts_list`, and reports any that are not adopted.
- **Not built yet:** a lab scenario composing a server and two clients, asserting that the export names
only the granted nodes' addresses with `all_squash`, that a file written from a client is owned by the
share's owner, that a client's boot completes with the server stopped, and that an absent occasional
source leaves no failed unit.
- **Not built yet:** a probe in the controller's self-check that counts fstab lines of type `nfs` or
`cifs` on every node through `mounts_list`, and reports any that are not adopted.
- Until both exist, the modules' own tests and the safe rollout in design 52 are the check.
## References
+66 -20
View File
@@ -61,14 +61,26 @@ The glossary holds the three words.
- **Its setting `shares`.** A list of `name=path`, each optionally `:ro`, separated by spaces: on the
home server the library and the downloads. A share's path is the operator's data (ADR 0051); the
module never creates, chowns, chmods or removes anything under it.
- **What it refuses.** A path that does not exist or is not a directory, `/`, and a path that holds a
backup repository at its root or one level below. A refused share is said by the tools and makes the
module unhealthy; the others are exported.
- **Its setting `grants`.** Which node may mount which share, and how: `library=<workstation>:rw,<laptop>:ro`.
`none` (the default) grants nothing. A share is exported to a node only when `grants` names it and the
node's `mounts` asks for it (its bucket `shares`, field `wants`); the access is the lower of the two.
The node's address comes from the controller's list of machines, rendered into the module's own file,
and must be inside the private network's range; never from what the node says.
- **What it refuses.** A path that does not exist or is not a directory, `/`, a folder owned by root
(uid or gid 0), and a folder that is, is inside, or holds within four levels a backup repository; a look
below that is cut short by its bound refuses rather than exports. A clean look is kept for an hour.
Every check runs on the real folder after any symbolic link. A refused share is said by the tools and
makes the module unhealthy; the others are exported.
- **Its file.** `/etc/exports.d/mesh.exports`, written by the module's own process, which runs as root
and looks every 30 s. One line per share: the path, to `${machine:mesh-range}` and nothing else, with
`rw` or `ro`, `sync`, `secure`, `no_subtree_check`, `all_squash`, and `anonuid` and `anongid` read from
the path's owner. `exportfs -ra` runs when the file changes or a share is missing from the kernel's
and looks every 30 s. One line per share: the real path (after any symbolic link), to the addresses of
the nodes the server grants it to and that ask for it, and to nobody else, with
`rw` or `ro`, `sync`, `secure`, `no_subtree_check`, `all_squash`, `anonuid` and `anongid` read from
the folder's owner, and `mp` (or `mp=<its filesystem's mount point>`), so nothing is served from an
unmounted dataset's empty folder. `exportfs -ra` runs when the file changes or a share is missing from the kernel's
table. The filesystem's own id is kept; none is forced.
- **Unassigned.** The process is stopped after its configuration is removed (the node-engine removes a
module's resources in reverse order); finding it gone, it removes its export file and has the kernel
drop the exports. A restart or a push keeps them.
- **The service.** `nfs-server.service`, running and enabled. NFS versions 2, 3 and 4.0 are turned off
in the module's drop-in in `/etc/nfs.conf.d`, so only 2049/tcp is needed, declared in `listens` with
`from: mesh`.
@@ -93,6 +105,13 @@ The glossary holds the three words.
`password=`, issued outside the mesh and given with `secret accept <node> mounts smb-credentials
--from <file>`. The process writes one credentials file per source, readable by root only. A source
with no section is said with that remedy and is not mounted.
- **What it refuses.** A mount point at, under or above any data the mesh keeps on this machine (the
controller's `data` verb, asked over the bus; until it is read nothing new is set up), a system path,
a mount point where something else is mounted or a folder that is not empty, and a share this machine
serves itself. It never stops a mount whose mounted filesystem is not its own source, and a setting
entry it cannot read removes nothing.
- **What it asks for.** Its bucket `shares` carries `wants`: each share this machine asks of which server,
read-write or read-only. The server decides; the mount takes the access the server gave this machine.
- **Reading the server.** The module reads the server's bucket `exports` (its manifest's `reads`) for
each share's path and access: its tools, which reach the bus, write what they read to a file of the
module's, and the root process takes only a name, an absolute path and the access from it. A share
@@ -147,22 +166,49 @@ Each says the share, the node and since when: *"The home server's share `library
from the workstation since 19:21."* An absent occasional source raises nothing; `mounts_list` says
*"the share `retro-games` is not present since 18:02"*.
## 5. Migration
## 5. The safe rollout
1. Build and deliver the two seats, `nfs-server` and `mounts`.
2. Assign `nfs-server` to the home server with the library and the downloads. Check `exports` and
`test` from the workstation.
3. `adopt` on the home server clears the library's `sharenfs` property. The underlay range leaves the
export table; the private network's range is in it.
4. Assign `mounts` to the workstation. Its unit pair for the library takes precedence over the unit
generated from fstab. Check `mounts_list` and a file written from the workstation.
5. `adopt` on the workstation comments out the library's fstab line, keeping a copy of the file.
6. Accept the gaming device's credential into the vault with `secret accept`, add the gaming device as an
occasional source, and `adopt` its fstab line.
7. Assign `mounts` to the laptop, read-only.
Read-only first, one node first, and every step checked before the next. Each step names how it is
undone. Every `adopt` is a person's act. Nothing here touches the library's files, owners or modes, or
the media modules' local bind mounts on the home server.
Every `adopt` is a person's act. Nothing in this migration touches the library's files, owners or
modes, or the media modules' bind mounts.
**Before anything:** the seats are merged and the controller that knows them is running; the two
modules' pull request is green against it. On the home server, `node-backup.backed-up` shows a good
night for Plex's own data, and `nfs-server`'s dry runs are read: `exports` lists only the intended
folders, none refused for a reason not understood.
1. **Assign `nfs-server` to the home server, read-only.** Its `shares` names the library only, and its
`grants` names the workstation read-only (`library=<workstation>:ro`). Check `exports`: one line, the
workstation's address only, `ro`, `all_squash` to the library's owner, `mp`. The ZFS property's own
export to the home network stays as it is (the packet filter already blocks it).
*Undo:* unassign `nfs-server`; its export file goes with it.
2. **Assign `mounts` to the workstation, read-only.** Its `shares` is `library=<mount point>:ro`. Check
`node-mounts.list`: the pair is armed, the share reads `armed` then `mounted` after a listing, and the
files are there. Its unit pair takes the place of the unit generated from fstab.
*Undo:* unassign `mounts`; its unit pairs go, and the fstab line is still there.
3. **Watch for a day.** `mounts_health` and `nfs_health` stay healthy; a reboot of the workstation
completes with the home server's share reachable and with it stopped (the share reads `unreachable`
and the boot does not wait).
4. **Read-write.** Change the grant to `rw` and the workstation's setting to drop `:ro`. Write one test
file from the workstation, check its owner on the home server is the library's owner, and remove it.
*Undo:* set both back to `ro`.
5. **Adopt the workstation's library line**: `node-mounts.adopt` dry run, read it, then with
`confirm` and a reason. *Undo:* the dated copy of fstab beside it.
6. **Adopt the ZFS property**: `node-nfs-server.adopt` dry run, then confirmed. The home network's
export goes; the mesh's stays. *Undo:* `zfs set sharenfs=<the old value>` by a person, recorded as a
hand-act.
7. **The downloads**, the same way: read-only first, then read-write.
8. **The gaming device**: the operator gives `smb-credentials` with `secret accept`; the source is added
to `sources`; `node-mounts.list` says it `absent` while the device is off, and it mounts on access
when it is on. Then its fstab line is adopted.
9. **The laptop**, read-only, granted read-only on the home server.
**Root.** Both modules' tools act as root through `sudo -n`, and the `sudo` module grants the operator
account every command, so the modules' own checks are the only line before root. Narrowing sudo is a
separate decision.
**Stop at once** when a share reads `failed` for a reason not understood, when a dry run would touch a
line or a dataset other than the one asked for, or when anything under the library changes owner.
## Not in this design