From 3edafd78ab3f1addef7b9ec01518bc1c42efb6ab Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 17:23:52 +0200 Subject: [PATCH 1/5] Research 038, ADR 0263 and to-be 52: let the mesh own shares and mounts, so no node depends on a hand-written export or fstab line --- .../00-overview.md | 78 +++++++++ .../01-evidence.md | 134 ++++++++++++++++ .../02-options.md | 150 ++++++++++++++++++ ...n-demand-and-only-nodes-that-ask-get-it.md | 136 ++++++++++++++++ 03-DESIGN/01-to-be/52-shares-and-mounts.md | 145 +++++++++++++++++ 03-DESIGN/01-to-be/README.md | 1 + 6 files changed, 644 insertions(+) create mode 100644 01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md create mode 100644 01-RESEARCH/038-shares-and-mounts-the-mesh-manages/01-evidence.md create mode 100644 01-RESEARCH/038-shares-and-mounts-the-mesh-manages/02-options.md create mode 100644 02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md create mode 100644 03-DESIGN/01-to-be/52-shares-and-mounts.md diff --git a/01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md b/01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md new file mode 100644 index 00000000..48025484 --- /dev/null +++ b/01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md @@ -0,0 +1,78 @@ +--- +status: active +initiated: 2026-10-08 +touches: + - 04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md + - 04-ISSUES/154-a-machines-own-network-is-not-a-reach/00-report.md + - 01-RESEARCH/027-the-system-layer-as-modules/02-candidates-and-questions.md + - 03-DESIGN/01-to-be/26-the-seats.md + - 03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md + - 02-DECISIONS/0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md + - 02-DECISIONS/0051-shared-data-is-the-operators.md + - 02-DECISIONS/0092-an-operator-delivers-a-pair-credential.md + - the catalogue's module manifests + - the controller's seat set +became: [] +--- + +# 038 — Shares and mounts the mesh manages + +## What is investigated + +How the mesh shares a directory from one node and mounts it on the others, so that no machine +carries a hand-written export or a hand-written mount line. Three kinds of source are in scope: + +1. **A node's share.** The home server shares its storage pool over NFS. A node that wants the files + mounts them. +2. **A device outside the mesh that is not always there.** A gaming device on the home network shares + a folder over SMB, and only while it is switched on. A USB disk is the same case. +3. **What the mesh should show and say** when a mount works, when it does not, and when its source is + simply away. + +## Why + +On 2026-10-08 the operator found two failed mounts on the workstation, and said two things: + +- about the media library: *"This should be handled by the mesh. If a node exposes an NFS share, it + should get configured on all the nodes in their fstab?"* +- about the gaming device's share: *"This mount only works if I plug in my recalbox machine."* + +They also set a requirement: **the home server shares its storage pool over NFS**, the workstation's +library mount comes from that share, the export reaches only the mesh's private network, it squashes +root, and the media modules' data (the Plex library is sacred) must not be put at risk. + +[Issue 169](../../04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md) +predicted the first failure on 2026-09-30: nothing declares the share's ports, so the packet filter +closes them. This effort measures what happened, and proposes the decision issue 169 asked for. + +## What was found + +The detail is in [01-evidence.md](01-evidence.md). In short: + +1. **The library mount times out because the home server's packet filter drops NFS.** The server + exports the library and its NFS service runs. The mesh's filter accepts only what a module + `listens` on, and no module listens on 2049, 111 or 445. The server's NFS counters read zero I/O + since its last boot six days earlier, so no client has used either share since then. +2. **The gaming device's mount fails because the gaming device is off.** It is an SMB share on the device itself, + not a disk. The fstab line has no `nofail` and no automount, so each boot records a failed unit. + The line also carries the share's password in a world-readable file. +3. **The export is not where a person would look.** The server's `/etc/exports` holds only comments. + The export comes from a property of the ZFS dataset, which writes its own file in `/etc/exports.d`. + The library is a child dataset of the pool, so an export of the pool's root does not show the + library's files unless it crosses into child filesystems. +4. **User ids do not match across nodes.** The library is owned by a service account and group. The + operator's account has the same id (1000) on every node, but on the home server that id belongs to + a different account, which owns the pool's backup repository. NFS with `sec=sys` trusts the client's + id. A plain export of the whole pool would give the workstation's operator account the backup + repository. +5. **No module, and until now no tool, covers any of this.** No module in either catalogue exports or + mounts anything. No mesh tool read a machine's fstab, its mount units or its exports, so this effort + read them through the login shell. A read-only `mounts` module now proposes those tools. + +## Where it goes + +[02-options.md](02-options.md) weighs the options. The proposed decision is +[ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md), +and the to-be design is +[52 — Shares and mounts](../../03-DESIGN/01-to-be/52-shares-and-mounts.md). Both wait for the +operator's word. The open questions are listed at the end of the options. diff --git a/01-RESEARCH/038-shares-and-mounts-the-mesh-manages/01-evidence.md b/01-RESEARCH/038-shares-and-mounts-the-mesh-manages/01-evidence.md new file mode 100644 index 00000000..fd7945b5 --- /dev/null +++ b/01-RESEARCH/038-shares-and-mounts-the-mesh-manages/01-evidence.md @@ -0,0 +1,134 @@ +# 038 — Evidence + +Measured on 2026-10-08 on the four nodes, read only, through the mesh's tools: `node-service-manager` +(`failed`, `status`, `journal`), `node-uplink.links`, `node-packet-filter.rules`, +`disk-load.disk_filesystems`, `docker.docker_list`, and `node-login-shell.execute` for the files no +tool read (§7). Addresses are given in documentation ranges: the home network (the underlay) as +`192.0.2.0/24`. Node roles: the **home server**, the **workstation**, the **laptop**, the **control-node**. + +## 1. The workstation's two failed mounts + +`node-service-manager.failed` on the workstation lists two failed system units, both generated from +`/etc/fstab` (their unit file is under `/run/systemd/generator`, and `mesh_declared` is false): + +| unit | the fstab line (password removed) | what the journal says | +|---|---|---| +| the library | `: nfs defaults,_netdev 0 0` | `Mounting timed out. Terminating.` 90 s after boot, at every boot since 2026-10-04; the last successful mount was at the boot of 2026-08-23 | +| the gaming device | `///Share cifs username=root,password=…,iocharset=utf8,sec=ntlmssp 0 0` | `mount error(113): could not connect to ` at every boot | + +Neither line has `nofail` or `x-systemd.automount`. Both name the source by its underlay address. +`remote-fs.target` is inactive on the workstation, because both of its members failed. + +## 2. Why the library times out + +- The home server exports the library, and its NFS and Samba services run. `nfs-server` and `smb` + are active and enabled, and ports 2049, 111 and 445 are listening on every address. +- **The mesh's packet filter drops them.** The mesh's table on the home server has `policy drop` on + its input chain for the underlay link and the private network link. It accepts a port only where a + module `listens` on it ([ADR 0045](../../02-DECISIONS/0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md)). + No rule names 2049, 111 or 445. +- From the workstation, a TCP connection to the home server's port 2049 fails, and one to port 22 on + the same address succeeds. +- The home server's NFS counters (`/proc/net/rpc/nfsd`) read `io 0 0` since its boot on 2026-10-04. + No established connection to 2049 or 445 exists. **Nobody has used either share since that boot.** + +This is the failure [issue 169](../../04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md) +predicted: the shares close silently once the filter is the sum of what modules listen on. + +## 3. What the home server exports, and from where + +- `/etc/exports` holds only the distribution's comments; it was last written on 2026-09-20. +- The export comes from ZFS. The library dataset has `sharenfs=rw=@192.0.2.0/24` and `sharesmb=on`. + ZFS writes `/etc/exports.d/zfs.exports` itself, marked *do not edit*. The kernel's export table + (`/var/lib/nfs/etab`) holds one entry: the library, to the underlay range, `rw,sync,root_squash, + no_all_squash,secure,no_subtree_check,sec=sys`. +- The Samba configuration has one share, `media`, on the library, `valid users = media`, + `force user = media`, writable. +- **The pool is two filesystems.** The pool is mounted at its root, and the library is a child dataset + mounted under it. An NFS export of the pool's root does not show the library's files unless the + export crosses into child filesystems (`crossmnt`); without it, a client sees an empty directory + where the library should be. + +## 4. What lives on the pool, and who uses it + +The pool's root holds five directories: + +| directory | owner | what it is | +|---|---|---| +| the library (child dataset, about 90 TiB used) | the media account and group, `0775` | movies, series, anime, music, audiobooks and more | +| downloads | the media account and group | the download clients' and the library managers' working area | +| archive | root | the operator's archive | +| a second Plex folder | the operator's account on the home server | an older library folder | +| the backup repository | the operator's account on the home server, `0700` | the restic repository the backup seat writes | + +Plex's own configuration and database live on a separate disk, not on the pool. + +Eight running containers bind paths on the pool, all of them as plain bind mounts of local paths: +Plex (the library's folders, all read-only), the library managers for series, movies, music and +subtitles, two download clients, and one automation app. **Exporting the pool over NFS changes none of +these paths and none of their owners.** An export adds a way in; it does not move or chown anything. +The only way an export endangers this data is by what it lets a client do (§5). + +## 5. User and group ids across the nodes + +| node | id 1000 | the media account | the media group | +|---|---|---|---| +| home server | a separate account, which owns the backup repository and the older Plex folder | exists, owns the library | exists | +| workstation | the operator | exists, same ids as on the server | exists; the operator is a member | +| laptop | the operator | absent | absent | +| control-node | the operator | absent | absent | + +The library managers on the home server run as the media account and group, set by their `puid` and +`pgid` settings. + +With `sec=sys`, the server believes the id a client sends; `root_squash` changes only root. So: + +- **A client's operator account (1000) is the home server's other account (1000).** An export of the + pool's root would let it read and write the backup repository and the older Plex folder. +- **A file written by the laptop's operator lands owned by 1000**, not by the media account, and the + library managers may then fail to rename or delete it. +- A device on the underlay that is not a node can claim any id. That is a reason to export to the + private network only, where every address is a node. + +## 6. The gaming device's share + +- The gaming device is a separate machine on the underlay, a retro-gaming device that shares a folder named + `Share` over SMB. It is not a USB disk. The router's DNS knows its name. It was switched off during + the survey and did not answer a ping. +- The workstation has `cifs-utils`, `avahi` and `nss-mdns`, and its `hosts:` line contains + `mdns4_minimal`. **The packet filter drops inbound mDNS** on every node, as the `avahi` module's + README already records. So discovery by mDNS cannot see the gaming device arrive today. Waiting for it + would first need a local-link reach for `listens`, which is the question of + [issue 154](../../04-ISSUES/154-a-machines-own-network-is-not-a-reach/00-report.md). +- The share's password is written in `/etc/fstab`, which is mode `0644`. Every account on the + workstation can read it. + +## 7. What the catalogue and the tools cover + +- **No module exports or mounts.** No manifest among the 117 modules of the core catalogue and the 12 + of the media catalogue writes an export, a Samba share, a mount unit or an fstab line. +- **`disk-load.disk_filesystems`** lists NFS and CIFS filesystems that are mounted now, with their space. + It cannot see a mount that failed, an fstab line, a mount unit or an export. +- **`node-service-manager`** shows a mount unit's state and journal, but not what it mounts or with + which options. +- **No tool read a machine's fstab, its mount and automount units with their sources, or its NFS and + SMB exports.** This effort read them with `node-login-shell.execute`, which runs a shell command as + the operator account. That is a work-around for a missing tool, so a read-only `mounts` module with + two tools, `mounts_list` and `mounts_exports`, was opened in the catalogue alongside this effort. It + redacts passwords in mount options and never touches a network mount point, so a hung server cannot + hang the tool. + +## 8. Prior work + +- **Issue 169** (open, the operator's proposal of 2026-09-30): two seats, `nfs-share` and `smb-share`, + because the two protocols share an intent and not a contract; a consumer module that writes a + systemd `.mount` unit from its binding; the modules that use the files never learn about NFS; the + data-loss case of a writer starting before its mount; and identity crossing the wire. +- **Research 027**, option M2: each client module writes its own systemd mount and automount unit, and + `/etc/fstab` stays the machine's. +- **Research 026**, the tools list: `exports`, `shares`, `clients`, `reload` on the server; + `mounts`, `mount`, `unmount`, `test` on the client. +- **ADR 0092**: a credential for something outside the mesh is an operator-delivered secret, accepted + into the vault (`secret accept`) and never replaced by a made one. +- **`${machine:mesh-range}`** already exists as a placeholder in two manifests, so an export line can + name the private network's range without naming a node. diff --git a/01-RESEARCH/038-shares-and-mounts-the-mesh-manages/02-options.md b/01-RESEARCH/038-shares-and-mounts-the-mesh-manages/02-options.md new file mode 100644 index 00000000..25294f03 --- /dev/null +++ b/01-RESEARCH/038-shares-and-mounts-the-mesh-manages/02-options.md @@ -0,0 +1,150 @@ +# 038 — Options + +Each question below lists the options weighed and the one recommended. The recommendations together +are [ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md). + +## Q1. Who writes the export + +| option | for | against | +|---|---|---| +| A. Keep the ZFS `sharenfs` property, and have a module declare only the ports | nothing new on disk | the export lives in a dataset property, beside the operator's data; a module would be setting properties on data the mesh does not own (ADR 0051); the options are one string with no room for squashing | +| B. A module owns `/etc/exports` whole | one file, one owner | the file is the machine's; a second exporter on the same node would collide | +| **C. A module, `nfs-server`, writes its own file in `/etc/exports.d`, and the ZFS property is turned off when the module's export is live** | the mesh owns exactly its file; the distribution reads the directory; unassigning removes the file | a migration step: the property must be cleared by an act, not by hand | + +**Recommended: C.** The module holds a node seat, `nfs-share`, so one module exports on a node, and it +provides the provision `nfs-share` (issue 169's proposal). Samba is left as found, with its own seat +`smb-share` later; nothing on the mesh uses it, and the gaming device's SMB share is a client case (Q6). + +## Q2. What is exported, and to whom + +The operator asked for the pool's root. Exporting it plainly has two problems (evidence §3 and §5): the +library is a child filesystem, so the export needs `crossmnt` to show it; and with `sec=sys`, any client +account with id 1000 becomes the server's own 1000, which owns the backup repository. + +| option | for | against | +|---|---|---| +| A. The pool's root, `rw,root_squash`, to the underlay range | what the operator literally asked; what exists today (but for the library only) | gives the backup repository to every client's operator account; open to every device on the underlay, which can claim any id | +| B. The pool's root, `rw,crossmnt,all_squash`, every client mapped to the media account, to the private network only | one share, as asked; every file written lands owned by the media account; the `0700` backup repository is unreadable to the media account, so it stays closed | every client writes as one account, so there is no per-person permission on the share; top-level folders other than the library are writable by every client | +| **C. One share per subtree: the library and the downloads, each `rw,all_squash` to the media account, to the private network only; the backup repository and the archive not exported** | the backup repository is not reachable at all; ownership is right; a consumer mounts only what it needs | two shares instead of one; a later folder needs a new share entry (one setting) | +| D. Kerberos (`sec=krb5`) with id mapping | real per-person identity | needs a KDC and keytabs on every node; far beyond the problem | + +**Recommended: C, with B as the operator's alternative if one share is wanted.** Both use +`all_squash` with `anonuid` and `anongid` set to the owner of the exported path, read on the server +(`${access::uid}`, from issue 153's fix), never written as numbers in a manifest. Both export to +`${machine:mesh-range}` only, so every client is a node, reached over the private network. `root_squash` +is implied by `all_squash`. `sync`, `secure`, `no_subtree_check` stay as they are today. The server +serves NFS version 4 only, so only port 2049 is opened, `from: mesh`; 111 and the mountd ports stay +closed. + +**What this does not touch:** the local bind mounts of Plex and the library managers (evidence §4), +the datasets' owners and modes, and the Samba share. + +## Q3. How a node mounts a share + +| option | for | against | +|---|---|---| +| A. The mesh writes lines into `/etc/fstab` | the file people know | one shared file and one ordered list; the node-engine can place a marked block but every line competes with the operator's | +| B. `autofs` with its own maps | mature automounter | a second daemon doing what systemd already does | +| **C. A `.mount` unit and an `.automount` unit per share, written by one module, `mounts`, which holds the node seat `node-mounts`** | a unit is a unit: its state, its journal, its health; an automount never holds the boot; unassigning removes it; no shared file | units have to be named by the escaped mount path, which the module computes | + +**Recommended: C.** The `.automount` is enabled under `remote-fs.target`; the `.mount` is not enabled +on its own, so nothing mounts at boot and nothing fails at boot. The mount is made at first access and +released after an idle time. The options carried: + +- `x-systemd.automount` semantics through the `.automount` unit, `TimeoutIdleSec` (default 10 minutes); +- a mount timeout, `TimeoutSec` on the `.mount` (default 30 s for a node's share, 10 s for an + occasional one), instead of systemd's 90 s; +- `_netdev` and an order after the private network's link is up; +- for NFS, `vers=4.2,hard,timeo=150,retrans=3` on a read-write share (no silent write loss) and + `soft` allowed for a read-only one, as a setting. + +An automount also closes issue 169's data-loss case: the mount point is always an autofs mount, so a +process that writes there either triggers the mount or gets an error. It never writes into the empty +directory underneath. + +## Q4. Which nodes get a share + +The operator suggested every node. Weighed: + +| option | for | against | +|---|---|---| +| A. Every node mounts every share | no assignment to make | the control-node is the stably reachable machine the controller runs on, and gains a writable path into the library for no use; the laptop, behind a corporate VPN client, would try a mount whenever a file manager looks at the path | +| B. A node mounts a share only when a module on it requires it | precise | a person who wants the files in their file manager has no module that requires them | +| **C. A node mounts what its `mounts` assignment lists; the list may be set once at the mesh-wide layer** | opt-in per node, and "every node" is one setting at the mesh-wide layer, with a node's own layer to leave a share out | two layers to read when asking why a node has a share; `mounts_list` says which layer gave it | + +**Recommended: C, assigned to the workstation and the laptop, not to the control-node.** A module that needs +the files (a player, an indexer) requires `nfs-share` as well; the controller then adds the share to +that node's `mounts` unit set, so B is the special case of C, not a second way. + +## Q5. Health + +| case | finding | condition | +|---|---|---| +| a node's share, automount armed, server answers | healthy | none | +| a node's share, server's 2049 unreachable from this node for 5 minutes | not healthy | `share-unreachable`, a warning: *"The home server's share `library` is not reachable from the workstation since 19:21."* | +| a node's share, the mount failed with an error other than a timeout | not healthy | the same condition, with the error | +| an occasional source, absent | **absent**, a state, not a finding | none | +| an occasional source, present but the mount fails | not healthy | `share-unreachable` | +| the server: the export table lacks a declared export, or `nfs-server` is down | not healthy | the module's ordinary health condition | + +The `mounts` module's code probes each source's port on a timer and keeps each share's state in its +own key-value bucket (ADR 0201): `mounted`, `armed`, `absent`, `unreachable`, with since when. Its +health reads that bucket. Nobody reads its files. + +## Q6. Occasional sources + +The gaming device is not a node, so it cannot hold a module or provide a provision. Weighed: + +| option | for | against | +|---|---|---| +| A. A fstab line with `nofail,x-systemd.automount` | small | a hand-written line again, and the password stays in the file | +| **B. An external source in the `mounts` assignment's settings: protocol, name, share, mount point, presence `occasional`, and the name of a credential** | the same unit pair and tools as a node's share; the password goes to the vault | a share with no provider, so no binding: the module checks the name itself | +| C. Wait for the device by mDNS and mount when it appears | mounts before anyone asks | the packet filter drops mDNS today (evidence §6); depends on issue 154 | + +**Recommended: B now, C when issue 154 gives `listens` a local-link reach.** With an automount, nothing +has to happen when the gaming device appears: the next access mounts it. The probe only keeps the state +honest, so `mounts_list` says *"the share `retro-games` is not present since 18:02"*. A USB disk is the same kind of source, +with presence taken from its device unit (`BindsTo=` on the disk's by-uuid device) instead of a probe. + +**The credential:** the share's password is an operator-delivered secret, accepted into the vault with +`secret accept` (ADR 0092), and reaches the node as a file the node-engine unseals (ADR 0086). The unit +names it as `credentials=`. Nothing in a repository or in `/etc/fstab` holds it. + +## Q7. Taking over the existing fstab lines + +The two lines are the operator's. The node-engine's own `.mount` unit in `/etc/systemd/system` already +takes precedence over the generated unit of the same name, but the line would still be read by every +other tool and person. + +**Recommended:** the `mounts` module gets an act, `adopt`, that for one mount point: + +1. checks a unit pair the module placed is armed for that mount point, and refuses if not; +2. comments out the fstab line, with a marker naming the module and the date, and keeps a copy of the + file beside it; +3. reloads the service manager and says what it did. + +It runs only on a person's word. The library line is adopted after the home server's share works over +the private network; the gaming device's after its credential is accepted. + +## Q8. Over the bus + +Every exchange between the modules is over NATS (research 036): + +- the controller wires provider and consumer through the provision; the binding carries `at` (the + server's private-network name), `path` and `options`; +- `nfs-server` keeps its exports and the clients it sees in its bucket, and answers `test` (does it + export this path to this node) by request and reply; +- `mounts` keeps each share's state in its bucket; a view or the controller watches it; +- no module reads another's files (ADR 0260). + +## Open questions for the operator + +1. **The pool's root, or one share per folder?** Recommended: one per folder (Q2 option C), so the + backup repository is never exported. +2. **Every node, or the nodes that ask?** Recommended: only the workstation and the laptop, with a + mesh-wide setting available (Q4). +3. **Underlay clients.** The private network only is recommended. Does anything on the home network + that is not a node (a TV, a media player) still need the NFS or SMB share? Today the filter closes + both, so nothing has been able to use them since 2026-10-04. +4. **Read-write from the laptop.** Should the laptop mount the library read-only? Recommended: yes, + read-only, since it travels. diff --git a/02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md b/02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md new file mode 100644 index 00000000..163d8344 --- /dev/null +++ b/02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md @@ -0,0 +1,136 @@ +--- +topic: what runs on it +status: proposed +date: 2026-10-08 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0051-shared-data-is-the-operators.md +--- + +# 263. A share is a provision, a node mounts it on demand through one module, and only the nodes that ask get it + +## Context + +On 2026-10-08 two mounts on the workstation had failed at every boot since 2026-10-04 +([research 038](../01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md), evidence §1–§2): + +- **The media library over NFS timed out** after 90 s. The home server exports it and its NFS service + runs, but the mesh's packet filter drops port 2049, because no module listens on it (ADR 0045). The + server's NFS counters read zero I/O since its boot six days earlier. This is the silent closure + [issue 169](../04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md) + predicted on 2026-09-30. +- **A gaming device's SMB share failed** with "host unreachable", because the gaming device was off. The + fstab line has no `nofail` and no automount, and it holds the share's password in a file every + account can read. + +The operator asked that the mesh handle this: a node that shares a directory should have it configured +on the nodes that use it. They also required that the home server share its storage pool over NFS, to +the private network only, with root squashed, and without risk to the media modules' data. + +What was measured bears on the shape (evidence §3–§5): + +- The export comes from a ZFS dataset property, not from `/etc/exports`. +- The library is a child filesystem of the pool, so an export of the pool's root shows an empty + directory there unless it crosses filesystems. +- With `sec=sys` the server believes the client's user id. The operator's account is id 1000 on every + node, and on the home server id 1000 is another account, which owns the backup repository. +- Eight containers bind paths on the pool locally. An export changes none of their paths or owners. +- No module exports or mounts anything, and no tool read fstab, mount units or exports. + +## Considered Options + +The full comparison is in research 038's [options](../01-RESEARCH/038-shares-and-mounts-the-mesh-manages/02-options.md). +Rejected: + +1. **Mesh-written lines in `/etc/fstab`.** One file shared with the operator, and one line per + mount with everything in an option string. Rejected for a unit per mount. +2. **`autofs` with its own maps.** A second daemon doing what systemd's automount already does. +3. **Keep the ZFS `sharenfs` property and only open the port.** It keeps the export inside a property + of the operator's data (ADR 0051), and it has no room for squashing every client to the library's + owner. +4. **Export to the home network (the underlay).** Every device there can claim any user id, and the + underlay has no reach in the mesh yet (issue 154). +5. **Kerberos (`sec=krb5`) for real identities.** It needs a KDC and keytabs on every node, for shares + that have one owner. +6. **Every node mounts every share.** The control-node is the stably reachable machine the controller runs on, and would gain a writable path + into the library for no use. The laptop, behind a VPN client, would try a mount whenever something + looks at the path. Kept as a setting, not as the default. +7. **One `file-share` seat for NFS and SMB.** The two share an intent, not a contract (issue 169). + +## Decision + +1. **A node's share is a provision.** The module `nfs-server` holds the node seat `nfs-share` and + provides the provision `nfs-share`. Its assignment's settings list the shares: a name, the path + (an `access`, ADR 0051: the module never creates, chowns or removes it), and read-only or + read-write. It writes its own file in `/etc/exports.d`, never `/etc/exports`, and listens on 2049/tcp + from the private network. It serves NFS version 4 only, so no other port opens. +2. **Every export squashes every client to the share's owner.** `all_squash`, with `anonuid` and + `anongid` read from the exported path's owner on the server (`${access::uid}`), never written as + numbers. The clients are `${machine:mesh-range}` and nothing else. A file written from any node lands + owned by the library's owner, and no client acts as any other account on the server. +3. **The home server exports one share per folder, not the pool's root**: the library and the + downloads. The backup repository and the archive are not exported. (If the operator wants the pool's + root as one share, it is exported with `crossmnt` and the same squashing, which keeps the `0700` + backup repository closed. That is the alternative this record names, and the operator's to choose.) +4. **A node mounts through one module.** The module `mounts` holds the node seat `node-mounts`. For each + share it writes a `.mount` unit and an `.automount` unit, never an fstab line. Only the automount is + enabled, so nothing mounts at boot and nothing can fail the boot. Each mount has an idle timeout + (default 10 minutes) and a mount timeout (default 30 s; 10 s for an occasional source). A read-write + NFS share is mounted `hard`. A mount point is always an autofs mount, so nothing writes into the + empty directory under it (issue 169's data-loss case). +5. **Only the nodes that ask get a share.** A node mounts what its `mounts` assignment lists, plus every + share a module on that node requires. The list may be set once at the mesh-wide settings layer, which + is how "every node" is said, and a node's own layer can leave a share out. The proposal is to assign + `mounts` to the workstation and the laptop, and not to the control-node. +6. **An occasional source is a share with presence `occasional`.** A device that is not a node, such as + the gaming device or a USB disk, is described in the `mounts` assignment's settings. Its absence is a + state, `absent`, not a fault. It is mounted on access when it is there, and it never fails the boot or + leaves a failed unit. Mounting when a device appears on the network waits for issue 154 (mDNS is + dropped by the packet filter today). +7. **A credential for a share is an operator-delivered secret** (ADR 0092), accepted into the vault and + unsealed on the node as a file (ADR 0086), which the mount unit names. No password is written in a + repository, a manifest or `/etc/fstab`. +8. **Health:** a share that should always be reachable and is not, for 5 minutes, raises one plain + condition, `share-unreachable`, naming the share, the node and since when. An occasional source that + is absent raises nothing. The server's health is its service and its export table holding every + declared export. +9. **Over the bus.** The controller wires the two through the provision. `nfs-server` keeps its exports + and clients, and `mounts` keeps each share's state, in their own key-value buckets (ADR 0201). They + ask each other by request and reply. Neither reads the other's files (ADR 0260). +10. **Taking over a hand-written line is an act.** The `mounts` module's `adopt` comments out one fstab + line, keeping a copy, only after its own unit pair for that mount point is armed, and only on a + person's word. The ZFS `sharenfs` property is cleared the same way, by `nfs-server`, once its own + export is live. + +## Consequences + +- The workstation's library comes back over the private network, and the two failed units stop + appearing at boot. +- Two node seats join the mesh's closed set, `nfs-share` and `node-mounts`. That is a controller + change. +- What got harder: every client writes to a share as one account, so there is no per-person + permission on it. Traffic to the library crosses the private network's encryption even on the home + network; the workstation is on Wi-Fi, so the tunnel is not the limit there, and this is measured at + build time. +- The underlay's own devices lose the NFS share the property gave them. They have had no access since + 2026-10-04 anyway, because the filter has closed it. Giving them access back is issue 154's question. +- Samba stays as found, and so does its closed port. An `smb-share` seat follows when something on the + mesh needs it. + +## How it is checked + +- The `mounts` module's `mounts_list` shows every mount point's source, its unit pair, its state and + which settings layer asked for it. `mounts_exports` shows the server's export table. A test in the + module renders the unit pair from a binding and asserts the options above. +- The lab composes a server and two clients and asserts: the export names only the private network's + range with `all_squash`; a file written from a client is owned by the share's owner; the client's + boot completes with the server stopped; and an occasional source that is absent leaves no failed unit. +- The controller's self-check counts fstab lines of type `nfs` or `cifs` on every node through + `mounts_list`, and reports any that are not adopted. + +## References + +- [Research 038](../01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md), its evidence and options +- [Issue 169](../04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md), [issue 154](../04-ISSUES/154-a-machines-own-network-is-not-a-reach/00-report.md), [issue 153](../04-ISSUES/153-an-adopted-machines-data-cannot-be-placed-where-it-is/00-report.md) +- [ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md), [ADR 0051](0051-shared-data-is-the-operators.md), [ADR 0086](0086-a-secret-reaches-a-process-as-a-file.md), [ADR 0092](0092-an-operator-delivers-a-pair-credential.md), [ADR 0201](0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md), [ADR 0240](0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md), [ADR 0260](0260-a-bar-block-shows-a-value-its-module-keeps-on-the-bus-and-no-module-reads-anothers-files.md) +- [52 — Shares and mounts](../03-DESIGN/01-to-be/52-shares-and-mounts.md) diff --git a/03-DESIGN/01-to-be/52-shares-and-mounts.md b/03-DESIGN/01-to-be/52-shares-and-mounts.md new file mode 100644 index 00000000..20d5f098 --- /dev/null +++ b/03-DESIGN/01-to-be/52-shares-and-mounts.md @@ -0,0 +1,145 @@ +--- +layer: to-be +status: proposed +code: [] +updated: 2026-10-08 +decisions: + - 02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md + - 02-DECISIONS/0051-shared-data-is-the-operators.md + - 02-DECISIONS/0092-an-operator-delivers-a-pair-credential.md +--- + +# 52 — Shares and mounts + +**A node that shares a directory does it through one module, which provides the share as a provision +and opens its port to the private network only. A node that wants the files gets them through one +module, which writes a mount unit and an automount unit per share, so nothing mounts at boot and +nothing fails at boot. Only the nodes that ask get a share. A device that is not always there is a +share whose absence is a state, not a fault.** +([ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md), +from [research 038](../../01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md).) + +A **share** is a directory one machine offers to others over the network. A **mount** is a share made +visible at a path on one node. An **occasional source** is a share on a device that comes and goes. +These three words are proposed for the glossary with this design. + +## The parts + +``` + home server workstation, laptop (the nodes that ask) + + nfs-server (seat nfs-share, node) mounts (seat node-mounts, node) + ├ /etc/exports.d/ ├ .automount enabled, idle 10 min + │ one line per share, to the ├ .mount not enabled, timeout 30 s + │ private network's range only, │ What = binding at:path, options + │ all_squash to the share's owner ├ credentials file (vault, ADR 0086), + ├ nfs-server.service, NFS v4 only │ for an SMB source only + ├ listens 2049/tcp from mesh ├ its process: probes each source, + └ its bucket: exports, clients │ keeps state in its bucket + ▲ └ tools: mounts_list, mounts_exports, + │ provision nfs-share mount, unmount, test, adopt + │ (the controller wires it; ▲ + │ binding: at, path, options) │ settings: shares wanted, occasional + └──────────────── request/reply on the bus ─────────┘ sources (mesh-wide and node layers) + + gaming device (not a node) ── SMB share ──► an occasional source in the workstation's mounts settings +``` + +| part | where | does | +|---|---|---| +| seats `nfs-share` and `node-mounts` | the controller's closed set | node seats, capacity 1, with their verbs | +| `nfs-server` | the catalogue | holds `nfs-share`, provides the provision, writes its export file, runs the service, opens 2049 to the private network | +| `mounts` | the catalogue | holds `node-mounts`, writes a unit pair per share, keeps each share's state, serves the tools | +| the binding | the controller | gives a consumer the server's private-network name, the path and the options | +| the credential | the vault | an operator-delivered secret per SMB source, unsealed on the node as a file | + +## 1. The server: `nfs-server` + +- **Settings.** A list of shares, each with a name, the path as an `access` (the operator's data; the + module never creates, chowns or removes it), and `read-write` or `read-only`. +- **Its file.** One file in `/etc/exports.d`, written by the node-engine. One line per share: + the path, to `${machine:mesh-range}`, with `rw` or `ro`, `sync`, `secure`, `no_subtree_check`, + `all_squash`, and `anonuid` and `anongid` taken from the path's owner on the server + (`${access::uid}` and `:gid`). The pool's root is exported only if the operator asks, and then + with `crossmnt` as well. +- **The service.** `nfs-server.service`, running and enabled, restarted when the file changes. + NFS version 4 only, set in the module's drop-in for `nfs.conf`. So only 2049/tcp is needed, declared in + `listens` with `from: mesh`. +- **The provision.** `nfs-share`, one per share, with `path` and `options` in what it serves. +- **Health.** The service is active, and the kernel's export table holds every line of the module's + file. +- **Its bucket.** The exports as the kernel holds them, and the clients that have each mounted, with + since when. +- **Tools.** `exports` (r), `clients` (r), `reload` (a), and `test` (r): does this server export this + path to this node. +- **Taking over.** An act, `adopt`, clears a dataset's `sharenfs` property for a path the module now + exports, on a person's word, and says what it did. + +## 2. The client: `mounts` + +- **Settings.** `shares`: the names of the `nfs-share` provisions this node mounts, each with a mount + point and optional `read-only`. `sources`: occasional sources, each with a protocol (`smb` or `disk`), + a name or a disk's UUID, a share name, a mount point, and the name of its credential. Both may be set + at the mesh-wide layer, which is how "every node" is said, and changed at a node's own layer. +- **A share a module requires.** When a module on the node requires `nfs-share`, the controller adds that + share to the node's `mounts` set, with the mount point that module's assignment gives. +- **The unit pair.** For each share, a `.mount` and an `.automount`, named by the escaped mount point and + written in `/etc/systemd/system` by the node-engine: + - the `.automount` is enabled under `remote-fs.target`, with `TimeoutIdleSec=10min`; + - the `.mount` is not enabled; it has `TimeoutSec=30s` (10 s for an occasional source), `_netdev`, + and an order after the private network's link; + - for NFS: `vers=4.2`, `hard,timeo=150,retrans=3` when read-write, `soft` allowed when read-only; + - for SMB: `credentials=`, `uid` and `gid` of the node's account that uses it; + - for a disk: `BindsTo=` the disk's device unit, so the mount goes with the disk. +- **States**, kept in its bucket per share, with since when: `armed` (automount waiting), `mounted`, + `absent` (an occasional source not there), `unreachable` (should be there and is not), `failed` + (the mount was tried and refused, with the error). +- **Its process.** Probes each source's port every 30 s, and the device unit for a disk. It never + touches a mount point, so a hung server cannot hang it. +- **Health.** Every automount is armed, and every share that is not occasional is `armed` or `mounted`. + An occasional source that is `absent` is healthy. +- **Tools.** `mounts_list` (r), `mounts_exports` (r), `test` (r), `mount` and `unmount` (a), `adopt` (a). + +## 3. Which nodes get which share + +| node | gets | why | +|---|---|---| +| home server | none; it holds `nfs-server` | the files are local; the media modules keep their local bind mounts | +| workstation | the library and the downloads, read-write; the gaming device as an occasional source | the operator works on the library there | +| laptop | the library, read-only | it travels; the automount means it tries only when something looks | +| control-node | none | the stably reachable machine the controller runs on; it has no use for the library | + +## 4. Conditions + +| condition | raised when | cleared when | +|---|---|---| +| `share-unreachable` (warning) | a share that is not occasional has been `unreachable` or `failed` for 5 minutes | the next probe reaches it | +| the server's health | the export table lacks a declared line, or the service is down | the next look finds both right | + +Each says the share, the node and since when: *"The home server's share `library` is not reachable +from the workstation since 19:21."* An absent occasional source raises nothing; `mounts_list` says +*"the share `retro-games` is not present since 18:02"*. + +## 5. Migration + +1. Build and deliver `mounts` (read-only first, already proposed) and `nfs-server`, and add the two seats. +2. Assign `nfs-server` to the home server with the library and the downloads. Check `exports` and + `test` from the workstation. +3. `adopt` on the home server clears the library's `sharenfs` property. The underlay range leaves the + export table; the private network's range is in it. +4. Assign `mounts` to the workstation. Its unit pair for the library takes precedence over the unit + generated from fstab. Check `mounts_list` and a file written from the workstation. +5. `adopt` on the workstation comments out the library's fstab line, keeping a copy of the file. +6. Accept the gaming device's credential into the vault with `secret accept`, add the gaming device as an + occasional source, and `adopt` its fstab line. +7. Assign `mounts` to the laptop, read-only. + +Every `adopt` is a person's act. Nothing in this migration touches the library's files, owners or +modes, or the media modules' bind mounts. + +## Not in this design + +- **Samba.** Kept as found. An `smb-share` seat follows when a node needs it. +- **Clients on the underlay.** NFS for devices that are not nodes waits on issue 154. +- **Mounting when a device appears.** Waits on a local-link reach (issue 154), because the packet + filter drops mDNS. The automount already mounts on the next access. diff --git a/03-DESIGN/01-to-be/README.md b/03-DESIGN/01-to-be/README.md index 215b5137..47a585da 100644 --- a/03-DESIGN/01-to-be/README.md +++ b/03-DESIGN/01-to-be/README.md @@ -51,6 +51,7 @@ document is written and this one's status becomes `implemented`. | [`49-the-mesh-in-domains.md`](49-the-mesh-in-domains.md) | **In progress.** Every concept belongs to one of ten domains, which owns its one word; the glossary is organised by them and is the authority, a retired word is named on its replacement's line with its scope, and two checks hold this repository's documents and the catalogue's tool descriptions to it | [ADR 0244](../../02-DECISIONS/0244-the-mesh-is-described-in-domains-and-one-word-names-one-thing.md), [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md), [ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md) | | [`50-split-dns-on-a-machine-with-a-vpn-client.md`](50-split-dns-on-a-machine-with-a-vpn-client.md) | **In progress.** A machine whose VPN client pushes servers of its own runs a resolver of its own, on a node seat, that routes the VPN's domains over its link and every other name to the mesh's resolvers; the VPN client's module carries the adapter | [ADR 0247](../../02-DECISIONS/0247-a-machine-with-a-vpn-client-routes-names-by-domain-through-a-resolver-of-its-own.md), [ADR 0223](../../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md) | | [`51-the-registries-keep-what-is-named.md`](51-the-registries-keep-what-is-named.md) | **In progress.** The artifact store, the package registry and each machine's container runtime get tools that say what they hold, what names each thing and what could go; every removal is a dry run unless a person asks and says why, and what decides is always a record | [ADR 0251](../../02-DECISIONS/0251-the-registries-say-what-they-hold-and-keep-what-is-named-and-a-machine-keeps-the-image-it-runs-and-the-one-before.md), [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md) | +| [`52-shares-and-mounts.md`](52-shares-and-mounts.md) | **Proposed.** A node shares a directory through `nfs-server`, which provides it as a provision to the private network only, squashing every client to the share's owner; a node that asks mounts it through `mounts`, a mount unit and an automount per share, so nothing mounts or fails at boot; a device that comes and goes is an occasional source whose absence is a state | [ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md), [ADR 0051](../../02-DECISIONS/0051-shared-data-is-the-operators.md), [ADR 0092](../../02-DECISIONS/0092-an-operator-delivers-a-pair-credential.md) | ## Not yet written From 34899385ec9e467c71572078c5bc29f4e7fde0f7 Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 18:23:23 +0200 Subject: [PATCH 2/5] Accept ADR 0263 with the operator's choices: one share per folder so backups are never exported, mesh machines only --- 00-META/glossary.md | 9 +- .../00-overview.md | 16 ++- .../02-options.md | 18 +-- ...n-demand-and-only-nodes-that-ask-get-it.md | 60 +++++---- 03-DESIGN/01-to-be/52-shares-and-mounts.md | 126 ++++++++++-------- 03-DESIGN/01-to-be/README.md | 2 +- .../00-report.md | 9 +- 7 files changed, 142 insertions(+), 98 deletions(-) diff --git a/00-META/glossary.md b/00-META/glossary.md index e177d8f2..bd9eae52 100644 --- a/00-META/glossary.md +++ b/00-META/glossary.md @@ -219,7 +219,7 @@ and Health and repair. **Decided by** ADR 0100, 0126, 0176, 0181, 0207, 0221, 02 **Purpose.** To resolve which module serves a provision for which consumer, and to wire the two. **Recorded by** the controller. **Upstream of** Data; downstream of Module, Placement and Identity. -**Decided by** ADR 0027, 0084, 0138, 0225, 0230. +**Decided by** ADR 0027, 0084, 0138, 0225, 0230, 0263. **Uses:** module, seat, credential (Identity and access). - **provision** — a service one module `provides` and others `require`; the mesh resolves a provider @@ -234,6 +234,13 @@ and Health and repair. **Decided by** ADR 0100, 0126, 0176, 0181, 0207, 0221, 02 reaches ([ADR 0138](../02-DECISIONS/0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md)). - **retire** — a provider stops serving a consumer the mesh no longer asks for; what it kept is deleted only by a person ([ADR 0230](../02-DECISIONS/0230-a-consumer-the-mesh-stops-asking-for-is-retired-and-deleted-only-by-a-person.md)). +- **share** — a folder one node offers to the mesh's other nodes over the private network: the provision + `nfs-share`, provided by the holder of the node seat `node-nfs-server`, one share per folder + ([ADR 0263](../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md)). +- **mount** — a share, or an occasional source, made visible at a path on one node by the holder of the + node seat `node-mounts`, as a mount unit and an automount unit, mounted on access (ADR 0263). +- **occasional source** — a share on a device that is not a node and comes and goes, such as a USB disk; + its absence is a state, *not present*, never a fault (ADR 0263). ## Identity and access — who may do what diff --git a/01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md b/01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md index 48025484..0298b153 100644 --- a/01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md +++ b/01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md @@ -1,5 +1,5 @@ --- -status: active +status: graduated initiated: 2026-10-08 touches: - 04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md @@ -12,7 +12,9 @@ touches: - 02-DECISIONS/0092-an-operator-delivers-a-pair-credential.md - the catalogue's module manifests - the controller's seat set -became: [] +became: + - 03-DESIGN/01-to-be/52-shares-and-mounts.md + - 02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md --- # 038 — Shares and mounts the mesh manages @@ -71,8 +73,10 @@ The detail is in [01-evidence.md](01-evidence.md). In short: ## Where it goes -[02-options.md](02-options.md) weighs the options. The proposed decision is +[02-options.md](02-options.md) weighs the options. The decision is [ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md), -and the to-be design is -[52 — Shares and mounts](../../03-DESIGN/01-to-be/52-shares-and-mounts.md). Both wait for the -operator's word. The open questions are listed at the end of the options. +and the design is [52 — Shares and mounts](../../03-DESIGN/01-to-be/52-shares-and-mounts.md). + +On 2026-10-08 the operator chose **one share per folder**, so the backups are never exported, and +**mesh machines only**, over the private network. Which nodes mount a share, and whether the laptop +mounts read-only, is decided at assignment. diff --git a/01-RESEARCH/038-shares-and-mounts-the-mesh-manages/02-options.md b/01-RESEARCH/038-shares-and-mounts-the-mesh-manages/02-options.md index 25294f03..df6fb9c5 100644 --- a/01-RESEARCH/038-shares-and-mounts-the-mesh-manages/02-options.md +++ b/01-RESEARCH/038-shares-and-mounts-the-mesh-manages/02-options.md @@ -1,7 +1,7 @@ # 038 — Options Each question below lists the options weighed and the one recommended. The recommendations together -are [ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md). +became [ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md). ## Q1. Who writes the export @@ -137,14 +137,10 @@ Every exchange between the modules is over NATS (research 036): - `mounts` keeps each share's state in its bucket; a view or the controller watches it; - no module reads another's files (ADR 0260). -## Open questions for the operator +## The operator's answers (2026-10-08) -1. **The pool's root, or one share per folder?** Recommended: one per folder (Q2 option C), so the - backup repository is never exported. -2. **Every node, or the nodes that ask?** Recommended: only the workstation and the laptop, with a - mesh-wide setting available (Q4). -3. **Underlay clients.** The private network only is recommended. Does anything on the home network - that is not a node (a TV, a media player) still need the NFS or SMB share? Today the filter closes - both, so nothing has been able to use them since 2026-10-04. -4. **Read-write from the laptop.** Should the laptop mount the library read-only? Recommended: yes, - read-only, since it travels. +1. **The pool's root, or one share per folder?** One share per folder (Q2 option C), so the backup + repository is never exported. +2. **Underlay clients?** Mesh machines only, over the private network. +3. **Every node, or the nodes that ask? Read-write from the laptop?** Decided at assignment, which is the + operator's; the recommendation (Q4 option C; the laptop read-only) stands in the design. diff --git a/02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md b/02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md index 163d8344..e4d9298f 100644 --- a/02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md +++ b/02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md @@ -1,6 +1,6 @@ --- topic: what runs on it -status: proposed +status: accepted date: 2026-10-08 deciders: jochen reconstructed: false @@ -56,47 +56,61 @@ Rejected: into the library for no use. The laptop, behind a VPN client, would try a mount whenever something looks at the path. Kept as a setting, not as the default. 7. **One `file-share` seat for NFS and SMB.** The two share an intent, not a contract (issue 169). +8. **The pool's root as one share**, with `crossmnt` and every client mapped to the library's owner. That + keeps the `0700` backup repository closed to clients, but it still exports it, and every other folder + at the root becomes writable from every node. The operator chose one share per folder on 2026-10-08, + so the backups are never exported. +9. **A provider's paths written into its binding.** A binding carries what the provider's manifest + `serves`, which is the same for every machine; a share's path is a setting of one assignment. The + consumer asks the server for it over the bus instead, so no controller change is needed for it. ## Decision -1. **A node's share is a provision.** The module `nfs-server` holds the node seat `nfs-share` and - provides the provision `nfs-share`. Its assignment's settings list the shares: a name, the path - (an `access`, ADR 0051: the module never creates, chowns or removes it), and read-only or - read-write. It writes its own file in `/etc/exports.d`, never `/etc/exports`, and listens on 2049/tcp +1. **A node's share is a provision.** The module `nfs-server` holds the node seat `node-nfs-server` and + provides the provision `nfs-share`. Its assignment's setting `shares` lists the shares: a name, the + path, and read-only or read-write. The path is the operator's data (ADR 0051): the module never + creates, chowns, chmods or removes anything under it. It writes its own file in `/etc/exports.d`, never `/etc/exports`, and listens on 2049/tcp from the private network. It serves NFS version 4 only, so no other port opens. 2. **Every export squashes every client to the share's owner.** `all_squash`, with `anonuid` and - `anongid` read from the exported path's owner on the server (`${access::uid}`), never written as + `anongid` read by the module from the exported path's owner on the server, never written as numbers. The clients are `${machine:mesh-range}` and nothing else. A file written from any node lands owned by the library's owner, and no client acts as any other account on the server. -3. **The home server exports one share per folder, not the pool's root**: the library and the - downloads. The backup repository and the archive are not exported. (If the operator wants the pool's - root as one share, it is exported with `crossmnt` and the same squashing, which keeps the `0700` - backup repository closed. That is the alternative this record names, and the operator's to choose.) +3. **One share per folder, never a pool's root** (the operator's choice, 2026-10-08): on the home server, + the library and the downloads. The backup repository and the archive are not exported. The module + refuses a share that is `/` or that holds a backup repository at its path or one level below, so a + setting cannot export one by mistake. 4. **A node mounts through one module.** The module `mounts` holds the node seat `node-mounts`. For each share it writes a `.mount` unit and an `.automount` unit, never an fstab line. Only the automount is enabled, so nothing mounts at boot and nothing can fail the boot. Each mount has an idle timeout (default 10 minutes) and a mount timeout (default 30 s; 10 s for an occasional source). A read-write NFS share is mounted `hard`. A mount point is always an autofs mount, so nothing writes into the empty directory under it (issue 169's data-loss case). -5. **Only the nodes that ask get a share.** A node mounts what its `mounts` assignment lists, plus every - share a module on that node requires. The list may be set once at the mesh-wide settings layer, which - is how "every node" is said, and a node's own layer can leave a share out. The proposal is to assign - `mounts` to the workstation and the laptop, and not to the control-node. +5. **Only the nodes that ask get a share.** A node mounts what its `mounts` assignment's setting `shares` + lists. The list may be set once at the mesh-wide settings layer, which is how "every node" is said, + and a node's own layer can leave a share out. Which nodes are assigned, and whether the laptop mounts + read-only, is decided at assignment, which is the operator's; the recommendation is the workstation + and the laptop (read-only), not the control-node. 6. **An occasional source is a share with presence `occasional`.** A device that is not a node, such as - the gaming device or a USB disk, is described in the `mounts` assignment's settings. Its absence is a + the gaming device or a USB disk, is described in the `mounts` assignment's setting `sources`. Its absence is a state, `absent`, not a fault. It is mounted on access when it is there, and it never fails the boot or leaves a failed unit. Mounting when a device appears on the network waits for issue 154 (mDNS is dropped by the packet filter today). -7. **A credential for a share is an operator-delivered secret** (ADR 0092), accepted into the vault and - unsealed on the node as a file (ADR 0086), which the mount unit names. No password is written in a +7. **A credential for a share is an operator-delivered secret** (ADR 0092): the `mounts` module's own + secret `smb-credentials`, issued outside the mesh (ADR 0228), given with `secret accept`, kept by the + vault and unsealed on the node as a file (ADR 0086). The module writes one credentials file per source + from it, readable by root only, which the mount unit names. No password is written in a repository, a manifest or `/etc/fstab`. 8. **Health:** a share that should always be reachable and is not, for 5 minutes, raises one plain - condition, `share-unreachable`, naming the share, the node and since when. An occasional source that + condition, naming the share, the node and since when. It is the `mounts` module's health (ADR 0240), + judged by its tool `mounts_health`, whose finding is *share unreachable*; no new kind of condition is + added to the controller. An occasional source that is absent raises nothing. The server's health is its service and its export table holding every declared export. -9. **Over the bus.** The controller wires the two through the provision. `nfs-server` keeps its exports - and clients, and `mounts` keeps each share's state, in their own key-value buckets (ADR 0201). They - ask each other by request and reply. Neither reads the other's files (ADR 0260). +9. **Over the bus.** The controller wires the two through the provision: the binding gives the consumer + the server's name. The consumer then asks the server's seat, `node-nfs-server.exports`, by request and + reply for each share's path and access, so a share's path is said once, on the server. `nfs-server` + keeps its exports, and `mounts` keeps each share's state, in their own key-value buckets (ADR 0201). + Neither reads the other's files (ADR 0260). 10. **Taking over a hand-written line is an act.** The `mounts` module's `adopt` comments out one fstab line, keeping a copy, only after its own unit pair for that mount point is armed, and only on a person's word. The ZFS `sharenfs` property is cleared the same way, by `nfs-server`, once its own @@ -106,8 +120,8 @@ Rejected: - The workstation's library comes back over the private network, and the two failed units stop appearing at boot. -- Two node seats join the mesh's closed set, `nfs-share` and `node-mounts`. That is a controller - change. +- Two node seats join the mesh's closed set, `node-nfs-server` and `node-mounts`. That is a controller + change. The provision `nfs-share` needs none. - What got harder: every client writes to a share as one account, so there is no per-person permission on it. Traffic to the library crosses the private network's encryption even on the home network; the workstation is on Wi-Fi, so the tunnel is not the limit there, and this is measured at diff --git a/03-DESIGN/01-to-be/52-shares-and-mounts.md b/03-DESIGN/01-to-be/52-shares-and-mounts.md index 20d5f098..6251e2a3 100644 --- a/03-DESIGN/01-to-be/52-shares-and-mounts.md +++ b/03-DESIGN/01-to-be/52-shares-and-mounts.md @@ -1,7 +1,10 @@ --- layer: to-be -status: proposed -code: [] +status: in-progress +code: + - mesh-controller internal/catalogue (the seats node-nfs-server and node-mounts) + - mesh-catalog modules/nfs-server + - mesh-catalog modules/mounts updated: 2026-10-08 decisions: - 02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md @@ -21,17 +24,17 @@ from [research 038](../../01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00- A **share** is a directory one machine offers to others over the network. A **mount** is a share made visible at a path on one node. An **occasional source** is a share on a device that comes and goes. -These three words are proposed for the glossary with this design. +The glossary holds the three words. ## The parts ``` home server workstation, laptop (the nodes that ask) - nfs-server (seat nfs-share, node) mounts (seat node-mounts, node) + nfs-server (seat node-nfs-server) mounts (seat node-mounts, node) ├ /etc/exports.d/ ├ .automount enabled, idle 10 min │ one line per share, to the ├ .mount not enabled, timeout 30 s - │ private network's range only, │ What = binding at:path, options + │ private network's range only, │ What = :, options │ all_squash to the share's owner ├ credentials file (vault, ADR 0086), ├ nfs-server.service, NFS v4 only │ for an SMB source only ├ listens 2049/tcp from mesh ├ its process: probes each source, @@ -39,7 +42,7 @@ These three words are proposed for the glossary with this design. ▲ └ tools: mounts_list, mounts_exports, │ provision nfs-share mount, unmount, test, adopt │ (the controller wires it; ▲ - │ binding: at, path, options) │ settings: shares wanted, occasional + │ binding: the server's name) │ settings: shares wanted, occasional └──────────────── request/reply on the bus ─────────┘ sources (mesh-wide and node layers) gaming device (not a node) ── SMB share ──► an occasional source in the workstation's mounts settings @@ -47,61 +50,73 @@ These three words are proposed for the glossary with this design. | part | where | does | |---|---|---| -| seats `nfs-share` and `node-mounts` | the controller's closed set | node seats, capacity 1, with their verbs | -| `nfs-server` | the catalogue | holds `nfs-share`, provides the provision, writes its export file, runs the service, opens 2049 to the private network | +| seats `node-nfs-server` and `node-mounts` | the controller's closed set | node seats, capacity 1, with their verbs | +| `nfs-server` | the catalogue | holds `node-nfs-server`, provides the provision, writes its export file, runs the service, opens 2049 to the private network | | `mounts` | the catalogue | holds `node-mounts`, writes a unit pair per share, keeps each share's state, serves the tools | -| the binding | the controller | gives a consumer the server's private-network name, the path and the options | -| the credential | the vault | an operator-delivered secret per SMB source, unsealed on the node as a file | +| the binding | the controller | gives a consumer the server's name; the consumer asks the server's seat for each share's path and access over the bus | +| the credential | the vault | the `mounts` module's own secret `smb-credentials`, issued outside the mesh and given with `secret accept`, unsealed on the node as a file | ## 1. The server: `nfs-server` -- **Settings.** A list of shares, each with a name, the path as an `access` (the operator's data; the - module never creates, chowns or removes it), and `read-write` or `read-only`. -- **Its file.** One file in `/etc/exports.d`, written by the node-engine. One line per share: - the path, to `${machine:mesh-range}`, with `rw` or `ro`, `sync`, `secure`, `no_subtree_check`, - `all_squash`, and `anonuid` and `anongid` taken from the path's owner on the server - (`${access::uid}` and `:gid`). The pool's root is exported only if the operator asks, and then - with `crossmnt` as well. -- **The service.** `nfs-server.service`, running and enabled, restarted when the file changes. - NFS version 4 only, set in the module's drop-in for `nfs.conf`. So only 2049/tcp is needed, declared in - `listens` with `from: mesh`. -- **The provision.** `nfs-share`, one per share, with `path` and `options` in what it serves. -- **Health.** The service is active, and the kernel's export table holds every line of the module's - file. -- **Its bucket.** The exports as the kernel holds them, and the clients that have each mounted, with - since when. -- **Tools.** `exports` (r), `clients` (r), `reload` (a), and `test` (r): does this server export this - path to this node. -- **Taking over.** An act, `adopt`, clears a dataset's `sharenfs` property for a path the module now - exports, on a person's word, and says what it did. +- **Its setting `shares`.** A list of `name=path`, each optionally `:ro`, separated by spaces: on the + home server the library and the downloads. A share's path is the operator's data (ADR 0051); the + module never creates, chowns, chmods or removes anything under it. +- **What it refuses.** A path that does not exist or is not a directory, `/`, and a path that holds a + backup repository at its root or one level below. A refused share is said by the tools and makes the + module unhealthy; the others are exported. +- **Its file.** `/etc/exports.d/mesh.exports`, written by the module's own process, which runs as root. + One line per share: the path, to `${machine:mesh-range}` and nothing else, with `rw` or `ro`, `sync`, + `secure`, `no_subtree_check`, `all_squash`, a stable `fsid`, and `anonuid` and `anongid` read from the + path's owner. `exportfs -ra` runs when the file changes. +- **The service.** `nfs-server.service`, running and enabled. NFS version 3 is turned off in the + module's drop-in for `nfs.conf`, so only 2049/tcp is needed, declared in `listens` with `from: mesh`. +- **The provision.** `nfs-share`, at the mesh's scope, with no identity: the binding gives the + consumer the server's name, and the consumer asks for the rest. +- **Health.** Its tool `nfs_health`: the service is active, every share is in the kernel's export table, + and no share was refused. +- **Its bucket.** `exports`, one entry per machine: the shares as the kernel holds them. +- **Its seat's verbs.** `exports` (r), `clients` (r), `test` (r), `reload` (a) and `adopt` (a). `adopt` + clears a ZFS dataset's `sharenfs` property for a path the module now exports, only once the mesh's + export of it is in the kernel's table, and only with a person's confirmation and reason. ## 2. The client: `mounts` -- **Settings.** `shares`: the names of the `nfs-share` provisions this node mounts, each with a mount - point and optional `read-only`. `sources`: occasional sources, each with a protocol (`smb` or `disk`), - a name or a disk's UUID, a share name, a mount point, and the name of its credential. Both may be set - at the mesh-wide layer, which is how "every node" is said, and changed at a node's own layer. -- **A share a module requires.** When a module on the node requires `nfs-share`, the controller adds that - share to the node's `mounts` set, with the mount point that module's assignment gives. -- **The unit pair.** For each share, a `.mount` and an `.automount`, named by the escaped mount point and - written in `/etc/systemd/system` by the node-engine: - - the `.automount` is enabled under `remote-fs.target`, with `TimeoutIdleSec=10min`; - - the `.mount` is not enabled; it has `TimeoutSec=30s` (10 s for an occasional source), `_netdev`, - and an order after the private network's link; - - for NFS: `vers=4.2`, `hard,timeo=150,retrans=3` when read-write, `soft` allowed when read-only; - - for SMB: `credentials=`, `uid` and `gid` of the node's account that uses it; - - for a disk: `BindsTo=` the disk's device unit, so the mount goes with the disk. -- **States**, kept in its bucket per share, with since when: `armed` (automount waiting), `mounted`, - `absent` (an occasional source not there), `unreachable` (should be there and is not), `failed` - (the mount was tried and refused, with the error). -- **Its process.** Probes each source's port every 30 s, and the device unit for a disk. It never - touches a mount point, so a hung server cannot hang it. -- **Health.** Every automount is armed, and every share that is not occasional is `armed` or `mounted`. - An occasional source that is `absent` is healthy. -- **Tools.** `mounts_list` (r), `mounts_exports` (r), `test` (r), `mount` and `unmount` (a), `adopt` (a). +- **Its setting `shares`.** A list of `name=mountpoint`, each optionally `:ro`: the `nfs-share` shares + this node mounts, by the server's name for them. +- **Its setting `sources`.** Occasional sources: `name=smb:///@` or + `name=disk:@`, each optionally `:ro`. +- **Its secret `smb-credentials`.** One section per SMB source, `[name]` with `username=` and + `password=`, issued outside the mesh and given with `secret accept mounts smb-credentials + --from `. The process writes one credentials file per source, readable by root only. A source + with no section is said with that remedy and is not mounted. +- **Asking the server.** The process asks `node-nfs-server.exports` on the server named by the binding, + over the bus, for each share's path and access. A share the server does not export is said, not + guessed. +- **The unit pair.** For each share and source, a `.mount` and an `.automount`, named by the escaped + mount point, written in `/etc/systemd/system` by the module's own process with a marker line. It never + touches a unit without its marker, and it removes its own pairs that are no longer asked for: + - the `.automount` is enabled under `remote-fs.target`, with an idle time of 10 minutes; + - the `.mount` is not enabled; its mount timeout is 30 s, 10 s for an occasional source; it starts + after the network is online; + - NFS: version 4.2, `hard,timeo=150,retrans=3` when read-write; `ro,soft` when read-only; + - SMB: `credentials=`, `iocharset=utf8`, `nofail`; + - a disk: bound to the disk's device unit, so the mount goes with the disk. +- **States**, per share, with since when, in its bucket `shares` (one entry per machine): `armed`, + `mounted`, `absent` (an occasional source not there), `unreachable` (should be there and is not), + `failed` (the mount was tried and refused, with the error). +- **Its probes.** Every 30 s, a connection to the server's port (2049, or 445 for SMB) with a short + timeout, and the device for a disk. It never touches a mount point, so a hung server cannot hang it. +- **Health.** Its tool `mounts_health`: every automount is armed, and no share that is not occasional + has been `unreachable` or `failed` for 5 minutes. An absent occasional source is healthy and said as + *not present since 18:02*. +- **Its seat's verbs.** `list` (r), `test` (r), `mount` (a), `unmount` (a), `adopt` (a); its tools + `mounts_list` and `mounts_exports` (r) stay. ## 3. Which nodes get which share +Which nodes are assigned `mounts`, and with which settings, is decided at assignment, which is the +operator's. The recommendation: + | node | gets | why | |---|---|---| | home server | none; it holds `nfs-server` | the files are local; the media modules keep their local bind mounts | @@ -113,8 +128,11 @@ These three words are proposed for the glossary with this design. | condition | raised when | cleared when | |---|---|---| -| `share-unreachable` (warning) | a share that is not occasional has been `unreachable` or `failed` for 5 minutes | the next probe reaches it | -| the server's health | the export table lacks a declared line, or the service is down | the next look finds both right | +| the `mounts` module's health, finding *share unreachable* | a share that is not occasional has been `unreachable` or `failed` for 5 minutes, or an automount is not armed | the next look finds it reached and armed | +| the `nfs-server` module's health | a share is missing from the kernel's export table, a share was refused, or the service is down | the next look finds all three right | + +Both are the module's health under ADR 0240, judged by its tool; the controller raises and clears the +condition as for any module, and adds no kind of its own. Each says the share, the node and since when: *"The home server's share `library` is not reachable from the workstation since 19:21."* An absent occasional source raises nothing; `mounts_list` says @@ -122,7 +140,7 @@ from the workstation since 19:21."* An absent occasional source raises nothing; ## 5. Migration -1. Build and deliver `mounts` (read-only first, already proposed) and `nfs-server`, and add the two seats. +1. Build and deliver the two seats, `nfs-server` and `mounts`. 2. Assign `nfs-server` to the home server with the library and the downloads. Check `exports` and `test` from the workstation. 3. `adopt` on the home server clears the library's `sharenfs` property. The underlay range leaves the diff --git a/03-DESIGN/01-to-be/README.md b/03-DESIGN/01-to-be/README.md index 47a585da..0ad621dc 100644 --- a/03-DESIGN/01-to-be/README.md +++ b/03-DESIGN/01-to-be/README.md @@ -51,7 +51,7 @@ document is written and this one's status becomes `implemented`. | [`49-the-mesh-in-domains.md`](49-the-mesh-in-domains.md) | **In progress.** Every concept belongs to one of ten domains, which owns its one word; the glossary is organised by them and is the authority, a retired word is named on its replacement's line with its scope, and two checks hold this repository's documents and the catalogue's tool descriptions to it | [ADR 0244](../../02-DECISIONS/0244-the-mesh-is-described-in-domains-and-one-word-names-one-thing.md), [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md), [ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md) | | [`50-split-dns-on-a-machine-with-a-vpn-client.md`](50-split-dns-on-a-machine-with-a-vpn-client.md) | **In progress.** A machine whose VPN client pushes servers of its own runs a resolver of its own, on a node seat, that routes the VPN's domains over its link and every other name to the mesh's resolvers; the VPN client's module carries the adapter | [ADR 0247](../../02-DECISIONS/0247-a-machine-with-a-vpn-client-routes-names-by-domain-through-a-resolver-of-its-own.md), [ADR 0223](../../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md) | | [`51-the-registries-keep-what-is-named.md`](51-the-registries-keep-what-is-named.md) | **In progress.** The artifact store, the package registry and each machine's container runtime get tools that say what they hold, what names each thing and what could go; every removal is a dry run unless a person asks and says why, and what decides is always a record | [ADR 0251](../../02-DECISIONS/0251-the-registries-say-what-they-hold-and-keep-what-is-named-and-a-machine-keeps-the-image-it-runs-and-the-one-before.md), [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md) | -| [`52-shares-and-mounts.md`](52-shares-and-mounts.md) | **Proposed.** A node shares a directory through `nfs-server`, which provides it as a provision to the private network only, squashing every client to the share's owner; a node that asks mounts it through `mounts`, a mount unit and an automount per share, so nothing mounts or fails at boot; a device that comes and goes is an occasional source whose absence is a state | [ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md), [ADR 0051](../../02-DECISIONS/0051-shared-data-is-the-operators.md), [ADR 0092](../../02-DECISIONS/0092-an-operator-delivers-a-pair-credential.md) | +| [`52-shares-and-mounts.md`](52-shares-and-mounts.md) | **In progress.** A node shares one folder at a time through `nfs-server`, which provides it as a provision to the private network only, mapping every client to the folder's owner; a node that asks mounts it through `mounts`, a mount unit and an automount per share, so nothing mounts or fails at boot; a device that comes and goes is an occasional source whose absence is a state | [ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md), [ADR 0051](../../02-DECISIONS/0051-shared-data-is-the-operators.md), [ADR 0092](../../02-DECISIONS/0092-an-operator-delivers-a-pair-credential.md) | ## Not yet written diff --git a/04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md b/04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md index 45580f51..a6bc7ac4 100644 --- a/04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md +++ b/04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md @@ -1,11 +1,11 @@ --- -status: open +status: located opened: 2026-09-30 located-in: - mesh-catalog (no module shares a path over the network) - hq 02-DECISIONS (a file-share seat, per ADR 0126, is a module's own to define) fixed-by: -amended-design: +amended-design: [03-DESIGN/01-to-be/52-shares-and-mounts.md] --- # 169 — A machine shares its files, and the mesh does not know @@ -129,3 +129,8 @@ the mounted tree, answers it on the consumer's side too. Re-checked 2026-10-08: still holds. No module in either catalogue shares a path over the network, and no `nfs-share` or `smb-share` seat exists. + +**Decided 2026-10-08:** [ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md), +from [research 038](../../01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md), which measured +the closure this report predicted: the workstation's library mount has timed out at every boot since +2026-10-04. The seats are `node-nfs-server` and `node-mounts`, the provision `nfs-share`; Samba waits. From f4b79ba7c2a853982883ca28deddc106bf04a438 Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 18:37:15 +0200 Subject: [PATCH 3/5] Design 52 says how the modules were built: the server's bucket carries each share's path, two root processes, settings default to none --- ...n-demand-and-only-nodes-that-ask-get-it.md | 11 ++--- 03-DESIGN/01-to-be/52-shares-and-mounts.md | 41 +++++++++++-------- 2 files changed, 31 insertions(+), 21 deletions(-) diff --git a/02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md b/02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md index e4d9298f..3f351c5f 100644 --- a/02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md +++ b/02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md @@ -62,7 +62,8 @@ Rejected: so the backups are never exported. 9. **A provider's paths written into its binding.** A binding carries what the provider's manifest `serves`, which is the same for every machine; a share's path is a setting of one assignment. The - consumer asks the server for it over the bus instead, so no controller change is needed for it. + consumer reads it from the server's bucket on the bus instead, so no controller change is needed for + it. ## Decision @@ -107,10 +108,10 @@ Rejected: is absent raises nothing. The server's health is its service and its export table holding every declared export. 9. **Over the bus.** The controller wires the two through the provision: the binding gives the consumer - the server's name. The consumer then asks the server's seat, `node-nfs-server.exports`, by request and - reply for each share's path and access, so a share's path is said once, on the server. `nfs-server` - keeps its exports, and `mounts` keeps each share's state, in their own key-value buckets (ADR 0201). - Neither reads the other's files (ADR 0260). + the server's name. `nfs-server` keeps its exports in its key-value bucket (ADR 0201), and `mounts` + reads it (`reads`) for each share's path and access, so a share's path is said once, on the server. + `mounts` keeps each share's state in its own bucket. The seat's `exports` verb answers the same by + request and reply. Neither reads the other's files (ADR 0260). 10. **Taking over a hand-written line is an act.** The `mounts` module's `adopt` comments out one fstab line, keeping a copy, only after its own unit pair for that mount point is armed, and only on a person's word. The ZFS `sharenfs` property is cleared the same way, by `nfs-server`, once its own diff --git a/03-DESIGN/01-to-be/52-shares-and-mounts.md b/03-DESIGN/01-to-be/52-shares-and-mounts.md index 6251e2a3..1fd51752 100644 --- a/03-DESIGN/01-to-be/52-shares-and-mounts.md +++ b/03-DESIGN/01-to-be/52-shares-and-mounts.md @@ -64,17 +64,21 @@ The glossary holds the three words. - **What it refuses.** A path that does not exist or is not a directory, `/`, and a path that holds a backup repository at its root or one level below. A refused share is said by the tools and makes the module unhealthy; the others are exported. -- **Its file.** `/etc/exports.d/mesh.exports`, written by the module's own process, which runs as root. - One line per share: the path, to `${machine:mesh-range}` and nothing else, with `rw` or `ro`, `sync`, - `secure`, `no_subtree_check`, `all_squash`, a stable `fsid`, and `anonuid` and `anongid` read from the - path's owner. `exportfs -ra` runs when the file changes. -- **The service.** `nfs-server.service`, running and enabled. NFS version 3 is turned off in the - module's drop-in for `nfs.conf`, so only 2049/tcp is needed, declared in `listens` with `from: mesh`. +- **Its file.** `/etc/exports.d/mesh.exports`, written by the module's own process, which runs as root + and looks every 30 s. One line per share: the path, to `${machine:mesh-range}` and nothing else, with + `rw` or `ro`, `sync`, `secure`, `no_subtree_check`, `all_squash`, and `anonuid` and `anongid` read from + the path's owner. `exportfs -ra` runs when the file changes or a share is missing from the kernel's + table. The filesystem's own id is kept; none is forced. +- **The service.** `nfs-server.service`, running and enabled. NFS versions 2, 3 and 4.0 are turned off + in the module's drop-in in `/etc/nfs.conf.d`, so only 2049/tcp is needed, declared in `listens` with + `from: mesh`. - **The provision.** `nfs-share`, at the mesh's scope, with no identity: the binding gives the - consumer the server's name, and the consumer asks for the rest. + consumer the server's name, and the consumer reads the rest from the server's bucket. Its consumers' + data class is `none`: the folders are the operator's, and the mesh backs up nothing through it. - **Health.** Its tool `nfs_health`: the service is active, every share is in the kernel's export table, and no share was refused. -- **Its bucket.** `exports`, one entry per machine: the shares as the kernel holds them. +- **Its bucket.** `exports`, one entry per machine, keyed by the node's name: the answer of `exports`, + put by the module's tools every 30 s. The root process does not reach the bus; its tools do. - **Its seat's verbs.** `exports` (r), `clients` (r), `test` (r), `reload` (a) and `adopt` (a). `adopt` clears a ZFS dataset's `sharenfs` property for a path the module now exports, only once the mesh's export of it is in the kernel's table, and only with a person's confirmation and reason. @@ -82,18 +86,22 @@ The glossary holds the three words. ## 2. The client: `mounts` - **Its setting `shares`.** A list of `name=mountpoint`, each optionally `:ro`: the `nfs-share` shares - this node mounts, by the server's name for them. + this node mounts, by the server's name for them. `none` (the default) mounts none. - **Its setting `sources`.** Occasional sources: `name=smb:///@` or - `name=disk:@`, each optionally `:ro`. + `name=disk:@`, each optionally `:ro`. `none` (the default) is none. - **Its secret `smb-credentials`.** One section per SMB source, `[name]` with `username=` and `password=`, issued outside the mesh and given with `secret accept mounts smb-credentials --from `. The process writes one credentials file per source, readable by root only. A source with no section is said with that remedy and is not mounted. -- **Asking the server.** The process asks `node-nfs-server.exports` on the server named by the binding, - over the bus, for each share's path and access. A share the server does not export is said, not - guessed. +- **Reading the server.** The module reads the server's bucket `exports` (its manifest's `reads`) for + each share's path and access: its tools, which reach the bus, write what they read to a file of the + module's, and the root process takes only a name, an absolute path and the access from it. A share + the server does not export is said, not guessed, and a silent server removes nothing. +- **Two processes, both root.** `apply` writes the unit pairs and the SMB credentials files and arms + the automounts; `watch` probes each source and keeps the states. The tools act as root through + `sudo -n` for `mount`, `unmount` and `adopt`, and say so when it is refused. - **The unit pair.** For each share and source, a `.mount` and an `.automount`, named by the escaped - mount point, written in `/etc/systemd/system` by the module's own process with a marker line. It never + mount point, written in `/etc/systemd/system` by `apply` with a marker line. It never touches a unit without its marker, and it removes its own pairs that are no longer asked for: - the `.automount` is enabled under `remote-fs.target`, with an idle time of 10 minutes; - the `.mount` is not enabled; its mount timeout is 30 s, 10 s for an occasional source; it starts @@ -106,8 +114,9 @@ The glossary holds the three words. `failed` (the mount was tried and refused, with the error). - **Its probes.** Every 30 s, a connection to the server's port (2049, or 445 for SMB) with a short timeout, and the device for a disk. It never touches a mount point, so a hung server cannot hang it. -- **Health.** Its tool `mounts_health`: every automount is armed, and no share that is not occasional - has been `unreachable` or `failed` for 5 minutes. An absent occasional source is healthy and said as +- **Health.** `apply` by its unit; `watch` by the tool `mounts_health`: every automount is armed, every + share the settings name could be set up, `watch` looked in the last 2 minutes, and no share that is not + occasional has been `unreachable` or `failed` for 5 minutes. An absent occasional source is healthy and said as *not present since 18:02*. - **Its seat's verbs.** `list` (r), `test` (r), `mount` (a), `unmount` (a), `adopt` (a); its tools `mounts_list` and `mounts_exports` (r) stay. From bffd0f90d06ca7d16256c4d56c8d5bb57fe0a647 Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 20:21:48 +0200 Subject: [PATCH 4/5] ADR 0263 and design 52 after review: what each export promises, per-node grants, refusals, not-built checks, and a read-only-first rollout --- ...n-demand-and-only-nodes-that-ask-get-it.md | 38 +++++--- 03-DESIGN/01-to-be/52-shares-and-mounts.md | 86 ++++++++++++++----- 2 files changed, 91 insertions(+), 33 deletions(-) diff --git a/02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md b/02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md index 3f351c5f..a197ae5b 100644 --- a/02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md +++ b/02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md @@ -72,22 +72,32 @@ Rejected: path, and read-only or read-write. The path is the operator's data (ADR 0051): the module never creates, chowns, chmods or removes anything under it. It writes its own file in `/etc/exports.d`, never `/etc/exports`, and listens on 2049/tcp from the private network. It serves NFS version 4 only, so no other port opens. -2. **Every export squashes every client to the share's owner.** `all_squash`, with `anonuid` and - `anongid` read by the module from the exported path's owner on the server, never written as - numbers. The clients are `${machine:mesh-range}` and nothing else. A file written from any node lands - owned by the library's owner, and no client acts as any other account on the server. +2. **Every export maps every client to the share's owner.** `all_squash`, with `anonuid` and `anongid` + read by the module from the exported folder's owner on the server (after following any symbolic link + to the real folder), never written as numbers. A share owned by root is refused, so no client is ever + mapped to root. A file written from any node lands owned by the folder's owner. **What this does not + promise:** a client acts as that owner, so it can do whatever the owner may do with what it reaches. + Where a share is a folder inside a larger filesystem, NFS cannot fully confine a client that forges + file handles to that folder; it can reach the rest of that filesystem as the owner. That is why + rule 3 refuses a share at or under a backup repository, or with one below it, and why the export + carries `mp`, so it is served only while its filesystem is mounted. 3. **One share per folder, never a pool's root** (the operator's choice, 2026-10-08): on the home server, the library and the downloads. The backup repository and the archive are not exported. The module - refuses a share that is `/` or that holds a backup repository at its path or one level below, so a - setting cannot export one by mistake. + refuses a share that is `/`, is owned by root, is inside a backup repository, or holds one within four + levels below (refusing when it cannot finish looking), all judged on the real folder after any + symbolic link, so a setting cannot export one by mistake. 4. **A node mounts through one module.** The module `mounts` holds the node seat `node-mounts`. For each share it writes a `.mount` unit and an `.automount` unit, never an fstab line. Only the automount is enabled, so nothing mounts at boot and nothing can fail the boot. Each mount has an idle timeout (default 10 minutes) and a mount timeout (default 30 s; 10 s for an occasional source). A read-write NFS share is mounted `hard`. A mount point is always an autofs mount, so nothing writes into the empty directory under it (issue 169's data-loss case). -5. **Only the nodes that ask get a share.** A node mounts what its `mounts` assignment's setting `shares` - lists. The list may be set once at the mesh-wide settings layer, which is how "every node" is said, +5. **Only the nodes that ask, and that the server grants, get a share.** A share is exported to one + node's address only when the server's setting `grants` names that node for that share and the node's + `mounts` assignment asks for it; the access is the lower of the two, decided on the server. Addresses + come from the controller's list of machines, never from what a client says. A share nobody is granted + is exported to nobody: the control-node, like any node, gets nothing unless both name it. A node mounts + what its `mounts` assignment's setting `shares` lists. The list may be set once at the mesh-wide settings layer, which is how "every node" is said, and a node's own layer can leave a share out. Which nodes are assigned, and whether the laptop mounts read-only, is decided at assignment, which is the operator's; the recommendation is the workstation and the laptop (read-only), not the control-node. @@ -137,11 +147,13 @@ Rejected: - The `mounts` module's `mounts_list` shows every mount point's source, its unit pair, its state and which settings layer asked for it. `mounts_exports` shows the server's export table. A test in the module renders the unit pair from a binding and asserts the options above. -- The lab composes a server and two clients and asserts: the export names only the private network's - range with `all_squash`; a file written from a client is owned by the share's owner; the client's - boot completes with the server stopped; and an occasional source that is absent leaves no failed unit. -- The controller's self-check counts fstab lines of type `nfs` or `cifs` on every node through - `mounts_list`, and reports any that are not adopted. +- **Not built yet:** a lab scenario composing a server and two clients, asserting that the export names + only the granted nodes' addresses with `all_squash`, that a file written from a client is owned by the + share's owner, that a client's boot completes with the server stopped, and that an absent occasional + source leaves no failed unit. +- **Not built yet:** a probe in the controller's self-check that counts fstab lines of type `nfs` or + `cifs` on every node through `mounts_list`, and reports any that are not adopted. +- Until both exist, the modules' own tests and the safe rollout in design 52 are the check. ## References diff --git a/03-DESIGN/01-to-be/52-shares-and-mounts.md b/03-DESIGN/01-to-be/52-shares-and-mounts.md index 1fd51752..d9eb11ba 100644 --- a/03-DESIGN/01-to-be/52-shares-and-mounts.md +++ b/03-DESIGN/01-to-be/52-shares-and-mounts.md @@ -61,14 +61,26 @@ The glossary holds the three words. - **Its setting `shares`.** A list of `name=path`, each optionally `:ro`, separated by spaces: on the home server the library and the downloads. A share's path is the operator's data (ADR 0051); the module never creates, chowns, chmods or removes anything under it. -- **What it refuses.** A path that does not exist or is not a directory, `/`, and a path that holds a - backup repository at its root or one level below. A refused share is said by the tools and makes the - module unhealthy; the others are exported. +- **Its setting `grants`.** Which node may mount which share, and how: `library=:rw,:ro`. + `none` (the default) grants nothing. A share is exported to a node only when `grants` names it and the + node's `mounts` asks for it (its bucket `shares`, field `wants`); the access is the lower of the two. + The node's address comes from the controller's list of machines, rendered into the module's own file, + and must be inside the private network's range; never from what the node says. +- **What it refuses.** A path that does not exist or is not a directory, `/`, a folder owned by root + (uid or gid 0), and a folder that is, is inside, or holds within four levels a backup repository; a look + below that is cut short by its bound refuses rather than exports. A clean look is kept for an hour. + Every check runs on the real folder after any symbolic link. A refused share is said by the tools and + makes the module unhealthy; the others are exported. - **Its file.** `/etc/exports.d/mesh.exports`, written by the module's own process, which runs as root - and looks every 30 s. One line per share: the path, to `${machine:mesh-range}` and nothing else, with - `rw` or `ro`, `sync`, `secure`, `no_subtree_check`, `all_squash`, and `anonuid` and `anongid` read from - the path's owner. `exportfs -ra` runs when the file changes or a share is missing from the kernel's + and looks every 30 s. One line per share: the real path (after any symbolic link), to the addresses of + the nodes the server grants it to and that ask for it, and to nobody else, with + `rw` or `ro`, `sync`, `secure`, `no_subtree_check`, `all_squash`, `anonuid` and `anongid` read from + the folder's owner, and `mp` (or `mp=`), so nothing is served from an + unmounted dataset's empty folder. `exportfs -ra` runs when the file changes or a share is missing from the kernel's table. The filesystem's own id is kept; none is forced. +- **Unassigned.** The process is stopped after its configuration is removed (the node-engine removes a + module's resources in reverse order); finding it gone, it removes its export file and has the kernel + drop the exports. A restart or a push keeps them. - **The service.** `nfs-server.service`, running and enabled. NFS versions 2, 3 and 4.0 are turned off in the module's drop-in in `/etc/nfs.conf.d`, so only 2049/tcp is needed, declared in `listens` with `from: mesh`. @@ -93,6 +105,13 @@ The glossary holds the three words. `password=`, issued outside the mesh and given with `secret accept mounts smb-credentials --from `. The process writes one credentials file per source, readable by root only. A source with no section is said with that remedy and is not mounted. +- **What it refuses.** A mount point at, under or above any data the mesh keeps on this machine (the + controller's `data` verb, asked over the bus; until it is read nothing new is set up), a system path, + a mount point where something else is mounted or a folder that is not empty, and a share this machine + serves itself. It never stops a mount whose mounted filesystem is not its own source, and a setting + entry it cannot read removes nothing. +- **What it asks for.** Its bucket `shares` carries `wants`: each share this machine asks of which server, + read-write or read-only. The server decides; the mount takes the access the server gave this machine. - **Reading the server.** The module reads the server's bucket `exports` (its manifest's `reads`) for each share's path and access: its tools, which reach the bus, write what they read to a file of the module's, and the root process takes only a name, an absolute path and the access from it. A share @@ -147,22 +166,49 @@ Each says the share, the node and since when: *"The home server's share `library from the workstation since 19:21."* An absent occasional source raises nothing; `mounts_list` says *"the share `retro-games` is not present since 18:02"*. -## 5. Migration +## 5. The safe rollout -1. Build and deliver the two seats, `nfs-server` and `mounts`. -2. Assign `nfs-server` to the home server with the library and the downloads. Check `exports` and - `test` from the workstation. -3. `adopt` on the home server clears the library's `sharenfs` property. The underlay range leaves the - export table; the private network's range is in it. -4. Assign `mounts` to the workstation. Its unit pair for the library takes precedence over the unit - generated from fstab. Check `mounts_list` and a file written from the workstation. -5. `adopt` on the workstation comments out the library's fstab line, keeping a copy of the file. -6. Accept the gaming device's credential into the vault with `secret accept`, add the gaming device as an - occasional source, and `adopt` its fstab line. -7. Assign `mounts` to the laptop, read-only. +Read-only first, one node first, and every step checked before the next. Each step names how it is +undone. Every `adopt` is a person's act. Nothing here touches the library's files, owners or modes, or +the media modules' local bind mounts on the home server. -Every `adopt` is a person's act. Nothing in this migration touches the library's files, owners or -modes, or the media modules' bind mounts. +**Before anything:** the seats are merged and the controller that knows them is running; the two +modules' pull request is green against it. On the home server, `node-backup.backed-up` shows a good +night for Plex's own data, and `nfs-server`'s dry runs are read: `exports` lists only the intended +folders, none refused for a reason not understood. + +1. **Assign `nfs-server` to the home server, read-only.** Its `shares` names the library only, and its + `grants` names the workstation read-only (`library=:ro`). Check `exports`: one line, the + workstation's address only, `ro`, `all_squash` to the library's owner, `mp`. The ZFS property's own + export to the home network stays as it is (the packet filter already blocks it). + *Undo:* unassign `nfs-server`; its export file goes with it. +2. **Assign `mounts` to the workstation, read-only.** Its `shares` is `library=:ro`. Check + `node-mounts.list`: the pair is armed, the share reads `armed` then `mounted` after a listing, and the + files are there. Its unit pair takes the place of the unit generated from fstab. + *Undo:* unassign `mounts`; its unit pairs go, and the fstab line is still there. +3. **Watch for a day.** `mounts_health` and `nfs_health` stay healthy; a reboot of the workstation + completes with the home server's share reachable and with it stopped (the share reads `unreachable` + and the boot does not wait). +4. **Read-write.** Change the grant to `rw` and the workstation's setting to drop `:ro`. Write one test + file from the workstation, check its owner on the home server is the library's owner, and remove it. + *Undo:* set both back to `ro`. +5. **Adopt the workstation's library line**: `node-mounts.adopt` dry run, read it, then with + `confirm` and a reason. *Undo:* the dated copy of fstab beside it. +6. **Adopt the ZFS property**: `node-nfs-server.adopt` dry run, then confirmed. The home network's + export goes; the mesh's stays. *Undo:* `zfs set sharenfs=` by a person, recorded as a + hand-act. +7. **The downloads**, the same way: read-only first, then read-write. +8. **The gaming device**: the operator gives `smb-credentials` with `secret accept`; the source is added + to `sources`; `node-mounts.list` says it `absent` while the device is off, and it mounts on access + when it is on. Then its fstab line is adopted. +9. **The laptop**, read-only, granted read-only on the home server. + +**Root.** Both modules' tools act as root through `sudo -n`, and the `sudo` module grants the operator +account every command, so the modules' own checks are the only line before root. Narrowing sudo is a +separate decision. + +**Stop at once** when a share reads `failed` for a reason not understood, when a dry run would touch a +line or a dataset other than the one asked for, or when anything under the library changes owner. ## Not in this design From d05c9d7ca4c7844da3bee427983c1016668dd6cf Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 20:22:05 +0200 Subject: [PATCH 5/5] Name the safe order of taking shares into use without the retired word --- ...a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md | 2 +- 03-DESIGN/01-to-be/52-shares-and-mounts.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md b/02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md index a197ae5b..5f578fab 100644 --- a/02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md +++ b/02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md @@ -153,7 +153,7 @@ Rejected: source leaves no failed unit. - **Not built yet:** a probe in the controller's self-check that counts fstab lines of type `nfs` or `cifs` on every node through `mounts_list`, and reports any that are not adopted. -- Until both exist, the modules' own tests and the safe rollout in design 52 are the check. +- Until both exist, the modules' own tests and taking it into use safely, as design 52 §5 orders it, are the check. ## References diff --git a/03-DESIGN/01-to-be/52-shares-and-mounts.md b/03-DESIGN/01-to-be/52-shares-and-mounts.md index d9eb11ba..b76e7500 100644 --- a/03-DESIGN/01-to-be/52-shares-and-mounts.md +++ b/03-DESIGN/01-to-be/52-shares-and-mounts.md @@ -166,7 +166,7 @@ Each says the share, the node and since when: *"The home server's share `library from the workstation since 19:21."* An absent occasional source raises nothing; `mounts_list` says *"the share `retro-games` is not present since 18:02"*. -## 5. The safe rollout +## 5. Taking it into use, safely Read-only first, one node first, and every step checked before the next. Each step names how it is undone. Every `adopt` is a person's act. Nothing here touches the library's files, owners or modes, or