novox/hq
7
0
Fork 0
Code Issues Pull Requests 5 Releases Wiki Activity

Research 038, ADR 0263 and to-be 52: let the mesh own shares and mounts, so no node depends on a hand-written export or fstab line
mesh/merge-gate pass: the change touches no module of the mesh's graph
Details
mesh/repo-check pass: its merge-check.sh passed
Details
mesh/delivery-group group feat/mounts-module rejected: a member's own check failed
Details
mesh/delivery superseded: a newer head of the same pull request
Details

Browse Source
This commit is contained in:
jochen
2026-10-08 17:23:52 +02:00
parent 72e0a851cf
commit 3edafd78ab
6 changed files with 644 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md
+78
View File
@@ -0,0 +1,78 @@
---
status: active
initiated: 2026-10-08
touches:
- 04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md
- 04-ISSUES/154-a-machines-own-network-is-not-a-reach/00-report.md
- 01-RESEARCH/027-the-system-layer-as-modules/02-candidates-and-questions.md
- 03-DESIGN/01-to-be/26-the-seats.md
- 03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md
- 02-DECISIONS/0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md
- 02-DECISIONS/0051-shared-data-is-the-operators.md
- 02-DECISIONS/0092-an-operator-delivers-a-pair-credential.md
- the catalogue's module manifests
- the controller's seat set
became: []
---
# 038 — Shares and mounts the mesh manages
## What is investigated
How the mesh shares a directory from one node and mounts it on the others, so that no machine
carries a hand-written export or a hand-written mount line. Three kinds of source are in scope:
1. **A node's share.** The home server shares its storage pool over NFS. A node that wants the files
mounts them.
2. **A device outside the mesh that is not always there.** A gaming device on the home network shares
a folder over SMB, and only while it is switched on. A USB disk is the same case.
3. **What the mesh should show and say** when a mount works, when it does not, and when its source is
simply away.
## Why
On 2026-10-08 the operator found two failed mounts on the workstation, and said two things:
- about the media library: *"This should be handled by the mesh. If a node exposes an NFS share, it
should get configured on all the nodes in their fstab?"*
- about the gaming device's share: *"This mount only works if I plug in my recalbox machine."*
They also set a requirement: **the home server shares its storage pool over NFS**, the workstation's
library mount comes from that share, the export reaches only the mesh's private network, it squashes
root, and the media modules' data (the Plex library is sacred) must not be put at risk.
[Issue 169](../../04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md)
predicted the first failure on 2026-09-30: nothing declares the share's ports, so the packet filter
closes them. This effort measures what happened, and proposes the decision issue 169 asked for.
## What was found
The detail is in [01-evidence.md](01-evidence.md). In short:
1. **The library mount times out because the home server's packet filter drops NFS.** The server
exports the library and its NFS service runs. The mesh's filter accepts only what a module
`listens` on, and no module listens on 2049, 111 or 445. The server's NFS counters read zero I/O
since its last boot six days earlier, so no client has used either share since then.
2. **The gaming device's mount fails because the gaming device is off.** It is an SMB share on the device itself,
not a disk. The fstab line has no `nofail` and no automount, so each boot records a failed unit.
The line also carries the share's password in a world-readable file.
3. **The export is not where a person would look.** The server's `/etc/exports` holds only comments.
The export comes from a property of the ZFS dataset, which writes its own file in `/etc/exports.d`.
The library is a child dataset of the pool, so an export of the pool's root does not show the
library's files unless it crosses into child filesystems.
4. **User ids do not match across nodes.** The library is owned by a service account and group. The
operator's account has the same id (1000) on every node, but on the home server that id belongs to
a different account, which owns the pool's backup repository. NFS with `sec=sys` trusts the client's
id. A plain export of the whole pool would give the workstation's operator account the backup
repository.
5. **No module, and until now no tool, covers any of this.** No module in either catalogue exports or
mounts anything. No mesh tool read a machine's fstab, its mount units or its exports, so this effort
read them through the login shell. A read-only `mounts` module now proposes those tools.
## Where it goes
[02-options.md](02-options.md) weighs the options. The proposed decision is
[ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md),
and the to-be design is
[52 — Shares and mounts](../../03-DESIGN/01-to-be/52-shares-and-mounts.md). Both wait for the
operator's word. The open questions are listed at the end of the options.
01-RESEARCH/038-shares-and-mounts-the-mesh-manages/01-evidence.md
+134
View File
@@ -0,0 +1,134 @@
# 038 — Evidence
Measured on 2026-10-08 on the four nodes, read only, through the mesh's tools: `node-service-manager`
(`failed`, `status`, `journal`), `node-uplink.links`, `node-packet-filter.rules`,
`disk-load.disk_filesystems`, `docker.docker_list`, and `node-login-shell.execute` for the files no
tool read (§7). Addresses are given in documentation ranges: the home network (the underlay) as
`192.0.2.0/24`. Node roles: the **home server**, the **workstation**, the **laptop**, the **control-node**.
## 1. The workstation's two failed mounts
`node-service-manager.failed` on the workstation lists two failed system units, both generated from
`/etc/fstab` (their unit file is under `/run/systemd/generator`, and `mesh_declared` is false):
| unit | the fstab line (password removed) | what the journal says |
|---|---|---|
| the library | `<home server>:<library> <same path> nfs defaults,_netdev 0 0` | `Mounting timed out. Terminating.` 90 s after boot, at every boot since 2026-10-04; the last successful mount was at the boot of 2026-08-23 |
| the gaming device | `//<device>/Share <mountpoint> cifs username=root,password=…,iocharset=utf8,sec=ntlmssp 0 0` | `mount error(113): could not connect to <device>` at every boot |
Neither line has `nofail` or `x-systemd.automount`. Both name the source by its underlay address.
`remote-fs.target` is inactive on the workstation, because both of its members failed.
## 2. Why the library times out
- The home server exports the library, and its NFS and Samba services run. `nfs-server` and `smb`
are active and enabled, and ports 2049, 111 and 445 are listening on every address.
- **The mesh's packet filter drops them.** The mesh's table on the home server has `policy drop` on
its input chain for the underlay link and the private network link. It accepts a port only where a
module `listens` on it ([ADR 0045](../../02-DECISIONS/0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md)).
No rule names 2049, 111 or 445.
- From the workstation, a TCP connection to the home server's port 2049 fails, and one to port 22 on
the same address succeeds.
- The home server's NFS counters (`/proc/net/rpc/nfsd`) read `io 0 0` since its boot on 2026-10-04.
No established connection to 2049 or 445 exists. **Nobody has used either share since that boot.**
This is the failure [issue 169](../../04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md)
predicted: the shares close silently once the filter is the sum of what modules listen on.
## 3. What the home server exports, and from where
- `/etc/exports` holds only the distribution's comments; it was last written on 2026-09-20.
- The export comes from ZFS. The library dataset has `sharenfs=rw=@192.0.2.0/24` and `sharesmb=on`.
ZFS writes `/etc/exports.d/zfs.exports` itself, marked *do not edit*. The kernel's export table
(`/var/lib/nfs/etab`) holds one entry: the library, to the underlay range, `rw,sync,root_squash,
no_all_squash,secure,no_subtree_check,sec=sys`.
- The Samba configuration has one share, `media`, on the library, `valid users = media`,
`force user = media`, writable.
- **The pool is two filesystems.** The pool is mounted at its root, and the library is a child dataset
mounted under it. An NFS export of the pool's root does not show the library's files unless the
export crosses into child filesystems (`crossmnt`); without it, a client sees an empty directory
where the library should be.
## 4. What lives on the pool, and who uses it
The pool's root holds five directories:
| directory | owner | what it is |
|---|---|---|
| the library (child dataset, about 90 TiB used) | the media account and group, `0775` | movies, series, anime, music, audiobooks and more |
| downloads | the media account and group | the download clients' and the library managers' working area |
| archive | root | the operator's archive |
| a second Plex folder | the operator's account on the home server | an older library folder |
| the backup repository | the operator's account on the home server, `0700` | the restic repository the backup seat writes |
Plex's own configuration and database live on a separate disk, not on the pool.
Eight running containers bind paths on the pool, all of them as plain bind mounts of local paths:
Plex (the library's folders, all read-only), the library managers for series, movies, music and
subtitles, two download clients, and one automation app. **Exporting the pool over NFS changes none of
these paths and none of their owners.** An export adds a way in; it does not move or chown anything.
The only way an export endangers this data is by what it lets a client do (§5).
## 5. User and group ids across the nodes
| node | id 1000 | the media account | the media group |
|---|---|---|---|
| home server | a separate account, which owns the backup repository and the older Plex folder | exists, owns the library | exists |
| workstation | the operator | exists, same ids as on the server | exists; the operator is a member |
| laptop | the operator | absent | absent |
| control-node | the operator | absent | absent |
The library managers on the home server run as the media account and group, set by their `puid` and
`pgid` settings.
With `sec=sys`, the server believes the id a client sends; `root_squash` changes only root. So:
- **A client's operator account (1000) is the home server's other account (1000).** An export of the
pool's root would let it read and write the backup repository and the older Plex folder.
- **A file written by the laptop's operator lands owned by 1000**, not by the media account, and the
library managers may then fail to rename or delete it.
- A device on the underlay that is not a node can claim any id. That is a reason to export to the
private network only, where every address is a node.
## 6. The gaming device's share
- The gaming device is a separate machine on the underlay, a retro-gaming device that shares a folder named
`Share` over SMB. It is not a USB disk. The router's DNS knows its name. It was switched off during
the survey and did not answer a ping.
- The workstation has `cifs-utils`, `avahi` and `nss-mdns`, and its `hosts:` line contains
`mdns4_minimal`. **The packet filter drops inbound mDNS** on every node, as the `avahi` module's
README already records. So discovery by mDNS cannot see the gaming device arrive today. Waiting for it
would first need a local-link reach for `listens`, which is the question of
[issue 154](../../04-ISSUES/154-a-machines-own-network-is-not-a-reach/00-report.md).
- The share's password is written in `/etc/fstab`, which is mode `0644`. Every account on the
workstation can read it.
## 7. What the catalogue and the tools cover
- **No module exports or mounts.** No manifest among the 117 modules of the core catalogue and the 12
of the media catalogue writes an export, a Samba share, a mount unit or an fstab line.
- **`disk-load.disk_filesystems`** lists NFS and CIFS filesystems that are mounted now, with their space.
It cannot see a mount that failed, an fstab line, a mount unit or an export.
- **`node-service-manager`** shows a mount unit's state and journal, but not what it mounts or with
which options.
- **No tool read a machine's fstab, its mount and automount units with their sources, or its NFS and
SMB exports.** This effort read them with `node-login-shell.execute`, which runs a shell command as
the operator account. That is a work-around for a missing tool, so a read-only `mounts` module with
two tools, `mounts_list` and `mounts_exports`, was opened in the catalogue alongside this effort. It
redacts passwords in mount options and never touches a network mount point, so a hung server cannot
hang the tool.
## 8. Prior work
- **Issue 169** (open, the operator's proposal of 2026-09-30): two seats, `nfs-share` and `smb-share`,
because the two protocols share an intent and not a contract; a consumer module that writes a
systemd `.mount` unit from its binding; the modules that use the files never learn about NFS; the
data-loss case of a writer starting before its mount; and identity crossing the wire.
- **Research 027**, option M2: each client module writes its own systemd mount and automount unit, and
`/etc/fstab` stays the machine's.
- **Research 026**, the tools list: `exports`, `shares`, `clients`, `reload` on the server;
`mounts`, `mount`, `unmount`, `test` on the client.
- **ADR 0092**: a credential for something outside the mesh is an operator-delivered secret, accepted
into the vault (`secret accept`) and never replaced by a made one.
- **`${machine:mesh-range}`** already exists as a placeholder in two manifests, so an export line can
name the private network's range without naming a node.
01-RESEARCH/038-shares-and-mounts-the-mesh-manages/02-options.md
+150
View File
@@ -0,0 +1,150 @@
# 038 — Options
Each question below lists the options weighed and the one recommended. The recommendations together
are [ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md).
## Q1. Who writes the export
| option | for | against |
|---|---|---|
| A. Keep the ZFS `sharenfs` property, and have a module declare only the ports | nothing new on disk | the export lives in a dataset property, beside the operator's data; a module would be setting properties on data the mesh does not own (ADR 0051); the options are one string with no room for squashing |
| B. A module owns `/etc/exports` whole | one file, one owner | the file is the machine's; a second exporter on the same node would collide |
| **C. A module, `nfs-server`, writes its own file in `/etc/exports.d`, and the ZFS property is turned off when the module's export is live** | the mesh owns exactly its file; the distribution reads the directory; unassigning removes the file | a migration step: the property must be cleared by an act, not by hand |
**Recommended: C.** The module holds a node seat, `nfs-share`, so one module exports on a node, and it
provides the provision `nfs-share` (issue 169's proposal). Samba is left as found, with its own seat
`smb-share` later; nothing on the mesh uses it, and the gaming device's SMB share is a client case (Q6).
## Q2. What is exported, and to whom
The operator asked for the pool's root. Exporting it plainly has two problems (evidence §3 and §5): the
library is a child filesystem, so the export needs `crossmnt` to show it; and with `sec=sys`, any client
account with id 1000 becomes the server's own 1000, which owns the backup repository.
| option | for | against |
|---|---|---|
| A. The pool's root, `rw,root_squash`, to the underlay range | what the operator literally asked; what exists today (but for the library only) | gives the backup repository to every client's operator account; open to every device on the underlay, which can claim any id |
| B. The pool's root, `rw,crossmnt,all_squash`, every client mapped to the media account, to the private network only | one share, as asked; every file written lands owned by the media account; the `0700` backup repository is unreadable to the media account, so it stays closed | every client writes as one account, so there is no per-person permission on the share; top-level folders other than the library are writable by every client |
| **C. One share per subtree: the library and the downloads, each `rw,all_squash` to the media account, to the private network only; the backup repository and the archive not exported** | the backup repository is not reachable at all; ownership is right; a consumer mounts only what it needs | two shares instead of one; a later folder needs a new share entry (one setting) |
| D. Kerberos (`sec=krb5`) with id mapping | real per-person identity | needs a KDC and keytabs on every node; far beyond the problem |
**Recommended: C, with B as the operator's alternative if one share is wanted.** Both use
`all_squash` with `anonuid` and `anongid` set to the owner of the exported path, read on the server
(`${access:<id>:uid}`, from issue 153's fix), never written as numbers in a manifest. Both export to
`${machine:mesh-range}` only, so every client is a node, reached over the private network. `root_squash`
is implied by `all_squash`. `sync`, `secure`, `no_subtree_check` stay as they are today. The server
serves NFS version 4 only, so only port 2049 is opened, `from: mesh`; 111 and the mountd ports stay
closed.
**What this does not touch:** the local bind mounts of Plex and the library managers (evidence §4),
the datasets' owners and modes, and the Samba share.
## Q3. How a node mounts a share
| option | for | against |
|---|---|---|
| A. The mesh writes lines into `/etc/fstab` | the file people know | one shared file and one ordered list; the node-engine can place a marked block but every line competes with the operator's |
| B. `autofs` with its own maps | mature automounter | a second daemon doing what systemd already does |
| **C. A `.mount` unit and an `.automount` unit per share, written by one module, `mounts`, which holds the node seat `node-mounts`** | a unit is a unit: its state, its journal, its health; an automount never holds the boot; unassigning removes it; no shared file | units have to be named by the escaped mount path, which the module computes |
**Recommended: C.** The `.automount` is enabled under `remote-fs.target`; the `.mount` is not enabled
on its own, so nothing mounts at boot and nothing fails at boot. The mount is made at first access and
released after an idle time. The options carried:
- `x-systemd.automount` semantics through the `.automount` unit, `TimeoutIdleSec` (default 10 minutes);
- a mount timeout, `TimeoutSec` on the `.mount` (default 30 s for a node's share, 10 s for an
occasional one), instead of systemd's 90 s;
- `_netdev` and an order after the private network's link is up;
- for NFS, `vers=4.2,hard,timeo=150,retrans=3` on a read-write share (no silent write loss) and
`soft` allowed for a read-only one, as a setting.
An automount also closes issue 169's data-loss case: the mount point is always an autofs mount, so a
process that writes there either triggers the mount or gets an error. It never writes into the empty
directory underneath.
## Q4. Which nodes get a share
The operator suggested every node. Weighed:
| option | for | against |
|---|---|---|
| A. Every node mounts every share | no assignment to make | the control-node is the stably reachable machine the controller runs on, and gains a writable path into the library for no use; the laptop, behind a corporate VPN client, would try a mount whenever a file manager looks at the path |
| B. A node mounts a share only when a module on it requires it | precise | a person who wants the files in their file manager has no module that requires them |
| **C. A node mounts what its `mounts` assignment lists; the list may be set once at the mesh-wide layer** | opt-in per node, and "every node" is one setting at the mesh-wide layer, with a node's own layer to leave a share out | two layers to read when asking why a node has a share; `mounts_list` says which layer gave it |
**Recommended: C, assigned to the workstation and the laptop, not to the control-node.** A module that needs
the files (a player, an indexer) requires `nfs-share` as well; the controller then adds the share to
that node's `mounts` unit set, so B is the special case of C, not a second way.
## Q5. Health
| case | finding | condition |
|---|---|---|
| a node's share, automount armed, server answers | healthy | none |
| a node's share, server's 2049 unreachable from this node for 5 minutes | not healthy | `share-unreachable`, a warning: *"The home server's share `library` is not reachable from the workstation since 19:21."* |
| a node's share, the mount failed with an error other than a timeout | not healthy | the same condition, with the error |
| an occasional source, absent | **absent**, a state, not a finding | none |
| an occasional source, present but the mount fails | not healthy | `share-unreachable` |
| the server: the export table lacks a declared export, or `nfs-server` is down | not healthy | the module's ordinary health condition |
The `mounts` module's code probes each source's port on a timer and keeps each share's state in its
own key-value bucket (ADR 0201): `mounted`, `armed`, `absent`, `unreachable`, with since when. Its
health reads that bucket. Nobody reads its files.
## Q6. Occasional sources
The gaming device is not a node, so it cannot hold a module or provide a provision. Weighed:
| option | for | against |
|---|---|---|
| A. A fstab line with `nofail,x-systemd.automount` | small | a hand-written line again, and the password stays in the file |
| **B. An external source in the `mounts` assignment's settings: protocol, name, share, mount point, presence `occasional`, and the name of a credential** | the same unit pair and tools as a node's share; the password goes to the vault | a share with no provider, so no binding: the module checks the name itself |
| C. Wait for the device by mDNS and mount when it appears | mounts before anyone asks | the packet filter drops mDNS today (evidence §6); depends on issue 154 |
**Recommended: B now, C when issue 154 gives `listens` a local-link reach.** With an automount, nothing
has to happen when the gaming device appears: the next access mounts it. The probe only keeps the state
honest, so `mounts_list` says *"the share `retro-games` is not present since 18:02"*. A USB disk is the same kind of source,
with presence taken from its device unit (`BindsTo=` on the disk's by-uuid device) instead of a probe.
**The credential:** the share's password is an operator-delivered secret, accepted into the vault with
`secret accept` (ADR 0092), and reaches the node as a file the node-engine unseals (ADR 0086). The unit
names it as `credentials=<that file>`. Nothing in a repository or in `/etc/fstab` holds it.
## Q7. Taking over the existing fstab lines
The two lines are the operator's. The node-engine's own `.mount` unit in `/etc/systemd/system` already
takes precedence over the generated unit of the same name, but the line would still be read by every
other tool and person.
**Recommended:** the `mounts` module gets an act, `adopt`, that for one mount point:
1. checks a unit pair the module placed is armed for that mount point, and refuses if not;
2. comments out the fstab line, with a marker naming the module and the date, and keeps a copy of the
file beside it;
3. reloads the service manager and says what it did.
It runs only on a person's word. The library line is adopted after the home server's share works over
the private network; the gaming device's after its credential is accepted.
## Q8. Over the bus
Every exchange between the modules is over NATS (research 036):
- the controller wires provider and consumer through the provision; the binding carries `at` (the
server's private-network name), `path` and `options`;
- `nfs-server` keeps its exports and the clients it sees in its bucket, and answers `test` (does it
export this path to this node) by request and reply;
- `mounts` keeps each share's state in its bucket; a view or the controller watches it;
- no module reads another's files (ADR 0260).
## Open questions for the operator
1. **The pool's root, or one share per folder?** Recommended: one per folder (Q2 option C), so the
backup repository is never exported.
2. **Every node, or the nodes that ask?** Recommended: only the workstation and the laptop, with a
mesh-wide setting available (Q4).
3. **Underlay clients.** The private network only is recommended. Does anything on the home network
that is not a node (a TV, a media player) still need the NFS or SMB share? Today the filter closes
both, so nothing has been able to use them since 2026-10-04.
4. **Read-write from the laptop.** Should the laptop mount the library read-only? Recommended: yes,
read-only, since it travels.
02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md
+136
View File
@@ -0,0 +1,136 @@
---
topic: what runs on it
status: proposed
date: 2026-10-08
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0051-shared-data-is-the-operators.md
---
# 263. A share is a provision, a node mounts it on demand through one module, and only the nodes that ask get it
## Context
On 2026-10-08 two mounts on the workstation had failed at every boot since 2026-10-04
([research 038](../01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md), evidence §1–§2):
- **The media library over NFS timed out** after 90 s. The home server exports it and its NFS service
runs, but the mesh's packet filter drops port 2049, because no module listens on it (ADR 0045). The
server's NFS counters read zero I/O since its boot six days earlier. This is the silent closure
[issue 169](../04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md)
predicted on 2026-09-30.
- **A gaming device's SMB share failed** with "host unreachable", because the gaming device was off. The
fstab line has no `nofail` and no automount, and it holds the share's password in a file every
account can read.
The operator asked that the mesh handle this: a node that shares a directory should have it configured
on the nodes that use it. They also required that the home server share its storage pool over NFS, to
the private network only, with root squashed, and without risk to the media modules' data.
What was measured bears on the shape (evidence §3–§5):
- The export comes from a ZFS dataset property, not from `/etc/exports`.
- The library is a child filesystem of the pool, so an export of the pool's root shows an empty
directory there unless it crosses filesystems.
- With `sec=sys` the server believes the client's user id. The operator's account is id 1000 on every
node, and on the home server id 1000 is another account, which owns the backup repository.
- Eight containers bind paths on the pool locally. An export changes none of their paths or owners.
- No module exports or mounts anything, and no tool read fstab, mount units or exports.
## Considered Options
The full comparison is in research 038's [options](../01-RESEARCH/038-shares-and-mounts-the-mesh-manages/02-options.md).
Rejected:
1. **Mesh-written lines in `/etc/fstab`.** One file shared with the operator, and one line per
mount with everything in an option string. Rejected for a unit per mount.
2. **`autofs` with its own maps.** A second daemon doing what systemd's automount already does.
3. **Keep the ZFS `sharenfs` property and only open the port.** It keeps the export inside a property
of the operator's data (ADR 0051), and it has no room for squashing every client to the library's
owner.
4. **Export to the home network (the underlay).** Every device there can claim any user id, and the
underlay has no reach in the mesh yet (issue 154).
5. **Kerberos (`sec=krb5`) for real identities.** It needs a KDC and keytabs on every node, for shares
that have one owner.
6. **Every node mounts every share.** The control-node is the stably reachable machine the controller runs on, and would gain a writable path
into the library for no use. The laptop, behind a VPN client, would try a mount whenever something
looks at the path. Kept as a setting, not as the default.
7. **One `file-share` seat for NFS and SMB.** The two share an intent, not a contract (issue 169).
## Decision
1. **A node's share is a provision.** The module `nfs-server` holds the node seat `nfs-share` and
provides the provision `nfs-share`. Its assignment's settings list the shares: a name, the path
(an `access`, ADR 0051: the module never creates, chowns or removes it), and read-only or
read-write. It writes its own file in `/etc/exports.d`, never `/etc/exports`, and listens on 2049/tcp
from the private network. It serves NFS version 4 only, so no other port opens.
2. **Every export squashes every client to the share's owner.** `all_squash`, with `anonuid` and
`anongid` read from the exported path's owner on the server (`${access:<id>:uid}`), never written as
numbers. The clients are `${machine:mesh-range}` and nothing else. A file written from any node lands
owned by the library's owner, and no client acts as any other account on the server.
3. **The home server exports one share per folder, not the pool's root**: the library and the
downloads. The backup repository and the archive are not exported. (If the operator wants the pool's
root as one share, it is exported with `crossmnt` and the same squashing, which keeps the `0700`
backup repository closed. That is the alternative this record names, and the operator's to choose.)
4. **A node mounts through one module.** The module `mounts` holds the node seat `node-mounts`. For each
share it writes a `.mount` unit and an `.automount` unit, never an fstab line. Only the automount is
enabled, so nothing mounts at boot and nothing can fail the boot. Each mount has an idle timeout
(default 10 minutes) and a mount timeout (default 30 s; 10 s for an occasional source). A read-write
NFS share is mounted `hard`. A mount point is always an autofs mount, so nothing writes into the
empty directory under it (issue 169's data-loss case).
5. **Only the nodes that ask get a share.** A node mounts what its `mounts` assignment lists, plus every
share a module on that node requires. The list may be set once at the mesh-wide settings layer, which
is how "every node" is said, and a node's own layer can leave a share out. The proposal is to assign
`mounts` to the workstation and the laptop, and not to the control-node.
6. **An occasional source is a share with presence `occasional`.** A device that is not a node, such as
the gaming device or a USB disk, is described in the `mounts` assignment's settings. Its absence is a
state, `absent`, not a fault. It is mounted on access when it is there, and it never fails the boot or
leaves a failed unit. Mounting when a device appears on the network waits for issue 154 (mDNS is
dropped by the packet filter today).
7. **A credential for a share is an operator-delivered secret** (ADR 0092), accepted into the vault and
unsealed on the node as a file (ADR 0086), which the mount unit names. No password is written in a
repository, a manifest or `/etc/fstab`.
8. **Health:** a share that should always be reachable and is not, for 5 minutes, raises one plain
condition, `share-unreachable`, naming the share, the node and since when. An occasional source that
is absent raises nothing. The server's health is its service and its export table holding every
declared export.
9. **Over the bus.** The controller wires the two through the provision. `nfs-server` keeps its exports
and clients, and `mounts` keeps each share's state, in their own key-value buckets (ADR 0201). They
ask each other by request and reply. Neither reads the other's files (ADR 0260).
10. **Taking over a hand-written line is an act.** The `mounts` module's `adopt` comments out one fstab
line, keeping a copy, only after its own unit pair for that mount point is armed, and only on a
person's word. The ZFS `sharenfs` property is cleared the same way, by `nfs-server`, once its own
export is live.
## Consequences
- The workstation's library comes back over the private network, and the two failed units stop
appearing at boot.
- Two node seats join the mesh's closed set, `nfs-share` and `node-mounts`. That is a controller
change.
- What got harder: every client writes to a share as one account, so there is no per-person
permission on it. Traffic to the library crosses the private network's encryption even on the home
network; the workstation is on Wi-Fi, so the tunnel is not the limit there, and this is measured at
build time.
- The underlay's own devices lose the NFS share the property gave them. They have had no access since
2026-10-04 anyway, because the filter has closed it. Giving them access back is issue 154's question.
- Samba stays as found, and so does its closed port. An `smb-share` seat follows when something on the
mesh needs it.
## How it is checked
- The `mounts` module's `mounts_list` shows every mount point's source, its unit pair, its state and
which settings layer asked for it. `mounts_exports` shows the server's export table. A test in the
module renders the unit pair from a binding and asserts the options above.
- The lab composes a server and two clients and asserts: the export names only the private network's
range with `all_squash`; a file written from a client is owned by the share's owner; the client's
boot completes with the server stopped; and an occasional source that is absent leaves no failed unit.
- The controller's self-check counts fstab lines of type `nfs` or `cifs` on every node through
`mounts_list`, and reports any that are not adopted.
## References
- [Research 038](../01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md), its evidence and options
- [Issue 169](../04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md), [issue 154](../04-ISSUES/154-a-machines-own-network-is-not-a-reach/00-report.md), [issue 153](../04-ISSUES/153-an-adopted-machines-data-cannot-be-placed-where-it-is/00-report.md)
- [ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md), [ADR 0051](0051-shared-data-is-the-operators.md), [ADR 0086](0086-a-secret-reaches-a-process-as-a-file.md), [ADR 0092](0092-an-operator-delivers-a-pair-credential.md), [ADR 0201](0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md), [ADR 0240](0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md), [ADR 0260](0260-a-bar-block-shows-a-value-its-module-keeps-on-the-bus-and-no-module-reads-anothers-files.md)
- [52 — Shares and mounts](../03-DESIGN/01-to-be/52-shares-and-mounts.md)
03-DESIGN/01-to-be/52-shares-and-mounts.md
+145
View File
@@ -0,0 +1,145 @@
---
layer: to-be
status: proposed
code: []
updated: 2026-10-08
decisions:
- 02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md
- 02-DECISIONS/0051-shared-data-is-the-operators.md
- 02-DECISIONS/0092-an-operator-delivers-a-pair-credential.md
---
# 52 — Shares and mounts
**A node that shares a directory does it through one module, which provides the share as a provision
and opens its port to the private network only. A node that wants the files gets them through one
module, which writes a mount unit and an automount unit per share, so nothing mounts at boot and
nothing fails at boot. Only the nodes that ask get a share. A device that is not always there is a
share whose absence is a state, not a fault.**
([ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md),
from [research 038](../../01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md).)
A **share** is a directory one machine offers to others over the network. A **mount** is a share made
visible at a path on one node. An **occasional source** is a share on a device that comes and goes.
These three words are proposed for the glossary with this design.
## The parts
```
home server workstation, laptop (the nodes that ask)
nfs-server (seat nfs-share, node) mounts (seat node-mounts, node)
├ /etc/exports.d/<its file> ├ <path>.automount enabled, idle 10 min
│ one line per share, to the ├ <path>.mount not enabled, timeout 30 s
│ private network's range only, │ What = binding at:path, options
│ all_squash to the share's owner ├ credentials file (vault, ADR 0086),
├ nfs-server.service, NFS v4 only │ for an SMB source only
├ listens 2049/tcp from mesh ├ its process: probes each source,
└ its bucket: exports, clients │ keeps state in its bucket
▲ └ tools: mounts_list, mounts_exports,
│ provision nfs-share mount, unmount, test, adopt
│ (the controller wires it; ▲
│ binding: at, path, options) │ settings: shares wanted, occasional
└──────────────── request/reply on the bus ─────────┘ sources (mesh-wide and node layers)
gaming device (not a node) ── SMB share ──► an occasional source in the workstation's mounts settings
```
| part | where | does |
|---|---|---|
| seats `nfs-share` and `node-mounts` | the controller's closed set | node seats, capacity 1, with their verbs |
| `nfs-server` | the catalogue | holds `nfs-share`, provides the provision, writes its export file, runs the service, opens 2049 to the private network |
| `mounts` | the catalogue | holds `node-mounts`, writes a unit pair per share, keeps each share's state, serves the tools |
| the binding | the controller | gives a consumer the server's private-network name, the path and the options |
| the credential | the vault | an operator-delivered secret per SMB source, unsealed on the node as a file |
## 1. The server: `nfs-server`
- **Settings.** A list of shares, each with a name, the path as an `access` (the operator's data; the
module never creates, chowns or removes it), and `read-write` or `read-only`.
- **Its file.** One file in `/etc/exports.d`, written by the node-engine. One line per share:
the path, to `${machine:mesh-range}`, with `rw` or `ro`, `sync`, `secure`, `no_subtree_check`,
`all_squash`, and `anonuid` and `anongid` taken from the path's owner on the server
(`${access:<id>:uid}` and `:gid`). The pool's root is exported only if the operator asks, and then
with `crossmnt` as well.
- **The service.** `nfs-server.service`, running and enabled, restarted when the file changes.
NFS version 4 only, set in the module's drop-in for `nfs.conf`. So only 2049/tcp is needed, declared in
`listens` with `from: mesh`.
- **The provision.** `nfs-share`, one per share, with `path` and `options` in what it serves.
- **Health.** The service is active, and the kernel's export table holds every line of the module's
file.
- **Its bucket.** The exports as the kernel holds them, and the clients that have each mounted, with
since when.
- **Tools.** `exports` (r), `clients` (r), `reload` (a), and `test` (r): does this server export this
path to this node.
- **Taking over.** An act, `adopt`, clears a dataset's `sharenfs` property for a path the module now
exports, on a person's word, and says what it did.
## 2. The client: `mounts`
- **Settings.** `shares`: the names of the `nfs-share` provisions this node mounts, each with a mount
point and optional `read-only`. `sources`: occasional sources, each with a protocol (`smb` or `disk`),
a name or a disk's UUID, a share name, a mount point, and the name of its credential. Both may be set
at the mesh-wide layer, which is how "every node" is said, and changed at a node's own layer.
- **A share a module requires.** When a module on the node requires `nfs-share`, the controller adds that
share to the node's `mounts` set, with the mount point that module's assignment gives.
- **The unit pair.** For each share, a `.mount` and an `.automount`, named by the escaped mount point and
written in `/etc/systemd/system` by the node-engine:
- the `.automount` is enabled under `remote-fs.target`, with `TimeoutIdleSec=10min`;
- the `.mount` is not enabled; it has `TimeoutSec=30s` (10 s for an occasional source), `_netdev`,
and an order after the private network's link;
- for NFS: `vers=4.2`, `hard,timeo=150,retrans=3` when read-write, `soft` allowed when read-only;
- for SMB: `credentials=<the unsealed file>`, `uid` and `gid` of the node's account that uses it;
- for a disk: `BindsTo=` the disk's device unit, so the mount goes with the disk.
- **States**, kept in its bucket per share, with since when: `armed` (automount waiting), `mounted`,
`absent` (an occasional source not there), `unreachable` (should be there and is not), `failed`
(the mount was tried and refused, with the error).
- **Its process.** Probes each source's port every 30 s, and the device unit for a disk. It never
touches a mount point, so a hung server cannot hang it.
- **Health.** Every automount is armed, and every share that is not occasional is `armed` or `mounted`.
An occasional source that is `absent` is healthy.
- **Tools.** `mounts_list` (r), `mounts_exports` (r), `test` (r), `mount` and `unmount` (a), `adopt` (a).
## 3. Which nodes get which share
| node | gets | why |
|---|---|---|
| home server | none; it holds `nfs-server` | the files are local; the media modules keep their local bind mounts |
| workstation | the library and the downloads, read-write; the gaming device as an occasional source | the operator works on the library there |
| laptop | the library, read-only | it travels; the automount means it tries only when something looks |
| control-node | none | the stably reachable machine the controller runs on; it has no use for the library |
## 4. Conditions
| condition | raised when | cleared when |
|---|---|---|
| `share-unreachable` (warning) | a share that is not occasional has been `unreachable` or `failed` for 5 minutes | the next probe reaches it |
| the server's health | the export table lacks a declared line, or the service is down | the next look finds both right |
Each says the share, the node and since when: *"The home server's share `library` is not reachable
from the workstation since 19:21."* An absent occasional source raises nothing; `mounts_list` says
*"the share `retro-games` is not present since 18:02"*.
## 5. Migration
1. Build and deliver `mounts` (read-only first, already proposed) and `nfs-server`, and add the two seats.
2. Assign `nfs-server` to the home server with the library and the downloads. Check `exports` and
`test` from the workstation.
3. `adopt` on the home server clears the library's `sharenfs` property. The underlay range leaves the
export table; the private network's range is in it.
4. Assign `mounts` to the workstation. Its unit pair for the library takes precedence over the unit
generated from fstab. Check `mounts_list` and a file written from the workstation.
5. `adopt` on the workstation comments out the library's fstab line, keeping a copy of the file.
6. Accept the gaming device's credential into the vault with `secret accept`, add the gaming device as an
occasional source, and `adopt` its fstab line.
7. Assign `mounts` to the laptop, read-only.
Every `adopt` is a person's act. Nothing in this migration touches the library's files, owners or
modes, or the media modules' bind mounts.
## Not in this design
- **Samba.** Kept as found. An `smb-share` seat follows when a node needs it.
- **Clients on the underlay.** NFS for devices that are not nodes waits on issue 154.
- **Mounting when a device appears.** Waits on a local-link reach (issue 154), because the packet
filter drops mDNS. The automount already mounts on the next access.
03-DESIGN/01-to-be/README.md
+1
View File
@@ -51,6 +51,7 @@ document is written and this one's status becomes `implemented`.
| [`49-the-mesh-in-domains.md`](49-the-mesh-in-domains.md) | **In progress.** Every concept belongs to one of ten domains, which owns its one word; the glossary is organised by them and is the authority, a retired word is named on its replacement's line with its scope, and two checks hold this repository's documents and the catalogue's tool descriptions to it | [ADR 0244](../../02-DECISIONS/0244-the-mesh-is-described-in-domains-and-one-word-names-one-thing.md), [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md), [ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md) |
| [`50-split-dns-on-a-machine-with-a-vpn-client.md`](50-split-dns-on-a-machine-with-a-vpn-client.md) | **In progress.** A machine whose VPN client pushes servers of its own runs a resolver of its own, on a node seat, that routes the VPN's domains over its link and every other name to the mesh's resolvers; the VPN client's module carries the adapter | [ADR 0247](../../02-DECISIONS/0247-a-machine-with-a-vpn-client-routes-names-by-domain-through-a-resolver-of-its-own.md), [ADR 0223](../../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md) |
| [`51-the-registries-keep-what-is-named.md`](51-the-registries-keep-what-is-named.md) | **In progress.** The artifact store, the package registry and each machine's container runtime get tools that say what they hold, what names each thing and what could go; every removal is a dry run unless a person asks and says why, and what decides is always a record | [ADR 0251](../../02-DECISIONS/0251-the-registries-say-what-they-hold-and-keep-what-is-named-and-a-machine-keeps-the-image-it-runs-and-the-one-before.md), [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md) |
| [`52-shares-and-mounts.md`](52-shares-and-mounts.md) | **Proposed.** A node shares a directory through `nfs-server`, which provides it as a provision to the private network only, squashing every client to the share's owner; a node that asks mounts it through `mounts`, a mount unit and an automount per share, so nothing mounts or fails at boot; a device that comes and goes is an occasional source whose absence is a state | [ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md), [ADR 0051](../../02-DECISIONS/0051-shared-data-is-the-operators.md), [ADR 0092](../../02-DECISIONS/0092-an-operator-delivers-a-pair-credential.md) |
## Not yet written