Merge pull request 'ADR 0263 (accepted), research 038, design 52: a share is a provision, a node mounts it on demand' (#221) from feat/mounts-module into main
This commit was merged in pull request #221.
This commit is contained in:
+8
-1
@@ -224,7 +224,7 @@ and Health and repair. **Decided by** ADR 0100, 0126, 0176, 0181, 0207, 0221, 02
|
||||
|
||||
**Purpose.** To resolve which module serves a provision for which consumer, and to wire the two.
|
||||
**Recorded by** the controller. **Upstream of** Data; downstream of Module, Placement and Identity.
|
||||
**Decided by** ADR 0027, 0084, 0138, 0225, 0230.
|
||||
**Decided by** ADR 0027, 0084, 0138, 0225, 0230, 0263.
|
||||
**Uses:** module, seat, credential (Identity and access).
|
||||
|
||||
- **provision** — a service one module `provides` and others `require`; the mesh resolves a provider
|
||||
@@ -239,6 +239,13 @@ and Health and repair. **Decided by** ADR 0100, 0126, 0176, 0181, 0207, 0221, 02
|
||||
reaches ([ADR 0138](../02-DECISIONS/0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md)).
|
||||
- **retire** — a provider stops serving a consumer the mesh no longer asks for; what it kept is deleted
|
||||
only by a person ([ADR 0230](../02-DECISIONS/0230-a-consumer-the-mesh-stops-asking-for-is-retired-and-deleted-only-by-a-person.md)).
|
||||
- **share** — a folder one node offers to the mesh's other nodes over the private network: the provision
|
||||
`nfs-share`, provided by the holder of the node seat `node-nfs-server`, one share per folder
|
||||
([ADR 0263](../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md)).
|
||||
- **mount** — a share, or an occasional source, made visible at a path on one node by the holder of the
|
||||
node seat `node-mounts`, as a mount unit and an automount unit, mounted on access (ADR 0263).
|
||||
- **occasional source** — a share on a device that is not a node and comes and goes, such as a USB disk;
|
||||
its absence is a state, *not present*, never a fault (ADR 0263).
|
||||
|
||||
## Identity and access — who may do what
|
||||
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
status: graduated
|
||||
initiated: 2026-10-08
|
||||
touches:
|
||||
- 04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md
|
||||
- 04-ISSUES/154-a-machines-own-network-is-not-a-reach/00-report.md
|
||||
- 01-RESEARCH/027-the-system-layer-as-modules/02-candidates-and-questions.md
|
||||
- 03-DESIGN/01-to-be/26-the-seats.md
|
||||
- 03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md
|
||||
- 02-DECISIONS/0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md
|
||||
- 02-DECISIONS/0051-shared-data-is-the-operators.md
|
||||
- 02-DECISIONS/0092-an-operator-delivers-a-pair-credential.md
|
||||
- the catalogue's module manifests
|
||||
- the controller's seat set
|
||||
became:
|
||||
- 03-DESIGN/01-to-be/52-shares-and-mounts.md
|
||||
- 02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md
|
||||
---
|
||||
|
||||
# 038 — Shares and mounts the mesh manages
|
||||
|
||||
## What is investigated
|
||||
|
||||
How the mesh shares a directory from one node and mounts it on the others, so that no machine
|
||||
carries a hand-written export or a hand-written mount line. Three kinds of source are in scope:
|
||||
|
||||
1. **A node's share.** The home server shares its storage pool over NFS. A node that wants the files
|
||||
mounts them.
|
||||
2. **A device outside the mesh that is not always there.** A gaming device on the home network shares
|
||||
a folder over SMB, and only while it is switched on. A USB disk is the same case.
|
||||
3. **What the mesh should show and say** when a mount works, when it does not, and when its source is
|
||||
simply away.
|
||||
|
||||
## Why
|
||||
|
||||
On 2026-10-08 the operator found two failed mounts on the workstation, and said two things:
|
||||
|
||||
- about the media library: *"This should be handled by the mesh. If a node exposes an NFS share, it
|
||||
should get configured on all the nodes in their fstab?"*
|
||||
- about the gaming device's share: *"This mount only works if I plug in my recalbox machine."*
|
||||
|
||||
They also set a requirement: **the home server shares its storage pool over NFS**, the workstation's
|
||||
library mount comes from that share, the export reaches only the mesh's private network, it squashes
|
||||
root, and the media modules' data (the Plex library is sacred) must not be put at risk.
|
||||
|
||||
[Issue 169](../../04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md)
|
||||
predicted the first failure on 2026-09-30: nothing declares the share's ports, so the packet filter
|
||||
closes them. This effort measures what happened, and proposes the decision issue 169 asked for.
|
||||
|
||||
## What was found
|
||||
|
||||
The detail is in [01-evidence.md](01-evidence.md). In short:
|
||||
|
||||
1. **The library mount times out because the home server's packet filter drops NFS.** The server
|
||||
exports the library and its NFS service runs. The mesh's filter accepts only what a module
|
||||
`listens` on, and no module listens on 2049, 111 or 445. The server's NFS counters read zero I/O
|
||||
since its last boot six days earlier, so no client has used either share since then.
|
||||
2. **The gaming device's mount fails because the gaming device is off.** It is an SMB share on the device itself,
|
||||
not a disk. The fstab line has no `nofail` and no automount, so each boot records a failed unit.
|
||||
The line also carries the share's password in a world-readable file.
|
||||
3. **The export is not where a person would look.** The server's `/etc/exports` holds only comments.
|
||||
The export comes from a property of the ZFS dataset, which writes its own file in `/etc/exports.d`.
|
||||
The library is a child dataset of the pool, so an export of the pool's root does not show the
|
||||
library's files unless it crosses into child filesystems.
|
||||
4. **User ids do not match across nodes.** The library is owned by a service account and group. The
|
||||
operator's account has the same id (1000) on every node, but on the home server that id belongs to
|
||||
a different account, which owns the pool's backup repository. NFS with `sec=sys` trusts the client's
|
||||
id. A plain export of the whole pool would give the workstation's operator account the backup
|
||||
repository.
|
||||
5. **No module, and until now no tool, covers any of this.** No module in either catalogue exports or
|
||||
mounts anything. No mesh tool read a machine's fstab, its mount units or its exports, so this effort
|
||||
read them through the login shell. A read-only `mounts` module now proposes those tools.
|
||||
|
||||
## Where it goes
|
||||
|
||||
[02-options.md](02-options.md) weighs the options. The decision is
|
||||
[ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md),
|
||||
and the design is [52 — Shares and mounts](../../03-DESIGN/01-to-be/52-shares-and-mounts.md).
|
||||
|
||||
On 2026-10-08 the operator chose **one share per folder**, so the backups are never exported, and
|
||||
**mesh machines only**, over the private network. Which nodes mount a share, and whether the laptop
|
||||
mounts read-only, is decided at assignment.
|
||||
@@ -0,0 +1,134 @@
|
||||
# 038 — Evidence
|
||||
|
||||
Measured on 2026-10-08 on the four nodes, read only, through the mesh's tools: `node-service-manager`
|
||||
(`failed`, `status`, `journal`), `node-uplink.links`, `node-packet-filter.rules`,
|
||||
`disk-load.disk_filesystems`, `docker.docker_list`, and `node-login-shell.execute` for the files no
|
||||
tool read (§7). Addresses are given in documentation ranges: the home network (the underlay) as
|
||||
`192.0.2.0/24`. Node roles: the **home server**, the **workstation**, the **laptop**, the **control-node**.
|
||||
|
||||
## 1. The workstation's two failed mounts
|
||||
|
||||
`node-service-manager.failed` on the workstation lists two failed system units, both generated from
|
||||
`/etc/fstab` (their unit file is under `/run/systemd/generator`, and `mesh_declared` is false):
|
||||
|
||||
| unit | the fstab line (password removed) | what the journal says |
|
||||
|---|---|---|
|
||||
| the library | `<home server>:<library> <same path> nfs defaults,_netdev 0 0` | `Mounting timed out. Terminating.` 90 s after boot, at every boot since 2026-10-04; the last successful mount was at the boot of 2026-08-23 |
|
||||
| the gaming device | `//<device>/Share <mountpoint> cifs username=root,password=…,iocharset=utf8,sec=ntlmssp 0 0` | `mount error(113): could not connect to <device>` at every boot |
|
||||
|
||||
Neither line has `nofail` or `x-systemd.automount`. Both name the source by its underlay address.
|
||||
`remote-fs.target` is inactive on the workstation, because both of its members failed.
|
||||
|
||||
## 2. Why the library times out
|
||||
|
||||
- The home server exports the library, and its NFS and Samba services run. `nfs-server` and `smb`
|
||||
are active and enabled, and ports 2049, 111 and 445 are listening on every address.
|
||||
- **The mesh's packet filter drops them.** The mesh's table on the home server has `policy drop` on
|
||||
its input chain for the underlay link and the private network link. It accepts a port only where a
|
||||
module `listens` on it ([ADR 0045](../../02-DECISIONS/0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md)).
|
||||
No rule names 2049, 111 or 445.
|
||||
- From the workstation, a TCP connection to the home server's port 2049 fails, and one to port 22 on
|
||||
the same address succeeds.
|
||||
- The home server's NFS counters (`/proc/net/rpc/nfsd`) read `io 0 0` since its boot on 2026-10-04.
|
||||
No established connection to 2049 or 445 exists. **Nobody has used either share since that boot.**
|
||||
|
||||
This is the failure [issue 169](../../04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md)
|
||||
predicted: the shares close silently once the filter is the sum of what modules listen on.
|
||||
|
||||
## 3. What the home server exports, and from where
|
||||
|
||||
- `/etc/exports` holds only the distribution's comments; it was last written on 2026-09-20.
|
||||
- The export comes from ZFS. The library dataset has `sharenfs=rw=@192.0.2.0/24` and `sharesmb=on`.
|
||||
ZFS writes `/etc/exports.d/zfs.exports` itself, marked *do not edit*. The kernel's export table
|
||||
(`/var/lib/nfs/etab`) holds one entry: the library, to the underlay range, `rw,sync,root_squash,
|
||||
no_all_squash,secure,no_subtree_check,sec=sys`.
|
||||
- The Samba configuration has one share, `media`, on the library, `valid users = media`,
|
||||
`force user = media`, writable.
|
||||
- **The pool is two filesystems.** The pool is mounted at its root, and the library is a child dataset
|
||||
mounted under it. An NFS export of the pool's root does not show the library's files unless the
|
||||
export crosses into child filesystems (`crossmnt`); without it, a client sees an empty directory
|
||||
where the library should be.
|
||||
|
||||
## 4. What lives on the pool, and who uses it
|
||||
|
||||
The pool's root holds five directories:
|
||||
|
||||
| directory | owner | what it is |
|
||||
|---|---|---|
|
||||
| the library (child dataset, about 90 TiB used) | the media account and group, `0775` | movies, series, anime, music, audiobooks and more |
|
||||
| downloads | the media account and group | the download clients' and the library managers' working area |
|
||||
| archive | root | the operator's archive |
|
||||
| a second Plex folder | the operator's account on the home server | an older library folder |
|
||||
| the backup repository | the operator's account on the home server, `0700` | the restic repository the backup seat writes |
|
||||
|
||||
Plex's own configuration and database live on a separate disk, not on the pool.
|
||||
|
||||
Eight running containers bind paths on the pool, all of them as plain bind mounts of local paths:
|
||||
Plex (the library's folders, all read-only), the library managers for series, movies, music and
|
||||
subtitles, two download clients, and one automation app. **Exporting the pool over NFS changes none of
|
||||
these paths and none of their owners.** An export adds a way in; it does not move or chown anything.
|
||||
The only way an export endangers this data is by what it lets a client do (§5).
|
||||
|
||||
## 5. User and group ids across the nodes
|
||||
|
||||
| node | id 1000 | the media account | the media group |
|
||||
|---|---|---|---|
|
||||
| home server | a separate account, which owns the backup repository and the older Plex folder | exists, owns the library | exists |
|
||||
| workstation | the operator | exists, same ids as on the server | exists; the operator is a member |
|
||||
| laptop | the operator | absent | absent |
|
||||
| control-node | the operator | absent | absent |
|
||||
|
||||
The library managers on the home server run as the media account and group, set by their `puid` and
|
||||
`pgid` settings.
|
||||
|
||||
With `sec=sys`, the server believes the id a client sends; `root_squash` changes only root. So:
|
||||
|
||||
- **A client's operator account (1000) is the home server's other account (1000).** An export of the
|
||||
pool's root would let it read and write the backup repository and the older Plex folder.
|
||||
- **A file written by the laptop's operator lands owned by 1000**, not by the media account, and the
|
||||
library managers may then fail to rename or delete it.
|
||||
- A device on the underlay that is not a node can claim any id. That is a reason to export to the
|
||||
private network only, where every address is a node.
|
||||
|
||||
## 6. The gaming device's share
|
||||
|
||||
- The gaming device is a separate machine on the underlay, a retro-gaming device that shares a folder named
|
||||
`Share` over SMB. It is not a USB disk. The router's DNS knows its name. It was switched off during
|
||||
the survey and did not answer a ping.
|
||||
- The workstation has `cifs-utils`, `avahi` and `nss-mdns`, and its `hosts:` line contains
|
||||
`mdns4_minimal`. **The packet filter drops inbound mDNS** on every node, as the `avahi` module's
|
||||
README already records. So discovery by mDNS cannot see the gaming device arrive today. Waiting for it
|
||||
would first need a local-link reach for `listens`, which is the question of
|
||||
[issue 154](../../04-ISSUES/154-a-machines-own-network-is-not-a-reach/00-report.md).
|
||||
- The share's password is written in `/etc/fstab`, which is mode `0644`. Every account on the
|
||||
workstation can read it.
|
||||
|
||||
## 7. What the catalogue and the tools cover
|
||||
|
||||
- **No module exports or mounts.** No manifest among the 117 modules of the core catalogue and the 12
|
||||
of the media catalogue writes an export, a Samba share, a mount unit or an fstab line.
|
||||
- **`disk-load.disk_filesystems`** lists NFS and CIFS filesystems that are mounted now, with their space.
|
||||
It cannot see a mount that failed, an fstab line, a mount unit or an export.
|
||||
- **`node-service-manager`** shows a mount unit's state and journal, but not what it mounts or with
|
||||
which options.
|
||||
- **No tool read a machine's fstab, its mount and automount units with their sources, or its NFS and
|
||||
SMB exports.** This effort read them with `node-login-shell.execute`, which runs a shell command as
|
||||
the operator account. That is a work-around for a missing tool, so a read-only `mounts` module with
|
||||
two tools, `mounts_list` and `mounts_exports`, was opened in the catalogue alongside this effort. It
|
||||
redacts passwords in mount options and never touches a network mount point, so a hung server cannot
|
||||
hang the tool.
|
||||
|
||||
## 8. Prior work
|
||||
|
||||
- **Issue 169** (open, the operator's proposal of 2026-09-30): two seats, `nfs-share` and `smb-share`,
|
||||
because the two protocols share an intent and not a contract; a consumer module that writes a
|
||||
systemd `.mount` unit from its binding; the modules that use the files never learn about NFS; the
|
||||
data-loss case of a writer starting before its mount; and identity crossing the wire.
|
||||
- **Research 027**, option M2: each client module writes its own systemd mount and automount unit, and
|
||||
`/etc/fstab` stays the machine's.
|
||||
- **Research 026**, the tools list: `exports`, `shares`, `clients`, `reload` on the server;
|
||||
`mounts`, `mount`, `unmount`, `test` on the client.
|
||||
- **ADR 0092**: a credential for something outside the mesh is an operator-delivered secret, accepted
|
||||
into the vault (`secret accept`) and never replaced by a made one.
|
||||
- **`${machine:mesh-range}`** already exists as a placeholder in two manifests, so an export line can
|
||||
name the private network's range without naming a node.
|
||||
@@ -0,0 +1,146 @@
|
||||
# 038 — Options
|
||||
|
||||
Each question below lists the options weighed and the one recommended. The recommendations together
|
||||
became [ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md).
|
||||
|
||||
## Q1. Who writes the export
|
||||
|
||||
| option | for | against |
|
||||
|---|---|---|
|
||||
| A. Keep the ZFS `sharenfs` property, and have a module declare only the ports | nothing new on disk | the export lives in a dataset property, beside the operator's data; a module would be setting properties on data the mesh does not own (ADR 0051); the options are one string with no room for squashing |
|
||||
| B. A module owns `/etc/exports` whole | one file, one owner | the file is the machine's; a second exporter on the same node would collide |
|
||||
| **C. A module, `nfs-server`, writes its own file in `/etc/exports.d`, and the ZFS property is turned off when the module's export is live** | the mesh owns exactly its file; the distribution reads the directory; unassigning removes the file | a migration step: the property must be cleared by an act, not by hand |
|
||||
|
||||
**Recommended: C.** The module holds a node seat, `nfs-share`, so one module exports on a node, and it
|
||||
provides the provision `nfs-share` (issue 169's proposal). Samba is left as found, with its own seat
|
||||
`smb-share` later; nothing on the mesh uses it, and the gaming device's SMB share is a client case (Q6).
|
||||
|
||||
## Q2. What is exported, and to whom
|
||||
|
||||
The operator asked for the pool's root. Exporting it plainly has two problems (evidence §3 and §5): the
|
||||
library is a child filesystem, so the export needs `crossmnt` to show it; and with `sec=sys`, any client
|
||||
account with id 1000 becomes the server's own 1000, which owns the backup repository.
|
||||
|
||||
| option | for | against |
|
||||
|---|---|---|
|
||||
| A. The pool's root, `rw,root_squash`, to the underlay range | what the operator literally asked; what exists today (but for the library only) | gives the backup repository to every client's operator account; open to every device on the underlay, which can claim any id |
|
||||
| B. The pool's root, `rw,crossmnt,all_squash`, every client mapped to the media account, to the private network only | one share, as asked; every file written lands owned by the media account; the `0700` backup repository is unreadable to the media account, so it stays closed | every client writes as one account, so there is no per-person permission on the share; top-level folders other than the library are writable by every client |
|
||||
| **C. One share per subtree: the library and the downloads, each `rw,all_squash` to the media account, to the private network only; the backup repository and the archive not exported** | the backup repository is not reachable at all; ownership is right; a consumer mounts only what it needs | two shares instead of one; a later folder needs a new share entry (one setting) |
|
||||
| D. Kerberos (`sec=krb5`) with id mapping | real per-person identity | needs a KDC and keytabs on every node; far beyond the problem |
|
||||
|
||||
**Recommended: C, with B as the operator's alternative if one share is wanted.** Both use
|
||||
`all_squash` with `anonuid` and `anongid` set to the owner of the exported path, read on the server
|
||||
(`${access:<id>:uid}`, from issue 153's fix), never written as numbers in a manifest. Both export to
|
||||
`${machine:mesh-range}` only, so every client is a node, reached over the private network. `root_squash`
|
||||
is implied by `all_squash`. `sync`, `secure`, `no_subtree_check` stay as they are today. The server
|
||||
serves NFS version 4 only, so only port 2049 is opened, `from: mesh`; 111 and the mountd ports stay
|
||||
closed.
|
||||
|
||||
**What this does not touch:** the local bind mounts of Plex and the library managers (evidence §4),
|
||||
the datasets' owners and modes, and the Samba share.
|
||||
|
||||
## Q3. How a node mounts a share
|
||||
|
||||
| option | for | against |
|
||||
|---|---|---|
|
||||
| A. The mesh writes lines into `/etc/fstab` | the file people know | one shared file and one ordered list; the node-engine can place a marked block but every line competes with the operator's |
|
||||
| B. `autofs` with its own maps | mature automounter | a second daemon doing what systemd already does |
|
||||
| **C. A `.mount` unit and an `.automount` unit per share, written by one module, `mounts`, which holds the node seat `node-mounts`** | a unit is a unit: its state, its journal, its health; an automount never holds the boot; unassigning removes it; no shared file | units have to be named by the escaped mount path, which the module computes |
|
||||
|
||||
**Recommended: C.** The `.automount` is enabled under `remote-fs.target`; the `.mount` is not enabled
|
||||
on its own, so nothing mounts at boot and nothing fails at boot. The mount is made at first access and
|
||||
released after an idle time. The options carried:
|
||||
|
||||
- `x-systemd.automount` semantics through the `.automount` unit, `TimeoutIdleSec` (default 10 minutes);
|
||||
- a mount timeout, `TimeoutSec` on the `.mount` (default 30 s for a node's share, 10 s for an
|
||||
occasional one), instead of systemd's 90 s;
|
||||
- `_netdev` and an order after the private network's link is up;
|
||||
- for NFS, `vers=4.2,hard,timeo=150,retrans=3` on a read-write share (no silent write loss) and
|
||||
`soft` allowed for a read-only one, as a setting.
|
||||
|
||||
An automount also closes issue 169's data-loss case: the mount point is always an autofs mount, so a
|
||||
process that writes there either triggers the mount or gets an error. It never writes into the empty
|
||||
directory underneath.
|
||||
|
||||
## Q4. Which nodes get a share
|
||||
|
||||
The operator suggested every node. Weighed:
|
||||
|
||||
| option | for | against |
|
||||
|---|---|---|
|
||||
| A. Every node mounts every share | no assignment to make | the control-node is the stably reachable machine the controller runs on, and gains a writable path into the library for no use; the laptop, behind a corporate VPN client, would try a mount whenever a file manager looks at the path |
|
||||
| B. A node mounts a share only when a module on it requires it | precise | a person who wants the files in their file manager has no module that requires them |
|
||||
| **C. A node mounts what its `mounts` assignment lists; the list may be set once at the mesh-wide layer** | opt-in per node, and "every node" is one setting at the mesh-wide layer, with a node's own layer to leave a share out | two layers to read when asking why a node has a share; `mounts_list` says which layer gave it |
|
||||
|
||||
**Recommended: C, assigned to the workstation and the laptop, not to the control-node.** A module that needs
|
||||
the files (a player, an indexer) requires `nfs-share` as well; the controller then adds the share to
|
||||
that node's `mounts` unit set, so B is the special case of C, not a second way.
|
||||
|
||||
## Q5. Health
|
||||
|
||||
| case | finding | condition |
|
||||
|---|---|---|
|
||||
| a node's share, automount armed, server answers | healthy | none |
|
||||
| a node's share, server's 2049 unreachable from this node for 5 minutes | not healthy | `share-unreachable`, a warning: *"The home server's share `library` is not reachable from the workstation since 19:21."* |
|
||||
| a node's share, the mount failed with an error other than a timeout | not healthy | the same condition, with the error |
|
||||
| an occasional source, absent | **absent**, a state, not a finding | none |
|
||||
| an occasional source, present but the mount fails | not healthy | `share-unreachable` |
|
||||
| the server: the export table lacks a declared export, or `nfs-server` is down | not healthy | the module's ordinary health condition |
|
||||
|
||||
The `mounts` module's code probes each source's port on a timer and keeps each share's state in its
|
||||
own key-value bucket (ADR 0201): `mounted`, `armed`, `absent`, `unreachable`, with since when. Its
|
||||
health reads that bucket. Nobody reads its files.
|
||||
|
||||
## Q6. Occasional sources
|
||||
|
||||
The gaming device is not a node, so it cannot hold a module or provide a provision. Weighed:
|
||||
|
||||
| option | for | against |
|
||||
|---|---|---|
|
||||
| A. A fstab line with `nofail,x-systemd.automount` | small | a hand-written line again, and the password stays in the file |
|
||||
| **B. An external source in the `mounts` assignment's settings: protocol, name, share, mount point, presence `occasional`, and the name of a credential** | the same unit pair and tools as a node's share; the password goes to the vault | a share with no provider, so no binding: the module checks the name itself |
|
||||
| C. Wait for the device by mDNS and mount when it appears | mounts before anyone asks | the packet filter drops mDNS today (evidence §6); depends on issue 154 |
|
||||
|
||||
**Recommended: B now, C when issue 154 gives `listens` a local-link reach.** With an automount, nothing
|
||||
has to happen when the gaming device appears: the next access mounts it. The probe only keeps the state
|
||||
honest, so `mounts_list` says *"the share `retro-games` is not present since 18:02"*. A USB disk is the same kind of source,
|
||||
with presence taken from its device unit (`BindsTo=` on the disk's by-uuid device) instead of a probe.
|
||||
|
||||
**The credential:** the share's password is an operator-delivered secret, accepted into the vault with
|
||||
`secret accept` (ADR 0092), and reaches the node as a file the node-engine unseals (ADR 0086). The unit
|
||||
names it as `credentials=<that file>`. Nothing in a repository or in `/etc/fstab` holds it.
|
||||
|
||||
## Q7. Taking over the existing fstab lines
|
||||
|
||||
The two lines are the operator's. The node-engine's own `.mount` unit in `/etc/systemd/system` already
|
||||
takes precedence over the generated unit of the same name, but the line would still be read by every
|
||||
other tool and person.
|
||||
|
||||
**Recommended:** the `mounts` module gets an act, `adopt`, that for one mount point:
|
||||
|
||||
1. checks a unit pair the module placed is armed for that mount point, and refuses if not;
|
||||
2. comments out the fstab line, with a marker naming the module and the date, and keeps a copy of the
|
||||
file beside it;
|
||||
3. reloads the service manager and says what it did.
|
||||
|
||||
It runs only on a person's word. The library line is adopted after the home server's share works over
|
||||
the private network; the gaming device's after its credential is accepted.
|
||||
|
||||
## Q8. Over the bus
|
||||
|
||||
Every exchange between the modules is over NATS (research 036):
|
||||
|
||||
- the controller wires provider and consumer through the provision; the binding carries `at` (the
|
||||
server's private-network name), `path` and `options`;
|
||||
- `nfs-server` keeps its exports and the clients it sees in its bucket, and answers `test` (does it
|
||||
export this path to this node) by request and reply;
|
||||
- `mounts` keeps each share's state in its bucket; a view or the controller watches it;
|
||||
- no module reads another's files (ADR 0260).
|
||||
|
||||
## The operator's answers (2026-10-08)
|
||||
|
||||
1. **The pool's root, or one share per folder?** One share per folder (Q2 option C), so the backup
|
||||
repository is never exported.
|
||||
2. **Underlay clients?** Mesh machines only, over the private network.
|
||||
3. **Every node, or the nodes that ask? Read-write from the laptop?** Decided at assignment, which is the
|
||||
operator's; the recommendation (Q4 option C; the laptop read-only) stands in the design.
|
||||
+163
@@ -0,0 +1,163 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-10-08
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0051-shared-data-is-the-operators.md
|
||||
---
|
||||
|
||||
# 263. A share is a provision, a node mounts it on demand through one module, and only the nodes that ask get it
|
||||
|
||||
## Context
|
||||
|
||||
On 2026-10-08 two mounts on the workstation had failed at every boot since 2026-10-04
|
||||
([research 038](../01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md), evidence §1–§2):
|
||||
|
||||
- **The media library over NFS timed out** after 90 s. The home server exports it and its NFS service
|
||||
runs, but the mesh's packet filter drops port 2049, because no module listens on it (ADR 0045). The
|
||||
server's NFS counters read zero I/O since its boot six days earlier. This is the silent closure
|
||||
[issue 169](../04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md)
|
||||
predicted on 2026-09-30.
|
||||
- **A gaming device's SMB share failed** with "host unreachable", because the gaming device was off. The
|
||||
fstab line has no `nofail` and no automount, and it holds the share's password in a file every
|
||||
account can read.
|
||||
|
||||
The operator asked that the mesh handle this: a node that shares a directory should have it configured
|
||||
on the nodes that use it. They also required that the home server share its storage pool over NFS, to
|
||||
the private network only, with root squashed, and without risk to the media modules' data.
|
||||
|
||||
What was measured bears on the shape (evidence §3–§5):
|
||||
|
||||
- The export comes from a ZFS dataset property, not from `/etc/exports`.
|
||||
- The library is a child filesystem of the pool, so an export of the pool's root shows an empty
|
||||
directory there unless it crosses filesystems.
|
||||
- With `sec=sys` the server believes the client's user id. The operator's account is id 1000 on every
|
||||
node, and on the home server id 1000 is another account, which owns the backup repository.
|
||||
- Eight containers bind paths on the pool locally. An export changes none of their paths or owners.
|
||||
- No module exports or mounts anything, and no tool read fstab, mount units or exports.
|
||||
|
||||
## Considered Options
|
||||
|
||||
The full comparison is in research 038's [options](../01-RESEARCH/038-shares-and-mounts-the-mesh-manages/02-options.md).
|
||||
Rejected:
|
||||
|
||||
1. **Mesh-written lines in `/etc/fstab`.** One file shared with the operator, and one line per
|
||||
mount with everything in an option string. Rejected for a unit per mount.
|
||||
2. **`autofs` with its own maps.** A second daemon doing what systemd's automount already does.
|
||||
3. **Keep the ZFS `sharenfs` property and only open the port.** It keeps the export inside a property
|
||||
of the operator's data (ADR 0051), and it has no room for squashing every client to the library's
|
||||
owner.
|
||||
4. **Export to the home network (the underlay).** Every device there can claim any user id, and the
|
||||
underlay has no reach in the mesh yet (issue 154).
|
||||
5. **Kerberos (`sec=krb5`) for real identities.** It needs a KDC and keytabs on every node, for shares
|
||||
that have one owner.
|
||||
6. **Every node mounts every share.** The control-node is the stably reachable machine the controller runs on, and would gain a writable path
|
||||
into the library for no use. The laptop, behind a VPN client, would try a mount whenever something
|
||||
looks at the path. Kept as a setting, not as the default.
|
||||
7. **One `file-share` seat for NFS and SMB.** The two share an intent, not a contract (issue 169).
|
||||
8. **The pool's root as one share**, with `crossmnt` and every client mapped to the library's owner. That
|
||||
keeps the `0700` backup repository closed to clients, but it still exports it, and every other folder
|
||||
at the root becomes writable from every node. The operator chose one share per folder on 2026-10-08,
|
||||
so the backups are never exported.
|
||||
9. **A provider's paths written into its binding.** A binding carries what the provider's manifest
|
||||
`serves`, which is the same for every machine; a share's path is a setting of one assignment. The
|
||||
consumer reads it from the server's bucket on the bus instead, so no controller change is needed for
|
||||
it.
|
||||
|
||||
## Decision
|
||||
|
||||
1. **A node's share is a provision.** The module `nfs-server` holds the node seat `node-nfs-server` and
|
||||
provides the provision `nfs-share`. Its assignment's setting `shares` lists the shares: a name, the
|
||||
path, and read-only or read-write. The path is the operator's data (ADR 0051): the module never
|
||||
creates, chowns, chmods or removes anything under it. It writes its own file in `/etc/exports.d`, never `/etc/exports`, and listens on 2049/tcp
|
||||
from the private network. It serves NFS version 4 only, so no other port opens.
|
||||
2. **Every export maps every client to the share's owner.** `all_squash`, with `anonuid` and `anongid`
|
||||
read by the module from the exported folder's owner on the server (after following any symbolic link
|
||||
to the real folder), never written as numbers. A share owned by root is refused, so no client is ever
|
||||
mapped to root. A file written from any node lands owned by the folder's owner. **What this does not
|
||||
promise:** a client acts as that owner, so it can do whatever the owner may do with what it reaches.
|
||||
Where a share is a folder inside a larger filesystem, NFS cannot fully confine a client that forges
|
||||
file handles to that folder; it can reach the rest of that filesystem as the owner. That is why
|
||||
rule 3 refuses a share at or under a backup repository, or with one below it, and why the export
|
||||
carries `mp`, so it is served only while its filesystem is mounted.
|
||||
3. **One share per folder, never a pool's root** (the operator's choice, 2026-10-08): on the home server,
|
||||
the library and the downloads. The backup repository and the archive are not exported. The module
|
||||
refuses a share that is `/`, is owned by root, is inside a backup repository, or holds one within four
|
||||
levels below (refusing when it cannot finish looking), all judged on the real folder after any
|
||||
symbolic link, so a setting cannot export one by mistake.
|
||||
4. **A node mounts through one module.** The module `mounts` holds the node seat `node-mounts`. For each
|
||||
share it writes a `.mount` unit and an `.automount` unit, never an fstab line. Only the automount is
|
||||
enabled, so nothing mounts at boot and nothing can fail the boot. Each mount has an idle timeout
|
||||
(default 10 minutes) and a mount timeout (default 30 s; 10 s for an occasional source). A read-write
|
||||
NFS share is mounted `hard`. A mount point is always an autofs mount, so nothing writes into the
|
||||
empty directory under it (issue 169's data-loss case).
|
||||
5. **Only the nodes that ask, and that the server grants, get a share.** A share is exported to one
|
||||
node's address only when the server's setting `grants` names that node for that share and the node's
|
||||
`mounts` assignment asks for it; the access is the lower of the two, decided on the server. Addresses
|
||||
come from the controller's list of machines, never from what a client says. A share nobody is granted
|
||||
is exported to nobody: the control-node, like any node, gets nothing unless both name it. A node mounts
|
||||
what its `mounts` assignment's setting `shares` lists. The list may be set once at the mesh-wide settings layer, which is how "every node" is said,
|
||||
and a node's own layer can leave a share out. Which nodes are assigned, and whether the laptop mounts
|
||||
read-only, is decided at assignment, which is the operator's; the recommendation is the workstation
|
||||
and the laptop (read-only), not the control-node.
|
||||
6. **An occasional source is a share with presence `occasional`.** A device that is not a node, such as
|
||||
the gaming device or a USB disk, is described in the `mounts` assignment's setting `sources`. Its absence is a
|
||||
state, `absent`, not a fault. It is mounted on access when it is there, and it never fails the boot or
|
||||
leaves a failed unit. Mounting when a device appears on the network waits for issue 154 (mDNS is
|
||||
dropped by the packet filter today).
|
||||
7. **A credential for a share is an operator-delivered secret** (ADR 0092): the `mounts` module's own
|
||||
secret `smb-credentials`, issued outside the mesh (ADR 0228), given with `secret accept`, kept by the
|
||||
vault and unsealed on the node as a file (ADR 0086). The module writes one credentials file per source
|
||||
from it, readable by root only, which the mount unit names. No password is written in a
|
||||
repository, a manifest or `/etc/fstab`.
|
||||
8. **Health:** a share that should always be reachable and is not, for 5 minutes, raises one plain
|
||||
condition, naming the share, the node and since when. It is the `mounts` module's health (ADR 0240),
|
||||
judged by its tool `mounts_health`, whose finding is *share unreachable*; no new kind of condition is
|
||||
added to the controller. An occasional source that
|
||||
is absent raises nothing. The server's health is its service and its export table holding every
|
||||
declared export.
|
||||
9. **Over the bus.** The controller wires the two through the provision: the binding gives the consumer
|
||||
the server's name. `nfs-server` keeps its exports in its key-value bucket (ADR 0201), and `mounts`
|
||||
reads it (`reads`) for each share's path and access, so a share's path is said once, on the server.
|
||||
`mounts` keeps each share's state in its own bucket. The seat's `exports` verb answers the same by
|
||||
request and reply. Neither reads the other's files (ADR 0260).
|
||||
10. **Taking over a hand-written line is an act.** The `mounts` module's `adopt` comments out one fstab
|
||||
line, keeping a copy, only after its own unit pair for that mount point is armed, and only on a
|
||||
person's word. The ZFS `sharenfs` property is cleared the same way, by `nfs-server`, once its own
|
||||
export is live.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The workstation's library comes back over the private network, and the two failed units stop
|
||||
appearing at boot.
|
||||
- Two node seats join the mesh's closed set, `node-nfs-server` and `node-mounts`. That is a controller
|
||||
change. The provision `nfs-share` needs none.
|
||||
- What got harder: every client writes to a share as one account, so there is no per-person
|
||||
permission on it. Traffic to the library crosses the private network's encryption even on the home
|
||||
network; the workstation is on Wi-Fi, so the tunnel is not the limit there, and this is measured at
|
||||
build time.
|
||||
- The underlay's own devices lose the NFS share the property gave them. They have had no access since
|
||||
2026-10-04 anyway, because the filter has closed it. Giving them access back is issue 154's question.
|
||||
- Samba stays as found, and so does its closed port. An `smb-share` seat follows when something on the
|
||||
mesh needs it.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- The `mounts` module's `mounts_list` shows every mount point's source, its unit pair, its state and
|
||||
which settings layer asked for it. `mounts_exports` shows the server's export table. A test in the
|
||||
module renders the unit pair from a binding and asserts the options above.
|
||||
- **Not built yet:** a lab scenario composing a server and two clients, asserting that the export names
|
||||
only the granted nodes' addresses with `all_squash`, that a file written from a client is owned by the
|
||||
share's owner, that a client's boot completes with the server stopped, and that an absent occasional
|
||||
source leaves no failed unit.
|
||||
- **Not built yet:** a probe in the controller's self-check that counts fstab lines of type `nfs` or
|
||||
`cifs` on every node through `mounts_list`, and reports any that are not adopted.
|
||||
- Until both exist, the modules' own tests and taking it into use safely, as design 52 §5 orders it, are the check.
|
||||
|
||||
## References
|
||||
|
||||
- [Research 038](../01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md), its evidence and options
|
||||
- [Issue 169](../04-ISSUES/169-a-machine-shares-its-files-and-the-mesh-does-not-know/00-report.md), [issue 154](../04-ISSUES/154-a-machines-own-network-is-not-a-reach/00-report.md), [issue 153](../04-ISSUES/153-an-adopted-machines-data-cannot-be-placed-where-it-is/00-report.md)
|
||||
- [ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md), [ADR 0051](0051-shared-data-is-the-operators.md), [ADR 0086](0086-a-secret-reaches-a-process-as-a-file.md), [ADR 0092](0092-an-operator-delivers-a-pair-credential.md), [ADR 0201](0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md), [ADR 0240](0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md), [ADR 0260](0260-a-bar-block-shows-a-value-its-module-keeps-on-the-bus-and-no-module-reads-anothers-files.md)
|
||||
- [52 — Shares and mounts](../03-DESIGN/01-to-be/52-shares-and-mounts.md)
|
||||
@@ -0,0 +1,218 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-controller internal/catalogue (the seats node-nfs-server and node-mounts)
|
||||
- mesh-catalog modules/nfs-server
|
||||
- mesh-catalog modules/mounts
|
||||
updated: 2026-10-08
|
||||
decisions:
|
||||
- 02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md
|
||||
- 02-DECISIONS/0051-shared-data-is-the-operators.md
|
||||
- 02-DECISIONS/0092-an-operator-delivers-a-pair-credential.md
|
||||
---
|
||||
|
||||
# 52 — Shares and mounts
|
||||
|
||||
**A node that shares a directory does it through one module, which provides the share as a provision
|
||||
and opens its port to the private network only. A node that wants the files gets them through one
|
||||
module, which writes a mount unit and an automount unit per share, so nothing mounts at boot and
|
||||
nothing fails at boot. Only the nodes that ask get a share. A device that is not always there is a
|
||||
share whose absence is a state, not a fault.**
|
||||
([ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md),
|
||||
from [research 038](../../01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md).)
|
||||
|
||||
A **share** is a directory one machine offers to others over the network. A **mount** is a share made
|
||||
visible at a path on one node. An **occasional source** is a share on a device that comes and goes.
|
||||
The glossary holds the three words.
|
||||
|
||||
## The parts
|
||||
|
||||
```
|
||||
home server workstation, laptop (the nodes that ask)
|
||||
|
||||
nfs-server (seat node-nfs-server) mounts (seat node-mounts, node)
|
||||
├ /etc/exports.d/<its file> ├ <path>.automount enabled, idle 10 min
|
||||
│ one line per share, to the ├ <path>.mount not enabled, timeout 30 s
|
||||
│ private network's range only, │ What = <server>:<path>, options
|
||||
│ all_squash to the share's owner ├ credentials file (vault, ADR 0086),
|
||||
├ nfs-server.service, NFS v4 only │ for an SMB source only
|
||||
├ listens 2049/tcp from mesh ├ its process: probes each source,
|
||||
└ its bucket: exports, clients │ keeps state in its bucket
|
||||
▲ └ tools: mounts_list, mounts_exports,
|
||||
│ provision nfs-share mount, unmount, test, adopt
|
||||
│ (the controller wires it; ▲
|
||||
│ binding: the server's name) │ settings: shares wanted, occasional
|
||||
└──────────────── request/reply on the bus ─────────┘ sources (mesh-wide and node layers)
|
||||
|
||||
gaming device (not a node) ── SMB share ──► an occasional source in the workstation's mounts settings
|
||||
```
|
||||
|
||||
| part | where | does |
|
||||
|---|---|---|
|
||||
| seats `node-nfs-server` and `node-mounts` | the controller's closed set | node seats, capacity 1, with their verbs |
|
||||
| `nfs-server` | the catalogue | holds `node-nfs-server`, provides the provision, writes its export file, runs the service, opens 2049 to the private network |
|
||||
| `mounts` | the catalogue | holds `node-mounts`, writes a unit pair per share, keeps each share's state, serves the tools |
|
||||
| the binding | the controller | gives a consumer the server's name; the consumer asks the server's seat for each share's path and access over the bus |
|
||||
| the credential | the vault | the `mounts` module's own secret `smb-credentials`, issued outside the mesh and given with `secret accept`, unsealed on the node as a file |
|
||||
|
||||
## 1. The server: `nfs-server`
|
||||
|
||||
- **Its setting `shares`.** A list of `name=path`, each optionally `:ro`, separated by spaces: on the
|
||||
home server the library and the downloads. A share's path is the operator's data (ADR 0051); the
|
||||
module never creates, chowns, chmods or removes anything under it.
|
||||
- **Its setting `grants`.** Which node may mount which share, and how: `library=<workstation>:rw,<laptop>:ro`.
|
||||
`none` (the default) grants nothing. A share is exported to a node only when `grants` names it and the
|
||||
node's `mounts` asks for it (its bucket `shares`, field `wants`); the access is the lower of the two.
|
||||
The node's address comes from the controller's list of machines, rendered into the module's own file,
|
||||
and must be inside the private network's range; never from what the node says.
|
||||
- **What it refuses.** A path that does not exist or is not a directory, `/`, a folder owned by root
|
||||
(uid or gid 0), and a folder that is, is inside, or holds within four levels a backup repository; a look
|
||||
below that is cut short by its bound refuses rather than exports. A clean look is kept for an hour.
|
||||
Every check runs on the real folder after any symbolic link. A refused share is said by the tools and
|
||||
makes the module unhealthy; the others are exported.
|
||||
- **Its file.** `/etc/exports.d/mesh.exports`, written by the module's own process, which runs as root
|
||||
and looks every 30 s. One line per share: the real path (after any symbolic link), to the addresses of
|
||||
the nodes the server grants it to and that ask for it, and to nobody else, with
|
||||
`rw` or `ro`, `sync`, `secure`, `no_subtree_check`, `all_squash`, `anonuid` and `anongid` read from
|
||||
the folder's owner, and `mp` (or `mp=<its filesystem's mount point>`), so nothing is served from an
|
||||
unmounted dataset's empty folder. `exportfs -ra` runs when the file changes or a share is missing from the kernel's
|
||||
table. The filesystem's own id is kept; none is forced.
|
||||
- **Unassigned.** The process is stopped after its configuration is removed (the node-engine removes a
|
||||
module's resources in reverse order); finding it gone, it removes its export file and has the kernel
|
||||
drop the exports. A restart or a push keeps them.
|
||||
- **The service.** `nfs-server.service`, running and enabled. NFS versions 2, 3 and 4.0 are turned off
|
||||
in the module's drop-in in `/etc/nfs.conf.d`, so only 2049/tcp is needed, declared in `listens` with
|
||||
`from: mesh`.
|
||||
- **The provision.** `nfs-share`, at the mesh's scope, with no identity: the binding gives the
|
||||
consumer the server's name, and the consumer reads the rest from the server's bucket. Its consumers'
|
||||
data class is `none`: the folders are the operator's, and the mesh backs up nothing through it.
|
||||
- **Health.** Its tool `nfs_health`: the service is active, every share is in the kernel's export table,
|
||||
and no share was refused.
|
||||
- **Its bucket.** `exports`, one entry per machine, keyed by the node's name: the answer of `exports`,
|
||||
put by the module's tools every 30 s. The root process does not reach the bus; its tools do.
|
||||
- **Its seat's verbs.** `exports` (r), `clients` (r), `test` (r), `reload` (a) and `adopt` (a). `adopt`
|
||||
clears a ZFS dataset's `sharenfs` property for a path the module now exports, only once the mesh's
|
||||
export of it is in the kernel's table, and only with a person's confirmation and reason.
|
||||
|
||||
## 2. The client: `mounts`
|
||||
|
||||
- **Its setting `shares`.** A list of `name=mountpoint`, each optionally `:ro`: the `nfs-share` shares
|
||||
this node mounts, by the server's name for them. `none` (the default) mounts none.
|
||||
- **Its setting `sources`.** Occasional sources: `name=smb://<host>/<share>@<mountpoint>` or
|
||||
`name=disk:<uuid>@<mountpoint>`, each optionally `:ro`. `none` (the default) is none.
|
||||
- **Its secret `smb-credentials`.** One section per SMB source, `[name]` with `username=` and
|
||||
`password=`, issued outside the mesh and given with `secret accept <node> mounts smb-credentials
|
||||
--from <file>`. The process writes one credentials file per source, readable by root only. A source
|
||||
with no section is said with that remedy and is not mounted.
|
||||
- **What it refuses.** A mount point at, under or above any data the mesh keeps on this machine (the
|
||||
controller's `data` verb, asked over the bus; until it is read nothing new is set up), a system path,
|
||||
a mount point where something else is mounted or a folder that is not empty, and a share this machine
|
||||
serves itself. It never stops a mount whose mounted filesystem is not its own source, and a setting
|
||||
entry it cannot read removes nothing.
|
||||
- **What it asks for.** Its bucket `shares` carries `wants`: each share this machine asks of which server,
|
||||
read-write or read-only. The server decides; the mount takes the access the server gave this machine.
|
||||
- **Reading the server.** The module reads the server's bucket `exports` (its manifest's `reads`) for
|
||||
each share's path and access: its tools, which reach the bus, write what they read to a file of the
|
||||
module's, and the root process takes only a name, an absolute path and the access from it. A share
|
||||
the server does not export is said, not guessed, and a silent server removes nothing.
|
||||
- **Two processes, both root.** `apply` writes the unit pairs and the SMB credentials files and arms
|
||||
the automounts; `watch` probes each source and keeps the states. The tools act as root through
|
||||
`sudo -n` for `mount`, `unmount` and `adopt`, and say so when it is refused.
|
||||
- **The unit pair.** For each share and source, a `.mount` and an `.automount`, named by the escaped
|
||||
mount point, written in `/etc/systemd/system` by `apply` with a marker line. It never
|
||||
touches a unit without its marker, and it removes its own pairs that are no longer asked for:
|
||||
- the `.automount` is enabled under `remote-fs.target`, with an idle time of 10 minutes;
|
||||
- the `.mount` is not enabled; its mount timeout is 30 s, 10 s for an occasional source; it starts
|
||||
after the network is online;
|
||||
- NFS: version 4.2, `hard,timeo=150,retrans=3` when read-write; `ro,soft` when read-only;
|
||||
- SMB: `credentials=<its file>`, `iocharset=utf8`, `nofail`;
|
||||
- a disk: bound to the disk's device unit, so the mount goes with the disk.
|
||||
- **States**, per share, with since when, in its bucket `shares` (one entry per machine): `armed`,
|
||||
`mounted`, `absent` (an occasional source not there), `unreachable` (should be there and is not),
|
||||
`failed` (the mount was tried and refused, with the error).
|
||||
- **Its probes.** Every 30 s, a connection to the server's port (2049, or 445 for SMB) with a short
|
||||
timeout, and the device for a disk. It never touches a mount point, so a hung server cannot hang it.
|
||||
- **Health.** `apply` by its unit; `watch` by the tool `mounts_health`: every automount is armed, every
|
||||
share the settings name could be set up, `watch` looked in the last 2 minutes, and no share that is not
|
||||
occasional has been `unreachable` or `failed` for 5 minutes. An absent occasional source is healthy and said as
|
||||
*not present since 18:02*.
|
||||
- **Its seat's verbs.** `list` (r), `test` (r), `mount` (a), `unmount` (a), `adopt` (a); its tools
|
||||
`mounts_list` and `mounts_exports` (r) stay.
|
||||
|
||||
## 3. Which nodes get which share
|
||||
|
||||
Which nodes are assigned `mounts`, and with which settings, is decided at assignment, which is the
|
||||
operator's. The recommendation:
|
||||
|
||||
| node | gets | why |
|
||||
|---|---|---|
|
||||
| home server | none; it holds `nfs-server` | the files are local; the media modules keep their local bind mounts |
|
||||
| workstation | the library and the downloads, read-write; the gaming device as an occasional source | the operator works on the library there |
|
||||
| laptop | the library, read-only | it travels; the automount means it tries only when something looks |
|
||||
| control-node | none | the stably reachable machine the controller runs on; it has no use for the library |
|
||||
|
||||
## 4. Conditions
|
||||
|
||||
| condition | raised when | cleared when |
|
||||
|---|---|---|
|
||||
| the `mounts` module's health, finding *share unreachable* | a share that is not occasional has been `unreachable` or `failed` for 5 minutes, or an automount is not armed | the next look finds it reached and armed |
|
||||
| the `nfs-server` module's health | a share is missing from the kernel's export table, a share was refused, or the service is down | the next look finds all three right |
|
||||
|
||||
Both are the module's health under ADR 0240, judged by its tool; the controller raises and clears the
|
||||
condition as for any module, and adds no kind of its own.
|
||||
|
||||
Each says the share, the node and since when: *"The home server's share `library` is not reachable
|
||||
from the workstation since 19:21."* An absent occasional source raises nothing; `mounts_list` says
|
||||
*"the share `retro-games` is not present since 18:02"*.
|
||||
|
||||
## 5. Taking it into use, safely
|
||||
|
||||
Read-only first, one node first, and every step checked before the next. Each step names how it is
|
||||
undone. Every `adopt` is a person's act. Nothing here touches the library's files, owners or modes, or
|
||||
the media modules' local bind mounts on the home server.
|
||||
|
||||
**Before anything:** the seats are merged and the controller that knows them is running; the two
|
||||
modules' pull request is green against it. On the home server, `node-backup.backed-up` shows a good
|
||||
night for Plex's own data, and `nfs-server`'s dry runs are read: `exports` lists only the intended
|
||||
folders, none refused for a reason not understood.
|
||||
|
||||
1. **Assign `nfs-server` to the home server, read-only.** Its `shares` names the library only, and its
|
||||
`grants` names the workstation read-only (`library=<workstation>:ro`). Check `exports`: one line, the
|
||||
workstation's address only, `ro`, `all_squash` to the library's owner, `mp`. The ZFS property's own
|
||||
export to the home network stays as it is (the packet filter already blocks it).
|
||||
*Undo:* unassign `nfs-server`; its export file goes with it.
|
||||
2. **Assign `mounts` to the workstation, read-only.** Its `shares` is `library=<mount point>:ro`. Check
|
||||
`node-mounts.list`: the pair is armed, the share reads `armed` then `mounted` after a listing, and the
|
||||
files are there. Its unit pair takes the place of the unit generated from fstab.
|
||||
*Undo:* unassign `mounts`; its unit pairs go, and the fstab line is still there.
|
||||
3. **Watch for a day.** `mounts_health` and `nfs_health` stay healthy; a reboot of the workstation
|
||||
completes with the home server's share reachable and with it stopped (the share reads `unreachable`
|
||||
and the boot does not wait).
|
||||
4. **Read-write.** Change the grant to `rw` and the workstation's setting to drop `:ro`. Write one test
|
||||
file from the workstation, check its owner on the home server is the library's owner, and remove it.
|
||||
*Undo:* set both back to `ro`.
|
||||
5. **Adopt the workstation's library line**: `node-mounts.adopt` dry run, read it, then with
|
||||
`confirm` and a reason. *Undo:* the dated copy of fstab beside it.
|
||||
6. **Adopt the ZFS property**: `node-nfs-server.adopt` dry run, then confirmed. The home network's
|
||||
export goes; the mesh's stays. *Undo:* `zfs set sharenfs=<the old value>` by a person, recorded as a
|
||||
hand-act.
|
||||
7. **The downloads**, the same way: read-only first, then read-write.
|
||||
8. **The gaming device**: the operator gives `smb-credentials` with `secret accept`; the source is added
|
||||
to `sources`; `node-mounts.list` says it `absent` while the device is off, and it mounts on access
|
||||
when it is on. Then its fstab line is adopted.
|
||||
9. **The laptop**, read-only, granted read-only on the home server.
|
||||
|
||||
**Root.** Both modules' tools act as root through `sudo -n`, and the `sudo` module grants the operator
|
||||
account every command, so the modules' own checks are the only line before root. Narrowing sudo is a
|
||||
separate decision.
|
||||
|
||||
**Stop at once** when a share reads `failed` for a reason not understood, when a dry run would touch a
|
||||
line or a dataset other than the one asked for, or when anything under the library changes owner.
|
||||
|
||||
## Not in this design
|
||||
|
||||
- **Samba.** Kept as found. An `smb-share` seat follows when a node needs it.
|
||||
- **Clients on the underlay.** NFS for devices that are not nodes waits on issue 154.
|
||||
- **Mounting when a device appears.** Waits on a local-link reach (issue 154), because the packet
|
||||
filter drops mDNS. The automount already mounts on the next access.
|
||||
@@ -51,6 +51,7 @@ document is written and this one's status becomes `implemented`.
|
||||
| [`49-the-mesh-in-domains.md`](49-the-mesh-in-domains.md) | **In progress.** Every concept belongs to one of ten domains, which owns its one word; the glossary is organised by them and is the authority, a retired word is named on its replacement's line with its scope, and two checks hold this repository's documents and the catalogue's tool descriptions to it | [ADR 0244](../../02-DECISIONS/0244-the-mesh-is-described-in-domains-and-one-word-names-one-thing.md), [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md), [ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md) |
|
||||
| [`50-split-dns-on-a-machine-with-a-vpn-client.md`](50-split-dns-on-a-machine-with-a-vpn-client.md) | **In progress.** A machine whose VPN client pushes servers of its own runs a resolver of its own, on a node seat, that routes the VPN's domains over its link and every other name to the mesh's resolvers; the VPN client's module carries the adapter | [ADR 0247](../../02-DECISIONS/0247-a-machine-with-a-vpn-client-routes-names-by-domain-through-a-resolver-of-its-own.md), [ADR 0223](../../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md) |
|
||||
| [`51-the-registries-keep-what-is-named.md`](51-the-registries-keep-what-is-named.md) | **In progress.** The artifact store, the package registry and each machine's container runtime get tools that say what they hold, what names each thing and what could go; every removal is a dry run unless a person asks and says why, and what decides is always a record | [ADR 0251](../../02-DECISIONS/0251-the-registries-say-what-they-hold-and-keep-what-is-named-and-a-machine-keeps-the-image-it-runs-and-the-one-before.md), [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md) |
|
||||
| [`52-shares-and-mounts.md`](52-shares-and-mounts.md) | **In progress.** A node shares one folder at a time through `nfs-server`, which provides it as a provision to the private network only, mapping every client to the folder's owner; a node that asks mounts it through `mounts`, a mount unit and an automount per share, so nothing mounts or fails at boot; a device that comes and goes is an occasional source whose absence is a state | [ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md), [ADR 0051](../../02-DECISIONS/0051-shared-data-is-the-operators.md), [ADR 0092](../../02-DECISIONS/0092-an-operator-delivers-a-pair-credential.md) |
|
||||
|
||||
## Not yet written
|
||||
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
---
|
||||
status: open
|
||||
status: located
|
||||
opened: 2026-09-30
|
||||
located-in:
|
||||
- mesh-catalog (no module shares a path over the network)
|
||||
- hq 02-DECISIONS (a file-share seat, per ADR 0126, is a module's own to define)
|
||||
fixed-by:
|
||||
amended-design:
|
||||
amended-design: [03-DESIGN/01-to-be/52-shares-and-mounts.md]
|
||||
---
|
||||
|
||||
# 169 — A machine shares its files, and the mesh does not know
|
||||
@@ -129,3 +129,8 @@ the mounted tree, answers it on the consumer's side too.
|
||||
|
||||
Re-checked 2026-10-08: still holds. No module in either catalogue shares a path over the network, and no
|
||||
`nfs-share` or `smb-share` seat exists.
|
||||
|
||||
**Decided 2026-10-08:** [ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md),
|
||||
from [research 038](../../01-RESEARCH/038-shares-and-mounts-the-mesh-manages/00-overview.md), which measured
|
||||
the closure this report predicted: the workstation's library mount has timed out at every boot since
|
||||
2026-10-04. The seats are `node-nfs-server` and `node-mounts`, the provision `nfs-share`; Samba waits.
|
||||
|
||||
Reference in New Issue
Block a user