Design 42 zfs-owns-zfs: the apply writes /etc/hostid and reads it back, as ADR 0679 version 17 says

No template writes the four binary bytes of /etc/hostid, so it is no resource
and the composer cannot see a second writer: the apply guards it by the pool's
label, reads it back, is unhealthy on a mismatch, and keeps the file on leave.

03-DESIGN/01-to-be/42-milestones/zfs-owns-zfs.md

@@ -160,13 +160,20 @@ capabilities of §4 (a reader's capability the delivery plan derives, ADR 0614
     `/etc/zfs/zed.d` is not read while the drop-in holds. The mesh creates no symlinks. The node-engine restarts
     `zfs-zed.service` on a change of the drop-in, a handler or `zed.rc` (`restart-on`). Checked by the manifest test (no
     link resource, each handler's condition).
-15. **The node's ZFS id** (ADR 0679 §1, §3, §4). `/etc/hostid` holds four bytes, which no template writes, so the apply
+15. **The node's ZFS id** (ADR 0679 §1, §3, §4, §17). `/etc/hostid` holds four bytes, binary, which no template writes,
-    (§18) writes it with ZFS's own writer, `zgenhostid -f <value>`, only while `hostid` is set, and only when the pool's
+    so it is no resource of the manifest and the apply (§18) writes it with ZFS's own writer, `zgenhostid -f <value>`,
-    label (`zdb -C <pool>`, its `hostid`) is 0 or equals the value: then the next boot imports without `zpool import
+    only while `hostid` is set, and only when the pool's label (`zdb -C <pool>`, its `hostid`) is 0 or equals the value:
-    -f`. Otherwise it writes nothing and emits `config.refused` naming why ("the pool's import would need -f"). Before
+    then the next boot imports without `zpool import -f`. When the label shows another value, or cannot be read, it
-    the Approve, the value's description and `zfs_config` say the change needs a check of the pool's import at the next
+    writes nothing and emits `config.refused` naming why ("the pool's import would need -f", "the pool's label cannot be
-    boot; the module never reboots. After the reboot, `zfs_pools` shows the pool imported. Emits `config.applied` with
+    read"). Before the Approve, the value's description and `zfs_config` say the change needs a check of the pool's
-    target `file`. Checked by §31 (unset: no `zgenhostid`; a label of another id: refused; 0 or equal: one run).
+    import at the next boot; the module never reboots. After the reboot, `zfs_pools` shows the pool imported. Emits
+    `config.applied` with target `file`. The apply reads `/etc/hostid` back at every start and after every write: while
+    `hostid` is set, a value there other than the set one emits `config.refused` naming it, because the node-engine's
+    composer cannot see a second writer of a file that is no resource. Each refusal of this line makes `zfs_health`
+    unhealthy (§29), naming the key and why, until an apply finds the label and the file agreeing with `hostid`, or
+    `hostid` is unset. No resource or template of the manifest writes `/etc/hostid`. Checked by §31 (unset: no
+    `zgenhostid`; a label of another id, or one that cannot be read: refused; 0 or equal: one run; a file holding
+    another value: refused; unhealthy, then healthy again; no manifest resource for the file).
 
 ## The pool
 
@@ -194,7 +201,7 @@ capabilities of §4 (a reader's capability the delivery plan derives, ADR 0614
       waits-on, since}.
     A declared device that is not present is a wait. Each change is read back from `zpool status -P` and emitted.
     Checked by §31's convergence tests (the empty list among them) and acceptance test 2.
-18. **The apply: who, when, and what wakes a wait** (ADR 0679 §9). A `process` resource `zfs-config-apply`, `run:
+18. **The apply: who, when, and what wakes a wait** (ADR 0679 §9, §17). A `process` resource `zfs-config-apply`, `run:
     ./zfs-tools apply`, as root, oneshot, ordered after `zfs-import.target`, enabled at boot, with `restart-on` the
     config file (§9); it converges §15 to §17 and writes the ARC values live. It holds an exclusive lock on
     `/run/zfs-tools/apply.lock`; a second start waits for it. It converges nothing while a scrub or resilver runs on
@@ -207,7 +214,7 @@ capabilities of §4 (a reader's capability the delivery plan derives, ADR 0614
     to `/sys/module/zfs/parameters/zfs_arc_max` and `zfs_arc_min` as well as to §12's file, and reads both back; each
     `module-options` entry whose live value differs is a warning naming it, "takes effect at the next load of the
     kernel module", never success and never unhealthy.
-19. **The call-site test** (ADR 0679 §7). Every command the bundle runs is built by one of its argument functions, and
+19. **The call-site test** (ADR 0679 §7, §17). Every command the bundle runs is built by one of its argument functions, and
     the test (`TestTheBundleRunsOnlyReadingVerbsSnapshotAndScrub`, renamed `TestTheBundleRunsOnlyTheFormsOfADR0679`)
     allows only: `zpool add <pool> cache|log`, `zpool remove` of a cache or log device, `zpool set`, `zfs set` and `zfs
     inherit` of a listed property, `zfs snapshot`, `zfs set mesh:schedule=<kind>` on a ZFS snapshot named `mesh-`,
@@ -217,7 +224,9 @@ capabilities of §4 (a reader's capability the delivery plan derives, ADR 0614
     data vdevs, spares, special and dedup devices, `upgrade`, a dataset's destroy, `rollback`, `rename` and `clear` are
     never run (each an act of its own, asked of the operator). The same test pins the module's other root forms:
     `wipefs --no-act <device>`, `zdb -l <device>`, `zdb -C <pool>`, `zgenhostid -f <value>`, `udevadm monitor --udev
-    --subsystem-match=block` and `systemctl show`. Fate: the README's list of allowed forms is replaced by this one.
+    --subsystem-match=block` and `systemctl show`, and refuses every other `zgenhostid` or `zdb` form: `zgenhostid`
+    only as `-f` with the value §11 validated, `zdb` only as these two reads. Fate: the README's list of allowed forms
+    is replaced by this one.
 
 ## The scheduled run and ZFS snapshots
 
@@ -302,22 +311,27 @@ capabilities of §4 (a reader's capability the delivery plan derives, ADR 0614
     `zfs.conf`, `/etc/zfs/mesh-zed.d` and the drop-in are removed, and before leaving `zfs_config` says that the next
     load of the kernel module uses ZFS's defaults. The distribution's scrub and trim timers are put back as the
     module's declaration found them, started and enabled again where it stopped or disabled them. Pool and dataset
-    properties and devices stay as the last apply set them, also on a rollback (§8).
+    properties and devices stay as the last apply set them, also on a rollback (§8). `/etc/hostid` stays, and the
+    module says so before it leaves (`zfs_config`): it is no resource the node-engine removes, and removing it would
+    change the node's hostid, so that the pool's next import could need `zpool import -f` (ADR 0679 §15).
 
 ## How it is checked
 
-31. **Unit tests**, table-driven, one test per area, within the module's 500: no values gives no `zpool` or `zfs`
+31. **Unit tests**, table-driven, one test per area, within the module's 500: no values gives no `zpool` or `zfs` write
-    write but the due scrub; each convergence plan from parsed `zpool status` and `zfs get` output (an add, a kernel
+    but the due scrub; each convergence plan from parsed `zpool status` and `zfs get` output (an add, a kernel name to
-    name to its stable name, a removal, an empty list removing nothing, a log removal refused while not healthy, a
+    its stable name, a removal, an empty list removing nothing, a log removal refused while not healthy, a `log-mirror`
-    `log-mirror` change, a wait during a scrub); every refused form (a newline, quote, `$`, backtick or semicolon in
+    change, a wait during a scrub); every refused form (a newline, quote, `$`, backtick or semicolon in each field; a
-    each field; a parameter or property not on its list; a `sharenfs` value; `-f`; a device with a signature or another
+    parameter or property not on its list; a `sharenfs` value; `-f`; a device with a signature or another pool's label;
-    pool's label; a hostid of the wrong form or a label of another id); the call-site test of §19; the lock (a second
+    a hostid of the wrong form, a label of another id or one that cannot be read, and a hostid file holding another
-    apply waits, a run with the lock held does nothing); the apply started only by `config-changed`, never a timer;
+    value than the set one, each unhealthy and healthy again once the label and the file agree or `hostid` is unset);
-    one scheduled resource; both ARC writes read back; applying from the last config file with the bus down; the
+    the call-site test of §19, refusing any other `zgenhostid` or `zdb` form and a `zgenhostid` value not validated; the
-    three templates (config file, options file, `zed.rc`); pruning never removes a ZFS snapshot whose name, property
+    lock (a second apply waits, a run with the lock held does nothing); the apply started only by `config-changed`,
-    or ledger entry does not fit, nor one with a hold or clone, and removes every surplus in one run; no ZFS snapshot
+    never a timer; one scheduled resource; both ARC writes read back; applying from the last config file with the bus
-    taken without a ledger record; `snapshots` back to `none` removes nothing; `zfs_snapshot` refuses a `mesh-`
+    down; the three templates (config file, options file, `zed.rc`); pruning never removes a ZFS snapshot whose name,
+    property or ledger entry does not fit, nor one with a hold or clone, and removes every surplus in one run; no ZFS
+    snapshot taken without a ledger record; `snapshots` back to `none` removes nothing; `zfs_snapshot` refuses a `mesh-`
     name; a scheduled `snapshot.taken.schedule` does not wake a person's waiter; a parameter awaiting a load is a
-    warning; the `.pacnew` warning; each change emits its event with before and after; one test per tool. The module
+    warning; the `.pacnew` warning; each change emits its event with before and after; one test per tool; no resource or
-    adds no integration test; §5 is the milestone's, and the main file's live tests stand for the module whole, said
+    template of the manifest for `/etc/hostid`; the hostid converged at the apply's start after `zfs-import.target`; the
-    in the pull request.
+    leave keeps `/etc/hostid` and says so. The module adds no integration test; §5 is the milestone's, and the main
+    file's live tests stand for the module whole, said in the pull request.

03-DESIGN/01-to-be/42-the-machines-modules-in-order.md

@@ -170,7 +170,8 @@ named ZFS snapshot, a scrub's start, pause and stop.
   the states of ZFS's import, mount, share, volume and target units and of the distribution's scrub and trim timers;
   and, for the one pool its config value `pool` names, that pool's cache and log devices, its properties and its
   datasets' properties from fixed lists. The node-engine's composer refuses a second module owning one of its paths
-  (ADR 0593 §7, ADR 0051 §9).
+  (ADR 0593 §7, ADR 0051 §9), all but `/etc/hostid`, which is no resource: the module's apply reads it back instead
+  (ADR 0679 §1, §17).
 - **What it never writes** (ADR 0679 §1, §2): the package's own `/etc/default/zfs` and `/etc/zfs/zed.d` with its
   `zed.rc`, and what ZFS writes itself; it reads and shows them. A dataset's `sharenfs` is the seat
   `node-nfs-server`'s, `sharesmb` the SMB module's; the pool's files are the operator's (ADR 0051); the weekly
@@ -178,10 +179,11 @@ named ZFS snapshot, a scrub's start, pause and stop.
 - **An unset or empty value changes nothing** (ADR 0679 §3): every new config value is "not managed" until the
   operator sets it through mesh-config, on the Approve for a trusted value (ADR 0348 §4, §8). Placing the new version
   on a node runs no `zpool` or `zfs` write but the scrub its schedule already runs, and places none of its new files.
-- **Who applies** (ADR 0679 §9): the node-engine places the packages, files and units as the module's resources,
+- **Who applies** (ADR 0679 §9, §17): the node-engine places the packages, files and units as the module's resources,
-  some only while a value holds; the module's root process `zfs-tools apply` converges the pool's devices and
+  some only while a value holds, all but `/etc/hostid`; the module's root process `zfs-tools apply` converges the pool's
-  properties at its start and on `config-changed`, never on a timer; the module's one scheduled run, under the same
+  devices and properties at its start and on `config-changed`, never on a timer, and writes `/etc/hostid` with ZFS's own
-  lock, starts the scrub and the trim when due and takes and removes the scheduled ZFS snapshots (ADR 0679 §8).
+  writer only while `hostid` is set and the pool's label agrees; the module's one scheduled run, under the same lock,
+  starts the scrub and the trim when due and takes and removes the scheduled ZFS snapshots (ADR 0679 §8).
 - **What it says** (ADR 0679 §10, §11, §12): every change, refusal and wait is an event, each kind on a subject of
   its own, on an events stream asked for by template; the last change per value, the schedule's pause and every
   scheduled ZFS snapshot are kept on the bus in its own states and ledger (ADR 0595). Its health names a refusal or a
@@ -190,7 +192,8 @@ named ZFS snapshot, a scrub's start, pause and stop.
 - **What it never does** (ADR 0679 §7): it converges no data device, spare, special or dedup device, upgrades no
   pool, destroys no dataset, rolls back, renames and clears nothing; each is an act of its own, decided separately.
 - **When it leaves** (ADR 0679 §15): ZFS keeps running (the import, mount and ZFS target are never stopped), the
-  packages stay, its own files are removed and the distribution's timers are put back as found.
+  packages stay, its own files are removed but `/etc/hostid`, which stays and is said, and the distribution's timers
+  are put back as found.
 - **Its prerequisites** (ADR 0679 §16): three node-engine capabilities, built first in the same milestone under issue
   1979, with their readers in mesh-sdk and the manifest validator.
 
@@ -200,7 +203,7 @@ flowchart LR
   config -->|the module's config file| engine[the node-engine]
   engine -->|packages, files, units| node[ZFS on the node]
   engine -->|restarts on its config file| apply[zfs-tools apply, root]
-  apply -->|zpool add, remove, set; zfs set, inherit| node
+  apply -->|zpool add, remove, set; zfs set, inherit; zgenhostid| node
   sched[the scheduled run] -->|scrub, trim, ZFS snapshots and their removal| node
   follower[the event follower] -->|scrub ended, device appeared: start| apply
   apply -->|config.applied, refused, waiting| stream[(events stream)]