novox/hq
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Resolve the host lifecycle's open items, and say how the host is delivered

Browse Source 
The upgrade question turned out to be a delivery question, so 0058 answers
both.

Today's third silo runs once per node and sends each one a command to install
and start. That is where the as-is records a package install that 404ed from
every mirror while the job went green, an image pull failure that did not fail
the deploy, and a verify stage that was built and never scheduled because it
was missing from a list.

The shape underneath all of those is that the thing reporting success was not
the thing doing the work. Meanwhile ADR 0037 has given every node a component
that applies state, reads back and reports -- so two mechanisms now change a
node and only one checks its work.

0058: a pipeline ends when the declaration is updated. Deploy stops sending
commands to nodes and becomes one write. The host applies it on its next
reconcile, and the host cannot report success it did not verify. The verify
stage disappears as a stage, which is the point -- verification stops being a
step that can be left off a list.

A pipeline result now means "the declaration is updated, and here is which
nodes have applied it". It does not wait for every node, because a node may be
legitimately switched off for a week. Outstanding is reported separately from
failed, since conflating them is how the old system produced a stall with no
error anywhere.

The host is delivered by exactly this path and needs no new resource type: a
`file` writes the package manager's config pointing at the mesh's repository, a
`package` names the version. Added a step I had missed -- before exiting for a
restart, the host runs the new binary once. A package can install something
that does not execute here, and that turns "the node never came back" into "the
apply failed and said why".

Six open items resolved: re-enrolment is decided when the token is issued and
revokes the previous identity; the mesh keeps a recovery copy of what each node
reports it owns, which un-strands the orphans; last-contact is reported with no
threshold, because a laptop off for three weeks is doing nothing wrong;
adoption always completes but a failed line makes a node ineligible for
assignment; a briefing is a structured document whose outcome is computed from
its lines; and the token is printed once and carried by hand, which is the
property that makes it worth anything.

Still open and named: automatic rollback of a host version that will not start.

0057 and 0058 are both proposed.
This commit is contained in:
jschoubben
2026-08-27 21:53:38 +02:00
parent 2204b01909
commit aeea2a9f9a
2 changed files with 306 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
03-DESIGN/01-to-be/09-the-node-lifecycle.md
+176 -36
View File
@@ -196,24 +196,6 @@ runtime because a declaration changed would stop every container on the node.
---
## Updating the host itself
A declaration too — `package: nox-mesh-host`, at a version. The package manager writes the new
binary; the running process is undisturbed, because Unix keeps the running executable's inode.
**Then the host exits, and the supervisor restarts it on the new binary.** After the apply
completes, never during it; only when the executable actually changed; exit zero, so a restart
is what happens next rather than a failure a supervisor backs off from.
**The host never asks the service manager to restart it.** That is the host killing itself
part-way through an apply. It stops by finishing.
**A version-skewed fleet is normal**, because each node restarts when its own apply finishes.
What a node reports must be the **running** version, not the installed one, or the mesh will
believe an upgrade landed before it took effect.
---
## enrolled ⇄ disconnected
Not a failure. Not degraded. A situation
@@ -285,26 +267,184 @@ If `/var/lib/mesh-host/state.json` is lost — a reinstall, a replaced disk —
**its record of what it owns**, not its ability to work. It re-enrols, receives the declaration
again, and re-applies it.
**What does not come back is removal.** Resources it applied under an older declaration and no
longer holds a record of become unowned: the host will not touch them, because it never touches
what it did not create. They sit there, unmanaged, indefinitely.
**Without help, what does not come back is removal.** Resources applied under an older
declaration, whose record is gone, become unowned: the host will not touch them, because it
never touches what it did not create. They would sit there, unmanaged, indefinitely.
The store is therefore the one piece of node state that matters, and *how it is protected* is
not designed.
**So the mesh keeps a copy of what each node reports it owns**, refreshed on every apply report,
and hands it back on a rebuild — see [Protecting the store](#protecting-the-store). The store
remains locally authoritative for *operating*; the copy exists only for this.
---
## Open
## Upgrading the host
- **Re-enrolling as the same node.** A machine that lost its identity gets a new token — but
whether the mesh treats it as the same node or a new one is an operator's decision today, and
nothing supports either.
- **Protecting the store.** Above. Its loss is silent and permanent.
- **How long disconnected, and who is told.** The fact is not held anywhere.
- **Whether a `failed` adoption line still lets adoption complete.** Reopened by
[research 012](../../01-RESEARCH/012-the-minimum-viable-node/00-overview.md) and not decided:
*flags inform* was decided about conflicts, and a failure is different in kind.
- **What a briefing looks like.** It is the first thing a session on a new node sees, which
makes it an interface rather than a log.
- **Where the enrolment token comes from, operationally.** A person carries it. Nothing says how
it is generated, shown, or transported, and it is now the only secret in adoption.
The host is delivered like anything else
([ADR 0058](../../02-DECISIONS/0058-delivery-ends-in-a-declaration.md)), and this is worth
walking through because tier 0 looks like it should be special and is not.
```
push to mesh-host
│
├─ build go build → one static binary
├─ publish packaged, into the mesh's own package repository
└─ deploy each node's declaration now names the new version
│
└─ every host applies it on its next reconcile
```
**Compared with today.** The current pipeline's third silo runs *once per node* and sends each
one a command to install and start. That is where the as-is records a package install that
404ed while the job went green. Here deploy is **one write** — the declaration changes — and the
installing is the host's ordinary work, which reads back before it records anything.
**The repository is reachable because a declaration made it so.** A `file` resource writes the
package manager's configuration pointing at the mesh's repository; a `package` resource names
the version. Both ordinary shapes, applied by the same host. **No new resource type**, which is
the test of whether this is really uniform.
### The restart
```
1 pacman installs the new binary the running process is untouched —
Unix keeps the running executable's inode
2 the host verifies the new binary runs `nox-mesh-host version`, as a subprocess
3 it finishes the apply and reports never mid-way
4 it exits 0 having finished, not having been stopped
5 systemd restarts it on the new binary
6 the new host reconciles on start trigger 1, confirming the machine still matches
```
**Step 2 is the one to insist on.** A package can install a binary that does not execute here —
wrong architecture, a libc that is not present. Running it once before committing to a restart
turns "the node never came back" into "the apply failed and said why". It is the same read-back
rule the rest of the host already follows, applied to the one resource that is the host.
**The host never asks the service manager to restart it.** That is the host stopping itself
part-way through an apply. It stops by finishing.
**A fleet upgrades over an interval, not at an instant**, because each node restarts when its
own apply completes. A node must therefore report the version it is **running**, not the one
installed — otherwise the mesh believes an upgrade landed at step 1.
**What is not solved: a new version that crashes on start.** systemd will restart it, back off,
and the node is stuck on a binary that does not run. The previous package is still in the
package manager's cache, so a person at the machine can downgrade — but there is no automatic
rollback, and designing one means the host judging its own health, which is the kind of
self-reference the rest of this document avoids. **Named, not solved.**
---
## Resolved
The items this document opened, with the reasoning, because each was open for a reason.
### Re-enrolling as the same node
**A token is issued *for* a node record, and that is where the question is answered.**
```
mesh-control token issue --node workstation # this machine is that node again
mesh-control token issue --new # a machine the mesh has not seen
```
The host does not need to know which it is. It presents a token and receives an identity; what
that identity is bound to was decided when the token was made.
**Issuing a re-enrolment token revokes the previous identity for that node**, and that is not
housekeeping. Two live identities for one node record is the stolen-laptop case with the thief's
credentials still valid — the case
[`retiring a node`](#enrolled--hosted-retiring-a-node) says is answered by revocation.
### Protecting the store
**The host reports what it owns, and the mesh keeps the last report.**
The store stays locally authoritative — a node operates from its own copy and needs nothing to
do so ([ADR 0036](../../02-DECISIONS/0036-a-node-is-a-managed-machine.md)). What changes is that
the mesh holds a **copy for recovery**, refreshed on every apply report.
So a node that loses its state file re-enrols, receives both the declaration *and* the record of
what it previously owned, and can then remove what is no longer declared. The orphans that used
to be permanently stranded are recoverable.
**This is a backup, never a source.** The host never reads it to decide anything; it is handed
back only on a store rebuild, and a node that disagrees with it wins, because the node is the
one that can see the machine.
### How long disconnected, and who is told
**The mesh records last contact per node; the node records time since it last linked.** Both,
because they answer different questions — the mesh's is *have I heard from it*, the node's is
*how stale am I*, and a node reporting the second on reconnect is how a long absence gets
noticed at all.
**No threshold and no alarm.** A laptop switched off for three weeks is doing nothing wrong, and
a mesh that alerted on it would train people to ignore the alert. It is a **reported fact** —
`last seen 4 days ago` beside every node — and what counts as too long is a judgement for
whoever is looking, not a constant in the design.
### Whether a failed adoption line blocks
**Adoption always completes. A node with a `failed` line is a node, and it is not eligible for
assignment until the failure is resolved.**
This keeps *flags inform, they do not block*
([research 012](../../01-RESEARCH/012-the-minimum-viable-node/00-overview.md)) exactly where it
was decided — for **conflicts**, where the mesh chose deliberately and the machine works — and
gives **failures** the different treatment they need, because a failure is not *we chose* but
*we could not*.
The distinction is between **joining** and **being given work**. Refusing to join makes a
machine in use unadoptable, which is the outcome that rule exists to prevent. Placing work on a
machine where something the mesh needed never happened produces a module that is installed and
does not work — [04-ISSUES/007](../../04-ISSUES/007-an-installed-package-is-not-a-capability/00-report.md)
arriving from the adoption side.
### What a briefing is
**A structured document with prose in it**, held in the node's state and reported to the mesh.
It is the first thing a session on a new node reads, which makes it an interface.
```
outcome kept derived from the worst line below, never stated separately
node workstation
adopted 2026-08-27T14:02Z
ok container runtime docker 27.0, adopted; original config kept at <path>
kept storage driver machine has overlay2, the mesh wanted btrfs — machine wins
unknown firewall ruleset could not be parsed
failed package database locked by another process
what to look at
The storage driver disagreement is preference, not requirement, so nothing is broken.
The package database was locked; nothing was installed. Re-run adoption when it is free.
```
**The outcome is computed from the lines**, so a briefing cannot read *fine* while carrying a
failed line. Two independently written fields drift, and that drift is the fault this repository
keeps cataloguing.
### Where the enrolment token comes from
**`mesh-control token issue` prints it once**, to the person running it. Single-use, and it
expires whether used or not ([ADR 0039](../../02-DECISIONS/0039-the-link-is-the-security-boundary.md)).
It is carried by hand — read off a screen, pasted into a terminal. That is not a gap in the
design, it is the design: its authenticity comes from the channel it travelled, which is what
lets a node verify a mesh it has never spoken to
([ADR 0051](../../02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md)). A token emailed,
committed, or dropped in shared storage has lost the only property that makes it worth carrying.
**On the first node it comes from the control plane that was raised two commands ago**, which is
the same command against a mesh that is one machine old.
---
## Still open
- **Automatic rollback of a bad host version.** Above. A node can be left on a binary that will
not start, and recovering needs a person at the machine.
- **How a previous declaration is retained and chosen**, which is what rollback of anything else
would use ([ADR 0058](../../02-DECISIONS/0058-delivery-ends-in-a-declaration.md)).
- **A node returning after months** applies a very large jump in one go. Correct, and untested.