The mesh knows who is out of touch

09-the-node-lifecycle asks for this in as many words -- *how long it has been
disconnected is a fact the mesh must hold, and nothing holds it today. Without
it, a node running last month's assignments looks exactly like one that is
current.* Now it holds it.

`node list` says "here", "out of touch 4m", or "never spoken", and the third is
kept distinct from the second on purpose: a node that has never spoken did not
finish joining, and a node last heard from a month ago is running a month-old
picture of the mesh. Those need different responses from a person.

A bare word that a node is there moves last_seen and touches nothing else. It
is not an account of what the machine holds, and recording it as one would
replace the recovery copy with an empty list every minute -- so a rebuilding
node would then be told it owns nothing and remove whatever it found. There is
a test for exactly that.

Heard is silent in the log. A node saying it is there every minute would fill
the log with the ordinary case, and a log where the ordinary case is loud is a
log nobody reads.

Verified in the lab across the threshold, both directions.
This commit is contained in:
2026-08-29 20:32:27 +02:00
parent fc1417be72
commit f0cff88172
6 changed files with 153 additions and 9 deletions
+36 -1
View File
@@ -206,7 +206,7 @@ func nodeCommand(ctx context.Context, args []string) error {
return nil
}
for _, n := range nodes {
fmt.Printf("%-20s %s added %s\n", n.Name, n.ID, n.Created.Format(time.RFC3339))
fmt.Printf("%-20s %-14s %s\n", n.Name, heardFrom(n), n.ID)
}
return nil
@@ -633,3 +633,38 @@ func overlayPush(ctx context.Context, inv *inventory.Inventory) error {
fmt.Printf("\n%d of %d node(s) told\n", sent, len(nodes))
return nil
}
// SilentFor is how long a node may be quiet before the mesh says so.
//
// A node speaks every minute, so three of them missed is a gap rather than a slow one. The number
// is not the point — being able to say "out of touch" at all is, and nothing could before.
const SilentFor = 3 * time.Minute
// heardFrom says when a node was last heard from, in a form somebody can act on.
//
// "never" and "an hour ago" are different answers and are kept different. A node that has never
// spoken did not finish joining; a node last heard from an hour ago is running an hour-old
// picture of the mesh — and until this existed both looked exactly like a node that is current.
func heardFrom(n inventory.Node) string {
silent, ever := n.Silent()
switch {
case !ever:
return "never spoken"
case silent > SilentFor:
return "out of touch " + roughly(silent)
default:
return "here"
}
}
// roughly is a duration a person reads rather than parses.
func roughly(d time.Duration) string {
switch {
case d < time.Hour:
return fmt.Sprintf("%dm", int(d.Minutes()))
case d < 48*time.Hour:
return fmt.Sprintf("%dh", int(d.Hours()))
default:
return fmt.Sprintf("%dd", int(d.Hours()/24))
}
}