The node-engine asks again a subscription the bus refused, so its seat answers; backups and health follow what runs (issue 774) #179

Merged
jschoubben merged 4 commits from fix/sweep-2026-10-30 into main 2026-10-30 18:13:06 +00:00
4 Commits
Author SHA1 Message Date
jschoubben b9c24a7b42 Read the vault's structured move-in refusal and take a value it holds another of as superseded, so the move is not asked again for ever (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-host@ee71ba4dea81
mesh/delivery-group group fix/sweep-2026-10-30 merged: every member is merged
mesh/check checking b9c24a7b42cb on 737bce9c9448 | novox/mesh-host#179 in group fix/sweep-2026-10-30
mesh-vault (mesh-catalog f3b13de2) answers a move it refuses with `refused: {reason, held}` and `why`, and
no read-back. The engine read only the read-back, so every such row failed "answered with no read-back"
and the controller asked it again on every pass.

Quick fix: treat any answer without a read-back as done. Thorough fix (chosen): read `refused`. For
holds-another-value the vault's value is the live one: the move answers done, superseded, at the held
generation with the held record in its words, so the controller records the row and stops asking; the
journal says it once per secret. Any other reason fails in the vault's own words.
2026-10-30 18:36:38 +01:00
jschoubben 744bfec4a0 Hand the health loop's judges what runs once a module settles, so a resource its new composition drops is no longer watched (issue 774)
restic's composition dropped its process restic.measure on g14 and shanks at 16:14; the engine removed it
and said it down on every look after. The judges were set only when facts were said, and the facts said
inside an apply read the module's composition from before it (the agent records the new one only after
the apply returns), so the removed resource was put back. Judge.Set itself prunes what is not declared.

Quick fix: set the judges after the restic apply. Thorough fix (chosen): one function hands every judge
(liveness, resolver, units, accounts, libraries) what is declared; it runs from the facts as before and
now also whenever a module settles (applied or removed) or is taken as running, so every removed health
entry of every module is pruned at recompose.
2026-10-30 18:12:56 +01:00
jschoubben 8bff438e5e Ask a data module's backup again while nothing on the machine answers it, within three minutes, so a tool-runner restart no longer holds the module silently (issue 774)
On 2026-10-30 ace's redis, postgres and restic (14:03), step-ca (13:55) and mesh-vault on novox (15:22)
were held on a single backup ask that met no responder while the tool runner restarted.

Quick fix: ask node-backup.now twice. Thorough fix (chosen): every node-backup ask (backed-up and now)
is asked again with the engine's backoff while it is known never to have been delivered (ErrNotAsked),
bounded at three minutes; an ask that may have been delivered (a timeout, a refusal) is never asked
twice, so no backup starts twice. Past the bound the backup fails saying how often and how long it was
asked, the module is held and its have carries those words for the assigner's ask and condition. This
was the engine's only one-shot seat ask.
2026-10-30 18:09:16 +01:00
jschoubben 7ecf7ba356 Ask a refused subscription again on a new link and announce the node seat as discovery reads it, so node-engine.root-free is answered again (issue 774)
Every engine's node-seat subscriptions were refused at 12:10 by a user list composed without them. nats
asks a refused subscription again only on a new connection, and a user-list reload makes none, so the
seat answered nobody all afternoon and the messenger held every Telegram ask. The endpoints also lacked
the kind and scope discovery reads, so the console said nothing in the mesh is called node-engine.

Quick fix: restart the engines. Thorough fix (chosen, the second time a stale user list left an engine
deaf): a refused subscription of any kind is said and the link is opened again after a pause (1 min,
doubling to 30 min), so every subscription the engine makes is asked again; the seat's endpoints carry
kind, seat, scope, module, description and schema like every tool runner's.
2026-10-30 18:06:52 +01:00