The node-engine asks again a subscription the bus refused, so its seat answers; backups and health follow what runs (issue 774) #179

2026-10-30T17:45:40Z

mesh-tools commented

2026-10-30 17:45:40 +00:00

A subscription the bus refuses is asked again: the link is opened again after a pause (1 min doubling to 30 min). Every engine had been deaf since 12:10 on 2026-10-30, which held every Telegram ask from agents and merges. The seat's endpoints carry kind and scope, so <node>/node-engine.<verb> resolves.
Backup asks that reached no one are asked again for up to 3 minutes, then fail loudly into the module's have.
Health judges follow what is declared when a module settles, so a removed process (restic.measure) is no longer watched.
moveIntoTheVault reads the vault's structured refusal: "holds another value" is recorded as superseded and said once.
No mesh-sdk pin change.

Part of the fix group "sweep 2026-10-30" (issue 774, design 47's release): one branch fix/sweep-2026-10-30 in mesh-sdk, mesh-catalog, mesh-delivery, mesh-controller, mesh-host and mesh-tools. Merge mesh-sdk first; the others pin mesh-sdk a3718126 (v0.1.14-0.20261030174137-a37181261c24).
Recovery mode: the operator merges by hand. Every fix names its quick and thorough form in its commit message; each test fails without its fix.

Reviewed-by: rev-sweep@shanks at b9c24a7b42

- **A subscription the bus refuses is asked again**: the link is opened again after a pause (1 min doubling to 30 min). Every engine had been deaf since 12:10 on 2026-10-30, which held every Telegram ask from agents and merges. The seat's endpoints carry kind and scope, so `<node>/node-engine.<verb>` resolves. - **Backup asks** that reached no one are asked again for up to 3 minutes, then fail loudly into the module's have. - **Health judges follow what is declared** when a module settles, so a removed process (restic.measure) is no longer watched. - **moveIntoTheVault reads the vault's structured refusal**: "holds another value" is recorded as superseded and said once. No mesh-sdk pin change. --- Part of the fix group "sweep 2026-10-30" (issue 774, design 47's release): one branch `fix/sweep-2026-10-30` in mesh-sdk, mesh-catalog, mesh-delivery, mesh-controller, mesh-host and mesh-tools. Merge mesh-sdk first; the others pin mesh-sdk a3718126 (v0.1.14-0.20261030174137-a37181261c24). Recovery mode: the operator merges by hand. Every fix names its quick and thorough form in its commit message; each test fails without its fix. Reviewed-by: rev-sweep@shanks at b9c24a7b42cb2db5fe9fe32e8176b303c1cf2239

mesh-tools added 4 commits 2026-10-30 17:45:40 +00:00

Ask a refused subscription again on a new link and announce the node seat as discovery reads it, so node-engine.root-free is answered again (issue 774) 7ecf7ba356

Every engine's node-seat subscriptions were refused at 12:10 by a user list composed without them. nats
asks a refused subscription again only on a new connection, and a user-list reload makes none, so the
seat answered nobody all afternoon and the messenger held every Telegram ask. The endpoints also lacked
the kind and scope discovery reads, so the console said nothing in the mesh is called node-engine.

Quick fix: restart the engines. Thorough fix (chosen, the second time a stale user list left an engine
deaf): a refused subscription of any kind is said and the link is opened again after a pause (1 min,
doubling to 30 min), so every subscription the engine makes is asked again; the seat's endpoints carry
kind, seat, scope, module, description and schema like every tool runner's.

Ask a data module's backup again while nothing on the machine answers it, within three minutes, so a tool-runner restart no longer holds the module silently (issue 774) 8bff438e5e

On 2026-10-30 ace's redis, postgres and restic (14:03), step-ca (13:55) and mesh-vault on novox (15:22)
were held on a single backup ask that met no responder while the tool runner restarted.

Quick fix: ask node-backup.now twice. Thorough fix (chosen): every node-backup ask (backed-up and now)
is asked again with the engine's backoff while it is known never to have been delivered (ErrNotAsked),
bounded at three minutes; an ask that may have been delivered (a timeout, a refusal) is never asked
twice, so no backup starts twice. Past the bound the backup fails saying how often and how long it was
asked, the module is held and its have carries those words for the assigner's ask and condition. This
was the engine's only one-shot seat ask.

Hand the health loop's judges what runs once a module settles, so a resource its new composition drops is no longer watched (issue 774) 744bfec4a0

restic's composition dropped its process restic.measure on g14 and shanks at 16:14; the engine removed it
and said it down on every look after. The judges were set only when facts were said, and the facts said
inside an apply read the module's composition from before it (the agent records the new one only after
the apply returns), so the removed resource was put back. Judge.Set itself prunes what is not declared.

Quick fix: set the judges after the restic apply. Thorough fix (chosen): one function hands every judge
(liveness, resolver, units, accounts, libraries) what is declared; it runs from the facts as before and
now also whenever a module settles (applied or removed) or is taken as running, so every removed health
entry of every module is pruned at recompose.

Read the vault's structured move-in refusal and take a value it holds another of as superseded, so the move is not asked again for ever (issue 774)

mesh/delivery merged: source.main moved to novox/mesh-host@ee71ba4dea81

Details

mesh/delivery-group group fix/sweep-2026-10-30 merged: every member is merged

Details

mesh/check checking b9c24a7b42cb on 737bce9c9448 | novox/mesh-host#179 in group fix/sweep-2026-10-30

b9c24a7b42

mesh-vault (mesh-catalog f3b13de2) answers a move it refuses with `refused: {reason, held}` and `why`, and
no read-back. The engine read only the read-back, so every such row failed "answered with no read-back"
and the controller asked it again on every pass.

Quick fix: treat any answer without a read-back as done. Thorough fix (chosen): read `refused`. For
holds-another-value the vault's value is the live one: the move answers done, superseded, at the held
generation with the held record in its words, so the controller records the row and stops asking; the
journal says it once per secret. Any other reason fails in the vault's own words.

mesh-tools commented

2026-10-30 17:45:54 +00:00

Delivery novox/mesh-host@b9c24a7b42cb — merged since 2026-10-30T18:13:08Z

Source — source.main moved to novox/mesh-host@ee71ba4dea. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Group fix/sweep-2026-10-30: novox/mesh-catalog@2455445dab, novox/mesh-delivery@e0b56c83f9, novox/mesh-host@b9c24a7b42, novox/mesh-sdk@a37181261c, novox/mesh-tools@316a5a2fc3

composed together: fail — the checker's verdict at the members' heads (1e1a3f6ab540): shanks

Transitions

2026-10-30 17:45 (new) → proposed (announced): novox/mesh-host#179's head announced
2026-10-30 17:45 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-10-30 18:13 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-host@b9c24a7b42cb` — **merged** since 2026-10-30T18:13:08Z **Source** — source.main moved to novox/mesh-host@ee71ba4dea81. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Group** `fix/sweep-2026-10-30`: novox/mesh-catalog@2455445dabee, novox/mesh-delivery@e0b56c83f94c, novox/mesh-host@b9c24a7b42cb, novox/mesh-sdk@a37181261c24, novox/mesh-tools@316a5a2fc317 - composed together: fail — the checker's verdict at the members' heads (1e1a3f6ab540): shanks **Transitions** - 2026-10-30 17:45 (new) → proposed (announced): novox/mesh-host#179's head announced - 2026-10-30 17:45 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-10-30 18:13 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools commented

2026-10-30 18:06:17 +00:00

mesh-delivery: the operator's Approve of this pull request's merge into main is ended: its head 4e2b8f32ceea is no longer the pull request's head, or the pull request closed. Nothing is merged on it. The operator is asked again once the checks pass at the new head.

jschoubben merged commit ee71ba4dea into main

2026-10-30 18:13:06 +00:00

jschoubben deleted branch fix/sweep-2026-10-30

2026-10-30 18:13:07 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-host#179