Take the placement files out again, so every node returns to the controller's compositions (recovery, issue 1126) #521

2026-10-27T08:23:39Z

mesh-tools commented

2026-10-27 08:23:39 +00:00

Recovery for a blocking fault after mesh-catalog #520 (the placement seed) merged at 08:20:34Z.

Symptom (novox's node-engine journal, 09:21-09:22 CET, every node alike): 'this node composes its own modules: the catalogue places it' then 'nats: permissions violation: Permissions Violation for Publish to "$JS.API.CONSUMER.CREATE.KV_node-engine_have..$KV.node-engine_have.>"' and 'the node agent stopped: the agent cannot watch its compositions: ... watching > in node-engine_have: context deadline exceeded — beginning again in 4s/8s/16s/32s/1m4s'. The fold fix (mesh-host #152) made the self-composer watch every have ('>'), which the node's grant does not allow. So no node composes or applies anything; what runs keeps running (mesh-host, nats, mesh-wireguard healthy on every node at 08:22Z), but no delivery can reach a node, not even the fix.

Why this recovers: mesh-host cmd/mesh-host/nodeagent.go selfComposer returns nil when placement. is deleted, so at the agent's next retry (at most 2 minutes) each node applies the controller's compositions again. The fix of the watch (two patterns within the grant: '.node' and '.') is then delivered the normal way, and the seed is opened again after it.

This removes only placement/{ace,g14,novox,shanks}.json, which #520 added. Its check cannot pass (the nodes' shadow composers fail the same watch); merge it on the operator's Approve without waiting for the check.

Reviewed-by: lead-47@shanks at b95b0a4750 (a revert of #520's files only; no independent review: a blocking recovery, said in the report)

Recovery for a blocking fault after mesh-catalog #520 (the placement seed) merged at 08:20:34Z. Symptom (novox's node-engine journal, 09:21-09:22 CET, every node alike): 'this node composes its own modules: the catalogue places it' then 'nats: permissions violation: Permissions Violation for Publish to "$JS.API.CONSUMER.CREATE.KV_node-engine_have.<id>.$KV.node-engine_have.>"' and 'the node agent stopped: the agent cannot watch its compositions: ... watching > in node-engine_have: context deadline exceeded — beginning again in 4s/8s/16s/32s/1m4s'. The fold fix (mesh-host #152) made the self-composer watch every have ('>'), which the node's grant does not allow. So no node composes or applies anything; what runs keeps running (mesh-host, nats, mesh-wireguard healthy on every node at 08:22Z), but no delivery can reach a node, not even the fix. Why this recovers: mesh-host cmd/mesh-host/nodeagent.go selfComposer returns nil when placement.<node> is deleted, so at the agent's next retry (at most 2 minutes) each node applies the controller's compositions again. The fix of the watch (two patterns within the grant: '*.node' and '<node>.*') is then delivered the normal way, and the seed is opened again after it. This removes only placement/{ace,g14,novox,shanks}.json, which #520 added. Its check cannot pass (the nodes' shadow composers fail the same watch); merge it on the operator's Approve without waiting for the check. Reviewed-by: lead-47@shanks at b95b0a4750444161613661ee57dd242d039efda8 (a revert of #520's files only; no independent review: a blocking recovery, said in the report)

mesh-tools added 1 commit 2026-10-27 08:23:39 +00:00

Take the placement files out again: the node agents cannot watch their haves with '>' (refused by their grant), so no node can compose its own; without its placement key each node returns to the controller's compositions (issue 1126)

mesh/delivery merged: source.main moved to novox/mesh-catalog@c05aa5e6eafd

Details

mesh/check checking b95b0a475044 on c05aa5e6eafd

b95b0a4750

mesh-tools commented

2026-10-27 08:23:42 +00:00

Delivery novox/mesh-catalog@b95b0a475044 — merged since 2026-10-27T14:06:37Z

Source — source.main moved to novox/mesh-catalog@c05aa5e6ea. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Transitions

2026-10-27 08:23 (new) → proposed (announced): novox/mesh-catalog#521's head announced
2026-10-27 08:23 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-10-27 14:06 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-catalog@b95b0a475044` — **merged** since 2026-10-27T14:06:37Z **Source** — source.main moved to novox/mesh-catalog@c05aa5e6eafd. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Transitions** - 2026-10-27 08:23 (new) → proposed (announced): novox/mesh-catalog#521's head announced - 2026-10-27 08:23 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-10-27 14:06 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools merged commit c05aa5e6ea into main

2026-10-27 14:06:35 +00:00

mesh-tools deleted branch revert/1126-seed

2026-10-27 14:06:35 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-catalog#521