Take the placement files out again, so every node returns to the controller's compositions (recovery, issue 1126) #521

2026-10-27T08:23:39Z

mesh-tools commented

2026-10-27 08:23:39 +00:00

Recovery for a blocking fault after mesh-catalog #520 (the placement seed) merged at 08:20:34Z.

Symptom (novox's node-engine journal, 09:21-09:22 CET, every node alike): 'this node composes its own modules: the catalogue places it' then 'nats: permissions violation: Permissions Violation for Publish to "$JS.API.CONSUMER.CREATE.KV_node-engine_have..$KV.node-engine_have.>"' and 'the node agent stopped: the agent cannot watch its compositions: ... watching > in node-engine_have: context deadline exceeded — beginning again in 4s/8s/16s/32s/1m4s'. The fold fix (mesh-host #152) made the self-composer watch every have ('>'), which the node's grant does not allow. So no node composes or applies anything; what runs keeps running (mesh-host, nats, mesh-wireguard healthy on every node at 08:22Z), but no delivery can reach a node, not even the fix.

Why this recovers: mesh-host cmd/mesh-host/nodeagent.go selfComposer returns nil when placement. is deleted, so at the agent's next retry (at most 2 minutes) each node applies the controller's compositions again. The fix of the watch (two patterns within the grant: '.node' and '.') is then delivered the normal way, and the seed is opened again after it.

This removes only placement/{ace,g14,novox,shanks}.json, which #520 added. Its check cannot pass (the nodes' shadow composers fail the same watch); merge it on the operator's Approve without waiting for the check.

Reviewed-by: lead-47@shanks at b95b0a4750 (a revert of #520's files only; no independent review: a blocking recovery, said in the report)

Recovery for a blocking fault after mesh-catalog #520 (the placement seed) merged at 08:20:34Z. Symptom (novox's node-engine journal, 09:21-09:22 CET, every node alike): 'this node composes its own modules: the catalogue places it' then 'nats: permissions violation: Permissions Violation for Publish to "$JS.API.CONSUMER.CREATE.KV_node-engine_have.<id>.$KV.node-engine_have.>"' and 'the node agent stopped: the agent cannot watch its compositions: ... watching > in node-engine_have: context deadline exceeded — beginning again in 4s/8s/16s/32s/1m4s'. The fold fix (mesh-host #152) made the self-composer watch every have ('>'), which the node's grant does not allow. So no node composes or applies anything; what runs keeps running (mesh-host, nats, mesh-wireguard healthy on every node at 08:22Z), but no delivery can reach a node, not even the fix. Why this recovers: mesh-host cmd/mesh-host/nodeagent.go selfComposer returns nil when placement.<node> is deleted, so at the agent's next retry (at most 2 minutes) each node applies the controller's compositions again. The fix of the watch (two patterns within the grant: '*.node' and '<node>.*') is then delivered the normal way, and the seed is opened again after it. This removes only placement/{ace,g14,novox,shanks}.json, which #520 added. Its check cannot pass (the nodes' shadow composers fail the same watch); merge it on the operator's Approve without waiting for the check. Reviewed-by: lead-47@shanks at b95b0a4750444161613661ee57dd242d039efda8 (a revert of #520's files only; no independent review: a blocking recovery, said in the report)

mesh-tools added 1 commit 2026-10-27 08:23:39 +00:00

Take the placement files out again: the node agents cannot watch their haves with '>' (refused by their grant), so no node can compose its own; without its placement key each node returns to the controller's compositions (issue 1126)

mesh/delivery ready: it is merged on the Approve

Details

mesh/check failed: ace: ace would stop running node with the candidate placement, which the change does not intend; novox would stop running node with…

b95b0a4750

mesh-tools commented

2026-10-27 08:23:42 +00:00

Delivery novox/mesh-catalog@b95b0a475044 — ready since 2026-10-27T08:23:42Z

Transitions

2026-10-27 08:23 (new) → proposed (announced): novox/mesh-catalog#521's head announced
2026-10-27 08:23 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-catalog@b95b0a475044` — **ready** since 2026-10-27T08:23:42Z **Transitions** - 2026-10-27 08:23 (new) → proposed (announced): novox/mesh-catalog#521's head announced - 2026-10-27 08:23 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

Some required checks failed

mesh/delivery ready: it is merged on the Approve

Details

mesh/check failed: ace: ace would stop running node with the candidate placement, which the change does not intend; novox would stop running node with…

You are not authorized to merge this pull request.

This pull request can be merged automatically.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin revert/1126-seed:revert/1126-seed

git checkout revert/1126-seed

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-catalog#521