Never lose the bus to a replace of its own container (issue 1945) #138

2026-10-25T11:22:00Z

mesh-tools commented

2026-10-25 11:22:00 +00:00

On 2026-10-25 at 11:23 and again at 11:53, novox's node-engine applied novox's own composition, which carries nats. The files the bus container reads (nats.program-snapshot, nats.program-entrypoint) had changed, so it replaced the container mesh-broker-nats: docker rm -f, then docker run. Removing the container took the bus away; the apply ran on the link's context, so losing the link cancelled it and the docker call was killed ("docker exited -1:"). docker run was never reached. The mesh was down 11:23-12:04.

What changes:

An apply or removal runs in a context the loss of the bus cannot end, bounded to 30 minutes (cmd/mesh-host/nodeagent.go, applyContext).
A running container is set aside (renamed .replaced and stopped), never removed, until its replacement runs; if the new one does not come up it is removed, the old one is renamed back and started, and the apply fails loudly. A .replaced left by an engine that died mid-replace is the next apply's set-aside container.
A command killed by a signal says so and why.

Tests (each seen failing on the old code): TestAnApplyOutlivesTheLinksContext, TestAnApplyIsBounded, TestAContainerLeftAsideIsPutBackOrRemovedByTheNextApply, TestACommandKilledByItsContextSaysWhy, TestAContainerWhoseReplacementFailsIsPutBack, TestAReplacementThatRunsRemovesTheContainerSetAside.

Safe sequence for novox after this merges (the operator): see /home/jochen/projects/novox/worktrees/fix-r6-bus-replace-pr.md, steps 1-5 (rename the hand-started bus aside, start the engine, wait for this build on novox, give the bus its name back, restart the engine once).

Tries: try-1792927137646163363 passed on 175fb45 (849 unit, 852 integration).
Reviewed-by: reviewer-r6m@shanks at 175fb45c19
Pushed-by: lead-47@shanks

On 2026-10-25 at 11:23 and again at 11:53, novox's node-engine applied novox's own composition, which carries nats. The files the bus container reads (nats.program-snapshot, nats.program-entrypoint) had changed, so it replaced the container mesh-broker-nats: `docker rm -f`, then `docker run`. Removing the container took the bus away; the apply ran on the link's context, so losing the link cancelled it and the docker call was killed ("docker exited -1:"). `docker run` was never reached. The mesh was down 11:23-12:04. What changes: 1. An apply or removal runs in a context the loss of the bus cannot end, bounded to 30 minutes (cmd/mesh-host/nodeagent.go, applyContext). 2. A running container is set aside (renamed <name>.replaced and stopped), never removed, until its replacement runs; if the new one does not come up it is removed, the old one is renamed back and started, and the apply fails loudly. A <name>.replaced left by an engine that died mid-replace is the next apply's set-aside container. 3. A command killed by a signal says so and why. Tests (each seen failing on the old code): TestAnApplyOutlivesTheLinksContext, TestAnApplyIsBounded, TestAContainerLeftAsideIsPutBackOrRemovedByTheNextApply, TestACommandKilledByItsContextSaysWhy, TestAContainerWhoseReplacementFailsIsPutBack, TestAReplacementThatRunsRemovesTheContainerSetAside. Safe sequence for novox after this merges (the operator): see /home/jochen/projects/novox/worktrees/fix-r6-bus-replace-pr.md, steps 1-5 (rename the hand-started bus aside, start the engine, wait for this build on novox, give the bus its name back, restart the engine once). Tries: try-1792927137646163363 passed on 175fb45 (849 unit, 852 integration). Reviewed-by: reviewer-r6m@shanks at 175fb45c19e3e790ac2f97949af55c39d551caaf Pushed-by: lead-47@shanks

mesh-tools added 3 commits 2026-10-25 11:22:00 +00:00

Run an apply in a context the bus's loss cannot end, and say why a killed command was killed (issue 1945) b99d0b45bf

Set a running container aside until its replacement runs, and put it back when the replacement does not come up (issue 1945) 219169dbd9

Take a container left set aside as the next apply's, and bound an apply to 30 minutes (issue 1945)

mesh/delivery merged: source.main moved to novox/mesh-host@3e3e43de5214

Details

mesh/check pass 175fb45c19e3 on 3e3e43de5214 | unit 299 run, 1042 kept, 5 skipped | integration 302 run, 1043 kept, 4 skipped | over bound: unit (slow…

175fb45c19

mesh-tools commented

2026-10-25 11:22:05 +00:00

Delivery novox/mesh-host@175fb45c19e3 — merged since 2026-10-25T11:25:13Z

Source — source.main moved to novox/mesh-host@3e3e43de52. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Transitions

2026-10-25 11:22 (new) → proposed (announced): novox/mesh-host#138's head announced
2026-10-25 11:24 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail
2026-10-25 11:25 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-host@175fb45c19e3` — **merged** since 2026-10-25T11:25:13Z **Source** — source.main moved to novox/mesh-host@3e3e43de5214. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Transitions** - 2026-10-25 11:22 (new) → proposed (announced): novox/mesh-host#138's head announced - 2026-10-25 11:24 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail - 2026-10-25 11:25 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools merged commit 3e3e43de52 into main

2026-10-25 11:25:12 +00:00

mesh-tools deleted branch fix/r6-bus-replace

2026-10-25 11:25:12 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-host#138