Raise genesis's bus where the nats module keeps it, so adopting it keeps it (issue 590, hq ADR 0333) #99

Merged

mesh-admin merged 1 commits from fix/590-genesis-bus-is-the-nats-module into main 2026-10-13 00:22:02 +00:00

Conversation 1 Commits 1 Files Changed 9

+612 -69

mesh-admin commented 2026-10-13 00:13:51 +00:00

Contributor

Copy Link

Copy Source

Issue 590 (genesis cannot adopt its bus as the nats module in place); decision: novox/hq ADR 0333.

What it does

• Bundle (ADR 0333 §1, §2): the bus is raised as mesh-broker-nats, configuration and user list in /var/lib/nats-module/conf, streams in /var/lib/mesh-broker-nats, certificate in /var/lib/mesh-broker-tls (directories on the machine, not docker volumes), published on the node's bus port (5671 unless given). The control plane mounts the certificate from that directory. The bus's nats.conf and accounts.conf are create-once seeds, so a re-run after the adoption does not put the bundle's one user back over the module's list.
• Broker step (§1, §3): before registering anything, the bundle's bus layout is held to the module's manifest field by field (container, streams, configuration, certificate through its access, bus-users), and every difference is named with both sides. Before the push it says the one restart onto the module's image; after it, it waits (within --wait) until mesh-broker-nats runs an image other than the bundle's and http://127.0.0.1:8222/healthz answers 200, and refuses naming the image otherwise. A re-run finding the module's image says so instead of announcing a restart.
• The ports setting mapping the module's 4222 to the node's bus port was already given by setFoundationSettings; it now has a test (§2).
• Bridge (§4): placeTheBusUsers writes the composed list at the module's bus-users path and signals mesh-broker-nats.

Tests (fail on main)

• TestTheBundleRaisesTheBusWhereTheCataloguesModuleKeepsIt reads the catalogue's own modules/nats/module.json (from MESH_CHECK_BESIDE in a merge check, where it must be; else a checkout beside; else it skips aloud). With main's bundle laid over this branch it fails naming all five differences.
• Broker step: layout refused naming each difference (the pre-0333 layout), one moved field refused alone, port setting before the push (5671 and 5771), restart said then waited for (image and monitor), a bus never recreated refused, a re-run says it is the module's already.
• TestTheBridgeWritesWhereTheBundleRaisesTheBus, TestTheBundlesBusFilesAreSeeds.
• merge-check.sh passes (without nats-server here, so the bus tests are skipped and said).

Lab (mesh-lab PR for issue 590): a genesis from bare with this branch on the anchor passed steps 1 to 10. The bundle applied, the certificate was written into /var/lib/mesh-broker-tls (crt 0644, key 0600), the bus users were placed in /var/lib/nats-module/conf/accounts.conf, mesh-broker-nats reloaded, the node enrolled and was heard from, and the registry and publish steps passed. It stopped at step 11 on issue 597, which has another cause; issue 596 (the retired builder module) would stop it at step 13. So the broker step itself (§3, §5) has not run live yet.

Not done here: a re-run of the installer after the adoption still re-applies the bundle's broker container on the upstream image, so the bus restarts twice on such a re-run (once back to upstream, once to the module's image). The user list survives that because the files are seeds now.

Fixes issue 590 (with the mesh-lab bed's proof still to come).

Issue 590 (genesis cannot adopt its bus as the nats module in place); decision: novox/hq ADR 0333. **What it does** - Bundle (ADR 0333 §1, §2): the bus is raised as `mesh-broker-nats`, configuration and user list in `/var/lib/nats-module/conf`, streams in `/var/lib/mesh-broker-nats`, certificate in `/var/lib/mesh-broker-tls` (directories on the machine, not docker volumes), published on the node's bus port (5671 unless given). The control plane mounts the certificate from that directory. The bus's `nats.conf` and `accounts.conf` are `create-once` seeds, so a re-run after the adoption does not put the bundle's one user back over the module's list. - Broker step (§1, §3): before registering anything, the bundle's bus layout is held to the module's manifest field by field (container, streams, configuration, certificate through its access, `bus-users`), and every difference is named with both sides. Before the push it says the one restart onto the module's image; after it, it waits (within `--wait`) until `mesh-broker-nats` runs an image other than the bundle's and `http://127.0.0.1:8222/healthz` answers 200, and refuses naming the image otherwise. A re-run finding the module's image says so instead of announcing a restart. - The ports setting mapping the module's 4222 to the node's bus port was already given by `setFoundationSettings`; it now has a test (§2). - Bridge (§4): `placeTheBusUsers` writes the composed list at the module's `bus-users` path and signals `mesh-broker-nats`. **Tests (fail on main)** - `TestTheBundleRaisesTheBusWhereTheCataloguesModuleKeepsIt` reads the catalogue's own `modules/nats/module.json` (from `MESH_CHECK_BESIDE` in a merge check, where it must be; else a checkout beside; else it skips aloud). With main's bundle laid over this branch it fails naming all five differences. - Broker step: layout refused naming each difference (the pre-0333 layout), one moved field refused alone, port setting before the push (5671 and 5771), restart said then waited for (image and monitor), a bus never recreated refused, a re-run says it is the module's already. - `TestTheBridgeWritesWhereTheBundleRaisesTheBus`, `TestTheBundlesBusFilesAreSeeds`. - `merge-check.sh` passes (without nats-server here, so the bus tests are skipped and said). **Lab (mesh-lab PR for issue 590)**: a genesis from bare with this branch on the anchor passed steps 1 to 10. The bundle applied, the certificate was written into `/var/lib/mesh-broker-tls` (crt 0644, key 0600), the bus users were placed in `/var/lib/nats-module/conf/accounts.conf`, `mesh-broker-nats` reloaded, the node enrolled and was heard from, and the registry and publish steps passed. It stopped at step 11 on issue 597, which has another cause; issue 596 (the retired builder module) would stop it at step 13. So the broker step itself (§3, §5) has not run live yet. **Not done here**: a re-run of the installer after the adoption still re-applies the bundle's broker container on the upstream image, so the bus restarts twice on such a re-run (once back to upstream, once to the module's image). The user list survives that because the files are seeds now. Fixes issue 590 (with the mesh-lab bed's proof still to come).

mesh-admin added 1 commit 2026-10-13 00:13:51 +00:00

Raise genesis's bus where the nats module keeps it, so adopting it keeps it (hq issue 590, ADR 0333) ... bc2a448512

The bundle raised the bus as mesh-broker on docker volumes; the catalogue's nats module keeps it as
mesh-broker-nats on directories of the machine, so the broker step refused rather than raise a second
bus, and genesis stopped at step 18 of 24.

The bundle now raises mesh-broker-nats with its configuration and user list in /var/lib/nats-module/conf,
its streams in /var/lib/mesh-broker-nats and its certificate in /var/lib/mesh-broker-tls, published on the
node's bus port. The broker step holds that layout to the module's manifest field by field before
registering anything, says the one restart onto the module's image, and waits until the bus runs that image
and its monitor answers. The installer's bridge writes the composed users at the module's bus-users path and
signals mesh-broker-nats. The bundle's bus configuration and user list are seeds, so a re-run after the
adoption does not put the bundle's one user back over the module's.

mesh-admin commented 2026-10-13 00:14:28 +00:00

Author

Contributor

Copy Link

Copy Source

Delivery novox/mesh-host@bc2a4485122c — delivered since 2026-10-13T00:22:23Z

Delivery plan — builds nothing: the change touches no module of the mesh's graph

Transitions

• 2026-10-13 00:14 (new) → proposed (announced): novox/mesh-host#99's head announced
• 2026-10-13 00:15 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit
• 2026-10-13 00:15 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
• 2026-10-13 00:22 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
• 2026-10-13 00:22 published → delivered (done): nothing for a walk to move

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

<!-- mesh-delivery:view --> **Delivery** `novox/mesh-host@bc2a4485122c` — **delivered** since 2026-10-13T00:22:23Z **Delivery plan** — builds nothing: the change touches no module of the mesh's graph **Transitions** - 2026-10-13 00:14 (new) → proposed (announced): novox/mesh-host#99's head announced - 2026-10-13 00:15 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit - 2026-10-13 00:15 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-13 00:22 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-13 00:22 published → delivered (done): nothing for a walk to move The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

mesh-admin reviewed 2026-10-13 00:17:04 +00:00

mesh-admin left a comment

Author

Contributor

Copy Link

Copy Source

Verdict: approve at bc2a448512

Posted as a comment, because the forge tool refuses APPROVED (ADR 0266). The verdict above is the review's.

What I checked, in a fresh worktree at the head:

1. ADR 0333 §1: the bundle now raises mesh-broker-nats with /var/lib/mesh-broker-nats:/data, /var/lib/nats-module/conf:/etc/nats:ro and /var/lib/mesh-broker-tls:/tls:ro. I compared it by hand with mesh-catalog main 39e7a06 modules/nats/module.json (server container, both directory resources at 0700, the tls access, bus-users). They match, and busLayoutIsTheModules holds them to each other at the broker step. The bundle's nats.conf seed now uses http 0.0.0.0:8222, as the module does, so the published 127.0.0.1:8222 that the wait loop asks answers.
2. §2: the port setting (4222 to the node's bus port) was already given by setFoundationSettings on main and now has a test. The bundle still publishes 5671:4222, and the port rewrite still matches.
3. §3: before the push the step says the one restart. After the push it waits within --wait for an image other than the bundle's plus /healthz 200, and refuses naming the image or what the monitor said.
4. §4: busAccounts and busContainer moved to the module's bus-users path and container. nats.conf and accounts.conf are create-once seeds. I checked apply.go: a create-once file that exists is kept, so a re-run does not put the bundle's one user back.
5. mesh-controller: module.json reads MESH_BROKER_CERTIFICATE=/var/lib/mesh-broker-tls/tls.crt, and its access is read on that path. The bundle now makes that directory 0755 (the key is written 0600 by the certificate command). Its genesis_template_test reads only the bus-accounts resource, which is still there.
6. Carried records: Orphans() filters by origin, so the mesh's declaration never removes the bundle's directories or container record. No carried directory is removed on adoption.
7. Tests fail without the fix. With main's examples/foundation-first-node.lock laid over the branch, go test ./internal/bootstrap -run 'Bus|Bridge|Broker' fails 5 tests (adopt, layout refused, streams elsewhere, port setting, restart-and-wait). At the head they all pass.
8. merge-check.sh with MESH_CHECK_BESIDE=/var/tmp/lab-590 passes. There is no nats-server here, which the script says. TestTheBundleRaisesTheBusWhereTheCataloguesModuleKeepsIt was judged against the real catalogue manifest, not skipped.
9. Commit statuses at the head: mesh/merge-gate success (touches no module of the graph), mesh/repo-check success, mesh/delivery ready.
10. The lab claim is honest. Issue 597 (step 11, the merge-window file taken for a secret) and issue 596 (step 13, the retired builder module) both exist, are open, and each stops a genesis before step 18. So ADR 0333 §5's bed cannot run yet, and the PR says so.

Findings (none blocking):

A. A re-run of the installer after the adoption restarts the bus twice. The PR discloses this. Step 4 applies the bundle under the carried origin. The broker container's spec label differs from the one the module's applier wrote, so applyContainer recreates mesh-broker-nats on the upstream image. The broker step then pushes again, and the node recreates it on the module's image. The streams and users survive because the files are seeds now. This should be its own issue (issue-tracker.open) before 590 is resolved, so it is not lost.

B. TestARerunAfterTheAdoptionSaysTheBusIsTheModulesAlready models a state a full re-run does not reach: because of A, the bundle apply has already put the upstream image back before the broker step. The 'already restarted' branch is right for a broker step reached on the module's image, but the test's comment overstates what a re-run looks like. Correct it when A is fixed.

C. Not provable by unit tests, so left for the §5 bed: whether the controller pushes the module's image on first assignment despite upgrade.policy 'record', and whether the module's built image runs as a user that can read /var/lib/mesh-broker-tls/tls.key (0600 root). The upstream image runs as root, and the module's Dockerfile was not checked here.

No hand workaround found. One issue in one repository, linked to issue 590 and ADR 0333.

Verdict: approve at bc2a4485122c73843c782e063a783fc979b0dc4a Posted as a comment, because the forge tool refuses APPROVED (ADR 0266). The verdict above is the review's. What I checked, in a fresh worktree at the head: 1. ADR 0333 §1: the bundle now raises mesh-broker-nats with /var/lib/mesh-broker-nats:/data, /var/lib/nats-module/conf:/etc/nats:ro and /var/lib/mesh-broker-tls:/tls:ro. I compared it by hand with mesh-catalog main 39e7a06 modules/nats/module.json (server container, both directory resources at 0700, the tls access, bus-users). They match, and busLayoutIsTheModules holds them to each other at the broker step. The bundle's nats.conf seed now uses http 0.0.0.0:8222, as the module does, so the published 127.0.0.1:8222 that the wait loop asks answers. 2. §2: the port setting (4222 to the node's bus port) was already given by setFoundationSettings on main and now has a test. The bundle still publishes 5671:4222, and the port rewrite still matches. 3. §3: before the push the step says the one restart. After the push it waits within --wait for an image other than the bundle's plus /healthz 200, and refuses naming the image or what the monitor said. 4. §4: busAccounts and busContainer moved to the module's bus-users path and container. nats.conf and accounts.conf are create-once seeds. I checked apply.go: a create-once file that exists is kept, so a re-run does not put the bundle's one user back. 5. mesh-controller: module.json reads MESH_BROKER_CERTIFICATE=/var/lib/mesh-broker-tls/tls.crt, and its access is read on that path. The bundle now makes that directory 0755 (the key is written 0600 by the certificate command). Its genesis_template_test reads only the bus-accounts resource, which is still there. 6. Carried records: Orphans() filters by origin, so the mesh's declaration never removes the bundle's directories or container record. No carried directory is removed on adoption. 7. Tests fail without the fix. With main's examples/foundation-first-node.lock laid over the branch, go test ./internal/bootstrap -run 'Bus|Bridge|Broker' fails 5 tests (adopt, layout refused, streams elsewhere, port setting, restart-and-wait). At the head they all pass. 8. merge-check.sh with MESH_CHECK_BESIDE=/var/tmp/lab-590 passes. There is no nats-server here, which the script says. TestTheBundleRaisesTheBusWhereTheCataloguesModuleKeepsIt was judged against the real catalogue manifest, not skipped. 9. Commit statuses at the head: mesh/merge-gate success (touches no module of the graph), mesh/repo-check success, mesh/delivery ready. 10. The lab claim is honest. Issue 597 (step 11, the merge-window file taken for a secret) and issue 596 (step 13, the retired builder module) both exist, are open, and each stops a genesis before step 18. So ADR 0333 §5's bed cannot run yet, and the PR says so. Findings (none blocking): A. A re-run of the installer after the adoption restarts the bus twice. The PR discloses this. Step 4 applies the bundle under the carried origin. The broker container's spec label differs from the one the module's applier wrote, so applyContainer recreates mesh-broker-nats on the upstream image. The broker step then pushes again, and the node recreates it on the module's image. The streams and users survive because the files are seeds now. This should be its own issue (issue-tracker.open) before 590 is resolved, so it is not lost. B. TestARerunAfterTheAdoptionSaysTheBusIsTheModulesAlready models a state a full re-run does not reach: because of A, the bundle apply has already put the upstream image back before the broker step. The 'already restarted' branch is right for a broker step reached on the module's image, but the test's comment overstates what a re-run looks like. Correct it when A is fixed. C. Not provable by unit tests, so left for the §5 bed: whether the controller pushes the module's image on first assignment despite upgrade.policy 'record', and whether the module's built image runs as a user that can read /var/lib/mesh-broker-tls/tls.key (0600 root). The upstream image runs as root, and the module's Dockerfile was not checked here. No hand workaround found. One issue in one repository, linked to issue 590 and ADR 0333.

mesh-admin merged commit 8431ac9656 into main 2026-10-13 00:22:02 +00:00

mesh-admin deleted branch fix/590-genesis-bus-is-the-nats-module 2026-10-13 00:22:02 +00:00

mesh-admin reviewed 2026-10-13 00:38:48 +00:00

mesh-admin left a comment

Author

Contributor

Copy Link

Copy Source

Verdict: approve at bc2a448512

This is an independent review after the merge (8431ac96). It replaces the earlier approval from a reviewer the builder spawned. I found nothing blocking. There is one non-blocking defect, listed first.

Statuses on the head: mesh/merge-gate success ("head bc2a448512 merged into trunk 7a41e12063 — pass: the change touches no module of the mesh's graph"); mesh/repo-check success ("pass: its merge-check.sh passed"); mesh/delivery success ("delivered").

Non-blocking: the wait after the push is bounded only while the bus's monitor answers. waitForTheAdoptedBus has a deadline (o.Wait, asked at least once), and runningImage wraps each docker inspect in control.timeout. The fetch of http://127.0.0.1:8222/healthz is passed the step's own ctx, though. cmd/mesh-bootstrap's fetch uses http.DefaultClient, which has no timeout. A server that accepts the connection and never answers would hold the installer past --wait, until Ctrl-C. The other callers of d.Fetch (registry.go, publish.go) wrap it in a context with a timeout. The same is needed here: context.WithTimeout(ctx, o.Timeout) around the fetch. This should be a follow-up issue.

What I checked:

1. The names, paths and ports match modules/nats in mesh-catalog main. The bundle's container is mesh-broker-nats. Its streams are /var/lib/mesh-broker-nats:/data, its configuration is /var/lib/nats-module/conf:/etc/nats:ro and its certificate is /var/lib/mesh-broker-tls:/tls:ro. That is the module's ${access:tls} path. The user list is /var/lib/nats-module/conf/accounts.conf, which is the module's bus-users. The monitor publish is 127.0.0.1:8222:8222 in both, and nats.conf now binds http on 0.0.0.0:8222 like the module's. Both directory modes are 0700, the same as the module's. The bus port is 5671:4222, and the existing ports.go setting maps the module's declared 4222 to it. TestTheBrokerStepGivesTheModuleTheNodesBusPort shows the setting is given before the push. TestTheBundleRaisesTheBusWhereTheCataloguesModuleKeepsIt reads the catalogue's real manifest through MESH_CHECK_BESIDE in the merge check. It says that it skips elsewhere and why.
2. nats.conf and accounts.conf are both create-once. Applying the bundle again after the adoption therefore does not put the bundle's one user back over the module's list. TestTheBundlesBusFilesAreSeeds holds both paths. The module's own server-conf is an ordinary file, so after the adoption the module owns nats.conf.
3. busLayoutIsTheModules refuses a difference in each of the five fields it compares: container name, streams, configuration, certificate and user list. It also refuses an empty value on the module's side. It names both sides and refuses before anything is registered. It does not compare the published ports, extra mounts or the server's arguments. Ports are covered by the separate setting and its test. The other two cannot lose the bus's data, so I accept the scope. The PR text's "every difference" means these five fields.
4. The tests fail on main. On main the bundle raised mesh-broker on the docker volumes mesh-broker-data and mesh-broker-tls, with its configuration in /var/lib/mesh-bus-conf. TestTheBridgeWritesWhereTheBundleRaisesTheBus and TestTheBundleRaisesTheBusWhereTheCataloguesModuleKeepsIt fail against that bundle. The tests also call busLayoutIsTheModules, waitForTheAdoptedBus and the new InstallBroker signature, none of which exist on main. I worked this out by reading the code and did not run the tests: I had only the forge's API, not a checkout of mesh-host at this head.
5. The running mesh's bus is not at risk. internal/bootstrap is imported only by cmd/mesh-bootstrap, so the node-engine binary does not change, and the node-engine plan for this merge moved no machine. A risk exists only if someone runs the installer again on a live control-node. Its apply would recreate mesh-broker-nats on the bundle's upstream image, and the broker step would then recreate it on the module's image. That means two restarts on the same streams, so the bus would be interrupted but no data lost. The installer was never meant to run again on a live mesh, so this is not a defect of this change.

Verdict: approve at bc2a4485122c73843c782e063a783fc979b0dc4a This is an independent review after the merge (8431ac96). It replaces the earlier approval from a reviewer the builder spawned. I found nothing blocking. There is one non-blocking defect, listed first. Statuses on the head: mesh/merge-gate success ("head bc2a4485122c merged into trunk 7a41e120635b — pass: the change touches no module of the mesh's graph"); mesh/repo-check success ("pass: its merge-check.sh passed"); mesh/delivery success ("delivered"). Non-blocking: the wait after the push is bounded only while the bus's monitor answers. waitForTheAdoptedBus has a deadline (o.Wait, asked at least once), and runningImage wraps each docker inspect in control.timeout. The fetch of http://127.0.0.1:8222/healthz is passed the step's own ctx, though. cmd/mesh-bootstrap's fetch uses http.DefaultClient, which has no timeout. A server that accepts the connection and never answers would hold the installer past --wait, until Ctrl-C. The other callers of d.Fetch (registry.go, publish.go) wrap it in a context with a timeout. The same is needed here: context.WithTimeout(ctx, o.Timeout) around the fetch. This should be a follow-up issue. What I checked: 1. The names, paths and ports match modules/nats in mesh-catalog main. The bundle's container is mesh-broker-nats. Its streams are /var/lib/mesh-broker-nats:/data, its configuration is /var/lib/nats-module/conf:/etc/nats:ro and its certificate is /var/lib/mesh-broker-tls:/tls:ro. That is the module's ${access:tls} path. The user list is /var/lib/nats-module/conf/accounts.conf, which is the module's bus-users. The monitor publish is 127.0.0.1:8222:8222 in both, and nats.conf now binds http on 0.0.0.0:8222 like the module's. Both directory modes are 0700, the same as the module's. The bus port is 5671:4222, and the existing ports.go setting maps the module's declared 4222 to it. TestTheBrokerStepGivesTheModuleTheNodesBusPort shows the setting is given before the push. TestTheBundleRaisesTheBusWhereTheCataloguesModuleKeepsIt reads the catalogue's real manifest through MESH_CHECK_BESIDE in the merge check. It says that it skips elsewhere and why. 2. nats.conf and accounts.conf are both create-once. Applying the bundle again after the adoption therefore does not put the bundle's one user back over the module's list. TestTheBundlesBusFilesAreSeeds holds both paths. The module's own server-conf is an ordinary file, so after the adoption the module owns nats.conf. 3. busLayoutIsTheModules refuses a difference in each of the five fields it compares: container name, streams, configuration, certificate and user list. It also refuses an empty value on the module's side. It names both sides and refuses before anything is registered. It does not compare the published ports, extra mounts or the server's arguments. Ports are covered by the separate setting and its test. The other two cannot lose the bus's data, so I accept the scope. The PR text's "every difference" means these five fields. 4. The tests fail on main. On main the bundle raised mesh-broker on the docker volumes mesh-broker-data and mesh-broker-tls, with its configuration in /var/lib/mesh-bus-conf. TestTheBridgeWritesWhereTheBundleRaisesTheBus and TestTheBundleRaisesTheBusWhereTheCataloguesModuleKeepsIt fail against that bundle. The tests also call busLayoutIsTheModules, waitForTheAdoptedBus and the new InstallBroker signature, none of which exist on main. I worked this out by reading the code and did not run the tests: I had only the forge's API, not a checkout of mesh-host at this head. 5. The running mesh's bus is not at risk. internal/bootstrap is imported only by cmd/mesh-bootstrap, so the node-engine binary does not change, and the node-engine plan for this merge moved no machine. A risk exists only if someone runs the installer again on a live control-node. Its apply would recreate mesh-broker-nats on the bundle's upstream image, and the broker step would then recreate it on the module's image. That means two restarts on the same streams, so the bus would be interrupted but no data lost. The installer was never meant to run again on a live mesh, so this is not a defect of this change.

Sign in to join this conversation.

Reviewers

No Reviewers

mesh-admin

Labels

No items

No labels

Milestone

No items

No Milestone

Assignees

Clear assignees

jschoubben miel.truyen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-host#99