Upgrade the nodes one at a time on the bus, with a package record, a kernel hold, an after-upgrade check, a rollback and a fallback kernel (upgrade safety, issues 1840 and 1841) #450

Merged

mesh-tools merged 47 commits from feat/upgrade-safety into main 2026-10-22 18:34:16 +00:00

Conversation 1 Commits 47 Files Changed 69

+10534 -447

mesh-tools commented 2026-10-22 16:55:33 +00:00

Member

Copy Link

Copy Source

Milestone upgrade-safety (ADR 0507, ADR 0508, ADR 0510), one delivery group with mesh-sdk, mesh-controller and hq on the branch feat/upgrade-safety. R4e's fourth prerequisite (ADR 0513 §1).

modules/pacman: each node's packages on the bus (1840); upgrades as runs kept on the bus, a package record taken and read back before each, a kernel held when an installed DKMS module cannot follow it, an after-upgrade check, a rollback to a record, and weekly rounds one node at a time in the mesh-level upgrade-order, passing a node only on the operator's Approve (1841); a root helper (pacman-root) with a closed list of verbs on a socket only the keeper's account opens, no sudo rule (ADR 0510); health read from the bus (folds 1579). modules/kernel: a fallback kernel (linux-lts) on every boot menu, the default entry kept (GRUB_TOP_LEVEL names the booted image), a found kernel-copying hook taken over. predefined.json gains upgrade-order.

After the delivery, until the operator sets upgrade-order, every node's upgrade health says why no round starts (ADR 0508 §5).

Checks: pacman and kernel unit suites, particulars and paths checks; try build-1792685932013031447 passed (trunk 835bf22b). Reviewed in three rounds (GRUB default entry, a test's exactness, the re-pin).

Reviewed-by: reviewer-us@shanks at 86ba54b06e

Milestone upgrade-safety (ADR 0507, ADR 0508, ADR 0510), one delivery group with mesh-sdk, mesh-controller and hq on the branch feat/upgrade-safety. R4e's fourth prerequisite (ADR 0513 §1). modules/pacman: each node's packages on the bus (1840); upgrades as runs kept on the bus, a package record taken and read back before each, a kernel held when an installed DKMS module cannot follow it, an after-upgrade check, a rollback to a record, and weekly rounds one node at a time in the mesh-level `upgrade-order`, passing a node only on the operator's Approve (1841); a root helper (pacman-root) with a closed list of verbs on a socket only the keeper's account opens, no sudo rule (ADR 0510); health read from the bus (folds 1579). modules/kernel: a fallback kernel (linux-lts) on every boot menu, the default entry kept (GRUB_TOP_LEVEL names the booted image), a found kernel-copying hook taken over. predefined.json gains `upgrade-order`. After the delivery, until the operator sets `upgrade-order`, every node's upgrade health says why no round starts (ADR 0508 §5). Checks: pacman and kernel unit suites, particulars and paths checks; try build-1792685932013031447 passed (trunk 835bf22b). Reviewed in three rounds (GRUB default entry, a test's exactness, the re-pin). Reviewed-by: reviewer-us@shanks at 86ba54b06e09d1b11cf9c6f9da6f3a04c1ded7d7

mesh-tools added 47 commits 2026-10-22 16:55:34 +00:00

Declare the pacman keeper and its bus shapes, so upgrades run as rounds on the bus instead of a weekly timer on every node at once (issue 1841/S1, with 1840/S5's manifest) ... eb717cc514

The keeper process runs under its own account with a bus account of its own,
keeps the six states and emits the nine events of ADR 0507 and ADR 0508; its
config (upgrade-order, record-days, round-start-minutes) lands in its config
file and restarts it. shapes.go fixes every key's value and event body as the
interface the keeper and the verbs share. The weekly timer and its service are
dropped, which the node-engine removes.

Take the SDK that lets a process put, list and watch its states (issue 1840/S5) 802c72e4ed

issue 1841/S15: place the fallback kernel's boot entry from the loader the machine booted with ... 0f4e631cdd

ADR 0508 §9 and §10: systemd-boot gets an entry from the booted command line, GRUB a regenerated menu only when its kernels change, a found kernel-copying hook is taken over, a full boot partition fails loudly.

issue 1838: name the kernel module's fixtures by what they show (as in the fix's own branch) 2cb2e27cdf

issue 1841/S14: declare the fallback kernel and its headers after the boot-entries unit ... e4373768af

ADR 0508 §9: every DKMS module is built for the fallback kernel too, and its entry is in place first.

issue 1841/S16: kernel_boot answers the loader, entries, kernels, booted entry and the fallback ... bc77afa6bb

ADR 0508 §9: the operator sees whether every boot menu lists the fallback kernel, and the room left on /boot.

Keep every upgrade as a run on the bus and read the log's past upgrades once, so pacman_upgrades answers every node's history without asking a machine (issue 1841/S2) efac1b4856

Merge branch 'feat/upgrade-safety' into p1 c0bc0d4826

Take the node's package record before a run and read it back, refusing a run whose packages are not current, so a rollback always has a true record to return to (issue 1841/S3) 0bb6584437

issue 1840/S6: keep the node's packages in installed, read at the keeper's start and at every released lock ... 1de9a903c8

The keeper reads pacman's database into installed.<node> and writes it only when it differs from the key,
says each change as package.changed, and keeps the last value when the list is over the bus's cap, marking
the node not current until it fits again; the health names the size. The loop runs over injected states,
events, clock, watch and machine, and leaves hooks (start, after a transaction, state watches, wakes) for the
upgrade parts of ADR 0508.

Merge branch 'feat/upgrade-safety' into p2 e4fb4cc793

Hold a kernel an installed DKMS module's own bound excludes, with its headers, so an upgrade never leaves a machine whose storage driver cannot load (issue 1841/S4) b015576a13

issue 1841/S4: format the gate's patterns as gofmt does, which the merge check requires f40fa6e8e0

issue 1840/S6: emit through the same EventSink as the upgrade parts, so one connection serves both 79ce72a7b7

Merge branch 'feat/upgrade-safety' into p2 5cc401683c

issue 1840/S7: prove updates is read at start, after a transaction, after the mirrors and daily, and written only on change ... 56b285d3e8

A failed read keeps the last list and says reachable false, so a list the node can no longer confirm is
never shown as current, and a first read that fails is never an empty list taken for an answer.

Carry out an upgrade as an ask its node's keeper takes, through the root helper, and follow a run to its end after a restart, so a hand upgrade is a recorded, checked run like any other (issue 1841/S5) 95e69276c3

Merge branch 'feat/upgrade-safety' into p1 ac4d156cb7

issue 1840/S8: answer the package view from the bus, so every node is read at once and none is asked ... 62437f8964

pacman_packages answers one package on every node, or each node's counts, its explicit packages no module
declares and its foreign ones apart, the versions that differ and the packages on some nodes only, with a
node over the bus's cap marked not current. pacman_updates, pacman_installed and pacman_foreign read the
keeper's states instead of the machine; pacman_updates says when each node's list last changed.

Merge branch 'feat/upgrade-safety' into p2 32c66a98c1

issue 1840/S9: prove pacman's output puts no command line or environment on the bus ... 9651a8b2da

A fixture of pacman and checkupdates output holding command lines, environment variables and messages
leaves only names, versions and repositories in installed, updates and package.changed, and no command
the keeper runs names a package, a foreign one least of all.

Take the SDK whose process can delete a key (issue 1840/S1) 9511074f40

issue 1840/S10: run the keeper whole against a real bus, so its key, its event and another instance's view are proved together ... b7f3664839

Written and compiled under the integration tag; the merge check runs it.

issue 1841/S6: check after every upgrade the files it left, the news since the last one, units newly failed and a restart needed, so the operator sees what an upgrade asks of them 900a2da3e3

Merge branch 'feat/upgrade-safety' into p1 75f3f52589

Merge branch 'feat/upgrade-safety' into p2 ... c93f77ac19

# Conflicts:
#	modules/pacman/go.mod

issue 1840/S6: take the SDK's process delete for the not-current mark, now that the process state has one c1a9ad2abc

Merge branch 'feat/upgrade-safety' into p2 b7a4a1a6b0

issue 1841/S7: name the processes still mapping a replaced file and fail the check on a DKMS module not built for an installed kernel, so a machine is never left unable to load its drivers unnoticed 2b2517e4a5

Merge branch 'feat/upgrade-safety' into p1 ff0f43ea2f

issue 1841/S5: join the upgrade parts to the keeper's loop, so a node's log, records, unfinished runs and asks are taken up by its own keeper ... d70135bf08

At its start the keeper reads the log into runs once, prunes old records and carries unfinished runs on;
after every released lock it carries them on again; it takes each ask at <node>.* and prunes records daily.
The root helper's client is a constructor (newRootActs) the helper replaces; until then every act is
refused, named.

Merge branch 'feat/upgrade-safety' into p2 dc703d1511

issue 1841/S5: refuse the root helper's new maps act too until its client is built in 71f47c3291

issue 1841/S8: upgrade the nodes one at a time in a weekly round in upgrade-order, stopping at a failure and passing a node only on the operator's Approve, so one bad upgrade never reaches every machine f550f1c6b2

issue 1841/S10: return a node to a run's record, saying first what each package goes back to and from where ... 43fccb843c

pacman_rollback plans from the bus: each package now -> record from the node's cache, else the archive's
exact version, else the archive's repository on the record's date; added packages listed and removed only
when asked; foreign packages untouched. apply is an ask the node's keeper runs as a rollback run through the
root helper's new Rollback act (RootActs, upgrades.go), with rollback.started, rollback.finished and the
after-upgrade check (TakeAsk's rollback case, transaction.go).

Merge branch 'feat/upgrade-safety' into p2 ... f339d0aab7

# Conflicts:
#	modules/pacman/cmd/pacman-tools/main.go
#	modules/pacman/cmd/pacman-tools/transaction.go
#	modules/pacman/module.json

issue 1841/S9: carry out the upgrade's root acts through a root helper on a socket only the keeper opens, so no sudo rule lets the keeper's account run anything as root (ADR 0510) 28ef6e9e27

issue 1841/S8: join the round to the keeper's loop, so a node starts the moment the node before it passed and a pass is answered from the operator's channel 276422a85b

issue 1841/S11: judge the upgrades' health from the runs, rounds and held kernels on the bus, so a node that slept through a run is judged by what happened, not by a unit's age (folds issue 1579) f8ac498d64

issue 1841/S12: prove the round on a real bus with two made-up nodes, so the second node is seen to start only after the first passed and never after a failed check ea8e0def6e

issue 1841/S13: describe the package view, the runs, the round, the root helper and the health read from the bus in pacman's README e65c8c8961

Merge remote-tracking branch 'origin/main' into feat/upgrade-safety 2149d8f3e7

issue 1841/S2: leave the login manager's configuration path out of pacman's captured log, which the paths check refuses in another module's fixture f40269caec

issue 1841/S15: put the booted kernel first when GRUB's menu is generated again, so the fallback kernel, sorted first by version, never becomes the default entry (ADR 0508 §9) 76a0a726f0

Take the SDK at the milestone's mesh-sdk head f8b8f3a6 in pacman, which the delivery group's members must pin exactly (issue 1840) 0270027bf9

Merge remote-tracking branch 'origin/main' into feat/upgrade-safety af91a7e660

Take the SDK at the milestone's mesh-sdk head 8892a117 in pacman, which holds main's SDK and the milestone's (issue 1840) 86ba54b06e

mesh-tools commented 2026-10-22 16:55:43 +00:00

Author

Member

Copy Link

Copy Source

Delivery novox/mesh-catalog@86ba54b06e09 — failed since 2026-10-22T18:50:19Z

Covered: covered after its failure: a later walk that contains it was delivered: plan-1792694058124013462 (novox/mesh-catalog at 2f05e707 into main, done, contains its merge 2f05e707)

Delivery plan — builds claude-code, dunst, kernel, pacman → ace, g14, novox, shanks; no bus step

• tier 0: claude-code, dunst, kernel, pacman
• ace: receives claude-code, kernel, pacman
• g14: receives claude-code, dunst, kernel, pacman
• novox: receives claude-code, kernel, pacman
• shanks: receives claude-code, dunst, kernel, pacman
• claude-code keeps data (ADR 0233): what it holds is backed up before it moves
• grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before claude-code, dunst, kernel, pacman is judged on its first machine

Group feat/upgrade-safety: novox/hq@333e73830c, novox/mesh-catalog@86ba54b06e, novox/mesh-controller@abe356a262, novox/mesh-sdk@8892a11768

• composed together: pass — every machine composes with the change as it did without (4 of 4 compose)

Machines

• mesh-controller on every machine running it: sent
• mesh-sdk on every machine running it: sent
• build-agent on every machine running it: sent
• kernel on every machine running it: sent
• mesh-tools on every machine running it: sent
• pacman on every machine running it: sent
• audit-logger on every machine running it: sent
• baserow on every machine running it: sent
• bazarr on every machine running it: sent
• bookshelf on every machine running it: sent
• confluence on every machine running it: sent
• gitea on every machine running it: sent
• gitlab on every machine running it: sent
• grafana on every machine running it: sent
• home-assistant on every machine running it: sent
• icecast on every machine running it: sent
• influxdb on every machine running it: sent
• jackett on every machine running it: sent
• jira on every machine running it: sent
• kometa on every machine running it: sent
• lab on every machine running it: sent
• letta on every machine running it: sent
• lidarr on every machine running it: sent
• mesh-catalog on every machine running it: sent
• model-usage on every machine running it: sent
• mosquitto on every machine running it: sent
• netcheck on every machine running it: sent
• nextcloud on every machine running it: sent
• nodered on every machine running it: sent
• ombi on every machine running it: sent
• openai-consumer on every machine running it: sent
• radarr on every machine running it: sent
• redis on every machine running it: sent
• route-adapter on every machine running it: sent
• searxng on every machine running it: sent
• showcase on every machine running it: sent
• sonarr on every machine running it: sent
• tautulli on every machine running it: sent
• umami on every machine running it: sent
• unifi on every machine running it: sent
• zsh on every machine running it: sent

Transitions

• 2026-10-22 16:55 (new) → proposed (announced): novox/mesh-catalog#450's head announced
• 2026-10-22 17:14 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit
• 2026-10-22 17:14 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
• 2026-10-22 18:16 ready → proposed (trunk moved): out of date: judged against trunk 835bf22b7d, the trunk is now b5f812270eae; judged again on a recheck or when it is merged
• 2026-10-22 18:20 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit
• 2026-10-22 18:20 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
• 2026-10-22 18:36 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
• 2026-10-22 18:36 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path — not a failed walk that carried a later commit for it, after which it waits for a walk of its own commit
• 2026-10-22 18:50 delivering → failed (failed): its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

<!-- mesh-delivery:view --> **Delivery** `novox/mesh-catalog@86ba54b06e09` — **failed** since 2026-10-22T18:50:19Z **Covered**: covered after its failure: a later walk that contains it was delivered: plan-1792694058124013462 (novox/mesh-catalog at 2f05e707 into main, done, contains its merge 2f05e707) **Delivery plan** — builds claude-code, dunst, kernel, pacman → ace, g14, novox, shanks; no bus step - tier 0: claude-code, dunst, kernel, pacman - ace: receives claude-code, kernel, pacman - g14: receives claude-code, dunst, kernel, pacman - novox: receives claude-code, kernel, pacman - shanks: receives claude-code, dunst, kernel, pacman - claude-code keeps data (ADR 0233): what it holds is backed up before it moves - grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before claude-code, dunst, kernel, pacman is judged on its first machine **Group** `feat/upgrade-safety`: novox/hq@333e73830c31, novox/mesh-catalog@86ba54b06e09, novox/mesh-controller@abe356a262e8, novox/mesh-sdk@8892a1176800 - composed together: pass — every machine composes with the change as it did without (4 of 4 compose) **Machines** - mesh-controller on every machine running it: sent - mesh-sdk on every machine running it: sent - build-agent on every machine running it: sent - kernel on every machine running it: sent - mesh-tools on every machine running it: sent - pacman on every machine running it: sent - audit-logger on every machine running it: sent - baserow on every machine running it: sent - bazarr on every machine running it: sent - bookshelf on every machine running it: sent - confluence on every machine running it: sent - gitea on every machine running it: sent - gitlab on every machine running it: sent - grafana on every machine running it: sent - home-assistant on every machine running it: sent - icecast on every machine running it: sent - influxdb on every machine running it: sent - jackett on every machine running it: sent - jira on every machine running it: sent - kometa on every machine running it: sent - lab on every machine running it: sent - letta on every machine running it: sent - lidarr on every machine running it: sent - mesh-catalog on every machine running it: sent - model-usage on every machine running it: sent - mosquitto on every machine running it: sent - netcheck on every machine running it: sent - nextcloud on every machine running it: sent - nodered on every machine running it: sent - ombi on every machine running it: sent - openai-consumer on every machine running it: sent - radarr on every machine running it: sent - redis on every machine running it: sent - route-adapter on every machine running it: sent - searxng on every machine running it: sent - showcase on every machine running it: sent - sonarr on every machine running it: sent - tautulli on every machine running it: sent - umami on every machine running it: sent - unifi on every machine running it: sent - zsh on every machine running it: sent **Transitions** - 2026-10-22 16:55 (new) → proposed (announced): novox/mesh-catalog#450's head announced - 2026-10-22 17:14 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit - 2026-10-22 17:14 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-22 18:16 ready → proposed (trunk moved): out of date: judged against trunk 835bf22b7d40, the trunk is now b5f812270eae; judged again on a recheck or when it is merged - 2026-10-22 18:20 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit - 2026-10-22 18:20 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-22 18:36 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-22 18:36 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path — not a failed walk that carried a later commit for it, after which it waits for a walk of its own commit - 2026-10-22 18:50 delivering → failed (failed): its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

mesh-tools merged commit 2f05e70734 into main 2026-10-22 18:34:16 +00:00

mesh-tools deleted branch feat/upgrade-safety 2026-10-22 18:34:16 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

mesh/ready

Reviewed and ready: mesh-delivery merges at the marked head once its checks pass (hq ADR 0364)

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

agent-ace agent-g14 agent-novox agent-shanks jschoubben mesh-tools miel.truyen nox-agent

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-catalog#450