Builds, rollouts and assignments live on the bus, each assignment naming its commit, build and rollout (design 47 milestone 5, issues 1089-1095) #378

2026-10-18T01:39:27Z

mesh-tools commented

2026-10-18 01:39:27 +00:00

Design 47 milestone 5, delivery 1 of 2 (issue 981; ADR 0408): builds, rollouts and assignments become keys of the buckets mesh-controller_builds, _rollouts and _assignments on the bus. They are written by the lease holder by compare-and-set, keep 64 revisions, and no key is ever removed.

1089: package internal/chain, the three buckets, and migration 0104 (chain_moved; it also drops seat_holding's foreign key to assignment). The controller has no consolidated schema, so migration 0104 alone carries the change.
1090, 1091: builds on the bus, a build request is the build in state asked, and never-handed-over requests are asked again once.
1092: rollouts and the merges they hold (held) on the bus, read from one watched view. A merged event is acknowledged only after its write (ADR 0407 §3).
1093: assignments on the bus. An unassign writes ended (ADR 0408 §4), and the store's mirror stops once the assignments have moved. route-proxy reads through mesh-sdk f185aed.
1095: each assignment names its build, artifact or package, commit and rollout, and each rollout lists the assignments it changed (acceptance 10).
The move at start: per kind, counted, re-read and settled under a share lock, refused loudly (condition chain-move-refused) on a count that differs or a value over 1 MiB. Until a kind is recorded moved, its table is read. It needs the controller's new bus grants, and retries every 30 s until they arrive.
Delivery 2 (issues 1094, 1096) drops the tables and folds pending assignments, once the move is read back live.

The first merge gate at b56c1d00 failed on six integration tests (build-1792287575196340959). 0149a3d7 fixes them, and the same reviewer approved it in the cycle after a failed merge gate.

One delivery group with mesh-catalog and mesh-sdk (branch feat/chain-on-the-bus).

Reviewed-by: reviewer-chain@shanks at 0149a3d7de

Design 47 milestone 5, delivery 1 of 2 (issue 981; ADR 0408): builds, rollouts and assignments become keys of the buckets mesh-controller_builds, _rollouts and _assignments on the bus. They are written by the lease holder by compare-and-set, keep 64 revisions, and no key is ever removed. - 1089: package internal/chain, the three buckets, and migration 0104 (`chain_moved`; it also drops seat_holding's foreign key to assignment). The controller has no consolidated schema, so migration 0104 alone carries the change. - 1090, 1091: builds on the bus, a build request is the build in state `asked`, and never-handed-over requests are asked again once. - 1092: rollouts and the merges they hold (`held`) on the bus, read from one watched view. A merged event is acknowledged only after its write (ADR 0407 §3). - 1093: assignments on the bus. An unassign writes `ended` (ADR 0408 §4), and the store's mirror stops once the assignments have moved. route-proxy reads through mesh-sdk f185aed. - 1095: each assignment names its build, artifact or package, commit and rollout, and each rollout lists the assignments it changed (acceptance 10). - The move at start: per kind, counted, re-read and settled under a share lock, refused loudly (condition chain-move-refused) on a count that differs or a value over 1 MiB. Until a kind is recorded moved, its table is read. It needs the controller's new bus grants, and retries every 30 s until they arrive. - Delivery 2 (issues 1094, 1096) drops the tables and folds pending assignments, once the move is read back live. The first merge gate at b56c1d00 failed on six integration tests (build-1792287575196340959). 0149a3d7 fixes them, and the same reviewer approved it in the cycle after a failed merge gate. One delivery group with mesh-catalog and mesh-sdk (branch feat/chain-on-the-bus). Reviewed-by: reviewer-chain@shanks at 0149a3d7de202a983e50630e3eaf3b3f0bb075a7

mesh-tools added 11 commits 2026-10-18 01:39:28 +00:00

Add the chain package and its buckets: builds, rollouts and assignments with history 64 (issue 1089) fc55785956

internal/chain holds typed Build, Rollout and Assignment and a Backend with no delete or purge, in memory and on the bus. The builds and rollouts buckets are asserted with history 64, no age and values up to just under a mebibyte; the assignments bucket gets the same, keeping its marker age because a stream cannot be made to disallow message ages. Migration 0104 makes the one-row chain_moved switch. The controller keeps no consolidated schema, so 0104 is the whole schema change. Nothing reads the buckets yet.

Say where the assignments bucket is declared why it keeps its marker age (issue 1089) 36c4ce605e

Keep builds in the builds bucket and move the build rows there at start (issue 1090) f89d870308

Once chain_moved records the kind, RecordBuild and every build read go through the chain (reuse key and published added); before that the tables are read as before. The serving controller copies the rows at its start, holding the lease: idempotent per key, rows counted against keys, the largest value measured, a refusal (urgent condition, nothing switched) on a count that differs or a value over the bus's largest, and a re-read when the table took a row meanwhile. ForTest attaches the chain in memory with builds moved; ForTestBefore leaves it unmoved. Build requests, gates and the other kinds are still in their tables.

Keep a build request as the build in state asked (issue 1091) e0fb241263

A request is the build's record under the id its outcome will carry, in state asked; the outcome takes its place and keeps the request beside it, so assign, the pending assignments and the planner read the builds bucket alone. The bucket removes no key, so a request past thirty days is not listed instead of deleted. The move folds build_request into the same records (one kind, builds, counted against both tables) and marks the requests never handed over to be asked again once: claimed before it is asked and settled after, so a crash loses the ask rather than making it twice. Also tests the chain buckets' options on a real bus, the assignments' mirror options included.

Keep rollouts and the merges they answer in the rollouts bucket (issue 1092) 275ddd76e8

Once chain_moved records the kind, a plan is a key of the rollouts bucket under its id and the open batch is the same key in state assembling. A save refuses a copy another save moved since (the plan's own Revision, kept in its value, counts saves) and keeps the merges the record holds, so a copy read before a merge moved cannot lay it back. The merges a batch or walk answers are the record's held field, with the columns the table had; a merge no plan answers stays, flagged, in the record that held it last, or in the record kept for those. A merge moved to a plan is written to the receiving record first and then taken from the giving one; one found in both is read from the one it entered last and repaired at the next cut. Plans and merges are read from a view kept by one watch of the bucket. A bus that does not answer, a lease lost and a key moved are now not-now like the store being away, so a merge whose plan could not be written is held and handed over again, never acknowledged. The start moves release_plan and batched_merge into the bucket, one kind, counted against both tables.

Keep assignments in the assignments bucket and move the rows there at start (issue 1093) 0e4feb0d3a

Once chain_moved records the kind, Assign, Unassign and Assigned write and read the keys <node>.<module>, and every join of the store that read assignments is made from the bucket: where a module runs, what is behind, what a machine awaits, a given value due for replacement, taking a module, and whether a module may be forgotten (refused while a standing assignment names it). An unassignment ends the key with a marker and keeps its earlier values, as it removed the row; no ended or pending mark is written yet. The generation rises with each change, and a seat held by an assignment goes with it: the foreign key from seat_holding is dropped by migration 0104 and the controller says both. The provisioner's plan is kept in the values by a pass of the tick that writes that one field by compare-and-set; the assignment-changed event is said from what the bucket holds, whoever wrote it. The start moves the rows with what each machine was last sent, keeping a plan the mirror put in a value.

End an unassigned key with ended and remove no key of the chain (issue 1093) cb00bbbfcf

ADR 0408 section 4 says no key of the chain is removed, a delete marker included. The Backend loses Remove and the Chain RemoveAssignment. An unassignment, and the end of a value only pulled in by another, write the value again with ended set, by compare-and-set; assigning it again writes a fresh standing value over it. A value with ended or a state is no assignment (Assignment.Stands) to the assignment lists, the joins, forgetting a module, seat holdings, the given values, the move and the assignment-changed event.

Name the commit, build and rollout on the assignment and the assignments on the rollout (issue 1095) 703badcf5a

A send that records what a machine was sent writes, in the assignment of each module it carried a build of, the build, what it published, the commit, the rollout that sent it and since when: filled where the assignment names nothing, replaced when the commit moved, never blanked by a push by hand. A plan's sends are made as the plan, so they name it, and the plan lists the assignments each send changed with the build and commit before and after, the builds it carried and the steps taken. The builds, plans and node verbs say the links. node.sent_builds is written and read as before; SentBuilds from the assignments follows with the delivery that drops the column. Acceptance 10 is a test that enters by a merge heard and cut into a plan and writes no key by hand.

Try the move once before serving and again while serving; read each commit's build once when moving assignments (issue 1095) a47c63799e

The first try of the move to the bus is made before the controller serves, and a failure of the kind that is tried again no longer holds the start: it is tried every half minute while the controller serves from the tables. Moving the assignments looked up the build of each commit by scanning the bucket of builds for each of them; it reads the builds once.

Review round 1 of milestone 5: the move settles changed rows under the lock; reports counted by kind; route-proxy at the new SDK; the plan pass never revives an ended assignment; the store's mirror is refused once moved (issues 1090, 1092, 1093) e16cb4207b

1090/1092: the move tests pick each report by kind and expect three, all already, on the second run. 1090-1093: under the share lock the switch reads the rows again and writes every key whose value differs from its row (and ends an assignment the store dropped) before recording the kind, so an update the old controller made after the first pass is not lost. 1093: route-proxy is pinned at mesh-sdk f185aedb and vendored; the controller uses only the SDK's asks package. KeepAssignmentPlans decides each key by keepStep, which never makes or revives an assignment. AssignmentsMirrorToWrite refuses once the assignments are moved.

The store's mirror of the assignments may not write while they are switching or moved, asked before every write (issue 1093)

mesh/merge-gate waits for its group's composed check

Details

mesh/repo-check head b56c1d00a3ff merged into trunk 54070f5bc32b beside mesh-catalog#357@68e3326d — fail: its merge-check.sh failed: --- FAIL: TestOneReadO…

Details

mesh/delivery-group group feat/chain-on-the-bus rejected: a member's own check failed

Details

mesh/delivery superseded: a newer head of the same pull request

Details

b56c1d00a3

Review round 2. The writer's leave to write is the lease's and AssignmentsMirrorMayWrite, a flag set before the settle of the assignments' switch, kept after the commit and cleared only if the switch fails, so a pass that read its source before the switch lays and purges nothing after it.

mesh-tools commented

2026-10-18 01:39:32 +00:00

Delivery novox/mesh-controller@0149a3d7de20 — stopped since 2026-10-18T09:47:30Z

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

tier 0: mesh-controller, route-proxy
tier 1: build-agent
ace: receives route-proxy, build-agent
g14: receives build-agent
novox: receives mesh-controller, route-proxy, build-agent
shanks: receives build-agent
route-proxy provides route: its consumers are sent again after it
route-proxy keeps data (ADR 0233): what it holds is backed up before it moves
build-agent keeps data (ADR 0233): what it holds is backed up before it moves
grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before route-proxy, build-agent is judged on its first machine

Group feat/chain-on-the-bus: novox/mesh-catalog@3bc4215d87

composed together: pass — every machine composes with the change as it did without (4 of 4 compose)

Transitions

2026-10-18 01:50 (new) → proposed (announced): novox/mesh-controller#378's head announced
2026-10-18 01:53 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit
2026-10-18 01:53 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
2026-10-18 02:18 ready → proposed (recheck): Its mesh/repo-check has stood pending since a sibling's new head at 01:56Z, and only its own check sets it. Since 02:04Z each composed pass (5 so far) set the merge gate only. Each refused merge then asked the composed check again (issue 1082's path), a loop that never ends. Recheck asks its own check (ADR 0421 §12 exception: a check that never answers). Cause recorded on issue 1049.
2026-10-18 02:21 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit
2026-10-18 02:21 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
2026-10-18 09:38 ready → proposed (recheck): Its group feat/chain-on-the-bus was stopped when the plan of mesh-catalog#357 was stopped from Telegram at 09:00Z; the readers it needs were since delivered by rebuilds (releases done 09:31-09:37Z on novox and ace), and mesh-sdk#32 was closed. The operator asked to go on; it is checked again so its Approve is asked on its own.
2026-10-18 09:39 proposed → proposed (re-ask): delivery.novox/mesh-controller_0149a3d7de20.stalled — re-asked as
2026-10-18 09:44 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit
2026-10-18 09:44 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
2026-10-18 09:47 ready → stopped (closed): the pull request closed unmerged

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-controller@0149a3d7de20` — **stopped** since 2026-10-18T09:47:30Z **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller, route-proxy - tier 1: build-agent - ace: receives route-proxy, build-agent - g14: receives build-agent - novox: receives mesh-controller, route-proxy, build-agent - shanks: receives build-agent - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before route-proxy, build-agent is judged on its first machine **Group** `feat/chain-on-the-bus`: novox/mesh-catalog@3bc4215d872c - composed together: pass — every machine composes with the change as it did without (4 of 4 compose) **Transitions** - 2026-10-18 01:50 (new) → proposed (announced): novox/mesh-controller#378's head announced - 2026-10-18 01:53 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit - 2026-10-18 01:53 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-18 02:18 ready → proposed (recheck): Its mesh/repo-check has stood pending since a sibling's new head at 01:56Z, and only its own check sets it. Since 02:04Z each composed pass (5 so far) set the merge gate only. Each refused merge then asked the composed check again (issue 1082's path), a loop that never ends. Recheck asks its own check (ADR 0421 §12 exception: a check that never answers). Cause recorded on issue 1049. - 2026-10-18 02:21 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit - 2026-10-18 02:21 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-18 09:38 ready → proposed (recheck): Its group feat/chain-on-the-bus was stopped when the plan of mesh-catalog#357 was stopped from Telegram at 09:00Z; the readers it needs were since delivered by rebuilds (releases done 09:31-09:37Z on novox and ace), and mesh-sdk#32 was closed. The operator asked to go on; it is checked again so its Approve is asked on its own. - 2026-10-18 09:39 proposed → proposed (re-ask): delivery.novox/mesh-controller_0149a3d7de20.stalled — re-asked as - 2026-10-18 09:44 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit - 2026-10-18 09:44 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-18 09:47 ready → stopped (closed): the pull request closed unmerged The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

jschoubben added 1 commit 2026-10-18 01:49:58 +00:00

Fix the six tests the merge gate refused: assignments beside pulled-in modules, migration tests on the tables, the event compared as JSON, the restarted move settled to the table (issues 1090, 1092, 1093, 1095)

mesh/delivery-group group feat/chain-on-the-bus stopped: a member was stopped

Details

mesh/merge-gate head 0149a3d7de20 merged into trunk 54070f5bc32b — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…

Details

mesh/repo-check head 0149a3d7de20 merged into trunk 54070f5bc32b — pass: its merge-check.sh passed

Details

mesh/delivery delivered

Details

0149a3d7de

mesh-tools commented

2026-10-18 01:56:20 +00:00

mesh-delivery: the operator's Approve of this pull request's merge into main is ended: a head it covers moved on: the heads it covers are no longer the group's heads. Nothing is merged on it. The operator is asked again once the checks pass at the new head.

mesh-tools commented

2026-10-18 09:26:30 +00:00

mesh-delivery: the operator's Approve of this pull request's merge into main is ended: a head it covers moved on: the heads it covers are no longer the group's heads. Nothing is merged on it. The operator is asked again once the checks pass at the new head.

mesh-tools referenced this pull request

2026-10-18 09:43:39 +00:00

Builds, rollouts and assignments live on the bus, each assignment naming its commit, build and rollout (design 47 milestone 5, issues 1089-1095) #379

mesh-tools closed this pull request

2026-10-18 09:47:29 +00:00

mesh-tools referenced this pull request

2026-10-18 10:01:05 +00:00

Builds, rollouts and assignments live on the bus, each assignment naming its commit, build and rollout (design 47 milestone 5, issues 1089-1095) #379

mesh-tools referenced this pull request

2026-10-20 13:57:48 +00:00

Catch up merges the controller did not hear, on the operator's Approve of each (ADR 0457, issue 1336) #407

mesh-tools referenced this pull request