Commit Graph
894 Commits
Author SHA1 Message Date
jschoubben 315c622386 Lay the builds out before the start reads them, so the controller does not exit on its own start (issue 1121)
mesh/merge-gate waiting in the build queue since 14:36
mesh/delivery superseded: a newer head of the same pull request
Delivery 2 (#381) gated every chain read on the layout marker and laid the
builds out beside the start; a read of the start then met the gate, the
controller exited, and systemd started it again every five seconds from
14:34 on 2026-10-18. The layout now runs once inside attachTheChain, before
anything reads; it skips keys laid out before, so it takes seconds, and a
failure is retried beside the serving as before.
2026-10-18 14:36:04 +02:00
jschoubben 68db18bdfc Make the second merge check pass: the gate's mesh has a chain of its own; the tests follow delivery 2's meaning (issues 1121, 1108, 1109)
mesh/merge-gate head 68db18bdfc1b merged into trunk da6d038e07c0 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 68db18bdfc1b merged into trunk da6d038e07c0 — pass: its merge-check.sh passed
mesh/delivery delivered
The merge gate raises the snapshot's mesh in a throwaway store and now gives it a chain in memory, since the chain lives only on the bus. A module with no commit is carried by no send (a send is read from the commits on the assignments' keys), so the grants step flags only a module with a commit as newly assigned. The tests that sent a module with no build record one, the merge test waits for the merges' consumer before publishing and for the last look of a window before the next test, the pending test follows a chain that deletes nothing, and a wait's settling is a value of its key.
2026-10-18 14:21:35 +02:00
jschoubben e76ba966b3 Merge fix/1121-holds-released: delivery 2 on the builds' new layout (issue 1121)
Delivery 2 deleted the move and chain_moved, so the layout of the builds under their modules is made by the controller's start (LayBuildsOut), recorded by the key layout.done in the builds bucket, retried every half minute with an urgent condition while it fails; the chain on the bus is refused until it exists. Its source readers use the module and pointer reads.
2026-10-18 13:39:24 +02:00
jschoubben c795915315 Keep builds on the bus under their module, so no lookup reads the bucket whole (issue 1121, ADR 0427)
mesh/merge-gate head c79591531575 merged into trunk aa02e7bf4e78 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head c79591531575 merged into trunk aa02e7bf4e78 — pass: its merge-check.sh passed
mesh/delivery delivered
Milestone 5 moved 8,599 build records onto the bus under bare build ids, each
carrying its module's whole manifest. Every build lookup then read the whole
bucket, and a delivery plan's send did that per module per machine while it
held the plans and the machine's declaration: the send of build-agent never
finished, and every other act and the status waited behind it.

Builds now live under <module>.build.<id> (requests under req.<id>, id.<id>
names a build's module), with pointer keys newest, worked, held,
commit.<sha> and source.<reuse key> that move only to a newer build and are
read as hints, repaired when wrong. Manifests are kept once in the same
bucket under manifest.sha256-<hex>, so no grant the running bus lacks is
needed. The old keys are laid out once at start, idempotently, and left
unread. The calls a plan look makes on the silent clocks while it holds the
plans are bounded, since the vendored kv.Keys ignores its context.
2026-10-18 13:32:31 +02:00
jschoubben d095a3dcef Copy node.sent_builds onto the assignments' keys once, and keep the column; list the new store tests (issues 1109, 1096)
Delivery 1 wrote no commit on the key of a pulled-in module, so after the upgrade those modules fell out of SentBuilds. At the start of the lease holder a copy fills every key that names no commit (creating a standing pulled-in key where none stands), overwrites nothing, empties each column it copied, and raises an urgent condition while it fails. Migration 0105 no longer drops node.sent_builds (issue 1112 does).
2026-10-18 05:39:28 +02:00
jschoubben 5b08eb09be Integration tests: no table of the chain in the store after a live delivery; migration 0105 refuses unmoved rows and drops the moved (issue 1111) 2026-10-18 05:33:42 +02:00
jschoubben 472744bbaa Migration 0105 drops the tables of the chain behind a guard; a source scan proves no code names them (issue 1096)
The migration raises first unless chain_moved names builds, rollouts and assignments for every kind that holds rows, and unless pending_assignment holds no open or unanswered row; then it drops build_request, build, batched_merge, release_plan, pending_assignment, assignment, raise_assignment_generation(), node.sent_builds and chain_moved. build_gate stays.
2026-10-18 05:32:14 +02:00
jschoubben 9859923664 Delete the dead table paths, the move, the store's assignments mirror and its tick (issue 1110)
Every method of builds, build requests, rollouts, merges and assignments reads and writes the chain alone: the else branches that read the tables, the movers and MoveChain with their tests, chain_moved reads, ChainReads and the flags that switched a kind, the store's assignments mirror (KeepAssignmentPlans needs none of it: it uses only the AssignmentMirrored type, mirrors.go, and AssignmentValues, provisioning.go) and its tick, the move code of the controller's start, and ForTest's chain_moved row. The listed store tests of the move go with their functions; the tests of the readers, requests, assignments and plans that lived beside them stay in chain_store_test.go. Tests that replayed old data migrations over the build, assignment and pin tables are removed with the tables. The static test that no plan is written but through SavePlansMoving now reads the plan's chain path.
2026-10-18 05:30:19 +02:00
jschoubben fdab58ae4f Read what a node was sent from its assignment keys; a pulled-in module keeps a standing key; a hand send is a time (issue 1109)
SentBuilds answers from the keys of the node's assignments that name a commit, known when the node was sent anything and no key says its commit unknown. A send that names its builds writes the commit, build and rollout on each carried module's key (a module nothing assigns gets a standing key marked pulled-in, never a state) and clears what every other key of the node named; a send by hand writes sent-unknown, a time, on every key and clears their commits. A failed write of these fails the record of the send. node.sent_builds is no longer written. The plan pass ends a pulled-in key only when no send carries it.
2026-10-18 05:22:28 +02:00
jschoubben 3b989d8bb3 Callers of the pending assignments on the bucket: a replaced wait keeps its condition; one read of the key through assign (issue 1108)
A wait that ended unmade and whose condition nobody answered is carried by the wait that replaces it, so the operator is never left without the condition for a module still not on its machine, and taking it back, its condition and the orphan sweep read it there. The condition id stays node.module.<id> with the string id. clearOrphaned says a taken-back wait too. TestAnAssignmentWaitingForItsBuildIsOneKeyOfTheAssignmentsBucket enters through the assign verb and reads the key before and after the build's outcome.
2026-10-18 05:19:32 +02:00
jschoubben 88ea361bf6 Fold a pending assignment into the value of its assignment key (issue 1094)
A wait for a build is the value of <node>.<module> with state pending and a pending object (id p-<unix nanoseconds>, a string), so every reader that skips a value with a state skips it, and the controller's own lists skip it through Stands. The transitions are compare-and-set on the key's revision (claim, release, settle, repoint, mark); assigning over a wait carries it into the value that stands, applied when it was waiting; a plan pass never overwrites it; one open wait per machine and module, and a key that holds the module refuses a new one. The table is no longer read or written, ForgetEndedPending is gone (no key is removed), and a wait of a machine the store dropped is none. Callers change type only; their conditions are the next issue.
2026-10-18 05:17:30 +02:00
jschoubben 0149a3d7de Fix the six tests the merge gate refused: assignments beside pulled-in modules, migration tests on the tables, the event compared as JSON, the restarted move settled to the table (issues 1090, 1092, 1093, 1095)
mesh/delivery-group group feat/chain-on-the-bus stopped: a member was stopped
mesh/merge-gate head 0149a3d7de20 merged into trunk 54070f5bc32b — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 0149a3d7de20 merged into trunk 54070f5bc32b — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-18 03:48:35 +02:00
jschoubben b56c1d00a3 The store's mirror of the assignments may not write while they are switching or moved, asked before every write (issue 1093)
mesh/merge-gate waits for its group's composed check
mesh/repo-check head b56c1d00a3ff merged into trunk 54070f5bc32b beside mesh-catalog#357@68e3326d — fail: its merge-check.sh failed: --- FAIL: TestOneReadO…
mesh/delivery-group group feat/chain-on-the-bus rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
Review round 2. The writer's leave to write is the lease's and AssignmentsMirrorMayWrite, a flag set before the settle of the assignments' switch, kept after the commit and cleared only if the switch fails, so a pass that read its source before the switch lays and purges nothing after it.
2026-10-18 03:38:25 +02:00
jschoubben e16cb4207b Review round 1 of milestone 5: the move settles changed rows under the lock; reports counted by kind; route-proxy at the new SDK; the plan pass never revives an ended assignment; the store's mirror is refused once moved (issues 1090, 1092, 1093)
1090/1092: the move tests pick each report by kind and expect three, all already, on the second run. 1090-1093: under the share lock the switch reads the rows again and writes every key whose value differs from its row (and ends an assignment the store dropped) before recording the kind, so an update the old controller made after the first pass is not lost. 1093: route-proxy is pinned at mesh-sdk f185aedb and vendored; the controller uses only the SDK's asks package. KeepAssignmentPlans decides each key by keepStep, which never makes or revives an assignment. AssignmentsMirrorToWrite refuses once the assignments are moved.
2026-10-18 03:36:05 +02:00
jschoubben a47c63799e Try the move once before serving and again while serving; read each commit's build once when moving assignments (issue 1095)
The first try of the move to the bus is made before the controller serves, and a failure of the kind that is tried again no longer holds the start: it is tried every half minute while the controller serves from the tables. Moving the assignments looked up the build of each commit by scanning the bucket of builds for each of them; it reads the builds once.
2026-10-18 03:23:49 +02:00
jschoubben 703badcf5a Name the commit, build and rollout on the assignment and the assignments on the rollout (issue 1095)
A send that records what a machine was sent writes, in the assignment of each module it carried a build of, the build, what it published, the commit, the rollout that sent it and since when: filled where the assignment names nothing, replaced when the commit moved, never blanked by a push by hand. A plan's sends are made as the plan, so they name it, and the plan lists the assignments each send changed with the build and commit before and after, the builds it carried and the steps taken. The builds, plans and node verbs say the links. node.sent_builds is written and read as before; SentBuilds from the assignments follows with the delivery that drops the column. Acceptance 10 is a test that enters by a merge heard and cut into a plan and writes no key by hand.
2026-10-18 03:22:57 +02:00
jschoubben 0e4feb0d3a Keep assignments in the assignments bucket and move the rows there at start (issue 1093)
Once chain_moved records the kind, Assign, Unassign and Assigned write and read the keys <node>.<module>, and every join of the store that read assignments is made from the bucket: where a module runs, what is behind, what a machine awaits, a given value due for replacement, taking a module, and whether a module may be forgotten (refused while a standing assignment names it). An unassignment ends the key with a marker and keeps its earlier values, as it removed the row; no ended or pending mark is written yet. The generation rises with each change, and a seat held by an assignment goes with it: the foreign key from seat_holding is dropped by migration 0104 and the controller says both. The provisioner's plan is kept in the values by a pass of the tick that writes that one field by compare-and-set; the assignment-changed event is said from what the bucket holds, whoever wrote it. The start moves the rows with what each machine was last sent, keeping a plan the mirror put in a value.
2026-10-18 03:15:44 +02:00
jschoubben 275ddd76e8 Keep rollouts and the merges they answer in the rollouts bucket (issue 1092)
Once chain_moved records the kind, a plan is a key of the rollouts bucket under its id and the open batch is the same key in state assembling. A save refuses a copy another save moved since (the plan's own Revision, kept in its value, counts saves) and keeps the merges the record holds, so a copy read before a merge moved cannot lay it back. The merges a batch or walk answers are the record's held field, with the columns the table had; a merge no plan answers stays, flagged, in the record that held it last, or in the record kept for those. A merge moved to a plan is written to the receiving record first and then taken from the giving one; one found in both is read from the one it entered last and repaired at the next cut. Plans and merges are read from a view kept by one watch of the bucket. A bus that does not answer, a lease lost and a key moved are now not-now like the store being away, so a merge whose plan could not be written is held and handed over again, never acknowledged. The start moves release_plan and batched_merge into the bucket, one kind, counted against both tables.
2026-10-18 03:06:50 +02:00
jschoubben e0fb241263 Keep a build request as the build in state asked (issue 1091)
A request is the build's record under the id its outcome will carry, in state asked; the outcome takes its place and keeps the request beside it, so assign, the pending assignments and the planner read the builds bucket alone. The bucket removes no key, so a request past thirty days is not listed instead of deleted. The move folds build_request into the same records (one kind, builds, counted against both tables) and marks the requests never handed over to be asked again once: claimed before it is asked and settled after, so a crash loses the ask rather than making it twice. Also tests the chain buckets' options on a real bus, the assignments' mirror options included.
2026-10-18 02:53:39 +02:00
jschoubben f89d870308 Keep builds in the builds bucket and move the build rows there at start (issue 1090)
Once chain_moved records the kind, RecordBuild and every build read go through the chain (reuse key and published added); before that the tables are read as before. The serving controller copies the rows at its start, holding the lease: idempotent per key, rows counted against keys, the largest value measured, a refusal (urgent condition, nothing switched) on a count that differs or a value over the bus's largest, and a re-read when the table took a row meanwhile. ForTest attaches the chain in memory with builds moved; ForTestBefore leaves it unmoved. Build requests, gates and the other kinds are still in their tables.
2026-10-18 02:49:51 +02:00
jschoubben b6f8aef4d4 Keep a no-holder machine as ADR 0404 says; prove the cut takes the backups before its first tier (issue 1086, review round 1)
mesh/merge-gate head b6f8aef4d4f1 merged into trunk 47f9290b37aa — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head b6f8aef4d4f1 merged into trunk 47f9290b37aa — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-18 01:57:00 +02:00
jschoubben 7103e33294 A plan's cut takes the backups before asking its first tier, and a machine it cannot ask holds it (issue 1086)
The cut asked tier 0 itself, so the step before a first tier never ran and gitea was sent to novox with no restore point. A machine with no backup holder is now a failed backup that holds the plan.
2026-10-18 01:54:10 +02:00
jschoubben 08e000bb56 Read the backup holders of every machine, and the notes fixture's data at a resource (issues 1065, 1066)
mesh/merge-gate waits for its group's composed check
mesh/repo-check head 08e000bb56aa merged into trunk 7c9c187ca6e2 beside mesh-catalog#353@c898a937 — fail: its merge-check.sh failed: --- FAIL: TestADeliver…
mesh/delivery-group group feat/backup-before-send rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
2026-10-17 23:04:52 +02:00
jschoubben a39d6ce78d Fail a backup whose holder cannot be asked at the bound, not retry forever (issue 1066)
mesh/merge-gate head a39d6ce78d19 merged into trunk 7c9c187ca6e2 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head a39d6ce78d19 merged into trunk 7c9c187ca6e2 — fail: its merge-check.sh failed: --- FAIL: TestADeliveryThatMovesABackedUpModuleWaitsFor…
mesh/delivery-group group feat/backup-before-send rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
2026-10-17 22:53:59 +02:00
jschoubben c531c78eb5 Hold a plan whose backup failed or stayed silent, say it, and ask again by a verb (issue 1066) 2026-10-17 22:48:28 +02:00
jschoubben 9a20547dbd Ask the backup seats before the first tier (issue 1065) 2026-10-17 22:45:09 +02:00
jschoubben 64415f8984 Merge branch 'feat/one-group-one-plan' into g/977 2026-10-17 21:37:37 +02:00
jschoubben 07b050790e Review of issue 977: keep the plan's own order for its gates, hold a group only for a composed mesh-delivery, keep mark-ready one release, compare with the batch's start, fail a lone cycle (issue 977) 2026-10-17 21:37:37 +02:00
jschoubben 66dc501fc4 Split the unit level from the integration level in cmd, builder, link, catalogue, inventory; cheap fixtures for the archive-entry bound, the registry waits and the catalogue copy (issue 895) 2026-10-17 21:32:23 +02:00
jschoubben aac4bae5a4 A group member's own check runs its repository's check alone, a cycle fails the composed check, and mark-ready leaves the delivery seat (issue 977, part D, ADR 0403) 2026-10-17 21:22:55 +02:00
jschoubben 8b3a9163d0 Hold a group's batch open for its last member and keep both kinds of member in one batch and one plan (issue 977, part C, ADR 0418) 2026-10-17 21:06:33 +02:00
jschoubben e43d52c1ea Draw a plan's order per module from the graph and the manifests' change, and read no after: line (issue 977, part B, ADR 0419) 2026-10-17 20:57:30 +02:00
jschoubben b752b1f1df Remove the release-branch judge from the controller, the wire and the build seat (issue 977, part A, ADR 0401) 2026-10-17 20:43:06 +02:00
jschoubben d977f64ec9 Read an empty source ref as the trunk and say a merge into another branch starts no delivery (issue 977, part A, issue 889) 2026-10-17 20:37:58 +02:00
mesh-tools d5d5449835 Merge pull request 'build-agent and route-proxy live whole in mesh-controller (issue 1019, ADR 0420)' (#370) from feat/whole-modules into main 2026-10-17 17:31:58 +00:00
mesh-tools 34197afa01 Judge machines with the node-engine that knows unknown and knows (issue 1024, ADR 0422)
mesh/merge-gate head 34197afa0181 merged into trunk c43772fd94eb — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 34197afa0181 merged into trunk c43772fd94eb — pass: its merge-check.sh passed
mesh/delivery delivered
The gate composes every machine with the node-engine's own validator, vendored at mesh-host of 2026-10-15, which refuses the `unknown` and `knows` keys mesh-host #117 added; every catalogue manifest using them was refused on every machine (mesh-catalog #349). The pin moves to #117's merge, 6469e8238443, delivered; only the declaration validator changes in vendor/. A test holds the gate to taking such a directory.
2026-10-17 16:43:49 +00:00
jschoubben 0125ccd7d2 A member's own merge gate sees a move through the core clones beside it (issue 1043)
mesh/merge-gate out of date: judged beside mesh-catalog#350 at c47d94925811, it is now at 795806efa7ad; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged beside mesh-catalog#350 at c47d94925811, it is now at 795806efa7ad; judged again on a recheck or when it is merged
mesh/delivery-group group feat/whole-modules checking: composed check out of date: novox/mesh-controller@0125ccd7d200 was judged against trunk c43772fd94eb, th…
mesh/delivery superseded: a newer head of the same pull request
2026-10-17 18:40:34 +02:00
jschoubben 48f5a7d3a9 module check refuses an artifact context naming another repository (issue 1019)
mesh/delivery-group group feat/whole-modules rejected: its order contradicts itself: novox/mesh-catalog@c47d94925811 and novox/mesh-controller@48f5a7d3a99a: bu…
mesh/merge-gate head 48f5a7d3a99a merged into trunk c43772fd94eb beside mesh-catalog#350@c47d9492 — fail: builds mesh-controller, route-proxy, new: modules…
mesh/repo-check head 48f5a7d3a99a merged into trunk c43772fd94eb beside mesh-catalog#350@c47d9492 — fail: its merge-check.sh failed: --- FAIL: TestTheBuild…
mesh/delivery superseded: a newer head of the same pull request
2026-10-17 18:18:22 +02:00
jschoubben 4864a43413 The merge gate refuses an artifact context naming another repository (issue 1019) 2026-10-17 18:17:10 +02:00
jschoubben 7ee2b42230 issue 1025: the nested-module check is a file of its own, and the gate test keeps the app module in its repository
mesh/merge-gate head 7ee2b42230dc merged into trunk b8c810a3499c beside mesh-catalog#348@40eed3a0 — pass: builds build-agent, mesh-controller, route-proxy …
mesh/repo-check head 7ee2b42230dc merged into trunk b8c810a3499c beside mesh-catalog#348@40eed3a0 — pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh):…
mesh/delivery delivered
mesh/delivery-group group feat/move-a-module delivered: every member is delivered
2026-10-17 17:27:46 +02:00
jschoubben e42becb479 issue 1025: the bus test makes the controller's consumers before it publishes mesh-delivery's word
mesh/merge-gate head e42becb479a4 merged into trunk b8c810a3499c beside mesh-catalog#348@40eed3a0 — pass: builds build-agent, mesh-controller, route-proxy …
mesh/repo-check head e42becb479a4 merged into trunk b8c810a3499c beside mesh-catalog#348@40eed3a0 — fail: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh):…
mesh/delivery-group group feat/move-a-module rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
2026-10-17 17:18:56 +02:00
jschoubben 43b59c5dff issue 1025: the merge check vets and tests every nested Go module (ADR 0420 §5) 2026-10-17 15:58:46 +02:00
jschoubben 81715273d3 issue 1025: the operator's Approve of a reviewed group registers a moved module from its new repository (ADR 0420 §7) 2026-10-17 15:56:49 +02:00
jschoubben 085b661936 issue 1025: the merge gate judges a group's move of a module as a move (ADR 0420 §6) 2026-10-17 15:43:28 +02:00
jschoubben 21414307c8 issue 1025: a context naming the module's own repository is its own build source (ADR 0420 §4) 2026-10-17 15:41:35 +02:00
jschoubben 0f21fe7888 issue 974: review 2: a check stopped by its key says no verdict, an error judges nothing, and an unacknowledged check goes to a second builder
mesh/merge-gate head 0f21fe7888be merged into trunk 7d80155d1636 beside mesh-catalog#345@94f65ccc — pass: builds build-agent, mesh-controller, route-proxy …
mesh/repo-check head 0f21fe7888be merged into trunk 7d80155d1636 beside mesh-catalog#345@94f65ccc — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/events-and-queues delivered: every member is delivered
2026-10-17 14:12:50 +02:00
jschoubben a83b921af0 issue 974: acceptance 5: two merges within a minute leave each open pull request one queued check 2026-10-17 14:03:26 +02:00
jschoubben 3d09d5235f issue 974: the checks are queued by pull request on two work-queue streams, taken after a delivery build, dropped when judged, stopped when their key moves; the reordering and the forge-asking end are gone 2026-10-17 14:01:33 +02:00
jschoubben f883830975 issue 974: two work-queue streams for the checks of pull requests (CHECKS_READY, CHECKS_OTHER), their workers, grants and writer 2026-10-17 13:53:22 +02:00
jschoubben 96325c295c issue 974: the serving controller follows its own plan-moved, so a plan moved from the terminal is looked at 2026-10-17 13:47:47 +02:00