Commit Graph
42 Commits
Author SHA1 Message Date
jschoubben 3e8bd9b0de Builder: count only deliverable asks ahead, drop the old message of an ask asked again, and read a seat's queue (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-delivery@bcb5aafdbe80
mesh/check pass 3e8bd9b0def2 on bcb5aafdbe80 | unit 168 run, 472 kept | integration 178 run, 538 kept | composition passed on shanks
mesh-delivery #51's re-ask never fired for mesh-build-node.4e7b118e: messages the
SEAT_NODE_BUILD_GO workers had given up on stay on the work queue, and each counted
as waiting ahead, so the ask was never asked again (note 62). A message ahead now
waits only when no node wrote its step and no ask of its kind asked after it was
taken; the old message of an ask asked again is deleted from its stream; and
builder_seat_queue (answered by the builder through its queues state, as only its
account may read the stream) shows each message and whether it still waits.
2026-10-26 01:19:30 +01:00
jschoubben 9db15c7cf4 Builder and checker read tools read only the keys they need, in parallel, at most 1000 a call (issue 1827)
mesh/delivery merged: source.main moved to novox/mesh-delivery@b842b9422d2a
mesh/check checking 9db15c7cf4fc on b842b9422d2a
builder_stuck, checker_kept, checker_verdicts, checker_status, checker_previews, checker_pull and checker_try listed a
bucket of thousands of keys and read each value (builder_stuck also its history) one round trip at a time before
filtering, so they timed out at 30 s. Now keys are narrowed by what the key carries first, history is read only for rows
kept, values are read sixteen at a time, at most 1000 a call with the answer saying it was cut, steps newest first, a new
try reads only the newest 64 tries, and each tool logs the values read and the time taken. Same kind as the view's fix
(PR 52); the helper is copied, not put in mesh-sdk, as a few lines do not earn a group delivery.
2026-10-26 00:39:08 +01:00
jschoubben c623e8d235 Builder: ask again an ask its stream can no longer answer, and give it tools to read and ask again (issue 1945)
A build ask its seat's stream gave up on, dropped or ended with no step was
waited for with no end, and on restart taken over and waited for again: no
build node would ever answer it, and nothing could ask it again (trunk key
of mesh-build-node at 4e7b118e, building since 19:06; seven keys at 18:44).

At the step bound, a context's bound and no timer of its own, the builder
reads what the bus says of an ask with no step: gone from its stream, or
first in line while a node took a later ask. Such an ask is asked again
under a new id the key names, logged and said as build.asked-again, at most
three askings, then the key fails with why. The bus's max-deliveries
advisory would tell it sooner, but the builder is granted neither it nor the
workers' information, and the controller is not changed here.

builder_stuck reads every waiting key with its asks, askings and steps;
builder_ask_again writes the new reasks state, which the builder watches
and answers.
2026-10-25 22:29:03 +01:00
jschoubben 8d38c6bac9 Write the source revision a candidate was previewed onto, so the builder merges its preview onto that trunk head
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: integration: mesh-delivery-indexer/cmd/mesh-delivery-indexer.TestIndexerCountsOnlyTheAssignedBuildNodesToolchains: within 5 s of n2…
Candidate.Base was never written non-zero (review round 2): the builder's base lookup would have found no trunk for any
live candidate and built no preview at all. The indexer now writes, whenever it writes a candidate's fingerprints, the gate's
source revision whose trunk heads it read the previews onto; that head of the pull request's repository is the preview's
trunk. No mesh-sdk change: the field and its meaning were there. The builder's test now takes the candidate the indexer's
own test writes (testdata/candidate-written.json, kept equal to it), not a hand-made one.
2026-10-25 21:16:20 +01:00
jschoubben 9afef800c0 Merge a candidate's preview onto the trunk head it was previewed onto, not the module value's commit
The builder took a candidate build's merge base from the module's value commit. Now that a value keeps the commit its
fingerprint first appeared at, that commit can be many merges behind the trunk, so a preview would be built onto a stale
trunk (review round 1). The base is now the head of the pull request's repository in the gate's source revision the
candidate names (Candidate.Base): the builder reads mesh-delivery-gate.source, watched with its history (a direct get of
one revision is not what a read grants), and a candidate heard before its revision is built once the revision is heard.
2026-10-25 21:13:59 +01:00
jschoubben f0c5be8c30 Say which artifacts the store check asks about: bundles and archives, whose blobs the collector deletes (issue 1937) 2026-10-24 04:19:17 +02:00
jschoubben 80a0183567 Build a done key again when the artifact store no longer holds its blob, and on the controller's artifacts-gone (issue 1937)
The store's nightly collector deleted the bare blobs of node-tools', mesh-host's and the forge bridge's done keys, which
the controller had never taken in, and the builder kept them done: their next delivery would refuse. The builder now asks
the controller's held before it leaves a done trunk key done (once per key per process, at its start and on each module
value), claims a key whose blob is gone building as a new revision and says so in its log and the key's step, and hears
artifacts-gone, said when a take-in found a gone blob. No timer: a start, a value or an event.
2026-10-24 04:17:01 +02:00
jschoubben b83af04f0c Read an ask's stream at take-over by its name, as the broker grants it, and wait for the ask when that read fails rather than fail a key whose build may still finish (issue 1910)
mesh/delivery-group group feat/r4g rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: novox: novox/mesh-catalog#464: checks/health/go.mod names mesh-sdk a9560890ee26, which is neither on mesh-sdk's trunk nor the group…
2026-10-23 16:59:37 +02:00
jschoubben 357e28f31d Merge remote-tracking branch 'origin/feat/r4g' into feat/r4g
# Conflicts:
#	modules/mesh-delivery-builder/internal/build/run_test.go
2026-10-23 16:46:05 +02:00
jschoubben 5ddc221833 Bound a build or check ask's step from its start, and drop the checker's whole-check bound, so asks waiting in the streams during a large rebuild are never failed and asked twice (issue 1887, review round 2 item 1; ADR 0525 §3, §6) 2026-10-23 16:45:35 +02:00
jschoubben 84b2c92ccf Put again, under its own id, an ask a build key names that neither its stream holds nor any node ran, so a builder that stopped between naming and putting it leaves no key waiting for an ask nobody put (issue 1910) 2026-10-23 16:39:56 +02:00
jschoubben 9f27e2b1f8 Build each build module's part of a fingerprint as one ask on its seat's stream, named in the key before it is put, so the builder chooses no node, holds no capacity and never asks twice (issue 1887 A4a, A4b, A4c, ADR 0525 §1-§6) 2026-10-23 15:57:03 +02:00
jschoubben 9fad90ee89 Say that the builder writes references only once each build module answered one (issue 1886, ADR 0522 review)
mesh/merge-gate head 9fad90ee890a merged into trunk b714eab995a5 — pass: builds mesh-delivery-builder → novox; no bus step; the change touches no module of…
mesh/repo-check head 9fad90ee890a merged into trunk b714eab995a5 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/1886-builder-says-references merged: every member is merged
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
The references come from the build modules' own pushes, so a new builder beside
an old build module writes none; saying it at start would send the controller
to keys that fail not stored.
2026-10-23 00:52:50 +02:00
jschoubben 7cc97cd6b6 Name a candidate's named paths in its fetch, and prove on a real bus that claude-code's manifest is built (issues 1758 and 1751) 2026-10-22 18:20:59 +02:00
jschoubben 4bb04a47f9 Build each fingerprint on one node and drop the second-node retake, as ADR 0478 §4 has it from R4e (issue 1587, B9)
A key done on one node before R4e is not built again for a second; a key failed for differing digests stays as it is.
2026-10-22 18:18:59 +02:00
jschoubben 7651e47cc6 Write a module with no artifact a done key with no artifact, so the controller has a done key to register it from (issue 1613, ADR 0478 §1, ADR 0501 §1)
Moves the mesh-sdk pin to feat/r4e-sdk 9cb4d59, which accepts such a key and names delivery.WaitsFor; the builder's
package wait now uses that shared prefix (B7).
2026-10-22 18:03:02 +02:00
jschoubben 957afa6926 Ask the fetch with mesh-sdk's FetchRequest, naming a module's directory and the paths its kept inputs hold, so a module naming its paths is built from those alone (issue 1758, ADR 0499 §4; the lead's fetch interface) 2026-10-22 18:02:08 +02:00
jschoubben 2e567b9b30 Let a key whose lock names an unpublished package version wait queued, naming it, and build it once the builder's own done key publishes it (issue 1587, B7, ADR 0500 §6)
The prefix 'waits for: ' is kept in the builder until mesh-sdk's delivery.WaitsFor lands on feat/r4e-sdk.
2026-10-22 18:01:33 +02:00
jschoubben 851811a318 Build a candidate's done package key again once the trunk names it, so the package is published from the trunk (issue 1587, B6, ADR 0498 §6) 2026-10-22 18:00:19 +02:00
jschoubben 2399a4f4cc Build a TypeScript package with the Node build module, published only for a trunk fingerprint; a version taken fails the key until the fingerprint changes (issue 1587, B5, ADR 0498 §5, ADR 0500 §4) 2026-10-22 17:59:43 +02:00
jschoubben 081f32d6c9 Copy a pinned upstream image with mesh-build-image's copy step, with no fetch, so its key names the registry's reference (issue 1587, B4, ADR 0498 §4) 2026-10-22 17:59:19 +02:00
jschoubben c26f909152 Send a pinned upstream image to mesh-build-image's copy and fail an unpinned one 'not pinned:' until its fingerprint changes; refusals use mesh-sdk's RefusedKind (issue 1587, B3, ADR 0498 §4) 2026-10-22 17:58:33 +02:00
jschoubben 9871ed2d09 Build a key done before R4e again once, so every bundle and archive the controller delivers names its stored reference (issue 1587, B2, ADR 0498 §8) 2026-10-22 17:57:35 +02:00
jschoubben f26ff8f8bb Write a key done only when every bundle and archive names where it is stored, a candidate's too, so a merge builds nothing its check built (issue 1587, B1, ADR 0498 §1-§3)
Pins mesh-sdk feat/r4e-sdk a90f523 (R4e's group member). A trunk key's builds are asked to publish; a build answering
no reference fails the key 'not stored:', and a step's own 'not stored:', 'not pinned:', 'version taken:' or
'refused kind:' opens the key's error.
2026-10-22 17:57:01 +02:00
jschoubben db62e82e26 Write a failed publish of a merged image to the key; read mesh-build-image's captured steps (review r2)
mesh/repo-check head db62e82e26d7 merged into trunk 47ab8cd3cf45 beside mesh-catalog#435@7c5d11e5 — pass: its merge-check.sh passed
mesh/merge-gate head db62e82e26d7 merged into trunk 47ab8cd3cf45 — pass: group feat/r4c9-image-build composed: every machine composes with the change as it…
mesh/delivery delivered
mesh/delivery-group group feat/r4c9-image-build merged: every member is merged
mesh/check pass db62e82e26d7 on 2b635e2138ae | unit 366 run 0 kept | int 405 run 0 kept | clone 9.2s unit 191ms int 6.7s+44s | composition not yet run
2026-10-22 10:37:21 +02:00
jschoubben 12f61998c5 Publish a merged candidate's image once; a retaken key builds only what is missing (review r1) 2026-10-22 10:33:52 +02:00
jschoubben bbc690539b Build an image on one node through mesh-build-image; only a trunk key publishes (issue 1705) 2026-10-22 10:19:26 +02:00
jschoubben 7214bf4368 Fail a candidate's build when its fetch answers files in conflict or no merge, so the trunk's build is not stored under the candidate's fingerprint (review D4) 2026-10-21 21:50:30 +02:00
jschoubben 66908d9bbc The builder builds the fingerprints a candidate carries, once each, from the head merged onto the trunk (issue 1684) 2026-10-21 21:28:38 +02:00
jschoubben e554094b74 The builder uses mesh-sdk's buildrun package for the caller, the steps, the places, the presence, toolchain and have sets and the choice of nodes, so the checker shares them (issue 1671) 2026-10-21 21:25:00 +02:00
jschoubben 46702d1af3 Keep the builder off a node whose capacity setting is 0, and start the setting at shanks=3,ace=2,g14=0,novox=0 (issue 1661)
mesh/merge-gate head 46702d1af311 merged into trunk 2000c841bf70 — pass: builds mesh-delivery-builder, sent nowhere; no bus step; every machine composes wi…
mesh/repo-check head 46702d1af311 merged into trunk 2000c841bf70 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 13:46:14 +02:00
jschoubben dcf62b0413 Make the builder's capacity a module setting, hold back a failed second-node build until the nodes change, and close at start a key left building for a fingerprint the module no longer has (issues 1661, 1664, 1660) 2026-10-21 13:34:39 +02:00
jschoubben bfc64ac65e Build an archive with no toolchain, mark digests that differ and a one-node key so each is handled, reserve a place when a node is chosen, choose only nodes whose build module is healthy, and take over any key found building (issue 1607, review round 4)
mesh/merge-gate head bfc64ac65ea1 merged into trunk 26de6bc62558 — pass: builds mesh-delivery-indexer, new: modules/mesh-delivery-builder → novox; no bus s…
mesh/repo-check head bfc64ac65ea1 merged into trunk 26de6bc62558 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 13:15:28 +02:00
jschoubben 27e4c3891f Run at most a node's build capacity of build steps on it at once, and hold a step back for a place with no timer (issue 1607, round 3) 2026-10-21 12:41:42 +02:00
jschoubben f555a05cfb Build a key only with the toolchain its fingerprint names, accept '.' as a root module's path, and choose only online nodes (issues 1615, 1607 review) 2026-10-21 11:36:32 +02:00
jschoubben 0b0cfb6e37 Test the builder whole over the real bus with fake build modules, and look again at a value that arrives while its key is built (issue 1607, AC6, AC7) 2026-10-21 11:25:13 +02:00
jschoubben 46c9f33d64 Build each new-form fingerprint through the build modules on two nodes and write its key done or failed (issue 1607, AC7, AC8) 2026-10-21 11:23:39 +02:00
jschoubben 71bb5a5fb3 Watch the build modules' toolchains and steps and keep the toolchains by node, with no timer (issue 1607, AC6) 2026-10-21 11:21:46 +02:00
jschoubben 3298ad58d3 Run a build module's long step: call it on its node, read the result at its steps key, bound the wait (issue 1607, AC5) 2026-10-21 11:21:14 +02:00
jschoubben 74d11f95a2 Choose build nodes by the agreed toolchain and fewest running steps, and agree the two nodes' digests (issue 1607, AC4) 2026-10-21 11:20:31 +02:00
jschoubben 1e6d00b4c7 Claim a build key by create and compare-and-set, never twice, retry a failed key except a refused kind (issue 1607, AC3) 2026-10-21 11:20:09 +02:00
jschoubben 87e550681f Plan a manifest's artifacts for the build modules and refuse image, upstream and package by kind (issue 1607, AC2) 2026-10-21 11:19:38 +02:00