Link a build's bases into its repository before pushing it (issue 694) #318
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: novox/mesh-controller#318
Reference in New Issue
Block a user
What
Before the build agent pushes an image it built, it mounts every blob of each base the build stood on (for this machine's platform) from the base's repository into the image's repository, through the registry API (
Registry.LendBases, new optionalBaseLenderinterface; the scratch publisher forwards it under the scratch prefix). A failure to mount is said in the build log and the push is still tried.Why
Genesis stopped at step 16 of 24 (base):
docker push 127.0.0.1:5000/mesh-tools-go/buildfailed with "not all of them are available locally",NotFound: content digest sha256:e2de9651…. That digest is the first (alpine) layer of the linux/amd64 manifest ofgolang@sha256:8ac98ca5…, the base mesh-tools-go declares, and the same base the control plane was built on earlier in genesis.The bed's docker is a fresh 29.7.2 install, which keeps images in containerd's image store; the live nodes run docker 29.8.2 on the classic
overlay2store (read with docker_daemon_config on novox and shanks). On containerd's store a base layer whose unpacked snapshot already exists is never fetched, so the built image references a compressed layer this machine does not hold. moby's push (daemon/containerd/image_push.go) mounts missing blobs only from distribution-source labels, which a locally built image does not carry. But containerd's pusher asks the target repository first (HEAD per blob) and reads local content only for what it lacks. Once the base's blobs are in the image's repository, the push needs nothing of them from the store, whichever store the runtime keeps. On the classic store nothing changes: the push finds them there.Tested
TestABuildsBaseIsLinkedIntoItsRepositoryBeforeThePush: this platform's blobs are mounted (no bytes moved), another platform's are not, and non-image or foreign build arguments are ignored. A second run mounts nothing.TestABaseTheRegistryWillNotMountIsAnError: a refused mount is reported as an error.TestAnImageIsPushedOnlyAfterItsBasesAreLent: a build lends its bases before it pushes; this test fails without the change.go test ./internal/builder ./cmd/mesh-builderandgo vet ./...pass.Fixes issue 694.
Delivery
novox/mesh-controller@8933864a49d0— delivered since 2026-10-14T01:24:23ZDelivery plan — builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step
Machines
Transitions
The commit's note under
refs/notes/mesh-plankeeps every transition:git log --notes=mesh-plan.Merge check at
8933864aJudged: head
8933864a49d0merged into trunkdf9e5728a925; read beside it: mesh-catalogebf7f3773dc7, mesh-controller-maindf9e5728a925, mesh-host1a2677882380, mesh-tools954e1a53eef9mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller)mesh/repo-check: PASS — its merge-check.sh passedDelivery plan — builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step
Run by the build seat on ace as
build-1791939622585982399(builds --log build-1791939622585982399).Lab run run-2026-10-14T01-00-02-463Z on shanks (genesis-keeps-its-bus, mesh-controller at
8933864a, everything else main): step 16 (base) built mesh-tools and mesh-tools-go with no push error, step 17 adopted postgres, and genesis reached step 18 (broker). Step 18 stopped on a different fault:the nats module's manifest could not be read: open /opt/mesh-catalog/modules/nats/module.json: no such file or directory, although mesh-catalog ebf7f377 has modules/nats. The bed's rerun then stopped at step 14 on a stale pacman index (postgresql-libs 404). Neither is this change.Verdict: APPROVE at
8933864a49. No blocking findings; three minor notes below.Independent review of issue 694 (a build's bases are linked into its repository before the push).
Checks at this head, as the forge keeps them:
8933864a49merged into trunkdf9e5728a9— pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step" (4 of 4 compose)Tests (run niced, on a clone):
go vetandgo test ./internal/builder ./cmd/mesh-builderpass at the head and with the head merged into maindf9e572. On main: with only lend_test.go, the package does not compile (Registry.LendBases undefined), so the two registry tests fail; with lend.go and lend_test.go but without the builder.go wiring, TestAnImageIsPushedOnlyAfterItsBasesAreLent fails behaviourally (the build did onlypush bus/server, the want begins withlend bus/server …). So all three fail on main.Access model. The registry is plain HTTP with no per-repository authorisation (Registry's own comment: the private network is the authentication). The mount uses the standard distribution API (POST /blobs/uploads/?mount=&from=), which the registry only honours when
fromholds the blob, and a mount moves no bytes. It mounts only into the image's own repository (scratch-prefixed through scratchPublisher), and only the blobs of the bases the build itself stood on — content the build agent already read and which the pushed image references anyway. The mirror code already mounts across repositories the same way (source.mountFrom). Nothing becomes readable that was not before. A 202 (registry declined to mount) is cancelled with a DELETE, so no upload is left open.Live mesh. On the classic overlay2 store the push would HEAD each blob and mount or upload it anyway; now the blobs are already there and the push skips them. The image and its digest are unchanged. Cost: one manifest GET per base (two for an index) and a HEAD per blob, plus one new build-log line. The only state difference is note (b) below.
Skipping. buildArgValues reads only the --build-arg values standingOn produced; LendBases skips anything not
name@sha256:…(the@novox/mesh-sdk@1.2.3and tag cases are tested), a repository equal to the target, and a manifest this registry does not hold (404 is not an error). Index entries are filtered on runtime.GOOS/GOARCH, and an entry without a platform (attestations are unknown/unknown) is skipped. The build agent runs on the docker daemon's machine, so the Go runtime's platform is the build's platform. The variant is ignored, which only matters on arm, where at worst a sibling variant's blobs are also mounted.Log-and-continue is sound. The push stays the authority, and it fails loudly naming the missing blob when lending was needed and did not happen. A lending failure never makes a wrong image. The registry client waits for a registry held still, as the push does, so no new hang appears.
Minor notes (not blocking):
(a) LendBases drops the host of the base reference and looks the repository up on r.Address. Today every base is a reference into this registry, so this is right. A foreign-host reference with the same repository name here would mount the same bytes, because mounts are by digest. Comparing where.base with r.Address would make that explicit.
(b) Every base is lent, including one a multi-stage recipe uses only in a build stage, and each base's config blob too. So the image repository gains links to blobs its manifests never mark. No bytes are duplicated and registry GC marks by manifests, but once a mirror is collected those links can outlive their blobs. A HEAD on them then answers not-found, which costs one more lending, so this is harmless.
(c) The first failed mount returns, so the bases after it are not lent. That is acceptable because the push is tried anyway, but the log line says "could not all be linked" without saying how many were linked first.
The lab run on shanks (comment above) shows step 16 (base) pushed mesh-tools-go without the issue 694 error; the later stop at step 18 is unrelated.