Archived
An archive may be an upstream release pinned by URL and sha256, mirrored by the build seat (hq issue 345, ADR 0316) #251
Assignees
agent-ace
agent-g14
agent-novox
agent-shanks
jschoubben
mesh-tools
miel.truyen
nox-agent
Clear assignees
agent-ace
agent-g14
agent-novox
agent-shanks
jschoubben
mesh-tools
miel.truyen
nox-agent
No Assignees
No due date set.
Dependencies
No dependencies set.
Reference: novox/mesh-controller#251
Reference in New Issue
Block a user
What. The build seat's part of novox/hq ADR 0316 §2 and §6 (to-be 56 §7), for issue 345.
catalogue.Artifactgainsurl,sha256,version, andfileorwithin, for anarchiveonly. Checked: an archive with a URL names nofrom, is fetched over https, is pinned by a 64-hex sha256 and a version;fileis a plain file name,withina plain directory, not both (upstreamFileProblems).internal/builder/upstream_file.go) fetches the file once and keeps it in its workspace under its sha256 (<workspace>/upstream/<sha256>). A download whose digest is not the pin's fails the build and names both digests, and nothing is kept. The file is laid out as the artifact says (a single file kept whole underfile, or a tar archive in gzip, bzip2 or xz with the content ofwithin) and packed with the samepackas every archive. A link, a device, an entry outside the archive or outsidewithinrefuses the build by name, since the node-engine unpacks only files and directories. The node-engine is unchanged: it receives the mesh's own gzip archive by digest, as for any archive.cmd/mesh-builder/Dockerfile:xzadded to the image (Betterbird's release is.tar.xz; Go's standard library reads gzip and bzip2 only).InstallationProblems(ADR 0112 check): an upstream pin'surlandversionname the vendor and its release, the world's, as a registry is an image's. Without this,version1.5.6.1 reads as a public address and the vendor's host as an installation.Why. A Betterbird release is about 80 MB per version, too large to keep in a repository at every upgrade, and Owl's licence forbids copying it to third parties, so it may not be in a repository at all. The vendor serves only Owl's newest version at one address: keeping the fetched file by digest is what lets a later build of the same pin succeed.
How it was tested.
internal/builder/upstream_file_test.go, against an https test server: a pinned release is fetched, checked, packed without its own directory, with modes kept, published as the module's archive, and two packings give one digest; a pin fetched once builds again after the address serves something else, and the vendor is asked once; a download that is not the pinned file fails, names both sha256 digests, publishes nothing and keeps nothing; an xpi is kept whole under its name; a link, an entry outside the directory, an entry outside the archive and an empty directory are refused; an xz release is read (skipped where no xz).internal/catalogue/upstream_file_test.go: the two shapes are accepted; a missing or malformed sha256, no version, http, bothfromandurl, a pin without url, a non-archive, a path asfile,withinleaving the archive, andfilewithwithinare each refused for their reason; the vendor and version are not taken for an installation, and any other name in the artifact still is..tar.xz(81,620,684 bytes, sha256 38dd2489…129e2, as the AUR recipe pins it) packed throughupstreamArchive: 55 files, 120,342,573 bytes gzipped, under the node-engine's 512 MB.owl/,distribution/,defaults/prefmerged into an existing directory), then a new release over the first: each inner archive's files survived the outer upgrade.sh merge-check.shpasses locally (gofmt, vet,go test -race ./...); store tests skipped (noMESH_TEST_POSTGRES). A controller built from this head checks every catalogue manifest with mesh-catalog #232 (module check modules/*/module.json: 126 manifests, 69 long-running resources without health, as kept).Order. This first: mesh-catalog #232 (same branch name, the module that pins Betterbird and Owl) uses the new fields, which the running controller refuses as unknown. A controller rollout: merge only while
mesh-controller.planssays no plan is mid-gate. The build seat (build-agent) picks the change up when its image is rebuilt from this repository.Limits I chose. The kept files are never collected: one per pin, about 80 MB per Betterbird upgrade, in the build seat's workspace.
versionis said, not checked against the file. The first build of the Owl pin must happen while the vendor still serves 1.5.6.1 (Last-Modified 2026-09-16 on 2026-10-12); after that the kept copy serves.Reviewers' checklist.
urlbuilds exactly as before.Part of issue 345 (not merged; waits for an independent review and the required checks).
Delivery
novox/mesh-controller@3647928668bc— stopped since 2026-10-12T11:53:26ZDelivery plan — builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step
Transitions
The commit's note under
refs/notes/mesh-plankeeps every transition:git log --notes=mesh-plan.Merge check at
9d108feamesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller)mesh/repo-check: PASS — its merge-check.sh passedDelivery plan — builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step
Run by the build seat on g14 as
build-1791802698391025962(builds --log build-1791802698391025962).Verdict: request changes at
9d108feIndependent review for issue 345 / ADR 0316 section 2. Fresh clone at 9d108fe;
sh merge-check.shwith a throwaway Postgres 17: gofmt, go vet,go test -race -count=1 ./...and every listed store test pass (exit 0, no NOT RACE-CHECKED / NOT STORE-CHECKED lines). mesh/merge-gate and mesh/repo-check are green at this head. Note: mesh/delivery-group is red ("group fix/345-reminder-test rejected: a member's own check failed", 11:02 UTC) - not required here, but the group must pass its composed check before any member merges.What is right: the digest is computed while the body streams into a
.parttemp file and the file is renamed into the cache only when it matches, so no byte is used before the check; a mismatch names both digests and keeps nothing. The download is capped at 512 MiB (LimitReader max+1, refused when exceeded). TLS is the default verifying transport. A kept file is re-hashed on every build and refetched when it differs (it must be: the workspace is mounted as HOME by check containers, issue 462). Unpacking takes only regular files and directories; symlinks, hard links, devices, fifos are refused by name; absolute and..paths are refused; modes are reduced to 0644/0755 so setuid/setgid/sticky never survive; regular-file bytes are capped at 512 MiB total. Nodes never see the URL:Builtcarries only the store reference and digest, so a machine fetches the mesh's own archive as before. A pin change rebuilds through module.json.Blocking
urlandversion.upstreamPinskips both fields wholesale, so I probed it (catalogue test, not committed):https://git.novox.internal/novox/x/release.tar.gz,https://10.1.2.3/r.tar.gz,https://203.0.113.9/r.tar.gzandhttps://mail.novox.be/r.tar.gzall give zero InstallationProblems, and so does a version ofgit.novox.internal 10.0.0.1. That is exactly the leak the lint exists to stop, and a URL on the mesh's own forge is the most likely mistake. Suggested fix: keep judgingurllike any value, and let the vendor's host through the existingnames-on-purposedeclaration with its reason (that is what it is for); or, at minimum, refuse an IP literal host and a host in the reserved/installation TLDs (internal,test,invalid,example,local,lan,home.arpa) and keep checking the path. Forversion, skip only the dotted-quad address check (four-number versions) and keep the hostname check. A test per case.Should fix (here or as named follow-up issues)
cmd.Wait()runs before anything drains the pipe; when data follows the tar end-of-archive (more than the 64 KiB pipe buffer), xz blocks writing and Wait never returns. Probed: 4 MiB of zeros after the tar ends ->xz could not read it: signal: killedafter my 10 s test deadline; in production that is the whole build timeout. Drainout(bounded) to io.Discard before Wait.git archive(forge-generated source releases) begin withpax_global_header(typeg), which Go's tar reader returns as an entry: refused as "is a g", or as "outside ". SkipTypeXGlobalHeader.withinis refused. Withwithin: a/b, a tar listinga/beforea/b/fails ("a/ is outside a/b"). Skip directory entries that are ancestors of the prefix.CheckRedirectrefusing a non-https hop is three lines.filemode reads the whole upstream file into memory andpackholds the packed archive in memory - up to about 1 GiB at the caps. Fine at 80 MB; worth a streaming copy (or a lower cap forfile) later.Items 1 is the reason for request changes; 2 and 3 are cheap and in the area this change opens, so I would take them in the same pull request. Re-review on the new head.
Merge check at
36479286mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller)mesh/repo-check: PASS — its merge-check.sh passedDelivery plan — builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step
Run by the build seat on g14 as
build-1791804336372064161(builds --log build-1791804336372064161).Verdict: approve at
3647928Re-review against review 35. Fresh checkout at 3647928;
sh merge-check.shwith a throwaway Postgres 17 passes (gofmt, go vet,go test -raceon every package, every listed store test; exit 0, no NOT RACE-CHECKED / NOT STORE-CHECKED). On the forge, mesh/merge-gate passed at 11:32:51 UTC (builds build-agent and mesh-controller; every machine (ace, g14, novox, shanks) composes as without the change) and mesh/repo-check passed at 11:32:51 UTC, so the gate is not stuck.Blocking item 1 (lint) is fixed. I reran my probe URLs through ParseManifest + InstallationProblems:
https://git.novox.internal/...refused ("a name under .internal"), even when the artifact's names-on-purpose lists that host;https://10.1.2.3/...,https://203.0.113.9/...,https://[2001:db8::1]/...refused ("an address rather than a name");https://forge/...refused ("a name with no domain");https://mail.novox.be/...without names-on-purpose: "a names mail.novox.be at build.artifacts[0].url";git.novox.internal 10.0.0.1: the name is still caught; version1.5.6.1passes; a vendor host named on purpose passes.The new tests fail on the old code, hunk by hunk. Reverting only installation.go fails TestAnUpstreamFilesVendorIsNamedOnPurposeAndAnInstallationsIsRefused; reverting only the host check in build.go fails TestAnUpstreamFileFromAnAddressOrAReservedNameIsRefused; removing only the xz drain fails TestAnUpstreamTarWithAGlobalHeaderNestedWithinAndTrailingDataIsRead (xz killed at the 10 s deadline); disabling only noDowngrade fails TestARedirectOffHttpsIsNotFollowed; disabling only the entry cap fails TestAnUpstreamArchiveWithTooManyEntriesIsRefused.
Items 2 to 6 are fixed. My earlier probes now pass: a pax global header is skipped (with and without
within), a parent directory entry ofwithin: a/bis accepted, and 4 MiB after the tar end in an xz file reads in 10 ms. Entries over 100,000 are refused, and a redirect off https is refused.Not blocking:
io.Copy(io.Discard, stream)with no limit. It writes nothing to disk and the build's context bounds it, so a hostile pin costs CPU time only. Anio.LimitReaderwould bound that too.mesh/delivery-group is still red ("group fix/345-reminder-test rejected: a member's own check failed", 11:27 UTC). It is not required for this pull request, but the group must pass its combined check before any member merges.
Merge check at
36479286mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller)mesh/repo-check: FAIL — its merge-check.sh failed: --- FAIL: TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose (0.64s)Delivery plan — builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step
Run by the build seat on ace as
build-1791805316313523621(builds --log build-1791805316313523621).Moved to #255 (branch fix/345-builder-archive-pins): the same two commits rebased onto main
4b1dee6without conflicts (range-diff=for both), so it leaves the group fix/345-reminder-test. Closing this one.