Give the base build several mirrors and retry a failed pacman step (issue 700) #106

2026-10-14T05:39:41Z

mesh-admin commented

2026-10-14 05:39:41 +00:00

What

base build writes /etc/pacman.d/mirrorlist into the builder before anything is installed, then reads it back. The list is six mirrors in order (MIRRORS): Arch's own geo-routed mirror, then netcologne, fau.de, liteserver.nl and leaseweb, then the stock image's mirrors.kernel.org last. pacman tries the next server for each file it could not fetch.
Every pacman step goes through pacmanIn. A failed step is run again after 10 s, 30 s and 90 s (four attempts in all). Each failure says which mirror failed, read from pacman's own words. When the last attempt fails, the build stops with an error naming every mirror that failed.
buildBaseImage takes its hypervisor calls and its clock as an optional argument, so a test can run a whole base build without incus.

Design choice

The list is fixed rather than ranked at build time. rankmirrors and reflector are packages that would have to be installed first, from the one mirror that is the problem, and they give a different list on every build. Each listed mirror answered for core.db on 2026-10-14. A mirror that goes away costs a fallback, not a build. The reason is written above MIRRORS in src/lifecycle/base.ts.

Why

The stock images:archlinux/current names only mirrors.kernel.org. On 2026-10-14 the lab's base rebuild on shanks (lab_base) failed twice in a morning: timeouts installing docker, then a 500 on wireguard-tools.

Tested

npm test: 187 pass, and the typecheck passes. The 5 new tests in test/base-mirrors.test.ts fail on main, which has no mirrorlist, no pacmanIn and no injectable dependencies. They cover:
- the list: at least 4 hosts, kernel.org last, no single quote, written before the first pacman;
- a step that fails twice and is retried after a growing pause, saying which mirror failed;
- a step that keeps failing: four attempts, then an error naming every mirror;
- a whole base build that rides out one failure on every package;
- reading the failed mirrors from what pacman said.
After the merge, the base image is rebuilt on shanks with lab_base {"rebuild": true}, which builds from mesh-lab main.

Fixes issue 700.

## What - `base build` writes `/etc/pacman.d/mirrorlist` into the builder before anything is installed, then reads it back. The list is six mirrors in order (`MIRRORS`): Arch's own geo-routed mirror, then netcologne, fau.de, liteserver.nl and leaseweb, then the stock image's mirrors.kernel.org last. pacman tries the next server for each file it could not fetch. - Every pacman step goes through `pacmanIn`. A failed step is run again after 10 s, 30 s and 90 s (four attempts in all). Each failure says which mirror failed, read from pacman's own words. When the last attempt fails, the build stops with an error naming every mirror that failed. - `buildBaseImage` takes its hypervisor calls and its clock as an optional argument, so a test can run a whole base build without incus. ## Design choice The list is fixed rather than ranked at build time. rankmirrors and reflector are packages that would have to be installed first, from the one mirror that is the problem, and they give a different list on every build. Each listed mirror answered for core.db on 2026-10-14. A mirror that goes away costs a fallback, not a build. The reason is written above `MIRRORS` in src/lifecycle/base.ts. ## Why The stock images:archlinux/current names only mirrors.kernel.org. On 2026-10-14 the lab's base rebuild on shanks (`lab_base`) failed twice in a morning: timeouts installing docker, then a 500 on wireguard-tools. ## Tested - `npm test`: 187 pass, and the typecheck passes. The 5 new tests in test/base-mirrors.test.ts fail on main, which has no mirrorlist, no `pacmanIn` and no injectable dependencies. They cover: - the list: at least 4 hosts, kernel.org last, no single quote, written before the first pacman; - a step that fails twice and is retried after a growing pause, saying which mirror failed; - a step that keeps failing: four attempts, then an error naming every mirror; - a whole base build that rides out one failure on every package; - reading the failed mirrors from what pacman said. - After the merge, the base image is rebuilt on shanks with `lab_base {"rebuild": true}`, which builds from mesh-lab main. Fixes issue 700.

mesh-admin added 1 commit 2026-10-14 05:39:41 +00:00

Give the base build several mirrors and retry a failed pacman step (issue 700)

mesh/merge-gate head b053dc66bf9d merged into trunk c0b1225f6d7d — pass: the change touches no module of the mesh's graph

Details

mesh/repo-check head b053dc66bf9d merged into trunk c0b1225f6d7d — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…

Details

mesh/delivery delivered

Details

b053dc66bf

The stock Arch image names one mirror, and the base build failed twice in
a morning when mirrors.kernel.org timed out and then answered 500. Write a
fixed list of six mirrors before anything is installed, so pacman falls
back per file, and retry a failed step three times with a growing pause,
saying which mirror failed.

mesh-admin commented

2026-10-14 05:40:11 +00:00

Delivery novox/mesh-lab@b053dc66bf9d — delivered since 2026-10-14T05:46:06Z

Delivery plan — builds nothing: the change touches no module of the mesh's graph

Transitions

2026-10-14 05:40 (new) → proposed (announced): novox/mesh-lab#106's head announced
2026-10-14 05:44 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit
2026-10-14 05:44 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
2026-10-14 05:46 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
2026-10-14 05:46 published → delivered (done): nothing for a walk to move

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-lab@b053dc66bf9d` — **delivered** since 2026-10-14T05:46:06Z **Delivery plan** — builds nothing: the change touches no module of the mesh's graph **Transitions** - 2026-10-14 05:40 (new) → proposed (announced): novox/mesh-lab#106's head announced - 2026-10-14 05:44 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit - 2026-10-14 05:44 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-14 05:46 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-14 05:46 published → delivered (done): nothing for a walk to move The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

mesh-admin reviewed

2026-10-14 05:45:38 +00:00

mesh-admin left a comment

Verdict: sound, mergeable at b053dc66bf.

Independent review for issue 700 (the base build fails on one bad mirror).

Checks at this head: mesh/merge-gate success ("PASS — the change touches no module of the mesh's graph"), mesh/repo-check success ("PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), replays/merge-check.sh (go), asks/merge-check.sh (go)"), mesh/delivery success ("ready: waits for its merge"). No mesh/delivery-group is set; this pull request is in no group.

Mirrors. All six are https and official Arch mirrors. geo.mirror.pkgbuild.com is Arch's own; netcologne, ftp.fau.de, liteserver.nl, leaseweb and kernel.org are on the Arch mirror list. Each uses the standard $repo/os/$arch layout. A HEAD for core/os/x86_64/core.db from shanks at 07:45 returned 200 from five of them. mirrors.kernel.org answered 301 to mirrors.edge.kernel.org, which pacman follows. liteserver's core.db is a day older (12 Oct 18:53, against 13 Oct 17:51 on the others); as a fourth fallback that only costs a retry to the next mirror. The written list is single-quoted, so $repo and $arch stay literal for pacman, and it is read back before the first pacman step.
Signature checking is untouched. Nothing in src/ names SigLevel or pacman.conf, and only /etc/pacman.d/mirrorlist is written. A mirror can serve a stale or broken package, but not one that passes the keyring's check.
The retry is bounded. pacmanIn throws once attempt >= PACMAN_BACKOFF_MS.length, so it makes at most 4 tries with waits of 10, 30 and 90 s. Each try keeps its 600 s timeout, so the worst case for one step is about 42 minutes. The test "a step that keeps failing stops the build" holds this at 4 runs. Every failure is retried, including ones that are not transient (a file conflict, say); that costs time only, and each retry is logged with its reason.
The tests fail on main. test/base-mirrors.test.ts against main's src fails at import (no export MIRRORS), so this proves only that the code is new. Its 5 tests exercise the behaviour through injected incus and sleep: the order of the list, the retries and backoff, the stop after 4, a whole build with one failure per package (10 pacman runs), and the mirror names read from what pacman said.

At this head, niced: typecheck clean; npm test 187 pass, 0 fail, 2 skipped. No lab bed ran; the base build itself was not run.

Reviewed by an independent reviewer on g14.

Verdict: sound, mergeable at b053dc66bf9d. Independent review for issue 700 (the base build fails on one bad mirror). Checks at this head: mesh/merge-gate success ("PASS — the change touches no module of the mesh's graph"), mesh/repo-check success ("PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), replays/merge-check.sh (go), asks/merge-check.sh (go)"), mesh/delivery success ("ready: waits for its merge"). No mesh/delivery-group is set; this pull request is in no group. 1. Mirrors. All six are https and official Arch mirrors. geo.mirror.pkgbuild.com is Arch's own; netcologne, ftp.fau.de, liteserver.nl, leaseweb and kernel.org are on the Arch mirror list. Each uses the standard `$repo/os/$arch` layout. A HEAD for core/os/x86_64/core.db from shanks at 07:45 returned 200 from five of them. mirrors.kernel.org answered 301 to mirrors.edge.kernel.org, which pacman follows. liteserver's core.db is a day older (12 Oct 18:53, against 13 Oct 17:51 on the others); as a fourth fallback that only costs a retry to the next mirror. The written list is single-quoted, so `$repo` and `$arch` stay literal for pacman, and it is read back before the first pacman step. 2. Signature checking is untouched. Nothing in src/ names SigLevel or pacman.conf, and only /etc/pacman.d/mirrorlist is written. A mirror can serve a stale or broken package, but not one that passes the keyring's check. 3. The retry is bounded. pacmanIn throws once attempt >= PACMAN_BACKOFF_MS.length, so it makes at most 4 tries with waits of 10, 30 and 90 s. Each try keeps its 600 s timeout, so the worst case for one step is about 42 minutes. The test "a step that keeps failing stops the build" holds this at 4 runs. Every failure is retried, including ones that are not transient (a file conflict, say); that costs time only, and each retry is logged with its reason. 4. The tests fail on main. test/base-mirrors.test.ts against main's src fails at import (no export MIRRORS), so this proves only that the code is new. Its 5 tests exercise the behaviour through injected incus and sleep: the order of the list, the retries and backoff, the stop after 4, a whole build with one failure per package (10 pacman runs), and the mirror names read from what pacman said. At this head, niced: typecheck clean; npm test 187 pass, 0 fail, 2 skipped. No lab bed ran; the base build itself was not run. Reviewed by an independent reviewer on g14.

mesh-admin merged commit f024be6e04 into main

2026-10-14 05:45:50 +00:00

mesh-admin deleted branch fix/700-the-base-build-has-mirrors

2026-10-14 05:45:50 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-lab#106