novox/mesh-catalog
7
0
Fork 0
Code Issues Pull Requests 7 Actions Packages Projects Releases Wiki Activity

lab: name the base image's age, refuse a stale one, and rebuild it (issue 698) #288

Merged
mesh-admin merged 2 commits from fix/698-the-lab-names-its-base-age into main 2026-10-14 05:12:38 +00:00
Conversation 7 Commits 2 Files Changed 3
+501 -20
mesh-admin commented 2026-10-14 01:21:35 +00:00
Contributor
Copy Link
Copy Source

What

  • lab_base (new): reads the base image mesh-lab/base (when it was built, its age in days, the bound, whether it is stale) and the last three rebuilds, with the newest one's log. By default it only reads. With rebuild: true it clones mesh-lab's main from the forge, installs it the way a run does, and runs src/cli.ts base build in the background, answering at once. Call it again for the build's state and the image's new age. It refuses while a run or another rebuild is going.
  • lab_run refuses while the base is being rebuilt, and refuses an image older than 14 days, naming the remedy (lab_base {"rebuild": true}). A started run says the base's age.
  • lab_check says the base's age, and fails on an image past the bound.
  • The age is read from incus image list mesh-lab/base --format json (uploaded_at, which a publish sets). A missing image, or one whose age cannot be read, is reported but not refused: a bed that needs it says so itself, and one that does not still runs.

Design limit

The bound is 14 days (BASE_MAX_AGE_DAYS, written down at the head of tools/base.ts). The image that failed was 44 days old. A rebuild takes minutes, so a short bound costs little, and refusing a run is cheaper than a bed that fails an hour in for a reason that is not the change under test.

Why

A bed machine carries the package database its base image was built with, and Arch's mirrors stop serving those versions within weeks. In run-2026-10-14T01-00-02-463Z, genesis-keeps-its-bus stopped at step 14 on a 404 for postgresql-libs: the database dated from 2026-08-31. Nothing said the image's age, nothing refused it, and no mesh tool could rebuild it.

Tested

  • node --test --experimental-strip-types 'test/*.test.ts' in modules/lab: 32 pass. The 8 new tests in test/base.test.ts fail on main, which has no tools/base.ts and no lab_base. They cover reading the age, the 14-day refusal and its remedy, a missing or unreadable image, the refusal during a rebuild, the rebuild script (mesh-lab's base build, with the credential removed before npm), rebuild status and lost rebuilds, and the wiring in index.ts (lab_run asks before it starts; lab_base reads unless asked and never rebuilds under a run).
  • Type-checked with tsc using the flags merge-check-typescript.sh uses.
  • Not run against incus here: the lab runs only on shanks. Once delivered, the coordinator rebuilds the base with lab_base and reruns the genesis bed.

Fixes issue 698.

## What - **lab_base** (new): reads the base image `mesh-lab/base` (when it was built, its age in days, the bound, whether it is stale) and the last three rebuilds, with the newest one's log. By default it only reads. With `rebuild: true` it clones mesh-lab's main from the forge, installs it the way a run does, and runs `src/cli.ts base build` in the background, answering at once. Call it again for the build's state and the image's new age. It refuses while a run or another rebuild is going. - **lab_run** refuses while the base is being rebuilt, and refuses an image older than **14 days**, naming the remedy (`lab_base {"rebuild": true}`). A started run says the base's age. - **lab_check** says the base's age, and fails on an image past the bound. - The age is read from `incus image list mesh-lab/base --format json` (`uploaded_at`, which a publish sets). A missing image, or one whose age cannot be read, is reported but not refused: a bed that needs it says so itself, and one that does not still runs. ## Design limit The bound is 14 days (`BASE_MAX_AGE_DAYS`, written down at the head of tools/base.ts). The image that failed was 44 days old. A rebuild takes minutes, so a short bound costs little, and refusing a run is cheaper than a bed that fails an hour in for a reason that is not the change under test. ## Why A bed machine carries the package database its base image was built with, and Arch's mirrors stop serving those versions within weeks. In run-2026-10-14T01-00-02-463Z, genesis-keeps-its-bus stopped at step 14 on a 404 for postgresql-libs: the database dated from 2026-08-31. Nothing said the image's age, nothing refused it, and no mesh tool could rebuild it. ## Tested - `node --test --experimental-strip-types 'test/*.test.ts'` in modules/lab: 32 pass. The 8 new tests in test/base.test.ts fail on main, which has no tools/base.ts and no lab_base. They cover reading the age, the 14-day refusal and its remedy, a missing or unreadable image, the refusal during a rebuild, the rebuild script (mesh-lab's base build, with the credential removed before npm), rebuild status and lost rebuilds, and the wiring in index.ts (lab_run asks before it starts; lab_base reads unless asked and never rebuilds under a run). - Type-checked with tsc using the flags merge-check-typescript.sh uses. - Not run against incus here: the lab runs only on shanks. Once delivered, the coordinator rebuilds the base with lab_base and reruns the genesis bed. Fixes issue 698.
mesh-admin added 1 commit 2026-10-14 01:21:35 +00:00
lab: name the base image's age, refuse a stale one, and rebuild it (issue 698)
mesh/merge-gate head ff9d3ee6a06c merged into trunk ebf7f3773dc7 — pass: builds lab → shanks; no bus step; every machine composes with the change as it did…
Details
mesh/repo-check head ff9d3ee6a06c merged into trunk ebf7f3773dc7 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-…
Details
mesh/delivery superseded: a newer head of the same pull request
Details
ff9d3ee6a0 
A bed machine carries the package database its base image was built with,
and the mirrors stop serving it within weeks: genesis-keeps-its-bus failed
on a 404 from an image 44 days old, with nothing saying its age and no mesh
tool to rebuild it. lab_check and lab_run now say the age, lab_run refuses
an image older than 14 days and names the remedy, and lab_base reads the
image and rebuilds it with mesh-lab's base build when asked.
mesh-admin commented 2026-10-14 01:21:57 +00:00
Author
Contributor
Copy Link
Copy Source

Delivery novox/mesh-catalog@0fcc560fecd6 — delivered since 2026-10-14T05:14:27Z

Delivery plan — builds lab → shanks; no bus step

  • tier 0: lab
  • shanks: receives lab
  • lab keeps data (ADR 0233): what it holds is backed up before it moves
  • grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before lab is judged on its first machine

Machines

  • lab on shanks: passed — healthy 1 times over 26s, by a leaf module's rule: one reading (novox/hq ADR 0304)
  • lab on the rest running it: sent

Transitions

  • 2026-10-14 01:34 (new) → proposed (announced): novox/mesh-catalog#288's head announced
  • 2026-10-14 01:39 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit
  • 2026-10-14 01:39 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
  • 2026-10-14 05:06 ready → proposed (trunk moved): out of date: judged against trunk ebf7f3773d, the trunk is now e846c289c878; judged again on a recheck or when it is merged
  • 2026-10-14 05:08 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit
  • 2026-10-14 05:08 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
  • 2026-10-14 05:13 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
  • 2026-10-14 05:13 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path — not a failed walk that carried a later commit for it, after which it waits for a walk of its own commit
  • 2026-10-14 05:14 delivering → delivered (done): its walk is done, carrying its own commit: every machine of its deploy plan passed, or was left as its policy says

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

<!-- mesh-delivery:view --> **Delivery** `novox/mesh-catalog@0fcc560fecd6` — **delivered** since 2026-10-14T05:14:27Z **Delivery plan** — builds lab → shanks; no bus step - tier 0: lab - shanks: receives lab - lab keeps data (ADR 0233): what it holds is backed up before it moves - grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before lab is judged on its first machine **Machines** - lab on shanks: passed — healthy 1 times over 26s, by a leaf module's rule: one reading (novox/hq ADR 0304) - lab on the rest running it: sent **Transitions** - 2026-10-14 01:34 (new) → proposed (announced): novox/mesh-catalog#288's head announced - 2026-10-14 01:39 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit - 2026-10-14 01:39 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-14 05:06 ready → proposed (trunk moved): out of date: judged against trunk ebf7f3773dc7, the trunk is now e846c289c878; judged again on a recheck or when it is merged - 2026-10-14 05:08 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit - 2026-10-14 05:08 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-14 05:13 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-14 05:13 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path — not a failed walk that carried a later commit for it, after which it waits for a walk of its own commit - 2026-10-14 05:14 delivering → delivered (done): its walk is done, carrying its own commit: every machine of its deploy plan passed, or was left as its policy says The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.
mesh-admin commented 2026-10-14 01:26:31 +00:00
Author
Contributor
Copy Link
Copy Source

Merge check at ff9d3ee6
Judged: head ff9d3ee6a06c merged into trunk ebf7f3773dc7; read beside it: mesh-controller 2959c8143536, mesh-controller-main 2959c8143536, mesh-host 1a2677882380, mesh-tools 954e1a53eef9

  • mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: lab)
  • mesh/repo-check: PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)

Delivery plan — builds lab → shanks; no bus step

  • tier 0: lab
  • shanks: receives lab
  • lab keeps data (ADR 0233): what it holds is backed up before it moves
  • grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before lab is judged on its first machine

Run by the build seat on novox as build-1791940906819255991 (builds --log build-1791940906819255991).

--- the gate: lab
lab: ok, keeps work (cache); WARNING: runtime-env ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339)
long-running resources without health: 0
files asking for a setting without saying whether it is trusted: 1
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-14T01:08:55Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-14T01:08:55Z

modules the change moves:
  - lab (changed)

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 1 module(s) in 1 tier(s): lab
--- its merge-check.sh (go toolchain)
ok  	git.novox.be/novox/mesh-catalog/checks/words	0.008s
words: 12 retired words and 70 daemons' lines, none in the 46836 strings of 719 files
ok  	git.novox.be/novox/mesh-catalog/checks/agenthome	0.021s
agent-home: no archive and no file written into under the agent's home
ok  	git.novox.be/novox/mesh-catalog/checks/secrets	0.020s
secrets: no setting can replace a vault secret in a mergeable file (1 known, nested)
ok  	git.novox.be/novox/mesh-catalog/checks/paths	0.011s
paths: no module names another's state on its machine (3 known, listed to go)
long-running resources without health: 66, as kept
ok  	git.novox.be/novox/mesh-catalog/checks/deps	0.632s
modules/lab is TypeScript: merge-check-typescript.sh type-checks and tests it
--- its merge-check-typescript.sh (typescript toolchain)
type-checking modules/lab (TypeScript)
testing modules/lab (TypeScript)
✔ the base image's age is read from when this machine made it (13.824263ms)
✔ an image older than the bound refuses a run, naming the tool that rebuilds it (0.26576ms)
✔ an image within the bound runs (0.16286ms)
✔ a missing image, or one whose age cannot be read, is said and not refused (0.18302ms)
✔ no run starts while the base image is rebuilt (0.147741ms)
✔ a rebuild is mesh-lab's base build from the forge's main, with no secret in its script (1.541031ms)
✔ a rebuild is known by its status, and is not a run (18.948589ms)
✔ lab_run refuses on the base before it starts anything, lab_check names its age, lab_base reads unless asked (0.589141ms)
✔ a checkout with a lock file at its root is installed there (2.770433ms)
✔ a checkout whose package lives one directory down is installed there, as mesh-tools' node-tools (0.881631ms)
✔ a package.json with no lock file is named, never installed (0.478521ms)
✔ a run installs and builds mesh-tools in node-tools, where its lock file is (11.210431ms)
✔ a run installs a checkout with its lock file at the root there, and builds only when asked (7.960538ms)
✔ a run fails at its setup, naming the checkout, when nothing of it can be installed (12.121292ms)
✔ a run checks out mesh-tools-go and says where its checkouts are (0.956061ms)
✔ the forge credential is the bound account and its sealed secret, for the forge's own origin (0.795901ms)
✔ a run clones through git's credential store, and its script carries no secret (1.317361ms)
✔ git presents the stored credential to the forge, and only when one is given (17.039257ms)
✔ a stopped script removes the credential, whatever ends it (16.458627ms)
✔ the script removes the credential as soon as its clones are done (0.54117ms)
✔ credentials a crashed run left are removed, but not the running one's (1.555052ms)
✔ the lab asks the package registry for read only (0.39258ms)
✔ each check gets a fresh directory of its own (0.477831ms)
﹣ what earlier checks left is cleared, and what cannot be is named rather than tripped over (0.07036ms) # running as root, which no directory mode stops from removing anything
✔ leftover checks are cleared, whoever runs the check (0.787641ms)
✔ the lab's tools bundle names its bound credential and binding in its env (0.19926ms)
✔ a run names the package registry by the address the registry gives its tarballs (21.074191ms)
✔ a registry that does not give an address the lab takes leaves the forge's, and the reason is said (5.457235ms)
✔ a run's npm configuration names the registry it was given, and the forge's own address only without one (0.479921ms)
✔ a tarball is taken at the forge's public root, and at no other port or scheme of its host (1.040871ms)
✔ the public root is read from the binding, and only an https root at the npm path is taken (1.613002ms)
✔ a forge setting that is not http(s) trusts no origin of its own (0.58673ms)
ℹ tests 32
ℹ suites 0
ℹ pass 31
ℹ fail 0
ℹ cancelled 0
ℹ skipped 1
ℹ todo 0
ℹ duration_ms 231.3396
**Merge check** at `ff9d3ee6` Judged: head `ff9d3ee6a06c` merged into trunk `ebf7f3773dc7`; read beside it: mesh-controller `2959c8143536`, mesh-controller-main `2959c8143536`, mesh-host `1a2677882380`, mesh-tools `954e1a53eef9` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: lab) - `mesh/repo-check`: **PASS** — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript) **Delivery plan** — builds lab → shanks; no bus step - tier 0: lab - shanks: receives lab - lab keeps data (ADR 0233): what it holds is backed up before it moves - grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before lab is judged on its first machine Run by the build seat on novox as `build-1791940906819255991` (`builds --log build-1791940906819255991`). ``` --- the gate: lab lab: ok, keeps work (cache); WARNING: runtime-env ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339) long-running resources without health: 0 files asking for a setting without saying whether it is trusted: 1 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-14T01:08:55Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-14T01:08:55Z modules the change moves: - lab (changed) machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 1 module(s) in 1 tier(s): lab --- its merge-check.sh (go toolchain) ok git.novox.be/novox/mesh-catalog/checks/words 0.008s words: 12 retired words and 70 daemons' lines, none in the 46836 strings of 719 files ok git.novox.be/novox/mesh-catalog/checks/agenthome 0.021s agent-home: no archive and no file written into under the agent's home ok git.novox.be/novox/mesh-catalog/checks/secrets 0.020s secrets: no setting can replace a vault secret in a mergeable file (1 known, nested) ok git.novox.be/novox/mesh-catalog/checks/paths 0.011s paths: no module names another's state on its machine (3 known, listed to go) long-running resources without health: 66, as kept ok git.novox.be/novox/mesh-catalog/checks/deps 0.632s modules/lab is TypeScript: merge-check-typescript.sh type-checks and tests it --- its merge-check-typescript.sh (typescript toolchain) type-checking modules/lab (TypeScript) testing modules/lab (TypeScript) ✔ the base image's age is read from when this machine made it (13.824263ms) ✔ an image older than the bound refuses a run, naming the tool that rebuilds it (0.26576ms) ✔ an image within the bound runs (0.16286ms) ✔ a missing image, or one whose age cannot be read, is said and not refused (0.18302ms) ✔ no run starts while the base image is rebuilt (0.147741ms) ✔ a rebuild is mesh-lab's base build from the forge's main, with no secret in its script (1.541031ms) ✔ a rebuild is known by its status, and is not a run (18.948589ms) ✔ lab_run refuses on the base before it starts anything, lab_check names its age, lab_base reads unless asked (0.589141ms) ✔ a checkout with a lock file at its root is installed there (2.770433ms) ✔ a checkout whose package lives one directory down is installed there, as mesh-tools' node-tools (0.881631ms) ✔ a package.json with no lock file is named, never installed (0.478521ms) ✔ a run installs and builds mesh-tools in node-tools, where its lock file is (11.210431ms) ✔ a run installs a checkout with its lock file at the root there, and builds only when asked (7.960538ms) ✔ a run fails at its setup, naming the checkout, when nothing of it can be installed (12.121292ms) ✔ a run checks out mesh-tools-go and says where its checkouts are (0.956061ms) ✔ the forge credential is the bound account and its sealed secret, for the forge's own origin (0.795901ms) ✔ a run clones through git's credential store, and its script carries no secret (1.317361ms) ✔ git presents the stored credential to the forge, and only when one is given (17.039257ms) ✔ a stopped script removes the credential, whatever ends it (16.458627ms) ✔ the script removes the credential as soon as its clones are done (0.54117ms) ✔ credentials a crashed run left are removed, but not the running one's (1.555052ms) ✔ the lab asks the package registry for read only (0.39258ms) ✔ each check gets a fresh directory of its own (0.477831ms) ﹣ what earlier checks left is cleared, and what cannot be is named rather than tripped over (0.07036ms) # running as root, which no directory mode stops from removing anything ✔ leftover checks are cleared, whoever runs the check (0.787641ms) ✔ the lab's tools bundle names its bound credential and binding in its env (0.19926ms) ✔ a run names the package registry by the address the registry gives its tarballs (21.074191ms) ✔ a registry that does not give an address the lab takes leaves the forge's, and the reason is said (5.457235ms) ✔ a run's npm configuration names the registry it was given, and the forge's own address only without one (0.479921ms) ✔ a tarball is taken at the forge's public root, and at no other port or scheme of its host (1.040871ms) ✔ the public root is read from the binding, and only an https root at the npm path is taken (1.613002ms) ✔ a forge setting that is not http(s) trusts no origin of its own (0.58673ms) ℹ tests 32 ℹ suites 0 ℹ pass 31 ℹ fail 0 ℹ cancelled 0 ℹ skipped 1 ℹ todo 0 ℹ duration_ms 231.3396 ```
mesh-admin left a comment
Author
Contributor
Copy Link
Copy Source

Verdict: changes needed at ff9d3ee6a0: a rebuild can still race a run, because both tools check and then await before they start.

Independent review for issue 698 (the lab's base image grows stale and beds fail on mirror 404s).

Checks at this head: mesh/merge-gate success ("PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: lab)"), mesh/repo-check success ("PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)"), mesh/delivery success ("ready: waits for its merge"). mesh/delivery-group: none set; this pull request is in no delivery group.

  1. The race (blocking). lab_run calls running() and runRefusal(base, buildingBase(work)), then await registryOf(forge) (a fetch to the forge), and only then start(), which writes the run's status. lab_base calls buildingBase() and running(), then await registryOf(forge), and only then startBaseBuild(). The tool runner is one event loop, so during either await the other tool's checks pass: a lab_run and a lab_base {rebuild: true} called together both start, and base build replaces the image a run is about to raise its machines from. Fix: await the registry first, then check and start with no await in between (check and status write are then atomic in one process). The same move closes the old window between two lab_runs. The test that guards this reads index.ts as text and checks the order of strings, which cannot see an await; please test the handlers' behaviour instead.

  2. The credential is handled the way runs handle it: written by writeCredential, used through credentialArgs (credential.helper=store --file=...), removed by credentialGuard's traps, by the rm after the clone and by the wrapper; the clone URL carries no secret, the script has no set -x, and build.log holds only the clone's HEAD and the install/build output. One gap: cleanCredentials matches only ^\.run-.*\.git-credentials$, so a .base-*.git-credentials file left by a build killed with SIGKILL or a reboot is never removed. Extend the pattern, keeping the file of a build still in progress.

  3. Missing or unreadable age. lab_run stays open when the age is unknown, says so in its answer, and lets a bed that needs the image fail on its own. I accept that for runs. But lab_check then answers ok: true with only a line saying "its age is not known", so the check that should catch issue 698 passes quietly when incus cannot list images or gives no time. Make lab_check fail when the image exists and its age is unknown, or when incus cannot list. Also: a build whose status says building but has no pid (spawn failed, or the process died between the two writes in startBaseBuild) is never marked lost, so lab_run would refuse every run until someone edits the status by hand. Mark a building status with no pid, older than a few minutes, as lost.

  4. Smaller points. ageDays is floored, so an image 14.9 days old still runs (the bound is effectively under 15 days); say that, or compare in milliseconds.

  5. Overlap with #287. Both change modules/lab/tools/runs.ts: #287 changes RunStatus and readStatus in the middle of the file, and #288 appends the base-build functions at its end. I merged #287's head into this head locally: the merge was clean and the lab tests passed (36 pass, 0 fail). They do not interact: listRuns reads only run-* directories, and base builds keep their own status files.

Tests: niced, modules/lab at this head 32 pass, 0 fail; test/base.test.ts fails on main at import (tools/base.ts does not exist). No lab bed ran.

Reviewed by an independent reviewer on g14.

Verdict: changes needed at ff9d3ee6a06c: a rebuild can still race a run, because both tools check and then await before they start. Independent review for issue 698 (the lab's base image grows stale and beds fail on mirror 404s). Checks at this head: mesh/merge-gate success ("PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: lab)"), mesh/repo-check success ("PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)"), mesh/delivery success ("ready: waits for its merge"). mesh/delivery-group: none set; this pull request is in no delivery group. 1. The race (blocking). lab_run calls running() and runRefusal(base, buildingBase(work)), then `await registryOf(forge)` (a fetch to the forge), and only then start(), which writes the run's status. lab_base calls buildingBase() and running(), then `await registryOf(forge)`, and only then startBaseBuild(). The tool runner is one event loop, so during either await the other tool's checks pass: a lab_run and a lab_base {rebuild: true} called together both start, and `base build` replaces the image a run is about to raise its machines from. Fix: await the registry first, then check and start with no await in between (check and status write are then atomic in one process). The same move closes the old window between two lab_runs. The test that guards this reads index.ts as text and checks the order of strings, which cannot see an await; please test the handlers' behaviour instead. 2. The credential is handled the way runs handle it: written by writeCredential, used through credentialArgs (credential.helper=store --file=...), removed by credentialGuard's traps, by the rm after the clone and by the wrapper; the clone URL carries no secret, the script has no set -x, and build.log holds only the clone's HEAD and the install/build output. One gap: cleanCredentials matches only `^\.run-.*\.git-credentials$`, so a `.base-*.git-credentials` file left by a build killed with SIGKILL or a reboot is never removed. Extend the pattern, keeping the file of a build still in progress. 3. Missing or unreadable age. lab_run stays open when the age is unknown, says so in its answer, and lets a bed that needs the image fail on its own. I accept that for runs. But lab_check then answers ok: true with only a line saying "its age is not known", so the check that should catch issue 698 passes quietly when incus cannot list images or gives no time. Make lab_check fail when the image exists and its age is unknown, or when incus cannot list. Also: a build whose status says building but has no pid (spawn failed, or the process died between the two writes in startBaseBuild) is never marked lost, so lab_run would refuse every run until someone edits the status by hand. Mark a building status with no pid, older than a few minutes, as lost. 4. Smaller points. ageDays is floored, so an image 14.9 days old still runs (the bound is effectively under 15 days); say that, or compare in milliseconds. 5. Overlap with #287. Both change modules/lab/tools/runs.ts: #287 changes RunStatus and readStatus in the middle of the file, and #288 appends the base-build functions at its end. I merged #287's head into this head locally: the merge was clean and the lab tests passed (36 pass, 0 fail). They do not interact: listRuns reads only run-* directories, and base builds keep their own status files. Tests: niced, modules/lab at this head 32 pass, 0 fail; test/base.test.ts fails on main at import (tools/base.ts does not exist). No lab bed ran. Reviewed by an independent reviewer on g14.
jschoubben added 1 commit 2026-10-14 01:34:49 +00:00
lab: one job at a time with no await between check and start, and no rebuild left holding anything (issue 698, review 256)
mesh/merge-gate head 0fcc560fecd6 merged into trunk e846c289c878 — pass: builds lab → shanks; no bus step; every machine composes with the change as it did…
Details
mesh/repo-check head 0fcc560fecd6 merged into trunk e846c289c878 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-…
Details
mesh/delivery delivered
Details
0fcc560fec 
A run and a rebuild each asked the package registry between checking for
the other and writing their status, so both could start and a bed could
raise machines from an image being replaced. The registry is asked first
now. A hard-killed rebuild's credential is cleaned like a run's; a job that
never recorded its process is lost rather than blocking every run; and
lab_check reports a missing or unreadable base image.
mesh-admin commented 2026-10-14 01:34:54 +00:00
Author
Contributor
Copy Link
Copy Source

Review 256, all four points, at 0fcc560f:

  1. Race. Runs and rebuilds now start through launchRun and launchRebuild (tools/runs.ts). Each asks the package registry first, then reads what is going and writes the new job's status with nothing awaited in between, so in the one runtime no other call can run between the check and the start. The string-order test is gone. In its place, a run and a rebuild asked at once: exactly one starts (both creation orders) and two runs, or two rebuilds, asked at once: one of each starts run the calls concurrently against a registry that answers after 20 ms. With the old order (check, then await, then start) put back into launchRun, both tests fail.
  2. Credentials. cleanCredentials now also removes .base-*.git-credentials and takes a list to keep. lab_check keeps the running run's and the running rebuild's. A rebuild also clears leftovers before writing its own, as a run does.
  3. lab_check now reports a missing image, or one whose age cannot be read, as a problem (baseProblem). lab_run still refuses only a stale image.
  4. Lost jobs. A rebuild or a run whose status says it is going and records no process is now lost, and holds nothing up. Tested for both.

Also: tools/base.ts is folded into tools/runs.ts, because a test importing a file that imports its sibling by .js cannot run under strip-types. 38 tests pass, and tsc passes with the repository check's flags.

Review 256, all four points, at 0fcc560f: 1. **Race.** Runs and rebuilds now start through `launchRun` and `launchRebuild` (tools/runs.ts). Each asks the package registry first, then reads what is going and writes the new job's status with nothing awaited in between, so in the one runtime no other call can run between the check and the start. The string-order test is gone. In its place, `a run and a rebuild asked at once: exactly one starts` (both creation orders) and `two runs, or two rebuilds, asked at once: one of each starts` run the calls concurrently against a registry that answers after 20 ms. With the old order (check, then await, then start) put back into launchRun, both tests fail. 2. **Credentials.** `cleanCredentials` now also removes `.base-*.git-credentials` and takes a list to keep. lab_check keeps the running run's and the running rebuild's. A rebuild also clears leftovers before writing its own, as a run does. 3. **lab_check** now reports a missing image, or one whose age cannot be read, as a problem (`baseProblem`). lab_run still refuses only a stale image. 4. **Lost jobs.** A rebuild or a run whose status says it is going and records no process is now lost, and holds nothing up. Tested for both. Also: tools/base.ts is folded into tools/runs.ts, because a test importing a file that imports its sibling by `.js` cannot run under strip-types. 38 tests pass, and tsc passes with the repository check's flags.
mesh-admin commented 2026-10-14 01:39:06 +00:00
Author
Contributor
Copy Link
Copy Source

Merge check at 0fcc560f
Judged: head 0fcc560fecd6 merged into trunk ebf7f3773dc7; read beside it: mesh-controller 2959c8143536, mesh-controller-main 2959c8143536, mesh-host 1a2677882380, mesh-tools 954e1a53eef9

  • mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: lab)
  • mesh/repo-check: PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)

Delivery plan — builds lab → shanks; no bus step

  • tier 0: lab
  • shanks: receives lab
  • lab keeps data (ADR 0233): what it holds is backed up before it moves
  • grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before lab is judged on its first machine

Run by the build seat on novox as build-1791941693045741553 (builds --log build-1791941693045741553).

--- the gate: lab
lab: ok, keeps work (cache); WARNING: runtime-env ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339)
long-running resources without health: 0
files asking for a setting without saying whether it is trusted: 1
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-14T01:31:20Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-14T01:31:20Z

modules the change moves:
  - lab (changed)

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 1 module(s) in 1 tier(s): lab
--- its merge-check.sh (go toolchain)
ok  	git.novox.be/novox/mesh-catalog/checks/words	0.014s
words: 12 retired words and 70 daemons' lines, none in the 46832 strings of 718 files
ok  	git.novox.be/novox/mesh-catalog/checks/agenthome	0.017s
agent-home: no archive and no file written into under the agent's home
ok  	git.novox.be/novox/mesh-catalog/checks/secrets	0.013s
secrets: no setting can replace a vault secret in a mergeable file (1 known, nested)
ok  	git.novox.be/novox/mesh-catalog/checks/paths	0.007s
paths: no module names another's state on its machine (3 known, listed to go)
long-running resources without health: 66, as kept
ok  	git.novox.be/novox/mesh-catalog/checks/deps	0.238s
modules/lab is TypeScript: merge-check-typescript.sh type-checks and tests it
--- its merge-check-typescript.sh (typescript toolchain)
type-checking modules/lab (TypeScript)
testing modules/lab (TypeScript)
✔ the base image's age is read from when this machine made it (2.117664ms)
✔ an image older than the bound refuses a run, naming the tool that rebuilds it (0.39872ms)
✔ an image within the bound runs (0.19548ms)
✔ a missing image, or one whose age cannot be read, is said and not refused (0.22062ms)
✔ no run starts while the base image is rebuilt (0.261721ms)
✔ a rebuild is mesh-lab's base build from the forge's main, with no secret in its script (0.336861ms)
✔ a rebuild is known by its status, and is not a run (44.581503ms)
✔ a run and a rebuild asked at once: exactly one starts (43.756541ms)
✔ two runs, or two rebuilds, asked at once: one of each starts (42.694399ms)
✔ a run is refused on a stale base, and starts nothing (20.940609ms)
✔ a rebuild or a run that never recorded its process is lost, and holds nothing up (1.449222ms)
✔ a rebuild's credential left behind is cleaned, and a running one's is kept (1.296463ms)
✔ lab_check's problem: a stale, missing or unread base image; a fresh one is none (1.776103ms)
✔ lab_run and lab_base start through the gate, and lab_check names what is wrong with the base (0.625931ms)
✔ a checkout with a lock file at its root is installed there (3.926197ms)
✔ a checkout whose package lives one directory down is installed there, as mesh-tools' node-tools (1.202692ms)
✔ a package.json with no lock file is named, never installed (0.810972ms)
✔ a run installs and builds mesh-tools in node-tools, where its lock file is (13.696875ms)
✔ a run installs a checkout with its lock file at the root there, and builds only when asked (7.321424ms)
✔ a run fails at its setup, naming the checkout, when nothing of it can be installed (18.287164ms)
✔ a run checks out mesh-tools-go and says where its checkouts are (0.952892ms)
✔ the forge credential is the bound account and its sealed secret, for the forge's own origin (0.931132ms)
✔ a run clones through git's credential store, and its script carries no secret (0.983232ms)
✔ git presents the stored credential to the forge, and only when one is given (27.223041ms)
✔ a stopped script removes the credential, whatever ends it (19.414016ms)
✔ the script removes the credential as soon as its clones are done (0.399631ms)
✔ credentials a crashed run left are removed, but not the running one's (0.979952ms)
✔ the lab asks the package registry for read only (0.412231ms)
✔ each check gets a fresh directory of its own (0.532441ms)
﹣ what earlier checks left is cleared, and what cannot be is named rather than tripped over (0.07951ms) # running as root, which no directory mode stops from removing anything
✔ leftover checks are cleared, whoever runs the check (1.053692ms)
✔ the lab's tools bundle names its bound credential and binding in its env (0.2135ms)
✔ a run names the package registry by the address the registry gives its tarballs (23.536114ms)
✔ a registry that does not give an address the lab takes leaves the forge's, and the reason is said (6.134352ms)
✔ a run's npm configuration names the registry it was given, and the forge's own address only without one (0.456221ms)
✔ a tarball is taken at the forge's public root, and at no other port or scheme of its host (1.621283ms)
✔ the public root is read from the binding, and only an https root at the npm path is taken (2.083154ms)
✔ a forge setting that is not http(s) trusts no origin of its own (0.721291ms)
ℹ tests 38
ℹ suites 0
ℹ pass 37
ℹ fail 0
ℹ cancelled 0
ℹ skipped 1
ℹ todo 0
ℹ duration_ms 349.623021
**Merge check** at `0fcc560f` Judged: head `0fcc560fecd6` merged into trunk `ebf7f3773dc7`; read beside it: mesh-controller `2959c8143536`, mesh-controller-main `2959c8143536`, mesh-host `1a2677882380`, mesh-tools `954e1a53eef9` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: lab) - `mesh/repo-check`: **PASS** — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript) **Delivery plan** — builds lab → shanks; no bus step - tier 0: lab - shanks: receives lab - lab keeps data (ADR 0233): what it holds is backed up before it moves - grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before lab is judged on its first machine Run by the build seat on novox as `build-1791941693045741553` (`builds --log build-1791941693045741553`). ``` --- the gate: lab lab: ok, keeps work (cache); WARNING: runtime-env ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339) long-running resources without health: 0 files asking for a setting without saying whether it is trusted: 1 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-14T01:31:20Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-14T01:31:20Z modules the change moves: - lab (changed) machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 1 module(s) in 1 tier(s): lab --- its merge-check.sh (go toolchain) ok git.novox.be/novox/mesh-catalog/checks/words 0.014s words: 12 retired words and 70 daemons' lines, none in the 46832 strings of 718 files ok git.novox.be/novox/mesh-catalog/checks/agenthome 0.017s agent-home: no archive and no file written into under the agent's home ok git.novox.be/novox/mesh-catalog/checks/secrets 0.013s secrets: no setting can replace a vault secret in a mergeable file (1 known, nested) ok git.novox.be/novox/mesh-catalog/checks/paths 0.007s paths: no module names another's state on its machine (3 known, listed to go) long-running resources without health: 66, as kept ok git.novox.be/novox/mesh-catalog/checks/deps 0.238s modules/lab is TypeScript: merge-check-typescript.sh type-checks and tests it --- its merge-check-typescript.sh (typescript toolchain) type-checking modules/lab (TypeScript) testing modules/lab (TypeScript) ✔ the base image's age is read from when this machine made it (2.117664ms) ✔ an image older than the bound refuses a run, naming the tool that rebuilds it (0.39872ms) ✔ an image within the bound runs (0.19548ms) ✔ a missing image, or one whose age cannot be read, is said and not refused (0.22062ms) ✔ no run starts while the base image is rebuilt (0.261721ms) ✔ a rebuild is mesh-lab's base build from the forge's main, with no secret in its script (0.336861ms) ✔ a rebuild is known by its status, and is not a run (44.581503ms) ✔ a run and a rebuild asked at once: exactly one starts (43.756541ms) ✔ two runs, or two rebuilds, asked at once: one of each starts (42.694399ms) ✔ a run is refused on a stale base, and starts nothing (20.940609ms) ✔ a rebuild or a run that never recorded its process is lost, and holds nothing up (1.449222ms) ✔ a rebuild's credential left behind is cleaned, and a running one's is kept (1.296463ms) ✔ lab_check's problem: a stale, missing or unread base image; a fresh one is none (1.776103ms) ✔ lab_run and lab_base start through the gate, and lab_check names what is wrong with the base (0.625931ms) ✔ a checkout with a lock file at its root is installed there (3.926197ms) ✔ a checkout whose package lives one directory down is installed there, as mesh-tools' node-tools (1.202692ms) ✔ a package.json with no lock file is named, never installed (0.810972ms) ✔ a run installs and builds mesh-tools in node-tools, where its lock file is (13.696875ms) ✔ a run installs a checkout with its lock file at the root there, and builds only when asked (7.321424ms) ✔ a run fails at its setup, naming the checkout, when nothing of it can be installed (18.287164ms) ✔ a run checks out mesh-tools-go and says where its checkouts are (0.952892ms) ✔ the forge credential is the bound account and its sealed secret, for the forge's own origin (0.931132ms) ✔ a run clones through git's credential store, and its script carries no secret (0.983232ms) ✔ git presents the stored credential to the forge, and only when one is given (27.223041ms) ✔ a stopped script removes the credential, whatever ends it (19.414016ms) ✔ the script removes the credential as soon as its clones are done (0.399631ms) ✔ credentials a crashed run left are removed, but not the running one's (0.979952ms) ✔ the lab asks the package registry for read only (0.412231ms) ✔ each check gets a fresh directory of its own (0.532441ms) ﹣ what earlier checks left is cleared, and what cannot be is named rather than tripped over (0.07951ms) # running as root, which no directory mode stops from removing anything ✔ leftover checks are cleared, whoever runs the check (1.053692ms) ✔ the lab's tools bundle names its bound credential and binding in its env (0.2135ms) ✔ a run names the package registry by the address the registry gives its tarballs (23.536114ms) ✔ a registry that does not give an address the lab takes leaves the forge's, and the reason is said (6.134352ms) ✔ a run's npm configuration names the registry it was given, and the forge's own address only without one (0.456221ms) ✔ a tarball is taken at the forge's public root, and at no other port or scheme of its host (1.621283ms) ✔ the public root is read from the binding, and only an https root at the npm path is taken (2.083154ms) ✔ a forge setting that is not http(s) trusts no origin of its own (0.721291ms) ℹ tests 38 ℹ suites 0 ℹ pass 37 ℹ fail 0 ℹ cancelled 0 ℹ skipped 1 ℹ todo 0 ℹ duration_ms 349.623021 ```
mesh-admin left a comment
Author
Contributor
Copy Link
Copy Source

Verdict: sound, mergeable at 0fcc560fecd6; every point of review 256 is answered.

Re-review for issue 698 (the lab's base image grows stale).

Checks at this head: mesh/merge-gate success ("PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: lab)"), mesh/repo-check success ("PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)"), mesh/delivery success ("ready: waits for its merge"). mesh/delivery-group: none set, because this pull request is in no delivery group.

  1. The race is fixed. launchRun and launchRebuild await the registry first; from there to begin() nothing is awaited, so in the one runtime the check and the status write are one step. The two concurrency tests ("a run and a rebuild asked at once: exactly one starts", in both orders, and "two runs, or two rebuilds, asked at once") prove it: I moved the await back between the check and the start, and both tests failed (0 pass, 2 fail); with the head as it is, they pass.
  2. Credentials: cleanCredentials now matches .run- and .base- files and keeps a list; lab_check keeps the files of the run and the rebuild still going, and both start() and startBaseBuild() clear what a crash left.
  3. lab_check now fails on a base image that is missing or whose age cannot be read (baseProblem); lab_run still refuses only a stale one, which is the design. A job whose status says it is going with no recorded process is lost at once; that is safe because start() and startBaseBuild() write the status and the pid synchronously, with nothing awaited between. Not changed, and not blocking: ageDays is still floored, so the bound is under 15 days rather than 14.
  4. Overlap with #287: I merged #287's new head acf8a766 into this head locally. tools/runs.ts merges cleanly, and the lab tests of both together pass: 44 pass, 0 fail. At this head alone, niced: 38 pass, 0 fail.

Merge order with the issue 693 group: either order works, because the files merge cleanly.

Reviewed by an independent reviewer on g14; no lab bed ran.

Verdict: sound, mergeable at 0fcc560fecd6; every point of review 256 is answered. Re-review for issue 698 (the lab's base image grows stale). Checks at this head: mesh/merge-gate success ("PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: lab)"), mesh/repo-check success ("PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)"), mesh/delivery success ("ready: waits for its merge"). mesh/delivery-group: none set, because this pull request is in no delivery group. 1. The race is fixed. launchRun and launchRebuild await the registry first; from there to begin() nothing is awaited, so in the one runtime the check and the status write are one step. The two concurrency tests ("a run and a rebuild asked at once: exactly one starts", in both orders, and "two runs, or two rebuilds, asked at once") prove it: I moved the await back between the check and the start, and both tests failed (0 pass, 2 fail); with the head as it is, they pass. 2. Credentials: cleanCredentials now matches `.run-` and `.base-` files and keeps a list; lab_check keeps the files of the run and the rebuild still going, and both start() and startBaseBuild() clear what a crash left. 3. lab_check now fails on a base image that is missing or whose age cannot be read (baseProblem); lab_run still refuses only a stale one, which is the design. A job whose status says it is going with no recorded process is lost at once; that is safe because start() and startBaseBuild() write the status and the pid synchronously, with nothing awaited between. Not changed, and not blocking: ageDays is still floored, so the bound is under 15 days rather than 14. 4. Overlap with #287: I merged #287's new head acf8a766 into this head locally. tools/runs.ts merges cleanly, and the lab tests of both together pass: 44 pass, 0 fail. At this head alone, niced: 38 pass, 0 fail. Merge order with the issue 693 group: either order works, because the files merge cleanly. Reviewed by an independent reviewer on g14; no lab bed ran.
mesh-admin commented 2026-10-14 05:08:09 +00:00
Author
Contributor
Copy Link
Copy Source

Merge check at 0fcc560f
Judged: head 0fcc560fecd6 merged into trunk e846c289c878; read beside it: mesh-controller 2959c8143536, mesh-controller-main 2959c8143536, mesh-host 1a2677882380, mesh-tools 954e1a53eef9

  • mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: lab)
  • mesh/repo-check: PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)

Delivery plan — builds lab → shanks; no bus step

  • tier 0: lab
  • shanks: receives lab
  • lab keeps data (ADR 0233): what it holds is backed up before it moves
  • grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before lab is judged on its first machine

Run by the build seat on shanks as build-1791954344997736163 (builds --log build-1791954344997736163).

--- the gate: lab
lab: ok, keeps work (cache); WARNING: runtime-env ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339)
long-running resources without health: 0
files asking for a setting without saying whether it is trusted: 1
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-14T01:31:20Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-14T01:31:20Z

modules the change moves:
  - lab (changed)

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 1 module(s) in 1 tier(s): lab
--- its merge-check.sh (go toolchain)
ok  	git.novox.be/novox/mesh-catalog/checks/words	0.003s
words: 12 retired words and 70 daemons' lines, none in the 46860 strings of 718 files
ok  	git.novox.be/novox/mesh-catalog/checks/agenthome	0.013s
agent-home: no archive and no file written into under the agent's home
ok  	git.novox.be/novox/mesh-catalog/checks/secrets	0.012s
secrets: no setting can replace a vault secret in a mergeable file (1 known, nested)
ok  	git.novox.be/novox/mesh-catalog/checks/paths	0.006s
paths: no module names another's state on its machine (3 known, listed to go)
long-running resources without health: 66, as kept
ok  	git.novox.be/novox/mesh-catalog/checks/deps	0.207s
modules/lab is TypeScript: merge-check-typescript.sh type-checks and tests it
--- its merge-check-typescript.sh (typescript toolchain)
type-checking modules/lab (TypeScript)
testing modules/lab (TypeScript)
✔ the base image's age is read from when this machine made it (2.19386ms)
✔ an image older than the bound refuses a run, naming the tool that rebuilds it (0.231433ms)
✔ an image within the bound runs (0.131372ms)
✔ a missing image, or one whose age cannot be read, is said and not refused (0.140632ms)
✔ no run starts while the base image is rebuilt (0.208293ms)
✔ a rebuild is mesh-lab's base build from the forge's main, with no secret in its script (0.290114ms)
✔ a rebuild is known by its status, and is not a run (10.494564ms)
✔ a run and a rebuild asked at once: exactly one starts (42.314131ms)
✔ two runs, or two rebuilds, asked at once: one of each starts (42.295ms)
✔ a run is refused on a stale base, and starts nothing (20.162386ms)
✔ a rebuild or a run that never recorded its process is lost, and holds nothing up (0.936793ms)
✔ a rebuild's credential left behind is cleaned, and a running one's is kept (0.822441ms)
✔ lab_check's problem: a stale, missing or unread base image; a fresh one is none (0.131582ms)
✔ lab_run and lab_base start through the gate, and lab_check names what is wrong with the base (0.330854ms)
✔ a checkout with a lock file at its root is installed there (2.550794ms)
✔ a checkout whose package lives one directory down is installed there, as mesh-tools' node-tools (0.834142ms)
✔ a package.json with no lock file is named, never installed (0.495197ms)
✔ a run installs and builds mesh-tools in node-tools, where its lock file is (7.546343ms)
✔ a run installs a checkout with its lock file at the root there, and builds only when asked (4.702725ms)
✔ a run fails at its setup, naming the checkout, when nothing of it can be installed (13.348603ms)
✔ a run checks out mesh-tools-go and says where its checkouts are (0.814861ms)
✔ the forge credential is the bound account and its sealed secret, for the forge's own origin (0.876642ms)
✔ a run clones through git's credential store, and its script carries no secret (1.098955ms)
✔ git presents the stored credential to the forge, and only when one is given (12.027655ms)
✔ a stopped script removes the credential, whatever ends it (14.589261ms)
✔ the script removes the credential as soon as its clones are done (0.329485ms)
✔ credentials a crashed run left are removed, but not the running one's (0.74035ms)
✔ the lab asks the package registry for read only (0.343734ms)
✔ each check gets a fresh directory of its own (0.395055ms)
﹣ what earlier checks left is cleared, and what cannot be is named rather than tripped over (0.055271ms) # running as root, which no directory mode stops from removing anything
✔ leftover checks are cleared, whoever runs the check (0.778571ms)
✔ the lab's tools bundle names its bound credential and binding in its env (0.151962ms)
✔ a run names the package registry by the address the registry gives its tarballs (19.054202ms)
✔ a registry that does not give an address the lab takes leaves the forge's, and the reason is said (3.938365ms)
✔ a run's npm configuration names the registry it was given, and the forge's own address only without one (0.333575ms)
✔ a tarball is taken at the forge's public root, and at no other port or scheme of its host (1.084935ms)
✔ the public root is read from the binding, and only an https root at the npm path is taken (1.497931ms)
✔ a forge setting that is not http(s) trusts no origin of its own (0.571448ms)
✔ a run whose only test skipped as NOT REACHED, from a lab of before, reads as failed with the reason (1.496481ms)
✔ the lab's own NOT PASSED and skipped lines are read, a test name's punctuation and all (0.68571ms)
✔ a run that passed tests and skipped one for a declared reason stays passed, its skip listed (0.666839ms)
✔ a run that ran no test at all is not passed (0.110731ms)
✔ a line a test printed that looks like the suite's is not read as the suite's (0.124471ms)
✔ not reached is matched in any case (0.112771ms)
ℹ tests 44
ℹ suites 0
ℹ pass 43
ℹ fail 0
ℹ cancelled 0
ℹ skipped 1
ℹ todo 0
ℹ duration_ms 218.058181
**Merge check** at `0fcc560f` Judged: head `0fcc560fecd6` merged into trunk `e846c289c878`; read beside it: mesh-controller `2959c8143536`, mesh-controller-main `2959c8143536`, mesh-host `1a2677882380`, mesh-tools `954e1a53eef9` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: lab) - `mesh/repo-check`: **PASS** — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript) **Delivery plan** — builds lab → shanks; no bus step - tier 0: lab - shanks: receives lab - lab keeps data (ADR 0233): what it holds is backed up before it moves - grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before lab is judged on its first machine Run by the build seat on shanks as `build-1791954344997736163` (`builds --log build-1791954344997736163`). ``` --- the gate: lab lab: ok, keeps work (cache); WARNING: runtime-env ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339) long-running resources without health: 0 files asking for a setting without saying whether it is trusted: 1 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-14T01:31:20Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-14T01:31:20Z modules the change moves: - lab (changed) machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 1 module(s) in 1 tier(s): lab --- its merge-check.sh (go toolchain) ok git.novox.be/novox/mesh-catalog/checks/words 0.003s words: 12 retired words and 70 daemons' lines, none in the 46860 strings of 718 files ok git.novox.be/novox/mesh-catalog/checks/agenthome 0.013s agent-home: no archive and no file written into under the agent's home ok git.novox.be/novox/mesh-catalog/checks/secrets 0.012s secrets: no setting can replace a vault secret in a mergeable file (1 known, nested) ok git.novox.be/novox/mesh-catalog/checks/paths 0.006s paths: no module names another's state on its machine (3 known, listed to go) long-running resources without health: 66, as kept ok git.novox.be/novox/mesh-catalog/checks/deps 0.207s modules/lab is TypeScript: merge-check-typescript.sh type-checks and tests it --- its merge-check-typescript.sh (typescript toolchain) type-checking modules/lab (TypeScript) testing modules/lab (TypeScript) ✔ the base image's age is read from when this machine made it (2.19386ms) ✔ an image older than the bound refuses a run, naming the tool that rebuilds it (0.231433ms) ✔ an image within the bound runs (0.131372ms) ✔ a missing image, or one whose age cannot be read, is said and not refused (0.140632ms) ✔ no run starts while the base image is rebuilt (0.208293ms) ✔ a rebuild is mesh-lab's base build from the forge's main, with no secret in its script (0.290114ms) ✔ a rebuild is known by its status, and is not a run (10.494564ms) ✔ a run and a rebuild asked at once: exactly one starts (42.314131ms) ✔ two runs, or two rebuilds, asked at once: one of each starts (42.295ms) ✔ a run is refused on a stale base, and starts nothing (20.162386ms) ✔ a rebuild or a run that never recorded its process is lost, and holds nothing up (0.936793ms) ✔ a rebuild's credential left behind is cleaned, and a running one's is kept (0.822441ms) ✔ lab_check's problem: a stale, missing or unread base image; a fresh one is none (0.131582ms) ✔ lab_run and lab_base start through the gate, and lab_check names what is wrong with the base (0.330854ms) ✔ a checkout with a lock file at its root is installed there (2.550794ms) ✔ a checkout whose package lives one directory down is installed there, as mesh-tools' node-tools (0.834142ms) ✔ a package.json with no lock file is named, never installed (0.495197ms) ✔ a run installs and builds mesh-tools in node-tools, where its lock file is (7.546343ms) ✔ a run installs a checkout with its lock file at the root there, and builds only when asked (4.702725ms) ✔ a run fails at its setup, naming the checkout, when nothing of it can be installed (13.348603ms) ✔ a run checks out mesh-tools-go and says where its checkouts are (0.814861ms) ✔ the forge credential is the bound account and its sealed secret, for the forge's own origin (0.876642ms) ✔ a run clones through git's credential store, and its script carries no secret (1.098955ms) ✔ git presents the stored credential to the forge, and only when one is given (12.027655ms) ✔ a stopped script removes the credential, whatever ends it (14.589261ms) ✔ the script removes the credential as soon as its clones are done (0.329485ms) ✔ credentials a crashed run left are removed, but not the running one's (0.74035ms) ✔ the lab asks the package registry for read only (0.343734ms) ✔ each check gets a fresh directory of its own (0.395055ms) ﹣ what earlier checks left is cleared, and what cannot be is named rather than tripped over (0.055271ms) # running as root, which no directory mode stops from removing anything ✔ leftover checks are cleared, whoever runs the check (0.778571ms) ✔ the lab's tools bundle names its bound credential and binding in its env (0.151962ms) ✔ a run names the package registry by the address the registry gives its tarballs (19.054202ms) ✔ a registry that does not give an address the lab takes leaves the forge's, and the reason is said (3.938365ms) ✔ a run's npm configuration names the registry it was given, and the forge's own address only without one (0.333575ms) ✔ a tarball is taken at the forge's public root, and at no other port or scheme of its host (1.084935ms) ✔ the public root is read from the binding, and only an https root at the npm path is taken (1.497931ms) ✔ a forge setting that is not http(s) trusts no origin of its own (0.571448ms) ✔ a run whose only test skipped as NOT REACHED, from a lab of before, reads as failed with the reason (1.496481ms) ✔ the lab's own NOT PASSED and skipped lines are read, a test name's punctuation and all (0.68571ms) ✔ a run that passed tests and skipped one for a declared reason stays passed, its skip listed (0.666839ms) ✔ a run that ran no test at all is not passed (0.110731ms) ✔ a line a test printed that looks like the suite's is not read as the suite's (0.124471ms) ✔ not reached is matched in any case (0.112771ms) ℹ tests 44 ℹ suites 0 ℹ pass 43 ℹ fail 0 ℹ cancelled 0 ℹ skipped 1 ℹ todo 0 ℹ duration_ms 218.058181 ```
mesh-admin merged commit af5cf53a34 into main 2026-10-14 05:12:38 +00:00
mesh-admin deleted branch fix/698-the-lab-names-its-base-age 2026-10-14 05:12:39 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
mesh-admin
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
jschoubben miel.truyen
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: novox/mesh-catalog#288