The assigner never keeps a want back because a provider is unhealthy, and the canary gates only modules that run (issue 1960) #110

2026-10-29T15:05:42Z

mesh-tools commented

2026-10-29 15:05:42 +00:00

Why

Recovery of novox (2026-10-29). de-spiegel is held on novox (an unresolved config value), and the hold spread: mailu ("it reads what de-spiegel contributes"), then route-proxy, nftables and postgres were held too, so every site behind route-proxy is down. The assigner kept wants back for de-spiegel and link2pay ("rule 5: no provider of route runs healthy") and photos ("no provider of s3-bucket runs healthy"). The operator: one module should never block another. mesh-sdk #128 stopped the spread to consumers and collectors but kept dependents() in compose/frombus/answer.go, which held the provider of what a held module contributes to, transitively.

What changes

mesh-sdk: dependents, heldBehind and the held-behind path in heldAsToday go. A held module offers the others what it last offered (its contributes and listens from the manifest of the build it runs), and only the provision that does not compose is left out.
mesh-delivery (assigner): rule 5 holds no want; an unhealthy provider is recorded and said once. Rule 3 (the canary) skips a first want only for a node that does not run the module; moves of a running module keep the canary order.
mesh-host: pins mesh-sdk dbcaa29 (the composer it runs) and expects the proxy composed beside a held de-spiegel.
This departs from ADR 0429 §6 / design 47 (canary first and every provider healthy for every want); a decision record follows.

Quick fix or thorough fix

Quick: drop dependents() and skip the two rules for these three modules.
Thorough, chosen: no spread anywhere, the held module's offer read from what runs, per-provision leave-out, and no want held back by an unhealthy provider for any module.

How it is checked

Rows added in existing tests (no new test functions): de-spiegel held while mailu and the proxy compose; the assigner gives de-spiegel and link2pay wants with route-proxy down, photos a want with minio unhealthy, and an old-form running node still waits for the canary.
Live: on novox mailu, route-proxy, nftables and postgres compose; de-spiegel, link2pay and photos get wants; the sites behind route-proxy answer (de-spiegel's own names wait for its config value).

Group: delivery group fix/1960-no-hold-spread (mesh-sdk, mesh-delivery, mesh-host).

Reviewed-by: rev-hold@shanks at d54eab4052

## Why Recovery of novox (2026-10-29). de-spiegel is held on novox (an unresolved config value), and the hold spread: mailu ("it reads what de-spiegel contributes"), then route-proxy, nftables and postgres were held too, so every site behind route-proxy is down. The assigner kept wants back for de-spiegel and link2pay ("rule 5: no provider of route runs healthy") and photos ("no provider of s3-bucket runs healthy"). The operator: one module should never block another. mesh-sdk #128 stopped the spread to consumers and collectors but kept `dependents()` in compose/frombus/answer.go, which held the provider of what a held module contributes to, transitively. ## What changes - mesh-sdk: `dependents`, `heldBehind` and the held-behind path in `heldAsToday` go. A held module offers the others what it last offered (its contributes and listens from the manifest of the build it runs), and only the provision that does not compose is left out. - mesh-delivery (assigner): rule 5 holds no want; an unhealthy provider is recorded and said once. Rule 3 (the canary) skips a first want only for a node that does not run the module; moves of a running module keep the canary order. - mesh-host: pins mesh-sdk dbcaa29 (the composer it runs) and expects the proxy composed beside a held de-spiegel. - This departs from ADR 0429 §6 / design 47 (canary first and every provider healthy for every want); a decision record follows. ## Quick fix or thorough fix - Quick: drop `dependents()` and skip the two rules for these three modules. - Thorough, chosen: no spread anywhere, the held module's offer read from what runs, per-provision leave-out, and no want held back by an unhealthy provider for any module. ## How it is checked - Rows added in existing tests (no new test functions): de-spiegel held while mailu and the proxy compose; the assigner gives de-spiegel and link2pay wants with route-proxy down, photos a want with minio unhealthy, and an old-form running node still waits for the canary. - Live: on novox mailu, route-proxy, nftables and postgres compose; de-spiegel, link2pay and photos get wants; the sites behind route-proxy answer (de-spiegel's own names wait for its config value). Group: delivery group fix/1960-no-hold-spread (mesh-sdk, mesh-delivery, mesh-host). Reviewed-by: rev-hold@shanks at d54eab4052d132a120b83356092f1d5e7db738ca

mesh-tools added 1 commit 2026-10-29 15:05:43 +00:00

Never keep a want back for an unhealthy provider, nor a first want for the canary where nothing runs: the module is wanted and its provider down is said (issue 1960)

mesh/delivery merged: source.main moved to novox/mesh-delivery@5e931eac191a

Details

mesh/delivery-group group fix/1960-no-hold-spread merged: every member is merged

Details

mesh/check pass d54eab4052d1 on c1b4533201c8 | unit 3318 run, 1951 kept, 38 skipped | integration 3994 run, 2125 kept, 38 skipped | over bound: unit, …

d54eab4052

On novox on 2026-10-29 the assigner wrote no want for de-spiegel and link2pay (rule 5: no provider of route runs
healthy) nor photos (rule 5: no provider of s3-bucket), and none where no node of a module was online (rule 3). A
module with no want is on no list and nothing applies it; a site runs while the proxy in front of it is down.

- Quick fix: skip rules 3 and 5 for the three modules' first wants. Leaves rule 5 holding every move and the
  canary holding new assignments.
- Thorough fix, chosen: rule 5 holds no want; it is said per node (Outcome.Unreached, journaled once per words by the
  pass). The canary (rule 3) no longer holds the first want of a node that does not run the module. A node that runs
  it keeps the canary order: at an older composed and granted fingerprint of the new form its first want stays at what
  it runs; at an old-form build it waits for the canary as before (issue 1945); at an older fingerprint with no
  composition or grant it gets F. This departs from ADR 0429 §6, which makes every provider healthy a rule of every want.

Tests: rows changed in the existing rule tests, and rows for today's modules (route-proxy down: de-spiegel and
link2pay wanted; minio not healthy: photos moves; novox offline: a first want written, a move kept back).

mesh-tools commented

2026-10-29 15:05:46 +00:00

Delivery novox/mesh-delivery@d54eab4052d1 — merged since 2026-10-29T15:35:30Z

Source — source.main moved to novox/mesh-delivery@5e931eac19. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Group fix/1960-no-hold-spread: novox/mesh-controller@020c6a99b0, novox/mesh-delivery@d54eab4052, novox/mesh-host@ece607e215, novox/mesh-sdk@dbcaa29a1a

composed together: pass — the checker's verdict at the members' heads (f0569710e57f)

Transitions

2026-10-29 15:05 (new) → proposed (announced): novox/mesh-delivery#110's head announced
2026-10-29 15:05 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-10-29 15:35 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-delivery@d54eab4052d1` — **merged** since 2026-10-29T15:35:30Z **Source** — source.main moved to novox/mesh-delivery@5e931eac191a. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Group** `fix/1960-no-hold-spread`: novox/mesh-controller@020c6a99b035, novox/mesh-delivery@d54eab4052d1, novox/mesh-host@ece607e21522, novox/mesh-sdk@dbcaa29a1a38 - composed together: pass — the checker's verdict at the members' heads (f0569710e57f) **Transitions** - 2026-10-29 15:05 (new) → proposed (announced): novox/mesh-delivery#110's head announced - 2026-10-29 15:05 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-10-29 15:35 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools commented

2026-10-29 15:19:03 +00:00

mesh-delivery: the operator's Approve of this pull request's merge into main is ended: a head it covers moved on: the heads it covers are no longer the group's heads. Nothing is merged on it. The operator is asked again once the checks pass at the new head.

mesh-tools merged commit 5e931eac19 into main

2026-10-29 15:35:29 +00:00

mesh-tools deleted branch fix/1960-no-hold-spread

2026-10-29 15:35:29 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-delivery#110