mesh-provisioner: never ask to retire access on an incomplete read #599

2026-11-02T22:23:25Z

mesh-tools commented

2026-11-02 22:23:25 +00:00

Issue 940 (a provider asks the operator to remove most of what it holds instead of saying it is a fault), met a third time on 2026-11-02 22:55: mesh-provisioner's first compose after a new membership read an incomplete picture, took 21 web apps (6 on novox, 15 on ace) for handing nothing, set their route keys retiring and asked the operator to remove their route access; 3 s later every plan came back. Only 3 of 21 asks reached the router (three open per asker); the rest were refused silently yet marked asked. The operator pressed Keep; nothing was removed.

Quick fix (wait for the first load) against thorough fix (the kind), thorough chosen:
(a) a pass judges nothing until placement, wants, haves, indexer modules and published compositions have each loaded once; a failed haves or placement read fails the pass; a module whose composition is not published at its build holds only its own plan and keys; a placed module with no known build is held;
(b) a key whose consumer still has a want or a have on the node is never retired;
(c) a pass that would retire most of a node's consumers retires and asks nothing and says it on the new provisioner_health tool (runtime-health), naming the likely cause; provisioner_compose takes retire-many: true as the operator's word;
(d) at most three asks stand open, the rest wait and are raised as earlier ones end; an ask the router has no record of a minute later is raised again;
and a hold lasting over 45 minutes at the same build (longer than the 30-minute build step bound) shows on provisioner_health, naming node.module, build and since when.

Tests: two table-driven tests; every new case fails on main and passes on this head; gofmt, vet with and without the integration tag, the module's unit and integration tests, and sdkjudge, health and manifests as the running checker places siblings pass. The running indexer's judge (mesh-delivery 073e3fab, mesh-sdk 7c015b6) refuses none of 157 manifests at this head. Review: three rounds (a node-wide hold narrowed to the module; a pass-counted alarm timed). Known limit, said: a newly placed module whose build waits in the queue for more than 45 minutes shows unhealthy though nothing is wrong.

The head's last commit is empty: the first check failed only because ace's root disk was full (No space left on device in mesh-mail's and mesh-knowledge's store tests); the operator asked it be checked again, and no verb re-asks a failed check.

Reviewed-by: review-1126@shanks at 134baf0e3b

https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL

Issue 940 (a provider asks the operator to remove most of what it holds instead of saying it is a fault), met a third time on 2026-11-02 22:55: mesh-provisioner's first compose after a new membership read an incomplete picture, took 21 web apps (6 on novox, 15 on ace) for handing nothing, set their route keys retiring and asked the operator to remove their route access; 3 s later every plan came back. Only 3 of 21 asks reached the router (three open per asker); the rest were refused silently yet marked asked. The operator pressed Keep; nothing was removed. Quick fix (wait for the first load) against thorough fix (the kind), thorough chosen: (a) a pass judges nothing until placement, wants, haves, indexer modules and published compositions have each loaded once; a failed haves or placement read fails the pass; a module whose composition is not published at its build holds only its own plan and keys; a placed module with no known build is held; (b) a key whose consumer still has a want or a have on the node is never retired; (c) a pass that would retire most of a node's consumers retires and asks nothing and says it on the new provisioner_health tool (runtime-health), naming the likely cause; provisioner_compose takes retire-many: true as the operator's word; (d) at most three asks stand open, the rest wait and are raised as earlier ones end; an ask the router has no record of a minute later is raised again; and a hold lasting over 45 minutes at the same build (longer than the 30-minute build step bound) shows on provisioner_health, naming node.module, build and since when. Tests: two table-driven tests; every new case fails on main and passes on this head; gofmt, vet with and without the integration tag, the module's unit and integration tests, and sdkjudge, health and manifests as the running checker places siblings pass. The running indexer's judge (mesh-delivery 073e3fab, mesh-sdk 7c015b6) refuses none of 157 manifests at this head. Review: three rounds (a node-wide hold narrowed to the module; a pass-counted alarm timed). Known limit, said: a newly placed module whose build waits in the queue for more than 45 minutes shows unhealthy though nothing is wrong. The head's last commit is empty: the first check failed only because ace's root disk was full (No space left on device in mesh-mail's and mesh-knowledge's store tests); the operator asked it be checked again, and no verb re-asks a failed check. Reviewed-by: review-1126@shanks at 134baf0e3b8fe8c1de25285874ed33f3b5ad4a6f https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL

mesh-tools added 3 commits 2026-11-02 22:23:25 +00:00

mesh-provisioner: judge no retirement from a partial picture (issue 940) a1d98c2438

At 2026-11-02 22:55:58 the provisioner's first pass after a new membership read
part of the picture, took the modules missing from it as handing nothing, set
21 route keys of running web apps retiring and asked the operator to remove
them; 3 s later the next pass put every plan back. Only 3 asks reached the
router; the other 18 were refused yet counted asked. Third of its kind (937,
1332), so the kind is fixed, not the instance.

Quick fix: wait for the watched reads' first load before any pass.
Thorough fix, chosen because the kind recurred three times and its obvious
answer would cut running apps off their routes:
- no pass judges before placement, wants, haves, the indexer's modules and the
  published compositions have loaded once; a read error fails the pass
  instead of reading as empty;
- while a module's composition is not published at its build, no key on that
  node turns retiring and no plan is removed; a placed module with no known
  build is held, not counted as handing nothing;
- a key whose consumer still has a want or a have on the node and no plan read
  in this pass is left as it is;
- a pass that would retire most of a node's consumers at once (more than 3, or
  more than half and at least 2) retires and asks nothing and says it as a
  fault through provisioner_health (runtime-health, so a condition names the
  likely cause); provisioner_compose retire-many is the operator's word that
  such a removal is meant;
- at most 3 asks stand open (the router's bound per asker); the rest wait and
  are raised as one ends, and an ask the router holds no record of a minute on
  is raised again.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL

mesh-provisioner: an unpublished composition holds only its own module (issue 940, review 1) 9239177efd

A node-wide hold while any module's composition was unpublished meant one
placed module whose newest build never completes blocked every retirement on
its node for good, said only as a line of the pass. The incomplete first read
is already covered by the load gate, and an unpublished module's own plan and
keys are already kept as held. So the node-wide hold goes; a removed consumer
retires beside an unpublished module, and a module held unpublished for more
than two passes in a row is said on provisioner_health with its build.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL

mesh-provisioner: say a lasting unpublished hold by time, not by passes (issue 940, review 2)

mesh/delivery closed: a newer head of the same pull request

Details

mesh/check failed (60): integration: mesh-knowledge/cmd/mesh-knowledge.TestARecordThatNeedsTheOperatorIsAc…; and 59 more | 7c78fd2e0c4f on f469eb387a30

7c78fd2e0c

A pass runs on every change of its node, and a newly placed module stays
unpublished for the minutes its build takes, so counting passes made every
new placement unhealthy while nothing was wrong. The health now says a hold
only once a module has stayed unpublished at one build for longer than 45
minutes (the build step's bound is 30), with since when; a published module
or another build starts the clock again.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL

mesh-tools commented

2026-11-02 22:23:27 +00:00

Delivery novox/mesh-catalog@134baf0e3b8f — merged since 2026-11-02T23:26:51Z

Source — source.main moved to novox/mesh-catalog@0b2d393b11. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Transitions

2026-11-02 22:52 (new) → proposed (announced): novox/mesh-catalog#599's head announced
2026-11-02 22:52 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-11-02 23:26 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-catalog@134baf0e3b8f` — **merged** since 2026-11-02T23:26:51Z **Source** — source.main moved to novox/mesh-catalog@0b2d393b1100. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Transitions** - 2026-11-02 22:52 (new) → proposed (announced): novox/mesh-catalog#599's head announced - 2026-11-02 22:52 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-11-02 23:26 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

jschoubben added 1 commit 2026-11-02 22:52:46 +00:00

Check again: the previous check failed only on ace's full root disk

mesh/delivery merged: source.main moved to novox/mesh-catalog@0b2d393b1100

Details

mesh/check being carried to 134baf0e3b8f on 0b2d393b1100 | from f469eb387a30

134baf0e3b

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL

mesh-tools commented

2026-11-02 22:52:50 +00:00

mesh-delivery: the operator's Approve of this pull request's merge into main is ended: its head 7c78fd2e0c is no longer the pull request's head, or the pull request closed. Nothing is merged on it. The operator is asked again once the checks pass at the new head.

mesh-delivery: the operator's Approve of this pull request's merge into main is ended: its head 7c78fd2e0c4f is no longer the pull request's head, or the pull request closed. Nothing is merged on it. The operator is asked again once the checks pass at the new head.

mesh-tools merged commit 0b2d393b11 into main

2026-11-02 23:26:49 +00:00

mesh-tools deleted branch fix/940-no-retire-on-partial-read

2026-11-02 23:26:49 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-catalog#599