A provider whose build does not offer the provision it was asked for must
not be counted as having delivered it; the provisioner reads the build's
manifest (the builds read serves only this offer check and makes no plan,
ADR 0593 section 8) before it takes the provider's have as the outcome.
mesh/check failed (60): integration: mesh-knowledge/cmd/mesh-knowledge.TestARecordThatNeedsTheOperatorIsAc…; and 59 more | 7c78fd2e0c4f on f469eb387a30
A pass runs on every change of its node, and a newly placed module stays
unpublished for the minutes its build takes, so counting passes made every
new placement unhealthy while nothing was wrong. The health now says a hold
only once a module has stayed unpublished at one build for longer than 45
minutes (the build step's bound is 30), with since when; a published module
or another build starts the clock again.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
A node-wide hold while any module's composition was unpublished meant one
placed module whose newest build never completes blocked every retirement on
its node for good, said only as a line of the pass. The incomplete first read
is already covered by the load gate, and an unpublished module's own plan and
keys are already kept as held. So the node-wide hold goes; a removed consumer
retires beside an unpublished module, and a module held unpublished for more
than two passes in a row is said on provisioner_health with its build.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
At 2026-11-02 22:55:58 the provisioner's first pass after a new membership read
part of the picture, took the modules missing from it as handing nothing, set
21 route keys of running web apps retiring and asked the operator to remove
them; 3 s later the next pass put every plan back. Only 3 asks reached the
router; the other 18 were refused yet counted asked. Third of its kind (937,
1332), so the kind is fixed, not the instance.
Quick fix: wait for the watched reads' first load before any pass.
Thorough fix, chosen because the kind recurred three times and its obvious
answer would cut running apps off their routes:
- no pass judges before placement, wants, haves, the indexer's modules and the
published compositions have loaded once; a read error fails the pass
instead of reading as empty;
- while a module's composition is not published at its build, no key on that
node turns retiring and no plan is removed; a placed module with no known
build is held, not counted as handing nothing;
- a key whose consumer still has a want or a have on the node and no plan read
in this pass is left as it is;
- a pass that would retire most of a node's consumers at once (more than 3, or
more than half and at least 2) retires and asks nothing and says it as a
fault through provisioner_health (runtime-health, so a condition names the
likely cause); provisioner_compose retire-many is the operator's word that
such a removal is meant;
- at most 3 asks stand open (the router's bound per asker); the rest wait and
are raised as one ends, and an ask the router holds no record of a minute on
is raised again.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
The node-engine alone composes (ADR 0593 §6, §8), so the provisioner stops
composing with frombus and fromruntime and reads each module's plan from
composed-<node> (read as mesh-host.composed): at its want, or the build its
have runs, and at the indexer's newest build, so a key ahead of its want
still carries its plan entry. A refused or not yet published composition
holds the module with its last plan kept. The golden test proves the plans
and provisions are today's, captured from the composing holder before the
change. It no longer reads the builds or the resolved values.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
A reader on an older mesh-sdk refuses a provision field it does not know,
so a key a plan names carries no entry: such readers skip only a key ahead
of its consumer's want, which they could never bind (issue 1126 review 1).
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
mesh-provisioner keys provisions from a consumer's wanted and newest builds
but writes the wanted build's plan alone, so a requirement the newest build
adds was never bound. Each key now carries its plan entry, and a key ahead of
its want says in its why that no plan names it yet and its provider binds it
from the entry (issue 1126 review 1).
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
provisioner_compose took 2.6-13.6 s per node with nothing changed, CPU
bound. The node is now composed at its wanted and its newest builds over
one read (frombus.PlansHeldTwice, whose composer resolves each machine
once), its own plans, provisions and refusals and the providers' entries
are read from the watched copy, its own writes taken in at once, and a
provider's manifest once by digest: no read by key per composition is
left. The result says where the time went, phase by phase, and a slow
composition that changed nothing says it in the log.
BenchmarkComposeNode times it over a mesh captured with the read tools
(MESH_COMPOSE_CAPTURE): novox 11.7 s to 0.49 s, ace 0.49 s, g14 and
shanks 0.39 s; plans and provisions identical to before over nine
compositions of the four nodes.
Needs mesh-sdk feat/compose-once (5d26c65); go.mod still pins the old
mesh-sdk until the group pins its head.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
ADR 0580 clause 2 as revised: the holder hears haves by watch as it hears resolved values, so a have's change recomposes
its node from the copy it already holds.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
Composing one node read every placement, have, module and resolved key one at a time and took past 30 s with the bus
local (ADR 0580 clause 2): the holder now reads them from mesh-sdk's watched copy, its watch handlers queue the nodes a
change touches after the copy took it, and a resolved value's change recomposes its node. mesh-sdk pinned at ba8b8f2.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
mesh/check checking b5475cfa5c3d on 395a37dddab9 | novox/mesh-catalog#571 in group fix/held-module-and-assigner-contract
mesh-healers slept 2 s before its first subscribe to the keeper's conditions; its whole test delivered
condition-raised once the lease was read, often before that, and the event was dropped: "never: the would-act
written" in about two runs of three on main, failing the group check. Eight modules had the same shape.
Quick fix: sleep after a failed subscribe in mesh-healers alone.
Thorough fix (chosen, the kind is in eight modules): every loop that asks the runtime for events or a watch
waits for stdio.Served() and subscribes at once, sleeping only after a refusal (mesh-healers, mesh-config,
mesh-mail, mesh-issues, mesh-provisioner, mesh-watcher, mesh-delivery-gate); mesh-knowledge, whose pinned
mesh-sdk has no Served, subscribes first and sleeps after a refusal. The healers' whole test delivers the
event only once the bundle has subscribed it, as the runtime's consumer holds it until then.
mesh/delivery-group group fix/held-module-and-assigner-contract rejected: composed together: the checker's verdict at the members' heads (c82f18b5e6bb): integr…
mesh/check failed: integration: mesh-healers/cmd/mesh-healers.TestTheHealersWholeActOnTheBusAndBrake: never: the would-act written | b8ce4f0c0ac5 on 3…
The sweep of 2026-10-30 20:10: <node>.claude-code.gitea-read and <node>.mesh-build-node.npm-package-registry failed on
all four nodes with "gitea does not serve": gitea's provider helper answers only forge-account and refuses every other
bind into gitea.bound, and the provisioner took any entry of a module declaring `bound` as the outcome. And
novox.gitea.route, novox.keycloak.route and <node>.claude-code.mcp-endpoint stood asked: route-proxy and node-tools keep
entries the manifest did not read.
Quick fix: add route-proxy.bound and node-tools.bound to the reads. Thorough fix: judge each offer by who answers it
(answered-by: helper reads the entries, any other offer of a module that names a helper is judged by its have), and say
on gitea's forge-account that the helper answers it. Chose both: the reads alone leave gitea's two provisions failed.
mesh-provisioner, mesh-config, mesh-issues and mesh-healers each wrote a watch-again loop (`watched`). mesh-sdk
61b67d61 adds KeptState.Follow, which asks again with backoff and says each give-up, and its watches now answer a
change at once and run the handler beside the runtime (the provisioner's "names no watch of this bundle"). The four
modules use it; their loops are gone. Built and tested against the mesh-sdk group head through a go.work that is not
committed: their mesh-sdk pins move to that head with the group.
novox since 15:36 (nats's built manifest) and ace since 15:37 (baserow's config.json) had no plans at all, so the
vault refused every secret their clients asked. The holder now composes with frombus.PlansHeld: a module whose build
or plan refuses is held, its why kept in the new state refused and read by provisioner_plans, its last plan and its
provisions left as they were, and every other module on the node composed and written. Pins mesh-sdk 59311407.
mesh/delivery-group group feat/774-l3-reland rejected: composed together: the checker's verdict at the members' heads (b48c90fea31f): integration: mesh-self-ch…
mesh/check failed: integration: mesh-self-check/cmd/mesh-self-check.TestTheSelfCheckWholeRunsOnTheBus: the run answered {Run:doctor-1793351737-1 At:20…
One commit with exactly the tree of the reviewed head 35b4509c0a, on main 12c2574c9d, so two compiled programs added and removed on the branch leave no trace in history (issue 1577).
jschoubben
mesh-tools