mesh-provisioner-client's report now carries failing-since (mesh-catalog 3aae6c08), kept while a module
stays failed however its words change. S21 measures a failed placing from it, falling back to the
report's time from an older client; the hysteresis of 6ffcea4f (an open condition stays while any
failure stands) is kept. This is the thorough fix that commit named.
Research 072 piece: milestone 3 (health leaves the controller): the judgement reads the owner's state.
The sweep's controller changes use mesh-sdk's go/kvlimit and the ReadsAnnouncedBy that grants named
readers. At ef1ec1c9 mesh-config is a named reader of the controller's lease (its take-over reads it,
ADR 0561 §3), so the read test now expects it granted and a module not named refused. The captured SDK
(testdata/beside, conformance/events) is taken again at the pinned commit, as CAPTURED says; its file
is unchanged.
mesh-sdk #136 (42988a7) keeps the node-engine's manifest out of every placed user in buscompose: where a
placement names it, no user, no process and nothing the tool runner carries; the node's account alone
writes the engine's state, and a second writer has the writers table refuse the whole list (met twice on
2026-10-30: mesh-host_acts, then mesh-host_health). The controller still composes the bus's users with
its own copy (internal/broker Users), which lacked the guard. The same two predicates, Composed and
CarriedByRunner, now guard it.
Read on the bus at 18:30: the bus follows the grants part's list (mesh-grants_user-list.sealed), so the
controller's copy is not what the bus loads; it still composes the bus module's files and the controller's
own grants, where a refusal would stop that composition.
Quick fix: the same guard here. Thorough fix: delete the controller's copy of the bus composer and use
buscompose (about 18,000 lines of internal/broker with its tests, its genesis list and its golden file).
Chose quick: the controller is being deprecated and the grants part already holds the list; the copy goes
with the controller's composition of the bus module, not in a sweep.
Research 072 piece: 03-domains, mesh-grants (membership and grants leave the controller): until the copy is
removed, it composes as the SDK does.
At 17:44:53 the controller said 30 calls (tools, nodes, modules) were "kept in memory only: the bus is
not taking calls' records". No write had failed (no "could not be kept" line): a burst of about 200 read
calls in one second queued a begin and a finish each into a channel of 256 while the one writer put
them one by one, and the channel overflowed.
The queue now holds the latest state of each waiting call (a finish replaces its own waiting begin, as
only the last state is kept anyway), in first-queued order, bounded at 8192 calls; the writer takes the
whole queue at each wake. A burst now costs one write per call and drops nothing.
Quick fix: a larger channel. Thorough fix: coalesce by call, so the writes a burst asks are halved and
the bound is spent on calls, not states. Chose thorough; both are small.
Research 072 piece: §2's "read calls said" row: the calls bucket is the controller's own state on the
bus, kept from events (each call's change), never re-read.
At 16:36:51 the vault handed own/novox/nats/broker-nats-tools to the controller as a plain random own
secret, and owes the bus-account hashes of it and of the controller's broker-address and
broker-management for good ("holds no account the controller hands (not JSON)"). The controller decided
that: its send asks the vault (`get`) for every own secret of a core module, a bus process's account
among them, before the account is issued; the vault, holding none, makes one.
The send now leaves out of that ask a process's account nobody issued (the same rule as its warning
line); the machine keeps the controller's stand-in until the account is issued and handed to the vault.
The vault side (not minting an account-shaped name, and replacing the random value already held) is
mesh-catalog's mesh-vault, with build-catalog.
Quick fix: this filter on the ask. Thorough fix: process accounts minted by the vault itself (ADR 0113 §6,
research 072 milestone 6), so no part asks for one before it exists. Chose quick: milestone 6 is not
started, and the filter removes the cause on the controller's side now.
Research 072 piece: milestone 6 (identity and bus accounts to the vault): until then the controller asks
the vault only for what exists.
mesh-wireguard is provided by the mesh (sdkdelivery.ProvidedByMesh): no repository builds it and no
fingerprint will ever name it, yet every composition said it composes "with the node's own: no
fingerprint of the new form names them", and the switch's list counted it "not known". Both now leave
such a module out; its resources still go with the node's own composition, as before.
Quick and thorough are one here: the SDK names the modules the mesh provides, and every line that judges
fingerprints filters by it.
Research 072 piece: §6 milestone 2 (the composer reacts; its words say only what a change can fix).
The controller's other writer of the 16 KiB bucket of merges, the keeper of mesh-delivery's merge words,
called every put failure "not now". A word too large would be retried as the forge's merges were. It now
puts through kvlimit.Put: a value that does not fit is refused before it is put and returned as an answer
(the events consumer takes it, max-deliver 5 then a dead letter), and only another failure is retried.
It is the one write of the controller whose failure was retried as "not now" (searched: every Put,
Create and Update wrapped in ErrTryAgain).
Needs the group's mesh-sdk head (go/kvlimit, 7b02f63 or later).
Quick fix: none smaller. Thorough fix: every bucket write through kvlimit; this is the only one whose
refusal was retried, the others fail once and say so. No new test: kvlimit is tested in mesh-sdk, and an
integration test here would push the repository further over its budget (rules-are-weighed).
Research 072 piece: 03-domains (Control shrinks): the merge words stay the controller's only until the
registration of a moved module leaves it.
2536 reads of mesh-controller.lease were refused on 2026-10-30 (mesh-vault, mesh-grants, the assigner,
artifact-store-tools, the watchman, mesh-config). The controller composed the bus's users with its own
copy of a reader-less judgement (internal/broker ReadsAnnounced), which knows no named readers: the lease
is announced readable by nobody and read by the parts named for it (ADR 0547 clause 9, ADR 0561 §3). The
copy is deleted; BusRecords judges each read for its module with buscompose.ReadsAnnouncedBy. Each
refusal was also printed on every composition, about 420 times a read; it is now said once per process.
mesh-config's read of the lease stays refused: it reads the lease for its take-over (hand-back ends at
the lease write, ADR 0561 §3) and is not among the named readers. That is mesh-sdk's NamedReaders,
routed to the lead.
Quick fix: mark the lease readable in the controller's announcement (tried and reverted: it would open
the lease to every module and contradict ADR 0561 §3). Thorough fix: one judgement, the SDK's, for every
composer. Chose thorough.
Research 072 piece: 03-domains, mesh-grants (bus membership and grants leave the controller): the
controller composes reads with the same SDK function the grants part uses.
About 35 conditions from 10-20 to 10-27 still showed as current. Read on the bus at 18:06: S23's three
failed plans (last observed 10-24) and D14's four stalled deliveries (last observed 10-24) come from a
signals row and a probe this controller no longer has; nothing reconciles their source, so nothing clears
them. The other old ones are observed every tick and are not stale by the controller: S9's dead letters
are still held, S15's hand acts are still kept, and DM's 26 stuck keys are the watchman's (it never writes
a key whose watched key aged out as moved) — routed to mesh-delivery.
Each watchdogs tick now clears every open condition whose source has a row's or a probe's form (S12, D4,
DW, D-root) and is in neither the signals table nor the probe registry, saying the check was removed.
Sources of other forms are left to their own word.
Quick fix: clear the seven by hand. Thorough fix: the controller judges its own retired checks on every
tick, so the next removed row or probe clears what it raised. Chose thorough: rows and probes are being
moved out of the controller one by one (ADR 0564), each removal would leave this again.
Research 072 piece: milestone 3 (health leaves the controller): the conditions keeper drops what no check
of its own observes, so the move to mesh-conditions starts from a bucket of live conditions.
machine.novox.apply-failing cleared at 16:56 while the vault refused every secret there: the provisioner
client says a failure again whenever its words change, with a new time, and S21 measured each module's
failure from that newest time, so every new wording hid the failure for one apply interval (raised again
17:01). The same holds for the node-engine's own failed apply, whose since restarts with new words.
The bound now gates raising only: once the condition is open (read from the keeper's open conditions on
the bus), it stays while any module's apply fails on that machine, and clears when none does.
Quick fix: this hysteresis on the open condition in the controller. Thorough fix: the provisioner client
carries the start of a failure run (failing-since) in its report, so every reader measures from the true
start; that is mesh-catalog's mesh-provisioner-client, routed to the lead. Chose the quick fix here,
because the check moves to mesh-watchdogs (ADR 0564) and the condition's own state already proves the run.
Research 072 piece: milestone 3 (health leaves the controller): the judgement reads bus state, the open
condition, rather than a report's time.
The controller-merges consumer (no max-deliver, one at a time) held the de-spiegel merge e25e1beb from
10-29 09:39: keepForgeMerge put an announcement with its Paths into delivery-merges (16 KiB values), got
10054, retried "not now" forever, and 167 merges queued behind it. Every five-minute hold outran the deaf
bound, so core.controller.deaf flapped.
Merges are mesh-delivery's: the gate merges on the Approve and the forge bridge says each merge on its
stream FORGE (pull request, head, merge commit), kept a week. The controller's copy was a leftover, read
only at a moved module's registration (ADR 0420 §7, ADR 0432). That read now finds the merge on FORGE;
the consumer, its follow, its handler and its hold delays are removed, and every assertion removes the
retired consumer from the bus.
Kind fix for the watchdog: the event loop records when a message it held back is due again, and deafness
counts from then, so no hold longer than the bound (on any consumer) reads as deafness.
Quick fix: strip Paths from the kept announcement and add max-deliver with a dead letter. Thorough fix:
remove the duplicate duty and make the deaf watchdog hold-aware. Chose thorough: the controller is being
deprecated and the merges already have an owner; the hold-vs-deaf confusion is a kind, not one consumer.
Research 072 piece: 03-domains (Control shrinks to placement and composition; the delivery parts own
what a merge is) and milestone 3 (health signals stop sharing the controller's loop assumptions).
The release's fourth check failed TestModuleCheckPassesTheCatalogue with 27 problems. The checker clones the catalogue
beside a controller's tests at its trunk, even in a group holding a catalogue pull request (mesh-delivery checker
internal/check/beside.go besideFor, ADR 0484 clause 4). The release's sdk refuses an offer stating no credential form,
and the trunk's catalogue states none until the release's catalogue merges.
- Quick fix (not chosen): recapture alone, which the test never reads while MESH_CHECK_BESIDE is set.
- Chosen: the test reads the captured copy, now at the group's catalogue f3a57dcc, through beside.CapturedDir. Every other
beside test still reads the clone. The check itself is unchanged: every rule, every manifest. Moving CAPTURED to the
catalogue a change is released with is part of that change.
Run as the checker runs it (catalogue at main, mesh-host at main, mesh-sdk at the pin, mesh-controller's trunk
beside): the whole unit and integration suite passes, 1706 tests, 8 skipped. The test also passes with the catalogue at
f3a57dcc beside, and with nothing beside.
mesh/delivery-group group feat/774-l3-reland rejected: composed together: the checker's verdict at the members' heads (8bad7efb3e7a): unit: github.com/novox/me…
mesh/check failed: unit: github.com/novox/mesh-controller/cmd/mesh-controller.TestModuleCheckPassesTheCatalogue: the catalogue does not pass its own c…
The conflicts were go.mod, go.sum and testdata/beside/CAPTURED alone, taken from the release. 2da0de7 is 6e4a301's code;
its conformance events are unchanged.
78c9ec8 is sdk main 307b623 (tree of main's pin dbcaa29) with one parse-only commit: an offer with no form still means
pair. The fixture changes the full sdk forced are reverted to main's; the sdk's conformance events are captured at
78c9ec8 and unchanged. TestModuleCheckPassesTheCatalogue passes, on the captured copy and on mesh-catalog main 96b129e5
cloned beside. Whole unit and integration suites: 1706 tests, 8 skipped for a registry, a container, a refusing bus or a
lab's run.
No code of ours changes; the sdk's conformance events are unchanged and captured at 6e4a301. The fixtures of
identity_bound_test and issued_test say the credential form their rows meant (pair, none), since the sdk no longer reads
a missing form as pair (6b8c694).
NOT GREEN: TestModuleCheckPassesTheCatalogue fails. The sdk refuses every offer that states no credential form, and
main's catalogue states none (27 offers). Recapturing cannot fix it: this pin and main's catalogue cannot both stand.
The sdk's conformance events are unchanged at 6e4a301 and captured so. The whole integration suite passes at this head
against a local store and bus: 1706 tests, 8 skipped for a registry, a container, a refusing bus or a lab's run.
The sdk now refuses an offer that says no credential form (6b8c694, ADR 0570). So:
- the catalogue copy in testdata/beside is captured again at the release branch's catalogue 8ae8e88d, whose offers each
say one, and the sdk's conformance events at 007b6fd, as CAPTURED asks;
- the fixtures of identity_bound_test and issued_test say the form their rows meant: pair for a login of each
consumer's own, none for a keyless provision. No behaviour of the controller changed.
The controller on dbcaa29 parses manifests strictly and refuses `bus` on an artifact and `max-age-days` on a log, both
in the re-land group's catalogue. The SDK's conformance events are captured again at the pinned commit, as
testdata/beside/CAPTURED asks; they are unchanged.
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: the compose ask mvsg85b34d25b4 never ende…
mesh/delivery-group group feat/1126-placement-seat merged: every member is merged
mesh/check being carried to 8f847318963c on 0fbb5ef35991 | from 2dafbe60e939
A plain delete marker on the lease key is also written by a command's own
lease and by a take given back over the floor; counted as a handover, it
would record a crashed holder's epoch released and say nothing. Release now
writes the marker itself, compare-and-set on the key's revision, with the
header Mesh-Lease-Released naming its epoch, and the next holder ends only
that epoch as released. A takeover whose predecessor cannot be read is now
raised as an expiry, not only printed.
The 2026-10-26 replacements showed the lease was given back, but last: the
shutdown took 10 s, the gate had closed at half the key's age, and the release
recorded the epoch released only through the gate, so the next holder found
it expired and S12 said it, urgent, for an hour.
Now the stop signal gives the key back first and records the epoch released
whenever the delete succeeded; a candidate reads the key's delete marker on
the bus to tell a handover from an expiry; and a real takeover (lost or
expired) is raised and cleared once by the controller that took it, instead
of an hour of S12 observation.
Research 072 piece moved: the lease's takeover read from its own key on the
bus and said as an event (03-domains, mesh-controller row; 04-findings 2,
the watchdogs' age reading).
Quick fix weighed: drop the gate check from the release record only; it
leaves the release last in the shutdown and the hour-long alarm for a real
takeover.
mesh/check being carried to 8d58afea6cb8 on 2dafbe60e939 | from b611f7375a47
The server counts a consumer's pending after it acknowledges the publish, so a read at once failed the try of
main on a loaded ace (TestAResetOfTheEventsConsumerLeavesTheMergesAndTheirConsumerAlone, 2026-10-26 22:24 UTC).
Fixed for the kind, not the one test: all six reads of a positive count go through the package's own pendingOf,
whose deadline was 1 s.
Research 'The controller, NATS-native': none; this changes tests only, not a part of the controller.
mesh/check failed: unit: github.com/novox/mesh-controller/cmd/mesh-controller.TestModuleCheckPassesTheCatalogue: the catalogue does not pass its own c…
mesh/delivery-group group fix/1126-vault-seat-own merged: every member is merged
mesh/check failed: unit: ..test: ✔ a transient empty list for four passes retires nothing; five stable passes retire, data kept (290.408814ms) | 4e9e5…
mesh/check failed: ace: novox/mesh-catalog#505 does not merge into the trunk: files in conflict: .gitignore | 7c8a2e03dd8e on 2ecfbb3a29cb | unit 1245…
mesh/check pass 9ad8715871ea on 2ecfbb3a29cb | carried from e1a55f099b4e | unit 1023 run, 171 kept, 2 skipped | integration 1701 run, 214 kept, 9 skip…
The controller is the only composer, and its pin (10b6e202) predates mesh-sdk #95: without the bump the
build runtimes never get yields {priority idle, cpu-share} or MESH_BUILD_CPU_SHARE (ADR 0553, ADR 0554,
change B). The operator said yes to this one change of the frozen controller on 2026-10-26 15:54.
A composition at both pins over every node's assignment differs only in mesh-build-go, mesh-build-node and
mesh-build-image (the yields and the share word); every other composition is byte-identical. The SDK's
conformance capture moves with the pin, as testdata/beside/CAPTURED asks.
Research 072 (the controller, NATS-native): no piece moved; a pin bump the operator allowed.
The checker hears from DEAD_LETTERS which of its asks a build seat gave up on, and ends the try or asks the check again;
the controller composes today's user list, so its copy of the composer carries the grant too. NATS-native research piece:
none moved; the composition itself moves to mesh-grants (ADR 0547), and this keeps the copy in step with mesh-sdk.
jschoubben
mesh-tools