The controller's other writer of the 16 KiB bucket of merges, the keeper of mesh-delivery's merge words,
called every put failure "not now". A word too large would be retried as the forge's merges were. It now
puts through kvlimit.Put: a value that does not fit is refused before it is put and returned as an answer
(the events consumer takes it, max-deliver 5 then a dead letter), and only another failure is retried.
It is the one write of the controller whose failure was retried as "not now" (searched: every Put,
Create and Update wrapped in ErrTryAgain).
Needs the group's mesh-sdk head (go/kvlimit, 7b02f63 or later).
Quick fix: none smaller. Thorough fix: every bucket write through kvlimit; this is the only one whose
refusal was retried, the others fail once and say so. No new test: kvlimit is tested in mesh-sdk, and an
integration test here would push the repository further over its budget (rules-are-weighed).
Research 072 piece: 03-domains (Control shrinks): the merge words stay the controller's only until the
registration of a moved module leaves it.
2536 reads of mesh-controller.lease were refused on 2026-10-30 (mesh-vault, mesh-grants, the assigner,
artifact-store-tools, the watchman, mesh-config). The controller composed the bus's users with its own
copy of a reader-less judgement (internal/broker ReadsAnnounced), which knows no named readers: the lease
is announced readable by nobody and read by the parts named for it (ADR 0547 clause 9, ADR 0561 §3). The
copy is deleted; BusRecords judges each read for its module with buscompose.ReadsAnnouncedBy. Each
refusal was also printed on every composition, about 420 times a read; it is now said once per process.
mesh-config's read of the lease stays refused: it reads the lease for its take-over (hand-back ends at
the lease write, ADR 0561 §3) and is not among the named readers. That is mesh-sdk's NamedReaders,
routed to the lead.
Quick fix: mark the lease readable in the controller's announcement (tried and reverted: it would open
the lease to every module and contradict ADR 0561 §3). Thorough fix: one judgement, the SDK's, for every
composer. Chose thorough.
Research 072 piece: 03-domains, mesh-grants (bus membership and grants leave the controller): the
controller composes reads with the same SDK function the grants part uses.
About 35 conditions from 10-20 to 10-27 still showed as current. Read on the bus at 18:06: S23's three
failed plans (last observed 10-24) and D14's four stalled deliveries (last observed 10-24) come from a
signals row and a probe this controller no longer has; nothing reconciles their source, so nothing clears
them. The other old ones are observed every tick and are not stale by the controller: S9's dead letters
are still held, S15's hand acts are still kept, and DM's 26 stuck keys are the watchman's (it never writes
a key whose watched key aged out as moved) — routed to mesh-delivery.
Each watchdogs tick now clears every open condition whose source has a row's or a probe's form (S12, D4,
DW, D-root) and is in neither the signals table nor the probe registry, saying the check was removed.
Sources of other forms are left to their own word.
Quick fix: clear the seven by hand. Thorough fix: the controller judges its own retired checks on every
tick, so the next removed row or probe clears what it raised. Chose thorough: rows and probes are being
moved out of the controller one by one (ADR 0564), each removal would leave this again.
Research 072 piece: milestone 3 (health leaves the controller): the conditions keeper drops what no check
of its own observes, so the move to mesh-conditions starts from a bucket of live conditions.
machine.novox.apply-failing cleared at 16:56 while the vault refused every secret there: the provisioner
client says a failure again whenever its words change, with a new time, and S21 measured each module's
failure from that newest time, so every new wording hid the failure for one apply interval (raised again
17:01). The same holds for the node-engine's own failed apply, whose since restarts with new words.
The bound now gates raising only: once the condition is open (read from the keeper's open conditions on
the bus), it stays while any module's apply fails on that machine, and clears when none does.
Quick fix: this hysteresis on the open condition in the controller. Thorough fix: the provisioner client
carries the start of a failure run (failing-since) in its report, so every reader measures from the true
start; that is mesh-catalog's mesh-provisioner-client, routed to the lead. Chose the quick fix here,
because the check moves to mesh-watchdogs (ADR 0564) and the condition's own state already proves the run.
Research 072 piece: milestone 3 (health leaves the controller): the judgement reads bus state, the open
condition, rather than a report's time.
The controller-merges consumer (no max-deliver, one at a time) held the de-spiegel merge e25e1beb from
10-29 09:39: keepForgeMerge put an announcement with its Paths into delivery-merges (16 KiB values), got
10054, retried "not now" forever, and 167 merges queued behind it. Every five-minute hold outran the deaf
bound, so core.controller.deaf flapped.
Merges are mesh-delivery's: the gate merges on the Approve and the forge bridge says each merge on its
stream FORGE (pull request, head, merge commit), kept a week. The controller's copy was a leftover, read
only at a moved module's registration (ADR 0420 §7, ADR 0432). That read now finds the merge on FORGE;
the consumer, its follow, its handler and its hold delays are removed, and every assertion removes the
retired consumer from the bus.
Kind fix for the watchdog: the event loop records when a message it held back is due again, and deafness
counts from then, so no hold longer than the bound (on any consumer) reads as deafness.
Quick fix: strip Paths from the kept announcement and add max-deliver with a dead letter. Thorough fix:
remove the duplicate duty and make the deaf watchdog hold-aware. Chose thorough: the controller is being
deprecated and the merges already have an owner; the hold-vs-deaf confusion is a kind, not one consumer.
Research 072 piece: 03-domains (Control shrinks to placement and composition; the delivery parts own
what a merge is) and milestone 3 (health signals stop sharing the controller's loop assumptions).
mesh/check checking 8b32e04e2f8b on 6fe9efa71219 | novox/mesh-controller#482 in group feat/774-l3-reland
The release's fourth check failed TestModuleCheckPassesTheCatalogue with 27 problems. The checker clones the catalogue
beside a controller's tests at its trunk, even in a group holding a catalogue pull request (mesh-delivery checker
internal/check/beside.go besideFor, ADR 0484 clause 4). The release's sdk refuses an offer stating no credential form,
and the trunk's catalogue states none until the release's catalogue merges.
- Quick fix (not chosen): recapture alone, which the test never reads while MESH_CHECK_BESIDE is set.
- Chosen: the test reads the captured copy, now at the group's catalogue f3a57dcc, through beside.CapturedDir. Every other
beside test still reads the clone. The check itself is unchanged: every rule, every manifest. Moving CAPTURED to the
catalogue a change is released with is part of that change.
Run as the checker runs it (catalogue at main, mesh-host at main, mesh-sdk at the pin, mesh-controller's trunk
beside): the whole unit and integration suite passes, 1706 tests, 8 skipped. The test also passes with the catalogue at
f3a57dcc beside, and with nothing beside.
mesh/delivery-group group feat/774-l3-reland rejected: composed together: the checker's verdict at the members' heads (8bad7efb3e7a): unit: github.com/novox/me…
mesh/check failed: unit: github.com/novox/mesh-controller/cmd/mesh-controller.TestModuleCheckPassesTheCatalogue: the catalogue does not pass its own c…
The conflicts were go.mod, go.sum and testdata/beside/CAPTURED alone, taken from the release. 2da0de7 is 6e4a301's code;
its conformance events are unchanged.
mesh/check pass d96542cd2503 on e04846e9fa49 | unit 9174 run, 546 kept, 51 skipped | integration 9952 run, 659 kept, 60 skipped | over bound: unit, in…
78c9ec8 is sdk main 307b623 (tree of main's pin dbcaa29) with one parse-only commit: an offer with no form still means
pair. The fixture changes the full sdk forced are reverted to main's; the sdk's conformance events are captured at
78c9ec8 and unchanged. TestModuleCheckPassesTheCatalogue passes, on the captured copy and on mesh-catalog main 96b129e5
cloned beside. Whole unit and integration suites: 1706 tests, 8 skipped for a registry, a container, a refusing bus or a
lab's run.
No code of ours changes; the sdk's conformance events are unchanged and captured at 6e4a301. The fixtures of
identity_bound_test and issued_test say the credential form their rows meant (pair, none), since the sdk no longer reads
a missing form as pair (6b8c694).
NOT GREEN: TestModuleCheckPassesTheCatalogue fails. The sdk refuses every offer that states no credential form, and
main's catalogue states none (27 offers). Recapturing cannot fix it: this pin and main's catalogue cannot both stand.
The sdk's conformance events are unchanged at 6e4a301 and captured so. The whole integration suite passes at this head
against a local store and bus: 1706 tests, 8 skipped for a registry, a container, a refusing bus or a lab's run.
The sdk now refuses an offer that says no credential form (6b8c694, ADR 0570). So:
- the catalogue copy in testdata/beside is captured again at the release branch's catalogue 8ae8e88d, whose offers each
say one, and the sdk's conformance events at 007b6fd, as CAPTURED asks;
- the fixtures of identity_bound_test and issued_test say the form their rows meant: pair for a login of each
consumer's own, none for a keyless provision. No behaviour of the controller changed.
The controller on dbcaa29 parses manifests strictly and refuses `bus` on an artifact and `max-age-days` on a log, both
in the re-land group's catalogue. The SDK's conformance events are captured again at the pinned commit, as
testdata/beside/CAPTURED asks; they are unchanged.
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: the compose ask mvsg85b34d25b4 never ende…
mesh/check being carried to 8f847318963c on 0fbb5ef35991 | from 2dafbe60e939
A plain delete marker on the lease key is also written by a command's own
lease and by a take given back over the floor; counted as a handover, it
would record a crashed holder's epoch released and say nothing. Release now
writes the marker itself, compare-and-set on the key's revision, with the
header Mesh-Lease-Released naming its epoch, and the next holder ends only
that epoch as released. A takeover whose predecessor cannot be read is now
raised as an expiry, not only printed.
The 2026-10-26 replacements showed the lease was given back, but last: the
shutdown took 10 s, the gate had closed at half the key's age, and the release
recorded the epoch released only through the gate, so the next holder found
it expired and S12 said it, urgent, for an hour.
Now the stop signal gives the key back first and records the epoch released
whenever the delete succeeded; a candidate reads the key's delete marker on
the bus to tell a handover from an expiry; and a real takeover (lost or
expired) is raised and cleared once by the controller that took it, instead
of an hour of S12 observation.
Research 072 piece moved: the lease's takeover read from its own key on the
bus and said as an event (03-domains, mesh-controller row; 04-findings 2,
the watchdogs' age reading).
Quick fix weighed: drop the gate check from the release record only; it
leaves the release last in the shutdown and the hour-long alarm for a real
takeover.
mesh/check being carried to 8d58afea6cb8 on 2dafbe60e939 | from b611f7375a47
The server counts a consumer's pending after it acknowledges the publish, so a read at once failed the try of
main on a loaded ace (TestAResetOfTheEventsConsumerLeavesTheMergesAndTheirConsumerAlone, 2026-10-26 22:24 UTC).
Fixed for the kind, not the one test: all six reads of a positive count go through the package's own pendingOf,
whose deadline was 1 s.
Research 'The controller, NATS-native': none; this changes tests only, not a part of the controller.
mesh/check failed: unit: github.com/novox/mesh-controller/cmd/mesh-controller.TestModuleCheckPassesTheCatalogue: the catalogue does not pass its own c…
Moves no piece of research 'The controller, NATS-native': a repin for a seat verb list, no controller code changed.
mesh/check failed: unit: ..test: ✔ a transient empty list for four passes retires nothing; five stable passes retire, data kept (290.408814ms) | 4e9e5…
mesh/check failed: ace: novox/mesh-catalog#505 does not merge into the trunk: files in conflict: .gitignore | 7c8a2e03dd8e on 2ecfbb3a29cb | unit 1245…
mesh/check pass 9ad8715871ea on 2ecfbb3a29cb | carried from e1a55f099b4e | unit 1023 run, 171 kept, 2 skipped | integration 1701 run, 214 kept, 9 skip…
The controller is the only composer, and its pin (10b6e202) predates mesh-sdk #95: without the bump the
build runtimes never get yields {priority idle, cpu-share} or MESH_BUILD_CPU_SHARE (ADR 0553, ADR 0554,
change B). The operator said yes to this one change of the frozen controller on 2026-10-26 15:54.
A composition at both pins over every node's assignment differs only in mesh-build-go, mesh-build-node and
mesh-build-image (the yields and the share word); every other composition is byte-identical. The SDK's
conformance capture moves with the pin, as testdata/beside/CAPTURED asks.
Research 072 (the controller, NATS-native): no piece moved; a pin bump the operator allowed.
The checker hears from DEAD_LETTERS which of its asks a build seat gave up on, and ends the try or asks the check again;
the controller composes today's user list, so its copy of the composer carries the grant too. NATS-native research piece:
none moved; the composition itself moves to mesh-grants (ADR 0547), and this keeps the copy in step with mesh-sdk.
mesh/check being carried to 23226384e529 on 08a261ac1a6d | from 350f8ebdfe5f
The stub answered the trunk for every commit, so since registering on the
fingerprint (666d1b68, #465) mesh-host, registered at an older commit, looked
already at the trunk's fingerprint: the take-in only moved its commit, no build
was recorded, and the fingerprints never named the new node-engine. The real
indexer holds no value at a commit it never indexed.
(cherry picked from commit 779533764b)
mesh/check checking 6927b02d4d58 on fda4859a95b8 | novox/mesh-controller#469 in group fix/1790-grants-create-buckets
The operator's one more grant in the frozen controller's composer (2026-10-26 08:54), beside the
memberships grant of 22:31 (ADR 0547 clause 1 and 5): mesh-grants' principal may publish
$JS.API.STREAM.CREATE.<stream> and $JS.API.STREAM.INFO.<stream> for each bucket
(KV_<module>_<state>), log (LOG_<module>_<log>) and module stream a placed module's manifest
declares, each by its own name, and nothing wider. The writers table lets it create, never update
or delete. Byte-equal with mesh-sdk's buscompose (fix/1790-grants-create-buckets), whose expected
lists this code wrote; the same test file holds both copies. Nothing else changes, and the mesh-sdk
pin stays.
Research 072: moves the grants piece (the grants part split out of the controller) one step, the
creation of a module's declared bus objects; the controller's raise at start stays until it loses
its lease.
Issue: 1790
jschoubben
mesh-tools