Remove the leftover merges consumer behind the false deaf alarm; apply-failing stays raised while a failure stands (issue 774) #484

2026-10-30T17:45:39Z

mesh-tools commented

2026-10-30 17:45:39 +00:00

Each fix names the piece of the research "The controller, NATS-native" it moves.

Merges consumer removed (03-domains: the delivery parts own merges): merges are read from the forge bridge's stream FORGE; a held message is not deafness until it is due (the false core.controller.deaf).
apply-failing stays raised while any failure stands, and is measured from failing-since (milestone 3).
Conditions whose check is gone are cleared (S23, D14) (milestone 3).
Reads judged for their reader through mesh-sdk's ReadsAnnouncedBy; the controller's copy is deleted; each refusal logged once (03-domains, mesh-grants).
Call records keep each call's latest state under bursts; unissued process accounts are not asked of the vault (the random nats-tools value); the composer copy carries the engine guard; modules the mesh provides are left out of fingerprint lines; superseded in the move answer.
Pinned to mesh-sdk a3718126; captured SDK refreshed.

Part of the fix group "sweep 2026-10-30" (issue 774, design 47's release): one branch fix/sweep-2026-10-30 in mesh-sdk, mesh-catalog, mesh-delivery, mesh-controller, mesh-host and mesh-tools. Merge mesh-sdk first; the others pin mesh-sdk a3718126 (v0.1.14-0.20261030174137-a37181261c24).
Recovery mode: the operator merges by hand. Every fix names its quick and thorough form in its commit message; each test fails without its fix.

Reviewed-by: rev-sweep@shanks at 4e2b8f32ce

Each fix names the piece of the research "The controller, NATS-native" it moves. - **Merges consumer removed** (03-domains: the delivery parts own merges): merges are read from the forge bridge's stream FORGE; a held message is not deafness until it is due (the false `core.controller.deaf`). - **apply-failing stays raised** while any failure stands, and is measured from `failing-since` (milestone 3). - **Conditions whose check is gone are cleared** (S23, D14) (milestone 3). - **Reads judged for their reader** through mesh-sdk's `ReadsAnnouncedBy`; the controller's copy is deleted; each refusal logged once (03-domains, mesh-grants). - **Call records** keep each call's latest state under bursts; **unissued process accounts** are not asked of the vault (the random nats-tools value); the composer copy carries the engine guard; modules the mesh provides are left out of fingerprint lines; `superseded` in the move answer. - Pinned to mesh-sdk a3718126; captured SDK refreshed. --- Part of the fix group "sweep 2026-10-30" (issue 774, design 47's release): one branch `fix/sweep-2026-10-30` in mesh-sdk, mesh-catalog, mesh-delivery, mesh-controller, mesh-host and mesh-tools. Merge mesh-sdk first; the others pin mesh-sdk a3718126 (v0.1.14-0.20261030174137-a37181261c24). Recovery mode: the operator merges by hand. Every fix names its quick and thorough form in its commit message; each test fails without its fix. Reviewed-by: rev-sweep@shanks at 4e2b8f32ceea476396eab84f70e24ec1a91f46c0

mesh-tools added 14 commits 2026-10-30 17:45:40 +00:00

Retire the controller's merges consumer and read merges from the forge bridge's stream, so a held merge can never again queue every merge or look like a deaf controller (issue 774) 48d9f22fa5

The controller-merges consumer (no max-deliver, one at a time) held the de-spiegel merge e25e1beb from
10-29 09:39: keepForgeMerge put an announcement with its Paths into delivery-merges (16 KiB values), got
10054, retried "not now" forever, and 167 merges queued behind it. Every five-minute hold outran the deaf
bound, so core.controller.deaf flapped.

Merges are mesh-delivery's: the gate merges on the Approve and the forge bridge says each merge on its
stream FORGE (pull request, head, merge commit), kept a week. The controller's copy was a leftover, read
only at a moved module's registration (ADR 0420 §7, ADR 0432). That read now finds the merge on FORGE;
the consumer, its follow, its handler and its hold delays are removed, and every assertion removes the
retired consumer from the bus.

Kind fix for the watchdog: the event loop records when a message it held back is due again, and deafness
counts from then, so no hold longer than the bound (on any consumer) reads as deafness.

Quick fix: strip Paths from the kept announcement and add max-deliver with a dead letter. Thorough fix:
remove the duplicate duty and make the deaf watchdog hold-aware. Chose thorough: the controller is being
deprecated and the merges already have an owner; the hold-vs-deaf confusion is a kind, not one consumer.

Research 072 piece: 03-domains (Control shrinks to placement and composition; the delivery parts own
what a merge is) and milestone 3 (health signals stop sharing the controller's loop assumptions).

Keep apply-failing raised while any failure stands, so a failure said again in new words never clears it (issue 774) 6ffcea4f1d

machine.novox.apply-failing cleared at 16:56 while the vault refused every secret there: the provisioner
client says a failure again whenever its words change, with a new time, and S21 measured each module's
failure from that newest time, so every new wording hid the failure for one apply interval (raised again
17:01). The same holds for the node-engine's own failed apply, whose since restarts with new words.

The bound now gates raising only: once the condition is open (read from the keeper's open conditions on
the bus), it stays while any module's apply fails on that machine, and clears when none does.

Quick fix: this hysteresis on the open condition in the controller. Thorough fix: the provisioner client
carries the start of a failure run (failing-since) in its report, so every reader measures from the true
start; that is mesh-catalog's mesh-provisioner-client, routed to the lead. Chose the quick fix here,
because the check moves to mesh-watchdogs (ADR 0564) and the condition's own state already proves the run.

Research 072 piece: milestone 3 (health leaves the controller): the judgement reads bus state, the open
condition, rather than a report's time.

Clear a condition whose check no longer exists, so a removed row or probe leaves no alarm open for good (issue 774) d63eb83ed3

About 35 conditions from 10-20 to 10-27 still showed as current. Read on the bus at 18:06: S23's three
failed plans (last observed 10-24) and D14's four stalled deliveries (last observed 10-24) come from a
signals row and a probe this controller no longer has; nothing reconciles their source, so nothing clears
them. The other old ones are observed every tick and are not stale by the controller: S9's dead letters
are still held, S15's hand acts are still kept, and DM's 26 stuck keys are the watchman's (it never writes
a key whose watched key aged out as moved) — routed to mesh-delivery.

Each watchdogs tick now clears every open condition whose source has a row's or a probe's form (S12, D4,
DW, D-root) and is in neither the signals table nor the probe registry, saying the check was removed.
Sources of other forms are left to their own word.

Quick fix: clear the seven by hand. Thorough fix: the controller judges its own retired checks on every
tick, so the next removed row or probe clears what it raised. Chose thorough: rows and probes are being
moved out of the controller one by one (ADR 0564), each removal would leave this again.

Research 072 piece: milestone 3 (health leaves the controller): the conditions keeper drops what no check
of its own observes, so the move to mesh-conditions starts from a bucket of live conditions.

Judge each read for its reader with mesh-sdk's ReadsAnnouncedBy, so the parts named for the lease are granted it, and say a refusal once (issue 774) d543ff7126

2536 reads of mesh-controller.lease were refused on 2026-10-30 (mesh-vault, mesh-grants, the assigner,
artifact-store-tools, the watchman, mesh-config). The controller composed the bus's users with its own
copy of a reader-less judgement (internal/broker ReadsAnnounced), which knows no named readers: the lease
is announced readable by nobody and read by the parts named for it (ADR 0547 clause 9, ADR 0561 §3). The
copy is deleted; BusRecords judges each read for its module with buscompose.ReadsAnnouncedBy. Each
refusal was also printed on every composition, about 420 times a read; it is now said once per process.

mesh-config's read of the lease stays refused: it reads the lease for its take-over (hand-back ends at
the lease write, ADR 0561 §3) and is not among the named readers. That is mesh-sdk's NamedReaders,
routed to the lead.

Quick fix: mark the lease readable in the controller's announcement (tried and reverted: it would open
the lease to every module and contradict ADR 0561 §3). Thorough fix: one judgement, the SDK's, for every
composer. Chose thorough.

Research 072 piece: 03-domains, mesh-grants (bus membership and grants leave the controller): the
controller composes reads with the same SDK function the grants part uses.

Put mesh-delivery's merge words through mesh-sdk's kvlimit, so a word too large for its bucket is refused once and never retried (issue 774) e6ecc2d15e

The controller's other writer of the 16 KiB bucket of merges, the keeper of mesh-delivery's merge words,
called every put failure "not now". A word too large would be retried as the forge's merges were. It now
puts through kvlimit.Put: a value that does not fit is refused before it is put and returned as an answer
(the events consumer takes it, max-deliver 5 then a dead letter), and only another failure is retried.
It is the one write of the controller whose failure was retried as "not now" (searched: every Put,
Create and Update wrapped in ErrTryAgain).

Needs the group's mesh-sdk head (go/kvlimit, 7b02f63 or later).

Quick fix: none smaller. Thorough fix: every bucket write through kvlimit; this is the only one whose
refusal was retried, the others fail once and say so. No new test: kvlimit is tested in mesh-sdk, and an
integration test here would push the repository further over its budget (rules-are-weighed).

Research 072 piece: 03-domains (Control shrinks): the merge words stay the controller's only until the
registration of a moved module leaves it.

Leave modules the mesh provides itself out of the lines that say a module lacks a fingerprint (issue 774) 07cecd287c

mesh-wireguard is provided by the mesh (sdkdelivery.ProvidedByMesh): no repository builds it and no
fingerprint will ever name it, yet every composition said it composes "with the node's own: no
fingerprint of the new form names them", and the switch's list counted it "not known". Both now leave
such a module out; its resources still go with the node's own composition, as before.

Quick and thorough are one here: the SDK names the modules the mesh provides, and every line that judges
fingerprints filters by it.

Research 072 piece: §6 milestone 2 (the composer reacts; its words say only what a change can fix).

Ask the vault for a bus process's account only once it is issued, so the vault never makes a random value under an account's name (issue 774) c421629f6e

At 16:36:51 the vault handed own/novox/nats/broker-nats-tools to the controller as a plain random own
secret, and owes the bus-account hashes of it and of the controller's broker-address and
broker-management for good ("holds no account the controller hands (not JSON)"). The controller decided
that: its send asks the vault (`get`) for every own secret of a core module, a bus process's account
among them, before the account is issued; the vault, holding none, makes one.

The send now leaves out of that ask a process's account nobody issued (the same rule as its warning
line); the machine keeps the controller's stand-in until the account is issued and handed to the vault.
The vault side (not minting an account-shaped name, and replacing the random value already held) is
mesh-catalog's mesh-vault, with build-catalog.

Quick fix: this filter on the ask. Thorough fix: process accounts minted by the vault itself (ADR 0113 §6,
research 072 milestone 6), so no part asks for one before it exists. Chose quick: milestone 6 is not
started, and the filter removes the cause on the controller's side now.

Research 072 piece: milestone 6 (identity and bus accounts to the vault): until then the controller asks
the vault only for what exists.

Queue one state per call for the bus, bounded far above a burst, so a burst of read calls no longer drops call records (issue 774) 519434241a

At 17:44:53 the controller said 30 calls (tools, nodes, modules) were "kept in memory only: the bus is
not taking calls' records". No write had failed (no "could not be kept" line): a burst of about 200 read
calls in one second queued a begin and a finish each into a channel of 256 while the one writer put
them one by one, and the channel overflowed.

The queue now holds the latest state of each waiting call (a finish replaces its own waiting begin, as
only the last state is kept anyway), in first-queued order, bounded at 8192 calls; the writer takes the
whole queue at each wake. A burst now costs one write per call and drops nothing.

Quick fix: a larger channel. Thorough fix: coalesce by call, so the writes a burst asks are halved and
the bound is spent on calls, not states. Chose thorough; both are small.

Research 072 piece: §2's "read calls said" row: the calls bucket is the controller's own state on the
bus, kept from events (each call's change), never re-read.

Compose no placed user from the node-engine's manifest in the controller's bus composer too, as mesh-sdk #136 does (issue 774) 8c3d372322

mesh-sdk #136 (42988a7) keeps the node-engine's manifest out of every placed user in buscompose: where a
placement names it, no user, no process and nothing the tool runner carries; the node's account alone
writes the engine's state, and a second writer has the writers table refuse the whole list (met twice on
2026-10-30: mesh-host_acts, then mesh-host_health). The controller still composes the bus's users with
its own copy (internal/broker Users), which lacked the guard. The same two predicates, Composed and
CarriedByRunner, now guard it.

Read on the bus at 18:30: the bus follows the grants part's list (mesh-grants_user-list.sealed), so the
controller's copy is not what the bus loads; it still composes the bus module's files and the controller's
own grants, where a refusal would stop that composition.

Quick fix: the same guard here. Thorough fix: delete the controller's copy of the bus composer and use
buscompose (about 18,000 lines of internal/broker with its tests, its genesis list and its golden file).
Chose quick: the controller is being deprecated and the grants part already holds the list; the copy goes
with the controller's composition of the bus module, not in a sweep.

Research 072 piece: 03-domains, mesh-grants (membership and grants leave the controller): until the copy is
removed, it composes as the SDK does.

Pin the group's mesh-sdk ef1ec1c9 and capture its conformance files there, so the sweep builds on kvlimit and the named readers (issue 774) 77c54445dc

The sweep's controller changes use mesh-sdk's go/kvlimit and the ReadsAnnouncedBy that grants named
readers. At ef1ec1c9 mesh-config is a named reader of the controller's lease (its take-over reads it,
ADR 0561 §3), so the read test now expects it granted and a module not named refused. The captured SDK
(testdata/beside, conformance/events) is taken again at the pinned commit, as CAPTURED says; its file
is unchanged.

Judge apply-failing by the client's failing-since, so a failure said in new words is measured from when it began (issue 774) cf4ca55596

mesh-provisioner-client's report now carries failing-since (mesh-catalog 3aae6c08), kept while a module
stays failed however its words change. S21 measures a failed placing from it, falling back to the
report's time from an older client; the hysteresis of 6ffcea4f (an open condition stays while any
failure stands) is kept. This is the thorough fix that commit named.

Research 072 piece: milestone 3 (health leaves the controller): the judgement reads the owner's state.

Say rightly which own secrets the core ask leaves out: broker-address and broker-management are no bus accounts (issue 774) 93d167f0e8

The comment of c421629f named the controller's broker-address and broker-management as process accounts.
They are not: no process of the controller declares a bus; broker-address is the bus's address, read by
broker.FromEnvironment, and broker-management the retired AMQP management login, read by nothing. The
filter never touched them; only the words were wrong.

Pin the group's final mesh-sdk 206f794e with its captured conformance files, and read the engine's superseded word on a secret move (issue 774) dd366f8a3f

The captured SDK (testdata/beside, conformance/events) is taken again at the pinned commit, as CAPTURED
says. The controller's mirror of MoveSecretAnswer gains `superseded` (mesh-host b9c24a7): the vault holds
another value, the live one, and the machine's older value is not moved and not asked again. The field
test names it. A superseded row is still counted with the moved ones: counting it apart needs a field and
its words in nodeMove, more than the one line asked.

Pin mesh-sdk a3718126, whose bus retry never asks again a call its callee answered (issue 774)

mesh/delivery closed: the pull request closed unmerged

Details

mesh/delivery-group group fix/sweep-2026-10-30 rejected: composed together: the checker's verdict at the members' heads (9a9df8453781): g14

Details

mesh/check failed: g14: novox/mesh-catalog#565: modules/mesh-vault/mesh-vault: a compiled program added in 2066021fca5f and removed later: it stays in…

4e2b8f32ce

mesh-tools commented

2026-10-30 17:45:54 +00:00

Delivery novox/mesh-controller@4e2b8f32ceea — closed since 2026-10-30T18:06:14Z

Group fix/sweep-2026-10-30: novox/mesh-catalog@2455445dab, novox/mesh-delivery@e0b56c83f9, novox/mesh-host@b9c24a7b42, novox/mesh-sdk@a37181261c, novox/mesh-tools@316a5a2fc3

Transitions

2026-10-30 17:45 (new) → proposed (announced): novox/mesh-controller#484's head announced
2026-10-30 17:45 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-10-30 18:06 ready → closed (closed): the pull request closed unmerged

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-controller@4e2b8f32ceea` — **closed** since 2026-10-30T18:06:14Z **Group** `fix/sweep-2026-10-30`: novox/mesh-catalog@2455445dabee, novox/mesh-delivery@e0b56c83f94c, novox/mesh-host@b9c24a7b42cb, novox/mesh-sdk@a37181261c24, novox/mesh-tools@316a5a2fc317 **Transitions** - 2026-10-30 17:45 (new) → proposed (announced): novox/mesh-controller#484's head announced - 2026-10-30 17:45 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-10-30 18:06 ready → closed (closed): the pull request closed unmerged The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools closed this pull request

2026-10-30 18:06:13 +00:00

mesh-tools commented

2026-10-30 18:06:16 +00:00

mesh-delivery: the operator's Approve of this pull request's merge into main is ended: its head 4e2b8f32ce is no longer the pull request's head, or the pull request closed. Nothing is merged on it. The operator is asked again once the checks pass at the new head.

mesh-delivery: the operator's Approve of this pull request's merge into main is ended: its head 4e2b8f32ceea is no longer the pull request's head, or the pull request closed. Nothing is merged on it. The operator is asked again once the checks pass at the new head.

This repo is archived. You cannot comment on pull requests.

1 Participants

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-controller#484