Take new builds in from a durable queue on the bus, register on the fingerprint, and issue a node's bus accounts when its assignment stands (issue 1945, ADR 0543, ADR 0544) #465

2026-10-25T17:35:36Z

mesh-tools commented

2026-10-25 17:35:36 +00:00

Research 072 pieces moved: milestone 2's take-in on a bounded pull consumer (ADR 0544), and the grants piece (memberships and credentials heard from placement, ADR 0543).

Why

On 2026-10-25 every new build waited 30-60 minutes for its bus account (assigner rule 6): the take-in was one 16-slot queue, silent while waiting, three composer passes per module, an alphabetical start walk that every restart threw away, and a catalogue merge that changed only the commit re-took-in ~200 modules. nextcloud #478 waited 70 minutes. Separately, assigning an already-built module to a new node (ace, 17:19) never issued that node's bus account, so it crash-looped and the assigner held the version mesh-wide.

What

ADR 0544: two durable pull consumers mesh-controller-take-in (indexer modules bucket, deliver last per subject; builder builds bucket, deliver new); batches of up to 16 with the catalogue read once and two composer passes; a key with nothing new composes nothing; a module whose fingerprint is unchanged when only the commit moved is recorded with one UPDATE (RecordBuiltFrom), not composed; refusals wait 30 s doubling to 10 min; MaxAckPending 1000 so no refused key holds others; synchronous acks; on stop every held key is refused with no delay so the next controller resumes at once; backlog in status and the log; take-in-stalled when doable work does not settle for 5 min (cleared at the first healthy look, also after a restart); take-in-not-consumed per bucket. The alphabetical start walk and the follower slots are gone.
ADR 0543: the assignments bucket's watch issues missing accounts once (mint-if-absent, one winner seals), composes the bus node and writes the grants for a newly standing assignment; the start look runs once per process.

Delivery note

First start of this release: the bus refuses the new acks until novox's composition carries the new grant (composeAtStart composes it early); keys come back after the ack wait. Expect up to ~10 min before the take-in backlog drains, then seconds per new build.

Tests

go test -race ./... (untagged) and go vet -tags integration ./... pass. Unit tests 1072 -> 1063 (budget rule: net fewer). Recovery mode: no try; the operator merges by hand.

Issue: issue 1945 (notes 34-36), issue 1277.

Reviewed-by: reviewer-grant@shanks at 3e39ccf958
Reviewed-by: reviewer-takein@shanks at 666d1b682e

Research 072 pieces moved: milestone 2's take-in on a bounded pull consumer (ADR 0544), and the grants piece (memberships and credentials heard from placement, ADR 0543). ## Why On 2026-10-25 every new build waited 30-60 minutes for its bus account (assigner rule 6): the take-in was one 16-slot queue, silent while waiting, three composer passes per module, an alphabetical start walk that every restart threw away, and a catalogue merge that changed only the commit re-took-in ~200 modules. nextcloud #478 waited 70 minutes. Separately, assigning an already-built module to a new node (ace, 17:19) never issued that node's bus account, so it crash-looped and the assigner held the version mesh-wide. ## What - ADR 0544: two durable pull consumers `mesh-controller-take-in` (indexer modules bucket, deliver last per subject; builder builds bucket, deliver new); batches of up to 16 with the catalogue read once and two composer passes; a key with nothing new composes nothing; a module whose fingerprint is unchanged when only the commit moved is recorded with one UPDATE (RecordBuiltFrom), not composed; refusals wait 30 s doubling to 10 min; MaxAckPending 1000 so no refused key holds others; synchronous acks; on stop every held key is refused with no delay so the next controller resumes at once; backlog in `status` and the log; `take-in-stalled` when doable work does not settle for 5 min (cleared at the first healthy look, also after a restart); `take-in-not-consumed` per bucket. The alphabetical start walk and the follower slots are gone. - ADR 0543: the assignments bucket's watch issues missing accounts once (mint-if-absent, one winner seals), composes the bus node and writes the grants for a newly standing assignment; the start look runs once per process. ## Delivery note First start of this release: the bus refuses the new acks until novox's composition carries the new grant (composeAtStart composes it early); keys come back after the ack wait. Expect up to ~10 min before the take-in backlog drains, then seconds per new build. ## Tests `go test -race ./...` (untagged) and `go vet -tags integration ./...` pass. Unit tests 1072 -> 1063 (budget rule: net fewer). Recovery mode: no try; the operator merges by hand. Issue: issue 1945 (notes 34-36), issue 1277. Reviewed-by: reviewer-grant@shanks at 3e39ccf9580146088f335fe8c4a46de2c658914e Reviewed-by: reviewer-takein@shanks at 666d1b682e7f2b2ce7ffd25b2674e416b1459499

mesh-tools added 9 commits 2026-10-25 17:35:36 +00:00

Issue a node's bus accounts and grant when its assignment comes to stand (issue 1945, ADR 0543) ca5f74eadb

ADR 0539 clause 7 issues a module's credential and grant only when its build is done.
A module built long before and assigned to one more node never has a build done again:
on 2026-10-25 at 17:19 mesh-build-node and mesh-build-go were assigned to ace and composed
for it, the bus's user list never named ace's accounts, and both runtimes restarted every
5 s (one with no credential, one refused by the bus). The assign act composes the assigned
node alone, never the node holding the bus, and writes no grant.

So a write of the assignments bucket is the event: when a key comes to stand (new, or
ended or pending before), the lease holder issues the module's accounts on that node that
are not minted (never re-issued), composes the node holding the bus and writes the grants,
through the composer the done keys' take-ins use. A rewrite of a standing assignment does
nothing; the start's look settles every standing one, one after another. A step of
research 072's grants (memberships and credentials heard from placement).

Mint an assignment's account once, whoever finds it missing first (issue 1945, ADR 0543, review) 9ff47f6aaa

The assign act and the assignments' watch both checked whether an account was
minted and then minted it, and minting overwrites the hash: interleaved, the
bus kept one caller's hash and the node was sealed the other's password, the
crash loop this branch fixes. Both now mint through one statement that inserts
the row only when none stands; the caller whose insert wins seals, the other
seals nothing. issueMissingProcessAccounts mints the same way.

A watch opened again after one ended handed the standing assignments as the
start's look, so one that came to stand in the gap got no bus composition or
grant when its account already stood: only the process's first watch is the
start's look now. The start's look reads the catalogue once and ends with one
composition of the bus's node and the grants; a controller not holding the
lease is told so and tries again rather than dropping the key.

Tests folded to stay within the budget: the assignment tests are three table
tests, the process account and mayIssue tests folded into one of them, and
the concurrent mint is a case of the inventory's rotation test.
Research 072: the grants piece (memberships and credentials heard from placement).

Take done keys in from a durable pull consumer, a batch at a time (issue 1945, ADR 0544) bfb271a91b

The take-in watched the builder's and the indexer's buckets, started a follower
per module that retried on a 30 s timer, and at start took every catalogued
module in alphabetically: on 2026-10-25 a restart took in about 2 modules a
minute, about 190 followers queued behind 16 slots, and every take-in composed
two or three times even with nothing new, so a new build waited 30 minutes and
more for its credential.

Now two durable consumers named mesh-controller-take-in read the indexer's
modules bucket (every module's last trunk value when the consumer is made, then
each write) and the builder's builds bucket (each write), at most 16 keys
unacknowledged each. The bus keeps what is not taken in: the start's look is the
consumers' own catch-up, and a restart costs only what was left unacknowledged.
A batch reads the catalogue once, takes each module in once, and asks the
composer twice: once for its modules' nodes, once for the bus's node with the
grants. A module registered at its trunk whose compositions and grant exist, or
that this process took in at the same trunk, composes nothing; a key delivered
again is taken in whole. A wait for the builder's or indexer's next write is
acknowledged, a wait on a read or a failure refused with a 30 s delay. The
backlog is in status, and a backlog standing still for 5 minutes raises
take-in-stalled. The controller's account is granted the two consumers' acks.

The follower tests go with the followers: twelve tests become three.
Research 072: the take-ins on a bounded pull consumer (milestone 4, done keys
alone) and the start's look composing only what changed (milestone 2).

Bound the take-in's two consumers to 16 keys in flight together, 8 each (ADR 0544 §1, its review) 94bb74966f

Two consumers of 16 each allowed 32 take-ins on the store at once, twice
issue 1945's bound. A batch takes up to 16. Research 072: the take-ins on a
bounded pull consumer.

Let only a watch that read every standing assignment be the start's look (issue 1945, ADR 0543, review) 3e39ccf958

The flag that made the first watch the start's look was cleared after every
watch, even one that failed to open or ended before its Caught mark: the next
watch then handed every standing assignment as fresh, about 300 settles at once
with a catalogue read each, or dropped the keys the cut watch had handed. The
flag is now cleared by the start's look itself, and a watch cut before it opens
the next with nothing heard. Two rows of the existing table test cover it.
Research 072: the grants piece (memberships and credentials heard from placement).

Merge branch 'fix/1945-credential-on-assignment' into fix/1945-take-in-consumer e7673e3bd3

Keep refused keys from holding the take-in's slots, and judge its stall by acknowledgements (issue 1945, ADR 0544, review) 850f4b4d26

A refused key stays unacknowledged until its delay ends, so with 8 a consumer
a handful refused for long (a manifest word this controller does not read
yet, a node that keeps failing) stopped that consumer for good, the build that
reads the word among what it held back. Each consumer now leaves up to 1000
unacknowledged; the store's bound is the one loop's batch of 16. A refusal
waits 30 s, doubling each time up to 10 minutes.

The stall watch judged movement by deliveries, which rise with every
redelivery: it now counts the keys this process acknowledged. A consumer that
cannot be made raises take-in-stalled instead of a log line. On stop the
gathered batch and a key in hand are refused with no delay, so the next
controller has them at once. A trunk value with no fingerprint of the new form
is said once, as every wait for a write.
Research 072: the take-ins on a bounded pull consumer.

Judge the take-in's stall by keys settled with work to do, and clear its alarms after a restart (issue 1945, ADR 0544, review 2) bf20d9d8cf

A refused key stays unacknowledged during its delay, so one waiting key on a
quiet mesh raised take-in-stalled, cleared at the next ack and raised again.
The watch now counts the keys the loop settled (acknowledged or refused, as
the bus took them) and calls it a stall only with work to do, keys not
delivered yet or a batch handed and not settled, and nothing settled for 5
minutes: a hung loop stays loud. Its first look always says what it found, so
a stall raised before a restart is cleared once this controller moves. A
consumer not made raises take-in-not-consumed under its own id per bucket,
cleared when that consumer is made. Acks are synchronous, so only those the
bus took count; on stop the iterator is drained and what it held, and a batch
cut by the stop, are refused with no delay.
Research 072: the take-ins on a bounded pull consumer.

Register on the fingerprint, not the commit; count the take-in's attempts as settled (issue 1945, ADR 0544, review 3)

mesh/delivery held for a person: merged without a passing check: only a person decides what follows

Details

mesh/check checking 666d1b682e7f on 045f25bd1353

666d1b682e

registeredAt compared commits, so a catalogue merge that moved only the commit
(2df05030 at 18:02, a85d9037 before it) took every catalogue module in again
and composed it: about 200 take-ins, some 25 minutes, on every mesh-catalog
merge, which held nextcloud for an hour. A module whose trunk fingerprint is
the one it is registered at now has the new commit recorded (built_from and
source_head, as a registration moves them) and composes nothing unless its
composition or grant is missing. Readers of the registered commit: the
fingerprints read the commit's build or, with none, the indexer's value at it,
which names the same fingerprint; status, modules, the board and the facts
read it as the source's commit, which it is; the upgrade path counts it as
built from that commit, which it is.

The stall watch counted only settles the bus accepted, so each refused
acknowledgement at a first start left work held for good: it now counts the
loop's every attempt. The acknowledgement and refusals at the stop are sent
past the stop, and the connection closes only after the readers and the loop
have ended.
Research 072: the take-ins on a bounded pull consumer.

mesh-tools commented

2026-10-25 17:35:40 +00:00

Delivery novox/mesh-controller@666d1b682e7f — held since 2026-10-25T18:06:49Z

Held for a person: merged without a passing check: only a person decides what follows — the merge reached the trunk without a passing check.

Source — source.main moved to novox/mesh-controller@045f25bd13. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Transitions

2026-10-25 17:35 (new) → proposed (announced): novox/mesh-controller#465's head announced
2026-10-25 17:42 proposed → rejected (refused): a verdict names this commit and the trunk's head as its trunk commit, and the gate failed or could not run, or the repository's own check failed
2026-10-25 18:06 rejected → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides what follows

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-controller@666d1b682e7f` — **held** since 2026-10-25T18:06:49Z Held for a person: merged without a passing check: only a person decides what follows — the merge reached the trunk without a passing check. **Source** — source.main moved to novox/mesh-controller@045f25bd1353. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Transitions** - 2026-10-25 17:35 (new) → proposed (announced): novox/mesh-controller#465's head announced - 2026-10-25 17:42 proposed → rejected (refused): a verdict names this commit and the trunk's head as its trunk commit, and the gate failed or could not run, or the repository's own check failed - 2026-10-25 18:06 rejected → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides what follows The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

jschoubben merged commit 045f25bd13 into main

2026-10-25 18:06:47 +00:00

jschoubben deleted branch fix/1945-take-in-consumer

2026-10-25 18:06:50 +00:00

mesh-tools referenced this pull request

2026-10-25 18:37:16 +00:00

Kept compositions follow config changes from a durable consumer, so going back to an earlier build keeps today's config (issue 1945, ADR 0545) #466

mesh-tools referenced this pull request

2026-10-25 22:31:07 +00:00

The bus composer grants mesh-grants publishing memberships, nothing more (ADR 0547, the operator's one grant of 2026-10-25 22:31) #468

Sign in to join this conversation.