mesh/delivery-group group fix/1945-credential-when-row-without-seal held: a member is held: novox/mesh-controller@fbf14b68dbab (merged without a passing check:…
Migration 0108 marks every standing account sealed, ace's stale mesh-build-go
rows from 17:19 among them, and the forgetting ran only at a later unassign:
the next assignment to ace would have issued nothing again.
- The start's look first forgets, from each node's own resolution (assigned
and required), the accounts of every module that no longer resolves on its
node, and says each; only from a reading that holds assignments, and a node
whose resolution is unknown is left alone.
- Unassign forgets a module's accounts only once it no longer resolves on the
node (a module another requires keeps running), and never the node's own
provisioner account; the forgetting moved from the inventory to the act.
- Forgetting holds each account's lock across its delete, so an issue in flight
never writes a row back after it.
- A module whose accounts are not all sealed is granted its sealed accounts
alone, and its grant is dropped when none is: an older grant naming ace no
longer lets the assigner want it.
- An issue's locked section has a deadline of two minutes, since it holds a
connection of the store's pool across the vault and the bus.
Tests: cases in the existing tables.
Research 072: the grants piece (memberships and credentials heard from placement).
Met a second time live on 2026-10-25 at 20:04 (note 35): mesh-build-go was
assigned to ace again, and its account rows, minted at 17:19, had outlived the
unassignment. The issue minted only when no row stood, so it minted and sealed
nothing; the node had no credential and the bus no user, and the module
crash-looped 20 times. The watch still said "the grants written".
- A bus account now says when its credential was sealed to its node (migration
0108; every account standing before is taken as sealed). An issue under the
assign act, the assignments' watch or a registration mints only an account
that is not issued, a row with no seal counting as none, under the store's
lock on the account across its look, mint and seal, and marks it sealed once
sealed. A seal that fails leaves it unsealed for the next issue.
- Unassigning a module forgets its bus accounts and their credentials on that
node, so an assignment again issues them afresh.
- Only sealed accounts count as minted for a composition and are named in a
grant, so the assigner, which now wants a node only once the grant names its
account (mesh-delivery, same branch), holds the node until it is.
- The assignment's log line says which accounts were issued and sealed, or that
they stood issued and sealed.
The process-only issue drops the atomic insert of the review of 9ff47f6a for
the lock, which also covers the seal. Tests: cases in the existing tables.
Research 072: the grants piece (memberships and credentials heard from placement).
registeredAt compared commits, so a catalogue merge that moved only the commit
(2df05030 at 18:02, a85d9037 before it) took every catalogue module in again
and composed it: about 200 take-ins, some 25 minutes, on every mesh-catalog
merge, which held nextcloud for an hour. A module whose trunk fingerprint is
the one it is registered at now has the new commit recorded (built_from and
source_head, as a registration moves them) and composes nothing unless its
composition or grant is missing. Readers of the registered commit: the
fingerprints read the commit's build or, with none, the indexer's value at it,
which names the same fingerprint; status, modules, the board and the facts
read it as the source's commit, which it is; the upgrade path counts it as
built from that commit, which it is.
The stall watch counted only settles the bus accepted, so each refused
acknowledgement at a first start left work held for good: it now counts the
loop's every attempt. The acknowledgement and refusals at the stop are sent
past the stop, and the connection closes only after the readers and the loop
have ended.
Research 072: the take-ins on a bounded pull consumer.
The start composed every kept composition of every module on every node, each a
whole node composed in memory under the node's lock: 10 to 50 minutes at a
start, holding take-ins back. Now the writes of mesh-config_resolved are read
from the durable consumer mesh-controller-config-follow, acknowledged once the
change is composed, so a restart composes again only what was written since
the last change handled, and a first start nothing it did not hear. The node's
lock is taken for each write alone. A consumer that cannot be made or read
raises config-not-followed and is tried again a minute later, cleared once it
reads. The controller's account is granted that consumer's acknowledgements
where mesh-config serves, as its verb is.
Research 072: milestone 2, the composer reacting to the change of an input.
The take-in watched the builder's and the indexer's buckets, started a follower
per module that retried on a 30 s timer, and at start took every catalogued
module in alphabetically: on 2026-10-25 a restart took in about 2 modules a
minute, about 190 followers queued behind 16 slots, and every take-in composed
two or three times even with nothing new, so a new build waited 30 minutes and
more for its credential.
Now two durable consumers named mesh-controller-take-in read the indexer's
modules bucket (every module's last trunk value when the consumer is made, then
each write) and the builder's builds bucket (each write), at most 16 keys
unacknowledged each. The bus keeps what is not taken in: the start's look is the
consumers' own catch-up, and a restart costs only what was left unacknowledged.
A batch reads the catalogue once, takes each module in once, and asks the
composer twice: once for its modules' nodes, once for the bus's node with the
grants. A module registered at its trunk whose compositions and grant exist, or
that this process took in at the same trunk, composes nothing; a key delivered
again is taken in whole. A wait for the builder's or indexer's next write is
acknowledged, a wait on a read or a failure refused with a 30 s delay. The
backlog is in status, and a backlog standing still for 5 minutes raises
take-in-stalled. The controller's account is granted the two consumers' acks.
The follower tests go with the followers: twelve tests become three.
Research 072: the take-ins on a bounded pull consumer (milestone 4, done keys
alone) and the start's look composing only what changed (milestone 2).
The assign act and the assignments' watch both checked whether an account was
minted and then minted it, and minting overwrites the hash: interleaved, the
bus kept one caller's hash and the node was sealed the other's password, the
crash loop this branch fixes. Both now mint through one statement that inserts
the row only when none stands; the caller whose insert wins seals, the other
seals nothing. issueMissingProcessAccounts mints the same way.
A watch opened again after one ended handed the standing assignments as the
start's look, so one that came to stand in the gap got no bus composition or
grant when its account already stood: only the process's first watch is the
start's look now. The start's look reads the catalogue once and ends with one
composition of the bus's node and the grants; a controller not holding the
lease is told so and tries again rather than dropping the key.
Tests folded to stay within the budget: the assignment tests are three table
tests, the process account and mayIssue tests folded into one of them, and
the concurrent mint is a case of the inventory's rotation test.
Research 072: the grants piece (memberships and credentials heard from placement).
ADR 0539 clause 7 issues a module's credential and grant only when its build is done.
A module built long before and assigned to one more node never has a build done again:
on 2026-10-25 at 17:19 mesh-build-node and mesh-build-go were assigned to ace and composed
for it, the bus's user list never named ace's accounts, and both runtimes restarted every
5 s (one with no credential, one refused by the bus). The assign act composes the assigned
node alone, never the node holding the bus, and writes no grant.
So a write of the assignments bucket is the event: when a key comes to stand (new, or
ended or pending before), the lease holder issues the module's accounts on that node that
are not minted (never re-issued), composes the node holding the bus and writes the grants,
through the composer the done keys' take-ins use. A rewrite of a standing assignment does
nothing; the start's look settles every standing one, one after another. A step of
research 072's grants (memberships and credentials heard from placement).
At 16:07:36 the builder rewrote about 120 done keys at once. Each woke a
take-in of its own with no limit, and each take-in read the rest of the
mesh through restOfTheMeshOf, which opened a licences and an identity
pool per call. novox's store ran out of connections at 16:08 and refused
everyone, the controller's retries keeping it full.
- The serving controller lends its stores by inventory (shareStores);
restOfTheMeshOf reads on them instead of opening pools per call, and a
verb run as its own process still opens its own. stores opens each
store once under a lock, since readers now share it. serve uses the
shared identity store instead of a second pool.
- The watch's take-ins and the start's look share one cap of 16 take-ins
at once (takeInSlots); a take-in waiting for its next try holds none.
- store.Ready backs off on a full store (SQLSTATE 53300), doubling its
wait up to 5 s, and says once that the store is full.
Issues 1945 and 1277.
A node-engine newer than the registered one, or a node asleep on an older
have, made the done test false after the start deleted NODES; the bus's
health check, probes D6 and D7 and healer H3 then expected the stream and
every node's consumer again, raised them urgent and made them anew. Release
6 sends no declaration, so a stream made again helps no node: a gone stream
is retired. The advisory watchdog also reads a consumer whose stream is gone
as gone instead of failing.
mesh/delivery-group group fix/r6-node-composition held: a member is held: novox/mesh-sdk@10b6e202320f (merged without a passing check: only a person decides wh…
The silent mid-tier plan test main changed stays deleted with the plans (ADR 0539); the SDK is pinned to the feat/r6
head that merged mesh-sdk's main, 748c564c, whose hand path tool is now mesh-hand-compose.
Every grant #456 gives was already feat/r6's, by the SDK's names: each conflict takes feat/r6's side, with main's
test of the user list before the switch kept beside feat/r6's, and the duplicate assertion of the digests' bucket
dropped.
Assignments and builds live on the bus (ADR 0408), so a fixture written before the chain was attached was not what
the serving controller read; and push, which the row test used, went with the plans.
With no send left nothing else writes a missed grant, so the take-in of a done key returns it and tries again every
half minute under an urgent condition until it is written. plan-moved and sent go with the plans. The controller's
first-start deletions are said on a granted subject, a node hears only its backup holder, and the operator's account
writes the hand path, an object replaced included (ADR 0539 clauses 2, 7, 10 and 13).
The object store reads an object's metadata and purges its old chunks when it replaces one, so the hand path is
granted the direct and message gets and the purge of each node's object stream; a test on a bus running the composed
user list walks the hand path end to end.
The new controller says its first start's deletions as an event, a node-engine hears only its backup holder, and the
way back right after the switch, a want and a composition written at the terminal, runs on the user list today's
controller composes (ADR 0539 clauses 2, 10 and 13). The operator is the administrator person, the one second
writer ADR 0539 names.
The switch to release 6 is today's controller's last plan, and the bus runs the user list it composes until the new
controller writes one: so the new controller's writes of each node's compositions and their digests, each new
node-engine's reads of its own compositions and wants and its backup holder's verbs and word, and the assigner's read
of the digests must be granted by this controller already (ADR 0539 clauses 2, 6, 9 and 16; the review's A1).
A node away during the switch finds its last declaration on the stream; once every node's have names the
node-engine registered, no node reads one, so the stream and the nodes' consumers on it go, said once, and a raise
leaves them out (ADR 0539 clause 11).
With no plan left to name a build or a send to keep the plans, a want names what a collection must keep, and the
composition is where the provisioner's plans are composed for the assignments mirror (ADR 0539 clauses 1 and 5);
the assignments no longer name a node's build, so the controller replaced is the newest earlier build of it.
With no plan left to take a done key in, a write of the builder's keys registers the new fingerprint, composes the
module's nodes and the bus's user list, and writes its grant, so the assigner's rules can name it (ADR 0539 clauses
5 and 7). The new controller composes every node at its start, and the controller it replaced at its own
fingerprint, the operator's way back (clause 10); the merge window's, silent sends' and building plans' buckets are
no longer asserted and are deleted once, each said as the event bucket-deleted (clause 13). The assigner may read
the compositions' digests, and a node-engine hears its backup holder's backed-up.
Each act that changed what a node runs now writes that node's compositions, and its wants follow by the assigner's
rules; a bus upgrade composes the bus's machines and answers the want the operator writes, since the assigner never
moves the bus (ADR 0539 clauses 5, 7 and 15). A node-engine may ask its own backup holder before a data module moves.
Every act that sent a declaration now writes compositions instead, by compare-and-set under the lease, and a
module's compositions are kept for its five newest fingerprints and whatever its want or have names (ADR 0539
clauses 1 to 5 and 14). A module the mesh provides itself, which no want can name, rides in the node's own.
mesh/check failed: shanks: novox/mesh-delivery#35: modules/mesh-delivery-assigner/go.mod names mesh-sdk bc0c25e51052, which is neither on mesh-sdk's t…
mesh/delivery-group group fix/1937-done-key-blobs rejected: composed together: the checker's verdict at the members' heads (dbd9c3771cd8): integration: github.…
chainBuildByID answered a trimmed build without SourceFingerprint since builds moved to the builds bucket (issue 1090), so
currentFingerprints read every fingerprint empty and grantsFrom skipped every module: mesh-controller_grants has never held
a key, and TestTheGrantOfAnIssuedCredentialIsOnTheBus failed on main.
mesh/delivery-group group fix/1937-done-key-blobs rejected: composed together: the checker's verdict at the members' heads (c20fdcb8c623): integration: github.…
A done key the controller never took in holds no manifest, and the store's nightly collector deleted its bare blob:
node-tools', mesh-host's and the forge bridge's keys name blobs that answer 404. Since issue 1933 the take-in refuses such a
key, and nothing built it again. held (in the serving process, HEADs only) lets the builder ask before it reuses a key; a
refused take-in now names every gone archive and says artifacts-gone with the module, key, fingerprint and references.
A sweep read what it keeps before what was ever recorded, so a take-in's
build recorded between the two reads, by a concurrent take-in off the
plans' hold, was eligible and let go of seconds after the take-in read
it; a new module's build is unkept until it is registered. Now the sweep
reads the records first and keeps what a build recorded within the hour
or an open plan names, and names every reference it lets go of; a take-in
holds every artifact in the store before recording, failing on one the
store lacks; and RecordBuild clears artifact_collected as RecordMirrored
does.
jschoubben