Commit Graph
2439 Commits
Author SHA1 Message Date
jschoubben 2407bfb9ff Compose kept compositions again when a module's config changes (issue 1945, ADR 0545)
The controller composed a module only at the fingerprint it is registered at,
so each of the up to five compositions kept for going back carried the config
of the moment it was made: a want back to an earlier mesh-build-go on g14 put
its capacity back to 0. Nothing composed a node when mesh-config's resolved
values changed, either.

Now a write of mesh-config_resolved (<node>.<module>.<key>), heard from that
bucket's watch and gathered for a second, drops the controller's 5 s reading
of the values, composes the node, and composes every kept fingerprint of the
touched modules on that node again from its own build's manifest with today's
inputs, writing only the compositions whose digest differs. A starting
controller does it once for every node, for a change no controller heard.
The two replaced-controller tests are one table test with the kept cases.
Research 072: milestone 2, the composer reacting to the change of an input.
2026-10-25 18:14:03 +01:00
jschoubben 850f4b4d26 Keep refused keys from holding the take-in's slots, and judge its stall by acknowledgements (issue 1945, ADR 0544, review)
A refused key stays unacknowledged until its delay ends, so with 8 a consumer
a handful refused for long (a manifest word this controller does not read
yet, a node that keeps failing) stopped that consumer for good, the build that
reads the word among what it held back. Each consumer now leaves up to 1000
unacknowledged; the store's bound is the one loop's batch of 16. A refusal
waits 30 s, doubling each time up to 10 minutes.

The stall watch judged movement by deliveries, which rise with every
redelivery: it now counts the keys this process acknowledged. A consumer that
cannot be made raises take-in-stalled instead of a log line. On stop the
gathered batch and a key in hand are refused with no delay, so the next
controller has them at once. A trunk value with no fingerprint of the new form
is said once, as every wait for a write.
Research 072: the take-ins on a bounded pull consumer.
2026-10-25 18:10:35 +01:00
jschoubben e7673e3bd3 Merge branch 'fix/1945-credential-on-assignment' into fix/1945-take-in-consumer 2026-10-25 18:07:30 +01:00
jschoubben 3e39ccf958 Let only a watch that read every standing assignment be the start's look (issue 1945, ADR 0543, review)
The flag that made the first watch the start's look was cleared after every
watch, even one that failed to open or ended before its Caught mark: the next
watch then handed every standing assignment as fresh, about 300 settles at once
with a catalogue read each, or dropped the keys the cut watch had handed. The
flag is now cleared by the start's look itself, and a watch cut before it opens
the next with nothing heard. Two rows of the existing table test cover it.
Research 072: the grants piece (memberships and credentials heard from placement).
2026-10-25 18:07:21 +01:00
jschoubben 94bb74966f Bound the take-in's two consumers to 16 keys in flight together, 8 each (ADR 0544 §1, its review)
Two consumers of 16 each allowed 32 take-ins on the store at once, twice
issue 1945's bound. A batch takes up to 16. Research 072: the take-ins on a
bounded pull consumer.
2026-10-25 18:01:18 +01:00
jschoubben bfb271a91b Take done keys in from a durable pull consumer, a batch at a time (issue 1945, ADR 0544)
The take-in watched the builder's and the indexer's buckets, started a follower
per module that retried on a 30 s timer, and at start took every catalogued
module in alphabetically: on 2026-10-25 a restart took in about 2 modules a
minute, about 190 followers queued behind 16 slots, and every take-in composed
two or three times even with nothing new, so a new build waited 30 minutes and
more for its credential.

Now two durable consumers named mesh-controller-take-in read the indexer's
modules bucket (every module's last trunk value when the consumer is made, then
each write) and the builder's builds bucket (each write), at most 16 keys
unacknowledged each. The bus keeps what is not taken in: the start's look is the
consumers' own catch-up, and a restart costs only what was left unacknowledged.
A batch reads the catalogue once, takes each module in once, and asks the
composer twice: once for its modules' nodes, once for the bus's node with the
grants. A module registered at its trunk whose compositions and grant exist, or
that this process took in at the same trunk, composes nothing; a key delivered
again is taken in whole. A wait for the builder's or indexer's next write is
acknowledged, a wait on a read or a failure refused with a 30 s delay. The
backlog is in status, and a backlog standing still for 5 minutes raises
take-in-stalled. The controller's account is granted the two consumers' acks.

The follower tests go with the followers: twelve tests become three.
Research 072: the take-ins on a bounded pull consumer (milestone 4, done keys
alone) and the start's look composing only what changed (milestone 2).
2026-10-25 17:59:20 +01:00
jschoubben 9ff47f6aaa Mint an assignment's account once, whoever finds it missing first (issue 1945, ADR 0543, review)
The assign act and the assignments' watch both checked whether an account was
minted and then minted it, and minting overwrites the hash: interleaved, the
bus kept one caller's hash and the node was sealed the other's password, the
crash loop this branch fixes. Both now mint through one statement that inserts
the row only when none stands; the caller whose insert wins seals, the other
seals nothing. issueMissingProcessAccounts mints the same way.

A watch opened again after one ended handed the standing assignments as the
start's look, so one that came to stand in the gap got no bus composition or
grant when its account already stood: only the process's first watch is the
start's look now. The start's look reads the catalogue once and ends with one
composition of the bus's node and the grants; a controller not holding the
lease is told so and tries again rather than dropping the key.

Tests folded to stay within the budget: the assignment tests are three table
tests, the process account and mayIssue tests folded into one of them, and
the concurrent mint is a case of the inventory's rotation test.
Research 072: the grants piece (memberships and credentials heard from placement).
2026-10-25 17:51:06 +01:00
jschoubben ca5f74eadb Issue a node's bus accounts and grant when its assignment comes to stand (issue 1945, ADR 0543)
ADR 0539 clause 7 issues a module's credential and grant only when its build is done.
A module built long before and assigned to one more node never has a build done again:
on 2026-10-25 at 17:19 mesh-build-node and mesh-build-go were assigned to ace and composed
for it, the bus's user list never named ace's accounts, and both runtimes restarted every
5 s (one with no credential, one refused by the bus). The assign act composes the assigned
node alone, never the node holding the bus, and writes no grant.

So a write of the assignments bucket is the event: when a key comes to stand (new, or
ended or pending before), the lease holder issues the module's accounts on that node that
are not minted (never re-issued), composes the node holding the bus and writes the grants,
through the composer the done keys' take-ins use. A rewrite of a standing assignment does
nothing; the start's look settles every standing one, one after another. A step of
research 072's grants (memberships and credentials heard from placement).
2026-10-25 17:37:03 +01:00
jschoubben c3db00ceac Merge pull request 'Compose the packages mirror only on a change, end every serve goroutine before serve returns, and raise no alarm for a cancellation (issue 1945)' (#464) from fix/r6-mirror-on-change-2 into main
Reviewed-on: #464
2026-10-25 16:20:55 +00:00
jschoubben 8e02c4726e Clear a module's fingerprint-not-composed on every successful take-in, and stop serve's goroutines before an error return undoes anything (issue 1945)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check pass 8e02c4726eb4 on c3db00ceac25 | unit 981 run, 171 kept, 2 skipped | integration 1641 run, 232 kept, 7 skipped | over bound: integration…
A success cleared the condition only when the same follower had raised it, so
one raised before a restart or by another follower stayed open after the take-in
worked, and the operator read that a build needed a look. Clearing a condition
that is not open costs one read. On an error return, serve now ends its
goroutines before the keeper closes and before handActConn and the bus are put
back (the review of fix 12).
2026-10-25 17:09:35 +01:00
jschoubben cc5c9ae888 Count the take-ins and every other background work in the serving group, so serve's stop waits for them too (issue 1945)
inBackground ran the take-ins of done keys (now in #463's slots), the start's
look, the given values' replacement and the announced reads beside serve without
serve waiting for them: a take-in could outlive its controller as the loops did.
It now takes the context and is counted in its serving group as well, and a
serve that is stopping starts none.
2026-10-25 16:36:49 +01:00
jschoubben c7e31640cf Raise no condition for a take-in, a layout or a put cut by a cancellation (issue 1945)
Thirty urgent fingerprint-not-composed conditions were live on 2026-10-25, each a
done key's take-in a stop or a restart cut with "context canceled". A
cancellation is no fault of the build, the chain or the bucket: a take-in or a
layout whose controller stops returns and the next one does it; one whose own
step was cancelled is tried again without a condition; a send cut by a
cancellation counts as no failure to put its files.

The put test now starts and leaves the node's failure count at nothing, so a
second run of it (-count=2) finds no count left by the first.
2026-10-25 16:30:58 +01:00
jschoubben c75d726205 End every goroutine serve starts before serve returns, so no loop outlives its controller (issue 1945)
The try of the on-change mirror failed with every test passed: goroutines a
served test started outlived its serve and met the next test's globals and its
closed stores. Under -race the served tests show it as a race on conditionsFrom
(watchTheMesh writing it while settlingPending, already running, read it), and
a take-in cut by the stop raised fingerprint-not-composed, a kind with no plain
words, which fails the package after its PASS line.

- serve counts every goroutine it and its parts start (serving_group.go) and
  ends and waits for them before anything else is undone: the lease's watch,
  the pending settler, the status summary, the facts, the mirrors and the new
  declared-packages keeper with its two bucket watches, the declaration sweep,
  the read-call counter, the watches' closers, the have watches, the vault move
  on a report, and the watchdogs, doctor, healers and asker under their own
  group.
- The condition store opens before anything that reads conditionsFrom starts;
  askerFrom is an atomic pointer, read from the keeper's goroutines.
- A done key's take-in cut by the stop raises nothing, and fingerprint-not-composed
  has plain words.
2026-10-25 16:30:58 +01:00
jschoubben a4cd5ba375 Compose the declared-packages mirror only when what it composes from changed (issue 1945)
The mirror composed every node in full on the record mirrors' 10 s tick; a
pass takes about 45 s on the control node, so the controller never rested
(130-175 % CPU, issue 1945 note 22). It now passes on a watch of the
compositions' digests and of the assignments, one pass at a time with the
wakes of a pass coalesced into one after it (at least 30 s apart), skips the
composition when the stores' Input digest is unchanged, and keeps a safety
pass every 10 minutes that composes nothing when nothing changed.
2026-10-25 16:30:58 +01:00
jschoubben 62802cd966 Merge pull request 'Share the controller's stores and take in at most 16 done keys at once, so a burst of builds cannot fill the store (issue 1277, issue 1945)' (#463) from fix/store-pool-flood into main
Reviewed-on: #463
2026-10-25 15:29:58 +00:00
jschoubben e9a885cda9 Share the controller's stores with its take-ins and cap take-ins at 16
mesh/check checking e9a885cda925 on 62802cd96602
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
At 16:07:36 the builder rewrote about 120 done keys at once. Each woke a
take-in of its own with no limit, and each take-in read the rest of the
mesh through restOfTheMeshOf, which opened a licences and an identity
pool per call. novox's store ran out of connections at 16:08 and refused
everyone, the controller's retries keeping it full.

- The serving controller lends its stores by inventory (shareStores);
  restOfTheMeshOf reads on them instead of opening pools per call, and a
  verb run as its own process still opens its own. stores opens each
  store once under a lock, since readers now share it. serve uses the
  shared identity store instead of a second pool.
- The watch's take-ins and the start's look share one cap of 16 take-ins
  at once (takeInSlots); a take-in waiting for its next try holds none.
- store.Ready backs off on a full store (SQLSTATE 53300), doubling its
  wait up to 5 s, and says once that the store is full.

Issues 1945 and 1277.
2026-10-25 16:23:55 +01:00
mesh-tools 2814262ac0 Merge pull request 'Keep the declarations retired once their stream is gone, so the bus is not called unhealthy for a stream release 6 deleted (issue 1945)' (#462) from fix/r6-nodes-stream-gone into main 2026-10-25 14:25:11 +00:00
jschoubben 0a536d7b6d Keep the declarations retired once their stream is gone, so nothing expects, asserts or makes NODES again (issue 1945)
mesh/check pass 0a536d7b6d79 on 2bcd85d2df4c | unit 966 run, 172 kept, 2 skipped | integration 1622 run, 236 kept, 7 skipped | over bound: integration…
mesh/delivery merged: source.main moved to novox/mesh-controller@2814262ac0fa
A node-engine newer than the registered one, or a node asleep on an older
have, made the done test false after the start deleted NODES; the bus's
health check, probes D6 and D7 and healer H3 then expected the stream and
every node's consumer again, raised them urgent and made them anew. Release
6 sends no declaration, so a stream made again helps no node: a gone stream
is retired. The advisory watchdog also reads a consumer whose stream is gone
as gone instead of failing.
2026-10-25 15:01:09 +01:00
mesh-tools 2bcd85d2df Merge pull request 'Never put a module whose want names its own composition into the node's own, and read a re-registered module's fingerprint from the indexer (issue 1945)' (#461) from fix/r6-own-composition-dupes into main 2026-10-25 13:37:34 +00:00
jschoubben ac34bce49f Take a module's fingerprint from the indexer when its commit has no build, and never compose a module its want names into the node's own (issue 1945)
mesh/check pass ac34bce49f58 on 2bcd85d2df4c | unit 966 run, 171 kept, 2 skipped | integration 1625 run, 232 kept, 7 skipped | over bound: integration…
mesh/delivery merged: source.main moved to novox/mesh-controller@2bcd85d2df4c
A catalogue merge that leaves a module's sources alone registers it at the new
commit from the same done key, whose build record stands at its first commit,
so the commit had no build and the module no fingerprint of the new form: it
went into the node's own composition while its want named its own, and every
node declared its files twice. currentFingerprints now reads the indexer's
trunk value at the commit when no build points at it (1945/AC15), and a module
a want or have names by its own composition is left out of the node's own
(1945/AC16).
2026-10-25 14:10:51 +01:00
mesh-tools af2b0844a1 Merge pull request 'Compose each machine once per pass, so take-ins of done builds stop colliding and every new build gets its composition and credential (issue 1945)' (#460) from fix/r6-compose-once into main 2026-10-25 12:30:32 +00:00
jschoubben 3b4cb69e84 Take again a replacement of given values asked while one ran on the node (issue 1945)
mesh/check pass 3b4cb69e84dd on af2b0844a1fc | unit 963 run, 171 kept, 2 skipped | integration 1622 run, 232 kept, 7 skipped | over bound: integration…
mesh/delivery merged: source.main moved to novox/mesh-controller@af2b0844a1fc
2026-10-25 13:02:49 +01:00
jschoubben c47a590463 The start pass reads a module's look-again mark after letting go, and hands it on to takeDoneKey (issue 1945) 2026-10-25 13:00:43 +01:00
jschoubben f35aa35541 Look at a module's mark again after letting go of it, so a wake just before is taken (issue 1945) 2026-10-25 12:55:33 +01:00
jschoubben 8114cee2af Give each caller of a pass its own nodes' errors and write the grants whenever asked; a wake during a take-in makes it look once more (issue 1945)
mesh/check pass 8114cee2af47 on 0fff3efd4e0b | unit 959 run, 172 kept, 2 skipped | integration 1615 run, 236 kept, 7 skipped | over bound: integration…
mesh/delivery closed: a newer head of the same pull request
2026-10-25 12:44:28 +01:00
jschoubben baa8f58473 Compose each node once per pass in one composer for the done keys' take-ins, one composition of a node at a time, and the start's look side by side (issue 1945) 2026-10-25 12:39:59 +01:00
jschoubben 0470869fbb Keep the digest of the composition the bucket holds, following another writer's composition written in between (issue 1945) 2026-10-25 12:36:00 +01:00
jschoubben 8a3c0cd022 Say each wait of a done key's take-in once with its reason and look again every 30 s, one follower per module (issue 1945) 2026-10-25 12:35:42 +01:00
mesh-tools 0fff3efd4e Merge pull request 'Register every done key once at the start, and write a node's own composition last (issue 1945)' (#459) from fix/r6-fp2-seen into main 2026-10-25 04:21:27 +00:00
jschoubben eafd1883f6 Write a node's own composition last, and not in a pass where a module's composition failed (issue 1945)
mesh/check pass eafd1883f65c on 0fff3efd4e0b | unit 953 run, 171 kept, 2 skipped | integration 1612 run, 232 kept, 7 skipped | over bound: integration…
mesh/delivery merged: source.main moved to novox/mesh-controller@0fff3efd4e0b
2026-10-25 02:57:24 +02:00
jschoubben e67916bbd8 Take every catalogued module's done key once at the start, beside the updates-only watch, so a key written or interrupted across a restart is registered (issue 1945) 2026-10-25 02:44:50 +02:00
jschoubben 7039c1b4b3 Merge pull request 'Keep moved modules' filled paths in the node's own composition, put every file object, grant access-act and the provisioner's have read (issue 1944)' (#458) from fix/r6-node-composition into main
Reviewed-on: #458
2026-10-25 00:17:42 +00:00
jschoubben 3dadc46ee6 Grant each provisioner client a read of its node's own have, so it refills when the node's own composition moves (issue 1945)
mesh/delivery checking
mesh/check checking 3dadc46ee6cc on ba4d15657343 | novox/mesh-controller#458 in group fix/r6-node-composition
mesh/delivery-group group fix/r6-node-composition held: a member is held: novox/mesh-sdk@10b6e202320f (merged without a passing check: only a person decides wh…
2026-10-25 01:58:27 +02:00
jschoubben d8aa565fee Keep the filled paths of the modules moved into a node's own composition, which the node-engine otherwise wrote empty (issue 1945) 2026-10-25 01:47:57 +02:00
jschoubben 82c89bec65 Split compositions from the inline body so their large files' objects are put, and refuse a composition naming an object the bucket lacks (issue 1945) 2026-10-25 01:42:28 +02:00
jschoubben b1f758ed71 Grant each node-engine its own access-act ask, which it has subscribed to since ADR 0325 and was refused at every connect (issue 1945) 2026-10-25 01:42:28 +02:00
mesh-tools ba4d156573 Merge pull request 'R6: the controller writes signed per-node compositions and their digests; plans, pushes and their verbs go (issue 1945)' (#457) from feat/r6 into main 2026-10-24 23:00:30 +00:00
jschoubben 9c38aed789 Read the declaration a served-key test checks from the composition, since the plan verb is gone (issue 1945)
mesh/delivery-group group feat/r6 merged: every member is merged
mesh/check checking 9c38aed78905 on bdb8193456e1 | novox/mesh-controller#457 in group feat/r6
mesh/delivery delivering: 0 machine step(s) passed
2026-10-25 00:06:06 +02:00
jschoubben a3f1d0460d Wait for the whole start pass before serving a served test, so a rotation test never reads it midway (issue 1945) 2026-10-24 23:48:51 +02:00
jschoubben 0af12b369b Wait for the done key's grant, which the take-in writes after the compositions (issue 1945) 2026-10-24 23:45:33 +02:00
jschoubben e338b1f0f4 Re-capture the converged declaration for release 6: every resource says its module (issue 1945)
readNode sets ReadsModules, since release 6 composes only for node-engines that read a module per resource (issue 1145);
the failure says the bytes on its first line, the one the verdict keeps (issue 1921).
2026-10-24 23:21:28 +02:00
jschoubben e42818a0ab Merge main (the R4g finish: reads-outside markers) into feat/r6 (issue 1945)
The silent mid-tier plan test main changed stays deleted with the plans (ADR 0539); the SDK is pinned to the feat/r6
head that merged mesh-sdk's main, 748c564c, whose hand path tool is now mesh-hand-compose.
2026-10-24 23:04:56 +02:00
jschoubben bdb8193456 Merge pull request 'R4g finish: reads-outside markers (issue 1935)' (#454) from feat/r4g-finish into main
Reviewed-on: #454
2026-10-24 20:57:07 +00:00
jschoubben ca25e4ab45 Give back everything serve sets once the served tests' controller stops (issue 1945)
The served mark in the environment made every later test in the process read as the serving controller's: the
registration test was refused as "never through a verb". The fixture now restores the mark, the hand-act log's
connection, the keeper, the status and the event publishers serve sets.
2026-10-24 22:54:00 +02:00
jschoubben 7a64075702 Let a later test's verb run after the served controller stops (issue 1945)
serve marks the process as handing over once it stops, and refuses every verb from then on; the served tests run it in
the test process, so the connection test's conditions verb was refused without its own process ever starting.
2026-10-24 22:44:12 +02:00
jschoubben 9abe0b59fa Give back the connection name a test's in-process command line set (issue 1945)
runLine runs the controller's dispatch in the test process, which names every later connection after the line's verb:
the proposals tests left "mesh-controller verb settings", and the conditions verb's test later in the same process
read that name instead of its own.
2026-10-24 22:37:46 +02:00
jschoubben 183d7a99ff Assert the controller's buckets in the builder-on-the-bus fixture (issue 1945)
A merge the controller hears is now kept in the bucket of the merges (ADR 0539 §17); the test that sends one passed
only when an earlier test on the shared bus had made that bucket.
2026-10-24 22:23:11 +02:00
jschoubben fad3d7ee71 Let go of the hand-act log's connection when serving ends (issue 1945)
A controller served in the test process left it pointing at a closed connection, so the next test's conditions verb
failed with "connection closed".
2026-10-24 22:13:52 +02:00
jschoubben 3c50e80972 Merge main (#456, the pre-switch grants) into feat/r6 (issue 1945)
Every grant #456 gives was already feat/r6's, by the SDK's names: each conflict takes feat/r6's side, with main's
test of the user list before the switch kept beside feat/r6's, and the duplicate assertion of the digests' bucket
dropped.
2026-10-24 22:02:32 +02:00
jschoubben 6e4bca7fce Run the served rotation's verb on the bus chain the controller keeps, as its child process does (issue 1945) 2026-10-24 21:48:30 +02:00