Commit Graph
243 Commits
Author SHA1 Message Date
jschoubben e40b1e70db Follow config from a durable consumer, compose again only what changed since, and hold the node only for a write (issue 1945, ADR 0545, review)
The start composed every kept composition of every module on every node, each a
whole node composed in memory under the node's lock: 10 to 50 minutes at a
start, holding take-ins back. Now the writes of mesh-config_resolved are read
from the durable consumer mesh-controller-config-follow, acknowledged once the
change is composed, so a restart composes again only what was written since
the last change handled, and a first start nothing it did not hear. The node's
lock is taken for each write alone. A consumer that cannot be made or read
raises config-not-followed and is tried again a minute later, cleared once it
reads. The controller's account is granted that consumer's acknowledgements
where mesh-config serves, as its verb is.
Research 072: milestone 2, the composer reacting to the change of an input.
2026-10-25 18:27:43 +01:00
jschoubben bfb271a91b Take done keys in from a durable pull consumer, a batch at a time (issue 1945, ADR 0544)
The take-in watched the builder's and the indexer's buckets, started a follower
per module that retried on a 30 s timer, and at start took every catalogued
module in alphabetically: on 2026-10-25 a restart took in about 2 modules a
minute, about 190 followers queued behind 16 slots, and every take-in composed
two or three times even with nothing new, so a new build waited 30 minutes and
more for its credential.

Now two durable consumers named mesh-controller-take-in read the indexer's
modules bucket (every module's last trunk value when the consumer is made, then
each write) and the builder's builds bucket (each write), at most 16 keys
unacknowledged each. The bus keeps what is not taken in: the start's look is the
consumers' own catch-up, and a restart costs only what was left unacknowledged.
A batch reads the catalogue once, takes each module in once, and asks the
composer twice: once for its modules' nodes, once for the bus's node with the
grants. A module registered at its trunk whose compositions and grant exist, or
that this process took in at the same trunk, composes nothing; a key delivered
again is taken in whole. A wait for the builder's or indexer's next write is
acknowledged, a wait on a read or a failure refused with a 30 s delay. The
backlog is in status, and a backlog standing still for 5 minutes raises
take-in-stalled. The controller's account is granted the two consumers' acks.

The follower tests go with the followers: twelve tests become three.
Research 072: the take-ins on a bounded pull consumer (milestone 4, done keys
alone) and the start's look composing only what changed (milestone 2).
2026-10-25 17:59:20 +01:00
jschoubben 0a536d7b6d Keep the declarations retired once their stream is gone, so nothing expects, asserts or makes NODES again (issue 1945)
mesh/check pass 0a536d7b6d79 on 2bcd85d2df4c | unit 966 run, 172 kept, 2 skipped | integration 1622 run, 236 kept, 7 skipped | over bound: integration…
mesh/delivery merged: source.main moved to novox/mesh-controller@2814262ac0fa
A node-engine newer than the registered one, or a node asleep on an older
have, made the done test false after the start deleted NODES; the bus's
health check, probes D6 and D7 and healer H3 then expected the stream and
every node's consumer again, raised them urgent and made them anew. Release
6 sends no declaration, so a stream made again helps no node: a gone stream
is retired. The advisory watchdog also reads a consumer whose stream is gone
as gone instead of failing.
2026-10-25 15:01:09 +01:00
jschoubben 3dadc46ee6 Grant each provisioner client a read of its node's own have, so it refills when the node's own composition moves (issue 1945)
mesh/check checking 3dadc46ee6cc on ba4d15657343 | novox/mesh-controller#458 in group fix/r6-node-composition
mesh/delivery checking
mesh/delivery-group group fix/r6-node-composition held: a member is held: novox/mesh-sdk@10b6e202320f (merged without a passing check: only a person decides wh…
2026-10-25 01:58:27 +02:00
jschoubben b1f758ed71 Grant each node-engine its own access-act ask, which it has subscribed to since ADR 0325 and was refused at every connect (issue 1945) 2026-10-25 01:42:28 +02:00
jschoubben e42818a0ab Merge main (the R4g finish: reads-outside markers) into feat/r6 (issue 1945)
The silent mid-tier plan test main changed stays deleted with the plans (ADR 0539); the SDK is pinned to the feat/r6
head that merged mesh-sdk's main, 748c564c, whose hand path tool is now mesh-hand-compose.
2026-10-24 23:04:56 +02:00
jschoubben bdb8193456 Merge pull request 'R4g finish: reads-outside markers (issue 1935)' (#454) from feat/r4g-finish into main
Reviewed-on: #454
2026-10-24 20:57:07 +00:00
jschoubben 3c50e80972 Merge main (#456, the pre-switch grants) into feat/r6 (issue 1945)
Every grant #456 gives was already feat/r6's, by the SDK's names: each conflict takes feat/r6's side, with main's
test of the user list before the switch kept beside feat/r6's, and the duplicate assertion of the digests' bucket
dropped.
2026-10-24 22:02:32 +02:00
jschoubben dfd81d8daf Re-capture the catalogue at mesh-catalog feat/r6 3a1017f5 and excuse a derived subject whose module lives in another repository (issue 1945) 2026-10-24 20:37:04 +02:00
jschoubben f20d221a60 Return a grant not written and retry the take-in under a condition; drop the plans' events; grant bucket-deleted, restic's backup word and the operator's hand path (issue 1945; the review's D1, F1, H1 and its notes)
With no send left nothing else writes a missed grant, so the take-in of a done key returns it and tries again every
half minute under an urgent condition until it is written. plan-moved and sent go with the plans. The controller's
first-start deletions are said on a granted subject, a node hears only its backup holder, and the operator's account
writes the hand path, an object replaced included (ADR 0539 clauses 2, 7, 10 and 13).
2026-10-24 18:30:24 +02:00
jschoubben bd8ea8ad4f Format the hand path's live test (issue 1945)
mesh/delivery delivered
mesh/check failed: integration: github.com/novox/mesh-controller/cmd/mesh-controller.TestAPlanPassesATierWhoseMachineNeverReportsAfterTheBoundAndSends…
2026-10-24 18:25:54 +02:00
jschoubben 1350c428ed Let the operator's hand path replace a file over 64 KiB in a node's object bucket (issue 1945; the review of A1, H1)
The object store reads an object's metadata and purges its old chunks when it replaces one, so the hand path is
granted the direct and message gets and the purge of each node's object stream; a test on a bus running the composed
user list walks the hand path end to end.
2026-10-24 18:25:46 +02:00
jschoubben 7f8e5a821c Grant the controller's bucket-deleted, narrow a node's backup word to restic's, and grant the operator release 6's hand path ahead of the switch (issue 1945; the review of A1)
The new controller says its first start's deletions as an event, a node-engine hears only its backup holder, and the
way back right after the switch, a want and a composition written at the terminal, runs on the user list today's
controller composes (ADR 0539 clauses 2, 10 and 13). The operator is the administrator person, the one second
writer ADR 0539 names.
2026-10-24 18:21:58 +02:00
jschoubben 3ae2d6e17c Grant release 6's compositions, wants and backups ahead of its switch (issue 1945)
The switch to release 6 is today's controller's last plan, and the bus runs the user list it composes until the new
controller writes one: so the new controller's writes of each node's compositions and their digests, each new
node-engine's reads of its own compositions and wants and its backup holder's verbs and word, and the assigner's read
of the digests must be granted by this controller already (ADR 0539 clauses 2, 6, 9 and 16; the review's A1).
2026-10-24 18:14:45 +02:00
jschoubben b4a62e9e6e Delete the declarations' stream once every node runs the new node-engine, and assert it no more (issue 1945, B13)
A node away during the switch finds its last declaration on the stream; once every node's have names the
node-engine registered, no node reads one, so the stream and the nodes' consumers on it go, said once, and a raise
leaves them out (ADR 0539 clause 11).
2026-10-24 17:28:10 +02:00
jschoubben 66650e315c Take a registered module's new done key in, compose it and grant it; at start delete the plans' buckets and compose every node and the replaced controller (issue 1945, B3, B5, B13)
With no plan left to take a done key in, a write of the builder's keys registers the new fingerprint, composes the
module's nodes and the bus's user list, and writes its grant, so the assigner's rules can name it (ADR 0539 clauses
5 and 7). The new controller composes every node at its start, and the controller it replaced at its own
fingerprint, the operator's way back (clause 10); the merge window's, silent sends' and building plans' buckets are
no longer asserted and are deleted once, each said as the event bucket-deleted (clause 13). The assigner may read
the compositions' digests, and a node-engine hears its backup holder's backed-up.
2026-10-24 17:11:01 +02:00
jschoubben c40e15268e Compose where a declaration was sent: secrets, credentials, acceptances, the hub's peers, adoption, assignments and the bus's user list (issue 1945, B2, B6)
Each act that changed what a node runs now writes that node's compositions, and its wants follow by the assigner's
rules; a bus upgrade composes the bus's machines and answers the want the operator writes, since the assigner never
moves the bus (ADR 0539 clauses 5, 7 and 15). A node-engine may ask its own backup holder before a data module moves.
2026-10-24 16:59:06 +02:00
jschoubben aee0a2718f Write each module's composition per node, signed, with its digest and the move's judgement beside it (issue 1945, B1)
Every act that sent a declaration now writes compositions instead, by compare-and-set under the lease, and a
module's compositions are kept for its five newest fingerprints and whatever its want or have names (ADR 0539
clauses 1 to 5 and 14). A module the mesh provides itself, which no want can name, rides in the node's own.
2026-10-24 16:54:11 +02:00
jschoubben dd2d8b91be Merge main into feat/r4g-finish (#453)
mesh/delivery-group group feat/r4g-finish rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: shanks: novox/mesh-delivery#35: modules/mesh-delivery-assigner/go.mod names mesh-sdk bc0c25e51052, which is neither on mesh-sdk's t…
2026-10-24 13:26:24 +02:00
jschoubben 8831c74f70 Mark the packages whose tests read files outside their directory, found by the review (issue 1935) 2026-10-24 04:38:51 +02:00
jschoubben 3b51d3f86e Answer held and say artifacts-gone, so the builder rebuilds a done key whose blob the artifact store no longer holds (issue 1937)
mesh/delivery-group group fix/1937-done-key-blobs rejected: composed together: the checker's verdict at the members' heads (c20fdcb8c623): integration: github.…
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: integration: github.com/novox/mesh-controller/cmd/mesh-controller.TestAWarrantForRetryRetriesAFailedPlanInTheStoreAndEndsItsConditi…
A done key the controller never took in holds no manifest, and the store's nightly collector deleted its bare blob:
node-tools', mesh-host's and the forge bridge's keys name blobs that answer 404. Since issue 1933 the take-in refuses such a
key, and nothing built it again. held (in the serving process, HEADs only) lets the builder ask before it reuses a key; a
refused take-in now names every gone archive and says artifacts-gone with the module, key, fingerprint and references.
2026-10-24 04:12:02 +02:00
jschoubben c9a7102bc5 Set aside a retired seat's consume the trunk catalogue still holds, and say it, so a group that retires a seat passes its check
mesh/delivery failed: its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine
mesh/check checking c9a7102bc53b on 791786d5b87f | novox/mesh-controller#448 in group feat/r4g
mesh/delivery-group group feat/r4g merged: every member is merged
2026-10-23 18:05:06 +02:00
jschoubben 13842a7234 Grant the builder the build seats' streams' information and messages by name, so its take-over at start can tell an ask its key names is on its stream (issue 1910)
mesh/delivery-group group feat/r4g rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: g14: novox/mesh-catalog#464: checks/health/go.mod names mesh-sdk a9560890ee26, which is neither on mesh-sdk's trunk nor the group's…
2026-10-23 16:59:03 +02:00
jschoubben 758c8e84e5 Bound the build seats' workers' deliveries, so an ask no run ends is offered at most five times even when the holder that says it is poison dies too (issue 1907) 2026-10-23 16:42:13 +02:00
jschoubben ff543d3def Take the watchman's grant on the build seats back: a waiting ask has no pulse, a taken one is read from the steps (issue 1908) 2026-10-23 16:35:28 +02:00
jschoubben 3c247e80af Fix the view grant test's consumer subjects (issue 1887 A8) 2026-10-23 16:32:43 +02:00
jschoubben bb0d5e17db Grant the watchman the build seats' accept subjects and the view the seat workers' information, each by name, so the backlog and the waiting-ask pulse read live (issue 1887 A8) 2026-10-23 16:32:32 +02:00
jschoubben 5b5cd6160a Merge branch 'feat/r4g-asks' into feat/r4g 2026-10-23 16:20:32 +02:00
jschoubben df70ca43a7 Take the controller's publish grants on node-build-agent and mesh-build-machine away, with the cancelled sets and the redelivery of its own asks: it asks no build seat (ADR 0525 §10, C3a) 2026-10-23 16:13:43 +02:00
jschoubben eb8286fc68 Merge feat/r4g-assigner into feat/r4g: lane B's grants bucket and its writer beside lane C's deletions
The controller's readable states, its bucket lists and the composed golden file keep lane C's removal of the
checks-running and plans-building buckets and the build agent's check subjects, and add lane B's grants bucket.
2026-10-23 15:59:52 +02:00
jschoubben e419d0182f Delete the builder, the build asks, the build queue and the merge gate: the builder's keys are the only way a module is built (issues 1851, 1299, 1318; C2b-C5, C10)
The controller asks no build of anybody (ADR 0501 §5, ADR 0525 clause 6, ADR 0503 §3). Every module of a walk is taken from the
builder's done key; a key not done yet waits, named in the plan, and never fails it.

- C2b/C2c: internal/builder and cmd/mesh-controller's asks go (askABuild, onlyTheBuildersOwn, buildOneAsked, takeIn from an
  outcome, buildAgentsOwn, the builder's say and its fallback, the daemon hearing an outcome). The server follows the retired
  seats' built subjects only to let go of them, because the installer's carried user list names them.
- C2d: replay, queue, cancel, clear, kill, pause, resume, delivery-check, check-here and merge-gate are refused by name.
- C3: the check queues (CHECKS_READY, CHECKS_OTHER) and their workers, the checks-running and plans-building buckets and
  their grants, the build queue and its hold (plans_building.go, queue.go, link/holds.go, builds_nats.go, checks_nats.go).
- C4: merge_gate*.go, the pull request's repo-check path, the Checked event and its layers; a pull request asks the
  controller nothing but to tell mesh-config what its head declares.
- C5: the builds bucket is written only by key_takein.
- C10 (help and refusals): the terminal's help lists only what is served.

Integration tests of the plan's tiers, sends, stops and retries now run on a stand-in builder with a done key for every
module (walk_keys_scaffold_integration_test.go); the tests of the asks, the queue, the merge gate and the replays are deleted.
2026-10-23 15:55:15 +02:00
jschoubben 5ca4f14479 Derive a worker per kind for the build seats and grant their users to put asks, so a build node pulls delivery, check and recheck in order and a session cannot ask (issue 1887 A6, ADR 0525 §2) 2026-10-23 15:54:45 +02:00
jschoubben 548681cc4e Declare the controller's grants bucket, written by the controller alone and readable through mesh-controller.grants, so the assigner can read its sixth rule's key (issue 1394, subtask C9a; ADR 0528 clause 2); the SDK pin moves to the layouts' commit, captured beside 2026-10-23 15:39:40 +02:00
jschoubben a0b2d98002 Merge main (upgrade safety's declared packages, issue 1840) and pin the R4e mesh-sdk member's final head 875c680d, re-vendored here and in modules/route-proxy; the controller announces declared-packages readable (issue 1790, D1; E) 2026-10-22 20:44:59 +02:00
jschoubben d36e0ea278 Announce the controller's states, keep every module's word, and grant a read only where its owner announced the state readable; the copied list of the controller's mirrors goes (issue 1790, D1-D4, ADR 0495, ADR 0513 §3) 2026-10-22 19:05:04 +02:00
jschoubben b6ed18197b Vendor the R4e mesh-sdk member's head with the states announcement (issue 1790; re-pinned to the final head in E) 2026-10-22 19:05:04 +02:00
jschoubben b3786cc67e Merge remote-tracking branch 'origin/main' into feat/upgrade-safety
# Conflicts:
#	go.mod
#	go.sum
#	testdata/beside/CAPTURED
#	vendor/modules.txt
2026-10-22 14:25:33 +02:00
jschoubben 21954ad1bc issue 1840/S3b: prove a process may watch its own state's bucket and no other (AC9) 2026-10-22 14:23:58 +02:00
jschoubben 485eb44de4 issue 1840/S3: keep each node's declared packages on the bus, so the pacman verbs can say what a node declares 2026-10-22 14:23:58 +02:00
jschoubben 499ff5c218 Move the route proxy into its module and run it from the trust image, on mesh-sdk's public membership (issues 1633, 1634, 1635)
mesh/repo-check out of date: judged beside mesh-catalog#434 at c948f3a61c74, it is now at 7451e7edf4e5; judged again on a recheck or when it is merged
mesh/merge-gate group feat/r4d: composed again for the group's new heads
mesh/delivery superseded: a newer head of the same pull request
mesh/delivery-group group feat/r4d rejected: a member's own check failed
mesh/check failed: unit: i3/cmd/i3-tools.TestTheMainFileAndEveryModulesContributionLoadTogether: manifest_test.go:247: expected the launcher, clipboar…
A bundle is built only from its module's directory (ADR 0480 clause 4), so the proxy leaves
examples/ and the controller's internal/broker; the broker keeps no second copy of what the
proxy shares with mesh-sdk. The image and its Dockerfile go, and a test holds that only
build-agent declares one (clause 11).
2026-10-22 09:53:56 +02:00
jschoubben b9ec1919b8 Name plans-building among the controller's readable states and pin the SDK that does (issue 1687, ADR 0483 clause 2)
mesh/repo-check head b9ec1919b865 merged into trunk a3c965b9a0d5 beside mesh-catalog#423@12ff1cfb — pass: its merge-check.sh passed
mesh/merge-gate head b9ec1919b865 merged into trunk a3c965b9a0d5 — pass: group feat/the-checker composed: every machine composes with the change as it did …
mesh/delivery delivered
mesh/delivery-group group feat/the-checker merged: every member is merged
2026-10-21 22:47:04 +02:00
jschoubben 79e2923dc8 Merge remote-tracking branch 'origin/fix/1652-checks-wait' into feat/the-checker
# Conflicts:
#	internal/broker/testdata/composed.conf
#	internal/link/builds_nats.go
2026-10-21 22:01:42 +02:00
jschoubben b62fbeabbf Ask a held move and the builds held after a failed release, and say them on the plan's line (issue 1732, 1320) 2026-10-21 19:43:36 +02:00
jschoubben a14dc04b7b Hold the build seat's checks while a delivery builds, and let one check that waited 30 minutes through
ADR 0483: the controller keeps a key per plan in its build phase; holders read it and
take no new check while one is there, but for the oldest check past 30 minutes, one
at a time by a let-through key. The queue says why checks wait.
2026-10-21 19:32:47 +02:00
jschoubben a3b4e0601d Take mesh-sdk at 86f9a1a5 and name each node's bucket with its FileBucketFor, refusing a node name that could carry a wildcard (issues 1696, 1697; ADR 0488) 2026-10-21 16:05:24 +02:00
jschoubben da364c2730 Give each node a bucket of its own for the files that travel by digest, and keep the sweep from removing an object a send relies on (issues 1696, 1697; review of ADR 0486 step 3)
A literal * in a published subject matches the grant's *, so in one shared bucket a node could read every node's files. Each node now
has declaration-files-<node>, which its account reads by name and the controller writes; the sweep judges each bucket against that
node's latest declaration, under the node's lock with the send, and an object the sweep marked is put again by the next send. Putting
that fails send after send raises a condition.
2026-10-21 15:43:36 +02:00
jschoubben ceaa616d01 Send files over 64 KiB by digest through the bus's object bucket to nodes that read them, grant it, and warn at 75 percent (issues 1696, 1697, 1698; ADR 0486 step 3)
A node that reports file-references gets the digest and size of each file over 64 KiB that may travel so, and the content is put in
the bucket declaration-files before the declaration is sent; others get every content inline. The controller alone writes the
bucket and node-engines read an object by name (the four grants the node-engine's own fetch makes, no list, no write). An object
goes a day after no node's latest declaration names it. The near-limit condition warns at 75 percent, and a test holds the control
node's declaration, in the reference form, under half the bus's largest message.
2026-10-21 15:16:11 +02:00
mesh-tools 233db89f29 Count only the controller's own consumers in the deaf look, not the check workers a builder holds (issue 1063) 2026-10-20 23:14:45 +00:00
mesh-tools e7c6abfd0e Test the grants of the bridge's file read and the indexer's manifests bucket (issues 1448, 1449, design 47 R4c-2)
mesh/repo-check head e7c6abfd0e3c merged into trunk f2d7bc94800d — pass: its merge-check.sh passed
mesh/merge-gate head e7c6abfd0e3c merged into trunk f2d7bc94800d — pass: group feat/r4c2-new-keys composed: every machine composes with the change as it di…
mesh/delivery delivered
mesh/delivery-group group feat/r4c2-new-keys merged: every member is merged
2026-10-20 18:56:37 +00:00
mesh-tools bfe7fe1530 Pin mesh-sdk at 14fd5cb6 (R4a's merge) and take the catalogue, overlay and validator from it, s to inventory (issue 1381, 4 of 6 import swaps)
go.mod and go.sum move to mesh-sdk 14fd5cb68fd1, the commit on its main that holds go/catalogue, go/overlay and go/validate (ADR 0453 clause 2); vendor/modules.txt follows in the last of the six commits.
2026-10-20 08:26:01 +00:00